Oracle RAC Administration – Part 9: Hands on administration

Brief intro


As a continuing part of the hands-on articles, we will take a deeper look at such things as errors in the installation (which are actually not errors), ESX host tuning for time synchronization (without which the whole RHEL RAC installation means nothing) and some SRVCTL commands.


Errors come and errors go


Let’s take a look at a couple of screen shots and some of the typical Cluster Ready Services (CRS) errors and tricks to bring your RAC services and applications online.


This is a typical placement error that I get on every installation. I think it may have to do with the time issue; we will come to that. If you are using an ESX server to test/develop your RAC, then the information to test and fix your time synchronization issues will certainly come very handy.




It really doesn’t mean a thing. I do run into this every time I do my virtual machine restart.

[[email protected] ~]$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
————————————————————
ora….SM1.asm application ONLINE UNKNOWN vm1rh4
ora….H4.lsnr application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.gsd application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.ons application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.vip application ONLINE ONLINE vm1rh4
ora….SM2.asm application ONLINE UNKNOWN vm2rh4
ora….H4.lsnr application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.gsd application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.ons application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.vip application ONLINE ONLINE vm2rh4
ora….SM3.asm application ONLINE OFFLINE
ora….H4.lsnr application ONLINE OFFLINE
ora.vm3rh4.gsd application ONLINE UNKNOWN vm3rh4
ora.vm3rh4.ons application ONLINE UNKNOWN vm3rh4
ora.vm3rh4.vip application ONLINE UNKNOWN vm1rh4
ora….SM4.asm application ONLINE OFFLINE
ora….H4.lsnr application ONLINE OFFLINE
ora.vm4rh4.gsd application ONLINE ONLINE vm4rh4
ora.vm4rh4.ons application ONLINE ONLINE vm4rh4
ora.vm4rh4.vip application ONLINE UNKNOWN vm4rh4

This is really a VM issue. I am using a 1.2 G vMEM, 2vCPUs, and decent SCSI Virtual Disks (VMFS-2 file system on a VMDK format). Restarting the services does not go as expected. Let’s see what happens if you try to do a crs_stat –stop and then a crs_stop –start.

[[email protected] ~]$ $ORA_CRS_HOME/bin/crs_stop -all
Attempting to stop `ora.vm4rh4.gsd` on member `vm4rh4`
Attempting to stop `ora.vm4rh4.ons` on member `vm4rh4`
Stop of `ora.vm4rh4.gsd` on member `vm4rh4` succeeded.
Stop of `ora.vm4rh4.ons` on member `vm4rh4` succeeded.
Attempting to stop `ora.vm2rh4.LISTENER_VM2RH4.lsnr` on member `vm2rh4`
Attempting to stop `ora.vm1rh4.LISTENER_VM1RH4.lsnr` on member `vm1rh4`
Stop of `ora.vm2rh4.LISTENER_VM2RH4.lsnr` on member `vm2rh4` succeeded.
Attempting to stop `ora.vm2rh4.ASM2.asm` on member `vm2rh4`
Stop of `ora.vm1rh4.LISTENER_VM1RH4.lsnr` on member `vm1rh4` succeeded.
Attempting to stop `ora.vm1rh4.ASM1.asm` on member `vm1rh4`
Stop of `ora.vm2rh4.ASM2.asm` on member `vm2rh4` succeeded.
Attempting to stop `ora.vm2rh4.vip` on member `vm2rh4`
Stop of `ora.vm2rh4.vip` on member `vm2rh4` succeeded.
Stop of `ora.vm1rh4.ASM1.asm` on member `vm1rh4` succeeded.
Attempting to stop `ora.vm1rh4.vip` on member `vm1rh4`
Stop of `ora.vm1rh4.vip` on member `vm1rh4` succeeded.

As you can see, it just does not restart all of the services when we do the crs_stat –t.

[[email protected] ~]$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
————————————————————
ora….SM1.asm application OFFLINE OFFLINE
ora….H4.lsnr application OFFLINE OFFLINE
ora.vm1rh4.gsd application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.ons application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.vip application OFFLINE OFFLINE
ora….SM2.asm application OFFLINE OFFLINE
ora….H4.lsnr application OFFLINE OFFLINE
ora.vm2rh4.gsd application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.ons application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.vip application OFFLINE OFFLINE
ora….SM3.asm application ONLINE OFFLINE
ora….H4.lsnr application ONLINE OFFLINE
ora.vm3rh4.gsd application ONLINE UNKNOWN vm3rh4
ora.vm3rh4.ons application ONLINE UNKNOWN vm3rh4
ora.vm3rh4.vip application ONLINE UNKNOWN vm1rh4
ora….SM4.asm application ONLINE OFFLINE
ora….H4.lsnr application ONLINE OFFLINE
ora.vm4rh4.gsd application OFFLINE OFFLINE
ora.vm4rh4.ons application OFFLINE OFFLINE
ora.vm4rh4.vip application ONLINE UNKNOWN vm4rh4

And this is bizarre , of course, I will test it on a ESX 3.0 with more capacity and see if it vanishes, but now the task is to start all the services one by one, which is not easy when you have “ora….H4.lsnr” names. So, when you do the following, you get full names of the services.

[[email protected] ~]$ $ORA_CRS_HOME/bin/crs_stat

In my case, it gave these and now you have the full names.

NAME=ora.brianic.brianic1.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic2.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic3.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic4.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.db
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.fokeserv.brianic1.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.brianic.fokeserv.brianic2.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.brianic.fokeserv.brianic3.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.brianic.fokeserv.brianic4.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.brianic.fokeserv.cs
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm1rh4.ASM1.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm1rh4.LISTENER_VM1RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm1rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.vm1rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.vm1rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.ASM2.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.LISTENER_VM2RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.vm2rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.vm2rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.ASM3.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.LISTENER_VM3RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.vm3rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.vm3rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.ASM4.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.LISTENER_VM4RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm4rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm4rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

Tarry Singh
I have been active in several industries since 1991. While working in the maritime industry I have worked for several Fortune 500 firms such as NYK, A.P. Møller-Mærsk Group. I made a career switch, emigrated, learned a new language and moved into the IT industry starting 2000. Since then I have been a Sr. DBA, (Technical) Project Manager, Sr. Consultant, Infrastructure Specialist (Clustering, Load Balancing, Networks, Databases) and (currently) Virtualization/Cloud Computing Expert and Global Sourcing in the IT industry. My deep understanding of multi-cultural issues (having worked across the globe) and international exposure has not only helped me successfully relaunch my career in a new industry but also helped me stay successful in what I do. I believe in "worknets" and "collective or swarm intelligence". As a trainer (technical as well as non-technical) I have trained staff both on national and international level. I am very devoted, perspicacious and hard working.

Latest Articles