Brief intro
As a continuing part of the hands-on articles, we will take a deeper look at such things as errors in the installation (which are actually not errors), ESX host tuning for time synchronization (without which the whole RHEL RAC installation means nothing) and some SRVCTL commands.
Errors come and errors go
Let’s take a look at a couple of screen shots and some of the typical Cluster Ready Services (CRS) errors and tricks to bring your RAC services and applications online.
This is a typical placement error that I get on every installation. I think it may have to do with the time issue; we will come to that. If you are using an ESX server to test/develop your RAC, then the information to test and fix your time synchronization issues will certainly come very handy.
It really doesn’t mean a thing. I do run into this every time I do my virtual machine restart.
[oracle@vm1rh4 ~]$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
————————————————————
ora….SM1.asm application ONLINE UNKNOWN vm1rh4
ora….H4.lsnr application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.gsd application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.ons application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.vip application ONLINE ONLINE vm1rh4
ora….SM2.asm application ONLINE UNKNOWN vm2rh4
ora….H4.lsnr application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.gsd application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.ons application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.vip application ONLINE ONLINE vm2rh4
ora….SM3.asm application ONLINE OFFLINE
ora….H4.lsnr application ONLINE OFFLINE
ora.vm3rh4.gsd application ONLINE UNKNOWN vm3rh4
ora.vm3rh4.ons application ONLINE UNKNOWN vm3rh4
ora.vm3rh4.vip application ONLINE UNKNOWN vm1rh4
ora….SM4.asm application ONLINE OFFLINE
ora….H4.lsnr application ONLINE OFFLINE
ora.vm4rh4.gsd application ONLINE ONLINE vm4rh4
ora.vm4rh4.ons application ONLINE ONLINE vm4rh4
ora.vm4rh4.vip application ONLINE UNKNOWN vm4rh4
This is really a VM issue. I am using a 1.2 G vMEM, 2vCPUs, and decent SCSI Virtual Disks (VMFS-2 file system on a VMDK format). Restarting the services does not go as expected. Let’s see what happens if you try to do a crs_stat –stop and then a crs_stop –start.
[oracle@vm1rh4 ~]$ $ORA_CRS_HOME/bin/crs_stop -all
Attempting to stop `ora.vm4rh4.gsd` on member `vm4rh4`
Attempting to stop `ora.vm4rh4.ons` on member `vm4rh4`
Stop of `ora.vm4rh4.gsd` on member `vm4rh4` succeeded.
Stop of `ora.vm4rh4.ons` on member `vm4rh4` succeeded.
Attempting to stop `ora.vm2rh4.LISTENER_VM2RH4.lsnr` on member `vm2rh4`
Attempting to stop `ora.vm1rh4.LISTENER_VM1RH4.lsnr` on member `vm1rh4`
Stop of `ora.vm2rh4.LISTENER_VM2RH4.lsnr` on member `vm2rh4` succeeded.
Attempting to stop `ora.vm2rh4.ASM2.asm` on member `vm2rh4`
Stop of `ora.vm1rh4.LISTENER_VM1RH4.lsnr` on member `vm1rh4` succeeded.
Attempting to stop `ora.vm1rh4.ASM1.asm` on member `vm1rh4`
Stop of `ora.vm2rh4.ASM2.asm` on member `vm2rh4` succeeded.
Attempting to stop `ora.vm2rh4.vip` on member `vm2rh4`
Stop of `ora.vm2rh4.vip` on member `vm2rh4` succeeded.
Stop of `ora.vm1rh4.ASM1.asm` on member `vm1rh4` succeeded.
Attempting to stop `ora.vm1rh4.vip` on member `vm1rh4`
Stop of `ora.vm1rh4.vip` on member `vm1rh4` succeeded.
As you can see, it just does not restart all of the services when we do the crs_stat –t.
[oracle@vm1rh4 ~]$ $ORA_CRS_HOME/bin/crs_stat -t
Name Type Target State Host
————————————————————
ora….SM1.asm application OFFLINE OFFLINE
ora….H4.lsnr application OFFLINE OFFLINE
ora.vm1rh4.gsd application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.ons application ONLINE UNKNOWN vm1rh4
ora.vm1rh4.vip application OFFLINE OFFLINE
ora….SM2.asm application OFFLINE OFFLINE
ora….H4.lsnr application OFFLINE OFFLINE
ora.vm2rh4.gsd application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.ons application ONLINE UNKNOWN vm2rh4
ora.vm2rh4.vip application OFFLINE OFFLINE
ora….SM3.asm application ONLINE OFFLINE
ora….H4.lsnr application ONLINE OFFLINE
ora.vm3rh4.gsd application ONLINE UNKNOWN vm3rh4
ora.vm3rh4.ons application ONLINE UNKNOWN vm3rh4
ora.vm3rh4.vip application ONLINE UNKNOWN vm1rh4
ora….SM4.asm application ONLINE OFFLINE
ora….H4.lsnr application ONLINE OFFLINE
ora.vm4rh4.gsd application OFFLINE OFFLINE
ora.vm4rh4.ons application OFFLINE OFFLINE
ora.vm4rh4.vip application ONLINE UNKNOWN vm4rh4
And this is bizarre , of course, I will test it on a ESX 3.0 with more capacity and see if it vanishes, but now the task is to start all the services one by one, which is not easy when you have “ora….H4.lsnr” names. So, when you do the following, you get full names of the services.
[oracle@vm1rh4 ~]$ $ORA_CRS_HOME/bin/crs_stat
In my case, it gave these and now you have the full names.
NAME=ora.brianic.brianic1.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic2.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic3.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.brianic4.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.db
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.brianic.fokeserv.brianic1.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.brianic.fokeserv.brianic2.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.brianic.fokeserv.brianic3.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.brianic.fokeserv.brianic4.srv
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.brianic.fokeserv.cs
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm1rh4.ASM1.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm1rh4.LISTENER_VM1RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm1rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.vm1rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm1rh4
NAME=ora.vm1rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.ASM2.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.LISTENER_VM2RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm2rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.vm2rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm2rh4
NAME=ora.vm2rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.ASM3.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.LISTENER_VM3RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm3rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.vm3rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm3rh4
NAME=ora.vm3rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.ASM4.asm
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.LISTENER_VM4RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.vm4rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm4rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vm4rh4
NAME=ora.vm4rh4.vip
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE