Brief intro
In our last article, we spoke about certain CRS_ commands and syntax. We should also try to see what happens when our cluster suddenly starts to act weird and we really do not know where to get started and how to proceed. But first let’s continue our introduction to the remaining crs_ commands.
Introduction to CRS
crs_start
This command sets all the applications or the specified applications ONLINE, and attempts to start the specified registered applications or application resources. You should have full admin privileges for this.
Syntax
crs_start resource_name [...] [-c cluster_node] [-q] [-f] crs_start -all [-q] crs_start [USR_attribute_name=value] [...] resource_name [-c node_name] [-q]
crs_stop
This command stops a resource on the specified node. Checking your crs_stat always helps to verify which application/resource we want to stop.
Syntax
crs_stop resource_name [...] [-f] [-q] crs_stop -c cluster_node [...] [-q] crs_stop -all [-q] crs_stop [USR_attribute_name=value] [...] resource_name [-q] -c cluster_node [...]
crs_unregister
The crs_unregister command removes the registration information of Oracle Clusterware resources.
Syntax
crs_unregister resource_name [...] [-q]
Correcting our RAC problem
We have seen the CRS commands and syntax, now lets look at the practical aspect of it.
I was having some trouble while running my RAC on RHEL4 (I think the problem primarily arose because I supplied my virtual machines with inadequate memory. When I noticed that I was getting some alerts on my SoRAC tool, I looked at the status of my RAC.
oracle@vmora01rh4 ~]$ cd /u01/app/oracle/product/10.2.0/crs/bin [oracle@vmora01rh4 bin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.fokerac.db application ONLINE ONLINE vmora02rh4 ora....c1.inst application ONLINE ONLINE vmora01rh4 ora....c2.inst application OFFLINE UNKNOWN vmora02rh4 ora....serv.cs application ONLINE ONLINE vmora02rh4 ora....ac1.srv application ONLINE ONLINE vmora01rh4 ora....ac2.srv application ONLINE OFFLINE ora....SM1.asm application ONLINE ONLINE vmora01rh4 ora....H4.lsnr application ONLINE ONLINE vmora01rh4 ora....rh4.gsd application ONLINE UNKNOWN vmora01rh4 ora....rh4.ons application ONLINE UNKNOWN vmora01rh4 ora....rh4.vip application ONLINE ONLINE vmora01rh4 ora....SM2.asm application OFFLINE UNKNOWN vmora02rh4 ora....H4.lsnr application OFFLINE UNKNOWN vmora02rh4 ora....rh4.gsd application ONLINE UNKNOWN vmora02rh4 ora....rh4.ons application OFFLINE UNKNOWN vmora02rh4 ora....rh4.vip application ONLINE ONLINE vmora02rh4
As you can see above, some of the applications are UNKNOWN or OFFLINE, either of which is not good for my RAC.
The crs_stat command gives you the names of the applications, which you might need to shut down some applications manually, in order to shut the whole cluster down and restart it.
[oracle@vmora01rh4 bin]$ crs_stat NAME=ora.fokerac.db TYPE=application TARGET=ONLINE STATE=ONLINE on vmora02rh4 NAME=ora.fokerac.fokerac1.inst TYPE=application TARGET=ONLINE STATE=ONLINE on vmora01rh4 NAME=ora.fokerac.fokerac2.inst TYPE=application TARGET=OFFLINE STATE=OFFLINE NAME=ora.fokerac.fokeserv.cs TYPE=application TARGET=ONLINE STATE=ONLINE on vmora02rh4 NAME=ora.fokerac.fokeserv.fokerac1.srv TYPE=application TARGET=ONLINE STATE=ONLINE on vmora01rh4 NAME=ora.fokerac.fokeserv.fokerac2.srv TYPE=application TARGET=ONLINE STATE=OFFLINE NAME=ora.vmora01rh4.ASM1.asm TYPE=application TARGET=ONLINE STATE=ONLINE on vmora01rh4 NAME=ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr TYPE=application TARGET=ONLINE STATE=ONLINE on vmora01rh4 NAME=ora.vmora01rh4.gsd TYPE=application TARGET=ONLINE STATE=UNKNOWN on vmora01rh4 NAME=ora.vmora01rh4.ons TYPE=application TARGET=ONLINE STATE=UNKNOWN on vmora01rh4 NAME=ora.vmora01rh4.vip TYPE=application TARGET=ONLINE STATE=ONLINE on vmora01rh4 NAME=ora.vmora02rh4.ASM2.asm TYPE=application TARGET=OFFLINE STATE=UNKNOWN on vmora02rh4 NAME=ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr TYPE=application TARGET=OFFLINE STATE=UNKNOWN on vmora02rh4 NAME=ora.vmora02rh4.gsd TYPE=application TARGET=ONLINE STATE=UNKNOWN on vmora02rh4 NAME=ora.vmora02rh4.ons TYPE=application TARGET=OFFLINE STATE=UNKNOWN on vmora02rh4 NAME=ora.vmora02rh4.vip TYPE=application TARGET=ONLINE STATE=ONLINE on vmora02rh4
I could have also attempted to stop them all using crs_stop –all, but it normally throws enough errors to force you do it manually one by one.
[oracle@vmora01rh4 bin]$ crs_stop -all Attempting to stop `ora.vmora01rh4.ons` on member `vmora01rh4` Attempting to stop `ora.vmora02rh4.ons` on member `vmora02rh4` `ora.vmora02rh4.ons` on member `vmora02rh4` has experienced an unrecoverable failure. Human intervention required to resume its availability. Stop of `ora.vmora01rh4.ons` on member `vmora01rh4` succeeded. Attempting to stop `ora.vmora01rh4.ASM1.asm` on member `vmora01rh4` Attempting to stop `ora.fokerac.fokerac2.inst` on member `vmora02rh4` `ora.fokerac.fokerac2.inst` on member `vmora02rh4` has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop `ora.fokerac.fokerac2.inst` on member `vmora02rh4` `ora.fokerac.fokerac2.inst` on member `vmora02rh4` has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop `ora.vmora02rh4.vip` on member `vmora02rh4` Stop of `ora.vmora02rh4.vip` on member `vmora02rh4` succeeded. Stop of `ora.vmora01rh4.ASM1.asm` on member `vmora01rh4` succeeded. Attempting to stop `ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr` on member `vmora01rh4` Stop of `ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr` on member `vmora01rh4` succeeded. Attempting to stop `ora.vmora01rh4.vip` on member `vmora01rh4` Stop of `ora.vmora01rh4.vip` on member `vmora01rh4` succeeded. CRS-0216: Could not stop resource 'ora.vmora02rh4.ASM2.asm'. CRS-0216: Could not stop resource 'ora.vmora02rh4.ons'. CRS-0216: Could not stop resource 'ora.vmora02rh4.vip'.
For the very same reason we will go ahead and do it our way. Therefore, we need to stop our instances first.
[oracle@vmora01rh4 bin]$ srvctl stop instance -d fokerac -i fokerac1 [oracle@vmora01rh4 bin]$ srvctl stop instance -d fokerac -i fokerac2
Check our status
[oracle@vmora01rh4 bin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.fokerac.db application OFFLINE OFFLINE ora....c1.inst application OFFLINE OFFLINE ora....c2.inst application OFFLINE OFFLINE ora....serv.cs application ONLINE UNKNOWN vmora02rh4 ora....ac1.srv application OFFLINE OFFLINE ora....ac2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....H4.lsnr application OFFLINE OFFLINE ora....rh4.gsd application ONLINE UNKNOWN vmora01rh4 ora....rh4.ons application OFFLINE OFFLINE ora....rh4.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE UNKNOWN vmora02rh4 ora....H4.lsnr application OFFLINE UNKNOWN vmora02rh4 ora....rh4.gsd application ONLINE UNKNOWN vmora02rh4 ora....rh4.ons application OFFLINE UNKNOWN vmora02rh4 ora....rh4.vip application OFFLINE OFFLINE
Stop the service
[oracle@vmora01rh4 bin]$ srvctl stop service -d fokerac -s fokeserv
Check status again
[oracle@vmora01rh4 bin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.fokerac.db application OFFLINE OFFLINE ora....c1.inst application OFFLINE OFFLINE ora....c2.inst application OFFLINE OFFLINE ora....serv.cs application OFFLINE OFFLINE ora....ac1.srv application OFFLINE OFFLINE ora....ac2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....H4.lsnr application OFFLINE OFFLINE ora....rh4.gsd application ONLINE UNKNOWN vmora01rh4 ora....rh4.ons application OFFLINE OFFLINE ora....rh4.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE UNKNOWN vmora02rh4 ora....H4.lsnr application OFFLINE UNKNOWN vmora02rh4 ora....rh4.gsd application ONLINE UNKNOWN vmora02rh4 ora....rh4.ons application OFFLINE UNKNOWN vmora02rh4 ora....rh4.vip application OFFLINE OFFLINE
OK, so we need to stop those applications now.
[oracle@vmora01rh4 bin]$ crs_stop ora.vmora01rh4.gsd Attempting to stop `ora.vmora01rh4.gsd` on member `vmora01rh4` Stop of `ora.vmora01rh4.gsd` on member `vmora01rh4` succeeded. [oracle@vmora01rh4 bin]$ crs_stop ora.vmora02rh4.ASM2.asm Attempting to stop `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` Stop of `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` succeeded. [oracle@vmora01rh4 bin]$ crs_stop ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr Attempting to stop `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` Stop of `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` succeeded. [oracle@vmora01rh4 bin]$ crs_stop ora.vmora02rh4.gsd Attempting to stop `ora.vmora02rh4.gsd` on member `vmora02rh4` Stop of `ora.vmora02rh4.gsd` on member `vmora02rh4` succeeded. [oracle@vmora01rh4 bin]$ crs_stop ora.vmora02rh4.ons Attempting to stop `ora.vmora02rh4.ons` on member `vmora02rh4` Stop of `ora.vmora02rh4.ons` on member `vmora02rh4` succeeded.
Check status
[oracle@vmora01rh4 bin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.fokerac.db application OFFLINE OFFLINE ora....c1.inst application OFFLINE OFFLINE ora....c2.inst application OFFLINE OFFLINE ora....serv.cs application OFFLINE OFFLINE ora....ac1.srv application OFFLINE OFFLINE ora....ac2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....H4.lsnr application OFFLINE OFFLINE ora....rh4.gsd application OFFLINE OFFLINE ora....rh4.ons application OFFLINE OFFLINE ora....rh4.vip application OFFLINE OFFLINE ora....SM2.asm application OFFLINE OFFLINE ora....H4.lsnr application OFFLINE OFFLINE ora....rh4.gsd application OFFLINE OFFLINE ora....rh4.ons application OFFLINE OFFLINE ora....rh4.vip application OFFLINE OFFLINE
OK all set , now lets bring them all online.
[oracle@vmora01rh4 bin]$ crs_start -all Attempting to start `ora.vmora02rh4.vip` on member `vmora02rh4` Attempting to start `ora.vmora01rh4.vip` on member `vmora01rh4` Start of `ora.vmora02rh4.vip` on member `vmora02rh4` succeeded. Start of `ora.vmora01rh4.vip` on member `vmora01rh4` succeeded. Attempting to start `ora.vmora01rh4.ASM1.asm` on member `vmora01rh4` Attempting to start `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` Start of `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` succeeded. Attempting to start `ora.fokerac.fokerac2.inst` on member `vmora02rh4` Start of `ora.vmora01rh4.ASM1.asm` on member `vmora01rh4` succeeded. Attempting to start `ora.fokerac.fokerac1.inst` on member `vmora01rh4` Start of `ora.fokerac.fokerac2.inst` on member `vmora02rh4` succeeded. Attempting to start `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` Start of `ora.fokerac.fokerac1.inst` on member `vmora01rh4` succeeded. Attempting to start `ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr` on member `vmora01rh4` Start of `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` succeeded. Start of `ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr` on member `vmora01rh4` succeeded. CRS-1002: Resource 'ora.vmora02rh4.ons' is already running on member 'vmora02rh4' CRS-1002: Resource 'ora.vmora01rh4.ons' is already running on member 'vmora01rh4' Attempting to start `ora.fokerac.fokeserv.fokerac1.srv` on member `vmora01rh4` Attempting to start `ora.vmora01rh4.gsd` on member `vmora01rh4` Attempting to start `ora.fokerac.db` on member `vmora01rh4` Attempting to start `ora.fokerac.fokeserv.fokerac2.srv` on member `vmora02rh4` Attempting to start `ora.fokerac.fokeserv.cs` on member `vmora02rh4` Attempting to start `ora.vmora02rh4.gsd` on member `vmora02rh4` Start of `ora.fokerac.fokeserv.fokerac2.srv` on member `vmora02rh4` succeeded. Start of `ora.fokerac.fokeserv.cs` on member `vmora02rh4` succeeded. Start of `ora.fokerac.db` on member `vmora01rh4` succeeded. Start of `ora.vmora02rh4.gsd` on member `vmora02rh4` succeeded. Start of `ora.vmora01rh4.gsd` on member `vmora01rh4` succeeded. Start of `ora.fokerac.fokeserv.fokerac1.srv` on member `vmora01rh4` succeeded. *CRS-0223: Resource 'ora.vmora01rh4.ons' has placement error. CRS-0223: Resource 'ora.vmora02rh4.ons' has placement error.
*Don’t bother about those errors, as they just did not report back to us in the sequence they were started by the clusterware.
[oracle@vmora01rh4 bin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.fokerac.db application ONLINE ONLINE vmora01rh4 ora....c1.inst application ONLINE ONLINE vmora01rh4 ora....c2.inst application ONLINE ONLINE vmora02rh4 ora....serv.cs application ONLINE ONLINE vmora02rh4 ora....ac1.srv application ONLINE ONLINE vmora01rh4 ora....ac2.srv application ONLINE ONLINE vmora02rh4 ora....SM1.asm application ONLINE ONLINE vmora01rh4 ora....H4.lsnr application ONLINE ONLINE vmora01rh4 ora....rh4.gsd application ONLINE ONLINE vmora01rh4 ora....rh4.ons application ONLINE ONLINE vmora01rh4 ora....rh4.vip application ONLINE ONLINE vmora01rh4 ora....SM2.asm application ONLINE ONLINE vmora02rh4 ora....H4.lsnr application ONLINE ONLINE vmora02rh4 ora....rh4.gsd application ONLINE ONLINE vmora02rh4 ora....rh4.ons application ONLINE ONLINE vmora02rh4 ora....rh4.vip application ONLINE ONLINE vmora02rh4 [oracle@vmora01rh4 bin]$
Conclusion:
The power and strength of a command or any other tool lies not in its intelligence, it lies in our ability to do some effective troubleshooting with them. This article has discussed how the CRS commands can be used to fix/troubleshoot the quirkiness of our RAC.
