dcsimg

Oracle RAC Administration - Part 2: CRS commands

June 15, 2006

Brief intro

In our last article, we spoke about certain CRS_ commands and syntax. We should also try to see what happens when our cluster suddenly starts to act weird and we really do not know where to get started and how to proceed. But first let's continue our introduction to the remaining crs_ commands.

Introduction to CRS

crs_start

This command sets all the applications or the specified applications ONLINE, and attempts to start the specified registered applications or application resources. You should have full admin privileges for this.

Syntax

crs_start resource_name  [...] [-c cluster_node] [-q] [-f]
crs_start -all [-q]
crs_start [USR_attribute_name=value] [...] 
 resource_name [-c node_name] [-q]

crs_stop

This command stops a resource on the specified node. Checking your crs_stat always helps to verify which application/resource we want to stop.

Syntax

crs_stop resource_name [...] [-f] [-q]
crs_stop -c cluster_node [...] [-q]
crs_stop -all [-q]
crs_stop [USR_attribute_name=value] [...] resource_name [-q]
-c cluster_node [...]

crs_unregister

The crs_unregister command removes the registration information of Oracle Clusterware resources.

Syntax

crs_unregister resource_name [...] [-q]

Correcting our RAC problem

We have seen the CRS commands and syntax, now lets look at the practical aspect of it.
I was having some trouble while running my RAC on RHEL4 (I think the problem primarily arose because I supplied my virtual machines with inadequate memory. When I noticed that I was getting some alerts on my SoRAC tool, I looked at the status of my RAC.

oracle@vmora01rh4 ~]$ cd /u01/app/oracle/product/10.2.0/crs/bin
[oracle@vmora01rh4 bin]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.fokerac.db application    ONLINE    ONLINE    vmora02rh4
ora....c1.inst application    ONLINE    ONLINE    vmora01rh4
ora....c2.inst application    OFFLINE   UNKNOWN   vmora02rh4
ora....serv.cs application    ONLINE    ONLINE    vmora02rh4
ora....ac1.srv application    ONLINE    ONLINE    vmora01rh4
ora....ac2.srv application    ONLINE    OFFLINE
ora....SM1.asm application    ONLINE    ONLINE    vmora01rh4
ora....H4.lsnr application    ONLINE    ONLINE    vmora01rh4
ora....rh4.gsd application    ONLINE    UNKNOWN   vmora01rh4
ora....rh4.ons application    ONLINE    UNKNOWN   vmora01rh4
ora....rh4.vip application    ONLINE    ONLINE    vmora01rh4
ora....SM2.asm application    OFFLINE   UNKNOWN   vmora02rh4
ora....H4.lsnr application    OFFLINE   UNKNOWN   vmora02rh4
ora....rh4.gsd application    ONLINE    UNKNOWN   vmora02rh4
ora....rh4.ons application    OFFLINE   UNKNOWN   vmora02rh4
ora....rh4.vip application    ONLINE    ONLINE    vmora02rh4

As you can see above, some of the applications are UNKNOWN or OFFLINE, either of which is not good for my RAC.

The crs_stat command gives you the names of the applications, which you might need to shut down some applications manually, in order to shut the whole cluster down and restart it.

[oracle@vmora01rh4 bin]$ crs_stat
NAME=ora.fokerac.db
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vmora02rh4
NAME=ora.fokerac.fokerac1.inst
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vmora01rh4
NAME=ora.fokerac.fokerac2.inst
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE
NAME=ora.fokerac.fokeserv.cs
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vmora02rh4
NAME=ora.fokerac.fokeserv.fokerac1.srv
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vmora01rh4
NAME=ora.fokerac.fokeserv.fokerac2.srv
TYPE=application
TARGET=ONLINE
STATE=OFFLINE
NAME=ora.vmora01rh4.ASM1.asm
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vmora01rh4
NAME=ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vmora01rh4
NAME=ora.vmora01rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vmora01rh4
NAME=ora.vmora01rh4.ons
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vmora01rh4
NAME=ora.vmora01rh4.vip
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vmora01rh4
NAME=ora.vmora02rh4.ASM2.asm
TYPE=application
TARGET=OFFLINE
STATE=UNKNOWN on vmora02rh4
NAME=ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr
TYPE=application
TARGET=OFFLINE
STATE=UNKNOWN on vmora02rh4
NAME=ora.vmora02rh4.gsd
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on vmora02rh4
NAME=ora.vmora02rh4.ons
TYPE=application
TARGET=OFFLINE
STATE=UNKNOWN on vmora02rh4
NAME=ora.vmora02rh4.vip
TYPE=application
TARGET=ONLINE
STATE=ONLINE on vmora02rh4

I could have also attempted to stop them all using crs_stop –all, but it normally throws enough errors to force you do it manually one by one.

[oracle@vmora01rh4 bin]$ crs_stop -all
Attempting to stop `ora.vmora01rh4.ons` on member `vmora01rh4`
Attempting to stop `ora.vmora02rh4.ons` on member `vmora02rh4`
`ora.vmora02rh4.ons` on member `vmora02rh4` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Stop of `ora.vmora01rh4.ons` on member `vmora01rh4` succeeded.
Attempting to stop `ora.vmora01rh4.ASM1.asm` on member `vmora01rh4`
Attempting to stop `ora.fokerac.fokerac2.inst` on member `vmora02rh4`
`ora.fokerac.fokerac2.inst` on member `vmora02rh4` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4`
`ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4`
`ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.fokerac.fokerac2.inst` on member `vmora02rh4`
`ora.fokerac.fokerac2.inst` on member `vmora02rh4` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4`
`ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
Attempting to stop `ora.vmora02rh4.vip` on member `vmora02rh4`
Stop of `ora.vmora02rh4.vip` on member `vmora02rh4` succeeded.
Stop of `ora.vmora01rh4.ASM1.asm` on member `vmora01rh4` succeeded.
Attempting to stop `ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr` on member `vmora01rh4`
Stop of `ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr` on member `vmora01rh4` succeeded.
Attempting to stop `ora.vmora01rh4.vip` on member `vmora01rh4`
Stop of `ora.vmora01rh4.vip` on member `vmora01rh4` succeeded.
CRS-0216: Could not stop resource 'ora.vmora02rh4.ASM2.asm'.
CRS-0216: Could not stop resource 'ora.vmora02rh4.ons'.
CRS-0216: Could not stop resource 'ora.vmora02rh4.vip'.

For the very same reason we will go ahead and do it our way. Therefore, we need to stop our instances first.

[oracle@vmora01rh4 bin]$ srvctl stop instance -d fokerac -i fokerac1
[oracle@vmora01rh4 bin]$ srvctl stop instance -d fokerac -i fokerac2

Check our status

[oracle@vmora01rh4 bin]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.fokerac.db application    OFFLINE   OFFLINE
ora....c1.inst application    OFFLINE   OFFLINE
ora....c2.inst application    OFFLINE   OFFLINE
ora....serv.cs application    ONLINE    UNKNOWN   vmora02rh4
ora....ac1.srv application    OFFLINE   OFFLINE
ora....ac2.srv application    OFFLINE   OFFLINE
ora....SM1.asm application    OFFLINE   OFFLINE
ora....H4.lsnr application    OFFLINE   OFFLINE
ora....rh4.gsd application    ONLINE    UNKNOWN   vmora01rh4
ora....rh4.ons application    OFFLINE   OFFLINE
ora....rh4.vip application    OFFLINE   OFFLINE
ora....SM2.asm application    OFFLINE   UNKNOWN   vmora02rh4
ora....H4.lsnr application    OFFLINE   UNKNOWN   vmora02rh4
ora....rh4.gsd application    ONLINE    UNKNOWN   vmora02rh4
ora....rh4.ons application    OFFLINE   UNKNOWN   vmora02rh4
ora....rh4.vip application    OFFLINE   OFFLINE

Stop the service

[oracle@vmora01rh4 bin]$ srvctl stop service -d fokerac -s fokeserv

Check status again

[oracle@vmora01rh4 bin]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.fokerac.db application    OFFLINE   OFFLINE
ora....c1.inst application    OFFLINE   OFFLINE
ora....c2.inst application    OFFLINE   OFFLINE
ora....serv.cs application    OFFLINE   OFFLINE
ora....ac1.srv application    OFFLINE   OFFLINE
ora....ac2.srv application    OFFLINE   OFFLINE
ora....SM1.asm application    OFFLINE   OFFLINE
ora....H4.lsnr application    OFFLINE   OFFLINE
ora....rh4.gsd application    ONLINE    UNKNOWN   vmora01rh4
ora....rh4.ons application    OFFLINE   OFFLINE
ora....rh4.vip application    OFFLINE   OFFLINE
ora....SM2.asm application    OFFLINE   UNKNOWN   vmora02rh4
ora....H4.lsnr application    OFFLINE   UNKNOWN   vmora02rh4
ora....rh4.gsd application    ONLINE    UNKNOWN   vmora02rh4
ora....rh4.ons application    OFFLINE   UNKNOWN   vmora02rh4
ora....rh4.vip application    OFFLINE   OFFLINE

OK, so we need to stop those applications now.

[oracle@vmora01rh4 bin]$ crs_stop ora.vmora01rh4.gsd
Attempting to stop `ora.vmora01rh4.gsd` on member `vmora01rh4`
Stop of `ora.vmora01rh4.gsd` on member `vmora01rh4` succeeded.
[oracle@vmora01rh4 bin]$ crs_stop ora.vmora02rh4.ASM2.asm
Attempting to stop `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4`
Stop of `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` succeeded.
[oracle@vmora01rh4 bin]$ crs_stop ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr
Attempting to stop `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4`
Stop of `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` succeeded.
[oracle@vmora01rh4 bin]$ crs_stop ora.vmora02rh4.gsd
Attempting to stop `ora.vmora02rh4.gsd` on member `vmora02rh4`
Stop of `ora.vmora02rh4.gsd` on member `vmora02rh4` succeeded.
[oracle@vmora01rh4 bin]$ crs_stop ora.vmora02rh4.ons
Attempting to stop `ora.vmora02rh4.ons` on member `vmora02rh4`
Stop of `ora.vmora02rh4.ons` on member `vmora02rh4` succeeded.

Check status

[oracle@vmora01rh4 bin]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.fokerac.db application    OFFLINE   OFFLINE
ora....c1.inst application    OFFLINE   OFFLINE
ora....c2.inst application    OFFLINE   OFFLINE
ora....serv.cs application    OFFLINE   OFFLINE
ora....ac1.srv application    OFFLINE   OFFLINE
ora....ac2.srv application    OFFLINE   OFFLINE
ora....SM1.asm application    OFFLINE   OFFLINE
ora....H4.lsnr application    OFFLINE   OFFLINE
ora....rh4.gsd application    OFFLINE   OFFLINE
ora....rh4.ons application    OFFLINE   OFFLINE
ora....rh4.vip application    OFFLINE   OFFLINE
ora....SM2.asm application    OFFLINE   OFFLINE
ora....H4.lsnr application    OFFLINE   OFFLINE
ora....rh4.gsd application    OFFLINE   OFFLINE
ora....rh4.ons application    OFFLINE   OFFLINE
ora....rh4.vip application    OFFLINE   OFFLINE

OK all set , now lets bring them all online.

[oracle@vmora01rh4 bin]$ crs_start -all
Attempting to start `ora.vmora02rh4.vip` on member `vmora02rh4`
Attempting to start `ora.vmora01rh4.vip` on member `vmora01rh4`
Start of `ora.vmora02rh4.vip` on member `vmora02rh4` succeeded.
Start of `ora.vmora01rh4.vip` on member `vmora01rh4` succeeded.
Attempting to start `ora.vmora01rh4.ASM1.asm` on member `vmora01rh4`
Attempting to start `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4`
Start of `ora.vmora02rh4.ASM2.asm` on member `vmora02rh4` succeeded.
Attempting to start `ora.fokerac.fokerac2.inst` on member `vmora02rh4`
Start of `ora.vmora01rh4.ASM1.asm` on member `vmora01rh4` succeeded.
Attempting to start `ora.fokerac.fokerac1.inst` on member `vmora01rh4`
Start of `ora.fokerac.fokerac2.inst` on member `vmora02rh4` succeeded.
Attempting to start `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4`
Start of `ora.fokerac.fokerac1.inst` on member `vmora01rh4` succeeded.
Attempting to start `ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr` on member `vmora01rh4`
Start of `ora.vmora02rh4.LISTENER_VMORA02RH4.lsnr` on member `vmora02rh4` succeeded.
Start of `ora.vmora01rh4.LISTENER_VMORA01RH4.lsnr` on member `vmora01rh4` succeeded.
CRS-1002: Resource 'ora.vmora02rh4.ons' is already running on member 'vmora02rh4'
CRS-1002: Resource 'ora.vmora01rh4.ons' is already running on member 'vmora01rh4'
Attempting to start `ora.fokerac.fokeserv.fokerac1.srv` on member `vmora01rh4`
Attempting to start `ora.vmora01rh4.gsd` on member `vmora01rh4`
Attempting to start `ora.fokerac.db` on member `vmora01rh4`
Attempting to start `ora.fokerac.fokeserv.fokerac2.srv` on member `vmora02rh4`
Attempting to start `ora.fokerac.fokeserv.cs` on member `vmora02rh4`
Attempting to start `ora.vmora02rh4.gsd` on member `vmora02rh4`
Start of `ora.fokerac.fokeserv.fokerac2.srv` on member `vmora02rh4` succeeded.
Start of `ora.fokerac.fokeserv.cs` on member `vmora02rh4` succeeded.
Start of `ora.fokerac.db` on member `vmora01rh4` succeeded.
Start of `ora.vmora02rh4.gsd` on member `vmora02rh4` succeeded.
Start of `ora.vmora01rh4.gsd` on member `vmora01rh4` succeeded.
Start of `ora.fokerac.fokeserv.fokerac1.srv` on member `vmora01rh4` succeeded.
*CRS-0223: Resource 'ora.vmora01rh4.ons' has placement error.
CRS-0223: Resource 'ora.vmora02rh4.ons' has placement error.

*Don’t bother about those errors, as they just did not report back to us in the sequence they were started by the clusterware.

[oracle@vmora01rh4 bin]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.fokerac.db application    ONLINE    ONLINE    vmora01rh4
ora....c1.inst application    ONLINE    ONLINE    vmora01rh4
ora....c2.inst application    ONLINE    ONLINE    vmora02rh4
ora....serv.cs application    ONLINE    ONLINE    vmora02rh4
ora....ac1.srv application    ONLINE    ONLINE    vmora01rh4
ora....ac2.srv application    ONLINE    ONLINE    vmora02rh4
ora....SM1.asm application    ONLINE    ONLINE    vmora01rh4
ora....H4.lsnr application    ONLINE    ONLINE    vmora01rh4
ora....rh4.gsd application    ONLINE    ONLINE    vmora01rh4
ora....rh4.ons application    ONLINE    ONLINE    vmora01rh4
ora....rh4.vip application    ONLINE    ONLINE    vmora01rh4
ora....SM2.asm application    ONLINE    ONLINE    vmora02rh4
ora....H4.lsnr application    ONLINE    ONLINE    vmora02rh4
ora....rh4.gsd application    ONLINE    ONLINE    vmora02rh4
ora....rh4.ons application    ONLINE    ONLINE    vmora02rh4
ora....rh4.vip application    ONLINE    ONLINE    vmora02rh4
[oracle@vmora01rh4 bin]$

Conclusion:

The power and strength of a command or any other tool lies not in its intelligence, it lies in our ability to do some effective troubleshooting with them. This article has discussed how the CRS commands can be used to fix/troubleshoot the quirkiness of our RAC.

» See All Articles by Columnist Tarry Singh








The Network for Technology Professionals

Search:

About Internet.com

Legal Notices, Licensing, Permissions, Privacy Policy.
Advertise | Newsletters | E-mail Offers