Top Monitoring RAC Commands

Top Monitoring RAC Commands

Check the status of the cluster

[oracle@rac1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[oracle@rac1 ~]$ crsctl check cluster
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

[oracle@rac1 ~]$crsctl check cluster -all
**************************************************************
rac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

Check the status of the services on the cluster

[oracle@rac1 ~]$ crsctl status resource -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
ONLINE ONLINE rac1
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ora.asm
ONLINE ONLINE rac1
ora.gsd
OFFLINE OFFLINE rac1
ora.net1.network
ONLINE ONLINE rac1
ora.ons
ONLINE ONLINE rac1
ora.registry.acfs
ONLINE ONLINE rac1
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1
ora.cvu
1 ONLINE ONLINE rac1
ora.oc4j
1 ONLINE ONLINE rac1
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 OFFLINE OFFLINE
ora.scan1.vip
1 ONLINE ONLINE rac1
ora.testrac.db
1 OFFLINE OFFLINE Instance Shutdown

Stop crs on all nodes (only one node up in this example)

[root@rac1 bin]# ./crsctl stop cluster -all
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘rac1’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘rac1’
CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘rac1’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.rac1.vip’ on ‘rac1’
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘rac1’
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.rac1.vip’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.cvu’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.registry.acfs’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.oc4j’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘rac1’
CRS-2677: Stop of ‘ora.asm’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘rac1’
CRS-2677: Stop of ‘ora.ons’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘rac1’
CRS-2677: Stop of ‘ora.net1.network’ on ‘rac1’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘rac1’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘rac1’
CRS-2677: Stop of ‘ora.asm’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘rac1’
CRS-2677: Stop of ‘ora.evmd’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘rac1’
CRS-2677: Stop of ‘ora.cssd’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘rac1’
CRS-2677: Stop of ‘ora.diskmon’ on ‘rac1’ succeeded

Stop crs on specifc node

[root@rac1 bin]# ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘rac1’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘rac1’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘rac1’
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.registry.acfs’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘rac1’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.rac1.vip’ on ‘rac1’
CRS-2677: Stop of ‘ora.rac1.vip’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘rac1’
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.cvu’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.oc4j’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.registry.acfs’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘rac1’
CRS-2677: Stop of ‘ora.asm’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘rac1’
CRS-2677: Stop of ‘ora.ons’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘rac1’
CRS-2677: Stop of ‘ora.net1.network’ on ‘rac1’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘rac1’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.crf’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘rac1’
CRS-2677: Stop of ‘ora.asm’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘rac1’
CRS-2677: Stop of ‘ora.evmd’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘rac1’
CRS-2677: Stop of ‘ora.cssd’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘rac1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘rac1’
CRS-2677: Stop of ‘ora.diskmon’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.gpnpd’ on ‘rac1’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘rac1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.

Disable CRS from starting on reboot

[root@rac1 bin]# ./crsctl disable has
CRS-4621: Oracle High Availability Services autostart is disabled.

Enable CRS to start on reboot

[root@rac1 bin]# ./crsctl enable has
CRS-4622: Oracle High Availability Services autostart is enabled.

Start crs on specifc node

[root@rac1 bin]# ./crsctl start has
CRS-4123: Oracle High Availability Services has been started.

Check database status

[oracle@rac1 ~]$ srvctl status database -d RACDB
Instance RACDB1 is running on node rac1
Instance RACDB2 is running on node rac2
Stop an instance

[oracle@rac1 ~]$ srvctl stop instance -i RACDB1 -d RACDB

[oracle@rac1 ~]$srvctl status database -d RACDB
Instance RACDB1 is not running on node rac1
Instance RACDB2 is running on node rac2

Start an instance

[oracle@rac1 ~]$ srvctl start instance -i RACDB1 -d RACDB

[oracle@rac1 ~]$ srvctl status database -d RACDB
Instance RACDB1 is running on node rac1
Instance RACDB2 is running on node rac2

Stop the database

[oracle@rac1 ~]$ srvctl stop database -d RACDB

[oracle@rac1 ~]$ srvctl status database -d RACDB
Instance RACDB1 is not running on node rac1
Instance RACDB2 is not running on node rac2

Start the database

[oracle@rac1 ~]$ srvctl start database -d RACDB

[oracle@rac1 ~]$ srvctl status database -d RACDB
Instance RACDB1 is running on node rac1
Instance RACDB2 is running on node rac2

Stop the listener on a Node

[oracle@rac1 ~]$ srvctl status listener -n rac1
Listener LISTENER is enabled on node(s): rac1
Listener LISTENER is not running on node(s): rac1

Start the listener on a Node

[oracle@rac1 ~]$ srvctl status listener -n rac1
Listener LISTENER is enabled on node(s): rac1
Listener LISTENER is running on node(s): rac1

Check the voting disk
[oracle@rac1 ~]$ crsctl query cssvotedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 91b18024f9674f83bf5117d3c0996a05 (ORCL:VOL1) [DATA]
Located 1 voting disk(s).

Query the network addresses

[oracle@rac1 ~]$ oifcfggetif
eth0 192.168.245.0 global public
eth1 192.168.126.0 global cluster_interconnect

Show OCR backup – the voting file (disk) is also backed up with the OCR backup

[root@rac1 bin]# ./ocrconfig -showbackup
rac1 2012/07/17 07:47:17 /u01/app/11.2.0/grid/cdata/rac-cluster/backup00.ocr
rac1 2012/07/17 03:47:15 /u01/app/11.2.0/grid/cdata/rac-cluster/backup01.ocr
rac1 2012/07/16 23:47:13 /u01/app/11.2.0/grid/cdata/rac-cluster/backup02.ocr
rac1 2012/07/16 19:47:11 /u01/app/11.2.0/grid/cdata/rac-cluster/day.ocr
rac1 2012/07/16 19:47:11 /u01/app/11.2.0/grid/cdata/rac-cluster/week.ocr
rac1 2012/03/23 11:30:19 /u01/app/11.2.0/grid/cdata/rac-cluster/backup_20120323_113019.ocr

Take a manual backup of the OCR

[root@rac1 bin]# ./ocrconfig -manualbackup
rac1 2012/07/17 10:08:28 /u01/app/11.2.0/grid/cdata/rac-cluster/backup_20120717_100828.ocr
rac1 2012/07/17 10:07:39 /u01/app/11.2.0/grid/cdata/rac-cluster/backup_20120717_100739.ocr
rac1 2012/03/23 11:30:19 /u01/app/11.2.0/grid/cdata/rac-cluster/backup_20120323_113019.ocr

COMMAND                                      DESCRIPTION

---------------------------------------------------------------------

crsctl status resource –t                      Clusterware Resource Status Check

srvctl status database -d ORCL        STATUS OF DATABASE

srvctl stop listener -l LISTENER_NAME      STOP A LISTENER

srvctl start listener -l LISTENER_NAME      START A LISTENER

crsctl stop has                                    stop all the clusterware services/ resources on specific node (including DB and listener) (run as root)

crsctl start has                                    start all the clusterware services/ resources on specific node (including DB and listener) (run as root)

crsctl stop cluster –all            to stop csr services on all nodes of clusterware (run as root)

crsctl start cluster –all           to start crs services on all nodes of clusterware (run as root)

crsctl check has                   to check if ohasd is running/ stopped (run as root)

crsctl enable has                  enable Oracle High Availability Services autostart (run as root)

crsctl disable has                 disable Oracle High Availability Services autostart (run as root)

crsctlconfig has                    check if Oracle High Availability Services autostart is enabled/ disabled (run as root)

srvctl status nodeapps         to check the status of services on all nodes

crsctl stop crs                                      stop all the cluster ware services/ resources ON THAT NODE! (run as root)

crsctl start crs                      start all the cluster ware services/ resources ON THAT NODE! (run as root)

cluvfy comp scan –verbose                Verifying scan status scan_listener                        

srvctlconfigscan_listener                    Verifying scan port

srvctl relocate scan -i 1 -n NODE1    Relocate scan listener 1 to the mentioned node

/etc/init.d/init.crs start                        START CRS PROCESS (AS ROOT USER)

srvctl start asm -n NODE1                START ASM INSTANCE on node 1

srvctl start asm -n NODE2                 START ASM INSTANCE on node 2

srvctl start database -d ORCL          START DATABASE

srvctl start instance -d ORCL -i ORCL1    START first INSTANCE (skip it if running 'start database' as that will start both instances)

srvctl start instance -d ORCL -i ORCL2    START second INSTANCE (skip it if running 'start database', as taht will start both instances)

srvctl start nodeapps -n NODE1                      START NODEAPPS on NODE1

srvctl start nodeapps -n NODE2                      START NODEAPPS ON NODE2

srvctl stop database -d ORCL                          STOP DATABASE 

srvctl stop instance -d ORCL -i ORCL1     STOP first INSTANCE (skip it if running 'stop database' as that will 

                                                                        stop both instances)

srvctl stop instance -d ORCL -i ORCL2     STOP second INSTANCE (skip it if running 'stop database' as that will

                                                                       stop both instances)

srvctl stop asm -n NODE1                 STOP ASM INSTANCES on NODE 1 (In 11G , we have OCR on ASM

                                                             so we cannot stop ASM, but if you have OCR in NON-ASM you should stop it)

srvctl stop asm -n NODE2                 STOP ASM INSTANCES on NODE 2 (In 

11G , we have OCR on ASM so we cannot stop ASM, but if you have OCR in NON-ASM 

you should stop it)

srvctl stop nodeapps -n NODE1                       STOP NODEAPPS  on NODE 1

srvctl stop nodeapps -n NODE2                       STOP NODEAPPS  on NODE 2

/etc/init.d/init.crsstop                                         STOP CRS PROCESSES (AS ROOT USER)

Thanks (Comments please)