With 12c, restore and recover of the voting disks and the OCR is quite simple since Oracle remembers the exact physical place of the OCR and voting disks on the ASM.
This scenario assumes:
In nutshell, the procedure is:
Corruption is done via the “dd” command:
[root@lparaca grid_12201]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 51ffb7b88b334f26bf482590e403d674 (/dev/sdaa1) [VTDISK] <- Disk to corrupt 2. ONLINE f8f88c54edec4f4cbfda6f9676c6bde6 (/dev/sdy1) [VTDISK] 3. ONLINE b88a12b132a94fafbf64befc48683a82 (/dev/sdz1) [VTDISK] Located 3 voting disk(s). [root@lparaca grid_12201]# dd if=/dev/zero of=/dev/sdaa1 bs=2048 count=10 10+0 records in 10+0 records out 20480 bytes (20 kB) copied, 7.4799e-05 s, 274 MB/s
You can check the corruption by checking the header of the corrupted disk and a normal disk:
[root@lparaca grid_12201]# kfed read /dev/sdaa1 | egrep 'name|size|type' kfbh.type: 0 ; 0x002: KFBTYP_INVALID KFED-00322: invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0] [root@lparaca grid_12201]# od /dev/sdaa1 | head 0000000 000000 000000 000000 000000 000000 000000 000000 000000 * 0050000 101001 001003 000005 000000 000000 100000 104516 106017 0050020 000000 000000 000000 000000 000000 000000 000000 000000 0050040 002500 000000 000000 000000 000010 000010 000014 000014 0050060 000020 000020 000024 000024 000030 000030 000034 000034 0050100 000050 006050 000000 000000 001050 000040 000006 000000 0050120 000000 000000 000000 000000 000000 000000 000000 000000 * 0051100 000000 000000 000000 000000 002050 000050 000006 000000 [root@lparaca client]# od -c /dev/sdaa1 | head 0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0050000 001 202 003 002 005 \0 \0 \0 \0 \0 \0 200 N 211 017 214 0050020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0050040 @ 005 \0 \0 \0 \0 \0 \0 \b \0 \b \0 \f \0 \f \0 0050060 020 \0 020 \0 024 \0 024 \0 030 \0 030 \0 034 \0 034 \0 0050100 ( \0 ( \f \0 \0 \0 \0 ( 002 \0 006 \0 \0 \0 0050120 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0051100 \0 \0 \0 \0 \0 \0 \0 \0 ( 004 ( \0 006 \0 \0 \0 [root@lparaca client]#
You can compare this result with:
[root@lparaca grid_12201]# kfed read /dev/sdy1 | egrep 'name|size|type' kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfdhdb.dskname: VTDISK_0001 ; 0x028: length=11 kfdhdb.grpname: VTDISK ; 0x048: length=6 kfdhdb.fgname: VTDISK_0001 ; 0x068: length=11 kfdhdb.secsize: 512 ; 0x0b8: 0x0200 kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 kfdhdb.ausize: 4194304 ; 0x0bc: 0x00400000 kfdhdb.dsksize: 1279 ; 0x0c4: 0x000004ff [root@lparaca grid_12201]# od /dev/sdy1 | head 0000000 101001 000401 000000 000000 000001 100000 051547 003637 0000020 000000 000000 000000 000000 000000 000000 000000 000000 0000040 051117 046103 044504 045523 000000 000000 000000 000000 0000060 000000 000000 000000 000000 000000 000000 000000 000000 0000100 000400 006040 000001 001402 052126 044504 045523 030137 0000120 030060 000061 000000 000000 000000 000000 000000 000000 0000140 000000 000000 000000 000000 052126 044504 045523 000000 0000160 000000 000000 000000 000000 000000 000000 000000 000000 0000200 000000 000000 000000 000000 052126 044504 045523 030137 0000220 030060 000061 000000 000000 000000 000000 000000 000000 [root@lparaca client]# od -c /dev/sdaa1 | head 0000000 001 202 001 001 \0 \0 \0 \0 \0 \0 \0 200 037 225 031 = 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0 0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000100 \0 001 \f \0 \0 002 003 V T D I S K _ 0 0000120 0 0 0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000140 \0 \0 \0 \0 \0 \0 \0 \0 V T D I S K \0 \0 0000160 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000200 \0 \0 \0 \0 \0 \0 \0 \0 V T D I S K _ 0 0000220 0 0 0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 [root@lparaca client]#
CRS should be brought down to all instances in the cluster, feel free to use the -f option in case of errors:
[root@lparaca grid_12201]# crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lparaca' CRS-2673: Attempting to stop 'ora.crsd' on 'lparaca' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'lparaca' CRS-2673: Attempting to stop 'ora.orcl.db' on 'lparaca' CRS-2677: Stop of 'ora.orcl.db' on 'lparaca' succeeded CRS-2673: Attempting to stop 'ora.VTDISK.dg' on 'lparaca' CRS-2673: Attempting to stop 'ora.DATA.dg' on 'lparaca' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'lparaca' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'lparaca' CRS-2677: Stop of 'ora.VTDISK.dg' on 'lparaca' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'lparaca' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'lparaca' CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'lparaca' succeeded CRS-2673: Attempting to stop 'ora.lparaca.vip' on 'lparaca' CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'lparaca' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'lparaca' CRS-2677: Stop of 'ora.asm' on 'lparaca' succeeded CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'lparaca' CRS-2677: Stop of 'ora.lparaca.vip' on 'lparaca' succeeded CRS-2677: Stop of 'ora.scan1.vip' on 'lparaca' succeeded CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'lparaca' succeeded CRS-2672: Attempting to start 'ora.scan1.vip' on 'lparacb' CRS-2672: Attempting to start 'ora.lparaca.vip' on 'lparacb' ----
Start the CRS on ONLY ONE node exclusively and without CRS:
[root@lparaca client]# crsctl start crs -excl -nocrs CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start 'ora.evmd' on 'lparaca' CRS-2672: Attempting to start 'ora.mdnsd' on 'lparaca' CRS-2676: Start of 'ora.evmd' on 'lparaca' succeeded CRS-2676: Start of 'ora.mdnsd' on 'lparaca' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'lparaca' CRS-2676: Start of 'ora.gpnpd' on 'lparaca' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lparaca' CRS-2672: Attempting to start 'ora.gipcd' on 'lparaca' CRS-2676: Start of 'ora.cssdmonitor' on 'lparaca' succeeded CRS-2676: Start of 'ora.gipcd' on 'lparaca' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'lparaca' CRS-2672: Attempting to start 'ora.diskmon' on 'lparaca' CRS-2676: Start of 'ora.diskmon' on 'lparaca' succeeded CRS-2676: Start of 'ora.cssd' on 'lparaca' succeeded CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lparaca' CRS-2672: Attempting to start 'ora.ctssd' on 'lparaca' CRS-2676: Start of 'ora.ctssd' on 'lparaca' succeeded CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lparaca' succeeded CRS-2672: Attempting to start 'ora.asm' on 'lparaca' CRS-2676: Start of 'ora.asm' on 'lparaca' succeeded
OCR is restore as follows:
[root@lparaca client]# ocrconfig -showbackup lparaca 2018/01/31 10:56:24 +DATA:/lparac-scan/OCRBACKUP/backup_20180131_105624.ocr.307.966855385 2960767134 [root@lparaca client]# ocrconfig -restore +DATA:/lparac-scan/OCRBACKUP/backup_20180131_105624.ocr.307.966855385 [root@lparaca client]#
[root@lparaca grid_12201]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. OFFLINE 51ffb7b88b334f26bf482590e403d674 () [] 2. ONLINE f8f88c54edec4f4cbfda6f9676c6bde6 (/dev/sdy1) [VTDISK] 3. ONLINE b88a12b132a94fafbf64befc48683a82 (/dev/sdz1) [VTDISK] Located 3 voting disk(s).
The diskgroup should be re-created OR the voting disks should be relocated to other diskgroup, whichever you choose. Since we have to re-establish our connection to the corrupted disk, I preferred to re-create the disk group:
[root@lparaca grid_12201]# crsctl delete css votedisk +VTDISK CRS-4611: Successful deletion of voting disk +VTDISK.
[root@lparaca grid_12201]# su - oracle Last login: Thu Feb 1 05:39:24 EST 2018 on pts/0 [oracle@lparaca ~]$ . oraenv ORACLE_SID = [oracle] ? +ASM1 The Oracle base has been set to /u01/app/oracle/product [oracle@lparaca ~]$ sqlplus / as sysasm SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 1 05:40:39 2018 Copyright (c) 1982, 2016, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production SQL> drop diskgroup VTDISK force including contents; Diskgroup dropped. SQL> CREATE DISKGROUP VTDisk NORMAL REDUNDANCY DISK '/dev/sdaa1' SIZE 5119M DISK '/dev/sdy1' SIZE 5119M DISK '/dev/sdz1' SIZE 5119M ATTRIBUTE 'compatible.asm'='12.2.0.1','au_size'='4M'; 2 3 4 5 Diskgroup created. SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
Restore of the voting disks is also simple:
[root@lparaca client]# crsctl replace votedisk +VTDISK Successful addition of voting disk 77385c1044b34f9abf0c0682934b219d. Successful addition of voting disk 608ce542beee4f18bf2c85a6eaca39f4. Successful addition of voting disk 219485dcd7e04fecbf18b010157b6464. Successfully replaced voting disk group with +VTDISK. CRS-4266: Voting file(s) successfully replaced [root@lparaca client]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 77385c1044b34f9abf0c0682934b219d (/dev/sdaa1) [VTDISK] 2. ONLINE 608ce542beee4f18bf2c85a6eaca39f4 (/dev/sdy1) [VTDISK] 3. ONLINE 219485dcd7e04fecbf18b010157b6464 (/dev/sdz1) [VTDISK] Located 3 voting disk(s).
[root@lparaca grid_12201]# crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lparaca' CRS-2673: Attempting to stop 'ora.mdnsd' on 'lparaca' CRS-2673: Attempting to stop 'ora.gpnpd' on 'lparaca' CRS-2673: Attempting to stop 'ora.ctssd' on 'lparaca' CRS-2673: Attempting to stop 'ora.evmd' on 'lparaca' CRS-2673: Attempting to stop 'ora.asm' on 'lparaca' CRS-2677: Stop of 'ora.mdnsd' on 'lparaca' succeeded CRS-2677: Stop of 'ora.ctssd' on 'lparaca' succeeded CRS-2677: Stop of 'ora.evmd' on 'lparaca' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'lparaca' succeeded CRS-2677: Stop of 'ora.asm' on 'lparaca' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'lparaca' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'lparaca' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'lparaca' CRS-2677: Stop of 'ora.cssd' on 'lparaca' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'lparaca' CRS-2677: Stop of 'ora.gipcd' on 'lparaca' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lparaca' has completed CRS-4133: Oracle High Availability Services has been stopped.
Start CRS again normally on all nodes:
[root@lparaca client]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@lparaca client]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [root@lparaca client]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....SM.lsnr ora....er.type ONLINE ONLINE lparaca ora.DATA.dg ora....up.type ONLINE ONLINE lparaca ora....ER.lsnr ora....er.type ONLINE ONLINE lparaca ora....N1.lsnr ora....er.type ONLINE ONLINE lparacb ora....N2.lsnr ora....er.type ONLINE ONLINE lparaca ora....N3.lsnr ora....er.type ONLINE ONLINE lparaca ora.MGMTLSNR ora....nr.type ONLINE ONLINE lparaca ora.VTDISK.dg ora....up.type ONLINE ONLINE lparaca ora.asm ora.asm.type ONLINE ONLINE lparaca ora.cvu ora.cvu.type ONLINE ONLINE lparaca ora....SM1.asm application ONLINE ONLINE lparaca ora....CA.lsnr application ONLINE ONLINE lparaca ora....aca.ons application ONLINE ONLINE lparaca ora....aca.vip ora....t1.type ONLINE ONLINE lparaca ora....SM2.asm application ONLINE ONLINE lparacb ora....CB.lsnr application ONLINE ONLINE lparacb ora....acb.ons application ONLINE ONLINE lparacb ora....acb.vip ora....t1.type ONLINE ONLINE lparacb ora....network ora....rk.type ONLINE ONLINE lparaca ora.ons ora.ons.type ONLINE ONLINE lparaca ora.orcl.db ora....se.type ONLINE ONLINE lparacb ora.qosmserver ora....er.type ONLINE ONLINE lparaca ora.scan1.vip ora....ip.type ONLINE ONLINE lparacb ora.scan2.vip ora....ip.type ONLINE ONLINE lparaca ora.scan3.vip ora....ip.type ONLINE ONLINE lparaca [root@lparaca client]#
Since the ASM parameter file should reside in ASM, it can happen that the ASM should be restored as well. However in such case , both the ASM Spfile and the password file should be restored as follows:
[oracle@lparaca dbs]$ sqlplus / as sysasm SQL*Plus: Release 12.2.0.1.0 Production on Sun Feb 4 12:07:04 2018 Copyright (c) 1982, 2016, Oracle. All rights reserved. Connected to an idle instance. SQL> startup pfile='/u01/app/oracle/grid_12201/dbs/init+ASM1.ora'; ASM instance started Total System Global Area 1140850688 bytes Fixed Size 8629704 bytes Variable Size 1107055160 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@lparaca dbs]$ asmcmd ASMCMD> spget +DATA/lparac-scan/ASMPARAMETERFILE/registry.253.967204631 ASMCMD> exit [oracle@lparaca dbs]$ [oracle@lparaca dbs]$ sqlplus / as sysasm SQL*Plus: Release 12.2.0.1.0 Production on Sun Feb 4 12:08:55 2018 Copyright (c) 1982, 2016, Oracle. All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production SQL> create spfile='+DATA' from pfile; File created. SQL>exit [oracle@lparaca dbs]$ asmcmd -p ASMCMD [+] > spget +DATA/lparac-scan/ASMPARAMETERFILE/registry.253.967205371 ASMCMD [+] > ls -lart +DATA/lparac-scan/ASMPARAMETERFILE/registry.253.967205371 WARNING:option 'a' is deprecated for 'ls' please use 'absolutepath' WARNING:option 'r' is deprecated for 'ls' please use 'reverse' Type Redund Striped Time Sys Name ASMPARAMETERFILE UNPROT COARSE FEB 04 12:00:00 Y none => registry.253.967205371 ASMCMD [+] >
The ASM Password file can really be pain in the….arm. In order to solve teh ASM password there are couple problems which you should address: (e.g. MOS: 2139591.1) In a nutshell, you should:
The creation of a new user and new password file is fairly easy:
ASMCMD> pwdelete --asm OPW-00022: The password file does not exist. ASMCMD-9462: could not delete password file ASMCMD> exit
Create a new password file either via:
[oracle@lparaca ~]$ orapwd file=+DATA/orapwASM asm=y Enter password for SYS: [oracle@lparaca ~]$
or
[oracle@lparaca admin]$ asmcmd -p ASMCMD [+] > pwcreate --asm '+DATA/orapwasm' 'PASSWORD' ASMCMD [+] > pwget --asm +DATA/orapwasm ASMCMD [+] > exit [oracle@lparaca admin]$
Create a new user with SYSDBA and SYSASM Roles
[oracle@lparaca ~]$ asmcmd
cdASMCMD> +DATA
ASMCMD> ls -lart
WARNING:option 'a' is deprecated for 'ls'
please use 'absolutepath'
WARNING:option 'r' is deprecated for 'ls'
please use 'reverse'
Type Redund Striped Time Sys Name
Y ASM/
Y lparac-scan/
PASSWORD UNPROT COARSE FEB 04 14:00:00 N orapwasm => +DATA/ASM/PASSWORD/pwdasm.256.967212023
ASMCMD> exit
[oracle@lparaca ~]$ asmcmd
ASMCMD> lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE FALSE
ASMCMD> exit
[oracle@lparaca ~]$ asmcmd orapwusr --add CRSUSER__ASM_001
Enter password: *********
[oracle@lparaca ~]$
[oracle@lparaca ~]$
[oracle@lparaca ~]$ asmcmd orapwusr --grant sysasm CRSUSER__ASM_001
[oracle@lparaca ~]$ asmcmd orapwusr --grant sysdba CRSUSER__ASM_001
[oracle@lparaca ~]$ asmcmd lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE FALSE
CRSUSER__ASM_001 TRUE FALSE TRUE
[oracle@lparaca admin]$
This step was pain in the….arm because I was missing the PATH to the credential domain:
[oracle@lparaca ~]$ crsctl query credmaint -path ASM/Self -credtype userpass Path Credtype ID Attrs <- No path :)
So to solve that, I assumed that the path displayed in the trace file of the 2nd node is correct and just run with it.
ags-2 0x0, usrFlags 0x0 }, name 'traceLevel', val 0x7f6d3e910004, len 4, flags 0x0
2018-02-04 15:05:43.229 : CSSCLNT:1049790208: clsssinit: initialized context: (0x7f6d182e3650) flags 0x115
2018-02-04 15:05:43.230 : CSSCLNT:1049790208: clsssterm: terminating context (0x7f6d182e3650)
2018-02-04 15:05:43.230 : CLSNS:1049790208: clsns_SetTraceLevel:trace level set to 1.
2018-02-04 15:05:43.231 : default:1049790208: Inited LSF context: 0x7f6d18491ca0
2018-02-04 15:05:43.232 : CLSCRED:1049790208: clsCredCommonInit: Inited singleton credctx.
2018-02-04 15:05:43.232 : CLSCRED:1049790208: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.
2018-02-04 15:05:43.250 : USRTHRD:1049790208: {0:5:3} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS
2018-02-04 15:05:43.254 : USRTHRD:1049790208: {0:5:3} 8033 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS
2018-02-04 15:05:43.284 : CLSCRED:1049790208: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.1684dac9c18e7fbbbf9c6b4ccd6e0694.root not found <- This one
2018-02-04 15:05:43.284 : USRTHRD:1049790208: {0:5:3} 7755 Error 4 opening dom root in 0x7f6d1826c940
2018-02-04 15:05:43.287 :GIPCXCPT:1049790208: gipcInternalSetAttribute: failed during gipcInternalSetAttribute, ret gipcretInvalidAttribute (5)
2018-02-04 15:05:43.287 :GIPCXCPT:1049790208: gipcSetAttributeNativeF [clscrsconGipcConnect : clscrscon.c : 655]: EXCEPTION[ ret gipcretInvalidAttribute (5) ] failure for obj 0x7f6d1838c460 [00000000000b6145] { gipcEndpoint : localAddr '', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp (nil) status 13flags 0x20000000, flags-2 0x0, usrFlags 0x0 }, name 'traceLevel', val 0x7f6d3e910004, len 4, flags 0x0
[oracle@lparaca admin]$ crsctl get credmaint -path /ASM/Self/1684dac9c18e7fbbbf9c6b4ccd6e0694 -credtype userpass -id 0 -attr user -local CRSUSER__ASM_001 [oracle@lparaca admin]$ crsctl get credmaint -path /ASM/Self/1684dac9c18e7fbbbf9c6b4ccd6e0694 -credtype userpass -id 0 -attr passwd -local rjJgN3D1qk7cBMk1UnkbQqkILaV58 <- This damn hash is IMPORTANT.
[oracle@lparaca admin]$ asmcmd orapwusr --modify CRSUSER__ASM_001 Enter password: *****************************
[root@lparacb trace]# crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lparacb' CRS-2673: Attempting to stop 'ora.ctssd' on 'lparacb' CRS-2673: Attempting to stop 'ora.evmd' on 'lparacb' CRS-2673: Attempting to stop 'ora.storage' on 'lparacb' CRS-2673: Attempting to stop 'ora.crf' on 'lparacb' CRS-2673: Attempting to stop 'ora.mdnsd' on 'lparacb' CRS-2673: Attempting to stop 'ora.gpnpd' on 'lparacb' CRS-2677: Stop of 'ora.crf' on 'lparacb' succeeded CRS-2677: Stop of 'ora.evmd' on 'lparacb' succeeded CRS-2677: Stop of 'ora.ctssd' on 'lparacb' succeeded CRS-2677: Stop of 'ora.storage' on 'lparacb' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'lparacb' CRS-2677: Stop of 'ora.asm' on 'lparacb' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'lparacb' CRS-2677: Stop of 'ora.mdnsd' on 'lparacb' succeeded CRS-2677: Stop of 'ora.gpnpd' on 'lparacb' succeeded CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'lparacb' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'lparacb' CRS-2677: Stop of 'ora.cssd' on 'lparacb' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'lparacb' CRS-2677: Stop of 'ora.gipcd' on 'lparacb' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lparacb' has completed CRS-4133: Oracle High Availability Services has been stopped. [root@lparacb trace]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. [root@lparacb trace]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [root@lparacb trace]#
[root@lparacb trace]# crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....SM.lsnr ora....er.type ONLINE ONLINE lparaca ora.DATA.dg ora....up.type ONLINE ONLINE lparaca ora....ER.lsnr ora....er.type ONLINE ONLINE lparaca ora....N1.lsnr ora....er.type ONLINE ONLINE lparacb ora....N2.lsnr ora....er.type ONLINE ONLINE lparaca ora....N3.lsnr ora....er.type ONLINE ONLINE lparaca ora.MGMTLSNR ora....nr.type ONLINE ONLINE lparaca ora.VTDISK.dg ora....up.type ONLINE ONLINE lparaca ora.asm ora.asm.type ONLINE ONLINE lparaca ora.cvu ora.cvu.type ONLINE ONLINE lparaca ora....SM1.asm application ONLINE ONLINE lparaca ora....CA.lsnr application ONLINE ONLINE lparaca ora....aca.ons application ONLINE ONLINE lparaca ora....aca.vip ora....t1.type ONLINE ONLINE lparaca ora....SM2.asm application ONLINE ONLINE lparacb ora....CB.lsnr application ONLINE ONLINE lparacb ora....acb.ons application ONLINE ONLINE lparacb ora....acb.vip ora....t1.type ONLINE ONLINE lparacb ora....network ora....rk.type ONLINE ONLINE lparaca ora.ons ora.ons.type ONLINE ONLINE lparaca ora.qosmserver ora....er.type ONLINE ONLINE lparaca ora.scan1.vip ora....ip.type ONLINE ONLINE lparacb ora.scan2.vip ora....ip.type ONLINE ONLINE lparaca ora.scan3.vip ora....ip.type ONLINE ONLINE lparaca [root@lparacb trace]#