oracle_grid_sa_br_ocr_12c

Overview

With 12c, restore and recover of the voting disks and the OCR is quite simple since Oracle remembers the exact physical place of the OCR and voting disks on the ASM.

This scenario assumes:

  • The ASM Spfile is on separate diskgroup than the voting disks(Check “Restore ASM” if they are on the same)
  • The voting disk is on NORMAL redundancy and has 3 disks only.

In a Nutshell

In nutshell, the procedure is:

  1. Corrupt the data. (P.S. don't do that :D This is only to show you what happens IF it is corrupted)
  2. Stop CRS on ALL Nodes
  3. Start CRS exclusively on one node
  4. Restore OCR
  5. Restore voting disks

Corrupt the Voting Disks

Corruption is done via the “dd” command: 

[root@lparaca grid_12201]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   51ffb7b88b334f26bf482590e403d674 (/dev/sdaa1) [VTDISK] <- Disk to corrupt
 2. ONLINE   f8f88c54edec4f4cbfda6f9676c6bde6 (/dev/sdy1) [VTDISK]
 3. ONLINE   b88a12b132a94fafbf64befc48683a82 (/dev/sdz1) [VTDISK]
Located 3 voting disk(s).
[root@lparaca grid_12201]# dd if=/dev/zero of=/dev/sdaa1 bs=2048 count=10
10+0 records in
10+0 records out
20480 bytes (20 kB) copied, 7.4799e-05 s, 274 MB/s

Corrupted Disk

You can check the corruption by checking the header of the corrupted disk and a normal disk: 

[root@lparaca grid_12201]# kfed read /dev/sdaa1 | egrep 'name|size|type'

kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
KFED-00322: invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[root@lparaca grid_12201]# od /dev/sdaa1 | head
0000000 000000 000000 000000 000000 000000 000000 000000 000000
*
0050000 101001 001003 000005 000000 000000 100000 104516 106017
0050020 000000 000000 000000 000000 000000 000000 000000 000000
0050040 002500 000000 000000 000000 000010 000010 000014 000014
0050060 000020 000020 000024 000024 000030 000030 000034 000034
0050100 000050 006050 000000 000000 001050 000040 000006 000000
0050120 000000 000000 000000 000000 000000 000000 000000 000000
*
0051100 000000 000000 000000 000000 002050 000050 000006 000000


[root@lparaca client]# od -c /dev/sdaa1 | head
0000000  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
0050000 001 202 003 002 005  \0  \0  \0  \0  \0  \0 200   N 211 017 214
0050020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0050040   @ 005  \0  \0  \0  \0  \0  \0  \b  \0  \b  \0  \f  \0  \f  \0
0050060 020  \0 020  \0 024  \0 024  \0 030  \0 030  \0 034  \0 034  \0
0050100   (  \0   (  \f  \0  \0  \0  \0   ( 002      \0 006  \0  \0  \0
0050120  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
*
0051100  \0  \0  \0  \0  \0  \0  \0  \0   ( 004   (  \0 006  \0  \0  \0
[root@lparaca client]#

Normal Disk

You can compare this result with: 

[root@lparaca grid_12201]# kfed read /dev/sdy1 | egrep 'name|size|type'
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dskname:             VTDISK_0001 ; 0x028: length=11
kfdhdb.grpname:                  VTDISK ; 0x048: length=6
kfdhdb.fgname:              VTDISK_0001 ; 0x068: length=11
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  4194304 ; 0x0bc: 0x00400000
kfdhdb.dsksize:                    1279 ; 0x0c4: 0x000004ff

[root@lparaca grid_12201]# od /dev/sdy1 | head
0000000 101001 000401 000000 000000 000001 100000 051547 003637
0000020 000000 000000 000000 000000 000000 000000 000000 000000
0000040 051117 046103 044504 045523 000000 000000 000000 000000
0000060 000000 000000 000000 000000 000000 000000 000000 000000
0000100 000400 006040 000001 001402 052126 044504 045523 030137
0000120 030060 000061 000000 000000 000000 000000 000000 000000
0000140 000000 000000 000000 000000 052126 044504 045523 000000
0000160 000000 000000 000000 000000 000000 000000 000000 000000
0000200 000000 000000 000000 000000 052126 044504 045523 030137
0000220 030060 000061 000000 000000 000000 000000 000000 000000



[root@lparaca client]# od -c /dev/sdaa1 | head
0000000 001 202 001 001  \0  \0  \0  \0  \0  \0  \0 200 037 225 031   =
0000020  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000040   O   R   C   L   D   I   S   K  \0  \0  \0  \0  \0  \0  \0  \0
0000060  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000100  \0 001      \f  \0  \0 002 003   V   T   D   I   S   K   _   0
0000120   0   0   0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000140  \0  \0  \0  \0  \0  \0  \0  \0   V   T   D   I   S   K  \0  \0
0000160  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000200  \0  \0  \0  \0  \0  \0  \0  \0   V   T   D   I   S   K   _   0
0000220   0   0   0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
[root@lparaca client]#

Shutdown CRS

CRS should be brought down to all instances in the cluster, feel free to use the -f option in case of errors: 

[root@lparaca grid_12201]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lparaca'
CRS-2673: Attempting to stop 'ora.crsd' on 'lparaca'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'lparaca'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'lparaca'
CRS-2677: Stop of 'ora.orcl.db' on 'lparaca' succeeded
CRS-2673: Attempting to stop 'ora.VTDISK.dg' on 'lparaca'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'lparaca'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'lparaca'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'lparaca'
CRS-2677: Stop of 'ora.VTDISK.dg' on 'lparaca' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'lparaca' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'lparaca'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'lparaca' succeeded
CRS-2673: Attempting to stop 'ora.lparaca.vip' on 'lparaca'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'lparaca' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'lparaca'
CRS-2677: Stop of 'ora.asm' on 'lparaca' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'lparaca'
CRS-2677: Stop of 'ora.lparaca.vip' on 'lparaca' succeeded
CRS-2677: Stop of 'ora.scan1.vip' on 'lparaca' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'lparaca' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'lparacb'
CRS-2672: Attempting to start 'ora.lparaca.vip' on 'lparacb'
----

Start CRS stack Exclusively

Start the CRS on ONLY ONE node exclusively and without CRS: 

[root@lparaca client]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'lparaca'
CRS-2672: Attempting to start 'ora.mdnsd' on 'lparaca'
CRS-2676: Start of 'ora.evmd' on 'lparaca' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'lparaca' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'lparaca'
CRS-2676: Start of 'ora.gpnpd' on 'lparaca' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'lparaca'
CRS-2672: Attempting to start 'ora.gipcd' on 'lparaca'
CRS-2676: Start of 'ora.cssdmonitor' on 'lparaca' succeeded
CRS-2676: Start of 'ora.gipcd' on 'lparaca' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'lparaca'
CRS-2672: Attempting to start 'ora.diskmon' on 'lparaca'
CRS-2676: Start of 'ora.diskmon' on 'lparaca' succeeded
CRS-2676: Start of 'ora.cssd' on 'lparaca' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'lparaca'
CRS-2672: Attempting to start 'ora.ctssd' on 'lparaca'
CRS-2676: Start of 'ora.ctssd' on 'lparaca' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'lparaca' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'lparaca'
CRS-2676: Start of 'ora.asm' on 'lparaca' succeeded

Restore the OCR

OCR is restore as follows: 

[root@lparaca client]# ocrconfig -showbackup
lparaca     2018/01/31 10:56:24     +DATA:/lparac-scan/OCRBACKUP/backup_20180131_105624.ocr.307.966855385 2960767134

[root@lparaca client]# ocrconfig -restore +DATA:/lparac-scan/OCRBACKUP/backup_20180131_105624.ocr.307.966855385
[root@lparaca client]#

Check the voting disks

[root@lparaca grid_12201]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. OFFLINE  51ffb7b88b334f26bf482590e403d674 () []
 2. ONLINE   f8f88c54edec4f4cbfda6f9676c6bde6 (/dev/sdy1) [VTDISK]
 3. ONLINE   b88a12b132a94fafbf64befc48683a82 (/dev/sdz1) [VTDISK]
Located 3 voting disk(s).

Drop and Recreate Diskgroup

The diskgroup should be re-created OR the voting disks should be relocated to other diskgroup, whichever you choose. Since we have to re-establish our connection to the corrupted disk, I preferred to re-create the disk group:

Delete the voting disks from the corrupter diskgroup

[root@lparaca grid_12201]# crsctl delete css votedisk +VTDISK
CRS-4611: Successful deletion of voting disk +VTDISK.

Re-create the ASM Diskgroup

[root@lparaca grid_12201]# su - oracle
Last login: Thu Feb  1 05:39:24 EST 2018 on pts/0
[oracle@lparaca ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base has been set to /u01/app/oracle/product
[oracle@lparaca ~]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Thu Feb 1 05:40:39 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> drop diskgroup VTDISK force including contents;

Diskgroup dropped.

SQL> CREATE DISKGROUP VTDisk NORMAL REDUNDANCY
DISK '/dev/sdaa1' SIZE 5119M
DISK '/dev/sdy1' SIZE 5119M
DISK '/dev/sdz1' SIZE 5119M
ATTRIBUTE 'compatible.asm'='12.2.0.1','au_size'='4M';  2    3    4    5

Diskgroup created.

SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

Restore of the Voting Disks

Restore of the voting disks is also simple: 

[root@lparaca client]# crsctl replace votedisk +VTDISK
Successful addition of voting disk 77385c1044b34f9abf0c0682934b219d.
Successful addition of voting disk 608ce542beee4f18bf2c85a6eaca39f4.
Successful addition of voting disk 219485dcd7e04fecbf18b010157b6464.
Successfully replaced voting disk group with +VTDISK.
CRS-4266: Voting file(s) successfully replaced
[root@lparaca client]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   77385c1044b34f9abf0c0682934b219d (/dev/sdaa1) [VTDISK]
 2. ONLINE   608ce542beee4f18bf2c85a6eaca39f4 (/dev/sdy1) [VTDISK]
 3. ONLINE   219485dcd7e04fecbf18b010157b6464 (/dev/sdz1) [VTDISK]
Located 3 voting disk(s).

Shutdown CRS

[root@lparaca grid_12201]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lparaca'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'lparaca'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'lparaca'
CRS-2673: Attempting to stop 'ora.ctssd' on 'lparaca'
CRS-2673: Attempting to stop 'ora.evmd' on 'lparaca'
CRS-2673: Attempting to stop 'ora.asm' on 'lparaca'
CRS-2677: Stop of 'ora.mdnsd' on 'lparaca' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'lparaca' succeeded
CRS-2677: Stop of 'ora.evmd' on 'lparaca' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'lparaca' succeeded
CRS-2677: Stop of 'ora.asm' on 'lparaca' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'lparaca'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'lparaca' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'lparaca'
CRS-2677: Stop of 'ora.cssd' on 'lparaca' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'lparaca'
CRS-2677: Stop of 'ora.gipcd' on 'lparaca' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lparaca' has completed
CRS-4133: Oracle High Availability Services has been stopped.

Start CRS

Start CRS again normally on all nodes: 

[root@lparaca client]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@lparaca client]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@lparaca client]# crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM.lsnr ora....er.type ONLINE    ONLINE    lparaca
ora.DATA.dg    ora....up.type ONLINE    ONLINE    lparaca
ora....ER.lsnr ora....er.type ONLINE    ONLINE    lparaca
ora....N1.lsnr ora....er.type ONLINE    ONLINE    lparacb
ora....N2.lsnr ora....er.type ONLINE    ONLINE    lparaca
ora....N3.lsnr ora....er.type ONLINE    ONLINE    lparaca
ora.MGMTLSNR   ora....nr.type ONLINE    ONLINE    lparaca
ora.VTDISK.dg  ora....up.type ONLINE    ONLINE    lparaca
ora.asm        ora.asm.type   ONLINE    ONLINE    lparaca
ora.cvu        ora.cvu.type   ONLINE    ONLINE    lparaca
ora....SM1.asm application    ONLINE    ONLINE    lparaca
ora....CA.lsnr application    ONLINE    ONLINE    lparaca
ora....aca.ons application    ONLINE    ONLINE    lparaca
ora....aca.vip ora....t1.type ONLINE    ONLINE    lparaca
ora....SM2.asm application    ONLINE    ONLINE    lparacb
ora....CB.lsnr application    ONLINE    ONLINE    lparacb
ora....acb.ons application    ONLINE    ONLINE    lparacb
ora....acb.vip ora....t1.type ONLINE    ONLINE    lparacb
ora....network ora....rk.type ONLINE    ONLINE    lparaca
ora.ons        ora.ons.type   ONLINE    ONLINE    lparaca
ora.orcl.db    ora....se.type ONLINE    ONLINE    lparacb
ora.qosmserver ora....er.type ONLINE    ONLINE    lparaca
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    lparacb
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    lparaca
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    lparaca
[root@lparaca client]#

Restore ASM

Restore the ASM Instance

Since the ASM parameter file should reside in ASM, it can happen that the ASM should be restored as well. However in such case , both the ASM Spfile and the password file should be restored as follows:

Startup with pfile

[oracle@lparaca dbs]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Sun Feb 4 12:07:04 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>  startup pfile='/u01/app/oracle/grid_12201/dbs/init+ASM1.ora';
ASM instance started

Total System Global Area 1140850688 bytes
Fixed Size                  8629704 bytes
Variable Size            1107055160 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

Re-create the spfile

[oracle@lparaca dbs]$ asmcmd
ASMCMD> spget
+DATA/lparac-scan/ASMPARAMETERFILE/registry.253.967204631
ASMCMD> exit
[oracle@lparaca dbs]$
[oracle@lparaca dbs]$ sqlplus / as sysasm

SQL*Plus: Release 12.2.0.1.0 Production on Sun Feb 4 12:08:55 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> create spfile='+DATA' from pfile;

File created.

SQL>exit

[oracle@lparaca dbs]$ asmcmd -p
ASMCMD [+] > spget
+DATA/lparac-scan/ASMPARAMETERFILE/registry.253.967205371
ASMCMD [+] > ls -lart +DATA/lparac-scan/ASMPARAMETERFILE/registry.253.967205371
WARNING:option 'a' is deprecated for 'ls'
please use 'absolutepath'

WARNING:option 'r' is deprecated for 'ls'
please use 'reverse'

Type              Redund  Striped  Time             Sys  Name
ASMPARAMETERFILE  UNPROT  COARSE   FEB 04 12:00:00  Y    none => registry.253.967205371
ASMCMD [+] >

ASM Password File

The ASM Password file can really be pain in the….arm. In order to solve teh ASM password there are couple problems which you should address: (e.g. MOS: 2139591.1) In a nutshell, you should:

  • Create new user
  • Grant him SYSDBA and SYSASM
  • Modify the hash password ← This step was a real nightmare, you will see later why.

Create new user

The creation of a new user and new password file is fairly easy:

First be sure that all password files are deleted


ASMCMD> pwdelete --asm
OPW-00022: The password file does not exist.
ASMCMD-9462: could not delete password file
ASMCMD> exit

Create a new password file either via: 


[oracle@lparaca ~]$ orapwd file=+DATA/orapwASM asm=y

Enter password for SYS:
[oracle@lparaca ~]$

or 

[oracle@lparaca admin]$ asmcmd -p
ASMCMD [+] >  pwcreate --asm  '+DATA/orapwasm' 'PASSWORD'
ASMCMD [+] > pwget --asm
+DATA/orapwasm
ASMCMD [+] > exit
[oracle@lparaca admin]$

Create a new user with SYSDBA and SYSASM Roles 

[oracle@lparaca ~]$ asmcmd
cdASMCMD>  +DATA
ASMCMD> ls -lart
WARNING:option 'a' is deprecated for 'ls'
please use 'absolutepath'

WARNING:option 'r' is deprecated for 'ls'
please use 'reverse'

Type      Redund  Striped  Time             Sys  Name
                                            Y    ASM/
                                            Y    lparac-scan/
PASSWORD  UNPROT  COARSE   FEB 04 14:00:00  N    orapwasm => +DATA/ASM/PASSWORD/pwdasm.256.967212023
ASMCMD> exit
[oracle@lparaca ~]$ asmcmd
ASMCMD> lspwusr
Username sysdba sysoper sysasm
     SYS   TRUE    TRUE  FALSE
ASMCMD> exit
[oracle@lparaca ~]$ asmcmd orapwusr --add CRSUSER__ASM_001
Enter password: *********
[oracle@lparaca ~]$
[oracle@lparaca ~]$
[oracle@lparaca ~]$ asmcmd orapwusr --grant sysasm CRSUSER__ASM_001
[oracle@lparaca ~]$ asmcmd orapwusr --grant sysdba CRSUSER__ASM_001
[oracle@lparaca ~]$ asmcmd lspwusr
        Username sysdba sysoper sysasm
             SYS   TRUE    TRUE  FALSE
CRSUSER__ASM_001   TRUE   FALSE   TRUE
[oracle@lparaca admin]$

Extract the hash password

This step was pain in the….arm because I was missing the PATH to the credential domain: 

[oracle@lparaca ~]$ crsctl query credmaint -path ASM/Self -credtype userpass
Path                                           Credtype   ID   Attrs
<- No path :)

So to solve that, I assumed that the path displayed in the trace file of the 2nd node is correct and just run with it.

Tracefile (2nd Node)

ags-2 0x0, usrFlags 0x0 }, name 'traceLevel', val 0x7f6d3e910004, len 4, flags 0x0
2018-02-04 15:05:43.229 : CSSCLNT:1049790208: clsssinit: initialized context: (0x7f6d182e3650) flags 0x115
2018-02-04 15:05:43.230 : CSSCLNT:1049790208: clsssterm: terminating context (0x7f6d182e3650)
2018-02-04 15:05:43.230 :   CLSNS:1049790208: clsns_SetTraceLevel:trace level set to 1.
2018-02-04 15:05:43.231 : default:1049790208: Inited LSF context: 0x7f6d18491ca0
2018-02-04 15:05:43.232 : CLSCRED:1049790208: clsCredCommonInit: Inited singleton credctx.
2018-02-04 15:05:43.232 : CLSCRED:1049790208: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.
2018-02-04 15:05:43.250 : USRTHRD:1049790208: {0:5:3} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS

2018-02-04 15:05:43.254 : USRTHRD:1049790208: {0:5:3} 8033 Error 4 querying length of attr ASM_STATIC_DISCOVERY_ADDRESS

2018-02-04 15:05:43.284 : CLSCRED:1049790208: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.1684dac9c18e7fbbbf9c6b4ccd6e0694.root not found   <- This one
2018-02-04 15:05:43.284 : USRTHRD:1049790208: {0:5:3} 7755 Error 4 opening dom root in 0x7f6d1826c940

2018-02-04 15:05:43.287 :GIPCXCPT:1049790208:  gipcInternalSetAttribute: failed during gipcInternalSetAttribute, ret gipcretInvalidAttribute (5)
2018-02-04 15:05:43.287 :GIPCXCPT:1049790208:  gipcSetAttributeNativeF [clscrsconGipcConnect : clscrscon.c : 655]: EXCEPTION[ ret gipcretInvalidAttribute (5) ]  failure for obj 0x7f6d1838c460 [00000000000b6145] { gipcEndpoint : localAddr '', remoteAddr '', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj (nil), sendp (nil) status 13flags 0x20000000, flags-2 0x0, usrFlags 0x0 }, name 'traceLevel', val 0x7f6d3e910004, len 4, flags 0x0

Extract the cdentials

[oracle@lparaca admin]$ crsctl get credmaint -path /ASM/Self/1684dac9c18e7fbbbf9c6b4ccd6e0694 -credtype userpass -id 0 -attr user -local
CRSUSER__ASM_001
[oracle@lparaca admin]$ crsctl get credmaint -path /ASM/Self/1684dac9c18e7fbbbf9c6b4ccd6e0694 -credtype userpass -id 0 -attr passwd -local
rjJgN3D1qk7cBMk1UnkbQqkILaV58     <- This damn hash is IMPORTANT.

Change the password using the hash

[oracle@lparaca admin]$ asmcmd orapwusr --modify CRSUSER__ASM_001
Enter password: *****************************

Restart the CRS on the 2nd node

[root@lparacb trace]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'lparacb'
CRS-2673: Attempting to stop 'ora.ctssd' on 'lparacb'
CRS-2673: Attempting to stop 'ora.evmd' on 'lparacb'
CRS-2673: Attempting to stop 'ora.storage' on 'lparacb'
CRS-2673: Attempting to stop 'ora.crf' on 'lparacb'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'lparacb'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'lparacb'
CRS-2677: Stop of 'ora.crf' on 'lparacb' succeeded
CRS-2677: Stop of 'ora.evmd' on 'lparacb' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'lparacb' succeeded
CRS-2677: Stop of 'ora.storage' on 'lparacb' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'lparacb'
CRS-2677: Stop of 'ora.asm' on 'lparacb' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'lparacb'
CRS-2677: Stop of 'ora.mdnsd' on 'lparacb' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'lparacb' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'lparacb' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'lparacb'
CRS-2677: Stop of 'ora.cssd' on 'lparacb' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'lparacb'
CRS-2677: Stop of 'ora.gipcd' on 'lparacb' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'lparacb' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@lparacb trace]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@lparacb trace]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@lparacb trace]#

Check

[root@lparacb trace]# crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM.lsnr ora....er.type ONLINE    ONLINE    lparaca
ora.DATA.dg    ora....up.type ONLINE    ONLINE    lparaca
ora....ER.lsnr ora....er.type ONLINE    ONLINE    lparaca
ora....N1.lsnr ora....er.type ONLINE    ONLINE    lparacb
ora....N2.lsnr ora....er.type ONLINE    ONLINE    lparaca
ora....N3.lsnr ora....er.type ONLINE    ONLINE    lparaca
ora.MGMTLSNR   ora....nr.type ONLINE    ONLINE    lparaca
ora.VTDISK.dg  ora....up.type ONLINE    ONLINE    lparaca
ora.asm        ora.asm.type   ONLINE    ONLINE    lparaca
ora.cvu        ora.cvu.type   ONLINE    ONLINE    lparaca
ora....SM1.asm application    ONLINE    ONLINE    lparaca
ora....CA.lsnr application    ONLINE    ONLINE    lparaca
ora....aca.ons application    ONLINE    ONLINE    lparaca
ora....aca.vip ora....t1.type ONLINE    ONLINE    lparaca
ora....SM2.asm application    ONLINE    ONLINE    lparacb
ora....CB.lsnr application    ONLINE    ONLINE    lparacb
ora....acb.ons application    ONLINE    ONLINE    lparacb
ora....acb.vip ora....t1.type ONLINE    ONLINE    lparacb
ora....network ora....rk.type ONLINE    ONLINE    lparaca
ora.ons        ora.ons.type   ONLINE    ONLINE    lparaca
ora.qosmserver ora....er.type ONLINE    ONLINE    lparaca
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    lparacb
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    lparaca
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    lparaca
[root@lparacb trace]#

  • oracle_grid_sa_br_ocr_12c.txt
  • Last modified: 2019/10/18 20:04
  • by 127.0.0.1