=====Overview===== Now, disk in ASM diskgroup CAN get corrupted and depends on the way it was corrupted, MAYBE it can be fixed. From my experience there are couple ways how this can happen: * OS Overlapped with the ASM disk * Manual Corruption * Oracle Bug So let's see what we can do: =====Oracle Bug===== Now the disk can get corrupted for a lot of reasons and that's why it is always good to have NORMAL or HIGH redundancy. In either case, let's see what happens if we can fix the corruption: [root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfod status=TRUE asm_diskstring='/dev/oradev' disk=ALL -------------------------------------------------------------------------------- Disk Size Header Path User Group ================================================================================ 1: 10236 Mb MEMBER /dev/oradev/ora-data1 oracle dba 2: 25595 Mb CANDIDATE /dev/oradev/ora-data10 oracle dba 3: 10236 Mb MEMBER /dev/oradev/ora-data2 oracle dba 4: 10236 Mb MEMBER /dev/oradev/ora-data3 oracle dba 5: 10236 Mb MEMBER /dev/oradev/ora-data4 oracle dba 6: 10236 Mb MEMBER /dev/oradev/ora-data5 oracle dba 7: 10236 Mb CANDIDATE /dev/oradev/ora-data6 oracle dba 8: 51199 Mb MEMBER /dev/oradev/ora-data7 oracle dba 9: 25595 Mb CANDIDATE /dev/oradev/ora-data8 oracle dba 10: 25595 Mb MEMBER /dev/oradev/ora-data9 oracle dba 11: 10236 Mb MEMBER /dev/oradev/ora-fra1 oracle dba 12: 10236 Mb MEMBER /dev/oradev/ora-fra2 oracle dba 13: 10236 Mb MEMBER /dev/oradev/ora-fra3 oracle dba 14: 10236 Mb MEMBER /dev/oradev/ora-fra4 oracle dba 15: 10236 Mb MEMBER /dev/oradev/ora-fra5 oracle dba 16: 10236 Mb CANDIDATE /dev/oradev/ora-fra6 oracle dba <- Corrupted, it shows is a candidate 17: 5114 Mb MEMBER /dev/oradev/ora-fra7 oracle dba 18: 25595 Mb MEMBER /dev/oradev/ora-fra8 oracle dba 19: 25595 Mb CANDIDATE /dev/oradev/ora-fra9 oracle dba -------------------------------------------------------------------------------- ORACLE_SID ORACLE_HOME ================================================================================ +ASM /u01/app/oracle/product/11.3.0/grid Disk number "fra6" is a corrupted one, even though it shows it as a candidate. This can be seen from: [root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfed read /dev/oradev/ora-fra6 aun=0 blkn=0 kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 0 ; 0x001: 0x00 kfbh.type: 0 ; 0x002: KFBTYP_INVALID kfbh.datfmt: 0 ; 0x003: 0x00 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 0 ; 0x008: file=0 kfbh.check: 0 ; 0x00c: 0x00000000 kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 2B2530EBB400 00000000 00000000 00000000 00000000 [................] Repeat 31 times 2B2530EBB600 4542414C 454E4F4C 00000001 00000000 [LABELONE........] 2B2530EBB610 5133D14F 00000020 324D564C 31303020 [O.3Q ...LVM2 001] 2B2530EBB620 3336334E 314A4643 49585056 63396C5A [N363CFJ1VPXIZl9c] 2B2530EBB630 576B484A 306C6455 4C4B6F52 58567A33 [JHkWUdl0RoKL3zVX] 2B2530EBB640 7FCB3400 00000002 00030000 00000000 [.4..............] 2B2530EBB650 00000000 00000000 00000000 00000000 [................] 2B2530EBB660 00000000 00000000 00001000 00000000 [................] 2B2530EBB670 0002F000 00000000 00000000 00000000 [................] 2B2530EBB680 00000000 00000000 00000000 00000000 [................] Repeat 215 times That can be repaired as follows: [root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfed repair /dev/oradev/ora-fra6 aun=0 blkn=0 [root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfed read /dev/oradev/ora-fra6 aun=0 blkn=0 kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: blk=0 kfbh.block.obj: 2147483653 ; 0x008: disk=5 kfbh.check: 2061566458 ; 0x00c: 0x7ae101fa kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8 kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000 kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000 In the summary, now this disk appears as a member: [root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfod status=TRUE asm_diskstring='/dev/oradev' disk=ALL -------------------------------------------------------------------------------- Disk Size Header Path User Group ================================================================================ 1: 10236 Mb MEMBER /dev/oradev/ora-data1 oracle dba 2: 25595 Mb CANDIDATE /dev/oradev/ora-data10 oracle dba 3: 10236 Mb MEMBER /dev/oradev/ora-data2 oracle dba 4: 10236 Mb MEMBER /dev/oradev/ora-data3 oracle dba 5: 10236 Mb MEMBER /dev/oradev/ora-data4 oracle dba 6: 10236 Mb MEMBER /dev/oradev/ora-data5 oracle dba 7: 10236 Mb CANDIDATE /dev/oradev/ora-data6 oracle dba 8: 51199 Mb MEMBER /dev/oradev/ora-data7 oracle dba 9: 25595 Mb CANDIDATE /dev/oradev/ora-data8 oracle dba 10: 25595 Mb MEMBER /dev/oradev/ora-data9 oracle dba 11: 10236 Mb MEMBER /dev/oradev/ora-fra1 oracle dba 12: 10236 Mb MEMBER /dev/oradev/ora-fra2 oracle dba 13: 10236 Mb MEMBER /dev/oradev/ora-fra3 oracle dba 14: 10236 Mb MEMBER /dev/oradev/ora-fra4 oracle dba 15: 10236 Mb MEMBER /dev/oradev/ora-fra5 oracle dba 16: 10236 Mb MEMBER /dev/oradev/ora-fra6 oracle dba <- Now a member 17: 5114 Mb MEMBER /dev/oradev/ora-fra7 oracle dba 18: 25595 Mb MEMBER /dev/oradev/ora-fra8 oracle dba 19: 25595 Mb CANDIDATE /dev/oradev/ora-fra9 oracle dba -------------------------------------------------------------------------------- ORACLE_SID ORACLE_HOME ================================================================================ +ASM /u01/app/oracle/product/11.3.0/grid