oracle_grid_sa_troubleshooting_asm_diskheader_corruption

Now, disk in ASM diskgroup CAN get corrupted and depends on the way it was corrupted, MAYBE it can be fixed. From my experience there are couple ways how this can happen:

  • OS Overlapped with the ASM disk
  • Manual Corruption
  • Oracle Bug

So let's see what we can do:

Now the disk can get corrupted for a lot of reasons and that's why it is always good to have NORMAL or HIGH redundancy. In either case, let's see what happens if we can fix the corruption:

Check the header of the disk

[root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfod status=TRUE asm_diskstring='/dev/oradev' disk=ALL
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                     User     Group
================================================================================
   1:      10236 Mb MEMBER    /dev/oradev/ora-data1                    oracle   dba
   2:      25595 Mb CANDIDATE /dev/oradev/ora-data10                   oracle   dba
   3:      10236 Mb MEMBER    /dev/oradev/ora-data2                    oracle   dba
   4:      10236 Mb MEMBER    /dev/oradev/ora-data3                    oracle   dba
   5:      10236 Mb MEMBER    /dev/oradev/ora-data4                    oracle   dba
   6:      10236 Mb MEMBER    /dev/oradev/ora-data5                    oracle   dba
   7:      10236 Mb CANDIDATE /dev/oradev/ora-data6                    oracle   dba
   8:      51199 Mb MEMBER    /dev/oradev/ora-data7                    oracle   dba
   9:      25595 Mb CANDIDATE /dev/oradev/ora-data8                    oracle   dba
  10:      25595 Mb MEMBER    /dev/oradev/ora-data9                    oracle   dba
  11:      10236 Mb MEMBER    /dev/oradev/ora-fra1                     oracle   dba
  12:      10236 Mb MEMBER    /dev/oradev/ora-fra2                     oracle   dba
  13:      10236 Mb MEMBER    /dev/oradev/ora-fra3                     oracle   dba
  14:      10236 Mb MEMBER    /dev/oradev/ora-fra4                     oracle   dba
  15:      10236 Mb MEMBER    /dev/oradev/ora-fra5                     oracle   dba
  16:      10236 Mb CANDIDATE /dev/oradev/ora-fra6                     oracle   dba  <- Corrupted, it shows is a candidate
  17:       5114 Mb MEMBER    /dev/oradev/ora-fra7                     oracle   dba
  18:      25595 Mb MEMBER    /dev/oradev/ora-fra8                     oracle   dba
  19:      25595 Mb CANDIDATE /dev/oradev/ora-fra9                     oracle   dba
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /u01/app/oracle/product/11.3.0/grid

Disk number “fra6” is a corrupted one, even though it shows it as a candidate. This can be seen from:

Corrupted header

[root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfed read /dev/oradev/ora-fra6 aun=0 blkn=0
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
2B2530EBB400 00000000 00000000 00000000 00000000  [................]
        Repeat 31 times
2B2530EBB600 4542414C 454E4F4C 00000001 00000000  [LABELONE........]
2B2530EBB610 5133D14F 00000020 324D564C 31303020  [O.3Q ...LVM2 001]
2B2530EBB620 3336334E 314A4643 49585056 63396C5A  [N363CFJ1VPXIZl9c]
2B2530EBB630 576B484A 306C6455 4C4B6F52 58567A33  [JHkWUdl0RoKL3zVX]
2B2530EBB640 7FCB3400 00000002 00030000 00000000  [.4..............]
2B2530EBB650 00000000 00000000 00000000 00000000  [................]
2B2530EBB660 00000000 00000000 00001000 00000000  [................]
2B2530EBB670 0002F000 00000000 00000000 00000000  [................]
2B2530EBB680 00000000 00000000 00000000 00000000  [................]
  Repeat 215 times

That can be repaired as follows:

Repair not 0 AU corruption in ASM Disk

[root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfed repair /dev/oradev/ora-fra6 aun=0 blkn=0
[root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfed read /dev/oradev/ora-fra6 aun=0 blkn=0
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:              2147483653 ; 0x008: disk=5
kfbh.check:                  2061566458 ; 0x00c: 0x7ae101fa
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000

In the summary, now this disk appears as a member:

ASM Disk Summary

[root@oracrs ~]# /u01/app/oracle/product/11.3.0/grid/bin/kfod status=TRUE asm_diskstring='/dev/oradev' disk=ALL
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                     User     Group
================================================================================
   1:      10236 Mb MEMBER    /dev/oradev/ora-data1                    oracle   dba
   2:      25595 Mb CANDIDATE /dev/oradev/ora-data10                   oracle   dba
   3:      10236 Mb MEMBER    /dev/oradev/ora-data2                    oracle   dba
   4:      10236 Mb MEMBER    /dev/oradev/ora-data3                    oracle   dba
   5:      10236 Mb MEMBER    /dev/oradev/ora-data4                    oracle   dba
   6:      10236 Mb MEMBER    /dev/oradev/ora-data5                    oracle   dba
   7:      10236 Mb CANDIDATE /dev/oradev/ora-data6                    oracle   dba
   8:      51199 Mb MEMBER    /dev/oradev/ora-data7                    oracle   dba
   9:      25595 Mb CANDIDATE /dev/oradev/ora-data8                    oracle   dba
  10:      25595 Mb MEMBER    /dev/oradev/ora-data9                    oracle   dba
  11:      10236 Mb MEMBER    /dev/oradev/ora-fra1                     oracle   dba
  12:      10236 Mb MEMBER    /dev/oradev/ora-fra2                     oracle   dba
  13:      10236 Mb MEMBER    /dev/oradev/ora-fra3                     oracle   dba
  14:      10236 Mb MEMBER    /dev/oradev/ora-fra4                     oracle   dba
  15:      10236 Mb MEMBER    /dev/oradev/ora-fra5                     oracle   dba
  16:      10236 Mb MEMBER    /dev/oradev/ora-fra6                     oracle   dba  <- Now a member
  17:       5114 Mb MEMBER    /dev/oradev/ora-fra7                     oracle   dba
  18:      25595 Mb MEMBER    /dev/oradev/ora-fra8                     oracle   dba
  19:      25595 Mb CANDIDATE /dev/oradev/ora-fra9                     oracle   dba
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /u01/app/oracle/product/11.3.0/grid
  • oracle_grid_sa_troubleshooting_asm_diskheader_corruption.txt
  • Last modified: 2020/03/19 15:37
  • by andonovj