oracle_goldengate_hadoop_hbase [My wiki]

Again, we have to configure:

Start HBase
Replicat config
HBase Handler Properties File

So let's get going.

Start HBase, HDFS and Yarn

[oracle@hostname ~]$ start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /opt/hadoop/logs/hadoop-oraclenamenode-
edvmr1p0.out
localhost: starting datanode, logging to /opt/hadoop/logs/hadoop-oracledatanode-
edvmr1p0.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/logs/hadooporacle-
secondarynamenode-edvmr1p0.out
[oracle@hostname ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/logs/yarn-oracleresourcemanager-
edvmr1p0.out
localhost: starting nodemanager, logging to /opt/hadoop/logs/yarn-oraclenodemanager-
edvmr1p0.out
[oracle@hostname ~]$ start-hbase.sh
localhost: starting zookeeper, logging to /opt/hbase/logs/hbase-oraclezookeeper-
edvmr1p0.out
starting master, logging to /opt/hbase/logs/hbase-oracle-master-edvmr1p0.out
starting regionserver, logging to /opt/hbase/logs/hbase-oracle-1-regionserveredvmr1p0.
out
[oracle@hostname ~]$
[oracle@hostname ~]$ $JAVA_HOME/bin/jps
21538 SecondaryNameNode
21346 DataNode
22148 NodeManager
22613 HMaster
22502 HQuorumPeer
21159 NameNode
23386 Jps
21851 ResourceManager
22747 HRegionServer
[oracle@hostname ~]$

Configure HBase Handler Properties File (TRG_OGGHIME/dirprm/base.properties

gg.handlerlist=hbase
gg.handler.hbase.type=hbase
gg.handler.hbase.hBaseColumnFamilyName=cf
gg.handler.hbase.keyValueDelimiter=CDATA[=]
gg.handler.hbase.keyValuePairDelimiter=CDATA[,]
gg.handler.hbase.encoding=UTF-8
gg.handler.hbase.pkUpdateHandling=delete-insert
gg.handler.hbase.nullValueRepresentation=CDATA[NULL]
gg.handler.hbase.authType=none
gg.handler.hbase.includeTokens=false
gg.handler.hbase.mode=tx
goldengate.userexit.timestamp=utc
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE
gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec
gg.classpath=/opt/hbase/lib/*:/opt/hbase/conf/:
jvm.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar

Now, we can configure and start the REPLICAT

Configure Replicat

[oracle@edvmr1p0 oggtrg]$ ggsci

Oracle GoldenGate Command Interpreter
Version 12.2.0.1.160823 OGGCORE_OGGADP.12.2.0.1.0_PLATFORMS_161019.1437
Linux, x64, 64bit (optimized), Generic on Oct 19 2016 16:01:40
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2016, Oracle and/or its affiliates. All rights reserved.

GGSCI (edvmr1p0) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           

GGSCI (edvmr1p0) 2> view param mgr

Port 7909
DynamicPortList 20100-20199
PurgeOldExtracts ./dirdat/pe*, UseCheckPoints, MinKeepHours 2
Autostart Replicat R*
AUTORESTART Replicat *, WaitMinutes 1, Retries 3


GGSCI (edvmr1p0) 3> view param rhbase

REPLICAT rhbase
TARGETDB LIBFILE libggjava.so SET property=dirprm/hbase.properties
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 10000
MAP oggsrc.*, TARGET oggsrc.*;


GGSCI (edvmr1p0) 4> add replicat rhbase, exttrail ./dirdat/hb
REPLICAT added.


GGSCI (edvmr1p0) 5> start rhbase

Sending START request to MANAGER ...
REPLICAT RHBASE starting


GGSCI (edvmr1p0) 6> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING                                           
REPLICAT    RUNNING    RHBASE      00:00:00      00:00:09    


GGSCI (edvmr1p0) 7>

So let's check if that will work. We will do two tests:

Test on Insert Operation
Test on Update Operation

Insert data on the source

[oracle@edvmr1p0 oggsrc]$ sqlplus oggsrc/oracle@orcl

SQL*Plus: Release 12.1.0.2.0 Production on Wed Nov 11 15:45:59 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Last Successful login time: Wed Nov 11 2020 15:24:15 +00:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> insert into product_prod select * from product where product_id < 1001;

1000 rows created.

SQL> commit;

Commit complete.

SQL>

Check on HBase:

Check on the HBase

[oracle@edvmr1p0 ~]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.3, rbd63744624a26dc3350137b564fe746df7a721a4, Mon Aug 29 15:13:42 PDT 2016

hbase(main):001:0> list
TABLE                                                                                                                        
OGGSRC:PRODUCT_PROD                     <- Our table                                                                                         
1 row(s) in 0.2120 seconds

=> ["OGGSRC:PRODUCT_PROD"]
hbase(main):002:0> 
base(main):002:0> scan 'OGGSRC:PRODUCT_PROD'
ROW COLUMN+CELL
1 column=cf:PRODUCT_CODE, timestamp=1486380784291,
value=RJM75LEM3UY
1 column=cf:PRODUCT_ID, timestamp=1486380784291,
value=1
1 column=cf:PRODUCT_NAME, timestamp=1486380784291,
value=eu, eleifend nec, malesuada
1 column=cf:SUPPLIER_ID, timestamp=1486380784291,
value=1
10 column=cf:PRODUCT_CODE, timestamp=1486380784291,
value=IEF94KVH9EU
10 column=cf:PRODUCT_ID, timestamp=1486380784291,
..............................................................

On the Source database, update the table in any way you want:

Update source table

SQL> update product_prod set product_name = 'OGG Big Data HBase' where product_id=999;

1 row updated.

SQL> commit;

Commit complete.

SQL>

On the HBase that should be visible:

Check HBase

hbase(main):010:0* get 'OGGSRC:PRODUCT_PROD','999'
COLUMN                           CELL                                                                                        
 cf:PRODUCT_CODE                 timestamp=1605109589905, value=MTA34WRW7FB                                                  
 cf:PRODUCT_ID                   timestamp=1605111265717, value=999                        <- Our ID (where clause)                                   
 cf:PRODUCT_NAME                 timestamp=1605111265717, value=OGG Big Data HBase         <- Our change (set clause)                       
 cf:SUPPLIER_ID                  timestamp=1605109589905, value=999                                            
4 row(s) in 0.0100 seconds

hbase(main):011:0>

Configure Destination

Start HBase

Configure HBase Handler

Test

Test on Insert Operation

Test on Update Operation