Show pageOld revisionsBacklinksODT exportBack to top This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. =====Configure Destination===== Again, we have to configure: * Start HBase * Replicat config * HBase Handler Properties File So let's get going. ====Start HBase==== <Code:bash|Start HBase, HDFS and Yarn> [oracle@hostname ~]$ start-dfs.sh Starting namenodes on [localhost] localhost: starting namenode, logging to /opt/hadoop/logs/hadoop-oraclenamenode- edvmr1p0.out localhost: starting datanode, logging to /opt/hadoop/logs/hadoop-oracledatanode- edvmr1p0.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/logs/hadooporacle- secondarynamenode-edvmr1p0.out [oracle@hostname ~]$ start-yarn.sh starting yarn daemons starting resourcemanager, logging to /opt/hadoop/logs/yarn-oracleresourcemanager- edvmr1p0.out localhost: starting nodemanager, logging to /opt/hadoop/logs/yarn-oraclenodemanager- edvmr1p0.out [oracle@hostname ~]$ start-hbase.sh localhost: starting zookeeper, logging to /opt/hbase/logs/hbase-oraclezookeeper- edvmr1p0.out starting master, logging to /opt/hbase/logs/hbase-oracle-master-edvmr1p0.out starting regionserver, logging to /opt/hbase/logs/hbase-oracle-1-regionserveredvmr1p0. out [oracle@hostname ~]$ [oracle@hostname ~]$ $JAVA_HOME/bin/jps 21538 SecondaryNameNode 21346 DataNode 22148 NodeManager 22613 HMaster 22502 HQuorumPeer 21159 NameNode 23386 Jps 21851 ResourceManager 22747 HRegionServer [oracle@hostname ~]$ </Code> ====Configure HBase Handler==== <Code:bash |Configure HBase Handler Properties File (TRG_OGGHIME/dirprm/base.properties> gg.handlerlist=hbase gg.handler.hbase.type=hbase gg.handler.hbase.hBaseColumnFamilyName=cf gg.handler.hbase.keyValueDelimiter=CDATA[=] gg.handler.hbase.keyValuePairDelimiter=CDATA[,] gg.handler.hbase.encoding=UTF-8 gg.handler.hbase.pkUpdateHandling=delete-insert gg.handler.hbase.nullValueRepresentation=CDATA[NULL] gg.handler.hbase.authType=none gg.handler.hbase.includeTokens=false gg.handler.hbase.mode=tx goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec gg.classpath=/opt/hbase/lib/*:/opt/hbase/conf/: jvm.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar </Code> Now, we can configure and start the REPLICAT <Code:bash|Configure Replicat> [oracle@edvmr1p0 oggtrg]$ ggsci Oracle GoldenGate Command Interpreter Version 12.2.0.1.160823 OGGCORE_OGGADP.12.2.0.1.0_PLATFORMS_161019.1437 Linux, x64, 64bit (optimized), Generic on Oct 19 2016 16:01:40 Operating system character set identified as UTF-8. Copyright (C) 1995, 2016, Oracle and/or its affiliates. All rights reserved. GGSCI (edvmr1p0) 1> info all Program Status Group Lag at Chkpt Time Since Chkpt MANAGER RUNNING GGSCI (edvmr1p0) 2> view param mgr Port 7909 DynamicPortList 20100-20199 PurgeOldExtracts ./dirdat/pe*, UseCheckPoints, MinKeepHours 2 Autostart Replicat R* AUTORESTART Replicat *, WaitMinutes 1, Retries 3 GGSCI (edvmr1p0) 3> view param rhbase REPLICAT rhbase TARGETDB LIBFILE libggjava.so SET property=dirprm/hbase.properties REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 10000 MAP oggsrc.*, TARGET oggsrc.*; GGSCI (edvmr1p0) 4> add replicat rhbase, exttrail ./dirdat/hb REPLICAT added. GGSCI (edvmr1p0) 5> start rhbase Sending START request to MANAGER ... REPLICAT RHBASE starting GGSCI (edvmr1p0) 6> info all Program Status Group Lag at Chkpt Time Since Chkpt MANAGER RUNNING REPLICAT RUNNING RHBASE 00:00:00 00:00:09 GGSCI (edvmr1p0) 7> </Code> =====Test===== So let's check if that will work. We will do two tests: - Test on Insert Operation - Test on Update Operation ====Test on Insert Operation==== <Code:bash|Insert data on the source> [oracle@edvmr1p0 oggsrc]$ sqlplus oggsrc/oracle@orcl SQL*Plus: Release 12.1.0.2.0 Production on Wed Nov 11 15:45:59 2020 Copyright (c) 1982, 2014, Oracle. All rights reserved. Last Successful login time: Wed Nov 11 2020 15:24:15 +00:00 Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options SQL> insert into product_prod select * from product where product_id < 1001; 1000 rows created. SQL> commit; Commit complete. SQL> </Code> Check on HBase: <Code:bash|Check on the HBase> [oracle@edvmr1p0 ~]$ hbase shell SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 1.2.3, rbd63744624a26dc3350137b564fe746df7a721a4, Mon Aug 29 15:13:42 PDT 2016 hbase(main):001:0> list TABLE OGGSRC:PRODUCT_PROD <- Our table 1 row(s) in 0.2120 seconds => ["OGGSRC:PRODUCT_PROD"] hbase(main):002:0> base(main):002:0> scan 'OGGSRC:PRODUCT_PROD' ROW COLUMN+CELL 1 column=cf:PRODUCT_CODE, timestamp=1486380784291, value=RJM75LEM3UY 1 column=cf:PRODUCT_ID, timestamp=1486380784291, value=1 1 column=cf:PRODUCT_NAME, timestamp=1486380784291, value=eu, eleifend nec, malesuada 1 column=cf:SUPPLIER_ID, timestamp=1486380784291, value=1 10 column=cf:PRODUCT_CODE, timestamp=1486380784291, value=IEF94KVH9EU 10 column=cf:PRODUCT_ID, timestamp=1486380784291, .............................................................. </Code> ====Test on Update Operation==== On the Source database, update the table in any way you want: <Code:bash|Update source table> SQL> update product_prod set product_name = 'OGG Big Data HBase' where product_id=999; 1 row updated. SQL> commit; Commit complete. SQL> </Code> On the HBase that should be visible: <Code:bash|Check HBase> hbase(main):010:0* get 'OGGSRC:PRODUCT_PROD','999' COLUMN CELL cf:PRODUCT_CODE timestamp=1605109589905, value=MTA34WRW7FB cf:PRODUCT_ID timestamp=1605111265717, value=999 <- Our ID (where clause) cf:PRODUCT_NAME timestamp=1605111265717, value=OGG Big Data HBase <- Our change (set clause) cf:SUPPLIER_ID timestamp=1605109589905, value=999 4 row(s) in 0.0100 seconds hbase(main):011:0> </Code> oracle_goldengate_hadoop_hbase.txt Last modified: 2020/11/22 15:08by andonovj