Differences

This shows you the differences between two versions of the page.

--- oracle_goldengate_apache_kafka [2020/11/12 12:42] – [Configure Kafka Properties File] andonovj
+++ oracle_goldengate_apache_kafka [2020/11/12 12:51] (current) – andonovj
@@ Line 75: / Line 75: @@
+====Configure Golden Gate(Replicat)====
+Again we have to configure the Golden Gate Replicat:
+<Code:bash|Configure Golden Gate Replicat>
+[oracle@edvmr1p0 dirprm]$ trg
+[oracle@edvmr1p0 oggtrg]$ ggsci
+Oracle GoldenGate Command Interpreter
+Version 12.2.0.1.160823 OGGCORE_OGGADP.12.2.0.1.0_PLATFORMS_161019.1437
+Linux, x64, 64bit (optimized), Generic on Oct 19 2016 16:01:40
+Operating system character set identified as UTF-8.
+Copyright (C) 1995, 2016, Oracle and/or its affiliates. All rights reserved.
+GGSCI (edvmr1p0) 1> info all
+Program     Status      Group       Lag at Chkpt  Time Since Chkpt
+MANAGER     RUNNING
+GGSCI (edvmr1p0) 2> edit param rkafka
+GGSCI (edvmr1p0) 3> view param rkafka
+REPLICAT rkafka
+TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.properties
+REPORTCOUNT EVERY 1 MINUTES, RATE
+GROUPTRANSOPS 10000
+MAP OGGSRC.*, TARGET OGGTRG.*;
+GGSCI (edvmr1p0) 4> add replicat rkafka, exttrail ./dirdat/kf
+REPLICAT added.
+GGSCI (edvmr1p0) 5> start rkafka
+Sending START request to MANAGER ...
+REPLICAT RKAFKA starting
+GGSCI (edvmr1p0) 6> info all
+Program     Status      Group       Lag at Chkpt  Time Since Chkpt
+MANAGER     RUNNING
+REPLICAT    STARTING    RKAFKA      00:00:00      00:00:06
+GGSCI (edvmr1p0) 7> info all
+Program     Status      Group       Lag at Chkpt  Time Since Chkpt
+MANAGER     RUNNING
+REPLICAT    RUNNING     RKAFKA      00:00:00      00:00:08
+GGSCI (edvmr1p0) 8>
+</Code>
+=====Test=====
+To test the replication, we will insert some rows and see how they are being replicated.
+<Code:bash|Insert data on Oracle Source DB>
+[oracle@edvmr1p0 oggsrc]$ sqlplus oggsrc/oracle@orcl
+SQL*Plus: Release 12.1.0.2.0 Production on Thu Nov 12 12:46:13 2020
+Copyright (c) 1982, 2014, Oracle.  All rights reserved.
+Last Successful login time: Thu Nov 12 2020 12:26:00 +00:00
+Connected to:
+Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
+With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
+SQL> insert into customer_prod select * from customer where customer_id < 101;
+rows created.
+SQL> commit;
+Commit complete.
+SQL>
+</Code>
+Check if the data is replicated between: GG Extract -> GG Replicat
+<Code:bash|Check GG Replication>
+--Source:
+GGSCI (edvmr1p0) 4> send priex,stats
+Sending STATS request to EXTRACT PRIEX ...
+Start of Statistics at 2020-11-12 12:46:37.
+DDL replication statistics (for all trails):
+*** Total statistics since extract started     ***
+	Operations                		           6.00
+Output to ./dirdat/in:
+Extracting from OGGSRC.CUSTOMER_PROD to OGGSRC.CUSTOMER_PROD:
+*** Total statistics since 2020-11-12 12:46:30 ***
+	Total inserts                   	         100.00
+	Total updates                   	           0.00
+	Total deletes                   	           0.00
+	Total discards                  	           0.00
+	Total operations                	         100.00
+*** Daily statistics since 2020-11-12 12:46:30 ***
+	Total inserts                   	         100.00
+	Total updates                   	           0.00
+	Total deletes                   	           0.00
+	Total discards                  	           0.00
+	Total operations                	         100.00
+*** Hourly statistics since 2020-11-12 12:46:30 ***
+	Total inserts                   	         100.00
+	Total updates                   	           0.00
+	Total deletes                   	           0.00
+	Total discards                  	           0.00
+	Total operations                	         100.00
+*** Latest statistics since 2020-11-12 12:46:30 ***
+	Total inserts                   	         100.00
+	Total updates                   	           0.00
+	Total deletes                   	           0.00
+	Total discards                  	           0.00
+	Total operations                	         100.00
+End of Statistics.
+GGSCI (edvmr1p0) 5>
+--Target
+GGSCI (edvmr1p0) 8> send rkafka,stats
+Sending STATS request to REPLICAT RKAFKA ...
+Start of Statistics at 2020-11-12 12:46:47.
+Replicating from OGGSRC.CUSTOMER_PROD to OGGTRG.CUSTOMER_PROD:
+*** Total statistics since 2020-11-12 12:46:34 ***
+	Total inserts                   	         100.00
+	Total updates                   	           0.00
+	Total deletes                   	           0.00
+	Total discards                  	           0.00
+	Total operations                	         100.00
+*** Daily statistics since 2020-11-12 12:46:34 ***
+	Total inserts                   	         100.00
+	Total updates                   	           0.00
+	Total deletes                   	           0.00
+	Total discards                  	           0.00
+	Total operations                	         100.00
+*** Hourly statistics since 2020-11-12 12:46:34 ***
+	Total inserts                   	         100.00
+	Total updates                   	           0.00
+	Total deletes                   	           0.00
+	Total discards                  	           0.00
+	Total operations                	         100.00
+*** Latest statistics since 2020-11-12 12:46:34 ***
+	Total inserts                   	         100.00
+	Total updates                   	           0.00
+	Total deletes                   	           0.00
+	Total discards                  	           0.00
+	Total operations                	         100.00
+End of Statistics.
+GGSCI (edvmr1p0) 9>
+</Code>
+Finally, we can consume the data on Kafka:
+<Code:bash|Consume the data from Kafka>
+[oracle@edvmr1p0 ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic oggbgtopic --from-beginning --zookeeper localhost:2181
+{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.075000","pos":"00000000000000002372","after":{"CUSTOMER_ID":1,"FIRST_NAME":"Jaden","LAST_NAME":"Poole","EMAIL_ADDRESS":"[email protected]","SSN":"16060407-7404","ADDRESS":"3640 Ac St.","CITY":"San Juan de Dios","ZIP_CODE":"31014","CUSTOMER_TYPE":1}}{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.076000","pos":"00000000000000002658","after":{"CUSTOMER_ID":2,"FIRST_NAME":"Jack","LAST_NAME":"Cox","EMAIL_ADDRESS":"[email protected]","SSN":"16040918-3654","ADDRESS":"Ap #305-8153 Libero Ave","CITY":"Ca��as","ZIP_CODE":"40813","CUSTOMER_TYPE":4}}{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.077000","pos":"00000000000000002912","after":{"CUSTOMER_ID":3,"FIRST_NAME":"Anastasia","LAST_NAME":"Stewart","EMAIL_ADDRESS":"[email protected]","SSN":"16030603-2822","ADDRESS":"282-2584 Consectetuer Street","CITY":"San Juan (San Juan de Tib��s)","ZIP_CODE":"50409","CUSTOMER_TYPE":7}}{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.077001","pos":"00000000000000003198","after":{"CUSTOMER_ID":4,"FIRST_NAME":"Rowan","LAST_NAME":"Fleming","EMAIL_ADDRESS":"[email protected]","SSN":"16700223-3166","ADDRESS":"7594 Adipiscing. Ave","CITY":"San Vicente","ZIP_CODE":"40102","CUSTOMER_TYPE":2}}{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.077002","pos":"00000000000000003448","after":{"CUSTOMER_ID":5,"FIRST_NAME":"Adena","LAST_NAME":"Williamson","EMAIL_ADDRESS":"[email protected]","SSN":
+</Code>
+So, we have configured replication:
+Oracle RDBMS -> Golden Gate (Extract) -> Golden Gate (Replicat) -> Apache Kafka
 =====Appendix=====