oracle_goldengate_apache_kafka

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
oracle_goldengate_apache_kafka [2020/11/12 12:36] – created andonovjoracle_goldengate_apache_kafka [2020/11/12 12:51] (current) – andonovj
Line 2: Line 2:
 Before we do anything, we have to start Kafka first, so let's do it: Before we do anything, we have to start Kafka first, so let's do it:
  
 +====Start Kafka====
 <Code:bash|Start kafka> <Code:bash|Start kafka>
 [oracle@edvmr1p0 ~]$ kafka-server-start.sh $KAFKA_HOME/config/server.properties [oracle@edvmr1p0 ~]$ kafka-server-start.sh $KAFKA_HOME/config/server.properties
Line 29: Line 30:
 </Code> </Code>
  
 +====Configure Kafka Properties File====
 +<Code:bash|Configure Kafka Properties File (e.g. TRG_OGGHOME/dirprm/kafka.properties)>
 +[oracle@edvmr1p0 dirprm]$ cat ~/labs/practice05/replicat/prac_5_1/kafka.properties 
  
 +gg.handlerlist=kafka
 +gg.handler.kafka.type=kafka
 +gg.handler.kafka.KafkaProducerConfigFile=kafka_producer.properties
 +gg.handler.kafka.TopicName=oggbgtopic
 +gg.handler.kafka.format=json
 +gg.handler.kafka.BlockingSend=false
 +gg.handler.kafka.includeTokens=false
 +gg.handler.kafka.mode =tx
 +
 +goldengate.userexit.timestamp=utc
 +goldengate.userexit.writers=javawriter
 +javawriter.stats.display=TRUE
 +javawriter.stats.full=TRUE
 +
 +gg.log=log4j
 +gg.log.level=INFO
 +
 +gg.report.time=30sec
 +gg.classpath=dirprm/:/opt/kafka/libs/*
 +jvm.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar 
 +
 +
 +</Code>
 +
 +====Configure Kafka Producer Properties file====
 +<Code:bash|Kafka Producer Properties file (e.g. TRG_OGGHOME/dirprm/kafka_producer.properties)>
 +bootstrap.servers=localhost:9092
 +acks=1
 +compression.type=gzip
 +reconnect.backoff.ms=1000
 +
 +value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
 +key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
 +batch.size=102400
 +linger.ms=10000
 +
 +max.request.size = 5024000
 +send.buffer.bytes = 5024000
 +</Code>
 +
 +
 +====Configure Golden Gate(Replicat)====
 +Again we have to configure the Golden Gate Replicat:
 +
 +<Code:bash|Configure Golden Gate Replicat>
 +[oracle@edvmr1p0 dirprm]$ trg
 +[oracle@edvmr1p0 oggtrg]$ ggsci
 +
 +Oracle GoldenGate Command Interpreter
 +Version 12.2.0.1.160823 OGGCORE_OGGADP.12.2.0.1.0_PLATFORMS_161019.1437
 +Linux, x64, 64bit (optimized), Generic on Oct 19 2016 16:01:40
 +Operating system character set identified as UTF-8.
 +
 +Copyright (C) 1995, 2016, Oracle and/or its affiliates. All rights reserved.
 +
 +
 +
 +GGSCI (edvmr1p0) 1> info all
 +
 +Program     Status      Group       Lag at Chkpt  Time Since Chkpt
 +
 +MANAGER     RUNNING                                           
 +
 +
 +GGSCI (edvmr1p0) 2> edit param rkafka
 +
 +
 +
 +GGSCI (edvmr1p0) 3> view param rkafka
 +
 +REPLICAT rkafka
 +TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.properties
 +REPORTCOUNT EVERY 1 MINUTES, RATE
 +GROUPTRANSOPS 10000
 +MAP OGGSRC.*, TARGET OGGTRG.*;
 +
 +
 +GGSCI (edvmr1p0) 4> add replicat rkafka, exttrail ./dirdat/kf
 +REPLICAT added.
 +
 +
 +GGSCI (edvmr1p0) 5> start rkafka
 +
 +Sending START request to MANAGER ...
 +REPLICAT RKAFKA starting
 +
 +
 +GGSCI (edvmr1p0) 6> info all
 +
 +Program     Status      Group       Lag at Chkpt  Time Since Chkpt
 +
 +MANAGER     RUNNING                                           
 +REPLICAT    STARTING    RKAFKA      00:00:00      00:00:06    
 +
 +
 +GGSCI (edvmr1p0) 7> info all
 +
 +Program     Status      Group       Lag at Chkpt  Time Since Chkpt
 +
 +MANAGER     RUNNING                                           
 +REPLICAT    RUNNING     RKAFKA      00:00:00      00:00:08    
 +
 +
 +GGSCI (edvmr1p0) 8> 
 +
 +</Code>
 +
 +=====Test=====
 +To test the replication, we will insert some rows and see how they are being replicated.
 +
 +<Code:bash|Insert data on Oracle Source DB>
 +[oracle@edvmr1p0 oggsrc]$ sqlplus oggsrc/oracle@orcl
 +
 +SQL*Plus: Release 12.1.0.2.0 Production on Thu Nov 12 12:46:13 2020
 +
 +Copyright (c) 1982, 2014, Oracle.  All rights reserved.
 +
 +Last Successful login time: Thu Nov 12 2020 12:26:00 +00:00
 +
 +Connected to:
 +Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
 +With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
 +
 +SQL> insert into customer_prod select * from customer where customer_id < 101;
 +
 +100 rows created.
 +
 +SQL> commit;
 +
 +Commit complete.
 +
 +SQL> 
 +</Code>
 +
 +Check if the data is replicated between: GG Extract -> GG Replicat
 +
 +<Code:bash|Check GG Replication>
 +--Source:
 +GGSCI (edvmr1p0) 4> send priex,stats
 +
 +Sending STATS request to EXTRACT PRIEX ...
 +
 +Start of Statistics at 2020-11-12 12:46:37.
 +
 +DDL replication statistics (for all trails):
 +
 +*** Total statistics since extract started     ***
 + Operations                           6.00
 +
 +Output to ./dirdat/in:
 +
 +Extracting from OGGSRC.CUSTOMER_PROD to OGGSRC.CUSTOMER_PROD:
 +
 +*** Total statistics since 2020-11-12 12:46:30 ***
 + Total inserts                             100.00
 + Total updates                               0.00
 + Total deletes                               0.00
 + Total discards                             0.00
 + Total operations                         100.00
 +
 +*** Daily statistics since 2020-11-12 12:46:30 ***
 + Total inserts                             100.00
 + Total updates                               0.00
 + Total deletes                               0.00
 + Total discards                             0.00
 + Total operations                         100.00
 +
 +*** Hourly statistics since 2020-11-12 12:46:30 ***
 + Total inserts                             100.00
 + Total updates                               0.00
 + Total deletes                               0.00
 + Total discards                             0.00
 + Total operations                         100.00
 +
 +*** Latest statistics since 2020-11-12 12:46:30 ***
 + Total inserts                             100.00
 + Total updates                               0.00
 + Total deletes                               0.00
 + Total discards                             0.00
 + Total operations                         100.00
 +
 +End of Statistics.
 +
 +
 +GGSCI (edvmr1p0) 5> 
 +
 +--Target
 +GGSCI (edvmr1p0) 8> send rkafka,stats
 +
 +Sending STATS request to REPLICAT RKAFKA ...
 +
 +Start of Statistics at 2020-11-12 12:46:47.
 +
 +Replicating from OGGSRC.CUSTOMER_PROD to OGGTRG.CUSTOMER_PROD:
 +
 +*** Total statistics since 2020-11-12 12:46:34 ***
 + Total inserts                             100.00
 + Total updates                               0.00
 + Total deletes                               0.00
 + Total discards                             0.00
 + Total operations                         100.00
 +
 +*** Daily statistics since 2020-11-12 12:46:34 ***
 + Total inserts                             100.00
 + Total updates                               0.00
 + Total deletes                               0.00
 + Total discards                             0.00
 + Total operations                         100.00
 +
 +*** Hourly statistics since 2020-11-12 12:46:34 ***
 + Total inserts                             100.00
 + Total updates                               0.00
 + Total deletes                               0.00
 + Total discards                             0.00
 + Total operations                         100.00
 +
 +*** Latest statistics since 2020-11-12 12:46:34 ***
 + Total inserts                             100.00
 + Total updates                               0.00
 + Total deletes                               0.00
 + Total discards                             0.00
 + Total operations                         100.00
 +
 +End of Statistics.
 +
 +
 +GGSCI (edvmr1p0) 9> 
 +</Code>
 +
 +Finally, we can consume the data on Kafka:
 +
 +<Code:bash|Consume the data from Kafka>
 +[oracle@edvmr1p0 ~]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic oggbgtopic --from-beginning --zookeeper localhost:2181
 +{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.075000","pos":"00000000000000002372","after":{"CUSTOMER_ID":1,"FIRST_NAME":"Jaden","LAST_NAME":"Poole","EMAIL_ADDRESS":"[email protected]","SSN":"16060407-7404","ADDRESS":"3640 Ac St.","CITY":"San Juan de Dios","ZIP_CODE":"31014","CUSTOMER_TYPE":1}}{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.076000","pos":"00000000000000002658","after":{"CUSTOMER_ID":2,"FIRST_NAME":"Jack","LAST_NAME":"Cox","EMAIL_ADDRESS":"[email protected]","SSN":"16040918-3654","ADDRESS":"Ap #305-8153 Libero Ave","CITY":"Ca��as","ZIP_CODE":"40813","CUSTOMER_TYPE":4}}{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.077000","pos":"00000000000000002912","after":{"CUSTOMER_ID":3,"FIRST_NAME":"Anastasia","LAST_NAME":"Stewart","EMAIL_ADDRESS":"[email protected]","SSN":"16030603-2822","ADDRESS":"282-2584 Consectetuer Street","CITY":"San Juan (San Juan de Tib��s)","ZIP_CODE":"50409","CUSTOMER_TYPE":7}}{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.077001","pos":"00000000000000003198","after":{"CUSTOMER_ID":4,"FIRST_NAME":"Rowan","LAST_NAME":"Fleming","EMAIL_ADDRESS":"[email protected]","SSN":"16700223-3166","ADDRESS":"7594 Adipiscing. Ave","CITY":"San Vicente","ZIP_CODE":"40102","CUSTOMER_TYPE":2}}{"table":"OGGTRG.CUSTOMER_PROD","op_type":"I","op_ts":"2020-11-12 12:46:28.000148","current_ts":"2020-11-12T12:46:34.077002","pos":"00000000000000003448","after":{"CUSTOMER_ID":5,"FIRST_NAME":"Adena","LAST_NAME":"Williamson","EMAIL_ADDRESS":"[email protected]","SSN":
 +</Code>
 +
 +So, we have configured replication:
 +Oracle RDBMS -> Golden Gate (Extract) -> Golden Gate (Replicat) -> Apache Kafka
  
 =====Appendix===== =====Appendix=====
  • oracle_goldengate_apache_kafka.1605184617.txt.gz
  • Last modified: 2020/11/12 12:36
  • by andonovj