Configure Destination

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows.

We have to configure:

Flume.properties file

Flume RPC client properties file

Configure Flume Properties

In the target OGG for big data, create the following files

Create Flume.properties file (TRG_OGGHOME/dirprm/flume.properties)

Configure Flume RPC Client

To configure the client, we have to create custom-flume-rpc.properties in the same file:

Create customer RPC File (e.g. TRG_OGGHOME/dirprm/custom-flume-rpc.properties)

Start Flume

Start Flume

Configure the GG Replicat

Again, we have to configure the GG with the replicat:

Configure GoldenGate Replicat

Test

To test replication, we will simply insert some records and see if we can see them replicated in the log

Insert rows

Then, we can check the stats from the GoldenGate

Check GoldenGate Stats

Now, we can also check on the flume:

Check Apache Flume log

We can of course check the files on the HDFS as well:

Check HDFS

Niiice, so we have replicated 20 rows between: Oracle RDBMS → Golden Gate (Extract) → Golden Gate (replicat) → Apache Flume

Appendix

Flume config (/opt/flume/flume.conf)

Log4J properties file (/opt/flume/conf/log4j.properties)