Overview

In this section we will replicate from Oracle → Golden Gate → HDFS

Configuration Destination

Create HDFS Properties File (TRG_GGHOME/dirprm/hdfs.properties)

Now bare in mind that since the following settings:

rollOnMetadataChange is set to TRUE

pkUpdateHandling is set to delete-insert

That means if the table structure changes (add/remove column), the HDFS will create new JSON. Furthermore, because of the 2nd parameter, when you update, that will do pretty much the same, as it will delete the record and insert new one. Furthermore, since the table is without PK, HDFS will be forced to track all the columns into a pseudo-PK column. We will modify that at the end of the section to see different results.

Create Replicat

Create Replication

Test

To test, we can connect to the Source database and issue insert:

Insert Data

Check the replication:

Check source and dest statistics

We can check in the HDFS as well:

Check in the HDFS

With the current configuration, the HDFS will create a new directory (under HDFS) for each changed table. For example if I update a new table, I will recieve two directories as follows:

Check HDFS Structure

Modify Configuration

Let's test new configuration, by setting:

gg.handler.hdfs.rollOnMetadataChange=false

gg.handler.hdfs.format.pkUpdateHandling=update

That will force the HDFS to NOT create new JSON on metadata change and on update. Once we do that change we have to restart the replicat:

Restart Replicat