Overview

In this section we will replicate from Oracle → Golden Gate → HDFS

Configuration Destination

Create HDFS Properties File (TRG_GGHOME/dirprm/hdfs.properties)

Now bare in mind that since the following settings:

rollOnMetadataChange is set to TRUE

pkUpdateHandling is set to delete-insert

That means if the table structure changes (add/remove column), the HDFS will create new JSON. Furthermore, because of the 2nd parameter, when you update, that will do pretty much the same, as it will delete the record and insert new one. Furthermore, since the table is without PK, HDFS will be forced to track all the columns into a pseudo-PK column. We will modify that at the end of the section to see different results.

Create Replicat

Create Replication

Test

To test, we can connect to the Source database and issue insert:

Insert Data

Check the replication:

Check source and dest statistics

We can check in the HDFS as well:

Check in the HDFS

With the current configuration, the HDFS will create a new directory (under HDFS) for each changed table. For example if I update a new table, I will recieve two directories as follows:

Check HDFS Structure

Modify Configuration

Let's test new configuration, by setting:

gg.handler.hdfs.rollOnMetadataChange=false

gg.handler.hdfs.format.pkUpdateHandling=update

In $TRG_GGHOME/dirprm/hdfs.properties

That will force the HDFS to NOT create new JSON on metadata change and on update. Once we do that change we have to restart the replicat:

Restart Replicat

Now, if we add new column or modify the data, we will not rollover new JSON file until we don't reach the 1G file size defined in:

gg.handler.hdfs.maxFileSize=1g

These settings should cause the creation of a new JSON File:

Create new Column

Check HDFS files

After this point, whatever we do to the table, as long as we don't exceed the 1G of size, it should grow, let's test it. We can examine the additional of a new column with a simple cat of the JSON

Check the JSON

So now, if I add a new column (yet again) it SHOULDN'T create new JSON. When we are at it, let's add a PK for that table as well.

Create PK & new column (yet again)

Now, we can check again for the JSONs:

Check if HDFS, created new JSON

No new JSON, let's examine again the last one:

Examine HDFS

Do you notice that in the last record, we have way less values than the rest, that is becasue I create a PK and HDFS wasn't forced to include all columns (in pseudo-PK column) but used the PK (Product_id) :)