Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

In a nutshell, you say data to it and you can do something with that data. Here we will process data from Oracle through Kafka to big data.

So for example, you can have: Oracle → Kafka → Hadoop/Hbase

To start Kafka, we can use the following script:

[oracle@edvmr1p0 ~]$ kafka-server-start.sh
USAGE: /opt/kafka/bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*
[oracle@edvmr1p0 ~]$ kafka-server-start.sh $KAFKA_HOME/config/server.properties
[2020-11-11 09:33:54,319] INFO KafkaConfig values: 
	advertised.host.name = null
	metric.reporters = []
	quota.producer.default = 9223372036854775807
	offsets.topic.num.partitions = 50
	log.flush.interval.messages = 9223372036854775807
	auto.create.topics.enable = true
	controller.socket.timeout.ms = 30000
	log.flush.interval.ms = null
	principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
	replica.socket.receive.buffer.bytes = 65536
	min.insync.replicas = 1
.........

[oracle@edvmr1p0 config]$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 -topic ogg12cBigData
Created topic "ogg12cBigData".

[oracle@edvmr1p0 config]$ kafka-console-producer.sh --broker-list localhost:9092 --topic ogg12cBigData
Testing Kafka in the context of OGG 12c for Big Data                  <- First Message

Second message sent to the Kafka cluster                              <- Second Message