Kafka - Single broker setup

Apache Kafka

Apache Kafka
  -> Distributed Streaming platform
  -> Publish and subscribe to streams of records, similar to a message queue or enterprise message system
  -> Store streams of records in a fault-tolerant durable way
      - Keeping multiple copies of data - Replication
  -> Replication ( Leader - follower model)
      - One broker is selected as a Leader
      - Producer will connect to Leader and send data
      - Consumer request Leader for required data
      - Depends on Replication factor, Followers will copy data from Leader
  -> Process stream of records as they occur

Key Components in Kafka
  -> Producer : An application that sends a data/message
  -> Message  : Small piece of data( Array of bites)
  -> Consumer : An application that receive data/message
  -> Brokers  : Kafka server
  -> Cluster  : Group of instances and each will act as one broker
  -> Topic    : Unique name to a data stream
  -> Partitions : Distributing topic data into multiple partitions
  -> Offset   : Sequence number given to a message
  -> Consumer Group : Group of Consumers acting as single

Apache Kafka setup

1. Download tar file from apache.org
   curl -k https://www-us.apache.org/dist/kafka/2.3.1/kafka-2.3.1-src.tgz > kafka-2.3.1-src.tgz
   tar -xzf kafka-2.3.1-src.tgz

2. Update server.properties files

   Broker.id=1
   listeners = PLAINTEXT://:9092
   log.dirs=/home/bigdata/opt/kafka/kafka-2.3.1/logs

3. Zookeeper setup
   - Coordination service for a distributed system
 
   wget https://archive.apache.org/dist/zookeeper/zookeeper-3.5.6/apache-zookeeper-3.5.6-bin.tar.gz
   tar -xvf apache-zookeeper-3.5.6-bin.tar.gz

4. Update Zoo.cgf file

   tickTime=200
   dataDir=/home/bigdata/opt/zookeeper/apache-zookeeper-3.5.6/data
   clientPort=2181
   maxClientCnxns=60
   initLimit=10
   syscLimit=5

5. Start Zookeeper services
   
   bin/Zookeeper-server-start.sh config/zookeeper.properties

6. Starting Kafka Broker server
   bin/kafka-server-start.sh config/server.properties

7. Creating a Topic 
   bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partition 1 --topic Demo

8. Starting Producer console to publish messages into Topic
   bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Demo

9. Starting Consumer console to consume messages from Topic
   bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic Demo --from-beginning


Producer - Callback and acknowledge methods

1. Send and Forget
   Send a message to broker and wont check whether the message is delivered successfully or not

2. Synchronous send
   Send a message and wait for response
     Success : we get record metadata object
     Fail : Exception

3. Asynchronous send
   Send a message and provide callback function to receive acknowledge. We can record message details for later analysis.