How to Install Apache Kafka on Ubuntu 22.04 LTS

In this article We are going to perform How to Install Apache Kafka on Ubuntu 22.04 LTS and also covered creating systemd unit file for zookeeper and Kafka service, create a topic in Kafka, send messages in Kafka, Start consumer in Kafka.

What is Kafka ?

Kafka is an open source software which provides a framework for storing, reading and analysing streaming data.

Being open source means that it is essentially free to use and has a large network of users and developers who contribute towards updates, new features and offering support for new users.

Kafka is designed to be run in a “distributed” environment, which means that rather than sitting on one user’s computer, it runs across several (or many) servers, leveraging the additional processing power and storage capacity that this brings.

Why do you need Kafka it depends on your use case

Some use cases I can think of :

1. you can have a streaming app that captures logs of all the apps in a company and all can be redirected to common logging space.
2. Real-time transactions you can leverage kafka.
3. You can build set of micro services that perform complex business logic to communicate through Kafka .

You can easily integrate kafka to various big data processing frameworks like apache spark , Apache storm seamlessly.

You can know more about it here : Apache Kafka

Prerequisites

• Create Ubuntu Instance 22.04 LTS
• SSH Access with Sudo privileges
• Firewall Port 9092
• JDK 1.8 or higher version

Install OpenJDK on Ubuntu 22.04 LTS

To update system packages

sudo apt-get update

you can install OpenJDK 8 or OpenJDK 11

sudo apt install openjdk-8-jdk

or

sudo apt install openjdk-11-jdk

To check the java version, Here I have install installed Oracle Java 11 on my system.

java -version

Output:

Step #1:Install Apache Kafka on Ubuntu 22.04 LTS

To download the Kafka binary from offical website. Please use this Kafka official download page and to prompts to download page and you can download Kafka using wget

sudo wget https://downloads.apache.org/kafka/3.5.0/kafka_2.12-3.5.0.tgz

Now to un-tar or Unzip the archive file and move to another location:

sudo tar xzf kafka_2.12-3.5.0.tgz

sudo mv kafka_2.12-3.5.0 /opt/kafka

Step #2:Creating Zookeeper and Kafka Systemd Unit Files in Ubuntu 22.04 LTS

Create the systemd unit file for zookeeper service

sudo nano  /etc/systemd/system/zookeeper.service

paste the below lines

/etc/systemd/system/zookeeper.service
[Unit]
Description=Apache Zookeeper service
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Reload the daemon to take effect

sudo systemctl daemon-reload

Create the systemd unit file for kafka service

sudo nano /etc/systemd/system/kafka.service

paste the below lines

[Unit]
Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

Note: Modify the value of JAVA_HOME value, If the path of your Java installation is different path like If your java version is 1.8 the use Environment=”JAVA_HOME=/opt/jdk/jdk1.8.0_251″ this path.

Reload the daemon to take effect

sudo systemctl daemon-reload

Step #3:To Start ZooKeeper and Kafka Service and Check its Status

Lets start zookeeper service first

sudo systemctl start zookeeper

Check the status of  zookeeper service if it started

sudo systemctl status zookeeper

Output:

Start the kafka service

sudo systemctl start kafka

Check the status of  kafka service if it started

sudo systemctl status kafka

Output:

To Start Kafka And ZooKeeper Server in the background(without creating systemd unit file)

We have shell script to run the kafka and zookeeper server in backend:

Create a file named kafkastart.sh and copy the below script:

#!/bin/bash
sudo nohup /opt/kafka/bin/zookeeper-server-start.sh -daemon /opt/kafka/config/zookeeper.properties > /dev/null 2>&1 &
sleep 5
sudo nohup /opt/kafka/bin/kafka-server-start.sh -daemon /opt/kafka/config/server.properties > /dev/null 2>&1 &

After give the executable permissions to the file:

sudo chmod +x kafkastart.sh

Successfully We have covered how to install apache kafka on ubuntu 22.04 LTS.

Step #4:Creating Topic in Kafka

Now we will create a topic named as “DevOps” with a single replicaton-factor and partition:

cd /opt/kafka

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic DevopsHint

Output:

Explanation of command

• –create :- It is used for create a new topic
• –replication-factor :- It is used for how many copies of data will be created.
• –partitions :- It is used for set the partitions options as the number of brokers you want your data to be split between.
• –topic :- It is used for name of the topic

To check the list of topics created.

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Output:

Step #5:To send some messages using Kafka

To send some messages for created Topic.

sudo bin/kafka-console-producer.sh —broker-list localhost:9092 —topic DevopsHint

its prompt for messages to type:

> hello world!
> How are you?
> This is DevopsHint
> Bye…

Step #6:To Start consumer in Kafka

Using below command we can see the list of messages:

sudo bin/kafka-console-consumer.sh ---bootstrap-server localhost:9092 ---topic DevopsHint ---from-beginning

Output

Step #7:To Connect Kafka from remote machine

To connect, create Topic and send messages from remote server. Please follow below steps.

Go to the below path:

cd /opt/kafka/config

Now look for server.properties and make some configuration changes:

sudo vi server.properties

In this properties file uncomment as mentioned  below:

listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://<HOST IP>:9092

Step #8:To Delete any Topics in Kafka

If you want to delete any created  topic use below command:

sudo bin/kafka-topics.sh --delete localhost:9092 --topic <anytopic>

We have covered How to Install Apache Kafka on Ubuntu 22.04 LTS.

Conclusion:

In this article, We have covered ,How to Install Apache Kafka on Ubuntu 22.04 LTS and also covered creating systemd unit file for zookeeper and kafka service, start zookeeper, start kafka, create a topic, To send some messages, To start consumer.

Related Articles:

How to Install NGINX on Ubuntu 22.04 LTS