Skip to main content
Top
Review meshIQ Products (G2.com)

Setting up Kafka without ZooKeeper: Kafka Raft (KRaft) (Three-node cluster)

Kafka Raft (KRaft) is designed to allow you to run Kafka without a dependency on ZooKeeper.

To set up Kafka without ZooKeeper:

  1. Download and install the binary Scala 2.13 - kafka_2.13-3.x.x.tgz (asc, sha512).
  2. Untar it to the Kafka location and modify the soft link:
cd /opt/nastel/kafka
tar -zxvf kafka_2.13-3.x.x.tgz
unlink current

To link the new Kafka package:

ln -s kafka_2.13-3.x.x/ current
  1. Enter cd current/config/kraft.
  2. Add the following lines in the server.properties file of each Kafka broker:
For broker 1:
node.id=0
process.roles=broker, controller
inter.broker.listener.name=PLAINTEXT 
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://: 9092, CONTROLLER://:19092
log.dirs=/tmp/serverl/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT: PLAINTEXT, SSL: SSL, SASL PLAINTEXT: SASL_ PLAINTEX T, SASL_SSL: SASL_SSL
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094

For broker 2:
node.id=1
process.roles=broker, controller
inter.broker.listener.name=PLAINTEXT 
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://: 9092, CONTROLLER://:19093
log.dirs=/tmp/serverl/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT: PLAINTEXT, SSL: SSL, SASL PLAINTEXT: SASL_ PLAINTEX T, SASL_SSL: SASL_SSL
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094

For broker 3:
node.id=2
process.roles=broker, controller
inter.broker.listener.name=PLAINTEXT 
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://: 9092, CONTROLLER://:19094
log.dirs=/tmp/serverl/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT: PLAINTEXT, SSL: SSL, SASL PLAINTEXT: SASL_ PLAINTEX T, SASL_SSL: SASL_SSL
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094
  1. Generate a uuid by using the following command: 
    ./bin/kafka-storage.sh random-uuid

The uuid should be the same for all the clusters (in meta.properties file in Kafka-combine-logs).

An example of the resultant uuid is provided below. 

jwuHZEcQSO2dA3m2tZhjIQ

The basic command to format locations based on each server's property file is as follows: 
./bin/kafka-storage.sh format -t jwuHZEcQSO2dA3m2tZhjIQ -c ../config/kraft/server.properties
In the command above, -t is used to format storage with the generated UUID when running Kafka in KRaft mode; -c indicates the configuration file to format the storage.

Start Kafka

When Kafka uses KRaft, Kafka must start using a specific location. To start Kafka, be sure to use the modified startup script below. 

Start Kafka by running the following:

./bin/kafka-server-start.sh -daemon ../config/kraft/server.properties

Create topics

To create Kafka topics, Kafka must be running. See the Start Kafka section above.

Create Kafka topics using the jKool Kafka topics creation script under AutoPilotM6. The location of the script is:

/opt/nastel/AutoPilotM6/jkool/scripts

Execute the following script to create topics:

create-kafka-topics.sh