Deploy Standalone Kafka 4

This page explains how to deploy a standalone Kafka 4 cluster in KRaft mode: get the Kafka binaries, configure the controller and broker, initialize storage, start the cluster, and create, write, and read events from a topic.

Get Kafka

Download the standalone binaries of Kafka 4.3.0 and extract them:

$ tar -xzf kafka_2.13-4.3.0.3.3.6.5-1009.tgz
$ cd kafka_2.13-4.3.0.3.3.6.5-1009

Start the Kafka environment

Kafka4 requires Java to run. Ensure that Java 17 is installed by running the following command:

$ java -version

Set JAVA_HOME to point to Java 17. Kafka 4.3.0 only runs in KRaft mode. To get started, follow the steps below.

Kafka with KRaft

KRaft mode is Kafka's newer architecture that eliminates the dependency on ZooKeeper. In KRaft mode, Kafka manages metadata natively using its Raft-based consensus protocol.

Step 1: Configure the Kafka controllers

Controllers in KRaft mode are responsible for managing metadata across the cluster.

Edit the config/kraft-controller.properties file to define your controller configurations:

# Example configuration for a controller (config/controller.properties):
process.roles=controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://localhost:9093
log.dirs=/tmp/kraft-controller-logs

Step 2: Configure the Kafka broker

Once the controllers are set up, configure the brokers, which handle message storage and processing.

Edit the config/broker.properties file to define your broker configurations:

# Example configuration for a broker (config/broker.properties):
process.roles=broker
node.id=2
listeners=PLAINTEXT://localhost:9092
controller.quorum.voters=1@localhost:9093
log.dirs=/tmp/kraft-broker-logs

Step 3: Initialize the Kafka storage

In KRaft mode, Kafka uses a unique cluster.id to manage metadata. Before you start any services, generate and initialize the storage directories for the Kafka controllers and brokers:

# Generate a cluster UUID on any one of the nodes:
$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
 
# Format log directories for the controller
$ bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/controller.properties
 
# Format log directories for the broker
$ bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/broker.properties

Step 4: Start the Kafka controllers

Once the configuration is complete, start the Kafka controller services. In KRaft mode, start the controllers first, because they manage the metadata for the brokers.

$ bin/kafka-server-start.sh config/controller.properties

Step 5: Start the broker

Open another terminal session and run the broker service:

$ bin/kafka-server-start.sh config/broker.properties

Create a topic to store your events

A topic can be thought of as a folder in a file system, and individual events as files stored within that folder.

Before you can produce and consume events, create a topic. Open a new terminal and run the following command:

$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

Kafka's command-line tools offer several options for various operations. Run the kafka-topics.sh command without arguments to get a list of these options. For example, to get details about the newly created topic, such as its partition count, run:

$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092

Topic: quickstart-events       TopicId: NPmZHyhbR9y00wMglMH2sg
PartitionCount: 1       ReplicationFactor: 1    Configs:
Topic: quickstart-events Partition: 0    Leader: 0    Replicas: 0    Isr: 0

Write events into the topic

A Kafka client interacts with the Kafka brokers over the network to write or read events. When the brokers receive events, they store them in a durable, fault-tolerant way, ensuring the data remains available for as long as necessary — even indefinitely, if configured.

To write events to your Kafka topic, use the Kafka console producer. Each line of input you provide is written as a separate event to the specified topic.

Run the following command to start the console producer client and write events:

$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092

Start typing messages, for example:

>This is my first event
>This is my second event

Each line is treated as an individual event and sent to the quickstart-events topic. To stop the producer client, press Ctrl-C at any time.

Read the events

To read the events you've written to your Kafka topic, use the Kafka console consumer. This client connects to the Kafka brokers, retrieves the events, and displays them in the terminal.

Open a new terminal session and run the following command to start the console consumer:

$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092

The console outputs the events you previously produced:

This is my first event
This is my second event

As with the producer, you can stop the consumer client by pressing Ctrl-C at any time.

Terminate the Kafka environment

Once you've reached the end of the quickstart, you can tear down the Kafka environment, or continue exploring it.

Stop the producer and consumer clients with Ctrl-C, if you haven't done so already.
Stop the KRaft broker and controller with Ctrl-C.

If you also want to delete the data from your local Kafka environment, including any events you created, run:

$ rm -rf /tmp/kraft-controller-logs /tmp/kraft-broker-logs