Acceldata
 ODP
Search documentation… ⌘K

Upgrade Standalone Kafka 4

This page explains how to start a standalone Kafka 3 cluster in KRaft mode, validate its functionality, and perform a rolling upgrade from Kafka 3.7.2.3.3.6.x-x to Kafka 4.3.0.3.3.6.x-x.

Start Kafka (3.7.2.3.3.6.x-x) in KRaft mode

For details about the ODP standalone binaries, see Accessing Acceldata Repositories.

Step 1: Configure the Kafka controllers

Controllers in KRaft mode are responsible for managing metadata across the cluster.

Edit the config/kraft/controller.properties file to define your controller configurations:

# Example configuration for a controller (config/kraft/controller.properties):
process.roles=controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://localhost:9093
log.dirs=/tmp/kraft-controller-logs

Step 2: Configure the Kafka broker

Once the controllers are set up, configure the brokers, which handle message storage and processing.

Edit the config/kraft/broker.properties file to define your broker configurations:

# Example configuration for a broker (config/kraft/broker.properties):
process.roles=broker
node.id=2
listeners=PLAINTEXT://localhost:9092
controller.quorum.voters=1@localhost:9093
log.dirs=/tmp/kraft-broker-logs

Step 3: Initialize the Kafka storage

In KRaft mode, Kafka uses a unique cluster.id to manage metadata. Before you start any services, generate and initialize the storage directories for the Kafka controllers and brokers:

# Generate a cluster UUID on any one of the nodes:
$ KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
 
# Format log directories for the controller
$ bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/controller.properties
 
# Format log directories for the broker
$ bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/broker.properties

Kafka3 requires Java to run. Ensure that Java 11 is installed by running the following command:

$ java -version

Set JAVA_HOME to point to Java 11. Ensure Kafka3 is running in KRaft mode. If it's running in ZooKeeper mode, follow the Zookeeper to Kraft Migration guide to migrate from ZooKeeper to KRaft before you perform the Kafka upgrade.

Step 4: Start the Kafka controllers

Once the configuration is complete, start the Kafka controller services. In KRaft mode, start the controllers first, because they manage the metadata for the brokers.

$ bin/kafka-server-start.sh config/kraft/controller.properties

Step 5: Start the broker

Open another terminal session and run the broker service:

$ bin/kafka-server-start.sh config/kraft/broker.properties

Once all the services are running, you have a basic Kafka environment ready for use.

Pre-upgrade validation

Before you proceed with the upgrade, validate the Kafka setup by completing the following steps.

Step 1: List topics

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Example output:

__consumer_offsets
quickstart-events

Step 2: Create a test topic

bin/kafka-topics.sh --create --topic Test1 --bootstrap-server localhost:9092

Example output:

Created topic Test1

Step 3: Write events into the topic

bin/kafka-console-producer.sh --topic Test1 --bootstrap-server localhost:9092

Input:

>This is my first event
>This is my second event

Terminate with Ctrl-C.

Step 4: Read events from the topic

bin/kafka-console-consumer.sh --topic Test1 --from-beginning --bootstrap-server localhost:9092

Example output:

This is my first event
This is my second event

Upgrade options

For details about upgrading Kafka 3 to Kafka 4, see the following pages: