Acceldata
 Pulse
Search documentation… ⌘K

Configure Pulse to Monitor Standalone Kafka 3

This page describes how to configure Pulse to monitor a standalone Kafka 3 deployment, supporting both KRaft and ZooKeeper modes. This configuration enables real-time observability and health monitoring of Kafka components.

Introduced

This is supported from Pulse 4.0.0 onwards.

Run Accelo Config Command

On a Pulse node, run the following command.

accelo config cluster

Answer the CLI Questions

Upon running the accelo config cluster command, Pulse CLI prompts with the following questions.

Questions asked by CLI

Guidelines for answering the questions

Select the cluster type

Select the cluster type as Standalone. [x] Stand-Alone [ ] Ambari [ ] Cloudera

Select the Kafka 3 Standalone component.

[ ] Kafka

[ ] NiFi

[ ] Spark

[x] Kafka 3

If your Kafka 3 deployment uses KRaft mode, select Kafka 3.

If your Kafka 3 deployment uses ZooKeeper mode, select Kafka.

Based on your environment, select the appropriate component and enter the required details.

In this document, the scenario assumes a Kafka 3 deployment with KRaft mode. However, the remaining steps are the same for both modes.

Enter Your Cluster's Display Name

Enter the cluster name.

Enter Kafka 3 Version

Enter the Kafka 3 version.

Enter one of the Kafka bootstrap server's URIs (e.g., host1.kf.com:6667):

Enter one of the Kafka 3 bootstrap server URIs

Select the detected Kafka 3 brokers after Pulse auto-detects them.

✓ Detected kafka3 brokers: kafka101.acceldata.dvl:9092 kafka102.acceldata.dvl:9092 kafka103.acceldata.dvl:9092 ✓ Total number of brokers: 3

Select the security type used by your Kafka cluster.

On Pulse 4.0.3 or earlier:

  • None
  • Plain/SASL
  • Scram

    • Based on the selection of each security type, the CLI prompts with a set of questions to enter the details.

    Enter the Kraft Controller URLs (comma separated with host & port, e.g., krafthost1:19092,krafthost2:19092)

    Enter the Kraft Controller URLs (e.g., krafthost1:19092,krafthost2:19092).

    Do you want to monitor KafkaMM2 [y/n]

    Enter the Kafka MirrorMaker nodes (comma-separated hosts, e.g., mm2kafka1,mm2kafka2)

    Type y if you want to monitor Kafka Mirror Maker 2 and enter the host names.

    Do you want to monitor Kafka Connect [y/n]

    Enter the Kafka Connect nodes (comma-separated hosts, e.g., kafkaconnect1,kafkaconnect2):

    Type y if you want to monitor Kafka Connect and enter the host names.

    Do you want to monitor Kafka Cruise Control [y/n]

    Enter the Kafka Cruise Control nodes (comma-separated hosts, e.g., kafkaconnect1,kafkaconnect2)

    Type y if you want to monitor Kafka Cruise Control and enter the host names.

    Do you want to monitor Schema Registry [y/n]

    Enter the Schema Registry nodes (comma-separated hosts, e.g., schemaregistry1,schemaregistry2)

    Type y if you want to monitor the Schema Registry and enter the host names.

    Add JMX ports for data collection

    Enter the Kafka3 Broker Port: : 9092

    Enter the Kafka JMX Port (Ex, 9999): 9999

    Enter the Kafka3 MirrorMaker JMX Port (Ex, 9998): 9998

    Enter the Cruise Control3 JMX Port (default: 56667): 56667

    Enter the Kafka3 Connect JMX Port (default: 9994): 9994

    Enter the Schema Registry JMX Port (default: 8996)::8996

    Select logs for installed/enabled components.

    Select the logs for components that are installed/enabled in your target cluster:

    [ x ] kraftbroker_err

    [ x ] kafka3_connect_err, kern

    [ x ] kafka3_controller

    [ x ] kafka3_connect

    [ x ] kafka3_cruise_control

    [ x ] registry_err

    [ x ] kafka3_log_cleaner

    [ x ] kafka3_server

    [ x ] kafka3_server_gc

    [ x ] kraftcontroller_err

    [ x ] syslog, registry_server

    [ x ] kafka3_state_change

    [ x ] kafka3_cruise_control_err

    Add the Component Log Directory

    Add the component log directory by creating an override.yml file (if not exist) in $AcceloHome/work/<clustername>/ and specifying the required configuration settings as below.

    log_locations:
kafka3_connect:
- path: /path_to/connect.log
type: DEFAULT
kafka3_connect_err:
- path: /path_to/kafka-connect.err
type: KAFKA_ERR
kafka3_controller:
- path: /path_to/controller.log
type: DEFAULT
kafka3_cruise_control:
- path: /path_to/kafkacruisecontrol.log
type: DEFAULT
kafka3_cruise_control_err:
- path: /path_to/cruise-control.err
type: KAFKA_ERR
kafka3_log_cleaner:
- path: /path_to/log-cleaner.log
type: DEFAULT
kafka3_server:
- path: /path_to/server.log
type: DEFAULT
kafka3_server_gc:
- path: /path_to/kafkaServer-gc.log
type: DEFAULT
kafka3_state_change:
- path: /path_to/state-change.log
type: DEFAULT
kraftbroker_err:
- path: /path_to/kraft-broker.err
type: DEFAULT
kraftcontroller_err:
- path: /path_to/kraft-controller.err
type: DEFAULT
registry_err:
- path: /path_to/registry.err
type: REGISTRY
registry_server:
- path: /path_to/registry.log
type: REGISTRY
Run the following command to update the vars.yml file with the new configuration.

    accelo reconfig cluster

    Verify the Configuration File

    Once the configuration is complete, you can verify the final setup in the following path.

    $AcceloHome/config/acceldata<cluster name>.conf

    Run Accelo Deploy Core Command

    Run the following command. Once the deployment is complete, the Pulse UI will be up and running.

    accelo deploy core