Configure Pulse to Monitor Standalone Kafka
This page describes how to configure Pulse to monitor a standalone Kafka. This configuration enables real-time observability and health monitoring of Kafka components.
Configure the Pulse Core Component for Standalone Kafka Cluster
Note
Pulse supports only Kafka versions 1.0.0 and higher.
- Run the following command to get started:
accelo config cluster
- The Acceldata CLI asks you for information about your environment. The following table lists the questions asked by Acceldata CLI and a guide to help you answer the questions.
Questions asked by CLI | Guidelines for answering the questions |
Is this current node an 'Edge Node' of the cluster? [y/n]: | Type y if the node on which you are installing is a Edge Node. Else type n. |
Which distribution do you use? | Select Standalone |
Stand-alone configuration selected: | Displays the configuration selected. |
- The CLI asks you for few more information to generate the configuration file.
Questions asked by CLI | Guidelines for answering the questions |
Enter Your Cluster’s Name: | Enter the name of the cluster. |
Enter Your Cluster’s Display Name: | Enter the name you want for the cluster to display |
Select the components you would like to install a) MEMSQL b) KAFKA | Select the components that you want to install. You can use the arrows to move, space to select, and type in to filter |
Is your Kafka version greater than '0.11.0'? [y/n] | Type y if the Kafka version is greater than V0.11.0. |
Enter one of the Kafka bootstrap server's URI | Enter the Kafka boot server URL. Example: host1.kf.com:6667 |
What security do you use? a. None b. Plan/SSL c. Kerberos | Select your security type. |
Do you use TLS ? a. No b. Yes | Select Yes if you use TLS or else select NO. |
Detected Kafka brokers: | Based on your responses CLI would detect the Kafka brokers. |
Is the above the information correct? [y/n]: | Type y to confirm if the detected Kafka brokers are correct. |
Enter the Zookeeper Server URLs (comma separated with http/https & port): | Enter the Zookeeper server URL and use comma separate http, https, and the port. |
Enter Kafka's Zookeeper Chroot Name (enter '/' for root): | Enter the Kafka Zookeeper roots |
Would you like to continue with the above configuration? [y/n]: | Type y if you want to continue with the configuration or else type n. |
Would you like to enable LDAP? [y/n]: | Type y if you want to enable LDAP or else type n. |
Is Kerberos enabled in this cluster? [y/n]: | Enter your Kerberos Keytab username (Must have required HDFS permissions) |
Enter the cluster name to use (MUST be all lowercase & unique): | Enter the cluster name to be used. Ensure that the name should be in all lowercase and unique. |
You will see the following message.
INFO: Trying to generate host-roles-map file ... ✓ Success
INFO: Trying to generate alert-endpoints file ...
INFO: Trying to generate FSAnalytics scripts ...
WARN: Cannot find any HDFS installations from this cluster. Skipping the FS Analytics configuration.
INFO: Edit the config files 'ad-core-connectors.yaml' and 'ad-fsanalyticsv2-connector.yaml'
ERROR: Cannot find the required hadoop conf directory at the path '/etc/hadoop/conf'
IMPORTANT: Please make sure this directory exists at the path '/etc/hadoop/conf' and contains the files 'core-site.xml' and 'hdfs-site.xml'
INFO: Please run 'accelo deploy core' to deploy APM core using this configuration.
- You may be asked to answer the following questions if Kerberos is enabled.
Questions asked by CLI | Guidelines for answering the questions |
Would you like to continue configuring Kerberos? [y/n]: | Type y if you want to continue configuring the Kerberos |
Authentication realm | Add the authentication realm. Example: AZ.PULSE.COM |
KDC address | Add the KDC address. Example: host1.kdc.com:88 |
Kerberos principal | Add the Kerberos principal. Example: admin/admin@AZ.PULSE.COM |
KDC admin server address | Add the KDC administration server address. Example: host1.kdcadmin.com:749 |
Configure Kafka when Security Is Disabled
If security is not enabled in your standalone Kafka or Kafka 3 cluster, update the acceldata.conf and ad-kafka-connector.yml configuration files as described below.
- Update the acceldata.conf file.
Update consumerSecurityProtocol and securityProtocol:
- From "SASL_PLAINTEXT"
- To "PLAINTEXT".
enableConsumerGroupCache = true
batchSize = 5
zk_secure = false
consumerSecurityProtocol = "PLAINTEXT"
filterConsumerGroups = ""
filterConsumerGroupsState = ""
securityProtocol = "PLAINTEXT"
SASLEnabled = "false"
TLSEnabled = "false"
- Create the ad-kafka-connector.yml file (if not present).
Run the following command to generate the configuration file:
accelo admin makeconfig ad-kafka-connector
- Update the ad-kafka-connector.yml file.
In the $AcceloHome/config/docker/addons/ad-kafka-connector.yml file, set KERBEROS_ENABLED to false.
version: "2"
services:
ad-kafka-connector:
image: ad-kafka-connector
container_name: ""
environment:
- MONGO_URI=<URI>
- MONGO_ENCRYPTED=true
- MONGO_SECRET=<Secret>
- KERBEROS_ENABLED=false
- OTEL_JAVAAGENT_ENABLED=false
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/hosts:/etc/hosts:ro
ulimits: {}
ports: []
depends_on: []
opts: {}
restart: ""
extra_hosts: []
network_alias: []
label: Kafka Connector
Checking the Configuration
Open the files located in the address /data01/acceldata/config/acceldata<CLUSTER_NAME>.conf_ to verify if all the configurations are correct.