Configure Pulse to Monitor Multiple Kafka clusters under a single Hadoop Cluster

Steps

1. Add Multiple Kafka Clusters in Config

Edit:

<acceldata_clustername>.conf
Under kafka.connectors, define multiple clusters:

kafka.connectors = [
{
name = "kafka-0.11-connector"
trigger = "scheduled"
service = "kafka"
maxFailRetries = 5
clusters = [
{
name = "kafka-01"
servers = "host1:9092,host2:9092,host3:9092"
zk_servers = "host1:2181,host2:2181,host3:2181/kafka"
securityProtocol = "SASL_PLAINTEXT"
},
{
name = "kafka-02"
servers = "host4:9092"
zk_servers = "host1:2181,host2:2181,host3:2181/kafka12"
securityProtocol = "SASL_PLAINTEXT"
}
]
}
]
Key Notes

• Keep the first cluster name the same as the Hadoop cluster name (avoids reconfiguring all agents).
• Ensure:
  • Unique name per Kafka cluster
  • Correct zk_servers (different chroot if shared ZK)

1. Push Config to Database

accelo admin database push-config

1. Restart Kafka Connector

accelo restart ad-kafka-connector

1. Define Target Hosts for Reconfiguration

Edit/create:

$ACCELDATA_HOME/work/<cluster_name>/hydra_hosts_override.yml
Example:

hosts:
add:
- host1
- host2
- host3
remove:
- host4
- host5
- host6
Purpose:

• add → nodes to be reassigned to the new Kafka cluster
• remove → excluded nodes

1. Update JMX Cluster Mapping

Edit:

override.yml

static:
jmx_rmi_db: <kafka-cluster-name>_jmx
Example:

kafka-02_jmx

1. Reconfigure Agents

accelo reconfig cluster
This updates:

• Kafka JMX cluster name
• Only for hosts in add list

1. Repeat for Additional Kafka Clusters

Repeat Steps 4–6 for each Kafka cluster.

Validation Checklist

1. Check the db_name in /opt/pulse/jmx/config/enabled/vm_kafka.json file.
2. Check the cluster name in JMX metrics through Dashplot studio.