Ambari High Availability Setup

This document outlines the process for configuring high availability (HA) for Ambari Server using an active-passive setup. It also includes instructions for optional load balancing of agents with HAProxy.

Prerequisites

Two identically configured Ambari server instances.
A shared external database for both Ambari server instances.
A load balancer (such as HAProxy).

Steps to Setup

Perform the following steps:

1. Set Up Ambari Servers

Deploy two Ambari server instances.
Ensure both instances have identical configurations, including Ambari property files.
Connect both servers to an external database to maintain consistency.
Use ambari-server setup command to configure both instances in the same manner.

2. Install and Configure HAProxy

If HAProxy is not installed, use sudo yum install haproxy -y.
Modify the HAProxy configuration file at /etc/haproxy/haproxy.cfg.
Replace existing frontend and backend settings with:

frontend ambari_frontend
bind *:8080
mode http
default_backend ambari_backend
backend ambari_backend
mode http
balance roundrobin
option httpchk GET /
server ambari_server1 <Ambari_Server_1_IP>:8080 check
server ambari_server2 <Ambari_Server_2_IP>:8080 backup check
In this configuration:

ambari_server1 is the primary server.
ambari_server2 serves as the backup, receiving traffic only if ambari_server1 is down.
The backup designation indicates ambari_server2 is a passive server.

3. Restart HAProxy

Apply the new settings by restarting HAProxy:

sudo systemctl restart haproxy
sudo systemctl enable haproxy
sudo systemctl status haproxy

4. Optional: Configure HAProxy Logging

Set up logging by installing rsyslog and configuring logging for HAProxy:

sudo yum install rsyslog -y
echo '$ModLoad imudp' | sudo tee /etc/rsyslog.d/haproxy.conf
echo '$UDPServerRun 514' | sudo tee -a /etc/rsyslog.d/haproxy.conf
echo '$template HaproxyLog,"/var/log/haproxy.log"' | sudo tee -a /etc/rsyslog.d/haproxy.conf
echo ':programname, startswith, "haproxy" -?HaproxyLog' | sudo tee -a /etc/rsyslog.d/haproxy.conf
sudo systemctl restart rsyslog

Check the logs at /var/log/haproxy.log to confirm proper logging.

5. Update Ambari Agent Configuration

Direct Ambari agents to connect to the active server using the provided script:

#!/bin/bash
# Ambari Server Details
AMBARISERVER=$(hostname -f)
USER=admin
PASSWORD=admin
PORT=8080
PROTOCOL=http
# Load Balancer Details
ACTIVE_AMBARI=odp1-mar2024.adsre.com
# Get the cluster name
CLUSTER=$(curl -ksu $USER:$PASSWORD -i -H 'X-Requested-By: ambari' $PROTOCOL://$AMBARISERVER:$PORT/api/v1/clusters | grep -o '"cluster_name" : "[^"]*' | awk -F'"' '{print $4}')
# Get the list of all hosts in the cluster
HOSTS=($(curl -su $USER:$PASSWORD $PROTOCOL://$AMBARISERVER:$PORT/api/v1/clusters/$CLUSTER/hosts | grep -o '"host_name" : "[^"]*' | awk -F'"' '{print $4}'))
# Loop through each host and update ambari-agent.ini
for HOST in "${HOSTS[@]}"; do
# Get current hostname from ambari-agent.ini on remote host
CURRENT_HOSTNAME=$(ssh $HOST "grep 'hostname=' /etc/ambari-agent/conf/ambari-agent.ini | cut -d'=' -f2")
# Check if the hostname needs to be updated
if [ "$CURRENT_HOSTNAME" != "$ACTIVE_AMBARI" ]; then
# Backup the original ambari-agent.ini file
ssh $HOST "sudo cp -f /etc/ambari-agent/conf/ambari-agent.ini /etc/ambari-agent/conf/ambari-agent.ini.bak"
# Update hostname in ambari-agent.ini on remote host
ssh $HOST "sed -i 's/^hostname=.*/hostname=$ACTIVE_AMBARI/' /etc/ambari-agent/conf/ambari-agent.ini"
# Restart ambari-agent on remote host
ssh $HOST "sudo systemctl restart ambari-agent"
echo "Hostname on $HOST updated to $ACTIVE_AMBARI. Ambari agent restarted."
else
echo "Hostname on $HOST is already set to $ACTIVE_AMBARI. No action needed."
fi
done
If Ambari Server is unavailable, use the script to redirect agents to the operational server.

Conclusion

Following these steps ensures the successful implementation of high availability for the Ambari Server with HAProxy, enhancing both availability and reliability of your managed cluster.

Known Limitations

Heartbeat Lost After Failover: If the first Ambari Server goes down, HAProxy will redirect the request to the second Ambari Server, but you may observe "Heartbeat Lost" in the Ambari UI. Use 5. Update Ambari Agent Configuration (Step 5) to address any heartbeat issues post-failover.
Access Denied for Database User: Modify database permissions to allow access from the HA host if you see the following error:

Caused by: java.sql.SQLException: Access denied``for``user 'ambari'@'ambari-ha.adsre.com' (using password: YES)

Solution:

grant all privileges on *.* to 'ambari'@'ambari-ha.adsre.com' indentified by 'bigdata' with grant option ;

No Database Selected: Verify that server.jdbc.url in ambari.properties specifies the correct database name, if you see the following error:

Caused by: java.sql.SQLException: No database selected