Ambari Server Upgrade

You can perform the following steps to upgrade the Ambari Server.

The following steps have to be performed on the installed Ambari Server node.

Stop Infra-Solr Service

Stop the Infra-Solr service from the Ambari UI by navigating to Ambari > Infra Solr > Actions > Stop.

Stop Ambari Server and Agents

Stop the Ambari server and agents by running the following command.

ambari-server stop
ambari-agent stop
Stop the Ambari Agents on all nodes in the cluster.

Configuration Backup

Take a backup of the Ambari-server configs.

cp -r /etc/ambari-server/config /etc/ambari-server/conf_bkp

Back up Ambari Database

mysqldump --databases ambari > /tmp/ambari-server_db_bckp/ambari_mysql_bckp.sql

Erase the Ambari Server

For RHEL:

yum erase ambari-server -y
For Ubuntu:

apt remove ambari-server -y

Add Ambari Repo on Cluster Nodes

Back up the Old Ambari Repository

For RHEL:

mv /etc/yum.repos.d/ambari.repo /tmp/
For Ubuntu:

mv /etc/apt/sources.list.d/ambari.list /tmp/

Update the Repository Files

Retrieve the latest Ambari repository details from Ambari Repositories and add it to all the cluster nodes.

Update ambari.repo (RedHat) or ambari.list (Ubuntu).

For RHEL:

vim /etc/yum.repos.d/ambari.repo

[ambari]
async = 1
baseurl = https://mirror.odp.acceldata.dev/ODP/rhel/Ambari-3.0.0.1-1/
gpgcheck = 0
name = ambari Version - ambari-3.0.0.1-1
For Ubuntu:

vim /etc/apt/sources.list.d/ambari.list

deb https://mirror.odp.acceldata.dev/ODP/ubuntu20/Ambari-3.0.0.1-1/ ODP main
deb https://mirror.odp.acceldata.dev/ODP/ubuntu20/Ambari-3.0.0.1-1/ ODP-UTILS main

List the New Packages

Verify the latest Ambari package for the version - 3.0.0.1-1.

For RHEL:

yum clean all
yum list ambari-*
For Ubuntu 20:

apt clean all
apt list ambari-*

Install the Updated Ambari Packages

For RHEL:

yum clean all
yum install ambari-server -y
For Ubuntu:

apt clean
apt install ambari-server -y

Restore the Configuration Files

Copy ambari.properties and password.dat to /etc/ambari-server/conf/.

Once the installation is successful, make sure to set up the following versions.

Set up Java 17

Make sure to install and update the default Java version as 17 on all the cluster nodes.

For RHEL:

yum install java-17-openjdk.x86_64
yum install java-17-openjdk-devel.x86_64
For Ubuntu:

apt-get install openjdk-17-jdk openjdk-17-jdk-headless

update-alternatives --config java

There are 2 choices for the alternative java (providing /usr/bin/java).

Selection Path Priority Status

------------------------------------------------------------

* 0 /usr/lib/jvm/java-17-openjdk-amd64/bin/java 1111 auto mode

1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode

2 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java 1081 manual mode

Press <enter> to keep the current choice[*], or type selection number: 1

Set up Python 3.11

yum install python3.11
Make sure to install and update the default Python version as 3.11 on all the cluster nodes.

update-alternatives --config python
There are 4 programs which provide 'python'.
Selection Command
-----------------------------------------------
* 1 /usr/libexec/no-python
2 /usr/bin/python3
3 /usr/bin/python2
+ 4 /usr/bin/python3.11
Enter to keep the current selection[+], or type selection number: 4

Update JSVC

Make sure to Instal the latest version of the bigtop-jsvc-1.2.4 on all the cluster nodes.

For RHEL:

yum install bigtop-jsvc* -y
For Ubuntu:

apt install bigtop-jsvc

Restore Config Files

Restore the ambari.properties and password.dat files, which are backed up at this step: Configuration Backup on this page.

Restore the ambari.properties file

cp /etc/ambari-server/conf_bkp/ambari.properties /etc/ambari-server/conf/
Please override the file by prompting 'y'.

Restore the password.dat file

cp /etc/ambari-server/conf_bkp/password.dat /etc/ambari-server/conf/

Update the java.home and stack.java.home

In this case, set JAVA_HOME to /usr/lib/jvm/java-17-openjdk

File path - /etc/ambari-server/conf/ambari.properties | grep -i java.home

java.home=/usr/lib/jvm/java-17-openjdk
stack.java.home=/usr/lib/jvm/java-17-openjdk
ambari.java.home=/usr/lib/jvm/java-17-openjdk
mpacks-v2.staging.path=/var/lib/ambari-server/resources/mpacks-v2

Update the Ambari Database with New Tables and Columns

MySQL

mysql -u ambari -pambari
USE ambari;
CREATE TABLE IF NOT EXISTS registries(
id BIGINT NOT NULL,
registy_name VARCHAR(255) NOT NULL,
registry_type VARCHAR(255) NOT NULL,
registry_uri VARCHAR(255) NOT NULL,
CONSTRAINT PK_registries PRIMARY KEY (id));
CREATE TABLE IF NOT EXISTS mpacks(
id BIGINT NOT NULL,
mpack_name VARCHAR(255) NOT NULL,
mpack_version VARCHAR(255) NOT NULL,
mpack_uri VARCHAR(255),
registry_id BIGINT,
CONSTRAINT PK_mpacks PRIMARY KEY (id),
CONSTRAINT uni_mpack_name_version UNIQUE(mpack_name, mpack_version),
CONSTRAINT FK_registries FOREIGN KEY (registry_id) REFERENCES registries(id));
ALTER TABLE stack ADD COLUMN mpack_id BIGINT NULL;
ALTER TABLE stack ADD CONSTRAINT FK_mpacks FOREIGN KEY (mpack_id) REFERENCES mpacks(id);
ALTER TABLE upgrade ADD COLUMN upgrade_package_stack VARCHAR(255) NOT NULL;

MariaDB

mysql -u ambari -pambari
USE ambari;
CREATE TABLE IF NOT EXISTS registries(
id BIGINT NOT NULL,
registy_name VARCHAR(255) NOT NULL,
registry_type VARCHAR(255) NOT NULL,
registry_uri VARCHAR(255) NOT NULL,
CONSTRAINT PK_registries PRIMARY KEY (id));
CREATE TABLE IF NOT EXISTS mpacks(
id BIGINT NOT NULL,
mpack_name VARCHAR(255) NOT NULL,
mpack_version VARCHAR(255) NOT NULL,
mpack_uri VARCHAR(255),
registry_id BIGINT,
CONSTRAINT PK_mpacks PRIMARY KEY (id),
CONSTRAINT uni_mpack_name_version UNIQUE(mpack_name, mpack_version),
CONSTRAINT FK_registries FOREIGN KEY (registry_id) REFERENCES registries(id));
ALTER TABLE stack ADD COLUMN mpack_id BIGINT NULL;
ALTER TABLE stack ADD CONSTRAINT FK_mpacks FOREIGN KEY (mpack_id) REFERENCES mpacks(id);
ALTER TABLE upgrade ADD COLUMN upgrade_package_stack VARCHAR(255) NOT NULL;

-- 0) Use Ambari schema
USE ambari;
-- 1) (Optional) Confirm current FK names on affected tables
SELECT TABLE_NAME, CONSTRAINT_NAME
FROM information_schema.TABLE_CONSTRAINTS
WHERE CONSTRAINT_SCHEMA = DATABASE()
AND CONSTRAINT_TYPE = 'FOREIGN KEY'
AND TABLE_NAME IN ('servicecomponentdesiredstate', 'hostcomponentdesiredstate', 'hostcomponentstate')
ORDER BY TABLE_NAME, CONSTRAINT_NAME;
-- 2) Drop child FKs first, then parent FK
ALTER TABLE hostcomponentdesiredstate DROP FOREIGN KEY hstcmpnntdesiredstatecmpnntnme;
ALTER TABLE hostcomponentstate DROP FOREIGN KEY hstcomponentstatecomponentname;
ALTER TABLE servicecomponentdesiredstate DROP FOREIGN KEY srvccmponentdesiredstatesrvcnm;
-- 3) Alter service_name columns to VARCHAR(255)
ALTER TABLE servicecomponentdesiredstate MODIFY COLUMN service_name VARCHAR(255) NOT NULL;
ALTER TABLE hostcomponentdesiredstate MODIFY COLUMN service_name VARCHAR(255) NOT NULL;
ALTER TABLE hostcomponentstate MODIFY COLUMN service_name VARCHAR(255) NOT NULL;
-- 4) Recreate FKs
ALTER TABLE servicecomponentdesiredstate
ADD CONSTRAINT srvccmponentdesiredstatesrvcnm
FOREIGN KEY (service_name, cluster_id)
REFERENCES clusterservices (service_name, cluster_id);
ALTER TABLE hostcomponentdesiredstate
ADD CONSTRAINT hstcmpnntdesiredstatecmpnntnme
FOREIGN KEY (component_name, service_name, cluster_id)
REFERENCES servicecomponentdesiredstate (component_name, service_name, cluster_id);
ALTER TABLE hostcomponentstate
ADD CONSTRAINT hstcomponentstatecomponentname
FOREIGN KEY (component_name, service_name, cluster_id)
REFERENCES servicecomponentdesiredstate (component_name, service_name, cluster_id);
-- 5) Verify column definitions
SHOW COLUMNS FROM servicecomponentdesiredstate LIKE 'service_name';
SHOW COLUMNS FROM hostcomponentdesiredstate LIKE 'service_name';
SHOW COLUMNS FROM hostcomponentstate LIKE 'service_name';

Oracle 19C

sqlplus / as sysdba
ALTER SESSION SET CONTAINER=<DB Name>;
ALTER SESSION SET CURRENT_SCHEMA = <UserName>;
CREATE TABLE registries(
id NUMBER(19) NOT NULL,
registy_name VARCHAR2(255) NOT NULL,
registry_type VARCHAR2(255) NOT NULL,
registry_uri VARCHAR2(255) NOT NULL,
CONSTRAINT PK_registries PRIMARY KEY (id)
);
CREATE TABLE mpacks(
id NUMBER(19) NOT NULL,
mpack_name VARCHAR2(255) NOT NULL,
mpack_version VARCHAR2(255) NOT NULL,
mpack_uri VARCHAR2(255),
registry_id NUMBER(19),
CONSTRAINT PK_mpacks PRIMARY KEY (id),
CONSTRAINT uni_mpack_name_version UNIQUE(mpack_name, mpack_version),
CONSTRAINT FK_registries FOREIGN KEY (registry_id) REFERENCES registries(id)
);
ALTER TABLE stack ADD mpack_id NUMBER(19) NULL;
ALTER TABLE stack ADD CONSTRAINT FK_mpacks FOREIGN KEY (mpack_id) REFERENCES mpacks(id);

Postgres

Ambari Server Upgrade Fails on PostgreSQL Due to Missing ambari Schema

Summary:

When Ambari is backed by PostgreSQL and ambari-server upgrade is executed, the Java SchemaUpgradeHelper process fails immediately because it attempts to set the PostgreSQL search path to a schema named ambari, but that schema does not exist. All Ambari tables are instead located in the default public schema.

This issue is specific to PostgreSQL schema behavior.

Other database platforms do not encounter this problem because:

• MySQL treats the database and schema as the same object, so no separate schema creation is required.
• Oracle automatically creates a schema for each database user.
• PostgreSQL requires schemas to be explicitly created. Unless a schema is manually created beforehand, tables are created in the default public schema.

SchemaUpgradeHelper:483 - Exception occurred during upgrade

org.apache.ambari.server.AmbariException: ERROR: schema "ambari" does not exist

Caused by: org.postgresql.util.PSQLException: ERROR: schema "ambari" does not exist

at AbstractUpgradeCatalog.changePostgresSearchPath(AbstractUpgradeCatalog.java:397)

A secondary Python-level error may also appear:

ambari_simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
This is not the root cause. The Java upgrade process terminated unexpectedly and returned empty output, causing the Python wrapper to fail while attempting to parse the response.
Root Cause
The Ambari ambari.properties contains the property:

server.jdbc.postgres.schema=ambari
This causes AbstractUpgradeCatalog.changePostgresSearchPath() to execute:

SET search_path TO ambari;
The upgrade fails because the ambari schema does not exist. The Ambari PostgreSQL DDL was run before the schema was created, so PostgreSQL placed all Ambari tables in the default public schema.
The database should have been initialized with:

CREATE SCHEMA ambari AUTHORIZATION ambari;
ALTER ROLE ambari SET search_path TO ambari, public;
Diagnosis

1. Read the connection details from ambari.properties

DB_HOST=$(grep server.jdbc.hostname /etc/ambari-server/conf/ambari.properties | cut -d= -f2)
DB_PASS=$(cat /etc/ambari-server/conf/password.dat)
echo "Host: $DB_HOST"

1. Check which schemas exist

PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c "\dn"
-- Expected (broken state):
--   Name  |       Owner
-- --------+-------------------
--  public | pg_database_owner
-- Expected (correct state after fix):
--   Name  |  Owner
-- --------+--------
--  ambari | ambari
--  public | pg_database_owner

1. Check where the tables are located

PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c "
SELECT table_schema, count(*) AS table_count
FROM information_schema.tables
WHERE table_schema NOT IN ('information_schema','pg_catalog')
GROUP BY table_schema;"
##-- Broken state:  public | 113
##-- Fixed state:   ambari | 113
Fix Procedure

Set connection variables and verify connectivity:

DB_HOST=$(grep server.jdbc.hostname /etc/ambari-server/conf/ambari.properties | cut -d= -f2)
DB_PASS=$(cat /etc/ambari-server/conf/password.dat)
# Verify connection
PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c "SELECT 1;"

1. Create the ambari schema and move all objects

Create the schema, move tables and sequences, and set the search path:

PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 << 'EOF'
-- Create schema
CREATE SCHEMA IF NOT EXISTS ambari AUTHORIZATION ambari;
-- Move all tables from public to ambari
DO $$
DECLARE r RECORD;
BEGIN
  FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public'
  LOOP
    EXECUTE 'ALTER TABLE public.' || quote_ident(r.tablename) || ' SET SCHEMA ambari';
    RAISE NOTICE 'Moved table: %', r.tablename;
  END LOOP;
END;
$$;
-- Move all sequences from public to ambari
DO $$
DECLARE r RECORD;
BEGIN
  FOR r IN SELECT sequence_name FROM information_schema.sequences
               WHERE sequence_schema = 'public'
  LOOP
    EXECUTE 'ALTER SEQUENCE public.' || quote_ident(r.sequence_name) || ' SET SCHEMA ambari';
    RAISE NOTICE 'Moved sequence: %', r.sequence_name;
  END LOOP;
END;
$$;
-- Set permanent search path for ambari role
ALTER ROLE ambari SET search_path TO ambari, public;
EOF

1. Verify

Verify all tables are now in the ambari schema:

# All tables should now be in ambari schema
PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c "
SELECT table_schema, count(*) AS table_count
FROM information_schema.tables
WHERE table_schema NOT IN ('information_schema','pg_catalog')
GROUP BY table_schema;"
# Expected output:
#  table_schema | table_count
# --------------+-------------
#  ambari       |         113
# Verify search path is set
PGPASSWORD=$DB_PASS psql -U ambari -d ambari -h $DB_HOST -p 5432 -c "SHOW search_path;"
# Expected: ambari, public

1. Retry the upgrade

ambari-server upgrade -v
Why does this only affect PostgreSQL?

Prevention

For new Ambari PostgreSQL deployments, create the schema before running the Ambari DDL:

CREATE DATABASE ambari;
CREATE USER ambari WITH PASSWORD 'your_password';
\c ambari
CREATE SCHEMA ambari AUTHORIZATION ambari;
ALTER ROLE ambari SET search_path TO ambari, public;
Then run the Ambari DDL:

psql -U ambari -d ambari -f Ambari-DDL-Postgres-CREATE.sql
Correct Setup Checklist:
Before running Ambari PostgreSQL DDL, confirm that:

• Database ambari exists.
• User ambari exists.
• password.dat contains the correct password.
• Schema ambari exists.
• Schema ambari is owned by user ambari.
• ALTER ROLE ambari SET search_path TO ambari, public has been run.
• Ambari tables are created in the ambari schema, not public.

Upgrade the Ambari Server Database Schema

ambari-server upgrade -v
Upon running the command, the response will be as shown in the screenshot.
Make sure to take the database backup as mentioned earlier and enter 'y' here.

Start the Ambari Server

Once started, you are able to access the Ambari UI using the same url as it was earlier.

ambari-server start

Access the Ambari UI

Use your browser to access the Ambari UI.

Check the Upgraded Version

To verify the upgraded Ambari version, navigate to Admin → About in the top-right corner of the Ambari UI.

All the services also must be working normally without any downtime.

Known Issues

Pip dependency missing

[root@ce19 ~]# ambari-server upgrade
Using python /usr/bin/python3
Upgrading ambari-server
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 28, in <module>
from ambari_commons.exceptions import FatalException, NonFatalException
File "/usr/lib/ambari-server/lib/ambari_commons/init.py", line 21, in <module>
from ambari_commons.os_check import OSCheck, OSConst
File "/usr/lib/ambari-server/lib/ambari_commons/os_check.py", line 25, in <module>
import distro
ModuleNotFoundError: No module named 'distro'

Workaround:

pip3 install distro
If still hitting the distro module not found error, follow the below steps:
Step 1:

1. Download the get-pip.py script:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

1. Install pip for Python 3.11:

/usr/bin/python3.11 get-pip.py
Step 2: Install the distro module using pip3.11
Once `pip` is installed for Python 3.11, you can now use it to install the missing `distro` module:

/usr/bin/python3.11 -m pip install distro
Step 3: Verify the installation.
After installing the distro module, verify that it’s correctly installed:

/usr/bin/python3.11 -c "import distro"
The ambari-server upgrade -v upgrade is successfully done.

Mpacks uninstallation not done

Traceback (most recent call last):
File "/usr/lib/ambari-server/lib/ambari_simplejson/scanner.py", line 49, in _scan_once
nextchar = string[idx]
IndexError: string index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 1083, in <module>
mainBody()
File "/usr/sbin/ambari-server.py", line 1053, in mainBody
main(options, args, parser)
File "/usr/sbin/ambari-server.py", line 999, in main
action_obj.execute()
File "/usr/sbin/ambari-server.py", line 78, in execute
self.fn(*self.args, **self.kwargs)
File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade
retcode = run_schema_upgrade(args)
File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade
upgrade_response = json.loads(stdout)
File "/usr/lib/ambari-server/lib/ambari_simplejson/init.py", line 538, in loads
return _default_decoder.decode(s)
File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 378, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 408, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
File "/usr/lib/ambari-server/lib/ambari_simplejson/scanner.py", line 93, in scan_once
return _scan_once(string, idx)
File "/usr/lib/ambari-server/lib/ambari_simplejson/scanner.py", line 51, in _scan_once
raise JSONDecodeError(errmsg, string, idx)
ambari_simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Workaround: Uninstall all the ambari-mpacks.

ambari-server uninstall-mpack --mpack-name=<installed mpack>