Upgrade Mpacks

From the 3.2.3.7-x release onwards, the following Mpack services are supported for upgrade:

• Spark3
  • Spark3-355
  • Spark3-351
  • Spark3-333
• Pinot
• Impala
• Kafka3
• Kafka2 add-on services (Connect, Cruise Control, Mirror Maker2)

Step 1: Prepare Spark3 Mpacks

Spark3 Mpack upgrades support environments that use multiple Spark3 versions.

Spark3 uses a unified Mpack in earlier releases, where a single Mpack supports the following Spark3 versions:

• Spark3-355
• Spark3-351
• Spark3-333

The default version is Spark3-355.

If the existing environment uses multiple Spark3 versions, complete the following steps on the Ambari Server host to retain the corresponding Spark3 versions during the platform upgrade.

Step 1.1: Navigate to the Mpack Directory

On the Ambari Server host, run:

cd /var/lib/ambari-server/resources/mpacks

Step 1.2: Create the Spark3-351 and Spark3-333 Mpack Directories

Create copies of the Spark3-355 Mpack directory for Spark3-351 and Spark3-333:

cp -r spark3-ambari-3.5.5.mpack-3.5.5 spark3-ambari-3.5.1.mpack-3.5.1
cp -r spark3-ambari-3.5.5.mpack-3.5.5 spark3-ambari-3.3.3.mpack-3.3.3

Step 1.3: Verify the Mpack Directories

Verify that the required Spark3 Mpack directories are available:

ls -lrth /var/lib/ambari-server/resources/mpacks/spark3-ambari-3.*
Verify that the following directories are listed:

spark3-ambari-3.5.5.mpack-3.5.5
spark3-ambari-3.5.1.mpack-3.5.1
spark3-ambari-3.3.3.mpack-3.3.3
Expected Result: The Mpack directories for all Spark3 versions used in the cluster are available.

Step 2: Update the Spark3 Mpack Metadata

Update the name and version properties in the mpack.json files for the copied Spark3-351 and Spark3-333 Mpack directories.

Step 2.1 Update the Spark3-351 Mpack

Open the following file:

vim /var/lib/ambari-server/resources/mpacks/spark3-ambari-3.5.1.mpack-3.5.1/mpack.json
Locate the following properties:

{
"type": "full-release",
"name": "spark3-ambari-3.5.5.mpack",
"version": "3.5.5",
"description": "Ambari Management Pack for Spark3 by Acceldata"
}
Update the name and version properties for Spark3-351:

{
"type": "full-release",
"name": "spark3-ambari-3.5.1.mpack",
"version": "3.5.1",
"description": "Ambari Management Pack for Spark3 by Acceldata"
}
Save the changes.

Step 2.2: Update the Spark3-333 Mpack

Open the following file:

vim /var/lib/ambari-server/resources/mpacks/spark3-ambari-3.3.3.mpack-3.3.3/mpack.json
Locate the following properties:

{
"type": "full-release",
"name": "spark3-ambari-3.5.5.mpack",
"version": "3.5.5",
"description": "Ambari Management Pack for Spark3 by Acceldata"
}
Update the name and version properties for Spark3-333:

{
"type": "full-release",
"name": "spark3-ambari-3.3.3.mpack",
"version": "3.3.3",
"description": "Ambari Management Pack for Spark3 by Acceldata"
}
Save the changes.
Expected Result: The mpack.json files contain the correct Mpack name and version for Spark3-351 and Spark3-333.

Step 3: Restart Ambari Server

After updating the Spark3 Mpack metadata, restart Ambari Server:

ambari-server restart
Verify that Ambari Server starts successfully before proceeding.
Expected Result: Ambari Server restarts successfully with the updated Spark3 Mpack configuration.

Step 4. Download the Mpack Packages

On the Ambari Server host, create a directory to store the Mpack packages:

mkdir -p /root/mpacks
cd /root/mpacks
Download the required Mpack packages for the services and Spark3 versions deployed in the cluster.
For the example environment, download the packages for:

• Spark3-355
• Spark3-351
• Spark3-333
• Pinot
• Impala
• Kafka3

Use the appropriate Mpack download locations for the target ODP release.

For example:

wget <mpack_download_url>
Repeat the command for each required Mpack package.

Step 5: Verify the Downloaded Packages

Run:

ls -lrth
For the example 3.2.3.7-2011 release, the downloaded packages include:

ambari-mpacks-spark3_3.5.1.3.2.3.7-2011.tar.gz
ambari-mpacks-spark3_3.3.3.3.2.3.7-2011.tar.gz
ambari-mpacks-spark3.3.2.3.7-2011.tar.gz
ambari-mpacks-pinot-1.3.0.3.2.3.7-2011.tar.gz
ambari-mpacks-kafka3-3.7.2.3.2.3.7-2011.tar.gz
ambari-mpacks-impala-4.4.0.3.2.3.7-2011.tar.gz

Step 5: Upgrade the Mpacks

Upgrade each Mpack that corresponds to a service or Spark3 version deployed in the cluster.

Step 5.1: Upgrade Spark3-333

Run:

ambari-server upgrade-mpack \
--mpack=ambari-mpacks-spark3_3.3.3.3.2.3.7-2011.tar.gz
Verify that the command completes successfully.

Step 5.2: Upgrade Spark3-351

Run:

ambari-server upgrade-mpack \
--mpack=ambari-mpacks-spark3_3.5.1.3.2.3.7-2011.tar.gz
Verify that the command completes successfully.

Step 5.3: Upgrade Spark3-355

Run:

ambari-server upgrade-mpack \
--mpack=ambari-mpacks-spark3.3.2.3.7-2011.tar.gz
Verify that the command completes successfully.

Step 5.4: Upgrade Pinot

Run:

ambari-server upgrade-mpack \
--mpack=ambari-mpacks-pinot-1.3.0.3.2.3.7-2011.tar.gz
Verify that the command completes successfully.

Step 5.5: Upgrade Kafka3

Run:

ambari-server upgrade-mpack \
--mpack=ambari-mpacks-kafka3-3.7.2.3.2.3.7-2011.tar.gz
Verify that the command completes successfully.

Step 5.6: Upgrade Impala

Run:

ambari-server upgrade-mpack \
--mpack=ambari-mpacks-impala-4.4.0.3.2.3.7-2011.tar.gz
Verify that the command completes successfully.
For each Mpack upgrade, confirm that the following message is displayed:

Ambari Server 'upgrade-mpack' completed successfully.

Step 6: Restart Ambari Server

After upgrading all required Mpacks, restart Ambari Server:

ambari-server restart
Verify that Ambari Server starts successfully.
Expected Result: Ambari Server restarts successfully with the upgraded Mpacks.