Install Multiple Livy 3

Prerequisites

Before installing multiple Livy3 versions, ensure:

• ✅ Ambari Server 2.7.8.6-2/3
• ✅ ODP 3.0+ stack installed
• ✅ Required dependencies: HDFS, YARN, HIVE
• ✅ Corresponding Spark3 versions installed
  • Livy 3.5.5 requires Spark 3.5.5
  • Livy 3.3.3 requires Spark 3.3.3
  • Livy 3.5.1 requires Spark 3.5.1
• ✅ Network ports available:
  • Port 8355 for Livy 3.5.5 (recommended)
  • Port 8351 for Livy 3.5.1
  • Port 8333 for Livy 3.3.3
• ✅ Sufficient cluster resources for multiple Livy servers

Step 1: Obtain the Management Pack

The Livy management packs are bundled with the corresponding Spark3 mpacks:

# Livy is included in Spark3 mpacks
ambari-mpacks-odp-5566-spark3-3.5.5.3.2.3.7-2.tar.gz
ambari-mpacks-odp-5551-spark3-3.5.1.3.2.3.7-2.tar.gz
ambari-mpacks-odp-5333-spark3-3.3.3.3.2.3.7-2.tar.gz

Step 2: Install the Management Packs

On your Ambari Server, run:

# Install Spark 3.5.5 mpack (includes Livy 3.5.5)
ambari-server install-mpack \
--mpack=./ambari-mpacks-spark3-3.5.5.3.2.3.7-2.tar.gz \
--verbose
# Install Spark 3.3.3 mpack (includes Livy 3.3.3)
ambari-server install-mpack \
--mpack=./ambari-mpacks-spark3-3.3.3.3.2.3.7-2.tar.gz \
--verbose
# Install Spark 3.5.1 mpack (includes Livy 3.5.1)
ambari-server install-mpack \
--mpack=./ambari-mpacks-spark3-3.5.1.3.2.3.7-2.tar.gz \
--verbose
# Restart Ambari Server
ambari-server restart
Expected Output:

INFO: Installing management pack
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Management pack installed successfully

Step 3: Add Spark3 Services with Livy via Ambari UI

Adding Livy3 3.5.5 (Recommended)

1. Log in to Ambari Web UI
   • Navigate to http://<ambari-server>:8080
   • Log in with admin credentials
2. Add Service
   • Click "Actions" → "Add Service"
   • Select "Spark3 3.5.5" from the service list
   • Click "Next"
3. Assign Components
   • History Server: Select host(s) for Spark History Server
   • Livy Server: ✅ Enable and select host(s) for Livy3 Server
   • Thrift Server: (Optional) Select host(s) for Thrift Server
   • Client: Select hosts that need Spark client libraries
   • Click "Next"
4. Configure Service Review default configurations. Key Livy settings:
   • Livy Server Port: 8355 (default for 3.5.5)
   • Log Directory: /var/log/livy3
   • PID Directory: /var/run/livy3
   • Spark Home: /usr/odp/current/spark3-client
   • Click "Next"
5. Deploy Service
   • Review deployment plan
   • Click "Deploy"
   • Monitor installation progress
   • Click "Complete" when finished

Adding Livy3 3.3.3 or 3.5.1

Repeat the above steps, selecting:

• Spark 3.3.3 (includes Livy 3.3.3)
• Spark 3.5.1 (includes Livy 3.5.1)

Step 4: Configure Port Allocation

To avoid port conflicts, configure unique ports for each Livy instance:

Configure via Ambari UI:

1. Navigate to Services → SPARK3_3_3_3 (or respective version)
2. Click "Configs" tab
3. Find "Advanced livy3-3.3.3-conf" section
4. Set livy.server.port = 8333
5. Save changes
6. Restart Livy Server

Step 5: Verify Installation

Check Services in Ambari UI

Navigate to the Ambari dashboard and verify:

• ✅ All installed Livy3 services show "Started" status
• ✅ All components are healthy (green indicators)
• ✅ No port conflicts reported

Verify Livy REST API Endpoints

Access each Livy server's REST API:

# Livy 3.5.5
curl http://<host>:8355/version
curl http://<host>:8355/sessions
# Livy 3.3.3
curl http://<host>:8333/version
curl http://<host>:8333/sessions
# Livy 3.5.1
curl http://<host>:8351/version
curl http://<host>:8351/sessions
Each endpoint should return the correct Livy version information:

{
"version": "0.8.0-incubating-SNAPSHOT",
"user": "livy",
"spark": "3.5.5"
}

Verify Client Installation

On client nodes:

# Check installed packages
rpm -qa | grep livy3
# Verify configuration directories
ls -la /etc/livy3/conf
ls -la /etc/livy3_3_3_3/conf
ls -la /etc/livy3_3_5_1/conf
# Check Livy binaries
ls -la /usr/odp/current/livy3-server
ls -la /usr/odp/current/livy3_3_3_3-server
ls -la /usr/odp/current/livy3_3_5_1-server