Troubleshooting Livy 3

Issue: Livy Server Not Visible in Ambari UI

Symptoms: Livy component doesn't appear when adding Spark service

Solution:

# Verify mpack installation
ambari-server list-mpacks
# Expected output should include spark3 mpacks
# spark3-ambari-3.5.5.mpack
# spark3-ambari-3.3.3.mpack
# Clear Ambari cache
ambari-server stop
rm -rf /var/lib/ambari-server/cache/*
ambari-server start

Issue: Port Conflict - "Address already in use"

Symptoms: Livy server fails to start with port binding error

Error Message:

Exception in thread "main" java.io.IOException: Failed to bind to /0.0.0.0:8333
Caused by: java.net.BindException: Address already in use
Solution:

1. Check what's using the port:

sudo netstat -tulpn | grep 8333
# or
sudo lsof -i :8333

1. Update Livy configuration:

# Edit configuration via Ambari UI:
# Services → SPARK3_3_3_3 → Configs → Advanced livy3-3.3.3-conf
# Change: livy.server.port = 8334 (or another available port)
# Or edit directly:
vi /etc/livy3_3_3_3/conf/livy.conf
# Update: livy.server.port = 8334

1. Restart Livy Server:

# Via Ambari UI or:
ambari-agent restart

Issue: Wrong Spark Version Used

Symptoms: Livy sessions use incorrect Spark version

Solution:

1. Verify SPARK_HOME in livy-env.sh:

# For Livy 3.3.3, should point to Spark 3.3.3
cat /etc/livy3_3_3_3/conf/livy-env.sh | grep SPARK_HOME
# Expected: export SPARK_HOME=/usr/odp/current/spark3_3_3_3-client

1. Update via Ambari:
   • Services → SPARK3_3_3_3 → Configs → Advanced livy3-3.3.3-env
   • Set correct SPARK_HOME path
   • Save and restart Livy
2. Verify Spark version in session:

# Create test session
SESSION_ID=$(curl -X POST http://localhost:8356/sessions \
-H "Content-Type: application/json" \
-d '{"kind":"spark"}' | jq -r '.id')
# Check Spark version
curl http://localhost:8356/sessions/$SESSION_ID | jq '.appInfo.sparkUiUrl'

Issue: Session Creation Fails

Symptoms: REST API returns error when creating sessions

Common Errors and Solutions:

# Check if Livy server is running
curl http://<host>:8333/version

# Check Livy logs

tail -f /var/log/livy333/livy-livy-server.out

{
"conf": {
"spark.yarn.queue": "existing-queue-name"
}
}

# Check YARN capacity
yarn application -status

# Reduce resource requirements

curl -X POST http://<host>:8333/sessions \

-H "Content-Type: application/json" \

-d '{

"kind": "spark",

"conf": {

"spark.executor.memory": "1g",

"spark.executor.cores": 1,

"spark.executor.instances": 1

}

}'

Issue: Sessions Timeout Unexpectedly

Symptoms: Livy sessions terminate before completion

Solution:

Increase session timeout:

# /etc/livy3_3_3_3/conf/livy.conf
livy.server.session.timeout = 2h
livy.server.session.timeout-check = true
livy.server.session.timeout-check.interval = 60000 # 60 seconds
Or disable timeout for long-running jobs:

livy.server.session.timeout = 0 # Disable timeout

Issue: Livy UI Shows Wrong Version

Symptoms: Livy REST API reports incorrect version

Solution:

1. Verify correct binaries are being used:

ps aux | grep livy | grep java
# Should show correct path like /usr/odp/current/livy3_3_3_3-server

1. Check symlinks:

ls -la /usr/odp/current/ | grep livy3

1. Restart Livy with the correct LIVY_HOME:

export LIVY_HOME=/usr/odp/current/livy3_3_3_3-server
$LIVY_HOME/bin/livy-server restart

Session Pooling