Acceldata
 ODP
Search documentation… ⌘K

Troubleshooting Spark 3

Issue: Service not visible in Ambari UI

Symptoms: Spark version doesn't appear in "Add Service" wizard

Solution:

# Verify mpack installation
ambari-server list-mpacks
# Expected output should include:
# spark3-ambari-3.5.5.mpack
# Clear Ambari cache
ambari-server stop
rm -rf /var/lib/ambari-server/cache/*
ambari-server start

Issue: History Server shows the wrong Spark version

Symptoms: Port 18083 shows Spark 3.5.5 instead of 3.3.3

Solution:

  1. Verify configuration files exist:
ls -la /etc/spark3_3_3_3/conf/
  1. Check port configuration:
grep "spark.history.ui.port" /etc/spark3_3_3_3/conf/spark-defaults.conf
  1. Restart History Server from Ambari UI

Issue: spark-submit fails to find the correct version

Symptoms: Application uses the wrong Spark version

Solution:

Set environment variables explicitly:

export SPARK_HOME=/usr/odp/current/spark3_3_3_3-client export SPARK_CONF_DIR=/etc/spark3_3_3_3/conf /usr/odp/current/spark3_3_3_3-client/bin/spark-submit ...

Issue: Port conflict

Symptoms: History Server fails to start

Solution:

  1. Check port availability:
netstat -tuln | grep 1808
  1. Verify port configuration in Ambari
  2. Ensure no other services use ports 18082-18084