Acceldata
 ODP
Search documentation… ⌘K

Troubleshoot Spark 3 Dynamic Allocation and Shuffle Service Issues After ODP Upgrade

After upgrading from Spark2 to Spark3 “on ODP”, Spark applications may fail when dynamic allocation and external shuffle service are enabled.

Symptoms can include:

  • Spark jobs hanging
  • Executor allocation failures
  • Shuffle-related exceptions
  • PySpark startup failures
  • Spark service check failures

Symptoms

Applications run successfully without dynamic allocation:

spark.dynamicAllocation.enabled=false
but fail when the following settings are enabled:

spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.minExecutors
spark.dynamicAllocation.maxExecutors
spark.shuffle.service.enabled=true

Cause

During Spark2 → Spark3 migration, shuffle service configuration may not fully align with the Spark3 deployment.

Common causes include:

Incorrect Shuffle Classpath

yarn.nodemanager.aux-services.spark3_shuffle.classpath
does not point to the Spark3 shuffle libraries.

Incorrect Shuffle Service Port

spark.shuffle.service.port
is configured with a non-functional port.

Legacy Spark2 Components

Spark2 symlinks remain present and interfere with Spark3 client execution.

Resolution

Update Shuffle Port

Configure:

spark.shuffle.service.port=7337
Configure the location for jar files for the external shuffle service yarn.nodemanager.aux-services.spar.

yarn.nodemanager.aux-services.spark3_shuffle.classpath=/usr/odp/current/spark3-client/aux/*
Restart:

Spark Services
YARN Services

Validate Dynamic Allocation

Re-enable:

spark.dynamicAllocation.enabled=true
spark.shuffle.service.enabled=true
Submit a Spark application and verify that executor allocation functions normally.

PySpark Failure After Spark3 Migration

Symptoms

Multiple versions of Spark are installed
but SPARK_MAJOR_VERSION is not set
followed by:

TypeError: code() argument 13 must be str, not int
Cause
PySpark is launching the Spark2 runtime instead of Spark3.
Resolution
  • Verify Spark2 is no longer required.
  • Remove obsolete Spark2 references:
spark2-client
spark2-historyserver
spark2-thriftserver
Restart Spark services.
Validate:

spark-shell
and

pyspark
start successfully.

Validation

Run:

spark-shell
Run:

pyspark
Submit:

spark-submit
with dynamic allocation enabled.
Confirm:
  • Executors are allocated successfully.
  • Shuffle operations complete successfully.
  • No Livy or YARN errors are reported.

Best Practices

  • Remove obsolete Spark2 components after migration.
  • Verify shuffle service configuration before enabling dynamic allocation.
  • Validate Spark shell, PySpark, and Spark submit workflows after upgrade.
  • Test production jobs before enabling dynamic allocation in production.

Summary

Spark3 upgrades may expose issues related to external shuffle services, dynamic allocation, and legacy Spark2 references.

Correcting the Spark3 shuffle configuration, validating the shuffle service port, and removing obsolete Spark2 components typically resolve these issues and restore normal Spark operation.