Using Multiple Spark Versions

Submitting Jobs to Specific Versions

Using spark-submit

Spark 3.5.5:

/usr/odp/current/spark3-client/bin/spark-submit \
--class com.example.MyApp \
--master yarn \
--deploy-mode cluster \
myapp.jar
Spark 3.3.3:

/usr/odp/current/spark3_3_3_3-client/bin/spark-submit \
--class com.example.MyApp \
--master yarn \
--deploy-mode cluster \
myapp.jar
Spark 3.5.1:

/usr/odp/current/spark3_3_5_1-client/bin/spark-submit \
--class com.example.MyApp \
--master yarn \
--deploy-mode cluster \
myapp.jar

Setting Environment Variables

Add to your application launcher script:

For Spark 3.5.5:

export SPARK_HOME=/usr/odp/current/spark3-client export SPARK_CONF_DIR=/etc/spark3/conf
For Spark 3.3.3:

export SPARK_HOME=/usr/odp/current/spark3_3_3_3-client export SPARK_CONF_DIR=/etc/spark3_3_3_3/conf
For Spark 3.5.1:

export SPARK_HOME=/usr/odp/current/spark3_3_5_1-client export SPARK_CONF_DIR=/etc/spark3_3_5_1/conf