Run Spark with Velox

You can use Velox with Spark 3.5.5 and Spark 4.1.1 to accelerate Spark SQL workloads. Gluten provides the Spark plugin that offloads supported SQL operations to the Velox native execution engine.

To enable Gluten and Velox, you must configure:

spark.plugins=org.apache.gluten.GlutenPlugin

Without this configuration, Velox isn't enabled.

Prerequisites

Before you submit an application:

1. Set SPARK_HOME to the client for the Spark version that you want to use.
   For Spark 3.5.5:
   

   export SPARK_HOME=/usr/odp/current/spark3-client

   
   For Spark 4.1.1:
   

   export SPARK_HOME=/usr/odp/current/spark4-client

2. Use Parquet or ORC data for better performance.
3. Enable off-heap memory for Velox:
   

   spark.memory.offHeap.enabled=true

4. Configure sufficient off-heap memory for the YARN container.
   The examples on this page use 2g. For larger workloads, increase the value as required.

Run Spark 3.5.5 with Velox

Gluten JAR

On ODP, the Gluten bundle is typically available in the Spark 3 jars directory:

/usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar

Submit a Spark application

Set SPARK_HOME:

export SPARK_HOME=/usr/odp/current/spark3-client

On JDK 11 environments, configure the required Java options:

JAVAOPTS='-Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.misc=ALL-UNNAMED'

Submit the application:

$SPARK_HOME/bin/spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --name my-app-spark355-velox \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=2g \
  --conf spark.gluten.enabled=true \
  --conf spark.gluten.sql.columnar.backend.lib=velox \
  --conf spark.sql.autoBroadcastJoinThreshold=-1 \
  --conf spark.driver.extraJavaOptions="$JAVAOPTS" \
  --conf spark.executor.extraJavaOptions="$JAVAOPTS" \
  --class com.example.MyApp \
  /path/to/my-app.jar

Run Spark SQL

export SPARK_HOME=/usr/odp/current/spark3-client
 
JAVAOPTS='-Dio.netty.tryReflectionSetAccessible=true --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.misc=ALL-UNNAMED'
 
$SPARK_HOME/bin/spark-sql --master yarn \
  --name my-sql-spark355-velox \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=2g \
  --conf spark.gluten.enabled=true \
  --conf spark.gluten.sql.columnar.backend.lib=velox \
  --conf spark.sql.autoBroadcastJoinThreshold=-1 \
  --conf spark.driver.extraJavaOptions="$JAVAOPTS" \
  --conf spark.executor.extraJavaOptions="$JAVAOPTS" \
  -e "SELECT 1"

Disable broadcast joins

On some JDK 11 environments, Gluten columnar broadcast operations can fail with the following error:

DirectByteBuffer.<init>(long, int) not available

To avoid this issue, the preceding examples disable broadcast joins:

spark.sql.autoBroadcastJoinThreshold=-1

You can enable broadcast joins if your JDK and Gluten build support them.

Run Spark 4.1.1 with Velox

Spark 4.1.1 has the following requirements that differ from Spark 3.5.5:

• Spark 4.1.1 requires JDK 17 for the YARN ApplicationMaster and executors.
• The Gluten JAR might not be in the Spark 4 jars directory. In this case, specify it by using --jars.

Configure JDK 17

Set the Spark 4 client:

export SPARK_HOME=/usr/odp/current/spark4-client

Set the JDK 17 installation directory:

export JDK17=/usr/lib/jvm/java-17-openjdk

If multiple JDK 17 installations are available, specify the complete installation path.

For example:

export JDK17=/usr/lib/jvm/java-17-openjdk-17.0.20.1.1-1.1.el8_10.x86_64

Locate the Gluten JAR

The Gluten JAR is typically available at a path similar to:

/usr/odp/3.3.6.5-<build>/spark4/gluten/gluten-velox-bundle-spark4.1_2.13-linux_amd64-1.7.0.3.3.6.5.jar

The build number can vary depending on your ODP installation.

You can locate the JAR by running:

export GLUTEN=$(ls /usr/odp/*/spark4/gluten/gluten-velox-bundle-spark4.1*.jar 2>/dev/null | head -1)
echo "Using GLUTEN=$GLUTEN"

Submit a Spark application

$SPARK_HOME/bin/spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --name my-app-spark411-velox \
  --jars "$GLUTEN" \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=2g \
  --conf spark.gluten.enabled=true \
  --conf spark.gluten.sql.columnar.backend.lib=velox \
  --conf spark.yarn.appMasterEnv.JAVA_HOME="$JDK17" \
  --conf spark.executorEnv.JAVA_HOME="$JDK17" \
  --class com.example.MyApp \
  /path/to/my-app.jar

Run Spark SQL

export SPARK_HOME=/usr/odp/current/spark4-client
export JDK17=/usr/lib/jvm/java-17-openjdk
 
export GLUTEN=$(ls /usr/odp/*/spark4/gluten/gluten-velox-bundle-spark4.1*.jar 2>/dev/null | head -1)
 
$SPARK_HOME/bin/spark-sql --master yarn \
  --name my-sql-spark411-velox \
  --jars "$GLUTEN" \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=2g \
  --conf spark.gluten.enabled=true \
  --conf spark.gluten.sql.columnar.backend.lib=velox \
  --conf spark.yarn.appMasterEnv.JAVA_HOME="$JDK17" \
  --conf spark.executorEnv.JAVA_HOME="$JDK17" \
  -e "SELECT 1"

Velox configuration

The following configurations apply to both Spark versions:

You can also configure the following optional properties:

spark.sql.adaptive.enabled=true
spark.gluten.sql.columnar.backend.velox.spillEnabled=true
spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill

Verify that Velox is enabled

After you submit an application, verify that Gluten and Velox loaded successfully.

1. Open the Spark UI.
2. Go to Environment.
3. Verify that the following configuration is present:
   

   spark.plugins=org.apache.gluten.GlutenPlugin

4. Run df.explain(true) or EXPLAIN EXTENDED.
5. Check the execution plan for Gluten operators, such as:
   

   GlutenScan
   GlutenFilter
   GlutenProject

6. Check the driver logs for messages that indicate that Gluten and Velox loaded successfully.

If the execution plan contains only standard Spark operators and doesn't contain Gluten operators, verify the Gluten configuration and JAR location.

Troubleshoot Velox

Disable Velox

To run an application without Velox, remove the spark.plugins and Gluten-specific configurations.

Alternatively, disable Gluten for the application:

--conf spark.gluten.enabled=false

The application then runs using the standard Spark execution engine.