Gluten with Velox

Apache Gluten with Velox backend is a native execution plugin for Apache Spark that accelerates SQL query performance without requiring any code changes to your existing Spark applications.

Key Benefits

• Zero Code Changes – Works with existing Spark SQL, DataFrame, and Dataset APIs
• Significant Performance Gains – 2–5× faster on analytical workloads
• Drop-in Replacement – Simply add JAR and configuration
• Transparent Acceleration – Automatically offloads supported operations to native Velox engine
• Production Ready – Battle-tested on TPC-DS, TPC-H benchmarks

What is Gluten?

Apache Gluten is a Spark plugin that offloads SQL query execution from JVM to native C++ execution engines. It acts as a “glue” layer between Apache Spark and vectorized execution engines like Velox.

Architecture Overview

┌─────────────────────────────────────────────────┐
│ Your Spark Application │
│ (SQL, DataFrame, Dataset APIs) │
│ NO CHANGES NEEDED │
└─────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────┐
│ Apache Spark SQL Engine │
│ (Query Planning & Optimization) │
└─────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────┐
│ Gluten Plugin Layer │
│ (Translates Spark Plans to Native Plans) │
└─────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────┐
│ Velox Execution Engine (C++) │
│ (Vectorized Processing, SIMD, Zero-Copy) │
└─────────────────────────────────────────────────┘

Why Velox?

Velox is Meta's unified execution engine that powers:

• Meta's data warehouse (Presto)
• Stream processing systems
• Machine learning infrastructure

Key Features

• Vectorized execution – processes data in batches
• SIMD instructions – hardware-accelerated operations
• Advanced codegen – runtime code generation
• Memory efficiency – Arrow-based zero-copy design

How It Works

Transparent Query Acceleration

When you enable Gluten, Spark automatically:

1. Analyzes your SQL query plan
2. Identifies operations that can be accelerated (scans, filters, joins, aggregations)
3. Offloads those operations to the Velox native engine
4. Falls back to Spark for unsupported operations
5. Returns results – your application receives the same output, just faster!

Supported Operations

✅ Fully Supported

• Table scans (Parquet, ORC, CSV)
• Filters and predicates
• Projections
• Hash joins (inner, left, right, full outer, semi, anti)
• Hash aggregations (sum, avg, min, max, count)
• Sort operations
• Window functions
• String operations
• Date/time operations
• Math functions

⚠️ Partial Support

• Some complex UDFs
• Certain window function combinations
• Specialized data types

❌ Not Supported

• Custom data sources
• Complex nested UDFs
• Some exotic data types

Performance Expectations

Based on TPC-DS and production workloads:

Real-world Examples

• TPC-DS Query 72: 45s → 12s (3.75× speedup)
• TPC-DS Query 95: 120s → 35s (3.4× speedup)

Getting Started

Prerequisites

Step 1: Obtain Gluten JAR

Contact your Acceldata account manager or download from:

gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar
Place JAR in:

/usr/odp/current/spark3-client/jars/

Step 2: Enable Gluten (No Code Changes!)

Option A: Via spark-submit

spark-submit --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox your-application.jar

Option B: Via spark-defaults.conf

Add to $SPARK_HOME/conf/spark-defaults.conf:

spark.jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar
spark.plugins org.apache.gluten.GlutenPlugin
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 8g
spark.gluten.sql.enabled true
spark.gluten.sql.columnar.backend.lib velox

Option C: Programmatic Configuration (Scala/Java)

val spark = SparkSession.builder()
.appName("MyApp")
.config("spark.plugins", "org.apache.gluten.GlutenPlugin")
.config("spark.memory.offHeap.enabled", "true")
.config("spark.memory.offHeap.size", "8g")
.config("spark.gluten.sql.enabled", "true")
.config("spark.gluten.sql.columnar.backend.lib", "velox")
.getOrCreate()
// Your existing code runs unchanged!
spark.sql("SELECT * FROM large_table WHERE value > 100").show()

Option D: PySpark

from pyspark.sql import SparkSession
spark = SparkSession.builder .appName("MyApp") .config("spark.plugins", "org.apache.gluten.GlutenPlugin") .config("spark.memory.offHeap.enabled", "true") .config("spark.memory.offHeap.size", "8g") .config("spark.gluten.sql.enabled", "true") .config("spark.gluten.sql.columnar.backend.lib", "velox") .getOrCreate()
# Your existing code runs unchanged!
df = spark.sql("SELECT * FROM large_table WHERE value > 100")
df.show()

Step 3: Run Your Application

That's it! Your application runs exactly as before, but faster.

Configuration Guide

Minimal Configuration (Quick Start)

For JDK 11

spark-submit --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" your-application.jar

For JDK 8

spark-submit --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox your-application.jar

Recommended Production Configuration

spark-submit --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 16g --executor-cores 4 --num-executors 10 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar # Core Gluten Settings
--conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox # Memory Configuration
--conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=12g --conf spark.gluten.memory.fraction=0.7 # Shuffle Configuration
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 # Adaptive Query Execution
--conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true # Velox-Specific Settings
--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill # Java Options (JDK 11 only)
--conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" your-application.jar

TPC-DS Benchmark Configuration (Tested)

This is the exact configuration used by Acceldata for TPC-DS benchmarking.

For JDK 11

spark-sql --driver-memory 3g --executor-memory 12g --executor-cores 3 --num-executors 9 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=128MB --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.sql.columnar.backend.velox=true --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 --conf spark.driver.extraJavaOptions="--illegal-access=permit -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.executor.extraJavaOptions="--illegal-access=permit -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED" --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.gluten.sql.columnar.backend.velox.memCacheSize=4g --conf spark.sql.inMemoryColumnarStorage.compressed=true --conf spark.sql.inMemoryColumnarStorage.batchSize=50000 --conf spark.gluten.sql.columnar.forceShuffledHashJoin=false --conf spark.gluten.memory.fraction=0.7 --conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=32768 --conf spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled=true --conf spark.gluten.sql.columnar.preferColumnar=true --conf spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize=1048576 --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB --conf spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold=1GB --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.execution.arrow.pyspark.enabled=true --database tpcds_parquet -f file:///home/hive/hive-testbench/spark-queries-tpcds/query42.sql

For JDK 8

spark-sql --driver-memory 3g --executor-memory 12g --executor-cores 3 --num-executors 9 --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=128MB --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --conf spark.gluten.sql.columnar.backend.lib=velox --conf spark.gluten.sql.columnar.backend.velox=true --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.shuffle.codec=lz4 --conf spark.sql.adaptive.enabled=true --conf spark.sql.adaptive.coalescePartitions.enabled=true --conf spark.sql.adaptive.skewJoin.enabled=true --conf spark.sql.adaptive.localShuffleReader.enabled=true --conf spark.gluten.sql.columnar.backend.velox.memCacheSize=4g --conf spark.sql.inMemoryColumnarStorage.compressed=true --conf spark.sql.inMemoryColumnarStorage.batchSize=50000 --conf spark.gluten.sql.columnar.forceShuffledHashJoin=false --conf spark.gluten.memory.fraction=0.7 --conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=32768 --conf spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled=true --conf spark.gluten.sql.columnar.preferColumnar=true --conf spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize=1048576 --conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB --conf spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold=1GB --conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.execution.arrow.pyspark.enabled=true --database tpcds_parquet -f file:///home/hive/hive-testbench/spark-queries-tpcds/query42.sql

Performance Tuning – Core Gluten Parameters

1. Memory Configuration

• spark.memory.offHeap.enabled – enable off-heap memory
• spark.memory.offHeap.size – off-heap memory size (recommend 8–12g per executor)
• spark.gluten.memory.fraction – fraction of off-heap for Gluten (recommend ~0.7)

Rule of thumb

2. Shuffle Configuration

• spark.shuffle.manager = org.apache.spark.shuffle.sort.ColumnarShuffleManager
• spark.gluten.sql.columnar.shuffle.codec = lz4 (or zstd)

3. Velox Backend Settings

Key parameters to tune:

• spark.gluten.sql.columnar.backend.velox.maxBatchSize = 16384–32768
• spark.gluten.sql.columnar.backend.velox.memCacheSize = 2–4g
• spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize = 1048576–4194304
• spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled = true (join-heavy workloads)

4. Spilling Configuration

--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/mnt/ssd/velox-spill

5. Adaptive Query Execution (AQE)

--conf spark.sql.adaptive.enabled=true
--conf spark.sql.adaptive.coalescePartitions.enabled=true
--conf spark.sql.adaptive.skewJoin.enabled=true
--conf spark.sql.adaptive.localShuffleReader.enabled=true
--conf spark.sql.adaptive.advisoryPartitionSizeInBytes=128MB

Monitoring and Validation

Verify Gluten is Active

Method 1 – Spark UI

• Open Spark UI: http://<driver-host>:4040
• Environment tab → check:
• spark.plugins = org.apache.gluten.GlutenPlugin
• spark.gluten.sql.enabled = true

Method 2 – Query Plan

val df = spark.sql("SELECT * FROM large_table WHERE id > 100")
df.explain(true)
// Look for: GlutenColumnarToRow, GlutenScan, GlutenFilter, GlutenProject

df = spark.sql("SELECT * FROM large_table WHERE id > 100")
df.explain(True)
# Look for Gluten operators in the physical plan
Method 3 – Logs

grep -i "gluten" /var/log/spark/spark-driver.log
# "GlutenPlugin is loaded"
# "Velox backend is initialized"

Performance Metrics

• Query execution time (Spark UI → SQL tab)
• Shuffle read/write size and time
• GC time
• Off-heap memory usage and spilling
• CPU utilization

Example comparison

# Without Gluten
spark-sql --database mydb -e "SELECT COUNT(*) FROM large_table"
# With Gluten
spark-sql --jars gluten-velox-bundle-*.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --database mydb -e "SELECT COUNT(*) FROM large_table"

Benchmark Your Workload

#!/bin/bash
# Baseline (without Gluten)
echo "=== Running WITHOUT Gluten ==="
time spark-sql --database mydb -f my_query.sql > baseline.out
# With Gluten
echo "=== Running WITH Gluten ==="
time spark-sql --jars /usr/odp/current/spark3-client/jars/gluten-velox-bundle-*.jar --conf spark.plugins=org.apache.gluten.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g --conf spark.gluten.sql.enabled=true --database mydb -f my_query.sql > gluten.out
# Compare results
diff baseline.out gluten.out # Should be identical

Best Practices

1. Data Format Recommendations

Best performance

• Parquet (columnar, compressed)
• ORC (columnar, compressed)

Good

• CSV (large files)
• JSON (structured)

Poor

• Small files (<128MB)
• Plain uncompressed text

-- Convert to Parquet for best results
CREATE TABLE optimized_table
USING PARQUET
AS SELECT * FROM original_table;

2. Partition Strategy

• Partition by date/time for time-series
• Aim for 128–256MB file/partition size
• Avoid thousands of tiny files

-- Good partitioning
CREATE TABLE events (
id BIGINT,
event_time TIMESTAMP,
data STRING
)
USING PARQUET
PARTITIONED BY (date DATE)
LOCATION '/data/events';

3. Query Optimization

Do

• Use WHERE predicates for pushdown
• Use partition pruning
• SELECT only needed columns
• Enable bloom filters for joins

Avoid

• SELECT * on wide tables
• Complex nested UDFs
• Excessive small shuffles

4. Resource Allocation

# Small jobs (< 1TB)
--executor-memory 8g
--executor-cores 4
--conf spark.memory.offHeap.size=6g
# Medium jobs (1–10TB)
--executor-memory 16g
--executor-cores 4
--conf spark.memory.offHeap.size=12g
# Large jobs (> 10TB)
--executor-memory 32g
--executor-cores 8
--conf spark.memory.offHeap.size=24g

5. Fallback Strategy

# Disable Gluten if needed
--conf spark.gluten.sql.enabled=false
# Or omit Gluten JAR and plugin configs entirely

Troubleshooting

Issue 1: “Plugin GlutenPlugin could not be loaded”

Symptoms

ERROR PluginContainer: Error initializing plugin org.apache.gluten.GlutenPlugin
Fix

• Verify JAR path:

ls -lh /usr/odp/current/spark3-client/jars/gluten-velox-bundle-*.jar

• Ensure JAR is included in --jars
• Check Spark version compatibility (Gluten 1.4.0 ↔ Spark 3.5.x)

Issue 2: Native Library Load Failure

Symptoms

java.lang.UnsatisfiedLinkError: no velox in java.library.path
Fix

unzip -l gluten-velox-bundle-*.jar | grep "\.so$"
cat /proc/cpuinfo | grep avx2 # check AVX2 support

Issue 3: Out of Memory (OOM)

Fix

• Increase off-heap:

--conf spark.memory.offHeap.size=16g

• Enable spilling:

--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true --conf spark.gluten.sql.columnar.backend.velox.spillPath=/tmp/velox-spill

• Reduce batch size:

--conf spark.gluten.sql.columnar.backend.velox.maxBatchSize=16384

Issue 4: Result Mismatch vs Vanilla Spark

Fix

• Check data types (decimals, timestamps)
• Ensure query is supported by Gluten
• Temporarily disable Gluten:

spark.conf.set("spark.gluten.sql.enabled", "false")
// run query
spark.conf.set("spark.gluten.sql.enabled", "true")

Issue 5: Slower with Gluten

• Very small data (<100MB) → use vanilla Spark
• Excessive ColumnarToRow conversions → inspect plan and simplify query
• Increase spark.memory.offHeap.size
• Tune spark.gluten.sql.columnar.shuffle.codec (try lz4 or zstd)

Debugging Tips

Enable more logging:

--conf spark.gluten.sql.columnar.backend.velox.logLevel=INFO --conf spark.sql.adaptive.logLevel=DEBUG
Check plans:

val df = spark.sql("YOUR_QUERY")
df.explain(mode = "extended")
df.queryExecution.debug.codegen()
Inspect Gluten-related configs:

spark.sparkContext.getConf.getAll
.filter(_._1.contains("gluten"))
.foreach(println)

FAQ

General

Q: Do I need to rewrite my Spark applications? A: No. Gluten works transparently with existing Spark SQL, DataFrame, and Dataset code.

Q: Will query results be the same? A: Yes. If you see differences, treat them as bugs and escalate.

Q: Does Gluten work with PySpark / SparkR / Scala / Java? A: Yes.

Q: Does Gluten support UDFs? A: Built-in functions are supported; custom UDFs usually fall back to Spark.

Q: What about Structured Streaming? A: Works for SQL operations in micro-batch mode.

Performance

• Typical speedup: 2–5× on analytical workloads
• Limited gains on very small queries (<100MB)
• Fully compatible with dynamic allocation

Technical

• File formats: Parquet, ORC, CSV, JSON (best with Parquet / ORC)
• Works with Hive metastore and Hive tables
• Compatible with Delta Lake / Iceberg / Hudi through Spark
• Storage: S3, HDFS, ADLS, GCS, etc.

Disable for a specific query:

spark.conf.set("spark.gluten.sql.enabled", "false")
// run query
spark.conf.set("spark.gluten.sql.enabled", "true")

Compatibility & Deployment

• Spark versions: 3.3.x, 3.4.x, 3.5.x (Gluten 1.4.0)
• Java: 8 and 11
• Cluster managers: YARN, Kubernetes, Standalone
• Arch: x86_64 (ARM64 planned)

Rollback: remove Gluten JAR + plugin configs. Works in notebooks (Jupyter, Zeppelin) via Spark conf.

Quick Reference

Essential Config

# Minimum
--jars gluten-velox-bundle-*.jar
--conf spark.plugins=org.apache.gluten.GlutenPlugin
--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=8g
--conf spark.gluten.sql.enabled=true
--conf spark.gluten.sql.columnar.backend.lib=velox
# Recommended
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
--conf spark.sql.adaptive.enabled=true
--conf spark.gluten.sql.columnar.backend.velox.spillEnabled=true
# JDK 11 only
--conf spark.driver.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED"
--conf spark.executor.extraJavaOptions="--add-opens java.base/java.lang=ALL-UNNAMED"

Quick Checks

# Plugin loaded?
grep "GlutenPlugin" spark-driver.log
# Plan shows Gluten operators?
spark.sql("YOUR_QUERY").explain()
# Look for GlutenScan / GlutenFilter / GlutenProject

Appendix – Key Gluten Parameters

• spark.gluten.sql.enabled – master enable/disable switch
• spark.gluten.sql.columnar.backend.lib – backend engine (velox)
• spark.gluten.memory.fraction – fraction of off-heap for Gluten
• spark.gluten.sql.columnar.preferColumnar – prefer columnar exec
• spark.gluten.sql.columnar.forceShuffledHashJoin – force hash join
• spark.gluten.sql.columnar.shuffle.codec – shuffle codec (lz4)
• spark.gluten.sql.columnar.backend.velox.maxBatchSize – rows per batch
• spark.gluten.sql.columnar.backend.velox.memCacheSize – cache size
• spark.gluten.sql.columnar.backend.velox.spillEnabled – enable spilling
• spark.gluten.sql.columnar.backend.velox.spillPath – spill directory
• spark.gluten.sql.columnar.backend.velox.bloomFilterEnabled – bloom filters
• spark.gluten.sql.columnar.backend.velox.aggregationPreferredSize – aggregation table size

Support and Resources

• Apache Gluten GitHub – project source and issues
• Velox GitHub – execution engine documentation
• Apache Spark Tuning Guide – general Spark performance best practices