Connect to a Spark Connect Server from a Laptop

You can connect to a Spark 4.1 Connect server running on an ODP cluster from your laptop without installing a JVM. For a Python client, you need only Python and pip.

For information about Spark Connect, see Spark Connect Overview 4.1.1.

Prerequisites

Before you connect to the Spark Connect server, verify the following requirements:

• Python 3.9 or later. This configuration is validated with Python 3.11 and 3.12.
• Network connectivity from your laptop to the Spark Connect server on port 15005.
• The hostname or IP address of the Spark Connect server.

For example, the server used in this procedure is:

10.101.11.114:15005

If the Spark Connect server is bound only to the loopback address (127.0.0.1), configure an SSH tunnel before connecting. For more information, see Troubleshooting.

Install the Python Client

The full PySpark package is approximately 1.3 GB because it includes the JVM JAR files required to run Spark locally.

For a Spark Connect-only client, you don't need the full PySpark distribution. Install the lightweight client packages instead.

1. Create a Python virtual environment:

python3 -m venv ~/spark4-connect

1. Activate the virtual environment:

source ~/spark4-connect/bin/activate

1. Upgrade pip:

pip install --upgrade pip

1. Install the PySpark client package:

pip install https://mirror.odp.acceldata.dev/v2/standalone_binaries/3.3.6.5-1012/pyspark_client-4.1.1.3.3.6.5-1012.tar.gz

1. Install the Spark Connect package:

pip install https://mirror.odp.acceldata.dev/v2/standalone_binaries/3.3.6.5-1012/pyspark_connect-4.1.1.3.3.6.5-1012.tar.gz

1. Verify the installed PySpark version:

python -c "import pyspark; print(pyspark.__version__)"

1. Expected output:

4.1.1

The complete client installation is approximately 30 MB, including NumPy, pandas, PyArrow, gRPC, and the Spark client packages.

Connect Using Python

Use SparkSession.builder.remote() to connect to the Spark Connect server.

For example:

from pyspark.sql import SparkSession
 
spark = SparkSession.builder.remote(
    "sc://10.101.11.114:15005"
).getOrCreate()
 
print("count:", spark.range(1000).count())
 
df = spark.range(1_000_000).selectExpr(
    "id",
    "id % 100 as bucket"
)
 
df.groupBy("bucket").count().orderBy("bucket").limit(5).show()
 
spark.stop()

Save the code as demo.py.

Activate the virtual environment:

source ~/spark4-connect/bin/activate

Run the application:

python demo.py

Connect Using IntelliJ IDEA and Scala

To connect from a Scala application, create a Maven or Gradle project in IntelliJ IDEA.

For a Maven project, add the following dependency:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-connect-client-jvm_2.13</artifactId>
    <version>4.1.1.3.3.6.5-1012</version>
</dependency>

Use the following example to create a Spark Connect session:

import org.apache.spark.sql.SparkSession
 
object Demo extends App {
  val spark = SparkSession.builder()
    .remote("sc://10.101.11.114:15005")
    .getOrCreate()
 
  println("count: " + spark.range(1000).count())
 
  spark.stop()
}

Configure the IntelliJ IDEA run configuration to use JVM 17.

No local Spark installation is required. The client communicates with the Spark Connect server by using gRPC over TCP.

Troubleshoot Connection Issues

Authentication Token Is Not Provided

You might receive the following error:

UNAUTHENTICATED: No authentication token provided

This error occurs when spark.connect.authenticate.token is configured with a nonempty value on the Spark Connect server.

To resolve the issue, use one of the following options:

• Ask the cluster administrator to remove the spark.connect.authenticate.token value in Ambari > Services > Spark4 > Configs > Advanced spark4-connect, and then restart the Spark Connect server.
• Obtain the configured token from the cluster administrator and include it in the connection URL.

For example:

spark = SparkSession.builder.remote(
    "sc://10.101.11.114:15005/;token=<token>"
).getOrCreate()

TLS Handshake Fails

You might receive an error similar to the following:

Handshake failed with error SSL_ERROR_SSL ... WRONG_VERSION_NUMBER

This error can occur when you specify ;token= in the connection URL. PySpark automatically enables TLS when a token is specified, but the server might be using plaintext gRPC.

To resolve the issue, either:

• Remove the token from the connection URL if authentication isn't required.
• Configure TLS on the Spark Connect server if token-based authentication is required.

Connection Is Refused

If the client can't establish a TCP connection, the Spark Connect server might be bound to 127.0.0.1 instead of 0.0.0.0.

Ask the cluster administrator to set the following property in Ambari and restart the Spark Connect server:

spark.connect.grpc.binding.address=0.0.0.0

Alternatively, create an SSH tunnel:

ssh -i ~/Cursor/git_pr/spark-test/us_dc_vast2.pem -N \
    -L 15005:127.0.0.1:15005 acceldata@10.101.11.114

Then, connect to the local endpoint:

spark = SparkSession.builder.remote(
    "sc://localhost:15005"
).getOrCreate()

Access to Kerberos-Protected HDFS Data Fails

Spark Connect doesn't propagate Kerberos credentials from the client system to the Spark Connect server.

File operations run by using the identity of the user or principal that runs the Spark Connect server, typically the spark user.

To access HDFS data owned by another user, use one of the following approaches:

• Ask the cluster administrator to run a dedicated Spark Connect server for the required user.
• Configure the required HDFS or Ranger permissions for the Spark Connect server user.
• Stage the required data by using a standard spark-submit application before accessing it through Spark Connect.

Choose a PySpark Package

Use the following table to determine which package to install:

Known Limitations

• Scala Ammonite shell: spark-connect-shell requires an interactive TTY. Spark 4.1 doesn't support running the shell in scripted mode through a heredoc.