Livy 3 Limitations

Independent Livy Instances ⚠️

Impact:

• Each Livy version runs as a separate process
• No session sharing between Livy instances
• Client applications must connect to the correct endpoint

Consideration:

• Plan client routing carefully
• Use load balancers if needed for each version
• Document endpoint URLs for teams

Port Management

Impact:

• Each Livy instance requires a unique port
• Firewall rules must allow access to all ports
• Client applications need port-specific configuration

Recommended Port Allocation:

Spark Version Coupling

Impact:

• Each Livy version should use its corresponding Spark version
• Mixing versions can cause compatibility issues
• The configuration must point to the correct Spark installation

# Ensure SPARK_HOME matches Livy version
# For Livy 3.3.3:
SPARK_HOME=/usr/odp/current/spark3_3_3_3-client

# For Livy 3.5.5:

SPARK_HOME=/usr/odp/current/spark3-client

API Compatibility

Impact:

• Different Livy versions may have API differences
• Client applications may need version-specific code
• Behavioral changes between versions

Mitigation:

• Review Livy release notes for API changes
• Test client applications against the target Livy version
• Maintain version-specific client libraries if needed

Python2 Compatibility in Spark 3 with Livy3

Impact:

• Spark 3’s PySpark code requires Python 3.
• If Livy initializes a Spark 3 session using Python 2, a SyntaxError: invalid syntax occurs.
• Spark session initialization fails because Python 2 cannot parse Python 3–only language features (e.g., type annotations like Union[str, float]).

Error with Python 2

Squirrel

ERROR PythonInterpreter: Process has died with 1

ERROR PythonInterpreter: Traceback (most recent call last):

File "/tmp/9175131608716314470", line 726, in <module>

sys.exit(main())

File "/tmp/9175131608716314470", line 570, in main

exec('from pyspark.sql import HiveContext', global_dict)

File "<string>", line 1, in <module>

File "/usr/odp/current/spark3-client/python/lib/pyspark.zip/pyspark/__init__.py", line 75

def since(version: Union[str, float]) -> Callable[[_F], _F]:

^

SyntaxError: invalid syntax

Mitigation / Workaround:

• Configure Livy to use Python 3 by adding the following to livy3-env

Squirrel

export PYSPARKPYTHON=/usr/bin/python3

export PYSPARK DRIVER_PYTHON=/usr/bin/python3

• Restart the Livy service after applying these changes.
• Ensures both Spark driver and executors run Python 3, allowing Spark 3 sessions to start successfully.