Usability

Structured Logging Framework

Spark 4.1.1 outputs logs in structured JSON format for easy ingestion by log aggregation tools (ELK, Splunk, Datadog).

spark.log.structuredLogging.enabled=true

spark = SparkSession.builder \
.config("spark.log.structuredLogging.enabled", "true") \
.appName("Structured Logging") \
.getOrCreate()

Error Class Framework

Standardized, actionable error messages that explain what failed, why, and how to fix it.

Before (Spark 3.x):

AnalysisException: cannot resolve 'user_id' given input columns: [id, name, value]
After (Spark 4.1.1):

[UNRESOLVED_COLUMN.WITH_SUGGESTION] Column 'user_id' does not exist.
Did you mean one of the following? [id, name, value]
Example error message for boolean expression misuse:

[INVALID_BOOL_EXPRESSION] Cannot convert column into bool: please use '&' for 'and',
'|' for 'or', '~' for 'not' when building DataFrame boolean expressions.

Behavior Change Process

Spark 4.1.1 follows a formal Behavior Change Process, documenting all breaking changes with opt-out flags where applicable.

Key behavior changes from Spark 3.x → 4.1.1: