Spark Reference

Spark 4.1.0 Release Notes

https://spark.apache.org/releases/spark-release-4-1-0.html

Real-Time Mode SPIP

https://issues.apache.org/jira/browse/SPARK-52330

Python Data Source API

https://spark.apache.org/docs/latest/api/python/user_guide/sql/python_data_source.html

Python UDTF

https://spark.apache.org/docs/latest/api/python/user_guide/sql/python_udtf.html

Apache Arrow in PySpark

https://spark.apache.org/docs/latest/api/python/user_guide/sql/arrow_pandas.html

Error classes in PySpark

https://spark.apache.org/docs/latest/api/python/development/errors.html

Spark 4.x Migration Guide

https://spark.apache.org/docs/latest/core-migration-guide.html

SPARK-47240 Structured Logging

https://issues.apache.org/jira/browse/SPARK-47240

Scalar UDFs

https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html

Delta Lake 4.0

https://delta.io/blog/delta-lake-4-0-preview/

XML Connector

https://github.com/databricks/spark-xml

KLL Sketch (SPARK-54199)

https://issues.apache.org/jira/browse/SPARK-54199

Theta Sketch (SPARK-52407)

https://issues.apache.org/jira/browse/SPARK-52407

Python Worker Logging (SPARK-53754)

https://issues.apache.org/jira/browse/SPARK-53754