Acceldata
 ODP
Search documentation… ⌘K

Integrate Delta Lake with Hive 4.0.0

This document outlines the steps to integrate Delta Lake with Hive 4.0.0. The integration involves building a custom Delta Lake jar, deploying it to Hive, and using it to create and manage Delta tables.

Prerequisites

  • SBT: Scala Build Tool (SBT) for building the Delta Lake jar.
  • Java 8+: Required for running SBT and Hive.

Steps to Integrate Delta Lake with Hive

  • Clone the Delta Lake Repository

Clone the Delta Lake repository with the specific branch for integration.

git clone https://github.com/acceldata-io/delta.git
cd delta
git checkout ODP-1697
  • Build the Delta Lake Jar

Navigate to the Delta Lake repository root and use SBT to build the jar.

build/sbt hiveAssembly/assembly
  • Locate the Built Jar

After building, locate the jar in the following directory:

delta/connectors/hive-assembly/target/scala-2.12/delta-hive-assembly_2.12-3.2.1-SNAPSHOT.jar
  • Deploy the Jar to Hive

Copy the Jar: Place the delta-hive-assembly_2.12-3.2.1-SNAPSHOT.jar into the Hive lib directory. For example:

cp delta/connectors/hive-assembly/target/scala-2.12/delta-hive-assembly_2.12-3.2.1-SNAPSHOT.jar $HIVE_HOME/lib/
** Restart Hive: Restart the Hive service to apply the changes. **

sudo systemctl restart hive-server2
  • Create a Delta Table in Hive

Open Hive Shell: Start the Hive shell.

hive
Create the Delta Table: Run the following command to create an external Delta table.

CREATE EXTERNAL TABLE deltaTable1(
ts BIGINT,
uuid STRING,
rider STRING,
driver STRING,
fare DOUBLE,
city STRING
)
STORED BY 'io.delta.hive.DeltaStorageHandler'
LOCATION 'hdfs:///warehouse/tablespace/external/hive/my_delta_table';
  • Verify Table Creation

After running the CREATE EXTERNAL TABLE command, you should see a success message indicating that the table has been created.

  • Write Data to the Delta Table

: Writing to a Delta table directly through Hive is not supported. You must use Spark for data insertion.

Attempting to insert data directly via Hive results in an error:

INSERT INTO deltaTable1 VALUES
(1693400000000, 'uuid1', 'rider1', 'driver1', 15.5, 'city1'),
(1693400001000, 'uuid2', 'rider2', 'driver2', 20.0, 'city2');
Error Message:

java.lang.UnsupportedOperationException: Writing to a Delta table in Hive is not supported. Please use Spark to write.
  • Use Spark for Data Insertion

To insert data into the Delta table, use Spark:

val spark = SparkSession.builder().appName("DeltaLakeExample").getOrCreate()
val data = Seq(
(1693400000000L, "uuid1", "rider1", "driver1", 15.5, "city1"),
(1693400001000L, "uuid2", "rider2", "driver2", 20.0, "city2")
)
val df = spark.createDataFrame(data).toDF("ts", "uuid", "rider", "driver", "fare", "city")
df.write.format("delta").mode("append").save("hdfs:///warehouse/tablespace/external/hive/my_delta_table")
As part of the Delta Lake integration (ODP-1697), we do support querying Delta tables via Hive using the internal Delta-Hive connector JAR, specifically built for Hive 4 compatibility in ODP 3.3.6.x.However, please note the following key limitations:
  • Read-only support: Only SELECT and CREATE operations are supported from Hive.
  • No write support: INSERT, UPDATE, DELETE, or any DML operations on Delta tables from Hive are not supported, by design. This is enforced by the DeltaOutputFormat class in the connector, which throws UnsupportedOperationException for any write attempt.
  • Only EXTERNAL tables are supported: The Delta table must be pre-created using Spark with proper transaction logs (_delta_log), and then referenced in Hive using STORED BY 'io.delta.hive.DeltaStorageHandler'.
  • Delta Lake connector from the open-source Delta GitHub repo does not yet support Hive 4, hence we use our internally built version

Download the JAR using the link: https://repo1.acceldata.dev/repository/odp-staging-snapshot/io/delta/delta-hive-assemb[…]APSHOT/delta-hive-assembly_2.12-3.2.1-20250731.110833-1.jar