Use Hive Warehouse Connector with Spark 4

Hive Warehouse Connector (HWC) enables Spark 4 applications to access and operate on Hive tables, including Hive managed transactional tables.

This page describes how to configure HWC with Spark 4.1.1, connect to HiveServer2, and perform basic Hive operations.

Prerequisites

Before you begin, ensure that your environment meets the following requirements:

HWC JAR

The HWC JAR for Spark 4 is available at:

/usr/odp/3.3.6.4-1/hive_warehouse_connector_spark4/hive-warehouse-connector-spark4-assembly-1.0.0.jar
 

Authenticate with Kerberos

On a Kerberos-enabled cluster, obtain a Kerberos ticket before you start Spark Shell.

Run:

kinit -kt /etc/security/keytabs/spark.headless.keytab \
  spark-rl8iter1@ADSRE.COM
 

Replace the principal with the Spark principal for your environment.

Verify the Kerberos ticket:

klist
 

Verify that the output contains a valid ticket for the Spark principal.

Start Spark Shell with HWC

Set SPARK_MAJOR_VERSION to 4 and start Spark Shell with the required HWC configurations:

SPARK_MAJOR_VERSION=4 spark-shell \
  --conf spark.sql.hive.hiveserver2.jdbc.url.principal=hive/_HOST@ADSRE.COM \
  --conf spark.sql.extensions=com.acceldata.spark.sql.rule.Extensions \
  --conf spark.datasource.hive.warehouse.read.mode=JDBC_CLUSTER \
  --conf spark.kryo.registrator=com.qubole.spark.hiveacid.util.HiveAcidKyroRegistrator \
  --conf spark.sql.hive.hiveserver2.jdbc.url="jdbc:hive2://rl8-27.acceldata.ce:2181,rl8-26.acceldata.ce:2181,rl8-28.acceldata.ce:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" \
  --jars /usr/odp/3.3.6.4-1/hive_warehouse_connector_spark4/hive-warehouse-connector-spark4-assembly-1.0.0.jar
 

Replace the HiveServer2 JDBC URL and Kerberos principal with the values for your environment.

HWC configuration

After Spark Shell starts, verify that it uses Spark 4.1.1, Scala 2.13, and Java 17.

Create a Hive Warehouse session

Import the required HWC classes:

import com.acceldata.hwc.HiveWarehouseSession
import com.acceldata.hwc.HiveWarehouseSession._
 

Create the Hive Warehouse session:

val hive = HiveWarehouseSession.session(spark).build()
 

Use the hive session to perform Hive operations.

List databases

Run:

hive.showDatabases().show()
 

Verify that the expected Hive databases are displayed.

You can also list databases by using executeQuery:

hive.executeQuery("SHOW DATABASES").show()
 

List tables

Run:

hive.showTables().show()
 

To select a database and list its tables, run:

hive.executeQuery("USE default").show()
hive.executeQuery("SHOW TABLES").show()
 

Replace default with the required Hive database.

View a table schema

To view the columns and data types for a table, run:

hive.executeQuery("DESC employeedata").show()
 

Replace employeedata with the required table name.

Create a table

The following example creates an ORC table:

hive.executeQuery(
  "CREATE TABLE IF NOT EXISTS spark4_test " +
  "(id INT, name STRING) STORED AS ORC"
).show()
 

Insert data

Insert data into the table:

hive.executeQuery(
  "INSERT INTO spark4_test VALUES " +
  "(1, 'Alice'), (2, 'Bob'), (3, 'Charlie')"
).show()
 

Read table data

Query the table:

hive.executeQuery(
  "SELECT * FROM spark4_test"
).show()
 

To verify the number of rows, run:

hive.executeQuery(
  "SELECT COUNT(*) AS total FROM spark4_test"
).show()
 

Troubleshoot HWC

Multiple values for the principal

You might receive an error indicating that multiple values are configured for the principal property.

This issue occurs when you specify the HiveServer2 Kerberos principal in both:

spark.sql.hive.hiveserver2.jdbc.url.principal
 

and the JDBC URL:

principal=<HIVE_PRINCIPAL>
 

Specify the principal in only one location.

We recommend using:

spark.sql.hive.hiveserver2.jdbc.url.principal
 

and omitting principal from the JDBC URL.

Unsupported mechanism type PLAIN

The following error can occur when executors attempt to establish a non-Kerberos SASL connection to HiveServer2:

Unsupported mechanism type PLAIN
 

Verify that the HiveServer2 principal is configured:

spark.sql.hive.hiveserver2.jdbc.url.principal
 

Alternatively, specify the principal in the HiveServer2 JDBC URL.

GSS initiate failed

The following error can occur when executors don't have valid Kerberos credentials to establish a GSSAPI connection with HiveServer2:

GSS initiate failed
 

This issue can occur with JDBC_CLUSTER mode because executors establish their own JDBC connections.

To resolve the issue, obtain a valid Kerberos ticket before you start Spark Shell:

kinit -kt /etc/security/keytabs/spark.headless.keytab \
  spark-rl8iter1@ADSRE.COM
 

Alternatively, configure HWC to use DIRECT_READER_V2:

--conf spark.datasource.hive.warehouse.read.mode=DIRECT_READER_V2
 

With DIRECT_READER_V2, executors read data from HDFS by using delegation tokens instead of establishing JDBC connections.