Hive
This guide covers configuring Hive to access S3A storage at the session level, enabling users to work with S3 buckets without requiring cluster-wide configuration changes or service restarts.
Configuration Methods
Method 1: Session-Level Configuration (Recommended)
This method uses secure credential files (JCEKS) stored on HDFS.
Step 1: Create a credential file on HDFS
hadoop credential create fs.s3a.access.key -value <access_key> \
-provider jceks://hdfs@<namenode>:8020/user/hive/s3.jceks
hadoop credential create fs.s3a.secret.key -value <secret_key> \
-provider jceks://hdfs@<namenode>:8020/user/hive/s3.jceks
Step 2: Add to hive-site.xml
hive.security.authorization.sqlstd.confwhitelist.append=fs\\.s3a\\.path\\.style\\.access|fs\\.s3a\\.security\\.credential\\.provider\\.path|fs\\.s3a\\.bucket\\..*\\.security\\.credential\\.provider\\.path
Step 3: Set session properties in Beeline
set fs.s3a.endpoint=s3.<region>.amazonaws.com;
set metaconf:fs.s3a.endpoint=s3.<region>.amazonaws.com;
set fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks;
set metaconf:fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks;
Method 2: Static Configuration (Cluster-Wide)
Add the following properties to HDFS, Hive, Tez, and Spark XML configuration files:
<property>
<name>fs.s3a.access.key</name>
<value>YOUR_ACCESS_KEY</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>YOUR_SECRET_KEY</value>
</property>
<property>
<name>fs.s3a.endpoint</name>
<value>YOUR_S3_ENDPOINT</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<value>true</value>
</property>
Update hive.conf.hidden.list in Custom hiveserver2-site to exclude S3A credentials:
hive.conf.hidden.list=javax.jdo.option.ConnectionPassword,\
hive.server2.keystore.password,\
fs.s3a.proxy.password,\
dfs.adls.oauth2.credential,\
fs.adl.oauth2.credential
Warning
Static configuration exposes credentials in plain text across the cluster and is not recommended for multi-tenant environments.
Known Limitations
- Multiple Buckets Accessing multiple S3 buckets within a single session requires additional HMS changes and is currently not supported.
- Impala Session-level S3 credential configuration is not supported.
- Spark SQL INSERT S3 credentials do not propagate to HMS. Use the Spark DataFrame API instead.
Support Matrix
Supported Versions
- ODP 3.2: 3.2.3.5-2 and later
- ODP 3.3: 3.3.6.3-101 and later
Operation | Hive (Beeline) | Spark (Direct) | Spark + HWC |
Create Table | ✅ | ❌ | ✅ |
Insert (Static/Dynamic) | ✅ | ⚠️* | ✅ |
Show Partitions | ✅ | ✅ | ✅ |
MSCK Repair | ✅ | ❌ | ✅ |
Alter Table | ✅ | ❌ | ✅ |
Drop Table | ✅ | ❌ | ✅ |
Alter / Drop Partition | ✅ | ❌ | ✅ |
Legend
- ✅ Supported
- ❌ Not Supported
- ⚠️* Use DataFrame API; run MSCK REPAIR in Beeline after insert
Spark + HWC Example
import com.acceldata.hwc.HiveWarehouseSession
val hive = HiveWarehouseSession.session(spark).build()
hive.execute("set fs.s3a.endpoint=s3.<region>.amazonaws.com")
hive.execute("set metaconf:fs.s3a.endpoint=s3.<region>.amazonaws.com")
hive.execute("set fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks")
hive.execute("set metaconf:fs.s3a.security.credential.provider.path=jceks://hdfs@<namenode>:8020/user/hive/s3.jceks")
Spark DataFrame Write (Workaround for INSERT Limitation)
df.write.mode("append").insertInto("table_name")