Configuring ODP with GCS

To configure access to your GCS buckets from your cluster, start by adding the GCS connector jar. Ensure to include the gcs-connector-hadoop3-3.1.17-shaded.jar, as it not only contains the classes and resources for the GCS Connector for Hadoop but also includes its dependencies.

Once you have downloaded the jar mentioned above to your Hadoop cluster, you can go ahead with the following configuration steps to include the gcs-connector jar into the classpath for the relevant files.

Configuration Steps for Respective Classpaths

Hadoop-env.sh

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/odp/3.3.6.3-1/hadoop/gcs-connector-hadoop3-3.1.17-shaded.jar

Hive-env.sh

export HIVE_AUX_JARS_PATH=/usr/odp/current/hive-client/gcs-connector-hadoop3-3.1.17-shaded.jar

Mapreduce.application.classpath

/usr/odp/3.3.6.3-1/hadoop/gcs-connector-hadoop3-3.1.17-shaded.jar

Core-Site.xml

<property>
<name>fs.gs.impl</name>
<value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem</value>
<description>The FileSystem for gs: (GCS) uris.</description>
</property>
<property>
<name>fs.AbstractFileSystem.gs.impl</name>
<value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
</property>

Configure Access using Core-Site.xml

Core-Site.xml

<property>
<name>google.cloud.auth.type</name>
<value>SERVICE_ACCOUNT_JSON_KEYFILE</value>
</property>
<property>
<name>google.cloud.auth.service.account.json.keyfile</name>
<value>PATH TO JSON KEYFILE</value>
</property>

Configure Access using Hadoop Credential JCEKS

hdfs dfs -mkdir /app/gcpclient
hadoop credential create fs.gs.auth.service.account.email -provider jceks://user/dataget.jceks -value "SERVICE ACCOUNT EMAIL"
hadoop credential create fs.gs.auth.service.account.private.key.id -provider jceks://hdfs/app/gcpclient/dataget.jceks -value "PRIVATE KEY ID VALUE"
hadoop credential create fs.gs.auth.service.account.private.key -provider jceks://hdfs/app/dataget.jceks -value "ACCOUNT PRIVATE KEY"​
hadoop fs -chown -R hdfs:hdfs /app/gcpclient
hadoop fs -chown 500 /app/gcpclient
hadoop fs -chown 400 /app/gcpclient/dataget.jceks
After successfully crafting the JCEKS file, you can validate its functionality by attempting to access your GCS buckets, as shown in the code below.

hadoop fs -Dhadoop.security.credential.provider.path=jceks://hdfs/app/gcpclient/dataget.jceks -ls gs://trialodp/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/odp/3.3.6.3-1/hadoop/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/odp/3.3.6.3-1/hadoop-hdfs/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/odp/3.3.6.3-1/tez/lib/slf4j-reload4j-1.7.35.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
Jan 10, 2024 5:42:35 PM com.google.cloud.hadoop.fs.gcs.GhfsStorageStatistics updateStats
WARNING: Detected potential high latency for operation op_get_file_status. latencyMs=1553; previousMaxLatencyMs=0; operationCount=1; context=gs://trialodp/
Jan 10, 2024 5:42:35 PM com.google.cloud.hadoop.fs.gcs.GhfsStorageStatistics updateStats
WARNING: Detected potential high latency for operation op_glob_status. latencyMs=1633; previousMaxLatencyMs=0; operationCount=1; context=path=gs://trialodp/; pattern=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase$$Lambda$8/1628998132@39ce27f2
Found 1 items
drwx------ - hdfs hdfs 0 2024-01-10 16:29 gs://trialodp/testfolder

Access to Google Cloud Storage via ADC

Validate connectivity between ODP 3.3.6.x and Google Cloud Storage, and confirm whether the bundled Google GCS connector supports Application Default Credentials.

Scope

Validated

• ODP 3.3.6.x connectivity to GCS
• Hadoop filesystem access using gs://
• GCS connector availability and filesystem implementation
• ADC authentication using GOOGLE_APPLICATION_CREDENTIALS
• Configuration using:

google.cloud.auth.type=APPLICATION_DEFAULT

Not Validated

• Workload Identity Federation
• fs.gs.auth.type=APPLICATION_DEFAULT
• fs.gs.auth.type=ACCESS_TOKEN_PROVIDER
• Spark BigQuery Connector
• Hive BigQuery Storage Handler
• BigQuery read/write operations

Environment

Connector Discovery

The GCS connector was found under the Spark client jars:

find /usr/odp/ -name "*gcs*jar"
Example Output:

/usr/odp/3.3.6.2-1/spark3/jars/gcs-connector-hadoop3-3.1.17-shaded.jar
Because the connector was not available in the Hadoop runtime classpath by default, it was manually copied into the Hadoop client library path:

cp /usr/odp/3.3.6.2-1/spark3/jars/gcs-connector-hadoop3-3.1.17-shaded.jar \ /usr/odp/3.3.6.3-1/hadoop/client/lib/

GCP Setup

The following GCP resources were created:

• GCS test bucket
• GCP service account
• Service account JSON credential file
• Required IAM permissions on the bucket

The credential file was placed on the ODP host:

mkdir -p /etc/gcp
cp odp-gcs-sa.json /etc/gcp/
chmod 400 /etc/gcp/odp-gcs-sa.json

Network Validation

Connectivity from ODP to GCS was verified:

curl -I https://storage.googleapis.com
Output:

HTTP/2 400
This confirms:

• DNS resolution is successful
• HTTPS connectivity is available
• No firewall restrictions are blocking access to GCS endpoints

Hadoop Configuration

core-site.xml

1. In Ambari UI, go to HDFS → Configs → Custom core-site.xml.
2. In Custom core-site.xml, add the following properties:

fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS google.cloud.auth.type=APPLICATION_DEFAULT
hadoop-env.sh
Configure the following variable:

export GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/odp-gcs-sa.json

Configure Workload Identity Federation

Use Google Workload Identity Federation (WIF) to authenticate ODP with Google Cloud Storage (GCS) without using long-lived service account keys.

Before You Begin

Ensure that you have:

• A configured Google Cloud Workload Identity Pool and Provider.
• A WIF credential configuration file.
• Access to the GCS bucket that you want ODP to access.

1. Download the GCS Connector

Download the latest supported GCS connector:

https://repo1.maven.org/maven2/com/google/cloud/bigdataoss/gcs-connector/3.1.17/gcs-connector-3.1.17-shaded.jar

1. Replace the GCS Connector

Copy the connector JAR to every node that accesses Google Cloud Storage, such as:

• Edge nodes
• Hadoop client nodes
• Spark gateway or driver nodes
• HiveServer2 nodes

For example:

cp gcs-connector-3.1.17-shaded.jar <connector-directory>
If you are replacing an existing connector, use the same connector version on all applicable nodes.

1. Verify the Hadoop Configuration

Verify that the following properties are configured in the core-site.xml file.

<property>
<name>fs.gs.impl</name>
<value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem</value>
</property>
<property>
<name>fs.AbstractFileSystem.gs.impl</name>
<value>com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS</value>
</property>
<property>
<name>google.cloud.auth.type</name>
<value>APPLICATION_DEFAULT</value>
</property>
No additional Workload Identity Federation-specific Hadoop properties are required.

1. Configure the Hadoop Classpath

Configure the Hadoop classpath to use the updated connector.

export HADOOP_CLASSPATH=/path/to/gcs-connector-3.1.17-shaded.jar

1. Configure Spark Jobs

If Spark jobs access Google Cloud Storage, complete one of the following actions:

• Install the connector JAR on every Spark node.
• Pass the connector JAR when submitting the Spark job.

For example:

spark-submit \
--jars /path/to/gcs-connector-3.1.17-shaded.jar \
...

1. Configure Workload Identity Federation Credentials

Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the Workload Identity Federation credential configuration file.

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/wif-config.json

1. Validate the Configuration

Run the following command to verify that ODP can access the GCS bucket using Workload Identity Federation.

hadoop fs -ls gs://<bucket-name>
If the configuration is successful, the command lists the contents of the specified bucket.

Validation Steps

1. Verify GCS Connector Classes

Command:

jar tf gcs-connector-hadoop3-3.1.17-shaded.jar | grep GoogleHadoopFileSystem
Output:

com/google/cloud/hadoop/fs/gcs/GoogleHadoopFileSystem.class
Result:
The connector contains the required Hadoop filesystem implementation class.

1. Validate Hadoop Filesystem Access to GCS

Command:

hadoop fs -ls gs://odptestlab01
Output:

Found 1 items -rwx------ 3 hdfs hdfs 9 2026-06-10 02:20 gs://odptestlab01/Test.txt
Result: Hadoop successfully listed objects from the GCS bucket.

Findings

The validation confirms that:

• ODP 3.3.6.x can successfully connect to Google Cloud Storage.
• The bundled GCS connector supports:

google.cloud.auth.type=APPLICATION_DEFAULT

• Application Default Credentials are successfully consumed using:

GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/odp-gcs-sa.json

• Hadoop filesystem access to GCS using gs:// is functional.

Additional Connector Inspection

The connector JAR was inspected and contains classes related to:

• Access token providers
• External account credentials
• STS token exchange
• Identity pool subject token suppliers

Examples:

com/google/cloud/hadoop/util/AccessTokenProvider.class com/google/auth/oauth2/ExternalAccountCredentials.class com/google/auth/oauth2/StsTokenExchangeRequest.class com/google/auth/oauth2/IdentityPoolSubjectTokenSupplier.class
These components are commonly associated with:

• Application Default Credentials
• Workload Identity Federation
• External account authentication
• Token exchange authentication

Limitations

Workload Identity Federation was not tested end-to-end. The presence of WIF-related authentication classes indicates potential support, but this should not be treated as validated until tested using a GCP-generated WIF credential configuration file.

Example not tested:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/wif-config.json

Conclusion

The lab validation successfully confirmed that ODP 3.3.6.x can access Google Cloud Storage using Application Default Credentials (ADC) with:

google.cloud.auth.type=APPLICATION_DEFAULT
and

export GOOGLE_APPLICATION_CREDENTIALS=<credential-file>
While Workload Identity Federation (WIF) was not tested end-to-end, inspection of the bundled GCS connector indicates the presence of Google authentication libraries commonly used for WIF and external account authentication flows.
A future validation can be performed using a GCP-generated WIF credential configuration file (wif-config.json) to confirm end-to-end WIF support.