Configure CDP Hive and Tez

This page describes how to configure Hive and Tez in CDP clusters so that Pulse can collect query statistics, performance metrics, and lineage information.

Configure CDP Hive for Pulse

View Hive Table Details in Pulse

To display Hive table details with data in the Pulse UI:

1. Enable automatic statistics gathering. In the hive-site.xml file, set the following properties to true:

hive.stats.autogather
hive.stats.column.autogather

• This allows Hive to compute table statistics automatically.

1. Compute statistics manually (optional). You can also run the following Hive command to compute table statistics manually:

ANALYZE TABLE <table_name> COMPUTE STATISTICS;

Enable JMX for Hive Metastore and HiveServer2

For Hive Metastore Server:

1. Navigate to Hive > Java Configuration Options.
2. Update the property with the following values.

{{JAVA_GC_ARGS}} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8009
For HiveServer2:

1. Navigate to Tez > Java Configuration Options.
2. Update the property with the following values.

{{JAVA_GC_ARGS}} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008

Place Hive Hook JARs

• Get the Hive hook JARs from the Acceldata team (refer to the version mapping table).
• Place the JARs on all edge and HiveServer2 nodes in the local path:

/opt/acceldata

Hook Version Mapping

• Ensure the hook directory is readable and executable by all users.
• X.Y.Z >> For details about supported hook versions on each Pulse version, see Supported Hook Versions.

Update Hive Environment with Hook JAR

In CM, search for:

• Hive Service Environment Advanced Configuration Snippet (Safety Valve)
• Hive on Tez Service Environment Advanced Configuration Snippet (Safety Valve)

Add the following property:

AUX_CLASSPATH=${AUX_CLASSPATH}:/opt/acceldata/ad-hive-hook_cdp_3.1.3-assembly-x.y.z.jar

Update Hive Site Properties

In CM, search for:

• Hive Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml
• Hive on Tez Service Advanced Configuration Snippet (Safety Valve) for hive-site.xml

Switch to XML view and add the following properties:

<property>
<name>ad.cluster</name>
<value>[cluster_name]</value>
</property>
<property>
<name>ad.events.streaming.servers</name>
<value>[PULSE_IP]:19009</value>
</property>
<property>
<name>hive.exec.failure.hooks</name>
<value>io.acceldata.hive.AdHiveHook</value>
<description>for Acceldata APM</description>
</property>
<property>
<name>hive.exec.post.hooks</name>
<value>io.acceldata.hive.AdHiveHook</value>
<description>for Acceldata APM</description>
</property>
<property>
<name>hive.exec.pre.hooks</name>
<value>io.acceldata.hive.AdHiveHook</value>
<description>for Acceldata APM</description>
</property>

Restart Hive Services

• Restart affected Hive components.
• Deploy the updated client configuration.

Configure CDP Tez for Pulse

Place Tez Hook JARs

1. Obtain the Hive hook JARs from the Acceldata team (refer to the version mapping table).
2. Log in to any HDFS client node.

Update Tez Tarball

Update the Tez tarball with the Pulse hook JAR.

# Create a directory
mkdir -p tez_pack/ && cd tez_pack

# Take backup of existing tez tarball in HDFS /tmp

hdfs dfs -cp /user/tez/<tez_version>/tez.tar.gz /tmp

# Download tez tarball from HDFS to local, switch to accesible user

hdfs dfs -get /user/tez/<tez_version>/tez.tar.gz .

# Unpack the tarball

tar -zxvf tez.tar.gz

# Copy Pulse hook jar to tez libs/

cp </location../../pulse_hook.jar> ./lib/

# Package tez tarball

tar -cvzf /tmp/tez.tar.gz .

# Upload back and provide right permissions and ownership

hdfs dfs -put -f /tmp/tez.tar.gz /user/tez/<tez_version>/tez.tar.gz

hdfs dfs -chown tez:hadoop /user/tez/<tez_version>/tez.tar.gz

hdfs dfs -chmod 755 /user/tez/<tez_version>/tez.tar.gz

Update Tez Site Properties

1. In CM, search for: Tez Client Advanced Configuration Snippet (Safety Valve) for tez-conf/tez-site.xml
2. Switch to XML view and add the required Pulse properties.

<property>
<name>ad.cluster</name>
<value>[cluster_name]</value>
</property>
<property>
<name>ad.events.streaming.servers</name>
<value>[PULSE_IP]:19009</value>
</property>
<property>
<name>tez.history.logging.service.class</name>
<value>io.acceldata.hive.AdTezEventsNatsClient</value>
</property>

Result

The CDP cluster services are configured with Pulse hook JARs and JMX properties. Pulse can collect metrics, capture query data, and integrate with Hive and Tez.