Configure HDP Hive, Tez, and Sqoop

This page describes how to configure Hive, Tez, and Sqoop services in HDP clusters to enable Pulse to collect query statistics, performance metrics, and lineage information.

Configure HDP Hive for Pulse

This section describes how to configure Hive so that Pulse can collect query data and metrics.

Configure HiveServer2 JMX

In the Ambari UI:

1. Navigate to Hive > Configs > Advanced hive-env > hive-env template.
2. In hive-envtemplate``, choose and add one of the following JMX configurations based on your security requirements.

Enable JMX without Security on JMX Remote Port

To enable JMX port without any security, add the following parameters at the end of the file:

if [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"
fi

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"
fi

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password> -Dcom.sun.management.jmxremote.port=8008"
fi

Configure Hive Metastore JMX

In the Ambari UI:

1. Navigate to Hive > Configs > Advanced hive-env > hive-env template.
2. In hive-envtemplate``, choose and add one of the following JMX configurations based on your security requirements.

Enable JMX without Security on JMX Remote Port

To enable JMX port without any security, add the following parameters at the end of the file:

if [ "$SERVICE" = "metastore" ]; then
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8009"
fi

Enable Basic Authentication on JMX Remote Port (Optional)

To enable basic authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "metastore" ]; then
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=8008"
fi

Enable TLS/SSL on JMX Remote Port (Optional)

To enable TLS/SSL authentication on the JMX remote port, add the following parameters:

if [ "$SERVICE" = "metastore" ]; then
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=</path/to/jmxremote.access/file> -Dcom.sun.management.jmxremote.password.file=</path/to/jmxremote.password/file> -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Djavax.net.ssl.keyStore=</path/to/keystore.jks/file> -Djavax.net.ssl.keyStorePassword=<Keystore Password> -Djavax.net.ssl.trustStore=</path/to/truststore.jks/file> -Djavax.net.ssl.trustStorePassword=<Truststore Password> -Dcom.sun.management.jmxremote.port=8008"
fi

Place Hive Hook JARs

Pulse uses Hive hooks to capture query statistics.

Hook JAR Mapping

Steps to Place Hive Hook JARs

1. Get the Hive hook JARs from Acceldata (see mapping below).
2. Place the JARs on all edge nodes, HiveServer2, and Hive interactive nodes in:

/opt/acceldata

1. Ensure the hook directory is readable and executable by all users.
2. Add the hook JARs to AUX_CLASSPATH:

For example:

export AUX_CLASSPATH=/opt/acceldata/ad-hive-hook_hdp_1.2.x-assembly-x.y.x.jar

• For Hive Interactive, navigate to Hive > Configs > Advanced hive-interactive-env, add the following hook jar.

For example:

export AUX_CLASSPATH=/opt/acceldata/ad-hive-hook_hdp_2.1.x-assembly-x.y.z.jar

hive.exec.failure.hooks
hive.exec.pre.hooks
hive.exec.post.hooks

Configure HDP Tez for Pulse

This section describes how to configure Tez so that Pulse can collect query events.

Prerequisites

• Obtain the Hive hook JARs from the Acceldata team.
• Review the mapping table to identify the correct JAR version for your HDP distribution. For details, see the section Hook JAR Mapping on this page.

Steps for Configuration

1. Identify the hook JAR location.

• For Hive 3.x on HDP 3.x (example: HDP 3.1.4):

Use the JAR ad-hive-hook_hdp_3.1.0.3.1.4.0-315-assembly-x.y.z.jar on the HDFS path:

/hdp/apps/${hdp.version}/tez/tez.tar.gz

• For Hive 1.x on HDP 2.6.x: Use the JAR ad-hive-hook_hdp_1.2.x-assembly-x.y.z.jar on the HDFS path.

$plugin[89wocz3mp33]

• For Hive 2.x (LLAP) on HDP 2.6.x:

Use the JAR ad-hive-hook_hdp_2.1.x-assembly-x.y.z.jar on the HDFS path:

/hdp/apps/${hdp.version}/tez_hive2/tez.tar.gz

1. Update the Tez tarball with the Pulse hook JAR.

# Create a directory
mkdir -p tez_pack/ && cd tez_pack

# Take backup of existing tez tarball in HDFS /tmp

hdfs dfs -cp /hdp/apps/<cluster_version>/tez/tez.tar.gz /tmp

# Download tez tarball from HDFS to local, switch to accesible user

hdfs dfs -get /hdp/apps/<cluster_version>/tez/tez.tar.gz .

# Unpack the tarball

tar -zxvf tez.tar.gz

# Copy Pulse hook jar to tez libs/

cp </location../../pulse_hook.jar> ./lib/

# Package tez tarball

tar -cvzf /tmp/tez.tar.gz .

# Upload back and provide right permissions and ownership

hdfs dfs -put -f /tmp/tez.tar.gz /hdp/apps/<cluster_version>/tez/tez.tar.gz

hdfs dfs -chown hdfs:hadoop /hdp/apps/<cluster_version>/tez/tez.tar.gz

hdfs dfs -chmod 755 /hdp/apps/<cluster_version>/tez/tez.tar.gz

1. Update the tez-site properties.
   • In the Ambari UI, navigate to: Tez > Configs > Custom tez-site.
   • Add or update the following properties:

tez.history.logging.service.class=io.acceldata.hive.AdTezEventsNatsClient
ad.events.streaming.servers (PULSE_IP:19009)
ad.cluster (your cluster name, ex: ad_hdp3_dev)

• Optional (Hive 3.x only): The property ad.hdfs.sink is set to true by default. If set to false, Tez will not publish query metadata proto logs to HDFS.

Configure HDP Sqoop for Pulse

This section describes how to configure Sqoop so that Pulse can collect query data.

1. Copy and place the Hive hook JARs (from the table above) into the Sqoop classpath directory:

Example path:

/usr/hdp/current/sqoop-client/lib

1. For clusters with LLAP (Hive Interactive) enabled, place both Hive 1.2.x and Hive 2.1.x hook JARs in the classpath.

Result

• Hive exposes metrics over the configured port.
• Hive, Tez, and Sqoop are configured with JMX monitoring, hook JARs, and Pulse integration for query and job statistics.