Acceldata
 Pulse
Search documentation… ⌘K

Configure HDP Spark

This page helps you enable Spark metrics via the Graphite sink, allowing Pulse to capture Spark application and JVM metrics.

Configure Spark Metrics Sink

In the Ambari UI:

  1. Navigate to Spark > Configs > Advanced spark2-metrics-properties.
  2. Add the following properties:

Bash

# Graphite sink class

*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink

# Location of your graphite instance

*.sink.graphite.host=localhost

*.sink.graphite.port=12003

*.sink.graphite.protocol=tcp

*.sink.graphite.prefix=spark.metrics

*.sink.graphite.period=20

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource

worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource

driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource

executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Info

This change requires Pulse Node Agent running on all spark clients.

Steps for Spark Hook Update

  1. In the Ambari UI, navigate to Spark > Configs > Custom spark-defaults.
  2. Add or update the following properties:

Bash

spark.ad.connector.context yarn;<ClusterName>;admin;test

spark.ad.nats.server.host.list http://<nats1>:19009,http://<nats2>:19009

spark.extraListeners io.acceldata.sparkstats.AdSparkListener

spark.ad.events.url <pulse_ip>:19009

  1. Save the configuration changes.
  2. Remove the existing Spark hook JARs from the $SPARK_HOME/jars directory on all Spark client nodes.
  3. Copy the compatible Spark hook JARs (version 2.1.2) to the $SPARK_HOME/jars directory on all Spark client nodes. For the supported hook version on each Pulse release, see Supported Hook Versions.
  4. Restart the affected Spark services.
  5. Submit a test Spark job and verify that Spark events and metrics are successfully reported to Pulse.

Additional Notes

  • On edge nodes not managed by Ambari, manually append the above properties to:

Bash

/etc/spark2/conf/metrics.properties

  • Ensure the following properties are enabled in spark-defaults.conf for all Spark jobs:

Bash

spark.eventLog.enabled=true

spark.eventLog.dir=hdfs:///spark2-history/

  • Apply the same properties in managed configurations for applications running on Spark 1.x and 3.x.

Result

  • Spark sends metrics (including JVM metrics) to Graphite for Pulse ingestion.