Acceldata
 Pulse
Search documentation… ⌘K

Configure HDP Spark

This page helps you enable Spark metrics via the Graphite sink, allowing Pulse to capture Spark application and JVM metrics.

Configure Spark Metrics Sink

In the Ambari UI:

  1. Navigate to Spark > Configs > Advanced spark2-metrics-properties.
  2. Add the following properties:
# Graphite sink class
*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
# Location of your graphite instance
*.sink.graphite.host=localhost
*.sink.graphite.port=12003
*.sink.graphite.protocol=tcp
*.sink.graphite.prefix=spark.metrics
*.sink.graphite.period=20
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Info

This change requires Pulse Node Agent running on all spark clients.

Additional Notes

  • On edge nodes not managed by Ambari, manually append the above properties to:
/etc/spark2/conf/metrics.properties
  • Ensure the following properties are enabled in spark-defaults.conf for all Spark jobs:
spark.eventLog.enabled=true
spark.eventLog.dir=hdfs:///spark2-history/
  • Apply the same properties in managed configurations for applications running on Spark 1.x and 3.x.

Result

  • Spark sends metrics (including JVM metrics) to Graphite for Pulse ingestion.