Acceldata
 Pulse
Search documentation… ⌘K

Configure CDH Spark

This page describes configuring Spark metrics in CM to enable JMX monitoring and Graphite reporting for Spark and Spark2 clients.

Steps for Configuration

  1. In the CM UI, navigate to Service > Configuration tab.
  2. In the search bar, type the following, then update the parameters with the appropriate values.
  • Spark Client Advanced Configuration Snippet (Safety Valve) → maps to spark-conf/spark-defaults.conf.
  • Spark2 Client Advanced Configuration Snippet (Safety Valve) → maps to spark2-conf/spark-defaults.conf.
spark.metrics.conf.*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
spark.metrics.conf.*.sink.graphite.host=localhost
spark.metrics.conf.*.sink.graphite.port=12003
spark.metrics.conf.*.sink.graphite.protocol=tcp
spark.metrics.conf.*.sink.graphite.prefix=spark.metrics
spark.metrics.conf.*.sink.graphite.period=20
spark.metrics.conf.*.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Info

For details on configuring other details for Spark, see Additional Notes.

Result

Spark jobs will send JVM and application metrics to Graphite, enabling Pulse to collect, visualize, and analyze Spark performance.