Acceldata
 Pulse
Search documentation… ⌘K

Configure CDP Spark

This page describes configuring Spark metrics in CM to enable JMX monitoring and Graphite reporting for Spark and Spark2 clients.

Steps for Configuration

  1. In the CM UI, navigate to Service > Configuration tab.

  2. In the search bar, type the following, then update the parameters with the appropriate values.

  • Spark Client Advanced Configuration Snippet (Safety Valve) → maps to spark-conf/spark-defaults.conf.

  • Spark2 Client Advanced Configuration Snippet (Safety Valve) → maps to spark2-conf/spark-defaults.conf.

spark.metrics.conf.*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
spark.metrics.conf.*.sink.graphite.host=localhost
spark.metrics.conf.*.sink.graphite.port=12003
spark.metrics.conf.*.sink.graphite.protocol=tcp
spark.metrics.conf.*.sink.graphite.prefix=spark.metrics
spark.metrics.conf.*.sink.graphite.period=20
spark.metrics.conf.*.source.jvm.class=org.apache.spark.metrics.source.JvmSource

spark.ad.connector.context yarn;<ClusterName>;admin;test
spark.ad.nats.server.host.list http://<nats1>:19009,http://<nats2>:19009
spark.extraListeners io.acceldata.sparkstats.AdSparkListener
spark.ad.events.url <pulse_ip>:19009

Result

Spark jobs will send JVM and application metrics to Graphite, enabling Pulse to collect, visualize, and analyze Spark performance.