Acceldata
 Pulse
Search documentation… ⌘K

Configure ODP Spark 2 and 3

This page helps you enable Spark metrics via the Graphite sink, allowing Pulse to capture Spark application and JVM metrics.

Configure Spark Metrics Sink

In the Ambari UI:

  1. Navigate to Spark > Configs > Advanced spark2-metrics-properties.
  2. Update the following parameters:

Bash

# Graphite sink class

*.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink

# Location of your graphite instance

*.sink.graphite.host=localhost

*.sink.graphite.port=12003

*.sink.graphite.protocol=tcp

*.sink.graphite.prefix=spark.metrics

*.sink.graphite.period=20

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource

worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource

driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource

executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource

Steps for Spark Hook Update

  1. In the Ambari UI, navigate to Spark > Configs > Custom spark-defaults.
  2. Add or update the following properties:

Bash

spark.ad.connector.context yarn;<ClusterName>;admin;test

spark.ad.nats.server.host.list http://<nats1>:19009,http://<nats2>:19009

spark.extraListeners io.acceldata.sparkstats.AdSparkListener

spark.ad.events.url <pulse_ip>:19009

  1. Save the configuration changes.
  2. Remove the existing Spark hook JARs from the $SPARK_HOME/jars directory on all Spark client nodes.
  3. Copy the compatible Spark hook JARs (For example, version 2.1.2) to the $SPARK_HOME/jars directory on all Spark client nodes. For the supported hook version on each Pulse release, see Supported Hook Versions.
  4. Restart the affected Spark services.
  5. Submit a test Spark job and verify that Spark events and metrics are successfully reported to Pulse.

Result

  • Spark 2.x/3.x sends metrics (including JVM metrics) to Graphite for Pulse ingestion.