Acceldata
Acceldata
Search docs… ⌘K
Last updated:

Single Node Standalone Cluster Guide (Kafka/Druid/Nifi)

Kafka

Configure the Pulse Core Component for Standalone Kafka Cluster

Note

Pulse supports only Kafka versions 1.0.0 and higher.

  1. Run the following command to get started: accelo config cluster
  1. The Acceldata CLI asks you for information about your environment. The following table lists the questions asked by Acceldata CLI and a guide to help you answer the questions.

Questions asked by CLI

Guidelines for answering the questions

Is this current node an 'Edge Node' of the cluster? [y/n]:

Type y if the node on which you are installing is a Edge Node. Else type n.

Which distribution do you use?

Select Standalone

Stand-alone configuration selected:

Displays the configuration selected.

  1. The CLI asks you for few more information to generate the configuration file.

Questions asked by CLI

Guidelines for answering the questions

Enter Your Cluster’s Name:

Enter the name of the cluster.

Enter Your Cluster’s Display Name:

Enter the name you want for the cluster to display

Select the components you would like to install\n\n\n\na) MEMSQL\n\n\n\nb) KAFKA

Select the components that you want to install. You can use the arrows to move, space to select, and type in to filter

Is your Kafka version greater than '0.11.0'? [y/n]

Type y if the Kafka version is greater than V0.11.0.

Enter one of the Kafka bootstrap server's URI

Enter the Kafka boot server URL. Example: host1.kf.com:6667

What security do you use?\n\na. None\n\nb. Plan/SSL\n\nc. Kerberos

Select your security type.

Do you use TLS ?\n\na. No\n\nb. Yes

Select Yes if you use TLS or else select NO.

Detected Kafka brokers:

Based on your responses CLI would detect the Kafka brokers.

Is the above the information correct? [y/n]:

Type y to confirm if the detected Kafka brokers are correct.

Enter the Zookeeper Server URLs (comma separated with http/https & port):

Enter the Zookeeper server URL and use comma separate http, https, and the port.

Enter Kafka's Zookeeper Chroot Name (enter '/' for root):

Enter the Kafka Zookeeper roots

Would you like to continue with the above configuration? [y/n]:

Type y if you want to continue with the configuration or else type n.

Would you like to enable LDAP? [y/n]:

Type y if you want to enable LDAP or else type n.

Is Kerberos enabled in this cluster? [y/n]:

Enter your Kerberos Keytab username (Must have required HDFS permissions)

Enter the cluster name to use (MUST be all lowercase & unique):

Enter the cluster name to be used. Ensure that the name should be in all lowercase and unique.

You will see the following message.

INFO: Trying to generate host-roles-map file ...  ✓ SuccessINFO: Trying to generate alert-endpoints file ...INFO: Trying to generate FSAnalytics scripts ...WARN: Cannot find any HDFS installations from this cluster. Skipping the FS Analytics configuration.INFO: Edit the config files 'ad-core-connectors.yaml' and 'ad-fsanalyticsv2-connector.yaml'ERROR: Cannot find the required hadoop conf directory at the path '/etc/hadoop/conf'IMPORTANT: Please make sure this directory exists at the path '/etc/hadoop/conf' and contains the files 'core-site.xml' and 'hdfs-site.xml'INFO: Please run 'accelo deploy core' to deploy APM core using this configuration.
  1. You may be asked to answer the following questions if Kerberos is enabled.

Questions asked by CLI

Guidelines for answering the questions

Would you like to continue configuring Kerberos? [y/n]:

Type y if you want to continue configuring the Kerberos

Authentication realm

Add the authentication realm. Example: AZ.PULSE.COM

KDC address

Add the KDC address. Example: host1.kdc.com:88

Kerberos principal

Add the Kerberos principal. Example: admin/admin@AZ.PULSE.COM

KDC admin server address

Add the KDC administration server address. Example: host1.kdcadmin.com:749

Checking the Configuration

Open the files located in the address /data01/acceldata/config/acceldata<CLUSTER_NAME>.conf_ to verify if all the configurations are correct.

DRUID (HDP or Standalone Cluster)

  1. SSH to all druid hosts and create a directory druid-accel-extensions under /usr/hdp/<cluster_version>/druid/extensions
  2. Make following changes in druid advanced configs in Ambari UI
    • Add druid-accel-extensions in config druid.extensions.loadList=["druid-accel-extensions"...
    • Update or add druid.emitter=acceldata / example for any custom composing emitters druid.emitter.composing.emitters=["ambari-metrics","kafka","acceldata"]
    • druid.monitoring.emissionPeriod=PT15S
    • druid.emitter.acceldata.hostname=<pulse_host>
    • druid.emitter.acceldata.port=19009
    • druid.emitter.acceldata.databaseName=<cluster_name>
    • druid.emitter.acceldata.adStreamingUrl=http://{{pulse_host}}:19005
    • druid.emitter.acceldata.dbUser=<db-user>
    • druid.emitter.acceldata.dbPassword=<password>
    • druid.emitter.acceldata.group=<cluster_name>
    • druid.emitter.acceldata.monitor=druid
    • druid.emitter.acceldata.maxQueueSize=500
  3. Add the monitors that you want to monitor in the Druid Monitors. Ensure to add service-specific monitors in service specific-configs as mentioned in the following configurations. Prepare a separate config groups in Ambari for common(default), broker, and historical.

Common Configs:

druid.monitoring.monitors=["io.druid.server.metrics.EventReceiverFirehoseMonitor","io.druid.java.util.metrics.JvmThreadsMonitor","io.druid.java.util.metrics.JvmCpuMonitor","io.druid.client.cache.CacheMonitor","io.druid.java.util.metrics.SysMonitor","io.druid.java.util.metrics.JvmMonitor"]

Broker Configs:

druid.monitoring.monitors=["io.druid.client.cache.CacheMonitor","io.druid.java.util.metrics.SysMonitor","io.druid.java.util.metrics.JvmMonitor","io.druid.java.util.metrics.JvmCpuMonitor","io.druid.java.util.metrics.JvmThreadsMonitor","io.druid.server.metrics.QueryCountStatsMonitor"]

Historical Configs:

druid.monitoring.monitors=["io.druid.client.cache.CacheMonitor", "io.druid.java.util.metrics.SysMonitor", "io.druid.java.util.metrics.JvmMonitor", "io.druid.java.util.metrics.JvmCpuMonitor", "io.druid.java.util.metrics.JvmThreadsMonitor","io.druid.server.metrics.HistoricalMetricsMonitor"]
  1. Restart the affected components after the configuration changes.
  2. Check on the Influx DB for new measurements getting added and also validate the service logs on the Druid for any errors after the restart activity.

NiFi

Configuring the Apache NiFi

Follow the steps to setup the Apache NiFi in Pulse:

  1. Obtain the ad-nifi-1.0.nar download link from Acceldata team using the provided credentials
  2. Add the NiFi NAR files to all the NiFi daemons in the location Nifi library classpath. example : /usr/hdf/<hdf_version>/nifi/lib

Check for NiFi library location where AD NAR files needs to be placed.

  1. Restart the NiFi Services.
  2. Login to NiFi UI, navigate to the AccelDataReportingTask in the controller settings.

  1. Set the properties into the fields.

Field

Description

Monitor Group Name

Enter the name of cluster name as defined during Pulse installation

AccelData Influx Metric Collector URL

Add the VictoriaMetrics insert URL of the AccelData Nifi Metric Collector

Elasticsearch Concurrent Threads

Set the number. The number of threads to push the data to Elasticsearch

Elasticsearch Batch Size

Set the Elasticsearch Batch size per Thread

Max Provenance Events

Set the number to send a trigger when the maximum events occurs

Influx Batch Size

Set the number of Influx Batch size

Influx Batch Flush Duration

Set the duration of the influx batch flush in milliseconds.

Checking the Configuration

Check to see if the configuration is working as expected by following the steps:

  1. Check if the VictoriaMetrics is updated with the prefix <groupname>_nifi.
  2. The Elastic has the index with the name as Group Name + '-nifi-'.

Pulse Installation

Prerequisite

Create a Nifihosts file. The file's content will be as follows:

  1. Each line of the file will have a Nifi host and port.

Example:

nifihosts1:9090
nifihosts2:9090

  1. If the port is not given, the default port is used. For example, 9090 for alerts endpoints usage.

Configure the Pulse Core Component

  1. Run accelo config cluster

✔ Is the 'Database Service' up and running? [y/n]: : n█
✔ Stand-Alone
✔ Nifi
Enter Your Cluster's Display Name: : ad_nifi█
Enter the cluster name to use (MUST be all lowercase & unique): : ad_nifi█
Enter the cluster name to use (MUST be all lowercase & unique): : ad_nifi█
INFO: Creating Post dirs.
✔ Enter the hosts file path for Nifi (MUST formatted, one IP/host per line): : /data01/acceldata/nifihosts█
Discovered NIFI Hosts: path for Nifi (MUST formatted, one IP/host per line): : /data01/acceld
   ✓  nifi1000.sre.axl.iti:8888
   ✓  nifi1001.sre.axl.iti:8888
   ✓  nifi1002.sre.axl.iti:8888
Would you like to continue with the above NIFI nodes? [y/n]: : Y█
✓ acceldata.conf file generated successfully.
INFO: Creating post config files
INFO: Writing the .dist files
INFO: Clustername : ad_nifi
INFO: Performing PreCheck of Files
INFO: Setting the active cluster
Creating hydra inventory
SSH Key Algorithm used (RSA/DSA)?: : RSA█
Which user should connect over SSH: : root█
SSH private key file path for connecting to hosts: : /data01/clusterkey█
plat01.dev.axl.iti is the IP address of the Pulse Server, Is this correct? [Y/N]: : Y█
Would you like to enable NTP Stats? [y/n]: : y█
✔ Would you like to setup LogSearch? [y/n]: : Y█
? Select the logs for components that are installed/enabled in your target cluster:  nifi, kern, syslog
✓ Generated the vars.yml file successfully
Configuring notifications
✓ Generated the notifications.yml file successfully


Info

 Please run 'accelo deploy core' to deploy APM core using this configuration.