Deploy Alerts Service on a Separate Node

This topic describes how to deploy ad-alerts as a standalone add-on on a separate node. You can deploy ad-alerts on a separate node when high CPU usage by the Alerts service causes resource constraints on the node where it is currently running.

Prerequisites

Before deploying ad-alerts as a standalone add-on, ensure that the following requirements are met:

• You are using Pulse 3.5.0 or later, or Pulse 3.4.3 or later.
• Alerts (Agents must be configured) are not currently deployed. If it is already deployed, run accelo deploy addons and unselect Alerts (Agents must be configured) to uninstall it.
• ad-director is already deployed.
• The standalone node is configured. If it is not configured, run accelo set standalone after completing the initial accelo init prompt.

Deploy ad-alerts as a Standalone Add-on

Follow these steps to deploy ad-alerts as a standalone add-on.

1. Configure ad-alerts

On the standalone node, run the following command to generate ad-alerts.yml if the file does not already exist:

accelo admin makeconfig ad-alerts

If Alerts was previously deployed on another node and ad-alerts.yml exists under $AcceloHome/config/docker/addons on that node, copy the file to the same location on the standalone node:

Path: $AcceloHome/config/docker/addons

Open ad-alerts.yml in a text editor and update the following fields:

• ALERTS_DB_URI: Set it to the value of the MONGO_URI environment variable. By default, this value is encrypted.
• NOTIFICATIONS_HOST: Set it to the hostname of the node where ad-notifications is deployed.
• NATS_SERVER_HOST_LIST: Set it to <Pulse_Node_IP_or_hostname>:19009.
• ANOMALY_DETECTION_ENDPOINT: http://<Pulse Node IP or hostname>:29000

For example:
[root@plat03:addons (ad-default)]$ diff -y --suppress-common-lines ad-alerts.yml.original ad-alerts.yml
- ALERTS_DB_URI=<existing_value>             | - ALERTS_DB_URI=<updated_value>
- NOTIFICATIONS_HOST=ad-notifications        | - NOTIFICATIONS_HOST=plat01.acceldata.dvl
- NATS_SERVER_HOST_LIST=<existing_value>     | - NATS_SERVER_HOST_LIST=<Pulse_Node_IP_or_hostname>:19009
- ANOMALY_DETECTION_ENDPOINT=http:<existing_value> | - ANOMALY_DETECTION_ENDPOINT=http://<Pulse Node IP or hostname>:29000

Save the changes.

2. Update the Cluster Configuration

On the core node, back up the following cluster configuration file:

$AcceloHome/config/acceldata_<cluster_name>.conf

Open the file in a text editor and locate the connections section.

For example:
# General dependencies are expressed as accessible service url's. Spark, Yarn,
# ATS, Hive Metastore, Spark and Webhdfs urls are required.
# At this point in time, AccelData does not support compression by default.

group {
  name = "odp_zoro"
  displayName = "odp_zoro"
  originalName = "ODP_Zoro"
  distroType = "HWX"
  onprem = true
  active = true
  priority = 1000

  connections = {
    mongo = [
      ...

Under connections, make the following changes.

Mongo

For all Mongo connections:

• uri: Change the value to the ALERTS_DB_URI value configured in the previous step.

Elastic

For each applicable Elastic connection:

• host: Change the value from ad-elastic to the hostname of the node where Log Search is deployed.
• port: Change the value from 9200 to 19013.

VM

For readURL:

• Replace ad-vmselect with the hostname of the node where the core services are deployed.
• Replace port 8481 with 19042.

Save the changes.

For example:
[root@plat01:config (ad-default)]$ diff -y --suppress-common-lines acceldata_odp_zoro.conf.orig acceldata_odp_zoro.conf
uri = "<existing_value>"                         | uri = "<ALERTS_DB_URI>"
uri = "<existing_value>"                         | uri = "<ALERTS_DB_URI>"

host = "ad-elastic"                              | host = "plat01.acceldata.dvl"
port = 9200                                      | port = 19013

host = "ad-elastic"                              | host = "plat01.acceldata.dvl"
port = 9200                                      | port = 19013

host = "ad-elastic"                              | host = "plat01.acceldata.dvl"
port = 9200                                      | port = 19013

readURL = "http://ad-vmselect:8481/select/..."   | readURL = "http://plat01.acceldata.dvl:19042/select/..."

3. Push the Cluster Configuration

Run the following command on the core node to push the updated configuration:

accelo admin database push-config -a

4. Configure ad-core

On the core node, run the following command to generate ad-core.yml if the file does not already exist:

accelo admin makeconfig ad-core

Open ad-core.yml in a text editor and locate the ad-graphql section:

ad-graphql:
  image: ad-graphql
  container_name: ""
  environment:
    ...

Under environment, add the following environment variables:

• ALERTS_IP: Set it to the hostname of the standalone node where ad-alerts will be deployed.
• ALERTS_PORT: Set it to 19015.

For example:
[root@plat01:docker (ad-default)]$ diff -y --suppress-common-lines ad-core.yml.orig ad-core.yml
> - ALERTS_IP=plat03.acceldata.dvl
> - ALERTS_PORT=19015

Save the changes.

5. Package the Alerts Working Directories

On the core node, navigate to $AcceloHome:

cd $AcceloHome

Run the following command:

tar -cvf work_dir.tar.gz $(ls -d work/*/alerts)

• The command creates work_dir.tar.gz under $AcceloHome.
• Copy work_dir.tar.gz to $AcceloHome on the standalone node.

You can use SCP, an NFS export, or another supported file-transfer method to copy the file.

6. Extract the Alerts Working Directories

On the standalone node, navigate to $AcceloHome:

cd $AcceloHome

Extract the archive:

tar -xvf work_dir.tar.gz

7. Restart the Core Services

On the core node, restart all services:

accelo restart all -d

8. Deploy Alerts

On the standalone node, run:

accelo deploy addons

When prompted, select Alerts (Agents must be configured).

Reconfigure a Cluster After Deploying ad-alerts on a Separate Node

When you run either of the following commands:

accelo reconfig cluster

or:

accelo reconfig cluster -a

Perform the following additional steps to ensure that the standalone ad-alerts deployment uses the updated cluster configuration.

1. Update readURL in:

$AcceloHome/config/acceldata_<cluster_name>.conf

2. Push the updated configuration:

accelo admin database push-config -a

3. On the core node, navigate to $AcceloHome and create an archive of the Alerts working directories:

cd $AcceloHome
tar -cvf work_dir.tar.gz $(ls -d work/*/alerts)

4. Copy work_dir.tar.gz to $AcceloHome on the standalone node.
5. On the standalone node, navigate to $AcceloHome and extract the archive:

cd $AcceloHome
tar -xvf work_dir.tar.gz

6. Restart ad-alerts on the standalone node:
   

accelo restart ad-alerts

Update Alerts and Notifications Configuration

When you run accelo config alerts notifications, perform the command on the appropriate core node and then synchronize the updated Alerts working directories with the standalone node.

1. On the core node, navigate to $AcceloHome and create an archive of the Alerts working directories:

cd $AcceloHome
tar -cvf work_dir.tar.gz $(ls -d work/*/alerts)

2. Copy work_dir.tar.gz to $AcceloHome on the standalone node.
3. On the standalone node, navigate to $AcceloHome and extract the archive:

cd $AcceloHome
tar -xvf work_dir.tar.gz

4. Restart ad-alerts on the standalone node:

accelo restart ad-alerts