Disaster Recovery Strategy for MongoDB and VictoriaDB

Prerequisites

• A minimum of three nodes to enable High Availability are required
• MongoDB Version 6 is required for this setup

Steps to Deploy Pulse on New Node

1. Set the MongoDB URI:

export MONGO_URI="<paste previous command output here>"

1. Enable MongoDB Encryption:

export MONGO_ENCRYPTED=true

1. Configure Pulse as a Standalone Node:

export PULSE_SA_NODE=true

1. Create necessary directory structure:

mkdir -p /data01/acceldata/

1. Move the Pulse executable to the designated directory:

mv accelo.linux /data01/acceldata/accelo

1. Navigate to the Pulse directory:

cd /data01/acceldata/

1. Initialize Pulse:

./accelo init

1. Source the environment variables:

source /etc/profile.d/ad.sh

1. Reinitialize Pulse:

accelo init

1. Retrieve Pulse information:

accelo info

1. Log in to Pulse Docker:

accelo login docker

Add Host Entries

Add the following entries to the /etc/hosts file of all the MongoDB nodes, including the Pulse server:

<Pulse-Server-IP> ad-db
<Second-Node-IP> ad-db2
<Third-Node-IP> ad-db3

Set New MongoDB Nodes

1. Retrieve the accelo binary and initiate it with the same version tag as the Pulse version on the primary node.

accelo init

1. Login to the Pulse Registry:

Docker

docker network create ad-default

1. Create a new Docker network:

Docker

docker network create ad-default

1. Manually download the Mongo image:

Docker

docker pull 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>

Configuring the Primary Node (Pulse Server)

1. Edit Configuration File: In the config/docker/ad-core.yaml file, add the following new fields to the ad-db service section:

• Add the following fields to the environment block:

ENABLE_MONGODB_RS=true
MONGODB_RS_NAME=addb0
IS_MONGODB_PRIMARY=true

• Add the following fields to the volumes block:

/etc/localtime:/etc/localtime:ro
/root/acceldata/data/db:/data/db
/root/acceldata/config/db/mongokey:/mongokey
Add the following fields to the extra_hosts block:

• Add the following fields to the extra_hosts block:

ad-db2:<Second-Node-IP>
ad-db3:<Third-Node-IP>

1. Run Secondary Data Replica: Replace <Pulse-Server-IP>, <Third-Node-IP>, and <PULSE-VERSION> with the appropriate values and execute:

docker run -d --name ad-db3_default -p 27017:27017 -e MONGO_DATA_DIR=/data/db -e MONGO_LOG_DIR=/dev/null -e ENABLE_MONGODB_RS=true -e MONGODB_RS_NAME=addb0 -v /data01/acceldata/config/db/mongokey:/mongokey -v /etc/localtime:/etc/localtime:ro -v /data01/acceldata/data/db:/data/db --restart=unless-stopped --net=ad-default --network-alias=ad-db3 --add-host ad-db:<Pulse-Server-IP> --add-host ad-db2:<Second-Node-IP> 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>

1. Run Third Data Replica: Replace <Pulse-Server-IP>, <Second-Node-IP>, and <PULSE-VERSION> with the appropriate values and execute:

docker run -d --name ad-db3_default -p 27017:27017 -e MONGO_DATA_DIR=/data/db -e MONGO_LOG_DIR=/dev/null -e ENABLE_MONGODB_RS=true -e MONGODB_RS_NAME=addb0 -v /data01/acceldata/config/db/mongokey:/mongokey -v /etc/localtime:/etc/localtime:ro -v /data01/acceldata/data/db:/data/db --restart=unless-stopped --net=ad-default --network-alias=ad-db3 --add-host ad-db:<Pulse-Server-IP> --add-host ad-db2:<Second-Node-IP> 191579300362.dkr.ecr.us-east-1.amazonaws.com/acceldata/ad-database:<PULSE-VERSION>

1. Login to Primary Node's MongoDB Shell & Add Replica Members:

chown 1000:1000 /data01/acceldata/config/db/mongokey
docker exec -it ad-db_default bash
mongosh mongodb://admin:ACCELROOT_01082018@localhost:27017/admin
db.adminCommand({"replSetReconfig":{_id: "addb0",version: 2,members: [{ _id : 0, host : "ad-db:27017" },{ _id : 1, host : "ad-db2:27017" },{ _id : 2, host : "ad-db3:27017" }]}})

1. Check Replica Status: Run rs.status() to check the status of the replica.
2. Wait for Stable State: Wait for the MongoDB cluster to stabilize.
3. Update Mongodb Encrypted URI: Replace the MongoDB encrypted URI in all YAML files under config/docker/ and acceldata_acceluster.conf file with the provided value.

Unencrypted string for reference:

mongodb://accel:ACCELUSER_01082018@ad-db:27017,ad-db2:27017,ad-db3:27017/admin?replicaSet=addb0

VictoriaDB Setup for Backup and Restoration

wget "https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.97.1/vmutils-linux-amd64-v1.97.1.tar.gz "

1. Extract the contents of the above tar file:

tar -xvf vmutils-linux-amd64-v1.97.1.tar.gz

1. Remove unnecessary binaries, except vmbackup and vmrestore.
2. Execute the shell script:

source /etc/profile.d/ad.sh

1. Create a new directory for vmdbbackup as below:

mkdir $AcceloHome/data/vmdbbackup
chown 1000:1000 $AcceloHome/data/vmdbbackup
su - $(id -un 1000)

1. Open up access to the ad-vmstorage port by modifying the ad-core.yml file as shown below.

ad-vmstorage:
image: ad-vmstorage
container_name: ""
environment:
- ARGS=--storageDataPath=/storage
volumes:
- /etc/localtime:/etc/localtime:ro
- /data01/acceldata/data/vmdb:/storage
ulimits: {}
ports:
- 19099:8482
depends_on: []
opts: {}
restart: always
extra_hosts: []
network_alias: []

1. Restart vmstorage container using:

accelo restart ad-vmstorage

1. Execute the following command. This command initiates an incremental backup process. Incremental backups are conducted when the destination (-dst) parameter refers to an existing backup directory. After the initial backup is completed, subsequent backups will always target this existing backup directory.

vmbackup-prod -storageDataPath=/data01/acceldata/data/vmdb -snapshot.createURL=http://localhost:19099/snapshot/create -dst=fs:///data01/acceldata/data/vmdbbackup

1. Configure the crontab to execute this task once daily, preferably in the morning:

crontab -u user -e
0 1 * * * /data01/acceldata/vmbackup-prod -storageDataPath=/data01/acceldata/data/vmdb -snapshot.createURL=http://localhost:19099/snapshot/create -dst=fs:///data01/acceldata/data/vmdbbackup

Steps to Restore

To restore from backup, perform the following:

1. Stop the vmstorage container, if it already running, and run the restore command:

docker stop ad-vmstorage_default
vmrestore-prod -src=fs:///data01/acceldata/data/vmdbbackup -storageDataPath=/data01/acceldata/data/vmdb

1. Run accelo deploy core
2. Check the logs of the container using docker logs -f ad-vmstorage_default

Troubleshooting

• If the backup process is slow, consider adjusting the value of the -concurrency flag to a higher setting. This will boost the number of simultaneous workers responsible for uploading data to the backup storage.
• If vmrestore or vmbackup encounters an interruption due to a temporary error, simply restart it using the same arguments. The process will resume from where it left off, continuing the restore or backup operation.