Pulse Multi-Node Rancher Based Deployment

This document explains the process of deploying Pulse on a single node Kubernetes (K8s). This deployment is performed by using Rancher.

Prerequisites

• All the servers must be part of the same network.
• You must manually set the hostname of the servers you create by executing the command hostnamectl set-hostname pulk8s01.sre.axl
• Do ssh copy-id in all the list of IPs which is provided from datacenter.
  • If you do not have a key pair in your system, you must execute the command ssh-keygen.
  • ssh-copy-id root@<<IP address>>
• Install Kubectl from the official Kubernetes documentation.

Rancher Cluster Setup in K8s

To set up a rancher based cluster:

1. Clone the GIT repository git@bitbucket.org:acceldata/ad-platform-infrastructure.git.
2. Copy the public key to the internal_infra folder. You can navigate to this folder by executing the following command. You must use the same key pair that was generated above.

cd ad-platform-infrastructure/ansible/internal_infra

1. Edit the docker.yml, content. You must replace the id_rsa.pub, with the name of the file as shown in the following block

# Change
authorized_key:
        user: docker
        state: present
        key: "{{ lookup('file', 'id_rsa.pub') }}"
# To
authorized_key:
        user: docker
        state: present
        key: "{{ lookup('file', 'popk8s.pub') }}"

1. Edit the docker password as shown in the following block.

# Change
name: Add a new user named docker
      user:
        name: docker
        groups: docker
        password:
# To
name: Add a new user named docker
      user:
        name: docker
        groups: docker
        password: 24242sfsfad##$

1. Save and exit the docker.yml file.
2. Edit the hosts.ini file with the fully qualified domain name (FQDN) and the Internet Protocol (IP) and update the key. You can accomplish this by executing the following block.

[popk8s]
popk8smultinode1000.qe.axl.iti  ansible_host=<<ip address>>
popk8smultinode1001.qe.axl.iti  ansible_host=<<ip address>>
popk8smultinode1002.qe.axl.iti  ansible_host=<<ip address>>
[all:vars]
ansible_user=root
ansible_python_interpreter=/usr/bin/python
ansible_ssh_private_key_file=~/.ssh/popk8s

1. Test if the ping command works, based on the group name provided. You can accomplish this by executing the following command.

ansible -m ping popk8s

1. Execute the following command to install docker and ntp in all the machines that are part of the group popk8s. You must execute the command in all the machines that are part of the popk8s group.

ansible-playbook docker.yml --extra-vars="nodes=popk8s"

Install Rancher

1. Download and install Ranched by executing the following block of code.

wget https://github.com/rancher/rke/releases/download/v1.3.3/rke_linux-amd64
chmod +x rke_linux-amd64
mv rke_linux-amd64 /usr/local/bin/rke
rke --version
>>rke version v1.3.3

1. Configure RKE/Rancher to bring up Kubernetes cluster. You can accomplish this by executing the following block.

# Create a directory by executing the following command.
mkdir popk8smultinodecluster
# Change the directory by executing the following command.
bash-3.2$ cd popk8smultinodecluster/
#enter password
bash-3.2$ pwd
/<<filepath>>/popk8smultinodecluster
#Configuratrion command
bash-3.2$ rke config
# You are asked the following questions during installation. You can view the sample responses given by Acceldata. You can modify the responses as per your organization requirements.
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/popk8s
[+] Number of Hosts [1]: 3
[+] SSH Address of host (1) [none]: <<ip address>>
[+] SSH Port of host (1) [22]:
[+] SSH Private Key of host (<<ip address>>) [none]: ~/.ssh/popk8s
[+] SSH User of host (<<ip address>>) [ubuntu]: docker
[+] Is host (<<ip address>>) a Control Plane host (y/n)? [y]: y
[+] Is host (<<ip address>>) a Worker host (y/n)? [n]: y
[+] Is host (<<ip address>>) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (<<ip address>>) [none]:
[+] Internal IP of host (<<ip address>>) [none]:
[+] Docker socket path on host (<<ip address>>) [/var/run/docker.sock]:
[+] SSH Address of host (2) [none]: <<ip address>>
[+] SSH Port of host (2) [22]:
[+] SSH Private Key Path of host (<<ip address>>) [none]: ~/.ssh/popk8s
[+] SSH User of host (<<ip address>>) [ubuntu]: docker
[+] Is host (<<ip address>>) a Control Plane host (y/n)? [y]:
[+] Is host (<<ip address>>) a Worker host (y/n)? [n]: y
[+] Is host (<<ip address>>) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (<<ip address>>) [none]:
[+] Internal IP of host (<<ip address>>) [none]:
[+] Docker socket path on host (<<ip address>>) [/var/run/docker.sock]:
[+] SSH Address of host (3) [none]: <<ip address>>
[+] SSH Port of host (3) [22]:
[+] SSH Private Key Path of host (<<ip address>>) [none]: ~/.ssh/popk8s
[+] SSH User of host (<<ip address>>) [ubuntu]: docker
[+] Is host (<<ip address>>) a Control Plane host (y/n)? [y]: y
[+] Is host (<<ip address>>) a Worker host (y/n)? [n]: y
[+] Is host (<<ip address>>) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (<<ip address>>) [none]:
[+] Internal IP of host (<<ip address>>) [none]:
[+] Docker socket path on host (<<ip address>>) [/var/run/docker.sock]:
[+] Network Plugin Type (flannel, calico, weave, canal, aci) [canal]: weave
[+] Authentication Strategy [x509]:
[+] Authorization Mode (rbac, none) [rbac]:
[+] Kubernetes Docker image [rancher/hyperkube:v1.21.7-rancher1]:
[+] Cluster domain [cluster.local]:
[+] Service Cluster IP Range [<<ip address>>/16]:
[+] Enable PodSecurityPolicy [n]:
[+] Cluster Network CIDR [<<ip address>>/16]:
[+] Cluster DNS Service IP [<<ip address>>]:
[+] Add addon manifest URLs or YAML files [no]:
bash-3.2$ ls
cluster.yml popk8s

1. Edit the cluster.yml file. You must replace the entire ingress block the following block.

ingress:
  provider: none

1. Create the K8s cluster by executing the following code.

bash-3.2$ rke up
You get the following message on screen.

INFO[0000] Running RKE version: v1.3.3
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [<<ip address>>]
INFO[0000] [dialer] Setup tunnel for host [<<ip address>>]
INFO[0000] [dialer] Setup tunnel for host [<<ip address>>]
INFO[0001] Checking if container [cluster-state-deployer] is running on host [<<ip address>>], try #1
INFO[0001] Pulling image [rancher/rke-tools:v0.1.78] on host [<<ip address>>], try #1
If you encounter any error in getting the K8s cluster up, you must execute the following command in each node and reboot the node.

rm -rf /etc/kubernetes/
rm -rf /var/lib/kubelet/
rm -rf /var/lib/etcd/
docker rmi $(docker images -a -q)
docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)
You must now test the connectivity to the Kubernetes cluster. You may encounter an error with the message The connection to the server localhost:8080 was refused - did you specify the right host or port? . If you get this error, you must add the following command to .bashrc.

export KUBECONFIG=/root/popk8smulti/kube_config_cluster.yml

1. You can check the list of hosts by executing the following command.

kubectl get nodes
To execute the kubectl command, you must install Kubectl tool. You can follow this documentation to install Kubectl.

Metallb and Nginx Setup

1. Download the keyz binary for ECR secret and replace the following file For MAC system.

https://bitbucket.org/acceldata/ad-automation-test/src/Dev/ops/popk8scluster/

1. Check out the Bitbucket repository https://bitbucket.org/acceldata/ad-service/src/development/.

bash-3.2$ pwd
/<<filepath>>/helm-controller
bash-3.2$ bash-3.2$ ls .scripts/keyz
.scripts/keyz
bash-3.2$ chmod +x ./.scripts/keyz
##### Verification that you are using the keys relevant for MAC
bash-3.2$ file .scripts/keyz
.scripts/keyz: Mach-O 64-bit executable x86_64

1. Open the helmfile-ha.yml or helmfile.yml file.
2. Comment the lines of code as shown in the following block.

hooks:
- events: ["prepare", "presync"]
showlogs: true
command: ".scripts/ecr-secret-reset.sh"
args: ["{{ .Namespace }}"]
# - events: ["prepare"]
# showlogs: true
# command: ".scripts/clone-package.sh"
# args: ["{{ .Namespace }}"]
# - events: ["prepare"]
# command: ".scripts/add-package.sh"
# - events: ["cleanup"]
# command: ".scripts/remove-package.sh"

Nginx Deployment

If you are planning multiple Pulse installations and if each installation has a different namespace, you must execute the steps in the Pulse Multiple Namespace Deployment document and then proceed with the steps.

1. Execute the following commands to install the helmfile.

wget (appropriate link for your system from https://github.com/roboll/helmfile/releases/tag/v0.143.3)
chmod +x helmfile_linux-amd64
mv helmfile_linux_amd64 /usr/local/bin/helmfile
helmfile --version
helmfile version v0.143.3

1. Execute the following commands to install the helm.

wget https://get.helm.sh/helm-v3.9.4-linux-amd64.tar.gz
tar -xvf helm-v3.9.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm

1. Execute the following commands in to install K9s.

mkdir k9s-install
cd k9s-install/
curl -sS https://webinstall.dev/k9s | bash
You get the response as shown in the following block

[root@pulsek8smain ~]# mkdir k9s-install
[root@pulsek8smain ~]# cd k9s-install/
[root@pulsek8smain k9s-install]# curl -sS https://webinstall.dev/k9s | bash
Thanks for using webi to install 'k9s@stable' on 'Linux/x86_64'.
Have a problem? Experience a bug? Please let us know:
https://github.com/webinstall/webi-installers/issues
Lovin' it? Say thanks with a Star on GitHub:
https://github.com/webinstall/webi-installers
Downloading k9s from
https://github.com/derailed/k9s/releases/download/v0.26.3/k9s_Linux_x86_64.tar.gz
Saved as /root/Downloads/webi/k9s/0.26.3/k9s_Linux_x86_64.tar.gz
Extracting /root/Downloads/webi/k9s/0.26.3/k9s_Linux_x86_64.tar.gz
Installing to /root/.local/opt/k9s-v0.26.3/bin/k9s
Installed 'k9s v0.26.3' as /root/.local/bin/k9s
PATH.env updated with:
export PATH="/root/.local/bin:$PATH"
TO FINISH: copy, paste & run the following command:
source ~/.config/envman/PATH.env
(newly opened terminal windows will update automatically)

1. Execute the following commands to complete the installation.

source ~/.config/envman/PATH.env
k9s
You can now use the helmfile by executing the following commands.

helmfile --interactive --file helmfile-ha.yaml -l type=network -e on-prem/ha --state-values-set 'host=remote','containers.tagName=2.1.8' -n <namespace-of-new-pulse> sync
helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l app=ad-nginx-controller --state-values-set 'containers.tagName=2.1.9','host=remote' -n <namespace-of-new-pulse> sync
You can verify the nginx deployment by executing the following commands.

kubectl get po -n <name-of-new namespace> -o wide
kubectl get svc -n <name-of-new namespace> -o wide
## You must be able to view a list of nginx related pods and services.

Metallb Deployment

1. Open the helm-controller values.yaml (the file location is helm-controller/ha/ad-mettlb/values.yaml)
2. Ensure you use your subnet to specify the range and not some other subnet.

bash-3.2$ vi values.yaml
configInline:
address-pools:
- name: default
protocol: layer2
addresses:
- <<ip address>>-<<ip address>> ## Enter the right subnet here.
bash-3.2$ helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l app=ad-metallb --state-values-set 'containers.tagName=2.1.9','host=remote' -n <name of namespace> sync

1. Execute the commands in the following block to deploy Metallb.

bash-3.2$ helm repo add metallb https://metallb.github.io/metallb
"metallb" has been added to your repositories
bash-3.2$ helm fetch metallb/metallb --version=0.11.0
bash-3.2$ ls
README.md helmfile-ha.yaml helmfile.yaml metallb-0.11.0.tgz on-prem resolv.conf
bash-3.2$ pwd
/root/ad-service/ops/helm-controller
bash-3.2$ tar -xvf metallb-0.11.0.tgz
x metallb/Chart.yaml
x metallb/values.yaml
x metallb/values.schema.json
x metallb/templates/NOTES.txt
x metallb/templates/_helpers.tpl
x metallb/templates/config.yaml
x metallb/templates/controller.yaml
x metallb/templates/podmonitor.yaml
x metallb/templates/prometheusrules.yaml
x metallb/templates/psp.yaml
x metallb/templates/rbac.yaml
x metallb/templates/service-accounts.yaml
x metallb/templates/speaker.yaml
x metallb/.helmignore
x metallb/README.md
x metallb/policy/controller.rego
x metallb/policy/rbac.rego
x metallb/policy/speaker.rego
bash-3.2$ cd metallb
bash-3.2$ ls
Chart.yaml README.md policy templates values.schema.json values.yaml
## Create the following file in the metallb folder
bash-3.2$ cat metallb.yaml
configInline:
address-pools:
- name: default
protocol: layer2
addresses:
- <<ip address>>-<<ip address>>
bash-3.2$ kubectl create namespace metallb
namespace/metallb created
bash-3.2$ helm install metallb . -f metallb.yaml -n <name of namespace>

1. Verify if metallb pods and nginix services get an External IP from the range provided.

kubectl get po -n <name of namespace>
kubectl get svc -n <name of namespace> -o wide

Rook-CEPH Deployment

Execute the following commands.

bash-3.2$ helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l app=ad-rook-ceph-cluster --state-values-set 'host=remote' -n <name of namespace> sync
Conclusion rook-chef pods should be steady/Running
bash-3.2$ kubectl get po --namespace rook-ceph -w
bash-3.2$ kubectl get cephcluster -n <name of namespace>

Deployment of Pulse DB, Core, and Add-ons

If you are planning multiple Pulse installations and if each installation has a different namespace, you must execute the steps in the Pulse Multiple Namespace Deployment document and then proceed with the steps.

1. Execute the following command for DB deployment.

bash-3.2$ pwd
/<<filepath>>/helm-controller
bash-3.2$ helm repo add bitnami https://charts.bitnami.com/bitnami
bash-3.2$ helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l type=db --state-values-set 'host=remote','containers.tagName=2.1.9-210622' -n <name of namespace> sync

1. Execute the following command for deployment of accelo and deployer.

#### Accelo it is:
helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l app=ad-accelo --state-values-set 'host=remote','containers.tagName=2.1.9-210622' -n <name of namespace> sync
Un-comment the statements from helmfile-ha.yml and helmfile.yml

The following sample commands must be used to push license file, keytab, certs, krb5.conf to pvc.

For HDP or Kafka standalone (ansible deployment), you must generate an SSH key and ensure that you are able to perform passwordless secure login to the cluster nodes, and then copy the key to the ad-accelo pod. sample command. The sample command is as follows.

bash-3.2$ kubectl cp /<<filepath>>/ad-accelo-configure-59b7c6dfc6-wnqhk:/data01/acceldata/

MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXosC8vfVkGYGWGPNnX64ZVS/C0fUKWTwgrNMA6uLlQYcnbP46QSEcXxPDDUBIGh7fY=
MONGO_ENCRYPTED=true
FSANALYTICS_URI=http://ad-fsanalyticsv2-connector:19027
HYDRA_SERVER_URL=http://ad-hydra:19072

1. Configure core in the accelo pod by executing the following command.

accelo init
## If successful, you receive the following message.
✓ Done, AccelData Init Successful.

1. In the pod execute the following command.

accelo config cluster
You get the message as shown in the following block.

bash-5.1# accelo config cluster
INFO: Creating a Mongo Client
ERROR: mongo: no documents in result
ERROR: Cannot Find the active cluster from the MongoDB
ERROR: mongo: no documents in result
ERROR: Cannot Find the active cluster from the MongoDB
INFO: Configuring the cluster ...
The deployer asks the following questions. You can respond to them according to your configurations.

The following block shows sample responses for each of the above questions.

bash-5.1# accelo config cluster
INFO: Creating a Mongo Client
INFO: Configuring the cluster ...
Is the 'Database Service' up and running? [y/n]: : y█
INFO: Setting up configuration ...
Is this current node an 'Edge Node' of the cluster? [y/n]: : n█
✔ HDP
Enter Your Cluster's Display Name: : advisa_test█
✔ Enter Ambari URL (with http/https): : http://<<ip address>>:8080█
✔ Enter Ambari URL (with http/https): : http://<<ip address>>:8080█
The hostname for the Ambari is : : <<ip address>>
Enter Ambari Username: : admin█
IMPORTANT: This password will be securely encrypted and stored in this machine.
Enter Ambari User Password: : *****█
Contacting Ambari .... ✓ Success
Trying to fetch available services ... ✓ Success
Do you have hive metastore enabled (y/n)?: : y█
1m: y█
?
Select the hive metastore type: mysql/mariadb
Enter the hive metastore Database Name : : hive█
✔ Enter the hive metastore DB URI (example: db.ad.com:3306): : jdbc:mysql://
visa1000.sre.axl.i✔
Enter the hive metastore DB Username : : hive█
Enter the hive metastore DB Password : : ****█
Enter the hive metastore Server Timezone (Example: IST/UTC) : : IST█
✓ Success
Trying to fetch cluster hosts ... ✓ Success
The sample discovered configurations window is as follows. You can also find the next set of questions asked.

---------------------------Discovered configurations----------------------------------------
✓ Cluster Type: HDP
✓ HDP Version: 3.1.0
✓ Discovered Cluster Name: advisa_test
✓ Discovered Services:
✓ ACCUMULO: 1.7.0
✓ SUPERSET: 0.23.0
✓ STORM: 1.2.1
✓ LOGSEARCH: 0.5.0
✓ HDFS: 3.1.1.3.1
✓ KNOX: 1.0.0
✓ RANGER: 1.2.0.3.1
✓ AMBARI_INFRA_SOLR: 0.1.0
✓ SMARTSENSE: 1.5.1.2.7.4.0-118
✓ SQOOP: 1.4.7
✓ OOZIE: 4.4.0
✓ MAPREDUCE2: 3.1.1
✓ PIG: 0.16.0
✓ SPARK2: 2.3.0
✓ DRUID: 0.12.1
✓ TEZ: 0.9.1
✓ RANGER_KMS: 1.2.0.3.1
✓ ATLAS: 1.1.0
✓ HIVE: 3.1.0
✓ ZEPPELIN: 0.8.0
✓ KAFKA: 2.0.0
✓ HBASE: 2.0.2
✓ ZOOKEEPER: 3.4.6
✓ YARN: 3.1.1
✓ KERBEROS: 1.10.3-30
✓ AMBARI_METRICS: 0.1.0
✓ Yarn RM URI: http://visa1000.sre.axl.iti:8088,http://visa1001.sre.axl.iti:8088
✓ MapReduce Job History URI: http://visa1001.sre.axl.iti:19888
✓ Yarn ATS URI: http://visa1001.sre.axl.iti:8188
✓ HDFS Namenode URI: webhdfs://nameservice1
✓ Hive Metastore URI: thrift://visa1000.sre.axl.iti:9083
✗ Hive LLAP is not enabled
✓ Spark History Server URI: http://visa1001.sre.axl.iti:18081
✓ Kafka Broker URI: http://visa1002.sre.axl.iti:6667
✓ Zookeeper Server URI: http://visa1000.sre.axl.iti:2181,http://visa1001.sre.axl.iti:2181,http://visa1002.sre.axl.iti:2181
✔ Would you like to continue with the above configuration? [y/n]: : y█
Is Kerberos enabled in this cluster? [y/n]: : y█
ERROR: CANNOT FIND KERBEROS REALM FROM API
Would you like to enter the kerberos realm? [y/n]: : y█
Enter your Kerberos realm (Ex: CDH.DEMO.COM)?: : ACCELVISA.COM█
Enter your Kerberos keytab username (Must have required HDFS permissions): : hdfs█
Enter the cluster name to use (MUST be all lowercase & unique): : advisa_test█t█
✔ Enter the cluster name to use (MUST be all lowercase & unique): : advisa_test█
This cluster name 'advisa_test' already exists. Would you like to reconfigure it [Y/N]?: : Y█
This cluster name 'advisa_test' already exists. Would you like to reconfigure it [Y/N]?: : Y█
INFO: Pushing the active cluster config to the MongoDB
IMPORTANT: If you have TLS/SSL enabled for your services, copy the java keystore files 'jssecacerts' and 'cacerts' into the directory '/data01/acceldata/config/security/'
INFO: Edit the config files 'ad-core-connectors.yaml' and 'ad-fsanalyticsv2-connector.yaml'
Please Copy the files core-site.xml, hdfs-site.xml to /data01/acceldata/work/advisa_test/hadoop/conf locationINFO: Please run 'accelo deploy core' to deploy APM core using this configuration.
INFO: Trying to generate alert-endpoints file ...
✓ AMBARI GRAFANA
✓ HBASE
INFO: Cannot find any secondary namenode hosts from Ambari.
✓ HDFS
ERROR: Cannot find any Hive Interactive server ports from Ambari.
✓ HIVE
✓ KAFKA
✓ MAPREDUCE
WARN: Cannot find any Spark hive thrift servers from Ambari.
WARN: Cannot find any Spark Livy2 servers from Ambari.
✓ SPARK2
✓ YARN
✓ ZOOKEEPER
INFO: Trying to generate host-roles-map file ... ✓ Success
INFO: Trying to generate the pulse.yaml file...
INFO: Trying to generate FSAnalytics scripts ...
✔ike to continue configuring kerberos? [y/n]: : y█[0m: y█
Setting up kerberos..
WARN: 'ping' executable not found in this machine. So accelo will NOT check the
Kerberos server connectivity.
Enter the Authentication realm: : ACCELVISA.COM█
Enter the Authentication realm: : ACCELVISA.COM█
Enter the KDC address (eg: host1.kdc.com:88): : <<ip address>>.kdc.com:88█
Enter the principal: : hdfs@ACCELVISA.COM█
✗ Enter full path to the Keytab file (eg: /root/hdfs.keytab): : /data01/acceldata/hdfs.headless.ke✗
Enter full path to the Keytab file (eg: /root/hdfs.keytab): : /data01/acceldata/hdfs.headless.ke✗
Enter full path to the Keytab file (eg: /root/hdfs.keytab): : /data01/acceldata/hdfs.headless.ke✗
Enter full path to the Keytab file (eg: /root/hdfs.keytab): : /data01/acceldata/hdfs.headless.ke✔
Enter full path to the Keytab file (eg: /root/hdfs.keytab): : /data01/acceldata/hdfs.headless.ke✔
Enter full path to the Keytab file (eg: /root/hdfs.keytab): : /data01/acceldata/hdfs.headless.ke
Enter full path to the Keytab file (eg: /root/hdfs.keytab): : /data01/acceldata/hdfs.headless.keytab█
Enter the KDC admin server address (eg: host1.kdcadmin.com:749): : <<ip address>>.kdcadmin.com:749█
Enter the krb5Conf file path: : /data01/acceldata/krb5.conf█
✓ Done, Kerberos setup completed.
INFO: Configuring Pulse agent ...
SSH Key Algorithm used (RSA/DSA)?: : RSA█
SSH private key file path for connecting to hosts: : /data01/acceldata/krb5.conf█
Which user should connect over SSH: : root█
✔ <<ip address>> is the IP address of the AccelData APM Server, Is this correct? [Y/N]: : N█
What is the IP address of the AccelData APM Server: : <<ip address>>
INFO: Setting up the configuration ...
Do you want to install component specific agents? [Y/N]: : Y█
? Select the components you would like to install: Zookeeper, Hdfs, Hbase, Yarn, HiveServer2, Metastore, MapReduce2
Does zookeeper_server has kerberos enabled?: [y/n]: : y█
Does namenode has kerberos enabled?: [y/n]: : y█
✔ Does datanode has kerberos enabled?: [y/n]: : y█
Does journalnode has kerberos enabled?: [y/n]: : y█
Does hbase_regionserver has kerberos enabled?: [y/n]: : y█
Does hbase_master has kerberos enabled?: [y/n]: : y█
Does nodemanager has kerberos enabled?: [y/n]: : y█
Does resourcemanager has kerberos enabled?: [y/n]: : y█
Does hive_metastore has kerberos enabled?: [y/n]: : y█
Would you like to enable NTP Stats? [y/n]: : n█
Do the cluster nodes have java binary in the OS PATH [Y/N]: : Y█
Would you like to enable NIFI Support? [y/n]: : n█
Enter the JMX Port for hive_server: : 8009█
Enter the JMX Port for hive_metastore: : 8010█
Enter the JMX Port for zookeeper_server: : 8989█
[1m: 8989█
Enter the Kafka JMX Port (Ex: 9999): : 9999█
Would you like to install Kapxy? [y/n]: : y█
Would you like to install Kldagent? [y/n]: : n█
✔ Would you like to setup LogSearch? [y/n]: : y█
Select services for log search:
? Select components: hdfs_datanode, hdfs_namenode, yarn_nodemanager, yarn_resourcemanager, yarn_timelineserver, yarn_timelinereader, mapred_historyserver, hbase_regionserver, hbase_master, kafka_server, kafka_controller, kafka_server_gc, kafka_state_change, kafka_log_cleaner, kafka_err, metastore, hive_server, yarn_application, hdfs_audit, zookeeper, syslog, kern
Would you like to setup Acceldata Hooks? [y/n]: : n█
✓ Done, agents configuration completed.
INFO: Pushing Acceldata Conf file into the Database
Creating group monitors [==================================================================>-------------] 83.33%SKIPPED: notification.yml file was not found in the alerts dir.
Done [================================================================================================] 100.00% 1s
Push completed successfully!
INFO: Pushing vars.yml file to Acceldata DB
INFO: Pushing hydra_hosts.yml file to Acceldata DB
INFO: Pushing the hydra_hosts.yml to mongodb
INFO: Updating the Epoch Time
INFO: Reloading the Hydra Server
ERROR: Cannot connect to the Hydra server. Because: Get "http://ad-hydra:19072/reload": dial tcp: lookup ad-hydra on <<ip address>>:53: no such host
✓ Done, Configuration of Pulse Completed.
Now to deploy core components, Run: ./accelo deploy core.

• The IP addresses for Hydra is same as the container name, To Fix this issue, create a override.yml file in the /data01/acceldata/work/<Cluster_Name>/ location.
• The cluster name for Victoria metrics is obtained from the acceldata__advisa__test.conf. You must replace it in override.yml file.
• You must add the following environment.

export PULSE_HOSTNAME=<<ip address>>
The content of override.yml file is as follows.

base:
ad_stream_url: http://<LOADBALANCER_IP>:19005
jmx_rmi_url: http://<LOADBALANCER_IP>:19043/insert/<CLUSTER_HASH>/influx
tsdb_url: http://<LOADBALANCER_IP>:19043/insert/<CLUSTER_HASH>/influx
logstash_url: <LOADBALANCER_IP>:19012
yarn_app_es_urls: http://<LOADBALANCER_IP>:19013
components:
impala_agent_collector_url: http://<LOADBALANCER_IP>:19005/impala/batch/queries
hydra:
   base_url: http://<LOADBALANCER_IP>:19072

Mongo DB URI Correction

1. Execute the following commands in accelo pod.

bash-5.1# env | grep -i mongo
MONGO_ENCRYPTED=true
MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXosC8vfVkGYGWGPNnX64ZVS/C0fUKWTwgrNMA6uLlQYcnbP46QSEcXxPDDUBIGh7fY=

1. Replace the MONGO_URI=ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXosC8vfVkGYGWGPNnX64ZVSp9yHgErQknPBAfYZ9cOG1A== in the /data01/acceldata/config/acceldata<clusterName>.conf file.

group {
name = "k8skafka"
displayName = "k8skafka"
originalName = "k8skafka"
distroType = "SAKafka"
onprem = true
active = true
priority = 1000
connections = {
mongo = [
{
name = "default"
uri = "ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXosC8vfVkGYGWGPNnX64ZVSp9yHgErQknPBAfYZ9cOG1A=="
encrypted = true
secret = "Ah+MqxeIjflxE8u+/wcqWA=="
dbName = "k8skafka"
},
{
name = "alerts"
uri = "ZN4v8cuUTXYvdnDJIDp+R8Z+ZsVXXjv8zDOvh8UwQXosC8vfVkGYGWGPNnX64ZVSp9yHgErQknPBAfYZ9cOG1A=="
encrypted = true
secret = "Ah+MqxeIjflxE8u+/wcqWA=="
dbName = "alerts"
}
],
For ansible based deployment of hydra (HDP/kafka), fix hydra_server_ ur manually by executing the following code block.

bash-5.1# cat /data01/acceldata/work/k8skafka/hydra_hosts.yml
cluster:
hosts:
kafka27server1000.qe.axl.iti: ""
kafka27server1001.qe.axl.iti: ""
kafka27server1002.qe.axl.iti: ""
vars:
ansible_user: root
ansible_python_interpreter: /usr/bin/python
ansible_ssh_private_key_file: /ansible/work/clusterkey
hydra_server_url: http://<LOADBALANCER_IP>:19072
hydra_log_level: ""
hydra_agent_heartbeat: "10"
hydra_parcel_mode: false
hydra_hostname_method: CMD
cluster_name: k8skafka
ansible_pipelining: false
ansible_ssh_common_args: ""
ansible_HostKeyAlgorithms: ""
ansible_PubkeyAcceptedKeyTypes: ""
enable_nifi: false

1. Execute the following command.

accelo admin database push-config

Deploy Core Services

1. Uncomment the following statements from helmfile-ha/helmfile.yml file, You must perform this step for all core and addon services. You must retain the comment if you wish to deploy rook-ceph, ad-accelo.

hooks:
- events: ["prepare", "presync"]
showlogs: true
command: ".scripts/ecr-secret-reset.sh"
args: ["{{ .Namespace }}"]
# - events: ["prepare"]
# showlogs: true
# command: ".scripts/clone-package.sh"
# args: ["{{ .Namespace }}"]
# - events: ["prepare"]
# command: ".scripts/add-package.sh"
# - events: ["cleanup"]
# command: ".scripts/remove-package.sh"

1. The version/build number currently used is containers.tagName=2.1.9-200622. You must execute the following command.

bash-3.2$ helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l type=core --state-values-set 'host=remote','containers.tagName=2.1.9-200622' -n <name of namespace> sync

Deploy Add-on Services

1. Execute the following command to deploy the add on services.

bash-3.2$ helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l type=addon --state-values-set 'host=remote','containers.tagName=2.1.9-200622' -n <name of namespace> sync
You can now access the Pulse UI from the following URL.

http://<External IP for Nginix SVC>:4000

FSA Load

You must execute these commands after configuring the add-on services.

1. Connect to ad-accelo-configure, pod and run the following command.

bash-5.1# accelo k8s fsa
## This command allows admin operations on the AccelData database
Usage:
accelo k8s fsa [command]
Available Commands:
load Loads Acceldata FSAnalytics Data
reports Reports Acceldata FSAnalytics Data
Flags:
-h, --help help for fsa
Use "accelo k8s fsa [command] --help" for more information about a command.
bash-5.1#

1. Execute the folloiwng to configure notifications.

## Same steps as usual, enable webhook, email or any other notifications that you want to
bash-5.1# accelo config alerts notifications
INFO: Creating a Mongo Client
✔ Enter the JODA Timezone value (Example: Asia/Jakarta): : Asia/Kolkata█

1. Validate if the ad-hydra server container is running and then execute the following command.

accelo reconfig cluster -m ENV

Other Configurations

Installation and Uninstallation of agents for HDP/Kafka (ansible based deployment). Once sync is run you must delete it before running with “uninstall tag“.

1. Ensure that the /ad-service/ops/helm-controller/helmfile-ha.yaml file has the following content.

- name: ad-deployer
labels: # Arbitrary key value pairs for filtering releases
app: ad-deployer
namespace: "{{ .Namespace }}"
chart: ../helm-charts/ad-deployer
version: ~0.0.1
values:
- "{{.Environment.Name }}/ad-deployer/values.yaml"
- "{{.Environment.Name }}/ad-deployer/env-vars.yaml"
set:
- name: tags
value: "--tags install,kapxy_check"
# value: "--tags uninstall"
- name: ansible_become_pass
value: "{{ .Values.ansible_become_pass }}"
- name: ansible_ssh_pass
value: "{{ .Values.ansible_ssh_pass }}"
- name: clusterName
value: "{{ .Values.clusterName }}"
- name: containers.tagName
value: {{ .Values.containers.tagName }}
Below is the command that takes things into affect.
helmfile --interactive --file helmfile-ha.yaml -l app=ad-deployer -e on-prem/ha --state-values-set 'host=remote','containers.tagName=2.1.9-260722','clusterName=kafkapop' -n <name of namespace> delete
helmfile --interactive --file helmfile-ha.yaml -l app=ad-deployer -e on-prem/ha --state-values-set 'host=remote','containers.tagName=2.1.9-260722','clusterName=kafkapop' -n <name of namespace> sync
: clusterName is the same name that you provided in the accelo config cluster.

1. Execute the commands in the following block to monitor weave scope.

kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')"
bash-3.2$ kubectl get po -n weave
NAME READY STATUS RESTARTS AGE
weave-scope-agent-9kkz5 1/1 Running 0 20m
weave-scope-agent-jjxb7 1/1 Running 0 20m
weave-scope-agent-shhk8 1/1 Running 0 20m
weave-scope-app-588b789d4-pmxw7 1/1 Running 0 20m
weave-scope-cluster-agent-5d7745c9c9-2vwz9 1/1 Running 0 20m
bash-3.2$ kubectl port-forward -n weave "$(kubectl get -n weave pod --selector=weave-scope-component=app -o jsonpath='{.items..metadata.name}')" 4040
Forwarding from 127.0.0.1:4040 -> 4040
Forwarding from [::1]:4040 -> 4040
You can access the weave UI from the http://localhost:4040 UI.

1. Execute the following command to delete the add-ons.

bash-3.2$ helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l type=addon --state-values-set 'host=remote','containers.tagName=2.1.9-200622' -n <name of namespace> delete

1. To deploy a new version of the build, execute the following command (Consider that the current version is 2.1.9-200622 and you want to upgrade to 2.1.9-200822).

bash-3.2$ helmfile --interactive --file helmfile-ha.yaml -e on-prem/ha -l type=addon --state-values-set 'host=remote','containers.tagName=2.1.9-200822' -n <name of namespace> sync

Configurations for Kafka Standalone

For Kafka you need not deploy all the add-on services. You must add a new selection kafkastandalone: true.

bash-3.2$ cat helmfile-ha.yaml | grep -B2 standalone
app: ad-dashplot
type: addon
kafkastandalone: true
--
app: ad-director
type: addon
kafkastandalone: true
--
app: ad-alerts
type: addon
kafkastandalone: true
--
app: ad-kafka-connector
type: addon
kafkastandalone: true
--
app: ad-ldap
type: addon
kafkastandalone: true
--
app: ad-notifications
type: addon
kafkastandalone: true
--
app: ad-pg
type: addon
kafkastandalone: true

1. Execute the following commands to deploy addons for Kafka.

bash-3.2$ helmfile --interactive --log-level debug --file helmfile-ha.yaml -e on-prem/ha -l kafkastandalone=true,type=addon --state-values-set 'host=remote','containers.tagName=2.1.9-290722' -n <name of namespace> sync

1. The override.yaml file must have the following content. This file requires a access reconfigured cluster after the changes are done. This required for logs agent to understand and read the log files for Kafka.

bash-5.1# pwd
/data01/acceldata/work/k8skafka
bash-5.1# cat override.yml
log_locations:
kafka_controller:
- path: /kafka27/kafka_2.12-2.7.2/logs/controller.log
type: DEFAULT
kafka_err:
- path: /kafka27/kafka_2.12-2.7.2/logs/kafka.err
type: KAFKA_ERR
kafka_log_cleaner:
- path: /kafka27/kafka_2.12-2.7.2/logs/log-cleaner.log
type: DEFAULT
kafka_server:
- path: /kafka27/kafka_2.12-2.7.2/logs/server.log
type: DEFAULT
kafka_server_gc:
- path: /kafka27/kafka_2.12-2.7.2/logs/kafkaServer-gc.log.*.current
type: DEFAULT
kafka_state_change:
- path: /kafka27/kafka_2.12-2.7.2/logs/state-change.log
type: DEFAULT
kern:
- path: /var/log/kern.log
type: DATESTAMP
syslog:
- path: /var/log/syslog,/var/log/messages
type: DATESTAMP
zookeeper:
- path: /kafka27/kafka_2.12-2.7.2/logs/zookeeper.log
type: DEFAULT