Start the ODP Stack Services

Start All Services Manually

Once the Ambari upgrade is done, start all the services manually and make sure it all started.

JDK 11 Env Changes

The following flags must be added after upgrading Infra-Solr, Hadoop, HBase, Hive, and Druid env files, before starting the services in a JDK11 environment.

Hadoop-env

The Java 11 flags have been addressed in the ${else} block, which needs to be updated after the upgrade is completed during the service start process.

File location: Log on to the Ambari UI → HDFS → configs → Advanced -> Advanced hadoop-env -> hadoop-env template.

{% elif java_version == 8 %}
SHARED_HDFS_NAMENODE_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseG1GC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{namenode_opt_newsize}} -XX:MaxNewSize={{namenode_opt_maxnewsize}} -Xlog:gc*,gc+heap=debug,gc+phases=debug:file={{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'`:time,level,tags -XX:InitiatingHeapOccupancyPercent=70 -Xms{{namenode_heapsize}} -Xmx{{namenode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT"
export HDFS_NAMENODE_OPTS="${SHARED_HDFS_NAMENODE_OPTS} -XX:OnOutOfMemoryError=\"/usr/odp/current/hadoop-hdfs-namenode/bin/kill-name-node\" -Dorg.mortbay.jetty.Request.maxFormContentSize=-1 ${HDFS_NAMENODE_OPTS}"
export HDFS_DATANODE_OPTS="-server -XX:ParallelGCThreads=4 -XX:+UseG1GC -XX:OnOutOfMemoryError=\"/usr/odp/current/hadoop-hdfs-datanode/bin/kill-data-node\" -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -XX:NewSize=200m -XX:MaxNewSize=200m -Xlog:gc*,gc+heap=debug,gc+phases=debug:file=/var/log/hadoop/$USER/gc.log-`date +'%Y%m%d%H%M'`:time,level,tags -Xms{{dtnode_heapsize}} -Xmx{{dtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HDFS_DATANODE_OPTS} -XX:InitiatingHeapOccupancyPercent=70" export HDFS_SECONDARYNAMENODE_OPTS="${SHARED_HDFS_NAMENODE_OPTS} -XX:OnOutOfMemoryError=\"/usr/odp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node\" ${HDFS_SECONDARYNAMENODE_OPTS}"
# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx${HADOOP_HEAPSIZE}m $HADOOP_CLIENT_OPTS"
{% else %}
SHARED_HDFS_NAMENODE_OPTS="-server -XX:ParallelGCThreads=8 -XX:+UseG1GC -XX:ErrorFile={{hdfs_log_dir_prefix}}/$USER/hs_err_pid%p.log -XX:NewSize={{namenode_opt_newsize}} -XX:MaxNewSize={{namenode_opt_maxnewsize}} -Xlog:gc*,gc+heap=debug,gc+phases=debug:file={{hdfs_log_dir_prefix}}/$USER/gc.log-`date +'%Y%m%d%H%M'`:time,level,tags -XX:InitiatingHeapOccupancyPercent=70 -Xms{{namenode_heapsize}} -Xmx{{namenode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT"
export HDFS_NAMENODE_OPTS="${SHARED_HDFS_NAMENODE_OPTS} -XX:OnOutOfMemoryError=\"/usr/odp/current/hadoop-hdfs-namenode/bin/kill-name-node\" -Dorg.mortbay.jetty.Request.maxFormContentSize=-1 ${HDFS_NAMENODE_OPTS}"
export HDFS_DATANODE_OPTS="-server -XX:OnOutOfMemoryError=\"/usr/odp/current/hadoop-hdfs-datanode/bin/kill-data-node\" -XX:ErrorFile=/var/log/hadoop/$USER/hs_err_pid%p.log -verbose:gc -Xms{{dtnode_heapsize}} -Xmx{{dtnode_heapsize}} -Dhadoop.security.logger=INFO,DRFAS -Dhdfs.audit.logger=INFO,DRFAAUDIT ${HDFS_DATANODE_OPTS} "
export HDFS_SECONDARYNAMENODE_OPTS="${SHARED_HDFS_NAMENODE_OPTS} -XX:OnOutOfMemoryError=\"/usr/odp/current/hadoop-hdfs-secondarynamenode/bin/kill-secondary-name-node\" ${HDFS_SECONDARYNAMENODE_OPTS}"
{% endif %}

Hbase-env

Update the HBase Java 11 flags as shown below.

File location: Log on to the Ambari UI → HBase → configs → Advanced hbase-env -> hbase-env template.

{% if java_version == 8 %}
# For Java version 8
export SERVER_GC_OPTS="-verbose:gc -XX:-PrintGCCause -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:{{log_dir}}/gc.log-`date +'%Y%m%d%H%M'` -XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution"
{% else %}
# For Java 11 and above
export SERVER_GC_OPTS="-Xlog:gc*,gc+heap=info,gc+age=debug:file={{log_dir}}/gc.log-`date +'%Y%m%d%H%M'`:time,uptime:filecount=10,filesize=200M -XX:+UseG1GC"
{% endif %}

Hive-env

Update the Hive Java 11 flags in two places as shown below.

• metastore
• hiveserver2

1. File location: Log on to the Ambari UI → hive → configs → Advanced hive-env -> hive-env template.

# The heap size of the jvm, and jvm args stared by hive shell script can be controlled via:
java_version=$(java -version 2>&1 | awk -F[\".] '/version/ {print $2}')
if [ "$SERVICE" = "metastore" ]; then
export HADOOP_HEAPSIZE={{hive_metastore_heapsize}} # Setting for HiveMetastore
if [ "$java_version" -eq 8 ]; then
export HADOOP_OPTS="$HADOOP_OPTS -Xloggc:{{hive_log_dir}}/hivemetastore-gc-%t.log -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath={{hive_log_dir}}/hms_heapdump.hprof -Dhive.log.dir={{hive_log_dir}} -Dhive.log.file=hivemetastore.log"
else
export HADOOP_OPTS="$HADOOP_OPTS -Xlog:gc*,gc+heap=debug,gc+phases=debug:file={{hive_log_dir}}/hivemetastore-gc-%t.log:time,level,tags:filecount=10,filesize=10M -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath={{hive_log_dir}}/hms_heapdump.hprof -Dhive.log.dir={{hive_log_dir}} -Dhive.log.file=hivemetastore.log"
fi
fi
if [ "$SERVICE" = "hiveserver2" ]; then
export HADOOP_HEAPSIZE={{hive_heapsize}} # Setting for HiveServer2 and Client
if [ "$java_version" -eq 8 ]; then
# For Java 8
export HADOOP_OPTS="$HADOOP_OPTS -Xloggc:{{hive_log_dir}}/hiveserver2-gc-%t.log -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath={{hive_log_dir}}/hs2_heapdump.hprof -Dhive.log.dir={{hive_log_dir}} -Dhive.log.file=hiveserver2.log"
else
# For Java 11 and above
export HADOOP_OPTS="$HADOOP_OPTS -Xlog:gc*,gc+heap=debug,gc+phases=debug:file={{hive_log_dir}}/hiveserver2-gc-%t.log:time,level,tags:filecount=10,filesize=10M -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath={{hive_log_dir}}/hs2_heapdump.hprof -Dhive.log.dir={{hive_log_dir}} -Dhive.log.file=hiveserver2.log"
fi
fi
2. File location - Log in to the Ambari UI → hive → configs → Advanced->General

hive.tez.java.opts = -server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -Xlog:gc*,gc+heap=debug,gc+phases=debug:stdout:time,level,tags

Infra-solr-env

Update the Infra-Solr Java11 flags as shown below.

File location: Log on to the Ambari UI → Infra Solr → configs → Advanced Infr-Solr-env -> Infra Solr-env template.

if [ "$java_version" -lt 11 ]; then
# For JDK 8 and below
infra_solr_gc_log_opts="-verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=15 -XX:GCLogFileSize=200M"
GC_LOG_OPTS="{{infra_solr_gc_log_opts}} -Xloggc:{{infra_solr_log_dir}}/solr_gc.log"
infra_solr_gc_tune="-XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled"
else
# For JDK 11 and above
# Garbage Collection Logging
GC_LOG_OPTS="-Xlog:gc*=info:file={{infra_solr_log_dir}}/solr_gc.log:time,uptime"
# Use separate options to handle log file rotation and size
LOG_ROTATION_OPTS="-XX:NumberOfGCLogFiles=15 -XX:GCLogFileSize=200M"
# Garbage Collection Tuning
infra_solr_gc_tune="-XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+DisableExplicitGC"
# Combine all options
JAVA_OPTS="${GC_LOG_OPTS} ${LOG_ROTATION_OPTS} ${infra_solr_gc_tune}"
fi

Druid-env

Update the Druid Java-11 flags as shown below.

File location: Log on to the Ambari UI → Druid → configs → Advanced druid-env.

druid.broker.jvm.opts=--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Djava.security.auth.login.config={druid_jaas_file}

druid.coordinator.jvm.opts=--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Djava.security.auth.login.config={druid_jaas_file}

druid.historical.jvm.opts=--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Djava.security.auth.login.config={druid_jaas_file}

druid.middlemanager.jvm.opts=--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Djava.security.auth.login.config={druid_jaas_file}

druid.overlord.jvm.opts=--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Djava.security.auth.login.config={druid_jaas_file}

druid.router.jvm.opts=--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Djava.security.auth.login.config={druid_jaas_file}

Tez-env

Update the Tez Java11 flags as shown below.

File location: Log on to the Ambari UI → Tez → configs → Advanced tez-env -> tez-env template.

# Tez specific configuration
export TEZ_CONF_DIR={{config_dir}}
if [ "$java_version" -eq 8 ]; then
# Java 8 JVM options
export tez.am.launch.cmd-opts="-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB{{heap_dump_opts}}"
export tez.task.launch.cmd-opts="-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB{{heap_dump_opts}}"
else
# Java 11 JVM options
export tez.am.launch.cmd-opts="-Xlog:gc*:file={{gc_log_path}}/tez_am_gc.log:time,uptime,level,tags:filecount=10,filesize=10M -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB{{heap_dump_opts}}"
export tez.task.launch.cmd-opts="-Xlog:gc*:file={{gc_log_path}}/tez_task_gc.log:time,uptime,level,tags:filecount=10,filesize=10M -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB{{heap_dump_opts}}"
fi
# Set HADOOP_HOME to point to a specific hadoop install directory
export HADOOP_HOME=${HADOOP_HOME:-{{hadoop_home}}}
# The java implementation to use.
export JAVA_HOME={{java64_home}}
Following are the reference templates for making Java 11 runtime flags:

1. Druid-env-template
2. Hadoop-env-template
3. Hbase-env-template
4. Hive-env-template
5. Infra-solr-env-template

Known Issues

During the ODP upgrade, where Ambari-2.7.9.0 has already been upgraded, the following services might experience downtime. This downtime interrupts the ODP stack upgrade when using the express method.

Affected Services:

• Ranger
• Ranger KMS
• HDFS
• Kafka

Ranger

Issue 1: Ranger Admin service start

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/ODP/3.0/services/RANGER/package/scripts/ranger_admin.py", line 258, in <module>
RangerAdmin().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 351, in execute
method(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 1009, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/stacks/ODP/3.0/services/RANGER/package/scripts/ranger_admin.py", line 114, in start
self.configure(env, upgrade_type=upgrade_type, setup_db=params.stack_supports_ranger_setup_db_on_start)
File "/var/lib/ambari-agent/cache/stacks/ODP/3.0/services/RANGER/package/scripts/ranger_admin.py", line 151, in configure
setup_ranger_xml.ranger('ranger_admin', upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/stacks/ODP/3.0/services/RANGER/package/scripts/setup_ranger_xml.py", line 49, in ranger
setup_ranger_admin(upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/stacks/ODP/3.0/services/RANGER/package/scripts/setup_ranger_xml.py", line 207, in setup_ranger_admin
content=InlineTemplate(params.admin_logback_content),
File "/usr/lib/ambari-agent/lib/resource_management/core/source.py", line 149, in __init__
super(InlineTemplate, self).__init__(name, extra_imports, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/source.py", line 136, in __init__
self.template = self.template_env.get_template(self.name)
File "/usr/lib/ambari-agent/lib/ambari_jinja2/environment.py", line 717, in get_template
return self._load_template(name, self.make_globals(globals))
File "/usr/lib/ambari-agent/lib/ambari_jinja2/environment.py", line 691, in _load_template
template = self.loader.load(self, name, globals)
File "/usr/lib/ambari-agent/lib/ambari_jinja2/loaders.py", line 115, in load
source, filename, uptodate = self.get_source(environment, name)
ValueError: too many values to unpack (expected 3)
Solution:

1. Log in to the backend database for the Ambari-server.
   • In this case, MySQL is used as the Ambari server’s backend database.
2. Execute the following SQL query to identify the required configurations.

SELECT * FROM (
SELECT
cc.config_id,
cc.type_name,
service_name,
sc.version,
sc.note,
(SELECT MAX(version) FROM ambari.serviceconfig s
WHERE s.service_name = sc.service_name) AS max_version
FROM
ambari.serviceconfig sc
INNER JOIN
ambari.serviceconfigmapping scm
ON sc.service_config_id = scm.service_config_id
INNER JOIN
ambari.clusterconfig cc
ON cc.config_id = scm.config_id
WHERE
sc.service_name IN ("RANGER_KMS", "RANGER")
AND sc.version = (SELECT MAX(version) FROM ambari.serviceconfig s
WHERE s.service_name = sc.service_name)
) AS res
WHERE type_name IN ("kms-log4j", "admin-log4j", "tagsync-log4j", "usersync-log4j");

1. The Expected Response is as follows.

+-----------+----------------+--------------+---------+----------------------------------------+-------------+
| config_id | type_name | service_name | version | note | max_version |
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
| 232 | kms-log4j | RANGER_KMS | 4 | Enabling Kerberos for added components | 4 |
| 218 | admin-log4j | RANGER | 4 | | 4 |
| 228 | tagsync-log4j | RANGER | 4 | | 4 |
| 229 | usersync-log4j | RANGER | 4 | | 4 |
+-----------+----------------+--------------+---------+----------------------------------------+-------------+

1. Update the required configurations.

• Ranger-admin service: Update admin-log4j to admin-logback

UPDATE ambari.clusterconfig
SET type_name = "admin-logback"
WHERE config_id = 218;

• Ranger tagsync service: Update tagsync-log4j to tagsync-logback

UPDATE ambari.clusterconfig
SET type_name = "tagsync-logback"
WHERE config_id = 228;

• Ranger usersync service: Update usersync-log4j to usersync-logback

UPDATE ambari.clusterconfig
SET type_name = "usersync-logback"
WHERE config_id = 229;

1. Restart the Ambari-server.

ambari-server restart

Issue 2: Ranger Admin service start

2024-10-06 11:19:26,414 [I] java patch PatchForOzoneServiceDefConfigUpdate_J10051 is being applied..
ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML document. See previously reported errors.
at ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:70)
at ch.qos.logback.core.joran.GenericXMLConfigurator.populateSaxEventRecorder(GenericXMLConfigurator.java:184)
at ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:158)
at ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:122)
at ch.qos.logback.core.joran.GenericXMLConfigurator.doConfigure(GenericXMLConfigurator.java:65)
at ch.qos.logback.classic.util.DefaultJoranConfigurator.configureByResource(DefaultJoranConfigurator.java:68)
at ch.qos.logback.classic.util.DefaultJoranConfigurator.configure(DefaultJoranConfigurator.java:35)
at ch.qos.logback.classic.util.ContextInitializer.invokeConfigure(ContextInitializer.java:128)
at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:103)
at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:66)
at ch.qos.logback.classic.spi.LogbackServiceProvider.initializeLoggerContext(LogbackServiceProvider.java:52)
at ch.qos.logback.classic.spi.LogbackServiceProvider.initialize(LogbackServiceProvider.java:41)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:196)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:183)
at org.slf4j.LoggerFactory.getProvider(LoggerFactory.java:486)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:472)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:421)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:447)
at org.apache.ranger.patch.BaseLoader.<clinit>(BaseLoader.java:40)
Caused by: org.xml.sax.SAXParseException; systemId: file:/usr/odp/current/ranger-admin/ews/webapp/WEB-INF/classes/conf/logback.xml; lineNumber: 2; columnNumber: 1; Content is not allowed in prolog.
at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1243)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635)
at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324)
at ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents(SaxEventRecorder.java:64)
... 18 more
Solution:
Log on to the Ambari UI → ranger → configs → Advanced → Advanced admin-logback
Replace the admin-logback-xml template with the one below.

<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<appender name="xa_log_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger-admin-${hostname}-${user}.log</file>
<append>true</append>
<encoder>
<pattern>%date [%thread] %level{5} [%file:%line] %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger-admin-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="sql_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger_admin_sql.log</file>
<append>true</append>
<encoder>
<pattern>%d [%t] %-5p %C{6} \(%F:%L\) %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_admin_sql.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="perf_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger_admin_perf.log</file>
<append>true</append>
<encoder>
<pattern>%d [%t] %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_admin_perf.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="patch_logger" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<append>true</append>
<file>{{admin_log_dir}}/ranger_db_patch.log</file>
<encoder>
<pattern>%d [%t] %-5p %C{6} \(%F:%L\) %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_db_patch.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<logger name="xa" additivity="false" level="info">
<appender-ref ref="xa_log_appender"/>
</logger>
<logger name="jdbc.connection" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="com.mchange" additivity="false" level="warn" />
<logger name="org.apache.ranger.perf" additivity="false" level="info">
<appender-ref ref="perf_appender"/>
</logger>
<logger name="jdbc.audit" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.apache.ranger.patch" additivity="false" level="info">
<appender-ref ref="patch_logger"/>
</logger>
<logger name="jdbc.resultset" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.springframework" additivity="false" level="warn">
<appender-ref ref="patch_logger"/>
</logger>
<logger name="jdbc.sqltiming" additivity="false" level="warn">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.hibernate.SQL" additivity="false" level="warn">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.apache.ranger" additivity="false" level="info">
<appender-ref ref="xa_log_appender"/>
</logger>
<logger name="jdbc.sqlonly" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<root level="warn">
<appender-ref ref="xa_log_appender"/>
</root>
</configuration>
You can now proceed with the Ranger service restart from the Ambari UI. It must start smoothly.

Ranger KMS

During the cluster upgrade (after Ambari is upgraded to 2.7.9.0 but before the ODP stack is upgraded), the Ranger-KMS service might not start due to the following error:

raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'kms-logback' was not found in configurations dictionary!
Kms-log4j is replaced with kms-logback starting from version 3.2.3.x.
Solution:
Log in to the backend database for the Ambari server.
In this case, MySQL is the backend database for the Ambari server.

select * from (SELECT
cc.config_id,
cc.type_name,
service_name,
sc.version,
sc.note,
(SELECT MAX(version) FROM ambari.serviceconfig s
WHERE s.service_name = sc.service_name) AS max_version
FROM
ambari.serviceconfig sc
INNER JOIN
ambari.serviceconfigmapping scm
ON
sc.service_config_id = scm.service_config_id
INNER JOIN
ambari.clusterconfig cc
ON
cc.config_id = scm.config_id
WHERE
sc.service_name IN ("RANGER_KMS", "RANGER") and sc.version=(SELECT MAX(version) FROM ambari.serviceconfig s
WHERE s.service_name = sc.service_name)) as res where type_name in ("kms-log4j", "admin-log4j", "tagsync-log4j", "usersync-log4j");

• Response:

+-----------+----------------+--------------+---------+----------------------------------------+-------------+
| config_id | type_name | service_name | version | note | max_version |
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
| 232 | kms-log4j | RANGER_KMS | 4 | Enabling Kerberos for added components | 4 |
| 218 | admin-log4j | RANGER | 4 | | 4 |
| 228 | tagsync-log4j | RANGER | 4 | | 4 |
| 229 | usersync-log4j | RANGER | 4 | | 4 |
+-----------+----------------+--------------+---------+----------------------------------------+-------------+
Ranger-Admin Service Required Change
Update kms-log4j to kms-logback:

UPDATE ambari.clusterconfig
SET type_name = "kms-logback"
WHERE config_id = 232;
From the Ambari-Server host, perform the following steps:

1. Edit the File:

Path: /var/lib/ambari-server/resources/stacks/ODP/3.0/services/RANGER_KMS/package/scripts/kms.py

Locate the following lines:

File(os.path.join(params.kms_conf_dir, "kms-log4j.properties"),
owner=params.kms_user,
group=params.kms_group,
content=InlineTemplate(params.kms_log4j),
mode=0644
)
Comment them out as shown below:

"""
File(os.path.join(params.kms_conf_dir, "kms-log4j.properties"),
owner=params.kms_user,
group=params.kms_group,
content=InlineTemplate(params.kms_log4j),
mode=0644
)
"""

1. Restart the Ambari-Server.

ambari-server restart
You can now proceed with restarting the Ranger-KMS service from the Ambari UI. It must start smoothly.

Update the Ranger Configuration

1. Log in to the Ambari UI and navigate to RANGER → CONFIGS -> ADVANCED -> Advanced admin-logback, update the following content, and save.

<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<appender name="xa_log_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger-admin-${hostname}-${user}.log</file>
<append>true</append>
<encoder>
<pattern>%date [%thread] %level{5} [%file:%line] %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger-admin-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="sql_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger_admin_sql.log</file>
<append>true</append>
<encoder>
<pattern>%d [%t] %-5p %C{6} \(%F:%L\) %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_admin_sql.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="perf_appender" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{admin_log_dir}}/ranger_admin_perf.log</file>
<append>true</append>
<encoder>
<pattern>%d [%t] %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_admin_perf.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="patch_logger" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<append>true</append>
<file>{{admin_log_dir}}/ranger_db_patch.log</file>
<encoder>
<pattern>%d [%t] %-5p %C{6} \(%F:%L\) %msg%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{admin_log_dir}}/ranger_db_patch.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<logger name="xa" additivity="false" level="info">
<appender-ref ref="xa_log_appender"/>
</logger>
<logger name="jdbc.connection" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="com.mchange" additivity="false" level="warn" />
<logger name="org.apache.ranger.perf" additivity="false" level="info">
<appender-ref ref="perf_appender"/>
</logger>
<logger name="jdbc.audit" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.apache.ranger.patch" additivity="false" level="info">
<appender-ref ref="patch_logger"/>
</logger>
<logger name="jdbc.resultset" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.springframework" additivity="false" level="warn">
<appender-ref ref="patch_logger"/>
</logger>
<logger name="jdbc.sqltiming" additivity="false" level="warn">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.hibernate.SQL" additivity="false" level="warn">
<appender-ref ref="sql_appender"/>
</logger>
<logger name="org.apache.ranger" additivity="false" level="info">
<appender-ref ref="xa_log_appender"/>
</logger>
<logger name="jdbc.sqlonly" additivity="false" level="error">
<appender-ref ref="sql_appender"/>
</logger>
<root level="warn">
<appender-ref ref="xa_log_appender"/>
</root>
</configuration>

1. On the Ambari UI, navigate to RANGER → CONFIGS -> ADVANCED -> Advanced tagsync-logback, update with the following content, and save.

<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<appender name="logFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{tagsync_log_dir}}/tagsync.log</file>
<encoder>
<pattern>%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %L %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{tagsync_log_dir}}/tagsync.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<root level="info">
<appender-ref ref="logFile"/>
</root>
</configuration>

1. On the Ambari UI, navigate to RANGER → CONFIGS -> ADVANCED -> Advanced usersync-logback, update with the following content, and save.

<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<appender name="logFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<file>{{usersync_log_dir}}/usersync-${hostname}-${user}.log</file>
<encoder>
<pattern>%d{dd MMM yyyy HH:mm:ss} %5p %c{1} [%t] - %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>{{usersync_log_dir}}/usersync-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<root level="info">
<appender-ref ref="logFile"/>
</root>
</configuration>

1. Restart the Ranger service from the Ambari UI.

Update the Ranger KMS Configuration

1. Log in to the Ambari UI and navigate to RANGER → CONFIGS -> ADVANCED -> Advanced admin-logback, update the following content, and save.

<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration scan="true">
<appender name="kms-audit" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<Append>true</Append>
<File>${kms.log.dir}/kms-audit-${hostname}-${user}.log</File>
<encoder>
<pattern>%d{ISO8601} %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${kms.log.dir}/kms-audit-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<appender name="kms-metric" class="ch.qos.logback.core.FileAppender">
<!--See also http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<Append>false</Append>
<File>${kms.log.dir}/ranger_kms_metric_data_for_${metric.type}.log</File>
<encoder>
<pattern>%m%n</pattern>
</encoder>
</appender>
<appender name="kms" class="ch.qos.logback.core.rolling.RollingFileAppender">
<!--See http://logback.qos.ch/manual/appenders.html#RollingFileAppender-->
<!--and http://logback.qos.ch/manual/appenders.html#TimeBasedRollingPolicy-->
<!--for further documentation-->
<File>${kms.log.dir}/ranger-kms-${hostname}-${user}.log</File>
<Append>true</Append>
<encoder>
<pattern>%d{ISO8601} %-5p [%t] %c{1} \(%F:%L\) - %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${kms.log.dir}/ranger-kms-${hostname}-${user}.log.%d{yyyy-MM-dd}</fileNamePattern>
<maxHistory>15</maxHistory>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
</appender>
<logger name="com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator" level="OFF"/>
<logger name="kms-audit" additivity="false" level="INFO">
<appender-ref ref="kms-audit"/>
</logger>
<logger name="org.apache.hadoop" level="INFO"/>
<logger name="org.apache.hadoop.conf" level="INFO"/>
<logger name="org.apache.hadoop.crypto.key.kms.server.KMSMetricUtil" level="INFO">
<appender-ref ref="kms-metric"/>
</logger>
<logger name="org.apache.ranger" level="INFO"/>
<root level="WARN">
<appender-ref ref="kms"/>
</root>
</configuration>

1. Restart the Ranger KMS service from the Ambari UI.

HDFS

Name node or any service fails - rangerlookup not found

2024-09-25 12:38:32,687 - get_user_call_output returned (0, '{"statusCode":400,"msgDesc":"Operation denied. User name: rangerlookup specified in policy does not exist in ranger admin."}', ' % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1918 0 124 100 1794 2883 41720 --:--:-- --:--:-- --:--:-- 44604')
2024-09-25 12:41:28,288 - get_user_call_output returned (0, '{"statusCode":400,"msgDesc":"Operation denied. User name: rangerlookup specified in policy does not exist in ranger admin."}', ' % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 780 0 124 100 656 1675 8864 --:--:-- --:--:-- --:--:-- 10540')
Solution:
Add rangerlookup in the Ranger users list.

Data Node Start Issue

In upgraded clusters, the DataNodes fail to start and hang forever without much logging:

2024-10-07 15:14:58,346 - Writing File['/usr/odp/3.3.6.0-1/hadoop/conf/hadoop-metrics2.properties'] because contents don't match
2024-10-07 15:14:58,346 - File['/usr/odp/3.3.6.0-1/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0o755}
2024-10-07 15:14:58,346 - File['/usr/odp/3.3.6.0-1/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2024-10-07 15:14:58,347 - File['/etc/hadoop/conf/topology_mappings.data'] {'content': Template('topology_mappings.data.j2'), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0o644, 'only_if': 'test -d /etc/hadoop/conf'}
2024-10-07 15:14:58,349 - Writing File['/etc/hadoop/conf/topology_mappings.data'] because contents don't match
2024-10-07 15:14:58,349 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'mode': 0o755, 'only_if': 'test -d /etc/hadoop/conf'}
2024-10-07 15:14:58,351 - Skipping unlimited key JCE policy check and setup since the Java VM is not managed by Ambari
2024-10-07 15:14:58,605 - Using hadoop conf dir: /usr/odp/3.3.6.0-1/hadoop/conf
2024-10-07 15:14:58,605 - Stack Feature Version Info: Cluster Stack=3.3, Command Stack=None, Command Version=3.3.6.0-1, Upgrade Direction=upgrade -> 3.3.6.0-1
2024-10-07 15:14:58,611 - Using hadoop conf dir: /usr/odp/3.3.6.0-1/hadoop/conf
There will be no more logging after that, will be timed out later.
Solution:
Log on to the Ambari UI → HDFS → configs → Advanced → custom hdfs-rbf-site

dfs.federation.router.rpc-address=0.0.0.0:20010
Complete the configurations template for custom hdfs-rbf-site.

DFSRouter component configs list

If the router component is to be used in the ODP cluster, add the below configurations.

Log on to the Ambari UI → HDFS → configs → Advanced → custom hdfs-rbf-site.

dfs.federation.router.admin-address=0.0.0.0:8111
dfs.federation.router.admin-bind-host=0.0.0.0
dfs.federation.router.admin.enable=true
dfs.federation.router.admin.handler.count=1
dfs.federation.router.cache.ttl=1m
dfs.federation.router.client.allow-partial-listing=true
dfs.federation.router.client.mount-status.time-out=1s
dfs.federation.router.client.reject.overload=false
dfs.federation.router.client.retry.max.attempts=3
dfs.federation.router.client.thread-size=32
dfs.federation.router.connect.max.retries.on.timeouts=0
dfs.federation.router.connect.timeout=2s
dfs.federation.router.connection.clean.ms=10000
dfs.federation.router.connection.creator.queue-size=100
dfs.federation.router.connection.min-active-ratio=0.5f
dfs.federation.router.connection.pool-size=1
dfs.federation.router.connection.pool.clean.ms=60000
dfs.federation.router.dn-report.cache-expire=10s
dfs.federation.router.dn-report.time-out=1000
dfs.federation.router.file.resolver.client.class=org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver
dfs.federation.router.handler.count=10
dfs.federation.router.handler.queue.size=100
dfs.federation.router.heartbeat-state.interval=5s
dfs.federation.router.heartbeat.enable=true
dfs.federation.router.heartbeat.interval=5000
dfs.federation.router.http-address=0.0.0.0:50071
dfs.federation.router.http-bind-host=0.0.0.0
dfs.federation.router.http.enable=true
dfs.federation.router.https-address=0.0.0.0:50072
dfs.federation.router.https-bind-host=0.0.0.0
dfs.federation.router.metrics.class=org.apache.hadoop.hdfs.server.federation.metrics.FederationRPCPerformanceMonitor
dfs.federation.router.metrics.enable=true
dfs.federation.router.monitor.localnamenode.enable=true
dfs.federation.router.mount-table.cache.enable=true
dfs.federation.router.mount-table.cache.update=true
dfs.federation.router.mount-table.cache.update.client.max.time=5m
dfs.federation.router.mount-table.cache.update.timeout=1m
dfs.federation.router.mount-table.max-cache-size=10000
dfs.federation.router.namenode.heartbeat.enable=true
dfs.federation.router.namenode.resolver.client.class=org.apache.hadoop.hdfs.server.federation.resolver.MembershipNamenodeResolver
dfs.federation.router.quota-cache.update.interval=60s
dfs.federation.router.quota.enable=false
dfs.federation.router.reader.count=1
dfs.federation.router.reader.queue.size=100
dfs.federation.router.rpc-address=0.0.0.0:20010
dfs.federation.router.rpc-bind-host=0.0.0.0
dfs.federation.router.rpc.enable=true
dfs.federation.router.safemode.enable=true
dfs.federation.router.safemode.expiration=3m
dfs.federation.router.safemode.extension=30s
dfs.federation.router.secret.manager.class=org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl
dfs.federation.router.store.connection.test=60000
dfs.federation.router.store.driver.class=org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreZooKeeperImpl
dfs.federation.router.store.enable=true
dfs.federation.router.store.membership.expiration=300000
dfs.federation.router.store.membership.expiration.deletion=-1
dfs.federation.router.store.router.expiration=5m
dfs.federation.router.store.router.expiration.deletion=-1
dfs.federation.router.store.serializer=org.apache.hadoop.hdfs.server.federation.store.driver.impl.StateStoreSerializerPBImpl

Node manager start failed

plete.
2024-09-25 12:00:41,360 ERROR nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(965)) - Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:482)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:758)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:327)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:494)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:962)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1042)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
at org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.getInstance(AuxiliaryServiceWithCustomClassLoader.java:165)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.createAuxServiceFromLocalClasspath(AuxServices.java:242)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.createAuxService(AuxServices.java:271)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.initAuxService(AuxServices.java:452)
Solution:

• Remove the following from yarn configurations:
  • spark_shuffle from yarn.nodemanager.aux-services
• Remove the following from Custom yarn-site:

yarn.nodemanager.aux-services.spark2_shuffle.class
yarn.nodemanager.aux-services.spark2_shuffle.classpath
yarn.nodemanager.aux-services.spark_shuffle.classpath
yarn.nodemanager.aux-services.spark_shuffle.class

Kafka

After completing the Ambari upgrade, during the express upgrade, Kafka needs to be up and running in a healthy mode. However, Kafka might be in a stopped state due to the following errors:

raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'kafka-connect-distributed' was not found in configurations dictionary!
These issues occur because Kafka Connect and MirrorMaker were added in this release.
Solution
To start Kafka, add the following configurations in Ambari:

1. Custom kafka-mirrormaker2-common
   • replication.factor =1
2. Custom kafka-mirrormaker2-destination
   • replication.enabled=true
   • destination.cluster.name=dest
   • topics.whitelist=.*
3. Custom kafka-mirrormaker2-source
   • replication.enabled=true
   • source.cluster.name=source
   • topics.whitelist=.*
4. Custom kafka-connect-distributed
   • bootstrap.servers=localhost:6667

Proceed with restarting the Kafka service. It must start smoothly.