Acceldata
 ODP
Search documentation… ⌘K

Tez and MapReduce Service Checks Fail After ODP Upgrade

After upgrading an ODP cluster, Tez and MapReduce service checks may fail with errors indicating that YARN cannot download required application resources from HDFS.

This issue typically occurs when the Tez or MapReduce application archives stored in HDFS are missing, corrupted, or have a file size of zero bytes.

Symptoms

Tez Service Check Failure

Failed to download resource
hdfs:///odp/apps/<version>/tez/tez.tar.gz
org.apache.hadoop.yarn.exceptions.YarnException:
Download and unpack failed

MapReduce Service Check Failure

Failed to download resource
hdfs:///odp/apps/<version>/mapreduce/mapreduce.tar.gz
org.apache.hadoop.yarn.exceptions.YarnException:
Download and unpack failed
Applications may repeatedly fail with:

AM Container exited with exitCode: -1000

Cause

Tez and MapReduce rely on application archives stored in HDFS under:

/odp/apps/<version>/tez/
/odp/apps/<version>/mapreduce/
During service startup or service checks, YARN distributes these archives to containers.
If the archive:
  • Does not exist
  • Is corrupted
  • Has a size of 0 bytes

YARN cannot unpack the application resources and the service check fails.

Verify the Issue

Check the file sizes:

hdfs dfs -ls /odp/apps/<version>/tez
hdfs dfs -ls /odp/apps/<version>/mapreduce
Example of a problematic archive:

-rw-r--r-- 0 hdfs hdfs 0 tez.tar.gz
-rw-r--r-- 0 hdfs hdfs 0 mapreduce.tar.gz

Resolution

Restore MapReduce Archive

Copy the archive from a healthy local installation:

hdfs dfs -put -f \
/usr/odp/<version>/hadoop/mapreduce.tar.gz \
/odp/apps/<version>/mapreduce/

Restore Tez Archive

hdfs dfs -put -f \
/usr/odp/<version>/tez/tez.tar.gz \
/odp/apps/<version>/tez/
Verify:

hdfs dfs -ls /odp/apps/<version>/tez
hdfs dfs -ls /odp/apps/<version>/mapreduce
Confirm the files are no longer zero bytes.

Validation

Test Tez

Run the Tez service check from Ambari.

Test MapReduce

yarn jar \
/usr/odp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar \
pi 2 10
Both service checks should complete successfully.

Best Practices

  • Verify /odp/apps/<version>/ contents immediately after upgrade.
  • Validate archive file sizes before running service checks.
  • Ensure HDFS permissions remain intact.
  • Keep a backup of application archives prior to major upgrades.