YARN Spark Shuffle Services

Update the YARN NodeManager auxiliary service configuration to use the Spark 3.5.5 shuffle service.

1. In the Ambari UI, go to YARN > Configs > Custom yarn-site.
2. Locate the yarn.nodemanager.aux-services property.
3. Replace spark3_shuffle with spark_shuffle_355.
   

   yarn.nodemanager.aux-services=spark_shuffle_355

4. If Spark 3.3.3 or Spark 3.5.1 services are installed during the upgrade, add the corresponding shuffle services:
   • Spark 3.3.3: spark_shuffle_333
   • Spark 3.5.1: spark_shuffle_351
5. Save the configuration.
6. Restart the affected NodeManagers.