Troubleshooting

Master Not Starting

# Check master logs
 
tail -f /var/log/celeborn/celeborn-master-*.log

Common causes:

• Port conflict — ensure ports 9097, 9098, and 9872 are available
• Java heap too small — increase CELEBORN_MASTER_MEMORY
• Ratis storage permission issue — check /var/lib/celeborn/ratis ownership

Worker Not Registering

# Check worker logs
 
tail -f /var/log/celeborn/celeborn-worker-*.log

Common causes:

• Cannot connect to master — verify network connectivity to port 9097
• Storage directory permission issues — ensure celeborn user has write access
• Disk full or unhealthy — check disk health and free space

Shuffle Failures

# Increase RPC and commit timeouts
celeborn.rpc.askTimeout=300s
celeborn.worker.commitFiles.timeout=300s
 
# Enable replication to prevent data loss on worker failure
spark.celeborn.client.push.replicate.enabled=true

Diagnostic REST API Commands

# Check master status and HA role
curl http://master:9098/api/v1/masters
 
# List all workers and their health status
curl http://master:9098/api/v1/workers
 
# List active shuffles
curl http://master:9098/api/v1/shuffles
 
# Health check
curl http://master:9098/ping

Log Locations