Troubleshooting

Master Not Starting

# Check master logs
tail -f /var/log/celeborn/celeborn-master-*.log
Common causes:

• Port conflict — ensure ports 9097, 9098, and 9872 are available
• Java heap too small — increase CELEBORN_MASTER_MEMORY
• Ratis storage permission issue — check /var/lib/celeborn/ratis ownership

Worker Not Registering

# Check worker logs
tail -f /var/log/celeborn/celeborn-worker-*.log
Common causes:

• Cannot connect to master — verify network connectivity to port 9097
• Storage directory permission issues — ensure celeborn user has write access
• Disk full or unhealthy — check disk health and free space

Shuffle Failures

# Increase RPC and commit timeouts
celeborn.rpc.askTimeout=300s
celeborn.worker.commitFiles.timeout=300s
# Enable replication to prevent data loss on worker failure
spark.celeborn.client.push.replicate.enabled=true

Diagnostic REST API Commands

# Check master status and HA role
curl http://master:9098/api/v1/masters
# List all workers and their health status
curl http://master:9098/api/v1/workers
# List active shuffles
curl http://master:9098/api/v1/shuffles
# Health check
curl http://master:9098/ping

Log Locations