Prerequisites for Celeborn
System Requirements
Component | Minimum | Recommended |
|---|---|---|
Master Nodes | 1 | 3 (for HA) |
Worker Nodes | 1 | 3 or more |
CPU per Worker | 4 cores | 8–16 cores |
Memory per Worker | 4 GB heap + 4 GB off-heap | 8 GB heap + 16 GB off-heap |
Disk per Worker | 100 GB SSD | 500 GB+ SSD / NVMe |
Network | 1 Gbps | 10 Gbps |
Software Requirements
- ODP Stack 3.3.6.x or higher
- Ambari 2.7.x or higher
- Java 8 or Java 11
- Hadoop 3.x (required for HDFS storage backend)
Network Ports
Port | Service | Protocol | Description |
|---|---|---|---|
9097 | Master RPC | TCP | Client / Worker to Master communication |
9098 | Master HTTP | TCP | Web UI and REST API |
9872 | Ratis | TCP | Master-to-Master Raft consensus |
9094 | Worker RPC | TCP | Client to Worker data transfer |
9096 | Worker HTTP | TCP | Worker Web UI and Prometheus metrics |
19097 | Master Internal | TCP | Internal port when auth is enabled |