Capacity Planning
Hardware Resource Ratio
Recommended balanced ratio per worker node for optimal Celeborn performance:
Resource | Ratio |
|---|---|
vCPUs | 2 |
Memory (GB) | 5 |
Network (Gbps) | 2 |
Disk I/O (KIOps) | 1 |
Worker Sizing Formula
Worker Count Calculation
Formula: Worker Count = (Max Concurrent Shuffle Size × 2) / Node Usable Disk Space
Example: Max concurrent shuffle = 1 TB, Node disk = 500 GB
→ Workers needed = (1,000 GB × 2) / 500 GB = 4 workers
Memory Recommendations by Workload
Workload Profile | Worker Heap | Worker Off-Heap |
|---|---|---|
Light (< 100 GB shuffle) | 2 GB | 4 GB |
Medium (100 GB – 1 TB) | 4 GB | 8 GB |
Heavy (> 1 TB) | 8 GB | 16 GB |
HDFS / S3 backend | 8 GB | 32 GB |
Storage Backend Recommendations
Storage Type | Performance | Notes |
|---|---|---|
NVMe SSD | Best | Lowest latency — recommended for hot shuffle data |
SAS SSD | Good | Good balance of cost and performance |
HDD | Acceptable | Use multiple disks per worker for parallelism |
HDFS | Moderate | Higher latency but infinite capacity — ideal for cold/large shuffles |
S3 / Object Store | Lower | Highest latency, lowest cost — use for overflow or archival |