Deploy Celeborn with Ambari Mpack
Install the Celeborn Mpack
Run the following commands on the Ambari server host:
Step 1: Stop Ambari Server
ambari-server stop
Step 2: Install the Celeborn mpack
Download the Celeborn mpack from the Mpacks Link.
ambari-server install-mpack --mpack=/path/to/celeborn-ambari-mpack-0.6.2.tar.gz
Step 3: Restart Ambari Server
ambari-server restart
Add Celeborn Service via Ambari UI
- Navigate to Ambari UI → Services → Add Service
- Select Celeborn from the service list and click Next
- Assign hosts for each component:
- Celeborn Master: 1 host (non-HA) or 3 hosts (HA mode)
- Celeborn Worker: 3 or more worker hosts
- Celeborn Client: All nodes running Spark or Flink executors
- Configure properties (see Section 6) and deploy
- Verify via Master Web UI: http://<master-host>:9098
Basic Ambari Configuration
celeborn-defaults
Property | Default | Description |
celeborn.master.port | 9097 | Master RPC port |
celeborn.master.http.port | 9098 | Master HTTP / UI port |
celeborn.worker.storage.dirs | /var/lib/celeborn/shuffle | Worker shuffle storage directories |
celeborn.metrics.enabled | true | Enable Prometheus metrics collection |
celeborn-env
Variable | Default | Description |
celeborn_master_memory | 1024 MB | Master JVM heap size |
celeborn_worker_memory | 1024 MB | Worker JVM heap size |
celeborn_worker_offheap_memory | 4096 MB | Worker off-heap (direct) memory |