Acceldata
 ODP
Search documentation… ⌘K

Configure Storage

Local Disk Storage

# Multiple disks with disk type annotation
celeborn.worker.storage.dirs=/mnt/nvme1:disktype=SSD,/mnt/nvme2:disktype=SSD,/mnt/hdd1:disktype=HDD
celeborn.storage.availableTypes=SSD,HDD

HDFS Storage

The Real-World Use Case for HDFS in Celeborn

HDFS storage in Celeborn is designed for:

  1. Disk exhaustion fallback - When local disk is full, new files go to HDFS
  2. Not for primary tiered storage - The current architecture doesn't support MEMORY → HDFS eviction when local disk is available
celeborn.storage.availableTypes=HDFS
celeborn.storage.hdfs.dir=hdfs://namenode:8020/celeborn
# Kerberos authentication for HDFS
celeborn.storage.hdfs.kerberos.principal=celeborn/_HOST@REALM
celeborn.storage.hdfs.kerberos.keytab=/etc/security/keytabs/celeborn.keytab
# HDFS performance tuning (buffer must be >= 4 MB)
celeborn.worker.flusher.hdfs.buffer.size=4m
celeborn.worker.sortPartition.threads=64
celeborn.worker.commitFiles.threads=128
celeborn.rpc.askTimeout=240s

S3 / Object Storage

celeborn.storage.availableTypes=S3
celeborn.storage.s3.dir=s3a://bucket-name/celeborn
celeborn.storage.s3.endpoint.region=us-east-1
# S3 performance (larger buffer recommended)
celeborn.worker.flusher.s3.buffer.size=6m
celeborn.worker.flusher.s3.threads=8
For S3-compatible storage (Ceph, VAST, MinIO), configure the S3A endpoint in Hadoop core-site.xml:

<property>
<name>fs.s3a.endpoint</name>
<value>http://your-s3-endpoint:port</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<value>true</value>
</property>

Tiered Storage

Configure eviction policies for hybrid SSD-to-HDFS tiering:

celeborn.storage.availableTypes=SSD,HDFS
celeborn.worker.storage.storagePolicy.createFilePolicy=SSD,HDFS
celeborn.worker.storage.storagePolicy.evictPolicy=SSD|HDFS