Worker Tags

Worker tags allow fine-grained routing of shuffle jobs to specific worker subsets. Common use cases include:

• Resource Isolation: Dedicate workers to specific tenants or applications
• Hardware Segmentation: Route jobs to SSD-equipped nodes vs. HDD nodes
• Environment Separation: Separate production and staging worker pools
• Rolling Upgrades: Tag workers by version for controlled, safe upgrades

Enable Worker Tags

celeborn.tags.enabled=true

Tag Expression Syntax

# AND condition: all tags must match
env=production,region=us-east,high-io
 
# Advanced TagsQL mode
 
celeborn.tags.useTagsQL=true
env:production region:{us-east,us-west} env:!sandbox

dynamicConfig.yaml — Tag Example

- level: SYSTEM
 
 config:
 
 celeborn.tags.preferClientTagsExpr: false
 
 celeborn.tags.tagsExpr: 'env=production'
 
 tags:
 
 env=production:
 
 - 'worker1:9094'
 
 - 'worker2:9094'
 
 env=staging:
 
 - 'worker3:9094'
 
 high-io:
 
 - 'worker1:9094'
 
 
 
- tenantId: analytics
 
 level: TENANT
 
 config:
 
 celeborn.tags.tagsExpr: 'env=production,high-io'