Worker Tags

Worker tags allow fine-grained routing of shuffle jobs to specific worker subsets. Common use cases include:

• Resource Isolation: Dedicate workers to specific tenants or applications
• Hardware Segmentation: Route jobs to SSD-equipped nodes vs. HDD nodes
• Environment Separation: Separate production and staging worker pools
• Rolling Upgrades: Tag workers by version for controlled, safe upgrades

Enable Worker Tags

celeborn.tags.enabled=true

Tag Expression Syntax

# AND condition: all tags must match
env=production,region=us-east,high-io
# Advanced TagsQL mode
celeborn.tags.useTagsQL=true
env:production region:{us-east,us-west} env:!sandbox

dynamicConfig.yaml — Tag Example

- level: SYSTEM
config:
celeborn.tags.preferClientTagsExpr: false
celeborn.tags.tagsExpr: 'env=production'
tags:
env=production:
- 'worker1:9094'
- 'worker2:9094'
env=staging:
- 'worker3:9094'
high-io:
- 'worker1:9094'
- tenantId: analytics
level: TENANT
config:
celeborn.tags.tagsExpr: 'env=production,high-io'