OpenLineage Integration

ADOC supports OpenLineage as a standard for bringing pipeline observability into your environment without instrumenting your jobs with an SDK. This page explains what OpenLineage is, how ADOC interprets OpenLineage events, and how pipeline and asset metadata is derived from them.

ADOC currently supports the following pipeline orchestrators through OpenLineage:

• Airflow
• Spark
• dbt Core

What is OpenLineage

OpenLineage is an open framework for collecting and analyzing data lineage. It defines a standard for recording job execution metadata and specifies three core concepts:

• Run: A specific instance of a job execution.
• Job: A process that performs extract, transform, or load (ETL) operations.
• Dataset: A static data source or destination that a job reads from or writes to.

OpenLineage provides a uniform way to describe these objects and supports adding custom metadata through Facets, which define metadata schemas for a Run, Job, or Dataset. OpenLineage includes a set of default facets, and you can define custom facets as part of your OpenLineage setup.

How ADOC interprets OpenLineage events

ADOC maps each OpenLineage core object to a platform entity:

The Run and Job objects vary depending on the orchestrator that emits them:

When a run starts in a supported orchestrator — for example, triggering the DAG Finance_HR_Argentina in Airflow — ADOC creates a pipeline and begins tracking it. Subsequent runs of the same DAG are recorded as pipeline runs under the existing pipeline rather than creating new pipelines each time.

OpenLineage naming convention

In OpenLineage, every entity — jobs, datasets, and so on — is organized within its own namespace. Understanding this convention is useful when filtering or searching for pipelines in ADOC by namespace or job name.

• Airflow DAG: Uniquely identified as {namespace}.{Airflow DAG Name}
• Task within a DAG: Identified as {namespace}.{Airflow DAG Name}.{task id}
• Dataset: Follows source-specific naming conventions. For example, a BigQuery table is identified as bigquery://{project id}.{dataset name}.{table name}

For the full list of naming conventions by source, see OpenLineage naming conventions.

Asset correlation

When an OpenLineage event includes dataset metadata, ADOC checks whether that dataset already exists as an asset in the Data Catalog. If a match is found, ADOC links the pipeline run to the corresponding asset, so the dataset appears in lineage and reliability views alongside the rest of your pipeline.

Asset correlation is currently supported for datasets from the following sources:

• AWS S3
• BigQuery
• GCS
• Snowflake
• Redshift
• Glue
• Pub/Sub
• Iceberg
• PostgreSQL

Datasets from other sources still appear in pipeline runs and lineage graphs, but are not automatically correlated to an existing Catalog asset.

Pipeline tags

When ADOC receives OpenLineage events from Airflow or Spark pipelines, it adds the tags in those events to the matching pipeline. Tags often arrive as single strings, such as Region:US or billing, so ADOC uses the Pipeline Tag Ingestion settings to map each tag to a key and a value.

How tags are mapped

When Tag Key-Value Parsing is on, ADOC maps tags as follows. The examples use the default separator (:) and default key (source_tags).

When Tag Key-Value Parsing is off, ADOC uses the whole tag as both the key and the value. For example, Region:US appears as Region:US: Region:US.

Configure tag mapping

The settings apply across your tenant. To change them, you need permission to create or edit pipelines.

1. In the left navigation, click Settings.
2. On the ADOC Settings tab, under Pipelines, click Pipeline Tag Ingestion.
3. Configure the following settings, and then click Save Changes.

To restore a setting to its default value, click Reset to default next to it, and then click Save Changes.

Changes apply from the next pipeline run. For example, if you change Tag Default Key to app_tags, the next run adds the tag billing as app_tags: billing.

Tag updates on each run

On each run, ADOC updates the pipeline's tags to match the tags in the OpenLineage events. If a tag is removed from the source pipeline, or no longer matches because the settings changed, ADOC removes it from the pipeline. The tag still exists in ADOC, but it's no longer attached to the pipeline.

View, filter, and add tags

To see a pipeline's tags, use the Tags column on the Pipelines page. To filter pipelines by tag, select More Filters > Tags.

You can also attach tags to a pipeline manually. Create the tag in Data Observability Cloud > Governance > Tags, and then edit the pipeline to attach it. For more information, see Tags.

What's next

• To instrument Airflow or Python pipelines directly with the Acceldata SDK instead of OpenLineage, see Airflow and Python in the Developer Guide.

• To view and manage pipelines created through OpenLineage, see Working with Pipelines.
• To understand core pipeline concepts such as runs, spans, and jobs, see Pipeline Observability.
• To explore column-level lineage across your data assets, see Lineage.