AWS Step Functions

AWS Step Functions is a serverless workflow orchestration service native to AWS, used to coordinate tasks such as AWS Lambda functions, API calls, and other AWS services into a single workflow, defined as a state machine. ADOC supports AWS Step Functions as a pipeline source, so you can observe Step Function executions in ADOC alongside your other pipeline sources such as Airflow, Azure Data Factory, and dbt.

AWS Step Functions are defined using the Amazon States Language (ASL), in either JSONPath or JSONata query language. For the complete language reference, see the Amazon States Language specification.

How the integration works

ADOC observes a Step Function by listening for the events AWS Step Functions emits to Amazon EventBridge when an execution starts and when it reaches a terminal state:

• An Amazon EventBridge rule listens for Step Functions execution events.
• The rule's target is an AWS Lambda function that processes these events and emits OpenLineage events to your ADOC Control Plane.
• When a Step Function execution starts, the corresponding OpenLineage event is sent to ADOC.
• When the execution reaches a terminal state (SUCCESS, ABORTED, or FAILED), the Lambda function processes the full execution details and emits the corresponding OpenLineage events.

Setting up the integration

Configuring this integration requires access to your AWS account to create the following resources.

• Create an Amazon EventBridge rule that listens for AWS Step Functions execution status events. You can scope the rule to specific state machines using their ARN. For example:

{
"source": ["aws.states"],
"detail-type": ["Step Functions Execution Status Change"],
"detail": {
"stateMachineArn": [
"arn:aws:states:ap-south-1:241571801844:stateMachine:TriggerStateMachine"
]
}
}

• Set the rule's target to an AWS Lambda function. This function uses boilerplate code (Python) to invoke the AWS State Machine OpenLineage Adapter SDK, maintained by Acceldata. The SDK is installed as a Lambda layer; the Lambda function itself only invokes the public methods the SDK exposes.
• Configure the following environment variables on the Lambda function:

Viewing Step Function Pipelines in ADOC

Once configured, a Step Function appears in ADOC as a pipeline, and its states appear as job nodes on the pipeline canvas — the same way tasks appear for other pipeline sources.

Map and Parallel states use ADOC's nested job model: clicking a Map or Parallel node opens a nested canvas showing its sub-states, with breadcrumb navigation back to the parent state machine (for example, TriggerStateMachine > MyMap > MyParallel).

You can filter pipelines by source on the Pipelines page: select AWS Step Functions from the Pipeline Source filter, alongside other supported sources such as Airflow, Autosys, Azure Data Factory, dbt, and Fivetran.

Additional execution metadata is surfaced through span events, giving you more context on each state's execution within the pipeline.

Linking Parent and Child Step Functions

When one Step Function triggers another — either through a direct API invocation or through a Step Functions StartExecution state within a state machine definition — you can link the two executions into a single connected pipeline in ADOC by passing parent run context as input to the triggered (child) Step Function.

JSONata state machines:

Json

"aws.states.openlineage.parentRunId": "{% $states.context.Execution.Name %}",

"aws.states.openlineage.parentJobName": "{% $states.context.State.Name & '.' & $states.context.StateMachine.Name %}",

"aws.states.openlineage.parentJobNamespace": ""

JSONPath state machines:

Json

"aws.states.openlineage.parentRunId.$": "$$.Execution.Name",

"aws.states.openlineage.parentJobName.$": "States.Format('{}:{}', $$.State.Name, $$.StateMachine.Name)",

"aws.states.openlineage.parentJobNamespace.$": ""

The parentJobNamespace value must match the OpenLineage Namespace configured on your Lambda function.

Once this convention is followed, ADOC links the parent and child Step Functions into a single connected pipeline view. This is the same pattern used to link Airflow to downstream Spark or dbt runs.

Viewing Dataset-Level Lineage

To see the datasets read and written by a Step Function's states, upload a JSON mapping file to Amazon S3 describing each state's inputs and outputs, and grant your Lambda function read access to that S3 location.

Json

{

  "": {

    "state_name": "",

    "inputs": [{"namespace": "snowflake://{organization name}-{account name}", "name": "{database}.{schema}.{table}"}],

    "outputs": [{"namespace": "mysql://{host}:{port}", "name": "{database}.{table}"}]

  }

}

ADOC correlates the dataset names in this file with assets already onboarded in the Catalog.

Retries

ADOC applies the same retry-visualization model used for other orchestrators, such as Airflow. When a state retries, its job node appears once on the pipeline canvas; each retry attempt appears as a separate span, with the retry attempt number appended to the span name (for example, transform_task, then transform_task-1, transform_task-2).

Limitations

• Only Standard Step Functions are supported. Express Step Functions are not supported.
• The pipeline view is not a live reflection of an in-progress execution. It updates once the Step Function execution reaches a terminal state (SUCCESS, ABORTED, or FAILED).
• ADOC Pipeline observability does not support pipeline run-level retries. AWS Step Functions' native Redrive capability, which resumes an execution from its point of failure, is not reflected in ADOC.

What's next

• [Pipelines] – Learn about pipeline observability in ADOC.
• [Pipeline Run Details] – View job nodes, timelines, and automations for a pipeline run.
• [Lineage] – Explore lineage across systems and assets.