Airflow Dashboard
You can use the following Airflow metrics displayed on the Airflow > Dashboard page to analyze them.
Metric | Submetric | Description |
|---|---|---|
Airflow Dag Processing Stats | Average Parse Time | The average amount of time taken to parse each file. |
Files to be Scanned | The total number of files that need to be processed. | |
Filesystem Updates | Any changes or updates made to the file system during the parsing process. | |
Parse Errors | The number of errors encountered while parsing files. | |
Airflow Dag Run Stats | Average Dag Dependency Check | The average time taken to check dependencies between tasks in a DAG. |
Average Dag Success Rate | The average percentage of DAGs that are completed successfully. | |
Average Dag Failed State | The average number of DAGs that enter a failed state. | |
Average Dag Schedule Delay | The average delay experienced in starting DAGs according to their scheduled time. | |
Airflow Dataset Stats | Orphaned Datasets | The datasets that are no longer associated with any Active DAGs. |
Dags Triggered | Tracks the number of times a DAG has been initiated for execution, indicating the frequency of workflow runs. | |
Dataset Updates | The dataset updates are changes to datasets that DAGs can depend on. | |
Airflow Pool Stats | Deferred Pool Stats | Number of tasks waiting for external conditions to proceed. |
Open Pool Stats | Available capacity in a pool for additional tasks. | |
Queued Pool Stats | Number of tasks waiting in the queue for execution. | |
Running Pool Stats | Number of tasks currently executing in the pool. | |
Starving Pool Stats | Number of tasks unable to run due to a lack of available resources. | |
Airflow Task Finish Instance Stats | Failure | The Task execution failed. |
Success | The task was completed successfully. | |
Deferred | The task is waiting for an external trigger to continue. | |
Running | The task is currently executing. | |
Queued | The task is waiting in line for execution. | |
Removed | The task was removed before execution. | |
Scheduled | The task is planned for execution at a later time. | |
Skipped | The task was intentionally not executed. | |
Rescheduled | The task was postponed to run at a different time. | |
Upstream | The task is dependent on upstream tasks before it can run. | |
Rescheduled | The task was postponed to run at a different time. | |
Upstream | The task is dependent on upstream tasks before it can run. | |
Airflow Executor Stats | DAGs | Directed Acyclic Graphs that define workflows in Airflow. |
Executor Open Slots | Available slots in the executor for running new tasks. | |
Airflow Dag Time Stats | Average Dag Queue Time | The average time a DAG spends waiting in the queue before starting. |
Average Dag Run Time | The average time it takes for a DAG to complete its execution. | |
Airflow Scheduler Stats | Time Spent on Critical Section | Time spent by the scheduler in critical sections of code. |
Orphaned Tasks Adopted | Number of orphaned tasks that the scheduler has taken over. | |
Orphaned Tasks Cleared | Number of orphaned tasks that have been removed by the scheduler. | |
Loop Duration | Time taken for one complete cycle of the scheduler's loop. | |
Task Executable | Number of tasks determined to be ready for execution by the scheduler. | |
Task Starving | Number of tasks that are unable to execute due to resource constraints. |