Apache Airflow Logging Guide

Overview

Apache Airflow 2.8.1 generates logs for various components such as the Scheduler, Webserver, and Worker processes. These logs are essential for debugging and monitoring the system.

Log Storage Location

The Airflow logs are stored under the following directory:

_var_log_airflow_logs_
To list the logs available, use:

ls _var_log_airflow_logs_
This directory contains logs for various DAGs, the scheduler, and the webserver.

Scheduler Logs

The Airflow scheduler logs can be found in:

_var_log_airflow_logs_scheduler_
The logs are organized by date, with the latest logs available under:

_var_log_airflow_logs_scheduler_latest_
To inspect a specific log file, navigate to the corresponding date:

ls _var_log_airflow_logs_scheduler_2025-02-19
Example output:

clean_up.py.log

Webserver Logs

The Airflow webserver logs can be found in:

_var_log_airflow_logs_webserver-access.log
_var_log_airflow_logs_webserver-error.log
To check the last 100 lines of the webserver logs, use:

tail -n 100 _var_log_airflow_logs_webserver-access.log
For errors, check:

tail -n 100 _var_log_airflow_logs_webserver-error.log

DAG Logs

Each DAG execution logs its runs under:

_var_log_airflow_logs_dag_id=<dag_id>_run_id=<run_id>_task_id=<task_id>_attempt=<attempt>.log
Example:

ls _var_log_airflow_logs_dag_id=dataset_consumes_1_run_id=manual__2025-02-14T08:04:06.983781+00:00_task_id=consuming_1_
Output:

attempt=1.log
To view the logs of a task attempt:

cat _var_log_airflow_logs_dag_id=dataset_consumes_1_run_id=manual__2025-02-14T08:04:06.983781+00:00_task_id=consuming_1_attempt=1.log

Debugging with Systemd

Since Airflow components are managed using systemd, you can check the status and logs using:

Webserver

Check the webserver status:

systemctl status airflow-webserver
View the last 100 logs:

journalctl -u airflow-webserver -n 100 --no-pager

Scheduler

Check the scheduler status:

systemctl status airflow-webserver
View the last 100 logs:

journalctl -u airflow-webserver -n 100 --no-pager

Worker (if applicable)

If using Celery workers, check their status:

systemctl status airflow-worker
View the last 100 logs:

journalctl -u airflow-worker -n 100 --no-pager