Airflow Migration Tutorial#

This tutorial demonstrates using dagster-airlift to migrate an Airflow DAG to Dagster.

Using dagster-airlift we can

Observe Airflow DAGs and their execution history with no changes to Airflow code
Model and observe assets orchestrated by Airflow with no changes to Airflow code
Enable a migration process that
- Can be done task-by-task in any order with minimal coordination
- Has task-by-task rollback to reduce risk
- That retains Airflow DAG structure and execution history during the migration

Process#

This is a high level overview of the steps to migrate an Airflow DAG to Dagster:

Peer
- Observe an Airflow instance from within a Dagster Deployment via the Airflow REST API.
- This loads every Airflow DAG as an asset definition and creates a sensor that polls Airflow for execution history.
Observe
- Add a mapping that maps the Airflow DAG and task id to a basket of definitions that you want to observe. (e.g. render the full lineage the dbt models an Airflow task orchestrates)
- The sensor used for peering also polls for task execution history, and adds materializations to an observed asset when its corresponding task successfully executes
Migrate
- Selectively move execution of Airflow tasks to Dagster Software Defined Assets