Ask AI

You are viewing an unreleased or outdated version of the documentation

View Latest Documentation

Airlift#

Airlift is a toolkit for integrating Dagster and Airflow.

  • Observe Airflow instances from within Dagster
  • Accelerate the migration of Airflow DAGs to Dagster assets with opinionated tooling.

Compatibility#

REST API Availability#

Airlift depends on the availability of Airflow’s REST API. Airflow’s REST API was made stable in its 2.0 release (Dec 2020) and was introduced experimentally in 1.10 in August 2018. Currently Airflow requires the availability of the REST API.

  • OSS: Stable as of 2.00
  • MWAA
    • Note: only available in Airflow 2.4.3 or later on MWAA.
  • Cloud Composer: No limitations as far as we know.
  • Astronomer: No limitations as far as we know.

Airflow Migration Tutorial#

In this tutorial, we'll use dagster-airlift to migrate an Airflow DAG to Dagster assets.

By the end of the tutorial, you'll understand how to use dagster-airlift to enable a migration process that

  • Can be done task-by-task in any order with minimal coordination
  • Has task-by-task rollback to reduce risk
  • Retains Airflow DAG structure and execution history during the migration

Click here to get started.

Airflow Federation Tutorial#

In this tutorial, we'll use dagster-airlift to observe DAGs from multiple Airflow instances, and federate execution between them using Dagster as a centralized control plane.

Click here to get started.

Airlift Operator Migation Reference#

In this reference, we'll explain how to migrate common Airflow operators to Dagster.

Click here to get started.

References#