Operator migration guides: Migrating usage of BashOperator for dbt#

In this page, we'll explain migrating an Airflow BashOperator that runs a dbt command to Dagster.

Background#

In Airflow, you might have a BashOperator that runs a dbt command. For example, you might have a task that runs dbt run to build your dbt models.

from airflow.operators.bash import BashOperator

run_dbt_model = BashOperator(task_id="build_dbt_models", bash_command="dbt run")

Dagster equivalent#

The Dagster equivalent is to instead use the dagster-dbt library to run commands against your dbt project. Here would be the equivalent code in Dagster:

from dagster_dbt import DbtCliResource, DbtProject, dbt_assets

from dagster import AssetExecutionContext

project = DbtProject(project_dir="path/to/dbt_project")


@dbt_assets(manifest=project.manifest_path, project=project)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["run"], context=context).stream()

Migrating the operator#

Migrating the operator breaks down into a few steps:

1. Making the dbt project available to both your Airflow and Dagster deployments.
2. Writing a @dbt_asset-decorated function which runs your dbt commands.
3. Using dagster-airlift to proxy execution of the original task to Dagster.

Step 1: Making the dbt project available & building manifest#

First, you'll need to make the dbt project available to the Dagster runtime and build the manifest.

• If you're building your Dagster deployment in a monorepo alongside your dbt and Airflow projects, you can follow this guide: Monorepo setup.
• If you're deploying within a separate repository, you can follow this guide: Separate repository setup.

Step 2: Writing a @dbt_asset-decorated function#

Once your dbt project is available, you can write a function that runs your dbt commands using the dbt_assets decorator and DbtCliResource. Most dbt CLI commands and flags are supported - to see a full guide on using dbt_assets, check out the dagster-dbt guide.

Step 3: Using dagster-airlift to proxy execution#

Finally, you can use dagster-airlift to proxy the execution of the original task to Dagster. The dagster-airlift migration guide details this process.