Dagster & dbt quickstart#

Select one of the following to load your dbt project into Dagster:

• Option 1: Create a Dagster project in a single file
• Option 2: Create a new Dagster project that wraps a dbt project by using the dagster-dbt command line interface (CLI)
• Option 3: Use an existing Dagster project

Option 1: Create a single Dagster file#

Running your dbt project with Dagster can be easily done after creating a single file. For this example, let's consider a basic use case - say you want to represent your dbt models as Dagster assets and run them daily at midnight.

With your text editor of choice, create a Python file in the same directory as your dbt project directory and add the following code. Note that since this file contains all Dagster definitions required for your code location, it is recommended to name it definitions.py.

The following code assumes that your Python file and dbt project directory are adjacent in the same directory. If that's not the case, make sure to update the RELATIVE_PATH_TO_MY_DBT_PROJECT constant so that it points to your dbt project.

from pathlib import Path

from dagster import AssetExecutionContext, Definitions
from dagster_dbt import (
    DbtCliResource,
    DbtProject,
    build_schedule_from_dbt_selection,
    dbt_assets,
)

RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my_dbt_project"

my_project = DbtProject(
    project_dir=Path(__file__)
    .joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
    .resolve(),
)
my_project.prepare_if_dev()


@dbt_assets(manifest=my_project.manifest_path)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()


my_schedule = build_schedule_from_dbt_selection(
    [my_dbt_assets],
    job_name="materialize_dbt_models",
    cron_schedule="0 0 * * *",
    dbt_select="fqn:*",
)

defs = Definitions(
    assets=[my_dbt_assets],
    schedules=[my_schedule],
    resources={
        "dbt": DbtCliResource(project_dir=my_project),
    },
)

Option 2: Create a new Dagster project#

This approach uses the dagster-dbt CLI to create a new Dagster project and wrap it around a dbt project. Running this command requires two arguments:

• --project-name - The name of the Dagster project to be created. In our example, this will be my_dagster_project.
• --dbt-project-dir - The path to the dbt project. In our example, this will be ./my_dbt_project, which means our current location is in the directory where my_dbt_project is located.

In our example, our command would look like this:

dagster-dbt project scaffold --project-name my_dagster_project --dbt-project-dir ./my_dbt_project

This command will create a new directory called my_dagster_project/ inside the current directory. The new my_dagster_project/ directory will contain a set of files that define a Dagster project to load the dbt project provided in ./my_dbt_project.

Option 3: Use an existing Dagster project#

Existing Dagster projects can also be used to run a dbt project. To do this, you'll need to:

1. Use the dagster-dbt library to add DbtProject and @dbt_assets objects
2. Add the new objects to your Dagster project's Definitions object

Note: This example assumes that your existing Dagster project includes both assets.py and definitions.py files, among other required files like setup.py and pyproject.toml. For example, your project might look like this:

my_dagster_project
├── __init__.py
├── assets.py
├── definitions.py
├── pyproject.toml
├── setup.cfg
└── setup.py

1. Change directories to the Dagster project directory:

   cd my_dagster_project/
   

2. Create a Python file named project.py and add the following code:

   from pathlib import Path
   
   from dagster_dbt import DbtProject
   
   RELATIVE_PATH_TO_MY_DBT_PROJECT = "./my_dbt_project"
   
   my_project = DbtProject(
       project_dir=Path(__file__)
       .joinpath("..", RELATIVE_PATH_TO_MY_DBT_PROJECT)
       .resolve(),
   )
   my_project.prepare_if_dev()
   

   The DbtProject object is a representation of the dbt project that assists with manifest.json preparation.

3. In your project's assets.py file, add the following code:

   from dagster import AssetExecutionContext
   from dagster_dbt import DbtCliResource, dbt_assets
   
   from .project import my_project
   
   @dbt_assets(manifest=my_project.manifest_path)
   def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
       yield from dbt.cli(["build"], context=context).stream()
   

   The @dbt_assets decorator allows Dagster to create a definition for how to compute a set of dbt resources, described by a manifest.json.

4. In your project's definitions.py file, update the Definitions object to include the newly created objects:

   from dagster import Definitions
   from dagster_dbt import DbtCliResource
   
   from .assets import my_dbt_assets
   from .project import my_project
   
   defs = Definitions(
       ...,
       assets=[
           ...,
           # Add the dbt assets alongside your other asset
           my_dbt_assets,
       ],
       resources={
           ...: ...,
           # Add the dbt resource alongside your other resources
           "dbt": DbtCliResource(project_dir=my_project),
       },
   )
   

With these changes, your existing Dagster project is ready to run your dbt project.

How you start the UI depends on which approach you took to load your dbt project into Dagster:

• If you created a single Dagster file, use Option 1
• If you created a new Dagster project or used an existing project, use Option 2

Option 1: From a Dagster file#

1. Locate the Dagster file containing your definitions. If you created a single Dagster file in the previous section (Option 1), this file will be definitions.py.

2. To start Dagster's UI, run the following:

   dagster dev -f definitions.py
   

   Which will result in output similar to:

   Serving dagster-webserver on http://127.0.0.1:3000 in process 70635
   

Option 2: From a Dagster project#

1. Change directories to the Dagster project directory:

   cd my_dagster_project/
   

2. To start Dagster's UI, run the following:

   dagster dev
   

   Which will result in output similar to:

   Serving dagster-webserver on http://127.0.0.1:3000 in process 70635