Ask AI

You are viewing an unreleased or outdated version of the documentation

View Latest Documentation

Orchestration on Docker

APIs

dagster_docker.DockerRunLauncher RunLauncher[source]

Config Schema:
image (dagster.StringSource, optional):

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:
url (dagster.StringSource):

username (dagster.StringSource):

password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

Launches runs in a Docker container.

dagster_docker.docker_executor ExecutorDefinition[source]

Config Schema:
image (dagster.StringSource, optional):

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:
url (dagster.StringSource):

username (dagster.StringSource):

password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

retries (selector, optional):

Whether retries are enabled or not. By default, retries are enabled.

Default Value:
{
    "enabled": {}
}
Config Schema:
enabled (strict dict, optional):
Default Value:
{}
disabled (strict dict, optional):
Default Value:
{}
max_concurrent (dagster.IntSource, optional):

Limit on the number of containers that will run concurrently within the scope of a Dagster run. Note that this limit is per run, not global.

tag_concurrency_limits (List[strict dict], optional):

A set of limits that are applied to steps with particular tags. If a value is set, the limit is applied to only that key-value pair. If no value is set, the limit is applied across all values of that key. If the value is set to a dict with applyLimitPerUniqueValue: true, the limit will apply to the number of unique values for that key. Note that these limits are per run, not global.

experimental This API may break in future versions, even between dot releases.

Executor which launches steps as Docker containers.

To use the docker_executor, set it as the executor_def when defining a job:

from dagster_docker import docker_executor

from dagster import job

@job(executor_def=docker_executor)
def docker_job():
    pass

Then you can configure the executor with run config as follows:

execution:
  config:
    registry: ...
    network: ...
    networks: ...
    container_kwargs: ...

If you’re using the DockerRunLauncher, configuration set on the containers created by the run launcher will also be set on the containers that are created for each step.

Ops

dagster_docker.docker_container_op = <dagster._core.definitions.op_definition.OpDefinition object>[source]

Config Schema:
image (dagster.StringSource):

The image in which to run the Docker container.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:
url (dagster.StringSource):

username (dagster.StringSource):

password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

entrypoint (List[String], optional):

The ENTRYPOINT for the Docker container

command (List[String], optional):

The command to run in the container within the launched Docker container.

experimental This API may break in future versions, even between dot releases.

An op that runs a Docker container using the docker Python API.

Contrast with the docker_executor, which runs each Dagster op in a Dagster job in its own Docker container.

This op may be useful when:

  • You need to orchestrate a command that isn’t a Dagster op (or isn’t written in Python)

  • You want to run the rest of a Dagster job using a specific executor, and only a single op in docker.

For example:

from dagster_docker import docker_container_op

from dagster import job

first_op = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["echo HELLO"],
    },
    name="first_op",
)
second_op = docker_container_op.configured(
    {
        "image": "busybox",
        "command": ["echo GOODBYE"],
    },
    name="second_op",
)

@job
def full_job():
    second_op(first_op())

You can create your own op with the same implementation by calling the execute_docker_container function inside your own op.

dagster_docker.execute_docker_container(context, image, entrypoint=None, command=None, networks=None, registry=None, env_vars=None, container_kwargs=None)[source]

experimental This API may break in future versions, even between dot releases.

This function is a utility for executing a Docker container from within a Dagster op.

Parameters:

  • image (str) – The image to use for the launched Docker container.

  • entrypoint (Optional[Sequence[str]]) – The ENTRYPOINT to run in the launched Docker container. Default: None.

  • command (Optional[Sequence[str]]) – The CMD to run in the launched Docker container. Default: None.

  • networks (Optional[Sequence[str]]) – Names of the Docker networks to which to connect the launched container. Default: None.

  • registry – (Optional[Mapping[str, str]]): Information for using a non local/public Docker registry. Can have “url”, “username”, or “password” keys.

  • env_vars (Optional[Sequence[str]]) – List of environemnt variables to include in the launched container. ach can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the calling environment.

  • container_kwargs (Optional[Dict[str[Any]]]) – key-value pairs that can be passed into containers.create in the Docker Python API. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

Pipes

class dagster_docker.PipesDockerClient(env=None, registry=None, context_injector=None, message_reader=None)[source]

experimental This API may break in future versions, even between dot releases.

A pipes client that runs external processes in docker containers.

By default context is injected via environment variables and messages are parsed out of the log stream, with other logs forwarded to stdout of the orchestration process.

Parameters:

  • env (Optional[Mapping[str, str]]) – An optional dict of environment variables to pass to the container.

  • register (Optional[Mapping[str, str]]) – An optional dict of registry credentials to login to the docker client.

  • context_injector (Optional[PipesContextInjector]) – A context injector to use to inject context into the docker container process. Defaults to PipesEnvContextInjector.

  • message_reader (Optional[PipesMessageReader]) – A message reader to use to read messages from the docker container process. Defaults to DockerLogsMessageReader.

class dagster_docker.PipesDockerLogsMessageReader(*args, **kwargs)[source]

experimental This API may break in future versions, even between dot releases.