For a decade, the industry has been trapped in the YAML-CI Paradox: we use a static, untyped configuration language to define some of the most complex, dynamic, and critical logic in our software delivery lifecycle. Whether it is GitHub Actions, GitLab CI, or CircleCI, the result is the same — a fragmented mess of shell scripts wrapped in proprietary YAML tags, impossible to test locally without a runner, and prone to "it works in the cloud" failures.

Dagger breaks this cycle by treating the CI pipeline as a first-class software artifact. Instead of defining a sequence of steps in YAML, Dagger allows you to define your pipeline in a general-purpose language (Go, TypeScript, Python) that communicates with a vertically integrated engine via a GraphQL API. This enables the pipeline to be portable, typed, and locally executable.

The Architectural Blueprint: The Dagger Engine

At its core, Dagger is not a CI provider; it is a CI Engine. It abstracts the container runtime, the scheduler, and the cache into a single execution model. The shift from Dagger v0 to v1.0 introduces the Module Architecture, which replaces custom SDK scripts with a standardized, discoverable system of functional blocks.

The Dagger execution flow follows a strict request-response cycle:


+-----------------+       +-------------------+       +-----------------------+
|  Dagger SDK      | ----> | GraphQL API       | ----> | Dagger Engine        |
| (TS / Go / Py)   |       | (Request DAG)     |       | (Theseus Scheduler)   |
+-----------------+       +-------------------+       +-----------+-----------+
                                                                  |
                                                                  v
                                                      +-----------------------+
                                                      | Container Sandbox      |
                                                      | (BuildKit / Theseus)   |
                                                      +-----------+-----------+
                                                                  |
                                                                  v
                                                      +-----------------------+
                                                      | Global Content Cache  |
                                                      | (Layer-based / Remote) |
                                                      +-----------------------+

1. The SDK Layer: The developer writes code in their language of choice. The SDK does not execute the build; it constructs a Directed Acyclic Graph (DAG) of operations. For example, a call to container.From("node").WithExec(["npm install"]) simply adds a node to the graph.

2. The GraphQL API: The SDK sends this graph to the Dagger Engine. This allows the engine to optimize the execution, prune unused branches, and verify if a specific node's result is already present in the cache.

3. The Engine (Theseus): Dagger's internal runtime, Theseus, manages the lifecycle of the sandboxes. It replaces the traditional BuildKit approach with a more integrated model that couples the scheduler and the container runtime, reducing the latency of "cold starts" for individual pipeline steps.

Implementing the Module Pattern

The Module Pattern is the recommended way to build production-grade pipelines. A module is essentially a typed API for your infrastructure. Instead of every project rewriting the "Build and Push" logic, the platform team creates a build-module that exports a single function: deploy(source: Directory).

Below is a production-ready implementation of a Dagger module using the TypeScript SDK. This blueprint handles source ingestion, linting via an official module, building a production image, and pushing to a registry.


import { Module, Container, Directory, Secret } from "@dagger.io/dagger";

export default class PipelineModule extends Module {
  /**
   * The primary entry point for the project's CI pipeline.
   * Orchestrates linting, testing, and deployment.
   */
  async deploy(source: Directory, regToken: Secret): Promise<void> {
    // 1. Setup the base environment
    const base = Container.from("node:20-alpine")
      .withDirectory("/src", source)
      .withWorkdir("/src");

    // 2. Execute Linting (Using the native Dagger 'check' logic)
    // This step is cached unless source files change.
    await base
      .withExec(["npm", "install"])
      .withExec(["npm", "run", "lint"])
      .stdout();

    // 3. Build the Production Image
    const prodImage = base
      .withExec(["npm", "run", "build"])
      .withExec(["npm", "prune", "--production"]);

    // 4. Push to Registry
    // Dagger handles the authentication and layer upload automatically.
    await prodImage
      .publish("ghcr.io/my-org/my-app:latest", {
        token: regToken,
      });
  }
}

To execute this blueprint, the developer simply runs:


# Install the module
dagger module install ./my-pipeline-module

# Run the deploy function
dagger call deploy source=. regToken=env:GITHUB_TOKEN

Dagger vs. Traditional YAML CI

Feature GitHub Actions / GitLab CI Dagger (Module Pattern)
Configuration Proprietary YAML / Secret-leaking shell Typed TS/Go/Python (Strongly Typed)
Local Dev "Push and Pray" (Wait for Runner) dagger call (Same as CI)
Caching Manual Keys (actions/cache) Automatic, Per-Node (DAG-based)
Portability Locked to Provider (Runner API) Engine-Agnostic (Runs anywhere)
Testing CI-Run only Unit testable via standard SDK tests

Production Realities & Operational Toil

While Dagger solves the "YAML Hell" problem, it introduces its own set of production challenges that architects must account for:

  • The Container-in-Container Tax: Because Dagger executes every operation in a sandbox, there is a non-trivial overhead for very small tasks. For a simple "echo" command, the overhead of the Dagger Engine managing the sandbox is higher than a raw shell script.
  • Engine Memory Leaks: In long-running self-hosted engines, the accumulation of session schema servers can lead to heap growth. While fixed in v0.21.11, monitoring the dagger-engine memory usage is mandatory for platform teams.
  • The "Dagger Lock" Friction: Dagger uses a dagger.lock file to ensure module version consistency. In large monorepos, frequent updates to shared modules can lead to merge conflicts in the lock file, similar to package-lock.json.
  • Remote Cache Latency: If using a remote engine, the latency of sending the GraphQL DAG is negligible, but pulling large build contexts (directories) from the local machine to the remote engine can become a bottleneck if not optimized via .daggerignore.

Who Should Skip Dagger?

Dagger is a powerhouse for complex delivery cycles, but it is overkill for:

  • Trivial Pipelines: If your CI is just npm install && npm test, sticking to a simple GitHub Action is more efficient.
  • Resource-Constrained Runners: Dagger requires a running engine (daemon). If your CI runners have extremely tight memory limits (e.g., < 2GB), the engine's overhead may be prohibitive.
  • Teams Avoiding New Runtimes: If your organization has a strict policy against introducing new binary dependencies (the Dagger CLI) into the build environment.

References & Further Reading