Terraform State Management at Scale: Backends, Workspaces, and the Terragrunt Question

Terraform state is the most consequential database most platform teams operate without thinking of it as one. It is a JSON file that maps every resource instance in your configuration to a real object in a cloud API, and every plan you have ever run started by refreshing it against reality. Lose it and Terraform no longer knows which load balancer is yours; corrupt it and the next apply is an unguided missile. The backend you choose, the way you partition state across teams, and the locking discipline you enforce are not configuration details — they are the blast-radius architecture of your infrastructure practice.

Three things make this the right week to re-run the math. First, the S3 backend's DynamoDB-based locking — the pattern a decade of tutorials taught you to copy — is now formally deprecated: the current Terraform documentation states that DynamoDB-based locking "will be removed in a future minor version," and in the v1.17.1 source the dynamodb_table argument is marked Deprecated: true, emitting a deprecation diagnostic that points you at use_lockfile. Second, Terraform 1.17.0 (October 21, 2026) shipped -minimal-refresh, a planning option that only refreshes resources with proposed changes — a direct attack on the O(N) refresh tax that dominates plan time in large states. Third, the state orchestration layer is moving again: Terragrunt 1.2.0 (October 21, 2026) graduated eleven experiments including dependency-fetch-output-from-state, and OpenTofu 1.13.9 (November 5, 2026) carries native state and plan encryption. The defaults changed under everyone who set this up in 2022 and stopped reading changelogs.

This guide is the mechanics-first version: exactly what the lock does at the S3 API level, exactly what workspaces write to your bucket, exactly what leaks into state as plaintext, and exactly what Terragrunt adds — with the honest counter-arguments. One conclusion up front, because it frames everything: the dollar cost of storing state is a rounding error; the real costs are lock wedges, refresh time, coupling, and the secrets sitting in plaintext in an S3 object. Architecture accordingly.

What state actually is (and why it is load-bearing)

Terraform's own definition is precise and worth internalizing: state stores bindings between objects in a remote system and resource instances declared in your configuration, and the tool relies on it to know which real-world objects it created, to track metadata, and "to improve performance for large infrastructures" — the cache of record that lets Terraform plan without re-describing every object in the target account every time. Before any operation, Terraform runs a refresh to update the state against the real infrastructure; the plan you read is the diff between refreshed state and configuration. Locally, each workspace's state lives in terraform.tfstate with a one-generation backup at terraform.tfstate.backup. In a remote backend, state is held in memory only while an operation uses it — which is why a compromised bucket, not a compromised laptop, is the threat model that matters.

Two properties drive every scaling decision that follows:

• State is a JSON text file. Terraform's docs are blunt: do not edit it directly; the terraform state subcommands exist precisely so the CLI "insulates users from any format changes within the state itself" — the format can shift between versions while the CLI keeps working. Every hack that greps and rewrites the file by hand is one version bump away from disaster.
• Terraform expects a one-to-one mapping between configured resource instances and remote objects. The moment two states believe they own the same object, or no state does, you have drift, double-management, or an orphan — and partitioning (Section 4) is the tool that keeps ownership boundaries clean.

  CONFIG (.tf files)          STATE (JSON)                PROVIDER APIS
  ┌────────────────┐    ┌───────────────────┐    ┌──────────────────────┐
  │  variables,    │    │  object bindings, │    │  VPCs, subnets,      │
  │  resources,    │    │  attributes,      │    │  IAM roles, DBs...   │
  │  modules       │    │  outputs          │    │                      │
  └───────┬────────┘    └─────────┬─────────┘    └──────────┬───────────┘
          │                       │                         │
          │   1. terraform plan   │                         │
          │   ┌───────────────────▼──────────┐              │
          │   │ LOCK the state (.tflock)     │◄─── write    │
          │   │ REFRESH state vs APIs ────────┼──────────────┤ read each object
          │   │ DIFF config vs refreshed      │              │
          │   │ UNLOCK (plan, no state write) │              │
          │   └───────────────────────────────┘              │
          │                       │                         │
          │   2. terraform apply  │                         │
          │   ┌───────────────────▼──────────┐              │
          │   │ LOCK the state (.tflock)     │              │
          │   │ REFRESH (optional: -refresh=false,
          │   │         1.17: -minimal-refresh)             │
          │   │ CREATE / UPDATE / DELETE ────┼──────────────┤ mutate
          │   │ WRITE new state to S3 ───────┤              │
          │   │ UNLOCK (delete .tflock)     │              │
          │   └──────────────────────────────┘              │
          ▼                       ▼                         ▼

The refresh step is the quiet scale problem. A plan against a state with thousands of resource instances issues provider reads for the objects it refreshes; the wall-clock time grows with your estate, not with your change. That is the tax -minimal-refresh (v1.17.0) exists to cut — only resources with proposed changes get refreshed — and it trades freshness for speed in a way Section 8 dissects, because a plan computed against stale attributes is a plan with stale diffs.

The lock, end to end: what actually happens in S3

State locking is automatic on every operation that could write state; you never see it in normal output. If acquiring the lock fails, Terraform does not continue. The -lock=false flag exists and the documentation explicitly does not recommend it. That is the entire user-facing contract — the interesting part is what the backend does underneath, because that is what you will be debugging at 2 a.m. when a CI runner died mid-apply.

Since v1.10.0 (November 27, 2024), the S3 backend supports native state locking. The implementation in internal/backend/remote-state/s3/client.go is worth reading because it is beautifully small: acquiring the lock is a single conditional PutObject against your state bucket, uploading the lock document to the state key plus the suffix .tflock with the S3 conditional-write header If-None-Match: *. If no .tflock object exists, the PUT succeeds and you hold the lock; if one exists, S3 rejects the conditional write and Terraform reports "Error acquiring the state lock" with the holder's identity. There is no lock service, no extra table, no daemon — the lock is an object, and S3's conditional-PUT semantics are the atomic compare-and-set.

The lock document itself is the marshaled statemgr.LockInfo struct — seven fields, straight from the v1.17.1 source:

{
  "ID": "4f4a2f1e-9d3c-4b7a-a1d2-8e6f5b3c2a19",
  "Operation": "OperationTypePlan",
  "Info": "",
  "Who": "ci-runner-07@ip-10-4-1-42",
  "Version": "1.17.1",
  "Created": "2026-11-08T19:20:14.22104658Z",
  "Path": "network/prod/us-east-1.tfstate"
}

Because it is a plain object in your bucket, the lock is inspectable with tools you already have — aws s3api get-object on the .tflock key tells you exactly who is holding the lock, from where, since when, and with which Terraform version. The lock object is written with content type application/json, a SHA-256 checksum, and the same server-side encryption settings as your state object, so an SSE-KMS backend config encrypts the lock like the state.

Release is the mirror path: Terraform GETs the .tflock object, verifies the lock ID matches the one it holds, and issues a DeleteObject. Which leads to the failure mode everyone meets eventually: the stale lock. A hard-killed CI runner (OOM, spot reclamation, a cancelled pipeline) can exit between the PUT and the DELETE, leaving a lock object with no live owner. Every subsequent run then fails to acquire, and someone on your team will be copy-pasting terraform force-unlock from a runbook.

The deprecation arc, precisely dated

The legacy mechanism deserves its exact history, because ten thousand tutorials still teach it as if it were the design. DynamoDB-based locking predates native locking: Terraform stored a lock row in a table you provisioned, keyed by the LockID partition key (a String), acquiring the lock with a conditional PutItem. When v1.10.0 introduced S3 native locking, the changelog was explicit about the migration path: when both are configured, "locks will be acquired from both sources," and even then it promised that "in a future minor release of Terraform the DynamoDB locking mechanism and associated arguments will be deprecated." That future has arrived: the current backend documentation states that DynamoDB-based locking "is deprecated and will be removed in a future minor version," and the v1.17.1 backend schema marks dynamodb_table and friends Deprecated: true with a diagnostic that tells you to use use_lockfile instead. One historical footnote matters for bucket design: v1.10.1 (December 4, 2024) fixed lock files not being writable to buckets with S3 Object Lock enabled — worth knowing before you conclude that Object Lock and state locking are incompatible (they are not, on any current release).

The double-locking design is also documented in the source: when both mechanisms are configured, both locks must be acquired ("design decision: both must succeed"), and if the DynamoDB lock fails after the file lock succeeded, Terraform releases the file lock before returning the error. The practical takeaway for anyone still running a hybrid config during migration: your failure surface is the union of both mechanisms, so finish the migration — use_lockfile = true, drop dynamodb_table — rather than living in the both-locked state permanently.

  RUNNER A                              S3 BUCKET
  ─────────                             ─────────────────────────────────────────────
  terraform plan
    │
    ├─► PutObject  key=network/prod/us-east-1.tfstate.tflock
    │               If-None-Match: *   (conditional create)
    │               Body: LockInfo JSON
    │   ✅ 200 OK  → RUNNER A holds the lock
    │
  RUNNER B (same moment)
    ├─► PutObject  same key, If-None-Match: *
    │   ❌ 412 Precondition Failed
    │   → "Error acquiring the state lock"
    │     Lock Info: ID, Operation, Who, Version, Created
    │
  RUNNER A
    ├─► ... plan finishes (no state write) ...
    ├─► GetObject  key=*.tflock      → verify lock ID matches
    └─► DeleteObject key=*.tflock    → lock released

  FAILURE: runner A dies between PUT and DELETE
    └─► stale .tflock object blocks all runs
        → terraform state pull > backup.tfstate
        → aws s3api get-object ... tflock   (confirm Who/Created)
        → terraform force-unlock <lock-ID>   (deletes the .tflock object)

force-unlock is a controlled corruption window

terraform force-unlock LOCK_ID removes the lock "for the defined configuration" without modifying infrastructure — the flag takes the unique lock ID that Terraform prints in the lock error (the same ID field from the lock document, deliberately acting as a nonce "ensuring that locks and unlocks target the correct lock"), and the -force variant skips the confirmation prompt. The documentation's own warning is the right one: if you unlock while someone else genuinely holds the lock, you have created multiple writers against one state file — two applies interleaving writes means the second write's state becomes authoritative and the first apply's changes are silently unrecorded. Use it only to release your own dead lock, and make the runbook prove the lock is dead first: check Created is old, Who matches a dead runner, and the runner's logs actually terminated before you type the command.

# 1. Read the lock document — who holds it, since when, which version?
aws s3api get-object \
  --bucket acme-terraform-state-123456789012 \
  --key network/prod/us-east-1.tfstate.tflock \
  /tmp/stale-lock.json
cat /tmp/stale-lock.json | jq

# 2. Prove the holder is gone: check the runner/CI logs for that "Who"
#    and confirm the job exited between lock acquisition and release.
#    If Created is minutes old and the job is still alive — do NOT unlock.

# 3. Back up current state before touching anything
terraform state pull > /tmp/state-backup-$(date +%Y%m%d-%H%M%S).tfstate

# 4. Release the dead lock using the exact ID from the lock document
terraform force-unlock 4f4a2f1e-9d3c-4b7a-a1d2-8e6f5b3c2a19

# 5. Verify the lock object is actually gone
aws s3api head-object \
  --bucket acme-terraform-state-123456789012 \
  --key network/prod/us-east-1.tfstate.tflock || echo "lock released"

The S3 backend at production grade

The backend block is where most of the sharp edges live, because it accepts far more than the four lines tutorials show. The required arguments are exactly three — bucket, key, region — and everything else is optional policy. A production-grade configuration uses that surface deliberately:

terraform {
  backend "s3" {
    # --- required ---
    bucket = "acme-terraform-state-123456789012"
    key    = "network/prod/us-east-1.tfstate"
    region = "us-east-1"

    # --- locking: S3-native (DynamoDB locking is deprecated) ---
    use_lockfile = true

    # --- encryption: SSE-KMS on both state and lock objects ---
    encrypt    = true
    kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/11111111-2222-3333-4444-555555555555"

    # --- resilience: longer backoff beats CI flakes on throttling ---
    max_retries = 15
    retry_mode  = "adaptive"

    # --- identity: the plan/apply role in the TARGET account,
    #     so state reads and resource mutations are separable ---
    assume_role = {
      role_arn     = "arn:aws:iam::987654321098:role/TerraformApplyRole"
      session_name = "tf-network-prod"
      duration     = "1h"
    }

    # --- guardrails: never silently apply into the wrong account ---
    allowed_account_ids = ["987654321098"]
  }
}

Notes on the pieces that bite people:

• use_lockfile defaults to false. The single most important line in any migration. Setting it to true while keeping dynamodb_table gives you the double-lock regime; the clean end-state is lockfile-only.
• KMS requires key-policy discipline. With kms_key_id set, Terraform needs kms:Encrypt, kms:Decrypt, and kms:GenerateDataKey on that key — for the state object, the lock object, and every runner identity that assumes the role. A customer-managed key runs $1.00 per month plus $0.03 per 10,000 requests (verified from the KMS pricing page example) — i.e., nothing; there is no cost argument for SSE-S3.
• sse_customer_key is a footgun by design. It accepts a base64 key that must decode to 256 bits for SSE-C, and the docs warn that setting it inside a .tf file causes the value to be "persisted to disk in terraform.tfstate" — use the AWS_SSE_CUSTOMER_KEY environment variable instead. SSE-C is for compliance shops that refuse KMS; for everyone else kms_key_id is the grown-up option.
• assume_role duration bounds are 15 minutes to 12 hours. Long plans in big accounts with slow providers: give the session headroom, because a mid-apply credential expiry is a bad afternoon. The backend also supports chained roles (multiple assume_role blocks are assumed in order) and OIDC-based assume_role_with_web_identity for keyless CI.
• allowed_account_ids / forbidden_account_ids exist precisely to prevent "destruction of a live environment" from the wrong account. Cheap insurance; take it.

The IAM story for the state bucket follows directly from the documented API surface. With lockfile locking, the runner role needs, at minimum: s3:ListBucket on the bucket, s3:GetObject and s3:PutObject on the state key, and s3:GetObject, s3:PutObject, s3:DeleteObject on the lock file path (...<key>.tflock) — plus the KMS permissions if SSE-KMS is on. Scoped to one state prefix, that policy is:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyThisStatePrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-terraform-state-123456789012",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["network/prod/*"]
        }
      }
    },
    {
      "Sid": "ReadWriteStateObject",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::acme-terraform-state-123456789012/network/prod/us-east-1.tfstate"
    },
    {
      "Sid": "AcquireAndReleaseLockfile",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::acme-terraform-state-123456789012/network/prod/us-east-1.tfstate.tflock"
    },
    {
      "Sid": "UseStateEncryptionKey",
      "Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/11111111-2222-3333-4444-555555555555"
    }
  ]
}

Read-only consumers — the terraform_remote_state data source, dashboards, audit tooling — get s3:GetObject on the state key only. That is the entire reader policy, and it is what makes state sharing safe by construction: a consumer with no PutObject cannot participate in writes no matter how badly its own configuration drifts.

Workspaces: a key-layout feature, not a tenancy model

Terraform workspaces multiply states for one configuration, and the S3 backend implements them as pure key layout. The default workspace writes to exactly <key>; every non-default workspace writes to <workspace_key_prefix>/<workspace_name>/<key>, where the prefix defaults to env: (configurable via workspace_key_prefix). Ten backends support multiple workspaces at all — AzureRM, Consul, COS, GCS, Kubernetes, Local, OSS, Postgres, Remote, and S3 — which means any tooling that assumes "workspace = separate S3 object" is backend-specific by construction.

acme-terraform-state-123456789012
├── network/terraform.tfstate                  # workspace "default"
├── network/terraform.tfstate.tflock           # its lock object
├── env:/dev/network/terraform.tfstate         # workspace "dev"
├── env:/dev/network/terraform.tfstate.tflock
├── env:/prod/network/terraform.tfstate        # workspace "prod"
└── env:/prod/network/terraform.tfstate.tflock

The documentation's own warning is the part teams skip and regret: workspaces are not appropriate for system decomposition or deployments requiring separate credentials and access controls. All workspaces share one backend configuration — same bucket, same role, same lock family. Workspaces are the right tool for "the same configuration, N short-lived instances" (a sandbox per developer with throwaway stacks, or ephemeral test environments); they are the wrong tool for "networking and database teams must not be able to touch each other's state," which needs separate keys plus separate IAM paths, or separate accounts outright.

The multi-account layout HashiCorp actually recommends

The backend documentation's multi-account section is unusually opinionated, and it is worth following closely: the infrastructure Terraform uses should exist outside the infrastructure Terraform manages. That means a dedicated administrative AWS account containing your human IAM users, and — critically — the state bucket itself, so that no apply in any workload account can ever damage the storage holding the state for every environment. Environment accounts then contain one or more IAM roles that the administrative account's users assume; the state bucket policy grants only the administrative account's principals, and assume_role in each root's backend block picks up the environment-specific role. The blast radius of a compromised workload account stops at its own resources: no path back to state.

  ADMIN ACCOUNT (123456789012)              ENV ACCOUNT PROD (987654321098)
  ┌───────────────────────────────┐          ┌─────────────────────────────┐
  │ IAM users / SSO identities   │          │                             │
  │ IAM group "platform-eng"     │  assume   │ TerraformApplyRole          │
  │        │                      ├─────────►│  trust: admin acct only     │
  │        │ sts:AssumeRole       │          │  scoped to env resources    │
  │        ▼                      │          │                             │
  │ S3 acme-terraform-state-*    │          │  VPCs, EKS, RDS, IAM...     │
  │  ├─ network/prod/*.tfstate   │◄─────────┤  (state reads/writes via    │
  │  ├─ network/prod/*.tflock    │  s3 ops   │   admin-side credentials;   │
  │  ├─ data/prod/*.tfstate      │          │   mutations via the role)   │
  │  └─ (versioning ON, SSE-KMS, │          │                             │
  │      TLS-only, access logs)  │          │                             │
  └───────────────────────────────┘          └─────────────────────────────┘
        state bucket policy: admin principals only
        per-prefix IAM (Section 3) limits each root to its own key

One chicken-and-egg deserves its own sentence because it eats new teams: the state bucket cannot live in the state it stores. The bucket, its versioning, its KMS key, and its access policies are applied exactly once, from a root that either uses a different backend (a bootstrapped bucket in the admin account) or local state sealed into the bootstrap runbook. Everything else in this guide assumes that root exists, is boring, and is reviewed like a control plane — because it is one. When you move an existing root between backends, terraform init -migrate-state copies state to the new backend configuration (lock and all); re-run it per workspace, since each workspace migrates its own key. And when you change a backend argument in a way that alters the object address (region, key, bucket), treat it as a migration, not an edit.

Partitioning: the state file is the blast-radius unit

Every state design argument reduces to one question: what is the largest set of resources you are willing to lose, lock, or block all at once? The state file is the unit of locking (one lock per key), the unit of corruption (one damaged object), the unit of refresh time (state size drives plan time), and the unit of ownership (who can write it). Pick its boundaries deliberately:

The coupling between partitions is where architecture actually happens, and Terraform gives you exactly one built-in: the terraform_remote_state data source. Its contract is read-only output sharing — the consumer fetches the producer's state object from the backend and exposes the producer's root module outputs; nothing else crosses the boundary. HashiCorp's own framing is the "core infrastructure team" pattern: a networking root exposes VPC IDs, subnet IDs, NAT gateways, and every other root consumes them read-only. The security property comes from IAM asymmetry: producers hold write paths on their own state, consumers hold s3:GetObject only, and no amount of drift in a consumer can modify the producer's state.

# consumer root (e.g. an EKS cluster that needs the network)
data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-terraform-state-123456789012"
    key    = "network/prod/us-east-1.tfstate"
    region = "us-east-1"
  }
}

resource "aws_subnet" "cluster_nodes" {
  # outputs cross the partition boundary; resource internals do not
  vpc_id            = data.terraform_remote_state.network.outputs.vpc_id
  cidr_block        = data.terraform_remote_state.network.outputs.node_subnet_cidr
  availability_zone = "us-east-1a"
}

Two costs come with that contract, and they compound at scale. First, fan-out: a network state consumed by forty downstream roots means every one of those plans retrieves the network state object from S3 to read a handful of outputs — cheap in dollars, but it couples every consumer's plan to the producer's object and makes the producer's state a hard dependency of every downstream plan. Second, interface discipline: only root outputs are visible, so the producer's output block is an API whether you treat it as one or not. The mitigations that survive production are boring — version your output contract like any API, keep the producer's outputs minimal and stable, and prefer a real data store (the Terraform docs themselves point at Consul-style key stores as the generalization) when the shared value is consumed by non-Terraform tooling too. The documentation is also explicit that remote state is not the only delegation mechanism — and where operations coordination matters more than data sharing, HCP Terraform's queueing replaces the lock dance entirely: it centralizes operations so that runs are serialized per workspace and a new plan cannot even be created while an existing plan awaits approval.

Secrets live in state until you remove them

The uncomfortable fact, stated plainly in HashiCorp's own sensitive-data documentation: state and plan files "contain detailed information about your infrastructure, including resource attributes and metadata that can contain sensitive values, such as initial database passwords or API tokens." Not the variable declarations — the values. Marking a variable sensitive = true redacts it from CLI output and the HCP UI; it does not remove it from state. The state bucket is therefore a credential store with the blast radius of your entire estate, and it deserves the access policy of one.

The escape hatches are real, and version-gated:

• sensitive (v0.15+): display redaction only. The value remains in state.
• ephemeral values (v1.10+) — the ephemeral argument on variables and child module outputs, the ephemeral block, and ephemeral resources: available at runtime, "omitted from state and plan files entirely."
• Write-only arguments (v1.11+) on managed resources: values are used for the create/update call and never persisted — provider support varies, so check the registry for each resource.
• HCP Terraform "automatically encrypts state at rest and protects it with TLS in transit," which is the managed-platform answer if you do not want to build the IAM story yourself.

Everything else is bucket hardening, and it is cheap because you build it once. The module below is the complete production posture: versioning on (every apply is a recoverable version), SSE-KMS with a dedicated key, public access fully blocked, TLS-only transport, denial of unencrypted puts, access logging, and a prevent_destroy lifecycle that makes a stray destroy fail loudly:

resource "aws_s3_bucket" "state" {
  bucket = "acme-terraform-state-123456789012"

  # A destroyed state bucket is an unrecoverable outage; require a
  # deliberate, reviewed removal of this block from the config first.
  lifecycle {
    prevent_destroy = true
  }

  tags = {
    Purpose     = "terraform-state"
    DataClass   = "secrets" # state contains resource attributes incl. secrets
    Compliance  = "infra-critical"
  }
}

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.state.arn
      sse_algorithm     = "aws:kms"
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_iam_policy_document" "state" {
  statement {
    sid     = "DenyInsecureTransport"
    effect  = "Deny"
    actions = ["s3:*"]
    resources = [
      aws_s3_bucket.state.arn,
      "${aws_s3_bucket.state.arn}/*",
    ]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    condition {
      test     = "Bool"
      variable = "aws:SecureTransport"
      values   = ["false"]
    }
  }

  statement {
    sid       = "DenyUnEncryptedObjectUploads"
    effect    = "Deny"
    actions   = ["s3:PutObject"]
    resources = ["${aws_s3_bucket.state.arn}/*"]
    principals {
      type        = "*"
      identifiers = ["*"]
    }
    condition {
      test     = "StringNotEquals"
      variable = "s3:x-amz-server-side-encryption"
      values   = ["aws:kms"]
    }
  }
}

resource "aws_s3_bucket_policy" "state" {
  bucket = aws_s3_bucket.state.id
  policy = data.aws_iam_policy_document.state.json
}

resource "aws_s3_bucket_logging" "state" {
  bucket = aws_s3_bucket.state.id
  target_bucket = aws_s3_bucket.state_access_logs.arn # separate log bucket
  target_prefix  = "s3-state-access/"
}

resource "aws_s3_bucket" "state_access_logs" {
  bucket = "acme-tf-state-access-logs-123456789012"
}

resource "aws_kms_key" "state" {
  description             = "Terraform state encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true
}

Two operational notes on that module. Versioning is your rollback primitive: because every apply writes a new S3 object version, a corrupted or partially-written state can be recovered by copying the prior version back over the key — the recovery runbook in the next section uses it. And SSE-KMS means every state and lock read is a KMS Decrypt call under the hood: at $1.00 per month for the key and $0.03 per 10,000 requests, the cost is noise at any realistic apply volume, but the key policy must include every runner role or your CI fails in a way that looks like an S3 outage.

Import and state surgery at scale

Every brownfield platform meets the same problem: three years of console-clicked resources that Terraform now needs to own. The config-driven import path replaced the old imperative terraform import ceremony: write an import block binding the resource address to the real object's identifier, and the import is planned and applied like any other change — reviewable, repeatable, and part of the normal plan/apply cycle. The escape hatch that saves the first afternoon: write only the import block and run the plan with -generate-config-out=FILE to have Terraform draft the resource block for you. Import blocks are also nestable inside modules as of Terraform 1.16 — which matters at scale, because a 400-resource account import is a tree of module-level import blocks, not one monolithic list in the root. And 1.16 fixed the failure mode you would otherwise discover mid-import: an import target that exists in state but has no corresponding configuration now returns the correct error instead of a confusing one.

# import.tf — planned, reviewable, repeatable
import {
  to = aws_s3_bucket.artifacts
  id = "acme-artifacts-123456789012"
}

import {
  to = module.data_platform.aws_s3_bucket.query_results
  id = "acme-query-results-123456789012"
}

output "import_audit" {
  value = {
    artifacts     = aws_s3_bucket.artifacts.arn
    query_results = module.data_platform.aws_s3_bucket.query_results.arn
  }
}

# 1. Draft the resource blocks from the existing objects
terraform plan -generate-config-out=generated-resources.tf

# 2. Review the generated file, adjust names, move it into the module tree
#    (generated HCL is a starting point, not a reviewed artifact)

# 3. Plan the import itself — it appears as an ordinary planned change
terraform plan -out=import.tfplan

# 4. Apply; state now binds the real objects to the configured addresses
terraform apply import.tfplan

Once resources are bound, state surgery is the maintenance discipline — and the CLI subcommands are the only sanctioned interface: state list, state show (machine-readable with -json, added in 1.16), state identities (JSON identity listing, the newest of the family), state mv, state rm, state pull, state push, and state replace-provider. The two that prevent the most outages:

• terraform state mv is how you rename an address (a module refactor, a resource rename) without a destroy-and-recreate. Move the config, move the state binding, plan: if the plan is empty, you did it right.
• terraform state rm is how you hand an object back to the console or to another team's root — Terraform forgets it without touching it. The config-driven counterpart is the removed block, which the 1.15 series was still fixing edge cases for, so treat both paths as one-way doors and stage them in a plan first.

# 0. Preconditions: the root's lock is FREE (no runner holds it), and
#    you have already pulled the corrupt state for evidence.
terraform state pull > /tmp/state-corrupt-evidence.tfstate

# 1. Find the prior good version of the state object
aws s3api list-object-versions \
  --bucket acme-terraform-state-123456789012 \
  --prefix network/prod/us-east-1.tfstate \
  --query 'Versions[?IsLatest!=true].[VersionId,LastModified,Size]' \
  --output table

# 2. Restore the chosen version back over the live key (server-side copy,
#    no download round-trip; creates a new version, preserving the corrupt one)
aws s3api copy-object \
  --bucket acme-terraform-state-123456789012 \
  --key network/prod/us-east-1.tfstate \
  --copy-source "acme-terraform-state-123456789012/network/prod/us-east-1.tfstate?versionId=VERSION_ID_FROM_STEP_1"

# 3. Verify before the next apply: pull and compare against the evidence copy
terraform state pull | jq '.resources | length'
terraform state list | head -20

# 4. Run a plan against a large sample of the estate; an empty or
#    expected-only plan means the binding set is whole.

Recovering through S3 versioning (above) beats the state push route for one specific reason: push writes whatever you feed it as the new truth, while the S3 restore path copies a byte-identical prior version and keeps the corrupt one as evidence. Either way, the discipline is the same: lock free, evidence pulled, restore, verify with a plan before anyone applies.

Terragrunt: what it actually adds, and what it costs

Terragrunt is an orchestration and DRY layer that sits outside Terraform/OpenTofu: it keeps a terragrunt.hcl per "unit" (a Terraform root), generates backend boilerplate from shared config, and orders multi-unit operations along dependency edges. It does not change what state is — it changes how many roots you can operate without drowning in repeated backend blocks. Whether that trade is worth it is genuinely load-dependent, and the honest answer is below the mechanics.

The state-relevant mechanics are three. First, the remote_state block with a generate attribute writes the backend configuration into each unit as a generated .tf file — one canonical definition, every unit identical:

# common.hcl — included by every unit
remote_state {
  backend = "s3"

  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }

  config = {
    bucket         = "acme-terraform-state-123456789012"
    region         = "us-east-1"
    use_lockfile   = true
    encrypt        = true
    kms_key_id     = "arn:aws:kms:us-east-1:123456789012:key/11111111-2222-3333-4444-555555555555"
    max_retries    = 15
    retry_mode     = "adaptive"
  }
}

# live/prod/data-platform/terragrunt.hcl
include "root" {
  path = find_in_parent_folders("common.hcl")
}

terraform {
  source = "git::git@github.com:acme/infra-modules.git//data-platform?ref=v1.4.0"
}

# State key is unique per unit — each unit gets its own object and lock
locals {
  unit_path = path_relative_to_include()
}

remote_state {
  config = {
    key = "live/prod/data-platform/terraform.tfstate"
  }
}

# Outputs of the network unit flow in as inputs; mocks cover the first
# validate/plan runs before the dependency has ever been applied.
dependency "network" {
  config_path                            = "../network"
  mock_outputs_allowed_terraform_commands = ["validate", "plan"]
  mock_outputs = {
    vpc_id             = "vpc-mock"
    private_subnet_ids = ["subnet-mock-a", "subnet-mock-b"]
  }
}

inputs = {
  vpc_id             = dependency.network.outputs.vpc_id
  subnet_ids         = dependency.network.outputs.private_subnet_ids
  environment        = "prod"
}

Second, dependency resolution: Terragrunt builds the dependency graph from dependency blocks, and when you run a multi-unit operation it applies in topological order — network before data platform, always. The mock-output machinery is what makes cold starts possible: a dependency whose target has never been applied has no outputs, so you allow mocks for validate and plan with a merge strategy (no_merge by default, shallow and deep_map_only available) that decides how real outputs take over from mocks as the dependency lands. The v1.2 series added block-level iteration over dependencies (expansion with count/for_each), which is how a 3-region × 4-component matrix stops being twelve hand-written files.

Third, the optimization that matters most at fleet size: dependency-fetch-output-from-state, graduated to standard behavior in Terragrunt 1.2.0 (October 21, 2026). Instead of invoking terraform output on every dependency unit — which means a full init and provider handshake per dependency per operation — Terragrunt can read the dependency's outputs directly from its remote state, cutting graph resolution from minutes to seconds on large stacks. It is controlled per remote_state block with disable_dependency_optimization, and it composes with the lock-file discipline Terragrunt already handles for you: it generates and propagates each unit's .terraform.lock.hcl so provider versions stay pinned across environments — the one file, besides the generated backend, that you must commit back to version control. The 1.2.1 follow-up release (October 23, 2026) is worth a read even if you never touch its features: it re-implemented Terragrunt's source downloads through its own command handling and shipped faster parsing of shared configuration — both are direct responses to pain reported by large trees.

The stack layer is Terragrunt's answer to "which units, in which environments, from one place": a terragrunt.stack.hcl file declares units with sources, paths, and values; running it materializes each unit under a generated .terragrunt-stack/ directory; and a unit elsewhere in the repo can declare a dependency on the whole stack, consuming every unit's outputs namespaced by unit name. At the state level this changes nothing — every unit still gets its own S3 key and its own lock — it changes the coordination: one command runs the tree, topologically, with the outputs wired.

# live/prod/terragrunt.stack.hcl
unit "network" {
  source = "github.com/acme/infra-modules//networking/vpc?ref=v1.4.0"
  path   = "network"
  values = {
    cidr_block = "10.40.0.0/16"
  }
}

unit "data_platform" {
  source = "github.com/acme/infra-modules//data-platform?ref=v1.4.0"
  path   = "data-platform"
  values = {
    vpc_id  = "10.40.0.0/16"  # wired via dependency outputs in unit configs
    region  = "us-east-1"
  }
}

The counter-argument, stated as strongly as its proponents state the benefits: Terragrunt is an entire second configuration language and runtime between you and Terraform. Its HCL has its own functions, its own resolution order, its own strict controls, and a release cadence independent of Terraform's — v1.2.0 alone graduated eleven experiments and shipped breaking changes in the same release, which is the kind of sentence that should make a platform team schedule a regression week before upgrading. Debugging moves up a layer too: when a dependency output is wrong, you are now reading Terragrunt's graph resolution before you read Terraform's. The rule of thumb that has aged well: adopt Terragrunt when the number of environments times the number of roots makes hand-maintained backend and dependency wiring a measurable fraction of platform toil (roughly, beyond two environments and a dozen roots); skip it below that, because the wrapper will be more code than the problem. HashiCorp's own Stacks orchestration, actively shipping through the 1.14–1.17 series, is the vendor-native answer to the same problem — worth evaluating before adopting a third-party layer if you are already in that ecosystem.

The OpenTofu alternative: encrypt the file itself

Everything above assumes the bucket, the KMS key, and IAM do the protecting. OpenTofu (v1.13.9, November 5, 2026) adds a defense the bucket cannot provide: state and plan encryption in the file itself, via an encryption block in the Terraform/OpenTofu language that both OpenTofu and Terragrunt can configure. The design is key-provider based — static keys, remote providers such as AWS KMS, or derivative keys — with AES-GCM as the primary method, and it deliberately covers the gap S3 encryption does not: a file that escapes the bucket (a developer's local pull, a CI artifact, a misconfigured backup sync) is ciphertext, not a secrets dump.

terraform {
  encryption {
    key_provider "static" "state_key_v2" {
      key = "REPLACE_WITH_32_BYTE_HEX_MANAGED_IN_YOUR_KMS"
    }

    method "aes_gcm" "current" {
      key_provider = key_provider.static.state_key_v2
    }

    # Previous key kept only as a decrypt fallback: reads try the primary
    # method first, then fall back; writes always use the primary method.
    key_provider "static" "state_key_v1" {
      key = "REPLACE_WITH_PREVIOUS_KEY_DURING_ROTATION"
    }

    method "aes_gcm" "previous" {
      key_provider = key_provider.static.state_key_v1
    }

    state {
      method = method.aes_gcm.current
      fallback {
        method = method.aes_gcm.previous
      }
    }

    plan {
      method = method.aes_gcm.current
    }

    remote_state_data_sources {
      default {
        method = method.aes_gcm.current
      }
    }
  }
}

Be equally clear about what it does not do, because the design document is refreshingly honest about it: encryption protects data at rest only. It does not protect JSON or CLI output ("not a goal to encrypt any output on the standard output"), does not authenticate plan freshness or defend against replay (an attacker swapping in an old state file), and explicitly "is not a goal to protect the state file against the operator of the device running tofu" — whoever runs the tool can decrypt by definition. The fallback block is the rotation mechanism: tofu attempts decryption with the primary method and then falls back to the old key, while every write uses the primary — so rotation is one release of new writes, not a coordinated flip. And the Terragrunt bridge means fleet adoption is one attribute: the remote_state block's encryption map is transformed into the OpenTofu encryption configuration for every unit that shares the file.

The cost model, in verified dollars

Cost arguments about state are usually wrong in both directions — either "S3 is free, ignore it" or "KMS will bankrupt us." Here is the arithmetic for a concrete reference fleet (50 roots, 10 plan+apply cycles per root per day, 50 MB average state), computed with the verified pricing facts and the request model implied by the locking mechanics above:

Read the last row twice, because it is the finding most teams miss: versioned-state volume scales with apply frequency, not with state size. A 50 MB state applied 20 times a day accumulates 30 GB of versions per month; at 200 roots that is 6 TB of retained state versions, billed at S3 Standard rates for as long as you keep them. The fix is not to disable versioning — versioning is your corruption-recovery primitive (Section 6) — it is a lifecycle rule on noncurrent versions that caps retention deliberately: NoncurrentVersionExpiration keeping 90 days (or 200 versions) of recovery depth instead of forever. Everything else in the table is noise: the entire request-plus-KMS bill for a 50-root fleet is under $2/month, which is why backend choices should be made on locking semantics, IAM isolation, and recovery, and never on price.

Choosing a shape

Production realities: where this breaks at scale

• The stale-lock incident is a when, not an if. Hard-killed runners (spot reclamation, OOM, cancelled pipelines) leave .tflock objects with no owner, and every subsequent run fails with a lock error. The runbook in Section 2 is the mitigation; the prevention is CI runners that trap termination signals and a team rule that force-unlock requires reading the lock document first.
• force-unlock is a corruption window, not a button. Unlocking a live holder creates two writers; the second state write wins and the first apply's changes are silently unrecorded. The lock ID nonce exists to make you look at the holder before you shoot.
• Refresh time grows with the estate. Plans against big states spend their time on provider reads, not diffing. -minimal-refresh (v1.17.0) cuts it by refreshing only resources with proposed changes — but the diff you read is then computed against unrefreshed attributes, so a config-only plan can miss out-of-band changes made outside Terraform. That trade-off pairs naturally with a drift-detection discipline; we cover the detection side in our IaC drift-detection guide.
• Versioning is unbounded by default. 30 GB of retained state versions per root per month at the reference fleet's apply cadence; without a noncurrent-version lifecycle rule this grows forever. Cap it deliberately (Section 8's table).
• Remote-state fan-out couples every consumer to the producer's object. Forty consumers planning hourly against one network state means the producer's S3 object is on the critical path of every downstream plan — and its output schema is an API whether you manage it as one or not. Keep producer output surfaces small, versioned, and boring.
• Workspaces are a key-layout feature, not tenancy. All workspaces under one backend share identity and lock family; the docs say it outright — they are "not appropriate for system decomposition or deployments requiring separate credentials and access controls." Use separate keys plus per-prefix IAM when boundaries matter.
• Import waves change plan size dramatically. Bringing an existing account under management means an import plan that touches every resource; schedule it off-peak, per-domain, and expect the first applies to be slow. The 1.16 module-level import blocks exist precisely to keep such waves reviewable.
• CI lock contention looks like an outage. Concurrent pipelines retrying a locked state generate streams of 412s and alert noise. The backend's max_retries / retry_mode = "adaptive" absorb transient contention; the real fix is a queue (one in-flight run per root) rather than optimistic concurrency everywhere.
• The bucket policy is the whole security model for plaintext state. Sensitive marking redacts displays; only ephemeral values (1.10+) and write-only arguments (1.11+) actually leave state. Until your estate is migrated to those, treat read access to the state bucket as equivalent to admin access to every resource it describes — and audit it with that severity.

Who should skip this architecture

Honesty cuts both ways, so here is who should not build any of this. If you run fewer than a hundred resources for one team, the production-grade bucket module, per-prefix IAM, and partitioning strategy here is over-engineering — take the single-backend row from the decision table and spend the saved weekend on tests. If you are already on HCP Terraform, Spacelift, or Scalr, their managed state storage, locking, and queueing already implement most of Sections 2–4 for you; this guide is useful as an understanding of the machinery underneath, not as a migration target. If your estate is PaaS or serverless-shaped with a handful of resources, state size and lock contention will never be your problems — a plain backend and good variable hygiene suffice. And if you have genuinely evaluated Crossplane-style control planes or OpenTofu for other reasons, the state-management problem follows the tool — pick the tool for its provider coverage and team fit first, then apply the partitioning and locking principles, which are identical everywhere: one blast-radius unit per thing you can afford to lose, one lock per unit, write paths smaller than read paths, and a recovery plan you have actually rehearsed.

The 2026 defaults make the baseline cheap: S3-native locking means no DynamoDB table to provision, alarm on, or pay for; -minimal-refresh claws back plan time for the estates that need it; Terragrunt 1.2 makes hundred-unit trees practical if you need them; OpenTofu encrypts the artifact itself if your compliance posture demands it. None of that changes the architecture. State is a database with your credentials in it and your blast radius in it — size its partitions, guard its writes, rehearse its recovery, and re-read the changelog before assuming the tutorial you copied in 2022 is still true.