GitOps Secrets After the OpenAI Breach: SOPS, External Secrets, and the Vault Question
Sources
- Hacktron: Hacking OpenAI (heap overflow + SSO misconfiguration)
- External Secrets Operator v2.11.0 Release
- External Secrets Operator Documentation
- SOPS Documentation (getsops.io)
- Sealed Secrets (bitnami/sealed-secrets)
- HashiCorp Vault v2.1.1 Changelog
- Discourse Security Advisory GHSA-vhm9-85gw-x335 (RCE via malformed HEIF file)
- Debian DSA-6417-1: libheif security update
- Flux: Manage Kubernetes secrets with SOPS
- Secrets Store CSI Driver (Kubernetes SIG-Auth)
On July 25, 2026, a security research team chained two unremarkable failures into full read access to OpenAI’s internal repositories: a heap buffer overflow in libheif — the image decoder behind every HEIC/HEIF upload on the internet — and a single sign-on misconfiguration on community.openai.com that turned forum compromise into ChatGPT and Codex account takeover. The researchers demonstrated impact by having the employee’s own Codex agent open a pull request in OpenAI’s internal monorepo. The writeup went to the Hacker News front page on September 18 with over 400 points.
The lesson platform engineers keep relearning is that secrets infrastructure fails at its identity layer, not its cryptography layer. Nobody broke AES. They broke the assumption that a session token on a low-value property should not grant high-value access. This guide is about building GitOps secrets workflows where that assumption is already dead: SOPS with age, Sealed Secrets, External Secrets Operator, and the Vault integration question — with the identity controls that decide whether any of it survives contact with an attacker who already owns one of your services.
The Breach, In Platform Engineering Terms
The chain, mapped to the controls a platform team owns:
| Attack stage | What happened (per the Hacktron disclosure) | The control that should have stopped it |
|---|---|---|
| Dependency RCE | A heap buffer overflow in libheif 1.19.7, reachable through Discourse’s HEIC image-upload path (FastImage passed HEIF to ImageMagick’s magick command, exposing the parser). The vulnerable code had been fixed upstream without a CVE or security flag, so Debian 12/13 never backported it. |
Dependency scanning keyed to CVEs misses unflagged security fixes. Image-processing pipelines belong in ephemeral sandboxes — not just for cost, for blast radius. |
| Identity pivot | An SSO misconfiguration meant that anyone with code execution on the forum could hijack active ChatGPT/Codex sessions. Not a Discourse bug — the researchers stress any OpenAI-SSO-integrated service would have worked. | Session-token scoping, token binding, and short-lived credentials per service — not one ambient identity trust domain shared by every property. |
| Lateral movement | With an employee’s ChatGPT account, they reached the employee’s Codex, which was connected to OpenAI’s GitHub organization — enough to open a PR in the internal monorepo as a proof of concept. | Connector permissions: an agent should hold scoped, per-repo credentials — not an employee-wide integration that inherits every connection the human has. |
| Detection | Across the broader multi-company campaign, only Shopify detected the activity — after thousands of uploaded images repeatedly crashed their image processors. | If your image-decode service is crashing and you are not paging, your runtime security telemetry does not exist yet. |
None of this is exotic. Every stage maps to a question your secrets architecture answers implicitly — usually wrongly: when one service is compromised, what does the attacker get? The four GitOps secrets patterns below answer that question very differently.
The Four Patterns
The Landscape
- SOPS + age: secrets encrypted at rest in Git, decrypted by the GitOps controller in-cluster. Git stays the source of truth; the blast radius of a repo leak is low; the blast radius of a cluster identity leak is high.
- Sealed Secrets: asymmetric encryption per-secret against a controller-held key. Simpler mental model, but every secret is cluster-bound and the controller key is a crown jewel.
- External Secrets Operator (ESO): secrets never enter Git at all; the cluster pulls from AWS Secrets Manager, Vault, GCP Secret Manager, and ~20 more providers. Fresh at the cost of a runtime dependency.
- Secrets Store CSI: secrets are mounted as volumes, never becoming Kubernetes
Secretobjects at all unless you explicitly opt in to syncing them.
Pattern Git repo holds Cluster holds Provider holds
------------------ ------------------- ----------------------- ----------------------
Plaintext (NO) the actual secret the actual secret n/a
SOPS + age ciphertext decryption key + secret n/a
Sealed Secrets ciphertext (bound) sealing key + secret n/a
ESO nothing (a reference) the synced Secret the actual secret
CSI (mount-only) nothing (a reference) nothing on the API the actual secret
server (file only)
Attacker footholds, best case exposure:
- Repo access: SOPS/Sealed = ciphertext only. ESO/CSI = references only.
- etcd/cluster: everyone except CSI mount-only leaks the plaintext Secret.
- Workload RCE: identical for all four (the app must read the secret).
- SSO/identity: ESO's provider credentials are the newest crown jewel.
Note the asymmetry the breach exploits: platform teams pour effort into the Git leg (encrypting what lives in a repo) and almost none into the identity leg (what a stolen session can authenticate as). The Hacktron chain never touched a Kubernetes Secret. It went service → session → connected integration. Your secrets tooling is only one layer of that story.
SOPS + age: Encrypted-at-Source, Decrypted-in-Cluster
SOPS encrypts the values of a YAML file while leaving keys in plaintext, so diffs stay reviewable. The recommended identity backend in 2026 is age — the SOPS project itself recommends age over PGP where possible, and PGP keyrings in CI are an operational relic you should retire.
# Generate the age key that the GitOps controller will use to decrypt
age-keygen -o age.agekey
# Public key: age1helqcqsh9464r8chnwc2fzj8uv7vr5ntnsft0tn45v2xtz0hpfwq98cmsg
# The private key becomes a Kubernetes Secret in the GitOps namespace.
# The key file name must end in .agekey for SOPS to detect it as an age key:
cat age.agekey | kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin
# Encrypt a Secret manifest in place (values only, keys stay readable):
sops --age=age1helqcqsh9464r8chnwc2fzj8uv7vr5ntnsft0tn45v2xtz0hpfwq98cmsg \
--encrypt --encrypted-regex '^(data|stringData)$' --in-place basic-auth.yaml
Scale key selection with a .sops.yaml at the repo root — creation rules are evaluated sequentially and the first matching rule wins:
# .sops.yaml — rules are evaluated sequentially, first match wins
creation_rules:
# dev files: shared dev age key + KMS for break-glass
- path_regex: \.dev\.yaml$
age: age1s3cqcks5genc6ru8chl0hkkd04zmxvczsvdxq99ekffe4gmvjpzsedk23c
kms: 'arn:aws:kms:us-west-2:927034868273:key/fe86dd69-4132-404c-ab86-4269956b4500'
# prod files: prod-only key set — a dev laptop cannot decrypt prod
- path_regex: \.prod\.yaml$
age: age1qe5lxzzeppw5k79vxn3872272sgy224g2nzqlzy3uljs84say3yqgvd0sw
kms: 'arn:aws:kms:us-west-2:361527076523:key/5052f06a-5d3f-489e-b86c-57201e06f31e+arn:aws:iam::361527076523:role/sops-prod'
# catchall: deny-by-default by pointing at a KMS key nobody's laptop has
- kms: 'arn:aws:kms:us-west-2:142069644989:key/846cfb17-373d-49b9-8baf-f36b04512e47'
Wire decryption into Flux by pointing the Kustomization at the age secret:
flux create kustomization my-secrets \
--source=my-secrets \
--path=./clusters/cluster0 \
--prune=true \
--interval=10m \
--decryption-provider=sops \
--decryption-secret=sops-age
On the Argo CD side, the equivalent is KSOPS (kustomize-sops, current release v4.5.1) with the decryption key delivered to the repo server — which means the Argo CD repo server can now decrypt your entire estate, so treat it like a tier-0 asset: dedicated namespace, NetworkPolicy egress restrictions, and no dashboard exposure.
The Hidden Operational Costs
- Key distribution is the real product. SOPS-the-CLI is a weekend to learn; a key-rotation runbook that doesn't break 400 encrypted files across 12 repos is the actual project. Without
sops updatekeysdiscipline, teams accumulate orphaned keys that still decrypt old files. - Decryption key placement = your crown jewel moves into the cluster. A Flux kustomize-controller or Argo CD repo server with the age private key can decrypt every secret it can name. Compromise it and the ciphertext in Git is worthless as a defense.
- CI decrypts too, usually worse. If your pipeline runs
sops -dfor validation, the CI runner's identity is now in the blast radius. Pin CI to KMS-backed identities with short-lived credentials rather than exporting an age private key as a CI variable — that variable is one log-mistake away from a leak. - Rotation is manual-ish. SOPS rotates encryption metadata on save, but the secret value and the provider credential usage are yours to rotate. Nothing in the SOPS model reminds you a password is three years old.
Sealed Secrets: Asymmetric, Cluster-Bound
Sealed Secrets inverts the flow: you encrypt against the cluster's public key, before anything is committed. The controller is the only entity that can decrypt. The current project home is bitnami/sealed-secrets (it moved from bitnami-labs; latest release v0.40.0, September 2026).
# Install the controller (Helm):
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system \
--set-string fullnameOverride=sealed-secrets-controller \
sealed-secrets/sealed-secrets
# Create the plaintext Secret locally (never applied!):
echo -n bar | kubectl create secret generic mysecret \
--dry-run=client --from-file=foo=/dev/stdin -o json > mysecret.json
# Seal it against the controller's public cert:
kubeseal -f mysecret.json -w mysealedsecret.json
# mysealedsecret.json is now safe to commit
# In-cluster, the controller decrypts and creates the Secret:
kubectl create -f mysealedsecret.json
Scope control is the part teams get wrong. By default a SealedSecret is bound to its name and namespace; widen at your peril:
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: mysecret
namespace: payments
annotations:
# cluster-wide: ANY namespace can use this ciphertext. Resist.
# sealedsecrets.bitnami.com/cluster-wide: "true"
# namespace-wide: any name within this namespace. Sometimes right.
sealedsecrets.bitnami.com/namespace-wide: "true"
spec:
encryptedData:
foo: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq.....
The Controller Key Is the Whole Game
- Sealing keys renew every 30 days by default (“a new sealing key is created and appended to the set of active sealing keys”) — but old keys are kept so existing SealedSecrets keep decrypting. Your key registry is a long-lived crown jewel; back it up off-cluster or losing the cluster means losing every sealed secret permanently.
- No authentication by design. The docs are explicit: “anyone can create a SealedSecret containing any Secret they like” if the name/namespace matches. GitOps review discipline and RBAC on the CRD are the actual security boundary, not the crypto.
- Rotation is a re-seal, not a flag. To rotate a value you re-run kubeseal — with the ciphertext bound to name/namespace, renaming a team or migrating a namespace means re-sealing everything.
- One cluster = one key registry. Multi-cluster with Sealed Secrets means either sharing keys across clusters (blast radius grows) or per-cluster registries (every secret exists N times). This is the wall most teams hit before evaluating ESO.
External Secrets Operator: Secrets Never Touch Git
ESO removes secrets from Git entirely: the repo holds a reference, and the operator materializes the Secret in-cluster from a provider. External Secrets Operator v2.11.0 was released September 18, 2026 — the day this guide is being written — so the patterns below are current.
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
-n external-secrets --create-namespace \
--set installCRDs=true
The model separates what from how: the SecretStore holds provider access; the ExternalSecret says what to fetch:
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: aws-secretsmanager
namespace: payments
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
# Controller pod identity (IRSA) — no static credentials in the cluster.
# Omit spec.provider.aws.auth entirely to use the controller's IAM role:
role: arn:aws:iam::123456789012:role/external-secrets-payments
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: stripe-api-key
namespace: payments
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secretsmanager
kind: SecretStore
target:
name: stripe-api-key
creationPolicy: Owner
deletionPolicy: Retain # Delete | Merge | Retain; default is Retain
data:
- secretKey: api-key
remoteRef:
key: payments/stripe/prod
property: api-key
For Vault-backed estates, the same shape with Kubernetes auth — no static Vault token stored anywhere:
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: vault-backend
namespace: payments
spec:
provider:
vault:
server: "https://vault.acme.org"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "payments-eso"
serviceAccountRef:
name: "payments-eso
Vault validates the service account token via the TokenReview API — you must bind the system:auth-delegator ClusterRole to the service account ESO authenticates with, or every reconciliation fails with a 403 that looks exactly like a bad role. That is the single most common misconfiguration in this pattern; budget an hour for it the first time.
What ESO Buys You, and What It Costs
- Freshness for free-ish:
refreshIntervalre-pulls on schedule, and provider-side rotation lands in the cluster without a commit. This is the pattern’s headline win over SOPS and Sealed Secrets. - deletionPolicy semantics matter:
Deleteremoves the Secret if all provider secrets are deleted;Mergeremoves keys but not the Secret;Retain(the default) keeps it — and if a provider secret vanishes,Retainputs the ExternalSecret intoSecretSyncedErrorwhile the old Secret keeps running. For a payments credential, that default is usually what you want; know which one you set. - A new blast radius: with IRSA/Kubernetes-auth, the ESO controller’s service account identity can read every secret its IAM role or Vault role allows. The controller is now a tier-0 asset: an attacker who compromises it does not need your repo, your SSO, or your session tokens. Scope the IAM role by secret path prefix, not
Resource: "*". - Runtime dependency: ESO turns your secrets provider into cluster-critical infrastructure. If AWS Secrets Manager has a regional incident, new Secrets do not materialize and changed ones go stale. Existing Secrets keep working (they are plain Kubernetes Secrets) — degraded, not down — but design the alerting for
SecretSyncedErrorbefore you need it. - Git loses its auditability of values: the diff shows reference changes, not value changes. If your compliance story is “Git history proves who changed what,” that proof moves to the provider’s audit log. Verify your provider has one, and that you can query it.
The CSI Alternative: Never a Secret Object at All
The Secrets Store CSI driver (kubernetes-sigs, SIG-Auth subproject, current v1.6.1) mounts provider secrets as a volume and, by default, never creates a Kubernetes Secret object. In a world where etcd access or API-server credential theft is a realistic stage of an attack chain (see: every stage of the OpenAI chain being a surprise), that is a material hardening step for high-value secrets:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: vault-db-creds
namespace: payments
spec:
provider: vault
parameters:
vaultAddress: "https://vault.acme.org"
roleName: "payments-db"
objects: |
array:
- |
objectPath: "payments/db"
objectName: "prod-password"
objectVersion: ""
---
# Pod spec: the secret arrives as a file, never as an API object
# (unless secretObjects sync is explicitly enabled)
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "vault-db-creds"
The trade-off is application friction: your workload reads a file instead of an environment variable, the pod must be restarted (or the alpha rotation reconciler enabled) to refresh, and every framework that expects envFrom needs a shim. This is the right pattern for database credentials and signing keys; it is overkill for an API rate-limiter token.
Comparison: Choosing by Failure Mode
| Criterion | SOPS + age | Sealed Secrets | ESO (pull) | CSI mount-only |
|---|---|---|---|---|
| Plaintext in Git | No (ciphertext) | No (ciphertext) | Never in Git | Never in Git |
| Plaintext on API server / etcd | Yes | Yes | Yes (unless mount-only via CSI) | No (file only) |
| Automatic provider-side rotation sync | No | No | Yes (refreshInterval) | On remount / alpha rotation |
| Runtime dependency on provider | None (decrypt-time only) | None | Continuous | At pod start |
| Crown jewel | age key in-cluster / CI | Controller key registry (backup it!) | ESO service-account identity (IAM/Vault role) | Provider credentials |
| Multi-cluster story | Same key or per-cluster keys | Awkward (per-cluster registries or shared keys) | Clean (same provider, scoped roles) | Clean |
| Git audit trail of values | Yes (encrypted diffs) | Yes (ciphertext commits) | No (references only) | No |
| Practical sweet spot | Small teams, Git-first, low secret churn | Single cluster, Git-first, wants one mental model | Multi-cluster, cloud provider in play, rotation matters | High-value credentials; DB passwords, signing keys |
The Vault Integration Question
Where does HashiCorp Vault (current release v2.1.1, September 16, 2026 — a security-patch release updating golang.org/x/crypto, grpc, and apache/thrift for published advisories) fit? The honest answer for a platform team in 2026: Vault is a policy engine and central audit log, not a requirement.
- If you are on a cloud provider, the provider’s native secret store is usually enough. AWS Secrets Manager + IRSA gives you short-lived credentials, path-scoped IAM roles, and CloudTrail audit — with one less stateful, tier-0 service to operate. Vault’s value on top of that is dynamic database credentials, pki issuance, and cross-provider unification.
- Vault’s real differentiator is dynamic secrets: short-lived database credentials issued per-pod on request, not stored at all. If your compliance regime requires “no human ever sees a prod DB password,” that is a Vault (or equivalent) feature, and ESO’s Vault provider plus the CSI driver’s Vault provider both integrate with it.
- The operational cost is real: Vault is a stateful, quorum-sensitive service (Raft storage) whose unseal ceremony is a deliberate inconvenience. Treat a Vault outage as a tier-0 incident and rehearse unseal/restore. Teams that install Vault without that operational commitment end up with static tokens in Kubernetes Secrets — the worst of both worlds.
- OpenBao exists if the BUSL license is a blocker for your legal team; ESO added an OpenBao Kubernetes auth method in v2.10.0, so the integration path is the same as Vault’s.
The Controls That Actually Stop the OpenAI Chain
Your secrets tooling is table stakes. The Hacktron chain was stopped by none of it — it was stopped (eventually, by disclosure) by identity architecture. Five controls, in order of leverage:
- Scope every SSO integration. The researchers were explicit: the vulnerability was not Discourse-specific — “if any first-party or third-party service using the OpenAI SSO was compromised, it would lead to same access.” Audit every service that accepts your IdP: does a session on your forum really need transit to your code agents? Per-service OIDC clients with scoped claims, short-lived tokens, and no ambient trust.
- Bind agent/connector credentials to work, not to people. The final hop was an employee’s Codex connection to the GitHub org. Connected-integration permissions should be scoped per-repo and reviewable; “the agent inherits the human’s connections” is the misconfiguration, not the crypto.
- Sandbox untrusted file parsing. The entry point was an image decoder reachable from an upload form. If a service parses attacker-controlled files, it runs ephemerally with no ambient credentials — the same rule as the Discourse fix (they added ImageMagick sandboxing as defense in depth) applied at the platform layer.
- Patch dependencies on commit metadata, not just CVEs. The libheif fix existed upstream a year earlier without a CVE, so Debian never backported it and every downstream image inherited it. Your dependency pipeline needs a story for security-relevant fixes that never get a CVE — tracking upstream -security branches and maintainer advisories (DSA-6417-1 for Debian libheif is the precedent here) is the manual part nobody automates well.
- Page on crash loops in parsing paths. Only one company in the multi-month campaign detected the exploitation. Repeated crashes of an image processor were the tell. An alert on
container_cpu_crash_loopin parsing services is cheaper than any secrets migration on this list.
Who Should Skip What
- Skip SOPS if your secrets change faster than your Git flow tolerates, or you cannot invest in key rotation runbooks — stale ciphertext with orphaned keys is worse than a provider.
- Skip Sealed Secrets if you run more than one cluster or rename namespaces often; the re-seal tax compounds, and the controller-key backup is a single point of catastrophic failure.
- Skip ESO if your platform has zero tolerance for new runtime dependencies and your secret churn is near zero — SOPS + a strict .sops.yaml is a smaller system to operate.
- Skip Vault if you are single-cloud and do not need dynamic secrets — the provider’s secret manager plus IRSA/Workload Identity covers the threat model with one less stateful tier-0 service to babysit.
- Skip none of the identity work. The breach that ends your week will not brute-force your ciphertext; it will ride a session token from the least-secured service you forgot you owned.
Verdict
For a platform team starting from plaintext-in-a-private-repo in 2026: encrypt in Git with SOPS+age as the floor, adopt ESO with cloud-native identity (IRSA, Workload Identity, Vault Kubernetes auth) as the default for anything that rotates, and treat the CSI mount-only pattern for database and signing credentials as the ceiling. Then spend the time you saved on the part none of these tools ship: auditing every SSO integration and connected-agent permission — because the next Hacktron-style chain will walk through identity, not cryptography.