In platform engineering, we are taught that the reconciliation loop is our greatest safeguard. It is the "self-healing" promise of the control plane: if the world drifts from the desired state, the controller will fix it. But there is a dark side to this promise. When the desired state itself is wrong, the reconciliation loop becomes a high-velocity delivery mechanism for disaster.
In February 2025, Grafana Labs experienced this firsthand. A routine update to TLS policies—something that should have been a low-risk configuration tweak—resulted in the deletion of the load balancers serving roughly 25% of Grafana Cloud. The outage lasted 150 minutes. The root cause wasn't a bug in the cloud provider or a failed API call; it was a mutation blind spot in how their control plane handled resource composition.
The Failure Chain: From TLS Policy to Deleted Load Balancers
Grafana uses Crossplane to manage its cloud service provider (CSP) resources. In Crossplane, a Composition defines how a high-level "Composite Resource" (like a managed database or load balancer) is translated into the specific low-level resources required by the cloud provider.
To implement a TLS policy update, an engineer modified the merge-policy of a Crossplane composition. The goal was simple: allow a specific TLS policy to be overridden by the user while keeping the base configuration intact.
The failure followed a precise, logical sequence:
- The Edit: The
merge-policywas changed to allow overrides. However, this change unintentionally overrode a critical default value in the base configuration. - The Flip: Because of the override, the rendered state of the underlying Kubernetes
Serviceobjects flipped fromtype: LoadBalancertotype: ClusterIP. - The Execution: The reconciliation loop saw the "desired state" had changed. It didn't see this as a "deletion" of the load balancer, but as a "mutation" of the Service type.
- The Destruction: The Kubernetes Cloud Controller Manager (CCM) observed the change to
ClusterIP. SinceClusterIPservices do not require a cloud load balancer, the CCM dutifully deleted the physical load balancers in the cloud provider to match the new state.
sequenceDiagram
participant Eng as Engineer
participant CP as Crossplane Composition
participant K8s as K8s Service
participant CSP as Cloud Provider LB
Eng->>CP: Change merge-policy (TLS update)
Note over CP: Unintentional override of Service type
CP->>K8s: Update Service: LoadBalancer → ClusterIP
K8s->>CSP: "I no longer need a Cloud LB for this service"
CSP->>CSP: Delete Load Balancer
Note over CSP: 25% of Grafana Cloud traffic drops
The Architectural Lie: Why Deletion Protection Failed
The most unsettling part of this incident is that Grafana had deletion protection enabled at the Crossplane level. In most infrastructure-as-code (IaC) setups, "deletion protection" is the final safety net. It's the flag that says, "even if the API tells you to delete this, don't."
But deletion protection is designed to stop a DELETE request. In this case, there was no DELETE request for the Crossplane resource. The Crossplane resource stayed exactly where it was. It just told the underlying Kubernetes Service to change its type.
This is the Mutation Blind Spot. The control plane was protecting the abstraction (the Crossplane object), but it was blind to the consequence (the cloud provider deleting a resource because the abstraction changed). The protection was at the wrong layer. To stop this, you don't need protection at the control plane level; you need it at the CSP layer—explicit deletion protection on the actual cloud load balancer itself.
The Process Fallacy: "Waves" vs. "Systems"
Grafana uses a "wave" deployment strategy: Dev → Staging → Canary → Production. This is standard industry practice. However, this change bypassed the automated wave machinery and was deployed manually.
The "senior engineer" mistake here wasn't just bypassing the process; it was the belief that the process was the safety. When safety is "procedural" (i.e., "we agree to follow the waves"), it is a suggestion. When safety is "systemic" (i.e., "the CI blocks any deployment to more than one wave at a time"), it is a constraint.
Grafana's remediation was a move from process to system:
- CI Enforcement: A new check now prevents a deployment to more than one wave simultaneously.
- Rendered-State Diffing: They developed a "Managed Resources exporter" that renders the final cloud state before it is applied. This allows engineers to see that a "simple TLS change" is actually flipping a Service type from
LoadBalancertoClusterIPbefore the reconcile loop executes.
Verdict: Who Should Be Terrified?
If you are running a control plane (Crossplane, ACK, Config Connector) that manages your critical ingress path, you are susceptible to the mutation blind spot.
Who should skip this pattern: Teams that do not have the bandwidth to build a "rendered-state" validation pipeline. If you can't diff the final cloud manifest before the reconcile loop hits, you are trusting the internal logic of your composition engine with your entire availability. For smaller teams, simple Terraform/OpenTofu with strict prevent_destroy flags is safer because the plan phase is an explicit human gate.
Who should standardize on it: Platform teams building high-scale internal developer platforms (IDPs) who can enforce technical waves and CSP-layer protections. The power of the reconciliation loop is unmatched for scale, but only if you treat the loop as a high-velocity weapon that needs a physical safety catch.