Platform Monkey ctx

Incident Response for Platform Teams: Oncall, Runbooks, and Postmortems That Stick

· By

For the average application developer, an "incident" is a spike in 5xx errors or a slow API endpoint. For a Platform Engineer, an incident is a systemic failure of the substrate: a DNS loop in CoreDNS, a cascading failure of the CNI, or a compromised ServiceAccount escaping to the node. The blast radius is not a single feature; it is the entire fleet.

Most organizations treat Platform Incident Response (PIR) as a subset of Application IR, relying on the same PagerDuty rotations and the same "blameless" postmortem templates. This is a critical mistake. Platform IR requires a fundamentally different telemetry stack and a more aggressive approach to containment, as the attacker or the failure is often operating below the level of application logs.

The Technical Anatomy of a Platform IR Pipeline

A mature PIR process is determined by the quality of its telemetry before the incident ever fires. The goal is to minimize the gap between Time to Detect (TTD) and Time to Contain (TTC) by filtering noise in the kernel, not in the alerting tool.

Phase 1: High-Fidelity Detection (The eBPF Layer)

Traditional audit logs (like Kubernetes API server logs) tell you what happened to an object, but they don't tell you how it happened at the process level. To get forensic-grade visibility, you must move into the kernel using eBPF.

Tools like Falco and Cilium Tetragon allow you to monitor syscalls (exec, open, connect) in real-time. Instead of searching through gigabytes of logs after the fact, you define security policies that trigger alerts based on behavioral anomalies—such as a shell being spawned inside a production pod or an unexpected binary writing to /etc/shadow.

- rule: Unexpected Shell in Production desc: A shell was spawned in a production pod condition: proc.name = sh and container.id != host output: "Unexpected shell spawned (user=%user.name container_id=%container.id proc=%proc.name)" priority: Critical

Phase 2: Automated Containment (The Isolation Pattern)

The instinct of many engineers is to kubectl delete pod [name] the moment a compromise is detected. This is a forensic disaster. Deleting the pod destroys the process list, the volatile memory, and the local ephemeral storage—the very evidence you need for a root cause analysis.

The professional approach is Isolation. By applying a specific label (e.g., incident-status=quarantined) to the affected pod, you can trigger a pre-deployed, deny-all NetworkPolicy that severs all ingress and egress traffic while keeping the process alive for investigation.

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: quarantine-policy namespace: production spec: podSelector: matchLabels: incident-status: quarantined policyTypes: - Ingress - Egress # Empty ingress/egress lists mean all traffic is denied ingress: [] egress: []

To prevent automated controllers (like Deployments or ReplicaSets) from "healing" the pod by deleting it and starting a new one, you can use a Kyverno or OPA Gatekeeper policy to block the deletion of any pod carrying the quarantine label.

apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: protect-quarantined-pods spec: rules: - name: block-quarantine-deletion match: any: - resources: kinds: - Pod validate: message: "Cannot delete a pod marked for forensic analysis (incident-status=quarantined)." deny: conditions: all: - key: "NaN" operator: Equals value: quarantined

Phase 3: Forensic Provenance (The Correlation Matrix)

The hardest part of PIR is Attribution. In a cloud-native environment, a pod is a transient entity. Its name changes, its IP is reused, and its identity is a set of assumed IAM roles. To prove that "Pod A performed Action B," you need a three-plane correlation matrix.

The Correlation Flow:

If you only have the CloudTrail log, you know a node performed an action, but you don't know which pod was responsible. If you only have eBPF, you know a pod performed an action, but you can't prove what happened at the cloud provider level.

def correlate_incident(pod_uid, timestamp): # 1. Get runtime evidence from eBPF stream runtime_event = ebpf_store.query(pod_id=pod_uid, time=timestamp) # 2. Map Pod UID to Node and ServiceAccount via K8s Audit k8s_meta = k8s_audit.get_pod_metadata(pod_uid) node_id = k8s_meta.node_name sa_name = k8s_meta.service_account # 3. Find the Cloud Role Session for that node/SA at that time cloud_session = cloud_trail.find_session( resource=node_id, principal=sa_name, time=timestamp ) return { "evidence_chain": f"{pod_uid} -> {node_id} -> {cloud_session.id}", "action": runtime_event.syscall }

Compliance Mapping: Making IR Audit-Ready

For organizations pursuing SOC2, HIPAA, or PCI DSS, the incident response process is a primary piece of evidence. Auditors are not looking for a perfect record of zero incidents; they are looking for a verifiable trail of how incidents were handled.

The following table maps common platform IR activities to SOC2 CC7.3 controls:

Control | Required Evidence | Platform Artifact --------------------------------------------------------------------------------- CC7.3 (Detection) | Proof that alerts were triggered | Prometheus/Falco Alert History CC7.3 (Response) | Evidence of timely triage | Incident Timeline (Slack/Jira) CC7.3 (Containment) | Proof that threat was neutralized | NetworkPolicy Apply Logs / Git Commit CC7.3 (Post-Incident) | Evidence of root cause analysis | Signed Blameless Postmortem Document

To satisfy these requirements, the postmortem must be an immutable record. The gold standard is a Markdown file in a protected Git repository, merged via PR, and signed by both the Lead Engineer and the Security Officer.

Production Realities: Where the Process Breaks

In the real world, the "textbook" IR process often collapses under the weight of production scale.

Who Should Skip This

This level of forensic rigor is overkill for:

References & Further Reading