Managed Prometheus on AWS: AMP Pricing Re-Verified, the ADOT Pipeline, and Where the 90% Savings Is Real

Sources

Every few weeks another "we cut our observability bill by 90% with managed Prometheus on AWS" post makes the rounds. The latest is Expona's "Managed Prometheus on AWS: practical tips and trade-offs" (October 27, 2026), a genuinely useful practitioner's tour of Amazon Managed Service for Prometheus (AMP). It is also a useful specimen of how quickly AMP facts rot: its remote-write URL is the wrong path, its per-sample price is roughly 3× the current rate card, and its mental model of query billing does not match how AWS actually meters queries.

None of that makes the 90% claim wrong — against per-host APM vendors the arithmetic lands close to it. But the claim is only true in one direction, and this industry does a poor job of saying which. So this piece does the boring work: it re-verifies AMP's pricing, quotas, endpoints and collector mechanics against AWS's own current documentation, builds the collector architecture that actually earns the savings (ADOT on EKS, SigV4, PrivateLink), and runs the cost model in both directions — including the one where AMP loses to a couple of EC2 instances and the one where a Grafana wall quietly outbills your ingestion.

Two things changed recently that make this worth revisiting now. First, AMP went through a 2025–2026 maturity wave: the default active-series quota jumped from 10M to 50M per workspace (July 2025) (adjustable to 1.5 billion), query insights and per-query cost controls landed in May 2025, Random Cut Forest anomaly detection shipped in October 2025, and native histogram ingestion is now metered explicitly. Second, the ingestion price itself was restructured into descending volume tiers — which is why almost every blog number you remember is stale. All prices below were re-fetched from AWS's pricing pages on 2026-11-10; the comparison numbers for Datadog come from its live pricing list page the same day.

1. What AMP Actually Is (and Is Not)

AMP is a managed, multi-Availability-Zone Prometheus-compatible metrics store with a managed ruler (rule evaluation engine) and a managed Alertmanager. It is not a managed Prometheus server: nothing scrapes for you unless you opt into the agentless collector, and the thing you are buying is the remote-write endpoint, the storage behind it, and a PromQL engine with per-sample metering on both the write and read sides.

The service splits into two planes, and every quota, IAM policy and PrivateLink endpoint in this article hangs off that split:

The workspace is the unit of isolation, billing scope, quota scope and blast radius. Retention defaults to 150 days and is configurable to a maximum of 1095 days (three years). Everything older than that is deleted; there is no block shipping to S3, no compactor tuning, no object-storage lifecycle to design. That is the deal: you give up the Thanos/Mimir knob collection, and in exchange nobody on your team is on call for a time-series database.

POST /workspaces/ws-b226cc2a-a446-46a9-933a-ac50479a5568/api/v1/remote_write
Authorization: AUTHPARAMS                          (AWS SigV4, service "aps")
Content-Type: application/x-protobuf
Content-Encoding: snappy
X-Prometheus-Remote-Write-Version: 0.1.0           (remote write 1.0)

Valid URIs : /workspaces/workspaceId/api/v1/remote_write
Verbs      : POST
Query URL  : /workspaces/workspaceId/api/v1/query

Three properties of that endpoint deserve emphasis because they shape everything downstream. First, the path is /api/v1/remote_write — not /api/v1/write (that is the path of a self-hosted Prometheus receiver started with --web.enable-remote-write-receiver; AMP never used it, and the Expona post ships the wrong path in its example config). Second, requests are limited to 1 MB — batches, not streams. Third, ingestion refuses any sample older than one hour, full stop, non-negotiably; you cannot backfill yesterday's gap into AMP.

2. The Rate Card, Re-Verified

AMP bills four meters, and the tier structure has a quirk that surprises people on their first bill. Prices below are US East (N. Virginia) list rates re-read from the pricing page on 2026-11-10.

MeterRate (us-east-1)What it counts
Metric samples ingested$0.90 per 10M samples, first 2B/moEvery sample that lands on the remote-write endpoint. Descending tiers above 2B and 252B (tier rates render only in AWS's pricing calculator, but every published example prices the first tier at $0.90/10M)
Metrics stored$0.03 per GB-monthSamples (typically 1–2 bytes each) plus metric metadata (name + label keys/values), 1 GB = 230 bytes
Query Samples Processed (QSP)$0.10 per billion samplesEvery data point scanned by a PromQL query — dashboards, scripts, and managed rule evaluations all count. Not "per query"
Managed (agentless) collector$0.04 per collector-hour + $0.03 per 10M samples collectedOptional AWS-run scraper per EKS cluster; you can skip it entirely with your own collector
Free tier (monthly)40M samples ingested · 200B QSP · 10 GB storedPermanently available free tier in supported regions

Four rate-card facts that are routinely misquoted:

Two costs that never appear on the AMP rate card but land on your AWS bill anyway: Amazon Managed Grafana licenses you need to look at the data ($9 per active Editor per workspace per month, $5 per active Viewer, minimum one Editor even if nobody logs in), and interface VPC endpoints if you keep ingestion off the public internet ($0.01/hr per AZ per endpoint plus data processing, plus an STS endpoint requirement covered in section 3). Neither is optional in a production build.

3. Reference Architecture: EKS → ADOT → AMP → Managed Grafana

The architecture that earns the savings has five decisions baked in, and each one is a place teams go wrong:

  EKS cluster (us-east-1)                                    AWS managed planes
 ┌───────────────────────────────────────────────┐   ┌─────────────────────────────────────────────┐
 │                                               │   │                                             │
 │  targets: kubelet, cAdvisor, kube-state,      │   │   AMP workspace ws-abc123                   │
 │  app pods w/ prometheus.io/scrape annotations │   │  ┌───────────────────────────────────────┐   │
 │      │  scrape (in-cluster, HTTP)             │   │  │ dedup (cluster + __replica__ labels)   │   │
 │      ▼                                       │   │  │ ruler: rule groups (QSP-metered evals)│   │
 │  ADOT Collector DaemonSet  (one per node)    │   │  │ alert manager: routes → SNS only       │   │
 │  ┌──────────────────────────────────────┐     │   │  │ storage: 150d default, 1095d max      │   │
 │  │ prometheus receiver (full Prom config)│    │   │  └───────────────────────────────────────┘   │
 │  │   │ relabel: drop noise, cap series  │     │   │        ▲                    ▲               │
 │  │   ▼                                 │      │   │        │ POST /api/v1/       │ GET /api/v1/  │
 │  │ prometheusremotewrite exporter       │──────┼───┼──▶ remote_write (1MB cap) │ query+range    │
 │  │   + sigv4auth ext (service "aps")   │      │   │        │                    │               │
 │  └──────────────────────────────────────┘     │   │  ┌─────┴──────────┐   ┌────┴──────────────┐   │
 │        │ IRSA: amp-iamproxy-ingest-role       │   │  │ VPC endpoint    │   │ Amazon Managed   │   │
 │        │ aps:RemoteWrite on ws ARN           │   │  │ com.amazonaws.  │   │ Grafana workspace │   │
 │        ▼                                     │    │  │ us-east-1.      │   │ sigv4 datasource │   │
 │  VPC endpoint (PrivateLink, no public path)  │   │  │ aps-workspaces  │   │ $9/editor, $5/vwr│   │
 │  + STS endpoint (regional) for token refresh  │   │  └────────────────┘   └───────────────────┘   │
 └───────────────────────────────────────────────┘   │   CloudWatch: AWS/Usage + AWS/Prometheus      │
                                                     │   (DiscardedSamples, ActiveSeries, QSP, free) │
                                                     └─────────────────────────────────────────────┘

Decision 1 — who scrapes. Three options, and they are not equal. The agentless collector is AWS-run (an ENI per subnet, scraping through your VPC, remote-writing via a VPC endpoint, data never on the public internet) and is billed at $0.04/collector-hour plus $0.03 per 10M samples collected — on AWS's own worked example that is $32.44/month for a 10-node cluster, a 40% surcharge on top of that cluster's $80 ingestion bill. A customer-managed collector — ADOT or raw Prometheus in agent mode — pays no collector fee at all; you fund its CPU/memory inside your cluster instead. For a 100-node fleet the agentless collector's metered sample collection alone ($2.68/node-equivalent per month, scaling linearly) usually beats the convenience; run the numbers before defaulting to it. There is also a middle path: the EKS adot add-on (aws eks create-addon --addon-name adot), which gives you an AWS-patched ADOT operator without you hand-rolling the DaemonSet.

Decision 2 — network path. Remote write works over the public endpoint, but a private cluster should use an interface VPC endpoint. Two service names exist: com.amazonaws.<region>.aps-workspaces for the Prometheus-compatible APIs and com.amazonaws.<region>.aps for workspace management. The trap AWS documents explicitly: in a VPC without internet access, SigV4 still needs to mint credentials, so you must also create an STS interface endpoint and set it regionalized — without it, collectors start fine and then silently fail to authenticate.

Decision 3 — HA topology. AMP replicates across AZs in-region by itself; that is not your job. What is your job is deduplication if you run two collectors scraping the same targets: AMP dedups (and stops double-billing) only when HA pairs share a cluster external label and each instance carries a distinct __replica__ label. Without those labels you pay ingestion and storage twice and Grafana shows two of everything — AWS says this in bold: "If you do not set up deduplication, you will be charged for all data samples." Cross-region HA means remote-writing to a second workspace in another region (the OTel config for two exporters in one pipeline is in AWS's cross-region guide), and yes, that doubles ingestion and storage billing too.

Decision 4 — where rules run. AMP's managed ruler evaluates rule groups inside the workspace, which means rule evaluations are QSP-metered queries. AWS's own example: an alerting rule scanning 10,000 series every minute for a month processes 438M samples ≈ $0.04/month. The counter-intuitive consequence: moving alert rules into AMP is usually a cost reduction versus many dashboards polling the same expression, because one evaluation amortizes across every consumer. Pre-aggregate expensive expressions as recording rules in the workspace and point dashboards at the recording rule.

Decision 5 — cardinality control at the edge. The single highest-leverage line in your entire AMP build is the relabel/metric_relabel stage that drops high-cardinality series before remote write. Every series you drop there is 4 samples/minute (at a 15s interval) never ingested, never stored, never scanned by queries — three meters avoided at once. AWS's own cost guide's first two recommendations are exactly these: lengthen collection intervals and reduce active series.

4. The Build: Terraform, the ADOT DaemonSet, and a Prometheus Alternative

4.1 Provisioning the workspace (Terraform)

The workspace itself is trivial; the production value is in what surrounds it — KMS, vended logs, IAM scoped to one workspace ARN, rule groups as code, and the PrivateLink path. All resources below match the current aws_prometheus_* provider schema (aws_prometheus_workspace, aws_prometheus_rule_group_namespace, aws_prometheus_alert_manager_definition, aws_prometheus_scraper for the agentless collector):

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "aws_prometheus_workspace" "platform" {
  alias       = "platform-prod"
  kms_key_arn = aws_kms_key.amp.arn

  logging_configuration {
    # ARN must end with :* so AMP can write into the group
    log_group_arn = "${aws_cloudwatch_log_group.amp_vended.arn}:*"
  }

  tags = { env = "prod", team = "platform" }
}

resource "aws_prometheus_rule_group_namespace" "slo" {
  workspace_id = aws_prometheus_workspace.platform.id
  name         = "slo-rules"

  data = <<-EOT
groups:
  - name: api_slo
    interval: 60s
    rules:
      - record: api:p95_latency:5m
        expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
      - alert: ApiLatencyBudgetBurn
        expr: api:p95_latency:5m > 0.5
        for: 10m
        keep_firing_for: 20m
        labels:
          severity: critical
        annotations:
          summary: "API p95 latency above 500ms for 10m"
EOT
}

resource "aws_prometheus_alert_manager_definition" "sns" {
  workspace_id = aws_prometheus_workspace.platform.id

  definition = <<-EOT
alertmanager_config: |
  route:
    receiver: default
    group_by: [alertname, cluster]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
  receivers:
    - name: default
      sns_configs:
        - topic_arn: ${aws_sns_topic.alerts.arn}
          sigv4:
            region: us-east-1
EOT
}

# IRSA role for the ADOT collector: remote-write only, scoped to one workspace
resource "aws_iam_role" "adot_ingest" {
  name = "amp-iamproxy-ingest-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = { Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(var.oidc_provider_url, "https://", "")}" }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${replace(var.oidc_provider_url, "https://", "")}:sub" = "system:serviceaccount:adot-col:amp-iamproxy-ingest-service-account"
          "${replace(var.oidc_provider_url, "https://", "")}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy" "adot_ingest" {
  name = "amp-remote-write-only"
  role = aws_iam_role.adot_ingest.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Action = [
        "aps:RemoteWrite",
        "aps:GetSeries",
        "aps:GetLabels",
        "aps:GetMetricMetadata"
      ]
      Resource = aws_prometheus_workspace.platform.arn
    }]
  })
}

# Private ingestion path: data plane + control plane + the STS endpoint SigV4 requires
resource "aws_vpc_endpoint" "aps_workspaces" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${data.aws_region.current.name}.aps-workspaces"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.amp_vpce.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "aps_control" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${data.aws_region.current.name}.aps"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.amp_vpce.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "sts" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${data.aws_region.current.name}.sts"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.amp_vpce.id]
  private_dns_enabled = true
}

output "prometheus_endpoint" {
  value       = aws_prometheus_workspace.platform.prometheus_endpoint
  description = "Base URL; append /api/v1/remote_write or /api/v1/query"
}

Two schema notes verified against the provider docs: the logging_configuration.log_group_arn must end in :*, and rule/alert-manager payloads are the same YAML files standalone Prometheus uses — the alert-manager definition wraps them under an alertmanager_config: key and only routes to Amazon SNS topics in the same account. If you need PagerDuty, Opsgenie or Slack natively, AMP cannot be your Alertmanager; fan out from SNS instead.

4.2 The ADOT collector DaemonSet on EKS

ADOT is AWS's downstream OpenTelemetry Collector distribution — v0.51.0 at the time of writing (October 29, 2026). The pipeline is three components: a prometheus receiver (which accepts your existing Prometheus scrape config verbatim, including all relabeling), the prometheusremotewrite exporter, and the sigv4auth extension that signs every export request with credentials from the default AWS chain — on EKS, the IRSA role from the Terraform above.

apiVersion: v1
kind: Namespace
metadata:
  name: adot-col
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: adot-collector-conf
  namespace: adot-col
data:
  adot-collector-config: |
    receivers:
      prometheus:
        config:
          global:
            scrape_interval: 30s
            scrape_timeout: 10s
            # dedup labels: shared cluster name + distinct replica per collector
            external_labels:
              cluster: prod-us-east-1
              __replica__: replica-1
          scrape_configs:
          - job_name: 'kubernetes-service-endpoints'
            metric_relabel_configs:
            - source_labels: [__name__]
              regex: 'go_.*|process_.*'
              action: drop
            kubernetes_sd_configs:
            - role: endpoints
            tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: true
            bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
            relabel_configs:
            - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
              action: keep
              regex: true
    exporters:
      prometheusremotewrite:
        endpoint: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-CHANGEME/api/v1/remote_write"
        auth:
          authenticator: sigv4auth
        namespace: "adot"
    extensions:
      health_check:
      sigv4auth:
        service: "aps"
        region: "us-east-1"
    service:
      extensions: [health_check, sigv4auth]
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [prometheusremotewrite]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: amp-iamproxy-ingest-service-account
  namespace: adot-col
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/amp-iamproxy-ingest-role
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: adotcol-admin-role
rules:
  - apiGroups: [""]
    resources: [nodes, nodes/proxy, services, endpoints, pods]
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: adotcol-admin-role-binding
subjects:
  - kind: ServiceAccount
    name: amp-iamproxy-ingest-service-account
    namespace: adot-col
roleRef:
  kind: ClusterRole
  name: adotcol-admin-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: adot-collector
  namespace: adot-col
spec:
  selector:
    matchLabels:
      app: aws-adot
      component: adot-collector
  template:
    metadata:
      labels:
        app: aws-adot
        component: adot-collector
    spec:
      serviceAccountName: amp-iamproxy-ingest-service-account
      containers:
      - name: adot-collector
        image: public.ecr.aws/aws-observability/aws-otel-collector:v0.51.0
        command: ["/awscollector", "--config=/conf/adot-collector-config.yaml"]
        resources:
          requests:
            cpu: 200m
            memory: 400Mi
          limits:
            cpu: "1"
            memory: 2Gi
        ports:
        - containerPort: 13133
        volumeMounts:
        - name: adot-collector-config-vol
          mountPath: /conf
        livenessProbe:
          httpGet:
            path: /
            port: 13133
        readinessProbe:
          httpGet:
            path: /
            port: 13133
      volumes:
        - name: adot-collector-config-vol
          configMap:
            name: adot-collector-conf
            items:
              - key: adot-collector-config
                path: adot-collector-config.yaml

Three build details that the docs bury and every team hits:

4.3 The alternative: Prometheus in agent mode

If you would rather not introduce an OTel dependency, stock Prometheus works — AMP is just a remote-write target, and Prometheus has native SigV4 signing for it. Run it in agent mode (the --agent flag; no local query path, WAL-only) and point it at the correct endpoint:

global:
  scrape_interval: 30s
  external_labels:
    cluster: prod-us-east-1
    __replica__: replica-1

remote_write:
  - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-CHANGEME/api/v1/remote_write
    sigv4:
      region: us-east-1        # blank access_key/secret_key -> SDK default chain (IRSA on EKS)
    queue_config:
      capacity: 10000
      max_shards: 50
      min_shards: 2
      max_samples_per_send: 2000
      batch_send_deadline: 5s
      min_backoff: 500ms
      max_backoff: 5s
      retry_on_http_429: true  # default is false: without this, throttled writes are not retried
      sample_age_limit: 55m    # AMP refuses samples older than 1h; do not re-send them
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*|process_.*"
        action: drop

Field names matter here because the blog copies circulate the wrong ones: the current queue_config keys are batch_send_deadline, min_backoff, max_backoff, retry_on_http_429 and sample_age_limit — there is no batch_send_wait_ms key in Prometheus 3.x (the Expona example ships it; Prometheus would refuse that config). Two of these fields are load-bearing for AMP specifically: sample_age_limit: 55m aligns the client with AMP's 1-hour staleness refusal so a collector coming back online does not burn shards re-sending samples AMP will discard anyway, and retry_on_http_429: true opts into retrying throttled writes, which the Prometheus default deliberately does not do. Agent mode's own WAL retention truncates at 4 hours by default (--storage.agent.retention.max-time) — on a long outage your samples are gone client-side before AMP's 1-hour rule even gets a say.

4.4 Verifying the pipeline

AMP's query API is plain Prometheus over SigV4, so the fastest end-to-end check is awscurl — the same tool AWS uses in its ADOT guide:

# requires credentials with aps:QueryMetrics on the workspace ARN
awscurl --service="aps" --region="us-east-1" \
  "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-CHANGEME/api/v1/query?query=up"

# expect: {"status":"success","data":{"resultType":"vector","result":[...]}}

# workspace facts from the control plane (no charge; 10 TPS per region quota)
aws amp describe-workspace --workspace-id ws-CHANGEME --query 'workspace.prometheusEndpoint'

For dashboards, Amazon Managed Grafana talks to AMP with SigV4 using its workspace execution role — no static credentials anywhere. The same datasource works in self-managed Grafana via the standard SigV4 auth type, which is the common migration path: keep Grafana, swap the storage behind it.

5. The Cost Model, Computed Both Ways

Every number in this section is decimal-computed from the verified rate card above, using AWS's own workload assumptions (1,000 metrics per node, 744 hours per month, ~2 bytes stored per sample, ~100k samples per panel query). Where AWS publishes a worked example, we recompute it — and one discrepancy is worth knowing about: AWS's pricing page prints $80.93 ingestion for its 10-node example, but 892.8M samples at $0.90 per 10M is $80.35. The page's own arithmetic overshoots by $0.58/month. Use the formula, not the printed example.

Workload (AWS's 10-node example)MeterComputationMonthly cost
10 nodes × 1,000 metrics @ 30sIngestion892.8M samples × $0.90/10M$80.35
Same fleetStorage (month 1)3.34 GB × $0.03/GB$0.10
Storage (150-day equilibrium)10.05 GB × $0.03/GB$0.30
Query (1 user, 2h/day, 20 panels, 60s refresh)7.2B QSP × $0.10/B$0.72
Optional agentless collectorCollector + samples$0.04 × 744h + $0.03 × 89.28$32.44
Total, customer-managed collectoringestion + storage + query$81.17–$81.77

Now scale it to a realistic mid-size fleet — 20 nodes, same density, plus the surrounding stack (3 Grafana editors, 10 viewers, a modest interactive query load):

20-node EKS fleet on AMPComputationMonthly
Ingestion: 1.786B samples (tier 1, under the 2B threshold)1.786B × $0.90/10M$160.70
Storage at 150-day equilibrium50.2 GB × $0.03/GB$1.51
Query: 5 users × 2h/day × 20 panels @ 60s36B QSP × $0.10/B$3.60
Amazon Managed Grafana: 3 editors + 10 viewers3 × $9 + 10 × $5$77.00
AMP stack total$242.81

And the same fleet on Datadog list prices (fetched 2026-11-10) — note this is the honest, assumption-stated version, not a marketing slide:

Same 20-node fleet on Datadog (list prices)ComputationMonthly
Infrastructure Pro, on-demand20 hosts × $18$360.00
Custom metrics (upper bound: all 20k series billable)20,000 / 100 × $5$1,000.00
APM, on-demand20 hosts × $36$720.00
Infrastructure-only upper bound$1,360.00
Infrastructure + APM upper bound$2,080.00
Conservative floor (annual Infra Pro + annual APM, no custom metrics)20 × $15 + 20 × $31$920.00

So where does the "90% savings" land? Against the upper-bound Datadog bill, the AMP stack saves 88.3%. Against infrastructure-only pricing, 82.1%. Against the conservative annual-commit floor, 73.6%. The claim is directionally real but only reaches 90% against the most expensive way to buy the alternative — per-host APM with custom metrics. State your denominator before you quote a percentage.

Two cost behaviors to plan around rather than be surprised by:

Finally, the fact-check on the lead that inspired this piece: Expona's 5,000-series example at 15s intervals produces 864M samples/month. At their assumed $0.30 per million that is $259.20/month (they rounded to "~$270"). At the actual current tier-1 rate it is $77.76 — the stale price overstated AMP by 3.3×. The free tier (40M samples/month) covers about 4.5% of that single 10-node example cluster's volume, so ignore anyone calling AMP "basically free" — it is cheap, not free.

6. Fact-Check: The Lead's Claims vs. Primary Sources

The Expona post is worth reading for its trade-off framing; its configuration details are where it drifts. For a service that restructured pricing and quotas as aggressively as AMP did in 2025, this is what "verified against primary sources" buys you:

Claim in the lead postVerified reality (source)
Remote write URL ends in /api/v1/writeAMP's RemoteWrite URI is /api/v1/remote_write (AMP API reference). A config with /api/v1/write will 404
"Around $0.30 per million samples in most regions"Tier 1 is $0.90 per 10M = $0.09 per million (AMP pricing page, re-read 2026-11-10) — a 70% cut from the old flat rate
"AWS charges for… API queries… a dashboard with 20 panels… incurs 240 queries per hour"Billing is per Query Samples Processed, not per query; the same dashboard costs 240 × ~samples-scanned per hour (pricing page + query-insights guide)
"AWS handles deduplication and consolidation on the backend"Only when HA senders share a cluster label and carry distinct __replica__ labels; otherwise duplicates are billed in full (AMP dedup guide, in bold)
queue_config with batch_send_wait_ms, max_shards: 200No such key exists in Prometheus 3.x; the field is batch_send_deadline and the 3.x default max_shards is 50 (Prometheus configuration reference)
"the queue has finite capacity (default 10,000 samples)… you'll lose recent data"Directionally right but understated: capacity is per shard, and agent mode's WAL truncates at 4h by default — the client loses data before AMP's 1-hour staleness rule even applies
Grafana datasource with sigv4Auth: true via IAMCorrect — SigV4 with the workspace execution role (or default chain in self-managed Grafana) is the supported path

7. Governance at Scale: Label Sets, Query Insights, Rules

Once more than one team writes into a workspace, three AMP features do the work that per-tenant overrides do in Mimir:

Label sets with active-series limits. A label set is a label match expression (e.g. team=payments env=prod) with its own active-series cap. A tenant that explodes past its cap gets only its own series throttled — reason per_labelset_series_limit — while everyone else keeps ingesting. There is also a default bucket limit for series matching no label set, which is the line that saves you from the untagged service nobody claimed. Up to 100 label sets per workspace. CloudWatch vends ActiveSeriesPerLabelSet and DiscardedSamplesPerLabelSet so each tenant's approach to their cap is visible in isolation.

Query insights and per-query throttling. Two QSP thresholds per workspace: queries crossing the warning threshold return a warning header (rendered in Grafana's UI) and are still charged; queries crossing the error threshold are rejected and not charged. That combination — you only pay for queries that actually ran — is the right default posture for a shared workspace. Pair it with query logging, which ships per-query telemetry (expression, samples processed, and for dashboards the dashboardUid and panelId that issued it; for rules the namespace and rule name) to CloudWatch Logs above a QSP threshold you choose:

# log group names starting with /aws/vendedlogs/ avoid the 5120-character
# CloudWatch resource-policy limit that AMP must update on your behalf
aws amp create-query-logging-configuration \
  --workspace-id ws-CHANGEME \
  --destinations '[{"cloudWatchLogs":{"logGroupArn":"arn:aws:logs:us-east-1:111122223333:log-group:/aws/vendedlogs/amp/query-insights"},"filters":{"qspThreshold":10000000}}]'

# per-query limits applied to every query from a Grafana datasource:
# custom query parameters
#   max_samples_processed_warning_threshold=10000000
#   max_samples_processed_error_threshold=50000000

Rules inside the workspace. Rule groups are the same YAML format standalone Prometheus uses (plus AMP supports keep_firing_for), uploaded as namespaces and evaluated by the managed ruler at 30-second minimum intervals. Two workspace-level settings matter more than people expect: the rule query offset — new workspaces default to 60 seconds of evaluation delay to compensate for ingestion lag, and AWS recommends it be at least your out-of-order window, because a rule evaluated against a half-landed minute produces flapping alerts; and the out-of-order time window itself, 60 seconds by default, tunable 0–600. CloudWatch's OutOfOrderSampleAge metric with its p50/p99/max percentile dimension tells you empirically what your window should be.

# BAD: every dashboard panel re-scans raw histogram buckets
histogram_quantile(0.95,
  sum by (le, cluster, namespace) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

# GOOD: one ruler evaluation per minute writes 4 series;
# every panel reads 4 samples instead of thousands
http:p95_latency:5m{cluster="prod-us-east-1"}

8. Production Realities: Where AMP Bites

Throttling is all-or-nothing per request. AMP rate-limits ingestion with a token bucket (bucket capacity = your ingestion-rate quota; refill = the same rate per second). The part that surprises people: if a request arrives with more samples than the bucket currently holds, the entire request is rejected — 101 samples against 100 tokens loses all 101, no partial acceptance. A collector restarting after an outage with a full local buffer therefore dumps a large burst into an empty bucket and can lose whole requests. Smooth the restart: shard your remote-write queues (min_shards > 1), keep max_samples_per_send moderate, and set sample_age_limit: 55m so stale backfill never enters the queue at all. Discarded-by-throttle samples appear in CloudWatch as DiscardedSamples with reason rate_limited — alarm on it before your users notice.

Series-capacity autoscaling has a documented anti-spike rule. Below 5M series, AMP doubles capacity in 30-minute-average steps; above 5M it adds a 10M buffer. But throttle protection is conditional: if you more than double your 30-minute baseline, or exceed it by more than 10M, you throttle even below quota. AWS's guidance is explicit — ramp ingestion gradually, don't relabel 5M series in one deploy. The default 50M active-series quota (request up to 1.5B) auto-adjusts as 1/30 of your series limit for ingestion rate — the two move together.

Know your hard ceilings before the incident, not during. The quotas that actually bite in practice:

Quota (per workspace unless noted)DefaultAdjustable
Active series50M (max 1.5B)Yes
Ingestion rate1,666,666 samples/s (= 1/30 of series limit)Yes
RemoteWrite requests3,000/sYes
QueryMetrics requests300/sYes
Query scan ceiling50M samples / 12M series per 24h per query · 5 GB scannedNo
Query time range95 daysNo
Request size (ingest or query)1 MBNo
Samples older than 1 hourRefused outrightNo
Labels per series · combined label size150 · 7 KB150 yes · 7 KB no
Rules · min evaluation interval2,000 · 30sBoth yes
Concurrent alerts · silences1,000 · 1,000Both yes
Workspaces per region per account · control-plane API rate25 · 10 TPS per region25 yes · 10 yes

Two of those interact in a way worth doing math on: the ingestion-rate quota (1.67M samples/s) divided by the request-rate quota (3,000/s) is ~556 samples per request. Any collector averaging more than ~556 samples per remote-write POST saturates the sample-rate limit long before the request-rate limit — which is why batch-size tuning matters and why "we're nowhere near 3,000 requests per second" is not evidence you are safe from throttling. And note the docs drift: the AMP API reference still states "an ingestion rate of 70,000 samples per second" for RemoteWrite, a figure three years stale against the quotas page's 1,666,666 — trust the quotas page.

The discard taxonomy is your forensic record. When samples vanish, AMP tells you why, via CloudWatch DiscardedSamples/DiscardedSeries reason dimensions. The reasons you will actually meet:

Discard reasonMeaning
greater_than_max_sample_ageSample older than 1 hour — your collector fell behind; align sample_age_limit
rate_limitedToken bucket empty — burst smoothing or a quota increase
per_user_series_limitWorkspace active-series cap hit — cardinality event, alarm on ActiveSeries trend
per_labelset_series_limitOne tenant's label-set cap hit — blast radius worked as designed
per_metric_series_limitOne metric name exploded (usually an unbounded label like user_id)
new-value-for-timestampDuplicate timestamp with different value — often two writers claiming the same series
sample-out-of-order / sample-too-oldOutside the out-of-order window (default 60s) — check OutOfOrderSampleAge p99 before raising it
label_value_too_long · duplicate_label_names · missing_metric_nameMalformed producers — exporter bugs, not capacity

Native histograms get their own metering and their own discard reasons (populated buckets count as 0.25 samples each, empty buckets are free, resolution is auto-reduced past 200 buckets — which CloudWatch surfaces as NativeHistogramReducedResolutionCount).

The hidden-cost checklist nobody puts in the migration deck:

9. Who Should Skip AMP

Skip AMP if you are not AWS-all-in. Workspaces, quotas, PrivateLink, IAM and the collector model are region-locked AWS constructs. GCP and Azure each have their own managed Prometheus-flavored service; a multi-cloud fleet remote-writing into one vendor's regions is a latency and egress design smell. If you need one metrics plane for many clouds, self-hosted Mimir or VictoriaMetrics is the honest answer.

Skip AMP if you are OTLP-native end to end. AMP speaks Prometheus remote write 1.0 — protobuf, snappy, SigV4. It has no OTLP ingestion endpoint, so OTel-native agents must convert to Prometheus exposition format before shipping. If your fleet already emits OTLP and you will never run a Prometheus scrape config, Grafana Cloud, Mimir, or VictoriaMetrics accept OTLP natively and remove a translation hop.

Skip AMP if you need multi-year retention economics or single queries past 95 days. Retention tops out at 1,095 days and per-query time ranges at 95 days, and storage is billed at $0.03/GB-month — fine at months scale, unremarkable against S3-backed compactor economics at years scale with hundreds of billions of samples. That regime is Thanos/Mimir territory by construction.

Skip AMP if Alertmanager-as-a-service with Slack/PagerDuty receivers is a hard requirement. AMP's alert manager routes to Amazon SNS topics in the same account, full stop. Everything else is a bridge you build and own (SNS → Lambda → webhook). If you already operate a Alertmanager fan-out you like, keep it: evaluate rules in AMP, notify through your existing receiver via recording-rule → external watcher, or just point your Alertmanager at a Prometheus that scrapes AMP.

Skip AMP if your fleet is tiny and static. The free tier (40M samples/month) is real but small; below roughly a hundred million samples a month you are paying mostly for Grafana seats, and a single agent-mode Prometheus with 15-day local retention covers a lab cluster for the cost of one small instance — with the trade that you, not AWS, are on call for it. The moment "who patches the metrics database at 2 a.m." has a real answer that is you, AMP's economics start winning.

10. Verdict

AMP in late 2026 is a mature, boring, well-metered metrics store — which is the highest compliment available in observability. The pricing restructure means the savings claims circulating in blog posts are now understated against per-host APM vendors (the real range on list prices is roughly 73–88%, reaching ~90% only against the most expensive alternative configurations), and the new governance surfaces — label sets, query insights with uncharged rejected queries, QSP-metered rules — are the features that make a shared multi-team workspace actually governable rather than just cheap.

The honest procurement position: AMP's cost model rewards exactly the discipline a good platform team already has — controlled cardinality, deliberate scrape intervals, recording rules over repeated dashboard math — and punishes exactly the habits per-host pricing forgave. Migrate the discipline along with the data, or the 90% becomes 40% in a quarter.

Platform Monkey companion pieces: long-term metrics storage with Grafana Mimir (the self-hosted comparison point, including its 400-vs-429 data-loss trap), OpenTelemetry Collector architecture (the component doing the scraping and conversion here), Grafana Cloud vs Datadog pricing (the SaaS comparison frame), eBPF observability deep-dive (kernel-level metrics without exporters), and Prometheus 3.16.0 release notes (the scrape and remote-write client feeding this pipeline).

References & Further Reading