Managed Prometheus on AWS: AMP Pricing Re-Verified, the ADOT Pipeline, and Where the 90% Savings Is Real
Sources
- Amazon Managed Service for Prometheus pricing (rate card, tiers, worked examples)
- AMP user guide — understand and optimize costs
- AMP service quotas (active series, ingestion rate, query limits)
- Configure your AMP workspace (retention, label sets, offsets)
- RemoteWrite API reference (endpoint path and wire protocol)
- Workspace endpoints and dualstack api.aws URLs
- Set up metrics ingestion using AWS Distro for OpenTelemetry on EKS
- Cross-Region HA with ADOT and multiple AMP workspaces
- AMP with interface VPC endpoints (aps and aps-workspaces)
- Deduplicating high-availability metrics (cluster and replica labels)
- Managing the query cost in AMP (query insights, QSP thresholds)
- CloudWatch usage metrics vended by AMP (DiscardedSamples reasons)
- AMP alert manager (SNS-only routing, silences, templating)
- Prometheus configuration reference (remote_write, sigv4, queue_config)
- Prometheus command-line reference (agent mode and agent WAL flags)
- AWS Distro for OpenTelemetry collector (v0.51.0)
- Expona — Managed Prometheus on AWS: practical tips and trade-offs (the lead under review)
- Amazon Managed Grafana pricing (per active user licenses)
- Datadog pricing list (Infrastructure, APM, custom metrics rates)
Every few weeks another "we cut our observability bill by 90% with managed Prometheus on AWS" post makes the rounds. The latest is Expona's "Managed Prometheus on AWS: practical tips and trade-offs" (October 27, 2026), a genuinely useful practitioner's tour of Amazon Managed Service for Prometheus (AMP). It is also a useful specimen of how quickly AMP facts rot: its remote-write URL is the wrong path, its per-sample price is roughly 3× the current rate card, and its mental model of query billing does not match how AWS actually meters queries.
None of that makes the 90% claim wrong — against per-host APM vendors the arithmetic lands close to it. But the claim is only true in one direction, and this industry does a poor job of saying which. So this piece does the boring work: it re-verifies AMP's pricing, quotas, endpoints and collector mechanics against AWS's own current documentation, builds the collector architecture that actually earns the savings (ADOT on EKS, SigV4, PrivateLink), and runs the cost model in both directions — including the one where AMP loses to a couple of EC2 instances and the one where a Grafana wall quietly outbills your ingestion.
Two things changed recently that make this worth revisiting now. First, AMP went through a 2025–2026 maturity wave: the default active-series quota jumped from 10M to 50M per workspace (July 2025) (adjustable to 1.5 billion), query insights and per-query cost controls landed in May 2025, Random Cut Forest anomaly detection shipped in October 2025, and native histogram ingestion is now metered explicitly. Second, the ingestion price itself was restructured into descending volume tiers — which is why almost every blog number you remember is stale. All prices below were re-fetched from AWS's pricing pages on 2026-11-10; the comparison numbers for Datadog come from its live pricing list page the same day.
1. What AMP Actually Is (and Is Not)
AMP is a managed, multi-Availability-Zone Prometheus-compatible metrics store with a managed ruler (rule evaluation engine) and a managed Alertmanager. It is not a managed Prometheus server: nothing scrapes for you unless you opt into the agentless collector, and the thing you are buying is the remote-write endpoint, the storage behind it, and a PromQL engine with per-sample metering on both the write and read sides.
The service splits into two planes, and every quota, IAM policy and PrivateLink endpoint in this article hangs off that split:
- Control plane —
aps.<region>.amazonaws.com. Workspace CRUD, rule group namespaces, alert manager definitions, scrapers, query-logging configuration. Metered against a 10 transactions-per-second-per-region quota across all AMP control-plane APIs. IAM service prefix:aps. - Data plane —
aps-workspaces.<region>.amazonaws.com. The Prometheus-compatible APIs for one workspace:/api/v1/remote_write,/api/v1/query,/api/v1/query_range, series/labels/metadata endpoints. Every URL also exists in a dualstack IPv4+IPv6 form on theaps-workspaces.<region>.api.awsdomain.
The workspace is the unit of isolation, billing scope, quota scope and blast radius. Retention defaults to 150 days and is configurable to a maximum of 1095 days (three years). Everything older than that is deleted; there is no block shipping to S3, no compactor tuning, no object-storage lifecycle to design. That is the deal: you give up the Thanos/Mimir knob collection, and in exchange nobody on your team is on call for a time-series database.
POST /workspaces/ws-b226cc2a-a446-46a9-933a-ac50479a5568/api/v1/remote_write
Authorization: AUTHPARAMS (AWS SigV4, service "aps")
Content-Type: application/x-protobuf
Content-Encoding: snappy
X-Prometheus-Remote-Write-Version: 0.1.0 (remote write 1.0)
Valid URIs : /workspaces/workspaceId/api/v1/remote_write
Verbs : POST
Query URL : /workspaces/workspaceId/api/v1/query
Three properties of that endpoint deserve emphasis because they shape everything downstream. First, the path is /api/v1/remote_write — not /api/v1/write (that is the path of a self-hosted Prometheus receiver started with --web.enable-remote-write-receiver; AMP never used it, and the Expona post ships the wrong path in its example config). Second, requests are limited to 1 MB — batches, not streams. Third, ingestion refuses any sample older than one hour, full stop, non-negotiably; you cannot backfill yesterday's gap into AMP.
2. The Rate Card, Re-Verified
AMP bills four meters, and the tier structure has a quirk that surprises people on their first bill. Prices below are US East (N. Virginia) list rates re-read from the pricing page on 2026-11-10.
| Meter | Rate (us-east-1) | What it counts |
|---|---|---|
| Metric samples ingested | $0.90 per 10M samples, first 2B/mo | Every sample that lands on the remote-write endpoint. Descending tiers above 2B and 252B (tier rates render only in AWS's pricing calculator, but every published example prices the first tier at $0.90/10M) |
| Metrics stored | $0.03 per GB-month | Samples (typically 1–2 bytes each) plus metric metadata (name + label keys/values), 1 GB = 230 bytes |
| Query Samples Processed (QSP) | $0.10 per billion samples | Every data point scanned by a PromQL query — dashboards, scripts, and managed rule evaluations all count. Not "per query" |
| Managed (agentless) collector | $0.04 per collector-hour + $0.03 per 10M samples collected | Optional AWS-run scraper per EKS cluster; you can skip it entirely with your own collector |
| Free tier (monthly) | 40M samples ingested · 200B QSP · 10 GB stored | Permanently available free tier in supported regions |
Four rate-card facts that are routinely misquoted:
- The $0.30-per-million figure is dead. That was the old flat rate everyone memorized. Tier 1 is now $0.90 per 10M samples — $0.09 per million — a ~70% cut. Expona's worked example ("around $0.30 per million samples… about $270 per month") overstates the same workload by 3.3×; we recompute it below at $77.76.
- Your bill is front-loaded by design. Ingestion tiers are descending, tallied per payer account per region, and reset monthly — so early-month usage prices at the top tier and gets cheaper as you cross volume thresholds. AWS's own cost guide says this plainly: expect a higher daily burn in week one than in week four for identical usage.
- Queries are billed per sample scanned, not per request. A 20-panel dashboard on a 5-minute refresh does not incur "240 queries per hour"; it incurs 240 × (samples each panel scans). At roughly 100k samples per panel query, one always-on viewer with a 20-panel dashboard at 60s refresh costs about $8.64/month in QSP — before a single other user opens Grafana.
- Storage is the cheap meter — until retention equilibrium. AWS's pricing example counts one month of samples (~3.3 GB for their 10-node example). With the default 150-day retention, a steady fleet plateaus at ~5 months of samples in storage (the same example: ~10 GB, $0.30/mo). Still cheap; just don't model month-one storage as your steady state.
Two costs that never appear on the AMP rate card but land on your AWS bill anyway: Amazon Managed Grafana licenses you need to look at the data ($9 per active Editor per workspace per month, $5 per active Viewer, minimum one Editor even if nobody logs in), and interface VPC endpoints if you keep ingestion off the public internet ($0.01/hr per AZ per endpoint plus data processing, plus an STS endpoint requirement covered in section 3). Neither is optional in a production build.
3. Reference Architecture: EKS → ADOT → AMP → Managed Grafana
The architecture that earns the savings has five decisions baked in, and each one is a place teams go wrong:
EKS cluster (us-east-1) AWS managed planes
┌───────────────────────────────────────────────┐ ┌─────────────────────────────────────────────┐
│ │ │ │
│ targets: kubelet, cAdvisor, kube-state, │ │ AMP workspace ws-abc123 │
│ app pods w/ prometheus.io/scrape annotations │ │ ┌───────────────────────────────────────┐ │
│ │ scrape (in-cluster, HTTP) │ │ │ dedup (cluster + __replica__ labels) │ │
│ ▼ │ │ │ ruler: rule groups (QSP-metered evals)│ │
│ ADOT Collector DaemonSet (one per node) │ │ │ alert manager: routes → SNS only │ │
│ ┌──────────────────────────────────────┐ │ │ │ storage: 150d default, 1095d max │ │
│ │ prometheus receiver (full Prom config)│ │ │ └───────────────────────────────────────┘ │
│ │ │ relabel: drop noise, cap series │ │ │ ▲ ▲ │
│ │ ▼ │ │ │ │ POST /api/v1/ │ GET /api/v1/ │
│ │ prometheusremotewrite exporter │──────┼───┼──▶ remote_write (1MB cap) │ query+range │
│ │ + sigv4auth ext (service "aps") │ │ │ │ │ │
│ └──────────────────────────────────────┘ │ │ ┌─────┴──────────┐ ┌────┴──────────────┐ │
│ │ IRSA: amp-iamproxy-ingest-role │ │ │ VPC endpoint │ │ Amazon Managed │ │
│ │ aps:RemoteWrite on ws ARN │ │ │ com.amazonaws. │ │ Grafana workspace │ │
│ ▼ │ │ │ us-east-1. │ │ sigv4 datasource │ │
│ VPC endpoint (PrivateLink, no public path) │ │ │ aps-workspaces │ │ $9/editor, $5/vwr│ │
│ + STS endpoint (regional) for token refresh │ │ └────────────────┘ └───────────────────┘ │
└───────────────────────────────────────────────┘ │ CloudWatch: AWS/Usage + AWS/Prometheus │
│ (DiscardedSamples, ActiveSeries, QSP, free) │
└─────────────────────────────────────────────┘
Decision 1 — who scrapes. Three options, and they are not equal. The agentless collector is AWS-run (an ENI per subnet, scraping through your VPC, remote-writing via a VPC endpoint, data never on the public internet) and is billed at $0.04/collector-hour plus $0.03 per 10M samples collected — on AWS's own worked example that is $32.44/month for a 10-node cluster, a 40% surcharge on top of that cluster's $80 ingestion bill. A customer-managed collector — ADOT or raw Prometheus in agent mode — pays no collector fee at all; you fund its CPU/memory inside your cluster instead. For a 100-node fleet the agentless collector's metered sample collection alone ($2.68/node-equivalent per month, scaling linearly) usually beats the convenience; run the numbers before defaulting to it. There is also a middle path: the EKS adot add-on (aws eks create-addon --addon-name adot), which gives you an AWS-patched ADOT operator without you hand-rolling the DaemonSet.
Decision 2 — network path. Remote write works over the public endpoint, but a private cluster should use an interface VPC endpoint. Two service names exist: com.amazonaws.<region>.aps-workspaces for the Prometheus-compatible APIs and com.amazonaws.<region>.aps for workspace management. The trap AWS documents explicitly: in a VPC without internet access, SigV4 still needs to mint credentials, so you must also create an STS interface endpoint and set it regionalized — without it, collectors start fine and then silently fail to authenticate.
Decision 3 — HA topology. AMP replicates across AZs in-region by itself; that is not your job. What is your job is deduplication if you run two collectors scraping the same targets: AMP dedups (and stops double-billing) only when HA pairs share a cluster external label and each instance carries a distinct __replica__ label. Without those labels you pay ingestion and storage twice and Grafana shows two of everything — AWS says this in bold: "If you do not set up deduplication, you will be charged for all data samples." Cross-region HA means remote-writing to a second workspace in another region (the OTel config for two exporters in one pipeline is in AWS's cross-region guide), and yes, that doubles ingestion and storage billing too.
Decision 4 — where rules run. AMP's managed ruler evaluates rule groups inside the workspace, which means rule evaluations are QSP-metered queries. AWS's own example: an alerting rule scanning 10,000 series every minute for a month processes 438M samples ≈ $0.04/month. The counter-intuitive consequence: moving alert rules into AMP is usually a cost reduction versus many dashboards polling the same expression, because one evaluation amortizes across every consumer. Pre-aggregate expensive expressions as recording rules in the workspace and point dashboards at the recording rule.
Decision 5 — cardinality control at the edge. The single highest-leverage line in your entire AMP build is the relabel/metric_relabel stage that drops high-cardinality series before remote write. Every series you drop there is 4 samples/minute (at a 15s interval) never ingested, never stored, never scanned by queries — three meters avoided at once. AWS's own cost guide's first two recommendations are exactly these: lengthen collection intervals and reduce active series.
4. The Build: Terraform, the ADOT DaemonSet, and a Prometheus Alternative
4.1 Provisioning the workspace (Terraform)
The workspace itself is trivial; the production value is in what surrounds it — KMS, vended logs, IAM scoped to one workspace ARN, rule groups as code, and the PrivateLink path. All resources below match the current aws_prometheus_* provider schema (aws_prometheus_workspace, aws_prometheus_rule_group_namespace, aws_prometheus_alert_manager_definition, aws_prometheus_scraper for the agentless collector):
data "aws_caller_identity" "current" {}
data "aws_region" "current" {}
resource "aws_prometheus_workspace" "platform" {
alias = "platform-prod"
kms_key_arn = aws_kms_key.amp.arn
logging_configuration {
# ARN must end with :* so AMP can write into the group
log_group_arn = "${aws_cloudwatch_log_group.amp_vended.arn}:*"
}
tags = { env = "prod", team = "platform" }
}
resource "aws_prometheus_rule_group_namespace" "slo" {
workspace_id = aws_prometheus_workspace.platform.id
name = "slo-rules"
data = <<-EOT
groups:
- name: api_slo
interval: 60s
rules:
- record: api:p95_latency:5m
expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
- alert: ApiLatencyBudgetBurn
expr: api:p95_latency:5m > 0.5
for: 10m
keep_firing_for: 20m
labels:
severity: critical
annotations:
summary: "API p95 latency above 500ms for 10m"
EOT
}
resource "aws_prometheus_alert_manager_definition" "sns" {
workspace_id = aws_prometheus_workspace.platform.id
definition = <<-EOT
alertmanager_config: |
route:
receiver: default
group_by: [alertname, cluster]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receivers:
- name: default
sns_configs:
- topic_arn: ${aws_sns_topic.alerts.arn}
sigv4:
region: us-east-1
EOT
}
# IRSA role for the ADOT collector: remote-write only, scoped to one workspace
resource "aws_iam_role" "adot_ingest" {
name = "amp-iamproxy-ingest-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(var.oidc_provider_url, "https://", "")}" }
Action = "sts:AssumeRoleWithWebIdentity"
Condition = {
StringEquals = {
"${replace(var.oidc_provider_url, "https://", "")}:sub" = "system:serviceaccount:adot-col:amp-iamproxy-ingest-service-account"
"${replace(var.oidc_provider_url, "https://", "")}:aud" = "sts.amazonaws.com"
}
}
}]
})
}
resource "aws_iam_role_policy" "adot_ingest" {
name = "amp-remote-write-only"
role = aws_iam_role.adot_ingest.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = [
"aps:RemoteWrite",
"aps:GetSeries",
"aps:GetLabels",
"aps:GetMetricMetadata"
]
Resource = aws_prometheus_workspace.platform.arn
}]
})
}
# Private ingestion path: data plane + control plane + the STS endpoint SigV4 requires
resource "aws_vpc_endpoint" "aps_workspaces" {
vpc_id = var.vpc_id
service_name = "com.amazonaws.${data.aws_region.current.name}.aps-workspaces"
vpc_endpoint_type = "Interface"
subnet_ids = var.private_subnet_ids
security_group_ids = [aws_security_group.amp_vpce.id]
private_dns_enabled = true
}
resource "aws_vpc_endpoint" "aps_control" {
vpc_id = var.vpc_id
service_name = "com.amazonaws.${data.aws_region.current.name}.aps"
vpc_endpoint_type = "Interface"
subnet_ids = var.private_subnet_ids
security_group_ids = [aws_security_group.amp_vpce.id]
private_dns_enabled = true
}
resource "aws_vpc_endpoint" "sts" {
vpc_id = var.vpc_id
service_name = "com.amazonaws.${data.aws_region.current.name}.sts"
vpc_endpoint_type = "Interface"
subnet_ids = var.private_subnet_ids
security_group_ids = [aws_security_group.amp_vpce.id]
private_dns_enabled = true
}
output "prometheus_endpoint" {
value = aws_prometheus_workspace.platform.prometheus_endpoint
description = "Base URL; append /api/v1/remote_write or /api/v1/query"
}
Two schema notes verified against the provider docs: the logging_configuration.log_group_arn must end in :*, and rule/alert-manager payloads are the same YAML files standalone Prometheus uses — the alert-manager definition wraps them under an alertmanager_config: key and only routes to Amazon SNS topics in the same account. If you need PagerDuty, Opsgenie or Slack natively, AMP cannot be your Alertmanager; fan out from SNS instead.
4.2 The ADOT collector DaemonSet on EKS
ADOT is AWS's downstream OpenTelemetry Collector distribution — v0.51.0 at the time of writing (October 29, 2026). The pipeline is three components: a prometheus receiver (which accepts your existing Prometheus scrape config verbatim, including all relabeling), the prometheusremotewrite exporter, and the sigv4auth extension that signs every export request with credentials from the default AWS chain — on EKS, the IRSA role from the Terraform above.
apiVersion: v1
kind: Namespace
metadata:
name: adot-col
---
apiVersion: v1
kind: ConfigMap
metadata:
name: adot-collector-conf
namespace: adot-col
data:
adot-collector-config: |
receivers:
prometheus:
config:
global:
scrape_interval: 30s
scrape_timeout: 10s
# dedup labels: shared cluster name + distinct replica per collector
external_labels:
cluster: prod-us-east-1
__replica__: replica-1
scrape_configs:
- job_name: 'kubernetes-service-endpoints'
metric_relabel_configs:
- source_labels: [__name__]
regex: 'go_.*|process_.*'
action: drop
kubernetes_sd_configs:
- role: endpoints
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
exporters:
prometheusremotewrite:
endpoint: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-CHANGEME/api/v1/remote_write"
auth:
authenticator: sigv4auth
namespace: "adot"
extensions:
health_check:
sigv4auth:
service: "aps"
region: "us-east-1"
service:
extensions: [health_check, sigv4auth]
pipelines:
metrics:
receivers: [prometheus]
exporters: [prometheusremotewrite]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: amp-iamproxy-ingest-service-account
namespace: adot-col
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/amp-iamproxy-ingest-role
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: adotcol-admin-role
rules:
- apiGroups: [""]
resources: [nodes, nodes/proxy, services, endpoints, pods]
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: adotcol-admin-role-binding
subjects:
- kind: ServiceAccount
name: amp-iamproxy-ingest-service-account
namespace: adot-col
roleRef:
kind: ClusterRole
name: adotcol-admin-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: adot-collector
namespace: adot-col
spec:
selector:
matchLabels:
app: aws-adot
component: adot-collector
template:
metadata:
labels:
app: aws-adot
component: adot-collector
spec:
serviceAccountName: amp-iamproxy-ingest-service-account
containers:
- name: adot-collector
image: public.ecr.aws/aws-observability/aws-otel-collector:v0.51.0
command: ["/awscollector", "--config=/conf/adot-collector-config.yaml"]
resources:
requests:
cpu: 200m
memory: 400Mi
limits:
cpu: "1"
memory: 2Gi
ports:
- containerPort: 13133
volumeMounts:
- name: adot-collector-config-vol
mountPath: /conf
livenessProbe:
httpGet:
path: /
port: 13133
readinessProbe:
httpGet:
path: /
port: 13133
volumes:
- name: adot-collector-config-vol
configMap:
name: adot-collector-conf
items:
- key: adot-collector-config
path: adot-collector-config.yaml
Three build details that the docs bury and every team hits:
- The double-dollar rule. Because the OTel collector expands environment variables, any
$in your pasted Prometheus config must become$$— most commonly in relabelreplacement: $${1}://...targets. A single$silently swallows your replacement value. sigv4auth.servicemust beaps(the IAM service name), with the region set — AWS's own text: "You must specify the service to beaps." The exporter then needsauth: { authenticator: sigv4auth }to actually use it.- Scrape cost is a config decision. This manifest sets
scrape_interval: 30sand dropsgo_*/process_*boilerplate atmetric_relabel_configs— the pre-write stage. Halving the interval doubles ingestion cost with zero benefit for most SLO signals; AWS's cost guide ranks "reduce collection frequency" and "reduce active series" as the two effective levers.
4.3 The alternative: Prometheus in agent mode
If you would rather not introduce an OTel dependency, stock Prometheus works — AMP is just a remote-write target, and Prometheus has native SigV4 signing for it. Run it in agent mode (the --agent flag; no local query path, WAL-only) and point it at the correct endpoint:
global:
scrape_interval: 30s
external_labels:
cluster: prod-us-east-1
__replica__: replica-1
remote_write:
- url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-CHANGEME/api/v1/remote_write
sigv4:
region: us-east-1 # blank access_key/secret_key -> SDK default chain (IRSA on EKS)
queue_config:
capacity: 10000
max_shards: 50
min_shards: 2
max_samples_per_send: 2000
batch_send_deadline: 5s
min_backoff: 500ms
max_backoff: 5s
retry_on_http_429: true # default is false: without this, throttled writes are not retried
sample_age_limit: 55m # AMP refuses samples older than 1h; do not re-send them
write_relabel_configs:
- source_labels: [__name__]
regex: "go_.*|process_.*"
action: drop
Field names matter here because the blog copies circulate the wrong ones: the current queue_config keys are batch_send_deadline, min_backoff, max_backoff, retry_on_http_429 and sample_age_limit — there is no batch_send_wait_ms key in Prometheus 3.x (the Expona example ships it; Prometheus would refuse that config). Two of these fields are load-bearing for AMP specifically: sample_age_limit: 55m aligns the client with AMP's 1-hour staleness refusal so a collector coming back online does not burn shards re-sending samples AMP will discard anyway, and retry_on_http_429: true opts into retrying throttled writes, which the Prometheus default deliberately does not do. Agent mode's own WAL retention truncates at 4 hours by default (--storage.agent.retention.max-time) — on a long outage your samples are gone client-side before AMP's 1-hour rule even gets a say.
4.4 Verifying the pipeline
AMP's query API is plain Prometheus over SigV4, so the fastest end-to-end check is awscurl — the same tool AWS uses in its ADOT guide:
# requires credentials with aps:QueryMetrics on the workspace ARN
awscurl --service="aps" --region="us-east-1" \
"https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-CHANGEME/api/v1/query?query=up"
# expect: {"status":"success","data":{"resultType":"vector","result":[...]}}
# workspace facts from the control plane (no charge; 10 TPS per region quota)
aws amp describe-workspace --workspace-id ws-CHANGEME --query 'workspace.prometheusEndpoint'
For dashboards, Amazon Managed Grafana talks to AMP with SigV4 using its workspace execution role — no static credentials anywhere. The same datasource works in self-managed Grafana via the standard SigV4 auth type, which is the common migration path: keep Grafana, swap the storage behind it.
5. The Cost Model, Computed Both Ways
Every number in this section is decimal-computed from the verified rate card above, using AWS's own workload assumptions (1,000 metrics per node, 744 hours per month, ~2 bytes stored per sample, ~100k samples per panel query). Where AWS publishes a worked example, we recompute it — and one discrepancy is worth knowing about: AWS's pricing page prints $80.93 ingestion for its 10-node example, but 892.8M samples at $0.90 per 10M is $80.35. The page's own arithmetic overshoots by $0.58/month. Use the formula, not the printed example.
| Workload (AWS's 10-node example) | Meter | Computation | Monthly cost |
|---|---|---|---|
| 10 nodes × 1,000 metrics @ 30s | Ingestion | 892.8M samples × $0.90/10M | $80.35 |
| Same fleet | Storage (month 1) | 3.34 GB × $0.03/GB | $0.10 |
| Storage (150-day equilibrium) | 10.05 GB × $0.03/GB | $0.30 | |
| Query (1 user, 2h/day, 20 panels, 60s refresh) | 7.2B QSP × $0.10/B | $0.72 | |
| Optional agentless collector | Collector + samples | $0.04 × 744h + $0.03 × 89.28 | $32.44 |
| Total, customer-managed collector | ingestion + storage + query | $81.17–$81.77 | |
Now scale it to a realistic mid-size fleet — 20 nodes, same density, plus the surrounding stack (3 Grafana editors, 10 viewers, a modest interactive query load):
| 20-node EKS fleet on AMP | Computation | Monthly |
|---|---|---|
| Ingestion: 1.786B samples (tier 1, under the 2B threshold) | 1.786B × $0.90/10M | $160.70 |
| Storage at 150-day equilibrium | 50.2 GB × $0.03/GB | $1.51 |
| Query: 5 users × 2h/day × 20 panels @ 60s | 36B QSP × $0.10/B | $3.60 |
| Amazon Managed Grafana: 3 editors + 10 viewers | 3 × $9 + 10 × $5 | $77.00 |
| AMP stack total | $242.81 |
And the same fleet on Datadog list prices (fetched 2026-11-10) — note this is the honest, assumption-stated version, not a marketing slide:
| Same 20-node fleet on Datadog (list prices) | Computation | Monthly |
|---|---|---|
| Infrastructure Pro, on-demand | 20 hosts × $18 | $360.00 |
| Custom metrics (upper bound: all 20k series billable) | 20,000 / 100 × $5 | $1,000.00 |
| APM, on-demand | 20 hosts × $36 | $720.00 |
| Infrastructure-only upper bound | $1,360.00 | |
| Infrastructure + APM upper bound | $2,080.00 | |
| Conservative floor (annual Infra Pro + annual APM, no custom metrics) | 20 × $15 + 20 × $31 | $920.00 |
So where does the "90% savings" land? Against the upper-bound Datadog bill, the AMP stack saves 88.3%. Against infrastructure-only pricing, 82.1%. Against the conservative annual-commit floor, 73.6%. The claim is directionally real but only reaches 90% against the most expensive way to buy the alternative — per-host APM with custom metrics. State your denominator before you quote a percentage.
Two cost behaviors to plan around rather than be surprised by:
- Query cost can outrun ingestion. A 24/7 NOC wall — 10 viewers, 20 panels, 60-second refresh, ~100k samples per query — processes 864B samples per month: $86.40 in QSP, which is 35% of that fleet's query-plus-ingestion combined bill and more than half its ingestion line. Grafana caching and lower refresh rates on walls are cost controls, not cosmetic settings. AMP's query insights (section 6) exist precisely to catch this.
- The interval lever halves everything. Moving 20 nodes from 30s to 60s scrape intervals drops ingestion from $160.70 to $80.35 with zero architectural change. Most capacity-planning and SLO signals do not need 30s resolution; alert latency on a 5m-window rule is not meaningfully improved by it either.
Finally, the fact-check on the lead that inspired this piece: Expona's 5,000-series example at 15s intervals produces 864M samples/month. At their assumed $0.30 per million that is $259.20/month (they rounded to "~$270"). At the actual current tier-1 rate it is $77.76 — the stale price overstated AMP by 3.3×. The free tier (40M samples/month) covers about 4.5% of that single 10-node example cluster's volume, so ignore anyone calling AMP "basically free" — it is cheap, not free.
6. Fact-Check: The Lead's Claims vs. Primary Sources
The Expona post is worth reading for its trade-off framing; its configuration details are where it drifts. For a service that restructured pricing and quotas as aggressively as AMP did in 2025, this is what "verified against primary sources" buys you:
| Claim in the lead post | Verified reality (source) |
|---|---|
Remote write URL ends in /api/v1/write | AMP's RemoteWrite URI is /api/v1/remote_write (AMP API reference). A config with /api/v1/write will 404 |
| "Around $0.30 per million samples in most regions" | Tier 1 is $0.90 per 10M = $0.09 per million (AMP pricing page, re-read 2026-11-10) — a 70% cut from the old flat rate |
| "AWS charges for… API queries… a dashboard with 20 panels… incurs 240 queries per hour" | Billing is per Query Samples Processed, not per query; the same dashboard costs 240 × ~samples-scanned per hour (pricing page + query-insights guide) |
| "AWS handles deduplication and consolidation on the backend" | Only when HA senders share a cluster label and carry distinct __replica__ labels; otherwise duplicates are billed in full (AMP dedup guide, in bold) |
queue_config with batch_send_wait_ms, max_shards: 200 | No such key exists in Prometheus 3.x; the field is batch_send_deadline and the 3.x default max_shards is 50 (Prometheus configuration reference) |
| "the queue has finite capacity (default 10,000 samples)… you'll lose recent data" | Directionally right but understated: capacity is per shard, and agent mode's WAL truncates at 4h by default — the client loses data before AMP's 1-hour staleness rule even applies |
Grafana datasource with sigv4Auth: true via IAM | Correct — SigV4 with the workspace execution role (or default chain in self-managed Grafana) is the supported path |
7. Governance at Scale: Label Sets, Query Insights, Rules
Once more than one team writes into a workspace, three AMP features do the work that per-tenant overrides do in Mimir:
Label sets with active-series limits. A label set is a label match expression (e.g. team=payments env=prod) with its own active-series cap. A tenant that explodes past its cap gets only its own series throttled — reason per_labelset_series_limit — while everyone else keeps ingesting. There is also a default bucket limit for series matching no label set, which is the line that saves you from the untagged service nobody claimed. Up to 100 label sets per workspace. CloudWatch vends ActiveSeriesPerLabelSet and DiscardedSamplesPerLabelSet so each tenant's approach to their cap is visible in isolation.
Query insights and per-query throttling. Two QSP thresholds per workspace: queries crossing the warning threshold return a warning header (rendered in Grafana's UI) and are still charged; queries crossing the error threshold are rejected and not charged. That combination — you only pay for queries that actually ran — is the right default posture for a shared workspace. Pair it with query logging, which ships per-query telemetry (expression, samples processed, and for dashboards the dashboardUid and panelId that issued it; for rules the namespace and rule name) to CloudWatch Logs above a QSP threshold you choose:
# log group names starting with /aws/vendedlogs/ avoid the 5120-character
# CloudWatch resource-policy limit that AMP must update on your behalf
aws amp create-query-logging-configuration \
--workspace-id ws-CHANGEME \
--destinations '[{"cloudWatchLogs":{"logGroupArn":"arn:aws:logs:us-east-1:111122223333:log-group:/aws/vendedlogs/amp/query-insights"},"filters":{"qspThreshold":10000000}}]'
# per-query limits applied to every query from a Grafana datasource:
# custom query parameters
# max_samples_processed_warning_threshold=10000000
# max_samples_processed_error_threshold=50000000
Rules inside the workspace. Rule groups are the same YAML format standalone Prometheus uses (plus AMP supports keep_firing_for), uploaded as namespaces and evaluated by the managed ruler at 30-second minimum intervals. Two workspace-level settings matter more than people expect: the rule query offset — new workspaces default to 60 seconds of evaluation delay to compensate for ingestion lag, and AWS recommends it be at least your out-of-order window, because a rule evaluated against a half-landed minute produces flapping alerts; and the out-of-order time window itself, 60 seconds by default, tunable 0–600. CloudWatch's OutOfOrderSampleAge metric with its p50/p99/max percentile dimension tells you empirically what your window should be.
# BAD: every dashboard panel re-scans raw histogram buckets
histogram_quantile(0.95,
sum by (le, cluster, namespace) (
rate(http_request_duration_seconds_bucket[5m])
)
)
# GOOD: one ruler evaluation per minute writes 4 series;
# every panel reads 4 samples instead of thousands
http:p95_latency:5m{cluster="prod-us-east-1"}
8. Production Realities: Where AMP Bites
Throttling is all-or-nothing per request. AMP rate-limits ingestion with a token bucket (bucket capacity = your ingestion-rate quota; refill = the same rate per second). The part that surprises people: if a request arrives with more samples than the bucket currently holds, the entire request is rejected — 101 samples against 100 tokens loses all 101, no partial acceptance. A collector restarting after an outage with a full local buffer therefore dumps a large burst into an empty bucket and can lose whole requests. Smooth the restart: shard your remote-write queues (min_shards > 1), keep max_samples_per_send moderate, and set sample_age_limit: 55m so stale backfill never enters the queue at all. Discarded-by-throttle samples appear in CloudWatch as DiscardedSamples with reason rate_limited — alarm on it before your users notice.
Series-capacity autoscaling has a documented anti-spike rule. Below 5M series, AMP doubles capacity in 30-minute-average steps; above 5M it adds a 10M buffer. But throttle protection is conditional: if you more than double your 30-minute baseline, or exceed it by more than 10M, you throttle even below quota. AWS's guidance is explicit — ramp ingestion gradually, don't relabel 5M series in one deploy. The default 50M active-series quota (request up to 1.5B) auto-adjusts as 1/30 of your series limit for ingestion rate — the two move together.
Know your hard ceilings before the incident, not during. The quotas that actually bite in practice:
| Quota (per workspace unless noted) | Default | Adjustable |
|---|---|---|
| Active series | 50M (max 1.5B) | Yes |
| Ingestion rate | 1,666,666 samples/s (= 1/30 of series limit) | Yes |
| RemoteWrite requests | 3,000/s | Yes |
| QueryMetrics requests | 300/s | Yes |
| Query scan ceiling | 50M samples / 12M series per 24h per query · 5 GB scanned | No |
| Query time range | 95 days | No |
| Request size (ingest or query) | 1 MB | No |
| Samples older than 1 hour | Refused outright | No |
| Labels per series · combined label size | 150 · 7 KB | 150 yes · 7 KB no |
| Rules · min evaluation interval | 2,000 · 30s | Both yes |
| Concurrent alerts · silences | 1,000 · 1,000 | Both yes |
| Workspaces per region per account · control-plane API rate | 25 · 10 TPS per region | 25 yes · 10 yes |
Two of those interact in a way worth doing math on: the ingestion-rate quota (1.67M samples/s) divided by the request-rate quota (3,000/s) is ~556 samples per request. Any collector averaging more than ~556 samples per remote-write POST saturates the sample-rate limit long before the request-rate limit — which is why batch-size tuning matters and why "we're nowhere near 3,000 requests per second" is not evidence you are safe from throttling. And note the docs drift: the AMP API reference still states "an ingestion rate of 70,000 samples per second" for RemoteWrite, a figure three years stale against the quotas page's 1,666,666 — trust the quotas page.
The discard taxonomy is your forensic record. When samples vanish, AMP tells you why, via CloudWatch DiscardedSamples/DiscardedSeries reason dimensions. The reasons you will actually meet:
| Discard reason | Meaning |
|---|---|
greater_than_max_sample_age | Sample older than 1 hour — your collector fell behind; align sample_age_limit |
rate_limited | Token bucket empty — burst smoothing or a quota increase |
per_user_series_limit | Workspace active-series cap hit — cardinality event, alarm on ActiveSeries trend |
per_labelset_series_limit | One tenant's label-set cap hit — blast radius worked as designed |
per_metric_series_limit | One metric name exploded (usually an unbounded label like user_id) |
new-value-for-timestamp | Duplicate timestamp with different value — often two writers claiming the same series |
sample-out-of-order / sample-too-old | Outside the out-of-order window (default 60s) — check OutOfOrderSampleAge p99 before raising it |
label_value_too_long · duplicate_label_names · missing_metric_name | Malformed producers — exporter bugs, not capacity |
Native histograms get their own metering and their own discard reasons (populated buckets count as 0.25 samples each, empty buckets are free, resolution is auto-reduced past 200 buckets — which CloudWatch surfaces as NativeHistogramReducedResolutionCount).
The hidden-cost checklist nobody puts in the migration deck:
- Amazon Managed Grafana charges per active user per workspace, minimum one $9 Editor even in a month where nobody logs in. An API key or service account counts as a user at $9/$5. Ten idle-viewer SSO groups cost nothing until someone logs in — then they are $5 each.
- Cross-region HA (dual workspaces) doubles ingestion and storage. Dedup only protects you within a workspace.
- Kubernetes 1.28/1.29 clusters can emit their own
clusterlabel, which collides with the dedup label scheme AWS documents — check the HA FAQ before assuming your labels are unique. - Anomaly detectors are not free: each detector creates four new time series (the anomaly and confidence bands) that are ingested, stored and queryable like any other series.
- Query logging writes to CloudWatch Logs on your dime once past the vended-metrics free tier (the AMP usage metrics themselves — ActiveSeries, DiscardedSamples, QuerySamplesProcessed, RuleEvaluations — are vended at no charge).
- VPC endpoints bill $0.01/hour per endpoint per AZ; a three-AZ deployment with aps + aps-workspaces + STS is nine endpoint-AZs before you ingest a single sample.
9. Who Should Skip AMP
Skip AMP if you are not AWS-all-in. Workspaces, quotas, PrivateLink, IAM and the collector model are region-locked AWS constructs. GCP and Azure each have their own managed Prometheus-flavored service; a multi-cloud fleet remote-writing into one vendor's regions is a latency and egress design smell. If you need one metrics plane for many clouds, self-hosted Mimir or VictoriaMetrics is the honest answer.
Skip AMP if you are OTLP-native end to end. AMP speaks Prometheus remote write 1.0 — protobuf, snappy, SigV4. It has no OTLP ingestion endpoint, so OTel-native agents must convert to Prometheus exposition format before shipping. If your fleet already emits OTLP and you will never run a Prometheus scrape config, Grafana Cloud, Mimir, or VictoriaMetrics accept OTLP natively and remove a translation hop.
Skip AMP if you need multi-year retention economics or single queries past 95 days. Retention tops out at 1,095 days and per-query time ranges at 95 days, and storage is billed at $0.03/GB-month — fine at months scale, unremarkable against S3-backed compactor economics at years scale with hundreds of billions of samples. That regime is Thanos/Mimir territory by construction.
Skip AMP if Alertmanager-as-a-service with Slack/PagerDuty receivers is a hard requirement. AMP's alert manager routes to Amazon SNS topics in the same account, full stop. Everything else is a bridge you build and own (SNS → Lambda → webhook). If you already operate a Alertmanager fan-out you like, keep it: evaluate rules in AMP, notify through your existing receiver via recording-rule → external watcher, or just point your Alertmanager at a Prometheus that scrapes AMP.
Skip AMP if your fleet is tiny and static. The free tier (40M samples/month) is real but small; below roughly a hundred million samples a month you are paying mostly for Grafana seats, and a single agent-mode Prometheus with 15-day local retention covers a lab cluster for the cost of one small instance — with the trade that you, not AWS, are on call for it. The moment "who patches the metrics database at 2 a.m." has a real answer that is you, AMP's economics start winning.
10. Verdict
AMP in late 2026 is a mature, boring, well-metered metrics store — which is the highest compliment available in observability. The pricing restructure means the savings claims circulating in blog posts are now understated against per-host APM vendors (the real range on list prices is roughly 73–88%, reaching ~90% only against the most expensive alternative configurations), and the new governance surfaces — label sets, query insights with uncharged rejected queries, QSP-metered rules — are the features that make a shared multi-team workspace actually governable rather than just cheap.
The honest procurement position: AMP's cost model rewards exactly the discipline a good platform team already has — controlled cardinality, deliberate scrape intervals, recording rules over repeated dashboard math — and punishes exactly the habits per-host pricing forgave. Migrate the discipline along with the data, or the 90% becomes 40% in a quarter.
Platform Monkey companion pieces: long-term metrics storage with Grafana Mimir (the self-hosted comparison point, including its 400-vs-429 data-loss trap), OpenTelemetry Collector architecture (the component doing the scraping and conversion here), Grafana Cloud vs Datadog pricing (the SaaS comparison frame), eBPF observability deep-dive (kernel-level metrics without exporters), and Prometheus 3.16.0 release notes (the scrape and remote-write client feeding this pipeline).
References & Further Reading
- Amazon Managed Service for Prometheus pricing — rate card, volume tiers, and AWS's worked examples (re-read 2026-11-10)
- Understand and optimize costs in AMP — AWS's own cost-lever guidance: intervals and active series first
- AMP service quotas — the full quota table, capacity autoscaling, and token-bucket mechanics
- Configure your workspace — retention (150d default, 1095d max), label sets, out-of-order window, rule query offset
- RemoteWrite API reference — endpoint paths, wire headers, and the stale 70k/s figure
- Workspace endpoints — query/remote-write URLs and dualstack api.aws forms
- ADOT ingestion on EKS — the three-component pipeline, IRSA policy, and the $$-escaping rule
- Cross-region HA with ADOT — dual-exporter OTel config for two workspaces
- AMP with interface VPC endpoints — aps and aps-workspaces service names, STS requirement, endpoint policies
- Deduplicating HA metrics — cluster + __replica__ labels, and the double-billing warning
- Managing query cost — QSP warning/error thresholds and query logging configuration
- CloudWatch usage metrics — vended metrics and the full discard-reason taxonomy
- AMP alert manager — SNS-only receivers, silences API, templating
- AWS managed (agentless) collectors — ENI-per-subnet scraping, VPC-path remote write
- 50M default active series limit (July 2025) — quota history
- AMP anomaly detection (October 2025) — Random Cut Forest detectors and the four series they create
- Prometheus configuration reference — remote_write, sigv4, queue_config fields as of 3.x
- Prometheus command-line reference — --agent and agent WAL retention flags
- aws-otel-collector on GitHub — ADOT source, releases (v0.51.0, October 2026), example manifests
- Amazon EKS add-ons — the adot add-on and its cert-manager prerequisite
- Terraform aws_prometheus_workspace — workspace, rule group namespace, and alert manager definition resources
- Amazon Managed Grafana pricing — per-active-user Editor/Viewer licenses and minimums
- Datadog pricing list — Infrastructure, APM, and custom-metric rates used in the comparison (fetched 2026-11-10)
- Expona — Managed Prometheus on AWS — the October 2026 practitioner post this guide fact-checks