LLM Observability Tools in 2026: LangSmith vs. Langfuse vs. Arize AX vs. Datadog Agent Observability

Buyer's guide, verified November 7, 2026. Every price and product claim below was checked against the vendor's own pricing page or documentation on this date — where a vendor publishes nothing, we say so instead of guessing. This category repriced repeatedly through 2026, so treat every number as a starting point for your own verification, not a quote.

You are shopping for LLM observability because debugging agents from log lines has stopped scaling: multi-step traces where one failed hop poisons a whole run, evals you cannot correlate with the traces they scored, token spend nobody can attribute, and a growing suspicion that your bill is a function of telemetry nobody reads. This guide compares the four surfaces senior teams actually choose between in 2026 — LangSmith, Langfuse, Arize AX, and Datadog Agent Observability — plus the OpenTelemetry exit ramp that keeps any of them honest. Every verdict carries honest weaknesses and a who-should-skip, because a page that reads like an ad does not help you spend money well.

Why this comparison, right now

Three things changed in 2026 that invalidate most comparison content written before mid-year:

• Datadog renamed and repriced its LLM product. "LLM Observability" is now "Agent Observability" — the old product URL 301-redirects to the new page (verified live today). The current pricing list meters LLM spans with a free allowance footnoted at 40k LLM spans per month, and a Pro base of $160/month on annual commitment ($200 month-to-month, $240 on-demand) that includes the first 100k LLM spans, then $3.50 per 10k at the default 15-day retention. Third-party analysis checked against archived pricing pages (TrueFoundry) dates the switch to between February and July 2026 and documents the retired model: $8 per 10K LLM requests with a 100k/month contractual minimum — a floor that punished exactly the pilots this product is meant for.
• LangSmith capped SaaS retention. Per the official billing docs, starting September 14, 2026 the maximum long-lived trace retention for SaaS customers is 180 days (Enterprise can still customize per workspace). Traces now come in two tiers — base at 14-day retention, extended at 180-day — and several features silently upgrade traces to the pricier extended tier (details in the traps section).
• The OpenTelemetry GenAI conventions split out — and still have no release. All GenAI semantic conventions were deprecated in the core repository and moved to a dedicated repository in v1.42.0 (published June 12, 2026). As of today, that repository — created May 5, 2026 — has zero tagged releases, which means every vendor's "OTel-compatible" claim is a marketing statement, not a schema contract you can pin.

The billable-unit trap: one agent run, four different bills

The most expensive mistake in this category is comparing seat prices. Nobody here is really selling seats — they are selling units, and the units are different things:

• LangSmith bills per trace. A trace is "a single execution of your application — whether it's an agent, evaluator, or playground session" and "can include many individual steps." In practice: one agent run, one billable unit, however many LLM calls and tool hops it contains.
• Langfuse bills per unit, defined on its pricing page as "any tracing data point you send to our platform — this includes the trace (a complete application interaction), observations (individual steps within a trace: Spans, Events and Generations), and scores (evaluations of your AI outputs)."
• Datadog bills per LLM span. Their docs are explicit: "Agent Observability is metered and billed on the number of LLM spans ingested. An LLM span represents a single request to an LLM provider, so one agent workflow can produce several LLM spans." Tool, retrieval, and embedding spans do not bill — a genuinely favorable quirk for agent workloads.
• Arize AX caps per span. Plan limits are expressed in "trace spans" per month — all span kinds count against the cap.

ONE AGENT RUN
  [llm]---> [llm]---> [tool]---> [llm]---> [tool]---> [llm]---> [llm]---+
       5 LLM calls            2 tool steps              + 1 eval score

BILLABLE UNITS FOR THIS ONE RUN
  Vendor    Counts what?                        Units   List price (verified)
  ----------------------------------------------------------------------
  LangSmith whole trace (the run itself)            1    $0.005 per base trace
  Langfuse  trace + observations + scores          11    $8 per 100k units
  Datadog   LLM spans only (model calls)            5    $3.50 per 10k spans
  Arize AX  all traced spans                       10    plan cap, then quote

  Same telemetry. Same month. A 5.7x spread in what lands on the invoice.

At production volume this compounds. In our modeled production workload below (500k runs/month, 5 LLM calls each), the identical traffic costs $2,645/month on LangSmith Plus, $381/month on Langfuse Core, and $1,000/month on Datadog — before retention choices. That is a 6.9x spread for the same observability, driven almost entirely by unit definitions, not by feature gaps.

The 60-second comparison

LangSmith: the deepest tracing UI, priced per run

LangSmith is LangChain's platform: tracing, evaluation datasets, prompt hub, annotation queues, and in 2026 a growing agent-runtime surface (Deployments, Engine, Fleet, Sandboxes — all metered in LangChain Standard Units, where 1 LSU = $1). The tracing product is genuinely best-in-class for depth: nested agent graphs, per-step inputs/outputs, feedback threads, and tight integration if your stack is already LangChain-shaped.

The trap: traces bill at $0.005 per base trace (14-day retention) or effectively $0.010 fully extended (180-day), and the billing docs state that online evaluators and automation rules upgrade traces to extended retention by default ("retention extension is enabled by default for new online evaluators and automation rules"). In our production model, a team that leaves evaluators on default settings roughly doubles its trace bill — $2,645/month becomes $5,095/month — without changing a single line of code. Also note the seat definition: "A seat is a distinct user inside your organization. We consider the total number of users (including invited users) to determine the number of seats to bill."

Who should skip LangSmith: teams running high-volume, multi-call agent traffic on a thin budget — per-run billing is the most expensive unit model of the four at scale. Teams that need SaaS trace retention beyond 180 days for compliance. And teams not on LangChain frameworks who would pay the same trace rates without the framework ergonomics that justify them.

Langfuse: open-source core, unit economics that win on volume

Langfuse is the open-source-first option (37,127 GitHub stars; MIT-licensed core) with cloud plans that undercut everyone on volume. The unit is "any tracing data point" — traces, observations (spans, events, generations), and scores — with graduated rates that get cheaper as you grow.

The graduation is aggressive in your favor: $8 per 100k units from 100k-1M, $7 from 1M-10M, $6.50 from 10M-50M, and $6 beyond — identical rates on every paid plan, so you upgrade for retention, throughput, and compliance, never for cheaper volume. A sanity check that built confidence in these numbers: our cost model reproduces Langfuse's own published example exactly — 5M units on Pro totals $551/month, matching their pricing page to the dollar.

The self-host card is the strategic one: Langfuse OSS is MIT-licensed with unlimited usage and — unusually for this market — Enterprise SSO and RBAC included in the free self-hosted edition, deployable via Docker Compose or Kubernetes templates. The paid self-hosted Enterprise tier adds management APIs, retention management, audit logs, and support, and ships bundled with ClickHouse Cloud, BYOC, or private deployment. The trade: you own the ClickHouse/Postgres operational surface, and the cloud convenience (managed ingestion at 4k-20k req/min) is exactly what you give up.

Who should skip Langfuse: teams that want one vendor for the platform and the agents — Langfuse has no APM, infra, or log plane, so it is a second tool by design. And eval-heavy teams should model carefully: every score bills as a unit, so a program of online LLM-as-judge evals on every trace adds a 1-unit tax on top of each observation.

Arize AX: the best free toolkit, the shortest self-serve runway

Arize's story is two products. Phoenix is the open-source tracing, evaluation, and experimentation toolkit (currently v21.0.0 on PyPI, Elastic License 2.0) — genuinely excellent, local-first, and free forever. Arize AX is the managed platform wrapped around it, and its self-serve pricing is the simplest in the market:

The problem is what sits between Pro and Enterprise. Arize publishes no overage rate above the 50k-span Pro cap — there is no self-serve path to 100k spans, let alone production agentic volume. We modeled three workloads for this guide (below): even the pilot produces 100k spans/month, and every single scenario falls off the self-serve shelf into a sales conversation. That is a legitimate enterprise motion, but it means the $50 Pro plan is a staging tier, not a production tier — budget owners cannot model their bill from public data.

Who should skip Arize AX (the SaaS): anyone whose span volume is not already known to sit under 50k/month, and anyone who needs a published unit rate for FinOps sign-off. Who should still take Arize seriously: self-hosters — Phoenix OSS is the strongest free tracing and evals stack in this comparison, and moving from Phoenix to AX Enterprise later is the intended upgrade path.

Datadog Agent Observability: buy it for the correlation, mind the ladder

Datadog's renamed product (the old LLM Observability URL 301s here) meters only LLM spans — one billable span per request to a model provider. Every tool call, retrieval hop, and embedding lookup traces for free. For agent workloads that is the narrowest — most favorable — billing unit in the market, and the docs say so plainly: "one agent workflow can produce several LLM spans."

The free allowance (40k LLM spans/month, footnoted on the pricing list) plus the narrow billing unit make Datadog's entry cost attractive — a pilot under 40k spans pays nothing, and our 60k-span pilot model lands at the $160 floor. The bill grows on two axes instead: span volume, and the retention ladder, which is a 2.14x multiplier from 15-day to 90-day retention. At our scale scenario, 30-day retention adds $2,385/month over the 15-day rate. The adjacent SKUs stack too: AI Guard (prompt filtering, billed per input token) and GPU Monitoring are separate products that the Agent Observability UI will happily cross-sell. One more wrinkle from the pricing list footnotes: experiment retention is 3 months with an annual or month-to-month commitment, otherwise 15 days — a detail that bites teams evaluating on-demand.

The genuine, durable advantage is correlation: the same request traced from service activity through the agent to infrastructure, in one platform. If your org already standardizes on Datadog, Agent Observability is the lowest-friction answer, and the pricing list is at least fully published.

Who should skip it: teams not already on Datadog — you would be adopting a second observability platform to get an LLM feature, and the per-span economics only make sense inside an existing commitment. Also teams needing long trace retention on a budget: the 90-day rung costs 2.14x the default, and there is no published path beyond 90 days at all.

What it costs at three scales (modeled, November 7, 2026 rates)

Modeled — not benchmarked. We assume an agent product where each user-facing run makes 3-8 LLM calls and 2-6 tool steps; LangSmith bills the run, Langfuse bills trace + observations (we exclude eval scores), Datadog bills LLM-kind spans only, and Arize counts all traced spans. Seats are the platform-team headcount (LangSmith) — every other vendor charges $0 per user. Every number below is computed from the verified rates above with exact decimal arithmetic.

Per agent run at production volume (scenario B), that is $0.00529 on LangSmith, $0.00076 on Langfuse Core, $0.00200 on Datadog at 15-day retention. Same traffic, same telemetry, 6.9x spread — and the ordering is stable across all three scenarios: Langfuse cheapest per unit of work, Datadog in the middle (its LLM-only meter narrows the gap), LangSmith the most expensive per run despite being the cheapest to start.

Two second-order effects the table hides:

• Retention multipliers are real money. Keeping scenario B traces 90 days on Datadog costs $1,960/month versus $1,000 at 15 days. Keeping every LangSmith trace at extended retention (180 days) turns $2,645 into $5,095. Langfuse Core gives 90 days to everyone and Pro gives 3 years — retention is where Langfuse's price list quietly wins.
• Seats compound on LangSmith. The Plus plan's 10k included traces are per organization, not per seat — five platform engineers on the production scenario pay $195/month in seats before a single trace bills.

Hidden fees and traps we verified

• LangSmith's retention auto-upgrades are on by default. New online evaluators and automation rules extend matched traces to 180-day extended retention — "retention extension is enabled by default for new online evaluators and automation rules" — at double the base trace price. Turning it off is a per-evaluator checkbox, not a workspace setting.
• LangSmith seats include invited users. "We consider the total number of users (including invited users) to determine the number of seats to bill" — stakeholder viewers cost $39/month each unless you police the invite list.
• Langfuse evals bill as units. Every score — including automated LLM-as-judge evals — is a billable tracing data point. A judge model run per trace adds roughly one unit per trace on top of the observations.
• Langfuse ingestion is rate-limited per plan (1,000 / 4,000 / 20,000 requests per minute on Hobby / Core / Pro). Batch your exports or your ingestion — not your budget — decides your plan tier.
• Datadog experiment retention collapses to 15 days without a commitment (3 months with annual or month-to-month) — teams evaluating on-demand lose their eval history mid-experiment.
• Arize publishes no overage rate. Above 50k spans/month there is no rack rate to model against — the number you need for budgeting does not exist publicly.
• Everyone's "OTel-compatible" claim is not a contract — yet. The GenAI conventions live in a repository with zero releases; your traces contain vendor-shaped attributes either way (next section).

The OpenTelemetry exit ramp — and why it is not clean yet

The standard anti-lock-in advice is "just use OpenTelemetry and switch backends at will." The theory is sound: OTLP out of your SDKs, a collector in the middle, vendors and self-hosted backends interchangeable behind it. The practice, verified today against the GenAI conventions repository: all GenAI semantic conventions moved out of the core spec in v1.42.0 (June 12, 2026) into a dedicated repository that — as of November 7, 2026 — has 525 stars and zero tagged releases. Everything in it is still marked development. Between v1.27 (August 2024) and v1.42 the attribute names churned through at least six breaking generations (token count renames, gen_ai.system becoming gen_ai.provider.name, agent-span splits), and each vendor's instrumentation currently emits a different generation of those names.

                      +----------------------+
   agent SDKs  --->   |  otel-collector      |     pin schema_url per pipeline;
   (langfuse /        |  (batch, tail-sample |     normalize gen_ai.* generations
    openllmetry /     |  before fan-out)     |     before storage;
    phoenix otel)     +----------+-----------+
                                |
              +-----------------+------------------+
              |                 |                  |
     +--------v------+  +-------v--------+  +-------v--------+
     | vendor A      |  | vendor B       |  | self-hosted    |
     | (evals, UI)   |  | (dashboards)   |  | Phoenix/Langfuse|
     +---------------+  +----------------+  +----------------+
              |                 |                  |
              +-------- raw OTLP archive (S3 / ClickHouse) ---+
                       your schema of record; vendors become
                       replaceable views, not the system of record

The workable pattern for platform teams: keep the collector as your border, dual-write to a vendor for its eval/UI features while archiving raw OTLP to storage you own, and pin the schema version per pipeline. The self-host backstops are real and current: Langfuse OSS (MIT, unlimited usage, SSO and RBAC included) via Docker Compose or Kubernetes, and Phoenix OSS v21.0.0 (Elastic 2.0) as a pip-installable local-first option. What self-hosting actually moves is the cost center — from a per-unit bill to ClickHouse, Postgres, and an on-call rotation — which is why Langfuse's paid self-hosted Enterprise tier bundles ClickHouse Cloud/BYOC/private deployments rather than pretending the database is free.

Who should pick what — the bottom line

• Already a Datadog shop: Agent Observability. The correlation with APM/infra/logs is worth more than any price delta, the LLM-only meter is the narrowest in the market, and $3.50/10k at 15 days is competitive. Budget the retention ladder before you commit, and remember experiment history needs a commitment to persist.
• Eval-heavy agent startup on LangChain frameworks: LangSmith, with eyes open. The eval/trace/feedback loop is the tightest in the market, but model the per-run math first — at 500k runs/month you are paying a $0.005 tax per run, and evaluator-driven retention upgrades can double it. Set the retention opt-out on day one.
• Volume on a budget, or compliance retention: Langfuse. Cheapest at every modeled scale, 90-day retention on the $29 tier, 3 years on Pro with SOC 2/ISO 27001/HIPAA, and the MIT self-host with SSO/RBAC is the credible exit ramp that keeps the cloud bill negotiable. Add an APM for the platform around your agents — Langfuse will not do that part.
• Self-host-first platform team: Phoenix OSS or Langfuse OSS behind one collector, dual-writing raw OTLP to storage you own. You accept the operational surface in exchange for a $0 license and zero unit anxiety.
• Pilot under 40k LLM spans/month: start on Datadog's free allowance or Langfuse Hobby (50k units), not on a paid tier. Upgrade decisions belong after you know your real span volume.

Who should skip this entire category

• Teams who cannot state their monthly run or span volume. Every vendor in this market bills on telemetry units. If you cannot estimate your monthly spans within 2x, you are not ready to buy — instrument with a free tier or the OTel SDK first, measure a week, then shop.
• Single-model wrappers behind an AI gateway. If your LLM surface is one API route proxied through a gateway, gateway-level metrics and structured logs (covered in our AI gateway architecture guide) already give you latency, spend, and error rates per provider. Per-trace tooling on top of that is a want, not a need.
• Teams that need >180-day SaaS trace retention for compliance. LangSmith's new 180-day SaaS cap and Datadog's 90-day ladder rung both fail that requirement outright; you want self-hosted OSS with your own retention policy, or a data-warehouse-based archive fed from the collector.
• Anyone buying "OTel compatibility" as the decision. It is table stakes and simultaneously unfinished — the GenAI conventions are unreleased and everyone emits different generations of the attributes. Buy the eval workflow and the unit economics; treat portability as something you build with the collector, not something you purchase.

Bottom line: in November 2026 this market prices the same telemetry at spreads of 6-7x depending on the billable unit, and the vendors are repricing fast enough that any 2025-vintage comparison is wrong. Langfuse's unit rates and included retention win on pure economics, Datadog wins if correlation with an existing Datadog estate is the real requirement, LangSmith wins on tracing depth per dollar of attention — and loses it on deep-agent volume — while Arize's managed tiers only work below 50k spans/month, above which the only honest answer is Phoenix OSS self-hosted or a sales call. Whichever you pick, put the OpenTelemetry collector in front of it and archive the raw spans: in a market where the schema and the price lists both move quarterly, the archive is the only lock-in insurance that has actually shipped.