LLM Routers and Gateways After Jev: LiteLLM, OpenRouter, and the Decision-Only Workload Split

The second-most-upvoted infrastructure story on Hacker News this week (544 points as of September 19) was not a Kubernetes release or a cloud outage. It was a browser experiment that proves an uncomfortable point for anyone operating an LLM gateway: a large fraction of the traffic you route to chat-completion models never needed text generation at all.

The story has three legs, and all three matter to platform teams:

• TypeSafe AI shipped Jev (September 15) — the first public “System One Model.” It ingests unstructured state and emits typed, calibrated probabilistic decisions with a parallel sampler instead of autoregressive decoding. Input is priced at $0.042/MTok; output tokens are free because there are none in the text-generation sense. TypeSafe’s own workflow evals claim 193.6x faster and 444.6x cheaper against frontier LLM baselines — and, refreshingly, the blog post annotates every claim with a “Nuance” block admitting where the numbers are flattered.
• SemIf (formerly OpenJev) took Jev’s core claim — that a language model can answer a constrained choice by reading option logits instead of writing JSON — and rebuilt it as a local, in-browser experiment you can run on your own GPU with open checkpoints (Qwen3 0.6B, MiniCPM5 2B, Qwen3.5 4B). No waitlist, no backend, WebGPU only.
• Eider, a Rust inference server by Robert Daum (rdaum), is already serving a /v1/decisions endpoint from Gemma 4 on an NVIDIA DGX Spark: shared state evaluated once, one KV-cache fork per question, and a 64-row slice of the LM head that scores answer labels without ever materializing the full vocabulary. It is open source today.

Put the three together and a trend line emerges: decision-only inference is detaching from text generation as a separately priced, separately routed, separately served workload. Your LLM router — the thing you bought to normalize OpenAI/Anthropic/DeepSeek APIs, retry failures, and cap spend — was designed for a world with exactly one kind of call. This guide is about what changes when there are two.

The Cost Anatomy of a “Simple” Decision Call

Here is the standard way platforms make a model pick between three options today. You send a chat-completion request asking for JSON:

{
  "model": "claude-fable-5.1",
  "messages": [
    {"role": "system", "content": "Classify the ticket. Answer with JSON only."},
    {"role": "user", "content": "Ticket: payouts failing for three days..."}
  ],
  "response_format": {"type": "json_object"}
}

What you are actually paying for, mechanically: the model computes a next-token distribution over its entire vocabulary (~250K rows in modern tokenizers), samples one token, feeds it back, and repeats — serially — until it has written {"department": "billing", "confidence": 0.8}. Every step of that loop is a full forward pass through the network. You bought a 250,000-way choice machine to express a 3-way choice, then paid to serialize it through a parser that exists because JSON-in-prose is unreliable.

Now the same decision as a logit readout — what Jev, SemIf, and Eider all do in different ways: run the prompt through the model once, gather only the vocabulary rows corresponding to your allowed labels (“billing”, “technical”, “sales”), softmax over those few rows, and return the distribution. One forward pass. No decode loop. No parser. No schema violations — an output outside the label set is mathematically impossible, which is what TypeSafe means by “can’t hallucinate”: schema matching is guaranteed by construction, not by prompt discipline.

The list prices make the split concrete. All generation prices below are from the OpenRouter models API on September 19, 2026; the Jev input price is from TypeSafe’s announcement. The workload is the platonic platform decision call: 100 input tokens, 10 output tokens, one million calls a month.

Two honest caveats before the graph-axes crowd arrives. First, the Jev column assumes your entire decision workload is portable to one early-access vendor whose pricing TypeSafe itself flags as possibly subsidized (“we can’t prove it isn’t; we’ll need the long term”). Second, the comparison models here were not prompted to be efficient — but that is precisely the point: nobody’s production stack prompts for efficiency either. The 357x is the cost of using a generation API as a decision API, at list price, with the workloads platform teams actually run: guardrail checks, ticket triage, intent routing, canary verdicts, data-classification gates.

There is also a latency side that list prices hide. TypeSafe publishes 70–500ms end-to-end for Jev versus 3–329 seconds for frontier LLMs on decision-shaped queries — a 40x–200x gap on their evals. A 300ms ceiling is the difference between “AI feature” and “inline code path”. Every platform engineer who has tried to put an LLM check inside a request handler knows which side of that line matters.

Where the Gateway Sits — and What It Doesn’t Know

The typical self-hosted stack in 2026 looks like this:

                    ┌──────────────────────────────────────────────┐
                    │                LLM GATEWAY (LiteLLM)          │
  app services ────▶│  auth (virtual keys)  budgets  retries        │
  agents ──────────▶│  model routing       cooldown  spend tracking  │
  CI pipelines ────▶│  fallbacks           rpm/tpm   provider swap  │
                    └───────┬───────────────────┬──────────────────┘
                            │                   │
              ┌─────────────▼─────┐   ┌─────────▼───────────┐
              │  provider APIs    │   │  self-hosted vLLM   │
              │  (OpenRouter,     │   │  (OpenAI-compatible │
              │   Anthropic,     │   │   server, GPUs,     │
              │   OpenAI, ...)   │   │   structured output)│
              └───────────────────┘   └─────────────────────┘
                            │                   │
                            └───── ONE CALL TYPE: generation ─────┘

Every request — whether it needs 2,000 output tokens of reasoning or a single bit of “is this PII: yes/no” — enters the same queue, costs the same per-token meter, fails over to the same backends, and burns the same decode loop upstream. The gateway has model as its only steering primitive.

The decision-model shift adds a second, structurally different call type:

  app services ────┐
  agents ─────────┤   ┌─────────────────────────────────────────┐
  guardrails ─────┼──▶│  LLM GATEWAY                           │
  pipelines ──────┤   │  /v1/chat/completions  → generation pool│
                  │   │  /v1/decisions        → decision pool  │
                  │   │      (typed questions, probabilities,   │
                  │   │       no decode, sub-second budgets)   │
                  │   └───────────────┬─────────────────────────┘
                  │                   │
                  │        ┌──────────▼───────────┐
                  │        │ decision-native tier: │
                  │        │ Jev API, Eider-class  │
                  │        │ /v1/decisions, or     │
                  │        │ vLLM choice-constrained│
                  │        │ fallback for now      │
                  │        └──────────────────────┘

None of today’s popular gateways ship that second route as a first-class concept. But they give you every primitive you need to build it now — model groups, fallback chains, per-key budgets — and understanding those primitives is what the rest of this guide does, hands-on.

LiteLLM: Load Balancing and Failover, Tested on This Machine

LiteLLM (current stable: v1.101.0, released September 15, 2026) is the default answer for teams that want provider independence without shipping their own retry logic. Its router docs recommend simple-shuffle as the production default, with usage-based-routing-v2 (async) and latency-based-routing as opt-ins; in multi-instance deployments it tracks cooldowns and TPM/RPM in Redis, and enable_pre_call_checks makes it reject requests that would blow a deployment’s rate limits before the call leaves the process.

Docs are docs. We ran the thing. Setup: LiteLLM 1.101.0 from PyPI as a proxy, two mock OpenAI-compatible upstreams on localhost (each response body names the port that served it, so distribution is observable), one model group decision-mock with two deployments, simple-shuffle, num_retries: 2.

Phase 1 — both deployments healthy, 50 concurrent calls. The shuffle is close to even, no surprises:

$ python router_test.py
{"phase": "both-healthy", "distribution": {"served-by-8101": 27, "served-by-8102": 23}, "errors": 0}

Phase 2 — kill deployment B entirely (it returns HTTP 500 on every request). Same config, 30 sequential calls through the proxy:

28 served-by-8101
 2 litellm.InternalServerError: InternalServerError: OpenAIException -
   mock upstream failure. Received Model Group=decision-mock
 2 Available Model Group Fallbacks=None

That error string is the lesson of the whole experiment, so read it twice. Retries inside a model group are not cross-deployment failover. When the shuffle lands a request on the dead deployment and its two retries also land there (or on a still-cooling-down peer), LiteLLM gives up and surfaces the error — the router had no second model group to fall to, and it says so: Available Model Group Fallbacks=None. Two of thirty requests failed while a perfectly healthy deployment sat idle one line away in the same config file. The mock’s access log counted 32 failed attempts — the retries did fire — but retrying the same broken backend is availability theater.

Phase 3 — same setup plus an explicit cross-group fallback, which is the documented production pattern:

model_list:
  - model_name: decision-mock            # primary group: A + B
    litellm_params:
      model: openai/mock-a
      api_base: http://127.0.0.1:8101/v1
      api_key: sk-local-mock
      rpm: 1000
      tpm: 100000
  - model_name: decision-mock
    litellm_params:
      model: openai/mock-b
      api_base: http://127.0.0.1:8102/v1
      api_key: sk-local-mock
  - model_name: decision-mock-backup     # fallback group: healthy only
    litellm_params:
      model: openai/mock-a2
      api_base: http://127.0.0.1:8101/v1
      api_key: sk-local-mock

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  allowed_fails: 1        # a deployment goes to cooldown after 1 failure
  cooldown_time: 10

fallbacks:
  - decision-mock: ["decision-mock-backup"]

Result with deployment B still hard-down: 30 of 30 requests succeeded, all served by A. With allowed_fails: 1 the dead deployment entered cooldown almost immediately and the shuffle stopped selecting it; the explicit fallbacks entry closed the residual window where retries exhaust inside the group.

One more proxy behavior worth knowing before you rely on it: spend tracking. The /global/spend endpoints return {"error": "No db connected"} unless the proxy runs with a DATABASE_URL (Postgres) configured — spend, per-key budgets, and the admin UI all assume that database exists. In our throwaway test we deliberately ran DB-less, and the proxy told us plainly at the first spend query. Budget enforcement is not a config-file feature; it is a Postgres feature.

Virtual Keys: The Budget Control That Actually Works

The genuinely load-bearing LiteLLM feature for multi-tenant platforms is virtual keys: per-team, per-service, per-agent credentials minted at runtime against the proxy’s Postgres, each carrying its own model allowlist, spend cap, and rate limits. Generate one scoped to a single model group with a monthly budget:

curl http://llm-gateway.internal:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["decision-mock"],
    "max_budget": 50,
    "budget_duration": "30d",
    "rpm_limit": 600,
    "tpm_limit": 200000,
    "key_alias": "payments-triage-agent"
  }'

The returned key can call only decision-mock, dies at $50 of spend or its rate limits, and re-arms on its budget_duration window. The docs are explicit about the sharp edge: a key’s owner is whoever its user_id points at — the caller’s when minted by a non-admin, but proxy admins must pass user_id explicitly — and budget limits cascade from owner to key. Mis-attribute a service key and you have just given some human’s team budget to a cron job. Treat key minting as an audited, code-reviewed path, not something an on-call does from a laptop.

OpenRouter: When You Want the Routing Layer To Not Be Your Problem

OpenRouter is the hosted version of the same idea — one OpenAI-compatible endpoint, N upstream providers — plus a routing surface that is genuinely more expressive than most self-hosted setups. The provider object in the request body is the control plane:

{
  "model": "deepseek/deepseek-v4.1-flash",
  "provider": {
    "sort": "throughput",
    "allow_fallbacks": true,
    "require_parameters": true,
    "data_collection": "deny",
    "quantizations": ["fp8"],
    "preferred_max_latency": {"p99": 0.8},
    "max_price": {"prompt": 0.000002, "completion": 0.000008}
  },
  "messages": [{"role": "user", "content": "..."}]
}

The knobs worth internalizing, from their routing docs:

• Default strategy is price-based load balancing across the providers of a given model, weighted toward uptime. You do nothing, you get price-balanced multi-provider redundancy.
• sort accepts price, throughput, or latency, optionally with a partition — and the percentile preferences (preferred_min_throughput / preferred_max_latency) take objects like {"p50": 100, "p99": 2500} so you can demand good tails, not good averages. Tail latency is the correct metric for a decision tier sitting in a request path.
• :nitro appended to any model slug is the throughput-sort shortcut that also makes priority-tier endpoints eligible — a superset of sort: "throughput".
• require_parameters: true restricts to providers that support every parameter in your request — this is how you avoid the classic “my structured-output constraint silently vanished on failover” failure mode.
• data_collection: "deny" and zdr hard-exclude providers that train on or retain request data. For a decision tier ingesting customer state, this is not optional.
• BYOK — bring your own provider keys, encrypted at rest, with per-credential scoping and rotation — for the hybrid pattern: your negotiated provider rates for bulk, OpenRouter’s mesh for burst and failover.

The trade-off is the mirror of LiteLLM’s: you have outsourced the control plane, so the gateway itself is a dependency with its own incidents, and every request transits a third party. Platforms with data-residency obligations (OpenRouter does offer in-region routing on enterprise tiers, and BYOK keeps the direct provider relationship) should run the decision tier — which touches real user state — self-hosted, and treat hosted routing as the generation-tier convenience. That division is itself a preview of the two-tier split this article is about.

The Poor Man’s Decision Endpoint, Available Today in vLLM

You do not need Jev access or a DGX Spark to start splitting decision traffic. vLLM (current stable v0.29.0, September 9) has had constrained decoding for years, and the API got a long-overdue rename in v0.12.0: the old guided_* parameters are gone, replaced by a structured structured_outputs object. For a choice among fixed options:

from openai import OpenAI

client = OpenAI(base_url="http://vllm.internal:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="Qwen3.5-4B",
    messages=[
        {"role": "system", "content": "Pick exactly one label."},
        {"role": "user", "content": ticket_state},
    ],
    extra_body={"structured_outputs": {"choice": ["billing", "technical", "sales"]}},
    max_tokens=8,
)
print(resp.choices[0].message.content)   # one of the three labels, guaranteed

Behind the scenes vLLM masks the sampling distribution to your choice set and decodes a single token — the output is one of your labels by construction, no JSON parsing, no retry-on-malformed-JSON loop. Add the logprobs parameter to the same call and you also get the normalized probabilities over the allowed labels — which is a large fraction of what Jev sells, at the cost of one decode step rather than zero.

What this does not give you relative to the decision-native tier:

That middle column deserves emphasis because it is the sleeper story of the week. Eider’s Gemma 4 path implements the exact architecture the HN thread was hand-waving about: evaluate the shared state once, fork a KV-only child sequence per question, batch the question suffixes, and score answers with a compact 64-row slice of the tied embedding — the README is explicit that it “does not allocate or evaluate the full vocabulary head for each branch.” Answer types are noul (a probability), choice (an option plus all option probabilities), and score (an expected ordinal over a legend). Usage accounting is honest too: input usage includes the state and every question suffix; output usage is one answer token per question. And the author’s HN confession is the most platform-relevant sentence in the whole affair: “I don’t have the chutzpah to go creating PRs for vLLM to do the same.” The technique is sitting in a 24-star repo while every major inference server ignores it.

What SemIf Actually Proved (and What It Didn’t)

SemIf’s value is that it independently operationalizes the claim with open weights, in a browser, with no waitlist — and then publishes its own accuracy table next to TypeSafe’s. The page reports balanced accuracy on its authored benchmark, a “perturbed” variant, and equal-case agreement with TypeSafe’s public 102-row subset:

Read that honestly. A 3 GB open checkpoint reading logits lands within ~4 points of the hosted, RLCD-trained Jev on the agreement subset — close enough that the engineering question stops being “is decision-quality reachable” (it is) and becomes “what do calibration, consistency, and tail behavior look like under your real distribution.” That is a question your eval harness answers, not a vendor’s. The site is also admirably explicit about what its numbers do not mean: direct scores are a softmax over only the displayed option tokens, they “are not calibrated confidence”, and browser quantization may change both quality and speed. We could not run the timed demo ourselves — our headless browser has WebGPU APIs but no GPU adapter, and the site refused gracefully — so we report their methodology, not their timings.

The platform takeaway is not “run MiniCPM in Chrome.” It is that the expensive property — decision quality without decoding — is not exclusive to a frontier lab. It is an inference-server feature away from commodity hardware, and Eider has already written that server.

The Monday-Morning Configuration

Assume the two-tier split starting now, using primitives that exist today. Gateway config with a decision tier distinct from the generation tier:

model_list:
  # --- decision tier: small, cheap, latency-capped ---
  - model_name: decision
    litellm_params:
      model: openai/Qwen3.5-4B
      api_base: http://vllm-decision.internal:8000/v1
      api_key: os.environ/LOCAL_LLM_KEY
      rpm: 6000
      tpm: 600000
  - model_name: decision-fallback        # cross-group fallback, NOT just retries
    litellm_params:
      model: openai/Qwen3-0.6B
      api_base: http://vllm-decision-b.internal:8000/v1
      api_key: os.environ/LOCAL_LLM_KEY

  # --- generation tier: frontier, expensive, prompt-heavy ---
  - model_name: generation
    litellm_params:
      model: openrouter/anthropic/claude-fable-5.1
      api_key: os.environ/OPENROUTER_API_KEY

router_settings:
  routing_strategy: simple-shuffle       # docs-recommended production default
  num_retries: 2
  allowed_fails: 1
  cooldown_time: 10
  enable_pre_call_checks: true           # reject before violating rpm/tpm
  timeout: 8

fallbacks:
  - decision: ["decision-fallback"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL  # required for budgets/spend tracking

Client code that splits the call types explicitly — the five-line change that captures most of the win:

def classify_ticket(state: str) -> dict:
    """Decision call: one label set, logit-constrained, no JSON parsing."""
    r = client.chat.completions.create(
        model="decision",
        messages=[{"role": "user", "content": state}],
        extra_body={"structured_outputs": {"choice": LABELS}},
        logprobs=True,
        max_tokens=8,
    )
    label = r.choices[0].message.content
    probs = r.choices[0].logprobs.content[0].top_logprobs
    return {"label": label, "probs": {p.token: p.logprob for p in probs}}

And the guardrails that keep the tier honest:

• Mint a virtual key per caller with its own budget (max_budget + budget_duration) so a runaway retry loop in one service drains that service’s allocation, not the platform’s.
• Cap max_tokens on decision calls to single digits — a decision that starts generating prose is a misconfiguration, and the cap turns it into a loud error instead of a slow bleed.
• Alert on output tokens per decision call. The metric should sit at ~1. When it drifts, someone has pointed a generation workload at the decision tier.
• Keep the decision tier off any provider that trains on inputs — it sees user state by definition. Self-hosted, or data_collection: "deny"/zdr when hosted.
• Eval per tier. The decision tier needs its own golden set of state→label cases with measured calibration, not vibes. If a 4B model matches your labels 95% as well as a frontier model at 1/350th the cost, that is a routing decision, and you should make it with numbers you own.

Hidden Costs and Honest Trade-offs

Two tiers means two failure domains. You are now operating a GPU pool with a latency budget measured in tens of milliseconds. vLLM’s constrained decoding path, queueing under burst, and prefix-cache behavior on short prompts are all your problem now. The decision tier’s SLO must be engineered, not assumed — measure p99, not the demo.

Calibration is the real moat, and logit readout alone does not give it to you. Jev’s training objective (RLCD) optimizes for answers whose stated probabilities match observed frequencies. A raw softmax over label rows — SemIf’s approach, and vLLM’s via logprobs — is a score, not a calibrated probability. Eider ships a --decision-calibration FILE flag for exactly this reason and notes that confidence is “probability concentration, not answer correctness.” If your platform gates user-facing actions on these numbers (fraud, access, prioritization), uncalibrated scores are a quiet correctness bug at scale.

Vendor pricing for decision inference is unproven at durability. TypeSafe says the quiet part loudly: they cannot prove their price isn’t subsidized. Early access, single-digit public eval sets, and a founder-authored blog are not a supply contract. The architecture should assume today’s decision-tier prices are volatile in both directions.

Cardinality has a ceiling. Jev documents a 255-option limit, with a two-stage score-then-choose for higher-cardinality cases at some latency cost. Eider’s label head is 64 rows. Constrained decoding in vLLM handles large choice sets, but at some point you are describing a retrieval problem, not a decision problem — don’t route a 10,000-way SKU match through any of this.

Lock-in is asymmetric. LiteLLM’s whole value is normalizing providers away; a decision-native tier partially un-does that. Mitigate it the boring way: keep the request/response shape of your decision tier behind your own service interface (the classify_ticket function above, not a raw vendor client), so the tier below it — vLLM choice today, Eider-class server or a hosted Jev tomorrow — is a config change.

Who Should Skip This

• Teams under ~10M decision-ish calls a month. Your entire classification spend on DeepSeek Flash-tier models is tens of dollars. The engineering cost of a second tier exceeds years of savings. Use structured_outputs.choice on your existing backend and revisit when volume or latency requirements actually hurt.
• Teams whose LLM usage is genuinely generation-shaped — copilots, summarization, code review — with no request-path calls. There is nothing to split; a plain gateway remains correct.
• Teams without an eval harness. Routing decisions between a frontier model and a 4B label-reader without measured accuracy and calibration on your distribution is how you ship a regression with a beautiful cost dashboard. Build the eval first; this guide will still be true next quarter.
• Regulated shops that cannot ship user state to any external inference endpoint — the hosted versions of this pattern are off the table until your legal team finishes reading, and the self-hosted versions need GPU capacity you may not have. The vLLM choice-constrained pattern on your existing fleet is the only leg available to you today, and it is a good one.

Verdict

The gateways won. LiteLLM and OpenRouter are load-bearing infrastructure with real, tested behavior — but they were built to arbitrage generation, and the workload mix flowing through them is quietly bifurcating. This week moved the decision tier from vendor promise to reproducible fact three separate ways: a hosted model with published prices and honest nuance blocks, a browser lab proving open checkpoints reach within a few points of it, and an open-source server already implementing the logit-readout mechanics on real hardware.

The right platform response is not to bet on any of the three. It is to make the decision call a first-class route in your gateway — its own model group, its own fallback, its own budget, its own evals — and implement it today with constrained choice decoding on hardware you already run. Then, when the decision-native backends graduate from early access (or the vLLM PR finally gets written), the switch is a deployment detail behind an interface you own. The teams that learn the split now will be the ones whose bill graphs don’t care which way the pricing war breaks.