Verdict up front: Magnitude is the most architecturally interesting local inference engine to launch in 2026 — and the only one whose install refuses to run unless a root-owned package manager put it there. The Magnitude desktop app from magnitudedev/magnitude (YC S25, Apache-2.0, ~5.7k stars at launch) compiles and tunes its GPU kernels on your device instead of shipping precompiled ones, which is a genuinely different bet from llama.cpp or MLX. But we unpacked the actual Linux package, ran the real CLI, strace'd the parts that refused to start, and read the launch thread's benchmark dissent — and the "up to 2x faster than llama.cpp" headline inverts on Apple's newest silicon while the headless-server path is still gated behind behavior that will surprise every platform engineer who treats a .deb as an artifact you can inspect and run.

Executive Scorecard

DimensionScoreEvidence
Reliability6/1015 releases in 14 days (0.0.15 → 0.2.3); engine v3 ships while a v4 rewrite is in flight; an M5 Max user measured llama.cpp at roughly 2x Magnitude's speed on current builds, which the founder acknowledged as a Metal 4 matmul gap
DX7/10One-click connections for Pi, OpenCode, Hermes, Codex, Claude Code, Cline and more; a curated 15-model catalog; a coherent CLI — but the fit-assessment under-recommends models on hybrid setups, per user reports in the launch thread
Cost9/10Apache-2.0, no token costs, nothing leaves the machine; the only bill is RAM and GPU. No managed tier exists yet, so there is nothing to upsell you
Security7/10Loopback-only by default, Ed25519-signed updates, root-owned install admission, Host-header allowlist on network mode — but the Debian package hard-depends on pkexec, and the update installer shells into sudo

What Magnitude Actually Is (Including the Parts the Landing Page Doesn't Mention)

The product you install is three binaries plus an Electron shell: the magnitude CLI (an 85 MB Bun-compiled TypeScript binary), the magnitude-service "Agent Control Node" (93 MB), and a separately-downloaded Rust inference engine the docs call the ICN. The GitHub repo reports Rust as its primary language for a reason — the engine is where the interesting code lives.

The repo itself is an archaeology dig of engine generations, which tells you the real story of the pace here:

DirectoryGenerationStatus
old-inference/llama.cpp embedded through a pinned llama-cpp-rs bindings fork ("Inference Control Node")Frozen — the current engine's README calls it "the frozen previous implementation"
inference-v2/Python + MLX engineSelf-described as "an Apple Silicon inference engine" — Metal only
inference-v3/Rust + TileLang ("TileLang-native inference through Ops")Dense and routed Qwen 3.5 only — an earlier Rust detour
inference/v3: Rust + Seismic, CPU/Metal/CUDA/VulkanWhat ships today
inference-v4/Rust + Seismic rewrite"Under implementation; not yet a replacement for V3" — its README says so explicitly

Five engine generations in one repository, a public alpha series dating to June 2026 with the first stable release (0.1.0) on 2026-09-17, and launch on 2026-09-30. Whatever else is true, this is a codebase in motion — and note the irony the table hides: the frozen predecessor embedded llama.cpp through a bindings fork, and the product's whole pitch is replacing that with self-authored kernels.

The novel part is Seismic, their kernel authoring pipeline: numerical programs are authored once in a checked DSL, lowered to a structured IR, and then jointly selected, instantiated, and compiled to the backend — Metal as the reference backend, CUDA via on-device NVRTC compilation, Vulkan 1.3 compute shaders, and Cranelift CPU code. Per the founder, they "write the efficient high level kernel structure with tunable parameters, then it fits to whatever hardware it's actually running on" — structure is authored, parameters are fitted per device. On NVIDIA, model kernels are CUDA C++ compiled on your machine into CUBIN for your exact architecture via a bundled NVRTC 12.9 — no CUDA toolkit install required. That is the technical core of the "tuned on your device" claim, and it is real engineering, not marketing veneer.

Architecture Mechanics

  ┌────────────────────────── your machine ──────────────────────────┐
  │                                                                  │
  │  Agent (Pi / OpenCode / Hermes / Codex / Cline / anything OpenAI) │
  │        │  localhost:10100   OpenAI- or Anthropic-compatible      │
  │        ▼                                                         │
  │  ┌──────────────────────┐    owns    ┌──────────────────────────┐ │
  │  │  magnitude CLI (Bun) │───────────▶│ ACN: magnitude-service  │ │
  │  │  + Electron desktop │  parent-    │ "Agent Control Node"     │ │
  │  └──────────────────────┘  loss       └───────────┬──────────────┘ │
  │                                          spawns │ supervised      │
  │                                                 ▼                 │
  │  ~/.magnitude/config.json      ┌──────────────────────────────┐   │
  │  ~/.magnitude/models (GGUF/    │ ICN: magnitude-inference     │   │
  │  MLX weights, 15-model         │ Rust engine + Seismic        │   │
  │  curated catalog)              │  check → IR → selection →    │   │
  │                                │  Metal | CUDA(NVRTC) |       │   │
  │  /var/lib/magnitude-desktop/   │  Vulkan 1.3 | CPU(Cranelift) │   │
  │  installation.lock (root:root  └──────────────────────────────┘   │
  │  444 — the admission gate)                                       │
  └──────────────────────────────────────────────────────────────────┘