Model Weights Don't Belong in Container Images: The GKE Storage Pattern for LLM Fleets

Every platform team running LLM inference on Google Kubernetes Engine rediscovers the same wall at roughly the same time: a 70B-class model is ~140 GiB of safetensors, a pod that has to drag those bytes over the network before its first token is a pod that burns A100 money doing nothing, and an autoscaler that scales that is a cost center with a YAML interface. The fix is never “more GPU.” The fix is a storage layer that puts weights where the pod can read them at line rate.

This guide compares the four storage patterns that actually ship on GKE in October 2026 — Cloud Storage FUSE (with the new auto-tuned Profiles), Hyperdisk ML, Rapid Cache (the service Google quietly renamed from Anywhere Cache), and native Kubernetes Image Volumes, which went stable in v1.36 — with real numbers from Google's own published benchmark, honest cost math priced at us-central1 rates, and a named production decision where the “obvious” fast option was the wrong one.

The problem, in one number

Google's accelerated-platforms pod-startup study (Llama-3.3-70B served by vLLM on an A2-highgpu-8g node, 8×A100) opens with the failure mode: with an untuned GCS FUSE mount, pod startup is dominated by serial reads through a userspace filesystem, and the tuned configuration — parallel downloads, indefinite file cache, RAM-disk cache medium, a prefetch sidecar — cut startup latency by more than 7×. The study's own framing is the part platform teams should internalize: slow startup “makes autoscaling infeasible and requires workloads to be configured to the peak, wasting expensive resources.”

Here is the floor math for any 140 GiB model, pure transfer time only — no per-file open overhead, no metadata round-trips, no CUDA graph build:

140 GiB over   400 MiB/s  =  358.4 s   (~6 min)
140 GiB over  1400 MiB/s  =  102.4 s   (~1.7 min)
140 GiB over  2400 MiB/s  =   59.7 s   (~1 min)

GPU idle cost per cold load (a2-highgpu-8g, 8xA100,
us-central1 on-demand $29.38708/hr, verified Oct 2026):

  at  1400 MiB/s:  102.4 s idle  =  $0.84 per load
  at   400 MiB/s:  358.4 s idle  =  $2.93 per load

A 10-node fleet cold-loading once a day:
  1400 MiB/s path:  ~$251/month of pure GPU idle time
    400 MiB/s path:  ~$878/month of pure GPU idle time

That $0.84-versus-$2.93-per-load gap is the entire business case for the rest of this guide. Everything below is a way to move your fleet from the bottom row to the top row without buying storage you don't need.

Pattern 1 — Cloud Storage FUSE with Profiles: the default that got good

The bucket-plus-FUSE pattern is what most teams land on first: weights live in a Cloud Storage bucket, and the managed GCS FUSE CSI driver mounts it into the pod. Until this year the knock on it was fair: FUSE is a userspace gateway, tuning meant hand-editing a dozen mountOptions, and a mistuned cache could OOM-kill your own inference pod.

Google's answer shipped April 9, 2026: GKE Cloud Storage FUSE Profiles — three dynamically managed StorageClasses (gcsfusecsi-serving, gcsfusecsi-training, gcsfusecsi-checkpointing) where the driver scans the bucket, inspects the node's RAM/Local-SSD/accelerator inventory, and picks cache sizes and backing medium itself. The headline number from the launch post: a Qwen3-235B-A22B workload (480 GB) on TPUs went from 39 hours to 14 minutes of model-loading time on the serving profile. The Profiles also integrate Rapid Cache automatically for the serving path. The catch is a version gate: Profiles require GKE 1.35.1-gke.1616000 or later with the CSI driver enabled.

Under the hood, the tuned configuration the pod-startup study documents is still worth knowing, because you may be on a version without Profiles (or need to deviate). Four knobs do almost all of the work:

volumeAttributes:
  bucketName: llm-weights-us-central1
  mountOptions: "metadata-cache:ttl-secs:-1,metadata-cache:stat-cache-max-size-mb:-1,metadata-cache:type-cache-max-size-mb:-1,metadata-cache:negative-ttl-secs:0,file-cache:max-size-mb:-1,file-cache:cache-file-for-range-read:true,file-cache:enable-parallel-downloads:true,implicit-dirs,only-dir:llama-3.3-70b/"
  skipCSIBucketAccessCheck: "true"

Read that as three decisions: metadata cached forever (ttl-secs:-1 and unbounded stat/type caches — safe only because model buckets are immutable-by-convention, a new version is a new prefix), file cache unbounded and parallel (max-size-mb:-1 with parallel downloads), and scope pinned to one model directory (only-dir) so the cache can't be flooded by bucket-listing noise. Two supporting moves matter as much: back the cache with RAM disk (emptyDir with medium: Memory on the three CSI cache volumes — GPU nodes have hundreds of GB of underused RAM) or Local SSD, and run a prefetch sidecar that dds every *safetensors file to /dev/null in parallel (the study uses xargs -P 15) while vLLM is still initializing, so the network fetch overlaps with the serving stack's own boot sequence.

Two operational notes from the same docs, both load-bearing in production: create the bucket in the workload's region with hierarchical namespace enabled (faster list/stat semantics), and don't cache on the boot disk — Google explicitly warns this leads to “reduced performance and unexpected terminations.”

Pattern 2 — Hyperdisk ML: provisioned throughput, shared read-only

Hyperdisk ML is Google's purpose-built answer for exactly this workload: a block volume you provision with a throughput number (400 MiB/s to 2 TiB/s, in MiB/s units) and attach to many instances read-only. Each MiB/s of provisioned throughput comes with 16 IOPS built in — you can't buy IOPS separately. The GKE how-to wires it via a StorageClass with provisioned-throughput-on-create: "2400 Mi" and a read_ahead_kb=4096 mount option, hydrated once from GCS (Google's example uses a c3-standard-44 instance to populate the disk at max throughput) and then mounted by every inference pod.

The limits that bite in real fleets, straight from the spec page:

• Zonal, permanently. A Hyperdisk ML volume lives in one zone. Your fleet is pinned to that zone unless you replicate — the GKE how-to's multi-zone variant uses enable-multi-zone-provisioning with per-zone allowedTopologies, which means one volume per zone and re-hydration per volume.
• Throughput must be provisioned for concurrency. If more than 20 instances attach, you must provision at least 20 MiB/s per instance — 500 readers means ≥10,000 MiB/s. Under-provision and you've built a queue.
• Attach-rate ceiling: up to 100 new attachments per 30-second interval, and total readers are capped by size (2,500 for ≤512 GiB volumes, down to 30 instances for ≥16 TiB).
• It bills while idle. Google's own pricing note: charges incur “even if the volume isn't attached to any instances or if the instance is suspended or stopped.”
• A date-stamped compatibility trap: Hyperdisk ML volumes created before February 4, 2026 cannot attach to 4th-generation machines (C4, G4, etc.) — you must snapshot and re-create. If you inherited a disk, check its creation date before a node-pool refresh.

Pattern 3 — Rapid Cache: the zonal SSD cache in front of your bucket

Rapid Cache (still sold under the Anywhere Cache name in most tooling — the docs page now leads with “Anywhere Cache is now renamed Rapid Cache”) is an SSD-backed zonal read cache in front of a Cloud Storage bucket: up to 2.5 TB/s of zonal throughput, autoscaling capacity and bandwidth, TTL between 24 hours and 7 days, with an optional ingest-on-write mode so new weights land in the cache the moment they're uploaded instead of on first read. It requires zero application changes — the pod still reads the bucket; the cache intercepts. For model loading, Google's pitch is precise: store once in a multi-region bucket, read anywhere in-region at cache speed, and skip the multi-region data-transfer fees on every subsequent read.

It pairs naturally with the FUSE pattern (the gcsfusecsi-serving profile automates the integration) and it's the cheapest way to de-risk the cold path on fleets that don't justify a provisioned disk. Two honest caveats: cache creation is an asynchronous long-running operation that can take up to 48 hours, and the TTL clock runs from last read — a model nobody loads for 7 days falls out of the cache and the next load pays full ingest again. Also know what it doesn't cache: object metadata always goes to the bucket.

Pattern 4 — Kubernetes Image Volumes: OCI artifacts as volumes, now stable

The fourth option doesn't come from Google at all. Image Volumes (KEP-4639) let a pod mount an OCI artifact directly — volumes[*].image with a reference and a pullPolicy — and the container runtime unpacks it to a read-only mount. The feature went stable and enabled-by-default in Kubernetes v1.36 (alpha in 1.31, beta 1.33/1.35), and the Kubernetes 1.31 announcement blog was explicit about the intended use: “mount large language model weights in a pod alongside a model-server… separate the model specifications/content from the executables that process them.”

apiVersion: v1
kind: Pod
metadata:
  name: image-volume
spec:
  containers:
  - name: inference-server
    image: ghcr.io/acme/vllm-server:1.2
    volumeMounts:
    - name: weights
      mountPath: /models
      readOnly: true
  volumes:
  - name: weights
    image:
      reference: registry.acme.dev/models/llama-3.3-70b:2026-10-01
      pullPolicy: IfNotPresent

The semantics you're signing up for, per the current docs: the volume resolves at pod startup; Always fails the pod on a pull error, Never requires the artifact to already be on the node, IfNotPresent re-resolves on pod recreation so new remote content becomes available then. A pull failure blocks containers from starting — the docs warn it “may add significant delay.” On GKE specifically, treat this as the registry-native path: it rides containerd's image store and pull path (containerd's Image Volume support has been actively hardened through 2026, with mount-manager and cleanup fixes backported to the release/2.2 line), so weights benefit from node-local image caching and registry mirroring exactly like a container image layer would. What it does not give you is Hyperdisk-class shared throughput — every node pulls its own copy unless you front the registry with a cache — and the read-only mount means no partial-file range tricks, which is why it slots in as the portable, on-prem-compatible option rather than the highest-performance one.

The cost math, honestly

us-central1 (Iowa) rates, verified October 13, 2026 against Google's published pricing pages. Scenario: one 140 GiB model, a 10-node fleet, one full model load per node per day. The disk sizes and load frequencies are assumptions you should re-run for your fleet; the unit prices are not.

The pricing-unit ambiguity, stated plainly: Google's disk pricing page lists “Hyperdisk ML provisioned throughput” at $0.000164384 / 1 hour with no per-MiB/s unit on the line item, while the monthly table lists $0.137 / 1 month for the same SKU family in some regions and $0.12 / 1 month in us-central1 — again without a MiB/s denominator. Compare that to Hyperdisk Balanced, where the page explicitly says throughput is billed “in MBps.” For Hyperdisk ML, the honest reading of the spec page's own example (500 instances requiring ≥10,000 MiB/s provisioned) suggests the charge scales with provisioned throughput, and the flat $0.12/mo line cannot possibly cover a 10,000 MiB/s volume. Treat $0.12/mo as per-MiB/s until your own bill proves otherwise: budget ~$168/mo for 1400 MiB/s, and be pleasantly surprised if the invoice says $0.12. This is exactly the kind of unit ambiguity you resolve with a one-day test volume, not with a blog post — including this one.

The comparison that matters isn't storage cost at all. A 10-node fleet cold-loading daily at 400 MiB/s burns ~$878/month in idle A100 time; at 1400 MiB/s it burns ~$251/month. The delta (~$627/month) is 3× the worst-case Hyperdisk ML reading, ~16× the flat reading, and ~20× the all-in bucket-plus-cache path. Storage choice is GPU economics wearing a storage hat.

Architecture: where each piece sits

flowchart LR
    subgraph CI["Model release (CI)"]
        U["Upload weights version"] --> B
    end
    subgraph GCS["Cloud Storage"]
        B["Bucket (multi-region, HNS)
immutable per-version prefixes"]
        RC["Rapid Cache (zonal SSD)
TTL 24h-7d, ingest-on-write"]
        B --> RC
    end
    subgraph GKE["GKE cluster (zonal node pool)"]
        subgraph NODE["a2-highgpu-8g node"]
            P["vLLM pod"]
            FUSE["GCS FUSE CSI sidecar
file cache: RAM disk / Local SSD"]
            HD["Hyperdisk ML RO mount
1400 MiB/s provisioned"]
            P --> FUSE
            P -.-> HD
            IV["Image Volume
OCI artifact, containerd store"]
            P -.-> IV
        end
    end
    RC --> FUSE --> P
    REG["OCI registry
models as artifacts"] -.-> IV

Three data paths, three ownership stories: the FUSE+cache path keeps the bucket as the single source of truth and treats everything between bucket and GPU as disposable cache (safe with immutable per-version prefixes); the Hyperdisk path promotes one hydration into a provisioned, billed artifact (fastest per-pod floor, bills while idle, zonal); the Image Volume path makes the registry the source of truth and inherits containerd's pull machinery (portable across clouds and on-prem, one full copy per node).

The real decision record: when the fastest option lost

The most useful production artifact I found for this pattern is a decision record, not a benchmark. The Cloud Native Reference project (oGenki) published ADR-0021, “Cloud Storage FUSE for LLM model weights on GCP” (accepted August 26, 2026) — a named platform team choosing between exactly the options above for their gcp-0 reference platform, with the reasoning written down.

Their context: the platform is built, verified, and destroyed in the same session — every workstream ends with terraform destroy and a check that nothing bills afterward. Their AWS cluster (ADR-0004) had chosen S3 Files for weights; GCP has no single equivalent, so the decision was real. They weighed three options: (1) bucket + GCS FUSE only, (2) bucket + a Hyperdisk ML volume hydrated per deploy and destroyed at teardown, (3) Hyperdisk ML persisted between tests.

They chose option 1 — the slowest one. The reasoning is a masterclass in constraint-first architecture: “it is the only option whose resting state costs nothing but storage, which is the constraint that governs gcp-0. Options 2 and 3 buy cold-start speed, and cold-start speed is worth less here… Option 3 additionally bills while idle, which is disqualifying rather than merely expensive.” On Hyperdisk ML they also flag what the spec page confirms: it's zonal, so it “pins the platform to one zone in a way the bucket does not.” And the honesty clause every ADR should have: “Cold start is slower than aws-0's, by an amount nobody has measured. No claim is made about whether it meets the 90-second budget — asserting that without measuring is precisely the habit this programme has been correcting.”

That's the pattern's real shape: the same technology that wins for a persistent serving fleet loses for an ephemeral one. Hyperdisk ML's billing-while-idle and zonal pin are only problems if your platform survives longer than a model load; for a 24/7 inference fleet they're the price of a 102-second cold-load floor. For anything with a “nothing bills between tests” constraint, they're disqualifying.

Decision matrix

When NOT to use each pattern

• Skip Hyperdisk ML if your platform has any “nothing bills between runs” constraint — it charges while detached, stopped, or suspended (Google's own words), it's zonal, and its minimum useful size (the study's example needs a hydration instance) makes it a commitment device. Also skip it if readers attach and detach constantly: the 100-attachments-per-30s limit plus the 20 MiB/s-per-reader requirement make churny fleets pay for concurrency they don't use.
• Skip Rapid Cache for rarely-loaded models: the TTL (max 7 days, from last read) evicts cold entries, so you pay ingest ($0.0032/GiB) again and again for sporadic loads — a bucket alone is cheaper below a load frequency you should measure, not guess. Also skip it if you can't tolerate the async-create latency (up to 48 hours) in your provisioning path.
• Skip GCS FUSE for checkpoint-heavy training writes or when metadata churn is high — FUSE metadata ops still round-trip the bucket, and Google's own tuning docs assume immutable, read-heavy, per-version prefixes. A mistuned unbounded cache on a small node OOMs the pod; the Profiles feature exists precisely because hand-tuning this failed often enough to productize.
• Skip Image Volumes if you need concurrent-cold-start throughput (every node pulls independently — front the registry with a cache or accept N×140 GiB pulls), if your runtime predates support, or if you need range-reads of safetensors without materializing layers. And remember pullPolicy: Never fails pods whose layers aren't pre-pulled — it's an operations contract, not a cache hint.

The verdict

For a persistent LLM serving fleet on GKE in late 2026, the pattern that survived contact with production is layered, not exclusive: an immutable per-version bucket as the source of truth, the gcsfusecsi-serving Profile for the mount, Rapid Cache in the serving zones, and Hyperdisk ML only for the model whose cold-start frequency actually pays for provisioned throughput. Keep Image Volumes in your back pocket for portability and for platforms where the registry is already the artifact system of record. And whichever you pick, instrument pod startup time (scheduled→ready) as a first-class SLO — the 7× study gap between default and tuned is a config difference, not a hardware one, which means your first fleet is probably running at the bottom of that range right now.

One measurement discipline note, because this guide quotes prices: every rate here was verified against Google's live pricing pages on October 13, 2026 (disk pricing, storage pricing, GPU pricing), and Google re-prices constantly. The transfer-floor arithmetic is timeless; the dollar figures are not — re-run them before you commit budget. If your storage choice is being made by someone who has never divided model size by provisioned throughput, make them do that division first. Everything else in this guide is detail.