Broken access control has sat at the top of the OWASP Top 10 since 2021, and it stays there for a structural reason: every product team reimplements authorization as bespoke if role == "admin" logic, scatters it across services, and loses track of who can touch what long before the first audit. Google solved this class of problem in 2019 by centralizing it: Zanzibar, a globally replicated store of relationship tuples that answered permission questions for Calendar, Drive, Maps, and YouTube at 200 million lookups per second at peak, over two trillion ACLs, with a 95th-percentile latency under 10 milliseconds and 99.999% availability over three years of production use.
OpenFGA is the open-source descendant of that design: a CNCF incubating project (TOC vote, November 11, 2025) created by the Auth0/Okta FGA team, running in production at Auth0 since December 2021 and now used publicly by 37+ organizations including Grafana Labs, Docker, Canonical, Read AI, Wolt, and Headspace. It stores relationship tuples in PostgreSQL, MySQL, or SQLite and answers the two Zanzibar questions over an HTTP and gRPC API: is user U related to object O via relation R? and which objects of type T is user U related to via R?
This guide is about the hard part: not what OpenFGA is, but how you get there. Grafana Labs walked this exact road — from a 14-year-old access-control stack that had already been through four generations (Org roles in Grafana 1.0, 2014; fine-grained access control in 5.0, 2018; RBAC in 8.0, 2021; Grafana 9 in 2022) — to a multi-tenant OpenFGA deployment serving Grafana Cloud, embedded OSS, and on-prem from one model. Their Identity and Access team first experimented with OpenFGA in February 2024, made it a mainline integration in August 2024, and presented the migration at KubeCon EU 2025 in London. Everything in the migration playbook below is anchored in their documented lessons, in the CNCF TOC adopter interviews, and in the official OpenFGA documentation.
One honesty note before anything else: every API request, response, and error body shown in Section 2 was executed on this host against the real openfga server v1.22.1 (linux/amd64 release build 44054f5, sha256-verified download) with the in-memory datastore, driven with curl and the fga CLI v0.9.0. Nothing in the wire section is paraphrased from documentation screenshots. That matters for a migration guide: the exact response shapes and error codes are what your dual-write and shadow-check code will parse.
1. The Zanzibar Contract: Tuples, Rewrites, and What OpenFGA Ships
Zanzibar's core abstraction is the relationship tuple: user, relation, object. The tuple user:anne has relation viewer on folder:q3-planning is written as:
folder:q3-planning#viewer@user:anne
Three things make this a database and not a policy file:
- A typed schema (the paper's namespace configuration, OpenFGA's authorization model) declares which relations can exist on which types and how they rewrite into each other. viewer can be defined as direct users or the admin userset or the viewers of the parent folder.
- A tuple store indexed in both directions. The shipped PostgreSQL migrations make this concrete: the tuple table's primary key is (store, object_type, object_id, relation, _user) with partial indexes for direct-user and userset entries plus idx_reverse_lookup_user on (store, object_type, relation, _user) — the forward index answers check, the reverse index answers what-can-this-user-see. Every tuple write also lands in a changelog table keyed by (store, ulid, object_type), which is what the watch-style changes API reads.
- Userset rewrites. A relation's definition is a set-expression tree over this (direct tuples), computedUserset (another relation on the same object), and tupleToUserset (a relation on a related object — the parent-folder hop), combined with or, and, and but not.
OpenFGA implements the paper's Write, Read, Check, Expand, and Watch operations and extends the model in ways the paper does not have: ListObjects and ListUsers reverse queries, CEL-based conditions for attribute-style decisions, contextual tuples (up to 100 per request), and an experimental AuthZEN-compatible Access v1 API. It deliberately does not implement the paper's Spanner-backed global replication, and it does not yet ship Zookies — the paper's write-returned consistency tokens that let a client pin a check to the data version it just read. The consistency documentation is explicit that Zookies are future work, which has real consequences for read-after-write semantics that Section 6 covers. The model that everything below hangs off of is written in OpenFGA's DSL and looks like this:
model
schema 1.1
type user
type group
relations
define member: [user, group#member]
type folder
relations
define parent: [folder]
define admin: [user, group#member]
define editor: [user, group#member]
define viewer: [user, group#member, user:*] or admin or editor or viewer from parent
define can_create_dashboard: admin or editor
define can_share: admin
type dashboard
relations
define parent: [folder]
define owner: [user, group#member]
define editor: [user, group#member] or owner
define viewer: [user, group#member, user:*] or editor or viewer from parent
define can_edit: editor or editor from parent
define can_delete: owner or admin from parent
Read the rewrites the way you would read set algebra. A folder's viewer is: anyone directly granted (a user, every member of a group, or the type-wide wildcard user:* which marks the resource public), or any admin, or any editor, or anyone who is a viewer of the folder this folder hangs from via parent. The dashboard's viewer chains the same trick one level deeper. That single viewer from parent clause is tuple-to-userset: it is what makes hierarchical permission inheritance a property of the model instead of a recursive loop in your application code, and it is what Grafana used to replace folder-permission inheritance that previously lived in application logic.
The DSL compiles to an immutable, versioned JSON authorization model. The server and SDKs speak only JSON; the CLI does the conversion:
fga model transform --file model.fga
# produces schema_version 1.1 type_definitions JSON; the CLI also accepts
# .fga files directly in: fga store create --model model.fga
2. The Wire Contract, Verified Live Against v1.22.1
Migrations are built out of API calls, so this section is the real session, executed against openfga run with the in-memory datastore (fine for a lab; the production datastore is Postgres — Section 4). First, a store and a model:
$ curl -sS -X POST http://127.0.0.1:8080/stores -H "content-type: application/json" \
-d '{"name":"tenant-42"}'
{"id":"01M7H8T3HEC94NG915N63EVTMB", "name":"tenant-42", "created_at":"2026-11-16T04:10:57.454125525Z", "updated_at":"2026-11-16T04:10:57.454125525Z"}
$ curl -sS -X POST http://127.0.0.1:8080/stores/01M7H8T3HEC94NG915N63EVTMB/authorization-models \
-H "content-type: application/json" -d @model.json
{"authorization_model_id":"01M7H8T3HSTSS85RZPDP2CFER3"}
Store and model IDs are ULIDs — the model ID's timestamp is decodable, which is how you date a model version without extra bookkeeping. Authorization models are immutable: you never edit one, you write a new version and get a new ID. Pin that ID in configuration and pass it on every request — the docs recommend it for both correctness (a botched model rollout cannot silently affect traffic that pins the old ID) and latency (no latest-model lookup), and the same immutability is what makes zero-downtime model migrations possible (Section 5).
Now the tuples — group membership, a folder tree, an editor grant expressed as a userset, dashboards parented to folders, and one public dashboard via the wildcard. All ten go in one write call (the API caps at OPENFGA_MAX_TUPLES_PER_WRITE=100 tuples per request by default):
$ curl -sS -X POST http://127.0.0.1:8080/stores/01M7H8T3HEC94NG915N63EVTMB/write -H "content-type: application/json" -d '{
"writes": { "tuple_keys": [
{ "user": "user:anne", "relation": "member", "object": "group:platform" },
{ "user": "user:6", "relation": "admin", "object": "folder:root" },
{ "user": "folder:root", "relation": "parent", "object": "folder:q3" },
{ "user": "group:platform#member", "relation": "editor", "object": "folder:q3" },
{ "user": "folder:q3", "relation": "parent", "object": "dashboard:latency" },
{ "user": "user:bertram", "relation": "owner", "object": "dashboard:latency" },
{ "user": "folder:q3", "relation": "parent", "object": "dashboard:public" },
{ "user": "user:*", "relation": "viewer", "object": "dashboard:public" },
{ "user": "user:anne", "relation": "editor", "object": "dashboard:anne-owned" },
{ "user": "folder:q3", "relation": "parent", "object": "dashboard:anne-owned" }
], "on_duplicate": "ignore" },
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3"
}'
{}
# empty object = success. Note the userset user value: group:platform#member
# grants every current and future member of that group the editor relation.
Then the checks — each one exercising a different part of the rewrite machinery, all real responses:
$ curl -sS -X POST http://127.0.0.1:8080/stores/01M7H8T3HEC94NG915N63EVTMB/check -H "content-type: application/json" -d '{
"tuple_key": { "user": "user:anne", "relation": "viewer", "object": "folder:q3" },
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3"
}'
{"allowed":true, "resolution":""}
# anne is viewer via group:platform#member -> editor -> or-editor rewrite on viewer
$ curl -sS -X POST http://127.0.0.1:8080/check -d '{
"tuple_key": { "user": "user:6", "relation": "viewer", "object": "dashboard:latency" },
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"allowed":true, "resolution":""}
# TWO tuple-to-userset hops resolved on the fly:
# folder:root#admin -> folder:root#viewer -> folder:q3#viewer -> dashboard:latency#viewer
$ curl -sS -X POST http://127.0.0.1:8080/check -d '{
"tuple_key": { "user": "user:999", "relation": "viewer", "object": "dashboard:latency" },
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"allowed":false, "resolution":""}
$ curl -sS -X POST http://127.0.0.1:8080/check -d '{
"tuple_key": { "user": "user:999", "relation": "viewer", "object": "dashboard:public" },
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"allowed":true, "resolution":""}
# user:* wildcard: the public dashboard is visible to a user with zero tuples
The batch API exists for exactly the check-fan-out that listing pages need — one request, up to 50 checks by default (OPENFGA_MAX_CHECKS_PER_BATCH_CHECK), each correlated by an ID matching ^[\w\d-]{1,36}$:
$ curl -sS -X POST http://127.0.0.1:8080/stores/01M7H8T3HEC94NG915N63EVTMB/batch-check \
-H "content-type: application/json" -d '{
"checks": [
{ "tuple_key": { "user": "user:anne", "relation": "viewer", "object": "dashboard:latency" },
"correlation_id": "c1", "authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" },
{ "tuple_key": { "user": "user:6", "relation": "viewer", "object": "dashboard:latency" },
"correlation_id": "c2", "authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" },
{ "tuple_key": { "user": "user:999", "relation": "viewer", "object": "dashboard:public" },
"correlation_id": "c3", "authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" },
{ "tuple_key": { "user": "user:999", "relation": "viewer", "object": "dashboard:latency" },
"correlation_id": "c4", "authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }
]
}'
{
"result": {
"c1": { "allowed": true },
"c2": { "allowed": true },
"c3": { "allowed": true },
"c4": { "allowed": false }
}
}
The reverse queries behave as the docs promise — a check that returns true implies the object appears in the equivalent list-objects call:
$ curl -sS -X POST http://127.0.0.1:8080/stores/01M7H8T3HEC94NG915N63EVTMB/list-objects \
-H "content-type: application/json" -d '{
"user": "user:anne", "relation": "viewer", "type": "dashboard",
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3"
}'
{"objects":["dashboard:public","dashboard:anne-owned","dashboard:latency"]}
$ curl -sS -N -X POST http://127.0.0.1:8080/stores/01M7H8T3HEC94NG915N63EVTMB/streamed-list-objects \
-H "content-type: application/json" -d '{ "user": "user:anne", "relation": "viewer", "type": "dashboard",
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"result":{"object":"dashboard:public"}}
{"result":{"object":"dashboard:anne-owned"}}
{"result":{"object":"dashboard:latency"}}
ListUsers answers the inverse question (who can access this?) — with two sharp edges worth knowing before you write code against it. The object must be passed in structured {type, id} form and user_filters takes exactly one filter; get either wrong and the server does not return a helpful validation message:
# WRONG (flat string form) — actual error returned by v1.22.1:
$ curl -sS -X POST http://127.0.0.1:8080/list-users -d '{
"object": "dashboard:latency", "relation": "viewer",
"user_filters": [ { "type": "user" } ],
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"code":"validation_error","message":"syntax error (line 1:31): unexpected token \"dashboard:latency\""}
# RIGHT (structured form) — concrete users, resolved through the group and parent chain:
$ curl -sS -X POST http://127.0.0.1:8080/list-users -d '{
"object": { "type": "dashboard", "id": "latency" }, "relation": "viewer",
"user_filters": [ { "type": "user" } ],
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"users":[{"object":{"type":"user", "id":"anne"}}, {"object":{"type":"user", "id":"6"}}, {"object":{"type":"user", "id":"bertram"}}]}
# Userset filter — returns the group memberset instead of expanding it:
$ curl -sS -X POST http://127.0.0.1:8080/list-users -d '{
"object": { "type": "dashboard", "id": "latency" }, "relation": "viewer",
"user_filters": [ { "type": "group", "relation": "member" } ],
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"users":[{"userset":{"type":"group", "id":"platform", "relation":"member"}}]}
For debugging a model, Expand returns the full userset tree the resolver will walk — this is the tool you will use when a shadow-check discrepancy appears and you need to see why:
{
"tree": {
"root": {
"name": "dashboard:latency#viewer",
"union": {
"nodes": [
{ "name": "dashboard:latency#viewer", "leaf": { "users": { "users": [] } } },
{ "name": "dashboard:latency#viewer", "leaf": { "computed": { "userset": "dashboard:latency#editor" } } },
{ "name": "dashboard:latency#viewer", "leaf": { "tupleToUserset": {
"tupleset": "dashboard:latency#parent",
"computed": [ { "userset": "folder:q3#viewer" } ] } } }
]
}
}
}
}
Contextual tuples let a check carry ephemeral relationships that are never persisted — the hybrid mode that matters enormously during migration, because it lets you feed the new engine data it has not backfilled yet:
# user:carol has NO stored tuples anywhere. With a contextual membership:
$ curl -sS -X POST http://127.0.0.1:8080/check -d '{
"tuple_key": { "user": "user:carol", "relation": "viewer", "object": "folder:q3" },
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3",
"contextual_tuples": { "tuple_keys": [
{ "user": "user:carol", "relation": "member", "object": "group:platform" } ] }'
{"allowed":true, "resolution":""}
# Without it — same stored state, different answer:
$ curl -sS -X POST http://127.0.0.1:8080/check -d '{
"tuple_key": { "user": "user:carol", "relation": "viewer", "object": "folder:q3" },
"authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"allowed":false, "resolution":""}
CEL conditions are how OpenFGA covers attribute-based decisions without leaving the tuple model. The with <condition> type restriction makes the condition mandatory on every tuple of that relation, and check requests must then supply the condition's runtime parameters in context — miss one and the check fails closed with a validation error, not a silent allow:
type organization
relations
define admin: [user with non_expired_grant]
define can_manage_billing: admin
condition non_expired_grant(current_time: timestamp, grant_time: timestamp, grant_duration: duration) {
current_time < grant_time + grant_duration
}
$ curl -sS -X POST http://127.0.0.1:8080/write -d '{
"writes": { "tuple_keys": [ { "user": "user:peter", "relation": "admin",
"object": "organization:acme",
"condition": { "name": "non_expired_grant",
"context": { "grant_time": "2026-11-16T04:00:00Z", "grant_duration": "1h" } } } ] },
"authorization_model_id": "01M7H8Y0YE9ARJDNK953S17FYD"
}'
{}
$ curl -sS -X POST http://127.0.0.1:8080/check -d '{ "tuple_key": { "user": "user:peter",
"relation": "can_manage_billing", "object": "organization:acme" },
"authorization_model_id": "01M7H8Y0YE9ARJDNK953S17FYD",
"context": { "current_time": "2026-11-16T04:30:00Z" } }'
{"allowed":true, "resolution":""}
$ curl -sS -X POST http://127.0.0.1:8080/check -d '{ "tuple_key": { "user": "user:peter",
"relation": "can_manage_billing", "object": "organization:acme" },
"authorization_model_id": "01M7H8Y0YE9ARJDNK953S17FYD",
"context": { "current_time": "2026-11-16T06:00:00Z" } }'
{"allowed":false, "resolution":""}
$ curl -sS -X POST http://127.0.0.1:8080/check -d '{ "tuple_key": { "user": "user:peter",
"relation": "can_manage_billing", "object": "organization:acme" },
"authorization_model_id": "01M7H8Y0YE9ARJDNK953S17FYD",
"context": {} }'
{"code":"validation_error","message":"failed to evaluate relationship condition: 'non_expired_grant' - tuple 'organization:acme#admin@user:peter' is missing context parameters '[current_time]'"}
Finally, the two behaviors your migration code must handle defensively. The Write API is not idempotent by default — and the error it returns is the one your backfill retry logic will meet:
# Re-writing an existing tuple WITHOUT on_duplicate:"ignore" — actual response:
$ curl -sS -X POST http://127.0.0.1:8080/write -d '{
"writes": { "tuple_keys": [ { "user": "user:anne", "relation": "editor",
"object": "dashboard:anne-owned" } ] }, "authorization_model_id": "01M7H8T3HSTSS85RZPDP2CFER3" }'
{"code":"write_failed_due_to_invalid_input","message":"cannot write a tuple which already exists: user: 'user:anne', relation: 'editor', object: 'dashboard:anne-owned': tuple to be written already existed or the tuple to be deleted did not exist"}
# The changes feed (what powers reconciliation and local permission indexes):
$ curl -sS "http://127.0.0.1:8080/stores/01M7H8T3HEC94NG915N63EVTMB/changes"
{ "continuation_token": "MDFNN0g4VDNKN043QkQyODhOQjRDNFpCWkN8", "changes": [
{ "tuple_key": { "user": "user:anne", "relation": "member",
"object": "group:platform", "condition": null },
"operation": "TUPLE_OPERATION_WRITE",
"timestamp": "2026-11-16T04:10:57.479357854Z" },
{ "tuple_key": { "user": "user:6", "relation": "admin",
"object": "folder:root", "condition": null },
"operation": "TUPLE_OPERATION_WRITE",
"timestamp": "2026-11-16T04:10:57.479357854Z" } ] }
# And the migration command your Kubernetes Job will run, against the memory
# engine (Postgres applies the six shipped migrations instead):
$ openfga migrate --datastore-engine memory
2026-11-16T04:10:57.647Z INFO no migrations to run for `memory` datastore
That is the entire wire surface a migration touches: immutable models, transactional writes with explicit idempotency controls, check/batch-check for enforcement, expand for debugging, the two list queries for UI, conditions for ABAC, and a changelog you can tail. Everything after this point is architecture and orchestration built on those primitives.
3. Multi-Tenant Topology: Per-Tenant Stores vs. One Store with Tenant Types
Before any tuple moves, you make the one architectural decision that everything else inherits: where does the tenant boundary live? OpenFGA gives you two legitimate answers, and Grafana's team and the official docs each landed on a different one.
Option A — one store per tenant (Grafana's choice)
An OpenFGA store is a lightweight, namespaced partition of tuples, models, and assertions — creating one is a single API call (you saw it return in milliseconds in Section 2). Grafana assigns each tenant its own dedicated store inside a deployment that runs one OpenFGA instance set per cluster/region, fronted by a middleware chain that Jo Guerreiro's talk enumerated precisely:
- Authentication middleware — establishes the calling service identity (Grafana authenticates users with OIDC ID tokens whose aud carries the stack/tenant ID and whose sub carries the user, then uses scoped access tokens with NIST 8693-style token exchange for service-to-service calls).
- Tenant store mapping middleware — resolves the tenant ID from the token to the OpenFGA store ID. Grafana's example token carries "aud": "stack:1" and "sub": "user:6"; the middleware turns stack:1 into 01M7H8T3HEC94NG915N63EVTMB and every downstream call is scoped to that store.
- Permission mapping middleware — translates customer-facing permission concepts into the OpenFGA relation format, so callers never learn the model's internal vocabulary.
The payoff is structural. A cross-tenant leak requires the store-mapping middleware to be wrong — a single, auditable piece of code — rather than a modeling mistake anywhere in a shared graph. And when a noisy neighbor appears (Guerreiro's term is "superstar tenant": the one whose fan-out graph eats the database), the fix is surgical: move that tenant's store to a dedicated instance. You cannot do that with a shared store.
Option B — one store, tenant isolation in the model (the documented SaaS pattern)
The official multi-tenant SaaS pattern keeps one store and puts the boundary in the model itself: an organization type holds members, admins, and owners; every resource type (workspace, project, document) declares a parent relation pointing at the organization, and permission relations traverse that edge so membership grants access automatically. Isolation is a consequence of tuple topology: a user only has tuples in organizations they belong to, so list-objects naturally returns nothing cross-tenant — and cross-tenant sharing is still possible on the same model by granting an out-of-organization user a relation on a specific resource. Per-tenant custom roles work the same way: make role a first-class type whose assignee relation accepts users, groups, or other roles, and each tenant invents its own role:acme-billing-manager without model changes. The reference implementation of this pattern (with group nesting and list-users assertions) ships as the multitenant-rbac sample store.
| Dimension | Per-tenant stores (A) | Single store + org type (B) |
|---|---|---|
| Cross-tenant leak surface | Middleware routing only | Every relation in the model |
| Noisy neighbor remedy | Move one store to a dedicated instance | Throttling/concurrency limits shared globally |
| Per-tenant model versioning | Natural (each store pins its own model ID) | Possible but awkward (per-request model ID pinning) |
| Store/tenant lifecycle | Provisioning must create stores; metadata is minimal | One store; tenants are tuples |
| Monitoring cardinality | Per-store dashboards and alerts | Per-tenant metrics need tuple-derived labels |
| Cross-tenant sharing | Requires tuples in both stores or a shared store | Same model, no special case |
| Operational overhead | Higher (store sprawl, per-region placement) | Lower (one database, one rollout) |
Both are production-proven at scale. Grafana runs Option A because their legacy system was already per-stack and the store concept mapped cleanly onto Grafana Cloud's multi-tenant architecture; the docs pattern (Option B) is the right default for a B2B SaaS with hundreds-to-thousands of tenants where cross-tenant sharing is a product feature. Choose before you backfill — tuple schemas and backfill tooling differ between the two.
graph TD
U["End user / service principal"] --> GW["Platform API"]
GW --> AUTHN["Authn middleware: OIDC ID token, aud = stack:1"]
AUTHN --> MAP["Tenant store mapping: stack:1 → store 01M7H8T3HEC94NG915N63EVTMB"]
MAP --> PMAP["Permission mapping middleware"]
PMAP --> FGA["OpenFGA pods, eu-west-1"]
PMAP2["OpenFGA pods, us-east-1"]:::region
classDef region stroke-dasharray: 5 5
FGA --> PG[("PostgreSQL primary + secondary, same region")]
PMAP2 --> PG2[("PostgreSQL primary + secondary, same region")]
APP["Application writes: folder/dashboards/share grants"] -->|"transactional outbox"| FGA
APP -->|"legacy tables"| LEG[("Legacy ACL engine + tables")]
REC["Reconciliation service: periodic diff old vs new"]
FGA -->|"GET /changes + continuation token"| REC
LEG --> REC
MISM["Discrepancy log → alert + fix tickets"]
REC --> MISM
The scale numbers behind this architecture are public. Read AI runs one OpenFGA service handling 5,200 requests per second at peak with 20ms p99 latency (1.8ms average) over 5.3 billion stored tuples on PostgreSQL. Grafana replaced a single-tenant engine with OpenFGA spanning multi-tenant SaaS, embedded OSS, and on-prem. Docker centralized product permissions at 100-150 RPS after a parallel-run migration. Zuplo enforces authorization at the edge across multiple data centers with PostgreSQL global replication. These are not proofs that OpenFGA is magic — they are proofs that the tuple-store design, run on a well-tuned Postgres, holds at the scale most platforms will ever need.
4. Production Deployment: Postgres, Kubernetes, and the Flags That Matter
OpenFGA is a stateless Go binary over a database you already know how to operate. That is the entire deployment story — and it is why the "second database" objection in Section 8 is about cost, not difficulty. The production requirements, straight from the docs: co-locate the database in the same datacenter as the OpenFGA servers, dedicate it to OpenFGA (no shared application databases), and bootstrap it with openfga migrate so the indexes from Section 1 exist.
The Helm path
helm repo add openfga https://openfga.github.io/helm-charts
helm upgrade --install authz openfga/openfga \
--version 0.4.1 \
--namespace openfga --create-namespace \
--set image.tag=v1.22.1 \
--set replicaCount=2 \
--set datastore.engine=postgres \
--set datastore.uriSecret=openfga-postgres-credentials \
--set playground.enabled=false \
--set authn.method=preshared \
--set authn.preshared.keys[0]=$FGA_API_TOKEN \
--set metrics.enabled=true
Three chart facts worth knowing before you hit production. First, the bundled Bitnami PostgreSQL/MySQL subcharts are explicitly deprecated in the chart's values ("uses unsupported legacy archive") — bring your own Postgres via an operator like CloudnativePG or a managed service, which is also the chart's own recommendation. Second, the chart runs schema migrations as a Helm hook Job before the Deployment rolls; the newer openfga-operator chart (0.1.0, November 2026) runs migrations whenever the image or datastore settings change, and its migration.trigger value re-runs them after you rotate a secret to point at a new database. Third, the migration Job can use a dedicated service account with cloud IAM annotations for DDL permissions — use that instead of handing the app's service account DDL rights.
The raw manifests (what the chart renders, minus the subchart noise)
apiVersion: v1
kind: Secret
metadata:
name: openfga-postgres-credentials
namespace: openfga
type: Opaque
stringData:
uri: "postgres://openfga:S3cure-Password-From-Vault@openfga-postgres.openfga.svc.cluster.local:5432/openfga?sslmode=require"
---
apiVersion: v1
kind: Secret
metadata:
name: openfga-api-token
namespace: openfga
type: Opaque
stringData:
preshared_key: "preshared-key-rotated-via-external-secrets"
apiVersion: batch/v1
kind: Job
metadata:
name: openfga-migrate
namespace: openfga
spec:
backoffLimit: 3
ttlSecondsAfterFinished: 3600
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: openfga/openfga:v1.22.1
args: ["migrate"]
env:
- name: OPENFGA_DATASTORE_ENGINE
value: "postgres"
- name: OPENFGA_DATASTORE_URI
valueFrom:
secretKeyRef:
name: openfga-postgres-credentials
key: uri
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1"
memory: "512Mi"
apiVersion: apps/v1
kind: Deployment
metadata:
name: openfga
namespace: openfga
labels:
app: openfga
spec:
replicas: 2
selector:
matchLabels:
app: openfga
template:
metadata:
labels:
app: openfga
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2112"
spec:
containers:
- name: openfga
image: openfga/openfga:v1.22.1
args: ["run"]
ports:
- { name: http, containerPort: 8080 }
- { name: grpc, containerPort: 8081 }
- { name: metrics, containerPort: 2112 }
env:
# -- datastore: dedicated Postgres, connections divided across replicas.
# Database max_connections=100 / 2 replicas = 50 per pod.
- name: OPENFGA_DATASTORE_ENGINE
value: "postgres"
- name: OPENFGA_DATASTORE_URI
valueFrom:
secretKeyRef:
name: openfga-postgres-credentials
key: uri
- name: OPENFGA_DATASTORE_MAX_OPEN_CONNS
value: "50"
- name: OPENFGA_DATASTORE_MIN_OPEN_CONNS
value: "10" # baseline pool; must be >= MIN_IDLE_CONNS
- name: OPENFGA_DATASTORE_MAX_IDLE_CONNS
value: "10"
# -- authentication: preshared keys for service-to-service callers,
# or OIDC (authn.method=oidc) to verify caller JWTs directly.
- name: OPENFGA_AUTHN_METHOD
value: "preshared"
- name: OPENFGA_AUTHN_PRESHARED_KEYS
valueFrom:
secretKeyRef:
name: openfga-api-token
key: preshared_key
# -- caching: check subproblem + iterator caches with changelog-driven
# invalidation. This is the latency lever; see Section 6 for the
# consistency trade it buys.
- name: OPENFGA_CHECK_QUERY_CACHE_ENABLED
value: "true"
- name: OPENFGA_CHECK_ITERATOR_CACHE_ENABLED
value: "true"
- name: OPENFGA_CACHE_CONTROLLER_ENABLED
value: "true"
- name: OPENFGA_CHECK_CACHE_LIMIT
value: "10000"
- name: OPENFGA_CHECK_ITERATOR_CACHE_TTL
value: "30s"
# -- hardening and observability
- name: OPENFGA_PLAYGROUND_ENABLED
value: "false" # deprecated component; local dev only
- name: OPENFGA_LOG_FORMAT
value: "json"
- name: OPENFGA_LOG_LEVEL
value: "info"
- name: OPENFGA_METRICS_ENABLED
value: "true" # Prometheus on :2112/metrics
- name: OPENFGA_DATASTORE_METRICS_ENABLED
value: "true"
- name: OPENFGA_TRACE_ENABLED
value: "true"
- name: OPENFGA_TRACE_SAMPLE_RATIO
value: "0.2"
- name: OPENFGA_TRACE_OTLP_ENDPOINT
value: "otel-collector.observability.svc.cluster.local:4317"
readinessProbe:
exec:
command: ["/usr/local/bin/grpc_health_probe", "-addr=:8081"]
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
livenessProbe:
exec:
command: ["/usr/local/bin/grpc_health_probe", "-addr=:8081"]
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 12
resources:
requests:
cpu: "1"
memory: "1Gi"
limits:
cpu: "2"
memory: "2Gi"
---
apiVersion: v1
kind: Service
metadata:
name: openfga
namespace: openfga
spec:
selector:
app: openfga
ports:
- { name: http, port: 8080, targetPort: 8080 }
- { name: grpc, port: 8081, targetPort: 8081 }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: openfga
namespace: openfga
spec:
minAvailable: 1
selector:
matchLabels:
app: openfga
The health checks are not decorative: the gRPC health service (grpc.health.v1.Health/Check) is wired to datastore testing, and the HTTP server exposes /healthz — we verified both on the live server (the response is {"status":"SERVING"}). A pod that cannot reach Postgres will fail readiness, which is precisely what you want under a connection storm.
Connection-pool math, concurrency limits, and the flags you will actually tune
The docs' database guidance is unusually concrete: OPENFGA_DATASTORE_MAX_OPEN_CONNS should equal the database's max_connections divided across replicas (the worked example: Postgres at 100 connections, two OpenFGA instances, 50 each); OPENFGA_DATASTORE_MIN_IDLE_CONNS must not exceed OPENFGA_DATASTORE_MIN_OPEN_CONNS (the server validates it at startup); and idle-connection churn should be avoided by keeping connections around longer rather than shorter — re-establishing connections is the slow path that shows up as p99 spikes. When you observe database contention from aggressive queries, the concurrency guardrails are OPENFGA_MAX_CONCURRENT_READS_FOR_CHECK, OPENFGA_MAX_CONCURRENT_READS_FOR_LIST_OBJECTS, and the resolution-depth caps OPENFGA_RESOLVE_NODE_LIMIT (default 25 — a hard bound on how deep your hierarchy can resolve) and OPENFGA_RESOLVE_NODE_BREADTH_LIMIT (default 10). For Postgres you can also point reads at a replica with OPENFGA_DATASTORE_SECONDARY_URI — with the v1.22 routing caveats in Section 6.
Model as code
Stores and models are API objects, which means they are Terraform-able. Headspace ships its model and static enablement tuples through the OpenFGA Terraform provider so permission-schema changes go through the same review pipeline as infrastructure; the pattern is four blocks:
terraform {
required_providers {
openfga = {
source = "openfga/openfga"
}
}
}
provider "openfga" {
api_url = "https://authz.internal.example.com" # your OpenFGA HTTP endpoint
# credentials via the provider's documented auth options
}
resource "openfga_store" "tenant_42" {
name = "tenant-42"
}
data "openfga_authorization_model_document" "model" {
dsl = file("model.fga") # the same DSL file the fga CLI and CI tests use
}
resource "openfga_authorization_model" "current" {
store_id = openfga_store.tenant_42.id
model_json = data.openfga_authorization_model_document.model.result
}
The openfga_authorization_model_document data source is the piece that makes this sane: it converts the DSL to JSON and only emits a diff when the model changes semantically — formatting churn does not create new model versions. Note what Terraform does deliberately not manage here: relationship tuples written at runtime. Those belong to your application and migration tooling, not to state files — a distinction the source-of-truth guidance in the docs makes explicit (OpenFGA is the right system of record for fine-grained grants and role membership; entity hierarchies and search-relevant metadata belong in your application database).
5. The Migration Playbook: Orchestrating the Cut Over Without Breaking Authorization
Grafana's team framed their migration in three moves — schema, data migration, data retrieval — and then learned, in public, how much operational scaffolding those three moves require when the payload is 14 years of accumulated permission semantics across 24+ entity types. What follows is the full orchestration, five phases, each with a concrete gate. Every mechanism in it comes from their documented lessons, the adopter interviews, or the official migration guides — and the shadow/reconciliation scripts below are built on the exact wire semantics validated in Section 2.
stateDiagram-v2
[*] --> Phase0_Model_Tests
Phase0_Model_Tests --> Phase1_DualWrite : model green in CI
Phase1_DualWrite --> Phase2_Backfill : new writes landing in both systems
Phase2_Backfill --> Phase3_Shadow : tuple counts match legacy snapshot
Phase3_Shadow --> Phase4_Reconcile : discrepancy rate near zero
Phase4_Reconcile --> Phase5_Cutover : per-tenant flags armed
Phase5_Cutover --> LegacyContained : all reads on OpenFGA
LegacyContained --> [*]
Phase5_Cutover --> Phase4_Reconcile : flag flipped back = rollback
Phase3_Shadow --> Phase2_Backfill : model or backfill defects found
Phase 0 — Freeze the model, gate it in CI
The migration's real deliverable is the model. Grafana's most-cited potholes were all model-level: the number of relations to write, schema complexity directly affecting performance, and database-specific differences behind the same DSL. Their team's own retrospective — "schematizing our access control helped us find inconsistencies in our UX" — cuts both ways: the DSL will expose permission semantics your product teams forgot they had, and each discovery becomes scope. Budget for that, cut the verb list ruthlessly (reducing the number of access-control verbs was an explicit Grafana outcome), and then make the model a tested artifact before any data moves. The fga CLI runs model-plus-tuple test suites offline:
name: platform-access-model-tests
model: |
model
schema 1.1
type user
type group
relations
define member: [user, group#member]
type folder
relations
define parent: [folder]
define admin: [user, group#member]
define editor: [user, group#member]
define viewer: [user, group#member, user:*] or admin or editor or viewer from parent
define can_create_dashboard: admin or editor
define can_share: admin
type dashboard
relations
define parent: [folder]
define owner: [user, group#member]
define editor: [user, group#member] or owner
define viewer: [user, group#member, user:*] or editor or viewer from parent
define can_edit: editor or editor from parent
define can_delete: owner or admin from parent
tuples:
- user: user:anne
relation: member
object: group:platform
- user: user:6
relation: admin
object: folder:root
- user: folder:root
relation: parent
object: folder:q3
- user: group:platform#member
relation: editor
object: folder:q3
- user: folder:q3
relation: parent
object: dashboard:latency
- user: user:bertram
relation: owner
object: dashboard:latency
tests:
- name: viewer inherits through group membership
check:
- user: user:anne
object: folder:q3
assertions:
viewer: true
can_share: false
- name: folder admin inherits two hops down the tree
check:
- user: user:6
object: dashboard:latency
assertions:
viewer: true
- name: ungranted user denied on a private dashboard
check:
- user: user:999
object: dashboard:latency
assertions:
viewer: false
- name: owner can edit and delete, cannot share
check:
- user: user:bertram
object: dashboard:latency
assertions:
can_edit: true
can_delete: true
can_share: false
fga model test --tests model.fga.yaml
# prints a per-test pass/fail summary; a non-zero exit fails the build.
# The OpenFGA VS Code extension validates the DSL while you edit.
Phase 1 — Dual-write, from day one, in both directions of behavior
Grafana's first hard rule, learned from previous migrations being toggled back and forth: dual-write always. Every permission mutation — share, role change, folder move, team edit — writes to the legacy engine and to OpenFGA simultaneously, so that feature toggles can flip in either direction without state divergence. The corollary they insist on: when you fix an undesired corner case in the new system, fix it in the legacy system too, because that corner case is exactly why a customer will cling to the old behavior and force a toggle-back.
Mechanically, the write side is where the Section 2 semantics earn their keep: writes are transactional per call, capped at 100 tuples, and non-idempotent by default — your mutation path should use "on_duplicate": "ignore" and "on_missing": "ignore" so retries are safe, and handle the write_failed_due_to_invalid_input error class explicitly. For atomicity against your application database, use the transactional outbox pattern — Vitrolife's team commits the application row and the OpenFGA tuple mutation together in one Wolverine-managed outbox, with consumers draining the outbox into OpenFGA, bounding divergence to the outbox lag rather than hoping two writes happen to both succeed. Under write bursts (enrollment events, bulk imports), concurrent tuple writes collide; the SDK's conflict-resolution behavior (shipped in SDK generation 1.10, per Headspace's production account) is the mitigation.
Phase 2 — Backfill the history, then distrust it
Backfill is a batch program over the legacy tables: read, transform to tuples, write in batches of 100, verify. Two rules from the people who did it: don't trust one-time data migrations (Grafana saw upgrade/downgrade cycles on-prem roll back the schema while the new store kept newer state — they now verify state between the two stores continuously rather than once), and treat tuple count as a cost — in Guerreiro's words, "the best tuple is the one you don't have to write." Every relationship the model can derive (inheritance via parent, admin-implies-viewer rewrites) should never be backfilled as an explicit tuple.
#!/usr/bin/env bash
# backfill-openfga.sh
# legacy-acl-export.jsonl: one JSON object per line: {"user":"user:6","relation":"viewer","object":"dashboard:latency"}
set -euo pipefail
FGA_API_URL="${FGA_API_URL:?set FGA_API_URL}"
FGA_STORE_ID="${FGA_STORE_ID:?set FGA_STORE_ID}"
FGA_MODEL_ID="${FGA_MODEL_ID:?set FGA_MODEL_ID}"
LEGACY_EXPORT="${1:?usage: backfill-openfga.sh legacy-acl-export.jsonl}"
BATCH_SIZE=100
CHUNK_DIR="$(mktemp -d)"
split -l "$BATCH_SIZE" -d -a 4 --additional-suffix=.jsonl "$LEGACY_EXPORT" "$CHUNK_DIR/chunk-"
count=0
for chunk in "$CHUNK_DIR"/chunk-*.jsonl; do
jq -s --arg mid "$FGA_MODEL_ID" '{
writes: { tuple_keys: [.[] | {user: .user, relation: .relation, object: .object}],
on_duplicate: "ignore" },
authorization_model_id: $mid }' "$chunk" > "$CHUNK_DIR/batch.json"
curl -sS -X POST "$FGA_API_URL/stores/$FGA_STORE_ID/write" \
-H "content-type: application/json" \
-d @"$CHUNK_DIR/batch.json" > /dev/null
count=$((count + 1))
echo "batch $count written ($(basename "$chunk"))"
done
echo "backfill complete: $(ls "$CHUNK_DIR"/chunk-*.jsonl | wc -l) chunks, $BATCH_SIZE tuples per call max"
Verification is symmetric: export OpenFGA's state with the read API (or the CLI's fga tuple read), and diff against the legacy snapshot per relation. The tuple-count comparison per (type, relation) is your cheapest integrity signal; row-level diffs catch systematic transform bugs.
Phase 3 — Shadow calls: ask the new engine, ignore its answers
The docs' shadow-mode pattern and Grafana's rollout agree on the mechanics: while the legacy system keeps making the real decisions, every (or sampled) check also fires asynchronously at OpenFGA, discrepancies get logged, and the logs drive model and backfill fixes. Grafana went further and shadowed real production traffic from external users in a pre-production capacity before any read cutover — the highest-fidelity test data that exists. The harness below is the OpenFGA side of that pattern; the legacy side is whatever your current decision procedure is:
#!/usr/bin/env bash
# shadow-check.sh
# decisions-sample.jsonl: {"user":"user:6","relation":"viewer","object":"dashboard:latency","legacy":true}
set -euo pipefail
FGA_API_URL="${FGA_API_URL:?set FGA_API_URL}"
FGA_STORE_ID="${FGA_STORE_ID:?set FGA_STORE_ID}"
FGA_MODEL_ID="${FGA_MODEL_ID:?set FGA_MODEL_ID}"
SAMPLES="${1:?usage: shadow-check.sh decisions-sample.jsonl}"
total=0
mismatch=0
while IFS= read -r line; do
total=$((total + 1))
user="$(printf '%s' "$line" | jq -r '.user')"
relation="$(printf '%s' "$line" | jq -r '.relation')"
object="$(printf '%s' "$line" | jq -r '.object')"
legacy="$(printf '%s' "$line" | jq -r '.legacy')"
fga="$(curl -sS -X POST "$FGA_API_URL/stores/$FGA_STORE_ID/check" \
-H "content-type: application/json" \
-d "{\"tuple_key\":{\"user\":\"$user\",\"relation\":\"$relation\",\"object\":\"$object\"},\"authorization_model_id\":\"$FGA_MODEL_ID\"}" \
| jq -r '.allowed')"
if [ "$fga" != "$legacy" ]; then
mismatch=$((mismatch + 1))
printf 'MISMATCH user=%s relation=%s object=%s legacy=%s fga=%s\n' \
"$user" "$relation" "$object" "$legacy" "$fga" >&2
fi
done < "$SAMPLES"
echo "shadow sample: total=$total mismatches=$mismatch"
# For per-mismatch diagnosis: run expand on the object/relation pair to dump
# the userset tree OpenFGA walked, and compare it to the legacy grant chain.
Two operational notes. First, shadow calls measure the performance impact of adding an authorization network hop before it is load-bearing — capture OpenFGA check latency separately from application latency at this stage (the metrics endpoint histograms and tracing with a low sample ratio exist for exactly this). Second, discrepancy triage is where expand earns its keep: for every mismatch, expand the relation on the object and read the tree against what the legacy engine walked.
Phase 4 — The reconciliation loop: never fully trust either system again
Grafana's slides put it as a one-line diagram: a permission reconciliation loop. The lesson behind it: between dual-write start and read cutover, you have two live permission systems and months of window — on-prem customers upgrade and downgrade, toggles flip, backfills rerun. A periodic diff job between the two systems is not a nicety; it is the mechanism that lets you keep promising correctness to customers during a quarters-long migration. The changes feed is also the incremental transport: tail it with a persisted continuation token, and every tuple write/delete that OpenFGA has seen since the last pass is replayable against the legacy state without a full snapshot:
#!/usr/bin/env bash
# reconcile-tail.sh [type-filter]
# The legacy-side comparison is the integration seam: the loop below emits
# every change, and the consumer of each line encodes YOUR system's
# decision procedure. The OpenFGA side is complete.
set -euo pipefail
FGA_API_URL="${FGA_API_URL:?set FGA_API_URL}"
FGA_STORE_ID="${FGA_STORE_ID:?set FGA_STORE_ID}"
TOKEN_FILE="${TOKEN_FILE:-/var/lib/reconcile/$FGA_STORE_ID.token}"
TYPE_FILTER="${1:-dashboard}"
TOKEN=""
if [ -f "$TOKEN_FILE" ]; then TOKEN="$(cat "$TOKEN_FILE")"; fi
RESP="$(curl -sS -G "$FGA_API_URL/stores/$FGA_STORE_ID/changes" \
--data-urlencode "type=$TYPE_FILTER" \
--data-urlencode "continuation_token=$TOKEN")"
# Persist the token BEFORE processing, so a crash re-reads the same page.
printf '%s' "$RESP" | jq -r '.continuation_token' > "$TOKEN_FILE.tmp" && mv "$TOKEN_FILE.tmp" "$TOKEN_FILE"
printf '%s' "$RESP" | jq -r '.changes[]? | [(.operation // "unknown"), .tuple_key.user, .tuple_key.relation, .tuple_key.object] | @tsv' \
| while IFS=$'\t' read -r op user relation object; do
echo "change op=$op user=$user relation=$relation object=$object"
# Integration seam: evaluate the same (user, relation, object) against the
# legacy engine. Any disagreement is drift — alert, repair, and re-verify.
done
Run the loop on a schedule (the Grafana team's guidance is "verify the state often," and their reconciliation loop ran continuously through the migration), and treat a zero-drift window as one of the cutover gates — not as a reason to stop reconciling.
Phase 5 — Cutover by tenant, with the legacy system contained, not deleted
The final phase is boring on purpose: flip reads per tenant behind feature flags, watch error budgets and discrepancy logs, expand the percentage. Grafana's lessons are all discipline: feature toggles will be toggled (so dual-write must remain on), don't rush, and contain the legacy system — keep it working but route all new authorization development to OpenFGA, so the migration surface stops growing. Because dual-write never stopped, rollback is a flag flip, not a data recovery project.
Model evolution after cutover uses the same discipline, formalized by immutable models. A relation rename is the canonical worked example in the docs, and it is four steps: (1) write a backwards-compatible model where the new relation coexists with the old — a new model ID, not yet serving traffic; (2) write new tuples on the new relation; (3) copy existing tuples across with read-then-write (writes before deletes — access never disappears mid-migration), re-check, then delete the old tuples; (4) write a model without the obsolete relation and pin services to it. Stale tuples for deleted relations are ignored during evaluation but still cost retrieval time — clean them up. And because model IDs are ULIDs, you can decode any pinned ID's creation timestamp when you need to know when a tenant last moved forward.
Timeline honesty, from the only public multi-tenant migration of this scale: Grafana's first OpenFGA experiments ran in February 2024, mainline integration landed in August 2024, and the KubeCon EU 2025 talk in April 2025 described external production still shadowing real traffic. This is a multi-quarter program. The playbook keeps it safe, not fast.
6. Consistency: What the Caches Cost You, and What Zanzibar Would Have Given You
Authorization has a read-after-write reflex that most application data does not: share a dashboard with a user, and the user (and their auditor) expects the very next request to reflect it. This is where Zanzibar-grade systems do the most subtle work, and where OpenFGA's model needs to be understood precisely.
With caching disabled — the default — every query is strongly consistent: reads go to the datastore, and a write is visible to the next check. When you enable the latency stack from Section 4 (check subproblem cache, iterator cache, cache controller), the consistency modes start to matter:
| Consistency mode | Behavior | When to use |
|---|---|---|
| MINIMIZE_LATENCY (default) | Serves Check and (partially) ListObjects from cache when possible; bounded staleness on the order of the cache TTL | The steady-state read path, when you can tolerate a small revocation window |
| HIGHER_CONSISTENCY | Skips the caches and queries the datastore directly | Read-after-write paths — right after a permission mutation affecting the same user/object pair |
The cache controller narrows the staleness window without giving up the caches: it checks the changelog for recent writes and invalidates affected entries, so a revocation lands sooner than the TTL would suggest — at the cost of extra datastore queries (which is why it, too, has a rate-limiting TTL). The docs' recommended hybrid is to decide consistency per request from your own data: keep a modified_date on the protected row, and only send HIGHER_CONSISTENCY when the row changed within the cache's TTL window. That gets you cache-speed reads and correctness exactly where it matters, without paying the datastore round-trip on every check.
Two more consistency surfaces to know. First, read replicas: OPENFGA_DATASTORE_SECONDARY_URI routes reads to the replica except for HIGHER_CONSISTENCY requests — but as of v1.22, GetStore, ListStores, ReadAssertions, ReadAuthorizationModel(s), FindLatestAuthorizationModel, and ReadChanges always hit the replica when one is configured, so a lagging replica can bite you on read-after-write for those endpoints (model introspection right after a model write is the classic case). Second, OpenFGA does not yet implement Zookies — the write-returned consistency token from the paper that lets a client pin a check to the data version it just observed. The docs list it as future work. Until it exists, your fencing is the modified-date pattern above plus HIGHER_CONSISTENCY on hot mutation paths; do not assume a global causal ordering you did not build.
7. The Search Problem: What a User Can Find After Fine-Grained Permissions
The moment every dashboard is permissioned individually, search can no longer be a plain database query — you need the intersection of "matches the filter" (your database) and "the user can see it" (OpenFGA). The docs enumerate three strategies; Grafana's contribution is the pragmatic fourth:
- Search then check — filter and sort in your database, then /batch-check the results and drop what the user cannot see. Best when the query is narrow. Openlane made this work by overfetching candidates from Postgres (capped at 100 per page, up to 1,000 overfetched) and filtering with one batch-check round trip — after their first ListObjects-based implementation measured ~8-second worst-case latency.
- Filter by allowed IDs — call /list-objects first and push the returned IDs into the database query. Viable when the user can access roughly a thousand or fewer objects of the type; breaks down as the ID set grows, because a partial list cannot back a sorted search.
- Local permission index — consume the changes feed into your own index, intersect there, then still check each result to guard against un-indexed revocations. This is the docs' answer for Google-Drive-shaped workloads (many accessible objects, low percentage of the total) and the reason Grafana says a local index is "an extra component to deploy and maintain."
- Pragmatic pre-filtering (Grafana's addition) — filter aggressively in the database first with context you already have, like tenant membership from the ID token, and duplicate just enough model logic in the query to shrink the candidate set. This transforms a worst-case scenario (D) into an easy one, and it is why multi-tenant search does not need exotic infrastructure.
| Scenario | DB query results | Objects user can access | % of total | Recommended strategy |
|---|---|---|---|---|
| A — narrow search | Low | — | — | Search then check |
| B — small total pool | Low | Low | High | Search then check |
| C — few accessible objects | High | Low (~1k) | Low | Filter by allowed IDs, or local index |
| D — Drive-like | High | High | Low | Local index from changes, then check |
| E — mostly public | High | High | High | Search then check, or local index |
Remember the sharp edges from the live session when you build this: ListObjects results are bounded by both a deadline (default 3 seconds — OPENFGA_LIST_OBJECTS_DEADLINE) and a result cap (default 1,000 — OPENFGA_LIST_OBJECTS_MAX_RESULTS), and results are not sorted and not deterministic across calls. The streaming variant lifts the result cap and keeps only the deadline. Headspace's architectural lesson generalizes to all of this: their original model put the feature on top with all users below, so a check traversed the user population; inverting the graph so the user is the object and access resolves through small relations dropped end-to-end latency from 10-15 seconds to 10-15 milliseconds. Model shape is the first optimization; caching is the second; infrastructure is a distant third.
8. Production Realities: Where This Pattern Hurts
- Tuple count is a first-class cost. Grafana's warnings were specific: excessive tuples degrade performance, and schema complexity — not tuple count alone — is a performance variable. Every migration meeting should treat "can the model derive this?" as the default question for every proposed tuple. Intersections and exclusions (and, but not) are more expensive to evaluate than unions, per the docs — Headspace's four-parallel-batch-check wrapper exists because intersections over large sets cost real time.
- Resolution depth and breadth are capped, and your hierarchy must fit. OPENFGA_RESOLVE_NODE_LIMIT defaults to 25 — deeper hierarchies error out — and breadth is capped at 10 concurrent resolutions per level by default. A tenant with pathological folder nesting will hit these before your p99 dashboard shows anything.
- CEL condition evaluation has a cost budget (OPENFGA_MAX_CONDITION_EVALUATION_COST, default 100) — heavy expressions fail requests rather than silently succeeding. Condition misses fail closed, as Section 2's live error showed.
- The cache/consistency dial is yours to own. Caching is off by default (strong consistency, higher latency); turning the cache on introduces a bounded staleness window on revocations. If your security model requires instant revocation propagation, either keep the caches off, keep TTLs short, or use the modified-date + HIGHER_CONSISTENCY hybrid — and test it with a revocation SLO.
- MySQL is a second-class citizen. The README flags stricter tuple length limits on MySQL; Grafana found the Postgres adapter the mature one and standardized there; Read AI runs billions of tuples on Postgres. SQLite is beta (Grafana contributed it for embedded OSS use). Choose Postgres unless you have a reason you can defend.
- You are adopting a stateful system. A dedicated, co-located Postgres with correctly-sized connection pools (divided across replicas), its own backups, its own capacity planning, and its own upgrade cadence. Read AI folds OpenFGA upgrades into a monthly cadence and reports smooth version upgrades, but it is a standing operational responsibility — Grafana's team noted pluggable storage as a future need precisely because datastore operational fit varies by environment.
- The migration window is measured in quarters, not sprints. Dual-write debt, reconciliation jobs, and per-tenant cutover flags are live infrastructure for the entire program. Grafana's own arc — experiments February 2024, mainline August 2024, still shadowing production traffic publicly in April 2025 — is the honest baseline.
- Noisy neighbors are a tenant-shape problem. Per-tenant stores let you relocate a superstar tenant; a single shared store does not. In both topologies, concurrency-limit flags are your only in-place defense — plan the isolation story before the fan-out tenant arrives, not after.
- Contextual tuples are a hybrid, not a home. Capped at 100 per request, and the docs are blunt that overusing them forfeits the main benefit of the Zanzibar approach (not collecting permission data before each decision). Token-claim-derived contextual tuples also keep granting until token expiry even when the underlying claim changes — bound your token lifetimes accordingly (Vitrolife intersects Entra group claims as contextual tuples and reconciles hourly, accepting exactly this window).
- Model bugs are production security incidents, but they are also greppable. The flip side of schema-driven permissions: one wrong define line is a fleet-wide vulnerability — and also a one-line fix, testable offline, deployable by writing a new immutable model. Budget for model review the way you budget for database schema review; it deserves at least that ceremony.
9. Who Should Skip This Pattern
- Teams with static, coarse-grained RBAC that fits their IdP. If your authorization decisions are all "role X can do action Y" and your identity provider already evaluates that at the token level, a tuple store adds a stateful system to operate and a migration to run for zero new expressiveness. Keep it in the IdP.
- Single-tenant internal tools. One audience, one boundary — Kubernetes RBAC or a policy engine wired to your ingress is a weekend of work, not a quarter of migration. Multi-tenancy is the load-bearing reason for the Zanzibar shape.
- Policy-only workloads with no relationship dimension. If your decisions are attribute arithmetic over request context (time, IP, quota) with no sharing, inheritance, or hierarchy, a policy engine (OPA-style) is the right primitive — OpenFGA's conditions exist to decorate relationships, not to replace a policy engine.
- Teams who cannot staff the migration. The playbook's safety comes from dual-write, shadow traffic, and a reconciliation loop — all built and operated by you. If nobody owns that work for multiple quarters, a Zanzibar migration is not de-risked by the tool; it is just a risky migration.
- Products that need instant global replication today. OpenFGA is single-database per deployment; multi-region means you operating Postgres replication and accepting the v1.22 replica-routing caveats (Zuplo does exactly this — it is real, but it is your replication engineering, not Zanzibar's).
For everyone else — multi-tenant platforms with resource-level sharing, inheritance hierarchies, or a sprawl of bespoke permission logic — OpenFGA in 2026 is the strongest Zanzibar-lineage bet in the CNCF ecosystem: incubating with a real maintainer bench (Grafana Labs and GitPod engineers among them), five SDKs, a CLI with offline model tests, a Terraform provider, an official Helm chart plus an operator, and production users at billions of tuples. The migration is the price; the payoff is that permission questions stop being a per-service research project and become a single, testable, greppable artifact — the same consolidation Zanzibar bought Google in 2019, minus the Spanner bill.