Platform Monkey ctx

Argo CD 3.6.1: The Patch That Makes 3.6 Deployable — and the Diff Engine Flip Underneath

· By
Sources

Argo CD v3.6.1 is three commits: two UI fixes and a version bump. By itself, that is the most forgettable release of the quarter — unless you were one of the teams whose Argo CD dashboard went blank on November 4, in which case it is the only release of the quarter that matters. The patch landed just under sixteen hours after v3.6.0 went stable, and it is the first version of the 3.6 line that belongs in production. What broke, what the fix says about how the project ships, and the diff-engine change underneath 3.6 that matters more than both UI fixes — that is this release, examined properly.

If you are arriving fresh: we profiled Argo CD 3.6.0-rc1 in September, when the line was promising AppSet progressive-sync metrics and CNPG hibernation actions. Seven weeks and four release candidates later, stable shipped on November 3 carrying 86 features, 206 bug fixes, and 367 dependency updates — plus a regression that turned every reverse-proxied Argo CD UI into a white screen within a day.

Sixteen hours: how an HTML minifier broke sub-path deployments

The headline fix in v3.6.1 is PR #30763 by augustomenin: fix(ui): stop webpack minifying index.html so --rootpath/--basehref work. The failure mode is painfully specific, and worth understanding because it is a class of bug that eats any project which embeds a web asset and rewrites it at serve time.

Argo CD can serve its UI under an optional sub-path: set --rootpath (the reported case used server.rootpath: /tools/argocd in argocd-cmd-params-cm) and argocd-server rewrites the base tag of its embedded index.html so the browser resolves JavaScript bundles under your prefix instead of the site root. The rewrite is a single regular expression in server/server.go that matches the quoted form of the tag — and only the quoted form.

// the embedded base tag, as shipped inside each argocd-server image

v3.5.7   <base href="/">     quoted   - regex matches, rewritten to your rootpath
v3.6.0   <base href=/>       unquoted - regex misses, base stays "/"

// server/server.go (identical in both releases)
baseHRefRegex = regexp.MustCompile(`<base href="(.*?)">`)
                                     ^^ quoted form only

// the reporter's proof, run against the published images:
$ docker create quay.io/argoproj/argocd:v3.5.7   # then copy /usr/local/bin/argocd
$ grep -aoE '<base href[^>]*>' argocd-v3.5.7
<base href="/">                                   # matches the regex
$ grep -aoE '<base href[^>]*>' argocd-v3.6.0
<base href=/>                                    # does not match

// consequence: the browser requests /main.<hash>.js at the domain
// root, receives 404s, and renders a blank page behind the proxy.

Why the regex missed. The 3.6 line overhauled the UI build: lazy loading, virtual scrolling, and a React Compiler pass are all in the 3.6.0 changelog. Somewhere in that overhaul, the build's HTML minifier started stripping attribute quotes from index.html. An unquoted href is valid HTML and invisible to any test that requests the page with a real browser and a default configuration — but Argo CD's server-side rewrite was never a browser; it was a regex tuned to the old build's output. Two components that each worked in isolation composed into a blank page for every deployment serving the UI under anything other than the site root. The fix did not loosen the regex: it stopped webpack from minifying index.html at all, trading a few hundred bytes for an entire class of silent breakage.

The upstream report is a model of forensic quality: the reporter pulled the published container images for both versions and diffed the embedded asset directly, proving the tag shape changed between 3.5.7 and 3.6.0 while the serving regex stayed untouched. Every installation running the UI under a sub-path — the standard pattern for shared internal tooling domains — got a blank page on upgrade. Rolling back to v3.5.7 fixed it immediately, which is also your incident playbook if you already shipped 3.6.0.

2026-09-16   v3.6.0-rc1    first RC
2026-10-06   v3.6.0-rc2
2026-10-15   v3.6.0-rc3
2026-10-22   v3.6.0-rc4    final RC
2026-11-03   22:29   v3.6.0 stable   86 features, 206 fixes - blank-UI bug ships
2026-11-04   09:10   chart 10.12.0   app v3.6.0 published to the Helm repo
2026-11-04   12:42   PR #30768       rootpath fix cherry-picked to release-3.6
2026-11-04   13:21   PR #30771       sidebar fix cherry-picked to release-3.6
2026-11-04   14:20   v3.6.1 stable   15h 51m after v3.6.0
2026-11-04   15:25   chart 10.12.1   app v3.6.1 published

Two details matter for planning upgrade windows. First, the Helm chart published 10.12.0 (app v3.6.0) at 09:10 UTC on November 4, so the chart's broken window was roughly six hours — teams upgrading on a schedule never saw it; teams upgrading on reflex did. Second, the maintainer response time is the real signal here: report to patched stable in under a day is what a healthy CNCF project looks like under stress.

One honest smell from the release machinery, since we are in the business of noticing: v3.6.1's release notes still point readers at an "official blog post" that is the v3.0 release-candidate announcement. The release-branch template apparently has not been touched in three minors. Harmless — but a reminder that even graduated CNCF projects ship their release notes with tired-human energy.

The second fix: sidebar state

Cherry-pick PR #30771 keeps the sidebar layout in sync after toggling it, from shimagrawal. Cosmetic in isolation; representative in aggregate — the 3.6 UI shipped a layout overhaul, and the follow-up patches are exactly the state-desync bugs you expect a fresh frontend to shed in its first week. Note what v3.6.1 does not contain: no controller, repo-server, or API changes whatsoever. If you already run 3.6.0 with the UI served at the site root, there is no operational urgency here.

The real 3.6 headline: SMD is dead, and SSA now means SSD

The UI fixes are the excuse; the substance of this line is PR #29103 — Default to SSD with SSA and remove SMD — merged by pjiang-dev on September 9, closing issue #24134. The mechanics, per the official upgrade guide:

argocd.argoproj.io/compare-options: ServerSideDiff=false
        |
        v   explicit opt-out ALWAYS WINS
LEGACY CLIENT-SIDE DIFF (three-way / two-way, normalized)

otherwise, Server-Side Diff runs when ANY of these is true:
  [1] controller flag      controller.diff.server.side=true
  [2] compare annotation   argocd.argoproj.io/compare-options: ServerSideDiff=true
  [3] sync option          ServerSideApply=true            <-- new in 3.6

        ->  SERVER-SIDE DIFF: a real Server-Side Apply dry-run
             against kube-apiserver. Results are cached; recomputed
             only on refresh, new revision, Application spec change,
             or a resource's resourceVersion change.

none of [1..3]  ->  client-side diff (unchanged behavior)

What this costs you at scale. The dry-run requests shift real work onto kube-apiserver, proportional to the number of resources in Applications that use Server-Side Apply. The upgrade guide is explicit that diffs are cached — recomputed only on refresh, a new revision, an Application spec change, or a resource's resourceVersion change — so this is load shaping, not load removal. Size your control-plane headroom accordingly on fleets with large SSA estates. The payoff is correctness you cannot fake locally: validating admission webhooks now participate in the diff stage, so a resource your policy engine rejects surfaces as a diff error before sync time. If your runbooks treat diff errors as pure spec drift, update them — a diff can now fail for policy reasons, which is exactly the feedback loop you want, once you know to expect it.

If you need the old behavior for a specific hot Application — say, a very large SSA app whose diff dry-runs you cannot yet afford — the opt-out is per-Application and takes one annotation:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  namespace: argocd
  annotations:
    # Explicit opt-out: this app keeps the legacy client-side diff
    # even though it syncs with Server-Side Apply.
    argocd.argoproj.io/compare-options: ServerSideDiff=false
spec:
  project: default
  source:
    repoURL: https://github.com/acme/platform-gitops.git
    targetRevision: main
    path: clusters/prod/payments-api
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
    retry:
      limit: 2
      backoff:
        duration: 15s
        factor: 2
        maxDuration: 3m

The full strategy matrix — including the CLI's argocd app diff --server-side-diff behavior and the runner wiring — is documented in the diffing guide; the upstream pull request also carries the maintainers' selection truth table, worth reading before you bet a migration on this.

Hidden costs: what the upgrade page puts below the fold

Argo CD 3.6 is a normal minor in the sense that its upgrade page carries behavioral changes that are easy to miss if you skim for the word "breaking". None of these are fatal; all of them will burn someone:

Change in 3.6 Who feels it What to do
Repo-server reads OpenTelemetry collector headers from ARGOCD_REPO_SERVER_OTLP_HEADERS; the old ARGOCD_REPO_OTLP_HEADERS is no longer read. Custom repo-server Deployments or Helm values that set the old variable. Rename it. The shipped manifests already use the new key, so default installs are unaffected.
otlp.insecure now unilaterally decides TLS: with "true" (the default) the connection is always plaintext — certificate env vars no longer upgrade it — and with "false" an unreadable CA or incomplete client cert pair fails the component at startup. Anyone with OTLP certificate environment variables set on any component. Set otlp.insecure: "false" explicitly and validate cert mounts before the upgrade window. See the observability docs.
Tracing sampler is now parent-based and configurable (--otlp-sample-ratio, default 1.0). An inbound W3C traceparent marked "not sampled" is now honored — such requests are not recorded, even at ratio 1.0. Out-of-range flag values fail at startup; env-var values silently fall back to 1.0 with a warning. Teams fronting argocd-server with trace-propagating proxies or gateways. Audit upstream trace injection; set the same ratio on every component to keep traces whole.
argocd app diff --local --server-side-generate: the implicit --local-include defaults now also cover *.tpl and Chart.lock; passing --local-include replaces the whole set rather than appending. CLI-driven local-diff workflows over Helm charts. Re-specify the full pattern list when you override; the new defaults fix silent _helpers.tpl breakage, so prefer keeping them.
ApplicationSets with progressive sync now refresh-annotate all owned Applications and wait for every one to reconcile before progressing to the next step. Large AppSet fleets — hundreds of apps per set. Expect slower step progression. Budget the delay into rollout timing; the correctness win is not being rated on stale data.
GitOps Promoter resources (PromotionStrategy, ChangeTransferPolicy) report Healthy instead of Progressing while a promotion is in flight. Alerting or automation keyed on Progressing for Promoter resources. Key automation on health messages or Promoter status fields instead.
Health transition events gain a (Caused by ...) suffix listing up to three responsible resources; health checks may return a deletionMessage and the default check now lists pending finalizers. Anything parsing Argo CD event messages programmatically. Update parsers to tolerate the suffix. The feature itself is a gift: flapping health finally points at the resource driving it.
gitops-engine: KubectlOptionsRunner.Create() ignores its third argument because upstream kubectl changed at v0.37.0; removal is scheduled for the next major. Downstream consumers of gitops-engine, not Argo CD operators. Pin your gitops-engine dependency and patch wrappers before the next major.

What you actually get for the upgrade

Strip away the changelog noise and 3.6 delivers four things worth the trouble:

Smaller knobs that should have existed years ago: Redis Sentinel support via environment variables, an optional Redis key prefix for shared instances, an OCI generator for ApplicationSets, and a tpl function in AppSet go-templating. The ApplicationSet docs cover the generator surface.

Upgrade vectors

Version mapping first, because the chart is the trap: chart 10.12.0 carried app v3.6.0 — the broken one. Every chart since 10.12.1 carries appVersion v3.6.1, verified against the chart's own Chart.yaml.

App version Helm chart State
v3.5.7 10.11.7 Last 3.5 stable; a safe fallback if 3.6 blockers below apply to you.
v3.6.0 10.12.0 Superseded within a day; blank UI under sub-path deployments. Skip.
v3.6.1 10.12.1 The stable baseline this article is about.
v3.6.1 10.14.0 Current chart — same appVersion. Pin by appVersion, not chart version. 
# Helm: every chart since 10.12.1 carries app v3.6.1 (10.14.0 is current).
# Substitute your own release name, namespace, and repo name.
helm repo update argo
helm upgrade argocd argo/argo-cd --version 10.12.1 --namespace argocd --reuse-values

# Raw manifests, verbatim from the v3.6.1 release notes:
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.6.1/manifests/install.yaml

Worth pausing on that --server-side flag in Argo CD's own install command: the project now needs Server-Side Apply to install itself, which tells you where its defaults are going. Supply-chain posture is also solid — images are signed with cosign and ship SLSA Level 3 provenance for both images and CLI binaries; the verification procedure is documented for signed release assets, and images are published on quay.io.

If you serve the UI under a sub-path — the exact configuration 3.6.0 broke — verify the rewrite immediately after any upgrade in this line, using the reproduction from the upstream report:

# The sub-path configuration, in argocd-cmd-params-cm:
#   server.insecure: "true"
#   server.rootpath: /tools/argocd

# After upgrading, confirm the base tag carries your prefix:
curl -s -H 'Accept: text/html' http://argocd.example.com/tools/argocd/applications \
  | grep -o '<base[^>]*>'

#   v3.6.0 (broken)   <base href=/>        - UI assets requested at site root, 404
#   v3.6.1 (fixed)    the base tag is rewritten to your prefix (/tools/argocd/)

On the commercial side: Argo CD is an open-source CNCF graduated project with no vendor to buy it from directly. The supported path is Akuity — "built by the creators of Argo CD and Kargo", per its own homepage — which sells an enterprise control plane around the project. That is a plain informational link; we have no affiliate relationship with them.

Who should wait

Upgrade now if you are on 3.5.x and intend to stay current — the SSD flip is the correctness upgrade this minor is remembered for, and sub-path UI deployments have no choice but to skip 3.6.0 anyway. Schedule deliberately if you run ApplicationSets with progressive sync across hundreds of apps (the new reconciliation barrier delays step progression), custom OpenTelemetry pipelines (the env rename plus sampler semantics need a config pass first), or Promoter automation keyed on Progressing states. Stay put on the November 2 patch line — v3.5.7, v3.4.13, v3.3.19 — if any of the above needs more than one sprint; those releases took the same shared fixes backported across three minor lines, which is exactly what a healthy maintenance model looks like.

Credits

The 3.6.1 patch is small, but the credit chain is instructive: augustomenin reported the sub-path regression with image-level forensics and shipped the fix; shimagrawal landed the sidebar state fix; and the argo-cd-cherry-pick-bot got both onto release-3.6 inside a single morning. The 3.6 line itself belongs to a long contributor list — pjiang-dev (the SSD flip), rumstead and blakepettersson (the refresh-path and marshalling performance series), ranakan19 (AppSet progressive-sync telemetry), ppapapetrou76 (rollback RBAC), crenshaw-dev (release hygiene, Zizmor CI enforcement, deletion health messages), jwinters01 (the UI build overhaul), and yugstar, who alone landed a double-digit share of the new health checks.

Verdict: v3.6.1 is not a feature release — it is the release that makes 3.6 shippable. Run it if you run 3.5.x and intend to stay current. Read the SSD section before you roll out, because your diffs quietly changed engines this minor: what Argo CD shows you as "drift" is now computed by the API server itself, admission webhooks included. And keep v3.5.7 in your back pocket if the AppSet delays or the OTel renames need more than one sprint. The only wrong answer on the board is v3.6.0.