Kubernetes Environment Promotion with Git

If staging and production build their own images from the same branch, staging tested nothing.

That sentence is the whole lesson. Promotion is the discipline of moving one identified artifact toward production rather than independently producing something similar for each environment — and getting it right is what makes lower environments worth having.

Before you start

• Kubernetes Repository Structure — where environment configuration lives
• Container Image Tagging with Git — digests versus tags
• GitOps Explained — what the controller does with the result

Build once, promote

One artifact, promoted

A vertical sequence: a Git commit; one build producing an immutable image digest; deployment to development; promotion to staging by reference; promotion to production by reference.

One build. The artifact production runs is byte-identical to the one development tested.

Promotion changes a reference, not content. A one-line diff moving a digest from one environment’s configuration to the next.

Configuration differs per environment; the artifact does not. Replica counts, resource limits, connection strings and feature flags are environment values. The image is not, and an image built with an environment baked into it is an image that cannot be promoted at all — which is the deeper reason twelve-factor configuration matters here.

The anti-pattern is a pipeline that builds on merge to main for staging and builds again on a tag for production. Two builds, two digests, and any difference between them — a base image that moved, a dependency that resolved differently — is a difference nobody tested.

What gets promoted

Not just images.

Artifact	Promoted as
Container image	An image digest
Helm chart	A chart version, ideally a digest
Kubernetes manifests	A commit SHA or an OCI artifact digest
Terraform module	A version tag
Ansible collection	A version

The common property: an immutable reference to something that already exists and has been exercised somewhere. A promotion that produces something new is not a promotion.

The mechanics

With a config repository and environment overlays, promotion is a pull request changing one line.

apps/api/overlays/production/kustomization.yaml

images:
  - name: api
    newName: ghcr.io/example-org/api
    # v2.4.0
    digest: sha256:1111111111111111111111111111111111111111111111111111111111111111
    # v2.4.1
    digest: sha256:2222222222222222222222222222222222222222222222222222222222222222

One line, in one file, per environment. That is the property to design for. A promotion diff that touches several files is one a reviewer cannot check at a glance, and a diff that looks different every time is one where nobody notices when it is genuinely different.

The version comment is doing real work. A digest tells a reviewer nothing about what they are approving; # v2.4.1 tells them what to look up.

Kustomize: the images field, edited with kustomize edit set image.

Helm: a value, ideally image.digest rather than image.tag.

Plain manifests: the image field directly, which is why plain manifests get awkward with several environments.

Automating the pull request

Promotion should be proposed automatically and merged deliberately.

On a successful build, a workflow opens a pull request against the next environment’s overlay with the new digest. The pull request is the gate; the automation removes the tedium, not the decision.

Include what a reviewer needs in the description: the version, the digest, a link to the commit range since the currently deployed version, and the test results.

One pull request per environment. Not a single one updating all three, which removes the staging step entirely.

Do not auto-merge to production. Development, defensibly. Staging, sometimes. Production is where a human decides, and the environment protection mechanisms exist for exactly this.

Caution

The failure mode of automated promotion pull requests is that they all look identical, so nobody reads them. After the twentieth one-line digest bump, a reviewer approves on autopilot — including the one where the digest is from a build that failed its integration tests. The counter is putting the evidence in the description: what changed since the deployed version, and what passed. A promotion pull request should be quick to approve because the evidence is there, not because it looks like the last one.

Soak time

The part teams cut first and should not.

A promotion that spends four minutes in staging tested startup. Not memory growth, not connection pool exhaustion, not the behaviour under a daily traffic pattern, not the migration running against realistic data volumes.

Define what soak means per environment, and make it a rule rather than a habit. “Staging for at least one business day” is a policy somebody can point at when a release is being rushed.

Automate the waiting, not the judgement. A promotion pull request that cannot be merged until a check confirms the version has been running in the previous environment for N hours is a control. A convention that people should wait is not.

What to watch during soak: error rates, latency percentiles, memory trends, and anything the change specifically touched. If nobody is watching, the soak is a delay rather than a test.

Ordering and gates

build → dev (auto) → staging (auto PR, auto merge) → production (auto PR, human merge)

Development promotes automatically. The point is fast feedback.

Staging can promote automatically once development has soaked, if you trust your checks.

Production requires a human. Not because the automation is untrustworthy, but because somebody should be accountable for the decision and should have looked at what is changing.

Environment protection rules hold the actual gate — required reviewers, wait timers, branch restrictions. Covered in Pillar 4.

Skipping an environment should be possible and visible. Emergencies happen. A promotion straight to production should be a deliberate action that leaves a record, not something impossible that people work around by editing the manifest directly.

Rollback, precisely

The most important section, because the common claim is too simple.

The claim: with GitOps, rollback is git revert.

Where it is true. A stateless workload, an image digest change, no migration. Revert the promotion commit, the controller reconciles, the previous version runs. This is genuinely fast and genuinely auditable, and it covers a large fraction of changes.

Where it is not:

A database migration ran. Reverting the manifest gives old code against a migrated schema. The application may fail to start, or — worse — start and behave incorrectly.

A resource was deleted. Reverting recreates a resource with the same name. The data in the deleted volume is gone.

An external call was made. A webhook fired, a message was published, a payment was taken. Git has no record and no undo.

A CRD was removed. Every custom resource of that kind was deleted cluster-wide. Restoring the CRD restores the type, not the objects.

Data was written in the new format. Old code reading new data is a failure mode that is often worse than the original problem.

Danger

Reverting a Git commit does not guarantee infrastructure has been restored safely. It restores the declaration. Whether the system returns to its previous behaviour depends on whether the operations in between were reversible, and stateful ones are not. Every runbook that says “roll back by reverting the commit” should state which classes of change that actually covers — and what to do for the others.

The practical discipline: for any change, ask before merging whether reverting the commit is sufficient. If not, the pull request should say what the actual recovery procedure is. That sentence takes thirty seconds to write and is worth a great deal at 3am.

Backward-compatible changes are the real answer. A migration that adds a nullable column, deployed before the code that uses it, is revertible. A migration that drops a column is not. Expand-and-contract — expand the schema, deploy, migrate data, deploy code, contract later — makes each step individually revertible, and it is the discipline that makes “roll back” a true statement rather than a hope.

Rollback mechanics

When revert is the right tool:

git revert the promotion commit, not the application commit. The promotion commit is the one that changed what production runs.

Do not force-push or reset. A revert is a new commit that records both the change and the undo, which is what you want in the history of an environment.

Confirm the controller reconciled. Reverting the commit does not deploy anything; the controller does, on its interval. Watch it happen rather than assuming.

Know the previous digest before you need it. It is in the environment’s Git history, which is one of the reasons the digest belongs in a committed manifest rather than being resolved at deploy time.

Consider suspending reconciliation first if you need the cluster to stop changing while you diagnose. flux suspend or setting an Argo CD Application’s sync policy to none. Then resume deliberately.

When a promotion fails

The version reached the environment and it is not working. What the model gives you.

The controller will report it. A Deployment that cannot become ready shows as degraded, and that is the signal to act on rather than a user report.

The decision is revert or fix forward, and it should be made quickly rather than debated.

Revert when the previous version was known good, the change is reversible, and the cause is not obvious. Restoring service first and diagnosing afterwards is almost always right.

Fix forward when the fault is understood and small, the previous version has a problem of its own, or reverting is not actually safe — the migration case.

The thing that makes this fast is having decided in advance. A pull request whose description says “revertible: yes, no schema change” removes the debate. One that says “not revertible after the migration; recovery is restoring from the pre-deploy snapshot” tells you what to do instead.

Do not promote past a failure. A version that failed in staging must not be promoted to production because somebody is waiting for it. This sounds obvious and is the specific decision that gets made under pressure, usually with a reason that seems sound at the time.

Record what happened. A failed promotion is the most informative event your delivery pipeline produces, and the finding is usually about the checks rather than about the change — something reached staging that should have been caught earlier.

Progressive delivery

Beyond all-or-nothing promotion.

Canary and blue-green shift traffic gradually and roll back automatically on a metric threshold. Tools exist that integrate with both GitOps controllers, driving the rollout while the desired state stays declarative.

Where it helps genuinely: high-traffic services where a bad version’s impact scales with exposure, and where you have metrics good enough to make an automated judgement. Both conditions matter — a canary gated on a metric that is noisy at low traffic will roll back good releases and promote bad ones, which is worse than not having it.

Where it is overhead: low-traffic internal services, anything where a rollout takes longer than the time to notice and revert manually, and anywhere the metrics are not trustworthy enough to gate on.

It does not fix the stateful problem. A canary that rolls back after a migration ran leaves you exactly where the section above described. Progressive delivery reduces exposure to a bad version; it does not make irreversible operations reversible.

Promoting configuration, not just images

An image digest is the clearest case. Configuration changes need promoting too, and the model is less obvious.

A configuration change should travel the same path. A new environment variable, a changed resource limit, a feature flag — each should be exercised in development before production, for the same reason a code change should.

The mechanism is a commit, not a digest. With environment overlays, the same change is made in each overlay in turn. That is more manual than a digest bump and the discipline is identical: change dev, observe, change staging, observe, change production.

The temptation to skip is stronger, because a configuration change looks trivial. A resource limit reduced by a third looks like a one-word edit and is a change that will manifest under load, in production, three days later.

Where a change belongs in the base rather than the overlays — because it genuinely applies everywhere — promotion is harder to express, since editing the base changes all environments at once. Two workable answers: make the change in the base but behind a value each overlay sets, promoting the value; or accept that base changes are higher-risk and require correspondingly more review.

Configuration drift between environments is the cost of doing this badly. Every change applied to production and not backported to staging widens the gap, and eventually staging stops predicting anything. A periodic diff of the environment overlays — with every difference either explained or removed — is a cheap way to notice.

Coupled changes

The case that breaks the simple model: a change requiring an application version and a configuration change together.

The ordering problem. New code requires a new environment variable. Deploy the code first and it fails on a missing variable; deploy the configuration first and the old code ignores it. Either order is fine, and only one of them is fine in the wrong direction.

The general solution is backward compatibility. New code tolerates the variable being absent, and new configuration is harmless to old code. Then order does not matter and each step is independently revertible.

Where compatibility is genuinely impossible, the promotion is a single commit changing both — the digest and the configuration in one pull request, applied atomically by the controller. Slightly larger diff, no ordering window.

Across services it is harder. Service A’s new version requires Service B’s new API. That is a coordination problem no promotion mechanism solves, and the answer is the same as it has always been: version the interface, support both for a period, and promote the provider before the consumer.

The rule worth adopting: if a change cannot be promoted independently, say so in the pull request and name what it depends on. A promotion that silently requires another one is how a staging environment ends up in a state nobody can reproduce.

Common mistakes

Building per environment. The artifact tested is not the artifact shipped.

Promoting a tag rather than a digest. The tag can have moved in between.

One pull request updating all environments. Removes the staging step.

Auto-merging to production. Nobody accountable, nobody looked.

No soak time. A four-minute staging deployment tested startup.

Identical promotion pull requests with no evidence. Approved on autopilot.

Treating git revert as guaranteed rollback. It restores a declaration.

No stated recovery path for irreversible changes. Discovered at 3am.

Reverting the application commit rather than the promotion commit. Does not change what is deployed.

Making emergency promotion impossible. People edit manifests directly instead.

Knowing what is where

A promotion model is only usable if you can answer where each version currently is.

The repository is the record. Each environment’s overlay contains the digest it should be running, and git log on that file is the deployment history for that environment. This is one of the strongest arguments for the digest living in a committed manifest.

A one-command answer is worth building. A script that prints the digest and version comment from each environment’s overlay takes ten minutes to write and answers the question people ask most often:

for env in dev staging production; do
  printf '%-12s' "$env"
  grep -A1 'digest:' "apps/api/overlays/$env/kustomization.yaml" | head -2 | tr -d '\n'
  echo
done

The controller is the other half. The repository says what should be running; the controller says what is. If they disagree, that is either a promotion in flight or drift.

The gap between environments is a number worth watching. Production three versions behind staging means either a promotion process nobody is running or a set of changes nobody is confident about. Both are worth knowing.

Label the running workload with its version. app.kubernetes.io/version in the manifest means the cluster itself carries the answer, and kubectl get deploy -L app.kubernetes.io/version answers it without a repository checkout — useful when the person asking does not have one.

Multiple clusters per environment

Production is frequently more than one cluster, and promotion has to account for it.

Regions or availability zones. Several production clusters running the same workload. Promoting to “production” means promoting to all of them, and whether that happens simultaneously is a decision.

Simultaneous is simplest and riskiest. One commit, every production cluster reconciles, a bad version reaches everything at once.

Staged is safer and more work. One region first, soak, then the rest. Expressed as separate overlays or as clusters pinned to different revisions — the targetRevision mechanism in Argo CD, or a tag reference in a Flux GitRepository.

The trade-off is real. Staged rollout means the fleet is temporarily inconsistent, which matters for anything with shared state or a shared schema. Simultaneous means consistency and no blast-radius containment.

A reasonable default for a multi-region production: one region promotes first and soaks for a defined period, then the rest follow in one step. It contains the blast radius of an obviously bad version without leaving the fleet split for long.

Do not let clusters drift into different versions permanently. A fleet where three clusters are on different versions because promotions were abandoned halfway is a fleet nobody can reason about, and the version-reporting script above is how you notice.

Making promotion boring

The goal, and the properties that get you there.

A promotion should take one minute of a reviewer’s attention. If it takes longer, something is missing from the description or the diff is too large.

It should look the same every time, so that an unusual one is visible. Uniformity is a detection mechanism, not an aesthetic preference.

It should be obvious what is being promoted and from where. Version, digest, and the environment it has already run in.

It should be obvious what would happen if it is wrong. Revertible, or not, with the recovery path if not.

It should not require anybody to remember anything. The soak requirement is a check; the approval is an environment rule; the evidence is in the description.

Nobody should be able to bypass it quietly. Direct write access to the environment overlays removed, branch protection on the config repository, and the controller as the only thing applying to the cluster.

Get those six right and promotion stops being an event. That is the objective — a team that finds releases stressful usually has a promotion process that requires judgement and memory at the moment of highest pressure, and the fix is moving both earlier.

Mental model

Promotion moves an approved, immutable reference toward production through a controlled workflow. It never creates something new — if the artifact is different, nothing was promoted.

The corollary is the rollback rule: Git restores what was declared, quickly and auditably. Whether that restores the system depends on what the change did in the world, and that is a question about the change rather than about the tooling.

What you learned

• Build once; promote an immutable reference, never rebuild per environment
• A promotion diff should be one line in one file, with the version in a comment
• Automate the promotion pull request; keep the merge a decision, with evidence in the description
• Soak time must be a rule with a check behind it, or it is a habit that gets cut
• Production promotion requires a human and an environment protection rule
• git revert restores a declaration; migrations, deletions and external calls do not reverse
• Expand-and-contract migrations are what make “roll back” true rather than hopeful
• Revert the promotion commit, not the application commit

Exercise

Use a disposable local cluster and repository with a GitOps controller.

1. Set up dev and production overlays for one application, each with an image digest.

2. Build an image, note its digest, and promote it to dev with a one-line change. Watch the controller apply it.

3. Promote the same digest to production. Predict: is the diff identical in shape to the first?

4. Change the dev overlay to use a tag instead of a digest. Push a new image to that tag without changing the manifest. Predict: does anything deploy?

5. Revert the production promotion commit. Predict: how long until the controller reverts it, and what did you have to do?

6. Add a Job to the application that writes a file to a volume — a stand-in for a migration. Promote, let it run, then revert. Predict: does reverting undo the file?

7. Write the recovery procedure for step 6 as it would appear in a pull request description.

8. Delete the cluster.

Related lessons

GitOps Drift and ReconciliationWhat happens between promotions.

Infrastructure Promotion with Pull RequestsThe same pattern across all tools.

Container Image Tagging with GitWhy the digest is what travels.

EnvironmentsApproval gates and wait timers.