Infrastructure Promotion with Pull Requests

Promotion is the mechanism that makes lower environments worth having. Without it, staging tests one thing and production runs another, and the testing bought nothing.

The pattern generalises across every tool in this pillar: something identifiable and immutable, proven somewhere cheap, moves toward production through a pull request somebody approves.

Before you start

• Kubernetes Environment Promotion — the Kubernetes-specific case
• Environment Repositories — the boundary promotion crosses

The shape

Promotion through a pull request

A vertical sequence: a validated change proven in a lower environment; an automatically opened promotion pull request; a rendered environment diff; approval by an accountable person; merge; and automation or reconciliation applying it.

The property that makes this work is that a promotion changes a reference rather than content. A one-line diff moving an image digest, a module version, a chart version or a commit SHA. That sounds trivial and is the whole point: a reviewer can see precisely what is moving, and the thing reaching production is byte-identical to what was tested.

What gets promoted

Domain	Reference
Containers	An image digest
Helm	A chart version, ideally a digest
Kubernetes manifests	A commit SHA or an OCI artifact digest
Terraform modules	A version tag
Ansible roles and collections	A version
Configuration	A commit in a shared path

The common property: an immutable reference to something that already exists and has been exercised somewhere. A promotion producing something new is not a promotion.

The exception worth naming: a Terraform plan is not promotable. It is computed against state and the real world at a moment and is stale immediately. For Terraform, what promotes is a module version or a configuration commit, and the plan is recomputed per environment. That is a genuine limit of the artifact analogy and it is why Terraform environments treats promotion differently.

Automating the pull request

The pull request should be proposed automatically and merged deliberately.

name: Promote

on:
  workflow_dispatch:
    inputs:
      from:
        description: Source environment
        required: true
        type: choice
        options: [development, staging]
      to:
        description: Target environment
        required: true
        type: choice
        options: [staging, production]

permissions:
  contents: write
  pull-requests: write

jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7

      - name: Read the source version
        id: source
        run: |
          digest="$(yq '.images[0].digest' \
            "applications/api/overlays/${{ inputs.from }}/kustomization.yaml")"
          version="$(yq '.images[0].newTag // "unversioned"' \
            "applications/api/overlays/${{ inputs.from }}/kustomization.yaml")"
          echo "digest=$digest" >> "$GITHUB_OUTPUT"
          echo "version=$version" >> "$GITHUB_OUTPUT"

      - name: Apply it to the target
        run: |
          cd "applications/api/overlays/${{ inputs.to }}"
          kustomize edit set image "api@${{ steps.source.outputs.digest }}"

      - uses: peter-evans/create-pull-request@v8
        with:
          branch: promote/api-${{ inputs.to }}
          title: "Promote api to ${{ inputs.to }}: ${{ steps.source.outputs.version }}"
          body: |
            Promoting the version currently running in `${{ inputs.from }}`.

            **Version:** ${{ steps.source.outputs.version }}
            **Digest:** `${{ steps.source.outputs.digest }}`

            This digest has been running in `${{ inputs.from }}` and is byte-identical
            to what was tested there. No rebuild.
          labels: promotion

Reads the source environment’s current reference rather than the latest build. That distinction is the whole difference between promotion and deployment: you are moving what has been running and observed, not what was built most recently. A workflow that promotes the latest build has skipped the step that made the lower environment worthwhile.

One target per pull request. Not one updating every environment, which removes the intermediate step entirely.

A description containing the evidence. Version, digest, and where it has been running.

Labelled, so the population is visible and countable later — how many promotions happened, how many were reverted, and how long they typically waited for review.

Making them worth reading

The failure mode of automated promotion pull requests is that they all look identical, so nobody reads them.

After the twentieth one-line digest bump, a reviewer approves on autopilot — including the one whose source environment has been broken for a day.

Put the evidence in the description, not just the reference:

What changed since the deployed version. A commit range link. This is the single most useful addition and it is one API call.

How long it has been running in the source environment. “Running in staging for 26 hours” tells a reviewer whether it has soaked.

Whether the source environment is healthy. Promoting from an environment that is currently degraded is the specific mistake this catches.

What the checks said. A link to the run.

Anything unusual. A migration, a configuration change alongside the version bump, a first deployment after a long gap.

Caution

A promotion pull request that looks exactly like the last twenty is one a reviewer skims. Uniformity is useful for spotting the unusual, and it only works if the description carries enough information that an unusual one looks unusual. A body containing only a digest gives a reviewer nothing to notice.

Gating

Development promotes automatically on a successful build. Fast feedback is the point, and a development environment behind a manual gate is a development environment nobody uses.

Staging can promote automatically once development has soaked, if the checks are trustworthy.

Production requires a human, through an environment protection rule with required reviewers. Not because automation is untrustworthy — because somebody should be accountable and should have looked.

Soak requirements should be checks, not conventions. A promotion pull request that cannot merge until a check confirms the version has been running in the previous environment for N hours is a control. “We usually wait” is not, and it is the first thing dropped under pressure.

Health checks as merge gates. A check querying whether the source environment is currently healthy prevents promoting from a broken one — which is a specific and recurring mistake, because a promotion pull request opened yesterday says nothing about the state of the source environment today. The check is a few lines against your monitoring or your GitOps controller’s status, and it catches the case where somebody approves a promotion from an environment that started failing overnight.

Emergency promotion should be possible and visible. A path straight to production for a genuine emergency, leaving a record — a labelled pull request, a workflow dispatch with a reason field, an approval from somebody with the authority. Making it impossible does not prevent emergencies; it means people bypass the whole mechanism by editing manifests directly or applying by hand, which produces a change with no record at all. A visible exception is better than an invisible workaround.

The audit trail

What this produces, as a by-product.

Every environment’s history is git log on its directory. What version, when, who merged it.

Every promotion has an approver, recorded by the environment protection rule.

Every version traces to a build, which traces to a commit, which traces to a pull request and a review.

The gap between environments is visible — compare the references and you know which is behind.

The question this answers that nothing else does: “when did production start running this, and who decided?” That is a question asked after every incident and it is expensive to answer without this structure.

What it does not capture: anything that changed outside the process. A console edit, a script somebody ran, an operator writing a field. The promotion history records intent faithfully and says nothing about whether reality matches it — that is drift detection, and a team relying on promotion history alone as an account of its infrastructure will be wrong in exactly the cases that matter most.

Rollback, and its limits

The reason to be precise about what promotion is.

Reverting the promotion commit restores the previous reference. For a stateless workload with no schema change, that is a complete rollback: fast, auditable, and it works.

It is not a complete rollback when:

A migration ran. Old code against a migrated schema. The application may fail to start, or start and misbehave.

A resource was deleted. Recreated with the same name, without its data.

An external system was called. A webhook fired, a message published, a third party notified. Git has no undo.

Data was written in a new format. Old code reading it is a failure mode frequently worse than the original problem.

Danger

Reverting a Git commit does not guarantee infrastructure has been restored safely. It restores a declaration. Whether the system returns to its previous behaviour depends on whether the operations in between were reversible, and stateful ones are not. A runbook saying “roll back by reverting the promotion” should state which classes of change that covers and what to do for the others.

The discipline that makes it true more often: backward-compatible changes. A migration adding a nullable column, deployed before the code using it, is revertible. One dropping a column is not. Expand-and-contract makes each step individually revertible, and it is what turns “we can roll back” from a hope into a property.

Ask before merging, not after. “Is this revertible?” is a question for the promotion pull request, and the answer belongs in its description.

Promotion across tools

The pattern with each tool’s specifics.

Containers to Kubernetes. A digest in an overlay or a values file. The cleanest case: genuinely immutable, genuinely identical.

Helm charts. A chart version, or a digest for real immutability. Note that the chart and the application version move independently.

Terraform modules. A version tag in a source string. The module code is identical across environments; the plan is not, because state and existing resources differ. Read the plan per environment.

Ansible. A collection or role version in requirements.yml, plus running the playbook against the next inventory. Ansible has no artifact to promote beyond the version, and the promotion is as much about running it somewhere new as about changing a reference.

Raw configuration. A commit SHA where environments read from a shared path, or the change applied to each environment’s copy in turn. The second is more manual and keeps the differences visible, which is usually worth the tedium.

The common shape holds: something identified, proven somewhere, moved by a one-line change, reviewed, merged, applied.

Rendering the environment diff

The strongest addition to a promotion pull request, and it works for every tool in this pillar.

The problem: the source diff is one line. The effect on the target environment is not, and only the effect matters.

For Kubernetes: render the target overlay before and after and post the difference. A digest change usually produces exactly what you expect; when it does not — because the new version’s chart changed a resource, or a value now resolves differently — that is what a reviewer needs to see.

For Terraform: plan the target environment on the promotion pull request. The module version changed; the plan says what that means for this environment’s existing resources, which is different from what it meant for development’s.

For Helm: helm template with the target’s values, before and after.

Post it as a sticky comment, counts first, destroys called out, detail collapsed. The same treatment as Terraform plan surfacing, and the same reason: evidence that is inconvenient to read does not get read.

The value is highest exactly where promotion is riskiest. A digest bump on a stateless service produces a boring diff, which is reassuring. A module version bump that destroys and recreates a resource produces an alarming one, which is the point.

Where the tool cannot render the effect — Ansible, largely — the substitute is --check --diff against the target environment, which is weaker and better than nothing.

Promotion frequency

A parameter teams set implicitly and should set deliberately.

Frequent, small promotions are easier to review, easier to attribute when something breaks, and easier to revert. The change between environments is one thing.

Infrequent, large promotions batch many changes together. The review is harder, attribution is impossible, and a revert undoes work that was fine along with the work that was not.

The forces pushing toward infrequent: approval friction, soak requirements, a release process that feels heavyweight, and a team that finds promotions stressful.

Those forces are self-reinforcing. Stressful promotions get batched; batched promotions are riskier; riskier promotions are more stressful. Teams end up promoting monthly and finding it terrifying, having arrived there one reasonable decision at a time.

Breaking the cycle means making promotion boring rather than making it rarer: better evidence in the pull request, automated soak checks, a rendered diff, and a stated recovery path. Each reduces the judgement required at the moment of highest pressure.

The measure worth watching is how many changes are in a typical promotion. If it is more than a handful, the frequency is too low, and the reason is usually friction rather than policy.

Common mistakes

Rebuilding per environment. The artifact tested is not the artifact shipped.

Promoting the latest build rather than the running version. That is deployment, not promotion.

One pull request updating every environment. Removes the intermediate steps.

Auto-merging to production. Nobody accountable, nobody looked.

A description containing only a digest. Nothing for a reviewer to notice.

Soak time as a convention. First thing dropped under pressure.

Promoting from a degraded environment. A health check as a merge gate catches this.

Treating revert as guaranteed rollback. It restores a declaration.

No stated recovery path for irreversible changes. Discovered at 3am.

Making emergency promotion impossible. People bypass the mechanism entirely.

Coupled promotions

The case that breaks the one-line model: a change requiring two things to move together.

An application and its configuration. New code needs a new environment variable. Deploy the code first and it fails on the missing variable; deploy the configuration first and old code ignores it. Either order works if both are backward-compatible, and only one order works if neither is.

The general answer is compatibility. New code tolerates the variable being absent; new configuration is harmless to old code. Then order does not matter and each step is independently revertible. This is worth engineering for, because it converts a coordination problem into two ordinary promotions.

Where compatibility is genuinely impossible, the promotion is one pull request changing both — a slightly larger diff, applied atomically, with no window.

Across services it is harder. Service A’s new version requires Service B’s new API. No promotion mechanism solves this; the answer is versioning the interface, supporting both for a period, and promoting the provider before the consumer.

Across tools it is harder still. A Terraform change creating a queue and a Kubernetes change consuming it are two repositories or two directories with two pipelines. State the dependency in both pull requests, and promote the provider first.

The rule worth adopting: if a change cannot be promoted independently, say so in the pull request and name what it depends on. A promotion silently requiring another one is how an environment ends up in a state nobody can reproduce, and the person who discovers it is usually not the person who caused it.

Tracking what is where

A promotion model is only usable if you can answer where each version currently is.

The repository is the record. Each environment’s reference is in a committed file, and git log on that file is that environment’s deployment history.

Build a one-command answer. A script printing every environment’s current reference and version comment takes ten minutes and answers the question people ask most often.

Watch the gap. Production three versions behind staging means either nobody is promoting or nobody is confident. Both are worth knowing, and neither is visible without looking.

Label the running workload with its version so the cluster carries the answer too. That is what somebody without a repository checkout can query.

The two sources should agree. The repository says what should be running; the platform says what is. A disagreement is either a promotion in flight or drift, and telling them apart is the drift detection question.

When a promotion fails

The version reached the environment and it is not working.

The platform reports it first, if the monitoring is right — a degraded workload, a failed apply, an error rate. That should be the signal rather than a user report.

Revert or fix forward, decided quickly rather than debated.

Revert when the previous version was known good, the change is reversible, and the cause is not obvious. Restore service, diagnose afterwards.

Fix forward when the fault is understood and small, the previous version has a problem of its own, or reverting is not actually safe.

What makes this fast is having decided in advance. A pull request whose description says “revertible: yes, no schema change” removes the debate. One saying “not revertible after the migration; recovery is restoring from the pre-deploy snapshot” tells you what to do instead.

Do not promote past a failure. A version that failed in staging must not reach production because somebody is waiting. This sounds obvious and is the decision that gets made under pressure with a reason that seems sound at the time.

Record what happened. A failed promotion is the most informative event the pipeline produces, and the finding is usually about the checks rather than about the change — something reached staging that should have been caught earlier.

Then ask why it was not caught. That question is worth more than the incident report, and it is the one that improves the pipeline rather than the runbook.

Mental model

Promotion moves an approved, immutable reference toward production through a controlled workflow. It never creates something new — if the artifact is different, nothing was promoted.

The corollary about rollback follows directly: Git restores what was declared, quickly and auditably. Whether that restores the system is a question about the change, not about the tooling.

What you learned

• Promotion moves an immutable reference; producing something new is not promotion
• A promotion diff should be one line in one file, identical in shape every time
• Terraform plans cannot be promoted — they are computed per environment against state
• Automate the pull request; keep the merge a decision, with evidence in the description
• Uniformity is a detection mechanism only if the description carries enough to notice an anomaly
• Soak requirements and source-environment health should be merge checks, not conventions
• Reverting a promotion restores a declaration; migrations, deletions and external calls do not reverse
• Expand-and-contract is what makes “we can roll back” a true statement

Exercise

Use a disposable repository with two environment directories.

1. Set up development and production overlays, each referencing an image digest.

2. Write a promotion workflow reading development’s digest and opening a pull request against production.

3. Run it. Predict: how many lines does the diff contain?

4. Extend the description to include the commit range since production’s current version. Compare how much easier the pull request is to review.

5. Add a check that fails if the source environment’s version has been deployed for less than an hour. Try to promote immediately. Predict: does it block?

6. Add an environment protection rule requiring your approval on production. Merge the promotion and watch it wait.

7. Revert the promotion commit. Predict: what restores, and what would not have if the change had included a migration?

8. Delete the repository.

Related lessons

Kubernetes Environment PromotionThe Kubernetes-specific mechanics.

Environment RepositoriesThe boundary a promotion crosses.

Infrastructure Drift DetectionWhat changes outside the promotion path.

Terraform Modules with GitPromoting a module version.