Terraform CI established the rule: pull requests plan, they do not apply. This page is the other half — applying after merge, with the controls that make automated infrastructure changes defensible.
It also has an unusual amount to say about rollback, because Terraform is the place where the software industry’s normal rollback story quietly stops being true.
The apply workflow
Section titled “The apply workflow”on: push: branches: [main] paths: ['infra/**']
permissions: contents: read
concurrency: group: terraform-apply-production cancel-in-progress: false
jobs: apply: runs-on: ubuntu-latest environment: production permissions: contents: read id-token: write defaults: run: working-directory: ./infra
steps: - uses: actions/checkout@v7
- uses: hashicorp/setup-terraform@v4 with: terraform_version: "1.13.3"
- uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: arn:aws:iam::ACCOUNT_ID:role/terraform-apply-production aws-region: eu-west-1
- run: terraform init -input=false
- name: Plan run: terraform plan -input=false -no-color -out=tfplan
- name: Apply run: terraform apply -input=false -auto-approve tfplanFour details carry the safety of this file.
environment: production makes the job wait for whatever protection rules that environment has —
required reviewers, a wait timer, a branch restriction. Without it, -auto-approve means exactly what
it says and the merge is the only gate.
concurrency with cancel-in-progress: false queues concurrent applies rather than cancelling
one. Cancelling an apply mid-flight is the worst outcome available: Terraform may have created
resources it did not get to record in state, and the next run will try to create them again.
A separate apply role. The plan role from CI is read-only; this one can write. Keeping them distinct is what makes “pull requests cannot change infrastructure” a control rather than a promise.
terraform apply tfplan applies a saved plan, not the configuration.
Why applying a saved plan matters
Section titled “Why applying a saved plan matters”- run: terraform plan -input=false -out=tfplan- run: terraform apply -input=false -auto-approve tfplanWhat it doesApplies exactly the set of changes computed by the preceding plan, refusing if the state has moved since.
Why we run it`terraform apply` with no plan file re-plans first. Between the plan a human approved and the apply, the world can change — someone edits a resource by hand, another pipeline runs — and the apply then makes changes nobody reviewed.
Expected resultAn apply that either performs precisely the reviewed changes or fails with a state mismatch.
-auto-approve is safe here because a plan file is supplied: the approval it skips is the
interactive prompt confirming a plan that has already been fixed and reviewed. terraform apply -auto-approve with no plan file is a different and much less careful command.
State locking and what breaks it
Section titled “State locking and what breaks it”Terraform locks state during an apply so two runs cannot write simultaneously. Two failure modes appear in CI specifically.
A stale lock is left behind when a job is cancelled or the runner dies. The next run fails with a
lock error naming the holder. terraform force-unlock LOCK_ID clears it — after confirming no apply
is genuinely still running. Doing it reflexively while another apply is in flight is how state gets
corrupted.
Concurrent applies from different workflows on the same state will serialise on the lock, but
they will also produce confusing plans as each sees the other’s changes. The concurrency group
above prevents the GitHub-side case; if humans also run applies locally, agree on one path.
What rollback actually means
Section titled “What rollback actually means”This is the section that matters.
For an application, rollback means running the previous artifact, and the previous artifact still exists. For infrastructure, “rollback” means running a new apply that attempts to move the world back toward a previous description. Those are not the same operation, and treating them as the same is how a small incident becomes a large one.
Reverting the commit and applying gives you back the old configuration. Whether it gives back the old infrastructure depends entirely on what changed:
| Change | Re-applying the old config | Why |
|---|---|---|
| Instance count 3 → 5 | Recovers | Scaling down is symmetric |
| Security group rule added | Recovers | Removing a rule is symmetric |
| Tag or label changed | Recovers | Metadata is symmetric |
| Resource destroyed | Does not recover the data | A new resource is created, empty |
| Resource replaced | Creates another new one | New identifier; anything referencing the old one is still broken |
| Managed database deleted | Does not recover | Restore from backup, if one exists |
| Static IP released | May not recover | The address may already belong to someone else |
| DNS zone deleted | Does not recover cleanly | New nameservers; delegation must be updated |
| Certificate deleted | Recovers slowly | Reissue and revalidate |
The rows that recover share the property that the change was reversible in place. The rows that do not are the ones where the first apply destroyed something, and destruction is not undone by creation.
Practical consequences for the pipeline:
- Surface the destroy count in the pull request and in the apply job summary. It is the single most useful number Terraform produces.
- Use
prevent_destroyon stateful resources. It makes Terraform refuse to plan a destroy at all, converting a 3am incident into a build failure at review time. - Require a second reviewer for any change whose plan destroys something. GitHub environments
cannot conditionally require reviewers, so this is usually a
CODEOWNERSrule on the directories holding stateful resources. - Never automate
terraform destroyagainst a real environment. Ephemeral preview environments are the one reasonable exception, and they should live in a separate account or project.
Drift detection
Section titled “Drift detection”Infrastructure changes outside Terraform — a console edit during an incident, a resource created by another team. A scheduled plan makes that visible:
on: schedule: - cron: '0 6 * * 1-5'
jobs: drift: runs-on: ubuntu-latest permissions: contents: read id-token: write issues: write steps: - uses: actions/checkout@v7 - uses: hashicorp/setup-terraform@v4 with: terraform_version: "1.13.3" - uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: arn:aws:iam::ACCOUNT_ID:role/terraform-plan-readonly aws-region: eu-west-1 - run: terraform init -input=false - name: Detect drift run: terraform plan -input=false -no-color -detailed-exitcode-detailed-exitcode changes the exit codes to 0 for no changes, 2 for changes present, and 1
for an error. That makes “the world no longer matches the code” a distinguishable outcome that can
open an issue, rather than something indistinguishable from success.
Use the read-only role for drift detection. A scheduled job with apply permissions is a standing capability to change production on a timer.
Note that scheduled workflows are disabled automatically in repositories with no activity for 60
days, and that cron runs in UTC on a best-effort basis — a busy period can delay a run by many
minutes. Neither matters much for drift detection; both matter if you build something time-critical
on the same mechanism.
Exercise
Section titled “Exercise”-
Create a
productionenvironment with yourself as a required reviewer, and a separate apply role whose trust policy names that environment. -
Write the apply workflow above for a small, disposable stack. Merge a change that only adds a tag. Confirm the run waits for approval.
-
Read the plan output in the job summary and find the
Plan:line. Note the destroy count is 0. -
Now make a change that forces replacement of a resource — renaming something that is part of its identity. Confirm the destroy count is non-zero before approving, then decline the deployment.
-
Add
prevent_destroy = trueto a stateful resource and try to plan its removal. Confirm Terraform refuses at plan time. -
Add the scheduled drift workflow. Change something by hand in the console and confirm the next run exits with code 2.