The previous lessons covered the parts. This one assembles them, and the assembly has properties none of the parts have on their own.
The design goal is specific: a pipeline where the dangerous operation is structurally unable to run on untrusted input, and where every apply traces back to a merged commit and a named approver. That is achievable with ordinary GitHub features, and it is mostly a matter of getting the boundaries in the right places.
The shape
Section titled “The shape”A vertical sequence: a pull request; static checks needing no credentials; a plan using a read-only role; review of the posted plan; merge to the default branch; an apply job gated by an environment approval using a write role; and a recorded audit trail.
Two workflows, not one. A pull request workflow that can never apply, and a post-merge workflow that can. Keeping them in separate files makes the separation visible to anybody reading the repository, and makes it obvious in review when somebody proposes blurring it.
Why two separate workflows
Section titled “Why two separate workflows”The single most important structural decision.
A pull request contains code somebody proposed. terraform apply executes provider operations with real credentials. If one workflow can do both, then the conditions separating them are if: expressions — and an if: expression is one bad edit from being wrong.
Two workflows with different triggers and different permissions make the separation structural:
terraform-plan.yml | terraform-apply.yml | |
|---|---|---|
| Trigger | pull_request | push to default branch, or workflow_dispatch |
| Terraform operation | plan only | apply |
| Cloud role | Read-only | Write |
| Environment | None | Production, with required reviewers |
| Can it run on fork code? | Only with restrictions | Never |
The plan workflow
Section titled “The plan workflow”name: Terraform plan
on: pull_request: paths: ['environments/**', 'modules/**', '.github/workflows/terraform-*.yml']
permissions: contents: read pull-requests: write id-token: write
concurrency: group: tf-plan-${{ github.event.pull_request.number }} cancel-in-progress: true
jobs: static: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: hashicorp/setup-terraform@v4 with: terraform_version: ${{ vars.TERRAFORM_VERSION }} terraform_wrapper: false - run: terraform fmt -check -recursive -diff - name: Validate every root module run: | for dir in environments/*/; do echo "::group::validate $dir" terraform -chdir="$dir" init -backend=false -input=false terraform -chdir="$dir" validate echo "::endgroup::" done
plan: needs: static runs-on: ubuntu-latest environment: plan strategy: fail-fast: false matrix: env: [dev, staging, production] steps: - uses: actions/checkout@v7 - uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: ${{ vars.TF_PLAN_ROLE_ARN }} aws-region: eu-west-1 - uses: hashicorp/setup-terraform@v4 with: terraform_version: ${{ vars.TERRAFORM_VERSION }} terraform_wrapper: false - name: Plan id: plan run: | cd "environments/${{ matrix.env }}" terraform init -input=false set +e terraform plan -input=false -lock-timeout=5m -no-color -out=tfplan echo "exit=$?" >> "$GITHUB_OUTPUT" set -e terraform show -no-color tfplan > plan.txt terraform show -json tfplan > plan.jsonPoints worth reading carefully:
terraform_wrapper: false. The setup action’s wrapper alters exit codes and output, which breaks -detailed-exitcode handling and makes the raw plan harder to capture. Disable it and handle output yourself.
terraform -chdir= rather than cd. Cleaner for the validate loop, and it keeps the working directory predictable.
id-token: write. Required for OIDC. Without it the credential exchange fails with a message that does not obviously say so.
A pinned Terraform version. Two runners on different versions produce different plans. The version belongs in the workflow and should match what required_version allows.
environment: plan. Even the read-only role goes through an environment, so its secrets and variables are scoped and its use is logged.
No plan artifact upload here. tfplan contains sensitive attributes; see below.
Credentials
Section titled “Credentials”Three distinct identities, and conflating them removes the point of the separation.
Static checks: none. -backend=false means no state access.
Plan: a read-only role. Terraform refreshes real resources to compute a plan, so it needs read on the provider and read/write on the state backend’s lock — which is not zero, and is much smaller than apply.
Apply: a write role, assumable only from the apply workflow, on the default branch, in the production environment.
OIDC is what makes this enforceable rather than aspirational. The cloud role’s trust policy can require a specific repository, a specific ref, and a specific environment:
{ "Condition": { "StringEquals": { "token.actions.githubusercontent.com:sub": "repo:example-org/infrastructure:environment:production" } }}With that condition, a workflow running on a pull request branch cannot assume the apply role, regardless of what its YAML says. The separation stops depending on the repository’s configuration being correct and starts depending on the cloud provider’s authorisation, which is the boundary you want it on.
Use a wildcard in that sub claim carefully. repo:example-org/infrastructure:* matches every workflow in the repository including pull request runs, which discards the property just described.
The apply workflow
Section titled “The apply workflow”name: Terraform apply
on: push: branches: [main] paths: ['environments/**', 'modules/**'] workflow_dispatch: inputs: environment: description: Environment to apply required: true type: choice options: [dev, staging, production]
permissions: contents: read id-token: write
jobs: apply: runs-on: ubuntu-latest environment: ${{ inputs.environment || 'dev' }} concurrency: group: tf-apply-${{ inputs.environment || 'dev' }} cancel-in-progress: false steps: - uses: actions/checkout@v7 - uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: ${{ vars.TF_APPLY_ROLE_ARN }} aws-region: eu-west-1 - uses: hashicorp/setup-terraform@v4 with: terraform_version: ${{ vars.TERRAFORM_VERSION }} terraform_wrapper: false - name: Apply run: | cd "environments/${{ inputs.environment || 'dev' }}" terraform init -input=false terraform apply -input=false -auto-approve -lock-timeout=10mcancel-in-progress: false is the critical line. Cancelling an apply mid-operation can leave state locked and infrastructure half-changed. Queue; never cancel.
-auto-approve is correct here and nowhere else. There is no terminal to prompt at, and the human approval has already happened at the environment gate. It is not a shortcut past review; it is the acknowledgement that review happened earlier and in a better place.
Automatic dev, manual production. Merges apply to dev automatically. Staging and production go through workflow_dispatch with the environment’s required reviewers. That ordering is a policy choice, and it is worth being explicit about the trade: automatic apply narrows the window in which merged-but-unapplied configuration exists, which is itself a form of drift. Manual apply widens that window in exchange for a second human checkpoint. Teams with mature validation and small changes lean automatic; teams whose plans routinely contain surprises should not, and should treat that as a signal about their validation rather than their approval process.
Approvals belong to the environment
Section titled “Approvals belong to the environment”The most valuable GitHub feature in this workflow, and the most under-used.
An environment with required reviewers means the apply job pauses and waits for a named person. What that gets you:
A recorded decision. Who approved a production change, and when. This is the audit answer that is otherwise reconstructed from memory.
Scoped secrets and variables. The production role ARN is a production-environment variable. A dev job cannot read it.
A wait timer, if useful. A deliberate delay before production applies, giving somebody time to stop it. Useful for changes that are individually safe and collectively worth pausing on — a five-minute timer costs nothing on a Tuesday afternoon and has occasionally been the thing that let somebody cancel a change during an unrelated incident.
Branch restrictions. The environment can require that deployments come only from the default branch.
Combined with the OIDC trust condition above, the environment is doing real work: it is the thing the cloud provider’s trust policy names.
State locking and concurrency
Section titled “State locking and concurrency”Two independent mechanisms, and you need both.
State locking is Terraform’s, via the backend. It prevents two operations mutating the same state simultaneously, and it works regardless of what triggered them — a pipeline, a colleague’s laptop, or a scheduled drift check. It is why -lock-timeout matters: the second operation should queue rather than fail, because a lock error looks like a broken pipeline and gets retried, while a wait looks like a wait.
Workflow concurrency is GitHub’s. It prevents two runs starting. Group by environment so different environments proceed in parallel while the same one serialises.
The failure they jointly prevent: two applies interleaving on the same state, producing a state file that describes neither outcome. Recovering from that is a manual, unpleasant process, and it is entirely avoidable.
The plan artifact question
Section titled “The plan artifact question”Whether to apply the reviewed plan or re-plan at apply time.
Applying a saved plan guarantees that what was reviewed is what runs. Terraform refuses a plan whose state has moved on, which is a feature — it tells you the world changed. The cost: the plan file contains the same sensitive attributes as state, so it must be stored with matching controls. A workflow artifact readable by anybody with repository read access is not that.
Re-planning at apply time is simpler and always current, at the cost that the applied change is not literally the reviewed one.
The practical compromise most teams reach: re-plan for dev and staging; for production, save the plan to storage with appropriate access controls, and apply it. Do not upload plan files as ordinary workflow artifacts in either case.
Scheduled drift detection
Section titled “Scheduled drift detection”The pipeline above only runs when somebody changes something. Infrastructure changes without anybody changing anything, and a fourth workflow catches that.
name: Terraform drift
on: schedule: - cron: '0 6 * * 1-5' workflow_dispatch:
permissions: contents: read id-token: write issues: write
jobs: drift: runs-on: ubuntu-latest environment: plan strategy: fail-fast: false matrix: env: [staging, production] steps: - uses: actions/checkout@v7 - uses: aws-actions/configure-aws-credentials@v6 with: role-to-assume: ${{ vars.TF_PLAN_ROLE_ARN }} aws-region: eu-west-1 - uses: hashicorp/setup-terraform@v4 with: terraform_version: ${{ vars.TERRAFORM_VERSION }} terraform_wrapper: false - name: Detect drift run: | cd "environments/${{ matrix.env }}" terraform init -input=false set +e terraform plan -input=false -lock-timeout=5m -no-color -detailed-exitcode -out=tfplan code=$? set -e if [ "$code" = "2" ]; then terraform show -no-color tfplan | head -c 60000 > drift.txt echo "DRIFTED=true" >> "$GITHUB_ENV" elif [ "$code" != "0" ]; then exit "$code" fiThree design points:
It uses the plan role, not the apply role. Detection reads; it does not correct. A drift workflow that could apply is an unattended apply on a schedule.
It runs on the default branch. The question is whether reality matches what is merged, not what somebody is proposing.
Weekdays only, early. Drift discovered at 06:00 on Tuesday gets acted on. Drift discovered at 03:00 on Saturday produces an alert nobody reads until Monday, by which time it has been superseded.
What to do with a finding is a real decision rather than an obvious one, and it is the subject of infrastructure drift detection. The short version: open an issue, do not auto-apply. An automatic correction at 06:00 can undo somebody’s deliberate emergency fix from 03:00.
Reusable workflows across repositories
Section titled “Reusable workflows across repositories”Once several repositories run this pipeline, the workflows themselves become shared code with the same versioning problem as Terraform modules.
Extract the pipeline as a reusable workflow in a central repository, taking the environment and directory as inputs.
Consumers reference it by tag, not by branch:
jobs: plan: uses: example-org/workflows/.github/workflows/terraform-plan.yml@v3.1.0 with: environment: production directory: environments/production secrets: inherit@main here has the same defect as ?ref=main on a module: the central team merges something and every repository’s pipeline changes with no pull request anywhere to notice it. For a workflow that holds cloud credentials, that is a larger exposure than for a module.
Be careful with secrets: inherit. It passes every secret the calling workflow can see to the called workflow. Naming the specific secrets is more typing and much more explicit about what a shared workflow can reach.
The central workflow repository needs stricter review than its consumers. It runs with the credentials of every repository that calls it. CODEOWNERS, required reviews, and protected tags are all warranted.
What this produces
Section titled “What this produces”Assembled, the pipeline answers questions that are otherwise expensive:
“Why does this resource exist?” — the commit, its pull request, and the linked issue.
“Who approved changing production?” — the environment approval record.
“What did we think would happen?” — the plan comment on the pull request.
“When did it apply, and did it succeed?” — the workflow run.
“Has anything changed outside this process?” — the scheduled drift plan, which is the only one of these questions the pipeline cannot answer from its own history.
That is the auditability the pillar hub described. It is not a separate compliance activity bolted on; it is a by-product of the pipeline being shaped correctly.
Self-hosted runners
Section titled “Self-hosted runners”Many infrastructure pipelines need a self-hosted runner, because the state backend or the provider API sits inside a private network. It is a legitimate requirement with a security consequence worth stating plainly.
A self-hosted runner executes code from pull requests. On a public repository that is anybody’s code, running on a machine inside your network with whatever that machine can reach. GitHub’s own documentation is unambiguous that self-hosted runners should not be used with public repositories for exactly this reason.
On private repositories the exposure is smaller and real. Anybody who can open a pull request can run code on that machine.
The mitigations that matter:
Ephemeral runners. A runner that handles one job and is destroyed cannot carry state — a modified tool, a planted credential, a poisoned cache — from one job to the next. This is the single most valuable control.
Network scope, not machine scope. The runner should reach the state backend and the provider endpoint. Not the rest of the network. “It is inside the VPC” is not a security boundary.
No standing cloud credentials on the host. Use OIDC from the runner as you would from a hosted one. An instance profile attached to the runner is a credential every job inherits, including jobs from pull requests.
Separate runner pools by trust level. The pool that runs pull request plans and the pool that runs production applies should not be the same machines. If a pull request job can compromise a runner that later performs an apply, the separation established earlier in this lesson is gone.
If you can avoid self-hosted runners for the plan stage — by making the state backend reachable, or by planning against a provider endpoint that is public — do. Reserve them for the apply, where the input is a merged commit rather than a proposal.
Migrating an existing pipeline
Section titled “Migrating an existing pipeline”Few teams build this from scratch. Most have something that works and needs restructuring, and doing that safely has an order.
-
Split plan and apply into separate files first, before changing anything else. Keep the same behaviour. This is the structural change everything else depends on and it can be verified on its own.
-
Add the plan comment. Immediate, visible value, and it starts changing how reviews happen before any of the harder work lands.
-
Introduce OIDC for plan only. Lower risk than apply, and it proves the trust policy works.
-
Add the environment to the apply job, initially without required reviewers, so approvals and secret scoping are wired up before anybody is blocked.
-
Turn on required reviewers for production. This is the change people notice; do it once everything else is stable.
-
Move apply to OIDC, and narrow the trust condition to the environment.
-
Delete the long-lived cloud credentials from repository secrets. Not before this point, and definitely not after forgetting to.
-
Add required status checks so the pipeline cannot be skipped.
-
Add scheduled drift detection last — it is the least urgent and the most likely to produce noise while everything else is settling.
Step 7 is the one teams forget. A repository that has migrated to OIDC and still holds an unused access key in its secrets has not reduced its exposure; it has added a second mechanism alongside the first. Search your secrets for anything that is no longer referenced, and remove it.
Where AI fits, and where it does not
Section titled “Where AI fits, and where it does not”Copilot and agents are genuinely useful around this pipeline, in a narrow band.
Explaining a plan. A 400-line plan reduced to “this replaces the database because the engine version changed, and adds two security group rules” is a real saving, and it is a reading task — exactly what AI is good at.
Explaining a failure. A provider error message with a cause and a file to look at, rather than a stack of API responses.
Drafting the pull request description. The mechanical half — what changed, which environments — leaving the author to supply the intent.
Reviewing configuration against written standards. Encoded as review instructions, applied consistently.
Where it must not sit: anywhere that decides. Not as a required check, not gating an apply, not assessing whether a plan is safe. A plan summary is a reading aid for the reviewer; it is not the review, and a reviewer who reads the summary instead of the plan has swapped one skipped step for another.
The rule from agentic CI/CD holds exactly here: deterministic steps decide, AI steps inform. An AI-generated summary that is wrong should cost you a confusing comment, never an apply.
And the corollary that matters most in an infrastructure repository: AI-generated infrastructure changes go through the same plan, the same review and the same approval as any other change. A configuration that a model wrote is a proposal, and the fact that it was produced quickly is not evidence that it is correct.
Common mistakes
Section titled “Common mistakes”One workflow doing plan and apply. The separation becomes an if: expression.
A wildcard OIDC subject claim. Discards the property that pull requests cannot assume the apply role.
cancel-in-progress: true on apply. A cancelled apply leaves locked state and half-changed infrastructure.
Uploading tfplan as a normal artifact. Sensitive attributes, broadly readable.
Unpinned Terraform version. Different runners, different plans.
terraform_wrapper left enabled. Breaks exit-code handling and output capture.
Automatic production apply on merge, unexamined. Sometimes right; rarely a default anybody chose.
-lock=false to get past a lock. Removes the protection against state corruption.
No environment on the apply job. No approval record, unscoped secrets.
Handling failures mid-apply
Section titled “Handling failures mid-apply”An apply that fails halfway is the situation this whole pipeline exists to make survivable, and it is worth knowing what state you are in when it happens.
Terraform is not transactional. There is no rollback. If an apply creates six resources and fails on the seventh, those six exist and are recorded in state. That is the correct behaviour — the alternative would be destroying things that are working — but it means “the apply failed” does not mean “nothing happened”.
State is usually consistent. Terraform writes state as it goes, so a failed apply typically leaves state accurately describing what was created. The next plan will show the remaining work. This is the normal case and it recovers by fixing the cause and re-running.
Sometimes the lock is left held. A runner killed mid-apply — a cancelled workflow, a timeout, a spot instance reclaimed — can leave the lock in place with nothing holding it. terraform force-unlock <LOCK_ID> clears it, and should be run by a person who has confirmed in the workflow run log that the job genuinely terminated.
Occasionally state is genuinely wrong. A resource was created but the write to state failed. The next plan tries to create it again and the provider rejects it as already existing. terraform import reconciles this, and it is a manual operation requiring care.
The pipeline properties that make each of these recoverable:
Concurrency without cancellation, so a second run does not start on top of the first.
A logged, attributable run, so you can see exactly which step failed and what had completed.
A plan comment on the originating pull request, so you know what the apply was attempting.
An environment approval record, so it is clear who to talk to.
None of that prevents the failure. All of it turns a confusing situation into a legible one, which is the realistic goal.
Mental model
Section titled “Mental model”The pull request workflow proves what would happen. The apply workflow makes it happen. They must not be the same workflow, must not share credentials, and must not both be triggerable by proposed code.
Everything else — matrices, caching, comment formatting — is optimisation. That separation is the design.
What you learned
Section titled “What you learned”- Two workflows with different triggers and permissions, not one with conditionals
- OIDC trust conditions can bind the apply role to a specific environment, making the separation cloud-enforced
terraform_wrapper: falseis needed to handle exit codes and output correctlycancel-in-progress: falseon apply; cancelling mid-apply leaves locked state-auto-approveis correct only where the human approval already happened at an environment gate- State locking and workflow concurrency are different mechanisms and you need both
- Plan files are as sensitive as state and must not be ordinary workflow artifacts
- A
paths:filter can interact badly with required status checks — verify rather than assume
Exercise
Section titled “Exercise”Use a disposable repository. No cloud credentials — the local and null providers exercise the whole structure.
-
Create
environments/dev/andenvironments/production/with alocal_fileresource each. -
Add
terraform-plan.ymlrunning on pull requests with a matrix over both environments. Confirm both plan. -
Add
terraform-apply.ymlrunning on push to the default branch, with anenvironmentthat has you as a required reviewer. -
Open a pull request, merge it, and watch the apply job wait. Approve it. Check where the approval is recorded.
-
Add
concurrencywithcancel-in-progress: falseto apply. Merge twice quickly. Predict: does the second run cancel the first, or queue? -
Try adding an
applystep to the plan workflow behind anif:. Write down what a pull request editing that workflow file could do. Then remove it. -
Add
.github/workflows/toCODEOWNERSand confirm a change to the workflow requests the right reviewer. -
Delete the repository.