Infrastructure Drift Detection

Your repository is correct. Your infrastructure is something else. Nothing reports the difference unless something is looking for it.

Drift is not a failure of discipline — it is the normal state of any system that people can change, and the engineering question is not how to prevent it but how to know about it and what to do next.

Before you start

• GitOps Drift and Reconciliation — the Kubernetes-specific operational model
• Infrastructure Promotion with Pull Requests — the intended path

Detect or correct

The decision that shapes everything else, and it is genuinely a decision rather than an obvious answer.

Detection tells you something changed outside the process. Always valuable, always safe.

Automatic correction puts it back. Valuable where the declaration is authoritative; actively harmful where somebody is mid-incident and the controller keeps reverting their fix.

	Detect only	Auto-correct
Repository is authoritative	By convention	In fact
Emergency change	Persists until somebody acts	Reverted, possibly mid-incident
Someone must respond	Yes	No
Risk of surprise	Low	Real
Suits	Terraform, cloud, hosts	Kubernetes workloads

Kubernetes is where auto-correction fits best, because reconciliation is the platform’s own model and the objects are mostly recreatable.

Terraform auto-apply on drift is dangerous. A scheduled job with apply credentials, correcting a difference nobody has looked at, at whatever hour it runs. The plan might be a five-resource destroy caused by somebody’s console change. Detect, alert, and let a person decide.

Hosts sit in between. An Ansible playbook run on a schedule in check mode is detection; running it for real unattended is correction, and it is only safe for a genuinely small set of playbooks.

Danger

Automatic correction requires a suspend mechanism and the knowledge of how to use it, before you enable it. Otherwise the first serious incident includes a controller undoing the fix while somebody is watching to see whether it worked — and the person diagnosing rarely connects the recurrence to their own change having been reverted.

Drift by tool

Different mechanics, same question.

Terraform and OpenTofu. A scheduled plan on the default branch. Non-empty means either an unapplied merge or a real out-of-band change. -refresh-only distinguishes them: it shows how state differs from reality without proposing configuration changes, which is precisely the drift question separated from the pending-change question.

Kubernetes with a GitOps controller. Continuous, and free. The controller always knows. The work is alerting on it rather than detecting it.

Kubernetes without a controller. kubectl diff against rendered manifests, on a schedule. Needs cluster read access from wherever it runs.

Cloud configuration outside Terraform. Account-level settings, organisation policies, identity configuration, logging destinations — the things nobody put in Terraform because they were set up once during onboarding. The provider’s own configuration recorder or compliance service covers this, or a scheduled export compared against expectations. It is the category most often unmonitored, because it falls between every tool’s responsibility, and it is where the highest-consequence undetected changes tend to live.

Hosts. ansible-playbook --check --diff on a schedule. Reports what would change, which is what has drifted — with the caveat that check mode is imperfect and dependent tasks report inaccurately.

Container images. A running digest that does not match what the manifest declares. Usually a mutable tag that moved, and the fix is digests rather than detection.

The scheduled Terraform check

The most commonly missing piece, and it is a small workflow.

name: Drift

on:
  schedule:
    - cron: '0 6 * * 1-5'
  workflow_dispatch:

permissions:
  contents: read
  id-token: write
  issues: write

jobs:
  detect:
    runs-on: ubuntu-latest
    environment: plan
    strategy:
      fail-fast: false
      matrix:
        env: [staging, production]
    steps:
      - uses: actions/checkout@v7
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: ${{ vars.TF_PLAN_ROLE_ARN }}
          aws-region: eu-west-1
      - uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: ${{ vars.TERRAFORM_VERSION }}
          terraform_wrapper: false
      - name: Refresh-only plan
        id: drift
        run: |
          cd "environments/${{ matrix.env }}"
          terraform init -input=false
          set +e
          terraform plan -input=false -refresh-only -no-color -detailed-exitcode -out=tfplan
          code=$?
          set -e
          if [ "$code" = "2" ]; then
            terraform show -no-color tfplan | head -c 60000 > drift.txt
            echo "drifted=true" >> "$GITHUB_OUTPUT"
          elif [ "$code" != "0" ]; then
            exit "$code"
          fi

The plan role, not the apply role. Detection reads. A drift workflow that could apply is an unattended apply on a schedule.

-refresh-only asks the right question: how does state differ from reality, separate from what the configuration would change.

The default branch, because the question is whether reality matches what is merged.

Weekdays, early. Drift found at 06:00 on a Tuesday gets acted on. Drift found at 03:00 on a Saturday produces an alert nobody reads until Monday, by which time it has been superseded.

fail-fast: false so one environment’s failure does not hide another’s.

What to do with a finding

The part that determines whether detection is useful.

Open an issue, do not page. Drift is almost never an emergency, and paging on it trains people to ignore the alerts.

Include the diff. An issue saying “drift detected” sends somebody to a log; one containing the resources and the differences is actionable in a minute.

Assign it to the owning team, using the same map as CODEOWNERS.

Update rather than duplicate. Persistent drift that opens a new issue every morning is a channel people mute. One issue, updated, closed when resolved.

Have a triage step. Drift falls into a small number of categories, and knowing which determines the response.

The categories

Every drift finding is one of these, and the response differs.

An unapplied merge. The repository is ahead of reality. Not out-of-band change — apply it, and consider why the merge-to-apply gap was large enough to notice.

A deliberate emergency change. Somebody fixed something. The response is to bring the repository in line the same day, before the next apply silently reverts it.

An undeclared change nobody remembers. The interesting one. Something or somebody changed the system and there is no record anywhere. Worth an hour of investigation rather than a shrug: it is occasionally a decommissioned pipeline still holding credentials, occasionally an integration nobody knew was configured, and occasionally somebody who will remember once asked. All three are findings worth having, and the finding is more valuable than the correction.

Another system’s legitimate management. An autoscaler, an operator, a cloud service writing a field. Not drift; a field you should not be declaring. Remove it from your configuration rather than suppressing the finding.

A provider or API change. The provider now returns a field it did not, or a default changed. Usually resolved by a provider upgrade and worth recognising so nobody spends an afternoon on it.

Metadata churn. Timestamps, generated identifiers, computed fields. Noise, and the fix is configuring the detection to ignore them — carefully, and only once you know what they are.

The triage question: which of the six is this? Five have a routine response, and the third is the one worth an hour.

Reducing drift at the source

Detection finds it. These reduce how much there is.

Remove standing write access to production. The largest single reduction, and the hardest organisationally. Most drift comes from people who could make the change, doing so because it was faster than the process — and removing the ability is only reasonable once the process is fast enough to be the obvious choice. Doing it in the other order produces a team that cannot fix things and resents the platform.

Give people a fast path through the process. Drift is frequently a symptom of a process too slow for what somebody needed to do. If an urgent change takes two hours to merge and apply, people will bypass it — and that is a finding about the process rather than about the person.

Reduce the merge-to-apply gap. Automatic apply for low-risk environments, prompt apply for production. Every hour of gap is drift you created.

Do not declare fields you do not own. Autoscaler-managed replica counts, operator-managed annotations. This eliminates a whole category rather than suppressing it.

Make the emergency path explicit. A documented way to make an urgent change — suspend, change, record, backport the same day — means the change happens with a record rather than without one.

Backport the same day, without exception. A change made at 03:00 and backported that morning is a footnote. The same change backported in three weeks is an outage when something reverts it.

Alerting without noise

Alert on drift that persists past one detection cycle. Transient differences are usually a rollout or an apply in flight.

Alert on the detector itself failing. A drift check that has not run in a week reports no drift, which looks exactly like a healthy estate. This is the alert teams forget and the one that hides everything else.

Do not alert on every detection run. A channel receiving a message per successful check is a channel nobody reads.

Route by environment. Production drift to a channel people watch; development drift to a weekly digest.

Track the trend, not just the instances. Drift findings per week is a health metric. Rising means the process is being bypassed more; falling means it is working, or the detector broke.

Detection across the estate

The unifying view most teams lack.

One report answering “does reality match the repository”, per environment, across every tool. Terraform state, Kubernetes resources, cloud configuration and host state, in one place.

Nobody’s tool does this out of the box, because each tool reports its own domain. Assembling it is a small piece of platform work — each detector writes a status somewhere, and one job aggregates.

The value is in the aggregate. Individual findings get triaged; the aggregate answers whether the estate is under control, and it is the number worth showing somebody who is not a platform engineer.

Include coverage. “Zero drift” is only meaningful alongside “and we check all of it”. An estate with no detection for its cloud configuration has zero drift in the sense that nobody is looking.

Report the gaps honestly. Which parts of the estate have detection and which do not is the more useful half of the report, and it is what turns drift detection from a check into a programme.

Drift you cannot see

The categories no detector covers, and knowing them prevents false confidence.

Resources nothing declares. Terraform reports drift on resources in its state. A resource created by hand that Terraform has never known about produces no finding — it is invisible rather than drifted. Finding these means comparing what the provider has against what your state files contain, which is a different and harder exercise.

Accounts and projects nobody manages. An estate of nine cloud accounts where seven are under Terraform has two nobody is looking at.

Anything outside the tools’ scope. DNS held at a registrar, a third-party SaaS configuration, a firewall rule on a device nobody automated. Real infrastructure, and no detector covers it.

Resources a controller is not watching. A GitOps controller scoped to two namespaces knows nothing about the other six.

Configuration inside a running application. A feature flag toggled at runtime, a setting changed through an admin interface. Not infrastructure in the tools’ sense and frequently more consequential than a security group.

The honest position: drift detection covers the parts of the estate under management, and the parts outside are invisible rather than clean. The most useful drift report has two halves — what drifted, and what nobody is checking — and the second half is what turns detection into an actual programme.

The exercise worth running once: list everything your tools manage, then list everything that exists, and look at the difference. It takes a day and the findings are consistently uncomfortable.

Cost of drift

Making the case, because scheduled detection is work somebody has to justify.

Rebuild capability. A cluster or an account reconstructed from the repository comes back missing exactly the things that drifted. Every hour of unresolved drift is a gap between what you have and what you can recreate — which is only discovered during a recovery.

Environment comparability. Drift in staging and not production, or vice versa, means the environments stop predicting each other. That devalues every test you run in the lower one.

Incident diagnosis. “What changed?” answered from Git is fast; answered from Git plus an unknown set of manual changes is not.

Compounding. One undeclared resource is a footnote. Forty is an estate nobody can reason about, and it got there one reasonable decision at a time.

Silent reversion. The specific expensive failure: an emergency fix reverted weeks later by a routine apply, causing a recurrence nobody connects to a change made days earlier.

Audit. “Show that production matches what was reviewed” is a question with an easy answer or a very difficult one, depending entirely on whether anybody has been checking.

The framing that gets it funded: drift detection is not tidiness. It is the thing that makes your disaster recovery plan true, and most teams have never verified that theirs is.

Common mistakes

Auto-applying Terraform drift. An unattended apply on a schedule, on a plan nobody read.

Auto-correcting with no suspend procedure. The controller reverts an emergency fix.

Paging on drift. It is rarely urgent, and paging trains people to ignore it.

A new issue every morning. Muted.

No alert on the detector failing. Silence looks like health.

Suppressing findings you have not diagnosed. The symptom goes away; the cause does not.

Declaring fields another system manages. Permanent drift, and the fix is in your configuration.

Detection with no triage. Findings accumulate unread.

Never backporting emergency changes. Silently reverted later.

Reporting drift without reporting coverage. Zero drift where nobody is looking.

Bringing drift back into the repository

The response to most findings, and the mechanics differ by tool.

Terraform: a configuration change matching reality. If somebody added a security group rule in the console, add it to the configuration and confirm the plan is empty. That is the fix — the rule stays, and it is now declared.

Or terraform import, where the change created a resource Terraform does not know about. Bring it under management, then confirm the plan is empty.

Or revert the change, where it should not have been made. A deliberate decision rather than the default, and the person who made it should be part of it.

Kubernetes: a manifest change, then let the controller reconcile. With self-heal enabled the drift is already gone, so this is about capturing what was learned rather than restoring anything.

Hosts: a playbook change, then run it. Check mode first to confirm it produces the intended difference.

In every case, the verification is the same: after the fix, the detector should report nothing. An empty plan, a synced controller, a check run with no changes. That is the evidence the repository and reality agree, and it is the step people skip.

Do it the same day. The value of the fix decays quickly — the person who made the change forgets the details within a week, and the reason it was made within a month.

Record why, in the commit message. “Adding the rule that was applied manually during INC-1042” is the sentence that stops somebody removing it six months later as unexplained.

Building coverage

Most estates start with no detection and get there incrementally.

1. Start with production Terraform. Highest consequence, and a scheduled -refresh-only plan is an afternoon’s work.

2. Read the findings for a month before automating anything about them. You will learn what your estate’s normal drift looks like, and it is more than you expect.

3. Fix the noise. Fields another system manages, metadata churn. Each one removed makes the remaining findings more credible.

4. Add Kubernetes. If you run a GitOps controller this is alerting rather than detection.

5. Add the cloud configuration nobody’s tool covers. Usually the largest gap and the least visible.

6. Add hosts, with scheduled check-mode runs.

7. Aggregate, so one report answers the question for the estate.

8. Report coverage alongside findings, so “no drift” is qualified by “in the parts we check”.

Step 2 is the one to insist on. A team that automates a response before understanding what normal looks like builds automation around noise, and then either tunes it forever or turns it off.

Do not attempt everything at once. Detection for one environment, working and trusted, is worth more than partial coverage everywhere that nobody believes.

Mental model

Drift is the difference between what you declared and what exists. Detection tells you it happened; correction decides it should not have. Only the first is always safe, and only the second makes the repository authoritative in fact.

The framing that makes findings useful: drift is a report about your process. Every instance answers “what changed the system without going through Git”, and the answers are more informative than the differences themselves.

What you learned

• Detection is always safe; automatic correction is a separate decision needing a suspend mechanism
• Terraform auto-apply on drift is an unattended apply on an unread plan
• -refresh-only separates the drift question from the pending-change question
• Every finding is one of six categories, and five have a routine response
• A field another system manages is not drift — stop declaring it
• Alert on persistent drift and on the detector failing; never on every run
• Most drift is a symptom of a process too slow for what somebody needed to do
• “Zero drift” is meaningless without coverage; report the gaps

Exercise

Use a disposable cloud project or local providers, and a local cluster.

1. Apply a small Terraform configuration. Change one resource outside Terraform — a console edit, or local_file edited directly.

2. Run terraform plan. Predict: what does it propose?

3. Run terraform plan -refresh-only. Compare the two outputs. Predict: which one answers “what drifted”?

4. Add a scheduled drift workflow using the plan role. Trigger it manually and confirm it opens an issue.

5. In a cluster with a GitOps controller and self-heal off, change a Deployment with kubectl. Predict: what is reported, and how long does it persist?

6. Enable self-heal and repeat. Time the correction.

7. Add an HPA managing a Deployment whose manifest also declares replicas. Predict: how often is drift reported?

8. Remove replicas from the manifest and repeat.

9. Stop the drift detector. Predict: what does your monitoring show, and would you notice?

10. Delete everything.

Related lessons

Platform Engineering with GitThe pillar's capstone.

GitOps Drift and ReconciliationThe Kubernetes operational model.

Terraform Branching StrategyThe merge-to-apply gap.

Policy as Code with GitPreventing what detection would find.