GitOps Drift and Reconciliation

At 03:00 an engineer runs kubectl edit to stop an outage. It works. At 03:07 the controller reverts it and the outage resumes.

That sequence is the operational reality of GitOps, and a team that has not thought about it in advance experiences it at the worst possible moment. This lesson is about the model that produces it and how to work inside it.

Before you start

• GitOps Explained — the reconciliation loop
• Argo CD or Flux — the controller doing it

What drift actually is

Desired state is what the repository declares.

Actual state is what the cluster reports.

Drift is a persistent difference between them that the controller did not cause.

The word “persistent” is doing work. Clusters differ from their declarations constantly and most of it is not drift:

A rollout in progress. Two old pods, one new. Transient by design.

Status fields. Every resource has one, written by controllers. Never in the repository.

Defaulted fields. The API server fills in values you did not specify. Not drift; the resource simply has more fields than your manifest.

Autoscaler decisions. An HPA changing replicas is doing its job. This is drift only if your manifest also declares replicas, at which point you have two managers fighting over one field — and the fix is removing it from the manifest.

Mutating webhooks. A sidecar injected after your manifest was applied. The cluster legitimately contains something the repository does not.

Real drift is what remains: somebody edited a resource, a script changed something, a controller you forgot about is managing a field, or a resource exists that nothing declares.

Why it happens

Not carelessness, mostly.

Emergency changes. Somebody fixed production at 3am. This is the most common cause and frequently the correct action.

Debugging. Scaling a Deployment to zero to test something, and forgetting to undo it.

Another controller. An operator, a service mesh, a cloud controller writing annotations onto a Service. Legitimate, and it looks like drift unless you tell your GitOps controller to ignore those fields.

Automation nobody remembers. A cron job, a script, a decommissioned pipeline still holding credentials.

Manual creation. A resource somebody applied by hand a year ago that nothing declares. With pruning enabled this is the one that gets deleted, and it is frequently something load-bearing that nobody remembers creating.

The controller itself failing. A Kustomization that has not reconciled successfully for a week shows the cluster as it was, and the difference from the repository grows silently.

Self-heal, and the 3am problem

selfHeal: true in Argo CD, or Flux’s reconciliation interval doing the same thing: the controller reverts a manual change to a managed resource.

This is the property you adopted GitOps for. It makes the repository authoritative in fact rather than in policy. A convention that all changes go through Git decays; a mechanism that reverts changes that did not does not.

It also reverts your emergency fix, within one reconciliation interval, usually while the person who made it is still watching to see whether it worked.

Danger

The failure to plan for: an engineer changes a resource by hand during an incident, the fix works, and the controller reverts it minutes later. The outage resumes, the engineer does not connect it to their own fix having been undone, and diagnosis restarts from the beginning. Every team running self-heal should know how to suspend reconciliation before the incident that requires it.

Making an emergency change properly

The procedure, and it is short enough to memorise.

1. Suspend reconciliation for the affected resource. flux suspend kustomization NAME, or set the Argo CD Application’s sync policy to none. Do this first, before making the change.

2. Make the change. Fix the outage.

3. Say what you changed, where the team will see it. The incident channel, the ticket. This is the record that stops it being lost.

4. Bring the repository in line, the same day. A pull request making the same change, reviewed at whatever speed the situation warrants.

5. Merge and resume reconciliation. The controller applies the same change; nothing moves.

6. Verify. Confirm the reconciliation reports no changes, which proves the repository and the cluster now agree.

Step 1 is the one people skip, and skipping it is what produces the 3am scenario.

Step 4 has a deadline. A suspended resource is a resource nothing is managing: no self-healing, no drift detection, and no reconciliation of anything else declared in the same scope. A suspension nobody remembers is drift with the safety net switched off, and it will be discovered weeks later when a change that was merged appears never to have deployed. Same day, or the suspension itself becomes the problem.

If you cannot suspend first — because the outage is now — make the change, then suspend immediately, then continue from step 3. Better than losing the fix.

The alternative that also works: make the change in Git and let the controller apply it. For a change you can express as a commit and a merge, this is cleaner and leaves no suspension to remember. It is slower by the length of your review process, which during an incident may or may not be acceptable.

Detecting drift

Both controllers report it continuously. The question is whether anybody is looking.

Argo CD shows OutOfSync on the Application, with a per-resource diff.

Flux reports it in the Kustomization’s conditions, and applies the correction on the next interval.

Alert on persistent drift, not on every occurrence. A resource out of sync for more than an interval or two means something is fighting the controller or somebody made a change that has not reached Git.

Alert on the controller not reconciling. This is the one teams forget. A controller that has stopped shows the last state it saw, which looks identical to a healthy cluster with no drift. Every controller exposes a last-reconciled timestamp; alert when it goes stale.

Do not alert on every sync. Successful reconciliation is the normal case, and a channel full of it trains people to ignore the channel.

What is genuinely worth waking somebody for: the controller has not reconciled in an hour, or a production Application has been degraded for more than a few minutes. Drift itself is usually a working-hours conversation.

Ignoring what you do not own

Persistent, unfixable drift is usually a field somebody else legitimately manages.

Argo CD has ignoreDifferences with JSON pointers or jq expressions. Flux relies on server-side apply field ownership plus Kustomize patches.

The better fix is omitting the field. A Deployment manifest that does not declare replicas at all leaves the field to the HPA, and there is nothing to diff. ignoreDifferences is for fields you must declare that something else mutates.

Server-side apply is the principled version. The API server tracks which manager owns which field, so another manager’s field is not drift — it is somebody else’s business, recorded as such.

Do not use ignore rules to silence drift you have not diagnosed. A field changing for reasons nobody can identify is a finding. Suppressing it makes the symptom go away and leaves the cause, and it produces a controller reporting Synced while the cluster differs meaningfully from the repository — which is worse than visible drift, because the dashboard now says something untrue.

Pruning: the sharpest edge

Pruning deletes resources present in the cluster and absent from the source.

What it is for: removing a manifest should remove the resource. Without pruning, deleting a file leaves the resource running forever, and the repository stops describing the cluster.

What goes wrong: the source does not describe everything in the scope. A resource created by hand two years ago, an operator’s output, a namespace containing more than the controller knows about. Enabling pruning deletes the difference.

Enable it deliberately: run without it for a period, read what the controller reports as extra, account for every item, then enable.

Some things must never be pruned. A PersistentVolumeClaim whose deletion loses data. A CRD whose removal deletes every custom resource of that kind, cluster-wide, immediately. A namespace, whose deletion takes everything in it. Both controllers support annotations that exclude a resource from pruning, and the resources holding data should carry them regardless of how confident you are that the source is complete — the cost of the annotation is nothing and the cost of being wrong is unrecoverable.

Prune failures are a signal. A resource the controller cannot delete — a finalizer holding it — needs investigating rather than forcing.

Failure states

Things the controller can be in besides working.

Reconciliation erroring. An invalid manifest, a missing CRD, an RBAC denial. Visible and usually specific.

Reconciliation stuck. A resource with a finalizer that never completes, or a Kustomization waiting on a health check that will never pass. Looks like it is working and never finishes.

Reconciliation succeeding into a broken state. Manifests apply cleanly, pods crash-loop. Synced and Degraded — this is why sync status is not a health check.

The controller is down. Nothing reconciles, nothing changes, and everything looks exactly as it did. The cluster keeps running, which is a benign failure mode and an invisible one.

The controller is fighting another controller. A resource flapping every interval. Two managers, one field.

The source is unreachable. Authentication expired, a repository moved, network policy changed. The controller reports it; whether anybody sees it depends on alerting.

The one to instrument first is the fourth. It produces no errors and no symptoms until somebody makes a change and nothing happens, which may be days later.

Disaster recovery

The property that makes all of this worth the discipline, and the one nobody tests.

A cluster reconciled from a repository can be rebuilt. Provision a new cluster, bootstrap the controller, point it at the same source, and it converges to the same declared state. That is a genuinely strong recovery position and it is the strongest argument for keeping drift at zero.

What comes back: every declared resource. Deployments, Services, ConfigMaps, RBAC, network policies, ingress definitions.

What does not: anything that was drift. A resource somebody created by hand a year ago is not in the repository and will not exist in the rebuilt cluster. Every hour of unresolved drift is a gap between what you have and what you can rebuild.

What also does not: persistent data, secrets that live in a manager rather than the repository, and anything with cluster-generated identity. Those need their own recovery procedures, and the GitOps repository does not substitute for them.

Test it. Rebuild a non-production cluster from the repository, on a schedule. The findings are always the same shape — resources nobody declared, a bootstrap step nobody documented, a secret that was placed by hand — and finding them during a drill is very different from finding them during an incident.

The measure worth tracking: how long from an empty cluster to a working one, and how much manual intervention it needed. A team that can answer that has a recovery plan; one that assumes it works has a hypothesis.

Drift is the enemy of this property. Not because drift is untidy, but because it is precisely the set of things that will be missing.

Observability

What to put on a dashboard.

Signal	Why
Time since last successful reconciliation	Catches a stopped controller
Count of out-of-sync resources	Catches drift and fights
Reconciliation error rate	Catches broken manifests and permissions
Reconciliation duration	Catches a repository or cluster getting slow
Count of degraded resources	Catches applied-but-broken
Suspended resources	Catches a suspension nobody resumed

The last row is underrated. A suspension made during an incident and never undone is a resource nothing is managing, and nothing else on this list will tell you.

Correlate with deployments. Drift appearing at the same time as a deploy is usually the deploy — a defaulting behaviour or a webhook you have not accounted for. Drift appearing at 03:00 is usually a person, and the useful response is to find out what they were fixing rather than to note the drift.

Drift as information

A shift in framing that changes what teams do with it.

Drift is a report about your process, not just a technical difference. Every instance answers “what changed the cluster without going through Git”, and the answers are useful.

Frequent drift in one namespace means somebody is working around the process, which is usually because the process is too slow for what they need to do. That is a finding about the process.

Drift from another controller means your manifests declare fields you do not own. Fixable, permanently.

Drift after every deploy means a mutating webhook or a defaulting behaviour you have not accounted for.

Drift nobody can explain is the interesting one. Something is changing your cluster and you do not know what — that is worth an hour of investigation, and it occasionally turns out to be a decommissioned pipeline still holding credentials.

Zero drift, ever, is also information. Either the process is working well or the controller is not reconciling. Check which.

Fix the declaration, not the cluster

The operational shift GitOps requires, and the one that takes teams longest to internalise.

Before: something is wrong, you change the cluster, it is fixed.

After: something is wrong, you change the repository, the controller changes the cluster, it is fixed.

The second is slower for a single change and better for everything else. The reasons are worth being explicit about, because in the moment the first is very tempting.

The fix persists. A change made in the cluster is reverted at the next reconciliation. A change made in the repository is the new desired state.

The fix is reviewed, or at least recorded. Somebody else can see what was done and why.

The fix reaches other environments. A cluster edit fixes one cluster; a repository change can be promoted.

The fix survives a rebuild. A cluster recreated from the repository has it. A cluster edit does not exist anywhere.

The cost is latency, and that is the honest objection. A pull request, a review and a reconciliation interval is minutes when seconds matter. The suspend-fix-backport procedure exists precisely for those cases, and treating it as the exception rather than the norm is what keeps the model intact.

The habit to build: when you catch yourself about to run kubectl edit on something managed, ask whether a commit would be fast enough. Usually it is, and the times it is not are rarer than the instinct suggests.

The team norm worth stating out loud: a cluster edit is a decision to be undone later, and the person who makes it owns the backport. Written down, that converts an implicit expectation into an explicit one.

Common mistakes

Self-heal with no suspend procedure. The 3am scenario.

Not bringing an emergency change back into the repository. Silently reverted later.

Leaving a resource suspended. Nothing is managing it and nobody knows.

Alerting on every sync. Nobody reads the channel.

Not alerting on the controller being stopped. The invisible failure.

Declaring replicas on an autoscaled Deployment. Two managers, one field, permanent drift.

Using ignore rules to silence undiagnosed drift. A dashboard that says something untrue.

Pruning before the source is complete. Deletes what it does not know about.

Treating Synced as healthy. It means applied.

Not excluding data-holding resources from pruning. A PVC deleted by a tidy-up.

Reconciliation intervals

The number that determines how fast drift is corrected, and it is a trade rather than a setting to maximise.

Short intervals correct drift quickly and cost API server load, controller CPU, and — at scale — a meaningful amount of both.

Long intervals are cheaper and mean a manual change persists longer, which is sometimes what you want and usually is not.

Webhooks decouple the two concerns. A push-triggered reconciliation makes deployments immediate regardless of the interval, so the interval becomes purely the drift-correction latency rather than the deployment latency. Configuring a webhook is the single change that lets you set a longer interval without anybody noticing.

Set it per workload, not globally. A payment service where drift matters within a minute and a batch job where an hour is fine are different requirements, and applying the strictest one everywhere is how a cluster ends up doing a great deal of work to correct drift that is not there.

Manual reconciliation exists — flux reconcile, Argo CD’s refresh — which removes most of the pressure to set short intervals for the sake of responsiveness.

At scale the interval is a capacity decision. Two hundred applications at one-minute intervals is a different load profile from twenty, and the symptom of getting it wrong is a controller that falls behind — reconciliations queuing, and drift correction taking far longer than the configured interval suggests. Watch reconciliation duration, not just the setting.

Drift in a multi-tenant cluster

The dynamics change when several teams share a cluster and only some of it is reconciled.

Partial adoption is the normal state. Two namespaces under GitOps, six not, and a platform team gradually converting them. That is fine and it means “is the cluster drifted?” has no single answer.

Scope drift reporting to what is managed. A controller watching two namespaces should not report the other six as anything; a dashboard implying the whole cluster is drifted because most of it is unmanaged trains people to ignore it.

Cluster-scoped resources are the awkward case. ClusterRoles, CRDs and admission webhooks affect everybody. A tenant’s controller pruning a cluster-scoped resource another tenant relies on is a genuine risk, and it is the reason application projects should be forbidden from creating them at all.

Tenants drifting their own namespaces is their business, up to the point where it affects shared resources or the cluster’s capacity.

The conversion has an order. Adopt a namespace, report drift without correcting for a period, resolve what appears, then enable correction. Rushing it produces a team whose first experience of GitOps is their work being deleted.

Say who owns what, explicitly. A namespace with no clear owner and a controller reconciling it is a namespace where somebody will be surprised. The ownership map matters more here than the tooling.

Mental model

A reconciler continuously answers one question: does actual state match desired state? Drift is that answer being no. Self-heal is the controller acting on it. Both are the system working, including when the answer is inconvenient.

The operational discipline that follows: when you need the cluster to differ from the repository, say so explicitly by suspending reconciliation — and put it back, the same day. Everything that goes wrong with self-healing clusters is a variation of not doing that.

What you learned

• Drift is a persistent difference the controller did not cause; rollouts, status and defaulted fields are not drift
• Self-heal makes the repository authoritative in fact, and reverts emergency fixes within one interval
• Suspend reconciliation first, fix, record it, bring the repository in line the same day, resume, verify
• Alert on persistent drift and on the controller not reconciling; never on every sync
• The best fix for other-manager drift is omitting the field, not an ignore rule
• Ignore rules used to silence undiagnosed drift produce a dashboard that lies
• Exclude data-holding resources from pruning regardless of confidence
• A stopped controller looks exactly like a healthy cluster with no drift

Exercise

Use a disposable local cluster with a GitOps controller and self-heal enabled.

1. Deploy an application from a repository. Confirm the controller reports it synced.

2. Change the replica count with kubectl scale. Predict: how long until it reverts, and what did the controller report in between?

3. Suspend reconciliation, make the same change, and wait twice the interval. Predict: does it survive?

4. Resume reconciliation. Predict: what happens to your change?

5. Add an HPA managing the same Deployment while the manifest also declares replicas. Predict: what does the controller report, and how often?

6. Remove replicas from the manifest and repeat. Compare.

7. Stop the controller — scale its Deployment to zero. Change something in Git. Predict: what does your monitoring show, and would you notice?

8. Restart it and confirm it catches up.

9. Delete the cluster.

Related lessons

Infrastructure Drift DetectionThe same problem across Terraform and cloud configuration.

GitOps ExplainedThe reconciliation model this operates.

Kubernetes Environment PromotionIntentional change, versus drift.

Version Controlling AnsibleNext cluster — Ansible + Git.