GitOps Explained: Git, Kubernetes & Continuous Reconciliation

GitOps has been stretched to describe any deployment involving a repository. That is a shame, because the actual idea is specific and useful, and the loose usage costs you the vocabulary to describe what it adds.

This lesson uses the term precisely, following the OpenGitOps principles — and is equally precise about what does not qualify, including several things that are perfectly good architecture.

Before you start

• Kubernetes + Git — configuration in a repository
• Kubernetes, Git & GitOps — the cluster’s framing

The four principles

OpenGitOps publishes them as version 1.0.0. Verbatim:

1. Declarative — “A system managed by GitOps must have its desired state expressed declaratively.”

2. Versioned and Immutable — “Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.”

3. Pulled Automatically — “Software agents automatically pull the desired state declarations from the source.”

4. Continuously Reconciled — “Software agents continuously observe actual system state and attempt to apply the desired state.”

Two things are worth noticing immediately, because both are commonly misread.

The principles say “the source”, not “Git”. An OCI registry holding versioned, immutable artifacts satisfies principle 2 perfectly well — which is exactly what Flux’s OCIRepository source is built on. GitOps is not defined by the storage being Git, despite the name.

Principles 3 and 4 are what most “GitOps” pipelines lack. A workflow that runs kubectl apply on merge satisfies 1 and 2 completely. It does not pull, and it does not continuously reconcile — it acts once, on a trigger, and stops.

What qualifies

Setup	1. Declarative	2. Versioned	3. Pulled	4. Reconciled	GitOps?
Manifests in Git, kubectl apply by hand	Yes	Yes	No	No	No
Manifests in Git, CI applies on merge	Yes	Yes	No	No	No
Manifests in Git, Argo CD or Flux syncing	Yes	Yes	Yes	Yes	Yes
Terraform in Git, CI applies on merge	Yes	Yes	No	No	No
Helm chart in an OCI registry, Flux reconciling	Yes	Yes	Yes	Yes	Yes
Ansible playbooks in Git, run on merge	Partly	Yes	No	No	No

The two rows in bold are the ones that matter.

CI applying manifests on merge is a perfectly good deployment pipeline. It is not GitOps, and saying so is not a criticism — it is the difference that lets you reason about what a reconciler would add.

Terraform in Git with CI is Infrastructure as Code with CI/CD. Also good, also not GitOps. Terraform is declarative and versioned; nothing pulls it, and nothing continuously reconciles. Some tooling adds a reconciliation loop on top, at which point the question becomes interesting again.

Caution

The habit worth breaking: using “GitOps” to mean “we keep our YAML in Git”. If that is what the word means, there is no word left for the thing with the reconciliation loop — and the reconciliation loop is the part that changes how you operate.

Desired and actual state

The distinction the whole model rests on.

Desired state is what the repository says should be true. Three replicas, this image digest, this configuration.

Actual state is what the cluster reports. Two replicas because one is pending, an image somebody patched by hand, a config value edited during an incident.

They diverge constantly, and most divergence is benign — a pod restarting, a rollout in progress. The interesting divergence is the kind that persists.

Reconciliation is the loop that compares them and acts.

The reconciliation loop

A cycle: the agent reads desired state from the source; observes actual state from the cluster; compares them; applies the difference; and repeats continuously.

“Continuously” is doing the work in that diagram. A loop that runs on every merge is a pipeline. A loop that runs every few minutes regardless of whether anything changed is a reconciler, and the difference shows up when somebody edits the cluster directly.

What reconciliation actually gives you

Four properties, and they are why teams adopt this.

Self-healing. Somebody deletes a Deployment. Within a reconcile interval it comes back, because the desired state still says it should exist. Nobody had to notice, nobody had to be paged, and the recovery is identical whether the cause was a mistake, a script, or a controller in another system behaving badly.

Drift detection as a permanent state. The controller always knows whether the cluster matches the repository, and reports it as a status you can alert on. That is a continuously answered question rather than one somebody has to remember to ask, which in practice means it gets answered rather than not.

The repository becomes authoritative in fact, not just in policy. With a pipeline, “all changes go through Git” is a rule people follow. With reconciliation, a change that does not go through Git gets reverted — the rule enforces itself.

No inbound cluster credentials. The agent runs inside and reads outward. Nothing external needs write access to the cluster API, which removes a standing credential and a network path.

That third property is the one that changes team behaviour. It converts a convention into a mechanism, and conventions decay while mechanisms do not.

Pull and push

	Push (pipeline applies)	Pull (agent reconciles)
Cluster credentials	Held externally by CI	None external
Network direction	Inward	Outward
When state changes	On a trigger	Continuously
Manual change	Persists silently	Detected, optionally reverted
Ordered, imperative steps	Natural	Awkward
Multi-cluster	Credentials per cluster	An agent per cluster
Failure visibility	The pipeline run	Continuous status
Anything outside the cluster	Easy	Needs an operator

Push is better for ordered operations. A database migration that must complete before the new version starts is a sequence, and reconciliation models express sequences badly. Job ordering, sync waves and hooks exist to paper over this and they are more awkward than a pipeline step.

Pull is better for convergence. Anything where the correct behaviour is “make it look like this, and keep it looking like this”.

Most mature platforms run both, deliberately: a pipeline for migrations and one-off operations, a reconciler for workload state. The failure is running both with no rule about which is authoritative for a given resource, so they fight — the pipeline applies something and the reconciler reverts it every few minutes.

Git as source of truth — precisely

The phrase is used loosely enough to be misleading. What is actually true:

Git holds the desired state, and only the parts a reconciler is pointed at.

Git does not hold the actual state. The cluster does, and it contains a great deal Git never will: pod status, generated identifiers, defaulted fields, endpoint lists, autoscaler decisions, and everything the API server computes.

Git does not hold secrets, in any workable design.

Git does not hold runtime data. Databases, queues, object stores.

Git does not hold what other systems know. Cloud provider state, DNS held elsewhere, third-party configuration.

So a more accurate phrase is: Git is the source of truth for declared intent. The reconciler’s entire purpose presupposes that intent and reality differ — otherwise there would be nothing to reconcile.

Why the precision matters practically. A team that believes Git holds everything will look in the repository during an incident and conclude the system is fine. The repository being correct and the system being broken is an entirely ordinary situation, and the reconciler’s status is where you find out — not the repository’s contents.

The related error is treating the reconciler’s “synced” status as a health check. Synced means the declared resources were applied successfully. The application behind them can be crash-looping.

Sources other than Git

Principle 2 requires versioned, immutable storage with complete history. Git provides that and is not the only thing that does.

OCI registries. Manifests packaged as an OCI artifact, tagged and addressable by digest. Flux’s OCIRepository reads these natively. The properties are arguably stronger than Git’s: a digest is immutable in a way a tag is not, and a Git tag can be moved while a digest cannot.

Why a team would choose this. The cluster does not need Git access, which matters in restricted networks. Artifacts can be signed and verified as part of the pull. And it fits naturally where a build pipeline already publishes to a registry — the manifests travel the same path as the images.

What you give up. The commit history and the pull request as the human interface. In practice the usual arrangement keeps Git as the place people change things, with a pipeline packaging the result into an OCI artifact that the cluster consumes — Git for humans, OCI for machines.

Other sources exist too: object storage buckets are supported by some controllers, with versioning providing the immutability.

The distinction to hold on to: the principles are about properties, not products. Versioned, immutable, complete history, pullable by an agent. Anything with those properties can be the source.

Rollback, honestly

The claim: with GitOps, rollback is git revert.

Where it is true. Stateless workloads. Revert the commit that changed an image digest, the controller reconciles, the previous version is running. Fast, auditable, and it works.

Where it is not. Anything with state or an external side effect:

A database migration ran. Reverting the manifest gives you old code against a migrated schema. The application may not start; worse, it may start and misbehave.

A resource was deleted. Reverting recreates a resource with the same name. The data in the deleted PersistentVolume is gone.

An external system was called. A webhook fired, a message was published, a third party was notified. Git has no record and no undo.

A CRD was removed. Removing a CustomResourceDefinition deletes every custom resource of that kind, cluster-wide, immediately. Restoring the CRD gives you back the type and none of the objects. This is among the most destructive single-line changes possible in a Kubernetes repository, and it looks like deleting one file.

Danger

Reverting a Git commit does not guarantee infrastructure has been restored safely. It restores the declaration. Whether the system returns to its previous behaviour depends entirely on whether the operations in between were reversible — and stateful ones are not. Every runbook that says “roll back by reverting the commit” should say which changes that actually covers.

The honest formulation: Git gives you a fast, auditable way to restore a previous desired state. Operational recovery is a separate question that depends on what happened in between.

What GitOps does not solve

Worth stating, because the model is sometimes oversold.

It does not make your configuration correct. A reconciler faithfully applies a bad manifest, continuously.

It does not handle secrets. Desired state is in a repository; secrets cannot be. That is a genuinely hard problem with its own lesson.

It does not order things well. Sequences need workarounds.

It does not manage what it cannot see. Resources outside the reconciler’s scope drift freely, and the scope is narrower than people assume — a controller watching one namespace knows nothing about the rest of the cluster.

It does not remove the need for observability. A controller reporting “synced” means the manifests were applied, not that the application works.

It does not make emergency changes disappear. Somebody will change the cluster at 3am. The model’s contribution is that the change is detected — and, if self-heal is on, reverted, which is a problem you must plan for.

It is not free. A controller to run, upgrade and debug; a repository structure to design; a new failure mode where the controller itself is unhealthy and nothing is reconciling while everything looks fine. Teams underestimate the last one — a reconciler that has silently stopped is indistinguishable from a system with no drift, right up until somebody makes a change and nothing happens.

Adopting it

1. Get manifests into a repository first, and make sure they describe the cluster accurately. kubectl diff should be empty.

2. Remove direct write access so the repository is actually authoritative. Skipping this means running a reconciler alongside people who can still change things by hand.

3. Install a controller in a non-production cluster. Argo CD or Flux.

4. Start with sync but not pruning, and not self-heal. Let it apply, and watch what it reports as out of sync. This is the safest possible first configuration.

5. Read what it reports for a week. You will find drift you did not know about. That is the point, and it is why this step exists.

6. Enable pruning once you are confident the repository is complete. Pruning deletes resources it does not know about, and an incomplete repository plus pruning is an outage.

7. Enable self-heal last, and have a documented way to suspend it during an incident.

8. Then production, one namespace at a time.

Step 6 is the dangerous one. A controller with pruning enabled, pointed at a repository that does not describe everything in the namespace, deletes the difference.

Trust boundaries

The security argument for pull, stated precisely, because it is the part most often oversimplified.

In a push model, the CI system holds credentials that grant write access to the cluster API. Those credentials exist as a secret in a system outside the cluster. Anything that compromises the CI system — a malicious dependency in a workflow, a stolen token, a workflow file somebody modified — reaches the cluster.

In a pull model, the agent runs inside the cluster with a ServiceAccount. Nothing outside holds cluster write credentials. The cluster reaches out to read a repository, which needs only read access.

What that actually changes. The number of places a cluster credential exists drops to zero outside the cluster. The network path is outbound rather than inbound. A compromised CI system can still poison the repository — but that is a change that goes through review and appears in history, rather than a direct write nobody sees.

What it does not change. The agent itself is now a highly privileged component: it can apply anything in its scope. Compromising the agent is equivalent to compromising the credentials you removed. You have not eliminated the privilege; you have moved it inside the trust boundary and made it a Kubernetes RBAC problem rather than a secret-management problem.

Scope the agent’s RBAC. A reconciler with cluster-admin across every namespace has a blast radius equal to the cluster. Per-namespace agents, or one agent with per-application RBAC, is more work and considerably better.

The repository’s write access becomes the control. If anybody can merge to the branch the agent watches, anybody can change the cluster. Branch protection, required reviews and CODEOWNERS are load-bearing security controls in this model, not process preferences.

Verify what you pull, where you can. Both major controllers support verifying signed commits or signed OCI artifacts. That closes the gap where somebody with repository write access pushes directly, bypassing review.

GitOps beyond Kubernetes

The principles are not Kubernetes-specific, and it is worth knowing where the model does and does not transfer.

It works wherever there is a controller that can reconcile. Kubernetes has one built in, which is why it is the natural home. Crossplane and similar projects extend the same loop to cloud resources by representing them as Kubernetes objects, which is a real way to bring provisioning under the same model.

It does not transfer to Terraform by adding a pipeline. Terraform is declarative and versioned; without an agent pulling and continuously reconciling, two principles are missing. Tooling exists that adds a reconciliation loop around Terraform, and where it does the question becomes legitimate again.

It does not transfer to Ansible. Ansible is imperative, push-based and has no agent. Playbooks in Git with a pipeline is good practice and it is not GitOps, and the Ansible cluster says so directly.

Where the model genuinely does not fit: anything whose correct behaviour is a sequence rather than a state. Database migrations, one-off data corrections, coordinated multi-system cutovers. Trying to express these as desired state produces elaborate workarounds, and a pipeline is the honest tool.

The useful test for any system: can an agent read a declaration, observe the system, and act to close the gap — repeatedly, safely, without a human deciding the order? Where the answer is yes, the model fits. Where the order matters, it does not.

Common mistakes

Calling any Git-backed deployment GitOps. Costs you the word for the thing with reconciliation.

Enabling pruning before the repository is complete. Deletes what it does not know about.

Enabling self-heal without a suspend procedure. The controller reverts an emergency fix mid-incident.

Running a pipeline and a reconciler against the same resources. They fight.

Treating git revert as guaranteed rollback. It restores a declaration, not a system.

Committing autoscaler-managed replica counts. The repository and the HPA fight.

Assuming “synced” means “working”. It means the manifests were applied.

Not planning for the controller being unhealthy. A new single point of failure.

The audit trail

An underrated property, and one that arrives free.

Every change to declared state is a commit with an author, a timestamp, a message, and — if the repository is configured properly — a pull request with a review and an approval attached.

Every reconciliation is an event the controller records: what it applied, when, and whether it succeeded.

Together those answer questions that are otherwise expensive:

“When did this change?” — the commit that introduced it.

“Who approved it?” — the pull request.

“Why?” — the linked issue, if the team links issues.

“What was it before?” — the previous commit.

“Was it actually applied, and when?” — the controller’s history.

“Has anything changed that did not go through this?” — the drift the controller reports.

That last question is the one no push pipeline can answer, and it is frequently the one an auditor actually asks.

The caveat worth stating: this is an audit trail of declarations, not of the system. It records what was intended and when it was applied. It does not record what a pod did at runtime, and it does not capture a change somebody made and the controller silently reverted before anybody looked. Combining the controller’s events with cluster audit logs is what gives you the full picture, and only the first half comes for free.

Retention matters. A controller’s event history is typically short. If the audit trail is something you rely on, the durable part is Git plus your cluster’s audit log, and both need a retention policy somebody has actually set.

Mental model

GitOps is a control loop whose set point lives in version control. The repository declares what should be true; an agent inside the system continuously measures what is true and acts to close the gap.

Everything follows. Declarative because a control loop needs a set point rather than a sequence of actions. Versioned because you need to know what the set point was. Pulled because the agent must be able to read it without anything outside holding credentials. Continuously reconciled because a loop that runs once is not a loop.

What you learned

• The four OpenGitOps v1.0.0 principles: declarative, versioned and immutable, pulled automatically, continuously reconciled
• The principles say “the source” — an OCI registry qualifies; Git is not required
• CI applying manifests on merge satisfies two of four and is not GitOps
• Reconciliation gives self-healing, permanent drift detection, an authoritative repository and no inbound credentials
• Push suits ordered operations; pull suits convergence; mature platforms use both deliberately
• git revert restores a declaration, not a system — migrations, deletions and external calls do not reverse
• Enable sync first, pruning once the repository is complete, self-heal last

Exercise

Use a disposable local cluster — kind or minikube. No production cluster.

1. Create a repository with a Deployment. Apply it with kubectl apply -f. Note that no agent is involved.

2. Delete the Deployment with kubectl delete. Predict: does anything bring it back?

3. Install a GitOps controller and point it at the repository, with sync enabled and self-heal off.

4. Delete the Deployment again. Predict: what does the controller report, and does it act?

5. Enable self-heal. Delete it again. Predict: how long until it returns?

6. Change the replica count with kubectl scale — a manual change rather than a deletion. Predict: is that reported as drift?

7. Add a resource to the cluster that is not in the repository. Enable pruning. Predict: what happens to it?

8. Revert the commit that added the Deployment. Predict: what does the controller do, and would this have restored data if the workload had a volume?

9. Delete the cluster.

Practise it

Lab: ship an infrastructure change through a pull request takes a change from manifest to reconciliation, with the review checklist that makes it reviewable.

Related lessons

Argo CDApplications, sync, self-heal and pruning.

FluxSources, Kustomizations and HelmReleases.

GitOps Drift and ReconciliationThe operational model, including emergency changes.

Kubernetes Repository StructureWhat the controller points at.