Kubernetes + Git: Manifests, Review and Deployment

Kubernetes is already a reconciliation engine. You declare three replicas; a controller notices there are two and creates one.

That property is why Kubernetes and Git fit together so well, and why the fit is easy to get superficially right and substantively wrong. Putting YAML in a repository is the easy part. Making the repository the thing that actually determines cluster state is the work.

Before you start

• Kubernetes, Git & GitOps — the cluster’s framing
• Containers + Git — image identity, which deployment depends on
• Deploy to Kubernetes with GitHub Actions — the push model

What belongs in the repository

In Git	Not in Git
Deployments, Services, Ingresses	Live resource status
ConfigMaps with non-secret values	Secret values, in any encoding
Kustomize bases and overlays	Rendered output (unless deliberate)
Helm charts and values files	kubeconfig files
CRDs and custom resources	Anything with resourceVersion or uid
NetworkPolicies, RBAC	Generated tokens
Namespace definitions	metadata.creationTimestamp

Two rows need explaining.

Live status does not belong in Git, and the way it gets there is kubectl get -o yaml > deployment.yaml. That output includes status, metadata.uid, resourceVersion and creationTimestamp — fields the API server owns. Committing them produces a manifest that describes one cluster at one moment and applies badly to any other.

Secrets are the hard problem and get their own lesson. The short version: a Kubernetes Secret manifest holds base64, base64 is an encoding rather than encryption, and a data: field in a repository is a plaintext credential with extra steps.

Manifests, Kustomize or Helm

Three ways to get configuration into a repository, and the choice mostly determines how reviewable your diffs are.

Plain manifests are the most reviewable and the least reusable. The diff in a pull request is exactly what will be applied. For a small number of environments this is genuinely fine, and teams move off it earlier than they need to.

Kustomize overlays patches onto a base. No templating language, and kustomize build renders exactly what will be applied, so the diff review story stays good.

Helm templates charts with values. Much better for packaging something other people install; harder to review, because a values change and the manifests it produces are separated by a template.

Most repositories use more than one. Helm for third-party components — an ingress controller, a metrics stack — and Kustomize or plain manifests for their own workloads. That is a reasonable end state rather than an inconsistency.

Caution

Whatever you choose, the reviewable artifact must be the rendered output, not only the input. A one-line values change can alter forty resources. Rendering in CI and posting the diff on the pull request is the single highest-value addition to a Kubernetes repository, and it is the thing that makes Helm reviewable at all.

Rendering the diff

The practice that makes Kubernetes pull requests meaningful.

# What the change produces
kustomize build overlays/production > /tmp/new.yaml

# What is currently declared
git stash
kustomize build overlays/production > /tmp/old.yaml
git stash pop

diff -u /tmp/old.yaml /tmp/new.yaml

In CI, render both the base branch and the head, and post the difference as a pull request comment. What that gives a reviewer:

The actual resources changing, not the templating input.

Fields they did not expect, which is where the surprises live — a values change that also alters a label used by a selector, for example.

A count. “12 resources changed” at the top, with the detail collapsed underneath.

A comparison against the cluster is the stronger version. kubectl diff -f - shows the difference between the rendered manifests and what is actually running, which catches drift as well as the intended change — a diff containing changes the pull request did not make means somebody altered the cluster by hand. It needs cluster read access from CI, which is a real trade-off worth weighing rather than dismissing, and it is the closest Kubernetes has to terraform plan.

CI validation

A ladder, cheapest first.

YAML parses. Trivial and catches the indentation error.

Schema validation. Does this match the Kubernetes API schema for the target version? Catches a misspelled field, which YAML parsing does not — replicas: 3 under the wrong indentation is valid YAML and an invalid Deployment.

Rendering succeeds. kustomize build or helm template completes.

Policy. Resource limits present, no latest image tags, no privileged containers, required labels, no bare pods — policy as code. Running the same policies in CI and at admission is worth the duplication: CI gives fast feedback on a pull request, and admission catches anything that did not come through CI.

A server-side dry run. kubectl apply --dry-run=server sends the manifests to a real API server, which runs admission controllers and validation without persisting anything. This is the strongest check available and it needs cluster access.

The diff. Posted for a human.

Note the ordering: everything before the dry run is offline and needs no credentials. That is deliberate — a pull request from an untrusted source should get as far as it can without touching a cluster.

Which Kubernetes version

Schema validation is version-specific, and the version matters.

Pin the version CI validates against, and make it the version your clusters actually run. Validating against a newer version passes manifests your cluster will reject; validating against an older one fails on fields that are now valid.

API versions get removed. apiVersion values graduate — beta to stable — and the beta version is eventually removed. A manifest using a removed API applies fine until the cluster is upgraded and then does not apply at all. Deprecation detection in CI catches this before the upgrade rather than during it.

Upgrade the validation version before upgrading the cluster. Then CI tells you which manifests need changing while everything still works, rather than after the upgrade when nothing does.

Namespaces and boundaries

Namespaces are the primary isolation unit and the natural repository boundary.

One directory per namespace maps cleanly to CODEOWNERS, to RBAC, and to how people think about ownership.

Do not put namespace: in every manifest if you use Kustomize — set it once in the kustomization and let it apply. Manifests without a hard-coded namespace are reusable across environments.

Namespace-scoped and cluster-scoped resources are different. ClusterRoles, CRDs, StorageClasses and ValidatingWebhookConfigurations affect everything. They belong in a separate directory with stricter ownership, because a mistake in a cluster-scoped resource has no namespace boundary to contain it.

RBAC belongs in the repository and deserves review by somebody who actually reads RBAC. A ClusterRoleBinding granting cluster-admin is four lines, contains no alarming words, and is indistinguishable at a glance from a binding that grants read access to one namespace.

Push or pull

The architectural decision, covered fully in GitOps explained and worth stating here.

Push: a pipeline holds cluster credentials and runs kubectl apply. Simple, imperative, and the credentials live outside the cluster.

Pull: an agent inside the cluster reads the repository and reconciles continuously. Nothing outside needs write access, and manual changes are detected.

Neither is universally right. Push handles ordered operations — a migration that must complete before the new version starts — which reconciliation models express badly. Pull handles convergence and self-healing that push cannot express at all.

What you lose with push alone: a defined behaviour when somebody changes the cluster by hand. With push, a manual change persists silently until the next deploy, which may be weeks. That gap is the main argument for adding a reconciler.

Server-side apply and field ownership

A mechanism worth understanding because it explains several confusing behaviours.

Server-side apply tracks which actor owns which field. When you apply a manifest, the API server records that your field manager set those specific fields. Another actor changing a field you own produces a conflict rather than a silent overwrite.

Why it matters for a Git workflow: it is what lets an autoscaler own spec.replicas while your repository owns everything else in the same Deployment. Without it, every apply would reset the replica count and fight the autoscaler.

The practical consequence: omit fields you do not want to own. A manifest that specifies replicas: 3 claims that field. A manifest that omits it lets whoever else manages it — an HPA, or a GitOps controller configured to ignore it — keep control.

Conflicts surface as errors. “Apply failed with conflicts” means another manager owns a field you are trying to set. The honest resolutions are to stop setting it or to agree that you own it; forcing the conflict resolves it by taking ownership, which is sometimes right and should be a decision rather than a reflex.

GitOps controllers use this. Both Argo CD and Flux can apply server-side, and their drift detection interacts with field ownership — a field owned by another manager is not drift, it is somebody else’s business. Understanding this saves a great deal of confusion about why a controller reports a resource as synced when a field visibly differs from the manifest.

Labels and selectors

An area where a small mistake produces a large and confusing failure.

A Deployment’s spec.selector is immutable. Changing it on an existing Deployment is rejected by the API server, and the only path forward is deleting and recreating the Deployment — which means downtime, and which is why this is worth getting right initially.

Kustomize’s labels field can add labels to selectors, which is convenient and is the mechanism by which somebody accidentally changes an immutable field. A change that adds a common label across a base can alter selectors in every Deployment it touches, and the apply fails on all of them.

Recommended labels — app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/version, app.kubernetes.io/part-of, app.kubernetes.io/managed-by — are a convention worth following. Tooling reads them, and consistency across a cluster makes ad-hoc queries possible.

Keep the selector minimal and stable. Two labels that will never change. Put everything descriptive in metadata.labels, which is mutable, rather than in the selector, which is not.

The version label should not be in the selector. A selector matching on version means every version change is an immutable-field change. This is a mistake people make once.

What Git does not contain

Worth being precise, because “Git is the source of truth” gets repeated loosely.

Git holds declared desired state. The cluster holds a great deal more:

Status. Every resource has one, written by controllers.

Generated fields. UIDs, resource versions, timestamps, defaulted values the API server filled in.

Dynamically managed resources. Pods created by a Deployment. Endpoints maintained by a Service controller. Nothing declares these individually.

Autoscaler decisions. A replica count that an HPA is managing is not the number in your manifest, and committing the current number produces a fight between your repository and the autoscaler.

Anything created outside the repository. Which is the definition of drift.

The practical rule: declare what you intend, not what currently exists. Those differ, and the gap is the reconciler’s job rather than a problem with your manifests.

Reviewing a Kubernetes pull request

What to look at, in the order that catches the most for the least effort.

The rendered diff, not the source diff. Everything below assumes you are reading what will actually be applied.

Resources being deleted. A rendered diff that removes a resource means something will be pruned. If the reviewer did not expect that, stop.

Immutable fields. Selectors, spec.serviceName on a StatefulSet, most of a Job’s spec. A change to any of these means delete-and-recreate rather than an update, and for a StatefulSet that means the pods go away.

Resource requests and limits. A change here alters scheduling. Removing them entirely puts the workload in a lower quality-of-service class, which is the kind of change that looks harmless and shows up as evictions under pressure.

Anything with hostPath, hostNetwork, privileged, or added capabilities. Each is an escape from container isolation and each needs a reason.

RBAC. A ClusterRoleBinding is four lines and can grant everything. Read the verbs and the resources, not just the name.

Image references. A tag where a digest should be, or a change from a digest to a tag.

Probes. A readiness probe removed or a timeout increased affects rollout behaviour, and a liveness probe that is too aggressive causes restart loops under load.

Replica counts on autoscaled workloads. If an HPA manages it, the manifest should not set it.

Eight items, most of which are visible in the rendered diff without reading carefully. The ones that require actual thought are RBAC and immutable fields, and those are the two most worth having a code owner for.

ConfigMaps and rollouts

A behaviour that surprises people and has a standard solution.

Changing a ConfigMap does not restart the pods that mount it. The Deployment did not change, so nothing rolls. Pods that read the configuration at startup keep the old values indefinitely — until something unrelated causes a restart, at which point a subset of pods pick up a change made weeks earlier.

That is a genuinely bad failure mode: the change appears to have been applied, the cluster reports everything healthy, and the behaviour changes later at a moment nobody connects to it.

The standard fixes:

Generated names. Kustomize’s configMapGenerator appends a content hash to the name, so a content change produces a new ConfigMap name, which changes the Deployment that references it, which triggers a rollout. This is the cleanest mechanism and it is a good reason to use Kustomize even in a mostly-Helm repository.

A checksum annotation. Helm charts conventionally put a hash of the config into a pod template annotation, which changes the pod spec and triggers a rollout for the same reason.

Applications that watch for changes avoid the problem entirely by re-reading configuration at runtime. Best where available and not something you can retrofit onto an arbitrary application.

The one to avoid is remembering to restart the Deployment manually after a ConfigMap change. It works when somebody remembers and it is exactly the kind of step that gets skipped under time pressure — and its absence is invisible until much later.

The same applies to Secrets mounted as files or environment variables, with the additional wrinkle that a Secret rotated by an external operator changes without any commit at all.

Common mistakes

Committing kubectl get -o yaml output. Status, UIDs and timestamps that describe one cluster at one moment.

Reviewing template input rather than rendered output. A one-line change can alter forty resources.

Plaintext Secrets in the repository. Base64 is not encryption.

Committing a kubeconfig. A cluster credential in a repository.

Validating against the wrong Kubernetes version. Passes manifests the cluster rejects.

Cluster-scoped resources mixed with namespaced ones. No boundary to contain a mistake.

Committing a replica count an HPA manages. The repository and the autoscaler fight.

Hard-coded namespaces in every manifest. Not reusable across environments.

Assuming push deployment gives you drift detection. It does not; nothing is watching.

Getting started, from nothing

A cluster whose configuration lives in people’s shell history has a migration ahead of it, and the order matters.

1. Export what exists, then clean it. kubectl get -o yaml is the starting point and its output is not a manifest — strip status, metadata.uid, resourceVersion, creationTimestamp, generation, and the kubectl.kubernetes.io/last-applied-configuration annotation. Tools exist for this; doing it by hand once teaches you what the API server owns.

2. Get one namespace into the repository. Not the whole cluster. Pick something with low stakes and few resources.

3. Verify with kubectl diff. The cleaned manifests should produce an empty or near-empty diff against the live cluster. A large diff means the export was wrong, and finding that out on a low-stakes namespace is the point of step 2.

4. Apply from the repository and confirm nothing changes. This is the moment the repository becomes authoritative for that namespace.

5. Add CI validation. Schema, rendering, policy. Now further changes go through review.

6. Repeat per namespace, most important last.

7. Remove direct write access once a namespace is fully managed. Until this step, the repository is a description rather than a control — anybody can still change the cluster and nothing will notice.

8. Then consider a reconciler. Argo CD or Flux once the repository actually reflects the cluster. Pointing a reconciler at an incomplete repository means it prunes things you needed.

Step 8 is the one to resist rushing. A GitOps controller applied to a repository that does not yet describe the cluster completely will, if pruning is enabled, delete every resource it does not know about — which is most of them.

Mental model

Kubernetes reconciles the cluster toward what its API server holds. Git is where you decide what that should be. The repository declares intent; the cluster reports reality; something has to carry one to the other.

Whether that something is a pipeline pushing or an agent pulling is the next lesson’s subject. Either way, the repository’s job is to hold a clear, renderable, reviewable statement of intent — and to hold nothing that only the cluster can know.

What you learned

• Declare intent; never commit status, UIDs, timestamps or other API-server-owned fields
• Plain manifests, Kustomize and Helm trade reusability against reviewability
• The reviewable artifact is the rendered output, and rendering the diff in CI is the highest-value addition
• kubectl diff compares rendered manifests against the live cluster — the closest thing to a plan
• Validate against the Kubernetes version your clusters actually run, and upgrade validation first
• Cluster-scoped resources need stricter ownership than namespaced ones
• Push deployment gives you no drift detection; that is what a reconciler adds

Exercise

Use a disposable local cluster — kind or minikube. No production cluster, no real credentials.

1. Create a repository with a Deployment and a Service as plain manifests. Apply them with kubectl apply -f.

2. Run kubectl get deployment -o yaml > exported.yaml. Diff it against your original. Predict: how many fields did the API server add?

3. Change the replica count in your manifest and run kubectl diff -f deployment.yaml. Predict: what does it show?

4. Change the replica count with kubectl scale instead, without touching Git. Run kubectl diff again. Predict: does it detect the drift?

5. Apply your manifest again. Predict: does the manual change survive?

6. Add a CI step that runs schema validation against a specific Kubernetes version. Introduce a misspelled field and confirm it fails.

7. Add a --dry-run=server step and compare what it catches that offline validation did not.

8. Delete the cluster.

Related lessons

GitOps ExplainedThe four principles, and what qualifies.

Kubernetes Repository StructureLaying out clusters, apps and environments.

Kustomize + GitOverlays, patches and reviewable diffs.

Kubernetes Secrets and GitOpsThe problem base64 does not solve.