Kustomize + Git: Bases, Overlays and Reviewable Diffs

Kustomize’s advantage over templating is that its output is still YAML you can read, and its input is still YAML you can review.

That property is worth more in a GitOps repository than in almost any other setting, because the pull request diff and the thing the cluster receives are separated by one deterministic command rather than by a templating language.

Before you start

• Kubernetes + Git — what belongs in the repository
• Kubernetes Repository Structure — where bases and overlays live

Bases and overlays

apps/api/
├── base/
│   ├── kustomization.yaml
│   ├── deployment.yaml
│   ├── service.yaml
│   └── configmap.yaml
└── overlays/
    ├── dev/
    │   ├── kustomization.yaml
    │   └── replicas.yaml
    ├── staging/
    └── production/
        ├── kustomization.yaml
        ├── replicas.yaml
        └── resources.yaml

The base holds the resources. Overlays patch them per environment.

base/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml

labels:
  - pairs:
      app.kubernetes.io/name: api
    includeSelectors: true

overlays/production/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: production

resources:
  - ../../base

patches:
  - path: replicas.yaml
  - path: resources.yaml

images:
  - name: api
    newName: ghcr.io/example-org/api
    digest: sha256:0000000000000000000000000000000000000000000000000000000000000000

Render it and read the result:

kustomize build apps/api/overlays/production

That output is exactly what the cluster receives. No templating, no runtime resolution — which is why the diff review story works.

Current syntax, and what is deprecated

Kustomize v5 deprecated several fields. They still function in kustomize.config.k8s.io/v1beta1 and will not be included in a future v1 Kustomization API, so new configuration should use the replacements.

Deprecated	Use instead
patchesStrategicMerge	patches
patchesJson6902	patches
bases	resources
vars	replacements
commonLabels	labels

kustomize edit fix migrates a kustomization automatically, which is the least error-prone way to update an existing repository.

The unified patches field takes both strategic-merge and JSON 6902 patches, distinguished by their content, with an optional target selector:

patches:
  # Strategic merge, target inferred from the patch's own metadata
  - path: replicas.yaml

  # Inline JSON 6902, with an explicit target
  - target:
      kind: Deployment
      name: api
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits/memory
        value: 2Gi

  # Applied to everything matching the selector
  - target:
      kind: Deployment
      labelSelector: app.kubernetes.io/part-of=platform
    patch: |-
      - op: add
        path: /spec/template/spec/priorityClassName
        value: high-priority

The third form — a patch targeting a label selector rather than a name — is what makes patches more capable than what it replaced, and it is the reason to migrate beyond the deprecation.

Caution

labels with includeSelectors: true adds labels to a Deployment’s selector, and a Deployment’s selector is immutable. Adding a common label to a base with selectors enabled changes an immutable field on every Deployment it touches, and the apply fails on all of them. The recovery is deleting and recreating the Deployments, which means downtime. Set includeSelectors: true once, at the beginning, and never change the labels it includes.

Patching, in practice

Two mechanisms with different strengths.

Strategic merge is readable. You write a fragment of the resource with the fields you want changed:

overlays/production/replicas.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 6

Kustomize matches on kind and name and merges. This is the right default: the patch reads like the resource, and a reviewer can see what it does without knowing the mechanism.

The list problem. Strategic merge on lists behaves according to a merge key declared in the Kubernetes API’s own schema, and where there is no sensible key the behaviour surprises people. Container lists merge by name; environment variable lists merge by name; but args and command have no merge key and are replaced wholesale. A patch intending to add one argument silently removes the rest, the Deployment rolls, and the application starts with a command line nobody wrote.

JSON 6902 is precise. It addresses a specific path and states an operation:

  - target:
      kind: Deployment
      name: api
    patch: |-
      - op: add
        path: /spec/template/spec/containers/0/args/-
        value: --feature-flag=new-routing

Harder to read, exact about what it does, and the right tool for list manipulation and for removing a field entirely (op: remove).

Choose by what you are doing: replacing values, strategic merge; manipulating lists or removing fields, JSON 6902.

Images and promotion

The images field is what makes promotion a one-line change.

images:
  - name: api                              # As referenced in the base
    newName: ghcr.io/example-org/api        # Where it actually lives
    digest: sha256:0000000000000000000000000000000000000000000000000000000000000000

Use digest, not newTag, for anything deployed. The reasoning is that a tag can be reassigned and a digest cannot. What staging tested and production runs should be the same content.

The base should reference a bare name. image: api in the base, with each overlay supplying the real registry and version. That way the base is genuinely environment-independent, and a base that names a registry is a base you cannot reuse when the registry changes.

Promotion is then a diff of one line, in one file, per environment:

images:
  - name: api
    newName: ghcr.io/example-org/api
    digest: sha256:1111111111111111111111111111111111111111111111111111111111111111
    digest: sha256:2222222222222222222222222222222222222222222222222222222222222222

Keep the version in a comment beside it. A digest tells a reviewer nothing; # v2.4.1 tells them everything they need to decide.

kustomize edit set image makes that edit programmatically, which is what an automated promotion pull request runs.

ConfigMaps and rollouts

configMapGenerator solves a problem plain manifests have.

configMapGenerator:
  - name: api-config
    files:
      - config/application.yaml
    literals:
      - LOG_LEVEL=info

Kustomize appends a hash of the content to the generated name — api-config-7d9f2h4k8m — and rewrites every reference to it. A content change produces a new name, which changes the Deployment that references it, which triggers a rollout.

Why that matters: changing a ConfigMap in place does not restart the pods that mount it. Without generated names, a configuration change appears to apply and takes effect at some unpredictable later point when pods happen to restart. That is a genuinely bad failure mode, and this is the cleanest fix for it.

generatorOptions: disableNameSuffixHash: true turns it off, and should be used only where something outside Kustomize references the ConfigMap by a fixed name — at which point you have reintroduced the rollout problem and need another mechanism.

secretGenerator exists and generates Secrets the same way. It does not make the values secret: literals in a kustomization are plaintext in the repository, and files: references a file that is also in the repository. GitOps secrets covers what to do instead.

Components

For cross-cutting configuration that applies to some overlays and not others.

components/monitoring/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
  - target:
      kind: Deployment
    patch: |-
      - op: add
        path: /spec/template/metadata/annotations/prometheus.io~1scrape
        value: "true"

overlays/production/kustomization.yaml

components:
  - ../../../components/monitoring
  - ../../../components/high-availability

Components are composable in a way bases are not: an overlay can include several, and each contributes patches. The right tool for “production and staging get monitoring annotations, dev does not”.

The ~1 in that path is JSON Pointer escaping for a literal / in a key — prometheus.io/scrape becomes prometheus.io~1scrape. It catches everybody once.

Replacements

The successor to vars, and more capable than what it replaced.

The problem it solves: one value that must appear in several resources. A Service name referenced in a ConfigMap, a namespace referenced in an RBAC binding, an image tag referenced in an annotation. Duplicating it means it drifts.

replacements:
  - source:
      kind: Service
      name: api
      fieldPath: metadata.name
    targets:
      - select:
          kind: ConfigMap
          name: api-config
        fieldPaths:
          - data.UPSTREAM_SERVICE

The value is read from one resource and written into others at build time. Unlike vars, it can target arbitrary field paths and it works after other transformations have run, which is what makes it usable with name prefixes and generated names.

Use it sparingly. A repository with fifteen replacements is one where the rendered output cannot be predicted from reading the input, which undoes the main advantage Kustomize has over templating. Two or three, for genuinely shared values, is reasonable.

The common legitimate case is a generated ConfigMap or Secret name — the hash means you cannot write it literally, so a replacement is the only way for another resource to reference it correctly.

Nesting and composition

How far to nest, and where it stops being helpful.

Two levels is comfortable. A base and an overlay. Somebody reading the overlay can hold the base in their head.

Three levels is the practical limit. A shared base, a per-application base that patches it, and a per-environment overlay. Beyond this, predicting the output from reading requires a build.

Remote bases work and cost something. A resources entry can be a Git URL:

resources:
  - github.com/example-org/platform-bases//deployment?ref=v2.1.0

The same module versioning argument applies: pin the ref to a tag, never a branch. A branch reference means the platform team merging something changes what your overlay renders, with no commit in your repository.

Remote bases also mean a network fetch at build time, which affects CI reliability and means an offline kustomize build fails. Vendoring — copying the base in — trades upstream updates for determinism, and for a base that changes twice a year that is usually the right trade.

The composition failure to avoid: a base that exists only to be patched, whose own resources are never deployed as written. That is a template with extra steps, and the honest version is either fewer levels or Helm.

Reviewing the diff

The practice that makes any of this worthwhile.

kustomize build apps/api/overlays/production > /tmp/new.yaml
git stash
kustomize build apps/api/overlays/production > /tmp/old.yaml
git stash pop
diff -u /tmp/old.yaml /tmp/new.yaml

In CI, render the base branch and the head and post the difference as a pull request comment. What that catches that a source diff does not:

Unintended changes to other resources. A base change affecting overlays nobody thought about.

Immutable field changes. A label addition altering a selector.

Generated name changes. A ConfigMap hash changing, which means a rollout — sometimes intended, sometimes a surprise.

The scale of the change. “One line” in source can be forty resources rendered.

Do this for every overlay the change touches. A change to a base affects every environment consuming it, and rendering only production hides what dev will get — which is backwards, because dev is where you would rather find out.

With Argo CD and Flux

Both render Kustomize natively; you do not commit the output.

Argo CD detects a kustomization.yaml at the Application’s path and renders it. Overlay per Application, typically.

Flux uses a Kustomization resource whose path points at the overlay directory. Note the name collision: Flux’s Kustomization (kustomize.toolkit.fluxcd.io/v1) is a Flux custom resource that reconciles a path; Kustomize’s Kustomization (kustomize.config.k8s.io/v1beta1) is the file describing how to build it. They appear in the same repository and mean different things.

Rendered manifests as a pattern. Some teams commit the rendered output to a separate branch or directory, generated by CI, and point the controller at that. It makes the deployed state literally visible in Git and gives an exact diff per change, at the cost of generated content in the repository and a build step in the middle. Worth knowing about; not the default.

Kustomize or Helm

The comparison teams need, without a verdict.

	Kustomize	Helm
Mechanism	Patch YAML	Template YAML
Output predictable from input	Largely	Only by rendering
Diff review	Good	Needs helm template
Packaging for others	Poor	Designed for it
Versioned artifact	No	A chart
Conditionals and loops	No	Yes
Learning curve	Low	Moderate
Third-party components	Rare	The norm
Release lifecycle	None	Install, upgrade, rollback

The honest summary: Kustomize is better for configuration you own; Helm is better for software you distribute or consume.

Most repositories use both, and that is not a failure of consistency. Helm for the ingress controller and the monitoring stack because that is how they are published; Kustomize or plain manifests for your own services because you want the diff.

Kustomize can patch Helm output. Both Argo CD and Flux support rendering a chart and then applying Kustomize patches to the result, which handles the case where a chart does not expose the value you need. It works and it is a sign the chart is a poor fit — worth doing, worth not building a system around.

The decision that matters more than either: whether the rendered output is reviewed. A team using Kustomize badly and reviewing rendered diffs is in better shape than a team using Helm well and reviewing values files.

Migrating an existing repository

Moving off deprecated fields, or adopting Kustomize where plain manifests were duplicated per environment.

From duplicated manifests. Three near-identical directories, one per environment. The migration:

1. Diff the environments against each other. The common part is the base; the differences are the overlays. This diff is the whole design and it usually reveals differences nobody knew about.

2. Resolve the surprises first. A field that differs in production for no remembered reason is either important or abandoned, and you need to know which before you encode it.

3. Build the base from the common part, and overlays from the differences.

4. Render each overlay and diff against the original manifest for that environment. The diff must be empty. This is the verification step and skipping it means discovering the mismatch at apply time.

5. Apply from the new structure and confirm nothing changes.

6. Delete the old directories once nothing references them.

Step 4 is the migration. An empty diff proves the restructure is a no-op, and everything after it is routine.

From deprecated fields. kustomize edit fix in each directory, then render before and after and diff. The output should be identical; where it is not, the migration changed behaviour and needs looking at.

Do not combine the two migrations. Restructuring and migrating syntax at once means two sources of difference in the same diff.

Common mistakes

Deprecated fields in new configuration. patchesStrategicMerge and friends work and will not survive the API version change. Run kustomize edit fix.

Changing labels with includeSelectors: true. Immutable field, failed apply, delete-and-recreate to recover.

Strategic merge on args or command. Replaces the list rather than merging.

newTag instead of digest for deployed environments. Reintroduces mutability.

Disabling the name suffix hash without another rollout mechanism. Configuration changes that do not take effect.

secretGenerator with literals. Plaintext secrets in the repository.

Reviewing only the source diff. One line in, forty resources out.

Rendering only one overlay in CI. A base change affects all of them.

Deep base chains. Three levels of bases means nobody can predict what an overlay produces.

Validating in CI

Kustomize-specific checks worth running on every pull request.

kustomize build succeeds for every overlay. The most basic check and the one that catches a broken patch target — a patch whose target matches nothing fails silently in some versions and errors in others, so verify the behaviour of the version you pin.

Schema-validate the rendered output, not the input. The input is a kustomization; the output is Kubernetes resources, and only the output can be checked against the API schema.

Check for deprecated fields. kustomize build emits warnings for them. Failing CI on those warnings is how a repository stays migrated rather than drifting back.

Diff every affected overlay, not just one.

Pin the Kustomize version. Behaviour differs between versions — particularly around patch matching and the deprecated fields — and an unpinned version means a tool release changes your rendered output without a commit. This matters more than for most tools because the output goes straight to a cluster.

Note that kubectl kustomize and standalone kustomize can be different versions. The one embedded in kubectl lags the standalone release. A repository that renders correctly with one and not the other is a repository where somebody will lose an afternoon; state which is expected in the README and use the same one in CI.

A CI job doing those six things takes under a minute and catches essentially every mechanical Kustomize failure before a controller sees it.

Namespaces and name prefixes

Two transformers that seem convenient and have consequences worth knowing.

namespace: sets the namespace on every namespaced resource in the build, and updates references — a RoleBinding’s subject, for example. This is the right way to handle namespaces: set it once in the overlay, and leave every manifest namespace-free so it is reusable.

What it does not update: references inside opaque fields. A namespace written into a ConfigMap value, or into an application’s own configuration file, is a string Kustomize cannot see. Those need a replacement or a per-environment value.

namePrefix and nameSuffix rename resources and update references between them. Useful for running two instances of the same application in one namespace — a blue and a green, or a per-tenant deployment.

They also change the resource names in the cluster, which means adopting a prefix on an existing deployment creates new resources and orphans the old ones. With pruning enabled the old ones are deleted, which is a full replacement rather than an update. On a StatefulSet that means the volumes are detached and, depending on the reclaim policy, deleted.

Cluster-scoped resources are not namespaced, so namespace: does nothing for them. A ClusterRole appearing in several environments’ builds is the same object, and two environments both applying it means whichever reconciles last wins. Cluster-scoped resources belong in a separate build applied once, not in a per-environment overlay.

That last point catches teams running multiple environments in one cluster: the namespaced resources are properly isolated and the cluster-scoped ones silently are not.

Mental model

A base says what the application is. An overlay says how this environment differs. kustomize build turns the pair into exactly what the cluster receives — which is why the rendered output is the thing to review.

The discipline that follows: keep bases environment-agnostic, keep overlays small enough to read, and make CI show what the combination produces.

What you learned

• Bases hold resources; overlays patch them; kustomize build renders exactly what is applied
• patchesStrategicMerge, patchesJson6902, bases, vars and commonLabels are deprecated — kustomize edit fix migrates
• patches with a target selector can patch by label, which the deprecated fields could not
• labels with includeSelectors: true touches immutable selectors — set once, never change
• Strategic merge for values; JSON 6902 for list manipulation and removals
• images with a digest makes promotion a one-line, reviewable diff
• configMapGenerator’s name hash is what makes a config change actually roll pods
• Flux’s Kustomization and Kustomize’s Kustomization are different resources with the same name

Exercise

Use a disposable local cluster and repository.

1. Build a base with a Deployment and Service, and two overlays differing only in replica count.

2. Run kustomize build on both and diff the outputs. Predict: how many lines differ?

3. Add a strategic merge patch that adds one entry to the container’s args. Render it. Predict: were the existing arguments kept?

4. Redo the same change as a JSON 6902 patch with path: /.../args/-. Compare.

5. Add a configMapGenerator and render. Note the generated name. Change the content and render again. Predict: did the Deployment’s reference change?

6. Set disableNameSuffixHash: true and repeat. Predict: would a running pod pick up the change?

7. Add labels with includeSelectors: true to a base with an already-deployed Deployment. Apply. Predict: does it succeed?

8. Delete the cluster.

Related lessons

Helm + GitThe templating alternative, and when it fits.

Kubernetes Environment PromotionPromoting the digest in the images field.

Kubernetes Secrets and GitOpsWhy secretGenerator is not a solution.

FluxThe other Kustomization resource.