Flux GitOps: Complete Engineering Guide

Flux is not one thing. It is a set of controllers, each with its own custom resource, which you compose.

That is the main difference from Argo CD and it shapes everything else: there is no Application object holding the whole picture, and no bundled UI. What there is instead is a set of Kubernetes resources you can inspect, own, and reason about with the same tools you use for everything else in the cluster.

Before you start

• GitOps Explained — the principles and the reconciliation model
• Kubernetes Repository Structure — what the controllers point at

Availability

The controllers and their resources

Controller	Resource	API version	Job
source-controller	GitRepository	source.toolkit.fluxcd.io/v1	Fetch from Git
source-controller	OCIRepository	source.toolkit.fluxcd.io/v1	Fetch an OCI artifact
source-controller	Bucket	source.toolkit.fluxcd.io/v1	Fetch from object storage
kustomize-controller	Kustomization	kustomize.toolkit.fluxcd.io/v1	Build and apply manifests
helm-controller	HelmRelease	helm.toolkit.fluxcd.io/v2	Manage a Helm release
notification-controller	Alert, Provider, Receiver	notification.toolkit.fluxcd.io	Events in and out
image-reflector / image-automation	ImageRepository, ImagePolicy, ImageUpdateAutomation	image.toolkit.fluxcd.io	Scan tags, commit updates

The separation is the design. A source is fetched once and can be consumed by many Kustomizations. That means one clone serving twenty applications rather than twenty clones, and it means the fetch interval and the apply interval are configured independently — which matters more than it sounds, and is covered below.

Caution

Flux’s Kustomization (kustomize.toolkit.fluxcd.io/v1) and Kustomize’s own Kustomization (kustomize.config.k8s.io/v1beta1) are different resources with the same kind name. The first is a Flux resource telling a controller to reconcile a path; the second is the file in your repository describing how to build it. Both appear in the same repository. Read the apiVersion.

Sources

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: platform-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/example-org/platform-config.git
  ref:
    branch: main
  secretRef:
    name: platform-config-auth

interval is required and is how often the source is re-fetched. One minute is a reasonable default; a webhook via a Receiver makes it near-immediate and the interval becomes a fallback.

ref takes branch, tag, semver, name or commit. If ref is omitted entirely, the branch defaults to master — worth knowing, because a repository whose default is main and a GitRepository with no ref produces a confusing “reference not found”.

secretRef points at credentials. Read-only access is sufficient unless you use image automation, which writes back.

Other spec fields worth knowing: timeout (60 seconds by default), verify for commit signature verification, ignore for excluding paths, sparseCheckout for fetching only what you need in a large repository, recurseSubmodules, and provider for OIDC-based authentication.

OCIRepository reads manifests packaged as an OCI artifact rather than from Git:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: api-manifests
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/example-org/manifests/api
  ref:
    digest: sha256:0000000000000000000000000000000000000000000000000000000000000000
  verify:
    provider: cosign

This satisfies the GitOps principles fully — versioned, immutable, pulled — with two properties Git does not have: a digest reference that genuinely cannot move, and signature verification as part of the fetch. The usual arrangement keeps Git as the human interface and has a pipeline publish an OCI artifact the cluster consumes.

Kustomization

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: api-production
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/api/overlays/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: platform-config
  targetNamespace: production
  wait: true
  timeout: 5m
  dependsOn:
    - name: infrastructure
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: api
      namespace: production

prune is required and is a boolean. Flux makes you state it, which is a good design — there is no default to be surprised by. The same warning applies as everywhere: prune: true against a path that does not describe everything in the namespace deletes the difference.

interval has a minimum of 60 seconds. This is the reconciliation interval, distinct from the source’s fetch interval. Ten minutes is common for a Kustomization; the source polls more often.

sourceRef.kind can be GitRepository, OCIRepository, Bucket or ExternalArtifact.

dependsOn waits for another Kustomization to be ready. This is how ordering is expressed — infrastructure before applications, CRDs before the resources using them — and it is cleaner than Argo CD’s annotation-based sync waves because it is a first-class field on a resource.

wait: true makes the Kustomization report ready only when its resources are healthy, which is what makes dependsOn meaningful. Without it, “ready” means “applied”.

healthChecks names specific resources whose rollout status must succeed. More precise than wait: true and worth using for the resources that actually matter.

targetNamespace overrides the namespace for everything in the build, which lets one path serve several namespaces.

HelmRelease

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: api
  namespace: production
spec:
  interval: 10m
  chartRef:
    kind: OCIRepository
    name: api-chart
  values:
    replicaCount: 6
  valuesFrom:
    - kind: ConfigMap
      name: api-production-values
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
      remediateLastFailure: true
  driftDetection:
    mode: enabled

chartRef is the current recommendation, pointing at an OCIRepository — this is what the documentation suggests for production. The older chart.spec.sourceRef pattern still works and is what you will see in most existing examples and tutorials.

Flux maintains real Helm releases. Unlike Argo CD, which renders charts and applies the output, helm-controller runs Helm. helm list shows the releases, and Helm’s own upgrade and rollback semantics apply.

remediation retries a failed install or upgrade, and can roll back. This is a genuine capability Argo CD’s render-and-apply model does not have.

driftDetection compares the cluster against the release and corrects it — the self-heal equivalent for Helm-managed resources.

valuesFrom reads values from a ConfigMap or Secret, which is how environment values live outside the resource.

Bootstrap

Flux installs itself into your repository, which is unusual and deliberate.

flux bootstrap github \
  --owner=example-org \
  --repository=platform-config \
  --branch=main \
  --path=clusters/production \
  --personal=false

What it does: installs the controllers, commits their manifests to clusters/production/flux-system/ in your repository, and creates a GitRepository and Kustomization pointing at that path. From then on Flux reconciles its own installation — upgrading Flux is a commit.

The bootstrap is the one imperative step, and it is worth recording how it was run. A cluster somebody bootstrapped eighteen months ago with unknown flags is a cluster nobody can rebuild.

It needs a token with repository write access, because it commits. That is a one-off requirement; steady-state operation needs only read unless you use image automation.

--path scopes the cluster. A repository serving several clusters has a path per cluster, and each cluster’s Flux reconciles only its own — which is what stops one cluster applying another’s configuration.

Self-management has the obvious failure mode: a bad Flux upgrade breaks the thing that would roll it back. Keep the bootstrap command documented, because that is the recovery path.

Image automation

Flux can watch a registry and commit updated image references — genuinely useful, and a capability that changes the trust model.

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: api
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: api
  policy:
    semver:
      range: '>=2.0.0 <3.0.0'

Three resources work together: ImageRepository scans tags, ImagePolicy selects one, and ImageUpdateAutomation writes the result back to the repository as a commit.

The trade-off is real. Automatic image updates mean a new image reaches an environment without a human deciding. For a development environment that is exactly what you want. For production it removes the promotion gate this pillar has spent several lessons arguing for.

The compromise most teams reach: automation writes to development directly, and to a branch for higher environments so promotion is a pull request. The environment promotion lesson covers the shape.

It requires write access to the repository, which is a wider grant than steady-state Flux needs. Scope it to a deploy key limited to the specific repository, and be aware that a controller with repository write is a controller that can change desired state.

Notifications and events

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: on-failure
  namespace: flux-system
spec:
  providerRef:
    name: team-slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: '*'
    - kind: HelmRelease
      name: '*'

Provider and Alert send events outward. Receiver accepts webhooks inward, which is how a Git push triggers immediate reconciliation rather than waiting for the interval.

Alert on error severity, not on every event. A channel receiving a message per successful reconciliation is a channel nobody reads.

The alert people forget: the controllers themselves being unhealthy. A stopped Flux reconciles nothing and reports nothing, which looks identical to a cluster with no drift.

RBAC and multi-tenancy

Controllers run with a ServiceAccount, and by default that is broad. Flux supports impersonation: a Kustomization can specify serviceAccountName, and the apply runs as that ServiceAccount rather than the controller’s.

This is the multi-tenancy mechanism. A tenant’s Kustomization impersonates a ServiceAccount limited to their namespace, so a tenant cannot deploy cluster-scoped resources or reach another namespace regardless of what their manifests say.

Without impersonation, any manifest a Kustomization applies runs with the controller’s full permissions — which for a default install is substantial. Anyone who can merge to the watched path can create anything the controller can create.

Scope by namespace where you can. A Kustomization with targetNamespace and an impersonated ServiceAccount confined to it is a genuine boundary.

Adopting it on an existing cluster

The sequence, with the same caution about pruning as everywhere else.

1. Get manifests into a repository and verify with kubectl diff that they describe what is running.

2. Bootstrap into a non-production cluster first. Record the command.

3. Create one Kustomization with prune: false, pointing at a low-stakes path.

4. Check it reconciles cleanly. flux get kustomizations should show it ready. If the apply changes anything, the repository does not match the cluster — investigate before continuing.

5. Run for a week with prune: false. Watch what the controller does on each interval.

6. Enable prune: true once the path genuinely declares everything in the namespace.

7. Add dependsOn and wait: true where ordering matters, and verify the dependency actually blocks.

8. Add impersonation via serviceAccountName before letting other teams contribute paths.

9. Repeat per namespace, then per cluster.

10. Remove direct write access to reconciled namespaces.

Step 4 is where the honest state of your cluster becomes visible; step 6 is the one that can delete things. Neither should be rushed.

One Flux-specific note: because bootstrap commits to your repository, do this against a repository you are willing to have modified. It creates a flux-system directory and a commit, which is unsurprising once you know and startling if you do not.

Debugging

Flux’s CLI is where most diagnosis happens.

flux get all -A shows every Flux resource and its status. The first command to run.

flux logs --follow streams controller logs.

flux reconcile source git NAME forces an immediate fetch, and flux reconcile kustomization NAME --with-source re-fetches and re-applies. Useful when you do not want to wait for an interval.

flux suspend and flux resume stop and start reconciliation for a resource. This is what you use during an incident when you need the cluster to stop being corrected while somebody works on it by hand — and knowing the command exists before you need it is the point. A suspended resource stays suspended until somebody resumes it, so it also needs to be on the incident checklist as something to undo.

flux diff kustomization NAME --path ./path shows what a change would do before merging.

The status conditions on the resources themselves are the ground truth. kubectl describe kustomization NAME shows why it is not ready, and the message is usually specific.

The common failures: a source that cannot authenticate; a path that does not exist; a build that fails; an apply rejected by admission control; a dependsOn waiting on something that is not ready. Each reports distinctly in the resource’s conditions.

Repository layout for Flux

Flux’s model shapes the repository, and a layout that fits it looks slightly different from an Argo CD one.

clusters/
├── production/
│   ├── flux-system/          # Committed by bootstrap
│   ├── infrastructure.yaml   # A Kustomization
│   └── apps.yaml             # A Kustomization, dependsOn infrastructure
└── staging/

infrastructure/
├── controllers/              # Ingress, cert-manager — HelmReleases
└── configs/                  # ClusterIssuers, NetworkPolicies

apps/
├── base/
└── overlays/
    ├── production/
    └── staging/

clusters/<name>/ holds Flux resources, not workloads. A handful of Kustomization objects saying which paths to reconcile and in what order.

The rest of the tree holds what those paths point at. Applications and infrastructure, environment overlays underneath.

The split between controllers/ and configs/ matters more than it looks. A ClusterIssuer cannot be created before cert-manager’s CRDs exist, so they are two Kustomizations with a dependsOn between them. Putting both in one path produces a reconciliation that fails on the first pass and succeeds on the second — which works, and generates a stream of errors nobody should have to ignore.

One GitRepository serving many Kustomizations is the normal arrangement. One clone, many consumers, and each Kustomization reconciles on its own interval.

Adding a cluster is copying clusters/<name>/ and bootstrapping against it. The applications and infrastructure paths are shared; only the Flux resources differ.

Reconciliation intervals

More consequential in Flux than in Argo CD, because there are several and they compose.

The source interval governs how often the repository is fetched. One minute is common; with a Receiver webhook the fetch is immediate on push and the interval is a fallback.

The Kustomization interval governs how often the build is applied and drift is corrected. This is the one that determines how quickly a manual change is reverted.

They are independent. A one-minute source and a ten-minute Kustomization means a commit is fetched within a minute and applied within ten. That is usually fine and it confuses people who expect a push to deploy immediately.

Shorter is not better. Every interval is load on the API server and on the controllers. A cluster with two hundred Kustomizations at one-minute intervals is doing a great deal of work to correct drift that mostly is not there.

The interval is also the drift-correction latency. If self-healing within a minute matters for a particular workload, that Kustomization gets a short interval and the rest do not. Setting them all short to satisfy one requirement is the mistake.

flux reconcile exists for when you do not want to wait, which removes most of the pressure to set short intervals in the first place.

What Flux does not give you

Worth stating plainly, because the comparison with Argo CD usually turns on these.

No bundled web UI. There are third-party dashboards and there is no first-party one. Everything is kubectl and flux on the command line. For teams that want a console showing what is deployed where, this is the main objection — and for teams that want everything expressible as Kubernetes resources, it is the main attraction.

No single object holding the whole picture. An Argo CD Application is one resource describing a source, a destination and a policy. The Flux equivalent is a GitRepository plus a Kustomization, possibly plus a HelmRelease. More resources, more precision, more to write.

No built-in RBAC layer of its own. Argo CD has its own RBAC governing who can sync what. Flux uses Kubernetes RBAC — if you can edit a Kustomization, you can change what is deployed. That is simpler and it means access control is a cluster concern rather than a separate system to configure.

No sync button. Reconciliation happens on an interval or on flux reconcile. There is no UI action that a person with a browser can take, which is either a missing feature or a desirable property depending on your view of who should be able to deploy.

Less immediate visibility of drift. Flux reports it in resource conditions; you have to look, or alert. Argo CD shows it on a dashboard.

None of these are defects. They are the consequences of a design that treats reconciliation as a set of Kubernetes controllers rather than as an application with an interface, and the comparison works through which set of consequences suits which team.

Common mistakes

Following Flux v1 documentation. An archived, entirely different architecture.

v1beta1 or v1beta2 for sources and Kustomizations. Superseded — v1 for both, v2 for HelmRelease.

Omitting ref on a GitRepository. Defaults to the master branch.

prune: true before the path describes everything. Deletes the difference.

No wait: true with dependsOn. Ready means applied, not healthy, so the dependency does not do what you meant.

No impersonation in a multi-tenant cluster. Every manifest applies with the controller’s permissions.

Image automation writing directly to production. Removes the promotion gate.

Confusing the two Kustomization kinds. Read the apiVersion.

Alerting on every event. Nobody reads the channel.

No record of the bootstrap. The recovery path is undocumented.

Variable substitution

Flux’s answer to templating, and a feature that needs restraint.

A Kustomization can substitute variables into the manifests it builds, from a ConfigMap or Secret:

spec:
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
    substitute:
      cluster_name: production

Manifests reference them as ${cluster_name}.

The legitimate use is a genuinely per-cluster value that would otherwise force a separate overlay for each cluster — a region name, a cluster identifier, an ingress domain. One path serving thirty clusters that differ only in three values is exactly the case this solves.

The failure mode is turning your manifests into templates. A repository where kustomize build produces YAML full of unresolved ${...} is a repository where the rendered output is no longer readable, which loses the property that made Kustomize worth choosing. CI can no longer validate the output without knowing the substitutions.

The rule worth holding: substitute values, never structure. A variable that holds a name or a domain is fine. A variable that determines whether a resource exists is a conditional, and conditionals in a text-substitution system are how you get a repository nobody can reason about.

Test with substitutions applied. A CI job that renders with representative values catches the case where a variable is referenced and never defined — which otherwise fails at reconciliation time with a message about an unresolved variable.

Upgrading Flux

The controllers are components you operate, and Flux’s self-management makes the upgrade unusual.

flux check reports the installed version and whether the CLI matches. Run it before and after.

Upgrading is flux bootstrap again, with a newer CLI, against the same path. It updates the committed manifests, and Flux applies its own upgrade on the next reconciliation.

CRD changes are the thing to read about. Flux’s APIs have stabilised, and moving between older beta versions and v1 required migrating resources. Check the release notes for the range you are crossing.

Keep the CLI and the controllers in step. A CLI substantially newer than the cluster’s controllers produces confusing behaviour, and flux check warns about it.

Upgrade a non-production cluster first, and give it time. Controller regressions surface under real workloads rather than immediately.

If Flux is down, nothing reconciles and nothing breaks. The cluster keeps running exactly as it is. That is a benign failure mode worth stating explicitly, because the instinct during an outage is to start applying manifests by hand — which then drifts from the repository and gets reverted when Flux returns.

Secrets and Flux

The controllers need credentials, and Flux itself has an opinion about workload secrets.

Flux’s own credentials — the repository authentication, registry credentials for OCI sources, any provider tokens — are Kubernetes Secrets in flux-system. Whoever can read that namespace holds them, which makes namespace RBAC a real control rather than a formality.

Bootstrap creates the repository credential. For a deploy key, scope it to the one repository and make it read-only unless image automation needs write.

Flux has native SOPS support, which is the most commonly used answer to the workload-secret problem in Flux setups. A Kustomization can specify a decryption provider and a key, and encrypted files in the repository are decrypted at reconciliation time:

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

What that gives you: encrypted secrets committed alongside everything else, decrypted by a key the cluster holds and the repository does not. The key becomes the thing to protect, and key rotation becomes an operational task you own.

What it does not give you: anything about the key’s lifecycle. Rotation, access control and recovery are yours. GitOps secrets covers this properly, including the alternatives — an external secret operator pulling from a manager is a different trade with different failure modes.

Never commit an unencrypted Secret manifest because Flux will apply it exactly as written. Base64 in a data: field is an encoding, and the controller does not care.

Mental model

Flux is a set of Kubernetes controllers, each doing one job, composed through custom resources. A source fetches; a Kustomization builds and applies; a HelmRelease manages a release. Everything is a resource you can inspect with kubectl.

That composability is the trade: more resources to write than Argo CD’s single Application, and more precision about what each one does and what it is allowed to do.

What you learned

• Current APIs: source.toolkit.fluxcd.io/v1, kustomize.toolkit.fluxcd.io/v1, helm.toolkit.fluxcd.io/v2
• Flux’s Kustomization is not Kustomize’s Kustomization — check the apiVersion
• A GitRepository with no ref defaults to the master branch
• prune is a required boolean on a Kustomization; there is no default to be surprised by
• dependsOn plus wait: true is how ordering works, and without wait it means applied rather than healthy
• chartRef with an OCIRepository is the current recommendation for HelmRelease
• Flux maintains real Helm releases, with remediation and drift detection
• Impersonation via serviceAccountName is the multi-tenancy boundary
• Bootstrap commits Flux’s own manifests, so upgrading Flux is a commit

Exercise

Use a disposable local cluster — kind or minikube. No production cluster.

1. Bootstrap Flux against a disposable repository you own, scoped to a clusters/test path. Record the exact command.

2. Inspect what it committed to the repository. Predict: what manages Flux now?

3. Add a GitRepository without a ref. Predict: which branch does it try?

4. Add a Kustomization with prune: false pointing at a directory with a Deployment. Confirm it applies.

5. Delete a manifest from the directory. Predict: is the resource removed?

6. Set prune: true and repeat.

7. Add a second Kustomization with dependsOn on the first, but without wait: true. Predict: does it wait for healthy, or only for applied?

8. Run flux suspend kustomization NAME, change the manifest, and confirm nothing happens. Resume.

9. Delete the cluster and the repository.

Related lessons

Argo CD vs FluxChoosing between the two operating models.

Argo CDThe other controller.

GitOps Drift and ReconciliationSuspending reconciliation during an incident.

Helm + GitCharts consumed by HelmRelease.