Argo CD: Complete GitOps Guide

Argo CD makes the gap between your repository and your cluster visible, continuously, in a way nothing else in this pillar does.

That visibility is its main contribution and its main risk. A controller that can see the difference between declared and actual state is a controller that can close it — including at moments when somebody deliberately made the cluster differ.

Before you start

• GitOps Explained — the principles and the reconciliation model
• Kubernetes Repository Structure — what an Application points at

Availability

The Application

The central resource. It says where desired state lives and where it goes.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-production
  namespace: argocd
spec:
  project: production

  source:
    repoURL: https://github.com/example-org/platform-config.git
    targetRevision: main
    path: apps/api/overlays/production

  destination:
    server: https://kubernetes.default.svc
    namespace: production

  syncPolicy:
    automated:
      prune: false
      selfHeal: false
    syncOptions:
      - CreateNamespace=true

argoproj.io/v1alpha1 is the current API version for Application, AppProject and ApplicationSet. The v1alpha1 label has been stable for a long time and is not a sign of instability.

Applications live in the Argo CD namespace — argocd by default. This surprises people expecting them to live beside the workloads they deploy.

targetRevision can be a branch, a tag, a semver range or a commit SHA. A branch means the Application follows it and syncs whenever the branch moves; a tag or SHA pins it until somebody changes the Application. That distinction is the mechanism for staged rollouts across clusters, and it is the difference between a fleet where every cluster changes at once and one where changes travel deliberately.

path is the directory in the repository. Argo CD detects a kustomization.yaml or a Chart.yaml there and renders accordingly.

Sync policy: three separate decisions

The configuration that determines how much autonomy the controller has, and each part should be a deliberate choice.

automated — sync without a human pressing anything. Without it, Argo CD detects drift and waits.

prune — delete resources that exist in the cluster but not in the source. Without it, removing a manifest leaves the resource running.

selfHeal — revert manual changes to managed resources. Without it, drift is reported and left alone.

Configuration	Behaviour
No automated	Reports drift; a human syncs
automated only	Applies changes from the source; leaves manual changes and orphans alone
automated + prune	Also deletes what the source no longer declares
automated + prune + selfHeal	Full reconciliation — the repository wins, always

Danger

Enable these in order, with time between them. prune: true against a source that does not describe everything in the namespace deletes the difference — and “everything” includes resources created by other controllers, by an operator, or by a colleague last year. The safe sequence is automated alone for a week, then prune once you have confirmed the source is complete, then selfHeal.

selfHeal deserves its own thought. It is the property that makes the repository authoritative in fact rather than in policy — and it means an emergency kubectl edit at 3am is reverted within minutes, usually while somebody is still watching to see whether their fix worked. Drift and reconciliation covers operating with it, including how to suspend it.

Projects

AppProject is the boundary that stops one Application from being able to do anything.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  description: Production workloads

  sourceRepos:
    - https://github.com/example-org/platform-config.git

  destinations:
    - server: https://kubernetes.default.svc
      namespace: production
    - server: https://kubernetes.default.svc
      namespace: production-jobs

  clusterResourceWhitelist: []

  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota
    - group: rbac.authorization.k8s.io
      kind: ClusterRoleBinding

sourceRepos limits which repositories Applications in this project may use. Without it, an Application can point anywhere.

destinations limits which clusters and namespaces it may deploy to.

clusterResourceWhitelist: [] — an empty list — means this project may create no cluster-scoped resources. That is a strong and usually correct default for application projects; cluster-scoped resources belong to a platform project with different ownership.

Projects are the practical answer to blast radius. An Argo CD instance with everything in the default project has no boundaries at all — default permits every repository, every destination and every resource kind by design, because it exists to make the first Application work rather than to be safe. Moving off it is one of the highest-value configuration changes available and it is rarely done, because nothing prompts you.

Declarative Applications

Applications should be resources in the repository, not things created in the web UI.

The app-of-apps pattern. A root Application points at a directory containing Application manifests:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-apps
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://github.com/example-org/platform-config.git
    targetRevision: main
    path: clusters/production/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Adding an application is adding a file. Removing one is deleting a file — and with prune: true on the root, that deletes the Application, which with pruning on its policy deletes everything it deployed. That is the intended behaviour and it is worth knowing before somebody tidies a directory.

ApplicationSet generates Applications from a generator — a list, a directory, a set of clusters, a Git file pattern, a pull request. One definition producing many Applications.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: api-environments
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - env: dev
            namespace: dev
          - env: staging
            namespace: staging
  template:
    metadata:
      name: 'api-{{env}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/example-org/platform-config.git
        targetRevision: main
        path: 'apps/api/overlays/{{env}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{namespace}}'

Right at scale — many clusters, many tenants, many near-identical environments. Over-engineering for three environments you could list explicitly, and a generator producing unexpected Applications is harder to debug than three files.

Multiple sources

An Application can take a sources array rather than a single source. When sources is set, source is ignored.

The common legitimate use is a Helm chart from one place with values from another:

  sources:
    - repoURL: https://github.com/example-org/platform-config.git
      targetRevision: main
      ref: values
    - repoURL: oci://ghcr.io/example-org/charts
      chart: api
      targetRevision: 2.4.1
      helm:
        valueFiles:
          - $values/environments/production/values.yaml

The ref on the first source names it $values, which the second references. A source carrying a ref cannot also specify chart.

Argo CD’s own documentation warns against overuse. More than two or three entries is described as almost certainly abusing the feature, and where several sources produce the same resource the last one wins with a RepeatedResourceWarning. It is not a mechanism for grouping unrelated things.

The components

Knowing what runs helps when something is not working.

The application controller does the reconciliation: compares desired against actual, applies differences, computes health. This is the component whose resource usage grows with the number of Applications and resources.

The repo server clones sources and renders manifests — running Kustomize, Helm template, or reading plain YAML. Rendering is its job, so a slow render is a repo-server problem. It caches, and cache behaviour explains most “why has it not picked up my commit” questions.

The API server serves the UI, the CLI and the gRPC API, and enforces Argo CD’s own RBAC.

Redis caches state. It is not a database — losing it is recoverable, and the controller rebuilds. Worth knowing so that a Redis restart is not treated as an emergency.

Notifications and ApplicationSet controllers are optional components with their own lifecycles.

Where the bottlenecks appear: the repo server on repositories with many large renders, and the application controller on clusters with a large number of resources. Both scale horizontally with configuration, and reaching for that before diagnosing which one is slow is the usual mistake.

What this means for debugging: a sync that is slow to start is usually the repo server or a webhook that is not configured; a sync that is slow to complete is the application controller or the cluster API. Those have different fixes, and the UI does not distinguish them clearly.

Webhooks matter more than people expect. Without one, Argo CD polls the repository on an interval — three minutes by default — so a merge takes up to that long to be noticed. A webhook from the Git provider makes it near-immediate, and configuring one is the single easiest improvement to how the platform feels.

Health and sync status

Two different questions, and conflating them causes real confusion.

Sync status — does the cluster match the source? Synced or OutOfSync.

Health status — are the resources working? Healthy, Progressing, Degraded, Missing, Suspended.

Synced does not mean working. A Deployment applied exactly as declared, whose pods crash-loop, is Synced and Degraded. This is the single most common misreading of an Argo CD dashboard, and a team that alerts only on sync status is alerting on the wrong thing.

Health is computed per resource kind, with built-in logic for the standard ones and custom Lua health checks for CRDs. A custom resource with no health check reports as healthy regardless of what it is doing, which is worth knowing before you trust a green dashboard containing operators.

Alert on both, and on a third thing: whether the controller is reconciling at all. A controller that has stopped shows everything as it last saw it, which looks identical to a healthy synced state.

Sync waves and hooks

Ordering, for the cases where reconciliation’s unordered model does not fit.

Sync waves order resources within a sync using an annotation:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Lower numbers apply first. Useful for CRDs before custom resources, namespaces before what goes in them, and databases before applications.

Hooks run resources at a phase — PreSync, Sync, PostSync, SyncFail:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded

A PreSync Job is the usual home for a database migration.

Use these sparingly. They are the mechanism for expressing sequence in a system built around desired state, and a repository with elaborate wave numbering has usually encoded a deployment order that would be clearer as a pipeline. The push and pull discussion applies: some things are genuinely sequences.

Note the interaction with Helm hooks. Argo CD renders charts rather than running Helm, so Helm hooks are translated to Argo CD hooks. Behaviour can differ from what a chart’s author intended, and a chart relying on subtle hook ordering is worth testing under Argo CD specifically.

Repository credentials and RBAC

Repository credentials are Secrets in the Argo CD namespace with a particular label. They should be managed like any other secret — not committed in plaintext, which means the same GitOps secrets problem applies to Argo CD’s own configuration.

Read-only access is sufficient for the repository. Argo CD does not need to write, and the write-back features that do — image updating, for instance — should be a separate, narrowly scoped credential.

Argo CD’s RBAC governs who can do what in Argo CD: sync, override, delete, view. Map it to your identity provider’s groups, and be deliberate about who can sync production and who can override a sync.

The controller’s Kubernetes RBAC is separate and is the one that matters for blast radius. A controller with cluster-admin can do anything in the cluster. Scoping it per project or per namespace is more work and considerably better.

Two dangerous permissions to watch: the ability to sync with --force, which replaces resources rather than patching them, and the ability to edit an Application’s spec. The second is equivalent to changing what the cluster runs without touching the repository — which defeats the entire model, and is why Applications should be managed declaratively rather than through the UI.

Ignoring fields you do not own

A frequent source of permanent OutOfSync status, with a specific fix.

Some fields are legitimately managed by something other than your repository. An HPA owns spec.replicas. A mutating webhook injects a sidecar. A cloud controller writes an annotation onto a Service. Argo CD sees the difference and reports drift forever.

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: ""
      kind: Service
      jqPathExpressions:
        - '.metadata.annotations."service.beta.kubernetes.io/aws-load-balancer-arn"'

The better fix, where possible, is to omit the field from the manifest. A Deployment manifest that does not set replicas at all leaves the field to whoever manages it, and there is nothing to diff. ignoreDifferences is for cases where you must declare a field but something else mutates it.

Do not use it to silence drift you should investigate. A field being changed by something you cannot identify is a finding, not noise. Adding it to ignoreDifferences makes the symptom go away and leaves the cause.

Overuse produces a controller that reports Synced while the cluster differs meaningfully from the repository. That is worse than visible drift, because the dashboard now says something untrue.

Server-side apply and field management are the more principled version of the same idea, and both controllers support applying that way. Where field ownership is properly tracked, another manager’s field is not drift at all.

Notifications and observability

A controller nobody is watching is a controller whose failures are silent.

What to alert on, in order of value:

The controller is not reconciling. A stopped controller shows the last known state, which looks healthy. This is the alert teams forget and the one that hides everything else.

Sync failed. An apply that errored — an invalid manifest, a missing CRD, an RBAC denial.

Health degraded. Resources applied and not working.

Out of sync for longer than a threshold. With automated sync, persistent drift means something is fighting the controller. Without it, a long-standing OutOfSync means a change is waiting for somebody.

What not to alert on: every sync. A successful reconciliation is the normal case and alerting on it trains people to ignore the channel.

Argo CD Notifications sends on triggers to Slack, webhooks and similar, configured declaratively. Worth wiring to the pull request that caused a sync, so the person who merged sees the outcome.

Metrics matter more than notifications at scale. Reconciliation duration, sync failure rate and the number of out-of-sync Applications are the three that tell you whether the platform is healthy, and they belong on a dashboard rather than in a chat channel.

Adopting it on an existing cluster

The migration people get wrong, and the order that makes it safe.

1. Get the manifests into a repository first, and verify with kubectl diff that they describe what is running. Argo CD does not help with this and pointing it at an incomplete repository is where the damage happens.

2. Install Argo CD in the cluster, pinned, with its own namespace.

3. Create one Application for one low-stakes namespace, with no automated policy.

4. Look at what it reports. It should say Synced immediately. If it says OutOfSync, the repository does not match the cluster and that is the finding — investigate before going further.

5. Enable automated for that Application. Make a change in Git and watch it apply.

6. Run like that for a week. Watch what drifts. You will find changes nobody remembered making.

7. Enable prune once you are confident the repository declares everything in that namespace. Read the pending prune list before enabling, if the UI shows one.

8. Enable selfHeal last, and write down how to suspend it before you turn it on.

9. Repeat per namespace, most important last.

10. Remove direct write access to the namespaces Argo CD manages. Until this, the repository is a description rather than a control.

Step 4 is where the honest state of your cluster becomes visible, and step 7 is the one that can cause an outage. Neither should be rushed, and the week in step 6 is not padding — it is how you discover the resources somebody created by hand two years ago that nothing declares.

Common mistakes

prune: true before the source is complete. Deletes resources it does not know about.

selfHeal: true with no suspend procedure. Reverts an emergency fix mid-incident.

Creating Applications in the UI. The repository is no longer the source of truth for what runs.

Everything in the default project. No source, destination or resource boundaries.

clusterResourceWhitelist unset for application projects. Applications can create cluster-scoped resources.

Alerting on sync status only. Synced and Degraded is a common and unalarmed state.

Trusting health status for custom resources. No health check means always healthy.

Elaborate sync waves. A pipeline expressed as annotations.

cluster-admin for the controller. Blast radius equal to the cluster.

A floating install manifest. The controller upgrades itself unpredictably.

Multi-cluster

Argo CD manages clusters other than its own, and the choice of topology has real consequences.

A hub instance managing many clusters. One Argo CD, credentials for each target cluster, everything visible in one place. Operationally convenient and it means the hub holds write credentials for every cluster it manages — which is a concentration of privilege worth being deliberate about, and a single point of failure for deployment.

One instance per cluster. Each cluster reconciles itself. No cross-cluster credentials, blast radius contained, and no single dashboard. More instances to operate and upgrade.

The security difference is the one to weigh. The hub model reintroduces something the pull model was supposed to remove: a system holding write access to clusters from outside them. It is still better than a CI system doing it — the hub is inside a cluster, credentials are Kubernetes Secrets rather than CI secrets — and it is not the same as each cluster reconciling only itself.

Cluster credentials are Secrets in the Argo CD namespace. Whoever can read that namespace can reach every managed cluster.

Staged rollout across clusters is done with targetRevision. Development clusters follow main; production clusters pin a tag. Promotion is moving the tag, or a pull request changing the pinned revision per cluster. This is the mechanism that stops one bad commit reaching thirty clusters simultaneously, and a fleet where every cluster follows main has no such protection.

ApplicationSet with a cluster generator is how the hub model scales without hand-writing an Application per cluster per app.

Upgrading Argo CD

The controller is itself a component you operate, and its upgrades deserve the treatment infrastructure components get.

Pin the version. A floating install manifest means the controller upgrades when upstream publishes.

Manage it with itself, carefully. Argo CD can reconcile its own manifests, which is elegant and has an obvious failure mode: a bad upgrade breaks the thing that would roll it back. Keep the bootstrap path — the imperative command or Terraform module that installed it — documented and working, because that is the recovery route.

Read the upgrade notes between minor versions. CRD schema changes and behaviour changes both happen, and a CRD change applied by a controller that is mid-upgrade is a bad moment.

Upgrade a non-production instance first, and leave it for a while. Controller regressions tend to show up under load or on specific resource types rather than immediately.

Check custom health checks and resource customisations after upgrading. These are the parts most likely to interact with a version change.

Have a plan for the controller being down. Nothing reconciles, which means nothing deploys and nothing self-heals — the cluster keeps running exactly as it is. That is a benign failure mode and it is worth having said so out loud, because the instinct during an outage is to start applying manifests by hand, which then drifts from the repository.

The UI, and what it is for

Argo CD’s web interface is its most visible feature and the most likely to be misused.

What it is genuinely good for: seeing state. Which applications are out of sync, which are degraded, what the resource tree looks like, what the last sync did, and the diff between desired and actual for a specific resource. That view is hard to assemble any other way and it is the reason many teams choose Argo CD.

What it should not be used for: changing things. Creating an Application, editing a spec, or overriding a sync are all changes to what the cluster runs that leave no trace in the repository. The next reconciliation may revert them, or may not, depending on what was changed — and either outcome is confusing.

The sync button is the exception worth allowing. With automated off, somebody clicking sync is performing a deliberate promotion, and that is a legitimate human decision. Whether it should require a specific role is a real question, particularly for production.

Restrict the rest through RBAC. Read for most people, sync for those who deploy, and Application editing for essentially nobody. If an Application needs changing, that is a pull request.

Terminate and refresh are safe. Cancelling a stuck sync and forcing a re-check of the repository do not change desired state.

A useful team norm: the UI is where you look when something is wrong, and Git is where you change things. Teams that keep that line find the UI genuinely valuable; teams that do not end up with a cluster whose state has no explanation in any repository.

Mental model

An Application is a statement that this path in this repository should be running in this namespace. The sync policy decides how forcefully Argo CD asserts it, and the project decides how far that assertion can reach.

Three separate dials — automated, prune, self-heal — and a boundary. Set them deliberately, in that order, and the controller does exactly what you meant.

What you learned

• argoproj.io/v1alpha1 for Application, AppProject and ApplicationSet, living in the Argo CD namespace
• automated, prune and selfHeal are three separate decisions — enable them in that order
• AppProject limits source repositories, destinations and resource kinds; default permits everything
• App-of-apps makes adding an application a commit; deleting one with pruning removes everything it deployed
• sources (plural) supports a chart in one place and values in another; more than two or three is abuse
• Synced means applied, not working — alert on health and on the controller itself
• Custom resources without a health check always report healthy
• Sync waves and hooks express sequence, and elaborate use is a signal a pipeline would fit better

Exercise

Use a disposable local cluster — kind or minikube. No production cluster.

1. Install Argo CD at a pinned version. Create an Application pointing at a repository with a Deployment, with no automated policy.

2. Change the manifest in Git. Predict: does the cluster change, and what does the UI show?

3. Enable automated. Change it again. Predict: how long until it syncs?

4. Delete the Deployment with kubectl. Predict: does it come back with selfHeal: false?

5. Enable selfHeal and repeat. Time it.

6. Add a resource to the namespace that is not in the repository. Enable prune. Predict: what happens to it?

7. Break the application so its pods crash-loop, without changing the manifest. Predict: what are the sync and health statuses?

8. Create an AppProject with clusterResourceWhitelist: [] and try to deploy a ClusterRole through it.

9. Delete the cluster.

Related lessons

FluxThe other controller, with a different operating model.

Argo CD vs FluxChoosing between them.

GitOps Drift and ReconciliationOperating with self-heal enabled.

Kubernetes Environment PromotionMoving a digest between Applications.