Kubernetes Secrets and GitOps: Handling What Cannot Be Committed

GitOps says desired state lives in a versioned repository. Secrets cannot live in a repository. That is the problem, stated completely.

Everything below is a way of resolving that contradiction, and each resolution trades one risk for another. None of them makes the problem disappear.

Before you start

• GitOps Explained — why desired state must be readable by the controller
• Secret scanning — detecting what got committed anyway
• Rotating exposed credentials — the first step when it does

Base64 is not encryption

The single most important sentence in this lesson.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: c3VwZXItc2VjcmV0LXBhc3N3b3Jk

That value decodes in one command, with no key, by anybody:

echo c3VwZXItc2VjcmV0LXBhc3N3b3Jk | base64 -d

Base64 is an encoding, chosen so that arbitrary bytes survive YAML. It provides no confidentiality whatsoever. A Secret manifest committed to a repository is a plaintext credential that merely looks technical.

stringData is the same thing without the encoding step — Kubernetes base64-encodes it for you.

Danger

A Kubernetes Secret manifest in a Git repository is an exposed credential. Not “potentially exposed” — exposed, to everyone with repository read access, in every clone, in every fork, and permanently in history. If you find one, the response is rotation first, then removal, then history. The credential is valid throughout.

A second thing worth knowing: Kubernetes Secrets are not encrypted at rest in etcd by default either. Encryption at rest is a cluster configuration, and a cluster without it stores Secret values in etcd in a form anybody with etcd access can read. That is a separate problem from the Git one and it is worth confirming rather than assuming.

The approaches

Four families, each protecting against something different.

Approach	In the repository	Decrypted by	Protects against
External secret operator	A reference	Nothing — fetched at runtime	Repository exposure entirely
Encrypted in repository (SOPS)	Ciphertext	The cluster, with a key	Repository read access
Sealed secrets	Ciphertext for one cluster	That cluster’s controller	Repository read, cross-cluster reuse
Out of band	Nothing	Not applicable	Everything, at the cost of GitOps

External secret operators

The approach that resolves the contradiction rather than working around it.

An operator runs in the cluster, reads from a secret manager — a cloud provider’s, HashiCorp Vault, OpenBao — and creates Kubernetes Secrets from what it finds. The repository contains a reference, never a value.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: production/database
        property: password

Nothing secret is in the repository. The manifest names where to find a value, not the value.

Rotation happens in the secret manager. The operator picks it up on its refresh interval, with no commit and no deployment. This is the strongest practical argument for this approach: rotation stops being a code change, which means it can be automated, scheduled, and done during an incident by somebody who does not have repository write access.

Access control lives in the secret manager, with its own audit log answering who read what and when. That log is frequently the specific thing an auditor asks for, and it is the one capability neither of the in-repository approaches can offer — a decryption that happens inside a cluster leaves no central record of who triggered it.

The costs:

Another component to operate. An operator to install, upgrade and monitor.

A runtime dependency. If the secret manager is unreachable, new Secrets cannot be created and existing ones stop refreshing. A running cluster keeps running, because the Secrets already exist — but a new deployment during an outage of the secret manager will not get its credentials, and a scale-up event will schedule pods that cannot start. Worth knowing before it happens, and worth including in whatever dependency map your incident response uses.

Authentication is a bootstrap problem. The operator needs credentials for the secret manager. Workload identity — the cluster’s own service account authenticating to the cloud provider — solves this properly and is worth the setup; a static credential in a Secret means you have one secret protecting all the others.

The verdict: for most teams with a cloud secret manager already, this is the right answer.

Encrypted in the repository

Encrypt the values, commit the ciphertext, decrypt at reconciliation time. SOPS is the common tool, encrypting values while leaving keys and structure readable.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
stringData:
  password: ENC[AES256_GCM,data:REDACTED,iv:REDACTED,tag:REDACTED,type:str]
sops:
  kms:
    - arn: arn:aws:kms:eu-west-1:111111111111:key/EXAMPLE-KEY-ID

Structure stays readable and diffable. A pull request shows that the password changed without showing what to. This is a real advantage over encrypting whole files.

Flux has native SOPS support. A Kustomization with a decryption block decrypts at reconciliation. Argo CD needs a plugin or a preprocessing step.

Keys can be cloud KMS, so the cluster’s workload identity grants decryption and no key material sits anywhere.

The costs:

Key management is now yours. Rotation, access control, and a recovery plan for a lost key. Lose it and every encrypted value in history is unrecoverable.

Rotation requires a commit. Changing a credential means re-encrypting and merging, which is slower than changing it in a secret manager and considerably more visible.

Ciphertext in history is permanent. If a key is compromised, every value ever encrypted with it in that repository is compromised — including ones rotated years ago. That is a property people underestimate.

Everybody who edits secrets needs decryption access, which is a wider grant than you might intend — editing one value in a file means decrypting the file, so anybody who can change any secret can read all of them in that file. Splitting secrets across files by who needs to edit them limits this and produces a repository with a great many small encrypted files.

The verdict: good where a secret manager is unavailable or the operational dependency is unacceptable, and where you have somewhere to put the key.

Sealed secrets

A controller in the cluster holds a private key. You encrypt with the corresponding public key, producing a custom resource the controller converts into a Secret.

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    password: AgBy3i4OJSWK+PiTySYZZA9rO4...

Encryption is one-way for the author. You can create a sealed secret without being able to decrypt anything, which is genuinely useful — a developer can add a credential without holding the ability to read existing ones.

The ciphertext is scoped, by default to a namespace and name, so it cannot be reused elsewhere in the cluster.

The costs:

The controller’s private key is the whole security model. Back it up, or a cluster rebuild means re-creating every secret from source. Losing that key is the failure that turns a cluster rebuild into a scavenger hunt.

Ciphertext is cluster-specific. The same secret for three clusters is three sealed secrets, and disaster recovery into a new cluster requires the old key or re-sealing everything.

Rotation is a commit, as with SOPS.

The verdict: works well for a single cluster with a disciplined key backup, and gets awkward across a fleet — at which point the per-cluster ciphertext that made it safe becomes the thing making it tedious.

What each actually protects

Being precise, because these are frequently conflated.

All three protect against repository read access. Somebody who can clone the repository does not get credentials.

None protects against cluster access. The decrypted Secret exists in the cluster. Anybody who can read Secrets in that namespace has the value, regardless of how it got there. Kubernetes RBAC is what protects it, and a get secrets permission is the one to audit.

None protects against a compromised controller. The thing that decrypts can decrypt.

Only external secret operators make rotation cheap. The other two make it a commit, which means it happens less often.

Only external secret operators leave nothing in history. With SOPS or sealed secrets, the ciphertext is permanent.

Caution

The question worth asking about your own cluster: who can run kubectl get secret -n production -o yaml? That set of people has every credential in that namespace, whatever mechanism put them there. In many clusters the answer is far broader than anybody intended, and no amount of encryption in the repository changes it.

Rotation

The operational property that separates the approaches.

With an external operator: change the value in the secret manager. The operator refreshes and updates the Secret. No commit, no pull request, and the whole thing takes a minute.

With encryption in the repository: re-encrypt, commit, review, merge, reconcile. Minutes to hours depending on process, and it is visible in history — which is either a feature or an exposure of your rotation cadence, depending on who can read the repository.

Either way, pods do not automatically restart. A Secret’s value changing does not roll the Deployments mounting it. Applications that read secrets at startup keep the old value until something restarts them, which is the same problem ConfigMaps have and is more consequential here. Either use an application that re-reads, or trigger a rollout as part of rotation.

Rotate on a schedule, not on an incident. A team that has rotated a credential recently can do it under pressure. One that has never done it discovers the missing steps during the incident.

The bootstrap problem

Every approach has one: the first secret.

An external operator needs credentials for the secret manager. Workload identity is the answer — the cluster’s service account authenticating to the cloud provider, with no static credential anywhere.

SOPS needs a decryption key. Cloud KMS with workload identity, again, or a key placed at bootstrap.

Sealed secrets generates its own key, which then needs backing up somewhere that is not the cluster.

The pattern: use the platform’s identity mechanism so the first secret is a trust relationship rather than a credential. Where that is impossible, the bootstrap credential is a single high-value secret placed once, out of band, and rotated on a schedule.

Document how it was placed. A cluster whose bootstrap credential was set up eighteen months ago by somebody who has left is a cluster nobody can rebuild.

Choosing

The decision, by circumstance rather than by preference.

You already use a cloud secret manager. External secret operator. The integration exists, rotation is cheap, and nothing sensitive enters the repository.

You have no secret manager and no appetite to run one. SOPS with cloud KMS. The key lives in KMS, the cluster’s identity grants decryption, and there is no new service to operate.

Air-gapped or heavily restricted network. SOPS with a key you manage, or sealed secrets. An external operator needs to reach the manager.

A single cluster, a small team, low secret volume. Sealed secrets is proportionate, provided somebody owns backing up the key.

Many clusters. External operator, strongly. Sealed secrets’ per-cluster ciphertext becomes painful, and SOPS means every cluster needs decryption access to the same key.

Regulated, with an audit requirement on secret access. External operator — the secret manager’s audit log answers “who read this credential and when”, which neither of the other approaches can.

Something already chose for you. A platform product or an existing organisational standard. Consistency across teams is worth more than a marginally better fit.

What not to do: run two approaches because different teams picked differently. Two mechanisms means two key-management stories, two rotation procedures and two things to get wrong, for no benefit.

Secrets in the GitOps controller itself

Easily overlooked: the controller has its own credentials, and they are secrets in a cluster.

Repository authentication. A deploy key or token, stored as a Secret in the controller’s namespace. Read-only unless image automation needs write, and scoped to the one repository.

Registry credentials, where the controller pulls OCI artifacts or charts.

Decryption keys, for SOPS.

Cluster credentials, if you run Argo CD’s hub model — one namespace holding write access to every managed cluster.

The implication: whoever can read the controller’s namespace holds the keys to the platform. That namespace deserves tighter RBAC than most, and it is worth checking who currently has it.

The bootstrapping recursion is real. The controller’s own secrets cannot be managed by the mechanism the controller enables, because it needs them to start. They are placed at bootstrap, out of band, and they are the credentials most likely to be forgotten in a rotation schedule.

Write down what they are and when they were last rotated. A short list, in the platform’s documentation. This is the set of credentials nobody thinks about until an audit or an incident, and reconstructing it afterwards is slow.

Detecting what got committed

Prevention, in three layers, because the migration above assumes you can find the problem.

Push protection blocks recognised credential patterns at push time. It catches provider tokens and API keys reliably; it does not know that an arbitrary base64 string in a data: field is a password. Enable it anyway — it stops the largest category.

A CI check for Secret manifests. Cheap, specific and catches what pattern-matching cannot:

if grep -rlE '^kind:\s*Secret\b' --include='*.yaml' --include='*.yml' .    | grep -v '/sealed-' > /tmp/found.txt && [ -s /tmp/found.txt ]; then
  echo "Unencrypted Secret manifests found:" >&2
  cat /tmp/found.txt >&2
  exit 1
fi

Adjust the exclusion for whatever your encrypted files are called. The point is that a plain kind: Secret in the repository fails the build, which converts a policy into a control.

A pre-commit hook doing the same locally, so the feedback arrives before the push rather than after.

Reviewing what a reviewer can see. A pull request adding a Secret manifest should be obvious in the diff, and .github/CODEOWNERS covering the paths where secrets would appear means somebody who would recognise the problem is asked.

None of these is sufficient alone, and together they catch the accident — which is what almost all of these are. A determined bypass is a different problem, and the answer to that is the RBAC and rotation discipline above rather than a grep.

Common mistakes

Committing a Secret manifest. Base64 is not encryption.

Assuming etcd encrypts Secrets at rest. It is a cluster configuration, not a default.

Not auditing who can read Secrets in the cluster. The repository is protected and the cluster is not.

Losing the sealed secrets private key. Every secret must be recreated from source.

Treating a compromised SOPS key as a forward-only problem. Every value ever encrypted with it is in history.

Not rotating pods after rotating a secret. The old value keeps running.

A static credential for the secret operator. One secret protecting all the others.

Secrets in Helm values files. Committed, plaintext, and exactly where people put them.

No rotation practice. The first attempt happens during an incident.

Encrypting and then relaxing about cluster RBAC. The decrypted value is in the cluster.

Migrating to a managed approach

Most clusters start with Secrets applied by hand and arrive here with a mess to untangle.

1. Inventory what exists. kubectl get secrets -A and work out where each came from. Expect several nobody can account for; those are the interesting ones.

2. Search history for committed secrets. Every repository, including archived ones:

   git log --all -p -- '*.yaml' | grep -iE '^\+.*(kind: Secret|stringData|password:|token:)' | head -50

   Anything that turns up is an exposure, and rotation comes first.

3. Rotate everything you found, before any migration work. The credential is valid while you tidy.

4. Choose one mechanism and write down why.

5. Set up the bootstrap identity — workload identity to the secret manager, or the KMS key policy. Verify it works before migrating anything.

6. Migrate one non-critical secret end to end. Create it in the manager, add the reference, confirm the operator produces the Secret, confirm the application reads it.

7. Migrate the rest, most critical last, verifying each.

8. Remove the hand-applied Secrets only once the managed equivalent is confirmed working. Deleting first produces an outage.

9. Restrict who can create Secrets directly. Until this, the old path still exists and people will use it.

10. Schedule the first rotation. A mechanism nobody has exercised is a mechanism that does not work.

Steps 2 and 3 are the ones with a deadline. The migration can take weeks; an exposed credential should not wait for it.

Non-Kubernetes secrets

The same problem appears elsewhere in this pillar, and the answers rhyme.

Terraform state contains secrets as resource attributes. The answer is a remote backend with access control, and OpenTofu adds client-side encryption.

Ansible Vault encrypts values in a repository, with the vault password as the key that must not be committed. Structurally identical to SOPS, with the same key-management burden.

CI secrets live in the CI system, and the modern answer is OIDC — no standing credential, a short-lived token exchanged per run.

Container registry credentials for pulling private images, which are Kubernetes Secrets and therefore subject to everything above.

The pattern across all of them: the best answer is not to have a long-lived credential at all. Workload identity, OIDC and short-lived tokens remove the thing that needs protecting. Where a static secret is unavoidable, it belongs in a manager with access control and an audit log, referenced rather than copied.

The second-best answer is encryption with a key held elsewhere, which reduces the exposure to whoever holds the key.

The worst answer is a credential in a repository, in any encoding, in any tool.

Reducing how many secrets you have

The approach that beats managing them well: having fewer.

Workload identity instead of credentials. A pod authenticating to a cloud API through its ServiceAccount, exchanged for a short-lived token, needs no stored credential at all. Every cloud provider supports this for Kubernetes, and adopting it removes whole categories of secret rather than protecting them.

Short-lived database credentials. A secret manager that issues a credential valid for an hour, rotated automatically, is a very different risk from a static password that has been in production for three years. The application needs to handle re-authentication, which is a real change and usually a small one.

mTLS instead of shared tokens for service-to-service authentication. A service mesh issuing short-lived certificates removes the API keys services otherwise hold for each other.

OIDC in CI, which this pillar has mentioned repeatedly, for the same reason.

Question every static secret you find. For each one: does this need to exist, or is there an identity-based alternative? The answer is increasingly yes, and each one removed is one fewer to rotate, audit and worry about in a repository.

The realistic end state is not zero secrets. It is a small number of genuinely necessary ones — third-party API keys, credentials for systems that support nothing better — managed properly, with everything else handled by identity. That is a much smaller problem than the one most teams currently have.

Mental model

A secret must reach the cluster without passing through the repository. Either the repository holds a reference and something fetches the value, or it holds ciphertext and something decrypts it. Both move the problem to key or identity management, which is where it belongs.

The framing that prevents most mistakes: ask what each mechanism protects against, and what it does not. Every approach here protects the repository. None protects the cluster, and none removes the need to know who can read a Secret.

What you learned

• Base64 is an encoding; a Secret manifest in Git is a plaintext credential
• Kubernetes Secrets are not encrypted at rest in etcd by default
• External secret operators keep only a reference in the repository and make rotation cheap
• SOPS keeps structure diffable and makes key management your responsibility
• Sealed secrets allow write-only encryption and tie ciphertext to one cluster’s key
• None of these protects against cluster access — RBAC on Secrets is a separate audit
• Ciphertext in history is permanent; a compromised key exposes everything ever encrypted
• Rotating a Secret does not restart the pods using it

Exercise

Use a disposable local cluster. Placeholder values only — no real credentials.

1. Create a Secret manifest with stringData: {password: placeholder-value}. Apply it, then run kubectl get secret NAME -o yaml. Predict: what does the data field contain?

2. Decode it with base64 -d. Note that this needed no key.

3. Check whether your cluster has encryption at rest configured. Predict: before checking, what did you assume?

4. Install SOPS and encrypt a Secret manifest with an age key. Commit the ciphertext to a local repository. Predict: is the key name visible? Is the value?

5. Delete the key and try to decrypt. Predict: is the value recoverable?

6. Check which ServiceAccounts and users can get secrets in a namespace: kubectl auth can-i get secrets --as=SUBJECT -n NAMESPACE.

7. Change a Secret’s value while a pod mounts it as an environment variable. Predict: does the running pod see the new value?

8. Delete the cluster and the key material.

Related lessons

GitOps Drift and ReconciliationWhat the controller does with secrets it manages.

Rotating Exposed CredentialsWhen one reaches a repository.

Preventing .env File CommitsStopping it at the source.

Ansible Vault and GitThe same problem, a different tool.