GitHub Copilot CLI for Kubernetes

Kubernetes produces enormous quantities of structured diagnostic output that almost nobody enjoys reading. That makes it an excellent fit for an agent.

It also has a command surface where a single flag separates describing a resource from deleting a namespace, and where the same command against two contexts means two entirely different things.

Generating a command is not authorising a production change

This distinction matters more here than anywhere else in the pillar, because the commands are short, the context is invisible in the command text, and the effects are immediate.

{/* Reads */}
kubectl get pods -n production

{/* Changes production */}
kubectl delete pods -n production --all

Two commands, similar shape, entirely different consequences. Which cluster you are pointed at is not in either of them.

Before you start

• Copilot CLI for DevOps — the safeguards
• Deploy to Kubernetes — where changes belong
• Copilot CLI for Docker — the images this runs

Establish context first

Before anything else, every session.

kubectl config current-context

What it doesPrints the context the next kubectl command will use.

Why we run itAn agent running kubectl uses whatever context is active. Confirming which cluster that is takes a second and prevents the worst outcome in this lesson.

Expected resultThe current context name.

Better still, work in a shell whose context cannot be production. Separate kubeconfigs, a KUBECONFIG pointing only at development clusters, or a context bound to a view role.

The instruction worth opening an operational session with:

Before running any kubectl command, state which context and namespace you are using. Use read-only commands only until I ask otherwise.

Reading events and rollouts

The highest-value use, and where the output is genuinely hard to read.

{/* Everything about a pod, including its events */}
kubectl describe pod POD -n NAMESPACE

{/* Recent events across a namespace, most recent last */}
kubectl get events -n NAMESPACE --sort-by=.lastTimestamp

This deployment will not roll out. Here is kubectl describe deployment, the pod describe output and recent events. What is wrong?

The failures an agent identifies quickly, because they have distinctive signatures buried in verbose output:

ImagePullBackOff — wrong image name, wrong tag, a missing or misconfigured pull secret, or the wrong registry. The event message names which of those it is, and reading it is faster than guessing.

CrashLoopBackOff — the container starts and exits. The useful logs are from the previous instance:

kubectl logs POD -n NAMESPACE --previous

That --previous flag is the single most useful thing to know here and is frequently forgotten.

Pending with no node assignment — insufficient resources on any node, a taint without a matching toleration, a node selector or affinity rule matching nothing, or a persistent volume claim that cannot be bound. The events distinguish them, and the distinction determines whether the fix is capacity, scheduling configuration or storage.

OOMKilled — the container exceeded its memory limit and was killed. This is a limit problem until proven otherwise, and the reflex to go looking for an application memory leak wastes a great deal of time on services that simply need more than they were given.

Readiness probe failures — the container runs and never becomes ready. Usually a probe pointing at the wrong path or port, or an application whose start-up is slower than initialDelaySeconds allows, which presents as a deployment that never completes rather than as an error.

Writing manifests

Generated Kubernetes YAML has consistent gaps, all of which come from documentation examples being minimal.

No resource requests or limits. A pod without requests is scheduled anywhere and can starve its neighbours; without limits it can consume a node. This is the most common omission and the one with the widest operational consequence.

No security context. Running as root, writable root filesystem, all capabilities.

No probes. Without a readiness probe, traffic reaches a pod that is not ready. Without a liveness probe, a hung container stays in the rotation.

:latest image tags. Non-reproducible, and a pod restart can silently change the running version.

A manifest with the gaps closed:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        fsGroup: 10001
      containers:
        - name: api
          image: ghcr.io/YOUR_ORG/api@sha256:YOUR_DIGEST_HERE
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
          readinessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              port: 3000
            initialDelaySeconds: 30

Two decisions worth explaining.

runAsUser: 10001 is numeric, because runAsNonRoot checks the numeric UID. An image whose USER is a name rather than a number fails this check — which is why the Docker lesson uses a numeric USER.

A memory limit but no CPU limit. Memory is incompressible — exceeding it kills the container — so a limit prevents one pod taking down a node. CPU is compressible, and a CPU limit throttles rather than protects, frequently making latency worse. This is a deliberate and slightly contested choice, and it is the kind of thing a generated manifest will not reason about.

Putting these requirements in path-specific instructions scoped to **/*.yaml under your manifests directory means they apply without restating.

Networking problems

The category people struggle with most, and where a systematic reader helps because the diagnosis is a sequence of narrowing questions rather than one answer.

“Service A cannot reach service B” has a small number of causes, and the commands that distinguish them are all read-only:

Does the Service have endpoints?

kubectl get endpoints SERVICE -n NAMESPACE

No endpoints means the Service’s selector matches no ready pods — either the labels are wrong or the pods are not passing readiness. This is the single most common cause and the fastest to check.

Do the selector and the pod labels actually match?

kubectl get svc SERVICE -n NAMESPACE -o jsonpath='{.spec.selector}'
kubectl get pods -n NAMESPACE --show-labels

A selector of app: api and pods labelled app: api-server produce a Service that resolves and connects to nothing.

Is DNS resolving?

kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup SERVICE.NAMESPACE

Note that this creates a pod — a change, and one to make deliberately in a development cluster rather than reflexively in production.

Is a NetworkPolicy blocking it?

kubectl get networkpolicies -n NAMESPACE

Network policies are default-allow until one exists, then default-deny for what they select. A policy added for one workload frequently breaks another that nobody was thinking about.

Is it the port? The Service’s targetPort must match the container’s actual listening port, and the two are easy to get out of step.

An agent running the first four and correlating them reaches the cause quickly. The reason to know the sequence yourself is that you need to evaluate the conclusion, and “no endpoints” versus “policy blocking” lead to entirely different fixes.

Previews before applying

Kubernetes has two, and both are read-only.

{/* What would change, compared against the cluster */}
kubectl diff -f manifest.yaml

{/* Validate against the API server without persisting */}
kubectl apply -f manifest.yaml --dry-run=server

kubectl diff is the one to insist on. It shows the actual difference between your manifest and what is running, which catches the case where somebody changed something in the cluster directly and your apply would revert it.

--dry-run=server validates against the real API server — including admission controllers and defaulting — which --dry-run=client does not.

Run kubectl diff and explain what would change. Do not apply.

What not to delegate

kubectl apply against a live cluster. Changes belong in a reviewed pipeline with an audit trail, not an interactive session.

kubectl delete in any form. Especially with --all, a label selector, or a namespace. Deleting a namespace deletes everything in it, including persistent volume claims depending on the reclaim policy.

kubectl scale on production. A capacity decision with immediate effect.

kubectl rollout undo. Legitimate during an incident and a decision about which revision, made by somebody who knows what changed.

kubectl edit. Modifies live resources with no record in Git. This is the command that produces drift, and the drift reappears as a surprising kubectl diff months later.

kubectl exec into production. A shell in a production container, with whatever that pod’s service account can reach.

Anything with --force or --grace-period=0. Skips graceful shutdown.

Helm

The same pattern with an extra template layer.

{/* Render templates locally — no cluster contact */}
helm template myrelease ./chart --values values.yaml

{/* What an upgrade would change */}
helm diff upgrade myrelease ./chart --values values.yaml

helm template is read-only and safe to pre-approve. It is also the right first step for any “why is this value not taking effect” question — rendering shows what the templates actually produce, which is frequently not what the values file suggests.

Where an agent helps: explaining a chart somebody else wrote, tracing a value through nested templates, and finding why a conditional block is not rendering. Chart templating is fiddly and the errors are opaque.

Where to be careful: helm upgrade and helm rollback change the cluster. helm uninstall removes everything in the release.

A diagnostic sequence

The commands worth having an agent run, in order, when something is wrong. Each is read-only.

1. What is the state?

   kubectl get pods -n NAMESPACE -o wide

   Status, restart count, node, age. Restart count is the first signal.

2. Why is this pod unhappy?

   kubectl describe pod POD -n NAMESPACE

   The events at the bottom are the useful part.

3. What does the application say?

   kubectl logs POD -n NAMESPACE --tail 200
   kubectl logs POD -n NAMESPACE --previous --tail 200

4. What happened recently in this namespace?

   kubectl get events -n NAMESPACE --sort-by=.lastTimestamp | tail -30

5. Is it a resource problem?

   kubectl top pods -n NAMESPACE
   kubectl describe node NODE | grep -A5 'Allocated resources'

6. Did the deployment actually roll out?

   kubectl rollout status deployment/NAME -n NAMESPACE --timeout=10s
   kubectl rollout history deployment/NAME -n NAMESPACE

An agent running all six and correlating them produces a better first answer than a human running them one at a time — which is most of the value in this lesson. The judgement about what to do stays with you.

Worth asking explicitly afterwards:

Which of those outputs supports your conclusion? Quote the line.

Security review

Pattern-shaped and worth automating with a read-only agent:

• Containers running as root, or without runAsNonRoot
• privileged: true, or capabilities not dropped
• hostNetwork, hostPID or host path mounts
• Missing resource limits
• Secrets as environment variables rather than mounted files
• Service accounts with broad RBAC, or automountServiceAccountToken left on where unneeded
• :latest or unpinned images
• Missing network policies in a cluster that has them

Pair it with a deterministic policy engine in CI. An agent reading a manifest finds patterns; an admission controller enforces them on everything that arrives, which is the distinction between review and control.

GitOps, and where manifests should live

The structural answer to most of this lesson’s cautions.

In a GitOps arrangement, the cluster’s desired state lives in Git and a controller reconciles the cluster toward it. Nobody applies anything by hand — changes are pull requests, and the cluster follows the repository.

That changes what an agent is for, in a good way:

Manifest authoring becomes an ordinary code change. The agent writes YAML, you review a diff, the change goes through review and policy like anything else.

kubectl becomes read-only by default. There is no reason to apply, because applying is the controller’s job. That makes “read-only until I ask” the natural configuration rather than a discipline you have to maintain.

Drift becomes visible. A manual change is reverted by the controller and shows up as a reconciliation, rather than persisting silently until somebody runs kubectl diff.

The audit trail is the Git history. Who changed what, when, reviewed by whom — which an interactive session cannot provide.

Where a cluster is not managed this way, the same properties can be approximated by keeping manifests in Git and applying only from a pipeline. The point is not the particular tooling; it is that the path from intent to cluster runs through review, which is the control that makes everything else on this page less critical.

Instructions for manifest work

Path-specific instructions are worth their length here, because Kubernetes has many conventions that are not inferable and many minimal examples that are unsafe to copy.

.github/instructions/kubernetes.instructions.md

---
applyTo: "k8s/**/*.yaml"
---

Every workload manifest must include:

- `resources.requests` for cpu and memory, and `resources.limits` for memory.
  Do not set a cpu limit unless asked.
- `securityContext` with `runAsNonRoot: true` and a numeric `runAsUser`.
- Container `securityContext` with `allowPrivilegeEscalation: false`,
  `readOnlyRootFilesystem: true` and `capabilities.drop: ["ALL"]`.
- A readiness probe, and a liveness probe with a longer `initialDelaySeconds`.
- Images referenced by digest, never by `:latest`.
- Labels from our standard set: app, component, part-of, managed-by.

Never propose `kubectl apply`, `delete`, `edit`, `scale`, `exec`, `rollout undo`,
or anything with `--force` or `--grace-period=0`. Propose `kubectl diff` or
`--dry-run=server` instead and let me decide.

The last paragraph is the agent-specific half, and it addresses the failure modes this lesson is built around. As everywhere: this shapes rather than enforces. A preToolUse hook matching those verbs and denying them is the enforcement, and in a repository where people run agents against clusters it is worth the effort.

The first half is worth having regardless of AI — it is the manifest checklist, applied at authoring time rather than discovered in review.

Common mistakes

Not checking the context. The one that produces the worst outcomes, and it takes a second.

Applying from a session. Outside the audit trail, the approval and the scoped credentials.

Forgetting --previous on a crash loop. The current instance’s logs are empty; the previous one’s contain the error.

Generated manifests without limits or probes. The most common omission, and the widest consequence.

A non-numeric runAsUser. runAsNonRoot checks the UID.

Treating OOMKilled as an application leak. Check the limit before profiling.

kubectl edit on a live resource. Produces drift with no record.

Assuming --dry-run=client validates. It checks the manifest’s shape locally and never reaches the API server’s admission controllers or defaulting, so it passes things the cluster would reject.

Committing a Secret manifest with a real value. Base64 is not encryption, and it is now in Git.

Resource sizing

A question agents are asked constantly and answer with more confidence than the evidence supports.

What it can do: read your current usage from kubectl top, compare it against configured requests and limits, and identify obvious mismatches — a pod requesting 2Gi and using 80Mi, or one repeatedly hitting its limit.

What it cannot do: know your traffic pattern, your peak-to-average ratio, your growth, or how much headroom your organisation wants. Those determine the answer and none of them is in the cluster.

The useful framing:

Here is kubectl top pods over the last hour and the current requests and limits. Which are obviously mis-sized, in which direction, and what data would I need to size them properly?

That produces candidates and an honest statement of what is missing, rather than a number.

The specific trap: kubectl top is a point-in-time sample. Sizing from one observation of a service with a daily peak produces requests that are correct at 3am. Real sizing needs metrics over time, which is a monitoring system’s job — and an agent with access to one through MCP can query it, which is a much better position than reasoning from a single sample.

Secrets

A specific area where generated manifests get it wrong in a way that is easy to miss.

Kubernetes Secrets are base64-encoded, not encrypted. Anyone with read access to Secrets in a namespace can read them, and they are stored in etcd — encrypted at rest only if the cluster is configured for it. A generated manifest containing a Secret with a real value in it is a credential committed to your repository.

{/* Never commit this with a real value */}
apiVersion: v1
kind: Secret
stringData:
  DATABASE_PASSWORD: hunter2

The alternatives, none of which an agent will suggest unprompted:

A secrets operator that pulls from an external manager at runtime, so nothing sensitive is in Git.

Sealed secrets or an equivalent, where the committed form is encrypted and only the cluster can decrypt it.

Mounted as files rather than environment variables. Environment variables leak into crash dumps, logs and child processes; a mounted file is read by what needs it.

automountServiceAccountToken: false where a workload does not need to call the Kubernetes API, which most do not. The default mounts a token into every pod.

The instruction worth having:

Never generate a Secret manifest containing a real value. Reference an external secret source, and mount secrets as files rather than environment variables.

This connects directly to Pillar 5’s secret prevention — a Secret manifest with a value in it is the same problem as a committed .env, in a format that looks more official.

Mental model

Kubernetes tells you almost everything, in a format nobody wants to read. An agent is very good at the reading. The commands that change a cluster are short, similar-looking, and immediate — which is why the deciding stays with the person who knows which cluster they are pointed at.

What you learned

• Confirm the context before anything; it is not visible in any command
• --previous gets the logs from a crashed container’s prior instance
• ImagePullBackOff, CrashLoopBackOff, Pending and OOMKilled each have distinctive event signatures
• Generated manifests routinely omit resource limits, security contexts and probes
• runAsNonRoot checks the numeric UID, so runAsUser must be a number
• A memory limit protects the node; a CPU limit usually just throttles
• kubectl diff shows the real difference and catches manual drift
• --dry-run=server validates against admission controllers; client does not
• Apply, delete, scale, edit and exec belong outside an agent session

Exercise

Use a local cluster — kind, minikube or similar. Nothing here touches a real cluster.

1. Run kubectl config current-context. Predict: is it what you assumed?

2. Ask for a Deployment manifest with no qualification. Predict: does it include resource limits, probes and a security context?

3. Ask again specifying those requirements. Compare.

4. Apply the manifest with a deliberately wrong image name. Ask the agent to diagnose from describe and events. Predict: does it identify the cause?

5. Make the container exit immediately. Ask again. Predict: does it use --previous?

6. Set a memory limit lower than the application needs. Predict: what does the agent conclude — a leak, or the limit?

7. Change something with kubectl edit, then run kubectl diff against your manifest. Predict: does the drift show up?

8. Delete the cluster.

Related lessons

Deploy to KubernetesWhere cluster changes belong, with approvals.

Copilot CLI for DevOpsThe safeguards this lesson applies.

Copilot CLI for DockerThe images these manifests run.

Signing container imagesAdmission policy, and verifying what runs.