Agentic CI/CD

A pipeline is a machine for making the same decision the same way every time. That is its entire value.

Putting a model inside one is therefore a delicate operation, because a model does not make the same decision the same way every time. Done well, it adds analysis a deterministic step could not produce. Done badly, it makes your deployment gate probabilistic, which is a strange thing to have chosen on purpose.

Before you start

• GitHub Actions — the execution model
• Workflow security — permissions and the trust boundary
• AI coding agents explained — what bounds an agent

The rule

Deterministic steps decide. AI steps inform.

Everything in this lesson is an application of that. A test suite decides whether the build proceeds. A security scanner decides whether a vulnerability blocks. An AI analysis produces a comment somebody reads.

The test to apply to any proposed AI step in a pipeline: if this step is wrong, does something happen automatically? If yes, it is in the wrong place.

Danger

Never gate a deployment on an AI assessment. “The model judged this release low-risk” is not a release criterion — it is a probabilistic judgement standing where a deterministic check should be, and it fails in exactly the cases where the check mattered. Deployment gates are tests, scans, approvals and deterministic policy.

Where AI genuinely helps

Six places, all advisory, all producing something a person reads.

Explaining a failure. A CI failure with 400 lines of stack trace, reduced to “the migration test failed because the fixture expects a column added in the migration that this branch does not include.” Genuinely useful, and it saves the ten minutes somebody would spend scrolling.

Summarising a release. What changed between two tags, grouped by area, with the operationally significant items called out. See AI release engineering.

Flagging risk for a human. “This release contains a migration and a change to the auth middleware” is information that helps somebody decide. It is not a decision.

Triaging flaky tests. Reading failure history and grouping tests by likely cause. The output is a list somebody investigates.

Drafting an incident timeline. From commits, deploys and alerts, at the moment when nobody has time to write one. The draft is wrong in places and it is far easier to correct a draft than to reconstruct a sequence from four systems at two in the morning.

Reviewing infrastructure plan output. A terraform plan explained in prose, with destructive operations highlighted. The apply still requires a human approval and a reviewed pipeline.

The pattern across all six: the AI step reads something long and produces something short. That is where it is genuinely strong, and none of those outputs makes anything happen on its own.

Where it does not belong

As a required status check. A required check must be deterministic, or your merge gate is non-deterministic and a re-run can produce a different answer. Nobody wants to debug that.

Deciding whether to deploy. Covered above; it is the most damaging pattern in this area.

Auto-merging. An assessment of risk converted into an irreversible action.

Rolling back automatically on an AI judgement. A rollback is a deployment. Automate rollback on deterministic signals — error rate, health check, latency threshold — never on an interpretation.

Modifying infrastructure. terraform apply, kubectl apply, migrations against a real database. Route these through a reviewed pipeline that runs after a human approved the plan.

Holding production credentials. No AI step in a pipeline needs them. If yours appears to, the step is doing two jobs.

The shape of a good AI step

name: Explain CI failure

on:
  workflow_run:
    workflows: ["CI"]
    types: [completed]

permissions:
  contents: read
  actions: read
  pull-requests: write

jobs:
  explain:
    if: github.event.workflow_run.conclusion == 'failure'
    runs-on: ubuntu-latest
    continue-on-error: true
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
      - name: Collect the failure
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          gh run view "${{ github.event.workflow_run.id }}" \
            --log-failed > /tmp/failure.log
      - name: Explain it
        run: |
          copilot -p "Read /tmp/failure.log. In under 200 words, state which
            step failed, the most likely cause, and the file to look at.
            If the cause is not clear from the log, say so rather than
            guessing." \
            --allow-tool='shell(cat)' \
            > /tmp/explanation.md
      - name: Comment
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          gh pr comment "${{ github.event.workflow_run.pull_requests[0].number }}" \
            --body-file /tmp/explanation.md

Six deliberate properties:

It runs after CI, not inside it. The pipeline’s own result is already determined. Nothing here can change it.

continue-on-error: true. If the analysis fails, nothing else does. An advisory step that can break a pipeline is not advisory.

A timeout. An agent step without one can consume a runner indefinitely.

Narrow permissions. contents: read, actions: read, pull-requests: write. Nothing else, and certainly not contents: write.

A narrow tool list. Reading a file. That is the whole task.

“Say so rather than guessing.” The instruction that most improves output quality in this context. Without it, an unclear log produces a confident wrong answer, which is worse than no comment at all.

The pipeline

AI steps in a delivery pipeline

A vertical sequence: commit; build and test as a blocking deterministic stage; security scanning as blocking; AI analysis as an advisory non-blocking stage; human review; deployment approval; deploy; and post-deploy monitoring with deterministic rollback triggers.

The property to check in your own pipeline: remove every AI stage and it still works correctly. If removing one breaks the gate, the gate depended on it, and that is the design error this lesson is about.

Failure triage in practice

The highest-value AI step in most pipelines, and worth doing properly rather than as a one-line prompt.

The problem it solves. A failed CI run produces output measured in thousands of lines. The relevant part is often twenty of them, somewhere in the middle, and finding it is a skill that correlates strongly with how long you have worked on this codebase. New team members spend the longest.

What to feed it. Not the whole log — the failed steps only. gh run view --log-failed is substantially better input than the full log, because the noise is already gone and the model spends its attention on the part that matters.

What to ask for. A specific, short structure:

Read the failure log and answer, in this order:

1. Which step failed?
2. What is the immediate error?
3. What is the most likely cause, given the diff in this pull request?
4. Which file should be looked at first?

Keep it under 200 words. If the cause is not determinable from the log,
say "cause unclear from the log" and stop. Do not speculate about causes
you cannot see evidence for.

The last two sentences do most of the work. Without them, an unclear log yields a confident wrong answer that sends somebody to the wrong file — which is worse than no comment, because they trust it.

Give it the diff as well as the log. A failure explanation that can reference what changed is substantially more useful than one working from the log alone. git diff origin/main...HEAD into a file, and a tool permission to read it.

Known-flaky tests deserve a note. If your suite has tests that fail intermittently, saying so in the prompt stops the analysis confidently attributing a flake to the diff. A list of known-flaky test names is cheap context with a large effect.

Where it is weakest. Failures whose cause is not in the log at all — an infrastructure problem, a timeout under load, a race that only manifests under CI parallelism. The “say so rather than guessing” instruction is what keeps those cases honest, and it is the single most important line in the prompt.

Permissions

The most common way this goes wrong is not the AI at all — it is the workflow around it.

Scope GITHUB_TOKEN per job. Read what you read, write only where you comment. A default of permissions: read-all at workflow level with narrower per-job blocks is a good posture.

Never pull_request_target with a head checkout. It runs with repository secrets in the base context. Combining it with a checkout of untrusted pull request content hands those secrets to submitted code. This is the classic Actions failure and adding an AI step does not change it.

Pin actions to a commit SHA. A pipeline step that runs a model is a supply-chain surface like any other.

Separate the analysis job from anything privileged. A job that comments on pull requests and a job that deploys should not be the same job, should not share a token, and ideally should not be in the same workflow file.

No production credentials in an AI step. Stated three times in this lesson because it is the one that causes real damage.

Flaky test triage

A second high-value application, and one where the analysis is genuinely hard to do deterministically.

The problem. A test that fails one run in twenty is worse than a test that always fails, because it trains the team to re-run rather than investigate. Over a year a suite accumulates a dozen of them and CI stops being a signal.

What deterministic tooling gives you. Failure counts per test, which run failed, and on which commits. That is real data and it is where any analysis should start.

What the AI step adds. Reading the failure output across many occurrences of the same test and grouping them by apparent cause — a timeout, an ordering dependency, a shared fixture, a real intermittent bug. That grouping is the part that takes a person an afternoon and produces a shortlist worth acting on.

A weekly scheduled shape:

Collect. Query the last N runs for tests that both passed and failed on the same commit. This is deterministic and is the actual detection step.

Analyse. For each such test, feed the failure outputs and ask for a likely category and one suggestion for investigating it.

Report. One issue, or one comment on a tracking issue, listing the tests ranked by failure frequency with the analysis attached.

What it must not do:

Quarantine tests automatically. Skipping a flaky test on an interpretation of why it is flaky removes coverage on a guess, and quarantined tests are never un-quarantined.

Modify tests. The output is a report. A person decides whether the test or the code is wrong.

Close the issue when the test stops failing. A test that stopped failing might be fixed, might be skipped, might not have run.

The detection is deterministic and the categorisation is advisory — which is the same split as everywhere else in this lesson, applied to a problem where the advisory half is genuinely doing work.

Agents that open pull requests

A step beyond analysis: a scheduled workflow where an agent makes a change and opens a pull request.

Legitimate uses: dependency updates, generated documentation, formatting sweeps, mechanical migrations across many files.

The rules that make it safe:

It opens a pull request. It never pushes to a protected branch. No exceptions, no bypass rules.

Its permissions are minimal. Read, and write to its own branch. Not contents: write on the default branch.

Its tool list is scoped to the job. A dependency updater needs the package manager and the test command, not arbitrary shell.

It is scheduled, not triggered by untrusted input. A workflow an outsider can trigger by opening an issue is a workflow an outsider can aim.

Somebody reviews the output. A scheduled agent producing pull requests nobody reads is a queue, not automation.

Caution

A scheduled agent with contents: write and no branch protection is the single most dangerous configuration in this cluster. It has write access to your default branch, runs unattended, and nobody is watching. Branch protection with no bypass for the agent identity is what makes the rest of it acceptable.

Deployment: what actually gates

Since the rule is that AI never gates a deployment, it is worth being concrete about what does — because teams reaching for an AI risk assessment are usually filling a gap in this list.

Gate	Mechanism	Deterministic?
Tests pass	Required status check	Yes
Security scan clean	Code scanning as a required check	Yes
No vulnerable dependencies	Dependency review	Yes
Human approved the change	Required review in branch protection	Yes — a person decided
Human approved the deploy	Environment protection rule	Yes — a person decided
Deploy window respected	Environment wait timer or a scheduled trigger	Yes
Previous deploy healthy	A check against your monitoring	Yes

Environment protection rules are the mechanism most teams underuse. A required reviewer on a production environment means the deploy waits for a named person, and that approval is recorded. It is the correct home for “somebody senior should look at this before it goes out”, which is what an AI risk assessment is usually being asked to approximate.

Where the AI step belongs relative to these: before the approval, producing the summary that the approver reads. “This release includes a migration, a change to auth middleware and eleven dependency bumps” is exactly what somebody wants in front of them when deciding whether to click approve — and it is information, delivered to a person who then decides.

The failure to avoid: an approver who reads the AI summary instead of the change. The summary should shorten the approver’s route to the relevant part of the diff, not replace the diff. If your release approvals have become “read the generated summary, click approve”, the summary has quietly become the gate, without anybody having configured it that way.

Cost and runtime

Pipelines run constantly, and an AI step multiplies by the number of runs.

Scope by trigger. An explanation step that runs only on failure costs nothing on the ninety per cent of runs that pass.

Scope by branch. Release-note generation belongs on release branches, not on every push.

Set timeouts. A model step is the most likely thing in your pipeline to hang.

Cache what you can. The analysis usually needs only a log or a diff, not a full dependency install — and a job that skips npm ci entirely is both cheaper and faster to the comment.

Watch comment volume. A pipeline that comments on every run is a pipeline whose comments nobody reads, which is the same failure as everywhere else in this pillar.

Untrusted input in a pipeline

A pipeline step that reads a diff, an issue body or a pull request description is reading text somebody else wrote. On a public repository, anybody.

Why it is usually fine. The steps described in this lesson comment. They hold pull-requests: write and nothing else. Persuading one to produce a misleading comment achieves a misleading comment.

Why it stops being fine. The moment an AI step’s output feeds something that acts. A step that writes a label, and a later job that behaves differently based on that label, has built a path from submitted text to pipeline behaviour. That path is not obvious in either workflow file, because it is a property of the pair.

The check to run on your own pipeline: trace every AI step’s output. Where does it go? If it goes into a comment a human reads, that is the design. If it goes into a variable, a label, a file another job reads, or an output another step consumes, you have a control-flow dependency on an interpretation of untrusted text.

Structural defences:

Keep AI output terminal. It goes to a comment or a summary. Nothing downstream reads it.

Never ${{ }}-interpolate model output into a shell command. The same script-injection problem as interpolating a pull request title, with a longer and more creative source of text. Pass it through a file or an environment variable, never into the command line the shell will parse.

Separate the reading job from any privileged job. Different jobs, different tokens, no shared outputs.

Assume the text is adversarial on public repositories. Not because an attack is likely, but because the design that survives an adversarial reading is also the design that survives a confusing log.

The parallel with secure AI code review is exact: the defence is that the step has no authority, and the way you lose that defence is by wiring its output into something that does.

Common mistakes

Making an AI step a required check. Non-deterministic gate.

Gating deployment on an AI risk assessment. The most damaging pattern here.

Automatic rollback on an interpretation. Roll back on measured signals.

Broad workflow permissions. contents: write on a commenting job.

pull_request_target with a head checkout. Unrelated to AI, still the worst thing in the file.

No continue-on-error on advisory steps. An advisory step that breaks the build is not advisory.

No timeout. Agent steps hang.

A scheduled agent with default-branch write access. Unattended, unwatched, and unbounded.

Commenting on every run. Trains people to ignore the comments, including the useful ones.

Introducing it to an existing pipeline

The rollout order that avoids the failure modes above.

1. Start with failure explanation. It runs only on failure, it cannot affect a passing run, and its value is immediately visible. If it turns out to be unhelpful, deleting one workflow file undoes it.

2. Run it for a month without telling anybody to rely on it. The comments are there; nobody is expected to act on them. This gives you an honest read on whether they are useful, uncontaminated by people trying to make the experiment succeed.

3. Check the accuracy yourself. For twenty failures, was the explanation right? A step that is right most of the time is useful. One that is right half the time is worse than nothing, because it costs the reader more to check it than to read the log.

4. Add a second advisory step only if the first earned its place. Release summaries, or flaky test triage. One at a time.

5. Audit where the outputs go. Confirm every AI step’s result terminates in something a human reads. Do this before adding the third step, not after the tenth.

6. Never promote an advisory step to a gate. If a step becomes something the team relies on for a decision, that is the moment to ask what deterministic check should be making it instead.

What to watch for over time. Advisory steps become load-bearing by habit rather than by configuration. Nobody edits a workflow to make the failure explanation authoritative; people simply stop reading logs. Noticing that drift is a periodic conversation rather than a technical check, and it is worth having once a quarter alongside the rest of the pipeline review.

Mental model

The pipeline is the machine. The AI is a colleague who leaves notes on it.

Notes are useful. A well-written note explaining why the build failed saves real time. But nothing in the machine’s operation depends on the note being there or being right, and if the colleague is off sick the machine runs exactly as before.

The moment a note starts operating a lever, you no longer have a deterministic pipeline — you have a probabilistic one with a deterministic reputation, and that combination is worse than either honestly labelled alternative.

What you learned

• Deterministic steps decide; AI steps inform, and the test is whether a wrong step causes something to happen
• Good uses read something long and produce something short: failure explanations, release summaries, risk flags
• Never a required check, never a deployment gate, never an automatic rollback trigger
• Advisory steps need continue-on-error, a timeout and narrow permissions
• Removing every AI stage should leave a pipeline that still works correctly
• Scheduled agents open pull requests and never push to protected branches
• No AI step in a pipeline needs production credentials

Exercise

Use a disposable repository. No production credentials.

1. Add a workflow that runs on CI failure and comments an explanation. Scope permissions to contents: read, actions: read, pull-requests: write.

2. Break a test deliberately. Read the explanation. Predict: how much time would it have saved?

3. Remove continue-on-error and make the analysis step fail. Predict: what does the pipeline report now, and is that acceptable?

4. Try to add the analysis job as a required status check. Re-run it twice on the same commit. Predict: are the results identical?

5. Add a scheduled workflow where an agent opens a pull request for a mechanical change. Confirm it cannot push to the default branch.

6. Remove the branch protection rule and observe what the agent identity is then able to do. Restore it.

7. Delete the repository.

Related lessons

AI release engineeringRelease analysis that informs without deciding.

Repository maintenance agentsScheduled agents done safely.

Workflow securityPermissions, secrets and the fork boundary.

AI pull request automationThe same information-versus-decision split.