AI-Assisted Pull Request Automation: What to Automate and What to Keep Human

A pull request carries a lot of work that is not writing code: describing the change, labelling it, routing it, checking it is ready, chasing it when it stalls, and eventually merging it.

Most of that is mechanical. Some of it is judgement wearing a mechanical costume. This lesson is about telling them apart, because the automation that fails is invariably the automation that quietly took over a decision.

Before you start

• GitHub Actions — the execution layer
• Copilot code review — the review layer this complements
• Pull request summaries — the descriptions this automates

What to automate, and what not to

The line: automate the production of information; do not automate the decisions made from it.

Task	Automate?	Why
Drafting a description from the diff	Yes	Information, author edits it
Labelling by changed paths	Yes	Deterministic, cheap to correct
Requesting reviewers via CODEOWNERS	Yes	Already a rule
Flagging a missing test file	Yes	Observation, not a verdict
Summarising a large change for reviewers	Yes	Reading aid
Nudging a stale pull request	Yes	Reminder
Deciding a change is low-risk	No	Requires context
Approving	No	Accountability
Merging on AI assessment	No	The decision itself
Closing a pull request as unnecessary	No	Judgement about intent

Everything in the “yes” column produces something a human then acts on. Everything in the “no” column replaces the human.

Caution

Auto-merge triggered by an AI assessment of risk is the single most damaging pattern in this area. It converts a probabilistic judgement into an irreversible action, and it removes the layer that was supposed to catch the cases where the judgement was wrong. Auto-merge gated on deterministic checks — tests pass, required approvals present — is an entirely different thing and is fine.

Descriptions

The highest-value automation, because descriptions are written last, by a tired author, and read by everyone.

What works: generate a draft from the diff, populate it into the pull request body, and require the author to review it before requesting review.

A minimal workflow shape, using the CLI in a non-interactive prompt:

name: Draft PR description

on:
  pull_request:
    types: [opened]

permissions:
  contents: read
  pull-requests: write

jobs:
  describe:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - name: Generate draft
        run: |
          git diff origin/${{ github.base_ref }}...HEAD > /tmp/change.diff
          # Generation step writes /tmp/description.md
      - name: Post as a comment
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          gh pr comment "${{ github.event.pull_request.number }}" \
            --body-file /tmp/description.md

Two deliberate choices in that shape:

Post as a comment, not into the body. A comment is a draft the author can accept. Overwriting the body destroys anything the author already wrote, and authors who write good descriptions are exactly the ones you least want to overwrite.

Minimal permissions. contents: read and pull-requests: write. Nothing else. A description generator has no reason to hold anything more.

What the draft should contain: what changed, why (from the linked issue if there is one), what a reviewer should look at, and what is explicitly out of scope. The “why” is the part a model cannot reliably produce, and the part the author must supply.

Note

A generated description is a starting point that captures what changed. The author supplies why, and a description that never gets edited is usually a description that says nothing a reviewer could not get from the diff.

Labelling and routing

Deterministic where possible; AI only where the input is genuinely unstructured.

Path-based labelling needs no model. labeler with a configuration file maps changed paths to labels, and it is correct every time. Reach for a model here and you have added latency and cost to something a glob already solved.

Size labelling needs no model. Line counts.

Reviewer routing is CODEOWNERS. It is the mechanism, it integrates with branch protection, and it does not require anything to run.

Where a model genuinely helps: classifying an issue or a pull request whose input is prose. “Is this report a bug, a feature request or a support question” is a text classification problem over content nobody structured. Path globs cannot do it.

The rule worth holding: if a deterministic rule can do it, a deterministic rule should do it. Models are for unstructured input. Using one to read a file path is a cost with no corresponding benefit, and it introduces variability into something that was previously exact.

Merge readiness

Useful automation with a sharp boundary.

A readiness checklist is information. Tests pass, coverage did not drop, no TODO markers added, the changelog was updated, migrations have a rollback. Posting that as a comment helps everyone.

A readiness verdict is a decision. “This pull request is safe to merge” is not a checklist result. It is a judgement, and delegating it removes the layer that catches the case where the checklist was insufficient.

The distinction shows up in the wording. A comment saying “no changelog entry found for a change touching src/api/” is an observation somebody acts on. A comment saying “ready to merge” is a recommendation people will follow without reading, which is functionally an approval without the accountability.

Write the automation to report facts. Let the humans conclude.

The pipeline

The mechanics — listing, labelling, requesting reviewers, checking merge readiness — are ordinary API calls. Pull Request Automation with the GitHub API covers those endpoints; this section is about where AI fits around them.

Automated pull request lifecycle

A vertical sequence: a pull request is opened; mechanical steps apply labels, size and reviewer routing deterministically; generated steps draft a description and change summary that the author edits; verification runs build, tests and scanners as blocking deterministic checks; AI review applies criteria to the diff advisorily; human review covers approach, context and accountability and is required; and merge is gated on checks and approvals, never on an AI assessment.

Two properties of that pipeline are worth stating explicitly.

The blocking stages are deterministic. Build, tests, scanners, required approvals. Nothing probabilistic gates the merge.

The AI stages are advisory and come before the human one. They produce material the human uses. They do not replace the stage after them, and the pipeline is arranged so that removing every AI stage leaves a functioning process — which is the test of whether the automation is layered correctly.

Triage automation

The other side of the pull request lifecycle: the incoming stream that has to be sorted before anybody can work on it. Issues and pull requests both arrive as unstructured text, and this is where a model genuinely outperforms a rule.

What triage automation should produce:

A classification. Bug, feature request, question, duplicate-looking, spam. Applied as a label so the existing views work.

A completeness check. A bug report with no reproduction steps, no version and no environment is a report nobody can act on. A comment naming what is missing gets it filled in while the reporter is still present — which is the entire difference between a report that becomes a fix and one that dies in three weeks.

A likely-area guess. “This mentions the export path; the relevant code is under src/export/.” Useful, occasionally wrong, and cheap to correct.

A duplicate suggestion. “This resembles #1183.” Phrased as a suggestion, never as an action — a model closing an issue as duplicate when it is not is a bad experience for somebody who took the time to report something.

What triage automation must not do:

Close anything. Ever, on its own assessment. A wrongly closed issue is a person told their report does not matter, and the cost of that is not recovered by the time saved.

Assign to a person. Routing to a team label is fine; putting a name on it commits somebody else’s time on a guess.

Set priority. Priority is a statement about what the team will do next, and that is a decision made by people who know what else is in flight.

The shape that works: the automation reduces the reading, and a person does the sorting. A maintainer opening a triage queue where every item is labelled, has a completeness note and a likely area is doing a different job from one opening fifty unstructured reports — and it is still their job.

Note

Triage on a public repository reads text written by anybody, which is the same untrusted-input property covered in secure AI code review. The defence is identical: the automation has no authority. It labels and comments. Nothing it can be persuaded to output closes an issue, grants access or merges anything.

Stale pull request handling

A place automation helps and a place it does damage, depending on what it does.

Helpful: a comment after seven days of inactivity naming what the pull request is waiting on — a review, a change from the author, a failing check. That is information somebody needs.

Damaging: closing pull requests automatically after N days. The pull request was not abandoned; it was waiting for a review that never came, and closing it moves the failure from visible to invisible. Teams that do this end up with clean pull request lists and unresolved work.

The useful automation: a weekly digest of open pull requests grouped by what is blocking them. That turns a list into a set of actions, and it surfaces the pattern — “eleven pull requests waiting on the same reviewer” — which is a resourcing problem rather than a pull request problem.

A model helps here in summarising why each is stalled from the thread, where a status field alone cannot distinguish “waiting for review” from “author is mid-rework after review”.

Building it: the CLI in a workflow

The generation step in the earlier workflow was left as a comment. Here is the shape it takes in practice, using Copilot CLI in non-interactive mode.

      - name: Generate draft description
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          git diff "origin/${{ github.base_ref }}...HEAD" > /tmp/change.diff
          copilot -p "Read /tmp/change.diff and write a pull request description.
            Structure: what changed, what a reviewer should look at, what is out
            of scope. State facts from the diff only. Do not speculate about
            motivation. Do not claim the change is correct or tested."             --allow-tool='shell(cat)'             > /tmp/description.md

Four things in that invocation are deliberate:

-p for non-interactive. The workflow has no terminal. The prompt is the whole instruction.

A narrow --allow-tool. Reading a file is all this task needs. The default-deny posture means anything not listed cannot run, and a description generator with shell access to git push is a description generator that can push.

“State facts from the diff only.” Without this, generated descriptions confidently assert intent the diff does not contain — “improves performance” on a change that reorganises code. The instruction does not eliminate it, and it reduces it noticeably.

“Do not claim the change is correct or tested.” A description asserting that a change is tested, written by something that did not run the tests, is exactly the kind of claim that makes a reviewer relax. This is the editorial principle of the whole pillar applied to a single prompt line: a completed generation is not evidence of correctness.

What to do about a bad draft. Nothing automatic. The author reads it and either uses it or does not. A pipeline that retries until the output looks plausible has optimised for plausibility, which is not the property you want.

Caution

Run generation steps only for pull requests from the base repository, or accept that the diff being read is untrusted content. Neither is wrong — the generator has no authority — but be deliberate about it, and never combine it with pull_request_target and a head checkout.

Cost, noise and quota

Automation runs on every pull request, which multiplies whatever it costs.

Runner minutes. A generation step on every pull request open, on a busy repository, is real spend. Scope it — to certain paths, to non-draft pull requests, to pull requests over a size threshold.

Model usage. Consumes from your plan’s allowance. Check current terms rather than assuming; this lesson deliberately quotes no numbers.

Comment volume. The scarcest resource. Every automated comment competes for attention with the review comments that matter. Two automated comments per pull request is tolerable; six is a pull request nobody reads.

Consolidate. One comment that updates in place beats five comments appended over time. gh pr comment --edit-last, or a sticky comment action, keeps the thread readable.

Change summaries for reviewers

Distinct from the description, and more useful on large changes than most teams expect.

A description tells a reviewer what the author intended. A change summary tells them how to read the diff: which files carry the substance, which are mechanical, and what order to read them in.

Substantive (read these):
  src/billing/proration.ts        — the actual logic change
  src/billing/proration.test.ts   — new cases

Mechanical (skim):
  src/**/*.ts  (14 files)         — import path updates from the move
  package-lock.json               — regenerated

Suggested reading order:
  1. proration.ts, the calculate() function
  2. the new test cases
  3. everything else

On a 40-file pull request where 36 files are an automated rename, that summary is the difference between a reviewer finding the four files that matter and a reviewer giving up.

Why it beats the description for this job. The author knows which files matter and consistently fails to say so, because it is obvious to them. A summary generated from the diff has no such blind spot — it is reading the change cold, which is exactly the reviewer’s position.

The honest limitation. “Substantive” is a judgement, and the classification will sometimes put a consequential one-line change in the mechanical bucket. Present it as a suggested reading order rather than a claim about importance, and the failure mode is a reviewer reading in a slightly odd order rather than a reviewer skipping something.

Scope it to large pull requests. Below about fifteen files, the summary costs more attention than it saves. A size threshold in the workflow condition handles this.

Permissions

Automation that writes to pull requests needs permissions, and the defaults are usually too broad.

Scope the GITHUB_TOKEN per job. permissions: { contents: read, pull-requests: write } is enough for commenting and labelling. Do not grant write on contents to a workflow that only comments.

Do not use pull_request_target to run generation against fork content. It runs with repository secrets in the base context; combining it with a checkout of untrusted head content is the classic Actions failure and it is not made safer by the workflow being AI-related. See Workflow security.

Pin actions to a commit SHA. Automation touching pull requests is a supply-chain surface like any other.

No production credentials. A pull request automation workflow has no reason to hold cloud credentials, deployment keys or database access. If yours does, the workflow is doing two jobs and should be two workflows.

Keep the write surface to comments and labels. A pull request automation that can push commits, change branch protection or dismiss reviews has crossed from producing information into taking actions, and the permission block is where that crossing is visible.

Knowing whether the automation is worth keeping

Pull request automation accretes. Every workflow was a good idea once, and repositories end up with nine of them producing comments nobody has read in a year.

An annual pass, or whenever the pull request threads start looking crowded:

For each automated comment, ask when somebody last acted on one. Not “is it accurate” — accuracy is easy and irrelevant if nobody reads it. A correct comment nobody acts on is pure cost.

Check the generated descriptions people actually shipped. If they are unedited, either the generation is excellent or nobody is reading it. Compare a few against their diffs and you will know which.

Count the automated comments on a typical pull request. Above three, the review comments are competing with your own tooling for attention.

Look at what the automation was protecting against. A checklist item for “changelog updated” is worthwhile if changelog entries were being missed. If the team has internalised it, the check now fires on every pull request to report a thing that always passes, which is noise with a good origin story.

Ask whether any of it has started deciding. Automation drifts toward decision-making by small increments — a comment becomes a status check, a status check becomes required, a required check becomes the reason nobody reads the diff. Nothing announces the transition.

Delete rather than tune. An automation nobody acts on should be removed, not made more accurate. The attention it consumes is spent whether or not its output is right.

The healthy end state is fewer, better automations: one consolidated comment, deterministic checks that block, and a draft description the author edits. Everything else is a candidate for deletion, and most repositories are carrying several.

Common mistakes

Auto-merging on an AI assessment. Converts a probabilistic judgement into an irreversible action.

Overwriting the pull request body. Destroys what the author wrote, and the best authors lose the most.

Using a model where a glob would do. Cost, latency and variability for no gain.

Automated closure of stale pull requests. Moves a visible problem out of sight.

Six automated comments per pull request. Trains people to skip the whole section, including the review.

Over-broad workflow permissions. A commenting workflow with contents: write is a larger blast radius than the task needs.

Automating the description and never reading it. A description nobody edited says only what the diff already said.

Treating a readiness checklist as a verdict. The wording matters: report facts, do not conclude.

Mental model

Automation produces information. People make decisions.

Every worthwhile pull request automation is an application of that split. The description is information; whether it is accurate is the author’s call. The review is information; whether a finding matters is the reviewer’s call. The checklist is information; whether to merge is somebody’s responsibility.

The test for any proposed automation: if this runs and is wrong, does a human still have to notice, or does something happen? If something happens, it is on the wrong side of the line.

What you learned

• Automate the production of information; never automate approval, merge decisions or closure
• Post generated descriptions as comments, not into the body
• Deterministic rules beat models for path-based labelling, sizing and routing
• A readiness checklist reports facts; a readiness verdict is a decision in disguise
• Blocking stages must be deterministic; AI stages are advisory and sit before the human one
• Consolidate automated comments — attention is the scarcest resource in a pull request
• Scope workflow permissions narrowly and keep production credentials out entirely

Exercise

Use a disposable repository.

1. Add a workflow that posts a comment on pull request open with the changed file list and line counts. Scope permissions to contents: read and pull-requests: write.

2. Extend it to post a draft description as a comment. Open a pull request. Predict: how much of the draft would you change before using it?

3. Add path-based labelling with a configuration file, no model. Compare its accuracy and latency to a generated equivalent.

4. Add a readiness checklist comment reporting facts only — tests, changelog, migration rollback. Note how differently it reads from a comment that concludes “ready to merge”.

5. Change the checklist comment to update in place rather than appending. Push three times. Compare the thread.

6. Open the pull request from a fork and confirm the workflow’s permissions behave as you expect for fork pull requests.

7. Delete the repository.

Related lessons

GitHub Copilot code reviewThe review layer this pipeline surrounds.

AI pull request summariesWriting the descriptions this automates.

Agentic CI/CDWhere AI belongs in a pipeline, and where it does not.

Workflow securityPermissions and the fork trust boundary.