A pull request from a colleague comes with a person you can ask. A pull request from an agent comes with a summary written by the thing that wrote the code.
That is the whole difference, and it changes what review has to do. You cannot ask why. You cannot rely on the author having noticed something odd and mentioned it. The diff and the session log are the entire evidence, and reading them well is a learnable skill with a specific order.
Why green CI means less here
Section titled “Why green CI means less here”The instinct that a passing build is reassuring is calibrated on human-authored code, and the calibration does not transfer.
When a human writes code and CI passes, you have learned that a person who understood the requirement produced something the tests accept. The person is doing most of the work in that sentence.
When an agent writes code and CI passes, you have learned that the loop iterated until the signal went green. The loop was optimising for exactly that signal, which means a green result is closer to a tautology than to evidence.
The distinction becomes sharp when the agent also wrote the tests. A change where both the implementation and its tests came from the same session can be internally consistent and unrelated to what you asked for. Nothing in CI distinguishes that case.
The reframing: CI tells you the change is not obviously broken. It does not tell you the change is right, and with agent-authored work the gap between those is wider.
The reading order
Section titled “The reading order”Not top to bottom. In this order, because each step can end the review.
-
The file list. Before any code. Does the scope match the task? Count the files and ask whether that number makes sense for what you asked.
-
Anything under
.github/. Workflows, instruction files, agent definitions,CODEOWNERS. Almost never legitimate for a feature task, and the highest-consequence thing in any diff. -
Dependency changes. A new package is a supply-chain decision somebody should make deliberately.
-
The test diff. Were existing tests modified? Modified tests are the single highest-signal thing in an agent pull request.
-
The session log. Did setup succeed? Did tests actually run? Where did it change direction?
-
The implementation, against the task rather than against itself.
-
The question no tool answers: is this the change we wanted?
Steps one through four take about two minutes and catch most of what goes wrong. A reviewer who does those four consistently will out-perform one who reads every line in order.
Modified tests
Section titled “Modified tests”Worth its own section, because it is where the loop’s incentive is most visible.
An agent asked to make a failing build pass has two available moves: fix the code, or change the test. It does not have a preference for one over the other beyond what its instructions said, and both produce a green result.
What to check on any modified test:
Was the assertion weakened? An exact equality becoming a “contains” check. A specific value becoming “any number”. A case removed from a table.
Was a test deleted? Look at the removed lines, not just the added ones.
Was skip added? A skipped test still shows as not failing.
Was the expected value changed to the actual value? The clearest signal of a symptom fix, and it looks entirely reasonable in a diff unless you know what the value should be.
When is modifying a test legitimate? When the requirement changed and the test encoded the old requirement. That is a real case, and it should be stated explicitly in the pull request description. If the description does not mention that the requirement changed, and a test was modified, that is the thing to ask about.
Checking the tests against the requirement
Section titled “Checking the tests against the requirement”The step most reviewers skip, and the one that most often finds something.
Agent-written tests tend to mirror the implementation. The agent read the code it wrote and wrote tests asserting what that code does — which is a description, not a specification.
How to check it in thirty seconds, which is the whole cost of this step. Take the original task. For each requirement in it, find the test that would fail if that requirement were violated. If you cannot find one, the requirement is untested no matter how many tests exist.
The specific pattern to look for: tests that assert the shape of the output without asserting its
correctness. A test confirming that calculateProration() returns a number is not a test of proration.
The bug-fix version: a fix for a reported bug should have a test that fails without the fix. If the pull request does not contain one, the fix is unverified, and asking for one is a completely reasonable review comment. The quickest way to confirm it: check out the branch, revert the implementation change while keeping the test, and run it. A test that still passes is not testing the fix — and this takes a minute on any change where the fix matters enough to be worth the minute.
Scope drift
Section titled “Scope drift”The most common thing you will actually see, and the most benign.
Asked to fix a bug, the agent also reformatted the file, renamed two variables and reorganised imports. The fix is correct. The diff is four times larger than necessary.
Why it matters even though nothing is wrong: review cost scales with diff size, the real change is
harder to find, and future git blame on those lines points at an unrelated pull request.
What to do: ask for the unrelated changes to be reverted. That is a normal review comment, and the agent can act on it.
How to prevent it: an explicit out-of-scope statement in the task, and an AGENTS.md that says
formatting changes belong in their own pull request. Neither is enforcement, both reduce the frequency
substantially.
When to let it go: a formatting change confined to lines the fix already touched is fine. The objection is to the file that was reformatted for no reason, and to the rename that touched nine call sites in a pull request that was supposed to change one function.
Reading the pull request description
Section titled “Reading the pull request description”It was written by the thing that wrote the code, which makes it a useful summary and a poor attestation.
What it is good for: understanding the approach quickly. A description saying “invalidated the cache
in updateUser() following the pattern in order-service.ts” orients you in seconds and is almost
always an accurate account of what the diff does.
What it cannot tell you: whether that was the right thing to do. The description describes; it does not evaluate, even when its phrasing sounds evaluative.
Phrases to read as claims rather than facts:
| The description says | What it actually establishes |
|---|---|
| “Fixed the bug” | It changed code it believed was the cause |
| “All tests pass” | The tests it ran, passed |
| “Follows the existing pattern” | It found a pattern and believed it matched |
| “No breaking changes” | It did not notice one |
| “Improves performance” | Almost certainly unmeasured — check for a benchmark |
| “Handles edge cases” | Ask which ones, and look |
None of those is dishonest. They are a summary of what a session did, phrased the way summaries are phrased. The error is reading them as verification.
The most useful thing in a description is what it says it did not do. A description mentioning “did not update the migration because it is out of scope” is telling you something you would otherwise have to discover. When descriptions never contain that kind of statement, it usually means the task had no explicit scope boundary — which is a task-writing problem.
The failure modes, as review questions
Section titled “The failure modes, as review questions”Turning the failure modes into things you actually ask:
| Failure mode | The question to ask |
|---|---|
| Scope drift | Does the file list match the task? |
| Wrong problem | Does this address what the issue described? |
| Symptom fix | Was a test or an error path changed rather than the cause? |
| Confident incompleteness | Are the error paths, concurrency and migration cases handled? |
| Plausible invention | Does everything it calls exist? (Compilation usually catches this) |
| Stalled | Did the session hit a limit it worked around rather than solved? |
| Early-step cascade | Is the whole approach based on a correct reading? |
The last one is the hardest and the most valuable. A cascade produces a diff that is internally consistent and entirely wrong, and nothing in it looks broken. The only way to catch it is to reconstruct what the change is trying to do and check that against what you asked — which is the “is this the change we wanted” question, and it is why that step is on the list.
Time-boxing the review
Section titled “Time-boxing the review”Agent pull requests arrive faster than human ones, and the review is not shorter. That arithmetic produces a queue unless somebody manages it deliberately.
Set an expectation before reviewing. A well-specified bug fix should take five to ten minutes. If you are twenty minutes in and still working out what it did, that is information — the change is more complex than the task suggested, or the approach is wrong.
Close early rather than reviewing thoroughly. When the first two minutes show scope drift across fourteen files and a modified test, the useful response is to close it with a comment, not to review the implementation carefully. Reviewing a change you are going to reject in detail is spent effort with no return.
A rejection is cheap and a merge is not. The asymmetry is stronger than with human pull requests, because there is no colleague whose morale you are managing and no relationship cost to closing something. Use that. “This does not match what I asked for; I will rewrite the issue” is a complete and sufficient review.
Batch reviews if the volume is steady. Reviewing five agent pull requests in one sitting is faster than five interruptions, because the reading order becomes automatic and you keep the codebase in your head between them.
Track how long it actually takes. Teams routinely underestimate this when deciding how much to delegate, and the delegation rate that is sustainable is the one your review capacity supports rather than the one the agent can produce.
Accountability
Section titled “Accountability”The part that is not technical.
Somebody with write access owns every merged change. The person who assigned the task, or the person who approved the pull request. Not the agent — an agent cannot be accountable, and treating it as though it could is a category error with real consequences when something breaks.
“The agent wrote it” is not an explanation. Six months later, when the change causes an incident, the question is why the change was made and who decided it was correct. Both answers involve a person.
Approving without reading is worse here than elsewhere. With a human author, a rubber-stamp approval at least means somebody wrote the code deliberately. With an agent, an unread approval means nobody looked at the change at all.
The team norm worth setting explicitly: agent pull requests get the same review a change from an external contributor would get. Not less because CI is green, and not less because the volume went up. If the volume makes that impossible, delegate less.
The review comment loop
Section titled “The review comment loop”Agents can act on review feedback, which changes how you write it.
Be specific and bounded. “Use the existing formatMoney helper in src/money.ts rather than the
inline formatting on line 34” produces a correct follow-up. “This could be cleaner” does not.
One concern per comment. A comment containing four requests tends to produce a follow-up addressing two of them.
State the requirement, not just the objection. “This needs to handle the case where the order is already cancelled” tells it what to do. “This is wrong” does not.
Know when to stop. Three rounds on the same misunderstanding means the task was wrong. Close it, rewrite the issue, or take the branch over locally and finish it yourself. Iterating on a bad specification is the most reliable way to spend an afternoon and merge nothing.
Taking it over is not a failure. The agent got you to a starting point. Finishing the last twenty per cent yourself is often the fastest path, and there is no obligation to keep it in the loop for the part that needs judgement.
What the review teaches you upstream
Section titled “What the review teaches you upstream”The most valuable output of reviewing agent pull requests is not the pull requests. It is what the pattern of problems tells you about your own configuration.
Scope drift on most pull requests means your tasks are missing an out-of-scope statement, or your
AGENTS.md does not say that unrelated changes belong in their own pull request.
Tests that mirror the implementation means the task is not stating what should be true, only what is broken. Adding a “verify: a test asserting X” line to your issue template changes this.
Sessions with no verification means setup is failing in the agent environment. That is a one-time fix with a large payoff, and it is invisible unless somebody reads a session log.
The same convention violated repeatedly means the convention is not written where the agent reads it.
Move it into AGENTS.md.
Repeated wrong-problem outcomes means your issues are describing symptoms without causes, or assuming context that is not in the text.
Consistently correct output is information too — it means the tasks you are delegating are within the range that works, and you could probably widen it a little.
The habit worth building: whenever an agent pull request needs substantial rework, spend two minutes
asking whether the task, the AGENTS.md, or the environment would have prevented it. Most of the time one
of them would have, and that fix applies to every future session rather than to this one.
This is the same loop as tuning review instructions: the recurring correction is the signal, and the fix belongs upstream of where you noticed it.
Common mistakes
Section titled “Common mistakes”Treating green CI as evidence. The loop optimised for that signal.
Reading the diff top to bottom. The file list and the test diff catch more, faster.
Not reading the test diff. Where symptom fixes are visible.
Not checking tests against the requirement. Agent tests describe the implementation by default.
Skipping the session log. It says whether anything was verified at all.
Approving because the summary is confident. It was written by the thing that wrote the code.
Letting scope drift through. It costs review time on every future change to those lines.
Iterating past the third round. The specification is the problem.
Treating the agent as accountable. A person owns every merge.
When the agent is a teammate’s, not yours
Section titled “When the agent is a teammate’s, not yours”A case that arrives once a team is delegating routinely: reviewing a pull request produced by somebody else’s delegation.
You have less context than usual. You did not write the task, so you do not know what was asked. The first thing to do is read the linked issue — and if there is no linked issue, that is the review comment to leave before reading any code.
The assigner should be the primary reviewer. They know the intent. A second reviewer who does not is guessing at whether the change matches a task they never saw.
Ask about intent rather than inferring it. “The issue says invalidate on update; this also invalidates on read — was that intended?” is a better comment than an assumption in either direction.
Do not approve on the assumption that they reviewed it. The pattern where an agent produces a pull request, the assigner assumes the reviewer will check it, and the reviewer assumes the assigner already did, ends with nobody having read it. Explicit ownership prevents this, which is why the assigner owning the merge is worth stating as a team norm rather than leaving implicit.
Mental model
Section titled “Mental model”A contribution from a competent stranger who did not ask any questions.
They read the issue and nothing else. They followed it as written, including the parts you did not mean literally. They believe they finished. They cannot tell you what they were unsure about, because they were not unsure.
Review it the way you would review a first contribution from someone whose work you do not yet know: read the scope before the code, check that it does what was asked rather than something adjacent, verify the tests test the requirement, and do not let a confident description substitute for reading the change.
What you learned
Section titled “What you learned”- Green CI carries less information for agent-authored work, because the loop optimised for it
- Read the file list,
.github/changes, dependencies and the test diff first — two minutes, most of the value - Modified tests are the highest-signal thing in an agent pull request
- Check tests against the original requirement, not against the implementation
- A bug fix should include a test that fails without the fix
- Scope drift is common, benign and worth pushing back on anyway
- A person with write access is accountable for every merge; the agent cannot be
Exercise
Section titled “Exercise”Use a disposable repository with agent pull requests available.
-
Assign a bug fix with a reproducing test. When the pull request arrives, read the file list first. Predict: does the scope match?
-
Read the test diff. Were any existing tests modified? If so, examine each modification for weakening.
-
Take the original issue and, for each requirement, find the test that would fail if it were violated. Predict: is every requirement covered?
-
Check the session log for whether setup and tests actually ran.
-
Assign a task with no reproducing test. Compare how much longer the review takes.
-
Leave a bounded, specific review comment. Compare the follow-up commit to what a vague comment produces.
-
Deliberately let a review go three rounds on the same point. Note where it would have been faster to rewrite the task.
-
Delete the repository.
Related lessons
Section titled “Related lessons”Check your understanding
3 questions — each one asks you to predict what Git or GitHub will do, not to recall a flag.