Teaching an AI Reviewer to Disprove Itself Before It Comments

The first uncomfortable signal was not a bad AI code review comment. Bad comments are easy to dislike. The worse signal was that the review agent could be made less noisy with a prompt edit, but nobody could prove whether that edit had actually improved anything except the vibe. For AI review false positives, a better instruction is only half a control. The useful move is to make the reviewer try to disprove its own finding against the diff, then record what it removed so the team can inspect the tradeoff instead of admiring a cleaner comment thread.

This note is about a specific reviewer pipeline, not a general claim that agents need more verification. The system already generated findings from pull request diffs. The problem was narrower: repeated false positives were eroding reviewer trust, and ordinary prompt tuning gave us no durable way to tell whether we had reduced noise or merely changed its costume.

The decision I would carry forward is this: treat AI review noise as a product surface with policy, adversarial self-checking, and observability wired together. If any one of those is missing, the reviewer becomes hard to operate at staff level. It may still be useful, but you are left arguing from anecdotes.

Where AI review false positives actually came from

The v3.8 prompt change added a framework-agnostic "common false positives, do not flag" block. The categories were not exotic: deleted or unchanged context lines, generated or metadata files, thread-safety claims on non-shared state, validation that already happened at a boundary, wording-only comment nits, and similar review clutter.

That was a reasonable patch. It was also not enough.

A do-not-flag list asks the model to remember a negative policy while it is doing the generative work of finding issues. That is a lot to ask from one pass. The same model behavior that makes it useful for review, pattern completion over partial evidence, also makes it comfortable with a finding that sounds like a bug class it has seen before.

The concrete incident that forced the next step was a crafted smoke test against the real model in the pipeline, zai.glm-5. One candidate finding claimed a null dereference. The diff itself had added the guard:

if (!user) return null

The falsification filter was asked whether the diff disproved the finding. It returned remove:[1], which dropped that finding. In the same smoke test, a different finding about how a users map was populated could not be verified from the diff alone. That one stayed.

That asymmetry mattered. The filter was not asked to decide whether every finding was true. It had a narrower job: remove findings contradicted by the diff, and keep findings when the evidence was merely incomplete.

The filter is a second review contract, not a nicer prompt

A prompt-only approach would have been cheaper and easier to explain. Add the v3.8 false-positive guidance, watch a few reviews, and call the noise down. We rejected that as the rollout control because it leaves no inspected record of what the system chose not to say.

The falsification filter created a second contract:

1. The reviewer proposes findings.
2. The filter receives the diff and the findings.
3. The filter removes only findings the diff itself disproves.
4. Removed findings are persisted to the review trace.
5. Reports compare prompt versions by verdict mix, downgrade rate, findings, cost, and drops.

That last part is not administrative garnish. It changes the argument. Without saved drops, a quiet reviewer can look good while silently discarding useful comments. With saved drops, the team can inspect the disappeared findings and ask whether the filter is doing useful skepticism or just sanding off sharp edges.

The staff tradeoff is recall versus inspectable restraint

Defaulting the filter on was the interesting decision. A reasonable team might keep it behind a flag for longer, especially if code review comments trigger mandatory human process or if the cost of a missed bug is high. The default-on choice gave up some simplicity and some theoretical recall. A second pass can be wrong. A removed finding may have been clumsy but still useful.

The reason default-on made sense here was the evidence shape. The live smoke showed the desired asymmetry on the real model: drop what the diff disproves, keep what the diff cannot settle. The trace inspector already summarized recent review traces by prompt version, model-to-gate verdict, finding count, issue severity, tool calls, and cost. The follow-up change persisted finalReview.filter_drops inside the existing trace JSON, avoiding a migration while closing the audit gap.

That is the part I would not skip. If the filter drops are not persisted, the rollout becomes a trust exercise. You can count fewer comments, but you cannot distinguish better restraint from suppressed signal.

The recommendation stops being correct under a few conditions. If the filter uses a weaker model than the reviewer, it may become a lossy censor. If the team cannot inspect removed findings, default-on is too aggressive. If the review domain depends heavily on repository-wide invariants unavailable in the diff, the filter must be narrower, because "the diff disproves it" becomes a harder claim to make.

The observability changed the prompt conversation

The report work sounds mundane, which is often where the useful systems live. The script added a per-review drop column and per-prompt-version totals. Combined with the existing inspector, the team could compare prompt versions by verdict mix, average findings, downgrade and escalation counts, average cost, and filter drops.

That made prompt versions operationally comparable. v3.7 to v3.8 was no longer just "we added a common false positives block." v3.8 to v3.9 could be evaluated as a different behavior profile, especially when v3.9 added a contract-trace lens for cross-layer semantic mismatches. That lens targeted a different failure mode: a value that looks plausible at each layer but is wrong end-to-end.

The distinction matters because a repeated keyword is not a repeated finding. A false-positive policy, a falsification pass, and a contract-trace lens all touch review quality, but they do different jobs. Lumping them together under "better prompt" would make the system harder to reason about.

The sharp edge

The cost of this approach is that the reviewer now has more moving parts that need review themselves. The do-not-flag policy can become stale. The falsification filter can learn to remove awkward but correct findings. The trace report can encourage optimizing for the cleanest verdict mix rather than the most useful review.

The answer is not to add yet another rubric. It is to sample the drops as first-class review output. A removed finding deserves the same kind of inspection as an emitted one, especially after a prompt version change. Quiet failure is still failure; it just has better manners.

For senior teams using AI reviewers in real delivery work, the practical bar is not whether the agent can produce fewer comments. It is whether the system can explain what it chose not to say, and whether that explanation survives contact with the diff.