Code Review Agents Need Defect State, Not Comment History

Most AI code-review demos follow a clean sequence: give the model a diff, ask it to find problems, and publish the comments.

Real pull requests are not that clean.

A reviewer identifies a defect. The author pushes another commit. The relevant code moves to a different file. One part of the defect is fixed, another remains. An old comment becomes outdated. A later change reintroduces the original problem.

At that point, code review is no longer a single inference task. It is a state-management problem.

That distinction matters because an agent can produce an excellent review of each individual diff and still perform poorly over the lifetime of the pull request. It can forget an unresolved defect, repeat an obsolete comment, or accept a partial fix because the latest patch looks reasonable in isolation.

My view is simple:

A code-review agent should not treat comment history as its source of truth. It needs a durable model of every defect it is responsible for tracking.

I call that model a Defect Ledger.

What MCR-Bench makes visible

The recently published MCR-Bench is useful because it evaluates the part of code review that static benchmarks leave out: multiple rounds of interaction.

The benchmark contains 2,269 real-world multi-round review tasks across Python, Java, JavaScript, TypeScript, and C#. Each task includes fine-grained defect information and labels describing how a defect changes across review rounds.

The paper reports three findings that should concern anyone building code-review agents:

• Performance degrades as the number of review rounds increases.

This is not merely a context-window problem. Passing every historical diff and comment back to the model gives it more text, but not necessarily a reliable representation of what remains true.

Conversation history is an event log. It is not materialized state.

Comments are projections, not state

A GitHub review comment answers a communication question: what should a developer see at a particular line in a particular revision?

It does not reliably answer the engineering question: what is the current state of the underlying defect?

A comment can be:

• marked resolved even though the defect remains;

The comment should therefore be treated as a projection of internal review state. The defect itself needs an identity and lifecycle independent of any one comment, line number, or commit.

The Defect Ledger

The ledger is a durable store of normalized defect records. It can live in PostgreSQL, SQLite, a repository artifact, or another transactional store. The technology is less important than the contract.

Here is a simplified TypeScript representation:

type DefectStatus =
  | "open"
  | "possibly_fixed"
  | "verified"
  | "dismissed"
  | "reopened";

type CodeLocation = {
  path: string;
  startLine?: number;
  endLine?: number;
  symbol?: string;
  commitSha: string;
};

type Evidence = {
  kind: "code" | "test" | "trace" | "analysis";
  summary: string;
  commitSha: string;
  location?: CodeLocation;
};

type VerificationRun = {
  commitSha: string;
  method: "test" | "static_analysis" | "inspection" | "human";
  outcome: "passed" | "failed" | "inconclusive";
  evidence: string[];
  checkedAt: string;
};

interface DefectRecord {
  id: string;
  fingerprint: string;
  title: string;
  description: string;
  type: string;
  severity: "low" | "medium" | "high" | "critical";
  status: DefectStatus;
  introducedIn: string;
  lastObservedIn: string;
  locations: CodeLocation[];
  evidence: Evidence[];
  resolutionCriteria: string[];
  verificationHistory: VerificationRun[];
  relatedCommentIds: string[];
}

Several fields are especially important.

The id remains stable even when the code moves. The fingerprint helps reconcile a newly observed issue with an existing defect. resolutionCriteria records what must become true before closure. verificationHistory separates “the code changed” from “the defect was proven fixed.”

That last distinction is where many review workflows break down.

A defect is a state machine

The agent should not freely rewrite defect status from prose. State transitions need explicit rules.

• A newly supported finding enters open.

This prevents a common error: assuming that touching the affected code is equivalent to fixing the problem.

For a high-severity authorization defect, the resolution criteria might require all of the following:

1. authorization executes before the protected operation;

The model may propose that these conditions are satisfied. The harness should gather the evidence and decide whether the transition is allowed.

The review loop across commits

When a new commit arrives, the agent should not begin with a blank review prompt. The harness should construct a bounded review task from repository state and the unresolved ledger.

async function reviewRevision(input: {
  pullRequestId: string;
  previousSha: string;
  currentSha: string;
}) {
  const diff = await repository.diff(input.previousSha, input.currentSha);
  const active = await ledger.listActive(input.pullRequestId);

  const affected = selectPotentiallyAffected(active, diff);
  const remapped = await remapLocations(affected, diff, input.currentSha);

  for (const defect of remapped) {
    const result = await verifyDefect(defect, input.currentSha);
    await ledger.applyVerification(defect.id, result);
  }

  const candidates = await detectNewDefects({
    diff,
    repositoryContext: await retrieveRelevantContext(diff),
    knownDefects: remapped,
  });

  const reconciled = await reconcileCandidates(candidates, active);
  await ledger.upsertMany(reconciled);

  return publishReviewProjection(await ledger.changesSince(input.previousSha));
}

The sequence is intentional:

1. determine which existing defects may have changed;

This is more reliable than asking the model to reread every prior comment and reconstruct the lifecycle from scratch.

A three-commit example

Imagine a pull request adding an endpoint that returns customer invoices.

Commit A: the defect is opened

The first revision accepts a customerId from the request and queries invoices without verifying that the authenticated user belongs to that customer.

The agent creates DEF-1042, records the affected symbol, captures the unsafe data flow, and defines a resolution criterion: access must be authorized using the authenticated tenant identity before the query executes.

Commit B: the comment becomes stale, but the defect does not

The author refactors the endpoint into a service class and moves the query to another file. GitHub marks the original code location as outdated. A stateless reviewer may lose the finding because the original lines disappeared.

The ledger-backed reviewer sees that the changed symbols intersect with DEF-1042, remaps the evidence to the service method, and reruns verification. The new code contains a role check, but still trusts the request's customerId.

The defect remains open. The agent updates the existing finding instead of creating a duplicate.

Commit C: the fix is verified

The author derives the tenant identifier from the authenticated session and adds a negative cross-tenant test.

The harness executes the targeted test, inspects the relevant path, stores the evidence, and moves DEF-1042 from possibly_fixed to verified.

The important outcome is not that the agent remembered a comment. It maintained an engineering invariant across changing code.

Identity and reconciliation are the hard parts

A ledger does not eliminate ambiguity. It makes ambiguity explicit.

Two observations may refer to the same underlying defect even when their locations and wording differ. Conversely, similar-looking observations may represent separate defects requiring independent fixes.

I would use a layered reconciliation strategy:

1. deterministic signals such as repository, pull request, symbol, data-flow endpoints, rule identifier, and affected test;

The model can recommend a merge, but it should not silently collapse two high-severity defects without evidence. Merges and splits should be recorded as ledger events so they remain auditable.

The ledger is not model memory

It is tempting to describe this as “giving the agent memory.” That framing is too loose.

Model memory is usually retrieved context intended to influence the next response. The ledger is authoritative application state with schemas, invariants, versioning, and controlled transitions.

The model can read it and propose updates. The harness owns persistence and transition enforcement.

That separation is the same principle I use elsewhere in harness engineering:

The model proposes; the runtime verifies and commits.

Evaluating a ledger-backed reviewer

Static precision and recall still matter, but they do not measure lifecycle quality. I would compare stateless and ledger-backed reviewers across multi-commit pull requests using at least these metrics:

• Unresolved-defect recall: how often defects that remain valid are still recognized in later rounds.

The experiment should also measure latency and cost. Loading every historical artifact may improve recall while making routine reviews impractical. A ledger should reduce context by materializing only current defect state and retrieving historical evidence when it becomes relevant.

Practical safeguards

A production implementation needs several controls beyond the schema:

• optimistic concurrency or serialized updates so parallel review workers cannot overwrite each other;

These are not side concerns. Once a ledger influences whether software is approved, it becomes part of the software-delivery control plane.

The larger design lesson

MCR-Bench highlights a broader limitation in how we build agents. We often preserve a transcript and assume the model will recover the correct operational state from it later.

That works until the workflow becomes long-running, concurrent, or consequential.

Comments are valuable for collaboration. Conversation history is valuable for explanation. Neither should be the only source of truth for an evolving engineering object.

For code review, that object is the defect.

The agent can detect it. The model can explain it. The comment can communicate it.

But the harness must give it an identity, preserve its evidence, control its transitions, and verify when it is truly resolved.

That is how a code-review agent stops reviewing isolated snapshots and starts reviewing the pull request.

Source

• Dewu Zheng et al., “From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench”, submitted August 27, 2026.