Deep Code Review: Recall vs. Precision Benchmarks for Coding Agents

Deep code review prioritizes recall at the scan layer because published benchmarks show precision alone failing to predict how many real defects a reviewer catches.

Why Precision-First Became the Default

Precision-first tuning answers a measured trust problem. In the 2025 Stack Overflow Developer Survey, 46% of developers actively distrust the accuracy of AI tools against 33% who trust it. Two-thirds reported hitting "AI solutions that are almost right, but not quite."

Three vendors have published the reasoning. A March 5, 2026 GitHub changelog for Copilot code review describes the intended result as more targeted, higher-signal comments with less noise. OpenAI's post on verifying code at scale names the price, stating that its team accepted modestly reduced recall in exchange for high signal quality. Uber's uReview team wrote in an engineering post that comment quality matters far more than quantity, because a high false-positive rate leads engineers to tune out comments entirely.

All three assume a human reads first. Splitting broad detection from developer-facing comments changes that economics, and a companion guide covers the merge-boundary case for AI code review tuned that way. This article takes the narrower question of how the two metrics are measured and what published benchmarks compare.

Cosmos is Augment Code's unified cloud agents platform, generally available on paid plans, running agents across the software development lifecycle with shared context, persistent memory, and human checkpoints. Its Deep Code Review expert applies that model to pull request analysis.

Code Review Was Built for Human Readers

Human-centered review tools encode human attention limits, and those limits shaped the interface, the workflow, and the defaults more than bug coverage did. Six assumptions baked into tools from GitHub PRs to Gerrit to Phabricator follow from those limits, and each mismatches an AI agent in the reviewer role.

Reviewers have a hard ceiling on diff size. A SmartBear study of a Cisco Systems programming team, summarized in SmartBear's peer review guidance, found that the ability to find defects diminishes beyond 400 lines of code. The same guidance reports defect density dropping sharply above 500 LOC per hour and advises against reviewing for over 60 minutes at a stretch. Google's engineering practices tell authors to keep changelists small. An AI agent has no comparable working memory ceiling, so small-PR convention is evidence about reviewers, not code.

Reviewers carry known biases into the decision. Work on debiasing code review support, published at CHASE 2025, builds its prototypes on two assumptions from the bias literature. Where decision fatigue is triggered, "the reviewer tends to procrastinate." Under time pressure, a reviewer "tends to search for fast review approval instead of correct implementation." Both are design assumptions, not measured rates, and neither applies to an agent that does not tire.

Tooling has worked around those limits for a decade. Google built Tricorder, presented at ICSE 2015, to integrate static analysis into developer workflow, because some checks are worth running on every change and no reviewer will run them by hand.

The table below maps each assumption to the tool behavior it produced and to where an AI reviewer departs from it.

Diff-centric design limits codebase understanding because it centers the code delta over the resulting system behavior. A cognitive model of code review, published in Empirical Software Engineering in January 2026, drew on an ethnographic study of 10 developers across 34 reviews. It describes reviewers moving through an orientation phase that establishes context and rationale, then an analytical phase covering understanding, assessment, and planning. An agent bypasses the diff UI and evaluates the resulting state of the code.

Precision and Recall, Defined for an Agent Reader

Precision and recall describe one review output from two failure modes, noise and misses, and agent-first review changes which one is expensive. Each ratio uses true positives plus one error type, so tuning a single detector trades one against the other.

Precision = TP / (TP + FP) asks what fraction of flagged issues are real. Recall = TP / (TP + FN) asks what fraction of the real issues present the tool caught. Raising a classification threshold suppresses weak findings, which lifts precision and lowers recall in the same move.

The split tracks the approach. In REDO's evaluation on the SWEDE benchmark, Pyflakes and PyRight usually obtained higher precision while the LLM baseline obtained better recall, across Python runtime-error detection only.

Agent-first review changes the unit cost of a wrong candidate, because filtering happens before a comment exists. A January 2026 preprint, not yet peer reviewed, reports that a hybrid LLM and static-analysis technique eliminated 94 to 98% of false positives while holding recall high. That range spans backbone models on 433 alarms from one company's enterprise analyzer across three bug types, so it is one industrial result and not an industry rate. The false-negative side does not move, because a missed bug reaches production whoever reads the output first. Where a product places that filter is one of the sharper differences across the market of enterprise code generators.

Both metrics mean only as much as the measurement behind them, which is where published benchmarks diverge most.

How to Measure Recall and Precision on a Large Codebase

Measuring recall on a real repository requires a ground-truth set of the defects genuinely present, which no tool can supply about itself. Precision needs only the tool's own output plus a judgment on each finding, so it is cheap to measure and easy to inflate. Recall needs the denominator, built by hand.

Six decisions fix what the resulting numbers mean. Each moves the score with no change to the tool under test, which is why two published figures for one product often do not compare.

Denominator choice is where published figures mislead most often. An ISSTA 2024 study covered five C and C++ static analysis tools over 815 vulnerability-contributing commits behind 319 vulnerabilities in 92 projects. A single tool produced warnings in the vulnerable functions of only 52% of those commits, and 22% received no warning at the vulnerable function from any of the five. At least 76% of the warnings that did land there were irrelevant. The same tools measured at file granularity look far stronger, because a warning anywhere in a large file scores as coverage.

Two figures in one sentence can also belong to different populations, so a sample's subject and composition carry as much weight as its size.

What the 50-PR Benchmark Shows

Benchmark results across production codebases separate the two metrics cleanly. Ranked by F-score, the harmonic mean of the two, the tool with the best precision is not the tool with the best review quality, which is what a precision figure alone cannot tell a buyer.

The figures below come from Augment's own benchmark of seven tools, measuring Augment Code Review, its GitHub pull request product, against six competitors. Augment expanded and corrected the public dataset's golden comment set before scoring against it and adjusted severity, then published the corrected data and scripts. The numbers below should be read with that first-party involvement in view.

Source: Augment Code benchmark, 50 pull requests from Sentry, Grafana, Cal.com, Discourse, and Keycloak. Published December 11, 2025, updated June 18, 2026. Augment Code Review ran on GPT-5.2.

The 59% F-score at the top of the table comes from 65% precision paired with 55% recall, the only row above 50% on both axes. Codex Code Review leads on precision at 68% and places fourth of seven on F-score, because 29% recall leaves most real defects unflagged. Claude Code shows the reverse shape, reaching 51% recall on 23% precision, which points at its filtering stage. Neither figure is interpretable without the other.

What the table cannot show is the defect class no tested tool reached. Logic, authorization, and architectural defects surface only when several files, services, or execution paths are read together. That is a capability question before it is a tuning question, and this ranking of top AI coding assistants sets out which products attempt it.

Deep Code Review: The Recall-First Approach

Deep code review inverts the usual goal and prioritizes recall at the scan layer, on the reasoning that the first reader is an agent. Higher recall is only safe where three things hold together, and dropping any one turns volume into noise.

1. The agent reads the whole repository, not the diff. Bugs that escape file-level review share one property, appearing only under simultaneous reasoning across multiple files, services, or execution paths. OWASP's guide to business logic testing states that automation of business logic abuse cases "is not possible" and relies on knowledge of the complete business process. Cross-service API contract violations, authorization spanning several permission models, and race conditions between subsystems all need that architectural picture. The Context Engine behind Cosmos supplies it, semantically indexing and mapping relationships across hundreds of thousands of files, including how services connect and depend on each other.
2. Each finding arrives with its reasoning attached. Legacy linters emitted flags with no context, so every false positive cost a developer a full investigation. An agent can state why a pattern was flagged, what the impact would be, and under which conditions the concern applies, which cuts dismissal to seconds of reading. Cosmos also checks changes against repository guidance such as AGENTS.md and CLAUDE.md, so a team's stated conventions apply without being restated on every review.
3. Nothing reaches a developer without authorization. In Cosmos, the Deep Code Review expert runs exhaustive line-by-line analysis for objective correctness defects. The Pair Reviewer reconstructs intent, architecture, and tradeoffs, then posts comments or a verdict only after a human authorizes them. Approval never merges the change, so the final merge decision stays with a person.

Those three conditions describe the scan layer. The layer above it decides who reads a candidate finding, and that is where recall-first review either works or floods.

How Humans Stay in the Loop

Recall-first review produces more candidate findings than precision-first review does, so the design question moves from how much to detect to who sees what and when. Without a deliberate answer, broad scanning recreates the alert fatigue precision-first tuning was built to avoid.

BitsAI-CR, published at FSE 2025, splits its generation pipeline into two stages specifically to address low precision. Its authors note that a model's tendency toward false positives and hallucinations requires a validation mechanism, and that technically correct but practically superfluous comments are their own failure mode. Reviewer independence is a related control, and a companion guide covers it as adversarial code review and the maker-checker split. Four mechanisms in Cosmos hold candidate volume away from developers, working at configuration, routing, publication, and learning.

• Objective scope by configuration. Teams define review guidelines in a repository YAML file carrying severity levels and glob-scoped areas, and Augment Code Review's review guidance points those rules at objective issues that cause bugs or expose vulnerabilities.
• Risk-based routing before human attention. The Risk Analyzer classifies every change that is new or ready for review, applies the auto-approval policy to intrinsically low-risk changes, and routes the rest to human review with an explanation of what needs attention.
• Human authorization as the publication gate. The Pair Reviewer prepares a briefing and answers the reviewer's questions, then posts finalized comments or a verdict once the reviewer authorizes it.
• Learning from what reviewers did. The Memory Manager distills merged changes, comments, replies, reactions, and Pair Reviewer sessions into per-repository knowledge later reviews draw on, so a dismissed class of finding stops recurring.

Two failure modes survive all four. Alert fatigue leads developers to disable a tool whose noise outruns its value, and automation complacency leads them to rubber-stamp suggestions unread. Both are behavioral, so acceptance and dismissal rates belong in monitoring alongside defect counts.

How Cosmos's Deep Code Review Catches What Others Miss

The defect classes that cost the most to miss live between files, where a diff-scoped reviewer has no visibility at all. Cosmos runs its line-by-line correctness pass with the Context Engine's repository map behind it, which is what brings those classes into scope.

Design-level defects are roughly half the problem. Gary McGraw writes that security defects divide roughly 50/50 between bugs in the code and flaws in the design, and that top-ten lists concentrate on bugs at the expense of design-level problems. A reviewer with no model of what the software should prevent cannot evaluate that second half. Every one of the four categories below requires reading a second file to see the defect at all.

• Cross-service API contract violations: two services each pass review in isolation and break on interaction, which surfaces only when the reviewer traces call chains across the service boundary.
• Authorization logic spanning several permission models: OWASP's secure code review cheat sheet groups complex permission models and access control implementations under "Human Expertise Advantages," alongside business logic flaws and architecture security.
• Missing or incorrect failure handling across module boundaries: a HotOS 2019 study of 112 high-severity Azure production incidents attributes 31% of bug-induced incidents to failures in detecting or handling component faults. Among the fault-handling failures specifically, the handler ignored the error report in 35% of cases and over-reacted in another 35%.
• Workflow state bypass: the OWASP business logic abuse top ten names workflow step skipping among the flaws it addresses, in a category it describes as transcending technology stacks, from web applications and APIs to firmware and hardware platforms.

Tuning cannot reach any of these four, which is why scan scope is decided before any threshold is set.

What to Do Next

The choice between precision-first and recall-first tuning is a choice about which failure mode a team is willing to own, and the answer differs by codebase. A single-file utility gains little from full-repository scanning. A system with several services, layered permissions, and long workflow state machines is where the recall gap becomes shipped defects.

Start with the failures already on record. Pull the last quarter of production incidents and sort them by whether a diff-scoped reviewer could have seen the defect at all. That count is the size of the gap no amount of precision tuning will close. Then establish a baseline for escaped defects and review cycle time, run one team on recall-first review with agent-layer filtering in place, and compare those two numbers before switching more.

Try Cosmos →

Frequently Asked Questions

Related Reading

• 10 Open Source AI Code Review Tools Tested on a 450K-File Monorepo [2026 Rankings]
• Autonomous Code Review Platforms for Enterprise Teams
• 13 Best AI Coding Tools for Complex Codebases in 2026
• 12 Best Open Source Code Review Tools in 2026
• Google AI Code Review: 5 GCP Review Tools