c-CRAB：用可执行测试衡量代码审阅智能体，四工具合起来只过四成
Code Review Agent Benchmark
Yuntong Zhang Note: The first two authors contributed equally to this research. email: zhang.yuntong@u.nus.edu Affiliation: National University of Singapore , , Zhiyuan Pan Note: Work done while author is full-time at National University of Singapore. email: zy_pan@zju.edu.cn Affiliation: Zhejiang University , , Imam Nur Bani Yusuf email: inbyusuf@nus.edu.sg Note: Corresponding author. Affiliation: National University of Singapore , , Haifeng Ruan email: haifeng.ruan@u.nus.edu Affiliation: National University of Singapore , , Ridwan Shariffdeen email: shariffdeenr@acm.org Affiliation: SonarSource , and Abhik Roychoudhury email: abhik@nus.edu.sg Affiliation: National University of Singapore ,
Abstract
摘要
Software engineering agents have shown significant promise in coding tasks - either writing an entire application, or writing a code patch to improve a software project by remediating an issue. As AI agents permeate code writing, and generate huge volumes of code automatically - the matter of code quality comes front and centre. As the automatically generated code gets integrated into huge code-bases — the issue of code review and broadly quality assurance becomes important. In this paper, we take a fresh look at the problem and curate a code review dataset for AI agents to work with. Our dataset called c-CRAB (pronounced see-crab) can evaluate agents for code review tasks. Specifically given a pull-request (which could be coming from code generation agents or humans), if a code review agent produces a review, our evaluation framework can asses the reviewing capability of the code review agents. Our evaluation framework is used to evaluate the state of the art today - the open-source PR-agent, as well as commercial code review agents from Devin, Claude Code, and Codex.
软件工程智能体在编码任务上已展现显著潜力——无论是写完整应用，还是写补丁改进软件项目并修复问题。随着 AI 智能体渗入写代码、并自动生成海量代码，代码质量问题走到台前。当自动生成的代码并入庞大代码库，代码审阅与广义质量保障就变得关键。本文换个角度看问题，整理出一套供 AI 智能体使用的代码审阅数据集。我们的数据集名为 c-CRAB（读作 see-crab），可评估智能体在代码审阅任务上的表现。具体而言，给定一条拉取请求（可来自代码生成智能体或人类），若代码审阅智能体产出审阅，我们的评测框架可评估其审阅能力。我们用该框架评估当下前沿：开源 PR-agent，以及 Devin、Claude Code、Codex 的商业代码审阅智能体。
Our c-CRAB dataset is systematically constructed from human reviews - given a human review of a pull request instance we generate corresponding tests to evaluate the code review agent generated reviews. Such a benchmark construction gives us several insights. First of all, all of the existing review agents taken together can solve only around 40% of the c-CRAB tasks, indicating the potential to close this gap by future research. Secondly, we observe that the agent reviews often consider different aspects from the human reviews - indicating the potential for human-agent collaboration for code inspection and review that could be deployed in future software teams. Last but not the least, the agent generated tests from our data-set act as a held out test-suite and hence quality gate for agent generated reviews. What this will mean for future collaboration of code generation agents, test generation agents and code review agents - remains to be investigated.
c-CRAB 数据集系统性地从人类审阅构建——给定某条拉取请求实例的人类审阅，我们生成对应测试，用以评估代码审阅智能体产出的审阅。这样的基准构建带来几点洞见。首先，现有审阅智能体合在一起也只能解决约 40% 的 c-CRAB 任务，说明未来研究仍有很大空间。其次，我们观察到智能体审阅常关注与人类审阅不同的方面——提示未来软件团队可部署人—智能体协作做代码检查与审阅。最后，数据集中由智能体生成的测试可作为留出测试套件，从而成为智能体生成审阅的质量门。这对未来代码生成智能体、测试生成智能体与代码审阅智能体如何协作意味着什么，仍待研究。
1. Introduction
1. 引言
Keywords:
Code review is a fundamental quality assurance practice in modern software development. Before a proposed change is merged into the main codebase, developers inspect pull requests (PRs) to detect defects, enforce project standards, and maintain long-term code quality. Traditionally, this process relies on human reviewers who carefully examine pull requests (PRs) and provide feedback to authors. Recent progress in software engineering agents has significantly accelerated the process of writing code. AI systems can now generate large amounts of code automatically, both for implementing new features (Li et al., 2025b; Zhou et al., 2026) and for resolving issues (Xie et al., 2025; Zhang et al., 2024) in existing repositories. As these systems become increasingly integrated into development workflows, they contribute to a growing number of generated code and submitted PRs (Robbes et al., 2026; Pham and Ghaleb, 2026). However, the capacity of human reviewers has not scaled at the same pace (Hassan et al., 2025), creating a new bottleneck in the development pipeline: while code generation is increasingly automated, ensuring the quality of generated code still relies heavily on manual inspection.
关键词：
代码审阅是现代软件开发中的基本质量保障实践。拟议改动并入主代码库前，开发者检查拉取请求（PR），以发现缺陷、落实项目标准并维持长期代码质量。传统上，这一过程依赖人类审阅者仔细检查拉取请求并给作者反馈。软件工程智能体的进展显著加快了写代码：AI 系统现已能自动生成大量代码，既用于实现新功能 (Li et al., 2025b; Zhou et al., 2026)，也用于在现有仓库中解决问题 (Xie et al., 2025; Zhang et al., 2024)。随着这些系统越来越深地嵌入开发工作流，它们贡献了越来越多的生成代码与提交 PR (Robbes et al., 2026; Pham and Ghaleb, 2026)。然而人类审阅者的容量并未同速扩展 (Hassan et al., 2025)，开发管线出现新瓶颈：写代码日益自动化，保障生成代码质量仍大量依赖人工检查。
This challenge has motivated the development of automated code review agents that analyze pull requests and generate review feedback (Tang et al., 2024; Rasheed et al., 2024; Li et al., 2025a; Tantithamthavorn et al., 2026). Several tools and systems have recently been proposed to automatically produce review comments aimed at identifying bugs, design issues, or maintainability concerns. Yet a central question remains: to what extent do these automated review tools identify the same issues that human reviewers would raise?
这一挑战推动了自动化代码审阅智能体的发展：它们分析拉取请求并生成审阅反馈 (Tang et al., 2024; Rasheed et al., 2024; Li et al., 2025a; Tantithamthavorn et al., 2026)。近期已有若干工具与系统自动产出审阅评论，旨在发现 bug、设计问题或可维护性关切。但中心问题仍在：这些自动化审阅工具在多大程度上找出了人类审阅者也会提出的同一批问题？
Existing evaluation methods do not answer this question well. Most benchmarks compare generated comments against human-written reviews using textual overlap or embedding similarity metrics (Tufano et al., 2021; Tufano et al., 2022; Naik et al., 2025; Jiang et al., 2025; Hu et al., 2025). Such metrics primarily measure resemblance in wording rather than whether a review identifies a meaningful issue in the code. However, we posit that these evaluation approaches are not well suited for measuring how closely AI-generated reviews align with the concerns raised by human reviewers. Human review comments are often noisy artifacts of the review process: they may include clarification questions, subjective stylistic suggestions, or back-and-forth discussions between reviewers and authors. As a result, the wording of a comment does not always correspond directly to the underlying issue being raised. Measuring similarity between comments is therefore problematic. A review may correctly identify a valid issue using entirely different wording and still receive a low similarity score. Similar limitations also arise in other evaluation methods, such as localization metrics (Guo et al., 2025; Zhang et al., 2026) and LLM-as-a-judge (Guo et al., 2025). Localization metrics only assess whether an issue is identified at the same code location as the ground truth human reviews while ignoring the substance of the review. Furthermore, LLM-as-a-judge approaches can be sensitive to prompt design and inherent randomness, causing their judgments difficult to reproduce reliably.
现有评测方法回答不好这个问题。多数基准用文本重叠或嵌入相似度，把生成评论与人类写的审阅对比 (Tufano et al., 2021; Tufano et al., 2022; Naik et al., 2025; Jiang et al., 2025; Hu et al., 2025)。这类指标主要量措辞像不像，而不是审阅是否指出了代码里有意义的问题。我们认为这些做法不适合衡量 AI 生成审阅与人类审阅者关切的对齐程度。人类审阅评论往往是审阅过程中的嘈杂产物：可能含澄清问题、主观风格建议，或审阅者与作者的来回讨论。因此，评论措辞并不总直接对应被提出的底层问题。度量评论相似度因而成问题：审阅可能用完全不同的措辞正确指出有效问题，却仍得低相似度分。定位指标 (Guo et al., 2025; Zhang et al., 2026) 与 LLM-as-a-judge (Guo et al., 2025) 等其他方法也有类似局限。定位指标只评估问题是否落在与真值人类审阅相同的代码位置，而忽视审阅实质。此外，LLM-as-a-judge 对提示设计与固有随机性敏感，判断难以可靠复现。
In this paper, we revisit the problem of evaluating automated code review tools and introduce c-CRAB (pronounced see-crab), a benchmark designed to assess how well automated review tools identify the same issues that human reviewers raise in realistic PR settings. Instead of evaluating reviews solely based on similarity to reference comments, c-CRAB adopts a test-based evaluation. Specifically, human review feedback is systematically converted into executable tests that capture the underlying issues identified during the review process. These tests serve as objective evaluation oracles: if a review agent identifies an issue corresponding to a failing test, the issue can be verified automatically in the code.
本文重新审视自动化代码审阅工具的评测问题，并引入 c-CRAB（读作 see-crab）：一个基准，用于评估自动化审阅工具在真实 PR 场景中，能否找出人类审阅者提出的同一批问题。c-CRAB 不只靠与参考评论的相似度评审阅，而采用基于测试的评测。具体地，人类审阅反馈被系统性地转成可执行测试，捕捉审阅过程中指出的底层问题。这些测试充当客观评测预言：若审阅智能体指出与某条失败测试对应的问题，则可在代码中自动验证。
To evaluate a review tool, we provide the tool with a PR and collect the review comments it produces. A separate coding agent then revises the code in the PR based on the generated review comments from the review tools, and the revised version is executed against the curated tests. If the tests pass, this indicates that the review successfully identifies actionable issues that lead to correct code improvement. By doing so, c-CRAB measures the extent to which automated review tools discover issues originally identified by human reviewers. This evaluation therefore measures whether automated reviewers can raise issues that correspond to those raised by human reviewers in real PRs.
评测审阅工具时，我们把 PR 交给工具并收集其产出的审阅评论。另有一个编码智能体根据审阅工具生成的评论修订 PR 中的代码，修订版再对整理出的测试执行。若测试通过，说明审阅成功指出了可行动问题，并导向正确的代码改进。由此，c-CRAB 衡量自动化审阅工具在多大程度上发现了人类审阅者原先指出的问题。该评测因而衡量：自动化审阅者能否提出与真实 PR 中人类审阅者对应的问题。
Using c-CRAB, we evaluate several state-of-the-art review agents, including the open-source PR-Agent, and the review capabilities of Devin Review, Claude Code, and Codex. Our results show that current systems successfully identify only about 40% of the issues captured in the benchmark, indicating substantial room for improvement in automated code review. We further observe that AI-generated reviews often focus on different aspects of code compared to human reviewers. However, they remain valuable in identifying problems and suggesting improvements. This difference highlights complementary perspectives that may support future human-AI collaborative review workflows. In summary, the contributions of this paper are as follows:
我们用 c-CRAB 评估若干前沿审阅智能体，包括开源 PR-Agent，以及 Devin Review、Claude Code、Codex 的审阅能力。结果表明，当前系统只成功识别基准中约 40% 的问题，自动化代码审阅仍有很大改进空间。我们还观察到，AI 生成审阅常关注与人类审阅者不同的代码方面，但在发现问题与建议改进上仍有价值。这种差异凸显互补视角，或可支撑未来人—AI 协作审阅工作流。本文贡献总结如下：
- • We introduce c-CRAB, a benchmark dataset for evaluating automated code review agents on real-world pull requests.
- • We propose an evaluation framework that converts human review feedback into executable tests, enabling objective validation of review quality.
- • We evaluate several state-of-the-art review agents and show that current systems solve only about 40% of the benchmark tasks, indicating substantial room for improvement in automated code review.
- • We analyze differences between human and AI-generated reviews, revealing complementary perspectives that motivate future human-agent collaboration in code review.
- • 我们引入 c-CRAB，一个在真实世界拉取请求上评估自动化代码审阅智能体的基准数据集。
- • 我们提出把人类审阅反馈转成可执行测试的评测框架，使审阅质量可客观验证。
- • 我们评估若干前沿审阅智能体，表明当前系统只解决约 40% 的基准任务，自动化代码审阅仍有很大改进空间。
- • 我们分析人类与 AI 生成审阅的差异，揭示互补视角，推动未来代码审阅中的人—智能体协作。
2. Background and Related Work
2. 背景与相关工作
| Dataset | Year | Oracle source | Repo-level Support | Evaluation |
|---|---|---|---|---|
| Tufano-21 (Tufano et al., 2021) | 2021 | Mined OSS PR reviews | No | N-gram overlap |
| CodeReviewer (Li et al., 2022) | 2022 | Mined OSS PR reviews | No | N-gram overlap |
| Tufano-22 (Tufano et al., 2022) | 2022 | Mined OSS PR reviews | No | N-gram overlap |
| CRScore corpus (Naik et al., 2025) | 2025 | LLM + static analysis | No | Embedding similarity |
| ContextCRBench (Hu et al., 2025) | 2025 | Mined OSS PR reviews | No | N-gram overlap |
| SWE-CARE (Guo et al., 2025) | 2025 | Mined OSS PR reviews | Yes | Localization + N-gram overlap + LLM-as-a-judge |
| AACR-Bench (Zhang et al., 2026) | 2026 | Mined OSS PR reviews | Yes | Localization |
| RovoDev (Tantithamthavorn et al., 2026) | 2026 | Logs + user feedback | Yes | Online production metrics |
| Qodo Benchmark (Yanay and Fingerman, 2026) | 2026 | Mined OSS PR reviews | Yes | LLM-as-a-judge |
| CR-Bench (Pereira et al., 2026) | 2026 | Mined OSS PR issue description | Yes | LLM-as-a-judge |
| c-CRAB (Ours) | 2026 | Mined OSS PR reviews | Yes | Test-based |
Table 1. Comparison of automated code review benchmarks.
| 数据集 | 年份 | 预言来源 | 仓库级支持 | 评测 |
|---|---|---|---|---|
| Tufano-21 (Tufano et al., 2021) | 2021 | 挖掘的 OSS PR 审阅 | 否 | N-gram 重叠 |
| CodeReviewer (Li et al., 2022) | 2022 | 挖掘的 OSS PR 审阅 | 否 | N-gram 重叠 |
| Tufano-22 (Tufano et al., 2022) | 2022 | 挖掘的 OSS PR 审阅 | 否 | N-gram 重叠 |
| CRScore corpus (Naik et al., 2025) | 2025 | LLM + 静态分析 | 否 | 嵌入相似度 |
| ContextCRBench (Hu et al., 2025) | 2025 | 挖掘的 OSS PR 审阅 | 否 | N-gram 重叠 |
| SWE-CARE (Guo et al., 2025) | 2025 | 挖掘的 OSS PR 审阅 | 是 | 定位 + N-gram 重叠 + LLM-as-a-judge |
| AACR-Bench (Zhang et al., 2026) | 2026 | 挖掘的 OSS PR 审阅 | 是 | 定位 |
| RovoDev (Tantithamthavorn et al., 2026) | 2026 | 日志 + 用户反馈 | 是 | 在线生产指标 |
| Qodo Benchmark (Yanay and Fingerman, 2026) | 2026 | 挖掘的 OSS PR 审阅 | 是 | LLM-as-a-judge |
| CR-Bench (Pereira et al., 2026) | 2026 | 挖掘的 OSS PR 问题描述 | 是 | LLM-as-a-judge |
| c-CRAB（本文） | 2026 | 挖掘的 OSS PR 审阅 | 是 | 基于测试 |
表 1. 自动化代码审阅基准对比。
2.1. Automated Code Review Tools
2.1. 自动化代码审阅工具
Automated code review tools can be broadly categorized into static analysis-based approaches and AI-based approaches. Static analysis tools detect vulnerabilities and code quality issues based on pre-defined rules, patterns, or data-flow analysis. In contrast, AI-based review tools typically leverage LLMs to detect issues based on learned patterns during LLM’s training, as well as contextual knowledge provided through prompts or agentic scaffold. These review tools are typically triggered when a Pull Request (PR) is opened on GitHub, GitLab, or BitBucket. When triggered, AI-based review tools collect necessary context, such as the modified lines in the PR, the surrounding code context, and repository-level context such as dependency graphs. This collected context is then analyzed by an LLM to identify potential issues. Some AI-based review tools may perform additional post-processing, such as false positive filtering or severity ranking, before the reviews are presented to developers. As output, most review tools present the reviews as a summary comment or inline comments in the PR. In this paper, we focus specifically on the evaluation of AI-based code review tools.
自动化代码审阅工具大体可分为基于静态分析与基于 AI 两类。静态分析工具按预定义规则、模式或数据流分析发现漏洞与代码质量问题。相较之下，基于 AI 的审阅工具通常靠 LLM，依据训练中学到的模式以及通过提示或智能体脚手架提供的上下文知识来发现问题。这些审阅工具通常在 GitHub、GitLab 或 BitBucket 上打开拉取请求时触发。触发后，基于 AI 的工具收集必要上下文，例如 PR 中的改动行、周围代码上下文，以及依赖图等仓库级上下文。收集到的上下文再由 LLM 分析以识别潜在问题。部分工具还会做额外后处理，例如假阳性过滤或严重度排序，再把审阅呈现给开发者。输出上，多数审阅工具以 PR 中的汇总评论或行内评论呈现。本文聚焦于基于 AI 的代码审阅工具的评测。
2.2. Benchmarks for Automated Code Review
2.2. 自动化代码审阅基准
Table 1 summarizes the existing available benchmarks for code review. Early work primarily focuses on fine-grained settings, where review comments are aligned with localized code changes such as lines, methods, or diff hunks. Datasets, such as in (Li et al., 2022; Tufano et al., 2021; Tufano et al., 2022; Naik et al., 2025; Hu et al., 2025), construct large-scale pairs of code changes and human-written comments mined from open-source repositories. These datasets have been widely used to train and evaluate models for review comment generation. However, such datasets typically have limited context granularity, often isolating individual diff hunks and abstracting away the broader pull requests and repository context. Consequently, they do not fully capture the complexity of real-world code review. More recent benchmarks move toward PR-level and repository-aware evaluation. Datasets, such as in (Guo et al., 2025; Zhang et al., 2026; Tantithamthavorn et al., 2026), incorporate richer contextual signals, including full pull requests, surrounding code, and repository structure. These benchmarks better reflect practical deployment settings, where review tools must reason over larger contexts and diverse types of issues.
表 1 汇总现有可用的代码审阅基准。早期工作主要聚焦细粒度设定，把审阅评论与行、方法或 diff hunk 等局部代码改动对齐。如 (Li et al., 2022; Tufano et al., 2021; Tufano et al., 2022; Naik et al., 2025; Hu et al., 2025) 中的数据集，从开源仓库挖掘大规模「代码改动—人类评论」配对，广泛用于训练与评估审阅评论生成模型。但这类数据集通常上下文粒度有限，常孤立单个 diff hunk，并抽象掉更广的拉取请求与仓库上下文，因而不能充分刻画真实世界代码审阅的复杂性。更新的基准走向 PR 级与仓库感知评测。如 (Guo et al., 2025; Zhang et al., 2026; Tantithamthavorn et al., 2026) 纳入更丰富的上下文信号，包括完整拉取请求、周围代码与仓库结构。这些基准更好反映实际部署：审阅工具须在更大上下文与多样问题类型上推理。
Despite these progress, existing benchmarks differ in how they define evaluation oracles. Most rely on mined human review comments as reference signal, while more recent work introduces expert annotations (Zhang et al., 2026) or LLM-augmented labels (Naik et al., 2025) to improve data quality. Nevertheless, the majority of benchmarks still treat natural-language human comments as the primary ground truth, which introduces challenges due to noise, incompleteness, and variability in human review practices. Ours is the only dataset that depends on dynamic program behavior as witnessed by tests (albeit generated from human reviews).
尽管有这些进展，现有基准在如何定义评测预言上仍不同。多数依赖挖掘的人类审阅评论作参考信号，更新的工作引入专家标注 (Zhang et al., 2026) 或 LLM 增强标签 (Naik et al., 2025) 以提升数据质量。不过，绝大多数基准仍把自然语言人类评论当作主要真值，因而面临噪声、不完整与人类审阅实践变异带来的挑战。我们的是唯一依赖测试所见证的动态程序行为作预言的数据集（尽管测试由人类审阅生成）。
2.3. Evaluation Methods for Code Reviews
2.3. 代码审阅评测方法
A common strategy for evaluating automated code review systems compares generated comments against human-written reviews based on similarity with the human reviews, such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), chrF (Popovic, 2015), and embedding-based similarity (Tufano et al., 2021; Tufano et al., 2022; Jiang et al., 2025; Naik et al., 2025; Hu et al., 2025). While these metrics are easy to compute at scale, recent studies further show that human review comments used as references can be noisy or incomplete (Lu et al., 2025; Jiang et al., 2025).
评估自动化代码审阅系统的常见策略，是按与人类写的审阅的相似度比较生成评论，例如 BLEU (Papineni et al., 2002)、ROUGE (Lin, 2004)、chrF (Popovic, 2015) 以及基于嵌入的相似度 (Tufano et al., 2021; Tufano et al., 2022; Jiang et al., 2025; Naik et al., 2025; Hu et al., 2025)。这些指标易于大规模计算，但近期研究进一步表明，用作参考的人类审阅评论可能嘈杂或不完整 (Lu et al., 2025; Jiang et al., 2025)。
Several studies also uses localization to evaluate code review tools (Guo et al., 2025; Zhang et al., 2026). This metric measures whether automated code review tools identify concerns at the same locations as human reviewers. However, it evaluates only the location of identified issues, instead of the content or nature of the concerns raised. Another study (Tantithamthavorn et al., 2026) evaluates code review tools using online production metrics, such as acceptance rate. While these metrics more accurately reflect real-world usage, they are difficult to obtain because they require access to a production environment for data collection.
若干研究也用定位来评估代码审阅工具 (Guo et al., 2025; Zhang et al., 2026)。该指标衡量自动化工具是否在与人类审阅者相同的位置指出关切。但它只评估已识别问题的位置，而不评估关切的内容或性质。另有研究 (Tantithamthavorn et al., 2026) 用接受率等在线生产指标评估代码审阅工具。这些指标更贴近真实使用，但难以取得，因为需要生产环境访问来采集数据。
Another emerging paradigm uses large language models as judges to directly assess review quality (Zeng et al., 2025; Guo et al., 2025). In this setting, an LLM is prompted to assess the quality of generated review comments by comparing them with human-written feedback or by reasoning about the code changes and the identified issues (Zeng et al., 2025; Guo et al., 2025). Compared with lexical or embedding-based similarity metrics, LLM-based evaluation can consider richer contextual information and may better capture aspects such as correctness, relevance, and usefulness of review feedback. Although promising, LLM-as-judge approaches can suffer from bias, instability, and sensitivity to prompt design (Shi et al., 2025; Thakur et al., 2024). These factors make it difficult to ensure reproducibility and consistency when LLMs are used as evaluation oracles.
另一新兴范式用大语言模型作裁判，直接评估审阅质量 (Zeng et al., 2025; Guo et al., 2025)。此时提示 LLM，通过与人类写的反馈对比，或就代码改动与已识别问题推理，来评估生成审阅评论的质量 (Zeng et al., 2025; Guo et al., 2025)。相对词法或嵌入相似度，基于 LLM 的评测可考虑更丰富上下文，或更好捕捉正确性、相关性与有用性。虽有前景，LLM-as-judge 仍可能有偏置、不稳定与对提示设计敏感等问题 (Shi et al., 2025; Thakur et al., 2024)。这些因素使把 LLM 当评测预言时难以保证可复现与一致性。
In contrast, c-CRAB introduces a test-based evaluation framework for automated code review. Instead of comparing generated reviews against reference comments, we convert human review feedback into executable tests that capture the underlying issues identified by humans during the review process. Each benchmark instance includes an executable environment, enabling automated reviewers to be evaluated in a reproducible setting. These tests serve as objective evaluation oracles, allowing scalable, stable, and verifiable assessment of automated review agents.
相较之下，c-CRAB 为自动化代码审阅引入基于测试的评测框架。我们不把生成审阅与参考评论对比，而是把人类审阅反馈转成可执行测试，捕捉人类在审阅过程中指出的底层问题。每个基准实例含可执行环境，使自动化审阅者可在可复现设定下评测。这些测试充当客观评测预言，支持对自动化审阅智能体做可扩展、稳定、可验证的评估。
3. Motivation
3. 动机
First, we demonstrate that existing evaluation methods fail to capture whether automated reviews align with the intentions of human reviewers. We then show how our test-based evaluation in c-CRAB overcomes this limitation.
首先，我们说明现有评测方法未能捕捉自动化审阅是否对齐人类审阅者意图；随后说明 c-CRAB 中基于测试的评测如何克服这一局限。
3.1. Comparison with Similarity Metrics
3.1. 与相似度指标对比
Consider the pull request from python-telegram-bot[^1], which introduces a check to reject keyboard inputs with more than two dimensions.
考虑来自 python-telegram-bot[^1] 的拉取请求，它引入检查以拒绝超过两维的键盘输入。
if isinstance(keyboard[0][0], Sequence) and not isinstance(keyboard[0][0], (str, bytes))
Figure 1. The line of code under review.
图 1. 被审代码行。
A human reviewer raised the following concern on the line shown in Figure 1:
人类审阅者对图 1 所示行提出如下关切：
This comment identifies a robustness issue: the implementation assumes that nested elements are sequences and may raise an exception on malformed inputs (e.g., [1]) instead of safely rejecting them.
该评论指出稳健性问题：实现假定嵌套元素是序列，对畸形输入（例如 [1]）可能抛异常，而不是安全拒绝。
Now consider the review generated by Codex for the same patch:
再看 Codex 对同一补丁生成的审阅：
We can observe that this comment raises the same concern about indexing that can lead to runtime errors. However, the two comments describe the issue in different ways.
可以看出，这条评论提出了同一关切：索引可能导致运行时错误。但两条评论描述问题的方式不同。
| Metric | Value | Interpretation |
|---|---|---|
| BLEU-4 | 0.00 | The degree of n-gram overlap between human and automated tool reviews. |
| ROUGE-L | 7.02 | |
| chrF | 20.74 | |
| Embedding similarity | 54.59 | The relative distance between human and automated tool reviews in the embedding space. |
| Our Work | ||
| Test passed? | Yes | Whether the reviews successfully capture concerns raised by humans during review. |
Table 2. Comparison of c-CRAB against prior similarity-based metrics.
| 指标 | 值 | 解读 |
|---|---|---|
| BLEU-4 | 0.00 | 人类与自动化工具审阅之间的 n-gram 重叠程度。 |
| ROUGE-L | 7.02 | |
| chrF | 20.74 | |
| 嵌入相似度 | 54.59 | 人类与自动化工具审阅在嵌入空间中的相对距离。 |
| 本文 | ||
| 测试是否通过？ | 是 | 审阅是否成功捕捉人类在审阅中提出的关切。 |
表 2. c-CRAB 与先前基于相似度的指标对比。
Table 2 reports the similarity scores obtained using metrics commonly adopted in prior work on code review evaluation. Here, BLEU-4 measures the precision of up to 4-gram overlaps with a brevity penalty, ROUGE-L captures the longest common subsequence (LCS) to reflect sequence-level similarity, and chrF computes an F-score based on character-level 6-gram overlap. For the embedding similarity, we convert both reviews into an embedding vector using SentenceTransformers[^2] and compute the cosine similarity, following the prior work (Jiang et al., 2025). All scores are normalized to 0–100.
表 2 报告先前代码审阅评测常用指标上的相似度分数。此处 BLEU-4 衡量带简洁惩罚的最多 4-gram 重叠精确率，ROUGE-L 用最长公共子序列（LCS）反映序列级相似度，chrF 按字符级 6-gram 重叠算 F 分。嵌入相似度方面，我们用 SentenceTransformers[^2] 把两段审阅转成嵌入向量并算余弦相似度，遵循先前工作 (Jiang et al., 2025)。所有分数归一化到 0–100。
The result show that all similarity metrics yield low scores. The low scores on BLEU-4, ROUGE-L, and chrF are expected due to the minimal n-gram overlap between the human and automated tool reviews. The embedding-based similarity yields better score than chrF. However, the score remains moderate and cannot precisely capture that both reviews raise the same concern. This is because embedding-based similarity measures the proximity in the embedding space and the embedding representation is affected by how the reviews are expressed (e.g., word choices, sentence structure) (Antoniak and Mimno, 2018). As a result, two reviews that describes the same concern can still be relatively distant in the embedding space if they differ in phrasing.
结果显示所有相似度指标分数都低。BLEU-4、ROUGE-L、chrF 低分在预期之中，因为人类与自动化工具审阅之间 n-gram 重叠极少。基于嵌入的相似度优于 chrF，但仍属中等，无法精确捕捉两条审阅提出同一关切。因为嵌入相似度衡量嵌入空间中的邻近，而嵌入表示受表述方式影响（例如用词、句式） (Antoniak and Mimno, 2018)。结果是：两条描述同一关切的审阅若措辞不同，在嵌入空间中仍可能相对疏远。
3.2. Comparison with LLM-as-a-Judge
3.2. 与 LLM-as-a-Judge 对比
live_events_columns: ArrayField = ArrayField(models.CharField(max_length=200), null=True, blank=True)
Figure 2. The line of code under review.
Let us consider another pull request from posthog[^3]. As shown in Figure 2, a field with type ArrayField was added to a class. On this line of code change, a human reviewer raised the concern:
再看来自 posthog[^3] 的另一条拉取请求。如图 2，某类中新增了类型为 ArrayField 的字段。对这一行改动，人类审阅者提出关切：
Claude Code generated the following review comment:
Claude Code 生成了如下审阅评论：
Claude Code does a good job at defect detection and points out the same issue as human review: CharField in django library cannot handle arbitrarily long strings. However, the automated review comment is still not fully aligned with the human reviewer’s intent. Here, its suggested fix is different: it only proposes a trivial mitigation while the human reviewer suggested replacing CharField with TextField, which truly solves the root issue.
Claude Code 在缺陷检测上做得不错，指出了与人类审阅相同的问题：django 库中的 CharField 无法处理任意长字符串。但自动化审阅评论仍未完全对齐人类审阅者意图。此处建议的修复不同：它只提出琐碎缓解，而人类审阅者建议用 TextField 替换 CharField，真正解决根因。
For prior work that adopts vanilla LLM-as-a-Judge evaluation, LLM is typically used to judge whether an automated review points out the same issue as human reviews. However, as the above example shows, alignment with human intent means more than this, as the correctness and quality of suggested fix also plays an important role in code review, and previous LLM-as-a-Judge methods cannot distinguish between the above two review comments.
对采用朴素 LLM-as-a-Judge 评测的先前工作，LLM 通常被用来判断自动化审阅是否指出与人类审阅相同的问题。但如上例所示，与人类意图对齐不止于此：建议修复的正确性与质量在代码审阅中同样重要，而先前 LLM-as-a-Judge 方法无法区分上述两条审阅评论。
3.3. Our benchmark: c-CRAB
3.3. 我们的基准：c-CRAB
The key idea of c-CRAB is to evaluate review comments based on whether they lead to behaviorally correct fixes. During evaluation, we provide the generated review to a coding agent, which revises the patch accordingly. The revised patch is then executed against the synthesized test. If the revised patch passes the test, we consider the review useful, as it successfully captures the underlying issue originally identified by the human reviewer.
c-CRAB 的核心想法是：按审阅评论是否导向行为正确的修复来评估它们。评测时，我们把生成的审阅交给编码智能体，由其据此修订补丁。修订后的补丁再对合成测试执行。若修订补丁通过测试，我们视该审阅有用，因为它成功捕捉了人类审阅者原先指出的底层问题。
For the human review in Figure 1, we convert the human review into the executable test shown in Figure 3 to capture its intent.
对图 1 中的人类审阅，我们把人类审阅转成图 3 所示可执行测试，以捕捉其意图。
def test_check_keyboard():
"""
Test for a bug where check_keyboard_type tries to access keyboard[0][0] before validating that rows are sequences. For keyboard=[1], the function should not raise an exception and should return False.
"""
keyboard = [1]
try:
result = check_keyboard_type(keyboard)
except Exception as exc:
...
assert result is False
Figure 3. The test ensuring check_keyboard_type([1]) returns False instead of raising an exception.
图 3. 确保 check_keyboard_type([1]) 返回 False 而非抛异常的测试。
This test directly captures the reviewer’s concern by asserting that the function must not perform unsafe nested indexing before validating input structure. Unlike similarity-based metrics, which rely on textual overlap or semantic similarity of text, this test encodes the expected behavior as an executable specification. In this example, the review generated by the automated review tool enables the coding agent to revise the initial patch that passes the test. This indicates that, despite low textual similarity, the generated review successfully captures the same underlying issue as the human review and is therefore considered useful under our evaluation.
该测试直接捕捉审阅者关切：断言函数在校验输入结构前不得做不安全的嵌套索引。不同于依赖文本重叠或语义相似度的相似度指标，该测试把期望行为编码为可执行规格。本例中，自动化审阅工具生成的审阅使编码智能体修订初始补丁并通过测试。这说明尽管文本相似度低，生成审阅仍成功捕捉了与人类审阅相同的底层问题，因而在我们的评测下视为有用。
For the human review in Figure 2, the tool-generated review does not successfully guide the coding agent toward a patch that satisfies the executable test. This is because the test expects the presence of TextField in the source code, whereas the tool-generated review proposes an alternative fix by increasing the maximum length. Consequently, the revised patch fails to satisfy the test. In contrast, in our experiment, an LLM-as-a-judge considers both reviews to raise the same concern although both reviews suggesting different fixes. This ability to capture the precise review intent using executable tests is what makes c-CRAB different compared to LLM-as-a-judge. Moreover, another concern with the LLM-as-a-judge is the inherent randomness of its output that makes the evaluation not reliable.
对图 2 中的人类审阅，工具生成的审阅未能成功引导编码智能体得到满足可执行测试的补丁。因为测试期望源码中出现 TextField，而工具生成审阅提出的替代修复是增大最大长度。结果修订补丁未能满足测试。相较之下，我们实验中 LLM-as-a-judge 认为两条审阅提出同一关切，尽管建议的修复不同。用可执行测试捕捉精确审阅意图，正是 c-CRAB 相对 LLM-as-a-judge 的不同之处。此外，LLM-as-a-judge 输出固有的随机性也会使评测不可靠。
4. Methodology
4. 方法
In this section, we first present the high-level design of c-CRAB, followed by discussion on benchmark construction, evaluation procedure, and implementation details.
本节先给出 c-CRAB 的高层设计，再讨论基准构建、评测流程与实现细节。
4.1. c-CRAB Overview
4.1. c-CRAB 概览
The challenge in evaluating code review lies in the fact that the value of a review comment is not determined by how closely it matches human wording, but by whether it identifies an issue whose resolution improves the code. This observation motivates the design of c-CRAB, a benchmark that evaluates review comments based on whether they lead to verifiable improvements in the code.
评测代码审阅的难点在于：审阅评论的价值不取决于措辞与人类有多像，而取决于它是否指出一个修掉后能改进代码的问题。这一观察推动了 c-CRAB 的设计——按审阅评论是否导向可验证的代码改进来评估它们。
At a high level, each benchmark instance consists of a pull request containing a patch to be reviewed and a set of test cases derived from human review feedback. These tests encode issues identified by human reviewers. Each test is constructed to fail on the original patch and pass once the corresponding issue has been resolved. During evaluation, an automated review tool is provided with a PR instance and asked to analyze the PR and generate review comments. These comments are used as guidance for a coding agent to revise the code changes in the PR. The revised code is subsequently executed against the benchmark tests. Because each test represents an issue originally identified by human reviewers, passing a test indicates that the corresponding issue has been successfully addressed. Therefore, if a test fails on the original PR but passes after the revision, we attribute the improvement to the usefulness of the review feedback that guided the modification. c-CRAB measures review usefulness by whether generated feedback leads to resolving human-identified issues, as shown by passing of corresponding tests.
高层上，每个基准实例含一条待审拉取请求（含补丁）以及一组由人类审阅反馈导出的测试用例。这些测试编码人类审阅者指出的问题。每个测试构造为：在原始补丁上失败，在对应问题修复后通过。评测时，把 PR 实例交给自动化审阅工具，请其分析 PR 并生成审阅评论。这些评论再作为引导，供编码智能体修订 PR 中的代码改动。修订后的代码对基准测试执行。因每个测试代表人类审阅者原先指出的问题，通过测试即表明对应问题已成功处理。因此，若测试在原始 PR 上失败、修订后通过，我们将改进归因于引导修改的审阅反馈有用。c-CRAB 用对应测试是否通过，衡量生成反馈是否导向解决人类指出的问题。
4.2. Benchmark Curation
4.2. 基准整理
Figure 4. Idea behind Benchmark Construction.
图 4. 基准构建背后的想法。
Figure 4 illustrates our benchmark curation pipeline. For each pull request, the pipeline takes three artifacts as input: the pull request description, the candidate patch, and the associated human review comments. The goal of the pipeline is to construct a benchmark instance that preserves the PR context while providing a set of tests derived from human reviews. This goal is nontrivial for two reasons. First, raw code review discussions are inherently noisy: many comments are conversational or exploratory and therefore cannot be reliably translated into executable tests. Second, generated tests must be executed within a properly constructed environment that reproduces the repository’s dependencies and runtime conditions. Without such infrastructure, tests cannot be executed consistently during evaluation.
图 4 展示我们的基准整理管线。对每条拉取请求，管线以三件产物为输入：拉取请求描述、候选补丁，以及关联的人类审阅评论。目标是构建既保留 PR 上下文、又提供由人类审阅导出之测试集的基准实例。这一目标并不平凡，原因有二。其一，原始代码审阅讨论天生嘈杂：许多评论是对话式或探索性的，因而无法可靠转成可执行测试。其二，生成的测试必须在正确构建的环境中执行，该环境需复现仓库依赖与运行条件。没有这类基础设施，评测时测试无法一致执行。
To address these challenges, we design a structured curation pipeline consisting of four stages: review filtering, executable environment construction, converting NL comments to executable tests, and validation with coding agents. First, we filter review comments to retain only those that correspond to objectively verifiable issues. Second, we construct an executable environment for each PR instance to ensure that generated tests can be executed reliably. Third, we synthesize tests from the retained review comments. The key idea is to encode the issue raised by a human review comment as a test that fails on the candidate patch but passes once the issue is fixed. Finally, we perform additional validation to ensure that benchmark instances contain only comments that can be resolved by a coding agent. We describe each stage of the pipeline below.
为应对这些挑战，我们设计由四阶段组成的结构化整理管线：审阅过滤、可执行环境构建、把自然语言评论转成可执行测试，以及用编码智能体校验。首先，过滤审阅评论，只保留对应客观可验证问题的评论。其次，为每个 PR 实例构建可执行环境，确保生成测试能可靠执行。第三，从保留的审阅评论合成测试。关键想法是：把人类审阅评论提出的问题编码为「在候选补丁上失败、问题解决后通过」的测试。最后，做额外校验，确保基准实例只含编码智能体能解决的评论。下文分述各阶段。
4.2.1. Review Filtering
The objective of the filtering stage is to retain only review comments that can be translated into high-quality test cases serving as benchmark targets. In raw PR discussions, not every human comment reflects a concrete and actionable concern in the candidate patch. Many comments are conversational in nature, such as requests for clarification, acknowledgments, or praise. While such comments may still play an important role in real-world collaboration, they are not suitable for our benchmark because they cannot be mapped unambiguously to executable tests. Furthermore, these types of comments are generally not expected to be produced by automated code review tools. Including them would introduce noise into the curation pipeline and reduce the validity of the resulting tests. Therefore, the goal of the filtering stage is to identify review comments that express verifiable issues whose resolution can, in principle, be validated through execution.
4.2.1. 审阅过滤
过滤阶段的目标是：只保留能转成高质量测试用例、可作为基准靶标的审阅评论。在原始 PR 讨论中，并非每条人类评论都反映候选补丁中具体、可行动的关切。许多评论属对话性质，例如澄清请求、致谢或称赞。这类评论在真实协作中仍可能重要，但不适合我们的基准，因为无法 unambiguous 映射到可执行测试。此外，自动化代码审阅工具通常也不被期望产出这类评论。纳入它们会给整理管线引入噪声，并降低所得测试的效度。因此，过滤阶段的目标是识别表达可验证问题、且原则上可通过执行验证其解决的审阅评论。
To achieve this in a scalable manner, we design an automated filtering classifier by prompting an LLM. To define the filtering prompt, we first construct a small gold set through manual annotation, which is used to develop and refine the LLM prompt later. We randomly sample 100 review comments from the candidate pool, and two authors independently label each comment as either high-quality or low-quality for benchmark construction, providing a short justification for each label. A comment is labeled as high-quality if it identifies a specific, actionable, and objectively verifiable issue in the patch. Conversely, comments are labeled as low-quality if they lack a concrete action item, such as conversational exchanges, acknowledgments, or concerns that cannot be reliably validated through testing. After the independent annotation phase, the two authors compare their labels and resolve any disagreements through discussion, resulting in a set of ground truth labels for .
为可扩展地做到这一点，我们通过提示 LLM 设计自动化过滤分类器。定义过滤提示时，我们先经人工标注构建小规模金标集 ，用于后续开发与打磨 LLM 提示。我们从候选池随机抽 100 条审阅评论，两位作者独立把每条标为适合或不适合基准构建的高质量/低质量，并为每条标签提供简短理由。若评论指出补丁中具体、可行动且客观可验证的问题，则标为高质量。反之，若缺少具体行动项——例如对话往来、致谢，或无法经测试可靠验证的关切——则标为低质量。独立标注后，两位作者比较标签并通过讨论消歧，得到 的真值标签集。
Given this gold set with ground-truth labels, we iteratively refine an LLM prompt so that an LLM can make correct classifications in the gold set. This prompt refinement process continues until the prompt can make the LLM-based classification reach a predefined precision threshold. The resulting prompt is used to classify the remaining review comments outside of the gold set, filtering out low-quality instances.
有了带真值标签的金标集后，我们迭代打磨 LLM 提示，使 LLM 能在金标集上正确分类。提示打磨持续到基于 LLM 的分类达到预定义精确率阈值。所得提示用于分类金标集外的其余审阅评论，滤掉低质量实例。
4.2.2. Execution Environment Construction
An execution environment is essential in the curation of c-CRAB, as a generated test has to be executed and verified before being used as a reliable oracle for evaluating review comments. During this stage, we construct an isolated Docker environment for each pull request in the dataset. A successfully built environment is a Docker image that consists of pre-installed operating system and software dependencies, and a pre-configured source code repository, providing out-of-the-box scaffolding for subsequent pipeline stages.
4.2.2. 执行环境构建
执行环境对 c-CRAB 整理至关重要：生成的测试必须先执行并验证，才能作为评估审阅评论的可靠预言。本阶段为数据集中每条拉取请求构建隔离 Docker 环境。成功构建的环境是一个 Docker 镜像，含预装操作系统与软件依赖，以及预配置的源码仓库，为后续管线阶段提供开箱脚手架。
The detailed steps to build such environments are as follows. First, we generate scripts that clone the git repository and check out the commit to be reviewed. Second, we implement a heuristic script builder that automatically generates installation scripts by detecting build tools and inferring software dependencies and their versions from the corresponding specification files (e.g., setup.py and pyproject.toml in Python projects). These scripts, along with Dockerfiles that set up the operating system, form all necessary components required by execution environment construction. Finally, each built Docker image is tested to ensure that an editable installation of the repository exists and all dependencies are installed.
构建此类环境的详细步骤如下。首先，生成克隆 git 仓库并检出待审提交的脚本。其次，实现启发式脚本构建器：通过检测构建工具，并从对应规格文件（例如 Python 项目的 setup.py 与 pyproject.toml）推断软件依赖及其版本，自动生成安装脚本。这些脚本连同设置操作系统的 Dockerfile，构成执行环境构建所需的全部组件。最后，测试每个构建出的 Docker 镜像，确保仓库可编辑安装存在且全部依赖已安装。
In some cases, especially when it comes to historic pull requests, the automatically generated installation scripts can fail because the original specifications are too loose now. For example, some projects did not specify the versions or only specified lower bounds for some of the dependencies. To overcome the issue for these historic commits, we use an automated coding agent running in an isolated environment to audit the dependency requirements, pinpoint the accurate dependency versions and override the previously generated installation script.
某些情况——尤其是历史拉取请求——自动生成的安装脚本会失败，因为原始规格如今过松。例如，有些项目未指定版本，或只为部分依赖指定了下界。为克服这些历史提交的问题，我们在隔离环境中运行自动化编码智能体，审计依赖要求、精确定位准确依赖版本，并覆盖先前生成的安装脚本。
4.2.3. Converting NL Comments to Tests
The objective of this stage is to transform each retained natural-language review comment into an executable test that captures the issue raised by the human reviewer. For a review comment associated with file , the input to this stage consists of three components: (1) the natural-language comment and its corresponding diff hunk, (2) the repository state at the PR head commit (the before version), and (3) the repository state after the issue has been addressed (the after version). Given this information, we invoke an LLM-based loop to synthesize a test case that reflects the issue described in the comment. The goal is to generate a test case that fails in the before version and passes in the after version. The generated test therefore serves as an executable oracle representing the issue originally identified by the human reviewer.
4.2.3. 把自然语言评论转成测试
本阶段目标是：把每条保留的自然语言审阅评论变成可执行测试，捕捉人类审阅者提出的问题。对关联文件 的审阅评论 ，本阶段输入含三部分：（1）自然语言评论及其对应 diff hunk；（2）PR 头提交处的仓库状态（之前版本）；（3）问题解决后的仓库状态（之后版本）。给定这些信息，我们调用基于 LLM 的循环，合成反映评论所述问题的测试用例。目标是生成测试用例 ，在之前版本失败、在之后版本通过。因此生成的测试 充当可执行预言，代表人类审阅者原先指出的问题。
Before generating tests, we construct the contextual information required to interpret each review comment. Review comments in pull requests often refer to specific lines within a diff. Therefore, we align each comment with its corresponding code region using the diff metadata provided by the pull request. Then, we retrieve the full file contents of the affected file at both the before and after commits This context provides the LLM with sufficient information to understand the intended behavior and the modification that resolves the issue.
生成测试前，我们构建解释每条审阅评论所需的上下文信息。拉取请求中的审阅评论常引用 diff 内具体行。因此，我们用拉取请求提供的 diff 元数据，把每条评论与对应代码区域对齐。然后，我们检索受影响文件 在之前与之后提交处的完整文件内容。这些上下文为 LLM 提供足够信息，以理解期望行为与解决问题的修改。
Using the extracted context, we prompt an LLM to generate a candidate test case that captures the issue described in the comment. The prompt includes the review comment, the relevant diff hunk, and the file-level code context. After a candidate test is generated, we execute it against both the before and after repository states. The execution outcome determines whether the test satisfies the fail-then-pass requirement. If the test fails on the before version and passes on the after version, it is accepted as a valid oracle.
利用抽出的上下文，我们提示 LLM 生成捕捉评论所述问题的候选测试用例。提示含审阅评论、相关 diff hunk 与文件级代码上下文。候选测试生成后，我们分别在之前与之后仓库状态上执行。执行结果决定测试是否满足「先失败后通过」要求。若测试在之前版本失败且在之后版本通过，则接受为有效预言。
In practice, the initial generation may not satisfy the fail-then-pass requirement due to issues such as incorrect imports, runtime errors, or assertions that do not distinguish between the two repository states. To address this, we employ an execution-guided refinement loop. When a generated test does not satisfy the required behavior, the execution traces and error messages are fed back to the coding agent, which revises the test accordingly. This process continues for a fixed number of iterations until either a valid test is produced or the attempt budget is exhausted.
实践中，初次生成可能因错误导入、运行时错误，或无法区分两个仓库状态的断言，而不满足「先失败后通过」。为此，我们采用执行引导的打磨循环。当生成测试不满足所需行为时，把执行轨迹与错误信息反馈给编码智能体，由其据此修订测试。该过程固定迭代次数，直到产出有效测试或用尽尝试预算。
4.2.4. Validation with Coding Agents
The final stage of the curation pipeline ensures that each benchmark instance can be reliably resolved by the coding agent used in the evaluation phase when provided with the corresponding human review. This validation step is necessary because the evaluation protocol of c-CRAB relies on a coding agent to modify the patch according to generated review comments. Therefore, we must ensure that when the coding agent fails to make the generated test pass during evaluation, the failure can be attributed to the quality of the review feedback rather than limitations of the coding agent itself.
4.2.4. 用编码智能体校验
整理管线最后阶段确保：在评测阶段，当提供对应人类审阅时，每个基准实例都能被所用编码智能体可靠解决。这一校验步骤必要，因为 c-CRAB 的评测协议依赖编码智能体按生成的审阅评论修改补丁。因此，我们必须确保：评测时若编码智能体未能使生成测试通过，失败可归因于审阅反馈质量，而非编码智能体自身局限。
Concretely, for each instance, we ask a coding agent to improve the code using the human review comment as guidance. The agent is provided with the patch under review, the relevant file context, and the natural-language review comment that motivated the generated test. The tests produced during the test generation stage are not exposed to the agent and are used only for post-hoc verification. An instance is considered valid if the agent produces a revision that causes the corresponding test to pass, thereby satisfying the fail-then-pass property. If the agent fails to produce a revision that passes the test within a fixed number of attempts, the instance is discarded.
具体地，对每个实例，我们请编码智能体 以人类审阅评论为引导改进代码。智能体获得待审补丁、相关文件上下文，以及促生该测试的自然语言审阅评论。测试生成阶段产出的测试不对智能体暴露，仅用于事后验证。若智能体产出使对应测试通过的修订，从而满足「先失败后通过」性质，则实例视为有效。若在固定尝试次数内智能体未能产出通过测试的修订，则丢弃该实例。
4.3. Benchmark Usage
4.3. 基准用法
4.3.1. Workflow
The goal of the evaluation phase is to measure the usefulness of review feedback generated by automated code review tools. Specifically, we evaluate whether the issues raised by an automated review tool can guide the coding agent (the same coding agent used in Section 4.2.4) to improve the patch under review and cause the executable tests to pass.
4.3.1. 工作流
评测阶段的目标是衡量自动化代码审阅工具生成的审阅反馈是否有用。具体地，我们评估：自动化审阅工具提出的问题，能否引导编码智能体 （与第 4.2.4 节所用同一编码智能体）改进待审补丁，并使可执行测试通过。
The evaluation comprises two stages: review generation and review-guided revision. In the first stage, each automated review tool analyzes every instance in the final benchmark and generates review comments given the patch under review and PR description. In the second stage, the generated reviews are used to guide the coding agent to revise the patch. Specifically, we provide the agent with the patch under review and the review comments produced by the tool under evaluation. The agent attempts to modify the patch according to the generated reviews. The revised patch is then executed against the tests associated with the benchmark instance. If the revision causes a test to pass, the corresponding review feedback is considered aligned with human reviews, as it successfully guides the same agent to resolve the issue originally identified by human reviewers.
评测含两阶段：审阅生成与审阅引导修订。第一阶段，每个自动化审阅工具分析最终基准中的每个实例，给定待审补丁与 PR 描述，生成审阅评论。第二阶段，用生成的审阅引导编码智能体 修订补丁。具体地，我们把待审补丁与被测工具产出的审阅评论交给智能体。智能体尝试按生成审阅修改补丁。修订后的补丁再对基准实例关联的测试执行。若修订使某测试通过，则对应审阅反馈视为与人类审阅对齐，因为它成功引导同一智能体 解决了人类审阅者原先指出的问题。
4.3.2. Evaluation Metric
Formally, let denote the validated test set associated with benchmark instance , and let denote the patch produced by the coding agent when guided by the review findings generated by a tool. We define the instance-level performance as the pass rate of the validated tests:
4.3.2. 评测指标
形式上，令 表示基准实例 关联的已校验测试集，令 表示编码智能体 在某工具生成的审阅发现引导下产出的补丁。我们将实例级表现定义为已校验测试的通过率：
The overall performance of a review tool is measured by the aggregate test pass rate across benchmark instances. We note that the pass rate in c-CRAB reflects the extent to which review tools can identify issues found by human reviewers (i.e., the true positive rate when human reviews are treated as ground truth). Automated review tools may also generate other valuable comments that are not identified by human reviewers; however, like other existing benchmarks, c-CRAB does not directly evaluate these additional comments. To provide a more complete picture, we further discuss the quality of such additional reviews generated by automated tools in Section 5.2.
审阅工具的总体表现用跨基准实例的汇总测试通过率衡量。需注意：c-CRAB 中的通过率反映审阅工具能识别人类审阅者所发现问题的程度（即以人类审阅为真值时的真阳性率）。自动化审阅工具也可能生成人类审阅者未指出的其他有价值评论；不过，与其他现有基准一样，c-CRAB 不直接评估这些额外评论。为给出更完整图景，我们在第 5.2 节进一步讨论自动化工具生成的这类额外审阅的质量。
4.4. Benchmark Implementation
4.4. 基准实现
We build c-CRAB on top of SWE-CARE (Guo et al., 2025) because it already provides PR instances with the necessary commit metadata More generally, however, our pipeline is dataset-agnostic: any pull request that provides the required metadata could serve as a candidate instance for the curation process.
我们在 SWE-CARE (Guo et al., 2025) 之上构建 c-CRAB，因为它已提供带必要提交元数据的 PR 实例。更一般地，我们的管线与数据集无关：任何提供所需元数据的拉取请求都可作为整理过程的候选实例。
4.4.1. Implementation of the Curation Pipeline
We leverage GPT-5.2 to perform review filtering (Section 4.2.1). GPT-5.2 is also used for NL comments to tests conversion (Section 4.2.3), and up to three LLM attempts with execution feedback are allowed to generate the final test. For validation (Section 4.2.4), we use Claude Code with Sonnet-4.6 backend as the coding agent.
4.4.1. 整理管线实现
我们用 GPT-5.2 做审阅过滤（第 4.2.1 节）。GPT-5.2 也用于自然语言评论到测试的转换（第 4.2.3 节），并允许多至三次带执行反馈的 LLM 尝试以生成最终测试。校验（第 4.2.4 节）用 Claude Code，后端为 Sonnet-4.6，作为编码智能体。
4.4.2. Benchmark Statistics
| Pipeline Stage | # PR | # Comments |
|---|---|---|
| Initial Dataset | 671 | 1,313 |
| Review Filtering | 410 | 595 |
| Executable Environment Construction | 410 | 595 |
| Converting NL Comments to Tests | 339 | 481 |
| Validation with Coding Agents (Final) | 184 | 234 |
Table 3. The number of remaining PR instances and Review Comments after each processing step.
4.4.2. 基准统计
| 管线阶段 | # PR | # 评论 |
|---|---|---|
| 初始数据集 | 671 | 1,313 |
| 审阅过滤 | 410 | 595 |
| 可执行环境构建 | 410 | 595 |
| 自然语言评论转测试 | 339 | 481 |
| 用编码智能体校验 （最终） | 184 | 234 |
表 3. 各处理步骤后剩余的 PR 实例与审阅评论数。
| Statistic | Value |
|---|---|
| Pull Requests | |
| Repositories | 67 |
| Instances (PRs) | 184 |
| Avg. modified lines | 418.1 |
| Tests / Review Comments | |
| Total review comments (tests) | 234 |
| Behavioral | 42 (17.9%) |
| Structural | 192 (82.1%) |
| Avg. tests per instance | 1.27 |
| Avg. lines per test | 31.8 |
Table 4. Statistics of the final c-CRAB benchmark.
| 统计 | 值 |
|---|---|
| 拉取请求 | |
| 仓库 | 67 |
| 实例（PR） | 184 |
| 平均改动行数 | 418.1 |
| 测试 / 审阅评论 | |
| 审阅评论（测试）总数 | 234 |
| 行为型 | 42 (17.9%) |
| 结构型 | 192 (82.1%) |
| 每实例平均测试数 | 1.27 |
| 每测试平均行数 | 31.8 |
表 4. 最终 c-CRAB 基准统计。
Table 3 summarizes the number of remaining pull requests and review comments after each stage of the curation pipeline. Overall, the resulting benchmark contains 184 pull request instances and 234 validated review comments, each associated with at least one executable test. The reduction in number of comments from initial dataset is expected, given the strict requirements imposed at each stage of the pipeline. The final dataset represents a carefully curated subset of the original data and supports a faithful evaluation of review usefulness. The detailed statistics of the curated benchmark is shown in Table 4. The 234 tests in c-CRAB are further categorized into behavioral and structural. Behavioral tests import and execute the tested code at runtime. They invoke the tested functions with specific inputs, and check outputs or verify exceptions. On the other hand, structural tests inspect source code text, matches patterns, and check API surfaces to determine whether desired code changes have been made.
表 3 汇总整理管线各阶段后剩余的拉取请求与审阅评论数。总体而言，所得基准含 184 个拉取请求实例与 234 条已校验审阅评论，每条至少关联一个可执行测试。相对初始数据集评论数的缩减在预期之中，因为管线各阶段要求严格。最终数据集是原始数据经仔细整理的子集，支持对审阅有用性的忠实评测。整理后基准的详细统计见表 4。c-CRAB 中的 234 个测试进一步分为行为型与结构型。行为型测试在运行时导入并执行被测代码：用特定输入调用被测函数，检查输出或验证异常。结构型测试则检查源码文本、匹配模式，并检查 API 表面，以判断期望的代码改动是否已做出。
4.4.3. Test Suites Quality Assurance
To assess the quality of the test cases, we randomly sampled 50 instances and two authors independently evaluate whether each test faithfully captures the concern expressed in the corresponding human review comment. For each instance, annotators were provided with the generated test, the original review comment, and the relevant diff hunk, and were allowed to check the original pull request when additional context was necessary. The annotators achieved an agreement rate of 84%, indicating that the generated tests generally align well with human-identified issues. The remaining disagreement cases arise from partial coverage of the review comment and occasional overfitting of tests to specific implementations.
4.4.3. 测试套件质量保障
为评估测试用例质量，我们随机抽 50 个实例，两位作者独立评估每个测试是否忠实捕捉对应人类审阅评论所表达的关切。对每个实例，标注者获得生成测试、原始审阅评论与相关 diff hunk，必要时可查看原始拉取请求。标注者一致率达 84%，表明生成测试大体与人类指出的问题对齐良好。其余分歧来自对审阅评论的部分覆盖，以及测试偶发对特定实现过拟合。
5. Results
5. 结果
We aim to answer the following research questions:
我们旨在回答以下研究问题：
- • RQ1: How do different automated review tools perform on c-CRAB? This RQ quantitatively evaluates how effectively automated review tools can capture issues raised by humans, as measured by their ability to help a coding agent pass tests derived from human reviews.
- • RQ2: How does reviews generated from the automated review tools differ compared to human reviews? The goal in this RQ is to analyze the types of reviews raised by automated tools and humans and analyze the differences between them.
- • RQ1：不同自动化审阅工具在 c-CRAB 上表现如何？ 该 RQ 定量评估自动化审阅工具捕捉人类提出问题的有效程度，度量为其帮助编码智能体通过由人类审阅导出之测试的能力。
- • RQ2：自动化审阅工具生成的审阅与人类审阅有何不同？ 该 RQ 目标是分析自动化工具与人类提出的审阅类型，并分析其间差异。
5.1. Experimental Settings
5.1. 实验设置
5.1.1. Evaluated Tools
We select four representative automated code review tools spanning both open-source and proprietary solutions that are actively used in practice: Claude Code[^4], Codex[^5], Devin Review[^6], and PR-Agent[^7]. PR-Agent is an open-source tool, while the remaining tools are proprietary solutions. We run each review tool to produce per-instance review comments on the 184 PR instances in c-CRAB.
5.1.1. 被评工具
我们选取四个有代表性的自动化代码审阅工具，覆盖实践中活跃使用的开源与专有方案：Claude Code[^4]、Codex[^5]、Devin Review[^6] 与 PR-Agent[^7]。PR-Agent 为开源工具，其余为专有方案。我们在 c-CRAB 的 184 个 PR 实例上运行每个审阅工具，产出按实例的审阅评论。
5.1.2. Evaluation Configuration and Environment
We keep the coding agent used in benchmark curation validation (Section 4.2.4) and review evaluation (Section 4.3.1) consistent. Both steps use Claude Code with a fixed Sonnet-4.6 backend. All experiments were conducted on a Ubuntu 20.04 server.
5.1.2. 评测配置与环境
我们保持基准整理校验（第 4.2.4 节）与审阅评测（第 4.3.1 节）所用编码智能体一致。两步均用 Claude Code，固定 Sonnet-4.6 后端。全部实验在 Ubuntu 20.04 服务器上进行。
5.2. RQ1: Evaluation Scores on c-CRAB
5.2. RQ1：c-CRAB 上的评测分数
Table 5 presents the performance of automated code review tools on c-CRAB. Recall that each test encodes a concern originally raised by a human reviewer. Therefore, a higher pass rate indicates that the review comments produced by a tool more often guide the coding agent toward changes that address concerns similar to those identified by human reviewers.
表 5 给出自动化代码审阅工具在 c-CRAB 上的表现。回顾：每个测试编码人类审阅者原先提出的关切。因此，更高通过率表明工具产出的审阅评论更常引导编码智能体做出能应对与人类审阅者类似关切的改动。
| # Comments | Pass Rate (%) | ||||
|---|---|---|---|---|---|
| Reviewer | Total | Avg/PR | Behavioral | Structural | Overall |
| Automated Tool | |||||
| Claude-Code | 1336 | 7.3 | 38.1 | 30.7 | 32.1 |
| Codex | 324 | 1.8 | 38.1 | 16.1 | 20.1 |
| Devin | 1344 | 7.3 | 31.0 | 23.4 | 24.8 |
| PR-Agent | 524 | 2.8 | 38.1 | 19.8 | 23.1 |
| Human | 234 | 1.3 | 100 | 100 | 100 |
Table 5. Performance comparison of automated review tools.
| # 评论 | 通过率 (%) | ||||
|---|---|---|---|---|---|
| 审阅者 | 总计 | 平均/PR | 行为型 | 结构型 | 总体 |
| 自动化工具 | |||||
| Claude-Code | 1336 | 7.3 | 38.1 | 30.7 | 32.1 |
| Codex | 324 | 1.8 | 38.1 | 16.1 | 20.1 |
| Devin | 1344 | 7.3 | 31.0 | 23.4 | 24.8 |
| PR-Agent | 524 | 2.8 | 38.1 | 19.8 | 23.1 |
| 人类 | 234 | 1.3 | 100% | 100% | 100% |
表 5. 自动化审阅工具表现对比。
Figure 5. Overlap of passed test for each review tool.
图 5. 各审阅工具通过测试的重叠。
- Clear gap between human and automated review tools. The results show that the automated review tools achieve pass rates ranging from 20.1% to 32.1%, whereas human reviewers achieve a pass rate of 100%. Figure 5 shows the overlap of the passed tests across the four review tools. Considering the union across all four tools, 97 out of the 234 tests were passed by at least one tool (41.5%), which shows there is still a substantial gap between AI-generated reviews and human reviews, even when multiple tools are combined.
- 人类与自动化审阅工具之间差距清晰。 结果显示，自动化审阅工具通过率在 20.1% 到 32.1% 之间，而人类审阅者通过率为 100%。图 5 展示四个审阅工具通过测试的重叠。考虑四工具并集，234 个测试中有 97 个至少被一个工具通过（41.5%），说明即便多工具合并，AI 生成审阅与人类审阅之间仍有实质差距。
- Interpreting the gap: automated tools do not raise the same concerns as humans. This gap should be interpreted carefully. The lower pass rates do not necessarily imply that the generated reviews are low quality and not useful. Instead, they suggest that the reviews produced by automated tools are less aligned with the types of issues that human reviewers typically raise. This difference indicates that automated review tools and humans may be complementary, opening opportunities to combine both within the software development life cycle. We investigate this difference further in RQ2.
- 解读差距：自动化工具并未提出与人类相同的关切。 这一差距应谨慎解读。较低通过率并不必然意味着生成审阅质量低、无用。相反，它们表明自动化工具产出的审阅较少对齐人类审阅者通常提出的问题类型。这一差异提示自动化审阅工具与人类可能互补，为在软件开发生命周期中二者结合打开空间。我们在 RQ2 中进一步考察这一差异。
| Tool | # Comments | # Useful | Useful % |
|---|---|---|---|
| Claude Code | 40 | 31 | 78% |
| Codex | 8 | 7 | 88% |
| Devin | 26 | 22 | 85% |
| PR-Agent | 18 | 17 | 94% |
| Total | 92 | 77 | 84% |
Table 6. Manually labelled usefulness of review comments generated by four AI code review tools across six PRs.
| 工具 | # 评论 | # 有用 | 有用 % |
|---|---|---|---|
| Claude Code | 40 | 31 | 78% |
| Codex | 8 | 7 | 88% |
| Devin | 26 | 22 | 85% |
| PR-Agent | 18 | 17 | 94% |
| 合计 | 92 | 77 | 84% |
表 6. 六个 PR 上四个 AI 代码审阅工具生成审阅评论的人工标注有用性。
- Reviews from automated review tools are useful. To better interpret the low pass rates, we manually inspected a sample of 92 review comments from the evaluated tools on six randomly selected PRs. Two annotators independently examined each PR along with its generated review comments, with access to the full repository. They then labeled each comment as useful or not, where useful comments are those that point out valid issues or suggest improvements to the code. Disagreements were resolved through discussion. The result in Table 6 shows that most comments are useful. This finding indicates that the low pass rates are not a result of poor-quality reviews, but rather because automated review tools tend to highlight different concerns during the review process compared to human reviewers. We investigate this difference further in RQ2.
- 自动化审阅工具的审阅是有用的。 为更好解读低通过率，我们人工检查了六个随机选取 PR 上被评工具的 92 条审阅评论样本。两位标注者独立检查每个 PR 及其生成审阅评论，并可访问完整仓库。随后把每条评论标为有用或无用；有用评论指指出有效问题或建议代码改进的评论。分歧经讨论解决。表 6 结果表明多数评论有用。这一发现表明，低通过率并非审阅质量差的结果，而是因为自动化审阅工具在审阅过程中倾向突出与人类审阅者不同的关切。我们在 RQ2 中进一步考察这一差异。
5.3. RQ2: Categories of Review Comments
5.3. RQ2：审阅评论类别
Figure 6. Categories of review comments generated by four tools and comparison with human reviews.
图 6. 四个工具生成的审阅评论类别及与人类审阅的对比。
Figure 7. Pass rate of automated review tools grouped by the category of the executable tests.
图 7. 按可执行测试类别分组的自动化审阅工具通过率。
For RQ2, we aim to analyze the alignment between human and automated code reviews by assigning a category label to each review comment. The categorization is done manually with the following two-staged protocol. In stage 1, two authors individually perform preliminary inspections on human reviews in the base SWE-CARE dataset, and arrived at the following categories.
对 RQ2，我们旨在通过分析为每条审阅评论赋类别标签，来分析人类与自动化代码审阅的对齐。分类按如下两阶段协议人工完成。阶段 1，两位作者各自对基础 SWE-CARE 数据集中的人类审阅做初步检查，并得到以下类别。
(1) Functional Correctness: Concerns about incorrect or unexpected high-level behavior, regressions, semantic mismatches, or functional bugs.
(2) Testing: Concerns about incorrect, missing, fragile or flaky test functions, weak assertions, lack of test coverage, or requests improvements of test functions or files.
(3) Robustness: Concerns about implementation-level edge cases, such as handling of corner cases in input and control flow.
(4) Compatibility: Concerns about version, software or hardware compatibility.
(5) Documentation: Concerns about correctness, wording, structure, or clarity in docs, examples, links, or code comments.
(6) Design: Concerns about interaction with other parts of the software, such as API interface signature and return values.
(7) Error Handling: Concerns about warning/error messages or exception behavior.
(8) Maintainability: Concerns about readability, information logging, code smells, code complexity, code organization, and consistency with repository-specific coding practices.
(9) Efficiency: Concerns about overhead and computational efficiency.
(10) Security: Concerns about unsafe behavior, security exposure, or safety issues.
(1) 功能正确性： 关乎不正确或意外的高层行为、回归、语义不匹配或功能 bug。
(2) 测试： 关乎不正确、缺失、脆弱或不稳定的测试函数、弱断言、测试覆盖不足，或要求改进测试函数/文件。
(3) 稳健性： 关乎实现级边界情况，例如输入与控制流中的边角处理。
(4) 兼容性： 关乎版本、软件或硬件兼容。
(5) 文档： 关乎文档、示例、链接或代码注释中的正确性、措辞、结构或清晰度。
(6) 设计： 关乎与软件其他部分的交互，例如 API 接口签名与返回值。
(7) 错误处理： 关乎警告/错误消息或异常行为。
(8) 可维护性： 关乎可读性、信息日志、代码异味、代码复杂度、代码组织，以及与仓库特定编码实践的一致性。
(9) 效率： 关乎开销与计算效率。
(10) 安全： 关乎不安全行为、安全暴露或安全问题。
In stage 2, the same authors individually assign each review comment in the dataset to one of the categories defined in stage 1. Then they discuss and resolve the differences.
阶段 2，同一批作者各自把数据集中每条审阅评论赋到阶段 1 定义的某一类，再讨论并消歧差异。
- The distribution of review categories produced by automated tools differs from that of human reviewers. Figure 6 presents the percentage distribution of review comment categories from automated code review tools. Automated tools produce proportions of comments on security, efficiency, compatibility, and error handling that are comparable to those of humans; however, they address design, documentation, and maintainability substantially less frequently. A primary reason for this under representation is that these aspects are often highly specific to individual code repositories and reflect coding practices accumulated over time. This observation highlights a potential gap in automated tools’ understanding of repository-level coding conventions. From another perspective, automated tools tend to raise concerns related to robustness and testing more frequently than humans, suggesting a greater sensitivity to potential corner cases. While such complementary suggestions can be beneficial, the resulting false positives may also substantially increase the burden on maintainers during software development.
- 自动化工具产出的审阅类别分布与人类审阅者不同。 图 6 给出自动化代码审阅工具审阅评论类别的百分比分布。自动化工具在安全、效率、兼容性与错误处理上的评论比例与人类相当；但在设计、文档与可维护性上明显更少。这一不足的一个主要原因是：这些方面往往高度依赖个别代码仓库，并反映长期积累的编码实践。这一观察凸显自动化工具在理解仓库级编码约定上的潜在缺口。从另一角度看，自动化工具比人类更常提出与稳健性和测试相关的关切，提示对潜在边角情况更敏感。这类互补建议可能有益，但由此产生的假阳性也可能在开发中显著加重维护者负担。
- Automated tools capture human-identified issues with varying efficacy across categories. While previous analysis examins what types of issues automated tools tend to raise, we now investigate whether these tools successfully capture the issues that human reviewers identified per category. To this end, Figure 7 reports the pass rate of automated review tools when benchmark instances are grouped by the category of the corresponding human review comments. Recall that each benchmark instance is associated with an executable test derived from a human review. Since human reviews are categorized during the previous labeling process, each executable test inherits the same category. Therefore, the pass rate for each category reflects how often automated reviews successfully capture issues identified by human reviewers. The results show that automated tools do not capture human concerns uniformly across categories. Overall, across tools, high pass rate is observed on robustness and functional correctness. One possible explanation is that these categories are more localized in the patch under review and this makes them easier to capture. In contrast, low pass rate is observed on documentation and design. The pass rate on maintainability is also limited, ranging from 7.9% to 27.0%. These categories often require on broader repository context and project-specific conventions, which current repositories do not document this well and current tools do not utilize this well.
- 自动化工具捕捉人类指出问题的效力随类别变化。 先前分析考察自动化工具倾向提出何种问题，我们现在考察这些工具是否按类别成功捕捉人类审阅者指出的问题。为此，图 7 报告按对应人类审阅评论类别分组时，自动化审阅工具的通过率。回顾：每个基准实例关联由人类审阅导出的可执行测试。因人类审阅在先前标注中已分类，每个可执行测试继承同一类别。因此，每类通过率反映自动化审阅成功捕捉人类审阅者指出问题的频率。结果显示，自动化工具并非均匀捕捉各类人类关切。总体而言，跨工具在稳健性与功能正确性上通过率较高。一种可能解释是：这些类别在待审补丁中更局部，因而更易捕捉。相较之下，文档与设计上通过率低。可维护性上的通过率也有限，介于 7.9% 到 27.0%。这些类别往往需要更广的仓库上下文与项目特定约定，而当前仓库对此记录不佳，当前工具对此利用也不佳。
6. Actionable Insights
6. 可行动洞见
a. Automated review tools should be viewed as complements to human reviewers rather than replacements. In RQ1, all evaluated tools achieve substantially lower pass rates than human reviewers, despite often producing more review comments. Further analysis in RQ2 suggests that this gap is at least partly explained by differences in the types of issues raised: automated tools tend to emphasize certain categories (e.g., robustness and testing) while does not cover well others that humans frequently identify. Taken together, these findings point to the possibility for division of labor in future review workflows.
a. 自动化审阅工具应视为人类审阅者的补充，而非替代。 在 RQ1 中，所有被评工具的通过率都远低于人类审阅者，尽管往往产出更多审阅评论。RQ2 的进一步分析表明，这一差距至少部分可由提出问题类型的差异解释：自动化工具倾向强调某些类别（例如稳健性与测试），而对人类常指出的其他类别覆盖不佳。合起来看，这些发现指向未来审阅工作流中分工的可能。
b. Repository-specific context is a likely missing ingredient for improving automated review tools. In RQ2, automated tools rarely raise issues related to maintainability, design, and documentation, categories that often depend on knowledge specific to a repository. A concrete example can be found in a human review from the Lightning-AI project[^8], where the reviewer suggested using os.getenv(’COLAB_GPU’) instead of IS_COLAB to maintain consistency with existing code. This is not a generic correctness issue; rather, it reflects a local coding convention that is obvious to maintainers but difficult for an automated reviewer to infer from the patch alone. This observation has two implications. For practitioners, documenting repository-specific rules in accessible artifacts, such as architecture notes, style guides, or agent-facing instructions (e.g., AGENTS.md (Lulla et al., 2026)), may improve the effectiveness of automated review tools. For tool developers, review agents should be grounded in richer project context, including architecture documents, prior review history, naming conventions, and similar code patterns, so that generated feedback can better align with human expectations.
b. 仓库特定上下文很可能是改进自动化审阅工具所缺的一环。 在 RQ2 中，自动化工具很少提出与可维护性、设计与文档相关的问题——这些类别往往依赖仓库特定知识。一个具体例子来自 Lightning-AI 项目[^8] 的人类审阅：审阅者建议用 os.getenv('COLAB_GPU') 替代 IS_COLAB，以与现有代码保持一致。这不是通用正确性问题；它反映的是对维护者显然、但自动化审阅者难以仅从补丁推断的本地编码约定。这一观察有两点含义。对实践者而言，把仓库特定规则写进可访问产物——例如架构说明、风格指南或面向智能体的说明（如 AGENTS.md (Lulla et al., 2026)）——或可提升自动化审阅工具效力。对工具开发者而言，审阅智能体应扎根于更丰富的项目上下文，包括架构文档、先前审阅历史、命名约定与相似代码模式，使生成反馈更好对齐人类期望。
c. Tests can be useful artifacts for building code review agents. A key advantage of our benchmark is that it evaluates review comments based on whether they lead to code changes that resolve issues identified during the review process, as captured by executable tests. This provides a clear and objective optimization target for evaluation, allowing automated review tools to be assessed using execution-based signals rather than relying solely on textual similarity or subjective LLM-as-a-judge assessments. More broadly, executable tests also suggest a useful future direction for training review systems, since they reward issue identification that leads to verifiable improvements in code.
c. 测试可以是构建代码审阅智能体的有用产物。 我们基准的一个关键优势是：按审阅评论是否导向解决审阅过程中指出的问题（由可执行测试捕捉）来评估它们。这为评测提供清晰、客观的优化目标，使自动化审阅工具可用基于执行的信号评估，而不只依赖文本相似度或主观的 LLM-as-a-judge。更广地，可执行测试也为训练审阅系统提示有用方向：它们奖励导向可验证代码改进的问题识别。
7. Threats to Validity
7. 效度威胁
a. Internal Validity. Our evaluation relies on a coding agent to revise patches based on review comments. Therefore, the outcomes may reflect not only the quality of the reviews but also the capability of the agent. To mitigate this, we validate benchmark instances using human-written comments prior to inclusion, and the same coding agent is used in both the validation and the evaluation pipeline. This ensures that all retained instances are solvable.
a. 内部效度。 我们的评测依赖编码智能体按审阅评论修订补丁。因此，结果可能既反映审阅质量，也反映智能体能力。为缓解这一点，我们在纳入前用人类写的评论校验基准实例，并在校验与评测管线中使用同一编码智能体。这确保所有保留实例可解。
b. External Validity. Our benchmark is constructed from SWE-CARE and includes 184 pull request instances with 234 verifiable oracles across 56 repositories. While this curation improves reliability, it may limit representativeness with respect to the broader software ecosystem. Nevertheless, the dataset can be easily extended, as the pipeline can be applied to new pull request instances.
b. 外部效度。 我们的基准由 SWE-CARE 构建，含 184 个拉取请求实例与跨 56 个仓库的 234 个可验证预言。虽整理提升了可靠性，相对更广软件生态的代表性可能受限。不过，数据集易于扩展，因为管线可应用于新的拉取请求实例。
c. Construct Validity. The filtering stage in our pipeline leverages an LLM-based classifier, which may retain borderline comments. To mitigate this, we leverage a strict fail-then-pass criterion during test generation and verify whether each problem can be solved during the validation phase using the original human comments. Moreover, manual analysis of the reviews involve human judgment. To mitigate this, two authors independently performed labeling.
c. 构念效度。 我们管线的过滤阶段依赖基于 LLM 的分类器，可能保留边界评论。为缓解这一点，我们在测试生成时采用严格的「先失败后通过」标准，并在校验阶段用原始人类评论验证每个问题是否可解。此外，审阅的人工分析涉及人类判断。为缓解这一点，两位作者独立完成标注。
8. Conclusion
8. 结论
The rapid advancement of AI-based code generation has made generating code becomes increasingly easy and scalable. On the other hand, the process of reviewing the code remains a bottleneck. Several automated code review tools have been proposed, but evaluating their effectiveness is an open challenge.
基于 AI 的代码生成快速发展，使写代码日益容易、可扩展。另一方面，审阅代码的过程仍是瓶颈。已有若干自动化代码审阅工具被提出，但评估其效力仍是开放挑战。
In this paper, we revisit the problem of evaluating code review tools. We propose c-CRAB, a benchmark designed to assess how well automated review tools identify the same issues that human reviewers raise in realistic PR settings. c-CRAB adopts a test-based evaluation, where it converts human review comments into executable tests. Our empirical results show that current state-of-the-art review tools identify only a fraction of the issues captured in the benchmark. This finding highlights the limitations in existing code review tools. Further analysis reveals that tool-generated reviews often highlight different aspects of code compared to human reviewers. This suggests that human and automated review tools are complementary to each other. Looking forward, our findings suggest several directions for future research, such as integrating automated tools into human-centric review workflows and improving the ability of code review tools to better capture relevant context in the codebase.
本文重新审视代码审阅工具的评测问题。我们提出 c-CRAB：一个基准，用于评估自动化审阅工具在真实 PR 场景中，能否找出人类审阅者提出的同一批问题。c-CRAB 采用基于测试的评测，把人类审阅评论转成可执行测试。实证结果表明，当前前沿审阅工具只识别基准中一部分问题。这一发现凸显现有代码审阅工具的局限。进一步分析显示，工具生成的审阅常突出与人类审阅者不同的代码方面。这提示人类与自动化审阅工具彼此互补。展望未来，我们的发现建议若干研究方向，例如把自动化工具接入以人为中心的审阅工作流，并提升代码审阅工具更好捕捉代码库中相关上下文的能力。
Data Availability
数据可用性
Our replication package, including source code, benchmark datasets and results, is available at https://github.com/c-CRAB-Benchmark.
我们的复现包，包括源码、基准数据集与结果，见 https://github.com/c-CRAB-Benchmark。
Disclaimer
声明
Acknowledgements.
The views and conclusions expressed in this paper are those of the authors alone and do not represent the official policies or endorsements of SonarSource. Furthermore, the findings presented herein are independent and should not be interpreted as an evaluation of the quality of products at SonarSource.
致谢。
本文表达的观点与结论仅代表作者本人，不代表 SonarSource 的官方政策或背书。此外，文中发现独立于 SonarSource，不应解读为对该公司产品质量的评估。
References
参考文献
Antoniak and Mimno (2018) M. Antoniak and D. Mimno Evaluating the stability of embedding-based word similarities . Trans. Assoc. Comput. Linguistics 6 , pp. 107–119 . Cited by: §3.1 .
Guo et al. (2025) H. Guo, X. Zheng, Z. Liao, H. Yu, P. Di, Z. Zhang, and H. Dai CodeFuse-cr-bench: A comprehensiveness-aware benchmark for end-to-end code review evaluation in python projects . CoRR abs/2509.14856 . Cited by: §1 , §2.2 , §2.3 , §2.3 , Table 1 , §4.4 .
Hassan et al. (2025) A. E. Hassan, H. Li, D. Lin, B. Adams, T. Chen, Y. Kashiwa, and D. Qiu Agentic software engineering: foundational pillars and a research roadmap . CoRR abs/2509.06216 . Cited by: §1 .
Hu et al. (2025) R. Hu, X. Wang, X. Wen, Z. Zhang, B. Jiang, P. Gao, C. Peng, and C. Gao Benchmarking llms for fine-grained code review with enriched context in practice . CoRR abs/2511.07017 . Cited by: §1 , §2.2 , §2.3 , Table 1 .
Jiang et al. (2025) Y. Jiang, H. Liu, T. Chen, F. Fan, C. Dong, K. Liu, and L. Zhang Deep assessment of code review generation approaches: beyond lexical similarity . CoRR abs/2501.05176 . Cited by: §1 , §2.3 , §3.1 .
Li et al. (2025a) S. Li, D. Wang, P. Thongtanunam, Z. Wang, J. Yu, and J. Chen Issue-oriented agent-based framework for automated review comment generation . CoRR abs/2511.00517 . Cited by: §1 .
Li et al. (2025b) W. Li, X. Zhang, Z. Guo, S. Mao, W. Luo, G. Peng, Y. Huang, H. Wang, and S. Li FEA-bench: A benchmark for evaluating repository-level code generation for feature implementation . In ACL (1) , pp. 17160–17176 . Cited by: §1 .
Li et al. (2022) Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu, and N. Sundaresan Automating code review activities by large-scale pre-training . In ESEC/SIGSOFT FSE , pp. 1035–1047 . Cited by: §2.2 , Table 1 .
Lin (2004) C. Lin ROUGE: a package for automatic evaluation of summaries . In Text Summarization Branches Out , Barcelona, Spain , pp. 74–81 . External Links: Link Cited by: §2.3 .
Lu et al. (2025) J. Lu, X. Li, Z. Hua, L. Yu, S. Cheng, L. Yang, F. Zhang, and C. Zuo DeepCRCEval: revisiting the evaluation of code review comment generation . In FASE , Lecture Notes in Computer Science , Vol. 15693 , pp. 43–64 . Cited by: §2.3 .
Lulla et al. (2026) J. L. Lulla, S. Mohsenimofidi, M. Galster, J. M. Zhang, S. Baltes, and C. Treude On the impact of agents.md files on the efficiency of AI coding agents . CoRR abs/2601.20404 . External Links: Link , Document , 2601.20404 Cited by: §6 .
Naik et al. (2025) A. Naik, M. Alenius, D. Fried, and C. P. Rosé CRScore: grounding automated evaluation of code review comments in code claims and smells . In NAACL (Long Papers) , pp. 9049–9076 . Cited by: §1 , §2.2 , §2.2 , §2.3 , Table 1 .
Papineni et al. (2002) K. Papineni, S. Roukos, T. Ward, and W. Zhu Bleu: a method for automatic evaluation of machine translation . In ACL , pp. 311–318 . Cited by: §2.3 .
Pereira et al. (2026) K. Pereira, N. Sinha, R. Ghosh, and D. Dutta CR-bench: evaluating the real-world utility of ai code review agents . External Links: 2603.11078 , Link Cited by: Table 1 .
Pham and Ghaleb (2026) D. Pham and T. A. Ghaleb Code change characteristics and description alignment: A comparative study of agentic versus human pull requests . CoRR abs/2601.17627 . Cited by: §1 .
Popovic (2015) M. Popovic ChrF: character n-gram f-score for automatic MT evaluation . In WMT@EMNLP , pp. 392–395 . Cited by: §2.3 .
Rasheed et al. (2024) Z. Rasheed, M. A. Sami, M. Waseem, K. Kemell, X. Wang, A. Nguyen, K. Systä, and P. Abrahamsson AI-powered code review with llms: early results . CoRR abs/2404.18496 . Cited by: §1 .
Robbes et al. (2026) R. Robbes, T. Matricon, T. Degueule, A. C. Hora, and S. Zacchiroli Agentic much? adoption of coding agents on github . CoRR abs/2601.18341 . Cited by: §1 .
Shi et al. (2025) L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. Vosoughi Judging the judges: A systematic study of position bias in llm-as-a-judge . In IJCNLP-AACL (long papers) , pp. 292–314 . Cited by: §2.3 .
Tang et al. (2024) X. Tang, K. Kim, Y. Song, C. Lothritz, B. Li, S. Ezzini, H. Tian, J. Klein, and T. F. Bissyandé CodeAgent: autonomous communicative agents for code review . In EMNLP , pp. 11279–11313 . Cited by: §1 .
Tantithamthavorn et al. (2026) C. K. Tantithamthavorn, Y. Zou, A. Wong, M. Gupta, Z. Wang, M. Buller, R. Jiang, M. Watson, M. Jeong, K. Chen, and M. Wu RovoDev code reviewer: A large-scale online evaluation of llm-based code review automation at atlassian . CoRR abs/2601.01129 . Cited by: §1 , §2.2 , §2.3 , Table 1 .
Thakur et al. (2024) A. S. Thakur, K. Choudhary, V. S. Ramayapally, S. Vaidyanathan, and D. Hupkes Judging the judges: evaluating alignment and vulnerabilities in llms-as-judges . CoRR abs/2406.12624 . External Links: Link , Document , 2406.12624 Cited by: §2.3 .
Tufano et al. (2022) R. Tufano, S. Masiero, A. Mastropaolo, L. Pascarella, D. Poshyvanyk, and G. Bavota Using pre-trained models to boost code review automation . In ICSE , pp. 2291–2302 . Cited by: §1 , §2.2 , §2.3 , Table 1 .
Tufano et al. (2021) R. Tufano, L. Pascarella, M. Tufano, D. Poshyvanyk, and G. Bavota Towards automating code review activities . In ICSE , pp. 163–174 . Cited by: §1 , §2.2 , §2.3 , Table 1 .
Xie et al. (2025) C. Xie, B. Li, C. Gao, H. Du, W. Lam, D. Zou, and K. Chen SWE-fixer: training open-source llms for effective and efficient github issue resolution . In ACL (Findings) , Findings of ACL , Vol. ACL 2025 , pp. 1123–1139 . Cited by: §1 .
Yanay and Fingerman (2026) T. Yanay and B. Fingerman How we built a real-world benchmark for ai code review (Website) Note: Qodo blog post External Links: Link Cited by: Table 1 .
Zeng et al. (2025) Z. Zeng, R. Shi, K. Han, Y. Li, K. Sun, Y. Wang, Z. Yu, R. Xie, W. Ye, and S. Zhang Benchmarking and studying the llm-based code review . CoRR abs/2509.01494 . Cited by: §2.3 .
Zhang et al. (2026) L. Zhang, Y. Yu, M. Yu, X. Guo, Z. Zhuang, G. Rong, D. Shao, H. Shen, H. Kuang, Z. Li, B. Wang, G. Zhang, B. Xiang, and X. Xu AACR-bench: evaluating automatic code review with holistic repository-level context . External Links: 2601.19494 , Link Cited by: §1 , §2.2 , §2.2 , §2.3 , Table 1 .
Zhang et al. (2024) Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury AutoCodeRover: autonomous program improvement . In ISSTA , pp. 1592–1604 . Cited by: §1 .
Zhou et al. (2026) Q. Zhou, J. Zhang, H. Wang, R. Hao, J. Wang, M. Han, Y. Yang, S. Wu, F. Pan, L. Fan, D. Tu, and Z. Zhang FeatureBench: benchmarking agentic coding for complex feature development . External Links: 2602.10975 , Link Cited by: §1 .