SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback

SWE-PRBench：用拉取请求反馈衡量 AI 代码审阅质量

Deepak Kumar Affiliation:  Independent Researcher Email:  deepak.kumar@foundryhq.ai

Abstract

摘要

We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Evaluated against an LLM-as-judge framework validated at $\kappa{=}0.75$, 8 frontier models detect only 15–31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks. Pull requests are drawn from active open-source repositories, filtered from 700 candidates using a Repository Quality Score, and evaluated under three frozen context configurations : diff only (config_A), diff with file content (config_B), and full context (config_C) : enabling systematic ablation of context provision strategies. All 8 models degrade monotonically from config_A to config_C even when context is provided via structured semantic layers including AST-extracted function context and import graph resolution. The dominant mechanism is a collapse of Type2_Contextual issue detection at config_B, consistent with attention dilution in long contexts [18]: a structured 2,000-token diff-with-summary prompt outperforms a 2,500-token full-context prompt enriched with execution context, behaviour mapping, and test signatures across all 8 models. The top four models are statistically indistinguishable ($\bar{s}$ 0.147–0.153) while a clear tier gap separates them from the remaining four ($\bar{s}\leq 0.113$). Dataset, contexts, annotations, and evaluation harness are released publicly.

我们提出 SWE-PRBench：一个含 350 条拉取请求、带人工标注真值的基准，用于评估 AI 代码审阅质量。在经 $\kappa{=}0.75$ 校验的 LLM-as-judge 框架下，8 个前沿模型在仅含 diff 的配置上只能检出 15%–31% 的人工标出问题，说明尽管代码生成基准成绩亮眼，AI 代码审阅仍远低于人类专家。拉取请求来自活跃开源仓库，从 700 个候选里用仓库质量分（Repository Quality Score）筛选，并在三种冻结上下文配置下评测：仅 diff（config_A）、diff 加文件内容（config_B）、完整上下文（config_C），以便系统消融上下文供给策略。即便通过 AST 抽取函数上下文与导入图解析等结构化语义层提供上下文，8 个模型仍从 config_A 到 config_C 单调下降。主导机制是 config_B 上 Type2_Contextual 问题检出的崩塌，与长上下文中的注意力稀释一致 [18]：一份结构化、约 2,000 token 的「diff+摘要」提示，在全部 8 个模型上胜过一份约 2,500 token、补上执行上下文、行为映射与测试签名的完整上下文提示。前四名模型在统计上无法区分（$\bar{s}$ 0.147–0.153），与其余四名（$\bar{s}\leq 0.113$）之间存在清晰档差。数据集、上下文、标注与评测编排外壳均已公开发布。

Keywords. code review automation, large language models, software engineering benchmarks, pull request analysis, context-induced performance degradation, LLM-as-judge evaluation.

关键词。 代码审阅自动化、 大语言模型、 软件工程基准、 拉取请求分析、 上下文诱发的性能退化、 LLM-as-judge 评估。

1 Introduction

1 引言

We introduce SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. Existing benchmarks measure whether models can produce correct code [3, 4, 5]; none measure whether a model can evaluate proposed code changes as an expert reviewer would. We address this gap. Code review is a judgment task: the reviewer is presented with a diff, not a broken system; the goal is identifying and explaining problems in someone else’s work, not generating a solution. Ground truth is distributed across human expert judgments rather than captured by pass/fail tests, and quality varies simultaneously across depth, actionability, and factual accuracy [1, 9]. A model can score highly on SWE-Bench while being unreliable as a code reviewer. Review bottlenecks are well-documented [2]; automating the evaluation task requires a benchmark that measures it directly.

我们提出 SWE-PRBench：含 350 条拉取请求、带人工标注真值的基准，用于评估 AI 代码审阅质量。现有基准衡量模型能否生成正确代码 [3, 4, 5]；没有哪个衡量模型能否像专家审阅者那样评价拟议改动。我们补上这一缺口。代码审阅是判断任务：审阅者面对的是 diff，不是坏掉的系统；目标是发现并解释别人工作中的问题，而不是自己写出解法。真值分散在人类专家判断里，而不是 pass/fail 测试；质量同时体现在深度、可操作性与事实准确性上 [1, 9]。模型可以在 SWE-Bench 上拿高分，却当不好代码审阅者。审阅瓶颈早有文献记载 [2]；要自动化「评价」这一任务，需要能直接度量它的基准。

SWE-PRBench provides 350 pull requests selected from 700 candidates via a Repository Quality Score, evaluated under three frozen context configurations of increasing richness: diff only (config_A), diff with file content (config_B), and full context (config_C). Eight frontier models are evaluated using an LLM-as-judge framework validated at $\kappa{=}0.75$. Baseline results establish that no model detects more than 31% of human-flagged issues, and all 8 models degrade monotonically as context expands: a 2,000-token structured diff-with-summary prompt outperforms a 2,500-token full-context prompt across every model tested.

SWE-PRBench 提供 350 条从 700 个候选中按仓库质量分选出的拉取请求，在三种信息量递增的冻结上下文配置下评测：仅 diff（config_A）、diff 加文件内容（config_B）、完整上下文（config_C）。用经 $\kappa{=}0.75$ 校验的 LLM-as-judge 框架评估 8 个前沿模型。基线结果表明，没有模型能检出超过 31% 的人工标出问题，且 8 个模型都随上下文扩充而单调下降：约 2,000 token 的结构化「diff+摘要」提示，在每个被测模型上都胜过约 2,500 token 的完整上下文提示。

We address the following research questions: (RQ1) What fraction of human-flagged issues can current LLMs detect, and how do models differ in hallucination rate? (RQ2) Does additional code context improve review quality, and in what direction does performance move? (RQ3) Is context degradation uniform across capability tiers and difficulty types? (RQ4) What categories of issues remain consistently beyond current model capability?

我们回答以下研究问题：（RQ1） 当前 LLM 能检出多大比例的人工标出问题，模型间幻觉率差多少？（RQ2） 额外代码上下文是否提升审阅质量，性能往哪个方向走？（RQ3） 上下文退化在能力档次与难度类型上是否均匀？（RQ4） 哪些类别的问题始终超出当前模型能力？

This paper makes the following contributions.

本文贡献如下。

• • Dataset. 350 PRs with human-annotated ground truth across 6 languages and 3 difficulty types (Type1_Direct, Type2_Contextual, Type3_Latent).
• • Context configurations. Three frozen, reproducible structures (config_A, config_B, config_C) enabling systematic context ablation.
• • Evaluation protocol. A multi-dimensional scorer with LLM-as-judge classification and bipartite matching, validated at $\kappa{=}0.75$ and cross-validated under a second judge ($\kappa{=}0.616$).
• • Finding. All 8 models degrade monotonically from config_A to config_C. The primary mechanism is attention dilution [18]: unstructured file content harms Type2_Contextual detection specifically, indicating that attention representation rather than content selection is the binding constraint for context-aware code review.
• • Infrastructure. A public SWE-PRBench dataset on HuggingFace (https://huggingface.co/datasets/foundry-ai/swe-prbench), evaluation harness on GitHub (https://github.com/FoundryHQ-AI/swe-prbench), to support community benchmarking and extension.

• • 数据集。 350 条带人工标注真值的 PR，覆盖 6 种语言与 3 种难度类型（Type1_Direct、Type2_Contextual、Type3_Latent）。
• • 上下文配置。 三种冻结、可复现的结构（config_A、config_B、config_C），支持系统消融上下文。
• • 评测协议。 多维打分器，含 LLM-as-judge 分类与二分图匹配，经 $\kappa{=}0.75$ 校验，并在第二裁判下交叉验证（$\kappa{=}0.616$）。
• • 发现。 8 个模型均从 config_A 到 config_C 单调下降。主因是注意力稀释 [18]：非结构化文件内容尤其伤害 Type2_Contextual 检出，说明对上下文敏感的代码审阅而言，约束在注意力表示，而不在内容筛选。
• • 基础设施。 公开 HuggingFace 数据集（https://huggingface.co/datasets/foundry-ai/swe-prbench）与 GitHub 评测编排外壳（https://github.com/FoundryHQ-AI/swe-prbench），供社区基准与扩展。

Table 1 summarises how SWE-PRBench differs from the closest existing code review datasets and evaluation frameworks.

表 1 概括 SWE-PRBench 与最接近的代码审阅数据集、评测框架有何不同。

Property	CodeReviewer	DeepCRCEval	RovoDev	SWE-PRBench (Ours)
Primary contribution	Model + dataset	Eval metrics	Production tool	Dataset + protocol
Ground truth source	Synthetic pairs	Generated	CRR metric only	Human reviewers*∗*
Source links retained	No	No	N/A	Yes
Difficulty taxonomy	None	None	None	3 types
Context configs	None	None	None	3 frozen
Issue detection eval	No	No	Partial	Yes
Judge validated	No	No	No	$\kappa{=}0.75$
Public dataset	Partial	No	No	Yes

Table 1: Comparison of SWE-PRBench with related code review evaluation resources.

属性	CodeReviewer	DeepCRCEval	RovoDev	SWE-PRBench（本文）
主要贡献	模型+数据集	评测指标	生产工具	数据集+协议
真值来源	合成配对	生成	仅 CRR 指标	人类审阅者*∗*
保留源链接	否	否	不适用	是
难度分类	—	—	—	3 类
上下文配置	—	—	—	3 种冻结
问题检出评测	否	否	部分	是
裁判经校验	否	否	否	$\kappa{=}0.75$
公开数据集	部分	否	否	是

表 1：SWE-PRBench 与相关代码审阅评测资源对比。

2 Related Work

2 相关工作

2.1 Code Generation and Repair Benchmarks

2.1 代码生成与修复基准

SWE-Bench [3] tasks models with resolving GitHub issues by producing correct patches; top models score around 70% on the verified subset. SWE-Bench Pro [4] increases difficulty through GPL-licensed repositories and human-augmented specifications, with top models scoring approximately 23%. SWE Atlas [5] extends evaluation to deep codebase comprehension through agentic exploration. All three benchmarks measure code production. None measures code evaluation: the cognitive task of identifying problems in someone else’s proposed work requires different capabilities [1] and is not predicted by generation benchmark scores.

SWE-Bench [3] 要求模型通过写出正确补丁来解决 GitHub issue；顶尖模型在 verified 子集上约 70%。SWE-Bench Pro [4] 用 GPL 仓库与人工增强规格抬高难度，顶尖约 23%。SWE Atlas [5] 通过智能体式探索把评测扩展到深代码库理解。三者都度量代码生产。没有一个度量代码评价：在别人拟议工作中找问题，需要不同能力 [1]，也不能用生成基准分数预测。

2.2 Code Review Automation

2.2 代码审阅自动化

Automated code review has evolved from static analysis tools to pre-trained models. T5CR [6] and CodeReviewer [7] use T5-based architectures for review generation and code refinement; Tufano et al. [8] found that direct LLM use surpasses these on refinement but lags on review generation. LAURA [16] improves review generation by up to 38% IH-Score through retrieval-augmented context, demonstrating that context curation matters. DeepCRCEval [13] proposes LLM-based metrics to replace BLEU for evaluating generated review text. Our work is complementary: where DeepCRCEval evaluates the quality of generated review text given a fixed model, SWE-PRBench evaluates whether a model identifies the correct issues at all, a prerequisite question that precedes comment quality.

自动化代码审阅已从静态分析走到预训练模型。T5CR [6] 与 CodeReviewer [7] 用基于 T5 的架构做审阅生成与代码精炼；Tufano 等人 [8] 发现直接用 LLM 在精炼上超过这些模型，但在审阅生成上落后。LAURA [16] 靠检索增强上下文把审阅生成 IH-Score 最多提升 38%，说明上下文整理很重要。DeepCRCEval [13] 提出用 LLM 指标替代 BLEU 来评生成审阅文本。我们的工作互补：DeepCRCEval 在固定模型下评生成审阅文本质量，SWE-PRBench 则评模型是否首先找对了问题——这是评论质量之前的前置问题。

Existing code review datasets. The dominant dataset resource is the CodeReviewer dataset [7], released alongside the CodeT5-based model. Prior model papers including Tufano et al. [6, 8] used training data not released as standalone evaluation benchmarks. All existing datasets have been shown to contain approximately 25–32% low-quality entries despite filtering [8]. Critically, no existing dataset retains PR context beyond the diff itself, source links are removed, and none is structured for evaluation of issue detection capability. SWE-PRBench addresses all three gaps: it retains full PR context, links to source repositories, and defines ground truth in terms of human-flagged issues rather than syntactic diff similarity.

现有代码审阅数据集。 主流资源是 CodeReviewer 数据集 [7]，与基于 CodeT5 的模型一并发布。包括 Tufano 等人 [6, 8] 在内的先前模型论文所用训练数据并未作为独立评测基准发布。即便经过过滤，现有数据集仍约有 25%–32% 低质量条目 [8]。关键在于：现有数据集都不保留 diff 之外的 PR 上下文，源链接被去掉，也未按「问题检出能力」来组织评测。SWE-PRBench 补上这三点：保留完整 PR 上下文、链到源仓库，并把真值定义为人工标出的问题，而非句法上的 diff 相似度。

2.3 LLM-as-Judge Methodology

2.3 LLM-as-Judge 方法

MT-Bench [10] established that GPT-4 as evaluator achieves over 80% agreement with human raters on open-ended tasks. G-Eval [11] extended this with chain-of-thought evaluation. JudgeBench [17] showed that many models perform near-randomly on challenging factual judgment tasks, motivating our choice of a high-capability fixed judge model validated against an explicit rubric ($\kappa{=}0.75$, Section 5.4) and cross-validated under a second judge ($\kappa{=}0.616$, Section 5.5). LAURA’s finding that curated retrieval-augmented context helps [16] is consistent with our result: context benefit depends on curation. Our config_B represents the uncurated baseline against which retrieval-based approaches should be measured.

MT-Bench [10] 表明 GPT-4 作评委在开放任务上与人类评分者一致率超 80%。G-Eval [11] 用思维链评测加以扩展。JudgeBench [17] 显示许多模型在困难事实判断上接近随机，这促使我们选用高能力固定裁判模型，对照显式量表校验（$\kappa{=}0.75$，第 5.4 节），并在第二裁判下交叉验证（$\kappa{=}0.616$，第 5.5 节）。LAURA 关于整理过的检索增强上下文有帮助的发现 [16] 与我们一致：上下文是否有益取决于整理。我们的 config_B 代表未整理基线，检索类方法应以之为对照。

3 The SWE-PRBench Dataset

3 SWE-PRBench 数据集

3.1 Repository Selection and Quality Scoring

3.1 仓库选择与质量打分

Not all repositories have substantive review cultures. We define a Repository Quality Score (RQS) aggregating five components (Table 2) to select only repositories where human experts engage in technical discussion before merging. Repositories scoring below 60/100 are excluded. The median RQS of retained repositories is 74.

并非所有仓库都有实质审阅文化。我们定义仓库质量分 (RQS)，汇总五个分量（表 2），只保留合并前有人类专家技术讨论的仓库。低于 60/100 的排除。保留仓库的 RQS 中位数为 74。

Component	What it measures	Max pts
Review culture	Share of substantive human review comments	30
PR recency	Merged PRs in last 90 days	25
Test quality	Test files, CI presence, coverage tooling	20
PR volume	Average monthly merged PRs over 6 months	15
Contamination	Inverse of star count	10
Total		100

Table 2: Repository Quality Score (RQS) components.

分量	衡量什么	满分
审阅文化	实质人类审阅评论占比	30
PR 时效	近 90 天已合并 PR	25
测试质量	测试文件、CI、覆盖率工具	20
PR 体量	6 个月月均已合并 PR	15
污染风险	星标数的逆	10
合计		100

表 2：仓库质量分（RQS）各分量。

Review culture (max 30) is the primary signal: we sample the 30 most recent merged PRs within 90 days and score by the fraction of substantive human comments (human author, $\geq$10 words, substance keywords or code references). PR recency ensures active maintenance. Test quality signals reliable CI infrastructure. PR volume distinguishes sustained from burst activity. Contamination penalises high-star repositories likely to appear in training corpora.

审阅文化（满分 30）是主信号：我们抽样近 90 天内最近 30 条已合并 PR，按实质人类评论占比打分（人类作者、$\geq$10 词、含实质关键词或代码引用）。PR 时效保证仓库仍在维护。测试质量反映可靠 CI。PR 体量区分持续活跃与突发活跃。污染项惩罚高星仓库——它们更可能进入训练语料。

3.2 PR Collection and Filtering

3.2 PR 采集与过滤

Ground truth for SWE-PRBench consists of review comments written by human engineers during the actual review process on merged pull requests. Comments are collected after the fact from GitHub’s review API; no comments are generated, synthesised, or modified during dataset construction. This distinguishes SWE-PRBench from prior code review datasets that derive ground truth from model-generated or heuristically-filtered comments [7, 6].

SWE-PRBench 的真值是：已合并拉取请求在真实审阅过程中由人类工程师写下的审阅评论。评论事后从 GitHub 审阅 API 采集；构建数据集时不生成、不合成、不改写任何评论。这使 SWE-PRBench 有别于以模型生成或启发式过滤评论作真值的先前代码审阅数据集 [7, 6]。

For each qualifying repository we collect merged pull requests using the GitHub GraphQL API [19] for PR metadata, reviews, and file-level information, and the GitHub REST API separately to fetch full unified diff patches (which GraphQL does not return). The collection window spans six months prior to the benchmark freeze date, with a minimum age of 30 days to ensure that review cycles have completed. We collect up to 30 PRs per repository, yielding an initial pool of approximately 3,000 raw pull requests.

对每个合格仓库，我们用 GitHub GraphQL API [19] 采集已合并拉取请求的元数据、审阅与文件级信息，再用 GitHub REST API 单独拉取完整统一 diff 补丁（GraphQL 不返回）。采集窗口为基准冻结日前六个月，且 PR 至少已满 30 天，以确保审阅周期结束。每仓最多 30 条，初始约 3,000 条原始拉取请求。

Raw pull requests are passed through a ten-stage filtering pipeline before entering the dataset. Stages filter: (1) merged-only status; (2) at least two substantive human comments; (3) at least one non-test file changed; (4) not documentation-only changes; (5) not automated dependency updates (Dependabot, Renovate); (6) not AI-dominated reviews, defined as more than 30% of comments originating from known AI review bots or matching AI-generated comment signatures; (7) diff parseable without API errors; (8) base commit accessible for file content retrieval; (9) repository still public at collection time; and (10) PR Review Value Score (RVS, Section 3.3) at or above 0.3.

原始拉取请求经十级过滤管线后才进入数据集。各级过滤：（1） 仅已合并；（2） 至少两条实质人类评论；（3） 至少改动一个非测试文件；（4） 非纯文档改动；（5） 非自动化依赖更新（Dependabot、Renovate）；（6） 非 AI 主导审阅——定义为超过 30% 评论来自已知 AI 审阅机器人或匹配 AI 生成评论特征；（7） diff 可解析且无 API 错误；（8） 基提交可访问以取文件内容；（9） 采集时仓库仍公开；（10） PR 审阅价值分（RVS，第 3.3 节）不低于 0.3。

Bot and AI detection is explicit. The is_bot_username() function checks reviewer logins against a curated list of known AI review accounts (coderabbitai, github-copilot[bot], cursor-bugbot, qodo-merge-pro[bot], and 15 additional entries), and also matches accounts ending in [bot] or containing -bot. The is_ai_generated_comment() function detects AI-generated comment bodies via structural patterns: three or more Markdown H2 headers, three or more table lines, known AI review phrases, or length exceeding 2,000 characters with two or more headers. PRs where the AI comment ratio exceeds the 30% threshold are excluded from the dataset, ensuring that ground truth reflects genuine human engineering judgment rather than AI-assisted review [14].

机器人与 AI 检测是显式的。is_bot_username() 对照已知 AI 审阅账号列表（coderabbitai、github-copilot[bot]、cursor-bugbot、qodo-merge-pro[bot] 及另外 15 项），并匹配以 [bot] 结尾或含 -bot 的账号。is_ai_generated_comment() 用结构模式检测 AI 生成正文：三个及以上 Markdown H2、三行及以上表格、已知 AI 审阅套话，或长度超 2,000 字符且含两个及以上标题。AI 评论占比超 30% 的 PR 被排除，确保真值反映真实人类工程判断，而非 AI 辅助审阅 [14]。

After all filtering stages, the 700 candidate pull requests are scored with RQS and the 350 highest-quality PRs are retained as the final SWE-PRBench dataset.

全部过滤后，700 个候选拉取请求再按 RQS 打分，保留质量最高的 350 条作为最终 SWE-PRBench 数据集。

3.3 PR Review Value Score

3.3 PR 审阅价值分

Repository-level quality filtering via RQS is necessary but not sufficient: individual PR quality varies substantially even within high-RQS repositories. We define a PR Review Value Score (RVS) to quantify the ground-truth signal carried by each pull request and to weight its contribution to aggregate benchmark metrics.

仅靠仓库级 RQS 过滤必要但不充分：即便在高 RQS 仓库内，单条 PR 质量仍差异很大。我们定义 PR 审阅价值分 (RVS)，量化每条拉取请求携带的真值信号，并用于加权其对汇总基准指标的贡献。

RVS is a weighted sum of five components, each normalised to $[0,1]$:

RVS 是五个分量的加权和，每个归一化到 $[0,1]$：

$$\text{RVS}=0.25\cdot d_{\text{depth}}+0.20\cdot d_{\text{complexity}}+0.20\cdot d_{\text{discussion}}+0.15\cdot d_{\text{test}}+0.20\cdot d_{\text{bug}} \tag{1}$$

where $d_{\text{depth}}$ captures substantive comment count, thread count, and change-request events (normalised to 20); $d_{\text{complexity}}$ reflects file count, lines changed, and cross-directory scope; $d_{\text{discussion}}$ counts reply threads and unique reviewers; $d_{\text{test}}$ scores test modification signal (1.0 if tests are modified, 0.5 if added, 0 otherwise); and $d_{\text{bug}}$ scores bug-fix signal from title and label patterns. In the aggregate scoring formula, RVS contributes through a difficulty weight of $\log(\text{human\_comments}+1)$, ensuring that a PR with 25 expert-level review comments contributes more to aggregate metrics than a PR with three comments.

其中 $d_{\text{depth}}$ 刻画实质评论数、线程数与变更请求事件（归一化到 20）；$d_{\text{complexity}}$ 反映文件数、改动行数与跨目录范围；$d_{\text{discussion}}$ 计回复线程与独立审阅者；$d_{\text{test}}$ 打测试改动信号（改测试为 1.0、新增为 0.5、否则 0）；$d_{\text{bug}}$ 据标题与标签模式打缺陷修复信号。在汇总打分公式中，RVS 通过难度权重 $\log(\text{human\_comments}+1)$ 进入，确保有 25 条专家级审阅评论的 PR 对汇总指标的贡献大于只有三条评论的 PR。

3.4 Difficulty Taxonomy

3.4 难度分类

We classify each pull request into one of three difficulty types based on where the evidence for a reviewable issue resides in the codebase [1]. Classification uses the is_in_diff field of human reviewer comments, cross-referenced against diff hunk line ranges parsed from the raw unified diff.

我们按可审阅问题证据落在代码库何处，把每条拉取请求归入三种难度之一 [1]。分类使用人类审阅评论的 is_in_diff 字段，并对照从原始统一 diff 解析出的 hunk 行范围交叉核对。

Type1_Direct (34%). The issue is directly visible in the changed lines. A reviewer needs only the diff to identify it. Example: missing info < 0 error handling in a diff hunk that only checks info > 0. Type1 detection rate should be stable across all three context configurations.

Type1_Direct（34%）。 问题在改动行上直接可见。审阅者只需 diff 就能发现。例：某 hunk 只检查 info > 0，却缺少 info < 0 的错误处理。Type1 检出率在三种上下文配置下应保持稳定。

Type2_Contextual (40%). The issue requires understanding how changed code interacts with surrounding unchanged code in the same file. Example: a height-to-hash edge case where the bug is in the new logic, but understanding why it is wrong requires seeing the existing blockchain interface. Type2 detection rate should improve from config_A to config_B.

Type2_Contextual（40%）。 问题需要理解改动代码如何与同文件未改动代码交互。例：高度到哈希的边界情况——缺陷在新逻辑里，但要看出为何错，必须看现有区块链接口。Type2 检出率应从 config_A 到 config_B 提升。

Type3_Latent_Candidate (26%). The issue resides in files that import or depend on the changed files. Example: a Cython change that exposes a test teardown pattern issue in a file that uses the changed module. Type3 issues require the full context provided in config_C and, by design, test whether models can reason across file boundaries : a capability that current context windows only partially support.

Type3_Latent_Candidate（26%）。 问题落在导入或依赖被改文件的文件里。例：一次 Cython 改动暴露了使用该模块的文件里测试 teardown 模式问题。Type3 需要 config_C 提供的完整上下文，并刻意检验模型能否跨文件边界推理——当前上下文窗口只部分支持这一能力。

This taxonomy directly drives our ablation: if context provides signal rather than noise, Type2 detection should improve from config_A to config_B, and Type3 should only become accessible in config_C. Our results show the opposite: Type2 scores collapse at config_B across all models, establishing that the current context structure does not help contextual issues and actively harms performance on them.

这一分类直接驱动消融：若上下文提供的是信号而非噪声，Type2 检出应从 config_A 升到 config_B，Type3 应只在 config_C 才可及。结果却相反：所有模型在 config_B 上 Type2 分数崩塌，说明当前上下文结构帮不了上下文类问题，反而主动伤害它们上的表现。

3.5 Contamination Mitigation

3.5 污染缓解

Model training corpora may include GitHub pull requests, creating a risk that benchmark PRs are partially memorised rather than genuinely reasoned over [3, 4]. We apply four complementary mitigations. First, all PRs are sourced from a six-month collection window ending at the benchmark freeze date, ensuring recency relative to the training cutoffs of evaluated models. Second, the RQS contamination component penalises high-star repositories, which are more likely to be crawled for pretraining. Third, we over-sample from GPL-licensed repositories following the approach of SWE-Bench Pro [4]. Fourth, and most importantly, Type3_Latent issues require cross-file reasoning that cannot be resolved by retrieving a memorised patch : a model must genuinely trace the dependency relationship at inference time. We embed each PR using a sentence-level encoder and compute cosine similarity against known benchmark repositories [3]; PRs exceeding a similarity threshold of 0.85 are excluded. We cannot fully eliminate contamination risk, and recommend that future evaluations gate on PRs merged after the training cutoff of each evaluated model.

模型训练语料可能包含 GitHub 拉取请求，带来基准 PR 被部分背下而非真正推理的风险 [3, 4]。我们采用四项互补缓解。其一，所有 PR 来自截止基准冻结日的六个月窗口，相对被测模型训练截止尽量新。其二，RQS 污染分量惩罚高星仓库——它们更可能被预训练爬取。其三，按 SWE-Bench Pro [4] 做法，过采样 GPL 许可仓库。其四，也最重要：Type3_Latent 问题需要跨文件推理，无法靠检索一段背下的补丁解决——模型必须在推理时真正追踪依赖关系。我们用句级编码器嵌入每条 PR，对已知基准仓库算余弦相似度 [3]；超过 0.85 阈值的排除。污染风险无法完全消除，建议未来评测按各模型训练截止门控「合并时间更晚」的 PR。

3.6 Dataset Statistics

3.6 数据集统计

Table 3 shows the construction funnel from raw collection to the final released dataset.

表 3 给出从原始采集到最终发布数据集的构建漏斗。

Stage	Count	Retention
Raw PRs collected	$\sim$3,000	100%
After 10-stage hard filter	700	23%
After RVS $\geq$ 0.35	350	12%
Released (annotated)	350	—

Table 3: Dataset construction funnel.

阶段	数量	留存
原始采集 PR	$\sim$3,000	100%
十级硬过滤后	700	23%
RVS $\geq$ 0.35 后	350	12%
发布（已标注）	350	—

表 3：数据集构建漏斗。

Table 4 summarises the final SWE-PRBench dataset. The 350 pull requests are sourced from 65 of the 100 repositories that passed the RQS threshold; the remaining 35 repositories contributed no PRs meeting the RVS $\geq 0.35$ floor. Per-repository PR counts range from 1 to 30 (mean 5.4). The dataset is Python-dominant, reflecting the language distribution of high-RQS open-source repositories, with Python accounting for 242 PRs (69.1%), followed by JavaScript (37, 10.6%), Go (35, 10.0%), TypeScript (21, 6.0%), and Java (15, 4.3%). By PR type, feature PRs are the most common (214, 61.1%), followed by bug fixes (118, 33.7%), refactors (9), performance changes (8), and security fixes (1).

表 4 汇总最终 SWE-PRBench。350 条拉取请求来自通过 RQS 门槛的 100 个仓库中的 65 个；其余 35 个仓库没有贡献达到 RVS $\geq 0.35$ 下限的 PR。每仓 PR 数 1 到 30（均值 5.4）。数据集以 Python 为主，反映高 RQS 开源仓库的语言分布：Python 242 条（69.1%），其次 JavaScript（37，10.6%）、Go（35，10.0%）、TypeScript（21，6.0%）、Java（15，4.3%）。按 PR 类型，功能类最多（214，61.1%），其次缺陷修复（118，33.7%）、重构（9）、性能（8）、安全修复（1）。

The difficulty distribution reflects the classification method described in Section 3.4: 232 PRs (66.3%) are Type1_Direct, 75 (21.4%) are Type2_Contextual, and 43 (12.3%) are Type3_Latent_Candidate. The skew toward Type1 is expected : most review comments reference changed lines directly : and the 12.3% Type3 rate provides meaningful signal for testing cross-file reasoning capability. Table 5 shows the full cross-tabulation of difficulty by language.

难度分布按第 3.4 节分类方法：232 条（66.3%）为 Type1_Direct，75 条（21.4%）为 Type2_Contextual，43 条（12.3%）为 Type3_Latent_Candidate。偏向 Type1 在预期之中——多数审阅评论直接指向改动行——而 12.3% 的 Type3 率仍为跨文件推理能力提供有意义信号。表 5 给出难度×语言完整交叉表。

Pull request size ranges broadly. Lines added span 5 to 797 (mean 218.2, median 158), lines removed span 0 to 532 (mean 62.6, median 22), and files changed span 1 to 83 (mean 9.2, median 6). The upper bound of 797 added lines reflects the hard filter ceiling described in Section 3.2; the wide range ensures the benchmark covers both focused single-function patches and broader multi-file refactors. All PRs have an RVS score at or above the 0.35 threshold. The median RVS of the retained set is 0.52.

拉取请求规模跨度大。新增行 5 到 797（均值 218.2，中位数 158），删除行 0 到 532（均值 62.6，中位数 22），改动文件 1 到 83（均值 9.2，中位数 6）。新增行上限 797 对应第 3.2 节硬过滤天花板；宽范围保证基准既覆盖聚焦的单函数补丁，也覆盖更广的多文件重构。全部 PR 的 RVS 不低于 0.35。保留集 RVS 中位数为 0.52。

All raw pull requests, rejection logs, RQS scores, RVS breakdowns, and difficulty classifications are released publicly alongside the benchmark to support reproducibility and dataset extension.

全部原始拉取请求、拒绝日志、RQS 分数、RVS 分解与难度分类均随基准公开发布，以支持可复现与数据集扩展。

Property	Value
Total PRs	350
Candidate pool (before RVS cut)	700
Repositories (RQS-qualified)	100
Repositories (contributing PRs)	65
PRs per contributing repo (min/mean/max)	1 / 5.4 / 30
Language distribution
Python	242 (69.1%)
JavaScript	37 (10.6%)
Go	35 (10.0%)
TypeScript	21 (6.0%)
Java	15 (4.3%)
PR type distribution
Feature	214 (61.1%)
Bug fix	118 (33.7%)
Refactor	9 (2.6%)
Performance	8 (2.3%)
Security	1 (0.3%)
Difficulty distribution
Type1_Direct	232 (66.3%)
Type2_Contextual	75 (21.4%)
Type3_Latent	43 (12.3%)
PR size
Lines added (min/mean/median/max)	5 / 218.2 / 158 / 797
Lines removed (min/mean/median/max)	0 / 62.6 / 22 / 532
Files changed (min/mean/median/max)	1 / 9.2 / 6 / 83
RVS threshold	$\geq 0.35$
Median RVS (retained set)	0.52
Median RQS (retained repos)	74

Table 4: SWE-PRBench dataset statistics (350 PRs from 65 of 100 RQS-qualified repositories).

属性	值
PR 总数	350
候选池（RVS 截断前）	700
仓库（RQS 合格）	100
仓库（贡献 PR）	65
每贡献仓 PR 数（最小/均值/最大）	1 / 5.4 / 30
语言分布
Python	242 (69.1%)
JavaScript	37 (10.6%)
Go	35 (10.0%)
TypeScript	21 (6.0%)
Java	15 (4.3%)
PR 类型分布
功能	214 (61.1%)
缺陷修复	118 (33.7%)
重构	9 (2.6%)
性能	8 (2.3%)
安全	1 (0.3%)
难度分布
Type1_Direct	232 (66.3%)
Type2_Contextual	75 (21.4%)
Type3_Latent	43 (12.3%)
PR 规模
新增行（最小/均值/中位数/最大）	5 / 218.2 / 158 / 797
删除行（最小/均值/中位数/最大）	0 / 62.6 / 22 / 532
改动文件（最小/均值/中位数/最大）	1 / 9.2 / 6 / 83
RVS 阈值	$\geq 0.35$
RVS 中位数（保留集）	0.52
RQS 中位数（保留仓）	74

表 4：SWE-PRBench 数据集统计（350 条 PR，来自 100 个 RQS 合格仓库中的 65 个）。

Language	Type1	Type2	Type3	Total
Python	160	50	32	242
JavaScript	24	7	6	37
Go	24	8	3	35
TypeScript	13	7	1	21
Java	11	3	1	15
All	232	75	43	350

Table 5: Difficulty type by language.

语言	Type1	Type2	Type3	合计
Python	160	50	32	242
JavaScript	24	7	6	37
Go	24	8	3	35
TypeScript	13	7	1	21
Java	11	3	1	15
全部	232	75	43	350

表 5：难度类型按语言分布。

Each PR is stored as a JSON record containing metadata, scoring signals, reproducibility anchors (base and head commit SHAs), ground truth human review comments, and the raw unified diff patch. The full schema and a representative record are in Appendix A.

每条 PR 存为 JSON 记录，含元数据、打分信号、可复现锚点（基提交与头提交 SHA）、真值人类审阅评论，以及原始统一 diff 补丁。完整模式与一条代表性记录见附录 A。

4 Context Configurations

4 上下文配置

4.1 The V1 Problem

4.1 V1 问题

A naive collect-dump-truncate approach to context construction fails for three reasons: (1) token budgets are dominated by test files (60–70% on test-heavy PRs), leaving implementation files barely represented; (2) truncation at a fixed limit bisects diff hunks, producing syntactically incomplete fragments; (3) no explicit relationship between files is represented, so the agent cannot distinguish which unchanged file is related to which changed line. The V2 context builder addresses content selection through AST-based function extraction, import graph resolution, and behaviour mapping. As the results in Section 6 show, performance degrades monotonically even with V2, indicating that attention representation rather than content selection is the binding constraint for context-aware code review evaluation.

朴素的「采集—倾倒—截断」式上下文构造因三点失败：（1） token 预算被测试文件主导（测试重 PR 上占 60%–70%），实现文件几乎没空间；（2） 固定上限截断会切碎 diff hunk，得到句法不完整片段；（3） 文件间关系未显式表示，智能体分不清哪个未改文件与哪行改动相关。V2 上下文构建器通过基于 AST 的函数抽取、导入图解析与行为映射来解决内容筛选。如第 6 节结果所示，即便用上 V2，性能仍单调下降，说明对上下文敏感的代码审阅评测而言，约束在注意力表示，而不在内容筛选。

4.2 Three Fixed Configurations

4.2 三种固定配置

Three configurations are frozen at benchmark publication. Table 6 summarises them. config_A mirrors the information in a GitHub PR email notification. config_B mirrors the GitHub PR web view. config_C mirrors a reviewer working in a full IDE with the repository checked out. No configuration may be changed after results are reported, as they define the experimental conditions.

三种配置在基准发布时冻结。表 6 汇总。config_A 对应 GitHub PR 邮件通知里的信息。config_B 对应 GitHub PR 网页视图。config_C 对应审阅者在完整 IDE 中检出仓库工作。结果报告后不得改动任一配置，因为它们定义实验条件。

Config	Layers	Analogue	Budget
config_A	Task focus (L0), Key changes summary (L1), Diff (L2), Minimal metadata (L6)	GitHub PR email notification	2,000 tokens
config_B	+ Execution context (L3), Behaviour mapping (L4)	GitHub PR web view with file context	2,200 tokens
config_C	+ Test signatures (L5)	Reviewer with test suite access	2,500 tokens

Table 6: The three frozen context configurations. Budgets reflect actual measured token counts from the pipeline; the three configs differ in layer composition, not in token volume (range: 2,000–2,500 tokens).

配置	层	类比	预算
config_A	任务焦点（L0）、关键改动摘要（L1）、Diff（L2）、最少元数据（L6）	GitHub PR 邮件通知	2,000 tokens
config_B	+ 执行上下文（L3）、行为映射（L4）	带文件上下文的 GitHub PR 网页	2,200 tokens
config_C	+ 测试签名（L5）	能访问测试套件的审阅者	2,500 tokens

表 6：三种冻结上下文配置。预算来自管线实测 token 数；三种配置差在层组成，不在 token 体量（范围：2,000–2,500 tokens）。

4.3 V2 Improvements Over V1

4.3 V2 相对 V1 的改进

Four targeted improvements in V2 address the V1 failure modes.

V2 用四项针对性改进应对 V1 失败模式。

• • Key changes summary (200 tokens). An LLM-generated summary of principal changes inserted before the diff. This was the highest-return-on-investment change, measurably reducing hallucination rate even in config_A.
• • Behaviour mapping layer (100–150 tokens). For Go and TypeScript PRs with configuration-to-implementation propagation chains, V2 injects an explicit mapping of environment variable bindings and propagation paths, addressing the Type2/Type3 detection gap for non-Python languages.
• • Hard no-truncation rule. If adding the next file would require mid-structure truncation, that file is dropped entirely and its budget allocation rolls over. Every included fragment is syntactically complete.
• • Test noise reduction. Test files are included in body-stripped form: only def, assert, class, and @fixture lines are retained, targeting a ratio of 40% implementation diff, 30% execution context, 20% test signatures, and 10% metadata.

• • 关键改动摘要（200 tokens）。 在 diff 前插入 LLM 生成的主要改动摘要。这是投资回报最高的改动，即便在 config_A 也明显降低幻觉率。
• • 行为映射层（100–150 tokens）。 对存在「配置到实现」传播链的 Go 与 TypeScript PR，V2 注入环境变量绑定与传播路径的显式映射，补非 Python 语言上的 Type2/Type3 检出缺口。
• • 硬性不截断规则。 若加入下一文件会导致结构中途截断，则整文件丢弃并把预算滚入后续。凡纳入的片段句法完整。
• • 测试噪声削减。 测试文件以去正文形式纳入：只保留 def、assert、class 与 @fixture 行，目标配比为 40% 实现 diff、30% 执行上下文、20% 测试签名、10% 元数据。

Unlike V1’s collect-dump-truncate approach, V2 addresses the content selection problem. The persistence of the A$>$B$>$C pattern in V2 results therefore indicates that attention representation, not content selection, is the binding constraint. Full layer specification details are in Appendix B.

不同于 V1 的「采集—倾倒—截断」，V2 解决内容筛选问题。V2 结果里 A$>$B$>$C 模式仍在，因此说明注意力表示——而非内容筛选——才是约束。完整层规格见附录 B。

4.4 Reproducibility

4.4 可复现性

All pre-built contexts are released on HuggingFace as frozen artifacts at pipeline version v0.4.1 (https://huggingface.co/datasets/foundry-ai/swe-prbench). The was_truncated field is always false for V2 contexts by the no-truncation invariant.

全部预构建上下文以管线版本 v0.4.1 的冻结产物发布在 HuggingFace（https://huggingface.co/datasets/foundry-ai/swe-prbench）。因不截断不变量，V2 上下文的 was_truncated 字段恒为 false。

5 The SWE-PRBench Evaluation Protocol

5 SWE-PRBench 评测协议

The evaluation harness reads frozen context fixtures and makes independent agent and judge calls per record; the pipeline and all outputs are released at version v0.4.1. Agent and judge calls are fully decoupled: judge prompts can be updated and the full evaluation re-run without rebuilding contexts.

评测编排外壳读取冻结上下文夹具，对每条记录独立调用智能体与裁判；管线与全部输出以版本 v0.4.1 发布。智能体与裁判调用完全解耦：可更新裁判提示并重跑完整评测，而无需重建上下文。

5.1 Agent Setup

5.1 智能体设置

Each model receives a fixed system prompt ($\approx$200 tokens) instructing it to identify only issues traceable to a specific line in the provided diff or context, to avoid inventing behaviour not visible in the shown code, and to return output as a valid JSON array with severity labels P0 (production bug), P1 (fix before merge), or P2 (credible risk), targeting 4–6 issues per PR. We evaluate 8 frontier models: Claude Haiku 4.5, Claude Sonnet 4.6, GPT-4o, GPT-4o-mini, DeepSeek V3, Mistral Large 3, Mistral Small, and Llama 3.3 70B (via Groq). Models are called via provider APIs at temperature 0 with independent per-record requests. If JSON extraction fails after one retry, the record is hard-zeroed (overall_score = 0); judge parse fallbacks halve the score (val *= 0.5).

每个模型收到固定系统提示（约 200 tokens），要求只标出能追溯到所给 diff 或上下文中具体行的问题，避免编造所示代码中看不见的行为，并以合法 JSON 数组返回，严重度标为 P0（生产缺陷）、P1（合并前须修）或 P2（可信风险），每条 PR 目标 4–6 个问题。我们评测 8 个前沿模型：Claude Haiku 4.5、Claude Sonnet 4.6、GPT-4o、GPT-4o-mini、DeepSeek V3、Mistral Large 3、Mistral Small，以及 Llama 3.3 70B（经 Groq）。经提供商 API 调用，温度 0，每记录独立请求。若一次重试后仍无法抽取 JSON，该记录硬置零（overall_score = 0）；裁判解析回退则分数减半（val *= 0.5）。

5.2 Judge Design and Classification Rubric

5.2 裁判设计与分类量表

Agent comments are classified by a fixed judge model (GPT-5.2 in the final run; an earlier 20-PR validation used Claude Sonnet 4.6, with the same $\kappa{=}0.75$ result confirming judge consistency). The judge assigns each comment one of three labels: CONFIRMED (same underlying issue as a human comment, same file/area, same type of code change required); PLAUSIBLE (factually correct observation about visible code, but not in ground truth, since human reviews are not exhaustive [1]); or FABRICATED (references code not in context, or makes factually incorrect claims). PLAUSIBLE is not penalised as hallucination. Full criteria are published in RUBRIC.md alongside the dataset (Appendix C).

智能体评论由固定裁判模型分类（终跑用 GPT-5.2；更早的 20-PR 校验用 Claude Sonnet 4.6，同样得到 $\kappa{=}0.75$，确认裁判一致性）。裁判给每条评论三标签之一：CONFIRMED（与某条人类评论同一底层问题、同一文件/区域、需同类代码改动）；PLAUSIBLE（对可见代码的事实正确观察，但不在真值中——人类审阅并不穷尽 [1]）；或 FABRICATED（引用上下文没有的代码，或作事实错误断言）。PLAUSIBLE 不按幻觉惩罚。完整标准与数据集一并发布在 RUBRIC.md（附录 C）。

A two-step consistency constraint is enforced: if agent comment $X$ is CONFIRMED against human comment $H$, then $H$ is marked CAUGHT. We enforce 1-to-1 bipartite matching using all-mpnet-base-v2 sentence embeddings to prevent duplicate credit; pairs below affinity 0.30 are discarded. The 0.30 threshold was calibrated against the rubric validation set; values below 0.30 produced no judge-CONFIRMED pairs that also passed blind human review. Matching and affinity formula details are in Appendix C.

强制两步一致性：若智能体评论 $X$ 相对人类评论 $H$ 为 CONFIRMED，则 $H$ 标为 CAUGHT。用 all-mpnet-base-v2 句嵌入做一对一二分图匹配，防止重复计分；亲和度低于 0.30 的配对丢弃。0.30 阈值在量表校验集上标定；低于 0.30 时，没有任何裁判标为 CONFIRMED 且通过盲测人工复核的配对。匹配与亲和度公式细节见附录 C。

5.3 Scoring Formula

5.3 打分公式

The overall score for a single pull request is:

单条拉取请求的总分是：

$$s=\max\!\left(0,\,\min\!\left(1,\;0.40\cdot r+0.25\cdot p+0.15\cdot a+0.10\cdot q^{*}+0.05\cdot e-0.25\cdot h-0.15\cdot\rho-0.10\cdot\phi\right)\right) \tag{2}$$

where $r$ = recall, $p$ = precision, $a$ = semantic alignment of matched pairs, $q^{*}$ = actionability (discounted 80% when no comments are confirmed), $e$ = efficiency (confirmed / total agent comments), $h$ = hallucination rate (FABRICATED fraction), $\rho$ = redundancy penalty, and $\phi$ = excess plausible penalty above threshold $\tau{=}0.70$. The aggregate benchmark score is the difficulty-weighted mean: $S=\sum_{i}s_{i}w_{i}/\sum_{i}w_{i}$ where $w_{i}=\log(\text{human\_comments}_{i}+1)$.

其中 $r$ = 召回，$p$ = 精确率，$a$ = 匹配对语义对齐，$q^{*}$ = 可操作性（无确认评论时折扣 80%），$e$ = 效率（确认数 / 智能体评论总数），$h$ = 幻觉率（FABRICATED 占比），$\rho$ = 冗余惩罚，$\phi$ = 超过阈值 $\tau{=}0.70$ 的过量 plausible 惩罚。汇总基准分为难度加权均值：$S=\sum_{i}s_{i}w_{i}/\sum_{i}w_{i}$，其中 $w_{i}=\log(\text{human\_comments}_{i}+1)$。

5.4 Rubric Validation ($\kappa=0.75$)

5.4 量表校验（$\kappa=0.75$）

We validate that the judge operationalises the published rubric by sampling 30 judge classifications (10 per category, stratified) and applying the rubric independently using a blind template that omits the judge’s verdict. Cohen’s Kappa between judge labels and rubric-derived labels: $\kappa{=}0.75$ (substantial agreement, $N{=}30$, exact agreement 83.3%).

我们校验裁判是否落实已发布量表：分层抽样 30 条裁判分类（每类 10），用省略裁判裁决的盲模板独立套用量表。裁判标签与量表导出标签的 Cohen Kappa：$\kappa{=}0.75$（实质性一致，$N{=}30$，精确一致率 83.3%）。

5.5 Cross-Judge Validation

5.5 交叉裁判校验

To confirm the A$>$B$>$C ordering is not a judge artefact, we reran the judge step on a stratified 10-PR sample (batch_03, GPT-4o-mini agent outputs, $N{=}127$ comments) using Claude Sonnet 4.6. Overall inter-judge agreement: 78.7% ($\kappa{=}0.616$). Both judges independently produce the same A$>$B$>$C ordering (Table 7).

为确认 A$>$B$>$C 排序不是裁判伪迹，我们在分层 10-PR 样本上重跑裁判步（batch_03，GPT-4o-mini 智能体输出，$N{=}127$ 条评论），裁判换 Claude Sonnet 4.6。总体裁判间一致率：78.7%（$\kappa{=}0.616$）。两位裁判独立给出同一 A$>$B$>$C 排序（表 7）。

Config	$N$	Agreement	GPT-5.2 proxy	Sonnet proxy
config_A	46	78.3%	$-$0.035	$-$0.033
config_B	40	72.5%	$-$0.063	$-$0.045
config_C	41	85.4%	$-$0.066	$-$0.070
Overall	127	78.7% ($\kappa{=}0.616$)	direction: A$>$B$>$C confirmed

Table 7: Cross-judge agreement by context configuration ($N{=}127$ comments, 10-PR sample, GPT-4o-mini agent). Both judges reproduce A$>$B$>$C.

配置	$N$	一致率	GPT-5.2 代理	Sonnet 代理
config_A	46	78.3%	$-$0.035	$-$0.033
config_B	40	72.5%	$-$0.063	$-$0.045
config_C	41	85.4%	$-$0.066	$-$0.070
总体	127	78.7%（$\kappa{=}0.616$）	方向：A$>$B$>$C 确认

表 7：按上下文配置的交叉裁判一致率（$N{=}127$ 条评论，10-PR 样本，GPT-4o-mini 智能体）。两位裁判均复现 A$>$B$>$C。

The lowest agreement is at config_B (72.5%), consistent with the Type2_Contextual collapse: agent comments generated with file content are more specific and therefore harder to classify unambiguously. The highest agreement is at config_C (85.4%), where increased hallucination makes FABRICATED classifications more clear-cut.

一致率最低在 config_B（72.5%），与 Type2_Contextual 崩塌一致：带文件内容生成的智能体评论更具体，因而更难 unambiguous 分类。一致率最高在 config_C（85.4%），幻觉增多使 FABRICATED 分类更干脆。

6 Baseline Evaluation on SWE-PRBench

6 SWE-PRBench 基线评测

6.1 Experimental Setup

6.1 实验设置

We evaluate on a 100-PR stratified sample: Type1_Direct (40 PRs), Type2_Contextual (40), and Type3_Latent (20). Type2 and Type3 are deliberately over-sampled relative to their 21% and 13% dataset prevalence to ensure statistical power for per-difficulty analysis; per-difficulty scores are therefore reported separately and the weighted aggregate $\bar{s}$ uses the actual 100-PR counts.

我们在 100 条分层抽样 PR 上评测：Type1_Direct（40）、Type2_Contextual（40）、Type3_Latent（20）。相对数据集中约 21% 与 13% 的占比，刻意过采样 Type2 与 Type3，以保证按难度分析的统计效力；因此按难度分数单独报告，加权汇总 $\bar{s}$ 使用实际 100 条计数。

6.2 Leaderboard Results

6.2 排行榜结果

Table 8 establishes that SWE-PRBench is non-trivial: no model detects more than 31% of human-flagged issues on any configuration, and the mean detection rate across all 8 models on config_A is approximately 26%. These results quantify the gap between current AI code review capability and human expert performance. Figure 1 shows 95% bootstrap confidence intervals ($N{=}300$, $B{=}10{,}000$). The top four models (Haiku, Sonnet, DeepSeek, Mistral Large; $\bar{s}$ 0.147–0.153) are not statistically distinguishable, while the rank-4/rank-5 gap (0.147 vs. 0.113) is statistically significant.

表 8 表明 SWE-PRBench 非平凡：任一配置上没有模型检出超过 31% 的人工标出问题，config_A 上 8 模型平均检出率约 26%。这些结果量化了当前 AI 代码审阅能力与人类专家的差距。图 1 给出 95% bootstrap 置信区间（$N{=}300$，$B{=}10{,}000$）。前四名（Haiku、Sonnet、DeepSeek、Mistral Large；$\bar{s}$ 0.147–0.153）在统计上无法区分，而第 4/5 名差距（0.147 vs. 0.113）具统计显著性。

Model	$s_{A}$	$s_{B}$	$s_{C}$	$\bar{s}$	DRA	FPRavg
Claude Haiku 4.5*‡*	0.172	0.129	0.126	0.153	0.306	0.346
Claude Sonnet 4.6*†*	0.190	0.135	0.122	0.152	0.297	0.227
DeepSeek V3	0.181	0.132	0.118	0.150	0.312	0.315
Mistral Large 3	0.170	0.125	0.135	0.147	0.305	0.353
GPT-4o	0.134	0.110	0.090	0.113	0.220	0.193
GPT-4o-mini	0.110	0.095	0.093	0.108	0.210	0.353
Mistral Small	0.131	0.080	0.091	0.106	0.257	0.251
Llama 3.3 70B*§*	0.088	0.071	0.065	0.079	0.223	0.417
† 1 parse failure (hard-zeroed). ‡ 11 parse failures (hard-zeroed; removing them raises $\bar{s}$ by $\approx$0.006, no rank change).
§ 30 judge parse fallbacks (score $\times 0.5$).

Table 8: SWE-PRBench leaderboard: 100-PR sample, 8 models, sorted by mean difficulty-weighted composite $\bar{s}$ across all configurations. Parse failures in footnotes. Best per column in bold.

模型	$s_{A}$	$s_{B}$	$s_{C}$	$\bar{s}$	DRA	FPRavg
Claude Haiku 4.5*‡*	0.172	0.129	0.126	0.153	0.306	0.346
Claude Sonnet 4.6*†*	0.190	0.135	0.122	0.152	0.297	0.227
DeepSeek V3	0.181	0.132	0.118	0.150	0.312	0.315
Mistral Large 3	0.170	0.125	0.135	0.147	0.305	0.353
GPT-4o	0.134	0.110	0.090	0.113	0.220	0.193
GPT-4o-mini	0.110	0.095	0.093	0.108	0.210	0.353
Mistral Small	0.131	0.080	0.091	0.106	0.257	0.251
Llama 3.3 70B*§*	0.088	0.071	0.065	0.079	0.223	0.417
† 1 次解析失败（硬置零）。‡ 11 次解析失败（硬置零；去掉后 $\bar{s}$ 约升 $\approx$0.006，名次不变）。
§ 30 次裁判解析回退（分数 $\times 0.5$）。

表 8：SWE-PRBench 排行榜：100-PR 样本、8 模型，按各配置难度加权综合均值 $\bar{s}$ 排序。解析失败见脚注。每列最优加粗。

Figure 1: SWE-PRBench leaderboard with 95% bootstrap CIs. Top four models cluster ($\bar{s}$ 0.147–0.153); clear tier gap at rank 4/5.

图 1：带 95% bootstrap 置信区间的 SWE-PRBench 排行榜。前四名成簇（$\bar{s}$ 0.147–0.153）；第 4/5 名有清晰档差。

DeepSeek V3 achieves the highest raw detection on config_A (DR = 0.312) and outperforms all OpenAI models at approximately $0.28/M input tokens versus$2.50 for GPT-4o [21]: Tier 1 performance at roughly 9$\times$ lower cost. GPT-4o achieves the lowest hallucination rate (FPR = 0.193) but its detection rate lags all upper-tier competitors.

DeepSeek V3 在 config_A 上原始检出最高（DR = 0.312），并以约 $0.28/M 输入 token 相对 GPT-4o 的$2.50 [21] 超过全部 OpenAI 模型：约 9$\times$ 更低成本拿到第一档表现。GPT-4o 幻觉率最低（FPR = 0.193），但检出率落后所有上层竞争者。

6.3 Context Configuration Behaviour

6.3 上下文配置行为

Figure 2: Composite score by model and context configuration. All 8 models degrade A$\to$C; the A$\to$B drop is the largest step.

图 2：按模型与上下文配置的综合分。8 个模型均 A$\to$C 下降；A$\to$B 落差最大。

All 8 models degrade monotonically from config_A to config_C (Figure 2). Critically, the three configurations differ in layer composition, not in token volume: config_A and config_C differ by only 500 tokens (2,000 vs. 2,500). The A$>$B$>$C degradation therefore cannot be attributed to context length alone. Rather, it implicates the specific content added: execution context and behaviour mapping (config_B) and test signatures (config_C) each introduce additional signal that models cannot reliably integrate, even within a modest token budget. The V2 context builder uses AST-based function extraction and import graph resolution, ruling out content selection as the cause. The failure is in attention representation: once relevant context is provided as a flat token sequence alongside the diff, models cannot reliably direct attention to the changed lines. The Type2_Contextual collapse is the primary diagnostic: Sonnet drops from 0.22 to 0.10 on contextual issues when execution context is added at config_B, and DeepSeek drops from 0.20 to 0.10, consistent with the attention dilution documented by Liu et al. [18]. Table 9 shows the full difficulty-by-configuration breakdown.

8 个模型均从 config_A 到 config_C 单调下降（图 2）。关键在于：三种配置差在层组成，不在 token 体量——config_A 与 config_C 只差 500 tokens（2,000 vs. 2,500）。因此 A$>$B$>$C 退化不能单归因于上下文长度。更该指向具体新增内容：执行上下文与行为映射（config_B）以及测试签名（config_C）各自引入模型无法可靠整合的额外信号，即便 token 预算不大。V2 上下文构建器用基于 AST 的函数抽取与导入图解析，排除了内容筛选作为原因。失败在注意力表示：一旦相关上下文作为扁平 token 序列与 diff 并列，模型无法可靠把注意力导向改动行。Type2_Contextual 崩塌是主要诊断：Sonnet 在 config_B 加入执行上下文后，上下文类问题上从 0.22 掉到 0.10；DeepSeek 从 0.20 掉到 0.10，与 Liu 等人 [18] 记载的注意力稀释一致。表 9 给出完整的难度×配置分解。

Figure 3: Difficulty $\times$ config heatmap for three models. The Type2 collapse at config_B is visible across all three.

图 3：三个模型的难度 $\times$ 配置热图。config_B 上的 Type2 崩塌在三者上都可见。

Model	Difficulty	A	B	C	Trend
Claude Sonnet 4.6	Type1	0.18	0.19	0.14	stable
Type2	0.22	0.10	0.09	collapse
Type3	0.17	0.13	0.14	stable
DeepSeek V3	Type1	0.21	0.19	0.16	gradual
Type2	0.20	0.10	0.10	collapse
Type3	0.12	0.12	0.11	stable
GPT-4o-mini	Type1	0.15	0.13	0.15	stable
Type2	0.09	0.08	0.06	gradual
Type3	0.10	0.10	0.11	stable

Table 9: Composite score by difficulty and config for three representative models. Collapse: $>$50% drop from config_A to config_B.

模型	难度	A	B	C	趋势
Claude Sonnet 4.6	Type1	0.18	0.19	0.14	稳定
Type2	0.22	0.10	0.09	崩塌
Type3	0.17	0.13	0.14	稳定
DeepSeek V3	Type1	0.21	0.19	0.16	渐降
Type2	0.20	0.10	0.10	崩塌
Type3	0.12	0.12	0.11	稳定
GPT-4o-mini	Type1	0.15	0.13	0.15	稳定
Type2	0.09	0.08	0.06	渐降
Type3	0.10	0.10	0.11	稳定

表 9：三个代表模型按难度与配置的综合分。崩塌：从 config_A 到 config_B 下降 $>$50%。

6.4 Hallucination Profiles

6.4 幻觉画像

Figure 4: Mean hallucination rate (FPR) across config_A/B/C. GPT-4o is most precise (0.193); Llama fabricates at 0.417.

图 4：config_A/B/C 上的平均幻觉率（FPR）。GPT-4o 最精确（0.193）；Llama 编造率 0.417。

Figure 4 shows a clear precision-recall tradeoff: Haiku and DeepSeek achieve higher detection but at higher hallucination cost; Sonnet and GPT-4o produce more reliable comments with lower FPR. The composite scorer correctly captures this: Sonnet ranks second despite lower raw DR than Haiku, because the hallucination and redundancy penalties offset Haiku’s recall advantage. Batch ordering is stable across all 10 batches, confirming the ranking is not a sample artefact.

图 4 呈现清晰的精确率—召回权衡：Haiku 与 DeepSeek 检出更高，幻觉代价也更高；Sonnet 与 GPT-4o 评论更可靠、FPR 更低。综合打分器正确捕捉这一点：尽管原始 DR 低于 Haiku，Sonnet 仍排第二，因为幻觉与冗余惩罚抵消了 Haiku 的召回优势。10 个批次排序稳定，确认排名不是样本伪迹。

Practical deployment guidelines. (1) Use diff-only context (config_A): adding file content does not help and actively harms performance across all tested models. (2) Prioritise models by hallucination rate: GPT-4o (FPR = 0.193) and Sonnet (FPR = 0.227) produce more reliable comments than higher-recall alternatives, reducing triage burden. (3) DeepSeek V3 provides Tier 1 detection at roughly 9$\times$ lower API cost than GPT-4o with acceptable precision.

实用部署指引。（1） 用仅 diff 上下文（config_A）：加文件内容无益，且在全部被测模型上主动伤害表现。（2） 按幻觉率优先选模型：GPT-4o（FPR = 0.193）与 Sonnet（FPR = 0.227）比高召回替代更可靠，减轻分拣负担。（3） DeepSeek V3 以相对 GPT-4o 约 9$\times$ 更低 API 成本提供第一档检出，精确率可接受。

6.5 Benchmark Properties

6.5 基准性质

RQ	Question	Answer	Evidence
1	Detection rate and FPR	15–31% on config_A; FPR 0.193–0.417	Table 8
2	Context direction	All models degrade A$>$B$>$C	Figure 2
3	Uniformity	Universal direction; Type2 drives the A$\to$B drop	Table 9
4	Hard categories	Type2 collapses at config_B; Type3 near-zero	Figure 3

Table 10: Summary of benchmark properties and research question answers.

RQ	问题	答案	证据
1	检出率与 FPR	config_A 上 15%–31%；FPR 0.193–0.417	表 8
2	上下文方向	全部模型 A$>$B$>$C 下降	图 2
3	均匀性	方向普遍；Type2 驱动 A$\to$B 落差	表 9
4	困难类别	Type2 在 config_B 崩塌；Type3 近零	图 3

表 10：基准性质与研究问题答案摘要。

7 Dataset Analysis

7 数据集分析

Table 5 shows the full difficulty-by-language distribution. Python has the highest absolute Type3 count (32 of 43, 74.4%), consistent with Python’s dynamic import patterns. Go and TypeScript have the highest proportional Type2 rates (22.9% and 33.3%), consistent with interface contracts and type propagation creating contextual issues. PR size spans 5–797 added lines (mean 218.2) and 1–83 files changed (mean 9.2), covering both focused patches and broad refactors.

表 5 给出完整难度×语言分布。Python 的 Type3 绝对数最高（43 中的 32，74.4%），与 Python 动态导入模式一致。Go 与 TypeScript 的 Type2 比例最高（22.9% 与 33.3%），与接口契约与类型传播易产生上下文问题一致。PR 规模跨 5–797 新增行（均值 218.2）与 1–83 改动文件（均值 9.2），既覆盖聚焦补丁也覆盖广重构。

A SWE-PRBench task is hard in three structurally distinct ways: Type1 tasks require identifying a fault in changed lines without hallucinating non-existent bugs; Type2 tasks require understanding changed code in relation to surrounding context, but the current flat-token representation makes this harder with more context; Type3 tasks require cross-file dependency reasoning from import structure alone. The AI ceiling is clear: the best model (Haiku, DRA = 0.306) detects roughly one in three direct issues. If a second human reviewer achieves 50–70% recall [1], the gap to human performance is approximately 20–40 percentage points.

SWE-PRBench 任务在三种结构上不同地难：Type1 要在改动行上找故障且不编造不存在的 bug；Type2 要理解改动代码与周围上下文的关系，但当前扁平 token 表示使「更多上下文」反而更难；Type3 要从导入结构 alone 做跨文件依赖推理。AI 天花板很清楚：最佳模型（Haiku，DRA = 0.306）大约检出三分之一的直接问题。若第二位人类审阅者召回在 50%–70% [1]，与人类表现的差距约 20–40 个百分点。

8 Limitations

8 局限

Scope of the context degradation finding. The V2 context builder uses AST extraction and import graph resolution, not raw file dumping. The degradation therefore implicates attention representation rather than content selection. Context provision strategies encoding relevance at the token level, such as explicit changed-vs-unchanged boundary markers or interleaved annotations, may produce different outcomes and are a direct priority for future work.

上下文退化发现的范围。 V2 上下文构建器用 AST 抽取与导入图解析，而非原始文件倾倒。因此退化指向注意力表示，而非内容筛选。在 token 级编码相关性的上下文供给策略——例如显式的改动/未改动边界标记或交错标注——可能得到不同结果，是未来工作的直接优先项。

Language and sample scope. The dataset is Python-dominant (69.1%) and evaluated on a 100-PR sample from the 350-PR full corpus. Results for non-Python languages and per-difficulty sub-group statistics should be interpreted with this in mind. Full 350-PR evaluation is deferred to the camera-ready version.

语言与样本范围。 数据集以 Python 为主（69.1%），并在 350 条全库中的 100 条样本上评测。非 Python 语言结果与按难度子组统计应据此解读。完整 350 条评测留到 camera-ready 版本。

Evaluation methodology. The primary judge is GPT-5.2. Cross-judge validation confirms the A$>$B$>$C direction under Sonnet 4.6 ($\kappa{=}0.616$), but absolute score values differ between judges. The potential for mild judge-family bias cannot be excluded without cross-family validation on the full leaderboard. No human performance baseline is available; the 20–40 percentage-point gap estimate to human recall is derived from prior literature [1].

评测方法。 主裁判是 GPT-5.2。交叉裁判校验确认 Sonnet 4.6 下 A$>$B$>$C 方向（$\kappa{=}0.616$），但绝对分值因裁判而异。在未对完整排行榜做跨家族校验前，不能排除轻微裁判家族偏置。没有人类表现基线；相对人类召回约 20–40 个百分点差距来自先前文献 [1]。

Contamination. Despite recency filtering, RQS contamination weighting, and GPL preference, we cannot fully exclude the possibility that some benchmark PRs appeared in model pretraining. Type3 issues require cross-file reasoning that cannot be recalled from a memorised patch, providing partial contamination resistance for the hardest difficulty tier.

污染。 尽管有时效过滤、RQS 污染加权与 GPL 偏好，仍无法完全排除部分基准 PR 出现在模型预训练中。Type3 问题需要跨文件推理，无法从背下的补丁召回，为最难档提供部分抗污染性。

9 Conclusion

9 结论

We introduced SWE-PRBench, a benchmark of 350 pull requests with human-annotated ground truth for evaluating AI code review quality. The dataset, pre-built context artifacts and evaluation harness are released publicly:

我们提出 SWE-PRBench：含 350 条拉取请求、带人工标注真值的基准，用于评估 AI 代码审阅质量。数据集、预构建上下文产物与评测编排外壳均已公开发布：

• • Dataset: https://huggingface.co/datasets/foundry-ai/swe-prbench
• • Evaluation harness: https://github.com/FoundryHQ-AI/swe-prbench

• • 数据集： https://huggingface.co/datasets/foundry-ai/swe-prbench
• • 评测编排外壳： https://github.com/FoundryHQ-AI/swe-prbench

Baseline results on 8 frontier models establish that no model detects more than 31% of human-flagged issues on any configuration, confirming that AI code review remains far from human expert performance. The context configuration ablation reveals that even structured semantic context — built via AST extraction, import graph resolution, and behaviour mapping — degrades performance relative to diff-only prompts. This implicates attention representation rather than content selection as the binding constraint: models cannot reliably distinguish changed from unchanged lines once both appear in a flat token sequence. We invite the community to submit results to the SWE-PRBench leaderboard, extend the dataset to additional languages and domains, and develop context representation strategies — such as explicit changed-vs-unchanged boundary markers or retrieval-augmented context — that close the gap between current AI performance and human expert review quality.

8 个前沿模型上的基线表明，任一配置上没有模型检出超过 31% 的人工标出问题，确认 AI 代码审阅远未达到人类专家水平。上下文配置消融显示：即便是经 AST 抽取、导入图解析与行为映射构建的结构化语义上下文，相对仅 diff 提示也会拉低表现。这说明约束在注意力表示而非内容筛选：一旦改动与未改动行同现于扁平 token 序列，模型无法可靠区分二者。我们邀请社区向 SWE-PRBench 排行榜提交结果，把数据集扩展到更多语言与领域，并开发上下文表示策略——例如显式改动/未改动边界标记或检索增强上下文——以缩小当前 AI 表现与人类专家审阅质量的差距。

Reproducibility Note. Scores reported in this paper reflect pipeline version v0.4.1 with GPT-5.2 as judge at temperature $=0$. Frontier model APIs do not guarantee full determinism at temperature $=0$, so minor score variation across independent runs is expected. The two-tier ranking structure and A$>$B$>$C ordering are stable across runs and confirmed by cross-judge validation ($\kappa=0.616$, Section 5.5).

可复现说明。 本文分数对应管线版本 v0.4.1，裁判为 GPT-5.2，温度 $=0$。前沿模型 API 在温度 $=0$ 下也不保证完全确定性，独立重跑可有轻微分数波动。两档排名结构与 A$>$B$>$C 排序跨次稳定，并经交叉裁判校验确认（$\kappa=0.616$，第 5.5 节）。

References

参考文献

[1] A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in Proc. ICSE , 2013, pp. 712–721.

[2] J. Corbet, “Kernel development: Who does it, how they do it, and what it means,” LWN.net , Jan. 2016. [Online]. Available: https://lwn.net/Articles/671120/

[3] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” in Proc. ICLR , 2024.

[4] Scale AI, “SWE-Bench Pro: A harder benchmark for software engineering agents,” Technical Report, Scale AI, 2025.

[5] Scale AI, “SWE Atlas: Codebase question answering for software engineering,” Technical Report, Scale AI, 2026.

[6] M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Using pre-trained models to boost code review automation,” in Proc. ICSE , 2022, pp. 1–12.

[7] Z. Li, S. Lu, S. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu, and N. Sundaresan, “Automating code review activities by large-scale pre-training,” in Proc. ESEC/FSE , 2022, pp. 1035–1047.

[8] R. Tufano, S. Gualandi, M. Aghajani, M. Monperrus, and A. Bacchelli, “Code review automation: Strengths and weaknesses of the state of the art,” IEEE Transactions on Software Engineering , vol. 50, no. 5, 2024.

[9] O. Kononenko, O. Baysal, and M. W. Godfrey, “Code review quality: How developers see it,” in Proc. ICSE , 2016, pp. 1028–1038.

[10] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-Bench and chatbot arena,” in Proc. NeurIPS , 2023.

[11] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-Eval: NLG evaluation using GPT-4 with better human alignment,” in Proc. EMNLP , 2023, pp. 2511–2522.

[12] J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement , vol. 20, no. 1, pp. 37–46, 1960.

[13] J. Lu, D. Li, B. Yu, and L. Zhang, “DeepCRCEval: Revisiting the evaluation of code review comment generation,” arXiv preprint arXiv:2412.18291 , 2025.

[14] M. Golzadeh, A. Decan, D. Legay, and T. Mens, “A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments,” Journal of Systems and Software , vol. 175, 2021.

[15] A. Bosu, M. Greiler, and C. Bird, “Characteristics of useful code reviews: An empirical study at Microsoft,” in Proc. MSR , 2015, pp. 146–156.

[16] Y. Zhang, Y. Zhang, Z. Sun, Y. Jiang, and H. Liu, “LAURA: Enhancing code review generation with context-enriched retrieval-augmented LLM,” arXiv preprint arXiv:2512.01356 , 2024.

[17] R. Tan, J. Fu, T. Pang, L. Du, M. Lin, and C. Gao, “JudgeBench: A benchmark for evaluating LLM-based judges,” arXiv preprint arXiv:2410.12784 , 2024.

[18] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics , vol. 12, pp. 157–173, 2024.

[19] GitHub, “GitHub GraphQL API,” 2024. [Online]. Available: https://docs.github.com/en/graphql

[20] GitHub, “GitHub REST API: Comparing commits,” 2024. [Online]. Available: https://docs.github.com/en/rest/commits/commits

[21] OpenAI, “OpenAI API pricing,” 2024. [Online]. Available: https://openai.com/api/pricing/

Appendix A PR Record Schema

附录 A PR 记录模式

Each SWE-PRBench pull request is stored as a JSON record with five field categories. Metadata: task_id, repo, pr_number, language, pr_type, difficulty, merged_at. Scoring signals: rvs_score and five component scores (review_depth, code_complexity, discussion_signal, test_change_signal, bug_fix_signal), plus lines_added, lines_removed, files_changed. Reproducibility anchors: base_commit and head_commit SHA hashes pinning the exact file state at review time. Ground truth: human_review_comments array with per-comment fields for author, body, path, line, diff hunk, and reply status; plus num_substantive_comments, num_unique_reviewers, has_requested_changes, ai_comments_removed. Diff: diff_patch as the raw unified diff retrieved from the GitHub REST API; agent_input is populated by the context builder at evaluation time.

每条 SWE-PRBench 拉取请求存为含五类字段的 JSON 记录。元数据： task_id、repo、pr_number、language、pr_type、difficulty、merged_at。打分信号： rvs_score 与五个分量分（review_depth、code_complexity、discussion_signal、test_change_signal、bug_fix_signal），以及 lines_added、lines_removed、files_changed。可复现锚点： base_commit 与 head_commit SHA，钉住审阅时的精确文件状态。真值： human_review_comments 数组，含作者、正文、路径、行、diff hunk、回复状态等逐条字段；另有 num_substantive_comments、num_unique_reviewers、has_requested_changes、ai_comments_removed。Diff： diff_patch 为 GitHub REST API 取回的原始统一 diff；agent_input 由上下文构建器在评测时填充。

A representative record (dask/dask PR #12221) is available in the HuggingFace dataset card. This PR carries RVS = 0.334, placing it below the 0.35 inclusion threshold; it is provided as a boundary illustration of the quality cut, not as a dataset member.

HuggingFace 数据集卡提供一条代表性记录（dask/dask PR #12221）。该 PR 的 RVS = 0.334，低于 0.35 纳入阈值；用作质量截断的边界示例，而非数据集成员。

Appendix B Context Builder Layer Specification

附录 B 上下文构建器层规格

The V2 context builder assembles each configuration through a layered pipeline with budget rollover (unused tokens flow to subsequent layers). Layer 1a (metadata, 200 tokens fixed): PR title, repository, difficulty type, language, file count, truncated description; never includes RVS score, ground truth, or post-merge data. Layer 1b (key changes summary, 200 tokens): LLM-generated natural-language summary of principal changes inserted before the diff. Layer 2 (diff, 800–1,200 tokens): per-file hunks sorted by importance $=0.5\times\text{lines\_changed}+0.3\times\text{comment\_density}+0.2\times\text{function\_changes}$; production files before test files; at most 4 files. Layer 3 (changed files, 400–800 tokens): AST extraction for Python (get_innermost_enclosing_nodes plus relevant imports), cluster-based line windows for other languages; always uses base_commit, never head_commit; excludes test files. Layer 4a (related files, 100–300 tokens): import graph resolution for Python; GitHub code search for others; empty with not_applicable_type1 flag for Type1 PRs. Layer 4b (test files, 150–300 tokens): body-stripped, retaining def, async def, class, assert, and @fixture lines only; Go/TypeScript PRs also receive a behaviour mapping layer (100–150 tokens) making config-to-implementation propagation chains explicit.

V2 上下文构建器经分层管线组装各配置，并带预算滚动（未用 token 流入后续层）。第 1a 层（元数据，固定 200 tokens）：PR 标题、仓库、难度类型、语言、文件数、截断描述；从不包含 RVS 分、真值或合并后数据。第 1b 层（关键改动摘要，200 tokens）：LLM 生成的主要改动自然语言摘要，插在 diff 前。第 2 层（diff，800–1,200 tokens）：按文件 hunk，按重要性 $=0.5\times\text{lines\_changed}+0.3\times\text{comment\_density}+0.2\times\text{function\_changes}$ 排序；生产文件先于测试文件；最多 4 个文件。第 3 层（改动文件，400–800 tokens）：Python 用 AST 抽取（get_innermost_enclosing_nodes 加相关导入），其他语言用基于簇的行窗口；始终用 base_commit，从不用 head_commit；排除测试文件。第 4a 层（相关文件，100–300 tokens）：Python 用导入图解析，其他用 GitHub 代码搜索；Type1 PR 为空并带 not_applicable_type1 标志。第 4b 层（测试文件，150–300 tokens）：去正文，只保留 def、async def、class、assert 与 @fixture 行；Go/TypeScript PR 另收行为映射层（100–150 tokens），显式给出配置到实现的传播链。

Appendix C Evaluation Protocol Detail

附录 C 评测协议细节

CONFIRMED criteria (all must hold): (1) the agent identifies a specific issue in the code; (2) a human reviewer identified the same underlying issue; (3) the issue concerns the same file or functional area; (4) the agent concern would lead to the same kind of code change as the human concern. PLAUSIBLE: grounded in visible code, factually correct, reasonable engineering concern, but not matched to any human comment. FABRICATED (any sufficient): references code not in context; makes factually incorrect claims; describes a non-existent bug; invents method signatures or variable names.

CONFIRMED 标准（须全部成立）：（1）智能体指出代码中的具体问题；（2）人类审阅者指出同一底层问题；（3）问题关涉同一文件或功能区域；（4）智能体关切会导致与人类关切同类的代码改动。PLAUSIBLE： 立足可见代码、事实正确、属合理工程关切，但未匹配到任何人类评论。FABRICATED（任一即足）： 引用上下文没有的代码；作事实错误断言；描述不存在的 bug；编造方法签名或变量名。

Matching detail. Similarity uses all-mpnet-base-v2 sentence embeddings. Stage 1: judge-CONFIRMED pairs are sorted by combined actionability-and-similarity signal; the top-scoring 1-to-1 assignment is selected greedily. Stage 2: unmatched non-FABRICATED comments are evaluated against remaining human comments via pairwise affinity $=\text{cosine}(e_{a},e_{h})+0.15\cdot\mathbf{1}[\text{same file}]+\delta_{\text{line}}$, clamped to $[0,1]$. Pairs below 0.30 are discarded. The threshold was calibrated against the 30-sample rubric validation set; values below 0.30 produced no judge-CONFIRMED pairs that also passed blind human review. Agent comments unmatched after both stages and not FABRICATED are reclassified to PLAUSIBLE.

匹配细节。 相似度用 all-mpnet-base-v2 句嵌入。阶段 1：裁判标为 CONFIRMED 的配对按可操作性与相似度的组合信号排序，贪心取最高分一对一赋值。阶段 2：未匹配的非 FABRICATED 评论相对剩余人类评论算成对亲和度 $=\text{cosine}(e_{a},e_{h})+0.15\cdot\mathbf{1}[\text{same file}]+\delta_{\text{line}}$，钳到 $[0,1]$。低于 0.30 的配对丢弃。阈值在 30 样本量表校验集上标定；低于 0.30 时，没有任何裁判标为 CONFIRMED 且通过盲测人工复核的配对。两阶段后仍未匹配且非 FABRICATED 的智能体评论改标为 PLAUSIBLE。