{weijun.luo, kelvin.luu, mohamed.elfeki}@scale.com

Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

维护面向能力日益增强智能体的基准：未应得通过的检测与修复（Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes）

Weijun Luo Affiliation:  Scale AI    Kelvin Luu Affiliation:  Scale AI    Xinyi Liu Affiliation:  Scale AI    Guangze Luo Affiliation:  Scale AI    Miguel Romero Calvo Affiliation:  Scale AI    Soham Dan Affiliation:  Scale AI    Daniel Yue Zhang Affiliation:  Scale AI    Ying Liu Affiliation:  Scale AI    Mohamed Elfeki Affiliation:  Scale AI

Abstract

摘要

Agentic benchmarks guide model selection and training. Yet an agent can pass a task without demonstrating the intended capability. Such outcomes constitute unearned passes, their proportion among all passes defines the integrity gap. As agents improve, benchmark surfaces that once seemed harmless can become exploitable, making benchmark validity an ongoing maintenance problem. We introduce a process-verification framework that audits passing trajectories, distinguishes evidenced reward hacking from verifier weakness, and localizes exploitable surfaces for repair. Across 3,810 passing trajectories from 29 model-benchmark cohorts, confirmed violations often increase with model generation but not monotonically. On SWEBench Pro V1.0, confirmed violation rates rise from 24% to 73% between Opus 4.7 and Fable 5 on matched tasks; later cohorts fall to 11% for Fable 5.1 and 0% for GPT-6 Astra. These comparisons are descriptive: configurations were not normalized, and the latest models also pass fewer exploitable tasks. Violations concentrate around a small set of recurring surfaces, especially unintended access to reference solutions through git history. Three repair case studies across two benchmarks show why blocking a recorded exploit is insufficient: the same protected information can remain accessible through another route. Therefore, we combine minimal patches with exploit replay and fresh agent evaluation, auditing new passes under the original standard. No evaluated attempt against the final patches reached the protected channel, and every post-patch pass was judged legitimate. Benchmark integrity requires ongoing maintenance: audit passing behavior, repair the enabling surface, and re-evaluate both exploit access and legitimate solvability.

智能体基准指导模型选型与训练。但智能体可以在未展示预期能力的情况下通过任务。这类结果称为未应得通过（unearned passes），其在全部通过中的占比定义为完整性缺口（integrity gap）。随着智能体变强，一度无害的基准表面可能变得可利用，使基准有效性变成持续维护问题。我们提出过程核验框架：审计通过轨迹，区分有证据的奖励黑客与核验器弱点，并定位可利用表面以便修复。在来自 29 个模型—基准队列的 3,810 条通过轨迹上，已确认违规往往随模型代际上升，但并非单调。在 SWEBench Pro V1.0 上，匹配任务的确认违规率从 Opus 4.7 的 24% 升至 Fable 5 的 73%；更晚队列降至 Fable 5.1 的 11% 与 GPT-6 Astra 的 0%。这些比较是描述性的：配置未归一化，且最新模型通过的可利用任务也更少。违规集中在少数反复出现的表面，尤其是经 git 历史意外接触参考解。跨两个基准的三项修复案例表明：仅封堵已记录利用路径不够——同一受保护信息仍可能经另一路由可达。因此我们将最小补丁与利用重放、新鲜智能体评测结合，按原标准审计新的通过。对最终补丁的评测中，没有任何尝试触达受保护通道，且补丁后每一次通过均被判为正当。基准完整性需要持续维护：审计通过行为、修复使能表面，并同时再评测利用可达性与正当可解性。

Surface-pass

trajectory

commands, tools, results

Taxonomy

pre-scan

evidence flags, no LLM

Two judges

primary + adversarial

Causal gate

effective, grade-relevant

Escalation

cross-model, trust check

Ternary label

unearned-violation / unearned / legitimate

Channel map

corroborated + fault side

Control surface

channel vs. its addresses

Minimal seal

smallest sufficient edit

Replay probe

rebuilt task, A/B

Live pass@$k$

re-measure, same judges

Figure 1: Detection, localization, attribution, and remediation workflow. Passing trajectories are classified as legitimate, unearned, and unearned violations. A violation requires evidence of deliberate, grade-relevant exploitation; an unearned pass alone does not. Confirmed violations are mapped to the surface that enabled it. The remediation separates the identified channel from the addresses used to reach it, applies the smallest deterministic edit at the effective control surface, and verifies closure by rebuilding the task and replaying its recorded exploit while varying only that control. The patched task is then rerun live at pass@$k$ and re-judged with the same pipeline, returning to detection as ordinary input (dashed). The loop is closed by the dashed return: a patched task is accepted only after being judged again under the standard that defined the original violation. Gray: deterministic; Green: large language-model (LLM) judges; blue: remediation.

表面通过 → 轨迹（命令、工具、结果）→ 分类体系预扫描（证据标记，无 LLM）→ 双裁判（主裁判 + 对抗）→ 因果门控（有效、与评分相关）→ 升级（跨模型信任检查）→ 三元标签（未应得违规 / 未应得 / 正当）→ 通道映射（佐证 + 过错侧）→ 控制表面（通道 vs 其地址）→ 最小封堵（最小充分编辑）→ 重放探针（重建任务，A/B）→ 在线 pass@$k$（重测，同一裁判）。

图 1：检测、定位、归因与修复工作流。通过轨迹被标为正当、未应得、未应得违规。违规需要故意且与评分相关的利用证据；仅有未应得通过不足。已确认违规映射到使其可能的表面。修复将已识别通道与到达它的地址分离，在有效控制表面施加最小确定性编辑，并通过重建任务、仅变动该控制来重放已记录利用以验证封闭。补丁后任务再以 pass@$k$ 在线重跑，用同一管线再裁判，作为普通输入回到检测（虚线）。虚线回路闭合：补丁任务须在定义原违规的同一标准下再被裁判后才被接受。灰：确定性；绿：大语言模型裁判；蓝：修复。

1 Introduction

1 引言

Agentic benchmarks increasingly identify more capable models and shape behaviors during training. Their value depends on whether a passing score reflects successful completion of the intended task. Most benchmark verifiers evaluate observable outcomes, such as whether expected files exist, tests pass, or specified conditions are satisfied. However, they do not reflect whether the agent reached that outcome through an admissible process. This distinction creates an important failure mode for agentic systems: an agent can satisfy the verifier while bypassing the intended task, such as by accessing unintended oracle information, exploiting evaluation infrastructure, or tailoring an output to weaknesses in the grader. A verifier may also accept work that is simply wrong or incomplete. We refer to such seemingly successful outcomes as unearned passes. When these outcomes influence leaderboards, model selection, or reinforcement-learning rewards, benchmark scores may reward exploitation of the evaluation rather than the capability the evaluation was intended to measure [2, 3, 8, 4]. Whether agentic benchmark scores measure the capabilities they are intended to measure has become an active concern [14, 10, 6], with reward hacking identified as one important mechanism [1].

智能体基准日益用于识别更强模型，并在训练中塑造行为。其价值取决于通过分数是否反映预期任务的成功完成。多数基准核验器评估可观察结果，例如期望文件是否存在、测试是否通过或指定条件是否满足，却不反映智能体是否经可接受过程到达该结果。这为智能体系统制造了重要失败模式：智能体可以满足核验器却绕过预期任务，例如接触意外预言机信息、利用评测基础设施，或针对打分器弱点定制输出。核验器也可能接受错误或不完整的工作。我们把这类看似成功的结果称为未应得通过。当它们影响榜单、选型或强化学习奖励时，基准分数可能奖励对评测的利用，而非评测本欲测量的能力 [2, 3, 8, 4]。智能体基准分数是否测到预期能力已成为活跃关切 [14, 10, 6]，奖励黑客是重要机制之一 [1]。

Outcome verification alone cannot reliably distinguish these cases. First, the programmatic verifiers are necessarily limited to the conditions they encode and can therefore miss shortcuts they were not designed to detect. Second, language-model graders broaden the range of behaviors that can be evaluated, but they introduce additional limitations, including inconsistency and a tendency to assign unearned credit [7]. More fundamentally, increasingly capable agents may discover exploitable opportunities that were inconsequential when a benchmark was created. Benchmark validity is therefore not necessarily a fixed property. As agents become more capable, previously benign benchmark surfaces may become exploitable. Detecting unearned passes is consequently only part of the problem. A trustworthy evaluation system must also identify the surface that enabled the exploit for subsequent remediation. From this perspective, benchmark integrity becomes an ongoing maintenance problem.

仅靠结果核验无法可靠区分这些情况。其一，程序化核验器必然限于其编码的条件，因此会漏掉未设计去检测的捷径。其二，语言模型打分器拓宽可评行为范围，但也引入不一致、给予未应得信用等局限 [7]。更根本的是，能力更强的智能体可能发现基准创建时无关紧要的可利用机会。基准有效性因此不一定是固定属性。随着智能体变强，先前无害的表面可能变得可利用。检测未应得通过只是问题的一部分；可信评测系统还必须识别使利用成为可能的表面以便后续修复。由此视角，基准完整性变成持续维护问题。

There are three directions of work addressing this problem. The first audits recorded trajectories post hoc. [29] report 30.4% reward hacking on RE-Bench versus 0.7% on HCAST for o3, suggesting scoring-function visibility as one possible contributor to the gap; Cursor audited 731 Opus-4.8-Max trajectories with an auditor blinded to the verifier outcome, finding 57% upstream lookup and 9% git-history mining [8]. Cursor also seals git and network access and measures the effect through benchmark score changes, whereas we re-audit post-patch trajectories and distinguish closure of a recorded route from closure of the underlying information channel. The Reward-Hacking Benchmark pairs a taxonomy with an automated classifier  [17], and AgentProcessBench likewise evaluates step-level process quality rather than relying only on final outcomes [12]. AgentLens studies “lucky passes,” where passing trajectories exhibit weak or inefficient processes [33]; these are not equivalent to our UNEARNED category in this work, although lucky passes caused by insufficient verification or incomplete implementations may also qualify as unearned under our framework. Delphik’s Coding Index audit similarly analyzes recorded trajectories and distinguishes exploit attempts from load-bearing cases [34], and organizes failures by exploit channel and corresponding defense. The additional step in our work is to carry confirmed failures into a closed remediation loop: localize the effective control surface, apply a seal, replay the recorded exploit, and then re-evaluate fresh trajectories.

应对该问题有三个方向。其一事后审计已记录轨迹。[29] 报告 o3 在 RE-Bench 上奖励黑客 30.4%、在 HCAST 上仅 0.7%，提示评分函数可见性可能是差距之一；Cursor 对 731 条 Opus-4.8-Max 轨迹做对核验结果盲审的审计，发现 57% 上游查找与 9% git 历史挖掘 [8]。Cursor 还封堵 git 与网络并经基准分数变化测效果，而我们再审计补丁后轨迹，并区分“封闭已记录路由”与“封闭底层信息通道”。Reward-Hacking Benchmark 将分类体系与自动分类器配对 [17]，AgentProcessBench 同样评估步骤级过程质量而非仅最终结果 [12]。AgentLens 研究“幸运通过”——通过轨迹过程弱或低效 [33]；这与本文 UNEARNED 不完全等价，尽管因核验不足或不完整实现导致的幸运通过也可能落入未应得。Delphik 的 Coding Index 审计同样分析轨迹并区分利用尝试与承重案例 [34]，按利用通道与对应防御组织失败。我们的额外一步是把已确认失败带入闭环修复：定位有效控制表面、施加封堵、重放已记录利用，再评测新鲜轨迹。

These studies motivate our focus on linking trajectory-level judgments to targeted repairs and validating those repairs through fresh evaluation. The second direction builds tasks whose design makes exploitation measurable: ImpossibleBench constructs task variants in which the intended specification and tests cannot both be satisfied, so that a pass necessarily exploits the evaluation [22]; SpecBench measures the gap between visible and held-out suites [21]; EvilGenie compares reward-hacking detection strategies in programming tasks [13]; and hack-verifiable environments instrument the environment so that exploitation is decidable [15]. These yield cleaner ground truth but require evaluation settings designed to make exploitation identifiable and so do not directly address heterogeneous benchmarks already producing published numbers. The third direction audits benchmarks before exploitation appears: BenchJack argues that post-hoc monitoring is inherently reactive because a flaw may remain undiscovered until an agent exploits it, and scored near-perfect on nine of ten benchmarks without solving the intended tasks [19]; BenchGuard automates benchmark auditing for inconsistencies and defects [18], and [23] give construction-time guidance. Scanning enumerates what a benchmark permits; trajectory auditing establishes which opportunities deployed models actually used and therefore which surfaces affected realized scores. The hacker-fixer loop [28] audits 1,968 tasks across five benchmarks, finds 16% hackable from the task description alone, and uses a fixer to reject discovered exploits while a solver confirms that legitimate solutions still pass, driving KernelBench attack success from 62% to 0%. Hacker-fixer generates adversarial exploits and validates hardened verifiers by reducing attack success while preserving legitimate solver performance [28]. Recent work of MiMo-V2.6 deploys a dedicated hack agent to probe prepared RL environments for residual leaks: it checks known exploit routes and searches for previously unseen ones and iteratively refining cleanup and access restrictions [35]. Similarly, DeepSeek’s DSec reports production RL agents seeking answers through unintended environment paths [36]; they mitigated these behaviors with file/socket access controls and task-specific network policies and emphasized continuous observability and hardening as models evolve. Recent experience porting Agents’ Last Exam’s Linux CLI tasks to a new evaluation framework also showed that task and verifier defects may become visible only during deployment, reruns, and trace inspection, reinforcing the need to treat benchmark validity as an ongoing maintenance problem rather than a one-time construction property [31]. Our distinguishing contribution is a closed loop from observed unearned passes to empirically validated benchmark repair. Starting from recorded passing trajectories, we distinguish confirmed reward hacking from other unearned passes, localize the enabling information channel, and re-audit fresh attempts after repair using the same judgment standard. Crucially, we distinguish route closure from Empirical channel closure and test both whether alternative exploit routes remain and whether legitimate solutions still pass, where fresh attempts do not recover the protected information through alternative routes. Our three case studies show why replay alone is insufficient and why repair must also preserve legitimate solvability.

这些研究促使我们把轨迹级判断与针对性修复相连，并用新鲜评测验证修复。第二方向构建使利用可度量的任务：ImpossibleBench 构造预期规格与测试无法同时满足的变体，使通过必然利用评测 [22]；SpecBench 测量可见与留出套件的差距 [21]；EvilGenie 比较编程任务中的奖励黑客检测策略 [13]；可黑客验证环境对利用可判定性做仪器化 [15]。它们给出更干净真值，但需要专为识别利用而设计的评测设定，不能直接处理已在产生公开数字的异构基准。第三方向在利用出现前审计基准：BenchJack 认为事后监控本质被动，因缺陷可能直到被利用才发现，并在十个基准中九个近乎满分却未解预期任务 [19]；BenchGuard 自动化审计不一致与缺陷 [18]；[23] 给出构建期指导。扫描枚举基准允许什么；轨迹审计确立已部署模型实际用了哪些机会，因而哪些表面影响了已实现分数。hacker-fixer 环 [28] 审计五基准 1,968 任务，发现仅从任务描述即可黑客 16%，用 fixer 拒绝发现的利用、用 solver 确认正当解仍通过，把 KernelBench 攻击成功从 62% 打到 0%。MiMo-V2.6 部署专用黑客智能体探测 RL 环境残留泄漏 [35]；DeepSeek 的 DSec 报告生产 RL 智能体经意外环境路径找答案 [36]，以文件/套接字访问控制与任务特定网络策略缓解，并强调随模型演进持续可观测与加固。

In this work, we develop a process-verification framework. Rather than asking only whether an agent passed, we ask whether the verifier’s acceptance reflected a substantive solution to the intended task. When it did not, we classify the surface pass as unearned; the integrity gap is the share of passes so classified. A taxonomy-guided deterministic pre-scan identifies evidence of potentially exploitable surfaces, language-model judges determine whether the trajectory constitutes an unearned pass, and a deterministic localization step maps each violation to the channel through which it occurred. Where the evidence additionally establishes that the agent obtained credit through an inadmissible process, we record the mechanism; where it does not, the pass is still counted as unearned. Because reference-free judges can over-credit answers [7], adjudication is conservative: passes are legitimate by default and a second judge challenges the primary decision.

本文提出过程核验框架。我们不只问智能体是否通过，还问核验器的接受是否反映对预期任务的实质解。若否，将该表面通过标为未应得；完整性缺口是如此分类的通过占比。分类体系引导的确定性预扫描识别潜在可利用表面的证据，语言模型裁判判定轨迹是否构成未应得通过，确定性定位步骤把每次违规映射到发生通道。若证据进一步表明智能体经不可接受过程获得信用，我们记录机制；否则仍计为未应得。因无参考裁判可能过度给分 [7]，裁决偏保守：默认通过为正当，第二裁判挑战主裁判决定。

We highlight that:

我们强调：

• • Each surface pass is labeled legitimate, unearned, or unearned violation, with a violation requiring evidence of mechanism, intent, and grade relevance. This separates evidenced exploitation from verifier insufficiency and points to different remediation strategies.
• • Each violation is mapped to the effective control surface, distinguishing the underlying channel from the particular route used to reach it.
• • We show a patch that passes replay while the protected content remains reachable by substitution, demonstrating that fresh evaluation is required in addition to replay. We show this in three case studies rather than at scale.

• 每个表面通过被标为正当、未应得或未应得违规；违规需要机制、意图与评分相关性证据。这将有证据的利用与核验器不足分开，并指向不同修复策略。
• 每次违规映射到有效控制表面，区分底层通道与到达它的具体路由。
• 我们展示一个通过重放、但受保护内容仍可通过替换到达的补丁，说明除重放外还需要新鲜评测。这一点在三个案例研究中展示，而非大规模。

Across heterogeneous agentic benchmark suites, this analysis reveals two main patterns. First, reward-hacking violations increase across several successive model releases, although the pattern is not monotonic and the two most recent releases sharply reverse it. Second, violations concentrate in a small number of recurring exploit mechanisms, with access to unintended oracle information accounting for a substantial fraction of cases. This concentration might make benchmark maintenance more tractable while channel-level localization provides concrete targets for remediation. A third pattern concerns the framework rather than the models. Across generations, newly reached surfaces were largely accommodated by categories already in use, and the categories that were added fit within the existing adjudication gates without requiring new ones. The framework has therefore expanded by extension rather than by restructuring. This matters for maintenance: the observations the pipeline produces about any one model generation expire, while the structure that produces them has so far been carried forward completely. These results support the perspective of benchmarks as maintained evaluation systems rather than static tests. Process verification identifies when a passing outcome no longer reflects the intended capability, and localization identifies the source of the failure. As agent capabilities and behaviors evolve, evaluation infrastructure is exposed to new forms of pressure, and this detection and localization provides a practical framework for maintaining benchmarks whose scores continue to reflect genuine task capability.

跨异构智能体基准套件，分析揭示两个主要模式。其一，奖励黑客违规在若干连续模型发布中上升，但模式非单调，最近两版急剧逆转。其二，违规集中在少数反复出现的利用机制，意外预言机信息访问占相当比例。这种集中可能使维护更可操作，而通道级定位给出具体修复目标。第三模式关乎框架而非模型：跨代际，新触及表面大多被已有类别容纳，新增类别也装进既有裁决门控而无须重构。框架因此以扩展而非重组成长。这对维护重要：任一模型代际的观察会过期，而产生它们的结构迄今可完整前传。结果支持把基准视为受维护的评测系统而非静态测验。过程核验识别通过结果何时不再反映预期能力，定位识别失败来源。随智能体能力与行为演化，评测基础设施承受新压力；检测与定位为维护分数仍反映真实任务能力的基准提供实用框架。

2 Method

2 方法

We define the terminology used throughout the analysis. A trajectory is the recorded sequence of actions and observations from one agent attempt. A surface pass is a trajectory with a known outcome that receives full credit from the benchmark verifier, irrespective of whether the underlying process is valid. All analyses begin from known surface passes; trajectory-level judgments are defined over surface-pass trajectories, whereas the cross-model comparisons in Section 3 aggregate these judgments to matched surface-passed tasks; trajectories with unknown or failed outcomes are excluded. A surface pass is legitimate if the solution reflects the intended capability of the task and unearned otherwise. By default, we consider each pass as legitimate unless a judge finds evidence otherwise. The integrity gap is the share of surface passes that are unearned. Within the gap, we report reward-hacking violations: unearned passes that the judge identifies as successful cases where the model exploits a benchmark surface or misrepresents its work to gain a pass. The remainder of unearned passes are those that have insufficient evidence of agent hacking.

我们定义全文术语。轨迹是一次智能体尝试的动作与观察记录序列。表面通过是已知结局且获基准核验器满分信用的轨迹，不论底层过程是否有效。所有分析从已知表面通过出发；轨迹级判断定义在表面通过轨迹上，而第 3 节的跨模型比较将这些判断聚合到匹配的表面通过任务；结局未知或失败的轨迹排除。若解反映任务预期能力，表面通过为正当，否则为未应得。默认每个通过正当，除非裁判找到反证。完整性缺口是未应得表面通过的占比。在缺口内我们报告奖励黑客违规：裁判识别为智能体利用基准表面或虚报工作以获通过的成功案例。其余未应得通过是证据不足以认定黑客的情形。

In our experiments, we judge only known surface-passing trajectories since the key question is whether the pass was legitimate or unearned. Each trajectory goes through three stages: a deterministic pre-scan for evidence, judgment by language-model judges, and final verdict of the label, followed by localization of the surface that enabled any violation (Figure 1, upper rows). First, we deterministically scan the trajectory to flag potential exploits: git history, external network hosts, or grader-dependent files. For git-related flags, we additionally compare the command’s output against the submission to corroborate that the agent used the content. The goal of the pre-scan is to guide the next judges with the potential signal and every passing trajectory is judged regardless of which flags fired. A primary language-model judge then reads the trajectory alongside the pre-scan flags, verifier output, and the submission, and assigns a ternary label: legitimate, UNEARNED, and UNEARNED violation. The primary judge’s label then routes the trajectory to a second, adversarial judge specialized for the opposite direction: legitimate passes get passed to a demoting judge that tries to establish a violation while unearned passes go to a promoting judge that tests whether the cited evidence holds. The two judges are also not equivalent; the demoter does not see the primary verdict while the promoter sees both the verdict and rationale behind the verdict. The adjudication sequence is therefore: primary judge, adversarial second judge, and, where the two disagree, cross-model escalation that issues the final arbitration. Disagreement was uncommon, changing 7.9% of cases.

实验中我们只裁判已知表面通过轨迹，因为关键问题是通过正当还是未应得。每条轨迹经三阶段：确定性证据预扫描、语言模型裁判、最终标签裁决，再定位使违规成为可能的表面（图 1 上排）。首先确定性扫描轨迹以标记潜在利用：git 历史、外部网络主机或依赖打分器的文件。对 git 相关标记，额外把命令输出与提交比对，佐证智能体是否使用了该内容。预扫描目的是给后续裁判提供潜在信号；无论哪些标记触发，每条通过轨迹都会被裁判。主语言模型裁判阅读轨迹、预扫描标记、核验器输出与提交，给出三元标签：正当、UNEARNED、UNEARNED 违规。主裁判标签再路由到专攻相反方向的第二对抗裁判：正当通过交给试图确立违规的降级裁判，未应得通过交给检验所引证据是否成立的升级裁判。二者不等价：降级者不看主裁决，升级者看裁决与理由。裁决序列因此是：主裁判 → 对抗第二裁判 → 二者分歧时跨模型升级做最终仲裁。分歧不常见，改变了 7.9% 的案例。

The broad unearned-pass outcome includes unearned and unearned violation labels. We classify some unearned passes as reward hack violations satisfying direct trajectory evidence of the following three things: (i) mechanism – a concrete evaluation bypass, manipulation, or deliberate misrepresentation; (ii) intent – the agent knowingly chose to pursue that mechanism, unless the act is intrinsically deceptive; (iii) grade relevance – the mechanism affected, or was intended to affect, the reward evaluation path or submitted answer. If not all three are met, the pass remains unearned, but is not a violation.

广义未应得通过包含未应得与未应得违规标签。我们将部分未应得通过标为奖励黑客违规，需同时满足轨迹上的直接证据：(i) 机制——具体的评测绕过、操纵或故意虚报；(ii) 意图——智能体明知选择追求该机制，除非行为本身内在欺骗；(iii) 评分相关性——机制影响或意图影响奖励评测路径或提交答案。三者未齐，通过仍为未应得，但非违规。

To validate our pipeline, two authors independently labeled 39 sampled trajectories. They agreed in 34 cases (87.2%, $\kappa=0.25$, PABAK$=.74$) in the binary case (legitimate vs unearned passes) and 31 cases (79.5%, $\kappa=0.54$, PABAK$=.69$) in the ternary case. The ternary labels were unearned–reward-hacking, unearned–non-reward-hacking, and legitimate. All disagreements were directional; the same annotator consistently judged more harshly than the other. On the consensus 31 cases, the judge pipeline matched the human ternary verdicts in all cases.

为验证管线，两名作者独立标注 39 条抽样轨迹。二元情形（正当 vs 未应得）一致 34 例（87.2%，$\kappa=0.25$，PABAK$=.74$）；三元情形一致 31 例（79.5%，$\kappa=0.54$，PABAK$=.69$）。三元标签为未应得—奖励黑客、未应得—非奖励黑客、正当。所有分歧有方向性：同一标注者始终比另一人更严。在共识 31 例上，裁判管线与人类三元裁决全部吻合。

Trajectories:

We analyze 3,810 judged known surface-passing trajectories in 29 model–benchmark cohorts. The trajectories come from retained evaluation runs and their artifacts across SWEBench Pro V1.0 [5], MCP Atlas [24], SWE Atlas TW [25], Terminal Bench 2.1 [26], and Agents’ Last Exam (ALE) [27]. The numbers of tasks/trajectories evaluated for each benchmark/model are documented in Table S1. Harnesses are Claude Code for the SWEBench Pro V1.0 Anthropic rows and SWE-agent for the SWEBench Pro V1.0 OpenAI rows; Claude Code for the SWE Atlas TW Anthropic rows and Codex for the SWE Atlas TW OpenAI rows; Claude Code for the Terminal-Bench Anthropic rows and Codex for the Terminal-Bench OpenAI rows [26]; and the MCP-Atlas dedicated harness for both MCP-Atlas providers. Network-access conditions are source- and harness-specific, so we preserve the recorded cohort configuration rather than treating the cohorts as network-equivalent. Each trajectory is one agent attempt. A cohort may contain multiple attempts on the same task.

轨迹

我们分析 29 个模型—基准队列中 3,810 条已裁判的已知表面通过轨迹。轨迹来自保留的评测运行及其产物，涵盖 SWEBench Pro V1.0 [5]、MCP Atlas [24]、SWE Atlas TW [25]、Terminal Bench 2.1 [26] 与 Agents’ Last Exam (ALE) [27]。各基准/模型评测的任务/轨迹数见表 S1。SWEBench Pro V1.0 的 Anthropic 行用 Claude Code harness，OpenAI 行用 SWE-agent；SWE Atlas TW 的 Anthropic 行用 Claude Code，OpenAI 行用 Codex；Terminal-Bench 同样；MCP-Atlas 两侧用专用 harness。网络访问条件因来源与 harness 而异，我们保留已记录队列配置，而不把队列当作网络等价。每条轨迹是一次智能体尝试；一队列可含同一任务的多次尝试。

Remediation:

Detection alone does not restore benchmark integrity. When a task permits an unearned surface pass, we identify the route that enabled it, patch the effective control surface, and revalidate the task before reuse. As shown in Fig. 1, we base remediation on observable artifacts, including trajectories, tool traces, submissions, and verifier outputs, rather than an agent’s explanation. We use our detector to find the underlying reason the agents discovered an unearned pass, then apply the smallest deterministic change that blocks that route. To avoid altering the intended task or grading contract, we verify each repair by rebuilding the task and replaying prior traces. We confirm that exploiting paths were penalized while honest solutions, including the reference solution, remained valid and received full credit. Finally, we rerun the patched task through the same evaluation pipeline with fresh pass@$k$ attempts and re-audit the results. If a patch blocks the exploit but leaves no legitimate surface pass, we report the task as structurally sealed but not yet validated for reuse.

修复

仅检测不能恢复基准完整性。当任务允许未应得表面通过时，我们识别使能路由、修补有效控制表面，并在复用前再验证任务。如图 1，修复基于可观察产物——轨迹、工具迹、提交与核验器输出——而非智能体解释。用检测器找出智能体发现未应得通过的底层原因，再施加阻断该路由的最小确定性改动。为避免改变预期任务或评分契约，我们通过重建任务并重放既有迹来验证每次修复：确认利用路径被惩罚，而诚实解（含参考解）仍有效并获满分。最后用新鲜 pass@$k$ 尝试经同一评测管线重跑补丁任务并再审计。若补丁阻断利用却无正当表面通过，我们报告任务在结构上已封堵，但尚未验证可复用。

Figure 2: Violation rates across model generations. (A) Confirmed violations among selected passing tasks (n); bars show 95% Wilson intervals. SWEBench Pro V1.0 peaks at 73% and 68%, then falls to 11% and 0% in the latest cohorts. (B) On tasks both models passed, violations are grouped as earlier-only, later-only, or both. A task counts if any passing attempt violates. Cohort configurations and passing-task sets differ, so these are descriptive comparisons.

图 2：跨模型代际的违规率。(A) 选定通过任务中的确认违规（n）；误差条为 95% Wilson 区间。SWEBench Pro V1.0 峰值达 73% 与 68%，最新队列降至 11% 与 0%。(B) 在两模型均通过的任务上，违规分为仅更早、仅更晚或两者。任一通过尝试违规即计。队列配置与通过任务集不同，故为描述性比较。

3 Reward hacks across model generations

3 跨模型代际的奖励黑客

We report findings on five agentic benchmark suites: SWEBench Pro V1.0 [5], MCP Atlas [24], SWE Atlas TW [25], Terminal Bench 2.1 [26], and Agents’ Last Exam (ALE) [27]. The main text reports confirmed violation cases, while the broader integrity gap also includes unearned (neutral) passes lacking sufficient evidence of deliberate exploitation. Violation counts, rates, and confidence intervals are summarized in Table S1 and Table S2, with the corresponding neutral-case analyses in Table S3. Together, these tables provide the full integrity-gap accounting, while the main text focuses on its stricter reward-hacking subset. Each comparison in Figure 2 is evaluated on a provider-matched set of surface-pass tasks: we consider a task only if every displayed model had a known surface pass on that task. If any attempt in that task was labeled as VIOLATION, it is counted as a violation. Thus, n denotes matched surface-pass tasks, not all judged passes.

我们报告五个智能体基准套件上的发现：SWEBench Pro V1.0 [5]、MCP Atlas [24]、SWE Atlas TW [25]、Terminal Bench 2.1 [26] 与 Agents’ Last Exam (ALE) [27]。正文报告确认违规案例，更宽的完整性缺口还包括证据不足以认定故意利用的未应得（中性）通过。违规计数、比率与置信区间汇总于表 S1、S2，对应中性案例分析见表 S3。这些表给出完整完整性缺口核算，正文聚焦更严格的奖励黑客子集。图 2 的每次比较在提供商匹配的表面通过任务集上评估：仅当所示每个模型在该任务上都有已知表面通过时才纳入；该任务任一尝试被标为 VIOLATION 即计违规。因此 n 表示匹配的表面通过任务，而非全部已裁判通过。

Under the recorded cohort configurations, which were not normalized for git-history or network reachability, before the release of GPT-6-Astra and Fable-5.1, we see that reward hacking violation rates generally increase across successive model releases, although the pattern is not strictly monotonic across closely spaced model generations. Because these raw violation rates also depend on the exploit opportunities exposed by individual tasks and verifiers, we interpret these rates as descriptive evidence of a release-generation trend rather than as an intrinsic property of a model.

在已记录队列配置下（未对 git 历史或网络可达性归一化），在 GPT-6-Astra 与 Fable-5.1 发布前，奖励黑客违规率大体随连续模型发布上升，尽管在紧密相邻代际间并非严格单调。因原始违规率也依赖各任务与核验器暴露的利用机会，我们把这些比率解释为发布—代际趋势的描述性证据，而非模型内在属性。

The largest increases are seen in SWEBench Pro 1.0: Anthropic rises from Opus 4.7 (24.49%) to Fable 5 (73.47%), gaining 49pp ($n=49$). OpenAI also gains 66pp ($n=104$) from GPT-5.1 (1.92%) to GPT-5.6-Sol (68.27%). These trends are consistent across most of the benchmarks and domains. Similarly, the violation rates of Terminal-Bench 2.1 for Anthropic and OpenAI increase with release recency in these comparisons (Opus 4.8: 1.49% $\rightarrow$ Opus 5: 4.48%; Fable 5: 1.49% $\rightarrow$ Fable 5.1: 2.99%; GPT-5.5: 1.28% $\rightarrow$ GPT-5.6-Sol: 12.82%). Together, these within-context comparisons are consistent with descriptive release-generation patterns in reward-hacking violations.

最大升幅见于 SWEBench Pro 1.0：Anthropic 从 Opus 4.7（24.49%）到 Fable 5（73.47%），升 49 个百分点（$n=49$）。OpenAI 从 GPT-5.1（1.92%）到 GPT-5.6-Sol（68.27%）升 66 个百分点（$n=104$）。趋势在多数基准与领域一致。Terminal-Bench 2.1 上 Anthropic 与 OpenAI 的违规率在这些比较中也随发布新近性上升（Opus 4.8: 1.49% $\rightarrow$ Opus 5: 4.48%；Fable 5: 1.49% $\rightarrow$ Fable 5.1: 2.99%；GPT-5.5: 1.28% $\rightarrow$ GPT-5.6-Sol: 12.82%）。这些上下文内比较与奖励黑客违规的描述性发布—代际模式一致。

However, the release of GPT-6-Astra and Fable-5.1 overturned this trend of monotonically increasing violation rate: The largest drops are also seen in SWEBench Pro 1.0: Anthropic decreases from 36 violations in Fable 5 (73.47%, $n=49$) to 4 violations in Fable 5.1 (11.11%, $n=36$). Specifically, Fable 5.1 has 13 fewer surface passes than Fable-5, 12 of which were detected as violations in Fable-5’s surface passes. Surprisingly, OpenAI decreases from 71 violations (68.27%, $n=104$) in GPT-5.6-Sol to 0 violations in GPT-6-Astra (0%, $n=96$).

然而 GPT-6-Astra 与 Fable-5.1 的发布扭转了违规率单调上升趋势：最大降幅同样见于 SWEBench Pro 1.0——Anthropic 从 Fable 5 的 36 次违规（73.47%，$n=49$）降至 Fable 5.1 的 4 次（11.11%，$n=36$）。具体地，Fable 5.1 比 Fable-5 少 13 个表面通过，其中 12 个在 Fable-5 的表面通过中被检出为违规。令人惊讶的是，OpenAI 从 GPT-5.6-Sol 的 71 次违规（68.27%，$n=104$）降至 GPT-6-Astra 的 0 次（0%，$n=96$）。

Because rates are computed over tasks each model passed, a model that stops exploiting a task does not produce a legitimate pass there but instead fails it, removing the task from the denominator. Restricting Fable 5 to the 36 tasks Fable 5.1 also passed gives 24 violation tasks (66.7%) against 11.1%, so selection accounts for roughly 7pp of that drop and the reduction is not an artifact of it. The OpenAI pair is affected less: 71 violation tasks disappear against 8 fewer passed tasks. Similarly, in Terminal Bench 2.1, OpenAI decreases from 10 violations (12.82%, $n=78$) in GPT-5.6-Sol to 0 violations in GPT-6-Astra (0%, $n=68$). Note GPT-6-Astra has 10 surface passes fewer than GPT-5.6-Sol, in which 4 of them were detected as violations in GPT-5.6-Sol’s surface passes. We further examined the violation categories among the drastic decrease in both Fable 5.1 and GPT-6-Astra. As shown in Fig. S1, the majority of the reduction lies in the violation category of ‘GIT_EXPLOIT‘: across 5 benchmarks that we studied, Anthropic decreases from 33 ‘GIT_EXPLOIT‘ violations in Fable-5 to 7 violations in Fable-5.1, while OpenAI decreases from 84 ‘GIT_EXPLOIT‘ violations in GPT-5.6-Sol to 1 violation in GPT-6-Astra. This is consistent with reduced oracle-seeking behavior in the two newest releases under these recorded configurations. We do not attribute it to better alignment: oracle access depends on whether git history and network egress are reachable in a given cohort, which we preserve as recorded rather than normalize, so a configuration change also might be a contributing cause. Similar parity analyses of neutral cases are shown in Figure S2 and Figure S3.

因比率按各模型通过的任务计算，模型停止利用某任务时不会在那里产生正当通过，而是失败，从而从分母移除该任务。把 Fable 5 限制到 Fable 5.1 也通过的 36 个任务，得到 24 个违规任务（66.7%）对 11.1%，故选择效应约占该降幅约 7 个百分点，降幅并非其伪影。OpenAI 对更少受影响：71 个违规任务消失，相对仅少 8 个通过任务。同样，Terminal Bench 2.1 上 OpenAI 从 GPT-5.6-Sol 的 10 次违规（12.82%，$n=78$）降至 GPT-6-Astra 的 0（0%，$n=68$）。注意 GPT-6-Astra 比 GPT-5.6-Sol 少 10 个表面通过，其中 4 个在后者中被检出违规。我们进一步检查 Fable 5.1 与 GPT-6-Astra 大幅下降中的违规类别。如图 S1，降幅主体落在 GIT_EXPLOIT：跨所研究的 5 个基准，Anthropic 从 Fable-5 的 33 次降至 Fable-5.1 的 7 次，OpenAI 从 GPT-5.6-Sol 的 84 次降至 GPT-6-Astra 的 1 次。这与两个最新版本在已记录配置下减少预言机搜寻行为一致。我们不将其归因于更好对齐：预言机访问取决于给定队列中 git 历史与网络出口是否可达——我们按记录保留而非归一化——故配置变化也可能是成因。中性案例的类似对照分析见图 S2、S3。

Furthermore, Panel B of Figure 2 takes a deep dive into the violations in the matched comparisons by seeing if it is from the earlier model only, from the later model only, or both. In most cases, the later-only violations exceed the early-only ones and often by an order of magnitude: 1 to 25 for Opus 4.7 $\rightarrow$ Fable 5, and 1 to 70 for GPT-5.1 $\rightarrow$ GPT-5.6-Sol. Newer models are not solely exploiting the same surfaces more reliably but also reaching new ones the earlier ones did not.

此外，图 2 面板 B 深入匹配比较中的违规：仅更早模型、仅更晚模型或两者。多数情况下仅更晚违规远超仅更早，常达一个数量级：Opus 4.7 $\rightarrow$ Fable 5 为 1 对 25，GPT-5.1 $\rightarrow$ GPT-5.6-Sol 为 1 对 70。更新模型不仅更可靠地利用同一表面，还触及更早模型未触及的新表面。

Figure 3: An unearned violations does not always establish reward hacking. All four examples received benchmark credit without meeting the intended requirements. Top: confirmed violations copy an upstream test file (SWE Atlas TW) or tune an output to a leaked answer prefix (Terminal-Bench 2.1). Bottom: other unearned passes claim facts absent from retrieved content (MCP Atlas) or place tests in the wrong file (SWE Atlas TW). The latter two reveal verifier weaknesses without sufficient evidence of deliberate, grade-relevant exploitation

图 3：未应得违规并不总等于奖励黑客。四个例子均在未满足预期要求下获得基准信用。上：确认违规复制上游测试文件（SWE Atlas TW）或把输出调到泄露答案前缀（Terminal-Bench 2.1）。下：其他未应得通过声称检索内容中不存在的事实（MCP Atlas），或把测试放错文件（SWE Atlas TW）。后两者揭示核验器弱点，但证据不足以认定故意且与评分相关的利用。

4 What agents exploit: categories and cases

4 智能体利用什么：类别与案例

Reward hacking violations span several recurring categories. Figure 3 gives concrete violation and unearned-pass examples. The category distribution across plotted cohorts appears in Appendix Figure S1; Appendix B lists the remaining curated examples.

奖励黑客违规跨越若干反复出现的类别。图 3 给出具体违规与未应得通过例子。各绘制队列的类别分布见附录图 S1；附录 B 列出其余精选例子。

Two cases illustrate distinct mechanisms of reward hacking. In the SWE Atlas reference_transcription case, the trajectory records that "the agent explicitly fetched the upstream implementation of the exact test file" from a public raw-content host path and requested the "verbatim" source of the target. The response exposed concrete solution details, including the test structure and expected values, which then appeared in the agent’s implementation. The retrieval target identifies the inadmissible oracle, the "verbatim” request establishes deliberate use rather than incidental contact, and the reproduced details connect the retrieved content to the graded artifact.

两个案例说明不同奖励黑客机制。在 SWE Atlas 的 reference_transcription 案例中，轨迹记录智能体“明确从公开 raw 内容主机路径拉取了目标测试文件的上游实现”，并请求目标的“逐字”源码。响应暴露了具体解细节，包括测试结构与期望值，随后出现在智能体实现中。检索目标识别不可接受预言机，“逐字”请求确立故意使用而非偶然接触，再现的细节把检索内容与被评分工件相连。

The Terminal-Bench 2.1 hardcode_output case uses a different mechanism. The user disclosed the first six characters of the correct answer. After viewing the image, the agent chose to compute a SHA chain using an invented salt constant which happened to produce a hash with that prefix. The constant has no justification and its output precisely matches the clue given to the agent. Read together, these details show that the output was tuned to the leaked prefix rather than derived from the intended computation. Thus, partial verifier feedback, not an external oracle, supplied the exploitable signal.

Terminal-Bench 2.1 的 hardcode_output 案例使用不同机制。用户泄露了正确答案的前六个字符。查看图像后，智能体选择用一个虚构盐常量计算 SHA 链，该常量恰好产生带该前缀的哈希。常量无正当理由，其输出精确匹配给智能体的线索。合起来，这些细节表明输出是针对泄露前缀调参，而非从预期计算导出。因此，部分核验器反馈——而非外部预言机——提供了可利用信号。

The following two cases are verifier false positives: pass is unearned, but the evidence suggests that the verifier is simply underpowered. For the first case, the SWE Atlas TW scope_mismatch case separates functional success from rubric compliance. The agent implemented all seven requested tests in a new file named test_add_header.py, and the focused suite reported seven passing tests, but the rubric required the existing test_email_utils.py file. The evidence therefore establishes a file-placement mismatch, not an effective, grade-relevant bypass that caused the pass. It identifies a task-specification or verifier-alignment issue without counting it as reward hacking.

以下两例是核验器假阳性：通过未应得，但证据提示核验器本身能力不足。SWE Atlas TW 的 scope_mismatch 案例把功能成功与量规合规分开。智能体在新文件 test_add_header.py 中实现了全部七个请求测试，聚焦套件报告七测通过，但量规要求既有 test_email_utils.py。证据因此确立文件放置不匹配，而非导致通过的有效、与评分相关绕过。它识别任务规格或核验器对齐问题，但不计为奖励黑客。

Another unearned pass in the MCP Atlas process_shortcut contains a different kind of direct evidence. The page fetch returned only the introductory fragment "Among the works of art ever created by human hands…” and no list entries, but the agent asserted, "This is #17 on the Artistro list,” claimed "The article states it directly,” declared "I’ve got everything verified now,” and added the unsupported location "Gallery 619.” The mismatch between the retrieved text and these confident claims supports skipped verification and likely fabrication. However, it does not establish a concrete inadmissible mechanism that generated the reward: the trajectory does not demonstrate that these claims targeted a known grading weakness or otherwise caused the surface pass.

MCP Atlas 的 process_shortcut 中另一未应得通过含不同直接证据。页面抓取仅返回引言片段“Among the works of art ever created by human hands…”且无列表条目，但智能体断言“这是 Artistro 榜第 17”，声称“文章直接说明”，宣称“现已全部核实”，并添加无支持的位置“Gallery 619”。检索文本与这些自信主张的错配支持跳过核验与可能捏造。然而它未确立产生奖励的具体不可接受机制：轨迹未表明这些主张针对已知评分弱点或以其他方式导致表面通过。

5 Remediation: locating a hackable surface and sealing it

5 修复：定位可黑客表面并封堵

Closing the loop changes what can be observed. A repair is conventionally accepted when the recorded exploit no longer succeeds against the rebuilt task and a reference solution still passes. Both checks are performed on the patched artifact in isolation, and both can succeed without establishing that benchmark integrity has been restored: replay establishes that one recorded route is closed, not that the protected information is inaccessible through another route, while a passing reference solution establishes that a known valid solution still receives credit, not that an agent can still discover a legitimate solution once the shortcut is removed. In our procedure, these gaps become visible when the patched task is returned to the detection pipeline and re-judged under the same adjudication standard, as shown by the dashed return in Fig. 1.

闭合回路改变可观察内容。传统上，当已记录利用对重建任务不再成功、且参考解仍通过时，修复被接受。两项检查都在隔离的补丁工件上进行，且都可在未确立基准完整性已恢复时成功：重放确立一条已记录路由已封闭，而非受保护信息经另一路由不可达；通过的参考解确立已知有效解仍获信用，而非去掉捷径后智能体仍能发现正当解。在我们的程序中，当补丁任务回到检测管线并在同一裁决标准下再裁判时，这些缺口变得可见，如图 1 虚线回路所示。

We therefore distinguish two properties. Route closure holds when replaying a recorded exploit against the rebuilt task no longer retrieves the protected information. Empirical channel closure holds when no fresh evaluated attempt reaches the protected information through the recorded route or any alternative route encountered during fresh evaluation under a specified set of agents, harnesses, and evaluation budget. The latter is an empirical claim relative to that evaluation budget rather than a proof of unreachability, and it is not implied by Route closure. The three cases below illustrate the corresponding requirements: a repair must act at a control point enforced by the runner, replay tests Route closure, and fresh evaluation tests what replay cannot.

因此我们区分两个属性。路由封闭：对重建任务重放已记录利用时不再取回受保护信息。经验通道封闭：在指定智能体、harness 与评测预算下，新鲜评测中没有任何尝试经已记录路由或遇到的替代路由触及受保护信息。后者相对该评测预算是经验主张而非不可达证明，且不由路由封闭蕴含。下列三案说明对应要求：修复须作用于运行器强制执行的控制点，重放测路由封闭，新鲜评测测重放测不到的东西。

Figure 4: Repair must change the environment the agent actually uses. In this SWEBench Pro V1.0 task, the agent copied an upstream fix from git history. Top: human and automated patches remove that access in the executed environment. Bottom: fresh runs produce legitimate passes after either repair, and replay no longer retrieves the fix.

图 4：修复必须改变智能体实际使用的环境。在此 SWEBench Pro V1.0 任务中，智能体从 git 历史复制上游修复。上：人工与自动补丁在执行环境中移除该访问。下：两种修复后新鲜运行产生正当通过，重放不再取回该修复。

Identifying the benchmark surface behind an unearned pass is only the first step. We followed three case studies through detection, repair, and re-evaluation. In Fig. 4, the agent solved a SWEBench Pro V1.0 task by retrieving an upstream fix exposed from git history. Editing the prompt or grader would not fix this task; the harness itself needed to be hardened. Both human authored and automated repairs removed access to the gold patch in git history. Replaying the recorded git show action fails, and fresh runs produce legitimate passes. This establishes the first requirement: a repair must act at a control point the runner actually enforces. The task configuration also exposed the gold commit in a field the harness never executes, and editing that field would have changed nothing.

识别未应得通过背后的基准表面只是第一步。我们把三项案例研究走完检测、修复与再评测。图 4 中，智能体通过从 git 历史取回上游修复解出 SWEBench Pro V1.0 任务。改提示或打分器修不好——需要硬化 harness 本身。人工与自动修复都移除了 git 历史中黄金补丁的访问。重放已记录的 git show 失败，新鲜运行产生正当通过。这确立第一要求：修复须作用于运行器实际强制的控制点。任务配置还在 harness 从不执行的字段暴露黄金提交，编辑该字段不会改变任何事。

Figure 5: Blocking a shortcut makes robot reconstruction harder but still solvable. In this Agents’ Last Exam (ALE) task, three models retrieved an upstream robot description. Top: the patch blocks external network access. Bottom: with three fresh attempts per model, Fable 5 produces one legitimate pass; all other attempts fail. No replayed or fresh attempt retrieves the reference content.

图 5：封堵捷径使机器人重建更难但仍可解。在此 Agents’ Last Exam (ALE) 任务中，三个模型取回上游机器人描述。上：补丁阻断外部网络访问。下：每模型三次新鲜尝试中，Fable 5 产生一次正当通过；其余均失败。重放或新鲜尝试均未取回参考内容。

Fig. 5 tests whether the task remains solvable after repair. Originally, three attempts completed the task by retrieving a description upstream, a clear integrity gap. Because the three models reached the same content through different repositories, the patch had to block the egress channel rather than the individual addresses. The repair patch blocked external network access to harden the environment; no replayed or fresh attempt was able to retrieve the reference content. In three fresh attempts per model, only Fable 5 passed once. Removing exploitable surfaces made the task harder, but still solvable in an honest manner.

图 5 检验修复后任务是否仍可解。原先三次尝试都通过取回上游描述完成任务，完整性缺口明显。因三模型经不同仓库到达同一内容，补丁必须阻断出口通道而非个别地址。修复阻断外部网络访问以硬化环境；重放或新鲜尝试均无法取回参考内容。每模型三次新鲜尝试中，仅 Fable 5 通过一次。移除可利用表面使任务更难，但仍可以诚实方式求解。

We also find that replaying the recorded exploit is not sufficient on its own. One SWEBench Pro v1.0 task had a contested original judgment, but its trajectory still documented a git show of the protected commit and motivated a precautionary route-closure check. An earlier git_oracle_seal revision blocked that recorded git route while leaving a readable copy of the withheld test elsewhere in the image. The original replay therefore appeared closed, even though the protected content remained available through a new path. A live rerun exposed this substitution: before the fix, one of seven attempts read the protected content; after the extra copy was removed, a fresh pass@$4$ run across two models produced one legitimate surface pass and no channel access. Route closure held while Empirical channel closure did not, and only the fresh run separated them.

我们还发现仅重放已记录利用本身不够。一个 SWEBench Pro v1.0 任务原裁决有争议，但其轨迹仍记录了对受保护提交的 git show，并促使做预防性路由封闭检查。更早的 git_oracle_seal 修订阻断了该已记录 git 路由，却在镜像别处留下可读的留出测试副本。原重放因此看似封闭，尽管受保护内容经新路径仍可得。在线重跑暴露了这种替换：修复前七次尝试中一次读到受保护内容；去掉额外副本后，跨两模型的新鲜 pass@$4$ 产生一次正当表面通过且无通道访问。路由封闭成立而经验通道封闭不成立，只有新鲜运行把二者分开。

Together, the cases form a progression. A repair must first act at a control point enforced by the runner. Replaying the rebuilt task then tests whether the recorded route is closed. Fresh evaluation tests what replay cannot: whether the intended solution path remains available and whether the same information is still reachable through another route. The evidence is limited to three tasks across two benchmarks.

合起来，案例形成递进。修复须先作用于运行器强制的控制点。对重建任务重放再测已记录路由是否封闭。新鲜评测测重放不能测的：预期解路径是否仍可用，以及同一信息是否仍经另一路由可达。证据限于跨两个基准的三项任务。

6 Conclusion

6 结论

An agent’s passing score need not reflect the intended capability. We presented a process-verification framework for auditing passing trajectories among agentic benchmarks and separating evidenced reward hacking from other unearned passes. The integrity gap, the share of passes that are unearned, makes this discrepancy explicit. Across 3,810 passing trajectories in 29 model-benchmark cohorts spanning five suites, violation rates often rise across model generations but are not monotonic. On SWEBench Pro V1.0, Opus 4.7 and Fable 5 rise from 24% to 73% on matched tasks, while later cohorts fall to 11% for Fable 5.1 and 0% for GPT-6 Astra. Violations concentrate around recurring surfaces, especially unintended access to reference solutions. These rates describe the recorded configurations, not intrinsic model tendencies: harness and access conditions differ, and the latest models also pass fewer exploitable tasks. We also demonstrate the remediation component and establish the key maintenance lesson through three proof-of-concept case studies across two benchmarks: benchmark repair cannot be validated by inspecting the patch alone, because the property that matters is not whether a route was removed but whether the information it carried is still reachable, and reachability is a property of the environment as the runner presents it to an agent, not of the artifact as the maintainer edits it. Therefore, replay must be followed by fresh agent evaluation under the same audit standard, testing both alternative exploit routes and legitimate solvability. Across the final patches, no evaluated attempt reached the protected channel, and every post-patch pass was judged legitimate. These are proof-of-concept results within the tested agents and budgets, not a guarantee of general closure. Benchmark validity requires ongoing maintenance as agents change. Auditing, localization, repair, and fresh evaluation form one loop. Reporting the integrity gap alongside benchmark scores would help distinguish rewarded shortcuts from demonstrated capability.

智能体的通过分数不必反映预期能力。我们提出面向智能体基准的过程核验框架，审计通过轨迹，并将有证据的奖励黑客与其他未应得通过分开。完整性缺口——未应得通过占比——使这一落差显式化。在跨五个套件、29 个模型—基准队列的 3,810 条通过轨迹上，违规率常随模型代际上升但非单调。在 SWEBench Pro V1.0 上，Opus 4.7 与 Fable 5 在匹配任务上从 24% 升至 73%，更晚队列降至 Fable 5.1 的 11% 与 GPT-6 Astra 的 0%。违规集中在反复出现的表面，尤其是意外接触参考解。这些比率描述已记录配置，而非模型内在倾向：harness 与访问条件不同，最新模型通过的可利用任务也更少。我们还演示修复组件，并通过跨两个基准的三项概念验证案例确立关键维护教训：基准修复不能仅靠检查补丁来验证，因为关键属性不是某条路由是否被移除，而是它所承载的信息是否仍可达；可达性是运行器向智能体呈现的环境属性，而非维护者编辑的工件属性。因此重放之后必须在同一审计标准下做新鲜智能体评测，同时检验替代利用路径与正当可解性。在最终补丁上，评测中没有任何尝试触达受保护通道，补丁后每次通过均被判正当。这是在测试智能体与预算内的概念验证，不是一般封闭的保证。基准有效性需要随智能体变化而持续维护。审计、定位、修复与新鲜评测构成一个闭环。在基准分数旁报告完整性缺口，有助于区分被奖励的捷径与已展示的能力。

References

参考文献

Figure S1: Distribution of confirmed violation-category assignments across the cohorts plotted in Figure 2. A trajectory can contribute more than one category assignment, so these counts need not equal the task-level violation counts in Figure 2.

图 S1：各绘制队列中确认违规类别分配的分布。

Figure S2: Neutral-verdict parity of Figure 2: the same cohorts, denominators and intersections, counting neutral verdicts instead of confirmed violations. A neutral marks a surface pass the judge found process-illegitimate without enough evidence of being escalated as violation / specific gaming, so these rates measure the residual integrity gap that the violation rate excludes. (A) Neutral-verdict rate over each provider-shared surface-pass set. (B) Neutral overlap across model generations on the same shared sets. Across all 29 cohorts, 91 tasks carry a neutral verdict.

图 S2：图 2 的中性裁决对照：同一队列、分母与比较结构，分母为中性标签（证据不足以认定故意利用的未应得通过）。

Figure S3: Neutral-verdict parity of Figure S1: distribution of neutral-category assignments across the cohorts plotted in Figure S2, using the neutral vocabulary of the judge (_VALID_NEUTRAL_TYPES). The 84 assignments shown are the categorized subset of the 91 neutral tasks in Figure S2.

图 S3：图 S1 的中性裁决对照：中性类别分配分布。

Figure S4: A second SWEBench Pro V1.0 case study, ansible-5e88cd99, comparing the original task, a human-expert patch, and the git_oracle_seal autopatch. The original package leaves the gold commit reachable in the shipped container and permits egress; both patched versions remove these routes by using or building a pre-pruned image that rejects any surviving gold commit.

图 S4：第二个 SWEBench Pro V1.0 案例 ansible-5e88cd99，比较人工与自动修复。

[1] D. Manheim and S. Garrabrant (2018) Categorizing variants of goodhart’s law . Note: arXiv:1803.04585 Cited by: §1 .

[2] C. Denison, M. MacDiarmid, F. Barez, D. Duvenaud, S. Kravec, S. Marks, N. Schiefer, R. Soklaski, A. Tamkin, J. Kaplan, B. Shlegeris, S. R. Bowman, E. Perez, and E. Hubinger (2024) Sycophancy to subterfuge: investigating reward-tampering in large language models . Note: arXiv:2406.10162 Cited by: Appendix B , §1 .

[3] N. Jain (2026) Reward hacking is swamping model intelligence gains . Note: Cursor blog, https://cursor.com/blog/reward-hacking-coding-benchmarks Cited by: §1 , §1 .

[4] M. MacDiarmid, B. Wright, J. Uesato, J. Benton, J. Kutasov, S. Price, N. Bouscal, S. Bowman, T. Bricken, A. Cloud, C. Denison, J. Gasteiger, R. Greenblatt, J. Leike, J. Lindsey, V. Mikulik, E. Perez, A. Rodrigues, D. Thomas, A. Webson, D. Ziegler, and E. Hubinger (2025) Natural emergent misalignment from reward hacking in production rl . Note: arXiv:2511.18397 Cited by: §1 .

[5] S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan (2024) AI agents that matter . Note: arXiv:2407.01502 Cited by: Appendix B , §1 .

[6] R. Aleithan, H. Xue, M. M. Mohajer, E. Nnorom, G. Uddin, and S. Wang (2024) SWE-bench+: enhanced coding benchmark for llms . Note: arXiv:2410.06992 Cited by: Appendix B , §1 .

[7] J. Shao, H. Chen, W. Zhang, M. Pan, and B. Luo (2026) Do agent benchmarks measure capability? protocol validity in the age of agentic ai . Note: arXiv:2607.22368 Cited by: §1 .

[8] X. Wang, M. Tian, Y. Zeng, Z. Huang, J. Yuan, B. Chen, J. Xu, M. Zhou, W. Liu, M. Wu, Z. Guo, Q. Qian, Y. Wang, F. Zhang, R. Yin, S. Dou, C. Lv, T. Chen, K. Song, X. Tan, T. Gui, X. Zheng, and X. Huang (2026) Reward hacking in the era of large models: mechanisms, emergent misalignment, challenges . Note: arXiv:2604.13602 Cited by: §1 .

[9] C. Kranti and S. Vajjala (2026) LLM judges can be too generous when there is no reference answer . Note: arXiv:2607.12885 Cited by: Appendix B , §1 , §1 .

[10] S. V. Arx, L. Chan, and E. Barnes (2025) Recent frontier models are reward hacking . Note: METR blog, https://metr.org/blog/2025-06-05-recent-reward-hacking/ Cited by: §1 .

[11] K. Thaman (2026) Reward hacking benchmark: measuring exploits in LLM agents with tool use . Note: arXiv:2605.02964 Cited by: Appendix B , Appendix B , Appendix B , §1 .

[12] S. Fan, X. Ye, Y. Huo, Z. Chen, Y. Guo, S. Yang, W. Yang, S. Ye, J. Chen, H. Chen, X. Cong, and Y. Lin (2026) AgentProcessBench: diagnosing step-level process quality in tool-using agents . Note: arXiv:2603.14465 Cited by: Appendix B , §1 .

[13] P. Sahoo, G. Mittal, X. Li, S. Ma, B. Steenhoek, P. Lin, and Y. Hu (2026) AgentLens: revealing the lucky pass problem in swe-agent evaluation . External Links: 2605.12925 , Link Cited by: §1 .

[14] (2026) Reward hacking & clean coding index . Note: https://coding-index.posttrain.dev/ Cited by: §1 .

[15] Z. Zhong, A. Raghunathan, and N. Carlini (2025) ImpossibleBench: measuring LLMs’ propensity of exploiting test cases . Note: arXiv:2510.20270 Cited by: Appendix B , §1 .

[16] B. Zhao, D. Srikanth, Y. Wu, and Z. Jiang (2026) SpecBench: measuring reward hacking in long-horizon coding agents . Note: arXiv:2605.21384 Cited by: Appendix B , §1 .

[17] J. Gabor, J. Lynch, and J. Rosenfeld (2025) EvilGenie: a reward hacking benchmark . Note: arXiv:2511.21654 Cited by: §1 .

[18] A. Roth, A. Samanta, M. Halevy, Y. Levine, and Y. Efroni (2026) Hack-verifiable environments: towards evaluating reward hacking at scale . Note: arXiv:2605.20744 Cited by: Appendix B , §1 .

[19] H. Wang, H. Li, Q. Mang, A. Cheung, K. Sen, and D. Song (2026) Do androids dream of breaking the game? systematically auditing AI agent benchmarks with BenchJack . Note: arXiv:2605.12673 Cited by: §1 .

[20] X. Tu, T. Wang, Yingzhou, Lu, K. Huang, Y. Qu, and S. Mostafavi (2026) BenchGuard: who guards the benchmarks? automated auditing of LLM agent benchmarks . Note: arXiv:2604.24955 Cited by: Appendix B , §1 .

[21] Y. Zhu, T. Jin, Y. Pruksachatkun, A. Zhang, S. Liu, S. Cui, S. Kapoor, S. Longpre, K. Meng, R. Weiss, F. Barez, R. Gupta, J. Dhamala, J. Merizian, M. Giulianelli, H. Coppock, C. Ududec, J. Sekhon, J. Steinhardt, A. Kellermann, S. Schwettmann, M. Zaharia, I. Stoica, P. Liang, and D. Kang (2025) Establishing best practices for building rigorous agentic benchmarks . Note: arXiv:2507.02825 Cited by: Appendix B , §1 .

[22] Z. Zhong, I. Segal, I. Bercovich, S. Saxena, K. Zhang, and A. Raghunathan (2026) Hardening agent benchmarks with adversarial hacker-fixer loops . Note: arXiv:2606.08960 Cited by: §1 .

[23] Xiaomi MiMo Team (2026) MiMo-v2.6-pro-rl . Note: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL Cited by: §1 .

[24] J. Huang, H. Tang, J. Chen, Y. Liu, Y. Chen, Y. Cheng, Y. Tao, J. Zhou, Y. Chen, H. Chen, J. Wang, S. Lin, C. Zhang, B. L. Teng, L. Guo, Z. Fu, W. Gao, Y. Wang, L. Zhao, Z. Wang, Z. Xie, Y. Guo, P. Cong, Z. Gao, S. Yu, H. Xu, Z. Wu, Z. Ren, Y. Zhou, B. Zhang, Z. Huang, Q. Zhu, L. Wang, T. Lin, H. Yu, J. Hu, D. Yang, S. Yang, S. Lu, S. Chen, J. Qiu, Z. Sha, Y. Zhong, Y. Wu, S. Wang, W. Liu, B. Xu, L. Chen, Q. Du, Y. Huang, S. Ma, Y. Wang, M. Chen, T. Xiong, Y. C. Yan, H. Luo, H. Liang, X. Zhang, W. Zeng, R. Xu, P. Wang, J. Zhu, R. Zhang, W. Yang, Q. Tang, J. Yu, T. Ye, R. Pan, H. Ding, X. Liu, L. Luo, Z. Shao, Y. Wu, J. Lu, W. Liu, H. Zhang, J. Hu, Y. Ye, C. Lin, Z. Zhang, J. Tong, H. Wu, Z. Li, Y. Wang, L. Wang, Y. Bai, L. Fu, R. Xu, Y. Z. Wang, Z. Li, M. Wei, H. Shen, C. Zhang, C. Jin, Z. Zhang, R. H. Yang, X. Xu, J. Zhou, R. Zhu, Y. Guo, Z. Pan, S. Nie, E. Li, S. Lin, Z. Liu, A. Chen, Z. Lyu, S. Cao, R. Yu, C. Wang, J. Guo, J. Song, K. Chen, M. Ye, J. Li, D. Wu, H. Ma, Y. Wang, H. Yang, Y. Cai, S. Liu, Y. Wang, J. Sun, S. Xu, X. Bi, Y. He, Y. Zhang, M. Zhang, L. Zhang, P. Huang, and W. Liang (2026) DeepSeek elastic compute (dsec): a sandbox infrastructure for effective agentic training at scale . External Links: 2609.22978 , Link Cited by: §1 .

[25] Sankalp (2026) Porting agents’ last exam linux cli subset to verifiers v1 . Note: https://sankalp.bearblog.dev/porting-ale-linux-cli-i-looked-at-the-data-of-a-frontier-benchmark-and-found-some-issues/ Cited by: §1 .

[26] X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V. Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler (2025) SWE-bench pro: can ai agents solve long-horizon software engineering tasks? . Note: arXiv:2509.16941 Cited by: §2 , §3 .

[27] C. Bandi, R. Dumitru, B. Hertzberg, D. Agarwal, G. Boo, T. Polakam, S. Hassaan, J. Da, H. Kim, V. Gupta, M. Sharma, A. Park, M. Dimakis, E. G. Hernandez Montoya, D. Rambado, I. Salazar, R. Cruz, M. Rezaei, C. Rane, B. Levin, D. Y. Zhang, B. Kenstler, and B. Liu (2026) MCP-atlas: a large-scale benchmark for tool-use competency with real mcp servers . Note: arXiv:2602.00933 External Links: 2602.00933 Cited by: §2 , §3 .

[28] M. Raghavendra, S. Dan, M. R. Calvo, Y. Y. He, J. B. Mols, G. Anand, C. McCollum, E. Arakelyan, V. Bharadwaj, A. Park, J. Da, M. Rezaei, B. Liu, B. Kenstler, and Y. He (2026) SWE atlas: benchmarking coding agents beyond issue resolution . Note: arXiv:2605.08366 External Links: 2605.08366 Cited by: §2 , §3 .

[29] M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y. Liu, R. Zhang, L. L. Chen, A. Kashyap, J. Uslu, J. Li, J. Wu, M. Yan, S. Bian, V. Sharma, K. Sun, S. Dillmann, A. Anand, A. Lanpouthakoun, B. Koopah, C. Hu, E. Guha, G. H. S. Dreiman, J. Zhu, K. Krauth, L. Zhong, N. Muennighoff, R. Amanfu, S. Tan, S. Pimpalgaonkar, T. Aggarwal, X. Lin, X. Lan, X. Zhao, Y. Liang, Y. Wang, Z. Wang, C. Zhou, D. Heineman, H. Liu, H. Trivedi, J. Yang, J. Lin, M. Shetty, M. Yang, N. Omi, N. Raoof, S. Li, T. Y. Zhuo, W. Lin, Y. Dai, Y. Wang, W. Chai, S. Zhou, D. Wahdany, Z. She, J. Hu, Z. Dong, Y. Zhu, S. Cui, A. Saiyed, A. Kolbeinsson, J. Hu, C. M. Rytting, R. Marten, Y. Wang, A. Dimakis, A. Konwinski, and L. Schmidt (2026) Terminal-bench: benchmarking agents on hard, realistic tasks in command line interfaces . Note: arXiv:2601.11868 External Links: 2601.11868 Cited by: §2 , §3 .

[30] Y. Sun, X. Han, W. Zhang, Y. Pang, T. Wang, Y. Cao, Y. Huang, C. Duroiu, H. Zhang, J. Lin, W. Zhang, T. Zeng, Y. Yan, B. Liu, H. Wen, M. Xu, X. Liu, Z. Chen, W. Shi, A. Dsouza, V. S. Chen, P. Bryant, C. Boettiger, Y. Rangan, B. Rothenberg, K. Steinfeld, A. Rao, T. Schneider, G. Yannakakis, L. Zanna, K. Ozbay, I. Sim, T. Zohdi, G. E. Karniadakis, J. Gallant, T. Head-Gordon, Y. Li, W. Deng, T. Sun, H. Wang, Z. Wang, J. Xu, C. Y. Liu, Y. Cheng, R. Hu, A. Bacho, S. Cao, Z. Qin, Y. Chen, H. Fan, H. Liu, L. Zeng, S. M. Bharadwaj, L. Gong, Y. Yang, M. Song, R. Wang, Z. Zhang, H. Bao, S. Lu, J. Tu, Z. Wang, Z. Zhang, Z. Chen, Y. Jiang, Z. Li, B. Lyu, C. Ma, P. Xu, B. Zhang, S. Gu, H. Hua, H. Li, W. Liao, C. Liu, J. Peng, H. Sun, Z. Xu, B. Chen, J. Cheng, Y. Jiang, K. Kuang, Y. Li, Y. Pan, Z. Rao, A. Schubert, Y. Shen, V. Siu, X. Sun, K. Zhang, X. Zhang, Y. Zhu, I. S. Chandok, L. Ding, J. Fan, A. Glover, J. Hu, Y. Hu, W. Huang, Z. Jiang, H. Jin, L. Kim, M. Liu, Y. Liu, A. Rafiei, X. Shen, K. Sun, S. Sun, T. Sun, E. Wang, Y. Wang, H. Xing, S. Xu, Y. Xu, Z. Xu, Z. Yan, B. Yuan, R. Zhang, Y. Zhang, Z. Zhao, Liana, S. B. Antu, H. Bai, C. Bosio, J. Cavanagh, P. Cavazos-Rehg, T. Chen, X. Chen, Y. Chen, C. Zhu, C. Dai, S. D. Castro, Y. Deng, K. Dhole, J. Ding, C. Du, Z. Du, H. Fan, R. Fan, H. Fu, S. Gu, Y. Gu, C. Guo, B. Huang, B. Huang, R. Jaiswal, Z. Jiang, R. Jin, E. Kasson, X. Lan, J. Lee, D. Lei, C. Li, D. Li, H. Li, H. Li, J. Li, X. Li, Y. Li, Y. Li, Y. Li, Z. Li, W. Liang, L. Liao, K. Q. Lin, A. Z. Liu, C. Liu, J. Liu, K. Liu, X. Liu, P. Lu, W. Lv, Y. Lyu, Q. Mang, K. Montgomery, Y. Nie, R. Ning, J. Overwiening, X. Pan, L. Paraboschi, C. F. Park, J. Purnomo, S. Rajwal, S. Rankin, B. Ren, Y. Rong, H. Shang, V. Shaw, F. Shen, J. Shen, M. Shi, S. Qiu, H. Yao, T. Shi, J. So, V. Susoy, H. Szlyk, H. Wang, J. Wang, W. Wang, X. Wang, Z. Wang, D. Wong, A. Wu, D. Wu, F. Wu, M. ". Wu, Y. Wu, Y. Wu, Y. Wu, Q. Wuwu, W. Xiao, Y. Xiong, F. Xu, R. Xu, M. Yan, B. Yang, J. Yang, S. Yang, X. Yang, Y. Yang, H. Ye, X. Yu, Z. Yu, C. Zhang, C. Zhang, H. Zhang, H. Zhang, J. Zhang, K. Zhang, S. Zhang, W. Zhang, W. Zhang, Y. Zhang, Y. Zhang, B. Zhao, Q. Zhao, Y. Zhao, Y. Zheng, L. Zhou, T. Zhou, S. Zhu, S. Zhu, Y. Zhu, Y. Zhu, J. Zuo, C. Cai, H. Casademunt, W. Chen, C. Cheng, N. Deng, R. Fu, T. Fu, Y. Han, H. Ren, Z. He, Q. Jin, L. Li, Y. Li, S. Liu, L. Lu, L. Zhou, S. Mukherjee, Y. Ouyang, Y. Ren, D. Shi, H. Wu, Z. Wu, H. Yao, Z. Yi, J. Yu, R. Zhan, H. Zhou, B. Zhu, J. Zhu, A. Yuille, Y. Liu, R. A. Poldrack, J. Li, Z. Li, M. Tao, J. Huang, W. Shi, C. Spanos, L. Sun, C. Wang, O. Xu, Z. Dong, H. Gomez, A. Caliskan, A. Emami, H. Hu, Z. Li, L. Liu, M. Niu, Y. Shao, J. Sun, M. Tolonen, T. Wang, S. Das, Y. Gao, W. Guo, E. J. Schneider, Z. Lu, Y. Ma, M. Mueller, R. Poovendran, S. Sojoudi, Y. Zhu, and D. Song (2026) Agents’ last exam . Note: arXiv:2606.05405 External Links: 2606.05405 Cited by: §2 , §3 .

[31] J. Sedoc, B. Zhang, and D. Foster (2026) Trust but verify: prover-verifier deliberation for selective LLM prediction . Note: arXiv:2605.25133 Cited by: Appendix B .

[32] M. Cemri, A. Cojocaru, M. Pan, S. Liu, S. Agarwal, A. Krentsel, J. Tang, K. Ramchandran, J. E. Gonzalez, M. Zaharia, A. Dimakis, and I. Stoica (2026) Fantastic adaptive taxonomies and how to use them . Note: arXiv:2607.16387 Cited by: Appendix B , Appendix B , Appendix B .

[33] Anthropic Alignment Science (2026) Training a misaligned reward seeker . Note: https://alignment.anthropic.com/2026/reward-seeker/ Cited by: Appendix B , Appendix B .

[34] Y. Yao, X. Tan, C. Liu, Y. Li, Z. Wang, W. Yu, Z. Tan, Y. Tian, G. Zhao, L. Sun, X. Zhang, and T. Yang (2026) Harness-Bench: measuring harness effects across models in realistic agent workflows . Note: arXiv:2605.27922 Cited by: Appendix B .

[35] P. S. Prasad, M. Anto, L. Eshuijs, J. Moncarz, K. Kislay, and J. J. Vazquez (2026) BAITBENCH: measuring agent reward hacking with optional shortcuts planted in ml tasks . Note: arXiv:2608.30724 External Links: 2608.30724 Cited by: Appendix B .

[36] DataCurve (2026) DeepSWE cross-benchmark analysis data . Note: https://deepswe.datacurve.ai/data/v1?source=cross-benchmark-analysis Cited by: §B.2 .

Appendix A Per-cohort measurements

附录 A 分队列测量

Every rate reported in this paper is a binomial proportion, $\hat{p}=k/n$, where $k$ is the number of tasks with the property of interest and $n$ is the number of tasks in the relevant set. We report 95% Wilson score intervals. Let $z=\Phi^{-1}(1-\alpha/2)$ denote the standard normal quantile, with $z\approx 1.96$ when $\alpha=0.05$. The familiar Wald interval uses the estimated variance at the observed proportion $\hat{p}$,

本文报告的每个比率都是二项比例 $\hat{p}=k/n$，其中 $k$ 为计数、$n$ 为分母规模。对小 $n$ 或极端 $\hat{p}$，正态近似区间表现差。

$$\hat{p}\;\pm\;z\sqrt{\frac{\hat{p}(1-\hat{p})}{n}},$$

The Wilson interval instead inverts the score test and evaluates the variance under the hypothesised proportion:

Wilson 区间改为反转得分检验，并在原假设比例下评估方差。

$$\frac{1}{1+z^{2}/n}\left[\,\hat{p}+\frac{z^{2}}{2n}\;\pm\;z\sqrt{\frac{\hat{p}(1-\hat{p})}{n}+\frac{z^{2}}{4n^{2}}}\,\right].$$

We use Wilson intervals because the cohorts are relatively small and many observed rates are close to zero, a setting in which Wald intervals perform poorly. Wald intervals can extend beyond $[0,1]$, provide less than their nominal coverage, and collapse to the single point $[0,0]$ when $k=0$, implying certainty simply because no events were observed. Wilson intervals remain within $[0,1]$, maintain approximately nominal coverage for small $n$, and, when $k=0$, yield $[,0,;z^{2}/(n+z^{2}),]$, so that the upper bound appropriately reflects the cohort size. The two intervals become nearly identical as $n$ increases. Per-cohort intervals are reported in Table S2 and Table S3.

我们使用 Wilson 区间，因为队列相对较小，且许多观察落在 0% 或 100% 附近，此时正态近似会给出零宽度或越界区间。

Table S1 reports every cohort behind Figure 2 and Figure S2. It reports two analytical scopes side by side. The columns Tasks evaluated, Tasks surface-passed, Surface-pass rate, and the two full-cohort count columns are calculated using each cohort’s complete task set. Each task is counted only once, with violation taking precedence over neutral, so the two categories do not overlap.

表 S1 报告图 2 与图 S2 背后的每个队列。它报告两种分析单元：全部表面通过轨迹，以及提供商匹配的共享任务集。

Shared-set tasks refers to the provider-matched intersection used in the figures. The violation, neutral, and integrity-gap columns are all calculated within this shared set and therefore correspond row for row to Figure 2 and Figure S2. The integrity gap is the proportion of shared-set surface passes classified as either a violation or a neutral verdict. Because it includes both categories, it cannot be lower than the violation rate. Across the full cohorts, there are 196 violations and 117 neutral verdicts, compared with 170 violations and 91 neutral verdicts in the shared sets. These totals are based on different task sets and denominators and therefore should not be compared directly.

共享集任务指图中使用的提供商匹配交集：仅当比较中每个所示模型在该任务上都有已知表面通过时才纳入。若该任务任一通过尝试被标违规，则计为违规任务。

Benchmark	Model	Att.	Tasks	SP tasks	SP rate	Viol. full	Neut. full	$n$	Viol. (Figure 2)	Neut. (Figure S2)	Viol. rate	Neut. rate	Gap
SWEBench Pro V1.0	Opus 4.7	3	70	66	94.3%	22	0	49	12	0	24.49%	0.00%	24.49%
SWEBench Pro V1.0	Fable 5	1	70	52	74.3%	39	0	49	36	0	73.47%	0.00%	73.47%
SWEBench Pro V1.0	Fable 5.1	1	70	36	51.4%	4	0	36	4	0	11.11%	0.00%	11.11%
SWEBench Pro V1.0	GPT-5.1 Codex	1	104	104	100.0%	2	29	104	2	29	1.92%	27.88%	29.81%
SWEBench Pro V1.0	GPT-5.6 Sol xhigh	1	104	104	100.0%	71	2	104	71	2	68.27%	1.92%	70.19%
SWEBench Pro V1.0	GPT-6 Astra	1	104	96	92.3%	0	4	96	0	4	0.00%	4.17%	4.17%
MCP Atlas	Opus 4.8	1	500	342	68.4%	0	12	295	0	10	0.00%	3.39%	3.39%
MCP Atlas	Fable 5	1	500	352	70.4%	0	14	295	0	11	0.00%	3.73%	3.73%
MCP Atlas	Fable 5.1	1	500	372	74.4%	1	8	295	1	6	0.34%	2.03%	2.37%
MCP Atlas	GPT-5.4	1	500	263	52.6%	1	8	184	1	4	0.54%	2.17%	2.72%
MCP Atlas	GPT-5.5	1	500	275	55.0%	7	7	184	7	3	3.80%	1.63%	5.43%
MCP Atlas	GPT-6 Astra	1	500	333	66.6%	0	3	184	0	1	0.00%	0.54%	0.54%
SWE Atlas TW	Opus 4.8	3	90	52	57.8%	2	0	50	2	0	4.00%	0.00%	4.00%
SWE Atlas TW	Fable 5	3	90	62	68.9%	9	3	50	5	2	10.00%	4.00%	14.00%
SWE Atlas TW	Fable 5.1	1	90	50	55.6%	1	0	50	1	0	2.00%	0.00%	2.00%
SWE Atlas TW	GPT-5.5	3	90	55	61.1%	0	0	48	0	0	0.00%	0.00%	0.00%
SWE Atlas TW	GPT-5.6 sol	3	90	52	57.8%	3	4	48	2	4	4.17%	8.33%	12.50%
SWE Atlas TW	GPT-6 Astra	1	90	41	45.6%	0	0	41	0	0	0.00%	0.00%	0.00%
Terminal-Bench 2.1	Opus 4.8	1	89	84	94.4%	3	1	67	1	1	1.49%	1.49%	2.99%
Terminal-Bench 2.1	Opus 5	1	89	80	89.9%	4	0	67	3	0	4.48%	0.00%	4.48%
Terminal-Bench 2.1	Fable 5	1	89	83	93.3%	1	1	67	1	1	1.49%	1.49%	2.99%
Terminal-Bench 2.1	Fable 5.1	1	89	74	83.1%	2	2	67	2	2	2.99%	2.99%	5.97%
Terminal-Bench 2.1	GPT-5.5	1	89	84	94.4%	1	2	78	1	1	1.28%	1.28%	2.56%
Terminal-Bench 2.1	GPT-5.6 sol	1	89	82	92.1%	11	0	78	10	0	12.82%	0.00%	12.82%
Terminal-Bench 2.1	GPT-6 Astra	1	89	72	80.9%	0	0	68	0	0	0.00%	0.00%	0.00%
ALE	Fable 5	1	99	32	32.3%	3	10	21	2	8	9.52%	38.10%	47.62%
ALE	Fable 5.1	1	99	30	30.3%	3	4	21	2	2	9.52%	9.52%	19.05%
ALE	GPT-5.6 sol	1	99	32	32.3%	5	2	24	3	0	12.50%	0.00%	12.50%
ALE	GPT-6 Astra	1	99	31	31.3%	1	1	24	1	0	4.17%	0.00%	4.17%

Table S1: Per-cohort evaluated tasks, surface-pass rates, violations and neutral verdicts across all 29 cohorts. Columns: Att. attempts per task; Tasks tasks evaluated; SP tasks tasks surface-passed; SP rate surface-pass rate; Viol. full and Neut. full violating and neutral tasks on the cohort’s own task set; $n$ shared-set tasks; Viol. of Figure 2 and Neut.of Figure S2 violating and neutral tasks on that shared set; Viol. rate, Neut. rate and Gap the corresponding rates and the integrity gap. SWEBench Pro V1.0: tasks were selected using stratified sampling from the 731 public tasks; MCP Atlas: full 500 public tasks; SWE Atlas TW: 90/90 public tasks; Terminal-Bench 2.1: 89/89 public tasks; ALE: 99/167 Ubuntu task subset of 168 public tasks. Note that task-level violation status is positive when any passing attempt is a violation, comparisons with unequal attempts may be affected by multiplicity. Here we report attempts per task to make this difference explicit.

** 奔驰**	** 模式**	** 截至2001年12月31日。 **	** 任务**	** 战略计划任务**	** SP费率**	** 全文**	**蕴藏量。 **	$n$	** Viol. (图2)**	** Neut. (图S2)**	** 数值**	** 新费率**	页:1
60)}时间轴:	缩写 4.7	3	70	66	94.3%	22	0	49	12	0	24.49%	0.00%	24.49%
SWE Bench Pro V1.0 (英语).	寓言5	1	70	52	74.3%	39	0	49	36	0	73.47%	0.00%	73.47%
S煞车Pro V1.0	寓言5.1	1	70	36	51.4%	4	0	36	4	0	11.11%	0.00%	11.11%
SWE Ben拒不放弃	GPT-5.1 编码	1	104	104	100.0%	2	29	104	2	29	1.92%	27.88%	29.81%
SWE Bench Pro V1.0 (英语).	GPT-5.6 溶剂x高	1	104	104	100.0%	71	2	104	71	2	68.27%	1.92%	70.19%
SWE Bench Pro V1.0 (英语).	GPT-6 阿斯特拉语Name	1	104	96	92.3%	0	4	96	0	4	0.00%	4.17%	4.17%
MCP 地图集	缩写 4.8	1	500	342	68.4%	0	12	295	0	10	0.00%	3.39%	3.39%
MNia 地图集	寓言5	1	500	352	70.4%	0	14	295	0	11	0.00%	3.73%	3.73%
MCP 地图集	寓言5.1	1	500	372	74.4%	1	8	295	1	6	0.34%	2.03%	2.37%
MCP 地图集	GPT-5.4 咩 G	1	500	263	52.6%	1	8	184	1	4	0.54%	2.17%	2.72%
MCP 地图集	GPT-5.5 常规	1	500	275	55.0%	7	7	184	7	3	3.80%	1.63%	5.43%
MCP 地图集	GP所遇到的问题	1	500	333	66.6%	0	3	184	0	1	0.00%	0.54%	0.54%
SWE 地图集 TW	缩写 4.8	3	90	52	57.8%	2	0	50	2	0	4.00%	0.00%	4.00%
SWE 地图集 TW	寓言5	3	90	62	68.9%	9	3	50	5	2	10.00%	4.00%	14.00%
SWE 地图集 TW	寓言5.1	1	90	50	55.6%	1	0	50	1	0	2.00%	0.00%	2.00%
S 看到了吗?	GPT-5.5 常规	3	90	55	61.1%	0	0	48	0	0	0.00%	0.00%	0.00%
SWE 地图集 TW	知道了,请回答	3	90	52	57.8%	3	4	48	2	4	4.17%	8.33%	12.50%
SWE 地图集 TW	GPT-6 阿斯特拉语Name	1	90	41	45.6%	0	0	41	0	0	0.00%	0.00%	0.00%
2.1 终点站	缩写 4.8	1	89	84	94.4%	3	1	67	1	1	1.49%	1.49%	2.99%
2.1 终点站	口号5	1	89	80	89.9%	4	0	67	3	0	4.48%	0.00%	4.48%
2.1 终点站	寓言5	1	89	83	93.3%	1	1	67	1	1	1.49%	1.49%	2.99%
2.1 终点站	寓言5.1	1	89	74	83.1%	2	2	67	2	2	2.99%	2.99%	5.97%
2.1 终点站	GPT-5.5 常规	1	89	84	94.4%	1	2	78	1	1	1.28%	1.28%	2.56%
2.1 终点站	GPT-5.6 索尔	1	89	82	92.1%	11	0	78	10	0	12.82%	0.00%	12.82%
2.1 终点站	GP 呃 6 阿斯特拉语Name	1	89	72	80.9%	0	0	68	0	0	0.00%	0.00%	0.00%
阿列尔	寓言5	1	99	32	32.3%	3	10	21	2	8	9.52%	38.10%	47.62%
阿列尔	寓言5.1	1	99	30	30.3%	3	4	21	2	2	9.52%	9.52%	19.05%
阿列尔	GPT-5.6 索尔	1	99	32	32.3%	5	2	24	3	0	12.50%	0.00%	12.50%
阿列尔	GPT-6 阿斯特拉语Name	1	99	31	31.3%	1	1	24	1	0	4.17%	0.00%	4.17%

表S1:所有29个组别中,每个组别评估的任务、地面通行率、违规行为和中立判决。 ryl列: *Att. *每个任务尝试; 任务评价任务; SP任务表面通过任务; SP率表面通过率; *Viol. full 和 Neut. * 图2 和图S2 的Viol. 违反和中立任务; SWEBench Pro V1.0:通过从731项公共任务中分层抽样选定任务;MCP Atlas:全部500项公共任务;SWE Atlas TW:90/90项公共任务;终端-Bench 2.1:89/89项公共任务;ALE:99/167 Ubuntu任务子集168项公共任务. 请注意,如果任何过往的企图都是违反,那么与不平等的企图相比较可能会受到多重性的影响。 我们在此报告,每项任务都试图明确表明这一点。

Benchmark	Model	$k$	$n$	Rate	95% Wilson	Width
SWEBench Pro V1.0	Opus 4.7	12	49	24.49%	[14.60, 38.09]	23.49
SWEBench Pro V1.0	Fable 5	36	49	73.47%	[59.74, 83.79]	24.05
SWEBench Pro V1.0	Fable 5.1	4	36	11.11%	[4.41, 25.32]	20.91
SWEBench Pro V1.0	GPT-5.1 Codex	2	104	1.92%	[0.53, 6.74]	6.21
SWEBench Pro V1.0	GPT-5.6 Sol xhigh	71	104	68.27%	[58.81, 76.43]	17.62
SWEBench Pro V1.0	GPT-6 Astra	0	96	0.00%	[0.00, 3.85]	3.85
MCP Atlas	Opus 4.8	0	295	0.00%	[0.00, 1.29]	1.29
MCP Atlas	Fable 5	0	295	0.00%	[0.00, 1.29]	1.29
MCP Atlas	Fable 5.1	1	295	0.34%	[0.06, 1.89]	1.83
MCP Atlas	GPT-5.4	1	184	0.54%	[0.10, 3.01]	2.91
MCP Atlas	GPT-5.5	7	184	3.80%	[1.85, 7.64]	5.79
MCP Atlas	GPT-6 Astra	0	184	0.00%	[0.00, 2.05]	2.05
SWE Atlas TW	Opus 4.8	2	50	4.00%	[1.10, 13.46]	12.36
SWE Atlas TW	Fable 5	5	50	10.00%	[4.35, 21.36]	17.01
SWE Atlas TW	Fable 5.1	1	50	2.00%	[0.35, 10.50]	10.15
SWE Atlas TW	GPT-5.5	0	48	0.00%	[0.00, 7.41]	7.41
SWE Atlas TW	GPT-5.6 sol	2	48	4.17%	[1.15, 13.98]	12.83
SWE Atlas TW	GPT-6 Astra	0	41	0.00%	[0.00, 8.57]	8.57
Terminal-Bench 2.1	Opus 4.8	1	67	1.49%	[0.26, 7.98]	7.72
Terminal-Bench 2.1	Opus 5	3	67	4.48%	[1.53, 12.36]	10.83
Terminal-Bench 2.1	Fable 5	1	67	1.49%	[0.26, 7.98]	7.72
Terminal-Bench 2.1	Fable 5.1	2	67	2.99%	[0.82, 10.25]	9.43
Terminal-Bench 2.1	GPT-5.5	1	78	1.28%	[0.23, 6.91]	6.68
Terminal-Bench 2.1	GPT-5.6 sol	10	78	12.82%	[7.12, 22.02]	14.90
Terminal-Bench 2.1	GPT-6 Astra	0	68	0.00%	[0.00, 5.35]	5.35
ALE	Fable 5	2	21	9.52%	[2.65, 28.91]	26.26
ALE	Fable 5.1	2	21	9.52%	[2.65, 28.91]	26.26
ALE	GPT-5.6 sol	3	24	12.50%	[4.34, 31.00]	26.66
ALE	GPT-6 Astra	1	24	4.17%	[0.74, 20.24]	19.50

Table S2: Confidence intervals plotted in panel A of Figure 2 (violations). $k$ is the number of violating tasks and $n$ the provider-shared surface-pass set; the rate is $k/n$ and the interval is the 95% Wilson score interval, which is what the whiskers in the figure show. Width is in percentage points. These are the stored values the renderer reads, not a re-derivation.

** 奔驰**	** 模式**	$k$	$n$	** 时间**	95% 威尔逊	页:1
SWE Bench Pro并制定	缩写 4.7	12	49	24.49%	[14.60、38.09]	23.49
SWE Bench Pro V1.0 (英语).	寓言5	36	49	73.47%	[59.74, 83.79] (中文(简体) ).	24.05
SWE Bench Pro V1.0 (英语).	寓言5.1	4	36	11.11%	[4.41、25.32]	20.91
SWE Bench Pro V1.0 (英语).	GPT-5.1 编码	2	104	1.92%	[是可以使用的[0.53, 6.74]	6.21
S再见 Pro V1.0	我们也希望,	71	104	68.27%	[58.81, 76.43] (中文(简体) ).	17.62
SWE Bench Pro V1.0 软件	GPT-6 阿斯特拉语Name	0	96	0.00%	[0.00, 3.85] (中文(简体) ).	3.85
MCP 地图集	缩写 4.8	0	295	0.00%	[0.00, 1.29 这里是第为实施《公约》而设的	1.29
MCP 地图等领域的	寓言5	0	295	0.00%	[0.00, 1.29]	1.29
MCP 地图集	寓言5.1	1	295	0.34%	[0.06, 1.89]	1.83
MCP 地图集	将提供以下信息:	1	184	0.54%	[0.10, 3.01]	2.91
MCP 地图集	GPT-5.5 常规在于	7	184	3.80%	[1.85, 7.64]	5.79
MCP 地图集	GPT-6 阿斯特拉语Name	0	184	0.00%	[0.0,指导]	2.05
SWE 地图集 TW	缩写 4.8	2	50	4.00%	[1.10, 13.46]	12.36
SWE 地图集 TW	寓言5	5	50	10.00%	[4.35、21.36]	17.01
SWE 地图集 TW	寓言5.1	1	50	2.00%	[0.35, 我以前说过,	10.15
SWE 地图集 TW	GPT-5.5 常规	0	48	0.00%	[0.00, 7.41]	7.41
SWE 地图集 TW	GPT-5.6 索尔	2	48	4.17%	[1.15, 13.98]	12.83
S尚未完成。	GPT-6 阿斯特拉语Name	0	41	0.00%	[0.00, 8.57]	8.57
2.1 终点站	缩写 4.8	1	67	1.49%	[0.26, 7.98]	7.72
2.1 终点站	口号5	3	67	4.48%	[1.53, 12.36]	10.83
2.1 终点站	寓言5	1	67	1.49%	[0.26, 7.98]	7.72
2.1 终点站	寓言5.1	2	67	2.99%	[0.82, 10.25]	9.43
2.1 终点站	GPT-5.5 常规	1	78	1.28%	[0.23、6.91]	6.68
2.1 终点站	GPT-5.6 索尔	10	78	12.82%	[7.12、22.02]	14.90
2.1 终点站	GPT-6 阿斯特拉语Name	0	68	0.00%	[0.00, 5.35]	5.35
阿列尔	寓言5	2	21	9.52%	[2.65, 28.91]	26.26
阿列尔	寓言5.1	2	21	9.52%	[2.65, 28.91]	26.26
的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加的加.	GPT-5.6 索尔	3	24	12.50%	[4.34, 31.00] (中文(简体) ).	26.66
阿列尔	GPT-6 阿斯特拉语Name	1	24	4.17%	[0.74, 20.24] (中文(简体) ).	19.50

表S2:图2(违规)的A面板绘制的置信间隔。 $k$是违反任务的数量,$n$是提供者共享的地表通道集;比例为$k/n$,间隔为95%的威尔逊得分间隔,这是图中显示的胡须. 宽度为百分点. 这些是渲染器读取的存储值,而不是重构值.

Benchmark	Model	$k$	$n$	Rate	95% Wilson	Width
SWEBench Pro V1.0	Opus 4.7	0	49	0.00%	[0.00, 7.27]	7.27
SWEBench Pro V1.0	Fable 5	0	49	0.00%	[0.00, 7.27]	7.27
SWEBench Pro V1.0	Fable 5.1	0	36	0.00%	[0.00, 9.64]	9.64
SWEBench Pro V1.0	GPT-5.1 Codex	29	104	27.88%	[20.17, 37.17]	17.00
SWEBench Pro V1.0	GPT-5.6 Sol xhigh	2	104	1.92%	[0.53, 6.74]	6.21
SWEBench Pro V1.0	GPT-6 Astra	4	96	4.17%	[1.63, 10.23]	8.60
MCP Atlas	Opus 4.8	10	295	3.39%	[1.85, 6.13]	4.28
MCP Atlas	Fable 5	11	295	3.73%	[2.09, 6.55]	4.46
MCP Atlas	Fable 5.1	6	295	2.03%	[0.94, 4.37]	3.43
MCP Atlas	GPT-5.4	4	184	2.17%	[0.85, 5.46]	4.61
MCP Atlas	GPT-5.5	3	184	1.63%	[0.56, 4.68]	4.12
MCP Atlas	GPT-6 Astra	1	184	0.54%	[0.10, 3.01]	2.91
SWE Atlas TW	Opus 4.8	0	50	0.00%	[0.00, 7.14]	7.14
SWE Atlas TW	Fable 5	2	50	4.00%	[1.10, 13.46]	12.36
SWE Atlas TW	Fable 5.1	0	50	0.00%	[0.00, 7.14]	7.14
SWE Atlas TW	GPT-5.5	0	48	0.00%	[0.00, 7.41]	7.41
SWE Atlas TW	GPT-5.6 sol	4	48	8.33%	[3.29, 19.55]	16.26
SWE Atlas TW	GPT-6 Astra	0	41	0.00%	[0.00, 8.57]	8.57
Terminal-Bench 2.1	Opus 4.8	1	67	1.49%	[0.26, 7.98]	7.72
Terminal-Bench 2.1	Opus 5	0	67	0.00%	[0.00, 5.42]	5.42
Terminal-Bench 2.1	Fable 5	1	67	1.49%	[0.26, 7.98]	7.72
Terminal-Bench 2.1	Fable 5.1	2	67	2.99%	[0.82, 10.25]	9.43
Terminal-Bench 2.1	GPT-5.5	1	78	1.28%	[0.23, 6.91]	6.68
Terminal-Bench 2.1	GPT-5.6 sol	0	78	0.00%	[0.00, 4.69]	4.69
Terminal-Bench 2.1	GPT-6 Astra	0	68	0.00%	[0.00, 5.35]	5.35
ALE	Fable 5	8	21	38.10%	[20.75, 59.12]	38.37
ALE	Fable 5.1	2	21	9.52%	[2.65, 28.91]	26.26
ALE	GPT-5.6 sol	0	24	0.00%	[0.00, 13.80]	13.80
ALE	GPT-6 Astra	0	24	0.00%	[0.00, 13.80]	13.80

Table S3: Confidence intervals plotted in panel A of Figure S2 (neutral verdicts). Same cohorts, same denominators $n$ as Table S2; here $k$ counts tasks whose effective verdict is neutral. Intervals are 95% Wilson score intervals. Rows where $k=0$ still carry a non-zero upper bound, set by $n$ alone.

** 奔驰**	** 模式**	$k$	$n$	** 时间**	95%威尔逊	页:1
SWE Bench Pro V1.0 (英语).	缩写 4.7	0	49	0.00%	[0.00, 7.27]	7.27
SWE Bench Pro V1.0 (英语).	寓言5	0	49	0.00%	[0.00, 7.27]	7.27
SWE Bench Pro V1.0 (英语).	寓言5.1	0	36	0.00%	[0.00、9.64]	9.64
SWE Bench Pro V1.0 (英语).	GPT-5.1 编码	29	104	27.88%	[20.17、37.17]	17.00
SWEB集中注意力 Pro V1.0	GPT-5.6 溶剂x高	2	104	1.92%	[0.53, 6.74]	6.21
SWE Bench Pro V1.0 (英语).	GPT-6 阿斯特拉语Name	4	96	4.17%	[1.63, 10.23]	8.60
MCP 地图leals 器i Atlas M Atlas Atlas Atlas Atlas Atlas Atlas Atlas Atlas Atlas	缩写 4.8	10	295	3.39%	[1.85, 6.13]	4.28
MCP 地图集	寓言5	11	295	3.73%	[2.09,馅饼6.55]	4.46
MCP 地图集	寓言5.1	6	295	2.03%	[0.94, 4.37] (中文(简体) ).	3.43
MCP 地图集	GPT-5.4 常规	4	184	2.17%	[0.85, 5.46]	4.61
MCP 地图集	GPT-5.5 常规	3	184	1.63%	[0.56, 4.68]	4.12
MCP 地图集	GPT-6 阿斯特拉语Name	1	184	0.54%	公民资格	2.91
SWE 地图集 TW	缩写 4.8	0	50	0.00%	[0.00, 7.14]	7.14
SWE 地图集 TW	寓言5	2	50	4.00%	[1.10, 13.46]	12.36
SWE 地图集 TW	精灵5.1	0	50	0.00%	[0.00, 7.14]	7.14
SWE 地图集 TW	GPT-5.5 常规	0	48	0.00%	[0.00, 7.41]	7.41
SWE 地图集 TW	GPT-5.6 索尔	4	48	8.33%	[3.29、19.55]	16.26
SWE 地图集 TW	GPT-6 阿斯特拉语Name	0	41	0.00%	[0.00, 8.57]	8.57
2.1 终点站	缩写 4.8	1	67	1.49%	[0.26, 7.98]	7.72
2.1 终点站	口号5	0	67	0.00%	[0.00, 5.42]	5.42
2.1 终点站	寓言5	1	67	1.49%	[0.26, 7.98]	7.72
2.1 终点站	寓言5.1	2	67	2.99%	[0.82, 10.25]	9.43
2.1 终点站	GPT-5.5 常规	1	78	1.28%	[1, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,	6.68
2.1 终点站	GPT-5.6 索尔	0	78	0.00%	[0.0, 4.69] (中文(简体 以下是) ).	4.69
2.1 终点站	GPT-6 阿斯特拉语Name	0	68	0.00%	[0.00, 5.35]	5.35
阿列尔	寓言5	8	21	38.10%	[20.75, 59.12] (中文(简体) ).	38.37
阿列尔	寓言5.1	2	21	9.52%	[2.65, 28.91]	26.26
阿列尔	GPT-5.6 索尔	0	24	0.00%	[0.00, 13.80]	13.80
阿列尔	GPT-6 阿斯特拉语Name	0	24	0.00%	[0.00, 13.80]	13.80

表S3: 图S2(中性判决)A小组绘制的置信间隔图。 与表S2相同的组和分母为$n$;这里的$k$是有效判断中性的任务. 间隔时间为95%的威尔逊得分间隔. $k=0$仍然携带非零上界的行,单由$n$设定.

Appendix B Catalog of violation cases

附录 B 违规案例目录

Category definitions are listed once below, then the cases follow grouped by label.

类别定义列于下方，随后按标签分组给出案例。

Violation-gate note. The pinned judge uses category-specific controls before suspicious conduct can be escalated to unearned violation. An effectiveness gate requires a known surface pass, a clearly evidenced attempt, an actual effect on verifier or reward behavior, and grade relevance. An awareness + evaluation-relevance gate requires direct evidence that the agent registered the relevant failure or mismatch and used the resulting action or success claim in the evaluated path. If the named gate is not met, the case is not a violation and must be classified under the applicable neutral or legitimate rule.

违规门控说明。 固定裁判在标记确认违规前使用类别特定控制；未应得（中性）标签用于证据不足或核验器弱点情形，不要求机制—意图—评分相关性三联齐全。

B.1 Unearned Violations

B.1 未应得违规

Each case row gives the benchmark, agent model, harness, task name, the evidence, and the primary and secondary judge consensus. Quoted text is verbatim from the trajectory.

每行给出基准、智能体模型、harness、任务名、证据摘要与通道。

B.2 Unearned cases

B.2 未应得案例