SWE-Residual: Evaluating Starting-State Robustness of Coding Agents

SWE-Residual：评测编程智能体的起始状态鲁棒性（SWE-Residual: Evaluating Starting-State Robustness of Coding Agents）

Jinze Li Affiliation:  The University of Hong Kong   Email:  lijinze-hku@connect.hku.hk    Weijie Liu Affiliation:  The University of Hong Kong   Email:  liuwj0817@connect.hku.hk    Jinfeng Xu Affiliation:  The University of British Columbia Email:  shuo.yang@connect.hku.hk    Shuo Yang Affiliation:  The University of Hong Kong   Email:  zifeishan@tencent.com    Xin Yang Affiliation:  University of Tokyo   Email:  ethanyjwang@tencent.com    Yang Zhang Affiliation:  University of North Texas Email:  jinfeng.xu@ubc.ca    Yijun Wang Affiliation:  Tencent   Email:  xin.yang@weblab.t.u-tokyo.ac.jp    Zifei Shan † † thanks: Corresponding authors. Affiliation:  Tencent   Email:  yang.zhang@unt.edu    Edith C.H. Ngai † † footnotemark: Affiliation:  The University of Hong Kong   Email:  chngai@eee.hku.hk

Abstract

摘要

Software engineering (SWE) benchmarks are the primary way to measure the capabilities of coding agents: an agent is placed in a code repository or machine environment, given a request, and judged by hidden tests. These benchmarks almost always start the agent from a clean state that contains no earlier work on the task. In practice, however, agents often continue work that a developer or another agent has already done and reported as complete, with no indication of what remains incorrect or missing. We introduce SWE-Residual, a benchmark for Starting-State Robustness: the ability to maintain performance on a task when the starting state already contains earlier work on that task. SWE-Residual contains 198 Residual States drawn from 111 tasks in DeepSWE and Terminal-Bench, where the earlier work modifies a code repository or a whole machine environment. Each Residual State is the state a predecessor agent left behind after submitting its work as complete, although the hidden tests still fail. The evaluated agent receives only the original request and this state, without the predecessor’s trajectory or any sign that the attempt failed; because the request and hidden tests are unchanged, the starting state is the only difference from the source benchmark. Across five frontier agents, success drops by 20.6–53.5 percentage points on DeepSWE and 20.6–34.9 points on Terminal-Bench relative to a Clean Start, and agent rankings on DeepSWE reverse. Failed continuations show a pattern we call acceptance bias: agents treat the inherited work as the definition of what remains to be done instead of checking it against the request. On DeepSWE, in 58.9–88.2% of failed runs, they submit with exactly the same tests failing, although they pass most of these tests from a Clean Start. SWE-Residual thus exposes a capability gap that clean-start evaluation does not capture. Codes are available at https://github.com/userljz/SWE-Residual.

软件工程（SWE）基准是衡量编程智能体能力的主要方式：把智能体放进代码仓库或整机环境，给出请求，用隐藏测试判定。这些基准几乎总是从干净状态起步——不含该任务的任何既有工作。实践中，智能体却常接着开发者或另一智能体已声称完成的工作继续干，且得不到“还缺什么、错在哪”的提示。我们提出 SWE-Residual，评测起始状态鲁棒性（Starting-State Robustness）：当起始状态已含该任务的既有工作时，能否保持表现。基准含 198 个残留状态，来自 DeepSWE 与 Terminal-Bench 的 111 个任务；既有工作分别改仓库或整机环境。每个残留状态是前任智能体把工作提交为完成后留下的状态，尽管隐藏测试仍失败。被评智能体只收到原请求与该状态，看不到前任轨迹，也得不到失败信号；请求与隐藏测试不变，故与源基准的唯一差别是起始状态。五个前沿智能体上，相对干净起步，DeepSWE 成功率下降 20.6–53.5 个百分点，Terminal-Bench 下降 20.6–34.9 个百分点，且 DeepSWE 上排名翻转。失败续作呈现我们称为**接受偏置（acceptance bias）**的模式：智能体把继承工作当作“还剩什么要做”的定义，而不对照请求检查。DeepSWE 失败运行中 58.9–88.2% 最终失败的测试集合与起步时完全相同，尽管同一智能体从干净起步能通过其中多数测试。SWE-Residual 因而暴露干净起步评测捕捉不到的能力缺口。代码：https://github.com/userljz/SWE-Residual。

1 Introduction

1 引言

Coding agents are language models that read files, run commands, and edit code, and they are now used for software tasks that take hours of work (Yang et al., 2024; Deng et al., 2025; Huang et al., 2026). As these agents have become more capable, a growing number of benchmarks have been built to measure their capabilities, from resolving issues in code repositories to operating a whole machine through the command line (Jimenez et al., 2024; Huang et al., 2026; Merrill et al., 2026). These benchmarks differ in their tasks, but they evaluate an agent in the same way. The agent is placed in a starting state, such as a code repository or a machine, and given a request; it works until it declares the task done, and it succeeds if a set of hidden tests then passes. Scores obtained in this way are now the main evidence of how capable a coding agent is.

编程智能体是能读文件、跑命令、改代码的语言模型，已被用于耗时数小时的软件任务（Yang et al., 2024；Deng et al., 2025；Huang et al., 2026）。随其变强，越来越多基准被用来测能力——从解决仓库级 issue 到经命令行操作整机（Jimenez et al., 2024；Huang et al., 2026；Merrill et al., 2026）。任务不同，评法却一致：把智能体放进起始状态，给请求，用智能体看不到的隐藏测试判定最终状态。

However, these benchmarks typically start the agent from a clean initial state, one that contains no earlier work on the task the agent is given. We call this a Clean Start. Yet in practice, agents often start from states that already contain earlier work on the same task. Development is routinely handed over between developers (Khatoonabadi et al., 2023), and increasingly between agents: a user may return in a new session to work that an earlier agent declared done, without the earlier session’s context, and in multi-agent systems one agent builds on output that another has submitted as final. In each case, the agent inherits the artifacts of earlier work, such as modified code, newly created files, or installed software. Agents frequently declare success on tasks they have not completed (Advani, 2026), so the work handed to the next agent or session often looks finished but is not. We study handoffs in which this work was submitted as complete but does not satisfy the request, and the agent receives no indication of what is incomplete or incorrect: it must work out from the state alone what has been done and what is still missing.

然而这些基准通常从干净初始状态起步，不含该任务的任何既有工作。我们称之为干净起步（Clean Start）。实践中智能体却常从已含同任务既有工作的状态开始。开发在开发者之间交接是常态（Khatoonabadi et al., 2023），智能体之间也日益如此：用户可能在新会话回到更早智能体声称完成的工作，却没有更早会话上下文；多智能体系统中一个智能体建立在另一个已提交为完成的输出之上。

We call the ability to complete a task from a state that already contains such earlier work Starting-State Robustness. Most benchmarks cannot measure this ability, because a Clean Start never presents the agent with earlier work, so two agents with the same benchmark score may still differ widely in whether they can finish what someone else has started, and, as we show, their ranking can even reverse. This ability cannot be taken for granted. On one DeepSWE task, for example, an agent inspects the inherited changes, runs the package tests, and submits without changing a single file, although the same agent passes the 11 failing hidden tests from a Clean Start.

我们把从已含此类既有工作的状态完成任务的能力称为起始状态鲁棒性。多数基准测不到它，因为干净起步从不呈现既有工作，于是两智能体基准分数相同，接手他人未竟工作的能力仍可能天差地别；我们还将表明排名甚至会翻转。这一能力不能想当然。例如在某 DeepSWE 任务上，智能体检查继承改动、跑包测试后一个文件都未改就提交，尽管隐藏测试仍失败。

Figure 1: Left: Standard benchmarks evaluate an agent from the original repository (Clean Start). SWE-Residual evaluates the same agent on the same task from the repository left by a predecessor agent that submitted its work as complete but failed the hidden tests (Residual Start), with the same request and hidden tests and no information about the earlier attempt. Right: Success rates of the five evaluated agents under both settings, averaged over DeepSWE and Terminal-Bench.

图 1：左：标准基准从原始仓库（干净起步）评测智能体。SWE-Residual 在同一任务上从前任智能体留下的仓库评测（残留起步）——前任把工作提交为完成但隐藏测试失败；请求与隐藏测试相同，且无更早尝试信息。右：五个被评智能体在两种设置下的成功率，对 DeepSWE 与 Terminal-Bench 取平均。

To address this gap, we introduce SWE-Residual, a benchmark for Starting-State Robustness, as shown in Figure 1. A Residual State is the state a predecessor agent leaves behind after submitting its work on a task from an existing benchmark as complete, although the hidden tests still fail; starting an agent from it is a Residual Start. Such a state looks finished: the work is coherent, often passes the predecessor’s own checks, and on DeepSWE the median Residual State already passes 98.4% of the hidden tests; yet parts of the request remain missing or incorrectly implemented, and the evaluated agent must determine which parts of the apparent solution to trust and what work remains. Each instance of SWE-Residual pairs a Residual State with the original request and hidden tests. Because these are unchanged, the same agent can attempt each task from a Clean Start and from a Residual Start. We report success from Residual Starts and the change from Clean Start to Residual Start on the same tasks.

为填补该缺口，我们提出面向起始状态鲁棒性的基准 SWE-Residual（图 1）。残留状态（Residual State）是前任智能体在既有基准任务上把工作提交为完成后留下的状态，尽管隐藏测试仍失败；从它起步称为残留起步（Residual Start）。这种状态看起来像做完了：工作连贯，常能通过前任自检，在 DeepSWE 上中位残留状态已通过 98.4% 的隐藏测试；但请求仍有缺失或错误实现，被评智能体必须发现并修。

We build SWE-Residual on DeepSWE, where the inherited work modifies a code repository, and Terminal-Bench, where it modifies a machine’s files, installed software, and configuration, using Residual States left by predecessor agents from three model families (DeepSeek, Qwen, and GLM). Across five frontier agents, success drops by 20.6–53.5 percentage points on DeepSWE and 20.6–34.9 points on Terminal-Bench relative to Clean Start, and agent rankings on DeepSWE reverse, so strong clean-start performance does not guarantee Starting-State Robustness. We observe a recurring pattern we call acceptance bias: the evaluated agent treats the inherited work as the definition of what remains to be done, instead of checking it against the original request. It inspects the inherited changes but keeps most of them and rarely changes the files the predecessor left out. In 58.9–88.2% of failed continuations on DeepSWE, exactly the same tests fail at the end as at the start, even though the same agent passes most of these tests from a Clean Start.

我们在 DeepSWE（继承工作改仓库）与 Terminal-Bench（继承工作改机器文件、已装软件与配置）上构建 SWE-Residual，残留状态来自 DeepSeek、Qwen、GLM 三族前任智能体。五个前沿智能体上，相对干净起步，DeepSWE 成功率下降 20.6–53.5 个百分点，Terminal-Bench 下降 20.6–34.9 个百分点，且 DeepSWE 排名翻转，故强干净起步表现不保证起始状态鲁棒性。我们观察到反复出现的接受偏置：被评智能体把继承工作当作“还剩什么要做”的定义，而不对照原请求检查。它会查看继承改动，但大多保留，很少改前任漏掉的文件。DeepSWE 失败续作中 58.9–88.2% 最终失败测试集与起步时完全相同，尽管同一智能体从干净起步能通过其中多数。

In summary, our contributions are as follows.

贡献总结如下。

• • We identify Starting-State Robustness, the ability to maintain performance on a task when the starting state already contains earlier work on that task, as an evaluation dimension that complements conventional clean-start evaluation.
• • We release SWE-Residual, comprising 198 Residual States produced by predecessor agents from three model families (DeepSeek, Qwen, and GLM) on tasks from DeepSWE and Terminal-Bench. We also provide Residualization, a pipeline for constructing and validating these states that enables the benchmark to be extended to additional predecessor agents and source benchmarks.
• • Across five frontier agents, success drops substantially from a Clean Start to Residual States, and failed continuations show a pattern we call acceptance bias: agents treat the inherited work as the definition of what remains to be done instead of checking it against the original request.

• 我们提出起始状态鲁棒性——当起始状态已含该任务既有工作时保持表现的能力——作为补充传统干净起步评测的维度。
• 我们发布 SWE-Residual，含 DeepSeek、Qwen、GLM 三族前任在 DeepSWE 与 Terminal-Bench 任务上产生的 198 个残留状态，并提供 Residualization 管线以便扩展到更多前任与源基准。
• 五个前沿智能体上，从干净起步到残留状态成功率大幅下降；失败续作呈现接受偏置：智能体把继承工作当作待办定义，而不对照原请求检查。

2 Related Work

2 相关工作

Coding-agent benchmarks.

Rapid progress in LLMs has spurred substantial research on coding agents for automated software engineering. Such agents are commonly evaluated by placing them in a predefined software environment, providing a task specification, and evaluating whether their resulting changes satisfy execution-based tests. The SWE-bench series (Jimenez et al., 2024; Deng et al., 2025) established this paradigm for resolving real-world repository-level issues, and subsequent benchmarks have extended it to more programming languages (Zan et al., 2025; Rashid et al., 2025; Guo et al., 2025), multimodal inputs (Yang et al., 2025), and longer-horizon tasks (Huang et al., 2026; Zhou et al., 2026; Miserendino et al., 2025; Le et al., 2026). Beyond code repositories, Terminal-Bench (Merrill et al., 2026) extends agent evaluation to whole machine environments in which agents must manipulate files, dependencies, configurations, and command-line tools. A related line gives an agent a sequence of requests on one repository and studies how the code accumulated for earlier requests affects later ones (Chen et al., 2026; Lam et al., 2026; Shastry et al., 2026; Orlanski et al., 2026). Despite differing in task scope and environment, these benchmarks share the assumption that each task begins from a clean starting state containing no prior work on the task itself. This leaves performance on states modified by an earlier attempt at the same task unmeasured.

编程智能体基准

LLM 的快速进展推动了面向自动软件工程的编程智能体研究。常见评法是把智能体放进预定义软件环境，给出任务规格，用基于执行的测试评估其改动是否满足。SWE-bench 系列（Jimenez et al., 2024；Deng et al., 2025）为解决真实仓库级 issue 确立了这一范式，后续基准扩展到更多语言与设定。

Benchmarks beyond clean-start evaluation.

Recent research has begun to relax the clean-start assumption by evaluating coding agents in code or machine environments already modified by prior attempts. Handoff Debt (KC and Budathoki, 2026) and Handoff Tax (Ganz et al., 2026) study takeover from intermediate agent checkpoints, examining how different forms of predecessor context affect the handoff cost of resuming a task. OmniCode (Sonwane et al., 2026) and FeedbackEval (Dai et al., 2025) assess whether agents can comprehend and repair code given explicit compiler, test, or LLM feedback. Recovery-Bench (Tan et al., 2025) replays failed Terminal-Bench runs of a weaker agent and evaluates whether another agent can recover from the resulting environment, finding that more context from the failed attempt can hurt. FixedBench (Gloaguen et al., 2026) studies the reverse situation, in which the reported issue is already resolved, and finds that agents often make unnecessary changes. Together, these works show that prior attempts can substantially affect subsequent agent behavior and completion cost. Table 1 summarizes how their settings differ from SWE-Residual, which studies a handoff after a predecessor has submitted its work as complete, although the original task remains unsolved, and pairs each Residual State with a Clean Start of the same agent on the same task.

超越干净起步的基准

近期工作开始放松干净起步假设，在已被先前尝试修改的代码或机器环境中评测编程智能体。Handoff Debt（KC and Budathoki, 2026）与 Handoff Tax（Ganz et al., 2026）研究从中间智能体检查点接手，考察不同形式的前任上下文如何影响续作成本。OmniCode（Sonwane et al., 2026）与 FeedbackEval（Dai et al., 2025）评估智能体在给定显式编译器、测试或 LLM 反馈时能否理解并修复代码。

Relation to Recovery-Bench.

Recovery-Bench asks whether an agent can recover from a failed attempt: its states come from failed Terminal-Bench runs, whether or not the predecessor submitted its work, and the task is framed as recovery from that attempt. SWE-Residual instead keeps only final states that were submitted as complete and still fail the original verifier; the submission is a selection criterion, and the evaluated agent receives only the request and the inherited state, without the submission or the verdict. The question therefore shifts from recovering after a known failure to finding, in work that appears done, what the original request still requires.

与 Recovery-Bench 的关系

Recovery-Bench 问智能体能否从失败尝试恢复：其状态来自失败的 Terminal-Bench 运行，不论前任是否提交，任务被框为从该尝试恢复。SWE-Residual 只保留提交为完成且仍使原核验器失败的最终状态；提交是筛选准则，被评智能体只收到请求与继承状态，看不到提交或裁决。问题因此从“已知失败后的恢复”转为“在看似完成、且无失败信号的状态下继续”。

Why agents accept flawed work.

Our analysis of acceptance bias relates to known limitations of language models. Without external feedback, models often fail to correct their own reasoning (Huang et al., 2024; Kamoi et al., 2024), and they locate errors less reliably than they fix them once the location is given (Tyen et al., 2024). They also struggle to notice what is missing from an input (Fu et al., 2025), and they tend to build on earlier mistakes instead of revisiting them (Zhang et al., 2024; Laban et al., 2026), even declaring success on tasks they have not completed (Advani, 2026). Human developers who reuse an existing artifact show a similar anchoring effect (Parsons and Saunders, 2004). SWE-Residual measures what these tendencies cost when one agent continues another agent’s work.

为何智能体接受有缺陷的工作

我们对接受偏置的分析关联语言模型已知局限。无外部反馈时，模型常无法纠正自身推理（Huang et al., 2024；Kamoi et al., 2024），定位错误也不如在给定位置后修复可靠（Tyen et al., 2024）。它们也难以注意输入中缺失什么（Fu et al., 2025），并倾向于建立在更早错误之上而非回头（Zhang et al., 2024；Laban et al., 2026），甚至对未完成任务宣称成功（Advani, 2026）。

Benchmark	Environment	Prior work	Submitted as done	Submitted	as done	Failed	Blind continuation	Blind	continuation
Submitted
as done
Blind
continuation
SWE-bench (Jimenez et al., 2024)	Repository	✗	–	–	–
Terminal-Bench (Merrill et al., 2026)	Machine	✗	–	–	–
Handoff Debt (KC and Budathoki, 2026)	Repository	✓	✗	✗	✗
Handoff Tax (Ganz et al., 2026)	Repository	✓	✗	✗	✗
FeedbackEval (Dai et al., 2025)	Repository	✓	✗	✓	✗
OmniCode (Sonwane et al., 2026)	Repository	✓	✗	✓	✗
Recovery-Bench (Tan et al., 2025)	Machine	✓	✗	✓	✗
SWE-Residual (ours)	Repository & Machine	✓	✓	✓	✓

Submitted
as done

Blind
continuation

Table 1: A comparison of SWE-Residual to existing coding agent benchmarks. Prior work: the starting state contains residual artifacts from another agent’s earlier attempt. Submitted as done: the predecessor submitted the prior work as complete, rather than being interrupted or selected without regard to submission. Failed: the prior work does not fully satisfy the task specification. Blind continuation: the evaluated agent receives only the resulting state and task request, with no explicit feedback, trajectory, or results of the prior attempt. ✓ yes, ✗ no, – not applicable.

基准	环境	先前的工作	已提交	已提交	已执行	在其任务范围内失败	盲人继续	盲人	继续
已提交
已执行
盲人
继续
SWE-bench(Jimenez等人,2024年)	仓库	✗	- ――	- ――	- ――
终点-奔驰(Merrill等人,2026年)	机器	✗	- 说吧	- 说吧	- 说吧
握手债务(KC和Budathoki,2026年)	仓库	✓	✗	✗	✗
手税(Ganz等人,2026年)	仓库	也受到打击	✗	✗	✗
梢(Dai等人,2025年)	仓库	✓	✗	✓	✗
OmniCode(Sonwane等人,2026年)	仓库	✓	✗	✓	✗
恢复-奔驰(Tan等人,2025年)	机器	✓	✗	✓	✗
** SWE-Residual(我们)**	仓库机器( M)	✓	✓	✓	✓

已提交
已执行

盲人
继续

表1:SWE与现有编码代理基准的比较。 * 优先工作*:起始状态包含另一个智能体早先尝试的残余文物。 * 照此提交*:前任提交了完整的先前的工作,而不是在不考虑提交的情况下中断或选定。 * 失败*:先前的工作不能完全满足任务要求。 *Blind continuation *:被评估的代理只收到所得出的状态和任务请求,没有事先尝试的明确反馈,轨迹,或结果. 是的,没有,——不适用。

3 SWE-Residual

This section formalizes the SWE-Residual evaluation setting and describes how we construct and validate its starting states.

本节形式化 SWE-Residual 评测设定，并描述如何构造与验证起始状态。

3.1 Task Formulation

3.1 任务形式化

Clean Start.

An evaluation of task $T_{i}$ consists of a request $q_{i}$, a starting state $S_{i}$, and a verifier $V_{i}$. The starting state describes the environment that the agent can observe and modify when it begins: in DeepSWE, this is the code repository; in Terminal-Bench, it is the entire machine environment, including files, installed software, configurations, and running processes. The verifier $V_{i}$ evaluates the final state using a set of hidden tests that are never exposed to the agent.

干净起步

对任务 $T_{i}$ 的评测由请求 $q_{i}$、起始状态 $S_{i}$ 与核验器 $V_{i}$ 组成。起始状态描述智能体开始时可观察与修改的环境：在 DeepSWE 是代码仓库；在 Terminal-Bench 是整机环境，含文件、已装软件、配置与运行中进程。核验器 $V_{i}$ 用永不暴露给智能体的隐藏测试集评估最终状态。

For an agent $A$, let $F_{A}(q_{i},S_{i})$ denote the final state produced by agent $A$ in a run when the agent either declares the task complete or exhausts the evaluation budget. We define the outcome of the run as

对智能体 $A$，令 $F_{A}(q_{i},S_{i})$ 表示智能体在一次运行中声明任务完成或耗尽评测预算时产生的最终状态。运行结局定义为

$$y_{A,i}(S_{i})=V_{i}\left(q_{i},F_{A}(q_{i},S_{i})\right)\in\{0,1\}, \tag{1}$$

where $1$ indicates success and $0$ otherwise. We denote the clean starting state of task $i$ by $S_{0,i}$, which contains no work from a prior attempt on $q_{i}$. A run under the standard clean-start setting therefore has outcome $y_{A,i}(S_{0,i})$.

其中 $1$ 表示成功，$0$ 否则。记任务 $i$ 的干净起始状态为 $S_{0,i}$，不含对 $q_{i}$ 的先前尝试工作。标准干净起步设置下的结局即为 $y_{A,i}(S_{0,i})$。

Residual States.

A Residual State is the final state left by a prior attempt at the task that was submitted as complete but failed. Let $\tilde{A}$ denote the predecessor agent that attempts task $T_{i}$ from the clean state $S_{0,i}$. We define a Residual State as

残留状态

残留状态是先前尝试把任务提交为完成但失败后留下的最终状态。令 $\tilde{A}$ 为从干净状态 $S_{0,i}$ 尝试任务 $T_{i}$ 的前任智能体。残留状态定义为

$$S_{r,i}=F_{\tilde{A}}(q_{i},S_{0,i}),\quad\text{where $\tilde{A}$ declares the task complete and}\ y_{\tilde{A},i}(S_{0,i})=0. \tag{2}$$

That is, $S_{r,i}$ is the state left by the predecessor agent $\tilde{A}$ that attempted the same request $q_{i}$ and submitted its work as complete, but did not pass the hidden tests.

亦即，$S_{r,i}$ 是前任 $\tilde{A}$ 对同一请求 $q_{i}$ 尝试后留下的状态：它把工作提交为完成，但未通过隐藏测试。

Starting-State Robustness.

We use Starting-State Robustness to refer to an agent’s ability to maintain its performance when the same request $q_{i}$ is attempted from a Residual State $S_{r,i}$ rather than the clean state $S_{0,i}$. The verifier is held fixed, so the starting state is the only change.

起始状态鲁棒性

起始状态鲁棒性指智能体在从残留状态 $S_{r,i}$ 而非干净状态 $S_{0,i}$ 尝试同一请求 $q_{i}$ 时保持表现的能力。核验器固定，唯一起始状态改变。

3.2 Benchmark Construction

3.2 基准构造

We call our process for constructing validated Residual States from attempts by predecessor agents Residualization. Residualization consists of four stages. We apply this process to DeepSWE (Huang et al., 2026) and Terminal-Bench (Merrill et al., 2026), while keeping each task’s request and verifier unchanged.

我们把从前任智能体尝试构造并验证残留状态的过程称为 Residualization，分四阶段。应用于 DeepSWE（Huang et al., 2026）与 Terminal-Bench（Merrill et al., 2026），各任务请求与核验器保持不变。

Stage I: Trajectory generation.

We use three predecessor agents based on DeepSeek-V4-Flash-0731, Qwen-3.8-27B, and GLM-5.3: recent open-weight models of different sizes that are inexpensive to run at scale and capable enough that their work resembles what agents hand over in practice. For each task $T_{i}$, a predecessor agent $\tilde{A}$ starts from the clean state $S_{0,i}$ and attempts the request $q_{i}$ under a generous time limit of eight hours per run, producing a final state $F_{\tilde{A}}(q_{i},S_{0,i})$ with outcome $y_{\tilde{A},i}(S_{0,i})$. For each run, we record the full trajectory, termination reason, whether the predecessor agent declared completion, and the verifier outcome, together with the information needed to reconstruct its final state: a binary-safe patch of all repository changes on DeepSWE, and the ordered sequence of terminal interactions on Terminal-Bench.

阶段 I：轨迹生成

我们使用基于 DeepSeek-V4-Flash-0731、Qwen-3.8-27B 与 GLM-5.3 的三个前任智能体：不同规模的近期开权模型，规模化运行成本低，且能力足以使其工作接近实践中的交接物。对每个任务 $T_{i}$，前任 $\tilde{A}$ 从干净状态 $S_{0,i}$ 出发，在每跑八小时的宽裕时限下尝试请求 $q_{i}$，产生最终状态与结局。我们记录完整轨迹以便后续审查。

Stage II: Automatic filtering.

Not every generated run yields a suitable candidate Residual State: some make no persistent changes, some terminate because of infrastructure failures, and others exhaust the time limit before the agent declares the task complete. We therefore apply rule-based filtering before detailed human review. For each generated run, we retain only those that satisfy three criteria: (a) Submitted but failing: the predecessor agent declares the task complete, but $y_{\tilde{A},i}(S_{0,i})=0$. (b) Changed: the final state $F_{\tilde{A}}(q_{i},S_{0,i})$ differs from $S_{0,i}$ after removing incidental execution artifacts such as build outputs, caches, and logs. (c) No infrastructure failure: the run does not terminate because of failures unrelated to the task, such as model-request errors, container crashes, or unavailable network resources. Duplicate final states produced for the same task are retained only once.

阶段 II：自动过滤

并非每次生成运行都产生合适候选：有的无持久改动，有的因基础设施失败终止，有的在声明完成前耗尽时限。因此在人工细审前做基于规则的过滤，仅保留满足三项准则的运行：（a）已提交但失败：前任声明完成但 $y=0$。（b）有改动：最终状态相对干净状态有差异。（c）非基础设施失败：排除环境崩溃等。

Stage III: Human review.

Agent trajectories can span hundreds of steps, making direct step-by-step inspection impractical. Two human reviewers therefore independently assess each candidate using the request $q_{i}$, the changes from the clean state $S_{0,i}$ to the candidate state $F_{\tilde{A}}(q_{i},S_{0,i})$, and a GLM-5.3 summary of the predecessor trajectory that describes the intentions behind the agent’s actions and its overall solution strategy. A candidate is retained only if it satisfies two criteria: (a) Task-directed: the summarized strategy reflects a coherent attempt to satisfy the requirements in $q_{i}$, rather than unrelated or incidental changes. (b) Task–verifier consistent: the request and hidden tests specify consistent requirements for task completion.

阶段 III：人工审查

智能体轨迹可跨数百步，逐步检查不现实。两名人工审阅者独立评估每个候选，材料包括请求 $q_{i}$、从干净状态到候选的改动，以及 GLM-5.3 对前任轨迹的摘要（意图与总体策略）。仅当满足：（a）任务导向：摘要策略反映对请求的连贯尝试；（b）可接手：状态对未见前任轨迹的智能体可理解且可继续。

Stage IV: State reconstruction.

The retained candidates are saved artifacts rather than reusable environments, so we reconstruct each one from the clean state $S_{0,i}$ and store it in a reproducible form. On DeepSWE, we apply the saved patch at the task’s base commit without creating a commit, so the evaluated agent observes the modified working tree without the commit history that produced it. On Terminal-Bench, we replay the recorded interactions in a fresh task environment and checkpoint the resulting machine state, including live processes. We validate that each reconstructed state remains usable and that the original verifier still returns $V_{i}(q_{i},S_{r,i})=0$ (Appendix B). During evaluation, each rollout starts from an independent instance of $S_{r,i}$ and receives only the original request $q_{i}$; the prior trajectory, predecessor outputs, and failure information are not explicitly supplied. We refer to this protocol as blind continuation.

阶段 IV：状态重建

保留候选是保存的产物而非可复用环境，故从干净状态 $S_{0,i}$ 重建并以可复现形式存储。DeepSWE 上在任务基提交应用保存补丁但不创建提交，使被评智能体看到修改后工作树而无产生它的提交历史。Terminal-Bench 上在新鲜任务环境重放记录交互并检查点机器状态（含活动进程）。验证每个重建状态仍使原核验器失败，并与过滤时的最终状态一致。

4 Benchmark Characteristics

4 基准特征

SWE-Residual contains 198 validated Residual States derived from 111 tasks across DeepSWE and Terminal-Bench. Each state comes from a prior attempt submitted as complete, passes our inclusion review, and is verified to fail the original task verifier after reconstruction. Table 2 summarizes the benchmark’s coverage, predecessor composition, prior-attempt effort, and initial test outcomes.

SWE-Residual 含 198 个经验证残留状态，来自 DeepSWE 与 Terminal-Bench 共 111 个任务。每个状态来自已提交为完成的先前尝试，通过纳入审查，并在重建后验证仍使原任务核验器失败。表 2 汇总覆盖面、前任构成、先前尝试投入与初始测试结果。

	DeepSWE	Terminal-Bench
Source benchmark tasks	113	89
Source tasks covered	90	21
Residual States	165	33
from DeepSeek	62	17
from Qwen	72	11
from GLM	31	5
Tasks with one / two / three states	31 / 43 / 16	11 / 8 / 2
Prior-attempt length (steps)	134	24
Predecessor execution time (min)	47.7	15.5
Test pass rate (mean / median, %)	91.6 / 98.4	47.6 / 60.0
States with 1–3 tests still failing	88 (53.3%)	33 (100%)

Table 2: Statistics of SWE-Residual. Task counts refer to unique source tasks. Prior-attempt steps and execution time are medians over retained states; steps count recorded agent turns. Test pass rates include all tests used for task acceptance and are summarized with equal weight per state.

	DeepSWE	Terminal-Bench
源基准任务数	113	89
覆盖源任务数	90	21
残留状态数	165	33
来自 DeepSeek	62	17
来自 Qwen	72	11
来自 GLM	31	5
一/二/三个状态的任务数	31 / 43 / 16	11 / 8 / 2
先前尝试长度（步，中位）	134	24
前任执行时间（分钟，中位）	47.7	15.5
测试通过率（均值 / 中位，%）	91.6 / 98.4	47.6 / 60.0
仍有 1–3 个测试失败的状态	88（53.3%）	33（100%）

表 2：SWE-Residual 统计。任务数为唯一源任务。先前尝试长度与执行时间为中位数。

SWE-Residual covers 90 of DeepSWE’s 113 tasks, spanning 74 repositories and five programming languages, and 21 of Terminal-Bench’s 89 tasks, spanning nine task categories. The retained states reflect substantial prior attempts: the median predecessor run takes 134 agent steps on DeepSWE and 24 on Terminal-Bench, and DeepSWE states change a median of nine files. On Terminal-Bench, the inherited work extends beyond files: retained states include installed software, a compiled application, and running QEMU instances that the evaluated agent must work around or build on. Many states are close to passing: on DeepSWE, the median state passes 98.4% of the verifier’s tests and 53.3% of states have only one to three tests still failing, yet every retained state fails overall. Appendix A gives further statistics.

SWE-Residual 覆盖 DeepSWE 113 任务中的 90 个（跨 74 仓库、五种语言），以及 Terminal-Bench 89 任务中的 21 个（九类）。保留状态反映充分先前尝试：DeepSWE 中位前任运行 134 智能体步、Terminal-Bench 24 步；DeepSWE 状态中位改九个文件。Terminal-Bench 上继承工作超出文件：含已装软件、已编译应用与运行中的 QEMU 实例，被评智能体须绕过或在其上构建。许多状态接近通过：DeepSWE 中位已过 98.4% 隐藏测试，53.3% 的状态仅剩 1–3 个失败测试。

5 Experiments

5 实验

We first measure how success changes when the starting state moves from the clean state $S_{0,i}$ to a Residual State $S_{r,i}$, across five frontier agents and across predecessor agents. We then ask whether Clean Start successes carry over to Residual Starts, and finally analyze how agents behave when working from Residual States.

我们先测量起始状态从干净 $S_{0,i}$ 迁到残留 $S_{r,i}$ 时成功率如何变化，跨五个前沿智能体与各前任；再问干净起步成功是否迁移到残留起步；最后分析智能体在残留状态下的行为。

5.1 Experimental Setup

5.1 实验设置

Agents and harnesses.

We evaluate five frontier models as coding agents: Gemini-3.8-Flash, Qwen-3.8-Flash, GLM-5.3-Flash, GPT-6-Sol, and Kimi-K2.8-Preview. We use DeepSWE v1.1 and Terminal-Bench 2.1 as the source benchmarks. Following their evaluation setups, we use mini-swe-agent (Yang et al., 2024) as the agent harness for DeepSWE and Terminus-2 (Merrill et al., 2026) for Terminal-Bench, with all experiments orchestrated through Harbor (Team, 2026). Each model is evaluated with the highest available reasoning effort. Per-run time limits and other execution budgets follow the default settings of the corresponding source benchmark. Additional details, including sampling parameters, model-specific runtime settings, and infrastructure configurations, are provided in Appendix C.

智能体与编排外壳。

我们评估五个前沿模型作为编码智能体：Gemini-3.8-Flash、Qwen-3.8-Flash、GLM-5.3-Flash、GPT-6-Sol 与 Kimi-K2.8-Preview。源基准为 DeepSWE v1.1 与 Terminal-Bench 2.1。按各自评测设置，DeepSWE 使用 mini-swe-agent（Yang et al., 2024）作为智能体编排外壳，Terminal-Bench 使用 Terminus-2（Merrill et al., 2026）；全部实验由 Harbor（Team, 2026）编排。每个模型都在可用的最高推理力度下评估。单次运行时限及其他执行预算遵循对应源基准的默认设置。采样参数、模型特定运行设置与基础设施配置等更多细节见附录 C。

Metrics.

Our primary metric is the success rate under Residual Start. For tasks with multiple states, we first average their binary success outcomes within the task, then average across tasks so that each source task receives equal weight. As a matched reference, we report Clean Start success on the same source tasks. The request, verifier, harness, prompt, and evaluation budget are identical between the two conditions; only the starting state changes. We define Residual Gap as the Clean Start success rate minus the Residual Start success rate, measured in percentage points. A positive gap indicates lower success under Residual Start. We refer to a run that starts from a Residual State as a continuation.

量子仪

我们的首要衡量标准是剩余启动计划的成功率。 对于有多个状态的任务,我们首先在任务中平均它们的二进制成功结果,然后在任务中平均,这样每个源任务就能得到相等的权重. 作为匹配的参考,我们报告在相同的源任务上"干净起步"成功. 请求、核查、利用、及时和评价预算在两个条件之间是相同的;只有起始状态发生变化。 我们把“响应差距”定义为“清洁启动成功率减去残余启动成功率”,以百分点衡量。 一个积极的差距表明,在 " 余留事项启动 " 方案下,成功率较低。 我们把从剩余国家开始的一次逃亡称为 " 持续 " 。

5.2 Performance on SWE-Residual

5.2 在 SWE-Residual 上的表现

We evaluate five frontier agents under blind continuation. Table 3 reports Residual Start success with each source task weighted equally, alongside the matched Clean Start reference and the Residual Gap. Residual Start success ranges from 13.0% to 33.5% on DeepSWE and from 12.7% to 36.5% on Terminal-Bench. GPT-6-Sol performs best under Residual Start on both tracks, with success rates of 33.5% and 36.5%, respectively. Even the highest-scoring agent therefore leaves a large fraction of inherited states unresolved.

我们评估了五个边境特工 视而不见 表3报告 " 剩余启动 " 的成功,每项来源任务均匀加权,同时对应的 " 清洁启动 " 参考和 " 剩余缺口 " 。 在DeepSWE上,剩余启动成功率从13.0%到33.5%,在Bench终端上,从12.7%到36.5%。 GPT-6-Sol在两条轨道的剩余启动下表现最好,成功率分别为33.5%和36.5%。 因此,即使是得分最高的智能体也留下了很大一部分继承国家未解决的问题。

Track	Agent	Residual (%)	Ref. Clean (%)	Gap
SWE-Residual (DeepSWE)	Gemini-3.8-Flash	20.0	72.2	52.2
Qwen-3.8-Flash	30.6	51.1	20.6
GLM-5.3-Flash	15.4	68.9	53.5
GPT-6-Sol	33.5	65.6	32.0
Kimi-K2.8-Preview	13.0	57.8	44.8
SWE-Residual (Terminal-Bench)	Gemini-3.8-Flash	14.3	47.6	33.3
Qwen-3.8-Flash	12.7	33.3	20.6
GLM-5.3-Flash	14.3	38.1	23.8
GPT-6-Sol	36.5	71.4	34.9
Kimi-K2.8-Preview	20.6	42.9	22.2

Table 3: Performance on the two SWE-Residual tracks. Residual reports success averaged first over Residual States within each task, then equally across tasks. Ref. Clean reports success on the same underlying tasks. Gap is Ref. Clean minus Residual, in percentage points, computed before rounding. Bold indicates the highest residual and clean success rates within each track.

音轨	探员	残存(%)	参考文献:清洁(%)	差距
SWE-Residual (深交所)	双子座-3.8-闪光	20.0	页:1	52.2
Quen-3.8-闪光	30.6	51.1	20.6
应是GLM-5.3-闪光	15.4	68.9	53.5
GPT-6- 苏尔	页:1	65.6	32.0
Kimi- K2.8- 预览	13.0	57.8	44.8
SWE-Residual( 任期- 本奇)	双子座-3.8-闪光	14.3	47.6	33.3
Quen-3.8-闪光	12.7	33.3	20.6
GLM-5.3-闪光	14.3	38.1	23.8
GPT-6- 苏尔	页:1	咕	34.9
Kimi- K2.8- 预览	20.6	42.9	22.2

表3:SWE-Residual两条轨道的性能调查报告. 剩余报告的成功率首先高于每项任务的剩余国家,然后在各项任务之间相同。 参考:清洁报告在同一基本任务上取得了成功。 空白是参考文献,以四舍五入前计算的百分比乘以净值。 粗体表示每一轨道的剩余和清洁成功率最高。

Inheriting earlier work costs every agent 20.6 to 53.5 points.

Every evaluated agent performs worse under Residual Start than under the matched Clean Start reference. The Residual Gap ranges from 20.6 to 53.5 percentage points on DeepSWE and from 20.6 to 34.9 points on Terminal-Bench. For example, Gemini-3.8-Flash falls from 72.2% to 20.0% on DeepSWE, while GPT-6-Sol falls from 71.4% to 36.5% on Terminal-Bench.

继承先前的工作每名智能体花费20.6至53.5分。

每个被评价的毒剂在"残留启动"下的表现都比对应的"清洁启动"参考文献更糟糕. DepenSWE的剩余差距在20.6%至53.5个百分点之间,Bench终点站的剩余差距在20.6%至34.9个百分点之间。 例如,双子座-3.8-Flash在DeepSWE上从72.2%下降到20.0%,而GPT-6-Sol则在Terminal-Bench上从71.4%下降到36.5%。

The decline occurs although the median DeepSWE state already passes 98.4% of the verifier’s tests (Section 4), and it appears on both tracks, from inherited repository changes to modified machine environments.

虽然DeepSWE的中位数已经通过98.4%的验证者测试(第4节),但下降还是发生了,它出现在两个轨道上,从继承的寄存器改变到修改的机器环境。

Clean-start rankings reverse on DeepSWE.

On DeepSWE, Gemini-3.8-Flash and GLM-5.3-Flash rank first and second under Clean Start, but third and fourth under Residual Start, behind GPT-6-Sol and Qwen-3.8-Flash. In particular, GLM-5.3-Flash outperforms Qwen-3.8-Flash by 17.8 percentage points under Clean Start, whereas Qwen-3.8-Flash outperforms GLM-5.3-Flash by 15.2 points under Residual Start. The DeepSWE ranking reversal shows that clean-start and residual-start success can rank agents differently. Absolute residual performance and the drop from Clean Start also capture different aspects of the results: GPT-6-Sol achieves the highest residual success rate on both tracks, while Qwen-3.8-Flash has the smallest Residual Gap.

在DeepSWE上,清洁启动排名逆转.

在DeepSWE上,双子座-3.8-Flash和GLM-5.3-Flash在Clean Start下排名第一和第二,但在残存Start下排名第三和第四,在GPT-6-Sol和Quen-3.8-Flash后排名第四. 特别是GLM-5.3-Flash在Clean Start下优于Quen-3.8-Flash17.8个百分点,而Quen-3.8-Flash在Clean Start下优于GLM-5.3-Flash15.2个百分点. DeepSWE排名逆转显示,清洁启动和剩余启动成功可以对特工进行不同的排名. 绝对剩余性能和从Clean Start的下降也捕捉了不同方面的结果:GPT-6-Sol在两轨都实现了最高的剩余成功率,而Quen-3.8-Flash的剩余性差距最小.

The drop holds for states from every predecessor agent.

To examine whether the performance decline persists across predecessor agents, we group all Residual States from DeepSWE and Terminal-Bench by the agent that produced them. Within each group, we compare Residual Start success with the matched Clean Start reference on the same underlying tasks, weighting each task equally. Table 8 in Appendix E.1 shows that all five evaluated agents perform worse under Residual Start in every predecessor group, with Residual Gaps ranging from 19.0 to 51.8 percentage points. The decline therefore occurs across states produced by all three predecessor agents. Since the groups cover different task sets, differences in their absolute scores may also reflect task difficulty.

{\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}每个前特工的状态都会下降

为了审查前身智能体的性能是否持续下降,我们把所有残余国家从DeepSWE和Terminal-Bench归类为产生它们的智能体。 在每一组中,我们把 " 剩余启动 " 的成功与 " 清洁启动 " 对应的参考信息进行比较,以相同的基础任务为基础,对每项任务进行同等加权。 附录E.1的表8显示,所有5个经评估的智能体在前几类中都表现较差,其残留差距在19.0至51.8个百分点之间。 因此,所有三个前身智能体产生的各州都出现下降。 由于各组涵盖不同的任务组合,其绝对分数的差异也可能反映出任务难度.

5.3 Success Does Not Carry Over from Clean Start

5.3 干净起步的成功无法迁移到残留起步

We next ask whether an agent can still complete a task from a Residual State when it succeeds on the same task from Clean Start. For each agent, we select its observed Clean Start successes and examine its Residual Start outcomes on those tasks. Figure 2 shows the resulting failure and success rates.

我们接下来要问,当一个智能体从 " 干净起步 " 成功完成同样的任务时,是否仍然能够完成剩余国家的任务。 对于每个智能体,我们选择其观察到的 " 清洁启动 " 成功事例,并审查这些任务的剩余启动成果。 图2显示由此产生的失败率和成功率。

Figure 2: Residual Start outcomes on tasks that the same agent successfully completes under Clean Start. Red indicates failure and green indicates success. Outcomes are averaged over Residual States within each task, then equally across tasks. Each agent is evaluated on its own set of observed Clean Start successes.

图2:同一剂在Clean Start下成功完成的任务的剩余启动结果. 红色表示失败,绿色表示成功. 每项任务的成果平均比剩余国家多,然后在任务之间平均。 每种制剂都根据各自观察到的 " 清洁启动 " 成功经验进行评估。

We run this analysis on three agents: Gemini-3.8-Flash, Qwen-3.8-Flash, and GLM-5.3-Flash. Each has a conditional failure rate above 60% on both tracks, with rates from 63.4% to 79.0% on DeepSWE and from 61.9% to 81.3% on Terminal-Bench. For example, Gemini-3.8-Flash fails at a rate of 75.4% on DeepSWE tasks it completes under Clean Start, while GLM-5.3-Flash fails at a rate of 81.3% on its corresponding Terminal-Bench tasks.

我们用三种药剂来分析:双子座-3.8-Flash,Quen-3.8-Flash,和GLM-5.3-Flash. 每条轨道的有条件故障率均高于60%,在DeepSWE的故障率为63.4%至79.0%,在终端-奔驰的故障率为61.9%至81.3%。 例如,双子座-3.8-Flash在DeepSWE任务上以75.4%的速度在Clean Start下完成,而GLM-5.3-Flash在相应的终端-Bench任务上以81.3%的速度失败.

On tasks each agent completes from a Clean Start, most task-weighted Residual Start outcomes are failures. Success on the original task often fails to carry over once earlier work is inherited. Gemini-3.8-Flash and GLM-5.3-Flash, the strongest of the three under Clean Start, fail most often from inherited states on both tracks.

在每一剂从清洁启动完成的任务上,大多数任务加权的残留启动结果都是失败的. 一旦继承了以前的工作,原任务的成功往往无法延续。 双子座-3.8-Flash和GLM-5.3-Flash是Clean Start下的三个中最强的,两轨上继承的状态最经常失败.

5.4 Acceptance Bias

5.4 接受偏置

From a Clean Start, agents pass most of the tests that a Residual State fails; from the Residual State itself, most of these tests still fail at the end. We observe a recurring pattern in their continuations, which we call acceptance bias: once earlier work is present, the agent accepts it as the definition of what remains to be done, in place of the request. It continues inside the earlier work and leaves what that work got wrong or left out as it found it. We examine this on DeepSWE, whose per-test verifier outcomes, final patches, and reference solutions make each part of this pattern measurable (Table 4).

从 " 干净的开端 " 开始,智能体通过一个残余国家失败的大多数测试;从残余国家本身来看,这些测试大多数在结束时仍然失败。 我们发现,在继续工作时,一再出现一种模式,我们称之为 " 接受偏置 " :一旦出现较早的工作,智能体就接受它作为有待完成的工作的定义,以取代请求。 它继续从事早先的工作,并留下它发现的工作出错或遗漏的东西。 我们在DeepSWE上研究了这一点,其经测试的核查结果、最后补丁和参考解决办法使这一模式的每个部分都能衡量(表4)。

			Initially failing		Left-out files
Agent	Gap	Same failing	Clean	Residual	Kept	Clean	Residual	Steps
GLM-5.3-Flash	53.5	78.9	87.3	23.4	95.7	40.5	06.7	0.57
Gemini-3.8-Flash	52.2	76.8	88.4	27.5	97.2	43.5	09.4	0.70
Kimi-K2.8-Preview	44.8	88.2	77.6	14.4	98.6	45.7	8.1	0.56
GPT-6-Sol	32.0	58.9	84.1	47.7	91.1	41.2	15.0	1.02
Qwen-3.8-Flash	20.6	72.0	72.8	39.1	86.4	47.9	14.6	0.84

Table 4: Acceptance bias on DeepSWE, with agents sorted by Residual Gap. Same failing: failed continuations that end with exactly the tests that failed in the Residual State. Initially failing: share of those tests that the agent passes from each start. Kept: lines written by the predecessor agent that remain after failed continuations. Left-out files: runs that change a file changed by the reference solution but not by the predecessor agent, over states where both runs are normally submitted. Steps: median ratio of Residual to Clean steps. Normally submitted runs; task-level averages in % (Gap in points).

			初始失败		左侧文件
探员	差距	同样的失败	干净	剩余	固定	干净	剩余	步骤
GLM-5.3-闪光	53.5	78.9	87.3	23.4	95.7	40.5	06.7	0.57
双子听众3.8-Flash	52.2	76.8	88.4	27.5	97.2	43.5	09.4	0.70
Kimi- K2.8- 预览	44.8	88.2	77.6	14.4	98.6	45.7	8.1	0.56
GPT-6- 苏尔	32.0	58.9	84.1	47.7	91.1	41.2	15.0	1.02
Quen-3.8-闪光	20.6	72.0	72.8	39.1	86.4	47.9	14.6	0.84

表4:对DeepSWE的接受偏置,智能体按残留漏洞分类. * 同样的失败*:以在余留事项国家失败的测试为目的的失败延续。 * 初步失败*:从每次试验开始,智能体通过试验的比例。 Kept:在失败续作后,由前代理写成的行. Left-out文件:运行以更改一个由引用解决方案改变的文件,而不是由前身代理更改,在通常同时提交运行的状态上运行. 步骤:残余物与清洁步骤的中位比。 通常提交运行;任务级别平均值以%(积分计)计。

Inherited failures survive, although agents can fix them.

In 58.9% to 88.2% of failed continuations, exactly the same verifier tests fail at the end as at the start. Across failed continuations, agents repair 10.4% to 20.1% of the initially failing tests and break almost none of the passing ones. These tests are not beyond the agents: on the same tasks, the same agent passes 72.8% to 88.4% of them from a Clean Start, against 14.4% to 47.7% from the Residual State. The tests become hard only once a failing version of the work is present.

继承的失败依然存在 虽然特工可以修复它们

在58.9%到88.2%的失败续作中,最后的验证测试与开始时完全相同. 在失败的续作中,智能体修复了10.4%到20.1%的最初失败的测试,并且几乎没有突破任何通过测试. 这些测试没有超出药剂 Tess:在同一任务中,同一种药剂从一个干净的开始就从72.8%到88.4%,而残余国家从14.4%到47.7%。 只有在作品的失败版本出现后,测试才变得艰难.

The earlier work sets the scope of the continuation.

Agents do not overlook the earlier work: every continuation inspects the inherited changes, through git status, git diff, or the modified files. They then work inside it. Discarding the inherited changes and starting over from the base commit is always possible, and no continuation does so. Failed continuations keep 86.4% to 98.6% of the lines the predecessor agent wrote, and, except for GPT-6-Sol, whose ratio is 1.02, continuations take 0.56 to 0.84 times as many steps as runs from a Clean Start. The clearest evidence comes from the files that the earlier work left out, that is, files changed by the reference solution but not by the predecessor agent. On the same states, agents change these files in 40.5% to 47.9% of their Clean Start runs but in only 6.7% to 15.0% of their Residual Start runs, a difference of 26.2 to 37.6 points. The files are not unseen: agents open them in 37.6% to 64.0% of Residual Start runs, and leave them unchanged.

• 先前的工作确定了继续工作的范围。

代理不会忽略先前的作品:每次续作都会通过git状态,git diff,或修改后的文件检查所继承的更改. 然后他们在里面工作。 放弃继承的改变,从基础承诺开始总是可能的,任何延续都不可能. 失败的续作保留了前代理所写行数的86.4%至98.6%,除了GPT-6-Sol的比为1.02外,续作的步数是"干净的开始"的0.56至0.84倍. 最清楚的证据来自先前的工作留下的文件,即由参考解决方案改变的文件,而不是由前身智能体改变的文件. 在同一州,智能体将这些文件的40.5%改为47.9%的 " 清洁启动 " 运行,但仅将6.7%改为15.0%的 " 残余启动 " 运行,差额为26.2至37.6分。 档案不为人所知:代理商在剩余启动运行中以37.6%至64.0%的速度打开,并保持不变。

The more an agent accepts, the more it loses.

The three agents with the largest gaps, GLM-5.3-Flash, Gemini-3.8-Flash, and Kimi-K2.8-Preview, show every signature most strongly: they keep the most of the earlier work, most often leave the same tests failing, change the left-out files least, and take the fewest steps. GPT-6-Sol and Qwen-3.8-Flash, whose gaps are 32.0 and 20.6 points, show each signature more weakly. The same holds within agents: for Gemini-3.8-Flash, Qwen-3.8-Flash, and GLM-5.3-Flash, continuations that keep more of the earlier work than the agent’s median succeed 14.0 to 22.7 points less often than those that keep less. GPT-6-Sol fails differently. In 11 of 30 sampled failed continuations it names the defect in the earlier work and tries to repair it, which Gemini-3.8-Flash, Qwen-3.8-Flash, and GLM-5.3-Flash each do in at most 2 of 30. Among these four agents, its failed continuations also repair the most initially failing tests (20.1%, against 10.4% to 13.7%) and least often end with the same failures (58.9%). Its failures are more often failed repairs than accepted work. Its Residual Gap is nonetheless 32.0 points: recognizing that the inherited work is wrong recovers part of the lost performance, and repairing it accounts for the rest.

智能体接受得越多 就越输

GLM-5.3-Flash,双子座-3.8-Flash,和Kimi-K2.8-Preview这三种漏洞最大的代理,最强烈地显示每个签名:它们保存了大部分早期的工作,最常留下相同的测试失败,对左侧文件的更改最少,并采取了最少的步骤. GPT-6-Sol和Qwen-3.8-Flash的空档分别为32.0和20.6分,分别较弱显示每个签名. 同样的情况也存在于智能体内部:对于双子座-3.8-Flash,Quen-3.8-Flash,以及GLM-5.3-Flash,延续了比智能体的中位数成功14.0到22.7分少的早期工作. GPT-6-Sol故障不同. 在30个抽样失败的续作中的11个中,它将早期作品中的缺陷命名并试图修复,双子座-3.8-Flash,Quen-3.8-Flash,以及GLM-5.3-Flash在30个作品中最多各做了2个. 在这四种药剂中,其失败的续作还修复了最初始的失败测试(20.1%,对10.4%至13.7%),最不常见的是以同样的失败(58.9%)结束. 它的失败往往比被接受的工程更是失败的修复. 尽管如此,其残存差距为32.0分:认识到所继承的作品错误地收回了部分失物招领的功绩,而修复它则占其余部分。

Case study.

A run of Gemini-3.8-Flash on onedump shows the pattern in its plainest form. The Residual State fails 11 verifier tests, all of which the same agent passes from a Clean Start. From the Residual State, the agent inspects the inherited changes, runs the package tests, which all pass, and submits after 43 steps without changing a single file. Its final message describes the earlier work as its own: “I have implemented the requested encryption package.” The reference solution changes encryption/encryption.go, a file the predecessor agent never touched; the evaluated agent leaves it unchanged, while its own Clean Start run changes it. Qwen-3.8-Flash and GLM-5.3-Flash also fail from this state, each leaving the same 11 tests failing.

案例研究。

双子座-3.8-Flash on on dudump 显示最简单的模式. 残留国未进行11次核查测试,所有测试都是通过 " 清洁启动 " 的。 从残存国,智能体检查所继承的更改,运行包测试,全部通过,43步骤后提交,不更改一个文件. 其最后的讯息将先前的工作描述为自己的作品:“我已经执行了所要求的加密软件包。” 引用解决方案更改了加密/加密.go,一个前任代理从未触及过的文件;被评估的代理未更改,而其自己的Clean Start运行会更改它. Quen-3.8-Flash和GLM-5.3-Flash也因这个状态而失败,它们各自留下相同的11个测试失败.

6 Conclusion

6 结论

SWE-Residual measures how well coding agents continue work that a predecessor submitted as complete but that fails the hidden tests, given only the original request and the inherited state. Across five frontier agents, inheriting such work lowers success by 20.6 to 53.5 percentage points on the same tasks, most Clean Start successes do not carry over, and clean-start rankings can reverse. The failures share a pattern we call acceptance bias: agents treat the inherited work as the definition of what remains to be done and leave what it got wrong or left out in place; the more an agent accepts, the more it loses. Clean-start scores therefore overstate how well an agent can take over work in progress, and Starting-State Robustness deserves to be evaluated alongside them. Residualization applies to any benchmark with hidden tests, and SWE-Residual provides a testbed for agents that check inherited work against the request before building on it.

SWE-Residual 衡量编程智能体在只拿到原请求与继承状态时，能否继续前任提交为完成、却仍使隐藏测试失败的工作。五个前沿智能体上，继承此类工作使同任务成功率下降 20.6 至 53.5 个百分点；多数干净起步成功无法迁移，干净起步排名可翻转。失败共享我们称为接受偏置的模式：智能体把继承工作当作待办定义，把其做错或漏掉的部分留在原地；接受越多，损失越大。干净起步分数因此高估智能体接手进行中工作的能力，起始状态鲁棒性值得与之并列评测。Residualization 适用于任何带隐藏测试的基准，SWE-Residual 为“先对照请求检查继承工作再在其上构建”的智能体提供试验台。

AI Use Statement

AI 使用声明

Generative AI tools were used for typo revising, and were not used for method design or experimental analysis. In the benchmark construction, GLM-5.3 generates summaries of predecessor trajectories to support human review (Section 3.2); all inclusion decisions are made by human reviewers.

基因AI工具用于重排修订,不用于方法设计或实验分析. 在基准构建中,GLM-5.3生成前作轨迹摘要,以支持人类审查(第3.2节);所有纳入决定都由人类审查员做出.

Ethics Statement

伦理声明

Residual States contain unsuccessful implementations produced during software task attempts. The evaluation protocol uses isolated environments for executing these research artifacts. Distribution follows the terms of the source repositories and benchmarks. Upon publication, we will release all Residual States, the artifacts needed to reconstruct them, the Residualization pipeline, the evaluation code, and per-run outcomes.

剩余国家包含在软件任务尝试期间产生的未成功实施。 评估协议使用孤立的环境执行这些研究文物. 分发遵循源寄存器和基准的规定。 一旦公布,我们将公布所有剩余国家、重建这些国家所需的文物、剩余化管道、评价代码和每项成果。

Reproducibility Statement

可复现性声明

Section 3 specifies the construction and paired evaluation protocol. Appendix A reports benchmark statistics, Appendix B describes state transfer, and Appendices C–E give the experimental configurations and additional analyses. Reconstruction manifests, filtering records, and per-run outcomes will be publicly released upon publication.

第3节规定了施工和配对评价协议。 附录A报告基准统计,附录B描述国家转移,附录C-E给出实验配置和补充分析. 重建清单、过滤记录和每个运行结果将在公布后公布。

References

参考文献

Advani (2026) L. Advani From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents . arXiv preprint arXiv:2606.09863 . External Links: Link Cited by: §1 , §2 .

Chen et al. (2026) J. Chen, X. Xu, H. Wei, C. Chen, and B. Zhao SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration . arXiv preprint arXiv:2603.03823 . External Links: Link Cited by: §2 .

Dai et al. (2025) D. Dai, M. Liu, A. Li, J. Cao, Y. Wang, C. Wang, X. Peng, and Z. Zheng FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks . arXiv preprint arXiv:2504.06939 . Cited by: §2 , Table 1 .

Deng et al. (2025) X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, V. Bharadwaj, J. Holm, R. Aluri, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? . arXiv preprint arXiv:2509.16941 . External Links: Link Cited by: §1 , §2 .

Fu et al. (2025) H. Y. Fu, A. Shrivastava, J. Moore, P. West, C. Tan, and A. Holtzman Absence Bench: Language Models Can’t See What’s Missing . In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track , External Links: Link Cited by: §2 .

Ganz et al. (2026) R. Ganz, M. S. Nacson, A. Kalyanpur, and R. Litman The Handoff Tax: Continuing Non-Native Trajectories in LLM Agents . arXiv preprint arXiv:2608.24358 . External Links: Link Cited by: §2 , Table 1 .

Gloaguen et al. (2026) T. Gloaguen, N. Mündler, M. Müller, V. Raychev, and M. Vechev Coding Agents Don’t Know When to Act . arXiv preprint arXiv:2605.07769 . External Links: Link Cited by: §2 .

Guo et al. (2025) L. Guo, W. Tao, R. Jiang, Y. Wang, J. Chen, X. Liu, Y. Ma, M. Mao, H. Zhang, and Z. Zheng OmniGIRL: a multilingual and multimodal benchmark for GitHub issue resolution . Proc. ACM Softw. Eng. 2 ( ISSTA ). External Links: Link , Document Cited by: §2 .

Huang et al. (2024) J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou Large Language Models Cannot Self-Correct Reasoning Yet . In International Conference on Learning Representations , External Links: Link Cited by: §2 .

Huang et al. (2026) W. Huang, C. Lee, L. Tng, and S. Ge DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks . arXiv preprint arXiv:2607.07946 . External Links: Link Cited by: §1 , §2 , §3.2 .

Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan SWE-bench: Can Language Models Resolve Real-World GitHub Issues? . In International Conference on Learning Representations , External Links: Link Cited by: §1 , §2 , Table 1 .

Kamoi et al. (2024) R. Kamoi, Y. Zhang, N. Zhang, J. Han, and R. Zhang When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs . Transactions of the Association for Computational Linguistics 12 , pp. 1417–1440 . External Links: Link Cited by: §2 .

KC and Budathoki (2026) D. KC and A. Budathoki Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Tasks . arXiv preprint arXiv:2606.02875 . External Links: Link Cited by: §2 , Table 1 .

Khatoonabadi et al. (2023) S. Khatoonabadi, D. E. Costa, R. Abdalkareem, and E. Shihab On wasted contributions: understanding the dynamics of contributor-abandoned pull requests–a mixed-methods study of 10 large open-source projects . ACM Transactions on Software Engineering and Methodology 32 ( 1 ), pp. 1–39 . Cited by: §1 .

Laban et al. (2026) P. Laban, H. Hayashi, Y. Zhou, and J. Neville LLMs Get Lost In Multi-Turn Conversation . In International Conference on Learning Representations , External Links: Link Cited by: §2 .

Lam et al. (2026) M. H. Lam, C. Wang, H. Liu, J. Xiao, H. Li, J. Huang, T. Y. Zhuo, and M. R. Lyu SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades . arXiv preprint arXiv:2605.14415 . External Links: Link Cited by: §2 .

Le et al. (2026) T. Le, M. V. T. Thai, D. N. Manh, H. P. Nhat, and N. D. Q. Bui SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios . arXiv preprint arXiv:2512.18470 . External Links: Link Cited by: §2 .

Merrill et al. (2026) M. Merrill, A. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Shin, T. Walshe, E. K. Buchanan, et al. Terminal-bench: benchmarking agents on hard, realistic tasks in command line interfaces . In International Conference on Learning Representations , Vol. 2026 , pp. 40903–40986 . Cited by: §1 , §2 , Table 1 , §3.2 , §5.1 .

Miserendino et al. (2025) S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? . arXiv preprint arXiv:2502.12115 . External Links: Link Cited by: §2 .

Orlanski et al. (2026) G. Orlanski, D. Roy, A. Yun, C. Shin, A. Gu, A. Ge, D. Adila, N. Roberts, F. Sala, and A. Albarghouthi SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks . arXiv preprint arXiv:2603.24755 . External Links: Link Cited by: §2 .

Parsons and Saunders (2004) J. Parsons and C. Saunders Cognitive Heuristics in Software Engineering: Applying and Extending Anchoring and Adjustment to Artifact Reuse . IEEE Transactions on Software Engineering 30 ( 12 ), pp. 873–888 . External Links: Link Cited by: §2 .

Rashid et al. (2025) M. S. Rashid, C. Bock, Y. Zhuang, A. Buchholz, T. Esler, S. Valentin, L. Franceschi, M. Wistuba, P. T. Sivaprasad, W. J. Kim, A. Deoras, G. Zappella, and L. Callot SWE-PolyBench: A Multi-Language Benchmark for Repository Level Evaluation of Coding Agents . arXiv preprint arXiv:2504.08703 . External Links: Link Cited by: §2 .

Shastry et al. (2026) K. A. Shastry, G. Senrayan, S. Satapara, P. Panda, and C. Devaguptapu Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution . arXiv preprint arXiv:2604.03035 . External Links: Link Cited by: §2 .

Sonwane et al. (2026) A. Sonwane, E. Tu, W. Lu, C. Beger, C. Larsen, D. Dhar, S. Alford, R. Chen, R. Pattanayak, T. A. Dang, G. Chen, G. Geng, K. Ellis, and S. Dutta OmniCode: A Benchmark for Evaluating Software Development Agents . In Findings of the Association for Computational Linguistics: ACL 2026 , M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.) , San Diego, California, United States , pp. 40634–40661 . External Links: Link , Document , ISBN 979-8-89176-395-1 Cited by: §2 , Table 1 .

Tan et al. (2025) S. Tan, K. Lin, K. Sen, and M. Zaharia Recovery-Bench: Evaluating Agentic Recovery from Mistakes . In NeurIPS Workshop on Evaluating the Evolving LLM Lifecycle , External Links: Link Cited by: §2 , Table 1 .

Team (2026) H. F. Team Harbor: a framework for evaluating and optimizing agents and models in container environments . Zenodo . Cited by: §5.1 .

Tyen et al. (2024) G. Tyen, H. Mansoor, V. Carbune, P. Chen, and T. Mak LLMs Cannot Find Reasoning Errors, but Can Correct Them Given the Error Location . In Findings of the Association for Computational Linguistics: ACL 2024 , External Links: Link Cited by: §2 .

Yang et al. (2024) J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering . In Advances in Neural Information Processing Systems , External Links: Link Cited by: §1 , §5.1 .

Yang et al. (2025) J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. R. Narasimhan, D. Yang, S. I. Wang, and O. Press SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? . In International Conference on Learning Representations , External Links: Link Cited by: §2 .

Zan et al. (2025) D. Zan, Z. Huang, W. Liu, H. Chen, S. Xin, L. Zhang, Q. Liu, A. Li, L. Chen, X. Zhong, S. Liu, Y. Xiao, L. Chen, Y. Zhang, J. Su, T. Liu, R. Long, M. Ding, and L. Xiang Multi-SWE-bench: a multilingual benchmark for issue resolving . In Advances in Neural Information Processing Systems , D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.) , Vol. 38, Main Conference , pp.  . External Links: Document , Link Cited by: §2 .

Zhang et al. (2024) M. Zhang, O. Press, W. Merrill, A. Liu, and N. A. Smith How Language Model Hallucinations Can Snowball . In International Conference on Machine Learning , External Links: Link Cited by: §2 .

Zhou et al. (2026) Q. Zhou, J. Zhang, H. Wang, R. Hao, J. Wang, M. Han, Y. Yang, S. Wu, F. Pan, L. Fan, D. Tu, and Z. Zhang FeatureBench: Benchmarking Agentic Coding for Complex Feature Development . In International Conference on Learning Representations , External Links: Link Cited by: §2 .

Appendix A Benchmark Statistics

附录 A 基准统计

This appendix expands the summary in Section 4; Table 2 lists the main statistics.

缪克文:本附录扩充了第4节的摘要;表2列出了主要的统计数据.

Coverage and provenance.

SWE-Residual covers 90 of DeepSWE’s 113 tasks and 21 of Terminal-Bench’s 89 tasks. The DeepSWE tasks span 74 repositories and five programming languages: TypeScript, Go, Python, JavaScript, and Rust. The Terminal-Bench tasks span nine official categories, including software engineering, scientific computing, data science, security, and system administration. Both tracks contain states produced by all three predecessor families: DeepSeek, Qwen, and GLM. Of the 111 source tasks, 42 have one retained state, 51 have two, and 18 have three. Tasks with multiple states provide different inherited implementations or environments under the same original request and verifier.

覆盖范围和来源。

SWE-Residual涵盖了DeepSWE的113项任务中的90项和终端-奔驰的89项任务中的21项. DeepSWE的任务跨越74个寄存器和5个编程语言:TypeScript,Go,Python,JavaScript和Rust. 终端-奔驰任务涵盖9个正式类别,包括软件工程,科学计算,数据科学,安全和系统管理. 两条音轨都包含所有三个前世系产生的状态:DeepSeek,Quen,以及GLM. 在111个源任务中,42个有1个保留状态,51个有2个,18个有3个. 具有多个状态的任务在同一原始请求和验证人下提供不同的继承执行或环境.

Extent of prior work.

The retained states reflect substantial prior attempts. The median predecessor run contains 134 agent steps on DeepSWE and 24 on Terminal-Bench, with interquartile ranges of 102–167 and 15–49, respectively. Median agent execution times are 47.7 and 15.5 minutes, including model and tool execution but excluding environment setup and verification. These measurements describe the prior attempts that produced the retained states.

• 先前工作的程度。

保留状态反映了以前的重大尝试。 前置中位数运行在DeepSWE上包含134个代理步数,在Terminal-Bench上包含24个代理步数,间距分别为102–167和15–49. 中间代理执行时间为47.7分钟和15.5分钟,包括模型和工具执行,但不包括环境设置和核实. 这些测量方法描述了以前产生保留状态的尝试。

DeepSWE states change a median of nine files, with medians of 1,164 added text lines and 10 deleted text lines. The median patch size is 44.6 KiB. These changes can include source code, tests, documentation, and configuration. In Terminal-Bench, inherited work can additionally include installed software and running processes; retained examples include a compiled application and running QEMU instances. The median prior attempt contains 45 terminal interactions, including polling and control inputs. The two tracks therefore capture both repository-level changes and work that extends into the machine environment.

DeepSWE声明更改9个文件的中位数,中位数为1,164个新增文本行和10个删除文本行. 中间补丁尺寸为44.6 KiB. 这些修改可以包括源代码,测试,文档,以及配置. 在Terminal-Bench中,继承的工作还可以包括安装的软件和运行过程;保留的例子包括编译的应用程序和运行QEMU实例. 前一次尝试的中位数包含45次终端互动,包括投票和控制投入。 因此,这两条轨道既捕捉寄存器级的变化,又捕捉延伸到机器环境的工作.

Progress in Residual States.

We characterize each inherited state by the fraction of the original verifier’s acceptance tests that it passes before continuation. For DeepSWE, these include both target tests for the requested functionality and regression tests for existing behavior; for Terminal-Bench, they include all task acceptance tests. We compute the pass rate for each state and summarize these rates with equal weight across states. The mean and median are 91.6% and 98.4% on DeepSWE, and 47.6% and 60.0% on Terminal-Bench, respectively.

• 剩余国家的进展。

我们用原始核查者的验收测试的一小部分来描述每个继承状态,测试在继续前通过。 对于DeepSWE来说,这既包括对所请求的功能进行目标测试,也包括对现有行为的回归测试;对于Terminal-Bench来说,它们包括所有任务接受测试. 我们计算每个州的通过率,并以各州同等的权重来总结这些率。 DeepSWE的平均值和中位数分别为91.6%和98.4%,Bench终端的平均值和中位数分别为47.6%和60.0%.

On DeepSWE, 88 of 165 states (53.3%) have one to three tests still failing, while 147 states (89.1%) preserve all tests that passed in the clean state. On Terminal-Bench, 25 of 33 states (75.8%) pass at least one test. Twenty-five states have one failing test and eight have two. Of the eight states that pass no tests, seven are evaluated by a single test, which provides no finer-grained measure of partial progress.

在DeepSWE上,165个州中有88个州(53.3%)仍有1至3个测试失败,而147个州(89.1%)保留了在清洁状态中通过的所有测试. 在Bench终端上,33个州中有25个州(75.8%)至少通过了一次测试。 25个州有一个测试失败,8个州有两个测试失败. 在未通过测试的8个州中,有7个州通过单一测试进行评估,该测试没有更精确地衡量部分进展。

These results show that many inherited states already satisfy part of the original verifier, although every retained state still fails overall. The pass rates describe the current state rather than the fraction of work completed by the predecessor agent, since some tests already pass in the clean state. Test suites also differ in granularity and coverage, and even a single failing test may require substantial further work. Our evaluation examines whether agents can identify and complete the work that remains.

这些结果显示,许多继承的州已经满足了部分原核实者的要求,尽管每个保留州仍然总体失败. 通过率描述的是当前戮力状态,而不是前代理完成的一小部分,因为一些测试已经在清洁状态中通过. 测试套件在颗粒度和覆盖范围上也有所不同,即使是一次失败测试也可能需要做大量进一步的工作. 我们的评估审查智能体是否能够确定和完成剩余的工作。

Appendix B Residual State Construction and Transfer

附录 B 残留状态构造与迁移

This appendix describes how each Residual State is reconstructed from a predecessor run and what the evaluated agent can observe when it starts from that state. Under both Clean Start and Residual Start, the evaluated agent receives the same request, system prompt, harness, and budget; only the starting state differs.

本附录说明每个残余国家是如何从前一个运行中重建的,以及评估的智能体从该状态开始时可观察到什么。 在 " 清洁启动 " 和 " 残余启动 " 下,被评估的代理机构收到相同的请求、系统即时、使用和预算;只有起始状态不同。

B.1 DeepSWE

During Stage I, we save all changes that the predecessor agent made relative to the task’s base commit as a binary-safe patch, covering modified, added, and deleted files, including binary files. The patch retains everything the predecessor left in the repository, including build outputs, caches, and logs; such incidental artifacts are ignored only when Stage II decides whether a run made substantive changes, and they are not removed from the reconstructed state. To reconstruct the state, we start from the task’s original Docker image at the base commit and apply the patch to the working tree without creating a commit. The Git history therefore matches the clean state. The inherited changes appear as uncommitted modifications, visible through commands such as git status and git diff, and any commits the predecessor made during its run are flattened into these modifications, so their commit messages are not transferred. We impose no restriction on Git operations, so an evaluated agent can discard the inherited changes and reimplement the task from the base commit, within the same budget as any other strategy. None of the recorded DeepSWE continuations discarded the inherited work in its entirety.

在第一阶段,我们保存前身代理相对于任务基础承诺所做的所有修改,作为二进制安全补丁,覆盖修改,添加和删除的文件,包括二进制文件. 补丁保留了寄存器中留下的所有前身,包括构建输出,缓存,以及日志;这些附带的文物只有在第二阶段决定运行是否进行了实质性的更改时才会被忽略,并且不会从重建状态中移除. 为了重建状态, 我们从任务在基地的原始 Docker 图像开始, 在不创建承诺的情况下将补丁应用到工作树上 。 因此,吉特历史与清洁状态相符。 所继承的更改以未承诺的更改形式出现,通过git状态和git diff等命令可见,任何在运行中作出的约定都被平整为这些更改,因此其承诺消息不被转移. 我们对Git业务没有限制,因此经评估的智能体可以在与任何其他战略相同的预算范围内,放弃继承的改变,重新执行基地承诺的任务。 所记录的DeepSWE续作都没有完全抛弃了继承的作品.

Files that the predecessor wrote inside the repository, including its own tests, documentation, and notes, remain part of the Residual State, since they are part of the earlier work. The hidden tests used by the verifier are stored outside the repository and are never exposed to the evaluated agent. Tests written or modified by the predecessor remain in the working tree, and the evaluated agent may inspect or change them. Final outcomes are determined by the source benchmark’s verifier and acceptance tests, independently of any tests modified by either agent.

前任在存放处内撰写的文件,包括自己的测试、文件和笔记,仍然是余留事项国家的一部分,因为它们是早先工作的一部分。 核查者所使用的隐蔽测试被存储在寄存器之外,从未暴露于被评估的代理. 由前身书写或者修改的检验仍留在工作树上,被评审的智能体可以检查或者变更. 最终结果由源基准的核实和验收测试确定,而不受任一智能体修改的任何测试的影响。

B.2 Terminal-Bench

In Terminal-Bench, earlier work can change the whole machine, including files, installed packages, configuration, and running processes. During Stage I, we record the ordered sequence of terminal commands and interactions sent by the predecessor agent. To reconstruct the state, we replay this sequence in a fresh task environment and create a Docker checkpoint using CRIU[^1], which preserves the container’s filesystem and live process state. Each continuation restores an independent instance from this checkpoint. For states with a running QEMU instance, such as those described in Section 4, we verify after restoration that the VM process and task-relevant services remain operational.

在Terminal-Bench中,早期的工作可以改变整个机器,包括文件,安装的软件包,配置,以及运行的过程. 在第一阶段,我们记录了由前代理发送的终端命令和相互作用的顺序。 为了重建状态, 我们在新的任务环境中重放此序列, 并使用 CRIU [^1] 创建 Docker 检查点, 以保存容器的文件系统和活的进程状态 。 每次续作都会从这个检查站恢复一个独立的实例. 对于有运行中的QEMU实例的国家,例如第4节所述的国家,我们在恢复后核实VM进程和与任务有关的服务仍在运作。

The evaluated agent receives the original request and the restored machine state. Shell history and other artifacts remain accessible as part of that state; explicit handoff notes and verifier feedback are absent.

被评价的代理接收原始请求和恢复的机器状态. 壳体历史和其他文物作为该州的一部分,仍然可以查阅;没有明确的交割说明和核查反馈。

B.3 Validation

B.3 验证

Each reconstructed state is validated before evaluation. A state is usable if it can be instantiated as an independent environment for a rollout: the harness can connect to it, the agent can execute commands and work on the task, and the original verifier can be run on the resulting state. We also check that the original verifier still fails on it, i.e., $V_{i}(q_{i},S_{r,i})=0$. A failure must be a valid verdict, meaning that the verifier executed the tests and returned a result; a zero reward caused by an infrastructure or verification error is not treated as a failed attempt, either for a predecessor run or for a reconstructed state. Table 5 reports how many candidates were excluded at each step. Every rollout starts from an independent copy of the validated state.

每个重建状态在评测前都会做验证。若某状态能实例化为一次 rollout 的独立环境，则视为可用：编排外壳可以连接到它，智能体可以执行命令并继续任务，原始验证器也能在结果状态上运行。我们还检查原始验证器在其上仍然失败，即 $V_{i}(q_{i},S_{r,i})=0$。失败必须是有效判定，意味着验证器执行了测试并返回了结果；因基础设施或验证错误导致的零奖励，无论对前驱运行还是对重建状态，都不算作失败尝试。表 5 报告了每一步排除了多少候选。每次 rollout 都从已验证状态的独立副本启动。

	DeepSWE	Terminal-Bench	Total
Passed human review	170	37	207
Patch application or command replay failed	2	1	3
Environment unusable for rollouts	1	2	3
No valid verifier result, or no longer failing	2	1	3
Included in SWE-Residual	165	33	198

Table 5: Reconstruction and validation of the candidates that passed human review.

	深水	终极奔驰	共计
通过人类审查	170	37	207
补丁应用程序或命令重放失败	2	1	3
无法推出的环境	1	2	3
无有效的验证结果, 或不再失败	2	1	3
列入SWE-Residual	165	33	198

表5:通过人文审查的候选人的重建和鉴定.

B.4 Information Available to the Evaluated Agent

B.4 被评智能体可得信息

Table 6 summarizes what the evaluated agent receives under each condition. The two conditions differ only in the starting state and in the earlier work it contains.

表6汇总了被评估的制剂在每种条件下得到的惠益。 这两个条件仅在起始状态和它包含的早期工作上有所不同.

Item	Clean Start	Residual Start
Starting state	Original repository or machine	Residual State
Request	Original request	Original request
System prompt, harness, and budget	Source benchmark defaults	Source benchmark defaults
Earlier work inside the state	None	Retained, including code, tests, documentation, and notes written by the predecessor
Predecessor trajectory, summaries, or handoff notes	–	Not explicitly supplied; traces left in the state, such as shell history, may be accessible
Indication that an earlier attempt exists or failed	–	Not provided
Stage III review materials (trajectory summaries)	–	Not provided
Hidden tests and reference solution	Not provided	Not provided

Table 6: Information available to the evaluated agent under Clean Start and Residual Start. Both conditions use the same request, system prompt, harness, and budget.

项目	清除开始	剩余启动
开始状态	原始仓库或机器	残留状态
请求	原始请求	原始请求
系统迅速、利用和预算	源基准默认值	源基准默认值
州内较早的工作	无	保留,包括编码、测试、文件和前身编写的说明
预留器轨迹、摘要或交接便条	- 说吧	未明确提供; 状态中留下的痕迹, 如 shell 历史, 可访问
表明先前的尝试存在或失败	- 说吧	85. 未提供
第三阶段审查材料(讲座摘要)	- 说吧	未提供
隐藏测试和参考解决方案	未提供	未提供

表6:在 " 清洁启动 " 和 " 残留启动 " 项下向受评估的制剂提供的信息。 这两种条件都使用同样的要求、系统即时、利用和预算。

Appendix C Evaluation Configuration and Metrics

附录 C 评测配置与指标

C.1 Agents and Harnesses

C.1 智能体与框架

Table 7 lists the models used as predecessor agents and as evaluated agents. Evaluated agents use mini-swe-agent on DeepSWE and Terminus-2 on Terminal-Bench, with all runs orchestrated through Harbor, following the evaluation setups of the source benchmarks. Predecessor agents use the same harnesses in Stage I. Evaluated agents run under the default per-run budget of each source benchmark, identical for Clean Start and Residual Start. Predecessor agents instead run under a generous time limit of eight hours per run, so that as many attempts as possible reach a submission.

表7列出了用作前置制剂和被评价制剂的模型。 评估的特工在DeepSWE和Terminus-2上使用微型瑞典特工,在评估源基准设置后,所有特工都通过港湾运行。 Precessor代理在第一阶段使用同样的套装。 评价的代理在每一个源基准的默认每个运行预算下运行,与清洁启动和残余启动相同。 而先发制人智能体的运行时间是每次8小时的宽限期,以便尽可能多的尝试达到呈文.

We distinguish task failures from infrastructure failures. If an evaluated agent exhausts its budget without solving the task, the run counts as a task failure. If a run is interrupted by an infrastructure problem, such as a transient network, model-service, or container failure, we restart it from an independent instance of the same initial state. If verification fails for infrastructure reasons after the agent has finished, we rerun verification on the saved final state when possible and otherwise repeat the run from its initial state. A verifier error is never recorded as a task failure. The same policy applies to Clean Start and Residual Start.

我们区分任务失败和基础设施失败。 如果被评价的智能体在未解决任务的情况下耗尽预算,运行算作任务失败. 如果运行被一个视情况变化的网络,模型服务,或者容器故障等基础设施问题打断,我们从同一个初始状态的独立实例重新开始运行。 如果在代理完成后由于基础设施原因核查失败,我们尽可能重新对保存的最终状态进行核查,否则重复运行其初始状态。 核查错误从未记录为任务失败 。 同样的政策也适用于清洁启动和残余启动。

Role	Model	Reasoning and sampling settings
Predecessor	DeepSeek-V4-Flash-0731	Highest reasoning effort; default sampling (temperature 1.0)
Predecessor	Qwen-3.8-27B	Highest reasoning effort; default sampling (temperature 1.0)
Predecessor	GLM-5.3	Highest reasoning effort; default sampling (temperature 1.0)
Evaluated	Gemini-3.8-Flash	Highest reasoning effort; default sampling (temperature 1.0)
Evaluated	Qwen-3.8-Flash	Highest reasoning effort; default sampling (temperature 1.0)
Evaluated	GLM-5.3-Flash	Highest reasoning effort; default sampling (temperature 1.0)
Evaluated	GPT-6-Sol	Highest reasoning effort; default sampling (temperature 1.0)
Evaluated	Kimi-K2.8-Preview	Highest reasoning effort; default sampling (temperature 1.0)

Table 7: Models used in SWE-Residual. Predecessor agents produce Residual States in Stage I; evaluated agents are run from Clean Starts and Residual Starts.

作用	型号	理由和取样设置
预处理器	深层Seek-V4-Flash-0731	最高推理努力;默认取样(温度1.0)
预领先者	文-3.8-27B号	最高推理努力;默认取样(温度1.0)
预处理器	GLM-5.3号卫星	最高推理努力;默认取样(温度1.0)
评估	双子座-3.8-闪光	最高推理努力;默认取样(温度1.0)
评估	Quen-3.8-闪光	最高推理努力;并对其进行默认取样(温度1.0)
评估	GLM-5.3-闪光	最高推理努力;默认取样(温度1.0)
评估	GPT-6- 苏尔	最高推理努力;默认取样(温度1.0)
评估	K即便是K. 2.8-预览	最高推理努力;默认取样(温度1.0)

表7:SWE-Residual中使用的模式. 预处理剂在第一阶段产生残留状态;经评估的制剂来自清洁启动和残留启动。

C.2 Metrics

C.2 指标

Let $\mathcal{T}$ be the source tasks covered by a track and $N=|\mathcal{T}|$. For task $i$, let $\mathcal{R}_{i}$ be its set of Residual States, with $K_{i}=|\mathcal{R}_{i}|\in\{1,2,3\}$, and let $y_{A,i}(S)\in\{0,1\}$ be the verifier outcome of a run of agent $A$ from state $S$ (Section 3.1). The Residual Start success rate averages outcomes within each task and then across tasks, so that every source task receives equal weight:

让$\mathcal{T}$成为轨道和$N=|\mathcal{T}|$覆盖的来源任务. 对于任务$i$,让$\mathcal{R}_{i}$成为它的一组残余国家,拥有$K_{i}=|\mathcal{R}_{i}|\in\{1,2,3\}$,让$y_{A,i}(S)\in\{0,1\}$成为从$S$州运行的$A$剂的核实结果(第3.1节)。 剩余启动成功率平均为每项任务的结果,然后是跨任务的结果,因此每项源任务都得到同等的权重:

$$\mathrm{Res}(A)=\frac{1}{N}\sum_{i\in\mathcal{T}}\frac{1}{K_{i}}\sum_{S\in\mathcal{R}_{i}}y_{A,i}(S). \tag{3}$$

The matched Clean Start reference uses the same tasks:

匹配的 Clean Start 参考使用同样的任务 :

$$\mathrm{Clean}(A)=\frac{1}{N}\sum_{i\in\mathcal{T}}y_{A,i}(S_{0,i}). \tag{4}$$

The Residual Gap is $100\cdot(\mathrm{Clean}(A)-\mathrm{Res}(A))$ percentage points, computed before rounding.

剩余差距为$100\cdot(\mathrm{Clean}(A)-\mathrm{Res}(A))$个百分点,在四舍五入前计算。

Conditional failure rate.

Section 5.3 restricts the analysis to tasks that the agent completes from a Clean Start, $\mathcal{C}_{A}=\{i\in\mathcal{T}:y_{A,i}(S_{0,i})=1\}$, and reports

有条件故障率。

第5.3节将分析限制在毒剂从$\mathcal{C}_{A}=\{i\in\mathcal{T}:y_{A,i}(S_{0,i})=1\}$清洁启动完成的任务,并报告

$$\mathrm{CondFail}(A)=\frac{1}{|\mathcal{C}_{A}|}\sum_{i\in\mathcal{C}_{A}}\frac{1}{K_{i}}\sum_{S\in\mathcal{R}_{i}}\bigl(1-y_{A,i}(S)\bigr). \tag{5}$$

Because $\mathcal{C}_{A}$ depends on the agent, each agent is evaluated on its own task set.

因为$\mathcal{C}_{A}$依赖于代理,所以每个代理都根据自己的任务集进行评估.

Predecessor groups.

Table 8 pools DeepSWE and Terminal-Bench and weights each source task equally, so the Overall column covers all 111 source tasks. For a predecessor family $g$, the group contains the tasks with at least one Residual State from $g$; within each such task, only the states produced by $g$ are averaged, and the Clean Start reference is computed on the same task set. Groups therefore cover different tasks, and their absolute scores are not directly comparable.

能使预赛者组。

表8 集合 DeepSWE 和 Terminal-Bench 以及权重均匀的每个源任务,因此总体列涵盖了所有111个源任务. 对于一个前家庭$g$,该组包含着至少一个来自$g$的残留状态的任务;在每项任务中,只有$g$产生的状态是平均的,而"干净起步"的参考依据是相同的任务集计算. 因此,团体承担不同的任务,其绝对分数不具有直接可比性。

Appendix D Measuring Acceptance Bias

附录 D 测量接受偏置

The analysis in Section 5.4 is limited to DeepSWE, where per-test verifier outcomes, final patches, and reference solutions are available for every run.

第5.4节中的分析仅限于DeepSWE,每跑一次都提供每个测试的核查结果、最后补丁和参考解决方案。

D.1 Run Selection

D.1 运行选择

Table 4 uses normally submitted runs, i.e., runs that end with the agent declaring the task complete and for which the verifier returns a valid result. These include both successful runs and valid failures, and exclude runs that exhaust the budget or end in an infrastructure failure. In the success rates of Table 3, runs that exhaust the budget count as task failures. Each metric is computed per run, averaged within each task, and then averaged across tasks. Steps is the exception and is reported as a median ratio.

表4使用通常提交的运行,即以智能体宣布任务完成并核查者返回有效结果的方式运行该端。 其中包括成功运行和有效运行失败,不包括用尽预算或因基础设施故障而结束的运行。 在表3的成功率中,将预算算作任务失败。 每个度量衡按每次运行计算,在每项任务内平均,然后按任务平均。 步骤是例外,按中位数比率报告。

D.2 Metric Definitions

D.2 指标定义

For a state $S$, let $\mathcal{F}(S)$ denote the set of verifier tests that fail on $S$. A continuation from a Residual State $S_{r}$ ends in a final state $S^{\prime}$ and is a failed continuation if the verifier outcome is 0.

对于一个状态$S$,让$\mathcal{F}(S)$表示在$S$上失败的一组验证测试. 残留状态$S_{r}$的延续鹊在最终状态$S^{\prime}$中结束,如果验证结果为0.

Same failing.

The fraction of failed continuations with $\mathcal{F}(S^{\prime})=\mathcal{F}(S_{r})$, i.e., that end with exactly the tests that failed in the Residual State.

同样的失败。

使用$\mathcal{F}(S^{\prime})=\mathcal{F}(S_{r})$的失败续作的一小部分,即以在残留国失败的测试为确切的结束。

Initially failing.

For each Residual State $S_{r}$, the fraction of tests in $\mathcal{F}(S_{r})$ that pass at the end of the agent’s Clean Start run on the same task (Clean) and at the end of its continuation from $S_{r}$ (Residual). This column covers all normally submitted runs, including successful ones; the repair rate reported in Section 5.4 is the same quantity restricted to failed continuations.

最初失败。

对于每个残余国家$S_{r}$,在$\mathcal{F}(S_{r})$试验中,在毒剂清洁启动(Clean)结束时和在从$S_{r}$(residual)继续试验结束时,通过试验的一小部分在同一任务上运行。 本栏涵盖所有正常提交的运行量,包括成功运行量;第5.4节中报告的修复率与限于失败的续航量相同。

Kept.

For a continuation from $S_{r}$, let $P$ be the set of text lines that the predecessor added or replaced relative to the base commit; lines the predecessor deleted are not included. Every DeepSWE Residual State has a non-empty $P$. Kept is the fraction of lines in $P$ that remain in the diff of the final state against the base commit, using exact, order-preserving line matching within each file. Table 4 reports Kept over failed continuations.

{\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}别这样

对于$S_{r}$的续作,让$P$成为前身相对于基数承诺所添加或替换的一组文字行;删除的行不包含. 每个 DeepSWE 残留状态都有一个非空的$P$。 Kept 是$P$中保留在最终状态的 diff 中的线条的一小部分, 以精确的, 保留顺序的线条匹配于每个文件中。 表4报告拖延了持续失败。

Left-out files.

For a Residual State $S_{r}$, let $L$ be the set of files changed by the reference solution but not by the predecessor. We consider states with non-empty $L$ for which both the agent’s Clean Start run and its continuation from $S_{r}$ are normally submitted. Clean is the fraction of Clean Start runs that change at least one file in $L$; Residual is the fraction of continuations that do so. A continuation opens a file in $L$ if one of its actions exposes the file’s contents to the agent. This includes content-reading commands such as cat, sed, head, and tail, grep or rg output that shows file contents, and editor views recorded by the harness. Commands that only list paths, such as ls, find, rg --files, or grep -l, do not count. Neither do git status and git diff --stat; a git diff counts only if its output shows the file’s contents.

左倾文件。

对于一个剩余国$S_{r}$,让$L$成为参考解决方案更改的一组文件,而不是前身更改的一组文件。 我们考虑的是非空的$L$状态, Clean是Clean Start运行中至少改变$L$中一个文件的一小部分;剩余部分是这样做的续作的一小部分. 继续打开$L$中的文件, 如果其中一项动作将文件内容暴露给代理 。 这包括内容读取命令,如猫,塞德,头部,以及尾部,grep或rg输出,显示文件内容,以及编程器记录的编辑视图. 只列出路径的命令, 如 Is, 查找, rg -- files, 或 grep - l, 不计数 。 git 状态和 git diff - stat 都不是; git diff 只有在输出显示文件内容时才算数 。

Steps.

For each continuation, the ratio of its step count to the step count of the agent’s Clean Start run on the same task, where a step is one recorded agent turn, as in Table 2. We average the ratios over the Residual States of each task and report the median over tasks, using normally submitted continuations.

步骤。

对于每次续作,其步数与代理商的"清洁启动"步数之比都运行在同一任务上,其中一步是记录的代理转折,如表2. 我们利用通常提交的续稿,平均衡量每项任务在剩余国家的比率,并报告相对于任务的中位数。

Inspection of inherited changes.

A continuation inspects the inherited changes if it runs git status or git diff, or reads a file that the predecessor modified.

• 检查继承的变化。

如果运行 git 状态或 git diff , 或读取前身修改的文件, 继续检查所继承的更改 。

Split by Kept.

To compare continuations within an agent, we compute Kept for every normally submitted continuation, including successful ones, and split them at the agent’s median.

由 Kept 分割 。

为了比较代理商内部的续作,我们计算每一次正常提交的续作,包括成功的续作,然后按代理商的中位数进行分割。

Appendix E Additional Results

附录 E 补充结果

E.1 Performance by Predecessor Agent

E.1 按前任智能体的表现

Table 8 reports the per-predecessor results discussed in Section 5.2: Residual Start and matched Clean Start success for each evaluated agent, overall and within the states produced by each predecessor agent.

表8汇报了第5.2节讨论的每个预处理器的结果:每个被评估的制剂的残留启动和对应的清洁启动成功率,总体以及每个前制剂产生的状态。

Evaluated agent	Overall	Predecessor agent
DeepSeek	Qwen	GLM
Clean	Residual	Clean	Residual	Clean	Residual	Clean	Residual
Gemini-3.8-Flash	67.6	18.9	62.0	19.0	68.7	16.9	44.4	5.6
Qwen-3.8-Flash	47.7	27.2	40.5	21.5	51.8	32.5	38.9	8.3
GLM-5.3-Flash	63.1	15.2	57.0	11.4	62.7	15.7	33.3	2.8
GPT-6-Sol	66.7	34.1	63.3	32.9	62.7	34.9	47.2	19.4
Kimi-K2.8-Preview	55.0	14.4	51.9	12.7	55.4	09.6	38.9	11.1

Table 8: Performance by predecessor agent, pooling DeepSWE and Terminal-Bench. All values are success rates (%). Each group uses its own matched Clean Start reference, with each source task weighted equally. Overall includes all Residual States; the remaining groups include states produced by the indicated predecessor agent. DeepSeek, Qwen, and GLM denote DeepSeek-V4-Flash-0731, Qwen-3.8-27B, and GLM-5.3, respectively.

评估智能体	总表	事前智能体
深层搜索	温	GLM 图像
干净	剩余	干净	剩余	干净	剩余	干净	剩余
双子座-3.8-闪光	67.6	18.9	62.0	19.0	68.7	16.9	44.4	5.6
Quen-3.8-闪光	47.7	27.2	40.5	21.5	51.8	32.5	38.9	8.3
GLM-5.3-闪光	63.1	15.2	57.0	11.4	62.7	15.7	33.3	2.8
GPT-6- 苏尔	66.7	34.1	63.3	32.9	62.7	34.9	47.2	19.4
Kimi- K2.8- 预览	55.0	14.4	51.9	12.7	55.4	09.6	38.9	11.1

表8 按前身代理,集合DeepSWE和终端-奔驰的性能. 所有数值均为成功率(%)。 每个组使用自己匹配的"清洁启动"参考,每个源任务均匀加权. 总体而言,这包括所有剩余国家;其余类别包括指定的前身智能体产生的国家。 DeepSeek,Quen,和GLM分别表示DeepSeek-V4-Flash-0731,Quen-3.8-27B和GLM-5.3.

Appendix F Limitations

附录 F 局限

Our measurements cover 198 states from 111 tasks, two source benchmarks, and three predecessor families. Extending the collection to human-produced work and other handoff settings would show how the observed gaps vary with the source of inherited work. The acceptance-bias analysis uses DeepSWE, where per-test outcomes and final patches are available.

我们的测量包括111项任务中的198项状况、两个来源基准和三个前家庭。 将收集工作扩大到人类生产的工作和其他交接环境,将表明所观察到的差距与继承工作的来源有何不同。 验收-偏差分析采用DeepSWE,其中有每个测试结果和最终补丁.