换外壳几乎不抬通过率，却能把账单差到三倍

What Does a Harness Buy? Tokens, Mostly.

外壳到底买到了什么？主要是 Token。

Yangze Liu Affiliation: Shandong University Email: yangze2@illinois.edu Zhongyi Han † † thanks: Corresponding author. Affiliation: Shandong University Email: zhongyi.han@sdu.edu.cn

Abstract

摘要

\times

\times

1 Introduction

1 引言

Coding agents have turned into a product category, and the product is a model plus a harness. The harness, which part of the literature calls a scaffold, is everything around the model: the system prompt, the tools it can call, the way tool output is fed back, and the way the conversation is trimmed when it grows too long. It is also the part that changes most: popular harnesses ship more than two releases a day (Ben Sghaier et al., 2026), most of it prompts, tool surfaces, and context management rather than the model.

编程智能体已经变成一个产品品类，而产品就是「模型 + 外壳」。外壳在部分文献里也叫脚手架，指模型周围的一切：系统提示、可调用的工具、工具输出如何回灌，以及对话过长时如何裁剪。它也是变化最快的部分：流行外壳每天发版超过两次（Ben Sghaier 等，2026），多数更新是提示词、工具界面与上下文管理，而不是模型本身。

\times

\times

\times

\times

We measure it. We take three production harnesses as they ship, Claude Code, mini-SWE-agent, and OpenCode (Anthropic, 2026; Lieret et al., 2026; Anomaly Innovations, 2026), pin their versions, and run the same models through all three on SWE-bench Verified (Jimenez et al., 2024; OpenAI, 2024). The models are Qwen3.6-35B-A3B, Qwen3.8-27B, DeepSeek-V4-Flash, GLM-5.3-Flash, and HY4-Preview, with Claude Opus 5 added inside Claude Code as a strong anchor. Of the 492 Verified tasks that run in our offline setup, two sets play different roles. The 45 tasks in the two hardest annotated difficulty bands, all of them, are where we expect a harness to matter most. The remaining 447 give the power to bound a small effect. Ten rerun pairs of the same configuration let the harness gap be read next to the rerun gap, all runs are offline, and every arm is billed under one price list from its own token counts.

我们来测。取三套按原样出货的生产外壳——Claude Code、mini-SWE-agent、OpenCode（Anthropic，2026；Lieret 等，2026；Anomaly Innovations，2026）——钉死版本，让同一批模型在三套外壳上跑 SWE-bench Verified（Jimenez 等，2024；OpenAI，2024）。模型为 Qwen3.6-35B-A3B、Qwen3.8-27B、DeepSeek-V4-Flash、GLM-5.3-Flash、HY4-Preview，并在 Claude Code 内加入 Claude Opus 5 作为强锚点。线下环境能跑通的 492 道 Verified 题里，两套集合角色不同：标注难度最高的两档共 45 道题，是我们预期外壳最该起作用的地方；其余 447 道用来给小效应定界。十组同一配置的重跑对，让外壳差距可以紧挨着重跑差距来读；全部离线；每一臂按自己的 token 计数、在同一价目表下计费。

Held to that standard, the harness barely moves pass rate (Figure 1).

按这个标准，外壳几乎推不动通过率（图 1）。

(a) (b) Figure 1: Same model, different harness. (a) Input tokens per task against pass rate on the 45 hard tasks, one run of every model under every harness. Within a model the harnesses spread two- to three-fold in tokens and stay inside the gap that 45 tasks catch only half the time. (b) Paired pass-rate difference for each pair of harnesses on each model and task set, with 95% intervals for the paired difference; vertical ticks mark the rerun range where a configuration was run more than once. Filled points are the pairs that clear the noise, and all of them are OpenCode trailing on the 447-task pool.

(a) (b) 图 1： 同一模型、不同外壳。 (a) 在 45 道难题上，每题输入 token 对通过率；每个模型在每套外壳下各跑一次。模型内部，外壳之间的 token 可差两到三倍，通过率仍落在「45 道题只有一半机会抓住」的差距之内。(b) 每个模型、每套任务上，两两外壳的成对通过率差，并给出成对差的 95% 区间；竖线标出同一配置多次重跑的范围。实心点是压过噪声的配对，全部是 OpenCode 在 447 题池上落后。

On the hard tasks, the spread across the three harnesses is two to five tasks out of 45 for every model, and rerunning one harness moves it by up to three. The two flip the same share of tasks, 13% at the median, and the tasks a harness wins in one run are not the tasks it wins in the next. On the 447 tasks, Claude Code and mini-SWE-agent are equivalent within five points on both models we ran there. We also looked where a harness effect should hide if it existed: a vendor’s own harness on its own model scored inside the range of the other three.

在难题上，三套外壳对每个模型的差距是 45 题里差 2-5 题，而重跑一套外壳最多也能挪 3 题。两者翻掉的任务比例中位数同为 13%；某一外壳这一轮赢的题，下一轮未必还是那些。在 447 题上，Claude Code 与 mini-SWE-agent 在我们跑过的两个模型上都相差不超过五分。我们也查了「若真有外壳效应该藏在哪」：厂商用自家外壳跑自家模型，成绩仍落在另外三套外壳的区间内。

What does clear the noise is a loss. OpenCode trails by up to 9 points on the large pool, and half of that gap sits in runs its output cap cut short before a patch was written. Every other harness failure we can name is of the same kind: a hard stop at the context window, a crash at the step cap, a search tool that could not run offline. Every harness effect we can name is a way to lose a task, and none we measured is a way to win one.

真正压过噪声的是损失。OpenCode 在大池上最多落后 9 分，约一半差距来自输出上限在写出补丁前就把运行截断。我们能点名的其他外壳失败也是同一类：撞上上下文窗口硬停、步数上限崩溃、搜索工具离线跑不起来。能叫得出名字的外壳效应，都是「怎样丢掉一道题」，没有我们测到「怎样多赢一道题」。

\times

\times

The rerun data also settle what a harness comparison can resolve. At the discordance we observe, 45 tasks catch a 13-point gap half the time and no gap at 80% power, and 447 tasks resolve about 5 points. Many of the published harness gains sit below the first number, and some sit below the second as well.

重跑数据也框定了这类对比能分辨什么。按观测到的不一致率，45 道题一半机会抓住 13 分差距，80% 功效下抓不住零差距；447 道题大约能分辨 5 分。许多已发表的外壳增益低于前一个数，有的连后一个数也够不上。

2 Related Work

2 相关工作

\times

\times

What is inside a harness. A source-code taxonomy of coding agents lists twelve design dimensions and asks for exactly the control we use, a fixed model with the scaffold varied (Rombaut, 2026). The three harnesses we run sit at different points of that taxonomy. mini-SWE-agent gives the model a single bash tool and no context management, and reaches SWE-agent’s score in about a hundred lines of Python (Yang et al., 2024; Lieret et al., 2026). Claude Code and OpenCode add typed file tools, long system prompts, and automatic compaction (Anthropic, 2026; Anomaly Innovations, 2026). Component studies show that individual pieces matter under pressure: a context-management package nearly doubles a small model’s score in a 20k-token window (Lewis, 2026), and restricting the tool surface changes cost by task regime and agent while pass rates stay tied (Yang et al., 2026). A component study on SWE-bench Verified varies context management, action space, and planning inside one lightweight harness over 176 matched settings, and finds that context management pays as the window tightens and that capable models do as well with a bash-only tool set at lower cost (Fan et al., 2026). We ask the complementary question: what the whole production harness, as users run it, changes against a rerun of itself. Models fine-tuned on one scaffold’s trajectories lose as much as 29 points when moved to another, while base models vary far less (Thangarajah et al., 2026). Our check of a vendor’s own harness on its own model is the same question asked of an API model.

外壳里面有什么。 一份编程智能体的源码分类列出十二个设计维度，并恰好要求我们这种对照：固定模型、只变脚手架（Rombaut，2026）。我们跑的三套外壳落在该分类的不同位置。mini-SWE-agent 只给模型一个 bash 工具、不做上下文管理，用大约百行 Python 就能摸到 SWE-agent 的分数（Yang 等，2024；Lieret 等，2026）。Claude Code 与 OpenCode 增加了带类型的文件工具、长系统提示，以及自动压缩（Anthropic，2026；Anomaly Innovations，2026）。组件研究显示，压力下单件也会起作用：一套上下文管理包在 20k token 窗口里几乎把小模型分数翻倍（Lewis，2026）；收紧工具面会按任务形态与智能体改变成本，通过率却仍绑在一起（Yang 等，2026）。另一项在 SWE-bench Verified 上的组件研究，在轻量外壳内对 176 组匹配设置变动上下文管理、动作空间与规划，发现窗口收紧时上下文管理更值钱，而强模型用纯 bash 工具集也能以更低成本打出相近通过率（Fan 等，2026）。我们问的是互补问题：用户原样跑的整套生产外壳，相对「重跑它自己」到底改变了什么。在某一脚手架轨迹上微调的模型，换到另一脚手架可掉多达 29 分，而基座模型波动小得多（Thangarajah 等，2026）。我们检查「厂商自家外壳 + 自家模型」，问的是同一问题在 API 模型上的版本。

Noise and power in agent evaluation. The case for error bars on evaluations has been made for language models in general (Miller, 2024) and for agents in particular, where a score reported without its cost or its variance rewards the wrong things (Kapoor et al., 2025). Ten reruns of one model-scaffold pair on SWE-bench Verified spread by 2.2 to 6.0 points, with standard deviations above 1.5 points even at temperature zero, so reported gains of 2 to 3 points can be noise (Bjarnason et al., 2026). A generalizability-theory decomposition finds that the agent main effect explains under 3% of outcome variance while agent-by-task interaction explains 7 to 23% (Srinivasan, 2026), and a pre-registered comparison on GAIA fits the scaffold-by-model interaction directly and finds it significant (Starace, 2026). We take the rerun spread as the yardstick, and report every harness gap next to it with the sample size it would take to resolve.

智能体评测中的噪声与功效。 评测要带误差条的论证，对语言模型总体成立（Miller，2024），对智能体尤其成立：只报分数、不报成本与方差，会奖励错误的东西（Kapoor 等，2025）。同一模型-脚手架对在 SWE-bench Verified 上重跑十次，分数散开 2.2-6.0 分，即便温度为零标准差也常高于 1.5 分，因此宣称涨 2-3 分可能只是噪声（Bjarnason 等，2026）。用概化理论分解发现，智能体主效应解释不到 3% 的结果方差，智能体\times任务交互却解释 7-23%（Srinivasan，2026）；一项在 GAIA 上预注册的对比直接拟合脚手架\times模型交互并发现显著（Starace，2026）。我们把重跑离散当作尺子，每个外壳差距都挨着它报告，并给出要分辨它需要多大样本量。

3 Setup

3 实验设置

Harnesses. We run three production harnesses as they ship, with the version pinned per arm: Claude Code 2.1 (releases 2.1.259 to 2.1.273 in the scored arms; 2.1.263 on GLM and Opus), mini-SWE-agent 2.4.6, and OpenCode 1.18 (1.18.30 and 1.18.31 in the scored arms; 1.18.30 on GLM) (Anthropic, 2026; Lieret et al., 2026; Anomaly Innovations, 2026). The three span the design space of Section 2: one minimal harness and two full products, one closed and one open. They differ in what they give the model. mini-SWE-agent exposes one bash tool, trims only very long observations, and has no context management beyond a hard stop at the model’s window. Claude Code and OpenCode expose typed file, search, and edit tools, prepend long system prompts, and compact the transcript automatically when it grows. None of them sends sampling parameters, so decoding is whatever the provider does by default. We hold one budget fixed across all three: a step is one model call, and every trial gets 300 of them. Wall-clock limits are a safety net, and on the locally served Qwen arms they are not equal: Claude Code, whose steps against vLLM run about twice as slow, could run for 36,000 s where OpenCode’s own soft timeout stopped it at 5,760 s (Appendix A). On the GLM model, where the API accepts a reasoning-effort setting, all three harnesses pin it to high. Everything else is left as the harness authors set it, and Table A1 in the appendix lists what that is. We also run DeepSeek’s own command-line harness, dsh in the tables, on DeepSeek’s model as a home-field check.

外壳。 我们按出货原样跑三套生产外壳，并按臂钉死版本：Claude Code 2.1（计分臂为 2.1.259-2.1.273；GLM 与 Opus 用 2.1.263）、mini-SWE-agent 2.4.6、OpenCode 1.18（计分臂为 1.18.30 与 1.18.31；GLM 用 1.18.30）（Anthropic，2026；Lieret 等，2026；Anomaly Innovations，2026）。三者覆盖第 2 节的设计空间：一套最小外壳与两套完整产品，一闭源一开源。它们给模型的东西不同。mini-SWE-agent 只暴露一个 bash 工具，只裁极长观测，除模型窗口硬停外没有上下文管理。Claude Code 与 OpenCode 暴露带类型的文件、搜索与编辑工具，前置长系统提示，并在 transcript 变长时自动压缩。三者都不传采样参数，解码完全跟厂商默认。我们把预算在三者间钉死：一步等于一次模型调用，每次试验 300 步。墙钟限制是安全网；本地服务的 Qwen 臂上并不相等：Claude Code 对 vLLM 的每步大约慢一倍，可跑到 36,000 s，而 OpenCode 自带软超时在 5,760 s 就停（附录 A）。在接受 reasoning-effort 设置的 GLM 模型上，三套外壳都钉在 high。其余一律按外壳作者默认，附录表 A1 列出具体项。我们还把 DeepSeek 自己的命令行外壳 dsh（表中简称）跑在 DeepSeek 模型上，作为主场对照。

Models. The ladder has five models that run in all three harnesses: Qwen3.6-35B-A3B and Qwen3.8-27B, served locally with vLLM 0.27.1 in bfloat16 precision, and DeepSeek-V4-Flash (release 0731), GLM-5.3-Flash, and HY4-Preview, called through their vendor APIs (DS Flash, GLM Flash, and HY4 for short in figures and wide tables). The three API models are among the most used on OpenRouter: in its trailing-30-day ranking by tokens processed through 2026-09-20, DeepSeek-V4-Flash-0731 ranks second, HY4-Preview third, and GLM-5.3-Flash fourth (OpenRouter, 2026). Claude Opus 5 runs inside Claude Code only, through the harness’s own channel, and anchors the top of the ladder. Harbor 0.22.0 (Harbor Framework Team, 2026) orchestrates the runs, one container per trial, and runs the benchmark’s own tests to score each one.

模型。 阶梯上有五个在三套外壳都跑的模型：用 vLLM 0.27.1、bfloat16 本地服务的 Qwen3.6-35B-A3B 与 Qwen3.8-27B；以及经厂商 API 调用的 DeepSeek-V4-Flash（0731 版）、GLM-5.3-Flash、HY4-Preview（图与宽表中简称 DS Flash、GLM Flash、HY4）。三个 API 模型都在 OpenRouter 用量前列：按截至 2026-09-20 的近 30 天 token 处理量，DeepSeek-V4-Flash-0731 第二，HY4-Preview 第三，GLM-5.3-Flash 第四（OpenRouter，2026）。Claude Opus 5 只在 Claude Code 内经其自有通道运行，锚定阶梯顶端。Harbor 0.22.0（Harbor Framework Team，2026）编排运行，每次试验一个容器，并用基准自带测试打分。

Tasks. SWE-bench Verified (Jimenez et al., 2024; OpenAI, 2024) has 500 tasks, and 8 of them need a public web service to verify and are dropped, leaving 492. From these we take two disjoint sets. hard45 is every task in the two hardest annotated difficulty bands, 1 to 4 hours and over 4 hours, 42 and 3 tasks, taken whole so that there is no sampling to tune. pool447 is the remaining 447 tasks. Every arm on hard45 runs the same 45 tasks. On pool447 we run the two Qwen models, which we serve ourselves. The agent phase has no network beyond the model endpoint. An audit of our early web-enabled runs found that 71% of DeepSeek and Qwen trials had fetched the upstream fix from GitHub, so those runs are discarded and the condition is enforced at the network layer (Appendix D). Offline, OpenCode’s search tools fail because they download a binary at first use. We place it ahead of time and report that arm, marked rg in the tables.

任务。 SWE-bench Verified（Jimenez 等，2024；OpenAI，2024）共 500 道题，其中 8 道需要公网服务才能验证，去掉后剩 492。我们取两个不相交集合。hard45 是标注难度最高两档的全部题目（1-4 小时与超过 4 小时，分别为 42 与 3 道），整档纳入，不做可调采样。pool447 是其余 447 道。hard45 上每一臂跑同一 45 题。pool447 上我们跑自己服务的两个 Qwen 模型。智能体阶段除模型端点外无外网。对早期开网运行的审计发现，71% 的 DeepSeek 与 Qwen 试验从 GitHub 拉取了上游修复，故那些运行作废，并在网络层强制离线（附录 D）。离线时 OpenCode 的搜索工具会因首次使用要下载二进制而失败；我们预先放好二进制并报告该臂，表中标为 rg。

\delta

\delta

4 Results

4 结果

4.1 The harness barely moves pass rate

4.1 外壳几乎推不动通过率

A-B

Model	Pair $A-B$	$n$	$A$	$B$	$b$	$c$	$\Delta$ (pp)	McNemar $p$	Verdict
Qwen3.6-35B-A3B	CC $-$ mini	445	310	318	26	34	$-$ 1.8 [ $-$ 5.2, +1.6]	.37	equivalent
	CC $-$ OC	445	310	297	43	30	+2.9 [ $-$ 0.9, +6.7]	.16	inconclusive
	mini $-$ OC	445	318	297	45	24	+4.7 [+1.1, +8.4]	.02	inferior $B$
Qwen3.8-27B	CC $-$ mini	444	347	353	17	23	$-$ 1.4 [ $-$ 4.2, +1.5]	.43	equivalent
	CC $-$ OC	444	347	312	52	17	+7.9 [+4.3, +11.5]	$<$ .001	inferior $B$
	mini $-$ OC	445	353	312	57	16	+9.2 [+5.5, +12.9]	$<$ .001	inferior $B$

A-B

模型	配对 $A-B$	$n$	$A$	$B$	$b$	$c$	$\Delta$ (pp)	McNemar $p$	判定
Qwen3.6-35B-A3B	CC $-$ mini	445	310	318	26	34	$-$ 1.8 [ $-$ 5.2, +1.6]	.37	等价
	CC $-$ OC	445	310	297	43	30	+2.9 [ $-$ 0.9, +6.7]	.16	无结论
	mini $-$ OC	445	318	297	45	24	+4.7 [+1.1, +8.4]	.02	$B$ 更差
Qwen3.8-27B	CC $-$ mini	444	347	353	17	23	$-$ 1.4 [ $-$ 4.2, +1.5]	.43	等价
	CC $-$ OC	444	347	312	52	17	+7.9 [+4.3, +11.5]	$<$ .001	$B$ 更差
	mini $-$ OC	445	353	312	57	16	+9.2 [+5.5, +12.9]	$<$ .001	$B$ 更差

\pm

\pm

{}^{\,\times 2}

Model	Claude Code	mini-SWE-agent	OpenCode (rg)
Qwen3.6-35B-A3B	11/45 ${}^{\,\times 2}$ 24% [14, 39]	15/45 ${}^{\,\times 2}$ 33% [21, 48]	11/45 ${}^{\,\times 2}$ 24% [14, 39]
Qwen3.8-27B	19/43 ${}^{\,\times 2}$ 44% [30, 59]	17/45 ${}^{\,\times 2}$ 38% [25, 52]	17/45 ${}^{\,\times 2}$ 38% [25, 52]
DeepSeek-V4-Flash	21/45 ${}^{\,\times 2}$ 47% [33, 61]	22/45 ${}^{\,\times 1}$ 49% [35, 63]	19/45 ${}^{\,\times 1}$ 42% [29, 57]
GLM-5.3-Flash	22/45 ${}^{\,\times 1}$ 49% [35, 63]	20/45 ${}^{\,\times 1}$ 44% [31, 59]	18/45 ${}^{\,\times 1}$ 40% [27, 55]
HY4-Preview	23/45 ${}^{\,\times 1}$ 51% [37, 65]	26/45 ${}^{\,\times 1}$ 58% [43, 71]	21/45 ${}^{\,\times 1}$ 47% [33, 61]
Claude Opus 5	36/45 ${}^{\,\times 3}$ 80% [66, 89]	—	—

{}^{\,\times 2}

模型	Claude Code	mini-SWE-agent	OpenCode (rg)
Qwen3.6-35B-A3B	11/45 ${}^{\,\times 2}$ 24% [14, 39]	15/45 ${}^{\,\times 2}$ 33% [21, 48]	11/45 ${}^{\,\times 2}$ 24% [14, 39]
Qwen3.8-27B	19/43 ${}^{\,\times 2}$ 44% [30, 59]	17/45 ${}^{\,\times 2}$ 38% [25, 52]	17/45 ${}^{\,\times 2}$ 38% [25, 52]
DeepSeek-V4-Flash	21/45 ${}^{\,\times 2}$ 47% [33, 61]	22/45 ${}^{\,\times 1}$ 49% [35, 63]	19/45 ${}^{\,\times 1}$ 42% [29, 57]
GLM-5.3-Flash	22/45 ${}^{\,\times 1}$ 49% [35, 63]	20/45 ${}^{\,\times 1}$ 44% [31, 59]	18/45 ${}^{\,\times 1}$ 40% [27, 55]
HY4-Preview	23/45 ${}^{\,\times 1}$ 51% [37, 65]	26/45 ${}^{\,\times 1}$ 58% [43, 71]	21/45 ${}^{\,\times 1}$ 47% [33, 61]
Claude Opus 5	36/45 ${}^{\,\times 3}$ 80% [66, 89]	—	—

p

p

4.2 The noise floor and what it can resolve

4.2 噪声地板与它能分辨什么

Figure 2: Swapping the harness flips as many tasks as rerunning the same harness. Each point is one pair of runs on the same tasks, and the value is the share of tasks whose outcome differs between the two. Rerun pairs share model, harness, and settings. Harness pairs share the model. Model pairs share the harness. Filled points are hard45, open points pool447; bars are medians.

图 2： 换外壳翻掉的题，和重跑同一外壳一样多。 每个点是同一批题上的一对运行，取值是两轮结果不同的题占比。重跑对共享模型、外壳与设置；外壳对共享模型；模型对共享外壳。实心点为 hard45，空心点为 pool447；横线为中位数。

n

n

The harness’s flips do not replicate at the task level either. If a harness won a fixed set of tasks, a split in run 1 would repeat in run 2. On the six harness pairs on hard45 run twice on both sides, three on each Qwen model, the split repeats no more often than it reverses: 5 tasks split the same way and 7 the opposite way, the rest tie, and a coin gives that count under either test in Appendix F. A real effect leaves a different trace: for the model swap under the same harness, 10 of 15 such tasks split the same way both times, even though Qwen3.8-27B leads Qwen3.6-35B-A3B by only two to ten tasks on this set. On hard45 the harness gap behaves like a fresh draw of the same noise, with no set of tasks that one harness wins.

外壳的翻转在任务级也不复现。若某外壳稳定赢下一组固定题，第 1 轮的分裂应在第 2 轮重现。hard45 上六组外壳对（两边都跑了两次，每个 Qwen 三个）里，分裂重现的次数并不多于反向：同向 5 题、反向 7 题，其余打平；附录 F 的两种检验下，抛硬币也能给出这个计数。真效应会留下不同痕迹：同一外壳下换模型时，15 道这类题里有 10 道两次同向分裂，尽管 Qwen3.8-27B 相对 Qwen3.6-35B-A3B 在这套题上只多赢 2-10 题。在 hard45 上，外壳差距表现得像同噪声的一次新抽样，不存在「某一外壳专赢」的题目集合。

The flip rates set what a comparison of this kind can resolve. With 14% of tasks discordant between two arms, 45 tasks catch a 12.9-point gap half the time and no gap at 80% power, and 447 tasks detect 5.2 points at that power (Figure A2 in the appendix). So the two-to-five-task gaps in Table 2 cannot be told from zero. By the same scale, a reported gain of a few points needs a paired pool larger than ours, and one read off a single run of a few dozen tasks is inside the noise.

翻转率框定了这类对比能分辨什么。两臂之间约 14% 的题不一致时，45 道题一半机会抓住 12.9 分差距，80% 功效下抓不住零差距；447 道题在该功效下能检出 5.2 分（附录图 A2）。因此表 2 里差 2-5 题的差距无法与零区分。同一尺子下，宣称涨几分需要比我们更大的成对池；从几十道题单次运行读出的增益，落在噪声里。

4.3 A harness can lose tasks

4.3 外壳可以丢掉任务

Figure 3: How trials end. One bar per model and harness, split into pass, fail with the model stopping on its own, output-token cutoff, context overflow, step or time cap or crash, and trials lost to infrastructure, which are not scored. dsh is DeepSeek’s own harness.

图 3： 试验如何结束。 每个模型与外壳一根柱，拆成：通过、模型自行停下的失败、输出 token 截断、上下文溢出、步数/时间上限或崩溃，以及未计分的基础设施丢失。dsh 是 DeepSeek 自家外壳。

The one harness effect that clears the noise is OpenCode losing tasks, and the trajectories say how (Figure 3). The first way is an output cap with no recovery: when the model’s reasoning runs past the provider’s cap in the middle of a step, OpenCode does not re-prompt, and a trial cut on its final step ends with whatever is on disk, which is usually nothing. On pool447 with Qwen3.8-27B, 33 OpenCode trials fail this way, where Claude Code, which hits the same cap on the same model and re-prompts the model to continue, loses one. Scoring only the tasks where OpenCode was not cut off roughly halves its gap to the other two harnesses (Table A7), and transplanting Claude Code’s recovery prompt into OpenCode recovers tasks at a rate the affected tasks cannot tell from zero (Appendix H). The rest of OpenCode’s gap goes with a second habit: under OpenCode the model stops sooner. On Qwen3.6-35B-A3B over the pool, where no trial is cut off, OpenCode ends after a median of 28 model calls against 71 under mini-SWE-agent and 55 under Claude Code. Its gap to mini-SWE-agent is still +4.7 points. OpenCode also stops soonest on GLM-5.3-Flash (Table A4). The data do not say what in OpenCode makes the model stop sooner, only that shorter runs and lost tasks travel together.

唯一压过噪声的外壳效应，是 OpenCode 在丢题，轨迹说明了怎么丢（图 3）。第一种方式是没有恢复的输出上限：模型推理在某一步中途撞上厂商上限时，OpenCode 不会再提示继续；若截断发生在最后一步，磁盘上通常什么补丁都没有。pool447 上 Qwen3.8-27B 有 33 次 OpenCode 试验这样失败；Claude Code 在同一模型上也会撞同一上限，但会再提示模型继续，只丢 1 次。只计 OpenCode 未被截断的题，它相对另外两套的差距大约减半（表 A7）；把 Claude Code 的恢复提示移植进 OpenCode，能挽回一些题，但受影响题数分辨不出与零的差别（附录 H）。OpenCode 其余差距对应第二种习惯：在它下面模型更早停。pool 上 Qwen3.6-35B-A3B 没有任何试验被截断，OpenCode 中位 28 次模型调用就结束，而 mini-SWE-agent 为 71、Claude Code 为 55。它相对 mini-SWE-agent 仍差 +4.7 分。OpenCode 在 GLM-5.3-Flash 上也停得最早（表 A4）。数据说不清 OpenCode 里什么让模型更早停，只说明更短的运行与丢题结伴出现。

The other failure modes are also losses, and each traces to one design choice. mini-SWE-agent has no context management, so a long trial runs into the model’s window, which on Qwen3.8-27B happens on 11 of the 45 hard tasks and 18 of the pool. Most of those overflows are symptoms: the other two harnesses mostly failed those tasks too, so a compaction step that recovered the ones another harness passed would move mini-SWE-agent by 6.7 points on the hard tasks and less on the pool, and on rerun most overflow again (Appendix K). An overflow mostly marks a task the model was already losing. The two full harnesses fail at their own limits instead: Claude Code and OpenCode compact the transcript and lose at most one trial to the window, and Claude Code loses at most two trials per arm at the step cap. OpenCode hits the wall clock on six of the 45 hard tasks on Qwen3.8-27B, where it had a sixth of Claude Code’s time, and dropping those tasks does not shrink its gap there (Table A7). OpenCode as shipped also cannot run its search tools offline, and the shipped arm sits lower than the reported arm on one Qwen model and higher on the other (same table). None of these is a way to win a task. The pairs that clear the noise, the filled points in Figure 1b, all point the same way: toward a harness losing.

其他失败模式也是损失，且各自对应一个设计选择。mini-SWE-agent 没有上下文管理，长试验会撞模型窗口：Qwen3.8-27B 上 hard45 有 11/45、大池有 18 次。多数溢出是症状：另外两套外壳在这些题上多半也失败了，因此即便压缩能挽回「另一外壳通过」的那些，hard45 上也只相当于给 mini-SWE-agent 挪 6.7 分，大池上更少；重跑时多数再次溢出（附录 K）。溢出多半标出模型已经在输的题。两套完整外壳则败在各自限制上：Claude Code 与 OpenCode 会压缩 transcript，因窗口丢掉的试验最多各 1 次；Claude Code 因步数上限每臂最多丢 2 次。OpenCode 在 Qwen3.8-27B 的 hard45 上有 6/45 撞墙钟——它只有 Claude Code 约六分之一的时间——去掉这些题也缩不了那里的差距（表 A7）。原样出货的 OpenCode 还无法离线跑搜索工具；出货臂相对报告臂，在一个 Qwen 上更低、在另一个上更高（同表）。这些都不是赢题的办法。压过噪声的配对——图 1b 的实心点——全都指向同一方向：某套外壳在丢。

4.4 The harness sets the bill

4.4 外壳定下账单

\times

\times

The per-step weight is not what the tools return (Table A4 in the appendix). Per call, the three harnesses hand the model about the same amount of text, at the mean and at the 90th percentile. What differs is the preamble each harness carries into every call, its system prompt, tool schemas, and message wrapper: 16,581 tokens in Claude Code, 7,025 in OpenCode, and 829 in mini-SWE-agent, paid again on every step. Claude Code also keeps the model’s own reasoning in the transcript, so a run accumulates many times the model text of a mini-SWE-agent run, and it takes the most steps. Adding these terms to the tool returns and call arguments approximately reconstructs the measured input per step in all three arms (Appendix G gives the gaps). What separates the harnesses’ per-step weight is their own text, not the repository’s.

每步重量不是工具返回的内容（附录表 A4）。每次调用，三套外壳交给模型的文本量在均值与 90 分位上都差不多。差别在于每一步都带着的前缀——系统提示、工具 schema 与消息包装：Claude Code 16,581 token，OpenCode 7,025，mini-SWE-agent 829，每一步再付一遍。Claude Code 还把模型自己的推理留在 transcript 里，因此一轮累计的模型文本可达 mini-SWE-agent 的许多倍，并且它步数也最多。把这些项加到工具返回与调用参数上，大致能还原三臂测得的每步输入（附录 G 给出残差）。把外壳每步重量分开的，是它们自己的文本，不是仓库的文本。

Figure 4: What the model reads at each step. Input tokens of every call against the step index, one faint line per trial and a bold median over the trials still running at that step, drawn while at least half of them are. The harnesses differ first in where the line starts, which is the preamble, and less in how fast it climbs. The cliffs are compactions.

图 4： 模型每一步读到什么。 每次调用的输入 token 对步序号；每条淡线是一次试验，粗线是该步仍在跑的试验的中位数（至少一半试验仍在跑时绘制）。外壳首先差在线条起点（前缀），爬升速度差得较少。悬崖是压缩。

Read step by step, the same accounting shows the harnesses separating at the first call (Figure 4). The input the model reads rises along a near-straight line in every arm, so a harness is described by where its line starts and how fast it climbs. The harnesses differ first in the start: on GLM-5.3-Flash the first call carries 17.1k tokens under Claude Code, 7.6k under OpenCode, and 1.4k under mini-SWE-agent, and the same three sizes on DeepSeek-V4-Flash. The slopes sit within a factor of one and a half of one another on both models, and the reasoning Claude Code keeps in the transcript accounts for part of its steeper climb (Appendix G). On DeepSeek-V4-Flash the three lines run nearly parallel, so Claude Code’s per-step input ends up tied with mini-SWE-agent’s and the bill is decided by the step count. Claude Code compacts in a fifth to a third of its trials, at the cliffs in the figure, and mini-SWE-agent never compacts and overflows instead. The first call, the growth per step, and the step count are the three numbers a harness’s bill turns on.

逐步看，同一套账说明外壳在第一次调用就分开了（图 4）。模型读到的输入在每一臂都近似直线上升，因此一套外壳可用起点与斜率刻画。差别首先在起点：GLM-5.3-Flash 上第一次调用 Claude Code 带 17.1k token，OpenCode 7.6k，mini-SWE-agent 1.4k；DeepSeek-V4-Flash 上也是同样三档。两模型上斜率彼此相差不到 1.5 倍；Claude Code 更陡的爬升，部分来自它留在 transcript 里的推理（附录 G）。DeepSeek-V4-Flash 上三条线近乎平行，于是 Claude Code 的每步输入最终与 mini-SWE-agent 持平，账单由步数决定。Claude Code 在五分之一到三分之一的试验里压缩（图中的悬崖）；mini-SWE-agent 从不压缩，而是溢出。第一次调用、每步增长、步数——这三项决定外壳的账单。

\times

\times

4.5 What we looked for and did not find

4.5 我们找过、但没找到的

A vendor’s own harness on its own model buys no score advantage. DeepSeek’s harness on DeepSeek-V4-Flash passes 21 of 45, the same as Claude Code and one fewer than mini-SWE-agent, at 1.04 CNY per trial against Claude Code’s 0.98 (Appendix B). Home field is inside the range on score and at the top of it on bill.

厂商自家外壳跑自家模型，买不到分数优势。DeepSeek 外壳在 DeepSeek-V4-Flash 上通过 21/45，与 Claude Code 相同、比 mini-SWE-agent 少 1，每试验 1.04 元对 Claude Code 的 0.98（附录 B）。主场在分数上落在区间内，在账单上落在区间顶端。

\times

\times

p

p

\times

\times

p

p

5 Discussion and Conclusion

5 讨论与结论

What a harness buys, on this evidence, is fewer ways to lose a task, and a bill, not pass rate. Every harness effect we could measure is a way to lose a task the model would otherwise solve: an output cap with no recovery, a context window with no compaction, a search tool that does not run offline. We did not find a harness winning tasks, which puts the engineering where the losses are. The one change that might move a score in our grid is a recovery step for OpenCode’s output cutoff, which Claude Code already has. Transplanted, it recovered tasks at a rate that 41 tasks cannot tell from zero, because the runaway reasoning behind the cut recurs on only about 15% of reruns of the same task. The three harnesses differ most in the parts that get the most attention, the prompt, the tool surface, the compaction policy, and none of those differences buys a score gain.

按这些证据，外壳买到的是更少的丢题路径，以及一张账单，而不是通过率。我们能量到的每一种外壳效应，都是「把模型本可解的题弄丢」：没有恢复的输出上限、没有压缩的上下文窗口、离线跑不了的搜索工具。我们没找到外壳在赢题，于是工程该盯着损失。网格里唯一可能挪分的改动，是给 OpenCode 的输出截断加恢复步——Claude Code 已有。移植后挽回的题，在 41 题尺度上分不清与零的差别，因为导致截断的失控推理大约只在同一题 15% 的重跑里再现。三套外壳差别最大的，恰是最受关注的部分——提示、工具面、压缩策略——而这些差别都买不到涨分。

The thinking interaction is the one lead we would chase. Switching thinking off on Qwen3.6-35B-A3B costs mini-SWE-agent 11 points and moves the other two by less than the noise. On HY4-Preview, however, the same switch inside Claude Code costs 11 points, as much as the widest harness gap on that model, and cuts output tokens by more than half. The two results suggest that dependence on explicit reasoning is specific to the model and harness pair, but our grid is too small to identify the mechanism.

与思考的交互是我们会继续追的线索。在 Qwen3.6-35B-A3B 上关掉思考，mini-SWE-agent 掉 11 分，另外两套动幅小于噪声。但在 HY4-Preview 上，同一开关在 Claude Code 内也掉 11 分——与该模型上最宽外壳差距同量级——并把输出 token 砍掉一半以上。两结果暗示：对显式推理的依赖是模型与外壳的配对属性；我们的网格太小，还辨不出机制。

Two limits on scope. Our results come from one benchmark. The one-model check on Terminal-Bench 2.1 (Appendix L) is a transfer check, and general-purpose agents may be a different regime. And the closed anchor model runs in Claude Code only, so the three-harness ladder stops at HY4-Preview and cannot tell whether a closed frontier model shows the same gaps.

范围有两处限制。结果来自一个基准。Terminal-Bench 2.1 上的单模型检查（附录 L）只是迁移抽查；通用智能体可能是另一套制度。闭源锚点模型只在 Claude Code 里跑，三外壳阶梯止于 HY4-Preview，无法判断闭源前沿模型是否呈现同样差距。

\times

\times

Ethics statement

伦理声明

All tasks come from the public SWE-bench Verified benchmark, and no human subjects or personal data are involved. The upstream fixes that networked agents retrieved are public GitHub content; we report retrieval rates by model to inform evaluation practice and make no claim that any vendor trained on or intended to use the test data. API keys and credentials were removed from all released files.

全部任务来自公开的 SWE-bench Verified 基准，不涉及人类受试者或个人数据。联网智能体取回的上游修复是公开 GitHub 内容；我们按模型报告拉取率以供评测实践参考，并不声称任何厂商训练过或意图使用测试数据。所有发布文件已去除 API 密钥与凭证。

Reproducibility statement

可复现性声明

Every number in the paper is generated from one table of trials by the scripts that draw the figures and tables, and Appendix A lists the harness versions, the settings each arm ran with, and how the offline condition was enforced. The trial table, the analysis scripts, and the Harbor job files, with credentials removed, are available at https://github.com/YangzeLiu/what-does-a-harness-buy . The full trajectories of the mini-SWE-agent and OpenCode runs, and the tool-call sequences and results of the Claude Code runs, will be released with the paper.

文中每个数字都由同一张试验表、经绘制图表的脚本生成；附录 A 列出外壳版本、各臂设置，以及离线条件如何强制。试验表、分析脚本与 Harbor 任务文件（已去凭证）见 https://github.com/YangzeLiu/what-does-a-harness-buy 。mini-SWE-agent 与 OpenCode 运行的完整轨迹，以及 Claude Code 运行的工具调用序列与结果，将随论文一并发布。

AI use statement

AI 使用声明

In this work, we used generative AI tools to implement standard methods, to assist with data analysis, and to draft and edit parts of the paper. The agent trajectories this paper measures are model output by design and are the object of study itself. We have reviewed all AI-assisted work: the analysis scripts were rerun by the authors against the trial table behind every table and figure, and all drafted and edited text was read and approved by the authors. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

本工作中，我们使用生成式 AI 工具实现标准方法、辅助数据分析，并起草与编辑部分文稿。本文度量的智能体轨迹本身就是模型输出，也是研究对象。我们审阅了全部 AI 辅助工作：分析脚本由作者对照支撑每张表与图的试验表重跑；所有起草与编辑文本均经作者阅读并批准。我们对最终内容负责，包括在生成式 AI 协助下产生的文本、主张或制品。

References

参考文献

Lieret et al. (2026) Kilian Lieret, John Yang, and SWE-agent contributors. mini-SWE-agent. https://github.com/SWE-agent/mini-swe-agent, 2026. Version 2.4.6. Mahmud et al. (2026) Hasan Najib Mahmud, Shreya Gupta, Isha Chaudhary, Nathaniel Enis, et al. A jagged frontier: Evaluating robustness of code agents to semantics-preserving transformations. arXiv preprint arXiv:2608.18389 , 2026. Miller (2024) Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640 , 2024. OpenAI (2024) OpenAI. Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/, 2024. OpenRouter (2026) OpenRouter. AI model rankings. https://openrouter.ai/rankings, 2026. Trailing 30 days by tokens processed, usage data through 2026-09-20; archived 2026-09-21 at https://web.archive.org/web/20260921030005/https://openrouter.ai/rankings . Rombaut (2026) Benjamin Rombaut. Inside the scaffold: A source-code taxonomy of coding agent architectures. arXiv preprint arXiv:2604.03515 , 2026. Srinivasan (2026) Vasundra Srinivasan. Deployment decision reliability: A generalizability-theory framework for sizing long-horizon agent evaluations. arXiv preprint arXiv:2608.11323 , 2026. Starace (2026) Jason Starace. Scaffold effects on GAIA: A controlled comparison. arXiv preprint arXiv:2606.08529 , 2026. Thangarajah et al. (2026) Kishanthan Thangarajah, Boyuan Chen, and Ahmed E. Hassan. DCAS: Decoupling CLI agent scaffolding to internalize planning across scaffolds. In IEEE/ACM International Conference on Automated Software Engineering (ASE) , 2026. Vats & Golev (2026) Naman Vats and Oleg Golev. The scaffold effect in coding agents: Harness choice as a hidden variable in coding-agent evaluation. arXiv preprint arXiv:2607.22585 , 2026.

Wang et al. (2025) Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations (ICLR) , 2025. Xia et al. (2025) Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying LLM-based software engineering agents. In ACM International Conference on the Foundations of Software Engineering (FSE) , 2025. Yang et al. (2026) Hong Yang, Qi Yu, and Travis Desell. When does restricting a coding agent to execute_code help? a regime × \times agent-design ablation. arXiv preprint arXiv:2607.10569 , 2026. Yang et al. (2024) John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS) , 2024. Yao et al. (2026) Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, et al. Harness-bench: Measuring harness effects across models in realistic agent workflows. arXiv preprint arXiv:2605.27922 , 2026. Zhang et al. (2026) Yunbei Zhang, Janet Wang, Jihun Hamm, Yingqiang Ge, Weijie Xu, and Chandan K. Reddy. Stop comparing LLM agents without disclosing the harness. arXiv preprint arXiv:2605.23950 , 2026. Zheng et al. (2026) Mengyu Zheng, Kai Han, et al. Claw-SWE-bench: A benchmark for evaluating OpenClaw-style agent harnesses on coding tasks. arXiv preprint arXiv:2606.12344 , 2026. Zhipu AI (2026) Zhipu AI. ZCode: the official harness for GLM-5.3. https://zhuanlan.zhihu.com/p/2070504528416337941 , 2026. Vendor post, 11 Aug 2026.

Appendix

附录

Appendix A What was held fixed

附录 A 哪些条件被钉死

Table A1 lists the configuration of each harness as run. Three settings were aligned across the harnesses on purpose; the rest were left as shipped.

表 A1 列出各外壳的实际运行配置。有三项设置是故意跨外壳对齐的；其余一律按出货原样。

\times

	Claude Code	mini-SWE-agent	OpenCode
Version (pinned per arm)a	2.1.259–2.1.273 (2.1.263 on GLM and Opus)	2.4.6	1.18.30–1.18.31 (1.18.30 on GLM)
Sampling	none sent (no flag)	none sent	none sent
Thinking / effort	GLM: effort high (CLI flag); Qwen: effort xhigh via env (vLLM compat., a no-op)b; DS: provider default	GLM: reasoning_effort = high (request body); Qwen/DS: default	GLM: reasoningEffort = high (model config); Qwen/DS: default
Tools	built-in set (Bash, Read, Edit, Write, Grep…); WebSearch disallowed	bash only	bash, read, edit, write, glob, grep, webfetch, task; ripgrep pre-placed (rg arms)
Context management	auto-compaction: 110k window (env override) on the Qwen arms, client default elsewherec	none; per-observation head/tail trim (10k chars); hard stop at 131k	prune (40k tool-output tail) + auto-compaction at limit.input minus a 20k reserve (78k); input limit 98k, output limit 32k
Termination	--max-turns 300; soft timeout (exit 124)	step_limit=300; no soft timeout	build.steps=300; soft timeout (exit 124)
Wall clock (safety net)d	Qwen arms: 12 $\times$ the task’s agent timeout; vendor-API arms: the batch’s single soft timeout	Qwen arms: 6 $\times$ the task’s agent timeout; vendor-API arms: the same soft timeout	Qwen arms: 6 $\times$ the task’s agent timeout; vendor-API arms: the same soft timeout
Network (agent phase)	allowlist: model API host + 172.17.0.1	same	same
Runner	Harbor 0.22.0, one container per trial, verifier after the agent exits	same	same
a the versions of the arms scored in Tables 2 and 1, read off every scored trial; OpenCode 1.18.25–1.18.29 appear only in the as-shipped OpenCode arm of Table A7 and in the ablations. b a compatibility setting, not a treatment: xhigh is the Qwen chat template’s own default, and vLLM’s Anthropic-compatible /v1/messages endpoint ignores effort altogether (verified by a single-prompt probe against the same serving stack); Claude Code’s reasoning per step is within 3–6% of OpenCode’s on the same model, and OpenCode sends no effort at all, so all three harnesses see the same thinking budget on the Qwen arms. c CLAUDE_CODE_AUTO_COMPACT_WINDOW ${=}$ 110000 is set on the Qwen arms only, where it fires at a measured median of 78k tokens and so matches OpenCode by construction; on the vendor-API arms Claude Code keeps its own default window and compacts much later (on GLM-5.3-Flash at a median of 168k prompt tokens, Table A4). d on the locally served Qwen arms the limit is a multiple of the task’s agent timeout, six times for mini-SWE-agent and OpenCode and twelve for Claude Code; on the vendor-API arms (DeepSeek-V4-Flash, GLM-5.3-Flash, Opus 5) one soft timeout applies to all three harnesses, 2,760 s on DeepSeek and Opus and 5,760 s on GLM.

\times

	Claude Code	mini-SWE-agent	OpenCode
版本（按臂钉死）a	2.1.259–2.1.273 (2.1.263 on GLM and Opus)	2.4.6	1.18.30–1.18.31 (1.18.30 on GLM)
采样	未发送（无标志）	未发送	未发送
思考 / effort	GLM: effort high (CLI flag); Qwen: effort xhigh via env (vLLM compat., a no-op)b; DS: provider default	GLM: reasoning_effort = high (request body); Qwen/DS: default	GLM: reasoningEffort = high (model config); Qwen/DS: default
工具	built-in set (Bash, Read, Edit, Write, Grep…); WebSearch 禁用	仅 bash	bash, read, edit, write, glob, grep, webfetch, task; 预先放置 ripgrep（rg 臂）
上下文管理	auto-compaction: 110k window (env override) on the Qwen arms, client default elsewherec	none; per-observation head/tail trim (10k chars); hard stop at 131k	prune (40k tool-output tail) + auto-compaction at limit.input minus a 20k reserve (78k); input limit 98k, output limit 32k
终止条件	--max-turns 300; soft timeout (exit 124)	step_limit=300; no soft timeout	build.steps=300; soft timeout (exit 124)
墙钟（安全网）d	Qwen arms: 12 $\times$ the task’s agent timeout; vendor-API arms: the batch’s single soft timeout	Qwen arms: 6 $\times$ the task’s agent timeout; vendor-API arms: the same soft timeout	Qwen arms: 6 $\times$ the task’s agent timeout; vendor-API arms: the same soft timeout
网络（智能体阶段）	白名单：模型 API 主机 + 172.17.0.1	同上	同上
运行器	Harbor 0.22.0, one container per trial, verifier after the agent exits	同上	同上

\times

\times

Context window. The two Qwen models are served with a 131k-token window. OpenCode is configured with an input limit of 98k tokens and an output limit of 32k, so the two together fit the window; it compacts when the conversation exceeds the input limit minus a 20k reserve, at about 78k tokens. On the Qwen arms Claude Code’s auto-compaction window is set to 110k, which fires at a measured median of 78k tokens, matching OpenCode by construction; on the vendor-API arms Claude Code runs with its default window and compacts much later, on GLM-5.3-Flash at a median of 168k prompt tokens. mini-SWE-agent has no compaction and runs into the model’s window; that outcome is reported as its own column.

上下文窗口。 两个 Qwen 模型以 131k token 窗口服务。OpenCode 配置输入上限 98k、输出上限 32k，两者合计装进窗口；对话超过「输入上限减 20k 预留」时压缩，大约在 78k token。Qwen 臂上 Claude Code 的自动压缩窗口设为 110k，实测中位在 78k 触发，按构造与 OpenCode 对齐；厂商 API 臂上 Claude Code 用默认窗口、压缩晚得多，在 GLM-5.3-Flash 上中位约 168k prompt token。mini-SWE-agent 没有压缩，会撞上模型窗口；该结果单独成列报告。

Network. The agent phase of every reported run has no network beyond the model endpoint and the host’s package mirror. On the servers this is Harbor’s allowlist mode, which puts the trial container behind an egress sidecar. The Opus runs, which must run on a local machine with the subscription channel, use an internal Docker network and a phase-aware proxy that admits only the model API during the agent phase and any host during verification; every denial is logged, and a content-level audit found no scored trial in any offline arm that retrieved GitHub content. Two harness-specific consequences of the offline condition are handled explicitly. OpenCode downloads ripgrep at first use, so its grep and glob tools fail offline; the reported arm pre-places the binary, and the shipped arm is kept for Table A7. mini-SWE-agent calls the model without streaming, and the egress sidecar drops plain-HTTP connections that do not answer within 15 seconds, so its local-model arms go through a TLS forwarder on the host; this changes transport, not sampling.

网络。 所有报告运行的智能体阶段，除模型端点与主机软件包镜像外无外网。服务器上这是 Harbor 的白名单模式，把试验容器放在出口 sidecar 之后。必须走订阅通道、在本机跑的 Opus，使用内部 Docker 网络与分阶段代理：智能体阶段只放行模型 API，验证阶段放行任意主机；每次拒绝都记日志；内容级审计未发现任何离线臂的计分试验取回过 GitHub 内容。离线条件对外壳的两处影响单独处理。OpenCode 首次使用会下载 ripgrep，因此 grep/glob 离线会失败；报告臂预先放好二进制，出货臂保留给表 A7。mini-SWE-agent 非流式调模型，出口 sidecar 会丢掉 15 秒内无应答的明文 HTTP，故其本地模型臂经主机上的 TLS 转发；这改的是传输，不是采样。

Other fixed conditions. OpenCode selects its system prompt by model-name pattern, and every model in this study falls into its default prompt. Claude Code runs with a fresh configuration directory per trial and with web search disallowed; against non-Anthropic endpoints its Grep tool is unavailable, and those arms search through Bash. Thinking is on by default in every reported arm, including the three HY4-Preview arms, which are called through the vendor’s API. The Qwen3.6-35B-A3B thinking-off arms of Appendix J switch it off at the serving layer, so the switch is the same for all three harnesses; the HY4-Preview thinking-off arm switches it off through the API’s thinking field, which the endpoint treats as on when it is missing, and it runs in Claude Code only. For the HY4-Preview Claude Code arms the reverse proxy folds Claude Code’s mid-conversation system messages into the preceding user message and sets the thinking field explicitly, which aligns the vendor endpoint with the defaults of the Anthropic API; the content and the order of the conversation are unchanged. Claude Code sends a reasoning-effort setting of xhigh to every endpoint; vLLM’s Anthropic-compatible endpoint ignores it, and on the Qwen arms the reasoning per step under Claude Code is within a few percent of OpenCode’s, which sends none.

其他固定条件。 OpenCode 按模型名模式选择系统提示，本研究所有模型都落在它的默认提示。Claude Code 每次试验用全新配置目录，并禁用网页搜索；对接非 Anthropic 端点时其 Grep 工具不可用，那些臂经 Bash 搜索。所有报告臂默认开启思考，包括经厂商 API 调用的三个 HY4-Preview 臂。附录 J 的 Qwen3.6-35B-A3B 关思考臂在服务层关闭，因此三套外壳开关相同；HY4-Preview 关思考臂经 API 的 thinking 字段关闭（缺省时端点视为开启），且只在 Claude Code 内跑。HY4-Preview 的 Claude Code 臂上，反向代理把 Claude Code 对话中途的系统消息折进前一条用户消息，并显式设置 thinking 字段，使厂商端点与 Anthropic API 默认对齐；对话内容与顺序不变。Claude Code 向每个端点发送 reasoning-effort=xhigh；vLLM 的 Anthropic 兼容端点忽略它；Qwen 臂上 Claude Code 每步推理量与完全不发送 effort 的 OpenCode 相差几个百分点以内。

Appendix B DeepSeek’s own harness

附录 B DeepSeek 自家外壳

The home-field arm runs DeepSeek’s command-line harness, release 0.1.2-rc.1, on DeepSeek-V4-Flash over hard45, driven by Harbor like the other three and under the same soft timeout. All 45 trials were scored, none was lost to infrastructure, and 21 passed, the same count as Claude Code on the same model. Six trials ended with a non-zero exit from the harness, three of them after a passing patch. The harness takes 111.2 model calls per trial, between OpenCode’s 88.2 and mini-SWE-agent’s 141.1 on this model, and costs 1.04 CNY per trial at DeepSeek’s list price, two fen above mini-SWE-agent, the most expensive of the three general harnesses on this model. A vendor’s own harness lands inside the range of the three general ones on score and at the top of it on bill.

主场臂跑 DeepSeek 命令行外壳（发布 0.1.2-rc.1），在 DeepSeek-V4-Flash 上过 hard45，与另外三套一样由 Harbor 驱动、同一软超时。45 次试验全部计分，无基础设施丢失，通过 21 次，与同模型上的 Claude Code 相同。六次试验以外壳非零退出结束，其中三次已有通过的补丁。该外壳每试验 111.2 次模型调用，介于同模型上 OpenCode 的 88.2 与 mini-SWE-agent 的 141.1 之间；按 DeepSeek 价目表每试验 1.04 元，比三套通用外壳里最贵的 mini-SWE-agent 贵两分钱。厂商自家外壳在分数上落在三套通用外壳区间内，在账单上落在区间顶端。

Appendix C Task pools and coverage

附录 C 任务池与覆盖

Table A2 traces the denominators from the 500 tasks of SWE-bench Verified to the two task sets, and counts, for the canonical run of every cell, the trials that were lost to infrastructure and the trials that ended at a limit. Table A3 lists every cell of the grid with its runs, its usable count, and the jobs it came from. The eight dropped tasks are the psf/requests instances, whose verifier calls a public web service. One further task, django-11133, served as the smoke test for every arm and is filed under the hard45 label in the job tree while belonging to pool447; it has no scored hard45 row, so the hard45 universe is 45 tasks and disjoint from pool447. Two scikit-learn tasks time out in the verifier in most arms, which is why pool447 cells have 444 to 446 scored tasks. Eight tasks in three cells have two usable trials for the same run; the latest-finishing one is taken, and taking the earliest instead changes no count.

表 A2 从 SWE-bench Verified 的 500 道题追溯两个任务集的分母，并统计每个格子规范运行中因基础设施丢失、以及撞上限结束的试验数。表 A3 列出网格每个格子的运行次数、可用计数与来源任务。去掉的 8 道是 psf/requests 实例，其验证器会调公网服务。另有一道 django-11133 作为每臂冒烟测试，在任务树里打在 hard45 标签下，但实际属于 pool447；它没有计分的 hard45 行，因此 hard45 宇宙是 45 道、与 pool447 不相交。两道 scikit-learn 题在多数臂的验证器上超时，故 pool447 各格有 444-446 道计分题。三个格子里有八道题对同一次运行有两次可用试验；取最晚结束的那次，若改取最早，计数不变。

-

(a) Denominator ledger
SWE-bench Verified: 500 tasks
$-$ 8 psf/requests instances (verifier depends on httpbin.org, hangs offline) $\rightarrow$ pool492 $=$ 492
$\rightarrow$ hard45 $=$ 45 (whole difficulty bands 1–4 h and $>$ 4 h: 42 $+$ 3; zero sampling degrees of freedom) $+$ pool447 $=$ 447 (the complementary remainder, disjoint)
django__django-11133 is filed under the hard45 label as the universal smoke task but belongs to pool447; it has no usable hard45 row, so the scored hard45 universe is 45.
(b) What the denominator loses, canonical run-1 trials
Subset	Model	Arm	$n$	ok	void	overfl.	t/out	step	out-cut	other
hard45	Qwen3.6	CC	45	45	0	0	0	1	0	0
		mini	45	45	0	2	0	0	0	0
		OC-rg	45	45	0	0	0	0	0	0
	Qwen3.8	CC	45	43	2	1	0	0	0	0
		mini	45	45	0	11	0	0	0	0
		OC-rg	45	45	0	0	6	0	0	0
	DS Flash	CC	45	45	0	0	0	0	0	0
		mini	45	45	0	0	0	0	0	0
		OC-rg	45	45	0	0	0	0	0	1
	GLM Flash	CC	45	45	0	0	1	0	0	0
		mini	45	45	0	0	0	0	0	0
		OC-rg	45	45	0	0	0	0	0	0
	HY4	CC	45	45	0	0	0	0	0	0
		mini	45	45	0	0	0	0	0	0
		OC-rg	45	45	0	0	0	0	0	0
	Opus 5	CC	45	45	0	0	0	0	0	0
pool447	Qwen3.6	CC	447	445	2	0	0	2	0	0
		mini	447	445	2	6	0	0	0	0
		OC-rg	447	446	1	0	0	0	0	0
	Qwen3.8	CC	447	444	3	1	0	0	1	0
		mini	447	445	2	18	1	0	0	2
		OC-rg	447	445	2	0	0	0	33	0
Classes follow analysis/paper_numbers.py:outcome() on the canonical run-1 trial of every task. void = infra exception (not scored as an outcome, removed from the denominator); overflow = ContextWindowExceeded or the extractor’s context-overflow flag; timeout = SoftTimeout / AgentTimeoutError; step-cap = StepCapReached; out-cut = the run ended on the 32 k output-token ceiling (OpenCode: a terminal step_finish with reason=length, from analysis/cutoff_audit.txt; Claude Code: a fatal OutputTokenExceededError). Overflow, timeout, step-cap and out-cut trials are kept and scored as failures in every main table; column (b) says how many they are. other = NonZeroAgentExitCodeError, the remaining member of the cap set paper_numbers.CAP_EXC. The wall-clock limit is not symmetric on the locally served Qwen arms: the six OpenCode timeouts on Qwen3.8-27B $\times$ hard45 are a 5,760 s limit against Claude Code’s 36,000 s on the same model, and all six were still making tool calls, with a patch already on disk, when they were stopped (none near the 300-step cap).

-

(a) 分母台账
SWE-bench Verified: 500 题
$-$ 8 psf/requests instances (verifier depends on httpbin.org, hangs offline) $\rightarrow$ pool492 $=$ 492
$\rightarrow$ hard45 $=$ 45 (整档难度带 1–4 h and $>$ 4 h: 42 $+$ 3; 无可调采样自由度) $+$ pool447 $=$ 447 (互补剩余，不相交)
django__django-11133 is filed under the hard45 label as the universal smoke task but belongs to pool447; it has no usable hard45 row, so the scored hard45 universe is 45.
(b) 分母损失了什么（规范 run-1 试验）
子集	模型	臂	$n$	ok	void	溢出	超时	step	输出截断	其他
hard45	Qwen3.6	CC	45	45	0	0	0	1	0	0
		mini	45	45	0	2	0	0	0	0
		OC-rg	45	45	0	0	0	0	0	0
	Qwen3.8	CC	45	43	2	1	0	0	0	0
		mini	45	45	0	11	0	0	0	0
		OC-rg	45	45	0	0	6	0	0	0
	DS Flash	CC	45	45	0	0	0	0	0	0
		mini	45	45	0	0	0	0	0	0
		OC-rg	45	45	0	0	0	0	0	1
	GLM Flash	CC	45	45	0	0	1	0	0	0
		mini	45	45	0	0	0	0	0	0
		OC-rg	45	45	0	0	0	0	0	0
	HY4	CC	45	45	0	0	0	0	0	0
		mini	45	45	0	0	0	0	0	0
		OC-rg	45	45	0	0	0	0	0	0
	Opus 5	CC	45	45	0	0	0	0	0	0
pool447	Qwen3.6	CC	447	445	2	0	0	2	0	0
		mini	447	445	2	6	0	0	0	0
		OC-rg	447	446	1	0	0	0	0	0
	Qwen3.8	CC	447	444	3	1	0	0	1	0
		mini	447	445	2	18	1	0	0	2
		OC-rg	447	445	2	0	0	0	33	0
类别依据 analysis/paper_numbers.py:outcome() on the canonical run-1 trial of every task. void = 基础设施异常（不计为结果，移出分母）; overflow = ContextWindowExceeded or the extractor’s context-overflow flag; timeout = SoftTimeout / AgentTimeoutError; step-cap = StepCapReached; 输出截断 = the run ended on the 32 k output-token ceiling (OpenCode: a terminal step_finish with reason=length, from analysis/cutoff_audit.txt; Claude Code: a fatal OutputTokenExceededError). Overflow, timeout, step-cap and 输出截断 trials are kept and scored as failures in every main table; column (b) says how many they are. 其他 = NonZeroAgentExitCodeError, the remaining member of the cap set paper_numbers.CAP_EXC. 本地服务的 Qwen 臂上墙钟限制不对称： the six OpenCode timeouts on Qwen3.8-27B $\times$ hard45 是 5,760 s 上限对同模型 Claude Code 的 36,000 s；六次被停时仍在调工具、磁盘上已有补丁（都远离 300 步上限）。

n

Model	Subset	Arm	Run	$n$	ok	pass	void	Status	Jobs
Qwen3.6	hard45	CC	1	45	45	11	0	complete	cc-qwen36-hard45-nw
		CC	2	45	45	10	0	complete	cc-qwen36-hard45-nw-k2
		mini	1	45	45	15	0	complete	mini-qwen36-hard45-nw (+ -r, -r2)
		mini	2	45	45	13	0	complete	mini-qwen36-hard45-nw-k2
		OC-rg	1	45	45	11	0	complete	oc-qwen36-hard45-nw-rg (+ -r)
		OC-rg	2	45	45	11	0	complete	oc-qwen36-hard45-nw-rg-k2
	pool447	CC	1	447	445	310	2	partial (445/447)	cc-qwen36-pool447-nw (+ -r, -r2)
		mini	1	447	445	318	2	partial (445/447)	mini-qwen36-pool447-nw-c (+ mini-qwen36-pool447-nw, mini-qwen36-pool447-nw-r, mini-qwen36-pool447-nw-r2)
		mini	2	447	445	313	2	partial (445/447)	mini-qwen36-pool447-nw-k2-c2 (+ mini-qwen36-pool447-nw-k2, mini-qwen36-pool447-nw-k2-c)
		OC-rg	1	447	446	297	1	partial (446/447)	oc-qwen36-pool447-nw-rg (+ -r, -r2, -r3)
Qwen3.8	hard45	CC	1	45	43	19	2	partial (43/45)	cc-qwen-hard45-nw (+ -r)
		CC	2	45	45	20	0	complete	cc-qwen-hard45-nw (+ -r)
		mini	1	45	45	17	0	complete	mini-qwen-hard45-nw (+ -r)
		mini	2	45	45	18	0	complete	mini-qwen-hard45-nw
		OC-rg	1	45	45	17	0	complete	oc-qwen-hard45-nw-rg
		OC-rg	2	45	44	14	1	partial (44/45)	oc-qwen-hard45-nw-rg
	pool447	CC	1	447	444	347	3	partial (444/447)	cc-qwen-pool447-nw
		mini	1	447	445	353	2	partial (445/447)	mini-qwen-pool447-nw (+ -r, -r2, -r3)
		OC-rg	1	447	445	312	2	partial (445/447)	oc-qwen-pool447-nw-rg
DS Flash	hard45	CC	1	45	45	21	0	complete	cc-ds-hard45-nw (+ -r)
		CC	2	45	45	23	0	complete	cc-ds-hard45-nw-k2 (+ -r)
		mini	1	45	45	22	0	complete	mini-ds-hard45-nw (+ -r)
		OC-rg	1	45	45	19	0	complete	oc-ds-hard45-nw-rg (+ -r, -r2)
GLM Flash	hard45	CC	1	45	45	22	0	complete	cc-glm53f-hard45-nw
		mini	1	45	45	20	0	complete	mini-glm53f-hard45b-nw
		OC-rg	1	45	45	18	0	complete	oc-glm53f-hard45-nw-rg (+ oc-glm53f-r1-nw-rg)
HY4	hard45	CC	1	45	45	23	0	complete	cc-hy4-hard45-nw-r
		mini	1	45	45	26	0	complete	mini-hy4-hard45-nw
		OC-rg	1	45	45	21	0	complete	oc-hy4-hard45-nw-rg (+ -r)
Opus 5	hard45	CC	1	45	45	36	0	complete	cc-opus-hard45-nw (+ -r, -r2, -r3)
		CC	2	45	45	35	0	complete	cc-opus-hard45-nw-k2 (+ -r, -r2)
		CC	3	45	45	35	0	complete	cc-opus-hard45-nw-k3 (+ -r)
Arms: CC = Claude Code, mini = mini-SWE-agent, OC = OpenCode as shipped, OC-rg = OpenCode with ripgrep pre-placed (the canonical OpenCode arm wherever it exists). Not-excluded jobs only; smoke and crashed batches carry exclude=1 in analysis/extract_trials.py and never enter a cell. Run: the job-level repeat, except in the Qwen3.8 hard45 and Qwen3.6/3.8 $k{=}2$ jobs, where the within-job attempt index is the run index, so one job appears under both runs. void = scored trial lost to one of the 14 infra exception types (main_table.INFRA); the task is then absent from the paired analyses. Arms outside the main grid — nothink, midsys, c2-*, cc-v1.0.100, the dsh harness and every web-enabled arm — are excluded from this matrix and reported in their own appendix sections.

n

模型	子集	臂	运行	$n$	ok	pass	void	状态	任务
Qwen3.6	hard45	CC	1	45	45	11	0	complete	cc-qwen36-hard45-nw
		CC	2	45	45	10	0	complete	cc-qwen36-hard45-nw-k2
		mini	1	45	45	15	0	complete	mini-qwen36-hard45-nw (+ -r, -r2)
		mini	2	45	45	13	0	complete	mini-qwen36-hard45-nw-k2
		OC-rg	1	45	45	11	0	complete	oc-qwen36-hard45-nw-rg (+ -r)
		OC-rg	2	45	45	11	0	complete	oc-qwen36-hard45-nw-rg-k2
	pool447	CC	1	447	445	310	2	partial (445/447)	cc-qwen36-pool447-nw (+ -r, -r2)
		mini	1	447	445	318	2	partial (445/447)	mini-qwen36-pool447-nw-c (+ mini-qwen36-pool447-nw, mini-qwen36-pool447-nw-r, mini-qwen36-pool447-nw-r2)
		mini	2	447	445	313	2	partial (445/447)	mini-qwen36-pool447-nw-k2-c2 (+ mini-qwen36-pool447-nw-k2, mini-qwen36-pool447-nw-k2-c)
		OC-rg	1	447	446	297	1	partial (446/447)	oc-qwen36-pool447-nw-rg (+ -r, -r2, -r3)
Qwen3.8	hard45	CC	1	45	43	19	2	partial (43/45)	cc-qwen-hard45-nw (+ -r)
		CC	2	45	45	20	0	complete	cc-qwen-hard45-nw (+ -r)
		mini	1	45	45	17	0	complete	mini-qwen-hard45-nw (+ -r)
		mini	2	45	45	18	0	complete	mini-qwen-hard45-nw
		OC-rg	1	45	45	17	0	complete	oc-qwen-hard45-nw-rg
		OC-rg	2	45	44	14	1	partial (44/45)	oc-qwen-hard45-nw-rg
	pool447	CC	1	447	444	347	3	partial (444/447)	cc-qwen-pool447-nw
		mini	1	447	445	353	2	partial (445/447)	mini-qwen-pool447-nw (+ -r, -r2, -r3)
		OC-rg	1	447	445	312	2	partial (445/447)	oc-qwen-pool447-nw-rg
DS Flash	hard45	CC	1	45	45	21	0	complete	cc-ds-hard45-nw (+ -r)
		CC	2	45	45	23	0	complete	cc-ds-hard45-nw-k2 (+ -r)
		mini	1	45	45	22	0	complete	mini-ds-hard45-nw (+ -r)
		OC-rg	1	45	45	19	0	complete	oc-ds-hard45-nw-rg (+ -r, -r2)
GLM Flash	hard45	CC	1	45	45	22	0	complete	cc-glm53f-hard45-nw
		mini	1	45	45	20	0	complete	mini-glm53f-hard45b-nw
		OC-rg	1	45	45	18	0	complete	oc-glm53f-hard45-nw-rg (+ oc-glm53f-r1-nw-rg)
HY4	hard45	CC	1	45	45	23	0	complete	cc-hy4-hard45-nw-r
		mini	1	45	45	26	0	complete	mini-hy4-hard45-nw
		OC-rg	1	45	45	21	0	complete	oc-hy4-hard45-nw-rg (+ -r)
Opus 5	hard45	CC	1	45	45	36	0	complete	cc-opus-hard45-nw (+ -r, -r2, -r3)
		CC	2	45	45	35	0	complete	cc-opus-hard45-nw-k2 (+ -r, -r2)
		CC	3	45	45	35	0	complete	cc-opus-hard45-nw-k3 (+ -r)
Arms: CC = Claude Code, mini = mini-SWE-agent, OC = OpenCode as shipped, OC-rg = OpenCode with ripgrep pre-placed (the canonical OpenCode arm wherever it exists). Not-excluded jobs only; smoke and crashed batches carry exclude=1 in analysis/extract_trials.py and never enter a cell. Run: the job-level repeat, except in the Qwen3.8 hard45 and Qwen3.6/3.8 $k{=}2$ jobs, where the within-job attempt index is the run index, so one job appears under both runs. void = scored trial lost to one of the 14 infra exception types (main_table.INFRA); the task is then absent from the paired analyses. Arms outside the main grid — nothink, midsys, c2-*, cc-v1.0.100, the dsh harness and every web-enabled arm — are excluded from this matrix and reported in their own appendix sections.

Appendix D Leakage audit of the web-enabled runs

附录 D 开网运行的泄漏审计

Before the offline condition was enforced, the DeepSeek and Qwen arms ran with network access. An audit of those runs scanned tool-call arguments for retrieval of the upstream fix: a GitHub search hit, a commit patch, a pull-request diff, or the post-fix file from the raw content host. 312 of 439 trials had fetched the fix, 152 of them citing the task’s own issue or pull-request number, and 310 of the 312 went on to edit a file after the lookup. The Claude models did this in 4 of 445 trials. Pass rate followed: 96% of the scored DeepSeek and Qwen trials that retrieved the fix passed, against 71% of those that never touched the network. All web-enabled pass-rate results were discarded, and the condition was moved from the prompt to the network layer.

在强制离线之前，DeepSeek 与 Qwen 臂可访问网络。审计扫描工具调用参数，查找是否取回上游修复：GitHub 搜索命中、commit 补丁、PR diff，或 raw 内容主机上的修后文件。439 次试验中有 312 次取回了修复，其中 152 次引用了该题自己的 issue 或 PR 编号；312 次里有 310 次在查找后继续编辑文件。Claude 模型在 445 次试验中只有 4 次这样做。通过率随之变化：取回修复的 DeepSeek/Qwen 计分试验通过率 96%，从未触网的为 71%。所有开网通过率结果作废，条件从提示词挪到网络层强制。

Appendix E Per-task outcomes

附录 E 逐题结果

Figure A1 shows every hard45 task under every model and harness. Rows are sorted by how many arms solved the task.

图 A1 展示 hard45 每道题在每个模型与外壳下的结果。行按有多少臂解出该题排序。

Figure A1: hard45, task by task. Canonical run of every cell. Eight tasks are solved by no arm and four by every arm.

图 A1： hard45，逐题。 每个格子的规范运行。八道题无臂解出，四道题所有臂都解出。

Appendix F Resolution of a paired comparison

附录 F 成对比较的分辨率

(b+c)/n

(b+c)/n

\alpha

\alpha

Appendix G Where the per-step input goes

附录 G 每步输入花在哪里

-

-

Figure A3: Where the bill comes from. Top: tokens per trial on DeepSeek-V4-Flash and GLM-5.3-Flash, split into cache-miss input, cache-hit input, and output including reasoning. Bottom: the same tokens priced at the vendor’s list price. Annotations give the cache-hit share, steps per trial, and input tokens per step for the three harnesses.

图 A3： 账单从哪来。 上：DeepSeek-V4-Flash 与 GLM-5.3-Flash 每试验 token，拆成缓存未命中输入、缓存命中输入、以及含推理的输出。下：同一批 token 按厂商价目计价。标注给出三套外壳的缓存命中占比、每试验步数与每步输入 token。

+

(a) Tool-return size per call (chars)
Harness	calls	mean	median	p90	Largest sources
Claude Code	4,403	1,237	392	3,095	Bash 68%, Read 30%
mini-SWE-agent	3,787	1,186	461	3,121	bash 100%
OpenCode	2,633	1,103	311	3,049	read 49%, bash 39%, grep 9%

(b) Per-step input budget (tokens, hard45 means over 45 runs)
	Claude Code	mini-SWE-agent	OpenCode
Fixed cost per calla	17,134	1,382	7,579
of which the issue statement	554	554	554
of which harness preamble	16,581	829	7,025
Model text $+$ reasoning per runb	58,359	1,715	21,900
Tool returns (observations) per runb,c	31,557	44,871	16,781
Tool-call arguments per runb	11,719	14,554	5,130
chars per token (empirical)b	3.84	2.22	3.85
Steps per run	88.1	80.2	55.1
Runs with an auto-compaction (of 45)d	8	0	0
Reconstructed input per step	80,060	39,933	33,068
Measured input per stepe	75,286	45,897	35,039
GLM-5.3-Flash on hard45, 45 runs per harness (cc-glm53f-hard45-nw, mini-glm53f-hard45b-nw, oc-glm53f-hard45-nw-rg); the model, its settings and the task set are identical across the three columns, so every difference here belongs to the harness. a the first call’s prompt tokens, i.e. what every later call also pays before any transcript: the harness’s system prompt and tool schemas, its own user-message wrapper, and the issue statement. The split subtracts the issue statement at 3.7 chars/token, measured on the first user turn, which OpenCode and Claude Code receive verbatim and identically task by task (2,049 chars $=$ 554 tokens on average); mini-SWE-agent’s user turn additionally carries its instance template (a constant 2,860 chars), which is counted as harness preamble. The harnesses’ own system prompts are in neither the trajectory nor the agent log, so the preamble is a residual. b characters appended to the conversation over one run, converted at the arm’s own empirical rate (fitted on step-to-step $\Delta$ prompt_tokens vs. $\Delta$ chars): Claude Code and OpenCode carry prose at $\approx$ 3.8 chars/token, mini-SWE-agent’s raw shell output is token-dense at 2.2. Everything a step appends is re-sent with every later request; the fit only closes for Claude Code if its reasoning blocks are re-sent too, which we infer from the fit rather than observe directly. c Claude Code’s observation sizes are read from the raw stream log (agent/claude-code.txt), because the trajectory converter appends the SDK’s tool_use_result metadata to every observation and roughly doubles it; that blob was never sent to the model. mini-SWE-agent and OpenCode trajectories match their raw logs and are used as stored. d auto-compaction boundaries recorded by the Claude Code SDK; mini-SWE-agent and OpenCode write no such record and never reset in these runs, so Claude Code’s reconstruction is an upper bound and the other two are exact. On this arm Claude Code runs with its own default auto-compaction window — the 110k override of Table A1 is set on the Qwen arms only — and compacts at a median of 168k prompt tokens, where OpenCode compacts at limit.input minus its 20k reserve, i.e. at 78k, which is why the compaction counts differ. e input tokens reported by the provider, averaged over all steps of all 45 runs.

+

(a) 每次调用的工具返回大小（字符）
Harness	calls	mean	median	p90	Largest sources
Claude Code	4,403	1,237	392	3,095	Bash 68%, Read 30%
mini-SWE-agent	3,787	1,186	461	3,121	bash 100%
OpenCode	2,633	1,103	311	3,049	read 49%, bash 39%, grep 9%

(b) 每步输入预算（token，hard45 上 45 次运行均值）
	Claude Code	mini-SWE-agent	OpenCode
Fixed cost per calla	17,134	1,382	7,579
of which the issue statement	554	554	554
of which harness preamble	16,581	829	7,025
Model text $+$ reasoning per runb	58,359	1,715	21,900
Tool returns (observations) per runb,c	31,557	44,871	16,781
Tool-call arguments per runb	11,719	14,554	5,130
chars per token (empirical)b	3.84	2.22	3.85
Steps per run	88.1	80.2	55.1
Runs with an auto-compaction (of 45)d	8	0	0
Reconstructed input per step	80,060	39,933	33,068
Measured input per stepe	75,286	45,897	35,039
GLM-5.3-Flash on hard45, 45 runs per harness (cc-glm53f-hard45-nw, mini-glm53f-hard45b-nw, oc-glm53f-hard45-nw-rg); the model, its settings and the task set are identical across the three columns, so every difference here belongs to the harness. a the first call’s prompt tokens, i.e. what every later call also pays before any transcript: the harness’s system prompt and tool schemas, its own user-message wrapper, and the issue statement. The split subtracts the issue statement at 3.7 chars/token, measured on the first user turn, which OpenCode and Claude Code receive verbatim and identically task by task (2,049 chars $=$ 554 tokens on average); mini-SWE-agent’s user turn additionally carries its instance template (a constant 2,860 chars), which is counted as harness preamble. The harnesses’ own system prompts are in neither the trajectory nor the agent log, so the preamble is a residual. b characters appended to the conversation over one run, converted at the arm’s own empirical rate (fitted on step-to-step $\Delta$ prompt_tokens vs. $\Delta$ chars): Claude Code and OpenCode carry prose at $\approx$ 3.8 chars/token, mini-SWE-agent’s raw shell output is token-dense at 2.2. Everything a step appends is re-sent with every later request; the fit only closes for Claude Code if its reasoning blocks are re-sent too, which we infer from the fit rather than observe directly. c Claude Code’s observation sizes are read from the raw stream log (agent/claude-code.txt), because the trajectory converter appends the SDK’s tool_use_result metadata to every observation and roughly doubles it; that blob was never sent to the model. mini-SWE-agent and OpenCode trajectories match their raw logs and are used as stored. d auto-compaction boundaries recorded by the Claude Code SDK; mini-SWE-agent and OpenCode write no such record and never reset in these runs, so Claude Code’s reconstruction is an upper bound and the other two are exact. On this arm Claude Code runs with its own default auto-compaction window — the 110k override of Table A1 is set on the Qwen arms only — and compacts at a median of 168k prompt tokens, where OpenCode compacts at limit.input minus its 20k reserve, i.e. at 78k, which is why the compaction counts differ. e input tokens reported by the provider, averaged over all steps of all 45 runs.

Appendix H The output cutoff and a recovery transplant

附录 H 输出截断与恢复移植

OpenCode’s provider configuration caps a single response at 32k output tokens, and the trajectories show the cut landing at exactly that size in the middle of a reasoning block. On pool447 with Qwen3.8-27B, 38 of 447 OpenCode trials contain such a cut and in 34 of them it is the final step, all but one of which fail. Those trials pass at 7.9% against 75.9% for the rest, and 32 of the 34 end with the repository untouched, after a median of 11 steps. Claude Code on the same model and pool is cut in 52 of 447 trials, 66 events in all; 64 of them are followed by the harness’s own resume prompt and the session continues, and one trial ends on the cap. Its trials with a cut pass at 69% against 79% without. mini-SWE-agent never reaches the cap; its largest single step is 13k output tokens.

OpenCode 的厂商配置把单次响应封顶在 32k 输出 token；轨迹显示截断正好落在该长度、且常在推理块中途。pool447 上 Qwen3.8-27B 的 447 次 OpenCode 试验中有 38 次含此类截断，其中 34 次发生在最后一步，除 1 次外全部失败。这些试验通过率 7.9%，其余为 75.9%；34 次里有 32 次结束时仓库未被改动，中位只走了 11 步。同模型同池上 Claude Code 有 52/447 次被截断，共 66 次事件；其中 64 次后外壳会发出自己的续跑提示并继续会话，只有 1 次试验死在上限上。有截断的试验通过率 69%，无截断为 79%。mini-SWE-agent 从未触顶；单步最大输出约 13k token。

p

p

Appendix I Claude Code settings on Claude Opus 5

附录 I Claude Code 在 Claude Opus 5 上的设置

\times

\times

k

Setting	CC version	Solved / 45	flips	McNemar	output	reas.	steps	USD/trial
		$k$	95% CI (%)	/135	$p$ vs. s1	tok.	tok.		(mean)
Control, run 1	2.1.263	36	[66.2, 89.1]	—	—	15,984	4,916	34	1.83
Control, run 2	2.1.263	35	[63.7, 87.5]	—	1.00	12,777	5,927	32	1.51
Control, run 3	2.1.263	35	[63.7, 87.5]	—	1.00	14,930	5,345	39	1.60
Bash-only, run 1	2.1.263	36	[66.2, 89.1]	16/135	1.00	13,539	5,378	31	1.42
Bash-only, run 2	2.1.263	37	[68.7, 90.7]	7/135	1.00	12,250	5,853	28	1.63
No shell	2.1.263	34	[61.3, 85.8]	4/135	0.50	16,734	10,031	26	1.86
Claude Code 1.0.100	1.0.100	34	[61.3, 85.8]	12/135	0.62	3,351	0	26	0.73
Claude Opus 5 through Claude Code on hard45, 45 tasks per row, no web access; every row is one run, and all 45 tasks are usable in every row. Control is the arm of the main tables (three independent runs). Bash-only removes the file-editing tools with --disallowedTools (Edit,Write,MultiEdit,NotebookEdit,WebSearch), so the model can only edit through the shell; it was run twice. No shell removes execution instead (--disallowedTools WebSearch,Bash), leaving the editing tools. Claude Code 1.0.100 is the npm release of 2025-09-01, one year older than the 2.1.263 the other six rows run, with the job YAML otherwise identical. Solved is tasks with reward 1 out of the usable trials, with a Wilson 95% interval. flips counts tasks whose outcome differs from a control run, pooled over all three runs ( $3\times 45$ ); the within-control floor — the same count between the three control runs themselves — is 6/135 $=$ 4.4%, so no setting here separates itself from rerun noise by more than about a factor of two. McNemar is the two-sided exact binomial on the discordant pairs against control run 1. output tok. and reas. tok. are per-trial medians of the provider’s output-token and reasoning-token counts; 1.0.100 predates extended thinking being on by default, which is where its 4 $\times$ smaller output and its zero reasoning come from. USD/trial is the mean of the cost_usd Claude Code reports for itself, i.e. Anthropic list price for the tokens used; these runs were served by a Claude Code subscription and were not billed per token, so the column is a token-weighted effort index, not spend.

k

设置	CC version	解出 / 45	翻转	McNemar	输出	reas.	steps	USD/trial
		$k$	95% CI (%)	/135	$p$ vs. s1	tok.	tok.		(mean)
Control, run 1	2.1.263	36	[66.2, 89.1]	—	—	15,984	4,916	34	1.83
Control, run 2	2.1.263	35	[63.7, 87.5]	—	1.00	12,777	5,927	32	1.51
Control, run 3	2.1.263	35	[63.7, 87.5]	—	1.00	14,930	5,345	39	1.60
Bash-only, run 1	2.1.263	36	[66.2, 89.1]	16/135	1.00	13,539	5,378	31	1.42
Bash-only, run 2	2.1.263	37	[68.7, 90.7]	7/135	1.00	12,250	5,853	28	1.63
No shell	2.1.263	34	[61.3, 85.8]	4/135	0.50	16,734	10,031	26	1.86
Claude Code 1.0.100	1.0.100	34	[61.3, 85.8]	12/135	0.62	3,351	0	26	0.73
Claude Opus 5 through Claude Code on hard45, 45 tasks per row, no web access; every row is one run, and all 45 tasks are usable in every row. Control is the arm of the main tables (three independent runs). Bash-only removes the file-editing tools with --disallowedTools (Edit,Write,MultiEdit,NotebookEdit,WebSearch), so the model can only edit through the shell; it was run twice. No shell removes execution instead (--disallowedTools WebSearch,Bash), leaving the editing tools. Claude Code 1.0.100 is the npm release of 2025-09-01, one year older than the 2.1.263 the other six rows run, with the job YAML otherwise identical. Solved is tasks with reward 1 out of the usable trials, with a Wilson 95% interval. flips counts tasks whose outcome differs from a control run, pooled over all three runs ( $3\times 45$ ); the within-control floor — the same count between the three control runs themselves — is 6/135 $=$ 4.4%, so no setting here separates itself from rerun noise by more than about a factor of two. McNemar is the two-sided exact binomial on the discordant pairs against control run 1. output tok. and reas. tok. are per-trial medians of the provider’s output-token and reasoning-token counts; 1.0.100 predates extended thinking being on by default, which is where its 4 $\times$ smaller output and its zero reasoning come from. USD/trial is the mean of the cost_usd Claude Code reports for itself, i.e. Anthropic list price for the tokens used; these runs were served by a Claude Code subscription and were not billed per token, so the column is a token-weighted effort index, not spend.

Appendix J Thinking on and off

附录 J 开/关思考

-

-

\to

\to

\Delta

Harness	solved / attempts	per-task mean	$\Delta$	up /	sign	med. steps	med. output
	on	off	on $\to$ off	(pp)	down	$p$	on $\to$ off	on $\to$ off
Claude Code	21/90	22/87	0.233 $\to$ 0.256	+2.2	7 / 5	0.77	103.5 $\to$ 126	27,761 $\to$ 24,550
mini-SWE-agent	28/90	18/90	0.311 $\to$ 0.200	$-$ 11.1	2 / 10	0.04	106.5 $\to$ 98	25,198 $\to$ 20,582
OpenCode	15/88	16/86	0.148 $\to$ 0.182	+3.4	7 / 5	0.77	56 $\to$ 78.5	14,018 $\to$ 15,949.5
HY4-Preview, Claude Code only, one run per side, thinking switched at the API
Claude Code	23/45	18/45	0.511 $\to$ 0.400	$-$ 11.1	4 / 9	0.27	75 $\to$ 90	43,189 $\to$ 15,001
Qwen3.6-35B-A3B on hard45, no web access, two independent runs per side and per harness (90 attempts), thinking switched off at the server, not in the harness: a chat template that closes the think block before generation, no reasoning parser, and the token banned by a logits processor, so all three harnesses see the same model. solved counts attempts with reward 1 over the attempts that are usable (scored, no infra exception); 7 thinking-off and 2 thinking-on attempts were lost to verifier faults (verifier_incomplete, one VerifierTimeoutError) and were not rerun, which is why some denominators are below 90 and why the OpenCode pairing is over 44 tasks: django-13449 voided in both thinking-off attempts. per-task mean is the mean outcome of a task over the usable attempts of that side, and the comparison is paired over the tasks both sides scored ( $n$ $=$ 45, 45 and 44); $\Delta$ is off $-$ on in percentage points. up / down counts the tasks whose per-task mean rose / fell; the sign test is the two-sided exact binomial on those tasks, the remaining tasks being ties. This is one $k{=}2$ observation on 45 tasks. Three caveats. Both OpenCode arms here run without the pre-placed ripgrep, because the thinking-off arm had none either; they are therefore not the OC-rg arm of the main tables. The Claude Code arms carry reasoning effort xhigh, its setting throughout this paper. And mini-SWE-agent’s reasoning_chars are visible narration duplicated from the message text, not hidden thinking, so its reasoning length is not comparable with the other two and is omitted here. The second block is a different experiment: HY4-Preview through the vendor’s API under Claude Code, one run per side on the same 45 tasks, with thinking switched off by setting the API’s thinking field to disabled — the endpoint treats a missing field as on — instead of at the serving layer. With one run per side a per-task mean is 0 or 1, so up / down are the discordant tasks and the two-sided sign test on them is McNemar’s exact test. At Tencent’s list price the trial costs 2.86 CNY with thinking on and 2.14 with it off, from mean output of 43.9k and 17.5k tokens per trial.

\Delta

外壳	解出 / 尝试	每题均值	$\Delta$	up /	sign	med. steps	med. output
	on	off	on $\to$ off	(pp)	down	$p$	on $\to$ off	on $\to$ off
Claude Code	21/90	22/87	0.233 $\to$ 0.256	+2.2	7 / 5	0.77	103.5 $\to$ 126	27,761 $\to$ 24,550
mini-SWE-agent	28/90	18/90	0.311 $\to$ 0.200	$-$ 11.1	2 / 10	0.04	106.5 $\to$ 98	25,198 $\to$ 20,582
OpenCode	15/88	16/86	0.148 $\to$ 0.182	+3.4	7 / 5	0.77	56 $\to$ 78.5	14,018 $\to$ 15,949.5
HY4-Preview, Claude Code only, one run per side, thinking switched at the API
Claude Code	23/45	18/45	0.511 $\to$ 0.400	$-$ 11.1	4 / 9	0.27	75 $\to$ 90	43,189 $\to$ 15,001
Qwen3.6-35B-A3B on hard45, no web access, two independent runs per side and per harness (90 attempts), thinking switched off at the server, not in the harness: a chat template that closes the think block before generation, no reasoning parser, and the token banned by a logits processor, so all three harnesses see the same model. solved counts attempts with reward 1 over the attempts that are usable (scored, no infra exception); 7 thinking-off and 2 thinking-on attempts were lost to verifier faults (verifier_incomplete, one VerifierTimeoutError) and were not rerun, which is why some denominators are below 90 and why the OpenCode pairing is over 44 tasks: django-13449 voided in both thinking-off attempts. per-task mean is the mean outcome of a task over the usable attempts of that side, and the comparison is paired over the tasks both sides scored ( $n$ $=$ 45, 45 and 44); $\Delta$ is off $-$ on in percentage points. up / down counts the tasks whose per-task mean rose / fell; the sign test is the two-sided exact binomial on those tasks, the remaining tasks being ties. This is one $k{=}2$ observation on 45 tasks. Three caveats. Both OpenCode arms here run without the pre-placed ripgrep, because the thinking-off arm had none either; they are therefore not the OC-rg arm of the main tables. The Claude Code arms carry reasoning effort xhigh, its setting throughout this paper. And mini-SWE-agent’s reasoning_chars are visible narration duplicated from the message text, not hidden thinking, so its reasoning length is not comparable with the other two and is omitted here. The second block is a different experiment: HY4-Preview through the vendor’s API under Claude Code, one run per side on the same 45 tasks, with thinking switched off by setting the API’s thinking field to disabled — the endpoint treats a missing field as on — instead of at the serving layer. With one run per side a per-task mean is 0 or 1, so up / down are the discordant tasks and the two-sided sign test on them is McNemar’s exact test. At Tencent’s list price the trial costs 2.86 CNY with thinking on and 2.14 with it off, from mean output of 43.9k and 17.5k tokens per trial.

Appendix K Sensitivity of the paired differences

附录 K 成对差的敏感性

Table A7 recomputes every paired difference under four changes: dropping the trials that overflowed the context window, scoring per protocol (dropping tasks where either arm ended at a limit or was lost), using OpenCode as shipped instead of the arm with ripgrep in place, and substituting the second run for the first wherever one exists. No conclusion of the paper changes under any of them. On the pool, the only entries that move by more than 3 points are the two OpenCode gaps on Qwen3.8-27B under per-protocol scoring, which is the cutoff effect of Appendix H. On hard45, eleven entries move by more than 3 points, all inside that set’s resolution: the largest is the shipped OpenCode arm on Qwen3.6-35B-A3B, 11 points worse than the arm with ripgrep in place, and one sign flips, Claude Code minus mini-SWE-agent on DeepSeek-V4-Flash under the second run, by one task each way. mini-SWE-agent’s overflows on Qwen3.8-27B mostly fall on tasks the other harnesses also fail: of the 11 hard tasks it lost that way, 3 were passed by either other harness, and of the 18 lost on the pool, 6, so recovering every one of them would move it by 6.7 and 1.3 points, and on rerun 7 of the 11 overflow again.

表 A7 在四种改动下重算每个成对差：去掉上下文溢出试验；按协议计分（任一侧撞上限或丢失的题都去掉）；用出货版 OpenCode 替代预置 ripgrep 的臂；凡有第二次运行则用它替换第一次。论文结论在任一改动下都不变。大池上，唯一变动超过 3 分的是按协议计分时 Qwen3.8-27B 上两个 OpenCode 差距，正是附录 H 的截断效应。hard45 上有十一项变动超过 3 分，都落在该集分辨率内：最大的是 Qwen3.6-35B-A3B 上出货 OpenCode 臂，比预置 ripgrep 臂差 11 分；有一处符号翻转——DeepSeek-V4-Flash 上第二次运行的 Claude Code 减 mini-SWE-agent——两边各差一题。Qwen3.8-27B 上 mini-SWE-agent 的溢出，多半落在另外两套也失败的题上：hard 上因此丢掉的 11 题里，只有 3 题被另外任一外壳通过；大池丢掉的 18 题里只有 6 题；即便全部挽回，也只相当于挪 6.7 与 1.3 分；重跑时 11 次溢出有 7 次再现。

\Delta

Model	Pair	baseline $\Delta$ (pp) [95% CI] ( $n$ )	no ovfl.	per-prot.	OC ship	run 2
hard45
Qwen3.6	CC $-$ mini	-8.9 [-23.9, +6.6] (45)	-9.343	-6.844	—	-6.7
	CC $-$ OC	+0.0 [-11.8, +11.8] (45)	$=$	+0.044	+11.1	-2.2
	mini $-$ OC	+8.9 [-4.0, +21.5] (45)	+9.343	$=$	+20.0	+4.4
Qwen3.8	CC $-$ mini	+7.0 [-5.5, +19.1] (43)	+0.032	+7.0	—	+4.445
	CC $-$ OC	+7.0 [-1.8, +15.5] (43)	+7.142	+8.137	+4.7	+11.444
	mini $-$ OC	+0.0 [-11.2, +11.2] (45)	+5.934	+0.039	-2.2	+6.844
DS Flash	CC $-$ mini	-2.2 [-10.5, +6.2] (45)	$=$	$=$	—	+2.2∗
	CC $-$ OC	+4.4 [-6.7, +15.3] (45)	$=$	+4.544	+0.0	+8.9∗
	mini $-$ OC	+6.7 [-3.6, +16.6] (45)	$=$	+6.844	+2.2	—
GLM Flash	CC $-$ mini	+4.4 [-6.7, +15.3] (45)	$=$	+6.844	—	—
	CC $-$ OC	+8.9 [-2.2, +19.5] (45)	$=$	+11.444	—	—
	mini $-$ OC	+4.4 [-6.7, +15.4] (45)	$=$	$=$	—	—
HY4	CC $-$ mini	-6.7 [-18.2, +5.2] (45)	$=$	$=$	—	—
	CC $-$ OC	+4.4 [-6.7, +15.3] (45)	$=$	$=$	—	—
	mini $-$ OC	+11.1 [-0.7, +22.2] (45)	$=$	$=$	—	—
pool447
Qwen3.6	CC $-$ mini	-1.8 [-5.2, +1.6] (445)	-1.8439	-1.8443	—	-0.7∗
	CC $-$ OC	+2.9 [-0.9, +6.7] (445)	$=$	+3.2443	+5.0421	—
	mini $-$ OC	+4.7 [+1.1, +8.4] (445)	+5.0439	+4.7	+6.7421	+3.6∗
Qwen3.8	CC $-$ mini	-1.4 [-4.2, +1.5] (444)	-2.4425	-1.4440	—	—
	CC $-$ OC	+7.9 [+4.3, +11.5] (444)	+7.9443	+3.4410	+5.9	—
	mini $-$ OC	+9.2 [+5.5, +12.9] (445)	+10.5427	+4.6409	+7.0	—
Every cell is the paired difference (first harness $-$ second) over the tasks both arms scored, computed by paper_numbers.paired on the canonical trials; the baseline column is the number in Fig. 1b / Table 2 (intention-to-treat: overflow, cap and cut trials all count as failures). no ovfl.: tasks where either arm hit the context window are dropped. per-prot. (per protocol): tasks where either arm ended on a step cap, a timeout, an output cut or an infra void are dropped. $=$ marks a variant that is defined but drops no task, i.e. reproduces the baseline exactly. OC ship: the as-shipped OpenCode arm (no pre-placed ripgrep) replaces OC-rg. run 2: the second independent run replaces run 1. — means the variant does not exist for that cell (no such trial, no as-shipped arm, no second run). ∗ only one of the two arms has a second run; the other stays at run 1. A superscript is the variant’s $n$ where it differs from the baseline’s.

\Delta

模型	配对	基准 $\Delta$ (pp) [95% CI] ( $n$ )	无溢出	按协议	OC ship	run 2
hard45
Qwen3.6	CC $-$ mini	-8.9 [-23.9, +6.6] (45)	-9.343	-6.844	—	-6.7
	CC $-$ OC	+0.0 [-11.8, +11.8] (45)	$=$	+0.044	+11.1	-2.2
	mini $-$ OC	+8.9 [-4.0, +21.5] (45)	+9.343	$=$	+20.0	+4.4
Qwen3.8	CC $-$ mini	+7.0 [-5.5, +19.1] (43)	+0.032	+7.0	—	+4.445
	CC $-$ OC	+7.0 [-1.8, +15.5] (43)	+7.142	+8.137	+4.7	+11.444
	mini $-$ OC	+0.0 [-11.2, +11.2] (45)	+5.934	+0.039	-2.2	+6.844
DS Flash	CC $-$ mini	-2.2 [-10.5, +6.2] (45)	$=$	$=$	—	+2.2∗
	CC $-$ OC	+4.4 [-6.7, +15.3] (45)	$=$	+4.544	+0.0	+8.9∗
	mini $-$ OC	+6.7 [-3.6, +16.6] (45)	$=$	+6.844	+2.2	—
GLM Flash	CC $-$ mini	+4.4 [-6.7, +15.3] (45)	$=$	+6.844	—	—
	CC $-$ OC	+8.9 [-2.2, +19.5] (45)	$=$	+11.444	—	—
	mini $-$ OC	+4.4 [-6.7, +15.4] (45)	$=$	$=$	—	—
HY4	CC $-$ mini	-6.7 [-18.2, +5.2] (45)	$=$	$=$	—	—
	CC $-$ OC	+4.4 [-6.7, +15.3] (45)	$=$	$=$	—	—
	mini $-$ OC	+11.1 [-0.7, +22.2] (45)	$=$	$=$	—	—
pool447
Qwen3.6	CC $-$ mini	-1.8 [-5.2, +1.6] (445)	-1.8439	-1.8443	—	-0.7∗
	CC $-$ OC	+2.9 [-0.9, +6.7] (445)	$=$	+3.2443	+5.0421	—
	mini $-$ OC	+4.7 [+1.1, +8.4] (445)	+5.0439	+4.7	+6.7421	+3.6∗
Qwen3.8	CC $-$ mini	-1.4 [-4.2, +1.5] (444)	-2.4425	-1.4440	—	—
	CC $-$ OC	+7.9 [+4.3, +11.5] (444)	+7.9443	+3.4410	+5.9	—
	mini $-$ OC	+9.2 [+5.5, +12.9] (445)	+10.5427	+4.6409	+7.0	—
Every cell is the paired difference (first harness $-$ second) over the tasks both arms scored, computed by paper_numbers.paired on the canonical trials; the baseline column is the number in Fig. 1b / Table 2 (intention-to-treat: overflow, cap and cut trials all count as failures). no ovfl.: tasks where either arm hit the context window are dropped. per-prot. (per protocol): tasks where either arm ended on a step cap, a timeout, an output cut or an infra void are dropped. $=$ marks a variant that is defined but drops no task, i.e. reproduces the baseline exactly. OC ship: the as-shipped OpenCode arm (no pre-placed ripgrep) replaces OC-rg. run 2: the second independent run replaces run 1. — means the variant does not exist for that cell (no such trial, no as-shipped arm, no second run). ∗ only one of the two arms has a second run; the other stays at run 1. A superscript is the variant’s $n$ where it differs from the baseline’s.

Appendix L A transfer check on Terminal-Bench 2.1

附录 L Terminal-Bench 2.1 上的迁移抽查

p

p

What Does a Harness Buy? Tokens, Mostly.

外壳到底买到了什么？主要是 Token。

Abstract

摘要

1 Introduction

1 引言

2 Related Work

2 相关工作

Harness comparisons.

外壳对比。

What is inside a harness.

外壳里面有什么。

Noise and power in agent evaluation.

智能体评测中的噪声与功效。

3 Setup

3 实验设置

Harnesses.

外壳。

Models.

模型。

Tasks.

任务。

Runs and scoring.

运行与计分。

4 Results

4 结果

4.1 The harness barely moves pass rate

4.1 外壳几乎推不动通过率

4.2 The noise floor and what it can resolve

4.2 噪声地板与它能分辨什么

4.3 A harness can lose tasks

4.3 外壳可以丢掉任务

4.4 The harness sets the bill

4.4 外壳定下账单

4.5 What we looked for and did not find

4.5 我们找过、但没找到的

5 Discussion and Conclusion

5 讨论与结论

Ethics statement

伦理声明

Reproducibility statement

可复现性声明

AI use statement

AI 使用声明

References

参考文献

Appendix

附录

Appendix A What was held fixed

附录 A 哪些条件被钉死

Budget.

预算。

Context window.

上下文窗口。

Network.

网络。

Other fixed conditions.

其他固定条件。

Appendix B DeepSeek’s own harness

附录 B DeepSeek 自家外壳

Appendix C Task pools and coverage

附录 C 任务池与覆盖

Appendix D Leakage audit of the web-enabled runs

附录 D 开网运行的泄漏审计

Appendix E Per-task outcomes

附录 E 逐题结果

Appendix F Resolution of a paired comparison

附录 F 成对比较的分辨率

Appendix G Where the per-step input goes

附录 G 每步输入花在哪里

Appendix H The output cutoff and a recovery transplant

附录 H 输出截断与恢复移植

Appendix I Claude Code settings on Claude Opus 5

附录 I Claude Code 在 Claude Opus 5 上的设置

Appendix J Thinking on and off

附录 J 开/关思考

Appendix K Sensitivity of the paired differences

附录 K 成对差的敏感性

Appendix L A transfer check on Terminal-Bench 2.1

附录 L Terminal-Bench 2.1 上的迁移抽查