On the estimation and validity of AI time horizons—a statistical look at the METR plot

论 AI 时间跨度的估计与效度：从统计学角度审视 METR 图

Drew T. Nguyen, William Fithian

Drew T. Nguyen、William Fithian

Department of Statistics, UC Berkeley

加州大学伯克利分校统计系

Abstract

摘要

METR’s 50% time horizon measures the human completion time of software tasks that an AI solves with 50% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that converts human time to AI difficulty; it is nearly flat in a region from 2–30 min but close to linear elsewhere. Hence, a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of $10\times$. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons’ construct validity. We suggest that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks.

METR 的 50% 时间跨度（time horizon）指的是：AI 有 50% 的概率能完成的软件任务，人类完成它要花多长时间。这让 AI 能力可以用直观的单位来表达。我们在 228 个任务和 26 个 AI 上，用样条和项目反应理论（item-response theory, IRT）重新计算时间跨度，放宽了“任务对 AI 的难度与人类耗时的对数呈线性关系”这一假设。拟合出的样条可以理解为一个把人类耗时换算成 AI 难度的函数：它在 2–30 分钟区间几乎是平的，在其他区间接近线性。因此，时间跨度从 3 分钟跳到 30 分钟，比从 30 分钟跳到 5 小时容易得多，尽管两者同样是 $10\times$。总体而言，我们给出了一组时间跨度点估计，在交叉验证的一整套适当评分规则（proper scoring rule）下表现更好；我们还给出了用于评估时间跨度构念效度（construct validity）的诊断图。我们建议把时间跨度和诊断图放在一起解读，尤其是在有人提出新的基于时间跨度的基准、或者现有基准纳入更长任务的时候。

1 Introduction

1 引言

How quickly are AI capabilities advancing? This fundamental question underlies the large bets and policy discussions regarding AI infrastructure today. Among researchers, opinions diverge wildly; high-profile pieces such as AI 2027 [Kokotajlo et al., 2025] and AI as Normal Technology [Narayanan and Kapoor, 2025] offer starkly contrasting views. However, the work of METR, a research non-profit in Berkeley, CA, stands out as a key source of guidance. Called “the world’s informal A.I. umpire” by the New York Times [Witt, 2025], they provide compelling evidence that, in a certain sense, AI capabilities are advancing very quickly indeed.

AI 能力进步得有多快？这个基本问题，是当下围绕 AI 基础设施的巨额押注和政策讨论的底层问题。研究者之间的看法分歧极大，《AI 2027》（AI 2027）[Kokotajlo et al., 2025] 和《作为普通技术的 AI》（AI as Normal Technology）[Narayanan and Kapoor, 2025] 这类引人注目的文章给出了截然相反的判断。在这种情况下，加州伯克利的非营利研究机构 METR 的工作成了重要的参考。《纽约时报》称它为“全世界非正式的 AI 裁判”[Witt, 2025]。METR 提供了有力的证据，表明在某种意义上，AI 能力确实进步得非常快。

According to METR’s “time horizons” methodology, each of $228$ hand-designed software engineering tasks is assigned a task length, or “human time”—the length of time it takes a professional software engineer to successfully execute it. Then, crucially, each major AI release is assigned a measurement called its time horizon—the human time of tasks that it can solve autonomously with 50% probability, over repeated stochastic runs. An AI’s time horizon hence captures its capability at solving software tasks in some precisely defined sense, linked to the suite of tasks.

按照 METR 的“时间跨度”方法，$228$ 个人工设计的软件工程任务各有一个任务长度，也叫“人类耗时”（human time），即一名专业软件工程师成功完成这个任务所需的时间。关键的一步是，每个重要的 AI 版本都会得到一个称为时间跨度的测量值：在多次随机重复运行中，它有 50% 的概率能自主完成的任务所对应的人类耗时。因此，一个 AI 的时间跨度在某种精确定义的意义上刻画了它完成软件任务的能力，而这种刻画依附于这套任务。

For example, GPT-4’s estimated time horizon is 4 minutes. This means, approximately, that on a software engineering task that takes a human 4 minutes to solve (such as “Google this fact”), GPT-4 autonomously completes it with 50% probability. The longer an AI’s time horizon, the longer the tasks it can solve at this 50% rate.

例如，GPT-4 的时间跨度估计为 4 分钟。粗略地说，这意味着对于一个人类需要 4 分钟完成的软件工程任务（比如“用 Google 查一个事实”），GPT-4 有 50% 的概率能自主完成。AI 的时间跨度越长，它能以 50% 成功率完成的任务就越长。

METR’s time horizon chart, computed with the baseline shared-slope logistic method.

METR 的时间跨度图，用基线的共享斜率逻辑回归方法计算。

A conditional success plot of $\Pr(\text{success}\mid\text{human time})$ based on the baseline method, which is a more complete summary of the fit than 50% time horizons. The time horizons can be read off by locating where the dotted line intersects with any plotted point, and inspecting that point’s time label.

基于基线方法的条件成功率图，纵轴为 $\Pr(\text{success}\mid\text{human time})$，它比 50% 时间跨度更完整地概括了拟合结果。找到虚线与某个数据点的交点，查看该点的时间标签，就能读出时间跨度。

An in-sample comparison of the fitted $\Pr(\text{success}\mid\text{human time})$ for GPT-4o. Model 2 is closer.

GPT-4o 拟合的 $\Pr(\text{success}\mid\text{human time})$ 的样本内比较。模型 2 更接近。

The time-to-difficulty conversion plot, using Model 2’s conversion function $f$ and fitted standard deviation $s$. Each of the 228 points represents a task, plotted with its human time and estimated AI difficulty. The AI difficulty scale is arbitrary up to affine reparametrization, so we plot in standardized units. On these tasks log human time has a good signal to noise ratio (SNR) for AI difficulty overall, while having a notable kink from 2–30 min. The baseline assumes linearity.

耗时—难度换算图，使用模型 2 的换算函数 $f$ 和拟合的标准差 $s$。228 个点各代表一个任务，横轴为人类耗时，纵轴为估计的 AI 难度。AI 难度的尺度只确定到仿射变换，因此我们用标准化单位作图。在这些任务上，人类耗时的对数总体上对 AI 难度有良好的信噪比（SNR），但在 2–30 分钟之间有一个明显的拐折。基线假设二者呈线性关系。

Time horizons under Model 2. The scoring rule-based evaluations in Section 2.5 suggest that the estimates in the shaded region are particularly better; they are also the most different.

模型 2 下的时间跨度。第 2.5 节基于评分规则的评估表明，阴影区域内的估计改进尤其明显，它们也是差别最大的估计。

A conditional success plot based on Model 2. Beyond just estimating 50% time horizon, scoring rule-based evaluations also suggest that Model 2 predicts success better overall compared to the baseline, indicating that (f) may better summarize the data compared to (b).

基于模型 2 的条件成功率图。除了估计 50% 时间跨度，基于评分规则的评估还表明，模型 2 对成功的整体预测优于基线，说明 (f) 可能比 (b) 更好地概括了数据。

Figure 1: Summary of contributions. In this work we propose new point estimates for time horizons with an in-sample fit comparison (first column) as well as diagnostic charts for construct validity (second column). Details of how the first column was computed are described in Section 2, including a description of Model 2, and details and interpretation for the second column are in Section 3. (a) METR’s time horizon chart, computed with the baseline shared-slope logistic method. (b) A conditional success plot of $\Pr(\text{success}\mid\text{human time})$ based on the baseline method, which is a more complete summary of the fit than 50% time horizons. The time horizons can be read off by locating where the dotted line intersects with any plotted point, and inspecting that point’s time label. (c) An in-sample comparison of the fitted $\Pr(\text{success}\mid\text{human time})$ for GPT-4o. Model 2 is closer. (d) The time-to-difficulty conversion plot, using Model 2’s conversion function $f$ and fitted standard deviation $s$. Each of the 228 points represents a task, plotted with its human time and estimated AI difficulty. The AI difficulty scale is arbitrary up to affine reparametrization, so we plot in standardized units. On these tasks log human time has a good signal to noise ratio (SNR) for AI difficulty overall, while having a notable kink from 2–30 min. The baseline assumes linearity. (e) Time horizons under Model 2. The scoring rule-based evaluations in Section 2.5 suggest that the estimates in the shaded region are particularly better; they are also the most different. (f) A conditional success plot based on Model 2. Beyond just estimating 50% time horizon, scoring rule-based evaluations also suggest that Model 2 predicts success better overall compared to the baseline, indicating that (f) may better summarize the data compared to (b).

图 1：本文贡献概览。本文提出新的时间跨度点估计，并给出样本内拟合比较（第一列），以及用于检验构念效度的诊断图（第二列）。第一列的计算细节见第 2 节，其中包括对模型 2 的说明；第二列的细节和解读见第 3 节。(a) METR 的时间跨度图，用基线的共享斜率逻辑回归方法计算。(b) 基于基线方法的条件成功率图，纵轴为 $\Pr(\text{success}\mid\text{human time})$，它比 50% 时间跨度更完整地概括了拟合结果。找到虚线与某个数据点的交点，查看该点的时间标签，就能读出时间跨度。(c) GPT-4o 拟合的 $\Pr(\text{success}\mid\text{human time})$ 的样本内比较。模型 2 更接近。(d) 耗时—难度换算图，使用模型 2 的换算函数 $f$ 和拟合的标准差 $s$。228 个点各代表一个任务，横轴为人类耗时，纵轴为估计的 AI 难度。AI 难度的尺度只确定到仿射变换，因此我们用标准化单位作图。在这些任务上，人类耗时的对数总体上对 AI 难度有良好的信噪比（SNR），但在 2–30 分钟之间有一个明显的拐折。基线假设二者呈线性关系。(e) 模型 2 下的时间跨度。第 2.5 节基于评分规则的评估表明，阴影区域内的估计改进尤其明显，它们也是差别最大的估计。(f) 基于模型 2 的条件成功率图。除了估计 50% 时间跨度，基于评分规则的评估还表明，模型 2 对成功的整体预测优于基线，说明 (f) 可能比 (b) 更好地概括了数据。

Using METR’s public data, we reproduce[^1] METR’s time-horizon plot in Figure 1(a). Originally released in March 2025, it was updated point by point with each new AI release until ending with Claude Mythos in May 2026. The plot depicts an exponential increase in the time horizons, convincing many of AI’s overall exponential advance—the New York Times compared it to Moore’s law [Roose, 2026]—and, presumably, convincing many of AI’s future investment returns as well. Meanwhile, writing from the perspective of AI safety, Steinhardt [2026] relates it to the Keeling Curve, noting that, where the original Keeling Curve demonstrated rapidly rising atmospheric CO2 concentrations and oriented global strategy, the time-horizon plot could serve the same role for AI risk. Although the prospect of slowing down AI development became U.S. front page news in summer 2026, the time-horizon plot helped catalyze advocacy for a slowdown before then—including from figures such as Turing Award winner Yoshua Bengio and U.S. Senator Bernie Sanders.

我们用 METR 的公开数据，在图 1(a) 中复现[^1]了 METR 的时间跨度图。这张图最初发布于 2025 年 3 月，此后每有新 AI 发布就补上一个点，最后一个点是 2026 年 5 月的 Claude Mythos。图中的时间跨度呈指数增长，让许多人相信 AI 整体在指数式进步（《纽约时报》把它比作摩尔定律 [Roose, 2026]），想必也让许多人相信 AI 未来的投资回报。另一方面，Steinhardt [2026] 从 AI 安全的角度把它和基林曲线（Keeling Curve）相比：最初的基林曲线展示了大气 CO2 浓度的快速上升，并为全球战略定下了方向，时间跨度图也可能为 AI 风险起到同样的作用。放缓 AI 发展的话题在 2026 年夏天登上美国报纸头版，而在此之前，时间跨度图已经推动了放缓 AI 的倡议，倡议者包括图灵奖得主 Yoshua Bengio 和美国参议员 Bernie Sanders 等人。

[^1] Strictly speaking, we use the shared-slope logistic approach described by Barry [2026] in a METR research note.

[^1] 严格地说，我们使用的是 Barry [2026] 在一篇 METR 研究笔记中描述的共享斜率逻辑回归方法。

Unlike Moore’s law or the Keeling Curve, however, the quantity plotted on the y-axis is not a direct measurement of a physical quantity, but rather a statistical estimate which can be critiqued and improved. Indeed, the time-horizon methodology can be viewed as an application of item-response theory (IRT) techniques with human time as a covariate, allowing “software engineering capability” to be measured in the highly interpretable units of human time. This move—anchoring capability in an external scale—is the time-horizon plot’s key idea.

但与摩尔定律或基林曲线不同，这张图纵轴上的量不是对某个物理量的直接测量，而是一个统计估计，可以被批评，也可以被改进。实际上，时间跨度方法可以看作项目反应理论（IRT）技术的一种应用：以人类耗时为协变量，用人类耗时这一非常直观的单位来衡量“软件工程能力”。把能力锚定在一个外部尺度上，正是时间跨度图的核心想法。

In this work, we critique the time-horizon plot by reanalyzing METR’s data and fitting new models. Our methods and framing may be useful for any implementation of IRT methodology using an external anchoring scale. We answer two questions:

本文重新分析 METR 的数据并拟合新的模型，以此审视时间跨度图。我们的方法和框架可能适用于任何借助外部锚定尺度来实施 IRT 方法的场景。我们回答两个问题：

• What is the METR plot trying to estimate, and can a better estimate be devised? (Section 2)

• METR 图想要估计的是什么？能否设计出更好的估计？（第 2 节）

• How useful are time horizons for measuring AI capabilities? How can we diagnose this? (Section 3)

• 时间跨度在衡量 AI 能力上有多大用处？怎样诊断这一点？（第 3 节）

Though we are not the first to critique the METR plot, we are unaware of any critiques that fully claim a better estimator, or that discuss whether and under what conditions they are substantively useful.

我们不是第一个批评 METR 图的人，但据我们所知，此前还没有哪项批评完整地给出了更好的估计量，或者讨论过时间跨度是否、在什么条件下具有实质意义。

In Section 2, we improve the estimation of the time horizons on held-out data. Our main proposal is two statistical models, Model 1 and Model 2, that lead to new estimates; we quantify their improvement via scoring rule criteria. An in-sample diagnostic showing the improved fit of Model 2 relative to the baseline is presented in Figure 1(c); its time horizon estimates are in Figure 1(e).

在第 2 节中，我们改进了时间跨度在留出数据上的估计。我们的主要方案是两个统计模型，即模型 1 和模型 2，它们给出新的估计；我们用评分规则标准量化其改进。图 1(c) 给出了样本内诊断，显示模型 2 比基线拟合得更好；其时间跨度估计见图 1(e)。

Next, when and whether time horizons themselves are useful is a question of their construct validity, and it depends crucially on the relationship between a task’s human time and the latent difficulty of the task itself. Our Model 2, an item-response theory model, estimates this relationship.

其次，时间跨度本身在什么时候有用、是否有用，是它的构念效度问题，而这关键取决于任务的人类耗时与任务本身的潜在难度之间的关系。我们的模型 2 是一个项目反应理论模型，可以估计这一关系。

Therefore in Section 3, using Model 2, we propose two plots to help diagnose the construct validity of the time horizons: a “time-to-difficulty conversion” plot and a “conditional success trajectory” plot. Here “conditional” means conditional on human time.

因此，在第 3 节中，我们用模型 2 提出两种帮助诊断时间跨度构念效度的图：“耗时—难度换算”图和“条件成功率轨迹”图。这里的“条件”指以人类耗时为条件。

Figure 1(d) shows the time-to-difficulty conversion plot, as well as the time-to-difficulty conversion function $f$. The plot displays estimates of each of the 228 tasks’ AI difficulty, which are non-linear in log human time. In particular, we highlight a flat region between 2–30 minutes, which also can be seen in Figure 1(c); this both raises the question of just how meaningful a time horizon jump within that region is, while also suggesting that a jump cannot have the same meaning there as it does elsewhere.

图 1(d) 展示了耗时—难度换算图，以及耗时—难度换算函数 $f$。图中给出 228 个任务各自的 AI 难度估计，它们与人类耗时的对数呈非线性关系。我们特别指出 2–30 分钟之间有一段平坦区域，这在图 1(c) 中也能看到。这既引出一个问题：时间跨度在这一区域内的跃升究竟有多大意义；也说明这一区域内的跃升不可能与其他区域的跃升具有相同的含义。

Figures 1(b) and (f) show our conditional success trajectory plots, though we believe that Figure 1(f) is a better fit to the data than (b). This figure suggests that a time horizon jump from 4 min to 15 min may be quite small indeed.

图 1(b) 和 (f) 展示了我们的条件成功率轨迹图，我们认为图 1(f) 比 (b) 更贴合数据。这张图表明，时间跨度从 4 分钟跳到 15 分钟，实际进步可能相当小。

Finally, in Section 4 we discuss what this may mean for the time horizon methodology when potentially extended to longer tasks or applied to other domains. Overall, we view our work as providing a constructive critique for benchmarks like METR’s. We improve the time horizon estimates, while also providing a lens to assist in their interpretation.

最后，我们在第 4 节讨论，如果时间跨度方法扩展到更长的任务或应用到其他领域，这些发现可能意味着什么。总体而言，我们把本文看作对 METR 这类基准的建设性批评：既改进了时间跨度估计，也提供了一个辅助解读它的视角。

Note on terminology

术语说明

The word “model” is used in AI contexts to refer to a large language model, and in applied statistics contexts to refer to a statistical model or fit. Here, we will reserve “model” for the latter, and refer to large language models as “AIs”.

“模型”（model）一词在 AI 语境中指大语言模型，在应用统计语境中指统计模型或拟合。本文中“模型”只取后一种含义，大语言模型一律称为“AI”。

1.1 Prior work

1.1 已有工作

Abstractly, item-response theory [Baker, 2001] is the body of statistical techniques suited for jointly learning respondent ability and item difficulty from outcomes observed for an item-respondent pair. METR’s work as well as ours can be thought of as an application of this kind of theory.

抽象地说，项目反应理论 [Baker, 2001] 是一套统计技术，用于根据观测到的“题目—作答者”配对结果，同时学习作答者能力和题目难度。METR 的工作和我们的工作都可以看作这类理论的应用。

The original time-horizon measurement was introduced by Kwa et al. [2025], but the basic idea of pinning a unitless measure of ability, from IRT, to an externally interpretable quantity has previously been proposed; see, for example, the Lexile Framework for reading [Stenner, 2023].

时间跨度测量最早由 Kwa et al. [2025] 提出，但把 IRT 中无单位的能力度量锚定到一个外部可解释量上，这一基本想法以前就有人提出过，例如阅读领域的 Lexile 分级框架 [Stenner, 2023]。

Since 2025, there have been many critiques of Kwa et al. [2025] on Substack, LessWrong, and the EA Forum [shash42, 2025, Witkin, 2026, Gregory Lewis, 2026]. Statistical critiques and extensions of the time-horizon plot include Moss [2026] and METR’s own research notes [Kwa, 2026, Barry, 2026]. Substantive construct validity critiques of the time horizon concept in domains outside of software tasks include Kwa and Cheng [2025], Mertens et al. [2026].

自 2025 年以来，Substack、LessWrong 和 EA 论坛上出现了许多对 Kwa et al. [2025] 的批评 [shash42, 2025, Witkin, 2026, Gregory Lewis, 2026]。对时间跨度图的统计批评和扩展包括 Moss [2026] 以及 METR 自己的研究笔记 [Kwa, 2026, Barry, 2026]。针对软件任务之外领域中时间跨度概念的实质性构念效度批评，包括 Kwa and Cheng [2025] 和 Mertens et al. [2026]。

A major alternative measure of AI progress, similarly based on item-response theory, is the Epoch Capabilities Index [Ho et al., 2025]. Unlike the time-horizon approach, they do not pin an AI’s capability to an externally interpretable measure; this allows them to combine many benchmarks into one measure, at the cost of a less interpretable index.

另一个重要的 AI 进步度量是 Epoch 能力指数（Epoch Capabilities Index）[Ho et al., 2025]，它同样基于项目反应理论。与时间跨度方法不同，它不把 AI 能力锚定到外部可解释的度量上；这样做可以把许多基准合并成一个度量，代价是指数的可解释性较差。

2 The time-horizon estimation problem

2 时间跨度的估计问题

2.1 The statistical framework

2.1 统计框架

What problem is the time-horizon plot solving? Within the framework of a statistical model, it depicts estimates of a well-specified quantity, as we shall see in Definition 2.1.

时间跨度图要解决的是什么问题？在统计模型的框架内，它描绘的是一个定义明确的量的估计，见定义 2.1。

Suppose we are able to draw tasks $(\texttt{task}_{i})_{i=1}^{I}$ from some task distribution $\mathcal{D}$, and fix AIs $(\texttt{AI}_{j})_{j=1}^{J}$. We think of the tasks as units existing in some space tasks, and AIs in AIs.

假设我们能从某个任务分布 $\mathcal{D}$ 中抽取任务 $(\texttt{task}_{i})_{i=1}^{I}$，并固定一组 AI $(\texttt{AI}_{j})_{j=1}^{J}$。我们把任务看作空间 tasks 中的单元，把 AI 看作空间 AIs 中的单元。

Each task is associated with certain covariates, in particular associated with a human time $T_{1},\dots,T_{I}>0$ and a task family ID. On the other hand, each AI is associated with a release date $D_{1}<\dots<D_{J}$. Let $\mathcal{P}:\textbf{tasks}\times\textbf{AIs}\to[0,1]$ be the function that associates each $(\texttt{task},\texttt{AI})$ pair with the probability that AI solves task. Denote $p_{ij}:=\mathcal{P}(\texttt{task}_{i},\texttt{AI}_{j})$.

每个任务带有一些协变量，尤其是人类耗时 $T_{1},\dots,T_{I}>0$ 和任务族 ID。另一方面，每个 AI 带有一个发布日期 $D_{1}<\dots<D_{J}$。设 $\mathcal{P}:\textbf{tasks}\times\textbf{AIs}\to[0,1]$ 是把每个 $(\texttt{task},\texttt{AI})$ 配对映射到该 AI 完成该任务的概率的函数，记 $p_{ij}:=\mathcal{P}(\texttt{task}_{i},\texttt{AI}_{j})$。

Given $(\texttt{task}_{i})_{i=1}^{I}$ and $(\texttt{AI}_{j})_{j=1}^{J}$, assume we have access to a mechanism to draw iid samples $Y_{ij}=(Y_{ij1},\dots,Y_{ijn_{ij}})$, such that $\mathbb{E}Y_{ij1}=p_{ij}$, which models multiple trials of re-running the AI on the same prompt. Here $Y_{ij}$ represents the $n_{ij}$ runs where $\texttt{AI}_{j}$ attempts $\texttt{task}_{i}$, and $Y_{ijr}\in\{0,1\}$ is the success or failure of the $r$th run.

给定 $(\texttt{task}_{i})_{i=1}^{I}$ 和 $(\texttt{AI}_{j})_{j=1}^{J}$，假设我们有一种机制可以抽取独立同分布样本 $Y_{ij}=(Y_{ij1},\dots,Y_{ijn_{ij}})$，满足 $\mathbb{E}Y_{ij1}=p_{ij}$，用来刻画在同一提示上多次重跑 AI 的试验。这里 $Y_{ij}$ 表示 $\texttt{AI}_{j}$ 尝试 $\texttt{task}_{i}$ 的 $n_{ij}$ 次运行，$Y_{ijr}\in\{0,1\}$ 是第 $r$ 次运行的成功或失败。

(Despite this iid assumption, approximation errors in the functional form of any practical model specification will induce correlation among trials in the runset $Y_{ij}$ even for the same AI-task pair when conditioning on latent features. We return to this point when introducing our Model 2.)

（尽管有这一独立同分布假设，任何实际模型设定在函数形式上的近似误差，都会使运行集 $Y_{ij}$ 中的各次试验在以潜在特征为条件时仍然相关，即使是同一个 AI—任务配对也不例外。介绍模型 2 时我们会回到这一点。）

Define the success curve $p_{j}(t)=\mathbb{E}_{\texttt{task}_{i}\sim\mathcal{D}}[Y_{ij1}\mid T_{i}=t]$. We make the structural assumption that, for our chosen AIs and task distribution $\mathcal{D}$, this function is strictly monotone decreasing in $t$, so that its inverse is always defined.

定义成功曲线 $p_{j}(t)=\mathbb{E}_{\texttt{task}_{i}\sim\mathcal{D}}[Y_{ij1}\mid T_{i}=t]$。我们作一个结构性假设：对选定的 AI 和任务分布 $\mathcal{D}$，这个函数关于 $t$ 严格单调递减，因此其反函数总是有定义的。

Now, we can define the oracle quantity we would like to estimate, from the data $(Y_{ijr})$ and $(T_{i})$.

现在可以定义我们想从数据 $(Y_{ijr})$ 和 $(T_{i})$ 中估计的理想量。

Definition 2.1.

定义 2.1.

For $q\in(0,1)$, the $q$-time horizon of $\texttt{AI}_{j}$ is

对 $q\in(0,1)$，$\texttt{AI}_{j}$ 的 $q$-时间跨度为

$$\mathbf{th}_{q}(\texttt{AI}_{j}):=p_{j}^{-1}(q).$$

This is the human time of tasks from $\mathcal{D}$ that $\texttt{AI}_{j}$ can solve with probability $q$. To learn this quantity for each AI it suffices, therefore, to learn the regression function $p_{j}(t)$, for each $\texttt{AI}_{j}$. The time-horizon plots of Figures 1(a) and (e) are then a scatter of points $(D_{1},\widehat{\mathbf{th}}_{q}(\texttt{AI}_{1})),\dots,(D_{J},\widehat{\mathbf{th}}_{q}(\texttt{AI}_{J}))$.

这是 $\texttt{AI}_{j}$ 能以概率 $q$ 完成的、来自 $\mathcal{D}$ 的任务的人类耗时。因此，要得到每个 AI 的这个量，只需对每个 $\texttt{AI}_{j}$ 学习回归函数 $p_{j}(t)$。图 1(a) 和 (e) 的时间跨度图就是点 $(D_{1},\widehat{\mathbf{th}}_{q}(\texttt{AI}_{1})),\dots,(D_{J},\widehat{\mathbf{th}}_{q}(\texttt{AI}_{J}))$ 的散点图。

Importantly for the interpretation of the METR plot, the release dates $D_{1},\dots,D_{J}$ are used nowhere in the model; the observed exponential growth is entirely empirical and we do not challenge it here.

对解读 METR 图来说很重要的一点是：模型中完全没有用到发布日期 $D_{1},\dots,D_{J}$；观察到的指数增长完全是经验结果，本文不对它提出质疑。

2.2 Implementing the framework

2.2 框架的实施

In order to implement this framework in the real world, we require the following ingredients: tasks $(\texttt{task}_{i})_{i=1}^{I}$, thought to be sampled from some relevant task distribution $\mathcal{D}$; a setup for humans and AIs to attempt the tasks, and hence measure each task’s human time, $(T_{i})_{i=1}^{I}$, as well as the task-AI success indicators $Y_{ij}$ for each task-AI pair $(i,j)$; a technique for estimating the marginal success probability $p_{j}(t)$ for the $j$th AI, with confidence intervals; and finally, a method for evaluating whether our estimate had good performance.

要在现实中实施这一框架，需要以下几样东西：任务 $(\texttt{task}_{i})_{i=1}^{I}$，可以认为它们是从某个相关任务分布 $\mathcal{D}$ 中抽样得到的；一套让人类和 AI 尝试这些任务的设置，从而测出每个任务的人类耗时 $(T_{i})_{i=1}^{I}$，以及每个任务—AI 配对 $(i,j)$ 的成功指示 $Y_{ij}$；一种估计第 $j$ 个 AI 边际成功概率 $p_{j}(t)$ 并给出置信区间的技术；最后，一种评估估计表现好坏的方法。

In this section, we summarize strategies for each of these implementation components. In particular, we summarize METR’s approach for the first two components; see Kwa et al. [2025] for a fuller description. For the last two components (estimation and evaluation), we summarize general principles here and defer fuller exposition to Sections 2.3 and 2.4.

本节概述实施上述各部分的策略。前两部分我们概述 METR 的做法，更完整的描述见 Kwa et al. [2025]。后两部分（估计与评估）我们在这里概述一般原则，详细说明放在第 2.3 和 2.4 节。

Designing the tasks.

设计任务。

METR designed 228 tasks in their Time Horizon 1.1 framework, which were broadly software engineering relevant tasks such as “implement this function” or “train this classifier” written as a set of instructions, together with an automated function that scores the result. Each task was a member of one of three “task suites” with a unifying theme; each task was further classified into a task_family, of which there were 79.

METR 在其 Time Horizon 1.1 框架中设计了 228 个任务，大体上都是与软件工程相关的任务，例如“实现这个函数”或“训练这个分类器”，以一组指令的形式写成，并配有一个自动给结果打分的函数。每个任务属于三个有统一主题的“任务套件”之一；每个任务还被进一步归入一个任务族（task_family），共有 79 个任务族。

Attempting the tasks.

尝试任务。

For each task, a shared setup was used for both humans and AIs to attempt the task. This was concretely three things per task: a set of task instructions, a field to input the task solution, and a scorer function to judge success or failure.

每个任务都有一套人类和 AI 共用的尝试设置，具体包括三样东西：一组任务指令、一个填写任务答案的输入框，以及一个判断成功或失败的评分函数。

On the human side, multiple software engineers were recruited to attempt each task while being timed (typically ${\sim}4$ attempts per task); on attempts where the scorer judged a success, their time measurements were combined using the geometric mean. This value is called the human time of the task (METR’s “task length”), which we denote $T_{i}$ for $\texttt{task}_{i}$. However, these measurements were unobtainable or deemed invalid (e.g. too few successes) for 29% of the tasks, and in those cases, an expert’s value is used instead to set as $T_{i}$.

在人类一侧，每个任务招募多名软件工程师计时完成（通常每个任务约 ${\sim}4$ 次尝试）；对评分函数判为成功的尝试，用几何平均合并其用时。这个值称为任务的人类耗时（METR 所说的“任务长度”），对 $\texttt{task}_{i}$ 记为 $T_{i}$。不过，有 29% 的任务无法得到这样的测量值，或测量值被判为无效（例如成功次数太少），这些情况下改用专家给出的估值作为 $T_{i}$。

On the AI side, the $j$th AI was provided the instructions for $\texttt{task}_{i}$ as a prompt and run autonomously $n_{ij}$ times, and the scorer’s output recorded as $Y_{ij}=(Y_{ij1},\dots,Y_{ijn_{ij}})$ for each run. Note that we ignore the time AIs take to complete tasks.

在 AI 一侧，把 $\texttt{task}_{i}$ 的指令作为提示交给第 $j$ 个 AI，让它自主运行 $n_{ij}$ 次，并把每次运行的评分输出记为 $Y_{ij}=(Y_{ij1},\dots,Y_{ijn_{ij}})$。注意，我们不考虑 AI 完成任务所用的时间。

Fitting models and estimating time horizons.

拟合模型并估计时间跨度。

Here we summarize general principles for fitting models; concrete approaches are detailed in Section 2.3. The framework of Section 2.1 allows us to precisely define the statistical problem at hand: we want good estimates for $\mathbf{th}_{q}$, especially $\mathbf{th}_{0.5}(\texttt{AI}_{j})$ and $\mathbf{th}_{0.8}(\texttt{AI}_{j})$, for each $\texttt{AI}_{j}$. METR’s original approach, as well as ours, will be to first obtain estimates for $p_{j}(t)$ and then invert the function according to Definition 2.1. These per-AI success curves $p_{j}(t)$, also called “characteristic curves” in item-response theory, can be estimated in any way in principle, as long as they are ultimately evaluated with held-out data.

这里概述拟合模型的一般原则，具体做法详见第 2.3 节。第 2.1 节的框架让我们可以精确定义眼前的统计问题：我们希望对每个 $\texttt{AI}_{j}$ 得到 $\mathbf{th}_{q}$ 的良好估计，尤其是 $\mathbf{th}_{0.5}(\texttt{AI}_{j})$ 和 $\mathbf{th}_{0.8}(\texttt{AI}_{j})$。METR 最初的做法和我们的做法一样，都是先估计 $p_{j}(t)$，再按定义 2.1 求反函数。这些逐 AI 的成功曲线 $p_{j}(t)$ 在项目反应理论中也称为“特征曲线”，原则上可以用任何方式估计，只要最终用留出数据来评估。

These approaches are all based on the same tabular dataset, where a row consists of the AI’s ID $j$, the task’s index $i$, the run index $r$, the success indicator $Y_{ijr}$ for the AI’s attempt on that task, and finally the task’s human time $T_{i}$—we do not use the AI’s release date $D_{j}$ for fitting, but only for plotting.

这些做法都基于同一个表格数据集，每一行包括 AI 的 ID $j$、任务索引 $i$、运行索引 $r$、该 AI 在该任务上这次尝试的成功指示 $Y_{ijr}$，以及任务的人类耗时 $T_{i}$。我们不用 AI 的发布日期 $D_{j}$ 做拟合，只用它作图。

Evaluating models.

评估模型。

Finally, we summarize general principles for evaluating models; our evaluation metrics are defined in Section 2.4, with results in Section 2.5. Essentially, given estimates for the success curves $p_{1}(\cdot),\dots,p_{J}(\cdot)$, we compute these metrics to adjudicate between the different estimates and select the best. Our metrics consist of proper scoring rules and Murphy diagrams [Dimitriadis et al., 2023], which we computed with 5-fold cross validation; this can be thought of as an extension of the evaluations in Barry [2026]. We also plot in-sample visual diagnostics for our $p_{j}(t)$ estimates against empirical frequencies to assess the functional form, as done by Kwa et al. [2026] in their Figure 4.

最后，我们概述评估模型的一般原则；评估指标在第 2.4 节定义，结果见第 2.5 节。简单地说，给定成功曲线的估计 $p_{1}(\cdot),\dots,p_{J}(\cdot)$，我们计算这些指标，在不同估计之间做出裁决并选出最好的。我们的指标由适当评分规则和墨菲图（Murphy diagram）[Dimitriadis et al., 2023] 组成，用 5 折交叉验证计算；这可以看作对 Barry [2026] 中评估的扩展。我们还像 Kwa et al. [2026] 的图 4 那样，画出 $p_{j}(t)$ 估计与经验频率对比的样本内可视化诊断，以评估函数形式。

2.3 Model specifications

2.3 模型设定

In this section, we list the four main model specifications we ran. By model specification, we mean a model family and fitting procedure for the success curves $p_{j}(t)$. The first two were proposed by METR, and the last two are our contribution.

本节列出我们运行的四种主要模型设定。所谓模型设定，是指成功曲线 $p_{j}(t)$ 的模型族加上拟合过程。前两种由 METR 提出，后两种是我们的贡献。

Logistic regression.

逻辑回归。

This is the original headline model specification used by METR. For each $\texttt{AI}_{j}$, we assume for $p_{j}(t)$ the log-linear model

这是 METR 最初用于主要结果的模型设定。对每个 $\texttt{AI}_{j}$，我们假设 $p_{j}(t)$ 服从对数线性模型

$$\log\left(\frac{p_{j}(t)}{1-p_{j}(t)}\right)=\alpha_{j}-\beta_{j}\log_{2}(t)$$

with $\beta_{j}>0$. We estimate the parameters $\alpha_{j}$, $\beta_{j}$ for each $j$ with logistic regression, i.e. minimizing binary cross entropy, using $\log_{2}(T_{1}),\dots,\log_{2}(T_{I})$ as covariates. When the cross-entropy is unweighted, this is formally equivalent to maximum likelihood with the generative model $Y_{ijr}\mid T_{i}=t\sim\operatorname{Bern}(p_{j}(t))$ independently for all $i,j,r$.

其中 $\beta_{j}>0$。我们以 $\log_{2}(T_{1}),\dots,\log_{2}(T_{I})$ 为协变量，用逻辑回归（即最小化二元交叉熵）估计每个 $j$ 的参数 $\alpha_{j}$、$\beta_{j}$。交叉熵不加权时，这在形式上等价于在生成模型“对所有 $i,j,r$ 独立地有 $Y_{ijr}\mid T_{i}=t\sim\operatorname{Bern}(p_{j}(t))$”下做最大似然估计。

Such an independence assumption is certainly wrong, but this does not necessarily mean that the success curves are wrong. Also, tasks in a shared task family have correlated behavior, so following Kwa et al. [2025], the effect of large families was downweighted in training by using the sqrt-family weighting described in Section 2.4.1 on the cross-entropy loss.

这样的独立性假设肯定是错的，但这并不一定意味着成功曲线是错的。另外，同一任务族内的任务表现相关，因此我们沿用 Kwa et al. [2025] 的做法，在交叉熵损失上使用第 2.4.1 节描述的平方根任务族加权（sqrt-family weighting），降低大任务族在训练中的影响。

Logistic regression with shared slope (our baseline).

共享斜率的逻辑回归（我们的基线）。

This specification was tried by Barry [2026] in work for METR. It is the same as the previous, except we assume for $p_{j}(t)$ that

Barry [2026] 在为 METR 做的工作中尝试过这一设定。它与前一个设定相同，只是假设 $p_{j}(t)$ 满足

$$\log\left(\frac{p_{j}(t)}{1-p_{j}(t)}\right)=\alpha_{j}-\beta\log_{2}(t)$$

for a fixed $\beta>0$ common to each AI. Again, an unweighted fit is equivalent to maximum likelihood on a model where all the $Y_{ijr}$ are independent conditionally on the human times $T_{1},\dots,T_{I}$.

其中 $\beta>0$ 是所有 AI 共用的固定值。同样，不加权的拟合等价于在“以人类耗时 $T_{1},\dots,T_{I}$ 为条件、所有 $Y_{ijr}$ 相互独立”的模型上做最大似然估计。

Barry [2026] observes it has better performance in cross-validation on both Brier score and marginal log score, with sqrt-family weighting; we replicate this finding, and in fact it beats METR’s model spec on all but one of the 12 metrics in our proper scoring suite (Section 2.5). We hence use this spec as our performance baseline, rather than the logistic regression spec. The time horizons from this fit were displayed in Figure 1(a).

Barry [2026] 观察到，在平方根任务族加权下，它在交叉验证的 Brier 分数和边际对数分数上都表现更好；我们复现了这一结果，而且在我们的适当评分规则套件的 12 项指标中，它有 11 项胜过 METR 的模型设定（第 2.5 节）。因此我们用这一设定而不是逻辑回归设定作为性能基线。图 1(a) 展示的就是这一拟合得到的时间跨度。

Model 1: Shared monotone spline, with shared slope.

模型 1：共享单调样条，共享斜率。

This specification just replaces the linear dependence on $\log_{2}(t)$ with a monotone function. We assume for $p_{j}(t)$ that

这一设定只是把对 $\log_{2}(t)$ 的线性依赖换成一个单调函数。我们假设 $p_{j}(t)$ 满足

$$\log\left(\frac{p_{j}(t)}{1-p_{j}(t)}\right)=\alpha_{j}-\beta\,f(\log_{2}t),\qquad\beta>0.$$

We take $f$ to be a monotone natural cubic spline; specifically, a monotone natural cubic I-spline with nonnegative coefficients with 4 interior knots. Additionally, because of non-uniqueness with respect to affine transformations of $f(\cdot)\mapsto af(\cdot)+b$ where $a>0$, we make the identifiability assumption $f(\min\log_{2}T_{i})=\min\log_{2}T_{i}$ and $f(\max\log_{2}T_{i})=\max\log_{2}T_{i}$. Then we fit with binary cross entropy, with sqrt-family weighting; again, an unweighted fit is equivalent to maximum likelihood on a model where all the $Y_{ijr}$ are independent conditionally on the human times $T_{1},\dots,T_{I}$.

我们取 $f$ 为单调自然三次样条，具体来说是系数非负、有 4 个内部节点的单调自然三次 I 样条。此外，由于 $f$ 在仿射变换 $f(\cdot)\mapsto af(\cdot)+b$（$a>0$）下不唯一，我们作可识别性假设 $f(\min\log_{2}T_{i})=\min\log_{2}T_{i}$ 和 $f(\max\log_{2}T_{i})=\max\log_{2}T_{i}$。然后用平方根任务族加权的二元交叉熵拟合；同样，不加权的拟合等价于在“以人类耗时 $T_{1},\dots,T_{I}$ 为条件、所有 $Y_{ijr}$ 相互独立”的模型上做最大似然估计。

A monotone spline-based approach was previously considered by METR in Kwa [2026], and attempted in Barry [2026], but did not beat the baseline; this was likely because they were separate splines, rather than shared.

METR 此前在 Kwa [2026] 中考虑过基于单调样条的方法，Barry [2026] 也尝试过，但都没有胜过基线；这很可能是因为他们用的是各自独立的样条，而不是共享样条。

The single shared shape $f$ gives this specification its interpretation. It is an estimate of how AI difficulty varies with human time, common to all the AIs. Using item-response theory in Model 2, we can extend this interpretation further.

单一的共享形状 $f$ 赋予了这个设定可解释性：它估计的是 AI 难度如何随人类耗时变化，并且对所有 AI 通用。借助模型 2 中的项目反应理论，我们可以把这种解释进一步扩展。

Model 2: Explanatory IRT with family effects and overdispersion.

模型 2：带任务族效应和过度离散的解释性 IRT。

This specification is a fully generative model $\mathcal{M}$ for all the $Y_{ijr}$’s. We fit it with maximum likelihood, and subsequently derive the success curves. This model involves task families; let $F$ index the families, and let $F(i)$ be the task family of $\texttt{task}_{i}$. Let $\check{T}_{{F(i)}}$ be the geometric mean of human times in $F(i)$.

这一设定是所有 $Y_{ijr}$ 的完整生成模型 $\mathcal{M}$。我们用最大似然拟合它，再从中推导成功曲线。这个模型涉及任务族：令 $F$ 索引任务族，$F(i)$ 为 $\texttt{task}_{i}$ 所属的任务族，$\check{T}_{{F(i)}}$ 为 $F(i)$ 中人类耗时的几何平均。

In this model, each $\texttt{task}_{i}$ carries a latent difficulty $\theta_{i}\in\mathbb{R}$. Define

在这个模型中，每个 $\texttt{task}_{i}$ 带有一个潜在难度 $\theta_{i}\in\mathbb{R}$。定义

$$\tilde{p}_{ij}(\theta_{i}):=\Pr(Y_{ijr}=1\mid\theta_{i})$$

for the success probability of a single run conditional on the difficulties. We model this as

为以难度为条件时单次运行的成功概率。我们把它建模为

$$\log\left(\frac{\tilde{p}_{ij}(\theta_{i})}{1-\tilde{p}_{ij}(\theta_{i})}\right)=\alpha_{j}-\beta\theta_{i},$$

where the $\beta>0$ is shared among all AIs $j$. The latent difficulty is modeled as

其中 $\beta>0$ 由所有 AI $j$ 共享。潜在难度建模为

$$\theta_{i}=f(\log_{2}T_{i})+\xi_{F(i)}+\varepsilon_{i},\qquad\xi_{F}\sim\mathcal{N}\big(0,\ \tau(\log_{2}\check{T}_{F})^{2}\big),\qquad\varepsilon_{i}\sim\mathcal{N}\big(0,\ \sigma(\log_{2}T_{i})^{2}\big),$$

for some mean function $f$, family variance $\tau(\cdot)^{2}$, and idiosyncratic variance $\sigma(\cdot)^{2}$.

其中 $f$ 是均值函数，$\tau(\cdot)^{2}$ 是任务族方差，$\sigma(\cdot)^{2}$ 是任务自身的方差。

Even though we assume $Y_{ij1},\dots,Y_{ijn_{ij}}$ are iid, we do not model them as independent conditional on $\theta$, since the pair $(\theta_{i},\alpha_{j})$ is not a complete summary of the AI-task interaction, and conditioning on these induces correlation between trials for each AI-task pair. Hence we introduce a correlation parameter $\rho\in(0,1)$ shared by all AIs, put $\varphi=(1-\rho)/\rho$, and model the final probability $p_{ij}$ as drawn from

尽管我们假设 $Y_{ij1},\dots,Y_{ijn_{ij}}$ 独立同分布，但我们并不把它们建模为以 $\theta$ 为条件相互独立，因为 $(\theta_{i},\alpha_{j})$ 这一对数并不能完整概括 AI 与任务之间的交互，以它们为条件会在每个 AI—任务配对的各次试验之间引入相关性。因此，我们引入一个所有 AI 共享的相关参数 $\rho\in(0,1)$，令 $\varphi=(1-\rho)/\rho$，并把最终概率 $p_{ij}$ 建模为从下式中抽取：

$$p_{ij}\mid\theta_{i}\sim\operatorname{Beta}\big(\tilde{p}_{ij}(\theta_{i})\,\varphi,\ (1-\tilde{p}_{ij}(\theta_{i}))\,\varphi\big),\qquad Y_{ij1},\dots,Y_{ijn_{ij}}\mid p_{ij}\overset{\text{iid}}{\sim}\operatorname{Bern}(p_{ij}),$$

so that $\sum_{r=1}^{n_{ij}}Y_{ijr}$ is beta-binomial, allowing us to model it with higher variance than if it were only binomial (hence the name “overdispersion”). The source of this higher variance is the correlation $\operatorname{Corr}(Y_{ijr},Y_{ijr^{\prime}}\mid\theta_{i})=\rho$ for $r\neq r^{\prime}$.

这样 $\sum_{r=1}^{n_{ij}}Y_{ijr}$ 服从贝塔二项分布，使我们能用比单纯二项分布更大的方差来建模（“过度离散”一名即由此而来）。方差增大的来源是相关性 $\operatorname{Corr}(Y_{ijr},Y_{ijr^{\prime}}\mid\theta_{i})=\rho$（$r\neq r^{\prime}$）。

Fitting maximum likelihood under this model is not trivial. We implemented an EM algorithm and approximated the integrals with Gauss-Hermite quadrature. Note that, similar to Model 1, the likelihood is invariant to affine transformations of $\theta$, so we fix the scale with identifiability assumptions on $f$.

在这个模型下做最大似然拟合并不简单。我们实现了 EM 算法，并用高斯—埃尔米特求积近似积分。注意，与模型 1 类似，似然对 $\theta$ 的仿射变换不变，因此我们通过对 $f$ 的可识别性假设来固定尺度。

To compute $p_{j}(t)$ under this model, we must marginalize over the random effects:

要在这个模型下计算 $p_{j}(t)$，必须对随机效应求边际：

$$p_{j}(t)=\mathbb{E}\left[\left(1+\exp(-(\alpha_{j}-\beta\theta))\right)^{-1}\right],\qquad\theta\sim\mathcal{N}\!\left(f(\log_{2}t),\ \sigma(\log_{2}t)^{2}+\tau(\log_{2}t)^{2}\right), \tag{1}$$

where evaluating $\tau$ at $\log_{2}t$ is the convention for a fresh task from a fresh family whose mean human time is $t$. We evaluate this by quadrature and invert numerically for $\widehat{\mathbf{th}}_{q}$.

其中在 $\log_{2}t$ 处取 $\tau$ 的值，对应的约定是：一个来自新任务族的新任务，该任务族的平均人类耗时为 $t$。我们用数值求积计算它，再数值求逆得到 $\widehat{\mathbf{th}}_{q}$。

Remarks on Model 2.

关于模型 2 的说明。

This is an item-response theory model. Specifically, it is a kind of 1PL Rasch model due to the shared $\beta$. In general, item-response theory uses latent variables to model the success probability of a respondent on a test item. It captures correlations between responses that share a respondent or an item by modeling a shared latent trait that connects them—a student’s ability in a standardized testing setting, or, in our case, the task difficulty $\theta_{i}$, which connects $Y_{ij}$ and $Y_{ij^{\prime}}$ for $j\neq j^{\prime}$.

这是一个项目反应理论模型。具体来说，由于 $\beta$ 共享，它属于单参数（1PL）Rasch 模型。一般而言，项目反应理论用潜变量来建模作答者在某道测试题上的成功概率。对于共享同一作答者或同一题目的作答结果，它通过建模一个把它们联系起来的共享潜在特质来刻画其相关性：在标准化考试中，这个特质是学生的能力；在我们这里，则是任务难度 $\theta_{i}$，它把 $j\neq j^{\prime}$ 时的 $Y_{ij}$ 和 $Y_{ij^{\prime}}$ 联系起来。

It is an explanatory IRT here because $\theta_{i}$ is regressed on the task’s observed covariates: human time and task family, which “explain” the values of $\theta$. We use the same family of monotone spline $f$ as in Model 1, with the same identifiability assumption, and take $\sigma(\cdot)$ and $\tau(\cdot)$ to each be the exponential of a natural cubic spline with 2 interior knots.

这里之所以是解释性 IRT，是因为 $\theta_{i}$ 对任务的观测协变量（人类耗时和任务族）做了回归，由这些协变量“解释” $\theta$ 的取值。我们使用与模型 1 相同的单调样条族 $f$ 和相同的可识别性假设，并把 $\sigma(\cdot)$ 和 $\tau(\cdot)$ 各取为一个有 2 个内部节点的自然三次样条的指数。

Like in Model 1, the mean function $f$ accounts for non-linearities between difficulty and log human time, while the individual and family effects account for possible spread in task difficulties from their mean. In particular, the family effect $\xi_{F}$ accounts for correlations among tasks within a family, without the heuristic sqrt-family reweighting.

与模型 1 一样，均值函数 $f$ 刻画难度与人类耗时对数之间的非线性，而任务个体效应和任务族效应刻画任务难度围绕均值可能的分散。尤其是，任务族效应 $\xi_{F}$ 刻画了同一任务族内任务之间的相关性，不再需要启发式的平方根任务族重加权。

We account for dependence within the runset $Y_{ij1},\dots,Y_{ijn_{ij}}$ via the correlation parameter $\rho$. In the data, this dependence is actually substantial: assuming the generative model implied by the baseline spec, where all $Y_{ijr}$ are independent, then assuming the fitted $p_{ij}:=p_{j}(T_{i})$ were correct, we would predict that $\sim 60\%$ of vectors $Y_{ij}=(Y_{ij1},\dots,Y_{ijn_{ij}})$ were unanimous. However, it is actually 83% of them. The correlation parameter affects the final fit by downweighting AI-task pairs with many runs, as they carry redundant information.

我们通过相关参数 $\rho$ 处理运行集 $Y_{ij1},\dots,Y_{ijn_{ij}}$ 内部的依赖。在数据中，这种依赖实际上相当大：假设基线设定所隐含的生成模型成立，即所有 $Y_{ijr}$ 相互独立，再假设拟合出的 $p_{ij}:=p_{j}(T_{i})$ 是正确的，那么我们会预测约 $\sim 60\%$ 的向量 $Y_{ij}=(Y_{ij1},\dots,Y_{ijn_{ij}})$ 各次结果一致。但实际上有 83% 一致。相关参数会降低运行次数多的 AI—任务配对的权重，因为这些配对携带的信息是冗余的，从而影响最终拟合。

2.4 Evaluation with proper scoring rules

2.4 用适当评分规则评估

In this section, let $\mathcal{M}$ index modeling approaches, each of which supplies a success curve $p_{j,\mathcal{M}}(t)$ for every $\texttt{AI}_{j}$, which we assume to be monotone decreasing in $t$, so that the time horizons $\mathbf{th}_{q}^{\mathcal{M}}(\texttt{AI}_{j}):=p_{j,\mathcal{M}}^{-1}(q)$ are well defined. Let

本节中，令 $\mathcal{M}$ 索引各种建模方法，每种方法为每个 $\texttt{AI}_{j}$ 提供一条成功曲线 $p_{j,\mathcal{M}}(t)$，我们假设它关于 $t$ 单调递减，从而时间跨度 $\mathbf{th}_{q}^{\mathcal{M}}(\texttt{AI}_{j}):=p_{j,\mathcal{M}}^{-1}(q)$ 有良好定义。令

$$\mathbf{Y}=(Y_{ijr})_{i\leq I,\ j\leq J,\ r\leq n_{ij}},\qquad\mathbf{T}=(T_{1},\dots,T_{I})$$

collect, respectively, every run outcome in the table and every human time, so that the dataset is the pair $(\mathbf{T},\mathbf{Y})$.

分别汇集表中所有运行结果和所有人类耗时，于是数据集就是 $(\mathbf{T},\mathbf{Y})$ 这一对。

A scoring rule is then a function $S((\mathbf{T},\mathbf{Y}),\mathcal{M})$ of the data and the approach. We use the four scores below. The first three (marginal log score, Brier score, and elementary score) are oriented so that smaller is better, and we report each divided by its total weight $\sum_{i,j}n_{ij}w_{ij}$.

评分规则是数据和方法的函数 $S((\mathbf{T},\mathbf{Y}),\mathcal{M})$。我们使用下面四种分数。前三种（边际对数分数、Brier 分数和基本分数）都是越小越好，我们报告时把每种分数除以其总权重 $\sum_{i,j}n_{ij}w_{ij}$。

Marginal log score.

边际对数分数。

Also known as binary cross entropy. Writing $\hat{p}_{ij}:=p_{j,\mathcal{M}}(T_{i})$ for the predicted success probability, let

也就是二元交叉熵。把预测成功概率记为 $\hat{p}_{ij}:=p_{j,\mathcal{M}}(T_{i})$，令

$$S_{\mathrm{MLS}}((\mathbf{T},\mathbf{Y}),\mathcal{M})=\sum_{i=1}^{I}\sum_{j=1}^{J}w_{ij}\sum_{r=1}^{n_{ij}}\left(-\,Y_{ijr}\log\hat{p}_{ij}\;-\;(1-Y_{ijr})\log(1-\hat{p}_{ij})\right),$$

for some weights $w_{ij}>0$. This score only depends on $\mathcal{M}$ through the functions $p_{j,\mathcal{M}}(t)$.

其中 $w_{ij}>0$ 为权重。这个分数只通过函数 $p_{j,\mathcal{M}}(t)$ 依赖于 $\mathcal{M}$。

With all the weights equal to 1, minimizing this score in training is identical to maximizing log-likelihood when every $Y_{ijr}$ is assumed drawn independently. Even though such a model is certainly not a good fit in our application, that does not necessarily imply that the resulting success-curve functions $p_{j,\mathcal{M}}(T_{i})$ are not a good fit.

当所有权重都为 1 时，在训练中最小化这个分数，等同于在假设每个 $Y_{ijr}$ 独立抽取时最大化对数似然。尽管这样的模型在我们的应用中肯定拟合得不好，但这并不一定意味着由此得到的成功曲线函数 $p_{j,\mathcal{M}}(T_{i})$ 拟合得不好。

Brier score.

Brier 分数。

With the same definitions as in marginal log score, let

沿用边际对数分数中的定义，令

$$S_{\mathrm{BR}}((\mathbf{T},\mathbf{Y}),\mathcal{M})=\sum_{i=1}^{I}\sum_{j=1}^{J}w_{ij}\left(\sum_{r=1}^{n_{ij}}\left(Y_{ijr}-\hat{p}_{ij}\right)^{2}\right).$$

Like the marginal log score, this only depends on $\mathcal{M}$ through the functions $p_{j,\mathcal{M}}(t)$.

与边际对数分数一样，它只通过函数 $p_{j,\mathcal{M}}(t)$ 依赖于 $\mathcal{M}$。

Elementary binary score at level $q$.

水平 $q$ 下的基本二元分数。

For $q\in(0,1)$, define

对 $q\in(0,1)$，定义

$$\displaystyle S_{q}((\mathbf{T},\mathbf{Y}),\mathcal{M}) \\ \displaystyle=\sum_{i=1}^{I}\sum_{j=1}^{J}w_{ij}\sum_{r=1}^{n_{ij}}\left(q(1-Y_{ijr})1\{\hat{p}_{ij}>q\}+(1-q)Y_{ijr}1\{\hat{p}_{ij}<q\}\right) \\ \displaystyle=\sum_{i=1}^{I}\sum_{j=1}^{J}w_{ij}\sum_{r=1}^{n_{ij}}\Big(q(1-Y_{ijr})1\{T_{i}<\mathbf{th}_{q}^{\mathcal{M}}(\texttt{AI}_{j})\} \\ \displaystyle\hphantom{{}= \sum_{i = 1}^I \sum_{j = 1}^J w_{ij} \sum_{r = 1}^{n_{ij}} \Big(}+\,(1-q)Y_{ijr}1\{T_{i}>\mathbf{th}_{q}^{\mathcal{M}}(\texttt{AI}_{j})\}\Big)$$

assuming no ties between $\hat{p}_{ij}$ and $q$. This score only depends on $\mathcal{M}$ through the time horizons $\mathbf{th}_{q}$. In other words, this score directly measures the quality of the time horizon estimates.

这里假设 $\hat{p}_{ij}$ 与 $q$ 之间没有相等的情况。这个分数只通过时间跨度 $\mathbf{th}_{q}$ 依赖于 $\mathcal{M}$。换句话说，它直接衡量时间跨度估计的质量。

These scores are elementary in the sense that every proper scoring rule for a binary outcome is a nonnegative mixture of them [Schervish, 1989]; in particular, the Brier score satisfies $S_{\mathrm{BR}}=2\int S_{q}\,dq$. The curve $q\mapsto S_{q}((\mathbf{T},\mathbf{Y}),\mathcal{M})$ is the Murphy diagram of $\mathcal{M}$ [Ehm et al., 2016], which we also plot; one specification dominating another at every $q$ on the plot implies dominance under every proper score.

这些分数之所以称为“基本”分数，是因为二元结果的任何适当评分规则都是它们的非负混合 [Schervish, 1989]；特别地，Brier 分数满足 $S_{\mathrm{BR}}=2\int S_{q}\,dq$。曲线 $q\mapsto S_{q}((\mathbf{T},\mathbf{Y}),\mathcal{M})$ 就是 $\mathcal{M}$ 的墨菲图 [Ehm et al., 2016]，我们也把它画了出来；如果在图上的每个 $q$ 处，一个设定都优于另一个，那么它在每种适当评分规则下都占优。

At a single $q$, these scores are noisy, so for our evaluations we report the smoothed score $\tilde{S}_{q}((\mathbf{T},\mathbf{Y}),\mathcal{M}):=\frac{1}{0.1}\int_{-0.05}^{0.05}S_{q+\varepsilon}((\mathbf{T},\mathbf{Y}),\mathcal{M})\,d\varepsilon$. This can be thought of then as measuring an average of the quality of the time horizons $\mathbf{th}_{q+\varepsilon}$ within that window.

在单个 $q$ 处，这些分数噪声很大，因此评估时我们报告平滑后的分数 $\tilde{S}_{q}((\mathbf{T},\mathbf{Y}),\mathcal{M}):=\frac{1}{0.1}\int_{-0.05}^{0.05}S_{q+\varepsilon}((\mathbf{T},\mathbf{Y}),\mathcal{M})\,d\varepsilon$。可以认为它衡量的是该窗口内时间跨度 $\mathbf{th}_{q+\varepsilon}$ 的平均质量。

Full log-likelihood.

完整对数似然。

When a generative model is available, we can use this score. The (full) log-likelihood has the typical definition of

有生成模型时，可以使用这个分数。（完整）对数似然按通常方式定义为

$$S_{\mathrm{JLL}}((\mathbf{T},\mathbf{Y}),\mathcal{M}):=\log\Pr_{\mathcal{M}}(\mathbf{Y}\mid\mathbf{T}).$$

This scores the entire model $\mathcal{M}$, including correlations between the $Y$. In particular, it is possible to do poorly in estimating the $p_{j}(t)$ but still do well on the full log-likelihood.

它给整个模型 $\mathcal{M}$ 打分，包括各个 $Y$ 之间的相关性。特别地，一个模型可能对 $p_{j}(t)$ 估计得很差，却在完整对数似然上得分很好。

2.4.1 Evaluation weights

2.4.1 评估权重

The choice of weights $w_{ij}>0$ essentially is a choice of what distribution of prediction problems we would like to average over; it does not depend on the run index $r$.

选择权重 $w_{ij}>0$，本质上是选择我们想在什么样的预测问题分布上取平均；权重与运行索引 $r$ 无关。

Writing $F(i)$ for the task_family of $\texttt{task}_{i}$ and $n_{F}$ for the number of scored tasks belonging to family $F$, we use the following three weightings:

把 $\texttt{task}_{i}$ 所属的任务族记为 $F(i)$，把属于任务族 $F$ 的已评分任务数记为 $n_{F}$，我们使用以下三种加权：

$$w_{ij}=1/n_{ij}\ \ (\text{equal-task}),\qquad 1/(n_{F(i)}\,n_{ij})\ \ (\text{equal-family}),\qquad 1/(\sqrt{n_{F(i)}}\,n_{ij})\ \ (\text{sqrt-family}).$$

All of these schemes weight the AIs uniformly, but not the tasks. The equal task weighting targets a uniformly drawn task, adjusting for unequal numbers of runs. The equal family weighting instead targets a uniformly drawn task family, adjusting for unequal numbers of tasks in the family. Finally, the sqrt-family weighting is an intermediate used by METR for both fitting and evaluating their models.

这些方案对各 AI 的权重都是均匀的，对任务则不是。等任务加权（equal-task）以均匀抽取的任务为目标，校正运行次数不等的问题。等任务族加权（equal-family）则以均匀抽取的任务族为目标，校正各任务族任务数不等的问题。平方根任务族加权（sqrt-family）介于两者之间，METR 在拟合和评估模型时都使用它。

It is also possible to define versions of the scoring rules specific to each $\texttt{AI}_{j}$, by restricting the sum over $j$ to that single AI.

也可以为每个 $\texttt{AI}_{j}$ 单独定义评分规则，只需把对 $j$ 的求和限制在这一个 AI 上。

2.4.2 A suite of proper scoring rule metrics

2.4.2 一套适当评分规则指标

Our main numerical measure of model quality is based on crossing the above scoring rules with the weighting schemes, under 5-fold cross validation on task families. That is, we split the 79 task families into 5 approximately equally sized folds, and then evaluate marginal log score, Brier score, and the smoothed elementary binary score at $q=0.5$ and $q=0.8$ on each fold using a model trained from data out-of-fold. Crossing these four scores with the three weighting schemes gives 12 proper scoring rule metrics.

我们衡量模型质量的主要数值指标，是把上述评分规则与各种加权方案交叉组合，并在任务族上做 5 折交叉验证。也就是说，我们把 79 个任务族分成大小大致相等的 5 折，在每一折上用折外数据训练的模型，计算边际对数分数、Brier 分数，以及 $q=0.5$ 和 $q=0.8$ 处的平滑基本二元分数。四种分数与三种加权方案交叉，得到 12 项适当评分规则指标。

2.5 Fit results and diagnostics

2.5 拟合结果与诊断

The results of the scoring rule metrics are displayed in Figure 2, showing that the point estimates for our methods dominate the baseline on all metrics. Our chosen baseline itself dominates the original logistic regression specification on all metrics except one, namely equal-task $\tilde{S}_{q=0.5}$, so we do not report its results.

评分规则指标的结果见图 2：我们方法的点估计在所有指标上都优于基线。我们选用的基线本身，在除等任务加权 $\tilde{S}_{q=0.5}$ 之外的所有指标上都优于最初的逻辑回归设定，因此我们不再报告后者的结果。

Figure 2: Models 1 and 2 against the baseline across the scoring-rule suite. Each panel shows the paired difference in weighted per-run score between our model and the baseline. (Left is better for our models.) The confidence intervals use family as the unit $n=79$, at two nested levels: 80% thick, 95% thin. Point estimates for both of our models dominate the baseline at every displayed metric and weighting.

图 2：在评分规则套件上比较模型 1、模型 2 与基线。每个子图显示我们的模型与基线在加权单次运行分数上的配对差值（越靠左，我们的模型越好）。置信区间以任务族为单位（$n=79$），分两个嵌套水平：粗线为 80%，细线为 95%。在图中所示的每项指标和每种加权下，我们两个模型的点估计都优于基线。

We also plot the Murphy diagrams in Figure 3. For $q>0.4$, the baseline seems to be nearly dominated, partly explaining why we get favorable performance on Brier score and marginal log score, as they are just weighted mixtures of points on the Murphy diagram.

我们还在图 3 中画出了墨菲图。当 $q>0.4$ 时，基线几乎全面落后，这部分解释了为什么我们在 Brier 分数和边际对数分数上表现更好，因为这两种分数正是墨菲图上各点的加权混合。

Figure 3: Murphy diagrams of Models 1 and 2 against the baseline, with equal-family weighting. Left: Murphy diagrams of the baseline, Model 1, and Model 2. Right: Each model’s paired difference from the baseline ($\times 10^{-3}$), with its 95% band based on family $n=79$ units. For display, these curves are smoothed by a centered rolling mean over the $q$-grid ($\approx 0.03$ wide in $q$).

图 3：等任务族加权下模型 1、模型 2 与基线的墨菲图。左：基线、模型 1 和模型 2 的墨菲图。右：各模型与基线的配对差值（$\times 10^{-3}$），以及基于 $n=79$ 个任务族单位的 95% 置信带。为便于展示，这些曲线在 $q$ 网格上做了居中滚动平均平滑（窗口宽约 $\approx 0.03$）。

Additionally, we plot in-sample diagnostics in Figure 4 for six AIs with time horizons of 2–15 min. The shape of Models 1 and 2 bends to follow a flat region through roughly 2–30 minutes whose tasks Figure 1(d) shows have roughly equal difficulty, while the log-linear baseline is forced to cut straight through. Our fits substantially change the 50% time horizon estimates for these AIs, some of which go outside the original METR plot’s 95% confidence intervals.

此外，我们在图 4 中给出了六个时间跨度为 2–15 分钟的 AI 的样本内诊断。模型 1 和模型 2 的形状弯折过来，贴合大约 2–30 分钟的一段平坦区域（图 1(d) 显示这段区域内的任务难度大致相同），而对数线性的基线只能直线穿过。我们的拟合大幅改变了这些 AI 的 50% 时间跨度估计，其中一些甚至落到了原始 METR 图的 95% 置信区间之外。

A version separated by AIs is shown in the Appendix as Figure 9, showing that a large amount of the gain in the metrics was due to improvement in estimates for AIs with time horizons of 2–15 min, such as these six, which were most affected by the flat region of tasks with human time 2–30 min, as shown in Figure 1(d).

附录图 9 给出了按 AI 拆分的版本，显示指标上的提升很大一部分来自时间跨度为 2–15 分钟的 AI（例如这六个）的估计改进；如图 1(d) 所示，受人类耗时 2–30 分钟这段平坦区域影响最大的正是这些 AI。

Figure 4: In-sample fitted $p_{j}(t)$ against binned observed frequencies, for six AIs with time horizons of 2–15 min. Black points pool each AI’s runs within factor-4 human-time bins—width two in $\log_{2}$ minutes, with bin centers at 1 min, 4 min, 16 min, etc—plotted at the run-weighted mean human time within the bin. (The points are therefore unevenly spaced). Vertical bars are 95% Wald intervals $\hat{p}\pm 1.96\sqrt{\hat{p}(1-\hat{p})/N}$ over the bin’s $N$ runs.

图 4：六个时间跨度为 2–15 分钟的 AI 的样本内拟合 $p_{j}(t)$ 与分箱观测频率的对比。黑点把每个 AI 的运行按人类耗时以 4 倍为间隔分箱（在以分钟为单位的 $\log_{2}$ 尺度上宽度为 2，箱中心为 1 分钟、4 分钟、16 分钟等），画在箱内按运行次数加权的平均人类耗时处（因此各点间距不均）。竖线是基于箱内 $N$ 次运行的 95% Wald 区间 $\hat{p}\pm 1.96\sqrt{\hat{p}(1-\hat{p})/N}$。

3 Diagnostics for construct validity

3 构念效度诊断

3.1 Construct validity for capability anchoring

3.1 能力锚定的构念效度

We loosely define construct validity to be a property of a statistical estimand—whether, at the population level, it measures the construct that we wish it to measure. For example, when reading the METR plot one might assume, implicitly, that the construct of “AI software development capabilities” is well captured by the estimand of “50% time horizons on METR’s task suite”. But whether this is actually the case depends on how the human time of the suite’s tasks relates to their difficulty for an AI—the relationship between human time and AI difficulty.

我们粗略地把构念效度定义为统计估计目标（estimand）的一种性质：在总体层面上，它是否衡量了我们希望它衡量的构念。例如，读 METR 图时，人们可能默认“AI 软件开发能力”这一构念被“METR 任务套件上的 50% 时间跨度”这一估计目标很好地刻画了。但实际是否如此，取决于套件中任务的人类耗时与它们对 AI 的难度之间是什么关系，即人类耗时与 AI 难度的关系。

This consideration applies to using any interpretable, external feature besides human time. Let us refer to this key idea of time horizons as “capability anchoring”. Capability anchoring is in principle applicable to any benchmark dataset where $J$ AIs are set up to attempt $I$ tasks, or perhaps only a subset of them, and where the $I$ tasks have external annotations $T_{1},\dots,T_{I}$. The general approach is then roughly abstractable to the following steps:

除了人类耗时，使用任何可解释的外部特征都要考虑这一点。我们把时间跨度的这一核心想法称为“能力锚定”（capability anchoring）。原则上，能力锚定适用于任何这样的基准数据集：$J$ 个 AI 尝试 $I$ 个任务（也可能只尝试其中一部分），并且这 $I$ 个任务带有外部标注 $T_{1},\dots,T_{I}$。一般做法大致可以抽象为以下几步：

1. Learn task difficulties $\theta_{1},\dots,\theta_{I}$ and AI capabilities $\alpha_{1},\dots,\alpha_{J}$ jointly with some method (say, IRT), and hence learn a function $\tilde{p}(\theta_{i},\alpha_{j})$ as our best predictor of the chance that AI $j$ will succeed at task $i$.
2. Regress the learned task difficulties $\theta_{1},\dots,\theta_{I}$ on the external annotations $T_{1},\dots,T_{I}$ to learn a mapping $f$ from annotation to difficulty.
3. Define AI $j$’s “anchored capability” as a 50% horizon, via $\mathbf{th}_{0.5}:=f^{-1}(\theta_{j}^{\star})$, where $\theta_{j}^{\star}$ solves $\tilde{p}(\theta_{j}^{\star},\alpha_{j})=1/2$. (This is the annotation value whose predicted difficulty gives AI $j$ an even chance of success.) Use the “anchored capability” as an interpretable scale for the AI’s capability.

1. 用某种方法（比如 IRT）同时学习任务难度 $\theta_{1},\dots,\theta_{I}$ 和 AI 能力 $\alpha_{1},\dots,\alpha_{J}$，从而学到一个函数 $\tilde{p}(\theta_{i},\alpha_{j})$，作为 AI $j$ 完成任务 $i$ 的概率的最佳预测。
2. 把学到的任务难度 $\theta_{1},\dots,\theta_{I}$ 对外部标注 $T_{1},\dots,T_{I}$ 做回归，学到一个从标注到难度的映射 $f$。
3. 把 AI $j$ 的“锚定能力”定义为 50% 跨度，即 $\mathbf{th}_{0.5}:=f^{-1}(\theta_{j}^{\star})$，其中 $\theta_{j}^{\star}$ 满足 $\tilde{p}(\theta_{j}^{\star},\alpha_{j})=1/2$。（这是这样一个标注值：按它预测出的难度，AI $j$ 成功与失败的机会各半。）用“锚定能力”作为 AI 能力的可解释尺度。

In our setting, it is possible actually to follow these steps by using a Rasch model to perform Step 1 and a monotone spline to fit $f$ in Step 2. The function $p_{j}(t):=\tilde{p}(f(t),\alpha_{j})$ then essentially follows the specification of Model 1 but with a different way of learning the parameters; we found the result to be nearly competitive with both Model 1 and 2. However, we prefer to use these steps as a conceptual way to understand methods for capability anchoring: Model 1 and the baseline implicitly perform Steps 1 and 2 jointly, though the baseline assumes that the mapping $f$ is linear in log human time. Model 2’s EM algorithm alternates between Step 1 and Step 2 in the course of fitting, using the fitted $f$ to improve the estimates of difficulties and capabilities and then using those estimates to re-fit $f$.

在我们的设定中，确实可以按这些步骤来做：第 1 步用 Rasch 模型，第 2 步用单调样条拟合 $f$。这样得到的函数 $p_{j}(t):=\tilde{p}(f(t),\alpha_{j})$ 基本上就是模型 1 的设定，只是参数的学习方式不同；我们发现其结果与模型 1 和模型 2 都相差无几。不过，我们更愿意把这些步骤当作理解能力锚定方法的概念框架：模型 1 和基线隐含地把第 1 步和第 2 步合在一起做，只是基线假设映射 $f$ 关于人类耗时的对数是线性的。模型 2 的 EM 算法在拟合过程中交替执行第 1 步和第 2 步，用拟合出的 $f$ 改进难度和能力的估计，再用这些估计重新拟合 $f$。

Given this abstraction for capability anchoring, we posit that the construct validity of time horizons and capability anchoring in general comes from affirmative answers to the following “checklist” questions:

有了能力锚定的这一抽象，我们认为，时间跨度乃至一般能力锚定的构念效度，来自对以下“清单”问题的肯定回答：

1. Are the tasks representative of real tasks—does solving them mean the AI is “capable” in the sense that we wish?
2. Do two numbers $(\theta,\alpha)$ and the function $\tilde{p}$ model the difficulty-capability interaction well, or do more numbers/different specifications fit significantly better?
3. Does the annotation have an objective meaning? Is it the same objective meaning across different tasks? Does this objective meaning capture a desired construct? Relative to this objective meaning, is the annotation measured well, and if not, can we account for it?
4. (Predictiveness) Are the task annotations predictive of the difficulties, and hence AI success of the tasks?
5. (Comparability) Is the relationship between the annotation and the difficulties different depending on the annotation? For anchored capabilities, can we interpret equal differences among AIs equally? If not, can we account for it?

1. 这些任务是否代表真实任务？完成它们是否意味着 AI 具备我们所说的那种“能力”？
2. 两个数 $(\theta,\alpha)$ 加上函数 $\tilde{p}$，能否很好地刻画难度与能力之间的交互？还是用更多的数或不同的设定会拟合得明显更好？
3. 标注是否有客观含义？不同任务上的客观含义是否相同？这种客观含义是否刻画了想要的构念？相对于这种客观含义，标注是否测得准确？如果不准确，能否加以校正？
4. （预测性）任务标注能否预测任务难度，从而预测 AI 在任务上的成功？
5. （可比性）标注与难度之间的关系是否随标注取值而变化？对于锚定能力，AI 之间相等的差距能否作同等解读？如果不能，能否加以校正？

Many critiques of the construct validity of time horizons address the first three questions. In this work, we simply assume that answers to the first three questions are affirmative. Our diagnostic plots are then comparatively narrow—only addressing the last two questions of predictiveness and comparability.

许多对时间跨度构念效度的批评针对的是前三个问题。本文直接假定前三个问题的答案都是肯定的。因此我们的诊断图范围相对较窄，只处理后两个问题，即预测性和可比性。

The issue raised by predictiveness is this. In the setting of Section 2.1, suppose AIs A and B have 50% time horizons $T_{A}$ and $T_{B}$, with $T_{B}\gg T_{A}$, representing a large jump in the time horizon. Interpreting this as a large increase in capabilities seems misleading if both $p_{B}(T_{A})-p_{A}(T_{A})$ and $p_{B}(T_{B})-p_{A}(T_{B})$ are small, i.e. if human time is not predictive of AI success in $[T_{A},T_{B}]$. This would be reflected by $p_{B}(\cdot)$ and $p_{A}(\cdot)$ being flat in that region.

预测性引出的问题是这样的。在第 2.1 节的设定下，假设 AI A 和 B 的 50% 时间跨度分别为 $T_{A}$ 和 $T_{B}$，且 $T_{B}\gg T_{A}$，即时间跨度有一次大的跃升。如果 $p_{B}(T_{A})-p_{A}(T_{A})$ 和 $p_{B}(T_{B})-p_{A}(T_{B})$ 都很小，也就是说在 $[T_{A},T_{B}]$ 区间内人类耗时不能预测 AI 的成功，那么把这次跃升解读为能力的大幅提高就是误导性的。这种情况会表现为 $p_{B}(\cdot)$ 和 $p_{A}(\cdot)$ 在该区间内都是平的。