Faster Cascades via Speculative Decoding

用投机解码加速级联

Harikrishna Narasimhan    Wittawat Jitkrittum Affiliation:  Google Research, Mountain View     Google Research, New YorkCorresponding author: hnarasimhan@google.com[5pt]    Ankit Singh Rawat Affiliation:  Google Research, Mountain View     Google Research, New YorkCorresponding author: hnarasimhan@google.com[5pt]    Seungyeon Kim Affiliation:  Google Research, Mountain View     Google Research, New YorkCorresponding author: hnarasimhan@google.com[5pt]    Neha Gupta    Aditya Krishna Menon Affiliation:  Google Research, Mountain View     Google Research, New YorkCorresponding author: hnarasimhan@google.com[5pt]    Sanjiv Kumar Affiliation:  Google Research, Mountain View     Google Research, New YorkCorresponding author: hnarasimhan@google.com[5pt]

Abstract

摘要

Cascades and speculative decoding are two common approaches to improving language models’ inference efficiency. Both approaches involve interleaving models of different sizes, but via fundamentally distinct mechanisms: cascades employ a deferral rule that invokes the larger model only for “hard” inputs, while speculative decoding uses speculative execution to primarily invoke the larger model in parallel verification mode. These mechanisms offer different benefits: empirically, cascades are often capable of yielding better quality than even the larger model, while theoretically, speculative decoding offers a guarantee of quality-neutrality. In this paper, we leverage the best of both these approaches by designing new speculative cascading techniques that implement their deferral rule through speculative execution. We characterize the optimal deferral rule for our speculative cascades, and employ a plug-in approximation to the optimal rule. Through experiments with T5 models on benchmark language tasks, we show that the proposed approach yields better cost-quality trade-offs than cascading and speculative decoding baselines.

级联与投机解码是提升语言模型推理效率的两种常见做法。二者都交错使用不同规模的模型，但机制根本不同：级联用延迟规则仅在「难」输入上调用更大模型，而投机解码用投机执行，使更大模型主要以并行验证模式被调用。两种机制各有好处：经验上级联常能给出甚至优于更大模型的质量，而理论上级联投机解码可保证质量中性。本文通过设计新的投机级联技术，把二者长处结合起来：用投机执行实现延迟规则。我们刻画投机级联的最优延迟规则，并用插件近似实现该规则。在基准语言任务上用 T5 模型的实验表明，所提方法比级联与投机解码基线给出更好的代价–质量折中。

1 Introduction

1 引言

Large language models (LLMs) have demonstrated significant advances in quality on a range of natural language processing tasks [33, 34, 5, 2, 11, 48, 12, 41, 1, 43, 42], at the cost of a significant increase in inference latency. This has sparked a growing body of literature on reducing LMs’ inference costs without (overly) compromising on quality [15, 32, 36, 26, 7, 37, 40]. One such line of work involves constructing a family of models of various sizes (e.g., a small and large model), and suitably orchestrating amongst them to make a prediction. Two canonical instantiations of this strategy are model cascading [47, 28, 45, 23, 14, 8, 18, 13] and speculative decoding [39, 7, 26, 40, 49].

大语言模型（LLM）在一系列自然语言处理任务上质量显著提升 [33, 34, 5, 2, 11, 48, 12, 41, 1, 43, 42]，代价是推理延迟明显增加。这催生了大量在不过度牺牲质量的前提下降低 LM 推理成本的工作 [15, 32, 36, 26, 7, 37, 40]。其中一条路线是构造不同规模的模型族（例如一小一大），并在其间恰当编排以做预测。该策略的两种典型实例是模型级联 [47, 28, 45, 23, 14, 8, 18, 13] 与投机解码 [39, 7, 26, 40, 49]。

While similar in spirit, cascades and speculative decoding are fundamentally different in details. Cascades employ a deferral rule to identify “hard” inputs, and only invoke larger models on such inputs. For example, in a two-model cascade, one first invokes the smaller model, and uses its associated probability of the generated output to decide whether to defer to the larger model. By contrast, speculative decoding uses a small model to draft a block of tokens via standard auto-regressive decoding, which are then verified in parallel by a large model. One then accepts all drafted tokens until the first “implausible” one, which is rolled back based on the larger LM’s prediction.

尽管精神相近，级联与投机解码在细节上根本不同。级联用延迟规则识别「难」输入，只在此类输入上调用更大模型。例如在双模型级联中，先调用较小模型，再根据其生成输出的概率决定是否延迟到较大模型。相对地，投机解码用小模型经标准自回归解码起草一块 token，再由大模型并行验证；随后接受起草 token 直至第一个「不可信」者，并按更大 LM 的预测回滚。

Owing to their different mechnanisms, both methods have complementary strengths. Cascades seek to output distributions that have the best quality for a given cost budget, and are empirically observed to often yield better accuracies than even the individual models they are constructed with [22, 24] (§3). By contrast, speculative decoding is theoretically guaranteed to match the output distribution (or a close approximation thereof [44]), and are practically observed to provide impressive speed-ups [39, 7, 26, 40]. Given the complementary nature of these two approaches, a natural question that arises is whether we can leverage the best of both techniques.

因机制不同，两者优势互补。级联追求在给定代价预算下质量最好的输出分布，经验上常比构成它的单个模型更准 [22, 24]（§3）。相对地，投机解码在理论上保证匹配输出分布（或其近似 [44]），实践中也能带来可观加速 [39, 7, 26, 40]。鉴于二者互补，自然要问：能否兼得两者之长。

In this paper, we do so by designing new techniques for two-model cascades that implement their deferral rule in a speculative manner: we have the smaller model generate drafts auto-regressively, and the larger model execute in parallel on the drafts to decide whether or not to defer on themWe show that this speculative cascading approach yields better cost-quality trade-offs than both standard cascades and speculative decoding. In detail, we make the following contributions:

本文通过设计用投机方式实现延迟规则的双模型级联新技术来回答：让较小模型自回归生成草稿，较大模型在草稿上并行执行，以决定是否对其延迟。我们表明这种投机级联比标准级联与投机解码给出更好的代价–质量折中。具体贡献如下：

(i) We introduce a general recipe for speculative execution, where we seek to mimic a general target distribution that interleaves the drafter’s and verifier’s output distributions. Lossy speculative sampling [44] is a special case of this recipe for a particular target distribution (§4.1).
(ii) We show how common cascading deferral rules, such as Chow’s rule [10] and confidence-difference thresholding [22], can be implemented speculatively by plugging in the corresponding target distribution into the proposed framework. We refer to these as speculative cascades (§4.2).
(iii) We characterize the theoretically optimal deferral rule for a speculative cascade, and design a speculative cascading technique that implements a plug-in estimate to the optimal rule (Lem. 4).
(iv) Through experiments on benchmark language tasks, we show that speculative cascades constructed from T5 models [35] of different sizes are able to provide better cost-quality trade-offs than their sequential cascade and speculative decoding counterparts (§6).

(i) 我们给出投机执行的一般配方：目标是模仿交错起草者与验证者输出分布的一般目标分布。有损投机采样 [44] 是该配方在特定目标分布下的特例（§4.1）。
(ii) 我们说明如何把 Chow 规则 [10]、置信差阈值 [22] 等常见级联延迟规则，通过把相应目标分布插入所提框架而以投机方式实现；称之为投机级联（§4.2）。
(iii) 我们刻画投机级联的理论最优延迟规则，并设计实现该最优规则插件估计的投机级联技术（引理 4）。
(iv) 在基准语言任务上的实验表明，由不同规模 T5 [35] 构成的投机级联，能比其顺序级联与投机解码对应物给出更好的代价–质量折中（§6）。

2 A Tale of Two Efficient LM Inference Strategies

2 两种高效 LM 推理策略

Let $\mathscr{V}$ denote a finite vocabulary of tokens, with $\mathscr{V}^{*}$ the set of all sequences generated by this vocabulary. Let $\Delta_{\mathscr{V}}$ denote the set of all probability distributions over tokens in $\mathscr{V}$. Given an arbitrary length sequence $x=x_{1}x_{2}\ldots x_{L}\in\mathscr{V}^{*}$ and index $i\leq L$, denote by $x_{<i}=x_{1}x_{2}\ldots x_{i-1}$. A language model (LM) is a probability distribution over $\mathscr{V}^{*}$. Let $\mathbb{P}$ denote the ground-truth probability distribution over $\mathscr{V}^{*}$. This could be, for example, a distribution over prompt-response pairs that the LM may encounter during deployment, or a distribution of sequences used to pre-train the LM. We will measure the quality of an LM based on how closely it mimics $\mathbb{P}$.

令 $\mathscr{V}$ 为有限词表，$\mathscr{V}^{*}$ 为由该词表生成的全部序列集合。令 $\Delta_{\mathscr{V}}$ 为词表上全体概率分布的集合。对任意长度序列 $x=x_{1}x_{2}\ldots x_{L}\in\mathscr{V}^{*}$ 与指标 $i\leq L$，记 $x_{<i}=x_{1}x_{2}\ldots x_{i-1}$。语言模型（LM）是 $\mathscr{V}^{*}$ 上的概率分布。令 $\mathbb{P}$ 为 $\mathscr{V}^{*}$ 上的真实概率分布，例如部署时 LM 可能遇到的提示–回答对分布，或预训练所用序列分布。我们将按 LM 与 $\mathbb{P}$ 的接近程度度量质量。

Suppose we are provided two LMs $q$ and $p$, where $p$ is the larger model. Our goal is to design an inference strategy that selectively invokes $q$ and $p$ to trade-off between quality and latency (which may be approximated by the fraction of times that $p$ is invoked). We will denote by $q(x_{t}|x_{<t})$ the probability $q$ associates to token $x_{t}\in\mathscr{V}$ given prefix $x_{<t}\in\mathscr{V}^{t-1}$, and by $p(x_{t}|x_{<t})$ the same distribution from model $p$. Whenever it is clear from context, we will hide the conditioning on prefix $x_{<t}$, and use the shorthand $p_{t}(\cdot)$ for $p(\cdot|x_{<t})$ and $q_{t}(\cdot)$ for $q(\cdot|x_{<t})$.

假设有两个 LM $q$ 与 $p$，其中 $p$ 更大。目标是设计在二者间选择性调用的推理策略，以在质量与延迟（可用调用 $p$ 的比例近似）之间折中。记 $q(x_{t}|x_{<t})$ 为给定前缀 $x_{<t}\in\mathscr{V}^{t-1}$ 时 $q$ 赋给 token $x_{t}\in\mathscr{V}$ 的概率，$p(x_{t}|x_{<t})$ 同理。上下文清楚时省略对前缀 $x_{<t}$ 的条件，简写 $p_{t}(\cdot)$ 为 $p(\cdot|x_{<t})$、$q_{t}(\cdot)$ 为 $q(\cdot|x_{<t})$。

Cascades.

Cascades are an effective strategy to trade-off cost and quality by having the smaller model $q$ handle the “easy” samples, and the larger model $p$ handle the “difficult” samples [18, 50]. A common approach to cascading is confidence-based thresholding or Chow’s rule [10, 22], where we first run $q$ on the given input, and defer to $p$ when $q$’s confidence for its generated response is sufficiently low. This strategy is typically implemented at the sequence-level, where for a given prefix $x_{<m}$ we invoke $q$ to generate a complete response $x_{m}\ldots x_{m+n}$. We evaluate the predicted probability from $q$ for this response, and check whether it falls below a threshold $\alpha\in[0,1]$:

级联。

级联通过让较小模型 $q$ 处理「易」样本、较大模型 $p$ 处理「难」样本，有效折中代价与质量 [18, 50]。常见做法是基于置信的阈值或 Chow 规则 [10, 22]：先在给定输入上跑 $q$，当 $q$ 对其生成回答的置信足够低时延迟到 $p$。该策略通常在序列级实现：对给定前缀 $x_{<m}$，调用 $q$ 生成完整回答 $x_{m}\ldots x_{m+n}$，再评估 $q$ 对该回答的预测概率，检查是否低于阈值 $\alpha\in[0,1]$：

$$\displaystyle q(x_{m}\ldots x_{m+n}\,|\,x_{<m})<1-\alpha. \tag{1}$$

If the above holds, we defer to $p$ to generate a new response; otherwise, we retain $q$’s response. One may then tune the threshold to achieve a desired cost-quality trade-off. The literature also offers variants of Chow’s rule that use a more nuanced aggregation of per-token uncertainties [18].

若成立，则延迟到 $p$ 生成新回答；否则保留 $q$ 的回答。随后可调阈值以达到期望的代价–质量折中。文献中也有对逐 token 不确定性做更细致聚合的 Chow 规则变体 [18]。

Speculative decoding.

Speculative decoding is an alternate inference strategy that applies token-level interleaving between $q$ and $p$, resulting in provably matching the larger model quality at a reduced inference cost [39, 26]. Given a prefix $x_{<t}$, we sample $\gamma$ draft tokens $x_{t},\ldots,x_{t+\gamma-1}$ auto-regressively from $q$, and run $p$ in parallel on the $\gamma$ prefixes $x_{<t},\ldots,x_{<t+\gamma-1}$, and verify if the generated tokens can be accepted. We then rollback to the first token $t+j^{*}$ that was rejected, replace $x_{t+j^{*}}$with a new token, and repeat the drafting-verification process with $x_{<t+j^{*}+1}$ as the new prefix.

投机解码。

投机解码是另一种推理策略：在 $q$ 与 $p$ 之间做token 级交错，在降低推理成本的同时可证明地匹配更大模型的质量 [39, 26]。给定前缀 $x_{<t}$，从 $q$ 自回归采样 $\gamma$ 个草稿 token $x_{t},\ldots,x_{t+\gamma-1}$，再在 $\gamma$ 个前缀 $x_{<t},\ldots,x_{<t+\gamma-1}$ 上并行跑 $p$，验证生成的 token 是否可接受。然后回滚到第一个被拒绝的 token $t+j^{*}$，替换 $x_{t+j^{*}}$，并以 $x_{<t+j^{*}+1}$ 为新前缀重复起草–验证。

During the verification stage, a draft token $x_{t+j}$ generated by $q$ is accepted with probability $\min\left(1,\frac{p_{t+j}(x_{t+j})}{q_{t+j}(x_{t+j})}\right)$ and rejected otherwise, where recall the shorthand $q_{t+j}(\cdot)=q(\cdot|x_{<t+j})$ and $p_{t+j}(\cdot)=p(\cdot|x_{<t+j})$. A rejected token is then replaced by a new token sampled from a modified distribution $\operatorname{norm}\left(\max\left\{0,\,p_{t+j}(\cdot)-q_{t+j}(\cdot)\right\}\right),$ where $\text{norm}(\cdot)$ denotes normalization to sum to 1. This sampling process is provably equivalent to sampling $\gamma$ tokens auto-regressively from $p$ for prefix $x_{<t}$ [26]. We summarize this speculative sampling procedure in Algorithm 1. Each invocation of this algorithm generates at most $\gamma+1$ next tokens for a given prefix $x_{<t}$. One may run this algorithm multiple times to generate a complete output sequence.

验证阶段，由 $q$ 生成的草稿 token $x_{t+j}$ 以概率 $\min\left(1,\frac{p_{t+j}(x_{t+j})}{q_{t+j}(x_{t+j})}\right)$ 被接受，否则拒绝；其中 $q_{t+j}(\cdot)=q(\cdot|x_{<t+j})$、$p_{t+j}(\cdot)=p(\cdot|x_{<t+j})$。被拒绝的 token 再从修正分布 $\operatorname{norm}\left(\max\left\{0,\,p_{t+j}(\cdot)-q_{t+j}(\cdot)\right\}\right)$ 采样替换，其中 $\text{norm}(\cdot)$ 表示归一化使和为 1。该采样过程可证明等价于对前缀 $x_{<t}$ 从 $p$ 自回归采样 $\gamma$ 个 token [26]。算法 1 汇总该投机采样；每次调用至多为给定前缀 $x_{<t}$ 生成 $\gamma+1$ 个下一 token。可多次运行以生成完整输出序列。

In practice, one may employ a lossy variant [44] of the above sampling that allows some deviation from verifier’s distribution $p$. In this case, a draft token $x_{t+j}$ is accepted with probability $\min\left(1,\frac{p_{t+j}(x_{t+j})}{(1-\alpha)\cdot q_{t+j}(x_{t+j})}\right)$, where $\alpha\in[0,1)$ is a strictness parameter, with higher values indicating greater deviation from $p$. A rejected token may then be replaced by a token sampled from the residual distribution $\operatorname{norm}\left(\max\left\{0,\,\frac{1}{\beta}\cdot p_{t+j}(\cdot)-q_{t+j}(\cdot)\right\}\right),$ where $\beta\geq 1-\alpha$ is a parameter that depends on $\alpha$, $q$ and $p$. A common heuristic is to simply set $\beta=1$ [52].

实践中可采用上述采样的有损变体 [44]，允许相对验证者分布 $p$ 有一定偏离。此时草稿 token $x_{t+j}$ 以概率 $\min\left(1,\frac{p_{t+j}(x_{t+j})}{(1-\alpha)\cdot q_{t+j}(x_{t+j})}\right)$ 被接受，其中 $\alpha\in[0,1)$ 为严格度参数，越大表示相对 $p$ 偏离越大。被拒绝的 token 可从残差分布 $\operatorname{norm}\left(\max\left\{0,\,\frac{1}{\beta}\cdot p_{t+j}(\cdot)-q_{t+j}(\cdot)\right\}\right)$ 采样替换，其中 $\beta\geq 1-\alpha$ 依赖于 $\alpha$、$q$ 与 $p$。常见启发式是直接取 $\beta=1$ [52]。

Inference strategy	Deferral decision $\delta(q,p)$	Target distribution $\pi(x)$	Execution
SpecDecoding [26]	-	$p(x)$	Speculative
Lossy SpecDecoding [44]	-	$\max\{\min\{q(x),\frac{p(x)}{1-\alpha}\},\frac{p(x)}{\beta}\}$	Speculative
BiLD* [24]	$\mathbf{1}\big(\,D(q,p)>\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	Speculative
TokenCascade [Chow] [10]	$\mathbf{1}\big(\max_{v}q(v)<1-\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	Sequential
Oracle [Diff] [22]	$\mathbf{1}\big(\max_{v}q(v)<\max_{v}p(v)-\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	Oracle
SpecCascade [Chow]	$\mathbf{1}\big(\max_{v}q(v)<1-\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	Speculative
SpecCascade [Diff]	$\mathbf{1}\big(\max_{v}q(v)<\max_{v}p(v)-\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	Speculative
SpecCascade [OPT]	$\mathbf{1}\big(\max_{v}q(v)<\max_{v}p(v)-\alpha\cdot D_{\textrm{TV}}(p,q)\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	Speculative

Table 1: Target distributions associated with different inference algorithms, where $\alpha$ is a free parameter and $\beta\geq 1-\alpha$ is a parameter dependent on $\alpha$, $q$ and $p$. The last column indicates whether the execution is sequential (Algorithm 2), via an oracle (Algorithm 3), or speculative (Algorithm 5) [26]. The third row presents a variant of the BiLD algorithm of [24], where $D(q,p)$ is a measure of discrepancy between $q$ and $p$; the original algorithm differs in the use of a deterministic speculative decoding procedure with a dynamic draft window (see §5). See (6) for more details on $\delta$.

推理策略	延迟决策 $\delta(q,p)$	目标分布 $\pi(x)$	执行方式
SpecDecoding [26]	-	$p(x)$	投机
Lossy SpecDecoding [44]	-	$\max\{\min\{q(x),\frac{p(x)}{1-\alpha}\},\frac{p(x)}{\beta}\}$	投机
BiLD* [24]	$\mathbf{1}\big(\,D(q,p)>\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	投机
TokenCascade [Chow] [10]	$\mathbf{1}\big(\max_{v}q(v)<1-\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	顺序
Oracle [Diff] [22]	$\mathbf{1}\big(\max_{v}q(v)<\max_{v}p(v)-\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	神谕
SpecCascade [Chow]	$\mathbf{1}\big(\max_{v}q(v)<1-\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	投机
SpecCascade [Diff]	$\mathbf{1}\big(\max_{v}q(v)<\max_{v}p(v)-\alpha\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	投机
SpecCascade [OPT]	$\mathbf{1}\big(\max_{v}q(v)<\max_{v}p(v)-\alpha\cdot D_{\textrm{TV}}(p,q)\big)$	$(1-\delta)\cdot q(x)+\delta\cdot p(x)$	投机

表 1：不同推理算法关联的目标分布，其中 $\alpha$ 为自由参数，$\beta\geq 1-\alpha$ 依赖于 $\alpha$、$q$ 与 $p$。末列标明执行是顺序（算法 2）、经神谕（算法 3）还是投机（算法 5） [26]。第三行是 [24] 中 BiLD 算法的变体，$D(q,p)$ 为 $q$ 与 $p$ 的差异度量；原算法在使用带动态草稿窗口的确定性投机解码上有所不同（见 §5）。关于 $\delta$ 的更多细节见 (6)。

3 Cascades Meet Speculative Decoding

3 级联遇上投机解码

Both cascades and speculative decoding interleave models of different sizes to reduce inference cost, but fundamentally differ in the mechanisms they use. As a step towards comparing the strengths and weaknesses of these approaches, we first describe how one may design a token-level cascade.

级联与投机解码都交错不同规模模型以降低推理成本，但机制根本不同。作为比较二者优劣的一步，我们先说明如何设计 token 级级联。

3.1 Warm-up: Token-level cascades

3.1 热身：Token 级级联

It is straight-forward to extend the sequence-level Chow’s rule from §2 to form a token-level cascade between $q$ and $p$. For a prefix $x_{<t}$, we first compute the smaller model’s distribution $q(\cdot|x_{<t})$, and check whether $\max_{v\in\mathscr{V}}\,q(v|x_{<t})$ is below a pre-chosen threshold. if so, we evaluate $p(\cdot|x_{<t})$, and sample $x_{t}\sim p(\cdot|x_{<t})$; otherwise, we sample $x_{t}\sim q(\cdot|x_{<t})$.

把 §2 的序列级 Chow 规则扩展为 $q$ 与 $p$ 之间的token 级级联很直接。对前缀 $x_{<t}$，先计算较小模型分布 $q(\cdot|x_{<t})$，检查 $\max_{v\in\mathscr{V}}\,q(v|x_{<t})$ 是否低于预设阈值；若是，则评估 $p(\cdot|x_{<t})$ 并从中采样 $x_{t}$；否则从 $q(\cdot|x_{<t})$ 采样 $x_{t}$。

More generally, we may design a token-level deferral rule ${r}:\mathscr{V}^{t-1}\rightarrow\{0,1\}$ that takes the prefix $x_{<t}$ as input and outputs a binary decision, with ${r}(x_{<t})=1$ indicating that we defer to $p$ (i.e., draw a sample from $p$ rather than $q$). For example, token-level Chow’s rule can be written as:

更一般地，可设计 token 级延迟规则 ${r}:\mathscr{V}^{t-1}\rightarrow\{0,1\}$，以前缀 $x_{<t}$ 为输入输出二元决策，${r}(x_{<t})=1$ 表示延迟到 $p$（即从 $p$ 而非 $q$ 抽样）。例如，token 级 Chow 规则可写成：

$$\displaystyle{r}_{\rm\tt Chow}(x_{<t})=1~\iff~\max_{v\in\mathscr{V}}\,q(v|x_{<t})<1-\alpha, \tag{2}$$

where $\alpha$ is a cost parameter; the higher the value, the lower is the frequency of deferral to $p$. One may also use a confidence measure different from the maximum probability, such as the entropy of the small model’s probability distribution. We elaborate in §B that the choice of confidence measure would depend on the evaluation metric of interest; (2) is typically prescribed when the cascade’s quality is evaluated in terms of its accuracy against the ground-truth distribution on individual tokens, whereas entropy is prescribed when the metric of interest is the cross-entropy loss.

其中 $\alpha$ 为代价参数；越大则延迟到 $p$ 的频率越低。也可使用异于最大概率的置信度量，例如小模型概率分布的熵。我们在 §B 中说明置信度量的选择取决于关心的评测指标；(2) 通常在按真实分布上逐 token 准确率评测级联质量时采用，而交叉熵损失为指标时更宜用熵。

3.2 Optimal token-level cascade deferral

3.2 最优 token 级级联延迟

While Chow’s rule (2) is easy to implement, it can be sub-optimal if the smaller model’s maximum token probability is not reflective of which of the two models are better equipped to predict the next token for a given prefix [22]. Given this, it is natural to ask what the optimal deferral rule $r$ for a token-cascade looks like, and whether we can reasonably approximate this rule.

虽 Chow 规则 (2) 易于实现，但若较小模型的最大 token 概率不能反映两个模型中谁更擅长对给定前缀预测下一 token，则可能次优 [22]。因此自然要问：token 级联的最优延迟规则 $r$ 长什么样，以及能否合理近似。

For this, we must first specify an objective to minimize at each step $t$. Following the prior cascade literature [22, 18], a reasonable objective to minimize is the expected loss from the deferral rule against the ground-truth distribution $\mathbb{P}$, with an added cost for deferring to the larger model. We state this below for a fixed prefix $x_{<t}$, using the short-hand $q_{t}(\cdot)$ for $q(\cdot|x_{<t})$ and $p_{t}(\cdot)$ for $p(\cdot|x_{<t})$:

为此须先指定每步 $t$ 要最小化的目标。遵循先前级联文献 [22, 18]，合理目标是：相对真实分布 $\mathbb{P}$，延迟规则的期望损失，外加延迟到更大模型的代价。对固定前缀 $x_{<t}$，用简写 $q_{t}(\cdot)$、$p_{t}(\cdot)$ 陈述如下：

$$\displaystyle L_{\rm def}(r;x_{<t}) \\ \displaystyle=\mathbb{E}_{x_{t}\sim\mathbb{P}(\cdot|x_{<t})}\Big[\big(1-r(x_{<t})\big)\cdot\ell(x_{t},q_{t})+r(x_{<t})\cdot\big(\ell(x_{t},p_{t})+\alpha\big)\Big], \tag{3}$$

for a cost penalty $\alpha\geq 0$ and loss function $\ell:\mathscr{V}\times\Delta_{\mathscr{V}}\rightarrow\mathbb{R}_{+}$. Common choices for $\ell$ include the 0-1 loss $\ell_{\text{0-1}}(x_{t},q_{t})=\mathbf{1}\left(x_{t}\neq\operatorname{argmax}_{v}q_{t}(v)\right)$ and the log loss $\ell_{\log}(x_{t},q_{t})=-\log\left(q_{t}(x_{t})\right).$

其中代价惩罚 $\alpha\geq 0$，损失函数 $\ell:\mathscr{V}\times\Delta_{\mathscr{V}}\rightarrow\mathbb{R}_{+}$。$\ell$ 的常见选择包括 0-1 损失 $\ell_{\text{0-1}}(x_{t},q_{t})=\mathbf{1}\left(x_{t}\neq\operatorname{argmax}_{v}q_{t}(v)\right)$ 与对数损失 $\ell_{\log}(x_{t},q_{t})=-\log\left(q_{t}(x_{t})\right).$

Lemma 1(Optimal deferral for token-level cascades [22]).

The minimizer of (3) is of the form:

引理 1（Token 级级联的最优延迟 [22]）。

(3) 的最小化子形如：

$$\displaystyle r^{*}(x_{<t})=1~~\iff~~\mathbb{E}_{x_{t}\sim\mathbb{P}(\cdot|x_{<t})}\left[\ell(x_{t},q_{t})\right]\,>\,\mathbb{E}_{x_{t}\sim\mathbb{P}(\cdot|x_{<t})}\left[\ell(x_{t},p_{t})\right]+\alpha. \tag{4}$$

Intuitively, we compare the expected loss from $q$ with the expected cost of invoking $p$, and decide to defer when the latter is smaller. We note here that this optimization problem is set up for a fixed prefix $x_{<t}$. One may also consider the coupled optimization problem across all prefixes from 1 to $T$.

直观上，我们比较来自 $q$ 的期望损失与调用 $p$ 的期望代价，当后者更小时选择延迟。注意该优化是对固定前缀 $x_{<t}$ 设立的；也可考虑从 1 到 $T$ 所有前缀上的耦合优化问题。

Plug-in estimator for (4). The optimal rule in (4) requires computing expectations over the ground-truth distribution $\mathbb{P}(\cdot|x_{>t})$, which is not available during inference time. A common approach in the cascades literature is to replace the expected losses with the models’ confidence estimates [22]. For example, when $\ell=\ell_{\text{0-1}}$, it may be reasonable to use $1-\max_{v}q_{t}(v)$ as an estimate of the expected 0-1 loss $\mathbb{E}_{x_{t}\sim\mathbb{P}(\cdot|x_{<t})}\left[\ell_{\text{0-1}}(x_{t},q_{t})\right]$ and $1-\max_{v}p_{t}(v)$ as an estimate of $\mathbb{E}_{x_{t}\sim\mathbb{P}(\cdot|x_{<t})}\left[\ell_{\text{0-1}}(x_{t},q_{t})\right]$. The extent to which these estimates are accurate depend on how well $q$ and $p$ are calibrated [17]. The resulting plug-in estimator for (4) is given by:

(4) 的插件估计。 最优规则 (4) 需要对真实分布 $\mathbb{P}(\cdot|x_{<t})$ 求期望，推理时不可得。级联文献的常见做法是用模型置信估计替换期望损失 [22]。例如当 $\ell=\ell_{\text{0-1}}$ 时，可用 $1-\max_{v}q_{t}(v)$ 估计期望 0-1 损失 $\mathbb{E}_{x_{t}\sim\mathbb{P}(\cdot|x_{<t})}\left[\ell_{\text{0-1}}(x_{t},q_{t})\right]$，用 $1-\max_{v}p_{t}(v)$ 估计对 $p$ 的相应期望。这些估计准不准取决于 $q$ 与 $p$ 的校准程度 [17]。由此得到 (4) 的插件估计：

$$\displaystyle\textstyle\hat{r}_{\rm\tt Diff}(x_{<t})=1~~\iff~~\max_{v}q_{t}(v)\,<\,\max_{v}p_{t}(v)-\alpha. \tag{5}$$

Similarly, when $\ell=\ell_{\log}$, we may use the entropy $-\sum_{v}q_{t}(v)\cdot\log(q_{t}(v))$ from $q_{t}$ as an estimate of its expected log-loss, and similarly for $p_{t}$ (see §C).

类似地，当 $\ell=\ell_{\log}$ 时，可用 $q_{t}$ 的熵 $-\sum_{v}q_{t}(v)\cdot\log(q_{t}(v))$ 估计其期望对数损失，对 $p_{t}$ 同理（见 §C）。

Remark 1(Oracle deferral rules).

$\hat{r}_{\rm\tt Diff}$cannot be directly used in a token-level cascade, as it needs the larger model to be invoked at every position $t$. However, it serves as an oracle that allows to analyze the head-room available to improve upon Chow’s rule. See also Remark 2.

注 1（神谕延迟规则）。

$\hat{r}_{\rm\tt Diff}$不能直接用于token 级级联，因为每个位置 $t$ 都需调用更大模型。但它可作为神谕*，用于分析相对 Chow 规则的改进空间。另见注 2。*

3.3 When do token-level cascades outperform speculative decoding?

3.3 何时 token 级级联优于投机解码？

Token-level cascades and speculative decoding differ in the distribution over tokens they seek to mimic. Speculative decoding seeks to mimic the larger model’s output distribution (or an approximation to it). As a result, the quality of the sampled output is limited by the accuracy of the larger model. On the other hand, token-level cascades seek to output distributions that closely approximate the ground-truth label distribution, and can potentially yield better quality than even the larger model.

Token 级级联与投机解码所要模仿的 token 分布不同。投机解码追求模仿更大模型的输出分布（或其近似），因而采样输出质量受更大模型准确率限制。另一方面，token 级级联追求逼近真实标签分布的输出分布，有可能给出甚至优于更大模型的质量。

Indeed, speculative decoding would be ideal in applications where the verification model $p$ is uniformly better than the draft model $q$ on all inputs. However, when there are inputs where the draft model fares better than the verifier, one may want to retain the drafter’s predictions even when it disagrees with the verifier. Similarly, in settings where both the drafter and verifier fare poorly on some inputs (e.g., due to label noise), one may again want to ignore the disagreement between the drafter and verifier, and avoid triggering unnecessary roll-backs.

确实，若验证模型 $p$ 在所有输入上一致好于起草模型 $q$，投机解码很理想。但当存在起草者优于验证者的输入时，即使二者不一致，也可能想保留起草者的预测。同样，若二者在某些输入上都表现差（例如标签噪声），也可能想忽略分歧、避免触发不必要回滚。

As a concrete example, we consider token-level cascades of T5 models [35] of two different sizes finetuned on a WMT EN $\rightarrow$ DE translation [3] and an extreme summarization (XSum) task [31]. We construct these cascades using both (token-level) Chow’s rule in (2) and the oracle Diff rule in (5), and also apply speculative decoding with the smaller (larger) model as the drafter (verifier). In Figure 1, we plot quality as a function of fraction of samples deferred to the large model, as we vary the cost parameter $\alpha$. Note that with speculative decoding, each verification step is counted as a single call to the large model. While speculative decoding matches the quality of the large model (right-most point), the oracle rule yields significantly better quality on a wide range of operating points. Even Chow’s rule, which is sub-optimal for cascading [22], outperforms speculative decoding in a small region.

作为具体例子，我们在 WMT EN$\rightarrow$DE 翻译 [3] 与极端摘要（XSum） [31] 上微调两种规模的 T5 [35]，构造 token 级级联。级联分别用 (2) 的（token 级）Chow 规则与 (5) 的神谕 Diff 规则；同时以较小（较大）模型为起草者（验证者）做投机解码。图 1 在变化代价参数 $\alpha$ 时，画出质量相对延迟到大模型样本比例的曲线。注意投机解码中每次验证计为一次大模型调用。投机解码匹配大模型质量（最右点），而神谕规则在很宽的工作点上质量显著更好；即便对级联次优的 Chow 规则 [22]，也在一小段区域超过投机解码。

However, as also evident from the plots, token-level cascades require a significantly larger number of calls to the larger model to achieve the same quality. This is because token-level cascades are executed sequentially: whenever $q$ defers, we execute $p$ once to generate one next token for the prefix accumulated so far, and the control transfers back to $q$. In contrast, speculative decoding runs $p$ in scoring mode to verify multiple draft tokens from $q$ in parallel. Moreover, the stochastic verification algorithm in speculative decoding often results in fewer tokens from $q$ getting rejected compared to the deterministic deferral rules used in a cascade. These observations motivate a natural question: given their complementary strengths, how can we leverage the best of both these techniques?

但图中同样可见：token 级级联要达到同等质量，对更大模型的调用次数显著更多。因为 token 级级联顺序执行：一旦 $q$ 延迟，就调用一次 $p$ 为当前累积前缀生成一个下一 token，再把控制交回 $q$。相对地，投机解码以打分模式跑 $p$，并行验证来自 $q$ 的多个草稿 token。此外，投机解码的随机验证算法往往比级联的确定性延迟规则拒绝更少来自 $q$ 的 token。这些观察引出自然问题：鉴于二者互补，如何兼得其长？

Figure 1: Plot of quality as a function of deferral rate for cascades constructed from T5 models (under temperature sampling with $T=1$). The left-most point represents the small model and the right-most represents the large model. We compare token-level cascades constructed with Chow’s rule (Chow) and an oracle deferral rule (Diff), and speculative decoding with block size $\gamma=5$. We measure deferral rate in terms of the number of calls to the larger model; with a cascade, each call yields exactly one token, whereas with speculative decoding, a single call scores $\gamma$ draft tokens in parallel. While speculative decoding matches the quality of the large model (see dashed horizontal line), the oracle deferral rule yields significantly better quality on a range of deferral rates; this however comes at the cost of higher number of calls to the large model.

图 1：由 T5 构造的级联在延迟率下的质量曲线（温度采样 $T=1$）。最左为小模型，最右为大模型。比较用 Chow 规则（Chow）与神谕延迟规则（Diff）构造的 token 级级联，以及块大小 $\gamma=5$ 的投机解码。延迟率按对更大模型的调用次数计；级联中每次调用恰产生一个 token，而投机解码一次调用并行打分 $\gamma$ 个草稿 token。投机解码匹配大模型质量（见虚水平线），神谕延迟规则在一系列延迟率上质量显著更好；代价是对大模型调用次数更高。

4 Speculative Cascades: Leveraging the Best of Both Worlds

4 投机级联：兼得两者之长

In addressing the above question, we present our main contribution: a principled approach to combining the superior quality of cascades with the faster execution of speculative decoding approaches.

针对上述问题，我们给出主要贡献：一条把级联的更优质量与投机解码的更快执行相结合的原则性路径。

4.1 Speculative decoding with general target distributions

4.1 带一般目标分布的投机解码

We begin by considering a generic version of speculative sampling that seeks to mimic a general target distribution derived from the drafter’s and verifier’s distributions. In the proposed sampling procedure outlined in Algorithm 4, we sample tokens auto-regressively as before from the drafter’s distribution. During the verification step, however, we do not compare the drafter’s token probabilities against the verifier’s distribution. Instead, we use a user-specified target distribution $\pi=\mathbb{T}(q,p)\in\Delta_{\mathscr{V}}$ derived from the drafter’s and verifier’s distributions at position $t$, for some function $\mathbb{T}(\cdot,\cdot)$ that is inexpensive to compute. We accept a draft token $x_{t}$ when $q(x_{t})\leq\pi(x_{t})$ and reject it otherwise with probability $1-\frac{\pi(x_{t})}{q(x_{t})}$. Upon rejection, we re-sample from the residual distribution $\operatorname{norm}\left(\max\{0,\pi(\cdot)-q(\cdot)\}\right)$.

我们先考虑投机采样的一般版本：目标是模仿由起草者与验证者分布导出的一般目标分布。算法 4 给出的采样流程中，仍从起草者分布自回归采样 token；但在验证步，不再把起草者 token 概率与验证者分布直接比较，而是使用由位置 $t$ 处起草者与验证者分布导出、用户指定的目标分布 $\pi=\mathbb{T}(q,p)\in\Delta_{\mathscr{V}}$，其中 $\mathbb{T}(\cdot,\cdot)$ 计算廉价。当 $q(x_{t})\leq\pi(x_{t})$ 时接受草稿 token $x_{t}$，否则以概率 $1-\frac{\pi(x_{t})}{q(x_{t})}$ 拒绝；拒绝后从残差分布 $\operatorname{norm}\left(\max\{0,\pi(\cdot)-q(\cdot)\}\right)$ 重采样。

This general procedure not only encompasses standard speculative decoding [26] for $\mathbb{T}(q,p)=p$, but also includes lossy speculative decoding [44] as a special case:

该一般流程不仅在 $\mathbb{T}(q,p)=p$ 时涵盖标准投机解码 [26]，也将有损投机解码 [44] 作为特例：

Lemma 2.

Algorithm 4 reduces to the lossy speculative sampling procedure in [44] with parameters $\alpha$ and $\beta$ when $\mathbb{T}(q,p)(v)=\max\{\min\{q(v),\frac{p(v)}{1-\alpha}\},\frac{p(v)}{\beta}\}$.

引理 2。

当 $\mathbb{T}(q,p)(v)=\max\{\min\{q(v),\frac{p(v)}{1-\alpha}\},\frac{p(v)}{\beta}\}$ 时，算法 4 退化为 [44] 中参数为 $\alpha$ 与 $\beta$ 的有损投机采样。

4.2 From sequential to speculative cascades

4.2 从顺序级联到投机级联

Equipped with Algorithm 4, we now propose new cascading techniques that implement their deferral rule in a speculative manner. Recall that a token-level cascade of two models $q$ and $p$ is defined by a deferral rule $r:\mathscr{V}^{t-1}\rightarrow\{0,1\}$. For a prefix $x_{<t}$, the next-token distribution at position $t$ modeled by this cascade can be written as:

有了算法 4，我们提出以投机方式实现延迟规则的新级联技术。回忆双模型 $q$ 与 $p$ 的 token 级级联由延迟规则 $r:\mathscr{V}^{t-1}\rightarrow\{0,1\}$ 定义。对前缀 $x_{<t}$，该级联在位置 $t$ 建模的下一 token 分布可写成：

$$\pi(v)=(1-r(x_{<t}))\cdot q_{t}(v)+r(x_{<t})\cdot p_{t}(v).$$

In fact, for all the deferral rules described in §2, the resulting distribution can be described by a target distribution function $\mathbb{T}_{\delta}$ of the form:

事实上，对 §2 所述全部延迟规则，所得分布都可由形如下式的目标分布函数 $\mathbb{T}_{\delta}$ 描述：

$$\mathbb{T}_{\delta}(q,p)(v)=(1-\delta(q,p))\cdot q(v)+\delta(q,p)\cdot p(v), \tag{6}$$

for some function $\delta:\Delta_{\mathscr{V}}\times\Delta_{\mathscr{V}}\rightarrow\{0,1\}$ that maps distributions $(q,p)$ to a binary decision. For example, for Chow, $\delta(q,p)=\mathbf{1}\big(\max_{v}q(v)<1-\alpha\big)$, and for Diff, $\delta(q,p)=\mathbf{1}\big(\max_{v}q(v)<\max_{v}p(v)-\alpha\big).$ See Table 1 for a summary of target distributions for different deferral rules.

其中某函数 $\delta:\Delta_{\mathscr{V}}\times\Delta_{\mathscr{V}}\rightarrow\{0,1\}$ 把分布对 $(q,p)$ 映到二元决策。例如对 Chow，$\delta(q,p)=\mathbf{1}\big(\max_{v}q(v)<1-\alpha\big)$；对 Diff，$\delta(q,p)=\mathbf{1}\big(\max_{v}q(v)<\max_{v}p(v)-\alpha\big).$ 不同延迟规则的目标分布汇总见表 1。

Our proposal is to then invoke the speculative sampling procedure in Algorithm 4 with $\mathbb{T}_{\delta}$ as the target distribution function. We outline this generic speculative cascading approach in Algorithm 5, and contrast it with the sequential execution of a deferral rule in Algorithm 2.

我们的提案是：以 $\mathbb{T}_{\delta}$ 为目标分布函数调用算法 4 的投机采样。算法 5 概述该通用投机级联方法，并与算法 2 中延迟规则的顺序执行对照。

Remark 2(Exact implementation of oracle deferral ruleDiff).

Unlike a sequential cascade, where the larger model’s distribution $p$ is not available at the time the deferral decision is made (see Remark 1), with a speculative cascade, we can accommodate deferral rules likeDiffthat depend on both $q$ and $p$. This is because we run the larger model $p$ in parallel on drafts generated by the smaller model $q$, allowing us to compute both $p(\cdot)$ and $q(\cdot)$ on every prefix.

注 2（神谕延迟规则 Diff 的精确实现）。

So far we have considered deferral rules designed for sequential cascades. In what follows, we derive the optimal deferral rule $r$ for a speculative cascade, where we sample speculatively from a target distribution $\pi=(1-r(x_{<t}))\cdot q_{t}+r(x_{<t})\cdot p_{t}$ using $q_{t}$ as the drafter.

有趣的是，与顺序执行不同，投机执行天然允许实现像 Diff 这样的神谕延迟规则：验证阶段反正要并行评估起草者与验证者分布。

Models $q$, $p$, Prefix $x_{<t}$, Block size $\gamma$
$\mathbb{T}(q,p)\stackrel{{\scriptstyle.}}{{=}}p$
$\text{\tt{GenSpecSample}}(q,p,\mathbb{T},x_{<t},\gamma)$

Models $q$, $p$, Deferral logic ${\delta}$, Prefix $x_{<t}$
$q_{t}(\cdot)\stackrel{{\scriptstyle.}}{{=}}q(\cdot|x_{<t})$
if ${\delta}(q_{t},\emptyset)=0$ then
Sample $x_{t}\sim q_{t}(\cdot)$
else
$p_{t}(\cdot)\stackrel{{\scriptstyle.}}{{=}}p(\cdot|x_{<t})$;   Sample $x_{t}\sim p_{t}(\cdot)$
end if
$x_{t}$

Models $q$, $p$, Deferral logic ${\delta}$, Prefix $x_{<t}$
$q_{t}(\cdot)\stackrel{{\scriptstyle.}}{{=}}q(\cdot|x_{<t})$;    $p_{t}(\cdot)\stackrel{{\scriptstyle.}}{{=}}p(\cdot|x_{<t})$
if ${\delta}(q_{t},p_{t})=0$ then
Sample $x_{t}\sim q_{t}(\cdot)$
else
Sample $x_{t}\sim p_{t}(\cdot)$
end if
$x_{t}$

Models $q$, $p$, Target distr. $\mathbb{T}$, Prefix $x_{<t}$, Block size $\gamma$
$[\gamma]\equiv\{0,\ldots,\gamma\}$
Sample $\gamma$ tokens auto-regressively from $q$
for $j=0$ to $\gamma-1$ do
$q_{t+j}(\cdot)\stackrel{{\scriptstyle.}}{{=}}q(\cdot|x_{<t+j});~~~~$ $x_{t+j}\sim q_{t+j}(\cdot)$
end for
Run $p$ in parallel to score $\gamma$ draft tokens
$p_{t+j}(\cdot)\stackrel{{\scriptstyle.}}{{=}}p(\cdot|x_{<t+j}),~\forall j\in[\gamma]$
$\pi_{t+j}=\mathbb{T}(q_{t+j},p_{t+j})$
Find the earliest draft token that gets rejected
$a_{j}\sim\text{Ber}\left(\min\left\{1,\frac{\pi_{t+j}({x_{t+j}})}{q_{t+j}({x_{t+j}})}\right\}\right),~\forall j\in[\gamma-1];$    $a_{\gamma}=0$
$j^{*}=\min\{j\in[\gamma]\,:\,a_{j}=0\}$
Sample a new token from residual distribution
$p_{\rm res}(\cdot)=\begin{cases}\operatorname{norm}\hskip-1.0pt\left(\max\left\{0,\,\pi_{t+j^{*}}(\cdot)-q_{t+j^{*}}(\cdot)\right\}\right)&\text{if}~j^{*}<\gamma\\ p_{t+\gamma}(\cdot)&\text{else}\end{cases}$
Sample $x_{t+j^{*}}\sim p_{\rm res}(\cdot)$
$x_{t},\ldots,x_{t+j^{*}}$

Models $q$, $p$, Deferral logic $\delta$, Prefix $x_{<t}$, Block size $\gamma$
$\mathbb{T}_{\delta}(q,p)\stackrel{{\scriptstyle.}}{{=}}(1-\delta(q,p))\cdot q+\delta(q,p)\cdot p$
$\text{\tt{GenSpecSample}}(q,p,\mathbb{T}_{\delta},x_{<t},\gamma)$

Algorithm 1 SpecDecode

图示/表：投机级联与相关方法的算法对照（见原文算法 2–5）。顺序级联每延迟一次只前进一个 token；投机级联在草稿块上并行做延迟决策，并按目标分布 $\pi=\mathbb{T}_{\delta}(q,p)$ 做接受/残差重采样。

4.3 Deferral risk for speculative cascades

4.3 投机级联的延迟风险

As with sequential cascades (§2), we begin by defining an objective to minimize. We seek a deferral rule $r:\mathscr{V}^{t-1}\rightarrow\{0,1\}$ that minimizes a loss against the ground-truth distribution, while limiting the inference cost to be within a budget. (Per above, this deferral rule implicitly defines a target distribution $\pi$.) The inference cost crucially depends on how frequently a draft token is rejected in the verification phase, triggering a rollback. To this end, we derive the probability that a token sampled from $q$ is rejected during verification, for a target distribution resulting from a deferral rule $r$.

与顺序级联（§2）一样，我们先定义要最小化的目标。由于验证者以投机方式运行，拒绝草稿 token 的代价不仅来自调用更大模型，还来自回滚。事实上，每当草稿被拒绝，我们就从残差分布采样替换 token，并重启投机过程。因此，延迟风险应同时反映推理质量与草稿拒绝率。

Lemma 3.

For a given prefix $x_{<t}$, and target distribution $\pi=(1-r(x_{<t}))\cdot q_{t}+r(x_{<t})\cdot p_{t}$, the probability of a token drawn from draft distribution $q_{t}$ being rejected is equal to:

引理 3。

对给定前缀 $x_{<t}$，以及形如 $\pi_{t}=(1-r(x_{<t}))\cdot q_{t}+r(x_{<t})\cdot p_{t}$ 的目标分布，拒绝草稿 token 的概率为：

$$r(x_{<t})\cdot D_{\textup{{TV}}}(p_{t},q_{t}),$$

where $D_{\textup{{TV}}}(p,q)=\sum_{v\in\mathscr{V}}\max\{0,p(v)-q(v)\}$ is the TV distance between $p$ and $q$.

其中 $D_{\textup{{TV}}}(p,q)=\sum_{v\in\mathscr{V}}\max\{0,p(v)-q(v)\}$ 为全变差距离。

Intuitively, whenever $r(x_{<t})=0$, $\pi(v)=q_{t}(v)$ and therefore there is no rejection or roll-back; when $r(x_{<t})=1$, the rejection rate equals $D_{\textup{{TV}}}(p_{t},q_{t})$.

直观上，每当 $r(x_{<t})=0$，$\pi(v)=q_{t}(v)$，因而拒绝概率为 0；每当 $r(x_{<t})=1$，拒绝概率等于 $q_{t}$ 与 $p_{t}$ 之间的全变差距离。

For a fixed prefix $x_{<t}$, we formulate the goal of finding a solution to:

对固定前缀 $x_{<t}$，我们把目标表述为在延迟预算约束下最小化期望损失：

$$\displaystyle\min_{r} \\ \displaystyle~~\mathbb{E}_{x_{t}\sim\mathbb{P}(\cdot|x_{<t})}\Big[\big(1-r(x_{<t})\big)\cdot\ell(x_{t},q_{t})+r(x_{<t})\cdot\ell(x_{t},p_{t})\big)\Big] \\ \displaystyle\hskip 28.45274pt\text{s.t.}~~r(x_{<t})\cdot D_{\textrm{TV}}(p_{t},q_{t})\,\leq\,B, \tag{7}$$

for some budget $B>0$. Equivalently, one may minimize an unconstrained objective similar to (3), for suitable cost parameter $\alpha>0$ (see §C.4):

对某预算 $B>0$。等价地，可最小化带拉格朗日乘子 $\alpha\geq 0$ 的无约束目标：

$$\displaystyle\hskip-10.0ptL_{\rm spec}(r;x_{<t}) \\ \displaystyle=\mathbb{E}_{x_{t}\sim\mathbb{P}(\cdot|x_{<t})}\Big[\big(1-r(x_{<t})\big)\cdot\ell(x_{t},q_{t})+r(x_{<t})\cdot\big(\ell(x_{t},p_{t})+\alpha\cdot D_{\textrm{TV}}(p_{t},q_{t})\big)\Big],\hskip-5.0pt \tag{8}$$

Contrasting (8) with the deferral risk in (3) for a sequential cascade, a key difference is that the cost of deferring to the larger model is no longer a constant, but depends on the similarity between $q_{t}$ and $p_{t}$, as measured by the TV distance between them.

把 (8) 与顺序级联的延迟风险 (3) 对照，额外的 $D_{\textrm{TV}}(p_{t},q_{t})$ 项显式惩罚拒绝草稿 token 的代价。当 $q_{t}$ 与 $p_{t}$ 接近时该项较小，鼓励在分歧不大时仍延迟到 $p$；当二者差异大时惩罚更大，更倾向于信任 $q$。