Stealing Part of a Production Language Model

窃取生产级语言模型的一部分

Nicholas Carlini Affiliation:  Google DeepMind Correspondence to: nicholas@carlini.com    Daniel Paleka Affiliation:  ETH Zurich    Krishnamurthy (Dj) Dvijotham Affiliation:  Google DeepMind    Thomas Steinke Affiliation:  Google DeepMind    Jonathan Hayase Affiliation:  University of Washington    A. Feder Cooper Affiliation:  Google DeepMind    Katherine Lee Affiliation:  Google DeepMind    Matthew Jagielski Affiliation:  Google DeepMind    Milad Nasr Affiliation:  Google DeepMind    Arthur Conmy Affiliation:  Google DeepMind    Itay Yona Affiliation:  Google DeepMind    Eric Wallace Affiliation:  OpenAI    David Rolnick Affiliation:  McGill University    Florian Tramèr Affiliation:  ETH Zurich

Abstract

摘要

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI’s ChatGPT or Google’s PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under $20 USD, our attack extracts the entire projection matrix of OpenAI’s ada and babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under$2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

我们给出首个能从 OpenAI 的 ChatGPT、Google 的 PaLM-2 这类黑盒生产级语言模型中，抽出精确且非平凡信息的模型窃取攻击。具体而言，在常见的 API 访问下，攻击可恢复 transformer 的嵌入投影层（至多差对称性）。花费不到 20 美元，即可抽出 OpenAI 的 ada 与 babbage 语言模型的整张投影矩阵，从而首次确认这两个黑盒模型的隐藏维分别是 1024 与 2048。我们还恢复了 gpt-3.5-turbo 的确切隐藏维大小，并估计抽完整张投影矩阵的查询费用不到 2000 美元。文末讨论可能的防御与缓解，以及把攻击继续扩展的后续工作含义。

1 Introduction

1 引言

Keywords:

Little is publicly known about the inner workings of today’s most popular large language models, such as GPT-4, Claude 2, or Gemini. The GPT-4 technical report states it “contains no […] details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar” (OpenAI et al., 2023). Similarly, the PaLM-2 paper states that “details of [the] model size and architecture are withheld from external publication” (Anil et al., 2023). This secrecy is often ascribed to “the competitive landscape” (because these models are expensive to train) and the “safety implications of large-scale models” (OpenAI et al., 2023) (because it is easier to attack models when more information is available). Nevertheless, while these models’ weights and internal details are not publicly accessible, the models themselves are exposed via APIs.

关键词：

今天最常用的大语言模型——例如 GPT-4、Claude 2 或 Gemini——其内部机制几乎不对外公开。GPT-4 技术报告写明「不含……架构（含模型规模）、硬件、训练算力、数据集构造、训练方法或类似细节」（OpenAI et al., 2023）。PaLM-2 论文同样写「模型规模与架构细节不予对外发表」（Anil et al., 2023）。保密常被归因于「竞争格局」（训练昂贵）以及「大规模模型的安全影响」（OpenAI et al., 2023）：知道得越多，越容易攻击。尽管权重与内部细节不公开，模型本身却通过 API 对外提供。

In this paper we ask: how much information can an adversary learn about a production language model by making queries to its API? This is the question studied by the field of model stealing (Tramèr et al., 2016): the ability of an adversary to extract model weights by making queries its API.

本文问的是：对手通过对生产级语言模型的 API 发查询，究竟能学到多少信息？ 这正是模型窃取（Tramèr et al., 2016）所研究的问题：对手能否靠查询 API 抽出模型权重。

Contributions. We introduce an attack that can be applied to black-box language models, and allows us to recover the complete embedding projection layer of a transformer language model. Our attack departs from prior approaches that reconstruct a model in a bottom-up fashion, starting from the input layer. Instead, our attack operates top-down and directly extracts the model’s last layer. Specifically, we exploit the fact that the final layer of a language model projects from the hidden dimension to a (higher dimensional) logit vector. This final layer is thus low-rank, and by making targeted queries to a model’s API, we can extract its embedding dimension or its final weight matrix.

贡献。 我们提出一种可用于黑盒语言模型的攻击，能恢复 transformer 语言模型完整的嵌入投影层。与自底向上、从输入层起重构模型的既有路线不同，本攻击自上而下，直接抽取模型最后一层。利用的是：语言模型末层把隐藏维投影到（更高维的）logit 向量，因而末层低秩；通过对 API 做有针对性的查询，即可抽出嵌入维或最终权重矩阵。

Stealing this layer is useful for several reasons. First, it reveals the width of the transformer model, which is often correlated with its total parameter count. Second, it slightly reduces the degree to which the model is a complete “black-box”, which so might be useful for future attacks. Third, while our attack recovers only a (relatively small) part of the entire model, the fact that it is at all possible to steal any parameters of a production model is surprising, and raises concerns that extensions of this attack might be able to recover more information. Finally, recovering the model’s last layer (and thus hidden dimension) may reveal more global information about the model, such as relative size differences between different models.

窃取这一层有几方面用途。第一，它揭示 transformer 的宽度，往往与总参数量相关。第二，略微削弱模型作为完全「黑盒」的程度，或可服务后续攻击。第三，虽只恢复整模中相对较小的一块，但生产模型的任意参数竟可被窃取，本身就令人惊讶，也令人担心攻击扩展后能拿到更多信息。第四，恢复末层（从而知道隐藏维）可能暴露更全局的信息，例如不同模型之间的相对规模差。

Our attack is effective and efficient, and is applicable to production models whose APIs expose full logprobs, or a “logit bias”. This included Google’s PaLM-2 and OpenAI’s GPT-4 (Anil et al., 2023; OpenAI et al., 2023); after responsible disclosure, both APIs have implemented defenses to prevent our attack or make it more expensive. We extract the embedding layer of several OpenAI models with a mean squared error of $10^{-4}$ (up to unavoidable symmetries). We apply a limited form of our attack to gpt-3.5 at a cost of under $200 USD and, instead of recovering the full embedding layer, recover just the size of the embedding dimension.

攻击有效且高效，适用于暴露完整 logprobs 或「logit bias」的生产模型 API，包括 Google 的 PaLM-2 与 OpenAI 的 GPT-4（Anil et al., 2023；OpenAI et al., 2023）。负责任披露后，两家 API 都已加入防御，阻止本攻击或抬高其成本。我们抽出若干 OpenAI 模型的嵌入层，均方误差约 $10^{-4}$（在不可避免的对称意义下）。对 gpt-3.5 只跑有限形式攻击，花费不到 200 美元，不抽完整嵌入层，只恢复嵌入维大小。

Responsible disclosure. We shared our attack with all services we are aware of that are vulnerable to this attack. We also shared our attack with several other popular services, even if they were not vulnerable to our specific attack, because variants of our attack may be possible in other settings. We received approval from OpenAI prior to extracting the parameters of the last layers of their models, worked with OpenAI to confirm our approach’s efficacy, and then deleted all data associated with the attack. In response to our attack, OpenAI and Google have both modified their APIs to introduce mitigiations and defenses (like those that we suggest in Section 8) to make it more difficult for adversaries to perform this attack.

负责任披露。 凡我们已知易受本攻击的服务，均已分享攻击细节；对若干虽不直接受本攻击、但变体可能成立的热门服务，也做了通报。抽出 OpenAI 模型末层参数前已获其批准，并与对方确认方法有效，随后删除攻击相关全部数据。作为回应，OpenAI 与 Google 均已修改 API，引入缓解与防御（如第 8 节所建议），增加对手实施该攻击的难度。

2 Related Work

2 相关工作

Model stealing attacks (Tramèr et al., 2016) aim to recover the functionality of a black-box model, and optimize for one of two objectives (Jagielski et al., 2020):

模型窃取攻击（Tramèr et al., 2016）意在恢复黑盒模型的功能，优化目标通常二选一（Jagielski et al., 2020）：

1. Accuracy: the stolen model $\hat{f}$ should match the performance of the target model $f$ on some particular data domain. For example, if the target is an image classifier, we might want the stolen model to match the target’s overall accuracy on ImageNet.
2. Fidelity: the stolen model $\hat{f}$ should be functionally equivalent to the target model $f$ on all inputs. That is, for any valid input $p$, we want $\hat{f}(p)\approx f(p)$.

1. 准确率：窃取模型 $\hat{f}$ 应在特定数据域上匹配目标模型 $f$ 的表现。例如目标是图像分类器时，可能希望窃取模型在 ImageNet 上的整体准确率与目标一致。
2. 保真度：窃取模型 $\hat{f}$ 应在所有输入上与目标 $f$ 功能等价。即对任意合法输入 $p$，希望 $\hat{f}(p)\approx f(p)$。

In this paper, we focus on high-fidelity attacks. Most prior high-fidelity attacks exploit specific properties of deep neural networks with ReLU activations. Milli et al. (2019) first showed that if an attacker can compute gradients of a target two-layer ReLU model, then they can steal a nearly bit-for-bit equivalent model. Jagielski et al. (2020) observed that if the attacker only has query access to model outputs, they can approximate gradients with finite differences. Subsequent work extended these attacks to efficiently extract deeper ReLU models (Carlini et al., 2020; Rolnick & Kording, 2020; Shamir et al., 2023). Unfortunately, none of these approaches scale to production language models, because they (1) accept tokens as inputs (and so performing finite differences is intractable); (2) use activations other than ReLUs; (3) contain architectural components such as attention, layer normalization, residual connections, etc. that current attacks cannot handle; (4) are orders-of-magnitude larger than prior extracted models; and (5) expose only limited-precision outputs.

本文聚焦高保真攻击。多数既有高保真攻击利用带 ReLU 激活的深度网络性质。Milli et al.（2019）首先指出：若攻击者能计算目标两层 ReLU 模型的梯度，即可窃取近乎逐比特等价的模型。Jagielski et al.（2020）观察到：若只有对模型输出的查询，可用有限差分近似梯度。后续工作把攻击扩展到高效抽取更深的 ReLU 模型（Carlini et al., 2020；Rolnick & Kording, 2020；Shamir et al., 2023）。可惜这些方法都难扩展到生产语言模型，因为它们（1）输入是 token（有限差分不可行）；（2）激活不是 ReLU；（3）含注意力、层归一化、残差等现有攻击处理不了的组件；（4）比以往抽出的模型大几个数量级；（5）只暴露有限精度输出。

Other attacks aim to recover more limited information, or assume a stronger adversary. Wei et al. (2020) show that an adversary co-located on the same server as the LLM can recover the sizes of all hidden layers. Zanella-Beguelin et al. (2021) assume a model with a public pretrained encoder and a private final layer, and extract the final layer; our Section 4.2 is quite similar to their method. Others have attempted to recover model sizes by correlating performance on published benchmarks with model sizes in academic papers (Gao, 2021).

另一些攻击只恢复有限信息，或假设更强对手。Wei et al.（2020）表明：与 LLM 同机共址的对手可恢复全部隐藏层大小。Zanella-Beguelin et al.（2021）假设公开预训练编码器加私有末层，并抽取末层；我们第 4.2 节与其方法相当接近。也有人试图用公开基准表现与学术论文中的模型规模做相关，从而估计规模（Gao, 2021）。

3 Problem Formulation

3 问题形式化

We study models that take a sequence of tokens drawn from a vocabulary $\mathcal{X}$ as input. Let $\mathcal{P}\left({\mathcal{X}}\right)$ denote the space of probability distributions over $\mathcal{X}$. We study parameterized models $f_{\theta}:\mathcal{X}^{N}\to\mathcal{P}\left({\mathcal{X}}\right)$ that produce a probability distribution over the next output token, given an input sequence of $N$ tokens. The model has the following structure:

我们研究的模型以词表 $\mathcal{X}$ 上的 token 序列为输入。令 $\mathcal{P}\left({\mathcal{X}}\right)$ 表示 $\mathcal{X}$ 上的概率分布空间。参数化模型 $f_{\theta}:\mathcal{X}^{N}\to\mathcal{P}\left({\mathcal{X}}\right)$ 在给定 $N$ 个 token 的输入序列后，输出下一 token 的概率分布。模型结构如下：

$$\displaystyle f_{\theta}(p)=\mathsf{softmax}(\mathbf{W}\cdot g_{\theta}(p)), \tag{1}$$

where $g_{\theta}:\mathcal{X}^{N}\to\mathbb{R}^{h}$ is another parameterized model that computes hidden states, $\mathbf{W}$ is an $l\times h$ dimensional matrix (the embedding projection matrix), and $\mathsf{softmax}:\mathbb{R}^{l}\to[0,1]^{l}$ is the softmax function applied to the resulting logits:

其中 $g_{\theta}:\mathcal{X}^{N}\to\mathbb{R}^{h}$ 是计算隐藏状态的另一参数化模型，$\mathbf{W}$ 是 $l\times h$ 维矩阵（嵌入投影矩阵），$\mathsf{softmax}:\mathbb{R}^{l}\to[0,1]^{l}$ 作用于所得logits：

$$\mathsf{softmax}(\mathbf{z})=\left[\frac{e^{\mathbf{z}_{1}}}{\sum_{i=1}^{l}e^{\mathbf{z}_{i}}},\dots,\frac{e^{\mathbf{z}_{l}}}{\sum_{i=1}^{l}e^{\mathbf{z}_{i}}}\right]\;.$$

Note that the hidden dimension size is much smaller than the size of the token dictionary, i.e., $h\ll l$. For example, LLaMA (Touvron et al., 2023) chooses $h\in\{4096,5120,6656,8192\}$ and $l=32{,}000$, and there is a recent trend towards increasingly large token sizes; GPT-4, for example, has a $\approx$100,000 token vocabulary.

注意隐藏维远小于词表大小，即 $h\ll l$。例如 LLaMA（Touvron et al., 2023）取 $h\in\{4096,5120,6656,8192\}$、$l=32{,}000$；词表还在继续变大，例如 GPT-4 词表约 10 万。

Threat model.  Throughout the paper, we assume that the adversary does not have any additional knowledge about the model parameters. We assume access to a model $f_{\theta}$, hosted by a service provider and made available to users through a query interface (API) $\mathcal{O}$. We assume that $\mathcal{O}$ is a perfect oracle: given an input sequence $p$, it produces $y=\mathcal{O}\left({p}\right)$ without leaking any other information about $f_{\theta}$ than what can be inferred from $(p,y)$. For example, the adversary cannot infer anything about $f_{\theta}$ via timing side-channels or other details of the implementation of the query interface.

威胁模型。 全文假设对手对模型参数无额外先验知识。对手可访问服务方托管、经查询接口（API）$\mathcal{O}$ 暴露的模型 $f_{\theta}$。假设 $\mathcal{O}$ 是完美预言机：给定输入序列 $p$，输出 $y=\mathcal{O}\left({p}\right)$，除可由 $(p,y)$ 推断者外，不泄露 $f_{\theta}$ 的其他信息。例如对手不能靠计时侧信道或查询接口实现细节推断 $f_{\theta}$。

Different open-source and proprietary LLMs offer APIs with varying capabilities, which impact the ability to perform model extraction attacks and the choice of attack algorithm. A summary of the different APIs we study, and our motivation for doing so, is presented in Table 1. The logits API is a strawman threat model where the API provides logits for all tokens in the response to a given prompt. We begin with this toy setting, as the attack techniques we develop here can be reused in subsequent sections, where we will first reconstruct the logits from more limited information (e.g., log-probabilities for only the top few tokens) and then run the attack.

开源与专有 LLM 的 API 能力各异，会影响能否做模型抽取以及选用何种算法。我们研究的各类 API 及动机见表 1。Logits API 是教学用威胁模型：对给定 prompt，API 返回响应中所有 token 的 logits。我们从该玩具设定入手，因为这里发展的技术可复用到后续章节——先从更受限信息（例如仅 top 若干 token 的对数概率）重构 logits，再跑攻击。

API	Motivation
All Logits §4	Pedagogy & basis for next attacks
Top Logprobs, Logit-bias §5	Current LLM APIs (e.g., OpenAI)
No logprobs, Logit-bias §F	Potential future constrained APIs

Table 1: Summary of APIs

API	动机
全部 Logits §4	教学，并为后续攻击奠基
Top Logprobs、Logit-bias §5	当前 LLM API（如 OpenAI）
无 logprobs、有 Logit-bias §F	未来可能更受限的 API

表 1：API 概览

4 Extraction Attack for Logit-Vector APIs

4 面向 Logit 向量 API 的抽取攻击

In this section, we assume the adversary can directly view the logits that feed into the softmax function for every token in the vocabulary (we will later relax this assumption), i.e.,

本节假设对手可直接看到进入 softmax 的、词表中每个 token 的 logits（稍后放松该假设），即

$$\mathcal{O}\left({p}\right)\leftarrow\mathbf{W}\cdot g_{\theta}\left({p}\right)\;.$$

We develop new attack techniques that allow us to perform high-fidelity extraction of (a small part of) a transformer. Section 4.1 demonstrates how we can identify the hidden dimension $h$ using the logits API and Section 4.2 presents an algorithm that can recover the matrix $\mathbf{W}$.

我们发展新的攻击技术，对 transformer（的一小部分）做高保真抽取。第 4.1 节说明如何用 logits API 识别隐藏维 $h$；第 4.2 节给出恢复矩阵 $\mathbf{W}$ 的算法。

4.1 Warm-up: Recovering Hidden Dimensionality

4.1 热身：恢复隐藏维

We begin with a simple attack that allows an adversary to recover the size of the hidden dimension of a language model by making queries to the oracle $\mathcal{O}$ (Algorithm 1). The techniques we use to perform this attack will be the foundation for attacks that we further develop to perform complete extraction of the final embedding projection matrix.

先给出简单攻击：对手通过对预言机 $\mathcal{O}$ 发查询即可恢复语言模型隐藏维大小（算法 1）。完成该攻击所用技术，也是后续完整抽取末层嵌入投影矩阵的基础。

1: Oracle LLM $\mathcal{O}$ returning logits
2: Initialize $n$ to an appropriate value greater than $h$
3: Initialize an empty matrix $\mathbf{Q}=\mathbf{0}^{n\times l}$
4: for $i=1$ to $n$ do
5:   $p_{i}\leftarrow\texttt{RandPrefix}()$ $\triangleright$ Choose a random prompt
6:    $\mathbf{Q}_{i}\leftarrow\mathcal{O}(p_{i})$
7: end for
8: $\lambda_{1}\geq\lambda_{2}\geq\dots\geq\lambda_{n}\leftarrow\text{SingularValues}(\mathbf{Q})$
9: $\text{count}\leftarrow\argmax_{i}{\log\lVert\lambda_{i}\rVert-\log\lVert\lambda_{i+1}\rVert}$
10: return count

Algorithm 1 Hidden-Dimension Extraction Attack

1: 返回 logits 的预言机 LLM $\mathcal{O}$
2: 将 $n$ 初始化为大于 $h$ 的合适值
3: 初始化空矩阵 $\mathbf{Q}=\mathbf{0}^{n\times l}$
4: for $i=1$ to $n$ do
5:   $p_{i}\leftarrow\texttt{RandPrefix}()$ $\triangleright$ 选随机 prompt
6:    $\mathbf{Q}_{i}\leftarrow\mathcal{O}(p_{i})$
7: end for
8: $\lambda_{1}\geq\lambda_{2}\geq\dots\geq\lambda_{n}\leftarrow\text{SingularValues}(\mathbf{Q})$
9: $\text{count}\leftarrow\argmax_{i}{\log\lVert\lambda_{i}\rVert-\log\lVert\lambda_{i+1}\rVert}$
10: return count

算法 1 隐藏维抽取攻击

Intuition.  Suppose we query a language model on a large number of different random prefixes. Even though each output logit vector is an $l$-dimensional vector, they all actually lie in a $h$-dimensional subspace because the embedding projection layer up-projects from $h$-dimensions. Therefore, by querying the model “enough” (more than $h$ times) we will eventually observe new queries are linearly dependent of past queries. We can then compute the dimensionality of this subspace (e.g., with SVD) and report this as the hidden dimensionality of the model.

直觉。 假设用大量不同随机前缀查询语言模型。每个输出 logit 向量虽是 $l$ 维，却都落在 $h$ 维子空间里，因为嵌入投影层从 $h$ 维上投影。因此「够多」次查询（多于 $h$ 次）之后，新查询会对过去查询线性相关。可用 SVD 等算出该子空间维数，并报告为模型隐藏维。

Formalization.  The attack is based on the following straightforward mathematical result:

形式化。 攻击基于如下直接的数学结论：

Lemma 4.1.

Let $\mathbf{Q}\left({p_{1},\ldots p_{n}}\right)\in\mathbb{R}^{l\times n}$ denote the matrix with columns $\mathcal{O}\left({p_{1}}\right),\ldots,\mathcal{O}\left({p_{n}}\right)$ of query responses from the logit-vector API. Then

引理 4.1。

令 $\mathbf{Q}\left({p_{1},\ldots p_{n}}\right)\in\mathbb{R}^{l\times n}$ 表示以 logit 向量 API 的查询响应 $\mathcal{O}\left({p_{1}}\right),\ldots,\mathcal{O}\left({p_{n}}\right)$ 为列的矩阵。则

$$h\geq\text{rank}\left({\mathbf{Q}\left({p_{1},\ldots p_{n}}\right)}\right).$$

Further, if the matrix with columns $g_{\theta}\left({p_{i}}\right)$ ($i=1,...,n$) has rank $h$ and $\mathbf{W}$ has rank $h$, then

进而，若以 $g_{\theta}\left({p_{i}}\right)$（$i=1,...,n$）为列的矩阵秩为 $h$，且 $\mathbf{W}$ 秩为 $h$，则

$$h=\text{rank}\left({\mathbf{Q}\left({p_{1},\ldots p_{n}}\right)}\right).$$

Proof.

We have $\mathbf{Q}=\mathbf{W}\cdot\mathbf{H}$, where $\mathbf{H}$ is a $h\times n$ matrix whose columns are $g_{\theta}(p_{i})$ ($i=1,\ldots,n$). Thus, $h\geq\textrm{rank}\left({\mathbf{Q}}\right)$. Further, if $\mathbf{H}$ has rank $h$ (with the second assumption), then $h=\textrm{rank}\left({\mathbf{Q}}\right)$. ∎

证明。

有 $\mathbf{Q}=\mathbf{W}\cdot\mathbf{H}$，其中 $\mathbf{H}$ 是 $h\times n$ 矩阵，列是 $g_{\theta}(p_{i})$（$i=1,\ldots,n$）。故 $h\geq\textrm{rank}\left({\mathbf{Q}}\right)$。进而若 $\mathbf{H}$ 秩为 $h$（配合第二假设），则 $h=\textrm{rank}\left({\mathbf{Q}}\right)$。∎

Assumptions.  In Lemma 4.1, we assume that both the matrix with columns $g_{\theta}\left({p_{i}}\right)$ and the matrix $\mathbf{W}$ have rank $h$. These matrices have either $h$ rows or $h$ columns, so both have rank at most $h$. Moreover, it is very unlikely that they have rank $<h$: this would require the distribution of $g_{\theta}\left({p}\right)$ to be fully supported on a subspace of dimension $<h$ across all $p_{i}$ we query, or all $h\ll l$ columns of $\mathbf{W}$ to lie in the same $(h-1)$ dimensional subspace of $\mathbb{R}^{l}$ (the output space of logits). In practice we find this assumption holds for all larger models (Table 2) and when different normalization layers are used (Section B.1).

假设。 引理 4.1 假设以 $g_{\theta}\left({p_{i}}\right)$ 为列的矩阵与 $\mathbf{W}$ 秩均为 $h$。这些矩阵只有 $h$ 行或 $h$ 列，故秩至多为 $h$。秩 $<h$ 极不可能：那会要求对所有查询的 $p_i$，$g_{\theta}\left({p}\right)$ 的分布完全支撑在维数 $<h$ 的子空间上，或 $\mathbf{W}$ 全部 $h\ll l$ 列落在 $\mathbb{R}^{l}$（logits 输出空间）同一 $(h-1)$ 维子空间。实践中，对更大模型（表 2）以及使用不同归一化层时（附录 B.1），该假设都成立。

Practical considerations.  Since the matrix $\mathbf{Q}$ is not computed over the reals, but over floating-point numbers (possibly with precision as low as 16-bits or 8-bits for production neural networks), we cannot naively take the rank to be the number of linearly independent rows. Instead, we use a practical numerical rank of $\mathbf{Q}$, where we order the singular values $\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{n}$, and identify the largest multiplicative gap $\frac{\lambda_{i}}{\lambda_{i+1}}$ between consecutive singular values. A large multiplicative gap arises when we switch from large “actual” singular values to small singular values that arise from numerical imprecision. Figure 2 shows these gaps. Algorithm 1 describes this attack.

实践考量。 $\mathbf{Q}$ 并非在实数上精确计算，而是浮点（生产网络精度可能低至 16 位或 8 位），不能朴素地把秩取为线性无关行数。我们改用 $\mathbf{Q}$ 的实用数值秩：将奇异值排序 $\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{n}$，找连续奇异值之间最大的乘性间隙 $\frac{\lambda_{i}}{\lambda_{i+1}}$。从较大的「真实」奇异值切到数值误差造成的小奇异值时，会出现大间隙。图 2 展示这些间隙。算法 1 描述该攻击。

Figure 1: SVD can recover the hidden dimensionality of a model when the final output layer dimension is greater than the hidden dimension. Here we extract the hidden dimension (2048) of the Pythia 1.4B model. We can precisely identify the size by obtaining slightly over 2048 full logit vectors.

图 1：当末层输出维大于隐藏维时，SVD 可恢复模型隐藏维。这里抽出 Pythia 1.4B 的隐藏维（2048）。略多于 2048 个完整 logit 向量即可精确判定规模。

Experiments. In order to visualize the intuition behind this attack, Figure 1 illustrates an attack against the Pythia-1.4b LLM. Here, we plot the magnitude of the singular values of $\mathbf{Q}$ as we send an increasing number $n$ of queries to the model. When we send fewer than 2048 queries it is impossible to identify the dimensionality of the hidden space. This is because $n<h$, and so the $n\times l$ dimensional matrix $\mathbf{Q}$ has full rank and $n$ nontrivial singular values. But once we make more than $2048$ queries to the model, and thus $n>h$, the number of numerically significant singular values does not increase further; it is capped at exactly $2048$.

实验。 为可视化攻击直觉，图 1 展示对 Pythia-1.4b LLM 的攻击。我们画出随查询数 $n$ 增加时 $\mathbf{Q}$ 奇异值幅度。查询少于 2048 时无法识别隐藏空间维数：因 $n<h$，$n\times l$ 的 $\mathbf{Q}$ 满秩且有 $n$ 个非平凡奇异值。一旦查询超过 2048、$n>h$，数值显著的奇异值个数不再增加，恰好封顶在 2048。

In Figure 2 we plot the difference (in log-space) between subsequent singular values. As we can see, the largest difference occurs at (almost exactly) the 2048th singular value—the true hidden dimensionality of this model.

图 2 画出相邻奇异值（对数空间）之差。可见最大差距几乎正好出现在第 2048 个奇异值——该模型真实隐藏维。

We now analyze the efficacy of this attack across a wider range of models: GPT-2 (Radford et al., 2019) Small and XL, Pythia (Biderman et al., 2023) 1.4B and 6.9B, and LLaMA (Touvron et al., 2023) 7B and 65B. The results are in Table 2: our attack recovers the embedding size nearly perfectly, with an error of 0 or 1 in five out of six cases.

再在更广模型上检验：GPT-2（Radford et al., 2019）Small 与 XL、Pythia（Biderman et al., 2023）1.4B 与 6.9B、LLaMA（Touvron et al., 2023）7B 与 65B。结果见表 2：攻击几乎完美恢复嵌入规模，六例中五例误差为 0 或 1。

Figure 2: Our extraction attack recovers the hidden dimension by identifying a sharp drop in singular values, visualized as a spike in the difference between consecutive singular values. On Pythia-1.4B, a 2048 dimensional model, the spike occurs at 2047 values.

图 2：抽取攻击通过识别奇异值急剧下降来恢复隐藏维，在相邻奇异值差值上表现为尖峰。对 2048 维的 Pythia-1.4B，尖峰出现在约第 2047 个值处。

Our near perfect extraction has one exception: GPT-2 Small. On this 768 dimensional model, our attack reports a hidden dimension of 757. In Appendix A we show that this “failure” is caused by GPT-2 actually having an effective hidden dimensionality of $757$ despite having $768$ dimensions.

近乎完美抽取有一例外：GPT-2 Small。该 768 维模型上，攻击报告隐藏维 757。附录 A 说明这次「失败」是因为 GPT-2 虽有 768 维，有效隐藏维实际是 757。

Cheaper Dimension Extraction

Note that $l$ being exactly equal to the vocabulary size is not crucial. Formally, taking only $l^{\prime}<l$ rows of $\mathbf{Q}$ does not change the number of nonzero singular values, except in the unlikely case that the resulting submatrix is of smaller rank. Hence, we can choose a subset of $l^{\prime}$ tokens and extract the dimension from logits on these tokens alone, as long as $l^{\prime}>h$.

更便宜的维数抽取

注意 $l$ 恰好等于词表大小并非关键。形式上，只取 $\mathbf{Q}$ 的 $l'<l$ 行不改变非零奇异值个数，除非子矩阵秩意外变小。因此可选 $l'$ 个 token 的子集，仅用这些 token 上的 logits 抽维数，只要 $l'>h$。

4.2 Full Layer Extraction (Up to Symmetries)

4.2 完整层抽取（至多差对称性）

We extend the attack from the prior section to recover the final output projection matrix $\mathbf{W}$ that maps from the final hidden layer to the output logits.

把上一节攻击扩展为恢复把最终隐藏层映到输出 logits 的末层投影矩阵 $\mathbf{W}$。

Method: Let $\mathbf{Q}$ be as defined in Algorithm 1. Now rewrite $\mathbf{Q}=\mathbf{U}\cdot\mathbf{\Sigma}\cdot\mathbf{V}^{\top}$ with SVD. Previously we saw that the number of large enough singular values corresponded to the dimension of the model. But it turns out that the matrix $\mathbf{U}$ actually directly represents (a linear transformation of) the final layer! Specifically, we can show that $\mathbf{U}\cdot\mathbf{\Sigma}=\mathbf{W}\cdot\mathbf{G}$ for some $h\times h$ matrix $\mathbf{G}$ in the following lemma.

方法： 令 $\mathbf{Q}$ 如算法 1 所定义。将 $\mathbf{Q}=\mathbf{U}\cdot\mathbf{\Sigma}\cdot\mathbf{V}^{\top}$ 做 SVD。先前看到足够大的奇异值个数对应模型维数；实际上矩阵 $\mathbf{U}$ 直接表示（经线性变换后的）末层！具体地，可证明 $\mathbf{U}\cdot\mathbf{\Sigma}=\mathbf{W}\cdot\mathbf{G}$，其中 $\mathbf{G}$ 为某 $h\times h$ 矩阵，见下引理。

Lemma 4.2.

In the logit-API threat model, under the assumptions of Lemma  4.1:(i)The method above recovers $\tilde{\mathbf{W}}=\mathbf{W}\cdot\mathbf{G}$ for some $\mathbf{G}\in\mathbb{R}^{h\times h}$;(ii)With the additional assumption that $g_{\theta}(p)$ is a transformer with residual connections, it is impossible to extract $\mathbf{W}$ exactly.

引理 4.2。

在 logit-API 威胁模型下，于引理 4.1 的假设中：（i）上述方法恢复 $\tilde{\mathbf{W}}=\mathbf{W}\cdot\mathbf{G}$，其中某 $\mathbf{G}\in\mathbb{R}^{h\times h}$；（ii）若额外假设 $g_{\theta}(p)$ 是带残差连接的 transformer，则不可能精确抽出 $\mathbf{W}$。

Proof. See Appendix C. $\square$

证明。 见附录 C。$\square$

Note that we could also use $\mathbf{Q}=\mathbf{W}\cdot\mathbf{G}$ for $n=l$. The SVD construction above gains numerical precision if $n>l$.

也可对 $n=l$ 直接用 $\mathbf{Q}=\mathbf{W}\cdot\mathbf{G}$。若 $n>l$，上述 SVD 构造数值上更稳。

Experiments.

For the six models considered previously, we evaluate the attack success rate by comparing the root mean square (RMS) between our extracted matrix $\tilde{\mathbf{W}}=\mathbf{U}\cdot\mathbf{\Sigma}$ and the actual weight matrix, after allowing for a $h\times h$ affine transformation. Concretely, we solve the least squares system $\tilde{\mathbf{W}}\cdot\mathbf{G}\approx\mathbf{W}$ for $\mathbf{G}$, which reduces to $h$ linear least squares problems, each with $l$ equations and $h$ unknowns. Then, we report the RMS of $\mathbf{W}$ and $\tilde{\mathbf{W}}\cdot\mathbf{G}$.

实验。

对前述六个模型，比较抽出矩阵 $\tilde{\mathbf{W}}=\mathbf{U}\cdot\mathbf{\Sigma}$ 与真实权重矩阵在允许 $h\times h$ 仿射变换后的均方根误差（RMS），以评估攻击成功率。具体地，解最小二乘 $\tilde{\mathbf{W}}\cdot\mathbf{G}\approx\mathbf{W}$ 求 $\mathbf{G}$，化为 $h$ 个线性最小二乘，每个 $l$ 个方程、$h$ 个未知量。再报告 $\mathbf{W}$ 与 $\tilde{\mathbf{W}}\cdot\mathbf{G}$ 的 RMS。

The results are in Table 2. As a point of reference, the RMS between a randomly initialized model and the actual weights is $2\cdot 10^{-2}$, over $100$–$500\times$ higher than the error of our reconstruction.

结果见表 2。作参照：随机初始化模型与真实权重的 RMS 为 $2\cdot 10^{-2}$，比我们的重构误差高 $100$–$500\times$。

In Appendices C and H, we show that reconstruction is possible up an orthogonal transformation (approximately $h^{2}/2$ missing parameters, as opposed to $h^{2}$ for reconstruction up to an affine transformation), and that this is tight under some formal assumptions. However, we only have an efficient algorithm for reconstruction up to affine transformations.

附录 C 与 H 表明：可重构至差一个正交变换（约缺 $h^{2}/2$ 个参数，而仿射意义下缺 $h^{2}$ 个），并在某些形式假设下这是紧的。但我们目前只有高效算法做到仿射意义下的重构。

Model	Hidden Dim	Stolen Size	$\mathbf{W}$ RMS
GPT-2 Small (fp32)	$768$	757 $\pm$ 1	$4\cdot 10^{-4}$
GPT-2 XL (fp32)	$1600$	1599 $\pm$ 1	$6\cdot 10^{-4}$
Pythia-1.4 (fp16)	$2048$	2047 $\pm$ 1	$3\cdot 10^{-5}$
Pythia-6.9 (fp16)	$4096$	4096 $\pm$ 1	$4\cdot 10^{-5}$
LLaMA 7B (fp16)	$4096$	4096 $\pm$ 2	$8\cdot 10^{-5}$
LLaMA 65B (fp16)	$8192$	8192 $\pm$ 2	$5\cdot 10^{-5}$

Table 2: Our attack succeeds across a range of open-source models, at both stealing the model size, and also at reconstructing the output projection matrix (up to invariances; we show the root MSE).

模型	隐藏维	窃取规模	$\mathbf{W}$ RMS
GPT-2 Small (fp32)	$768$	757 $\pm$ 1	$4\cdot 10^{-4}$
GPT-2 XL (fp32)	$1600$	1599 $\pm$ 1	$6\cdot 10^{-4}$
Pythia-1.4 (fp16)	$2048$	2047 $\pm$ 1	$3\cdot 10^{-5}$
Pythia-6.9 (fp16)	$4096$	4096 $\pm$ 1	$4\cdot 10^{-5}$
LLaMA 7B (fp16)	$4096$	4096 $\pm$ 2	$8\cdot 10^{-5}$
LLaMA 65B (fp16)	$8192$	8192 $\pm$ 2	$5\cdot 10^{-5}$

表 2：攻击在多种开源模型上成功，既能窃取模型规模，也能重构输出投影矩阵（至多差不变性；给出均方根误差）。

5 Extraction Attack for Logit-Bias APIs

5 面向 Logit-Bias API 的抽取攻击

The above attack makes a significant assumption: that the adversary can directly observe the complete logit vector for each input. In practice, this is not true: no production model we are aware of provides such an API. Instead, for example, they provide a way for users to get the top-$K$ (by logit) token log probabilities. In this section we address this challenge.

上述攻击有一重要假设：对手能直接观察每次输入的完整 logit 向量。实践中并非如此：我们已知的生产模型都不提供这种 API。例如它们只让用户拿到按 logit 排序的 top-$K$ token 对数概率。本节处理这一挑战。

5.1 Description of the API

5.1 API 描述

In this section we develop attacks for APIs that return log probabilities for the top $K$ tokens (sorted by logits), and where the user can specify a real-valued bias $b\in\mathbb{R}^{|\mathcal{X}|}$ (the “logit bias”) to be added to the logits for specified tokens before the softmax, i.e.,

本节为返回 top $K$ token（按 logits 排序）对数概率、且用户可指定加到指定 token logits 上的实值偏置 $b\in\mathbb{R}^{|\mathcal{X}|}$（「logit bias」）的 API 发展攻击，即

$$\displaystyle\mathcal{O}(p,b)\leftarrow\mathsf{TopK}\left({\mathsf{logsoftmax}\left({\mathbf{W}g_{\theta}(p)+b}\right)}\right) \\ \displaystyle=\!\mathsf{TopK}\!\left(\!\mathbf{W}g_{\theta}(p)\!+\!b\!-\!\log\!\left(\!\sum_{i}\exp\!\left(\!\mathbf{W}g_{\theta}(p)\!+\!b\right)_{i}\!\right)\!\cdot\!\mathbf{1}\!\right)\!.$$

where $\mathsf{TopK}\left({\mathbf{z}}\right)$ returns the $K$ highest entries of $\mathbf{z}\in\mathbb{R}^{l}$ and their indices. Many APIs (prior to this paper) provided such an option for their state-of-the-art models (OpenAI, 2024; Google, 2024). In particular, the OpenAI API supports modifying logits for at most $300$ tokens, and the logit bias for each token is restricted to the range $[-100,100]$ (OpenAI, 2023).

其中 $\mathsf{TopK}\left({\mathbf{z}}\right)$ 返回 $\mathbf{z}\in\mathbb{R}^{l}$ 的 $K$ 个最高项及其下标。许多 API（在本文之前）为其先进模型提供此类选项（OpenAI, 2024；Google, 2024）。特别地，OpenAI API 最多支持修改 300 个 token 的 logits，且每个 token 的 logit bias 限制在 $[-100,100]$（OpenAI, 2023）。

All that remains is to show that we can uncover the full logit vector for distinct prompt queries through this API. In this section, we develop techniques for this purpose. Once we have recovered multiple complete logit vector, we can run the attack from Section 4.2 without modification.

剩下只需说明：经该 API 可为不同 prompt 查询揭开完整 logit 向量。本节为此发展技术。一旦恢复多个完整 logit 向量，即可不加修改地跑第 4.2 节攻击。

5.2 Evaluation Methodology

5.2 评估方法

Practical attacks must be efficient, both to keep the cost of extraction manageable and to bypass any rate limiters or other filters in the APIs. We thus begin with two cost definitions that we use to measure the efficacy of our attack.

实用攻击必须高效，既要控制抽取费用，也要绕过 API 的速率限制或其他过滤。我们先定义两个成本指标衡量攻击效果。

Token cost: the number of tokens the adversary sends to (or receives from) the model during the attack. Most APIs charge users per-token, so this metric represents the monetary cost of an attack (after scaling by the token cost).

Token 成本： 攻击期间对手发给（或从）模型收到的 token 数。多数 API 按 token 计费，故该指标（乘以单价后）代表金钱成本。

Query cost: the total duration of the attack. Most APIs place a limit on the number of queries an adversary can make in any given interval, and so some attacks may be faster but cost more (by sending more tokens per query).

查询成本： 攻击总时长。多数 API 限制给定时间窗内的查询次数，因而有的攻击更快但更贵（每次查询发送更多 token）。

In the remainder of this section we develop several attacks under varying attack assumptions and optimizing for either token cost, query cost, or both.

本节余下在不同攻击假设下发展若干攻击，分别或同时优化 token 成本、查询成本。

5.3 Extraction Attack for Top-5 Logit Bias APIs

5.3 面向 Top-5 Logit Bias API 的抽取攻击

We develop a technique to compute the logit vector for any prefix $p$ via a sequence of queries with varying logit biases.

我们发展一种技术：通过对不同 logit bias 的一串查询，计算任意前缀 $p$ 的 logit 向量。

To begin, suppose that the API returned the top $K$ logits. Then we could recover the complete logit vector for an arbitrary prompt $p$ by cycling through different choices for the logit bias and measuring the top-$k$ logits each time. In particular, for an API with top-5 logits we can send a sequence of queries

首先，假设 API 返回 top $K$ logits。那么可通过轮换不同 logit bias、每次测量 top-$k$ logits，恢复任意 prompt $p$ 的完整 logit 向量。对 top-5 logits 的 API，可发送一串查询

$$\mathcal{O}(p,b_{k}\!=\!b_{k+1}\!=\!\dots\!=\!b_{k+4}\!=\!B)\text{, for }k\!\in\!\{\!0,\!5,\!10,\!\dots\!,\!|\mathcal{X}|\!\}$$

with a large enough $B$. Each query thus promotes five different tokens $\{k,k+1,\ldots,k+4\}$ into the top-5, which allows us to observe their logits. By subtracting the bias $B$ and merging answers from all of these queries, we recover the entire logit vector.

其中 $B$ 足够大。每次查询把五个不同 token $\{k,k+1,\ldots,k+4\}$ 抬进 top-5，从而观察到它们的 logits。减去偏置 $B$ 并合并全部查询答案，即可恢复整条 logit 向量。

Unfortunately, we cannot use this attack directly because all production APIs we are aware of return logprobs (the log of the softmax output of the model) instead of the logits $z_{i}$. The problem now is that when we apply a logit bias $B$ to the $i$-th token and observe that token’s logprob, we get the value

可惜不能直接用该攻击：我们已知的生产 API 都返回 logprobs（模型 softmax 输出的对数），而非 logits $z_{i}$。问题在于：对第 $i$ 个 token 施加 logit bias $B$ 并观察其 logprob 时，得到的是

$$y_{i}^{B}=z_{i}+B-\log\big(\sum_{j\neq i}\exp(z_{j})+\exp(z_{i}+B)\big)$$

where $z_{i}$ are the original logits. We thus get an additional bias-dependent term which we need to deal with. We propose two approaches.

其中 $z_{i}$ 是原始 logits。因而多出一个依赖偏置的项需要处理。我们提出两种做法。

Our first approach relies on a common “reference” token that lets us learn the relative difference between all logits (this is the best we can hope for, since the softmax is invariant under additive shifts to the logits). Suppose the top token for a prompt is $R$, and we want to learn the relative difference between the logits of tokens $i$ and $R$. We add a large bias $B$ to token $i$ to push it to the top-5, and then observe the logprobs of both token $i$ and $R$. We have:

第一种依赖公共「参考」token，用来学习所有 logits 的相对差（这已是最好结果，因为 softmax 对 logits 的加性平移不变）。设某 prompt 的 top token 为 $R$，想学 token $i$ 与 $R$ 的 logit 相对差。给 token $i$ 加大偏置 $B$ 把它推进 top-5，再同时观察 token $i$ 与 $R$ 的 logprobs。有：

$$y_{R}^{B}-y_{i}^{B}-B=z_{R}-z_{i}\,.$$

Since we can observe 5 logprobs, we can compare the reference token $R$ to four tokens per query, by adding a large bias that pushes all four tokens into the top 5 (along with the reference token). We thus issue a sequence of queries

由于可观察 5 个 logprobs，每次查询可用参考 token $R$ 对比四个 token：给四个 token 都加大偏置，使它们连同参考 token 进入 top 5。于是发送一串查询

$$\mathcal{O}(p,b_{i}=b_{i+1}=b_{i+2}=b_{i+3}=B)$$

for $i\in\{0,4,8,\cdots,|\mathcal{X}|\}$. This recovers the logits up to the free parameter $z_{R}$ that we set to $0$.

对 $i\in\{0,4,8,\cdots,|\mathcal{X}|\}$。这样恢复的 logits 至多差自由参数 $z_{R}$，我们把它设为 $0$。

Query cost. This attack reveals the value of K-1 logits with each query to the model (the $K$-th being used as a reference point), for a cost of $1/(K-1)$ queries per logit.

查询成本。 该攻击每次模型查询揭示 $K-1$ 个 logit 值（第 $K$ 个用作参考），故每个 logit 的查询成本为 $1/(K-1)$。

In Appendix E we present a second, more sophisticated method that allows us to recover $K$ logits per query, i.e., a cost of $1/K$, by viewing each logprob we receive as a linear constraint on the original logits.

附录 E 给出第二种更精巧的方法：把每次收到的 logprob 视为对原始 logits 的线性约束，从而每次查询恢复 $K$ 个 logits，即成本 $1/K$。

Token cost. Recall that our attack requires that we learn the logits for several distinct prompts; and so each prompt must be at least one token long. Therefore, this attack costs at least two tokens per query (one input and one output), or a cost of $1/2$ for each token of output. But, in practice, many models (like gpt-3.5-turbo) include a few tokens of overhead along with every single query. This increases the token cost per logit to ${2+\Delta}\over 4$ where $\Delta$ is the number of overhead tokens; for gpt-3.5-turbo we report $\Delta=7$.

Token 成本。 回忆攻击需要学习若干不同 prompt 的 logits，每个 prompt 至少一 token 长。因此每次查询至少两 token（一入一出），或每个输出 token 成本 $1/2$。但实践中许多模型（如 gpt-3.5-turbo）每次查询还带少量开销 token，把每个 logit 的 token 成本抬到 ${2+\Delta}\over 4$，其中 $\Delta$ 为开销 token 数；对 gpt-3.5-turbo 我们报告 $\Delta=7$。

An improved cost-optimal attack.

It is possible to generalize the above attack to improve both the query cost and token cost. Instead of issuing queries to the model that reveal 4 or 5 logit values for a single generated token, we might instead hope to be able to send a multi-token query $[p_{0}\quad p_{1}\quad p_{2}\dots p_{n}]$ and then ask for the logprob vector for each prefix of the prompt $[p_{0}]$, $[p_{0}\quad p_{1}]$, $[p_{0}\quad p_{1}\quad p_{2}]$ etc. OpenAI’s API did allow for queries of this form in the past, by providing logprobs for prompt tokens as well as generated tokens by combining the logprob and echo parameters; this option has since been removed.

改进的成本最优攻击。

可推广上述攻击，同时改进 查询成本 与 token 成本。与其每次查询只为一个生成 token 揭示 4 或 5 个 logit 值，不如希望发送多 token 查询 $[p_{0}\quad p_{1}\quad p_{2}\dots p_{n}]$，并请求每个前缀 $[p_{0}]$、$[p_{0}\quad p_{1}]$、$[p_{0}\quad p_{1}\quad p_{2}]$ 等的 logprob 向量。OpenAI 的 API 过去确曾允许这类查询：把 logprob 与 echo 参数组合，即可对 prompt token 与生成 token 都给 logprobs；该选项现已移除。

Now, it is only possible to view logprobs of generated tokens. And since only the very last token is generated, we can only view four logprobs for this single longer query. This, however, presents a potential approach to reduce the query and token cost: if there were some way to cause the model to emit a specific sequence of tokens $[p_{n+1}\quad p_{n+2}\quad\dots\quad p_{n+m}]$, then we could inspect the logprob vector of each generated token.

现在只能看 生成 token 的 logprobs。由于只有最后一个 token 被生成，对这条更长查询也只能看到四个 logprobs。但这提示一条降低查询与 token 成本的路：若能使模型发出特定序列 $[p_{n+1}\quad p_{n+2}\quad\dots\quad p_{n+m}]$，即可检查每个生成 token 的 logprob 向量。

We achieve this as follows: we fix a token $x$ and four other tokens, and force the model to emit $[x\quad x\quad\ldots\quad x]$. Instead of supplying a logit bias of $B$ for each of the five tokens, we supply a logit bias of $B$ for token $x$, and $B^{\prime}<B$ for the other four tokens. If $B^{\prime}$ is large enough so that the other tokens will be brought into the top-5 outputs, we will still be able to learn the logits for those tokens. As long as $B^{\prime}$ is small enough so that the model will always complete the initial prompt $p_{0}$ with token $x$ (and not any other), then we will be able to collect the logits on several prompts of the form $[p_{0}\quad x\quad x\quad\ldots\quad x]$.

做法如下：固定 token $x$ 与另外四个 token，迫使模型发出 $[x\quad x\quad\ldots\quad x]$。不是给五个 token 都供 logit bias $B$，而是给 $x$ 供 $B$，给另外四个供 $B'<B$。若 $B'$ 大到足以把其他 token 抬进 top-5，仍可学到它们的 logits；只要 $B'$ 小到模型总会用 $x$（而非其他）补全初始 prompt $p_{0}$，就能在形如 $[p_{0}\quad x\quad x\quad\ldots\quad x]$ 的若干 prompt 上收集 logits。

Analysis. It is easy to see that the query cost of this attack is $1\over{4m}$, where $m$ is the expansion factor. Further, since each query requires $1+m$ tokens, the token cost is $1+m\over 4m$. (Or, $1+m+\Delta$ if the API has an overhead of $\Delta$ tokens.) Note that if $m=1$, i.e., there is no expansion, this attack reduces to our first attack and the analysis similarly gives a query cost of $\nicefrac{{1}}{{4}}$ and a token cost of $\nicefrac{{1}}{{2}}$.

分析。 易见该攻击查询成本为 $1\over{4m}$，其中 $m$ 为扩展因子。又因每次查询需 $1+m$ 个 token，token 成本为 $1+m\over 4m$。（若 API 有 $\Delta$ 开销 token，则为 $1+m+\Delta$。）注意若 $m=1$（无扩展），攻击退化为第一种，分析同样给出查询成本 $\nicefrac{{1}}{{4}}$、token 成本 $\nicefrac{{1}}{{2}}$。

	Dimension Extraction	Weight Matrix Extraction
Model	Size	# Queries	Cost (USD)	RMS	# Queries	Cost (USD)
OpenAI ada	$1024$ ✓	$<2\cdot 10^{6}$	$1	$5\cdot 10^{-4}$	$<2\cdot 10^{7}$	$4
OpenAI babbage	$2048$ ✓	$<4\cdot 10^{6}$	$2	$7\cdot 10^{-4}$	$<4\cdot 10^{7}$	$12
OpenAI babbage-002	$1536$ ✓	$<4\cdot 10^{6}$	$2	†	$<4\cdot 10^{6}$ †+	$12
OpenAI gpt-3.5-turbo-instruct	∗   ✓	$<4\cdot 10^{7}$	$200	†	$<4\cdot 10^{8}$ †+	$2,000*†+*
OpenAI gpt-3.5-turbo-1106	∗   ✓	$<4\cdot 10^{7}$	$800	†	$<4\cdot 10^{8}$ †+	$8,000*†+*

Table 3: Attack success rate on five different black-box models

	维数抽取			权重矩阵抽取
模型	规模	# 查询	费用 (USD)	RMS	# 查询	费用 (USD)
OpenAI ada	$1024$ ✓	$<2\cdot 10^{6}$	$1	$5\cdot 10^{-4}$	$<2\cdot 10^{7}$	$4
OpenAI babbage	$2048$ ✓	$<4\cdot 10^{6}$	$2	$7\cdot 10^{-4}$	$<4\cdot 10^{7}$	$12
OpenAI babbage-002	$1536$ ✓	$<4\cdot 10^{6}$	$2	†	$<4\cdot 10^{6}$ †+	$12
OpenAI gpt-3.5-turbo-instruct	∗   ✓	$<4\cdot 10^{7}$	$200	†	$<4\cdot 10^{8}$ †+	$2,000*†+*
OpenAI gpt-3.5-turbo-1106	∗   ✓	$<4\cdot 10^{7}$	$800	†	$<4\cdot 10^{8}$ †+	$8,000*†+*

表 3：五种黑盒模型上的攻击成功率

Attack	Logprobs	Bits of precision	Queries per logit
logprob-4 (§5.3)	top-5	$23.0$	$0.25$
logprob-5 (§E)	top-5	$11.5$	$0.64$
logprob-1 (§5.4)	top-1	$6.1$	$1.0$
binary search (§F.1)	✗	$7.2$	$10.0$
hyperrectangle (§F.2)	✗	$15.7$	$5.4$
one-of-n (§F.3)	✗	$18.0$	$3.7$

Table 4: Average error at recovering the logit vector for each of the logit-estimation attacks we develop. Our highest precision, and most efficient attack, recovers logits nearly perfectly; other attacks approach this level of precision but at a higher query cost.

攻击	Logprobs	精度比特	每 logit 查询数
logprob-4（§5.3）	top-5	$23.0$	$0.25$
logprob-5（§E）	top-5	$11.5$	$0.64$
logprob-1（§5.4）	top-1	$6.1$	$1.0$
二分搜索（§F.1）	✗	$7.2$	$10.0$
超矩形（§F.2）	✗	$15.7$	$5.4$
one-of-n（§F.3）	✗	$18.0$	$3.7$

表 4：各类 logit 估计攻击恢复 logit 向量的平均误差。精度最高且最高效的攻击几乎完美恢复 logits；其他攻击接近该精度，但查询成本更高。

5.4 Extraction Attack for top-1 Binary Logit Bias APIs

5.4 面向 top-1 二元 Logit Bias API 的抽取攻击

In light of our attacks, it is conceivable that model providers introduce restrictions on the above API. We now demonstrate that an attack is possible even if the API only returns the top logprob ($K=1$ in the API from Section 5.1), and the logit bias is constrained to only take one of two values.

鉴于本攻击，服务方可想象对上述 API 加限制。我们接着证明：即便 API 只返回 top logprob（第 5.1 节 API 中 $K=1$），且 logit bias 只能取两个值之一，攻击仍可能。

API. We place two following further restrictions on the logit bias API (Section 5.1): first, we set $K=1$, and only see the most likely token’s logprob; and second, each logit bias entry $b$ is constrained to be in $\{-1,0\}$. These constraints would completely prevent the attacks from the prior section. We believe this constraint is significantly tighter than any practical implementation would define.

API。 对 logit bias API（第 5.1 节）再加两条限制：第一，$K=1$，只看最可能 token 的 logprob；第二，每个 logit bias 项 $b$ 只能取 $\{-1,0\}$。这些约束会彻底挡住上一节攻击。我们相信这比任何实际部署会设的限制都紧得多。

Method. At first it may seem impossible to be able to learn any information about a token $t$ if it is not already the most likely token. However, note that if we query the model twice, once without any logit bias, and once with a logit bias of $-1$ for token $t$, then the top token will be slightly more likely with a bias of $-1$, with exactly how slight depending on the value of token $t$’s logprob. Specifically, in Appendix D we show the logprob equals $(\nicefrac{{1}}{{e}}-1)^{-1}(\exp(y_{\text{top}}-y^{\prime}_{\text{top}})-1)$ where $y_{\text{top}}$ and $y^{\prime}_{\text{top}}$ are the logprobs of the most likely token when querying with logit bias of $0$ and $-1$.

方法。 乍看若 token $t$ 本不是最可能者，似乎学不到任何关于它的信息。但注意：若查两次——一次无 logit bias，一次给 token $t$ 加 $-1$ 的 bias——则带 $-1$ bias 时 top token 会略更可能，幅度恰好取决于 token $t$ 的 logprob 取值。附录 D 具体证明 logprob 等于 $(\nicefrac{{1}}{{e}}-1)^{-1}(\exp(y_{\text{top}}-y'_{\text{top}})-1)$，其中 $y_{\text{top}}$ 与 $y'_{\text{top}}$ 分别是 bias 为 $0$ 与 $-1$ 时最可能 token 的 logprob。

Analysis. This attack requires $1$ query and token per logprob extracted. However, as we will show in the evaluation, this attack is much less numerically stable than the previously-discussed attacks, and so may require more queries to reach the same level of accuracy.

分析。 该攻击每抽出一个 logprob 需 1 次查询与 1 个 token。但如下文评估所示，数值稳定性远差于前述攻击，达到同等准确率可能需要更多查询。

6 Logprob-free attacks

6 无 Logprob 攻击

Due to space constraints, in Appendix F, we show we can still extract logits without logprob access, although with a higher cost.

篇幅所限，附录 F 说明：即使没有 logprob 访问，仍可抽出 logits，只是成本更高。

Intuitively, even without logprobs (as long as we still have logit bias) it is possible to perform binary search to increase and decrease the logits for every token until increasing any token by epsilon will make it the most likely. At this point, the logit bias vector corresponds directly to the (relative) logits of each token relative to every other.

直觉上，即便没有 logprobs（只要仍有 logit bias），也可对每个 token 做二分：上下调整 logits，直到再给任意 token 加一点 epsilon 就会让它成为最可能者。此时 logit bias 向量直接对应各 token 相对其他 token 的（相对）logits。

By performing the binary search one token at a time, we can develop an effective (but inefficient) attack that requires $N\log({B\over\epsilon})$ where $N$ is the number of logits, $B$ is an upper bound on the gap between any two logits and $\epsilon$ is the desired tolerance.

逐 token 做二分，可得到有效但低效的攻击，需 $N\log({B\over\epsilon})$，其中 $N$ 为 logit 个数，$B$ 为任意两 logit 间隙上界，$\epsilon$ 为期望容差。

An improved attack is possible by noticing that it is possible to perform binary search on multiple tokens in parallel. Because the adversary gets to view the arg-max sampled token, by modifying multiple tokens at the same time we can learn information faster and therefore improve attack efficiency.

注意到可对多个 token 并行二分，就能改进：对手能看到 arg-max 采样 token，同时改多个 token 可更快获得信息，从而提高攻击效率。

7 Evaluation

7 评估

We now study the efficacy of our practical stealing attack.

下面研究实用窃取攻击的效果。

7.1 Logit Validation

7.1 Logit 验证

We begin by validating that the attacks developed in the prior sections can effectively recover the full logit vector given a limited query interface. In Table 4 we report the average number of bits of agreement between the true logit vector and the recovered logit vector, as well as the (amortized) number of queries required to recover one full logit vector.

先验证前述各节攻击能否在受限查询接口下有效恢复完整 logit 向量。表 4 报告真实 logit 向量与恢复向量之间一致的平均比特数，以及恢复一条完整 logit 向量的（摊销）查询次数。

Generally, attacks that operate under stronger threat models have higher precision. But theoretical improvements are not always practical: the theoretically stronger attack from §E that learns 5 logprobs per query in practice requires more queries and recovers logits with lower fidelity. This is because this attack is numerically unstable: it requires a potentially ill-conditioned matrix, and therefore can require re-querying the API after adjusting the logit bias. Our strongest logprob-free attack is highly efficient, and recovers $18$ bits of precision at just $3.7$ queries per logit. In Appendix G we theoretically analyze how far this is from optimal, and find it is within a factor of two.

一般而言，威胁模型更强的攻击精度更高。但理论改进未必实用：§E 中理论上每次学 5 个 logprobs 的更强攻击，实践中反而查询更多、保真更低。因其数值不稳：需要可能病态的矩阵，调 logit bias 后可能要重查 API。我们最强的无 logprob 攻击很高效，仅 $3.7$ 次查询/logit 即可恢复 $18$ 比特精度。附录 G 理论分析其与最优的距离，发现在两倍以内。

7.2 Stealing Parts of Production Models

7.2 窃取生产模型的一部分

We now investigate our ability to steal production language models, focusing on five of OpenAI’s models available on 1 January 2024: ada, babbage, babbage-002, gpt-3.5-turbo-instruct, and gpt-3.5-turbo-1106. We selected these models because these were the only production models which were able to receive advance permission to attempt an extraction attack; we are exceptionally grateful to OpenAI for allowing us to perform this research using their models.

接着考察窃取生产语言模型的能力，聚焦 2024 年 1 月 1 日可用的五个 OpenAI 模型：ada、babbage、babbage-002、gpt-3.5-turbo-instruct、gpt-3.5-turbo-1106。选它们是因为当时仅这些生产模型获准提前尝试抽取攻击；非常感谢 OpenAI 允许我们用其模型做此研究。

Given the results from the prior section, we chose to implement the improved 4-logprob attack (Section 5.3) because it is both the most query efficient attack and also the most precise attack. Switching to a different attack algorithm would increase our total experiment cost significantly, and so we do not perform these ablation studies.

基于上一节结果，我们实现改进的 4-logprob 攻击（第 5.3 节），因其既最省查询也最精确。换其他算法会显著抬高总实验费用，故不做这些消融。

Both our hidden-dimension-stealing and entire-layer-stealing attack worked for all five of these models. The size we recover from the model perfectly matches the actual size of the original model, as confirmed by OpenAI. For the first three models, we report in Table 3 the size we recover because (1) the sizes of these models was never previously confirmed, but (2) they have now been deprecated and so disclosing the size is not harmful. In discussions with OpenAI, we decided to withhold disclosure of the size of gpt-3.5-turbo models, but we confirmed with them that the number our attack reported was accurate.

隐藏维窃取与整层窃取对五个模型均成功。恢复的规模与 OpenAI 确认的真实规模完全一致。对前三个模型，表 3 报告恢复规模，因为（1）这些规模此前从未被确认，但（2）它们现已弃用，披露规模无害。与 OpenAI 讨论后，我们决定不披露 gpt-3.5-turbo 模型的规模，但已与对方确认攻击报告的数字准确。

When running the full layer-stealing attack, we confirmed that our extracted weights are nearly identical to the actual weights, with error $<7\cdot 10^{-4}$, up to an $h\times h$ matrix product as discussed previously. Table 3 reports the RMS between our extracted weight matrix and the actual model weights, after “aligning” the two by an $h\times h$ transform.

跑完整层窃取时，确认抽出权重与真实权重近乎相同，误差 $<7\cdot 10^{-4}$（至多差前述 $h\times h$ 矩阵乘积）。表 3 报告抽出权重矩阵与真实权重在经 $h\times h$ 变换「对齐」后的 RMS。

8 Defenses

8 防御

It would be possible to prevent or mitigate this attack in a number of different ways, albeit with loss of functionality.

有多种方式可阻止或缓解本攻击，但都会损失部分功能。

8.1 Prevention

8.1 阻止

Remove logit bias. Perhaps the simplest defense would be to outright remove the logit bias parameter from the API. Unfortunately, there are several legitimate use cases of this parameter. For example, several works use logit bias in order to perform controlled or constrained generation (Jiang et al., 2023; Yang & Klein, 2021), to shift generation and mimic fine-tuning the model (Liu et al., 2024; Mitchell et al., 2024), or other reasons (Ren et al., 2023; Lee et al., 2022).

去掉 logit bias。 最简单的防御或许是干脆从 API 移除 logit bias 参数。可惜该参数有若干正当用途。例如有工作用 logit bias 做受控或受限生成（Jiang et al., 2023；Yang & Klein, 2021），用它偏移生成以模拟微调（Liu et al., 2024；Mitchell et al., 2024），或其他用途（Ren et al., 2023；Lee et al., 2022）。

Replace logit bias with a block-list. Instead of offering a logit bias, model developers could replace it with a block-list of tokens the model is prohibited from emitting. This would support (some) of the functionality discussed in the prior section, but would still prevent our attack.

用屏蔽列表替代 logit bias。 不提供 logit bias，而改为禁止模型发出某些 token 的屏蔽列表。可支持上一节部分功能，但仍能挡住本攻击。

Architectural changes. Instead of modifying the API, we could instead make changes to the model. Our attack only works because the hidden dimension $h$ is less than the output dimension $l$. This suggests a natural architectural defense: split the final layer into two layers, one that goes from $h\to t$ and then $t\to l$ where $t>l$ and a nonlinearity was placed in between. This is not very efficient though, as the last linear layer is large (quadratic in the vocabulary size).

架构改动。 也可不改 API 而改模型。本攻击成立是因为隐藏维 $h$ 小于输出维 $l$。这提示自然的架构防御：把末层拆成两层，先 $h\to t$ 再 $t\to l$，其中 $t>l$，中间加非线性。效率不佳，因最后线性层很大（对词表大小二次）。

Post-hoc altering the architecture. We can also modify the hidden dimension $h$ for the final layer after the model is trained. In particular, we can expand the dimensionality of $\mathbf{W}$ by concatenating extra weight vectors that are orthogonal to the original matrix. We set the singular values for these weights to be small enough to not materially affect the model’s predictions, while also being large enough to look realistic. Then, during the model’s forward pass, we concatenate a vector of random Gaussian noise to the final hidden vector $g_{\theta}(p)$ before multiplying by $\mathbf{W}$. Figure 5 shows an example of this, where we expand GPT-2 small to appear as if it was 1024 dimensional instead of 768 dimensions. This misleads the adversary into thinking that the model is wider than it actually is.

事后改架构。 也可在训练后修改末层的隐藏维 $h$。具体地，给 $\mathbf{W}$ 拼接与原矩阵正交的额外权重向量以扩大维数；把这些权重的奇异值设得足够小以免实质影响预测，又足够大以显得真实。前向时，在乘 $\mathbf{W}$ 前给最终隐藏向量 $g_{\theta}(p)$ 拼接随机高斯噪声向量。图 5 给出例子：把 GPT-2 small 扩成看起来像 1024 维而非 768 维，从而误导对手以为模型更宽。

8.2 Mitigations

8.2 缓解

Logit bias XOR logprobs. Our attack is $10\times$ cheaper when an adversary can supply both a logit bias and also view output logprobs. This suggests a natural mitigation: prohibit queries to the API that make use of both logit bias and logprobs at the same time. This type of defense is common in both the security and machine learning community: for example, in 2023 OpenAI removed the ability to combine both echo and logprobs, but with either alone being allowed; this defense would behave similarly.

Logit bias 与 logprobs 互斥。 对手能同时提供 logit bias 并查看输出 logprobs 时，本攻击便宜约 $10\times$。自然缓解：禁止同一查询同时使用 两者。安全与机器学习社区常见此类防御：例如 2023 年 OpenAI 去掉同时使用 echo 与 logprobs 的能力，但单独使用仍允许；本防御行为类似。

Noise addition. By adding a sufficient amount of noise to the output logits of any given query, it would be possible to prevent our attack. However, logit-noise has the potential to make models less useful. We perform some preliminary experiments on this direction in Appendix I.

加噪声。 给任意查询的输出 logits 加足够噪声，可阻止本攻击。但 logit 噪声可能降低模型有用性。附录 I 做了该方向的初步实验。

Rate limits on logit bias. Our attack requires that we are able to learn at least $h$ logit values for each prompt $p$. One defense would be to allow logit-bias queries to the model, but only allow $T=\tilde{h}/5$ logit bias queries for any given prompt $p$ to prevent an adversary from learning if a model has hidden dimension $\tilde{h}$ or smaller.

对 logit bias 限速。 本攻击要求对每个 prompt $p$ 至少学到 $h$ 个 logit 值。一种防御是仍允许 logit-bias 查询，但对任意给定 prompt $p$ 只允许 $T=\tilde{h}/5$ 次，以防对手判断模型隐藏维是否 $\leq\tilde{h}$。

Unfortunately this has several significant drawbacks: the threshold has to be independent of $h$ (or learning the threshold would reveal $h$); the system would need to maintain state of all user queries to the API; and preventing Sybil attacks requires a global pool of user queries, which can present significant privacy risks (Debenedetti et al., 2023).

可惜有几大缺点：阈值必须与 $h$ 无关（否则学阈值就暴露 $h$）；系统需维护用户对 API 的全部查询状态；防 Sybil 攻击需要全局用户查询池，会带来显著隐私风险（Debenedetti et al., 2023）。

Detect malicious queries. Instead of preventing any queries that might leak model weights, an alternate strategy could be to implement standard anti-abuse tools to detect any patterns of malicious behavior. Several proposals of this form exist for prior machine learning attacks, including model stealing (Juuti et al., 2019; Pal et al., 2021) and adversarial examples (Chen et al., 2020). Unfortunately, these defenses are often vulnerable to attack (Feng et al., 2023), and so any mitigation here would need to improve on the state-of-the-art to be truly robust.

检测恶意查询。 与其阻止任何可能泄露权重的查询，不如用标准反滥用工具 检测 恶意行为模式。针对既有机器学习攻击已有若干此类提案，包括模型窃取（Juuti et al., 2019；Pal et al., 2021）与对抗样本（Chen et al., 2020）。可惜这些防御常可被攻破（Feng et al., 2023），因而此处的缓解需超越现有水平才真正稳健。

9 Future Work

9 未来工作

We are motivated to study this problem not because we expect to be able to steal an entire production transformer model bit-for-bit, but because we hope to conclusively demonstrate that model stealing attacks are not just of academic concern but can be practically applied to the largest production models deployed today. We see a number of potential directions for improving on this attack.

研究该问题的动机，并非期望逐比特窃取整座生产 transformer，而是希望明确证明：模型窃取不只是学术关切，也可实际作用于今天部署的最大生产模型。我们看到若干改进本攻击的方向。

Breaking symmetry with quantized weights. Large production models are typically stored “quantized”, where each weight is represented in just 4 or 8 bits. In principle, this quantization could allow an adversary to recover a nearly bit-for-bit copy of the matrix $\mathbf{W}$: while there exist an infinite number of matrices $\mathbf{W}\cdot\mathbf{G}$, only one will be discretized properly. Unfortunately, this integer-constrained problem is NP-hard in general (similar problems are the foundation for an entire class of public key cryptosystems). But this need not imply that the problem is hard on all instances.

用量化权重打破对称。 大型生产模型通常「量化」存储，每个权重仅 4 或 8 比特。原则上，量化可使对手恢复近乎逐比特的 $\mathbf{W}$：虽有无穷多 $\mathbf{W}\cdot\mathbf{G}$，但只有一个会正确离散化。可惜该整数约束问题一般 NP 难（类似问题是一类公钥密码体制的基础）。但这不意味着每个实例都难。

Extending this attack beyond a single layer. Our attack recovers a single layer of a transformer. We see no obvious methodology to extend it beyond just a single layer, due to the non-linearity of the models. But we invite further research in this area.

把攻击扩展到不止一层。 本攻击恢复 transformer 的一层。因模型非线性，我们看不到明显办法扩到更多层，但欢迎该方向进一步研究。

Removing the logit bias assumption. All our attacks require the ability to pass a logit bias. Model providers including Google and OpenAI provided this capability when we began the writing of this paper, but this could change. (Indeed, it already has, as model providers begin implementing defenses to prevent this attack.) Other API parameters could give alternative avenues for learning logit information. For example, unconstrained temperature and top-k parameters could also leak logit values through a series of queries. In the long run, completely hiding the logit information might be challenging due both to public demand for the feature, and ability of adversaries to infer this information through other means.

去掉 logit bias 假设。 我们所有攻击都需要传入 logit bias。撰写本文之初 Google 与 OpenAI 等提供商具备该能力，但可能改变。（确实已经改变：提供方开始部署防御以阻止本攻击。）其他 API 参数也可能成为学习 logit 信息的途径。例如无约束的 temperature 与 top-k 也可经一串查询泄露 logit 值。长期看，因公众对功能的需求，以及对手经其他途径推断的能力，彻底隐藏 logit 信息或许很难。

Exploiting the stolen weights. Recovering a model’s embedding projection layer might improve other attacks against that model. Alternatively, an attacker could infer details about a provider’s finetuning API by observing changes (or the absence thereof) in the last layer. In this paper, we focus primarily on the model extraction problem and leave exploring downstream attacks to future work.

利用窃取的权重。 恢复模型的嵌入投影层或可改进针对该模型的其他攻击。或者，攻击者可通过观察末层变化（或无变化）推断提供方 微调 API 的细节。本文主要聚焦模型抽取，下游攻击留待未来。

Practical stealing of other model information. Existing high-fidelity model stealing attacks are “all-or-nothing” attacks that recover entire models, but only apply to small ReLU networks. We show that stealing partial information can be much more practical, even for state-of-the-art models. Future work may find that practical attacks can steal many more bits of information about current proprietary models.

实用地窃取其他模型信息。 既有高保真模型窃取多是「全有或全无」、恢复整模，但只适用于小型 ReLU 网络。我们表明窃取部分信息可以实用得多，即便对最先进模型亦然。未来工作或会发现：针对当前专有模型，实用攻击可窃取多得多的信息比特。

10 Conclusion

10 结论

As the field of machine learning matures, and models transition from research artifacts to production tools used by millions, the field of adversarial machine learning must also adapt. While it is certainly useful to understand the potential applicability of model stealing to three-layer 100-neuron ReLU-only fully-connected networks, at some point it becomes important to understand to what extent attacks can be actually applied to the largest production models.

随着机器学习走向成熟，模型从研究产物变为数百万人使用的生产工具，对抗机器学习研究也必须适应。了解模型窃取对三层、百神经元、仅 ReLU 的全连接网络的潜在适用性当然有用，但到某个时点，重要的是弄清攻击究竟能在多大程度上实际作用于最大的生产模型。

This paper takes one step in that direction. We give an existence proof that it is possible to steal one layer of a production language model. While there appear to be no immediate practical consequences of learning this layer, it represents the first time that any precise information about a deployed transformer model has been stolen. Two immediate open questions are (1) how hazardous these practical stealing attacks are and (2) whether they pose a greater threat to developers and the security of their models than black-box access already does via distillation or other approximate stealing attacks.

本文朝该方向迈出一步。我们给出存在性证明：可以窃取生产语言模型的一层。虽看似没有立刻的实用后果，但这是首次从已部署 transformer 中窃取到任何精确信息。两个直接未决问题是：（1）这些实用窃取攻击有多危险；（2）相对黑盒访问已可通过蒸馏或其他近似窃取带来的风险，它们对开发者及其模型安全构成的威胁是否更大。

Our attack also highlights how small design decisions influence the overall security of a system. Our attack works because of the seemingly-innocuous logit-bias and logprobs parameters made available by the largest machine learning service providers, including OpenAI and Google—although both have now implemented mitigations to prevent this attack or make it more expensive. Practitioners should strive to understand how system-level design decisions impact the safety and security of the full product.

本攻击也凸显：小设计决策如何影响系统整体安全。攻击之所以成立，是因为 OpenAI 与 Google 等最大机器学习服务商提供了看似无害的 logit-bias 与 logprobs 参数——尽管两家现已实施缓解以阻止本攻击或抬高其成本。实践者应努力理解系统级设计决策如何影响完整产品的安全与保障。

Overall, we hope our paper serves to further motivate the study of practical attacks on machine learning models, in order to ultimately develop safer and more reliable systems.

总的来说，希望本文进一步推动对机器学习模型实用攻击的研究，最终发展更安全、更可靠的系统。

Impact Statement

影响声明

This paper is the most recent in a line of work that demonstrates successful attacks on production models. As such, we take several steps to mitigate the near-term potential harms of this research. As discussed throughout the paper, we have worked closely with all affected products to ensure that mitigations are in place before disclosing this work. We have additionally sent advance copies of this paper to all potentially affected parties, even if we were unable to precisely verify our attack.

本文是展示对生产模型成功攻击的系列工作中最新一篇。因此我们采取若干步骤缓解近期潜在危害。如全文所述，我们与所有受影响产品密切合作，确保披露前已部署缓解。对所有可能受影响方，即便未能精确验证攻击，也发送了本文预印本。

Long-term, we believe that openly discussing vulnerabilities that have practical impact is an important strategy for ensuring safe machine learning. This vulnerability exists whether or not we report on it. Especially for attacks that are simple to identify (as evidenced by the concurrent work of Finlayson et al. (2024) that discovered this same vulnerability), malicious actors are also likely to discover the same vulnerability whether or not we report on it. By documenting it early, we can ensure future systems remain secure.

长期看，我们认为公开讨论有实际影响的漏洞，是保障机器学习安全的重要策略。无论是否报告，漏洞都存在。尤其对易于发现的攻击（并发工作 Finlayson et al.（2024）发现了同一漏洞即是证据），恶意行动者无论我们是否报告都可能发现。早文档化，才能保证未来系统更安全。

Acknowledgements

致谢

We are grateful to Andreas Terzis and the anonymous reviewers for comments on early drafts of this paper. We are grateful to Joshua Achiam for helping to write the code for post-hoc modifying the model architecture. We are also grateful to OpenAI for allowing us to attempt our extraction attack on their production models.

感谢 Andreas Terzis 与匿名审稿人对本文初稿的意见。感谢 Joshua Achiam 协助编写事后修改模型架构的代码。亦感谢 OpenAI 允许我们在其生产模型上尝试抽取攻击。

References

参考文献

Anil et al. (2023) Anil, R. et al. PaLM 2 Technical Report, 2023.

Ba et al. (2016) Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Biderman (2024) Biderman, S. Common LLM settings, 2024. URL https://rb.gy/2afqlw . Accessed February 1, 2024.

Biderman et al. (2023) Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning , 2023.

Cancedda (2024) Cancedda, N. Spectral filters, dark signals, and attention sinks, 2024.

Carlini et al. (2020) Carlini, N., Jagielski, M., and Mironov, I. Cryptanalytic extraction of neural network models. In Annual International Cryptology Conference , 2020.

Chen et al. (2020) Chen, S., Carlini, N., and Wagner, D. Stateful detection of black-box adversarial attacks. In Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence , 2020.

Chiu (2024) Chiu, J. openlogprobs, 2024. URL https://github.com/justinchiu/openlogprobs . Accessed February 1, 2024.

Debenedetti et al. (2023) Debenedetti, E., Severi, G., Carlini, N., Choquette-Choo, C. A., Jagielski, M., Nasr, M., Wallace, E., and Tramèr, F. Privacy side channels in machine learning systems. arXiv preprint arXiv:2309.05610 , 2023.

Dettmers et al. (2022) Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit Optimizers via Block-wise Quantization. ICLR , 2022.

Elhage et al. (2021) Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. 2021. URL https://transformer-circuits.pub/2021/framework/index.html .

Feng et al. (2023) Feng, R., Hooda, A., Mangaokar, N., Fawaz, K., Jha, S., and Prakash, A. Stateful defenses for machine learning models are not yet secure against black-box attacks. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , pp. 786–800, 2023.

Finlayson et al. (2024) Finlayson, M., Swayamdipta, S., and Ren, X. Logits of api-protected llms leak proprietary information. arXiv preprint arXiv:2403.09539 , 2024.

Gao (2021) Gao, L. On the sizes of OpenAI API models. https://blog.eleuther.ai/gpt3-model-sizes/ , 2021.

Google (2024) Google. Changelog 1.38.0, 2024. URL https://cloud.google.com/python/docs/reference/aiplatform/1.38.0/changelog . Accessed January 30, 2024.

Gurnee et al. (2024) Gurnee, W., Horsley, T., Guo, Z. C., Kheirkhah, T. R., Sun, Q., Hathaway, W., Nanda, N., and Bertsimas, D. Universal neurons in gpt2 language models, 2024.

Hayase et al. (2024) Hayase, J., Borevkovic, E., Carlini, N., Tramèr, F., and Nasr, M. Query-based adversarial prompt generation. arXiv preprint arXiv:2402.12329 , 2024.

Jagielski et al. (2020) Jagielski, M., Carlini, N., Berthelot, D., Kurakin, A., and Papernot, N. High accuracy and high fidelity extraction of neural networks. In USENIX Security Symposium , 2020.

Jiang et al. (2023) Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. In EMNLP , 2023.

Juuti et al. (2019) Juuti, M., Szyller, S., Marchal, S., and Asokan, N. PRADA: protecting against DNN model stealing attacks. In EuroS&P , 2019.

Lee et al. (2022) Lee, K.-H., Nachum, O., Yang, M. S., Lee, L., Freeman, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., and Mordatch, I. Multi-game decision transformers. In Advances in Neural Information Processing Systems , 2022.

Liu et al. (2024) Liu, A., Han, X., Wang, Y., Tsvetkov, Y., Choi, Y., and Smith, N. A. Tuning language models by proxy. arXiv preprint arXiv:2401.08565 , 2024.

Milli et al. (2019) Milli, S., Schmidt, L., Dragan, A. D., and Hardt, M. Model reconstruction from model explanations. In Proceedings of the Conference on Fairness, Accountability, and Transparency , 2019.

Mitchell et al. (2024) Mitchell, E., Rafailov, R., Sharma, A., Finn, C., and Manning, C. D. An emulator for fine-tuning large language models using small language models. In ICLR , 2024.

Morris et al. (2023) Morris, J. X., Zhao, W., Chiu, J. T., Shmatikov, V., and Rush, A. M. Language model inversion. arXiv preprint arXiv:2311.13647 , 2023.

OpenAI (2023) OpenAI. Using logit bias to define token probability, 2023. URL https://help.openai.com/en/articles/5247780-using-logit-bias-to-define-token-probability . Accessed Febraury 1, 2024.

OpenAI (2024) OpenAI. Create chat completion, 2024. URL https://platform.openai.com/docs/api-reference/chat/create . Accessed January 30, 2024.

OpenAI et al. (2023) OpenAI et al. GPT-4 Technical Report, 2023.

Pal et al. (2021) Pal, S., Gupta, Y., Kanade, A., and Shevade, S. Stateful detection of model extraction attacks. arXiv preprint arXiv:2107.05166 , 2021.

Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language Models are Unsupervised Multitask Learners. Technical report, OpenAI, 2019. URL https://rb.gy/tm8qh .

Rae et al. (2022) Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.-B., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., and Irving, G. Scaling language models: Methods, analysis and insights from training gopher, 2022.

Ren et al. (2023) Ren, J., Zhao, Y., Vu, T., Liu, P. J., and Lakshminarayanan, B. Self-evaluation improves selective generation in large language models. arXiv preprint arXiv:2312.09300 , 2023.

Rolnick & Kording (2020) Rolnick, D. and Kording, K. Reverse-engineering deep relu networks. In International Conference on Machine Learning , 2020.

Shamir et al. (2023) Shamir, A., Canales-Martinez, I., Hambitzer, A., Chavez-Saab, J., Rodrigez-Henriquez, F., and Satpute, N. Polynomial time cryptanalytic extraction of neural network models. arXiv preprint arXiv:2310.08708 , 2023.

Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Tramèr et al. (2016) Tramèr, F., Zhang, F., Juels, A., Reiter, M. K., and Ristenpart, T. Stealing machine learning models via prediction APIs. In USENIX Security Symposium , 2016.

Veit et al. (2016) Veit, A., Wilber, M. J., and Belongie, S. J. Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems , pp. 550–558, 2016.

Wei et al. (2020) Wei, J., Zhang, Y., Zhou, Z., Li, Z., and Al Faruque, M. A. Leaky DNN: Stealing deep-learning model secret with GPU context-switching side-channel. In IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) , 2020.

Yang & Klein (2021) Yang, K. and Klein, D. FUDGE: Controlled text generation with future discriminators. In Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), ACL , 2021.

Zanella-Beguelin et al. (2021) Zanella-Beguelin, S., Tople, S., Paverd, A., and Köpf, B. Grey-box extraction of natural language models. In International Conference on Machine Learning , pp. 12278–12286. PMLR, 2021.

Zhang & Sennrich (2019) Zhang, B. and Sennrich, R. Root mean square layer normalization. NeurIPS , 2019.

Appendix A What’s Going On With GPT-2 Small?

附录 A GPT-2 Small 出了什么事？

Our attack nearly perfectly extracts the model size of all models—except for GPT-2 Small where our extracted size of 757 is off by 11 from the correct 768. Why is this?

除 GPT-2 Small 抽出规模 757、与正确的 768 差 11 外，攻击对其余模型几乎完美抽出规模。为什么？

In Figure A we directly inspect this model’s final hidden activation vector across $10,000$ different model queries and perform SVD of the resulting activation matrix. We see that despite GPT-2 actually having 768 potential hidden neurons, there are only $757$ different activation directions. Thus, while this model is technically a 768 dimensional model, in practice it behaves as if it was a 757 (i.e, the rank of the embedding matrix is 757) dimensional model, and our attack has recovered this effective size.

图 A 中我们直接检查该模型在 $10{,}000$ 次不同查询上的最终隐藏激活向量，并对激活矩阵做 SVD。可见尽管 GPT-2 名义上有 768 个潜在隐藏神经元，激活方向却只有 $757$ 个。因此该模型技术上是 768 维，实践中却像 757 维（即嵌入矩阵秩为 757），攻击恢复的正是这一有效规模。

However, when running the model in higher float64 precision, we find that indeed all dimensions are used, but that the smallest dozen or so singular values are much smaller than the other singular values, an observation made by concurrent work (Cancedda, 2024).

但在更高精度 float64 下运行时，发现所有维确实都在用，只是最小约十几个奇异值远小于其余奇异值；并发工作（Cancedda, 2024）也观察到这一点。

$\alpha_{i}\leftarrow-B,\beta_{i}\leftarrow 0$
while $\beta_{i}-\alpha_{i}>\varepsilon$ do
if $\mathcal{O}\left({p,b=\{i:-\frac{\alpha_{i}+\beta_{i}}{2}\}}\right)=0$ then
$\beta_{i}\leftarrow\frac{\alpha_{i}+\beta_{i}}{2}$
else
$\alpha_{i}\leftarrow\frac{\alpha_{i}+\beta_{i}}{2}$
end if
Return $\frac{\alpha_{i}+\beta_{i}}{2}$
end while

$\alpha_{i}\leftarrow-B,\beta_{i}\leftarrow 0\quad\forall i=1,\ldots,N$
$\mathcal{C}=\{\mathsf{logit}:\mathsf{logit}_{i}-\mathsf{logit}_{0}\leq B\quad\forall i=1,\ldots,N\}$
for $T$ rounds do
$b_{i}\leftarrow-\frac{\alpha_{i}+\beta_{i}}{2}$ for $i=0,\ldots,N$
$k\leftarrow\mathcal{O}(p,b=\{{0:b_{0},1:b_{1},\dots,N:b_{N}}\})$
for $j\neq k$ do
$\mathcal{C}\leftarrow\mathcal{C}\cap\{\mathsf{logit}:\mathsf{logit}_{k}+b_{k}\geq\mathsf{logit}_{j}+b_{j}\}$
end for
for $i=0,\ldots,N$ do
$\alpha_{i}\leftarrow\displaystyle\min_{\mathsf{logit}\in\mathcal{C}}\mathsf{logit}_{i}-\mathsf{logit}_{0}$
$\beta_{i}\leftarrow\displaystyle\max_{\mathsf{logit}\in\mathcal{C}}\mathsf{logit}_{i}-\mathsf{logit}_{0}$
end for
end for
Return $[\alpha_{i},\beta_{i}]\quad\forall i\in\{0,\ldots,N\}$

0 .