LoRA: Low-Rank Adaptation of Large Language Models

Edward Hu Thanks:  Equal contribution.    Yelong ShenPhillip Wallis Zeyuan Allen-ZhuYuanzhi LiShean WangLu WangWeizhu Chen Microsoft Corporation {edwardhu, yeshe, phwallis, zeyuana,yuanzhil, swang, luw, wzchen}@microsoft.comyuanzhil@andrew.cmu.edu (Version 2)

Abstract

An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example – deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

自然语言处理的一个重要范式是：在通用领域数据上大规模预训练，再适配到特定任务或领域。随着我们预训练更大的模型，重新训练全部模型参数的全量微调变得越来越不可行。以 GPT-3 175B 为例——部署各自拥有 175B 参数的独立微调实例，代价高得难以承受。我们提出 Low-Rank Adaptation（LoRA）：冻结预训练模型权重，并向 Transformer 架构的每一层注入可训练的秩分解矩阵，从而大幅减少下游任务的可训练参数数量。相对用 Adam 微调的 GPT-3 175B，LoRA 可将可训练参数数减少约 10,000 倍，GPU 内存需求减少约 3 倍。尽管可训练参数更少、训练吞吐更高，且与 adapter 不同、不引入额外推理延迟，LoRA 在 RoBERTa、DeBERTa、GPT-2 与 GPT-3 上的模型质量仍与全量微调持平或更好。我们还对语言模型适配中的秩亏缺做了实证考察，以阐明 LoRA 有效性并对未来应用给出启示。我们发布便于把 LoRA 集成到 PyTorch 模型的软件包，并在 https://github.com/microsoft/LoRA 提供 RoBERTa、DeBERTa 与 GPT-2 的实现与模型检查点。

1 Introduction

Figure 1: Our reparametrization. We only train $A$ and $B$.

图 1：我们的重参数化。只训练 $A$ 与 $B$。

Many applications in natural language processing rely on adapting one large-scale, pre-trained language model to multiple downstream applications. Such adaptation is usually done via fine-tuning, which updates all the parameters of the pre-trained model. The major downside of fine-tuning is that the new model contains as many parameters as in the original model. As larger models are trained every few months, this changes from a mere “inconvenience” for GPT-2 (Radford et al. b) or RoBERTa large (Liu et al. 2019) to a critical deployment challenge for GPT-3 (Brown et al. 2020) with 175 billion trainable parameters.[^1]

自然语言处理的许多应用依赖把一个大规模预训练语言模型适配到多个下游应用。这种适配通常通过微调完成，即更新预训练模型的全部参数。微调的主要缺点是：新模型包含与原模型一样多的参数。随着每隔几个月就训练更大的模型，这对 GPT-2（Radford et al. b）或 RoBERTa large（Liu et al. 2019）还只是“不便”，对拥有 1750 亿可训练参数的 GPT-3（Brown et al. 2020）则变成关键部署挑战。[^1]

Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks. This way, we only need to store and load a small number of task-specific parameters in addition to the pre-trained model for each task, greatly boosting the operational efficiency when deployed. However, existing techniques often introduce inference latency (Houlsby et al. 2019; Rebuffi et al. 2017) by extending model depth or reduce the model’s usable sequence length (Li & Liang 2021; Lester et al. 2021; Hambardzumyan et al. 2020; Liu et al. 2021) (Section 3). More importantly, these method often fail to match the fine-tuning baselines, posing a trade-off between efficiency and model quality.

许多人试图只适配部分参数，或为新任务学习外部模块来缓解这一问题。这样，对每个任务我们只需在预训练模型之外存储并加载少量任务特定参数，从而在部署时大幅提升运维效率。但现有技术往往通过加深模型引入推理延迟（Houlsby et al. 2019; Rebuffi et al. 2017），或缩短模型可用序列长度（Li & Liang 2021; Lester et al. 2021; Hambardzumyan et al. 2020; Liu et al. 2021）（第 3 节）。更重要的是，这些方法常常无法匹配微调基线，从而在效率与模型质量之间形成权衡。

We take inspiration from Li et al. 2018a; Aghajanyan et al. 2020 which show that the learned over-parametrized models in fact reside on a low intrinsic dimension. We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation (LoRA) approach. LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen, as shown in Figure 1. Using GPT-3 175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288, making LoRA both storage- and compute-efficient.

我们受 Li et al. 2018a; Aghajanyan et al. 2020 启发：学得的过参数化模型实际上落在低本征维上。我们假设模型适配期间的权重变化也具有低“本征秩”，由此提出 Low-Rank Adaptation（LoRA）。如图 1 所示，LoRA 允许我们通过优化稠密层在适配期间变化量的秩分解矩阵，间接训练神经网络中的某些稠密层，同时保持预训练权重冻结。以 GPT-3 175B 为例，我们表明即便全秩（即 d）高达 12,288，极低的秩（即图 1 中的 r 可为 1 或 2）也已足够，从而使 LoRA 在存储与计算上都高效。

LoRA possesses several key advantages.

LoRA 具有若干关键优势。

• • A pre-trained model can be shared and used to build many small LoRA modules for different tasks. We can freeze the shared model and efficiently switch tasks by replacing the matrices $A$ and $B$ in Figure 1, reducing the storage requirement and task-switching overhead significantly.

• • 一个预训练模型可被共享，并为不同任务构建许多小型 LoRA 模块。我们可以冻结共享模型，通过替换图 1 中的矩阵 $A$ 与 $B$ 高效切换任务，显著降低存储需求与任务切换开销。

• • LoRA makes training more efficient and lowers the hardware barrier to entry by up to 3 times when using adaptive optimizers since we do not need to calculate the gradients or maintain the optimizer states for most parameters. Instead, we only optimize the injected, much smaller low-rank matrices.

• • LoRA 使训练更高效，并在使用自适应优化器时将硬件门槛最高降低约 3 倍：因为我们不必为大多数参数计算梯度或维护优化器状态，而只需优化注入的、小得多的低秩矩阵。

• • Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to a fully fine-tuned model, by construction.

• • 我们的简单线性设计允许在部署时把可训练矩阵与冻结权重合并，相对全量微调模型不引入推理延迟，这由构造保证。

• • LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning. We provide an example in Appendix E.

• • LoRA 与许多先前方法正交，并可与其中许多结合，例如 prefix-tuning。附录 E 给出一个例子。

Terminologies and Conventions

We make frequent references to the Transformer architecture and use the conventional terminologies for its dimensions. We call the input and output dimension size of a Transformer layer $d_{model}$. We use $W_{q}$, $W_{k}$, $W_{v}$, and $W_{o}$ to refer to the query/key/value/output projection matrices in the self-attention module. $W$ or $W_{0}$ refers to a pre-trained weight matrix and $\Delta W$ its accumulated gradient update during adaptation. We use $r$ to denote the rank of a LoRA module. We follow the conventions set out by (Vaswani et al. 2017; Brown et al. 2020) and use Adam (Loshchilov & Hutter 2019; Kingma & Ba 2017) for model optimization and use a Transformer MLP feedforward dimension $d_{ffn}=4\times d_{model}$.

Terminologies and Conventions

我们频繁引用 Transformer 架构，并对其维度使用常规术语。我们把 Transformer 层的输入与输出维数称为 $d_{model}$。用 $W_{q}$、$W_{k}$、$W_{v}$ 与 $W_{o}$ 指自注意力模块中的 query/key/value/output 投影矩阵。$W$ 或 $W_{0}$ 指预训练权重矩阵，$\Delta W$ 指其在适配期间累积的梯度更新。用 $r$ 表示 LoRA 模块的秩。我们遵循（Vaswani et al. 2017; Brown et al. 2020）的惯例，用 Adam（Loshchilov & Hutter 2019; Kingma & Ba 2017）做模型优化，并使用 Transformer MLP 前馈维 $d_{ffn}=4\times d_{model}$。

2 Problem Statement

While our proposal is agnostic to training objective, we focus on language modeling as our motivating use case. Below is a brief description of the language modeling problem and, in particular, the maximization of conditional probabilities given a task-specific prompt.

虽然我们的提议与训练目标无关，但我们以语言建模作为动机用例。以下简要描述语言建模问题，特别是给定任务特定提示时条件概率的最大化。

Suppose we are given a pre-trained autoregressive language model $P_{\Phi}(y|x)$ parametrized by $\Phi$. For instance, $P_{\Phi}(y|x)$ can be a generic multi-task learner such as GPT (Radford et al. b; Brown et al. 2020) based on the Transformer architecture (Vaswani et al. 2017). Consider adapting this pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language to SQL (NL2SQL). Each downstream task is represented by a training dataset of context-target pairs: $\mathcal{Z}=\{(x_{i},y_{i})\}_{i=1,..,N}$, where both $x_{i}$ and $y_{i}$ are sequences of tokens. For example, in NL2SQL, $x_{i}$ is a natural language query and $y_{i}$ its corresponding SQL command; for summarization, $x_{i}$ is the content of an article and $y_{i}$ its summary.

假设给定由 $\Phi$ 参数化的预训练自回归语言模型 $P_{\Phi}(y|x)$。例如，$P_{\Phi}(y|x)$ 可以是基于 Transformer 架构（Vaswani et al. 2017）的通用多任务学习器，如 GPT（Radford et al. b; Brown et al. 2020）。考虑把该预训练模型适配到下游条件文本生成任务，例如摘要、机器阅读理解（MRC）以及自然语言到 SQL（NL2SQL）。每个下游任务由上下文–目标对的训练数据集表示：$\mathcal{Z}=\{(x_{i},y_{i})\}_{i=1,..,N}$，其中 $x_{i}$ 与 $y_{i}$ 都是 token 序列。例如在 NL2SQL 中，$x_{i}$ 是自然语言查询，$y_{i}$ 是对应 SQL 命令；对摘要，$x_{i}$ 是文章内容，$y_{i}$ 是其摘要。

During full fine-tuning, the model is initialized to pre-trained weights $\Phi_{0}$ and updated to $\Phi_{0}+\Delta\Phi$ by repeatedly following the gradient to maximize the conditional language modeling objective:

在全量微调中，模型初始化为预训练权重 $\Phi_{0}$，并通过反复沿梯度上升最大化条件语言建模目标，更新为 $\Phi_{0}+\Delta\Phi$：

$$\displaystyle\max_{\Phi}\sum_{(x,y)\in\mathcal{Z}}\sum_{t=1}^{|y|}\text{log}\left(P_{\Phi}(y_{t}|x,y_{<t})\right) \tag{1}$$

One of the main drawbacks for full fine-tuning is that for each downstream task, we learn a different set of parameters $\Delta\Phi$ whose dimension $|\Delta\Phi|$ equals $|\Phi_{0}|$. Thus, if the pre-trained model is large (such as GPT-3 with $|\Phi_{0}|\approx 175\text{~Billion}$), storing and deploying many independent instances of fine-tuned models can be challenging, if at all feasible.

全量微调的主要缺点之一是：对每个下游任务，我们学到一组不同的参数 $\Delta\Phi$，其维数 $|\Delta\Phi|$ 等于 $|\Phi_{0}|$。因此，若预训练模型很大（如 GPT-3，$|\Phi_{0}|\approx 175\text{~Billion}$），存储并部署许多独立微调实例即便可行也极具挑战。

In this paper, we adopt a more parameter-efficient approach, where the task-specific parameter increment $\Delta\Phi=\Delta\Phi(\Theta)$ is further encoded by a much smaller-sized set of parameters $\Theta$ with $|\Theta|\ll|\Phi_{0}|$. The task of finding $\Delta\Phi$ thus becomes optimizing over $\Theta$:

本文采用更参数高效的方法：任务特定参数增量 $\Delta\Phi=\Delta\Phi(\Theta)$ 进一步由规模小得多的参数集 $\Theta$ 编码，且 $|\Theta|\ll|\Phi_{0}|$。于是寻找 $\Delta\Phi$ 变为对 $\Theta$ 优化：

$$\displaystyle\max_{\Theta}\sum_{(x,y)\in\mathcal{Z}}\sum_{t=1}^{|y|}\log\left({p_{\Phi_{0}+\Delta\Phi(\Theta)}(y_{t}|x,y_{<t}})\right) \tag{2}$$

In the subsequent sections, we propose to use a low-rank representation to encode $\Delta\Phi$ that is both compute- and memory-efficient. When the pre-trained model is GPT-3 175B, the number of trainable parameters $|\Theta|$ can be as small as $0.01\%$ of $|\Phi_{0}|$.

后续章节中，我们提出用低秩表示编码 $\Delta\Phi$，使其在计算与内存上都高效。当预训练模型为 GPT-3 175B 时，可训练参数数 $|\Theta|$ 可小至 $|\Phi_{0}|$ 的 $0.01\%$。

3 Aren’t Existing Solutions Good Enough?

The problem we set out to tackle is by no means new. Since the inception of transfer learning, dozens of works have sought to make model adaptation more parameter- and compute-efficient. See Section 6 for a survey of some of the well-known works. Using language modeling as an example, there are two prominent strategies when it comes to efficient adaptations: adding adapter layers (Houlsby et al. 2019; Rebuffi et al. 2017; Pfeiffer et al. 2021; Rücklé et al. 2020) or optimizing some forms of the input layer activations (Li & Liang 2021; Lester et al. 2021; Hambardzumyan et al. 2020; Liu et al. 2021). However, both strategies have their limitations, especially in a large-scale and latency-sensitive production scenario.

我们着手解决的问题绝非全新。自迁移学习诞生以来，已有数十项工作试图使模型适配更参数高效与计算高效。第 6 节综述其中一些知名工作。以语言建模为例，高效适配有两条突出策略：加入 adapter 层（Houlsby et al. 2019; Rebuffi et al. 2017; Pfeiffer et al. 2021; Rücklé et al. 2020），或优化输入层激活的某些形式（Li & Liang 2021; Lester et al. 2021; Hambardzumyan et al. 2020; Liu et al. 2021）。但两条策略都有局限，尤其在大规模、对延迟敏感的生产场景中。

Adapter Layers Introduce Inference Latency

There are many variants of adapters. We focus on the original design by Houlsby et al. 2019 which has two adapter layers per Transformer block and a more recent one by  Lin et al. 2020 which has only one per block but with an additional LayerNorm (Ba et al. 2016). While one can reduce the overall latency by pruning layers or exploiting multi-task settings (Rücklé et al. 2020; Pfeiffer et al. 2021), there is no direct ways to bypass the extra compute in adapter layers. This seems like a non-issue since adapter layers are designed to have few parameters (sometimes $<$1% of the original model) by having a small bottleneck dimension, which limits the FLOPs they can add. However, large neural networks rely on hardware parallelism to keep the latency low, and adapter layers have to be processed sequentially. This makes a difference in the online inference setting where the batch size is typically as small as one. In a generic scenario without model parallelism, such as running inference on GPT-2 (Radford et al. b) medium on a single GPU, we see a noticeable increase in latency when using adapters, even with a very small bottleneck dimension (Table 1).

Adapter Layers Introduce Inference Latency

Adapter 有许多变体。我们聚焦 Houlsby et al. 2019 的原始设计（每个 Transformer 块两个 adapter 层），以及 Lin et al. 2020 更近的设计（每块仅一个，但额外有 LayerNorm（Ba et al. 2016））。虽然可通过剪枝层或利用多任务设定降低总体延迟（Rücklé et al. 2020; Pfeiffer et al. 2021），但没有直接办法绕过 adapter 层中的额外计算。这看似不是问题，因为 adapter 层通过很小的瓶颈维设计成参数很少（有时 $<$1% 原模型），从而限制它们能增加的 FLOPs。但大型神经网络依赖硬件并行保持低延迟，而 adapter 层必须顺序处理。这在 batch size 通常小到 1 的在线推理设定中会有差别。在无模型并行的一般场景中，例如在单 GPU 上对 GPT-2（Radford et al. b）medium 做推理，即便瓶颈维很小，使用 adapter 也会带来可察觉的延迟增加。

Batch Size	32	16	1
Sequence Length	512	256	128
$|\Theta|$	0.5M	11M	11M
Fine-Tune/LoRA	1449.4$\pm$0.8	338.0$\pm$0.6	19.8$\pm$2.7
$\text{Adapter}^{\text{L}}$	1482.0$\pm$1.0 (+2.2%)	354.8$\pm$0.5 (+5.0%)	23.9$\pm$2.1 (+20.7%)
$\text{Adapter}^{\text{H}}$	1492.2$\pm$1.0 (+3.0%)	366.3$\pm$0.5 (+8.4%)	25.8$\pm$2.2 (+30.3%)

Table 1: Infernece latency of a single forward pass in GPT-2 medium measured in milliseconds, averaged over 100 trials. We use an NVIDIA Quadro RTX8000. “$|\Theta|$” denotes the number of trainable parameters in adapter layers. $\text{Adapter}^{\text{L}}$ and $\text{Adapter}^{\text{H}}$ are two variants of adapter tuning, which we describe in Section 5.1. The inference latency introduced by adapter layers can be significant in an online, short-sequence-length scenario. See the full study in Appendix B.

Batch Size	32	16	1
Sequence Length	512	256	128
$|\Theta|$	0.5M	11M	11M
Fine-Tune/LoRA	1449.4$\pm$0.8	338.0$\pm$0.6	19.8$\pm$2.7
$\text{Adapter}^{\text{L}}$	1482.0$\pm$1.0 (+2.2%)	354.8$\pm$0.5 (+5.0%)	23.9$\pm$2.1 (+20.7%)
$\text{Adapter}^{\text{H}}$	1492.2$\pm$1.0 (+3.0%)	366.3$\pm$0.5 (+8.4%)	25.8$\pm$2.2 (+30.3%)

表 1：GPT-2 medium 单次前向的推理延迟（毫秒），100 次试验平均。使用 NVIDIA Quadro RTX8000。“$|\Theta|$”表示 adapter 层中可训练参数数。$\text{Adapter}^{\text{L}}$ 与 $\text{Adapter}^{\text{H}}$ 是第 5.1 节描述的两种 adapter tuning 变体。在在线、短序列场景中，adapter 层引入的推理延迟可能显著。完整研究见附录 B。

This problem gets worse when we need to shard the model as done in Shoeybi et al. 2020; Lepikhin et al. 2020, because the additional depth requires more synchronous GPU operations such as AllReduce and Broadcast, unless we store the adapter parameters redundantly many times.

当我们需要像 Shoeybi et al. 2020; Lepikhin et al. 2020 那样对模型做分片时，问题更糟：额外深度需要更多同步 GPU 操作（如 AllReduce 与 Broadcast），除非我们把 adapter 参数冗余存储许多份。

Directly Optimizing the Prompt is Hard

The other direction, as exemplified by prefix tuning (Li & Liang 2021), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods. We defer the study on task performance to Section 5.

Directly Optimizing the Prompt is Hard

另一方向以 prefix tuning（Li & Liang 2021）为代表，面临不同挑战。我们观察到 prefix tuning 难以优化，且其表现随可训练参数非单调变化，这与原论文的类似观察一致。更根本的是，为适配预留一部分序列长度必然减少可用于处理下游任务的序列长度，我们怀疑这使得调 prompt 相对其他方法表现更弱。任务表现研究留到第 5 节。

4 Our Method

We describe the simple design of LoRA and its practical benefits. The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in our experiments as the motivating use case.

我们描述 LoRA 的简单设计及其实际收益。这里概述的原则适用于深度学习模型中的任意稠密层，尽管实验中我们只聚焦 Transformer 语言模型中的某些权重作为动机用例。

4.1 Low-Rank-Parametrized Update Matrices

A neural network contains many dense layers which perform matrix multiplication. The weight matrices in these layers typically have full-rank. When adapting to a specific task, Aghajanyan et al. 2020 shows that the pre-trained language models have a low “instrisic dimension” and can still learn efficiently despite a random projection to a smaller subspace. Inspired by this, we hypothesize the updates to the weights also have a low “intrinsic rank” during adaptation. For a pre-trained weight matrix $W_{0}\in\mathbb{R}^{d\times k}$, we constrain its update by representing the latter with a low-rank decomposition $W_{0}+\Delta W=W_{0}+BA$, where $B\in\mathbb{R}^{d\times r},A\in\mathbb{R}^{r\times k}$, and the rank $r\ll\min(d,k)$. During training, $W_{0}$ is frozen and does not receive gradient updates, while $A$ and $B$ contain trainable parameters. Note both $W_{0}$ and $\Delta W=BA$ are multiplied with the same input, and their respective output vectors are summed coordinate-wise. For $h=W_{0}x$, our modified forward pass yields:

神经网络包含许多执行矩阵乘法的稠密层。这些层中的权重矩阵通常满秩。适配到特定任务时，Aghajanyan et al. 2020 表明预训练语言模型具有低“本征维”，即便随机投影到更小子空间仍能高效学习。受此启发，我们假设适配期间权重更新也具有低“本征秩”。对预训练权重矩阵 $W_{0}\in\mathbb{R}^{d\times k}$，我们通过低秩分解约束其更新：$W_{0}+\Delta W=W_{0}+BA$，其中 $B\in\mathbb{R}^{d\times r}$，$A\in\mathbb{R}^{r\times k}$，且秩 $r\ll\min(d,k)$。训练时 $W_{0}$ 冻结、不接收梯度更新，而 $A$ 与 $B$ 含可训练参数。注意 $W_{0}$ 与 $\Delta W=BA$ 都与同一输入相乘，各自输出向量按坐标相加。对 $h=W_{0}x$，我们修改后的前向为：

$$h=W_{0}x+\Delta Wx=W_{0}x+BAx \tag{3}$$

We illustrate our reparametrization in Figure 1. We use a random Gaussian initialization for $A$ and zero for $B$, so $\Delta W=BA$ is zero at the beginning of training. We then scale $\Delta Wx$ by $\frac{\alpha}{r}$, where $\alpha$ is a constant in $r$. When optimizing with Adam, tuning $\alpha$ is roughly the same as tuning the learning rate if we scale the initialization appropriately. As a result, we simply set $\alpha$ to the first $r$ we try and do not tune it. This scaling helps to reduce the need to retune hyperparameters when we vary $r$ (Yang & Hu 2021).

我们在图 1 中说明重参数化。对 $A$ 使用随机高斯初始化，对 $B$ 置零，因而训练开始时 $\Delta W=BA$ 为零。随后把 $\Delta Wx$ 按 $\frac{\alpha}{r}$ 缩放，其中 $\alpha$ 是相对 $r$ 的常数。用 Adam 优化时，若适当缩放初始化，调节 $\alpha$ 大致等同于调节学习率。因此我们简单把 $\alpha$ 设为首次尝试的 $r$ 且不再调它。该缩放有助于在改变 $r$ 时减少重新调节超参的需要（Yang & Hu 2021）。

A Generalization of Full Fine-tuning.  A more general form of fine-tuning allows the training of a subset of the pre-trained parameters. LoRA takes a step further and does not require the accumulated gradient update to weight matrices to have full-rank during adaptation. This means that when applying LoRA to all weight matrices and training all biases[^2], we roughly recover the expressiveness of full fine-tuning by setting the LoRA rank $r$ to the rank of the pre-trained weight matrices. In other words, as we increase the number of trainable parameters [^3], training LoRA roughly converges to training the original model, while adapter-based methods converges to an MLP and prefix-based methods to a model that cannot take long input sequences.

A Generalization of Full Fine-tuning.　更一般的微调形式允许训练预训练参数的一个子集。LoRA 更进一步，不要求适配期间权重矩阵的累积梯度更新满秩。这意味着：当对所有权重矩阵应用 LoRA 并训练所有偏置[^2] 时，通过把 LoRA 秩 $r$ 设为预训练权重矩阵的秩，我们大致恢复全量微调的表达能力。换言之，随着可训练参数数增加[^3]，训练 LoRA 大致收敛到训练原模型；而基于 adapter 的方法收敛到一个 MLP，基于 prefix 的方法收敛到无法接受长输入序列的模型。

No Additional Inference Latency.  When deployed in production, we can explicitly compute and store $W=W_{0}+BA$ and perform inference as usual. Note that both $W_{0}$ and $BA$ are in $\mathbb{R}^{d\times k}$. When we need to switch to another downstream task, we can recover $W_{0}$ by subtracting $BA$ and then adding a different $B^{\prime}A^{\prime}$, a quick operation with very little memory overhead. Critically, this guarantees that we do not introduce any additional latency during inference compared to a fine-tuned model by construction.

No Additional Inference Latency.　在生产部署时，我们可以显式计算并存储 $W=W_{0}+BA$，并照常推理。注意 $W_{0}$ 与 $BA$ 都在 $\mathbb{R}^{d\times k}$ 中。需要切换到另一下游任务时，可通过减去 $BA$ 再加不同的 $B^{\prime}A^{\prime}$ 恢复 $W_{0}$，这是内存开销很小的快速操作。关键的是，这由构造保证：相对微调模型，我们在推理时不引入任何额外延迟。

4.2 Applying LoRA to Transformer

In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the number of trainable parameters. In the Transformer architecture, there are four weight matrices in the self-attention module ($W_{q},W_{k},W_{v},W_{o}$) and two in the MLP module. We treat $W_{q}$ (or $W_{k}$, $W_{v}$) as a single matrix of dimension $d_{model}\times d_{model}$, even though the output dimension is usually sliced into attention heads. We limit our study to only adapting the attention weights for downstream tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity and parameter-efficiency.We further study the effect on adapting different types of attention weight matrices in a Transformer in Section 7.1. We leave the empirical investigation of adapting the MLP layers, LayerNorm layers, and biases to a future work.

原则上，我们可对神经网络中任意权重矩阵子集应用 LoRA 以减少可训练参数数。在 Transformer 架构中，自注意力模块有四个权重矩阵（$W_{q},W_{k},W_{v},W_{o}$），MLP 模块有两个。即便输出维通常切成注意力头，我们仍把 $W_{q}$（或 $W_{k}$、$W_{v}$）视为维数为 $d_{model}\times d_{model}$ 的单个矩阵。为简洁与参数效率，我们把研究限制为只适配注意力权重用于下游任务，并冻结 MLP 模块（因而下游任务中不训练它们）。我们在第 7.1 节进一步研究所适配的注意力权重矩阵类型的影响。把 MLP 层、LayerNorm 层与偏置也纳入适配的实证考察留待未来工作。

Practical Benefits and Limitations.  The most significant benefit comes from the reduction in memory and storage usage. For a large Transformer trained with Adam, we reduce that VRAM usage by up to $2/3$ if $r\ll d_{model}$ as we do not need to store the optimizer states for the frozen parameters. On GPT-3 175B, we reduce the VRAM consumption during training from 1.2TB to 350GB. With $r=4$ and only the query and value projection matrices being adapted, the checkpoint size is reduced by roughly 10,000$\times$ (from 350GB to 35MB)[^4]. This allows us to train with significantly fewer GPUs and avoid I/O bottlenecks. Another benefit is that we can switch between tasks while deployed at a much lower cost by only swapping the LoRA weights as opposed to all the parameters. This allows for the creation of many customized models that can be swapped in and out on the fly on machines that store the pre-trained weights in VRAM. We also observe a 25% speedup during training on GPT-3 175B compared to full fine-tuning[^5] as we do not need to calculate the gradient for the vast majority of the parameters.

Practical Benefits and Limitations.　最显著收益来自内存与存储占用的降低。对用 Adam 训练的大型 Transformer，若 $r\ll d_{model}$，由于不必为冻结参数存储优化器状态，我们最高可将 VRAM 占用降低约 $2/3$。在 GPT-3 175B 上，训练时 VRAM 消耗从 1.2TB 降到 350GB。以 $r=4$ 且只适配 query 与 value 投影矩阵时，检查点大小约缩小 10,000$\times$（从 350GB 到 35MB）[^4]。这使我们能用显著更少的 GPU 训练并避免 I/O 瓶颈。另一收益是：部署时可仅交换 LoRA 权重而非全部参数，从而以低得多的代价在任务间切换。这允许创建许多定制模型，并在把预训练权重存于 VRAM 的机器上即时换入换出。我们还观察到：相对全量微调，在 GPT-3 175B 上训练加速约 25%[^5]，因为不必为绝大多数参数计算梯度。

LoRA also has its limitations. For example, it is not straightforward to batch inputs to different tasks with different $A$ and $B$ in a single forward pass, if one chooses to absorb $A$ and $B$ into $W$ to eliminate additional inference latency. Though it is possible to not merge the weights and dynamically choose the LoRA modules to use for samples in a batch for scenarios where latency is not critical.

LoRA 也有局限。例如，若选择把 $A$ 与 $B$ 吸收进 $W$ 以消除额外推理延迟，则不易在单次前向中把使用不同 $A$、$B$ 的不同任务输入批在一起。不过在延迟不关键的场景中，也可以不合并权重，并为 batch 中的样本动态选择所用的 LoRA 模块。

5 Empirical Experiments

We evaluate the downstream task performance of LoRA on RoBERTa (Liu et al. 2019), DeBERTa (He et al. 2021), and GPT-2 (Radford et al. b), before scaling up to GPT-3 175B (Brown et al. 2020). Our experiments cover a wide range of tasks, from natural language understanding (NLU) to generation (NLG). Specifically, we evaluate on the GLUE (Wang et al. 2019) benchmark for RoBERTa and DeBERTa. We follow the setup of Li & Liang 2021 on GPT-2 for a direct comparison and add WikiSQL (Zhong et al. 2017) (NL to SQL queries) and SAMSum (Gliwa et al. 2019) (conversation summarization) for large-scale experiments on GPT-3. See Appendix C for more details on the datasets we use. We use NVIDIA Tesla V100 for all experiments.

我们在 RoBERTa（Liu et al. 2019）、DeBERTa（He et al. 2021）与 GPT-2（Radford et al. b）上评估 LoRA 的下游任务表现，再扩展到 GPT-3 175B（Brown et al. 2020）。实验覆盖从自然语言理解（NLU）到生成（NLG）的广泛任务。具体地，对 RoBERTa 与 DeBERTa 我们在 GLUE（Wang et al. 2019）基准上评估。对 GPT-2 我们遵循 Li & Liang 2021 的设定以便直接比较，并为 GPT-3 上的大规模实验加入 WikiSQL（Zhong et al. 2017）（自然语言到 SQL）与 SAMSum（Gliwa et al. 2019）（对话摘要）。所用数据集更多细节见附录 C。全部实验使用 NVIDIA Tesla V100。

5.1 Baselines

To compare with other baselines broadly, we replicate the setups used by prior work and reuse their reported numbers whenever possible. This, however, means that some baselines might only appear in certain experiments.

为与其他基线广泛比较，我们复现先前工作使用的设定，并尽可能复用其报告数字。但这意味着某些基线可能只出现在部分实验中。

Fine-Tuning (FT) is a common approach for adaptation. During fine-tuning, the model is initialized to the pre-trained weights and biases, and all model parameters undergo gradient updates.A simple variant is to update only some layers while freezing others. We include one such baseline reported in prior work (Li & Liang 2021) on GPT-2, which adapts just the last two layers ($\textbf{FT}^{\textbf{Top2}}$).

Fine-Tuning (FT) 是常见的适配方法。微调时，模型初始化为预训练权重与偏置，全部模型参数接受梯度更新。简单变体是只更新某些层、冻结其余。我们纳入先前工作（Li & Liang 2021）在 GPT-2 上报告的一种基线：只适配最后两层（$\textbf{FT}^{\textbf{Top2}}$）。

Bias-only or BitFit is a baseline where we only train the bias vectors while freezing everything else. Contemporarily, this baseline has also been studied by BitFit (Zaken et al. 2021).

Bias-only or BitFit 是只训练偏置向量、冻结其余一切的基线。同期，该基线也被 BitFit（Zaken et al. 2021）研究。

Prefix-embedding tuning (PreEmbed) inserts special tokens among the input tokens. These special tokens have trainable word embeddings and are generally not in the model’s vocabulary. Where to place such tokens can have an impact on performance. We focus on “prefixing”, which prepends such tokens to the prompt, and “infixing”, which appends to the prompt; both are discussed in Li & Liang 2021. We use $l_{p}$ (resp. $l_{i}$) denote the number of prefix (resp. infix) tokens. The number of trainable parameters is $|\Theta|=d_{model}\times(l_{p}+l_{i})$.

Prefix-embedding tuning (PreEmbed) 在输入 token 之间插入特殊 token。这些特殊 token 有可训练词嵌入，一般不在模型词表中。放置位置可影响表现。我们聚焦“prefixing”（把此类 token 加在提示前）与“infixing”（加在提示后）；二者均在 Li & Liang 2021 中讨论。用 $l_{p}$（或 $l_{i}$）表示 prefix（或 infix）token 数。可训练参数数为 $|\Theta|=d_{model}\times(l_{p}+l_{i})$。

Prefix-layer tuning (PreLayer) is an extension to prefix-embedding tuning. Instead of just learning the word embeddings (or equivalently, the activations after the embedding layer) for some special tokens, we learn the activations after every Transformer layer. The activations computed from previous layers are simply replaced by trainable ones. The resulting number of trainable parameters is $|\Theta|=L\times d_{model}\times(l_{p}+l_{i})$, where $L$ is the number of Transformer layers.

Prefix-layer tuning (PreLayer) 是对 prefix-embedding tuning 的扩展。不只学习某些特殊 token 的词嵌入（或等价地，嵌入层后的激活），我们学习每个 Transformer 层后的激活。由前层算出的激活直接被可训练激活替换。所得可训练参数数为 $|\Theta|=L\times d_{model}\times(l_{p}+l_{i})$，其中 $L$ 为 Transformer 层数。

Adapter tuning as proposed in Houlsby et al. 2019 inserts adapter layers between the self-attention module (and the MLP module) and the subsequent residual connection. There are two fully connected layers with biases in an adapter layer with a nonlinearity in between. We call this original design $\textbf{Adapter}^{\textbf{H}}$. Recently, Lin et al. 2020 proposed a more efficient design with the adapter layer applied only after the MLP module and after a LayerNorm. We call it $\textbf{Adapter}^{\textbf{L}}$. This is very similar to another deign proposed in Pfeiffer et al. 2021, which we call $\textbf{Adapter}^{\textbf{P}}$. We also include another baseline call AdapterDrop (Rücklé et al. 2020) which drops some adapter layers for greater efficiency ($\textbf{Adapter}^{\textbf{D}}$). We cite numbers from prior works whenever possible to maximize the number of baselines we compare with; they are in rows with an asterisk (*) in the first column. In all cases, we have $|\Theta|=\hat{L}_{Adpt}\times(2\times d_{model}\times r+r+d_{model})+2\times\hat{L}_{LN}\times d_{model}$ where $\hat{L}_{Adpt}$ is the number of adapter layers and $\hat{L}_{LN}$ the number of trainable LayerNorms (e.g., in $\text{Adapter}^{\text{L}}$).

Adapter tuning 如 Houlsby et al. 2019 所提，在自注意力模块（以及 MLP 模块）与随后的残差连接之间插入 adapter 层。adapter 层中有两个带偏置的全连接层，中间有非线性。我们称该原始设计为 $\textbf{Adapter}^{\textbf{H}}$。近来 Lin et al. 2020 提出更高效设计：adapter 层仅加在 MLP 模块之后、并在 LayerNorm 之后。我们称之为 $\textbf{Adapter}^{\textbf{L}}$。这与 Pfeiffer et al. 2021 提出的另一设计非常相似，我们称之为 $\textbf{Adapter}^{\textbf{P}}$。我们还纳入另一基线 AdapterDrop（Rücklé et al. 2020），它丢掉部分 adapter 层以换取更高效率（$\textbf{Adapter}^{\textbf{D}}$）。为最大化比较的基线数量，我们尽可能引用先前工作数字；它们位于第一列带星号 (*) 的行。在所有情形中，$|\Theta|=\hat{L}_{Adpt}\times(2\times d_{model}\times r+r+d_{model})+2\times\hat{L}_{LN}\times d_{model}$，其中 $\hat{L}_{Adpt}$ 为 adapter 层数，$\hat{L}_{LN}$ 为可训练 LayerNorm 数。

LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices. As mentioned in Section 4.2, we only apply LoRA to $W_{q}$ and $W_{v}$ in most experiments for simplicity. The number of trainable parameters is determined by the rank $r$ and the shape of the original weights: $|\Theta|=2\times\hat{L}_{LoRA}\times d_{model}\times r$, where $\hat{L}_{LoRA}$ is the number of weight matrices we apply LoRA to.

LoRA 与现有权重矩阵并行地加入可训练的秩分解矩阵对。如第 4.2 节所述，多数实验中为简洁只对 $W_{q}$ 与 $W_{v}$ 应用 LoRA。可训练参数数由秩 $r$ 与原权重形状决定：$|\Theta|=2\times\hat{L}_{LoRA}\times d_{model}\times r$，其中 $\hat{L}_{LoRA}$ 是我们应用 LoRA 的权重矩阵数。

5.2 RoBERTa base/large

Model & Method	# Trainable
	Parameters	MNLI	SST-2	MRPC	CoLA	QNLI	QQP	RTE	STS-B	Avg.
$\text{RoB}_{\text{base}}$ (FT)*	125.0M	87.6	94.8	90.2	63.6	92.8	91.9	78.7	91.2	86.4
$\text{RoB}_{\text{base}}$ (BitFit)*	0.1M	84.7	93.7	92.7	62.0	91.8	84.0	81.5	90.8	85.2
$\text{RoB}_{\text{base}}$ ($\text{Adpt}^{\text{D}}$)*	0.3M	87.1$\pm$.0	94.2$\pm$.1	88.5$\pm$1.1	60.8$\pm$.4	93.1$\pm$.1	90.2$\pm$.0	71.5$\pm$2.7	89.7$\pm$.3	84.4
$\text{RoB}_{\text{base}}$ ($\text{Adpt}^{\text{D}}$)*	0.9M	87.3$\pm$.1	94.7$\pm$.3	88.4$\pm$.1	62.6$\pm$.9	93.0$\pm$.2	90.6$\pm$.0	75.9$\pm$2.2	90.3$\pm$.1	85.4
$\text{RoB}_{\text{base}}$ (LoRA)	0.3M	87.5$\pm$.3	95.1$\pm$.2	89.7$\pm$.7	63.4$\pm$1.2	93.3$\pm$.3	90.8$\pm$.1	86.6$\pm$.7	91.5$\pm$.2	87.2
$\text{RoB}_{\text{large}}$ (FT)*	355.0M	90.2	96.4	90.9	68.0	94.7	92.2	86.6	92.4	88.9
$\text{RoB}_{\text{large}}$ (LoRA)	0.8M	90.6$\pm$.2	96.2$\pm$.5	90.9$\pm$1.2	68.2$\pm$1.9	94.9$\pm$.3	91.6$\pm$.1	87.4$\pm$2.5	92.6$\pm$.2	89.0
$\text{RoB}_{\text{large}}$ ($\text{Adpt}^{\text{P}}$)$\dagger$	3.0M	90.2$\pm$.3	96.1$\pm$.3	90.2$\pm$.7	68.3$\pm$1.0	94.8$\pm$.2	91.9$\pm$.1	83.8$\pm$2.9	92.1$\pm$.7	88.4
$\text{RoB}_{\text{large}}$ ($\text{Adpt}^{\text{P}}$)$\dagger$	0.8M	90.5$\pm$.3	96.6$\pm$.2	89.7$\pm$1.2	67.8$\pm$2.5	94.8$\pm$.3	91.7$\pm$.2	80.1$\pm$2.9	91.9$\pm$.4	87.9
$\text{RoB}_{\text{large}}$ ($\text{Adpt}^{\text{H}}$)$\dagger$	6.0M	89.9$\pm$.5	96.2$\pm$.3	88.7$\pm$2.9	66.5$\pm$4.4	94.7$\pm$.2	92.1$\pm$.1	83.4$\pm$1.1	91.0$\pm$1.7	87.8
$\text{RoB}_{\text{large}}$ ($\text{Adpt}^{\text{H}}$)$\dagger$	0.8M	90.3$\pm$.3	96.3$\pm$.5	87.7$\pm$1.7	66.3$\pm$2.0	94.7$\pm$.2	91.5$\pm$.1	72.9$\pm$2.9	91.5$\pm$.5	86.4
$\text{RoB}_{\text{large}}$ (LoRA)$\dagger$	0.8M	90.6$\pm$.2	96.2$\pm$.5	90.2$\pm$1.0	68.2$\pm$1.9	94.8$\pm$.3	91.6$\pm$.2	85.2$\pm$1.1	92.3$\pm$.5	88.6
$\text{DeB}_{\text{XXL}}$ (FT)*	1500.0M	91.8	97.2	92.0	72.0	96.0	92.7	93.9	92.9	91.1
$\text{DeB}_{\text{XXL}}$ (LoRA)	4.7M	91.9$\pm$.2	96.9$\pm$.2	92.6$\pm$.6	72.4$\pm$1.1	96.0$\pm$.1	92.9$\pm$.1	94.9$\pm$.4	93.0$\pm$.2	91.3

Table 2: $\text{RoBERTa}_{\text{base}}$, $\text{RoBERTa}_{\text{large}}$, and $\text{DeBERTa}_{\text{XXL}}$ with different adaptation methods on the GLUE benchmark. We report the overall (matched and mismatched) accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. Higher is better for all metrics. * indicates numbers published in prior works. $\dagger$ indicates runs configured in a setup similar to Houlsby et al. 2019 for a fair comparison.

表 2：RoBERTa base/large 在 GLUE 上不同适配方法的结果（数字与模型名保留）。

Model & Method	# Trainable
	Parameters	MNLI	SST-2	MRPC	CoLA	QNLI	QQP	RTE	STS-B	Avg.
$\text{RoB}_{\text{base}}$ (FT)*	125.0M	87.6	94.8	90.2	63.6	92.8	91.9	78.7	91.2	86.4
$\text{RoB}_{\text{base}}$ (BitFit)*	0.1M	84.7	93.7	92.7	62.0	91.8	84.0	81.5	90.8	85.2
$\text{RoB}_{\text{base}}$ ($\text{Adpt}^{\text{D}}$)*	0.3M	87.1$\pm$.0	94.2$\pm$.1	88.5$\pm$1.1	60.8$\pm$.4	93.1$\pm$.1	90.2$\pm$.0	71.5$\pm$2.7	89.7$\pm$.3	84.4
$\text{RoB}_{\text{base}}$ ($\text{Adpt}^{\text{D}}$)*	0.9M	87.3$\pm$.1	94.7$\pm$.3	88.4$\pm$.1	62.6$\pm$.9	93.0$\pm$.2	90.6$\pm$.0	75.9$\pm$2.2	90.3$\pm$.1	85.4
$\text{RoB}_{\text{base}}$ (LoRA)	0.3M	87.5$\pm$.3	95.1$\pm$.2	89.7$\pm$.7	63.4$\pm$1.2	93.3$\pm$.3	90.8$\pm$.1	86.6$\pm$.7	91.5$\pm$.2	87.2
$\text{RoB}_{\text{large}}$ (FT)*	355.0M	90.2	96.4	90.9	68.0	94.7	92.2	86.6	92.4	88.9
$\text{RoB}_{\text{large}}$ (LoRA)	0.8M	90.6$\pm$.2	96.2$\pm$.5	90.9$\pm$1.2	68.2$\pm$1.9	94.9$\pm$.3	91.6$\pm$.1	87.4$\pm$2.5	92.6$\pm$.2	89.0
$\text{RoB}_{\text{large}}$ ($\text{Adpt}^{\text{P}}$)$\dagger$	3.0M	90.2$\pm$.3	96.1$\pm$.3	90.2$\pm$.7	68.3$\pm$1.0	94.8$\pm$.2	91.9$\pm$.1	83.8$\pm$2.9	92.1$\pm$.7	88.4
$\text{RoB}_{\text{large}}$ ($\text{Adpt}^{\text{P}}$)$\dagger$	0.8M	90.5$\pm$.3	96.6$\pm$.2	89.7$\pm$1.2	67.8$\pm$2.5	94.8$\pm$.3	91.7$\pm$.2	80.1$\pm$2.9	91.9$\pm$.4	87.9
$\text{RoB}_{\text{large}}$ ($\text{Adpt}^{\text{H}}$)$\dagger$	6.0M	89.9$\pm$.5	96.2$\pm$.3	88.7$\pm$2.9	66.5$\pm$4.4	94.7$\pm$.2	92.1$\pm$.1	83.4$\pm$1.1	91.0$\pm$1.7	87.8
$\text{RoB}_{\text{large}}$ ($\text{Adpt}^{\text{H}}$)$\dagger$	0.8M	90.3$\pm$.3	96.3$\pm$.5	87.7$\pm$1.7	66.3$\pm$2.0	94.7$\pm$.2	91.5$\pm$.1	72.9$\pm$2.9	91.5$\pm$.5	86.4
$\text{RoB}_{\text{large}}$ (LoRA)$\dagger$	0.8M	90.6$\pm$.2	96.2$\pm$.5	90.2$\pm$1.0	68.2$\pm$1.9	94.8$\pm$.3	91.6$\pm$.2	85.2$\pm$1.1	92.3$\pm$.5	88.6
$\text{DeB}_{\text{XXL}}$ (FT)*	1500.0M	91.8	97.2	92.0	72.0	96.0	92.7	93.9	92.9	91.1
$\text{DeB}_{\text{XXL}}$ (LoRA)	4.7M	91.9$\pm$.2	96.9$\pm$.2	92.6$\pm$.6	72.4$\pm$1.1	96.0$\pm$.1	92.9$\pm$.1	94.9$\pm$.4	93.0$\pm$.2	91.3

Table 2: $\text{RoBERTa}_{\text{base}}$, $\text{RoBERTa}_{\text{large}}$, and $\text{DeBERTa}_{\text{XXL}}$ with different adaptation methods on the GLUE benchmark. We report the overall (matched and mismatched) accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. Higher is better for all metrics. * indicates numbers published in prior works. $\dagger$ indicates runs configured in a setup similar to Houlsby et al. 2019 for a fair comparison.

RoBERTa (Liu et al. 2019) optimized the pre-training recipe originally proposed in BERT (Devlin et al. 2019a) and boosted the latter’s task performance without introducing many more trainable parameters. While RoBERTa has been overtaken by much larger models on NLP leaderboards such as the GLUE benchmark (Wang et al. 2019) in recent years, it remains a competitive and popular pre-trained model for its size among practitioners. We take the pre-trained RoBERTa base (125M) and RoBERTa large (355M) from the HuggingFace Transformers library (Wolf et al. 2020) and evaluate the performance of different efficient adaptation approaches on tasks from the GLUE benchmark. We also replicate Houlsby et al. 2019 and Pfeiffer et al. 2021 according to their setup. To ensure a fair comparison, we make two crucial changes to how we evaluate LoRA when comparing with adapters. First, we use the same batch size for all tasks and use a sequence length of 128 to match the adapter baselines. Second, we initialize the model to the pre-trained model for MRPC, RTE, and STS-B, not a model already adapted to MNLI like the fine-tuning baseline. Runs following this more restricted setup from Houlsby et al. 2019 are labeled with $\dagger$. The result is presented in Table 2 (Top Three Sections). See Section D.1 for details on the hyperparameters used.

RoBERTa（Liu et al. 2019）优化了 BERT（Devlin et al. 2019a）最初提出的预训练配方，在不引入许多额外可训练参数的情况下提升了后者的任务表现。尽管近年来 RoBERTa 已被更大模型在 GLUE 等 NLP 排行榜上超越，但就体量而言，它在从业者中仍是有竞争力且流行的预训练模型。我们从 HuggingFace Transformers 库（Wolf et al. 2020）取预训练 RoBERTa base（125M）与 RoBERTa large（355M），在 GLUE 基准任务上评估不同高效适配方法的表现。我们还按各自设定复现 Houlsby et al. 2019 与 Pfeiffer et al. 2021。为保证公平比较，与 adapter 比较时我们对 LoRA 评估做两处关键改动。第一，所有任务使用相同 batch size，序列长度 128，以匹配 adapter 基线。第二，对 MRPC、RTE 与 STS-B，模型初始化为预训练模型，而非像微调基线那样已适配到 MNLI 的模型。

5.3 DeBERTa XXL

DeBERTa (He et al. 2021) is a more recent variant of BERT that is trained on a much larger scale and performs very competitively on benchmarks such as GLUE (Wang et al. 2019) and SuperGLUE (Wang et al. 2020). We evaluate if LoRA can still match the performance of a fully fine-tuned DeBERTa XXL (1.5B) on GLUE. The result is presented in Table 2 (Bottom Section). See Section D.2 for details on the hyperparameters used.

DeBERTa（He et al. 2021）是 BERT 更近的变体，以更大尺度训练，并在 GLUE（Wang et al. 2019）与 SuperGLUE（Wang et al. 2020）等基准上表现极具竞争力。我们评估 LoRA 能否在 GLUE 上仍匹配全量微调 DeBERTa XXL（1.5B）的表现。结果见表 2（下段）。所用超参细节见第 D.2 节。

5.4 GPT-2 medium/large

Having shown that LoRA can be a competitive alternative to full fine-tuning on NLU, we hope to answer if LoRA still prevails on NLG models, such as GPT-2 medium and large (Radford et al. b). We keep our setup as close as possible to Li & Liang 2021 for a direct comparison. Due to space constraint, we only present our result on E2E NLG Challenge (Table 3) in this section. See Section F.1 for results on WebNLG (Gardent et al. 2017) and DART (Nan et al. 2020). We include a list of the hyperparameters used in Section D.3.

在表明 LoRA 可作为 NLU 上全量微调的有竞争力替代后，我们希望回答：LoRA 在 NLG 模型（如 GPT-2 medium 与 large（Radford et al. b））上是否仍然占优。我们尽可能保持与 Li & Liang 2021 接近的设定以便直接比较。受篇幅限制，本节只给出 E2E NLG Challenge 结果（表 3）。WebNLG 与 DART 的额外结果见第 F.1 节（表 10–11）。

Model & Method	# Trainable	E2E NLG Challenge
	Parameters	BLEU	NIST	MET	ROUGE-L	CIDEr
GPT-2 M (FT)*	354.92M	68.2	8.62	46.2	71.0	2.47
GPT-2 M ($\text{Adapter}^{\text{L}}$)*	0.37M	66.3	8.41	45.0	69.8	2.40
GPT-2 M ($\text{Adapter}^{\text{L}}$)*	11.09M	68.9	8.71	46.1	71.3	2.47
GPT-2 M ($\text{Adapter}^{\text{H}}$)	11.09M	67.3$\pm$.6	8.50$\pm$.07	46.0$\pm$.2	70.7$\pm$.2	2.44$\pm$.01
GPT-2 M ($\text{FT}^{\text{Top2}}$)*	25.19M	68.1	8.59	46.0	70.8	2.41
GPT-2 M (PreLayer)*	0.35M	69.7	8.81	46.1	71.4	2.49
GPT-2 M (LoRA)	0.35M	70.4$\pm$.1	8.85$\pm$.02	46.8$\pm$.2	71.8$\pm$.1	2.53$\pm$.02
GPT-2 L (FT)*	774.03M	68.5	8.78	46.0	69.9	2.45
GPT-2 L ($\text{Adapter}^{\text{L}}$)	0.88M	69.1$\pm$.1	8.68$\pm$.03	46.3$\pm$.0	71.4$\pm$.2	2.49$\pm$.0
GPT-2 L ($\text{Adapter}^{\text{L}}$)	23.00M	68.9$\pm$.3	8.70$\pm$.04	46.1$\pm$.1	71.3$\pm$.2	2.45$\pm$.02
GPT-2 L (PreLayer)*	0.77M	70.3	8.85	46.2	71.7	2.47
GPT-2 L (LoRA)	0.77M	70.4$\pm$.1	8.89$\pm$.02	46.8$\pm$.2	72.0$\pm$.2	2.47$\pm$.02

Table 3: GPT-2 medium (M) and large (L) with different adaptation methods on the E2E NLG Challenge. For all metrics, higher is better. LoRA outperforms several baselines with comparable or fewer trainable parameters. Confidence intervals are shown for experiments we ran. * indicates numbers published in prior works.

Model & Method	# Trainable	E2E NLG Challenge
	Parameters	BLEU	NIST	MET	ROUGE-L	CIDEr
GPT-2 M (FT)*	354.92M	68.2	8.62	46.2	71.0	2.47
GPT-2 M ($\text{Adapter}^{\text{L}}$)*	0.37M	66.3	8.41	45.0	69.8	2.40
GPT-2 M ($\text{Adapter}^{\text{L}}$)*	11.09M	68.9	8.71	46.1	71.3	2.47
GPT-2 M ($\text{Adapter}^{\text{H}}$)	11.09M	67.3$\pm$.6	8.50$\pm$.07	46.0$\pm$.2	70.7$\pm$.2	2.44$\pm$.01
GPT-2 M ($\text{FT}^{\text{Top2}}$)*	25.19M	68.1	8.59	46.0	70.8	2.41
GPT-2 M (PreLayer)*	0.35M	69.7	8.81	46.1	71.4	2.49
GPT-2 M (LoRA)	0.35M	70.4$\pm$.1	8.85$\pm$.02	46.8$\pm$.2	71.8$\pm$.1	2.53$\pm$.02
GPT-2 L (FT)*	774.03M	68.5	8.78	46.0	69.9	2.45
GPT-2 L ($\text{Adapter}^{\text{L}}$)	0.88M	69.1$\pm$.1	8.68$\pm$.03	46.3$\pm$.0	71.4$\pm$.2	2.49$\pm$.0
GPT-2 L ($\text{Adapter}^{\text{L}}$)	23.00M	68.9$\pm$.3	8.70$\pm$.04	46.1$\pm$.1	71.3$\pm$.2	2.45$\pm$.02
GPT-2 L (PreLayer)*	0.77M	70.3	8.85	46.2	71.7	2.47
GPT-2 L (LoRA)	0.77M	70.4$\pm$.1	8.89$\pm$.02	46.8$\pm$.2	72.0$\pm$.2	2.47$\pm$.02

表 3：GPT-2 medium (M) 与 large (L) 在 E2E NLG Challenge 上不同适配方法的结果。所有指标越高越好。LoRA 以可比或更少的可训练参数超过若干基线。置信区间对应我们运行的实验。* 表示先前工作发表的数字。

Model&Method	# Trainable	WikiSQL	MNLI-m	SAMSum
Parameters	Acc. (%)	Acc. (%)	R1/R2/RL
GPT-3 (FT)	175,255.8M	73.8	89.5	52.0/28.0/44.5
GPT-3 (BitFit)	14.2M	71.3	91.0	51.3/27.4/43.5
GPT-3 (PreEmbed)	3.2M	63.1	88.6	48.3/24.2/40.5
GPT-3 (PreLayer)	20.2M	70.1	89.5	50.8/27.3/43.5
GPT-3 ($\text{Adapter}^{\text{H}}$)	7.1M	71.9	89.8	53.0/28.9/44.8
GPT-3 ($\text{Adapter}^{\text{H}}$)	40.1M	73.2	91.5	53.2/29.0/45.1
GPT-3 (LoRA)	4.7M	73.4	91.7	53.8/29.8/45.9
GPT-3 (LoRA)	37.7M	74.0	91.6	53.4/29.2/45.1

Table 4: Performance of different adaptation methods on GPT-3 175B. We report the logical form validation accuracy on WikiSQL, validation accuracy on MultiNLI-matched, and Rouge-1/2/L on SAMSum. LoRA performs better than prior approaches, including full fine-tuning. The results on WikiSQL have a fluctuation around $\pm 0.5\%$, MNLI-m around $\pm 0.1\%$, and SAMSum around $\pm 0.2$/$\pm 0.2$/$\pm 0.1$ for the three metrics.

Model&Method	# Trainable	WikiSQL	MNLI-m	SAMSum
Parameters	Acc. (%)	Acc. (%)	R1/R2/RL
GPT-3 (FT)	175,255.8M	73.8	89.5	52.0/28.0/44.5
GPT-3 (BitFit)	14.2M	71.3	91.0	51.3/27.4/43.5
GPT-3 (PreEmbed)	3.2M	63.1	88.6	48.3/24.2/40.5
GPT-3 (PreLayer)	20.2M	70.1	89.5	50.8/27.3/43.5
GPT-3 ($\text{Adapter}^{\text{H}}$)	7.1M	71.9	89.8	53.0/28.9/44.8
GPT-3 ($\text{Adapter}^{\text{H}}$)	40.1M	73.2	91.5	53.2/29.0/45.1
GPT-3 (LoRA)	4.7M	73.4	91.7	53.8/29.8/45.9
GPT-3 (LoRA)	37.7M	74.0	91.6	53.4/29.2/45.1

表 4：GPT-3 175B 上不同适配方法的表现。我们报告 WikiSQL 逻辑形式验证准确率、MultiNLI-matched 验证准确率，以及 SAMSum 上的 Rouge-1/2/L。LoRA 优于包括全量微调在内的先前方法。WikiSQL 结果波动约 $\pm 0.5\%$，MNLI-m 约 $\pm 0.1\%$，SAMSum 三项约 $\pm 0.2$/$\pm 0.2$/$\pm 0.1$。

5.5 Scaling up to GPT-3 175B

As a final stress test for LoRA, we scale up to GPT-3 with 175 billion parameters. Due to the high training cost, we only report the typical standard deviation for a given task over random seeds, as opposed to providing one for every entry. See Section D.4 for details on the hyperparameters used.

作为对 LoRA 的最终压力测试，我们扩展到 1750 亿参数的 GPT-3。由于训练成本高，我们只报告给定任务上相对随机种子的典型标准差，而非为每个条目提供。所用超参见第 D.4 节。

As shown in Table 4, LoRA matches or exceeds the fine-tuning baseline on all three datasets. Note that not all methods benefit monotonically from having more trainable parameters, as shown in Figure 2. We observe a significant performance drop when we use more than 256 special tokens for prefix-embedding tuning or more than 32 special tokens for prefix-layer tuning. This corroborates similar observations in Li & Liang 2021. While a thorough investigation into this phenomenon is out-of-scope for this work, we suspect that having more special tokens causes the input distribution to shift further away from the pre-training data distribution. Separately, we investigate the performance of different adaptation approaches in the low-data regime in Section F.3.

如表 4 所示，LoRA 在全部三个数据集上匹配或超过微调基线。注意并非所有方法都随可训练参数增多而单调受益，见图 2。当我们为 prefix-embedding tuning 使用超过 256 个特殊 token，或为 prefix-layer tuning 使用超过 32 个特殊 token 时，观察到显著性能下降。这印证了 Li & Liang 2021 中的类似观察。对该现象的彻底研究超出本文范围，但我们怀疑更多特殊 token 会使输入分布进一步偏离预训练数据分布。另外，我们在第 F.3 节考察低数据设定下不同适配方法的表现。

Figure 2: GPT-3 175B validation accuracy vs. number of trainable parameters of several adaptation methods on WikiSQL and MNLI-matched. LoRA exhibits better scalability and task performance. See Section F.2 for more details on the plotted data points.

图 2：GPT-3 175B 上若干适配方法在 WikiSQL 与 MNLI-matched 上的验证准确率相对可训练参数数。LoRA 展现更好的可扩展性与任务表现。所绘数据点细节见第 F.2 节。

6 Related Works

Transformer Language Models.  Transformer (Vaswani et al. 2017) is a sequence-to-sequence architecture that makes heavy use of self-attention. Radford et al. a applied it to autoregressive language modeling by using a stack of Transformer decoders. Since then, Transformer-based language models have dominated NLP, achieving the state-of-the-art in many tasks. A new paradigm emerged with BERT (Devlin et al. 2019b) and GPT-2 (Radford et al. b) – both are large Transformer language models trained on a large amount of text – where fine-tuning on task-specific data after pre-training on general domain data provides a significant performance gain compared to training on task-specific data directly. Training larger Transformers generally results in better performance and remains an active research direction. GPT-3 (Brown et al. 2020) is the largest single Transformer language model trained to-date with 175B parameters.

Transformer Language Models.　Transformer（Vaswani et al. 2017）是大量使用自注意力的序列到序列架构。Radford et al. a 通过堆叠 Transformer 解码器把它用于自回归语言建模。此后，基于 Transformer 的语言模型主导了 NLP，在许多任务上达到当时最好水平。BERT（Devlin et al. 2019b）与 GPT-2（Radford et al. b）——二者都是在大量文本上训练的大型 Transformer 语言模型——带来新范式：相对直接在任务数据上训练，先在通用领域预训练再在任务数据上微调能带来显著性能增益。训练更大的 Transformer 通常带来更好表现，仍是活跃研究方向。GPT-3（Brown et al. 2020）是当时训练过的最大单一 Transformer 语言模型，有 175B 参数。

Prompt Engineering and Fine-Tuning.  While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily on the input prompt (Brown et al. 2020). This necessitates an empirical art of composing and formatting the prompt to maximize a model’s performance on a desired task, which is known as prompt engineering or prompt hacking. Fine-tuning retrains a model pre-trained on general domains to a specific task Devlin et al. 2019b; Radford et al. a. Variants of it include learning just a subset of the parameters Devlin et al. 2019b; Collobert & Weston 2008, yet practitioners often retrain all of them to maximize the downstream performance. However, the enormity of GPT-3 175B makes it challenging to perform fine-tuning in the usual way due to the large checkpoint it produces and the high hardware barrier to entry since it has the same memory footprint as pre-training.

Prompt Engineering and Fine-Tuning.　虽然 GPT-3 175B 只需少量额外训练样本就能适配其行为，结果严重依赖输入提示（Brown et al. 2020）。这催生了组合与格式化提示以最大化模型在目标任务上表现的经验艺术，称为 prompt engineering 或 prompt hacking。微调把在通用领域预训练的模型再训练到特定任务（Devlin et al. 2019b; Radford et al. a）。其变体包括只学习参数子集（Devlin et al. 2019b; Collobert & Weston 2008），但从业者常重训全部参数以最大化下游表现。然而 GPT-3 175B 的体量使其难以按常规方式微调：它产生巨大检查点，且内存占用与预训练相同，硬件门槛很高。

Parameter-Efficient Adaptation.  Many have proposed inserting adapter layers between existing layers in a neural network (Houlsby et al. 2019; Rebuffi et al. 2017; Lin et al. 2020). Our method uses a similar bottleneck structure to impose a low-rank constraint on the weight updates. The key functional difference is that our learned weights can be merged with the main weights during inference, thus not introducing any latency, which is not the case for the adapter layers (Section 3). A comtenporary extension of adapter is compacter (Mahabadi et al. 2021), which essentially parametrizes the adapter layers using Kronecker products with some predetermined weight sharing scheme. Similarly, combining LoRA with other tensor product-based methods could potentially improve its parameter efficiency, which we leave to future work. More recently, many proposed optimizing the input word embeddings in lieu of fine-tuning, akin to a continuous and differentiable generalization of prompt engineering (Li & Liang 2021; Lester et al. 2021; Hambardzumyan et al. 2020; Liu et al. 2021). We include comparisons with Li & Liang 2021 in our experiment section. However, this line of works can only scale up by using more special tokens in the prompt, which take up available sequence length for task tokens when positional embeddings are learned.

Parameter-Efficient Adaptation.　许多人提出在神经网络既有层之间插入 adapter 层（Houlsby et al. 2019; Rebuffi et al. 2017; Lin et al. 2020）。我们的方法使用类似瓶颈结构，对权重更新施加低秩约束。关键功能差别是：我们学到的权重可在推理时与主权重合并，因而不引入任何延迟，这对 adapter 层不成立（第 3 节）。同期的 adapter 扩展是 Compacter（Mahabadi et al. 2021），它本质上用带预定权重共享方案的 Kronecker 积参数化 adapter 层。类似地，把 LoRA 与其他基于张量积的方法结合有望进一步提升参数效率，我们留待未来工作。更近地，许多人提出优化输入词嵌入以代替微调，类似于 prompt engineering 的连续可微推广（Li & Liang 2021; Lester et al. 2021; Hambardzumyan et al. 2020; Liu et al. 2021）。实验部分纳入与 Li & Liang 2021 的比较。但这条工作线只能通过在提示中使用更多特殊 token 来扩展，而在学习位置嵌入时，这些 token 会占用可用于任务 token 的序列长度。

Low-Rank Structures in Deep Learning.  Low-rank structure is very common in machine learning. A lot of machine learning problems have certain intrinsic low-rank structure (Li et al. 2016; Cai et al. 2010; Li et al. 2018b; Grasedyck et al. 2013). Moreover, it is known that for many deep learning tasks, especially those with a heavily over-parametrized neural network, the learned neural network will enjoy low-rank properties after training (Oymak et al. 2019). Some prior works even explicitly impose the low-rank constraint when training the original neural network (Sainath et al. 2013; Povey et al. 2018; Zhang et al. 2014; Jaderberg et al. 2014; Zhao et al. 2016; Khodak et al. 2021; Denil et al. 2014); however, to the best of our knowledge, none of these works considers low-rank update to a frozen model for adaptation to downstream tasks. In theory literature, it is known that neural networks outperform other classical learning methods, including the corresponding (finite-width) neural tangent kernels (Allen-Zhu et al. 2019; Li & Liang 2018) when the underlying concept class has certain low-rank structure (Ghorbani et al. 2020; Allen-Zhu & Li 2019; Allen-Zhu & Li 2020a). Another theoretical result in Allen-Zhu & Li 2020b suggests that low-rank adaptations can be useful for adversarial training. In sum, we believe that our proposed low-rank adaptation update is well-motivated by the literature.

Low-Rank Structures in Deep Learning.　低秩结构在机器学习中非常常见。许多机器学习问题具有某种本征低秩结构（Li et al. 2016; Cai et al. 2010; Li et al. 2018b; Grasedyck et al. 2013）。此外，已知对许多深度学习任务——尤其是严重过参数化的神经网络——训练后学得的网络会具有低秩性质（Oymak et al. 2019）。一些先前工作甚至在训练原始神经网络时显式施加低秩约束（Sainath et al. 2013; Povey et al. 2018; Zhang et al. 2014; Jaderberg et al. 2014; Zhao et al. 2016; Khodak et al. 2021; Denil et al. 2014）；但据我们所知，这些工作都未考虑对冻结模型做低秩更新以适配下游任务。在理论文献中，已知当底层概念类具有某些低秩结构时，神经网络优于包括对应（有限宽）神经正切核在内的其他经典学习方法（Allen-Zhu et al. 2019; Li & Liang 2018; Ghorbani et al. 2020; Allen-Zhu & Li 2019; Allen-Zhu & Li 2020a）。Allen-Zhu & Li 2020b 的另一理论结果暗示低秩适配可对抗训练有用。总之，我们相信所提出的低秩适配更新有充分的文献动机。

7 Understanding the Low-Rank Updates

Given the empirical advantage of LoRA, we hope to further explain the properties of the low-rank adaptation learned from downstream tasks. Note that the low-rank structure not only lowers the hardware barrier to entry which allows us to run multiple experiments in parallel, but also gives better interpretability of how the update weights are correlated with the pre-trained weights. We focus our study on GPT-3 175B, where we achieved the largest reduction of trainable parameters (up to 10,000$\times$) without adversely affecting task performances.

鉴于 LoRA 的经验优势，我们希望进一步解释从下游任务学到的低秩适配的性质。注意低秩结构不仅降低硬件门槛、使我们能并行跑多个实验，还使更新权重如何与预训练权重相关更具可解释性。我们把研究聚焦于 GPT-3 175B，在此我们实现了最大的可训练参数缩减（最高约 10,000$\times$）且未损害任务表现。

We perform a sequence of empirical studies to answer the following questions: 1) Given a parameter budget constraint, which subset of weight matrices in a pre-trained Transformer should we adapt to maximize downstream performance? 2) Is the “optimal” adaptation matrix $\Delta W$ really rank-deficient? If so, what is a good rank to use in practice? 3) What is the connection between $\Delta W$ and $W$? Does $\Delta W$ highly correlate with $W$? How large is $\Delta W$ comparing to $W$?

我们做一系列实证研究以回答下列问题：1）给定参数预算约束，应适配预训练 Transformer 中哪一子集的权重矩阵以最大化下游表现？2）“最优”适配矩阵 $\Delta W$ 是否真的秩亏？若是，实践中用多大的秩合适？3）$\Delta W$ 与 $W$ 有何联系？$\Delta W$ 是否与 $W$ 高度相关？相对 $W$，$\Delta W$ 有多大？

We believe that our answers to question (2) and (3) shed light on the fundamental principles of using pre-trained language models for downstream tasks, which is a critical topic in NLP.

我们相信对问题 (2) 与 (3) 的回答，有助于阐明把预训练语言模型用于下游任务的基本原理——这是 NLP 中的关键议题。

7.1 Which Weight Matrices in Transformer Should We Apply LoRA to?

Given a limited parameter budget, which types of weights should we adapt with LoRA to obtain the best performance on downstream tasks? As mentioned in Section 4.2, we only consider weight matrices in the self-attention module. We set a parameter budget of 18M (roughly 35MB if stored in FP16) on GPT-3 175B, which corresponds to $r=8$ if we adapt one type of attention weights or $r=4$ if we adapt two types, for all 96 layers. The result is presented in Table 5.

给定有限参数预算，应把 LoRA 用在哪类权重上，才能在下游任务上取得最佳表现？如第 4.2 节所述，我们只考虑自注意力模块中的权重矩阵。我们在 GPT-3 175B 上设定 18M 参数预算（以 FP16 存储约 35MB），对应：若适配一类注意力权重则 $r=8$，若适配两类则 $r=4$，覆盖全部 96 层。结果见表 5。

	# of Trainable Parameters = 18M
Weight Type	$W_{q}$	$W_{k}$	$W_{v}$	$W_{o}$	$W_{q},W_{k}$	$W_{q},W_{v}$	$W_{q},W_{k},W_{v},W_{o}$
Rank $r$	8	8	8	8	4	4	2
WikiSQL ($\pm 0.5$%)	70.4	70.0	73.0	73.2	71.4	73.7	73.7
MultiNLI ($\pm 0.1$%)	91.0	90.8	91.0	91.3	91.3	91.3	91.7

Table 5: Validation accuracy on WikiSQL and MultiNLI after applying LoRA to different types of attention weights in GPT-3, given the same number of trainable parameters. Adapting both $W_{q}$ and $W_{v}$ gives the best performance overall. We find the standard deviation across random seeds to be consistent for a given dataset, which we report in the first column.

	# of Trainable Parameters = 18M
Weight Type	$W_{q}$	$W_{k}$	$W_{v}$	$W_{o}$	$W_{q},W_{k}$	$W_{q},W_{v}$	$W_{q},W_{k},W_{v},W_{o}$
Rank $r$	8	8	8	8	4	4	2
WikiSQL ($\pm 0.5$%)	70.4	70.0	73.0	73.2	71.4	73.7	73.7
MultiNLI ($\pm 0.1$%)	91.0	90.8	91.0	91.3	91.3	91.3	91.7

表 5：在可训练参数数相同的条件下，对 GPT-3 不同类型注意力权重应用 LoRA 后在 WikiSQL 与 MultiNLI 上的验证准确率。同时适配 $W_{q}$ 与 $W_{v}$ 总体表现最好。我们发现给定数据集上跨随机种子的标准差一致，报告于第一列。

Note that putting all the parameters in $\Delta W_{q}$ or $\Delta W_{k}$ results in significantly lower performance, while adapting both $W_{q}$ and $W_{v}$ yields the best result. This suggests that even a rank of four captures enough information in $\Delta W$ such that it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank.

注意：把全部参数放在 $\Delta W_{q}$ 或 $\Delta W_{k}$ 上会导致显著更低的表现，而同时适配 $W_{q}$ 与 $W_{v}$ 给出最佳结果。这表明即便秩为四也足以捕捉 $\Delta W$ 中的足够信息，因而更宜适配更多权重矩阵，而非用更大的秩只适配单一类型权重。

7.2 What is the Optimal Rank $r$ for LoRA?

We turn our attention to the effect of rank $r$ on model performance. We adapt $\{W_{q},W_{v}\}$, $\{W_{q},W_{k},W_{v},W_{c}\}$, and just $W_{q}$ for a comparison.

我们转而关注秩 $r$ 对模型表现的影响。我们适配 $\{W_{q},W_{v}\}$、$\{W_{q},W_{k},W_{v},W_{o}\}$，以及仅 $W_{q}$ 以便比较。

	Weight Type	$r=1$	$r=2$	$r=4$	$r=8$	$r=64$
WikiSQL($\pm 0.5$%)	$W_{q}$	68.8	69.6	70.5	70.4	70.0
$W_{q},W_{v}$	73.4	73.3	73.7	73.8	73.5
	$W_{q},W_{k},W_{v},W_{o}$	74.1	73.7	74.0	74.0	73.9
MultiNLI ($\pm 0.1$%)	$W_{q}$	90.7	90.9	91.1	90.7	90.7
$W_{q},W_{v}$	91.3	91.4	91.3	91.6	91.4
$W_{q},W_{k},W_{v},W_{o}$	91.2	91.7	91.7	91.5	91.4

Table 6: Validation accuracy on WikiSQL and MultiNLI with different rank $r$. To our surprise, a rank as small as one suffices for adapting both $W_{q}$ and $W_{v}$ on these datasets while training $W_{q}$ alone needs a larger $r$. We conduct a similar experiment on GPT-2 in Section H.2.

	Weight Type	$r=1$	$r=2$	$r=4$	$r=8$	$r=64$
WikiSQL($\pm 0.5$%)	$W_{q}$	68.8	69.6	70.5	70.4	70.0
$W_{q},W_{v}$	73.4	73.3	73.7	73.8	73.5
	$W_{q},W_{k},W_{v},W_{o}$	74.1	73.7	74.0	74.0	73.9
MultiNLI ($\pm 0.1$%)	$W_{q}$	90.7	90.9	91.1	90.7	90.7
$W_{q},W_{v}$	91.3	91.4	91.3	91.6	91.4
$W_{q},W_{k},W_{v},W_{o}$	91.2	91.7	91.7	91.5	91.4

表 6：不同秩 $r$ 在 WikiSQL 与 MultiNLI 上的验证准确率。令我们惊讶的是：在这些数据集上同时适配 $W_{q}$ 与 $W_{v}$ 时，小至 1 的秩已足够；而只训练 $W_{q}$ 需要更大的 $r$。我们在第 H.2 节对 GPT-2 做了类似实验。

Table 6shows that, surprisingly, LoRA already performs competitively with a very small $r$ (more so for $\{W_{q},W_{v}\}$ than just $W_{q}$). This suggests the update matrix $\Delta W$ could have a very small ‘‘intrinsic rank”.[^6] To further support this finding, we check the overlap of the subspaces learned by different choices of $r$ and by different random seeds. We argue that increasing $r$ does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.

表 6 表明：令人惊讶的是，极小的 $r$ 已使 LoRA 表现具有竞争力（对 $\{W_{q},W_{v}\}$ 尤甚于仅 $W_{q}$）。这暗示更新矩阵 $\Delta W$ 可能具有非常小的“本征秩”。[^6] 为进一步支持该发现，我们检查不同 $r$ 选择以及不同随机种子所学子空间的重叠。我们主张增大 $r$ 并不会覆盖更有意义的子空间，这表明低秩适配矩阵已足够。

Subspace similarity between different $r$.  Given $A_{r=8}$ and $A_{r=64}$ which are the learned adaptation matrices with rank $r=8$ and $64$ using the same pre-trained model, we perform singular value decomposition and obtain the right-singular unitary matrices $U_{A_{r=8}}$ and $U_{A_{r=64}}$.[^7] We hope to answer: how much of the subspace spanned by the top $i$ singular vectors in $U_{A_{r=8}}$ (for $1\leq i\leq 8$) is contained in the subspace spanned by top $j$ singular vectors of $U_{A_{r=64}}$ (for $1\leq j\leq 64$)? We measure this quantity with a normalized subspace similarity based on the Grassmann distance (See Appendix G for a more formal discussion)

Subspace similarity between different $r$.　给定用同一预训练模型学到的秩 $r=8$ 与 $64$ 的适配矩阵 $A_{r=8}$ 与 $A_{r=64}$，我们做奇异值分解并得到右奇异酉矩阵 $U_{A_{r=8}}$ 与 $U_{A_{r=64}}$。[^7] 我们希望回答：$U_{A_{r=8}}$ 中前 $i$ 个奇异向量张成的子空间（$1\leq i\leq 8$），有多少包含在 $U_{A_{r=64}}$ 前 $j$ 个奇异向量张成的子空间中（$1\leq j\leq 64$）？我们用基于 Grassmann 距离的归一化子空间相似度衡量该量（更形式化讨论见附录 G）