ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

ALBERT：用于语言表示自监督学习的轻量 BERT

Zhenzhong Lan 1    Mingda Chen 2    Sebastian Goodman 1    Kevin Gimpel 2 Piyush Sharma 1    Radu Soricut 1       1 Google Research     2 Toyota Technological Institute at Chicago    {lanzhzh, seabass, piyushsharma, rsoricut}@google.com           {mchen, kgimpel}@ttic.edu Thanks:  Work done as an intern at Google Research, driving data processing and downstream task evaluations.

Abstract

摘要

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT (Devlin et al. 2019). Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at https://github.com/google-research/ALBERT.

在预训练自然语言表示时增大模型规模，往往能提升下游任务表现。但到了一定程度，继续放大就会撞上 GPU/TPU 显存上限，训练时间也会拉长。为缓解这些问题，我们提出两种参数缩减技术，降低 BERT（Devlin et al. 2019）的显存占用并加快训练。大量实验表明，相对原始 BERT，我们的方法让模型更易继续放大。我们还使用关注句间连贯性的自监督损失，并证明它持续有助于需要多句输入的下游任务。最终，我们最好的模型在参数量少于 BERT-large 的前提下，在 GLUE、RACE 与 SQuAD 基准上刷新了当时最优结果。代码与预训练模型见 https://github.com/google-research/ALBERT。

1 Introduction

1 引言

Full network pre-training (Dai & Le 2015; Radford et al. 2018; Devlin et al. 2019; Howard & Ruder 2018) has led to a series of breakthroughs in language representation learning. Many nontrivial NLP tasks, including those that have limited training data, have greatly benefited from these pre-trained models. One of the most compelling signs of these breakthroughs is the evolution of machine performance on a reading comprehension task designed for middle and high-school English exams in China, the RACE test (Lai et al. 2017): the paper that originally describes the task and formulates the modeling challenge reports then state-of-the-art machine accuracy at $44.1\%$; the latest published result reports their model performance at $83.2\%$ (Liu et al. 2019); the work we present here pushes it even higher to $89.4\%$, a stunning $45.3\%$ improvement that is mainly attributable to our current ability to build high-performance pretrained language representations.

全网络预训练（Dai & Le 2015；Radford et al. 2018；Devlin et al. 2019；Howard & Ruder 2018）推动了语言表示学习的一系列突破。许多非平凡 NLP 任务——包括训练数据有限的任务——都从这些预训练模型中受益。最有说服力的迹象之一，是机器在为中国中学、高中英语考试设计的阅读理解任务 RACE（Lai et al. 2017）上的表现演变：原始论文报告当时最优机器准确率为 $44.1\%$；最新已发表结果报告为 $83.2\%$（Liu et al. 2019）；本文进一步推到 $89.4\%$，绝对提升达 $45.3\%$，主要归功于我们如今构建高性能预训练语言表示的能力。

Evidence from these improvements reveals that a large network is of crucial importance for achieving state-of-the-art performance (Devlin et al. 2019; Radford et al. 2019). It has become common practice to pre-train large models and distill them down to smaller ones (Sun et al. 2019; Turc et al. 2019) for real applications. Given the importance of model size, we ask: Is having better NLP models as easy as having larger models?

这些改进说明：要拿到最优表现，大网络至关重要（Devlin et al. 2019；Radford et al. 2019）。业界也常先预训练大模型，再蒸馏成小模型用于落地（Sun et al. 2019；Turc et al. 2019）。既然模型规模如此重要，我们追问：做更好的 NLP 模型，是否就等于做更大的模型？

An obstacle to answering this question is the memory limitations of available hardware. Given that current state-of-the-art models often have hundreds of millions or even billions of parameters, it is easy to hit these limitations as we try to scale our models. Training speed can also be significantly hampered in distributed training, as the communication overhead is directly proportional to the number of parameters in the model.

回答这个问题的障碍是硬件显存上限。当前最优模型动辄数亿乃至数十亿参数，继续放大很容易撞墙。在分布式训练里，通信开销与参数量成正比，训练速度也会被明显拖慢。

Existing solutions to the aforementioned problems include model parallelization (Shazeer et al. 2018; Shoeybi et al. 2019) and clever memory management (Chen et al. 2016; Gomez et al. 2017). These solutions address the memory limitation problem, but not the communication overhead. In this paper, we address all of the aforementioned problems, by designing A Lite BERT (ALBERT) architecture that has significantly fewer parameters than a traditional BERT architecture.

已有对策包括模型并行（Shazeer et al. 2018；Shoeybi et al. 2019）与巧妙的显存管理（Chen et al. 2016；Gomez et al. 2017）。它们缓解显存问题，却不解决通信开销。本文通过设计参数量远少于传统 BERT 的 A Lite BERT（ALBERT）架构，同时应对上述问题。

ALBERT incorporates two parameter reduction techniques that lift the major obstacles in scaling pre-trained models. The first one is a factorized embedding parameterization. By decomposing the large vocabulary embedding matrix into two small matrices, we separate the size of the hidden layers from the size of vocabulary embedding. This separation makes it easier to grow the hidden size without significantly increasing the parameter size of the vocabulary embeddings. The second technique is cross-layer parameter sharing. This technique prevents the parameter from growing with the depth of the network. Both techniques significantly reduce the number of parameters for BERT without seriously hurting performance, thus improving parameter-efficiency. An ALBERT configuration similar to BERT-large has 18x fewer parameters and can be trained about 1.7x faster. The parameter reduction techniques also act as a form of regularization that stabilizes the training and helps with generalization.

ALBERT 引入两种参数缩减技术，去掉放大预训练模型的主要障碍。其一是分解式嵌入参数化：把大词表嵌入矩阵拆成两个小矩阵，使隐层大小与词表嵌入大小解耦，从而在不明显增加词表嵌入参数的前提下扩大隐层。其二是跨层参数共享：参数量不再随网络深度线性增长。两种技术都在不明显伤性能的前提下大幅削减 BERT 参数量，提升参数效率。与 BERT-large 相近的 ALBERT 配置参数量约少 18 倍，训练约快 1.7 倍。参数缩减本身也起到正则化作用，稳定训练并有助于泛化。

To further improve the performance of ALBERT, we also introduce a self-supervised loss for sentence-order prediction (SOP). SOP primary focuses on inter-sentence coherence and is designed to address the ineffectiveness (Yang et al. 2019; Liu et al. 2019) of the next sentence prediction (NSP) loss proposed in the original BERT.

为进一步提升 ALBERT，我们还引入面向句子顺序预测（SOP）的自监督损失。SOP 主要关注句间连贯，旨在纠正原始 BERT 中下一句预测（NSP）损失已被证明效果不稳的问题（Yang et al. 2019；Liu et al. 2019）。

As a result of these design decisions, we are able to scale up to much larger ALBERT configurations that still have fewer parameters than BERT-large but achieve significantly better performance. We establish new state-of-the-art results on the well-known GLUE, SQuAD, and RACE benchmarks for natural language understanding. Specifically, we push the RACE accuracy to $89.4\%$, the GLUE benchmark to 89.4, and the F1 score of SQuAD 2.0 to 92.2.

基于这些设计，我们得以放大到参数仍少于 BERT-large、表现却显著更好的 ALBERT 配置，并在著名的 GLUE、SQuAD、RACE 自然语言理解基准上刷新最优结果。具体而言，RACE 准确率达到 $89.4\%$，GLUE 分数达到 89.4，SQuAD 2.0 的 F1 达到 92.2。

2 Related work

2 相关工作

2.1 Scaling Up Representation Learning for Natural Language

2.1 放大自然语言表示学习

Learning representations of natural language has been shown to be useful for a wide range of NLP tasks and has been widely adopted (Mikolov et al. 2013; Le & Mikolov 2014; Dai & Le 2015; Peters et al. 2018; Devlin et al. 2019; Radford et al. 2018; Radford et al. 2019). One of the most significant changes in the last two years is the shift from pre-training word embeddings, whether standard (Mikolov et al. 2013; Pennington et al. 2014) or contextualized (McCann et al. 2017; Peters et al. 2018), to full-network pre-training followed by task-specific fine-tuning (Dai & Le 2015; Radford et al. 2018; Devlin et al. 2019). In this line of work, it is often shown that larger model size improves performance. For example, Devlin et al. 2019 show that across three selected natural language understanding tasks, using larger hidden size, more hidden layers, and more attention heads always leads to better performance. However, they stop at a hidden size of 1024, presumably because of the model size and computation cost problems.

学习自然语言表示已被证明对广泛 NLP 任务有用，并被广泛采用（Mikolov et al. 2013；Le & Mikolov 2014；Dai & Le 2015；Peters et al. 2018；Devlin et al. 2019；Radford et al. 2018；Radford et al. 2019）。过去两年最显著的变化之一，是从预训练词嵌入——无论是标准嵌入（Mikolov et al. 2013；Pennington et al. 2014）还是上下文嵌入（McCann et al. 2017；Peters et al. 2018）——转向全网络预训练再做任务特定微调（Dai & Le 2015；Radford et al. 2018；Devlin et al. 2019）。这一路线上常观察到更大模型带来更好表现。例如 Devlin et al. 2019 在三个选定的自然语言理解任务上表明：更大隐层、更多层、更多注意力头总是更好。但他们止步于隐层大小 1024，大概是因为模型规模与计算成本。

It is difficult to experiment with large models due to computational constraints, especially in terms of GPU/TPU memory limitations. Given that current state-of-the-art models often have hundreds of millions or even billions of parameters, we can easily hit memory limits. To address this issue, Chen et al. 2016 propose a method called gradient checkpointing to reduce the memory requirement to be sublinear at the cost of an extra forward pass. Gomez et al. 2017 propose a way to reconstruct each layer’s activations from the next layer so that they do not need to store the intermediate activations. Both methods reduce the memory consumption at the cost of speed. Raffel et al. 2019 proposed to use model parallelization to train a giant model. In contrast, our parameter-reduction techniques reduce memory consumption and increase training speed.

受计算约束——尤其是 GPU/TPU 显存——很难试验大模型。当前最优模型常有数亿甚至数十亿参数，很容易触顶。Chen et al. 2016 提出梯度检查点，用额外一次前向把显存需求降到亚线性；Gomez et al. 2017 提出从下一层重建各层激活，从而不必存中间激活。两者都以速度换显存。Raffel et al. 2019 用模型并行训练巨型模型。相对地，我们的参数缩减技术既降显存又提训练速度。

2.2 Cross-layer parameter sharing

2.2 跨层参数共享

The idea of sharing parameters across layers has been previously explored with the Transformer architecture (Vaswani et al. 2017), but this prior work has focused on training for standard encoder-decoder tasks rather than the pretraining/finetuning setting. Different from our observations, Dehghani et al. 2018 show that networks with cross-layer parameter sharing (Universal Transformer, UT) get better performance on language modeling and subject-verb agreement than the standard transformer. Very recently, Bai et al. 2019 propose a Deep Equilibrium Model (DQE) for transformer networks and show that DQE can reach an equilibrium point for which the input embedding and the output embedding of a certain layer stay the same. Our observations show that our embeddings are oscillating rather than converging. Hao et al. 2019 combine a parameter-sharing transformer with the standard one, which further increases the number of parameters of the standard transformer.

在 Transformer 架构（Vaswani et al. 2017）上跨层共享参数的想法此前已有探索，但多面向标准编码器–解码器任务，而非预训练/微调设定。与我们的观察不同，Dehghani et al. 2018 表明跨层参数共享网络（Universal Transformer，UT）在语言建模与主谓一致上优于标准 Transformer。Bai et al. 2019 最近提出面向 Transformer 的深度均衡模型（DQE），并表明 DQE 可到达某一均衡点，使某层的输入嵌入与输出嵌入相同。我们的观察是：嵌入在振荡而非收敛。Hao et al. 2019 把参数共享 Transformer 与标准 Transformer 结合，进一步增加了标准 Transformer 的参数量。

2.3 Sentence Ordering Objectives

2.3 句子排序目标

ALBERT uses a pretraining loss based on predicting the ordering of two consecutive segments of text. Several researchers have experimented with pretraining objectives that similarly relate to discourse coherence. Coherence and cohesion in discourse have been widely studied and many phenomena have been identified that connect neighboring text segments (Hobbs 1979; Halliday & Hasan 1976; Grosz et al. 1995). Most objectives found effective in practice are quite simple. Skip-thought (Kiros et al. 2015) and FastSent (Hill et al. 2016) sentence embeddings are learned by using an encoding of a sentence to predict words in neighboring sentences. Other objectives for sentence embedding learning include predicting future sentences rather than only neighbors (Gan et al. 2017) and predicting explicit discourse markers (Jernite et al. 2017; Nie et al. 2019). Our loss is most similar to the sentence ordering objective of Jernite et al. 2017, where sentence embeddings are learned in order to determine the ordering of two consecutive sentences. Unlike most of the above work, however, our loss is defined on textual segments rather than sentences. BERT (Devlin et al. 2019) uses a loss based on predicting whether the second segment in a pair has been swapped with a segment from another document. We compare to this loss in our experiments and find that sentence ordering is a more challenging pretraining task and more useful for certain downstream tasks. Concurrently to our work, Wang et al. 2019 also try to predict the order of two consecutive segments of text, but they combine it with the original next sentence prediction in a three-way classification task rather than empirically comparing the two.

ALBERT 使用基于预测两段连续文本顺序的预训练损失。已有研究尝试过类似话语连贯性的预训练目标。话语中的连贯与衔接被广泛研究，并识别出许多连接相邻文本段的现象（Hobbs 1979；Halliday & Hasan 1976；Grosz et al. 1995）。实践中有效的目标大多很简单。Skip-thought（Kiros et al. 2015）与 FastSent（Hill et al. 2016）通过用一句编码预测邻句词来学习句嵌入。其他目标包括预测更远的未来句而非仅邻句（Gan et al. 2017），以及预测显式话语标记（Jernite et al. 2017；Nie et al. 2019）。我们的损失最接近 Jernite et al. 2017 的句子排序目标：学习句嵌入以判断两连续句的顺序。但与上述多数工作不同，我们的损失定义在文本段而非句子上。BERT（Devlin et al. 2019）用损失预测配对中第二段是否被换成另一文档的片段。我们在实验中与该损失对比，发现句子排序是更难、对某些下游任务更有用的预训练任务。与本文同期，Wang et al. 2019 也尝试预测两连续文本段的顺序，但他们把它与原始下一句预测合成三分类，而非对二者做经验对比。

3 The Elements of ALBERT

3 ALBERT 的要素

In this section, we present the design decisions for ALBERT and provide quantified comparisons against corresponding configurations of the original BERT architecture (Devlin et al. 2019).

本节给出 ALBERT 的设计决策，并相对原始 BERT 架构（Devlin et al. 2019）的对应配置做量化比较。

3.1 Model architecture choices

3.1 模型架构选择

The backbone of the ALBERT architecture is similar to BERT in that it uses a transformer encoder (Vaswani et al. 2017) with GELU nonlinearities (Hendrycks & Gimpel 2016). We follow the BERT notation conventions and denote the vocabulary embedding size as $E$, the number of encoder layers as $L$, and the hidden size as $H$. Following Devlin et al. 2019, we set the feed-forward/filter size to be $4H$ and the number of attention heads to be $H/64$.

ALBERT 的主干与 BERT 类似：使用带 GELU 非线性（Hendrycks & Gimpel 2016）的 Transformer 编码器（Vaswani et al. 2017）。沿用 BERT 记号，词表嵌入大小记为 $E$，编码器层数记为 $L$，隐层大小记为 $H$。遵循 Devlin et al. 2019，前馈/滤波器大小设为 $4H$，注意力头数设为 $H/64$。

There are three main contributions that ALBERT makes over the design choices of BERT.

相对 BERT 的设计选择，ALBERT 主要有三点贡献。

Factorized embedding parameterization.

In BERT, as well as subsequent modeling improvements such as XLNet (Yang et al. 2019) and RoBERTa (Liu et al. 2019), the WordPiece embedding size $E$ is tied with the hidden layer size $H$, i.e., $E\equiv H$. This decision appears suboptimal for both modeling and practical reasons, as follows.

分解式嵌入参数化。

在 BERT 以及后续如 XLNet（Yang et al. 2019）、RoBERTa（Liu et al. 2019）中，WordPiece 嵌入大小 $E$ 与隐层大小 $H$ 绑定，即 $E\equiv H$。从建模与实践两方面看，这一决定都不理想，理由如下。

From a modeling perspective, WordPiece embeddings are meant to learn context-independent representations, whereas hidden-layer embeddings are meant to learn context-dependent representations. As experiments with context length indicate (Liu et al. 2019), the power of BERT-like representations comes from the use of context to provide the signal for learning such context-dependent representations. As such, untying the WordPiece embedding size $E$ from the hidden layer size $H$ allows us to make a more efficient usage of the total model parameters as informed by modeling needs, which dictate that $H\gg E$.

从建模看，WordPiece 嵌入应学上下文无关表示，而隐层嵌入应学上下文相关表示。上下文长度实验（Liu et al. 2019）表明，BERT 类表示的力量来自用上下文提供学习上下文相关表示的信号。因此，解开 WordPiece 嵌入大小 $E$ 与隐层大小 $H$，能按建模需要更高效地使用总参数——建模需要往往要求 $H\gg E$。

From a practical perspective, natural language processing usually require the vocabulary size $V$ to be large.[^1] If $E\equiv H$, then increasing $H$ increases the size of the embedding matrix, which has size $V\times E$. This can easily result in a model with billions of parameters, most of which are only updated sparsely during training.

从实践看，自然语言处理通常需要大词表大小 $V$。[^1] 若 $E\equiv H$，增大 $H$ 就会增大规模为 $V\times E$ 的嵌入矩阵，很容易得到数十亿参数的模型，且其中多数在训练中仅被稀疏更新。

Therefore, for ALBERT we use a factorization of the embedding parameters, decomposing them into two smaller matrices. Instead of projecting the one-hot vectors directly into the hidden space of size $H$, we first project them into a lower dimensional embedding space of size $E$, and then project it to the hidden space. By using this decomposition, we reduce the embedding parameters from $O(V\times H)$ to $O(V\times E+E\times H)$. This parameter reduction is significant when $H\gg E$. We choose to use the same E for all word pieces because they are much more evenly distributed across documents compared to whole-word embedding, where having different embedding size (Grave et al. 2017; Baevski & Auli 2018; Dai et al. 2019 ) for different words is important.

因此 ALBERT 对嵌入参数做分解，拆成两个更小的矩阵。不是把 one-hot 向量直接投到大小为 $H$ 的隐空间，而是先投到大小为 $E$ 的低维嵌入空间，再投到隐空间。由此嵌入参数从 $O(V\times H)$ 降到 $O(V\times E+E\times H)$。当 $H\gg E$ 时缩减显著。我们对所有 word piece 使用相同的 $E$，因为相对整词嵌入，它们在文档间分布更均匀；整词嵌入下对不同词使用不同嵌入维（Grave et al. 2017；Baevski & Auli 2018；Dai et al. 2019）更重要。

Cross-layer parameter sharing.

For ALBERT, we propose cross-layer parameter sharing as another way to improve parameter efficiency. There are multiple ways to share parameters, e.g., only sharing feed-forward network (FFN) parameters across layers, or only sharing attention parameters. The default decision for ALBERT is to share all parameters across layers. All our experiments use this default decision unless otherwise specified. We compare this design decision against other strategies in our experiments in Sec. 4.5.

跨层参数共享。

ALBERT 还提出跨层参数共享，以提升参数效率。共享方式有多种，例如只共享前馈网络（FFN）参数，或只共享注意力参数。ALBERT 的默认是跨层共享全部参数。除非另有说明，我们的实验都用这一默认。第 4.5 节会把它与其他策略对比。

Similar strategies have been explored by Dehghani et al. 2018 (Universal Transformer, UT) and Bai et al. 2019 (Deep Equilibrium Models, DQE) for Transformer networks. Different from our observations, Dehghani et al. 2018 show that UT outperforms a vanilla Transformer. Bai et al. 2019 show that their DQEs reach an equilibrium point for which the input and output embedding of a certain layer stay the same. Our measurement on the L2 distances and cosine similarity show that our embeddings are oscillating rather than converging.

Dehghani et al. 2018（Universal Transformer，UT）与 Bai et al. 2019（Deep Equilibrium Models，DQE）对 Transformer 网络探索过类似策略。与我们不同，Dehghani et al. 2018 表明 UT 优于普通 Transformer；Bai et al. 2019 表明其 DQE 到达均衡点，某层输入与输出嵌入相同。我们对 L2 距离与余弦相似度的测量显示，嵌入在振荡而非收敛。

Figure 1: The L2 distances and cosine similarity (in terms of degree) of the input and output embedding of each layer for BERT-large and ALBERT-large.

图 1：BERT-large 与 ALBERT-large 各层输入、输出嵌入的 L2 距离与余弦相似度（以角度计）。

Figure 1 shows the L2 distances and cosine similarity of the input and output embeddings for each layer, using BERT-large and ALBERT-large configurations (see Table 1). We observe that the transitions from layer to layer are much smoother for ALBERT than for BERT. These results show that weight-sharing has an effect on stabilizing network parameters. Although there is a drop for both metrics compared to BERT, they nevertheless do not converge to 0 even after 24 layers. This shows that the solution space for ALBERT parameters is very different from the one found by DQE.

图 1 给出 BERT-large 与 ALBERT-large 配置（见表 1）下各层输入、输出嵌入的 L2 距离与余弦相似度。我们观察到 ALBERT 的层间过渡远比 BERT 平滑。这说明权值共享对稳定网络参数有作用。两项指标相对 BERT 都有下降，但即便经过 24 层也不会收敛到 0。这表明 ALBERT 参数的解空间与 DQE 找到的解空间很不相同。

Inter-sentence coherence loss.

In addition to the masked language modeling (MLM) loss (Devlin et al. 2019), BERT uses an additional loss called next-sentence prediction (NSP). NSP is a binary classification loss for predicting whether two segments appear consecutively in the original text, as follows: positive examples are created by taking consecutive segments from the training corpus; negative examples are created by pairing segments from different documents; positive and negative examples are sampled with equal probability. The NSP objective was designed to improve performance on downstream tasks, such as natural language inference, that require reasoning about the relationship between sentence pairs. However, subsequent studies (Yang et al. 2019; Liu et al. 2019) found NSP’s impact unreliable and decided to eliminate it, a decision supported by an improvement in downstream task performance across several tasks.

句间连贯损失。

除掩码语言建模（MLM）损失（Devlin et al. 2019）外，BERT 还使用下一句预测（NSP）损失。NSP 是二分类损失，预测两段是否在原文中连续出现：正例取训练语料中的连续段；负例把来自不同文档的段配对；正负例等概率采样。NSP 旨在提升自然语言推理等需要推理句对关系的下游任务。但后续研究（Yang et al. 2019；Liu et al. 2019）发现 NSP 影响不稳，决定去掉它，并在若干任务上观察到下游表现提升。

We conjecture that the main reason behind NSP’s ineffectiveness is its lack of difficulty as a task, as compared to MLM. As formulated, NSP conflates topic prediction and coherence prediction in a single task[^2]. However, topic prediction is easier to learn compared to coherence prediction, and also overlaps more with what is learned using the MLM loss.

我们推测 NSP 效果不佳的主因是相对 MLM 任务难度不足。按当前形式，NSP 把主题预测与连贯预测混在同一任务里[^2]。主题预测比连贯预测更容易学，也与 MLM 已学到的内容重叠更多。

We maintain that inter-sentence modeling is an important aspect of language understanding, but we propose a loss based primarily on coherence. That is, for ALBERT, we use a sentence-order prediction (SOP) loss, which avoids topic prediction and instead focuses on modeling inter-sentence coherence. The SOP loss uses as positive examples the same technique as BERT (two consecutive segments from the same document), and as negative examples the same two consecutive segments but with their order swapped. This forces the model to learn finer-grained distinctions about discourse-level coherence properties. As we show in Sec. 4.6, it turns out that NSP cannot solve the SOP task at all (i.e., it ends up learning the easier topic-prediction signal, and performs at random-baseline level on the SOP task), while SOP can solve the NSP task to a reasonable degree, presumably based on analyzing misaligned coherence cues. As a result, ALBERT models consistently improve downstream task performance for multi-sentence encoding tasks.

我们仍认为句间建模是语言理解的重要方面，但提出主要基于连贯的损失。亦即 ALBERT 使用句子顺序预测（SOP）损失，避开主题预测，专注句间连贯。SOP 的正例与 BERT 相同（同一文档中两连续段），负例则是把同一对连续段的顺序对调。这迫使模型学习更细粒度的话语级连贯性质。如第 4.6 节所示，NSP 完全解不开 SOP 任务（即最终只学到更易的主题预测信号，在 SOP 上接近随机基线），而 SOP 能在合理程度上解开 NSP（大概靠分析错位的连贯线索）。因此 ALBERT 在多句编码任务上持续提升下游表现。

3.2 Model setup

3.2 模型设定

We present the differences between BERT and ALBERT models with comparable hyperparameter settings in Table 1. Due to the design choices discussed above, ALBERT models have much smaller parameter size compared to corresponding BERT models.

表 1 给出超参设定可比时 BERT 与 ALBERT 的差异。由于上述设计，ALBERT 相对对应 BERT 的参数量小得多。

Model	Parameters	Layers	Hidden	Embedding	Parameter-sharing
BERT	base	108M	12	768	768	False
large	334M	24	1024	1024	False
ALBERT	base	12M	12	768	128	True
large	18M	24	1024	128	True
xlarge	60M	24	2048	128	True
xxlarge	235M	12	4096	128	True

Table 1: The configurations of the main BERT and ALBERT models analyzed in this paper.

Model	Parameters	Layers	Hidden	Embedding	Parameter-sharing
BERT	base	108M	12	768	768	False
large	334M	24	1024	1024	False
ALBERT	base	12M	12	768	128	True
large	18M	24	1024	128	True
xlarge	60M	24	2048	128	True
xxlarge	235M	12	4096	128	True

表 1：本文分析的主要 BERT 与 ALBERT 模型配置。

For example, ALBERT-large has about 18x fewer parameters compared to BERT-large, 18M versus 334M. An ALBERT-xlarge configuration with $H=2048$ has only 60M parameters and an ALBERT-xxlarge configuration with $H=4096$ has 233M parameters, i.e., around 70% of BERT-large’s parameters. Note that for ALBERT-xxlarge, we mainly report results on a 12-layer network because a 24-layer network (with the same configuration) obtains similar results but is computationally more expensive.

例如 ALBERT-large 相对 BERT-large 参数约少 18 倍：18M 对 334M。$H=2048$ 的 ALBERT-xlarge 仅 60M 参数；$H=4096$ 的 ALBERT-xxlarge 有 233M 参数，约为 BERT-large 的 70%。注意对 ALBERT-xxlarge，我们主要报告 12 层网络结果，因为同配置 24 层结果相近但计算更贵。

This improvement in parameter efficiency is the most important advantage of ALBERT’s design choices. Before we can quantify this advantage, we need to introduce our experimental setup in more detail.

参数效率的提升是 ALBERT 设计选择最重要的优势。在量化这一优势之前，需要更细致地介绍实验设定。

4 Experimental Results

4 实验结果

4.1 Experimental Setup

4.1 实验设定

To keep the comparison as meaningful as possible, we follow the BERT (Devlin et al. 2019) setup in using the BookCorpus (Zhu et al. 2015) and English Wikipedia (Devlin et al. 2019) for pretraining baseline models. These two corpora consist of around 16GB of uncompressed text. We format our inputs as “[CLS] $x_{1}$ [SEP] $x_{2}$ [SEP]”, where $x_{1}=x_{1,1},x_{1,2}\cdots$ and $x_{2}=x_{1,1},x_{1,2}\cdots$ are two segments.[^3] We always limit the maximum input length to 512, and randomly generate input sequences shorter than 512 with a probability of 10%. Like BERT, we use a vocabulary size of 30,000, tokenized using SentencePiece (Kudo & Richardson 2018) as in XLNet (Yang et al. 2019).

为尽量公平对比，我们沿用 BERT（Devlin et al. 2019）设定，用 BookCorpus（Zhu et al. 2015）与英文维基百科（Devlin et al. 2019）预训练基线模型。两语料未压缩文本约 16GB。输入格式为 “[CLS] $x_{1}$ [SEP] $x_{2}$ [SEP]”，其中 $x_{1}=x_{1,1},x_{1,2}\cdots$ 与 $x_{2}=x_{1,1},x_{1,2}\cdots$ 为两段。[^3] 最大输入长度始终限制为 512，并以 10% 概率随机生成短于 512 的序列。与 BERT 一样，词表大小 30,000，采用 SentencePiece（Kudo & Richardson 2018）分词，与 XLNet（Yang et al. 2019）相同。

We generate masked inputs for the MLM targets using $n$-gram masking (Joshi et al. 2019), with the length of each $n$-gram mask selected randomly. The probability for the length $n$ is given by

我们用 $n$-gram 掩码（Joshi et al. 2019）生成 MLM 目标的掩码输入，每个 $n$-gram 掩码长度随机选取。长度 $n$ 的概率为

$$p(n)=\frac{1/n}{\sum_{k=1}^{N}{1/k}}$$

We set the maximum length of $n$-gram (i.e., $n$) to be 3 (i.e., the MLM target can consist of up to a 3-gram of complete words, such as “White House correspondents”).

我们将 $n$-gram 最大长度（即 $n$）设为 3（即 MLM 目标最多可由完整词组成的 3-gram，例如 “White House correspondents”）。

All the model updates use a batch size of 4096 and a Lamb optimizer with learning rate 0.00176 (You et al. 2019). We train all models for 125,000 steps unless otherwise specified. Training was done on Cloud TPU V3. The number of TPUs used for training ranged from 64 to 512, depending on model size.

所有模型更新使用 batch size 4096，以及学习率 0.00176 的 Lamb 优化器（You et al. 2019）。除非另有说明，所有模型训练 125,000 步。训练在 Cloud TPU V3 上进行，TPU 数量依模型规模从 64 到 512 不等。

The experimental setup described in this section is used for all of our own versions of BERT as well as ALBERT models, unless otherwise specified.

除非另有说明，本节设定用于我们自己的全部 BERT 版本以及 ALBERT 模型。

4.2 Evaluation Benchmarks

4.2 评测基准

4.2.1 Intrinsic Evaluation

To monitor the training progress, we create a development set based on the development sets from SQuAD and RACE using the same procedure as in Sec. 4.1. We report accuracies for both MLM and sentence classification tasks. Note that we only use this set to check how the model is converging; it has not been used in a way that would affect the performance of any downstream evaluation, such as via model selection.

4.2.1 内在评测

为监控训练进度，我们按第 4.1 节相同流程，基于 SQuAD 与 RACE 的开发集构造开发集，并报告 MLM 与句子分类任务的准确率。注意该集合仅用于检查收敛，不通过模型选择等方式影响任何下游评测表现。

4.2.2 Downstream Evaluation

Following Yang et al. 2019 and Liu et al. 2019, we evaluate our models on three popular benchmarks: The General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2018), two versions of the Stanford Question Answering Dataset (SQuAD; Rajpurkar et al. 2016; Rajpurkar et al. 2018), and the ReAding Comprehension from Examinations (RACE) dataset (Lai et al. 2017). For completeness, we provide description of these benchmarks in Appendix A.3. As in (Liu et al. 2019), we perform early stopping on the development sets, on which we report all comparisons except for our final comparisons based on the task leaderboards, for which we also report test set results. For GLUE datasets that have large variances on the dev set, we report median over 5 runs.

4.2.2 下游评测

遵循 Yang et al. 2019 与 Liu et al. 2019，我们在三个流行基准上评测：通用语言理解评测（GLUE）基准（Wang et al. 2018）、斯坦福问答数据集两个版本（SQuAD；Rajpurkar et al. 2016；Rajpurkar et al. 2018），以及考试阅读理解（RACE）数据集（Lai et al. 2017）。为完整起见，附录 A.3 给出这些基准的说明。与（Liu et al. 2019）一样，我们在开发集上做早停，除最终基于任务排行榜的对比（同时报告测试集）外，其余对比均报告开发集结果。对开发集方差较大的 GLUE 数据集，报告 5 次运行的中位数。

4.3 Overall Comparison between BERT and ALBERT

4.3 BERT 与 ALBERT 的总体比较

We are now ready to quantify the impact of the design choices described in Sec. 3, specifically the ones around parameter efficiency. The improvement in parameter efficiency showcases the most important advantage of ALBERT’s design choices, as shown in Table 2: with only around 70% of BERT-large’s parameters, ALBERT-xxlarge achieves significant improvements over BERT-large, as measured by the difference on development set scores for several representative downstream tasks: SQuAD v1.1 (+1.9%), SQuAD v2.0 (+3.1%), MNLI (+1.4%), SST-2 (+2.2%), and RACE (+8.4%).

现在可以量化第 3 节设计选择——尤其是参数效率——的影响。参数效率的提升展示了 ALBERT 设计最重要的优势，见表 2：仅用约 BERT-large 70% 的参数，ALBERT-xxlarge 在若干代表性下游任务开发集上显著优于 BERT-large：SQuAD v1.1（+1.9%）、SQuAD v2.0（+3.1%）、MNLI（+1.4%）、SST-2（+2.2%）、RACE（+8.4%）。

Another interesting observation is the speed of data throughput at training time under the same training configuration (same number of TPUs). Because of less communication and fewer computations, ALBERT models have higher data throughput compared to their corresponding BERT models. If we use BERT-large as the baseline, we observe that ALBERT-large is about 1.7 times faster in iterating through the data while ALBERT-xxlarge is about 3 times slower because of the larger structure.

另一有趣观察是：在相同训练配置（相同 TPU 数量）下的训练时数据吞吐速度。由于通信更少、计算更少，ALBERT 相对对应 BERT 有更高数据吞吐。以 BERT-large 为基线，ALBERT-large 遍历数据约快 1.7 倍，而 ALBERT-xxlarge 因结构更大约慢 3 倍。

Model	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg	Speedup
BERT	base	108M	90.4/83.2	80.4/77.6	84.5	92.8	68.2	82.3	4.7x
large	334M	92.2/85.5	85.0/82.2	86.6	93.0	73.9	85.2	1.0
ALBERT	base	12M	89.3/82.3	80.0/77.1	81.6	90.3	64.0	80.1	5.6x
large	18M	90.6/83.9	82.3/79.4	83.5	91.7	68.5	82.4	1.7x
xlarge	60M	92.5/86.1	86.1/83.1	86.4	92.4	74.8	85.5	0.6x
xxlarge	235M	94.1/88.3	88.1/85.1	88.0	95.2	82.3	88.7	0.3x

Table 2: Dev set results for models pretrained over BookCorpus and Wikipedia for 125k steps. Here and everywhere else, the Avg column is computed by averaging the scores of the downstream tasks to its left (the two numbers of F1 and EM for each SQuAD are first averaged).

Model	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg	Speedup
BERT	base	108M	90.4/83.2	80.4/77.6	84.5	92.8	68.2	82.3	4.7x
large	334M	92.2/85.5	85.0/82.2	86.6	93.0	73.9	85.2	1.0
ALBERT	base	12M	89.3/82.3	80.0/77.1	81.6	90.3	64.0	80.1	5.6x
large	18M	90.6/83.9	82.3/79.4	83.5	91.7	68.5	82.4	1.7x
xlarge	60M	92.5/86.1	86.1/83.1	86.4	92.4	74.8	85.5	0.6x
xxlarge	235M	94.1/88.3	88.1/85.1	88.0	95.2	82.3	88.7	0.3x

表 2：在 BookCorpus 与维基百科上预训练 125k 步的开发集结果。此处及后文 Avg 列为左侧下游任务分数的平均（每个 SQuAD 的 F1 与 EM 先取平均）。

Next, we perform ablation experiments that quantify the individual contribution of each of the design choices for ALBERT.

接下来做消融实验，量化 ALBERT 各设计选择的单独贡献。

4.4 Factorized Embedding Parameterization

4.4 分解式嵌入参数化

Table 3 shows the effect of changing the vocabulary embedding size $E$ using an ALBERT-base configuration setting (see Table 1), using the same set of representative downstream tasks. Under the non-shared condition (BERT-style), larger embedding sizes give better performance, but not by much. Under the all-shared condition (ALBERT-style), an embedding of size 128 appears to be the best. Based on these results, we use an embedding size $E=128$ in all future settings, as a necessary step to do further scaling.

表 3 给出在 ALBERT-base 配置（见表 1）下改变词表嵌入大小 $E$ 的影响，下游任务集合相同。在非共享条件（BERT 风格）下，更大嵌入带来更好表现，但增益有限。在全共享条件（ALBERT 风格）下，嵌入大小 128 表现最好。据此，后续设定一律使用 $E=128$，作为继续放大的必要一步。

Model	$E$	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
ALBERT base not-shared	64	87M	89.9/82.9	80.1/77.8	82.9	91.5	66.7	81.3
128	89M	89.9/82.8	80.3/77.3	83.7	91.5	67.9	81.7
256	93M	90.2/83.2	80.3/77.4	84.1	91.9	67.3	81.8
768	108M	90.4/83.2	80.4/77.6	84.5	92.8	68.2	82.3
ALBERT base all-shared	64	10M	88.7/81.4	77.5/74.8	80.8	89.4	63.5	79.0
128	12M	89.3/82.3	80.0/77.1	81.6	90.3	64.0	80.1
256	16M	88.8/81.5	79.1/76.3	81.5	90.3	63.4	79.6
768	31M	88.6/81.5	79.2/76.6	82.0	90.6	63.3	79.8

Table 3: The effect of vocabulary embedding size on the performance of ALBERT-base.

Model	$E$	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
ALBERT base not-shared	64	87M	89.9/82.9	80.1/77.8	82.9	91.5	66.7	81.3
128	89M	89.9/82.8	80.3/77.3	83.7	91.5	67.9	81.7
256	93M	90.2/83.2	80.3/77.4	84.1	91.9	67.3	81.8
768	108M	90.4/83.2	80.4/77.6	84.5	92.8	68.2	82.3
ALBERT base all-shared	64	10M	88.7/81.4	77.5/74.8	80.8	89.4	63.5	79.0
128	12M	89.3/82.3	80.0/77.1	81.6	90.3	64.0	80.1
256	16M	88.8/81.5	79.1/76.3	81.5	90.3	63.4	79.6
768	31M	88.6/81.5	79.2/76.6	82.0	90.6	63.3	79.8

表 3：词表嵌入大小对 ALBERT-base 表现的影响。

4.5 Cross-layer parameter sharing

4.5 跨层参数共享

Table 4 presents experiments for various cross-layer parameter-sharing strategies, using an ALBERT-base configuration (Table 1) with two embedding sizes ($E=768$ and $E=128$). We compare the all-shared strategy (ALBERT-style), the not-shared strategy (BERT-style), and intermediate strategies in which only the attention parameters are shared (but not the FNN ones) or only the FFN parameters are shared (but not the attention ones).

表 4 给出多种跨层参数共享策略的实验，使用 ALBERT-base 配置（表 1）及两种嵌入大小（$E=768$ 与 $E=128$）。我们比较全共享（ALBERT 风格）、不共享（BERT 风格），以及只共享注意力参数（不共享 FFN）或只共享 FFN 参数（不共享注意力）的中间策略。

The all-shared strategy hurts performance under both conditions, but it is less severe for $E=128$ (-1.5 on Avg) compared to $E=768$ (-2.5 on Avg). In addition, most of the performance drop appears to come from sharing the FFN-layer parameters, while sharing the attention parameters results in no drop when $E=128$ (+0.1 on Avg), and a slight drop when $E=768$ (-0.7 on Avg).

全共享在两种条件下都会伤性能，但对 $E=128$ 更轻（Avg 上 -1.5）相对 $E=768$（Avg 上 -2.5）。此外，多数性能下降来自共享 FFN 层参数；共享注意力参数在 $E=128$ 时几乎无降（Avg 上 +0.1），在 $E=768$ 时略降（Avg 上 -0.7）。

There are other strategies of sharing the parameters cross layers. For example, We can divide the $L$ layers into $N$ groups of size $M$, and each size-$M$ group shares parameters. Overall, our experimental results shows that the smaller the group size $M$ is, the better the performance we get. However, decreasing group size $M$ also dramatically increase the number of overall parameters. We choose all-shared strategy as our default choice.

还有其他跨层共享策略。例如，可将 $L$ 层分成大小为 $M$ 的 $N$ 组，每组内共享参数。总体实验表明：组大小 $M$ 越小，表现越好；但减小 $M$ 也会急剧增加总参数量。我们选择全共享作为默认。

Model	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
ALBERT base $E$=768	all-shared	31M	88.6/81.5	79.2/76.6	82.0	90.6	63.3	79.8
shared-attention	83M	89.9/82.7	80.0/77.2	84.0	91.4	67.7	81.6
shared-FFN	57M	89.2/82.1	78.2/75.4	81.5	90.8	62.6	79.5
not-shared	108M	90.4/83.2	80.4/77.6	84.5	92.8	68.2	82.3
ALBERT base $E$=128	all-shared	12M	89.3/82.3	80.0/77.1	82.0	90.3	64.0	80.1
shared-attention	64M	89.9/82.8	80.7/77.9	83.4	91.9	67.6	81.7
shared-FFN	38M	88.9/81.6	78.6/75.6	82.3	91.7	64.4	80.2
not-shared	89M	89.9/82.8	80.3/77.3	83.2	91.5	67.9	81.6

Table 4: The effect of cross-layer parameter-sharing strategies, ALBERT-base configuration.

Model	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
ALBERT base $E$=768	all-shared	31M	88.6/81.5	79.2/76.6	82.0	90.6	63.3	79.8
shared-attention	83M	89.9/82.7	80.0/77.2	84.0	91.4	67.7	81.6
shared-FFN	57M	89.2/82.1	78.2/75.4	81.5	90.8	62.6	79.5
not-shared	108M	90.4/83.2	80.4/77.6	84.5	92.8	68.2	82.3
ALBERT base $E$=128	all-shared	12M	89.3/82.3	80.0/77.1	82.0	90.3	64.0	80.1
shared-attention	64M	89.9/82.8	80.7/77.9	83.4	91.9	67.6	81.7
shared-FFN	38M	88.9/81.6	78.6/75.6	82.3	91.7	64.4	80.2
not-shared	89M	89.9/82.8	80.3/77.3	83.2	91.5	67.9	81.6

表 4：跨层参数共享策略的影响（ALBERT-base 配置）。

4.6 Sentence order prediction (SOP)

4.6 句子顺序预测（SOP）

We compare head-to-head three experimental conditions for the additional inter-sentence loss: none (XLNet- and RoBERTa-style), NSP (BERT-style), and SOP (ALBERT-style), using an ALBERT-base configuration. Results are shown in Table 5, both over intrinsic (accuracy for the MLM, NSP, and SOP tasks) and downstream tasks.

我们在额外句间损失上直接对比三种实验条件：无（XLNet 与 RoBERTa 风格）、NSP（BERT 风格）、SOP（ALBERT 风格），均用 ALBERT-base 配置。结果见表 5，涵盖内在任务（MLM、NSP、SOP 准确率）与下游任务。

	Intrinsic Tasks	Downstream Tasks
SP tasks	MLM	NSP	SOP	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
None	54.9	52.4	53.3	88.6/81.5	78.1/75.3	81.5	89.9	61.7	79.0
NSP	54.5	90.5	52.0	88.4/81.5	77.2/74.6	81.6	91.1	62.3	79.2
SOP	54.0	78.9	86.5	89.3/82.3	80.0/77.1	82.0	90.3	64.0	80.1

Table 5: The effect of sentence-prediction loss, NSP vs. SOP, on intrinsic and downstream tasks.

	Intrinsic Tasks	Downstream Tasks
SP tasks	MLM	NSP	SOP	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
None	54.9	52.4	53.3	88.6/81.5	78.1/75.3	81.5	89.9	61.7	79.0
NSP	54.5	90.5	52.0	88.4/81.5	77.2/74.6	81.6	91.1	62.3	79.2
SOP	54.0	78.9	86.5	89.3/82.3	80.0/77.1	82.0	90.3	64.0	80.1

表 5：句子预测损失 NSP 与 SOP 对内在任务与下游任务的影响。

The results on the intrinsic tasks reveal that the NSP loss brings no discriminative power to the SOP task (52.0% accuracy, similar to the random-guess performance for the “None” condition). This allows us to conclude that NSP ends up modeling only topic shift. In contrast, the SOP loss does solve the NSP task relatively well (78.9% accuracy), and the SOP task even better (86.5% accuracy). Even more importantly, the SOP loss appears to consistently improve downstream task performance for multi-sentence encoding tasks (around +1% for SQuAD1.1, +2% for SQuAD2.0, +1.7% for RACE), for an Avg score improvement of around +1%.

内在任务结果表明：NSP 损失对 SOP 任务几乎没有判别力（准确率 52.0%，与 “None” 条件的随机猜测相近）。由此可断定 NSP 最终只建模了主题迁移。相对地，SOP 损失能较好地解开 NSP 任务（准确率 78.9%），并更好解开 SOP 任务（准确率 86.5%）。更重要的是，SOP 损失似乎持续提升多句编码下游任务（SQuAD1.1 约 +1%，SQuAD2.0 约 +2%，RACE 约 +1.7%），Avg 约 +1%。

4.7 What if we train for the same amount of time?

4.7 若按相同训练时间呢？

The speed-up results in Table 2 indicate that data-throughput for BERT-large is about 3.17x higher compared to ALBERT-xxlarge. Since longer training usually leads to better performance, we perform a comparison in which, instead of controlling for data throughput (number of training steps), we control for the actual training time (i.e., let the models train for the same number of hours). In Table 6, we compare the performance of a BERT-large model after 400k training steps (after 34h of training), roughly equivalent with the amount of time needed to train an ALBERT-xxlarge model with 125k training steps (32h of training).

表 2 的加速结果表明：相对 ALBERT-xxlarge，BERT-large 的数据吞吐约高 3.17 倍。由于更长训练通常带来更好表现，我们做了另一对比：不控制数据吞吐（训练步数），而控制实际训练时间（即让模型训练相同小时数）。表 6 比较训练 400k 步后的 BERT-large（训练约 34 小时），大致等价于训练 125k 步的 ALBERT-xxlarge（训练约 32 小时）。

Models	Steps	Time	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
BERT-large	400k	34h	93.5/87.4	86.9/84.3	87.8	94.6	77.3	87.2
ALBERT-xxlarge	125k	32h	94.0/88.1	88.3/85.3	87.8	95.4	82.5	88.7

Table 6: The effect of controlling for training time, BERT-large vs ALBERT-xxlarge configurations.

Models	Steps	Time	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
BERT-large	400k	34h	93.5/87.4	86.9/84.3	87.8	94.6	77.3	87.2
ALBERT-xxlarge	125k	32h	94.0/88.1	88.3/85.3	87.8	95.4	82.5	88.7

表 6：控制训练时间的影响：BERT-large 与 ALBERT-xxlarge 配置。

After training for roughly the same amount of time, ALBERT-xxlarge is significantly better than BERT-large: +1.5% better on Avg, with the difference on RACE as high as +5.2%.

在大致相同训练时间后，ALBERT-xxlarge 显著优于 BERT-large：Avg 上好 +1.5%，RACE 上差距高达 +5.2%。

4.8 Additional training data and dropout effects

4.8 额外训练数据与 dropout 影响

The experiments done up to this point use only the Wikipedia and BookCorpus datasets, as in (Devlin et al. 2019). In this section, we report measurements on the impact of the additional data used by both XLNet (Yang et al. 2019) and RoBERTa (Liu et al. 2019).

此前实验仅使用维基百科与 BookCorpus，与（Devlin et al. 2019）相同。本节报告 XLNet（Yang et al. 2019）与 RoBERTa（Liu et al. 2019）所用额外数据的影响。

Fig. 2(a) plots the dev set MLM accuracy under two conditions, without and with additional data, with the latter condition giving a significant boost. We also observe performance improvements on the downstream tasks in Table 7, except for the SQuAD benchmarks (which are Wikipedia-based, and therefore are negatively affected by out-of-domain training material).

图 2(a) 画出有无额外数据两种条件下的开发集 MLM 准确率，后者有显著提升。表 7 中下游任务也有改进，但 SQuAD 基准除外（基于维基百科，因此受域外训练材料负面影响）。

(a) Adding data

(a) 增加数据

	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
No additional data	89.3/82.3	80.0/77.1	81.6	90.3	64.0	80.1
With additional data	88.8/81.7	79.1/76.3	82.4	92.8	66.0	80.8

Table 7: The effect of additional training data using the ALBERT-base configuration.

	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
No additional data	89.3/82.3	80.0/77.1	81.6	90.3	64.0	80.1
With additional data	88.8/81.7	79.1/76.3	82.4	92.8	66.0	80.8

表 7：使用 ALBERT-base 配置时额外训练数据的影响。

We also note that, even after training for 1M steps, our largest models still do not overfit to their training data. As a result, we decide to remove dropout to further increase our model capacity. The plot in Fig. 2(b) shows that removing dropout significantly improves MLM accuracy. Intermediate evaluation on ALBERT-xxlarge at around 1M training steps (Table 8) also confirms that removing dropout helps the downstream tasks. There is empirical (Szegedy et al. 2017) and theoretical (Li et al. 2019) evidence showing that a combination of batch normalization and dropout in Convolutional Neural Networks may have harmful results. To the best of our knowledge, we are the first to show that dropout can hurt performance in large Transformer-based models. However, the underlying network structure of ALBERT is a special case of the transformer and further experimentation is needed to see if this phenomenon appears with other transformer-based architectures or not.

我们还注意到：即便训练到 1M 步，最大模型仍未过拟合训练数据。因此我们去掉 dropout 以进一步提升模型容量。图 2(b) 显示去掉 dropout 显著提升 MLM 准确率。约 1M 训练步时对 ALBERT-xxlarge 的中间评测（表 8）也证实去掉 dropout 有助于下游任务。已有经验（Szegedy et al. 2017）与理论（Li et al. 2019）证据表明，卷积神经网络中批归一化与 dropout 组合可能有害。据我们所知，我们首次表明 dropout 可能伤及大型 Transformer 类模型的表现。但 ALBERT 的底层网络结构是 Transformer 的特例，是否在其他 Transformer 架构上也出现这一现象，还需进一步实验。

	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
With dropout	94.7/89.2	89.6/86.9	90.0	96.3	85.7	90.4
Without dropout	94.8/89.5	89.9/87.2	90.4	96.5	86.1	90.7

Table 8: The effect of removing dropout, measured for an ALBERT-xxlarge configuration.

	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
With dropout	94.7/89.2	89.6/86.9	90.0	96.3	85.7	90.4
Without dropout	94.8/89.5	89.9/87.2	90.4	96.5	86.1	90.7

表 8：去掉 dropout 的影响（ALBERT-xxlarge 配置）。

4.9 Current State-of-the-art on NLU Tasks

4.9 NLU 任务上的当前最优

The results we report in this section make use of the training data used by Devlin et al. 2019, as well as the additional data used by Liu et al. 2019 and Yang et al. 2019. We report state-of-the-art results under two settings for fine-tuning: single-model and ensembles. In both settings, we only do single-task fine-tuning[^4]. Following Liu et al. 2019, on the development set we report the median result over five runs.

本节结果同时使用 Devlin et al. 2019 的训练数据，以及 Liu et al. 2019 与 Yang et al. 2019 的额外数据。我们在两种微调设定下报告最优结果：单模型与集成。两种设定都只做单任务微调[^4]。遵循 Liu et al. 2019，开发集上报告五次运行的中位数。

Models	MNLI	QNLI	QQP	RTE	SST	MRPC	CoLA	STS	WNLI	Avg
Single-task single models on dev
BERT-large	86.6	92.3	91.3	70.4	93.2	88.0	60.6	90.0	-	-
XLNet-large	89.8	93.9	91.8	83.8	95.6	89.2	63.6	91.8	-	-
RoBERTa-large	90.2	94.7	92.2	86.6	96.4	90.9	68.0	92.4	-	-
ALBERT (1M)	90.4	95.2	92.0	88.1	96.8	90.2	68.7	92.7	-	-
ALBERT (1.5M)	90.8	95.3	92.2	89.2	96.9	90.9	71.4	93.0	-	-
Ensembles on test (from leaderboard as of Sept. 16, 2019)
ALICE	88.2	95.7	90.7	83.5	95.2	92.6	69.2	91.1	80.8	87.0
MT-DNN	87.9	96.0	89.9	86.3	96.5	92.7	68.4	91.1	89.0	87.6
XLNet	90.2	98.6	90.3	86.3	96.8	93.0	67.8	91.6	90.4	88.4
RoBERTa	90.8	98.9	90.2	88.2	96.7	92.3	67.8	92.2	89.0	88.5
Adv-RoBERTa	91.1	98.8	90.3	88.7	96.8	93.1	68.0	92.4	89.0	88.8
ALBERT	91.3	99.2	90.5	89.2	97.1	93.4	69.1	92.5	91.8	89.4

Table 9: State-of-the-art results on the GLUE benchmark. For single-task single-model results, we report ALBERT at 1M steps (comparable to RoBERTa) and at 1.5M steps. The ALBERT ensemble uses models trained with 1M, 1.5M, and other numbers of steps.

Models	MNLI	QNLI	QQP	RTE	SST	MRPC	CoLA	STS	WNLI	Avg
Single-task single models on dev
BERT-large	86.6	92.3	91.3	70.4	93.2	88.0	60.6	90.0	-	-
XLNet-large	89.8	93.9	91.8	83.8	95.6	89.2	63.6	91.8	-	-
RoBERTa-large	90.2	94.7	92.2	86.6	96.4	90.9	68.0	92.4	-	-
ALBERT (1M)	90.4	95.2	92.0	88.1	96.8	90.2	68.7	92.7	-	-
ALBERT (1.5M)	90.8	95.3	92.2	89.2	96.9	90.9	71.4	93.0	-	-
Ensembles on test (from leaderboard as of Sept. 16, 2019)
ALICE	88.2	95.7	90.7	83.5	95.2	92.6	69.2	91.1	80.8	87.0
MT-DNN	87.9	96.0	89.9	86.3	96.5	92.7	68.4	91.1	89.0	87.6
XLNet	90.2	98.6	90.3	86.3	96.8	93.0	67.8	91.6	90.4	88.4
RoBERTa	90.8	98.9	90.2	88.2	96.7	92.3	67.8	92.2	89.0	88.5
Adv-RoBERTa	91.1	98.8	90.3	88.7	96.8	93.1	68.0	92.4	89.0	88.8
ALBERT	91.3	99.2	90.5	89.2	97.1	93.4	69.1	92.5	91.8	89.4

表 9：GLUE 基准上的最优结果。单任务单模型结果中，我们报告训练 1M 步（可与 RoBERTa 比）与 1.5M 步的 ALBERT。ALBERT 集成使用 1M、1.5M 及其他步数训练的模型。

The single-model ALBERT configuration incorporates the best-performing settings discussed: an ALBERT-xxlarge configuration (Table 1) using combined MLM and SOP losses, and no dropout. The checkpoints that contribute to the final ensemble model are selected based on development set performance; the number of checkpoints considered for this selection range from 6 to 17, depending on the task. For the GLUE (Table 9) and RACE (Table 10) benchmarks, we average the model predictions for the ensemble models, where the candidates are fine-tuned from different training steps using the 12-layer and 24-layer architectures. For SQuAD (Table 10), we average the prediction scores for those spans that have multiple probabilities; we also average the scores of the “unanswerable” decision.

单模型 ALBERT 配置采用前述最佳设定：ALBERT-xxlarge 配置（表 1），联合 MLM 与 SOP 损失，且无 dropout。进入最终集成的检查点按开发集表现选取；依任务不同，参与选择的检查点数量从 6 到 17 不等。对 GLUE（表 9）与 RACE（表 10），集成对来自不同训练步、12 层与 24 层架构微调得到的候选做预测平均。对 SQuAD（表 10），对有多个概率的跨度做预测分数平均；对“不可回答”决策的分数也取平均。

Both single-model and ensemble results indicate that ALBERT improves the state-of-the-art significantly for all three benchmarks, achieving a GLUE score of 89.4, a SQuAD 2.0 test F1 score of 92.2, and a RACE test accuracy of 89.4. The latter appears to be a particularly strong improvement, a jump of +17.4% absolute points over BERT (Devlin et al. 2019; Clark et al. 2019), +7.6% over XLNet (Yang et al. 2019), +6.2% over RoBERTa (Liu et al. 2019), and 5.3% over DCMI+ (Zhang et al. 2019), an ensemble of multiple models specifically designed for reading comprehension tasks. Our single model achieves an accuracy of $86.5\%$, which is still $2.4\%$ better than the state-of-the-art ensemble model.

单模型与集成结果均表明 ALBERT 在三个基准上显著提升最优水平：GLUE 分数 89.4，SQuAD 2.0 测试 F1 92.2，RACE 测试准确率 89.4。后者提升尤为突出：相对 BERT（Devlin et al. 2019；Clark et al. 2019）绝对提升 +17.4%，相对 XLNet（Yang et al. 2019）+7.6%，相对 RoBERTa（Liu et al. 2019）+6.2%，相对专为阅读理解设计的多模型集成 DCMI+（Zhang et al. 2019）+5.3%。我们的单模型准确率为 $86.5\%$，仍比当时最优集成高 $2.4\%$。

Models	SQuAD1.1 dev	SQuAD2.0 dev	SQuAD2.0 test	RACE test (Middle/High)
Single model (from leaderboard as of Sept. 23, 2019)
BERT-large	90.9/84.1	81.8/79.0	89.1/86.3	72.0 (76.6/70.1)
XLNet	94.5/89.0	88.8/86.1	89.1/86.3	81.8 (85.5/80.2)
RoBERTa	94.6/88.9	89.4/86.5	89.8/86.8	83.2 (86.5/81.3)
UPM	-	-	89.9/87.2	-
XLNet + SG-Net Verifier++	-	-	90.1/87.2	-
ALBERT (1M)	94.8/89.2	89.9/87.2	-	86.0 (88.2/85.1)
ALBERT (1.5M)	94.8/89.3	90.2/87.4	90.9/88.1	86.5 (89.0/85.5)
Ensembles (from leaderboard as of Sept. 23, 2019)
BERT-large	92.2/86.2	-	-	-
XLNet + SG-Net Verifier	-	-	90.7/88.2	-
UPM	-	-	90.7/88.2
XLNet + DAAF + Verifier	-	-	90.9/88.6	-
DCMN+	-	-	-	84.1 (88.5/82.3)
ALBERT	95.5/90.1	91.4/88.9	92.2/89.7	89.4 (91.2/88.6)

Table 10: State-of-the-art results on the SQuAD and RACE benchmarks.

Models	SQuAD1.1 dev	SQuAD2.0 dev	SQuAD2.0 test	RACE test (Middle/High)
Single model (from leaderboard as of Sept. 23, 2019)
BERT-large	90.9/84.1	81.8/79.0	89.1/86.3	72.0 (76.6/70.1)
XLNet	94.5/89.0	88.8/86.1	89.1/86.3	81.8 (85.5/80.2)
RoBERTa	94.6/88.9	89.4/86.5	89.8/86.8	83.2 (86.5/81.3)
UPM	-	-	89.9/87.2	-
XLNet + SG-Net Verifier++	-	-	90.1/87.2	-
ALBERT (1M)	94.8/89.2	89.9/87.2	-	86.0 (88.2/85.1)
ALBERT (1.5M)	94.8/89.3	90.2/87.4	90.9/88.1	86.5 (89.0/85.5)
Ensembles (from leaderboard as of Sept. 23, 2019)
BERT-large	92.2/86.2	-	-	-
XLNet + SG-Net Verifier	-	-	90.7/88.2	-
UPM	-	-	90.7/88.2
XLNet + DAAF + Verifier	-	-	90.9/88.6	-
DCMN+	-	-	-	84.1 (88.5/82.3)
ALBERT	95.5/90.1	91.4/88.9	92.2/89.7	89.4 (91.2/88.6)

表 10：SQuAD 与 RACE 基准上的最优结果。

5 Discussion

5 讨论

While ALBERT-xxlarge has less parameters than BERT-large and gets significantly better results, it is computationally more expensive due to its larger structure. An important next step is thus to speed up the training and inference speed of ALBERT through methods like sparse attention (Child et al. 2019) and block attention (Shen et al. 2018). An orthogonal line of research, which could provide additional representation power, includes hard example mining (Mikolov et al. 2013) and more efficient language modeling training (Yang et al. 2019). Additionally, although we have convincing evidence that sentence order prediction is a more consistently-useful learning task that leads to better language representations, we hypothesize that there could be more dimensions not yet captured by the current self-supervised training losses that could create additional representation power for the resulting representations.

尽管 ALBERT-xxlarge 参数少于 BERT-large 且结果显著更好，但因其更大结构，计算更昂贵。重要的下一步是通过稀疏注意力（Child et al. 2019）、块注意力（Shen et al. 2018）等方法加速 ALBERT 的训练与推理。正交研究方向——可能提供额外表示能力——包括难例挖掘（Mikolov et al. 2013）与更高效的语言建模训练（Yang et al. 2019）。此外，尽管我们有令人信服的证据表明句子顺序预测是更稳定有用、能带来更好语言表示的学习任务，我们仍假设当前自监督训练损失尚未捕捉的更多维度，可能为最终表示创造额外能力。

Acknowledgement

致谢

The authors would like to thank Beer Changpinyo, Nan Ding, Noam Shazeer, and Tomer Levinboim for discussion and providing useful feedback on the project; Omer Levy and Naman Goyal for clarifying experimental setup for RoBERTa; Zihang Dai for clarifying XLNet; Brandon Norick, Emma Strubell, Shaojie Bai, Chas Leichner, and Sachin Mehta for providing useful feedback on the paper; Jacob Devlin for providing the English and multilingual version of training data; Liang Xu, Chenjie Cao and the CLUE community for providing the training data and evaluation benechmark of the Chinese version of ALBERT models.

作者感谢 Beer Changpinyo、Nan Ding、Noam Shazeer 与 Tomer Levinboim 的讨论及对项目的有益反馈；感谢 Omer Levy 与 Naman Goyal 澄清 RoBERTa 实验设定；感谢 Zihang Dai 澄清 XLNet；感谢 Brandon Norick、Emma Strubell、Shaojie Bai、Chas Leichner 与 Sachin Mehta 对论文的有益反馈；感谢 Jacob Devlin 提供英文与多语版训练数据；感谢 Liang Xu、Chenjie Cao 与 CLUE 社区提供中文版 ALBERT 模型的训练数据与评测基准。

References

参考文献

Baevski & Auli (2018) Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853 , 2018.

Bai et al. (2019) Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In Neural Information Processing Systems (NeurIPS) , 2019.

Bar-Haim et al. (2006) Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second PASCAL recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment , volume 6, pp. 6–4. Venice, 2006.

Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In TAC , 2009.

Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , pp. 1–14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2001 . URL https://www.aclweb.org/anthology/S17-2001 .

Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016.

Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019.

Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc V Le. Bam! born-again multi-task networks for natural language understanding. arXiv preprint arXiv:1907.04829 , 2019.

Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop , pp. 177–190. Springer, 2005.

Dai & Le (2015) Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems , pp. 3079–3087, 2015.

Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 , 2019.

Dehghani et al. (2018) Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819 , 2018.

Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423 . URL https://www.aclweb.org/anthology/N19-1423 .

Dolan & Brockett (2005) William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005) , 2005. URL https://www.aclweb.org/anthology/I05-5002 .

Gan et al. (2017) Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, Xiaodong He, and Lawrence Carin. Learning generic sentence representations using convolutional neural networks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pp. 2390–2400, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1254 . URL https://www.aclweb.org/anthology/D17-1254 .

Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing , pp. 1–9, Prague, June 2007. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/W07-1401 .

Gomez et al. (2017) Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Advances in neural information processing systems , pp. 2214–2224, 2017.

Gong et al. (2019) Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International Conference on Machine Learning , pp. 2337–2346, 2019.

Grave et al. (2017) Edouard Grave, Armand Joulin, Moustapha Cissé, Hervé Jégou, et al. Efficient softmax approximation for gpus. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pp. 1302–1310. JMLR. org, 2017.

Grosz et al. (1995) Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics , 21(2):203–225, 1995. URL https://www.aclweb.org/anthology/J95-2003 .

Halliday & Hasan (1976) M.A.K. Halliday and Ruqaiya Hasan. Cohesion in English . Routledge, 1976.

Hao et al. (2019) Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. Modeling recurrence for transformer. Proceedings of the 2019 Conference of the North , 2019. doi: 10.18653/v1/n19-1122 . URL http://dx.doi.org/10.18653/v1/n19-1122 .

Hendrycks & Gimpel (2016) Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.08415 , 2016.

Hill et al. (2016) Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 1367–1377. Association for Computational Linguistics, 2016. doi: 10.18653/v1/N16-1162 . URL http://aclweb.org/anthology/N16-1162 .

Hobbs (1979) Jerry R. Hobbs. Coherence and coreference. Cognitive Science , 3(1):67–90, 1979.

Howard & Ruder (2018) Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 , 2018.

Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. First quora dataset release: Question pairs, January 2017. URL https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs .

Jernite et al. (2017) Yacine Jernite, Samuel R Bowman, and David Sontag. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557 , 2017.

Joshi et al. (2019) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529 , 2019.

Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Skip-thought vectors. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 , NIPS’15, pp. 3294–3302, Cambridge, MA, USA, 2015. MIT Press. URL http://dl.acm.org/citation.cfm?id=2969442.2969607 .

Kudo & Richardson (2018) Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 66–71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-2012 . URL https://www.aclweb.org/anthology/D18-2012 .

Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pp. 785–794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1082 . URL https://www.aclweb.org/anthology/D17-1082 .

Le & Mikolov (2014) Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In Proceedings of the 31st ICML , Beijing, China, 2014.

Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning , 2012.

Li et al. (2019) Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the disharmony between dropout and batch normalization by variance shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 2682–2690, 2019.

Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

McCann et al. (2017) Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30 , pp. 6294–6305. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/7209-learned-in-translation-contextualized-word-vectors.pdf .

Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems , pp. 3111–3119, 2013.

Nie et al. (2019) Allen Nie, Erin Bennett, and Noah Goodman. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 4497–4510, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1442 . URL https://www.aclweb.org/anthology/P19-1442 .

Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pp. 1532–1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1162 . URL https://www.aclweb.org/anthology/D14-1162 .

Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pp. 2227–2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1202 . URL https://www.aclweb.org/anthology/N18-1202 .

Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf , 2018.

Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8), 2019.

Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 , 2019.

Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pp. 2383–2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264 . URL https://www.aclweb.org/anthology/D16-1264 .

Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pp. 784–789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124 . URL https://www.aclweb.org/anthology/P18-2124 .

Shazeer et al. (2018) Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems , pp. 10414–10423, 2018.

Shen et al. (2018) Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. Bi-directional block self-attention for fast and memory-efficient sequence modeling. arXiv preprint arXiv:1804.00857 , 2018.

Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism, 2019.

Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pp. 1631–1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/D13-1170 .

Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. arXiv preprint arXiv:1908.09355 , 2019.

Szegedy et al. (2017) Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Conference on Artificial Intelligence , 2017.

Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. arXiv preprint arXiv:1908.08962 , 2019.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pp. 5998–6008, 2017.

Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pp. 353–355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-5446 . URL https://www.aclweb.org/anthology/W18-5446 .

Wang et al. (2019) Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. StructBERT: Incorporating language structures into pre-training for deep language understanding. arXiv preprint arXiv:1908.04577 , 2019.

Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471 , 2018.

Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pp. 1112–1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1101 . URL https://www.aclweb.org/anthology/N18-1101 .

Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 , 2019.

You et al. (2019) Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Reducing BERT pre-training time from 3 days to 76 minutes. arXiv preprint arXiv:1904.00962 , 2019.

Zhang et al. (2019) Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. DCMN+: Dual co-matching network for multi-choice reading comprehension. arXiv preprint arXiv:1908.11511 , 2019.

Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision , pp. 19–27, 2015.

Appendix A Appendix

附录 A 附录

A.1 Effect of network depth and width

A.1 网络深度与宽度的影响

In this section, we check how depth (number of layers) and width (hidden size) affect the performance of ALBERT. Table 11 shows the performance of an ALBERT-large configuration (see Table 1) using different numbers of layers. Networks with 3 or more layers are trained by fine-tuning using the parameters from the depth before (e.g., the 12-layer network parameters are fine-tuned from the checkpoint of the 6-layer network parameters).[^5] Similar technique has been used in Gong et al. 2019. If we compare a 3-layer ALBERT model with a 1-layer ALBERT model, although they have the same number of parameters, the performance increases significantly. However, there are diminishing returns when continuing to increase the number of layers: the results of a 12-layer network are relatively close to the results of a 24-layer network, and the performance of a 48-layer network appears to decline.

本节检查深度（层数）与宽度（隐层大小）如何影响 ALBERT 表现。表 11 给出 ALBERT-large 配置（见表 1）在不同层数下的表现。3 层及以上网络由上一深度的参数微调得到（例如 12 层网络参数由 6 层检查点微调）。[^5] Gong et al. 2019 用过类似技术。比较 3 层与 1 层 ALBERT：尽管参数量相同，表现显著上升。但继续加深收益递减：12 层与 24 层结果相对接近，48 层表现似乎下降。

Number of layers	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
1	18M	31.1/22.9	50.1/50.1	66.4	80.8	40.1	52.9
3	18M	79.8/69.7	64.4/61.7	77.7	86.7	54.0	71.2
6	18M	86.4/78.4	73.8/71.1	81.2	88.9	60.9	77.2
12	18M	89.8/83.3	80.7/77.9	83.3	91.7	66.7	81.5
24	18M	90.3/83.3	81.8/79.0	83.3	91.5	68.7	82.1
48	18M	90.0/83.1	81.8/78.9	83.4	91.9	66.9	81.8

Table 11: The effect of increasing the number of layers for an ALBERT-large configuration.

Number of layers	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
1	18M	31.1/22.9	50.1/50.1	66.4	80.8	40.1	52.9
3	18M	79.8/69.7	64.4/61.7	77.7	86.7	54.0	71.2
6	18M	86.4/78.4	73.8/71.1	81.2	88.9	60.9	77.2
12	18M	89.8/83.3	80.7/77.9	83.3	91.7	66.7	81.5
24	18M	90.3/83.3	81.8/79.0	83.3	91.5	68.7	82.1
48	18M	90.0/83.1	81.8/78.9	83.4	91.9	66.9	81.8

表 11：增加层数对 ALBERT-large 配置的影响。

A similar phenomenon, this time for width, can be seen in Table 12 for a 3-layer ALBERT-large configuration. As we increase the hidden size, we get an increase in performance with diminishing returns. At a hidden size of 6144, the performance appears to decline significantly. We note that none of these models appear to overfit the training data, and they all have higher training and development loss compared to the best-performing ALBERT configurations.

类似现象也出现在宽度上，见表 12 对 3 层 ALBERT-large 配置的结果。增大隐层大小带来性能提升，但收益递减。隐层大小 6144 时表现显著下降。我们注意到这些模型都未过拟合训练数据，且相对最佳 ALBERT 配置，它们的训练与开发损失都更高。

Hidden size	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
1024	18M	79.8/69.7	64.4/61.7	77.7	86.7	54.0	71.2
2048	60M	83.3/74.1	69.1/66.6	79.7	88.6	58.2	74.6
4096	225M	85.0/76.4	71.0/68.1	80.3	90.4	60.4	76.3
6144	499M	84.7/75.8	67.8/65.4	78.1	89.1	56.0	74.0

Table 12: The effect of increasing the hidden-layer size for an ALBERT-large 3-layer configuration.

Hidden size	Parameters	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
1024	18M	79.8/69.7	64.4/61.7	77.7	86.7	54.0	71.2
2048	60M	83.3/74.1	69.1/66.6	79.7	88.6	58.2	74.6
4096	225M	85.0/76.4	71.0/68.1	80.3	90.4	60.4	76.3
6144	499M	84.7/75.8	67.8/65.4	78.1	89.1	56.0	74.0

表 12：增大隐层大小对 3 层 ALBERT-large 配置的影响。

A.2 Do very wide ALBERT models need to be deep(er) too?

A.2 很宽的 ALBERT 是否也需要更深？

In Section A.1, we show that for ALBERT-large ($H$=1024), the difference between a 12-layer and a 24-layer configuration is small. Does this result still hold for much wider ALBERT configurations, such as ALBERT-xxlarge ($H$=4096)?

附录 A.1 表明，对 ALBERT-large（$H$=1024），12 层与 24 层差异很小。对更宽的配置如 ALBERT-xxlarge（$H$=4096），这一结论是否仍成立？

Number of layers	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
12	94.0/88.1	88.3/85.3	87.8	95.4	82.5	88.7
24	94.1/88.3	88.1/85.1	88.0	95.2	82.3	88.7

Table 13: The effect of a deeper network using an ALBERT-xxlarge configuration.

Number of layers	SQuAD1.1	SQuAD2.0	MNLI	SST-2	RACE	Avg
12	94.0/88.1	88.3/85.3	87.8	95.4	82.5	88.7
24	94.1/88.3	88.1/85.1	88.0	95.2	82.3	88.7

表 13：更深网络对 ALBERT-xxlarge 配置的影响。

The answer is given by the results from Table 13. The difference between 12-layer and 24-layer ALBERT-xxlarge configurations in terms of downstream accuracy is negligible, with the Avg score being the same. We conclude that, when sharing all cross-layer parameters (ALBERT-style), there is no need for models deeper than a 12-layer configuration.

答案见表 13。12 层与 24 层 ALBERT-xxlarge 在下游准确率上差异可忽略，Avg 相同。我们断定：在跨层全参数共享（ALBERT 风格）时，无需比 12 层更深的模型。

A.3 Downstream Evaluation Tasks

A.3 下游评测任务

GLUE

GLUE is comprised of 9 tasks, namely Corpus of Linguistic Acceptability (CoLA; Warstadt et al. 2018), Stanford Sentiment Treebank (SST; Socher et al. 2013), Microsoft Research Paraphrase Corpus (MRPC; Dolan & Brockett 2005), Semantic Textual Similarity Benchmark (STS; Cer et al. 2017), Quora Question Pairs (QQP; Iyer et al. 2017), Multi-Genre NLI (MNLI; Williams et al. 2018), Question NLI (QNLI; Rajpurkar et al. 2016), Recognizing Textual Entailment (RTE; Dagan et al. 2005; Bar-Haim et al. 2006; Giampiccolo et al. 2007; Bentivogli et al. 2009) and Winograd NLI (WNLI; Levesque et al. 2012). It focuses on evaluating model capabilities for natural language understanding. When reporting MNLI results, we only report the “match” condition (MNLI-m). We follow the finetuning procedures from prior work (Devlin et al. 2019; Liu et al. 2019; Yang et al. 2019) and report the held-out test set performance obtained from GLUE submissions. For test set submissions, we perform task-specific modifications for WNLI and QNLI as described by Liu et al. 2019 and Yang et al. 2019.

GLUE

GLUE 包含 9 个任务：语言可接受性语料（CoLA；Warstadt et al. 2018）、斯坦福情感树库（SST；Socher et al. 2013）、微软研究复述语料（MRPC；Dolan & Brockett 2005）、语义文本相似度基准（STS；Cer et al. 2017）、Quora 问句对（QQP；Iyer et al. 2017）、多体裁 NLI（MNLI；Williams et al. 2018）、问题 NLI（QNLI；Rajpurkar et al. 2016）、文本蕴含识别（RTE；Dagan et al. 2005；Bar-Haim et al. 2006；Giampiccolo et al. 2007；Bentivogli et al. 2009）以及 Winograd NLI（WNLI；Levesque et al. 2012）。它侧重评测自然语言理解能力。报告 MNLI 时只报告 “match” 条件（MNLI-m）。我们遵循先前工作（Devlin et al. 2019；Liu et al. 2019；Yang et al. 2019）的微调流程，并报告经 GLUE 提交得到的留出测试集表现。测试集提交中，对 WNLI 与 QNLI 按 Liu et al. 2019 与 Yang et al. 2019 做任务特定修改。

SQuAD

SQuAD is an extractive question answering dataset built from Wikipedia. The answers are segments from the context paragraphs and the task is to predict answer spans. We evaluate our models on two versions of SQuAD: v1.1 and v2.0. SQuAD v1.1 has 100,000 human-annotated question/answer pairs. SQuAD v2.0 additionally introduced 50,000 unanswerable questions. For SQuAD v1.1, we use the same training procedure as BERT, whereas for SQuAD v2.0, models are jointly trained with a span extraction loss and an additional classifier for predicting answerability (Yang et al. 2019; Liu et al. 2019). We report both development set and test set performance.

SQuAD

SQuAD 是基于维基百科的抽取式问答数据集。答案是上下文段落中的片段，任务是预测答案跨度。我们在两个版本上评测：v1.1 与 v2.0。SQuAD v1.1 有 100,000 个人工标注问答对；v2.0 额外引入 50,000 个不可回答问题。对 v1.1 使用与 BERT 相同的训练流程；对 v2.0，模型联合训练跨度抽取损失与预测可回答性的额外分类器（Yang et al. 2019；Liu et al. 2019）。我们同时报告开发集与测试集表现。

RACE

RACE is a large-scale dataset for multi-choice reading comprehension, collected from English examinations in China with nearly 100,000 questions. Each instance in RACE has 4 candidate answers. Following prior work (Yang et al. 2019; Liu et al. 2019), we use the concatenation of the passage, question, and each candidate answer as the input to models. Then, we use the representations from the “[CLS]” token for predicting the probability of each answer. The dataset consists of two domains: middle school and high school. We train our models on both domains and report accuracies on both the development set and test set.

RACE

RACE 是大规模多选阅读理解数据集，采集自中国英语考试，近 100,000 道题。每个实例有 4 个候选答案。遵循先前工作（Yang et al. 2019；Liu et al. 2019），我们将段落、问题与每个候选答案拼接作为模型输入，再用 “[CLS]” 词元表示预测各答案概率。数据集含初中与高中两个域。我们在两域上训练，并报告开发集与测试集准确率。

A.4 Hyperparameters

A.4 超参数

Hyperparameters for downstream tasks are shown in Table 14. We adapt these hyperparameters from Liu et al. 2019, Devlin et al. 2019, and Yang et al. 2019.

下游任务超参数见表 14。我们改编自 Liu et al. 2019、Devlin et al. 2019 与 Yang et al. 2019。

	LR	BSZ	ALBERT DR	Classifier DR	TS	WS	MSL
CoLA	1.00E-05	16	0	0.1	5336	320	512
STS	2.00E-05	16	0	0.1	3598	214	512
SST-2	1.00E-05	32	0	0.1	20935	1256	512
MNLI	3.00E-05	128	0	0.1	10000	1000	512
QNLI	1.00E-05	32	0	0.1	33112	1986	512
QQP	5.00E-05	128	0.1	0.1	14000	1000	512
RTE	3.00E-05	32	0.1	0.1	800	200	512
MRPC	2.00E-05	32	0	0.1	800	200	512
WNLI	2.00E-05	16	0.1	0.1	2000	250	512
SQuAD v1.1	5.00E-05	48	0	0.1	3649	365	384
SQuAD v2.0	3.00E-05	48	0	0.1	8144	814	512
RACE	2.00E-05	32	0.1	0.1	12000	1000	512

Table 14: Hyperparameters for ALBERT in downstream tasks. LR: Learning Rate. BSZ: Batch Size. DR: Dropout Rate. TS: Training Steps. WS: Warmup Steps. MSL: Maximum Sequence Length.

	LR	BSZ	ALBERT DR	Classifier DR	TS	WS	MSL
CoLA	1.00E-05	16	0	0.1	5336	320	512
STS	2.00E-05	16	0	0.1	3598	214	512
SST-2	1.00E-05	32	0	0.1	20935	1256	512
MNLI	3.00E-05	128	0	0.1	10000	1000	512
QNLI	1.00E-05	32	0	0.1	33112	1986	512
QQP	5.00E-05	128	0.1	0.1	14000	1000	512
RTE	3.00E-05	32	0.1	0.1	800	200	512
MRPC	2.00E-05	32	0	0.1	800	200	512
WNLI	2.00E-05	16	0.1	0.1	2000	250	512
SQuAD v1.1	5.00E-05	48	0	0.1	3649	365	384
SQuAD v2.0	3.00E-05	48	0	0.1	8144	814	512
RACE	2.00E-05	32	0.1	0.1	12000	1000	512

表 14：ALBERT 下游任务超参数。LR：学习率。BSZ：批大小。DR：Dropout 率。TS：训练步数。WS：预热步数。MSL：最大序列长度。