DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

DistilBERT：BERT 的蒸馏版——更小、更快、更便宜、更轻

Victor SANH    Lysandre DEBUT    Julien CHAUMOND    Thomas WOLF Affiliation:  Hugging Face Email:  {victor,lysandre,julien,thomas}@huggingface.co

Abstract

摘要

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

随着大规模预训练模型上的迁移学习在自然语言处理（NLP）中日益普及，在边缘设备或受计算训练／推理预算约束的条件下运行这些大模型仍然困难。本文提出一种预训练更小通用语言表示模型 DistilBERT 的方法，随后可在多种任务上微调并取得良好表现，接近更大的对应模型。此前工作多把蒸馏用于构建任务专用模型；我们则在预训练阶段利用知识蒸馏，证明可以把 BERT 模型体量缩小 40%，同时保留其 97% 的语言理解能力，并将推理速度提高 60%。为利用大模型在预训练中学到的归纳偏置，我们引入由语言建模、蒸馏与余弦距离损失组成的三重损失。更小、更快、更轻的模型预训练成本更低；我们在概念验证实验与对照性端侧研究中展示了其端侧计算能力。

1 Introduction

1 引言

Figure 1: Parameter counts of several recently released pretrained language models.

图 1：若干近期发布的预训练语言模型的参数量。

The last two years have seen the rise of Transfer Learning approaches in Natural Language Processing (NLP) with large-scale pre-trained language models becoming a basic tool in many NLP tasks (Devlin et al. 2018; Radford et al. 2019; Liu et al. 2019). While these models lead to significant improvement, they often have several hundred million parameters and current research[^1] on pre-trained models indicates that training even larger models still leads to better performances on downstream tasks.

过去两年，大规模预训练语言模型成为许多 NLP 任务的基础工具，迁移学习在自然语言处理中迅速兴起（Devlin et al. 2018；Radford et al. 2019；Liu et al. 2019）。这些模型带来显著提升，但往往有数亿参数；当前关于预训练模型的研究[^1]还表明，训练更大的模型仍能进一步提升下游任务表现。

The trend toward bigger models raises several concerns. First is the environmental cost of exponentially scaling these models’ computational requirements as mentioned in Schwartz et al. 2019; Strubell et al. 2019. Second, while operating these models on-device in real-time has the potential to enable novel and interesting language processing applications, the growing computational and memory requirements of these models may hamper wide adoption.

模型越做越大带来若干担忧。其一是计算需求指数级扩张带来的环境成本，Schwartz et al. 2019；Strubell et al. 2019 已有讨论。其二，尽管在设备上实时运行这些模型有望催生新颖的语言处理应用，不断增长的计算与内存需求可能阻碍广泛落地。

In this paper, we show that it is possible to reach similar performances on many downstream-tasks using much smaller language models pre-trained with knowledge distillation, resulting in models that are lighter and faster at inference time, while also requiring a smaller computational training budget. Our general-purpose pre-trained models can be fine-tuned with good performances on several downstream tasks, keeping the flexibility of larger models. We also show that our compressed models are small enough to run on the edge, e.g. on mobile devices.

本文表明：用知识蒸馏预训练出的更小语言模型，可在许多下游任务上达到相近表现；推理时更轻更快，训练计算预算也更小。这类通用预训练模型可在多个下游任务上微调并保持良好表现，同时保留大模型的灵活性。我们还表明，压缩后的模型足够小，可在边缘侧（例如移动设备）运行。

Using a triple loss, we show that a 40% smaller Transformer (Vaswani et al. 2017) pre-trained through distillation via the supervision of a bigger Transformer language model can achieve similar performance on a variety of downstream tasks, while being 60% faster at inference time. Further ablation studies indicate that all the components of the triple loss are important for best performances.

借助三重损失，我们证明：在更大 Transformer 语言模型监督下经蒸馏预训练、体积缩小 40% 的 Transformer（Vaswani et al. 2017），可在多种下游任务上达到相近表现，推理速度提高 60%。进一步消融表明，三重损失的各分量对最佳表现都很重要。

We have made the trained weights available along with the training code in the Transformers[^2] library from HuggingFace (Wolf et al. 2019).

我们已将训练权重与训练代码发布在 HuggingFace 的 Transformers[^2] 库中（Wolf et al. 2019）。

2 Knowledge distillation

2 知识蒸馏

Knowledge distillation (Bucila et al. 2006; Hinton et al. 2015) is a compression technique in which a compact model - the student - is trained to reproduce the behaviour of a larger model - the teacher - or an ensemble of models.

知识蒸馏（Bucila et al. 2006；Hinton et al. 2015）是一种压缩技术：训练紧凑的学生模型，去复现更大教师模型（或模型集成）的行为。

In supervised learning, a classification model is generally trained to predict an instance class by maximizing the estimated probability of gold labels. A standard training objective thus involves minimizing the cross-entropy between the model’s predicted distribution and the one-hot empirical distribution of training labels. A model performing well on the training set will predict an output distribution with high probability on the correct class and with near-zero probabilities on other classes. But some of these "near-zero" probabilities are larger than others and reflect, in part, the generalization capabilities of the model and how well it will perform on the test set[^3].

在监督学习中，分类模型通常通过最大化金标签的估计概率来预测实例类别。标准训练目标因此是最小化模型预测分布与训练标签 one-hot 经验分布之间的交叉熵。在训练集上表现良好的模型，会在正确类上给出高概率，在其他类上给出接近零的概率。但这些「接近零」的概率彼此并不相同，它们部分反映了模型的泛化能力以及在测试集上的表现[^3]。

Training loss The student is trained with a distillation loss over the soft target probabilities of the teacher: $L_{ce}=\sum_{i}t_{i}*\log(s_{i})$ where $t_{i}$ (resp. $s_{i}$) is a probability estimated by the teacher (resp. the student). This objective results in a rich training signal by leveraging the full teacher distribution. Following Hinton et al. 2015 we used a softmax-temperature: $p_{i}=\frac{\exp(z_{i}/T)}{\sum_{j}\exp(z_{j}/T)}$ where $T$ controls the smoothness of the output distribution and $z_{i}$ is the model score for the class $i$. The same temperature $T$ is applied to the student and the teacher at training time, while at inference, $T$ is set to 1 to recover a standard softmax.

训练损失 学生在教师的软目标概率上用蒸馏损失训练：$L_{ce}=\sum_{i}t_{i}*\log(s_{i})$，其中 $t_{i}$（相应地 $s_{i}$）是教师（学生）估计的概率。该目标利用教师的完整分布，提供更丰富的训练信号。遵循 Hinton et al. 2015，我们使用 softmax-temperature：$p_{i}=\frac{\exp(z_{i}/T)}{\sum_{j}\exp(z_{j}/T)}$，其中 $T$ 控制输出分布的平滑程度，$z_{i}$ 是类别 $i$ 的模型分数。训练时对学生与教师使用相同温度 $T$；推理时将 $T$ 设为 1，以恢复标准 softmax。

The final training objective is a linear combination of the distillation loss $L_{ce}$ with the supervised training loss, in our case the masked language modeling loss $L_{mlm}$ (Devlin et al. 2018). We found it beneficial to add a cosine embedding loss ($L_{cos}$) which will tend to align the directions of the student and teacher hidden states vectors.

最终训练目标是蒸馏损失 $L_{ce}$ 与监督训练损失的线性组合；在我们这里，监督损失是 掩码语言建模 损失 $L_{mlm}$（Devlin et al. 2018）。我们发现加入 余弦嵌入 损失（$L_{cos}$）有益，它倾向于对齐学生与教师隐状态向量的方向。

3 DistilBERT: a distilled version of BERT

3 DistilBERT：BERT 的蒸馏版

Student architecture In the present work, the student - DistilBERT - has the same general architecture as BERT. The token-type embeddings and the pooler are removed while the number of layers is reduced by a factor of 2. Most of the operations used in the Transformer architecture (linear layer and layer normalisation) are highly optimized in modern linear algebra frameworks and our investigations showed that variations on the last dimension of the tensor (hidden size dimension) have a smaller impact on computation efficiency (for a fixed parameters budget) than variations on other factors like the number of layers. Thus we focus on reducing the number of layers.

学生架构 本文中，学生 DistilBERT 与 BERT 总体架构相同。去掉 token-type embeddings 与 pooler，层数减半。Transformer 架构中的多数运算（线性层 与 层归一化）在现代线性代数框架里高度优化；我们的调查表明，在固定参数预算下，张量最后一维（隐层维度）的变化对计算效率的影响，小于层数等因素的变化。因此我们重点减少层数。

Student initialization In addition to the previously described optimization and architectural choices, an important element in our training procedure is to find the right initialization for the sub-network to converge. Taking advantage of the common dimensionality between teacher and student networks, we initialize the student from the teacher by taking one layer out of two.

学生初始化 除前述优化与架构选择外，训练流程中的关键是为学生子网络找到合适的初始化以便收敛。利用师生网络维度一致，我们从教师每隔一层取一层来初始化学生。

Distillation We applied best practices for training BERT model recently proposed in Liu et al. 2019. As such, DistilBERT is distilled on very large batches leveraging gradient accumulation (up to 4K examples per batch) using dynamic masking and without the next sentence prediction objective.

蒸馏 我们采用 Liu et al. 2019 近期提出的 BERT 训练最佳实践。DistilBERT 在大批量上蒸馏，借助梯度累积（每批最多 4K 样本），使用动态掩码，且不使用下一句预测目标。

Data and compute power We train DistilBERT on the same corpus as the original BERT model: a concatenation of English Wikipedia and Toronto Book Corpus (Zhu et al. 2015). DistilBERT was trained on 8 16GB V100 GPUs for approximately 90 hours. For the sake of comparison, the RoBERTa model (Liu et al. 2019) required 1 day of training on 1024 32GB V100.

数据与算力 DistilBERT 在与原版 BERT 相同的语料上训练：英文维基百科与 Toronto Book Corpus（Zhu et al. 2015）的拼接。DistilBERT 在 8 块 16GB V100 GPU 上训练约 90 小时。作为对照，RoBERTa（Liu et al. 2019）需要在 1024 块 32GB V100 上训练 1 天。

4 Experiments

4 实验

General Language Understanding We assess the language understanding and generalization capabilities of DistilBERT on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al. 2018), a collection of 9 datasets for evaluating natural language understanding systems. We report scores on the development sets for each task by fine-tuning DistilBERT without the use of ensembling or multi-tasking scheme for fine-tuning (which are mostly orthogonal to the present work). We compare the results to the baseline provided by the authors of GLUE: an ELMo (Peters et al. 2018) encoder followed by two BiLSTMs.[^4]

通用语言理解 我们在 General Language Understanding Evaluation（GLUE）基准（Wang et al. 2018）上评估 DistilBERT 的语言理解与泛化能力；该基准含 9 个数据集，用于评估自然语言理解系统。我们报告各任务开发集分数，微调 DistilBERT 时不使用集成或多任务微调方案（这些大多与本文正交）。并与 GLUE 作者提供的基线比较：ELMo（Peters et al. 2018）编码器后接两个 BiLSTM。[^4]

The results on each of the 9 tasks are showed on Table 1 along with the macro-score (average of individual scores). Among the 9 tasks, DistilBERT is always on par or improving over the ELMo baseline (up to 19 points of accuracy on STS-B). DistilBERT also compares surprisingly well to BERT, retaining 97% of the performance with 40% fewer parameters.

表 1 给出 9 个任务的结果以及宏平均分数（各任务分数的平均）。在 9 个任务上，DistilBERT 始终与 ELMo 基线持平或更好（在 STS-B 上最多高出 19 个准确率点）。DistilBERT 与 BERT 的对比也出人意料地接近：参数少 40%，仍保留 97% 的表现。

Model	Score	CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST-2	STS-B	WNLI
ELMo	68.7	44.1	68.6	76.6	71.1	86.2	53.4	91.5	70.4	56.3
BERT-base	79.5	56.3	86.7	88.6	91.8	89.6	69.3	92.7	89.0	53.5
DistilBERT	77.0	51.3	82.2	87.5	89.2	88.5	59.9	91.3	86.9	56.3

Table 1: DistilBERT retains 97% of BERT performance. Comparison on the dev sets of the GLUE benchmark. ELMo results as reported by the authors. BERT and DistilBERT results are the medians of 5 runs with different seeds.

Model	Score	CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST-2	STS-B	WNLI
ELMo	68.7	44.1	68.6	76.6	71.1	86.2	53.4	91.5	70.4	56.3
BERT-base	79.5	56.3	86.7	88.6	91.8	89.6	69.3	92.7	89.0	53.5
DistilBERT	77.0	51.3	82.2	87.5	89.2	88.5	59.9	91.3	86.9	56.3

表 1：DistilBERT 保留 BERT 97% 的表现。 在 GLUE 基准开发集上的比较。ELMo 结果取自作者报告；BERT 与 DistilBERT 为不同种子 5 次运行的中位数。

4.1 Downstream task benchmark

4.1 下游任务基准

Downstream tasks We further study the performances of DistilBERT on several downstream tasks under efficient inference constraints: a classification task (IMDb sentiment classification - Maas et al. 2011) and a question answering task (SQuAD v1.1 - Rajpurkar et al. 2016).

下游任务 我们进一步在高效推理约束下，在若干下游任务上研究 DistilBERT：分类任务（IMDb 情感分类，Maas et al. 2011）与问答任务（SQuAD v1.1，Rajpurkar et al. 2016）。

Model	IMDb	SQuAD
	(acc.)	(EM/F1)
BERT-base	93.46	81.2/88.5
DistilBERT	92.82	77.7/85.8
DistilBERT (D)	-	79.1/86.9

Model	# param.	Inf. time
	(Millions)	(seconds)
ELMo	180	895
BERT-base	110	668
DistilBERT	66	410

Table 2: DistilBERT yields to comparable performance on downstream tasks. Comparison on downstream tasks: IMDb (test accuracy) and SQuAD 1.1 (EM/F1 on dev set). D: with a second step of distillation during fine-tuning.

Model	IMDb	SQuAD
	(acc.)	(EM/F1)
BERT-base	93.46	81.2/88.5
DistilBERT	92.82	77.7/85.8
DistilBERT (D)	-	79.1/86.9

Model	# param.	Inf. time
	(Millions)	(seconds)
ELMo	180	895
BERT-base	110	668
DistilBERT	66	410

表 2：DistilBERT 在下游任务上表现相当。 下游任务比较：IMDb（测试准确率）与 SQuAD 1.1（开发集 EM/F1）。D：微调阶段再做一步蒸馏。

As shown in Table 3, DistilBERT is only 0.6% point behind BERT in test accuracy on the IMDb benchmark while being 40% smaller. On SQuAD, DistilBERT is within 3.9 points of the full BERT.

如表 3 所示，在 IMDb 基准测试准确率上 DistilBERT 仅落后 BERT 0.6 个百分点，同时小 40%。在 SQuAD 上，DistilBERT 与完整 BERT 相差 3.9 分以内。

We also studied whether we could add another step of distillation during the adaptation phase by fine-tuning DistilBERT on SQuAD using a BERT model previously fine-tuned on SQuAD as a teacher for an additional term in the loss (knowledge distillation). In this setting, there are thus two successive steps of distillation, one during the pre-training phase and one during the adaptation phase. In this case, we were able to reach interesting performances given the size of the model: 79.8 F1 and 70.4 EM, i.e. within 3 points of the full model.

我们还研究了能否在适配阶段再加一步蒸馏：用先前在 SQuAD 上微调好的 BERT 作为教师，在损失中增加一项（知识蒸馏）来微调 DistilBERT。此时存在两次相继蒸馏——一次在预训练，一次在适配。在这种设置下，以模型体量而言可达到有趣表现：79.8 F1 与 70.4 EM，即距完整模型约 3 分以内。

Size and inference speed

体积与推理速度

To further investigate the speed-up/size trade-off of DistilBERT, we compare (in Table 3) the number of parameters of each model along with the inference time needed to do a full pass on the STS-B development set on CPU (Intel Xeon E5-2690 v3 Haswell @2.9GHz) using a batch size of 1. DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.

为进一步考察 DistilBERT 的加速／体积权衡，我们在表 3 中比较各模型参数量，以及在 CPU（Intel Xeon E5-2690 v3 Haswell @2.9GHz）上以 batch size 1 对 STS-B 开发集完整跑一遍所需的推理时间。DistilBERT 比 BERT 少 40% 参数，比 BERT 快 60%。

On device computation We studied whether DistilBERT could be used for on-the-edge applications by building a mobile application for question answering. We compare the average inference time on a recent smartphone (iPhone 7 Plus) against our previously trained question answering model based on BERT-base. Excluding the tokenization step, DistilBERT is 71% faster than BERT, and the whole model weighs 207 MB (which could be further reduced with quantization). Our code is available[^5].

端侧计算 我们构建问答移动应用，研究 DistilBERT 能否用于边缘应用。在较新智能手机（iPhone 7 Plus）上，与先前基于 BERT-base 训练的问答模型比较平均推理时间。不计分词步骤，DistilBERT 比 BERT 快 71%，整模约 207 MB（量化还可再缩小）。代码已公开[^5]。

4.2 Ablation study

4.2 消融研究

In this section, we investigate the influence of various components of the triple loss and the student initialization on the performances of the distilled model. We report the macro-score on GLUE. Table 4 presents the deltas with the full triple loss: removing the Masked Language Modeling loss has little impact while the two distillation losses account for a large portion of the performance.

本节考察三重损失各分量与学生初始化对蒸馏模型表现的影响。报告 GLUE 宏平均分数。表 4 给出相对完整三重损失的差值：去掉 掩码语言建模 损失影响很小，而两项蒸馏损失贡献了大部分表现。

Ablation	Variation on GLUE macro-score
$\emptyset$ - $L_{cos}$ - $L_{mlm}$	-2.96
$L_{ce}$ - $\emptyset$ - $L_{mlm}$	-1.46
$L_{ce}$ - $L_{cos}$ - $\emptyset$	-0.31
Triple loss + random weights initialization	-3.69

Table 4: Ablation study. Variations are relative to the model trained with triple loss and teacher weights initialization.

Ablation	Variation on GLUE macro-score
$\emptyset$ - $L_{cos}$ - $L_{mlm}$	-2.96
$L_{ce}$ - $\emptyset$ - $L_{mlm}$	-1.46
$L_{ce}$ - $L_{cos}$ - $\emptyset$	-0.31
Triple loss + random weights initialization	-3.69

表 4：消融研究。 变分相对于用三重损失并以教师权重初始化训练的模型。

5 Related work

5 相关工作

Task-specific distillation Most of the prior works focus on building task-specific distillation setups. Tang et al. 2019 transfer fine-tune classification model BERT to an LSTM-based classifier. Chatterjee 2019 distill BERT model fine-tuned on SQuAD in a smaller Transformer model previously initialized from BERT. In the present work, we found it beneficial to use a general-purpose pre-training distillation rather than a task-specific distillation. Turc et al. 2019 use the original pretraining objective to train smaller student, then fine-tuned via distillation. As shown in the ablation study, we found it beneficial to leverage the teacher’s knowledge to pre-train with additional distillation signal.

任务专用蒸馏 先前工作多聚焦任务专用蒸馏设置。Tang et al. 2019 把微调后的 BERT 分类模型迁移到基于 LSTM 的分类器。Chatterjee 2019 把在 SQuAD 上微调的 BERT 蒸馏到先前从 BERT 初始化的更小 Transformer。本文发现，通用预训练蒸馏比任务专用蒸馏更有益。Turc et al. 2019 用原始预训练目标训练更小的学生，再经蒸馏微调。如消融所示，利用教师知识、在预训练中加入额外蒸馏信号是有益的。

Multi-distillation Yang et al. 2019 combine the knowledge of an ensemble of teachers using multi-task learning to regularize the distillation. The authors apply Multi-Task Knowledge Distillation to learn a compact question answering model from a set of large question answering models. An application of multi-distillation is multi-linguality: Tsai et al. 2019 adopts a similar approach to us by pre-training a multilingual model from scratch solely through distillation. However, as shown in the ablation study, leveraging the teacher’s knowledge with initialization and additional losses leads to substantial gains.

多重蒸馏 Yang et al. 2019 用多任务学习结合教师集成的知识来正则化蒸馏。作者将 多任务知识蒸馏 用于从一组大问答模型学习紧凑问答模型。多重蒸馏的一个应用是多语：Tsai et al. 2019 采取与我们类似的思路，仅通过蒸馏从零预训练多语模型。但如消融所示，借助初始化与额外损失利用教师知识可带来显著收益。

Other compression techniques have been studied to compress large models. Recent developments in weights pruning reveal that it is possible to remove some heads in the self-attention at test time without significantly degrading the performance Michel et al. 2019. Some layers can be reduced to one head. A separate line of study leverages quantization to derive smaller models (Gupta et al. 2015). Pruning and quantization are orthogonal to the present work.

其他压缩技术 也被用于压缩大模型。权重剪枝的近期进展表明，测试时可去掉自注意力中的部分头而不显著降性能（Michel et al. 2019）。某些层可减至单头。另一路线用量化得到更小模型（Gupta et al. 2015）。剪枝与量化与本文正交。

6 Conclusion and future work

6 结论与未来工作

We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities. We showed that a general-purpose language model can be successfully trained with distillation and analyzed the various components with an ablation study. We further demonstrated that DistilBERT is a compelling option for edge applications.

我们提出 DistilBERT——BERT 的通用预训练版本：小 40%、快 60%，并保留 97% 的语言理解能力。我们表明可用蒸馏成功训练通用语言模型，并用消融分析了各分量。我们进一步表明 DistilBERT 是边缘应用的有力选项。

References

参考文献

Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , 2018.

Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.

Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar S. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv , abs/1907.11692, 2019.

Schwartz et al. (2019) Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. ArXiv , abs/1907.10597, 2019.

Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACL , 2019.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , 2017.

Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Transformers: State-of-the-art natural language processing, 2019.

Bucila et al. (2006) Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD , 2006.

Hinton et al. (2015) Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv , abs/1503.02531, 2015.

Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV) , pages 19–27, 2015.

Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR , 2018.

Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL , 2018.

Wang et al. (2019) Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Najoung Kim, Phu Mon Htut, Thibault F’evry, Berlin Chen, Nikita Nangia, Haokun Liu, Anhad Mohananey, Shikha Bordia, Nicolas Patry, Ellie Pavlick, and Samuel R. Bowman. jiant 1.1: A software toolkit for research on general-purpose text understanding models. http://jiant.info/ , 2019.

Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL , 2011.

Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP , 2016.

Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from bert into simple neural networks. ArXiv , abs/1903.12136, 2019.

Chatterjee (2019) Debajyoti Chatterjee. Making neural machine reading comprehension faster. ArXiv , abs/1904.00796, 2019.

Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. ArXiv , abs/1908.08962, 2019.

Yang et al. (2019) Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with multi-task knowledge distillation for web-scale question answering system. ArXiv , abs/1904.09636, 2019.

Tsai et al. (2019) Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. Small and practical bert models for sequence labeling. In EMNLP-IJCNLP , 2019.

Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In NeurIPS , 2019.

Gupta et al. (2015) Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML , 2015.