RoBERTa: A Robustly Optimized BERT Pretraining Approach

RoBERTa：一种稳健优化的 BERT 预训练方法（RoBERTa: A Robustly Optimized BERT Pretraining Approach）

Yinhan Liu Thanks:  ˜˜Equal contribution.    Myle Ott    Naman Goyal    Jingfei Du    Mandar Joshi    Danqi Chen   Omer Levy   Mike Lewis   Luke Zettlemoyer  Veselin Stoyanov    [8pt] Paul G. Allen School of Computer Science & EngineeringUniversity of Washington, Seattle, WA {mandar90,lsz}@cs.washington.edu    [4pt] Facebook AI {yinhanliu,myleott,naman,jingfeidu,    danqi,omerlevy,mikelewis,lsz,ves}@fb.com

Yinhan Liu* Myle Ott* Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer Veselin Stoyanov
Paul G. Allen School of Computer Science & Engineering，华盛顿大学，西雅图；Facebook AI
{mandar90,lsz}@cs.washington.edu；{yinhanliu,myleott,naman,jingfeidu,danqi,omerlevy,mikelewis,lsz,ves}@fb.com

• 作者贡献相同

Abstract

摘要

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining Devlin et al. 2019 that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.[^1]

语言模型预训练带来了显著的性能提升，但不同方法之间难以仔细比较。训练计算昂贵，常在规模各异的私有数据上进行，而且我们将表明，超参数选择对最终结果影响很大。我们对 BERT 预训练（Devlin 等，2019）做了一次复现研究，仔细度量许多关键超参数与训练数据规模的影响。我们发现 BERT 明显训练不足，在充分训练后可以追平或超过其后发表的每个模型。我们最好的模型在 GLUE、RACE 与 SQuAD 上取得了当时最优结果。这些结果凸显了先前被忽视的设计选择的重要性，并对近期报道的改进来源提出了疑问。我们发布了模型与代码。

1 Introduction

1 引言

Self-training methods such as ELMo Peters et al. 2018, GPT Radford et al. 2018, BERT Devlin et al. 2019, XLM Lample and Conneau 2019, and XLNet Yang et al. 2019 have brought significant performance gains, but it can be challenging to determine which aspects of the methods contribute the most. Training is computationally expensive, limiting the amount of tuning that can be done, and is often done with private training data of varying sizes, limiting our ability to measure the effects of the modeling advances.

ELMo Peters等人2018年,GPT Radford等人2018年,BERT Devlin等人2019年,XLM Lample和Conneau2019年,XLNet Yang等人2019年等自训方法都带来了显著的性能收益,但确定方法的哪些方面贡献最大可能具有挑战性. 培训在计算上费用高昂,限制了可以完成的调试量,并且常常用大小不同的私人培训数据完成,限制了我们衡量建模进步效果的能力.

We present a replication study of BERT pretraining Devlin et al. 2019, which includes a careful evaluation of the effects of hyperparmeter tuning and training set size. We find that BERT was significantly undertrained and propose an improved recipe for training BERT models, which we call RoBERTa, that can match or exceed the performance of all of the post-BERT methods. Our modifications are simple, they include: (1) training the model longer, with bigger batches, over more data; (2) removing the next sentence prediction objective; (3) training on longer sequences; and (4) dynamically changing the masking pattern applied to the training data. We also collect a large new dataset (CC-News) of comparable size to other privately used datasets, to better control for training set size effects.

我们提出了BERT预训练Devlin等人2019年的复制研究,其中包括对超parmeter调制和训练设置尺寸的效果进行仔细评价. 我们发现BERT训练严重不足,并提出了一种更好的方法来训练BERT模型,我们称之为RoBERTa,它可以匹配或超过所有BERT后方法的性能. 我们的修改很简单,它们包括:(1) 训练模型更长,分批量较大,超过更多的数据;(2) 删除下一个句子预测目标;(3) 训练较长的序列;(4) 动态改变训练数据所用的遮掩模式。 我们还收集了与其他私人使用的数据集具有可比性的大型新数据集(CC-News),以更好地控制训练设置大小效应。

When controlling for training data, our improved training procedure improves upon the published BERT results on both GLUE and SQuAD. When trained for longer over additional data, our model achieves a score of 88.5 on the public GLUE leaderboard, matching the 88.4 reported by Yang et al. 2019. Our model establishes a new state-of-the-art on 4/9 of the GLUE tasks: MNLI, QNLI, RTE and STS-B. We also match state-of-the-art results on SQuAD and RACE. Overall, we re-establish that BERT’s masked language model training objective is competitive with other recently proposed training objectives such as perturbed autoregressive language modeling Yang et al. 2019.[^2]

在控制培训数据时,我们改进了的训练流程,改进了已公布的关于GLUE和SQuAD的BERT结果。 在接受超过额外数据的培训时,我们的模型在公开的GLUE领导板上取得了88.5的分数,匹配了杨等人2019年报告的88.4分. 我们的模式为GLUE任务的4/9建立了新的先进水平:MNLI、QNLI、RTE和STS-B。 我们还对准了SQuAD和RACE的最新结果。 总体而言,我们重新确定BERT的蒙面语言模型训练目标与最近提出的其他训练目标具有竞争力,如令人不安的自旋语言模型杨等人2019。

In summary, the contributions of this paper are: (1) We present a set of important BERT design choices and training strategies and introduce alternatives that lead to better downstream task performance; (2) We use a novel dataset, CC-News, and confirm that using more data for pretraining further improves performance on downstream tasks; (3) Our training improvements show that masked language model pretraining, under the right design choices, is competitive with all other recently published methods. We release our model, pretraining and fine-tuning code implemented in PyTorch Paszke et al. 2017.

简言之,本文件的贡献是:(1)我们提出一套重要的BERT设计选择和培训战略,并采用其他方法,使下游任务业绩得到改善;(2)我们使用一套新的数据集CC-News,并证实使用更多的数据进行预训练会进一步提高下游任务的业绩;(3)我们的培训改进表明,蒙面语言模型预训练,在正确的设计选择下,与所有其他最近公布的方法具有竞争力. 发布在PyTorch Paszke等人2017年实施的模型,预训练和微调代码.

2 Background

2 背景

In this section, we give a brief overview of the BERT Devlin et al. 2019 pretraining approach and some of the training choices that we will examine experimentally in the following section.

在本节中,我们简要概述了BERT Devlin等人2019年预训练方法以及我们将在下节进行实验性审查的一些培训选择.

2.1 Setup

2.1 设定

BERT takes as input a concatenation of two segments (sequences of tokens), $x_{1},\ldots,x_{N}$ and $y_{1},\ldots,y_{M}$. Segments usually consist of more than one natural sentence. The two segments are presented as a single input sequence to BERT with special tokens delimiting them: $[\mathit{CLS}],x_{1},\ldots,x_{N},[\mathit{SEP}],y_{1},\ldots,y_{M},[\mathit{EOS}]$. $M$ and $N$ are constrained such that $M+N<T$, where $T$ is a parameter that controls the maximum sequence length during training.

BERT将两个段(令牌序列)、$x_{1},\ldots,x_{N}$和$y_{1},\ldots,y_{M}$的组合作为输入。 片段通常由一个以上的自然句子组成. 这两个部分作为BERT的单一输入序列呈现,并配有特别的符号来划分它们:$[\mathit{CLS}],x_{1},\ldots,x_{N},[\mathit{SEP}],y_{1},\ldots,y_{M},[\mathit{EOS}]$. $M$和$N$受到制约,以至于$M+N<T$,其中$T$是训练期间控制最大序列长度的参数.

The model is first pretrained on a large unlabeled text corpus and subsequently finetuned using end-task labeled data.

该模型首先在大型无标签文本库上进行预训练,随后使用端任务标签数据进行微调.

2.2 Architecture

2.2 架构

BERT uses the now ubiquitous transformer architecture Vaswani et al. 2017, which we will not review in detail. We use a transformer architecture with $L$ layers. Each block uses $A$ self-attention heads and hidden dimension $H$.

BERT使用现在无所不在的变压器架构Vaswani et al. 2017,我们将不详细审查. 我们使用一个带有$L$层的变压器架构. 每个区块使用$A$自注头和隐藏尺寸$H$.

2.3 Training Objectives

2.3 训练目标

During pretraining, BERT uses two objectives: masked language modeling and next sentence prediction.

在预训练期间,BERT使用两个目标:蒙面语言模型和下句预测.

Masked Language Model (MLM)

A random sample of the tokens in the input sequence is selected and replaced with the special token $[\mathit{MASK}]$. The MLM objective is a cross-entropy loss on predicting the masked tokens. BERT uniformly selects 15% of the input tokens for possible replacement. Of the selected tokens, 80% are replaced with $[\mathit{MASK}]$, 10% are left unchanged, and 10% are replaced by a randomly selected vocabulary token.

• 假面语言模型(MLM)

在输入序列中随机抽取一个符号样本,并替换为特殊符号$[\mathit{MASK}]$ . MLM目标是在预测蒙面符时的交叉内向损失. BERT统一选择15%的输入符用于可能的替换. 所选的令牌中,80%被替换为$[\mathit{MASK}]$,10%未变,10%被随机选择的词汇令牌所取代.

In the original implementation, random masking and replacement is performed once in the beginning and saved for the duration of training, although in practice, data is duplicated so the mask is not always the same for every training sentence (see Section 4.1).

在最初的实施工作中,随机蒙面和替换在开始时一次进行,并在培训期间保存,尽管在实践中数据重复,因此每个培训句的蒙面并不总是相同的(见第4.1节)。

Next Sentence Prediction (NSP)

NSP is a binary classification loss for predicting whether two segments follow each other in the original text. Positive examples are created by taking consecutive sentences from the text corpus. Negative examples are created by pairing segments from different documents. Positive and negative examples are sampled with equal probability.

下一个判决预测(NSP)

NSP是一种二进制分类损失,用于预测原始文本中两个段是否互相跟踪. 正面的例子通过从文字体中连续句子来创造. 负例是通过从不同文档中对齐部分来创建的. 正面和负面实例的抽样概率相等。

The NSP objective was designed to improve performance on downstream tasks, such as Natural Language Inference Bowman et al. 2015, which require reasoning about the relationships between pairs of sentences.

NSP目标旨在改进下游任务的性能,如自然语言推理Bowman等人2015年,这需要对对句之间的关系进行推理.

2.4 Optimization

2.4 优化

BERT is optimized with Adam Kingma and Ba 2015 using the following parameters: $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=\text{1e-6}$ and $L_{2}$ weight decay of $0.01$. The learning rate is warmed up over the first 10,000 steps to a peak value of 1e-4, and then linearly decayed. BERT trains with a dropout of 0.1 on all layers and attention weights, and a GELU activation function Hendrycks and Gimpel 2016. Models are pretrained for $S=\text{1,000,000}$ updates, with minibatches containing $B=\text{256}$ sequences of maximum length $T=\text{512}$ tokens.

BERT与Adam Kingma和Ba 2015一起进行了优化,使用了以下参数: $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=\text{1e-6}$和 $L_{2}$重量衰减 $0.01$. 学习速度在最初的1万步到1e-4的峰值时会变暖,然后线性衰变. BERT列车各层和注意力重量的停放量为0.1,GELU激活功能Hendrycks和Gimpel 2016. 型号为$S=\text{1,000,000}$更新的预训练,小管包含最大长度的$B=\text{256}$序列 $T=\text{512}$令牌.

2.5 Data

2.5 数据

BERT is trained on a combination of BookCorpus Zhu et al. 2015 plus English Wikipedia, which totals 16GB of uncompressed text.[^3]

BERT接受过BookCorpus Zhu等人2015年加英语维基百科的组合训练,共16GB未压缩文本. [^3]

3 Experimental Setup

3 实验设置

In this section, we describe the experimental setup for our replication study of BERT.

本节介绍BERT复制研究的实验设置.

3.1 Implementation

3.1 实现

We reimplement BERT in fairseq Ott et al. 2019. We primarily follow the original BERT optimization hyperparameters, given in Section 2, except for the peak learning rate and number of warmup steps, which are tuned separately for each setting. We additionally found training to be very sensitive to the Adam epsilon term, and in some cases we obtained better performance or improved stability after tuning it. Similarly, we found setting $\beta_{2}=0.98$ to improve stability when training with large batch sizes.

我们在Fairseq Ott等人的2019年重新执行BERT. 我们主要遵循最初BERT优化的超参数,在第2节中给出,但峰值学习率和热量步骤数除外,每个设置都分别调制. 我们还发现训练对亚当·埃普西隆(Adam epsilon)一词非常敏感,在某些情况下,我们在调试后获得了更好的性能或更好的稳定性. 同样,我们发现设置$\beta_{2}=0.98$,以提高大批量培训时的稳定性。

We pretrain with sequences of at most $T=512$ tokens. Unlike Devlin et al. 2019, we do not randomly inject short sequences, and we do not train with a reduced sequence length for the first 90% of updates. We train only with full-length sequences.

我们预演的序列最多为$T=512$令牌. 与Devlin等人2019年不同的是,我们不会随机注入短序列,也不会对前90%的更新进行缩短序列长度的训练. 我们只用全长序列来训练

We train with mixed precision floating point arithmetic on DGX-1 machines, each with 8 $\times$ 32GB Nvidia V100 GPUs interconnected by Infiniband Micikevicius et al. 2018.

我们用混合精度浮点算术在DGX-1机上进行训练,每台机上由8$\times$32GB Nvidia V100 GPU由Infiniband Micikevicius等人进行互联互通.

3.2 Data

3.2 数据

BERT-style pretraining crucially relies on large quantities of text. Baevski et al. 2019 demonstrate that increasing data size can result in improved end-task performance. Several efforts have trained on datasets larger and more diverse than the original BERT Radford et al. 2019; Yang et al. 2019; Zellers et al. 2019. Unfortunately, not all of the additional datasets can be publicly released. For our study, we focus on gathering as much data as possible for experimentation, allowing us to match the overall quality and quantity of data as appropriate for each comparison.

BERT式预训练主要依靠大量文本。 Baevski 等人 2019 证明,数据规模的扩大可以导致终端任务性能的改善. 与最初的BERT Ra确定其拉德福德等人2019年相比,在数据集方面进行了多项培训;Yang等人2019年;Zellers等人2019年。 不幸的是,并非所有的额外数据集都能够公开发布。 就我们的研究而言,我们的重点是尽可能多地收集用于试验的数据,使我们能够在每次比较时酌情匹配数据的整体质量和数量。

We consider five English-language corpora of varying sizes and domains, totaling over 160GB of uncompressed text. We use the following text corpora:

我们认为,5个英语公司的规模和领域各不相同,总共超过160GB的未压缩文本。 我们使用以下文字:

• • BookCorpus Zhu et al. 2015 plus English Wikipedia. This is the original data used to train BERT. (16GB).
• • CC-News, which we collected from the English portion of the CommonCrawl News dataset Nagel 2016. The data contains 63 million English news articles crawled between September 2016 and February 2019. (76GB after filtering).[^4]
• • OpenWebText Gokaslan and Cohen 2019, an open-source recreation of the WebText corpus described in Radford et al. 2019. The text is web content extracted from URLs shared on Reddit with at least three upvotes. (38GB).[^5]
• • Stories, a dataset introduced in Trinh and Le 2018 containing a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas. (31GB).

-),BookCorpus Zhu等2015年+英语维基百科. 这是用于训练BERT. (16GB)的原始数据.
– CC-News,我们从CommonCrawl News的英语部分收集到了这集的Nagel 2016. 数据包含6300万篇在2016年9月至2019年2月期间爬行的英国新闻文章. (过滤后的76GB). [^4].

• OpenWebText Gokaslan和Cohen 2019,是拉德福德等人2019年描述的WebText的开源娱乐. 文本是从 Reddit 上共享的 URL 中提取的网络内容, 并且至少有3 个 Upvotes (38GB). [^5].
• Stories),Trinh和Le 2018引入的数据集,包含一个被过滤的CommonCrawl数据子集,以配合Winograd schemas的故事风格. (31GB).

3.3 Evaluation

3.3 评测

Following previous work, we evaluate our pretrained models on downstream tasks using the following three benchmarks.

做好前期工作后,我们利用以下三个基准,对下游任务先行先试的模式进行评估.

GLUE

The General Language Understanding Evaluation (GLUE) benchmark Wang et al. 2019b is a collection of 9 datasets for evaluating natural language understanding systems.[^6] Tasks are framed as either single-sentence classification or sentence-pair classification tasks. The GLUE organizers provide training and development data splits as well as a submission server and leaderboard that allows participants to evaluate and compare their systems on private held-out test data.

格卢

通用语言理解评价(GLUE)基准Wang等人 2019b是9个数据集的集合,用于评价自然语言理解系统. [^6] 各项任务要么是单一判决分类,要么是判刑-补偿分类。 你说GLUE组织者提供培训和开发数据分解,以及提交服务器和领导板,使参与者能够在私人持有的测试数据上评价和比较他们的系统.

For the replication study in Section 4, we report results on the development sets after finetuning the pretrained models on the corresponding single-task training data (i.e., without multi-task training or ensembling). Our finetuning procedure follows the original BERT paper Devlin et al. 2019.

关于第4节的复制研究,我们报告在调整相应单一任务培训数据(即不进行多任务培训或综艺培训)的预先培训模型后开发各套的成果。 我们的微调程序遵循了BERT最初的论文Devlin等人2019年.

In Section 5 we additionally report test set results obtained from the public leaderboard. These results depend on a several task-specific modifications, which we describe in Section 5.1.

在第五节中,我们另外报告测试结果来自公共领导板。 是吗? 这些结果取决于我们在第5.1节中描述的几项针对具体任务的修改。

SQuAD

The Stanford Question Answering Dataset (SQuAD) provides a paragraph of context and a question. The task is to answer the question by extracting the relevant span from the context. We evaluate on two versions of SQuAD: V1.1 and V2.0 Rajpurkar et al. 2016; Rajpurkar et al. 2018. In V1.1 the context always contains an answer, whereas in V2.0 some questions are not answered in the provided context, making the task more challenging.

QQ SQAD 主题

斯坦福问答数据集(SQuAD)提供了一个上下文段落和一个问题. 任务是通过从上下文中提取相关跨度来回答这个问题. 我们评价SQuAD的两个版本:V1.1和V2.0 Rajpurkar等2016年;Rajpurkar等2018年. 在V1.1中,上下文总是包含一个答案,而在V2.0中,有些问题没有在所提供的背景下得到回答,使任务更具挑战性。

For SQuAD V1.1 we adopt the same span prediction method as BERT Devlin et al. 2019. For SQuAD V2.0, we add an additional binary classifier to predict whether the question is answerable, which we train jointly by summing the classification and span loss terms. During evaluation, we only predict span indices on pairs that are classified as answerable.

就SQuAD V1.1而言,我们采用了与BERT Devlin等人2019年相同的跨度预测方法。 对于SQAD V2.0,我们增加一个额外的二进制分类器来预测这个问题是否可回答,我们通过对分类和跨损失条款进行总结来联合训练. 在评价期间,我们只预测被归类为可回答的对等指数。

RACE

The ReAding Comprehension from Examinations (RACE) Lai et al. 2017 task is a large-scale reading comprehension dataset with more than 28,000 passages and nearly 100,000 questions. The dataset is collected from English examinations in China, which are designed for middle and high school students. In RACE, each passage is associated with multiple questions. For every question, the task is to select one correct answer from four options. RACE has significantly longer context than other popular reading comprehension datasets and the proportion of questions that requires reasoning is very large.

\ nRACE(RACE)

ReAding Consultion from Experiences (RACE) Lai et al. 2017的任务是一个大规模的阅读理解数据集,有超过28000个段落和近10万个问题. 数据集采集自中国英语考试,为中学生和高中生设计. 在RACE中,每个段落都与多个问题相关. 对于每一个问题,任务是从四个选项中选择一个正确的答案. RACE的上下文大大长于其他通俗阅读理解数据集,需要推理的问题比例非常大.

4 Training Procedure Analysis

4 训练流程分析

This section explores and quantifies which choices are important for successfully pretraining BERT models. We keep the model architecture fixed.[^7] Specifically, we begin by training BERT models with the same configuration as BERT${}_{\textsc{base}}$ ($L=12$, $H=768$, $A=12$, 110M params).

本节探讨并量化哪些选择对成功预训练BERT模型很重要. 我们把模型架构固定下来。 [^7] 具体而言,我们首先培训BERT模型,其配置与BERT ${}_{\textsc{base}}$($L=12$,$H=768$,$A=12$,110M Parms)相同.

4.1 Static vs. Dynamic Masking

4.1 静态掩码与动态掩码

As discussed in Section 2, BERT relies on randomly masking and predicting tokens. The original BERT implementation performed masking once during data preprocessing, resulting in a single static mask. To avoid using the same mask for each training instance in every epoch, training data was duplicated 10 times so that each sequence is masked in 10 different ways over the 40 epochs of training. Thus, each training sequence was seen with the same mask four times during training.

如第2节所述,BERT依靠随机遮掩和预测符号. 最初的BERT执行在数据预处理过程中进行了一次遮掩,导致一个单一的静态遮掩. 为避免在每个时代的每个shino培训场合都使用同样的面具,培训数据重复了10次,这样在40个世纪的培训过程中,每个序列都以10种不同的方式被掩盖。 因此,在训练期间,每次训练都用同样的面具四次看到。

We compare this strategy with dynamic masking where we generate the masking pattern every time we feed a sequence to the model. This becomes crucial when pretraining for more steps or with larger datasets.

我们把这个策略和“动态遮罩”进行比较, 每次给模型输入序列时, 我们都会产生遮罩模式。 在预先培训更多步骤或使用更大的数据集时,这一点至关重要。

Results

Masking	SQuAD 2.0	MNLI-m	SST-2
reference	76.3	84.3	92.8
Our reimplementation:
static	78.3	84.3	92.5
dynamic	78.7	84.0	92.9

Table 1: Comparison between static and dynamic masking for BERT${}_{\textsc{base}}$. We report F1 for SQuAD and accuracy for MNLI-m and SST-2. Reported results are medians over 5 random initializations (seeds). Reference results are from Yang et al. 2019.

结果

• 马斯金* * * * SQuAD 2.0 * * * ** MNLI-m** * * * ** SST-2** * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ** * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ** * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
  |,|也.
  参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 参考文献: 76.3 参考文献: 参考文献: 84.3 参考文献: 参考文献: 92.8 参考文献: 参考文献:
• 我们的重新实施:* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ** * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
  静态 静态 静态 静态 静态 静态
  | 动态 | 78.7 | 84.0 | 92.9 |

表1:BERT ${}_{\textsc{base}}$的静态遮罩与动态遮罩的比较. 我们报告 SQuAD 的 F1 和 MNLI- m 和 SST-2 的精确度。 报告的结果是5个随机初始化(种子)以上的中位数。 参考结果来自杨等人 2019.

Table 1 compares the published BERT${}_{\textsc{base}}$ results from Devlin et al. 2019 to our reimplementation with either static or dynamic masking. We find that our reimplementation with static masking performs similar to the original BERT model, and dynamic masking is comparable or slightly better than static masking.

表1将公布的Devlin等人2019年的BERT ${}_{\textsc{base}}$结果与我们以静态或动态遮罩方式重新实施的结果进行比较. 我们发现,我们用静态遮罩的重新实施,其性能与原先的BERT模型相似,动态遮罩与静态遮罩相似或略好.

Given these results and the additional efficiency benefits of dynamic masking, we use dynamic masking in the remainder of the experiments.

鉴于这些结果和动态遮罩的额外效率好处,我们在其余实验中使用动态遮罩.

4.2 Model Input Format and Next Sentence Prediction

4.2 模型输入格式与下一句预测

Model	SQuAD 1.1/2.0	MNLI-m	SST-2	RACE
Our reimplementation (with NSP loss):
segment-pair	90.4/78.7	84.0	92.9	64.2
sentence-pair	88.7/76.2	82.9	92.1	63.0
Our reimplementation (without NSP loss):
full-sentences	90.4/79.1	84.7	92.5	64.8
doc-sentences	90.6/79.7	84.7	92.7	65.6
BERT${}_{\textsc{base}}$	88.5/76.3	84.3	92.8	64.3
XLNet${}_{\textsc{base}}$ (K = 7)	–/81.3	85.8	92.7	66.1
XLNet${}_{\textsc{base}}$ (K = 6)	–/81.0	85.6	93.4	66.7

Table 2: Development set results for base models pretrained over BookCorpus and Wikipedia. All models are trained for 1M steps with a batch size of 256 sequences. We report F1 for SQuAD and accuracy for MNLI-m, SST-2 and RACE. Reported results are medians over five random initializations (seeds). Results for BERT${}_{\textsc{base}}$ and XLNet${}_{\textsc{base}}$ are from Yang et al. 2019.

** 模式**	** SQuAD 1.1/2.0**	** MNLI-m** (韩语)	** 《大会正式记录,第五十八届会议,补编第6号》(A/59/6),第二章。	页:1
我们的重新执行(国家战略计划损失):
扇区	90.4/78.7	84.0	92.9	64.2
句子页	88.7/76.2	82.9	92.1	63.0
我们的重新实施(没有国家战略计划损失):
完全判决	90.4/79.1	84.7	92.5	64.8
医生判刑	90.6/79.7	84.7	92.7	65.6
BERT 第二十五届	88.5/76.3	84.3	92.8	64.3
XLNet ${}_{\textsc{base}}$(K=7)	-81.3) (中文(简体) ).	85.8	92.7	66.1
XLNet ${}_{\textsc{base}}$(K=6)	-/81.0 (中文(简体) ).	85.6	93.4	66.7

表2:BookCorpus和维基百科对基础模型进行预先培训的开发设定结果. 所有型号都为1M级步骤训练,批量大小为256个序列. 我们报告F1的SQuAD和MNLI-m、SST-2和RACE的准确性。 报告的结果为5个随机初始化(种子)以上的中位数. BERT ${}_{\textsc{base}}$和XLNet ${}_{\textsc{base}}$的结果来自杨等人2019年.

In the original BERT pretraining procedure, the model observes two concatenated document segments, which are either sampled contiguously from the same document (with $p=0.5$) or from distinct documents. In addition to the masked language modeling objective, the model is trained to predict whether the observed document segments come from the same or distinct documents via an auxiliary Next Sentence Prediction (NSP) loss.

在最初的BERT预训练程序中,模型观察到两个缩合的文件段,或者从同一文件(与$p=0.5$)或不同文件的毗连抽样. 除了蒙蔽的语言建模目标外,该模型还经过培训,通过辅助的"下句预测"(NSP)丢失来预测所观测到的文档片段是来自同一文件还是来自不同的文件.

The NSP loss was hypothesized to be an important factor in training the original BERT model. Devlin et al. 2019 observe that removing NSP hurts performance, with significant performance degradation on QNLI, MNLI, and SQuAD 1.1. However, some recent work has questioned the necessity of the NSP loss Lample and Conneau 2019; Yang et al. 2019; Joshi et al. 2019.

假设NSP损失是培训BERT模型的一个重要因素。 Devlin等人2019年观察到,移除NSP会伤害性能,在QNLI,MNLI,SQuAD 1.1. 然而,最近的一些作品质疑NSP损失Lample和Conneau 2019的必要性;Yang等人 2019;Joshi等人 2019.

To better understand this discrepancy, we compare several alternative training formats:

为了更好地理解这一差异,我们比较了几种备选培训形式:

• • segment-pair+nsp: This follows the original input format used in BERT Devlin et al. 2019, with the NSP loss. Each input has a pair of segments, which can each contain multiple natural sentences, but the total combined length must be less than 512 tokens.
• • sentence-pair+nsp: Each input contains a pair of natural sentences, either sampled from a contiguous portion of one document or from separate documents. Since these inputs are significantly shorter than 512 tokens, we increase the batch size so that the total number of tokens remains similar to segment-pair+nsp. We retain the NSP loss.
• • full-sentences: Each input is packed with full sentences sampled contiguously from one or more documents, such that the total length is at most 512 tokens. Inputs may cross document boundaries. When we reach the end of one document, we begin sampling sentences from the next document and add an extra separator token between documents. We remove the NSP loss.
• • doc-sentences: Inputs are constructed similarly to full-sentences, except that they may not cross document boundaries. Inputs sampled near the end of a document may be shorter than 512 tokens, so we dynamically increase the batch size in these cases to achieve a similar number of total tokens as full-sentences. We remove the NSP loss.

• 段-段+nsp: 这沿用了BERT Devlin等人2019年使用的原始输入格式,NSP丢失. 每个输入都有一对片段,每个片段可以包含多个自然句子,但总合长度必须小于512个令牌.
• 句子-pair+nsp:每个输入包含一对自然的判决,从一份文件的毗连部分或单独的文件中取样。 由于这些投入大大短于512个令牌,所以我们增加批量大小,使令牌总数保持与段-pair+nsp相似. 我们保留了NSP的损失。
• 完全判决:每份输入都用完整句子作为样本,从一份或多份文件中取出,总长度最多512个符号。 输入可能跨越文档边界。 当我们到达一个文档的结尾时,我们开始从下一个文档中取样句子,并在文档之间添加一个额外的分隔符符. 我们取消了NSP的损失。
• 判决:投入与完全判决相似,但不得跨越文件界限。 在文件末端采样的输入物可能短于512个令牌,因此我们动态地增加这些案件的批量大小,以达到与全刑相同的总令牌数量. 我们取消了NSP的损失。

Results

Table 2 shows results for the four different settings. We first compare the original segment-pair input format from Devlin et al. 2019 to the sentence-pair format; both formats retain the NSP loss, but the latter uses single sentences. We find that using individual sentences hurts performance on downstream tasks, which we hypothesize is because the model is not able to learn long-range dependencies.

结果

表2显示四种不同设置的结果。 我们首先将Devlin等人2019年最初的段-pair输入格式与句-pair格式进行比较;两种格式都保留了NSP损失,但后者使用单句. 我们发现使用单个句子会伤害下游任务上的性能,我们假设这是因为模型无法学习远距离依赖性.

We next compare training without the NSP loss and training with blocks of text from a single document (doc-sentences). We find that this setting outperforms the originally published BERT${}_{\textsc{base}}$ results and that removing the NSP loss matches or slightly improves downstream task performance, in contrast to Devlin et al. 2019. It is possible that the original BERT implementation may only have removed the loss term while still retaining the segment-pair input format.

接下来,我们将没有国家战略计划损失的培训与单一文件(判决书)的文本进行对比。 我们发现,这一表述超越了最初公布的BERT ${}_{\textsc{base}}$结果,** 取消NSP损失匹配或略微改善下游任务性能**,与Devlin等人2019年相比。 最初的BERT执行可能只是删除了损失的术语,同时仍然保留了片段-pair输入格式.

Finally we find that restricting sequences to come from a single document (doc-sentences) performs slightly better than packing sequences from multiple documents (full-sentences). However, because the doc-sentences format results in variable batch sizes, we use full-sentences in the remainder of our experiments for easier comparison with related work.

以最佳方式,我们最终发现,限制单一文件(判决书)中的序列,其表现略优于多份文件(判决书)中的包装序列。 然而,由于剂量判决格式产生可变批量大小,我们在剩余实验中使用全量判决,以便于与相关工作进行比较.

4.3 Training with large batches

4.3 大批次训练

Past work in Neural Machine Translation has shown that training with very large mini-batches can both improve optimization speed and end-task performance when the learning rate is increased appropriately Ott et al. 2018. Recent work has shown that BERT is also amenable to large batch training You et al. 2019.

过去在神经机器翻译表面上的研究表明,在学习率适当提高时,使用非常大型的微型枪管的培训既可以提高优化速度,也可以提高终端任务性能. Ott et al. 2018. 近期的研究表明,BERT也能够接受大批量的You等人2019年培训.

Devlin et al. 2019 originally trained BERT${}_{\textsc{base}}$ for 1M steps with a batch size of 256 sequences. This is equivalent in computational cost, via gradient accumulation, to training for 125K steps with a batch size of 2K sequences, or for 31K steps with a batch size of 8K.

Devlin等人 2019年最初为1M步训练BERT ${}_{\textsc{base}}$,批量大小为256个序列. 这在计算成本中,通过梯度积累,相当于对批量大小为2K序列的125K步或批量大小为8K的31K步的培训.

bsz	steps	lr	ppl	MNLI-m	SST-2
256	1M	1e-4	3.99	84.7	92.7
2K	125K	7e-4	3.68	85.2	92.9
8K	31K	1e-3	3.77	84.6	92.8

Table 3: Perplexity on held-out training data (ppl) and development set accuracy for base models trained over BookCorpus and Wikipedia with varying batch sizes (bsz). We tune the learning rate (lr) for each setting. Models make the same number of passes over the data (epochs) and have the same computational cost.

页:1	** 步骤**	页:1	页:1	** MNLI-m** (韩语)	** 《大会正式记录,第五十八届会议,补编第6号》(A/59/6),第二章。
256	1门	1个e-4	3.99	84.7	92.7
2K (韩语)	125K 导弹	7个e-4	** 3.68** (单位:千美元)	页:1	92.9 (单位:千美元)
八千块	平等地使用	1个e-3	3.77	84.6	92.8

表3:对停产培训数据(ppl)的困惑和在BookCorpus和Wikipedia上培训的基准模型的开发精度(bsz) 我们调整每个环境的学习率(lr)。 模型通过数据的次数相同(时间段),计算成本相同.

In Table 3 we compare perplexity and end-task performance of BERT${}_{\textsc{base}}$ as we increase the batch size, controlling for the number of passes through the training data. We observe that training with large batches improves perplexity for the masked language modeling objective, as well as end-task accuracy. Large batches are also easier to parallelize via distributed data parallel training,[^8] and in later experiments we train with batches of 8K sequences.

在表3中,我们比较了BERT ${}_{\textsc{base}}$的复杂性和最终任务性能,因为我们增加了批量规模,控制了通过培训数据的次数。 我们注意到,大批量的培训提高了蒙面语言建模目标的模糊性,以及最终任务的准确性。 通过分布式数据平行训练,大批次也比较容易平行,[^8] 在后来的实验中,我们用批次的8K序列进行训练.

Notably You et al. 2019 train BERT with even larger batche sizes, up to 32K sequences. We leave further exploration of the limits of large batch training to future work.

显著 You et al. 2019列车BERT的批量尺寸甚至更大,最高可达32K序列. 我们把对大批量培训限度的进一步探索留给今后的工作。

4.4 Text Encoding

4.4 文本编码

Byte-Pair Encoding (BPE) Sennrich et al. 2016 is a hybrid between character- and word-level representations that allows handling the large vocabularies common in natural language corpora. Instead of full words, BPE relies on subwords units, which are extracted by performing statistical analysis of the training corpus.

Byte-Pair Encoding (BPE) Sennrich et al. 2016是字符与字位代表的混合体,可以处理自然语言corpora中常见的大字典. BPE不全词,而是依靠子词单位,通过对培训书进行统计分析来提取.

BPE vocabulary sizes typically range from 10K-100K subword units. However, unicode characters can account for a sizeable portion of this vocabulary when modeling large and diverse corpora, such as the ones considered in this work. Radford et al. 2019 introduce a clever implementation of BPE that uses bytes instead of unicode characters as the base subword units. Using bytes makes it possible to learn a subword vocabulary of a modest size (50K units) that can still encode any input text without introducing any “unknown” tokens.

BPE词汇大小一般在10K-100K子词单位之间. 然而,Unicode字符在建模大而多样的corpora时可以占这个词汇的相当大一部分,如本作中考虑的. Radford et al. 2019引入了BPE的巧妙执行,使用字节而不是unicode字符作为底子词单元. 使用字节使得可以学习一个大小不大(50K单位)的子词词汇,在不引入任何"未知"符号的情况下仍然可以编码任何输入文本.

The original BERT implementation Devlin et al. 2019 uses a character-level BPE vocabulary of size 30K, which is learned after preprocessing the input with heuristic tokenization rules. Following Radford et al. 2019, we instead consider training BERT with a larger byte-level BPE vocabulary containing 50K subword units, without any additional preprocessing or tokenization of the input. This adds approximately 15M and 20M additional parameters for BERT${}_{\textsc{base}}$ and BERT${}_{\textsc{large}}$, respectively.

最初的BERT执行Devlin et al. 2019使用一个大小为30K的字符级BPE词汇,在预处理输入后用Heuristic sorderization规则学习. 在Ladford et al. 2019年之后,我们转而考虑训练BERT,其中含有50K子词单元的较大型的字节级BPE词汇,没有附加预处理或标注输入. 这为BERT ${}_{\textsc{base}}$和BERT ${}_{\textsc{large}}$分别增加了约15M和20M的附加参数.

Early experiments revealed only slight differences between these encodings, with the Radford et al. 2019 BPE achieving slightly worse end-task performance on some tasks. Nevertheless, we believe the advantages of a universal encoding scheme outweighs the minor degredation in performance and use this encoding in the remainder of our experiments. A more detailed comparison of these encodings is left to future work.

早期的实验只揭示了这些编码之间的细微差异,拉德福德等人的2019年BPE在一些任务上实现了略差的终端任务性能. 然而,我们认为,通用编码办法的优点超过了性能上的微小分解,在我们其余的实验中使用这种编码. 对这些编码进行更详细的比较,留待未来工作处理。

5 RoBERTa

Model	data	bsz	steps	SQuAD	MNLI-m	SST-2
(v1.1/2.0)
RoBERTa
with Books + Wiki	16GB	8K	100K	93.6/87.3	89.0	95.3
+ additional data (§3.2)	160GB	8K	100K	94.0/87.7	89.3	95.6
+ pretrain longer	160GB	8K	300K	94.4/88.7	90.0	96.1
+ pretrain even longer	160GB	8K	500K	94.6/89.4	90.2	96.4
BERT${}_{\textsc{large}}$
with Books + Wiki	13GB	256	1M	90.9/81.8	86.6	93.7
XLNet${}_{\textsc{large}}$
with Books + Wiki	13GB	256	1M	94.0/87.8	88.4	94.4
+ additional data	126GB	2K	500K	94.5/88.8	89.8	95.6

Table 4: Development set results for RoBERTa as we pretrain over more data (16GB $\rightarrow$ 160GB of text) and pretrain for longer (100K $\rightarrow$ 300K $\rightarrow$ 500K steps). Each row accumulates improvements from the rows above. RoBERTa matches the architecture and training objective of BERT${}_{\textsc{large}}$. Results for BERT${}_{\textsc{large}}$ and XLNet${}_{\textsc{large}}$ are from Devlin et al. 2019 and Yang et al. 2019, respectively. Complete results on all GLUE tasks can be found in the Appendix.

** 模式**	** 数据**	页:1	** 步骤**	** 《古兰经》**	** MNLI-m** (韩语)	** 《大会正式记录,第五十八届会议,补编第6号》(A/59/6),第二章。
(v1.1/2.0) (中文(简体) ).
RoBERTa
有书 + 维基	16GB (英语).	八千块	100K 车次	93.6/87.3	89.0	95.3
+ 额外数据(§3.2)	steria. kgm	八千块	100K 车次	94.0/87.7	89.3	95.6
+ 预训练时间更长	160GB (英语).	鸠8K	300K 车次	94.4/88.7	90.0	96.1
+ 预训练时间更长	160GB (英语).	八千块	500K 车次	94.6**89.4	** 第90.2段**	页:1
BERT 第二十五届
有书 + 维基	13GB (英语).	256	1门	90.9/81.8	86.6	93.7
XLNet 第二十届
有书 + 维基	13GB (英语).	256	1M讷	94.0/87.8	88.4	94.4
+ 额外数据	126GB (英语).	2K (韩语)	5 000人 5 000人 营地	94.5/88.8	89.8	95.6

表4: 当我们对更多数据(16GB$\rightarrow$160GB文本)进行预训练和对更长时间(100K$\rightarrow$300K$\rightarrow$500K步骤)进行预训练时,为RoBERTA制定开发结果。 每行从以上各行累积改进. RoBERTA符合BERT ${}_{\textsc{large}}$的建筑和训练目标。 BERT ${}_{\textsc{large}}$和XLNet ${}_{\textsc{large}}$的结果分别来自Devlin等人2019年和Yang等人2019年. 所有GLUE任务的完整结果可参见附录.

In the previous section we propose modifications to the BERT pretraining procedure that improve end-task performance. We now aggregate these improvements and evaluate their combined impact. We call this configuration RoBERTa for Robustly optimized BERT approach. Specifically, RoBERTa is trained with dynamic masking (Section 4.1), full-sentences without NSP loss (Section 4.2), large mini-batches (Section 4.3) and a larger byte-level BPE (Section 4.4).

在上一节中,我们提议修改BERT预训练程序,以改进最终任务的业绩。 我们现在汇总这些改进并评估其综合影响。 我们称这一配置为RbustlyoptimizedBERT****appromach。 具体地说,RoBERTa受过动态蒙面(第4.1节)、全刑而不丧失国家战略计划(第4.2节)、大型小枪炮(第4.3节)和较大的字节级BPE(第4.4节)的培训。

Additionally, we investigate two other important factors that have been under-emphasized in previous work: (1) the data used for pretraining, and (2) the number of training passes through the data. For example, the recently proposed XLNet architecture Yang et al. 2019 is pretrained using nearly 10 times more data than the original BERT Devlin et al. 2019. It is also trained with a batch size eight times larger for half as many optimization steps, thus seeing four times as many sequences in pretraining compared to BERT.

此外,我们调查了先前工作强调不足的另外两个重要因素:(1) 用于预训练的数据;(2) 培训通过的数据数量。 例如,最近提出的XLNet架构杨等人2019的预训练使用的数据比最初的BERT Devlin等人2019年多近10倍. 与BERT相比,它在预训练中还接受了批量大小8倍的训练,以达到一半的优化步骤,因此在预训练中看到的序列是4倍.

To help disentangle the importance of these factors from other modeling choices (e.g., the pretraining objective), we begin by training RoBERTa following the BERT${}_{\textsc{large}}$ architecture ($L=24$, $H=1024$, $A=16$, 355M parameters). We pretrain for 100K steps over a comparable BookCorpus plus Wikipedia dataset as was used in Devlin et al. 2019. We pretrain our model using 1024 V100 GPUs for approximately one day.

为了帮助将这些因素的重要性与其他建模选择(如预训练目标)区分开来,我们首先按照BERT ${}_{\textsc{large}}$架构($L=24$,$H=1024$,$A=16$,355M参数)来培训RoBERTA. 我们通过一个可比较的BookCorpus加维基百科数据集进行100K步骤的预选,该数据集在Devlin等人2019年使用. 我们用1024V100GPU预演我们的模型,大约一天.

Results

We present our results in Table 4. When controlling for training data, we observe that RoBERTa provides a large improvement over the originally reported BERT${}_{\textsc{large}}$ results, reaffirming the importance of the design choices we explored in Section 4.

结果

我们在表4中介绍了我们的成果。 在控制培训数据时,我们发现,RoBERTA比最初报告的BERT ${}_{\textsc{large}}$结果大有改进,重申了我们在第4节中探讨的设计选择的重要性。

Next, we combine this data with the three additional datasets described in Section 3.2. We train RoBERTa over the combined data with the same number of training steps as before (100K). In total, we pretrain over 160GB of text. We observe further improvements in performance across all downstream tasks, validating the importance of data size and diversity in pretraining.[^9]

其次,我们将这一数据与第3.2节所述三个补充数据集合并。 我们通过综合数据培训RoBERTa,培训步骤与以前(100K)相同。 总共我们预选了160GB的文本. 我们注意到,所有下游任务的业绩都有了进一步改善,证实了数据规模和培训前多样性的重要性。

	MNLI	QNLI	QQP	RTE	SST	MRPC	CoLA	STS	WNLI	Avg
Single-task single models on dev
BERT${}_{\textsc{large}}$	86.6/-	92.3	91.3	70.4	93.2	88.0	60.6	90.0	-	-
XLNet${}_{\textsc{large}}$	89.8/-	93.9	91.8	83.8	95.6	89.2	63.6	91.8	-	-
RoBERTa	90.2/90.2	94.7	92.2	86.6	96.4	90.9	68.0	92.4	91.3	-
Ensembles on test (from leaderboard as of July 25, 2019)
ALICE	88.2/87.9	95.7	90.7	83.5	95.2	92.6	68.6	91.1	80.8	86.3
MT-DNN	87.9/87.4	96.0	89.9	86.3	96.5	92.7	68.4	91.1	89.0	87.6
XLNet	90.2/89.8	98.6	90.3	86.3	96.8	93.0	67.8	91.6	90.4	88.4
RoBERTa	90.8/90.2	98.9	90.2	88.2	96.7	92.3	67.8	92.2	89.0	88.5

Table 5: Results on GLUE. All results are based on a 24-layer architecture. BERT${}_{\textsc{large}}$ and XLNet${}_{\textsc{large}}$ results are from Devlin et al. 2019 and Yang et al. 2019, respectively. RoBERTa results on the development set are a median over five runs. RoBERTa results on the test set are ensembles of single-task models. For RTE, STS and MRPC we finetune starting from the MNLI model instead of the baseline pretrained model. Averages are obtained from the GLUE leaderboard.

	页:1	页:1	页:1	页:1	页:1	** MRPC **	页:1	页:1	** 妇联**	页:1
通话单任务模式
BERT 第二十五届	86.6/-	92.3	91.3	70.4	93.2	88.0	60.6	90.0	-	-
XLNet 第二十届	89.8/-	93.9	91.8	83.8	95.6	89.2	63.6	91.8	-	-
可在RoBERTa	页:1	页:1	92.2 经常预算:	页:1	页:1	页:1	页:1	92.4 国家	91.3 经常预算:	-
* 参加测试(截至2019年7月25日从领导板)*
爱丽丝	88.2/87.9	95.7	定期向大会提交报告。	83.5	95.2	92.6	页:1	91.1	80.8	86.3
MT - 数字	87.9/87.4	96.0	89.9	86.3	96.5	92.7	68.4	91.1	89.0	87.6
XLNet 网络	90.2/89.8	98.6	90.3	86.3	** 第96 那么,第76段。	** 93.0 ** 中文无改动。	67.8	91.6	90.4 (单位:千美元)	88.4
RoBERTa	页:1		90.2	** ** 88.2 **	96.7	92.3	67.8	92.2 经常预算:	89.0	* ** 88.5**

表5:GLUE的结果。 所有结果都基于24层建筑. BERT ${}_{\textsc{large}}$和XLNet ${}_{\textsc{large}}$的结果分别来自Devlin等人2019年和Yang等人2019年. RoBERTA的开发结果在5次运行中位数。 RoBERTA在测试集上的结果是单任务模型的组合. 对于RTE,STS和MRPC,我们从MNLI模型开始微调,而不是基线预试模型. 平均值来自GLUE领导板.

Finally, we pretrain RoBERTa for significantly longer, increasing the number of pretraining steps from 100K to 300K, and then further to 500K. We again observe significant gains in downstream task performance, and the 300K and 500K step models outperform XLNet${}_{\textsc{large}}$ across most tasks. We note that even our longest-trained model does not appear to overfit our data and would likely benefit from additional training.

最后,我们为RoBERTA预训练时间大大延长,将预训练步骤从100K增加到300K,然后进一步增加到500K. 我们再次看到,下游任务业绩取得显著进展,大多数任务的300K和500K步骤模型都超过了XLNet ${}_{\textsc{large}}$。 我们注意到,即使我们训练时间最长的模型也似乎没有超出我们的数据,而且可能受益于更多的培训。

In the rest of the paper, we evaluate our best RoBERTa model on the three different benchmarks: GLUE, SQuaD and RACE. Specifically we consider RoBERTa trained for 500K steps over all five of the datasets introduced in Section 3.2.

在论文的其余部分,我们用三个不同的基准来评价我们最好的RoBERTa模式:GLUE,SQuaD和RACE. 具体地说,我们认为,RoBERTa在第3.2节介绍的所有五个数据集中,都经过了500K步骤的培训。

5.1 GLUE Results

5.1 GLUE 结果

For GLUE we consider two finetuning settings. In the first setting (single-task, dev) we finetune RoBERTa separately for each of the GLUE tasks, using only the training data for the corresponding task. We consider a limited hyperparameter sweep for each task, with batch sizes $\in\{16,32\}$ and learning rates $\in\{1e-5,2e-5,3e-5\}$, with a linear warmup for the first 6% of steps followed by a linear decay to 0. We finetune for 10 epochs and perform early stopping based on each task’s evaluation metric on the dev set. The rest of the hyperparameters remain the same as during pretraining. In this setting, we report the median development set results for each task over five random initializations, without model ensembling.

对于GLUE,我们考虑两个微调设置. 在第一个设置(* single-task, dev*)中,我们针对GLUE任务的每一项分别微调RoBERTa,只使用训练数据来完成相应的任务. 我们考虑对每项任务进行有限的超参数扫描,批量大小为$\in\{16,32\}$,学习速率为$\in\{1e-5,2e-5,3e-5\}$,前6%步骤的线性暖化,然后线性衰变为0。 我们根据每个任务在dev集上的评价指标来细化10个时代, 其余超参数与预训练时相同. 在此情况下,我们报告在5个随机初始化的基础上为每项任务设定的中位发展结果,而不采用模型组合。

In the second setting (ensembles, test), we compare RoBERTa to other approaches on the test set via the GLUE leaderboard. While many submissions to the GLUE leaderboard depend on multi-task finetuning, our submission depends only on single-task finetuning. For RTE, STS and MRPC we found it helpful to finetune starting from the MNLI single-task model, rather than the baseline pretrained RoBERTa. We explore a slightly wider hyperparameter space, described in the Appendix, and ensemble between 5 and 7 models per task.

在第二种设置(commbles, test )中,我们通过GLUE导板将RoBERTa与测试集上的其他方法进行比较. 虽然向GLUE领导板提交的很多文件都依赖于多任务微调,但**我们的提交仅依赖于单任务微调. 对RTE,STS和MRPC来说,我们发现从MNLI单任务模式开始微调,而不是基线预先训练的RoBERTa,很有帮助. 我们在附录中探索了一个略宽的超参数空间,每个任务共5至7个模型。

Task-specific modifications

Two of the GLUE tasks require task-specific finetuning approaches to achieve competitive leaderboard results.

任务特定修改

两项GLUE任务需要针对具体任务的微调方法纸张,以实现有竞争力的领导板结果.

QNLI: Recent submissions on the GLUE leaderboard adopt a pairwise ranking formulation for the QNLI task, in which candidate answers are mined from the training set and compared to one another, and a single (question, candidate) pair is classified as positive Liu et al. 2019b; Liu et al. 2019a; Yang et al. 2019. This formulation significantly simplifies the task, but is not directly comparable to BERT Devlin et al. 2019. Following recent work, we adopt the ranking approach for our test submission, but for direct comparison with BERT we report development set results based on a pure classification approach.

QNLI:关于GLUE领导板的近期提交材料采用QNLI任务的双向排名配方,其中选手的答案来自培训集并相互比较,单对(质疑,候选)被归类为阳性刘等人2019b;刘等人2019a;杨等人2019. 这种配方大大简化了任务,但不能直接与BERT Devlin等人2019年相比. 继最近的工作之后,我们在提交测试报告时采用了排名办法,但为了与BERT直接比较,我们报告发展的结果是基于纯粹的分类办法。

WNLI: We found the provided NLI-format data to be challenging to work with. Instead we use the reformatted WNLI data from SuperGLUE Wang et al. 2019a, which indicates the span of the query pronoun and referent. We finetune RoBERTa using the margin ranking loss from Kocijan et al. 2019. For a given input sentence, we use spaCy Honnibal and Montani 2017 to extract additional candidate noun phrases from the sentence and finetune our model so that it assigns higher scores to positive referent phrases than for any of the generated negative candidate phrases. One unfortunate consequence of this formulation is that we can only make use of the positive training examples, which excludes over half of the provided training examples.[^10]

我们发现所提供的NLI格式数据很难使用。 取而代之的是使用SuperGLUE Wang等人2019年a的重排WNLI数据,该数据表示查询代词和参考词的跨度. 我们利用来自Kocijan等人的比值排名损失来微调RoBERTa. 对于一个给定的输入句,我们使用spaCy Honnibal和Montani 2017从句子中提取额外的候选名词短语,并对我们的模型进行微调,这样它就会将比任何生成的负面候选短语的正面参考短语的分数分配得更高. 这一提法的一个不幸后果是,我们只能利用积极的培训实例,其中不包括所提供的培训实例的一半以上。

Results

We present our results in Table 5. In the first setting (single-task, dev), RoBERTa achieves state-of-the-art results on all 9 of the GLUE task development sets. Crucially, RoBERTa uses the same masked language modeling pretraining objective and architecture as BERT${}_{\textsc{large}}$, yet consistently outperforms both BERT${}_{\textsc{large}}$ and XLNet${}_{\textsc{large}}$. This raises questions about the relative importance of model architecture and pretraining objective, compared to more mundane details like dataset size and training time that we explore in this work.

结果

我们在表5中介绍了我们的成果。 在第一个设置(* single-task, dev*)中,RoBERTa在GLUE任务开发集的所有9集上都取得了最新的结果. 关键是,RoBERTA使用与BERT ${}_{\textsc{large}}$相同的蒙面语言模型预训练目标和架构,但一贯优于BERT ${}_{\textsc{large}}$和XLNet ${}_{\textsc{large}}$. 这提出了与我们在本作品中探索的数据集尺寸和训练时间等更普通的细节相比,模型架构和预训练目标的相对重要性问题.

In the second setting (ensembles, test), we submit RoBERTa to the GLUE leaderboard and achieve state-of-the-art results on 4 out of 9 tasks and the highest average score to date. This is especially exciting because RoBERTa does not depend on multi-task finetuning, unlike most of the other top submissions. We expect future work may further improve these results by incorporating more sophisticated multi-task finetuning procedures.

在第二个设定(*countbles, test *)中,我们向GLUE领导板提交RoBERTa,并在9项任务中的4项任务中取得最新成绩,也是迄今为止平均得分最高的. 这尤其令人兴奋,因为RoBERTa并不依赖于多任务微调,不像其他大多数顶级提交. 我们期望今后的工作能够通过采用更复杂的多任务微调程序来进一步改善这些成果。

5.2 SQuAD Results

5.2 SQuAD 结果

We adopt a much simpler approach for SQuAD compared to past work. In particular, while both BERT Devlin et al. 2019 and XLNet Yang et al. 2019 augment their training data with additional QA datasets, we only finetune RoBERTa using the provided SQuAD training data. Yang et al. 2019 also employed a custom layer-wise learning rate schedule to finetune XLNet, while we use the same learning rate for all layers.

与过去的工作相比,我们对SQuAD采取简单得多的方法。 特别是,虽然BERT Devlin等人2019年和XLNet Yang等人都用额外的QA数据集增加了培训数据,但** 我们只利用所提供的SQuAD培训数据对RoBERTa进行微调**. Yang 等人 2019 也采用自定义的分层学习率表来微调 XLNet,同时我们对所有各层使用相同的学习率.

For SQuAD v1.1 we follow the same finetuning procedure as Devlin et al. 2019. For SQuAD v2.0, we additionally classify whether a given question is answerable; we train this classifier jointly with the span predictor by summing the classification and span loss terms.

对于SQuAD v1.1,我们遵循与Devlin等人2019相同的微调程序. 对于SQuAD v2.0,我们额外分类某个问题是否可回答;我们通过汇总分类和跨损失条款来与跨度预测器一起训练这个分类器.

Results

Model	SQuAD 1.1	SQuAD 2.0
EM	F1	EM	F1
Single models on dev, w/o data augmentation
BERT${}_{\textsc{large}}$	84.1	90.9	79.0	81.8
XLNet${}_{\textsc{large}}$	89.0	94.5	86.1	88.8
RoBERTa	88.9	94.6	86.5	89.4
Single models on test (as of July 25, 2019)
XLNet${}_{\textsc{large}}$	86.3*†*	89.1*†*
RoBERTa	86.8	89.8
XLNet + SG-Net Verifier	87.0†**	89.9†**

Table 6: Results on SQuAD. ${\dagger}$ indicates results that depend on additional external training data. RoBERTa uses only the provided SQuAD data in both dev and test settings. BERT${}_{\textsc{large}}$ and XLNet${}_{\textsc{large}}$ results are from Devlin et al. 2019 and Yang et al. 2019, respectively.

结果

港口
|,|也.
额,额,额,额,额...
-=YTET -伊甸园字幕组=- 翻译:
|${}_{\textsc{large}}$| 84.1 | 90.9 | 79.0 | 81.8 |

• XLNet${}_{\textsc{large}}$** 89.0** * 94.5 * 86.1 * 88.8 * * *
  · RoBERTa · 88.9 · ** 94.6 ** · ** 86.5 ** · ** 89.4 ** ·
• 测试中的单项模型(截至2019年7月25日)* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ** * * ** * * * * * * ** * * ** * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
  XLNet${}_{\textsc{large}}$| 86.3*†* | 89.1*†* |
  芷叶(RoBERTa) |86.8 |89.8 | |
  XLNet + SG-Net 校验器 **87.0 → **89.9 + + + + + + + +

表6:SQUAD的结果 ${\dagger}$显示的结果取决于额外的外部培训数据. RoBERTa只在dev和测试设置中使用所提供的SQuAD数据. BERT ${}_{\textsc{large}}$和XLNet ${}_{\textsc{large}}$的结果分别来自Devlin等人2019年和Yang等人2019年.

We present our results in Table 6. On the SQuAD v1.1 development set, RoBERTa matches the state-of-the-art set by XLNet. On the SQuAD v2.0 development set, RoBERTa sets a new state-of-the-art, improving over XLNet by 0.4 points (EM) and 0.6 points (F1).

我们的结果见表6。 在SQuAD v1.1开发集上,RoBERTa匹配了XLNet最先进的设定. 在SQuAD v2.0开发集上,RoBERTa设定了一个新的最先进的,比XLNet改进0.4分(EM)和0.6分(F1).

We also submit RoBERTa to the public SQuAD 2.0 leaderboard and evaluate its performance relative to other systems. Most of the top systems build upon either BERT Devlin et al. 2019 or XLNet Yang et al. 2019, both of which rely on additional external training data. In contrast, our submission does not use any additional data.

我们还将RoBERTa提交公众SQuAD 2.0领导板,并比照其他系统评价其性能. 大多数顶级系统都建立在BERT Devlin等人2019或XLNet Yang等人2019的基础上,两者都依赖于额外的外部培训数据. 与此相反,我们提交的文件没有使用任何其他数据。

Our single RoBERTa model outperforms all but one of the single model submissions, and is the top scoring system among those that do not rely on data augmentation.

我们的单一的RoBERTa模型,除了一个模型的提交之外,都超过了其他所有模型,它是那些不依赖数据增强的模型中最高的评分系统.

5.3 RACE Results

5.3 RACE 结果

In RACE, systems are provided with a passage of text, an associated question, and four candidate answers. Systems are required to classify which of the four candidate answers is correct.

在RACE中,向系统提供了文本的段落、相关问题和四个候选答案。 需要建立系统,对四个候选答案中哪一个是正确的进行分类。

We modify RoBERTa for this task by concatenating each candidate answer with the corresponding question and passage. We then encode each of these four sequences and pass the resulting [CLS] representations through a fully-connected layer, which is used to predict the correct answer. We truncate question-answer pairs that are longer than 128 tokens and, if needed, the passage so that the total length is at most 512 tokens.

我们修改RoBERTa以完成这项任务,将每个候选人的回答与相应的问题和段落结合起来。 然后,我们将这四个序列中的每一个编码,并通过一个完全连接的层来传递由此产生的*[CLS]*表示,这个层被用来预测正确的答案. 我们缩短回答对 超过128个符号, 如果需要的话, 通道, 这样总长度最多512个符号。

Model	Accuracy	Middle	High
Single models on test (as of July 25, 2019)
BERT${}_{\textsc{large}}$	72.0	76.6	70.1
XLNet${}_{\textsc{large}}$	81.7	85.4	80.2
RoBERTa	83.2	86.5	81.3

Table 7: Results on the RACE test set. BERT${}_{\textsc{large}}$ and XLNet${}_{\textsc{large}}$ results are from Yang et al. 2019.

** 模式**	** 准确性**	** 中间**	** 高级**
测试时的单一型号(截至2019年7月25日)
BERT 第二十五届	72.0	76.6	70.1
XLNet 第二十届	81.7	85.4	80.2
RoBERTa	页:1	**充实了《大会正式记录,第五十七届会议,补编第6号》(A/59/6/Rev.1)。	页:1

表7: RACE测试结果。 BERT ${}_{\textsc{large}}$和XLNet ${}_{\textsc{large}}$的结果来自杨等人 2019.

Results on the RACE test sets are presented in Table 7. RoBERTa achieves state-of-the-art results on both middle-school and high-school settings.

表7列示了RACE成套测试的结果。 RoBERTa在初中和高中环境都取得了最先进的成绩.

6 Related Work

6 相关工作

Pretraining methods have been designed with different training objectives, including language modeling Dai and Le 2015; Peters et al. 2018; Howard and Ruder 2018, machine translation McCann et al. 2017, and masked language modeling Devlin et al. 2019; Lample and Conneau 2019. Many recent papers have used a basic recipe of finetuning models for each end task Howard and Ruder 2018; Radford et al. 2018, and pretraining with some variant of a masked language model objective. However, newer methods have improved performance by multi-task fine tuning Dong et al. 2019, incorporating entity embeddings Sun et al. 2019, span prediction Joshi et al. 2019, and multiple variants of autoregressive pretraining Song et al. 2019; Chan et al. 2019; Yang et al. 2019. Performance is also typically improved by training bigger models on more data Devlin et al. 2019; Baevski et al. 2019; Yang et al. 2019; Radford et al. 2019. Our goal was to replicate, simplify, and better tune the training of BERT, as a reference point for better understanding the relative performance of all of these methods.

预训练方法的设计有着不同的训练目标,包括语言模型Dai和Le2015;Peters等人2018;霍华德和Ruder 2018,机器翻译McCann等人2017,蒙面语言模型Devlin等人2019;Lample和Conneau2019. 许多近代论文都为每次结束任务霍华德和鲁德2018使用过微调模型的基本配方;拉德福德等人2018年,并预训练带有某种变体的蒙面语言模型目标. 然而,较新的方法通过多任务微调Dong等人2019,整合实体嵌入Sun等人2019,横幅预测Joshi等人2019,以及自递性预训练Song等人2019,Chan等人2019,Yang等人2019等多种变体,提高了性能. 性能也一般通过在更多的数据Devlin等人2019年,Baevski等人2019年,Yang等人2019年,Radford等人2019年等方面培训更大的模型来提高. 我们的目标是复制、简化和更好地调整BERT的培训,作为更好地了解所有这些方法的相对性能的参照点。

7 Conclusion

7 结论

We carefully evaluate a number of design decisions when pretraining BERT models. We find that performance can be substantially improved by training the model longer, with bigger batches over more data; removing the next sentence prediction objective; training on longer sequences; and dynamically changing the masking pattern applied to the training data. Our improved pretraining procedure, which we call RoBERTa, achieves state-of-the-art results on GLUE, RACE and SQuAD, without multi-task finetuning for GLUE or additional data for SQuAD. These results illustrate the importance of these previously overlooked design decisions and suggest that BERT’s pretraining objective remains competitive with recently proposed alternatives.

我们仔细评估一些设计决定 当预先训练BERT模型。 我们发现,通过对模型进行更长的培训,使更多的批次超过更多的数据,取消下一句的预测目标,就较长的序列进行培训,以及动态地改变对培训数据采用的遮掩模式,可以大大提高绩效。 我们改进的预训练程序,我们称之为RoBERTa,在GLUE,RACE和SQuAD上实现了最新的结果,没有GLUE的多任务微调或SQuAD的额外数据. 这些结果表明这些以前被忽视的设计决定的重要性,并表明BERT的培训前目标仍然与最近提出的替代办法有竞争力。

We additionally use a novel dataset, CC-News, and release our models and code for pretraining and finetuning at: https://github.com/pytorch/fairseq.

并发布我们的模型和代码, 用于预训练和微调:https://github.com/pytorch/fairseq。

References

参考文献

	MNLI	QNLI	QQP	RTE	SST	MRPC	CoLA	STS
RoBERTa${}_{\textsc{base}}$
+ all data + 500k steps	87.6	92.8	91.9	78.7	94.8	90.2	63.6	91.2
RoBERTa${}_{\textsc{large}}$
with Books + Wiki	89.0	93.9	91.9	84.5	95.3	90.2	66.3	91.6
+ additional data (§3.2)	89.3	94.0	92.0	82.7	95.6	91.4	66.1	92.2
+ pretrain longer 300k	90.0	94.5	92.2	83.3	96.1	91.1	67.4	92.3
+ pretrain longer 500k	90.2	94.7	92.2	86.6	96.4	90.9	68.0	92.4

Table 8: Development set results on GLUE tasks for various configurations of RoBERTa.

	页:1	页:1	页:1	页:1	页:1	** MRPC **	** 《一句话》。	页:1
RoBERTa
+ 所有数据+ 500k 步	87.6	92.8	91.9	78.7	94.8	90.2	63.6	91.2
RoBERTa
有书 + 维基	89.0	93.9	91.9	84.5	95.3	90.2	66.3	91.6
+ 立足于额外数据(§3.2)	89.3	94.0	92.0	82.7	95.6	页:1	66.1	92.2
+ 预排所给予的300k	90.0	94.5	92.2 经常预算:	83.3	96.1	91.1	67.4	92.3
+ 预装500k	** 第90.2段**	页:1	92.2 经常预算:	页:1	页:1	90.9	页:1	92.4 国家

表8 开发为RoBERTa的各种配置设定了GLUE任务的结果.

Hyperparam	RoBERTa${}_{\textsc{large}}$	RoBERTa${}_{\textsc{base}}$
Number of Layers	24	12
Hidden size	1024	768
FFN inner hidden size	4096	3072
Attention heads	16	12
Attention head size	64	64
Dropout	0.1	0.1
Attention Dropout	0.1	0.1
Warmup Steps	30k	24k
Peak Learning Rate	4e-4	6e-4
Batch Size	8k	8k
Weight Decay	0.01	0.01
Max Steps	500k	500k
Learning Rate Decay	Linear	Linear
Adam $\epsilon$	1e-6	1e-6
Adam $\beta_{1}$	0.9	0.9
Adam $\beta_{2}$	0.98	0.98
Gradient Clipping	0.0	0.0

Table 9: Hyperparameters for pretraining RoBERTa${}_{\textsc{large}}$ and RoBERTa${}_{\textsc{base}}$.

** 高级**	** 蛊惑人心*	** 蛊惑人心*
图层数	24	12
隐藏大小	1024	768
苟真内藏大小	4096	3072
性别平等问题	16	12
注意 过来	64	64
辍学	0.1	0.1
注意退学	0.1	0.1
温暖步骤	30k	24k
峰值学习率	页:1	6个e-4
批量大小	8k	8k
重量衰减	0.01	0.01
最大步骤	500k	500k
学习率下降	线性	线性
亚当·$\epsilon$	1个e-6	1个e-6
亚当·$\beta_{1}$	0.9	0.9
亚当·$\beta_{2}$	0.98	0.98
渐变剪接	0.0	0.0

表9:预训练RoBERTa ${}_{\textsc{large}}$和RoBERTa ${}_{\textsc{base}}$的超参数。

Hyperparam	RACE	SQuAD	GLUE
Learning Rate	1e-5	1.5e-5	{1e-5, 2e-5, 3e-5}
Batch Size	16	48	{16, 32}
Weight Decay	0.1	0.01	0.1
Max Epochs	4	2	10
Learning Rate Decay	Linear	Linear	Linear
Warmup ratio	0.06	0.06	0.06

Table 10: Hyperparameters for finetuning RoBERTa${}_{\textsc{large}}$ on RACE, SQuAD and GLUE.

** 高级**	文档	** 《古兰经》**	** 格卢**
学习率	1个e-5	1.5e-5 (单位:千美元)	[1e-5,2e-5,3e-5] (中文(简体) ).
批量大小	16	48	{16, 32}
重量衰减	0.1	0.01	0.1
最大 Epochs	4	2	10
学习率下降	线性	线性	线性
温和率	0.06	0.06	0.06

表 Sung 10:对RACE、SQuAD和GLUE上的RoBERTa ${}_{\textsc{large}}$进行微调的超参数。

Agirre et al. (2007) Eneko Agirre, Llu’is M‘arquez, and Richard Wicentowski, editors. 2007. Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007) .

Baevski et al. (2019) Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. arXiv preprint arXiv:1903.07785 .

Bar-Haim et al. (2006) Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment .

Bentivogli et al. (2009) Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge.

Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP) .

Chan et al. (2019) William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. 2019. KERMIT: Generative insertion-based modeling for sequences. arXiv preprint arXiv:1906.01604 .

Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment .

Dai and Le (2015) Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems (NIPS) .

Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Association for Computational Linguistics (NAACL) .

Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing .

Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197 .

Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing .

Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://web.archive.org/save/http://Skylion007.github.io/OpenWebTextCorpus .

Hamborg et al. (2017) Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science .

Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 .

Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.

Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 .

Iyer et al. (2016) Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. 2016. First quora dataset release: Question pairs. https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs .

Joshi et al. (2019) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529 .

Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) .

Kocijan et al. (2019) Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for winograd schema challenge. arXiv preprint arXiv:1905.06290 .

Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 .

Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 .

Levesque et al. (2011) Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning .

Liu et al. (2019a) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482 .

Liu et al. (2019b) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504 .

McCann et al. (2017) Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems (NIPS) , pages 6297–6308.

Micikevicius et al. (2018) Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In International Conference on Learning Representations .

Nagel (2016) Sebastian Nagel. 2016. Cc-news. http://web.archive.org/save/http://commoncrawl.org/2016/10/news-dataset-available .

Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq : A fast, extensible toolkit for sequence modeling. In North American Association for Computational Linguistics (NAACL): System Demonstrations .

Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT) .

Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop .

Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL) .

Radford et al. (2018) Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI.

Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI.

Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Association for Computational Linguistics (ACL) .

Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP) .

Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Association for Computational Linguistics (ACL) , pages 1715–1725.

Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP) .

Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning (ICML) .

Sun et al. (2019) Yu Stephanie Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xinlun Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223 .

Trinh and Le (2018) Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847 .

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems .

Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint 1905.00537 .

Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR) .

Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018. Neural network acceptability judgments. arXiv preprint 1805.12471 .

Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In North American Association for Computational Linguistics (NAACL) .

Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 .

You et al. (2019) Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. 2019. Reducing bert pre-training time from 3 days to 76 minutes. arXiv preprint arXiv:1904.00962 .

Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. arXiv preprint arXiv:1905.12616 .

Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.06724 .

Appendix for “RoBERTa: A Robustly Optimized BERT Pretraining Approach”

附录：“RoBERTa: A Robustly Optimized BERT Pretraining Approach”

Appendix A Full results on GLUE

附录 A GLUE 完整结果

In Table 8 we present the full set of development set results for RoBERTa. We present results for a large configuration that follows BERT${}_{\textsc{large}}$, as well as a base configuration that follows BERT${}_{\textsc{base}}$.

在表8中,我们介绍了RoBERTa的一整套发展成果。 我们介绍了BERT ${}_{\textsc{large}}$之后的大型配置的结果,以及BERT ${}_{\textsc{base}}$之后的基础配置。

Appendix B Pretraining Hyperparameters

附录 B 预训练超参数

Table 9 describes the hyperparameters for pretraining of RoBERTa${}_{\textsc{large}}$ and RoBERTa${}_{\textsc{base}}$

表9说明RoBERTa ${}_{\textsc{large}}$和RoBERTa ${}_{\textsc{base}}$的预训练超参数。

Appendix C Finetuning Hyperparameters

附录 C 微调超参数

Finetuning hyperparameters for RACE, SQuAD and GLUE are given in Table 10. We select the best hyperparameter values based on the median of 5 random seeds for each task.

表10列出了RACE、SQuAD和GLUE的微调超参数。 我们根据每个任务的5个随机种子的中位数来选择最佳超参数值.