BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

BART：用于自然语言生成、翻译与理解的去噪序列到序列预训练

Mike Lewis*    Yinhan Liu*    Naman Goyal*    Marjan Ghazvininejad Affiliation:  Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer Affiliation:  Facebook AI Email:  {mikelewis,yinhanliu,naman}@fb.com

Abstract

摘要

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa with comparable training resources on GLUE and SQuAD, achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 6 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also report ablation experiments that replicate other pretraining schemes within the BART framework, to better measure which factors most influence end-task performance.

我们提出 BART，一种用于预训练序列到序列模型的去噪自编码器。BART 的训练分为两步：(1) 用任意加噪函数腐蚀文本；(2) 学习模型重建原文。它采用标准的基于 Transformer 的神经机器翻译结构；尽管简单，却可看作对 BERT（双向编码器）、GPT（从左到右解码器）以及许多更新预训练方案的概括。我们评估多种加噪方式，发现最佳表现来自随机打乱原句顺序，并配合一种新的填空方案——用单个掩码标记替换文本跨度。BART 在微调文本生成时尤为有效，理解任务上同样出色：在可比训练资源下，GLUE 与 SQuAD 上与 RoBERTa 相当；在一系列抽象对话、问答与摘要任务上刷新结果，ROUGE 最多提升约 6；在仅目标语预训练下，机器翻译相对回译系统再增 1.1 BLEU。我们还报告在 BART 框架内复现其他预训练方案的消融，以更好度量哪些因素最影响端任务表现。

1 Introduction

1 引言

Self-supervised methods have achieved remarkable success in a wide range of NLP tasks  Mikolov et al. 2013; Peters et al. 2018; Devlin et al. 2019; Joshi et al. 2019; Yang et al. 2019; Liu et al. 2019. The most successful approaches have been variants of masked language models, which are denoising autoencoders that are trained to reconstruct text where a random subset of the words has been masked out. Recent work has shown gains by improving the distribution of masked tokens Joshi et al. 2019, the order in which masked tokens are predicted Yang et al. 2019, and the available context for replacing masked tokens Dong et al. 2019. However, these methods typically focus on particular types of end tasks (e.g. span prediction, generation, etc.), limiting their applicability.

自监督方法已在广泛 NLP 任务上取得显著成功（Mikolov et al. 2013；Peters et al. 2018；Devlin et al. 2019；Joshi et al. 2019；Yang et al. 2019；Liu et al. 2019）。最成功的一类是掩码语言模型变体，即训练去噪自编码器去重建随机掩掉一部分词后的文本。近期工作通过改进掩码 token 的分布（Joshi et al. 2019）、掩码 token 的预测顺序（Yang et al. 2019）以及替换掩码时可用的上下文（Dong et al. 2019）获得增益。但这些方法通常针对特定端任务（如跨度预测、生成等），限制了适用范围。

In this paper, we present BART, which pre-trains a model combining Bidirectional and Auto-Regressive Transformers. BART is a denoising autoencoder built with a sequence-to-sequence model that is applicable to a very wide range of end tasks. Pretraining has two stages (1) text is corrupted with an arbitrary noising function, and (2) a sequence-to-sequence model is learned to reconstruct the original text. BART uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and many other more recent pretraining schemes (see Figure 1).

本文提出 BART，预训练结合双向与自回归 Transformer 的模型。BART 是用序列到序列模型构建的去噪自编码器，适用于非常广泛的端任务。预训练分两阶段：(1) 用任意加噪函数腐蚀文本；(2) 学习序列到序列模型重建原文。BART 使用标准基于 Transformer 的神经机器翻译结构；尽管简单，却可看作对 BERT（双向编码器）、GPT（从左到右解码器）以及许多更新预训练方案的概括（见图 1）。

A key advantage of this setup is the noising flexibility; arbitrary transformations can be applied to the original text, including changing its length. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where arbitrary length spans of text (including zero length) are replaced with a single mask token. This approach generalizes the original word masking and next sentence prediction objectives in BERT by forcing the model to reason more about overall sentence length and make longer range transformations to the input.

该设定的一个关键优势是加噪灵活：可对原文施加任意变换，包括改变长度。我们评估多种加噪方式，发现最佳表现来自随机打乱原句顺序，并配合新的填空方案——用单个掩码标记替换任意长度（含零长度）文本跨度。该方法概括了 BERT 中的词掩码与下一句预测目标，迫使模型更多推理整体句长并对输入做更长程变换。

(a) BERT: Random tokens are replaced with masks, and the document is encoded bidirectionally. Missing tokens are predicted independently, so BERT cannot easily be used for generation.

(a) BERT：随机 token 被替换为掩码，文档双向编码。缺失 token 独立预测，因此 BERT 不易用于生成。

BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa Liu et al. 2019 with comparable training resources on GLUE Wang et al. 2018 and SQuAD Rajpurkar et al. 2016, and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks. For example, it improves performance by 6 ROUGE over previous work on XSum Narayan et al. 2018.

BART 在微调文本生成时尤为有效，理解任务上同样出色。在可比训练资源下，它在 GLUE（Wang et al. 2018）与 SQuAD（Rajpurkar et al. 2016）上与 RoBERTa（Liu et al. 2019）相当，并在一系列抽象对话、问答与摘要任务上刷新结果。例如在 XSum（Narayan et al. 2018）上相对先前工作提升 6 ROUGE。

BART also opens up new ways of thinking about fine tuning. We present a new scheme for machine translation where a BART model is stacked above a few additional transformer layers. These layers are trained to essentially translate the foreign language to noised English, by propagation through BART, thereby using BART as a pre-trained target-side language model. This approach improves performance over a strong back-translation MT baseline by 1.1 BLEU on the WMT Romanian-English benchmark.

BART 也打开了微调的新思路。我们提出一种机器翻译新方案：在 BART 之上再叠若干 Transformer 层。这些层经 BART 反向传播，实质学习把外语译成加噪英语，从而把 BART 当作预训练好的目标侧语言模型。该方法在 WMT 罗马尼亚–英语基准上，相对强回译 MT 基线提升 1.1 BLEU。

To better understand these effects, we also report an ablation analysis that replicates other recently proposed training objectives. This study allows us to carefully control for a number of factors, including data and optimization parameters, which have been shown to be as important for overall performance as the selection of training objectives Liu et al. 2019. We find that BART exhibits the most consistently strong performance across the full range of tasks we consider.

为更好理解这些效应，我们还报告复现近期其他训练目标的消融分析。该研究使我们能仔细控制数据与优化参数等因素——它们已被证明与训练目标选择同样重要（Liu et al. 2019）。我们发现，在所考虑的全部任务上，BART 表现最为一贯地强。

Figure 2: Transformations for noising the input that we experiment with. These transformations can be composed.

图 2：我们实验的输入加噪变换。这些变换可以组合。

2 Model

2 模型

BART is a denoising autoencoder that maps a corrupted document to the original document it was derived from. It is implemented as a sequence-to-sequence model with a bidirectional encoder over corrupted text and a left-to-right autoregressive decoder. For pre-training, we optimize the negative log likelihood of the original document.

BART 是把腐蚀文档映射回其来源原文的去噪自编码器。实现为序列到序列模型：双向编码器处理腐蚀文本，从左到右自回归解码器生成。预训练时优化原文的负对数似然。

2.1 Architecture

2.1 结构

BART uses the standard sequence-to-sequence Transformer architecture from Vaswani et al. 2017, except, following GPT, that we modify ReLU activation functions to GeLUs (Hendrycks & Gimpel 2016) and initialise parameters from $\mathcal{N}(0,0.02)$. For our base model, we use 6 layers in the encoder and decoder, and for our large model we use 12 layers in each. The architecture is closely related to that used in BERT, with the following differences: (1) each layer of the decoder additionally performs cross-attention over the final hidden layer of the encoder (as in the transformer sequence-to-sequence model); and (2) BERT uses an additional feed-forward network before word-prediction, which BART does not. In total, BART contains roughly 10% more parameters than the equivalently sized BERT model.

BART 采用 Vaswani et al. 2017 的标准序列到序列 Transformer 结构，但沿用 GPT，把 ReLU 激活改为 GeLU（Hendrycks & Gimpel 2016），并从 $\mathcal{N}(0,0.02)$ 初始化参数。base 模型编码器与解码器各 6 层，large 各 12 层。结构与 BERT 密切相关，差异在于：(1) 解码器每层额外对编码器最终隐层做交叉注意力（如 Transformer 序列到序列模型）；(2) BERT 在词预测前还有额外前馈网络，BART 没有。总体参数量约比同等规模 BERT 多 10%。

2.2 Pre-training BART

2.2 预训练 BART

BART is trained by corrupting documents and then optimizing a reconstruction loss—the cross-entropy between the decoder’s output and the original document. Unlike existing denoising autoencoders, which are tailored to specific noising schemes, BART allows us to apply any type of document corruption. In the extreme case, where all information about the source is lost, BART is equivalent to a language model.

BART 通过腐蚀文档并优化重建损失训练——解码器输出与原文之间的交叉熵。与针对特定加噪方案定制的既有去噪自编码器不同，BART 允许施加任意类型的文档腐蚀。极端情况下，若源端信息全部丢失，BART 等价于语言模型。

We experiment with several previously proposed and novel transformations, but we believe there is a significant potential for development of other new alternatives. The transformations we used are summarized below, and examples are shown in Figure 2.

我们实验了若干先前提出与新的变换，并相信开发其他新方案仍有很大空间。所用变换概述如下，示例见图 2。

Token Masking

Following BERT Devlin et al. 2019, random tokens are sampled and replaced with [MASK] elements.

Token 掩码

沿用 BERT（Devlin et al. 2019），随机采样 token 并替换为 [MASK] 元素。

Token Deletion

Random tokens are deleted from the input. In contrast to token masking, the model must decide which positions are missing inputs.

Token 删除

从输入中删除随机 token。与 token 掩码不同，模型必须判断哪些位置缺失输入。

Text Infilling

A number of text spans are sampled, with span lengths drawn from a Poisson distribution ($\lambda=3$). Each span is replaced with a single [MASK] token. 0-length spans correspond to the insertion of [MASK] tokens. Text infilling is inspired by SpanBERT Joshi et al. 2019, but SpanBERT samples span lengths from a different (clamped geometric) distribution, and replaces each span with a sequence of [MASK] tokens of exactly the same length. Text infilling teaches the model to predict how many tokens are missing from a span.

文本填空

采样若干文本跨度，跨度长度服从泊松分布（$\lambda=3$）。每个跨度替换为单个 [MASK] token。长度为 0 的跨度对应插入 [MASK]。文本填空受 SpanBERT（Joshi et al. 2019）启发，但 SpanBERT 从不同的（截断几何）分布采样跨度长度，并用等长 [MASK] 序列替换每个跨度。文本填空教会模型预测跨度中缺失了多少 token。

Sentence Permutation

A document is divided into sentences based on full stops, and these sentences are shuffled in a random order.

句子置换

按句号把文档分成句子，再随机打乱这些句子的顺序。

Document Rotation

A token is chosen uniformly at random, and the document is rotated so that it begins with that token. This task trains the model to identify the start of the document.

文档旋转

均匀随机选一个 token，把文档旋转到以该 token 开头。该任务训练模型识别文档起点。

(a) To use BART for classification problems, the same input is fed into the encoder and decoder, and the representation from the final output is used.

(a) 用 BART 做分类时，同一输入送入编码器与解码器，使用最终输出的表示。

3 Fine-tuning BART

3 微调 BART

The representations produced by BART can be used in several ways for downstream applications.

BART 产生的表示可用多种方式服务下游应用。

3.1 Sequence Classification Tasks

3.1 序列分类任务

For sequence classification tasks, the same input is fed into the encoder and decoder, and the final hidden state of the final decoder token is fed into new multi-class linear classifier. This approach is related to the CLS token in BERT; however we add the additional token to the end so that representation for the token in the decoder can attend to decoder states from the complete input (Figure 3(a)).

对序列分类，同一输入送入编码器与解码器，把最终解码器 token 的最终隐状态送入新的多类线性分类器。做法与 BERT 的 CLS token 相关；但我们把额外 token 加在末尾，使该 token 在解码器中的表示能关注完整输入的解码器状态（图 3(a)）。

3.2 Token Classification Tasks

3.2 Token 分类任务

For token classification tasks, such as answer endpoint classification for SQuAD, we feed the complete document into the encoder and decoder, and use the top hidden state of the decoder as a representation for each word. This representation is used to classify the token.

对 token 分类（如 SQuAD 的答案端点分类），把完整文档送入编码器与解码器，用解码器顶层隐状态作为每个词的表示，再据此分类该 token。

3.3 Sequence Generation Tasks

3.3 序列生成任务

Because BART has an autoregressive decoder, it can be directly fine tuned for sequence generation tasks such as abstractive question answering and summarization. In both of these tasks, information is copied from the input but manipulated, which is closely related to the denoising pre-training objective. Here, the encoder input is the input sequence, and the decoder generates outputs autoregressively.

由于 BART 有自回归解码器，可直接微调到抽象问答与摘要等序列生成任务。这两类任务都从输入复制信息并加以改写，与去噪预训练目标密切相关。此处编码器输入为输入序列，解码器自回归生成输出。

3.4 Machine Translation

3.4 机器翻译

We also explore using BART to improve machine translation decoders for translating into English. Previous work Edunov et al. 2019 has shown that models can be improved by incorporating pre-trained encoders, but gains from using pre-trained language models in decoders have been limited. We show that it is possible to use the entire BART model (both encoder and decoder) as a single pretrained decoder for machine translation, by adding a new set of encoder parameters that are learned from bitext (see Figure 3(b)).

我们还探索用 BART 改进译入英语的机器翻译解码器。先前工作（Edunov et al. 2019）表明接入预训练编码器可提升模型，但解码器侧使用预训练语言模型的收益有限。我们表明：可通过增加一组从双语文本学习的新编码器参数，把整个 BART（编码器与解码器）当作机器翻译的单一预训练解码器使用（见图 3(b)）。

More precisely, we replace BART’s encoder embedding layer with a new randomly initialized encoder. The model is trained end-to-end, which trains the new encoder to map foreign words into an input that BART can de-noise to English. The new encoder can use a separate vocabulary from the original BART model.

更精确地说，我们用新的随机初始化编码器替换 BART 的编码器嵌入层。端到端训练使新编码器学习把外语词映射成 BART 能去噪成英语的输入。新编码器可使用与原 BART 不同的词表。

We train the source encoder in two steps, in both cases backpropagating the cross-entropy loss from the output of the BART model. In the first step, we freeze most of BART parameters and only update the randomly initialized source encoder, the BART positional embeddings, and the self-attention input projection matrix of BART’s encoder first layer. In the second step, we train all model parameters for a small number of iterations.

源编码器分两步训练，两步都从 BART 模型输出回传交叉熵损失。第一步冻结 BART 大部分参数，只更新随机初始化的源编码器、BART 位置嵌入，以及 BART 编码器第一层自注意力的输入投影矩阵。第二步对全部模型参数再训少量迭代。

4 Comparing Pre-training Objectives

4 比较预训练目标

BART supports a much wider range of noising schemes during pre-training than previous work. We compare a range of options using base-size models (6 encoder and 6 decoder layers, with a hidden size of 768), evaluated on a representative subset of the tasks we will consider for the full large scale experiments in §5.

预训练时 BART 支持远比先前工作更广的加噪方案。我们用 base 规模模型（编码器与解码器各 6 层，隐层 768）比较一系列选项，并在第 5 节大规模实验将考虑的代表性任务子集上评估。

4.1 Comparison Objectives

4.1 比较目标

While many pre-training objectives have been proposed, fair comparisons between these have been difficult to perform, at least in part due to differences in training data, training resources, architectural differences between models, and fine-tuning procedures. We re-implement strong pre-training approaches recently proposed for discriminative and generation tasks. We aim, as much as possible, to control for differences unrelated to the pre-training objective. However, we do make minor changes to the learning rate and usage of layer normalisation in order to improve performance (tuning these separately for each objective). For reference, we compare our implementations with published numbers from BERT, which was also trained for 1M steps on a combination of books and Wikipedia data. We compare the following approaches:

尽管已有许多预训练目标被提出，公平比较仍困难，部分原因在于训练数据、训练资源、模型结构差异与微调流程不同。我们重新实现近期为判别与生成任务提出的强预训练方法，尽量控制与预训练目标无关的差异。不过为提升表现，仍对学习率与层归一化用法做小幅调整（按目标分别调）。作为参考，我们把实现与 BERT 的已发表数字比较——BERT 也在书籍与维基百科组合上训了 1M 步。比较以下方法：

Model	SQuAD 1.1	MNLI	ELI5	XSum	ConvAI2	CNN/DM
	F1	Acc	PPL	PPL	PPL	PPL
BERT Base Devlin et al. 2019	88.5	84.3	-	-	-	-
Masked Language Model	90.0	83.5	24.77	7.87	12.59	7.06
Masked Seq2seq	87.0	82.1	23.40	6.80	11.43	6.19
Language Model	76.7	80.1	21.40	7.00	11.51	6.56
Permuted Language Model	89.1	83.7	24.03	7.69	12.23	6.96
Multitask Masked Language Model	89.2	82.4	23.73	7.50	12.39	6.74
BART Base
w/ Token Masking	90.4	84.1	25.05	7.08	11.73	6.10
w/ Token Deletion	90.4	84.1	24.61	6.90	11.46	5.87
w/ Text Infilling	90.8	84.0	24.26	6.61	11.05	5.83
w/ Document Rotation	77.2	75.3	53.69	17.14	19.87	10.59
w/ Sentence Shuffling	85.4	81.5	41.87	10.93	16.67	7.89
w/ Text Infilling + Sentence Shuffling	90.8	83.8	24.17	6.62	11.12	5.41

Table 1: Comparison of pre-training objectives. All models are of comparable size and are trained for 1M steps on a combination of books and Wikipedia data. Entries in the bottom two blocks are trained on identical data using the same code-base, and fine-tuned with the same procedures. Entries in the second block are inspired by pre-training objectives proposed in previous work, but have been simplified to focus on evaluation objectives (see §4.1). Performance varies considerably across tasks, but the BART models with text infilling demonstrate the most consistently strong performance.

Model	SQuAD 1.1	MNLI	ELI5	XSum	ConvAI2	CNN/DM
	F1	Acc	PPL	PPL	PPL	PPL
BERT Base Devlin et al. 2019	88.5	84.3	-	-	-	-
Masked Language Model	90.0	83.5	24.77	7.87	12.59	7.06
Masked Seq2seq	87.0	82.1	23.40	6.80	11.43	6.19
Language Model	76.7	80.1	21.40	7.00	11.51	6.56
Permuted Language Model	89.1	83.7	24.03	7.69	12.23	6.96
Multitask Masked Language Model	89.2	82.4	23.73	7.50	12.39	6.74
BART Base
w/ Token Masking	90.4	84.1	25.05	7.08	11.73	6.10
w/ Token Deletion	90.4	84.1	24.61	6.90	11.46	5.87
w/ Text Infilling	90.8	84.0	24.26	6.61	11.05	5.83
w/ Document Rotation	77.2	75.3	53.69	17.14	19.87	10.59
w/ Sentence Shuffling	85.4	81.5	41.87	10.93	16.67	7.89
w/ Text Infilling + Sentence Shuffling	90.8	83.8	24.17	6.62	11.12	5.41

表 1：预训练目标比较。所有模型规模可比，均在书籍与维基百科组合上训 1M 步。底部两块条目用相同数据与同一代码库训练，并以相同流程微调。第二块受先前工作目标启发，但已简化以聚焦评估目标（见第 4.1 节）。各任务表现差异大，但带文本填空的 BART 模型表现最为一贯地强。

Language Model

Similarly to GPT (Radford et al. 2018), we train a left-to-right Transformer language model. This model is equivalent to the BART decoder, without cross-attention.

语言模型

类似 GPT（Radford et al. 2018），我们训练从左到右的 Transformer 语言模型。该模型等价于去掉交叉注意力的 BART 解码器。

Permuted Language Model

Based on XLNet (Yang et al. 2019), we sample 1/6 of the tokens, and generate them in a random order autoregressively. For consistency with other models, we do not implement the relative positional embeddings or attention across segments from XLNet.

置换语言模型

基于 XLNet（Yang et al. 2019），采样 1/6 token，按随机顺序自回归生成。为与其他模型一致，我们不实现 XLNet 的相对位置嵌入或跨段注意力。

Masked Language Model

Following BERT (Devlin et al. 2019), we replace 15% of tokens with [MASK] symbols, and train the model to independently predict the original tokens.

掩码语言模型

沿用 BERT（Devlin et al. 2019），用 [MASK] 符号替换 15% token，训练模型独立预测原 token。

Multitask Masked Language Model

As in UniLM (Dong et al. 2019), we train a Masked Language Model with additional self-attention masks. Self attention masks are chosen randomly in with the follow proportions: 1/6 left-to-right, 1/6 right-to-left, 1/3 unmasked, and 1/3 with the first 50% of tokens unmasked and a left-to-right mask for the remainder.

多任务掩码语言模型

如 UniLM（Dong et al. 2019），在掩码语言模型上再加自注意力掩码。自注意力掩码随机选取，比例为：1/6 从左到右、1/6 从右到左、1/3 无掩码、1/3 前 50% token 无掩码且其余用从左到右掩码。

Masked Seq-to-Seq

Inspired by MASS (Song et al. 2019), we mask a span containing 50% of tokens, and train a sequence to sequence model to predict the masked tokens.

掩码序列到序列

受 MASS（Song et al. 2019）启发，掩掉含 50% token 的跨度，训练序列到序列模型预测被掩 token。

For the Permuted LM, Masked LM and Multitask Masked LM, we use two-stream attention (Yang et al. 2019) to efficiently compute likelihoods of the output part of the sequence (using a diagonal self-attention mask on the output to predict words left-to-right).

对置换 LM、掩码 LM 与多任务掩码 LM，我们使用双流注意力（Yang et al. 2019）高效计算序列输出部分的似然（输出侧用对角自注意力掩码从左到右预测词）。

We experiment with (1) treating the task as a standard sequence-to-sequence problem, where the source input to the encoder and the target is the decoder output, or (2) adding the source as prefix to the target in the decoder, with a loss only on the target part of the sequence. We find the former works better for BART models, and the latter for other models.

我们尝试：(1) 把任务当作标准序列到序列，源端进编码器、目标为解码器输出；或 (2) 把源端作为前缀接到解码器目标前，只在目标部分计损失。发现前者对 BART 更好，后者对其他模型更好。

To most directly compare our models on their ability to model their fine-tuning objective (the log likelihood of the human text), we report perplexity in Table 1.

为最直接比较各模型对微调目标（人类文本的对数似然）的建模能力，表 1 报告困惑度。

4.2 Tasks

4.2 任务

SQuAD

Rajpurkar et al. 2016a an extractive question answering task on Wikipedia paragraphs. Answers are text spans extracted from a given document context. Similar to BERT Devlin et al. 2019, we use concatenated question and context as input to the encoder of BART, and additionally pass them to the decoder. The model includes classifiers to predict the start and end indices of each token.

SQuAD

Rajpurkar et al. 2016a 是维基百科段落上的抽取式问答。答案是从给定文档上下文抽取的文本跨度。类似 BERT（Devlin et al. 2019），我们将拼接的问题与上下文作为 BART 编码器输入，并额外送入解码器。模型含分类器预测每个 token 的起止索引。

MNLI

(Williams et al. 2017), a bitext classification task to predict whether one sentence entails another. The fine-tuned model concatenates the two sentences with appended an EOS token, and passes them to both the BART encoder and decoder. In contrast to BERT, the representation of the EOS token is used to classify the sentences relations.

MNLI

（Williams et al. 2017）是判断一句是否蕴含另一句的双句分类任务。微调模型把两句拼接并追加 EOS token，送入 BART 编码器与解码器。与 BERT 不同，用 EOS token 的表示分类句间关系。

ELI5

(Fan et al. 2019), a long-form abstractive question answering dataset. Models generate answers conditioned on the concatenation of a question and supporting documents.

ELI5

（Fan et al. 2019）是长篇抽象问答数据集。模型在问题与支持文档的拼接条件下生成答案。

XSum

(Narayan et al. 2018), a news summarization dataset with highly abstractive summaries.

XSum

（Narayan et al. 2018）是摘要高度抽象的新闻摘要数据集。

ConvAI2

(Dinan et al. 2019), a dialogue response generation task, conditioned on context and a persona.

ConvAI2

（Dinan et al. 2019）是以上下文与人设为条件的对话响应生成任务。

CNN/DM

(Hermann et al. 2015), a news summarization dataset. Summaries here are typically closely related to source sentences.

CNN/DM

（Hermann et al. 2015）是新闻摘要数据集。此处摘要通常与源句密切相关。

4.3 Results

4.3 结果

Results are shown in Table 1. Several trends are clear:

结果见表 1。若干趋势清晰：

Performance of pre-training methods varies significantly across tasks

The effectiveness of pre-training methods is highly dependent on the task. For example, a simple language model achieves the best ELI5 performance, but the worst SQUAD results.

预训练方法表现随任务显著变化

预训练方法的有效性高度依赖任务。例如，简单语言模型在 ELI5 上最好，但在 SQuAD 上最差。

Token masking is crucial

Pre-training objectives based on rotating documents or permuting sentences perform poorly in isolation. The successful methods either use token deletion or masking, or self-attention masks. Deletion appears to outperform masking on generation tasks.

Token 掩码至关重要

仅基于旋转文档或置换句子的预训练目标单独使用时表现很差。成功的方法要么使用 token 删除或掩码，要么使用自注意力掩码。删除在生成任务上似乎优于掩码。

Left-to-right pre-training improves generation

The Masked Language Model and the Permuted Language Model perform less well than others on generation, and are the only models we consider that do not include left-to-right auto-regressive language modelling during pre-training.

从左到右预训练改善生成

掩码语言模型与置换语言模型在生成上弱于其他，它们也是我们考虑的唯一在预训练中不含从左到右自回归语言建模的模型。

Bidirectional encoders are crucial for SQuAD

As noted in previous work Devlin et al. 2019, just left-to-right decoder performs poorly on SQuAD, because future context is crucial in classification decisions. However, BART achieves similar performance with only half the number of bidirectional layers.

双向编码器对 SQuAD 至关重要

如先前工作（Devlin et al. 2019）所指出，仅从左到右解码器在 SQuAD 上表现差，因为分类决策需要未来上下文。不过 BART 仅用一半数量的双向层即达相近表现。

The pre-training objective is not the only important factor

Our Permuted Language Model performs less well than XLNet Yang et al. 2019. Some of this difference is likely due to not including other architectural improvements, such as relative-position embeddings or segment-level recurrence.

预训练目标不是唯一重要因素

我们的置换语言模型弱于 XLNet（Yang et al. 2019）。部分差异可能来自未纳入相对位置嵌入或段级循环等其他结构改进。

Pure language models perform best on ELI5

The ELI5 dataset is an outlier, with much higher perplexities than other tasks, and is the only generation task where other models outperform BART. A pure language model performs best, suggesting that BART is less effective when the output is only loosely constrained by the input.

纯语言模型在 ELI5 上最好

ELI5 是离群点：困惑度远高于其他任务，也是唯一其他模型超过 BART 的生成任务。纯语言模型最好，说明当输出仅被输入松约束时，BART 效果较弱。

BART achieves the most consistently strong performance.

With the exception of ELI5, BART models using text-infilling perform well on all tasks.

BART 取得最一贯的强表现

除 ELI5 外，使用文本填空的 BART 模型在所有任务上都表现良好。

	SQuAD 1.1	SQuAD 2.0	MNLI	SST	QQP	QNLI	STS-B	RTE	MRPC	CoLA
	EM/F1	EM/F1	m/mm	Acc	Acc	Acc	Acc	Acc	Acc	Mcc
BERT	84.1/90.9	79.0/81.8	86.6/-	93.2	91.3	92.3	90.0	70.4	88.0	60.6
UniLM	-/-	80.5/83.4	87.0/85.9	94.5	-	92.7	-	70.9	-	61.1
XLNet	89.0/94.5	86.1/88.8	89.8/-	95.6	91.8	93.9	91.8	83.8	89.2	63.6
RoBERTa	88.9/94.6	86.5/89.4	90.2/90.2	96.4	92.2	94.7	92.4	86.6	90.9	68.0
BART	88.8/94.6	86.1/89.2	89.9/90.1	96.6	92.5	94.9	91.2	87.0	90.4	62.8

Table 2: Results for large models on SQuAD and GLUE tasks. BART performs comparably to RoBERTa and XLNet, suggesting that BART’s uni-directional decoder layers do not reduce performance on discriminative tasks.

	SQuAD 1.1	SQuAD 2.0	MNLI	SST	QQP	QNLI	STS-B	RTE	MRPC	CoLA
	EM/F1	EM/F1	m/mm	Acc	Acc	Acc	Acc	Acc	Acc	Mcc
BERT	84.1/90.9	79.0/81.8	86.6/-	93.2	91.3	92.3	90.0	70.4	88.0	60.6
UniLM	-/-	80.5/83.4	87.0/85.9	94.5	-	92.7	-	70.9	-	61.1
XLNet	89.0/94.5	86.1/88.8	89.8/-	95.6	91.8	93.9	91.8	83.8	89.2	63.6
RoBERTa	88.9/94.6	86.5/89.4	90.2/90.2	96.4	92.2	94.7	92.4	86.6	90.9	68.0
BART	88.8/94.6	86.1/89.2	89.9/90.1	96.6	92.5	94.9	91.2	87.0	90.4	62.8

表 2：大模型在 SQuAD 与 GLUE 任务上的结果。BART 与 RoBERTa、XLNet 相当，表明 BART 的单向解码器层并未削弱判别任务表现。

	CNN/DailyMail	XSum
	R1	R2	RL	R1	R2	RL
Lead-3	40.42	17.62	36.67	16.30	1.60	11.95
PTGEN See et al. 2017	36.44	15.66	33.42	29.70	9.21	23.24
PTGEN+COV See et al. 2017	39.53	17.28	36.38	28.10	8.02	21.72
UniLM	43.33	20.21	40.51	-	-	-
BERTSUMABS (Liu & Lapata 2019)	41.72	19.39	38.76	38.76	16.33	31.15
BERTSUMEXTABS (Liu & Lapata 2019)	42.13	19.60	39.18	38.81	16.50	31.27
BART	44.16	21.28	40.90	45.14	22.27	37.25

Table 3: Results on two standard summarization datasets. BART outperforms previous work on summarization on two tasks and all metrics, with gains of roughly 6 points on the more abstractive dataset.

	CNN/DailyMail	XSum
	R1	R2	RL	R1	R2	RL
Lead-3	40.42	17.62	36.67	16.30	1.60	11.95
PTGEN See et al. 2017	36.44	15.66	33.42	29.70	9.21	23.24
PTGEN+COV See et al. 2017	39.53	17.28	36.38	28.10	8.02	21.72
UniLM	43.33	20.21	40.51	-	-	-
BERTSUMABS (Liu & Lapata 2019)	41.72	19.39	38.76	38.76	16.33	31.15
BERTSUMEXTABS (Liu & Lapata 2019)	42.13	19.60	39.18	38.81	16.50	31.27
BART	44.16	21.28	40.90	45.14	22.27	37.25

表 3：两个标准摘要数据集上的结果。BART 在两项任务、全部指标上超过先前工作，在更抽象的数据集上约提升 6 点。

5 Large-scale Pre-training Experiments

5 大规模预训练实验

Recent work has shown that downstream performance can dramatically improve when pre-training is scaled to large batch sizes (Yang et al. 2019; Liu et al. 2019) and corpora. To test how well BART performs in this regime, and to create a useful model for downstream tasks, we trained BART using the same scale as the RoBERTa model.

近期工作表明，当预训练放大到大批次（Yang et al. 2019；Liu et al. 2019）与语料时，下游表现可显著提升。为测试 BART 在该设定下的表现并得到可用的下游模型，我们按与 RoBERTa 相同的规模训练 BART。

5.1 Experimental Setup

5.1 实验设定

We pre-train a large model with 12 layers in each of the encoder and decoder, and a hidden size of 1024. Following RoBERTa Liu et al. 2019, we use a batch size of 8000, and train the model for 500000 steps. Documents are tokenized with the same byte-pair encoding as GPT-2 Radford et al. 2019. Based on the results in Section §4, we use a combination of text infilling and sentence permutation. We mask 30% of tokens in each document, and permute all sentences. Although sentence permutation only shows significant additive gains on the CNN/DM summarization dataset, we hypothesised that larger pre-trained models may be better able to learn from this task. To help the model better fit the data, we disabled dropout for the final 10% of training steps. We use the same pre-training data as Liu et al. 2019, consisting of 160Gb of news, books, stories, and web text.

我们预训练 large 模型：编码器与解码器各 12 层，隐层 1024。沿用 RoBERTa（Liu et al. 2019），batch 大小 8000，训 500000 步。文档用与 GPT-2（Radford et al. 2019）相同的字节对编码分词。基于第 4 节结果，采用文本填空与句子置换的组合：每篇文档掩掉 30% token，并置换全部句子。尽管句子置换仅在 CNN/DM 摘要上有明显叠加收益，我们假设更大预训练模型或许更能从该任务中学习。为更好拟合数据，训练最后 10% 步关闭 dropout。预训练数据与 Liu et al. 2019 相同，约 160GB 新闻、书籍、故事与网页文本。

5.2 Discriminative Tasks

5.2 判别任务

Table 2 compares the performance of BART with several recent approaches on the well-studied SQuAD and GLUE tasks (Warstadt et al. 2018; Socher et al. 2013; Dolan & Brockett 2005; Agirre et al. 2007; Williams et al. 2018; Dagan et al. 2006; Levesque et al. 2011).

表 2 在充分研究的 SQuAD 与 GLUE 任务上比较 BART 与若干近期方法（Warstadt et al. 2018；Socher et al. 2013；Dolan & Brockett 2005；Agirre et al. 2007；Williams et al. 2018；Dagan et al. 2006；Levesque et al. 2011）。

The most directly comparable baseline is RoBERTa, which was pre-trained with the same resources, but a different objective. Overall, BART performs similarly, with only small differences between the models on most tasks. suggesting that BART’s improvements on generation tasks do not come at the expense of classification performance.

最直接可比的基线是 RoBERTa：预训练资源相同，但目标不同。总体而言 BART 表现相近，多数任务上差距很小，说明 BART 在生成任务上的提升并非以牺牲分类表现为代价。

5.3 Generation Tasks

5.3 生成任务

We also experiment with several text generation tasks. BART is fine-tuned as a standard sequence-to-sequence model from the input to the output text. During fine-tuning we use a label smoothed cross entropy loss (Pereyra et al. 2017), with the smoothing parameter set to 0.1. During generation, we set beam size as 5, remove duplicated trigrams in beam search, and tuned the model with min-len, max-len, length penalty on the validation set Fan et al. 2017.

我们还在若干文本生成任务上实验。BART 作为从输入到输出文本的标准序列到序列模型微调。微调使用标签平滑交叉熵（Pereyra et al. 2017），平滑参数 0.1。生成时 beam 大小 5，在 beam 搜索中去除重复三元组，并在验证集上调节 min-len、max-len 与长度惩罚（Fan et al. 2017）。

Summarization

To provide a comparison with the state-of-the-art in summarization, we present results on two summarization datasets, CNN/DailyMail and XSum, which have distinct properties.

摘要

为与摘要领域顶尖结果比较，我们在 CNN/DailyMail 与 XSum 两个性质不同的摘要数据集上报告结果。

Summaries in the CNN/DailyMail tend to resemble source sentences. Extractive models do well here, and even the baseline of the first-three source sentences is highly competitive. Nevertheless, BART outperforms all existing work.

CNN/DailyMail 的摘要往往类似源句。抽取式模型在此表现好，甚至前三句源句基线也很强。尽管如此，BART 仍超过所有既有工作。

In contrast, XSum is highly abstractive, and extractive models perform poorly. BART outperforms the best previous work, which leverages BERT, by roughly 6.0 points on all ROUGE metrics—representing a significant advance in performance on this problem. Qualitatively, sample quality is high (see §6).

相较之下，XSum 高度抽象，抽取式模型表现差。BART 在全部 ROUGE 指标上相对利用 BERT 的最佳先前工作约高 6.0 点——代表该问题上的显著进展。定性上样本质量高（见第 6 节）。

	ConvAI2
	Valid F1	Valid PPL
Seq2Seq + Attention	16.02	35.07
Best System	19.09	17.51
BART	20.72	11.85

Table 4: BART outperforms previous work on conversational response generation. Perplexities are renormalized based on official tokenizer for ConvAI2.

	ConvAI2
	Valid F1	Valid PPL
Seq2Seq + Attention	16.02	35.07
Best System	19.09	17.51
BART	20.72	11.85

表 4：BART 在对话响应生成上超过先前工作。困惑度按 ConvAI2 官方分词器重新归一化。

	ELI5
	R1	R2	RL
Best Extractive	23.5	3.1	17.5
Language Model	27.8	4.7	23.1
Seq2Seq	28.3	5.1	22.8
Seq2Seq Multitask	28.9	5.4	23.1
BART	30.6	6.2	24.3

Table 5: BART achieves state-of-the-art results on the challenging ELI5 abstractive question answering dataset. Comparison models are from Fan et al. 2019.

	ELI5
	R1	R2	RL
Best Extractive	23.5	3.1	17.5
Language Model	27.8	4.7	23.1
Seq2Seq	28.3	5.1	22.8
Seq2Seq Multitask	28.9	5.4	23.1
BART	30.6	6.2	24.3

表 5：BART 在具有挑战性的 ELI5 抽象问答数据集上取得顶尖结果。对照模型来自 Fan et al. 2019。

Dialogue

We evaluate dialogue response generation on ConvAI2 (Dinan et al. 2019), in which agents must generate responses conditioned on both the previous context and a textually-specified persona. BART outperforms previous work on two automated metrics.

对话

我们在 ConvAI2（Dinan et al. 2019）上评估对话响应生成：智能体须在先前上下文与文本给定人设条件下生成响应。BART 在两项自动指标上超过先前工作。

Abstractive QA

We use the recently proposed ELI5 dataset to test the model’s ability to generate long free-form answers. We find BART outperforms the best previous work by 1.2 ROUGE-L, but the dataset remains a challenging, because answers are only weakly specified by the question.

抽象问答

我们用近期提出的 ELI5 数据集测试模型生成长篇自由形式答案的能力。BART 相对最佳先前工作高 1.2 ROUGE-L，但该集仍具挑战，因为答案仅被问题弱指定。

5.4 Translation

5.4 翻译

	RO-EN
Baseline	36.80
Fixed BART	36.29
Tuned BART	37.96

Table 6: The performance (BLEU) of baseline and BART on WMT’16 RO-EN augmented with back-translation data. BART improves over a strong back-translation (BT) baseline by using monolingual English pre-training.

	RO-EN
Baseline	36.80
Fixed BART	36.29
Tuned BART	37.96

表 6：在加入回译数据的 WMT’16 RO-EN 上，基线与 BART 的表现（BLEU）。BART 通过单语英语预训练，相对强回译（BT）基线有提升。

Source Document (abbreviated)	BART Summary
The researchers examined three types of coral in reefs off the coast of Fiji … The researchers found when fish were plentiful, they would eat algae and seaweed off the corals, which appeared to leave them more resistant to the bacterium Vibrio coralliilyticus, a bacterium associated with bleaching. The researchers suggested the algae, like warming temperatures, might render the corals’ chemical defenses less effective, and the fish were protecting the coral by removing the algae.	Fisheries off the coast of Fiji are protecting coral reefs from the effects of global warming, according to a study in the journal Science.
Sacoolas, who has immunity as a diplomat’s wife, was involved in a traffic collision … Prime Minister Johnson was questioned about the case while speaking to the press at a hospital in Watford. He said, “I hope that Anne Sacoolas will come back … if we can’t resolve it then of course I will be raising it myself personally with the White House.”	Boris Johnson has said he will raise the issue of US diplomat Anne Sacoolas’ diplomatic immunity with the White House.
According to Syrian state media, government forces began deploying into previously SDF controlled territory yesterday. … On October 6, US President Donald Trump and Turkish President Recep Tayyip Erdoğan spoke on the phone. Then both nations issued statements speaking of an imminent incursion into northeast Syria … . On Wednesday, Turkey began a military offensive with airstrikes followed by a ground invasion.	Syrian government forces have entered territory held by the US-backed Syrian Democratic Forces (SDF) in response to Turkey’s incursion into the region.
This is the first time anyone has been recorded to run a full marathon of 42.195 kilometers (approximately 26 miles) under this pursued landmark time. It was not, however, an officially sanctioned world record, as it was not an ”open race” of the IAAF. His time was 1 hour 59 minutes 40.2 seconds. Kipchoge ran in Vienna, Austria. It was an event specifically designed to help Kipchoge break the two hour barrier.	Kenyan runner Eliud Kipchoge has run a marathon in less than two hours.
PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.	Power has been turned off to millions of customers in California as part of a power shutoff plan.

Table 7: Example summaries from the XSum-tuned BART model on WikiNews articles. For clarity, only relevant excerpts of the source are shown. Summaries combine information from across the article and prior knowledge.

Source Document (abbreviated)	BART Summary
The researchers examined three types of coral in reefs off the coast of Fiji … The researchers found when fish were plentiful, they would eat algae and seaweed off the corals, which appeared to leave them more resistant to the bacterium Vibrio coralliilyticus, a bacterium associated with bleaching. The researchers suggested the algae, like warming temperatures, might render the corals’ chemical defenses less effective, and the fish were protecting the coral by removing the algae.	Fisheries off the coast of Fiji are protecting coral reefs from the effects of global warming, according to a study in the journal Science.
Sacoolas, who has immunity as a diplomat’s wife, was involved in a traffic collision … Prime Minister Johnson was questioned about the case while speaking to the press at a hospital in Watford. He said, “I hope that Anne Sacoolas will come back … if we can’t resolve it then of course I will be raising it myself personally with the White House.”	Boris Johnson has said he will raise the issue of US diplomat Anne Sacoolas’ diplomatic immunity with the White House.
According to Syrian state media, government forces began deploying into previously SDF controlled territory yesterday. … On October 6, US President Donald Trump and Turkish President Recep Tayyip Erdoğan spoke on the phone. Then both nations issued statements speaking of an imminent incursion into northeast Syria … . On Wednesday, Turkey began a military offensive with airstrikes followed by a ground invasion.	Syrian government forces have entered territory held by the US-backed Syrian Democratic Forces (SDF) in response to Turkey’s incursion into the region.
This is the first time anyone has been recorded to run a full marathon of 42.195 kilometers (approximately 26 miles) under this pursued landmark time. It was not, however, an officially sanctioned world record, as it was not an ”open race” of the IAAF. His time was 1 hour 59 minutes 40.2 seconds. Kipchoge ran in Vienna, Austria. It was an event specifically designed to help Kipchoge break the two hour barrier.	Kenyan runner Eliud Kipchoge has run a marathon in less than two hours.
PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.	Power has been turned off to millions of customers in California as part of a power shutoff plan.

表 7：在 WikiNews 文章上、经 XSum 微调的 BART 摘要示例。为清晰起见，仅展示源文相关摘录。摘要综合全文信息与先验知识。

We also evaluated performance on WMT16 Romanian-English, augmented with back-translation data from Sennrich et al. 2016. We use a 6-layer transformer source encoder to map Romanian into a representation that BART is able to de-noise into English, following the approach introduced in §3.4. Experiment results are presented in Table 6. We compare our results against a baseline Transformer architecture Vaswani et al. 2017 with Transformer-large settings (the baseline row). We show the performance of both steps of our model in the fixed BART and tuned BART rows. For each row we experiment on the original WMT16 Romanian-English augmented with back-translation data. We use a beam width of 5 and a length penalty of $\alpha=1$. Preliminary results suggested that our approach was less effective without back-translation data, and prone to overfitting—future work should explore additional regularization techniques.

我们还在加入 Sennrich et al. 2016 回译数据的 WMT16 罗马尼亚–英语上评估。按第 3.4 节方法，用 6 层 Transformer 源编码器把罗马尼亚语映射为 BART 能去噪成英语的表示。结果见表 6。对照为 Transformer-large 设定的基线 Transformer（Vaswani et al. 2017）（基线行）。我们展示两步训练的固定 BART 与调谐 BART 行；每行实验均在原 WMT16 罗马尼亚–英语加回译数据上进行。beam 宽 5，长度惩罚 $\alpha=1$。初步结果表明，无回译数据时该方法效果较差且易过拟合——后续工作应探索额外正则。

6 Qualitative Analysis

6 定性分析

BART shows large improvements on summarization metrics, of up to 6 points over the prior state-of-the-art. To understand BART’s performance beyond automated metrics, we analyse its generations qualitatively.

BART 在摘要指标上提升很大，相对先前顶尖最多约 6 点。为理解超越自动指标的表现，我们对其生成做定性分析。

Table 7 shows example summaries generated by BART. Examples are taken from WikiNews articles published after the creation of the pre-training corpus, to eliminate the possibility of the events described being present in the model’s training data. Following Narayan et al. 2018, we remove the first sentence of the article prior to summarizing it, so there is no easy extractive summary of the document.

表 7 给出 BART 生成的摘要示例。样本取自预训练语料创建之后发表的 WikiNews 文章，以排除所述事件已出现在训练数据中的可能。沿用 Narayan et al. 2018，摘要前先去掉文章首句，使文档没有轻易的抽取式摘要。

Unsurprisingly, model output is fluent and grammatical English. However, model output is also highly abstractive, with few phrases copied from the input. The output is also generally factually accurate, and integrates supporting evidence from across the input document with background knowledge (for example, correctly completing names, or inferring that PG&E operates in California). In the first example, inferring that fish are protecting reefs from global warming requires non-trivial inference from the text. However, the claim that the work was published in Science is not supported by the source.

毫不意外，模型输出是流畅、合语法的英语。同时高度抽象，从输入复制的短语很少。输出大体事实准确，并把输入文档各处证据与背景知识整合（例如正确补全姓名，或推断 PG&E 在加州运营）。第一例中，推断鱼类在保护珊瑚礁免受全球变暖影响，需要对文本做非平凡推理。但「发表于 Science」的说法并无源文支持。

These samples demonstrate that the BART pretraining has learned a strong combination of natural language understanding and generation.

这些样本表明，BART 预训练学到了自然语言理解与生成的强组合。

7 Related Work

7 相关工作

Early methods for pretraining were based on language models. GPT Radford et al. 2018 only models leftward context, which is problematic for some tasks. ELMo Peters et al. 2018 concatenates left-only and right-only representations, but does not pre-train interactions between these features. Radford et al. 2019 demonstrated that very large language models can act as unsupervised multitask models.

早期预训练方法基于语言模型。GPT（Radford et al. 2018）只建模左侧上下文，对某些任务不利。ELMo（Peters et al. 2018）拼接仅左与仅右表示，但不预训练这些特征之间的交互。Radford et al. 2019 表明超大语言模型可充当无监督多任务模型。

BERT (Devlin et al. 2019) introduced masked language modelling, which allows pre-training to learn interactions between left and right context words. Recent work has shown that very strong performance can be achieved by training for longer (Liu et al. 2019), by tying parameters across layers (Lan et al. 2019), and by masking spans instead of words (Joshi et al. 2019). Predictions are not made auto-regressively, reducing the effectiveness of BERT for generation tasks.

BERT（Devlin et al. 2019）引入掩码语言建模，使预训练能学习左右上下文词之间的交互。近期工作表明：训更久（Liu et al. 2019）、跨层共享参数（Lan et al. 2019）、掩跨度而非词（Joshi et al. 2019）都能取得很强表现。预测非自回归，降低了 BERT 对生成任务的有效性。

UniLM (Dong et al. 2019) fine-tunes BERT with an ensemble of masks, some of which allow only leftward context. Like BART, this allows UniLM to be used for both generative and discriminative tasks. A difference is that UniLM predictions are conditionally independent, whereas BART’s are autoregressive. BART reduces the mismatch between pre-training and generation tasks, because the decoder is always trained on uncorrupted context.

UniLM（Dong et al. 2019）用一组掩码微调 BERT，其中部分只允许左侧上下文。与 BART 类似，这使 UniLM 可用于生成与判别任务。差异在于 UniLM 预测条件独立，而 BART 自回归。BART 缩小了预训练与生成任务之间的错配，因为解码器始终在未腐蚀上下文上训练。

MASS (Song et al. 2019) is perhaps the most similar model to BART. An input sequence where a contiguous span of tokens is masked is mapped to a sequence consisting of the missing tokens. MASS is less effective for discriminative tasks, because disjoint sets of tokens are fed into the encoder and decoder.

MASS（Song et al. 2019）或许是与 BART 最相似的模型。输入序列中连续跨度被掩码，映射为仅含缺失 token 的序列。MASS 对判别任务效果较弱，因为送入编码器与解码器的是不相交的 token 集合。

XL-Net Yang et al. 2019 extends BERT by predicting masked tokens auto-regressively in a permuted order. This objective allows predictions to condition on both left and right context. In contrast, the BART decoder works left-to-right during pre-training, matching the setting during generation.

XL-Net（Yang et al. 2019）扩展 BERT，按置换顺序自回归预测掩码 token。该目标使预测可条件于左右上下文。相较之下，BART 解码器在预训练时从左到右工作，与生成时设定一致。

Several papers have explored using pre-trained representations to improve machine translation. The largest improvements have come from pre-training on both source and target languages (Song et al. 2019; Lample & Conneau 2019), but this requires pre-training on all languages of interest. Other work has shown that encoders can be improved using pre-trained representations Edunov et al. 2019, but gains in decoders are more limited. We show how BART can be used to improve machine translation decoders.

若干工作探索用预训练表示改进机器翻译。最大提升来自在源语与目标语上都预训练（Song et al. 2019；Lample & Conneau 2019），但这需要在所有相关语言上预训练。其他工作表明编码器可用预训练表示改进（Edunov et al. 2019），但解码器侧收益更有限。我们展示如何用 BART 改进机器翻译解码器。

8 Conclusions

8 结论

We introduced BART, a pre-training approach that learns to map corrupted documents to the original. BART achieves similar performance to RoBERTa on discriminative tasks, while achieving new state-of-the-art results on a number of text generation tasks. Future work should explore new methods for corrupting documents for pre-training, perhaps tailoring them to specific end tasks.

我们提出 BART：学习把腐蚀文档映射回原文的预训练方法。BART 在判别任务上与 RoBERTa 相近，同时在多项文本生成任务上刷新结果。后续工作应探索预训练时腐蚀文档的新方法，或许按具体端任务定制。

References

参考文献

Agirre et al. (2007) Eneko Agirre, Llu’is M‘arquez, and Richard Wicentowski (eds.). Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007) . Association for Computational Linguistics, Prague, Czech Republic, June 2007.

Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment , pp. 177–190. Springer, 2006.

Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423 . URL https://www.aclweb.org/anthology/N19-1423 .

Dinan et al. (2019) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098 , 2019.

Dolan & Brockett (2005) William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing , 2005.

Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197 , 2019.

Edunov et al. (2019) Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained language model representations for language generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.

Fan et al. (2017) Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. arXiv preprint arXiv:1711.05217 , 2017.

Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190 , 2019.

Hendrycks & Gimpel (2016) Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems , pp. 1693–1701, 2015.

Joshi et al. (2019) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529 , 2019.

Lample & Conneau (2019) Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 , 2019.

Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019.

Levesque et al. (2011) Hector J Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning , volume 46, pp.  47, 2011.

Liu & Lapata (2019) Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345 , 2019.

Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745 , 2018.

Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548 , 2017.

Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365 , 2018.

Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf , 2018.

Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog , 1(8), 2019.

Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016.

See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368 , 2017.

Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers , 2016.

Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP , pp. 1631–1642, 2013.

Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning , 2019.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pp. 5998–6008, 2017.

Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805.12471 , 2018.

Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426 , 2017.

Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT , 2018.

Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 , 2019.