Longformer: The Long-Document Transformer

Longformer：长文档 Transformer

Iz Beltagy          Matthew E. Peters 1 1 footnotemark: 1          Arman Cohan 1 1 footnotemark: 1 Thanks:   Equal contribution. Affiliation:  Allen Institute for Artificial Intelligence, Seattle, WA, USA Email:  {beltagy,matthewp,armanc}@allenai.org

Abstract

摘要

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.[^1]

基于 Transformer 的模型难以处理长序列，因为自注意力的计算量随序列长度二次增长。为突破这一限制，我们提出 Longformer：其注意力机制随序列长度线性扩展，因而能轻松处理数千乃至更长的 token。Longformer 的注意力可作为标准自注意力的即插即用替代，并把局部窗口注意力与面向任务的全局注意力结合起来。沿袭此前长序列 Transformer 工作，我们在字符级语言建模上评估 Longformer，并在 text8 与 enwik8 上取得当时最优结果。与多数先前工作不同，我们还对 Longformer 做预训练，并在多种下游任务上微调。预训练后的 Longformer 在长文档任务上稳定优于 RoBERTa，并在 WikiHop 与 TriviaQA 上刷新纪录。最后我们给出面向长文档生成式序列到序列任务的变体 Longformer-Encoder-Decoder（LED），并在 arXiv 摘要数据集上验证其有效性。[^1]

1 Introduction

1 引言

Transformers Vaswani et al. 2017 have achieved state-of-the-art results in a wide range of natural language tasks including generative language modeling Dai et al. 2019; Radford et al. 2019 and discriminative language understanding Devlin et al. 2019. This success is partly due to the self-attention component which enables the network to capture contextual information from the entire sequence. While powerful, the memory and computational requirements of self-attention grow quadratically with sequence length, making it infeasible (or very expensive) to process long sequences.

Transformer（Vaswani et al. 2017）已在广泛自然语言任务上取得领先结果，包括生成式语言建模（Dai et al. 2019；Radford et al. 2019）与判别式语言理解（Devlin et al. 2019）。成功部分来自自注意力：网络可从整段序列捕获上下文。但自注意力的内存与计算随序列长度二次增长，使处理长序列变得不可行（或代价极高）。

Figure 1: Runtime and memory of full self-attention and different implementations of Longformer’s self-attention; Longformer-loop is non-vectorized, Longformer-chunk is vectorized, and Longformer-cuda is a custom cuda kernel implementations. Longformer’s memory usage scales linearly with the sequence length, unlike the full self-attention mechanism that runs out of memory for long sequences on current GPUs. Different implementations vary in speed, with the vectorized Longformer-chunk being the fastest. More details are in section 3.2.

图 1：完整自注意力与 Longformer 自注意力多种实现的运行时间与内存；Longformer-loop 为非向量化实现，Longformer-chunk 为向量化实现，Longformer-cuda 为自定义 CUDA 核。与在当前 GPU 上对长序列会耗尽内存的完整自注意力不同，Longformer 的内存用量随序列长度线性增长。不同实现速度不同，向量化的 Longformer-chunk 最快。更多细节见第 3.2 节。

To address this limitation, we present Longformer, a modified Transformer architecture with a self-attention operation that scales linearly with the sequence length, making it versatile for processing long documents (Fig 1). This is an advantage for natural language tasks such as long document classification, question answering (QA), and coreference resolution, where existing approaches partition or shorten the long context into smaller sequences that fall within the typical 512 token limit of BERT-style pretrained models. Such partitioning could potentially result in loss of important cross-partition information, and to mitigate this problem, existing methods often rely on complex architectures to address such interactions. On the other hand, our proposed Longformer is able to build contextual representations of the entire context using multiple layers of attention, reducing the need for task-specific architectures.

为应对这一限制，我们提出 Longformer：一种改进的 Transformer 结构，其自注意力随序列长度线性扩展，因而适合处理长文档（图 1）。这对长文档分类、问答（QA）与共指消解等自然语言任务很有利——既有方法常把长上下文切分或缩短到 BERT 类预训练模型典型的 512 token 限制之内。这种切分可能丢掉跨分区的重要信息；为缓解损失，现有方法往往依赖复杂结构来建模跨块交互。相比之下，Longformer 可用多层注意力建立整段上下文的表征，从而减少对任务专用结构的需求。

Recent work has addressed the computational inefficiency of Transformers on long sequences (see Tab. 1). However, they primarily focus on autoregressive language modeling (LM), while the application of long document transformers to document-level NLP tasks in the transfer learning setting Dai and Le 2015; Peters et al. 2018; Howard and Ruder 2018; Devlin et al. 2019 has remained largely unexplored. We address this gap and show that Longformer’s attention mechanism can act as a drop-in replacement for the self-attention mechanism in pretrained Transformers, and leads to gains across a suite of document NLP tasks.

近期工作已着手缓解 Transformer 在长序列上的计算低效（见表 1）。但它们主要聚焦自回归语言建模（LM），而长文档 Transformer 在迁移学习设定下对文档级 NLP 任务的应用（Dai and Le 2015；Peters et al. 2018；Howard and Ruder 2018；Devlin et al. 2019）仍大体未被探索。我们填补这一空白，并表明 Longformer 的注意力可作为预训练 Transformer 中自注意力的即插即用替代，从而在一系列文档 NLP 任务上带来提升。

Longformer’s attention mechanism is a combination of a windowed local-context self-attention and an end task motivated global attention that encodes inductive bias about the task. Through ablations and controlled trials we show both attention types are essential – the local attention is primarily used to build contextual representations, while the global attention allows Longformer to build full sequence representations for prediction.

Longformer 的注意力由面向局部上下文的窗口自注意力，与面向终端任务、编码任务归纳偏置的全局注意力组合而成。通过消融与对照试验，我们表明两类注意力都必不可少——局部注意力主要用于构建上下文表征，全局注意力则让 Longformer 为预测建立整段序列表征。

We first evaluate Longformer on autoregressive character-level language modeling using a combination of windowed and a new dilated attention pattern, allowing the model to process sequences of up to 32K characters on modern GPUs. We achieve state-of-the-art results on text8 and enwik8 benchmark datasets, demonstrating the effectiveness of Longformer in long document modeling.

我们首先在自回归字符级语言建模上评估 Longformer，结合窗口注意力与新的扩张注意力模式，使模型在现代 GPU 上可处理长达 32K 字符的序列。我们在 text8 与 enwik8 基准上取得当时最优结果，表明 Longformer 对长文档建模有效。

Then, to evaluate Longformer’s ability to replace the full self-attention operation of existing pretrained models, we pretrain it with the masked language modeling (MLM) objective, continuing from the RoBERTa Liu et al. 2019 released checkpoint. After pretraining, we apply it to downstream language tasks through finetuning and demonstrate that Longformer consistently outperforms RoBERTa on a wide range of document-level natural language tasks including text classification, QA, and coreference resolution, achieving state-of-the-art results on two of these datasets.

接着，为评估 Longformer 能否替代现有预训练模型中的完整自注意力，我们以掩码语言建模（MLM）目标，从 RoBERTa（Liu et al. 2019）发布的检查点继续预训练。预训练后经微调用于下游语言任务，结果表明 Longformer 在文本分类、问答与共指消解等广泛文档级自然语言任务上稳定优于 RoBERTa，并在其中两个数据集上取得当时最优。

We finally introduce a variant of Longformer which instead of an encoder-only Transformer architecture, it follows an encoder-decoder architecture similar to the original Transformer model Vaswani et al. 2017, and it is intended for sequence-to-sequence (seq2seq) learning Sutskever et al. 2014. We call this model Longformer-Encoder-Decoder (LED) that uses Longformer’s efficient attention pattern on the encoder network, allowing it to address long document seq2seq tasks such as summarization. We demonstrate the effectiveness of LED on the arXiv summarization dataset Cohan et al. 2018.

最后我们引入 Longformer 的一个变体：不再是仅编码器结构，而是沿袭原始 Transformer（Vaswani et al. 2017）的编码器–解码器结构，面向序列到序列（seq2seq）学习（Sutskever et al. 2014）。我们称该模型为 Longformer-Encoder-Decoder（LED）：编码器采用 Longformer 的高效注意力模式，从而可处理长文档 seq2seq 任务（如摘要）。我们在 arXiv 摘要数据集（Cohan et al. 2018）上验证了 LED 的有效性。

Model	attention	char-LM	other	pretrain
	matrix		tasks
Transformer-XL Dai et al. 2019	ltr	yes	no	no
Adaptive Span Sukhbaatar et al. 2019	ltr	yes	no	no
Compressive Rae et al. 2020	ltr	yes	no	no
[0.4pt/2pt] Reformer Kitaev et al. 2020	sparse	yes	no	no
Sparse Child et al. 2019	sparse	yes	no	no
Routing Roy et al. 2020	sparse	yes	no	no
[0.4pt/2pt] BP-Transformer Ye et al. 2019	sparse	yes	MT	no
Blockwise Qiu et al. 2019	sparse	no	QA	yes
[0.4pt/2pt] Our Longformer	sparse	yes	multiple	yes

Table 1: Summary of prior work on adapting Transformers for long documents. ltr: left-to-right.

模型	注意力	字符 LM	其他	预训练
	矩阵		任务
Transformer-XL（Dai et al. 2019）	ltr	是	否	否
Adaptive Span（Sukhbaatar et al. 2019）	ltr	是	否	否
Compressive（Rae et al. 2020）	ltr	是	否	否
Reformer（Kitaev et al. 2020）	稀疏	是	否	否
Sparse（Child et al. 2019）	稀疏	是	否	否
Routing（Roy et al. 2020）	稀疏	是	否	否
BP-Transformer（Ye et al. 2019）	稀疏	是	MT	否
Blockwise（Qiu et al. 2019）	稀疏	否	QA	是
本文 Longformer	稀疏	是	多项	是

表 1：面向长文档适配 Transformer 的既有工作概览。ltr：从左到右。

(a) Full $n^{2}$ attention

(a) 完整 $n^{2}$ 注意力

2 Related Work

2 相关工作

Long-Document Transformers

Tab. 1 summarizes recent prior work on long documents. Two types of self-attention approaches have been explored. The first is a left-to-right (ltr) approach that processes the document in chunks moving from left-to-right. While such models have been successful in autoregressive language modeling, they are unsuitable for transfer learning approaches with tasks that benefit from bidirectional context.

长文档 Transformer

表 1 汇总了近期长文档相关工作。自注意力大致有两类思路。第一类是从左到右（ltr）：按块自左向右处理文档。这类模型在自回归语言建模上成功，但不适合需要双向上下文的迁移学习任务。

Our work falls within the other general approach that defines some form of sparse attention pattern and avoids computing the full quadratic attention matrix multiplication. The model with the most similar attention pattern to ours is Sparse Transformer Child et al. 2019, which uses a form of dilated sliding window of blocks of size 8x8 provided by BlockSparse Gray et al. 2017. Our implementation (§3) also includes a custom CUDA kernel, but it is more flexible and maintainable than BlockSparse which is implemented in C++, and designed for a specific version of TensorFlow. We also introduce additional task motivated global attention patterns suitable for common NLP tasks (§3) and show they are essential for good performance in the transfer learning setting.

我们的工作属于另一类：定义某种稀疏注意力模式，避免计算完整二次注意力矩阵乘积。与我们注意力模式最接近的是 Sparse Transformer（Child et al. 2019）：它使用 BlockSparse（Gray et al. 2017）提供的 8×8 块扩张滑动窗口。我们的实现（§3）也包括自定义 CUDA 核，但比用 C++ 实现、且针对特定 TensorFlow 版本设计的 BlockSparse 更灵活、更易维护。我们还引入面向常见 NLP 任务的额外全局注意力模式（§3），并表明它们在迁移学习设定下对良好表现至关重要。

A few models tried tasks other than autoregressive language modeling, which is a step forward because arguably focusing on language modeling as the primary evaluation has led to the development of models with limited applicability. BP-Transformer Ye et al. 2019 evaluated on machine translation (MT), but didn’t explore the pretrain-finetune setting. Blockwise attention Qiu et al. 2019 pretrained their models and evaluated on question answering (QA). However, the evaluation is limited as it doesn’t include language modeling, and the QA datasets are of relatively short documents,[^2] therefore the effectiveness of this model on long document tasks remains unexplored.

少数模型尝试了自回归语言建模以外的任务，这是一步前进——可以说把语言建模当作主评估，已催生出适用范围有限的模型。BP-Transformer（Ye et al. 2019）在机器翻译（MT）上评估，但未探索预训练–微调设定。Blockwise 注意力（Qiu et al. 2019）预训练后在问答（QA）上评估。但其评估有限：不含语言建模，且 QA 数据集文档相对较短，[^2]因而该模型在长文档任务上的有效性仍未充分检验。

Task-specific Models for Long Documents

Many task-specific approaches have been developed to workaround the 512 limit of pretrained transformer models like BERT. The simplest approach just truncates the document, commonly used for classification Xie et al. 2019. Another approach chunks the document into chunks of length 512 (could be overlapping), processes each chunk separately, then combines the activations with a task specific model Joshi et al. 2019. A third approach popular for multihop and open domain QA tasks uses a two-stage model where the first stage retrieves relevant documents that are passed onto the second stage for answer extraction Clark and Gardner 2017; Chen et al. 2017. All of these approaches suffer from information loss due to truncation or cascading errors from the two stage approach. In contrast, Longformer can process long sequences without truncating or chunking, allowing us to adopt a much simpler approach that concatenates the available context and processes it in a single pass.

面向长文档的任务专用模型

许多任务专用方法绕开 BERT 类预训练 Transformer 的 512 限制。最简单是直接截断文档，分类中很常见（Xie et al. 2019）。另一种把文档切成长度 512 的块（可重叠），分别处理后用任务专用模型汇总激活（Joshi et al. 2019）。第三种在多跳与开放域 QA 中流行：两阶段模型，第一阶段检索相关文档，再交给第二阶段抽取答案（Clark and Gardner 2017；Chen et al. 2017）。这些方法都因截断或两阶段级联误差而损失信息。相比之下，Longformer 无需截断或分块即可处理长序列，因而可采用更简单的做法：把可用上下文拼接后单次前向处理。

A few contemporaneous works[^3] have explored similar ideas to Longformer using local + global attention in Transformers, and pre-training it for long document natural language tasks. In particular, ETC Ainslie et al. 2020 uses a similar local + global attention instead of full self-attention to scale Transformers to long documents. Different from Longformer, ETC uses relative position embeddings (which we only used for the Autoregressive LM setting), introduces an additional training objective (CPC loss) for pre-training, and configures global attention in a slightly different way. It shows strong results on several tasks including reading comprehension and classification. GMAT Gupta and Berant 2020 uses a similar idea of few global locations in the input serving as global memory. BigBird Zaheer et al. 2020 is an extension over ETC with evaluation on additional tasks, including summarization. Importantly, through theoretical analysis, BigBird shows that sparse Transformers are universal approximators of sequence functions and preserve these properties of the full self-attention.

若干同期工作[^3]探索了与 Longformer 类似的思路：在 Transformer 中使用局部+全局注意力，并预训练以服务长文档自然语言任务。尤其是 ETC（Ainslie et al. 2020）用类似的局部+全局注意力替代完整自注意力，以扩展到长文档。与 Longformer 不同，ETC 使用相对位置嵌入（我们仅在自回归 LM 设定中使用）、引入额外预训练目标（CPC 损失），并以略不同方式配置全局注意力。它在阅读理解与分类等任务上表现强劲。GMAT（Gupta and Berant 2020）使用类似想法：输入中少数全局位置充当全局记忆。BigBird（Zaheer et al. 2020）是 ETC 的扩展，并在含摘要在内的更多任务上评估。重要的是，BigBird 通过理论分析表明稀疏 Transformer 是序列函数的通用逼近器，并保留完整自注意力的这些性质。

3 Longformer

The original Transformer model has a self-attention component with $O(n^{2})$ time and memory complexity where $n$ is the input sequence length. To address this challenge, we sparsify the full self-attention matrix according to an “attention pattern” specifying pairs of input locations attending to one another. Unlike the full self-attention, our proposed attention pattern scales linearly with the input sequence, making it efficient for longer sequences. This section discusses the design and implementation of this attention pattern.

原始 Transformer 的自注意力时间与内存复杂度为 $O(n^{2})$，其中 $n$ 为输入序列长度。为应对这一挑战，我们按「注意力模式」稀疏化完整自注意力矩阵，该模式指定彼此互相关注的输入位置对。与完整自注意力不同，我们所提注意力模式随输入序列线性扩展，因而对更长序列高效。本节讨论该注意力模式的设计与实现。

3.1 Attention Pattern

3.1 注意力模式

Sliding Window

Given the importance of local context Kovaleva et al. 2019, our attention pattern employs a fixed-size window attention surrounding each token. Using multiple stacked layers of such windowed attention results in a large receptive field, where top layers have access to all input locations and have the capacity to build representations that incorporate information across the entire input, similar to CNNs Wu et al. 2019. Given a fixed window size $w$, each token attends to $\frac{1}{2}w$ tokens on each side (Fig. 2(b)). The computation complexity of this pattern is $O(n\times w)$, which scales linearly with input sequence length $n$. In a transformer with $\ell$ layers, the receptive field size at the top layer is $\ell\times w$ (assuming $w$ is fixed for all layers). Depending on the application, it might be helpful to use different values of $w$ for each layer to balance between efficiency and model representation capacity (§4.1).

滑动窗口

鉴于局部上下文的重要性（Kovaleva et al. 2019），我们的注意力模式对每个 token 使用固定大小的窗口注意力。堆叠多层此类窗口注意力可形成很大的感受野：顶层可访问全部输入位置，并有能力构建融合整段输入信息的表征，类似 CNN（Wu et al. 2019）。给定固定窗口大小 $w$，每个 token 关注两侧各 $\frac{1}{2}w$ 个 token（图 2(b)）。该模式计算复杂度为 $O(n\times w)$，随输入序列长度 $n$ 线性增长。在具有 $\ell$ 层的 Transformer 中，顶层感受野大小为 $\ell\times w$（假设各层 $w$ 固定）。依应用而定，可为每层使用不同的 $w$，以在效率与表征能力之间折中（§4.1）。

Dilated Sliding Window

To further increase the receptive field without increasing computation, the sliding window can be “dilated”. This is analogous to dilated CNNs van den Oord et al. 2016 where the window has gaps of size dilation $d$ (Fig. 2(c)). Assuming a fixed $d$ and $w$ for all layers, the receptive field is $\ell\times d\times w$, which can reach tens of thousands of tokens even for small values of $d$.

扩张滑动窗口

为在不增加计算量的前提下进一步扩大感受野，滑动窗口可以「扩张」。这类似于扩张 CNN（van den Oord et al. 2016）：窗口中带有大小为扩张率 $d$ 的空隙（图 2(c)）。若各层固定 $d$ 与 $w$，感受野为 $\ell\times d\times w$，即便 $d$ 很小也可达数万 token。

In multi-headed attention, each attention head computes a different attention score. We found settings with different dilation configurations per head improves performance by allowing some heads without dilation to focus on local context, while others with dilation focus on longer context.

在多头注意力中，每个注意力头计算不同的注意力分数。我们发现：对各头使用不同扩张配置可提升表现——部分无扩张的头专注局部上下文，带扩张的头则关注更长距离上下文。

Global Attention

In state-of-the-art BERT-style models for natural language tasks, the optimal input representation differs from language modeling and varies by task. For masked language modeling (MLM), the model uses local context to predict the masked word, while for classification, the model aggregates the representation of the whole sequence into a special token ([CLS] in case of BERT). For QA, the question and document are concatenated, allowing the model to compare the question with the document through self-attention.

全局注意力

在面向自然语言任务的先进 BERT 风格模型中，最优输入表征不同于语言建模，并随任务而变。对掩码语言建模（MLM），模型用局部上下文预测被掩码词；对分类，模型把整段序列表征汇聚到特殊 token（BERT 中为 [CLS]）；对 QA，则把问题与文档拼接，使模型通过自注意力对照问题与文档。

In our case, the windowed and dilated attention are not flexible enough to learn task-specific representations. Accordingly, we add “global attention” on few pre-selected input locations. Importantly, we make this attention operation symmetric: that is, a token with a global attention attends to all tokens across the sequence, and all tokens in the sequence attend to it. Fig. 2(d) shows an example of a sliding window attention with global attention at a few tokens at custom locations. For example for classification, global attention is used for the [CLS] token while in QA global attention is provided on all question tokens. Since the number of such tokens is small relative to and independent of $n$ the complexity of the combined local and global attention is still $O(n)$. While specifying global attention is task specific, it is a easy way to add inductive bias to the model’s attention, and it is much simpler than existing task specific approaches that use complex architecture to combine information across smaller input chunks.

对我们而言，窗口与扩张注意力尚不足以灵活学习任务专用表征。因此我们在少数预选输入位置上加入「全局注意力」。重要的是，我们使该注意力对称：带全局注意力的 token 关注序列中全部 token，且序列中全部 token 也关注它。图 2(d) 给出在若干自定义位置加入全局注意力的滑动窗口示例。例如分类时对 [CLS] 使用全局注意力，QA 时则对全部问题 token 提供全局注意力。由于此类 token 数量相对 $n$ 很小且与 $n$ 无关，局部与全局注意力合起来的复杂度仍为 $O(n)$。虽说指定全局注意力是任务相关的，但这是向模型注意力注入归纳偏置的简便方式，也远比用复杂结构在较小输入块之间汇总信息的既有任务专用方法简单。

Linear Projections for Global Attention

Recall that given the linear projections $Q$, $K$, $V$, the Transformer model Vaswani et al. 2017 computes attention scores as follows:

全局注意力的线性投影

回顾：给定线性投影 $Q$、$K$、$V$，Transformer（Vaswani et al. 2017）按如下方式计算注意力分数：

$$\displaystyle\text{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V \tag{1}$$

We use two sets of projections, $Q_{s}$, $K_{s}$, $V_{s}$ to compute attention scores of sliding window attention, and $Q_{g}$, $K_{g}$, $V_{g}$ to compute attention scores for the global attention. The additional projections provide flexibility to model the different types of attention, which we show is critical for best performance on downstream tasks. $Q_{g}$, $K_{g}$, $V_{g}$ are all initialized with values that match $Q_{s}$, $K_{s}$, $V_{s}$.

我们使用两套投影：$Q_{s}$、$K_{s}$、$V_{s}$ 计算滑动窗口注意力分数，$Q_{g}$、$K_{g}$、$V_{g}$ 计算全局注意力分数。额外投影为建模不同类型注意力提供灵活性，我们表明这对下游任务取得最佳表现至关重要。$Q_{g}$、$K_{g}$、$V_{g}$ 均初始化为与 $Q_{s}$、$K_{s}$、$V_{s}$ 相同的值。

3.2 Implementation

3.2 实现

In regular transformers, attention scores are computed as in Eqn. 1. The expensive operation is the matrix multiplication $QK^{T}$ because both $Q$ and $K$ have $n$ (sequence length) projections. For Longformer, the dilated sliding window attention computes only a fixed number of the diagonals of $QK^{T}$. As shown in Fig. 1, this results in a linear increase in memory usage compared to quadratic increase for full self-attention. However, implementing it requires a form of banded matrix multiplication that is not supported in existing deep learning libraries like PyTorch/Tensorflow. Fig. 1 compares the performance of three different ways of implementing it: loop is a memory efficient PyTorch implementation that supports dilation but is unusably slow and only used for testing; chunks only supports the non-dilated case and is used for the pretraining/finetuning setting; and cuda is our fully functioning highly optimized custom CUDA kernel implemented using TVM Chen et al. 2018 and used for the language modeling experiments (see Appendix A for more details).

在常规 Transformer 中，注意力分数按式 1 计算。昂贵操作是矩阵乘法 $QK^{T}$，因为 $Q$ 与 $K$ 都有 $n$（序列长度）个投影。对 Longformer，扩张滑动窗口注意力只计算 $QK^{T}$ 的固定条数对角线。如图 1 所示，这使内存随序列线性增长，而完整自注意力为二次增长。但实现需要一种带状矩阵乘法，现有深度学习库（如 PyTorch/Tensorflow）并不直接支持。图 1 比较了三种实现：loop 是支持扩张、内存高效的 PyTorch 实现，但慢到几乎不可用，仅用于测试；chunks 仅支持非扩张情形，用于预训练/微调设定；cuda 是我们用 TVM（Chen et al. 2018）实现的功能完整、高度优化的自定义 CUDA 核，用于语言建模实验（更多细节见附录 A）。

4 Autoregressive Language Modeling

4 自回归语言建模

Autoregressive or left-to-right language modeling is loosely defined as estimating the probability distribution of an existing token/character given its previous tokens/characters in an input sequence. This task is considered one of the fundamental tasks in natural language and recent prior work on modeling long sequences using transformers has relied on this task as their primary evaluation Dai et al. 2019; Rae et al. 2020; Sukhbaatar et al. 2019. Similarly, we develop and evaluate our model on autoregressive language modeling.

自回归或从左到右语言建模可宽泛定义为：给定输入序列中某 token/字符之前的 token/字符，估计其概率分布。该任务被视为自然语言的基本任务之一；近期用 Transformer 建模长序列的工作也主要以此作为评估（Dai et al. 2019；Rae et al. 2020；Sukhbaatar et al. 2019）。同样，我们在自回归语言建模上开发并评估模型。

4.1 Attention Pattern

4.1 注意力模式

For autoregressive language modeling we use our dilated sliding window attention. Following Sukhbaatar et al. 2019 we use differing window sizes across the layers. In particular, we use small window sizes for the lower layers and increase window sizes as we move to higher layers. This allows the top layers to learn higher-level representation of the entire sequence while having the lower layers capture local information. In addition, it provides balance between efficiency (smaller window sizes are less computationally expensive due to fewer nonzero values) and performance (larger window sizes have richer representation power and often result in performance improvements).

对自回归语言建模，我们使用扩张滑动窗口注意力。沿袭 Sukhbaatar et al. 2019，我们在各层使用不同窗口大小：下层窗口较小，随层升高窗口增大。这使顶层可学习整段序列的高层表征，同时下层捕获局部信息。此外，它在效率（较小窗口非零值更少、计算更省）与表现（较大窗口表征更丰富、常带来提升）之间取得平衡。

We do not use dilated sliding windows for lower layers to maximize their capacity to learn and utilize the immediate local context. For the higher layers, we use a small amount of increasing dilation only on 2 heads. This gives the model the ability to directly attend to distant tokens without sacrificing local context.

下层不使用扩张滑动窗口，以最大化学习并利用紧邻局部上下文的能力。对更高层，我们仅在 2 个头上使用少量递增扩张。这使模型能直接关注远处 token，同时不牺牲局部上下文。

4.2 Experiment Setup

4.2 实验设置

To compare to prior work we focus on character-level LM (Mahoney 2009, text8 and enwik8;).

为与先前工作比较，我们聚焦字符级 LM（Mahoney 2009，text8 与 enwik8）。

Training

Ideally, we would like to train our model on the largest window size and sequence length we can fit in a modern GPU memory. However, we found that the model needs a large number of gradient updates to learn the local context first, before learning to utilize longer context. To accommodate this, we adopt a staged training procedure where we increase the attention window size and sequence length across multiple training phases. In particular, in the first phase we start with a short sequence length and window size, then on each subsequent phase, we double the window size and the sequence length, and halve the learning rate. This makes training fast, while keeping the slow part (longest sequences and window sizes) to the end. We train the model over 5 total phases with starting sequence length of 2,048 and ending sequence length of 23,040 on the last phase (see Appendix B for detailed configurations of each phase, and for all other hyperparameters).

训练

理想情况下，我们希望在现代 GPU 内存可容纳的最大窗口与序列长度上训练。但我们发现模型需要大量梯度更新先学会局部上下文，再学会利用更长上下文。为此我们采用分阶段训练：跨多个阶段增大注意力窗口与序列长度。具体地，第一阶段从较短序列与窗口开始，之后每阶段把窗口与序列长度加倍，并把学习率减半。这使训练更快，并把慢的部分（最长序列与窗口）留到最后。我们共训练 5 个阶段，起始序列长度 2,048，末阶段为 23,040（各阶段详细配置与其他超参见附录 B）。

Evaluation

We evaluate with sequences of length 32,256. Following Dai et al. 2019, we split the dataset into overlapping sequences of size 32,256 with a step of size 512, and report the performance on the last 512 tokens on the sequence.

评估

我们以长度 32,256 的序列评估。沿袭 Dai et al. 2019，把数据集切成大小 32,256、步长 512 的重叠序列，并报告序列末尾 512 个 token 上的表现。

4.2.1 Results

Tab. 2 and 3 summarize evaluation results on text8 and enwik8 datasets. We achieve a new state-of-the-art on both text8 and enwik8 using the small models with BPC of 1.10 and 1.00 on text8 and enwik8 respectively, demonstrating the effectiveness of our model.

4.2.1 结果

表 2 与表 3 汇总 text8 与 enwik8 上的评估。我们用小模型在 text8 与 enwik8 上分别达到 1.10 与 1.00 BPC 的新最优，表明模型有效。

For large models, given how expensive these experiments are, and following recent work Kitaev et al. 2020; Rae et al. 2020, we are only evaluating on enwik8. Tab. 3 shows that Longformer outperforms the comparable Transformer-XL model, matches the performance of the comparable Sparse Transformer Child et al. 2019, and matches or slightly underperforms recent models that have more than twice the number of parameters. It is worth noting that Adaptive Span Sukhbaatar et al. 2019 and Compressive Transformer Rae et al. 2020 are not good fit for the pretraining-finetuning paradigm as discussed in §2.

对大模型，鉴于实验昂贵，并沿袭近期工作（Kitaev et al. 2020；Rae et al. 2020），我们仅在 enwik8 上评估。表 3 显示 Longformer 优于可比的 Transformer-XL，与可比的 Sparse Transformer（Child et al. 2019）持平，并与参数量两倍以上的近期模型持平或略逊。值得注意的是，如 §2 所讨论，Adaptive Span（Sukhbaatar et al. 2019）与 Compressive Transformer（Rae et al. 2020）并不适合预训练–微调范式。

Model	#Param	Dev	Test
Dataset text8
T12 Al-Rfou et al. 2018	44M	-	1.18
Adaptive Sukhbaatar et al. 2019	38M	1.05	1.11
BP-Transformer Ye et al. 2019	39M	-	1.11
Our Longformer	41M	1.04	1.10
[0.2pt/0.2pt] Dataset enwik8
T12 Al-Rfou et al. 2018	44M	-	1.11
Transformer-XL Dai et al. 2019	41M	-	1.06
Reformer Kitaev et al. 2020	-	-	1.05
Adaptive Sukhbaatar et al. 2019	39M	1.04	1.02
BP-Transformer Ye et al. 2019	38M	-	1.02
Our Longformer	41M	1.02	1.00

Table 2: Small model BPC on text8 & enwik8

模型	#参数	Dev	Test
数据集 text8
T12（Al-Rfou et al. 2018）	44M	-	1.18
Adaptive（Sukhbaatar et al. 2019）	38M	1.05	1.11
BP-Transformer（Ye et al. 2019）	39M	-	1.11
本文 Longformer	41M	1.04	1.10
数据集 enwik8
T12（Al-Rfou et al. 2018）	44M	-	1.11
Transformer-XL（Dai et al. 2019）	41M	-	1.06
Reformer（Kitaev et al. 2020）	-	-	1.05
Adaptive（Sukhbaatar et al. 2019）	39M	1.04	1.02
BP-Transformer（Ye et al. 2019）	38M	-	1.02
本文 Longformer	41M	1.02	1.00

表 2：小模型在 text8 与 enwik8 上的 BPC

Model	#Param	Test BPC
Transformer-XL (18 layers)	88M	1.03
Sparse Child et al. 2019	$\approx$100M	0.99
Transformer-XL (24 layers)	277M	0.99
Adaptive Sukhbaatar et al. 2019	209M	0.98
Compressive Rae et al. 2020	277M	0.97
Routing Roy et al. 2020	$\approx$223M	0.99
Our Longformer	102M	0.99

Table 3: Performance of large models on enwik8

模型	#参数	Test BPC
Transformer-XL（18 层）	88M	1.03
Sparse（Child et al. 2019）	$\approx$100M	0.99
Transformer-XL（24 层）	277M	0.99
Adaptive（Sukhbaatar et al. 2019）	209M	0.98
Compressive（Rae et al. 2020）	277M	0.97
Routing（Roy et al. 2020）	$\approx$223M	0.99
本文 Longformer	102M	0.99

表 3：大模型在 enwik8 上的表现

4.2.2 Ablation Study

Model	Dev BPC
Decreasing $w$ (from 512 to 32)	1.24
Fixed $w$ (= 230)	1.23
Increasing $w$ (from 32 to 512)	1.21
No Dilation	1.21
Dilation on 2 heads	1.20

Table 4: Top: changing window size across layers. Bottom: with/without dilation (@ 150K steps on phase1)

4.2.2 消融研究

模型	Dev BPC
递减 $w$（从 512 到 32）	1.24
固定 $w$（= 230）	1.23
递增 $w$（从 32 到 512）	1.21
无扩张	1.21
在 2 个头上扩张	1.20

表 4：上：跨层改变窗口大小。下：有/无扩张（阶段 1 配置上训练 150K 步）

To show the importance of the design choices of our attention patterns, we tried different variants and report their controlled experiment results. To make the ablation study more manageable, we train each configuration for 150K steps[^4] with phase 1 configuration on a small model on text8, then report the BPC performance on the dev set.

为说明注意力模式设计选择的重要性，我们尝试不同变体并报告对照实验结果。为使消融更可管理，我们在 text8 上以阶段 1 配置对小模型训练各配置 150K 步[^4]，再报告开发集 BPC。

The top of Tab. 4 demonstrates the impact of different ways of configuring the window sizes per layer. We observe that increasing the window size from the bottom to the top layer leads to the best performance, arranging them in the reverse way leads to worse performance, and using a fixed window size (the average of window sizes of the other configuration) leads to a performance that it is in between. The bottom of Tab. 4 shows the impact of adding dilation. Adding some dilation to two heads leads to some improvement compared with no dilation at all.

表 4 上半部展示各层窗口大小配置的影响。我们观察到：自底向上增大窗口表现最好，反向排列更差，固定窗口（取其他配置窗口均值）则介于两者之间。表 4 下半部展示加入扩张的影响：在两个头上加入一定扩张，相对完全无扩张有所提升。

5 Pretraining and Finetuning

5 预训练与微调

Current state-of-the-art systems for many NLP tasks finetune a pretrained model with task supervision (e.g. BERT). One of our main motivations is to develop such a model suitable for long document tasks. To do so, we pretrained Longformer on a document corpus and finetune it for six tasks, including classification, QA and coreference resolution. The resulting model can process sequences up to 4,096 tokens long (8 times longer than BERT)[^5].

许多 NLP 任务上当前领先系统用任务监督微调预训练模型（如 BERT）。我们的主要动机之一是开发适合长文档任务的此类模型。为此，我们在文档语料上预训练 Longformer，并在含分类、问答与共指消解在内的六项任务上微调。所得模型可处理长达 4,096 token 的序列（为 BERT 的 8 倍）[^5]。

We pretrain Longformer with masked language modeling (MLM), where the goal is to recover randomly masked tokens in a sequence. Since MLM pretraining is expensive, we continue pretraining from the RoBERTa Liu et al. 2019 released checkpoint, while only making the minimal changes necessary to support Longformer’s attention mechanism. Note that our attention pattern can be plugged into any pretrained transformer model without the need to change the model architecture.

我们以掩码语言建模（MLM）预训练 Longformer，目标是恢复序列中随机掩码的 token。由于 MLM 预训练昂贵，我们从 RoBERTa（Liu et al. 2019）发布的检查点继续预训练，仅做支持 Longformer 注意力所需的最小改动。注意：我们的注意力模式可插入任意预训练 Transformer，而无需改动模型结构。

Attention Pattern

We use sliding window attention with window size of 512, therefore using the same amount of computation as RoBERTa.[^6]

注意力模式

我们使用窗口大小 512 的滑动窗口注意力，因而计算量与 RoBERTa 相当。[^6]

Position Embeddings

RoBERTa uses learned absolute position embeddings with the maximum position being 512. To support longer documents, we add extra position embeddings to support up to position 4,096. To leverage RoBERTa’s pretrained weights, instead of randomly initializing the new position embeddings, we initialize them by copying the 512 position embeddings from RoBERTa multiple times as analysis of BERT’s attention heads shows a strong learned bias to attending to local context, including the previous or next token Clark et al. 2019. Using the copy initialization preserves this local structure everywhere except at the partition boundaries. Despite its simplicity, we found this to be a very effective (see Tab. 5), allowing Longformer pretraining to rapidly converge with a small number of gradient updates.

位置嵌入

RoBERTa 使用学习的绝对位置嵌入，最大位置为 512。为支持更长文档，我们额外加入位置嵌入以支持到位置 4,096。为利用 RoBERTa 预训练权重，新位置嵌入不随机初始化，而是多次复制 RoBERTa 的 512 个位置嵌入——对 BERT 注意力头的分析显示其学到了强烈的局部上下文偏置，包括关注前一个或后一个 token（Clark et al. 2019）。复制初始化在除分区边界外处处保留这种局部结构。尽管简单，我们发现它非常有效（见表 5），使 Longformer 预训练能以少量梯度更新快速收敛。

Model	base	large
RoBERTa (seqlen: 512)	1.846	1.496
Longformer (seqlen: 4,096)	10.299	8.738
+ copy position embeddings	1.957	1.597
+ 2K gradient updates	1.753	1.414
+ 65K gradient updates	1.705	1.358
Longformer (train extra pos. embed. only)	1.850	1.504

Table 5: MLM BPC for RoBERTa and various pretrained Longformer configurations.

模型	base	large
RoBERTa（seqlen: 512）	1.846	1.496
Longformer（seqlen: 4,096）	10.299	8.738
+ 复制位置嵌入	1.957	1.597
+ 2K 梯度更新	1.753	1.414
+ 65K 梯度更新	1.705	1.358
Longformer（仅训练额外位置嵌入）	1.850	1.504

表 5：RoBERTa 与多种预训练 Longformer 配置的 MLM BPC。

Continued MLM Pretraining

We pretrain Longformer using fairseq Ott et al. 2019 on a corpus of long documents that we compiled (see Appendix C for corpus details). We train two model sizes, a base model and a large model. Both models are trained for 65K gradient updates with sequences length 4,096, batch size 64 ($2^{18}$ tokens), maximum learning rate of 3e-5, linear warmup of 500 steps, followed by a power 3 polynomial decay. The rest of the hyperparameters are the same as RoBERTa.

继续 MLM 预训练

我们用 fairseq（Ott et al. 2019）在自编长文档语料上预训练 Longformer（语料细节见附录 C）。训练 base 与 large 两种规模，序列长度 4,096、批大小 64（$2^{18}$ token）、最大学习率 3e-5、线性预热 500 步，随后按幂次 3 的多项式衰减，共 65K 梯度更新。其余超参与 RoBERTa 相同。

Wordpieces	WH	TQA	HQA	ON	IMDB	HY
avg.	1,535	6,589	1,316	506	300	705
95th pctl.	3,627	17,126	1,889	1,147	705	1,975

Table 6: Average and 95th percentile of context length of datasets in wordpieces. WH: WikiHop, TQA: TriviaQA, HQA: HotpotQA, ON: OntoNotes, HY: Hyperpartisan news

Wordpieces	WH	TQA	HQA	ON	IMDB	HY
平均	1,535	6,589	1,316	506	300	705
第 95 百分位	3,627	17,126	1,889	1,147	705	1,975

表 6：各数据集上下文长度（wordpieces）的平均值与第 95 百分位。WH: WikiHop，TQA: TriviaQA，HQA: HotpotQA，ON: OntoNotes，HY: Hyperpartisan 新闻

	QA	Coref.	Classification
Model	WikiHop	TriviaQA	HotpotQA	OntoNotes	IMDB	Hyperpartisan
RoBERTa-base	72.4	74.3	63.5	78.4	95.3	87.4
Longformer-base	75.0	75.2	64.4	78.6	95.7	94.8

Table 7: Summary of finetuning results on QA, coreference resolution, and document classification. Results are on the development sets comparing our Longformer-base with RoBERTa-base. TriviaQA, Hyperpartisan metrics are F1, WikiHop and IMDB use accuracy, HotpotQA is joint F1, OntoNotes is average F1.

	QA	共指	分类
模型	WikiHop	TriviaQA	HotpotQA	OntoNotes	IMDB	Hyperpartisan
RoBERTa-base	72.4	74.3	63.5	78.4	95.3	87.4
Longformer-base	75.0	75.2	64.4	78.6	95.7	94.8

表 7：问答、共指消解与文档分类微调结果汇总。结果在开发集上，比较 Longformer-base 与 RoBERTa-base。TriviaQA、Hyperpartisan 为 F1，WikiHop 与 IMDB 为准确率，HotpotQA 为联合 F1，OntoNotes 为平均 F1。

Tab. 5 shows the BPC on the development set of our training corpus. The first row shows a 1.846 BPC using RoBERTa-base, which is comparable to the 1.880 BPC reported on the RoBERTa paper on their corpus. This indicates our training corpus is from a distribution close to that used to train RoBERTa. The following two rows show the performance of Longformer before pretraining with randomly initialized position embeddings and with copied position embeddings. The significant difference indicates the importance of the copy initialization, and the relative small difference between the RoBERTa BPC and the initialized BPC indicates that our sliding window attention is working well with the RoBERTa weights. The following two rows show the impact of continuing pretraining. Traininig for 2K steps improves BPC from 1.957 to 1.753, which further decreases to 1.705 after 65K steps, demonstrating the model is learning to better utilize the sliding window attention and longer context. Similar patterns are observed with RoBERTa-large and Longformer-large.

表 5 给出训练语料开发集上的 BPC。第一行 RoBERTa-base 为 1.846 BPC，与 RoBERTa 论文在其语料上报告的 1.880 BPC 接近，说明我们的训练语料分布接近 RoBERTa 所用分布。随后两行给出预训练前 Longformer 在随机初始化位置嵌入与复制位置嵌入下的表现。显著差异说明复制初始化很重要；RoBERTa BPC 与初始化后 BPC 差距相对较小，说明滑动窗口注意力与 RoBERTa 权重配合良好。再往后两行展示继续预训练的影响：训练 2K 步把 BPC 从 1.957 降到 1.753，65K 步后再降到 1.705，表明模型在更好地利用滑动窗口注意力与更长上下文。RoBERTa-large 与 Longformer-large 上可见类似模式。

Frozen RoBERTa Weights

We also pretrained Longformer while freezing all RoBERTa weights, and only training the new position embeddings. The motivation for this configuration is to perfectly preserve the RoBERTa performance on short documents. This configuration has a BPC of 1.850 (down from 1.957 at initialization), but higher than 1.705 where all the weights are trainable.

冻结 RoBERTa 权重

我们还在冻结全部 RoBERTa 权重、仅训练新位置嵌入的设定下预训练 Longformer。动机是完整保留 RoBERTa 在短文档上的表现。该配置 BPC 为 1.850（相对初始化的 1.957 有所下降），但仍高于全部权重可训练时的 1.705。

6 Tasks

6 任务

We apply Longformer to multiple long document tasks, including QA, coreference resolution and classification. Tab. 6 shows the evaluation datasets have contexts significantly longer than 512 wordpieces. Our primary goal is to evaluate whether our attention mechanism can act as a replacement for the standard self-attention mechanism in BERT style models, and to perform controlled trials against a strong baseline. We are also interested in evaluating whether we can replace complicated task specific models necessitated by BERT’s limited context with simpler models that just concatenate all available context into a single sequence.

我们把 Longformer 用于多项长文档任务，包括问答、共指消解与分类。表 6 显示评估数据集的上下文远长于 512 wordpieces。主要目标是评估我们的注意力机制能否替代 BERT 风格模型中的标准自注意力，并与强基线做对照试验。我们也关心：能否用把全部可用上下文拼成单序列的更简单模型，替代因 BERT 上下文有限而必需的复杂任务专用模型。

Our baseline is a RoBERTa based model that breaks the context into the longest possible segment, passes each individually through RoBERTa, and concatenates the activations for further processing. For QA tasks, we also concatenate the question to each segment so that RoBERTa can condition it’s contextual representations of the context on the question. The Longformer variant replaces the RoBERTa self-attention mechanism with our windowed attention used during pretraining, plus a task motivated global attention. The global attention uses additional linear projections (§3.1).

基线是基于 RoBERTa 的模型：把上下文切成尽可能长的段，分别送入 RoBERTa，再拼接激活做后续处理。对 QA 任务，我们还把问题拼到每段，使 RoBERTa 能在问题条件下构建上下文表征。Longformer 变体用预训练时的窗口注意力替换 RoBERTa 自注意力，并加上面向任务的全局注意力。全局注意力使用额外线性投影（§3.1）。

6.1 Question answering

6.1 问答

We used three datasets: WikiHop Welbl et al. 2018, TriviaQA (Joshi et al. 2017, Wikipedia setting), and HotpotQA, (Yang et al. 2018, distractor setting).[^7]

我们使用三个数据集：WikiHop（Welbl et al. 2018）、TriviaQA（Joshi et al. 2017，Wikipedia 设定）以及 HotpotQA（Yang et al. 2018，干扰项设定）。[^7]

For WikiHop and TriviaQA we follow the simple QA model of BERT Devlin et al. 2019, and concatenate question and documents into one long sequence, run it through Longformer, then have a dataset-specific prediction layer. WikiHop uses a classification layer for the candidate while TriviaQA uses the loss function of Clark and Gardner 2017 to predict answer span. We include global attention to question tokens and answer candidates for WikiHop and to question tokens for TriviaQA.

对 WikiHop 与 TriviaQA，我们沿袭 BERT（Devlin et al. 2019）的简单 QA 模型：把问题与文档拼成一条长序列，经 Longformer，再接数据集专用预测层。WikiHop 对候选做分类层，TriviaQA 用 Clark and Gardner 2017 的损失函数预测答案跨度。我们对 WikiHop 的问题 token 与答案候选、以及对 TriviaQA 的问题 token 施加全局注意力。

HotpotQA is a multihop QA dataset that involves extracting answer spans and evidence sentences from 10 Wikipedia paragraphs, 2 of which are relevant and the rest are distractors. We use a two-stage model that first selects the most relevant paragraphs then passes them to a second stage for answer extraction. Both stages concatenate question and context into one sequence, run it through Longformer, then use task-specific prediction layers. We train the models in a multi-task way to predict relevant paragraphs, evidence sentences, answer spans and question types (yes/no/span) jointly. Note that this model is simpler than recent SOTA models that include complex task-specific architectures (e.g., Tu et al. 2019; Chen et al. 2019; Tu et al. 2020; Groeneveld et al. 2020). See Appendix D for further details about the models and hyperparameters.

HotpotQA 是多跳 QA 数据集，需从 10 段维基百科段落中抽取答案跨度与证据句，其中 2 段相关、其余为干扰。我们使用两阶段模型：先选出最相关段落，再交给第二阶段做答案抽取。两阶段都把问题与上下文拼成一条序列，经 Longformer，再接任务专用预测层。我们以多任务方式联合训练，预测相关段落、证据句、答案跨度与问题类型（是/否/跨度）。注意该模型比近期含复杂任务专用结构的 SOTA 模型更简单（例如 Tu et al. 2019；Chen et al. 2019；Tu et al. 2020；Groeneveld et al. 2020）。模型与超参细节见附录 D。

6.2 Coreference Resolution

6.2 共指消解

We use OntoNotes Pradhan et al. 2012, and the model from Joshi et al. 2019, a modification of the system from Lee et al. 2018 to replace ELMo with BERT. The Longformer system is a straightforward adaption of the baseline model by replacing RoBERTa with Longformer and extending the sequence length. We didn’t use global attention for this task.

我们使用 OntoNotes（Pradhan et al. 2012），以及 Joshi et al. 2019 的模型——该模型修改自 Lee et al. 2018，用 BERT 替换 ELMo。Longformer 系统是对基线的直接适配：用 Longformer 替换 RoBERTa 并扩展序列长度。该任务未使用全局注意力。

6.3 Document Classification

6.3 文档分类

We evaluate on IMDB Maas et al. 2011 and Hyperpartisan news detection Kiesel et al. 2019 datasets.[^8] IMDB is a standard sentiment classification datasets consisting of movie reviews. While most documents in this dataset are short, about 13.6% of them are larger than 512 wordpieces (Tab. 6). Documents in Hyperpartisan are relatively long, and it is small with only 645 documents making it a good test for Longformer’s ability to adapt to limited data. We use global attention on the [CLS] token.

我们在 IMDB（Maas et al. 2011）与 Hyperpartisan 新闻检测（Kiesel et al. 2019）上评估。[^8] IMDB 是标准情感分类数据集，由影评组成。多数文档较短，但约 13.6% 超过 512 wordpieces（表 6）。Hyperpartisan 文档相对较长，且仅有 645 篇，是检验 Longformer 在有限数据上适应能力的好测试。我们对 [CLS] token 使用全局注意力。

6.4 Results

6.4 结果

Main Result

Tab. 7 summarizes the results of all our finetuning experiments. We observe that Longformer consistently outperforms the RoBERTa baseline. Its performance gain is especially obvious for tasks that require long context such as WikiHop and Hyperpartisan. For TriviaQA, the improvement is more modest as the local context is often sufficient to answer the question. In the case of HotpotQA, the supporting fact auxiliary supervision allows models to easily find relevant contexts and then focus on local context, leading to smaller gains. This is contrasted with WikiHop that only includes distant supervision of intermediate reasoning chains, where our approach excels by reasoning over the entire context. On the IMDB and OntoNotes datasets the performance gains are smaller. For IMDB, the majority of the dataset consists of short documents and thus it is expected to see smaller improvements. For OntoNotes, we found that the distance between any two mentions is typically quite small so that a baseline that processes smaller chunks separately is able to stitch together mentions into coreference chains without considering cross chunk interactions.

主要结果

表 7 汇总全部微调实验。我们观察到 Longformer 稳定优于 RoBERTa 基线。对 WikiHop 与 Hyperpartisan 等需要长上下文的任务，提升尤为明显。对 TriviaQA，提升较温和，因为局部上下文往往已足以回答。对 HotpotQA，支持事实的辅助监督使模型容易找到相关上下文后再聚焦局部，因而增益较小。这与 WikiHop 形成对照：后者仅有中间推理链的弱监督，我们的方法可通过整段上下文推理而占优。在 IMDB 与 OntoNotes 上增益较小。对 IMDB，数据集主体是短文档，预期提升有限。对 OntoNotes，我们发现任意两指称之间的距离通常很小，因而分别处理较小块的基线也能在不考虑跨块交互的情况下把指称串成共指链。

Longformer-large for QA

We also evaluate the performance of Longformer-large on long context QA tasks. Tab. 8 shows that our Longformer-large achieves new state-of-the-art results[^9] on WikiHop and TriviaQA by large margins (3.6 and 4 points respectively), and for HotpotQA, it underperforms the current state-of-the-art Fang et al. 2020 by a point. Tab. 9 shows the detailed results of HotpotQA compared with published and unpublished concurrent models. Longformer places second on the published leaderboard, outperforming all other published results except for HGN Fang et al. 2020. All published top performing models in this task Tu et al. 2019; Fang et al. 2020; Shao et al. 2020 use GNNs Kipf and Welling 2017 or graph network of entities, which seem to encode an important inductive bias for the task and can potentially improve our results further. Nevertheless, Longformer performs strongly outperforming all other methods including the recent non-GNN methods Glaß et al. 2019; Shao et al. 2020; Groeneveld et al. 2020.

用于 QA 的 Longformer-large

我们还在长上下文 QA 任务上评估 Longformer-large。表 8 显示 Longformer-large 在 WikiHop 与 TriviaQA 上以较大幅度（分别为 3.6 与 4 分）取得新最优[^9]，在 HotpotQA 上则比当时最优（Fang et al. 2020）低 1 分。表 9 给出 HotpotQA 与已发表及未发表同期模型的详细对比。Longformer 在公开排行榜已发表结果中位列第二，仅次于 HGN（Fang et al. 2020），并优于其余已发表结果。该任务上所有已发表顶尖模型（Tu et al. 2019；Fang et al. 2020；Shao et al. 2020）都使用 GNN（Kipf and Welling 2017）或实体图网络，这似乎编码了任务重要的归纳偏置，并有可能进一步提升我们的结果。尽管如此，Longformer 仍强劲地超过包括近期非 GNN 方法在内的其他方法（Glaß et al. 2019；Shao et al. 2020；Groeneveld et al. 2020）。

Model	WikiHop	TriviaQA	HotpotQA
Current∗ SOTA	78.3	73.3	74.2
Longformer-large	81.9	77.3	73.2

Table 8: Leaderboard results of Longformer-large at time of submission (May 2020). All numbers are F1 scores.

模型	WikiHop	TriviaQA	HotpotQA
当时∗ SOTA	78.3	73.3	74.2
Longformer-large	81.9	77.3	73.2

表 8：提交时（2020 年 5 月）Longformer-large 的排行榜结果。数字均为 F1。

Model	ans.	supp.	joint
TAP 2 (ensemble) Glaß et al. 2019	79.8	86.7	70.7
SAE Tu et al. 2019	79.6	86.7	71.4
Quark (dev) Groeneveld et al. 2020	81.2	87.0	72.3
C2F Reader Shao et al. 2020	81.2	87.6	72.8
[0.4pt/2pt] Longformer-large	81.3	88.3	73.2
[0.4pt/2pt] ETC-large*†* Ainslie et al. 2020	81.2	89.1	73.6
GSAN-large*†*	81.6	88.7	73.9
HGN-large Fang et al. 2020	82.2	88.5	74.2

Table 9: HotpotQA results in distractor setting test set. Quark’s test results are not available. All numbers are F1 scores. † shows contemporaneous leaderboard submissions.

模型	答案	支持	联合
TAP 2（集成）（Glaß et al. 2019）	79.8	86.7	70.7
SAE（Tu et al. 2019）	79.6	86.7	71.4
Quark（dev）（Groeneveld et al. 2020）	81.2	87.0	72.3
C2F Reader（Shao et al. 2020）	81.2	87.6	72.8
Longformer-large	81.3	88.3	73.2
ETC-large*†*（Ainslie et al. 2020）	81.2	89.1	73.6
GSAN-large*†*	81.6	88.7	73.9
HGN-large（Fang et al. 2020）	82.2	88.5	74.2

表 9：HotpotQA 干扰项设定测试集结果。Quark 无测试结果。数字均为 F1。† 表示同期排行榜提交。

6.5 Ablations on WikiHop

6.5 WikiHop 上的消融

Model	Accuracy / $\Delta$
Longformer (seqlen: 4,096)	73.8
[0.2pt/0.4pt] RoBERTa-base (seqlen: 512)	72.4 / -1.4
Longformer (seqlen: 4,096, 15 epochs)	75.0 / +1.2
Longformer (seqlen: 512, attention: $n^{2}$)	71.7 / -2.1
Longformer (seqlen: 2,048)	73.1 / -0.7
Longformer (no MLM pretraining)	73.2 / -0.6
Longformer (no linear proj.)	72.2 / -1.6
Longformer (no linear proj. no global atten.)	65.5 / -8.3
Longformer (pretrain extra position embed. only)	73.5 / -0.3

Table 10: WikiHop development set ablations

模型	准确率 / $\Delta$
Longformer（seqlen: 4,096）	73.8
RoBERTa-base（seqlen: 512）	72.4 / -1.4
Longformer（seqlen: 4,096，15 epochs）	75.0 / +1.2
Longformer（seqlen: 512，注意力: $n^{2}$）	71.7 / -2.1
Longformer（seqlen: 2,048）	73.1 / -0.7
Longformer（无 MLM 预训练）	73.2 / -0.6
Longformer（无线性投影）	72.2 / -1.6
Longformer（无线性投影、无全局注意力）	65.5 / -8.3
Longformer（仅预训练额外位置嵌入）	73.5 / -0.3

表 10：WikiHop 开发集消融

Tab. 10 presents an ablation study for WikiHop on the development set. All results use Longformer-base, fine-tuned for five epochs with identical hyperparameters except where noted. Longformer benefits from longer sequences, global attention, separate projection matrices for global attention, MLM pretraining, and longer training. In addition, when configured as in RoBERTa-base (seqlen: 512, and $n^{2}$ attention) Longformer performs slightly worse then RoBERTa-base, confirming that performance gains are not due to additional pretraining. Performance drops slightly when using the RoBERTa model pretrained when only unfreezing the additional position embeddings, showing that Longformer can learn to use long range context in task specific fine-tuning with large training datasets such as WikiHop.

表 10 给出 WikiHop 开发集上的消融。除非另有说明，全部结果使用 Longformer-base，以相同超参微调五轮。Longformer 受益于更长序列、全局注意力、全局注意力的独立投影矩阵、MLM 预训练与更长训练。此外，当配置成与 RoBERTa-base 相同（seqlen: 512，且 $n^{2}$ 注意力）时，Longformer 略逊于 RoBERTa-base，确认提升并非来自额外预训练。仅解冻额外位置嵌入的 RoBERTa 预训练配置下表现略降，说明在 WikiHop 这类大规模训练数据上，Longformer 能在任务微调中学会利用长程上下文。

7 Longformer-Encoder-Decoder (LED)

7 Longformer-Encoder-Decoder（LED）

The original Transformer Vaswani et al. 2017 consisted of an encoder-decoder architecture, intended for sequence-to-sequence tasks Sutskever et al. 2014, such as summarization and translation. While encoder-only Transformers are effective on a variety of NLP tasks, pre-trained encoder-decoder Transformer models (e.g. BART Lewis et al. 2020 and T5 Raffel et al. 2020) have achieved strong results on tasks like summarization. Yet, such models can’t efficiently scale to seq2seq tasks with longer inputs.

原始 Transformer（Vaswani et al. 2017）采用编码器–解码器结构，面向摘要与翻译等序列到序列任务（Sutskever et al. 2014）。虽说仅编码器 Transformer 在多种 NLP 任务上有效，预训练的编码器–解码器 Transformer（如 BART Lewis et al. 2020 与 T5 Raffel et al. 2020）在摘要等任务上取得强结果。但这类模型难以高效扩展到输入更长的 seq2seq 任务。

To facilitate modeling long sequences for seq2seq learning, we propose a Longformer variant that has both the encoder and decoder Transformer stacks but instead of the full self-attention in the encoder, it uses the efficient local+global attention pattern of the Longformer. The decoder uses the full self-attention to the entire encoded tokens and to previously decoded locations. We call this model Longformer-Encoder-Decoder (LED) which scales linearly with the input. Since pre-training LED is expensive, we initialize LED parameters from the BART, and follow BART’s exact architecture in terms of number of layers and hidden sizes. The only difference is that to process longer inputs, we extend position embedding to 16K tokens (up from BART’s 1K tokens) and we initialize the new position embedding matrix by repeatedly copying BART’s 1K position embeddings 16 times as in Section 5 for RoBERTa. Following BART, we release two model sizes, LED-base and LED-large, which respectively have 6 and 12 layers in both encoder and decoder stacks.

为便于对长序列做 seq2seq 学习，我们提出同时具有编码器与解码器 Transformer 栈的 Longformer 变体：编码器不用完整自注意力，而用 Longformer 的高效局部+全局注意力；解码器则对全部编码 token 与已解码位置使用完整自注意力。我们称该模型为 Longformer-Encoder-Decoder（LED），其复杂度随输入线性扩展。由于预训练 LED 昂贵，我们从 BART 初始化 LED 参数，并在层数与隐藏维度上严格沿袭 BART 结构。唯一差异是：为处理更长输入，我们把位置嵌入扩展到 16K token（相对 BART 的 1K），并像第 5 节对 RoBERTa 那样，把 BART 的 1K 位置嵌入重复复制 16 次以初始化新矩阵。沿袭 BART，我们发布 LED-base 与 LED-large 两种规模，编码器与解码器栈分别有 6 层与 12 层。

We evaluate LED on the summarization task using the arXiv summarization dataset Cohan et al. 2018 which focuses on long document summarization in the scientific domain. The 90th percentile of document lengths is 14.5K tokens, making it an appropriate testbed for evaluating LED. LED’s encoder reads the document and its decoder generates the output summary. The encoder uses local attention with window size 1,024 tokens and global attention on the first token. The decoder uses full attention to the entire encoder and previously decoded locations. As standard in seq2seq models, LED is trained using teacher forcing on gold training summaries and uses beam search at inference.

我们在面向科学领域长文档摘要的 arXiv 摘要数据集（Cohan et al. 2018）上评估 LED。文档长度的第 90 百分位为 14.5K token，适合作为评估 LED 的测试床。LED 编码器读入文档，解码器生成摘要。编码器使用窗口大小 1,024 的局部注意力，并对首个 token 使用全局注意力；解码器对整个编码器输出与已解码位置使用完整注意力。与标准 seq2seq 一样，LED 在金标训练摘要上用 teacher forcing 训练，推理时用束搜索。

	R-1	R-2	R-L
Discourse-aware (Cohan et al. 2018)	35.80	11.05	31.80
Extr-Abst-TLM (Subramanian et al. 2020)	41.62	14.69	38.03
Dancer (Gidiotis and Tsoumakas 2020)	42.70	16.54	38.44
Pegasus (Zhang et al. 2020)	44.21	16.95	38.83
LED-large (seqlen: 4,096) (ours)	44.40	17.94	39.76
BigBird (seqlen: 4,096) (Zaheer et al. 2020)	46.63	19.02	41.77
[.4pt/1pt] LED-large (seqlen: 16,384) (ours)	46.63	19.62	41.83

Table 11: Summarization results of Longformer-Encoder-Decoder (LED) on the arXiv dataset. Metrics from left to right are ROUGE-1, ROUGE-2 and ROUGE-L.

	R-1	R-2	R-L
Discourse-aware（Cohan et al. 2018）	35.80	11.05	31.80
Extr-Abst-TLM（Subramanian et al. 2020）	41.62	14.69	38.03
Dancer（Gidiotis and Tsoumakas 2020）	42.70	16.54	38.44
Pegasus（Zhang et al. 2020）	44.21	16.95	38.83
LED-large（seqlen: 4,096）（本文）	44.40	17.94	39.76
BigBird（seqlen: 4,096）（Zaheer et al. 2020）	46.63	19.02	41.77
LED-large（seqlen: 16,384）（本文）	46.63	19.62	41.83

表 11：Longformer-Encoder-Decoder（LED）在 arXiv 数据集上的摘要结果。自左至右指标为 ROUGE-1、ROUGE-2 与 ROUGE-L。

Figure 3: ROUGE-1 and ROUGE-2 of LED when varying the input size (arXiv validation set).

图 3：改变输入长度时 LED 的 ROUGE-1 与 ROUGE-2（arXiv 验证集）。

Tab. 11 demonstrates the results of LED-large 16K on the arXiv summarization task. This model is merely initialized from BART, with no additional pre-training. We observe that LED achieves state-of-the-art results on arXiv, slightly outperforming BigBird Zaheer et al. 2020. Note that the BigBird summarization model supports sequence length of 4K tokens but starts from and continues pre-training Pegasus Zhang et al. 2020, a model specifically designed and pre-trained for summarization. With no pre-training or task-specific initialization, but with ability to process longer inputs, LED can slightly outperform BigBird. Further improvements should be possible through pre-training of LED. Fig. 3 further illustrates the importance of sequence length showing the ablility to process longer input significantly improves the results.

表 11 给出 LED-large 16K 在 arXiv 摘要任务上的结果。该模型仅从 BART 初始化，无额外预训练。我们观察到 LED 在 arXiv 上取得当时最优，略优于 BigBird（Zaheer et al. 2020）。注意 BigBird 摘要模型支持 4K token 序列长度，但自专为摘要设计并预训练的 Pegasus（Zhang et al. 2020）起步并继续预训练。在无预训练或任务专用初始化、却能处理更长输入的情况下，LED 仍可略胜 BigBird。进一步预训练 LED 应能带来更多提升。图 3 进一步说明序列长度的重要性：能处理更长输入显著改善结果。

8 Conclusion and Future Work

8 结论与未来工作

We present Longformer, a transformer-based model that is scalable for processing long documents and that makes it easy to perform a wide range of document-level NLP tasks without chunking/shortening the long input and without complex architecture to combine information across these chunks. Longformer employs an attention pattern that combines local and global information while also scaling linearly with the sequence length. Longformer achieves state-of-the-art results on the character-level language modeling tasks of text8 and enwik8. When pretrained, Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We further present LED, an encoder-decoder variant of Longformer for modeling sequence-to-sequence tasks, and achieve state-of-the-art results on the arXiv long document summarization task. For future work, we would like to study other pretraining objectives, especially for LED, increase the sequence length, and explore other tasks that might benefit from our model.

我们提出 Longformer：一种可扩展以处理长文档的基于 Transformer 的模型，使广泛文档级 NLP 任务无需切分/缩短长输入，也无需用复杂结构跨块汇总信息。Longformer 采用结合局部与全局信息、并随序列长度线性扩展的注意力模式。Longformer 在 text8 与 enwik8 字符级语言建模上取得当时最优；预训练后在长文档任务上稳定优于 RoBERTa，并在 WikiHop 与 TriviaQA 上刷新纪录。我们进一步提出面向序列到序列任务的编码器–解码器变体 LED，并在 arXiv 长文档摘要上取得当时最优。未来工作中，我们希望研究其他预训练目标（尤其对 LED）、增大序列长度，并探索可能受益于本模型的其他任务。

Acknowledgment

致谢

We would like to thank Noah Smith, Dan Weld, Dirk Groeneveld, Kyle Lo, Daniel King and Doug Downey for helpful discussions and feedback, and the AI2 infrastructure team for technical support.

感谢 Noah Smith、Dan Weld、Dirk Groeneveld、Kyle Lo、Daniel King 与 Doug Downey 的有益讨论与反馈，以及 AI2 基础设施团队的技术支持。

References

参考文献

Ainslie et al. (2020) Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 268–284, Online. Association for Computational Linguistics.

Al-Rfou et al. (2018) Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2018. Character-level language modeling with deeper self-attention. In AAAI .

Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL .

Chen et al. (2019) Jifan Chen, Shih-Ting Lin, and Greg Durrett. 2019. Multi-hop question answering via reasoning chains. arXiv preprint , abs/1910.02610.

Chen et al. (2018) Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In OSDI .

Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint , abs/1604.06174.

Child et al. (2019) Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint , abs/1904.10509.

Clark and Gardner (2017) Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. In ACL .

Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does bert look at? an analysis of bert’s attention. arXiv preprint , abs/1906.04341.

Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In NAACL-HLT 2018 .

Dai and Le (2015) Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NeurIPS .

Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL .

Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT .

Fang et al. (2020) Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 8823–8838, Online. Association for Computational Linguistics.

Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In MRQA workshop at EMNLP .

Gidiotis and Tsoumakas (2020) Alexios Gidiotis and Grigorios Tsoumakas. 2020. A divide-and-conquer approach to the summarization of academic articles. ArXiv , abs/2004.06190.

Glaß et al. (2019) Michael Glaß, Alfio Massimiliano Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, Gaudani Bhargav, Dinesh Garg, and Avirup Sil. 2019. Span selection pre-training for question answering. arXiv preprint , abs/1909.04120.

Gray et al. (2017) Scott Gray, Alec Radford, and Diederik P. Kingma. 2017. Gpu kernels for block-sparse weights.

Groeneveld et al. (2020) Dirk Groeneveld, Tushar Khot, Mausam, and Ashish Sabhwaral. 2020. A simple yet strong pipeline for HotpotQA. arXiv preprint , abs/2004.06753.

Gupta and Berant (2020) Ankit Gupta and Jonathan Berant. 2020. Gmat: Global memory augmentation for transformers. ArXiv , abs/2006.03274.

Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL .

Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL .

Joshi et al. (2019) Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In EMNLP-IJCNLP .

Kiesel et al. (2019) Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval-2019 task 4: Hyperpartisan news detection . In Proceedings of the 13th International Workshop on Semantic Evaluation , pages 829–839, Minneapolis, Minnesota, USA. Association for Computational Linguistics.

Kipf and Welling (2017) Thomas N Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. ICLR .

Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In ICLR .

Kovaleva et al. (2019) Olga V. Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. In EMNLP/IJCNLP .

Lee et al. (2018) Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In NAACL .

Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7871–7880, Online. Association for Computational Linguistics.

Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint , abs/1907.11692.

Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.

Mahoney (2009) Matt Mahoney. 2009. Large text compression benchmark.

van den Oord et al. (2016) Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. In SSW .

Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations .

Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL .

Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes . In Joint Conference on EMNLP and CoNLL - Shared Task , pages 1–40, Jeju Island, Korea. Association for Computational Linguistics.

Qiu et al. (2019) Jiezhong Qiu, Hao Ma, Omer Levy, Scott Yih, Sinong Wang, and Jie Tang. 2019. Blockwise self-attention for long document understanding. arXiv preprint , abs/1911.02972.

Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In ICLR .

Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1–140:67.

Roy et al. (2020) Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient content-based sparse attention with routing transformers. arXiv preprint , abs/2003.05997.

Shao et al. (2020) Nan Shao, Yiming Cui, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Is graph structure necessary for multi-hop reasoning? arXiv preprint , abs/2004.03096.

Subramanian et al. (2020) Sandeep Subramanian, Raymond Li, Jonathan Pilault, and C. Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In EMNLP .

Sukhbaatar et al. (2019) Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. In ACL .

Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS .

Trinh and Le (2018) Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. arXiv preprint , abs/1806.02847.

Tu et al. (2020) Ming Tu, Jinke Huang, Xiaodong He, and Bowen Zhou. 2020. Graph sequential network for reasoning over sequences. In NeurIPS Graph Representation Learning workshop .

Tu et al. (2019) Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bufang Zhou. 2019. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. arXiv preprint , abs/1911.00484.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS .

Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. TACL , 6:287–302.

Wu et al. (2019) Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. arXiv preprint , abs/1901.10430.

Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint , abs/1904.12848.

Xiong et al. (2020) Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shu xin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Li-Wei Wang, and Tie-Yan Liu. 2020. On layer normalization in the transformer architecture. arXiv preprint , abs/2002.04745.

Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP .

Ye et al. (2019) Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. 2019. BP-Transformer: Modelling long-range context via binary partitioning. arXiv preprint , abs/1911.04070.

Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, C. Alberti, S. Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, L. Yang, and A. Ahmed. 2020. Big bird: Transformers for longer sequences. ArXiv , abs/2007.14062.

Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS .

Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. ICML .

Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV , pages 19–27.

Appendix A Implementation Details

附录 A 实现细节

Implementing Longformer’s dilated sliding window attention requires a form of banded matrix multiplication (matrix multiplication where the output is all zero except certain diagonals) that is not directly supported in existing deep learning libraries like PyTorch/Tensorflow. Fig. 1 compares the runtime and memory of three different ways of implementing it. Longformer-loop is a naive implementation that computes each diagonal separately in a loop. It is memory efficient because it only computes the non-zero values, but it is unusably slow. We only use it for testing because it is easy to implement but don’t use it to run experiments. Longformer-chunks only supports the non-dilated case. It chunks $Q$ and $K$ into overlapping blocks of size $w$ and overlap of size $\frac{1}{2}w$, multiplies the blocks, then mask out the diagonals. This is very compute efficient because it uses a single matrix multiplication operation from PyTorch, but it consumes 2x the amount of memory a perfectly optimized implementation should consume because it computes some of the zero values. Because of the compute efficiency, this implementation is most suitable for the pretrain/finetune case. We didn’t find the increase in memory to be a problem for this setting. Longformer-cuda is a custom CUDA kernel that we implement using TVM Chen et al. 2018. It is a fully functioning implementation of our attention (not limited as Longformer-chunks), it is the most memory efficient, and it is as fast as the highly optimized full self-attention.[^10] We mainly use this implementation for the autoregressive language modeling experiments because of the memory efficiency (allows the longest sequences) and the support of dilation (needed for character-LM experiments).

实现 Longformer 的扩张滑动窗口注意力需要一种带状矩阵乘法（输出除若干对角线外全为零的矩阵乘法），现有深度学习库如 PyTorch/Tensorflow 并不直接支持。图 1 比较了三种实现的运行时间与内存。Longformer-loop 是朴素实现：在循环中分别计算每条对角线。它只计算非零值因而内存高效，但慢到几乎不可用。我们仅因其易实现而用于测试，实验中不用。Longformer-chunks 仅支持非扩张情形：把 $Q$ 与 $K$ 切成大小为 $w$、重叠 $\frac{1}{2}w$ 的重叠块，做块乘法后再掩掉对角线。它用单次 PyTorch 矩阵乘法，计算很高效，但因会算部分零值，内存约为完美优化实现的 2 倍。鉴于计算效率，该实现最适合预训练/微调；我们未发现此设定下内存增加成问题。Longformer-cuda 是我们用 TVM（Chen et al. 2018）实现的自定义 CUDA 核：功能完整（不受 Longformer-chunks 的限制），内存最省，速度可与高度优化的完整自注意力相当。[^10]我们主要在自回归语言建模实验中使用它，因其内存高效（允许最长序列）且支持扩张（字符 LM 实验需要）。

Tensor Virtual Machine (TVM)

We build our custom CUDA kernel using TVM Chen et al. 2018, a deep learning compiler stack that compiles high level description of a function into optimized device-specific code. Using TVM, we describe our banded matrix multiplication in high-level python constructs, then TVM generates the corresponding CUDA code and compiles it for GPUs.

Tensor Virtual Machine（TVM）

我们用 TVM（Chen et al. 2018）构建自定义 CUDA 核。TVM 是把函数高层描述编译为针对设备优化代码的深度学习编译栈。用 TVM 时，我们以高层 Python 构造描述带状矩阵乘法，再由 TVM 生成对应 CUDA 代码并为 GPU 编译。

Appendix B Character LM Hyperparameters

附录 B 字符 LM 超参数

We evaluate on text8 and enwik8, both contain 100M characters from Wikipedia split into 90M, 5M, 5M for train, dev, test. Our model only specifies how the self-attention component works, and it is agnostic to the other design choices for the transformer model. Our implementation is based on the Transformer-XL Dai et al. 2019 code[^11] with the memory mechanism disabled. We use relative position embeddings with sinusoidal weights as in Dai et al. 2019. We use two different model sizes, a small (12 layers, 512 hidden size) model as in Dai et al. 2019, and a large (30 layers, 512 hidden size) model as in Child et al. 2019. We employed mixed precision training (floating points 16 and 32) using apex[^12] to reduce memory consumption and speed-up training. However, we kept the attention computation in fp32 to avoid numerical instability issues.[^13] We used gradient checkpointing Chen et al. 2016 to reduce memory usage, and ran our experiments on 48GB RTX8000 GPUs. All hyperparameters and stage configurations are listed in Tab. 12. Our CUDA kernel supports the autoregressive mode where each token attends to a window of previous tokens only. Our implementation also includes a version of the relative position embedding that is compatible with our dilated sliding window attention.

我们在 text8 与 enwik8 上评估，二者均含来自维基百科的 1 亿字符，按 90M / 5M / 5M 划分训练/开发/测试。我们的模型只规定自注意力如何工作，与 Transformer 其他设计选择无关。实现基于 Transformer-XL（Dai et al. 2019）代码[^11]，并关闭记忆机制。我们使用与 Dai et al. 2019 相同的带正弦权重的相对位置嵌入。模型规模有两种：小模型（12 层、隐藏维度 512，同 Dai et al. 2019）与大模型（30 层、隐藏维度 512，同 Child et al. 2019）。我们用 apex[^12] 做混合精度训练（fp16 与 fp32）以降低内存并加速，但把注意力计算保持在 fp32 以避免数值不稳。[^13]我们用梯度检查点（Chen et al. 2016）降低内存，并在 48GB RTX8000 GPU 上实验。全部超参与阶段配置见表 12。我们的 CUDA 核支持自回归模式：每个 token 仅关注此前窗口内的 token。实现中还包含与扩张滑动窗口注意力兼容的相对位置嵌入版本。

We ran the small model experiments on 4 RTX8000 GPUs for 16 days. For the large model, we ran experiments on 8 RTX8000 GPUs for 13 days. Most of our hyperparameter search is similar to the ablation in Tab. 4 where we run the configuration for 150K steps on text8. We experimented with absolute position embeddings and learned position embeddings, dropout values of [0.1, 0.2] (small model) and [0.1, 0.4] (large model), pre-layernorm and post-layernorm Xiong et al. 2020, learning rate (LR) of phase1 of values [2.5e-5, 5e-4, 1e-4] constant and cosine LR schedules, and different configurations for dilation (on all heads, on 2 heads, no dilation). Number of gradient updates/phase reported in Tab. 12 is determined by running each phase until the validation BPC stops getting better.

小模型实验在 4 块 RTX8000 上跑 16 天；大模型在 8 块 RTX8000 上跑 13 天。多数超参搜索类似表 4 的消融：在 text8 上跑 150K 步。我们试过绝对位置嵌入与学习位置嵌入、dropout 取值 [0.1, 0.2]（小模型）与 [0.1, 0.4]（大模型）、pre-layernorm 与 post-layernorm（Xiong et al. 2020）、阶段 1 学习率 [2.5e-5, 5e-4, 1e-4]、常数与余弦 LR 日程，以及不同扩张配置（全部头、2 个头、无扩张）。表 12 中每阶段梯度更新数由各阶段验证 BPC 不再提升时确定。

Param	Value
Position Embeddings	Relative and Sinusoidal as in Dai et al. 2019
Small model config	12 layers, 8 heads, 512 hidden size as in Dai et al. 2019
Large model config	30 layers, 8 heads, 512 hidden size as in Child et al. 2019
Optimizer	AdamW
Dropout	0.2 (small model), 0.4 (large model)
Gradient clipping	0.25
Weight Decay	0.01
Layernorm Location	pre-layernorm Xiong et al. 2020
Activation	GeLU
Number of phases	5
Phase 1 window sizes	32 (bottom layer) - 8,192 (top layer)
Phase 5 window sizes	512 (bottom layer) - (top layer)
Phase 1 sequence length	2,048
Phase 5 sequence length	23,040 (gpu memory limit)
Phase 1 LR	0.00025
Phase 5 LR	000015625
Batch size per phase	32, 32, 16, 16, 16
#Steps per phase (small)	430K, 50k, 50k, 35k, 5k
#Steps per phase (large)	350K, 25k, 10k, 5k, 5k
Warmup	10% of the phase steps with maximum 10K steps
LR scheduler	constant throughout each phase
Dilation (small model)	0 (layers 0-5), 1 (layers 6-7), 2 (layers 8-9), 3 (layers 10-11)
Dilation (large model)	0 (layers 0-14), 1 (layers 15-19), 2 (layers 20-24), 3 (layers 25-29)
Dilation heads	2 heads only

Table 12: Hyperparameters for the best performing model for character-level language modeling

参数	取值
位置嵌入	相对且正弦，同 Dai et al. 2019
小模型配置	12 层、8 头、隐藏 512，同 Dai et al. 2019
大模型配置	30 层、8 头、隐藏 512，同 Child et al. 2019
优化器	AdamW
Dropout	0.2（小模型），0.4（大模型）
梯度裁剪	0.25
权重衰减	0.01
Layernorm 位置	pre-layernorm（Xiong et al. 2020）
激活	GeLU
阶段数	5
阶段 1 窗口大小	32（底层）– 8,192（顶层）
阶段 5 窗口大小	512（底层）–（顶层）
阶段 1 序列长度	2,048
阶段 5 序列长度	23,040（GPU 内存上限）
阶段 1 LR	0.00025
阶段 5 LR	0.000015625
每阶段批大小	32, 32, 16, 16, 16
每阶段步数（小）	430K, 50k, 50k, 35k, 5k
每阶段步数（大）	350K, 25k, 10k, 5k, 5k
预热	该阶段步数的 10%，最多 10K 步
LR 调度	每阶段内恒定
扩张（小模型）	0（层 0–5），1（层 6–7），2（层 8–9），3（层 10–11）
扩张（大模型）	0（层 0–14），1（层 15–19），2（层 20–24），3（层 25–29）
扩张头	仅 2 个头

表 12：字符级语言建模最佳模型超参数

Appendix C Pretraining Data

附录 C 预训练数据

Source	Tokens	Avg doc len
Books Zhu et al. 2015	0.5B	95.9K
English Wikipedia	2.1B	506
Realnews Zellers et al. 2019	1.8B	1.7K
Stories Trinh and Le 2018	2.1B	7.8K

Table 13: Pretraining data

来源	Token 数	平均文档长
Books（Zhu et al. 2015）	0.5B	95.9K
英文维基百科	2.1B	506
Realnews（Zellers et al. 2019）	1.8B	1.7K
Stories（Trinh and Le 2018）	2.1B	7.8K

表 13：预训练数据

In order to allow the model to learn long dependencies in pretraining, we compiled a corpus of long documents. Some of these data sources were also included in the original RoBERTa pretraining including the Books corpus Zhu et al. 2015 plus English Wikipedia. We additionally included one third of a subset of the Realnews dataset Zellers et al. 2019 with documents longer than 1,200 tokens as well as one third of the Stories Trinh and Le 2018 corpus. Our goal was to include a mix of long and short documents to both allow the model to learn longer dependencies while not to forget information from the original RoBERTa pretraining. The statistics of the pretraining data is shown in Tab. 13.

为使模型在预训练中学习长依赖，我们汇编了长文档语料。其中部分来源也包含在原始 RoBERTa 预训练中，包括 Books 语料（Zhu et al. 2015）与英文维基百科。我们额外纳入 Realnews（Zellers et al. 2019）中长度超过 1,200 token 的子集的三分之一，以及 Stories（Trinh and Le 2018）语料的三分之一。目标是混合长短文档，既让模型学习更长依赖，又不遗忘原始 RoBERTa 预训练信息。预训练数据统计见表 13。

Appendix D Task specific model details

附录 D 任务专用模型细节

All the QA and classification models are implemented using PyTorch-Lightning[^14]. We use the official train/dev/test splits of all datasets except for the Hyperpartisan news which we randomely split into 80/10/10 for train/dev/test.

全部问答与分类模型用 PyTorch-Lightning[^14] 实现。除 Hyperpartisan 新闻外，我们使用各数据集官方 train/dev/test 划分；Hyperpartisan 随机按 80/10/10 划分。

WikiHop

Instances in WikiHop consist of: a question, answer candidates (ranging from two candidates to 79 candidates), supporting contexts (ranging from three paragraphs to 63 paragraphs), and the correct answer. The dataset does not provide any intermediate annotation for the multihop reasoning chains, requiring models to instead infer them from the indirect answer supervision.

WikiHop

WikiHop 实例包含：一个问题、答案候选（2 到 79 个）、支持上下文（3 到 63 段）以及正确答案。数据集不提供多跳推理链的中间标注，模型须从间接的答案监督中推断。

To prepare the data for input to Longformer and RoBERTa, we first tokenize the question, answer candidates, and support contexts using RoBERTa’s wordpiece tokenizer. Then we concatenate the question and answer candidates with special tokens as [q] question [/q] [ent] candidate1 [/ent] ... [ent] candidateN [/ent]. The contexts are also concatenated using RoBERTa’s document delimiter tokens as separators: context1 ... contextM . The special tokens [q], [/q], [ent], [/ent] were added to the RoBERTa vocabulary and randomly initialized before task finetuning.

为准备 Longformer 与 RoBERTa 的输入，我们先用 RoBERTa 的 wordpiece 分词器对问题、答案候选与支持上下文分词。再把问题与答案候选用特殊 token 拼成 [q] question [/q] [ent] candidate1 [/ent] ... [ent] candidateN [/ent]。上下文也用 RoBERTa 的文档分隔 token 拼接： context1 ... contextM 。特殊 token [q]、[/q]、[ent]、[/ent] 加入 RoBERTa 词表，并在任务微调前随机初始化。

After preparing the input data, we compute activations from the top layer of each model as follows. We take the question and answer candidates and concatenate them to as much context as possible up to the model sequence length (512 for RoBERTa, 4,096 for Longformer), run the sequence through the model, collect the output activations, and repeat until all of the context is exhausted (for all models except Longformer-large, where we just include the first 4,096 length sequence due to memory requirements). Then all activations for all chunks are concatenated into one long sequence. In the case of Longformer, we use global attention to the entire question and answer candidate sequence.

准备好输入后，我们按如下方式计算各模型顶层激活：把问题与答案候选拼上尽可能多的上下文直至模型序列长度（RoBERTa 为 512，Longformer 为 4,096），跑通模型、收集输出激活，并重复直至上下文耗尽（Longformer-large 除外：因内存限制只取第一条长度 4,096 的序列）。再把各块激活拼成一条长序列。对 Longformer，我们对整段问题与答案候选序列使用全局注意力。

For prediction, we attach a linear layer to each [ent] that outputs a single logit, average over all logits for each candidate across the chunks, apply a softmax and use the cross entropy loss with the correct answer candidate.

预测时，我们在每个 [ent] 上接线性层输出单个 logit，对每个候选跨块平均全部 logit，再经 softmax，并以正确答案候选的交叉熵为损失。

Training used the Adam optimizer with linear warmup over 200 gradient updates to a maximum LR, and linear decay over the remainder of training. We used gradient accumulation to effective batch size of 32 instances, checking the development accuracy every 250 gradient updates and reported the maximum development accuracy. Other hyperparameters (dropout, weight decay) were identical to RoBERTa pretraining.

训练使用 Adam 优化器：在 200 次梯度更新内线性预热到最大学习率，再在剩余训练中线性衰减。我们用梯度累积使有效批大小为 32，每 250 次梯度更新检查开发准确率，并报告最高开发准确率。其余超参（dropout、权重衰减）与 RoBERTa 预训练相同。

In general, we ran minimal hyperparameter trials, but for fair comparison between Longformer and RoBERTa ran an identical hyperparameter search with Longformer-base and RoBERTa-base. This consisted of a grid search of LR in [2e-5, 3e-5, 5e-5] and number epochs in [5, 10, 15]. The best Longformer-base configuration used lr=3e-5, 15 epochs. We ran two hyperparameter trials for Longformer-large, lr=3e-5 and number epochs in [5, 15] (the 5 epoch model had higher dev accuracy of 77.6, and was the single model submitted to the public leaderboard for test set evaluation). All models were trained on a single RTX8000 GPU, with Longformer-base taking about a day for 5 epochs.

总体超参试验很少；但为公平比较 Longformer 与 RoBERTa，我们对 Longformer-base 与 RoBERTa-base 做了相同超参搜索：学习率在 [2e-5, 3e-5, 5e-5]、轮数在 [5, 10, 15] 的网格搜索。最佳 Longformer-base 配置为 lr=3e-5、15 轮。对 Longformer-large 做了两次试验：lr=3e-5，轮数取 [5, 15]（5 轮模型开发准确率更高，为 77.6，并作为提交公开排行榜测试评估的单一模型）。全部模型在单块 RTX8000 上训练，Longformer-base 约一天完成 5 轮。

TriviaQA

TriviaQA has more than 100K question, answer, document triplets for training. Documents are Wikipedia articles, and answers are named entities mentioned in the article. The span that answers the question is not annotated, but it is found using simple text matching.

TriviaQA

TriviaQA 训练集约有超过 10 万条问题–答案–文档三元组。文档为维基百科文章，答案为文中出现的命名实体。回答问题的跨度未标注，而是用简单文本匹配找到。

Similar to WikiHop, we tokenize the question and the document using RoBERTa’s tokenizer, then form the input as [s] question [/s] document [/s]. We truncate the document at 4,096 wordpiece to avoid it being very slow. Afterwards, we get the activations from RoBERTa and Longformer similar to WikiHop (discussed above). We use global attention on all question tokens.

与 WikiHop 类似，我们用 RoBERTa 分词器对问题与文档分词，再构成输入 [s] question [/s] document [/s]。为避免过慢，文档截断到 4,096 wordpieces。之后按与 WikiHop 相同方式（见上）从 RoBERTa 与 Longformer 取激活。我们对全部问题 token 使用全局注意力。

For prediction, we add one layer that predicts the beginning and end of the answer span. Because of the distant supervision nature of the training data (no gold answer spans), we use the loss function of Clark and Gardner 2017 which works like an OR that the model only needs to get one answer span right, not all of them.

预测时加入一层，预测答案跨度起止。由于训练数据是弱监督（无金标答案跨度），我们使用 Clark and Gardner 2017 的损失函数：类似 OR，模型只需答对其中一个答案跨度即可。

Hyperparameters of the best configuration are listed in Tab. 14. All other hyperparameters are similar to RoBERTa’s. For hyperparameter search, we only tuned LR for the RoBERTa baseline and tried rates [3e-5, 5e-5, 1e-4], then used the best, which is 3e-5, for all subsequent experiments with no further tuning. We trained the Longformer-large with the best configuration once and submitted its output to the leaderboard. We ran our experiments on 32GB V100 GPUs. Small model takes 1 day to train on 4 GPUs, while large model takes 1 day on 8 GPUs.

最佳配置超参见表 14。其余超参与 RoBERTa 相同。超参搜索中，我们仅调 RoBERTa 基线的学习率，尝试 [3e-5, 5e-5, 1e-4]，取最佳 3e-5 用于后续全部实验且不再调。我们用最佳配置训练一次 Longformer-large 并提交排行榜。实验在 32GB V100 上进行：小模型 4 卡约 1 天，大模型 8 卡约 1 天。

HotpotQA

HotpotQA dataset involves answering questions from a set of 10 paragraphs from 10 different Wikipedia articles where 2 paragraphs are relevant to the question and the rest are distractors. It includes 2 tasks of answer span extraction and evidence sentence identification. Our model for HotpotQA combines both answer span extraction and evidence extraction in one joint model. We found a higher performance using a two-stage Longformer model with similar setup that first identifies relevant paragraphs and then does find the final answer span and evidence.[^15] This is largely because removing the distracting paragraphs first reduces the noise for the final evidence and span detection as also found to be important by recent state-of-the-art methods in this dataset Fang et al. 2020. Similar to Wikihop and TriviaQA, to prepare the data for input to Longformer, we concatenate question and then all the 10 paragraphs in one long context. We particularly use the following input format with special tokens: “[CLS] [q] question [/q] $\langle$t$\rangle$ $\texttt{title}_{\texttt{1}}$ $\langle$/t$\rangle$ $\texttt{sent}_{\texttt{1,1}}$ [s] $\texttt{sent}_{\texttt{1,2}}$ [s] ... $\langle$t$\rangle$ $\texttt{title}_{\texttt{2}}$ $\langle$/t$\rangle$ $\texttt{sent}_{\texttt{2,1}}$ [s] $\texttt{sent}_{\texttt{2,2}}$ [s] ...” where [q], [/q], $\langle$t$\rangle$, $\langle$/t$\rangle$, [s], [p] are special tokens representing, question start and end, paragraph title start and end, and sentence, respectively. The special tokens were added to the Longformer vocabulary and randomly initialized before task finetuning. For Longformer, we use global attention to question tokens, paragraph title start tokens as well as sentence tokens. The model includes additional feedforward layers on top of paragraph title start tokens for prediction of relevant paragraphs, as well as sentence tokens for predicting evidence sentences. After training the first stage model, we predict relevant paragraph scores for both training and development set. We then keep up to 5 paragraphs whose raw score is higher than a pre-specified threshold (-3.0), and remove the other paragraphs from the context. We then train the second stage model on the resulting shortened context. For answer span extraction we use BERT’s QA model Devlin et al. 2019 with addition of a question type (yes/no/span) classification head over the first special token ([CLS]). For evidence extraction we apply 2 layer feedforward networks on top of the representations corresponding to sentence and paragraph tokens to get the corresponding evidence prediction scores and use binary cross entropy loss to train the model. At inference time for evidence extraction, we use a constrained decoding strategy similar to Groeneveld et al. 2020 that ensures that the evidence sentences come from exactly two paragraphs which is the setup of this dataset. We combine span, question classification, sentence, and paragraphs losses and train the model in a multitask way using linear combination of losses. Our experiments are done on RTX8000 GPUs and training each epoch takes approximately half a day on 4 GPUs. We trained the model using Adam optimizer with linear warmup (1000 steps) and linear decay. We used minimal hyperparameter tuning using LRs of 3e-5 and 5e-5 and epochs of 3 to 7 and found the model with LR of 3e-5 and 5 epochs to work best. We conduct the same hyperparameter search for the RoBERTa baseline as well. The rest of hyperparameters are reported in Tab 14.

HotpotQA

HotpotQA 要求从 10 篇不同维基文章的 10 段中回答问题，其中 2 段相关、其余为干扰。任务含答案跨度抽取与证据句识别两项。我们的模型在一个联合模型中结合二者。我们发现：使用设置类似的两阶段 Longformer——先识别相关段落，再找最终答案跨度与证据——表现更好。[^15]这很大程度上是因为先去掉干扰段落可降低最终证据与跨度检测的噪声，近期该数据集上的先进方法也强调这一点（Fang et al. 2020）。与 WikiHop、TriviaQA 类似，我们把问题与全部 10 段拼成一条长上下文，输入格式使用特殊 token：「[CLS] [q] question [/q] ⟨t⟩ title₁ ⟨/t⟩ sent₁,₁ [s] sent₁,₂ [s] ... ⟨t⟩ title₂ ⟨/t⟩ sent₂,₁ [s] sent₂,₂ [s] ...」，其中 [q]、[/q]、⟨t⟩、⟨/t⟩、[s]、[p] 分别表示问题起止、段落标题起止与句子。特殊 token 加入 Longformer 词表并在微调前随机初始化。对 Longformer，我们对问题 token、段落标题起始 token 以及句子 token 使用全局注意力。模型在段落标题起始 token 上另接前馈层以预测相关段落，在句子 token 上预测证据句。第一阶段训练后，我们为训练与开发集预测相关段落分数，保留原始分数高于预设阈值（-3.0）的至多 5 段，并从上下文中去掉其余段落，再在缩短后的上下文上训练第二阶段。答案跨度抽取沿用 BERT 的 QA 模型（Devlin et al. 2019），并在首个特殊 token（[CLS]）上增加问题类型（是/否/跨度）分类头。证据抽取对句子与段落 token 对应表征施加两层前馈网络得到证据预测分数，并以二元交叉熵训练。推理时证据抽取采用类似 Groeneveld et al. 2020 的约束解码，确保证据句恰好来自两段，以符合该数据集设定。我们将跨度、问题分类、句子与段落损失线性组合，以多任务方式训练。实验在 RTX8000 上进行，每轮约需 4 卡半天。用 Adam、线性预热（1000 步）与线性衰减训练。仅做最小超参调优：学习率 3e-5 与 5e-5、轮数 3 到 7，发现 lr=3e-5、5 轮最好。对 RoBERTa 基线做相同搜索。其余超参见表 14。

Param	WikiHop	TriviaQA	HotpotQA
Epochs	15	5	5
LR	3e-5	3e-5	5e-5
Warmup steps	200	1000	1000
Batch size	32	32	32
Optimizer	Adam	Adam	Adam

Table 14: Hyperparameters of the QA models. All models use a similar scheduler with linear warmup and decay.

参数	WikiHop	TriviaQA	HotpotQA
轮数	15	5	5
LR	3e-5	3e-5	5e-5
预热步数	200	1000	1000
批大小	32	32	32
优化器	Adam	Adam	Adam

表 14：QA 模型超参数。全部模型使用带线性预热与衰减的类似调度。

Coreference model details

The coreference model is a straightforward adaptation of the coarse-to-fine BERT based model from Joshi et al. 2019. After preprocessing each document with the RoBERTa wordpiece tokenizer, it splits each document into non-overlapping segments up to the maximum sequence length, then concatenates the activations for the coarse-to-fine clustering stage that forms coreference clusters. The maximum sequence length was 384 for RoBERTa-base, chosen after three trials from [256, 384, 512] using the default hyperparameters in the original implementation.[^16] For Longformer-base the sequence length was 4,096. Similar to the original implementation, different learning rates were used for the pretrained RoBERTa parameters and the randomly initialized task parameters. Using a larger learning rate in the task parameters allows the optimizer to adjust them farther from their randomly initialized values without destroying the information in the pretrained RoBERTa parameters.

共指模型细节

共指模型是对 Joshi et al. 2019 基于 BERT 的 coarse-to-fine 模型的直接适配。用 RoBERTa wordpiece 分词器预处理每篇文档后，把文档切成不超过最大序列长度的非重叠段，再拼接激活进入 coarse-to-fine 聚类以形成共指簇。RoBERTa-base 最大序列长度为 384，在原始实现默认超参下从 [256, 384, 512] 三次试验中选出。[^16] Longformer-base 序列长度为 4,096。与原实现类似，预训练 RoBERTa 参数与随机初始化的任务参数使用不同学习率。任务参数用较大学习率，使优化器能在不破坏预训练 RoBERTa 信息的前提下，把它们推离随机初始化。

Hyperparameter searches were minimal and consisted of grid searches of RoBERTa LR in [1e-5, 2e-5, 3e-5] and task LR in [1e-4, 2e-4, 3e-4] for both RoBERTa and Longformer for a fair comparison. The best configuration for Longformer-base was RoBERTa lr=1e-5, task lr=1e-4. All other hyperparameters were the same as in the original implementation. Training takes about 10 hours on a single GPU.

超参搜索很少，由 RoBERTa 学习率在 [1e-5, 2e-5, 3e-5]、任务学习率在 [1e-4, 2e-4, 3e-4] 的网格搜索组成，对 RoBERTa 与 Longformer 同样进行以公平比较。Longformer-base 最佳配置为 RoBERTa lr=1e-5、任务 lr=1e-4。其余超参与原实现相同。单卡训练约 10 小时。

Our implementation is a superhack that involves PyTorch and Tensorflow sharing a single process and GPU. To avoid re-implementing the complicated coarse-to-fine logic from Tensorflow in PyTorch (that involves a highly optimized custom GPU kernel originally released by Lee et al. 2018), we devised a system where the lower transformer portion of the model passes activations and gradients back and forth between PyTorch and Tensorflow. The input tensors are first run through the transformer in PyTorch, the activations are collected from the top layer, transferred from GPU to CPU then from CPU to Tensorflow and back to GPU to run the coarse-to-fine clustering and compute the loss. Then gradients are back propogated in Tensorflow to the top of the transformer and the process reversed to transfer them to PyTorch for back propogation through the remainder of the model. Separate optimizers are maintained with identical LR schedules for parameter updates. The overhead in this approach is minimal compared to the overall cost of running the model.

我们的实现是一套「超级黑科技」：让 PyTorch 与 Tensorflow 共享同一进程与 GPU。为避免把 Tensorflow 中复杂的 coarse-to-fine 逻辑（含 Lee et al. 2018 发布的高度优化自定义 GPU 核）重写到 PyTorch，我们设计了系统：模型下层 Transformer 部分在 PyTorch 与 Tensorflow 之间来回传递激活与梯度。输入张量先在 PyTorch 中跑过 Transformer，从顶层收集激活，经 GPU→CPU 再转入 Tensorflow 并回到 GPU，以运行 coarse-to-fine 聚类并计算损失；再在 Tensorflow 中反传到 Transformer 顶部，反向把梯度交回 PyTorch，完成模型其余部分的反向传播。参数更新由两个独立优化器维护，学习率日程相同。相对整模开销，该方案额外成本很小。

Text classification

For classification, following BERT, we used a simple binary cross entropy loss on top of a first [CLS] token with addition of global attention to [CLS]. We used Adam optimizer with batch sizes of 32 and linear warmup and decay with warmup steps equal to 0.1 of the total training steps. For both IMDB and Hyperpartisan news we did grid search of LRs [3e-5, 5e-5] and epochs [10, 15, 20] and found the model with [3e-5] and epochs 15 to work best. Experiments were done on a single RTX8000 GPU.

文本分类

对分类，沿袭 BERT，我们在第一个 [CLS] token 上使用简单的二元交叉熵损失，并对 [CLS] 施加全局注意力。使用 Adam 优化器，批大小 32，线性预热与衰减，预热步数为总训练步数的 0.1。对 IMDB 与 Hyperpartisan 新闻，我们在学习率 [3e-5, 5e-5] 与轮数 [10, 15, 20] 上做网格搜索，发现学习率 3e-5、15 轮最好。实验在单块 GPU 上完成。