Unsupervised Cross-lingual Representation Learning at Scale

大规模无监督跨语言表示学习

Alexis Conneau Kartikay Khandelwal 1 1 footnotemark: 1 Thanks:  Equal contribution.    Naman Goyal Vishrav Chaudhary Guillaume Wenzek Francisco Guzmán    Edouard Grave Myle Ott Luke Zettlemoyer Veselin Stoyanov Affiliation:  Facebook AI

Abstract

摘要

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code, data and models publicly available.[^†][^1]

本文表明，大规模预训练多语言语言模型能在广泛的跨语言迁移任务上带来显著提升。我们在一百种语言上训练基于 Transformer 的掩码语言模型，使用超过两太字节经过滤的 CommonCrawl 数据。模型命名为 XLM-R，在多种跨语言基准上显著优于多语言 BERT（mBERT），包括 XNLI 平均准确率 +14.6%、MLQA 平均 F1 +13%，以及 NER 上 F1 +2.4%。XLM-R 在低资源语言上表现尤为突出：相对先前 XLM 模型，斯瓦希里语 XNLI 准确率提升 15.7%，乌尔都语提升 11.4%。我们还详细实证分析了取得这些增益的关键因素，包括（1）正向迁移与容量稀释之间的权衡，以及（2）大规模下高资源与低资源语言的表现。最后我们首次表明，多语言建模可以不牺牲单语言表现；XLM-R 在 GLUE 与 XNLI 上与强单语言模型竞争力相当。代码、数据与模型将公开。[^†][^1]

1 Introduction

1 引言

The goal of this paper is to improve cross-lingual language understanding (XLU), by carefully studying the effects of training unsupervised cross-lingual representations at a very large scale. We present XLM-R a transformer-based multilingual masked language model pre-trained on text in 100 languages, which obtains state-of-the-art performance on cross-lingual classification, sequence labeling and question answering.

本文目标是改进跨语言语言理解（XLU），办法是仔细研究在极大规模下训练无监督跨语言表示的效果。我们提出 XLM-R：在 100 种语言文本上预训练的基于 Transformer 的多语言掩码语言模型，在跨语言分类、序列标注与问答上取得当时最优表现。

Multilingual masked language models (MLM) like mBERT Devlin et al. 2018 and XLM Lample and Conneau 2019 have pushed the state-of-the-art on cross-lingual understanding tasks by jointly pretraining large Transformer models Vaswani et al. 2017 on many languages. These models allow for effective cross-lingual transfer, as seen in a number of benchmarks including cross-lingual natural language inference  Bowman et al. 2015; Williams et al. 2017; Conneau et al. 2018, question answering Rajpurkar et al. 2016; Lewis et al. 2019, and named entity recognition Pires et al. 2019; Wu and Dredze 2019. However, all of these studies pre-train on Wikipedia, which provides a relatively limited scale especially for lower resource languages.

mBERT Devlin et al. 2018 与 XLM Lample and Conneau 2019 一类多语言掩码语言模型（MLM），通过在多种语言上联合预训练大型 Transformer Vaswani et al. 2017，把跨语言理解任务的水平往前推。这些模型支持有效的跨语言迁移，见于跨语言自然语言推断 Bowman et al. 2015; Williams et al. 2017; Conneau et al. 2018、问答 Rajpurkar et al. 2016; Lewis et al. 2019 以及命名实体识别 Pires et al. 2019; Wu and Dredze 2019 等基准。但上述研究都在维基百科上预训练，规模相对有限，对低资源语言尤甚。

In this paper, we first present a comprehensive analysis of the trade-offs and limitations of multilingual language models at scale, inspired by recent monolingual scaling efforts Liu et al. 2019. We measure the trade-off between high-resource and low-resource languages and the impact of language sampling and vocabulary size. The experiments expose a trade-off as we scale the number of languages for a fixed model capacity: more languages leads to better cross-lingual performance on low-resource languages up until a point, after which the overall performance on monolingual and cross-lingual benchmarks degrades. We refer to this tradeoff as the curse of multilinguality, and show that it can be alleviated by simply increasing model capacity. We argue, however, that this remains an important limitation for future XLU systems which may aim to improve performance with more modest computational budgets.

本文首先全面分析大规模多语言语言模型的权衡与局限，灵感来自近期单语言缩放工作 Liu et al. 2019。我们度量高资源与低资源语言之间的权衡，以及语言采样与词表大小的影响。实验揭示：固定模型容量下扩大语言数存在权衡——语言越多，低资源语言的跨语言表现先升后降，最终单语言与跨语言基准整体变差。我们称此为多语言诅咒（curse of multilinguality），并表明简单增大模型容量可缓解。但我们认为，对未来希望以更温和算力预算提升表现的 XLU 系统而言，这仍是重要限制。

Our best model XLM-RoBERTa (XLM-R) outperforms mBERT on cross-lingual classification by up to 23% accuracy on low-resource languages. It outperforms the previous state of the art by 5.1% average accuracy on XNLI, 2.42% average F1-score on Named Entity Recognition, and 9.1% average F1-score on cross-lingual Question Answering. We also evaluate monolingual fine tuning on the GLUE and XNLI benchmarks, where XLM-R obtains results competitive with state-of-the-art monolingual models, including RoBERTa Liu et al. 2019. These results demonstrate, for the first time, that it is possible to have a single large model for all languages, without sacrificing per-language performance. We will make our code, models and data publicly available, with the hope that this will help research in multilingual NLP and low-resource language understanding.

我们最好的模型 XLM-RoBERTa（XLM-R）在跨语言分类上相对 mBERT 最高可提升 23% 准确率（低资源语言）。相对先前最优，XNLI 平均准确率高 5.1%，命名实体识别平均 F1 高 2.42%，跨语言问答平均 F1 高 9.1%。我们也在 GLUE 与 XNLI 上评估单语言微调，XLM-R 与包括 RoBERTa Liu et al. 2019 在内的当时最优单语言模型竞争力相当。这些结果首次表明：可以有一个覆盖所有语言的大模型，而不牺牲各语言表现。代码、模型与数据将公开，以期推动多语言 NLP 与低资源语言理解研究。

2 Related Work

2 相关工作

From pretrained word embeddings (Mikolov et al. 2013b; Pennington et al. 2014) to pretrained contextualized representations (Peters et al. 2018; Schuster et al. 2019) and transformer based language models (Radford et al. 2018; Devlin et al. 2018), unsupervised representation learning has significantly improved the state of the art in natural language understanding. Parallel work on cross-lingual understanding (Mikolov et al. 2013a; Schuster et al. 2019; Lample and Conneau 2019) extends these systems to more languages and to the cross-lingual setting in which a model is learned in one language and applied in other languages.

从预训练词嵌入 (Mikolov et al. 2013b; Pennington et al. 2014) 到预训练上下文表示 (Peters et al. 2018; Schuster et al. 2019) 再到基于 Transformer 的语言模型 (Radford et al. 2018; Devlin et al. 2018)，无监督表示学习显著提升了自然语言理解水平。跨语言理解的并行工作 (Mikolov et al. 2013a; Schuster et al. 2019; Lample and Conneau 2019) 把这些系统扩展到更多语言，以及「在一种语言上学习、应用到其他语言」的跨语言设定。

Most recently, Devlin et al. 2018 and Lample and Conneau 2019 introduced mBERT and XLM - masked language models trained on multiple languages, without any cross-lingual supervision. Lample and Conneau 2019 propose translation language modeling (TLM) as a way to leverage parallel data and obtain a new state of the art on the cross-lingual natural language inference (XNLI) benchmark Conneau et al. 2018. They further show strong improvements on unsupervised machine translation and pretraining for sequence generation. Wu et al. 2019 shows that monolingual BERT representations are similar across languages, explaining in part the natural emergence of multilinguality in bottleneck architectures. Separately, Pires et al. 2019 demonstrated the effectiveness of multilingual models like mBERT on sequence labeling tasks. Huang et al. 2019 showed gains over XLM using cross-lingual multi-task learning, and Singh et al. 2019 demonstrated the efficiency of cross-lingual data augmentation for cross-lingual NLI. However, all of this work was at a relatively modest scale, in terms of the amount of training data, as compared to our approach.

最近，Devlin et al. 2018 与 Lample and Conneau 2019 引入 mBERT 与 XLM——在多种语言上训练、无需跨语言监督的掩码语言模型。Lample and Conneau 2019 提出翻译语言建模（TLM），利用平行数据，在跨语言自然语言推断（XNLI）基准 Conneau et al. 2018 上刷新纪录；他们还展示了对无监督机器翻译与序列生成预训练的显著改进。Wu et al. 2019 表明单语言 BERT 表示在语言间相似，部分解释了瓶颈架构中多语言性的自然涌现。另有 Pires et al. 2019 展示了 mBERT 一类多语言模型在序列标注上的有效性。Huang et al. 2019 用跨语言多任务学习相对 XLM 取得增益，Singh et al. 2019 展示了跨语言数据增强对跨语言 NLI 的效率。但相对我们的方法，上述工作在训练数据量上规模都较温和。

Figure 1: Amount of data in GiB (log-scale) for the 88 languages that appear in both the Wiki-100 corpus used for mBERT and XLM-100, and the CC-100 used for XLM-R. CC-100 increases the amount of data by several orders of magnitude, in particular for low-resource languages.

图 1：同时出现在 mBERT 与 XLM-100 所用 Wiki-100 语料、以及 XLM-R 所用 CC-100 中的 88 种语言的数据量（GiB，对数尺度）。CC-100 把数据量提高了几个数量级，对低资源语言尤为明显。

The benefits of scaling language model pretraining by increasing the size of the model as well as the training data has been extensively studied in the literature. For the monolingual case, Jozefowicz et al. 2016 show how large-scale LSTM models can obtain much stronger performance on language modeling benchmarks when trained on billions of tokens. GPT Radford et al. 2018 also highlights the importance of scaling the amount of data and RoBERTa Liu et al. 2019 shows that training BERT longer on more data leads to significant boost in performance. Inspired by RoBERTa, we show that mBERT and XLM are undertuned, and that simple improvements in the learning procedure of unsupervised MLM leads to much better performance. We train on cleaned CommonCrawls Wenzek et al. 2019, which increase the amount of data for low-resource languages by two orders of magnitude on average. Similar data has also been shown to be effective for learning high quality word embeddings in multiple languages Grave et al. 2018.

通过增大模型与训练数据来缩放语言模预训练，文献已有大量研究。单语言方面，Jozefowicz et al. 2016 表明在数十亿 token 上训练的大规模 LSTM 可在语言建模基准上强得多。GPT Radford et al. 2018 也强调数据规模，RoBERTa Liu et al. 2019 表明在更多数据上把 BERT 训更久可显著提升。受 RoBERTa 启发，我们表明 mBERT 与 XLM 训练不足，无监督 MLM 学习流程上的简单改进即可带来好得多的表现。我们在清洗后的 CommonCrawl Wenzek et al. 2019 上训练，平均把低资源语言数据量提高两个数量级。类似数据也被证明对学习多语言高质量词嵌入有效 Grave et al. 2018。

Several efforts have trained massively multilingual machine translation models from large parallel corpora. They uncover the high and low resource trade-off and the problem of capacity dilution (Johnson et al. 2017; Tan et al. 2019). The work most similar to ours is Arivazhagan et al. 2019, which trains a single model in 103 languages on over 25 billion parallel sentences. Siddhant et al. 2019 further analyze the representations obtained by the encoder of a massively multilingual machine translation system and show that it obtains similar results to mBERT on cross-lingual NLI. Our work, in contrast, focuses on the unsupervised learning of cross-lingual representations and their transfer to discriminative tasks.

多项工作从大型平行语料训练大规模多语言机器翻译模型，揭示了高/低资源权衡与容量稀释问题 (Johnson et al. 2017; Tan et al. 2019)。与我们最接近的是 Arivazhagan et al. 2019，在 103 种语言、逾 250 亿平行句上训练单一模型。Siddhant et al. 2019 进一步分析大规模多语言机器翻译编码器表示，表明其在跨语言 NLI 上与 mBERT 相近。我们的工作则聚焦无监督学习跨语言表示及其向判别任务的迁移。

3 Model and Data

3 模型与数据

In this section, we present the training objective, languages, and data we use. We follow the XLM approach Lample and Conneau 2019 as closely as possible, only introducing changes that improve performance at scale.

本节介绍训练目标、语言与所用数据。我们尽可能贴近 XLM 方法 Lample and Conneau 2019，仅引入有助于大规模表现的改动。

Masked Language Models.

We use a Transformer model Vaswani et al. 2017 trained with the multilingual MLM objective Devlin et al. 2018; Lample and Conneau 2019 using only monolingual data. We sample streams of text from each language and train the model to predict the masked tokens in the input. We apply subword tokenization directly on raw text data using Sentence Piece Kudo and Richardson 2018 with a unigram language model Kudo 2018. We sample batches from different languages using the same sampling distribution as Lample and Conneau 2019, but with $\alpha=0.3$. Unlike Lample and Conneau 2019, we do not use language embeddings, which allows our model to better deal with code-switching. We use a large vocabulary size of 250K with a full softmax and train two different models: XLM-R Base (L = 12, H = 768, A = 12, 270M params) and XLM-R (L = 24, H = 1024, A = 16, 550M params). For all of our ablation studies, we use a BERTBase architecture with a vocabulary of 150K tokens. Appendix B goes into more details about the architecture of the different models referenced in this paper.

掩码语言模型。

我们使用 Transformer 模型 Vaswani et al. 2017，以多语言 MLM 目标 Devlin et al. 2018; Lample and Conneau 2019 仅在单语言数据上训练。从各语言采样文本流，训练模型预测输入中被掩码的 token。我们用 Sentence Piece Kudo and Richardson 2018 配合 unigram 语言模型 Kudo 2018，直接在原始文本上做子词分词。按 Lample and Conneau 2019 的同一采样分布从不同语言抽批次，但取 $\alpha=0.3$。与 Lample and Conneau 2019 不同，我们不使用语言嵌入，以便更好处理语码转换。词表取大至 250K，配合全 softmax，训练两个模型：XLM-R Base（L = 12，H = 768，A = 12，2.70 亿参数）与 XLM-R（L = 24，H = 1024，A = 16，5.50 亿参数）。消融一律用 BERTBase 架构、15 万词表。附录 B 详述文中各模型架构。

Scaling to a hundred languages.

XLM-R is trained on 100 languages; we provide a full list of languages and associated statistics in Appendix A. Figure 1 specifies the iso codes of 88 languages that are shared across XLM-R and XLM-100, the model from Lample and Conneau 2019 trained on Wikipedia text in 100 languages.

扩展到一百种语言。

XLM-R 在 100 种语言上训练；附录 A 给出完整语言列表及相关统计。图 1 标出 XLM-R 与 XLM-100（Lample and Conneau 2019 在 100 种语言维基百科文本上训练的模型）共有的 88 种语言的 ISO 代码。

Compared to previous work, we replace some languages with more commonly used ones such as romanized Hindi and traditional Chinese. In our ablation studies, we always include the 7 languages for which we have classification and sequence labeling evaluation benchmarks: English, French, German, Russian, Chinese, Swahili and Urdu. We chose this set as it covers a suitable range of language families and includes low-resource languages such as Swahili and Urdu. We also consider larger sets of 15, 30, 60 and all 100 languages. When reporting results on high-resource and low-resource, we refer to the average of English and French results, and the average of Swahili and Urdu results respectively.

相对先前工作，我们替换部分语言为更常用者，例如罗马化印地语与繁体中文。消融中始终包含有分类与序列标注评测基准的 7 种语言：英语、法语、德语、俄语、中文、斯瓦希里语与乌尔都语。该集合覆盖合适的语系范围，并含斯瓦希里语、乌尔都语等低资源语言。我们也考虑 15、30、60 以及全部 100 种语言的更大集合。报告高资源与低资源结果时，分别指英语与法语的平均，以及斯瓦希里语与乌尔都语的平均。

Scaling the Amount of Training Data.

Following Wenzek et al. 2019 [^2], we build a clean CommonCrawl Corpus in 100 languages. We use an internal language identification model in combination with the one from fastText Joulin et al. 2017. We train language models in each language and use it to filter documents as described in Wenzek et al. 2019. We consider one CommonCrawl dump for English and twelve dumps for all other languages, which significantly increases dataset sizes, especially for low-resource languages like Burmese and Swahili.

扩大训练数据量。

沿用 Wenzek et al. 2019 [^2]，我们构建 100 种语言的干净 CommonCrawl 语料。语言识别结合内部模型与 fastText Joulin et al. 2017。在各语言上训练语言模型并按 Wenzek et al. 2019 过滤文档。英语用一次 CommonCrawl dump，其他语言用十二次 dump，显著增大数据量，对缅甸语、斯瓦希里语等低资源语言尤甚。

Figure 1 shows the difference in size between the Wikipedia Corpus used by mBERT and XLM-100, and the CommonCrawl Corpus we use. As we show in Section 5.3, monolingual Wikipedia corpora are too small to enable unsupervised representation learning. Based on our experiments, we found that a few hundred MiB of text data is usually a minimal size for learning a BERT model.

图 1 展示 mBERT 与 XLM-100 所用维基百科语料与我们所用 CommonCrawl 语料的规模差异。如第 5.3 节所示，单语言维基百科语料往往过小，难以支撑无监督表示学习。实验表明，通常需数百 MiB 文本才够学一个 BERT 模型。

4 Evaluation

4 评测

We consider four evaluation benchmarks. For cross-lingual understanding, we use cross-lingual natural language inference, named entity recognition, and question answering. We use the GLUE benchmark to evaluate the English performance of XLM-R and compare it to other state-of-the-art models.

我们考虑四项评测基准。跨语言理解用跨语言自然语言推断、命名实体识别与问答；用 GLUE 评估 XLM-R 的英语表现并与其他当时最优模型比较。

Cross-lingual Natural Language Inference (XNLI).

The XNLI dataset comes with ground-truth dev and test sets in 15 languages, and a ground-truth English training set. The training set has been machine-translated to the remaining 14 languages, providing synthetic training data for these languages as well. We evaluate our model on cross-lingual transfer from English to other languages. We also consider three machine translation baselines: (i) translate-test: dev and test sets are machine-translated to English and a single English model is used (ii) translate-train (per-language): the English training set is machine-translated to each language and we fine-tune a multiligual model on each training set (iii) translate-train-all (multi-language): we fine-tune a multilingual model on the concatenation of all training sets from translate-train. For the translations, we use the official data provided by the XNLI project.

跨语言自然语言推断（XNLI）。

XNLI 数据集提供 15 种语言的真实开发/测试集，以及真实英语训练集。训练集已机翻到其余 14 种语言，也为这些语言提供合成训练数据。我们评估从英语到其他语言的跨语言迁移。另考虑三种机器翻译基线：(i) translate-test：开发/测试集机翻到英语，只用单一英语模型；(ii) translate-train（按语言）：英语训练集机翻到各语言，在各训练集上分别微调多语言模型；(iii) translate-train-all（多语言）：在 translate-train 全部训练集的拼接上微调多语言模型。翻译使用 XNLI 项目提供的官方数据。

Named Entity Recognition.

For NER, we consider the CoNLL-2002 Sang 2002 and CoNLL-2003 Tjong Kim Sang and De Meulder 2003 datasets in English, Dutch, Spanish and German. We fine-tune multilingual models either (1) on the English set to evaluate cross-lingual transfer, (2) on each set to evaluate per-language performance, or (3) on all sets to evaluate multilingual learning. We report the F1 score, and compare to baselines from Lample et al. 2016 and Akbik et al. 2018.

命名实体识别。

NER 使用英语、荷兰语、西班牙语与德语上的 CoNLL-2002 Sang 2002 与 CoNLL-2003 Tjong Kim Sang and De Meulder 2003。多语言模型微调方式为：(1) 仅在英语集上以评测跨语言迁移；(2) 在各语言集上评测单语言表现；(3) 在全部集合上评测多语言学习。报告 F1，并与 Lample et al. 2016、Akbik et al. 2018 的基线比较。

Cross-lingual Question Answering.

We use the MLQA benchmark from Lewis et al. 2019, which extends the English SQuAD benchmark to Spanish, German, Arabic, Hindi, Vietnamese and Chinese. We report the F1 score as well as the exact match (EM) score for cross-lingual transfer from English.

跨语言问答。

使用 Lewis et al. 2019 的 MLQA 基准，把英语 SQuAD 扩展到西班牙、德、阿拉伯、印地、越南与中文。报告从英语跨语言迁移的 F1 与精确匹配（EM）。

GLUE Benchmark.

Finally, we evaluate the English performance of our model on the GLUE benchmark Wang et al. 2018 which gathers multiple classification tasks, such as MNLI Williams et al. 2017, SST-2 Socher et al. 2013, or QNLI Rajpurkar et al. 2018. We use BERTLarge and RoBERTa as baselines.

GLUE 基准。

最后在 GLUE Wang et al. 2018 上评估模型的英语表现，该基准汇集 MNLI Williams et al. 2017、SST-2 Socher et al. 2013、QNLI Rajpurkar et al. 2018 等多种分类任务。基线为 BERTLarge 与 RoBERTa。

5 Analysis and Results

5 分析与结果

In this section, we perform a comprehensive analysis of multilingual masked language models. We conduct most of the analysis on XNLI, which we found to be representative of our findings on other tasks. We then present the results of XLM-R on cross-lingual understanding and GLUE. Finally, we compare multilingual and monolingual models, and present results on low-resource languages.

本节全面分析多语言掩码语言模型。多数分析在 XNLI 上进行，我们发现它能代表其他任务上的结论。随后给出 XLM-R 在跨语言理解与 GLUE 上的结果，并比较多语言与单语言模型，以及低资源语言结果。

5.1 Improving and Understanding Multilingual Masked Language Models

5.1 改进并理解多语言掩码语言模型

Figure 2: The transfer-interference trade-off: Low-resource languages benefit from scaling to more languages, until dilution (interference) kicks in and degrades overall performance.

图 2：迁移–干扰权衡：低资源语言受益于扩展到更多语言，直到稀释（干扰）出现并拉低整体表现。

Figure 5: On the high-resource versus low-resource trade-off: impact of batch language sampling for XLM-100.

图 5：高资源与低资源权衡：批次语言采样对 XLM-100 的影响。

Much of the work done on understanding the cross-lingual effectiveness of mBERT or XLM Pires et al. 2019; Wu and Dredze 2019; Lewis et al. 2019 has focused on analyzing the performance of fixed pretrained models on downstream tasks. In this section, we present a comprehensive study of different factors that are important to pretraining large scale multilingual models. We highlight the trade-offs and limitations of these models as we scale to one hundred languages.

理解 mBERT 或 XLM 跨语言有效性的许多工作 Pires et al. 2019; Wu and Dredze 2019; Lewis et al. 2019 聚焦分析固定预训练模型在下游任务上的表现。本节全面研究对预训练大规模多语言模型重要的不同因素，突出扩展到一百种语言时的权衡与局限。

Transfer-dilution Trade-off and Curse of Multilinguality.

Model capacity (i.e. the number of parameters in the model) is constrained due to practical considerations such as memory and speed during training and inference. For a fixed sized model, the per-language capacity decreases as we increase the number of languages. While low-resource language performance can be improved by adding similar higher-resource languages during pretraining, the overall downstream performance suffers from this capacity dilution Arivazhagan et al. 2019. Positive transfer and capacity dilution have to be traded off against each other.

迁移–稀释权衡与多语言诅咒。

模型容量（参数量）受训练与推理时内存、速度等实际因素约束。固定大小模型下，语言数增加会使每语言可用容量下降。预训练时加入相似的高资源语言可改善低资源表现，但整体下游表现会受容量稀释拖累 Arivazhagan et al. 2019。正向迁移与容量稀释必须彼此权衡。

We illustrate this trade-off in Figure 2, which shows XNLI performance vs the number of languages the model is pretrained on. Initially, as we go from 7 to 15 languages, the model is able to take advantage of positive transfer which improves performance, especially on low resource languages. Beyond this point the curse of multilinguality kicks in and degrades performance across all languages. Specifically, the overall XNLI accuracy decreases from 71.8% to 67.7% as we go from XLM-7 to XLM-100. The same trend can be observed for models trained on the larger CommonCrawl Corpus.

图 2 展示该权衡：XNLI 表现相对预训练语言数。从 7 种增至 15 种时，模型能利用正向迁移提升表现，低资源语言尤甚。超过此点后多语言诅咒出现，所有语言表现下降。具体地，从 XLM-7 到 XLM-100，整体 XNLI 准确率从 71.8% 降至 67.7%。在更大规模 CommonCrawl 语料上训练的模型也呈同一趋势。

The issue is even more prominent when the capacity of the model is small. To show this, we pretrain models on Wikipedia Data in 7, 30 and 100 languages. As we add more languages, we make the Transformer wider by increasing the hidden size from 768 to 960 to 1152. In Figure 4, we show that the added capacity allows XLM-30 to be on par with XLM-7, thus overcoming the curse of multilinguality. The added capacity for XLM-100, however, is not enough and it still lags behind due to higher vocabulary dilution (recall from Section 3 that we used a fixed vocabulary size of 150K for all models).

模型容量较小时问题更突出。我们在 7、30、100 种语言的维基百科数据上预训练；随语言增多，把 Transformer 加宽，隐层从 768 增至 960 再至 1152。图 4 显示增加容量可使 XLM-30 与 XLM-7 持平，从而克服多语言诅咒。但对 XLM-100，增加的容量仍不够，仍因更高的词表稀释落后（回顾第 3 节：所有模型固定使用 15 万词表）。

High-resource vs Low-resource Trade-off.

The allocation of the model capacity across languages is controlled by several parameters: the training set size, the size of the shared subword vocabulary, and the rate at which we sample training examples from each language. We study the effect of sampling on the performance of high-resource (English and French) and low-resource (Swahili and Urdu) languages for an XLM-100 model trained on Wikipedia (we observe a similar trend for the construction of the subword vocab). Specifically, we investigate the impact of varying the $\alpha$ parameter which controls the exponential smoothing of the language sampling rate. Similar to Lample and Conneau 2019, we use a sampling rate proportional to the number of sentences in each corpus. Models trained with higher values of $\alpha$ see batches of high-resource languages more often. Figure 5 shows that the higher the value of $\alpha$, the better the performance on high-resource languages, and vice-versa. When considering overall performance, we found $0.3$ to be an optimal value for $\alpha$, and use this for XLM-R.

高资源与低资源权衡。

模型容量在语言间的分配由若干参数控制：训练集大小、共享子词词表大小，以及从各语言采样训练例的速率。我们研究采样对在维基百科上训练的 XLM-100 在高资源（英、法）与低资源（斯瓦希里、乌尔都）上表现的影响（子词词表构建上也观察到类似趋势）。具体考察控制语言采样率指数平滑的 $\alpha$ 参数。与 Lample and Conneau 2019 类似，采样率与各语料句子数成正比。$\alpha$ 越大，模型越常看到高资源语言批次。图 5 表明：$\alpha$ 越高，高资源表现越好，反之亦然。综合考虑整体表现，我们发现 $\alpha=0.3$ 较优，并用于 XLM-R。

Importance of Capacity and Vocabulary.

In previous sections and in Figure 4, we showed the importance of scaling the model size as we increase the number of languages. Similar to the overall model size, we argue that scaling the size of the shared vocabulary (the vocabulary capacity) can improve the performance of multilingual models on downstream tasks. To illustrate this effect, we train XLM-100 models on Wikipedia data with different vocabulary sizes. We keep the overall number of parameters constant by adjusting the width of the transformer. Figure 6 shows that even with a fixed capacity, we observe a 2.8% increase in XNLI average accuracy as we increase the vocabulary size from 32K to 256K. This suggests that multilingual models can benefit from allocating a higher proportion of the total number of parameters to the embedding layer even though this reduces the size of the Transformer. For simplicity and given the softmax computational constraints, we use a vocabulary of 250k for XLM-R.

容量与词表的重要性。

前几节与图 4 已表明，随语言数增加需同步放大模型。与整体模型大小类似，我们认为放大共享词表（词表容量）也可改善多语言模型下游表现。为说明，我们在维基百科数据上以不同词表大小训练 XLM-100，并通过调节 Transformer 宽度保持总参数恒定。图 6 显示即便固定总容量，词表从 32K 增至 256K 时 XNLI 平均准确率仍升 2.8%。这表明多语言模型可以把更高比例参数分给嵌入层而受益，即便这会缩小 Transformer。为简单并受 softmax 计算约束，XLM-R 使用 250k 词表。

We further illustrate the importance of this parameter, by training three models with the same transformer architecture (BERTBase) but with different vocabulary sizes: 128K, 256K and 512K. We observe more than 3% gains in overall accuracy on XNLI by simply increasing the vocab size from 128k to 512k.

我们进一步说明该参数：用同一 Transformer 架构（BERTBase）但词表分别为 128K、256K、512K 训练三个模型。仅把词表从 128k 增至 512k，XNLI 整体准确率提升逾 3%。

Larger-scale Datasets and Training.

As shown in Figure 1, the CommonCrawl Corpus that we collected has significantly more monolingual data than the previously used Wikipedia corpora. Figure 3 shows that for the same BERTBase architecture, all models trained on CommonCrawl obtain significantly better performance.

更大规模数据与训练。

如图 1，我们收集的 CommonCrawl 语料显著多于先前维基百科语料。图 3 显示：同一 BERTBase 架构下，所有在 CommonCrawl 上训练的模型都显著更好。

Apart from scaling the training data, Liu et al. 2019 also showed the benefits of training MLMs longer. In our experiments, we observed similar effects of large-scale training, such as increasing batch size (see Figure 7) and training time, on model performance. Specifically, we found that using validation perplexity as a stopping criterion for pretraining caused the multilingual MLM in Lample and Conneau 2019 to be under-tuned. In our experience, performance on downstream tasks continues to improve even after validation perplexity has plateaued. Combining this observation with our implementation of the unsupervised XLM-MLM objective, we were able to improve the performance of Lample and Conneau 2019 from 71.3% to more than 75% average accuracy on XNLI, which was on par with their supervised translation language modeling (TLM) objective. Based on these results, and given our focus on unsupervised learning, we decided to not use the supervised TLM objective for training our models.

除放大训练数据外，Liu et al. 2019 也展示了把 MLM 训更久的好处。实验中我们观察到类似的大规模训练效应，例如增大 batch（见图 7）与训练时间对表现的影响。具体地，我们发现用验证困惑度作预训练停止准则会导致 Lample and Conneau 2019 中的多语言 MLM 训练不足。经验上，即便验证困惑度已平台，下游任务仍可继续提升。结合这一观察与我们对无监督 XLM-MLM 目标的实现，可将 Lample and Conneau 2019 的 XNLI 平均准确率从 71.3% 提升到逾 75%，与其有监督翻译语言建模（TLM）目标持平。基于此，并鉴于我们聚焦无监督学习，决定训练时不使用有监督 TLM 目标。

Simplifying Multilingual Tokenization with Sentence Piece.

The different language-specific tokenization tools used by mBERT and XLM-100 make these models more difficult to use on raw text. Instead, we train a Sentence Piece model (SPM) and apply it directly on raw text data for all languages. We did not observe any loss in performance for models trained with SPM when compared to models trained with language-specific preprocessing and byte-pair encoding (see Figure 7) and hence use SPM for XLM-R.

用 Sentence Piece 简化多语言分词。

mBERT 与 XLM-100 使用的各语言专用分词工具使模型更难直接用于原始文本。我们改为训练 Sentence Piece 模型（SPM），对所有语言直接作用于原始文本。相对使用语言特定预处理与字节对编码的模型，SPM 训练模型未观察到性能损失（见图 7），故 XLM-R 采用 SPM。

5.2 Cross-lingual Understanding Results

5.2 跨语言理解结果

Based on these results, we adapt the setting of Lample and Conneau 2019 and use a large Transformer model with 24 layers and 1024 hidden states, with a 250k vocabulary. We use the multilingual MLM loss and train our XLM-R model for 1.5 Million updates on five-hundred 32GB Nvidia V100 GPUs with a batch size of 8192. We leverage the SPM-preprocessed text data from CommonCrawl in 100 languages and sample languages with $\alpha=0.3$. In this section, we show that it outperforms all previous techniques on cross-lingual benchmarks while getting performance on par with RoBERTa on the GLUE benchmark.

基于上述结果，我们沿用 Lample and Conneau 2019 的设定，使用 24 层、1024 隐状态、250k 词表的大型 Transformer。采用多语言 MLM 损失，在五百张 32GB Nvidia V100 上以 batch 8192 训练 XLM-R 共 150 万次更新。利用 CommonCrawl 中 100 种语言的 SPM 预处理文本，并以 $\alpha=0.3$ 采样语言。本节表明它在跨语言基准上全面优于先前技术，同时在 GLUE 上与 RoBERTa 持平。

Model	D	#M	#lg	en	fr	es	de	el	bg	ru	tr	ar	vi	th	zh	hi	sw	ur	Avg
Fine-tune multilingual model on English training set (Cross-lingual Transfer)
Lample and Conneau 2019	Wiki+MT	N	15	85.0	78.7	78.9	77.8	76.6	77.4	75.3	72.5	73.1	76.1	73.2	76.5	69.6	68.4	67.3	75.1
Huang et al. 2019	Wiki+MT	N	15	85.1	79.0	79.4	77.8	77.2	77.2	76.3	72.8	73.5	76.4	73.6	76.2	69.4	69.7	66.7	75.4
Devlin et al. 2018	Wiki	N	102	82.1	73.8	74.3	71.1	66.4	68.9	69.0	61.6	64.9	69.5	55.8	69.3	60.0	50.4	58.0	66.3
Lample and Conneau 2019	Wiki	N	100	83.7	76.2	76.6	73.7	72.4	73.0	72.1	68.1	68.4	72.0	68.2	71.5	64.5	58.0	62.4	71.3
Lample and Conneau 2019	Wiki	1	100	83.2	76.7	77.7	74.0	72.7	74.1	72.7	68.7	68.6	72.9	68.9	72.5	65.6	58.2	62.4	70.7
XLM-RBase	CC	1	100	85.8	79.7	80.7	78.7	77.5	79.6	78.1	74.2	73.8	76.5	74.6	76.7	72.4	66.5	68.3	76.2
XLM-R	CC	1	100	89.1	84.1	85.1	83.9	82.9	84.0	81.2	79.6	79.8	80.8	78.1	80.2	76.9	73.9	73.8	80.9
Translate everything to English and use English-only model (TRANSLATE-TEST)
BERT-en	Wiki	1	1	88.8	81.4	82.3	80.1	80.3	80.9	76.2	76.0	75.4	72.0	71.9	75.6	70.0	65.8	65.8	76.2
RoBERTa	Wiki+CC	1	1	91.3	82.9	84.3	81.2	81.7	83.1	78.3	76.8	76.6	74.2	74.1	77.5	70.9	66.7	66.8	77.8
Fine-tune multilingual model on each training set (TRANSLATE-TRAIN)
Lample and Conneau 2019	Wiki	N	100	82.9	77.6	77.9	77.9	77.1	75.7	75.5	72.6	71.2	75.8	73.1	76.2	70.4	66.5	62.4	74.2
Fine-tune multilingual model on all training sets (TRANSLATE-TRAIN-ALL)
Lample and Conneau 2019*†*	Wiki+MT	1	15	85.0	80.8	81.3	80.3	79.1	80.9	78.3	75.6	77.6	78.5	76.0	79.5	72.9	72.8	68.5	77.8
Huang et al. 2019	Wiki+MT	1	15	85.6	81.1	82.3	80.9	79.5	81.4	79.7	76.8	78.2	77.9	77.1	80.5	73.4	73.8	69.6	78.5
Lample and Conneau 2019	Wiki	1	100	84.5	80.1	81.3	79.3	78.6	79.4	77.5	75.2	75.6	78.3	75.7	78.3	72.1	69.2	67.7	76.9
XLM-RBase	CC	1	100	85.4	81.4	82.2	80.3	80.4	81.3	79.7	78.6	77.3	79.7	77.9	80.2	76.1	73.1	73.0	79.1
XLM-R	CC	1	100	89.1	85.1	86.6	85.7	85.3	85.9	83.5	83.2	83.1	83.7	81.5	83.7	81.6	78.0	78.1	83.6

Table 1: Results on cross-lingual classification. We report the accuracy on each of the 15 XNLI languages and the average accuracy. We specify the dataset D used for pretraining, the number of models #M the approach requires and the number of languages #lg the model handles. Our XLM-R results are averaged over five different seeds. We show that using the translate-train-all approach which leverages training sets from multiple languages, XLM-R obtains a new state of the art on XNLI of $83.6$% average accuracy. Results with † are from Huang et al. 2019.

Model	D	#M	#lg	en	fr	es	de	el	bg	ru	tr	ar	vi	th	zh	hi	sw	ur	Avg
在英语训练集上微调多语言模型（跨语言迁移)
Lample and Conneau 2019	Wiki+MT	N	15	85.0	78.7	78.9	77.8	76.6	77.4	75.3	72.5	73.1	76.1	73.2	76.5	69.6	68.4	67.3	75.1
Huang et al. 2019	Wiki+MT	N	15	85.1	79.0	79.4	77.8	77.2	77.2	76.3	72.8	73.5	76.4	73.6	76.2	69.4	69.7	66.7	75.4
Devlin et al. 2018	Wiki	N	102	82.1	73.8	74.3	71.1	66.4	68.9	69.0	61.6	64.9	69.5	55.8	69.3	60.0	50.4	58.0	66.3
Lample and Conneau 2019	Wiki	N	100	83.7	76.2	76.6	73.7	72.4	73.0	72.1	68.1	68.4	72.0	68.2	71.5	64.5	58.0	62.4	71.3
Lample and Conneau 2019	Wiki	1	100	83.2	76.7	77.7	74.0	72.7	74.1	72.7	68.7	68.6	72.9	68.9	72.5	65.6	58.2	62.4	70.7
XLM-RBase	CC	1	100	85.8	79.7	80.7	78.7	77.5	79.6	78.1	74.2	73.8	76.5	74.6	76.7	72.4	66.5	68.3	76.2
XLM-R	CC	1	100	89.1	84.1	85.1	83.9	82.9	84.0	81.2	79.6	79.8	80.8	78.1	80.2	76.9	73.9	73.8	80.9
全部译成英语并用纯英语模型（TRANSLATE-TEST)
BERT-en	Wiki	1	1	88.8	81.4	82.3	80.1	80.3	80.9	76.2	76.0	75.4	72.0	71.9	75.6	70.0	65.8	65.8	76.2
RoBERTa	Wiki+CC	1	1	91.3	82.9	84.3	81.2	81.7	83.1	78.3	76.8	76.6	74.2	74.1	77.5	70.9	66.7	66.8	77.8
在各语言训练集上微调多语言模型（TRANSLATE-TRAIN)
Lample and Conneau 2019	Wiki	N	100	82.9	77.6	77.9	77.9	77.1	75.7	75.5	72.6	71.2	75.8	73.1	76.2	70.4	66.5	62.4	74.2
在全部训练集上微调多语言模型（TRANSLATE-TRAIN-ALL)
Lample and Conneau 2019*†*	Wiki+MT	1	15	85.0	80.8	81.3	80.3	79.1	80.9	78.3	75.6	77.6	78.5	76.0	79.5	72.9	72.8	68.5	77.8
Huang et al. 2019	Wiki+MT	1	15	85.6	81.1	82.3	80.9	79.5	81.4	79.7	76.8	78.2	77.9	77.1	80.5	73.4	73.8	69.6	78.5
Lample and Conneau 2019	Wiki	1	100	84.5	80.1	81.3	79.3	78.6	79.4	77.5	75.2	75.6	78.3	75.7	78.3	72.1	69.2	67.7	76.9
XLM-RBase	CC	1	100	85.4	81.4	82.2	80.3	80.4	81.3	79.7	78.6	77.3	79.7	77.9	80.2	76.1	73.1	73.0	79.1
XLM-R	CC	1	100	89.1	85.1	86.6	85.7	85.3	85.9	83.5	83.2	83.1	83.7	81.5	83.7	81.6	78.0	78.1	83.6

表 1：跨语言分类结果。 报告 15 种 XNLI 语言各自准确率与平均准确率。标明预训练数据集 D、方法所需模型数 #M，以及模型覆盖语言数 #lg。XLM-R 结果对五个随机种子取平均。使用利用多语言训练集的 translate-train-all，XLM-R 在 XNLI 上取得 $83.6$% 平均准确率的新最优。带 † 的结果来自 Huang et al. 2019。

XNLI.

Table 1 shows XNLI results and adds some additional details: (i) the number of models the approach induces (#M), (ii) the data on which the model was trained (D), and (iii) the number of languages the model was pretrained on (#lg). As we show in our results, these parameters significantly impact performance. Column #M specifies whether model selection was done separately on the dev set of each language ($N$ models), or on the joint dev set of all the languages (single model). We observe a 0.6 decrease in overall accuracy when we go from $N$ models to a single model - going from 71.3 to 70.7. We encourage the community to adopt this setting. For cross-lingual transfer, while this approach is not fully zero-shot transfer, we argue that in real applications, a small amount of supervised data is often available for validation in each language.

XNLI。

表 1 给出 XNLI 结果并补充细节：(i) 方法导出的模型数（#M），(ii) 模型训练数据（D），(iii) 预训练语言数（#lg）。如结果所示，这些参数显著影响表现。#M 列标明是在各语言开发集上分别做模型选择（$N$ 个模型），还是在全部语言联合开发集上选择（单一模型）。从 $N$ 个模型到单一模型，整体准确率下降 0.6——从 71.3 到 70.7。我们鼓励社区采用后一设定。对跨语言迁移而言，该方法并非完全零样本，但我们认为实际应用中各语言往往有少量监督数据可用于验证。

XLM-R sets a new state of the art on XNLI. On cross-lingual transfer, XLM-R obtains 80.9% accuracy, outperforming the XLM-100 and mBERT open-source models by 10.2% and 14.6% average accuracy. On the Swahili and Urdu low-resource languages, XLM-R outperforms XLM-100 by 15.7% and 11.4%, and mBERT by 23.5% and 15.8%. While XLM-R handles 100 languages, we also show that it outperforms the former state of the art Unicoder (Huang et al. 2019) and XLM (MLM+TLM), which handle only 15 languages, by 5.5% and 5.8% average accuracy respectively. Using the multilingual training of translate-train-all, XLM-R further improves performance and reaches 83.6% accuracy, a new overall state of the art for XNLI, outperforming Unicoder by 5.1%. Multilingual training is similar to practical applications where training sets are available in various languages for the same task. In the case of XNLI, datasets have been translated, and translate-train-all can be seen as some form of cross-lingual data augmentation Singh et al. 2019, similar to back-translation Xie et al. 2019.

XLM-R 在 XNLI 上刷新纪录。跨语言迁移上，XLM-R 取得 80.9% 准确率，相对开源 XLM-100 与 mBERT 分别高 10.2% 与 14.6% 平均准确率。在斯瓦希里语与乌尔都语低资源语言上，相对 XLM-100 分别高 15.7% 与 11.4%，相对 mBERT 高 23.5% 与 15.8%。尽管 XLM-R 覆盖 100 种语言，它仍分别以 5.5% 与 5.8% 平均准确率超过仅覆盖 15 种语言的先前最优 Unicoder (Huang et al. 2019) 与 XLM（MLM+TLM）。使用 translate-train-all 的多语言训练，XLM-R 进一步提升到 83.6% 准确率，成为 XNLI 整体新最优，相对 Unicoder 高 5.1%。多语言训练类似实际应用中同一任务有多语言训练集的情形。XNLI 中数据经翻译，translate-train-all 可视为某种跨语言数据增强 Singh et al. 2019，类似回译 Xie et al. 2019。

Model	train	#M	en	nl	es	de	Avg
Lample et al. 2016	each	N	90.74	81.74	85.75	78.76	84.25
Akbik et al. 2018	each	N	93.18	90.44	-	88.27	-
mBERT*†*	each	N	91.97	90.94	87.38	82.82	88.28
en	1	91.97	77.57	74.96	69.56	78.52
XLM-RBase	each	N	92.25	90.39	87.99	84.60	88.81
en	1	92.25	78.08	76.53	69.60	79.11
all	1	91.08	89.09	87.28	83.17	87.66
XLM-R	each	N	92.92	92.53	89.72	85.81	90.24
en	1	92.92	80.80	78.64	71.40	80.94
all	1	92.00	91.60	89.52	84.60	89.43

Table 2: Results on named entity recognition on CoNLL-2002 and CoNLL-2003 (F1 score). Results with $\dagger$ are from Wu and Dredze 2019. Note that mBERT and XLM-R do not use a linear-chain CRF, as opposed to Akbik et al. 2018 and Lample et al. 2016.

Model	train	#M	en	nl	es	de	Avg
Lample et al. 2016	each	N	90.74	81.74	85.75	78.76	84.25
Akbik et al. 2018	each	N	93.18	90.44	-	88.27	-
mBERT*†*	each	N	91.97	90.94	87.38	82.82	88.28
en	1	91.97	77.57	74.96	69.56	78.52
XLM-RBase	each	N	92.25	90.39	87.99	84.60	88.81
en	1	92.25	78.08	76.53	69.60	79.11
all	1	91.08	89.09	87.28	83.17	87.66
XLM-R	each	N	92.92	92.53	89.72	85.81	90.24
en	1	92.92	80.80	78.64	71.40	80.94
all	1	92.00	91.60	89.52	84.60	89.43

表 2：CoNLL-2002 与 CoNLL-2003 上的命名实体识别结果（F1）。带 $\dagger$ 的结果来自 Wu and Dredze 2019。注意相对 Akbik et al. 2018 与 Lample et al. 2016，mBERT 与 XLM-R 未使用线性链 CRF。

Named Entity Recognition.

In Table 2, we report results of XLM-R and mBERT on CoNLL-2002 and CoNLL-2003. We consider the LSTM + CRF approach from Lample et al. 2016 and the Flair model from Akbik et al. 2018 as baselines. We evaluate the performance of the model on each of the target languages in three different settings: (i) train on English data only (en) (ii) train on data in target language (each) (iii) train on data in all languages (all). Results of mBERT are reported from Wu and Dredze 2019. Note that we do not use a linear-chain CRF on top of XLM-R and mBERT representations, which gives an advantage to Akbik et al. 2018. Without the CRF, our XLM-R model still performs on par with the state of the art, outperforming Akbik et al. 2018 on Dutch by $2.09$ points. On this task, XLM-R also outperforms mBERT by 2.42 F1 on average for cross-lingual transfer, and 1.86 F1 when trained on each language. Training on all languages leads to an average F1 score of 89.43%, outperforming cross-lingual transfer approach by 8.49%.

命名实体识别。

表 2 报告 XLM-R 与 mBERT 在 CoNLL-2002/2003 上的结果。基线取 Lample et al. 2016 的 LSTM + CRF 与 Akbik et al. 2018 的 Flair。三种设定评估目标语言表现：(i) 仅在英语数据上训练（en）；(ii) 在目标语言数据上训练（each）；(iii) 在全部语言数据上训练（all）。mBERT 结果引自 Wu and Dredze 2019。注意我们未在 XLM-R 与 mBERT 表示之上加线性链 CRF，这对 Akbik et al. 2018 有利。即便无 CRF，XLM-R 仍与当时最优持平，荷兰语上相对 Akbik et al. 2018 高 $2.09$ 分。该任务上 XLM-R 跨语言迁移平均 F1 相对 mBERT 高 2.42，按语言训练时高 1.86。在全部语言上训练得到平均 F1 89.43%，相对跨语言迁移高 8.49%。

Question Answering.

We also obtain new state of the art results on the MLQA cross-lingual question answering benchmark, introduced by Lewis et al. 2019. We follow their procedure by training on the English training data and evaluating on the 7 languages of the dataset. We report results in Table 3. XLM-R obtains F1 and accuracy scores of 70.7% and 52.7% while the previous state of the art was 61.6% and 43.5%. XLM-R also outperforms mBERT by 13.0% F1-score and 11.1% accuracy. It even outperforms BERT-Large on English, confirming its strong monolingual performance.

问答。

我们在 Lewis et al. 2019 提出的 MLQA 跨语言问答基准上也取得新最优。按其流程在英语训练数据上训练，在数据集的 7 种语言上评估。结果见表 3。XLM-R 取得 F1 70.7%、准确率 52.7%，而先前最优为 61.6% 与 43.5%。相对 mBERT，F1 高 13.0%，准确率高 11.1%。它甚至在英语上超过 BERT-Large，印证其强单语言表现。

Model	train	#lgs	en	es	de	ar	hi	vi	zh	Avg
BERT-Large*†*	en	1	80.2 / 67.4	-	-	-	-	-	-	-
mBERT*†*	en	102	77.7 / 65.2	64.3 / 46.6	57.9 / 44.3	45.7 / 29.8	43.8 / 29.7	57.1 / 38.6	57.5 / 37.3	57.7 / 41.6
XLM-15*†*	en	15	74.9 / 62.4	68.0 / 49.8	62.2 / 47.6	54.8 / 36.3	48.8 / 27.3	61.4 / 41.8	61.1 / 39.6	61.6 / 43.5
XLM-RBase	en	100	77.1 / 64.6	67.4 / 49.6	60.9 / 46.7	54.9 / 36.6	59.4 / 42.9	64.5 / 44.7	61.8 / 39.3	63.7 / 46.3
XLM-R	en	100	80.6 / 67.8	74.1 / 56.0	68.5 / 53.6	63.1 / 43.5	69.2 / 51.6	71.3 / 50.9	68.0 / 45.4	70.7 / 52.7

Table 3: Results on MLQA question answering We report the F1 and EM (exact match) scores for zero-shot classification where models are fine-tuned on the English Squad dataset and evaluated on the 7 languages of MLQA. Results with $\dagger$ are taken from the original MLQA paper Lewis et al. 2019.

Model	train	#lgs	en	es	de	ar	hi	vi	zh	Avg
BERT-Large*†*	en	1	80.2 / 67.4	-	-	-	-	-	-	-
mBERT*†*	en	102	77.7 / 65.2	64.3 / 46.6	57.9 / 44.3	45.7 / 29.8	43.8 / 29.7	57.1 / 38.6	57.5 / 37.3	57.7 / 41.6
XLM-15*†*	en	15	74.9 / 62.4	68.0 / 49.8	62.2 / 47.6	54.8 / 36.3	48.8 / 27.3	61.4 / 41.8	61.1 / 39.6	61.6 / 43.5
XLM-RBase	en	100	77.1 / 64.6	67.4 / 49.6	60.9 / 46.7	54.9 / 36.6	59.4 / 42.9	64.5 / 44.7	61.8 / 39.3	63.7 / 46.3
XLM-R	en	100	80.6 / 67.8	74.1 / 56.0	68.5 / 53.6	63.1 / 43.5	69.2 / 51.6	71.3 / 50.9	68.0 / 45.4	70.7 / 52.7

表 3：MLQA 问答结果。 报告零样本设定下的 F1 与 EM（精确匹配）：模型在英语 SQuAD 上微调，在 MLQA 的 7 种语言上评估。带 $\dagger$ 的结果取自原 MLQA 论文 Lewis et al. 2019。

5.3 Multilingual versus Monolingual

5.3 多语言对比单语言

In this section, we present results of multilingual XLM models against monolingual BERT models.

本节给出多语言 XLM 模型相对单语言 BERT 模型的结果。

GLUE: XLM-R versus RoBERTa.

Our goal is to obtain a multilingual model with strong performance on both, cross-lingual understanding tasks as well as natural language understanding tasks for each language. To that end, we evaluate XLM-R on the GLUE benchmark. We show in Table 4, that XLM-R obtains better average dev performance than BERTLarge by 1.6% and reaches performance on par with XLNetLarge. The RoBERTa model outperforms XLM-R by only 1.0% on average. We believe future work can reduce this gap even further by alleviating the curse of multilinguality and vocabulary dilution. These results demonstrate the possibility of learning one model for many languages while maintaining strong performance on per-language downstream tasks.

GLUE：XLM-R 对比 RoBERTa。

目标是得到在跨语言理解任务与各语言自然语言理解任务上都强的多语言模型。为此我们在 GLUE 上评估 XLM-R。表 4 显示 XLM-R 平均开发集表现相对 BERTLarge 高 1.6%，并与 XLNetLarge 持平。RoBERTa 平均仅高 XLM-R 1.0%。我们认为后续工作可通过缓解多语言诅咒与词表稀释进一步缩小差距。这些结果表明：可以为多种语言学习单一模型，同时在各语言下游任务上保持强表现。

Model	#lgs	MNLI-m/mm	QNLI	QQP	SST	MRPC	STS-B	Avg
BERTLarge*†*	1	86.6/-	92.3	91.3	93.2	88.0	90.0	90.2
XLNetLarge*†*	1	89.8/-	93.9	91.8	95.6	89.2	91.8	92.0
RoBERTa*†*	1	90.2/90.2	94.7	92.2	96.4	90.9	92.4	92.8
XLM-R	100	88.9/89.0	93.8	92.3	95.0	89.5	91.2	91.8

Table 4: GLUE dev results. Results with † are from Liu et al. 2019. We compare the performance of XLM-R to BERTLarge, XLNet and RoBERTa on the English GLUE benchmark.

Model	#lgs	MNLI-m/mm	QNLI	QQP	SST	MRPC	STS-B	Avg
BERTLarge*†*	1	86.6/-	92.3	91.3	93.2	88.0	90.0	90.2
XLNetLarge*†*	1	89.8/-	93.9	91.8	95.6	89.2	91.8	92.0
RoBERTa*†*	1	90.2/90.2	94.7	92.2	96.4	90.9	92.4	92.8
XLM-R	100	88.9/89.0	93.8	92.3	95.0	89.5	91.2	91.8

表 4：GLUE 开发集结果。 带 † 的结果来自 Liu et al. 2019。在英语 GLUE 上比较 XLM-R 与 BERTLarge、XLNet 与 RoBERTa。

XNLI: XLM versus BERT.

A recurrent criticism against multilingual models is that they obtain worse performance than their monolingual counterparts. In addition to the comparison of XLM-R and RoBERTa, we provide the first comprehensive study to assess this claim on the XNLI benchmark. We extend our comparison between multilingual XLM models and monolingual BERT models on 7 languages and compare performance in Table 5. We train 14 monolingual BERT models on Wikipedia and CommonCrawl (capped at 60 GiB), and two XLM-7 models. We increase the vocabulary size of the multilingual model for a better comparison. We found that multilingual models can outperform their monolingual BERT counterparts. Specifically, in Table 5, we show that for cross-lingual transfer, monolingual baselines outperform XLM-7 for both Wikipedia and CC by 1.6% and 1.3% average accuracy. However, by making use of multilingual training (translate-train-all) and leveraging training sets coming from multiple languages, XLM-7 can outperform the BERT models: our XLM-7 trained on CC obtains 80.0% average accuracy on the 7 languages, while the average performance of BERT models trained on CC is 77.5%. This is a surprising result that shows that the capacity of multilingual models to leverage training data coming from multiple languages for a particular task can overcome the capacity dilution problem to obtain better overall performance.

XNLI：XLM 对比 BERT。

对多语言模型的常见批评是其表现不如单语言对应物。除 XLM-R 与 RoBERTa 的比较外，我们在 XNLI 上首次全面评估该说法。我们把多语言 XLM 与单语言 BERT 的比较扩展到 7 种语言，结果见表 5。在维基百科与 CommonCrawl（上限 60 GiB）上训练 14 个单语言 BERT，以及两个 XLM-7；并增大多语言模型词表以便更好比较。我们发现多语言模型可以超过其单语言 BERT 对应物。具体地，表 5 显示跨语言迁移设定下，单语言基线相对 XLM-7 在维基与 CC 上分别高 1.6% 与 1.3% 平均准确率。但利用多语言训练（translate-train-all）与来自多种语言的训练集后，XLM-7 可超过 BERT：CC 上训练的 XLM-7 在 7 种语言上平均准确率 80.0%，而 CC 上 BERT 平均为 77.5%。这一意外结果表明：多语言模型利用多语言训练数据完成特定任务的能力，可以克服容量稀释，获得更好的整体表现。

Model	D	#vocab	en	fr	de	ru	zh	sw	ur	Avg
Monolingual baselines
BERT	Wiki	40k	84.5	78.6	80.0	75.5	77.7	60.1	57.3	73.4
CC	40k	86.7	81.2	81.2	78.2	79.5	70.8	65.1	77.5
Multilingual models (cross-lingual transfer)
XLM-7	Wiki	150k	82.3	76.8	74.7	72.5	73.1	60.8	62.3	71.8
CC	150k	85.7	78.6	79.5	76.4	74.8	71.2	66.9	76.2
Multilingual models (translate-train-all)
XLM-7	Wiki	150k	84.6	80.1	80.2	75.7	78	68.7	66.7	76.3
CC	150k	87.2	82.5	82.9	79.7	80.4	75.7	71.5	80.0

Table 5: Multilingual versus monolingual models (BERT-BASE). We compare the performance of monolingual models (BERT) versus multilingual models (XLM) on seven languages, using a BERT-BASE architecture. We choose a vocabulary size of 40k and 150k for monolingual and multilingual models.

Model	D	#vocab	en	fr	de	ru	zh	sw	ur	Avg
单语言基线
BERT	Wiki	40k	84.5	78.6	80.0	75.5	77.7	60.1	57.3	73.4
CC	40k	86.7	81.2	81.2	78.2	79.5	70.8	65.1	77.5
多语言模型（跨语言迁移）
XLM-7	Wiki	150k	82.3	76.8	74.7	72.5	73.1	60.8	62.3	71.8
CC	150k	85.7	78.6	79.5	76.4	74.8	71.2	66.9	76.2
多语言模型（translate-train-all）
XLM-7	Wiki	150k	84.6	80.1	80.2	75.7	78	68.7	66.7	76.3
CC	150k	87.2	82.5	82.9	79.7	80.4	75.7	71.5	80.0

表 5：多语言对比单语言模型（BERT-BASE）。 在七种语言上用 BERT-BASE 架构比较单语言（BERT）与多语言（XLM）模型。单语言与多语言词表分别取 40k 与 150k。

5.4 Representation Learning for Low-resource Languages

5.4 低资源语言的表示学习

We observed in Table 5 that pretraining on Wikipedia for Swahili and Urdu performed similarly to a randomly initialized model; most likely due to the small size of the data for these languages. On the other hand, pretraining on CC improved performance by up to 10 points. This confirms our assumption that mBERT and XLM-100 rely heavily on cross-lingual transfer but do not model the low-resource languages as well as XLM-R. Specifically, in the translate-train-all setting, we observe that the biggest gains for XLM models trained on CC, compared to their Wikipedia counterparts, are on low-resource languages; 7% and 4.8% improvement on Swahili and Urdu respectively.

表 5 中我们观察到：在斯瓦希里语与乌尔都语的维基百科上预训练，表现接近随机初始化；很可能因这些语言数据过小。相反，在 CC 上预训练最多可提升约 10 分。这印证我们的假设：mBERT 与 XLM-100 高度依赖跨语言迁移，但对低资源语言的建模不如 XLM-R。具体在 translate-train-all 设定下，相对维基对应物，CC 上训练的 XLM 最大增益出现在低资源语言：斯瓦希里语与乌尔都语分别提升 7% 与 4.8%。

6 Conclusion

6 结论

In this work, we introduced XLM-R, our new state of the art multilingual masked language model trained on 2.5 TB of newly created clean CommonCrawl data in 100 languages. We show that it provides strong gains over previous multilingual models like mBERT and XLM on classification, sequence labeling and question answering. We exposed the limitations of multilingual MLMs, in particular by uncovering the high-resource versus low-resource trade-off, the curse of multilinguality and the importance of key hyperparameters. We also expose the surprising effectiveness of multilingual models over monolingual models, and show strong improvements on low-resource languages.

本文介绍 XLM-R：在新建的 2.5 TB 干净 CommonCrawl 数据、100 种语言上训练的当时最优多语言掩码语言模型。结果表明它在分类、序列标注与问答上相对 mBERT、XLM 等先前多语言模型有显著增益。我们揭示了多语言 MLM 的局限，尤其是高/低资源权衡、多语言诅咒与关键超参数的重要性；也展示了多语言相对单语言模型的意外有效性，以及在低资源语言上的显著改进。

References

参考文献

Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING , pages 1638–1649.

Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019 .

Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP .

Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In EMNLP . Association for Computational Linguistics.

Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL .

Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In LREC .

Huang et al. (2019) Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. ACL .

Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. TACL , 5:339–351.

Joulin et al. (2017) Armand Joulin, Edouard Grave, and Piotr Bojanowski Tomas Mikolov. 2017. Bag of tricks for efficient text classification. EACL 2017 , page 427.

Jozefowicz et al. (2016) Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410 .

Kudo (2018) Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In ACL , pages 66–75.

Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. EMNLP .

Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition . In NAACL , pages 260–270, San Diego, California. Association for Computational Linguistics.

Lample and Conneau (2019) Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. NeurIPS .

Lewis et al. (2019) Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475 .

Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 .

Mikolov et al. (2013a) Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013a. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168 .

Mikolov et al. (2013b) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In NIPS , pages 3111–3119.

Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation . In EMNLP , pages 1532–1543.

Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. NAACL .

Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In ACL .

Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training . URL https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf .

Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).

Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 .

Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. ACL .

Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text . In EMNLP , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.

Sang (2002) Erik F Sang. 2002. Introduction to the conll-2002 shared task: Language-independent named entity recognition. CoNLL .

Schuster et al. (2019) Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. NAACL .

Siddhant et al. (2019) Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Arivazhagan, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. 2019. Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. AAAI .

Singh et al. (2019) Jasdeep Singh, Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2019. Xlda: Cross-lingual data augmentation for natural language inference and question answering. arXiv preprint arXiv:1905.11471 .

Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , pages 1631–1642.

Tan et al. (2019) Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. ICLR .

Tjong Kim Sang and De Meulder (2003) Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: language-independent named entity recognition. In CoNLL , pages 142–147. Association for Computational Linguistics.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems , pages 6000–6010.

Wang et al. (2018) Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 .

Wenzek et al. (2019) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2019. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359 .

Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2nd Workshop on Evaluating Vector-Space Representations for NLP .

Wu et al. (2019) Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Emerging cross-lingual structure in pretrained language models. ACL .

Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. EMNLP .

Xie et al. (2019) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848 .

Appendix

附录

Appendix A Languages and statistics for CC-100 used by XLM-R

附录 A XLM-R 所用 CC-100 的语言与统计

In this section we present the list of languages in the CC-100 corpus we created for training XLM-R. We also report statistics such as the number of tokens and the size of each monolingual corpus.

本节给出为训练 XLM-R 创建的 CC-100 语料中的语言列表，并报告各单语言语料的 token 数与规模等统计。

ISO code	Language	Tokens (M)	Size (GiB)	ISO code	Language	Tokens (M)	Size (GiB)
af	Afrikaans	242	1.3	lo	Lao	17	0.6
am	Amharic	68	0.8	lt	Lithuanian	1835	13.7
ar	Arabic	2869	28.0	lv	Latvian	1198	8.8
as	Assamese	5	0.1	mg	Malagasy	25	0.2
az	Azerbaijani	783	6.5	mk	Macedonian	449	4.8
be	Belarusian	362	4.3	ml	Malayalam	313	7.6
bg	Bulgarian	5487	57.5	mn	Mongolian	248	3.0
bn	Bengali	525	8.4	mr	Marathi	175	2.8
-	Bengali Romanized	77	0.5	ms	Malay	1318	8.5
br	Breton	16	0.1	my	Burmese	15	0.4
bs	Bosnian	14	0.1	my	Burmese	56	1.6
ca	Catalan	1752	10.1	ne	Nepali	237	3.8
cs	Czech	2498	16.3	nl	Dutch	5025	29.3
cy	Welsh	141	0.8	no	Norwegian	8494	49.0
da	Danish	7823	45.6	om	Oromo	8	0.1
de	German	10297	66.6	or	Oriya	36	0.6
el	Greek	4285	46.9	pa	Punjabi	68	0.8
en	English	55608	300.8	pl	Polish	6490	44.6
eo	Esperanto	157	0.9	ps	Pashto	96	0.7
es	Spanish	9374	53.3	pt	Portuguese	8405	49.1
et	Estonian	843	6.1	ro	Romanian	10354	61.4
eu	Basque	270	2.0	ru	Russian	23408	278.0
fa	Persian	13259	111.6	sa	Sanskrit	17	0.3
fi	Finnish	6730	54.3	sd	Sindhi	50	0.4
fr	French	9780	56.8	si	Sinhala	243	3.6
fy	Western Frisian	29	0.2	sk	Slovak	3525	23.2
ga	Irish	86	0.5	sl	Slovenian	1669	10.3
gd	Scottish Gaelic	21	0.1	so	Somali	62	0.4
gl	Galician	495	2.9	sq	Albanian	918	5.4
gu	Gujarati	140	1.9	sr	Serbian	843	9.1
ha	Hausa	56	0.3	su	Sundanese	10	0.1
he	Hebrew	3399	31.6	sv	Swedish	77.8	12.1
hi	Hindi	1715	20.2	sw	Swahili	275	1.6
-	Hindi Romanized	88	0.5	ta	Tamil	595	12.2
hr	Croatian	3297	20.5	-	Tamil Romanized	36	0.3
hu	Hungarian	7807	58.4	te	Telugu	249	4.7
hy	Armenian	421	5.5	-	Telugu Romanized	39	0.3
id	Indonesian	22704	148.3	th	Thai	1834	71.7
is	Icelandic	505	3.2	tl	Filipino	556	3.1
it	Italian	4983	30.2	tr	Turkish	2736	20.9
ja	Japanese	530	69.3	ug	Uyghur	27	0.4
jv	Javanese	24	0.2	uk	Ukrainian	6.5	84.6
ka	Georgian	469	9.1	ur	Urdu	730	5.7
kk	Kazakh	476	6.4	-	Urdu Romanized	85	0.5
km	Khmer	36	1.5	uz	Uzbek	91	0.7
kn	Kannada	169	3.3	vi	Vietnamese	24757	137.3
ko	Korean	5644	54.2	xh	Xhosa	13	0.1
ku	Kurdish (Kurmanji)	66	0.4	yi	Yiddish	34	0.3
ky	Kyrgyz	94	1.2	zh	Chinese (Simplified)	259	46.9
la	Latin	390	2.5	zh	Chinese (Traditional)	176	16.6

Table 6: Languages and statistics of the CC-100 corpus. We report the list of 100 languages and include the number of tokens (Millions) and the size of the data (in GiB) for each language. Note that we also include romanized variants of some non latin languages such as Bengali, Hindi, Tamil, Telugu and Urdu.

ISO code	语言	Token 数 (M)	规模 (GiB)	ISO code	语言	Token 数 (M)	规模 (GiB)
af	Afrikaans	242	1.3	lo	Lao	17	0.6
am	Amharic	68	0.8	lt	Lithuanian	1835	13.7
ar	Arabic	2869	28.0	lv	Latvian	1198	8.8
as	Assamese	5	0.1	mg	Malagasy	25	0.2
az	Azerbaijani	783	6.5	mk	Macedonian	449	4.8
be	Belarusian	362	4.3	ml	Malayalam	313	7.6
bg	Bulgarian	5487	57.5	mn	Mongolian	248	3.0
bn	Bengali	525	8.4	mr	Marathi	175	2.8
-	Bengali Romanized	77	0.5	ms	Malay	1318	8.5
br	Breton	16	0.1	my	Burmese	15	0.4
bs	Bosnian	14	0.1	my	Burmese	56	1.6
ca	Catalan	1752	10.1	ne	Nepali	237	3.8
cs	Czech	2498	16.3	nl	Dutch	5025	29.3
cy	Welsh	141	0.8	no	Norwegian	8494	49.0
da	Danish	7823	45.6	om	Oromo	8	0.1
de	German	10297	66.6	or	Oriya	36	0.6
el	Greek	4285	46.9	pa	Punjabi	68	0.8
en	English	55608	300.8	pl	Polish	6490	44.6
eo	Esperanto	157	0.9	ps	Pashto	96	0.7
es	Spanish	9374	53.3	pt	Portuguese	8405	49.1
et	Estonian	843	6.1	ro	Romanian	10354	61.4
eu	Basque	270	2.0	ru	Russian	23408	278.0
fa	Persian	13259	111.6	sa	Sanskrit	17	0.3
fi	Finnish	6730	54.3	sd	Sindhi	50	0.4
fr	French	9780	56.8	si	Sinhala	243	3.6
fy	Western Frisian	29	0.2	sk	Slovak	3525	23.2
ga	Irish	86	0.5	sl	Slovenian	1669	10.3
gd	Scottish Gaelic	21	0.1	so	Somali	62	0.4
gl	Galician	495	2.9	sq	Albanian	918	5.4
gu	Gujarati	140	1.9	sr	Serbian	843	9.1
ha	Hausa	56	0.3	su	Sundanese	10	0.1
he	Hebrew	3399	31.6	sv	Swedish	77.8	12.1
hi	Hindi	1715	20.2	sw	Swahili	275	1.6
-	Hindi Romanized	88	0.5	ta	Tamil	595	12.2
hr	Croatian	3297	20.5	-	Tamil Romanized	36	0.3
hu	Hungarian	7807	58.4	te	Telugu	249	4.7
hy	Armenian	421	5.5	-	Telugu Romanized	39	0.3
id	Indonesian	22704	148.3	th	Thai	1834	71.7
is	Icelandic	505	3.2	tl	Filipino	556	3.1
it	Italian	4983	30.2	tr	Turkish	2736	20.9
ja	Japanese	530	69.3	ug	Uyghur	27	0.4
jv	Javanese	24	0.2	uk	Ukrainian	6.5	84.6
ka	Georgian	469	9.1	ur	Urdu	730	5.7
kk	Kazakh	476	6.4	-	Urdu Romanized	85	0.5
km	Khmer	36	1.5	uz	Uzbek	91	0.7
kn	Kannada	169	3.3	vi	Vietnamese	24757	137.3
ko	Korean	5644	54.2	xh	Xhosa	13	0.1
ku	Kurdish (Kurmanji)	66	0.4	yi	Yiddish	34	0.3
ky	Kyrgyz	94	1.2	zh	Chinese (Simplified)	259	46.9
la	Latin	390	2.5	zh	Chinese (Traditional)	176	16.6

表 6：CC-100 语料的语言与统计。 列出 100 种语言，并给出各语言 token 数（百万）与数据规模（GiB）。注意也包含部分非拉丁文字语言的罗马化变体，如孟加拉、印地、泰米尔、泰卢固与乌尔都。

Appendix B Model Architectures and Sizes

附录 B 模型架构与规模

As we showed in section 5, capacity is an important parameter for learning strong cross-lingual representations. In the table below, we list multiple monolingual and multilingual models used by the research community and summarize their architectures and total number of parameters.

如第 5 节所示，容量是学习强跨语言表示的重要参数。下表列出研究社区使用的多种单语言与多语言模型，并汇总其架构与总参数量。

Model	#lgs	tokenization	L	$H_{m}$	$H_{ff}$	A	V	#params
BERTBase	1	WordPiece	12	768	3072	12	30k	110M
BERTLarge	1	WordPiece	24	1024	4096	16	30k	335M
mBERT	104	WordPiece	12	768	3072	12	110k	172M
RoBERTaBase	1	bBPE	12	768	3072	8	50k	125M
RoBERTa	1	bBPE	24	1024	4096	16	50k	355M
XLM-15	15	BPE	12	1024	4096	8	95k	250M
XLM-17	17	BPE	16	1280	5120	16	200k	570M
XLM-100	100	BPE	16	1280	5120	16	200k	570M
Unicoder	15	BPE	12	1024	4096	8	95k	250M
XLM-R Base	100	SPM	12	768	3072	12	250k	270M
XLM-R	100	SPM	24	1024	4096	16	250k	550M
GPT2	1	bBPE	48	1600	6400	32	50k	1.5B
wide-mmNMT	103	SPM	12	2048	16384	32	64k	3B
deep-mmNMT	103	SPM	24	1024	16384	32	64k	3B
T5-3B	1	WordPiece	24	1024	16384	32	32k	3B
T5-11B	1	WordPiece	24	1024	65536	32	32k	11B

Table 7: Details on model sizes. We show the tokenization used by each Transformer model, the number of layers L, the number of hidden states of the model $H_{m}$, the dimension of the feed-forward layer $H_{ff}$, the number of attention heads A, the size of the vocabulary V and the total number of parameters #params. For Transformer encoders, the number of parameters can be approximated by $4LH_{m}^{2}+2LH_{m}H_{ff}+VH_{m}$. GPT2 numbers are from Radford et al. 2019, mm-NMT models are from the work of Arivazhagan et al. 2019 on massively multilingual neural machine translation (mmNMT), and T5 numbers are from Raffel et al. 2019. While XLM-R is among the largest models partly due to its large embedding layer, it has a similar number of parameters than XLM-100, and remains significantly smaller that recently introduced Transformer models for multilingual MT and transfer learning. While this table gives more hindsight on the difference of capacity of each model, note it does not highlight other critical differences between the models.

Model	#lgs	tokenization	L	$H_{m}$	$H_{ff}$	A	V	#params
BERTBase	1	WordPiece	12	768	3072	12	30k	110M
BERTLarge	1	WordPiece	24	1024	4096	16	30k	335M
mBERT	104	WordPiece	12	768	3072	12	110k	172M
RoBERTaBase	1	bBPE	12	768	3072	8	50k	125M
RoBERTa	1	bBPE	24	1024	4096	16	50k	355M
XLM-15	15	BPE	12	1024	4096	8	95k	250M
XLM-17	17	BPE	16	1280	5120	16	200k	570M
XLM-100	100	BPE	16	1280	5120	16	200k	570M
Unicoder	15	BPE	12	1024	4096	8	95k	250M
XLM-R Base	100	SPM	12	768	3072	12	250k	270M
XLM-R	100	SPM	24	1024	4096	16	250k	550M
GPT2	1	bBPE	48	1600	6400	32	50k	1.5B
wide-mmNMT	103	SPM	12	2048	16384	32	64k	3B
deep-mmNMT	103	SPM	24	1024	16384	32	64k	3B
T5-3B	1	WordPiece	24	1024	16384	32	32k	3B
T5-11B	1	WordPiece	24	1024	65536	32	32k	11B

表 7：模型规模细节。 给出各 Transformer 使用的分词、层数 L、模型隐状态数 $H_{m}$、前馈层维度 $H_{ff}$、注意力头数 A、词表大小 V 与总参数量 #params。对 Transformer 编码器，参数量可近似为 $4LH_{m}^{2}+2LH_{m}H_{ff}+VH_{m}$。GPT2 数字来自 Radford et al. 2019，mm-NMT 来自 Arivazhagan et al. 2019 的大规模多语言神经机器翻译（mmNMT），T5 来自 Raffel et al. 2019。XLM-R 因大嵌入层而位居较大模型之列，但参数量与 XLM-100 相近，仍显著小于近期多语言机器翻译与迁移学习引入的 Transformer。本表有助于理解各模型容量差异，但未突出模型间其他关键差别。