ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

ELECTRA：把文本编码器当作判别器而非生成器来预训练（ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators）

Kevin Clark Affiliation:  Stanford University Email:  kevclark@cs.stanford.edu    Minh-Thang Luong Affiliation:  Google Brain Email:  thangluong@google.com    Quoc V. Le Affiliation:  Google Brain Email:  qvl@google.com    Christopher D. Manning Affiliation:  Stanford University & CIFAR Fellow Email:  manning@cs.stanford.edu

Abstract

摘要

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

诸如 BERT 的掩码语言模型（MLM）预训练方法，通过把部分词元替换为 [MASK] 来破坏输入，再训练模型恢复原文。与之相对，我们提出一种更具样本效率的预训练任务：替换词检测（replaced token detection）。我们先用小生成器网络生成似真的词元替换，再训练一个判别器去预测输入中每个位置是否被替换过。与 MLM 只从约 15% 被掩码词元学习不同，该判别任务是在所有输入词元上定义损失，因而更高效。在相同的模型规模、数据和算力下，用该任务预训练的上下文编码器大幅优于 BERT；同样的优异表现也出现在算力远小于 GPT 的小模型设定，以及与 RoBERTa 规模相当的大模型设定。

1 Introduction

1 引言

Current state-of-the-art representation learning methods for language can be viewed as learning denoising autoencoders (Vincent et al., 2008). They select a small subset of the unlabeled input sequence (typically 15%), mask the identities of those tokens (e.g., BERT; Devlin et al. (2019)) or attention to those tokens (e.g., XLNet; Yang et al. (2019)), and then train the network to recover the original input. While more effective than conventional language-model pre-training due to learning bidirectional representations, these masked language modeling (MLM) approaches incur a substantial compute cost because the network only learns from 15% of the tokens per example.

目前最先进的语言表述学习方法可视为学习自动编码器(Vincent等人,2008年)。 他们选择一个没有标签的输入序列的小子集(典型的15%),掩盖这些符号的身份(例如BERT;Devlin等人(2019年))或注意这些符号(例如XLNet;杨等人(2019年)),然后训练网络以恢复原始输入. 虽然由于学习双向表现而比常规语言模型预训练更有效,但是这些蒙面语言模型(MLM)方法却需要大量的计算成本,因为网络只学习15%的每个符号.

As an alternative, we propose replaced token detection, a pre-training task in which the model learns to distinguish real input tokens from plausible but synthetically generated replacements. Instead of masking, our method corrupts the input by replacing some tokens with samples from a proposal distribution, which is typically the output of a small masked language model. This corruption procedure solves a mismatch in BERT (although not in XLNet) where the network sees artificial [MASK] tokens during pre-training but not when being fine-tuned on downstream tasks. We then pre-train the network as a discriminator that predicts for every token whether it is an original or a replacement. In contrast, MLM trains the network as a generator that predicts the original identities of the corrupted tokens. A key advantage of our discriminative task is that the model learns from all input tokens instead of just the small masked-out subset, making it more computationally efficient. Although our approach is reminiscent of training the discriminator of a GAN, our method is not adversarial in that the generator producing corrupted tokens is trained with maximum likelihood due to the difficulty of applying GANs to text (Caccia et al., 2018).

作为一种替代办法,我们提议替换代号检测, 一种训练前的任务,即模型学习区分真实输入代号与假想但合成生成的代号. 我们的方法不是遮掩,而是通过用一个提案分发的样本来取代一些符号来腐蚀输入,这通常是一个小遮掩语言模型的输出. 这种腐败程序解决了BERT(虽然不是在XLNet)中的一种不匹配现象,即网络在预训练期间看到人造[MASK]符号,但在下游任务上微调时则不会. 然后我们作为歧视者对网络进行预训练, 预言它是原始的还是替代的。 相比之下,MLM将网络训练成一个预测腐烂的代币原始身份的生成器. 我们的歧视性任务的一个关键优势是,模型从所有输入符号中学习,而不是仅仅从小的遮掩出来的子集中学习,使其在计算上更有效率. 尽管我们的做法让人想起训练GAN歧视者,但是我们的方法并不是对抗性的,因为由于GANs在文字上应用的难度,生成腐烂的代币的生成者受到了最大可能性的训练(Caccia等人,2018年)。

We call our approach ELECTRA[^1] for “Efficiently Learning an Encoder that Classifies Token Replacements Accurately.” As in prior work, we apply it to pre-train Transformer text encoders (Vaswani et al., 2017) that can be fine-tuned on downstream tasks. Through a series of ablations, we show that learning from all input positions causes ELECTRA to train much faster than BERT. We also show ELECTRA achieves higher accuracy on downstream tasks when fully trained.

我们称我们的方针为“有效学习一个编码器,准确分类地分类托肯替换器”。 和之前的工作一样,我们将其应用于预列列车Transformer文字编码器(Vaswani等人,2017年),可以在下游任务上进行微调. 通过一系列的模拟,我们显示,从所有输入位置学习,使得ELECTRA的训练速度比BERT快得多. 我们还显示,ELECTRA在接受充分训练后,在下游任务上取得了更高的准确度.

Most current pre-training methods require large amounts of compute to be effective, raising concerns about their cost and accessibility. Since pre-training with more compute almost always results in better downstream accuracies, we argue an important consideration for pre-training methods should be compute efficiency as well as absolute downstream performance. From this viewpoint, we train ELECTRA models of various sizes and evaluate their downstream performance vs. their compute requirement. In particular, we run experiments on the GLUE natural language understanding benchmark (Wang et al., 2019) and SQuAD question answering benchmark (Rajpurkar et al., 2016). ELECTRA substantially outperforms MLM-based methods such as BERT and XLNet given the same model size, data, and compute (see Figure 1). For example, we build an ELECTRA-Small model that can be trained on 1 GPU in 4 days.[^2] ELECTRA-Small outperforms a comparably small BERT model by 5 points on GLUE, and even outperforms the much larger GPT model (Radford et al., 2018). Our approach also works well at large scale, where we train an ELECTRA-Large model that performs comparably to RoBERTa (Liu et al., 2019) and XLNet (Yang et al., 2019), despite having fewer parameters and using 1/4 of the compute for training. Training ELECTRA-Large further results in an even stronger model that outperforms ALBERT (Lan et al., 2019) on GLUE and sets a new state-of-the-art for SQuAD 2.0. Taken together, our results indicate that the discriminative task of distinguishing real data from challenging negative samples is more compute-efficient and parameter-efficient than existing generative approaches for language representation learning.

目前大多数训练前方法都需要大量计算才能有效,这引起了对其成本和可获得性的关切。 由于更计算好的预训练几乎总能产生更好的下游速率,我们主张,对预训练方法的重要考虑应当是计算效率和下游绝对性能。 从这个角度看,我们训练各种大小的ELECTRA模型,并评价其下游性能与计算要求. 特别是,我们进行了关于GLUE自然语言理解基准(Wang等人,2019年)和SQuAD问答基准(Rajpurkar等人,2016年)的实验。 ELECTRA在模型大小、数据和计算上都大大超过基于MLM的方法,如BERT和XLNet(见图1)。 例如,我们建造了一个ELECTRA-Small型号,可以在4天内在1GPU上进行训练. [1]. ^2),ELECTRA-Small在GLUE上以5分的比对小的BERT模型,甚至超过了更大的GPT模型(Radford等人,2018年). 尽管参数较少,而且使用1/4的计算法进行训练,但我们的做法也大范围地发挥作用,我们训练了一种与RoBERTa(Liu等人,2019年)和XLNet(Yang等人,2019年)具有可比性的ELECTRA-Large模型。 ELECTRA-Large训练进一步形成了一个比ALBERT(Lan等人,2019年)在GLUE上表现得更强的模型,并为SQuAD 2.0设定了新的最先进水平. 总而言之,我们的结果表明,区分真实数据与挑战性负面样本的区别性任务,比现有语言代表性学习的基因方法更具有计算效率和参数效率。

Figure 1: Replaced token detection pre-training consistently outperforms masked language model pre-training given the same compute budget. The left figure is a zoomed-in view of the dashed box.

图1:在计算预算相同的情况下,更换符号检测预训练的成绩始终高于掩盖的语言模式预训练。 左图是缩放的视线 显示破损的盒子。

2 Method

2 方法

Figure 2: An overview of replaced token detection. The generator can be any model that produces an output distribution over tokens, but we usually use a small masked language model that is trained jointly with the discriminator. Although the models are structured like in a GAN, we train the generator with maximum likelihood rather than adversarially due to the difficulty of applying GANs to text. After pre-training, we throw out the generator and only fine-tune the discriminator (the ELECTRA model) on downstream tasks.

图2:替换代号检测概况. 生成器可以是任何在符号上产生输出分布的模型,但我们通常使用一个与歧视者共同训练的小蒙面语言模型. 虽然模型的结构就像GAN一样,但是由于GANs在文本上的应用困难,我们以最大的可能性而不是对抗性来训练发电机. 经过前期训练后,我们抛出发电机,只在下游任务上微调歧视者(ELECTRA模型).

We first describe the replaced token detection pre-training task; see Figure 2 for an overview. We suggest and evaluate several modeling improvements for this method in Section 3.2.

我们首先描述更换的代号检测训练前任务;概览见图2。 我们建议并评估第3.2节中对这一方法的若干改进模式。

Our approach trains two neural networks, a generator $G$ and a discriminator $D$. Each one primarily consists of an encoder (e.g., a Transformer network) that maps a sequence on input tokens $\bm{x}=[x_{1},...,x_{n}]$ into a sequence of contextualized vector representations $h(\bm{x})=[h_{1},...,h_{n}]$. For a given position $t$, (in our case only positions where $x_{t}=\texttt{[MASK]}$), the generator outputs a probability for generating a particular token $x_{t}$ with a softmax layer:

我们的方法训练两个神经网络,一个发电机$G$和一个歧视者$D$。 每个都主要由一个编码器(如Transformer网络)组成,将输入符$\bm{x}=[x_{1},...,x_{n}]$上的序列映射为上下文的矢量表示符$h(\bm{x})=[h_{1},...,h_{n}]$的序列. 对于一个给定位置$t$,(就我们而言,只有$x_{t}=\texttt{[MASK]}$的位置),生成器输出出一个带有软max层的特定符号$x_{t}$的概率:

$$\displaystyle p_{G}(x_{t}|\bm{x})=\text{exp}\left(e(x_{t})^{T}h_{G}(\bm{x})_{t}\right)/\sum_{x^{\prime}}\exp\left(e(x^{\prime})^{T}h_{G}(\bm{x})_{t}\right)$$

where $e$ denotes token embeddings. For a given position $t$, the discriminator predicts whether the token $x_{t}$ is “real,” i.e., that it comes from the data rather than the generator distribution, with a sigmoid output layer:

其中$e$表示符号嵌入。 对于一个给定位置$t$, 歧义者预测该符号$x_{t}$是否是"真实的",即它来自数据而不是生成器分布,带有sigmoid输出层:

$$\displaystyle D(\bm{x},t)=\text{sigmoid}(w^{T}h_{D}(\bm{x})_{t})$$

The generator is trained to perform masked language modeling (MLM). Given an input $\bm{x}=[x_{1},x_{2},...,x_{n}]$, MLM first select a random set of positions (integers between 1 and $n$) to mask out $\bm{m}=[m_{1},...,m_{k}]$.[^3] The tokens in the selected positions are replaced with a [MASK] token: we denote this as $\bm{x}^{\text{masked}}=\textsc{replace}(\bm{x},\bm{m},\texttt{[MASK]})$. The generator then learns to predict the original identities of the masked-out tokens. The discriminator is trained to distinguish tokens in the data from tokens that have been replaced by generator samples. More specifically, we create a corrupted example $\bm{x}^{\text{corrupt}}$ by replacing the masked-out tokens with generator samples and train the discriminator to predict which tokens in $\bm{x}^{\text{corrupt}}$ match the original input $\bm{x}$. Formally, model inputs are constructed according to

生成器经过训练,可以进行蒙面语言建模(MLM). 在输入$\bm{x}=[x_{1},x_{2},...,x_{n}]$的情况下,MLM首先选择一组随机位置(插入到1到$n$之间)来遮掩$\bm{m}=[m_{1},...,m_{k}]$. [^3]. 选定职位上的符号被替换为[MASK]符号:我们将此表示为$\bm{x}^{\text{masked}}=\textsc{replace}(\bm{x},\bm{m},\texttt{[MASK]})$. 生成器随后学会了预测蒙面出征物的原始身份. 歧视者接受训练,以区分数据中的符号与已被生成器样本取代的符号。 更具体地说,我们创建了一个腐败的例子$\bm{x}^{\text{corrupt}}$,用发电机样品取代蒙 听我说,歧视者可以预测$\bm{x}^{\text{corrupt}}$中的哪些符号与原始输入的$\bm{x}$相符。 在形式上,模型投入按照

$$\displaystyle m_{i}\sim\text{unif}\{1,n\}\text{ for }i=1\text{ to }k\hskip 28.45274pt\bm{x}^{\text{masked}}=\textsc{replace}(\bm{x},\bm{m},\texttt{[MASK]}) \\ \displaystyle\hat{x}_{i}\sim p_{G}(x_{i}|\bm{x}^{\text{masked}})\text{ for }i\in\bm{m}\hskip 28.45274pt\bm{x}^{\text{corrupt}}=\textsc{replace}(\bm{x},\bm{m},\hat{\bm{x}})$$

and the loss functions are

损失的功能是

$$\displaystyle\mathcal{L}_{\text{MLM}}(\bm{x},\theta_{G})=\mathop{\mathbb{E}}\left(\sum_{i\in\bm{m}}-\log p_{G}(x_{i}|\bm{x}^{\text{masked}})\right) \\ \displaystyle\mathcal{L}_{\text{Disc}}(\bm{x},\theta_{D})=\mathop{\mathbb{E}}\left(\sum_{t=1}^{n}-\mathbbm{1}(x^{\text{corrupt}}_{t}=x_{t})\log D(\bm{x}^{\text{corrupt}},t)-\mathbbm{1}(x^{\text{corrupt}}_{t}\neq x_{t})\log(1-D(\bm{x}^{\text{corrupt}},t))\right)$$

Although similar to the training objective of a GAN, there are several key differences. First, if the generator happens to generate the correct token, that token is considered “real” instead of “fake”; we found this formulation to moderately improve results on downstream tasks. More importantly, the generator is trained with maximum likelihood rather than being trained adversarially to fool the discriminator. Adversarially training the generator is challenging because it is impossible to back-propagate through sampling from the generator. Although we experimented circumventing this issue by using reinforcement learning to train the generator (see Appendix F), this performed worse than maximum-likelihood training. Lastly, we do not supply the generator with a noise vector as input, as is typical with a GAN.

虽然与GAN的训练目标相似,但存在几个关键差异. 首先,如果生成者碰巧产生irgent,则该irgent被视为“real”而不是“fake”;我们发现,这种提法在下游任务方面略有改进。 更重要的是,发电机的训练可能性最大,而不是通过对抗性训练来愚弄歧视者。 逆向训练发电机具有挑战性,因为无法通过从发电机取样来后退。 尽管我们通过使用强化学习来训练发电机来绕过这个问题(见附录F),但这表现比极限相似性训练更糟糕. 最后,我们不向发电机提供噪音矢量作为输入,典型的GAN就是如此。

We minimize the combined loss

我们把损失降到最低

$$\displaystyle\min_{\theta_{G},\theta_{D}}\sum_{\bm{x}\in\mathcal{X}}\mathcal{L}_{\text{MLM}}(\bm{x},\theta_{G})+\lambda\mathcal{L}_{\text{Disc}}(\bm{x},\theta_{D})$$

over a large corpus $\mathcal{X}$ of raw text. We approximate the expectations in the losses with a single sample. We don’t back-propagate the discriminator loss through the generator (indeed, we can’t because of the sampling step). After pre-training, we throw out the generator and fine-tune the discriminator on downstream tasks.

超过一大块$\mathcal{X}$原始文本。 我们用一个样本来估算损失的预期值。 我们不会通过发电机支持歧视者的损失(事实上, 经过预训练后,我们抛出发电机,在下游任务上微调歧视者.

3 Experiments

3 实验

3.1 Experimental Setup

3.1 实验设置

We evaluate on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) and Stanford Question Answering (SQuAD) dataset (Rajpurkar et al., 2016). GLUE contains a variety of tasks covering textual entailment (RTE and MNLI) question-answer entailment (QNLI), paraphrase (MRPC), question paraphrase (QQP), textual similarity (STS), sentiment (SST), and linguistic acceptability (CoLA). See Appendix  C for more details on the GLUE tasks. Our evaluation metrics are Spearman correlation for STS, Matthews correlation for CoLA, and accuracy for the other GLUE tasks; we generally report the average score over all tasks. For SQuAD, we evaluate on versions 1.1, in which models select the span of text answering a question, and 2.0, in which some questions are unanswerable by the passage. We use the standard evaluation metrics of Exact-Match (EM) and F1 scores. For most experiments we pre-train on the same data as BERT, which consists of 3.3 Billion tokens from Wikipedia and BooksCorpus (Zhu et al., 2015). However, for our Large model we pre-trained on the data used for XLNet (Yang et al., 2019), which extends the BERT dataset to 33B tokens by including data from ClueWeb (Callan et al., 2009), CommonCrawl, and Gigaword (Parker et al., 2011). All of the pre-training and evaluation is on English data, although we think it would be interesting to apply our methods to multilingual data in the future.

我们评价通用语言理解评价(GLUE)基准(Wang等人,2019年)和斯坦福问答(SQuAD)数据集(Rajpurkar等人,2016年). GLUE包含各种任务,包括文本内涵(RTE和MNLI)问答(QNLI),释义(MRPC),释义(QQP),释义相似(STS),情感(SST),语言可接受(CoLA). 关于GLUE任务的更多详情,见附录C。 我们的评价衡量标准是STS的斯皮尔曼相关,CoLA的马修斯相关,以及其他GLUE任务的准确性;我们一般报告所有任务的平均得分. 对于SQuAD,我们评价1.1版本,其中模型选择回答一个问题的文本跨度,2.0版本中有些问题无法通过段落回答. 我们使用精确评分(EM)和F1分的标准评价指标。 对于大多数实验,我们对与BERT相同的数据进行预训练,由来自维基百科和BooksCorpus(Zhu等人,2015年)的3.3亿词元组成. 然而,对于我们的大模型,我们预先对XLNet(Yang等人,2019年)所使用的数据进行了训练,将BERT数据集扩展至33B词元,包括来自ClueWeb(Callan等人,2009年)、CommonCrawl和Gigaword的数据(Parker等人,2011年)。 所有的训练前和评估都是在英语数据上进行的,不过我们认为,今后将我们的方法应用于多语种数据将是很有趣的。

Our model architecture and most hyperparameters are the same as BERT’s. For fine-tuning on GLUE, we add simple linear classifiers on top of ELECTRA. For SQuAD, we add the question-answering module from XLNet on top of ELECTRA, which is slightly more sophisticated than BERT’s in that it jointly rather than independently predicts the start and end positions and has a “answerability” classifier added for SQuAD 2.0. Some of our evaluation datasets are small, which means accuracies of fine-tuned models can vary substantially depending on the random seed. We therefore report the median of 10 fine-tuning runs from the same pre-trained checkpoint for each result. Unless stated otherwise, results are on the dev set. See the appendix for further training details and hyperparameter values.

我们的模型架构和大多数超参数与BERT相同. 对于GLUE的微调,我们在ELECTRA上方增加了简单的线性分类器. 对于SQuAD,我们添加来自XLNet的回答模块在ELECTRA上方,这个模块比BERT的更精细一些,因为它联合而非独立预测起始和结束位置,并为SQuAD 2.0添加了一个"可回答性"分类器. 我们的一些评价数据集是小的,这意味着精细调整模型的精密度会因随机种子的不同而有很大差异. 因此,我们报告,10个微调的中位数来自同一预先训练过的检查站。 除非另有说明,结果在dev集上。 关于进一步训练的细节和超参数值,见附录。

3.2 Model Extensions

3.2 模型扩展

We improve our method by proposing and evaluating several extensions to the model. Unless stated otherwise, these experiments use the same model size and training data as BERT-Base.

我们改进了方法,提议并评估了该模式的若干延伸。 除非另有说明,这些实验使用了与BERT-Base相同的模型尺寸和训练数据.

Weight Sharing  We propose improving the efficiency of the pre-training by sharing weights between the generator and discriminator. If the generator and discriminator are the same size, all of the transformer weights can be tied. However, we found it to be more efficient to have a small generator, in which case we only share the embeddings (both the token and positional embeddings) of the generator and discriminator. In this case we use embeddings the size of the discriminator’s hidden states.[^4] The “input” and “output” token embeddings of the generator are always tied as in BERT.

** 重量分享** 我们建议通过发电机与歧视者分享权重来提高预训练的效率。 如果发电机与歧视器的尺寸相同,所有变压器的重量都可以捆绑. 然而,我们发现拥有一个小型发电机的效率更高,在这种情况下,我们只分享发电机和歧视者的嵌入(象征和位置嵌入). 在这种情况下,我们使用嵌入歧视者隐藏状态的大小。[^4] 发电机的“输入”和“输出”符号嵌入始终与BERT一样捆绑在一起。

We compare the weight tying strategies when the generator is the same size as the discriminator. We train these models for 500k steps. GLUE scores are 83.6 for no weight tying, 84.3 for tying token embeddings, and 84.4 for tying all weights. We hypothesize that ELECTRA benefits from tied token embeddings because masked language modeling is particularly effective at learning these representations: while the discriminator only updates tokens that are present in the input or are sampled by the generator, the generator’s softmax over the vocabulary densely updates all token embeddings. On the other hand, tying all encoder weights caused little improvement while incurring the significant disadvantage of requiring the generator and discriminator to be the same size. Based on these findings, we use tied embeddings for further experiments in this paper.

我们比较了当发电机大小与歧视者相同时的重量绑定策略。 我们训练这些模型500千步。 GLUE的分数是83.6分无重量捆绑,84.3分是捆绑符嵌入,84.4分是所有重量捆绑. 我们假设ELECTRA从捆绑的符号嵌入中获益,因为蒙面语言模型在学习这些符号方面特别有效:虽然歧视者只更新输入中存在的或者由生成器取样的符号,但生成器在词汇上的软max密集更新所有符号嵌入。 另一方面,将所有编码器重量捆绑起来,没有带来什么改善,同时造成了要求生成器和歧视器大小相同的严重缺点。 基于这些发现,我们用捆绑的嵌入物在本文中做进一步实验.

Smaller Generators  If the generator and discriminator are the same size, training ELECTRA would take around twice as much compute per step as training only with masked language modeling. We suggest using a smaller generator to reduce this factor. Specifically, we make models smaller by decreasing the layer sizes while keeping the other hyperparameters constant. We also explore using an extremely simple “unigram” generator that samples fake tokens according their frequency in the train corpus. GLUE scores for differently-sized generators and discriminators are shown in the left of Figure 3. All models are trained for 500k steps, which puts the smaller generators at a disadvantage in terms of compute because they require less compute per training step. Nevertheless, we find that models work best with generators 1/4-1/2 the size of the discriminator. We speculate that having too strong of a generator may pose a too-challenging task for the discriminator, preventing it from learning as effectively. In particular, the discriminator may have to use many of its parameters modeling the generator rather than the actual data distribution. Further experiments in this paper use the best generator size found for the given discriminator size.

** 小型发电机** 32. 如果发电机廖家和歧视性 48. 训练ELECTRA每步的计算量是只有蒙面语言模型的训练量的两倍。 我们建议使用一个较小的发电机来减少这一因素。 具体来说,我们通过降低层积而使模型变小,同时保持其他超参数不变. 我们还探索使用一种极其简单的“单格”发电机,在火车装箱中按频率对假信使进行取样。 图3左侧显示不同大小的发电机和歧视器的GLUE分数。 所有型号都经过500k级阶梯的训练,这使得较小的发电机在计算上处于劣势,因为每个训练级要求的计算较少. 尽管如此,我们发现,这种模型与1/4-1/2的发电机相比,歧视者的规模是最佳的。 我们推测,发电机太强可能对歧视者构成挑战性太强的任务,使其无法有效地学习。 特别是,歧视者可能不得不使用其许多参数,而不是实际的数据分布。 本文中的进一步实验使用为给定的歧义大小找到的最佳发电机尺寸.

Figure 3: Left: GLUE scores for different generator/discriminator sizes (number of hidden units). Interestingly, having a generator smaller than the discriminator improves results. Right: Comparison of different training algorithms. As our focus is on efficiency, the x-axis shows FLOPs rather than train steps (e.g., ELECTRA is trained for fewer steps than BERT because it includes the generator).

图3 左:GLUE对不同生成器/差异器大小的分数(隐藏单位数). 有趣的是,拥有一个比歧视者小的发电机会提高效果. 对:不同训练算法的比较. BERT的步数比BERT的步数要少,因为它包括了生成器。

Training Algorithms  Lastly, we explore other training algorithms for ELECTRA, although these did not end up improving results. The proposed training objective jointly trains the generator and discriminator. We experiment with instead using the following two-stage training procedure:

** 训练算法** 最后,我们为ELECTRA探索其他训练算法,尽管这些算法最终没有改善结果。 拟议的训练目标共同训练产生者和歧视者。 我们试验采用以下两个阶段的训练程序:

1. Train only the generator with $\mathcal{L}_{\text{MLM}}$ for $n$ steps.
2. Initialize the weights of the discriminator with the weights of the generator. Then train the discriminator with $\mathcal{L}_{\text{Disc}}$ for $n$ steps, keeping the generator’s weights frozen.

1. 只用$\mathcal{L}_{\text{MLM}}$对发电机进行$n$步骤的训练。
2. 将歧视者的权重与发电机的权重初始化。 然后用$\mathcal{L}_{\text{Disc}}$对歧视者进行$n$步骤的训练,使发电机的重量冻结.

Note that the weight initialization in this procedure requires having the same size for the generator and discriminator. We found that without the weight initialization the discriminator would sometimes fail to learn at all beyond the majority class, perhaps because the generator started so far ahead of the discriminator. Joint training on the other hand naturally provides a curriculum for the discriminator where the generator starts off weak but gets better throughout training. We also explored training the generator adversarially as in a GAN, using reinforcement learning to accommodate the discrete operations of sampling from the generator. See Appendix F for details.

请注意,这一程序中的权重初始化要求生成者和歧视者具有相同的尺寸。 我们发现,如果没有加权初始化,歧视者有时根本无法在多数阶层之外学习,或许是因为产生者远比歧视者早。 另一方面,联合训练自然为歧视者提供课程,因为发电机开始时很弱,但在整个训练期间会更好。 我们还探索对发电机进行对抗性训练,如在GAN中,利用强化学习来适应发电机的分散取样作业。 详情见附录F。

Results are shown in the right of Figure 3. During two-stage training, downstream task performance notably improves after the switch from the generative to the discriminative objective, but does not end up outscoring joint training. Although still outperforming BERT, we found adversarial training to underperform maximum-likelihood training. Further analysis suggests the gap is caused by two problems with adversarial training. First, the adversarial generator is simply worse at masked language modeling; it achieves 58% accuracy at masked language modeling compared to 65% accuracy for an MLE-trained one. We believe the worse accuracy is mainly due to the poor sample efficiency of reinforcement learning when working in the large action space of generating text. Secondly, the adversarially trained generator produces a low-entropy output distribution where most of the probability mass is on a single token, which means there is not much diversity in the generator samples. Both of these problems have been observed in GANs for text in prior work (Caccia et al., 2018).

结果见图3右边。 在两阶段训练中,下游任务绩效明显改善,因为从遗传目标转向歧视目标,但最终没有超过联合训练。 虽然BERT的表现仍然优于BERT,但我们发现对抗训练表现不佳的极限相似训练. 进一步的分析表明,这一差距是由对抗性训练的两个问题造成的。 第一,对抗力发生器在蒙面语言建模时更差;在蒙面语言建模时,其精度达到58%,而经过 MLE 训练的一台则达到65%. 我们认为,差的准确性主要是因为在生成文本的大动作空间工作时加强学习的样本效率差。 第二,经过对抗训练的发电机产生低通量输出分布,其中概率质量大部分在单一的符号上,这意味着发电机样本中差异不大. 这两个问题在GANs中都曾被观察到,用于前作的文字(Caccia等人,2018年).

3.3 Small Models

3.3 小模型

As a goal of this work is to improve the efficiency of pre-training, we develop a small model that can be quickly trained on a single GPU. Starting with the BERT-Base hyperparameters, we shortened the sequence length (from 512 to 128), reduced the batch size (from 256 to 128), reduced the model’s hidden dimension size (from 768 to 256), and used smaller token embeddings (from 768 to 128). To provide a fair comparison, we also train a BERT-Small model using the same hyperparameters. We train BERT-Small for 1.5M steps, so it uses the same training FLOPs as ELECTRA-Small, which was trained for 1M steps.[^5] In addition to BERT, we compare against two less resource-intensive pre-training methods based on language modeling: ELMo (Peters et al., 2018) and GPT (Radford et al., 2018).[^6] We also show results for a base-sized ELECTRA model comparable to BERT-Base.

由于这项工作的目标是提高预训练的效率,我们开发了一个小型模式,可以快速在单一GPU上进行训练. 从BERT-Base超参数开始,我们缩短了序列长度(从512到128),将批量大小(从256到128),将模型的隐藏尺寸大小(从768到256)降低,并使用了较小的代币嵌入(从768到128). 为了进行公平的比较,我们还使用同样的超参数来训练一种BERT-Small型号. 我们训练BERT-Small为1.5M步,因此它使用与ELECTRA-Small相同的训练FLOP,它训练了1M步. [^5] 除了BERT之外,我们比较了基于语言模型的两种资源密集程度较低的训练前方法:ELMo(Peters等人,2018年)和GPT(Radford等人,2018年)。 我们还展示了与BERT-Base类似的基型ELECTRA模型的结果.

Results are shown in Table 1. See Appendix D for additional results, including stronger small-sized and base-sized models trained with more compute. ELECTRA-Small performs remarkably well given its size, achieving a higher GLUE score than other methods using substantially more compute and parameters. For example, it scores 5 points higher than a comparable BERT-Small model and even outperforms the much larger GPT model. ELECTRA-Small is trained mostly to convergence, with models trained for even less time (as little as 6 hours) still achieving reasonable performance. While small models distilled from larger pre-trained transformers can also achieve good GLUE scores (Sun et al., 2019b; Jiao et al., 2019), these models require first expending substantial compute to pre-train the larger teacher model. The results also demonstrate the strength of ELECTRA at a moderate size; our base-sized ELECTRA model substantially outperforms BERT-Base and even outperforms BERT-Large (which gets 84.0 GLUE score). We hope ELECTRA’s ability to achieve strong results with relatively little compute will broaden the accessibility of developing and applying pre-trained models in NLP.

结果见表1。 额外成果见附录D,包括经过更多计算训练的更强大的小型和基型模型。 ELECTRA-Small根据其体积表现显著,比使用更多计算和参数的其他方法达到更高的GLUE分数. 例如,它的得分比可比的BERT-Small模型高5分,甚至超过了更大的GBT模型. ELECTRA-Small的训练大多是为了趋同,而模型的训练时间更短(短于6小时),仍能取得合理的性能. 虽然从更大的预训练变压器中蒸馏出来的小型模型也可以达到良好的GLUE分数(Sun等人,2019b;Jiao等人,2019),但这些模型首先需要花费大量计算来预训练更大的教师模型. 结果还显示了ELECTRA在中等尺寸的强度;我们基型的ELECTRA模型大大超过BERT-Base,甚至超过BERT-Large(它得到84.0 GLUE分数). 我们希望ELECTRA能够用相对较少的计算来取得强有力的成果,

Model	Train / Infer FLOPs	Speedup	Params	Train Time + Hardware	GLUE
ELMo	3.3e18 / 2.6e10	19x / 1.2x	96M	14d on 3 GTX 1080 GPUs	71.2
GPT	4.0e19 / 3.0e10	1.6x / 0.97x	117M	25d on 8 P6000 GPUs	78.8
BERT-Small	1.4e18 / 3.7e9	45x / 8x	14M	4d on 1 V100 GPU	75.1
BERT-Base	6.4e19 / 2.9e10	1x / 1x	110M	4d on 16 TPUv3s	82.2
ELECTRA-Small	1.4e18 / 3.7e9	45x / 8x	14M	4d on 1 V100 GPU	79.9
50% trained	7.1e17 / 3.7e9	90x / 8x	14M	2d on 1 V100 GPU	79.0
25% trained	3.6e17 / 3.7e9	181x / 8x	14M	1d on 1 V100 GPU	77.7
12.5% trained	1.8e17 / 3.7e9	361x / 8x	14M	12h on 1 V100 GPU	76.0
6.25% trained	8.9e16 / 3.7e9	722x / 8x	14M	6h on 1 V100 GPU	74.1
ELECTRA-Base	6.4e19 / 2.9e10	1x / 1x	110M	4d on 16 TPUv3s	85.1

Table 1: Comparison of small models on the GLUE dev set. BERT-Small/Base are our implementation and use the same hyperparameters as ELECTRA-Small/Base. Infer FLOPs assumes single length-128 input. Training times should be taken with a grain of salt as they are for different hardware and with sometimes un-optimized code. ELECTRA performs well even when trained on a single GPU, scoring 5 GLUE points higher than a comparable BERT model and even outscoring the much larger GPT model.

* 相互合作。	** 列车/推断FLOP**	页:1	** 巴勒斯坦**	** 火车时间+硬件**	** 格卢**
ELMO( 爱尔摩语)	3.3e18 / 2.6e10 (中文(简体) ).	19x / 1.2x	96马	14d 在3 GTX 1080 GPU上	71.2
常规	4.0e19 / 3.0e10 (中文(简体) ).	1.6x / 0.97x	117门	25d在8个P6000GPU上	78.8
贝特小	16 . 16	45x / 8x	14分钟	1台V100 GPU上的4d	75.1
贝特-贝斯	6.4e19 79.9e10	1x / 1x	110海里	4d对16 TPUv3s	82.2
ELECTRA 小号	1.4e18 / 3.7e9 (韩语)	45x / 8x	14分钟	1台V100 GPU上的4d	79.9
训练50%	7.1e17/3.7e9 (中文(简体) ).	90x / 8x	14分钟	2d在1台V100 GPU	79.0
训练25%	3.6e17 / 3.7e9 (中文(简体) ).	181x / 8x	14分钟	1d在1 V100 GPU上	77.7
12.5%的训练	1.8e17 / 3.7e9 (中文(简体) ).	361x / 8x	14分钟	业已使用1台V100 GPU的12小时	76.0
6.25%接受训练	8.9e16 / 3.7e9 (韩语)	722x / 8x	14分钟	1台V100 GPU上的6小时	74.1
ELECTRA- Base 数据	6.4e19 / 2.9e10 (中文(简体) ).	1x / 1x	110海里	4d对16 TPUv3s	85.1

表1:GLUE dev集小模型比较. BERT-Small/Base是我们执行的,使用与ELECTRA-Small/Base相同的超参数. 推断FLOPs假设单长-128输入. 训练时间应该用一粒盐来进行,因为它们是不同的硬件,有时还使用不优化的代码。 ELECTRA即使在接受单一GPU的训练时也表现良好,评分5GLUE比可比的BERT模型高,甚至评分要高得多的GBT模型.

3.4 Large Models

3.4 大模型

We train big ELECTRA models to measure the effectiveness of the replaced token detection pre-training task at the large scale of current state-of-the-art pre-trained Transformers. Our ELECTRA-Large models are the same size as BERT-Large but are trained for much longer. In particular, we train a model for 400k steps (ELECTRA-400K; roughly 1/4 the pre-training compute of RoBERTa) and one for 1.75M steps (ELECTRA-1.75M; similar compute to RoBERTa). We use a batch size 2048 and the XLNet pre-training data. We note that although the XLNet data is similar to the data used to train RoBERTa, the comparison is not entirely direct. As a baseline, we trained our own BERT-Large model using the same hyperparameters and training time as ELECTRA-400K.

我们训练了大型的ELECTRA模型,以测量更换的代号检测预训练任务在目前最先进的预训练Transformer的大规模上的效果. 我们的ELECTRA-Large型号与BERT-Large型号相同,但训练时间更长. 特别是,我们为400k级阶梯(ELECTRA-400K;约1/4为RoBERTa的预训练计算)和1.75M级阶梯(ELECTRA-1.75M;与RoBERTA相似的计算)训练一个模型. 使用批量尺寸2048和XLNet预训练数据. 我们注意到,尽管XLNet的数据与用于训练RoBERTa的数据相似,但比较并不完全直接. 作为基线,我们使用与ELECTRA-400K相同的超参数表和训练时间,训练了自己的BERT-LARG模型.

Results on the GLUE dev set are shown in Table 2. ELECTRA-400K performs comparably to RoBERTa and XLNet. However, it took less than 1/4 of the compute to train ELECTRA-400K as it did to train RoBERTa and XLNet, demonstrating that ELECTRA’s sample-efficiency gains hold at large scale. Training ELECTRA for longer (ELECTRA-1.75M) results in a model that outscores them on most GLUE tasks while still requiring less pre-training compute. Surprisingly, our baseline BERT model scores notably worse than RoBERTa-100K, suggesting our models may benefit from more hyperparameter tuning or using the RoBERTa training data. ELECTRA’s gains hold on the GLUE test set (see Table 3), although these comparisons are less apples-to-apples due to the additional tricks employed by the models (see Appendix B).

GLUE dev集的结果见表2. ELECTRA-400K的性能与RoBERTa和XLNet具有可比性. 然而,训练ELECTRA-400K的计算不到1/4,就像训练RoBERTa和XLNet一样,表明ELECTRA的样本效率收益是大规模的。 更长时间的训练ELECTRA(ELECTRA-1.75M)导致一种模式,在大多数GLUE任务上比他们得分,同时仍然需要较少的预训练计算. 令人惊讶的是,我们的基线BERT模型得分明显低于RoBERTa-100K,这表明我们的模型可能受益于更多的超参数调制或使用RoBERTA训练数据。 ELECTRA的增益在GLUE测试集(见表3)上持有,尽管由于模型使用的额外诡计(见附录B),这些比较较少苹果对苹果.

Model	Train FLOPs	Params	CoLA	SST	MRPC	STS	QQP	MNLI	QNLI	RTE	Avg.
BERT	1.9e20 (0.27x)	335M	60.6	93.2	88.0	90.0	91.3	86.6	92.3	70.4	84.0
RoBERTa-100K	6.4e20 (0.90x)	356M	66.1	95.6	91.4	92.2	92.0	89.3	94.0	82.7	87.9
RoBERTa-500K	3.2e21 (4.5x)	356M	68.0	96.4	90.9	92.1	92.2	90.2	94.7	86.6	88.9
XLNet	3.9e21 (5.4x)	360M	69.0	97.0	90.8	92.2	92.3	90.8	94.9	85.9	89.1
BERT (ours)	7.1e20 (1x)	335M	67.0	95.9	89.1	91.2	91.5	89.6	93.5	79.5	87.2
ELECTRA-400K	7.1e20 (1x)	335M	69.3	96.0	90.6	92.1	92.4	90.5	94.5	86.8	89.0
ELECTRA-1.75M	3.1e21 (4.4x)	335M	69.1	96.9	90.8	92.6	92.4	90.9	95.0	88.0	89.5

Table 2: Comparison of large models on the GLUE dev set. ELECTRA and RoBERTa are shown for different numbers of pre-training steps, indicated by the numbers after the dashes. ELECTRA performs comparably to XLNet and RoBERTa when using less than 1/4 of their pre-training compute and outperforms them when given a similar amount of pre-training compute. BERT dev results are from Clark et al. (2019).

** 模式**	** 训练 FLOPs**	** 巴勒斯坦**	页:1	页:1	** một以《公约》为主题的公约。	页:1	页:1	** 蒙古**	页:1	页:1	页:1
贝特	1.9e20 (0.27x) (中文(简体) ).	335海里	60.6	93.2	88.0	90.0	91.3	86.6	92.3	70.4	84.0
RoBERTa-100K	6.凳20 (0.90x) (中文(简体) ).	356海里	66.1	95.6	页:1	92.2	92.0	89.3	94.0	82.7	87.9
RoBERTa-500K	的人口为3.2e21(4.5x)	决定356M	68.0	96.4	90.9	92.1	92.2	90.2	94.7	86.6	88.9
XLNet 网络	3.9e21 (5.4x) (中文(简体) ).	360马力	69.0	页:1	90.8	92.2	92.3	90.8	94.9	85.9	89.1
BERT( 我们)	7.1e20(1x) (中文(简体) ).	335海里	67.0	95.9	89.1	91.2	91.5	89.6	93.5	79.5	87.2
ظاد ال ال ال ال .	7.1e20(1x) (中文(简体) ).	335海里	页:1	96.0	90.6	92.1	92.4 国家	90.5	94.5	86.8	89.0
电子数据交换-1.75M	3.1e21 (4.4x) (4.4x) (3.4e) (4.4x) (4.4x) (3.4e) (4.4e)) (4.4e) (4.4e) (4.4)) (4.4e) (4.4)) (4.4)) (4.4) (4.4)) (4.4)) (4.4) (4.4))) (4.4)) (4.4) (4.4))(4.4))(4.4))(3.e(4.e)(4.)(4.))(4.(4.)))(4.(4.))(4.(4.)))(4.(4.))(4.(4.)(4.))(4.(4.)(4.))(4.(4.)(4.)(4.)(4.)(4.(4.))(4.(4.)))(3.(e(4.(4.)(4.)(4.)(e)(e(4.)(4.)(e)(4.)(e)(e)(e)(e)((e)(e))(	335海里	69.1	96.9	90.8	92.6 国家	92.4 国家	页:1	页:1	** ** 88.0** (单位:千美元)	* ** 89.5**

表2:GLUE dev集上大型模型的比较. ELECTRA和RoBERTA显示不同的训练前步骤数量,以破折号后的数字表示. ELECTRA在使用不到1/4的预训练计算时,可比较地执行XLNet和RoBERTa,在给予类似数量的预训练计算时,其性能超过它们. BERT dev结果来自克拉克等人(2019年).

Model	Train FLOPs	CoLA	SST	MRPC	STS	QQP	MNLI	QNLI	RTE	WNLI	Avg.*	Score
BERT	1.9e20 (0.06x)	60.5	94.9	85.4	86.5	89.3	86.7	92.7	70.1	65.1	79.8	80.5
RoBERTa	3.2e21 (1.02x)	67.8	96.7	89.8	91.9	90.2	90.8	95.4	88.2	89.0	88.1	88.1
ALBERT	3.1e22 (10x)	69.1	97.1	91.2	92.0	90.5	91.3	–	89.2	91.8	89.0	–
XLNet	3.9e21 (1.26x)	70.2	97.1	90.5	92.6	90.4	90.9	–	88.5	92.5	89.1	–
ELECTRA	3.1e21 (1x)	71.7	97.1	90.7	92.5	90.8	91.3	95.8	89.8	92.5	89.5	89.4

Table 3: GLUE test-set results for large models. Models in this table incorporate additional tricks such as ensembling to improve scores (see Appendix B for details). Some models do not have QNLI scores because they treat QNLI as a ranking task, which has recently been disallowed by the GLUE benchmark. To compare against these models, we report the average score excluding QNLI (Avg.*) in addition to the GLUE leaderboard score (Score). “ELECTRA” and “RoBERTa” refer to the fully-trained ELECTRA-1.75M and RoBERTa-500K models.

** 模式**	** 训练 FLOPs**	页:1	页:1	** MRPC **	页:1	页:1	页:1	页:1	页:1	** 妇联**	页:1	** 记分**
贝特	1.9e20 (0.06x) (中文(简体) ).	60.5	94.9	85.4	86.5	89.3	86.7	92.7	70.1	65.1	79.8	80.5
RoBERTa	3.2e21 (1.02x) (中文(简体) ).	67.8	96.7	89.8	91.9	90.2	90.8	95.4	88.2	89.0	88.1	88.1
阿尔伯特	3.1e22(10杂种)	69.1	页:1	91.2 108.2 **	92.0	90.5	91.3 当我们看到时,就应注意:	- 说吧	89.2	91.8	89.0	- 说吧
XLNet 网络	3.9e21 (1.26x) (中文(简体) ).	70.2	页:1	90.5	92.6 国家	90.4	90.9	- 别这样	88.5	92.5 经常预算:	89.1	- 说吧
选举	3.1e21(1x) (中文(简体) ).	还将在2007年1月1日至2007年6月30日期间完成。	页:1	90.7	92.5	90.8 (单位:千美元)	91.3 经常预算:	方面的努力	页:1	92.5 经常预算:	* ** 89.5**	页:1

表3:大型模型的GLUE测试集结果. 本表中的模型包含一些额外的技巧,如提高分数的综艺(详见附录B). 一些模型没有QNLI的分数,因为它们将QNLI视为排位任务,这最近被GLUE基准否决. 与这些模型相比,我们报告除GLUE领导板积分(Score)外,不包括QNLI(Avg.*)的平均得分. “ELECTRA”和“ROBERTA”是指训练有素的ELECTRA-1.75M和RoBERTA-500K型号。

Model	Train FLOPs	Params	SQuAD 1.1 dev….	SQuAD 2.0 dev….	SQuAD 2.0 test
EM	F1	EM	F1	EM	F1
BERT-Base	6.4e19 (0.09x)	110M	80.8	88.5	–	–	–	–
BERT	1.9e20 (0.27x)	335M	84.1	90.9	79.0	81.8	80.0	83.0
SpanBERT	7.1e20 (1x)	335M	88.8	94.6	85.7	88.7	85.7	88.7
XLNet-Base	6.6e19 (0.09x)	117M	81.3	–	78.5	–	–	–
XLNet	3.9e21 (5.4x)	360M	89.7	95.1	87.9	90.6	87.9	90.7
RoBERTa-100K	6.4e20 (0.90x)	356M	–	94.0	–	87.7	–	–
RoBERTa-500K	3.2e21 (4.5x)	356M	88.9	94.6	86.5	89.4	86.8	89.8
ALBERT	3.1e22 (44x)	235M	89.3	94.8	87.4	90.2	88.1	90.9
BERT (ours)	7.1e20 (1x)	335M	88.0	93.7	84.7	87.5	–	–
ELECTRA-Base	6.4e19 (0.09x)	110M	84.5	90.8	80.5	83.3	–	–
ELECTRA-400K	7.1e20 (1x)	335M	88.7	94.2	86.9	89.6	–	–
ELECTRA-1.75M	3.1e21 (4.4x)	335M	89.7	94.9	88.0	90.6	88.7	91.4

Table 4: Results on the SQuAD for non-ensemble models.

** 模式**	** 训练 FLOPs**	** 巴勒斯坦**	** SQuAD 1.1 dev. **	** SQuAD 2.0 dev. ** (中文(简体) ).	** SQuAD 2.0测试**
电子邮件	F1	电子邮件	F1	电子邮件	F1
贝特-贝斯	6.4e19 (0.09x) (中文(简体) ).	110海里	80.8	88.5	- 说吧	- 说吧	- 说吧	- 说吧
贝特	1.9e20 (0.27x) (中文(简体) ).	335海里	84.1	90.9	79.0	81.8	80.0	83.0
弹簧	7.1e20(1x) (中文(简体) ).	335海里	88.8	94.6	85.7	88.7	85.7	88.7
中确定的 XLNet-基准	6.6e19 (0.09x) (中文(简体) ).	既然是一万一千七百万	81.3	- 说吧	78.5	- 说吧	- 说吧	- 说吧
XLNet 网络	3.9e21 (5.4x) (中文(简体) ).	360马力	89.7	页:1	87.9	页:1	87.9	90.7
RoBERTa-100K	6.4e20 (0.90x) (英语).	356海里	- 说吧	94.0	- 说吧	87.7	- 说吧	- 说吧
RoBERTa-500K	3.2e21 (4.5x) (中文(简体) ).	356海里	88.9	94.6	86.5	89.4	86.8	89.8
阿尔伯特	3.1e22(44x) (中文(简体) ).	235M型导弹	89.3	94.8	87.4	90.2	88.1	90.9
BERT( 我们)	7.1e20(1x) (中文(简体) ).	335海里	88.0	93.7	84.7	87.5	- 说吧	- 说吧
ELECTRA- Base 身亡	6.4e19 (0.09x) (中文(简体) ).	课题110M	84.5	90.8	80.5	83.3	- 说吧	- 说吧
ELECTRA-400K 数据	7.也可适用第7.1e20(1x)条。	335海里	88.7	94.2	86.9	89.6	- 说吧	- 说吧
ELECTRA-1.75M (英语).	3.1e21 (4.4x) (中文(简体) ).	335海里	页:1	94.9	** ** 88.0** (单位:千美元)	页:1	88.7 (单位:千美元)	页:1

表4:非综艺模式的SQuAD结果。

Results on SQuAD are shown in Table 4. Consistent, with the GLUE results, ELECTRA scores better than masked-language-modeling-based methods given the same compute resources. For example, ELECTRA-400K outperforms RoBERTa-100k and our BERT baseline, which use similar amounts of pre-training compute. ELECTRA-400K also performs comparably to RoBERTa-500K despite using less than 1/4th of the compute. Unsurprisingly, training ELECTRA longer improves results further: ELECTRA-1.75M scores higher than previous models on the SQuAD 2.0 benchmark. ELECTRA-Base also yields strong results, scoring substantially better than BERT-Base and XLNet-Base, and even surpassing BERT-Large according to most metrics. ELECTRA generally performs better at SQuAD 2.0 than 1.1. Perhaps replaced token detection, in which the model distinguishes real tokens from plausible fakes, is particularly transferable to the answerability classification of SQuAD 2.0, in which the model must distinguish answerable questions from fake unanswerable questions.

SQAD的结果见表4。 一致的是,与GLUE结果相比,ELECTRA的得分优于基于蒙面语言模型的方法,获得相同的计算资源. 例如,ELECTRA-400K优于ROBERTA-100k和我们的BERT基线,它们使用相似数量的预训练计算. ELECTRA-400K也具有与RoBERTa-500K的可比较性,尽管使用的计算不到四分之一. 毫不奇怪,训练ELECTRA更长的时间进一步提高了效果:ELECTRA-1.75M比SQuAD2.0基准上之前的模型高分. ELECTRA-Base也产生很强的结果,得分大大优于BERT-Base和XLNet-Base,甚至比BERT-Large还多,根据大多数衡量标准. ELECTRA在SQuAD 2.0中的表现一般优于1.1. 也许取代了符号检测,其中模型区分了真实符号与假象假象,特别可以被转移至SQuAD 2.0的可回答分类,其中模型必须区分可回答的问题与假的不可回答的问题.

3.5 Efficiency Analysis

3.5 效率分析

We have suggested that posing the training objective over a small subset of tokens makes masked language modeling inefficient. However, it isn’t entirely obvious that this is the case. After all, the model still receives a large number of input tokens even though it predicts only a small number of masked tokens. To better understand where the gains from ELECTRA are coming from, we compare a series of other pre-training objectives that are designed to be a set of “stepping stones” between BERT and ELECTRA.

我们建议,把训练目标放在一小撮代币上,使蒙面语言模特效率低下。 但情况并不完全明显。 毕竟,该模型仍然收到大量的输入符,尽管它预测只有少量的蒙面符. 为了更好地了解ELECTRA的收益来自何方,我们比较了一系列其他训练前目标,它们旨在成为BERT和ELECTRA之间的一组“踏步石”。

• • ELECTRA 15%: This model is identical to ELECTRA except the discriminator loss only comes from the 15% of the tokens that were masked out of the input. In other words, the sum in the discriminator loss $\mathcal{L}_{\text{Disc}}$ is over $i\in\bm{m}$ instead of from 1 to $n$.[^7]
• • Replace MLM: This objective is the same as masked language modeling except instead of replacing masked-out tokens with [MASK], they are replaced with tokens from a generator model. This objective tests to what extent ELECTRA’s gains come from solving the discrepancy of exposing the model to [MASK] tokens during pre-training but not fine-tuning.
• • All-Tokens MLM: Like in Replace MLM, masked tokens are replaced with generator samples. Furthermore, the model predicts the identity of all tokens in the input, not just ones that were masked out. We found it improved results to train this model with an explicit copy mechanism that outputs a copy probability $D$ for each token using a sigmoid layer. The model’s output distribution puts $D$ weight on the input token plus $1-D$ times the output of the MLM softmax. This model is essentially a combination of BERT and ELECTRA. Note that without generator replacements, the model would trivially learn to make predictions from the vocabulary for [MASK] tokens and copy the input for other ones.

• EELECTRA 15 QQ: 此型号与ELECTRA完全相同,但歧视者的损失仅来自输入中蒙面的15%的代币. 换言之,歧视者损失$\mathcal{L}_{\text{Disc}}$的金额超过$i\in\bm{m}$,而不是1至$n$。
• ** 更换MLM: 这个目标与蒙面语言模型相同,除了没有用[MASK]取代蒙面外的符号,而是用发电机模型中的符号代替. 这一客观的检验是,ELECTRA从解决在训练前但不进行微调时将该模型暴露在[MASK]标志上的不一致程度得到多少收益。
• ** 全托磁共振**: 和重置MLM一样,蒙面代币被替换为发电机样本. 此外,该模型还预测输入中所有符号的特性,而不仅仅是被掩盖的符号. 我们 73. 我们发现,通过一个明确的复制机制来训练这一模型的结果有所改善,这个机制将使用西格莫德层为每个符号输出一个复制概率$D$。 该模型的输出分布将$D$权重放在输入符上加上$1-D$倍于MLM软max的输出. 这个模型基本上是BERT和ELECTRA的组合. 注意没有生成器替换,该模型将微乎其微地学习从词汇中为[MASK]词元作出预测,并复制其他词元的输入.

Results are shown in Table 5. First, we find that ELECTRA is greatly benefiting from having a loss defined over all input tokens rather than just a subset: ELECTRA 15% performs much worse than ELECTRA. Secondly, we find that BERT performance is being slightly harmed from the pre-train fine-tune mismatch from [MASK] tokens, as Replace MLM slightly outperforms BERT. We note that BERT (including our implementation) already includes a trick to help with the pre-train/fine-tune discrepancy: masked tokens are replaced with a random token 10% of the time and are kept the same 10% of the time. However, our results suggest these simple heuristics are insufficient to fully solve the issue. Lastly, we find that All-Tokens MLM, the generative model that makes predictions over all tokens instead of a subset, closes most of the gap between BERT and ELECTRA. In total, these results suggest a large amount of ELECTRA’s improvement can be attributed to learning from all tokens and a smaller amount can be attributed to alleviating the pre-train fine-tune mismatch.

结果见表5。 首先,我们发现ELECTRA从所有输入符上定义一个损失而不只是一个子集受益很大:ELECTRA 15%的性能比ELECTRA要差得多. 第二,我们发现BERT的性能由于[MASK]符号的火车前微调不匹配而略有受损,因为替换MLM的性能略高于BERT. 我们注意到BERT(包括我们的执行)已经包含一个帮助解决预训练/精益求精差异的诡计:蒙面符被随机代号替换为10%的时间,并保持10%的时间不变. 然而,我们的结果表明,这些简单的修炼主义不足以充分解决这一问题。 最后,我们发现,All-Tokens MLM,即对所有符号而不是子集作出预测的生成模型,缩小了BERT和ELECTRA之间的大部分差距。 总而言之,这些结果表明,ELECTRA的大量改进可以归功于从所有符号中学习,而较少的改进可以归因于缓解训练前的微调不匹配。

Model	ELECTRA	All-Tokens MLM	Replace MLM	ELECTRA 15%	BERT
GLUE score	85.0	84.3	82.4	82.4	82.2

Table 5: Compute-efficiency experiments (see text for details).

** 模式**	选举	全托盘 MLM	替换 MLM	选举 15%	贝特
GLUE 分数	85.0	84.3	82.4	82.4	82.2

表5:计算效率实验(详见文本).

Figure 4: Left and Center: Comparison of BERT and ELECTRA for different model sizes. Right: A small ELECTRA model converges to higher downstream accuracy than BERT, showing the improvement comes from more than just faster training.

图4 左和中:比较不同型号大小的BERT和ELECTRA. 右:一个小型的ELECTRA模型与比BERT更高的下游精度汇合,显示改进来自不仅仅是更快的训练.

The improvement of ELECTRA over All-Tokens MLM suggests that the ELECTRA’s gains come from more than just faster training. We study this further by comparing BERT to ELECTRA for various model sizes (see Figure 4, left). We find that the gains from ELECTRA grow larger as the models get smaller. The small models are trained fully to convergence (see Figure 4, right), showing that ELECTRA achieves higher downstream accuracy than BERT when fully trained. We speculate that ELECTRA is more parameter-efficient than BERT because it does not have to model the full distribution of possible tokens at each position, but we believe more analysis is needed to completely explain ELECTRA’s parameter efficiency.

ELECTRA对全托肯斯MLM的改进表明,ELECTRA的收益不仅仅来自更快的训练. 我们通过比较各种模型大小的BERT和ELECTRA来进一步研究这个问题(见图4,左)。 我们发现从ELECTRA得到的收益随着模型的变小而增大. 小型模型经过充分训练后可以完全趋同(见图4,右),表明ELECTRA在经过充分训练后比BERT在下游的精度更高. 我们推测ELECTRA比BERT更具有参数效率,因为它不需要在每一个位置上模拟可能信物的全部分布,但我们认为需要更多的分析来完全解释ELECTRA的参数效率.

4 Related Work

4 相关工作

Self-Supervised Pre-training for NLP  Self-supervised learning has been used to learn word representations (Collobert et al., 2011; Pennington et al., 2014) and more recently contextual representations of words though objectives such as language modeling (Dai & Le, 2015; Peters et al., 2018; Howard & Ruder, 2018). BERT (Devlin et al., 2019) pre-trains a large Transformer (Vaswani et al., 2017) at the masked-language modeling task. There have been numerous extensions to BERT. For example, MASS (Song et al., 2019) and UniLM (Dong et al., 2019) extend BERT to generation tasks by adding auto-regressive generative training objectives. ERNIE (Sun et al., 2019a) and SpanBERT (Joshi et al., 2019) mask out contiguous sequences of token for improved span representations. This idea may be complementary to ELECTRA; we think it would be interesting to make ELECTRA’s generator auto-regressive and add a “replaced span detection” task. Instead of masking out input tokens, XLNet (Yang et al., 2019) masks attention weights such that the input sequence is auto-regressively generated in a random order. However, this method suffers from the same inefficiencies as BERT because XLNet only generates 15% of the input tokens in this way. Like ELECTRA, XLNet may alleviate BERT’s pretrain-finetune discrepancy by not requiring [MASK] tokens, although this isn’t entirely clear because XLNet uses two “streams” of attention during pre-training but only one for fine-tuning. Recently, models such as TinyBERT (Jiao et al., 2019) and MobileBERT (Sun et al., 2019b) show that BERT can effectively be distilled down to a smaller model. In contrast, we focus more on pre-training speed rather than inference speed, so we train ELECTRA-Small from scratch.

** NLP的自监督预训练** 自监督的学习被用于学习单词表达法(Collobert等人,2011年;Pennington等人,2014年),以及最近的编译法 通过语言建模等目标表达单词(Dai & Le,2015年;Peters等人,2018年;Howard & Ruder,2018年)。 BERT(Devlin等人,2019年)在蒙面语模具任务中预留了一个大型Transformer(Vaswani等人,2017年). 对BERT进行了多次扩展. 例如,MASS(Song等人,2019年)和UniLM(Dong等人,2019年)通过增加自动递归基因训练目标,将BERT扩展至生成任务. ERNIE(Sun等人,2019年a)和SpanBERT(Joshi等人,2019年)为改进跨度表示而掩盖了毗连序列的象征. 这个想法或许是对ELECTRA的补充;我们认为让ELECTRA的发电机具有自动递归性并增加一个"替换的跨段探测"任务将会很有趣. XLNet(Yang et al., 2019)没有掩盖输入符,而是掩盖了注意重量,以至于输入序列是按随机顺序自动递归生成的. 然而,这种方法和BERT一样效率低下,因为XLNet只以这种方式生成15%的输入符. XLNet与ELECTRA一样,通过不要求[MASK]词元,可以缓解BERT的预训练-finetune差异,虽然这一点并不完全清楚,因为XLNet在预训练期间使用两个"流"的注意力,而只有一个用于微调. 最近,TinyBERT(Jiao等人,2019年)和MobileBERT(Sun等人,2019年b)等模型显示,BERT可以有效地向下蒸馏到更小的模型. 相比之下,我们更注重训练前的速度,而不是推理速度,因此我们从零开始训练方面的活动EELCTRA-Small.

Generative Adversarial Networks  GANs (Goodfellow et al., 2014) are effective at generating high-quality synthetic data. Radford et al. (2016) propose using the discriminator of a GAN in downstream tasks, which is similar to our method. GANs have been applied to text data (Yu et al., 2017; Zhang et al., 2017), although state-of-the-art approaches still lag behind standard maximum-likelihood training (Caccia et al., 2018; Tevet et al., 2018). Although we do not use adversarial learning, our generator is particularly reminiscent of MaskGAN (Fedus et al., 2018), which trains the generator to fill in tokens deleted from the input.

** Generative Adversarial Networks** GANs(Goodfellow等人,2014年)在生成高质量合成数据方面是有效的. Radford等人(2016年)提议在下游任务中使用GAN的歧义,这与我们的方法相似. GANs已应用于文本数据(Yu等人,2017年;Zhang等人,2017年),尽管最先进的方法仍然落后于标准的最大相似度训练(Caccia等人,2018年;Tevet等人,2018年). 虽然我们不使用对抗性学习,但我们的生成器特别让人想起了MaskGAN(Fedus等人,2018年),它训练生成器以填充从输入中删除的符号.

Contrastive Learning  Broadly, contrastive learning methods distinguish observed data points from fictitious negative samples. They have been applied to many modalities including text (Smith & Eisner, 2005), images (Chopra et al., 2005), and video (Wang & Gupta, 2015; Sermanet et al., 2017) data. Common approaches learn embedding spaces where related data points are similar (Saunshi et al., 2019) or models that rank real data points over negative samples (Collobert et al., 2011; Bordes et al., 2013). ELECTRA is particularly related to Noise-Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010), which also trains a binary classifier to distinguish real and fake data points.

** 相互学习** 广义而言,对比性学习方法将观测到的数据点与虚构的负样品区分开来. 它们被应用于许多模式,包括文本(Smith & Eisner,2005年)、图像(Chopra等人,2005年)和视频(Wang & Gupta,2015年;Sermanet等人,2017年)数据。 共同方法学习嵌入空间,其中相关数据点相似(Saunshi等人,2019年),或将真实数据点排在负样本之上的模型(Collobert等人,2011年;Bordes等人,2013年)。 ELECTRA尤其与噪声-Contrastive估计(NCE)有关(Gutmann & Hyvärinen,2010年),它也训练一个二进制分类器,以区分真假数据点.

Word2Vec (Mikolov et al., 2013), one of the earliest pre-training methods for NLP, uses contrastive learning. In fact, ELECTRA can be viewed as a massively scaled-up version of Continuous Bag-of-Words (CBOW) with Negative Sampling. CBOW also predicts an input token given surrounding context and negative sampling rephrases the learning task as a binary classification task on whether the input token comes from the data or proposal distribution. However, CBOW uses a bag-of-vectors encoder rather than a transformer and a simple proposal distribution derived from unigram token frequencies instead of a learned generator.

Word2Vec(Mikolov等人,2013年)是NLP最早的预训练方法之一,使用对比学习. 事实上,ELECTRA可以看作是带有负采样功能的连续 Bag-of-Words(CBOW)大规模升级的版本. CBOW还预测了周围上下文给出的输入符,负采样将学习任务重新表述为二进制分类任务,即输入符是来自数据还是提议分布. 然而,CBOW使用一个袋式的气体编码器而不是一个变压器,以及从unigram词元频率而不是学习型生成器中衍生出来的简单的建议分配.

5 Conclusion

5 结论

We have proposed replaced token detection, a new self-supervised task for language representation learning. The key idea is training a text encoder to distinguish input tokens from high-quality negative samples produced by an small generator network. Compared to masked language modeling, our pre-training objective is more compute-efficient and results in better performance on downstream tasks. It works well even when using relatively small amounts of compute, which we hope will make developing and applying pre-trained text encoders more accessible to researchers and practitioners with less access to computing resources. We also hope more future work on NLP pre-training will consider efficiency as well as absolute performance, and follow our effort in reporting compute usage and parameter counts along with evaluation metrics.

我们提议更换符号检测,这是语文代表学习方面一项新的自监督的任务。 关键的想法是训练一个文本编码器,以区分输入符与小型生成器网络产生的高质量负样品. 与蒙面语言模型相比,我们的训练前目标更具有计算效率,并使得下游任务的表现更好. 它即使在使用相对较少量的计算时,也能发挥很好的作用,我们希望这将使开发和应用经过预先训练的文本编码器更容易为获得计算资源较少的研究人员和从业人员所利用. 我们还希望,今后关于NLP预训练的更多工作将考虑效率和绝对业绩,并跟踪我们努力报告计算使用量和参数计算以及评价指标。

Acknowledgements

致谢

We thank Allen Nie, Prajit Ramachandran, audiences at the CIFAR LMB meeting and U. de Montréal, and the anonymous reviewers for their thoughtful comments and suggestions. We thank Matt Peters for answering our questions about ELMo, Alec Radford for answers about GPT, Naman Goyal and Myle Ott for answers about RoBERTa, Zihang Dai for answers about XLNet, Zhenzhong Lan for answers about ALBERT, and Danqi Chen and Mandar Joshi for answers about SpanBERT. Kevin is supported by a Google PhD Fellowship.

我们感谢Allen Nie、Prajit Ramachandran、CIFAR LMB会议和U. de Montréal的听众以及匿名审查员提出了深思熟虑的意见和建议。 感谢马特·彼得斯(Matt Peters)回答关于ELMo的问题,阿莱克·拉德福德回答关于GPT的问题,纳曼·戈亚尔(Naman Goyal)和弥勒·奥特(Myle Ott)回答关于RoBERTa的问题,齐杭戴(Zihang Dai)回答关于XLNet的问题,禅宗兰回答关于ALBERT的问题,丹奇·陈(Danqi Chen)和曼达尔·乔希(Mandar Joshi)回答关于斯潘贝特的问题. 凯文得到谷歌博士所属机构的支持。

References

参考文献

Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NeurIPS , 2013.

Bose et al. (2018) Avishek Joey Bose, Huan Ling, and Yanshuai Cao. Adversarial contrastive estimation. In ACL , 2018.

Caccia et al. (2018) Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language GANs falling short. arXiv preprint arXiv:1811.02549 , 2018.

Callan et al. (2009) Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. Clueweb09 data set, 2009. URL https://lemurproject.org/clueweb09.php/ .

Cer et al. (2017) Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval@ACL , 2017.

Chopra et al. (2005) Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. CVPR , 2005.

Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. BAM! Born-again multi-task networks for natural language understanding. In ACL , 2019.

Collobert et al. (2011) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. Natural language processing (almost) from scratch. JMLR , 2011.

Dai & Le (2015) Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In NeurIPS , 2015.

Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , 2019.

Dolan & Brockett (2005) William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In IWP@IJCNLP , 2005.

Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NeurIPS , 2019.

Fedus et al. (2018) William Fedus, Ian J. Goodfellow, and Andrew M. Dai. MaskGAN: Better text generation via filling in the _______. In ICLR , 2018.

Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B. Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL@ACL , 2007.

Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.

Gutmann & Hyvärinen (2010) Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS , 2010.

Howard & Ruder (2018) Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In ACL , 2018.

Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. First Quora dataset release: Question pairs, 2017. URL https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs .

Jiao et al. (2019) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351 , 2019.

Joshi et al. (2019) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529 , 2019.

Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019.

Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR Workshop Papers , 2013.

Parker et al. (2011) Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.

Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP , 2014.

Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT , 2018.

Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088 , 2018.

Radford et al. (2016) Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR , 2016.

Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. https://blog.openai.com/language-unsupervised , 2018.

Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy S. Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP , 2016.

Saunshi et al. (2019) Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML , 2019.

Sermanet et al. (2017) Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. ICRA , 2017.

Smith & Eisner (2005) Noah A. Smith and Jason Eisner. Contrastive estimation: Training log-linear models on unlabeled data. In ACL , 2005.

Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP , 2013.

Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. In ICML , 2019.

Sun et al. (2019a) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223 , 2019a.

Sun et al. (2019b) Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https://openreview.net/forum?id=SJxjVaNKwB .

Tevet et al. (2018) Guy Tevet, Gavriel Habib, Vered Shwartz, and Jonathan Berant. Evaluating text gans as language models. In NAACL-HLT , 2018.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Vincent et al. (2008) Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML , 2008.

Wang et al. (2019) Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR , 2019.

Wang & Gupta (2015) Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. ICCV , 2015.

Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471 , 2018.

Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT , 2018.

Williams (1992) Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8(3-–4):229–256, 1992.

Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS , 2019.

Yu et al. (2017) Lantao Yu, Weinan Zhang, Jun Wang, and Yingrui Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In AAAI , 2017.

Zhang et al. (2017) Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. Adversarial feature matching for text generation. In ICML , 2017.

Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV , 2015.

Appendix A Pre-Training Details

附录 A 预训练细节

The following details apply to both our ELECTRA models and BERT baselines. We mostly use the same hyperparameters as BERT. We set $\lambda$, the weight for the discriminator objective in the loss to 50.[^8] We use dynamic token masking with the masked positions decided on-the-fly instead of during preprocessing. Also, we did not use the next sentence prediction objective proposed in the original BERT paper, as recent work has suggested it does not improve scores (Yang et al., 2019; Liu et al., 2019). For our ELECTRA-Large model, we used a higher mask percent (25 instead of 15) because we noticed the generator was achieving high accuracy with 15% masking, resulting in very few replaced tokens. We searched for the best learning rate for the Base and Small models out of [1e-4, 2e-4, 3e-4, 5e-4] and selected $\lambda$ out of [1, 10, 20, 50, 100] in early experiments. Otherwise we did no hyperparameter tuning beyond the experiments in Section 3.2. The full set of hyperparameters are listed in Table 6.

以下细节适用于我们的ELECTRA模型和BERT基线。 我们主要使用与BERT相同的超参数. 我们把$\lambda$, 歧视目标在损失中的权重定为50。 我们使用动态的标志性面具 与蒙面的姿势 决定飞行而不是在预处理。 另外,我们没有使用BERT论文原稿中提出的下一个句子预测目标,因为最近的工作表明它并没有提高分数(Yang等人,2019年;Liu等人,2019年). 对于我们的ELECTRA-Large模型,我们使用了更高的遮罩%(25而不是15),因为我们注意到发电机正以15%的遮罩实现高精度,导致很少的代号被替换. 我们从[1e-4,2e-4,3e-4,5e-4]中寻找基地和小型模型的最佳学习率,并在早期实验中从[1,10,20,50,100]中挑选出$\lambda$. 否则,我们除了3.2节的实验之外,没有做超参数调试。 表6列出了整套超参数。

Hyperparameter	Small	Base	Large
Number of layers	12	12	24
Hidden Size	256	768	1024
FFN inner hidden size	1024	3072	4096
Attention heads	4	12	16
Attention head size	64	64	64
Embedding Size	128	768	1024
Generator Size (multiplier for hidden-size,	1/4	1/3	1/4
FFN-size, and num-attention-heads)
Mask percent	15	15	25
Learning Rate Decay	Linear	Linear	Linear
Warmup steps	10000	10000	10000
Learning Rate	5e-4	2e-4	2e-4
Adam $\epsilon$	1e-6	1e-6	1e-6
Adam $\beta_{1}$	0.9	0.9	0.9
Adam $\beta_{2}$	0.999	0.999	0.999
Attention Dropout	0.1	0.1	0.1
Dropout	0.1	0.1	0.1
Weight Decay	0.01	0.01	0.01
Batch Size	128	256	2048
Train Steps (BERT/ELECTRA)	1.45M/1M	1M/766K	464K/400K

Table 6: Pre-train hyperparameters. We also train an ELECTRA-Large model for 1.75M steps (other hyperparameters are identical).

** 超参数**	** 小型**	页:1	页:1
层数	12	12	24
隐藏大小	256	768	1024
FFN 内隐藏大小	1024	3072	4096
注意指挥	4	12	16
注意头部大小	64	64	64
嵌入大小	128	768	1024
发电机大小(隐藏大小的乘数,	1/4	1/3	1/4
FFN 大小,和纳姆-注意头)
遮罩%	15	15	25
学习率下降	线性	线性	线性
温和步骤	10000	10000	10000
学习率	5个e-4	2e-4 (单位:千美元)	2e-4 (单位:千美元)
亚当·$\epsilon$	(单位:千美元)	(单位:千美元)	(单位:千美元)
亚当·$\beta_{1}$	0.9	0.9	0.9
亚当·$\beta_{2}$	0.999	0.999	0.999
注意退学	0.1	0.1	0.1
辍学	0.1	0.1	0.1
重量衰减	0.01	0.01	0.01
伎àn	128	256	2048
列车步骤(BERT/ELECTRA)	1.45M/1将人权纳入	1率先发射	464K/400K (韩语)

表6:火车前超参数。 我们还为1.75M级(其他超参数完全相同)训练一种ELECTRA-Large型号.

Appendix B Fine-Tuning Details

附录 B 微调细节

For Large-sized models, we used the hyperparameters from Clark et al. (2019) for the most part. However, after noticing that RoBERTa (Liu et al., 2019) uses more training epochs (up to 10 rather than 3) we searched for the best number of train epochs out of [10, 3] for each task. For SQuAD, we decreased the number of train epochs to 2 to be consistent with BERT and RoBERTa. For Base-sized models we searched for a learning rate out of [3e-5, 5e-5, 1e-4, 1.5e-4] and the layer-wise learning-rate decay out of [0.9, 0.8, 0.7], but otherwise used the same hyperparameters as for Large models. We found the small models benefit from a larger learning rate and searched for the best one out of [1e-4, 2e-4, 3e-4, 5e-3]. With the exception of number of train epochs, we used the same hyperparameters for all tasks. In contrast, previous research on GLUE such as BERT, XLNet, and RoBERTa separately searched for the best hyperparameters for each task. We expect our results would improve slightly if we performed the same sort of additional hyperparameter search. The full set of hyperparameters is listed in Table 7.

对于大尺寸的模型,我们大部分使用来自克拉克等人(2019年)的超参数. 然而,在注意到RoBERTa(Liu等人,2019年)使用更多训练时代(最多10个而不是3个)后,我们为每项任务搜索了最佳数量的列车时代[10,3]. 对于SQuAD,我们把列车的编组时间减少到2个,以与BERT和RoBERTA保持一致. 对于基质大小的模型,我们搜索了[3e-5,5e-5,1e-4,1.5e-4]的学习速率,以及[0.9,0.8,0.7]的分层学习速率衰减,但其他方法使用与大模型相同的超参数. 我们发现小模型得益于更大的学习率,并寻找[1e-4,2e-4,3e-4,5e-3]中最好的模型. 除了列车时代的数量之外,我们对所有任务都使用了相同的超参数. 相比之下,之前对GLUE的研究,如BERT,XLNet,和RoBERTa分别搜索每个任务的最佳超参数. 我们希望,如果我们进行同样的额外超参数搜索,结果会略有改善。 表7列出了整套超参数。

Hyperparameter	GLUE Value
Learning Rate	3e-4 for Small, 1e-4 for Base, 5e-5 for Large
Adam $\epsilon$	1e-6
Adam $\beta_{1}$	0.9
Adam $\beta_{2}$	0.999
Layerwise LR decay	0.8 for Base/Small, 0.9 for Large
Learning rate decay	Linear
Warmup fraction	0.1
Attention Dropout	0.1
Dropout	0.1
Weight Decay	0
Batch Size	32
Train Epochs	10 for RTE and STS, 2 for SQuAD, 3 for other tasks

Table 7: Fine-tune hyperparameters

** 超参数**	GLUE 值
学习率	小3e-4,基地1e-4,大5e-5
亚当·$\epsilon$	1个e-6
亚当·$\beta_{1}$	0.9
亚当·$\beta_{2}$	0.999
顺层 LR 衰变	基/小0.8,大0.9
学习率衰减	线性
温暖的分数	0.1
开放供退学	0.1
辍学	0.1
重量衰减	0
批量大小	32
火车 Epochs	RTE和STS各10个,SQuAD各2个,其他任务3个

表7:微调超参数

Following BERT, we do not show results on the WNLI GLUE task for the dev set results, as it is difficult to beat even the majority classifier using a standard fine-tuning-as-classifier approach. For the GLUE test set results, we apply the standard tricks used by many of the GLUE leaderboard submissions including RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019), and ALBERT (Lan et al., 2019). Specifically:

在BERT之后,我们并不显示WNLI GLUE任务对dev集结果的结果,因为即使大多数分类者也很难使用标准的微调-分级方法来击败. 对于GLUE测试设置结果,我们应用了GLUE领导板上许多提交文件所使用的标准技巧,包括RoBERTa(Liu等人,2019年),XLNet(Yang等人,2019年)和ALBERT(Lan等人,2019年). 具体而言:

• • For RTE and STS we use intermediate task training (Phang et al., 2018), starting from an ELECTRA checkpoint that has been fine-tuned on MNLI. For RTE, we found it helpful to combine this with a lower learning rate of 2e-5.
• • For WNLI, we follow the trick described in Liu et al. (2019) where we extract candidate antecedents for the pronoun using rules and train a model to score the correct antecedent highly. However, different from Liu et al. (2019), the scoring function is not based on MLM probabilities. Instead, we fine-tune ELECTRA’s discriminator so it assigns high scores to the tokens of the correct antecedent when the correct antecedent replaces the pronoun. For example, if the Winograd schema is “the trophy could not fit in the suitcase because it was too big,” we train the discriminator so it gives a high score to “trophy” in “the trophy could not fit in the suitcase because the trophy was too big” but a low score to “suitcase” in “the trophy could not fit in the suitcase because the suitcase was too big.”
• • For each task we ensemble the best 10 of 30 models fine-tuned with different random seeds but initialized from the same pre-trained checkpoint.

• 对RTE和STS来说,我们使用中间任务训练(Phang等人,2018年),起点是经过MNLI调整的ELECTRA检查站。 对于RTE来说,我们发现将这一点与较低的2e-5学习率结合起来是有益的.
• 对WNLI来说,我们遵循刘等人(2019年)描述的诡计,利用规则为代词提取候选人的先验,并训练一个模型来高分正确的先验. 然而,与刘等人(2019年)不同,评分功能并非基于MLM概率. 相反,我们微调了 ELECTRA 的 歧视者, 从而在正确的前代词取代代词时, 给正确的前代词分配高分。 例如,如果Winograd计划是“奖杯因太大而不能装进手提箱中”,那么我们就训练歧视者,使其在“奖杯因太大而不能装进手提箱中”得分高,但“奖杯因手提箱太大而不能装进手提箱中”得分低。
• 为每项任务,我们用不同的随机种子对30种型号中最好的10种进行精细调整,但从同一预先训练的检查站开始。

While these tricks do improve scores, they make having clear scientific comparisons more difficult because they require extra work to implement, require lots of compute, and make results less apples-to-apples because different papers implement the tricks differently. We therefore also report results for ELECTRA-1.75M with the only trick being dev-set model selection (best of 10 models), which is the setting BERT used to report results, in Table 8.

虽然这些技巧确实提高了分数,但是由于需要额外工作来执行,需要大量计算,并且由于不同的论文不同地执行这些技巧,使得结果较少苹果对应用,所以使得有清晰的科学比较变得更加困难. 因此,我们还报告ELECTRA-1.75M的结果,唯一的诀窍是dev-set模型选择(最佳10个模型),这是BERT用来报告结果的设置,见表8。

For our SQuAD 2.0 test set submission, we fine-tuned 20 models from the same pre-trained checkpoint and submitted the one with the best dev set score.

对于我们SQAD 2.0测试集的提交,我们从同一预先训练的检查点对20个模型进行了微调,并提交了最优秀的dev设定分数.

Appendix C Details about GLUE

附录 C 关于 GLUE 的细节

We provide further details about the GLUE benchmark tasks below

我们详细介绍GLUE基准任务如下:

• • CoLA: Corpus of Linguistic Acceptability (Warstadt et al., 2018). The task is to determine whether a given sentence is grammatical or not. The dataset contains 8.5k train examples from books and journal articles on linguistic theory.
• • SST: Stanford Sentiment Treebank (Socher et al., 2013). The tasks is to determine if the sentence is positive or negative in sentiment. The dataset contains 67k train examples from movie reviews.
• • MRPC: Microsoft Research Paraphrase Corpus (Dolan & Brockett, 2005). The task is to predict whether two sentences are semantically equivalent or not. The dataset contains 3.7k train examples from online news sources.
• • STS: Semantic Textual Similarity (Cer et al., 2017). The tasks is to predict how semantically similar two sentences are on a 1-5 scale. The dataset contains 5.8k train examples drawn from new headlines, video and image captions, and natural language inference data.
• • QQP: Quora Question Pairs (Iyer et al., 2017). The task is to determine whether a pair of questions are semantically equivalent. The dataset contains 364k train examples from the community question-answering website Quora.
• • MNLI: Multi-genre Natural Language Inference (Williams et al., 2018). Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither. The dataset contains 393k train examples drawn from ten different sources.
• • QNLI: Question Natural Language Inference; constructed from SQuAD (Rajpurkar et al., 2016). The task is to predict whether a context sentence contains the answer to a question sentence. The dataset contains 108k train examples from Wikipedia.
• • RTE: Recognizing Textual Entailment (Giampiccolo et al., 2007). Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis or not. The dataset contains 2.5k train examples from a series of annual textual entailment challenges.

• CoLA:中的一个语言可接受性公司(Warstadt等人,2018年)。 任务是确定某个句子是否是语法上的. 该数据集包含8.5k列车实例,来自关于语言理论的书籍和期刊文章.
• ** SST:斯坦福森蒂门特树岸(Socher等人,2013年)。 任务是确定句子是正面还是负面情绪. 数据集包含电影评论中的67k列车实例.
• MRPC: 微软研究参数Corpus(Dolan & Brockett,2005年)。 任务是预测两个句子在语义上是否等同. 该数据集包含3.7k列车实例来自在线新闻来源.
• STS:语义文字相似性(Cer等人,2017年)。 任务在于预测在1-5尺度上,两句的相似性如何. 数据集包含5.8k列车实例,从新的头条标题,视频和图像标题,以及自然语言推理数据中抽取.

• QQP:**Quora Question Pairs(Iyer等人,2017年)。 任务是确定一对问题是否在组成上等同. 数据集包含来自社区问答网站Quora的364k列车实例.
• ** MNLI**: 多流派自然语言推理(Williams等人,2018年)。 鉴于一个前提句和一个假说句,任务就是预测该前提是否包含假说,是否与假说相矛盾,或者两者都不矛盾. 该数据集包含从十个不同来源抽取的393k列车实例.
  • QQNLI:** 问题自然语言推理;由SQuAD建造(Rajpurkar等人,2016年)。 任务是预测一个上下文句子是否包含一个问题句的答案. 该数据集包含来自维基百科的108k列车实例.
  • RTE: 确认文字细节(Giampiccolo等人,2007年)。 鉴于前提句和假说句,任务是预测前提是否包含假说. 数据集包含从一系列年度文本引入挑战中获得的2.5k列车实例.

Appendix D Further results on GLUE

附录 D GLUE 上的更多结果

We report results for ELECTRA-Base and ELECTRA-Small on the GLUE test set in Table 8. Furthermore, we push the limits of base-sized and small-sized models by training them on the XLNet data instead of wikibooks and for much longer (4e6 train steps); these models are called ELECTRA-Base++ and ELECTRA-Small++ in the table. For ELECTRA-Small++ we also increased the sequence length to 512; otherwise the hyperparameters are the same as the ones listed in Table 6. Lastly, the table contains results for ELECTRA-1.75M without the tricks described in Appendix B. Consistent with dev-set results in the paper, ELECTRA-Base outperforms BERT-Large while ELECTRA-Small outperforms GPT in terms of average score. Unsurprisingly, the ++ models perform even better. The small model scores are even close to TinyBERT (Jiao et al., 2019) and MobileBERT (Sun et al., 2019b). These models learn from BERT-Base using sophisticated distillation procedures. Our ELECTRA models, on the other hand, are trained from scratch. Given the success of distilling BERT, we believe it would be possible to build even stronger small pre-trained models by distilling ELECTRA. ELECTRA appears to be particularly effective at CoLA. In CoLA the goal is to distinguish linguistically acceptable sentences from ungrammatical ones, which fairly closely matches ELECTRA’s pre-training task of identifying fake tokens, perhaps explaining ELECTRA’s strength at the task.

我们报告ELECTRA-Base和ELECTRA-Small在表8中设定的GLUE测试结果. 此外,我们通过在XLNet数据上而不是维基书上以及更长的时间(4e6列火车步骤)来推动基大小模型的极限;这些模型在表中被称为ELECTRA-Base++和ELECTRA-Small++. 对于ELECTRA-Small++,我们也将序列长度提高到512;否则超参数与表6所列的参数相同. 最后,该表载有ELECTRA-1.75M的结果,没有附录B中描述的诡计。 与文件中的dev-set结果一致,ELECTRA-Base的成绩优于BERT-Large,而ELECTRA-Small则优于GPT的平均得分。 毫不奇怪,++模型的表现甚至更好. 小模型分数甚至接近TinyBERT(Jiao等人,2019年)和MobileBERT(Sun等人,2019年b). 这些模型利用精密的蒸馏程序从BERT-Base学习. 另一方面,我们的ELECTRA模型是从零开始训练的. 鉴于蒸馏BERT的成功,我们认为可以通过蒸馏ELECTRA来建立更强大的小型的前训练模型. ELECTRA似乎在CoLA特别有效. 校对:Soup

Model	Train FLOPs	Params	CoLA	SST	MRPC	STS	QQP	MNLI	QNLI	RTE	Avg.
TinyBERT	6.4e19+ (45x+)	14.5M	51.1	93.1	82.6	83.7	89.1	84.6	90.4	70.0	80.6
MobileBERT	6.4e19+ (45x+)	25.3M	51.1	92.6	84.5	84.8	88.3	84.3	91.6	70.4	81.0
GPT	4.0e19 (29x)	117M	45.4	91.3	75.7	80.0	88.5	82.1	88.1	56.0	75.9
BERT-Base	6.4e19 (45x)	110M	52.1	93.5	84.8	85.8	89.2	84.6	90.5	66.4	80.9
BERT-Large	1.9e20 (135x)	335M	60.5	94.9	85.4	86.5	89.3	86.7	92.7	70.1	83.3
SpanBERT	7.1e20 (507x)	335M	64.3	94.8	87.9	89.9	89.5	87.7	94.3	79.0	85.9
ELECTRA-Small	1.4e18 (1x)	14M	54.6	89.1	83.7	80.3	88.0	79.7	87.7	60.8	78.0
ELECTRA-Small++	3.3e19 (18x)	14M	55.6	91.1	84.9	84.6	88.0	81.6	88.3	63.6	79.7
ELECTRA-Base	6.4e19 (45x)	110M	59.7	93.4	86.7	87.7	89.1	85.8	92.7	73.1	83.5
ELECTRA-Base++	3.3e20 (182x)	110M	64.6	96.0	88.1	90.2	89.5	88.5	93.1	75.2	85.7
ELECTRA-1.75M	3.1e21 (2200x)	330M	68.1	96.7	89.2	91.7	90.4	90.7	95.5	86.1	88.6

Table 8: Results for models on the GLUE test set. Only models with single-task finetuning (no ensembling, task-specific tricks, etc.) are shown.

模型	训练 FLOPs	参数量	CoLA	SST	MRPC	STS	QQP	MNLI	QNLI	RTE	平均
TinyBERT	6.4e19+ (45x+)	14.5M	51.1	93.1	82.6	83.7	89.1	84.6	90.4	70.0	80.6
MobileBERT	6.4e19+ (45x+)	25.3M	51.1	92.6	84.5	84.8	88.3	84.3	91.6	70.4	81.0
GPT	4.0e19 (29x)	117M	45.4	91.3	75.7	80.0	88.5	82.1	88.1	56.0	75.9
BERT-Base	6.4e19 (45x)	110M	52.1	93.5	84.8	85.8	89.2	84.6	90.5	66.4	80.9
BERT-Large	1.9e20 (135x)	335M	60.5	94.9	85.4	86.5	89.3	86.7	92.7	70.1	83.3
SpanBERT	7.1e20 (507x)	335M	64.3	94.8	87.9	89.9	89.5	87.7	94.3	79.0	85.9
ELECTRA-Small	1.4e18 (1x)	14M	54.6	89.1	83.7	80.3	88.0	79.7	87.7	60.8	78.0
ELECTRA-Small++	3.3e19 (18x)	14M	55.6	91.1	84.9	84.6	88.0	81.6	88.3	63.6	79.7
ELECTRA-Base	6.4e19 (45x)	110M	59.7	93.4	86.7	87.7	89.1	85.8	92.7	73.1	83.5
ELECTRA-Base++	3.3e20 (182x)	110M	64.6	96.0	88.1	90.2	89.5	88.5	93.1	75.2	85.7
ELECTRA-1.75M	3.1e21 (2200x)	330M	68.1	96.7	89.2	91.7	90.4	90.7	95.5	86.1	88.6

Table 8: Results for models on the GLUE test set. Only models with single-task finetuning (no ensembling, task-specific tricks, etc.) are shown.

Appendix E Counting FLOPs

附录 E 统计 FLOPs

We chose to measure compute usage in terms of floating point operations (FLOPs) because it is a measure agnostic to the particular hardware, low-level optimizations, etc. However, it is worth noting that in some cases abstracting away hardware details is a drawback because hardware-centered optimizations can be key parts of a model’s design, such as the speedup ALBERT (Lan et al., 2019) gets by tying weights and thus reducing communication overhead between TPU workers. We used TensorFlow’s FLOP-counting capabilities[^9] and checked the results with by-hand computation. We made the following assumptions:

我们选择从浮点操作(FLOPs)的角度来测量计算用量,因为它是对特定硬件,低水平优化等的不可知量. 然而,值得注意的是,在某些情况下,抽象去的硬件细节是一个缺点,因为以硬件为中心的优化可以是模型设计的关键部分,例如加速ALBERT(Lan等人,2019年)通过捆绑权重获得,从而减少TPU工人之间的通信间接费用. 我们使用 TensorFlow 的 FLOP 计算能力[^9] , 并用手工计算来检查结果 。 我们提出了以下假设:

• • An “operation” is a mathematical operation, not a machine instruction. For example, an exp is one op like an add, even though in practice the exp might be slower. We believe this assumption does not substantially change compute estimates because matrix-multiplies dominate the compute for most models. Similarly, we count matrix-multiplies as $2*m*n$ FLOPs instead of $m*n$ as one might if considering fused multiply-add operations.
• • The backwards pass takes the same number of FLOPs as the forward pass. This assumption is not exactly right (e.g., for softmax cross entropy loss the backward pass is faster), but importantly, the forward/backward pass FLOPs really are the same for matrix-multiplies, which is most of the compute anyway.
• • We assume “dense” embedding lookups (i.e., multiplication by a one-hot vector). In practice, sparse embedding lookups are much slower than constant time; on some hardware accelerators dense operations are actually faster than sparse lookups.

• “操作”是一种数学操作,而不是一种机器指示。 例如,exp是类似添加的操作,尽管实际上exp可能较慢. 我们认为,这一假设不会大大改变计算估计数,因为大多数模型的计算以矩阵乘法为主。 同样,我们计算矩阵乘法为$2*m*n$ FLOPs,而不是考虑引信乘法操作时的$m*n$。
• 后行通行证与前行通行证相同。 这个假设并不完全正确(例如,对于软马克斯交叉式的反向演算损失,后向传记更快),但重要的是,前向/后向传记的FLOP对矩阵-乘法来说确实是一样的,不管怎样,后者是大多数的计算.
• 我们假定“密集”嵌入式检查(即用一个热矢量乘法)。 在实践中,稀疏的嵌入式外观比恒定时间慢得多;在一些硬件加速器密集操作上,实际上比稀疏的外观更快.

Appendix F Adversarial Training

附录 F 对抗训练

Here we detail attempts to adversarially train the generator instead of using maximum likelihood. In particular we train the generator $G$ to maximize the discriminator loss $\mathcal{L}_{\text{Disc}}$. As our discriminator isn’t precisely the same as the discriminator of a GAN (see the discussion in Section 2), this method is really an instance of Adversarial Contrastive Estimation (Bose et al., 2018) rather than Generative Adversarial Training. It is not possible to adversarially train the generator by back-propagating through the discriminator (e.g., as in a GAN trained on images) due to the discrete sampling from the generator, so we use reinforcement learning instead.

我们在这里详细介绍了试图对发电机进行对抗性训练,而不是利用最大可能性。 特别是我们训练发电机$G$,以最大限度地减少歧视者的损失$\mathcal{L}_{\text{Disc}}$。 因为歧视者与歧视GAN(参见第2节的讨论)并不完全相同, 由于发电机的离散取样,不可能通过歧视者反向传播(如在图像训练的GAN中)对发电机进行对抗性训练,因此我们采用强化学习。

Our generator is different from most text generation models in that it is non-autogregressive: predictions are made independently. In other words, rather than taking a sequence of actions where each action generates a token, the generator takes a single giant action of generating all tokens simultaneously, where the probability for the action factorizes as the product of generator probabilities for each token. To deal with this enormous action space, we make the following simplifying assumption: that the discriminator’s prediction $D(\bm{x}^{\text{corrupt}},t)$ depends only on the token $x_{t}$ and the non-replaced tokens $\{x_{i}:i\not\in\bm{m}\}$, i.e., it does not depend on other generated tokens $\{\hat{x}_{i}:i\in\bm{m}\land i\neq t\}$. This isn’t too bad of an assumption because a relatively small number of tokens are replaced, and it greatly simplifies credit assignment when using reinforcement learning. Notationally, we show this assumption by (in a slight abuse of notation) by writing $D(\hat{x}_{t}|\bm{x}^{\text{masked}})$ for the discriminator predicting whether the generated token $\hat{x}_{t}$ equals the original token $x_{t}$ given the masked context $\bm{x}^{\text{masked}}$. A useful consequence of this assumption is that the discriminator score for non-replaced tokens ($D(x_{t}|\bm{x}^{\text{masked}})$ for $t\not\in\bm{m}$) is independent of $p_{G}$ because we are assuming it does not depend on any replaced token. Therefore these tokens can be ignored when training $G$ to maximize $\mathcal{L}_{\text{Disc}}$. During training we seek to find

我们的生成器与大多数文本生成模型不同,因为它是非自动的:预测是独立的. 换句话说,当每个动作生成一个符号时,生成者不采取一系列动作,而是采取一个同时生成所有符号的单个巨型动作,即动作的概率作为每个符号的生成概率的产物而成因. 为了处理这个巨大的行动空间,我们做了以下简化的假设:歧视者的预测$D(\bm{x}^{\text{corrupt}},t)$只取决于象征的$x_{t}$和没有替换的象征的$\{x_{i}:i\not\in\bm{m}\}$,即它不依赖于其他产生的代谢的$\{\hat{x}_{i}:i\in\bm{m}\land i\neq t\}$. 这在假设中并不差, 因为相对较少的代币被替换, 标注一下,我们通过(在略微滥用标记的情况下)为歧视者写出$D(\hat{x}_{t}|\bm{x}^{\text{masked}})$来显示这一假设,预测由于被遮蔽的上下文$\bm{x}^{\text{masked}}$,生成的$\hat{x}_{t}$是否等于原始的$x_{t}$. 这一假设的一个有益后果是,非替换的符号($D(x_{t}|\bm{x}^{\text{masked}})$ for $t\not\in\bm{m}$)的歧视性分数独立于$p_{G}$,因为我们假设它不依赖于任何替换的符号. 因此,在训练$G$以最大限度地实现$\mathcal{L}_{\text{Disc}}$时,可以忽略这些标志. 在训练期间,我们寻求找到

$$\displaystyle\argmax_{\theta_{G}}\mathcal{L}_{\text{Disc}}=\argmax_{\theta_{G}}\mathop{\mathbb{E}}_{\bm{x},\bm{m},\hat{\bm{x}}}\Bigg(\sum_{t=1}^{n}- \\ \displaystyle\mathbbm{1}(x^{\text{corrupt}}_{t}=x_{t})\log D(\bm{x}^{\text{corrupt}},t)- \\ \displaystyle\mathbbm{1}(x^{\text{corrupt}}_{t}\neq x_{t})\log(1-D(\bm{x}^{\text{corrupt}},t))\Bigg)$$

Using the simplifying assumption, we approximate the above by finding the argmax of

使用简化的假设,我们通过找到

$$\displaystyle\mathop{\mathbb{E}}{\bm{x},\bm{m},\hat{\bm{x}}}\Bigg(\sum{t\in\bm{m}}-\mathbbm{1}(\hat{x}{t}=x{t})\log D(\hat{x}|\bm{x}^{\text{masked}})-\mathbbm{1}(\hat{x}{t}\neq x{t})\log(1-D(\hat{x}|\bm{x}^{\text{masked}}))\Bigg) \ \displaystyle= \ \displaystyle\mathop{\mathbb{E}}{\bm{x},\bm{m}}\sum{t\in\bm{m}}\mathop{\mathbb{E}}{\hat{x}{t}\sim p_{G}}R(\hat{x}{t},\bm{x}) \ \displaystyle\text{where }R(\hat{x}{t},\bm{x})=\begin{cases}-\log D(\hat{x}{t}|\bm{x}^{\text{masked}})&\text{if}\ \hat{x}{t}=x_{t}\
-\log(1-D(\hat{x}_{t}|\bm{x}^{\text{masked}}))&\text{otherwise}\end{cases}$$

In short, the simplifying assumption allows us to decompose the loss over the individual generated tokens. We cannot directly find $\argmax_{\theta_{G}}$ using gradient ascent because it is impossible to back-propagate through discrete sampling of $\hat{x}$. Instead, we use policy gradient reinforcement learning (Williams, 1992). In particular, we use the REINFORCE gradient

简言之,简化的假设使我们能够将损失分解在单个生成的代币上. 我们无法使用梯度上升法直接找到$\argmax_{\theta_{G}}$,因为不可能通过对$\hat{x}$的离散采样来回推进。 相反,我们使用政策梯度强化学习(Williams,1992年)。 特别是,我们使用REINFORCE梯度

$$\displaystyle\nabla_{\theta_{G}}\mathcal{L}_{\text{Disc}}\approx\mathop{\mathbb{E}}_{\bm{x},\bm{m}}\sum_{t\in\bm{m}}\mathop{\mathbb{E}}_{\hat{x}_{t}\sim p_{G}}\nabla_{\theta_{g}}\log p_{G}(\hat{x}_{t}|\bm{x}^{\text{masked}})[R(\hat{x}_{t},\bm{x})-b(\bm{x}^{\text{masked}},t)]$$

Where $b$ is a learned baseline implemented as $b(\bm{x}^{\text{masked}},t)=-\log\text{sigmoid}(w^{T}h_{G}(\bm{x}^{\text{masked}})_{t})$ where $h_{G}(\bm{x}^{\text{masked}})$ are the outputs of the generator’s Transformer encoder. The baseline is trained with cross-entropy loss to match the reward for the corresponding position. We approximate the expectations with a single sample and learn $\theta_{G}$ with gradient ascent. Despite receiving no explicit feedback about which generated tokens are correct, we found the adversarial training resulted in a fairly accurate generator (for a 256-hidden-size generator, the adversarially trained one achieves 58% accuracy at masked language modeling while the same sized MLE generator gets 65%). However, using this generator did not improve over the MLE-trained one on downstream tasks (see the right of Figure 3 in the main paper).

如果$b$是作为$b(\bm{x}^{\text{masked}},t)=-\log\text{sigmoid}(w^{T}h_{G}(\bm{x}^{\text{masked}})_{t})$执行的学习基准,则$h_{G}(\bm{x}^{\text{masked}})$是发电机Transformer编码器的输出. 基线训练为交叉内损,与相应位置的奖励相匹配. 我们用单一的样本来估算预期值,用梯度上升来学习$\theta_{G}$。 尽管没有收到明确的反馈,但是我们发现对抗训练的结果是一个相当准确的发电机(对于256个隐藏大小的发电机,经过对抗训练的发电机在蒙面语言模型中达到58%的准确度,而同样的尺寸的MLE发电机得到65%的准确度)。 然而,使用这一生成器并没有改进下游任务中经过劳动与就业教育训练的生成器(见主要论文图3的权利)。

Appendix G Evaluating ELECTRA as a Masked Language Model

附录 G 把 ELECTRA 当作掩码语言模型评估

This sections details some initial experiments in evaluating ELECTRA as a masked language model. Using slightly different notation from the main paper, given a context $c$ consisting of a text sequence with one token $x$ masked-out, the discriminator loss can be written as

本节详述了评价ELECTRA作为一种蒙面语言模型的一些初步实验. 使用与主纸略微不同的注解,鉴于上下文$c$包含一个带有一个符号$x$的文字序列,歧视者的损失可以写成:

$$\displaystyle\mathcal{L}_{\text{Disc}}=-\sum\limits_{x\in\text{vocab}}\Big( \\ \displaystyle(1-p_{\text{mask}})p_{\text{data}}(x|c)\log D(x,c)+\text{\hskip 54.06023pt//unmasked token} \\ \displaystyle p_{\text{mask}}p_{\text{data}}(x|c)p_{G}(x|c)\log D(x,c)+\text{\hskip 48.36967pt//generator samples correct token} \\ \displaystyle p_{\text{mask}}(1-p_{\text{data}}(x|c))p_{G}(x|c)\log(1-D(x,c))\Big)\text{\hskip 5.69054pt//generator samples incorrect token}$$

Finding the critical points of this loss with respect to $D$ shows that for a fixed generator the optimal discriminator is

发现与$D$有关的这一损失的临界点表明,对于固定发电机来说,最佳歧视者是:

$$\displaystyle D(x,c)=p_{\text{data}}(x|c)(a+p_{G}(x|c))/(ap_{\text{data}}(x|c)+p_{G}(x|c))$$

which means

也就是说

$$\displaystyle p_{\text{data}}(x|c)=D(x,c)p_{G}(x|c)/(a(1-D(x,c))+p_{G}(x|c))$$

where $a=(1-p_{\text{mask}})/p_{\text{mask}}$ is the number of unmasked tokens for every masked token. We can use this expression to evaluate ELECTRA as a masked language model by selecting $\text{argmax}_{x\in\text{vocab}}D(x,c)p_{G}(x|c)/(a(1-D(x,c))+p_{G}(x|c))$ as the model’s prediction for a given context. In practice, selecting over the whole vocabulary is very expensive, so we instead take the argmax over the top 100 predictions from the generator.[^10] Using this method, we compared ELECTRA-Base and BERT-Base on the Wikipedia+BooksCorpus dataset. We found that BERT slightly outperformed ELECTRA at masked language modeling (77.9% vs 75.5% accuracy). It is possible that the assumption of an optimal discriminator, which is certainly far from correct, is harming ELECTRA’s accuracy under this evaluation scheme. However, perhaps it is not too surprising that a model like BERT that is trained specifically for generation performs better at generation while a model with a discriminative objective like ELECTRA is better at being fine-tuned on discriminative tasks. We think comparisons of BERT’s and ELECTRA’s MLM predictions might be an interesting way to uncover more about the differences between ELECTRA and BERT encoders in future work.

其中$a=(1-p_{\text{mask}})/p_{\text{mask}}$是每个蒙面标志的未蒙面标志的数量。 我们可以使用这个表达式来评价ELECTRA作为一个蒙面语言模型,选择$\text{argmax}_{x\in\text{vocab}}D(x,c)p_{G}(x|c)/(a(1-D(x,c))+p_{G}(x|c))$作为模型对特定上下文的预测. 在实践中,选择整个词汇非常昂贵,因此我们从生成器中取出最大100个预测值的正格。[^10] 使用这种方法,我们在维基百科+BooksCorpus数据集上比较了ELECTRA-Base和BERT-Base. 我们发现BERT在蒙面语言建模(77.9%对75.5%的精度)上的表现略高于ELECTRA. 假设一个最优的歧视者(这肯定远远不正确)有可能损害ELECTRA在这一评价办法下的准确性。逍遥影视公司(ELECTRA)等具有歧视性目标的模式更适合在歧视性任务上进行微调。 我们觉得比较BERT和ELECTRA的MLM预测可能是一种有趣的方法,

Appendix H Negative Results

附录 H 负结果

We briefly describe a few ideas that did not look promising in our initial experiments:

我们简要地描述了一些在我们最初的实验中看似没有希望的想法:

• • We initially attempted to make BERT more efficient by strategically masking-out tokens (e.g., masking our rarer tokens more frequently, or training a model to guess which tokens BERT would struggle to predict if they were masked out). This resulted in fairly minor speedups over regular BERT.
• • Given that ELECTRA seemed to benefit (up to a certain point) from having a weaker generator (see Section 3.2), we explored raising the temperature of the generator’s output softmax or disallowing the generator from sampling the correct token. Neither of these improved results.
• • We tried adding a sentence-level contrastive objective. For this task, we kept 20% of input sentences unchanged rather than noising them with the generator. We then added a prediction head to the model that predicted if the entire input was corrupted or not. Surprisingly, this slightly decreased scores on downstream tasks.

· 我们最初试图通过战略掩蔽符号使BERT更加有效(例如,更频繁地掩盖我们稀有的代号,或训练一个模型来猜出BERT的哪些代号如果蒙蔽的话会难以预测)。 这导致比普通的BERT速度小得多。

• 鉴于ELECTRA似乎(在一定时间内)由于向秘书处提供较弱的发电机而受益(见第3.2节),我们探索提高发电机输出软马克的温度,或不让发电机取样正确信号。 这些成果都没有改善。
• 我们试图增加一个句子层面的对比目标。 对于这项任务,我们保留了20%的输入句不变,而不是用生成器点头. 然后我们在模型中增加了一个预测头,预测整个输入是否被腐蚀. 令人惊讶的是,下游任务的分数略有下降。