BERTScore: Evaluating Text Generation with BERT | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
摘要1 引言2 问题定义与已有指标2.1 基于 $n$-gram 匹配的方法BLEUMETEOR其他相关指标2.2 基于编辑距离的指标2.3 基于嵌入的指标2.4 学习型指标3 BERTScore词元表示相似度度量BERTScore重要性加权基线缩放4 实验设置上下文嵌入模型机器翻译图像描述5 结果机器翻译图像描述速度6 稳健性分析7 讨论致谢参考文献附录 A 定性分析附录 B 表示层的选择附录 C MoverScore 的消融实验附录 D 抽象文本压缩上的补充实验附录 E 近期机器翻译模型的 BERTScore附录 F 补充结果F.1 WMT 相关性研究实验设置结果F.2 模型选择研究实验设置结果F.3 COCO 上的图像描述F.4 PAWS-QQP 上的稳健性分析
返回论文列表
目录

BERTScore: Evaluating Text Generation with BERT

收录于 ICLR 2020 会议论文;The Eighth International Conference on Learning Representations。公开 PDF 首页写明 Published as a conference paper at ICLR 2020被引 8,443 次
打开论文来源

BERTScore: Evaluating Text Generation with BERT

Tianyi Zhang∗†‡⋄^{\ast\dagger\ddagger\diamond}∗†‡⋄, Varsha Kishore∗‡^{\ast\ddagger}∗‡, Felix Wu∗‡^{\ast\ddagger}∗‡, Kilian Q. Weinberger†‡⋄^{\dagger\ddagger\diamond}†‡⋄, and Yoav Artzi‡§^{\ddagger\S}‡§

‡^{\ddagger}‡ Department of Computer Science and §^{\S}§ Cornell Tech, Cornell University

{vk352, fw245, kilian}@cornell.edu {yoav}@cs.cornell.edu
⋄^{\diamond}⋄ ASAPP Inc.
tzhang@asapp.com

∗^{\ast}∗ Equal contribution. †^{\dagger}† Work done at Cornell.

Abstract

摘要

We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

我们提出 BERTScore,这是一种用于文本生成的自动评测指标。与常见指标类似,BERTScore 会为候选句中的每个词元与参考句中的每个词元计算相似度分数。不同的是,它不以精确匹配为依据,而是用上下文嵌入来度量词元相似度。我们用 363 套机器翻译和图像描述系统的输出做了评测。BERTScore 与人工判断的相关性更好,模型选择表现也强于已有指标。最后,我们用一项对抗性释义检测任务表明,面对更难的例子,BERTScore 比已有指标更稳健。

1 Introduction

1 引言

Automatic evaluation of natural language generation, for example in machine translation and caption generation, requires comparing candidate sentences to annotated references. The goal is to evaluate semantic equivalence. However, commonly used methods rely on surface-form similarity only. For example, Bleu (Papineni et al., 2002), the most common machine translation metric, simply counts nnn-gram overlap between the candidate and the reference. While this provides a simple and general measure, it fails to account for meaning-preserving lexical and compositional diversity.

对自然语言生成做自动评测,例如机器翻译和图像描述,需要把候选句和已标注的参考句加以比较。目标是衡量语义是否等价。但常用方法往往只依赖表层形式的相似度。例如,机器翻译里最常用的指标 BLEU(Papineni et al., 2002)只是统计候选句和参考句之间的 nnn-gram 重叠。这种方法简单也通用,却无法刻画那些保持意义不变的词汇和组合多样性。

In this paper, we introduce BERTScore, a language generation evaluation metric based on pre-trained BERT contextual embeddings (Devlin et al., 2019). BERTScore computes the similarity of two sentences as a sum of cosine similarities between their tokens’ embeddings.

本文引入 BERTScore,这是一种基于预训练 BERT 上下文嵌入(Devlin et al., 2019)的语言生成评测指标。BERTScore 把两个句子的相似度,算成它们各个词元嵌入之间余弦相似度之和。

BERTScore addresses two common pitfalls in nnn-gram-based metrics (Banerjee & Lavie, 2005). First, such methods often fail to robustly match paraphrases. For example, given the reference people like foreign cars, Bleu and Meteor (Banerjee & Lavie, 2005) incorrectly give a higher score to people like visiting places abroad compared to consumers prefer imported cars. This leads to performance underestimation when semantically-correct phrases are penalized because they differ from the surface form of the reference. In contrast to string matching (e.g., in Bleu) or matching heuristics (e.g., in Meteor), we compute similarity using contextualized token embeddings, which have been shown to be effective for paraphrase detection (Devlin et al., 2019). Second, nnn-gram models fail to capture distant dependencies and penalize semantically-critical ordering changes (Isozaki et al., 2010). For example, given a small window of size two, Bleu will only mildly penalize swapping of cause and effect clauses (e.g. A because B instead of B because A), especially when the arguments A and B are long phrases. In contrast, contextualized embeddings are trained to effectively capture distant dependencies and ordering.

BERTScore 针对基于 nnn-gram 的指标里两类常见缺陷(Banerjee & Lavie, 2005)。第一,这类方法往往难以稳健地匹配释义。例如,给定参考句 people like foreign cars,BLEU 和 METEOR(Banerjee & Lavie, 2005)会错误地把 people like visiting places abroad 打得比 consumers prefer imported cars 更高。语义正确、只是字面形式与参考句不同的短语因此被扣分,系统表现会被低估。与字符串匹配(如 BLEU)或启发式匹配(如 METEOR)不同,我们用上下文相关的词元嵌入来计算相似度,这类表示已被证明对释义检测有效(Devlin et al., 2019)。第二,nnn-gram 模型抓不住远距离依赖,也难以对语义上至关重要的语序变化给出足够惩罚(Isozaki et al., 2010)。例如,在大小为二的短窗口下,把因果从句对调(如把 A because B 写成 B because A),尤其当论元 A 和 B 本身是较长短语时,BLEU 往往只轻轻扣一点分。相比之下,上下文嵌入在训练中就被要求有效捕捉远距离依赖和语序。

We experiment with BERTScore on machine translation and image captioning tasks using the outputs of 363 systems by correlating BERTScore and related metrics to available human judgments. Our experiments demonstrate that BERTScore correlates highly with human evaluations. In machine translation, BERTScore shows stronger system-level and segment-level correlations with human judgments than existing metrics on multiple common benchmarks and demonstrates strong model selection performance compared to Bleu. We also show that BERTScore is well-correlated with human annotators for image captioning, surpassing Spice, a popular task-specific metric (Anderson et al., 2016). Finally, we test the robustness of BERTScore on the adversarial paraphrase dataset PAWS (Zhang et al., 2019), and show that it is more robust to adversarial examples than other metrics. The code for BERTScore is available at https://github.com/Tiiiger/bert_score.

我们在机器翻译和图像描述任务上用 363 套系统的输出做实验,把 BERTScore 及相关指标与已有的人工判断做相关分析。实验表明,BERTScore 与人工评价高度相关。在机器翻译中,它在多个常用基准上的系统级和片段级人工判断相关性都强于已有指标,模型选择表现也明显好于 BLEU。我们还表明,在图像描述上 BERTScore 与人工标注同样相关良好,并超过了流行的任务专用指标 SPICE(Anderson et al., 2016)。最后,我们在对抗释义数据集 PAWS(Zhang et al., 2019)上测试稳健性,结果显示它比其他指标更能抵抗对抗样本。BERTScore 的代码发布在 https://github.com/Tiiiger/bert_score。

2 Problem Statement and Prior Metrics

2 问题定义与已有指标

Natural language text generation is commonly evaluated using annotated reference sentences. Given a reference sentence xxx tokenized to kkk tokens ⟨x1,…,xk⟩\langle x_{1},\dots,x_{k}\rangle⟨x1​,…,xk​⟩ and a candidate x^\hat{x}x^ tokenized to lll tokens ⟨x^1,…,x^l⟩\langle\hat{x}_{1},\dots,\hat{x}_{l}\rangle⟨x^1​,…,x^l​⟩, a generation evaluation metric is a function f(x,x^)∈\mathdsRf(x,\hat{x})\in\mathds{R}f(x,x^)∈\mathdsR. Better metrics have a higher correlation with human judgments. Existing metrics can be broadly categorized into using nnn-gram matching, edit distance, embedding matching, or learned functions.

自然语言文本生成通常用已标注的参考句来评测。给定经分词后含 kkk 个词元的参考句 x=⟨x1,…,xk⟩x=\langle x_{1},\dots,x_{k}\ranglex=⟨x1​,…,xk​⟩,以及含 lll 个词元的候选句 x^=⟨x^1,…,x^l⟩\hat{x}=\langle\hat{x}_{1},\dots,\hat{x}_{l}\ranglex^=⟨x^1​,…,x^l​⟩,生成评测指标是一个函数 f(x,x^)∈\mathdsRf(x,\hat{x})\in\mathds{R}f(x,x^)∈\mathdsR。更好的指标应与人工判断有更高的相关性。已有指标大致可分为 nnn-gram 匹配、编辑距离、嵌入匹配,以及学习得到的函数。

2.1 nnn-gram Matching Approaches

2.1 基于 nnn-gram 匹配的方法

The most commonly used metrics for generation count the number of nnn-grams that occur in the reference xxx and candidate x^\hat{x}x^. The higher the nnn is, the more the metric is able to capture word order, but it also becomes more restrictive and constrained to the exact form of the reference.

生成评测里最常用的一类指标,是统计参考句 xxx 和候选句 x^\hat{x}x^ 中共同出现的 nnn-gram 数量。nnn 越大,指标越能刻画词序,但也会变得更苛刻,更受参考句字面形式的约束。

Formally, let SxnS^{n}_{x}Sxn​ and Sx^nS^{n}_{\hat{x}}Sx^n​ be the lists of token nnn-grams (n∈Z+n\in\mathbb{Z}_{+}n∈Z+​) in the reference xxx and candidate x^\hat{x}x^ sentences. The number of matched nnn-grams is ∑w∈Sx^nI[w∈Sxn]\sum_{w\in S^{n}_{\hat{x}}}\mathbb{I}[w\in S^{n}_{x}]∑w∈Sx^n​​I[w∈Sxn​], where I[⋅]\mathbb{I}[\cdot]I[⋅] is an indicator function. The exact match precision (Exact-Pn\text{Exact-P}_{n}Exact-Pn​) and recall (Exact-Rn\text{Exact-R}_{n}Exact-Rn​) scores are:

形式化地,设 SxnS^{n}_{x}Sxn​ 和 Sx^nS^{n}_{\hat{x}}Sx^n​ 分别是参考句 xxx 与候选句 x^\hat{x}x^ 中词元 nnn-gram(n∈Z+n\in\mathbb{Z}_{+}n∈Z+​)的列表。匹配上的 nnn-gram 数量为 ∑w∈Sx^nI[w∈Sxn]\sum_{w\in S^{n}_{\hat{x}}}\mathbb{I}[w\in S^{n}_{x}]∑w∈Sx^n​​I[w∈Sxn​],其中 I[⋅]\mathbb{I}[\cdot]I[⋅] 是指示函数。精确匹配的准确率(Exact-Pn\text{Exact-P}_{n}Exact-Pn​)和召回率(Exact-Rn\text{Exact-R}_{n}Exact-Rn​)为:

Exact-Pn=∑w∈Sx^nI[w∈Sxn]∣Sx^n∣andExact-Rn=∑w∈SxnI[w∈Sx^n]∣Sxn∣.\displaystyle\text{Exact-P}_{n}=\frac{\sum_{w\in S^{n}_{\hat{x}}}\mathbb{I}[w\in S^{n}_{x}]}{\left\lvert S^{n}_{\hat{x}}\right\rvert}\quad\text{and}\quad\text{Exact-R}_{n}=\frac{\sum_{w\in S^{n}_{x}}\mathbb{I}[w\in S^{n}_{\hat{x}}]}{\left\lvert S^{n}_{x}\right\rvert}.Exact-Pn​=∣Sx^n​∣∑w∈Sx^n​​I[w∈Sxn​]​andExact-Rn​=∣Sxn​∣∑w∈Sxn​​I[w∈Sx^n​]​.

Several popular metrics build upon one or both of these exact matching scores.

若干常用指标,都建立在这两种精确匹配分数之一或二者之上。

Bleu

BLEU

The most widely used metric in machine translation is Bleu (Papineni et al., 2002), which includes three modifications to Exact-Pn\text{Exact-P}_{n}Exact-Pn​. First, each nnn-gram in the reference can be matched at most once. Second, the number of exact matches is accumulated for all reference-candidate pairs in the corpus and divided by the total number of nnn-grams in all candidate sentences. Finally, very short candidates are discouraged using a brevity penalty. Typically, Bleu is computed for multiple values of nnn (e.g. n=1,2,3,4n=1,2,3,4n=1,2,3,4) and the scores are averaged geometrically. A smoothed variant, SentBleu (Koehn et al., 2007) is computed at the sentence level. In contrast to Bleu, BERTScore is not restricted to maximum nnn-gram length, but instead relies on contextualized embeddings that are able to capture dependencies of potentially unbounded length.

机器翻译中用得最广的指标是 BLEU(Papineni et al., 2002),它在 Exact-Pn\text{Exact-P}_{n}Exact-Pn​ 上做了三处修改。第一,参考句中的每个 nnn-gram 最多只能被匹配一次。第二,先把语料中所有参考–候选对上的精确匹配数累加,再除以所有候选句中的 nnn-gram 总数。第三,用简洁惩罚抑制过短的候选句。实践中,BLEU 通常对多个 nnn(如 n=1,2,3,4n=1,2,3,4n=1,2,3,4)分别计算,再做几何平均。其平滑变体 SentBLEU(Koehn et al., 2007)则在句子级计算。与 BLEU 不同,BERTScore 不受最大 nnn-gram 长度限制,而是依赖能够捕捉潜在无界长度依赖的上下文嵌入。

Meteor

METEOR

Meteor (Banerjee & Lavie, 2005) computes Exact-P1\text{Exact-P}_{1}Exact-P1​ and Exact-R1\text{Exact-R}_{1}Exact-R1​ while allowing backing-off from exact unigram matching to matching word stems, synonyms, and paraphrases. For example, running may match run if no exact match is possible. Non-exact matching uses an external stemmer, a synonym lexicon, and a paraphrase table. Meteor 1.5 (Denkowski & Lavie, 2014) weighs content and function words differently, and also applies importance weighting to different matching types. The more recent Meteor++ 2.0 (Guo & Hu, 2019) further incorporates a learned external paraphrase resource. Because Meteor requires external resources, only five languages are supported with the full feature set, and eleven are partially supported. Similar to Meteor, BERTScore allows relaxed matches, but relies on BERT embeddings that are trained on large amounts of raw text and are currently available for 104 languages. BERTScore also supports importance weighting, which we estimate with simple corpus statistics.

METEOR(Banerjee & Lavie, 2005)计算 Exact-P1\text{Exact-P}_{1}Exact-P1​ 和 Exact-R1\text{Exact-R}_{1}Exact-R1​,并允许从精确的 unigram 匹配回退到词干、同义词和释义匹配。例如,若无法精确匹配,running 可以和 run 对齐。非精确匹配要借助外部词干器、同义词典和释义表。METEOR 1.5(Denkowski & Lavie, 2014)对实词和功能词给予不同权重,并对不同匹配类型做重要性加权。更晚近的 METEOR++ 2.0(Guo & Hu, 2019)还引入了学习得到的外部释义资源。因为 METEOR 依赖外部资源,完整功能只支持五种语言,另有十一种语言得到部分支持。与 METEOR 类似,BERTScore 也允许宽松匹配,但它依赖在大量原始文本上训练、目前覆盖 104 种语言的 BERT 嵌入。BERTScore 同样支持重要性加权,权值由简单的语料统计来估计。

Other Related Metrics

其他相关指标

NIST (Doddington, 2002) is a revised version of Bleu that weighs each nnn-gram differently and uses an alternative brevity penalty. Δ\DeltaΔBleu (Galley et al., 2015) modifies multi-reference Bleu by including human annotated negative reference sentences. chrF (Popović, 2015) compares character nnn-grams in the reference and candidate sentences. chrF++ (Popović, 2017) extends chrF to include word bigram matching. Rouge (Lin, 2004) is a commonly used metric for summarization evaluation. Rouge-nnn (Lin, 2004) computes Exact-Rn\text{Exact-R}_{n}Exact-Rn​ (usually n=1,2n=1,2n=1,2), while Rouge-LLL is a variant of Exact-R1\text{Exact-R}_{1}Exact-R1​ with the numerator replaced by the length of the longest common subsequence. CIDEr (Vedantam et al., 2015) is an image captioning metric that computes cosine similarity between tf{\rm tf}tf–idf{\rm idf}idf weighted nnn-grams. We adopt a similar approach to weigh tokens differently. Finally, Chaganty et al. (2018) and Hashimoto et al. (2019) combine automatic metrics with human judgments for text generation evaluation.

NIST(Doddington, 2002)是 BLEU 的修订版,它对每个 nnn-gram 赋予不同权重,并采用另一种简洁惩罚。Δ\DeltaΔBLEU(Galley et al., 2015)改造了多参考 BLEU,把人工标注的负向参考句也纳入计算。chrF(Popović, 2015)比较参考句和候选句中的字符 nnn-gram。chrF++(Popović, 2017)在 chrF 之上加入了词 bigram 匹配。ROUGE(Lin, 2004)是摘要评测中的常用指标。ROUGE-nnn(Lin, 2004)计算 Exact-Rn\text{Exact-R}_{n}Exact-Rn​(通常 n=1,2n=1,2n=1,2),而 ROUGE-L 是 Exact-R1\text{Exact-R}_{1}Exact-R1​ 的变体,把分子换成最长公共子序列的长度。CIDEr(Vedantam et al., 2015)是一种图像描述指标,计算 tf{\rm tf}tf–idf{\rm idf}idf 加权 nnn-gram 之间的余弦相似度。我们也采用类似思路,给不同词元赋不同权重。最后,Chaganty et al.(2018)和 Hashimoto et al.(2019)把自动指标与人工判断结合起来,用于生成评测。

2.2 Edit-distance-based Metrics

2.2 基于编辑距离的指标

Several methods use word edit distance or word error rate (Levenshtein, 1966), which quantify similarity using the number of edit operations required to get from the candidate to the reference. TER (Snover et al., 2006) normalizes edit distance by the number of reference words, and ITER (Panja & Naskar, 2018) adds stem matching and better normalization. PER (Tillmann et al., 1997) computes position independent error rate, CDER (Leusch et al., 2006) models block reordering as an edit operation. CharacTER (Wang et al., 2016) and EED (Stanchev et al., 2019) operate on the character level and achieve higher correlation with human judgements on some languages.

有一类方法使用词编辑距离或词错误率(Levenshtein, 1966),用把候选句改成参考句所需的编辑操作数来度量相似度。TER(Snover et al., 2006)用参考词数对编辑距离做归一化,ITER(Panja & Naskar, 2018)则加入词干匹配,并改进了归一化。PER(Tillmann et al., 1997)计算与位置无关的错误率,CDER(Leusch et al., 2006)把块重排也建模成一种编辑操作。CharacTER(Wang et al., 2016)和 EED(Stanchev et al., 2019)在字符级操作,并在部分语言上与人工判断取得了更高的相关性。

2.3 Embedding-based Metrics

2.3 基于嵌入的指标

Word embeddings (Mikolov et al., 2013; Pennington et al., 2014; Grave et al., 2018; Nguyen et al., 2017; Athiwaratkun et al., 2018) are learned dense token representations. MEANT 2.0 (Lo, 2017) uses word embeddings and shallow semantic parses to compute lexical and structural similarity. YiSi-1 (Lo et al., 2018) is similar to MEANT 2.0, but makes the use of semantic parses optional. Both methods use a relatively simple similarity computation, which inspires our approach, including using greedy matching (Corley & Mihalcea, 2005) and experimenting with a similar importance weighting to YiSi-1. However, we use contextual embeddings, which capture the specific use of a token in a sentence, and potentially capture sequence information. We do not use external tools to generate linguistic structures, which makes our approach relatively simple and portable to new languages. Instead of greedy matching, WMD (Kusner et al., 2015), WMDO (Chow et al., 2019), and SMS (Clark et al., 2019) propose to use optimal matching based on earth mover’s distance (Rubner et al., 1998). The tradeoff[^1] between greedy and optimal matching was studied by Rus & Lintean (2012). Sharma et al. (2018) compute similarity with sentence-level representations. In contrast, our token-level computation allows us to weigh tokens differently according to their importance.

词嵌入(Mikolov et al., 2013;Pennington et al., 2014;Grave et al., 2018;Nguyen et al., 2017;Athiwaratkun et al., 2018)是学习得到的稠密词元表示。MEANT 2.0(Lo, 2017)用词嵌入和浅层语义分析来计算词汇与结构相似度。YiSi-1(Lo et al., 2018)与 MEANT 2.0 类似,但把语义分析变成可选项。这两种方法的相似度计算都相对简单,也启发了我们的做法,包括使用贪心匹配(Corley & Mihalcea, 2005),以及尝试与 YiSi-1 类似的重要性加权。不同的是,我们使用上下文嵌入,它能刻画一个词元在具体句子中的用法,并有可能捕捉序列信息。我们不借助外部工具去生成语言结构,因此方法相对简单,也更容易移植到新语言。与贪心匹配不同,WMD(Kusner et al., 2015)、WMDO(Chow et al., 2019)和 SMS(Clark et al., 2019)主张基于土堆移动距离(Rubner et al., 1998)做最优匹配。贪心匹配与最优匹配之间的取舍[^1],Rus & Lintean(2012)曾经研究过。Sharma et al.(2018)则用句子级表示计算相似度。相比之下,我们在词元级计算,从而可以按重要性给不同词元加权。

[^1] We provide an ablation study of this design choice in Appendix C.

[^1] 我们在附录 C 中对这一设计选择做了消融实验。

2.4 Learned Metrics

2.4 学习型指标

Various metrics are trained to optimize correlation with human judgments. Beer (Stanojević & Sima’an, 2014) uses a regression model based on character nnn-grams and word bigrams. Blend (Ma et al., 2017) uses regression to combine 29 existing metrics. Ruse (Shimanaka et al., 2018) combines three pre-trained sentence embedding models. All these methods require costly human judgments as supervision for each dataset, and risk poor generalization to new domains, even within a known language and task (Chaganty et al., 2018). Cui et al. (2018) and Lowe et al. (2017) train a neural model to predict if the input text is human-generated. This approach also has the risk of being optimized to existing data and generalizing poorly to new data. In contrast, the model underlying BERTScore is not optimized for any specific evaluation task.

还有一类指标是训练出来、用来最大化与人工判断的相关性。BEER(Stanojević & Sima’an, 2014)以字符 nnn-gram 和词 bigram 为特征做回归。BLEND(Ma et al., 2017)用回归把 29 种已有指标组合起来。RUSE(Shimanaka et al., 2018)则组合了三种预训练的句子嵌入模型。这些方法都需要为每个数据集准备昂贵的人工判断作为监督,即便在已知语言和任务内部,也存在难以泛化到新领域的风险(Chaganty et al., 2018)。Cui et al.(2018)和 Lowe et al.(2017)训练神经网络来预测输入文本是否由人撰写。这条路同样有可能过拟合已有数据,难以泛化到新数据。相比之下,BERTScore 所依赖的模型并没有针对任何特定评测任务去优化。

3 BERTScore

Given a reference sentence x=⟨x1,…,xk⟩x=\langle x_{1},\dots,x_{k}\ranglex=⟨x1​,…,xk​⟩ and a candidate sentence x^=⟨x^1,…,x^l⟩\hat{x}=\langle\hat{x}_{1},\dots,\hat{x}_{l}\ranglex^=⟨x^1​,…,x^l​⟩, we use contextual embeddings to represent the tokens, and compute matching using cosine similarity, optionally weighted with inverse document frequency scores. Figure 1 illustrates the computation.

给定参考句 x=⟨x1,…,xk⟩x=\langle x_{1},\dots,x_{k}\ranglex=⟨x1​,…,xk​⟩ 和候选句 x^=⟨x^1,…,x^l⟩\hat{x}=\langle\hat{x}_{1},\dots,\hat{x}_{l}\ranglex^=⟨x^1​,…,x^l​⟩,我们用上下文嵌入表示各个词元,再用余弦相似度做匹配,并可按逆向文档频率加权。图 1 展示了这一计算过程。

Illustration of BERTScore recall computation with contextual embeddings, pairwise cosine similarity, maximum similarity, and optional IDF weighting

Figure 1: Illustration of the computation of the recall metric R\textscBERTR_{\textsc{BERT}}R\textscBERT​. Given the reference xxx and candidate x^\hat{x}x^, we compute BERT embeddings and pairwise cosine similarity. We highlight the greedy matching in red, and include the optional idf{\rm idf}idf importance weighting.

图 1:召回率指标 R\textscBERTR_{\textsc{BERT}}R\textscBERT​ 的计算示意。给定参考句 xxx 和候选句 x^\hat{x}x^,我们计算 BERT 嵌入以及成对余弦相似度。贪心匹配以红色标出,并包含可选的 idf{\rm idf}idf 重要性加权。

Token Representation

词元表示

We use contextual embeddings to represent the tokens in the input sentences xxx and x^\hat{x}x^. In contrast to prior word embeddings (Mikolov et al., 2013; Pennington et al., 2014), contextual embeddings, such as BERT (Devlin et al., 2019) and ELMo (Peters et al., 2018), can generate different vector representations for the same word in different sentences depending on the surrounding words, which form the context of the target word. The models used to generate these embeddings are most commonly trained using various language modeling objectives, such as masked word prediction (Devlin et al., 2019).

我们用上下文嵌入来表示输入句 xxx 和 x^\hat{x}x^ 中的词元。与以往的词嵌入(Mikolov et al., 2013;Pennington et al., 2014)不同,BERT(Devlin et al., 2019)和 ELMo(Peters et al., 2018)这类上下文嵌入,能够根据周围词所构成的语境,为同一词在不同句子中生成不同的向量表示。生成这些嵌入的模型,通常用各种语言建模目标来训练,例如掩码词预测(Devlin et al., 2019)。

We experiment with different models (Section 4), using the tokenizer provided with each model. Given a tokenized reference sentence x=⟨x1,…,xk⟩x=\langle x_{1},\dots,x_{k}\ranglex=⟨x1​,…,xk​⟩, the embedding model generates a sequence of vectors ⟨x1,…,xk⟩\langle\mathbf{x}_{1},\dots,\mathbf{x}_{k}\rangle⟨x1​,…,xk​⟩. Similarly, the tokenized candidate x^=⟨x^1,…,x^m⟩\hat{x}=\langle\hat{x}_{1},\dots,\hat{x}_{m}\ranglex^=⟨x^1​,…,x^m​⟩ is mapped to ⟨x^1,…,x^l⟩\langle\mathbf{\hat{x}}_{1},\dots,\mathbf{\hat{x}}_{l}\rangle⟨x^1​,…,x^l​⟩. The main model we use is BERT, which tokenizes the input text into a sequence of word pieces (Wu et al., 2016), where unknown words are split into several commonly observed sequences of characters. The representation for each word piece is computed with a Transformer encoder (Vaswani et al., 2017) by repeatedly applying self-attention and nonlinear transformations in an alternating fashion. BERT embeddings have been shown to benefit various NLP tasks (Devlin et al., 2019; Liu, 2019; Huang et al., 2019; Yang et al., 2019a).

我们试验了不同模型(第 4 节),并使用各模型自带的分词器。给定经分词的参考句 x=⟨x1,…,xk⟩x=\langle x_{1},\dots,x_{k}\ranglex=⟨x1​,…,xk​⟩,嵌入模型会生成向量序列 ⟨x1,…,xk⟩\langle\mathbf{x}_{1},\dots,\mathbf{x}_{k}\rangle⟨x1​,…,xk​⟩。类似地,分词后的候选句 x^=⟨x^1,…,x^m⟩\hat{x}=\langle\hat{x}_{1},\dots,\hat{x}_{m}\ranglex^=⟨x^1​,…,x^m​⟩ 被映射为 ⟨x^1,…,x^l⟩\langle\mathbf{\hat{x}}_{1},\dots,\mathbf{\hat{x}}_{l}\rangle⟨x^1​,…,x^l​⟩。我们使用的主模型是 BERT,它把输入文本切成一串词片段(Wu et al., 2016);未知词会被拆成若干常见的字符序列。每个词片段的表示由 Transformer 编码器(Vaswani et al., 2017)算出,方式是交替地反复施加自注意力和非线性变换。BERT 嵌入已被证明有益于多种 NLP 任务(Devlin et al., 2019;Liu, 2019;Huang et al., 2019;Yang et al., 2019a)。

Similarity Measure

相似度度量

The vector representation allows for a soft measure of similarity instead of exact-string (Papineni et al., 2002) or heuristic (Banerjee & Lavie, 2005) matching. The cosine similarity of a reference token xix_{i}xi​ and a candidate token x^j\hat{x}_{j}x^j​ is xi⊤x^j∥xi∥∥x^j∥\frac{\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j}}{\|\mathbf{x}_{i}\|\|\mathbf{\hat{x}}_{j}\|}∥xi​∥∥x^j​∥xi⊤​x^j​​. We use pre-normalized vectors, which reduces this calculation to the inner product xi⊤x^j\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j}xi⊤​x^j​. While this measure considers tokens in isolation, the contextual embeddings contain information from the rest of the sentence.

有了向量表示,就可以用一种软的相似度来代替精确字符串匹配(Papineni et al., 2002)或启发式匹配(Banerjee & Lavie, 2005)。参考词元 xix_{i}xi​ 与候选词元 x^j\hat{x}_{j}x^j​ 的余弦相似度为 xi⊤x^j∥xi∥∥x^j∥\frac{\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j}}{\|\mathbf{x}_{i}\|\|\mathbf{\hat{x}}_{j}\|}∥xi​∥∥x^j​∥xi⊤​x^j​​。我们使用预先归一化的向量,于是该计算简化为内积 xi⊤x^j\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j}xi⊤​x^j​。这一度量虽然是在孤立地看各个词元,但上下文嵌入本身已经包含了来自句中其余部分的信息。

BERTScore

The complete score matches each token in xxx to a token in x^\hat{x}x^ to compute recall, and each token in x^\hat{x}x^ to a token in xxx to compute precision. We use greedy matching to maximize the matching similarity score,[^2] where each token is matched to the most similar token in the other sentence. We combine precision and recall to compute an F1 measure. For a reference xxx and candidate x^\hat{x}x^, the recall, precision, and F1 scores are:

完整的分数会把 xxx 中的每个词元与 x^\hat{x}x^ 中的某个词元对齐以计算召回率,再把 x^\hat{x}x^ 中的每个词元与 xxx 中的某个词元对齐以计算准确率。我们采用贪心匹配来最大化匹配相似度,[^2] 也就是把每个词元对到另一句中最相似的那个词元。再把准确率和召回率合成 F1。对参考句 xxx 和候选句 x^\hat{x}x^,召回率、准确率和 F1 分别为:

[^2] We compare greedy matching with optimal assignment in Appendix C.

[^2] 我们在附录 C 中把贪心匹配与最优分配做了比较。

R\textscBERT=1∣x∣∑xi∈xmax⁡x^j∈x^xi⊤x^j    ,P\textscBERT=1∣x^∣∑x^j∈x^max⁡xi∈xxi⊤x^j    ,F\textscBERT=2P\textscBERT⋅R\textscBERTP\textscBERT+R\textscBERT    .R_{\textsc{BERT}}=\frac{1}{|x|}\sum_{x_{i}\in x}\max_{\hat{x}_{j}\in\hat{x}}\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j}\;\;,\quad P_{\textsc{BERT}}=\frac{1}{|\hat{x}|}\sum_{\hat{x}_{j}\in\hat{x}}\max_{x_{i}\in x}\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j}\;\;,\quad F_{\textsc{BERT}}=2\frac{P_{\textsc{BERT}}\cdot R_{\textsc{BERT}}}{P_{\textsc{BERT}}+R_{\textsc{BERT}}}\;\;.R\textscBERT​=∣x∣1​xi​∈x∑​x^j​∈x^max​xi⊤​x^j​,P\textscBERT​=∣x^∣1​x^j​∈x^∑​xi​∈xmax​xi⊤​x^j​,F\textscBERT​=2P\textscBERT​+R\textscBERT​P\textscBERT​⋅R\textscBERT​​.

Importance Weighting

重要性加权

Previous work on similarity measures demonstrated that rare words can be more indicative for sentence similarity than common words (Banerjee & Lavie, 2005; Vedantam et al., 2015). BERTScore enables us to easily incorporate importance weighting. We experiment with inverse document frequency (idf{\rm idf}idf) scores computed from the test corpus. Given MMM reference sentences {x(i)}i=1M\{x^{(i)}\}_{i=1}^{M}{x(i)}i=1M​, the idf{\rm idf}idf score of a word-piece token www is

已有关于相似度度量的工作表明,罕见词往往比常见词更能说明句子相似度(Banerjee & Lavie, 2005;Vedantam et al., 2015)。BERTScore 可以很方便地引入重要性加权。我们试验了在测试语料上计算的逆向文档频率(idf{\rm idf}idf)分数。给定 MMM 个参考句 {x(i)}i=1M\{x^{(i)}\}_{i=1}^{M}{x(i)}i=1M​,词片段 www 的 idf{\rm idf}idf 分数为

idf(w)=−log⁡1M∑i=1MI[w∈x(i)]    ,{\rm idf}(w)=-\log\frac{1}{M}\sum_{i=1}^{M}\mathbb{I}[w\in x^{(i)}]\;\;,idf(w)=−logM1​i=1∑M​I[w∈x(i)],

where I[⋅]\mathbb{I}[\cdot]I[⋅] is an indicator function. We do not use the full tf{\rm tf}tf-idf{\rm idf}idf measure because we process single sentences, where the term frequency (tf{\rm tf}tf) is likely 1. For example, recall with idf{\rm idf}idf weighting is

其中 I[⋅]\mathbb{I}[\cdot]I[⋅] 是指示函数。我们不使用完整的 tf{\rm tf}tf-idf{\rm idf}idf,因为处理的是单句,词频(tf{\rm tf}tf)几乎总是 1。例如,带 idf{\rm idf}idf 加权的召回率为

R\textscBERT=∑xi∈xidf(xi)max⁡x^j∈x^xi⊤x^j∑xi∈xidf(xi)    .R_{\textsc{BERT}}=\frac{\sum_{x_{i}\in x}{\rm idf}(x_{i})\max_{\hat{x}_{j}\in\hat{x}}\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j}}{\sum_{x_{i}\in x}{\rm idf}(x_{i})}\;\;.R\textscBERT​=∑xi​∈x​idf(xi​)∑xi​∈x​idf(xi​)maxx^j​∈x^​xi⊤​x^j​​.

Because we use reference sentences to compute idf{\rm idf}idf, the idf{\rm idf}idf scores remain the same for all systems evaluated on a specific test set. We apply plus-one smoothing to handle unknown word pieces.

由于 idf{\rm idf}idf 是用参考句计算的,在同一个测试集上评估的所有系统会共用同一组 idf{\rm idf}idf 分数。对未见过的词片段,我们采用加一平滑。

Baseline Rescaling

基线缩放

Because we use pre-normalized vectors, our computed scores have the same numerical range of cosine similarity (between −1-1−1 and 111). However, in practice we observe scores in a more limited range, potentially because of the learned geometry of contextual embeddings. While this characteristic does not impact BERTScore’s capability to rank text generation systems, it makes the actual score less readable. We address this by rescaling BERTScore with respect to its empirical lower bound bbb as a baseline. We compute bbb using Common Crawl monolingual datasets.[^3] For each language and contextual embedding model, we create 111M candidate-reference pairs by grouping two random sentences. Because of the random pairing and the corpus diversity, each pair has very low lexical and semantic overlapping.[^4] We compute bbb by averaging BERTScore computed on these sentence pairs. Equipped with baseline bbb, we rescale BERTScore linearly. For example, the rescaled value R^\textscBERT\hat{R}_{\textsc{BERT}}R^\textscBERT​ of R\textscBERTR_{\textsc{BERT}}R\textscBERT​ is:

因为使用了预先归一化的向量,我们算出的分数与余弦相似度具有相同的数值范围(在 −1-1−1 与 111 之间)。但在实践中,我们观察到分数往往落在更窄的区间里,这可能与上下文嵌入学到的几何结构有关。这一特性并不影响 BERTScore 对文本生成系统排序的能力,却会让具体读数不太好懂。为此,我们用经验下界 bbb 作为基线,对 BERTScore 做缩放。bbb 在 Common Crawl 单语数据集上估计。[^3] 对每种语言和每种上下文嵌入模型,我们把随机抽取的句子两两配对,构成 100 万个候选–参考对。由于配对是随机的,再加上语料本身多样,每对句子的词汇和语义重叠都非常低。[^4] 我们在这些句子对上计算 BERTScore 并取平均,得到 bbb。有了基线 bbb 之后,再对 BERTScore 做线性缩放。例如,R\textscBERTR_{\textsc{BERT}}R\textscBERT​ 缩放后的值 R^\textscBERT\hat{R}_{\textsc{BERT}}R^\textscBERT​ 为:

[^3] https://commoncrawl.org/

[^4] BLEU computed on these pairs is around zero.

[^4] 在这些句子对上算出的 BLEU 大约为零。

R^\textscBERT=R\textscBERT−b1−b    .\hat{R}_{\textsc{BERT}}=\frac{R_{\textsc{BERT}}-b}{1-b}\;\;.R^\textscBERT​=1−bR\textscBERT​−b​.

After this operation R^\textscBERT\hat{R}_{\textsc{BERT}}R^\textscBERT​ is typically between 000 and 111. We apply the same rescaling procedure for P\textscBERTP_{\textsc{BERT}}P\textscBERT​ and F\textscBERTF_{\textsc{BERT}}F\textscBERT​. This method does not affect the ranking ability and human correlation of BERTScore, and is intended solely to increase the score readability.

经过这一步,R^\textscBERT\hat{R}_{\textsc{BERT}}R^\textscBERT​ 通常落在 000 到 111 之间。我们对 P\textscBERTP_{\textsc{BERT}}P\textscBERT​ 和 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 采用同样的缩放手续。这种方法不影响 BERTScore 的排序能力和与人工判断的相关性,目的只是提高分数的可读性。

4 Experimental Setup

4 实验设置

We evaluate our approach on machine translation and image captioning.

我们在机器翻译和图像描述上评估这一方法。

Contextual Embedding Models

上下文嵌入模型

We evaluate twelve pre-trained contextual embedding models, including variants of BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), XLNet (Yang et al., 2019b), and XLM (Lample & Conneau, 2019). We present the best-performing models in Section 5. We use the 24-layer RoBERTalarge\text{RoBERTa}_{{\rm large}}RoBERTalarge​ model[^5] for English tasks, 12-layer \textscBERTchinese\textsc{BERT}_{{\rm chinese}}\textscBERTchinese​ model for Chinese tasks, and the 12-layer cased multilingual \textscBERTmulti\textsc{BERT}_{{\rm multi}}\textscBERTmulti​ model for other languages.[^6] We show the performance of all other models in Appendix F. Contextual embedding models generate embedding representations at every layer in the encoder network. Past work has shown that intermediate layers produce more effective representations for semantic tasks (Liu et al., 2019a). We use the WMT16 dataset (Bojar et al., 2016) as a validation set to select the best layer of each model (Appendix B).

我们评估了十二种预训练的上下文嵌入模型,包括 BERT(Devlin et al., 2019)、RoBERTa(Liu et al., 2019b)、XLNet(Yang et al., 2019b)和 XLM(Lample & Conneau, 2019)的多种变体。第 5 节报告表现最好的模型。英语任务使用 24 层的 RoBERTalarge\text{RoBERTa}_{{\rm large}}RoBERTalarge​ 模型,[^5] 中文任务使用 12 层的 \textscBERTchinese\textsc{BERT}_{{\rm chinese}}\textscBERTchinese​ 模型,其他语言使用 12 层、区分大小写的多语 \textscBERTmulti\textsc{BERT}_{{\rm multi}}\textscBERTmulti​ 模型。[^6] 其余模型的结果放在附录 F。上下文嵌入模型会在编码器网络的每一层都生成表示。已有工作表明,中间层对语义任务往往能给出更有效的表示(Liu et al., 2019a)。我们把 WMT16 数据集(Bojar et al., 2016)当作验证集,为每个模型选择最好的一层(附录 B)。

[^5] We use the tokenizer provided with each model. For all Hugging Face models that use the GPT-2 tokenizer, at the time of our experiments, the tokenizer adds a space to the beginning of each sentence.

[^5] 我们使用各模型自带的分词器。对所有使用 GPT-2 分词器的 Hugging Face 模型,在我们做实验时,该分词器会在每个句子开头加一个空格。

[^6] All the models used are from https://github.com/huggingface/pytorch-transformers.

[^6] 所用模型全部来自 https://github.com/huggingface/pytorch-transformers。

Machine Translation

机器翻译

Our main evaluation corpus is the WMT18 metric evaluation dataset (Ma et al., 2018), which contains predictions of 149 translation systems across 14 language pairs, gold references, and two types of human judgment scores. Segment-level human judgments assign a score to each reference-candidate pair. System-level human judgments associate each system with a single score based on all pairs in the test set. WMT18 includes translations from English to Czech, German, Estonian, Finnish, Russian, and Turkish, and from the same set of languages to English. We follow the WMT18 standard practice and use absolute Pearson correlation ∣ρ∣\lvert\rho\rvert∣ρ∣ and Kendall rank correlation τ\tauτ to evaluate metric quality, and compute significance with the Williams test (Williams, 1959) for ∣ρ∣\lvert\rho\rvert∣ρ∣ and bootstrap re-sampling for τ\tauτ as suggested by Graham & Baldwin (2014). We compute system-level scores by averaging BERTScore for every reference-candidate pair. We also experiment with hybrid systems by randomly sampling one candidate sentence from one of the available systems for each reference sentence (Graham & Liu, 2016). This enables system-level experiments with a higher number of systems. Human judgments of each hybrid system are created by averaging the WMT18 segment-level human judgments for the corresponding sentences in the sampled data. We compare BERTScores to one canonical metric for each category introduced in Section 2, and include the comparison with all other participating metrics from WMT18 in Appendix F.

我们的主评测语料是 WMT18 指标评测数据集(Ma et al., 2018),其中包含 14 个语言对上 149 套翻译系统的输出、标准参考译文,以及两类人工判断分数。片段级人工判断给每一对参考–候选打一个分;系统级人工判断则根据测试集中的全部句子对,给每个系统一个总分。WMT18 包含从英语到捷克语、德语、爱沙尼亚语、芬兰语、俄语和土耳其语的翻译,以及从这些语言到英语的翻译。我们遵循 WMT18 的惯例,用绝对皮尔逊相关 ∣ρ∣\lvert\rho\rvert∣ρ∣ 和肯德尔秩相关 τ\tauτ 来衡量指标质量,并按 Graham & Baldwin(2014)的建议,对 ∣ρ∣\lvert\rho\rvert∣ρ∣ 用 Williams 检验(Williams, 1959)做显著性,对 τ\tauτ 做自助重采样。系统级分数由每一对参考–候选的 BERTScore 取平均得到。我们还按 Graham & Liu(2016)的做法构造混合系统:对每个参考句,从已有系统中随机抽一句候选。这样可以在系统数量更多的条件下做系统级实验。每个混合系统的人工判断,由抽样数据中相应句子的 WMT18 片段级人工判断取平均得到。我们把 BERTScore 与第 2 节各类指标中的一个代表性指标比较,并在附录 F 中补充与 WMT18 全部参赛指标的比较。

In addition to the standard evaluation, we design model selection experiments. We use 10K hybrid systems super-sampled from WMT18. We randomly select 100 out of 10K hybrid systems, and rank them using the automatic metrics. We repeat this process 100K times. We report the percentage of the metric ranking agreeing with the human ranking on the best system (Hits@1). In Tables 23-28, we include two additional measures to the model selection study: (a) the mean reciprocal rank of the top metric-rated system according to the human ranking, and (b) the difference between the human score of the top human-rated system and that of the top metric-rated system.

在标准评测之外,我们还设计了模型选择实验。我们从 WMT18 超采样得到 1 万个混合系统,再从中随机抽出 100 个,用自动指标排序,并将这一过程重复 10 万次。我们报告指标排出的最优系统与人工排序一致的比例(Hits@1)。在表 23–28 中,模型选择研究还加了两项度量:(a)按人工排序,指标评为最优的那个系统的平均倒数排名;(b)人工评为最优的系统与指标评为最优的系统,两者人工分数之差。

Additionally, we report the same study on the WMT17 (Bojar et al., 2017) and the WMT16 (Bojar et al., 2016) datasests in Appendix F.[^7] This adds 202 systems to our evaluation.

此外,我们在附录 F 中对 WMT17(Bojar et al., 2017)和 WMT16(Bojar et al., 2016)数据集报告了同样的研究。[^7] 这为我们的评测又增加了 202 套系统。

[^7] For WMT16, we only conduct segment-level experiments on to-English pairs due to errors in the dataset.

[^7] 由于数据集中存在错误,对 WMT16 我们只做了到英语方向的片段级实验。

Image Captioning

图像描述

We use the human judgments of twelve submission entries from the COCO 2015 Captioning Challenge. Each participating system generates a caption for each image in the COCO validation set (Lin et al., 2014), and each image has approximately five reference captions. Following Cui et al. (2018), we compute the Pearson correlation with two system-level metrics: the percentage of captions that are evaluated as better or equal to human captions (M1) and the percentage of captions that are indistinguishable from human captions (M2). We compute BERTScore with multiple references by scoring the candidate with each available reference and returning the highest score. We compare with eight task-agnostic metrics: Bleu (Papineni et al., 2002), Meteor (Banerjee & Lavie, 2005), Rouge-L (Lin, 2004), CIDEr (Vedantam et al., 2015), BEER (Stanojević & Sima’an, 2014), EED (Stanchev et al., 2019), chrF++ (Popović, 2017), and CharacTER (Wang et al., 2016). We also compare with two task-specific metrics: Spice (Anderson et al., 2016) and Leic (Cui et al., 2018). Spice is computed using the similarity of scene graphs parsed from the reference and candidate captions. Leic is trained to predict if a caption is written by a human given the image.

我们使用 COCO 2015 Captioning Challenge 中十二个提交条目的人工判断。每个参赛系统都为 COCO 验证集(Lin et al., 2014)中的每张图像生成一句描述,每张图像大约有五句参考描述。按照 Cui et al.(2018)的做法,我们计算与两项系统级度量的皮尔逊相关:被评为优于或相当于人工描述的比例(M1),以及被评为与人工描述难以区分的比例(M2)。在有多个参考句时,BERTScore 对每条参考分别打分,再取最高分。我们与八种不限定任务的指标比较:BLEU(Papineni et al., 2002)、METEOR(Banerjee & Lavie, 2005)、ROUGE-L(Lin, 2004)、CIDEr(Vedantam et al., 2015)、BEER(Stanojević & Sima’an, 2014)、EED(Stanchev et al., 2019)、chrF++(Popović, 2017)和 CharacTER(Wang et al., 2016)。另外还与两种任务专用指标比较:SPICE(Anderson et al., 2016)和 LEIC(Cui et al., 2018)。SPICE 比较从参考描述和候选描述中解析出的场景图相似度。LEIC 则被训练成:在给定图像的情况下,预测一句描述是否由人撰写。

Table 1: Absolute Pearson correlations with system-level human judgments on WMT18. For each language pair, the left number is the to-English correlation, and the right is the from-English. We bold correlations of metrics not significantly outperformed by any other metric under Williams Test for that language pair and direction. The numbers in parenthesis are the number of systems used for each language pair and direction.

Metric en↔\leftrightarrow↔cs en↔\leftrightarrow↔de en↔\leftrightarrow↔et en↔\leftrightarrow↔fi en↔\leftrightarrow↔ru en↔\leftrightarrow↔tr en↔\leftrightarrow↔zh
(5/5) (16/16) (14/14) (9/12) (8/9) (5/8) (14/14)
BLEU .970/.995 .971/.981 .986/.975 .973/.962 .979/.983 .657/.826 .978/.947
ITER .975/.915 .990/.984 .975/.981 .996/.973 .937/.975 .861/.865 .980/.0–
RUSE .981/.0– .997/.0– .990/.0– .991/.0– .988/.0– .853/.0– .981/.0–
YiSi-1 .950/.987 .992/.985 .979/.979 .973/.940 .991/.992 .958/.976 .951/.963
PBERTP_{\text{BERT}}PBERT​ .980/.994 .998/.988 .990/.981 .995/.957 .982/.990 .791/.935 .981/.954
RBERTR_{\text{BERT}}RBERT​ .998/.997 .997/.990 .986/.980 .997/.980 .995/.989 .054/.879 .990/.976
FBERTF_{\text{BERT}}FBERT​ .990/.997 .999/.989 .990/.982 .998/.972 .990/.990 .499/.908 .988/.967
FBERTF_{\text{BERT}}FBERT​ (idf) .985/.995 .999/.990 .992/.981 .992/.972 .991/.991 .826/.941 .989/.973

表 1:与 WMT18 系统级人工判断的绝对皮尔逊相关。对每个语言对,左数为到英语的相关,右数为从英语出发的相关。在该语言对和方向上,按 Williams 检验未被任何其他指标显著超过的相关加粗。括号中的数字是每个语言对和方向所用的系统数。

指标 en↔\leftrightarrow↔cs en↔\leftrightarrow↔de en↔\leftrightarrow↔et en↔\leftrightarrow↔fi en↔\leftrightarrow↔ru en↔\leftrightarrow↔tr en↔\leftrightarrow↔zh
(5/5) (16/16) (14/14) (9/12) (8/9) (5/8) (14/14)
BLEU .970/.995 .971/.981 .986/.975 .973/.962 .979/.983 .657/.826 .978/.947
ITER .975/.915 .990/.984 .975/.981 .996/.973 .937/.975 .861/.865 .980/.0–
RUSE .981/.0– .997/.0– .990/.0– .991/.0– .988/.0– .853/.0– .981/.0–
YiSi-1 .950/.987 .992/.985 .979/.979 .973/.940 .991/.992 .958/.976 .951/.963
PBERTP_{\text{BERT}}PBERT​ .980/.994 .998/.988 .990/.981 .995/.957 .982/.990 .791/.935 .981/.954
RBERTR_{\text{BERT}}RBERT​ .998/.997 .997/.990 .986/.980 .997/.980 .995/.989 .054/.879 .990/.976
FBERTF_{\text{BERT}}FBERT​ .990/.997 .999/.989 .990/.982 .998/.972 .990/.990 .499/.908 .988/.967
FBERTF_{\text{BERT}}FBERT​ (idf) .985/.995 .999/.990 .992/.981 .992/.972 .991/.991 .826/.941 .989/.973

Table 2: Absolute Pearson correlations with system-level human judgments on WMT18. We use 10K hybrid super-sampled systems for each language pair and direction. For each language pair, the left number is the to-English correlation, and the right is the from-English. Bolding criteria is the same as in Table 1.

Metric en↔\leftrightarrow↔cs en↔\leftrightarrow↔de en↔\leftrightarrow↔et en↔\leftrightarrow↔fi en↔\leftrightarrow↔ru en↔\leftrightarrow↔tr en↔\leftrightarrow↔zh
BLEU .956/.993 .969/.977 .981/.971 .962/.958 .972/.977 .586/.796 .968/.941
ITER .966/.865 .990/.978 .975/.982 .989/.966 .943/.965 .742/.872 .978/.0–
RUSE .974/.0– .996/.0– .988/.0– .983/.0– .982/.0– .780/.0– .973/.0–
YiSi-1 .942/.985 .991/.983 .976/.976 .964/.938 .985/.989 .881/.942 .943/.957
PBERTP_{\text{BERT}}PBERT​ .965/.989 .995/.983 .990/.970 .976/.951 .976/.988 .846/.936 .975/.950
RBERTR_{\text{BERT}}RBERT​ .989/.995 .997/.991 .982/.979 .989/.977 .988/.989 .540/.872 .981/.980
FBERTF_{\text{BERT}}FBERT​ .978/.993 .998/.988 .989/.978 .983/.969 .985/.989 .760/.910 .981/.969
FBERTF_{\text{BERT}}FBERT​ (idf) .982/.995 .998/.988 .988/.979 .989/.969 .983/.987 .453/.877 .980/.963

表 2:与 WMT18 系统级人工判断的绝对皮尔逊相关。每个语言对和方向使用 1 万个超采样得到的混合系统。对每个语言对,左数为到英语的相关,右数为从英语出发的相关。加粗规则与表 1 相同。

指标 en↔\leftrightarrow↔cs en↔\leftrightarrow↔de en↔\leftrightarrow↔et en↔\leftrightarrow↔fi en↔\leftrightarrow↔ru en↔\leftrightarrow↔tr en↔\leftrightarrow↔zh
BLEU .956/.993 .969/.977 .981/.971 .962/.958 .972/.977 .586/.796 .968/.941
ITER .966/.865 .990/.978 .975/.982 .989/.966 .943/.965 .742/.872 .978/.0–
RUSE .974/.0– .996/.0– .988/.0– .983/.0– .982/.0– .780/.0– .973/.0–
YiSi-1 .942/.985 .991/.983 .976/.976 .964/.938 .985/.989 .881/.942 .943/.957
PBERTP_{\text{BERT}}PBERT​ .965/.989 .995/.983 .990/.970 .976/.951 .976/.988 .846/.936 .975/.950
RBERTR_{\text{BERT}}RBERT​ .989/.995 .997/.991 .982/.979 .989/.977 .988/.989 .540/.872 .981/.980
FBERTF_{\text{BERT}}FBERT​ .978/.993 .998/.988 .989/.978 .983/.969 .985/.989 .760/.910 .981/.969
FBERTF_{\text{BERT}}FBERT​ (idf) .982/.995 .998/.988 .988/.979 .989/.969 .983/.987 .453/.877 .980/.963

Table 3: Model selection accuracies (Hits@1) on WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10−310^{-3}10−3. We bold the highest numbers for each language pair and direction.

Metric en↔\leftrightarrow↔cs en↔\leftrightarrow↔de en↔\leftrightarrow↔et en↔\leftrightarrow↔fi en↔\leftrightarrow↔ru en↔\leftrightarrow↔tr en↔\leftrightarrow↔zh
BLEU .134/.151 .803/.610 .756/.618 .461/.088 .228/.519 .095/.029 .658/.515
ITER .154/.000 .814/.692 .742/.733 .475/.111 .234/.532 .102/.030 .673/.0–
RUSE .214/.0– .823/.0– .785/.0– .487/.0– .248/.0– .109/.0– .670/.0–
YiSi-1 .159/.178 .809/.671 .749/.671 .467/.230 .248/.544 .108/.398 .613/.594
PBERTP_{\text{BERT}}PBERT​ .173/.180 .706/.663 .764/.771 .498/.078 .255/.545 .140/.372 .661/.551
RBERTR_{\text{BERT}}RBERT​ .163/.184 .804/.730 .770/.722 .494/.148 .260/.542 .005/.030 .677/.657
FBERTF_{\text{BERT}}FBERT​ .175/.184 .824/.703 .769/.763 .501/.082 .262/.544 .142/.031 .673/.629
FBERTF_{\text{BERT}}FBERT​ (idf) .179/.178 .824/.722 .760/.764 .503/.082 .265/.539 .004/.030 .678/.595

表 3:WMT18 混合系统上的模型选择准确率(Hits@1)。我们报告 10 万次抽样的平均值,0.95 置信区间低于 10−310^{-3}10−3。每个语言对和方向上的最高值加粗。

指标 en↔\leftrightarrow↔cs en↔\leftrightarrow↔de en↔\leftrightarrow↔et en↔\leftrightarrow↔fi en↔\leftrightarrow↔ru en↔\leftrightarrow↔tr en↔\leftrightarrow↔zh
BLEU .134/.151 .803/.610 .756/.618 .461/.088 .228/.519 .095/.029 .658/.515
ITER .154/.000 .814/.692 .742/.733 .475/.111 .234/.532 .102/.030 .673/.0–
RUSE .214/.0– .823/.0– .785/.0– .487/.0– .248/.0– .109/.0– .670/.0–
YiSi-1 .159/.178 .809/.671 .749/.671 .467/.230 .248/.544 .108/.398 .613/.594
PBERTP_{\text{BERT}}PBERT​ .173/.180 .706/.663 .764/.771 .498/.078 .255/.545 .140/.372 .661/.551
RBERTR_{\text{BERT}}RBERT​ .163/.184 .804/.730 .770/.722 .494/.148 .260/.542 .005/.030 .677/.657
FBERTF_{\text{BERT}}FBERT​ .175/.184 .824/.703 .769/.763 .501/.082 .262/.544 .142/.031 .673/.629
FBERTF_{\text{BERT}}FBERT​ (idf) .179/.178 .824/.722 .760/.764 .503/.082 .265/.539 .004/.030 .678/.595

Table 4: Kendall correlations with segment-level human judgments on WMT18. For each language pair, the left number is the to-English correlation, and the right is the from-English. We bold correlations of metrics not significantly outperformed by any other metric under bootstrap sampling for that language pair and direction. The numbers in parenthesis are the number of candidate-reference sentence pairs for each language pair and direction.

Metric en↔\leftrightarrow↔cs en↔\leftrightarrow↔de en↔\leftrightarrow↔et en↔\leftrightarrow↔fi en↔\leftrightarrow↔ru en↔\leftrightarrow↔tr en↔\leftrightarrow↔zh
(5k/5k) (78k/ 20k) (57k/32k) (16k/10k) (10k/22k) (9k/1k) (33k/29k)
BLEU .233/.389 .415/.620 .285/.414 .154/.355 .228/.330 .145/.261 .178/.311
ITER .198/.333 .396/.610 .235/.392 .128/.311 .139/.291 -.029/.236 .144/.0–
RUSE .347/.0– .498/.0– .368/.0– .273/.0– .311/.0– .259/.0– .218/.0–
YiSi-1 .319/.496 .488/.691 .351/.546 .231/.504 .300/.407 .234/.418 .211/.323
PBERTP_{\text{BERT}}PBERT​ .387/.541 .541/.715 .389/.549 .283/.486 .345/.414 .280/.328 .248/.337
RBERTR_{\text{BERT}}RBERT​ .388/.570 .546/.728 .391/.594 .304/.565 .343/.420 .290/.411 .255/.367
FBERTF_{\text{BERT}}FBERT​ .404/.562 .550/.728 .397/.586 .296/.546 .353/.423 .292/.399 .264/.364
FBERTF_{\text{BERT}}FBERT​ (idf) .408/.553 .550/.721 .395/585 .293/.537 .346/.425 .296/.406 .260/.366

表 4:与 WMT18 片段级人工判断的肯德尔相关。对每个语言对,左数为到英语的相关,右数为从英语出发的相关。在该语言对和方向上,按自助抽样未被任何其他指标显著超过的相关加粗。括号中的数字是每个语言对和方向的候选–参考句对数。

指标 en↔\leftrightarrow↔cs en↔\leftrightarrow↔de en↔\leftrightarrow↔et en↔\leftrightarrow↔fi en↔\leftrightarrow↔ru en↔\leftrightarrow↔tr en↔\leftrightarrow↔zh
(5k/5k) (78k/ 20k) (57k/32k) (16k/10k) (10k/22k) (9k/1k) (33k/29k)
BLEU .233/.389 .415/.620 .285/.414 .154/.355 .228/.330 .145/.261 .178/.311
ITER .198/.333 .396/.610 .235/.392 .128/.311 .139/.291 -.029/.236 .144/.0–
RUSE .347/.0– .498/.0– .368/.0– .273/.0– .311/.0– .259/.0– .218/.0–
YiSi-1 .319/.496 .488/.691 .351/.546 .231/.504 .300/.407 .234/.418 .211/.323
PBERTP_{\text{BERT}}PBERT​ .387/.541 .541/.715 .389/.549 .283/.486 .345/.414 .280/.328 .248/.337
RBERTR_{\text{BERT}}RBERT​ .388/.570 .546/.728 .391/.594 .304/.565 .343/.420 .290/.411 .255/.367
FBERTF_{\text{BERT}}FBERT​ .404/.562 .550/.728 .397/.586 .296/.546 .353/.423 .292/.399 .264/.364
FBERTF_{\text{BERT}}FBERT​ (idf) .408/.553 .550/.721 .395/585 .293/.537 .346/.425 .296/.406 .260/.366

5 Results

5 结果

Machine Translation

机器翻译

Tables 1–3 show system-level correlation to human judgements, correlations on hybrid systems, and model selection performance. We observe that BERTScore is consistently a top performer. In to-English results, RUSE (Shimanaka et al., 2018) shows competitive performance. However, RUSE is a supervised method trained on WMT16 and WMT15 human judgment data. In cases where RUSE models were not made available, such as for our from-English experiments, it is not possible to use RUSE without additional data and training. Table 4 shows segment-level correlations. We see that BERTScore exhibits significantly higher performance compared to the other metrics. The large improvement over Bleu stands out, making BERTScore particularly suitable to analyze specific examples, where SentBleu is less reliable. In Appendix A, we provide qualitative examples to illustrate the segment-level performance difference between SentBleu and BERTScore. At the segment-level, BERTScore even significantly outperforms RUSE. Overall, we find that applying importance weighting using idf{\rm idf}idf at times provides small benefit, but in other cases does not help. Understanding better when such importance weighting is likely to help is an important direction for future work, and likely depends on the domain of the text and the available test data. We continue without idf{\rm idf}idf weighting for the rest of our experiments. While recall R\textscBERTR_{\textsc{BERT}}R\textscBERT​, precision P\textscBERTP_{\textsc{BERT}}P\textscBERT​, and F1 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ alternate as the best measure in different setting, F1 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ performs reliably well across all the different settings. Our overall recommendation is therefore to use F1. We present additional results using the full set of 351 systems and evaluation metrics in Tables 12–28 in the appendix, including for experiments with idf{\rm idf}idf importance weighting, different contextual embedding models, and model selection.

表 1–3 分别给出与人工判断的系统级相关、混合系统上的相关,以及模型选择表现。可以看到,BERTScore 始终名列前茅。在到英语的结果里,RUSE(Shimanaka et al., 2018)也有竞争力。不过 RUSE 是有监督方法,训练用的是 WMT16 和 WMT15 的人工判断数据。在未能提供 RUSE 模型的情形下,例如我们的从英语出发的实验,若不补充数据和训练就无法使用。表 4 给出片段级相关。BERTScore 显著高于其他指标。相对 BLEU 的提升尤其明显,因此它特别适合用来分析具体例子,而 SentBLEU 在这种场合不那么可靠。附录 A 给出了定性例子,用来说明 SentBLEU 与 BERTScore 在片段级上的表现差异。在片段级上,BERTScore 甚至显著超过了 RUSE。总体来看,用 idf{\rm idf}idf 做重要性加权有时会带来一点好处,有时则没有帮助。更好地理解这种加权何时有用,是值得继续研究的方向,很可能取决于文本领域和手头的测试数据。后文实验不再使用 idf{\rm idf}idf 加权。虽然召回率 R\textscBERTR_{\textsc{BERT}}R\textscBERT​、准确率 P\textscBERTP_{\textsc{BERT}}P\textscBERT​ 和 F1 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 会在不同设置下轮流成为最好的度量,但 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 在各种设置下都稳定地表现良好。因此我们总体上建议使用 F1。附录的表 12–28 给出了全部 351 套系统和评测指标上的补充结果,涵盖 idf{\rm idf}idf 重要性加权、不同上下文嵌入模型,以及模型选择实验。

Image Captioning

图像描述

Table 5 shows correlation results for the COCO Captioning Challenge. BERTScore outperforms all task-agnostic baselines by large margins. Image captioning presents a challenging evaluation scenario, and metrics based on strict nnn-gram matching, including Bleu and Rouge, show weak correlations with human judgments. idf{\rm idf}idf importance weighting shows significant benefit for this task, suggesting people attribute higher importance to content words. Finally, Leic (Cui et al., 2018), a trained metric that takes images as additional inputs and is optimized specifically for the COCO data and this set of systems, outperforms all other methods.

表 5 给出 COCO Captioning Challenge 上的相关结果。BERTScore 大幅度超过所有不限定任务的基线。图像描述是一个颇有难度的评测场景,依赖严格 nnn-gram 匹配的指标,包括 BLEU 和 ROUGE,与人工判断的相关性很弱。idf{\rm idf}idf 重要性加权在这项任务上有明显收益,说明人们更看重实词。最后,LEIC(Cui et al., 2018)作为一种把图像也作为额外输入、并专门针对 COCO 数据和这批系统做了优化的训练型指标,超过了所有其他方法。

Table 5: Pearson correlation on the 2015 COCO Captioning Challenge. The M1 and M2 measures are described in Section 4. Leic uses images as additional inputs. Numbers with ∗ are cited from Cui et al. (2018). We bold the highest correlations of task-specific and task-agnostic metrics.

Metric M1 M2
BLEU -0.019∗ -0.005∗
METEOR 0.606∗ 0.594∗
ROUGE-L 0.090∗ 0.096∗
CIDEr 0.438∗ 0.440∗
SPICE 0.759∗ 0.750∗
LEIC 0.939∗ 0.949∗
BEER 0.491 0.562
EED 0.545 0.599
chrF++ 0.702 0.729
CharacTER 0.800 0.801
P\textscBERTP_{\textsc{BERT}}P\textscBERT​ -0.105 -0.041
R\textscBERTR_{\textsc{BERT}}R\textscBERT​ 0.888 0.863
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 0.322 0.350
R\textscBERTR_{\textsc{BERT}}R\textscBERT​ (idf) 0.917 0.889

表 5:2015 年 COCO Captioning Challenge 上的皮尔逊相关。M1 和 M2 两项度量在第 4 节中说明。LEIC 把图像作为额外输入。带 ∗ 的数字引自 Cui et al.(2018)。任务专用与不限定任务的指标中,最高相关分别加粗。

指标 M1 M2
BLEU -0.019∗ -0.005∗
METEOR 0.606∗ 0.594∗
ROUGE-L 0.090∗ 0.096∗
CIDEr 0.438∗ 0.440∗
SPICE 0.759∗ 0.750∗
LEIC 0.939∗ 0.949∗
BEER 0.491 0.562
EED 0.545 0.599
chrF++ 0.702 0.729
CharacTER 0.800 0.801
P\textscBERTP_{\textsc{BERT}}P\textscBERT​ -0.105 -0.041
R\textscBERTR_{\textsc{BERT}}R\textscBERT​ 0.888 0.863
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 0.322 0.350
R\textscBERTR_{\textsc{BERT}}R\textscBERT​ (idf) 0.917 0.889

Speed

速度

Despite the use of a large pre-trained model, computing BERTScore is relatively fast. We are able to process 192.5 candidate-reference pairs/second using a GTX-1080Ti GPU. The complete WMT18 en-de test set, which includes 2,998 sentences, takes 15.6sec to process, compared to 5.4sec with SacreBLEU (Post, 2018), a common Bleu implementation. Given the sizes of commonly used test and validation sets, the increase in processing time is relatively marginal, and BERTScore is a good fit for using during validation (e.g., for stopping) and testing, especially when compared to the time costs of other development stages.

尽管用了大型预训练模型,计算 BERTScore 仍然相对较快。在一块 GTX-1080Ti GPU 上,我们每秒可以处理 192.5 对候选–参考。完整的 WMT18 英到德测试集共 2,998 句,耗时 15.6 秒;相比之下,常用的 BLEU 实现 SacreBLEU(Post, 2018)只要 5.4 秒。就常用测试集和验证集的规模而言,多出来的处理时间相对有限,BERTScore 适合在验证(例如用于早停)和测试时使用,尤其是和其他开发环节的时间成本相比。

6 Robustness Analysis

6 稳健性分析

We test the robustness of BERTScore using adversarial paraphrase classification. We use the Quora Question Pair corpus (Iyer et al., 2017, QQP;) and the adversarial paraphrases from the Paraphrase Adversaries from Word Scrambling dataset (Zhang et al., 2019, PAWS;). Both datasets contain pairs of sentences labeled to indicate whether they are paraphrases or not. Positive examples in QQP are real duplicate questions, while negative examples are related, but different questions. Sentence pairs in PAWS are generated through word swapping. For example, in PAWS, Flights from New York to Florida may be changed to Flights from Florida to New York and a good classifier should identify that these two sentences are not paraphrases. PAWS includes two parts: PAWSQQP, which is based on the QQP data, and PAWSWiki. We use the PAWSQQP development set which contains 667 sentences. For the automatic metrics, we use no paraphrase detection training data. We expect that pairs with higher scores are more likely to be paraphrases. To evaluate the automatic metrics on QQA, we use the first 5,000 sentences in the training set instead of the the test set because the test labels are not available. We treat the first sentence as the reference and the second sentence as the candidate.

我们用对抗性释义分类来测试 BERTScore 的稳健性。数据来自 Quora Question Pair 语料(Iyer et al., 2017,简称 QQP)以及 Paraphrase Adversaries from Word Scrambling 数据集中的对抗释义(Zhang et al., 2019,简称 PAWS)。两个数据集都包含成对句子,并标注它们是否构成释义。QQP 中的正例是真实的重复问题,负例则是相关但不相同的问题。PAWS 中的句子对通过词交换生成。例如,在 PAWS 里,Flights from New York to Florida 可能被改成 Flights from Florida to New York,好的分类器应当识别出这两句并不是释义。PAWS 包含两部分:基于 QQP 数据的 PAWSQQP_{\mathrm{QQP}}QQP​,以及 PAWSWiki。我们使用含 667 句的 PAWSQQP_{\mathrm{QQP}}QQP​ 开发集。对自动指标,我们不使用任何释义检测训练数据。我们期望分数更高的句子对更有可能是释义。在 QQP 上评测自动指标时,因为测试集标签不可用,我们改用训练集的前 5,000 句,而不是测试集。把第一句当作参考,第二句当作候选。

Table 6: Area under ROC curve (AUC) on QQP and PAWSQQP_{\mathrm{QQP}}QQP​ datasets. The scores of trained DecAtt (Parikh et al., 2016), DIIN (Gong et al., 2018), and fine-tuned BERT are reported by Zhang et al. (2019). Numbers with ∗ are scores on the held-out test set of QQP. We bold the highest correlations of task-specific and task-agnostic metrics.

Type Method QQP PAWSQQP_{\mathrm{QQP}}QQP​
Trained on QQP (supervised) DecAtt 0.939∗ 0.263
DIIN 0.952∗ 0.324
BERT 0.963∗ 0.351
Trained on QQP + PAWSQQP_{\mathrm{QQP}}QQP​ (supervised) DecAtt – 0.511
DIIN – 0.778
BERT – 0.831
Metric (Not trained on QQP or PAWSQQP_{\mathrm{QQP}}QQP​) BLEU 0.707 0.527
METEOR 0.755 0.532
ROUGE-L 0.740 0.536
chrF++ 0.577 0.608
BEER 0.741 0.564
EED 0.743 0.611
CharacTER 0.698 0.650
P\textscBERTP_{\textsc{BERT}}P\textscBERT​ 0.757 0.687
R\textscBERTR_{\textsc{BERT}}R\textscBERT​ 0.744 0.685
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 0.761 0.685
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ (idf) 0.777 0.693

表 6:QQP 和 PAWSQQP_{\mathrm{QQP}}QQP​ 数据集上的 ROC 曲线下面积(AUC)。已训练的 DecAtt(Parikh et al., 2016)、DIIN(Gong et al., 2018)以及微调 BERT 的分数由 Zhang et al.(2019)报告。带 ∗ 的数字是 QQP 留出测试集上的分数。任务专用与不限定任务的指标中,最高相关分别加粗。

类型 方法 QQP PAWSQQP_{\mathrm{QQP}}QQP​
在 QQP 上训练(有监督) DecAtt 0.939∗ 0.263
DIIN 0.952∗ 0.324
BERT 0.963∗ 0.351
在 QQP + PAWSQQP_{\mathrm{QQP}}QQP​ 上训练(有监督) DecAtt – 0.511
DIIN – 0.778
BERT – 0.831
指标(未在 QQP 或 PAWSQQP_{\mathrm{QQP}}QQP​ 上训练) BLEU 0.707 0.527
METEOR 0.755 0.532
ROUGE-L 0.740 0.536
chrF++ 0.577 0.608
BEER 0.741 0.564
EED 0.743 0.611
CharacTER 0.698 0.650
P\textscBERTP_{\textsc{BERT}}P\textscBERT​ 0.757 0.687
R\textscBERTR_{\textsc{BERT}}R\textscBERT​ 0.744 0.685
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 0.761 0.685
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ (idf) 0.777 0.693

Table 6 reports the area under ROC curve (AUC) for existing models and automatic metrics. We observe that supervised classifiers trained on QQP perform worse than random guess on PAWSQQP, which shows these models predict the adversarial examples are more likely to be paraphrases. When adversarial examples are provided in training, state-of-the-art models like DIIN (Gong et al., 2018) and fine-tuned BERT are able to identify the adversarial examples but their performance still decreases significantly from their performance on QQP. Most metrics have decent performance on QQP, but show a significant performance drop on PAWSQQP, almost down to chance performance. This suggests these metrics fail to to distinguish the harder adversarial examples. In contrast, the performance of BERTScore drops only slightly, showing more robustness than the other metrics.

表 6 报告了已有模型和自动指标的 ROC 曲线下面积(AUC)。我们看到,只在 QQP 上训练的有监督分类器,在 PAWSQQP_{\mathrm{QQP}}QQP​ 上的表现差于随机猜测,说明这些模型更倾向于把对抗样本判成释义。若训练中提供对抗样本,DIIN(Gong et al., 2018)和微调后的 BERT 这类当时最好的模型能够识别对抗样本,但表现仍比它们在 QQP 上明显下降。多数指标在 QQP 上还不错,但在 PAWSQQP_{\mathrm{QQP}}QQP​ 上会大幅下滑,几乎降到随机水平。这表明这些指标区分不了更难的对抗样本。相比之下,BERTScore 的表现只小幅下降,比其他指标更稳健。

7 Discussion

7 讨论

We propose BERTScore, a new metric for evaluating generated text against gold standard references. BERTScore is purposely designed to be simple, task agnostic, and easy to use. Our analysis illustrates how BERTScore resolves some of the limitations of commonly used metrics, especially on challenging adversarial examples. We conduct extensive experiments with various configuration choices for BERTScore, including the contextual embedding model used and the use of importance weighting. Overall, our extensive experiments, including the ones in the appendix, show that BERTScore achieves better correlation than common metrics, and is effective for model selection. However, there is no one configuration of BERTScore that clearly outperforms all others. While the differences between the top configurations are often small, it is important for the user to be aware of the different trade-offs, and consider the domain and languages when selecting the exact configuration to use. In general, for machine translation evaluation, we suggest using F\textscBERTF_{\textsc{BERT}}F\textscBERT​, which we find the most reliable. For evaluating text generation in English, we recommend using the 24-layer RoBERTalarge\text{RoBERTa}_{{\rm large}}RoBERTalarge​ model to compute BERTScore. For non-English language, the multilingual \textscBERTmulti\textsc{BERT}_{{\rm multi}}\textscBERTmulti​ is a suitable choice although BERTScore computed with this model has less stable performance on low-resource languages. We report the optimal hyperparameter for all models we experimented with in Appendix B

我们提出 BERTScore,用来把生成文本对照黄金标准参考句进行评测。它被有意设计得简单、不绑定特定任务,并且容易使用。我们的分析说明了 BERTScore 如何弥补常用指标的一些局限,尤其是在困难的对抗样本上。我们就 BERTScore 的多种配置做了广泛实验,包括所用的上下文嵌入模型,以及是否使用重要性加权。总体来看,包括附录在内的大量实验表明,BERTScore 的相关性优于常用指标,也适合做模型选择。不过,并没有一种配置明显压过所有其他配置。领先配置之间的差距往往不大,使用者仍有必要了解不同取舍,并在选定具体配置时考虑领域和语言。一般而言,对机器翻译评测,我们建议使用 F\textscBERTF_{\textsc{BERT}}F\textscBERT​,因为它最为稳妥。评测英语文本生成时,我们建议用 24 层的 RoBERTalarge\text{RoBERTa}_{{\rm large}}RoBERTalarge​ 来计算 BERTScore。对非英语语言,多语 \textscBERTmulti\textsc{BERT}_{{\rm multi}}\textscBERTmulti​ 是合适的选择,不过用该模型算出的 BERTScore 在低资源语言上的表现不够稳定。我们在附录 B 中报告了所试验全部模型的最优超参数。

Briefly following our initial preprint publication, Zhao et al. (2019) published a concurrently developed method related to ours, but with a focus on integrating contextual word embeddings with earth mover’s distance (Rubner et al., 1998, EMD;) rather than our simple matching process. They also propose various improvements compared to our use of contextualized embeddings. We study these improvements in Appendix C and show that integrating them into BERTScore makes it equivalent or better than the EMD-based approach. Largely though, the effect of the different improvements on BERTScore is more modest compared to their method. Shortly after our initial publication, YiSi-1 was updated to use BERT embeddings, showing improved performance (Lo, 2019). This further corroborates our findings. Other recent related work includes training a model on top of BERT to maximize the correlation with human judgments (Mathur et al., 2019) and evaluating generation with a BERT model fine-tuned on paraphrasing (Yoshimura et al., 2019). More recent work shows the potential of using BERTScore for training a summarization system (Li et al., 2019) and for domain-specific evaluation using SciBERT (Beltagy et al., 2019) to evaluate abstractive text summarization (Gabriel et al., 2019).

在我们最初的预印本发表后不久,Zhao et al.(2019)发表了一项并行发展、与我们相关的方法,重点是把上下文词嵌入与土堆移动距离(Rubner et al., 1998,简称 EMD)结合起来,而不是采用我们这种简单的匹配过程。他们还针对上下文嵌入的用法提出了若干改进。我们在附录 C 中研究了这些改进,并表明把它们并入 BERTScore 后,可以达到与基于 EMD 的方法相当或更好的结果。不过,这些改进对 BERTScore 的作用,总体要比对对方方法更温和。在我们最初发表后不久,YiSi-1 也更新为使用 BERT 嵌入,并显示出更好的表现(Lo, 2019)。这进一步印证了我们的发现。其他近期相关工作还包括:在 BERT 之上训练模型以最大化与人工判断的相关性(Mathur et al., 2019),以及用在释义任务上微调过的 BERT 来评测生成(Yoshimura et al., 2019)。更新的工作还展示了把 BERTScore 用于训练摘要系统的潜力(Li et al., 2019),以及用 SciBERT(Beltagy et al., 2019)做领域专用评测,以评估抽象摘要(Gabriel et al., 2019)。

In future work, we look forward to designing new task-specific metrics that use BERTScore as a subroutine and accommodate task-specific needs, similar to how Wieting et al. (2019) suggests to use semantic similarity for machine translation training. Because BERTScore is fully differentiable, it also can be incorporated into a training procedure to compute a learning loss that reduces the mismatch between optimization and evaluation objectives.

在未来工作中,我们希望设计以 BERTScore 为子程序、并能照顾任务特定需求的新指标,这与 Wieting et al.(2019)建议在机器翻译训练中使用语义相似度的思路类似。因为 BERTScore 完全可微,它也可以并入训练过程,用来计算一种缩小优化目标与评测目标差距的学习损失。

Acknowledgement

致谢

This research is supported in part by grants from the National Science Foundation (III-1618134, III-1526012, IIS1149882, IIS-1724282, TRIPODS-1740822, CAREER-1750499), the Office of Naval Research DOD (N00014-17-1-2175), and the Bill and Melinda Gates Foundation, SAP, Zillow, Workday, and Facebook Research. We thank Graham Neubig and David Grangier for for their insightful comments. We thank the Cornell NLP community including but not limited to Claire Cardie, Tianze Shi, Alexandra Schofield, Gregory Yauney, and Rishi Bommasani. We thank Yin Cui and Guandao Yang for their help with the COCO 2015 dataset.

本研究部分得到美国国家科学基金会(III-1618134、III-1526012、IIS1149882、IIS-1724282、TRIPODS-1740822、CAREER-1750499)、美国海军研究办公室 DOD(N00014-17-1-2175),以及比尔及梅琳达·盖茨基金会、SAP、Zillow、Workday 和 Facebook Research 的资助。我们感谢 Graham Neubig 和 David Grangier 提出的有见地的意见。我们也感谢康奈尔自然语言处理社区,包括但不限于 Claire Cardie、Tianze Shi、Alexandra Schofield、Gregory Yauney 和 Rishi Bommasani。感谢 Yin Cui 和 Guandao Yang 在 COCO 2015 数据集上给予的帮助。

References

参考文献

Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic propositional image caption evaluation. In ECCV , 2016.

Athiwaratkun et al. (2018) Ben Athiwaratkun, Andrew Wilson, and Anima Anandkumar. Probabilistic fasttext for multi-sense word embeddings. In ACL , 2018.

Banerjee & Lavie (2005) Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In IEEvaluation@ACL , 2005.

Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. ArXiv , 2019.

Bojar et al. (2016) Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. Results of the WMT16 metrics shared task. In WMT , 2016.

Bojar et al. (2017) Ondřej Bojar, Yvette Graham, and Amir Kamran. Results of the WMT17 metrics shared task. In WMT , 2017.

Chaganty et al. (2018) Arun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in natural language evalaution. In ACL , 2018.

Chow et al. (2019) Julian Chow, Lucia Specia, and Pranava Madhyastha. WMDO: Fluency-based word mover’s distance for machine translation evaluation. In WMT , 2019.

Clark et al. (2019) Elizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. Sentence mover’s similarity: Automatic evaluation for multi-sentence texts. In ACL , 2019.

Corley & Mihalcea (2005) Courtney Corley and Rada Mihalcea. Measuring the semantic similarity of texts. In ACL Workshop , EMSEE ’05, 2005.

Cui et al. (2018) Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge J. Belongie. Learning to evaluate image captioning. In CVPR , 2018.

Denkowski & Lavie (2014) Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In WMT@ACL , 2014.

Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , 2019.

Doddington (2002) George Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In HLT , 2002.

Dolan & Brockett (2005) William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In IWP , 2005.

Gabriel et al. (2019) Saadia Gabriel, Antoine Bosselut, Ari Holtzman, Kyle Lo, Asli Çelikyilmaz, and Yejin Choi. Cooperative generator-discriminator networks for abstractive summarization with narrative flow. ArXiv , 2019.

Galley et al. (2015) Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and William B. Dolan. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In ACL , 2015.

Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In ICML , 2017.

Gong et al. (2018) Yichen Gong, Heng Luo, and Jian Zhang. Natural language inference over interaction space. In ICLR , 2018.

Graham & Baldwin (2014) Yvette Graham and Timothy Baldwin. Testing for significance of increased correlation with human judgment. In EMNLP , 2014.

Graham & Liu (2016) Yvette Graham and Qun Liu. Achieving accurate conclusions in evaluation of automatic machine translation metrics. In NAACL , 2016.

Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. arXiv preprint arXiv:1802.06893 , 2018.

Guo & Hu (2019) Yinuo Guo and Junfeng Hu. Meteor++ 2.0: Adopt syntactic level paraphrase knowledge into machine translation evaluation. In WMT , 2019.

Hashimoto et al. (2019) Tatsu Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. In NAACL-HLT , 2019.

Huang et al. (2019) Chenyang Huang, Amine Trabelsi, and Osmar R Zaïane. ANA at semeval-2019 task 3: Contextual emotion detection in conversations through hierarchical LSTMs and BERT. arXiv preprint arXiv:1904.00132 , 2019.

Isozaki et al. (2010) Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. Automatic evaluation of translation quality for distant language pairs. In EMNLP , 2010.

Iyer et al. (2017) Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs. https://tinyurl.com/y2y8u5ed , 2017.

Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In ACL , 2007.

Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document distances. In ICML , 2015.

Lample & Conneau (2019) Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv , 2019.

Leusch et al. (2006) Gregor Leusch, Nicola Ueffing, and Hermann Ney. CDER: Efficient MT evaluation using block movements. In EACL , 2006.

Levenshtein (1966) Vladimir Iosifovich Levenshtein. Binary Codes Capable of Correcting Deletions, Insertions and Rever sals. Soviet Physics Doklady , 10, 1966.

Li et al. (2019) Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. Deep reinforcement learning with distributional semantic rewards for abstractive summarization. In EMNLP-IJCNLP , 2019.

Lin (2004) Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In ACL , 2004.

Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014.

Liu et al. (2019a) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855 , 2019a.

Liu (2019) Yang Liu. Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318 , 2019.

Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv , abs/1907.11692, 2019b.

Lo (2017) Chi-kiu Lo. MEANT 2.0: Accurate semantic mt evaluation for any output language. In WMT , 2017.

Lo (2019) Chi-kiu Lo. YiSi - a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In WMT , 2019.

Lo et al. (2018) Chi-kiu Lo, Michel Simard, Darlene Stewart, Samuel Larkin, Cyril Goutte, and Patrick Littell. Accurate semantic textual similarity for cleaning noisy parallel corpora using semantic machine translation evaluation metric: The NRC supervised submissions to the parallel corpus filtering task. In WMT , 2018.

Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. Towards an automatic Turing test: Learning to evaluate dialogue responses. In ACL , 2017.

Ma et al. (2017) Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. Blend: a novel combined mt metric based on direct assessment – casict-dcu submission to WMT17 metrics task. In WMT , 2017.

Ma et al. (2018) Qingsong Ma, Ondrej Bojar, and Yvette Graham. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In WMT , 2018.

Mathur et al. (2019) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In ACL , 2019.

Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NIPS , 2013.

Nguyen et al. (2017) Dai Quoc Nguyen, Dat Quoc Nguyen, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. A mixture model for learning multi-sense word embeddings. In ACL , 2017.

Ott et al. (2018) Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In WMT , 2018.

Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038 , 2019.

Panja & Naskar (2018) Joybrata Panja and Sudip Kumar Naskar. Iter: Improving translation edit rate through optimizable edit costs. In WMT , 2018.

Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL , 2002.

Parikh et al. (2016) Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In EMNLP , 2016.

Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP , 2014.

Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. Deep contextualized word representations. In NAACL-HLT , 2018.

Popović (2015) Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. In WMT@ACL , 2015.

Popović (2017) Maja Popović. chrf++: words helping character n-grams. In WMT , 2017.

Post (2018) Matt Post. A call for clarity in reporting BLEU scores. In WMT , 2018.

Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych. Alternative weighting schemes for elmo embeddings. arXiv preprint arXiv:1904.02954 , 2019.

Rubner et al. (1998) Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. A metric for distributions with applications to image databases. In ICCV . IEEE, 1998.

Rus & Lintean (2012) Vasile Rus and Mihai Lintean. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP . ACL, 2012.

Rücklé et al. (2018) Andreas Rücklé, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. Concatenated power mean word embeddings as universal cross-lingual sentence representations. arXiv , 2018.

Sharma et al. (2018) Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. arXiv preprint arXiv:1706.09799 , 2018.

Shimanaka et al. (2018) Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation. In WMT , 2018.

Snover et al. (2006) Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In AMTA , 2006.

Stanchev et al. (2019) Peter Stanchev, Weiyue Wang, and Hermann Ney. EED: Extended edit distance measure for machine translation. In WMT , 2019.

Stanojević & Sima’an (2014) Miloš Stanojević and Khalil Sima’an. Beer: Better evaluation as ranking. In WMT , 2014.

Tillmann et al. (1997) Christoph Tillmann, Stephan Vogel, Hermann Ney, Arkaitz Zubiaga, and Hassan Sawaf. Accelerated dp based search for statistical translation. In EUROSPEECH , 1997.

Toutanova et al. (2016) Kristina Toutanova, Chris Brockett, Ke M Tran, and Saleema Amershi. A dataset and evaluation metrics for abstractive compression of sentences and short paragraphs. In EMNLP , 2016.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , 2017.

Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image description evaluation. In CVPR , 2015.

Wang et al. (2016) Weiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. Character: Translation edit rate on character level. In WMT , 2016.

Wieting et al. (2019) John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. Beyond BLEU:training neural machine translation with semantic similarity. In ACL , 2019.

Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In ACL , 2018.

Williams (1959) Evan James Williams. Regression analysis . wiley, 1959.

Wu et al. (2019) Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR , 2019.

Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016.

Yang et al. (2019a) Wei Yang, Haotian Zhang, and Jimmy Lin. Simple applications of BERT for ad hoc document retrieval. arXiv preprint arXiv:1903.10972 , 2019a.

Yang et al. (2019b) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv , 2019b.

Yoshimura et al. (2019) Ryoma Yoshimura, Hiroki Shimanaka, Yukio Matsumura, Hayahide Yamagishi, and Mamoru Komachi. Filtering pseudo-references by paraphrasing for automatic evaluation of machine translation. In WMT , 2019.

Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130 , 2019.

Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In EMNLP , 2019.

Appendix A Qualitative Analysis

附录 A 定性分析

Table 7: Examples sentences where similarity ranks assigned by Human, F\textscBERTF_{\textsc{BERT}}F\textscBERT​, and Bleu differ significantly on WMT16 German-to-English evaluation task. xxx: gold reference, x^\hat{x}x^: candidate outputs of MT systems. Rankings assigned by Human, F\textscBERTF_{\textsc{BERT}}F\textscBERT​, and Bleu are shown in the right three columns. The sentences are ranked by the similarity, i.e. rank 1 is the most similar pair assigned by a score. An ideal metric should rank similar to humans.

Case No. Reference and Candidate Pairs Human F\textscBERTF_{\textsc{BERT}}F\textscBERT​ Bleu
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ >>> Bleu 1. xxx: At the same time Kingfisher is closing 60 B&Q outlets across the country 38 125 530
x^\hat{x}x^: At the same time, Kingfisher will close 60 B & Q stores nationwide
2. xxx: Hewlett-Packard to cut up to 30,000 jobs 119 39 441
x^\hat{x}x^: Hewlett-Packard will reduce jobs up to 30.000
3. xxx: According to opinion in Hungary, Serbia is “a safe third country". 23 96 465
x^\hat{x}x^: According to Hungarian view, Serbia is a “safe third country."
4. xxx: Experts believe November’s Black Friday could be holding back spending. 73 147 492
x^\hat{x}x^: Experts believe that the Black Friday in November has put the brakes on spending
5. xxx: And it’s from this perspective that I will watch him die. 37 111 414
x^\hat{x}x^: And from this perspective, I will see him die.
Bleu >>> F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 6. xxx: In their view the human dignity of the man had been violated. 500 470 115
x^\hat{x}x^: Look at the human dignity of the man injured.
8. xxx: For example when he steered a shot from Ideye over the crossbar in the 56th minute. 516 524 185
x^\hat{x}x^: So, for example, when he steered a shot of Ideye over the latte (56th).
7. xxx: A good prank is funny, but takes moments to reverse. 495 424 152
x^\hat{x}x^: A good prank is funny, but it takes only moments before he becomes a boomerang.
9. xxx: I will put the pressure on them and onus on them to make a decision. 507 471 220
x^\hat{x}x^: I will exert the pressure on it and her urge to make a decision.
10. xxx: Transport for London is not amused by this flyposting "vandalism." 527 527 246
x^\hat{x}x^: Transport for London is the Plaka animal "vandalism" is not funny.
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ >>> Human 11. xxx: One big obstacle to access to the jobs market is the lack of knowledge of the German language. 558 131 313
x^\hat{x}x^: A major hurdle for access to the labour market are a lack of knowledge of English.
12. xxx: On Monday night Hungary closed its 175 km long border with Serbia. 413 135 55
x^\hat{x}x^: Hungary had in the night of Tuesday closed its 175 km long border with Serbia.
13. xxx: They got nothing, but they were allowed to keep the clothes. 428 174 318
x^\hat{x}x^: You got nothing, but could keep the clothes.
14. xxx: A majority of Republicans don’t see Trump’s temperament as a problem. 290 34 134
x^\hat{x}x^: A majority of Republicans see Trump’s temperament is not a problem.
15. xxx:His car was still running in the driveway. 299 49 71
x^\hat{x}x^: His car was still in the driveway.
Human >>> F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 16. xxx: Currently the majority of staff are men. 77 525 553
x^\hat{x}x^: At the moment the men predominate among the staff.
17. xxx: There are, indeed, multiple variables at play. 30 446 552
x^\hat{x}x^: In fact, several variables play a role.
18. xxx: One was a man of about 5ft 11in tall. 124 551 528
x^\hat{x}x^: One of the men was about 1,80 metres in size.
19. xxx: All that stuff sure does take a toll. 90 454 547
x^\hat{x}x^: All of this certainly exacts its toll.
20. xxx: Wage gains have shown signs of picking up. 140 464 514
x^\hat{x}x^: Increases of wages showed signs of a recovery.

表 7:在 WMT16 德语到英语评测任务上,人工、F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 和 BLEU 给出的相似度排序有明显差异的例句。xxx:黄金参考句,x^\hat{x}x^:机器翻译系统的候选输出。人工、F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 和 BLEU 给出的排序列在右侧三栏。句子按相似度排序,即秩为 1 表示该分数认为最相似的一对。理想的指标给出的排序应接近人工排序。

情形 编号 参考句与候选句对 人工 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ BLEU
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ >>> Bleu 1. xxx: At the same time Kingfisher is closing 60 B&Q outlets across the country 38 125 530
x^\hat{x}x^: At the same time, Kingfisher will close 60 B & Q stores nationwide
2. xxx: Hewlett-Packard to cut up to 30,000 jobs 119 39 441
x^\hat{x}x^: Hewlett-Packard will reduce jobs up to 30.000
3. xxx: According to opinion in Hungary, Serbia is “a safe third country". 23 96 465
x^\hat{x}x^: According to Hungarian view, Serbia is a “safe third country."
4. xxx: Experts believe November’s Black Friday could be holding back spending. 73 147 492
x^\hat{x}x^: Experts believe that the Black Friday in November has put the brakes on spending
5. xxx: And it’s from this perspective that I will watch him die. 37 111 414
x^\hat{x}x^: And from this perspective, I will see him die.
Bleu >>> F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 6. xxx: In their view the human dignity of the man had been violated. 500 470 115
x^\hat{x}x^: Look at the human dignity of the man injured.
8. xxx: For example when he steered a shot from Ideye over the crossbar in the 56th minute. 516 524 185
x^\hat{x}x^: So, for example, when he steered a shot of Ideye over the latte (56th).
7. xxx: A good prank is funny, but takes moments to reverse. 495 424 152
x^\hat{x}x^: A good prank is funny, but it takes only moments before he becomes a boomerang.
9. xxx: I will put the pressure on them and onus on them to make a decision. 507 471 220
x^\hat{x}x^: I will exert the pressure on it and her urge to make a decision.
10. xxx: Transport for London is not amused by this flyposting "vandalism." 527 527 246
x^\hat{x}x^: Transport for London is the Plaka animal "vandalism" is not funny.
F\textscBERTF_{\textsc{BERT}}F\textscBERT​ >>> Human 11. xxx: One big obstacle to access to the jobs market is the lack of knowledge of the German language. 558 131 313
x^\hat{x}x^: A major hurdle for access to the labour market are a lack of knowledge of English.
12. xxx: On Monday night Hungary closed its 175 km long border with Serbia. 413 135 55
x^\hat{x}x^: Hungary had in the night of Tuesday closed its 175 km long border with Serbia.
13. xxx: They got nothing, but they were allowed to keep the clothes. 428 174 318
x^\hat{x}x^: You got nothing, but could keep the clothes.
14. xxx: A majority of Republicans don’t see Trump’s temperament as a problem. 290 34 134
x^\hat{x}x^: A majority of Republicans see Trump’s temperament is not a problem.
15. xxx:His car was still running in the driveway. 299 49 71
x^\hat{x}x^: His car was still in the driveway.
Human >>> F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 16. xxx: Currently the majority of staff are men. 77 525 553
x^\hat{x}x^: At the moment the men predominate among the staff.
17. xxx: There are, indeed, multiple variables at play. 30 446 552
x^\hat{x}x^: In fact, several variables play a role.
18. xxx: One was a man of about 5ft 11in tall. 124 551 528
x^\hat{x}x^: One of the men was about 1,80 metres in size.
19. xxx: All that stuff sure does take a toll. 90 454 547
x^\hat{x}x^: All of this certainly exacts its toll.
20. xxx: Wage gains have shown signs of picking up. 140 464 514
x^\hat{x}x^: Increases of wages showed signs of a recovery.

We study BERTScore and SentBleu using WMT16 German-to-English (Bojar et al., 2016). We rank all 560 candidate-reference pairs by human score, BERTScore, or SentBleu from most similar to least similar. Ideally, the ranking assigned by BERTScore and SentBleu should be similar to the ranking assigned by the human score.

我们用 WMT16 德语到英语数据(Bojar et al., 2016)来考察 BERTScore 和 SentBLEU。把全部 560 对候选–参考按人工分数、BERTScore 或 SentBLEU,从最相似排到最不相似。理想情况下,BERTScore 和 SentBLEU 给出的排序应当接近人工分数给出的排序。

Table 7 first shows examples where BERTScore and SentBleu scores disagree about the ranking for a candidate-reference pair by a large number. We observe that BERTScore is effectively able to capture synonyms and changes in word order. For example, the reference and candidate sentences in pair 3 are almost identical except that the candidate replaces opinion in Hungary with Hungarian view and switches the order of the quotation mark (“) and a. While BERTScore ranks the pair relatively high, SentBleu judges the pair as dissimilar, because it cannot match synonyms and is sensitive to the small word order changes. Pair 5 shows a set of changes that preserve the semantic meaning: replacing to cut with will reduce and swapping the order of 30,000 and jobs. BERTScore ranks the candidate translation similar to the human judgment, whereas SentBleu ranks it much lower. We also see that SentBleu potentially over-rewards nnn-gram overlap, even when phrases are used very differently. In pair 6, both the candidate and the reference contain the human dignity of the man. Yet the two sentences convey very different meaning. BERTScore agrees with the human judgment and ranks the pair low. In contrast, SentBleu considers the pair as relatively similar because of the significant word overlap.

表 7 首先给出了一些例子:在这些参考–候选对上,BERTScore 和 SentBLEU 的排序相差很大。可以看到,BERTScore 能够有效捕捉同义词和词序变化。例如,第 3 对里参考句和候选句几乎相同,只是候选把 opinion in Hungary 换成了 Hungarian view,并把引号(“)与 a 的顺序对调。BERTScore 把这对排得相对靠前,SentBLEU 却判它们不相似,因为它无法匹配同义词,也对细小的词序变化敏感。第 5 对展示了一组保持语义不变的改写:把 to cut 换成 will reduce,并交换 30,000 和 jobs 的顺序。BERTScore 给出的排序与人工判断接近,SentBLEU 则把它排得低得多。我们还看到,即便短语用法已经很不一样,SentBLEU 仍可能过分奖励 nnn-gram 重叠。在第 6 对中,候选和参考都含有 the human dignity of the man,但两句传达的意思很不同。BERTScore 与人工判断一致,把这对排得很低;SentBLEU 却因为大量词汇重叠,认为它们相对相似。

The bottom half of Table 7 shows examples where BERTScore and human judgments disagree about the ranking. We observe that BERTScore finds it difficult to detect factual errors. For example, BERTScore assigns high similarity to pair 11 when the translation replaces German language with English and pair 12 where the translation incorrectly outputs Tuesday when it is supposed to generate Monday. BERTScore also fails to identify that 5ft 11in is equivalent with 1.80 metres in pair 18. As a result, BERTScore assigns low similarity to the eighth pair in Table 7. SentBleu also suffers from these limitations.

表 7 的下半部分给出了 BERTScore 与人工判断排序不一致的例子。我们看到,BERTScore 难以发现事实错误。例如,第 11 对里译文把 German language 换成了 English,第 12 对本应生成 Monday 却错误输出 Tuesday,BERTScore 仍给出了很高的相似度。它也未能识别第 18 对中 5ft 11in 与 1.80 metres 等价,因此把表 7 中的第 8 对判得很不相似。SentBLEU 同样受这些局限影响。

Figure 2 visualizes the BERTScore matching of two pairs of candidate and reference sentences. The figure illustrates how F\textscBERTF_{\textsc{BERT}}F\textscBERT​ matches synonymous phrases, such as imported cars and foreign cars. We also see that F\textscBERTF_{\textsc{BERT}}F\textscBERT​ effectively matches words even given a high ordering distortion, for example the token people in the figure.

图 2 可视化了两对候选句与参考句的 BERTScore 匹配。图中可以看到 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 如何把 imported cars 和 foreign cars 这类同义短语对齐。我们还可以看到,即便词序被大幅打乱,F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 仍能有效匹配词语,例如图中的词元 people。

BERTScore visualization of word matching cosine similarities between candidate and reference sentences

Figure 2: BERTScore visualization. The cosine similarity of each word matching in P\textscBERTP_{\textsc{BERT}}P\textscBERT​ are color-coded.

图 2:BERTScore 可视化。P\textscBERTP_{\textsc{BERT}}P\textscBERT​ 中每次词匹配的余弦相似度用颜色编码。

Appendix B Representation Choice

附录 B 表示层的选择

As suggested by previous works (Peters et al., 2018; Reimers & Gurevych, 2019), selecting a good layer or a good combination of layers from the BERT model is important. In designing BERTScore, we use WMT16 segment-level human judgment data as a development set to facilitate our representation choice. For Chinese models, we tune with the WMT17 “en-zh” data because the language pair “en-zh” is not available in WMT16. In Figure 3, we plot the change of human correlation of F\textscBERTF_{\textsc{BERT}}F\textscBERT​ over different layers of BERT, RoBERTa, XLNet and XLM models. Based on results from different models, we identify a common trend that F\textscBERTF_{\textsc{BERT}}F\textscBERT​ computed with the intermediate representations tends to work better. We tune the number of layer to use for a range of publicly available models.[^8] Table 8 shows the results of our hyperparameter search.

正如已有工作所指出的(Peters et al., 2018;Reimers & Gurevych, 2019),从 BERT 模型中选好一层或一层的合适组合很重要。在设计 BERTScore 时,我们把 WMT16 片段级人工判断数据当作开发集,用来辅助选择表示。对中文模型,因为 WMT16 没有“en-zh”语言对,我们改用 WMT17 的“en-zh”数据来调参。图 3 画出了 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 与人工判断的相关性随 BERT、RoBERTa、XLNet 和 XLM 不同层的变化。综合不同模型的结果,可以看到一个共同趋势:用中间层表示算出的 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 往往更好。我们为一批公开模型分别调了所用层数。[^8] 表 8 给出了超参数搜索的结果。

[^8] https://huggingface.co/pytorch-transformers/pretrained_models.html

Table 8: Recommended layer of representation to use for BERTScore. The layers are chosen based on a held-out validation set (WMT16).

Model Total Number of Layers Best Layer
bert-base-uncased 12 9
bert-large-uncased 24 18
bert-base-cased-finetuned-mrpc 12 9
bert-base-multilingual-cased 12 9
bert-base-chinese 12 8
roberta-base 12 10
roberta-large 24 17
roberta-large-mnli 24 19
xlnet-base-cased 12 5
xlnet-large-cased 24 7
xlm-mlm-en-2048 12 7
xlm-mlm-100-1280 16 11

表 8:计算 BERTScore 时建议使用的表示层。层数根据留出验证集(WMT16)选定。

模型 总层数 最佳层
bert-base-uncased 12 9
bert-large-uncased 24 18
bert-base-cased-finetuned-mrpc 12 9
bert-base-multilingual-cased 12 9
bert-base-chinese 12 8
roberta-base 12 10
roberta-large 24 17
roberta-large-mnli 24 19
xlnet-base-cased 12 5
xlnet-large-cased 24 7
xlm-mlm-en-2048 12 7
xlm-mlm-100-1280 16 11

Pearson correlation of BERTScore F1 with human judgments across layers of BERT, RoBERTa, XLNet, XLM, and BERT-Chinese

Figure 3: Pearson correlation of F\textscBERTF_{\textsc{BERT}}F\textscBERT​ computed with different models, across different layers, with segment-level human judgments on the WMT16 to-English machine translation task. The WMT17 English-Chinese data is used for the BERT Chinese model. Layer 0 corresponds to using BPE embeddings. Consistently, correlation drops significantly in the final layers.

图 3:用不同模型、在不同层上计算的 F\textscBERTF_{\textsc{BERT}}F\textscBERT​,与 WMT16 到英语机器翻译任务上片段级人工判断的皮尔逊相关。BERT 中文模型使用的是 WMT17 英语–中文数据。第 0 层对应使用 BPE 嵌入。最后几层的相关性会明显下降,这一现象很稳定。

Appendix C Ablation Study of MoverScore

附录 C MoverScore 的消融实验

Word Mover’s Distance (Kusner et al., 2015, WMD;) is a semantic similarity metric that relies on word embeddings and optimal transport. MoverScore (Zhao et al., 2019) combines contextual embeddings and WMD for text generation evaluation. In contrast, BERTScore adopts a greedy approach to aggregate token-level information. In addition to using WMD for generation evaluation, Zhao et al. (2019) also introduce various other improvements. We do a detailed ablation study to understand the benefit of each improvement, and to investigate whether it can be applied to BERTScore. We use a 12-layer uncased BERT model on the WMT17 to-English segment-level data, the same setting as Zhao et al. (2019).

Word Mover’s Distance(Kusner et al., 2015,简称 WMD)是一种依赖词嵌入和最优传输的语义相似度指标。MoverScore(Zhao et al., 2019)把上下文嵌入与 WMD 结合起来,用于生成文本评测。相比之下,BERTScore 用贪心方法汇聚词元级信息。除了把 WMD 用于生成评测,Zhao et al.(2019)还引入了其他多种改进。我们做了细致的消融实验,以理解每一项改进的收益,并考察它能否用到 BERTScore 上。实验使用 12 层、不区分大小写的 BERT 模型,数据是 WMT17 到英语的片段级数据,设置与 Zhao et al.(2019)相同。

We identify several differences between MoverScore and BERTScore by analyzing the released source code. We isolate each difference, and mark it with a bracketed tag for our ablation study:

通过分析已发布的源代码,我们找出了 MoverScore 与 BERTScore 的若干差异。把每一处差异单独隔离出来,并加上方括号标签,以便做消融:

  1. [MNLI] Use a BERT model fine-tuned on MNLI (Williams et al., 2018).
  1. [MNLI] 使用在 MNLI(Williams et al., 2018)上微调过的 BERT 模型。
  1. [PMEANS] Apply power means (Rücklé et al., 2018) to aggregate the information of different layers.[^9]
  1. [PMEANS] 用幂平均(Rücklé et al., 2018)汇聚不同层的信息。[^9]

[^9] Zhao et al. (2019) uses the embeddings from the last five layers from BERT and L2-normalizes the embedding vectors at each layer before computing the P-MEANs and L2-normalizing the concatenated P-MEANs.

[^9] Zhao et al.(2019)使用 BERT 最后五层的嵌入,并在每一层先对向量做 L2 归一化,再计算 P-MEANS,最后对拼接后的 P-MEANS 再做一次 L2 归一化。

  1. [IDF-L] For reference sentences, instead of computing the idf{\rm idf}idf scores on the 560 sentences in the segment-level data ([IDF-S]), compute the idf{\rm idf}idf scores on the 3,005 sentences in the system-level data.
  1. [IDF-L] 对参考句,不再在片段级数据的 560 个句子上计算 idf{\rm idf}idf 分数([IDF-S]),而是在系统级数据的 3,005 个句子上计算。
  1. [SEP] For candidate sentences, recompute the idf{\rm idf}idf scores on the candidate sentences. The weighting of reference tokens are kept the same as in [IDF-S]
  1. [SEP] 对候选句,在候选句上重新计算 idf{\rm idf}idf 分数。参考词元的加权仍与 [IDF-S] 中相同。
  1. [RM] Exclude punctuation marks and sub-word tokens except the first sub-word in each word from the matching.
  1. [RM] 在匹配时排除标点,以及每个词中除第一个子词以外的子词词元。

We follow the setup of Zhao et al. (2019) and use their released fine-tuned BERT model to conduct the experiments. Table 9 shows the results of our ablation study. We report correlations for the two variants of WMD Zhao et al. (2019) study: unigrams (WMD1) and bigrams (WMD2). Our F\textscBERTF_{\textsc{BERT}}F\textscBERT​ corresponds to the vanilla setting and the importance weighted variant corresponds to the [IDF-S] setting. The complete MoverScore metric corresponds to [IDF-S]+[SEP]+[PMEANS]+[MNLI]+[RM]. We make several observations. First, for all language pairs except fi-en and lv-en, we can replicate the reported performance. For these two language pairs, Zhao et al. (2019) did not release their implementations at the time of publication.[^10] Second, we confirm the effectiveness of [PMEANS] and [MNLI]. In Appendix F, we study more pre-trained models and further corroborate this conclusion. However, the contribution of other techniques, including [RM] and [SEP], seems less stable. Third, replacing greedy matching with WMD does not lead to consistent improvement. In fact, oftentimes BERTScore is the better metric when given the same setup. In general, for any given language pair, BERTScore is always among the best performing ones. Given the current results, it is not clear tht WMD is better than greedy matching for text generation evaluation.

我们沿用 Zhao et al.(2019)的设置,并用他们发布的微调 BERT 模型做实验。表 9 给出消融结果。我们报告 Zhao et al.(2019)研究的两种 WMD 变体的相关性:unigram(WMD1)和 bigram(WMD2)。我们的 F\textscBERTF_{\textsc{BERT}}F\textscBERT​ 对应最简设置,带重要性加权的变体则对应 [IDF-S] 设置。完整的 MoverScore 指标对应 [IDF-S]+[SEP]+[PMEANS]+[MNLI]+[RM]。我们有几点观察。第一,除 fi-en 和 lv-en 外,所有语言对上我们都能复现所报告的表现。对这两个语言对,Zhao et al.(2019)在发表时并未放出实现。[^10] 第二,我们确认了 [PMEANS] 和 [MNLI] 的有效性。附录 F 中我们研究了更多预训练模型,并进一步印证了这一结论。不过,包括 [RM] 和 [SEP] 在内的其他技术,贡献看起来不够稳定。第三,用 WMD 替换贪心匹配并不会带来稳定的提升。事实上,在相同设置下,BERTScore 往往才是更好的指标。总体而言,对任意给定的语言对,BERTScore 总是名列前茅。就目前结果来看,还不能清楚说明 WMD 比贪心匹配更适合生成文本评测。

[^10] A public comment on the project page indicates that some of the techniques are not applied for these two language pairs (https://github.com/AIPHES/emnlp19-moverscore/issues/1).

[^10] 项目主页上的一条公开评论指出,这两个语言对并未应用其中部分技术(https://github.com/AIPHES/emnlp19-moverscore/issues/1)。

Table 9: Ablation Study of MoverScore and BERTScore using Pearson correlations on the WMT17 to-English segment-level data. Correlations that are not outperformed by others for that language pair under Williams Test are bolded. We observe that using WMD does not consistently improve BERTScore.

Ablation Metric cs-en de-en fi-en lv-en ru-en tr-en zh-en
Vanilla WMD1 0.628 0.655 0.795 0.692 0.701 0.715 0.699
WMD2 0.638 0.661 0.797 0.695 0.700 0.728 0.714
FBERTF_{{\text{{BERT}}}}FBERT​ 0.659 0.680 0.817 0.702 0.719 0.727 0.717
IDF-S WMD1 0.636 0.662 0.824 0.709 0.716 0.728 0.713
WMD2 0.643 0.662 0.821 0.708 0.712 0.732 0.715
FBERTF_{{\text{{BERT}}}}FBERT​ 0.657 0.681 0.823 0.713 0.725 0.718 0.711
IDF-L WMD1 0.633 0.659 0.825 0.708 0.716 0.727 0.715
WMD2 0.641 0.661 0.822 0.708 0.713 0.730 0.716
FBERTF_{{\text{{BERT}}}}FBERT​ 0.655 0.682 0.823 0.713 0.726 0.718 0.712
IDF-L + SEP WMD1 0.651 0.660 0.819 0.703 0.714 0.724 0.715
WMD2 0.659 0.662 0.816 0.702 0.712 0.729 0.715
FBERTF_{{\text{{BERT}}}}FBERT​ 0.664 0.681 0.818 0.709 0.724 0.716 0.710
IDF-L + SEP + RM WMD1 0.651 0.686 0.803 0.681 0.730 0.730 0.720
WMD2 0.664 0.687 0.797 0.679 0.728 0.735 0.718
FBERTF_{{\text{{BERT}}}}FBERT​ 0.659 0.695 0.800 0.683 0.734 0.722 0.712
IDF-L + SEP + PMEANS WMD1 0.658 0.663 0.820 0.707 0.717 0.725 0.712
WMD2 0.667 0.665 0.817 0.707 0.717 0.727 0.712
FBERTF_{{\text{{BERT}}}}FBERT​ 0.671 0.682 0.819 0.708 0.725 0.715 0.704
IDF-L + SEP + MNLI WMD1 0.659 0.679 0.822 0.732 0.718 0.746 0.725
WMD2 0.664 0.682 0.819 0.731 0.715 0.748 0.722
FBERTF_{{\text{{BERT}}}}FBERT​ 0.668 0.701 0.825 0.737 0.727 0.744 0.725
IDF-L + SEP + PMEANS + MNLI WMD1 0.672 0.686 0.831 0.738 0.725 0.753 0.737
WMD2 0.677 0.690 0.828 0.736 0.722 0.755 0.735
FBERTF_{{\text{{BERT}}}}FBERT​ 0.682 0.707 0.836 0.741 0.732 0.751 0.736
IDF-L + SEP + PMEANS + MNLI + RM WMD1 0.670 0.708 0.821 0.717 0.738 0.762 0.744
WMD2 0.679 0.709 0.814 0.716 0.736 0.762 0.738
FBERTF_{{\text{{BERT}}}}FBERT​ 0.676 0.717 0.824 0.719 0.740 0.757 0.738

表 9:在 WMT17 到英语片段级数据上,用皮尔逊相关对 MoverScore 和 BERTScore 做的消融实验。在该语言对上按 Williams 检验未被其他结果超过的相关加粗。可以看到,使用 WMD 并不能稳定地改进 BERTScore。

消融 指标 cs-en de-en fi-en lv-en ru-en tr-en zh-en
原始设置 WMD1 0.628 0.655 0.795 0.692 0.701 0.715 0.699
WMD2 0.638 0.661 0.797 0.695 0.700 0.728 0.714
FBERTF_{{\text{{BERT}}}}FBERT​ 0.659 0.680 0.817 0.702 0.719 0.727 0.717
IDF-S WMD1 0.636 0.662 0.824 0.709 0.716 0.728 0.713
WMD2 0.643 0.662 0.821 0.708 0.712 0.732 0.715
FBERTF_{{\text{{BERT}}}}FBERT​ 0.657 0.681 0.823 0.713 0.725 0.718 0.711
IDF-L WMD1 0.633 0.659 0.825 0.708 0.716 0.727 0.715
WMD2 0.641 0.661 0.822 0.708 0.713 0.730 0.716
FBERTF_{{\text{{BERT}}}}FBERT​ 0.655 0.682 0.823 0.713 0.726 0.718 0.712
IDF-L + SEP WMD1 0.651 0.660 0.819 0.703 0.714 0.724 0.715
WMD2 0.659 0.662 0.816 0.702 0.712 0.729 0.715
FBERTF_{{\text{{BERT}}}}FBERT​ 0.664 0.681 0.818 0.709 0.724 0.716 0.710
IDF-L + SEP + RM WMD1 0.651 0.686 0.803 0.681 0.730 0.730 0.720
WMD2 0.664 0.687 0.797 0.679 0.728 0.735 0.718
FBERTF_{{\text{{BERT}}}}FBERT​ 0.659 0.695 0.800 0.683 0.734 0.722 0.712
IDF-L + SEP + PMEANS WMD1 0.658 0.663 0.820 0.707 0.717 0.725 0.712
WMD2 0.667 0.665 0.817 0.707 0.717 0.727 0.712
FBERTF_{{\text{{BERT}}}}FBERT​ 0.671 0.682 0.819 0.708 0.725 0.715 0.704
IDF-L + SEP + MNLI WMD1 0.659 0.679 0.822 0.732 0.718 0.746 0.725
WMD2 0.664 0.682 0.819 0.731 0.715 0.748 0.722
FBERTF_{{\text{{BERT}}}}FBERT​ 0.668 0.701 0.825 0.737 0.727 0.744 0.725
IDF-L + SEP + PMEANS + MNLI WMD1 0.672 0.686 0.831 0.738 0.725 0.753 0.737
WMD2 0.677 0.690 0.828 0.736 0.722 0.755 0.735
FBERTF_{{\text{{BERT}}}}FBERT​ 0.682 0.707 0.836 0.741 0.732 0.751 0.736
IDF-L + SEP + PMEANS + MNLI + RM WMD1 0.670 0.708 0.821 0.717 0.738 0.762 0.744
WMD2 0.679 0.709 0.814 0.716 0.736 0.762 0.738
FBERTF_{{\text{{BERT}}}}FBERT​ 0.676 0.717 0.824 0.719 0.740 0.757 0.738

Appendix D Additional Experiments on Abstractive Text Compression

附录 D 抽象文本压缩上的补充实验

We use the human judgments provided from the MSR Abstractive Text Compression Dataset (Toutanova et al., 2016) to illustrate the applicability of BERTScore to abstractive text compression evaluation. The data includes three types of human scores: (a) meaning: how well a compressed text preserve the meaning of the original text; (b) grammar: how grammatically correct a compressed text is; and (c) combined: the average of the meaning and the grammar scores. We follow the experimental setup of Toutanova et al. (2016) and report Pearson correlation between BERTScore and the three types of human scores. Table 10 shows that RBERTR_{\text{BERT}}RBERT​ has the highest correlation with human meaning judgments, and PBERTP_{\text{BERT}}PBERT​ correlates highly with human grammar judgments. FBERTF_{\text{BERT}}FBERT​ provides a balance between the two aspects.

我们使用 MSR Abstractive Text Compression Dataset(Toutanova et al., 2016)提供的人工判断,来说明 BERTScore 也适用于抽象文本压缩评测。数据包含三类人工分数:(a)意义:压缩文本在多大程度上保留了原文的意思;(b)语法:压缩文本在语法上有多正确;(c)综合:意义分与语法分的平均。我们沿用 Toutanova et al.(2016)的实验设置,报告 BERTScore 与这三类人工分数的皮尔逊相关。表 10 显示,RBERTR_{\text{BERT}}RBERT​ 与人工给出的意义判断相关最高,PBERTP_{\text{BERT}}PBERT​ 则与语法判断高度相关。FBERTF_{\text{BERT}}FBERT​ 在这两方面之间取得平衡。

Table 10: Pearson correlations with human judgments on the MSR Abstractive Text Compression Dataset.

Type Metric Meaning Grammar Combined
BERTScore PBERTP_{\text{BERT}}PBERT​ 0.36 0.47 0.46
RBERTR_{\text{BERT}}RBERT​ 0.64 0.29 0.52
FBERTF_{\text{BERT}}FBERT​ 0.58 0.41 0.56
Common metrics Bleu 0.46 0.13 0.33
Meteor 0.53 0.11 0.36
ROUGE-L 0.51 0.16 0.38
SARI 0.50 0.15 0.37
Best metrics according to Toutanova et al. (2016) SKIP-2+Recall+MULT-PROB 0.59 N/A 0.51
PARSE-2+Recall+MULT-MAX N/A 0.35 0.52
PARSE-2+Recall+MULT-PROB 0.57 0.35 0.52

表 10:在 MSR Abstractive Text Compression Dataset 上与人工判断的皮尔逊相关。

类型 指标 意义 语法 综合
BERTScore PBERTP_{\text{BERT}}PBERT​ 0.36 0.47 0.46
RBERTR_{\text{BERT}}RBERT​ 0.64 0.29 0.52
FBERTF_{\text{BERT}}FBERT​ 0.58 0.41 0.56
Common metrics BLEU 0.46 0.13 0.33
Meteor 0.53 0.11 0.36
ROUGE-L 0.51 0.16 0.38
SARI 0.50 0.15 0.37
Best metrics according to Toutanova et al. (2016) SKIP-2+Recall+MULT-PROB 0.59 N/A 0.51
PARSE-2+Recall+MULT-MAX N/A 0.35 0.52
PARSE-2+Recall+MULT-PROB 0.57 0.35 0.52

Appendix E BERTScore of Recent MT Models

附录 E 近期机器翻译模型的 BERTScore

Table 11: Bleu scores and BERTScores of publicly available pre-trained MT models in fairseq (Ott et al., 2019). We show both rescaled scores marked with p^\hat{\phantom{p}}p^​ and raw BERTScores. ∗: trained on unconfirmed WMT data version, ∗∗: trained on WMT16 + ParaCrawl, ∗∗∗: trained on WMT16, +: trained by us using fairseq.

Task Model Bleu P^\textscBERT\hat{P}_{\textsc{BERT}}P^\textscBERT​ R^\textscBERT\hat{R}_{\textsc{BERT}}R^\textscBERT​ F^\textscBERT\hat{F}_{\textsc{BERT}}F^\textscBERT​ P\textscBERTP_{\textsc{BERT}}P\textscBERT​ R\textscBERTR_{\textsc{BERT}}R\textscBERT​ F\textscBERTF_{\textsc{BERT}}F\textscBERT​
WMT14 En-De ConvS2S (Gehring et al., 2017) 0.266 0.6099 0.6055 0.6075 0.8499 0.8482 0.8488
Transformer-big*∗∗* (Ott et al., 2018) 0.298 0.6587 0.6528 0.6558 0.8687 0.8664 0.8674
DynamicConv*∗∗∗* (Wu et al., 2019) 0.297 0.6526 0.6464 0.6495 0.8664 0.8640 0.8650
WMT14 En-Fr ConvS2S (Gehring et al., 2017) 0.408 0.6998 0.6821 0.6908 0.8876 0.8810 0.8841
Transformer-big (Ott et al., 2018) 0.432 0.7148 0.6978 0.7061 0.8932 0.8869 0.8899
DynamicConv (Wu et al., 2019) 0.432 0.7156 0.6989 0.7071 0.8936 0.8873 0.8902
IWSLT14 De-En Transformer-iwslt+ (Ott et al., 2019) 0.350 0.6749 0.6590 0.6672 0.9452 0.9425 0.9438
LightConv (Wu et al., 2019) 0.348 0.6737 0.6542 0.6642 0.9450 0.9417 0.9433
DynamicConv (Wu et al., 2019) 0.352 0.6770 0.6586 0.6681 0.9456 0.9425 0.9440

表 11:fairseq(Ott et al., 2019)中公开发布的预训练机器翻译模型的 BLEU 和 BERTScore。我们同时给出标有 p^\hat{\phantom{p}}p^​ 的缩放分数和原始 BERTScore。∗:训练数据的 WMT 版本未经确认;∗∗:在 WMT16 + ParaCrawl 上训练;∗∗∗:在 WMT16 上训练;+:我们用 fairseq 自行训练。

任务 模型 BLEU P^\textscBERT\hat{P}_{\textsc{BERT}}P^\textscBERT​ R^\textscBERT\hat{R}_{\textsc{BERT}}R^\textscBERT​ F^\textscBERT\hat{F}_{\textsc{BERT}}F^\textscBERT​ P\textscBERTP_{\textsc{BERT}}P\textscBERT​ R\textscBERTR_{\textsc{BERT}}R\textscBERT​ F\textscBERTF_{\textsc{BERT}}F\textscBERT​
WMT14 En-De ConvS2S (Gehring et al., 2017) 0.266 0.6099 0.6055 0.6075 0.8499 0.8482 0.8488
Transformer-big*∗∗* (Ott et al., 2018) 0.298 0.6587 0.6528 0.6558 0.8687 0.8664 0.8674
DynamicConv*∗∗∗* (Wu et al., 2019) 0.297 0.6526 0.6464 0.6495 0.8664 0.8640 0.8650
WMT14 En-Fr ConvS2S (Gehring et al., 2017) 0.408 0.6998 0.6821 0.6908 0.8876 0.8810 0.8841
Transformer-big (Ott et al., 2018) 0.432 0.7148 0.6978 0.7061 0.8932 0.8869 0.8899
DynamicConv (Wu et al., 2019) 0.432 0.7156 0.6989 0.7071 0.8936 0.8873 0.8902
IWSLT14 De-En Transformer-iwslt+ (Ott et al., 2019) 0.350 0.6749 0.6590 0.6672 0.9452 0.9425 0.9438
LightConv (Wu et al., 2019) 0.348 0.6737 0.6542 0.6642 0.9450 0.9417 0.9433
DynamicConv (Wu et al., 2019) 0.352 0.6770 0.6586 0.6681 0.9456 0.9425 0.9440

Table 11 shows the Bleu scores and the BERTScores of pre-trained machine translation models on WMT14 English-to-German, WMT14 English-to-French, IWSLT14 German-to-English task. We used publicly available pre-trained models from fairseq (Ott et al., 2019).[^11] Because a pre-trained Transformer model on IWSLT is not released, we trained our own using the fairseq library. We use multilingual cased \textscBERTbase\textsc{BERT}_{{\rm base}}\textscBERTbase​[^12] for English-to-German and English-to-French pairs, and English uncased \textscBERTbase\textsc{BERT}_{{\rm base}}\textscBERTbase​[^13] for German-to-English pairs. Interestingly, the gap between a DynamicConv (Wu et al., 2019) trained on only WMT16 and a Transformer (Ott et al., 2018) trained on WMT16 and ParaCrawl[^14] (about 30×\times× more training data) becomes larger when evaluated with BERTScore rather than Bleu.

表 11 给出了预训练机器翻译模型在 WMT14 英语到德语、WMT14 英语到法语,以及 IWSLT14 德语到英语任务上的 BLEU 和 BERTScore。我们使用了 fairseq(Ott et al., 2019)中公开发布的预训练模型。[^11] 由于 IWSLT 上的预训练 Transformer 模型并未发布,我们用 fairseq 库自行训练了一套。英语到德语和英语到法语使用区分大小写的多语 \textscBERTbase\textsc{BERT}_{{\rm base}}\textscBERTbase​,[^12] 德语到英语使用不区分大小写的英语 \textscBERTbase\textsc{BERT}_{{\rm base}}\textscBERTbase​。[^13] 有意思的是,只在 WMT16 上训练的 DynamicConv(Wu et al., 2019),与在 WMT16 和 ParaCrawl[^14] 上训练的 Transformer(Ott et al., 2018,训练数据大约多 30 倍)之间的差距,用 BERTScore 来评时比用 BLEU 更大。

[^11] Code and pre-trained model available at https://github.com/pytorch/fairseq.

[^11] 代码和预训练模型见 https://github.com/pytorch/fairseq。

[^12] Hash code: bert-base-multilingual-cased_L9_version=0.2.0

[^13] Hash code: roberta-large_L17_version=0.2.0

[^14] http://paracrawl.eu/download.html

Appendix F Additional Results

附录 F 补充结果

In this section, we present additional experimental results:

本节给出补充实验结果:

  1. Segment-level and system-level correlation studies on three years of WMT metric evaluation task (WMT16–18)
  1. 在连续三年 WMT 指标评测任务(WMT16–18)上的片段级和系统级相关研究
  1. Model selection study on WMT18 10K hybrid systems
  1. 在 WMT18 的 1 万个混合系统上的模型选择研究
  1. System-level correlation study on 2015 COCO captioning challenge
  1. 在 2015 年 COCO 图像描述挑战上的系统级相关研究
  1. Robustness study on PAWS-QQP.
  1. 在 PAWS-QQP 上的稳健性研究。

Following BERT (Devlin et al., 2019), a variety of Transformer-based (Vaswani et al., 2017) pre-trained contextual embeddings have been proposed and released. We conduct additional experiments with four types of pre-trained embeddings: BERT, XLM (Lample & Conneau, 2019), XLNet (Yang et al., 2019b), and RoBERTa (Liu et al., 2019b). XLM (Cross-lingual Language Model) is a Transformer pre-trained on the translation language modeling of predicting masked tokens from a pair of sentence in two different languages and masked language modeling tasks using multi-lingual training data. Yang et al. (2019b) modify the Transformer architecture and pre-train it on a permutation language modeling task resulting in some improvement on top of the original BERT when fine-tuned on several downstream tasks. Liu et al. (2019b) introduce RoBERTa (Robustly optimized BERT approach) and demonstrate that an optimized BERT model is comparable to or sometimes outperforms an XLNet on downstream tasks.

继 BERT(Devlin et al., 2019)之后,又有多种基于 Transformer(Vaswani et al., 2017)的预训练上下文嵌入被提出并发布。我们用四类预训练嵌入做了补充实验:BERT、XLM(Lample & Conneau, 2019)、XLNet(Yang et al., 2019b)和 RoBERTa(Liu et al., 2019b)。XLM(Cross-lingual Language Model)是在翻译语言建模(从一对两种语言的句子中预测被掩码的词元)以及使用多语训练数据的掩码语言建模任务上预训练的 Transformer。Yang et al.(2019b)修改了 Transformer 结构,并在排列语言建模任务上预训练,微调到若干下游任务时,相对原始 BERT 有一些提升。Liu et al.(2019b)提出 RoBERTa(Robustly optimized BERT approach),并表明经过优化的 BERT 模型在下游任务上可以媲美、有时还超过 XLNet。

We perform a comprehensive study with the following pre-trained contextual embedding models:[^15]

我们用下列预训练上下文嵌入模型做了全面研究:[^15]

[^15] Denoted by names specified at https://huggingface.co/pytorch-transformers/pretrained_models.html.

[^15] 名称按 https://huggingface.co/pytorch-transformers/pretrained_models.html 中的规定书写。

  • • BERT models: bert-base-uncased, bert-large-uncased, bert-based-chinese, bert-base-multilingual-cased, and bert-base-cased-mrpc
  • BERT 模型:bert-base-uncased、bert-large-uncased、bert-based-chinese、bert-base-multilingual-cased 和 bert-base-cased-mrpc
  • • RoBERTa models: roberta-base, roberta-large, and roberta-large-mnli
  • RoBERTa 模型:roberta-base、roberta-large 和 roberta-large-mnli
  • • XLNet models: xlnet-base-cased and xlnet-base-large
  • XLNet 模型:xlnet-base-cased 和 xlnet-base-large
  • • XLM models: xlm-mlm-en-2048 and xlm-mlm-100-1280
  • XLM 模型:xlm-mlm-en-2048 和 xlm-mlm-100-1280

F.1 WMT Correlation Study

F.1 WMT 相关性研究

Experimental setup

实验设置

Because of missing data in the released WMT16 dataset (Bojar et al., 2016), we are only able to experiment with to-English segment-level data, which contains the outputs of 50 different systems on 6 language pairs. We use this data as the validation set for hyperparameter tuning (Appendix B). Table 12 shows the Pearson correlations of all participating metrics and BERTScores computed with different pre-trained models. Significance testing for this dataset does not include the baseline metrics because the released dataset does not contain the original outputs from the baseline metrics. We conduct significance testing between BERTScore results only.

由于发布的 WMT16 数据集(Bojar et al., 2016)存在缺失,我们只能在到英语的片段级数据上做实验,其中包含 6 个语言对上 50 套不同系统的输出。我们把这批数据当作验证集,用于超参数调节(附录 B)。表 12 给出全部参赛指标,以及用不同预训练模型计算的 BERTScore 的皮尔逊相关。对该数据集做显著性检验时不包括基线指标,因为发布的数据中没有这些基线指标的原始输出。我们只在 BERTScore 的结果之间做显著性检验。

The WMT17 dataset (Bojar et al., 2017) contains outputs of 152 different translations on 14 language pairs. We experiment on the segment-level and system-level data on both to-English and from-English language pairs. We exclude fi-en data from the segment-level experiment due to an error in the released data. We compare our results to all participating metrics and perform standard significance testing as done by Bojar et al. (2017). Tables 13–16 show the results.

WMT17 数据集(Bojar et al., 2017)包含 14 个语言对上 152 套不同译文的输出。我们在到英语和从英语出发的语言对上,分别做片段级和系统级实验。由于发布数据有误,片段级实验排除了 fi-en。我们与全部参赛指标比较,并按 Bojar et al.(2017)的做法做标准显著性检验。结果见表 13–16。

The WMT18 dataset (Ma et al., 2018) contains outputs of 159 translation systems on 14 language pairs. In addition to the results in Tables 1–4, we complement the study with the correlations of all participating metrics in WMT18 and results from using different contextual models for BERTScore.

WMT18 数据集(Ma et al., 2018)包含 14 个语言对上 159 套翻译系统的输出。除了表 1–4 中的结果,我们还补充了 WMT18 全部参赛指标的相关性,以及用不同上下文模型计算 BERTScore 的结果。

Results

结果

Table 12–22 collectively showcase the effectiveness of BERTScore in correlating with human judgments. The improvement of BERTScore is more pronounced on the segment-level than on the system-level. We also see that more optimized or larger BERT models can produce better contextual representations (e.g., comparing FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ and FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​). In contrast, the smaller XLNet performs better than a large one. Based on the evidence in Figure 8 and Tables 12–22, we hypothesize that the permutation language task, though leading to a good set of model weights for fine-tuning on downstream tasks, does not necessarily produce informative pre-trained embeddings for generation evaluation. We also observe that fine-tuning pre-trained models on a related task, such as natural language inference (Williams et al., 2018), can lead to better human correlation in evaluating text generation. Therefore, for evaluating English sentences, we recommend computing BERTScore with a 24-layer RoBERTa model fine-tuned on the MNLI dataset. For evaluating Non-English sentences, both the multilingual BERT model and the XLM model trained on 100 languages are suitable candidates. We also recommend using domain- or language-specific contextual embeddings when possible, such as using BERT Chinese models for evaluating Chinese tasks. In general, we advise users to consider the target domain and languages when selecting the exact configuration to use.

表 12–22 共同展示了 BERTScore 与人工判断相关的有效性。提升在片段级上比在系统级上更明显。我们还看到,更充分优化或规模更大的 BERT 模型可以给出更好的上下文表示(例如比较 FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 和 FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​)。与此不同,较小的 XLNet 反而好于更大的那个。根据图 8 和表 12–22 中的证据,我们推测:排列语言建模任务虽然能得到一套适合在下游任务上微调的模型权重,但不一定能产生对生成评测有信息量的预训练嵌入。我们还观察到,在相关任务上微调预训练模型,例如自然语言推理(Williams et al., 2018),可以提高文本生成评测与人工判断的相关性。因此,评测英语句子时,我们建议用在 MNLI 数据集上微调过的 24 层 RoBERTa 来计算 BERTScore。评测非英语句子时,多语 BERT 模型以及在 100 种语言上训练的 XLM 模型都是合适的候选。若有可能,我们也建议使用领域或语言专用的上下文嵌入,例如用 BERT 中文模型来评测中文任务。总体而言,我们建议使用者在选定具体配置时,考虑目标领域和语言。

F.2 Model Selection Study

F.2 模型选择研究

Experimental setup

实验设置

Similar to Section 4, we use the 10K hybrid systems super-sampled from WMT18. We randomly select 100 out of 10K hybrid systems, rank them using automatic metrics, and repeat this process 100K times. We add to the results in the main paper (Table 3) performance of all participating metrics in WMT18 and results from using different contextual embedding models for BERTScore. We reuse the hybrid configuration and metric outputs released in WMT18. In addition to the Hits@1 measure, we evaluate the metrics using (a) mean reciprocal rank (MRR) of the top metric-rated system in human rankings, and (b) the absolute human score difference (Diff) between the top metric- and human-rated systems. Hits@1 captures a metric’s ability to select the best system. The other two measures quantify the amount of error a metric makes in the selection process. Tables 23–28 show the results from these experiments.

与第 4 节类似,我们使用从 WMT18 超采样得到的 1 万个混合系统。从中随机抽出 100 个,用自动指标排序,并将这一过程重复 10 万次。我们在正文表 3 的结果之外,补充了 WMT18 全部参赛指标的表现,以及用不同上下文嵌入模型计算 BERTScore 的结果。混合系统的配置和指标输出沿用 WMT18 发布的版本。除了 Hits@1,我们还用另外两项度量来评价这些指标:(a)指标评为最优的系统在人工排序中的平均倒数排名(MRR);(b)指标评为最优与人工评为最优的系统之间,人工分数的绝对差(Diff)。Hits@1 刻画指标选出最佳系统的能力,后两项则量化选择过程中的误差大小。这些实验的结果见表 23–28。

Results

结果

The additional results further support our conclusion from Table 3: BERTScore demonstrates better model selection performance. We also observe that the supervised metric RUSE displays strong model selection ability.

这些补充结果进一步支持我们从表 3 得出的结论:BERTScore 具有更好的模型选择表现。我们也看到,有监督指标 RUSE 同样显示出很强的模型选择能力。

F.3 Image Captioning on COCO

F.3 COCO 上的图像描述

We follow the experimental setup described in Section 4. Table 29 shows the correlations of several pre-trained contextual embeddings. We observe that precision-based methods such as Bleu and P\textscBERTP_{\textsc{BERT}}P\textscBERT​ are weakly correlated with human judgments on image captioning tasks. We hypothesize that this is because human judges prefer captions that capture the main objects in a picture for image captioning. In general, R\textscBERTR_{\textsc{BERT}}R\textscBERT​ has a high correlation, even surpassing the task-specific metric Spice Anderson et al. (2016). While the fine-tuned RoBERTa-Large model does not result in the highest correlation, it is one of the best metrics.

我们沿用第 4 节中的实验设置。表 29 给出若干预训练上下文嵌入的相关性。可以看到,BLEU 和 P\textscBERTP_{\textsc{BERT}}P\textscBERT​ 这类基于准确率的方法,在图像描述任务上与人工判断的相关性较弱。我们推测,这是因为在图像描述中,人工评委更偏好能够抓住画面主要对象的描述。总体来看,R\textscBERTR_{\textsc{BERT}}R\textscBERT​ 的相关性很高,甚至超过了任务专用指标 SPICE(Anderson et al., 2016)。微调后的 RoBERTa-Large 虽然不是相关最高的,但仍是最好的指标之一。

F.4 Robustness Analysis on PAWS-QQP

F.4 PAWS-QQP 上的稳健性分析

We present the full results of the robustness study described in Section 6 in Table 30. In general, we observe that BERTScore is more robust than other commonly used metrics. BERTScore computed with the 24-layer RoBERTa model performs the best. Fine-tuning RoBERTa-Large on MNLI (Williams et al., 2018) can significantly improve the robustness against adversarial sentences. However, a fine-tuned BERT on MRPC (Microsoft Research Paraphrasing Corpus) (Dolan & Brockett, 2005) performs worse than its counterpart.

表 30 给出第 6 节所述稳健性研究的完整结果。总体来看,BERTScore 比其他常用指标更稳健。用 24 层 RoBERTa 模型计算的 BERTScore 表现最好。在 MNLI(Williams et al., 2018)上微调 RoBERTa-Large,可以显著提高对对抗句子的稳健性。不过,在 MRPC(Microsoft Research Paraphrasing Corpus)(Dolan & Brockett, 2005)上微调过的 BERT,表现反而差于未做该微调的对应模型。

Table 12: Pearson correlations with segment-level human judgments on WMT16 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Setting Metric cs-en de-en fi-en ro-en ru-en tr-en
560 560 560 560 560 560
Unsupervised DPMFcomb 0.713 0.584 0.598 0.627 0.615 0.663
metrics-f 0.696 0.601 0.557 0.662 0.618 0.649
cobalt-f. 0.671 0.591 0.554 0.639 0.618 0.627
upf-coba. 0.652 0.550 0.490 0.616 0.556 0.626
MPEDA 0.644 0.538 0.513 0.587 0.545 0.616
chrF2 0.658 0.457 0.469 0.581 0.534 0.556
chrF3 0.660 0.455 0.472 0.582 0.535 0.555
chrF1 0.644 0.454 0.452 0.570 0.522 0.551
UoW-ReVaL 0.577 0.528 0.471 0.547 0.528 0.531
wordF3 0.599 0.447 0.473 0.525 0.504 0.536
wordF2 0.596 0.445 0.471 0.522 0.503 0.537
wordF1 0.585 0.435 0.464 0.508 0.497 0.535
SentBLEU 0.557 0.448 0.484 0.499 0.502 0.532
DTED 0.394 0.254 0.361 0.329 0.375 0.267
Supervised BEER 0.661 0.462 0.471 0.551 0.533 0.545
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.729 0.617 0.719 0.651 0.684 0.678
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.741 0.639 0.616 0.693 0.660 0.660
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.747 0.640 0.661 0.723 0.672 0.688
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (no idf) 0.723 0.638 0.662 0.700 0.633 0.696
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (no idf) 0.745 0.656 0.638 0.697 0.653 0.674
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (no idf) 0.747 0.663 0.666 0.714 0.662 0.703
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.697 0.618 0.614 0.676 0.62 0.695
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.723 0.636 0.587 0.667 0.648 0.664
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.725 0.644 0.617 0.691 0.654 0.702
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.713 0.613 0.630 0.693 0.635 0.691
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.727 0.631 0.573 0.666 0.642 0.662
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.735 0.637 0.620 0.700 0.658 0.697
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.756 0.671 0.701 0.723 0.678 0.706
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.768 0.684 0.677 0.720 0.686 0.699
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.774 0.693 0.705 0.736 0.701 0.717
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.758 0.653 0.704 0.734 0.685 0.705
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.771 0.680 0.661 0.718 0.687 0.692
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.774 0.678 0.700 0.740 0.701 0.711
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.738 0.642 0.671 0.712 0.669 0.671
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.745 0.669 0.645 0.698 0.682 0.653
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.761 0.674 0.686 0.732 0.697 0.689
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.751 0.626 0.678 0.723 0.685 0.668
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.744 0.652 0.638 0.699 0.685 0.657
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.767 0.653 0.688 0.737 0.705 0.685
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.757 0.702 0.709 0.735 0.721 0.676
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.765 0.713 0.686 0.718 0.714 0.676
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.780 0.724 0.728 0.753 0.738 0.709
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.771 0.682 0.705 0.727 0.714 0.681
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.762 0.695 0.683 0.711 0.708 0.678
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.786 0.704 0.727 0.747 0.732 0.711
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.777 0.718 0.733 0.744 0.729 0.747
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.790 0.731 0.702 0.741 0.727 0.732
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.795 0.736 0.733 0.757 0.744 0.756
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.794 0.695 0.731 0.752 0.732 0.747
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.792 0.706 0.694 0.737 0.724 0.733
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.804 0.710 0.729 0.760 0.742 0.754
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ 0.708 0.612 0.639 0.650 0.606 0.690
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ 0.728 0.630 0.617 0.645 0.621 0.675
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ 0.727 0.631 0.640 0.659 0.626 0.695
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ (idf) 0.726 0.618 0.655 0.678 0.629 0.700
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ (idf) 0.734 0.633 0.618 0.66 0.635 0.682
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ (idf) 0.739 0.633 0.649 0.681 0.643 0.702
PXL-NET–LARGEP_{\text{XL-NET--{LARGE}}}PXL-NET–LARGE​ 0.710 0.577 0.643 0.647 0.616 0.684
RXL-NET–LARGER_{\text{XL-NET--{LARGE}}}RXL-NET–LARGE​ 0.732 0.600 0.610 0.636 0.627 0.668
FXL-NET–LARGEF_{\text{XL-NET--{LARGE}}}FXL-NET–LARGE​ 0.733 0.600 0.643 0.655 0.637 0.691
PXL-NET–LARGEP_{\text{XL-NET--{LARGE}}}PXL-NET–LARGE​ (idf) 0.728 0.574 0.652 0.669 0.633 0.681
RXL-NET–LARGER_{\text{XL-NET--{LARGE}}}RXL-NET–LARGE​ (idf) 0.735 0.592 0.597 0.642 0.629 0.662
FXL-NET–LARGEF_{\text{XL-NET--{LARGE}}}FXL-NET–LARGE​ (idf) 0.742 0.592 0.643 0.670 0.645 0.685
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.688 0.569 0.613 0.645 0.583 0.659
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.715 0.603 0.577 0.645 0.609 0.644
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.713 0.597 0.610 0.657 0.610 0.668
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.728 0.576 0.649 0.681 0.604 0.683
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.730 0.597 0.591 0.659 0.622 0.669
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.739 0.594 0.636 0.682 0.626 0.691

表 12:与 WMT16 到英语译文的片段级人工判断的皮尔逊相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了样本数。

设置 指标 cs-en de-en fi-en ro-en ru-en tr-en
560 560 560 560 560 560
无监督 DPMFcomb 0.713 0.584 0.598 0.627 0.615 0.663
metrics-f 0.696 0.601 0.557 0.662 0.618 0.649
cobalt-f. 0.671 0.591 0.554 0.639 0.618 0.627
upf-coba. 0.652 0.550 0.490 0.616 0.556 0.626
MPEDA 0.644 0.538 0.513 0.587 0.545 0.616
chrF2 0.658 0.457 0.469 0.581 0.534 0.556
chrF3 0.660 0.455 0.472 0.582 0.535 0.555
chrF1 0.644 0.454 0.452 0.570 0.522 0.551
UoW-ReVaL 0.577 0.528 0.471 0.547 0.528 0.531
wordF3 0.599 0.447 0.473 0.525 0.504 0.536
wordF2 0.596 0.445 0.471 0.522 0.503 0.537
wordF1 0.585 0.435 0.464 0.508 0.497 0.535
SentBLEU 0.557 0.448 0.484 0.499 0.502 0.532
DTED 0.394 0.254 0.361 0.329 0.375 0.267
Supervised BEER 0.661 0.462 0.471 0.551 0.533 0.545
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.729 0.617 0.719 0.651 0.684 0.678
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.741 0.639 0.616 0.693 0.660 0.660
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.747 0.640 0.661 0.723 0.672 0.688
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (no idf) 0.723 0.638 0.662 0.700 0.633 0.696
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (no idf) 0.745 0.656 0.638 0.697 0.653 0.674
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (no idf) 0.747 0.663 0.666 0.714 0.662 0.703
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.697 0.618 0.614 0.676 0.62 0.695
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.723 0.636 0.587 0.667 0.648 0.664
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.725 0.644 0.617 0.691 0.654 0.702
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.713 0.613 0.630 0.693 0.635 0.691
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.727 0.631 0.573 0.666 0.642 0.662
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.735 0.637 0.620 0.700 0.658 0.697
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.756 0.671 0.701 0.723 0.678 0.706
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.768 0.684 0.677 0.720 0.686 0.699
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.774 0.693 0.705 0.736 0.701 0.717
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.758 0.653 0.704 0.734 0.685 0.705
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.771 0.680 0.661 0.718 0.687 0.692
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.774 0.678 0.700 0.740 0.701 0.711
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.738 0.642 0.671 0.712 0.669 0.671
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.745 0.669 0.645 0.698 0.682 0.653
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.761 0.674 0.686 0.732 0.697 0.689
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.751 0.626 0.678 0.723 0.685 0.668
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.744 0.652 0.638 0.699 0.685 0.657
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.767 0.653 0.688 0.737 0.705 0.685
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.757 0.702 0.709 0.735 0.721 0.676
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.765 0.713 0.686 0.718 0.714 0.676
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.780 0.724 0.728 0.753 0.738 0.709
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.771 0.682 0.705 0.727 0.714 0.681
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.762 0.695 0.683 0.711 0.708 0.678
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.786 0.704 0.727 0.747 0.732 0.711
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.777 0.718 0.733 0.744 0.729 0.747
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.790 0.731 0.702 0.741 0.727 0.732
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.795 0.736 0.733 0.757 0.744 0.756
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.794 0.695 0.731 0.752 0.732 0.747
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.792 0.706 0.694 0.737 0.724 0.733
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.804 0.710 0.729 0.760 0.742 0.754
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ 0.708 0.612 0.639 0.650 0.606 0.690
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ 0.728 0.630 0.617 0.645 0.621 0.675
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ 0.727 0.631 0.640 0.659 0.626 0.695
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ (idf) 0.726 0.618 0.655 0.678 0.629 0.700
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ (idf) 0.734 0.633 0.618 0.66 0.635 0.682
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ (idf) 0.739 0.633 0.649 0.681 0.643 0.702
PXL-NET–LARGEP_{\text{XL-NET--{LARGE}}}PXL-NET–LARGE​ 0.710 0.577 0.643 0.647 0.616 0.684
RXL-NET–LARGER_{\text{XL-NET--{LARGE}}}RXL-NET–LARGE​ 0.732 0.600 0.610 0.636 0.627 0.668
FXL-NET–LARGEF_{\text{XL-NET--{LARGE}}}FXL-NET–LARGE​ 0.733 0.600 0.643 0.655 0.637 0.691
PXL-NET–LARGEP_{\text{XL-NET--{LARGE}}}PXL-NET–LARGE​ (idf) 0.728 0.574 0.652 0.669 0.633 0.681
RXL-NET–LARGER_{\text{XL-NET--{LARGE}}}RXL-NET–LARGE​ (idf) 0.735 0.592 0.597 0.642 0.629 0.662
FXL-NET–LARGEF_{\text{XL-NET--{LARGE}}}FXL-NET–LARGE​ (idf) 0.742 0.592 0.643 0.670 0.645 0.685
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.688 0.569 0.613 0.645 0.583 0.659
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.715 0.603 0.577 0.645 0.609 0.644
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.713 0.597 0.610 0.657 0.610 0.668
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.728 0.576 0.649 0.681 0.604 0.683
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.730 0.597 0.591 0.659 0.622 0.669
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.739 0.594 0.636 0.682 0.626 0.691

Table 13: Absolute Pearson correlations with segment-level human judgments on WMT17 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Setting Metric cs-en de-en fi-en lv-en ru-en tr-en zh-en
560 560 560 560 560 560 560
Unsupervised chrF 0.514 0.531 0.671 0.525 0.599 0.607 0.591
chrF++ 0.523 0.534 0.678 0.520 0.588 0.614 0.593
MEANT 2.0 0.578 0.565 0.687 0.586 0.607 0.596 0.639
MEANT 2.0-nosrl 0.566 0.564 0.682 0.573 0.591 0.582 0.630
SentBLEU 0.435 0.432 0.571 0.393 0.484 0.538 0.512
TreeAggreg 0.486 0.526 0.638 0.446 0.555 0.571 0.535
UHH_TSKM 0.507 0.479 0.600 0.394 0.465 0.478 0.477
Supervised AutoDA 0.499 0.543 0.673 0.533 0.584 0.625 0.583
BEER 0.511 0.530 0.681 0.515 0.577 0.600 0.582
BLEND 0.594 0.571 0.733 0.577 0.622 0.671 0.661
BLEU2VEC 0.439 0.429 0.590 0.386 0.489 0.529 0.526
NGRAM2VEC 0.436 0.435 0.582 0.383 0.490 0.538 0.520
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.625 0.659 0.808 0.688 0.698 0.713 0.675
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.653 0.645 0.782 0.662 0.678 0.716 0.715
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.654 0.671 0.811 0.692 0.707 0.731 0.714
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.626 0.668 0.819 0.708 0.719 0.702 0.667
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.652 0.658 0.789 0.678 0.696 0.703 0.712
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.657 0.680 0.823 0.712 0.725 0.718 0.711
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.599 0.630 0.788 0.657 0.659 0.710 0.681
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.613 0.620 0.754 0.616 0.650 0.685 0.705
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.627 0.647 0.792 0.656 0.676 0.717 0.712
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.609 0.630 0.801 0.680 0.676 0.712 0.682
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.611 0.628 0.759 0.633 0.665 0.687 0.703
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.633 0.649 0.803 0.678 0.690 0.719 0.713
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.638 0.685 0.816 0.717 0.719 0.746 0.693
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.661 0.676 0.782 0.693 0.705 0.744 0.730
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.666 0.701 0.814 0.723 0.730 0.760 0.731
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.644 0.692 0.827 0.728 0.729 0.734 0.689
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.665 0.686 0.796 0.712 0.729 0.733 0.730
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.671 0.707 0.829 0.738 0.745 0.746 0.729
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.639 0.663 0.801 0.689 0.688 0.700 0.704
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.648 0.652 0.768 0.651 0.669 0.684 0.734
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.675 0.683 0.818 0.693 0.707 0.718 0.740
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.629 0.655 0.804 0.702 0.711 0.707 0.700
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.652 0.646 0.773 0.667 0.676 0.689 0.734
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.673 0.673 0.823 0.708 0.719 0.721 0.739
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.658 0.724 0.811 0.743 0.727 0.720 0.744
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.685 0.714 0.778 0.711 0.718 0.713 0.759
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.710 0.745 0.833 0.756 0.746 0.751 0.775
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.644 0.721 0.815 0.740 0.734 0.736 0.734
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.683 0.705 0.783 0.718 0.720 0.726 0.751
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.703 0.737 0.838 0.761 0.752 0.764 0.767
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.694 0.736 0.822 0.764 0.741 0.754 0.737
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.706 0.725 0.785 0.732 0.741 0.750 0.760
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.722 0.747 0.822 0.764 0.758 0.767 0.765
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.686 0.733 0.836 0.772 0.760 0.767 0.738
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.697 0.717 0.796 0.741 0.753 0.757 0.762
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.714 0.740 0.835 0.774 0.773 0.776 0.767
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.595 0.579 0.779 0.632 0.626 0.688 0.646
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.603 0.560 0.746 0.617 0.624 0.689 0.677
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.610 0.580 0.775 0.636 0.639 0.700 0.675
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.616 0.603 0.795 0.665 0.659 0.693 0.649
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.614 0.583 0.765 0.640 0.648 0.697 0.688
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.627 0.603 0.795 0.663 0.665 0.707 0.684
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.620 0.622 0.796 0.648 0.648 0.694 0.660
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.622 0.601 0.758 0.628 0.645 0.684 0.701
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.635 0.627 0.794 0.654 0.664 0.705 0.698
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.635 0.633 0.808 0.673 0.672 0.688 0.649
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.626 0.611 0.770 0.646 0.661 0.682 0.700
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.646 0.636 0.809 0.675 0.682 0.700 0.695
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.565 0.594 0.769 0.631 0.649 0.672 0.643
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.592 0.586 0.734 0.618 0.647 0.673 0.686
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.595 0.605 0.768 0.641 0.664 0.686 0.683
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.599 0.618 0.795 0.670 0.686 0.690 0.657
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.624 0.605 0.768 0.652 0.680 0.684 0.698
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.630 0.624 0.798 0.676 0.698 0.698 0.694

表 13:与 WMT17 到英语译文的片段级人工判断的绝对皮尔逊相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了样本数。

设置 指标 cs-en de-en fi-en lv-en ru-en tr-en zh-en
560 560 560 560 560 560 560
无监督 chrF 0.514 0.531 0.671 0.525 0.599 0.607 0.591
chrF++ 0.523 0.534 0.678 0.520 0.588 0.614 0.593
MEANT 2.0 0.578 0.565 0.687 0.586 0.607 0.596 0.639
MEANT 2.0-nosrl 0.566 0.564 0.682 0.573 0.591 0.582 0.630
SentBLEU 0.435 0.432 0.571 0.393 0.484 0.538 0.512
TreeAggreg 0.486 0.526 0.638 0.446 0.555 0.571 0.535
UHH_TSKM 0.507 0.479 0.600 0.394 0.465 0.478 0.477
Supervised AutoDA 0.499 0.543 0.673 0.533 0.584 0.625 0.583
BEER 0.511 0.530 0.681 0.515 0.577 0.600 0.582
BLEND 0.594 0.571 0.733 0.577 0.622 0.671 0.661
BLEU2VEC 0.439 0.429 0.590 0.386 0.489 0.529 0.526
NGRAM2VEC 0.436 0.435 0.582 0.383 0.490 0.538 0.520
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.625 0.659 0.808 0.688 0.698 0.713 0.675
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.653 0.645 0.782 0.662 0.678 0.716 0.715
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.654 0.671 0.811 0.692 0.707 0.731 0.714
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.626 0.668 0.819 0.708 0.719 0.702 0.667
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.652 0.658 0.789 0.678 0.696 0.703 0.712
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.657 0.680 0.823 0.712 0.725 0.718 0.711
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.599 0.630 0.788 0.657 0.659 0.710 0.681
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.613 0.620 0.754 0.616 0.650 0.685 0.705
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.627 0.647 0.792 0.656 0.676 0.717 0.712
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.609 0.630 0.801 0.680 0.676 0.712 0.682
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.611 0.628 0.759 0.633 0.665 0.687 0.703
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.633 0.649 0.803 0.678 0.690 0.719 0.713
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.638 0.685 0.816 0.717 0.719 0.746 0.693
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.661 0.676 0.782 0.693 0.705 0.744 0.730
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.666 0.701 0.814 0.723 0.730 0.760 0.731
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.644 0.692 0.827 0.728 0.729 0.734 0.689
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.665 0.686 0.796 0.712 0.729 0.733 0.730
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.671 0.707 0.829 0.738 0.745 0.746 0.729
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.639 0.663 0.801 0.689 0.688 0.700 0.704
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.648 0.652 0.768 0.651 0.669 0.684 0.734
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.675 0.683 0.818 0.693 0.707 0.718 0.740
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.629 0.655 0.804 0.702 0.711 0.707 0.700
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.652 0.646 0.773 0.667 0.676 0.689 0.734
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.673 0.673 0.823 0.708 0.719 0.721 0.739
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.658 0.724 0.811 0.743 0.727 0.720 0.744
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.685 0.714 0.778 0.711 0.718 0.713 0.759
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.710 0.745 0.833 0.756 0.746 0.751 0.775
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.644 0.721 0.815 0.740 0.734 0.736 0.734
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.683 0.705 0.783 0.718 0.720 0.726 0.751
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.703 0.737 0.838 0.761 0.752 0.764 0.767
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.694 0.736 0.822 0.764 0.741 0.754 0.737
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.706 0.725 0.785 0.732 0.741 0.750 0.760
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.722 0.747 0.822 0.764 0.758 0.767 0.765
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.686 0.733 0.836 0.772 0.760 0.767 0.738
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.697 0.717 0.796 0.741 0.753 0.757 0.762
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.714 0.740 0.835 0.774 0.773 0.776 0.767
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.595 0.579 0.779 0.632 0.626 0.688 0.646
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.603 0.560 0.746 0.617 0.624 0.689 0.677
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.610 0.580 0.775 0.636 0.639 0.700 0.675
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.616 0.603 0.795 0.665 0.659 0.693 0.649
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.614 0.583 0.765 0.640 0.648 0.697 0.688
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.627 0.603 0.795 0.663 0.665 0.707 0.684
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.620 0.622 0.796 0.648 0.648 0.694 0.660
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.622 0.601 0.758 0.628 0.645 0.684 0.701
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.635 0.627 0.794 0.654 0.664 0.705 0.698
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.635 0.633 0.808 0.673 0.672 0.688 0.649
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.626 0.611 0.770 0.646 0.661 0.682 0.700
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.646 0.636 0.809 0.675 0.682 0.700 0.695
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.565 0.594 0.769 0.631 0.649 0.672 0.643
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.592 0.586 0.734 0.618 0.647 0.673 0.686
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.595 0.605 0.768 0.641 0.664 0.686 0.683
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.599 0.618 0.795 0.670 0.686 0.690 0.657
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.624 0.605 0.768 0.652 0.680 0.684 0.698
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.630 0.624 0.798 0.676 0.698 0.698 0.694

Table 14: Absolute Pearson correlation (∣r∣|r|∣r∣) and Kendall correlation (τ\tauτ) with segment-level human judgments on WMT17 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Setting Metric en-cs en-de en-fi en-lv en-ru en-tr en-zh
32K 3K 3K 3K 560 247 560
τ\tauτ τ\tauτ τ\tauτ τ\tauτ ∣r∣|r|∣r∣ τ\tauτ ∣r∣|r|∣r∣
Unsupervised AutoDA 0.041 0.099 0.204 0.130 0.511 0.409 0.609
AutoDA-TECTO 0.336 - - - - - -
chrF 0.376 0.336 0.503 0.420 0.605 0.466 0.608
chrF+ 0.377 0.325 0.514 0.421 0.609 0.474 -
chrF++ 0.368 0.328 0.484 0.417 0.604 0.466 0.602
MEANT 2.0 - 0.350 - - - - 0.727
MEANT 2.0-nosrl 0.395 0.324 0.565 0.425 0.636 0.482 0.705
SentBLEU 0.274 0.269 0.446 0.259 0.468 0.377 0.642
TreeAggreg 0.361 0.305 0.509 0.383 0.535 0.441 0.566
Supervised BEER 0.398 0.336 0.557 0.420 0.569 0.490 0.622
BLEND - - - - 0.578 - -
BLEU2VEC 0.305 0.313 0.503 0.315 0.472 0.425 -
NGRAM2VEC - - 0.486 0.317 - - -
Pre-Trained PBERT–MultiP_{\text{BERT--{Multi}}}PBERT–Multi​ 0.412 0.364 0.561 0.435 0.606 0.579 0.759
RBERT–MultiR_{\text{BERT--{Multi}}}RBERT–Multi​ 0.443 0.430 0.587 0.480 0.663 0.571 0.804
FBERT–MultiF_{\text{BERT--{Multi}}}FBERT–Multi​ 0.440 0.404 0.587 0.466 0.653 0.587 0.806
PBERT–MultiP_{\text{BERT--{Multi}}}PBERT–Multi​ (idf) 0.411 0.328 0.568 0.444 0.616 0.555 0.741
RBERT–MultiR_{\text{BERT--{Multi}}}RBERT–Multi​ (idf) 0.449 0.416 0.591 0.479 0.665 0.579 0.796
FBERT–MultiF_{\text{BERT--{Multi}}}FBERT–Multi​ (idf) 0.447 0.379 0.588 0.470 0.657 0.571 0.793
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.406 0.383 0.553 0.423 0.562 0.611 0.722
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.446 0.436 0.587 0.458 0.626 0.652 0.779
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.444 0.424 0.577 0.456 0.613 0.628 0.778
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.419 0.367 0.557 0.427 0.571 0.595 0.719
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.450 0.424 0.592 0.464 0.632 0.644 0.770
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.448 0.419 0.580 0.459 0.617 0.644 0.771

表 14:与 WMT17 从英语出发译文的片段级人工判断的绝对皮尔逊相关(∣r∣|r|∣r∣)和肯德尔相关(τ\tauτ)。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了样本数。

设置 指标 en-cs en-de en-fi en-lv en-ru en-tr en-zh
32K 3K 3K 3K 560 247 560
τ\tauτ τ\tauτ τ\tauτ τ\tauτ ∣r∣|r|∣r∣ τ\tauτ ∣r∣|r|∣r∣
无监督 AutoDA 0.041 0.099 0.204 0.130 0.511 0.409 0.609
AutoDA-TECTO 0.336 - - - - - -
chrF 0.376 0.336 0.503 0.420 0.605 0.466 0.608
chrF+ 0.377 0.325 0.514 0.421 0.609 0.474 -
chrF++ 0.368 0.328 0.484 0.417 0.604 0.466 0.602
MEANT 2.0 - 0.350 - - - - 0.727
MEANT 2.0-nosrl 0.395 0.324 0.565 0.425 0.636 0.482 0.705
SentBLEU 0.274 0.269 0.446 0.259 0.468 0.377 0.642
TreeAggreg 0.361 0.305 0.509 0.383 0.535 0.441 0.566
Supervised BEER 0.398 0.336 0.557 0.420 0.569 0.490 0.622
BLEND - - - - 0.578 - -
BLEU2VEC 0.305 0.313 0.503 0.315 0.472 0.425 -
NGRAM2VEC - - 0.486 0.317 - - -
Pre-Trained PBERT–MultiP_{\text{BERT--{Multi}}}PBERT–Multi​ 0.412 0.364 0.561 0.435 0.606 0.579 0.759
RBERT–MultiR_{\text{BERT--{Multi}}}RBERT–Multi​ 0.443 0.430 0.587 0.480 0.663 0.571 0.804
FBERT–MultiF_{\text{BERT--{Multi}}}FBERT–Multi​ 0.440 0.404 0.587 0.466 0.653 0.587 0.806
PBERT–MultiP_{\text{BERT--{Multi}}}PBERT–Multi​ (idf) 0.411 0.328 0.568 0.444 0.616 0.555 0.741
RBERT–MultiR_{\text{BERT--{Multi}}}RBERT–Multi​ (idf) 0.449 0.416 0.591 0.479 0.665 0.579 0.796
FBERT–MultiF_{\text{BERT--{Multi}}}FBERT–Multi​ (idf) 0.447 0.379 0.588 0.470 0.657 0.571 0.793
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.406 0.383 0.553 0.423 0.562 0.611 0.722
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.446 0.436 0.587 0.458 0.626 0.652 0.779
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.444 0.424 0.577 0.456 0.613 0.628 0.778
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.419 0.367 0.557 0.427 0.571 0.595 0.719
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.450 0.424 0.592 0.464 0.632 0.644 0.770
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.448 0.419 0.580 0.459 0.617 0.644 0.771

Table 15: Absolute Pearson correlations with system-level human judgments on WMT17 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Setting Metric cs-en de-en fi-en lv-en ru-en tr-en zh-en
4 11 6 9 9 10 16
Unsupervised BLEU 0.971 0.923 0.903 0.979 0.912 0.976 0.864
CDER 0.989 0.930 0.927 0.985 0.922 0.973 0.904
CharacTER 0.972 0.974 0.946 0.932 0.958 0.949 0.799
chrF 0.939 0.968 0.938 0.968 0.952 0.944 0.859
chrF++ 0.940 0.965 0.927 0.973 0.945 0.960 0.880
MEANT 2.0 0.926 0.950 0.941 0.970 0.962 0.932 0.838
MEANT 2.0-nosrl 0.902 0.936 0.933 0.963 0.960 0.896 0.800
NIST 1.000 0.931 0.931 0.960 0.912 0.971 0.849
PER 0.968 0.951 0.896 0.962 0.911 0.932 0.877
TER 0.989 0.906 0.952 0.971 0.912 0.954 0.847
TreeAggreg 0.983 0.920 0.977 0.986 0.918 0.987 0.861
UHH_TSKM 0.996 0.937 0.921 0.990 0.914 0.987 0.902
WER 0.987 0.896 0.948 0.969 0.907 0.925 0.839
Supervised AutoDA 0.438 0.959 0.925 0.973 0.907 0.916 0.734
BEER 0.972 0.960 0.955 0.978 0.936 0.972 0.902
BLEND 0.968 0.976 0.958 0.979 0.964 0.984 0.894
BLEU2VEC 0.989 0.936 0.888 0.966 0.907 0.961 0.886
NGRAM2VEC 0.984 0.935 0.890 0.963 0.907 0.955 0.880
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.975 0.936 0.991 0.993 0.918 0.981 0.892
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.995 0.975 0.944 0.978 0.953 0.991 0.975
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.987 0.961 0.979 0.991 0.937 0.991 0.953
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.983 0.937 0.998 0.992 0.939 0.985 0.878
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.997 0.981 0.962 0.968 0.977 0.985 0.949
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.992 0.967 0.995 0.992 0.960 0.996 0.951
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.982 0.926 0.990 0.987 0.916 0.970 0.899
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.999 0.979 0.950 0.982 0.957 0.977 0.985
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.994 0.957 0.986 0.994 0.938 0.980 0.960
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.989 0.936 0.992 0.979 0.931 0.976 0.892
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.999 0.987 0.962 0.980 0.975 0.979 0.973
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.997 0.968 0.995 0.997 0.956 0.989 0.963
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.981 0.937 0.991 0.996 0.921 0.987 0.905
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.996 0.975 0.953 0.985 0.954 0.992 0.977
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.990 0.960 0.981 0.995 0.938 0.992 0.957
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.986 0.938 0.998 0.995 0.939 0.994 0.897
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.997 0.982 0.967 0.979 0.974 0.992 0.966
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.994 0.965 0.993 0.995 0.958 0.998 0.959
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.987 0.930 0.984 0.966 0.916 0.963 0.955
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.999 0.982 0.947 0.979 0.956 0.986 0.984
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.996 0.961 0.993 0.993 0.937 0.983 0.982
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.990 0.938 0.980 0.956 0.929 0.967 0.962
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.998 0.987 0.963 0.979 0.971 0.986 0.974
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.996 0.970 0.999 0.994 0.952 0.989 0.982
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.989 0.948 0.984 0.949 0.927 0.960 0.967
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.998 0.988 0.957 0.983 0.969 0.982 0.984
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.996 0.973 0.997 0.991 0.949 0.984 0.987
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.989 0.959 0.975 0.935 0.944 0.968 0.974
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.995 0.991 0.962 0.979 0.981 0.981 0.970
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.996 0.982 0.998 0.991 0.965 0.991 0.984
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.994 0.963 0.995 0.990 0.944 0.981 0.974
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.995 0.991 0.962 0.981 0.973 0.985 0.984
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.999 0.982 0.992 0.996 0.961 0.988 0.989
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.995 0.970 0.997 0.985 0.955 0.988 0.979
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.994 0.992 0.967 0.977 0.983 0.988 0.972
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.999 0.989 0.996 0.997 0.972 0.994 0.987
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.988 0.938 0.993 0.993 0.914 0.974 0.960
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.999 0.978 0.956 0.977 0.946 0.981 0.980
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.996 0.963 0.986 0.991 0.932 0.981 0.978
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.992 0.951 0.998 0.996 0.930 0.982 0.939
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.999 0.986 0.968 0.973 0.964 0.987 0.955
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.998 0.974 0.996 0.994 0.950 0.990 0.970
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.991 0.944 0.996 0.995 0.924 0.982 0.943
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.996 0.981 0.945 0.971 0.961 0.986 0.958
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.999 0.969 0.986 0.992 0.945 0.992 0.961
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.995 0.955 0.999 0.996 0.941 0.985 0.937
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.993 0.985 0.951 0.960 0.975 0.974 0.910
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 1.000 0.978 0.994 0.993 0.962 0.994 0.954
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.983 0.933 0.994 0.989 0.918 0.973 0.928
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.998 0.978 0.949 0.983 0.957 0.985 0.972
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.994 0.960 0.985 0.995 0.938 0.984 0.964
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.986 0.940 0.997 0.992 0.939 0.979 0.916
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.999 0.983 0.966 0.980 0.975 0.991 0.952
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.995 0.967 0.996 0.998 0.959 0.993 0.958

表 15:与 WMT17 到英语译文的系统级人工判断的绝对皮尔逊相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了系统数。

设置 指标 cs-en de-en fi-en lv-en ru-en tr-en zh-en
4 11 6 9 9 10 16
无监督 BLEU 0.971 0.923 0.903 0.979 0.912 0.976 0.864
CDER 0.989 0.930 0.927 0.985 0.922 0.973 0.904
CharacTER 0.972 0.974 0.946 0.932 0.958 0.949 0.799
chrF 0.939 0.968 0.938 0.968 0.952 0.944 0.859
chrF++ 0.940 0.965 0.927 0.973 0.945 0.960 0.880
MEANT 2.0 0.926 0.950 0.941 0.970 0.962 0.932 0.838
MEANT 2.0-nosrl 0.902 0.936 0.933 0.963 0.960 0.896 0.800
NIST 1.000 0.931 0.931 0.960 0.912 0.971 0.849
PER 0.968 0.951 0.896 0.962 0.911 0.932 0.877
TER 0.989 0.906 0.952 0.971 0.912 0.954 0.847
TreeAggreg 0.983 0.920 0.977 0.986 0.918 0.987 0.861
UHH_TSKM 0.996 0.937 0.921 0.990 0.914 0.987 0.902
WER 0.987 0.896 0.948 0.969 0.907 0.925 0.839
Supervised AutoDA 0.438 0.959 0.925 0.973 0.907 0.916 0.734
BEER 0.972 0.960 0.955 0.978 0.936 0.972 0.902
BLEND 0.968 0.976 0.958 0.979 0.964 0.984 0.894
BLEU2VEC 0.989 0.936 0.888 0.966 0.907 0.961 0.886
NGRAM2VEC 0.984 0.935 0.890 0.963 0.907 0.955 0.880
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.975 0.936 0.991 0.993 0.918 0.981 0.892
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.995 0.975 0.944 0.978 0.953 0.991 0.975
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.987 0.961 0.979 0.991 0.937 0.991 0.953
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.983 0.937 0.998 0.992 0.939 0.985 0.878
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.997 0.981 0.962 0.968 0.977 0.985 0.949
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.992 0.967 0.995 0.992 0.960 0.996 0.951
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.982 0.926 0.990 0.987 0.916 0.970 0.899
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.999 0.979 0.950 0.982 0.957 0.977 0.985
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.994 0.957 0.986 0.994 0.938 0.980 0.960
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.989 0.936 0.992 0.979 0.931 0.976 0.892
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.999 0.987 0.962 0.980 0.975 0.979 0.973
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.997 0.968 0.995 0.997 0.956 0.989 0.963
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.981 0.937 0.991 0.996 0.921 0.987 0.905
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.996 0.975 0.953 0.985 0.954 0.992 0.977
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.990 0.960 0.981 0.995 0.938 0.992 0.957
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.986 0.938 0.998 0.995 0.939 0.994 0.897
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.997 0.982 0.967 0.979 0.974 0.992 0.966
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.994 0.965 0.993 0.995 0.958 0.998 0.959
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.987 0.930 0.984 0.966 0.916 0.963 0.955
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.999 0.982 0.947 0.979 0.956 0.986 0.984
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.996 0.961 0.993 0.993 0.937 0.983 0.982
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.990 0.938 0.980 0.956 0.929 0.967 0.962
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.998 0.987 0.963 0.979 0.971 0.986 0.974
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.996 0.970 0.999 0.994 0.952 0.989 0.982
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.989 0.948 0.984 0.949 0.927 0.960 0.967
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.998 0.988 0.957 0.983 0.969 0.982 0.984
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.996 0.973 0.997 0.991 0.949 0.984 0.987
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.989 0.959 0.975 0.935 0.944 0.968 0.974
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.995 0.991 0.962 0.979 0.981 0.981 0.970
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.996 0.982 0.998 0.991 0.965 0.991 0.984
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.994 0.963 0.995 0.990 0.944 0.981 0.974
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.995 0.991 0.962 0.981 0.973 0.985 0.984
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.999 0.982 0.992 0.996 0.961 0.988 0.989
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.995 0.970 0.997 0.985 0.955 0.988 0.979
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.994 0.992 0.967 0.977 0.983 0.988 0.972
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.999 0.989 0.996 0.997 0.972 0.994 0.987
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.988 0.938 0.993 0.993 0.914 0.974 0.960
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.999 0.978 0.956 0.977 0.946 0.981 0.980
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.996 0.963 0.986 0.991 0.932 0.981 0.978
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.992 0.951 0.998 0.996 0.930 0.982 0.939
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.999 0.986 0.968 0.973 0.964 0.987 0.955
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.998 0.974 0.996 0.994 0.950 0.990 0.970
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.991 0.944 0.996 0.995 0.924 0.982 0.943
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.996 0.981 0.945 0.971 0.961 0.986 0.958
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.999 0.969 0.986 0.992 0.945 0.992 0.961
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.995 0.955 0.999 0.996 0.941 0.985 0.937
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.993 0.985 0.951 0.960 0.975 0.974 0.910
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 1.000 0.978 0.994 0.993 0.962 0.994 0.954
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.983 0.933 0.994 0.989 0.918 0.973 0.928
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.998 0.978 0.949 0.983 0.957 0.985 0.972
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.994 0.960 0.985 0.995 0.938 0.984 0.964
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.986 0.940 0.997 0.992 0.939 0.979 0.916
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.999 0.983 0.966 0.980 0.975 0.991 0.952
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.995 0.967 0.996 0.998 0.959 0.993 0.958

Table 16: Absolute Pearson correlations with system-level human judgments on WMT17 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Setting Metric en-cs en-de en-lv en-ru en-tr en-zh
14 16 17 9 8 11
Unsupervised BLEU 0.956 0.804 0.866 0.898 0.924 –
CDER 0.968 0.813 0.930 0.924 0.957 –
CharacTER 0.981 0.938 0.897 0.939 0.975 0.933
chrF 0.976 0.863 0.955 0.950 0.991 0.976
chrF++ 0.974 0.852 0.956 0.945 0.986 0.976
MEANT 2.0 – 0.858 – – – 0.956
MEANT 2.0-nosrl 0.976 0.770 0.959 0.957 0.991 0.943
NIST 0.962 0.769 0.935 0.920 0.986 –
PER 0.954 0.687 0.851 0.887 0.963 –
TER 0.955 0.796 0.909 0.933 0.967 –
TreeAggreg 0.947 0.773 0.927 0.921 0.983 0.938
UHH_TSKM – – – – – –
WER 0.954 0.802 0.906 0.934 0.956 –
Supervised AutoDA 0.975 0.603 0.729 0.850 0.601 0.976
BEER 0.970 0.842 0.930 0.944 0.980 0.914
BLEND – – – 0.953 – –
BLEU2VEC 0.963 0.810 0.859 0.903 0.911 –
NGRAM2VEC – – 0.862 – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.959 0.798 0.960 0.946 0.981 0.970
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.982 0.909 0.957 0.980 0.979 0.994
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.976 0.859 0.959 0.966 0.980 0.992
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.963 0.760 0.960 0.947 0.984 0.971
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.985 0.907 0.955 0.981 0.984 0.982
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.979 0.841 0.958 0.968 0.984 0.991
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.967 0.825 0.965 0.953 0.974 0.977
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.980 0.902 0.965 0.982 0.977 0.979
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.979 0.868 0.969 0.971 0.976 0.986
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.968 0.809 0.965 0.955 0.980 0.975
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.981 0.894 0.964 0.984 0.983 0.968
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.979 0.856 0.966 0.973 0.982 0.979

表 16:与 WMT17 从英语出发译文的系统级人工判断的绝对皮尔逊相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了系统数。

设置 指标 en-cs en-de en-lv en-ru en-tr en-zh
14 16 17 9 8 11
无监督 BLEU 0.956 0.804 0.866 0.898 0.924 –
CDER 0.968 0.813 0.930 0.924 0.957 –
CharacTER 0.981 0.938 0.897 0.939 0.975 0.933
chrF 0.976 0.863 0.955 0.950 0.991 0.976
chrF++ 0.974 0.852 0.956 0.945 0.986 0.976
MEANT 2.0 – 0.858 – – – 0.956
MEANT 2.0-nosrl 0.976 0.770 0.959 0.957 0.991 0.943
NIST 0.962 0.769 0.935 0.920 0.986 –
PER 0.954 0.687 0.851 0.887 0.963 –
TER 0.955 0.796 0.909 0.933 0.967 –
TreeAggreg 0.947 0.773 0.927 0.921 0.983 0.938
UHH_TSKM – – – – – –
WER 0.954 0.802 0.906 0.934 0.956 –
Supervised AutoDA 0.975 0.603 0.729 0.850 0.601 0.976
BEER 0.970 0.842 0.930 0.944 0.980 0.914
BLEND – – – 0.953 – –
BLEU2VEC 0.963 0.810 0.859 0.903 0.911 –
NGRAM2VEC – – 0.862 – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.959 0.798 0.960 0.946 0.981 0.970
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.982 0.909 0.957 0.980 0.979 0.994
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.976 0.859 0.959 0.966 0.980 0.992
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.963 0.760 0.960 0.947 0.984 0.971
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.985 0.907 0.955 0.981 0.984 0.982
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.979 0.841 0.958 0.968 0.984 0.991
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.967 0.825 0.965 0.953 0.974 0.977
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.980 0.902 0.965 0.982 0.977 0.979
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.979 0.868 0.969 0.971 0.976 0.986
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.968 0.809 0.965 0.955 0.980 0.975
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.981 0.894 0.964 0.984 0.983 0.968
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.979 0.856 0.966 0.973 0.982 0.979

Table 17: Kendall correlations with segment-level human judgments on WMT18 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Setting Metric cs-en de-en et-en fi-en ru-en tr-en zh-en
5K 78K 57K 16K 10K 9K 33K
Unsupervised CharacTER 0.256 0.450 0.286 0.185 0.244 0.172 0.202
ITER 0.198 0.396 0.235 0.128 0.139 -0.029 0.144
Meteor++ 0.270 0.457 0.329 0.207 0.253 0.204 0.179
SentBLEU 0.233 0.415 0.285 0.154 0.228 0.145 0.178
UHH_TSKM 0.274 0.436 0.300 0.168 0.235 0.154 0.151
YiSi-0 0.301 0.474 0.330 0.225 0.294 0.215 0.205
YiSi-1 0.319 0.488 0.351 0.231 0.300 0.234 0.211
YiSi-1 srl 0.317 0.483 0.345 0.237 0.306 0.233 0.209
Supervised BEER 0.295 0.481 0.341 0.232 0.288 0.229 0.214
BLEND 0.322 0.492 0.354 0.226 0.290 0.232 0.217
RUSE 0.347 0.498 0.368 0.273 0.311 0.259 0.218
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.349 0.522 0.373 0.264 0.325 0.264 0.232
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.370 0.528 0.378 0.291 0.333 0.257 0.244
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.373 0.531 0.385 0.287 0.341 0.266 0.243
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.352 0.524 0.382 0.27 0.326 0.277 0.235
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.368 0.536 0.388 0.300 0.340 0.284 0.244
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.375 0.535 0.393 0.294 0.339 0.289 0.243
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.343 0.520 0.365 0.247 0.333 0.25 0.227
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.370 0.524 0.373 0.277 0.34 0.261 0.244
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.366 0.529 0.377 0.271 0.342 0.263 0.242
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.348 0.522 0.371 0.25 0.318 0.256 0.224
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.379 0.531 0.383 0.285 0.339 0.266 0.242
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.373 0.534 0.383 0.274 0.342 0.275 0.242
PBERT–LARGEP_{\text{BERT--{LARGE}}}PBERT–LARGE​ 0.361 0.529 0.380 0.276 0.340 0.266 0.241
RBERT–LARGER_{\text{BERT--{LARGE}}}RBERT–LARGE​ 0.386 0.532 0.386 0.297 0.347 0.268 0.247
FBERT–LARGEF_{\text{BERT--{LARGE}}}FBERT–LARGE​ 0.402 0.537 0.390 0.296 0.344 0.274 0.252
PBERT–LARGEP_{\text{BERT--{LARGE}}}PBERT–LARGE​ (idf) 0.377 0.532 0.390 0.287 0.342 0.292 0.246
RBERT–LARGER_{\text{BERT--{LARGE}}}RBERT–LARGE​ (idf) 0.386 0.544 0.396 0.308 0.356 0.287 0.251
FBERT–LARGEF_{\text{BERT--{LARGE}}}FBERT–LARGE​ (idf) 0.388 0.545 0.399 0.309 0.358 0.300 0.257
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.368 0.53 0.371 0.274 0.318 0.265 0.235
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.383 0.536 0.376 0.283 0.336 0.253 0.245
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.391 0.540 0.383 0.273 0.339 0.270 0.249
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.379 0.528 0.372 0.261 0.314 0.265 0.232
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.389 0.539 0.384 0.288 0.332 0.267 0.245
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.400 0.540 0.385 0.274 0.337 0.277 0.247
PRoBERTa–LARGEP_{\text{RoBERTa--{LARGE}}}PRoBERTa–LARGE​ 0.387 0.541 0.389 0.283 0.345 0.280 0.248
RRoBERTa–LARGER_{\text{RoBERTa--{LARGE}}}RRoBERTa–LARGE​ 0.388 0.546 0.391 0.304 0.343 0.290 0.255
FRoBERTa–LARGEF_{\text{RoBERTa--{LARGE}}}FRoBERTa–LARGE​ 0.404 0.550 0.397 0.296 0.353 0.292 0.264
PRoBERTa–LARGEP_{\text{RoBERTa--{LARGE}}}PRoBERTa–LARGE​ (idf) 0.391 0.540 0.387 0.280 0.334 0.284 0.252
RRoBERTa–LARGER_{\text{RoBERTa--{LARGE}}}RRoBERTa–LARGE​ (idf) 0.386 0.548 0.394 0.305 0.338 0.295 0.252
FRoBERTa–LARGEF_{\text{RoBERTa--{LARGE}}}FRoBERTa–LARGE​ (idf) 0.408 0.550 0.395 0.293 0.346 0.296 0.260
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.397 0.549 0.396 0.299 0.351 0.295 0.253
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.404 0.553 0.393 0.313 0.351 0.279 0.253
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.418 0.557 0.402 0.312 0.362 0.290 0.258
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.414 0.552 0.399 0.301 0.349 0.306 0.249
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.412 0.555 0.400 0.316 0.357 0.289 0.258
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.417 0.559 0.403 0.309 0.357 0.307 0.258
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ 0.335 0.514 0.359 0.243 0.308 0.247 0.232
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ 0.351 0.515 0.362 0.261 0.311 0.227 0.232
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ 0.351 0.517 0.365 0.257 0.315 0.25 0.237
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ (idf) 0.339 0.516 0.366 0.258 0.307 0.261 0.236
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ (idf) 0.364 0.521 0.371 0.268 0.317 0.242 0.238
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ (idf) 0.355 0.524 0.374 0.265 0.320 0.261 0.241
PXL-NET–LARGEP_{\text{XL-NET--{LARGE}}}PXL-NET–LARGE​ 0.344 0.522 0.371 0.252 0.316 0.264 0.233
RXL-NET–LARGER_{\text{XL-NET--{LARGE}}}RXL-NET–LARGE​ 0.358 0.524 0.374 0.275 0.332 0.249 0.239
FXL-NET–LARGEF_{\text{XL-NET--{LARGE}}}FXL-NET–LARGE​ 0.357 0.530 0.380 0.265 0.334 0.263 0.238
PXL-NET–LARGEP_{\text{XL-NET--{LARGE}}}PXL-NET–LARGE​ (idf) 0.348 0.520 0.373 0.260 0.319 0.265 0.235
RXL-NET–LARGER_{\text{XL-NET--{LARGE}}}RXL-NET–LARGE​ (idf) 0.366 0.529 0.378 0.278 0.331 0.266 0.241
FXL-NET–LARGEF_{\text{XL-NET--{LARGE}}}FXL-NET–LARGE​ (idf) 0.375 0.530 0.382 0.274 0.332 0.274 0.240
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.349 0.516 0.366 0.244 0.310 0.259 0.233
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.358 0.518 0.364 0.264 0.320 0.244 0.237
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.358 0.525 0.373 0.259 0.322 0.258 0.238
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.355 0.527 0.374 0.254 0.311 0.28 0.238
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.362 0.528 0.376 0.274 0.333 0.26 0.24
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.367 0.531 0.382 0.273 0.330 0.275 0.246

表 17:与 WMT18 到英语译文的片段级人工判断的肯德尔相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了样本数。

设置 指标 cs-en de-en et-en fi-en ru-en tr-en zh-en
5K 78K 57K 16K 10K 9K 33K
无监督 CharacTER 0.256 0.450 0.286 0.185 0.244 0.172 0.202
ITER 0.198 0.396 0.235 0.128 0.139 -0.029 0.144
Meteor++ 0.270 0.457 0.329 0.207 0.253 0.204 0.179
SentBLEU 0.233 0.415 0.285 0.154 0.228 0.145 0.178
UHH_TSKM 0.274 0.436 0.300 0.168 0.235 0.154 0.151
YiSi-0 0.301 0.474 0.330 0.225 0.294 0.215 0.205
YiSi-1 0.319 0.488 0.351 0.231 0.300 0.234 0.211
YiSi-1 srl 0.317 0.483 0.345 0.237 0.306 0.233 0.209
Supervised BEER 0.295 0.481 0.341 0.232 0.288 0.229 0.214
BLEND 0.322 0.492 0.354 0.226 0.290 0.232 0.217
RUSE 0.347 0.498 0.368 0.273 0.311 0.259 0.218
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.349 0.522 0.373 0.264 0.325 0.264 0.232
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.370 0.528 0.378 0.291 0.333 0.257 0.244
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.373 0.531 0.385 0.287 0.341 0.266 0.243
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.352 0.524 0.382 0.27 0.326 0.277 0.235
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.368 0.536 0.388 0.300 0.340 0.284 0.244
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.375 0.535 0.393 0.294 0.339 0.289 0.243
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.343 0.520 0.365 0.247 0.333 0.25 0.227
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.370 0.524 0.373 0.277 0.34 0.261 0.244
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.366 0.529 0.377 0.271 0.342 0.263 0.242
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.348 0.522 0.371 0.25 0.318 0.256 0.224
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.379 0.531 0.383 0.285 0.339 0.266 0.242
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.373 0.534 0.383 0.274 0.342 0.275 0.242
PBERT–LARGEP_{\text{BERT--{LARGE}}}PBERT–LARGE​ 0.361 0.529 0.380 0.276 0.340 0.266 0.241
RBERT–LARGER_{\text{BERT--{LARGE}}}RBERT–LARGE​ 0.386 0.532 0.386 0.297 0.347 0.268 0.247
FBERT–LARGEF_{\text{BERT--{LARGE}}}FBERT–LARGE​ 0.402 0.537 0.390 0.296 0.344 0.274 0.252
PBERT–LARGEP_{\text{BERT--{LARGE}}}PBERT–LARGE​ (idf) 0.377 0.532 0.390 0.287 0.342 0.292 0.246
RBERT–LARGER_{\text{BERT--{LARGE}}}RBERT–LARGE​ (idf) 0.386 0.544 0.396 0.308 0.356 0.287 0.251
FBERT–LARGEF_{\text{BERT--{LARGE}}}FBERT–LARGE​ (idf) 0.388 0.545 0.399 0.309 0.358 0.300 0.257
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.368 0.53 0.371 0.274 0.318 0.265 0.235
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.383 0.536 0.376 0.283 0.336 0.253 0.245
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.391 0.540 0.383 0.273 0.339 0.270 0.249
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.379 0.528 0.372 0.261 0.314 0.265 0.232
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.389 0.539 0.384 0.288 0.332 0.267 0.245
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.400 0.540 0.385 0.274 0.337 0.277 0.247
PRoBERTa–LARGEP_{\text{RoBERTa--{LARGE}}}PRoBERTa–LARGE​ 0.387 0.541 0.389 0.283 0.345 0.280 0.248
RRoBERTa–LARGER_{\text{RoBERTa--{LARGE}}}RRoBERTa–LARGE​ 0.388 0.546 0.391 0.304 0.343 0.290 0.255
FRoBERTa–LARGEF_{\text{RoBERTa--{LARGE}}}FRoBERTa–LARGE​ 0.404 0.550 0.397 0.296 0.353 0.292 0.264
PRoBERTa–LARGEP_{\text{RoBERTa--{LARGE}}}PRoBERTa–LARGE​ (idf) 0.391 0.540 0.387 0.280 0.334 0.284 0.252
RRoBERTa–LARGER_{\text{RoBERTa--{LARGE}}}RRoBERTa–LARGE​ (idf) 0.386 0.548 0.394 0.305 0.338 0.295 0.252
FRoBERTa–LARGEF_{\text{RoBERTa--{LARGE}}}FRoBERTa–LARGE​ (idf) 0.408 0.550 0.395 0.293 0.346 0.296 0.260
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.397 0.549 0.396 0.299 0.351 0.295 0.253
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.404 0.553 0.393 0.313 0.351 0.279 0.253
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.418 0.557 0.402 0.312 0.362 0.290 0.258
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.414 0.552 0.399 0.301 0.349 0.306 0.249
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.412 0.555 0.400 0.316 0.357 0.289 0.258
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.417 0.559 0.403 0.309 0.357 0.307 0.258
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ 0.335 0.514 0.359 0.243 0.308 0.247 0.232
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ 0.351 0.515 0.362 0.261 0.311 0.227 0.232
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ 0.351 0.517 0.365 0.257 0.315 0.25 0.237
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ (idf) 0.339 0.516 0.366 0.258 0.307 0.261 0.236
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ (idf) 0.364 0.521 0.371 0.268 0.317 0.242 0.238
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ (idf) 0.355 0.524 0.374 0.265 0.320 0.261 0.241
PXL-NET–LARGEP_{\text{XL-NET--{LARGE}}}PXL-NET–LARGE​ 0.344 0.522 0.371 0.252 0.316 0.264 0.233
RXL-NET–LARGER_{\text{XL-NET--{LARGE}}}RXL-NET–LARGE​ 0.358 0.524 0.374 0.275 0.332 0.249 0.239
FXL-NET–LARGEF_{\text{XL-NET--{LARGE}}}FXL-NET–LARGE​ 0.357 0.530 0.380 0.265 0.334 0.263 0.238
PXL-NET–LARGEP_{\text{XL-NET--{LARGE}}}PXL-NET–LARGE​ (idf) 0.348 0.520 0.373 0.260 0.319 0.265 0.235
RXL-NET–LARGER_{\text{XL-NET--{LARGE}}}RXL-NET–LARGE​ (idf) 0.366 0.529 0.378 0.278 0.331 0.266 0.241
FXL-NET–LARGEF_{\text{XL-NET--{LARGE}}}FXL-NET–LARGE​ (idf) 0.375 0.530 0.382 0.274 0.332 0.274 0.240
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.349 0.516 0.366 0.244 0.310 0.259 0.233
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.358 0.518 0.364 0.264 0.320 0.244 0.237
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.358 0.525 0.373 0.259 0.322 0.258 0.238
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.355 0.527 0.374 0.254 0.311 0.28 0.238
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.362 0.528 0.376 0.274 0.333 0.26 0.24
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.367 0.531 0.382 0.273 0.330 0.275 0.246

Table 18: Kendall correlations with segment-level human judgments on WMT18 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of examples.

Setting Metric en-cs en-de en-et en-fi en-ru en-tr en-zh
5K 20K 32K 10K 22K 1K 29K
Unsupervised CharacTER 0.414 0.604 0.464 0.403 0.352 0.404 0.313
ITER 0.333 0.610 0.392 0.311 0.291 0.236 -
SentBLEU 0.389 0.620 0.414 0.355 0.330 0.261 0.311
YiSi-0 0.471 0.661 0.531 0.464 0.394 0.376 0.318
YiSi-1 0.496 0.691 0.546 0.504 0.407 0.418 0.323
YiSi-1 srl - 0.696 - - - - 0.310
Supervised BEER 0.518 0.686 0.558 0.511 0.403 0.374 0.302
BLEND - - - - 0.394 - -
Pre-Trained PBERT–MultiP_{\text{BERT--{Multi}}}PBERT–Multi​ 0.541 0.715 0.549 0.486 0.414 0.328 0.337
RBERT–MultiR_{\text{BERT--{Multi}}}RBERT–Multi​ 0.570 0.728 0.594 0.565 0.420 0.411 0.367
FBERT–MultiF_{\text{BERT--{Multi}}}FBERT–Multi​ 0.562 0.728 0.586 0.546 0.423 0.399 0.364
PBERT–MultiP_{\text{BERT--{Multi}}}PBERT–Multi​ (idf) 0.525 0.7 0.54 0.495 0.423 0.352 0.338
RBERT–MultiR_{\text{BERT--{Multi}}}RBERT–Multi​ (idf) 0.569 0.727 0.601 0.561 0.423 0.420 0.374
FBERT–MultiF_{\text{BERT--{Multi}}}FBERT–Multi​ (idf) 0.553 0.721 0.585 0.537 0.425 0.406 0.366
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.496 0.711 0.561 0.527 0.417 0.364 0.340
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.564 0.724 0.612 0.584 0.418 0.432 0.363
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.533 0.727 0.599 0.573 0.421 0.408 0.362
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.520 0.710 0.572 0.546 0.421 0.370 0.328
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.567 0.722 0.609 0.587 0.420 0.439 0.365
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.554 0.724 0.601 0.584 0.422 0.389 0.355

表 18:与 WMT18 从英语出发译文的片段级人工判断的肯德尔相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了样本数。

设置 指标 en-cs en-de en-et en-fi en-ru en-tr en-zh
5K 20K 32K 10K 22K 1K 29K
无监督 CharacTER 0.414 0.604 0.464 0.403 0.352 0.404 0.313
ITER 0.333 0.610 0.392 0.311 0.291 0.236 -
SentBLEU 0.389 0.620 0.414 0.355 0.330 0.261 0.311
YiSi-0 0.471 0.661 0.531 0.464 0.394 0.376 0.318
YiSi-1 0.496 0.691 0.546 0.504 0.407 0.418 0.323
YiSi-1 srl - 0.696 - - - - 0.310
Supervised BEER 0.518 0.686 0.558 0.511 0.403 0.374 0.302
BLEND - - - - 0.394 - -
Pre-Trained PBERT–MultiP_{\text{BERT--{Multi}}}PBERT–Multi​ 0.541 0.715 0.549 0.486 0.414 0.328 0.337
RBERT–MultiR_{\text{BERT--{Multi}}}RBERT–Multi​ 0.570 0.728 0.594 0.565 0.420 0.411 0.367
FBERT–MultiF_{\text{BERT--{Multi}}}FBERT–Multi​ 0.562 0.728 0.586 0.546 0.423 0.399 0.364
PBERT–MultiP_{\text{BERT--{Multi}}}PBERT–Multi​ (idf) 0.525 0.7 0.54 0.495 0.423 0.352 0.338
RBERT–MultiR_{\text{BERT--{Multi}}}RBERT–Multi​ (idf) 0.569 0.727 0.601 0.561 0.423 0.420 0.374
FBERT–MultiF_{\text{BERT--{Multi}}}FBERT–Multi​ (idf) 0.553 0.721 0.585 0.537 0.425 0.406 0.366
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.496 0.711 0.561 0.527 0.417 0.364 0.340
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.564 0.724 0.612 0.584 0.418 0.432 0.363
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.533 0.727 0.599 0.573 0.421 0.408 0.362
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.520 0.710 0.572 0.546 0.421 0.370 0.328
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.567 0.722 0.609 0.587 0.420 0.439 0.365
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.554 0.724 0.601 0.584 0.422 0.389 0.355

Table 19: Absolute Pearson correlations with system-level human judgments on WMT18 to-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Setting Metric cs-en de-en et-en fi-en ru-en tr-en zh-en
5 16 14 9 8 5 14
Unsupervised BLEU 0.970 0.971 0.986 0.973 0.979 0.657 0.978
CDER 0.972 0.980 0.990 0.984 0.980 0.664 0.982
CharacTER 0.970 0.993 0.979 0.989 0.991 0.782 0.950
ITER 0.975 0.990 0.975 0.996 0.937 0.861 0.980
Meteor++ 0.945 0.991 0.978 0.971 0.995 0.864 0.962
NIST 0.954 0.984 0.983 0.975 0.973 0.970 0.968
PER 0.970 0.985 0.983 0.993 0.967 0.159 0.931
TER 0.950 0.970 0.990 0.968 0.970 0.533 0.975
UHH_TSKM 0.952 0.980 0.989 0.982 0.980 0.547 0.981
WER 0.951 0.961 0.991 0.961 0.968 0.041 0.975
YiSi-0 0.956 0.994 0.975 0.978 0.988 0.954 0.957
YiSi-1 0.950 0.992 0.979 0.973 0.991 0.958 0.951
YiSi-1 srl 0.965 0.995 0.981 0.977 0.992 0.869 0.962
Supervised BEER 0.958 0.994 0.985 0.991 0.982 0.870 0.976
BLEND 0.973 0.991 0.985 0.994 0.993 0.801 0.976
RUSE 0.981 0.997 0.990 0.991 0.988 0.853 0.981
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.965 0.995 0.986 0.973 0.976 0.941 0.974
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.994 0.991 0.979 0.992 0.991 0.067 0.988
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.982 0.994 0.983 0.986 0.985 0.949 0.984
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.961 0.993 0.987 0.988 0.976 0.984 0.973
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.996 0.994 0.977 0.995 0.995 0.874 0.983
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.981 0.995 0.984 0.995 0.988 0.994 0.981
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.957 0.994 0.989 0.953 0.976 0.798 0.977
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.992 0.994 0.983 0.988 0.993 0.707 0.990
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.975 0.995 0.987 0.975 0.986 0.526 0.986
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.957 0.997 0.989 0.967 0.975 0.894 0.980
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.991 0.997 0.981 0.994 0.993 0.052 0.987
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.975 0.998 0.987 0.985 0.987 0.784 0.987
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.978 0.992 0.987 0.971 0.977 0.920 0.978
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.997 0.990 0.985 0.990 0.992 0.098 0.990
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.989 0.992 0.987 0.983 0.985 0.784 0.986
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.977 0.992 0.988 0.986 0.976 0.980 0.977
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.998 0.993 0.983 0.996 0.995 0.809 0.986
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.989 0.993 0.986 0.993 0.987 0.976 0.984
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.970 0.995 0.991 0.998 0.976 0.796 0.980
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.996 0.996 0.982 0.998 0.994 0.477 0.991
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.984 0.997 0.989 0.999 0.987 0.280 0.989
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.966 0.993 0.991 0.994 0.977 0.880 0.984
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.995 0.998 0.981 0.998 0.995 0.230 0.989
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.981 0.998 0.989 0.997 0.988 0.741 0.990
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.980 0.998 0.990 0.995 0.982 0.791 0.981
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.998 0.997 0.986 0.997 0.995 0.054 0.990
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.990 0.999 0.990 0.998 0.990 0.499 0.988
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.972 0.997 0.993 0.985 0.982 0.920 0.983
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.996 0.997 0.984 0.997 0.995 0.578 0.989
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.985 0.999 0.992 0.992 0.991 0.826 0.989
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.989 0.998 0.994 0.998 0.985 0.908 0.982
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 1.000 0.996 0.988 0.996 0.995 0.097 0.991
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.996 0.998 0.992 0.998 0.992 0.665 0.989
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.986 0.998 0.994 0.993 0.986 0.989 0.985
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.999 0.997 0.986 0.997 0.993 0.633 0.990
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.995 0.998 0.991 0.996 0.993 0.963 0.990
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.970 0.996 0.986 0.990 0.979 0.739 0.982
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.994 0.997 0.979 0.995 0.994 0.795 0.990
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.983 0.997 0.983 0.993 0.987 0.505 0.988
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.968 0.998 0.986 0.990 0.978 0.923 0.982
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.993 0.998 0.978 0.996 0.994 0.439 0.988
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.981 0.999 0.984 0.995 0.989 0.722 0.988
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.969 0.998 0.986 0.995 0.979 0.880 0.981
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.995 0.997 0.977 0.997 0.995 0.430 0.988
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.983 0.998 0.983 0.997 0.988 0.713 0.988
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.963 0.996 0.986 0.995 0.978 0.939 0.979
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.992 0.997 0.975 0.993 0.996 0.531 0.982
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.978 0.997 0.983 0.996 0.990 0.886 0.984
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.965 0.996 0.990 0.978 0.980 0.946 0.981
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.990 0.995 0.984 0.996 0.996 0.286 0.987
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.978 0.997 0.988 0.990 0.989 0.576 0.987
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.960 0.996 0.990 0.987 0.980 0.989 0.981
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.991 0.997 0.983 0.996 0.998 0.612 0.985
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.976 0.998 0.988 0.994 0.992 0.943 0.985

表 19:与 WMT18 到英语译文的系统级人工判断的绝对皮尔逊相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了系统数。

设置 指标 cs-en de-en et-en fi-en ru-en tr-en zh-en
5 16 14 9 8 5 14
无监督 BLEU 0.970 0.971 0.986 0.973 0.979 0.657 0.978
CDER 0.972 0.980 0.990 0.984 0.980 0.664 0.982
CharacTER 0.970 0.993 0.979 0.989 0.991 0.782 0.950
ITER 0.975 0.990 0.975 0.996 0.937 0.861 0.980
Meteor++ 0.945 0.991 0.978 0.971 0.995 0.864 0.962
NIST 0.954 0.984 0.983 0.975 0.973 0.970 0.968
PER 0.970 0.985 0.983 0.993 0.967 0.159 0.931
TER 0.950 0.970 0.990 0.968 0.970 0.533 0.975
UHH_TSKM 0.952 0.980 0.989 0.982 0.980 0.547 0.981
WER 0.951 0.961 0.991 0.961 0.968 0.041 0.975
YiSi-0 0.956 0.994 0.975 0.978 0.988 0.954 0.957
YiSi-1 0.950 0.992 0.979 0.973 0.991 0.958 0.951
YiSi-1 srl 0.965 0.995 0.981 0.977 0.992 0.869 0.962
Supervised BEER 0.958 0.994 0.985 0.991 0.982 0.870 0.976
BLEND 0.973 0.991 0.985 0.994 0.993 0.801 0.976
RUSE 0.981 0.997 0.990 0.991 0.988 0.853 0.981
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.965 0.995 0.986 0.973 0.976 0.941 0.974
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.994 0.991 0.979 0.992 0.991 0.067 0.988
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.982 0.994 0.983 0.986 0.985 0.949 0.984
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.961 0.993 0.987 0.988 0.976 0.984 0.973
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.996 0.994 0.977 0.995 0.995 0.874 0.983
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.981 0.995 0.984 0.995 0.988 0.994 0.981
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.957 0.994 0.989 0.953 0.976 0.798 0.977
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.992 0.994 0.983 0.988 0.993 0.707 0.990
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.975 0.995 0.987 0.975 0.986 0.526 0.986
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.957 0.997 0.989 0.967 0.975 0.894 0.980
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.991 0.997 0.981 0.994 0.993 0.052 0.987
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.975 0.998 0.987 0.985 0.987 0.784 0.987
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.978 0.992 0.987 0.971 0.977 0.920 0.978
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.997 0.990 0.985 0.990 0.992 0.098 0.990
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.989 0.992 0.987 0.983 0.985 0.784 0.986
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.977 0.992 0.988 0.986 0.976 0.980 0.977
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.998 0.993 0.983 0.996 0.995 0.809 0.986
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.989 0.993 0.986 0.993 0.987 0.976 0.984
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.970 0.995 0.991 0.998 0.976 0.796 0.980
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.996 0.996 0.982 0.998 0.994 0.477 0.991
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.984 0.997 0.989 0.999 0.987 0.280 0.989
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.966 0.993 0.991 0.994 0.977 0.880 0.984
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.995 0.998 0.981 0.998 0.995 0.230 0.989
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.981 0.998 0.989 0.997 0.988 0.741 0.990
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.980 0.998 0.990 0.995 0.982 0.791 0.981
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.998 0.997 0.986 0.997 0.995 0.054 0.990
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.990 0.999 0.990 0.998 0.990 0.499 0.988
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.972 0.997 0.993 0.985 0.982 0.920 0.983
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.996 0.997 0.984 0.997 0.995 0.578 0.989
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.985 0.999 0.992 0.992 0.991 0.826 0.989
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.989 0.998 0.994 0.998 0.985 0.908 0.982
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 1.000 0.996 0.988 0.996 0.995 0.097 0.991
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.996 0.998 0.992 0.998 0.992 0.665 0.989
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.986 0.998 0.994 0.993 0.986 0.989 0.985
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.999 0.997 0.986 0.997 0.993 0.633 0.990
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.995 0.998 0.991 0.996 0.993 0.963 0.990
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.970 0.996 0.986 0.990 0.979 0.739 0.982
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.994 0.997 0.979 0.995 0.994 0.795 0.990
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.983 0.997 0.983 0.993 0.987 0.505 0.988
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.968 0.998 0.986 0.990 0.978 0.923 0.982
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.993 0.998 0.978 0.996 0.994 0.439 0.988
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.981 0.999 0.984 0.995 0.989 0.722 0.988
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.969 0.998 0.986 0.995 0.979 0.880 0.981
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.995 0.997 0.977 0.997 0.995 0.430 0.988
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.983 0.998 0.983 0.997 0.988 0.713 0.988
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.963 0.996 0.986 0.995 0.978 0.939 0.979
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.992 0.997 0.975 0.993 0.996 0.531 0.982
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.978 0.997 0.983 0.996 0.990 0.886 0.984
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.965 0.996 0.990 0.978 0.980 0.946 0.981
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.990 0.995 0.984 0.996 0.996 0.286 0.987
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.978 0.997 0.988 0.990 0.989 0.576 0.987
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.960 0.996 0.990 0.987 0.980 0.989 0.981
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.991 0.997 0.983 0.996 0.998 0.612 0.985
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.976 0.998 0.988 0.994 0.992 0.943 0.985

Table 20: Absolute Pearson correlations with system-level human judgments on WMT18 from-English translations. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Setting Metric en-cs en-de en-et en-fi en-ru en-tr en-zh
5 16 14 12 9 8 14
Unsupervised BLEU 0.995 0.981 0.975 0.962 0.983 0.826 0.947
CDER 0.997 0.986 0.984 0.964 0.984 0.861 0.961
CharacTER 0.993 0.989 0.956 0.974 0.983 0.833 0.983
ITER 0.915 0.984 0.981 0.973 0.975 0.865 –
Meteor++ – – – – – – –
NIST 0.999 0.986 0.983 0.949 0.990 0.902 0.950
PER 0.991 0.981 0.958 0.906 0.988 0.859 0.964
TER 0.997 0.988 0.981 0.942 0.987 0.867 0.963
UHH_TSKM – – – – – – –
WER 0.997 0.986 0.981 0.945 0.985 0.853 0.957
YiSi-0 0.973 0.985 0.968 0.944 0.990 0.990 0.957
YiSi-1 0.987 0.985 0.979 0.940 0.992 0.976 0.963
YiSi-1 srl – 0.990 – – – – 0.952
Supervised BEER 0.992 0.991 0.980 0.961 0.988 0.965 0.928
BLEND – – – – 0.988 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.994 0.988 0.981 0.957 0.990 0.935 0.954
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.997 0.990 0.980 0.980 0.989 0.879 0.976
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.997 0.989 0.982 0.972 0.990 0.908 0.967
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.992 0.986 0.974 0.954 0.991 0.969 0.954
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.997 0.993 0.982 0.982 0.992 0.901 0.984
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.995 0.990 0.981 0.972 0.991 0.941 0.973
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.984 0.992 0.993 0.972 0.993 0.962 0.965
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.991 0.992 0.992 0.989 0.992 0.895 0.983
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.988 0.993 0.993 0.986 0.993 0.935 0.976
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.982 0.992 0.994 0.975 0.993 0.968 0.964
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.993 0.993 0.991 0.989 0.993 0.911 0.986
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.989 0.993 0.994 0.985 0.993 0.945 0.979

表 20:与 WMT18 从英语出发译文的系统级人工判断的绝对皮尔逊相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了系统数。

设置 指标 en-cs en-de en-et en-fi en-ru en-tr en-zh
5 16 14 12 9 8 14
无监督 BLEU 0.995 0.981 0.975 0.962 0.983 0.826 0.947
CDER 0.997 0.986 0.984 0.964 0.984 0.861 0.961
CharacTER 0.993 0.989 0.956 0.974 0.983 0.833 0.983
ITER 0.915 0.984 0.981 0.973 0.975 0.865 –
Meteor++ – – – – – – –
NIST 0.999 0.986 0.983 0.949 0.990 0.902 0.950
PER 0.991 0.981 0.958 0.906 0.988 0.859 0.964
TER 0.997 0.988 0.981 0.942 0.987 0.867 0.963
UHH_TSKM – – – – – – –
WER 0.997 0.986 0.981 0.945 0.985 0.853 0.957
YiSi-0 0.973 0.985 0.968 0.944 0.990 0.990 0.957
YiSi-1 0.987 0.985 0.979 0.940 0.992 0.976 0.963
YiSi-1 srl – 0.990 – – – – 0.952
Supervised BEER 0.992 0.991 0.980 0.961 0.988 0.965 0.928
BLEND – – – – 0.988 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.994 0.988 0.981 0.957 0.990 0.935 0.954
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.997 0.990 0.980 0.980 0.989 0.879 0.976
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.997 0.989 0.982 0.972 0.990 0.908 0.967
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.992 0.986 0.974 0.954 0.991 0.969 0.954
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.997 0.993 0.982 0.982 0.992 0.901 0.984
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.995 0.990 0.981 0.972 0.991 0.941 0.973
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.984 0.992 0.993 0.972 0.993 0.962 0.965
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.991 0.992 0.992 0.989 0.992 0.895 0.983
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.988 0.993 0.993 0.986 0.993 0.935 0.976
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.982 0.992 0.994 0.975 0.993 0.968 0.964
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.993 0.993 0.991 0.989 0.993 0.911 0.986
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.989 0.993 0.994 0.985 0.993 0.945 0.979

Table 21: Absolute Pearson correlations with human judgments on WMT18 to-English language pairs for 10K hybrid systems. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Setting Metric cs-en de-en et-en fi-en ru-en tr-en zh-en
10K 10K 10K 10K 10K 10K 10K
Unsupervised BLEU 0.956 0.969 0.981 0.962 0.972 0.586 0.968
CDER 0.964 0.980 0.988 0.976 0.974 0.577 0.973
CharacTER 0.960 0.992 0.975 0.979 0.984 0.680 0.942
ITER 0.966 0.990 0.975 0.989 0.943 0.742 0.978
Meteor++ 0.937 0.990 0.975 0.962 0.989 0.787 0.954
NIST 0.942 0.982 0.980 0.965 0.965 0.862 0.959
PER 0.937 0.982 0.978 0.983 0.955 0.043 0.923
TER 0.942 0.970 0.988 0.960 0.963 0.450 0.967
UHH_TSKM 0.943 0.979 0.987 0.974 0.973 0.443 0.972
WER 0.942 0.961 0.989 0.953 0.962 0.072 0.967
YiSi-0 0.947 0.992 0.972 0.969 0.982 0.863 0.950
YiSi-1 0.942 0.991 0.976 0.964 0.985 0.881 0.943
YiSi-1 srl 0.957 0.994 0.978 0.968 0.986 0.785 0.954
Supervised BEER 0.950 0.993 0.983 0.982 0.976 0.723 0.968
BLEND 0.965 0.990 0.982 0.985 0.986 0.724 0.969
RUSE 0.974 0.996 0.988 0.983 0.982 0.780 0.973
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.954 0.992 0.984 0.980 0.970 0.917 0.965
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.988 0.994 0.974 0.987 0.988 0.801 0.975
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.973 0.994 0.981 0.987 0.982 0.924 0.973
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.957 0.994 0.983 0.966 0.970 0.875 0.966
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.986 0.990 0.976 0.984 0.984 0.019 0.980
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.974 0.993 0.980 0.978 0.978 0.853 0.976
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.949 0.995 0.986 0.960 0.969 0.832 0.972
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.983 0.997 0.979 0.986 0.986 0.099 0.980
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.967 0.997 0.984 0.978 0.981 0.722 0.979
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.949 0.994 0.986 0.946 0.969 0.743 0.969
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.984 0.994 0.980 0.980 0.986 0.541 0.982
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.967 0.995 0.984 0.968 0.979 0.464 0.978
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.969 0.991 0.985 0.979 0.970 0.915 0.969
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.990 0.993 0.980 0.988 0.988 0.745 0.978
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.982 0.993 0.984 0.986 0.981 0.909 0.976
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.970 0.991 0.984 0.963 0.971 0.858 0.970
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.989 0.990 0.982 0.982 0.985 0.047 0.982
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.981 0.991 0.984 0.976 0.978 0.722 0.978
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.959 0.992 0.988 0.986 0.971 0.809 0.976
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.987 0.997 0.978 0.989 0.988 0.238 0.981
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.973 0.997 0.987 0.989 0.982 0.674 0.982
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.963 0.994 0.988 0.989 0.970 0.711 0.972
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.988 0.996 0.979 0.989 0.987 0.353 0.983
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.976 0.997 0.986 0.990 0.980 0.277 0.980
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.965 0.995 0.990 0.976 0.976 0.846 0.975
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.989 0.997 0.982 0.989 0.988 0.540 0.981
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.978 0.998 0.989 0.983 0.985 0.760 0.981
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.972 0.997 0.988 0.986 0.976 0.686 0.973
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.990 0.996 0.983 0.989 0.989 0.096 0.982
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.982 0.998 0.988 0.989 0.983 0.453 0.980
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.978 0.997 0.991 0.984 0.980 0.914 0.977
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.991 0.996 0.984 0.989 0.987 0.566 0.982
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.987 0.998 0.989 0.988 0.986 0.873 0.982
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.982 0.998 0.992 0.990 0.978 0.822 0.974
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.992 0.996 0.985 0.988 0.988 0.022 0.983
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.989 0.998 0.990 0.990 0.985 0.583 0.980
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.960 0.997 0.984 0.982 0.972 0.849 0.974
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.985 0.997 0.975 0.988 0.988 0.303 0.980
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.974 0.998 0.981 0.986 0.982 0.628 0.980
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.962 0.995 0.983 0.982 0.972 0.657 0.974
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.986 0.996 0.976 0.987 0.987 0.666 0.982
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.975 0.996 0.980 0.985 0.981 0.259 0.980
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.955 0.995 0.983 0.986 0.972 0.875 0.970
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.984 0.996 0.972 0.984 0.989 0.491 0.975
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.971 0.996 0.980 0.987 0.984 0.821 0.976
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.961 0.997 0.983 0.987 0.973 0.816 0.973
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.987 0.996 0.975 0.989 0.988 0.320 0.981
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.976 0.997 0.980 0.989 0.982 0.623 0.980
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.953 0.995 0.988 0.979 0.974 0.918 0.972
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.983 0.996 0.980 0.988 0.991 0.561 0.977
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.969 0.997 0.986 0.986 0.985 0.869 0.977
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.957 0.996 0.987 0.970 0.974 0.862 0.973
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.982 0.995 0.981 0.988 0.989 0.213 0.980
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.970 0.996 0.985 0.982 0.982 0.519 0.978

表 21:在 1 万个混合系统上,与 WMT18 到英语语言对人工判断的绝对皮尔逊相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了系统数。

设置 指标 cs-en de-en et-en fi-en ru-en tr-en zh-en
10K 10K 10K 10K 10K 10K 10K
无监督 BLEU 0.956 0.969 0.981 0.962 0.972 0.586 0.968
CDER 0.964 0.980 0.988 0.976 0.974 0.577 0.973
CharacTER 0.960 0.992 0.975 0.979 0.984 0.680 0.942
ITER 0.966 0.990 0.975 0.989 0.943 0.742 0.978
Meteor++ 0.937 0.990 0.975 0.962 0.989 0.787 0.954
NIST 0.942 0.982 0.980 0.965 0.965 0.862 0.959
PER 0.937 0.982 0.978 0.983 0.955 0.043 0.923
TER 0.942 0.970 0.988 0.960 0.963 0.450 0.967
UHH_TSKM 0.943 0.979 0.987 0.974 0.973 0.443 0.972
WER 0.942 0.961 0.989 0.953 0.962 0.072 0.967
YiSi-0 0.947 0.992 0.972 0.969 0.982 0.863 0.950
YiSi-1 0.942 0.991 0.976 0.964 0.985 0.881 0.943
YiSi-1 srl 0.957 0.994 0.978 0.968 0.986 0.785 0.954
Supervised BEER 0.950 0.993 0.983 0.982 0.976 0.723 0.968
BLEND 0.965 0.990 0.982 0.985 0.986 0.724 0.969
RUSE 0.974 0.996 0.988 0.983 0.982 0.780 0.973
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.954 0.992 0.984 0.980 0.970 0.917 0.965
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.988 0.994 0.974 0.987 0.988 0.801 0.975
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.973 0.994 0.981 0.987 0.982 0.924 0.973
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.957 0.994 0.983 0.966 0.970 0.875 0.966
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.986 0.990 0.976 0.984 0.984 0.019 0.980
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.974 0.993 0.980 0.978 0.978 0.853 0.976
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.949 0.995 0.986 0.960 0.969 0.832 0.972
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.983 0.997 0.979 0.986 0.986 0.099 0.980
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.967 0.997 0.984 0.978 0.981 0.722 0.979
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.949 0.994 0.986 0.946 0.969 0.743 0.969
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.984 0.994 0.980 0.980 0.986 0.541 0.982
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.967 0.995 0.984 0.968 0.979 0.464 0.978
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.969 0.991 0.985 0.979 0.970 0.915 0.969
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.990 0.993 0.980 0.988 0.988 0.745 0.978
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.982 0.993 0.984 0.986 0.981 0.909 0.976
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.970 0.991 0.984 0.963 0.971 0.858 0.970
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.989 0.990 0.982 0.982 0.985 0.047 0.982
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.981 0.991 0.984 0.976 0.978 0.722 0.978
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.959 0.992 0.988 0.986 0.971 0.809 0.976
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.987 0.997 0.978 0.989 0.988 0.238 0.981
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.973 0.997 0.987 0.989 0.982 0.674 0.982
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.963 0.994 0.988 0.989 0.970 0.711 0.972
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.988 0.996 0.979 0.989 0.987 0.353 0.983
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.976 0.997 0.986 0.990 0.980 0.277 0.980
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.965 0.995 0.990 0.976 0.976 0.846 0.975
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.989 0.997 0.982 0.989 0.988 0.540 0.981
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.978 0.998 0.989 0.983 0.985 0.760 0.981
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.972 0.997 0.988 0.986 0.976 0.686 0.973
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.990 0.996 0.983 0.989 0.989 0.096 0.982
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.982 0.998 0.988 0.989 0.983 0.453 0.980
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.978 0.997 0.991 0.984 0.980 0.914 0.977
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.991 0.996 0.984 0.989 0.987 0.566 0.982
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.987 0.998 0.989 0.988 0.986 0.873 0.982
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.982 0.998 0.992 0.990 0.978 0.822 0.974
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.992 0.996 0.985 0.988 0.988 0.022 0.983
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.989 0.998 0.990 0.990 0.985 0.583 0.980
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.960 0.997 0.984 0.982 0.972 0.849 0.974
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.985 0.997 0.975 0.988 0.988 0.303 0.980
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.974 0.998 0.981 0.986 0.982 0.628 0.980
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.962 0.995 0.983 0.982 0.972 0.657 0.974
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.986 0.996 0.976 0.987 0.987 0.666 0.982
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.975 0.996 0.980 0.985 0.981 0.259 0.980
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.955 0.995 0.983 0.986 0.972 0.875 0.970
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.984 0.996 0.972 0.984 0.989 0.491 0.975
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.971 0.996 0.980 0.987 0.984 0.821 0.976
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.961 0.997 0.983 0.987 0.973 0.816 0.973
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.987 0.996 0.975 0.989 0.988 0.320 0.981
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.976 0.997 0.980 0.989 0.982 0.623 0.980
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.953 0.995 0.988 0.979 0.974 0.918 0.972
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.983 0.996 0.980 0.988 0.991 0.561 0.977
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.969 0.997 0.986 0.986 0.985 0.869 0.977
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.957 0.996 0.987 0.970 0.974 0.862 0.973
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.982 0.995 0.981 0.988 0.989 0.213 0.980
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.970 0.996 0.985 0.982 0.982 0.519 0.978

Table 22: Absolute Pearson correlations with human judgments on WMT18 from-English language pairs for 10K hybrid systems. Correlations of metrics not significantly outperformed by any other for that language pair are highlighted in bold. For each language pair, we specify the number of systems.

Setting Metric en-cs en-de en-et en-fi en-ru en-tr en-zh
10K 10K 10K 10K 10K 10K 10K
Unsupervised BLEU 0.993 0.977 0.971 0.958 0.977 0.796 0.941
CDER 0.995 0.984 0.981 0.961 0.982 0.832 0.956
CharacTER 0.990 0.986 0.950 0.963 0.981 0.775 0.978
ITER 0.865 0.978 0.982 0.966 0.965 0.872 –
Meteor++ – – – – – – –
NIST 0.997 0.984 0.980 0.944 0.988 0.870 0.944
PER 0.987 0.979 0.954 0.904 0.986 0.829 0.950
TER 0.995 0.986 0.977 0.939 0.985 0.837 0.959
UHH_TSKM – – – – – – –
WER 0.994 0.984 0.977 0.942 0.983 0.824 0.954
YiSi-0 0.971 0.983 0.965 0.942 0.988 0.953 0.951
YiSi-1 0.985 0.983 0.976 0.938 0.989 0.942 0.957
YiSi-1 srl – 0.988 – – – – 0.948
Supervised BEER 0.990 0.989 0.978 0.959 0.986 0.933 0.925
BLEND – – – – 0.986 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.989 0.983 0.970 0.951 0.988 0.936 0.950
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.995 0.991 0.979 0.977 0.989 0.872 0.980
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.993 0.988 0.978 0.969 0.989 0.910 0.969
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.992 0.986 0.978 0.954 0.988 0.903 0.950
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.995 0.988 0.977 0.976 0.987 0.850 0.972
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.995 0.988 0.979 0.969 0.987 0.877 0.963
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.980 0.990 0.991 0.972 0.991 0.936 0.959
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.991 0.990 0.989 0.985 0.991 0.882 0.981
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.987 0.990 0.991 0.981 0.991 0.915 0.974
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.982 0.990 0.990 0.968 0.991 0.931 0.960
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.989 0.990 0.990 0.985 0.990 0.867 0.978
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.986 0.991 0.991 0.982 0.991 0.905 0.972

表 22:在 1 万个混合系统上,与 WMT18 从英语出发语言对人工判断的绝对皮尔逊相关。在该语言对上未被任何其他指标显著超过的相关加粗。我们为每个语言对注明了系统数。

设置 指标 en-cs en-de en-et en-fi en-ru en-tr en-zh
10K 10K 10K 10K 10K 10K 10K
无监督 BLEU 0.993 0.977 0.971 0.958 0.977 0.796 0.941
CDER 0.995 0.984 0.981 0.961 0.982 0.832 0.956
CharacTER 0.990 0.986 0.950 0.963 0.981 0.775 0.978
ITER 0.865 0.978 0.982 0.966 0.965 0.872 –
Meteor++ – – – – – – –
NIST 0.997 0.984 0.980 0.944 0.988 0.870 0.944
PER 0.987 0.979 0.954 0.904 0.986 0.829 0.950
TER 0.995 0.986 0.977 0.939 0.985 0.837 0.959
UHH_TSKM – – – – – – –
WER 0.994 0.984 0.977 0.942 0.983 0.824 0.954
YiSi-0 0.971 0.983 0.965 0.942 0.988 0.953 0.951
YiSi-1 0.985 0.983 0.976 0.938 0.989 0.942 0.957
YiSi-1 srl – 0.988 – – – – 0.948
Supervised BEER 0.990 0.989 0.978 0.959 0.986 0.933 0.925
BLEND – – – – 0.986 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.989 0.983 0.970 0.951 0.988 0.936 0.950
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.995 0.991 0.979 0.977 0.989 0.872 0.980
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.993 0.988 0.978 0.969 0.989 0.910 0.969
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.992 0.986 0.978 0.954 0.988 0.903 0.950
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.995 0.988 0.977 0.976 0.987 0.850 0.972
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.995 0.988 0.979 0.969 0.987 0.877 0.963
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.980 0.990 0.991 0.972 0.991 0.936 0.959
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.991 0.990 0.989 0.985 0.991 0.882 0.981
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.987 0.990 0.991 0.981 0.991 0.915 0.974
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.982 0.990 0.990 0.968 0.991 0.931 0.960
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.989 0.990 0.990 0.985 0.990 0.867 0.978
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.986 0.991 0.991 0.982 0.991 0.905 0.972

Table 23: Model selection accuracies (Hits@1) on to-English WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10−310^{-3}10−3. We bold the highest numbers for each language pair and direction.

Setting Metric cs-en de-en et-en fi-en ru-en tr-en zh-en
Unsupervised BLEU 0.135 0.804 0.757 0.460 0.230 0.096 0.661
CDER 0.162 0.795 0.764 0.493 0.234 0.087 0.660
CharacTER 0.146 0.737 0.696 0.496 0.201 0.082 0.584
ITER 0.152 0.814 0.746 0.474 0.234 0.100 0.673
Meteor++ 0.172 0.804 0.646 0.456 0.253 0.052 0.597
NIST 0.136 0.802 0.739 0.469 0.228 0.135 0.665
PER 0.121 0.764 0.602 0.455 0.218 0.000 0.602
TER 0.139 0.789 0.768 0.470 0.232 0.001 0.652
UHH_TSKM 0.191 0.803 0.768 0.469 0.240 0.002 0.642
WER 0.149 0.776 0.760 0.471 0.227 0.000 0.654
YiSi-0 0.148 0.780 0.703 0.483 0.229 0.106 0.629
YiSi-1 0.157 0.808 0.752 0.466 0.250 0.110 0.613
YiSi-1 srl 0.159 0.814 0.763 0.484 0.243 0.008 0.620
Supervised BEER 0.165 0.811 0.765 0.485 0.237 0.030 0.675
BLEND 0.184 0.820 0.779 0.484 0.254 0.003 0.611
RUSE 0.213 0.823 0.788 0.487 0.250 0.109 0.672
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.190 0.815 0.778 0.468 0.261 0.130 0.655
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.189 0.813 0.775 0.481 0.266 0.014 0.663
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.194 0.819 0.778 0.474 0.265 0.144 0.670
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.189 0.817 0.775 0.477 0.255 0.131 0.650
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.192 0.808 0.771 0.484 0.248 0.005 0.674
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.193 0.817 0.774 0.483 0.262 0.081 0.669
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.190 0.701 0.766 0.487 0.254 0.126 0.653
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.199 0.826 0.765 0.493 0.258 0.000 0.671
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.197 0.824 0.767 0.491 0.260 0.147 0.668
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.186 0.806 0.765 0.492 0.247 0.125 0.661
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.200 0.823 0.760 0.495 0.258 0.000 0.680
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.196 0.821 0.763 0.497 0.254 0.031 0.676
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.200 0.815 0.778 0.474 0.261 0.137 0.661
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.194 0.809 0.779 0.493 0.270 0.006 0.672
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.199 0.810 0.782 0.484 0.266 0.142 0.672
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.200 0.813 0.772 0.485 0.256 0.136 0.657
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.197 0.806 0.769 0.495 0.262 0.005 0.675
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.199 0.811 0.772 0.494 0.262 0.006 0.673
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.173 0.675 0.757 0.502 0.258 0.126 0.654
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.165 0.816 0.764 0.483 0.266 0.000 0.674
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.173 0.820 0.764 0.498 0.262 0.090 0.669
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.172 0.691 0.755 0.503 0.252 0.123 0.661
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.172 0.809 0.758 0.490 0.268 0.000 0.678
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.178 0.820 0.758 0.501 0.260 0.001 0.674
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.174 0.704 0.765 0.497 0.255 0.140 0.663
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.163 0.805 0.770 0.491 0.263 0.005 0.679
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.175 0.825 0.770 0.499 0.262 0.143 0.675
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.181 0.821 0.758 0.500 0.256 0.089 0.669
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.165 0.787 0.763 0.495 0.270 0.000 0.684
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.179 0.824 0.761 0.502 0.265 0.004 0.679
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.185 0.828 0.780 0.504 0.263 0.133 0.654
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.179 0.779 0.775 0.494 0.266 0.004 0.670
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.186 0.827 0.778 0.502 0.267 0.113 0.669
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.190 0.820 0.771 0.504 0.261 0.102 0.661
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.181 0.769 0.766 0.494 0.266 0.004 0.674
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.188 0.822 0.768 0.501 0.265 0.004 0.671
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.186 0.771 0.762 0.496 0.247 0.153 0.658
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.182 0.823 0.764 0.496 0.256 0.000 0.671
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.186 0.824 0.765 0.499 0.253 0.049 0.673
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.178 0.819 0.756 0.506 0.241 0.130 0.656
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.183 0.817 0.754 0.501 0.256 0.000 0.673
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.182 0.821 0.755 0.505 0.250 0.000 0.670
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.195 0.721 0.767 0.493 0.152 0.144 0.661
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.192 0.821 0.766 0.494 0.260 0.001 0.659
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.196 0.824 0.773 0.496 0.261 0.155 0.675
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.191 0.811 0.765 0.500 0.167 0.144 0.657
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.196 0.815 0.762 0.495 0.259 0.000 0.673
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.195 0.822 0.764 0.499 0.256 0.046 0.674
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.192 0.796 0.779 0.486 0.255 0.131 0.665
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.202 0.818 0.772 0.495 0.261 0.005 0.662
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.199 0.827 0.778 0.491 0.262 0.086 0.674
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.189 0.818 0.770 0.485 0.259 0.116 0.662
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.202 0.812 0.761 0.490 0.250 0.003 0.668
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.196 0.821 0.766 0.490 0.263 0.003 0.672

表 23:在到英语的 WMT18 混合系统上的模型选择准确率(Hits@1)。我们报告 10 万次抽样的平均值,0.95 置信区间低于 10−310^{-3}10−3。每个语言对和方向上的最高值加粗。

设置 指标 cs-en de-en et-en fi-en ru-en tr-en zh-en
无监督 BLEU 0.135 0.804 0.757 0.460 0.230 0.096 0.661
CDER 0.162 0.795 0.764 0.493 0.234 0.087 0.660
CharacTER 0.146 0.737 0.696 0.496 0.201 0.082 0.584
ITER 0.152 0.814 0.746 0.474 0.234 0.100 0.673
Meteor++ 0.172 0.804 0.646 0.456 0.253 0.052 0.597
NIST 0.136 0.802 0.739 0.469 0.228 0.135 0.665
PER 0.121 0.764 0.602 0.455 0.218 0.000 0.602
TER 0.139 0.789 0.768 0.470 0.232 0.001 0.652
UHH_TSKM 0.191 0.803 0.768 0.469 0.240 0.002 0.642
WER 0.149 0.776 0.760 0.471 0.227 0.000 0.654
YiSi-0 0.148 0.780 0.703 0.483 0.229 0.106 0.629
YiSi-1 0.157 0.808 0.752 0.466 0.250 0.110 0.613
YiSi-1 srl 0.159 0.814 0.763 0.484 0.243 0.008 0.620
Supervised BEER 0.165 0.811 0.765 0.485 0.237 0.030 0.675
BLEND 0.184 0.820 0.779 0.484 0.254 0.003 0.611
RUSE 0.213 0.823 0.788 0.487 0.250 0.109 0.672
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.190 0.815 0.778 0.468 0.261 0.130 0.655
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.189 0.813 0.775 0.481 0.266 0.014 0.663
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.194 0.819 0.778 0.474 0.265 0.144 0.670
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.189 0.817 0.775 0.477 0.255 0.131 0.650
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.192 0.808 0.771 0.484 0.248 0.005 0.674
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.193 0.817 0.774 0.483 0.262 0.081 0.669
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.190 0.701 0.766 0.487 0.254 0.126 0.653
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.199 0.826 0.765 0.493 0.258 0.000 0.671
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.197 0.824 0.767 0.491 0.260 0.147 0.668
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.186 0.806 0.765 0.492 0.247 0.125 0.661
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.200 0.823 0.760 0.495 0.258 0.000 0.680
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.196 0.821 0.763 0.497 0.254 0.031 0.676
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.200 0.815 0.778 0.474 0.261 0.137 0.661
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.194 0.809 0.779 0.493 0.270 0.006 0.672
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.199 0.810 0.782 0.484 0.266 0.142 0.672
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.200 0.813 0.772 0.485 0.256 0.136 0.657
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.197 0.806 0.769 0.495 0.262 0.005 0.675
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.199 0.811 0.772 0.494 0.262 0.006 0.673
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.173 0.675 0.757 0.502 0.258 0.126 0.654
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.165 0.816 0.764 0.483 0.266 0.000 0.674
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.173 0.820 0.764 0.498 0.262 0.090 0.669
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.172 0.691 0.755 0.503 0.252 0.123 0.661
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.172 0.809 0.758 0.490 0.268 0.000 0.678
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.178 0.820 0.758 0.501 0.260 0.001 0.674
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.174 0.704 0.765 0.497 0.255 0.140 0.663
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.163 0.805 0.770 0.491 0.263 0.005 0.679
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.175 0.825 0.770 0.499 0.262 0.143 0.675
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.181 0.821 0.758 0.500 0.256 0.089 0.669
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.165 0.787 0.763 0.495 0.270 0.000 0.684
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.179 0.824 0.761 0.502 0.265 0.004 0.679
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.185 0.828 0.780 0.504 0.263 0.133 0.654
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.179 0.779 0.775 0.494 0.266 0.004 0.670
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.186 0.827 0.778 0.502 0.267 0.113 0.669
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.190 0.820 0.771 0.504 0.261 0.102 0.661
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.181 0.769 0.766 0.494 0.266 0.004 0.674
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.188 0.822 0.768 0.501 0.265 0.004 0.671
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.186 0.771 0.762 0.496 0.247 0.153 0.658
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.182 0.823 0.764 0.496 0.256 0.000 0.671
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.186 0.824 0.765 0.499 0.253 0.049 0.673
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.178 0.819 0.756 0.506 0.241 0.130 0.656
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.183 0.817 0.754 0.501 0.256 0.000 0.673
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.182 0.821 0.755 0.505 0.250 0.000 0.670
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.195 0.721 0.767 0.493 0.152 0.144 0.661
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.192 0.821 0.766 0.494 0.260 0.001 0.659
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.196 0.824 0.773 0.496 0.261 0.155 0.675
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.191 0.811 0.765 0.500 0.167 0.144 0.657
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.196 0.815 0.762 0.495 0.259 0.000 0.673
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.195 0.822 0.764 0.499 0.256 0.046 0.674
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.192 0.796 0.779 0.486 0.255 0.131 0.665
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.202 0.818 0.772 0.495 0.261 0.005 0.662
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.199 0.827 0.778 0.491 0.262 0.086 0.674
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.189 0.818 0.770 0.485 0.259 0.116 0.662
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.202 0.812 0.761 0.490 0.250 0.003 0.668
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.196 0.821 0.766 0.490 0.263 0.003 0.672

Table 24: Mean Reciprocal Rank (MRR) of the top metric-rated system on to-English WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10−310^{-3}10−3. We bold the highest numbers for each language pair and direction.

Setting Metric cs-en de-en et-en fi-en ru-en tr-en zh-en
Unsupervised BLEU 0.338 0.894 0.866 0.666 0.447 0.265 0.799
CDER 0.362 0.890 0.870 0.689 0.451 0.256 0.799
CharacTER 0.349 0.854 0.814 0.690 0.429 0.254 0.739
ITER 0.356 0.901 0.856 0.676 0.454 0.278 0.811
Meteor++ 0.369 0.895 0.798 0.662 0.470 0.174 0.757
NIST 0.338 0.894 0.857 0.672 0.446 0.323 0.803
PER 0.325 0.866 0.771 0.663 0.435 0.021 0.754
TER 0.342 0.885 0.873 0.673 0.447 0.063 0.792
UHH_TSKM 0.387 0.894 0.873 0.671 0.460 0.063 0.788
WER 0.353 0.876 0.868 0.674 0.443 0.034 0.790
YiSi-0 0.344 0.881 0.834 0.681 0.452 0.275 0.776
YiSi-1 0.352 0.896 0.864 0.671 0.470 0.285 0.765
YiSi-1 srl 0.351 0.901 0.871 0.682 0.464 0.086 0.770
Supervised BEER 0.364 0.899 0.871 0.684 0.460 0.125 0.811
BLEND 0.382 0.904 0.880 0.681 0.473 0.077 0.767
RUSE 0.417 0.906 0.885 0.686 0.468 0.273 0.809
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.386 0.901 0.880 0.674 0.481 0.318 0.793
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.383 0.899 0.877 0.683 0.486 0.100 0.804
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.388 0.903 0.879 0.678 0.484 0.331 0.808
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.390 0.902 0.877 0.681 0.475 0.318 0.786
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.390 0.896 0.874 0.686 0.475 0.077 0.811
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.393 0.902 0.876 0.685 0.483 0.225 0.806
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.392 0.832 0.872 0.686 0.475 0.319 0.791
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.397 0.908 0.870 0.691 0.478 0.025 0.811
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.398 0.907 0.872 0.690 0.481 0.335 0.806
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.392 0.896 0.870 0.689 0.467 0.316 0.797
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.400 0.906 0.867 0.691 0.479 0.018 0.817
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.400 0.905 0.869 0.693 0.475 0.097 0.812
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.398 0.901 0.880 0.678 0.481 0.327 0.799
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.391 0.897 0.879 0.690 0.490 0.085 0.810
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.397 0.898 0.882 0.684 0.486 0.328 0.810
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.398 0.900 0.875 0.685 0.475 0.323 0.794
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.395 0.895 0.873 0.692 0.488 0.080 0.813
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.398 0.899 0.875 0.691 0.482 0.086 0.810
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.372 0.814 0.866 0.697 0.475 0.313 0.795
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.366 0.902 0.870 0.683 0.483 0.026 0.813
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.374 0.904 0.870 0.694 0.480 0.224 0.808
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.373 0.825 0.865 0.697 0.470 0.303 0.802
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.374 0.898 0.866 0.688 0.486 0.028 0.816
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.380 0.904 0.866 0.696 0.479 0.037 0.812
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.375 0.833 0.871 0.693 0.474 0.327 0.800
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.366 0.895 0.874 0.689 0.480 0.039 0.816
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.378 0.907 0.874 0.694 0.480 0.324 0.811
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.384 0.905 0.866 0.694 0.475 0.220 0.806
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.368 0.885 0.869 0.692 0.487 0.030 0.819
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.382 0.907 0.868 0.696 0.484 0.048 0.815
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.383 0.909 0.880 0.698 0.480 0.323 0.795
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.378 0.880 0.877 0.692 0.481 0.078 0.811
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.385 0.909 0.879 0.697 0.484 0.286 0.809
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.389 0.905 0.874 0.698 0.478 0.268 0.803
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.380 0.874 0.870 0.691 0.483 0.079 0.814
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.387 0.906 0.872 0.696 0.482 0.082 0.811
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.385 0.875 0.869 0.692 0.469 0.342 0.796
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.381 0.907 0.869 0.693 0.477 0.026 0.809
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.385 0.907 0.871 0.694 0.476 0.128 0.810
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.381 0.904 0.864 0.699 0.464 0.289 0.794
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.384 0.903 0.863 0.696 0.479 0.013 0.812
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.384 0.905 0.864 0.699 0.472 0.032 0.809
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.392 0.844 0.873 0.689 0.367 0.338 0.799
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.389 0.905 0.871 0.690 0.482 0.031 0.800
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.393 0.907 0.876 0.691 0.483 0.348 0.812
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.393 0.899 0.870 0.694 0.387 0.333 0.794
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.395 0.901 0.868 0.690 0.483 0.023 0.810
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.396 0.906 0.870 0.693 0.478 0.128 0.811
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.394 0.891 0.880 0.685 0.476 0.322 0.802
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.401 0.903 0.875 0.692 0.483 0.082 0.803
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.400 0.909 0.878 0.689 0.483 0.234 0.811
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.391 0.903 0.874 0.684 0.480 0.293 0.797
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.402 0.900 0.868 0.688 0.477 0.068 0.806
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.398 0.905 0.871 0.688 0.487 0.079 0.809

表 24:在到英语的 WMT18 混合系统上,指标评为最优的系统的平均倒数排名(MRR)。我们报告 10 万次抽样的平均值,0.95 置信区间低于 10−310^{-3}10−3。每个语言对和方向上的最高值加粗。

设置 指标 cs-en de-en et-en fi-en ru-en tr-en zh-en
无监督 BLEU 0.338 0.894 0.866 0.666 0.447 0.265 0.799
CDER 0.362 0.890 0.870 0.689 0.451 0.256 0.799
CharacTER 0.349 0.854 0.814 0.690 0.429 0.254 0.739
ITER 0.356 0.901 0.856 0.676 0.454 0.278 0.811
Meteor++ 0.369 0.895 0.798 0.662 0.470 0.174 0.757
NIST 0.338 0.894 0.857 0.672 0.446 0.323 0.803
PER 0.325 0.866 0.771 0.663 0.435 0.021 0.754
TER 0.342 0.885 0.873 0.673 0.447 0.063 0.792
UHH_TSKM 0.387 0.894 0.873 0.671 0.460 0.063 0.788
WER 0.353 0.876 0.868 0.674 0.443 0.034 0.790
YiSi-0 0.344 0.881 0.834 0.681 0.452 0.275 0.776
YiSi-1 0.352 0.896 0.864 0.671 0.470 0.285 0.765
YiSi-1 srl 0.351 0.901 0.871 0.682 0.464 0.086 0.770
Supervised BEER 0.364 0.899 0.871 0.684 0.460 0.125 0.811
BLEND 0.382 0.904 0.880 0.681 0.473 0.077 0.767
RUSE 0.417 0.906 0.885 0.686 0.468 0.273 0.809
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.386 0.901 0.880 0.674 0.481 0.318 0.793
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.383 0.899 0.877 0.683 0.486 0.100 0.804
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.388 0.903 0.879 0.678 0.484 0.331 0.808
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.390 0.902 0.877 0.681 0.475 0.318 0.786
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.390 0.896 0.874 0.686 0.475 0.077 0.811
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.393 0.902 0.876 0.685 0.483 0.225 0.806
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.392 0.832 0.872 0.686 0.475 0.319 0.791
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.397 0.908 0.870 0.691 0.478 0.025 0.811
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.398 0.907 0.872 0.690 0.481 0.335 0.806
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.392 0.896 0.870 0.689 0.467 0.316 0.797
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.400 0.906 0.867 0.691 0.479 0.018 0.817
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.400 0.905 0.869 0.693 0.475 0.097 0.812
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.398 0.901 0.880 0.678 0.481 0.327 0.799
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.391 0.897 0.879 0.690 0.490 0.085 0.810
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.397 0.898 0.882 0.684 0.486 0.328 0.810
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.398 0.900 0.875 0.685 0.475 0.323 0.794
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.395 0.895 0.873 0.692 0.488 0.080 0.813
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.398 0.899 0.875 0.691 0.482 0.086 0.810
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.372 0.814 0.866 0.697 0.475 0.313 0.795
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.366 0.902 0.870 0.683 0.483 0.026 0.813
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.374 0.904 0.870 0.694 0.480 0.224 0.808
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.373 0.825 0.865 0.697 0.470 0.303 0.802
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.374 0.898 0.866 0.688 0.486 0.028 0.816
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.380 0.904 0.866 0.696 0.479 0.037 0.812
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.375 0.833 0.871 0.693 0.474 0.327 0.800
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.366 0.895 0.874 0.689 0.480 0.039 0.816
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.378 0.907 0.874 0.694 0.480 0.324 0.811
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.384 0.905 0.866 0.694 0.475 0.220 0.806
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.368 0.885 0.869 0.692 0.487 0.030 0.819
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.382 0.907 0.868 0.696 0.484 0.048 0.815
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.383 0.909 0.880 0.698 0.480 0.323 0.795
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.378 0.880 0.877 0.692 0.481 0.078 0.811
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.385 0.909 0.879 0.697 0.484 0.286 0.809
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.389 0.905 0.874 0.698 0.478 0.268 0.803
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.380 0.874 0.870 0.691 0.483 0.079 0.814
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.387 0.906 0.872 0.696 0.482 0.082 0.811
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 0.385 0.875 0.869 0.692 0.469 0.342 0.796
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 0.381 0.907 0.869 0.693 0.477 0.026 0.809
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 0.385 0.907 0.871 0.694 0.476 0.128 0.810
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 0.381 0.904 0.864 0.699 0.464 0.289 0.794
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 0.384 0.903 0.863 0.696 0.479 0.013 0.812
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 0.384 0.905 0.864 0.699 0.472 0.032 0.809
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 0.392 0.844 0.873 0.689 0.367 0.338 0.799
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 0.389 0.905 0.871 0.690 0.482 0.031 0.800
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 0.393 0.907 0.876 0.691 0.483 0.348 0.812
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 0.393 0.899 0.870 0.694 0.387 0.333 0.794
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 0.395 0.901 0.868 0.690 0.483 0.023 0.810
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 0.396 0.906 0.870 0.693 0.478 0.128 0.811
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.394 0.891 0.880 0.685 0.476 0.322 0.802
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.401 0.903 0.875 0.692 0.483 0.082 0.803
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.400 0.909 0.878 0.689 0.483 0.234 0.811
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.391 0.903 0.874 0.684 0.480 0.293 0.797
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.402 0.900 0.868 0.688 0.477 0.068 0.806
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.398 0.905 0.871 0.688 0.487 0.079 0.809

Table 25: Absolute Difference (×100\times 100×100) of the top metric-rated and the top human-rated system on to-English WMT18 hybrid systems. Smaller difference signify higher agreement with human scores. We report the average of 100K samples and the 0.95 confidence intervals are below 10−310^{-3}10−3. We bold the lowest numbers for each language pair and direction.

Setting Metric cs-en de-en et-en fi-en ru-en tr-en zh-en
Unsupervised BLEU 3.85 0.45 1.01 2.17 2.34 4.48 3.19
CDER 3.88 0.43 0.87 1.33 2.30 4.58 3.43
CharacTER 3.77 0.49 0.94 2.07 2.25 4.07 3.37
ITER 3.55 0.46 1.25 1.43 4.65 3.11 2.92
Meteor++ 3.70 0.41 0.69 1.13 2.28 1.40 3.50
NIST 3.93 0.49 1.10 1.19 2.36 1.42 3.92
PER 2.02 0.46 1.71 1.49 2.25 4.22 3.20
TER 3.86 0.43 1.14 1.14 4.34 5.18 3.82
UHH_TSKM 3.98 0.40 1.27 1.10 2.23 4.26 3.47
WER 3.85 0.44 1.48 1.18 4.87 5.96 3.72
YiSi-0 3.81 0.48 0.72 1.20 1.75 1.40 3.44
YiSi-1 3.88 0.44 0.65 1.13 2.17 1.32 3.40
YiSi-1 srl 3.67 0.41 0.64 1.20 2.15 1.31 3.55
Supervised BEER 3.82 0.41 0.79 1.08 1.92 1.96 3.43
BLEND 3.77 0.41 0.66 1.09 2.21 1.28 3.46
RUSE 3.13 0.32 0.64 1.03 1.51 1.94 3.15
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 3.97 0.36 0.72 1.16 2.20 1.25 3.26
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 1.51 0.43 0.60 1.65 1.33 1.34 3.50
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 3.70 0.36 0.59 1.08 1.92 1.27 3.38
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 3.94 0.36 0.64 1.18 2.06 2.55 3.54
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 1.54 0.43 0.63 1.87 1.12 5.96 3.38
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 2.75 0.39 0.60 1.10 1.38 1.26 3.51
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 4.02 0.35 0.74 1.15 1.09 3.33 3.06
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 2.66 0.43 0.62 1.75 1.10 5.64 3.34
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 3.89 0.36 0.60 1.09 1.08 3.82 3.23
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 4.02 0.35 0.67 1.18 1.48 3.30 3.49
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 1.63 0.43 0.65 1.93 1.13 7.26 3.13
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 3.86 0.38 0.61 1.11 1.14 4.24 3.28
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 3.82 0.34 0.66 1.12 2.10 1.31 3.60
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 1.49 0.40 0.59 1.56 1.17 1.35 3.61
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 1.71 0.35 0.58 1.08 1.65 1.29 3.60
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 3.74 0.35 0.65 1.12 1.90 1.98 3.77
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 1.51 0.42 0.62 1.86 1.10 5.84 3.21
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 1.49 0.38 0.60 1.17 1.24 1.96 3.53
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 3.89 0.37 0.75 1.18 1.07 3.45 2.62
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 1.92 0.39 0.64 1.57 1.11 5.75 3.13
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 3.56 0.37 0.59 1.10 1.08 3.79 2.90
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 3.89 0.38 0.67 1.20 1.30 3.27 3.47
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 1.61 0.42 0.67 1.65 1.14 6.55 2.95
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 3.18 0.38 0.60 1.11 1.13 6.54 3.11
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 3.64 0.36 0.71 1.10 1.03 2.69 2.57
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 1.60 0.37 0.64 1.51 1.09 3.91 3.27
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 2.38 0.35 0.58 1.06 1.05 3.57 2.95
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 2.70 0.36 0.69 1.13 1.08 3.18 2.89
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 1.55 0.39 0.66 1.59 1.10 6.66 3.18
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 1.68 0.37 0.59 1.08 1.08 5.58 2.91
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 2.14 0.35 0.61 1.07 1.09 1.21 3.35
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 1.45 0.37 0.64 1.49 1.10 4.42 3.55
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 1.42 0.35 0.59 1.07 1.07 1.27 3.41
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 1.55 0.35 0.60 1.08 1.12 1.54 3.87
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 1.45 0.39 0.64 1.65 1.09 5.89 3.32
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 1.42 0.36 0.60 1.10 1.08 3.80 3.45
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 3.90 0.37 0.68 1.07 1.16 2.47 2.91
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 1.71 0.45 0.72 1.58 1.07 6.29 3.36
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 3.78 0.39 0.62 1.05 1.07 3.60 3.20
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 3.90 0.46 0.65 1.08 2.93 3.30 3.39
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 1.51 0.45 0.82 1.78 1.12 10.77 3.13
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 3.67 0.42 0.66 1.11 1.22 7.13 3.23
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 3.94 0.37 0.71 1.10 21.10 1.85 2.90
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 2.23 0.41 0.69 1.34 1.07 4.46 3.40
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 3.84 0.36 0.60 1.03 1.07 3.38 3.22
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 3.92 0.41 0.64 1.12 21.10 3.24 3.37
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 1.60 0.43 0.78 1.70 1.09 6.13 3.20
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 3.80 0.38 0.63 1.06 1.09 3.72 3.25
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 3.88 0.33 0.75 1.16 2.16 1.28 3.29
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 1.98 0.41 0.60 1.41 1.21 3.30 3.47
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 3.78 0.36 0.61 1.09 1.71 1.30 3.40
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 3.84 0.36 0.69 1.17 1.86 1.33 3.47
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 1.70 0.42 0.63 1.55 1.11 5.87 3.36
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 3.72 0.40 0.62 1.14 1.32 4.15 3.43

表 25:在到英语的 WMT18 混合系统上,指标评为最优与人工评为最优的系统之间人工分数的绝对差(×100\times 100×100)。差值越小,表示与人工分数越一致。我们报告 10 万次抽样的平均值,0.95 置信区间低于 10−310^{-3}10−3。每个语言对和方向上的最低值加粗。

设置 指标 cs-en de-en et-en fi-en ru-en tr-en zh-en
无监督 BLEU 3.85 0.45 1.01 2.17 2.34 4.48 3.19
CDER 3.88 0.43 0.87 1.33 2.30 4.58 3.43
CharacTER 3.77 0.49 0.94 2.07 2.25 4.07 3.37
ITER 3.55 0.46 1.25 1.43 4.65 3.11 2.92
Meteor++ 3.70 0.41 0.69 1.13 2.28 1.40 3.50
NIST 3.93 0.49 1.10 1.19 2.36 1.42 3.92
PER 2.02 0.46 1.71 1.49 2.25 4.22 3.20
TER 3.86 0.43 1.14 1.14 4.34 5.18 3.82
UHH_TSKM 3.98 0.40 1.27 1.10 2.23 4.26 3.47
WER 3.85 0.44 1.48 1.18 4.87 5.96 3.72
YiSi-0 3.81 0.48 0.72 1.20 1.75 1.40 3.44
YiSi-1 3.88 0.44 0.65 1.13 2.17 1.32 3.40
YiSi-1 srl 3.67 0.41 0.64 1.20 2.15 1.31 3.55
Supervised BEER 3.82 0.41 0.79 1.08 1.92 1.96 3.43
BLEND 3.77 0.41 0.66 1.09 2.21 1.28 3.46
RUSE 3.13 0.32 0.64 1.03 1.51 1.94 3.15
Pre-Trained PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 3.97 0.36 0.72 1.16 2.20 1.25 3.26
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 1.51 0.43 0.60 1.65 1.33 1.34 3.50
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 3.70 0.36 0.59 1.08 1.92 1.27 3.38
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 3.94 0.36 0.64 1.18 2.06 2.55 3.54
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 1.54 0.43 0.63 1.87 1.12 5.96 3.38
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 2.75 0.39 0.60 1.10 1.38 1.26 3.51
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 4.02 0.35 0.74 1.15 1.09 3.33 3.06
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 2.66 0.43 0.62 1.75 1.10 5.64 3.34
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 3.89 0.36 0.60 1.09 1.08 3.82 3.23
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 4.02 0.35 0.67 1.18 1.48 3.30 3.49
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 1.63 0.43 0.65 1.93 1.13 7.26 3.13
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 3.86 0.38 0.61 1.11 1.14 4.24 3.28
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 3.82 0.34 0.66 1.12 2.10 1.31 3.60
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 1.49 0.40 0.59 1.56 1.17 1.35 3.61
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 1.71 0.35 0.58 1.08 1.65 1.29 3.60
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 3.74 0.35 0.65 1.12 1.90 1.98 3.77
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 1.51 0.42 0.62 1.86 1.10 5.84 3.21
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 1.49 0.38 0.60 1.17 1.24 1.96 3.53
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 3.89 0.37 0.75 1.18 1.07 3.45 2.62
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 1.92 0.39 0.64 1.57 1.11 5.75 3.13
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 3.56 0.37 0.59 1.10 1.08 3.79 2.90
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 3.89 0.38 0.67 1.20 1.30 3.27 3.47
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 1.61 0.42 0.67 1.65 1.14 6.55 2.95
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 3.18 0.38 0.60 1.11 1.13 6.54 3.11
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 3.64 0.36 0.71 1.10 1.03 2.69 2.57
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 1.60 0.37 0.64 1.51 1.09 3.91 3.27
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 2.38 0.35 0.58 1.06 1.05 3.57 2.95
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 2.70 0.36 0.69 1.13 1.08 3.18 2.89
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 1.55 0.39 0.66 1.59 1.10 6.66 3.18
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 1.68 0.37 0.59 1.08 1.08 5.58 2.91
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 2.14 0.35 0.61 1.07 1.09 1.21 3.35
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 1.45 0.37 0.64 1.49 1.10 4.42 3.55
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 1.42 0.35 0.59 1.07 1.07 1.27 3.41
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 1.55 0.35 0.60 1.08 1.12 1.54 3.87
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 1.45 0.39 0.64 1.65 1.09 5.89 3.32
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 1.42 0.36 0.60 1.10 1.08 3.80 3.45
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ 3.90 0.37 0.68 1.07 1.16 2.47 2.91
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ 1.71 0.45 0.72 1.58 1.07 6.29 3.36
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ 3.78 0.39 0.62 1.05 1.07 3.60 3.20
PXLNET–BaseP_{\text{XLNET--Base}}PXLNET–Base​ (idf) 3.90 0.46 0.65 1.08 2.93 3.30 3.39
RXLNET–BaseR_{\text{XLNET--Base}}RXLNET–Base​ (idf) 1.51 0.45 0.82 1.78 1.12 10.77 3.13
FXLNET–BaseF_{\text{XLNET--Base}}FXLNET–Base​ (idf) 3.67 0.42 0.66 1.11 1.22 7.13 3.23
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ 3.94 0.37 0.71 1.10 21.10 1.85 2.90
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ 2.23 0.41 0.69 1.34 1.07 4.46 3.40
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ 3.84 0.36 0.60 1.03 1.07 3.38 3.22
PXLNET–LargeP_{\text{XLNET--Large}}PXLNET–Large​ (idf) 3.92 0.41 0.64 1.12 21.10 3.24 3.37
RXLNET–LargeR_{\text{XLNET--Large}}RXLNET–Large​ (idf) 1.60 0.43 0.78 1.70 1.09 6.13 3.20
FXLNET–LargeF_{\text{XLNET--Large}}FXLNET–Large​ (idf) 3.80 0.38 0.63 1.06 1.09 3.72 3.25
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 3.88 0.33 0.75 1.16 2.16 1.28 3.29
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 1.98 0.41 0.60 1.41 1.21 3.30 3.47
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 3.78 0.36 0.61 1.09 1.71 1.30 3.40
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 3.84 0.36 0.69 1.17 1.86 1.33 3.47
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 1.70 0.42 0.63 1.55 1.11 5.87 3.36
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 3.72 0.40 0.62 1.14 1.32 4.15 3.43

Table 26: Model selection accuracies (Hits@1) on to-English WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10−310^{-3}10−3. We bold the highest numbers for each language pair and direction.

Setting Metric en-cs en-de en-et en-fi en-ru en-tr en-zh
Unsupervised BLEU 0.151 0.611 0.617 0.087 0.519 0.029 0.515
CDER 0.163 0.663 0.731 0.081 0.541 0.032 0.552
CharacTER 0.135 0.737 0.639 0.492 0.543 0.027 0.667
ITER 0.000 0.691 0.734 0.112 0.534 0.031 –
Meteor++ – – – – – – –
NIST 0.182 0.662 0.549 0.083 0.537 0.033 0.553
PER 0.179 0.555 0.454 0.062 0.535 0.032 0.539
TER 0.175 0.657 0.550 0.065 0.545 0.029 0.551
UHH_TSKM – – – – – – –
WER 0.155 0.643 0.552 0.067 0.538 0.029 0.546
YiSi-0 0.154 0.674 0.622 0.356 0.523 0.383 0.600
YiSi-1 0.178 0.670 0.674 0.230 0.548 0.396 0.595
YiSi-1 srl – 0.708 – – – – 0.537
Supervised BEER 0.174 0.670 0.662 0.113 0.555 0.296 0.531
BLEND – – – – 0.559 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.181 0.665 0.771 0.077 0.550 0.373 0.550
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.184 0.728 0.722 0.146 0.544 0.031 0.657
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.185 0.703 0.764 0.081 0.548 0.032 0.629
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.175 0.713 0.769 0.080 0.542 0.031 0.549
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.177 0.725 0.752 0.178 0.538 0.031 0.628
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.178 0.721 0.766 0.081 0.543 0.030 0.594
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.175 0.669 0.748 0.079 0.550 0.314 0.582
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.195 0.671 0.770 0.222 0.555 0.034 0.658
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.187 0.670 0.775 0.099 0.552 0.034 0.615
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.163 0.664 0.750 0.091 0.550 0.288 0.578
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.191 0.681 0.770 0.231 0.548 0.033 0.645
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.180 0.672 0.774 0.127 0.550 0.033 0.616

表 26:在到英语的 WMT18 混合系统上的模型选择准确率(Hits@1)。我们报告 10 万次抽样的平均值,0.95 置信区间低于 10−310^{-3}10−3。每个语言对和方向上的最高值加粗。

设置 指标 en-cs en-de en-et en-fi en-ru en-tr en-zh
无监督 BLEU 0.151 0.611 0.617 0.087 0.519 0.029 0.515
CDER 0.163 0.663 0.731 0.081 0.541 0.032 0.552
CharacTER 0.135 0.737 0.639 0.492 0.543 0.027 0.667
ITER 0.000 0.691 0.734 0.112 0.534 0.031 –
Meteor++ – – – – – – –
NIST 0.182 0.662 0.549 0.083 0.537 0.033 0.553
PER 0.179 0.555 0.454 0.062 0.535 0.032 0.539
TER 0.175 0.657 0.550 0.065 0.545 0.029 0.551
UHH_TSKM – – – – – – –
WER 0.155 0.643 0.552 0.067 0.538 0.029 0.546
YiSi-0 0.154 0.674 0.622 0.356 0.523 0.383 0.600
YiSi-1 0.178 0.670 0.674 0.230 0.548 0.396 0.595
YiSi-1 srl – 0.708 – – – – 0.537
Supervised BEER 0.174 0.670 0.662 0.113 0.555 0.296 0.531
BLEND – – – – 0.559 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.181 0.665 0.771 0.077 0.550 0.373 0.550
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.184 0.728 0.722 0.146 0.544 0.031 0.657
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.185 0.703 0.764 0.081 0.548 0.032 0.629
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.175 0.713 0.769 0.080 0.542 0.031 0.549
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.177 0.725 0.752 0.178 0.538 0.031 0.628
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.178 0.721 0.766 0.081 0.543 0.030 0.594
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.175 0.669 0.748 0.079 0.550 0.314 0.582
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.195 0.671 0.770 0.222 0.555 0.034 0.658
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.187 0.670 0.775 0.099 0.552 0.034 0.615
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.163 0.664 0.750 0.091 0.550 0.288 0.578
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.191 0.681 0.770 0.231 0.548 0.033 0.645
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.180 0.672 0.774 0.127 0.550 0.033 0.616

Table 27: Mean Reciprocal Rank (MRR) of the top metric-rated system on to-English WMT18 hybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below 10−310^{-3}10−3. We bold the highest numbers for each language pair and direction.

Setting Metric en-cs en-de en-et en-fi en-ru en-tr en-zh
Unsupervised BLEU 0.363 0.764 0.766 0.323 0.714 0.205 0.666
CDER 0.371 0.803 0.851 0.319 0.729 0.210 0.700
CharacTER 0.346 0.853 0.781 0.667 0.732 0.205 0.809
ITER 0.044 0.825 0.853 0.365 0.717 0.210 –
Meteor++ – – – – – – –
NIST 0.393 0.803 0.710 0.326 0.726 0.211 0.698
PER 0.387 0.719 0.624 0.301 0.725 0.211 0.678
TER 0.384 0.798 0.708 0.305 0.733 0.209 0.695
UHH_TSKM – – – – – – –
WER 0.367 0.787 0.710 0.308 0.728 0.209 0.696
YiSi-0 0.370 0.811 0.775 0.553 0.715 0.602 0.753
YiSi-1 0.390 0.808 0.811 0.439 0.735 0.612 0.750
YiSi-1 srl – 0.835 – – – – 0.691
Supervised BEER 0.388 0.808 0.804 0.353 0.739 0.507 0.683
BLEND – – – – 0.742 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.395 0.805 0.876 0.314 0.736 0.586 0.694
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.401 0.849 0.844 0.368 0.732 0.212 0.802
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.400 0.832 0.872 0.317 0.735 0.214 0.775
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.390 0.839 0.875 0.320 0.730 0.213 0.691
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.395 0.847 0.864 0.398 0.727 0.212 0.776
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.395 0.844 0.873 0.319 0.730 0.212 0.739
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.391 0.808 0.862 0.316 0.735 0.522 0.733
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.413 0.809 0.876 0.435 0.738 0.216 0.803
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.404 0.809 0.878 0.333 0.737 0.216 0.767
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.377 0.805 0.863 0.326 0.735 0.497 0.729
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.409 0.816 0.876 0.444 0.733 0.214 0.793
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.396 0.810 0.878 0.355 0.735 0.214 0.767

表 27:在到英语的 WMT18 混合系统上,指标评为最优的系统的平均倒数排名(MRR)。我们报告 10 万次抽样的平均值,0.95 置信区间低于 10−310^{-3}10−3。每个语言对和方向上的最高值加粗。

设置 指标 en-cs en-de en-et en-fi en-ru en-tr en-zh
无监督 BLEU 0.363 0.764 0.766 0.323 0.714 0.205 0.666
CDER 0.371 0.803 0.851 0.319 0.729 0.210 0.700
CharacTER 0.346 0.853 0.781 0.667 0.732 0.205 0.809
ITER 0.044 0.825 0.853 0.365 0.717 0.210 –
Meteor++ – – – – – – –
NIST 0.393 0.803 0.710 0.326 0.726 0.211 0.698
PER 0.387 0.719 0.624 0.301 0.725 0.211 0.678
TER 0.384 0.798 0.708 0.305 0.733 0.209 0.695
UHH_TSKM – – – – – – –
WER 0.367 0.787 0.710 0.308 0.728 0.209 0.696
YiSi-0 0.370 0.811 0.775 0.553 0.715 0.602 0.753
YiSi-1 0.390 0.808 0.811 0.439 0.735 0.612 0.750
YiSi-1 srl – 0.835 – – – – 0.691
Supervised BEER 0.388 0.808 0.804 0.353 0.739 0.507 0.683
BLEND – – – – 0.742 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 0.395 0.805 0.876 0.314 0.736 0.586 0.694
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 0.401 0.849 0.844 0.368 0.732 0.212 0.802
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 0.400 0.832 0.872 0.317 0.735 0.214 0.775
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 0.390 0.839 0.875 0.320 0.730 0.213 0.691
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 0.395 0.847 0.864 0.398 0.727 0.212 0.776
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 0.395 0.844 0.873 0.319 0.730 0.212 0.739
PXLM–100P_{\text{XLM--100}}PXLM–100​ 0.391 0.808 0.862 0.316 0.735 0.522 0.733
RXLM–100R_{\text{XLM--100}}RXLM–100​ 0.413 0.809 0.876 0.435 0.738 0.216 0.803
FXLM–100F_{\text{XLM--100}}FXLM–100​ 0.404 0.809 0.878 0.333 0.737 0.216 0.767
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 0.377 0.805 0.863 0.326 0.735 0.497 0.729
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 0.409 0.816 0.876 0.444 0.733 0.214 0.793
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 0.396 0.810 0.878 0.355 0.735 0.214 0.767

Table 28: Absolute Difference (×100\times 100×100) of the top metric-rated and the top human-rated system on to-English WMT18 hybrid systems. Smaller difference indicate higher agreement with human scores. We report the average of 100K samples and the 0.95 confidence intervals are below 10−310^{-3}10−3. We bold the lowest numbers for each language pair and direction.

Setting Metric en-cs en-de en-et en-fi en-ru en-tr en-zh
Unsupervised BLEU 1.26 6.36 2.59 0.92 0.76 9.40 3.01
CDER 1.25 6.70 1.90 1.41 0.87 9.37 1.75
CharacTER 1.23 6.90 2.19 4.35 0.93 5.22 1.64
ITER 1.25 9.14 2.52 1.52 1.35 7.33 –
Meteor++ – – – – – – –
NIST 1.24 5.28 2.55 1.02 0.75 8.82 3.34
PER 1.25 6.62 4.92 7.43 0.68 9.76 2.31
TER 1.21 6.02 4.34 2.17 0.73 8.80 1.43
UHH_TSKM – – – – – – –
WER 1.22 6.15 4.19 2.43 0.72 9.28 1.49
YiSi-0 1.25 6.62 1.53 1.46 0.75 3.47 2.87
YiSi-1 1.22 6.27 1.21 1.13 0.71 3.51 3.33
YiSi-1 srl – 6.57 – – – – 3.71
Supervised BEER 1.21 5.96 1.84 0.77 0.74 3.36 1.96
BLEND – – – – 0.71 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 1.17 3.27 1.38 1.24 0.75 4.14 2.08
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 1.16 6.68 0.77 0.94 0.68 3.22 1.31
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 1.15 5.17 0.90 0.98 0.71 3.26 1.62
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 1.14 3.82 1.66 1.27 0.76 4.57 2.04
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 1.15 6.97 0.83 3.65 0.68 3.32 1.37
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 1.14 5.63 1.13 1.19 0.71 3.38 1.58
PXLM–100P_{\text{XLM--100}}PXLM–100​ 1.22 6.30 1.14 0.79 0.74 3.73 2.21
RXLM–100R_{\text{XLM--100}}RXLM–100​ 1.18 6.89 0.76 0.77 0.66 3.26 1.68
FXLM–100F_{\text{XLM--100}}FXLM–100​ 1.19 6.44 0.82 0.76 0.69 3.21 1.57
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 1.21 6.61 1.07 0.78 0.72 5.59 2.02
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 1.19 7.07 0.77 0.77 0.66 3.33 1.60
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 1.20 6.57 0.86 0.76 0.68 3.28 1.56

表 28:在到英语的 WMT18 混合系统上,指标评为最优与人工评为最优的系统之间人工分数的绝对差(×100\times 100×100)。差值越小,表示与人工分数越一致。我们报告 10 万次抽样的平均值,0.95 置信区间低于 10−310^{-3}10−3。每个语言对和方向上的最低值加粗。

设置 指标 en-cs en-de en-et en-fi en-ru en-tr en-zh
无监督 BLEU 1.26 6.36 2.59 0.92 0.76 9.40 3.01
CDER 1.25 6.70 1.90 1.41 0.87 9.37 1.75
CharacTER 1.23 6.90 2.19 4.35 0.93 5.22 1.64
ITER 1.25 9.14 2.52 1.52 1.35 7.33 –
Meteor++ – – – – – – –
NIST 1.24 5.28 2.55 1.02 0.75 8.82 3.34
PER 1.25 6.62 4.92 7.43 0.68 9.76 2.31
TER 1.21 6.02 4.34 2.17 0.73 8.80 1.43
UHH_TSKM – – – – – – –
WER 1.22 6.15 4.19 2.43 0.72 9.28 1.49
YiSi-0 1.25 6.62 1.53 1.46 0.75 3.47 2.87
YiSi-1 1.22 6.27 1.21 1.13 0.71 3.51 3.33
YiSi-1 srl – 6.57 – – – – 3.71
Supervised BEER 1.21 5.96 1.84 0.77 0.74 3.36 1.96
BLEND – – – – 0.71 – –
RUSE – – – – – – –
Pre-Trained PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ 1.17 3.27 1.38 1.24 0.75 4.14 2.08
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ 1.16 6.68 0.77 0.94 0.68 3.22 1.31
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ 1.15 5.17 0.90 0.98 0.71 3.26 1.62
PBERT–MultiP_{\text{BERT--Multi}}PBERT–Multi​ (idf) 1.14 3.82 1.66 1.27 0.76 4.57 2.04
RBERT–MultiR_{\text{BERT--Multi}}RBERT–Multi​ (idf) 1.15 6.97 0.83 3.65 0.68 3.32 1.37
FBERT–MultiF_{\text{BERT--Multi}}FBERT–Multi​ (idf) 1.14 5.63 1.13 1.19 0.71 3.38 1.58
PXLM–100P_{\text{XLM--100}}PXLM–100​ 1.22 6.30 1.14 0.79 0.74 3.73 2.21
RXLM–100R_{\text{XLM--100}}RXLM–100​ 1.18 6.89 0.76 0.77 0.66 3.26 1.68
FXLM–100F_{\text{XLM--100}}FXLM–100​ 1.19 6.44 0.82 0.76 0.69 3.21 1.57
PXLM–100P_{\text{XLM--100}}PXLM–100​ (idf) 1.21 6.61 1.07 0.78 0.72 5.59 2.02
RXLM–100R_{\text{XLM--100}}RXLM–100​ (idf) 1.19 7.07 0.77 0.77 0.66 3.33 1.60
FXLM–100F_{\text{XLM--100}}FXLM–100​ (idf) 1.20 6.57 0.86 0.76 0.68 3.28 1.56

Table 29: Pearson correlation on the 2015 COCO Captioning Challenge. The M1 and M2 measures are described in Section 4. We bold the best correlating task-specific and task-agnostic metrics in each setting Leic uses images as additional inputs. Numbers with ∗ are cited from Cui et al. (2018).

Metric M1 M2
Bleu-1 0.124∗ 0.135∗
Bleu-2 0.037∗ 0.048∗
Bleu-3 0.004∗ 0.016∗
Bleu-4 -0.019∗ -0.005∗
Meteor 0.606∗ 0.594∗
Rouge-L 0.090∗ 0.096∗
CIDEr 0.438∗ 0.440∗
Spice 0.759∗ 0.750∗
Leic 0.939∗ 0.949∗
BEER 0.491 0.562
EED 0.545 0.599
chrF++ 0.702 0.729
CharacTER 0.800 0.801
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.313 0.344
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.679 0.622
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.531 0.519
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.243 0.286
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.834 0.783
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.579 0.581
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.252 0.331
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.644 0.641
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.470 0.512
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.264 0.300
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.794 0.767
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.575 0.583
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.454 0.486
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.756 0.697
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.649 0.634
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.327 0.372
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.873 0.821
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.645 0.647
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ -0.223 -0.179
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.827 0.800
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.176 0.191
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) -0.256 -0.267
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.901 0.869
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.188 0.157
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ -0.105 -0.041
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.888 0.863
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.322 0.350
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.063 -0.011
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.917 0.889
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.519 0.453
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.129 0.208
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.820 0.823
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.546 0.592
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.081 0.099
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.906 0.875
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.605 0.596
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ -0.046 0.080
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ 0.409 0.506
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ 0.146 0.265
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ (idf) 0.006 0.145
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ (idf) 0.655 0.720
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ (idf) 0.270 0.391
PXLNet–LargeP_{\text{XLNet--Large}}PXLNet–Large​ -0.188 -0.115
RXLNet–LargeR_{\text{XLNet--Large}}RXLNet–Large​ 0.178 0.195
FXLNet–LargeF_{\text{XLNet--Large}}FXLNet–Large​ -0.014 0.036
PXLNet–LargeP_{\text{XLNet--Large}}PXLNet–Large​ (idf) -0.186 -0.072
RXLNet–LargeR_{\text{XLNet--Large}}RXLNet–Large​ (idf) 0.554 0.555
FXLNet–LargeF_{\text{XLNet--Large}}FXLNet–Large​ (idf) 0.151 0.234
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.230 0.220
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.333 0.263
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.297 0.243
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.266 0.275
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.700 0.640
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.499 0.470

表 29:2015 年 COCO Captioning Challenge 上的皮尔逊相关。M1 和 M2 两项度量在第 4 节中说明。我们在每种设置下,把任务专用与不限定任务的指标中相关最好的结果加粗。LEIC 把图像作为额外输入。带 ∗ 的数字引自 Cui et al.(2018)。

指标 M1 M2
Bleu-1 0.124∗ 0.135∗
Bleu-2 0.037∗ 0.048∗
Bleu-3 0.004∗ 0.016∗
Bleu-4 -0.019∗ -0.005∗
Meteor 0.606∗ 0.594∗
Rouge-L 0.090∗ 0.096∗
CIDEr 0.438∗ 0.440∗
Spice 0.759∗ 0.750∗
Leic 0.939∗ 0.949∗
BEER 0.491 0.562
EED 0.545 0.599
chrF++ 0.702 0.729
CharacTER 0.800 0.801
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.313 0.344
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.679 0.622
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.531 0.519
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.243 0.286
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.834 0.783
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.579 0.581
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.252 0.331
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.644 0.641
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.470 0.512
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.264 0.300
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.794 0.767
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.575 0.583
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.454 0.486
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.756 0.697
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.649 0.634
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.327 0.372
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.873 0.821
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.645 0.647
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ -0.223 -0.179
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.827 0.800
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.176 0.191
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) -0.256 -0.267
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.901 0.869
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.188 0.157
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ -0.105 -0.041
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.888 0.863
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.322 0.350
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.063 -0.011
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.917 0.889
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.519 0.453
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.129 0.208
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.820 0.823
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.546 0.592
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.081 0.099
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.906 0.875
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.605 0.596
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ -0.046 0.080
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ 0.409 0.506
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ 0.146 0.265
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ (idf) 0.006 0.145
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ (idf) 0.655 0.720
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ (idf) 0.270 0.391
PXLNet–LargeP_{\text{XLNet--Large}}PXLNet–Large​ -0.188 -0.115
RXLNet–LargeR_{\text{XLNet--Large}}RXLNet–Large​ 0.178 0.195
FXLNet–LargeF_{\text{XLNet--Large}}FXLNet–Large​ -0.014 0.036
PXLNet–LargeP_{\text{XLNet--Large}}PXLNet–Large​ (idf) -0.186 -0.072
RXLNet–LargeR_{\text{XLNet--Large}}RXLNet–Large​ (idf) 0.554 0.555
FXLNet–LargeF_{\text{XLNet--Large}}FXLNet–Large​ (idf) 0.151 0.234
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.230 0.220
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.333 0.263
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.297 0.243
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.266 0.275
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.700 0.640
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.499 0.470

Table 30: Area under ROC curve (AUC) on QQP and PAWSQQP datasets. The scores of trained DecATT (Parikh et al., 2016), DIIN (Gong et al., 2018), and fine-tuned BERT are reported by Zhang et al. (2019). We bold the best task-specific and task-agnostic metrics. Numbers with ∗ are scores on the held-out test set of QQP.

Type Method QQP PAWSQQP
Trained on QQP (supervised) DecAtt 0.939* 0.263
DIIN 0.952* 0.324
BERT 0.963* 0.351
Trained on QQP + PAWSQQP (supervised) DecAtt - 0.511
DIIN - 0.778
BERT - 0.831
Metric (Not trained on QQP or PAWSQQP) Bleu-1 0.737 0.402
Bleu-2 0.720 0.548
Bleu-3 0.712 0.527
Bleu-4 0.707 0.527
Meteor 0.755 0.532
Rouge-L 0.740 0.536
chrF++ 0.577 0.608
BEER 0.741 0.564
EED 0.743 0.611
CharacTER 0.698 0.650
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.750 0.654
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.739 0.655
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.755 0.654
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.766 0.665
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.752 0.665
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.770 0.664
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.742 0.615
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.729 0.617
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.746 0.614
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.752 0.618
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.737 0.619
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.756 0.617
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.752 0.706
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.740 0.710
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.756 0.707
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.766 0.713
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.751 0.718
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.769 0.714
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.746 0.657
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.736 0.656
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.751 0.654
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.760 0.666
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.745 0.666
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.765 0.664
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.757 0.687
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.744 0.685
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.761 0.685
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.773 0.691
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.757 0.697
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.777 0.693
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.763 0.767
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.750 0.772
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.766 0.770
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.783 0.756
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.767 0.764
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.784 0.759
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ 0.737 0.603
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ 0.731 0.607
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ 0.739 0.605
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ (idf) 0.751 0.625
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ (idf) 0.743 0.630
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ (idf) 0.751 0.626
PXLNet–LargeP_{\text{XLNet--Large}}PXLNet–Large​ 0.742 0.593
RXLNet–LargeR_{\text{XLNet--Large}}RXLNet–Large​ 0.734 0.598
FXLNet–LargeF_{\text{XLNet--Large}}FXLNet–Large​ 0.744 0.596
PXLNet–LargeP_{\text{XLNet--Large}}PXLNet–Large​ (idf) 0.759 0.604
RXLNet–LargeR_{\text{XLNet--Large}}RXLNet–Large​ (idf) 0.749 0.610
FXLNet–LargeF_{\text{XLNet--Large}}FXLNet–Large​ (idf) 0.760 0.606
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.734 0.600
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.725 0.604
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.737 0.602
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.757 0.596
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.745 0.603
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.759 0.600

表 30:QQP 和 PAWSQQP_{\mathrm{QQP}}QQP​ 数据集上的 ROC 曲线下面积(AUC)。已训练的 DecAtt(Parikh et al., 2016)、DIIN(Gong et al., 2018)以及微调 BERT 的分数由 Zhang et al.(2019)报告。任务专用与不限定任务的指标中最好的结果加粗。带 ∗ 的数字是 QQP 留出测试集上的分数。

类型 方法 QQP PAWSQQP
在 QQP 上训练(有监督) DecAtt 0.939* 0.263
DIIN 0.952* 0.324
BERT 0.963* 0.351
在 QQP + PAWSQQP_{\mathrm{QQP}}QQP​ 上训练(有监督) DecAtt - 0.511
DIIN - 0.778
BERT - 0.831
指标(未在 QQP 或 PAWSQQP_{\mathrm{QQP}}QQP​ 上训练) Bleu-1 0.737 0.402
Bleu-2 0.720 0.548
Bleu-3 0.712 0.527
Bleu-4 0.707 0.527
Meteor 0.755 0.532
Rouge-L 0.740 0.536
chrF++ 0.577 0.608
BEER 0.741 0.564
EED 0.743 0.611
CharacTER 0.698 0.650
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ 0.750 0.654
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ 0.739 0.655
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ 0.755 0.654
PBERT–BaseP_{\text{BERT--Base}}PBERT–Base​ (idf) 0.766 0.665
RBERT–BaseR_{\text{BERT--Base}}RBERT–Base​ (idf) 0.752 0.665
FBERT–BaseF_{\text{BERT--Base}}FBERT–Base​ (idf) 0.770 0.664
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ 0.742 0.615
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ 0.729 0.617
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ 0.746 0.614
PBERT–Base–MRPCP_{\text{BERT--Base--MRPC}}PBERT–Base–MRPC​ (idf) 0.752 0.618
RBERT–Base–MRPCR_{\text{BERT--Base--MRPC}}RBERT–Base–MRPC​ (idf) 0.737 0.619
FBERT–Base–MRPCF_{\text{BERT--Base--MRPC}}FBERT–Base–MRPC​ (idf) 0.756 0.617
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ 0.752 0.706
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ 0.740 0.710
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ 0.756 0.707
PBERT–LargeP_{\text{BERT--Large}}PBERT–Large​ (idf) 0.766 0.713
RBERT–LargeR_{\text{BERT--Large}}RBERT–Large​ (idf) 0.751 0.718
FBERT–LargeF_{\text{BERT--Large}}FBERT–Large​ (idf) 0.769 0.714
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ 0.746 0.657
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ 0.736 0.656
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ 0.751 0.654
PRoBERTa–BaseP_{\text{RoBERTa--Base}}PRoBERTa–Base​ (idf) 0.760 0.666
RRoBERTa–BaseR_{\text{RoBERTa--Base}}RRoBERTa–Base​ (idf) 0.745 0.666
FRoBERTa–BaseF_{\text{RoBERTa--Base}}FRoBERTa–Base​ (idf) 0.765 0.664
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ 0.757 0.687
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ 0.744 0.685
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ 0.761 0.685
PRoBERTa–LargeP_{\text{RoBERTa--Large}}PRoBERTa–Large​ (idf) 0.773 0.691
RRoBERTa–LargeR_{\text{RoBERTa--Large}}RRoBERTa–Large​ (idf) 0.757 0.697
FRoBERTa–LargeF_{\text{RoBERTa--Large}}FRoBERTa–Large​ (idf) 0.777 0.693
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ 0.763 0.767
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ 0.750 0.772
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ 0.766 0.770
PRoBERTa–Large–MNLIP_{\text{RoBERTa--Large--MNLI}}PRoBERTa–Large–MNLI​ (idf) 0.783 0.756
RRoBERTa–Large–MNLIR_{\text{RoBERTa--Large--MNLI}}RRoBERTa–Large–MNLI​ (idf) 0.767 0.764
FRoBERTa–Large–MNLIF_{\text{RoBERTa--Large--MNLI}}FRoBERTa–Large–MNLI​ (idf) 0.784 0.759
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ 0.737 0.603
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ 0.731 0.607
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ 0.739 0.605
PXLNet–BaseP_{\text{XLNet--Base}}PXLNet–Base​ (idf) 0.751 0.625
RXLNet–BaseR_{\text{XLNet--Base}}RXLNet–Base​ (idf) 0.743 0.630
FXLNet–BaseF_{\text{XLNet--Base}}FXLNet–Base​ (idf) 0.751 0.626
PXLNet–LargeP_{\text{XLNet--Large}}PXLNet–Large​ 0.742 0.593
RXLNet–LargeR_{\text{XLNet--Large}}RXLNet–Large​ 0.734 0.598
FXLNet–LargeF_{\text{XLNet--Large}}FXLNet–Large​ 0.744 0.596
PXLNet–LargeP_{\text{XLNet--Large}}PXLNet–Large​ (idf) 0.759 0.604
RXLNet–LargeR_{\text{XLNet--Large}}RXLNet–Large​ (idf) 0.749 0.610
FXLNet–LargeF_{\text{XLNet--Large}}FXLNet–Large​ (idf) 0.760 0.606
PXLM–EnP_{\text{XLM--En}}PXLM–En​ 0.734 0.600
RXLM–EnR_{\text{XLM--En}}RXLM–En​ 0.725 0.604
FXLM–EnF_{\text{XLM--En}}FXLM–En​ 0.737 0.602
PXLM–EnP_{\text{XLM--En}}PXLM–En​ (idf) 0.757 0.596
RXLM–EnR_{\text{XLM--En}}RXLM–En​ (idf) 0.745 0.603
FXLM–EnF_{\text{XLM--En}}FXLM–En​ (idf) 0.759 0.600