Neural Machine Translation of Rare Words with Subword Units Thanks: The research presented in this publication was conducted in cooperation with Samsung Electronics Polska sp. z o.o. - Samsung R&D Institute Poland.

用子词单元做稀有词的神经机器翻译 致谢：本出版物所报告的研究，是与 Samsung Electronics Polska sp. z o.o. - Samsung R&D Institute Poland 合作完成的。

Rico Sennrich    Barry Haddow    Alexandra Birch Affiliation: School of Informatics, University of Edinburgh Affiliation: {rico.sennrich,a.birch}@ed.ac.uk , bhaddow@inf.ed.ac.uk

Abstract

摘要

Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character $n$-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English$\to$German and English$\to$Russian by up to 1.1 and 1.3 Bleu, respectively.

神经机器翻译（NMT）模型通常在固定词表上运作，但翻译是一个开放词汇问题。先前工作处理未登录词翻译的办法，是退回到一本词典。本文引入一种更简单、也更有效的方法：把稀有词和未知词编码成子词单元的序列，从而使 NMT 模型具备开放词汇翻译的能力。其直觉是，有几类词可以通过比词更小的单位来翻译，例如专名（通过字符复制或转写）、复合词（通过组合翻译），以及同源词和借词（通过音系和形态变换）。我们讨论了不同分词技术是否合适，包括简单的字符 $n$ 元模型，以及一种基于 字节对编码 压缩算法的切分，并从经验上表明，在 WMT 15 的 English$\to$German 和 English$\to$Russian 翻译任务上，子词模型相对回退词典基线分别最高提升 1.1 和 1.3 Bleu。

1 Introduction

1 引言

Neural machine translation has recently shown impressive results [Kalchbrenner and Blunsom (2013, Sutskever et al. (2014, Bahdanau et al. (2015]. However, the translation of rare words is an open problem. The vocabulary of neural models is typically limited to $30\,000$–$50\,000$ words, but translation is an open-vocabulary problem, and especially for languages with productive word formation processes such as agglutination and compounding, translation models require mechanisms that go below the word level. As an example, consider compounds such as the German Abwasser|behandlungs|anlange ‘sewage water treatment plant’, for which a segmented, variable-length representation is intuitively more appealing than encoding the word as a fixed-length vector.

神经机器翻译近来取得了令人印象深刻的结果 [Kalchbrenner 与 Blunsom (2013, Sutskever 等人 (2014, Bahdanau 等人 (2015]。不过，稀有词的翻译仍是一个开放问题。神经模型的词表通常限制在 $30\,000$–$50\,000$ 个词，但翻译是开放词汇问题；尤其对具有黏着、复合这类能产构词过程的语言，翻译模型需要能够下到词级以下的机制。举个例子，像德语 Abwasser|behandlungs|anlange（“污水处理厂”）这样的复合词，用切分后的可变长表示，直观上就比把整个词编码成一个定长向量更有吸引力。

For word-level NMT models, the translation of out-of-vocabulary words has been addressed through a back-off to a dictionary look-up [Jean et al. (2015, Luong et al. (2015b]. We note that such techniques make assumptions that often do not hold true in practice. For instance, there is not always a 1-to-1 correspondence between source and target words because of variance in the degree of morphological synthesis between languages, like in our introductory compounding example. Also, word-level models are unable to translate or generate unseen words. Copying unknown words into the target text, as done by [Jean et al. (2015, Luong et al. (2015b], is a reasonable strategy for names, but morphological changes and transliteration is often required, especially if alphabets differ.

对词级 NMT 模型而言，未登录词的翻译是通过退回到词典查找来处理的 [Jean 等人 (2015, Luong 等人 (2015b]。我们指出，这类技术所做的假设在实践中往往并不成立。例如，由于语言之间形态综合程度不同，源词和目标词并不总是一一对应，就像我们开头那个复合词的例子。而且，词级模型无法翻译或生成未见过的词。把未知词复制到译文中，正如 [Jean 等人 (2015, Luong 等人 (2015b] 所做的那样，对姓名是一种合理策略，但往往还需要形态变化和转写，尤其是在字母表不同的时候。

We investigate NMT models that operate on the level of subword units. Our main goal is to model open-vocabulary translation in the NMT network itself, without requiring a back-off model for rare words. In addition to making the translation process simpler, we also find that the subword models achieve better accuracy for the translation of rare words than large-vocabulary models and back-off dictionaries, and are able to productively generate new words that were not seen at training time. Our analysis shows that the neural networks are able to learn compounding and transliteration from subword representations.

我们研究在子词单元这一层级上运作的 NMT 模型。主要目标是在 NMT 网络内部就对开放词汇翻译建模，而不再需要为稀有词准备回退模型。除了让翻译过程更简单之外，我们还发现，子词模型在稀有词翻译上的准确率高于大词表模型和回退词典，并且还能能产地生成训练时未见过的新词。我们的分析表明，神经网络能够从子词表示中学会复合和转写。

This paper has two main contributions:

本文有两项主要贡献：

• • We show that open-vocabulary neural machine translation is possible by encoding (rare) words via subword units. We find our architecture simpler and more effective than using large vocabularies and back-off dictionaries [Jean et al. (2015, Luong et al. (2015b].

• 我们表明，通过把（稀有）词编码成子词单元，开放词汇的神经机器翻译是可能的。与使用大词表和回退词典相比 [Jean 等人 (2015, Luong 等人 (2015b]，我们的结构更简单，也更有效。

• • We adapt byte pair encoding (BPE) [Gage (1994], a compression algorithm, to the task of word segmentation. BPE allows for the representation of an open vocabulary through a fixed-size vocabulary of variable-length character sequences, making it a very suitable word segmentation strategy for neural network models.

• 我们把一种压缩算法 字节对编码（BPE）[Gage (1994] 改造成一种分词任务。BPE 可以用一个固定大小、由可变长字符序列组成的词表，来表示开放词汇，因而非常适合作为神经网络模型的分词策略。

2 Neural Machine Translation

2 神经机器翻译

We follow the neural machine translation architecture by ?), which we will briefly summarize here. However, we note that our approach is not specific to this architecture.

我们沿用 Bahdanau 等人 (2015) 的神经机器翻译结构，下面作一简要概述。不过需要说明，我们的方法并不局限于这一结构。

The neural machine translation system is implemented as an encoder-decoder network with recurrent neural networks.

该神经机器翻译系统实现为一个带循环神经网络的编码器-解码器网络。

The encoder is a bidirectional neural network with gated recurrent units [Cho et al. (2014] that reads an input sequence $x=(x_{1},...,x_{m})$ and calculates a forward sequence of hidden states $(\overrightarrow{h}_{1},...,\overrightarrow{h}_{m})$, and a backward sequence $(\overleftarrow{h}_{1},...,\overleftarrow{h}_{m})$. The hidden states $\overrightarrow{h}_{j}$ and $\overleftarrow{h}_{j}$ are concatenated to obtain the annotation vector $h_{j}$.

编码器是一个带门控循环单元的双向神经网络 [Cho 等人 (2014]，它读入输入序列 $x=(x_{1},...,x_{m})$，计算前向隐状态序列 $(\overrightarrow{h}_{1},...,\overrightarrow{h}_{m})$ 和后向序列 $(\overleftarrow{h}_{1},...,\overleftarrow{h}_{m})$。把隐状态 $\overrightarrow{h}_{j}$ 与 $\overleftarrow{h}_{j}$ 拼接起来，就得到标注向量 $h_{j}$。

The decoder is a recurrent neural network that predicts a target sequence $y=(y_{1},...,y_{n})$. Each word $y_{i}$ is predicted based on a recurrent hidden state $s_{i}$, the previously predicted word $y_{i-1}$, and a context vector $c_{i}$. $c_{i}$ is computed as a weighted sum of the annotations $h_{j}$. The weight of each annotation $h_{j}$ is computed through an alignment model $\alpha_{ij}$, which models the probability that $y_{i}$ is aligned to $x_{j}$. The alignment model is a single-layer feedforward neural network that is learned jointly with the rest of the network through backpropagation.

解码器是一个循环神经网络，用来预测目标序列 $y=(y_{1},...,y_{n})$。每个词 $y_{i}$ 的预测依据是循环隐状态 $s_{i}$、先前预测的词 $y_{i-1}$，以及上下文向量 $c_{i}$。$c_{i}$ 计算为各标注 $h_{j}$ 的加权和。每个标注 $h_{j}$ 的权重由 对齐模型 $\alpha_{ij}$ 计算，该模型刻画 $y_{i}$ 与 $x_{j}$ 对齐的概率。对齐模型是一个单层前馈神经网络，与网络的其余部分通过反向传播联合学习。

A detailed description can be found in [Bahdanau et al. (2015]. Training is performed on a parallel corpus with stochastic gradient descent. For translation, a beam search with small beam size is employed.

详细描述见 [Bahdanau 等人 (2015]。训练在平行语料上用随机梯度下降进行。翻译时采用束较小的束搜索。

3 Subword Translation

3 子词翻译

The main motivation behind this paper is that the translation of some words is transparent in that they are translatable by a competent translator even if they are novel to him or her, based on a translation of known subword units such as morphemes or phonemes. Word categories whose translation is potentially transparent include:

本文的主要动机是：有些词的翻译是透明的，哪怕对一位合格译者来说它们是新词，只要能够翻译已知的子词单元，例如语素或音素，他或她仍然可以译出来。翻译可能透明的词类包括：

• • named entities. Between languages that share an alphabet, names can often be copied from source to target text. Transcription or transliteration may be required, especially if the alphabets or syllabaries differ. Example: Barack Obama (English; German) Барак Обама (Russian) \CJKnospace バラク・オバマ (ba-ra-ku o-ba-ma) (Japanese)

• 命名实体。在共用一套字母表的语言之间，姓名往往可以从源文复制到译文。若字母表或音节文字不同，则可能需要转写或音译。例如：Barack Obama（英语；德语）Барак Обама（俄语）バラク・オバマ (ba-ra-ku o-ba-ma)（日语）

• • cognates and loanwords. Cognates and loanwords with a common origin can differ in regular ways between languages, so that character-level translation rules are sufficient [Tiedemann (2012]. Example: claustrophobia (English) Klaustrophobie (German) Клаустрофобия (Klaustrofobiâ) (Russian)

• 同源词和借词。有共同来源的同源词和借词，在不同语言之间可以按规则的方式出现差异，因而字符级翻译规则就够用 [Tiedemann (2012]。例如：claustrophobia（英语）Klaustrophobie（德语）Клаустрофобия (Klaustrofobiâ)（俄语）

• • morphologically complex words. Words containing multiple morphemes, for instance formed via compounding, affixation, or inflection, may be translatable by translating the morphemes separately. Example: solar system (English) Sonnensystem (Sonne + System) (German) Naprendszer (Nap + Rendszer) (Hungarian)

• 形态复杂的词。包含多个语素的词，例如通过复合、加缀或屈折形成的词，也许可以靠分别翻译各个语素来翻译。例如：solar system（英语）Sonnensystem (Sonne + System)（德语）Naprendszer (Nap + Rendszer)（匈牙利语）

In an analysis of 100 rare tokens (not among the $50\,000$ most frequent types) in our German training data[^1], the majority of tokens are potentially translatable from English through smaller units. We find 56 compounds, 21 names, 6 loanwords with a common origin (emancipate$\to$emanzipieren), 5 cases of transparent affixation (sweetish ‘sweet’ + ‘-ish’ $\to$ süßlich ‘süß’ + ‘-lich’), 1 number and 1 computer language identifier.

在对我们德语训练数据里 100 个稀有符号（不在最频繁的 $50\,000$ 个类型之中）所做的分析中[^1]，大多数符号都有可能通过更小的单位从英语译出。我们找到 56 个复合词、21 个专名、6 个有共同来源的借词（emancipate$\to$emanzipieren）、5 处透明的加缀（sweetish ‘sweet’ + ‘-ish’ $\to$ süßlich ‘süß’ + ‘-lich’）、1 个数字和 1 个计算机语言标识符。

Our hypothesis is that a segmentation of rare words into appropriate subword units is sufficient to allow for the neural translation network to learn transparent translations, and to generalize this knowledge to translate and produce unseen words.[^2] We provide empirical support for this hypothesis in Sections 4 and 5. First, we discuss different subword representations.

我们的假说是：把稀有词切成合适的子词单元，就足以让神经翻译网络学会透明译法，并把这一知识推广到翻译和生成未见过的词。[^2] 我们在第 4 节和第 5 节为这一假说提供经验支持。首先，我们讨论不同的子词表示。

3.1 Related Work

3.1 相关工作

For Statistical Machine Translation (SMT), the translation of unknown words has been the subject of intensive research.

在统计机器翻译（SMT）中，未知词的翻译一直是密集研究的对象。

A large proportion of unknown words are names, which can just be copied into the target text if both languages share an alphabet. If alphabets differ, transliteration is required [Durrani et al. (2014]. Character-based translation has also been investigated with phrase-based models, which proved especially successful for closely related languages [Vilar et al. (2007, Tiedemann (2009, Neubig et al. (2012].

未知词里有很大一部分是姓名；如果两种语言共用字母表，就可以直接复制到译文中。如果字母表不同，则需要转写 [Durrani 等人 (2014]。基于短语的模型也研究过字符级翻译，在关系较近的语言上尤其成功 [Vilar 等人 (2007, Tiedemann (2009, Neubig 等人 (2012]。

The segmentation of morphologically complex words such as compounds is widely used for SMT, and various algorithms for morpheme segmentation have been investigated [Nießen and Ney (2000, Koehn and Knight (2003, Virpioja et al. (2007, Stallard et al. (2012]. Segmentation algorithms commonly used for phrase-based SMT tend to be conservative in their splitting decisions, whereas we aim for an aggressive segmentation that allows for open-vocabulary translation with a compact network vocabulary, and without having to resort to back-off dictionaries.

对复合词这类形态复杂的词做切分，在 SMT 中使用很广，各种语素切分算法都有人研究 [Nießen 与 Ney (2000, Koehn 与 Knight (2003, Virpioja 等人 (2007, Stallard 等人 (2012]。短语 SMT 里常用的切分算法，在决定是否切开时往往比较保守；而我们的目标是一种比较积极的切分，以便用紧凑的网络词表做开放词汇翻译，而不必再求助于回退词典。

The best choice of subword units may be task-specific. For speech recognition, phone-level language models have been used [Bazzi and Glass (2000]. ?) investigate subword language models, and propose to use syllables. For multilingual segmentation tasks, multilingual algorithms have been proposed [Snyder and Barzilay (2008]. We find these intriguing, but inapplicable at test time.

子词单元的最佳选择可能因任务而异。在语音识别中，使用过音素级语言模型 [Bazzi 与 Glass (2000]。Mikolov 等人 (2012) 研究了子词语言模型，并建议使用音节。对多语言切分任务，也有人提出过多语言算法 [Snyder 与 Barzilay (2008]。我们觉得这些想法很有吸引力，但在测试时并不适用。

Various techniques have been proposed to produce fixed-length continuous word vectors based on characters or morphemes [Luong et al. (2013, Botha and Blunsom (2014, Ling et al. (2015a, Kim et al. (2015]. An effort to apply such techniques to NMT, parallel to ours, has found no significant improvement over word-based approaches [Ling et al. (2015b]. One technical difference from our work is that the attention mechanism still operates on the level of words in the model by ?), and that the representation of each word is fixed-length. We expect that the attention mechanism benefits from our variable-length representation: the network can learn to place attention on different subword units at each step. Recall our introductory example Abwasserbehandlungsanlange, for which a subword segmentation avoids the information bottleneck of a fixed-length representation.

已有多种技术根据字符或语素来生成定长的连续词向量 [Luong 等人 (2013, Botha 与 Blunsom (2014, Ling 等人 (2015a, Kim 等人 (2015]。一项与我们并行、把这类技术用到 NMT 上的工作，发现相对词级方法没有显著提升 [Ling 等人 (2015b]。与我们工作的一个技术差别是：在 Ling 等人 (2015b) 的模型里，注意力机制仍然在词这一层级上运作，而且每个词的表示是定长的。我们预期注意力机制能从我们的可变长表示中获益：网络可以学会在每一步把注意力放在不同的子词单元上。回想开头那个 Abwasserbehandlungsanlange 的例子，子词切分可以避开定长表示的信息瓶颈。

Neural machine translation differs from phrase-based methods in that there are strong incentives to minimize the vocabulary size of neural models to increase time and space efficiency, and to allow for translation without back-off models. At the same time, we also want a compact representation of the text itself, since an increase in text length reduces efficiency and increases the distances over which neural models need to pass information.

神经机器翻译与基于短语的方法不同之处在于，它有很强的动机去尽量缩小神经模型的词表，以提高时间和空间效率，并允许在没有回退模型的情况下翻译。与此同时，我们也希望文本本身的表示尽量紧凑，因为文本变长会降低效率，并加大神经网络需要传递信息的距离。

A simple method to manipulate the trade-off between vocabulary size and text size is to use shortlists of unsegmented words, using subword units only for rare words. As an alternative, we propose a segmentation algorithm based on byte pair encoding (BPE), which lets us learn a vocabulary that provides a good compression rate of the text.

一种简单的办法来调节词表大小与文本长度之间的权衡，是对未切分的词使用短名单，只对稀有词使用子词单元。作为替代，我们提出一种基于字节对编码（BPE）的切分算法，让我们能够学到一个对文本压缩率很好的词表。

3.2 Byte Pair Encoding (BPE)

3.2 字节对编码（BPE）

Byte Pair Encoding (BPE) [Gage (1994] is a simple data compression technique that iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte. We adapt this algorithm for word segmentation. Instead of merging frequent pairs of bytes, we merge characters or character sequences.

字节对编码（BPE）[Gage (1994] 是一种简单的数据压缩技术，它迭代地把序列中最频繁的字节对替换成一个尚未使用的单个字节。我们把这一算法改用于分词。不是合并频繁的字节对，而是合并字符或字符序列。

Firstly, we initialize the symbol vocabulary with the character vocabulary, and represent each word as a sequence of characters, plus a special end-of-word symbol ‘$\cdot$’, which allows us to restore the original tokenization after translation. We iteratively count all symbol pairs and replace each occurrence of the most frequent pair (‘A’, ‘B’) with a new symbol ‘AB’. Each merge operation produces a new symbol which represents a character $n$-gram. Frequent character $n$-grams (or whole words) are eventually merged into a single symbol, thus BPE requires no shortlist. The final symbol vocabulary size is equal to the size of the initial vocabulary, plus the number of merge operations – the latter is the only hyperparameter of the algorithm.

首先，我们用字符词表初始化符号词表，并把每个词表示成字符序列，外加一个特殊的词尾符号 ‘$\cdot$’，以便在翻译之后还原原来的分词。我们迭代地统计所有符号对，并把最频繁的那一对（‘A’，‘B’）的每次出现都替换成新符号 ‘AB’。每一次合并都会产生一个表示字符 $n$ 元的新符号。频繁的字符 $n$ 元（或整个词）最终会被合并成单个符号，因此 BPE 不需要短名单。最终的符号词表大小等于初始词表大小再加上合并操作的次数——后者是该算法唯一的超参数。

import re, collections

def get_stats(vocab):
pairs = collections.defaultdict(int)
for word, freq in vocab.items():
symbols = word.split()
for i in range(len(symbols)-1):
pairs[symbols[i],symbols[i+1]] += freq
return pairs

def merge_vocab(pair, v_in):
v_out = {}
bigram = re.escape(' '.join(pair))
p = re.compile(r’(?<!\S)' + bigram + r’(?!\S)')
for word in v_in:
w_out = p.sub(''.join(pair), word)
v_out[w_out] = v_in[word]
return v_out

vocab = {'l o w ' : 5, 'l o w e r ' : 2,
'n e w e s t ':6, 'w i d e s t ':3}
num_merges = 10
for i in range(num_merges):
pairs = get_stats(vocab)
best = max(pairs, key=pairs.get)
vocab = merge_vocab(best, vocab)
print(best)

Algorithm 1 Learn BPE operations

import re, collections

def get_stats(vocab):
pairs = collections.defaultdict(int)
for word, freq in vocab.items():
symbols = word.split()
for i in range(len(symbols)-1):
pairs[symbols[i],symbols[i+1]] += freq
return pairs

def merge_vocab(pair, v_in):
v_out = {}
bigram = re.escape(' '.join(pair))
p = re.compile(r’(?<!\S)' + bigram + r’(?!\S)')
for word in v_in:
w_out = p.sub(''.join(pair), word)
v_out[w_out] = v_in[word]
return v_out

vocab = {'l o w ' : 5, 'l o w e r ' : 2,
'n e w e s t ':6, 'w i d e s t ':3}
num_merges = 10
for i in range(num_merges):
pairs = get_stats(vocab)
best = max(pairs, key=pairs.get)
vocab = merge_vocab(best, vocab)
print(best)

算法 1 学习 BPE 操作

For efficiency, we do not consider pairs that cross word boundaries. The algorithm can thus be run on the dictionary extracted from a text, with each word being weighted by its frequency. A minimal Python implementation is shown in Algorithm 1. In practice, we increase efficiency by indexing all pairs, and updating data structures incrementally.

为了提高效率，我们不考虑跨过词边界的符号对。因此，该算法可以在从文本中抽出的词典上运行，每个词按其频率加权。算法 1 给出一个最小的 Python 实现。实践中，我们通过给所有符号对建立索引、并增量更新数据结构，来进一步提高效率。

The main difference to other compression algorithms, such as Huffman encoding, which have been proposed to produce a variable-length encoding of words for NMT [Chitnis and DeNero (2015], is that our symbol sequences are still interpretable as subword units, and that the network can generalize to translate and produce new words (unseen at training time) on the basis of these subword units.

与其他已被提出、用来为 NMT 生成词的可变长编码的压缩算法（例如 Huffman 编码）[Chitnis 与 DeNero (2015] 相比，主要差别在于：我们的符号序列仍然可以当作子词单元来解释，而且网络可以在这些子词单元的基础上，推广去翻译和生成（训练时未见过的）新词。

r $\cdot$	$\rightarrow$	r$\cdot$
l o	$\rightarrow$	lo
lo w	$\rightarrow$	low
e r$\cdot$	$\rightarrow$	er$\cdot$

Figure 1: BPE merge operations learned from dictionary {‘low’, ‘lowest’, ‘newer’, ‘wider’}.

图 1：从词典 {‘low’，‘lowest’，‘newer’，‘wider’} 上学到的 BPE 合并操作。

Figure 1 shows a toy example of learned BPE operations. At test time, we first split words into sequences of characters, then apply the learned operations to merge the characters into larger, known symbols. This is applicable to any word, and allows for open-vocabulary networks with fixed symbol vocabularies.[^3] In our example, the OOV ‘lower’ would be segmented into ‘low er$\cdot$’.

图 1 给出一个学到的 BPE 操作的玩具例子。测试时，我们先把词切成字符序列，再应用学到的操作，把字符合并成更大的、已知的符号。这对任意词都适用，从而允许用固定的符号词表来做开放词汇网络。[^3] 在我们的例子中，未登录词 ‘lower’ 会被切成 ‘low er$\cdot$’。

We evaluate two methods of applying BPE: learning two independent encodings, one for the source, one for the target vocabulary, or learning the encoding on the union of the two vocabularies (which we call joint BPE).[^4] The former has the advantage of being more compact in terms of text and vocabulary size, and having stronger guarantees that each subword unit has been seen in the training text of the respective language, whereas the latter improves consistency between the source and the target segmentation. If we apply BPE independently, the same name may be segmented differently in the two languages, which makes it harder for the neural models to learn a mapping between the subword units. To increase the consistency between English and Russian segmentation despite the differing alphabets, we transliterate the Russian vocabulary into Latin characters with ISO-9 to learn the joint BPE encoding, then transliterate the BPE merge operations back into Cyrillic to apply them to the Russian training text.[^5]

我们评估了两种应用 BPE 的方法：分别学习两套独立编码，一套用于源语言，一套用于目标语言；或者在两个词表的并集上学习编码（我们称之为 joint BPE）。[^4] 前者的优点是在文本和词表大小上更紧凑，并且更能保证每个子词单元都在相应语言的训练文本中出现过；后者则能提高源端与目标端切分之间的一致性。如果独立地应用 BPE，同一个姓名在两种语言里可能会被切成不同的样子，这会让神经模型更难学会子词单元之间的对应。为了在字母表不同的情况下提高英语与俄语切分的一致性，我们先按 ISO-9 把俄语词表转写成拉丁字符，用来学习联合 BPE 编码，再把 BPE 合并操作转写回西里尔字母，应用到俄语训练文本上。[^5]

4 Evaluation

4 评估

We aim to answer the following empirical questions:

我们希望回答以下经验问题：

• • Can we improve the translation of rare and unseen words in neural machine translation by representing them via subword units?

• 通过把稀有词和未见词表示成子词单元，我们能否改善神经机器翻译对这些词的翻译？

• • Which segmentation into subword units performs best in terms of vocabulary size, text size, and translation quality?

• 就词表大小、文本长度和翻译质量而言，哪一种切成子词单元的办法最好？

We perform experiments on data from the shared translation task of WMT 2015. For English$\to$German, our training set consists of 4.2 million sentence pairs, or approximately 100 million tokens. For English$\to$Russian, the training set consists of 2.6 million sentence pairs, or approximately 50 million tokens. We tokenize and truecase the data with the scripts provided in Moses [Koehn et al. (2007]. We use newstest2013 as development set, and report results on newstest2014 and newstest2015.

我们在 WMT 2015 共享翻译任务的数据上做实验。对 English$\to$German，训练集包含 420 万句对，约 1 亿个 token。对 English$\to$Russian，训练集包含 260 万句对，约 5000 万个 token。我们用 Moses 提供的脚本对数据做分词和 truecase [Koehn 等人 (2007]。开发集使用 newstest2013，并在 newstest2014 和 newstest2015 上报告结果。

We report results with Bleu (mteval-v13a.pl), and chrF3 [Popović (2015], a character n-gram F3 score which was found to correlate well with human judgments, especially for translations out of English [Stanojević et al. (2015]. Since our main claim is concerned with the translation of rare and unseen words, we report separate statistics for these. We measure these through unigram F1, which we calculate as the harmonic mean of clipped unigram precision and recall.[^6]

我们报告 Bleu（mteval-v13a.pl）以及 chrF3 [Popović (2015]，后者是一种字符 n 元 F3 分数，被发现与人工评判相关性很好，尤其是从英语译出时 [Stanojević 等人 (2015]。由于我们的主要主张涉及稀有词和未见词的翻译，我们对这些词单独报告统计。衡量办法是 unigram F1，计算为截断的 unigram 精确率与召回率的调和平均。[^6]

We perform all experiments with Groundhog[^7] [Bahdanau et al. (2015]. We generally follow settings by previous work [Bahdanau et al. (2015, Jean et al. (2015]. All networks have a hidden layer size of 1000, and an embedding layer size of 620. Following ?), we only keep a shortlist of $\tau=30000$ words in memory.

全部实验都使用 Groundhog[^7] [Bahdanau 等人 (2015]。我们大体沿用先前工作的设定 [Bahdanau 等人 (2015, Jean 等人 (2015]。所有网络的隐层大小为 1000，嵌入层大小为 620。按照 Jean 等人 (2015) 的做法，我们只在内存中保留 $\tau=30000$ 个词的短名单。

During training, we use Adadelta [Zeiler (2012], a minibatch size of 80, and reshuffle the training set between epochs. We train a network for approximately 7 days, then take the last 4 saved models (models being saved every 12 hours), and continue training each with a fixed embedding layer (as suggested by [Jean et al. (2015]) for 12 hours. We perform two independent training runs for each models, once with cut-off for gradient clipping [Pascanu et al. (2013] of 5.0, once with a cut-off of 1.0 – the latter produced better single models for most settings. We report results of the system that performed best on our development set (newstest2013), and of an ensemble of all 8 models.

训练时我们使用 Adadelta [Zeiler (2012]，minibatch 大小为 80，并在各个 epoch 之间打乱训练集。每个网络大约训练 7 天，然后取最后 4 个保存的模型（每 12 小时保存一次），并按 [Jean 等人 (2015] 的建议，把嵌入层固定后再各自继续训练 12 小时。我们对每种模型做两次独立训练，一次把梯度裁剪 [Pascanu 等人 (2013] 的截断值设为 5.0，一次设为 1.0——后者在大多数设定下能得到更好的单模型。我们报告在开发集（newstest2013）上表现最好的那个系统的结果，以及全部 8 个模型集成的结果。

We use a beam size of 12 for beam search, with probabilities normalized by sentence length. We use a bilingual dictionary based on fast-align [Dyer et al. (2013]. For our baseline, this serves as back-off dictionary for rare words. We also use the dictionary to speed up translation for all experiments, only performing the softmax over a filtered list of candidate translations (like ?), we use $K=30000$; $K^{\prime}=10$).

束搜索的束大小为 12，概率按句子长度归一化。我们使用基于 fast-align [Dyer 等人 (2013] 的双语词典。对基线系统而言，它用作稀有词的回退词典。我们还在全部实验中用这本词典来加快翻译，只在过滤后的候选译文列表上做 softmax（与 Jean 等人 (2015) 一样，我们取 $K=30000$；$K^{\prime}=10$）。

4.1 Subword statistics

4.1 子词统计

Apart from translation quality, which we will verify empirically, our main objective is to represent an open vocabulary through a compact fixed-size subword vocabulary, and allow for efficient training and decoding.[^8]

除了将从经验上核实的翻译质量之外，我们的主要目标是用一个紧凑、固定大小的子词词表来表示开放词汇，并允许高效的训练与解码。[^8]

Statistics for different segmentations of the German side of the parallel data are shown in Table 1. A simple baseline is the segmentation of words into character $n$-grams.[^9] Character $n$-grams allow for different trade-offs between sequence length (# tokens) and vocabulary size (# types), depending on the choice of $n$. The increase in sequence length is substantial; one way to reduce sequence length is to leave a shortlist of the $k$ most frequent word types unsegmented. Only the unigram representation is truly open-vocabulary. However, the unigram representation performed poorly in preliminary experiments, and we report translation results with a bigram representation, which is empirically better, but unable to produce some tokens in the test set with the training set vocabulary.

平行数据德语一侧不同切分的统计见表 1。一个简单的基线是把词切成字符 $n$ 元。[^9] 字符 $n$ 元允许在序列长度（token 数）与词表大小（类型数）之间做不同的权衡，取决于 $n$ 的选择。序列长度的增加相当可观；一种缩短序列的办法，是让最频繁的 $k$ 个词类型保持不切分。只有 unigram 表示才真正是开放词汇的。不过，unigram 表示在初步实验中表现不佳，我们报告的是二元语法表示的翻译结果；后者在经验上更好，但无法用训练集词表产生测试集中的某些 token。

We report statistics for several word segmentation techniques that have proven useful in previous SMT research, including frequency-based compound splitting [Koehn and Knight (2003], rule-based hyphenation [Liang (1983], and Morfessor [Creutz and Lagus (2002]. We find that they only moderately reduce vocabulary size, and do not solve the unknown word problem, and we thus find them unsuitable for our goal of open-vocabulary translation without back-off dictionary.

我们还报告了若干在先前 SMT 研究中被证明有用的分词技术的统计，包括基于频率的复合词拆分 [Koehn 与 Knight (2003]、基于规则的连字符切分 [Liang (1983]，以及 Morfessor [Creutz 与 Lagus (2002]。我们发现，它们只能温和地缩小词表，并且解决不了未知词问题，因此不适合我们“不靠回退词典做开放词汇翻译”这一目标。

BPE meets our goal of being open-vocabulary, and the learned merge operations can be applied to the test set to obtain a segmentation with no unknown symbols.[^10] Its main difference from the character-level model is that the more compact representation of BPE allows for shorter sequences, and that the attention model operates on variable-length units.[^11] Table 1 shows BPE with $59\,500$ merge operations, and joint BPE with $89\,500$ operations.

BPE 达到了开放词汇这一目标，学到的合并操作可以应用到测试集上，得到不含未知符号的切分。[^10] 它与字符级模型的主要差别在于：BPE 更紧凑的表示允许更短的序列，而且注意力模型运作在可变长单元上。[^11] 表 1 给出了 $59\,500$ 次合并的 BPE，以及 $89\,500$ 次操作的联合 BPE。

In practice, we did not include infrequent subword units in the NMT network vocabulary, since there is noise in the subword symbol sets, e.g. because of characters from foreign alphabets. Hence, our network vocabularies in Table 2 are typically slightly smaller than the number of types in Table 1.

实践中，我们没有把低频子词单元纳入 NMT 网络词表，因为子词符号集合中存在噪声，例如来自外文字母表的字符。因此，表 2 中我们的网络词表通常略小于表 1 中的类型数。

segmentation	# tokens	# types	# UNK
none	100 m	$1\,750\,000$	1079
characters	550 m	$3000$	0
character bigrams	306 m	$20\,000$	34
character trigrams	214 m	$120\,000$	59
compound splitting△	102 m	$1\,100\,000$	643
morfessor*	109 m	$544\,000$	237
hyphenation⋄	186 m	$404\,000$	230
BPE	112 m	$63\,000$	0
BPE (joint)	111 m	$82\,000$	32
character bigrams	129 m	$69\,000$	34
(shortlist: $50\,000$)

Table 1: Corpus statistics for German training corpus with different word segmentation techniques. #UNK: number of unknown tokens in newstest2013. $\triangle$: [Koehn and Knight (2003]; *: [Creutz and Lagus (2002]; $\diamond$: [Liang (1983].

切分	token 数	类型数	UNK 数
none	100 m	$1\,750\,000$	1079
characters	550 m	$3000$	0
character bigrams	306 m	$20\,000$	34
character trigrams	214 m	$120\,000$	59
compound splitting△	102 m	$1\,100\,000$	643
morfessor*	109 m	$544\,000$	237
hyphenation⋄	186 m	$404\,000$	230
BPE	112 m	$63\,000$	0
BPE (joint)	111 m	$82\,000$	32
character bigrams	129 m	$69\,000$	34
(shortlist: $50\,000$)

表 1：采用不同分词技术时，德语训练语料的统计。#UNK：newstest2013 中未知 token 的数目。$\triangle$：[Koehn 与 Knight (2003]；*：[Creutz 与 Lagus (2002]；$\diamond$：[Liang (1983]。

4.2 Translation experiments

4.2 翻译实验

name	segmentation	shortlist	source	target	single	ens-8	single	ens-8	all	rare	OOV
syntax-based [Sennrich and Haddow (2015]					24.4	-	55.3	-	59.1	46.0	37.7
WUnk	-	-	$300\,000$	$500\,000$	20.6	22.8	47.2	48.9	56.7	20.4	0.0
WDict	-	-	$300\,000$	$500\,000$	22.0	24.2	50.5	52.4	58.1	36.8	36.8
C2-50k	char-bigram	$50\,000$	$60\,000$	$60\,000$	22.8	25.3	51.9	53.5	58.4	40.5	30.9
BPE-60k	BPE	-	$60\,000$	$60\,000$	21.5	24.5	52.0	53.9	58.4	40.9	29.3
BPE-J90k	BPE (joint)	-	$90\,000$	$90\,000$	22.8	24.7	51.7	54.1	58.5	41.8	33.6

Table 2: English$\to$German translation performance (Bleu, chrF3 and unigram F1) on newstest2015. Ens-8: ensemble of 8 models. Best NMT system in bold. Unigram F1 (with ensembles) is computed for all words ($n=44085$), rare words (not among top $50\,000$ in training set; $n=2900$), and OOVs (not in training set; $n=1168$).

			词表	Bleu	chrF3	unigram F1 (%)
名称	切分	短名单	源	目标	单模型	ens-8	单模型	ens-8	全部	稀有	OOV
syntax-based [Sennrich and Haddow (2015]					24.4	-	55.3	-	59.1	46.0	37.7
WUnk	-	-	$300\,000$	$500\,000$	20.6	22.8	47.2	48.9	56.7	20.4	0.0
WDict	-	-	$300\,000$	$500\,000$	22.0	24.2	50.5	52.4	58.1	36.8	36.8
C2-50k	char-bigram	$50\,000$	$60\,000$	$60\,000$	22.8	25.3	51.9	53.5	58.4	40.5	30.9
BPE-60k	BPE	-	$60\,000$	$60\,000$	21.5	24.5	52.0	53.9	58.4	40.9	29.3
BPE-J90k	BPE (joint)	-	$90\,000$	$90\,000$	22.8	24.7	51.7	54.1	58.5	41.8	33.6

表 2：English$\to$German 在 newstest2015 上的翻译表现（Bleu、chrF3 和 unigram F1）。Ens-8：8 个模型的集成。最好的 NMT 系统加粗。Unigram F1（用集成）分别对全部词（$n=44085$）、稀有词（不在训练集最频繁的 $50\,000$ 个之中；$n=2900$）以及 OOV（不在训练集中；$n=1168$）计算。

English$\to$German translation results are shown in Table 2; English$\to$Russian results in Table 3.

English$\to$German 的翻译结果见表 2；English$\to$Russian 的结果见表 3。

Our baseline WDict is a word-level model with a back-off dictionary. It differs from WUnk in that the latter uses no back-off dictionary, and just represents out-of-vocabulary words as UNK[^12]. The back-off dictionary improves unigram F1 for rare and unseen words, although the improvement is smaller for English$\to$Russian, since the back-off dictionary is incapable of transliterating names.

我们的基线 WDict 是一个带回退词典的词级模型。它与 WUnk 的差别在于，后者不使用回退词典，只是把未登录词表示成 UNK[^12]。回退词典改善了稀有词和未见词的 unigram F1，不过对 English$\to$Russian 的改善较小，因为回退词典无法转写姓名。

All subword systems operate without a back-off dictionary. We first focus on unigram F1, where all systems improve over the baseline, especially for rare words (36.8%$\rightarrow$41.8% for EN$\to$DE; 26.5%$\rightarrow$29.7% for EN$\to$RU). For OOVs, the baseline strategy of copying unknown words works well for English$\to$German. However, when alphabets differ, like in English$\to$Russian, the subword models do much better.

所有子词系统都不使用回退词典。我们先看 unigram F1：所有系统都超过了基线，尤其是在稀有词上（EN$\to$DE 为 36.8%$\to$41.8%；EN$\to$RU 为 26.5%$\to$29.7%）。对 OOV，把未知词从源文复制过去这一基线策略，对 English$\to$German 效果很好。不过当字母表不同时，例如 English$\to$Russian，子词模型要好得多。

Unigram F1 scores indicate that learning the BPE symbols on the vocabulary union (BPE-J90k) is more effective than learning them separately (BPE-60k), and more effective than using character bigrams with a shortlist of $50\,000$ unsegmented words (C2-50k), but all reported subword segmentations are viable choices and outperform the back-off dictionary baseline.

Unigram F1 分数表明，在词表并集上学习 BPE 符号（BPE-J90k）比分开学习（BPE-60k）更有效，也比使用带 $50\,000$ 个未切分词短名单的字符二元语法（C2-50k）更有效；不过所报告的各种子词切分都是可行的选择，并且都超过了回退词典基线。

Our subword representations cause big improvements in the translation of rare and unseen words, but these only constitute 9-11% of the test sets. Since rare words tend to carry central information in a sentence, we suspect that Bleu and chrF3 underestimate their effect on translation quality. Still, we also see improvements over the baseline in total unigram F1, as well as Bleu and chrF3, and the subword ensembles outperform the WDict baseline by 0.3–1.3 Bleu and 0.6–2 chrF3. There is some inconsistency between Bleu and chrF3, which we attribute to the fact that Bleu has a precision bias, and chrF3 a recall bias.

我们的子词表示在稀有词和未见词的翻译上带来很大提升，但这些词只占测试集的 9–11%。由于稀有词往往承载句子中的关键信息，我们怀疑 Bleu 和 chrF3 低估了它们对翻译质量的影响。尽管如此，我们在全部词的 unigram F1，以及 Bleu 和 chrF3 上，也都看到了对基线的提升；子词集成相对 WDict 基线高出 0.3–1.3 Bleu 和 0.6–2 chrF3。Bleu 与 chrF3 之间有一些不一致，我们将其归因于 Bleu 偏向精确率，而 chrF3 偏向召回率。

For English$\to$German, we observe the best Bleu score of 25.3 with C2-50k, but the best chrF3 score of 54.1 with BPE-J90k. For comparison to the (to our knowledge) best non-neural MT system on this data set, we report syntax-based SMT results [Sennrich and Haddow (2015]. We observe that our best systems outperform the syntax-based system in terms of Bleu, but not in terms of chrF3. Regarding other neural systems, ?) report a Bleu score of 25.9 on newstest2015, but we note that they use an ensemble of 8 independently trained models, and also report strong improvements from applying dropout, which we did not use. We are confident that our improvements to the translation of rare words are orthogonal to improvements achievable through other improvements in the network architecture, training algorithm, or better ensembles.

对 English$\to$German，我们观察到最好的 Bleu 是 C2-50k 的 25.3，但最好的 chrF3 是 BPE-J90k 的 54.1。为了与（据我们所知）这一数据集上最好的非神经 MT 系统比较，我们报告了基于句法的 SMT 结果 [Sennrich 与 Haddow (2015]。我们观察到，最好的系统在 Bleu 上超过了句法系统，但在 chrF3 上没有超过。至于其他神经系统，Luong 等人 (2015a) 在 newstest2015 上报告了 25.9 的 Bleu，不过需要说明，他们使用的是 8 个独立训练模型的集成，并且还报告了使用 dropout 带来的明显提升，而我们没有使用 dropout。我们有信心，我们对稀有词翻译的改进，与通过改进网络结构、训练算法或更好的集成所能获得的改进是正交的。

For English$\to$Russian, the state of the art is the phrase-based system by ?). It outperforms our WDict baseline by 1.5 Bleu. The subword models are a step towards closing this gap, and BPE-J90k yields an improvement of 1.3 Bleu, and 2.0 chrF3, over WDict.

对 English$\to$Russian，当时最好的是 Haddow 等人 (2015) 的基于短语的系统。它比我们的 WDict 基线高出 1.5 Bleu。子词模型是在缩小这一差距上迈出的一步，BPE-J90k 相对 WDict 提升了 1.3 Bleu 和 2.0 chrF3。

name	segmentation	shortlist	source	target	single	ens-8	single	ens-8	all	rare	OOV
phrase-based [Haddow et al. (2015]					24.3	-	53.8	-	56.0	31.3	16.5
WUnk	-	-	$300\,000$	$500\,000$	18.8	22.4	46.5	49.9	54.2	25.2	0.0
WDict	-	-	$300\,000$	$500\,000$	19.1	22.8	47.5	51.0	54.8	26.5	6.6
C2-50k	char-bigram	$50\,000$	$60\,000$	$60\,000$	20.9	24.1	49.0	51.6	55.2	27.8	17.4
BPE-60k	BPE	-	$60\,000$	$60\,000$	20.5	23.6	49.8	52.7	55.3	29.7	15.6
BPE-J90k	BPE (joint)	-	$90\,000$	$100\,000$	20.4	24.1	49.7	53.0	55.8	29.7	18.3

Table 3: English$\to$Russian translation performance (Bleu, chrF3 and unigram F1) on newstest2015. Ens-8: ensemble of 8 models. Best NMT system in bold. Unigram F1 (with ensembles) is computed for all words ($n=55654$), rare words (not among top $50\,000$ in training set; $n=5442$), and OOVs (not in training set; $n=851$).

			词表	Bleu	chrF3	unigram F1 (%)
名称	切分	短名单	源	目标	单模型	ens-8	单模型	ens-8	全部	稀有	OOV
phrase-based [Haddow et al. (2015]					24.3	-	53.8	-	56.0	31.3	16.5
WUnk	-	-	$300\,000$	$500\,000$	18.8	22.4	46.5	49.9	54.2	25.2	0.0
WDict	-	-	$300\,000$	$500\,000$	19.1	22.8	47.5	51.0	54.8	26.5	6.6
C2-50k	char-bigram	$50\,000$	$60\,000$	$60\,000$	20.9	24.1	49.0	51.6	55.2	27.8	17.4
BPE-60k	BPE	-	$60\,000$	$60\,000$	20.5	23.6	49.8	52.7	55.3	29.7	15.6
BPE-J90k	BPE (joint)	-	$90\,000$	$100\,000$	20.4	24.1	49.7	53.0	55.8	29.7	18.3

表 3：English$\to$Russian 在 newstest2015 上的翻译表现（Bleu、chrF3 和 unigram F1）。Ens-8：8 个模型的集成。最好的 NMT 系统加粗。Unigram F1（用集成）分别对全部词（$n=55654$）、稀有词（不在训练集最频繁的 $50\,000$ 个之中；$n=5442$）以及 OOV（不在训练集中；$n=851$）计算。

As a further comment on our translation results, we want to emphasize that performance variability is still an open problem with NMT. On our development set, we observe differences of up to 1 Bleu between different models. For single systems, we report the results of the model that performs best on dev (out of 8), which has a stabilizing effect, but how to control for randomness deserves further attention in future research.

关于翻译结果，我们还想强调：性能波动仍然是 NMT 中的一个开放问题。在开发集上，我们观察到不同模型之间最多可相差 1 Bleu。对单系统，我们报告的是（8 个之中）在开发集上表现最好的那个模型的结果，这有一定稳定作用，但如何控制随机性，值得在今后的研究中给予更多注意。

5 Analysis

5 分析

5.1 Unigram accuracy

5.1 Unigram 准确率

Our main claims are that the translation of rare and unknown words is poor in word-level NMT models, and that subword models improve the translation of these word types. To further illustrate the effect of different subword segmentations on the translation of rare and unseen words, we plot target-side words sorted by their frequency in the training set.[^13] To analyze the effect of vocabulary size, we also include the system C2-3/500k, which is a system with the same vocabulary size as the WDict baseline, and character bigrams to represent unseen words.

我们的主要主张是：词级 NMT 模型对稀有词和未知词的翻译很差，而子词模型能够改善这些词类的翻译。为了进一步说明不同子词切分对稀有词和未见词翻译的影响，我们按目标侧的词在训练集中的频率排序作图。[^13] 为了分析词表大小的影响，我们还加入了系统 C2-3/500k，它的词表大小与 WDict 基线相同，并用字符二元语法来表示未见词。

Figure 2 shows results for the English–German ensemble systems on newstest2015. Unigram F1 of all systems tends to decrease for lower-frequency words. The baseline system has a spike in F1 for OOVs, i.e. words that do not occur in the training text. This is because a high proportion of OOVs are names, for which a copy from the source to the target text is a good strategy for English$\to$German.

图 2 给出 English–German 集成系统在 newstest2015 上的结果。所有系统的 unigram F1 都倾向于随词频降低而下降。基线系统在 OOV，也就是训练文本中未出现的词上，F1 有一个尖峰。这是因为 OOV 中有很高比例是专名，对 English$\to$German 而言，从源文复制到译文是一种好策略。

The systems with a target vocabulary of $500\,000$ words mostly differ in how well they translate words with rank > $500\,000$. A back-off dictionary is an obvious improvement over producing UNK, but the subword system C2-3/500k achieves better performance. Note that all OOVs that the back-off dictionary produces are words that are copied from the source, usually names, while the subword systems can productively form new words such as compounds.

目标词表为 $500\,000$ 个词的系统，主要差别在于它们把频率排名 > $500\,000$ 的词译得有多好。回退词典相对产生 UNK 是一个明显的改进，但子词系统 C2-3/500k 取得了更好的表现。需要注意，回退词典产生的全部 OOV 都是从源文复制过来的词，通常是专名；而子词系统还能能产地构成复合词这样的新词。

For the $50\,000$ most frequent words, the representation is the same for all neural networks, and all neural networks achieve comparable unigram F1 for this category. For the interval between frequency rank $50\,000$ and $500\,000$, the comparison between C2-3/500k and C2-50k unveils an interesting difference. The two systems only differ in the size of the shortlist, with C2-3/500k representing words in this interval as single units, and C2-50k via subword units. We find that the performance of C2-3/500k degrades heavily up to frequency rank $500\,000$, at which point the model switches to a subword representation and performance recovers. The performance of C2-50k remains more stable. We attribute this to the fact that subword units are less sparse than words. In our training set, the frequency rank $50\,000$ corresponds to a frequency of 60 in the training data; the frequency rank $500\,000$ to a frequency of 2. Because subword representations are less sparse, reducing the size of the network vocabulary, and representing more words via subword units, can lead to better performance.

对最频繁的 $50\,000$ 个词，所有神经网络的表示是相同的，所有神经网络在这一类别上的 unigram F1 也相当。在频率排名 $50\,000$ 到 $500\,000$ 这一区间，C2-3/500k 与 C2-50k 的比较揭示了一个有意思的差别。这两个系统只在短名单大小上不同：C2-3/500k 把这一区间里的词表示成单个单元，C2-50k 则通过子词单元来表示。我们发现，C2-3/500k 的表现一直下降到频率排名 $500\,000$，到了这一点模型改用子词表示，表现才恢复过来。C2-50k 的表现则更为稳定。我们将其归因于子词单元比词更不稀疏。在我们的训练集中，频率排名 $50\,000$ 对应训练数据中的频次 60；频率排名 $500\,000$ 对应频次 2。正因为子词表示更不稀疏，缩小网络词表、让更多词通过子词单元来表示，反而可以带来更好的表现。

The F1 numbers hide some qualitative differences between systems. For English$\to$German, WDict produces few OOVs (26.5% recall), but with high precision (60.6%) , whereas the subword systems achieve higher recall, but lower precision. We note that the character bigram model C2-50k produces the most OOV words, and achieves relatively low precision of 29.1% for this category. However, it outperforms the back-off dictionary in recall (33.0%). BPE-60k, which suffers from transliteration (or copy) errors due to segmentation inconsistencies, obtains a slightly better precision (32.4%), but a worse recall (26.6%). In contrast to BPE-60k, the joint BPE encoding of BPE-J90k improves both precision (38.6%) and recall (29.8%).

F1 数字掩盖了系统之间的一些定性差别。对 English$\to$German，WDict 产生的 OOV 很少（召回率 26.5%），但精确率很高（60.6%）；子词系统则召回率更高，精确率更低。我们注意到，字符二元模型 C2-50k 产生的 OOV 词最多，在这一类别上精确率相对较低，为 29.1%。不过它在召回率上超过了回退词典（33.0%）。BPE-60k 因切分不一致而出现转写（或复制）错误，精确率略好（32.4%），召回率却更差（26.6%）。与 BPE-60k 相反，BPE-J90k 的联合 BPE 编码既提高了精确率（38.6%），也提高了召回率（29.8%）。

For English$\to$Russian, unknown names can only rarely be copied, and usually require transliteration. Consequently, the WDict baseline performs more poorly for OOVs (9.2% precision; 5.2% recall), and the subword models improve both precision and recall (21.9% precision and 15.6% recall for BPE-J90k). The full unigram F1 plot is shown in Figure 3.

对 English$\to$Russian，未知姓名很少能直接复制，通常需要转写。因此，WDict 基线在 OOV 上表现更差（精确率 9.2%；召回率 5.2%），而子词模型同时改善了精确率和召回率（BPE-J90k 的精确率为 21.9%，召回率为 15.6%）。完整的 unigram F1 图见图 3。

Figure 2: English$\to$German unigram F1 on newstest2015 plotted by training set frequency rank for different NMT systems.

图 2：不同 NMT 系统在 newstest2015 上的 English$\to$German unigram F1，按训练集频率排名作图。

Figure 3: English$\to$Russian unigram F1 on newstest2015 plotted by training set frequency rank for different NMT systems.

图 3：不同 NMT 系统在 newstest2015 上的 English$\to$Russian unigram F1，按训练集频率排名作图。

5.2 Manual Analysis

5.2 人工分析

Table 4 shows two translation examples for the translation direction English$\to$German, Table 5 for English$\to$Russian. The baseline system fails for all of the examples, either by deleting content (health), or by copying source words that should be translated or transliterated. The subword translations of health research institutes show that the subword systems are capable of learning translations when oversplitting (research$\rightarrow$Fo|rs|ch|un|g), or when the segmentation does not match morpheme boundaries: the segmentation Forschungs|instituten would be linguistically more plausible, and simpler to align to the English research institutes, than the segmentation Forsch|ungsinstitu|ten in the BPE-60k system, but still, a correct translation is produced. If the systems have failed to learn a translation due to data sparseness, like for asinine, which should be translated as dumm, we see translations that are wrong, but could be plausible for (partial) loanwords (asinine Situation$\rightarrow$Asinin-Situation).

表 4 给出 English$\to$German 方向的两个翻译例子，表 5 给出 English$\to$Russian 的例子。基线系统在全部例子上都失败了，要么删掉了内容（health），要么把本应翻译或转写的源词直接复制过去。health research institutes 的子词译文表明，子词系统即使切分切过了头（research$\to$Fo|rs|ch|un|g），或者切分并不落在语素边界上，仍然能够学会翻译：从语言学上看，Forschungs|instituten 这一切分更合理，也更容易与英语 research institutes 对齐，相比之下 BPE-60k 系统中的 Forsch|ungsinstitu|ten 更差，但仍然产生了正确译文。如果系统因为数据稀疏而没能学会某个译法，例如 asinine 本应译成 dumm，我们会看到虽然不对、但对（部分）借词来说可能说得通的译法（asinine Situation$\to$Asinin-Situation）。

The English$\to$Russian examples show that the subword systems are capable of transliteration. However, transliteration errors do occur, either due to ambiguous transliterations, or because of non-consistent segmentations between source and target text which make it hard for the system to learn a transliteration mapping. Note that the BPE-60k system encodes Mirzayeva inconsistently for the two language pairs (Mirz|ayeva$\rightarrow$Мир|за|ева Mir|za|eva). This example is still translated correctly, but we observe spurious insertions and deletions of characters in the BPE-60k system. An example is the transliteration of rakfisk, where a п is inserted and a к is deleted. We trace this error back to translation pairs in the training data with inconsistent segmentations, such as (p|rak|ri|ti$\rightarrow$пра|крит|и (pra|krit|i)), from which the translation (rak$\rightarrow$пра) is erroneously learned. The segmentation of the joint BPE system (BPE-J90k) is more consistent (pra|krit|i$\rightarrow$пра|крит|и (pra|krit|i)).

English$\to$Russian 的例子表明，子词系统具备转写能力。不过转写错误确实会出现，原因要么是转写有歧义，要么是源文与译文之间切分不一致，使系统难以学会转写映射。请注意，BPE-60k 系统对 Mirzayeva 在两个语言对上的编码并不一致（Mirz|ayeva$\to$Мир|за|ева Mir|za|eva）。这个例子仍然译对了，但我们在 BPE-60k 系统中观察到多余的字符插入和删除。一个例子是对 rakfisk 的转写，插入了一个 п，又删掉了一个 к。我们把这一错误追溯到训练数据中切分不一致的翻译对，例如（p|rak|ri|ti$\to$пра|крит|и (pra|krit|i)），系统由此错误地学会了（rak$\to$пра）这一对应。联合 BPE 系统（BPE-J90k）的切分更为一致（pra|krit|i$\to$пра|крит|и (pra|krit|i)）。

system	sentence
source	health research institutes
reference	Gesundheitsforschungsinstitute
WDict	Forschungsinstitute
C2-50k	Fo|rs|ch|un|gs|in|st|it|ut|io|ne|n
BPE-60k	Gesundheits|forsch|ungsinstitu|ten
BPE-J90k	Gesundheits|forsch|ungsin|stitute
source	asinine situation
reference	dumme Situation
WDict	asinine situation $\rightarrow$ UNK $\rightarrow$ asinine
C2-50k	as|in|in|e situation $\rightarrow$ As|in|en|si|tu|at|io|n
BPE-60k	as|in|ine situation $\rightarrow$ A|in|line-|Situation
BPE-J90K	as|in|ine situation $\rightarrow$ As|in|in-|Situation

Table 4: English$\to$German translation example. “|” marks subword boundaries.

系统	句子
source	health research institutes
reference	Gesundheitsforschungsinstitute
WDict	Forschungsinstitute
C2-50k	Fo|rs|ch|un|gs|in|st|it|ut|io|ne|n
BPE-60k	Gesundheits|forsch|ungsinstitu|ten
BPE-J90k	Gesundheits|forsch|ungsin|stitute
source	asinine situation
reference	dumme Situation
WDict	asinine situation $\rightarrow$ UNK $\rightarrow$ asinine
C2-50k	as|in|in|e situation $\rightarrow$ As|in|en|si|tu|at|io|n
BPE-60k	as|in|ine situation $\rightarrow$ A|in|line-|Situation
BPE-J90k	as|in|ine situation $\rightarrow$ As|in|in-|Situation

表 4：English$\to$German 翻译例子。“|” 标出子词边界。

system	sentence
source	Mirzayeva
reference	Мирзаева (Mirzaeva)
WDict	Mirzayeva|||| $\rightarrow$ UNK $\rightarrow$ Mirzayeva
C2-50k	Mi|rz|ay|ev|a $\rightarrow$ Ми|рз|ае|ва (Mi|rz|ae|va)
BPE-60k	Mirz|ayeva||| $\rightarrow$ Мир|за|ева (Mir|za|eva)
BPE-J90k	Mir|za|yeva|| $\rightarrow$ Мир|за|ева (Mir|za|eva)
source	rakfisk
reference	ракфиска (rakfiska)
WDict	rakfisk||| $\rightarrow$ UNK $\rightarrow$ rakfisk
C2-50k	ra|kf|is|k $\rightarrow$ ра|кф|ис|к (ra|kf|is|k)
BPE-60k	rak|f|isk| $\rightarrow$ пра|ф|иск (pra|f|isk)
BPE-J90k	rak|f|isk| $\rightarrow$ рак|ф|иска (rak|f|iska)

Table 5: English$\to$Russian translation examples. “|” marks subword boundaries.

系统	句子
source	Mirzayeva
reference	Мирзаева (Mirzaeva)
WDict	Mirzayeva|||| $\rightarrow$ UNK $\rightarrow$ Mirzayeva
C2-50k	Mi|rz|ay|ev|a $\rightarrow$ Ми|рз|ае|ва (Mi|rz|ae|va)
BPE-60k	Mirz|ayeva||| $\rightarrow$ Мир|за|ева (Mir|za|eva)
BPE-J90k	Mir|za|yeva|| $\rightarrow$ Мир|за|ева (Mir|za|eva)
source	rakfisk
reference	ракфиска (rakfiska)
WDict	rakfisk||| $\rightarrow$ UNK $\rightarrow$ rakfisk
C2-50k	ra|kf|is|k $\rightarrow$ ра|кф|ис|к (ra|kf|is|k)
BPE-60k	rak|f|isk| $\rightarrow$ пра|ф|иск (pra|f|isk)
BPE-J90k	rak|f|isk| $\rightarrow$ рак|ф|иска (rak|f|iska)

表 5：English$\to$Russian 翻译例子。“|” 标出子词边界。

6 Conclusion

6 结论

The main contribution of this paper is that we show that neural machine translation systems are capable of open-vocabulary translation by representing rare and unseen words as a sequence of subword units.[^14] This is both simpler and more effective than using a back-off translation model. We introduce a variant of byte pair encoding for word segmentation, which is capable of encoding open vocabularies with a compact symbol vocabulary of variable-length subword units. We show performance gains over the baseline with both BPE segmentation, and a simple character bigram segmentation.

本文的主要贡献是表明：通过把稀有词和未见词表示成子词单元的序列，神经机器翻译系统能够做开放词汇翻译。[^14] 这既比使用回退翻译模型更简单，也更有效。我们引入了一种用于分词的字节对编码变体，它能够用由可变长子词单元组成的紧凑符号词表，来编码开放词汇。我们表明，无论是 BPE 切分，还是简单的字符二元切分，都能相对基线带来性能提升。

Our analysis shows that not only out-of-vocabulary words, but also rare in-vocabulary words are translated poorly by our baseline NMT system, and that reducing the vocabulary size of subword models can actually improve performance. In this work, our choice of vocabulary size is somewhat arbitrary, and mainly motivated by comparison to prior work. One avenue of future research is to learn the optimal vocabulary size for a translation task, which we expect to depend on the language pair and amount of training data, automatically. We also believe there is further potential in bilingually informed segmentation algorithms to create more alignable subword units, although the segmentation algorithm cannot rely on the target text at runtime.

我们的分析表明，不仅未登录词，就连词表内的稀有词，也被我们的基线 NMT 系统译得很差；而且缩小子词模型的词表，实际上可以改善表现。在这项工作中，我们对词表大小的选择带有一定任意性，主要是为了便于与先前工作比较。今后研究的一个方向，是自动学出某个翻译任务的最优词表大小；我们预期这会依赖于语言对和训练数据量。我们也相信，用双语信息来指导切分、以得到更易对齐的子词单元，还有进一步的潜力，不过切分算法在运行时不能依赖目标文本。

While the relative effectiveness will depend on language-specific factors such as vocabulary size, we believe that subword segmentations are suitable for most language pairs, eliminating the need for large NMT vocabularies or back-off models.

尽管相对效果会依赖于词表大小这类语言相关因素，我们相信子词切分适合大多数语言对，从而不再需要很大的 NMT 词表或回退模型。

Acknowledgments

致谢

We thank Maja Popović for her implementation of chrF, with which we verified our re-implementation. The research presented in this publication was conducted in cooperation with Samsung Electronics Polska sp. z o.o. - Samsung R&D Institute Poland. This project received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement 645452 (QT21).

我们感谢 Maja Popović 提供的 chrF 实现，我们用它核实了自己的重新实现。本出版物所报告的研究，是与 Samsung Electronics Polska sp. z o.o. - Samsung R&D Institute Poland 合作完成的。本项目获得了欧盟地平线 2020 研究与创新计划的资助，资助协议号为 645452（QT21）。

References

参考文献

[Bahdanau et al. (2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the International Conference on Learning Representations (ICLR) .

[Bazzi and Glass (2000] Issam Bazzi and James R. Glass. 2000. Modeling out-of-vocabulary words for robust speech recognition. In Sixth International Conference on Spoken Language Processing, ICSLP 2000 / INTERSPEECH 2000 , pages 401–404, Beijing, China.

[Botha and Blunsom (2014] Jan A. Botha and Phil Blunsom. 2014. Compositional Morphology for Word Representations and Language Modelling. In Proceedings of the 31st International Conference on Machine Learning (ICML) , Beijing, China.

[Chitnis and DeNero (2015] Rohan Chitnis and John DeNero. 2015. Variable-Length Word Encodings for Neural Translation Models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) .

[Cho et al. (2014] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.

[Creutz and Lagus (2002] Mathias Creutz and Krista Lagus. 2002. Unsupervised Discovery of Morphemes. In Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning , pages 21–30. Association for Computational Linguistics.

[Durrani et al. (2014] Nadir Durrani, Hassan Sajjad, Hieu Hoang, and Philipp Koehn. 2014. Integrating an Unsupervised Transliteration Model into Statistical Machine Translation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014 , pages 148–153, Gothenburg, Sweden.

[Dyer et al. (2013] Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 644–648, Atlanta, Georgia. Association for Computational Linguistics.

[Gage (1994] Philip Gage. 1994. A New Algorithm for Data Compression. C Users J. , 12(2):23–38, February.

[Haddow et al. (2015] Barry Haddow, Matthias Huck, Alexandra Birch, Nikolay Bogoychev, and Philipp Koehn. 2015. The Edinburgh/JHU Phrase-based Machine Translation Systems for WMT 2015. In Proceedings of the Tenth Workshop on Statistical Machine Translation , pages 126–133, Lisbon, Portugal. Association for Computational Linguistics.

[Jean et al. (2015] Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On Using Very Large Target Vocabulary for Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1–10, Beijing, China. Association for Computational Linguistics.

[Kalchbrenner and Blunsom (2013] Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , Seattle. Association for Computational Linguistics.

[Kim et al. (2015] Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2015. Character-Aware Neural Language Models. CoRR , abs/1508.06615.

[Koehn and Knight (2003] Philipp Koehn and Kevin Knight. 2003. Empirical Methods for Compound Splitting. In EACL ’03: Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics , pages 187–193, Budapest, Hungary. Association for Computational Linguistics.

[Koehn et al. (2007] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the ACL-2007 Demo and Poster Sessions , pages 177–180, Prague, Czech Republic. Association for Computational Linguistics.

[Liang (1983] Franklin M. Liang. 1983. Word hy-phen-a-tion by com-put-er . Ph.D. thesis, Stanford University, Department of Linguistics, Stanford, CA.

[Ling et al. (2015a] Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015a. Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1520–1530, Lisbon, Portugal. Association for Computational Linguistics.

[Ling et al. (2015b] Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W. Black. 2015b. Character-based Neural Machine Translation. ArXiv e-prints , November.

[Luong et al. (2013] Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better Word Representations with Recursive Neural Networks for Morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013 , pages 104–113.

[Luong et al. (2015a] Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.

[Luong et al. (2015b] Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the Rare Word Problem in Neural Machine Translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 11–19, Beijing, China. Association for Computational Linguistics.

[Mikolov et al. (2012] Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Cernocký. 2012. Subword Language Modeling with Neural Networks. Unpublished.

[Neubig et al. (2012] Graham Neubig, Taro Watanabe, Shinsuke Mori, and Tatsuya Kawahara. 2012. Machine Translation without Words through Substring Alignment. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 1: Long Papers , pages 165–174.

[Nießen and Ney (2000] Sonja Nießen and Hermann Ney. 2000. Improving SMT quality with morpho-syntactic analysis. In 18th Int. Conf. on Computational Linguistics , pages 1081–1085.

[Pascanu et al. (2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013 , pages 1310–1318, Atlanta, USA.

[Popović (2015] Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.

[Sennrich and Haddow (2015] Rico Sennrich and Barry Haddow. 2015. A Joint Dependency Model of Morphological and Syntactic Structure for Statistical Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 2081–2087, Lisbon, Portugal. Association for Computational Linguistics.

[Snyder and Barzilay (2008] Benjamin Snyder and Regina Barzilay. 2008. Unsupervised Multilingual Learning for Morphological Segmentation. In Proceedings of ACL-08: HLT , pages 737–745, Columbus, Ohio. Association for Computational Linguistics.

[Stallard et al. (2012] David Stallard, Jacob Devlin, Michael Kayser, Yoong Keok Lee, and Regina Barzilay. 2012. Unsupervised Morphology Rivals Supervised Morphology for Arabic MT. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 2: Short Papers , pages 322–327.

[Stanojević et al. (2015] Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 Metrics Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation , pages 256–273, Lisbon, Portugal. Association for Computational Linguistics.

[Sutskever et al. (2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014 , pages 3104–3112, Montreal, Quebec, Canada.

[Tiedemann (2009] Jörg Tiedemann. 2009. Character-based PSMT for Closely Related Languages. In Proceedings of 13th Annual Conference of the European Association for Machine Translation (EAMT’09) , pages 12–19.

[Tiedemann (2012] Jörg Tiedemann. 2012. Character-Based Pivot Translation for Under-Resourced Languages and Domains. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics , pages 141–151, Avignon, France. Association for Computational Linguistics.

[Vilar et al. (2007] David Vilar, Jan-Thorsten Peter, and Hermann Ney. 2007. Can We Translate Letters? In Second Workshop on Statistical Machine Translation , pages 33–39, Prague, Czech Republic. Association for Computational Linguistics.

[Virpioja et al. (2007] Sami Virpioja, Jaakko J. Väyrynen, Mathias Creutz, and Markus Sadeniemi. 2007. Morphology-Aware Statistical Machine Translation Based on Morphs Induced in an Unsupervised Manner. In Proceedings of the Machine Translation Summit XI , pages 491–498, Copenhagen, Denmark.

[Zeiler (2012] Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. CoRR , abs/1212.5701.