Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

用 RNN 编码器–解码器学习短语表示，服务统计机器翻译

Kyunghyun Cho Affiliation:  Bart van Merriënboer   Caglar Gulcehre Affiliation:  Université de Montréal Email:  firstname.lastname@umontreal.ca Affiliation:  Dzmitry Bahdanau Affiliation:  Jacobs University, Germany Email:  d.bahdanau@jacobs-university.de    Fethi Bougares     Holger Schwenk Affiliation:  Université du Maine, France Email:  firstname.lastname@lium.univ-lemans.fr    Yoshua Bengio Affiliation:  Université de Montréal, CIFAR Senior Fellow Email:  find.me@on.the.web

Abstract

摘要

In this paper, we propose a novel neural network model called RNN Encoder–Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder–Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.

本文提出一种称为 RNN 编码器–解码器（RNN Encoder–Decoder）的新神经网络模型，由两个循环神经网络（RNN）组成。其中一个 RNN 把符号序列编码成固定长度的向量表示，另一个再把该表示解码成另一段符号序列。编码器与解码器联合训练，目标是最大化给定源序列时目标序列的条件概率。实验表明，把该模型算出的短语对条件概率作为已有对数线性模型中的额外特征，可以提高统计机器翻译系统的表现。定性分析还显示，该模型能够学到在语义和句法上都有意义的语言短语表示。

1 Introduction

1 引言

Deep neural networks have shown great success in various applications such as objection recognition (see, e.g., [\citenameKrizhevsky et al.2012]) and speech recognition (see, e.g., [\citenameDahl et al.2012]). Furthermore, many recent works showed that neural networks can be successfully used in a number of tasks in natural language processing (NLP). These include, but are not limited to, language modeling [\citenameBengio et al.2003], paraphrase detection [\citenameSocher et al.2011] and word embedding extraction [\citenameMikolov et al.2013]. In the field of statistical machine translation (SMT), deep neural networks have begun to show promising results. [\citenameSchwenk2012] summarizes a successful usage of feedforward neural networks in the framework of phrase-based SMT system.

深度神经网络已在多项任务上取得显著成功，例如物体识别（参见 [Krizhevsky et al., 2012]）和语音识别（参见 [Dahl et al., 2012]）。近来也有不少工作表明，神经网络能够胜任自然语言处理（NLP）中的多种任务，包括但不限于语言建模 [Bengio et al., 2003]、释义检测 [Socher et al., 2011] 以及词嵌入抽取 [Mikolov et al., 2013]。在统计机器翻译（SMT）领域，深度神经网络也开始显现出有希望的结果。[Schwenk, 2012] 综述了前馈神经网络在基于短语的 SMT 框架中的成功用法。

Along this line of research on using neural networks for SMT, this paper focuses on a novel neural network architecture that can be used as a part of the conventional phrase-based SMT system. The proposed neural network architecture, which we will refer to as an RNN Encoder–Decoder, consists of two recurrent neural networks (RNN) that act as an encoder and a decoder pair. The encoder maps a variable-length source sequence to a fixed-length vector, and the decoder maps the vector representation back to a variable-length target sequence. The two networks are trained jointly to maximize the conditional probability of the target sequence given a source sequence. Additionally, we propose to use a rather sophisticated hidden unit in order to improve both the memory capacity and the ease of training.

沿着把神经网络用于 SMT 这条研究线索，本文关注一种可作为传统基于短语的 SMT 系统组成部分的新网络结构。我们称之为 RNN 编码器–解码器：它由一对充当编码器和解码器的循环神经网络组成。编码器把变长的源序列映射成固定长度的向量，解码器再把该向量表示映射回变长的目标序列。两个网络联合训练，以最大化给定源序列时目标序列的条件概率。此外，我们还提出使用一种较为精巧的隐藏单元，以同时提高记忆容量并降低训练难度。

The proposed RNN Encoder–Decoder with a novel hidden unit is empirically evaluated on the task of translating from English to French. We train the model to learn the translation probability of an English phrase to a corresponding French phrase. The model is then used as a part of a standard phrase-based SMT system by scoring each phrase pair in the phrase table. The empirical evaluation reveals that this approach of scoring phrase pairs with an RNN Encoder–Decoder improves the translation performance.

带有这种新隐藏单元的 RNN 编码器–解码器，在英语到法语的翻译任务上接受了经验评估。我们训练该模型，学习从英语短语到对应法语短语的翻译概率，再把它嵌入标准的基于短语的 SMT 系统，对短语表中的每个短语对打分。实验表明，用 RNN 编码器–解码器为短语对打分，能够提高翻译性能。

We qualitatively analyze the trained RNN Encoder–Decoder by comparing its phrase scores with those given by the existing translation model. The qualitative analysis shows that the RNN Encoder–Decoder is better at capturing the linguistic regularities in the phrase table, indirectly explaining the quantitative improvements in the overall translation performance. The further analysis of the model reveals that the RNN Encoder–Decoder learns a continuous space representation of a phrase that preserves both the semantic and syntactic structure of the phrase.

我们还对训练好的 RNN 编码器–解码器做了定性分析，把它给出的短语分数与已有翻译模型的分数加以比较。分析显示，RNN 编码器–解码器更善于抓住短语表中的语言规律，这也可以间接解释整体翻译性能上的定量提升。进一步分析还表明，该模型学到的是短语的连续空间表示，并且同时保留了短语的语义结构与句法结构。

2 RNN Encoder–Decoder

2 RNN 编码器–解码器

2.1 Preliminary: Recurrent Neural Networks

2.1 预备：循环神经网络

A recurrent neural network (RNN) is a neural network that consists of a hidden state $\mathbf{h}$ and an optional output $\mathbf{y}$ which operates on a variable-length sequence $\mathbf{x}=(x_{1},\dots,x_{T})$. At each time step $t$, the hidden state $\mathbf{h}_{\left<t\right>}$ of the RNN is updated by

循环神经网络（RNN）包含隐藏状态 $\mathbf{h}$，以及可选的输出 $\mathbf{y}$，并在变长序列 $\mathbf{x}=(x_{1},\dots,x_{T})$ 上运行。在每个时间步 $t$，RNN 的隐藏状态 $\mathbf{h}_{\left<t\right>}$ 按如下方式更新

$$\displaystyle\mathbf{h}_{\left<t\right>}=f\left(\mathbf{h}_{\left<t-1\right>},x_{t}\right), \tag{1}$$

where $f$ is a non-linear activation function. $f$ may be as simple as an element-wise logistic sigmoid function and as complex as a long short-term memory (LSTM) unit [\citenameHochreiter and Schmidhuber1997].

其中 $f$ 是非线性激活函数。$f$ 既可以简单到逐元素的 logistic sigmoid 函数，也可以复杂到长短期记忆（LSTM）单元 [Hochreiter and Schmidhuber, 1997]。

An RNN can learn a probability distribution over a sequence by being trained to predict the next symbol in a sequence. In that case, the output at each timestep $t$ is the conditional distribution $p(x_{t}\mid x_{t-1},\dots,x_{1})$. For example, a multinomial distribution ($1$-of-$K$ coding) can be output using a softmax activation function

通过训练 RNN 去预测序列中的下一个符号，它可以学习序列上的概率分布。这时，每个时间步 $t$ 的输出就是条件分布 $p(x_{t}\mid x_{t-1},\dots,x_{1})$。例如，可以用 softmax 激活函数输出多项式分布（$1$-of-$K$ 编码）

$$\displaystyle p(x_{t,j}=1\mid x_{t-1},\dots,x_{1})=\frac{\exp\left(\mathbf{w}_{j}\mathbf{h}_{\left<t\right>}\right)}{\sum_{j^{\prime}=1}^{K}\exp\left(\mathbf{w}_{j^{\prime}}\mathbf{h}_{\left<t\right>}\right)}, \tag{2}$$

for all possible symbols $j=1,\dots,K$, where $\mathbf{w}_{j}$ are the rows of a weight matrix $\mathbf{W}$. By combining these probabilities, we can compute the probability of the sequence $\mathbf{x}$ using

对所有可能的符号 $j=1,\dots,K$ 成立，其中 $\mathbf{w}_{j}$ 是权重矩阵 $\mathbf{W}$ 的各行。把这些概率合起来，就可以用下式计算序列 $\mathbf{x}$ 的概率

$$\displaystyle p(\mathbf{x})=\prod_{t=1}^{T}p(x_{t}\mid x_{t-1},\dots,x_{1}). \tag{3}$$

From this learned distribution, it is straightforward to sample a new sequence by iteratively sampling a symbol at each time step.

有了学到的这个分布，只需在每个时间步依次采样一个符号，就可以直接生成新的序列。

2.2 RNN Encoder–Decoder

2.2 RNN 编码器–解码器

In this paper, we propose a novel neural network architecture that learns to encode a variable-length sequence into a fixed-length vector representation and to decode a given fixed-length vector representation back into a variable-length sequence. From a probabilistic perspective, this new model is a general method to learn the conditional distribution over a variable-length sequence conditioned on yet another variable-length sequence, e.g. $p(y_{1},\dots,y_{T^{\prime}}\mid x_{1},\dots,x_{T})$, where one should note that the input and output sequence lengths $T$ and $T^{\prime}$ may differ.

本文提出一种新的神经网络结构，它学习把变长序列编码成固定长度的向量表示，再把给定的固定长度向量解码回变长序列。从概率角度看，这一新模型是学习条件分布的一般方法：在一段变长序列的条件下，为另一段变长序列建模，例如 $p(y_{1},\dots,y_{T^{\prime}}\mid x_{1},\dots,x_{T})$。需要注意的是，输入与输出的序列长度 $T$ 和 $T^{\prime}$ 可以不同。

The encoder is an RNN that reads each symbol of an input sequence $\mathbf{x}$ sequentially. As it reads each symbol, the hidden state of the RNN changes according to Eq. (1). After reading the end of the sequence (marked by an end-of-sequence symbol), the hidden state of the RNN is a summary $\mathbf{c}$ of the whole input sequence.

编码器是一个 RNN，它按顺序读取输入序列 $\mathbf{x}$ 中的每个符号。每读入一个符号，RNN 的隐藏状态就按式 (1) 更新。读到序列末尾（由序列结束符标出）之后，RNN 的隐藏状态就成为整个输入序列的摘要 $\mathbf{c}$。

The decoder of the proposed model is another RNN which is trained to generate the output sequence by predicting the next symbol $y_{t}$ given the hidden state $\mathbf{h}_{\left<t\right>}$. However, unlike the RNN described in Sec. 2.1, both $y_{t}$ and $\mathbf{h}_{\left<t\right>}$ are also conditioned on $y_{t-1}$ and on the summary $\mathbf{c}$ of the input sequence. Hence, the hidden state of the decoder at time $t$ is computed by,

所提出模型的解码器是另一个 RNN，训练目标是生成输出序列：在给定隐藏状态 $\mathbf{h}_{\left<t\right>}$ 的条件下，预测下一个符号 $y_{t}$。不过，与第 2.1 节中的 RNN 不同，这里的 $y_{t}$ 和 $\mathbf{h}_{\left<t\right>}$ 还同时以 $y_{t-1}$ 以及输入序列的摘要 $\mathbf{c}$ 为条件。因此，解码器在时刻 $t$ 的隐藏状态由下式计算：

$$\displaystyle\mathbf{h}_{\left<t\right>}=f\left(\mathbf{h}_{\left<t-1\right>},y_{t-1},\mathbf{c}\right),$$

and similarly, the conditional distribution of the next symbol is

类似地，下一个符号的条件分布为

$$\displaystyle P(y_{t}|y_{t-1},y_{t-2},\ldots,y_{1},\mathbf{c})=g\left(\mathbf{h}_{\left<t\right>},y_{t-1},\mathbf{c}\right).$$

for given activation functions $f$ and $g$ (the latter must produce valid probabilities, e.g. with a softmax).

其中 $f$ 和 $g$ 是给定的激活函数（后者必须给出合法的概率，例如使用 softmax）。

Figure 1: An illustration of the proposed RNN Encoder–Decoder.

图 1：所提出的 RNN 编码器–解码器示意图。

See Fig. 1 for a graphical depiction of the proposed model architecture.

所提出模型结构的图示见图 1。

The two components of the proposed RNN Encoder–Decoder are jointly trained to maximize the conditional log-likelihood

所提出的 RNN 编码器–解码器 的两个组成部分联合训练，以最大化条件对数似然

$$\displaystyle\max_{\boldsymbol{\theta}}\frac{1}{N}\sum_{n=1}^{N}\log p_{{\boldsymbol{\theta}}}(\mathbf{y}_{n}\mid\mathbf{x}_{n}), \tag{4}$$

where ${\boldsymbol{\theta}}$ is the set of the model parameters and each $\left(\mathbf{x}_{n},\mathbf{y}_{n}\right)$ is an (input sequence, output sequence) pair from the training set. In our case, as the output of the decoder, starting from the input, is differentiable, we can use a gradient-based algorithm to estimate the model parameters.

其中 ${\boldsymbol{\theta}}$ 是模型参数集合，每个 $\left(\mathbf{x}_{n},\mathbf{y}_{n}\right)$ 是训练集中的一对（输入序列，输出序列）。在我们的设定中，解码器从输入出发的输出是可微的，因此可以用基于梯度的算法来估计模型参数。

Once the RNN Encoder–Decoder is trained, the model can be used in two ways. One way is to use the model to generate a target sequence given an input sequence. On the other hand, the model can be used to score a given pair of input and output sequences, where the score is simply a probability $p_{\boldsymbol{\theta}}(\mathbf{y}\mid\mathbf{x})$ from Eqs. (3) and (4).

RNN 编码器–解码器训练完成后，可以有两种用法。一种是在给定输入序列时生成目标序列；另一种是为给定的输入–输出序列对打分，分数就是式 (3) 和式 (4) 中的概率 $p_{\boldsymbol{\theta}}(\mathbf{y}\mid\mathbf{x})$。

2.3 Hidden Unit that Adaptively Remembers and Forgets

2.3 能自适应记忆与遗忘的隐藏单元

In addition to a novel model architecture, we also propose a new type of hidden unit ($f$ in Eq. (1)) that has been motivated by the LSTM unit but is much simpler to compute and implement.[^1] Fig. 2 shows the graphical depiction of the proposed hidden unit.

除了新的模型结构，我们还提出一种新的隐藏单元（即式 (1) 中的 $f$）。它的设计受到 LSTM 单元的启发，但计算和实现都简单得多。[^1]图 2 给出了所提出隐藏单元的示意图。

[^1]: The LSTM unit, which has shown impressive results in several applications such as speech recognition, has a memory cell and four gating units that adaptively control the information flow inside the unit, compared to only two gating units in the proposed hidden unit. For details on LSTM networks, see, e.g., [\citenameGraves2012].

[^1]：LSTM 单元已在语音识别等多项应用中取得很好的结果，它包含一个记忆单元以及四个门控单元，用来自适应地控制单元内部的信息流动；相比之下，所提出的隐藏单元只有两个门控单元。关于 LSTM 网络的细节，可参见 [Graves, 2012]。

Let us describe how the activation of the $j$-th hidden unit is computed. First, the reset gate $r_{j}$ is computed by

下面说明第 $j$ 个隐藏单元的激活如何计算。首先，重置门 $r_{j}$ 由下式得到

$$\displaystyle r_{j}=\sigma\left(\left[\mathbf{W}_{r}\mathbf{x}\right]_{j}+\left[\mathbf{U}_{r}\mathbf{h}_{\left<t-1\right>}\right]_{j}\right), \tag{5}$$

where $\sigma$ is the logistic sigmoid function, and $\left[.\right]_{j}$ denotes the $j$-th element of a vector. $\mathbf{x}$ and $\mathbf{h}_{t-1}$ are the input and the previous hidden state, respectively. $\mathbf{W}_{r}$ and $\mathbf{U}_{r}$ are weight matrices which are learned.

其中 $\sigma$ 是 logistic sigmoid 函数，$\left[.\right]_{j}$ 表示向量的第 $j$ 个元素。$\mathbf{x}$ 和 $\mathbf{h}_{t-1}$ 分别是输入和上一时刻的隐藏状态。$\mathbf{W}_{r}$ 和 $\mathbf{U}_{r}$ 是需要学习的权重矩阵。

Similarly, the update gate $z_{j}$ is computed by

类似地，更新门 $z_{j}$ 由下式计算

$$\displaystyle z_{j}=\sigma\left(\left[\mathbf{W}_{z}\mathbf{x}\right]_{j}+\left[\mathbf{U}_{z}\mathbf{h}_{\left<t-1\right>}\right]_{j}\right). \tag{6}$$

The actual activation of the proposed unit $h_{j}$ is then computed by

所提出单元的实际激活 $h_{j}$ 则由下式计算

$$\displaystyle h_{j}^{\left<t\right>}=z_{j}h_{j}^{\left<t-1\right>}+(1-z_{j})\tilde{h}_{j}^{\left<t\right>}, \tag{7}$$

where

其中

$$\displaystyle\tilde{h}_{j}^{\left<t\right>}=\phi\left(\left[\mathbf{W}\mathbf{x}\right]_{j}+\left[\mathbf{U}\left(\mathbf{r}\odot\mathbf{h}_{\left<t-1\right>}\right)\right]_{j}\right). \tag{8}$$

In this formulation, when the reset gate is close to 0, the hidden state is forced to ignore the previous hidden state and reset with the current input only. This effectively allows the hidden state to drop any information that is found to be irrelevant later in the future, thus, allowing a more compact representation.

在这一形式中，当重置门接近 0 时，隐藏状态被迫忽略上一时刻的隐藏状态，只按当前输入重新设置。这样，隐藏状态就可以丢掉后来被发现不再相关的信息，从而得到更紧凑的表示。

On the other hand, the update gate controls how much information from the previous hidden state will carry over to the current hidden state. This acts similarly to the memory cell in the LSTM network and helps the RNN to remember long-term information. Furthermore, this may be considered an adaptive variant of a leaky-integration unit [\citenameBengio et al.2013].

另一方面，更新门控制有多少来自上一时刻隐藏状态的信息会带到当前隐藏状态。它的作用类似于 LSTM 网络中的记忆单元，帮助 RNN 记住长期信息。此外，也可以把它看作泄漏积分单元的一种自适应变体 [Bengio et al., 2013]。

As each hidden unit has separate reset and update gates, each hidden unit will learn to capture dependencies over different time scales. Those units that learn to capture short-term dependencies will tend to have reset gates that are frequently active, but those that capture longer-term dependencies will have update gates that are mostly active.

由于每个隐藏单元都有各自的重置门和更新门，不同单元会学习捕捉不同时间尺度上的依赖。学会捕捉短期依赖的单元，其重置门往往会频繁处于激活状态；而捕捉更长期依赖的单元，其更新门则大多保持激活。

In our preliminary experiments, we found that it is crucial to use this new unit with gating units. We were not able to get meaningful result with an oft-used $\tanh$ unit without any gating.

在初步实验中，我们发现使用这种带门控的新单元至关重要。如果只用常见的、不加任何门控的 $\tanh$ 单元，我们无法得到有意义的结果。

Figure 2: An illustration of the proposed hidden activation function. The update gate $z$ selects whether the hidden state is to be updated with a new hidden state $\tilde{h}$. The reset gate $r$ decides whether the previous hidden state is ignored. See Eqs. (5)–(8) for the detailed equations of $r$, $z$, $h$ and $\tilde{h}$.

图 2：所提出的隐藏激活函数示意图。更新门 $z$ 选择是否用新的隐藏状态 $\tilde{h}$ 来更新当前隐藏状态。重置门 $r$ 决定是否忽略上一时刻的隐藏状态。$r$、$z$、$h$ 和 $\tilde{h}$ 的具体公式见式 (5)–(8)。

3 Statistical Machine Translation

3 统计机器翻译

In a commonly used statistical machine translation system (SMT), the goal of the system (decoder, specifically) is to find a translation $\mathbf{f}$ given a source sentence $\mathbf{e}$, which maximizes

在常用的统计机器翻译系统（SMT）中，系统（具体来说是解码器）的目标是：在给定源句 $\mathbf{e}$ 时，找到使下式最大的译文 $\mathbf{f}$

$$\displaystyle p(\mathbf{f}\mid\mathbf{e})\propto p(\mathbf{e}\mid\mathbf{f})p(\mathbf{f}),$$

where the first term at the right hand side is called translation model and the latter language model (see, e.g., [\citenameKoehn2005]). In practice, however, most SMT systems model $\log p(\mathbf{f}\mid\mathbf{e})$ as a log-linear model with additional features and corresponding weights:

其中等号右侧第一项称为翻译模型，第二项称为语言模型（参见 [Koehn, 2005]）。不过在实践中，大多数 SMT 系统把 $\log p(\mathbf{f}\mid\mathbf{e})$ 建成带有额外特征及相应权重的对数线性模型：

$$\displaystyle\log p(\mathbf{f}\mid\mathbf{e})=\sum_{n=1}^{N}w_{n}f_{n}(\mathbf{f},\mathbf{e})+\log Z(\mathbf{e}), \tag{9}$$

where $f_{n}$ and $w_{n}$ are the $n$-th feature and weight, respectively. $Z(\mathbf{e})$ is a normalization constant that does not depend on the weights. The weights are often optimized to maximize the BLEU score on a development set.

其中 $f_{n}$ 和 $w_{n}$ 分别是第 $n$ 个特征及其权重。$Z(\mathbf{e})$ 是与权重无关的归一化常数。这些权重通常在开发集上优化，以最大化 BLEU 分数。

In the phrase-based SMT framework introduced in [\citenameKoehn et al.2003] and [\citenameMarcu and Wong2002], the translation model $\log p(\mathbf{e}\mid\mathbf{f})$ is factorized into the translation probabilities of matching phrases in the source and target sentences.[^2] These probabilities are once again considered additional features in the log-linear model (see Eq. (9)) and are weighted accordingly to maximize the BLEU score.

在 [Koehn et al., 2003] 和 [Marcu and Wong, 2002] 提出的基于短语的 SMT 框架中，翻译模型 $\log p(\mathbf{e}\mid\mathbf{f})$ 被分解成源句与目标句中相匹配短语的翻译概率。[^2]这些概率同样作为式 (9) 中对数线性模型的额外特征，并被赋予相应权重，以最大化 BLEU 分数。

[^2]: Without loss of generality, from here on, we refer to $p(\mathbf{e}\mid\mathbf{f})$ for each phrase pair as a translation model as well.

[^2]：在不失一般性的前提下，后文也把每个短语对上的 $p(\mathbf{e}\mid\mathbf{f})$ 称作翻译模型。

Since the neural net language model was proposed in [\citenameBengio et al.2003], neural networks have been used widely in SMT systems. In many cases, neural networks have been used to rescore translation hypotheses ($n$-best lists) (see, e.g., [\citenameSchwenk et al.2006]). Recently, however, there has been interest in training neural networks to score the translated sentence (or phrase pairs) using a representation of the source sentence as an additional input. See, e.g., [\citenameSchwenk2012], [\citenameSon et al.2012] and [\citenameZou et al.2013].

自 [Bengio et al., 2003] 提出神经语言模型以来，神经网络已在 SMT 系统中得到广泛使用。在许多情形下，神经网络被用来对翻译假设（$n$-best 列表）重新打分（例如 [Schwenk et al., 2006]）。不过近来也有工作开始训练神经网络，利用源句的表示作为额外输入，直接为译句（或短语对）打分。可参见 [Schwenk, 2012]、[Son et al., 2012] 和 [Zou et al., 2013]。

3.1 Scoring Phrase Pairs with RNN Encoder–Decoder

3.1 用 RNN 编码器–解码器为短语对打分

Here we propose to train the RNN Encoder–Decoder (see Sec. 2.2) on a table of phrase pairs and use its scores as additional features in the log-linear model in Eq. (9) when tuning the SMT decoder.

在此我们提出：在短语对表上训练 RNN 编码器–解码器（见第 2.2 节），并在调优 SMT 解码器时，把它给出的分数作为式 (9) 对数线性模型中的额外特征。

When we train the RNN Encoder–Decoder, we ignore the (normalized) frequencies of each phrase pair in the original corpora. This measure was taken in order (1) to reduce the computational expense of randomly selecting phrase pairs from a large phrase table according to the normalized frequencies and (2) to ensure that the RNN Encoder–Decoder does not simply learn to rank the phrase pairs according to their numbers of occurrences. One underlying reason for this choice was that the existing translation probability in the phrase table already reflects the frequencies of the phrase pairs in the original corpus. With a fixed capacity of the RNN Encoder–Decoder, we try to ensure that most of the capacity of the model is focused toward learning linguistic regularities, i.e., distinguishing between plausible and implausible translations, or learning the “manifold” (region of probability concentration) of plausible translations.

训练 RNN 编码器–解码器时，我们忽略每个短语对在原始语料中的（归一化）频率。这样做有两个原因：（1）避免按归一化频率从大型短语表中随机抽取短语对所带来的计算开销；（2）确保模型不会仅仅学会按出现次数给短语对排序。这一选择的一个内在理由是：短语表中已有的翻译概率已经反映了短语对在原始语料中的频率。在 RNN 编码器–解码器容量固定的前提下，我们希望把模型的大部分容量用在学习语言规律上，也就是区分合理与不合理的译文，或学到合理译文所集中的“流形”（概率集中的区域）。

Once the RNN Encoder–Decoder is trained, we add a new score for each phrase pair to the existing phrase table. This allows the new scores to enter into the existing tuning algorithm with minimal additional overhead in computation.

RNN 编码器–解码器训练完成后，我们为短语表中的每个短语对追加一个新分数。这样，新分数可以进入已有的调参算法，而只增加极少的额外计算开销。

As Schwenk pointed out in [\citenameSchwenk2012], it is possible to completely replace the existing phrase table with the proposed RNN Encoder–Decoder. In that case, for a given source phrase, the RNN Encoder–Decoder will need to generate a list of (good) target phrases. This requires, however, an expensive sampling procedure to be performed repeatedly. In this paper, thus, we only consider rescoring the phrase pairs in the phrase table.

正如 Schwenk 在 [Schwenk, 2012] 中指出的，也可以用所提出的 RNN 编码器–解码器完全替换现有短语表。那时，对于给定的源短语，RNN 编码器–解码器需要生成一份（较好的）目标短语列表。不过这要求反复进行代价很高的采样。因此，本文只考虑对短语表中已有的短语对重新打分。

3.2 Related Approaches: Neural Networks in Machine Translation

3.2 相关方法：机器翻译中的神经网络

Before presenting the empirical results, we discuss a number of recent works that have proposed to use neural networks in the context of SMT.

在报告实验结果之前，我们先讨论若干近期把神经网络用于 SMT 的工作。

Schwenk in [\citenameSchwenk2012] proposed a similar approach of scoring phrase pairs. Instead of the RNN-based neural network, he used a feedforward neural network that has fixed-size inputs (7 words in his case, with zero-padding for shorter phrases) and fixed-size outputs (7 words in the target language). When it is used specifically for scoring phrases for the SMT system, the maximum phrase length is often chosen to be small. However, as the length of phrases increases or as we apply neural networks to other variable-length sequence data, it is important that the neural network can handle variable-length input and output. The proposed RNN Encoder–Decoder is well-suited for these applications.

Schwenk 在 [Schwenk, 2012] 中提出了类似的短语对打分方法。与基于 RNN 的网络不同，他使用的是前馈神经网络，输入大小固定（在他的设定中是 7 个词，较短短语用零填充），输出大小也固定（目标语言 7 个词）。当专门为 SMT 系统给短语打分时，最大短语长度通常选得较短。但随着短语变长，或把神经网络用到其他变长序列数据上，网络必须能够处理变长的输入和输出。所提出的 RNN 编码器–解码器正好适合这类应用。

Similar to [\citenameSchwenk2012], Devlin et al. [\citenameDevlin et al.2014] proposed to use a feedforward neural network to model a translation model, however, by predicting one word in a target phrase at a time. They reported an impressive improvement, but their approach still requires the maximum length of the input phrase (or context words) to be fixed a priori.

与 [Schwenk, 2012] 类似，Devlin 等人 [Devlin et al., 2014] 也提出用前馈神经网络来建模翻译模型，不过是一次预测目标短语中的一个词。他们报告了可观的提升，但该方法仍要求事先固定输入短语（或上下文词）的最大长度。

Although it is not exactly a neural network they train, the authors of [\citenameZou et al.2013] proposed to learn a bilingual embedding of words/phrases. They use the learned embedding to compute the distance between a pair of phrases which is used as an additional score of the phrase pair in an SMT system.

虽然训练的并不严格是神经网络，[Zou et al., 2013] 的作者提出学习词/短语的双语嵌入。他们用学到的嵌入计算短语对之间的距离，并把该距离作为 SMT 系统中短语对的额外分数。

In [\citenameChandar et al.2014], a feedforward neural network was trained to learn a mapping from a bag-of-words representation of an input phrase to an output phrase. This is closely related to both the proposed RNN Encoder–Decoder and the model proposed in [\citenameSchwenk2012], except that their input representation of a phrase is a bag-of-words. A similar approach of using bag-of-words representations was proposed in [\citenameGao et al.2013] as well. Earlier, a similar encoder–decoder model using two recursive neural networks was proposed in [\citenameSocher et al.2011], but their model was restricted to a monolingual setting, i.e. the model reconstructs an input sentence. More recently, another encoder–decoder model using an RNN was proposed in [\citenameAuli et al.2013], where the decoder is conditioned on a representation of either a source sentence or a source context.

在 [Chandar et al., 2014] 中，前馈神经网络被训练来学习从输入短语的词袋表示到输出短语的映射。这与所提出的 RNN 编码器–解码器以及 [Schwenk, 2012] 中的模型都很接近，不同之处在于他们的短语输入表示是词袋。[Gao et al., 2013] 也提出过类似的词袋表示方法。更早些，[Socher et al., 2011] 提出过用两个递归神经网络构成的类似编码器–解码器模型，但仅限于单语设定，也就是重构输入句子。更近一些，[Auli et al., 2013] 提出了另一种使用 RNN 的编码器–解码器，其解码器以源句或源上下文的表示为条件。

One important difference between the proposed RNN Encoder–Decoder and the approaches in [\citenameZou et al.2013] and [\citenameChandar et al.2014] is that the order of the words in source and target phrases is taken into account. The RNN Encoder–Decoder naturally distinguishes between sequences that have the same words but in a different order, whereas the aforementioned approaches effectively ignore order information.

所提出的 RNN 编码器–解码器与 [Zou et al., 2013]、[Chandar et al., 2014] 中方法的一个重要差别是：它考虑了源短语和目标短语中的词序。即便两个序列包含相同的词但顺序不同，RNN 编码器–解码器也能自然加以区分，而前述方法实际上忽略了顺序信息。

The closest approach related to the proposed RNN Encoder–Decoder is the Recurrent Continuous Translation Model (Model 2) proposed in [\citenameKalchbrenner and Blunsom2013]. In their paper, they proposed a similar model that consists of an encoder and decoder. The difference with our model is that they used a convolutional $n$-gram model (CGM) for the encoder and the hybrid of an inverse CGM and a recurrent neural network for the decoder. They, however, evaluated their model on rescoring the $n$-best list proposed by the conventional SMT system and computing the perplexity of the gold standard translations.

与所提出的 RNN 编码器–解码器最接近的，是 [Kalchbrenner and Blunsom, 2013] 提出的循环连续翻译模型（模型 2）。他们同样提出了由编码器和解码器组成的类似模型。与我们的差别在于：他们的编码器使用卷积 $n$-gram 模型（CGM），解码器则是反向 CGM 与循环神经网络的混合。不过，他们是在对传统 SMT 系统给出的 $n$-best 列表重新打分、以及计算标准译文的困惑度上评估该模型的。

4 Experiments

4 实验

We evaluate our approach on the English/French translation task of the WMT’14 workshop.

我们在 WMT’14 研讨会的英/法翻译任务上评估该方法。

4.1 Data and Baseline System

4.1 数据与基线系统

Large amounts of resources are available to build an English/French SMT system in the framework of the WMT’14 translation task. The bilingual corpora include Europarl (61M words), news commentary (5.5M), UN (421M), and two crawled corpora of 90M and 780M words respectively. The last two corpora are quite noisy. To train the French language model, about 712M words of crawled newspaper material is available in addition to the target side of the bitexts. All the word counts refer to French words after tokenization.

在 WMT’14 翻译任务的框架下，构建英/法 SMT 系统可利用的资源相当丰富。双语语料包括欧洲议会语料 Europarl（6100 万词）、新闻评论（550 万词）、联合国语料（4.21 亿词），以及规模分别为 9000 万词和 7.80 亿词的两份网络抓取语料。后两份语料噪声较大。训练法语语言模型时，除双语文本的目标端之外，还可使用约 7.12 亿词的抓取报纸文本。以上词数均指分词后的法语词。

It is commonly acknowledged that training statistical models on the concatenation of all this data does not necessarily lead to optimal performance, and results in extremely large models which are difficult to handle. Instead, one should focus on the most relevant subset of the data for a given task. We have done so by applying the data selection method proposed in [\citenameMoore and Lewis2010], and its extension to bitexts [\citenameAxelrod et al.2011]. By these means we selected a subset of 418M words out of more than 2G words for language modeling and a subset of 348M out of 850M words for training the RNN Encoder–Decoder. We used the test set newstest2012 and 2013 for data selection and weight tuning with MERT, and newstest2014 as our test set. Each set has more than 70 thousand words and a single reference translation.

人们普遍认为，把这些数据全部拼接起来训练统计模型，并不一定带来最优性能，而且会使模型极大、难以操作。更合理的做法是，针对给定任务聚焦于最相关的数据子集。我们采用了 [Moore and Lewis, 2010] 提出的数据选择方法，以及它在双语文本上的扩展 [Axelrod et al., 2011]。据此，我们从超过 20 亿词中选出 4.18 亿词用于语言建模，并从 8.50 亿词中选出 3.48 亿词用于训练 RNN 编码器–解码器。数据选择以及用 MERT 调权重时使用测试集 newstest2012 和 2013，newstest2014 作为我们的测试集。每个集合都有七万词以上，并带有单一参考译文。

For training the neural networks, including the proposed RNN Encoder–Decoder, we limited the source and target vocabulary to the most frequent 15,000 words for both English and French. This covers approximately 93% of the dataset. All the out-of-vocabulary words were mapped to a special token ($\left[\text{UNK}\right]$).

在训练包括所提出的 RNN 编码器–解码器在内的神经网络时，我们把英语和法语的源、目标词表都限制为各自最常见的 15,000 个词。这大约覆盖了数据集的 93%。所有未登录词都被映射到一个特殊记号 $\left[\text{UNK}\right]$。

The baseline phrase-based SMT system was built using Moses with default settings. This system achieves a BLEU score of 30.64 and 33.3 on the development and test sets, respectively (see Table 1).

基线基于短语的 SMT 系统使用 Moses 按默认设置构建。该系统在开发集和测试集上的 BLEU 分别为 30.64 和 33.3（见表 1）。

Models	BLEU
dev	test
Baseline	30.64	33.30
RNN	31.20	33.87
CSLM + RNN	31.48	34.64
CSLM + RNN + WP	31.50	34.54

Table 1: BLEU scores computed on the development and test sets using different combinations of approaches. WP denotes a word penalty, where we penalizes the number of unknown words to neural networks.

模型	BLEU
开发集	测试集
Baseline	30.64	33.30
RNN	31.20	33.87
CSLM + RNN	31.48	34.64
CSLM + RNN + WP	31.50	34.54

表 1：在开发集和测试集上、用不同方法组合算出的 BLEU 分数。WP 表示词惩罚，即对神经网络未知的词的数量加以惩罚。

4.1.1 RNN Encoder–Decoder

The RNN Encoder–Decoder used in the experiment had 1000 hidden units with the proposed gates at the encoder and at the decoder. The input matrix between each input symbol $x_{\left<t\right>}$ and the hidden unit is approximated with two lower-rank matrices, and the output matrix is approximated similarly. We used rank-100 matrices, equivalent to learning an embedding of dimension 100 for each word. The activation function used for $\tilde{h}$ in Eq. (8) is a hyperbolic tangent function. The computation from the hidden state in the decoder to the output is implemented as a deep neural network [\citenamePascanu et al.2014] with a single intermediate layer having 500 maxout units each pooling 2 inputs [\citenameGoodfellow et al.2013].

4.1.1 RNN 编码器–解码器

实验中使用的 RNN 编码器–解码器在编码器和解码器两侧都有 1000 个带有所提出门控的隐藏单元。每个输入符号 $x_{\left<t\right>}$ 与隐藏单元之间的输入矩阵用两个低秩矩阵近似，输出矩阵也同样近似。我们使用秩为 100 的矩阵，这等价于为每个词学习 100 维嵌入。式 (8) 中 $\tilde{h}$ 所用的激活函数是双曲正切。从解码器隐藏状态到输出的计算实现为一个深度神经网络 [Pascanu et al., 2014]，其中有一层含 500 个 maxout 单元的中间层，每个单元对 2 个输入做池化 [Goodfellow et al., 2013]。

All the weight parameters in the RNN Encoder–Decoder were initialized by sampling from an isotropic zero-mean (white) Gaussian distribution with its standard deviation fixed to $0.01$, except for the recurrent weight parameters. For the recurrent weight matrices, we first sampled from a white Gaussian distribution and used its left singular vectors matrix, following [\citenameSaxe et al.2014].

RNN 编码器–解码器中的全部权重参数，除循环权重外，都从标准差固定为 $0.01$ 的各向同性零均值（白）高斯分布中采样初始化。对于循环权重矩阵，我们先从白高斯分布中采样，再取其左奇异向量矩阵，做法遵循 [Saxe et al., 2014]。

We used Adadelta and stochastic gradient descent to train the RNN Encoder–Decoder with hyperparameters $\epsilon=10^{-6}$ and $\rho=0.95$ [\citenameZeiler2012]. At each update, we used 64 randomly selected phrase pairs from a phrase table (which was created from 348M words). The model was trained for approximately three days.

我们使用 Adadelta 和随机梯度下降训练 RNN 编码器–解码器，超参数为 $\epsilon=10^{-6}$、$\rho=0.95$ [Zeiler, 2012]。每次更新时，从短语表（由 3.48 亿词构建）中随机选取 64 个短语对。模型大约训练了三天。

Details of the architecture used in the experiments are explained in more depth in the supplementary material.

实验中所用结构的更多细节在补充材料中有更深入的说明。

Source	Translation Model	RNN Encoder–Decoder
at the end of the	[a la fin de la] [ŕ la fin des années] [être supprimés à la fin de la]	[à la fin du] [à la fin des] [à la fin de la]
for the first time	[r © pour la premirëre fois] [été donnés pour la première fois] [été commémorée pour la première fois]	[pour la première fois] [pour la première fois ,] [pour la première fois que]
in the United States and	[? aux **?**tats-Unis et] [été ouvertes aux États-Unis et] [été constatées aux États-Unis et]	[aux Etats-Unis et] [des Etats-Unis et] [des États-Unis et]
, as well as	[**?s , qu’] [?s , ainsi que] [?**re aussi bien que]	[, ainsi qu’] [, ainsi que] [, ainsi que les]
one of the most	[**?**t **?l’ un des plus] [?**l’ un des plus] [être retenue comme un de ses plus]	[l’ un des] [le] [un des]
(a) Long, frequent source phrases
Source	Translation Model	RNN Encoder–Decoder
, Minister of Communications and Transport	[Secrétaire aux communications et aux transports :] [Secrétaire aux communications et aux transports]	[Secrétaire aux communications et aux transports] [Secrétaire aux communications et aux transports :]
did not comply with the	[vestimentaire , ne correspondaient pas à des] [susmentionnée n’ était pas conforme aux] [présentées n’ étaient pas conformes à la]	[n’ ont pas respecté les] [n’ était pas conforme aux] [n’ ont pas respecté la]
parts of the world .	[© gions du monde .] [régions du monde considérées .] [région du monde considérée .]	[parties du monde .] [les parties du monde .] [des parties du monde .]
the past few days .	[le petit texte .] [cours des tout derniers jours .] [les tout derniers jours .]	[ces derniers jours .] [les derniers jours .] [cours des derniers jours .]
on Friday and Saturday	[vendredi et samedi à la] [vendredi et samedi à] [se déroulera vendredi et samedi ,]	[le vendredi et le samedi] [le vendredi et samedi] [vendredi et samedi]
(b) Long, rare source phrases

Table 2: The top scoring target phrases for a small set of source phrases according to the translation model (direct translation probability) and by the RNN Encoder–Decoder. Source phrases were randomly selected from phrases with 4 or more words. ? denotes an incomplete (partial) character. r is a Cyrillic letter ghe.

源短语	翻译模型	RNN 编码器–解码器
at the end of the	[a la fin de la] [ŕ la fin des années] [être supprimés à la fin de la]	[à la fin du] [à la fin des] [à la fin de la]
for the first time	[r © pour la premirëre fois] [été donnés pour la première fois] [été commémorée pour la première fois]	[pour la première fois] [pour la première fois ,] [pour la première fois que]
in the United States and	[? aux **?**tats-Unis et] [été ouvertes aux États-Unis et] [été constatées aux États-Unis et]	[aux Etats-Unis et] [des Etats-Unis et] [des États-Unis et]
, as well as	[**?s , qu’] [?s , ainsi que] [?**re aussi bien que]	[, ainsi qu’] [, ainsi que] [, ainsi que les]
one of the most	[**?**t **?l’ un des plus] [?**l’ un des] [être retenue comme un de ses plus]	[l’ un des] [le] [un des]
（a）较长且常见的源短语
源短语	翻译模型	RNN 编码器–解码器
, Minister of Communications and Transport	[Secrétaire aux communications et aux transports :] [Secrétaire aux communications et aux transports]	[Secrétaire aux communications et aux transports] [Secrétaire aux communications et aux transports :]
did not comply with the	[vestimentaire , ne correspondaient pas à des] [susmentionnée n’ était pas conforme aux] [présentées n’ étaient pas conformes à la]	[n’ ont pas respecté les] [n’ était pas conforme aux] [n’ ont pas respecté la]
parts of the world .	[© gions du monde .] [régions du monde considérées .] [région du monde considérée .]	[parties du monde .] [les parties du monde .] [des parties du monde .]
the past few days .	[le petit texte .] [cours des tout derniers jours .] [les tout derniers jours .]	[ces derniers jours .] [les derniers jours .] [cours des derniers jours .]
on Friday and Saturday	[vendredi et samedi à la] [vendredi et samedi à] [se déroulera vendredi et samedi ,]	[le vendredi et le samedi] [le vendredi et samedi] [vendredi et samedi]
（b）较长且罕见的源短语

表 2：对一小部分源短语，按翻译模型（直接翻译概率）和按 RNN 编码器–解码器各自打分最高的目标短语。源短语从长度为 4 词及以上的短语中随机选取。? 表示不完整（残缺）的字符。r 是西里尔字母 ghe。

4.1.2 Neural Language Model

In order to assess the effectiveness of scoring phrase pairs with the proposed RNN Encoder–Decoder, we also tried a more traditional approach of using a neural network for learning a target language model (CSLM) [\citenameSchwenk2007]. Especially, the comparison between the SMT system using CSLM and that using the proposed approach of phrase scoring by RNN Encoder–Decoder will clarify whether the contributions from multiple neural networks in different parts of the SMT system add up or are redundant.

4.1.2 神经语言模型

为了评估用所提出的 RNN 编码器–解码器为短语对打分是否有效，我们也尝试了更传统的做法：用神经网络学习目标语言模型（连续空间语言模型，CSLM）[Schwenk, 2007]。尤其是，把使用 CSLM 的 SMT 系统与使用 RNN 编码器–解码器为短语打分的系统加以比较，可以澄清：出现在 SMT 系统不同部位的多个神经网络，其贡献是相互叠加，还是彼此冗余。

We trained the CSLM model on 7-grams from the target corpus. Each input word was projected into the embedding space $\mathbb{R}^{512}$, and they were concatenated to form a 3072-dimensional vector. The concatenated vector was fed through two rectified layers (of size 1536 and 1024) [\citenameGlorot et al.2011]. The output layer was a simple softmax layer (see Eq. (2)). All the weight parameters were initialized uniformly between $-0.01$ and $0.01$, and the model was trained until the validation perplexity did not improve for 10 epochs. After training, the language model achieved a perplexity of 45.80. The validation set was a random selection of 0.1% of the corpus. The model was used to score partial translations during the decoding process, which generally leads to higher gains in BLEU score than n-best list rescoring [\citenameVaswani et al.2013].

我们在目标语料的 7-gram 上训练 CSLM。每个输入词被投影到嵌入空间 $\mathbb{R}^{512}$，再拼接成 3072 维向量。该拼接向量经过两层修正线性层（大小分别为 1536 和 1024）[Glorot et al., 2011]。输出层是简单的 softmax 层（见式 (2)）。全部权重参数在 $-0.01$ 与 $0.01$ 之间均匀初始化，模型一直训练到验证集困惑度连续 10 个 epoch 不再下降为止。训练结束后，该语言模型的困惑度为 45.80。验证集是从语料中随机抽取的 0.1%。该模型在解码过程中为部分译文打分，这通常比只对 $n$-best 列表重新打分能带来更大的 BLEU 提升 [Vaswani et al., 2013]。

To address the computational complexity of using a CSLM in the decoder a buffer was used to aggregate n-grams during the stack-search performed by the decoder. Only when the buffer is full, or a stack is about to be pruned, the n-grams are scored by the CSLM. This allows us to perform fast matrix-matrix multiplication on GPU using Theano [\citenameBergstra et al.2010, \citenameBastien et al.2012].

为了应对在解码器中使用 CSLM 的计算复杂度，我们在解码器进行栈搜索时用一个缓冲区汇聚 n-gram。只有当缓冲区已满，或某个栈即将被剪枝时，才由 CSLM 为这些 n-gram 打分。这样就可以用 Theano 在 GPU 上做快速的矩阵–矩阵乘法 [Bergstra et al., 2010, Bastien et al., 2012]。

Source	Samples from RNN Encoder–Decoder
at the end of the	[à la fin de la] ($\times 11$)
for the first time	[pour la première fois] ($\times 24$) [pour la première fois que] ($\times 2$)
in the United States and	[aux États-Unis et] ($\times 6$) [dans les États-Unis et] ($\times 4$)
, as well as	[, ainsi que] [,] [ainsi que] [, ainsi qu’] [et UNK]
one of the most	[l’ un des plus] ($\times 9$) [l’ un des] ($\times 5$) [l’ une des plus] ($\times 2$)
(a) Long, frequent source phrases
Source	Samples from RNN Encoder–Decoder
, Minister of Communications and Transport	[ , ministre des communications et le transport] ($\times 13$)
did not comply with the	[n’ tait pas conforme aux] [n’ a pas respect l’] ($\times 2$) [n’ a pas respect la] ($\times 3$)
parts of the world .	[arts du monde .] ($\times 11$) [des arts du monde .] ($\times 7$)
the past few days .	[quelques jours .] ($\times 5$) [les derniers jours .] ($\times 5$) [ces derniers jours .] ($\times 2$)
on Friday and Saturday	[vendredi et samedi] ($\times 5$) [le vendredi et samedi] ($\times 7$) [le vendredi et le samedi] ($\times 4$)
(b) Long, rare source phrases

Table 3: Samples generated from the RNN Encoder–Decoder for each source phrase used in Table 2. We show the top-5 target phrases out of 50 samples. They are sorted by the RNN Encoder–Decoder scores.

源短语	来自 RNN 编码器–解码器的样本
at the end of the	[à la fin de la] ($\times 11$)
for the first time	[pour la première fois] ($\times 24$) [pour la première fois que] ($\times 2$)
in the United States and	[aux États-Unis et] ($\times 6$) [dans les États-Unis et] ($\times 4$)
, as well as	[, ainsi que] [,] [ainsi que] [, ainsi qu’] [et UNK]
one of the most	[l’ un des plus] ($\times 9$) [l’ un des] ($\times 5$) [l’ une des plus] ($\times 2$)
（a）较长且常见的源短语
源短语	来自 RNN 编码器–解码器的样本
, Minister of Communications and Transport	[ , ministre des communications et le transport] ($\times 13$)
did not comply with the	[n’ tait pas conforme aux] [n’ a pas respect l’] ($\times 2$) [n’ a pas respect la] ($\times 3$)
parts of the world .	[arts du monde .] ($\times 11$) [des arts du monde .] ($\times 7$)
the past few days .	[quelques jours .] ($\times 5$) [les derniers jours .] ($\times 5$) [ces derniers jours .] ($\times 2$)
on Friday and Saturday	[vendredi et samedi] ($\times 5$) [le vendredi et samedi] ($\times 7$) [le vendredi et le samedi] ($\times 4$)
（b）较长且罕见的源短语

表 3：对表 2 中使用的每个源短语，由 RNN 编码器–解码器生成的样本。我们从 50 个样本中列出得分最高的 5 个目标短语，并按 RNN 编码器–解码器的分数排序。

Figure 3: The visualization of phrase pairs according to their scores (log-probabilities) by the RNN Encoder–Decoder and the translation model.

图 3：按 RNN 编码器–解码器和翻译模型给出的分数（对数概率）对短语对所作的可视化。

4.2 Quantitative Analysis

4.2 定量分析

We tried the following combinations:

我们尝试了以下组合：

1. Baseline configuration

1. 基线配置

2. Baseline + RNN

2. 基线 + RNN

3. Baseline + CSLM + RNN

3. 基线 + CSLM + RNN

4. Baseline + CSLM + RNN + Word penalty

4. 基线 + CSLM + RNN + 词惩罚

The results are presented in Table 1. As expected, adding features computed by neural networks consistently improves the performance over the baseline performance.

结果见表 1。正如所预期的，加入由神经网络计算的特征后，系统相对基线的表现持续得到提升。

The best performance was achieved when we used both CSLM and the phrase scores from the RNN Encoder–Decoder. This suggests that the contributions of the CSLM and the RNN Encoder–Decoder are not too correlated and that one can expect better results by improving each method independently. Furthermore, we tried penalizing the number of words that are unknown to the neural networks (i.e. words which are not in the shortlist). We do so by simply adding the number of unknown words as an additional feature the log-linear model in Eq. (9).[^3] However, in this case we were not able to achieve better performance on the test set, but only on the development set.

同时使用 CSLM 和来自 RNN 编码器–解码器的短语分数时，取得了最好的性能。这表明 CSLM 与 RNN 编码器–解码器的贡献相关程度并不高，可以预期通过分别改进这两种方法来获得更好的结果。此外，我们还尝试对神经网络未知的词（即不在短名单中的词）的数量加以惩罚。做法很简单：把未知词的个数作为式 (9) 对数线性模型中的额外特征。[^3]不过在这种情况下，我们只在开发集上得到了更好的结果，未能在测试集上取得更好的性能。

[^3]: To understand the effect of the penalty, consider the set of all words in the 15,000 large shortlist, SL. All words $x^{i}\notin\text{SL}$ are replaced by a special token $\left[\text{UNK}\right]$ before being scored by the neural networks. Hence, the conditional probability of any $x_{t}^{i}\notin\text{SL}$ is actually given by the model as

$$p\left(x_{t}=\left[\text{UNK}\right]\mid x_{<t}\right)=p\left(x_{t}\notin\text{SL}\mid x_{<t}\right)=\sum_{x_{t}^{j}\notin SL}p\left(x_{t}^{j}\mid x_{<t}\right)\geq p\left(x_{t}^{i}\mid x_{<t}\right),$$

where $x_{<t}$ is a shorthand notation for $x_{t-1},\dots,x_{1}$. As a result, the probability of words not in the shortlist is always overestimated. It is possible to address this issue by backing off to an existing model that contain non-shortlisted words (see [\citenameSchwenk2007]). In this paper, however, we opt for introducing a word penalty instead, which counteracts the word probability overestimation.

[^3]：要理解这一惩罚的作用，可考虑规模为 15,000 的短名单 SL 中的全部词。所有 $x^{i}\notin\text{SL}$ 的词在送入神经网络打分之前，都会被替换成特殊记号 $\left[\text{UNK}\right]$。因此，对任意 $x_{t}^{i}\notin\text{SL}$，模型实际给出的条件概率是

$$p\left(x_{t}=\left[\text{UNK}\right]\mid x_{<t}\right)=p\left(x_{t}\notin\text{SL}\mid x_{<t}\right)=\sum_{x_{t}^{j}\notin SL}p\left(x_{t}^{j}\mid x_{<t}\right)\geq p\left(x_{t}^{i}\mid x_{<t}\right),$$

其中 $x_{<t}$ 是 $x_{t-1},\dots,x_{1}$ 的简写。于是，不在短名单中的词的概率总是被高估。一种可能的解决办法是回退到包含非短名单词的已有模型（参见 [Schwenk, 2007]）。不过在本文中，我们选择引入词惩罚，以抵消这种词概率高估。

Figure 4: 2–D embedding of the learned word representation. The left one shows the full embedding space, while the right one shows a zoomed-in view of one region (color–coded). For more plots, see the supplementary material.

图 4：所学词表示的二维嵌入。左图展示完整嵌入空间，右图是其中一个区域的放大视图（按颜色编码）。更多图见补充材料。

Figure 5: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (5000 randomly selected points), while the other three figures show the zoomed-in view of specific regions (color–coded).

图 5：所学短语表示的二维嵌入。左上图展示完整表示空间（随机选取的 5000 个点），其余三张图是特定区域的放大视图（按颜色编码）。

4.3 Qualitative Analysis

4.3 定性分析

In order to understand where the performance improvement comes from, we analyze the phrase pair scores computed by the RNN Encoder–Decoder against the corresponding $p(\mathbf{f}\mid\mathbf{e})$ from the translation model. Since the existing translation model relies solely on the statistics of the phrase pairs in the corpus, we expect its scores to be better estimated for the frequent phrases but badly estimated for rare phrases. Also, as we mentioned earlier in Sec. 3.1, we further expect the RNN Encoder–Decoder which was trained without any frequency information to score the phrase pairs based rather on the linguistic regularities than on the statistics of their occurrences in the corpus.

为了理解性能提升从何而来，我们把 RNN 编码器–解码器算出的短语对分数，与翻译模型给出的相应 $p(\mathbf{f}\mid\mathbf{e})$ 加以对照。已有翻译模型完全依赖语料中短语对的统计量，因此可以预期：它对常见短语的分数估计较好，对罕见短语则估计较差。另外，正如第 3.1 节已经提到的，我们进一步预期：在完全不使用频率信息的情况下训练的 RNN 编码器–解码器，会更多地依据语言规律、而不是依据短语对在语料中的出现次数来打分。

We focus on those pairs whose source phrase is long (more than 3 words per source phrase) and frequent. For each such source phrase, we look at the target phrases that have been scored high either by the translation probability $p(\mathbf{f}\mid\mathbf{e})$ or by the RNN Encoder–Decoder. Similarly, we perform the same procedure with those pairs whose source phrase is long but rare in the corpus.

我们聚焦于源短语较长（每个源短语超过 3 个词）且常见的短语对。对每个这样的源短语，我们查看翻译概率 $p(\mathbf{f}\mid\mathbf{e})$ 或 RNN 编码器–解码器打分较高的那些目标短语。对源短语较长但在语料中罕见的短语对，我们也做同样的分析。

Table 2 lists the top-$3$ target phrases per source phrase favored either by the translation model or by the RNN Encoder–Decoder. The source phrases were randomly chosen among long ones having more than 4 or 5 words.

表 2 列出了翻译模型或 RNN 编码器–解码器各自更偏好的、每个源短语得分最高的 3 个目标短语。这些源短语是从长度超过 4 或 5 个词的较长短语中随机选取的。

In most cases, the choices of the target phrases by the RNN Encoder–Decoder are closer to actual or literal translations. We can observe that the RNN Encoder–Decoder prefers shorter phrases in general.

在大多数情况下，RNN 编码器–解码器所选的目标短语更接近实际或字面译文。还可以看到，RNN 编码器–解码器总体上更偏好较短的短语。

Interestingly, many phrase pairs were scored similarly by both the translation model and the RNN Encoder–Decoder, but there were as many other phrase pairs that were scored radically different (see Fig. 3). This could arise from the proposed approach of training the RNN Encoder–Decoder on a set of unique phrase pairs, discouraging the RNN Encoder–Decoder from learning simply the frequencies of the phrase pairs from the corpus, as explained earlier.

有意思的是，不少短语对在翻译模型和 RNN 编码器–解码器上的打分相近，但也有同样多的短语对打分差异极大（见图 3）。这可能来自我们提出的训练方式：只在一组互不重复的短语对上训练 RNN 编码器–解码器，从而抑制它去简单学习短语对在语料中的频率，如前所述。

Furthermore, in Table 3, we show for each of the source phrases in Table 2, the generated samples from the RNN Encoder–Decoder. For each source phrase, we generated 50 samples and show the top-five phrases accordingly to their scores. We can see that the RNN Encoder–Decoder is able to propose well-formed target phrases without looking at the actual phrase table. Importantly, the generated phrases do not overlap completely with the target phrases from the phrase table. This encourages us to further investigate the possibility of replacing the whole or a part of the phrase table with the proposed RNN Encoder–Decoder in the future.

此外，在表 3 中，我们针对表 2 里的每个源短语，展示了 RNN 编码器–解码器生成的样本。对每个源短语，我们生成 50 个样本，并按分数列出最高的五个短语。可以看到，即便不查阅实际的短语表，RNN 编码器–解码器也能提出形式良好的目标短语。重要的是，生成的短语与短语表中的目标短语并不完全重合。这促使我们在今后进一步考察：能否用所提出的 RNN 编码器–解码器替换整个或部分短语表。

4.4 Word and Phrase Representations

4.4 词表示与短语表示

Since the proposed RNN Encoder–Decoder is not specifically designed only for the task of machine translation, here we briefly look at the properties of the trained model.

由于所提出的 RNN 编码器–解码器并非专门只为机器翻译任务而设计，这里我们简要考察训练所得模型的性质。

It has been known for some time that continuous space language models using neural networks are able to learn semantically meaningful embeddings (See, e.g., [\citenameBengio et al.2003, \citenameMikolov et al.2013]). Since the proposed RNN Encoder–Decoder also projects to and maps back from a sequence of words into a continuous space vector, we expect to see a similar property with the proposed model as well.

人们已知，使用神经网络的连续空间语言模型能够学到在语义上有意义的嵌入（参见 [Bengio et al., 2003, Mikolov et al., 2013]）。所提出的 RNN 编码器–解码器同样把词序列投影到连续空间向量、再映射回来，因此我们预期该模型也会具有类似的性质。

The left plot in Fig. 4 shows the 2–D embedding of the words using the word embedding matrix learned by the RNN Encoder–Decoder. The projection was done by the recently proposed Barnes-Hut-SNE [\citenamevan der Maaten2013]. We can clearly see that semantically similar words are clustered with each other (see the zoomed-in plots in Fig. 4).

图 4 的左图展示了用 RNN 编码器–解码器学到的词嵌入矩阵所作的词的二维嵌入。投影采用了新近提出的 Barnes-Hut-SNE [van der Maaten, 2013]。可以清楚地看到，语义相近的词会聚集在一起（见图 4 中的放大图）。

The proposed RNN Encoder–Decoder naturally generates a continuous-space representation of a phrase. The representation ($\mathbf{c}$ in Fig. 1) in this case is a 1000-dimensional vector. Similarly to the word representations, we visualize the representations of the phrases that consists of four or more words using the Barnes-Hut-SNE in Fig. 5.

所提出的 RNN 编码器–解码器会自然地生成短语的连续空间表示。在本设定中，该表示（图 1 中的 $\mathbf{c}$）是一个 1000 维向量。与词表示类似，我们用 Barnes-Hut-SNE 对由四个或更多词组成的短语的表示作可视化，见图 5。

From the visualization, it is clear that the RNN Encoder–Decoder captures both semantic and syntactic structures of the phrases. For instance, in the bottom-left plot, most of the phrases are about the duration of time, while those phrases that are syntactically similar are clustered together. The bottom-right plot shows the cluster of phrases that are semantically similar (countries or regions). On the other hand, the top-right plot shows the phrases that are syntactically similar.

从可视化中可以清楚地看出，RNN 编码器–解码器同时抓住了短语的语义和句法结构。例如，在左下图中，大多数短语都与时间长短有关，而句法相近的短语会聚在一起。右下图展示的是语义相近的短语簇（国家或地区）。另一方面，右上图展示的则是句法相近的短语。

5 Conclusion

5 结论

In this paper, we proposed a new neural network architecture, called an RNN Encoder–Decoder that is able to learn the mapping from a sequence of an arbitrary length to another sequence, possibly from a different set, of an arbitrary length. The proposed RNN Encoder–Decoder is able to either score a pair of sequences (in terms of a conditional probability) or generate a target sequence given a source sequence. Along with the new architecture, we proposed a novel hidden unit that includes a reset gate and an update gate that adaptively control how much each hidden unit remembers or forgets while reading/generating a sequence.

本文提出了一种称为 RNN 编码器–解码器 的新神经网络结构，它能够学习从任意长度的序列到另一组、同样任意长度的序列的映射。所提出的 RNN 编码器–解码器既可以为序列对打分（以条件概率衡量），也可以在给定源序列时生成目标序列。与这一新结构一起，我们还提出了一种新的隐藏单元，其中包含重置门和更新门，在读取/生成序列时自适应地控制每个隐藏单元记住多少、遗忘多少。

We evaluated the proposed model with the task of statistical machine translation, where we used the RNN Encoder–Decoder to score each phrase pair in the phrase table. Qualitatively, we were able to show that the new model is able to capture linguistic regularities in the phrase pairs well and also that the RNN Encoder–Decoder is able to propose well-formed target phrases.

我们在统计机器翻译任务上评估了所提出的模型，用 RNN 编码器–解码器为短语表中的每个短语对打分。定性分析表明，新模型能够很好地抓住短语对中的语言规律，并且 RNN 编码器–解码器能够提出形式良好的目标短语。

The scores by the RNN Encoder–Decoder were found to improve the overall translation performance in terms of BLEU scores. Also, we found that the contribution by the RNN Encoder–Decoder is rather orthogonal to the existing approach of using neural networks in the SMT system, so that we can improve further the performance by using, for instance, the RNN Encoder–Decoder and the neural net language model together.

实验发现，RNN 编码器–解码器给出的分数能够提高整体翻译性能（以 BLEU 衡量）。我们还发现，RNN 编码器–解码器的贡献与 SMT 系统中已有的神经网络用法相当正交，因此可以把 RNN 编码器–解码器与神经语言模型等一起使用，以进一步提高性能。

Our qualitative analysis of the trained model shows that it indeed captures the linguistic regularities in multiple levels i.e. at the word level as well as phrase level. This suggests that there may be more natural language related applications that may benefit from the proposed RNN Encoder–Decoder.

对训练所得模型的定性分析表明，它确实在多个层次上抓住了语言规律，即同时包括词一级和短语一级。这提示我们，可能还有更多与自然语言相关的应用，能够从所提出的 RNN 编码器–解码器中获益。

The proposed architecture has large potential for further improvement and analysis. One approach that was not investigated here is to replace the whole, or a part of the phrase table by letting the RNN Encoder–Decoder propose target phrases. Also, noting that the proposed model is not limited to being used with written language, it will be an important future research to apply the proposed architecture to other applications such as speech transcription.

所提出的结构还有很大的改进和分析空间。本文没有考察的一条路径是：让 RNN 编码器–解码器提出目标短语，从而替换整个或部分短语表。另外，该模型并不限于书面语言，把这一结构应用到语音转写等其他任务上，将是一项重要的未来研究。

Acknowledgments

致谢

KC, BM, CG, DB and YB would like to thank NSERC, Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR. FB and HS were partially funded by the European Commission under the project MateCat, and by DARPA under the BOLT project.

KC、BM、CG、DB 和 YB 感谢 NSERC、Calcul Québec、Compute Canada、加拿大研究讲席以及 CIFAR。FB 和 HS 的部分工作由欧盟委员会 MateCat 项目以及 DARPA 的 BOLT 项目资助。

References

参考文献

[Auli et al.2013] Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. 2013. Joint language and translation modeling with recurrent neural networks. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1044–1054.

[Axelrod et al.2011] Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 355–362.

[Bastien et al.2012] Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.

[Bengio et al.2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res. , 3:1137–1155, March.

[Bengio et al.2013] Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu. 2013. Advances in optimizing recurrent networks. In Proceedings of the 38th International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2013) , May.

[Bergstra et al.2010] James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy) , June. Oral Presentation.

[Chandar et al.2014] Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. arXiv: 1402.1454 [cs.CL] , February.

[Dahl et al.2012] George E. Dahl, Dong Yu, Li Deng, and Alex Acero. 2012. Context-dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing , 20(1):33–42.

[Devlin et al.2014] Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, , and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the ACL 2014 Conference , ACL ’14, pages 1370–1380.

[Gao et al.2013] Jianfeng Gao, Xiaodong He, Wen tau Yih, and Li Deng. 2013. Learning semantic representations for the phrase translation model. Technical report, Microsoft Research.

[Glorot et al.2011] X. Glorot, A. Bordes, and Y. Bengio. 2011. Deep sparse rectifier neural networks. In AISTATS’2011 .

[Goodfellow et al.2013] Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Maxout networks. In ICML’2013 .

[Graves2012] Alex Graves. 2012. Supervised Sequence Labelling with Recurrent Neural Networks . Studies in Computational Intelligence. Springer.

[Hochreiter and Schmidhuber1997] S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation , 9(8):1735–1780.

[Kalchbrenner and Blunsom2013] Nal Kalchbrenner and Phil Blunsom. 2013. Two recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1700–1709.

[Koehn et al.2003] Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1 , NAACL ’03, pages 48–54.

[Koehn2005] P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Machine Translation Summit X , pages 79–86, Phuket, Thailand.

[Krizhevsky et al.2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25 (NIPS’2012) .

[Marcu and Wong2002] Daniel Marcu and William Wong. 2002. A phrase-based, joint probability model for statistical machine translation. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10 , EMNLP ’02, pages 133–139.

[Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26 , pages 3111–3119.

[Moore and Lewis2010] Robert C. Moore and William Lewis. 2010. Intelligent selection of language model training data. In Proceedings of the ACL 2010 Conference Short Papers , ACLShort ’10, pages 220–224, Stroudsburg, PA, USA.

[Pascanu et al.2014] R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio. 2014. How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014) , April.

[Saxe et al.2014] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. 2014. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014) , April.

[Schwenk et al.2006] Holger Schwenk, Marta R. Costa-Jussà, and José A. R. Fonollosa. 2006. Continuous space language models for the iwslt 2006 task. In IWSLT , pages 166–173.

[Schwenk2007] Holger Schwenk. 2007. Continuous space language models. Comput. Speech Lang. , 21(3):492–518, July.

[Schwenk2012] Holger Schwenk. 2012. Continuous space translation models for phrase-based statistical machine translation. In Martin Kay and Christian Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN) , pages 1071–1080.

[Socher et al.2011] Richard Socher, Eric H. Huang, Jeffrey Pennington, Andrew Y. Ng, and Christopher D. Manning. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems 24 .

[Son et al.2012] Le Hai Son, Alexandre Allauzen, and François Yvon. 2012. Continuous space translation models with neural networks. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , NAACL HLT ’12, pages 39–48, Stroudsburg, PA, USA.

[van der Maaten2013] Laurens van der Maaten. 2013. Barnes-hut-sne. In Proceedings of the First International Conference on Learning Representations (ICLR 2013) , May.

[Vaswani et al.2013] Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. Proceedings of the Conference on Empirical Methods in Natural Language Processing , pages 1387–1392.

[Zeiler2012] Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. Technical report, arXiv 1212.5701.

[Zou et al.2013] Will Y. Zou, Richard Socher, Daniel M. Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1393–1398.

Appendix A RNN Encoder–Decoder

附录 A RNN 编码器–解码器

In this document, we describe in detail the architecture of the RNN Encoder–Decoder used in the experiments.

本附录详细说明实验中使用的 RNN 编码器–解码器结构。

Let us denote an source phrase by $X=\left(\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{N}\right)$ and a target phrase by $Y=\left(\mathbf{y}_{1},\mathbf{y}_{2},\dots,\mathbf{y}_{M}\right)$. Each phrase is a sequence of $K$-dimensional one-hot vectors, such that only one element of the vector is $1$ and all the others are $0$. The index of the active ($1$) element indicates the word represented by the vector.

把源短语记为 $X=\left(\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{N}\right)$，目标短语记为 $Y=\left(\mathbf{y}_{1},\mathbf{y}_{2},\dots,\mathbf{y}_{M}\right)$。每个短语都是由 $K$ 维 one-hot 向量组成的序列，每个向量只有一个元素为 $1$，其余均为 $0$。取值为 $1$ 的那个元素的下标，就指示该向量所表示的词。

A.1 Encoder

A.1 编码器

Each word of the source phrase is embedded in a $500$-dimensional vector space: $e(\mathbf{x}_{i})\in\mathbb{R}^{500}$. $e(\mathbf{x})$ is used in Sec. 4.4 to visualize the words.

源短语中的每个词都被嵌入到 $500$ 维向量空间中：$e(\mathbf{x}_{i})\in\mathbb{R}^{500}$。第 4.4 节中对词的可视化使用的就是 $e(\mathbf{x})$。

The hidden state of an encoder consists of $1000$ hidden units, and each one of them at time $t$ is computed by

编码器的隐藏状态由 $1000$ 个隐藏单元组成，其中第 $j$ 个单元在时刻 $t$ 由下式计算

$$\displaystyle h_{j}^{\left<t\right>}=z_{j}h_{j}^{\left<t-1\right>}+(1-z_{j})\tilde{h}_{j}^{\left<t\right>},$$

where

其中

$$\displaystyle\tilde{h}_{j}^{\left<t\right>}= \\ \displaystyle\tanh\left(\left[\mathbf{W}e(\mathbf{x}_{t})\right]_{j}+\left[\mathbf{U}\left(\mathbf{r}\odot\mathbf{h}_{\left<t-1\right>}\right)\right]_{j}\right), \\ \displaystyle z_{j}= \\ \displaystyle\sigma\left(\left[\mathbf{W}_{z}e(\mathbf{x}_{t})\right]_{j}+\left[\mathbf{U}_{z}\mathbf{h}_{\left<t-1\right>}\right]_{j}\right), \\ \displaystyle r_{j}= \\ \displaystyle\sigma\left(\left[\mathbf{W}_{r}e(\mathbf{x}_{t})\right]_{j}+\left[\mathbf{U}_{r}\mathbf{h}_{\left<t-1\right>}\right]_{j}\right).$$

$\sigma$ and $\odot$ are a logistic sigmoid function and an element-wise multiplication, respectively. To make the equations uncluttered, we omit biases. The initial hidden state $h_{j}^{\left<0\right>}$ is fixed to $0$.

$\sigma$ 和 $\odot$ 分别表示 logistic sigmoid 函数和逐元素乘法。为使公式简洁，我们省略了偏置项。初始隐藏状态 $h_{j}^{\left<0\right>}$ 固定为 $0$。

Once the hidden state at the $N$ step (the end of the source phrase) is computed, the representation of the source phrase $\mathbf{c}$ is

一旦计算完第 $N$ 步（源短语末尾）的隐藏状态，源短语的表示 $\mathbf{c}$ 即为

$$\displaystyle\mathbf{c}=\tanh\left(\mathbf{V}\mathbf{h}^{\left<N\right>}\right).$$

A.1.1 Decoder

The decoder starts by initializing the hidden state with

A.1.1 解码器

解码器首先用下式初始化隐藏状态

$$\displaystyle{\mathbf{h}^{\prime}}^{\left<0\right>}=\tanh\left(\mathbf{V}^{\prime}\mathbf{c}\right),$$

where we will use $\cdot^{\prime}$ to distinguish parameters of the decoder from those of the encoder.

其中我们用 $\cdot^{\prime}$ 来区分解码器的参数与编码器的参数。

The hidden state at time $t$ of the decoder is computed by

解码器在时刻 $t$ 的隐藏状态由下式计算

$$\displaystyle{h^{\prime}}_{j}^{\left<t\right>}={z^{\prime}}_{j}{h^{\prime}}_{j}^{\left<t-1\right>}+(1-{z^{\prime}}_{j})\tilde{h^{\prime}}_{j}^{\left<t\right>},$$

where

其中

$$\displaystyle\tilde{h^{\prime}}_{j}^{\left<t\right>}= \\ \displaystyle\tanh\left(\left[\mathbf{W}^{\prime}e(\mathbf{y}_{t-1})\right]_{j}+{r^{\prime}}_{j}\left[\mathbf{U}^{\prime}{\mathbf{h}^{\prime}}_{\left<t-1\right>}+\mathbf{C}\mathbf{c}\right]\right), \\ \displaystyle{z^{\prime}}_{j}= \\ \displaystyle\sigma\left(\left[{\mathbf{W}^{\prime}}_{z}e(\mathbf{y}_{t-1})\right]_{j}+\left[{\mathbf{U}^{\prime}}_{z}{\mathbf{h}^{\prime}}_{\left<t-1\right>}\right]_{j}+\left[\mathbf{C}_{z}\mathbf{c}\right]_{j}\right), \\ \displaystyle{r^{\prime}}_{j}= \\ \displaystyle\sigma\left(\left[{\mathbf{W}^{\prime}}_{r}e(\mathbf{y}_{t-1})\right]_{j}+\left[{\mathbf{U}^{\prime}}_{r}{\mathbf{h}^{\prime}}_{\left<t-1\right>}\right]_{j}+\left[\mathbf{C}_{r}\mathbf{c}\right]_{j}\right),$$

and $e(\mathbf{y}_{0})$ is an all-zero vector. Similarly to the case of the encoder, $e(\mathbf{y})$ is an embedding of a target word.

并且 $e(\mathbf{y}_{0})$ 是全零向量。与编码器的情形类似，$e(\mathbf{y})$ 是目标词的嵌入。

Unlike the encoder which simply encodes the source phrase, the decoder is learned to generate a target phrase. At each time $t$, the decoder computes the probability of generating $j$-th word by

与只负责编码源短语的编码器不同，解码器被学习用来生成目标短语。在每个时刻 $t$，解码器按下式计算生成第 $j$ 个词的概率

$$\displaystyle p(y_{t,j}=1\mid\mathbf{y}_{t-1},\dots,\mathbf{y}_{1},X)=\frac{\exp\left({\mathbf{g}}_{j}\mathbf{s}_{\left<t\right>}\right)}{\sum_{j^{\prime}=1}^{K}\exp\left(\mathbf{g}_{j^{\prime}}\mathbf{s}_{\left<t\right>}\right)},$$

where the $i$-element of $\mathbf{s}_{\left<t\right>}$ is

其中 $\mathbf{s}_{\left<t\right>}$ 的第 $i$ 个元素为

$$\displaystyle s_{i}^{\left<t\right>}=\max\left\{{s^{\prime}}_{2i-1}^{\left<t\right>},{s^{\prime}}_{2i}^{\left<t\right>}\right\}$$

and

并且

$$\displaystyle{\mathbf{s}^{\prime}}^{\left<t\right>}=\mathbf{O}_{h}{\mathbf{h}^{\prime}}^{\left<t\right>}+\mathbf{O}_{y}\mathbf{y}_{t-1}+\mathbf{O}_{c}\mathbf{c}.$$

In short, the $s_{i}^{\left<t\right>}$ is a so-called maxout unit.

简而言之，$s_{i}^{\left<t\right>}$ 就是所谓的 maxout 单元。

For the computational efficiency, instead of a single-matrix output weight $\mathbf{G}$, we use a product of two matrices such that

出于计算效率的考虑，我们不用单一的输出权重矩阵 $\mathbf{G}$，而是使用两个矩阵的乘积，使得

$$\displaystyle\mathbf{G}=\mathbf{G}_{l}\mathbf{G}_{r},$$

where $\mathbf{G}_{l}\in\mathbb{R}^{K\times 500}$ and $\mathbf{G}_{r}\in\mathbb{R}^{500\times 1000}$.

其中 $\mathbf{G}_{l}\in\mathbb{R}^{K\times 500}$，$\mathbf{G}_{r}\in\mathbb{R}^{500\times 1000}$。

Appendix B Word and Phrase Representations

附录 B 词表示与短语表示

Here, we show enlarged plots of the word and phrase representations in Figs. 4–5.

这里给出图 4–5 中词表示与短语表示的放大图。

Figure 6: 2–D embedding of the learned word representation. The top left one shows the full embedding space, while the other three figures show the zoomed-in view of specific regions (color–coded).

图 6：所学词表示的二维嵌入。左上图展示完整嵌入空间，其余三张图是特定区域的放大视图（按颜色编码）。

Figure 7: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (1000 randomly selected points), while the other three figures show the zoomed-in view of specific regions (color–coded).

图 7：所学短语表示的二维嵌入。左上图展示完整表示空间（随机选取的 1000 个点），其余三张图是特定区域的放大视图（按颜色编码）。