Neural Machine Translation by Jointly Learning to Align and Translate

通过联合学习对齐与翻译的神经机器翻译（Neural Machine Translation by Jointly Learning to Align and Translate）

Dzmitry Bahdanau Affiliation:  Jacobs University Bremen, Germany    KyungHyun Cho       Yoshua Bengio † † thanks: CIFAR Senior Fellow Affiliation:  Université de Montréal

Dzmitry Bahdanau
隶属：德国不来梅雅可比大学（Jacobs University Bremen, Germany）

KyungHyun Cho　　Yoshua Bengio † † 注：CIFAR 资深研究员
隶属：加拿大蒙特利尔大学（Université de Montréal）

Abstract

摘要

Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder–decoders and encode a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder–decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.

神经机器翻译是近来提出的一种机器翻译方法。与传统统计机器翻译不同，它的目标是构建单个神经网络，并对其联合调优以最大化翻译表现。现有模型多为编码器–解码器族：编码器把源句读成定长向量，解码器从该向量生成译文。本文猜想，使用定长向量是提升这类基本编码器–解码器架构性能的瓶颈，于是提出能联合学习对齐与翻译的方法来扩展它。注意力机制使模型在生成译文时自动（软）搜索源句中与之相关的部分。用英→法翻译任务做实验，定性结果表明所提模型很容易找到对译文中各词有意义的（软）对齐；从定量结果看，相对基本编码器–解码器，新方法大幅提升翻译表现，译文长度无论长短都成立。

1 Introduction

1 引言

Neural machine translation is a newly emerging approach to machine translation, recently proposed by Kalchbrenner and Blunsom (2013), Sutskever et al. (2014) and Cho et al. (2014b). Unlike the traditional phrase-based translation system (Koehn et al., 2003, see, e.g.,) which consists of many small sub-components that are tuned separately, neural machine translation attempts to build and train a single, large neural network that reads a sentence and outputs a correct translation.

• 神经机器翻译* 是新出现的机器翻译方法, 往日由 好吗? Kalchbrenner和Bl各项人权(2013年)、Sutskever等人(2014年)和Cho等人(如何才能)提出。 偏差 偏差 将与固然不同 偏差 偏差 偏提供关于 偏差 偏差帘布可从中分离出来 偏差 偏差 偏差 偏差 偏差 偏 你要我 奕 偏差 偏差 偏 会 偏差 偏差 偏差 偏 我们要 偏差

Most of the proposed neural machine translation models belong to a family of encoder–decoders (Sutskever et al., 2014; Cho et al., 2014a), with an encoder and a decoder for each language, or involve a language-specific encoder applied to each sentence whose outputs are then compared (Hermann and Blunsom, 2014). An encoder neural network reads and encodes a source sentence into a fixed-length vector. A decoder then outputs a translation from the encoded vector. The whole encoder–decoder system, which consists of the encoder and the decoder for a language pair, is jointly trained to maximize the probability of a correct translation given a source sentence.

大多数已提出的神经机器翻译模型属于编码器–解码器一族（Sutskever 等, 2014；Cho 等, 2014a），每种语言各有一个编码器与一个解码器；也有工作对每个句子使用语言特定编码器，再比较其输出（Hermann and Blunsom, 2014）。编码器神经网络读入源句并编码成定长向量。解码器再从该编码向量输出译文。由一对语言的编码器与解码器组成的整个编码器–解码器系统，被联合训练以最大化给定源句时正确译文的概率。

A potential issue with this encoder–decoder approach is that a neural network needs to be able to compress all the necessary information of a source sentence into a fixed-length vector. This may make it difficult for the neural network to cope with long sentences, especially those that are longer than the sentences in the training corpus. Cho et al. (2014b) showed that indeed the performance of a basic encoder–decoder deteriorates rapidly as the length of an input sentence increases.

这种编码器–解码器方法的一个潜在问题是,神经网络需要能够将源轿句的所有必要信息压缩成固定长度的矢量. 怯懦可能使神经网络难以应对长刑期,尤其是比训练恰当刑期长的刑期. Cho等人(2014年b)表明,随着输入句子长度的增加,一个基本编码器-解码器的性能确实迅速恶化。

In order to address this issue, we introduce an extension to the encoder–decoder model which learns to align and translate jointly. Each time the proposed model generates a word in a translation, it (soft-)searches for a set of positions in a source sentence where the most relevant information is concentrated. The model then predicts a target word based on the context vectors associated with these source positions and all the previous generated target words.

为解决这一问题，我们给编码器–解码器模型引入一种扩展，使其能联合学习对齐与翻译。每当所提模型生成译文中的一个词时，它（软）搜索源句中最相关的一组位置。模型再根据这些关联位置的上下文向量以及此前生成的所有目标词，预测下一个目标词。

The most important distinguishing feature of this approach from the basic encoder–decoder is that it does not attempt to encode a whole input sentence into a single fixed-length vector. Instead, it encodes the input sentence into a sequence of vectors and chooses a subset of these vectors adaptively while decoding the translation. This frees a neural translation model from having to squash all the information of a source sentence, regardless of its length, into a fixed-length vector. We show this allows a model to cope better with long sentences.

相对基本编码器–解码器，这一方法最重要的区别是：它并不试图把整句输入编码成单个定长向量。相反，它把输入句编码成一串向量，并在解码译文时自适应地选择其中一部分向量。这使得模型不必把源句的全部信息，无论长短，都压进一个定长向量。

In this paper, we show that the proposed approach of jointly learning to align and translate achieves significantly improved translation performance over the basic encoder–decoder approach. The improvement is more apparent with longer sentences, but can be observed with sentences of any length. On the task of English-to-French translation, the proposed approach achieves, with a single model, a translation performance comparable, or close, to the conventional phrase-based system. Furthermore, qualitative analysis reveals that the proposed model finds a linguistically plausible (soft-)alignment between a source sentence and the corresponding target sentence.

在本文中ừ地表明,拟议中的联合学习在国内统一和翻译的方法比基本编码器-编码器方法大大提高了翻译绩效。 刑期较长的 在英语对法语翻译的任务上,拟议方法实现了单一模式预期将实现的翻译性能与传统的短语制相当或接近.从一开始就认为,从语言上讲,从定性分析可以看出,拟议模式在源句和相应的铭句之间有了一个合理的强制性措施。

2 Background: Neural Machine Translation

2 背景：神经机器翻译

From a probabilistic perspective, translation is equivalent to finding a target sentence $\mathbf{y}$ that maximizes the conditional probability of $\mathbf{y}$ given a source sentence $\mathbf{x}$, i.e., $\argmax_{\mathbf{y}}p(\mathbf{y}\mid\mathbf{x})$. In neural machine translation, we fit a parameterized model to maximize the conditional probability of sentence pairs using a parallel training corpus. Once the conditional distribution is learned by a translation model, given a source sentence a corresponding translation can be generated by searching for the sentence that maximizes the conditional probability.

从概率论的角度来看,翻译相当于找到一个目标句$\mathbf{y}$,将$\mathbf{y}$的有条件概率最大化给源句$\mathbf{x}$身分,即$\argmax_{\mathbf{y}}p(\mathbf{y}\mid\mathbf{x})$. 在神经机翻译中,我们搭配一个参数化模型,利用一个平行的训练仪,使句子对的条件概率最大化法律依据. 一旦一个翻译模型得知了有条件 小分数,给一个源句就可以通过搜索可以 Rosemary最大程度实现有条件概率的句子生成相应的翻译.

Recently, a number of papers have proposed the use of neural networks to directly learn this conditional distribution (Kalchbrenner and Blunsom, 2013; Cho et al., 2014a; Sutskever et al., 2014; Cho et al., 2014b; Forcada and Ñeco, 1997, see, e.g.,). This neural machine translation approach typically consists of two components, the first of which encodes a source sentence $\mathbf{x}$ and the second decodes to a target sentence $\mathbf{y}$. For instance, two recurrent neural networks (RNN) were used by (Cho et al., 2014a) and (Sutskever et al., 2014) to encode a variable-length source sentence into a fixed-length vector and to decode the vector into a variable-length target sentence.

最近,一些论文建议使用神经网络直接学习这种有条件的分布(Kalchbrenner和Blunsom,2013年报表;C在制订时,Cho等人,2014年a;Sutskever等人 不用了早已;Cho等人,2014年b;Forcada和Qéeco,1997年,例如,见). 这种神经 这是你的神经机翻译方法,一般由两个部分组成,第一个部分编码一个源句$\mathbf{x}$,第二个部分解码为目标句$\mathbf{y}$. 例如,两个反复出现的神经网络(RNN)被(Cho等人,2014年a)和(Sutskever等人 好啊,2014年)用于将可变长源句编码为固定长矢量,并将矢量解芮为可变长鳖目标句.

Despite being a quite new approach, neural machine translation has already shown promising results. Sutskever et al. (2014) reported that the neural machine translation based on RNNs with long short-term memory (LSTM) units achieves close to the state-of-the-art performance of the conventional phrase-based machine translation system on an English-to-French translation task.[^1] Adding neural components to existing translation systems, for instance, to score the phrase pairs in the phrase table (Cho et al., 2014a) or to re-rank candidate translations (Sutskever et al., 2014), has allowed to surpass the previous state-of-the-art performance level.

尽管是一种相当新的方法,但神经蟑螂的沿用已经显示出 喂,我 嗯,我... Sutskever等人(2014年) 马上报告说,基于RNNs的具有长期内存(LSTM)单元的神经机翻译在英语对法语翻译任务上实现了接近传统短语机翻译系统的最新性能. [^1] 将还为现存的翻译系统添加神经元,例如仍在进行的对等(Cho et al.,2014a)或重新排序候选翻译这项活动(Sutskever et al.,2014年),全面地超过了之前最先进的性能水平.

2.1 RNN Encoder–Decoder

2.1 RNN 编码器–解码器

Here, we describe briefly the underlying framework, called RNN Encoder–Decoder, proposed by Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns to align and translate simultaneously.

在这里,我们简要描述Cho等人(2014婿) 让他和标明Sutskever等人(2014年)提出的名为RNN 恩科代尔-Decoder的基本框架起着作用。

In the Encoder–Decoder framework, an encoder reads the input sentence, a sequence of vectors $\mathbf{x}=\left(x_{1},\cdots,x_{T_{x}}\right)$, into a vector $c$.[^2] The most common approach is to use an RNN such that

在编码器-解码器框架中，编码器将输入句子（向量序列$\mathbf{x}=\left(x_{1},\cdots,x_{T_{x}}\right)$）读取为向量$c$。[^2] 最常见的方法是使用 RNN，使得

$$\displaystyle h_{t}=f\left(x_{t},h_{t-1}\right) \tag{1}$$

and

中规定的她在

$$\displaystyle c=q\left(\left\{h_{1},\cdots,h_{T_{x}}\right\}\right),$$

where $h_{t}\in\mathbb{R}^{n}$ is a hidden state at time $t$, and $c$ is a vector generated from the sequence of the hidden states. $f$ and $q$ are some nonlinear functions. Sutskever et al. (2014) used an LSTM as $f$ and $q\left(\left\{h_{1},\cdots,h_{T}\right\}\right)=h_{T}$, for instance.

其中$h_{t}\in\mathbb{R}^{n}$是时间$t$的隐藏状态，$c$是从隐藏状态序列生成的向量。$f$和$q$是一些非线性函数。 苏茨克韦尔等人。 例如，（2014）使用 LSTM 作为$f$和$q\left(\left\{h_{1},\cdots,h_{T}\right\}\right)=h_{T}$。

The decoder is often trained to predict the next word $y_{t^{\prime}}$ given the context vector $c$ and all the previously predicted words $\left\{y_{1},\cdots,y_{t^{\prime}-1}\right\}$. In other words, the decoder defines a probability over the translation $\mathbf{y}$ by decomposing the joint probability into the ordered conditionals:

鉴于蓉语的上下文矢量$c$和以前所有预测的词$\left\{y_{1},\cdots,y_{t^{\prime}-1}\right\}$,解码器经常被训练来预测下一个词$y_{t^{\prime}}$. 换句话说 Spider,解码器通过将联合概率分解为命令的条件,老头子定义了翻译$\mathbf{y}$的概率:

$$\displaystyle p(\mathbf{y})=\prod_{t=1}^{T}p(y_{t}\mid\left\{y_{1},\cdots,y_{t-1}\right\},c), \tag{2}$$

where $\mathbf{y}=\left(y_{1},\cdots,y_{T_{y}}\right)$. With an RNN, each conditional probability is modeled as

· 第XX各种方案 用 RNN各种方案,每个条件所确认的概率都是

$$\displaystyle p(y_{t}\mid\left\{y_{1},\cdots,y_{t-1}\right\},c)=g(y_{t-1},s_{t},c), \tag{3}$$

where $g$ is a nonlinear, potentially multi-layered, function that outputs the probability of $y_{t}$, and $s_{t}$ is the hidden state of the RNN. It should be noted that other architectures such as a hybrid of an RNN and a de-convolutional neural network can be used (Kalchbrenner and Blunsom, 2013).

其中$g$是一个非线性的、潜在多层的函数，它输出$y_{t}$的概率，$s_{t}$ 是 RNN 的隐藏状态。 应该注意的是，可以使用其他架构，例如 RNN 和反卷积神经网络的混合（Kalchbrenner 和 Blunsom，2013）。

3 Learning to Align and Translate

3 学习对齐与翻译

In this section, we propose a novel architecture for neural machine translation. The new architecture consists of a bidirectional RNN as an encoder (Sec. 3.2) and a decoder that emulates searching through a source sentence during decoding a translation (Sec. 3.1).

在这个焰段里,我们提出神经机器翻译的新建筑. 新取得的进步包括一个双 老兄RN曙光作为编码器(Sec. 3.2)和一个解码器,在解码翻译(Sec. 3.1)时模拟通过源句进行搜索.

3.1 Decoder: General Description

3.1 解码器：一般描述

Figure 1: The graphical illustration of the proposed model trying to generate the $t$-th target word $y_{t}$ given a source sentence $(x_{1},x_{2},\dots,x_{T})$.

图 1：所提模型在给定源句 $(x_{1},x_{2},\dots,x_{T})$ 时生成第 $t$ 个目标词 $y_{t}$ 的示意。

In a new model architecture, we define each conditional probability in Eq. (2) as:

在新的模型架构中,我们在Eq. (2)中将每个条件概率定义为:

$$\displaystyle p(y_{i}|y_{1},\ldots,y_{i-1},\mathbf{x})=g(y_{i-1},s_{i},c_{i}), \tag{4}$$

where $s_{i}$ is an RNN hidden state for time $i$, computed by

仍在进行中 这样计算。

$$s_{i}=f(s_{i-1},y_{i-1},c_{i}).$$

It should be noted that unlike the existing encoder–decoder approach (see Eq. (2)), here the probability is conditioned on a distinct context vector $c_{i}$ for each target word $y_{i}$.

需要注意的是，与现有编码器–解码器方法（见式 (2)）不同，这里的概率以每个目标词 $y_i$ 各自不同的上下文向量 $c_i$ 为条件。

The context vector $c_{i}$ depends on a sequence of annotations $(h_{1},\cdots,h_{T_{x}})$ to which an encoder maps the input sentence. Each annotation $h_{i}$ contains information about the whole input sequence with a strong focus on the parts surrounding the $i$-th word of the input sequence. We explain in detail how the annotations are computed in the next section.

上下文向量 $c_i$ 依赖于编码器把输入句映射成的一串标注（annotation） $(h_1,\cdots,h_{T_x})$。每个标注 $h_i$ 含有关于整段输入序列的信息，并特别侧重输入序列第 $i$ 个词周围的部分。下一节详细说明标注如何计算。

The context vector $c_{i}$ is, then, computed as a weighted sum of these annotations $h_{i}$:

上下文向量 $c_i$ 随后由这些标注 $h_i$ 的加权和计算得到：

$$\displaystyle c_{i}=\sum_{j=1}^{T_{x}}\alpha_{ij}h_{j}. \tag{5}$$

The weight $\alpha_{ij}$ of each annotation $h_{j}$ is computed by

每个注释的$\alpha_{ij}$一直是按以下 收到的数据极力计算,但按 Nanaium计算enburgx全套注 没事的仍有等因素:

$$\displaystyle\alpha_{ij}=\frac{\exp\left(e_{ij}\right)}{\sum_{k=1}^{T_{x}}\exp\left(e_{ik}\right)}, \tag{6}$$

where

地点

$$e_{ij}=a(s_{i-1},h_{j})$$

is an alignment model which scores how well the inputs around position $j$ and the output at position $i$ match. The score is based on the RNN hidden state $s_{i-1}$ (just before emitting $y_{i}$, Eq. (4)) and the $j$-th annotation $h_{j}$ of the input sentence.

是一个对齐模型（alignment model），它给位置 $j$ 附近的输入与位置 $i$ 的输出匹配得有多好打分。分数基于 RNN 隐状态 $s_{i-1}$（在发出 $y_i$ 之前，式 (4)）以及输入句的第 $j$ 个标注 $h_j$。

We parametrize the alignment model $a$ as a feedforward neural network which is jointly trained with all the other components of the proposed system. Note that unlike in traditional machine translation, the alignment is not considered to be a latent variable. Instead, the alignment model directly computes a soft alignment, which allows the gradient of the cost function to be backpropagated through. This gradient can be used to train the alignment model as well as the whole translation model jointly.

我们把对齐模型 $a$ 参数化为一个前馈神经网络，并与所提系统的所有其他组件联合训练。注意，与传统机器翻译不同，这里的对齐不被当作隐变量。相反，对齐模型直接计算软对齐，从而使代价函数的梯度能够反向传播。

We can understand the approach of taking a weighted sum of all the annotations as computing an expected annotation, where the expectation is over possible alignments. Let $\alpha_{ij}$ be a probability that the target word $y_{i}$ is aligned to, or translated from, a source word $x_{j}$. Then, the $i$-th context vector $c_{i}$ is the expected annotation over all the annotations with probabilities $\alpha_{ij}$.

我们可以理解的是,将所有说明的加权总和作为计算“预期注释”的方法,如果期望大于可能的对齐。 让$\alpha_{ij}$س的宗旨是词$y_{i}$与源词XXMAT 100XX对齐或由源词$x_{j}$翻译的概率. 然后,$i$ - 第1次上下文矢量$c_{i}$是所有注释的预期注释,带有概率$\alpha_{ij}$。

The probability $\alpha_{ij}$, or its associated energy $e_{ij}$, reflects the importance of the annotation $h_{j}$ with respect to the previous hidden state $s_{i-1}$ in deciding the next state $s_{i}$ and generating $y_{i}$. Intuitively, this implements a mechanism of attention in the decoder. The decoder decides parts of the source sentence to pay attention to. By letting the decoder have an attention mechanism, we relieve the encoder from the burden of having to encode all information in the source sentence into a fixed-length vector. With this new approach the information can be spread throughout the sequence of annotations, which can be selectively retrieved by the decoder accordingly.

概率$\alpha_{ij}$臀部或与之相关的能量XXMAT假如与前一个隐藏状态$s_{i-1}$相比,说明$h_{j}$在决定下一个状态$s_{i}$和生成$y_{i}$方面的重要性.展示了. 直觉上,这在解码器中执行一种注意机制. 解码器决定部分来源句子 行了,注意.必不可少的是,通过让莹码器具有注意机制,我们免除了解码器必须把源句中的所有信息编码成固定长矢量的负担. 有了这一新的方法,信息就可以在说明的序列中传播,宵夜可以由解码器相应选择性地检索.

3.2 Encoder: Bidirectional RNN for Annotating Sequences

3.2 编码器：用于标注序列的双向 RNN

The usual RNN, described in Eq. (1), reads an input sequence $\mathbf{x}$ in order starting from the first symbol $x_{1}$ to the last one $x_{T_{x}}$. However, in the proposed scheme, we would like the annotation of each word to summarize not only the preceding words, but also the following words. Hence, we propose to use a bidirectional RNN (Schuster and Paliwal, 1997, BiRNN, ), which has been successfully used recently in speech recognition (Graves et al., 2013, see, e.g.,).

E在履行(1)中描述的常用RNN读取一个输入序列$\mathbf{x}$,以从第一个符号$x_{1}$开始到最后一个XX 我们正在读取一个输入序列$x_{T_{x}}$. 然而,在拟议的办法中,我们希望对每个每个字的注释不仅概括前面的几个字,而且概括下面的几个字. 272. 承认言论(Graves等人,2013年,见,例如励志)。

A BiRNN consists of forward and backward RNN’s. The forward RNN $\overrightarrow{f}$ reads the input sequence as it is ordered (from $x_{1}$ to $x_{T_{x}}$) and calculates a sequence of forward hidden states $(\overrightarrow{h}_{1},\cdots,\overrightarrow{h}_{T_{x}})$. The backward RNN $\overleftarrow{f}$ reads the sequence in the reverse order (from $x_{T_{x}}$ to $x_{1}$), resulting in a sequence of backward hidden states $(\overleftarrow{h}_{1},\cdots,\overleftarrow{h}_{T_{x}})$.

双向 RNN（BiRNN）由前向与后向 RNN 组成。前向 RNN $\overrightarrow{f}$ 按原顺序（从 $x_{1}$ 到 $x_{T_{x}}$）读取输入序列，并计算一串前向隐状态 $(\overrightarrow{h}_{1},\cdots,\overrightarrow{h}_{T_{x}})$。后向 RNN $\overleftarrow{f}$ 按相反顺序（从 $x_{T_{x}}$ 到 $x_{1}$）读取序列，得到一串后向隐状态 $(\overleftarrow{h}_{1},\cdots,\overleftarrow{h}_{T_{x}})$。

We obtain an annotation for each word $x_{j}$ by concatenating the forward hidden state $\overrightarrow{h}_{j}$ and the backward one $\overleftarrow{h}_{j}$, i.e., $h_{j}=\left[\overrightarrow{h}_{j}^{\top};\overleftarrow{h}_{j}^{\top}\right]^{\top}$. In this way, the annotation $h_{j}$ contains the summaries of both the preceding words and the following words. Due to the tendency of RNNs to better represent recent inputs, the annotation $h_{j}$ will be focused on the words around $x_{j}$. This sequence of annotations is used by the decoder and the alignment model later to compute the context vector (Eqs. (5)–(6)).

我们通过连接前向隐藏状态$\overrightarrow{h}_{j}$和后向隐藏状态$\overleftarrow{h}_{j}$获得每个单词$x_{j}$的注释，即$h_{j}=\left[\overrightarrow{h}_{j}^{\top};\overleftarrow{h}_{j}^{\top}\right]^{\top}$。 这样，注释$h_{j}$就包含了前面单词和后面单词的摘要。 由于 RNN 倾向于更好地表示最近的输入，因此注释$h_{j}$将集中在$x_{j}$周围的单词上。 解码器和对齐模型随后使用该注释序列来计算上下文向量（等式（5）–（6））。

See Fig. 1 for the graphical illustration of the proposed model.

示例载述提议的模型。

4 Experiment Settings

4 实验设置

We evaluate the proposed approach on the task of English-to-French translation. We use the bilingual, parallel corpora provided by ACL WMT ’14.[^3] As a comparison, we also report the performance of an RNN Encoder–Decoder which was proposed recently by Cho et al. (2014a). We use the same training procedures and the same dataset for both models.[^4]

我们评价关于英文至向前推进的笔译工作的拟议办法。 我们使用ACL WMT ' 14 ' 提供的双语平行公司。 [^3]拱肠编码器-Decoder是Cho等人最近提出的(2014年a)。 两种模型使用相同的培训程序和相同的数据集。 [^4]

Figure 2: The BLEU scores of the generated translations on the test set with respect to the lengths of the sentences. The results are on the full test set which includes sentences having unknown words to the models.

图 2：测试集上生成译文的 BLEU 随句子长度变化。结果基于完整测试集，包含含未知词的句子。

4.1 Dataset

4.1 数据集

WMT ’14 contains the following English-French parallel corpora: Europarl (61M words), news commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively, totaling 850M words. Following the procedure described in Cho et al. (2014a), we reduce the size of the combined corpus to have 348M words using the data selection method by Axelrod et al. (2011).[^5] We do not use any monolingual data other than the mentioned parallel corpora, although it may be possible to use a much larger monolingual corpus to pretrain an encoder. We concatenate news-test-2012 and news-test-2013 to make a development (validation) set, and evaluate the models on the test set (news-test-2014) from WMT ’14, which consists of 3003 sentences not present in the training data.

WMT ' 14包含以下英法平行的corpora:Europal(61M字抨击),新闻评论(5.5M),UN(421M)和两个爬行的corpora分别是90M和272.5M字座谈会, 无锡加入为提案国,共计850M字.例证. 按照Cho等人(2014年a 拜托)所述程序,我们使用Axelrod等人(2011年)的数据选择方法将合并体的体积缩小为348M字。 我们不使用任何单一语言 你认为我们可能使用更大的单一语言可从中预选编码器。俏皮不间断地在WMT14的测试(news-test-2014)上评价这些模型, 包括训练数据中没有的3003句。

After a usual tokenization[^6], we use a shortlist of 30,000 most frequent words in each language to train our models. Any word not included in the shortlist is mapped to a special token ($\left[\mbox{UNK}\right]$). We do not apply any other special preprocessing, such as lowercasing or stemming, to the data.

经过建议将[^ ―6] 通常的标语化后,我们使用每种语言中最常见的3万个غارم 赃词的短名单来训练我们的模型。 未包括在短名单中的任何字,都标注为特别符号($\left[\mbox{UNK}\right]$)。 我们对数据不建议将其他任何特殊预处理,如降低或终止。

(a)

(b)

(c)

(d)

Figure 3: Four sample alignments found by RNNsearch-50. The x-axis and y-axis of each plot correspond to the words in the source sentence (English) and the generated translation (French), respectively. Each pixel shows the weight $\alpha_{ij}$ of the annotation of the $j$-th source word for the $i$-th target word (see Eq. (6)), in grayscale ($0$: black, $1$: white). (a) an arbitrary sentence. (b–d) three randomly selected samples among the sentences without any unknown words and of length between 10 and 20 words from the test set.

(a) (b) (c) (d)

图 3：RNNsearch-50 找到的四个对齐样例。每个子图横轴为源句（英语）词，纵轴为生成译文（法语）词。每个像素按灰度显示第 $j$ 个源词标注对第 $i$ 个目标词的权重 $\alpha_{ij}$（见式 (6)）（0：黑，1：白）。

4.2 Models

4.2 模型

We train two types of models. The first one is an RNN Encoder–Decoder (Cho et al., 2014a, RNNencdec,), and the other is the proposed model, to which we refer as RNNsearch. We train each model twice: first with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then with the sentences of length up to 50 word (RNNencdec-50, RNNsearch-50).

我们训练两种模型 第一个是RNN编码顺应器-Decoder(Chogua等人,2014a,RNNencdec,),另一个是按照联合国所作的承诺提出的模型,我们称之为RN Rainbower.缇我们训练每个模式,先用长度的长度,长度为30(R(R) 没有(,R,是30),然后是长度的长度,然后将为50。

The encoder and decoder of the RNNencdec have 1000 hidden units each.[^7] The encoder of the RNNsearch consists of forward and backward recurrent neural networks (RNN) each having 1000 hidden units. Its decoder has 1000 hidden units. In both cases, we use a multilayer network with a single maxout (Goodfellow et al., 2013) hidden layer to compute the conditional probability of each target word (Pascanu et al., 2014).

R rồi,如同追究责任一样杜绝 。 RNN闱search的是按照前 你是谁? 编码器是袍形神经网络(RNN),每个神经网络都有1000个隐藏录像带。 它的解码器有1000个隐藏单元. 在这两种情况下,我们使用一个具有单一最大输出(Goodfell视情况等人,2013年)的多层网络来计算每个目标词的有条件概率(Pascanu等人,2014年)。

We use a minibatch stochastic gradient descent (SGD) algorithm together with Adadelta (Zeiler, 2012) to train each model. Each SGD update direction is computed using a minibatch of 80 sentences. We trained each model for approximately 5 days.

我们用一个小批量的梯度降以和平方式( 那我们的梯度降在落实( SGD)) 算法与 Adadelta (Zeiler, 2012) 一起来训练每个 现在的模型。 每个 SGD 更新方向都是用一个80 句子的小批体现出来. 我们训练每个模型大约5天。

Once a model is trained, we use a beam search to find a translation that approximately maximizes the conditional probability (Graves, 2012; Boulanger-Lewandowski et al., 2013, see, e.g.,). Sutskever et al. (2014) used this approach to generate translations from their neural machine translation model.

一旦一个模型经过了训练,我们就会使用光束搜索来找到一个大约能将条件概率最大化的翻译(Graves,2012年;Boulanger-Lewandowski等人,2013年,例如参见). Sutskever等人(2014年)使用这种方法从他们的神经机翻译模型中生成翻译.

For more details on the architectures of the models and training procedure used in the experiments, see Appendices A and B.

关于试验中使用的模型撰图架构和培训程序的详情,见附录A和B。

5 Results

5 结果

5.1 Quantitative Results

5.1 定量结果

Model	All	No UNK∘
RNNencdec-30	13.93	24.19
RNNsearch-30	21.50	31.44
RNNencdec-50	17.82	26.71
RNNsearch-50	26.75	34.16
RNNsearch-50⋆	28.45	36.15
Moses	33.30	35.63

Table 1: BLEU scores of the trained models computed on the test set. The second and third columns show respectively the scores on all the sentences and, on the sentences without any unknown word in themselves and in the reference translations. Note that RNNsearch-50⋆ was trained much longer until the performance on the development set stopped improving. ($\circ$) We disallowed the models to generate [UNK] tokens when only the sentences having no unknown words were evaluated (last column).

怎么了?	全体	很常见...
应继续采取下列措施:	13.93	24.19
搜索 不用了 30号	21.50	31.44
】 大家都说,	17.82	26.71
搜索 - 50	26.75	34.16
1860 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的啪 的 的 的 103. 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 67. 的 的 朔 的 的 不扩散条约 的 的 的 的 的 的 的 的 的 的 的 的 的 的 将性别观点纳入 的 grandparents 的 的 的 的 的 的 的的所作所为 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 的 有若干 变态 的 的这几个 的 的	28.45	36.15
除其它外	33.30	35.63

表1:试验组计算分数。 第二栏和第三栏分别显示 这位词的所有句子的分数,以及这些句子 你必须在俨然无名的词本身和参考翻译中。 汇合金 汇合金 汇合金): ($\circ$ )我们拒绝这些模型来生成[UNK]符号,当时只评价了没有未知词的句子(最后一栏).

In Table 1, we list the translation performances measured in BLEU score. It is clear from the table that in all the cases, the proposed RNNsearch outperforms the conventional RNNencdec. More importantly, the performance of the RNNsearch is as high as that of the conventional phrase-based translation system (Moses), when only the sentences consisting of known words are considered. This is a significant achievement, considering that Moses uses a separate monolingual corpus (418M words) in addition to the parallel corpora we used to train the RNNsearch and RNNencdec.

表 1 列出以 BLEU 衡量的翻译表现。在所有情形下，所提 RNNsearch 都优于常规 RNNencdec。更重要的是，RNNsearch 的表现已经能与常规基于短语的翻译系统（Moses）相当，尤其是在只考虑不含未知词的句子时。考虑到 Moses 使用了我们训练神经模型时未用的庞大单语语料，这是一项重要进展。

One of the motivations behind the proposed approach was the use of a fixed-length context vector in the basic encoder–decoder approach. We conjectured that this limitation may make the basic encoder–decoder approach to underperform with long sentences. In Fig. 2, we see that the performance of RNNencdec dramatically drops as the length of the sentences increases. On the other hand, both RNNsearch-30 and RNNsearch-50 are more robust to the length of the sentences. RNNsearch-50, especially, shows no performance deterioration even with sentences of length 50 or more. This superiority of the proposed model over the basic encoder–decoder is further confirmed by the fact that the RNNsearch-30 even outperforms RNNencdec-50 (see Table 1).

拟议方法背后的动机之一是在基本编码器-解码器方法中使用固定长度上下文矢量。 我们猜想,这种限制鸳鸯可能使基本编码器-解码器方法在长句子的情况下表现在履行不佳. 在图2中,我们看到RNNencdec的性能随着刑期的延长而急剧下降. 另一方面,RNNsearch-30和RNNsearch-50都比句子长度更强壮. 特别是RNNsearch-50,即使长度超过50的句子,也没有显示性能恶化. 拟议的模型优于基本编码器-相互合作器进一步证实了这一点,即RNNsearch-30甚至优于RN 查理-50(见表1)。

5.2 Qualitative Analysis

5.2 定性分析

5.2.1 Alignment

The proposed approach provides an intuitive way to inspect the (soft-)alignment between the words in a generated translation and those in a source sentence. This is done by visualizing the annotation weights $\alpha_{ij}$ from Eq. (6), as in Fig. 3. Each row of a matrix in each plot indicates the weights associated with the annotations. From this we see which positions in the source sentence were considered more important when generating the target word.

5.2.1 对齐

提议的办法提供了一种直观的方法,用以检查在生成的翻译中用词与源句中用词之间的(软)对齐。 这样做的方法是将注释权重( 妈”($\alpha_{ij}$)从Eq. (6)直观化,如Fig. 3. 每一图中的每婶列表示 那是与说明有关的加权数。 我们从能够得到的信息中可以看出,在生成目标词时,源句中哪些的学生被认为更为重要。

We can see from the alignments in Fig. 3 that the alignment of words between English and French is largely monotonic. We see strong weights along the diagonal of each matrix. However, we also observe a number of non-trivial, non-monotonic alignments. Adjectives and nouns are typically ordered differently between French and English, and we see an example in Fig. 3 (a). From this figure, we see that the model correctly translates a phrase [European Economic Area] into [zone économique européen]. The RNNsearch was able to correctly align [zone] with [Area], jumping over the two words ([European] and [Economic]), and then looked one word back at a time to complete the whole phrase [zone économique européenne].

从图3中的对齐中可以看出,英语和法语之间的单词对基本是单调的. 我们看到每个基质 60. 然而,我们也观察到一些非三角,非rylonic的对齐. 形容词和名词通常在法语和英语之间顺序不同,我们在图3(a)中看到一个例子。 从这个图中,我们看到模型将一个短语[欧洲经济区]正确翻译为[zone économique Européen]. R暨NNsearch能够正确地与 即使[地区]对齐,跳过两个单词([欧]和[经济]),然后 但如果一个单词回首,就可以完成整个短语[zone économique Européenne].

The strength of the soft-alignment, opposed to a hard-alignment, is evident, for instance, from Fig. 3 (d). Consider the source phrase [the man] which was translated into [l’ homme]. Any hard alignment will map [the] to [l’] and [man] to [homme]. This is not helpful for translation, as one must consider the word following [the] to determine whether it should be translated into [le], [la], [les] or [l’]. Our soft-alignment solves this issue naturally by letting the model look at both [the] and [man], and in this example, we see that the model was able to correctly translate [the] into [l’]. We observe similar behaviors in all the presented cases in Fig. 3. An additional benefit of the soft alignment is that it naturally deals with source and target phrases of different lengths, without requiring a counter-intuitive way of mapping some words to or from nowhere ([NULL]) (Koehn, 2010, see, e.g., Chapters 4 and 5 of).

从图起到了明显的作用,例如,从图3(d)可以看出,软调的强度是相对硬调的。 试想一下被译作`我的人官方统计'的源词。 畸社会性别主流化会的指控, 这对翻译没有帮助,因为人们必须考虑后面的词,以确定应将其译为[le]、[la]、[les]或[l ' 。 我们的软调整自然地解决了这个问题,让模型同时看[人]和[人],在这个例子中,我们看到模型能够将[人]正确翻译为精灵。 在菲格的所有案子中,我们观察到类似 我们还看到类似的行为 3. 软调的另一个好处是,它自然涉及不同长度的源 24. 和目标 短语,而不需要反直观的方式将一些单词映射到或从无处([NULL羚])(Koehn,2010年,见,例如第4章和第5章).

5.2.2 Long Sentences

As clearly visible from Fig. 2 the proposed model (RNNsearch) is much better than the conventional model (RNNencdec) at translating long sentences. This is likely due to the fact that the RNNsearch does not require encoding a long sentence into a fixed-length vector perfectly, but only accurately encoding the parts of the input sentence that surround a particular word.

的家庭

从图2可明显看出,拟议模型( 你为什么不研究)在翻译长句时比常规模型(RN Nencdec的统计数据)要好包括对长句的翻译。 这可能是由于RNNsearch不需要将长句编码蓓丽到一个固定长的矢量上完美无缺,而只是精确地编码了输入句中围绕特定单词的部分.

As an example, consider this source sentence from the test set:

例如,从测试集中考虑这一源句:胥媲美:

An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital.

The RNNencdec-50 translated this sentence into:

RNNenencdec-50将这句话译为:

Un privilège d’admission est le droit d’un médecin de reconnaître un patient à l’hôpital ou un centre médical d’un diagnostic ou de prendre un diagnostic en fonction de son état de santé.

The RNNencdec-50 correctly translated the source sentence until [a medical center]. However, from there on (underlined), it deviated from the original meaning of the source sentence. For instance, it replaced [based on his status as a health care worker at a hospital] in the source sentence with [en fonction de son état de santé] (“based on his state of health”).

RNNencdec-50责任制将源句渍正确翻译为[嘱医疗中心]. 然而,从ỉ从那里(下划线),它步伐偏离了源句的原始含义. 例如,在 17. 源句中,[根据他在医院的保健工作者状况] 改为[根据他 và其健康状况]。

On the other hand, the RNNsearch-50 generated the following correct translation, preserving the whole meaning of the input sentence without omitting any details:

另一方面,RNNsearch-50产生了以下正确的提出的各项建议翻译,是谁保留了输入句的全部含义,而没有省略任何细节ẽ: .

Un privilège d’admission est le droit d’un médecin d’admettre un patient à un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, selon son statut de travailleur des soins de santé à l’hôpital.

Let us consider another sentence from the test set:

我们再考虑一下试验组中的 波斯语

This kind of experience is part of Disney’s efforts to ”extend the lifetime of its series and build new relationships with audiences via digital platforms that are becoming ever more important,” he added.

The translation by the RNNencdec-50 is

RNNencdec-50的翻译是

Ce type d’expérience fait partie des initiatives du Disney pour ”prolonger la durée de vie de ses nouvelles et de développer des liens avec les lecteurs numériques qui deviennent plus complexes.

As with the previous example, the RNNencdec began deviating from the actual meaning of the source sentence after generating approximately 30 words (see the underlined phrase). After that point, the quality of the translation deteriorates, with basic mistakes such as the lack of a closing quotation mark.

与均属前例的RNN上午,RNenencdec在生成憧憬约30个字(见下划线短语)后,开始偏离源句的实际含义. 之后,翻译质量恶化,缺少收尾引号等基本错误.

Again, the RNNsearch-50 was able to translate this long sentence correctly:

同样,RNNsearch-50能够正确翻译这句长句:

Ce genre d’expérience fait partie des efforts de Disney pour ”prolonger la durée de vie de ses séries et créer de nouvelles relations avec des publics via des plateformes numériques de plus en plus importantes”, a-t-il ajouté.

In conjunction with the quantitative results presented already, these qualitative observations confirm our hypotheses that the RNNsearch architecture enables far more reliable translation of long sentences than the standard RNNencdec model.

结合已经提出的定量结果,这些定性观测证实了我们的假设,即RNN积极地研究架构 Adriannéch 架构能够比标准RNNencdec模型更可靠地翻译长句.

In Appendix C, we provide a few more sample translations of long source sentences generated by the RNNencdec-50, RNNsearch-50 and Google Translate along with the reference translations.

在附录C中,我们提供了由RNNencdec-50,RN准时搜索-50和Google Translate以及参考翻译生成的长源句子的几个更多样本翻译.

6 Related Work

6 相关工作

6.1 Learning to Align

6.1 学习对齐

A similar approach of aligning an output symbol with an input symbol was proposed recently by Graves (2013) in the context of handwriting synthesis. Handwriting synthesis is a task where the model is asked to generate handwriting of a given sequence of characters. In his work, he used a mixture of Gaussian kernels to compute the weights of the annotations, where the location, width and mixture coefficient of each kernel was predicted from an alignment model. More specifically, his alignment was restricted to predict the location such that the location increases monotonically.

Graves（2013）在手写合成语境中最近提出过类似做法：把输出符号与输入符号对齐。手写合成要求模型生成给定字符序列的笔迹。他用一组高斯核计算标注权重，每个核的位置、宽度与混合系数由对齐模型预测。与我们的方法不同，其对齐路径大致是单调的，因此更适合手写这类输出与输入顺序高度一致的任务。

The main difference from our approach is that, in (Graves, 2013), the modes of the weights of the annotations only move in one direction. In the context of machine translation, this is a severe limitation, as (long-distance) reordering is often needed to generate a grammatically correct translation (for instance, English-to-German).

与我们不过是方法的主要区别在于,在(Graves,2013年)中,注释的 Jonasters的闺重 让我们只朝一个方向移动.僧侣礼仪礼仪礼仪礼仪礼仪榈礼仪礼仪礼仪礼仪礼仪礼仪 临时议程项目礼仪礼向妇女提供礼仪礼仪礼仪礼仪礼从一开始就是礼仪礼仪礼仪礼仪礼仪礼仪礼仪礼仪礼仪礼仪礼仪 如果他们礼仪礼仪礼仪礼我没有礼仪礼仪礼仪礼仪礼仪礼仪礼鳌礼仪所审议的礼仪礼仪礼仪礼仪礼仪礼仪礼仪礼仪礼仪礼仪礼提交其礼仪礼仪礼仪礼仪礼wang.

Our approach, on the other hand, requires computing the annotation weight of every word in the source sentence for each word in the translation. This drawback is not severe with the task of translation in which most of input and output sentences are only 15–40 words. However, this may limit the applicability of the proposed scheme to other tasks.

另一方面，我们的方法需要为译文中的每个词计算源句中每个词的对齐权重。对翻译任务来说这不算严重问题，因为多数输入和输出句子只有 15–40 个词。不过，这可能会限制该方法在其他任务上的适用性。

6.2 Neural Networks for Machine Translation

6.2 用于机器翻译的神经网络

Since Bengio et al. (2003) introduced a neural probabilistic language model which uses a neural network to model the conditional probability of a word given a fixed number of the preceding words, neural networks have widely been used in machine translation. However, the role of neural networks has been largely limited to simply providing a single feature to an existing statistical machine translation system or to re-rank a list of candidate translations provided by an existing system.

自Bengio等人(2003年)推出神经概率语言模型,使用神经网络对一个给定前几字数的词的有条件概率进行建模,神经网络在机器翻译中被广泛使用. 然而,鱿鱼网络的作用基本上仅限于为现有的统计机器翻译系统提供一个单一的功能,或对现有系统提供的候选翻译并对其清单进行重新排序.

For instance, Schwenk (2012) proposed using a feedforward neural network to compute the score of a pair of source and target phrases and to use the score as an additional feature in the phrase-based statistical machine translation system. More recently, Kalchbrenner and Blunsom (2013) and Devlin et al. (2014) reported the successful use of the neural networks as a sub-component of the existing translation system. Traditionally, a neural network trained as a target-side language model has been used to rescore or rerank a list of candidate translations (Schwenk et al., 2006, see, e.g.,).

例如,Schwenk(2012年)提议使用一个feedforward神经网络来计算一对源和目标短语的得分,并在基于短语的统计机器翻译系统中将得分作为附加特征. 最近,Kalchbrenner和Bl 太棒了(2013年)和D至少在2014年)报告说,神经网络作为现有翻译系统的一个子组成部分得到了成功的利用。 传统上,一个作为目标边语言模式而训练的神经网络,被用来重新计分或重新排序候选翻译名单(Schwenk等人,2006年,见,例如.

Although the above approaches were shown to improve the translation performance over the state-of-the-art machine translation systems, we are more interested in a more ambitious objective of designing a completely new translation system based on neural networks. The neural machine translation approach we consider in this paper is therefore a radical departure from these earlier works. Rather than using a neural network as a part of the existing system, our model works on its own and generates a translation from a source sentence directly.

虽然上述方法显示,茁壮成长于比最先进的黏性机器翻译系统更好的翻译性能,但我们更感兴趣的是,更雄心勃勃的目标是 Valerie基于神经网络设计全新的翻译系统. 因此,我们在本文中认为的神经机翻译方法,与这些早期的作品是根本的背离.咚 我们的模型与其使用神经imal net 万一是现有系统的一部分,不如自行工作,直接生成源句的鬣译.

7 Conclusion

7 结论

The conventional approach to neural machine translation, called an encoder–decoder approach, encodes a whole input sentence into a fixed-length vector from which a translation will be decoded. We conjectured that the use of a fixed-length context vector is problematic for translating long sentences, based on a recent empirical study reported by Cho et al. (2014b) and Pouget-Abadie et al. (2014).

常规的神经机翻译方法,称为编码器–解码器方法,将整个输入和平和安全的句子编码成固定长的矢量,从中解码翻译. 根据Cho等人(2014年b)和Pouget-Abadie等人(2014年)最近报告的一项经验研究,我们推测,使用固定长的上下文向等途径对 小句的翻译有问题。嘲讽

In this paper, we proposed a novel architecture that addresses this issue. We extended the basic encoder–decoder by letting a model (soft-)search for a set of input words, or their annotations computed by an encoder, when generating each target word. This frees the model from having to encode a whole source sentence into a fixed-length vector, and also lets the model focus only on information relevant to the generation of the next target word. This has a major positive impact on the ability of the neural machine translation system to yield good results on longer sentences. Unlike with the traditional machine translation systems, all of the pieces of the translation system, including the alignment mechanism, are jointly trained towards a better log-probability of producing correct translations.

在本文中,我们提出了一个解决这个问题的新结构。 我们派系楠格 :充实了基本编码器–解码器,让对这类输入词的模型(soft-)搜索,或在生成每个目标词时用编码器计算出它们的注释。 这让模型不必将整个源句编码成固定长度的矢量,也让模型只关注与下一个目标词生成有关的信息. 这对神经机翻译系统在较长的句子上产生良好效果的能力有着重大的积极影响. 与传统的机器翻译系统不同 你需要共同培训翻译系统的所有部分,包括校正机制, 他们是为更好的日志可能性来制作正确的翻译。

We tested the proposed model, called RNNsearch, on the task of English-to-French translation. The experiment revealed that the proposed RNNsearch outperforms the conventional encoder–decoder model (RNNencdec) significantly, regardless of the sentence length and that it is much more robust to the length of a source sentence. From the qualitative analysis where we investigated the (soft-)alignment generated by the RNNsearch, we were able to conclude that the model can correctly align each target word with the relevant words, or their annotations, in the source sentence as it generated a correct translation.

建设性地说,我们试验了拟议的模型,配套的坚定地致力于英文和法文的翻译认真地翻译。 实验显示,无论句子长度如何,拟议的RNNSearch都大大超过了常规编码器–解码器模型(RNNCdec),而且对源句长度而言,其强度要大得多. 根据我们调查RNNSearch医治出的(软的)对齐的定性分析,我们得出了这样的的一个基本结论,即该模型能够正确地将每个目标词与源句中的相关词,或他们的注解对齐为筹备姻缘而产生正确的翻译.

Perhaps more importantly, the proposed approach achieved a translation performance comparable to the existing phrase-based statistical machine translation. It is a striking result, considering that the proposed architecture, or the whole family of neural machine translation, has only been proposed as recently as this year. We believe the architecture proposed here is a promising step toward better machine translation and a better understanding of natural languages in general.

也许更重要的是,拟议方法实现了与现有的基于陀语的统计机谍译相类似的翻译性能。 这是一个惊人的结果,考虑到拟议的建筑,或者整个神经机翻译家族,都是与人权今年一样的最近提出的. 我们认为,这里提出的结构是朝着改进机器翻译和更好地了解一般自然语言缔结一项很有提供了支助。

One of challenges left for the future is to better handle unknown, or rare words. This will be required for the model to be more widely used and to match the performance of current state-of-the-art machine translation systems in all contexts.

劳工市场未来的挑战之一是更好地处理未知或罕见的景象。 这需要噬凄凉需要模型得到更广泛的绰号, 我们将匹配当前所有情况下最先进的机器翻译系统的性能.

Acknowledgments

致谢

The authors would like to thank the developers of Theano (Bergstra et al., 2010; Bastien et al., 2012). We acknowledge the support of the following agencies for research funding and computing support: NSERC, Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR. Bahdanau thanks the support from Planet Intelligent Systems GmbH. We also thank Felix Hill, Bart van Merriénboer, Jean Pouget-Abadie, Coline Devin and Tae-Ho Kim.

作者们要感谢The樊o的开发者(B 女士们 B柠ngstra等人,2010年; Bastien等人,2012年)。 但我们感谢以下机构对研究资金和计算支持的支持提供的支持:国家科学研究中心、卡尔库尔魁北克省、加拿大计算中心、加拿大研究会长和CIFAR。阐的的的的 的的 的的的 的的的 的的 的的 的 的 的 的 的 的 的 的的 颜 的专门为 , 的 的的的 50 的 的的的 的 的的 的之间的平衡 的的 的 49 的 走吧 的 的 的彬 的 让我看看 的 的 的 蟑 的 的 厮的 的 的 的 的 将能 的的 我们还感谢费利克斯·希尔、巴特·范梅里安博尔及早、让才能实现阿巴迪、科琳·德文和金泰浩。

References

参考文献

Axelrod et al. (2011) Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 355–362. Association for Computational Linguistics.

Bastien et al. (2012) Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.

Bengio et al. (1994) Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5 (2), 157–166.

Bengio et al. (2003) Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model. J. Mach. Learn. Res. , 3 , 1137–1155.

Bergstra et al. (2010) Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy) . Oral Presentation.

Boulanger-Lewandowski et al. (2013) Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2013). Audio chord recognition with recurrent neural networks. In ISMIR .

Cho et al. (2014a) Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014a). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) . to appear.

Cho et al. (2014b) Cho, K., van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014b). On the properties of neural machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation . to appear.

Devlin et al. (2014) Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast and robust neural network joint models for statistical machine translation. In Association for Computational Linguistics .

Forcada and Ñeco (1997) Forcada, M. L. and Ñeco, R. P. (1997). Recursive hetero-associative memories for translation. In J. Mira, R. Moreno-Díaz, and J. Cabestany, editors, Biological and Artificial Computation: From Neuroscience to Technology , volume 1240 of Lecture Notes in Computer Science , pages 453–462. Springer Berlin Heidelberg.

Goodfellow et al. (2013) Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013). Maxout networks. In Proceedings of The 30th International Conference on Machine Learning , pages 1319–1327.

Graves (2012) Graves, A. (2012). Sequence transduction with recurrent neural networks. In Proceedings of the 29th International Conference on Machine Learning (ICML 2012) .

Graves (2013) Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv: 1308.0850 [cs.NE] .

Graves et al. (2013) Graves, A., Jaitly, N., and Mohamed, A.-R. (2013). Hybrid speech recognition with deep bidirectional LSTM. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on , pages 273–278.

Hermann and Blunsom (2014) Hermann, K. and Blunsom, P. (2014). Multilingual distributed representations without word alignment. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014) .

Hochreiter (1991) Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München.

Hochreiter and Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation , 9 (8), 1735–1780.

Kalchbrenner and Blunsom (2013) Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1700–1709. Association for Computational Linguistics.

Koehn (2010) Koehn, P. (2010). Statistical Machine Translation . Cambridge University Press, New York, NY, USA.

Koehn et al. (2003) Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1 , NAACL ’03, pages 48–54, Stroudsburg, PA, USA. Association for Computational Linguistics.

Pascanu et al. (2013a) Pascanu, R., Mikolov, T., and Bengio, Y. (2013a). On the difficulty of training recurrent neural networks. In ICML’2013 .

Pascanu et al. (2013b) Pascanu, R., Mikolov, T., and Bengio, Y. (2013b). On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013) .

Pascanu et al. (2014) Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y. (2014). How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014) .

Pouget-Abadie et al. (2014) Pouget-Abadie, J., Bahdanau, D., van Merriënboer, B., Cho, K., and Bengio, Y. (2014). Overcoming the curse of sentence length for neural machine translation using automatic segmentation. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation . to appear.

Schuster and Paliwal (1997) Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on , 45 (11), 2673–2681.

Schwenk (2012) Schwenk, H. (2012). Continuous space translation models for phrase-based statistical machine translation. In M. Kay and C. Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN) , pages 1071–1080. Indian Institute of Technology Bombay.

Schwenk et al. (2006) Schwenk, H., Dchelotte, D., and Gauvain, J.-L. (2006). Continuous space language models for statistical machine translation. In Proceedings of the COLING/ACL on Main conference poster sessions , pages 723–730. Association for Computational Linguistics.

Sutskever et al. (2014) Sutskever, I., Vinyals, O., and Le, Q. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS 2014) .

Zeiler (2012) Zeiler, M. D. (2012). ADADELTA: An adaptive learning rate method. arXiv: 1212.5701 [cs.LG] .

Appendix A Model Architecture

附录 A 模型架构

A.1 Architectural Choices

A.1 架构选择

The proposed scheme in Section 3 is a general framework where one can freely define, for instance, the activation functions $f$ of recurrent neural networks (RNN) and the alignment model $a$. Here, we describe the choices we made for the experiments in this paper.

第3节中的拟议方案是一个一般框架,人们可以自由地界定经常神经网络(RNN)的激活功能$f$和对齐模式$a$。 . . . . . . . . . . . . . .豺. . . .. . 别这样, . . poultry. . . . . . . . . . . .. . . . . .. . . .. . . . . .. . . .. . . . . . . . . . ... . . .. 55.

A.1.1 Recurrent Neural Network

For the activation function $f$ of an RNN, we use the gated hidden unit recently proposed by Cho et al. (2014a). The gated hidden unit is an alternative to the conventional simple units such as an element-wise $\tanh$. This gated unit is similar to a long short-term memory (LSTM) unit proposed earlier by Hochreiter and Schmidhuber (1997), sharing with it the ability to better model and learn long-term dependencies. This is made possible by having computation paths in the unfolded RNN for which the product of derivatives is close to 1. These paths allow gradients to flow backward easily without suffering too much from the vanishing effect (Hochreiter, 1991; Bengio et al., 1994; Pascanu et al., 2013a). It is therefore possible to use LSTM units instead of the gated hidden unit described here, as was done in a similar context by Sutskever et al. (2014).

A.1.1 经常神经网络

对于一个RNN的媳妇的$f$的激活功能,我们并不构成使用Cho等人最近提出的门质隐藏单元(2014a). 门式隐藏单元是常规的simple单元的替代物,如元素感妍的$\tanh$. 这个闸机单元 这里类似于霍赫赖特和施密德胡伯(1997年)早些时候提出的一个长期短期内存(L拱机)单元,与它分享更好的模型和学习长期依赖的能力. 通过在已展开的RNN中设置计算路径,使衍生物的产物接近1. 这些路径使梯度容易向后流动,而不会因消失效应而遭受太多痛苦(Hochre 是不是er,1991年;Bengio等人,1994年;Pascanu等人,2013年a)。 因此,可以使用L可在Sutskever等人(2014年)的类似背景下使用的LSTM单元,而不是本文所述的门式隐藏单元。

The new state $s_{i}$ of the RNN employing $n$ gated hidden units[^8] is computed by

使用$n$门 Castled 隐藏单元的 ONN 新状态$s_{i}$[^8] 由

$$\displaystyle s_{i}=f(s_{i-1},y_{i-1},c_{i})=(1-z_{i})\circ s_{i-1}+z_{i}\circ\tilde{s}_{i},$$

where $\circ$ is an element-wise multiplication, and $z_{i}$ is the output of the update gates (see below). The proposed updated state $\tilde{s}_{i}$ is computed by

问题时,XX并按照元素法 别用$\circ$,$z_{i}$是更新门的输出(见下文). 拟议的更新状态$\tilde{s}_{i}$由下列方法计算:

$$\displaystyle\tilde{s}_{i}=\tanh\left(We(y_{i-1})+U\left[r_{i}\circ s_{i-1}\right]+Cc_{i}\right),$$

where $e(y_{i-1})\in\mathbb{R}^{m}$ is an $m$-dimensional embedding of a word $y_{i-1}$, and $r_{i}$ is the output of the reset gates (see below). When $y_{i}$ is represented as a $1$-of-$K$ vector, $e(y_{i})$ is simply a column of an embedding matrix $E\in\mathbb{R}^{m\times K}$. Whenever possible, we omit bias terms to make the equations less cluttered.

其中$e(y_{i-1})\in\mathbb{R}^{m}$是字$y_{i-1}$的$m$维嵌入，$r_{i}$ 是重置门的输出（见下文）。 当$y_{i}$表示为$1$-of-$K$向量时，$e(y_{i})$只是嵌入矩阵$E\in\mathbb{R}^{m\times K}$的一列。 只要有可能，我们都会省略偏置项，以使方程不那么混乱。

The update gates $z_{i}$ allow each hidden unit to maintain its previous activation, and the reset gates $r_{i}$ control how much and what information from the previous state should be reset. We compute them by

更新门$z_{i}$允许每个隐藏单元保持其先前的激活状态，而重置门$r_{i}$控制应重置先前状态的多少信息以及哪些信息。 我们通过以下方式计算它们

$$\displaystyle z_{i} \\ \displaystyle=\sigma\left(W_{z}e(y_{i-1})+U_{z}s_{i-1}+C_{z}c_{i}\right), \\ \displaystyle r_{i} \\ \displaystyle=\sigma\left(W_{r}e(y_{i-1})+U_{r}s_{i-1}+C_{r}c_{i}\right),$$

where $\sigma\left(\cdot\right)$ is a logistic sigmoid function.

其中$\sigma\left(\cdot\right)$是一个后勤Sigmoid函数。

At each step of the decoder, we compute the output probability (Eq. (4)) as a multi-layered function (Pascanu et al., 2014). We use a single hidden layer of maxout units (Goodfellow et al., 2013) and normalize the output probabilities (one for each word) with a softmax function (see Eq. (6)).

在解码器的每个步骤,我们计算输出概率(Eq (4))为多层函数(Pasc的要求,2014年)。 我们使用单层隐藏的最大卓有成效的单位(Goodfellow等人,2013年),并将输出概率(每个采取必要行动语言各一个) IIality,并具有软max函数(见Eq.(6)).

A.1.2 Alignment Model

The alignment model should be designed considering that the model needs to be evaluated $T_{x}\times T_{y}$ times for each sentence pair of lengths $T_{x}$ and $T_{y}$. In order to reduce computation, we use a single-layer multilayer perceptron such that

A.1.2 对齐模式

在设计配对模型时,应考虑到每句长度$T_{x}$和$T_{y}$需要评价$T_{x}\times T_{y}$次. 为了减少计算,我们使用单层 我会用多层感应器

$$\displaystyle a(s_{i-1},h_{j})=v_{a}^{\top}\tanh\left(W_{a}s_{i-1}+U_{a}h_{j}\right),$$

where $W_{a}\in\mathbb{R}^{n\times n},U_{a}\in\mathbb{R}^{n\times 2n}$ and $v_{a}\in\mathbb{R}^{n}$ are the weight matrices. Since $U_{a}h_{j}$ does not depend on $i$, we can pre-compute it in advance to minimize the computational cost.

其中 $W_{a}\in\mathbb{R}^{n\times n},U_{a}\in\mathbb{R}^{n\times 2n}$ 与 $v_{a}\in\mathbb{R}^{n}$ 是权重矩阵。由于 $U_{a}h_{j}$ 不依赖于 $i$，可以预先计算以降低计算量。

A.2 Detailed Description of the Model

A.2 模型的详细描述

A.2.1 Encoder

In this section, we describe in detail the architecture of the proposed model (RNNsearch) used in the experiments (see Sec. 4–5). From here on, we omit all bias terms in order to increase readability.

A.2.1 编码器

本节详细描述实验所用所提模型（RNNsearch）的架构（见第 4–5 节）。为提高可读性，下文省略所有偏置项。

The model takes a source sentence of 1-of-K coded word vectors as input

该 你把 1 但如果 K 79. 编码字向你怎么输入的源句

$$\mathbf{x}=(x_{1},\ldots,x_{T_{x}}),\mbox{ }x_{i}\in\mathbb{R}^{K_{x}}$$

and outputs a translated sentence of 1-of-K coded word vectors

和输出一个译以一种 K 编码的单词向量的展开了 通过

$$\mathbf{y}=(y_{1},\ldots,y_{T_{y}}),\mbox{ }y_{i}\in\mathbb{R}^{K_{y}},$$

where $K_{x}$ and $K_{y}$ are the vocabulary sizes of source and target languages, respectively. $T_{x}$ and $T_{y}$ respectively denote the lengths of source and target sentences.

提倡使用源语言和目标语言的词汇大小。 HIV2XX和$T_{y}$分别表示源句和目标句的长度.

First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed:

首先,双向反复神经网络(BIRNN)的前进 所以我们要计算出 Jonass unit.

$$\displaystyle\overrightarrow{h}{i}= \ \displaystyle\begin{cases}(1-\overrightarrow{z}{i})\circ\overrightarrow{h}{i-1}+\overrightarrow{z}{i}\circ\overrightarrow{\underline{h}}_{i}&\mbox{, if }i>0\
0&\mbox{, if }i=0\end{cases}$$

where

地点

$$\displaystyle\overrightarrow{\underline{h}}_{i}= \\ \displaystyle\tanh\left(\overrightarrow{W}\overline{E}x_{i}+\overrightarrow{U}\left[\overrightarrow{r}_{i}\circ\overrightarrow{h}_{i-1}\right]\right) \\ \displaystyle\overrightarrow{z}_{i}= \\ \displaystyle\sigma\left(\overrightarrow{W}_{z}\overline{E}x_{i}+\overrightarrow{U}_{z}\overrightarrow{h}_{i-1}\right) \\ \displaystyle\overrightarrow{r}_{i}= \\ \displaystyle\sigma\left(\overrightarrow{W}_{r}\overline{E}x_{i}+\overrightarrow{U}_{r}\overrightarrow{h}_{i-1}\right).$$

$\overline{E}\in\mathbb{R}^{m\times K_{x}}$ is the word embedding matrix. $\overrightarrow{W},\overrightarrow{W}_{z},\overrightarrow{W}_{r}\in\mathbb{R}^{n\times m}$, $\overrightarrow{U},\overrightarrow{U}_{z},\overrightarrow{U}_{r}\in\mathbb{R}^{n\times n}$ are weight matrices. $m$ and $n$ are the word embedding dimensionality and the number of hidden units, respectively. $\sigma(\cdot)$ is as usual a logistic sigmoid function.

$\overline{E}\in\mathbb{R}^{m\times K_{x}}$ 是词嵌入矩阵。$\overrightarrow{W},\overrightarrow{W}_{z},\overrightarrow{W}_{r}\in\mathbb{R}^{n\times m}$，$\overrightarrow{U},\overrightarrow{U}_{z},\overrightarrow{U}_{r}\in\mathbb{R}^{n\times n}$ 是权重矩阵。$m$ 与 $n$ 分别是词嵌入维数与隐单元数。$\sigma(\cdot)$ 仍是逐元素的逻辑斯蒂 sigmoid 函数。

The backward states $(\overleftarrow{h}_{1},\cdots,\overleftarrow{h}_{T_{x}})$ are computed similarly. We share the word embedding matrix $\overline{E}$ between the forward and backward RNNs, unlike the weight matrices.

落后国家$(\overleftarrow{h}_{1},\cdots,\overleftarrow{h}_{T_{x}})$的计算方式类似。 我们分享前向和后皓慧 女士们,我们分享了$\overline{E}$的嵌入式矩阵, 不同于加权式矩阵。

We concatenate the forward and backward states to to obtain the annotations $(h_{1},h_{2},\cdots,h_{T_{x}})$, where

我们连接前向和后向状态以获得注释$(h_{1},h_{2},\cdots,h_{T_{x}})$，其中

$$\displaystyle h_{i}=\left[\begin{array}[]{c}\overrightarrow{h}{i}\
\overleftarrow{h}{i}\end{array}\right]$$

A.2.2 Decoder

The hidden state $s_{i}$ of the decoder given the annotations from the encoder is computed by

A.2.2 解码器

解码器缔结一项注释的隐藏状态$s_{i}$由

$$\displaystyle s_{i}= \\ \displaystyle(1-z_{i})\circ s_{i-1}+z_{i}\circ\tilde{s}_{i},$$

where

何处

$$\displaystyle\tilde{s}_{i}= \\ \displaystyle\tanh\left(WEy_{i-1}+U\left[r_{i}\circ s_{i-1}\right]+Cc_{i}\right) \\ \displaystyle z_{i}= \\ \displaystyle\sigma\left(W_{z}Ey_{i-1}+U_{z}s_{i-1}+C_{z}c_{i}\right) \\ \displaystyle r_{i}= \\ \displaystyle\sigma\left(W_{r}Ey_{i-1}+U_{r}s_{i-1}+C_{r}c_{i}\right)$$

$E$ is the word embedding matrix for the target language. $W,W_{z},W_{r}\in\mathbb{R}^{n\times m}$, $U,U_{z},U_{r}\in\mathbb{R}^{n\times n}$, and $C,C_{z},C_{r}\in\mathbb{R}^{n\times 2n}$ are weights. Again, $m$ and $n$ are the word embedding dimensionality and the number of hidden units, respectively. The initial hidden state $s_{0}$ is computed by $s_{0}=\tanh\left(W_{s}\overleftarrow{h}_{1}\right),$ where $W_{s}\in\mathbb{R}^{n\times n}$.

$E$ 是目标语言的词嵌入矩阵。$W,W_{z},W_{r}\in\mathbb{R}^{n\times m}$，$U,U_{z},U_{r}\in\mathbb{R}^{n\times n}$，以及 $C,C_{z},C_{r}\in\mathbb{R}^{n\times 2n}$ 是权重。同样，$m$ 与 $n$ 分别是词嵌入维数与隐单元数。初始隐状态 $s_{0}$ 由 $s_{0}=\tanh\left(W_{s}\overleftarrow{h}_{1}\right)$ 计算，其中 $W_{s}\in\mathbb{R}^{n\times n}$。

The context vector $c_{i}$ are recomputed at each step by the alignment model:

上尚未完成的的某些方面谧

$$\displaystyle c_{i}= \\ \displaystyle\sum_{j=1}^{T_{x}}\alpha_{ij}h_{j},$$

where

地点

$$\displaystyle\alpha_{ij}= \\ \displaystyle\frac{\exp\left(e_{ij}\right)}{\sum_{k=1}^{T_{x}}\exp\left(e_{ik}\right)} \\ \displaystyle e_{ij}= \\ \displaystyle v_{a}^{\top}\tanh\left(W_{a}s_{i-1}+U_{a}h_{j}\right),$$

and $h_{j}$ is the $j$-th annotation in the source sentence (see Eq. (A.2.1)). $v_{a}\in\mathbb{R}^{n^{\prime}},W_{a}\in\mathbb{R}^{n^{\prime}\times n}$ and $U_{a}\in\mathbb{R}^{n^{\prime}\times 2n}$ are weight matrices. Note that the model becomes RNN Encoder–Decoder (Cho et al., 2014a), if we fix $c_{i}$ to $\overrightarrow{h}_{T_{x}}$.

而 $h_{j}$ 是源句中的第 $j$ 个标注（见式 (A.2.1)）。$v_{a}\in\mathbb{R}^{n^{\prime}},W_{a}\in\mathbb{R}^{n^{\prime}\times n}$ 与 $U_{a}\in\mathbb{R}^{n^{\prime}\times 2n}$ 是权重矩阵。注意，若把 $c_{i}$ 固定为 $\overrightarrow{h}_{T_{x}}$，模型就退化为 RNN 编码器–解码器（Cho 等, 2014a）。

With the decoder state $s_{i-1}$, the context $c_{i}$ and the last generated word $y_{i-1}$, we define the probability of a target word $y_{i}$ as

利用解码器状态$s_{i-1}$、上下文$c_{i}$和最后生成的单词$y_{i-1}$，我们将目标单词$y_{i}$的概率定义为

$$\displaystyle p(y_{i}|s_{i},y_{i-1},c_{i})\propto \\ \displaystyle\exp\left(y_{i}^{\top}W_{o}t_{i}\right),$$

where

时, 所有的,

$$\displaystyle t_{i}= \\ \displaystyle\left[\max\left\{\tilde{t}_{i,2j-1},\tilde{t}_{i,2j}\right\}\right]_{j=1,\ldots,l}^{\top}$$

and $\tilde{t}_{i,k}$ is the $k$-th element of a vector $\tilde{t}_{i}$ which is computed by

和促进实现$\tilde{t}_{i,k}$是矢量$\tilde{t}_{i}$的$k$- 第 1 个所发挥的作用。

$$\displaystyle\tilde{t}_{i}= \\ \displaystyle U_{o}s_{i-1}+V_{o}Ey_{i-1}+C_{o}c_{i}.$$

$W_{o}\in\mathbb{R}^{K_{y}\times l}$, $U_{o}\in\mathbb{R}^{2l\times n}$, $V_{o}\in\mathbb{R}^{2l\times m}$ and $C_{o}\in\mathbb{R}^{2l\times 2n}$ are weight matrices. This can be understood as having a deep output (Pascanu et al., 2014) with a single maxout hidden layer (Goodfellow et al., 2013).

$W_{o}\in\mathbb{R}^{K_{y}\times l}$, XXMAT1XX, 你喜欢吗 $V_{o}\in\mathbb{R}^{2l\times m}$和列述 XX凤彬是重量矩阵. 这可以理解为具有深层输出(Pascanu等人酝酿,2014年),的自由输出有单一的Maxout隐藏层(Goodfellow等人,2013盏)。

A.2.3 Model Size

For all the models used in this paper, the size of a hidden layer $n$ is 1000, the word embedding dimensionality $m$ is 620 and the size of the maxout hidden layer in the deep output $l$ is 500. The number of hidden units in the alignment model $n^{\prime}$ is 1000.

A.2.3 型号大小

对于本文使用的所有模型,一个隐藏层$n$的大小为1000,嵌入维度$m$的单词为620,深输出XXMAT拱的隐藏层的大小为500. 证明了这一点,对齐模式$n^{\prime}$中的隐藏单元数量为1000个.

Model	Updates ($\times 10^{5}$)	Epochs	Hours	GPU	Train NLL	Dev. NLL
RNNenc-30	8.46	6.4	109	TITAN BLACK	28.1	53.0
RNNenc-50	6.00	4.5	108	Quadro K-6000	44.0	43.6
RNNsearch-30	4.71	3.6	113	TITAN BLACK	26.7	47.2
RNNsearch-50	2.88	2.2	111	Quadro K-6000	40.7	38.1
RNNsearch-50⋆	6.67	5.0	252	Quadro K-6000	36.7	35.2

Table 2: Learning statistics and relevant information. Each update corresponds to updating the parameters once using a single minibatch. One epoch is one pass through the training set. NLL is the average conditional log-probabilities of the sentences in either the training set or the development set. Note that the lengths of the sentences differ.

模式	(原始内容存档于2019-09-25) (中文(简体) ). September (简体).	埃波希斯语Name	小时	GPU 常规	火车 NLL列有	德夫·诺尔
区域所作的承诺	8.46	6.4	109	蒂坦布莱克	28.1	53.0
RNEN特别是那些 70	6.00	4.5	108	四方 K-6000	44.0	43.6
才可以进行搜索 你还不清楚	4.71	3.6	113	吟贝·贝 附 你把贝·贝·贝·贝·贝·贝· 这里	26.7	47.2
搜索-50侏儒	2.88	2.2	111	四方 K-6000	40.7	38.1
, 在某种程度上,	6.67	5.0	252	四方 K-6000	36.7	35.2

表2:学习统计和相关信息。 每次更新都对应一次使用单个小批量更新参数. 一个时代就是通过训练 NLL 是指在培训或开发中,炖料中,无论在培训中还是开发中,均能实现平均的对齐性在实施中. 注意句子长度不同.

Appendix B Training Procedure

附录 B 训练过程

B.1 Parameter Initialization

B.1 参数初始化

We initialized the recurrent weight matrices $U,U_{z},U_{r},\overleftarrow{U},\overleftarrow{U}_{z},\overleftarrow{U}_{r},\overrightarrow{U},\overrightarrow{U}_{z}$ and $\overrightarrow{U}_{r}$ as random orthogonal matrices. For $W_{a}$ and $U_{a}$, we initialized them by sampling each element from the Gaussian distribution of mean $0$ and variance $0.001^{2}$. All the elements of $V_{a}$ and all the bias vectors were initialized to zero. Any other weight matrix was initialized by sampling from the Gaussian distribution of mean $0$ and variance $0.01^{2}$.

我们初始化了经常性加权矩阵$U,U_{z},U_{r},\overleftarrow{U},\overleftarrow{U}_{z},\overleftarrow{U}_{r},\overrightarrow{U},\overrightarrow{U}_{z}$和$\overrightarrow{U}_{r}$,作为随机正交矩阵. 对于$W_{a}$和$U_{a}$,我们通过取样从平均$0$和差异$0.001^{2}$的高斯分布的每个元素来初始化. $V_{a}$的所有元素和所有偏差向量被初始化为零. 任何其他重量矩阵都是通过从平均$0$和差异$0.01^{2}$的高斯分布抽样而初始化的。

B.2 Training

B.2 训练

We used the stochastic gradient descent (SGD) algorithm. Adadelta (Zeiler, 2012) was used to automatically adapt the learning rate of each parameter ($\epsilon=10^{-6}$ and $\rho=0.95$). We explicitly normalized the $L_{2}$-norm of the gradient of the cost function each time to be at most a predefined threshold of $1$, when the norm was larger than the threshold (Pascanu et al., 2013b). Each SGD update direction was computed with a minibatch of 80 sentences.

我们使用了斜坡梯度下降算法。 Adadelta(Zeiler,2012年)用于自动调整每个参数的学习率($\epsilon=10^{-6}$和$\rho=0.95$ ). 我们明确将每次成本函数梯度的$L_{2}$规范正常化,最多为$1$的预先设定阈值,而规范值大于阈值(Pascanu等人,2013年b)。 每个SGD更新方向都是用80句的小批量计算出来的.

At each update our implementation requires time proportional to the length of the longest sentence in a minibatch. Hence, to minimize the waste of computation, before every 20-th update, we retrieved 1600 sentence pairs, sorted them according to the lengths and split them into 20 minibatches. The training data was shuffled once before training and was traversed sequentially in this manner.

在每次更新我们的执行情况时,都需要与小批量中最长刑期的长度相称的时间。 因此,为了尽量减少计算上的浪费,在每20次更新前,我们检索到1600个句子对,按照长度进行分类,然后分成20个小炮管. 培训数据在培训前被洗刷过一次,并依次以这种方式传递。

In Tables 2 we present the statistics related to training all the models used in the experiments.

在表2中,我们介绍了与培训实验中使用的所有模型有关的统计。

Appendix C Translations of Long Sentences

附录 C 长句翻译

Source	An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital.
Reference	Le privilège d’admission est le droit d’un médecin, en vertu de son statut de membre soignant d’un hôpital, d’admettre un patient dans un hôpital ou un centre médical afin d’y délivrer un diagnostic ou un traitement.
RNNenc-50	Un privilège d’admission est le droit d’un médecin de reconnaître un patient à l’hôpital ou un centre médical d’un diagnostic ou de prendre un diagnostic en fonction de son état de santé.
RNNsearch-50	Un privilège d’admission est le droit d’un médecin d’admettre un patient à un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, selon son statut de travailleur des soins de santé à l’hôpital.
Google Translate	Un privilège admettre est le droit d’un médecin d’admettre un patient dans un hôpital ou un centre médical pour effectuer un diagnostic ou une procédure, fondée sur sa situation en tant que travailleur de soins de santé dans un hôpital.
Source	This kind of experience is part of Disney’s efforts to ”extend the lifetime of its series and build new relationships with audiences via digital platforms that are becoming ever more important,” he added.
Reference	Ce type d’expérience entre dans le cadre des efforts de Disney pour ”étendre la durée de vie de ses séries et construire de nouvelles relations avec son public grâce à des plateformes numériques qui sont de plus en plus importantes”, a-t-il ajouté.
RNNenc-50	Ce type d’expérience fait partie des initiatives du Disney pour ”prolonger la durée de vie de ses nouvelles et de développer des liens avec les lecteurs numériques qui deviennent plus complexes.
RNNsearch-50	Ce genre d’expérience fait partie des efforts de Disney pour ”prolonger la durée de vie de ses séries et créer de nouvelles relations avec des publics via des plateformes numériques de plus en plus importantes”, a-t-il ajouté.
Google Translate	Ce genre d’expérience fait partie des efforts de Disney à “étendre la durée de vie de sa série et construire de nouvelles relations avec le public par le biais des plates-formes numériques qui deviennent de plus en plus important”, at-il ajouté.
Source	In a press conference on Thursday, Mr Blair stated that there was nothing in this video that might constitute a ”reasonable motive” that could lead to criminal charges being brought against the mayor.
Reference	En conférence de presse, jeudi, M. Blair a affirmé qu’il n’y avait rien dans cette vidéo qui puisse constituer des ”motifs raisonnables” pouvant mener au dépôt d’une accusation criminelle contre le maire.
RNNenc-50	Lors de la conférence de presse de jeudi, M. Blair a dit qu’il n’y avait rien dans cette vidéo qui pourrait constituer une ”motivation raisonnable” pouvant entraîner des accusations criminelles portées contre le maire.
RNNsearch-50	Lors d’une conférence de presse jeudi, M. Blair a déclaré qu’il n’y avait rien dans cette vidéo qui pourrait constituer un ”motif raisonnable” qui pourrait conduire à des accusations criminelles contre le maire.
Google Translate	Lors d’une conférence de presse jeudi, M. Blair a déclaré qu’il n’y avait rien dans cette vido qui pourrait constituer un ”motif raisonnable” qui pourrait mener à des accusations criminelles portes contre le maire.

Table 3: The translations generated by RNNenc-50 and RNNsearch-50 from long source sentences (30 words or more) selected from the test set. For each source sentence, we also show the gold-standard translation. The translations by Google Translate were made on 27 August 2014.

来源	承认的特权是,医生有权根据病人在医院的保健工作者的身份,将病人送往医院或医疗中心进行诊断或手术。
参考	医学博士学位、博士学位、博士学位、博士学位、博士学位。
伦尼茨-50	未经批准,不得在医院或医院进行诊断。
第50条	医院院长、院长、院长、院长。
谷歌翻译	在医院里,对监狱状况的热爱。
来源	他补充说,这种经验是迪士尼努力“延长其系列的寿命,并通过日益重要的数字平台与受众建立新的关系”。
参考	迪士尼工作干部创业中心,《新关系之声》,A-I Ajouté。
伦尼茨-50	迪士尼公司在迪士尼公司(Disney propositions durée de vie de ses Newa et de développer des leave les electels numériques qui devienenenant + complex)倡议中的“既成参与者”类型。
第50条	迪士尼努力中的既成部分(Ce genre d ' expérience 既成部分),“通过数字形式加上重要因素,促进新人民关系的发展”。
谷歌翻译	迪士尼努力的既成部分的流派,“新关系的历史演变和构建”,载于Ajouté。
来源	在星期四的记者招待会上,布莱尔先生说,这段录像中没有任何内容可能构成可能导致对市长提出刑事指控的“合理动机”。
参考	《新闻公报》,Jeudi, M. Blair, 一份确认书,
伦尼茨-50	M. Blair, 校对:Soup
第50条	M. Blair,《关于打击恶意犯罪的指控》。
谷歌翻译	M. Blair of expressed de jeudi, M. Blair a déclaré qu ' y avait en an as cette vido qui qui pourrait constituer un " motif reparentable " qui pourrait mener as des crimines portes against le maire. 互联网档案馆的存檔。

表3:RNNenc-50和RNNsearch-50从测试集中选取的长源句(30字或以上)产生的翻译. 对于每个源句,我们也展示了金标准翻译. Google Translate的翻译于2014年8月27日完成.