Sequence to Sequence Learning with Neural Networks

用神经网络做序列到序列学习（Sequence to Sequence Learning with Neural Networks）

Ilya Sutskever Affiliation:  Google Email:  ilyasu@google.com    Oriol Vinyals Affiliation:  Google Email:  vinyals@google.com    Quoc V. Le Affiliation:  Google Email:  qvl@google.com

Ilya Sutskever
所属机构：Google
电子邮箱：ilyasu@google.com

Oriol Vinyals
所属机构：Google
电子邮箱：vinyals@google.com

Quoc V. Le
所属机构：Google
电子邮箱：qvl@google.com

Abstract

摘要

Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT’14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM’s BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous best result on this task. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM’s performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.

深度神经网络（DNN）是强大的模型，在困难的学习任务上表现优异。尽管在有大规模标注训练集时 DNN 效果很好，它们却不能直接用来把序列映射成序列。本文提出一种对序列结构假设很少的通用端到端序列学习方法。我们用多层长短期记忆网络（LSTM）把输入序列映射到定长向量，再用另一个深层 LSTM 从该向量解码出目标序列。主要结果是：在 WMT’14 英语到法语翻译任务上，由五个深层 LSTM（共约 3.84 亿参数）组成的集成，用简单的从左到右束搜索解码，直接得到 34.8 的 BLEU，而当时基于短语的统计机器翻译基线为 33.3。单独使用时 LSTM 尚未追上当时最佳系统，但若用其对基线系统的 1000-best 列表重打分，可达 36.5 BLEU，接近此前最佳结果。LSTM 对词序也敏感：若在训练与测试时把源句词序反转，LSTM 的性能会显著提升。

1 Introduction

1 引言

Deep Neural Networks (DNNs) are extremely powerful machine learning models that achieve excellent performance on difficult problems such as speech recognition [13, 7] and visual object recognition [19, 6, 21, 20]. DNNs are powerful because they can perform arbitrary parallel computation for a modest number of steps. A surprising example of the power of DNNs is their ability to sort $N$ ​​​​ $N$-bit numbers using only 2 hidden layers of quadratic size [27]. So, while neural networks are related to conventional statistical models, they learn an intricate computation. Furthermore, large DNNs can be trained with supervised backpropagation whenever the labeled training set has enough information to specify the network’s parameters. Thus, if there exists a parameter setting of a large DNN that achieves good results (for example, because humans can solve the task very rapidly), supervised backpropagation will find these parameters and solve the problem.

深度神经网络（DNN）是极其强大的机器学习模型，可以在语音识别 [13, 7] 和视觉对象识别 [19, 6, 21, 20] 等难题上取得优异的性能。 DNN 非常强大，因为它们可以以适度的步骤执行任意并行计算。 DNN 强大功能的一个令人惊讶的例子是它们能够仅使用 2 个二次大小的隐藏层对$N$​​​​ $N$位数字进行排序 [27]。 因此，虽然神经网络与传统的统计模型相关，但它们学习复杂的计算。 此外，只要标记的训练集有足够的信息来指定网络参数，大型 DNN 就可以通过有监督的反向传播进行训练。 因此，如果存在一个大型 DNN 的参数设置可以取得良好的结果（例如，因为人类可以非常快速地解决任务），有监督反向传播将找到这些参数并解决问题。

Despite their flexibility and power, DNNs can only be applied to problems whose inputs and targets can be sensibly encoded with vectors of fixed dimensionality. It is a significant limitation, since many important problems are best expressed with sequences whose lengths are not known a-priori. For example, speech recognition and machine translation are sequential problems. Likewise, question answering can also be seen as mapping a sequence of words representing the question to a sequence of words representing the answer. It is therefore clear that a domain-independent method that learns to map sequences to sequences would be useful.

尽管 DNN 具有灵活性和强大功能，但它只能应用于输入和目标可以用固定维度的向量进行合理编码的问题。 这是一个很大的限制，因为许多重要的问题最好用长度未知的序列来表达。 例如，语音识别和机器翻译是顺序问题。 同样，问答也可以被视为将表示问题的单词序列映射到表示答案的单词序列。 因此，很明显，学习将序列映射到序列的独立于域的方法将是有用的。

Sequences pose a challenge for DNNs because they require that the dimensionality of the inputs and outputs is known and fixed. In this paper, we show that a straightforward application of the Long Short-Term Memory (LSTM) architecture [16] can solve general sequence to sequence problems. The idea is to use one LSTM to read the input sequence, one timestep at a time, to obtain large fixed-dimensional vector representation, and then to use another LSTM to extract the output sequence from that vector (fig. 1). The second LSTM is essentially a recurrent neural network language model [28, 23, 30] except that it is conditioned on the input sequence. The LSTM’s ability to successfully learn on data with long range temporal dependencies makes it a natural choice for this application due to the considerable time lag between the inputs and their corresponding outputs (fig. 1).

序列对 DNN 提出了挑战，因为它们要求输入和输出的维数已知且固定。 在本文中，我们证明了长短期记忆（LSTM）架构[16]的直接应用可以解决一般的序列到序列问题。 这个想法是使用一个 LSTM 读取输入序列，一次一个时间步，以获得大型固定维向量表示，然后使用另一个 LSTM 从该向量中提取输出序列（图 1）。 第二个 LSTM 本质上是一个循环神经网络语言模型 [28,23,30]，只不过它以输入序列为条件。 由于输入与其相应输出之间存在相当大的时间滞后，LSTM 能够成功学习具有长范围时间依赖性的数据，使其成为此应用程序的自然选择（图 1）。

There have been a number of related attempts to address the general sequence to sequence learning problem with neural networks. Our approach is closely related to Kalchbrenner and Blunsom [18] who were the first to map the entire input sentence to vector, and is related to Cho et al. [5] although the latter was used only for rescoring hypotheses produced by a phrase-based system. Graves [10] introduced a novel differentiable attention mechanism that allows neural networks to focus on different parts of their input, and an elegant variant of this idea was successfully applied to machine translation by Bahdanau et al. [2]. The Connectionist Sequence Classification is another popular technique for mapping sequences to sequences with neural networks, but it assumes a monotonic alignment between the inputs and the outputs [11].

已经进行了许多相关尝试来解决神经网络的一般序列到序列学习问题。 我们的方法与 Kalchbrenner 和 Blunsom [18] 密切相关，他们是第一个将整个输入句子映射到向量的人，并且与 Cho 等人相关。 [5]尽管后者仅用于重新评分基于短语的系统产生的假设。 Graves [10] 引入了一种新颖的可微分注意力机制，允许神经网络关注其输入的不同部分，并且这个想法的一个优雅变体被 Bahdanau 等人成功应用于机器翻译。 [2]。 连接主义序列分类是另一种使用神经网络将序列映射到序列的流行技术，但它假设输入和输出之间是单调对齐的[11]。

Figure 1: Our model reads an input sentence “ABC” and produces “WXYZ” as the output sentence. The model stops making predictions after outputting the end-of-sentence token. Note that the LSTM reads the input sentence in reverse, because doing so introduces many short term dependencies in the data that make the optimization problem much easier.

图 1：我们的模型读取输入句子“ABC”并生成“WXYZ”作为输出句子。 输出句末标记后，模型停止进行预测。 请注意，LSTM 反向读取输入句子，因为这样做会在数据中引入许多短期依赖性，从而使优化问题变得更加容易。

The main result of this work is the following. On the WMT’14 English to French translation task, we obtained a BLEU score of 34.81 by directly extracting translations from an ensemble of 5 deep LSTMs (with 384M parameters and 8,000 dimensional state each) using a simple left-to-right beam-search decoder. This is by far the best result achieved by direct translation with large neural networks. For comparison, the BLEU score of an SMT baseline on this dataset is 33.30 [29]. The 34.81 BLEU score was achieved by an LSTM with a vocabulary of 80k words, so the score was penalized whenever the reference translation contained a word not covered by these 80k. This result shows that a relatively unoptimized small-vocabulary neural network architecture which has much room for improvement outperforms a phrase-based SMT system.

本工作的主要结果如下。在 WMT’14 英语到法语翻译任务上，我们从一个由 5 个深层 LSTM（共约 3.84 亿参数，每个状态维 8000）组成的集成中，用简单的从左到右束搜索解码器直接抽出译文，得到 BLEU 34.81。这是大型神经网络直接翻译迄今最好的结果。对照之下，该数据集上 SMT 基线的 BLEU 为 33.30[29]。34.81 分由词表约 8 万词的 LSTM 取得，因此每当参考译文含未登录词时分数都会被惩罚。

Finally, we used the LSTM to rescore the publicly available 1000-best lists of the SMT baseline on the same task [29]. By doing so, we obtained a BLEU score of 36.5, which improves the baseline by 3.2 BLEU points and is close to the previous best published result on this task (which is 37.0 [9]).

最后，我们使用 LSTM 对同一任务的公开可用的 1000 个最佳 SMT 基线列表进行重新评分 [29]。 通过这样做，我们获得了 36.5 的 BLEU 分数，这将基线提高了 3.2 BLEU 分，接近之前发布的该任务的最佳结果（37.0 [9]）。

Surprisingly, the LSTM did not suffer on very long sentences, despite the recent experience of other researchers with related architectures [26]. We were able to do well on long sentences because we reversed the order of words in the source sentence but not the target sentences in the training and test set. By doing so, we introduced many short term dependencies that made the optimization problem much simpler (see sec. 2 and 3.3). As a result, SGD could learn LSTMs that had no trouble with long sentences. The simple trick of reversing the words in the source sentence is one of the key technical contributions of this work.

出人意料的是，尽管近期相关架构的研究者有过相反经验[26]，LSTM 在很长的句子上并未明显吃亏。我们能在长句上表现良好，是因为在训练与测试集中把源句词序反转、目标句保持原序。由此引入许多短程依赖，使优化问题容易得多（见第 2 节与 3.3 节）。于是 SGD 能学出对长句也不发怵的 LSTM。把源句词序反转这一简单技巧，是本工作的关键技术贡献之一。

A useful property of the LSTM is that it learns to map an input sentence of variable length into a fixed-dimensional vector representation. Given that translations tend to be paraphrases of the source sentences, the translation objective encourages the LSTM to find sentence representations that capture their meaning, as sentences with similar meanings are close to each other while different sentences meanings will be far. A qualitative evaluation supports this claim, showing that our model is aware of word order and is fairly invariant to the active and passive voice.

LSTM 的一个有用特性是它能够学习将可变长度的输入句子映射到固定维度的向量表示。 鉴于翻译往往是源句子的释义，翻译目标鼓励 LSTM 找到捕获其含义的句子表示，因为具有相似含义的句子彼此接近，而不同句子含义则相距较远。 定性评估支持了这一说法，表明我们的模型能够识别词序，并且对于主动语态和被动语态相当不变。

2 The model

2 模型

The Recurrent Neural Network (RNN) [31, 28] is a natural generalization of feedforward neural networks to sequences. Given a sequence of inputs $(x_{1},\ldots,x_{T})$, a standard RNN computes a sequence of outputs $(y_{1},\ldots,y_{T})$ by iterating the following equation:

循环神经网络 (RNN) [31, 28] 是前馈神经网络对序列的自然推广。 给定输入序列$(x_{1},\ldots,x_{T})$，标准 RNN 通过迭代以下等式来计算输出序列$(y_{1},\ldots,y_{T})$：

$$\displaystyle h_{t} \\ \displaystyle= \\ \displaystyle\mathrm{sigm}\left(W^{\mathrm{hx}}x_{t}+W^{\mathrm{hh}}h_{t-1}\right) \\ \displaystyle y_{t} \\ \displaystyle= \\ \displaystyle W^{\mathrm{yh}}h_{t}$$

The RNN can easily map sequences to sequences whenever the alignment between the inputs the outputs is known ahead of time. However, it is not clear how to apply an RNN to problems whose input and the output sequences have different lengths with complicated and non-monotonic relationships.

只要提前知道输入和输出之间的对齐情况，RNN 就可以轻松地将序列映射到序列。 然而，目前尚不清楚如何将 RNN 应用于输入和输出序列长度不同且关系复杂且非单调的问题。

The simplest strategy for general sequence learning is to map the input sequence to a fixed-sized vector using one RNN, and then to map the vector to the target sequence with another RNN (this approach has also been taken by Cho et al. [5]). While it could work in principle since the RNN is provided with all the relevant information, it would be difficult to train the RNNs due to the resulting long term dependencies (figure 1) [14, 4, 16, 15]. However, the Long Short-Term Memory (LSTM) [16] is known to learn problems with long range temporal dependencies, so an LSTM may succeed in this setting.

最简单的通用序列学习策略是：用一个 RNN 把输入序列映到定长向量，再用另一个 RNN 把该向量映成目标序列（这一思路也曾被 Cho 等人[5]与 Kalchbrenner 与 Blunsom[18]采用）。尽管原则上它能处理序列问题，但由于长程依赖造成的优化困难，用 DNN 成功训练这类模型曾极具挑战[26]。LSTM 的目的之一正是应对长程依赖，因此我们选择 LSTM 做这一序列到序列任务是自然的。

The goal of the LSTM is to estimate the conditional probability $p(y_{1},\ldots,y_{T^{\prime}}|x_{1},\ldots,x_{T})$ where $(x_{1},\ldots,x_{T})$ is an input sequence and $y_{1},\ldots,y_{T^{\prime}}$ is its corresponding output sequence whose length $T^{\prime}$ may differ from $T$. The LSTM computes this conditional probability by first obtaining the fixed-dimensional representation $v$ of the input sequence $(x_{1},\ldots,x_{T})$ given by the last hidden state of the LSTM, and then computing the probability of $y_{1},\ldots,y_{T^{\prime}}$ with a standard LSTM-LM formulation whose initial hidden state is set to the representation $v$ of $x_{1},\ldots,x_{T}$:

LSTM 的目标是估计条件概率 $p(y_{1},\ldots,y_{T^{\prime}}|x_{1},\ldots,x_{T})$，其中$(x_{1},\ldots,x_{T})$是输入序列，$y_{1},\ldots,y_{T^{\prime}}$ 是其相应的输出序列，其长度$T^{\prime}$可能与$T$不同。 LSTM 通过首先获得由 LSTM 最后一个隐藏状态给出的输入序列$(x_{1},\ldots,x_{T})$的固定维表示$v$来计算此条件概率，然后使用标准 LSTM-LM 公式计算$y_{1},\ldots,y_{T^{\prime}}$的概率，该公式的初始隐藏状态设置为$x_{1},\ldots,x_{T}$的表示$v$：

$$p(y_{1},\ldots,y_{T^{\prime}}|x_{1},\ldots,x_{T})=\prod_{t=1}^{T^{\prime}}p(y_{t}|v,y_{1},\ldots,y_{t-1}) \tag{1}$$

In this equation, each $p(y_{t}|v,y_{1},\ldots,y_{t-1})$ distribution is represented with a softmax over all the words in the vocabulary. We use the LSTM formulation from Graves [10]. Note that we require that each sentence ends with a special end-of-sentence symbol “$<$EOS$>$”, which enables the model to define a distribution over sequences of all possible lengths. The overall scheme is outlined in figure 1, where the shown LSTM computes the representation of “A”, “B”, “C”, “$<$EOS$>$” and then uses this representation to compute the probability of “W”, “X”, “Y”, “Z”, “$<$EOS$>$”.

在此等式中，每个$p(y_{t}|v,y_{1},\ldots,y_{t-1})$分布都用词汇表中所有单词的 softmax 表示。 我们使用 Graves [10] 的 LSTM 公式。 请注意，我们要求每个句子以特殊的句尾符号“$<$EOS$>$”结尾，这使得模型能够定义所有可能长度的序列上的分布。 整体方案如图 1 所示，其中所示的 LSTM 计算“A”、“B”、“C”、“$<$EOS$>$”的表示，然后使用该表示来计算“W”、“X”、“Y”、“Z”、“$<$EOS$>$”的概率。

Our actual models differ from the above description in three important ways. First, we used two different LSTMs: one for the input sequence and another for the output sequence, because doing so increases the number model parameters at negligible computational cost and makes it natural to train the LSTM on multiple language pairs simultaneously [18]. Second, we found that deep LSTMs significantly outperformed shallow LSTMs, so we chose an LSTM with four layers. Third, we found it extremely valuable to reverse the order of the words of the input sentence. So for example, instead of mapping the sentence $a,b,c$ to the sentence $\alpha,\beta,\gamma$, the LSTM is asked to map $c,b,a$ to $\alpha,\beta,\gamma$, where $\alpha,\beta,\gamma$ is the translation of $a,b,c$. This way, $a$ is in close proximity to $\alpha$, $b$ is fairly close to $\beta$, and so on, a fact that makes it easy for SGD to “establish communication” between the input and the output. We found this simple data transformation to greatly improve the performance of the LSTM.

我们的实际模型在三个重要方面与上述描述不同。 首先，我们使用了两种不同的 LSTM：一种用于输入序列，另一种用于输出序列，因为这样做会以可忽略的计算成本增加模型参数的数量，并且可以很自然地同时在多个语言对上训练 LSTM [18]。 其次，我们发现深层 LSTM 的性能明显优于浅层 LSTM，因此我们选择了四层 LSTM。 第三，我们发现颠倒输入句子的单词顺序非常有价值。 例如，不是将句子$a,b,c$映射到句子$\alpha,\beta,\gamma$，而是要求 LSTM 将$c,b,a$映射到$\alpha,\beta,\gamma$，其中$\alpha,\beta,\gamma$是$a,b,c$的翻译。 这样，$a$非常接近$\alpha$，$b$相当接近$\beta$，等等，这一事实使得 SGD 可以轻松地在输入和输出之间“建立通信”。 我们发现这种简单的数据转换可以极大地提高 LSTM 的性能。

3 Experiments

3 实验

We applied our method to the WMT’14 English to French MT task in two ways. We used it to directly translate the input sentence without using a reference SMT system and we it to rescore the n-best lists of an SMT baseline. We report the accuracy of these translation methods, present sample translations, and visualize the resulting sentence representation.

我们通过两种方式将我们的方法应用于 WMT’14 英语到法语 MT 任务。 我们用它来直接翻译输入句子，而不使用参考 SMT 系统，并用它来重新评分 SMT 基线的 n 个最佳列表。 我们报告这些翻译方法的准确性，提供示例翻译，并可视化生成的句子表示。

3.1 Dataset details

3.1 数据集细节

We used the WMT’14 English to French dataset. We trained our models on a subset of 12M sentences consisting of 348M French words and 304M English words, which is a clean “selected” subset from [29]. We chose this translation task and this specific training set subset because of the public availability of a tokenized training and test set together with 1000-best lists from the baseline SMT [29].

我们使用 WMT’14 英语到法语数据集。 我们在由 348M 法语单词和 304M 英语单词组成的 12M 句子子集上训练我们的模型，这是[29]中干净的“选择”子集。 我们选择这个翻译任务和这个特定的训练集子集是因为标记化训练和测试集以及来自基线 SMT [29] 的 1000 个最佳列表是公开可用的。

As typical neural language models rely on a vector representation for each word, we used a fixed vocabulary for both languages. We used 160,000 of the most frequent words for the source language and 80,000 of the most frequent words for the target language. Every out-of-vocabulary word was replaced with a special “UNK” token.

由于典型的神经语言模型依赖于每个单词的向量表示，因此我们对两种语言都使用了固定的词汇表。 我们使用了 160,000 个源语言最常用单词和 80,000 个目标语言最常用单词。 每个词汇表之外的单词都被替换为特殊的“UNK”标记。

3.2 Decoding and Rescoring

3.2 解码与打分

The core of our experiments involved training a large deep LSTM on many sentence pairs. We trained it by maximizing the log probability of a correct translation $T$ given the source sentence $S$, so the training objective is

我们实验的核心涉及在许多句子对上训练大型深度 LSTM。 我们通过在给定源句子$S$的情况下最大化正确翻译$T$的对数概率来训练它，因此训练目标是

$$1/|\mathcal{S}|\sum_{(T,S)\in\mathcal{S}}\log p(T|S)$$

where $\mathcal{S}$ is the training set. Once training is complete, we produce translations by finding the most likely translation according to the LSTM:

其中$\mathcal{S}$是训练集。 训练完成后，我们会根据 LSTM 查找最有可能的翻译来生成翻译：

$$\hat{T}=\arg\max_{T}p(T|S) \tag{2}$$

We search for the most likely translation using a simple left-to-right beam search decoder which maintains a small number $B$ of partial hypotheses, where a partial hypothesis is a prefix of some translation. At each timestep we extend each partial hypothesis in the beam with every possible word in the vocabulary. This greatly increases the number of the hypotheses so we discard all but the $B$ most likely hypotheses according to the model’s log probability. As soon as the “$<$EOS$>$” symbol is appended to a hypothesis, it is removed from the beam and is added to the set of complete hypotheses. While this decoder is approximate, it is simple to implement. Interestingly, our system performs well even with a beam size of 1, and a beam of size 2 provides most of the benefits of beam search (Table 1).

我们使用简单的从左到右波束搜索解码器来搜索最可能的翻译，该解码器维护少量$B$的部分假设，其中部分假设是某些翻译的前缀。 在每个时间步，我们用词汇表中的每个可能的单词扩展光束中的每个部分假设。 这大大增加了假设的数量，因此我们根据模型的对数概率丢弃除$B$最可能的假设之外的所有假设。 一旦“$<$ EOS $>$”符号被附加到假设中，它就会从梁中删除并添加到完整假设集中。 虽然该解码器是近似的，但实现起来很简单。 有趣的是，即使波束尺寸为 1，我们的系统也能表现良好，而尺寸为 2 的波束提供了波束搜索的大部分优点（表 1）。

We also used the LSTM to rescore the 1000-best lists produced by the baseline system [29]. To rescore an n-best list, we computed the log probability of every hypothesis with our LSTM and took an even average with their score and the LSTM’s score.

我们还使用 LSTM 对基线系统生成的 1000 个最佳列表进行重新评分 [29]。 为了重新评分 n-best 列表，我们使用 LSTM 计算每个假设的对数概率，并用它们的分数和 LSTM 的分数取平均数。

3.3 Reversing the Source Sentences

3.3 反转源句

While the LSTM is capable of solving problems with long term dependencies, we discovered that the LSTM learns much better when the source sentences are reversed (the target sentences are not reversed). By doing so, the LSTM’s test perplexity dropped from 5.8 to 4.7, and the test BLEU scores of its decoded translations increased from 25.9 to 30.6.

虽然 LSTM 能够解决具有长期依赖关系的问题，但我们发现当源句子反转（目标句子不反转）时，LSTM 学得更好。 通过这样做，LSTM 的测试困惑度从 5.8 下降到 4.7，解码翻译的测试 BLEU 分数从 25.9 增加到 30.6。

While we do not have a complete explanation to this phenomenon, we believe that it is caused by the introduction of many short term dependencies to the dataset. Normally, when we concatenate a source sentence with a target sentence, each word in the source sentence is far from its corresponding word in the target sentence. As a result, the problem has a large “minimal time lag” [17]. By reversing the words in the source sentence, the average distance between corresponding words in the source and target language is unchanged. However, the first few words in the source language are now very close to the first few words in the target language, so the problem’s minimal time lag is greatly reduced. Thus, backpropagation has an easier time “establishing communication” between the source sentence and the target sentence, which in turn results in substantially improved overall performance.

虽然我们对这种现象没有完整的解释，但我们认为这是由于向数据集引入了许多短期依赖性而引起的。 通常，当我们将源句子与目标句子连接时，源句子中的每个单词与目标句子中对应的单词相距甚远。 因此，该问题具有很大的“最小时滞”[17]。 通过颠倒源句子中的单词，源语言和目标语言中对应单词之间的平均距离保持不变。 然而，源语言中的前几个单词现在与目标语言中的前几个单词非常接近，因此问题的最小时间滞后大大减少。 因此，反向传播可以更轻松地在源句子和目标句子之间“建立通信”，从而显着提高整体性能。

Initially, we believed that reversing the input sentences would only lead to more confident predictions in the early parts of the target sentence and to less confident predictions in the later parts. However, LSTMs trained on reversed source sentences did much better on long sentences than LSTMs trained on the raw source sentences (see sec. 3.7), which suggests that reversing the input sentences results in LSTMs with better memory utilization.

最初，我们认为颠倒输入句子只会导致目标句子的前半部分的预测更自信，而后半部分的预测更不自信。 然而，在反向源句子上训练的 LSTM 在长句子上的表现比在原始源句子上训练的 LSTM 好得多（参见第 3.7 节），这表明反转输入句子会导致 LSTM 具有更好的内存利用率。

3.4 Training details

3.4 训练细节

We found that the LSTM models are fairly easy to train. We used deep LSTMs with 4 layers, with 1000 cells at each layer and 1000 dimensional word embeddings, with an input vocabulary of 160,000 and an output vocabulary of 80,000. Thus the deep LSTM uses 8000 real numbers to represent a sentence. We found deep LSTMs to significantly outperform shallow LSTMs, where each additional layer reduced perplexity by nearly 10%, possibly due to their much larger hidden state. We used a naive softmax over 80,000 words at each output. The resulting LSTM has 384M parameters of which 64M are pure recurrent connections (32M for the “encoder” LSTM and 32M for the “decoder” LSTM). The complete training details are given below:

我们发现 LSTM 模型相当容易训练。 我们使用 4 层的深度 LSTM，每层有 1000 个单元和 1000 维词嵌入，输入词汇量为 160,000，输出词汇量为 80,000。 因此深度LSTM使用8000个实数来表示一个句子。 我们发现深层 LSTM 的性能明显优于浅层 LSTM，其中每增加一层，困惑度就会降低近 10%，这可能是因为它们的隐藏状态要大得多。 我们在每个输出中使用了超过 80,000 个单词的朴素 softmax。 由此产生的 LSTM 有 384M 个参数，其中 64M 是纯循环连接（32M 用于“编码器”LSTM，32M​​ 用于“解码器”LSTM）。 完整的培训细节如下：

• • We initialized all of the LSTM’s parameters with the uniform distribution between -0.08 and 0.08
• • We used stochastic gradient descent without momentum, with a fixed learning rate of 0.7. After 5 epochs, we begun halving the learning rate every half epoch. We trained our models for a total of 7.5 epochs.
• • We used batches of 128 sequences for the gradient and divided it the size of the batch (namely, 128).
• • Although LSTMs tend to not suffer from the vanishing gradient problem, they can have exploding gradients. Thus we enforced a hard constraint on the norm of the gradient [10, 25] by scaling it when its norm exceeded a threshold. For each training batch, we compute $s=\left\|g\right\|_{2}$, where $g$ is the gradient divided by 128. If $s>5$, we set $g=\frac{5g}{s}$.
• • Different sentences have different lengths. Most sentences are short (e.g., length 20-30) but some sentences are long (e.g., length $>$ 100), so a minibatch of 128 randomly chosen training sentences will have many short sentences and few long sentences, and as a result, much of the computation in the minibatch is wasted. To address this problem, we made sure that all sentences in a minibatch are roughly of the same length, yielding a 2x speedup.

• • 我们初始化了 LSTM 的所有参数，使其均匀分布在 -0.08 和 0.08 之间
• • 我们使用无动量的随机梯度下降，固定学习率为0.7。 5 个 epoch 后，我们开始每半个 epoch 将学习率减半。 我们总共训练了 7.5 个时期的模型。
• • 我们使用128 个序列的批次作为梯度，并将其划分为批次的大小（即128）。
• • 尽管LSTM 往往不会遇到梯度消失问题，但它们可能会出现梯度爆炸。 因此，当梯度范数超过阈值时，我们通过缩放梯度 [10, 25] 来对梯度范数施加硬约束。 对于每个训练批次，我们计算$s=\left\|g\right\|_{2}$，其中$g$是除以 128 的梯度。如果是$s>5$，我们设置$g=\frac{5g}{s}$。
• • 不同的句子有不同的长度。 大多数句子都很短（例如，长度 20-30），但有些句子很长（例如，长度$>$100），因此由 128 个随机选择的训练句子组成的小批量将有很多短句子和很少的长句子，因此，小批量中的大部分计算都被浪费了。 为了解决这个问题，我们确保小批量中的所有句子的长度大致相同，从而实现 2 倍的加速。

3.5 Parallelization

3.5 并行化

A C++ implementation of deep LSTM with the configuration from the previous section on a single GPU processes a speed of approximately 1,700 words per second. This was too slow for our purposes, so we parallelized our model using an 8-GPU machine. Each layer of the LSTM was executed on a different GPU and communicated its activations to the next GPU / layer as soon as they were computed. Our models have 4 layers of LSTMs, each of which resides on a separate GPU. The remaining 4 GPUs were used to parallelize the softmax, so each GPU was responsible for multiplying by a $1000\times 20000$ matrix. The resulting implementation achieved a speed of 6,300 (both English and French) words per second with a minibatch size of 128. Training took about a ten days with this implementation.

深度 LSTM 的 C++ 实现（采用上一节中的配置）在单个 GPU 上的处理速度约为每秒 1,700 个字。 这对于我们的目的来说太慢了，因此我们使用 8-GPU 机器并行化我们的模型。 LSTM 的每一层都在不同的 GPU 上执行，并在计算后立即将其激活传递给下一个 GPU/层。 我们的模型有 4 层 LSTM，每层都驻留在单独的 GPU 上。 其余 4 个 GPU 用于并行化 softmax，因此每个 GPU 负责乘以$1000\times 20000$矩阵。 最终的实施实现了每秒 6,300 个单词（英语和法语）的速度，小批量大小为 128。此实施的培训大约需要十天。

3.6 Experimental Results

3.6 实验结果

We used the cased BLEU score [24] to evaluate the quality of our translations. We computed our BLEU scores using multi-bleu.pl[^1] on the tokenized predictions and ground truth. This way of evaluating the BELU score is consistent with [5] and [2], and reproduces the 33.3 score of [29]. However, if we evaluate the best WMT’14 system [9] (whose predictions can be downloaded from statmt.org\matrix) in this manner, we get 37.0, which is greater than the 35.8 reported by statmt.org\matrix.

我们使用 BLEU 评分 [24] 来评估我们翻译的质量。 我们使用 multi-bleu.pl[^1] 根据标记化预测和地面实况计算 BLEU 分数。 这种评估BELU分数的方式与[5]和[2]一致，并再现了[29]的33.3分数。 然而，如果我们以这种方式评估最好的 WMT’14 系统 [9]（其预测可以从 statmt.org\matrix 下载），我们得到 37.0，这大于 statmt.org\matrix 报告的 35.8。

The results are presented in tables 1 and 2. Our best results are obtained with an ensemble of LSTMs that differ in their random initializations and in the random order of minibatches. While the decoded translations of the LSTM ensemble do not outperform the best WMT’14 system, it is the first time that a pure neural translation system outperforms a phrase-based SMT baseline on a large scale MT task by a sizeable margin, despite its inability to handle out-of-vocabulary words. The LSTM is within 0.5 BLEU points of the best WMT’14 result if it is used to rescore the 1000-best list of the baseline system.

结果如表 1 和表 2 所示。我们的最佳结果是通过一组 LSTM 获得的，这些 LSTM 的随机初始化和小批量的随机顺序有所不同。 虽然 LSTM 集成的解码翻译并没有优于最好的 WMT’14 系统，但纯神经翻译系统首次在大规模 MT 任务上大幅优于基于短语的 SMT 基线，尽管它无法处理词汇表之外的单词。 如果使用 LSTM 对基线系统的 1000 个最佳列表进行重新评分，则 LSTM 与最佳 WMT’14 结果的 BLEU 点相差不超过 0.5 分。

Method	test BLEU score (ntst14)
Bahdanau et al. [2]	28.45
Baseline System [29]	33.30
Single forward LSTM, beam size 12	26.17
Single reversed LSTM, beam size 12	30.59
Ensemble of 5 reversed LSTMs, beam size 1	33.00
Ensemble of 2 reversed LSTMs, beam size 12	33.27
Ensemble of 5 reversed LSTMs, beam size 2	34.50
Ensemble of 5 reversed LSTMs, beam size 12	34.81

Table 1: The performance of the LSTM on WMT’14 English to French test set (ntst14). Note that an ensemble of 5 LSTMs with a beam of size 2 is cheaper than of a single LSTM with a beam of size 12.

方法	测试 BLEU（ntst14）
Bahdanau et al. [2]	28.45
基线系统 [29]	33.30
单个正向 LSTM，束宽 12	26.17
单个反序 LSTM，束宽 12	30.59
5 个反序 LSTM 集成，束宽 1	33.00
2 个反序 LSTM 集成，束宽 12	33.27
5 个反序 LSTM 集成，束宽 2	34.50
5 个反序 LSTM 集成，束宽 12	34.81

表 1：LSTM 在 WMT’14 英译法测试集（ntst14）上的表现。

Method	test BLEU score (ntst14)
Baseline System [29]	33.30
Cho et al. [5]	34.54
Best WMT’14 result [9]	37.0
Rescoring the baseline 1000-best with a single forward LSTM	35.61
Rescoring the baseline 1000-best with a single reversed LSTM	35.85
Rescoring the baseline 1000-best with an ensemble of 5 reversed LSTMs	36.5
Oracle Rescoring of the Baseline 1000-best lists	$\sim$45

Table 2: Methods that use neural networks together with an SMT system on the WMT’14 English to French test set (ntst14).

方法	测试 BLEU（ntst14）
基线系统 [29]	33.30
Cho et al. [5]	34.54
WMT’14 最佳结果 [9]	37.0
用单个正向 LSTM 重打基线 1000-best	35.61
用单个反序 LSTM 重打基线 1000-best	35.85
用 5 个反序 LSTM 集成重打基线 1000-best	36.5
对基线 1000-best 的 Oracle 重打分	$\sim$45

表 2：在 WMT’14 英译法测试集（ntst14）上，神经网络与 SMT 系统结合的方法。

3.7 Performance on long sentences

3.7 长句表现

We were surprised to discover that the LSTM did well on long sentences, which is shown quantitatively in figure 3. Table 3 presents several examples of long sentences and their translations.

我们惊讶地发现 LSTM 在长句子上表现良好，如图 3 定量显示。表 3 展示了长句子及其翻译的几个示例。

Type	Sentence
Our model	Ulrich UNK , membre du conseil d’ administration du constructeur automobile Audi ,
	affirme qu’ il s’ agit d’ une pratique courante depuis des années pour que les téléphones
	portables puissent être collectés avant les réunions du conseil d’ administration afin qu’ ils
	ne soient pas utilisés comme appareils d’ écoute à distance .
Truth	Ulrich Hackenberg , membre du conseil d’ administration du constructeur automobile Audi ,
	déclare que la collecte des téléphones portables avant les réunions du conseil , afin qu’ ils
	ne puissent pas être utilisés comme appareils d’ écoute à distance , est une pratique courante
	depuis des années .
Our model	“ Les téléphones cellulaires , qui sont vraiment une question , non seulement parce qu’ ils
	pourraient potentiellement causer des interférences avec les appareils de navigation , mais
	nous savons , selon la FCC , qu’ ils pourraient interférer avec les tours de téléphone cellulaire
	lorsqu’ ils sont dans l’ air ” , dit UNK .
Truth	“ Les téléphones portables sont véritablement un problème , non seulement parce qu’ ils
	pourraient éventuellement créer des interférences avec les instruments de navigation , mais
	parce que nous savons , d’ après la FCC , qu’ ils pourraient perturber les antennes-relais de
	téléphonie mobile s’ ils sont utilisés à bord ” , a déclaré Rosenker .
Our model	Avec la crémation , il y a un “ sentiment de violence contre le corps d’ un être cher ” ,
	qui sera “ réduit à une pile de cendres ” en très peu de temps au lieu d’ un processus de
	décomposition “ qui accompagnera les étapes du deuil ” .
Truth	Il y a , avec la crémation , “ une violence faite au corps aimé ” ,
	qui va être “ réduit à un tas de cendres ” en très peu de temps , et non après un processus de
	décomposition , qui “ accompagnerait les phases du deuil ” .

Table 3: A few examples of long translations produced by the LSTM alongside the ground truth translations. The reader can verify that the translations are sensible using Google translate.

类型	句子
我们的模型	Ulrich UNK，奥迪汽车制造商行政管理委员会成员，
	确认电话年报的实际操作情况
	便携式设备收集先进的管理委员会的聚会
	远距离使用通用服装。
真相	Ulrich Hackenberg，奥迪汽车制造商行政管理委员会成员，
	声明收集便携式电话，并在会议上召开会议，以供参考
	远距离使用通用服装是一种实用的方法
	旧事重提。
我们的模型	“ 手机电话，这是一个问题，不可解决问题”
	导航设备干扰的潜在原因
	请注意，请注意 FCC，请注意手机移动电话的干扰
	lorsqu'ils sont dans l'air”，dit UNK。
真相	“ 便携式电话确实存在问题，无法解决问题
	导航仪器干扰的主要活动
	FCC 后，我们的 savons 公园，这对天线的干扰
	罗森克宣称，移动电话是在边界上使用的。
我们的模型	火葬结束后，这是一种“对军团的暴力情绪”，
	qui sera “ réduit à une一堆 de cendres ” en très peu de temps au lieu d’un processus de
	分解“qui accompagnera les étapes du deuil”。
真相	Il y a , avec la crémation , “une Vieite au corps aimé ” ,
	qui va être “réduit à un tas de cendres ” en très peu de temps , et non après un processus de
	分解，即“accompagnerait les Phases du deuil”。

表 3：LSTM 生成的长翻译以及真实翻译的一些示例。 读者可以使用谷歌翻译来验证翻译是否合理。

3.8 Model Analysis

3.8 模型分析

Figure 2: The figure shows a 2-dimensional PCA projection of the LSTM hidden states that are obtained after processing the phrases in the figures. The phrases are clustered by meaning, which in these examples is primarily a function of word order, which would be difficult to capture with a bag-of-words model. Notice that both clusters have similar internal structure.

图 2：该图显示了处理图中的短语后获得的 LSTM 隐藏状态的二维 PCA 投影。 这些短语按含义进行聚类，在这些示例中，含义主要是词序的函数，而用词袋模型很难捕获词序。 请注意，两个簇具有相似的内部结构。

Figure 3: The left plot shows the performance of our system as a function of sentence length, where the x-axis corresponds to the test sentences sorted by their length and is marked by the actual sequence lengths. There is no degradation on sentences with less than 35 words, there is only a minor degradation on the longest sentences. The right plot shows the LSTM’s performance on sentences with progressively more rare words, where the x-axis corresponds to the test sentences sorted by their “average word frequency rank”.

图 3：左图显示了我们系统的性能与句子长度的函数关系，其中 x 轴对应于按长度排序的测试句子，并由实际序列长度标记。 少于 35 个单词的句子没有降级，最长的句子只有轻微降级。 右图显示了 LSTM 在稀有词逐渐增多的句子上的表现，其中 x 轴对应于按“平均词频排名”排序的测试句子。

One of the attractive features of our model is its ability to turn a sequence of words into a vector of fixed dimensionality. Figure 2 visualizes some of the learned representations. The figure clearly shows that the representations are sensitive to the order of words, while being fairly insensitive to the replacement of an active voice with a passive voice. The two-dimensional projections are obtained using PCA.

我们模型的一个吸引人的特征是它能够将单词序列转换为固定维度的向量。 图 2 可视化了一些学习到的表示形式。 该图清楚地表明，这些表示对单词的顺序很敏感，而对用被动语态替换主动语态相当不敏感。 二维投影是使用 PCA 获得的。

4 Related work

4 相关工作

There is a large body of work on applications of neural networks to machine translation. So far, the simplest and most effective way of applying an RNN-Language Model (RNNLM) [23] or a Feedforward Neural Network Language Model (NNLM) [3] to an MT task is by rescoring the n-best lists of a strong MT baseline [22], which reliably improves translation quality.

关于神经网络在机器翻译中的应用有大量的工作。 到目前为止，将 RNN 语言模型 (RNNLM) [23] 或前馈神经网络语言模型 (NNLM) [3] 应用到 MT 任务的最简单、最有效的方法是重新评分强 MT 基线 [22] 的 n 个最佳列表，这可靠地提高了翻译质量。

More recently, researchers have begun to look into ways of including information about the source language into the NNLM. Examples of this work include Auli et al. [1], who combine an NNLM with a topic model of the input sentence, which improves rescoring performance. Devlin et al. [8] followed a similar approach, but they incorporated their NNLM into the decoder of an MT system and used the decoder’s alignment information to provide the NNLM with the most useful words in the input sentence. Their approach was highly successful and it achieved large improvements over their baseline.

最近，研究人员开始研究将源语言信息纳入 NNLM 的方法。 这项工作的例子包括 Auli 等人。 [1]，将 NNLM 与输入句子的主题模型结合起来，提高了重新评分性能。 德夫林等人。 [8] 采用了类似的方法，但他们将 NNLM 合并到 MT 系统的解码器中，并使用解码器的对齐信息为 NNLM 提供输入句子中最有用的单词。 他们的方法非常成功，并且比他们的基线取得了巨大的进步。

Our work is closely related to Kalchbrenner and Blunsom [18], who were the first to map the input sentence into a vector and then back to a sentence, although they map sentences to vectors using convolutional neural networks, which lose the ordering of the words. Similarly to this work, Cho et al. [5] used an LSTM-like RNN architecture to map sentences into vectors and back, although their primary focus was on integrating their neural network into an SMT system. Bahdanau et al. [2] also attempted direct translations with a neural network that used an attention mechanism to overcome the poor performance on long sentences experienced by Cho et al. [5] and achieved encouraging results. Likewise, Pouget-Abadie et al. [26] attempted to address the memory problem of Cho et al. [5] by translating pieces of the source sentence in way that produces smooth translations, which is similar to a phrase-based approach. We suspect that they could achieve similar improvements by simply training their networks on reversed source sentences.

我们的工作与 Kalchbrenner 和 Blunsom [18] 密切相关，他们是第一个将输入句子映射到向量，然后再映射回句子的人，尽管他们使用卷积神经网络将句子映射到向量，这会丢失单词的顺序。 与这项工作类似，Cho 等人。 [5] 使用类似 LSTM 的 RNN 架构将句子映射到向量并映射回来，尽管他们的主要重点是将神经网络集成到 SMT 系统中。 巴达瑙等人。 [2] 还尝试使用神经网络进行直接翻译，该神经网络使用注意力机制来克服 Cho 等人在长句子上表现不佳的问题。 [5]并取得了令人鼓舞的成果。 同样，Pouget-Abadie 等人。 [26] 试图解决 Cho 等人的记忆问题。 [5] 通过以产生平滑翻译的方式翻译源句子的片段，这类似于基于短语的方法。 我们怀疑他们可以通过简单地在反向源句子上训练他们的网络来实现类似的改进。

End-to-end training is also the focus of Hermann et al. [12], whose model represents the inputs and outputs by feedforward networks, and map them to similar points in space. However, their approach cannot generate translations directly: to get a translation, they need to do a look up for closest vector in the pre-computed database of sentences, or to rescore a sentence.

端到端训练也是 Hermann 等人的重点。 [12]，其模型表示前馈网络的输入和输出，并将它们映射到空间中的相似点。 然而，他们的方法无法直接生成翻译：为了获得翻译，他们需要在预先计算的句子数据库中查找最接近的向量，或者重新评分句子。

5 Conclusion

5 结论

In this work, we showed that a large deep LSTM, that has a limited vocabulary and that makes almost no assumption about problem structure can outperform a standard SMT-based system whose vocabulary is unlimited on a large-scale MT task. The success of our simple LSTM-based approach on MT suggests that it should do well on many other sequence learning problems, provided they have enough training data.

在这项工作中，我们表明，词汇量有限且几乎不对问题结构做出任何假设的大型深度 LSTM 可以胜过基于标准 SMT 的系统（其词汇量在大规模 MT 任务中是无限的）。 我们基于 LSTM 的简单方法在 MT 上的成功表明，只要有足够的训练数据，它应该可以在许多其他序列学习问题上表现良好。

We were surprised by the extent of the improvement obtained by reversing the words in the source sentences. We conclude that it is important to find a problem encoding that has the greatest number of short term dependencies, as they make the learning problem much simpler. In particular, while we were unable to train a standard RNN on the non-reversed translation problem (shown in fig. 1), we believe that a standard RNN should be easily trainable when the source sentences are reversed (although we did not verify it experimentally).

我们对通过颠倒源句子中的单词所获得的改进程度感到惊讶。 我们的结论是，找到具有最多短期依赖性的问题编码非常重要，因为它们使学习问题变得更加简单。 特别是，虽然我们无法在非反向翻译问题上训练标准 RNN（如图 1 所示），但我们相信，当源句子反向时，标准 RNN 应该很容易训练（尽管我们没有通过实验验证）。

We were also surprised by the ability of the LSTM to correctly translate very long sentences. We were initially convinced that the LSTM would fail on long sentences due to its limited memory, and other researchers reported poor performance on long sentences with a model similar to ours [5, 2, 26]. And yet, LSTMs trained on the reversed dataset had little difficulty translating long sentences.

我们还对 LSTM 正确翻译很长句子的能力感到惊讶。 我们最初确信 LSTM 由于其有限的内存而无法处理长句子，而其他研究人员报告称，使用与我们类似的模型在长句子上表现不佳 [5,2,26]。 然而，在反向数据集上训练的 LSTM 翻译长句子几乎没有困难。

Most importantly, we demonstrated that a simple, straightforward and a relatively unoptimized approach can outperform an SMT system, so further work will likely lead to even greater translation accuracies. These results suggest that our approach will likely do well on other challenging sequence to sequence problems.

最重要的是，我们证明了一种简单、直接且相对未优化的方法可以优于 SMT 系统，因此进一步的工作可能会带来更高的翻译准确性。 这些结果表明我们的方法可能会在其他具有挑战性的序列到序列问题上表现良好。

6 Acknowledgments

6 致谢

We thank Samy Bengio, Jeff Dean, Matthieu Devin, Geoffrey Hinton, Nal Kalchbrenner, Thang Luong, Wolfgang Macherey, Rajat Monga, Vincent Vanhoucke, Peng Xu, Wojciech Zaremba, and the Google Brain team for useful comments and discussions.

我们感谢 Samy Bengio、Jeff Dean、Matthieu Devin、Geoffrey Hinton、Nal Kalchbrenner、Thang Luong、Wolfgang Macherey、Rajat Monga、Vincent Vanhoucke、Peng Xu、Wojciech Zaremba 以及 Google Brain 团队提供的有用评论和讨论。

References

参考文献

[1] M. Auli, M. Galley, C. Quirk, and G. Zweig. Joint language and translation modeling with recurrent neural networks. In EMNLP , 2013.

[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014.

[3] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. A neural probabilistic language model. In Journal of Machine Learning Research , pages 1137–1155, 2003.

[4] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks , 5(2):157–166, 1994.

[5] K. Cho, B. Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Arxiv preprint arXiv:1406.1078 , 2014.

[6] D. Ciresan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In CVPR , 2012.

[7] G. E. Dahl, D. Yu, L. Deng, and A. Acero. Context-dependent pre-trained deep neural networks for large vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing - Special Issue on Deep Learning for Speech and Language Processing , 2012.

[8] J. Devlin, R. Zbib, Z. Huang, T. Lamar, R. Schwartz, and J. Makhoul. Fast and robust neural network joint models for statistical machine translation. In ACL , 2014.

[9] Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. Edinburgh’s phrase-based machine translation systems for wmt-14. In WMT , 2014.

[10] A. Graves. Generating sequences with recurrent neural networks. In Arxiv preprint arXiv:1308.0850 , 2013.

[11] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML , 2006.

[12] K. M. Hermann and P. Blunsom. Multilingual distributed representations without word alignment. In ICLR , 2014.

[13] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine , 2012.

[14] S. Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Master’s thesis, Institut fur Informatik, Technische Universitat, Munchen , 1991.

[15] S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.

[16] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation , 1997.

[17] S. Hochreiter and J. Schmidhuber. LSTM can solve hard long time lag problems. 1997.

[18] N. Kalchbrenner and P. Blunsom. Recurrent continuous translation models. In EMNLP , 2013.

[19] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS , 2012.

[20] Q.V. Le, M.A. Ranzato, R. Monga, M. Devin, K. Chen, G.S. Corrado, J. Dean, and A.Y. Ng. Building high-level features using large scale unsupervised learning. In ICML , 2012.

[21] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998.

[22] T. Mikolov. Statistical Language Models based on Neural Networks . PhD thesis, Brno University of Technology, 2012.

[23] T. Mikolov, M. Karafiát, L. Burget, J. Cernockỳ, and S. Khudanpur. Recurrent neural network based language model. In INTERSPEECH , pages 1045–1048, 2010.

[24] K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. BLEU: a method for automatic evaluation of machine translation. In ACL , 2002.

[25] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063 , 2012.

[26] J. Pouget-Abadie, D. Bahdanau, B. van Merrienboer, K. Cho, and Y. Bengio. Overcoming the curse of sentence length for neural machine translation using automatic segmentation. arXiv preprint arXiv:1409.1257 , 2014.

[27] A. Razborov. On small depth threshold circuits. In Proc. 3rd Scandinavian Workshop on Algorithm Theory , 1992.

[28] D. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature , 323(6088):533–536, 1986.

[29] H. Schwenk. University le mans. http://www-lium.univ-lemans.fr/~schwenk/cslm_joint_paper/ , 2014. [Online; accessed 03-September-2014].

[30] M. Sundermeyer, R. Schluter, and H. Ney. LSTM neural networks for language modeling. In INTERSPEECH , 2010.

[31] P. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of IEEE , 1990.