Listen, Attend and Spell

Listen, Attend and Spell

William Chan Navdeep Jaitly, Quoc V. Le, Oriol Vinyals
Carnegie Mellon University Google Brain
williamchan@cmu.edu {ndjaitly,qvl,vinyals}@google.com
arXiv:1508.01211v2 [cs.CL] 20 Aug 2015

Abstract

摘要

We present Listen, Attend and Spell (LAS), a neural network that learns to tran-
scribe speech utterances to characters. Unlike traditional DNN-HMM models, this
model learns all the components of a speech recognizer jointly. Our system has
two components: a listener and a speller. The listener is a pyramidal recurrent net-
work encoder that accepts filter bank spectra as inputs. The speller is an attention-
based recurrent network decoder that emits characters as outputs. The network
produces character sequences without making any independence assumptions be-
tween the characters. This is the key improvement of LAS over previous end-to-
end CTC models. On a subset of the Google voice search task, LAS achieves a
word error rate (WER) of 14.1% without a dictionary or a language model, and
10.3% with language model rescoring over the top 32 beams. By comparison, the
state-of-the-art CLDNN-HMM model achieves a WER of 8.0%.

我们提出 Listen, Attend and Spell（LAS），这是一个把语音话语转写成字符的神经网络。与传统 DNN-HMM 模型不同，该模型联合学习语音识别器的全部组件。系统有两部分：listener（听）与 speller（拼）。listener 是接受滤波器组频谱作为输入的金字塔循环网络编码器；speller 是基于注意力、以字符为输出的循环网络解码器。网络在生成字符序列时不对字符间作独立性假设——这是 LAS 相对先前端到端 CTC 模型的关键改进。在 Google 语音搜索任务的一个子集上，LAS 在无词典、无语言模型时词错误率（WER）为 14.1%；对 top 32 束做语言模型重打分后为 10.3%。对比之下，当时最先进的 CLDNN-HMM 模型 WER 为 8.0%。

1 Introduction
Deep Neural Networks (DNNs) have led to improvements in various components of speech recog-
nizers. They are commonly used in hybrid DNN-HMM speech recognition systems for acoustic
modeling [1, 2, 3, 4, 5, 6]. DNNs have also produced significant gains in pronunciation models that
map words to phoneme sequences [7, 8]. In language modeling, recurrent models have been shown
to improve speech recognition accuracy by rescoring n-best lists [9]. Traditionally these compo-
nents – acoustic, pronunciation and language models – have all been trained separately, each with a
different objective. Recent work in this area attempts to rectify this disjoint training issue by design-
ing models that are trained end-to-end – from speech directly to transcripts [10, 11, 12, 13, 14, 15].
Two main approaches for this are Connectionist Temporal Classification (CTC) [10] and sequence
to sequence models with attention [16]. Both of these approaches have limitations that we try to
address: CTC assumes that the label outputs are conditionally independent of each other; whereas
the sequence to sequence approach has only been applied to phoneme sequences [14, 15], and not
trained end-to-end for speech recognition.
In this paper we introduce Listen, Attend and Spell (LAS), a neural network that improves upon the
previous attempts [12, 14, 15]. The network learns to transcribe an audio sequence signal to a word
sequence, one character at a time. Unlike previous approaches, LAS does not make independence
assumptions in the label sequence and it does not rely on HMMs. LAS is based on the sequence to
sequence learning framework with attention [17, 18, 16, 14, 15]. It consists of an encoder recurrent
neural network (RNN), which is named the listener, and a decoder RNN, which is named the speller.
The listener is a pyramidal RNN that converts low level speech signals into higher level features.
The speller is an RNN that converts these higher level features into output utterances by specifying
a probability distribution over sequences of characters using the attention mechanism [16, 14, 15].
The listener and the speller are trained jointly.
Key to our approach is the fact that we use a pyramidal RNN model for the listener, which reduces
the number of time steps that the attention model has to extract relevant information from. Rare and
out-of-vocabulary (OOV) words are handled automatically, since the model outputs the character

1 引言
深度神经网络（DNN）已改进语音识别器的多个组件。它们常用于混合 DNN-HMM 系统中的声学建模 [1, 2, 3, 4, 5, 6]。DNN 在把词映射到音素序列的发音模型上也带来显著收益 [7, 8]。在语言建模中，循环模型通过对 n-best 列表重打分提升识别精度 [9]。传统上，声学、发音与语言模型各自用不同目标分开训练。近期工作试图纠正这种割裂训练，设计从语音直接到转写的端到端模型 [10, 11, 12, 13, 14, 15]。两条主线是连接时序分类（CTC）[10] 与带注意力的序列到序列模型 [16]。二者都有局限，我们试图补上：CTC 假定标签输出条件独立；而序列到序列方法此前只用于音素序列 [14, 15]，尚未端到端训练完整语音识别。
本文提出 Listen, Attend and Spell（LAS），在先前尝试 [12, 14, 15] 上改进。网络把音频序列信号转写成词序列，一次一个字符。与先前方法不同，LAS 不对标签序列作独立性假设，也不依赖 HMM。LAS 基于带注意力的序列到序列学习框架 [17, 18, 16, 14, 15]，由名为 listener 的编码器 RNN 与名为 speller 的解码器 RNN 组成。listener 是把低层语音信号变成高层特征的金字塔 RNN；speller 借助注意力机制 [16, 14, 15]，在字符序列上给出概率分布。二者联合训练。
方法的关键是 listener 使用金字塔 RNN，从而减少注意力模型需要从中抽取信息的时间步数。稀有词与未登录词（OOV）自动处理，因为模型一次输出一个字符。

1
sequence, one character at a time. Another advantage of modeling characters as outputs is that the
network is able to generate multiple spelling variants naturally. For example, for the phrase “triple a”
the model produces both “triple a” and “aaa” in the top beams (see section 4.5). A model like CTC
may have trouble producing such diverse transcripts for the same utterance because of conditional
independence assumptions between frames.
In our experiments, we find that these components are necessary for LAS to work well. Without the
attention mechanism, the model overfits the training data significantly, in spite of our large training
set of three million utterances - it memorizes the training transcripts without paying attention to the
acoustics. Without the pyramid structure in the encoder side, our model converges too slowly - even
after a month of training, the error rates were significantly higher than the errors we report here.
Both of these problems arise because the acoustic signals can have hundreds to thousands of frames
which makes it difficult to train the RNNs. Finally, to reduce the overfitting of the speller to the
training transcripts, we use a sampling trick during training [19].
With these improvements, LAS achieves 14.1% WER on a subset of the Google voice search task,
without a dictionary or a language model. When combined with language model rescoring, LAS
achieves 10.3% WER. By comparison, the Google state-of-the-art CLDNN-HMM system achieves
8.0% WER on the same data set [20].

另一好处是以字符为输出时，网络能自然生成多种拼写变体。例如对短语 “triple a”，模型在 top 束中既产出 “triple a” 也产出 “aaa”（见第 4.5 节）。CTC 这类模型因帧间条件独立假设，可能难以对同一话语给出如此多样的转写。
实验中我们发现这些组件对 LAS 正常工作必不可少。没有注意力机制时，即便有三百万条话语的大规模训练集，模型仍严重过拟合——死记训练转写而不听声学。编码器侧没有金字塔结构时，收敛极慢：即便训练一个月，错误率仍显著高于本文报告。两难都源于声学信号可达数百到数千帧，RNN 难训。最后，为减轻 speller 对训练转写的过拟合，训练时采用采样技巧 [19]。
有了这些改进，LAS 在 Google 语音搜索任务子集上达到 14.1% WER（无词典、无语言模型）；结合语言模型重打分后为 10.3%。对比之下，Google 当时最先进的 CLDNN-HMM 系统在同一数据上为 8.0% WER [20]。

2 Related Work

2 相关工作

Even though deep networks have been successfully used in many applications, until recently, they
have mainly been used in classification: mapping a fixed-length vector to an output category [21].
For structured problems, such as mapping one variable-length sequence to another variable-length
sequence, neural networks have to be combined with other sequential models such as Hidden
Markov Models (HMMs) [22] and Conditional Random Fields (CRFs) [23]. A drawback of this
combining approach is that the resulting models cannot be easily trained end-to-end and they make
simplistic assumptions about the probability distribution of the data.
Sequence to sequence learning is a framework that attempts to address the problem of learning
variable-length input and output sequences [17]. It uses an encoder RNN to map the sequential
variable-length input into a fixed-length vector. A decoder RNN then uses this vector to produce
the variable-length output sequence, one token at a time. During training, the model feeds the
groundtruth labels as inputs to the decoder. During inference, the model performs a beam search to
generate suitable candidates for next step predictions.
Sequence to sequence models can be improved significantly by the use of an attention mechanism
that provides the decoder RNN more information when it produces the output tokens [16]. At each
output step, the last hidden state of the decoder RNN is used to generate an attention vector over
the input sequence of the encoder. The attention vector is used to propagate information from the
encoder to the decoder at every time step, instead of just once, as with the original sequence to
sequence model [17]. This attention vector can be thought of as skip connections that allow the
information and the gradients to flow more effectively in an RNN.
The sequence to sequence framework has been used extensively for many applications: machine
translation [24, 25], image captioning [26, 27], parsing [28] and conversational modeling [29]. The
generality of this framework suggests that speech recognition can also be a direct application [14,
15].

尽管深度网络已在许多应用中成功，直到不久前它们主要用于分类：把定长向量映射到类别 [21]。对把一条变长序列映到另一条变长序列这类结构化问题，神经网络须与隐马尔可夫模型（HMM）[22]、条件随机场（CRF）[23] 等序列模型结合。这种拼装的缺点是难以端到端训练，且对数据概率分布作过简假设。
序列到序列学习试图解决变长输入与输出的学习问题 [17]。它用编码器 RNN 把变长输入映成定长向量，再用解码器 RNN 一次一个词元地产生变长输出。训练时把真值标签喂给解码器；推理时用束搜索生成下一步候选。
加入注意力机制后，序列到序列模型可显著改进：解码器 RNN 在产生输出词元时能获得更多信息 [16]。每一步用解码器末隐状态在编码器输入序列上生成注意力向量，从而在每步（而非仅一次）从编码器向解码器传递信息，不同于原始序列到序列模型 [17]。该注意力向量可视为让信息与梯度在 RNN 中更有效流动的跳跃连接。
序列到序列框架已广泛用于机器翻译 [24, 25]、图像描述 [26, 27]、句法分析 [28] 与对话建模 [29]。其通用性提示语音识别也可直接作为应用 [14, 15]。

3 Model

3 模型

In this section, we will formally describe LAS which accepts acoustic features as in-
puts and emits English characters as outputs. Let x = (x1 , . . . , xT ) be our input se-
quence of filter bank spectra features, and let y = (hsosi, y1 , . . . , yS , heosi), yi ∈
{a, b, c, · · · , z, 0, · · · , 9, hspacei, hcommai, hperiodi, hapostrophei, hunki}, be the output se-
quence of characters. Here hsosi and heosi are the special start-of-sentence token, and end-of-
sentence tokens, respectively.

本节形式化描述 LAS：它接受声学特征为输入，输出英文字符。令 $x = (x_1, \ldots, x_T)$ 为滤波器组频谱特征输入序列，令 $y = (\langle\mathrm{sos}\rangle, y_1, \ldots, y_S, \langle\mathrm{eos}\rangle)$，$y_i \in \{a,b,c,\cdots,z,0,\cdots,9,\langle\mathrm{space}\rangle,\langle\mathrm{comma}\rangle,\langle\mathrm{period}\rangle,\langle\mathrm{apostrophe}\rangle,\langle\mathrm{unk}\rangle\}$ 为字符输出序列。其中 $\langle\mathrm{sos}\rangle$ 与 $\langle\mathrm{eos}\rangle$ 分别为句首与句末特殊词元。

2
We want to model each character output yi as a conditional distribution over the previous characters
y<i and the input signal x using the chain rule:
Y
P (y|x) = P (yi |x, y<i ) (1)
i

我们希望用链式法则，把每个字符输出 $y_i$ 建模为在先前字符 $y_{<i}$ 与输入信号 $x$ 上的条件分布：
$$P(y|x) = \prod_i P(y_i | x, y_{<i}) \tag{1}$$

Our Listen, Attend and Spell (LAS) model consists of two sub-modules: the listener and the speller.
The listener is an acoustic model encoder, whose key operation is Listen. The speller is an attention-
based character decoder, whose key operation is AttendAndSpell. The Listen function transforms
the original signal x into a high level representation h = (h1 , . . . , hU ) with U ≤ T , while the
AttendAndSpell function consumes h and produces a probability distribution over character se-
quences:
h = Listen(x) (2)
P (y|x) = AttendAndSpell(h, y) (3)

Listen, Attend and Spell（LAS）模型由两个子模块组成：listener 与 speller。listener 是声学模型编码器，关键操作为 Listen；speller 是基于注意力的字符解码器，关键操作为 AttendAndSpell。Listen 把原始信号 $x$ 变成高层表示 $h = (h_1, \ldots, h_U)$，$U \leq T$；AttendAndSpell 消费 $h$ 并在字符序列上给出概率分布：
$$h = \mathrm{Listen}(x) \tag{2}$$
$$P(y|x) = \mathrm{AttendAndSpell}(h, y) \tag{3}$$

Figure 1 visualizes LAS with these two components. We provide more details of these components
in the following sections.

图 1 可视化了带这两个组件的 LAS。下文给出更多细节。

Speller
y2 y3 y4 heosi Grapheme characters yi are
modelled by the
CharacterDistribution

Speller
$y_2$ $y_3$ $y_4$ $\langle\mathrm{eos}\rangle$ 字素字符 $y_i$ 由 CharacterDistribution 建模

c1 c2
AttentionContext creates
context vector ci from h
and si
h h h

$c_1$ $c_2$
AttentionContext 由 $h$ 与 $s_i$ 生成上下文向量 $c_i$
$h$ $h$ $h$

s1 s2

$s_1$ $s_2$

hsosi y2 y3 yS−1

$\langle\mathrm{sos}\rangle$ $y_2$ $y_3$ $y_{S-1}$

Long input sequence x is encoded with the pyramidal
h = (h1 , . . . , hU ) BLSTM Listen into shorter sequence h

长输入序列 $x$ 经金字塔 BLSTM Listen 编码为更短序列 $h = (h_1, \ldots, h_U)$

Listener
h1 h2 hU

Listener
$h_1$ $h_2$ $h_U$

x1 x2 x3 x4 x5 x6 x7 x8 xT

$x_1$ $x_2$ $x_3$ $x_4$ $x_5$ $x_6$ $x_7$ $x_8$ $x_T$

Figure 1: Listen, Attend and Spell (LAS) model: the listener is a pyramidal BLSTM encoding our input
sequence x into high level features h, the speller is an attention-based decoder generating the y characters
from h.

图 1：Listen, Attend and Spell（LAS）模型：listener 是把输入序列 $x$ 编成高层特征 $h$ 的金字塔 BLSTM；speller 是基于注意力、从 $h$ 生成字符 $y$ 的解码器。

3
3.1 Listen

3.1 Listen

The Listen operation uses a Bidirectional Long Short Term Memory RNN (BLSTM) [30, 31, 12]
with a pyramid structure. This modification is required to reduce the length U of h, from T , the
length of the input x, because the input speech signals can be hundreds to thousands of frames long.
A direct application of BLSTM for the operation Listen converged slowly and produced results
inferior to those reported here, even after a month of training time. This is presumably because the
operation AttendAndSpell has a hard time extracting the relevant information from a large number
of input time steps.
We circumvent this problem by using a pyramid BLSTM (pBLSTM) similar to the Clockwork RNN
[33]. In each successive stacked pBLSTM layer, we reduce the time resolution by a factor of 2. In a
typical deep BTLM architecture, the output at the i-th time step, from the j-th layer is computed as
follows:

Listen 操作使用带金字塔结构的双向长短期记忆 RNN（BLSTM）[30, 31, 12]。需要这一改动是为了把 $h$ 的长度 $U$ 相对输入 $x$ 的长度 $T$ 压短——输入语音信号可达数百到数千帧。若直接用 BLSTM 做 Listen，收敛很慢，即便训练一个月结果也劣于本文报告。推测是因为 AttendAndSpell 难以从大量时间步中抽取相关信息。
我们用类似 Clockwork RNN [33] 的金字塔 BLSTM（pBLSTM）绕开该问题。每个相继堆叠的 pBLSTM 层把时间分辨率减半。在典型深层 BTLM 架构中，第 $j$ 层第 $i$ 个时间步的输出计算如下：

hji = BLSTM(hji−1 , hij−1 ) (4)

$h^j_i = \mathrm{BLSTM}(h^{j}_{i-1}, h^{j-1}_i)$ (4)

In the pBLSTM model, we concatenate the outputs at consecutive steps of each layer before feeding
it to the next layer, i.e.:
h i
hji = pBLSTM(hji−1 , hj−1
2i , hj−1
2i+1 ) (5)

在 pBLSTM 中，我们把每层相邻步的输出拼接后再送入下一层，即：
$h^j_i = \mathrm{pBLSTM}(h^{j}_{i-1}, [h^{j-1}_{2i}, h^{j-1}_{2i+1}])$ (5)

In our model, we stack 3 pBLSTMs on top of the bottom BLSTM layer to reduce the time resolution
23 = 8 times. This allows the attention model (see next section) to extract the relevant information
from a smaller number of times steps. In addition to reducing the resolution, the deep architecture al-
lows the model to learn nonlinear feature representations of the data. See Figure 1 for a visualization
of the pBLSTM.
The pyramid structure also reduces the computational complexity. In the next section we show that
the attention mechanism over U features has a computational complexity of O(U S). Thus, reducing
U speeds up learning and inference significantly.

我们的模型在底层 BLSTM 之上堆叠 3 个 pBLSTM，把时间分辨率降为 $2^3 = 8$ 倍。这使注意力模型（见下节）可从更少时间步抽取相关信息。除降低分辨率外，深层结构也让模型学到数据的非线性特征表示。pBLSTM 可视化见图 1。
金字塔结构还降低计算复杂度。下节将说明，对 $U$ 个特征做注意力的复杂度为 $O(US)$。因此减小 $U$ 可显著加速学习与推理。

3.2 Attend and Spell

We now describe the AttendAndSpell function. The function is computed using an attention-based
LSTM transducer [16, 15]. At every output step, the transducer produces a probability distribution
over the next character conditioned on all the characters seen previously. The distribution for yi is
a function of the decoder state si and context ci . The decoder state si is a function of the previous
state si−1 , the previously emitted character yi−1 and context ci−1 . The context vector ci is produced
by an attention mechanism. Specifically,

下面描述 AttendAndSpell 函数。它由基于注意力的 LSTM transducer [16, 15] 计算。每一步，transducer 在已见全部先前字符的条件下，对下一字符给出概率分布。$y_i$ 的分布是解码器状态 $s_i$ 与上下文 $c_i$ 的函数。$s_i$ 依赖上一状态 $s_{i-1}$、上一发出字符 $y_{i-1}$ 与上一上下文 $c_{i-1}$。上下文向量 $c_i$ 由注意力机制产生。具体地：

ci = AttentionContext(si , h) (6)
si = RNN(si−1 , yi−1 , ci−1 ) (7)
P (yi |x, y<i ) = CharacterDistribution(si , ci ) (8)

$c_i = \mathrm{AttentionContext}(s_i, h)$ (6)
$s_i = \mathrm{RNN}(s_{i-1}, y_{i-1}, c_{i-1})$ (7)
$P(y_i | x, y_{<i}) = \mathrm{CharacterDistribution}(s_i, c_i)$ (8)

where CharacterDistribution is an MLP with softmax outputs over characters, and RNN is a 2
layer LSTM.
At each time step, i, the attention mechanism, AttentionContext generates a context vector, ci
encapsulating the information in the acoustic signal needed to generate the next character. The
attention model is content based - the contents of the decoder state si are matched to the contents
of hu representing time step u of h, to generate an attention vector αi . αi is used to linearly blend
vectors hu to create ci .
Specifically, at each decoder timestep i, the AttentionContext function computes the scalar energy
ei,u for each time step u, using vector hu ∈ h and si . The scalar energy ei,u is converted into a
probability distribution over times steps (or attention) αi using a softmax function. This is used to

其中 CharacterDistribution 是对字符做 softmax 的 MLP，RNN 是两层 LSTM。
在每一步 $i$，注意力机制 AttentionContext 生成上下文向量 $c_i$，封装生成下一字符所需的声学信息。注意力是基于内容的：用解码器状态 $s_i$ 的内容与表示 $h$ 中时间步 $u$ 的 $h_u$ 匹配，得到注意力向量 $\alpha_i$，再用 $\alpha_i$ 对向量 $h_u$ 做线性混合得到 $c_i$。
具体地，在每个解码时间步 $i$，AttentionContext 用向量 $h_u \in h$ 与 $s_i$ 为每个时间步 $u$ 计算标量能量 $e_{i,u}$。再经 softmax 把 $e_{i,u}$ 变成时间步上的概率分布（注意力）$\alpha_i$。由此：

4
create the context vector ci by linearly blending the listener features, hu , at different time steps:
ei,u = hφ(si ), ψ(hu )i (9)
exp(ei,u )
αi,u = P (10)
exp(ei,u )
Xu
ci = αi,u hu (11)
u

通过对不同时间步的 listener 特征 $h_u$ 做线性混合得到上下文向量 $c_i$：
$e_{i,u} = \langle \phi(s_i), \psi(h_u) \rangle$ (9)
$\alpha_{i,u} = \frac{\exp(e_{i,u})}{\sum_u \exp(e_{i,u})}$ (10)
$c_i = \sum_u \alpha_{i,u} h_u$ (11)

where φ and ψ are MLP networks. On convergence, the αi distribution is typically very sharp, and
focused on only a few frames of h; ci can be seen as a continuous bag of weighted features of h.
Figure 1 shows LAS architecture.

其中 $\phi$ 与 $\psi$ 是 MLP 网络。收敛后，$\alpha_i$ 分布通常非常尖锐，只聚焦于 $h$ 的少数几帧；$c_i$ 可视为 $h$ 的连续加权特征袋。图 1 展示了 LAS 架构。

3.3 Learning

3.3 学习

The Listen and AttendAndSpell functions can be trained jointly for end-to-end speech recognition.
The sequence to sequence methods condition the next step prediction on the previous characters [17,
16] and maximizes the log probability:
X
∗
max log P (yi |x, y<i ; θ) (12)
θ
i
∗
where y<i is the groundtruth of the previous characters.
However during inference, the groundtruth is missing and the predictions can suffer because the
model was not trained to be resilient to feeding in bad predictions at some time steps. To ameliorate
this effect, we use a trick that was proposed in [19]. During training, instead of always feeding in the
ground truth transcript for next step prediction, we sometimes sample from our previous character
distribution and use that as the inputs in the next step predictions:
ỹi ∼ CharacterDistribution(si , ci ) (13)
X
max log P (yi |x, ỹ<i ; θ) (14)
θ
i

Listen 与 AttendAndSpell 可联合训练，实现端到端语音识别。序列到序列方法把下一步预测条件于先前字符 [17, 16]，并最大化对数概率：
$$\max_\theta \sum_i \log P(y^*_i | x, y^*_{<i}; \theta) \tag{12}$$
其中 $y^*_{<i}$ 为先前字符的真值。
但推理时没有真值，若训练时从未见过错误的上一步预测，测试表现会受损。为缓解这一点，我们采用 [19] 提出的技巧：训练时不总是把真值转写喂给下一步预测，而是有时从上一字符分布采样，把采样结果作为下一步输入：
$$\tilde{y}_i \sim \mathrm{CharacterDistribution}(s_i, c_i) \tag{13}$$
$$\max_\theta \sum_i \log P(y_i | x, \tilde{y}_{<i}; \theta) \tag{14}$$

where ỹi−1 is the character chosen from the ground truth, or sampled from the model with a certain
sampling rate. Unlike [19], we do not use a schedule and simply use a constant sampling rate of
10% right from the start of training.
As the system is a very deep network it may appear that some type of pretraining would be required.
However, in our experiments, we found no need for pretraining. In particular, we attempted to
pretrain the Listen function with context independent or context dependent phonemes generated
from a conventional GMM-HMM system. A softmax network was attached to the output units
hu ∈ h of the listener and used to make multi-frame phoneme state predictions [34] but led to no
improvements. We also attempted to use the phonemes as a joint objective target [35], but found no
improvements.

其中 $\tilde{y}_{i-1}$ 以一定采样率来自真值或来自模型采样。与 [19] 不同，我们不用课程表，从训练一开始就用恒定 10% 采样率。
系统很深，看似需要某种预训练。但实验中我们发现不必预训练。具体地，我们曾尝试用传统 GMM-HMM 系统生成的上下文无关或上下文相关音素预训练 Listen 函数：在 listener 输出单元 $h_u \in h$ 上接 softmax，做多帧音素状态预测 [34]，但没有带来提升。我们也试过把音素作为联合目标 [35]，同样没有提升。

3.4 Decoding and Rescoring

3.4 解码与重打分

During inference we want to find the most likely character sequence given the input acoustics:
ŷ = arg max log P (y|x) (15)
y

推理时我们希望找给定输入声学下最可能的字符序列：
$$\hat{y} = \arg\max_y \log P(y|x) \tag{15}$$

Decoding is performed with a simple left-to-right beam search algorithm similar to [17]. We main-
tain a set of β partial hypotheses, starting with the start-of-sentence hsosi token. At each timestep,
each partial hypothesis in the beam is expanded with every possible character and only the β most
likely beams are kept. When the heosi token is encountered, it is removed from the beam and added
to the set of complete hypothesis. A dictionary can optionally be added to constrain the search space
to valid words, however we found that this was not necessary since the model learns to spell real
words almost all the time.
We have vast quantities of text data [36], compared to the amount of transcribed speech utterances.
We can use language models trained on text corpora alone similar to conventional speech systems

解码采用类似 [17] 的简单从左到右束搜索。我们维护一组大小为 $\beta$ 的部分假设，从句首词元 $\langle\mathrm{sos}\rangle$ 开始。每一步把束中每个部分假设用所有可能字符扩展，只保留最可能的 $\beta$ 条。遇到 $\langle\mathrm{eos}\rangle$ 时把它移出束并加入完整假设集合。可选地加入词典以把搜索限制在合法词上，但我们发现无此必要——模型几乎总会拼出真实单词。
相对已转写的语音话语量，我们有海量纯文本 [36]。可以像传统语音系统那样，用仅在文本语料上训练的语言模型

5
[37]. To do so we can rescore our beams with the language model. We find that our model has a
small bias for shorter utterances so we normalize our probabilities by the number of characters |y|c
in the hypothesis and combine it with a language model probability PLM (y):
log P (y|x)
s(y|x) = + λ log PLM (y) (16)
|y|c
where λ is our language model weight and can be determined by a held-out validation set.

对束做重打分 [37]。我们发现模型对较短话语略有偏好，于是用假设中的字符数 $|y|_c$ 归一化概率，再与语言模型概率 $P_{\mathrm{LM}}(y)$ 结合：
$$s(y|x) = \frac{\log P(y|x)}{|y|_c} + \lambda \log P_{\mathrm{LM}}(y) \tag{16}$$
其中 $\lambda$ 为语言模型权重，可由留出验证集确定。

4 Experiments
We used a dataset approximately three million Google voice search utterances (representing 2000
hours of data) for our experiments. Approximately 10 hours of utterances were randomly selected as
a held-out validation set. Data augmentation was performed using a room simulator, adding different
types of noise and reverberations; the noise sources were obtained from YouTube and environmental
recordings of daily events [20]. This increased the amount of audio data by 20 times. 40-dimensional
log-mel filter bank features were computed every 10ms and used as the acoustic inputs to the listener.
A separate set of 22K utterances representing approximately 16 hours of data were used as the test
data. A noisy test data set was also created using the same corruption strategy that was applied to
the training data. All training sets are anonymized and hand-transcribed, and are representative of
Googles speech traffic.
The text was normalized by converting all characters to lower case English alphanumerics (including
digits). The punctuations: space, comma, period and apostrophe were kept, while all other tokens
were converted to the unknown hunki token. As mentioned earlier, all utterances were padded with
the start-of-sentence hsosi and the end-of-sentence heosi tokens.
The state-of-the-art model on this dataset is a CLDNN-HMM system that was described in [20].
The CLDNN system achieves a WER of 8.0% on the clean test set and 8.9% on the noisy test set.
However, we note that the CLDNN uses unidirectional CLDNNs and would certainly benefit even
further from the use of a bidirectional CLDNN architecture.
For the Listen function we used 3 layers of 512 pBLSTM nodes (i.e., 256 nodes per direction) on
top of a BLSTM that operates on the input. This reduced the time resolution by 8 = 23 times. The
Spell function used a two layer LSTM with 512 nodes each. The weights were initialized with a
uniform distribution U(−0.1, 0.1).
Asynchronous Stochastic Gradient Descent (ASGD) was used for training our model [38]. A learn-
ing rate of 0.2 was used with a geometric decay of 0.98 per 3M utterances (i.e., 1/20-th of an epoch).
We used the DistBelief framework [38] with 32 replicas, each with a minibatch of 32 utterances.
In order to further speed up training, the sequences were grouped into buckets based on their frame
length [17].
The model was trained using groundtruth previous characters until results on the validation set
stopped improving. This took approximately two weeks. The model was decoded using beam width
β = 32 and achieved 16.2% WER on the clean test set and 19.0% WER on the noisy test set without
any dictionary or language model. We found that constraining the beam search with a dictionary had
no impact on the WER. Rescoring the top 32 beams with the same n-gram language model that was
used by the CLDNN system using a language model weight of λ = 0.008 improved the results for
the clean and noisy test sets to 12.6% and 14.7% respectively. Note that for convenience, we did not
decode with a language model, but rather only rescored the top 32 beams. It is possible that further
gains could have been achieved by using the language model during decoding.
As mentioned in Section 3.3, there is a mismatch between training and testing. During training
the model is conditioned on the correct previous characters but during testing mistakes made by
the model corrupt future predictions. We trained another model by sampling from our previous
character distribution with a probability of 10% (we did not use a schedule as described in [19]).
This improved our results on the clean and noisy test sets to 14.1% and 16.5% WER respectively
when no language model rescoring was used. With language model rescoring, we achevied 10.3%
and 12.0% WER on the clean and noisy test sets, respectively. Table 1 summarizes these results.
On the clean test set, this model is within 2.5% absolute WER of the state-of-the-art CLDNN-HMM
system, while on the noisy set it is less than 3.0% absolute WER worse. We suspect that convolu-

4 实验
实验使用约三百万条 Google 语音搜索话语（约 2000 小时）。随机留出约 10 小时作为验证集。用房间模拟器做数据增强，加入不同类型噪声与混响；噪声来自 YouTube 与日常环境录音 [20]，使音频数据量增至约 20 倍。每 10ms 计算 40 维 log-mel 滤波器组特征，作为 listener 的声学输入。另用约 2.2 万条话语（约 16 小时）作测试集，并用与训练相同的污染策略构造含噪测试集。全部训练集经匿名化与人工转写，代表 Google 语音流量。
文本归一化为小写英文数字字母（含数字）。保留空格、逗号、句号与撇号，其余词元转为未知词元 $\langle\mathrm{unk}\rangle$。如前所述，所有话语前后加上句首 $\langle\mathrm{sos}\rangle$ 与句末 $\langle\mathrm{eos}\rangle$。
该数据集上当时最先进模型是 [20] 描述的 CLDNN-HMM 系统：干净测试集 WER 8.0%，含噪测试集 8.9%。需注意该 CLDNN 为单向，若改双向架构还会进一步受益。
Listen 函数在输入上的 BLSTM 之上使用 3 层、每层 512 个 pBLSTM 节点（每方向 256），时间分辨率降为 $8=2^3$ 倍。Spell 函数为两层 LSTM，每层 512 节点。权重用均匀分布 $U(-0.1,0.1)$ 初始化。
训练用异步随机梯度下降（ASGD）[38]，学习率 0.2，每 300 万条话语（约 1/20 epoch）几何衰减 0.98。在 DistBelief 框架 [38] 上用 32 个副本，每个小批量 32 条话语。为进一步加速，按帧长把序列分桶 [17]。
模型先用真值先前字符训练，直到验证集不再提升，约两周。束宽 $\beta=32$ 解码时，干净测试集 16.2% WER、含噪 19.0% WER（无词典、无语言模型）。用词典约束束搜索对 WER 无影响。用与 CLDNN 相同的 n-gram 语言模型、权重 $\lambda=0.008$ 对 top 32 束重打分后，干净与含噪分别改进到 12.6% 与 14.7%。为方便起见我们未在解码期用语言模型，只重打分；若解码时也用 LM，可能还有增益。
如第 3.3 节所述，训练与测试存在错配：训练时条件于正确先前字符，测试时模型错误会污染后续预测。我们另训一模型，以 10% 概率从上一字符分布采样（不用 [19] 的课程表）。无 LM 重打分时，干净与含噪分别改进到 14.1% 与 16.5% WER；加上 LM 重打分后分别为 10.3% 与 12.0%。表 1 汇总这些结果。干净测试集上该模型与最先进 CLDNN-HMM 的绝对差距在 2.5% WER 内，含噪集上不到 3.0%。我们怀疑卷积滤波器还能进一步提升——相对非卷积架构，文献报告干净语音约相对改进 5% WER、含噪约 7% [20]。

6
Table 1: WER comparison on the clean and noisy Google voice search task. The CLDNN-HMM system is
the state-of-the-art system, the Listen, Attend and Spell (LAS) models are decoded with a beam size of 32.
Language Model (LM) rescoring was applied to our beams, and a sampling trick was applied to bridge the gap
between training and inference.

表 1：干净与含噪 Google 语音搜索任务上的 WER 比较。CLDNN-HMM 为当时最先进系统；Listen, Attend and Spell（LAS）模型以束宽 32 解码。对束施加语言模型（LM）重打分，并用采样技巧弥合训练与推理差距。

Model Clean WER Noisy WER
CLDNN-HMM [20] 8.0 8.9
LAS 16.2 19.0
LAS + LM Rescoring 12.6 14.7
LAS + Sampling 14.1 16.5
LAS + Sampling + LM Rescoring 10.3 12.0

模型 | 干净 WER | 含噪 WER
CLDNN-HMM [20] | 8.0 | 8.9
LAS | 16.2 | 19.0
LAS + LM 重打分 | 12.6 | 14.7
LAS + 采样 | 14.1 | 16.5
LAS + 采样 + LM 重打分 | 10.3 | 12.0

tional filters could lead to improved results, as they have been reported to improve performance by
5% relative WER on clean speech and 7% relative on noisy speech compared to non-convolutional
architectures [20].

（续上）卷积滤波器或可带来进一步改进，因相对非卷积架构，已有报告在干净语音上相对降低约 5% WER、在含噪语音上约 7% [20]。

4.1 Attention Visualization

4.1 注意力可视化

Figure 2: Alignments between character outputs and audio signal produced by the Listen, Attend and Spell
(LAS) model for the utterance “how much would a woodchuck chuck”. The content based attention mechanism
was able to identify the start position in the audio sequence for the first character correctly. The alignment
produced is generally monotonic without a need for any location based priors.

图 2：LAS 模型对话语 “how much would a woodchuck chuck” 产生的字符输出与音频信号对齐。基于内容的注意力机制能正确识别首字符在音频序列中的起始位置。对齐总体单调，无需基于位置的先验。

The content-based attention mechanism creates an explicit alignment between the characters and
audio signal. We can visualize the attention mechanism by recording the attention distribution on
the acoustic sequence at every character output timestep. Figure 2 visualizes the attention align-
ment between the characters and the filterbanks for the utterance “how much would a woodchuck
chuck”. For this particular utterance, the model learnt a monotonic distribution without any location
priors. The words “woodchuck” and “chuck” have acoustic similarities, the attention mechanism
was slightly confused when emitting “woodchuck” with a dilution in the distribution. The attention
model was also able to identify the start and end of the utterance properly.

基于内容的注意力在字符与音频信号之间建立显式对齐。可在每个字符输出时间步记录声学序列上的注意力分布来可视化。图 2 展示话语 “how much would a woodchuck chuck” 上字符与滤波器组之间的注意力对齐。对该话语，模型在无位置先验下学到了单调分布。“woodchuck” 与 “chuck” 声学相似，发出 “woodchuck” 时注意力略有分散。注意力模型也能正确识别话语起止。

7
In the following sections, we report results of control experiments that were conducted to understand
the effects of beam widths, utterance lengths and word frequency on the WER of our model.

以下各节报告控制实验，考察束宽、话语长度与词频对模型 WER 的影响。

4.2 Effects of Beam Width

4.2 束宽的影响

We investigate the correlation between the performance of the model and the width of beam search,
with and without the language model rescoring. Figure 3 shows the effect of the decode beam width,
β, on the WER for the clean test set. We see consistent WER improvements by increasing the beam
width up to 16, after which we observe no significant benefits. At a beam width of 32, the WER
is 14.1% and 10.3% after language model rescoring. Rescoring the top 32 beams with an oracle
produces a WER of 4.3% on the clean test set and 5.5% on the noisy test set.

我们考察有无语言模型重打分时，模型表现与束搜索宽度的关系。图 3 展示干净测试集上解码束宽 $\beta$ 对 WER 的影响。束宽增至 16 前 WER 持续改进，之后收益不明显。束宽 32 时 WER 为 14.1%，LM 重打分后 10.3%。对 top 32 束做 oracle 重打分，干净测试集 WER 4.3%，含噪 5.5%。

Beam Width vs. WER
20
WER
WER LM
WER Oracle
15
WER

束宽与 WER
纵轴为 WER；曲线含无 LM、LM 重打分与 oracle。

0
12 4 8 16 32
Beam Width
Figure 3: The effect of the decode beam width on WER for the clean Google voice search task. The reported
WERs are without a dictionary or language model, with language model rescoring and the oracle WER for
different beam widths. The figure shows that good results can be obtained even with a relatively small beam
size.

图 3：干净 Google 语音搜索任务上解码束宽对 WER 的影响。报告无词典/无 LM、LM 重打分以及不同束宽下的 oracle WER。图表明相对较小的束宽也能得到良好结果。

4.3 Effects of Utterance Length

4.3 话语长度的影响

We measure the performance of our model as a function of the number of words in the utterance. We
expect the model to do poorly on longer utterances due to limited number of long training utterances
in our distribution. Hence it is not surprising that longer utterances have a larger error rate. The
deletions dominate the error for long utterances, suggesting we may be missing out on words. It is
surprising that short utterances (e.g., 2 words or less) perform quite poorly. Here, the substitutions
and insertions are the main sources of errors, suggesting the model may split words apart.
Figure 4 also suggests that our model struggles to generalize to long utterances when trained on a
distribution of shorter utterances. It is possible location-based priors may help in these situations as
reported by [15].

我们按话语中词数衡量模型表现。由于训练分布中长话语有限，预期模型在更长话语上更差——事实也如此，更长话语错误率更高。长话语上删除错误占主导，提示可能漏词。出乎意料的是，很短话语（例如不超过 2 个词）表现也较差；此时替换与插入是主因，提示模型可能把词拆开。
图 4 也表明，在较短话语分布上训练时，模型难以泛化到长话语。如 [15] 所报，基于位置的先验在这些情形下或有帮助。

4.4 Word Frequency

4.4 词频

We study the performance of our model on rare words. We use the recall metric to indicate whether
a word appears in the utterance regardless of position (higher is better). Figure 5 reports the recall
of each word in the test distribution as a function of the word frequency in the training distribution.
Rare words have higher variance and lower recall while more frequent words typically have higher

我们研究模型在稀有词上的表现。用召回率表示某词是否出现在话语中（不论位置；越高越好）。图 5 报告测试分布中每个词的召回率相对其在训练分布中的词频。稀有词方差更大、召回更低，而更频繁的词通常召回更高。

8
 Utterance Length vs. Error
90
Data Distribution
80 Insertion
Deletion
Substitution
70 WER
WER LM
60 WER Oracle
Percentage

话语长度与误差
叠加数据分布；曲线含插入、删除、替换、WER、LM 重打分与 oracle。

50
40
30
20
10
0
5 10 15 20 25+
Number of Words in Utterance
Figure 4: The correlation between error rates (insertion, deletion, substitution and WER) and the number of
words in an utterance. The WER is reported without a dictionary or language model, with language model
rescoring and the oracle WER for the clean Google voice search task. The data distribution with respect to the
number of words in an utterance is overlaid in the figure. LAS performs poorly with short utterances despite
an abundance of data. LAS also fails to generalize well on longer utterances when trained on a distribution
of shorter utterances. Insertions and substitutions are the main sources of errors for short utterances, while
deletions dominate the error for long utterances.

图 4：错误率（插入、删除、替换与 WER）与话语词数的相关性。WER 报告无词典/无 LM、LM 重打分与 oracle（干净 Google 语音搜索）。图中叠加按词数的数据分布。尽管短话语数据充足，LAS 在其上仍较差；在较短话语分布上训练时，对更长话语也泛化不佳。短话语主错为插入与替换，长话语主错为删除。

recall. The word “and” occurs 85k times in the training set, however it has a recall of only 80% even
after language model rescoring. The word “and” is frequently mis-transcribed as “in” (which has
95% recall). This suggests improvements are needed in the language model. By contrast, the word
“walkerville” occurs just once in the training set but it has a recall of 100%. This suggests that the
recall for a word depends both on its frequency in the training set and its acoustic uniqueness.

词 “and” 在训练集出现 8.5 万次，即便 LM 重打分后召回也仅约 80%，常被误写成 “in”（召回 95%），提示语言模型仍需改进。相反，“walkerville” 在训练集只出现一次，召回却达 100%。说明某词召回既取决于训练频率，也取决于其声学独特性。

4.5 Interesting Decoding Examples

4.5 有趣的解码例子

In this section, we show the outputs of the model on several utterances to demonstrate the capabilities
of LAS. All the results in this section are decoded without a dictionary or a language model.
During our experiments, we observed that LAS can learn multiple spelling variants given the same
acoustics. Table 2 shows top beams for the utterance that includes “triple a”. As can be seen,
the model produces both “triple a” and “aaa” within the top four beams. The decoder is able to
generate such varied parses, because the next step prediction model makes no assumptions on the
probability distribution by using the chain rule decomposition. It would be difficult to produce such
differing transcripts using CTC due to the conditional independence assumptions, where p(yi |x)
is conditionally independent of p(yi+1 |x). Conventional DNN-HMM systems would require both
spellings to be in the pronunciation dictionary to generate both spelling permutations.
It can also be seen that the model produced “xxx” even though acoustically “x” is very different
from “a” - this is presumably because the language model overpowers the acoustic signal in this
case. In the training corpus “xxx” is a very common phrase and we suspect the language model

本节展示模型在若干话语上的输出，以说明 LAS 的能力。本节结果均在无词典、无语言模型下解码。
实验中我们观察到，给定相同声学时 LAS 能学到多种拼写变体。表 2 给出含 “triple a” 话语的 top 束：模型在前四束中既产出 “triple a” 也产出 “aaa”。解码器能生成如此多样的解析，是因为下一步预测模型用链式法则分解，不对概率分布作额外假设。CTC 因条件独立假设——$p(y_i|x)$ 条件独立于 $p(y_{i+1}|x)$——很难产出如此不同的转写。传统 DNN-HMM 系统则需把两种拼写都放进发音词典才能生成两种排列。
也可看到模型产出了 “xxx”，尽管声学上 “x” 与 “a” 很不同——推测此时语言模型压过了声学信号。训练集中 “xxx” 是很常见短语，我们怀疑

9
 Word Frequency vs. Recall
100
90
80
Word Recall Percentage 70
60
50
40
30
20
10
0
100 101 102 103 104 105 106
Word Frequency
Figure 5: The correlation between word frequency in the training distribution and recall in the test distribution.
In general, rare words report worse recall compared to more frequent words.

图 5：训练分布中的词频与测试分布中召回率的相关性。总体而言，稀有词召回低于更频繁的词。

Table 2: Example 1: “triple a” vs. “aaa” spelling variants.

表 2：例子 1：“triple a” 与 “aaa” 拼写变体。

Beam Text Log Probability WER
Truth call aaa roadside assistance - -
1 call aaa roadside assistance -0.5740 0.00
2 call triple a roadside assistance -1.5399 50.00
3 call trip way roadside assistance -3.5012 50.00
4 call xxx roadside assistance -4.4375 25.00

束 | 文本 | 对数概率 | WER
真值 | call aaa roadside assistance | - | -
1 | call aaa roadside assistance | -0.5740 | 0.00
2 | call triple a roadside assistance | -1.5399 | 50.00
3 | call trip way roadside assistance | -3.5012 | 50.00
4 | call xxx roadside assistance | -4.4375 | 25.00

implicit in the speller learns to associate “triple” with “xxx”. We note that “triple a” occurs 4 times
in the training distribution and “aaa” (when pronounced “triple a” rather than “a”-“a”-“a”) occurs
only once in the training distribution.
We are also surprised that the model is capable of handling utterances with repeated words despite
the fact that it uses content-based attention. Table 3 shows an example of an utterance with a repeated
word. Since LAS implements content-based attention, it is expected it to “lose its attention” during
the decoding steps and produce a word more or less times than the number of times the word was
spoken. As can be seen from this example, even though “seven” is repeated three times, the model
successfully outputs “seven” three times. This hints that location-based priors (e.g., location based
attention or location based regularization) may not be needed for repeated contents.

speller 中隐含的语言模型学会把 “triple” 与 “xxx” 关联。注意训练集中 “triple a” 出现 4 次，而按 “triple a” 而非 “a”-“a”-“a” 发音的 “aaa” 仅出现 1 次。
我们还惊讶地发现，尽管使用基于内容的注意力，模型仍能处理含重复词的话语。表 3 给出重复词例子。由于 LAS 实现基于内容的注意力，预期解码时可能“丢失注意力”，使某词出现次数多于或少于实际发音次数。本例中 “seven” 重复三次，模型仍成功输出三次。这提示对重复内容未必需要基于位置的先验（例如基于位置的注意力或正则）。

Table 3: Example 2: Repeated “seven”s.

表 3：例子 2：重复的 “seven”。

Beam Text Log Probability WER
Truth eight nine four minus seven seven seven - -
1 eight nine four minus seven seven seven -0.2145 0.00
2 eight nine four nine seven seven seven -1.9071 14.29
3 eight nine four minus seven seventy seven -4.7316 14.29
4 eight nine four nine s seven seven seven -5.1252 28.57

束 | 文本 | 对数概率 | WER
真值 | eight nine four minus seven seven seven | - | -
1 | eight nine four minus seven seven seven | -0.2145 | 0.00
2 | eight nine four nine seven seven seven | -1.9071 | 14.29
3 | eight nine four minus seven seventy seven | -4.7316 | 14.29
4 | eight nine four nine s seven seven seven | -5.1252 | 28.57

10
5 Conclusions

5 结论

We have presented Listen, Attend and Spell (LAS), an attention-based neural network that can di-
rectly transcribe acoustic signals to characters. LAS is based on the sequence to sequence framework
with a pyramid structure in the encoder that reduces the number of timesteps that the decoder has
to attend to. LAS is trained end-to-end and has two main components. The first component, the
listener, is a pyramidal acoustic RNN encoder that transforms the input sequence into a high level
feature representation. The second component, the speller, is an RNN decoder that attends to the
high level features and spells out the transcript one character at a time. Our system does not use
the concepts of phonemes, nor does it rely on pronunciation dictionaries or HMMs. We bypass the
conditional independence assumptions of CTC, and show how we can learn an implicit language
model that can generate multiple spelling variants given the same acoustics. To further improve
the results, we used samples from the softmax classifier in the decoder as inputs to the next step
prediction during training. Finally, we showed how a language model trained on additional text can
be used to rerank our top hypotheses.

我们提出了 Listen, Attend and Spell（LAS），这是一个能把声学信号直接转写成字符的基于注意力的神经网络。LAS 基于序列到序列框架，编码器采用金字塔结构以减少解码器需要关注的时间步。LAS 端到端训练，含两大组件：listener 是把输入序列变成高层特征表示的金字塔声学 RNN 编码器；speller 是关注这些高层特征并一次拼出一个字符的 RNN 解码器。系统不使用音素概念，也不依赖发音词典或 HMM。我们绕开 CTC 的条件独立假设，并展示如何学到隐含语言模型，从而在相同声学下生成多种拼写变体。为进一步改进结果，训练时把解码器 softmax 分类器的采样结果作为下一步预测的输入。最后，我们展示如何用额外文本训练的语言模型对 top 假设重排序。

Acknowledgements

致谢

We thank Tara Sainath, Babak Damavandi for helping us with the data, language models and for
helpful comments. We also thank Andrew Dai, Ashish Agarwal, Samy Bengio, Eugene Brevdo,
Greg Corrado, Andrew Dai, Jeff Dean, Rajat Monga, Christopher Olah, Mike Schuster, Noam
Shazeer, Ilya Sutskever, Vincent Vanhoucke and the Google Brain team for helpful comments, sug-
gestions and technical assistance.

感谢 Tara Sainath、Babak Damavandi 在数据、语言模型上的帮助与有益意见。也感谢 Andrew Dai、Ashish Agarwal、Samy Bengio、Eugene Brevdo、Greg Corrado、Andrew Dai、Jeff Dean、Rajat Monga、Christopher Olah、Mike Schuster、Noam Shazeer、Ilya Sutskever、Vincent Vanhoucke 以及 Google Brain 团队的有益意见、建议与技术支持。

References
[1] Nathaniel Morgan and Herve Bourlard. Continuous Speech Recognition using Multilayer Per-
ceptrons with Hidden Markov Models. In IEEE International Conference on Acoustics, Speech
and Signal Processing, 1990.
[2] Abdel-rahman Mohamed, George E. Dahl, and Geoffrey E. Hinton. Deep belief networks for
phone recognition. In Neural Information Processing Systems: Workshop on Deep Learning
for Speech Recognition and Related Applications, 2009.
[3] George E. Dahl, Dong Yu, Li Deng, and Alex Acero. Large vocabulary continuous speech
recognition with context-dependent dbn-hmms. In IEEE International Conference on Acous-
tics, Speech and Signal Processing, 2011.
[4] Abdel-rahman Mohamed, George E. Dahl, and Geoffrey Hinton. Acoustic modeling us-
ing deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing,
20(1):14–22, 2012.
[5] Navdeep Jaitly, Patrick Nguyen, Andrew W. Senior, and Vincent Vanhoucke. Application
of Pretrained Deep Neural Networks to Large Vocabulary Speech Recognition. In INTER-
SPEECH, 2012.
[6] Tara Sainath, Abdel-rahman Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep
Convolutional Neural Networks for LVCSR. In IEEE International Conference on Acoustics,
Speech and Signal Processing, 2013.
[7] Kanishka Rao, Fuchun Peng, Hasim Sak, and Francoise Beaufays. Grapheme-to-phoneme
conversion using long short-term memory recurrent neural networks. In IEEE International
Conference on Acoustics, Speech and Signal Processing, 2015.
[8] Kaisheng Yao and Geoffrey Zweig. Sequence-to-Sequence Neural Net Models for Grapheme-
to-Phoneme Conversion. 2015.
[9] Tomas Mikolov, Karafiat Martin, Burget Luka, Eernocky Jan, and Khudanpur Sanjeev. Recur-
rent neural network based language model. In INTERSPEECH, 2010.
[10] Alex Graves, Santiago Fernandez, Faustino Gomez, and Jurgen Schmiduber. Connectionist
Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Net-
works. In International Conference on Machine Learning, 2006.

参考文献
[1] Nathaniel Morgan and Herve Bourlard. Continuous Speech Recognition using Multilayer Perceptrons with Hidden Markov Models. In IEEE International Conference on Acoustics, Speech and Signal Processing, 1990.
[2] Abdel-rahman Mohamed, George E. Dahl, and Geoffrey E. Hinton. Deep belief networks for phone recognition. In Neural Information Processing Systems: Workshop on Deep Learning for Speech Recognition and Related Applications, 2009.
[3] George E. Dahl, Dong Yu, Li Deng, and Alex Acero. Large vocabulary continuous speech recognition with context-dependent dbn-hmms. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2011.
[4] Abdel-rahman Mohamed, George E. Dahl, and Geoffrey Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 20(1):14–22, 2012.
[5] Navdeep Jaitly, Patrick Nguyen, Andrew W. Senior, and Vincent Vanhoucke. Application of Pretrained Deep Neural Networks to Large Vocabulary Speech Recognition. In INTERSPEECH, 2012.
[6] Tara Sainath, Abdel-rahman Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep Convolutional Neural Networks for LVCSR. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2013.
[7] Kanishka Rao, Fuchun Peng, Hasim Sak, and Francoise Beaufays. Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2015.
[8] Kaisheng Yao and Geoffrey Zweig. Sequence-to-Sequence Neural Net Models for Grapheme-to-Phoneme Conversion. 2015.
[9] Tomas Mikolov, Karafiat Martin, Burget Luka, Eernocky Jan, and Khudanpur Sanjeev. Recurrent neural network based language model. In INTERSPEECH, 2010.
[10] Alex Graves, Santiago Fernandez, Faustino Gomez, and Jurgen Schmiduber. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In International Conference on Machine Learning, 2006.

11
[11] Alex Graves. Sequence Transduction with Recurrent Neural Networks. In International Con-
ference on Machine Learning: Representation Learning Workshop, 2012.
[12] Alex Graves and Navdeep Jaitly. Towards End-to-End Speech Recognition with Recurrent
Neural Networks. In International Conference on Machine Learning, 2014.
[13] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan
Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Ng. Deep Speech:
Scaling up end-to-end speech recognition. In http://arxiv.org/abs/1412.5567, 2014.
[14] Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. End-to-end Con-
tinuous Speech Recognition using Attention-based Recurrent NN: First Results. In Neural In-
formation Processing Systems: Workshop Deep Learning and Representation Learning Work-
shop, 2014.
[15] Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio.
Attention-Based Models for Speech Recognition. In http://arxiv.org/abs/1506.07503, 2015.
[16] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by
Jointly Learning to Align and Translate. In International Conference on Learning Representa-
tions, 2015.
[17] Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to Sequence Learning with Neural
Networks. In Neural Information Processing Systems, 2014.
[18] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares,
Holger Schwen, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-
Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural
Language Processing, 2014.
[19] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled Sampling for Se-
quence Prediction with Recurrent Neural Networks. In http://arxiv.org/abs/1506.03099, 2015.
[20] Tara N. Sainath, Oriol Vinyals, Andrew Senior, and Hasim Sak. Convolutional, Long Short-
Term Memory, Fully Connected Deep Neural Networks. In IEEE International Conference on
Acoustics, Speech and Signal Processing, 2015.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep
Convolutional Neural Networks. In Neural Information Processing Systems, 2012.
[22] Leonard E. Baum and Ted Petrie. Statistical Inference for Probabilistic Functions of Finite
State Markov Chains. The Annals of Mathematical Statistics, 37:1554–1563, 1966.
[23] John Lafferty, Andrew McCallum, and Fernando Pereira. Conditional Random Fields: Proba-
bilistic Models for Segmenting and Labeling Sequence Data. In International Conference on
Machine Learning, 2001.
[24] Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Ad-
dressing the Rare Word Problem in Neural Machine Translation. In Association for Computa-
tional Linguistics, 2015.
[25] Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On Using Very
Large Target Vocabulary for Neural Machine Translation. In Association for Computational
Linguistics, 2015.
[26] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and Tell: A Neural
Image Caption Generator. In IEEE Conference on Computer Vision and Pattern Recognition,
2015.
[27] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov,
Richard Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation
with Visual Attention. In International Conference on Machine Learning, 2015.
[28] Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey E. Hinton.
Grammar as a foreign language. In http://arxiv.org/abs/1412.7449, 2014.
[29] Oriol Vinyals and Quoc V. Le. A Neural Conversational Model. In International Conference
on Machine Learning: Deep Learning Workshop, 2015.
[30] Sepp Hochreiter and Jurgen Schmidhuber. Long Short-Term Memory. Neural Computation,
9(8):1735–1780, November 1997.

[11] Alex Graves. Sequence Transduction with Recurrent Neural Networks. In International Conference on Machine Learning: Representation Learning Workshop, 2012.
[12] Alex Graves and Navdeep Jaitly. Towards End-to-End Speech Recognition with Recurrent Neural Networks. In International Conference on Machine Learning, 2014.
[13] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Ng. Deep Speech: Scaling up end-to-end speech recognition. In http://arxiv.org/abs/1412.5567, 2014.
[14] Jan Chorowski, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results. In Neural Information Processing Systems: Workshop Deep Learning and Representation Learning Workshop, 2014.
[15] Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-Based Models for Speech Recognition. In http://arxiv.org/abs/1506.07503, 2015.
[16] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations, 2015.
[17] Ilya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to Sequence Learning with Neural Networks. In Neural Information Processing Systems, 2014.
[18] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwen, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Conference on Empirical Methods in Natural Language Processing, 2014.
[19] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In http://arxiv.org/abs/1506.03099, 2015.
[20] Tara N. Sainath, Oriol Vinyals, Andrew Senior, and Hasim Sak. Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2015.
[21] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Neural Information Processing Systems, 2012.
[22] Leonard E. Baum and Ted Petrie. Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics, 37:1554–1563, 1966.
[23] John Lafferty, Andrew McCallum, and Fernando Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In International Conference on Machine Learning, 2001.
[24] Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Addressing the Rare Word Problem in Neural Machine Translation. In Association for Computational Linguistics, 2015.
[25] Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On Using Very Large Target Vocabulary for Neural Machine Translation. In Association for Computational Linguistics, 2015.
[26] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and Tell: A Neural Image Caption Generator. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
[27] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In International Conference on Machine Learning, 2015.
[28] Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey E. Hinton. Grammar as a foreign language. In http://arxiv.org/abs/1412.7449, 2014.
[29] Oriol Vinyals and Quoc V. Le. A Neural Conversational Model. In International Conference on Machine Learning: Deep Learning Workshop, 2015.
[30] Sepp Hochreiter and Jurgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, November 1997.

12
[31] Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. Hybrid Speech Recognition with
Bidirectional LSTM. In Automatic Speech Recognition and Understanding Workshop, 2013.
[32] Salah Hihi and Yoshua Bengio. Hierarchical Recurrent Neural Networks for Long-Term De-
pendencies. In Neural Information Processing Systems, 1996.
[33] Jan Koutnik, Klaus Greff, Faustino Gomez, and Jurgen Schmidhuber. A Clockwork RNN. In
International Conference on Machine Learning, 2014.
[34] Navdeep Jaitly, Vincent Vanhoucke, and Geoffrey Hinton. Autoregressive product of multi-
frame predictions can improve the accuracy of hybrid models. In INTERSPEECH, 2014.
[35] Hasim Sak, Andrew Senior, Kanishka Rao, and Francoise Beaufays. Fast and Accurate Recur-
rent Neural Network Acoustic Models for Speech Recognition. In INTERSPEECH, 2015.
[36] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Repre-
sentations of Words and Phrases and their Compositionality. In Neural Information Processing
Systems, 2013.
[37] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra
Goel, Mirko Hannenmann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg
Stemmer, and Karel Vesely. The Kaldi Speech Recognition Toolkit. In Automatic Speech
Recognition and Understanding Workshop, 2011.
[38] Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z.
Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large
Scale Distributed Deep Networks. In Neural Information Processing Systems, 2012.

[31] Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. Hybrid Speech Recognition with Bidirectional LSTM. In Automatic Speech Recognition and Understanding Workshop, 2013.
[32] Salah Hihi and Yoshua Bengio. Hierarchical Recurrent Neural Networks for Long-Term Dependencies. In Neural Information Processing Systems, 1996.
[33] Jan Koutnik, Klaus Greff, Faustino Gomez, and Jurgen Schmidhuber. A Clockwork RNN. In International Conference on Machine Learning, 2014.
[34] Navdeep Jaitly, Vincent Vanhoucke, and Geoffrey Hinton. Autoregressive product of multi-frame predictions can improve the accuracy of hybrid models. In INTERSPEECH, 2014.
[35] Hasim Sak, Andrew Senior, Kanishka Rao, and Francoise Beaufays. Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition. In INTERSPEECH, 2015.
[36] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Neural Information Processing Systems, 2013.
[37] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannenmann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. The Kaldi Speech Recognition Toolkit. In Automatic Speech Recognition and Understanding Workshop, 2011.
[38] Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large Scale Distributed Deep Networks. In Neural Information Processing Systems, 2012.

13
A Alignment Examples
In this section, we give additional visualization examples of our model and the attention distribution.

A 对齐例子
本节给出模型与注意力分布的额外可视化例子。

Figure 6: The spelling variants of “aaa” vs “triple a” produces different attention distributions, both spelling
variants appear in our top beams. The ground truth is: “aaa emergency roadside service”.

图 6：“aaa” 与 “triple a” 的拼写变体产生不同注意力分布，两种变体都出现在 top 束中。真值为：“aaa emergency roadside service”。

14
Figure 7: The spelling variants of “st” vs “saint” produces different attention distributions, both spelling
variants appear in our top beams. The ground truth is: “st mary’s animal clinic”.

图 7：“st” 与 “saint” 的拼写变体产生不同注意力分布，两种变体都出现在 top 束中。真值为：“st mary's animal clinic”。

15
Figure 8: The phrase “cancel” is repeated three times. Note the parallel diagonals, the content attention
mechanism gets slightly confused however the model still emits the correct hypothesis. The ground truth is:
“cancel cancel cancel”.

图 8：短语 “cancel” 重复三次。注意平行对角线：内容注意力机制略有混淆，但模型仍发出正确假设。真值为：“cancel cancel cancel”。