SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

SpecAugment：一种用于自动语音识别的简单数据增强方法（SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition）

Abstract

摘要

We present SpecAugment, a simple data augmentation method for speech recognition. SpecAugment is applied directly to the feature inputs of a neural network (i.e., filter bank coefficients). The augmentation policy consists of warping the features, masking blocks of frequency channels, and masking blocks of time steps. We apply SpecAugment on Listen, Attend and Spell networks for end-to-end speech recognition tasks. We achieve state-of-the-art performance on the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work. On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language model, and 5.8% WER with shallow fusion with a language model. This compares to the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5’00 test set without the use of a language model, and 6.8%/14.1% with shallow fusion, which compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER.

我们提出 SpecAugment，一种用于语音识别的简单数据增强方法。SpecAugment 直接施加在神经网络的特征输入上（即滤波器组系数）。增强策略包括：对特征做时间扭曲、遮挡连续的频率通道块，以及遮挡连续的时间步块。我们在 Listen, Attend and Spell 网络上把 SpecAugment 用于端到端语音识别任务，在 LibriSpeech 960h 与 Switchboard 300h 任务上取得当时最优性能，超过此前所有工作。在 LibriSpeech 上，不使用语言模型时 test-other 达到 6.8% WER，与语言模型浅融合后为 5.8% WER；相较此前混合系统 7.5% WER 的最优结果有明显提升。在 Switchboard 上，不使用语言模型时 Hub5'00 测试集的 Switchboard/CallHome 部分为 7.2%/14.6%，浅融合后为 6.8%/14.1%，相较此前混合系统的 8.3%/17.3% WER 同样更优。

Index Terms: end-to-end speech recognition, data augmentation

索引词：端到端语音识别，数据增强

1 Introduction

1 引言

Deep Learning has been applied successfully to Automatic Speech Recognition (ASR) [1], where the main focus of research has been designing better network architectures, for example, DNNs [2], CNNs [3], RNNs [4] and end-to-end models [5, 6, 7]. However, these models tend to overfit easily and require large amounts of training data [8].

深度学习已成功应用于自动语音识别（ASR）[1]，研究重点主要在设计更好的网络架构，例如 DNN [2]、CNN [3]、RNN [4] 以及端到端模型 [5, 6, 7]。然而这些模型很容易过拟合，并且需要大量训练数据 [8]。

Data augmentation has been proposed as a method to generate additional training data for ASR. For example, in [9, 10], artificial data was augmented for low resource speech recognition tasks. Vocal Tract Length Normalization has been adapted for data augmentation in [11]. Noisy audio has been synthesised via superimposing clean audio with a noisy audio signal in [12]. Speed perturbation has been applied on raw audio for LVSCR tasks in [13]. The use of an acoustic room simulator has been explored in [14]. Data augmentation for keyword spotting has been studied in [15, 16]. Feature drop-outs have been employed for training multi-stream ASR systems [17]. More generally, learned augmentation techniques have explored different sequences of augmentation transformations that have achieved state-of-the-art performance in the image domain [18].

数据增强被提出来为 ASR 生成额外训练数据。例如，[9, 10] 为低资源语音识别任务增强了人工数据。[11] 把声道长度归一化改造成数据增强。 [12] 通过把干净音频与噪声信号叠加来合成带噪音频。[13] 在原始音频上做变速扰动用于 LVCSR 任务。[14] 探索了声学房间仿真器。[15, 16] 研究了关键词检出的数据增强。[17] 在多流 ASR 系统训练中使用了特征丢弃。更一般地，可学习的增强技术探索了不同增强变换序列，并在图像领域取得了当时最优性能 [18]。

Inspired by the recent success of augmentation in the speech and vision domains, we propose SpecAugment, an augmentation method that operates on the log mel spectrogram of the input audio, rather than the raw audio itself. This method is simple and computationally cheap to apply, as it directly acts on the log mel spectrogram as if it were an image, and does not require any additional data. We are thus able to apply SpecAugment online during training. SpecAugment consists of three kinds of deformations of the log mel spectrogram. The first is time warping, a deformation of the time-series in the time direction. The other two augmentations, inspired by “Cutout”, proposed in computer vision [19], are time and frequency masking, where we mask a block of consecutive time steps or mel frequency channels.

受语音与视觉领域近期增强工作的启发，我们提出 SpecAugment：一种作用在输入音频的 log mel 频谱图上、而不是原始音频上的增强方法。该方法简单且计算开销低，因为它把 log mel 频谱图当作图像直接处理，并且不需要任何额外数据。因此我们可以在训练过程中在线施加 SpecAugment。SpecAugment 包含对 log mel 频谱图的三类变形。第一类是时间扭曲，即沿时间方向对时间序列做变形。另外两类增强受计算机视觉中提出的 “Cutout” [19] 启发，分别是时间遮挡与频率遮挡：遮挡连续的一段时间步或连续的一段 mel 频率通道。

This approach while rudimentary, is remarkably effective and allows us to train end-to-end ASR networks, called Listen Attend and Spell (LAS) [6], to surpass more complicated hybrid systems, and achieve state-of-the-art results even without the use of Language Models (LMs). On LibriSpeech [20], we achieve 2.8% Word Error Rate (WER) on the test-clean set and 6.8% WER on the test-other set, without the use of an LM. Upon shallow fusion [21] with an LM trained on the LibriSpeech LM corpus, we are able to better our performance (2.5% WER on test-clean and 5.8% WER on test-other), improving the current state of the art on test-other by 22% relatively. On Switchboard 300h (LDC97S62) [22], we obtain 7.2% WER on the Switchboard portion of the Hub5’00 (LDC2002S09, LDC2003T02) test set, and 14.6% on the CallHome portion, without using an LM. Upon shallow fusion with an LM trained on the combined transcript of the Switchboard and Fisher (LDC200{4,5}T19) [23] corpora, we obtain 6.8%/14.1% WER on the Switchboard/Callhome portion.

这一做法虽然朴素，却非常有效，使我们能够训练名为 Listen Attend and Spell（LAS）[6] 的端到端 ASR 网络，超越更复杂的混合系统，甚至在不使用语言模型（LM）时也能取得当时最优结果。在 LibriSpeech [20] 上，不使用 LM 时我们在 test-clean 上达到 2.8% 词错误率（WER），在 test-other 上达到 6.8% WER。与在 LibriSpeech LM 语料上训练的语言模型做浅融合 [21] 后，性能进一步提升（test-clean 2.5% WER，test-other 5.8% WER），相对改进了当时 test-other 上的最优结果约 22%。在 Switchboard 300h（LDC97S62）[22] 上，不使用 LM 时，我们在 Hub5'00（LDC2002S09, LDC2003T02）测试集的 Switchboard 部分得到 7.2% WER，在 CallHome 部分得到 14.6%。与在 Switchboard 与 Fisher（LDC200{4,5}T19）[23] 合并转写上训练的 LM 浅融合后，Switchboard/CallHome 部分为 6.8%/14.1% WER。

2 Augmentation Policy

2 增强策略

We aim to construct an augmentation policy that acts on the log mel spectrogram directly, which helps the network learn useful features. Motivated by the goal that these features should be robust to deformations in the time direction, partial loss of frequency information and partial loss of small segments of speech, we have chosen the following deformations to make up a policy:

我们希望构造一种直接作用在 log mel 频谱图上的增强策略，以帮助网络学习有用特征。出于让这些特征对时间方向的变形、部分频率信息丢失以及小段语音丢失保持稳健的目标，我们选择了下列变形来组成策略：

1. Time warping is applied via the function sparse_image_warp of tensorflow. Given a log mel spectrogram with $\tau$ time steps, we view it as an image where the time axis is horizontal and the frequency axis is vertical. A random point along the horizontal line passing through the center of the image within the time steps $(W,\tau-W)$ is to be warped either to the left or right by a distance $w$ chosen from a uniform distribution from 0 to the time warp parameter $W$ along that line. We fix six anchor points on the boundary—the four corners and the mid-points of the vertical edges.
2. Frequency masking is applied so that $f$ consecutive mel frequency channels $[f_{0},f_{0}+f)$ are masked, where $f$ is first chosen from a uniform distribution from 0 to the frequency mask parameter $F$, and $f_{0}$ is chosen from $[0,\nu-f)$. $\nu$ is the number of mel frequency channels.
3. Time masking is applied so that $t$ consecutive time steps $[t_{0},t_{0}+t)$ are masked, where $t$ is first chosen from a uniform distribution from 0 to the time mask parameter $T$, and $t_{0}$ is chosen from $[0,\tau-t)$. • We introduce an upper bound on the time mask so that a time mask cannot be wider than $p$ times the number of time steps.

1. 时间扭曲通过 TensorFlow 的 sparse_image_warp 函数施加。给定具有 $\tau$ 个时间步的 log mel 频谱图，我们把它视为时间轴水平、频率轴垂直的图像。在穿过图像中心的水平线上、位于时间步 $(W,\tau-W)$ 范围内随机取一点，沿该线向左或向右扭曲距离 $w$，其中 $w$ 从 0 到时间扭曲参数 $W$ 的均匀分布中采样。我们在边界上固定六个锚点——四个角以及竖直边的中点。
2. 频率遮挡遮住 $f$ 个连续的 mel 频率通道 $[f_{0},f_{0}+f)$，其中 $f$ 先从 0 到频率遮挡参数 $F$ 的均匀分布中采样，$f_{0}$ 从 $[0,\nu-f)$ 中选取。$\nu$ 是 mel 频率通道数。
3. 时间遮挡遮住 $t$ 个连续时间步 $[t_{0},t_{0}+t)$，其中 $t$ 先从 0 到时间遮挡参数 $T$ 的均匀分布中采样，$t_{0}$ 从 $[0,\tau-t)$ 中选取。• 我们还对时间遮挡引入上限，使时间遮挡宽度不能超过时间步数的 $p$ 倍。

Figure 1 shows examples of the individual augmentations applied to a single input. The log mel spectrograms are normalized to have zero mean value, and thus setting the masked value to zero is equivalent to setting it to the mean value.

图 1 给出了在同一输入上分别施加各类增强的例子。log mel 频谱图被归一化为零均值，因此把被遮挡的值设为 0 等价于设为均值。

Figure 1: Augmentations applied to the base input, given at the top. From top to bottom, the figures depict the log mel spectrogram of the base input with no augmentation, time warp, frequency masking and time masking applied.

图 1：对顶部给出的原始输入施加的增强。自上而下依次为：无增强、时间扭曲、频率遮挡与时间遮挡后的 log mel 频谱图。

We can consider policies where multiple frequency and time masks are applied. The multiple masks may overlap. In this work, we mainly consider a series of hand-crafted policies, LibriSpeech basic (LB), LibriSpeech double (LD), Switchboard mild (SM) and Switchboard strong (SS) whose parameters are summarized in Table 1. In Figure 2, we show an example of a log mel spectrogram augmented with policies LB and LD.

我们也可以考虑施加多条频率遮挡与时间遮挡的策略。多条遮挡可以重叠。本文主要考虑一系列手调策略：LibriSpeech basic（LB）、LibriSpeech double（LD）、Switchboard mild（SM）与 Switchboard strong（SS），其参数汇总于表 1。图 2 展示了用策略 LB 与 LD 增强后的 log mel 频谱图示例。

Policy	$W$	$F$	$m_{F}$	$T$	$p$	$m_{T}$
None	0	0	-	0	-	-
LB	80	27	1	100	1.0	1
LD	80	27	2	100	1.0	2
SM	40	15	2	70	0.2	2
SS	40	27	2	70	0.2	2

Table 1: Augmentation parameters for policies. $m_{F}$ and $m_{T}$ denote the number of frequency and time masks applied.

策略	$W$	$F$	$m_{F}$	$T$	$p$	$m_{T}$
None	0	0	-	0	-	-
LB	80	27	1	100	1.0	1
LD	80	27	2	100	1.0	2
SM	40	15	2	70	0.2	2
SS	40	27	2	70	0.2	2

表 1：各策略的增强参数。$m_{F}$ 与 $m_{T}$ 分别表示施加的频率遮挡与时间遮挡条数。

Figure 2: Augmentation policies applied to the base input. From top to bottom, the figures depict the log mel spectrogram of the base input with policies None, LB and LD applied.

图 2：对原始输入施加的增强策略。自上而下依次为策略 None、LB 与 LD 作用后的 log mel 频谱图。

3 Model

3 模型

We use Listen, Attend and Spell (LAS) networks [6] for our ASR tasks. These models, being end-to-end, are simple to train and have the added benefit of having well-documented benchmarks [24, 25] that we are able to build upon to get our results. In this section, we review LAS networks and introduce some notation to parameterize them. We also introduce the learning rate schedules we use to train the networks, as they turn out to be an important factor in determining performance. We end with reviewing shallow fusion [21], which we have used to incorporate language models for further gains in performance.

我们的 ASR 任务使用 Listen, Attend and Spell（LAS）网络 [6]。这类端到端模型训练简单，并且已有记录完善的基准 [24, 25]，便于我们在其上取得结果。本节回顾 LAS 网络并引入参数化记号，同时介绍用于训练网络的学习率日程——事实证明它对性能至关重要。最后回顾浅融合 [21]，我们用它接入语言模型以进一步提升性能。

3.1 LAS Network Architectures

3.1 LAS 网络架构

We use Listen, Attend and Spell (LAS) networks [6] for end-to-end ASR studied in [25], for which we use the notation LAS-$d$-$w$. The input log mel spectrogram is passed in to a 2-layer Convolutional Neural Network (CNN) with max-pooling and stride of $2$. The output of the CNN is passes through an encoder consisting of $d$ stacked bi-directional LSTMs with cell size $w$ to yield a series of attention vectors. The attention vectors are fed into a 2-layer RNN decoder of cell dimension $w$, which yields the tokens for the transcript. The text is tokenized using a Word Piece Model (WPM) [26] of vocabulary size 16k for LibriSpeech and 1k for Switchboard. The WPM for LibriSpeech 960h is constructed using the training set transcripts. For the Switchboard 300h task, transcripts from the training set are combined with those of the Fisher corpus to construct the WPM. The final transcripts are obtained by a beam search with beam size 8. For comparison with [25], we note that their “large model” in our notation is LAS-4-1024.

我们使用 [25] 所研究的 Listen, Attend and Spell（LAS）网络 [6] 做端到端 ASR，并记作 LAS-$d$-$w$。输入的 log mel 频谱图先经过带最大池化、步长为 $2$ 的 2 层卷积神经网络（CNN）。CNN 输出再送入由 $d$ 层堆叠、单元大小为 $w$ 的双向 LSTM 组成的编码器，得到一系列注意力向量。注意力向量进入单元维度为 $w$ 的 2 层 RNN 解码器，输出转写的 token。文本用词片模型（WPM）[26] 分词，LibriSpeech 词表大小为 16k，Switchboard 为 1k。LibriSpeech 960h 的 WPM 用训练集转写构建；Switchboard 300h 任务则把训练集转写与 Fisher 语料转写合并后构建 WPM。最终转写由束宽为 8 的束搜索得到。为与 [25] 对照，他们的 “large model” 在我们的记号下是 LAS-4-1024。

3.2 Learning Rate Schedules

3.2 学习率日程

The learning rate schedule turns out to be an important factor in determining the performance of ASR networks, especially so when augmentation is present. Here, we introduce training schedules that serve two purposes. First, we use these schedules to verify that a longer schedule improves the final performance of the network, even more so with augmentation (Table 2). Second, based on this, we introduce very long schedules that are used to maximize the performance of the networks.

学习率日程被证明是决定 ASR 网络性能的重要因素，在有增强时尤其如此。这里我们引入的训练日程有两个目的。第一，用这些日程验证更长的日程能提升网络最终性能，在有增强时提升更明显（表 2）。第二，据此引入用于把网络性能推到极限的很长日程。

We use a learning rate schedule in which we ramp-up, hold, then exponentially decay the learning rate until it reaches $\nicefrac{{1}}{{100}}$ of its maximum value. The learning rate is kept constant beyond this point. This schedule is parameterized by three time stamps $(s_{r},s_{i},s_{f})$ – the step $s_{r}$ where the ramp-up (from zero learning rate) is complete, the step $s_{i}$ where exponential decay starts, and the step $s_{f}$ where the exponential decay stops.

我们使用这样一种学习率日程：先爬升，再保持，然后指数衰减，直到降到最大值的 $\nicefrac{1}{100}$。此后学习率保持恒定。该日程由三个时间戳 $(s_{r},s_{i},s_{f})$ 参数化——爬升（从零学习率）完成的步数 $s_{r}$、开始指数衰减的步数 $s_{i}$，以及指数衰减停止的步数 $s_{f}$。

There are two more factors that introduce time scales in our experiment. First, we turn on a variational weight noise [27] of standard deviation 0.075 at step $s_{\text{noise}}$ and keep it constant throughout training. Weight noise is introduced in the step interval $(s_{r},s_{i})$, i.e., during the high plateau of the learning rate.

实验中还有两个因素引入了时间尺度。第一，我们在步数 $s_{\text{noise}}$ 打开标准差为 0.075 的变分权重噪声 [27]，并在此后训练中保持不变。权重噪声在步数区间 $(s_{r},s_{i})$ 引入，也就是学习率的高平台阶段。

Second, we introduce uniform label smoothing [28] with uncertainty 0.1, i.e., the correct class label is assigned the confidence 0.9, while the confidence of the other labels are increased accordingly. As is commented on again later on, label smoothing can destabilize training for smaller learning rates, and we sometimes choose to turn it on only at the beginning of training and off when the learning rate starts to decay.

第二，我们引入不确定度为 0.1 的均匀标签平滑 [28]，即正确类标签置信度为 0.9，其余标签的置信度相应提高。后文还会提到，标签平滑会在较小学习率时使训练不稳，因此我们有时选择只在训练开始时打开，并在学习率开始衰减时关掉。

The two basic schedules we use, are given as the following:

我们使用的两个基本日程如下：

1. B(asic): $(s_{r},s_{\text{noise}},s_{i},s_{f})$ = (0.5k, 10k, 20k, 80k)
2. D(ouble): $(s_{r},s_{\text{noise}},s_{i},s_{f})$ = (1k, 20k, 40k, 160k)

1. B(asic，基本)：$(s_{r},s_{\text{noise}},s_{i},s_{f})$ = (0.5k, 10k, 20k, 80k)
2. D(ouble，加倍)：$(s_{r},s_{\text{noise}},s_{i},s_{f})$ = (1k, 20k, 40k, 160k)

As discussed further in section 5, we can improve the performance of the trained network by using a longer schedule. We thus introduce the following schedule:

如第 5 节进一步讨论的，使用更长的日程可以提升已训练网络的性能。因此我们再引入如下日程：

3. L(ong): $(s_{r},s_{\text{noise}},s_{i},s_{f})$ = (1k, 20k, 140k, 320k)

3. L(ong，长)：$(s_{r},s_{\text{noise}},s_{i},s_{f})$ = (1k, 20k, 140k, 320k)

which we use to train the largest model to improve performance. When using schedule L, label smoothing with uncertainty $0.1$ is introduced for time steps $<s_{i}=\text{140k}$ for LibriSpeech 960h, and is subsequently turned off. For Switchboard 300h, label smoothing is turned on throughout training.

我们用它训练最大模型以提升性能。使用日程 L 时，对 LibriSpeech 960h，不确定度为 $0.1$ 的标签平滑在时间步 $<s_{i}=140$k 时打开，随后关掉。对 Switchboard 300h，标签平滑在整个训练过程中保持打开。

3.3 Shallow Fusion with Language Models

3.3 与语言模型的浅融合

While we are able to get state-of-the-art results with augmentation, we can get further improvements by using a language model. We thus incorporate an RNN language model by shallow fusion for both tasks. In shallow fusion, the “next token” $\mathbf{y}^{*}$ in the decoding process is determined by

尽管仅靠增强我们已能取得当时最优结果，使用语言模型还能进一步改进。因此我们通过浅融合为两个任务接入 RNN 语言模型。在浅融合中，解码过程的“下一 token” $\mathbf{y}^{*}$ 由下式决定

$$\mathbf{y}^{*}=\underset{\mathbf{y}}{\text{argmax}}\left(\log P(\mathbf{y}|\mathbf{x})+\lambda\log P_{LM}(\mathbf{y})\right)\,, \tag{1}$$

i.e., by jointly scoring the token using the base ASR model and the language model. We also use a coverage penalty $c$ [29].

也就是用基础 ASR 模型与语言模型共同给 token 打分。我们还使用覆盖惩罚 $c$ [29]。

For LibriSpeech, we use a two-layer RNN with embedding dimension 1024 used in [25] for the LM, which is trained on the LibriSpeech LM corpus. We use identical fusion parameters ($\lambda=0.35$ and $c=0.05$) used in [25] throughout.

对 LibriSpeech，我们使用 [25] 中嵌入维度为 1024 的两层 RNN 作为 LM，并在 LibriSpeech LM 语料上训练。融合参数全程采用与 [25] 相同的设置（$\lambda=0.35$，$c=0.05$）。

For Switchboard, we use a two-layer RNN with embedding dimension 256, which is trained on the combined transcripts of the Fisher and Switchboard datasets. We find the fusion parameters via grid search by measuring performance on RT-03 (LDC2007S10). We discuss the fusion parameters used in individual experiments in section 4.2.

对 Switchboard，我们使用嵌入维度为 256 的两层 RNN，在 Fisher 与 Switchboard 数据集的合并转写上训练。融合参数通过对 RT-03（LDC2007S10）上的性能做网格搜索得到。各实验使用的融合参数在第 4.2 节讨论。

4 Experiments

4 实验

In this section, we describe our experiments on LibriSpeech and Switchboard with SpecAugment. We report state-of-the-art results that out-perform heavily engineered hybrid systems.

本节描述我们在 LibriSpeech 与 Switchboard 上使用 SpecAugment 的实验。我们报告超过精心设计的混合系统的当时最优结果。

4.1 LibriSpeech 960h

For LibriSpeech, we use the same setup as [25], where we use 80-dimensional filter banks with delta and delta-delta acceleration, and a 16k word piece model [26].

对 LibriSpeech，我们采用与 [25] 相同的设置：80 维滤波器组，并带一阶与二阶差分，以及 16k 词片模型 [26]。

The three networks LAS-4-1024, LAS-6-1024 and LAS-6-1280 are trained on LibriSpeech 960h with a combination of augmentation policies (None, LB, LD) and training schedules (B/D). Label smoothing was not applied in these experiments. The experiments were run with peak learning rate of $0.001$ and batch size of 512, on 32 Google Cloud TPU chips for 7 days. Other than the augmentation policies and learning rate schedules, all other hyperparameters were fixed, and no additional tuning was applied. We report test set numbers validated by the dev-other set in Table 2. We see that augmentation consistently improves the performance of the trained network, and that the benefit of a larger network and a longer learning rate schedule is more apparent with harsher augmentation.

三个网络 LAS-4-1024、LAS-6-1024 与 LAS-6-1280 在 LibriSpeech 960h 上，用增强策略（None、LB、LD）与训练日程（B/D）的组合进行训练。这些实验未使用标签平滑。峰值学习率为 $0.001$，批大小为 512，在 32 块 Google Cloud TPU 上训练 7 天。除增强策略与学习率日程外，其余超参数固定，未做额外调参。我们报告由 dev-other 集验证选出的测试集数字，见表 2。可以看到增强一致提升了已训练网络的性能，并且在更强增强下，更大网络与更长学习率日程的收益更明显。

Network	Sch	Pol	No LM	With LM
clean	other	clean	other
LAS-4-1024 [25]	B	-	4.7	13.4	3.6	10.3
LAS-4-1024	B	LB	3.7	10.0	3.4	8.3
B	LD	3.6	9.2	2.8	7.5
D	-	4.4	13.3	3.5	10.4
D	LB	3.4	9.2	2.7	7.3
D	LD	3.4	8.3	2.8	6.8
LAS-6-1024	D	-	4.5	13.1	3.6	10.3
D	LB	3.4	8.6	2.6	6.7
D	LD	3.2	8.0	2.6	6.5
LAS-6-1280	D	-	4.3	12.9	3.5	10.5
D	LB	3.4	8.7	2.8	7.1
D	LD	3.2	7.7	2.7	6.5

Table 2: LibriSpeech test WER (%) evaluated for varying networks, schedules and policies. First row from [25].

网络	日程	策略	无 LM	有 LM
clean	other	clean	other
LAS-4-1024 [25]	B	-	4.7	13.4	3.6	10.3
LAS-4-1024	B	LB	3.7	10.0	3.4	8.3
B	LD	3.6	9.2	2.8	7.5
D	-	4.4	13.3	3.5	10.4
D	LB	3.4	9.2	2.7	7.3
D	LD	3.4	8.3	2.8	6.8
LAS-6-1024	D	-	4.5	13.1	3.6	10.3
D	LB	3.4	8.6	2.6	6.7
D	LD	3.2	8.0	2.6	6.5
LAS-6-1280	D	-	4.3	12.9	3.5	10.5
D	LB	3.4	8.7	2.8	7.1
D	LD	3.2	7.7	2.7	6.5

表 2：在不同网络、日程与策略下评估的 LibriSpeech 测试 WER（%）。第一行来自 [25]。

We take the largest network, LAS-6-1280, and use schedule L (with training time $\sim$ 24 days) and policy LD to train the network to maximize performance. We turn label smoothing on for time steps $<140$k as noted before. The test set performance is reported by evaluating the checkpoint with best dev-other performance. State of the art performance is achieved by the LAS-6-1280 model, even without a language model. We can incorporate an LM using shallow fusion to further improve performance. The results are presented in Table 3.

我们取最大网络 LAS-6-1280，用日程 L（训练时间约 24 天）与策略 LD 训练，以尽量提高性能。如前述，在时间步 $<140$k 时打开标签平滑。测试集性能按在 dev-other 上表现最好的检查点报告。LAS-6-1280 模型即使不使用语言模型也达到了当时最优。我们还可用浅融合接入 LM 进一步提升，结果见表 3。

Method	No LM	With LM
	clean	other	clean	other
HMM
Panayotov et al., (2015) [20]			5.51	13.97
Povey et al., (2016) [30]			4.28
Han et al., (2017) [31]			3.51	8.58
Yang et al. (2018) [32]			2.97	7.50
CTC/ASG
Collobert et al., (2016) [33]	7.2
Liptchinsky et al., (2017) [34]	6.7	20.8	4.8	14.5
Zhou et al., (2018) [35]			5.42	14.70
Zeghidour et al., (2018) [36]			3.44	11.24
Li et al., (2019) [37]	3.86	11.95	2.95	8.79
LAS
Zeyer et al., (2018) [24]	4.87	15.39	3.82	12.76
Zeyer et al., (2018) [38]	4.70	15.20
Irie et al., (2019) [25]	4.7	13.4	3.6	10.3
Sabour et al., (2019) [39]	4.5	13.3
Our Work
LAS	4.1	12.5	3.2	9.8
LAS + SpecAugment	2.8	6.8	2.5	5.8

Table 3: LibriSpeech 960h WERs (%).

方法	无 LM	有 LM
	clean	other	clean	other
HMM
Panayotov et al., (2015) [20]			5.51	13.97
Povey et al., (2016) [30]			4.28
Han et al., (2017) [31]			3.51	8.58
Yang et al. (2018) [32]			2.97	7.50
CTC/ASG
Collobert et al., (2016) [33]	7.2
Liptchinsky et al., (2017) [34]	6.7	20.8	4.8	14.5
Zhou et al., (2018) [35]			5.42	14.70
Zeghidour et al., (2018) [36]			3.44	11.24
Li et al., (2019) [37]	3.86	11.95	2.95	8.79
LAS
Zeyer et al., (2018) [24]	4.87	15.39	3.82	12.76
Zeyer et al., (2018) [38]	4.70	15.20
Irie et al., (2019) [25]	4.7	13.4	3.6	10.3
Sabour et al., (2019) [39]	4.5	13.3
本文工作
LAS	4.1	12.5	3.2	9.8
LAS + SpecAugment	2.8	6.8	2.5	5.8

表 3：LibriSpeech 960h 的 WER（%）。

4.2 Switchboard 300h

For Switchboard 300h, we use the Kaldi [40] “s5c” recipe to process our data, but we adapt the recipe to use 80-dimensional filter banks with delta and delta-delta acceleration. We use a 1k WPM [26] to tokenize the output, constructed using the combined vocabulary of the Switchboard and Fisher transcripts.

对 Switchboard 300h，我们用 Kaldi [40] 的 “s5c” 配方处理数据，但改为使用带一阶与二阶差分的 80 维滤波器组。输出用 1k WPM [26] 分词，词表由 Switchboard 与 Fisher 转写的合并词汇构建。

We train LAS-4-1024 with policies (None, SM, SS) and schedule B. As before, we set the peak learning rate to $0.001$ and total batch size to 512, and train using 32 Google Cloud TPU chips. Here the experiments are run with and without label smoothing. Not having a canonical development set, we choose to evaluate the checkpoint at the end point of the training schedule, which we choose to be 100k steps for schedule B. We note that the training curve relaxes after the decay schedule is completed (step $s_{f}$), and the performance of the network does not vary much. The performance of various augmentation policies with and without label smoothing for Switchboard 300h is shown in Table 4. We see that label smoothing and augmentation have an additive effect for this corpus.

我们用策略（None、SM、SS）与日程 B 训练 LAS-4-1024。与前面相同，峰值学习率设为 $0.001$，总批大小 512，在 32 块 Google Cloud TPU 上训练。这里的实验分别在有、无标签平滑下进行。由于没有标准开发集，我们选择在训练日程终点评估检查点，对日程 B 取 100k 步。我们注意到衰减日程结束后（步数 $s_{f}$）训练曲线趋于平稳，网络性能变化不大。Switchboard 300h 上不同增强策略在有、无标签平滑时的性能见表 4。可以看到，对该语料标签平滑与增强具有叠加效果。

Policy	LS	SWBD	CH
-	$\times$	12.1	22.6
$\circ$	11.2	21.6
SM	$\times$	9.5	18.8
$\circ$	8.5	16.1
SS	$\times$	9.7	18.2
$\circ$	8.6	16.3

Table 4: Switchboard 300h WER (%) evaluated for LAS-4-1024 trained with schedule B with varying augmentation and Label Smoothing (LS) policies. No LMs have been used.

策略	LS	SWBD	CH
-	$\times$	12.1	22.6
$\circ$	11.2	21.6
SM	$\times$	9.5	18.8
$\circ$	8.5	16.1
SS	$\times$	9.7	18.2
$\circ$	8.6	16.3

表 4：用日程 B 训练的 LAS-4-1024 在不同增强与标签平滑（LS）策略下的 Switchboard 300h WER（%）。未使用 LM。

As with LibriSpeech 960h, we train LAS-6-1280 on the Switchboard 300h training set with schedule L (training time $\sim$ 24 days) to get state of the art performance. In this case, we find that turning label smoothing on throughout training benefits the final performance. We report the performance at the end of training time at 340k steps. We present our results in the context of other work in Table 5. We also apply shallow fusion with an LM trained on Fisher-Switchboard, whose fusion parameters are obtained by evaluating performance on the RT-03 corpus. Unlike the case for LibriSpeech, the fusion parameters do not transfer well between networks trained differently—the three entries in Table 5 were obtained by using fusion parameters $(\lambda,c)=$ (0.3, 0.05), (0.2, 0.0125) and (0.1, 0.025) respectively.

与 LibriSpeech 960h 相同，我们在 Switchboard 300h 训练集上用日程 L（训练时间约 24 天）训练 LAS-6-1280，以取得当时最优性能。此时我们发现在整个训练过程中打开标签平滑有利于最终性能。我们报告训练结束时（340k 步）的性能，并在表 5 中与其他工作对照。我们还与在 Fisher-Switchboard 上训练的 LM 做浅融合，融合参数通过在 RT-03 语料上评估性能得到。与 LibriSpeech 不同，融合参数在训练方式不同的网络之间迁移不佳——表 5 中三个条目分别使用融合参数 $(\lambda,c)=$ (0.3, 0.05)、(0.2, 0.0125) 与 (0.1, 0.025)。

Method	No LM	With LM
	SWBD	CH	SWBD	CH
HMM
Veselý et al., (2013) [41]			12.9	24.5
Povey et al., (2016) [30]			9.6	19.3
Hadian et al., (2018) [42]			9.3	18.9
Zeyer et al., (2018) [24]			8.3	17.3
CTC
Zweig et al., (2017) [43]	24.7	37.1	14.0	25.3
Audhkhasi et al., (2018) [44]	20.8	30.4
Audhkhasi et al., (2018) [45]	14.6	23.6
LAS
Lu et al., (2016) [46]	26.8	48.2	25.8	46.0
Toshniwal et al., (2017) [47]	23.1	40.8
Zeyer et al., (2018) [24]	13.1	26.1	11.8	25.7
Weng et al., (2018) [48]	12.2	23.3
Zeyer et al., (2018) [38]	11.9	23.7	11.0	23.1
Our Work
LAS	11.2	21.6	10.9	19.4
LAS + SpecAugment (SM)	7.2	14.6	6.8	14.1
LAS + SpecAugment (SS)	7.3	14.4	7.1	14.0

Table 5: Switchboard 300h WERs (%).

方法	无 LM	有 LM
	SWBD	CH	SWBD	CH
HMM
Veselý et al., (2013) [41]			12.9	24.5
Povey et al., (2016) [30]			9.6	19.3
Hadian et al., (2018) [42]			9.3	18.9
Zeyer et al., (2018) [24]			8.3	17.3
CTC
Zweig et al., (2017) [43]	24.7	37.1	14.0	25.3
Audhkhasi et al., (2018) [44]	20.8	30.4
Audhkhasi et al., (2018) [45]	14.6	23.6
LAS
Lu et al., (2016) [46]	26.8	48.2	25.8	46.0
Toshniwal et al., (2017) [47]	23.1	40.8
Zeyer et al., (2018) [24]	13.1	26.1	11.8	25.7
Weng et al., (2018) [48]	12.2	23.3
Zeyer et al., (2018) [38]	11.9	23.7	11.0	23.1
本文工作
LAS	11.2	21.6	10.9	19.4
LAS + SpecAugment (SM)	7.2	14.6	6.8	14.1
LAS + SpecAugment (SS)	7.3	14.4	7.1	14.0

表 5：Switchboard 300h 的 WER（%）。

5 Discussion

5 讨论

Time warping contributes, but is not a major factor in improving performance. In Table 6, we present three training results for which time warping, time masking and frequency masking have been turned off, respectively. We see that the effect time warping, while small, is still existent. Time warping, being the most expensive as well as the least influential of the augmentations discussed in this work, should be the first augmentation to be dropped given any budgetary limitations.

时间扭曲有贡献，但不是提升性能的主要因素。 表 6 给出了分别关掉时间扭曲、时间遮挡与频率遮挡的三组训练结果。可以看到时间扭曲的作用虽小，但确实存在。在本文讨论的增强里，时间扭曲计算最贵、影响也最小，若有预算限制，应首先去掉。

$W$	$F$	$m_{F}$	$T$	$p$	$m_{T}$	test-other	test
80	27	1	100	1.0	1	10.0	3.7
0	27	1	100	1.0	1	10.1	3.8
80	0	-	100	1.0	1	11.0	4.0
80	27	1	0	-	-	10.9	4.1

Table 6: Test set WER (%) evaluated without LM for network LAS-4-1024 trained with schedule B.

$W$	$F$	$m_{F}$	$T$	$p$	$m_{T}$	test-other	test
80	27	1	100	1.0	1	10.0	3.7
0	27	1	100	1.0	1	10.1	3.8
80	0	-	100	1.0	1	11.0	4.0
80	27	1	0	-	-	10.9	4.1

表 6：网络 LAS-4-1024 用日程 B 训练、不使用 LM 时的测试集 WER（%）。

Label smoothing introduces instability to training. We have noticed that the proportion of unstable training runs increases for LibriSpeech when label smoothing is applied with augmentation. This becomes more conspicuous while learning rate is being decayed, thus our introduction of a label smoothing schedule for training LibriSpeech, where labels are only smoothed in the initial phases of the learning rate schedule.

标签平滑会给训练带来不稳定性。 我们注意到，在 LibriSpeech 上把标签平滑与增强一起使用时，不稳定训练运行的比例会上升。这在学习率衰减阶段更明显，因此我们为 LibriSpeech 引入了标签平滑日程：只在学习率日程的初期阶段对标签做平滑。

Augmentation converts an over-fitting problem into an under-fitting problem. As can be observed from the training curves of the networks in Figure 3, the networks during training not only under-fit the loss and WER on the augmented training set, but also on the training set itself when trained on augmented data. This is in stark contrast to the usual situation where networks tend to over-fit to the training data. This is the major benefit of training with augmentation, as explained below.

增强把过拟合问题变成了欠拟合问题。 如图 3 中网络的训练曲线所示，在增强数据上训练时，网络不仅在增强训练集的损失与 WER 上欠拟合，在原始训练集本身上也欠拟合。这与网络通常会过拟合训练数据的常见情形形成鲜明对比。如下所述，这正是带增强训练的主要好处。

Figure 3: LAS-6-1280 on LibriSpeech with schedule D.

图 3：LAS-6-1280 在 LibriSpeech 上使用日程 D 的训练情况。

Common methods of addressing under-fitting yield improvements. We were able to make significant gains in performance by standard approaches to alleviate under-fitting—making larger networks and training longer. The current reported performance was obtained by the recursive process of applying a harsh augmentation policy, and then making wider, deeper networks and training them with longer schedules to address the under-fitting.

应对欠拟合的常用方法能够带来提升。 我们通过缓解欠拟合的标准做法——加大网络并延长训练——取得了显著性能收益。当前报告的性能，是通过反复施加更强的增强策略，再把网络加宽加深、用更长日程训练以应对欠拟合而得到的。

Remark on related works. We note that an augmentation similar to frequency masking has been studied in the context of CNN acoustic models in [49]. There, blocks of adjacent frequencies are pre-grouped into bins, which are randomly zeroed-out per minibatch. On the other hand, both the size and position of the frequency masks in SpecAugment are chosen stochastically, and differ for every input in the minibatch. More ideas for structurally omitting frequency data of spectrograms have been discussed in [50].

相关工作备注。 我们注意到，[49] 在 CNN 声学模型语境下研究过类似频率遮挡的增强。那里相邻频率被预先分到若干箱中，每个 minibatch 随机置零。而 SpecAugment 中频率遮挡的宽度与位置都是随机选取的，并且对 minibatch 中每个输入都不同。[50] 还讨论了更多从结构上省略频谱图频率数据的想法。

6 Conclusions

6 结论

SpecAugment greatly improves the performance of ASR networks. We are able to obtain state-of-the-art results on the LibriSpeech 960h and Switchboard 300h tasks on end-to-end LAS networks by augmenting the training set using simple hand-crafted policies, surpassing the performance of hybrid systems even without the aid of a language model. SpecAugment converts ASR from an over-fitting to an under-fitting problem, and we were able to gain performance by using bigger networks and training longer.

SpecAugment 大幅提升了 ASR 网络的性能。通过用简单的手调策略增强训练集，我们在端到端 LAS 网络上于 LibriSpeech 960h 与 Switchboard 300h 任务取得了当时最优结果，甚至在没有语言模型辅助时也超过了混合系统。SpecAugment 把 ASR 从过拟合问题转变成欠拟合问题，我们得以通过加大网络并延长训练来继续获得性能提升。

Acknowledgements: We would like to thank Yuan Cao, Ciprian Chelba, Kazuki Irie, Ye Jia, Anjuli Kannan, Patrick Nguyen, Vijay Peddinti, Rohit Prabhavalkar, Yonghui Wu and Shuyuan Zhang for useful discussions. We also thank György Kovács for introducing us to the works [49, 50].

致谢： 感谢 Yuan Cao、Ciprian Chelba、Kazuki Irie、Ye Jia、Anjuli Kannan、Patrick Nguyen、Vijay Peddinti、Rohit Prabhavalkar、Yonghui Wu 与 Shuyuan Zhang 的有益讨论。也感谢 György Kovács 向我们介绍了工作 [49, 50]。

References

参考文献

[1] G. Hinton, L. Deng, D. Yu, G. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, B. Kingsbury et al. , “Deep neural networks for acoustic modeling in speech recognition,” IEEE Signal processing magazine , vol. 29, 2012.

[2] G. Dahl, D. Yu, L. Deng, and A. Acero, “Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition,” IEEE Transactions on Audio, Speech, and Language Processing , vol. 20, Jan 2012.

[3] T. Sainath, A. rahman Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep Convolutional Neural Networks for LVCSR,” in ICASSP , 2013.

[4] A. Graves, A. rahman Mohamed, and G. Hinton, “Speech Recognition with Deep Recurrent Neural Networks,” in ICASSP , 2013.

[5] A. Graves and N. Jaitly, “Towards End-to-End Speech Recognition with Recurrent Neural Networks,” in ICML , 2014.

[6] W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition,” in ICASSP , 2016.

[7] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-End Attention-based Large Vocabulary Speech Recognition,” in ICASSP , 2016.

[8] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani, “State-of-the-art Speech Recognition With Sequence-to-Sequence Models,” in ICASSP , 2018.

[9] N. Kanda, R. Takeda, and Y. Obuchi, “Elastic spectral distortion for low resource speech recognition with deep neural networks,” in ASRU , 2013.

[10] A. Ragni, K. M. Knill, S. P. Rath, and M. J. F. Gales, “Data augmentation for low resource languages,” in INTERSPEECH , 2014.

[11] N. Jaitly and G. Hinton, “Vocal Tract Length Perturbation (VTLP) improves speech recognition,” in ICML Workshop on Deep Learning for Audio, Speech and Language Processing , 2013.

[12] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates, and A. Ng, “Deep Speech: Scaling up end-to-end speech recognition,” in arXiv , 2014.

[13] T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio Augmentation for Speech Recognition,” in INTERSPEECH , 2015.

[14] C. Kim, A. Misra, K. Chin, T. Hughes, A. Narayanan, T. Sainath, and M. Bacchiani, “Generation of large-scale simulated utterances in virtual rooms to train deep-neural networks for far-field speech recognition in Google Home,” in INTERSPEECH , 2017.

[15] R. Prabhavalkar, R. Alvarez, C. Parada, P. Nakkiran, and T. N. Sainath, “Automatic gain control and multi-style training for robust small-footprint keyword spotting with deep neural networks,” in ICASSP , 2015.

[16] A. Raju, S. Panchapagesan, X. Liu, A. Mandal, and N. Strom, “Data Augmentation for Robust Keyword Spotting under Playback Interference,” in arXiv , 2018.

[17] S. H. Mallidi and H. Hermansky, “Novel neural network based fusion for Multistream ASR,” in ICASSP , 2016.

[18] E. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation policies from data,” in CVPR , 2019.

[19] T. DeVries and G. Taylor, “Improved Regularization of Convolutional Neural Networks with Cutout,” in arXiv , 2017.

[20] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in ICASSP , 2015.

[21] Ç. Gülçehre, O. Firat, K. Xu, K. Cho, L. Barrault, H. Lin, F. Bougares, H. Schwenk, and Y. Bengio, “On using monolingual corpora in neural machine translation,” in arxiv , 2015.

[22] J. Godfrey, E. Holliman, and J. McDaniel, “SWITCHBOARD: telephone speech corpus for research and development,” in ICASSP , 1992.

[23] C. Cieri, D. Miller, and K. Walker, “The fisher corpus: a resource for the next generations of speech-to-text,” in LREC , 2004.

[24] A. Zeyer, K. Irie, R. Schlüter, and H. Ney, “Improved training of end-to-end attention models for speech recognition,” in INTERSPEECH , 2018.

[25] K. Irie, R. Prabhavalkar, A. Kannan, A. Bruguier, D. Rybach, and P. Nguyen, “Model Unit Exploration for Sequence-to-Sequence Speech Recognition,” in arXiv , 2019.

[26] M. Schuster and K. Nakajima, “Japanese and korean voice search,” in ICASSP , 2012.

[27] A. Graves, “Practical Variational Inference for Neural Networks,” in NIPS , 2011.

[28] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in CVPR , 2016.

[29] J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” in INTERSPEECH , 2017.

[30] D. Povey, V. Peddinti, D. Galvez, P. Ghahrmani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neural networks for ASR based on lattice-free MMI,” in INTERSPEECH , 2016.

[31] K. J. Han, A. Chandrashekaran, J. Kim, and I. Lane, “The CAPIO 2017 Conversational Speech Recognition System,” in arXiv , 2017.

[32] X. Yang, J. Li, and X. Zhou, “A novel pyramidal-FSMN architecture with lattice-free MMI for speech recognition,” in arXiv , 2018.

[33] R. Collobert, C. Puhrsch, and G. Synnaeve, “Wav2Letter: an End-to-End ConvNet-based Speech Recognition System,” in arXiv , 2016.

[34] V. Liptchinsky, G. Synnaeve, and R. Collobert, “Letter-Based Speech Recognition with Gated ConvNets,” in arXiv , 2017.

[35] Y. Zhou, C. Xiong, and R. Socher, “Improving End-to-End Speech Recognition with Policy Learning,” in ICASSP , 2018.

[36] N. Zeghidour, Q. Xu, V. Liptchinsky, N. Usunier, G. Synnaeve, and R. Collobert, “Fully Convolutional Speech Recognition,” in arXiv , 2018.

[37] J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T. Gadde, “Jasper: An End-to-End Convolutional Neural Acoustic Model,” in arXiv , 2019.

[38] A. Zeyer, A. Merboldt, R. Schlüter, and H. Ney, “A comprehensive analysis on attention models,” in NIPS: Workshop IRASL , 2018.

[39] S. Sabour, W. Chan, and M. Norouzi, “Optimal Completion Distillation for Sequence Learning,” in ICLR , 2019.

[40] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The Kaldi Speech Recognition Toolkit,” in ASRU , 2011.

[41] K. Vesely, A. Ghoshal, L. Burger, and D. Povey, “Sequence-discriminative training of deep neural networks,” in INTERSPEECH , 2013.

[42] H. Hadian, H. Sameti, D. Povey, and S. Khudanpur, “End-to-end speech recognition using lattice-free MMI,” in INTERSPEECH , 2018.

[43] G. Zweig, C. Yu, J. Droppo, and A. Stolcke, “Advances in All-Neural Speech Recognition,” in ICASSP , 2017.

[44] K. Audhkhasi, B. Ramabhadran, G. Saon, M. Picheny, and D. Nahamoo, “Direct Acoustics-to-Word Models for English Conversational Speech Recognition,” in INTERSPEECH , 2018.

[45] K. Audhkhasi, B. Kingsbury, B. Ramabhadran, G. Saon, and M. Picheny, “Building competitive direct acoustics-to-word models for english conversational speech recognition,” in ICASSP , 2018.

[46] L. Lu, X. Zhang, and S. Renals, “On training the recurrent neural network encoder-decoder for large vocabulary end-to-end speech recognition,” in ICASSP , 2016.

[47] S. Toshniwal, H. Tang, L. Lu, and K. Livescu, “Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition,” in INTERSPEECH , 2017.

[48] C. Weng, J. Cui, G. Wang, J. Wang, C. Yu, D. Su, and D. Yu, “Improving Attention Based Sequence-to-Sequence Models for End-to-End English Conversational Speech Recognition,” in INTERSPEECH , 2018.

[49] G. Kovács, L. Tóth, D. Van Compernolle, and S. Ganapathy, “Increasing the robustness of cnn acoustic models using autoregressive moving average spectrogram features and channel dropout,” Pattern Recognition Letters , vol. 100, pp. 44–50, 2017.

[50] L. Tóth, G. Kovács, and D. Van Compernolle, “A perceptually inspired data augmentation method for noise robust cnn acoustic models,” in SPECOM , 2018.