wav2vec: Unsupervised Pre-training for Speech Recognition

wav2vec：面向语音识别的无监督预训练

Steffen Schneider    Alexei Baevski    Ronan Collobert    Michael Auli Affiliation:  Facebook AI Research

Abstract

摘要

We explore unsupervised pre-training for speech recognition by learning representations of raw audio. wav2vec is trained on large amounts of unlabeled audio data and the resulting representations are then used to improve acoustic model training. We pre-train a simple multi-layer convolutional neural network optimized via a noise contrastive binary classification task. Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to $36\text{\,}\mathrm{\%}$ when only a few hours of transcribed data is available. Our approach achieves $2.43\text{\,}\mathrm{\%}$ WER on the nov92 test set. This outperforms Deep Speech 2, the best reported character-based system in the literature while using two orders of magnitude less labeled training data.[^1]

[^1]: The code is available as part of fairseq (https://github.com/pytorch/fairseq).

我们探索用无监督预训练来做语音识别，具体做法是从原始音频里学习表征。wav2vec 先在大量无标注音频上训练，再用得到的表征去改进声学模型的训练。我们预训练一个结构简单的多层卷积神经网络，优化目标是噪声对比二元分类。在 WSJ 上的实验表明，当只有少量转写数据时，相对一个较强的、基于字符、使用 log-mel 滤波器组特征的基线，词错误率最多可降低 $36\text{\,}\mathrm{\%}$。本方法在 nov92 测试集上做到 $2.43\text{\,}\mathrm{\%}$ 的词错误率，超过了文献中当时最好的字符系统 Deep Speech 2，而所用带标注训练数据要少两个数量级。

[^1]：代码已作为 fairseq 的一部分公开（https://github.com/pytorch/fairseq）。

1 Introduction

1 引言

Current state of the art models for speech recognition require large amounts of transcribed audio data to attain good performance (Amodei et al., 2016). Recently, pre-training of neural networks has emerged as an effective technique for settings where labeled data is scarce. The key idea is to learn general representations in a setup where substantial amounts of labeled or unlabeled data is available and to leverage the learned representations to improve performance on a downstream task for which the amount of data is limited. This is particularly interesting for tasks where substantial effort is required to obtain labeled data, such as speech recognition.

当前最先进的语音识别模型要达到好的表现，需要大量转写音频数据（Amodei et al., 2016）。近来，神经网络预训练成了一种在标注稀缺时很有效的做法。核心想法是：先在有大量标注或无标注数据的设定里学到通用表征，再把这些表征用到数据有限的下游任务上，把成绩做上去。对那些获取标注特别费力的任务，比如语音识别，这一点尤其有吸引力。

In computer vision, representations for ImageNet (Deng et al., 2009) and COCO (Lin et al., 2014) have proven to be useful to initialize models for tasks such as image captioning (Vinyals et al., 2016) or pose estimation (Pavllo et al., 2019). Unsupervised pre-training for computer vision has also shown promise (Doersch et al., 2015; Hénaff et al., 2019). In natural language processing (NLP), unsupervised pre-training of language models (Devlin et al., 2018; Radford et al., 2018; Baevski et al., 2019) improved many tasks such as text classification, phrase structure parsing and machine translation (Edunov et al., 2019; Lample & Conneau, 2019). In speech processing, pre-training has focused on emotion recogniton (Lian et al., 2018), speaker identification (Ravanelli & Bengio, 2018), phoneme discrimination (Synnaeve & Dupoux, 2016a; van den Oord et al., 2018) as well as transferring ASR representations from one language to another (Kunze et al., 2017). There has been work on unsupervised learning for speech but the resulting representations have not been applied to improve supervised speech recognition (Synnaeve & Dupoux, 2016b; Kamper et al., 2017; Chung et al., 2018; Chen et al., 2018; Chorowski et al., 2019).

在计算机视觉里，为 ImageNet（Deng et al., 2009）和 COCO（Lin et al., 2014）学到的表征，已被用来初始化图像描述（Vinyals et al., 2016）或姿态估计（Pavllo et al., 2019）等任务的模型。视觉上的无监督预训练同样显示出前景（Doersch et al., 2015; Hénaff et al., 2019）。在自然语言处理（NLP）里，语言模型的无监督预训练（Devlin et al., 2018; Radford et al., 2018; Baevski et al., 2019）改进了文本分类、短语结构分析和机器翻译等许多任务（Edunov et al., 2019; Lample & Conneau, 2019）。在语音处理中，预训练主要用在情感识别（Lian et al., 2018）、说话人识别（Ravanelli & Bengio, 2018）、音素判别（Synnaeve & Dupoux, 2016a; van den Oord et al., 2018），以及把自动语音识别的表征从一种语言迁到另一种语言（Kunze et al., 2017）。语音上也有无监督学习的工作，但得到的表征还没有被用来改进有监督语音识别（Synnaeve & Dupoux, 2016b; Kamper et al., 2017; Chung et al., 2018; Chen et al., 2018; Chorowski et al., 2019）。

In this paper, we apply unsupervised pre-training to improve supervised speech recognition. This enables exploiting unlabeled audio data which is much easier to collect than labeled data. Our model, wav2vec, is a convolutional neural network that takes raw audio as input and computes a general representation that can be input to a speech recognition system. The objective is a contrastive loss that requires distinguishing a true future audio sample from negatives (Collobert et al., 2011; Mikolov et al., 2013; van den Oord et al., 2018). Different to previous work (van den Oord et al., 2018), we move beyond frame-wise phoneme classification and apply the learned representations to improve strong supervised ASR systems. wav2vec relies on a fully convolutional architecture which can be easily parallelized over time on modern hardware compared to recurrent models used in previous work (§2).

本文将无监督预训练用到改进有监督语音识别上。这样就可以利用无标注音频，而收集无标注音频比收集标注数据容易得多。我们的模型 wav2vec 是一个卷积神经网络，输入原始音频，算出一种通用表征，再交给语音识别系统使用。目标函数是对比损失，要求模型把真正的未来音频样本和负例区分开（Collobert et al., 2011; Mikolov et al., 2013; van den Oord et al., 2018）。与以往工作（van den Oord et al., 2018）不同，我们不再停在逐帧音素分类，而是把学到的表征拿去改进较强的有监督自动语音识别系统。wav2vec 采用全卷积结构，相比以往工作里用的循环模型，更容易在现代硬件上沿时间维并行（§2）。

Experimental results on the WSJ benchmark demonstrate that pre-trained representations estimated on about $1,000$ hours of unlabeled speech can substantially improve a character-based ASR system and outperform the best character-based result in the literature, Deep Speech 2, improving WER from $3.1\text{\,}\mathrm{\%}$ to $2.43\text{\,}\mathrm{\%}$. On TIMIT, pre-training enables us to match the best reported result in the literature. In a simulated low-resource setup with only eight hours of transcribed audio data, wav2vec reduces WER by up to $36\text{\,}\mathrm{\%}$ compared to a baseline model that relies on labeled data only (§3, §4).

在 WSJ 基准上的实验表明，用大约 $1,000$ 小时无标注语音估计出的预训练表征，可以明显改进基于字符的自动语音识别系统，并超过文献中最好的字符结果 Deep Speech 2，把词错误率从 $3.1\text{\,}\mathrm{\%}$ 降到 $2.43\text{\,}\mathrm{\%}$。在 TIMIT 上，预训练使我们能够达到文献中当时最好的报告结果。在只有八小时转写音频的模拟低资源设定里，相对一个只依赖标注数据的基线模型，wav2vec 最多把词错误率降低 $36\text{\,}\mathrm{\%}$（§3，§4）。

Figure 1: Illustration of pre-training from audio data $\mathcal{X}$ which is encoded with two convolutional neural networks that are stacked on top of each other. The model is optimized to solve a next time step prediction task.

图 1：从音频数据 $\mathcal{X}$ 做预训练的示意。音频先由两个堆叠的卷积神经网络编码，模型再被优化去完成下一步预测任务。

2 Pre-training Approach

2 预训练方法

Given an audio signal as input, we optimize our model (§2.1) to predict future samples from a given signal context. A common problem with these approaches is the requirement to accurately model the data distribution $p(\mathbf{x})$, which is challenging. We avoid this problem by first encoding raw speech samples $\mathbf{x}$ into a feature representation $\mathbf{z}$ at a lower temporal frequency and then implicitly model a density ratio $p(\mathbf{z}_{i+k}|\mathbf{z}_{i}\dots\mathbf{z}_{i-r})/p(\mathbf{z}_{i+k})$ similar to van den Oord et al. (2018).

给定一段音频信号作为输入，我们优化模型（§2.1），让它根据给定的信号上下文预测未来样本。这类方法常见的一个困难是，必须准确建模数据分布 $p(\mathbf{x})$，而这并不容易。我们避开这个问题的办法是：先把原始语音样本 $\mathbf{x}$ 编码成时间频率更低的特征表示 $\mathbf{z}$，再隐式地建模密度比 $p(\mathbf{z}_{i+k}|\mathbf{z}_{i}\dots\mathbf{z}_{i-r})/p(\mathbf{z}_{i+k})$，做法与 van den Oord et al. (2018) 类似。

2.1 Model

2.1 模型

Our model takes raw audio signal as input and then applies two networks. The encoder network embeds the audio signal in a latent space and the context network combines multiple time-steps of the encoder to obtain contextualized representations (Figure 1). Both networks are then used to compute the objective function (§2.2).

我们的模型以原始音频信号为输入，然后接上两个网络。编码器网络把音频信号嵌入到潜空间，上下文网络再把编码器多个时间步的输出结合起来，得到上下文化表征（图 1）。这两个网络随后都用于计算目标函数（§2.2）。

Given raw audio samples $\mathbf{x}_{i}\in\mathcal{X}$, we apply the encoder network $f:\mathcal{X}\mapsto\mathcal{Z}$ parameterized as a five-layer convolutional network (van den Oord et al., 2018). Alternatively, one could use other architectures such as the trainable frontend of  Zeghidour et al. (2018a). The encoder layers have kernel sizes $(10,8,4,4,4)$ and strides $(5,4,2,2,2)$. The output of the encoder is a low frequency feature representation $\mathbf{z}_{i}\in\mathcal{Z}$ which encodes about $30\text{\,}\mathrm{ms}$ of $16\text{\,}\mathrm{kHz}$ of audio and the striding results in representations $\mathbf{z}_{i}$ every 10ms.

给定原始音频样本 $\mathbf{x}_{i}\in\mathcal{X}$，我们用参数化为五层卷积网络的编码器网络 $f:\mathcal{X}\mapsto\mathcal{Z}$ 来处理（van den Oord et al., 2018）。也可以改用其他结构，例如 Zeghidour et al. (2018a) 的可训练前端。编码器各层的核大小为 $(10,8,4,4,4)$，步幅为 $(5,4,2,2,2)$。编码器的输出是低频特征表示 $\mathbf{z}_{i}\in\mathcal{Z}$，大约编码 $16\text{\,}\mathrm{kHz}$ 音频中 $30\text{\,}\mathrm{ms}$ 的内容；由于步幅的设置，$\mathbf{z}_{i}$ 每隔 10ms 出现一次。

Next, we apply the context network $g:\mathcal{Z}\mapsto\mathcal{C}$ to the output of the encoder network to mix multiple latent representations $\mathbf{z}_{i}\dots\mathbf{z}_{i-v}$ into a single contextualized tensor $\mathbf{c}_{i}=g(\mathbf{z}_{i}\dots\mathbf{z}_{i-v})$ for a receptive field size $v$. The context network has nine layers with kernel size three and stride one. The total receptive field of the context network is about $210\text{\,}\mathrm{ms}$.

接下来，我们把上下文网络 $g:\mathcal{Z}\mapsto\mathcal{C}$ 接到编码器的输出上，把多个潜表征 $\mathbf{z}_{i}\dots\mathbf{z}_{i-v}$ 混合成一个上下文化张量 $\mathbf{c}_{i}=g(\mathbf{z}_{i}\dots\mathbf{z}_{i-v})$，其中 $v$ 是感受野大小。上下文网络有九层，核大小为三，步幅为一。上下文网络的总感受野大约是 $210\text{\,}\mathrm{ms}$。

The layers in both the encoder and context networks consist of a causal convolution with 512 channels, a group normalization layer and a ReLU nonlinearity. We normalize both across the feature and temporal dimension for each sample which is equivalent to group normalization with a single normalization group (Wu & He, 2018). We found it important to choose a normalization scheme that is invariant to the scaling and the offset of the input. This choice resulted in representations that generalize well across datasets.

编码器与上下文网络中的各层都由 512 通道的因果卷积、一层组归一化和一个 ReLU 非线性组成。我们对每个样本在特征和时间两个维度上一起做归一化，这等价于只有一个归一化组的组归一化（Wu & He, 2018）。我们发现，选择一种对输入的缩放和平移都保持不变的归一化方案很重要。这一选择使表征能够较好地在不同数据集之间通用。

For training on larger datasets, we also consider a model variant (“wav2vec large”) with increased capacity, using two additional linear transformations in the encoder and a considerably larger context network comprised of twelve layers with increasing kernel sizes $(2,3,\dots,13)$. We found it important to introduce skip connections in the aggregator to help convergence in this case. The total receptive field in the last context network layer is hereby increased to about $810\text{\,}\mathrm{ms}$.

在更大数据集上训练时，我们还考虑一种容量更大的模型变体（“wav2vec large”）：编码器里多加两层线性变换，上下文网络则明显加大，改成十二层，核大小递增为 $(2,3,\dots,13)$。在这种情况下，我们发现在聚合器里引入跳跃连接，对收敛很有帮助。由此，上下文网络最后一层的总感受野增大到大约 $810\text{\,}\mathrm{ms}$。

2.2 Objective

2.2 目标函数

We train the model to distinguish a sample $\mathbf{z}_{i+k}$ that is k steps in the future from distractor samples $\mathbf{\tilde{z}}$ drawn from a proposal distribution $p_{n}$, by minimizing the contrastive loss for each step $k=1,\dots,K$:

我们训练模型，使它能把未来 $k$ 步处的样本 $\mathbf{z}_{i+k}$ 与从提议分布 $p_{n}$ 中抽出的干扰样本 $\mathbf{\tilde{z}}$ 区分开。对每一步 $k=1,\dots,K$，最小化如下对比损失：

$$\mathcal{L}_{k}=-\sum_{i=1}^{T-k}\Big(\log\sigma(\mathbf{z}_{i+k}^{\top}h_{k}(\mathbf{c}_{i}))+\lambda\Exp_{\mathclap{\mathbf{\tilde{z}}\sim p_{n}}}\ [\log\sigma(-\mathbf{\tilde{z}}^{\top}h_{k}(\mathbf{c}_{i}))]\ \Big)\, \tag{1}$$

where we denote the sigmoid $\sigma(x)=1/(1+\exp(-x))$, and where $\sigma(\mathbf{z}_{i+k}^{\top}h_{k}(\mathbf{c}_{i}))$ is the probability of $\mathbf{z}_{i+k}$ being the true sample. We consider a step-specific affine transformation $h_{k}(\mathbf{c}_{i})=W_{k}\mathbf{c}_{i}+\mathbf{b}_{k}$ for each step $k$, that is applied to $\mathbf{c}_{i}$ (van den Oord et al., 2018). We optimize the loss $\mathcal{L}=\sum_{k=1}^{K}\mathcal{L}_{k}$, summing (1) over different step sizes. In practice, we approximate the expectation by sampling ten negatives examples by uniformly choosing distractors from each audio sequence, i.e., $p_{n}(\mathbf{z})=\frac{1}{T}$, where $T$ is the sequence length and we set $\lambda$ to the number of negatives.[^2]

[^2]: Similar to van den Oord et al. (2018), we found that sampling negatives from different sequences and speakers yields inferior results.

其中 sigmoid 记为 $\sigma(x)=1/(1+\exp(-x))$，而 $\sigma(\mathbf{z}_{i+k}^{\top}h_{k}(\mathbf{c}_{i}))$ 是 $\mathbf{z}_{i+k}$ 为真实样本的概率。对每一步 $k$，我们考虑一个步长专用的仿射变换 $h_{k}(\mathbf{c}_{i})=W_{k}\mathbf{c}_{i}+\mathbf{b}_{k}$，作用在 $\mathbf{c}_{i}$ 上（van den Oord et al., 2018）。我们优化损失 $\mathcal{L}=\sum_{k=1}^{K}\mathcal{L}_{k}$，也就是把式（1）按不同步长加总。实践中，我们用抽样来近似这个期望：从每条音频序列里均匀抽取十个干扰样本，即 $p_{n}(\mathbf{z})=\frac{1}{T}$，其中 $T$ 是序列长度，并把 $\lambda$ 设为负例个数。

[^2]：与 van den Oord et al. (2018) 类似，我们发现从不同序列和不同说话人里抽负例，效果反而更差。

After training, we input the representations $\mathbf{c}_{i}$ produced by the context network to the acoustic model instead of log-mel filterbank features.

训练结束后，我们把上下文网络产生的表征 $\mathbf{c}_{i}$ 输入声学模型，用来替代 log-mel 滤波器组特征。

3 Experimental Setup

3 实验设置

3.1 Data

3.1 数据

We consider the following corpora: For phoneme recognition on TIMIT (Garofolo et al., 1993b) we use the standard train, dev and test split where the training data contains just over three hours of audio data. Wall Street Journal (WSJ; Garofolo et al. (1993a); Woodland et al. (1994)) comprises about 81 hours of transcribed audio data. We train on si284, validate on nov93dev and test on nov92. Librispeech (Panayotov et al., 2015) contains a total of 960 hours of clean and noisy speech for training. For pre-training, we use either the full 81 hours of the WSJ corpus, an 80 hour subset of clean Librispeech, the full 960 hour Librispeech training set or a combination of all of them.

我们考虑以下语料。TIMIT（Garofolo et al., 1993b）上的音素识别使用标准的训练、开发和测试划分，训练数据刚过三小时。华尔街日报语料（WSJ；Garofolo et al., 1993a; Woodland et al., 1994）大约有 81 小时转写音频。我们在 si284 上训练，在 nov93dev 上验证，在 nov92 上测试。Librispeech（Panayotov et al., 2015）的训练集一共有 960 小时干净和带噪语音。预训练时，我们分别使用 WSJ 全部 81 小时、Librispeech 干净子集中的 80 小时、Librispeech 全部 960 小时训练集，或者它们的合集。

To train the baseline acoustic model we compute 80 log-mel filterbank coefficients for a $25\text{\,}\mathrm{ms}$ sliding window with stride $10\text{\,}\mathrm{ms}$. Final models are evaluated in terms of both word error rate (WER) and letter error rate (LER).

训练基线声学模型时，我们在 $25\text{\,}\mathrm{ms}$、步幅 $10\text{\,}\mathrm{ms}$ 的滑动窗口上计算 80 维 log-mel 滤波器组系数。最终模型同时用词错误率（WER）和字母错误率（LER）来评估。

3.2 Acoustic Models

3.2 声学模型

We use the wav2letter+​+ toolkit for training and evaluation of acoustic models (Pratap et al., 2018). For the TIMIT task, we follow the character-based wav2letter+​+ setup of Zeghidour et al. (2018a) which uses seven consecutive blocks of convolutions (kernel size $5$ with $1,000$ channels), followed by a PReLU nonlinearity and a dropout rate of $0.7$. The final representation is projected to a 39-dimensional phoneme probability. The model is trained using the Auto Segmentation Criterion (ASG; Collobert et al., 2016)) using SGD with momentum.

我们用 wav2letter++ 工具包来训练和评估声学模型（Pratap et al., 2018）。在 TIMIT 任务上，我们沿用 Zeghidour et al. (2018a) 基于字符的 wav2letter++ 设置：连续七块卷积（核大小为 $5$，通道数为 $1,000$），后面接 PReLU 非线性和 $0.7$ 的 dropout。最后把表示投影到 39 维的音素概率。模型用自动切分准则（ASG；Collobert et al., 2016）训练，优化器是带动量的 SGD。

Our baseline for the WSJ benchmark is the wav2letter+​+ setup described by Collobert et al. (2019) which is a 17 layer model with gated convolutions (Dauphin et al., 2017). The model predicts probabilities for 31 graphemes, including the standard English alphabet, the apostrophe and period, two repetition characters (e.g. the word ann is transcribed as an1), and a silence token (|) used as word boundary.

WSJ 基准上的基线是 Collobert et al. (2019) 描述的 wav2letter++ 设置，这是一个 17 层、带门控卷积的模型（Dauphin et al., 2017）。模型为 31 个字素预测概率，包括标准英文字母、撇号和句点、两个重复字符（例如单词 ann 被转写成 an1），以及用作词边界的静音记号（|）。

All acoustic models are trained on 8 Nvidia V100 GPUs using the distributed training implementations of fairseq and wav2letter+​+. When training acoustic models on WSJ, we use plain SGD with learning rate 5.6 as well as gradient clipping (Collobert et al., 2019) and train for 1,000 epochs with a total batch size of 64 audio sequences. We use early stopping and choose models based on validation WER after evaluating checkpoints with a 4-gram language model. For TIMIT we use learning rate 0.12, momentum 0.9 and we train for $1,000$ epochs on 8 GPUs with a batch size of 16 audio sequences.

所有声学模型都在 8 块 Nvidia V100 GPU 上训练，使用 fairseq 和 wav2letter++ 的分布式训练实现。在 WSJ 上训练声学模型时，我们使用普通 SGD，学习率为 5.6，并做梯度裁剪（Collobert et al., 2019），一共训练 1,000 个 epoch，总批大小为 64 条音频序列。我们采用早停，并在用四元语法语言模型评估过检查点之后，按验证集词错误率选模型。在 TIMIT 上，学习率为 0.12，动量为 0.9，我们在 8 块 GPU 上训练 $1,000$ 个 epoch，批大小为 16 条音频序列。

			nov93dev		nov92
			LER	WER	LER	WER
Deep Speech 2 (12K h labeled speech; Amodei et al., 2016)			–	4.42	–	3.1
Trainable frontend (Zeghidour et al., 2018a)			–	6.8	–	3.5
Lattice-free MMI (Hadian et al., 2018)			–	5.66†	–	2.8†
Supervised transfer-learning (Ghahremani et al., 2017)			–	4.99†	–	2.53†
4-gram LM (Heafield et al., 2013)
Baseline	–	–	3.32	8.57	2.19	5.64
wav2vec	Librispeech	$80\text{\,}\mathrm{h}$	3.71	9.11	2.17	5.55
wav2vec	Librispeech	$960\text{\,}\mathrm{h}$	2.85	7.40	1.76	4.57
wav2vec	Libri + WSJ	$1{,}041\text{\,}\mathrm{h}$	2.91	7.59	1.67	4.61
wav2vec large	Librispeech	$960\text{\,}\mathrm{h}$	2.73	6.96	1.57	4.32
Word ConvLM (Zeghidour et al., 2018b)
Baseline	–	–	2.57	6.27	1.51	3.60
wav2vec	Librispeech	$960\text{\,}\mathrm{h}$	2.22	5.39	1.25	2.87
wav2vec large	Librispeech	$960\text{\,}\mathrm{h}$	2.13	5.16	1.02	2.53
Char ConvLM (Likhomanenko et al., 2019)
Baseline	–	–	2.77	6.67	1.53	3.46
wav2vec	Librispeech	$960\text{\,}\mathrm{h}$	2.14	5.31	1.15	2.78
wav2vec large	Librispeech	$960\text{\,}\mathrm{h}$	2.11	5.10	0.99	2.43

Table 1: Replacing log-mel filterbanks (Baseline) by pre-trained embeddings improves WSJ performance on test (nov92) and validation (nov93dev) in terms of both LER and WER. We evaluate pre-training on the acoustic data of part of clean and full Librispeech as well as the combination of all of them. † indicates results with phoneme-based models.

			nov93dev		nov92
			LER	WER	LER	WER
Deep Speech 2（12K 小时标注语音；Amodei et al., 2016）			–	4.42	–	3.1
可训练前端（Zeghidour et al., 2018a）			–	6.8	–	3.5
无格 MMI（Hadian et al., 2018）			–	5.66†	–	2.8†
有监督迁移学习（Ghahremani et al., 2017）			–	4.99†	–	2.53†
四元语法语言模型（Heafield et al., 2013）
基线	–	–	3.32	8.57	2.19	5.64
wav2vec	Librispeech	$80\text{\,}\mathrm{h}$	3.71	9.11	2.17	5.55
wav2vec	Librispeech	$960\text{\,}\mathrm{h}$	2.85	7.40	1.76	4.57
wav2vec	Libri + WSJ	$1{,}041\text{\,}\mathrm{h}$	2.91	7.59	1.67	4.61
wav2vec large	Librispeech	$960\text{\,}\mathrm{h}$	2.73	6.96	1.57	4.32
词级卷积语言模型（Zeghidour et al., 2018b）
基线	–	–	2.57	6.27	1.51	3.60
wav2vec	Librispeech	$960\text{\,}\mathrm{h}$	2.22	5.39	1.25	2.87
wav2vec large	Librispeech	$960\text{\,}\mathrm{h}$	2.13	5.16	1.02	2.53
字符卷积语言模型（Likhomanenko et al., 2019）
基线	–	–	2.77	6.67	1.53	3.46
wav2vec	Librispeech	$960\text{\,}\mathrm{h}$	2.14	5.31	1.15	2.78
wav2vec large	Librispeech	$960\text{\,}\mathrm{h}$	2.11	5.10	0.99	2.43

表 1：用预训练嵌入替换 log-mel 滤波器组（基线）后，WSJ 在测试集（nov92）和验证集（nov93dev）上的字母错误率和词错误率都有改进。我们评估了在部分干净 Librispeech、完整 Librispeech 以及二者与 WSJ 合集的音频上做预训练的效果。† 表示音素模型的结果。

3.3 Decoding

3.3 解码

For decoding the emissions from the acoustic model we use a lexicon as well as a separate language model trained on the WSJ language modeling data only. We consider a 4-gram KenLM language model (Heafield et al., 2013), a word-based convolutional language model (Collobert et al., 2019), and a character based convolutional language model (Likhomanenko et al., 2019). We decode the word sequence $\mathbf{y}$ from the output of the context network $\mathbf{c}$ or log-mel filterbanks using the beam search decoder of Collobert et al. (2019) by maximizing

为了对声学模型的发射分数解码，我们使用一部词典，以及一个只在 WSJ 语言建模数据上训练的独立语言模型。我们考虑四元语法 KenLM 语言模型（Heafield et al., 2013）、词级卷积语言模型（Collobert et al., 2019），以及基于字符的卷积语言模型（Likhomanenko et al., 2019）。我们用 Collobert et al. (2019) 的束搜索解码器，从上下文网络的输出 $\mathbf{c}$ 或 log-mel 滤波器组中解码出词序列 $\mathbf{y}$，最大化

$$\max_{\mathbf{y}}f_{\text{AM}}(\mathbf{y}|\mathbf{c})+\alpha\log p_{\text{LM}}(\mathbf{y})+\beta|\mathbf{y}|-\gamma\sum_{i=1}^{T}[\pi_{i}=`|`] \tag{2}$$

where $f_{\text{AM}}$ is the acoustic model, $p_{\text{LM}}$ is the language model, $\pi=\pi_{1},...,\pi_{L}$ are the characters of $\mathbf{y}$. Hyper-parameters $\alpha$, $\beta$ and $\gamma$ are weights for the language model, the word penalty, and the silence penalty.

其中 $f_{\text{AM}}$ 是声学模型，$p_{\text{LM}}$ 是语言模型，$\pi=\pi_{1},...,\pi_{L}$ 是 $\mathbf{y}$ 的字符。超参数 $\alpha$、$\beta$ 和 $\gamma$ 分别是语言模型、词惩罚和静音惩罚的权重。

For decoding WSJ, we tune the hyperparameters $\alpha$, $\beta$ and $\gamma$ using a random search. Finally, we decode the emissions from the acoustic model with the best parameter setting for $\alpha$, $\beta$ and $\gamma$. We use a beam size of $4,000$ and beam score threshold of $250$ for the word based language models, and a beam size of $1,500$ with beam score threshold $40$ for the character based language model.

解码 WSJ 时，我们用随机搜索来调节超参数 $\alpha$、$\beta$ 和 $\gamma$。最后，用找到的最佳 $\alpha$、$\beta$ 和 $\gamma$，对声学模型的发射分数解码。对基于词的语言模型，束大小为 $4,000$，束分数阈值为 $250$；对基于字符的语言模型，束大小为 $1,500$，束分数阈值为 $40$。

3.4 Pre-training Models

3.4 预训练模型

The pre-training models are implemented in PyTorch in the fairseq toolkit (Ott et al., 2019). We optimize them with Adam (Kingma & Ba, 2015) and a cosine learning rate schedule (Loshchilov & Hutter, 2016) annealed over 40$\mathrm{k}$ update steps for both WSJ and the clean Librispeech training datasets or over 400$\mathrm{k}$ steps for full Librispeech. We start with a learning rate of $1\text{\times}{10}^{-7}$, and then gradually warm it up for 500 updates up to $5\text{\times}{10}^{-3}$ and then decay it following the cosine curve up to $1\text{\times}{10}^{-6}$. To compute the objective, we sample ten negatives for each of the $K=12$ tasks.

预训练模型在 fairseq 工具包中用 PyTorch 实现（Ott et al., 2019）。我们用 Adam（Kingma & Ba, 2015）优化，并采用余弦学习率日程（Loshchilov & Hutter, 2016）：在 WSJ 和干净 Librispeech 训练数据上退火 40$\mathrm{k}$ 步，在完整 Librispeech 上则退火 400$\mathrm{k}$ 步。学习率从 $1\text{\times}{10}^{-7}$ 起步，再用 500 次更新逐步升温到 $5\text{\times}{10}^{-3}$，随后沿余弦曲线衰减到 $1\text{\times}{10}^{-6}$。计算目标时，我们对 $K=12$ 个任务中的每一个抽取十个负例。

We train the first wav2vec variant on 8 GPUs and put audio sequences amounting up to 1.5M frames on each GPU. Sequences are grouped by length and we crop each to a maximum size of 150$\mathrm{k}$ frames, or the length of the shortest sequence in the batch, whichever is smaller. Cropping removes speech signal from either the beginning or the end of the sequence and we randomly decide the cropping offsets for each sample; we re-sample every epoch. This is a form of data augmentation but also ensures equal length of all sequences on a GPU and removes on average $25\text{\,}\mathrm{\%}$ of the training data. After cropping, the total effective batch size across GPUs is about 556 seconds of speech. For the large model variant, we train on 16 GPUs, doubling the effective batch size.

第一种 wav2vec 变体在 8 块 GPU 上训练，每块 GPU 上放置合计最多 150 万帧的音频序列。序列按长度分组，每条序列会被裁到最多 15 万帧，或该批次中最短序列的长度，以较小者为准。裁剪会去掉序列开头或末尾的语音信号，每条样本的裁剪偏移随机决定；每个 epoch 都会重新抽样。这既是一种数据增强，也能保证同一块 GPU 上的序列等长，平均会去掉 $25\text{\,}\mathrm{\%}$ 的训练数据。裁剪之后，各 GPU 合计的有效批大小大约是 556 秒语音。对于大模型变体，我们在 16 块 GPU 上训练，有效批大小翻倍。

4 Results

4 结果

Different to van den Oord et al. (2018), we evaluate the pre-trained representations directly on downstream speech recognition tasks. We measure speech recognition performance on the WSJ benchmark and simulate various low resource setups (§4.1). We also evaluate on the TIMIT phoneme recognition task (§4.2) and ablate various modeling choices (§4.3).

与 van den Oord et al. (2018) 不同，我们直接在下游语音识别任务上评估预训练表征。我们在 WSJ 基准上测量识别表现，并模拟多种低资源设定（§4.1）。我们也在 TIMIT 音素识别任务上评估（§4.2），并对若干建模选择做消融（§4.3）。

4.1 Pre-training for the WSJ benchmark

4.1 面向 WSJ 基准的预训练

Figure 2: Pre-training substanstially improves WER in simulated low-resource setups on the audio data of WSJ compared to wav2letter+​+ with log-mel filterbanks features (Baseline). Pre-training on the audio data of the full 960 h Librispeech dataset (wav2vec Libri) performs better than pre-training on the 81 h WSJ dataset (wav2vec WSJ).

图 2：在 WSJ 音频上模拟的低资源设定中，预训练显著降低了词错误率，对比的是使用 log-mel 滤波器组特征的 wav2letter++（基线）。在完整 960 小时 Librispeech 音频上预训练（wav2vec Libri），效果好于在 81 小时 WSJ 音频上预训练（wav2vec WSJ）。

We consider pre-training on the audio data (without labels) of WSJ, part of clean Librispeech (about $80\text{\,}\mathrm{h}$) and full Librispeech as well as a combination of all datasets (§3.1). For the pre-training experiments we feed the output of the context network to the acoustic model, instead of log-mel filterbank features.

我们考虑在 WSJ 的音频（不用标签）、部分干净 Librispeech（约 $80\text{\,}\mathrm{h}$）、完整 Librispeech 以及全部数据的合集上做预训练（§3.1）。在预训练实验里，我们把上下文网络的输出送给声学模型，而不再用 log-mel 滤波器组特征。

Table 1 shows that pre-training on more data leads to better accuracy on the WSJ benchmark. Pre-trained representations can substantially improve performance over our character-based baseline which is trained on log-mel filterbank features. This shows that pre-training on unlabeled audio data can improve over the best character-based approach, Deep Speech 2 (Amodei et al., 2016), by 0.67 WER on nov92. In comparison to Hadian et al. (2018), wav2vec performs as well as their phoneme-based model and wav2vec large outperforms it by 0.37 WER. The phoneme-based approach of Ghahremani et al. (2017) pre-trains on the labeled version of Librispeech and then fine-tunes on WSJ. wav2vec large still outperforms Ghahremani et al. (2017) despite a weaker baseline model and not using Librispeech transcriptions.

表 1 表明，预训练数据越多，WSJ 基准上的精度越高。预训练表征相对我们在 log-mel 滤波器组特征上训练的字符基线，可以明显提高表现。这说明，在无标注音频上预训练，可以超过最好的字符方法 Deep Speech 2（Amodei et al., 2016），在 nov92 上把词错误率再降 0.67。与 Hadian et al. (2018) 相比，wav2vec 不逊于他们的音素模型，wav2vec large 则还要再低 0.37 的词错误率。Ghahremani et al. (2017) 的音素方法是在带标注的 Librispeech 上预训练，再在 WSJ 上微调。即便基线更弱、也没有使用 Librispeech 转写，wav2vec large 仍然超过了 Ghahremani et al. (2017)。

What is the impact of pre-trained representations with less transcribed data? In order to get a better understanding of this, we train acoustic models with different amounts of labeled training data and measure accuracy with and without pre-trained representations (log-mel filterbanks). The pre-trained representations are trained on the full Librispeech corpus and we measure accuracy in terms of WER when decoding with a 4-gram language model. Figure 2 shows that pre-training reduces WER by $36\text{\,}\mathrm{\%}$ on nov92 when only about eight hours of transcribed data is available. Pre-training only on the audio data of WSJ (wav2vec WSJ) performs worse compared to the much larger Librispeech (wav2vec Libri). This further confirms that pre-training on more data is important to good performance. Similar to Hénaff et al. (2019), we noticed that fine-tuning the embedding network does not meaningfully improve performance while substantially increasing the acoustic model training time.

转写数据更少时，预训练表征会有什么影响？为了把这一点看清楚，我们用不同数量的标注训练数据来训练声学模型，并比较有无预训练表征（对照的是 log-mel 滤波器组）时的精度。预训练表征在完整 Librispeech 上训练，解码时配四元语法语言模型，用词错误率来衡量精度。图 2 显示，当只有大约八小时转写数据时，预训练把 nov92 上的词错误率降低了 $36\text{\,}\mathrm{\%}$。只在 WSJ 的音频上预训练（wav2vec WSJ），效果不如在大得多的 Librispeech 上预训练（wav2vec Libri）。这进一步确认，预训练数据越多，对取得好成绩越重要。与 Hénaff et al. (2019) 类似，我们注意到，再微调嵌入网络并不能有意义地提高表现，却会明显增加声学模型的训练时间。

4.2 Pre-training for TIMIT

4.2 面向 TIMIT 的预训练

	dev	test
CNN + TD-filterbanks (Zeghidour et al., 2018a)	15.6	18.0
Li-GRU + MFCC (Ravanelli et al., 2018)	–	16.7 $\pm$ 0.26
Li-GRU + FBANK (Ravanelli et al., 2018)	–	15.8 $\pm$ 0.10
Li-GRU + fMLLR (Ravanelli et al., 2018)	–	14.9 $\pm$ 0.27
Baseline	16.9 $\pm$ 0.15	17.6 $\pm$ 0.11
wav2vec (Librispeech 80h)	15.5 $\pm$ 0.03	17.6 $\pm$ 0.12
wav2vec (Librispeech 960h)	13.6 $\pm$ 0.20	15.6 $\pm$ 0.23
wav2vec (Librispeech + WSJ)	12.9 $\pm$ 0.18	14.7 $\pm$ 0.42

Table 2: Results for phoneme recognition on TIMIT in terms of PER. All our models use the CNN-8L-PReLU-do0.7 architecture (Zeghidour et al., 2018a).

	开发集	测试集
CNN + TD-filterbanks（Zeghidour et al., 2018a）	15.6	18.0
Li-GRU + MFCC（Ravanelli et al., 2018）	–	16.7 $\pm$ 0.26
Li-GRU + FBANK（Ravanelli et al., 2018）	–	15.8 $\pm$ 0.10
Li-GRU + fMLLR（Ravanelli et al., 2018）	–	14.9 $\pm$ 0.27
基线	16.9 $\pm$ 0.15	17.6 $\pm$ 0.11
wav2vec（Librispeech 80 小时）	15.5 $\pm$ 0.03	17.6 $\pm$ 0.12
wav2vec（Librispeech 960 小时）	13.6 $\pm$ 0.20	15.6 $\pm$ 0.23
wav2vec（Librispeech + WSJ）	12.9 $\pm$ 0.18	14.7 $\pm$ 0.42

表 2：TIMIT 上的音素识别结果，指标为音素错误率。我们的模型全部使用 CNN-8L-PReLU-do0.7 结构（Zeghidour et al., 2018a）。

On the TIMIT task we use a 7-layer wav2letter+​+ model with high dropout (§3; Synnaeve & Dupoux (2016b)). Table 2 shows that wav2vec pre-training on Librispeech and WSJ audio data can lead to results matching the state of the art. Accuracy steadily increases with more data for pre-training and the best accuracy is achieved with the largest amount of data for pre-training.

在 TIMIT 任务上，我们使用一个 7 层、高 dropout 的 wav2letter++ 模型（§3；Synnaeve & Dupoux, 2016b）。表 2 表明，在 Librispeech 和 WSJ 音频上做 wav2vec 预训练，可以达到与当时最先进结果相当的水平。预训练数据越多，精度稳步提高，最好成绩出现在预训练数据量最大的时候。

4.3 Ablations

4.3 消融实验

negatives	dev PER	train time (h)
1	16.3	6.1
2	15.8	6.3
5	15.9	8.2
10	15.5	10.5
20	15.7	15.3

Table 3: Effect of different number of negative samples during pre-training for TIMIT on the development set.

负例数	开发集音素错误率	训练时间（小时）
1	16.3	6.1
2	15.8	6.3
5	15.9	8.2
10	15.5	10.5
20	15.7	15.3

表 3：预训练时不同负例数量对 TIMIT 开发集的影响。

Crop size	dev PER
None (Avg. 207$\mathrm{k}$)	16.3
100$\mathrm{k}$	16.1
150$\mathrm{k}$	15.5
200$\mathrm{k}$	16.0

Table 4: Effect of different crop sizes (cf. Table 3).

裁剪长度	开发集音素错误率
不限制（平均 207$\mathrm{k}$）	16.3
100$\mathrm{k}$	16.1
150$\mathrm{k}$	15.5
200$\mathrm{k}$	16.0

表 4：不同裁剪长度的影响（参见表 3）。

# Tasks	dev PER
8	15.9
12	15.5
16	15.5

Table 5: Effect of different number of tasks $K$ (cf. Table 3).

任务数	开发集音素错误率
8	15.9
12	15.5
16	15.5

表 5：不同任务数 $K$ 的影响（参见表 3）。

In this section we analyze some of the design choices we made for wav2vec. We pre-train on the 80 hour subset of clean Librispeech and evaluate on TIMIT. Table 3 shows that increasing the number of negative samples only helps up to ten samples. Thereafter, performance plateaus while training time increases. We suspect that this is because the training signal from the positive samples decreases as the number of negative samples increases. In this experiment, everything is kept equal except for the number of negative samples.

本节分析我们为 wav2vec 做出的一些设计选择。我们在干净 Librispeech 的 80 小时子集上预训练，并在 TIMIT 上评估。表 3 表明，增加负例数量只在十个样本以内有帮助。再往上，表现就趋于平稳，训练时间却会增加。我们猜想，这是因为正样本提供的训练信号会随着负例增多而变弱。在这个实验里，除了负例数量，其他设置都保持不变。

Next, we analyze the effect of data augmentation through cropping audio sequences (§3.4). When creating batches we crop sequences to a pre-defined maximum length. Table 4 shows that a crop size of 150$\mathrm{k}$ frames results in the best performance. Not restricting the maximum length (None) gives an average sequence length of about 207$\mathrm{k}$ frames and results in the worst accuracy. This is most likely because the setting provides the least amount of data augmentation.

接下来，我们分析通过裁剪音频序列来做数据增强的效果（§3.4）。组批时，我们把序列裁到预先设定的最大长度。表 4 表明，裁剪长度为 15 万帧时效果最好。不限制最大长度（None）时，平均序列长度约为 20.7 万帧，精度也最差。这很可能是因为这种设置提供的数据增强最少。

Table 5 also shows that predicting more than 12 steps ahead in the future does not result in better performance and increasing the number of steps increases training time.

表 5 还表明，把未来预测的步数加到超过 12 步，并不能带来更好的表现，而增加步数会拉长训练时间。

5 Conclusions

5 结论

We introduce wav2vec, the first application of unsupervised pre-training to speech recognition with a fully convolutional model. Our approach achieves $2.43\text{\,}\mathrm{\%}$ WER on the test set of WSJ, a result that outperforms the next best known character-based speech recognition model in the literature (Amodei et al., 2016) while using two orders of magnitude less transcribed training data. We show that more data for pre-training improves performance and that this approach not only improves resource-poor setups, but also settings where all WSJ training data is used. In future work, we will investigate different architectures which is likely to further improve performance.

我们提出 wav2vec，这是把无监督预训练用于语音识别、并采用全卷积模型的首次尝试。本方法在 WSJ 测试集上做到 $2.43\text{\,}\mathrm{\%}$ 的词错误率，超过了文献中已知的次优字符语音识别模型（Amodei et al., 2016），而所用转写训练数据要少两个数量级。我们表明，预训练数据越多，表现越好；这一方法不仅改进资源匮乏的设定，在使用全部 WSJ 训练数据时同样有效。在未来工作中，我们将探索不同的网络结构，这很可能进一步提升表现。

Acknowledgements

致谢

We thank the Speech team at FAIR, especially Jacob Kahn, Vineel Pratap and Qiantong Xu for help with wav2letter+​+ experiments, and Tatiana Likhomanenko for providing convolutional language models for our experiments.

我们感谢 FAIR 语音团队，尤其是 Jacob Kahn、Vineel Pratap 和 Qiantong Xu 在 wav2letter++ 实验上提供的帮助，以及 Tatiana Likhomanenko 为我们的实验提供卷积语言模型。

References

参考文献

Amodei et al. (2016) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In Proc. of ICML , 2016.

Baevski et al. (2019) Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. arXiv , abs/1903.07785, 2019.

Chen et al. (2018) Yi-Chen Chen, Chia-Hao Shen, Sung-Feng Huang, Hung-yi Lee, and Lin-Shan Lee. Almost-unsupervised speech recognition with close-to-zero resource based on phonetic structures learned from very small unpaired speech and text data. arXiv , abs/1810.12566, 2018.

Chorowski et al. (2019) Jan Chorowski, Ron J. Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech representation learning using wavenet autoencoders. arXiv , abs/1901.08810, 2019.

Chung et al. (2018) Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James R. Glass. Unsupervised cross-modal alignment of speech and text embedding spaces. arXiv , abs/1805.07467, 2018.

Collobert et al. (2011) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. JMLR , 2011.

Collobert et al. (2016) Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve. Wav2letter: an end-to-end convnet-based speech recognition system. arXiv , abs/1609.03193, 2016.

Collobert et al. (2019) Ronan Collobert, Awni Hannun, and Gabriel Synnaeve. A fully differentiable beam search decoder. arXiv , abs/1902.06022, 2019.

Dauphin et al. (2017) Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In Proc. of ICML , 2017.

Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. of CVPR , 2009.

Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv , abs/1810.04805, 2018.

Doersch et al. (2015) Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In Proc. of ICCV , 2015.

Edunov et al. (2019) Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained language model representations for language generation. In Proc. of NAACL , 2019.

Garofolo et al. (1993a) John S. Garofolo, David Graff, Doug Paul, and David S. Pallett. CSR-I (WSJ0) Complete LDC93S6A. Web Download. Linguistic Data Consortium , 1993a.

Garofolo et al. (1993b) John S. Garofolo, Lori F. Lamel, William M. Fisher, Jonathon G. Fiscus, David S. Pallett, and Nancy L. Dahlgren. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM. Linguistic Data Consortium , 1993b.

Ghahremani et al. (2017) Pegah Ghahremani, Vimal Manohar, Hossein Hadian, Daniel Povey, and Sanjeev Khudanpur. Investigation of transfer learning for asr using lf-mmi trained neural networks. In Proc. of ASRU , 2017.

Hadian et al. (2018) Hossein Hadian, Hossein Sameti1, Daniel Povey, and Sanjeev Khudanpur. End-to-end speech recognition using lattice-free mmi. In Proc. of Interspeech , 2018.

Heafield et al. (2013) Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. Scalable modified Kneser-Ney language model estimation. In Proc. of ACL , 2013.

Hénaff et al. (2019) Olivier J. Hénaff, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aäron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv , abs/1905.09272, 2019.

Kamper et al. (2017) Herman Kamper, Aren Jansen, and Sharon Goldwater. A segmental framework for fully-unsupervised large-vocabulary speech recognition. Comput. Speech Lang. , 46(C), November 2017.

Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR , 2015.

Kunze et al. (2017) Julius Kunze, Lous Kirsch, Ilia Kurenkov, Andreas Krug, Jens Johannsmeier, and Sebastian Stober. Transfer learning for speech recognition on a budget. In Proc. of Workshop on Representation Learning for NLP , 2017.

Lample & Conneau (2019) Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv , abs/1901.07291, 2019.

Lian et al. (2018) Zheng Lian, Ya Li, Jianhua Tao, and Jian Huang. Improving speech emotion recognition via transformer-based predictive coding through transfer learning. arXiv , abs/1811.07691, 2018.

Likhomanenko et al. (2019) Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert. Who needs words? lexicon-free speech recognition. In Proc. of Interspeech , 2019.

Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. of ECCV , 2014.

Loshchilov & Hutter (2016) Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. arXiv , abs/1608.03983, 2016.

Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Proc. of NIPS , 2013.

Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations , 2019.

Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Proc. of ICASSP , pp. 5206–5210. IEEE, 2015.

Pavllo et al. (2019) Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Proc. of CVPR , 2019.

Pratap et al. (2018) Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert. wav2letter++: The fastest open-source speech recognition system. arXiv , abs/1812.07625, 2018.

Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf , 2018.

Ravanelli & Bengio (2018) Mirco Ravanelli and Yoshua Bengio. Learning speaker representations with mutual information. arXiv , abs/1812.00271, 2018.

Ravanelli et al. (2018) Mirco Ravanelli, Philemon Brakel, Maurizio Omologo, and Yoshua Bengio. Light gated recurrent units for speech recognition. IEEE Transactions on Emerging Topics in Computational Intelligence , 2(2):92–102, 2018.

Synnaeve & Dupoux (2016a) Gabriel Synnaeve and Emmanuel Dupoux. A temporal coherence loss function for learning unsupervised acoustic embeddings. In Proc. of SLTU , 2016a.

Synnaeve & Dupoux (2016b) Gabriel Synnaeve and Emmanuel Dupoux. A temporal coherence loss function for learning unsupervised acoustic embeddings. Procedia Computer Science , 81:95–100, 2016b.

van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv , abs/1807.03748, 2018.

Vinyals et al. (2016) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 MS COCO image captioning challenge. arXiv , abs/1609.06647, 2016.

Woodland et al. (1994) Philip C Woodland, Julian J Odell, Valtcho Valtchev, and Steve J Young. Large vocabulary continuous speech recognition using htk. In Proc. of ICASSP , 1994.

Wu & He (2018) Yuxin Wu and Kaiming He. Group normalization. arXiv , abs/1803.08494, 2018.

Zeghidour et al. (2018a) Neil Zeghidour, Nicolas Usunier, Iasonas Kokkinos, Thomas Schaiz, Gabriel Synnaeve, and Emmanuel Dupoux. Learning filterbanks from raw speech for phone recognition. In Proc. of (ICASSP) , 2018a.

Zeghidour et al. (2018b) Neil Zeghidour, Qiantong Xu, Vitaliy Liptchinsky, Nicolas Usunier, Gabriel Synnaeve, and Ronan Collobert. Fully convolutional speech recognition. arXiv , abs/1812.06864, 2018b.