wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

wav2vec 2.0：语音表征自监督学习框架

Alexei Baevski    Henry Zhou    Abdelrahman Mohamed    Michael Auli    {abaevski,henryzhou7,abdo,michaelauli}@fb.com    Facebook AI

Abstract

摘要

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.[^1]

我们首次表明：仅从语音音频学习强表征，再在转写语音上微调，就能在概念更简单的同时超过最好的半监督方法。wav2vec 2.0 在潜空间掩蔽语音输入，并在联合学习的潜表征量化之上求解对比任务。使用 Librispeech 全部标注数据时，在 clean／other 测试集上达到 1.8／3.3 WER。把标注数据降到一小时时，wav2vec 2.0 超过此前在 100 小时子集上的先进结果，而所用标注数据少 100 倍。仅用十分钟标注数据、并在 53k 小时无标注数据上预训练，仍达到 4.8／8.2 WER。这表明用有限标注数据做语音识别是可行的。[^1]

1 Introduction

1 引言

Neural networks benefit from large quantities of labeled training data. However, in many settings labeled data is much harder to come by than unlabeled data: current speech recognition systems require thousands of hours of transcribed speech to reach acceptable performance which is not available for the vast majority of the nearly 7,000 languages spoken worldwide [31]. Learning purely from labeled examples does not resemble language acquisition in humans: infants learn language by listening to adults around them - a process that requires learning good representations of speech.

神经网络受益于大量标注训练数据。但在许多设定中，标注数据远比无标注数据难获得：当前语音识别系统需要数千小时转写语音才能达到可接受表现，而这对全球近 7,000 种语言中的绝大多数并不具备 [31]。纯从标注样本学习也不像人类的语言习得：婴儿通过听周围成人说话来学语言——这一过程需要学习良好的语音表征。

In machine learning, self-supervised learning has emerged as a paradigm to learn general data representations from unlabeled examples and to fine-tune the model on labeled data. This has been particularly successful for natural language processing [43, 45, 9] and is an active research area for computer vision [20, 2, 36, 19, 6].

在机器学习中，自监督学习已成为从无标注样本学习通用数据表征、再在标注数据上微调模型的范式。这在自然语言处理中尤为成功 [43, 45, 9]，在计算机视觉中也是活跃研究方向 [20, 2, 36, 19, 6]。

In this paper, we present a framework for self-supervised learning of representations from raw audio data. Our approach encodes speech audio via a multi-layer convolutional neural network and then masks spans of the resulting latent speech representations [26, 56], similar to masked language modeling [9]. The latent representations are fed to a Transformer network to build contextualized representations and the model is trained via a contrastive task where the true latent is to be distinguished from distractors [54, 49, 48, 28]  (§ 2).

本文提出从原始音频数据自监督学习表征的框架。我们的方法用多层卷积神经网络编码语音音频，再掩蔽所得潜语音表征的若干跨度 [26, 56]，类似掩蔽语言模型 [9]。潜表征送入 Transformer 网络以构建上下文化表征，模型通过对比任务训练：要求把真实潜表征与干扰项区分开 [54, 49, 48, 28]（§ 2）。

As part of training, we learn discrete speech units [53, 32, 7, 18] via a gumbel softmax [24, 5] to represent the latent representations in the contrastive task (Figure 1) which we find to be more effective than non-quantized targets. After pre-training on unlabeled speech, the model is fine-tuned on labeled data with a Connectionist Temporal Classification (CTC) loss [14, 4] to be used for downstream speech recognition tasks (§ 3)

作为训练的一部分，我们用 Gumbel softmax [24, 5] 学习离散语音单元 [53, 32, 7, 18]，以在对比任务中表示潜表征（图 1）；我们发现这比非量化目标更有效。在无标注语音上预训练后，模型用连接时序分类（CTC）损失 [14, 4] 在标注数据上微调，以用于下游语音识别任务（§ 3）。

Figure 1: Illustration of our framework which jointly learns contextualized speech representations and an inventory of discretized speech units.

图 1：我们的框架示意：联合学习上下文化语音表征与离散化语音单元词表。

Previous work learned a quantization of the data followed by a contextualized representations with a self-attention model [5, 4], whereas our approach solves both problems end-to-end. Masking parts of the input with Transformer networks for speech has been explored  [4, 26], but prior work relies either on a two-step pipeline or their model is trained by reconstructing the filter bank input features. Other related work includes learning representations from auto-encoding the input data [52, 11] or directly predicting future timesteps [8].

先前工作先学习数据的量化，再用自注意力模型学习上下文化表征 [5, 4]，而我们的方法端到端同时解决这两个问题。用 Transformer 网络掩蔽语音输入的部分内容已被探索 [4, 26]，但先前工作要么依赖两步流水线，要么通过重建滤波器组输入特征来训练。其他相关工作包括通过对输入数据自编码学习表征 [52, 11]，或直接预测未来时间步 [8]。

Our results show that jointly learning discrete speech units with contextualized representations achieves substantially better results than fixed units learned in a prior step [4]. We also demonstrate the feasibility of ultra-low resource speech recognition: when using only 10 minutes of labeled data, our approach achieves word error rate (WER) 4.8/8.2 on the clean/other test sets of Librispeech. We set a new state of the art on TIMIT phoneme recognition as well as the 100 hour clean subset of Librispeech. Moreover, when we lower the amount of labeled data to just one hour, we still outperform the previous state of the art self-training method of [42] while using 100 times less labeled data and the same amount of unlabeled data. When we use all 960 hours of labeled data from Librispeech, then our model achieves 1.8/3.3 WER (§ 4, § 5).

结果表明：联合学习离散语音单元与上下文化表征，显著优于在先前步骤中学到的固定单元 [4]。我们还展示了超低资源语音识别的可行性：仅用 10 分钟标注数据时，我们的方法在 Librispeech 的 clean／other 测试集上达到词错误率（WER）4.8／8.2。我们在 TIMIT 音素识别以及 Librispeech 100 小时 clean 子集上刷新了先进水平。此外，把标注数据降到仅一小时时，我们仍超过此前 [42] 的自训练先进方法，而所用标注数据少 100 倍、无标注数据量相同。当我们使用 Librispeech 全部 960 小时标注数据时，模型达到 1.8／3.3 WER（§ 4，§ 5）。

2 Model

2 模型

Our model is composed of a multi-layer convolutional feature encoder $f:\mathcal{X}\mapsto\mathcal{Z}$ which takes as input raw audio $\mathcal{X}$ and outputs latent speech representations $\mathbf{z}_{1},\dots,\mathbf{z}_{T}$ for $T$ time-steps. They are then fed to a Transformer $g:\mathcal{Z}\mapsto\mathcal{C}$ to build representations $\mathbf{c}_{1},\dots,\mathbf{c}_{T}$ capturing information from the entire sequence [9, 5, 4]. The output of the feature encoder is discretized to $\mathbf{q}_{t}$ with a quantization module $\mathcal{Z}\mapsto\mathcal{Q}$ to represent the targets (Figure 1) in the self-supervised objective (§ 3.2). Compared to vq-wav2vec [5], our model builds context representations over continuous speech representations and self-attention captures dependencies over the entire sequence of latent representations end-to-end.

我们的模型由多层卷积特征编码器 $f:\mathcal{X}\mapsto\mathcal{Z}$ 组成，它以原始音频 $\mathcal{X}$ 为输入，输出 $T$ 个时间步的潜语音表征 $\mathbf{z}_{1},\dots,\mathbf{z}_{T}$。它们再送入 Transformer $g:\mathcal{Z}\mapsto\mathcal{C}$，以构建捕获整段序列信息的表征 $\mathbf{c}_{1},\dots,\mathbf{c}_{T}$ [9, 5, 4]。特征编码器的输出经量化模块 $\mathcal{Z}\mapsto\mathcal{Q}$ 离散化为 $\mathbf{q}_{t}$，以在自监督目标中表示目标（图 1）（§ 3.2）。相对 vq-wav2vec [5]，我们的模型在连续语音表征上构建上下文表征，且自注意力端到端捕获整段潜表征序列上的依赖。

Feature encoder.

The encoder consists of several blocks containing a temporal convolution followed by layer normalization [1] and a GELU activation function [21]. The raw waveform input to the encoder is normalized to zero mean and unit variance. The total stride of the encoder determines the number of time-steps $T$ which are input to the Transformer (§ 4.2).

特征编码器。

编码器由若干块组成，每块含时序卷积，后接层归一化 [1] 与 GELU 激活函数 [21]。输入编码器的原始波形被归一化为零均值、单位方差。编码器的总步长决定输入 Transformer 的时间步数 $T$（§ 4.2）。

Contextualized representations with Transformers.

The output of the feature encoder is fed to a context network which follows the Transformer architecture [55, 9, 33]. Instead of fixed positional embeddings which encode absolute positional information, we use a convolutional layer similar to [37, 4, 57] which acts as relative positional embedding. We add the output of the convolution followed by a GELU to the inputs and then apply layer normalization.

用 Transformer 做上下文化表征。

特征编码器的输出送入遵循 Transformer 架构的上下文网络 [55, 9, 33]。我们不用编码绝对位置信息的固定位置嵌入，而使用类似 [37, 4, 57] 的卷积层作为相对位置嵌入。我们将卷积输出经 GELU 后加到输入上，再施加层归一化。

Quantization module.

For self-supervised training we discretize the output of the feature encoder $\mathbf{z}$ to a finite set of speech representations via product quantization [25]. This choice led to good results in prior work which learned discrete units in a first step followed by learning contextualized representations [5]. Product quantization amounts to choosing quantized representations from multiple codebooks and concatenating them. Given $G$ codebooks, or groups, with $V$ entries $e\in\mathbb{R}^{V\times d/G}$, we choose one entry from each codebook and concatenate the resulting vectors $e_{1},\dots,e_{G}$ and apply a linear transformation $\mathbb{R}^{d}\mapsto\mathbb{R}^{f}$ to obtain $\mathbf{q}\in\mathbb{R}^{f}$.

量化模块。

对自监督训练，我们通过乘积量化 [25] 把特征编码器输出 $\mathbf{z}$ 离散化为有限的语音表征集合。这一选择在先前「先学离散单元、再学上下文化表征」的工作中效果良好 [5]。乘积量化相当于从多个码本中选取量化表征并拼接。给定 $G$ 个码本（或组），每个有 $V$ 个条目 $e\in\mathbb{R}^{V\times d/G}$，我们从每个码本选一个条目，拼接所得向量 $e_{1},\dots,e_{G}$，再施加线性变换 $\mathbb{R}^{d}\mapsto\mathbb{R}^{f}$ 得到 $\mathbf{q}\in\mathbb{R}^{f}$。

The Gumbel softmax enables choosing discrete codebook entries in a fully differentiable way [16, 24, 35]. We use the straight-through estimator [26] and setup $G$ hard Gumbel softmax operations [24]. The feature encoder output $\mathbf{z}$ is mapped to $\mathbf{l}\in\mathbb{R}^{G\times V}$ logits and the probabilities for choosing the $v$-th codebook entry for group $g$ are

Gumbel softmax 使以完全可微的方式选择离散码本条目成为可能 [16, 24, 35]。我们使用直通估计器 [26]，并设置 $G$ 个硬 Gumbel softmax 运算 [24]。特征编码器输出 $\mathbf{z}$ 被映射为 $\mathbf{l}\in\mathbb{R}^{G\times V}$ 的 logits，选择第 $g$ 组第 $v$ 个码本条目的概率为

$$p_{g,v}=\frac{\exp(l_{g,v}+n_{v})/\tau}{\sum_{k=1}^{V}\exp(l_{g,k}+n_{k})/\tau}, \tag{1}$$

where $\tau$ is a non-negative temperature, $n=-\log(-\log(u))$ and $u$ are uniform samples from $\mathcal{U}(0,1)$. During the forward pass, codeword $i$ is chosen by $i=\text{argmax}_{j}p_{g,j}$ and in the backward pass, the true gradient of the Gumbel softmax outputs is used.

其中 $\tau$ 是非负温度，$n=-\log(-\log(u))$，$u$ 是来自 $\mathcal{U}(0,1)$ 的均匀样本。前向传播中，码字 $i$ 由 $i=\text{argmax}_{j}p_{g,j}$ 选定；反向传播中使用 Gumbel softmax 输出的真实梯度。

3 Training

3 训练

To pre-train the model we mask a certain proportion of time steps in the latent feature encoder space (§ 3.1), similar to masked language modeling in BERT [9]. The training objective requires identifying the correct quantized latent audio representation in a set of distractors for each masked time step (§ 3.2) and the final model is fine-tuned on the labeled data (§ 3.3).

预训练时，我们在潜特征编码器空间掩蔽一定比例的时间步（§ 3.1），类似 BERT 中的掩蔽语言建模 [9]。训练目标要求为每个被掩蔽时间步在一组干扰项中识别出正确的量化潜音频表征（§ 3.2）；最终模型在标注数据上微调（§ 3.3）。

3.1 Masking

3.1 掩蔽

We mask a proportion of the feature encoder outputs, or time steps before feeding them to the context network and replace them with a trained feature vector shared between all masked time steps; we do not mask inputs to the quantization module. To mask the latent speech representations output by the encoder, we randomly sample without replacement a certain proportion $p$ of all time steps to be starting indices and then mask the subsequent $M$ consecutive time steps from every sampled index; spans may overlap.

我们掩蔽一定比例的特征编码器输出（或时间步），再将其送入上下文网络，并用所有被掩蔽时间步共享的、可训练特征向量替换它们；我们不掩蔽量化模块的输入。为掩蔽编码器输出的潜语音表征，我们无放回地随机采样全部时间步中一定比例 $p$ 作为起始下标，再从每个采样下标起掩蔽随后 $M$ 个连续时间步；跨度可以重叠。

3.2 Objective

3.2 目标

During pre-training, we learn representations of speech audio by solving a contrastive task $\mathcal{L}_{m}{}$ which requires to identify the true quantized latent speech representation for a masked time step within a set of distractors. This is augmented by a codebook diversity loss $\mathcal{L}_{d}{}$ to encourage the model to use the codebook entries equally often.

预训练期间，我们通过求解对比任务 $\mathcal{L}_{m}{}$ 学习语音音频表征：该任务要求为被掩蔽时间步在一组干扰项中识别出真实的量化潜语音表征。此外还有码本多样性损失 $\mathcal{L}_{d}{}$，以鼓励模型更均匀地使用码本条目。

$$\mathcal{L}=\mathcal{L}_{m}{}+\alpha\mathcal{L}_{d}{} \tag{2}$$

where $\alpha$ is a tuned hyperparameter.

其中 $\alpha$ 是调谐的超参数。

Contrastive Loss.

Given context network output $\mathbf{c}_{t}$ centered over masked time step $t$, the model needs to identify the true quantized latent speech representation $\mathbf{q}_{t}$ in a set of $K+1$ quantized candidate representations $\mathbf{\tilde{q}}\in\mathbf{Q}_{t}$ which includes $\mathbf{q}_{t}$ and $K$ distractors [23, 54]. Distractors are uniformly sampled from other masked time steps of the same utterance. The loss is defined as

对比损失。

给定以被掩蔽时间步 $t$ 为中心的上下文网络输出 $\mathbf{c}_{t}$，模型需要在包含 $\mathbf{q}_{t}$ 与 $K$ 个干扰项的 $K+1$ 个量化候选表征集合 $\mathbf{\tilde{q}}\in\mathbf{Q}_{t}$ 中识别出真实量化潜语音表征 $\mathbf{q}_{t}$ [23, 54]。干扰项从同一话语的其他被掩蔽时间步均匀采样。损失定义为

$$\mathcal{L}_{m}{}=-\log\frac{\exp(sim(\mathbf{c}_{t},\mathbf{q}_{t})/\kappa)}{\sum_{\mathbf{\tilde{q}}\sim\mathbf{Q}_{t}}\exp(sim(\mathbf{c}_{t},\mathbf{\tilde{q}})/\kappa)} \tag{3}$$

where we compute the cosine similarity $sim(\mathbf{a},\mathbf{b})=\mathbf{a}^{T}\mathbf{b}/\|\mathbf{a}\|\|\mathbf{b}\|$ between context representations and quantized latent speech representations [19, 6].

其中我们计算上下文表征与量化潜语音表征之间的余弦相似度 $sim(\mathbf{a},\mathbf{b})=\mathbf{a}^{T}\mathbf{b}/\|\mathbf{a}\|\|\mathbf{b}\|$ [19, 6]。

Diversity Loss.

The contrastive task depends on the codebook to represent both positive and negative examples and the diversity loss $\mathcal{L}_{d}$ is designed to increase the use of the quantized codebook representations [10]. We encourage the equal use of the $V$ entries in each of the $G$ codebooks by maximizing the entropy of the averaged softmax distribution $\mathbf{l}$ over the codebook entries for each codebook $\bar{p}_{g}$ across a batch of utterances; the softmax disribution does not contain the gumbel noise nor a temperature:[^2]

多样性损失。

对比任务依赖码本同时表示正例与负例，多样性损失 $\mathcal{L}_{d}$ 旨在提高量化码本表征的使用率 [10]。我们通过最大化一批话语上、各码本条目平均 softmax 分布 $\mathbf{l}$ 的熵 $\bar{p}_{g}$，鼓励每个码本的 $V$ 个条目被均等使用；该 softmax 分布不含 gumbel 噪声，也不含温度：[^2]

$$\mathcal{L}_{d}{}=\frac{1}{GV}\sum_{g=1}^{G}-H(\bar{p}_{g})=\frac{1}{GV}\sum_{g=1}^{G}\sum_{v=1}^{V}\bar{p}_{g,v}\log{\bar{p}_{g,v}} \tag{4}$$

3.3 Fine-tuning

3.3 微调

Pre-trained models are fine-tuned for speech recognition by adding a randomly initialized linear projection on top of the context network into $C$ classes representing the vocabulary of the task [4]. For Librispeech, we have 29 tokens for character targets plus a word boundary token. Models are optimized by minimizing a CTC loss [14] and we apply a modified version of SpecAugment [41] by masking to time-steps and channels during training which delays overfitting and significantly improves the final error rates, especially on the Libri-light subsets with few labeled examples.

预训练模型通过在上下文网络顶部添加随机初始化的线性投影到表示任务词表的 $C$ 个类别来微调，以用于语音识别 [4]。对 Librispeech，我们有 29 个字符目标记号外加一个词边界记号。模型通过最小化 CTC 损失 [14] 优化；训练时我们对时间步与通道施加改进版 SpecAugment [41] 掩蔽，以推迟过拟合并显著改善最终错误率，尤其在标注样本很少的 Libri-light 子集上。

4 Experimental Setup

4 实验设置

4.1 Datasets

4.1 数据集

As unlabeled data we consider the Librispeech corpus [40] without transcriptions containing 960 hours of audio (LS-960) or the audio data from LibriVox (LV-60k). For the latter we follow the pre-processing of [27] resulting in 53.2k hours of audio. We fine-tune on five labeled data settings: 960 hours of transcribed Librispeech, the train-clean-100 subset comprising 100 hours (100 hours labeled), as well as the Libri-light limited resource training subsets originally extracted from Librispeech, these are train-10h (10 hours labeled), train-1h (1 hour labeled), train-10min (10 min labeled). We follow the evaluation protocol of Libri-light for these splits and evaluate on the standard Librispech dev-other/clean and test-clean/other sets.

作为无标注数据，我们考虑不含转写的 Librispeech 语料 [40]，含 960 小时音频（LS-960），或来自 LibriVox 的音频数据（LV-60k）。对后者我们遵循 [27] 的预处理，得到 53.2k 小时音频。我们在五种标注数据设定上微调：960 小时转写 Librispeech、含 100 小时的 train-clean-100 子集（100 小时标注），以及原先从 Librispeech 抽取的 Libri-light 有限资源训练子集，即 train-10h（10 小时标注）、train-1h（1 小时标注）、train-10min（10 分钟标注）。对这些划分我们遵循 Libri-light 评估协议，并在标准 Librispeech dev-other／clean 与 test-clean／other 集合上评估。

We fine-tune the pre-trained models for phoneme recognition on the TIMIT dataset [13]. It contains five hours of audio recordings with detailed phoneme labels. We use the standard train, dev and test split and follow the standard protocol of collapsing phone labels to 39 classes.

我们在 TIMIT 数据集 [13] 上微调预训练模型以做音素识别。它含五小时带详细音素标签的音频录音。我们使用标准 train、dev 与 test 划分，并遵循把音素标签折叠为 39 类的标准协议。

4.2 Pre-training

4.2 预训练

Models are implemented in fairseq [39]. For masking, we sample $p=0.065$ of all time-steps to be starting indices and mask the subsequent $M=10$ time-steps. This results in approximately 49% of all time steps to be masked with a mean span length of 14.7, or 299ms (see Appendix A for more details on masking).

模型在 fairseq [39] 中实现。掩蔽时，我们采样全部时间步中 $p=0.065$ 作为起始下标，并掩蔽随后 $M=10$ 个时间步。这导致约 49% 的时间步被掩蔽，平均跨度长度为 14.7，即 299ms（掩蔽细节见附录 A）。

The feature encoder contains seven blocks and the temporal convolutions in each block have 512 channels with strides (5,2,2,2,2,2,2) and kernel widths (10,3,3,3,3,2,2). This results in an encoder output frequency of 49 hz with a stride of about 20ms between each sample, and a receptive field of 400 input samples or 25ms of audio. The convolutional layer modeling relative positional embeddings has kernel size 128 and 16 groups.

特征编码器含七个块，每块中的时序卷积有 512 个通道，步长为 (5,2,2,2,2,2,2)，核宽为 (10,3,3,3,3,2,2)。这使编码器输出频率为 49 Hz，样本间步长约 20ms，感受野为 400 个输入样本或 25ms 音频。建模相对位置嵌入的卷积层核大小为 128，分组数为 16。

We experiment with two model configurations which use the same encoder architecture but differ in the Transformer setup: Base contains 12 transformer blocks, model dimension 768, inner dimension (FFN) 3,072 and 8 attention heads. Batches are built by cropping 250k audio samples, or 15.6sec, from each example. Crops are batched together to not exceed 1.4m samples per GPU and we train on a total of 64 V100 GPUs for 1.6 days [38]; the total batch size is 1.6h.

我们试验两种模型配置，二者使用相同编码器架构但 Transformer 设置不同：Base 含 12 个 transformer 块，模型维 768，内维（FFN）3,072，8 个注意力头。批通过从每个样本裁剪 250k 音频样本（即 15.6 秒）构建。裁剪被组批以使每 GPU 不超过 1.4m 样本；我们在共 64 块 V100 GPU 上训练 1.6 天 [38]；总批量大小为 1.6 小时。

The Large model contains 24 transformer blocks with model dimension 1,024, inner dimension 4,096 and 16 attention heads. We crop 320k audio samples, or 20sec, with a limit of 1.2m samples per GPU and train on 128 V100 GPUs over 2.3 days for Librispeech and 5.2 days for LibriVox; the total batch size is 2.7h. We use dropout 0.1 in the Transformer, at the output of the feature encoder and the input to the quantization module. Layers are dropped at a rate of 0.05 for Base and 0.2 for Large  [22, 12]; there is no layer drop for LV-60k.

Large 模型含 24 个 transformer 块，模型维 1,024，内维 4,096，16 个注意力头。我们裁剪 320k 音频样本（即 20 秒），每 GPU 上限 1.2m 样本，在 128 块 V100 GPU 上训练：Librispeech 2.3 天，LibriVox 5.2 天；总批量大小为 2.7 小时。我们在 Transformer、特征编码器输出以及量化模块输入处使用 dropout 0.1。层丢弃率对 Base 为 0.05、对 Large 为 0.2 [22, 12]；LV-60k 不做层丢弃。

We optimize with Adam [29], warming up the learning rate for the first 8% of updates to a peak of $5\text{\times}{10}^{-4}$ for Base and $3\text{\times}{10}^{-4}$ for Large, and then linearly decay it. Large trains for 250k updates, Base for 400k updates, and Large on LV-60k for 600k updates. We use weight $\alpha=0.1$ for the diversity loss Equation 2. For the quantization module we use $G=2$ and $V=320$ for both models, resulting in a theoretical maximum of 102.4k codewords. Entries are of size $d/G=128$ for Base amd $d/G=384$ for Large. The Gumbel softmax temperature $\tau$ is annealed from 2 to a minimum of 0.5 for Base and 0.1 for Large by a factor of 0.999995 at every update. The temperature in the contrastive loss (Equation 3) is set to $\kappa=0.1$. For the smaller Librispeech dataset, we regularize the model by applying an L2 penalty to the activations of the final layer of the feature encoder and scale down the gradients for the encoder by a factor of 10. We also use a slightly different encoder architecture where we do not use layer normalization, and instead of normalizing the raw waveform, the output of the first encoder layer is normalized. In the contrastive loss we use $K=100$ distractors. We choose the training checkpoint with the lowest $\mathcal{L}_{m}{}$ on the validation set.

我们用 Adam [29] 优化，前 8% 更新把学习率预热到峰值：Base 为 $5\text{\times}{10}^{-4}$，Large 为 $3\text{\times}{10}^{-4}$，随后线性衰减。Large 训练 250k 次更新，Base 400k 次，LV-60k 上的 Large 600k 次。多样性损失（式 2）权重 $\alpha=0.1$。量化模块对两种模型都用 $G=2$、$V=320$，理论最大码字数 102.4k。条目大小对 Base 为 $d/G=128$，对 Large 为 $d/G=384$。Gumbel softmax 温度 $\tau$ 每个更新按因子 0.999995 从 2 退火到最小：Base 为 0.5，Large 为 0.1。对比损失（式 3）中的温度设为 $\kappa=0.1$。对较小的 Librispeech 数据集，我们对特征编码器最后一层的激活施加 L2 惩罚，并把编码器梯度缩小 10 倍。我们还使用略有不同的编码器架构：不用层归一化；不归一化原始波形，而归一化第一层编码器输出。对比损失中使用 $K=100$ 个干扰项。我们选择验证集上 $\mathcal{L}_{m}{}$ 最低的训练检查点。

4.3 Fine-tuning

4.3 微调

After pre-training we fine-tune the learned representations on labeled data and add a randomly initialized output layer on top of the Transformer to predict characters (Librispeech/Libri-light) or phonemes (TIMIT). For Libri-light, we train three seeds with two different learning rates (2e-5 and 3e-5) for all subsets and choose the configuration with lowest WER on dev-other subset decoded with the official 4-gram language model (LM) with beam 50 and fixed model weights (LM weight 2, word insertion penalty -1). For Base on the labeled 960h subset we use a learning rate of 1e-4.

预训练后，我们在标注数据上微调所学表征，并在 Transformer 顶部添加随机初始化的输出层以预测字符（Librispeech／Libri-light）或音素（TIMIT）。对 Libri-light，我们在所有子集上用两种学习率（2e-5 与 3e-5）各训练三个种子，并用官方 4-gram 语言模型（LM）以束宽 50、固定模型权重（LM 权重 2、词插入惩罚 -1）解码，选取在 dev-other 子集上 WER 最低的配置。对 Base 在 960h 标注子集上使用学习率 1e-4。

We optimize with Adam and a tri-state rate schedule where the learning rate is warmed up for the first 10% of updates, held constant for the next 40% and then linearly decayed for the remainder. Base uses a batch size of 3.2m samples per GPU and we fine-tune on 8 GPUs, giving a total batch size of 1,600sec. Large batches 1.28m samples on each GPU and we fine-tune on 24 GPUs, resulting in an effective batch size of 1,920sec. For the first 10k updates only the output classifier is trained, after which the Transformer is also updated. The feature encoder is not trained during fine-tuning. We mask the feature encoder representations with a strategy similar to SpecAugment [41] detailed in Appendix B.

我们用 Adam 与三阶段学习率日程优化：前 10% 更新预热学习率，随后 40% 保持恒定，其余线性衰减。Base 每 GPU 批量 3.2m 样本，在 8 块 GPU 上微调，总批量 1,600 秒。Large 每 GPU 批量 1.28m 样本，在 24 块 GPU 上微调，有效批量 1,920 秒。前 10k 次更新仅训练输出分类器，之后也更新 Transformer。微调期间不训练特征编码器。我们用类似 SpecAugment [41] 的策略掩蔽特征编码器表征，细节见附录 B。

4.4 Language Models and Decoding

4.4 语言模型与解码

We consider two types of language models (LM): a 4-gram model and a Transformer [3] trained on the Librispeech LM corpus. The Transformer LM is identical to [51] and contains 20 blocks, model dimension 1,280, inner dimension 6,144 and 16 attention heads. We tune the weights of the language model (interval $[0,5]$) and a word insertion penalty ($[-5,5]$) via Bayesian optimization[^3]: we run 128 trials with beam 500 for the 4-gram LM and beam 50 for the Transformer LM and choose the best set of weights according to performance on dev-other. Test performance is measured with beam 1,500 for the n-gram LM and beam 500 for the Transformer LM. We use the beam search decoder of [44].

我们考虑两类语言模型（LM）：4-gram 模型，以及在 Librispeech LM 语料上训练的 Transformer [3]。Transformer LM 与 [51] 相同，含 20 个块，模型维 1,280，内维 6,144，16 个注意力头。我们通过贝叶斯优化[^3] 调谐语言模型权重（区间 $[0,5]$）与词插入惩罚（$[-5,5]$）：对 4-gram LM 以束宽 500、对 Transformer LM 以束宽 50 运行 128 次试验，按 dev-other 表现选取最佳权重集。测试表现对 n-gram LM 用束宽 1,500、对 Transformer LM 用束宽 500 测量。我们使用 [44] 的束搜索解码器。

5 Results

5 结果

Model	Unlabeled	LM	dev		test
data	clean	other		clean	other
10 min labeled
Discrete BERT [4]	LS-960	4-gram	15.7	24.1		16.3	25.2
Base	LS-960	4-gram	8.9	15.7		9.1	15.6
		Transf.	6.6	13.2		6.9	12.9
Large	LS-960	Transf.	6.6	10.6		6.8	10.8
	LV-60k	Transf.	4.6	7.9		4.8	8.2
1h labeled
Discrete BERT [4]	LS-960	4-gram	8.5	16.4		9.0	17.6
Base	LS-960	4-gram	5.0	10.8		5.5	11.3
		Transf.	3.8	9.0		4.0	9.3
Large	LS-960	Transf.	3.8	7.1		3.9	7.6
	LV-60k	Transf.	2.9	5.4		2.9	5.8
10h labeled
Discrete BERT [4]	LS-960	4-gram	5.3	13.2		5.9	14.1
Iter. pseudo-labeling [58]	LS-960	4-gram+Transf.	23.51	25.48		24.37	26.02
	LV-60k	4-gram+Transf.	17.00	19.34		18.03	19.92
Base	LS-960	4-gram	3.8	9.1		4.3	9.5
		Transf.	2.9	7.4		3.2	7.8
Large	LS-960	Transf.	2.9	5.7		3.2	6.1
	LV-60k	Transf.	2.4	4.8		2.6	4.9
100h labeled
Hybrid DNN/HMM [34]	-	4-gram	5.0	19.5		5.8	18.6
TTS data augm. [30]	-	LSTM				4.3	13.5
Discrete BERT [4]	LS-960	4-gram	4.0	10.9		4.5	12.1
Iter. pseudo-labeling [58]	LS-860	4-gram+Transf.	4.98	7.97		5.59	8.95
	LV-60k	4-gram+Transf.	3.19	6.14		3.72	7.11
Noisy student [42]	LS-860	LSTM	3.9	8.8		4.2	8.6
Base	LS-960	4-gram	2.7	7.9		3.4	8.0
		Transf.	2.2	6.3		2.6	6.3
Large	LS-960	Transf.	2.1	4.8		2.3	5.0
	LV-60k	Transf.	1.9	4.0		2.0	4.0

Table 1: WER on the Librispeech dev/test sets when training on the Libri-light low-resource labeled data setups of 10 min, 1 hour, 10 hours and the clean 100h subset of Librispeech. Models use either the audio of Librispeech (LS-960) or the larger LibriVox (LV-60k) as unlabeled data. We consider two model sizes: Base (95m parameters) and Large (317m parameters). Prior work used 860 unlabeled hours (LS-860) but the total with labeled data is 960 hours and comparable to our setup.

模型	无标注	LM	dev		test
数据	clean	other		clean	other
10 分钟标注
Discrete BERT [4]	LS-960	4-gram	15.7	24.1		16.3	25.2
Base	LS-960	4-gram	8.9	15.7		9.1	15.6
		Transf.	6.6	13.2		6.9	12.9
Large	LS-960	Transf.	6.6	10.6		6.8	10.8
	LV-60k	Transf.	4.6	7.9		4.8	8.2
1 小时标注
Discrete BERT [4]	LS-960	4-gram	8.5	16.4		9.0	17.6
Base	LS-960	4-gram	5.0	10.8		5.5	11.3
		Transf.	3.8	9.0		4.0	9.3
Large	LS-960	Transf.	3.8	7.1		3.9	7.6
	LV-60k	Transf.	2.9	5.4		2.9	5.8
10 小时标注
Discrete BERT [4]	LS-960	4-gram	5.3	13.2		5.9	14.1
Iter. pseudo-labeling [58]	LS-960	4-gram+Transf.	23.51	25.48		24.37	26.02
	LV-60k	4-gram+Transf.	17.00	19.34		18.03	19.92
Base	LS-960	4-gram	3.8	9.1		4.3	9.5
		Transf.	2.9	7.4		3.2	7.8
Large	LS-960	Transf.	2.9	5.7		3.2	6.1
	LV-60k	Transf.	2.4	4.8		2.6	4.9
100 小时标注
Hybrid DNN/HMM [34]	-	4-gram	5.0	19.5		5.8	18.6
TTS data augm. [30]	-	LSTM				4.3	13.5
Discrete BERT [4]	LS-960	4-gram	4.0	10.9		4.5	12.1
Iter. pseudo-labeling [58]	LS-860	4-gram+Transf.	4.98	7.97		5.59	8.95
	LV-60k	4-gram+Transf.	3.19	6.14		3.72	7.11
Noisy student [42]	LS-860	LSTM	3.9	8.8		4.2	8.6
Base	LS-960	4-gram	2.7	7.9		3.4	8.0
		Transf.	2.2	6.3		2.6	6.3
Large	LS-960	Transf.	2.1	4.8		2.3	5.0
	LV-60k	Transf.	1.9	4.0		2.0	4.0

表 1：在 Libri-light 低资源标注设定（10 分钟、1 小时、10 小时）以及 Librispeech clean 100h 子集上训练时，Librispeech dev／test 集上的 WER。模型分别用 Librispeech 音频（LS-960）或更大的 LibriVox（LV-60k）作为无标注数据。我们考虑两种模型规模：Base（95m 参数）与 Large（317m 参数）。先前工作使用 860 小时无标注（LS-860），但与标注数据合计为 960 小时，与我们的设定可比。

5.1 Low-Resource Labeled Data Evaluation

5.1 低资源标注数据评估

We first evaluate our pre-trained models in settings where the amount of labeled data is limited to get a sense of how the representations learned on unlabeled data can improve low resource settings. If a pre-trained model captures the structure of speech, then it should require few labeled examples to fine-tune it for speech recognition. The models are pre-trained on the audio data of either Librispeech (LS-960) or LibriVox (LV-60k) and most results are obtained by decoding with a Transformer language model (Transf.); Appendix C shows results with no language model at all as well as with an n-gram language model.

我们首先在标注数据有限的设定中评估预训练模型，以了解在无标注数据上学到的表征如何改善低资源设定。若预训练模型捕获了语音结构，则应只需少量标注样本即可微调用于语音识别。模型在 Librispeech（LS-960）或 LibriVox（LV-60k）的音频数据上预训练，多数结果用 Transformer 语言模型（Transf.）解码；附录 C 给出完全不用语言模型以及使用 n-gram 语言模型的结果。

The Large model pre-trained on LV-60k and fine-tuned on only 10 minutes of labeled data achieves a word error rate of 5.2/8.6 on the Librispeech clean/other test sets. Ten minutes of labeled data corresponds to just 48 recordings with an average length of 12.5 seconds. This demonstrates that ultra-low resource speech recognition is possible with self-supervised learning on unlabeled data. Our approach of jointly learning discrete units and contextualized representations clearly improves over previous work which learned quantized audio units in a separate step [4], reducing WER by a about a third.

在 LV-60k 上预训练、仅在 10 分钟标注数据上微调的 Large 模型，在 Librispeech clean／other 测试集上达到词错误率 5.2／8.6。十分钟标注数据对应仅 48 段录音，平均长度 12.5 秒。这表明借助无标注数据上的自监督学习，超低资源语音识别是可行的。我们联合学习离散单元与上下文化表征的方法，明显优于先前分步学习量化音频单元的工作 [4]，WER 约降低三分之一。

A recent iterative self-training approach [42] represents the state of the art on the clean 100 hour subset of Librispeech but it requires multiple iterations of labeling, filtering, and re-training. Our approach is simpler: we pre-train on the unlabeled data and fine-tune on the labeled data. On the 100 hour subset of Librispeech, their method achieves WER 4.2/8.6 on test-clean/other which compares to WER 2.3/5.0 with the Large model in a like for like setup, a relative WER reduction of 45%/42%.

近期迭代自训练方法 [42] 代表 Librispeech clean 100 小时子集上的先进水平，但需要多轮标注、过滤与重训。我们的方法更简单：在无标注数据上预训练，在标注数据上微调。在 Librispeech 100 小时子集上，他们的方法在 test-clean／other 上达到 WER 4.2／8.6；在同等设定下 Large 模型为 WER 2.3／5.0，相对 WER 降低 45%／42%。

When the Large model uses an order of magnitude less labeled data (10h labeled), then it still achieves WER 3.2/6.1, an error reduction of 24%/29% relative to iterative self-training. Using only a single hour of labeled data, the same model achieves WER 3.9/7.6 which improves on both test-clean and test-other by 7%/12% - with two orders of magnitude less labeled data. We note that the Libri-light data splits contain both clean and noisy data leading to better accuracy on test-other compared to test-clean. Increasing model size reduces WER on all setups with the largest improvements on test-other (Base vs. Large both on LS-960) and increasing the amount of unlabeled training data also leads to large improvements (Large LS-960 vs. LV-60k).

当 Large 模型使用少一个数量级的标注数据（10h 标注）时，仍达到 WER 3.2／6.1，相对迭代自训练错误降低 24%／29%。仅用一小时标注数据时，同一模型达到 WER 3.9／7.6，在 test-clean 与 test-other 上分别改善 7%／12%——标注数据少两个数量级。我们注意到 Libri-light 数据划分同时含干净与嘈杂数据，因此相对 test-clean，在 test-other 上精度更好。增大模型规模在所有设定上降低 WER，在 test-other 上改善最大（同在 LS-960 上的 Base 对 Large）；增加无标注训练数据量也带来大幅改善（Large LS-960 对 LV-60k）。

5.2 High-Resource Labeled Data Evaluation on Librispeech

5.2 Librispeech 上的高资源标注数据评估

In this section we evaluate the performance when large quantities of labeled speech are available to assess the effectiveness of our approach in a high resource setup. Specifically, we fine-tune the same models as before on the full 960 hours of labeled Librispeech: Base and Large pre-trained on LS-960 as well as Large pre-trained on LV-60k.

本节在有大量标注语音可用时评估表现，以考察我们的方法在高资源设定中的有效性。具体地，我们把与前文相同的模型在全部 960 小时标注 Librispeech 上微调：在 LS-960 上预训练的 Base 与 Large，以及在 LV-60k 上预训练的 Large。

Table 2shows that our approach achieves WER 1.8/3.3 on test-clean/other on the full Librispeech benchmark. This is despite a weaker baseline architecture: supervised training of our architecture achieves WER 2.1/4.6 (Large - from scratch) compared to WER 1.9/4.1 for ContextNet [17], the baseline architecture of the state of the art [42]. We use a simple Transformer with CTC which does not perform as well as seq2seq models [51].

表 2 表明，我们的方法在完整 Librispeech 基准的 test-clean／other 上达到 WER 1.8／3.3。这是在较弱的基线架构下取得的：对我们架构做监督训练达到 WER 2.1／4.6（Large - 从零开始），而先进方法 [42] 的基线架构 ContextNet [17] 为 WER 1.9／4.1。我们使用简单的带 CTC 的 Transformer，表现不如 seq2seq 模型 [51]。

Note that the vocabulary of our acoustic model (characters) does not match the vocabulary of the LM (words) which delays feedback from the LM and is likely to be detrimental. Most recent work [51, 58, 17, 42] uses the better performing word pieces [50] for both models. Moreover, our result is achieved without any data balancing such as [42]. Finally, self-training is likely complimentary to pre-training and their combination may yield even better results. Appendix E presents a detailed error analysis of our pre-trained models in various labeled data setups.

注意我们的声学模型词表（字符）与 LM 词表（词）不匹配，这会推迟来自 LM 的反馈，并可能有害。多数近期工作 [51, 58, 17, 42] 对两种模型都使用表现更好的词片段 [50]。此外，我们的结果未使用如 [42] 那样的数据平衡。最后，自训练很可能与预训练互补，二者结合可能得到更好结果。附录 E 对我们在各种标注数据设定下的预训练模型给出详细错误分析。

Model	Unlabeled	LM	dev		test
data	clean	other		clean	other
Supervised
CTC Transf [51]	-	CLM+Transf.	2.20	4.94		2.47	5.45
S2S Transf.  [51]	-	CLM+Transf.	2.10	4.79		2.33	5.17
Transf. Transducer [60]	-	Transf.	-	-		2.0	4.6
ContextNet [17]	-	LSTM	1.9	3.9		1.9	4.1
Conformer [15]	-	LSTM	2.1	4.3		1.9	3.9
Semi-supervised
CTC Transf. + PL [51]	LV-60k	CLM+Transf.	2.10	4.79		2.33	4.54
S2S Transf. + PL [51]	LV-60k	CLM+Transf.	2.00	3.65		2.09	4.11
Iter. pseudo-labeling [58]	LV-60k	4-gram+Transf.	1.85	3.26		2.10	4.01
Noisy student [42]	LV-60k	LSTM	1.6	3.4		1.7	3.4
This work
Large - from scratch	-	Transf.	1.7	4.3		2.1	4.6
Base	LS-960	Transf.	1.8	4.7		2.1	4.8
Large	LS-960	Transf.	1.7	3.9		2.0	4.1
	LV-60k	Transf.	1.6	3.0		1.8	3.3

Table 2: WER on Librispeech when using all 960 hours of labeled data (cf. Table 1).

模型	无标注	LM	dev		test
数据	clean	other		clean	other
监督
CTC Transf [51]	-	CLM+Transf.	2.20	4.94		2.47	5.45
S2S Transf.  [51]	-	CLM+Transf.	2.10	4.79		2.33	5.17
Transf. Transducer [60]	-	Transf.	-	-		2.0	4.6
ContextNet [17]	-	LSTM	1.9	3.9		1.9	4.1
Conformer [15]	-	LSTM	2.1	4.3		1.9	3.9
半监督
CTC Transf. + PL [51]	LV-60k	CLM+Transf.	2.10	4.79		2.33	4.54
S2S Transf. + PL [51]	LV-60k	CLM+Transf.	2.00	3.65		2.09	4.11
Iter. pseudo-labeling [58]	LV-60k	4-gram+Transf.	1.85	3.26		2.10	4.01
Noisy student [42]	LV-60k	LSTM	1.6	3.4		1.7	3.4
本文
Large - 从零开始	-	Transf.	1.7	4.3		2.1	4.6
Base	LS-960	Transf.	1.8	4.7		2.1	4.8
Large	LS-960	Transf.	1.7	3.9		2.0	4.1
	LV-60k	Transf.	1.6	3.0		1.8	3.3

表 2：使用全部 960 小时标注数据时 Librispeech 上的 WER（对照表 1）。

5.3 Phoneme Recognition on TIMIT

5.3 TIMIT 上的音素识别

Next, we evaluate accuracy on TIMIT phoneme recognition by fine-tuning the pre-trained models on the labeled TIMIT training data. We fine-tune as for the 10 hour subset of Libri-light but do not use a language model. Table 3 shows that our approach can achieve a new state of the art on this dataset, reducing PER by a relative 23%/29% over the next best result on the dev/test sets. Appendix D shows an analysis of how the discrete latent speech representations related to phonemes. Other recent work on pre-training which evaluates on TIMIT includes [47] who solve multiple tasks to learn good representations of speech.

接下来，我们通过在标注 TIMIT 训练数据上微调预训练模型，评估音素识别精度。微调方式与 Libri-light 的 10 小时子集相同，但不使用语言模型。表 3 表明，我们的方法可在该数据集上刷新先进水平，相对次优结果在 dev／test 集上相对降低 PER 23%／29%。附录 D 分析离散潜语音表征与音素的关系。其他在 TIMIT 上评估的预训练近期工作包括 [47]，他们通过求解多任务来学习良好语音表征。

	dev PER	test PER
CNN + TD-filterbanks [59]	15.6	18.0
PASE+ [47]	-	17.2
Li-GRU + fMLLR [46]	–	14.9
wav2vec [49]	12.9	14.7
vq-wav2vec [5]	9.6	11.6
This work (no LM)
Large (LS-960)	7.4	8.3

Table 3: TIMIT phoneme recognition accuracy in terms of phoneme error rate (PER).

	dev PER	test PER
CNN + TD-filterbanks [59]	15.6	18.0
PASE+ [47]	-	17.2
Li-GRU + fMLLR [46]	–	14.9
wav2vec [49]	12.9	14.7
vq-wav2vec [5]	9.6	11.6
本文（无 LM）
Large (LS-960)	7.4	8.3

表 3：TIMIT 音素识别精度（音素错误率 PER）。

5.4 Ablations

5.4 消融

A difference to previous work [5, 4] is that we quantize the latent audio representations only for the contrastive loss, i.e., when latents are used as targets, but not when the latents are input to the Transformer network. We motivate this choice by an ablating for which we adopt a reduced training setup to increase experimental turn around: we pre-train Base on LS-960 for 250k updates with masking probability $p=0.075$, fine-tune on train-10h for 60k updates on a single GPU with 640k samples per batch, or 40 sec of speech audio. We report the average WER and standard deviation on the concatenation of dev-clean and dev-other (dev PER) for three seeds of fine-tuning.

与先前工作 [5, 4] 的一个差别是：我们只在对比损失中量化潜音频表征，即当潜表征用作目标时量化，而潜表征作为 Transformer 网络的输入时不量化。我们用消融说明这一选择：为加快实验周转，采用缩减训练设定——在 LS-960 上对 Base 预训练 250k 次更新，掩蔽概率 $p=0.075$，在 train-10h 上单 GPU 微调 60k 次更新，每批 640k 样本，即 40 秒语音音频。我们报告三次微调种子在拼接的 dev-clean 与 dev-other（dev-clean／other）上的平均 WER 与标准差。

Table 4 shows that our strategy of continuous inputs with quantized targets (Baseline) performs best. Continuous latent speech representations retain more information to enable better context representations and quantizing the target representations leads to more robust training. Quantizing the latents both in the input and the targets performs least well, and explains the lower performance of prior work [5, 4]. Continuous targets reduce the effectiveness of self-supervised training since targets can capture detailed artifacts of the current sequence, e.g. speaker and background information, which make the task easier and prevent the model from learning general representations beneficial to speech recognition. The training accuracy of identifying the correct latent audio representation increases from 62% to 78.0% when switching from quantized to continuous targets. Continuous inputs and continuous targets perform second best but various attempts to improve it did not lead to better results (see Appendix F for this experiment and other ablations on various hyperparameters).

表 4 表明，我们「连续输入、量化目标」（基线）的策略表现最好。连续潜语音表征保留更多信息以支持更好的上下文表征，而量化目标表征使训练更稳健。在输入与目标两端都量化表现最差，这也解释了先前工作 [5, 4] 较低的表现。连续目标会削弱自监督训练的有效性，因为目标可捕获当前序列的细节伪迹，例如说话人与背景信息，从而使任务变易，并阻止模型学习对语音识别有益的通用表征。从量化目标切到连续目标时，识别正确潜音频表征的训练精度从 62% 升到 78.0%。连续输入与连续目标表现次优，但多次尝试改进都未带来更好结果（该实验及其他各类超参数消融见附录 F）。

	avg. WER	std.
Continuous inputs, quantized targets (Baseline)	7.97	0.02
Quantized inputs, quantized targets	12.18	0.41
Quantized inputs, continuous targets	11.18	0.16
Continuous inputs, continuous targets	8.58	0.08

Table 4: Average WER and standard deviation on combined dev-clean/other of Librispeech for three training seeds. We ablate quantizing the context network input and the targets in the contrastive loss.

	平均 WER	标准差
连续输入、量化目标（基线）	7.97	0.02
量化输入、量化目标	12.18	0.41
量化输入、连续目标	11.18	0.16
连续输入、连续目标	8.58	0.08

表 4：三次训练种子在 Librispeech 合并 dev-clean／other 上的平均 WER 与标准差。我们消融对比损失中对上下文网络输入与目标的量化。

6 Conclusion

6 结论

We presented wav2vec 2.0, a framework for self-supervised learning of speech representations which masks latent representations of the raw waveform and solves a contrastive task over quantized speech representations. Our experiments show the large potential of pre-training on unlabeled data for speech processing: when using only 10 minutes of labeled training data, or 48 recordings of 12.5 seconds on average, we achieve a WER of 4.8/8.2 on test-clean/other of Librispeech.

我们提出 wav2vec 2.0，一个语音表征自监督学习框架：掩蔽原始波形的潜表征，并在量化语音表征上求解对比任务。实验显示无标注数据预训练对语音处理的巨大潜力：仅用 10 分钟标注训练数据（平均 12.5 秒的 48 段录音）时，我们在 Librispeech 的 test-clean／other 上达到 WER 4.8／8.2。

Our model achieves results which achieve a new state of the art on the full Librispeech benchmark for noisy speech. On the clean 100 hour Librispeech setup, wav2vec 2.0 outperforms the previous best result while using 100 times less labeled data. The approach is also effective when large amounts of labeled data are available. We expect performance gains by switching to a seq2seq architecture and a word piece vocabulary.

我们的模型在完整 Librispeech 基准的嘈杂语音上刷新先进水平。在 clean 100 小时 Librispeech 设定上，wav2vec 2.0 超过此前最佳结果，而所用标注数据少 100 倍。当有大量标注数据可用时，该方法同样有效。我们期望改用 seq2seq 架构与词片段词表会带来进一步增益。

Broader Impact

更广泛影响

There are around 7,000 languages in the world and many more dialects. However, for most of them no speech recognition technology exists since current systems require hundreds or thousands of hours of labeled data which is hard to collect for most languages. We have shown that speech recognition models can be built with very small amounts of annotated data at very good accuracy. We hope our work will make speech recognition technology more broadly available to many more languages and dialects.

世界上约有 7,000 种语言以及更多方言。但对其中大多数并不存在语音识别技术，因为当前系统需要数百或数千小时标注数据，而这在多数语言中难以收集。我们已表明：用极少标注数据即可构建精度很高的语音识别模型。希望我们的工作能使语音识别技术更广泛地惠及更多语言与方言。

Acknowledgments and Disclosure of Funding

致谢与资金披露

We thank Tatiana Likhomanenko and Qiantong Xu for helpful discussion and their help with wav2letter integration.

感谢 Tatiana Likhomanenko 与 Qiantong Xu 的有益讨论，以及他们在 wav2letter 集成上的帮助。

References

参考文献

Ba et al. [2016] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv , 2016.

Bachman et al. [2019] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. In Proc. of NeurIPS , 2019.

Baevski and Auli [2018] A. Baevski and M. Auli. Adaptive input representations for neural language modeling. In Proc. of ICLR , 2018.

Baevski et al. [2019] A. Baevski, M. Auli, and A. Mohamed. Effectiveness of self-supervised pre-training for speech recognition. arXiv , abs/1911.03912, 2019.

Baevski et al. [2020] A. Baevski, S. Schneider, and M. Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In Proc. of ICLR , 2020.

Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv , abs/2002.05709, 2020.

Chorowski et al. [2019] J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord. Unsupervised speech representation learning using wavenet autoencoders. arXiv , abs/1901.08810, 2019.

Chung et al. [2019] Y. Chung, W. Hsu, H. Tang, and J. R. Glass. An unsupervised autoregressive model for speech representation learning. arXiv , abs/1904.03240, 2019.

Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv , abs/1810.04805, 2018.

Dieleman et al. [2018] S. Dieleman, A. van den Oord, and K. Simonyan. The challenge of realistic music generation: modelling raw audio at scale. arXiv , 2018.

Eloff et al. [2019] R. Eloff, A. Nortje, B. van Niekerk, A. Govender, L. Nortje, A. Pretorius, E. Van Biljon, E. van der Westhuizen, L. van Staden, and H. Kamper. Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. arXiv , abs/1904.07556, 2019.

Fan et al. [2020] A. Fan, E. Grave, and A. Joulin. Reducing transformer depth on demand with structured dropout. In Proc. of ICLR , 2020.

Garofolo et al. [1993] J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM. Linguistic Data Consortium , 1993.

Graves et al. [2006] A. Graves, S. Fernández, and F. Gomez. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proc. of ICML , 2006.

Gulati et al. [2020] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang. Conformer: Convolution-augmented transformer for speech recognition. arXiv , 2020.

Gumbel [1954] E. J. Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures , volume 33. US Government Printing Office, 1954.

Han et al. [2020] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv , 2020.

Harwath et al. [2020] D. Harwath, W.-N. Hsu, and J. Glass. Learning hierarchical discrete linguistic units from visually-grounded speech. In Proc. of ICLR , 2020.

He et al. [2019] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. arXiv , abs/1911.05722, 2019.

Hénaff et al. [2019] O. J. Hénaff, A. Razavi, C. Doersch, S. M. A. Eslami, and A. van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv , abs/1905.09272, 2019.

Hendrycks and Gimpel [2016] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv , 2016.

Huang et al. [2016] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. arXiv , 2016.

Hyvärinen [2010] M. G. A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. of AISTATS , 2010.

Jang et al. [2016] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv , abs/1611.01144, 2016.

Jegou et al. [2011] H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. , 33(1):117–128, Jan. 2011.

Jiang et al. [2019] D. Jiang, X. Lei, W. Li, N. Luo, Y. Hu, W. Zou, and X. Li. Improving transformer-based speech recognition using unsupervised pre-training. arXiv , abs/1910.09932, 2019.

Kahn et al. [2020] J. Kahn et al. Libri-light: A benchmark for asr with limited or no supervision. In Proc. of ICASSP , 2020.

Kawakami et al. [2020] K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. van den Oord. Learning robust and multilingual speech representations. arXiv , 2020.

Kingma and Ba [2015] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR , 2015.

Laptev et al. [2020] A. Laptev, R. Korostik, A. Svischev, A. Andrusenko, I. Medennikov, and S. Rybin. You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation. arXiv , abs/2005.07157, 2020.

Lewis et al. [2016] M. P. Lewis, G. F. Simon, and C. D. Fennig. Ethnologue: Languages of the world, nineteenth edition. Online version: http://www.ethnologue.com , 2016.

Liu et al. [2019a] A. H. Liu, T. Tu, H. yi Lee, and L. shan Lee. Towards unsupervised speech recognition and synthesis with quantized speech representation learning. arXiv , 2019a.

Liu et al. [2019b] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019b.

Lüscher et al. [2019] C. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. Ney. Rwth asr systems for librispeech: Hybrid vs attention. In Interspeech 2019 , 2019.

Maddison et al. [2014] C. J. Maddison, D. Tarlow, and T. Minka. A* sampling. In Advances in Neural Information Processing Systems , pages 3086–3094, 2014.

Misra and van der Maaten [2019] I. Misra and L. van der Maaten. Self-supervised learning of pretext-invariant representations. arXiv , 2019.

Mohamed et al. [2019] A. Mohamed, D. Okhonko, and L. Zettlemoyer. Transformers with convolutional context for ASR. arXiv , abs/1904.11660, 2019.

Ott et al. [2018] M. Ott, S. Edunov, D. Grangier, and M. Auli. Scaling neural machine translation. In Proc. of WMT , 2018.

Ott et al. [2019] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations , 2019.

Panayotov et al. [2015] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Proc. of ICASSP , pages 5206–5210. IEEE, 2015.

Park et al. [2019] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Proc. of Interspeech , 2019.

Park et al. [2020] D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le. Improved noisy student training for automatic speech recognition. arXiv , abs/2005.09629, 2020.

Peters et al. [2018] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proc. of ACL , 2018.

Pratap et al. [2019] V. Pratap, A. Hannun, Q. Xu, J. Cai, J. Kahn, G. Synnaeve, V. Liptchinsky, and R. Collobert. Wav2letter++: A fast open-source speech recognition system. In Proc. of ICASSP , 2019.

Radford et al. [2018] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf , 2018.

Ravanelli et al. [2018] M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio. Light gated recurrent units for speech recognition. IEEE Transactions on Emerging Topics in Computational Intelligence , 2(2):92–102, 2018.

Ravanelli et al. [2020] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio. Multi-task self-supervised learning for robust speech recognition. arXiv , 2020.

Rivière et al. [2020] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux. Unsupervised pretraining transfers well across languages. arXiv , abs/2002.02848, 2020.

Schneider et al. [2019] S. Schneider, A. Baevski, R. Collobert, and M. Auli. wav2vec: Unsupervised pre-training for speech recognition. In Proc. of Interspeech , 2019.

Schuster and Nakajima [2012] M. Schuster and K. Nakajima. Japanese and korean voice search. In Proc. of ICASSP , 2012.

Synnaeve et al. [2020] G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert. End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures. arXiv , abs/1911.08460, 2020.

Tjandra et al. [2019] A. Tjandra, B. Sisman, M. Zhang, S. Sakti, H. Li, and S. Nakamura. Vqvae unsupervised unit discovery and multi-scale code2spec inverter for zerospeech challenge 2019. arXiv , 1905.11449, 2019.

van den Oord et al. [2017] A. van den Oord, O. Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems , pages 6306–6315, 2017.

van den Oord et al. [2018] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv , abs/1807.03748, 2018.

Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proc. of NIPS , 2017.

Wang et al. [2020] W. Wang, Q. Tang, and K. Livescu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. arXiv , 2020.

Wu et al. [2019] F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. In Proc. of ICLR , 2019.

Xu et al. [2020] Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert. Iterative pseudo-labeling for speech recognition. arXiv , 2020.

Zeghidour et al. [2018] N. Zeghidour, N. Usunier, I. Kokkinos, T. Schaiz, G. Synnaeve, and E. Dupoux. Learning filterbanks from raw speech for phone recognition. In Proc. of ICASSP , 2018.

Zhang et al. [2020] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss. arXiv , 2020.

Appendices

附录

Appendix A Masking distribution

附录 A 掩蔽分布

When choosing which time-steps to mask, each latent speech representation in an utterance is considered a candidate starting time-step with probability $p$ where $M$ is the length of each masked span starting from the respective time step; both are hyper-parameters. Sampled starting time steps are expanded to length $M$ and spans can overlap.

选择掩蔽哪些时间步时，话语中每个潜语音表征以概率 $p$ 被视为候选起始时间步，其中 $M$ 是从相应时间步起每个被掩蔽跨度的长度；二者都是超参数。采样得到的起始时间步扩展到长度 $M$，跨度可以重叠。

For a 15 sec long audio sample, the average mask length is 14.7 time-steps, corresponding to 299ms of audio, with a median of 10 time-steps, and a maximum of about 100 time steps; about 49% of all time-steps in the sample will be masked. A plot of the corresponding mask length distribution is shown in Figure 2 and an ablation of $M$ and $p$ as well as the effect of other masking strategies is shown in Table 5. Reducing $M$ results in increased prediction accuracy for the self-supervised but the task becomes trivial when spans with length one are masked, leading to poor performance on downstream speech recognition tasks. We also consider other masking strategies: w/o overlap uniform($a$,$b$) samples for each starting index a span length $M^{s}$ from interval $a$ to $b$ and masks the subsequent $M^{s}$ time-steps taking care not to overlap with existing spans; poisson($\lambda$) and normal($\mu$, $\sigma$) sample $M^{s}$ from Poisson and normal distributions.

对 15 秒长音频样本，平均掩蔽长度为 14.7 个时间步，对应 299ms 音频，中位数为 10 个时间步，最大约 100 个时间步；样本中约 49% 的时间步会被掩蔽。相应掩蔽长度分布见图 2；$M$ 与 $p$ 的消融以及其他掩蔽策略的效果见表 5。减小 $M$ 会提高自监督任务的预测精度，但当掩蔽长度为 1 的跨度时任务变得平凡，导致下游语音识别任务表现变差。我们还考虑其他掩蔽策略：无重叠 uniform($a$,$b$) 对每个起始下标从区间 $a$ 到 $b$ 采样跨度长度 $M^{s}$，并掩蔽随后 $M^{s}$ 个时间步，同时避免与已有跨度重叠；poisson($\lambda$) 与 normal($\mu$, $\sigma$) 分别从泊松与正态分布采样 $M^{s}$。

Figure 2: Mask length distribution for a 15 second sample with $p=0.065$ and $M=10$.

图 2：对 15 秒样本、$p=0.065$、$M=10$ 的掩蔽长度分布。

	avg WER	std
Baseline ($p=0.075$)	7.97	0.02
Mask length $M=8$	8.33	0.05
Mask length $M=12$	8.19	0.08
Mask length $M=15$	8.43	0.19
Mask probability $p=0.065$	7.95	0.08
Mask probability $p=0.06$	8.14	0.22
Mask w/o overlap, uniform(1,31)	8.39	0.02
Mask w/o overlap, uniform(10,30)	9.17	0.05
Mask w/o overlap, poisson(15)	8.13	0.04
Mask w/o overlap, normal(15, 10)	8.37	0.03
Mask w/o overlap, length 10	9.15	0.02
Mask w/o overlap, length 15	9.43	0.26

Table 5: Ablations on settings for the masking strategy during pre-training. When masking without overlap, we choose starting time steps with $p=0.037$ which results in the total number of masked tokens to match the baseline.

	平均 WER	标准差
基线 ($p=0.075$)	7.97	0.02
掩蔽长度 $M=8$	8.33	0.05
掩蔽长度 $M=12$	8.19	0.08
掩蔽长度 $M=15$	8.43	0.19
掩蔽概率 $p=0.065$	7.95	0.08
掩蔽概率 $p=0.06$	8.14	0.22
无重叠掩蔽，uniform(1,31)	8.39	0.02
无重叠掩蔽，uniform(10,30)	9.17	0.05
无重叠掩蔽，poisson(15)	8.13	0.04
无重叠掩蔽，normal(15, 10)	8.37	0.03
无重叠掩蔽，长度 10	9.15	0.02
无重叠掩蔽，长度 15	9.43	0.26

表 5：预训练掩蔽策略设定的消融。无重叠掩蔽时，我们以 $p=0.037$ 选择起始时间步，使被掩蔽记号总数与基线匹配。

Appendix B Fine-tuning Setup

附录 B 微调设置

During fine-tuning we apply a masking strategy to the feature encoder outputs similar to SpecAugment [41]: we randomly choose a number of starting time steps for which a span of ten subsequent time-steps is replaced with a mask embedding; spans may overlap and we use the same masked time step embedding as during pre-training. We also mask channels by choosing a number of channels as starting indices and then expand each one to cover the subsequent 64 channels. Spans may overlap and the selected channel spans are set to zero value. We use LayerDrop [22, 12] at a rate of 0.05 for Base and 0.1 for Large during fine-tuning.

微调期间，我们对特征编码器输出施加类似 SpecAugment [41] 的掩蔽策略：随机选择若干起始时间步，用掩蔽嵌入替换随后十个时间步的跨度；跨度可重叠，并使用与预训练相同的被掩蔽时间步嵌入。我们还掩蔽通道：选择若干通道作为起始下标，再把每个扩展以覆盖随后 64 个通道。跨度可重叠，被选通道跨度置零。微调时 Base 使用 LayerDrop [22, 12] 率 0.05，Large 为 0.1。

Table 6 summarizes the fine-tuning hyper-parameter settings used for the different labeled data setup. Table 7 shows the decoding parameters used for final evaluations of the various labeled data setups for Librispeech pre-trained models and Table 8 shows decoding parameters for LibriVox.

表 6 汇总不同标注数据设定使用的微调超参数。表 7 给出 Librispeech 预训练模型在各标注数据设定最终评估所用的解码参数；表 8 给出 LibriVox 的解码参数。

	timestep mask prob.	channel mask prob.	updates
10 min	0.075	0.008	12k
1 hour	0.075	0.004	13k
10 hours	0.065	0.004	20k
100 hours	0.05	0.008	50k
960 hours	0.05	0.0016	320k
TIMIT	0.065	0.012	40k

Table 6: Fine-tuning hyperparameters

	时间步掩蔽概率	通道掩蔽概率	更新次数
10 分钟	0.075	0.008	12k
1 小时	0.075	0.004	13k
10 小时	0.065	0.004	20k
100 小时	0.05	0.008	50k
960 小时	0.05	0.0016	320k
TIMIT	0.065	0.012	40k

表 6：微调超参数

	4gram LM weight	4gram word insert.	TransLM weight	TransLM word insert.
10 min	3.23	-0.26	1.20	-1.39
1 hour	2.90	-1.62	1.15	-2.08
10 hours	2.46	-0.59	1.06	-2.32
100 hours	2.15	-0.52	0.87	-1.00
960 hours	1.74	0.52	0.92	-0.86

Table 7: Decoding parameters for Librispeech subsets for models pre-trained on Librispeech

	4gram LM 权重	4gram 词插入	TransLM 权重	TransLM 词插入
10 分钟	3.23	-0.26	1.20	-1.39
1 小时	2.90	-1.62	1.15	-2.08
10 小时	2.46	-0.59	1.06	-2.32
100 小时	2.15	-0.52	0.87	-1.00
960 小时	1.74	0.52	0.92	-0.86

表 7：在 Librispeech 上预训练的模型在各 Librispeech 子集上的解码参数

	4gram LM weight	4gram word insert.	TransLM weight	TransLM word insert.
10 min	3.86	-1.18	1.47	-2.82
1 hour	3.09	-2.33	1.33	-0.69
10 hours	2.12	-0.90	0.94	-1.05
100 hours	2.15	-0.52	0.87	-1.00
960 hours	1.57	-0.64	0.90	-0.31

Table 8: Decoding parameters for Librispeech subsets for models pre-trained on Librivox.

	4gram LM 权重	4gram 词插入	TransLM 权重	TransLM 词插入
10 分钟	3.86	-1.18	1.47	-2.82
1 小时	3.09	-2.33	1.33	-0.69
10 小时	2.12	-0.90	0.94	-1.05
100 小时	2.15	-0.52	0.87	-1.00
960 小时	1.57	-0.64	0.90	-0.31

表 8：在 LibriVox 上预训练的模型在各 Librispeech 子集上的解码参数。

Appendix C Full results for Libri-light and Librispeech

附录 C Libri-light 与 Librispeech 完整结果

Model	Unlabeled	LM	dev		test
data	clean	other		clean	other
10 min labeled
Base	LS-960	None	46.1	51.5		46.9	50.9
		4-gram	8.9	15.7		9.1	15.6
		Transf.	6.6	13.2		6.9	12.9
Large	LS-960	None	43.0	46.3		43.5	45.3
		4-gram	8.6	12.9		8.9	13.1
		Transf.	6.6	10.6		6.8	10.8
Large	LV-60k	None	38.3	41.0		40.2	38.7
		4-gram	6.3	9.8		6.6	10.3
		Transf.	4.6	7.9		4.8	8.2
1h labeled
Base	LS-960	None	24.1	29.6		24.5	29.7
		4-gram	5.0	10.8		5.5	11.3
		Transf.	3.8	9.0		4.0	9.3
Large	LS-960	None	21.6	25.3		22.1	25.3
		4-gram	4.8	8.5		5.1	9.4
		Transf.	3.8	7.1		3.9	7.6
Large	LV-60k	None	17.3	20.6		17.2	20.3
		4-gram	3.6	6.5		3.8	7.1
		Transf.	2.9	5.4		2.9	5.8
10h labeled
Base	LS-960	None	10.9	17.4		11.1	17.6
		4-gram	3.8	9.1		4.3	9.5
		Transf.	2.9	7.4		3.2	7.8
Large	LS-960	None	8.1	12.0		8.0	12.1
		4-gram	3.4	6.9		3.8	7.3
		Transf.	2.9	5.7		3.2	6.1
Large	LV-60k	None	6.3	9.8		6.3	10.0
		4-gram	2.6	5.5		3.0	5.8
		Transf.	2.4	4.8		2.6	4.9
100h labeled
Base	LS-960	None	6.1	13.5		6.1	13.3
		4-gram	2.7	7.9		3.4	8.0
		Transf.	2.2	6.3		2.6	6.3
Large	LS-960	None	4.6	9.3		4.7	9.0
		4-gram	2.3	5.7		2.8	6.0
		Transf.	2.1	4.8		2.3	5.0
Large	LV-60k	None	3.3	6.5		3.1	6.3
		4-gram	1.8	4.5		2.3	4.6
		Transf.	1.9	4.0		2.0	4.0

Table 9: WER on the Librispeech dev/test sets when training on the Libri-light low-resource labeled data setups (cf. Table 1).

模型	无标注	LM	dev		test
数据	clean	other		clean	other
10 分钟标注
Base	LS-960	None	46.1	51.5		46.9	50.9
		4-gram	8.9	15.7		9.1	15.6
		Transf.	6.6	13.2		6.9	12.9
Large	LS-960	None	43.0	46.3		43.5	45.3
		4-gram	8.6	12.9		8.9	13.1
		Transf.	6.6	10.6		6.8	10.8
Large	LV-60k	None	38.3	41.0		40.2	38.7
		4-gram	6.3	9.8		6.6	10.3
		Transf.	4.6	7.9		4.8	8.2
1 小时标注
Base	LS-960	None	24.1	29.6		24.5	29.7
		4-gram	5.0	10.8		5.5	11.3
		Transf.	3.8	9.0		4.0	9.3
Large	LS-960	None	21.6	25.3		22.1	25.3
		4-gram	4.8	8.5		5.1	9.4
		Transf.	3.8	7.1		3.9	7.6
Large	LV-60k	None	17.3	20.6		17.2	20.3
		4-gram	3.6	6.5		3.8	7.1
		Transf.	2.9	5.4		2.9	5.8
10 小时标注
Base	LS-960	None	10.9	17.4		11.1	17.6
		4-gram	3.8	9.1		4.3	9.5
		Transf.	2.9	7.4		3.2	7.8
Large	LS-960	None	8.1	12.0		8.0	12.1
		4-gram	3.4	6.9		3.8	7.3
		Transf.	2.9	5.7		3.2	6.1
Large	LV-60k	None	6.3	9.8		6.3	10.0
		4-gram	2.6	5.5		3.0	5.8
		Transf.	2.4	4.8		2.6	4.9
100 小时标注
Base	LS-960	None	6.1	13.5		6.1	13.3
		4-gram	2.7	7.9		3.4	8.0
		Transf.	2.2	6.3		2.6	6.3
Large	LS-960	None	4.6	9.3		4.7	9.0
		4-gram	2.3	5.7		2.8	6.0
		Transf.	2.1	4.8		2.3	5.0
Large	LV-60k	None	3.3	6.5		3.1	6.3
		4-gram	1.8	4.5		2.3	4.6
		Transf.	1.9	4.0		2.0	4.0

表 9：在 Libri-light 低资源标注设定上训练时 Librispeech dev／test 集上的 WER（对照表 1）。

Model	Unlabeled	LM	dev		test
data	clean	other		clean	other
Large - from scratch	-	None	2.8	7.6		3.0	7.7
	-	4-gram	1.8	5.4		2.6	5.8
	-	Transf.	1.7	4.3		2.1	4.6
Base	LS-960	None	3.2	8.9		3.4	8.5
		4-gram	2.0	5.9		2.6	6.1
		Transf.	1.8	4.7		2.1	4.8
Large	LS-960	None	2.6	6.5		2.8	6.3
		4-gram	1.7	4.6		2.3	5.0
		Transf.	1.7	3.9		2.0	4.1
Large	LV-60k	None	2.1	4.5		2.2	4.5
		4-gram	1.4	3.5		2.0	3.6
		Transf.	1.6	3.0		1.8	3.3

Table 10: WER on Librispeech when using all 960 hours of Librispeech as labeled data (cf. Table 2).

模型	无标注	LM	dev		test
数据	clean	other		clean	other
Large - 从零开始	-	None	2.8	7.6		3.0	7.7
	-	4-gram	1.8	5.4		2.6	5.8
	-	Transf.	1.7	4.3		2.1	4.6
Base	LS-960	None	3.2	8.9		3.4	8.5
		4-gram	2.0	5.9		2.6	6.1
		Transf.	1.8	4.7		2.1	4.8
Large	LS-960	None	2.6	6.5		2.8	6.3
		4-gram	1.7	4.6		2.3	5.0
		Transf.	1.7	3.9		2.0	4.1
Large	LV-60k	None	2.1	4.5		2.2	4.5
		4-gram	1.4	3.5		2.0	3.6
		Transf.	1.6	3.0		1.8	3.3

表 10：使用全部 960 小时 Librispeech 作为标注数据时的 WER（对照表 2）。

Appendix D Analysis of Discrete Latent Speech Representations

附录 D 离散潜语音表征分析

Next, we investigate whether the discrete latent speech representations $\mathbf{q}_{t}$ learned by the quantizer relate to phonetic information: Using Large pre-trained on LV-60k and without any fine-tuning, we compute the discrete latents for the training data of TIMIT and compute the co-occurrence between human annotated phonemes and the latents. Ties are broken by choosing the phoneme which is most represented in the receptive field of $\mathbf{q}_{t}$. The training data contains 3696 utterances of average length 13.6 sec, or 563k discrete latents.

接下来，我们考察量化器学到的离散潜语音表征 $\mathbf{q}_{t}$ 是否与语音信息相关：使用在 LV-60k 上预训练、未经任何微调的 Large，我们计算 TIMIT 训练数据的离散潜变量，并计算人工标注音素与潜变量之间的共现。平局时选择在 $\mathbf{q}_{t}$ 感受野中占比最大的音素。训练数据含 3696 段话语，平均长度 13.6 秒，即 563k 个离散潜变量。

Figure 3plots $P(phoneme|\mathbf{q}_{t})$ and shows that many discrete latents appear to specialize in specific phonetic sounds. The silence phoneme (bcl) represents 22% of all human annotated speech data and is therefore also modeled by many different latents.

图 3 绘制 $P(phoneme|\mathbf{q}_{t})$，表明许多离散潜变量似乎专精于特定语音。静音音素（bcl）占全部人工标注语音数据的 22%，因此也被许多不同潜变量建模。

Figure 3: Visualization of the co-occurrence between discrete latent speech representations and phonemes. We plot the conditional probability $P(phoneme|\mathbf{q}_{t})$ on TIMIT train data. The y-axis shows the collapsed 39 classes of phonemes and the x-axis is over the different discrete latents.

图 3：离散潜语音表征与音素共现的可视化。我们在 TIMIT 训练数据上绘制条件概率 $P(phoneme|\mathbf{q}_{t})$。纵轴为折叠后的 39 类音素，横轴为不同离散潜变量。

Appendix E Speech Recognition Error Analysis

附录 E 语音识别错误分析

In this section we study the most common errors our models make when fine-tuned on different amounts of labeled data (Table 11). We also show transcriptions of a few relatively challenging utterances from the dev-clean subset of Librispeech (Table 12).

本节研究模型在不同标注数据量上微调时最常见的错误（表 11）。我们还展示 Librispeech 的 dev-clean 子集中若干相对困难话语的转写（表 12）。

We consider models with no lexicon or no language model decoding, marked None in Table 9: Larger capacity decreases error rates: Large on LS-960 improves the word error rate on dev-clean from 46.1 to 43 compared to Base. Increasing the amount of unlabeled training data further decreases the error rate to 33.8 for Large on LS-960.

我们考虑无词典或无语言模型解码的模型，在表 9 中记为 None：更大容量降低错误率——相对 Base，在 LS-960 上的 Large 把 dev-clean 上的词错误率从 46.1 降到 43。增加无标注训练数据量进一步把错误率降到 LS-960 上 Large 的 33.8。

In the ten minute labeled data setup, the model is still able to recognize basic units of speech: Table 11 shows that most errors are around spelling of words, e.g., omitting silent characters such as could $\rightarrow$ coud, know $\rightarrow$ now, or ignoring repeated letters such as still $\rightarrow$ stil, little $\rightarrow$ litle. The Large LV-60k model achieves WER 38.3 on dev-clean and adding a Transformer language model enables to choose more likely pronunciations during the search and gives a large WER improvement to 5.0.

在十分钟标注数据设定中，模型仍能识别基本语音单元：表 11 表明多数错误围绕单词拼写，例如省略静音字母如 could $\rightarrow$ coud、know $\rightarrow$ now，或忽略重复字母如 still $\rightarrow$ stil、little $\rightarrow$ litle。Large LV-60k 模型在 dev-clean 上达到 WER 38.3；加入 Transformer 语言模型使搜索中能选择更可能的发音，并把 WER 大幅改善到 5.0。

The ten minute models without lexicon and language model tend to spell words phonetically and omit repeated letters, e.g., will $\rightarrow$ wil (Table 11). Spelling errors decrease with more labeled data: with one hour of labeled data, slightly less common words move into the list of the most frequent errors, e.g., heaven and food are spelled phonetically. At ten hours, top errors include articles, e.g., a, the which are a common source of errors in speech recognition in general. There are also alternative spellings, color vs. colour as well as relatively rare words including person names, still spelled phonetically, e.g., phoebe $\rightarrow$ feeby.

十分钟、无词典与语言模型的模型倾向于按语音拼写单词并省略重复字母，例如 will $\rightarrow$ wil（表 11）。拼写错误随标注数据增多而减少：有一小时标注数据时，略不常见的词进入最频繁错误列表，例如 heaven 与 food 按语音拼写。到十小时，顶部错误包含冠词，例如 a、the，这在语音识别中一般也是常见错误来源。也有替拼，color 对 colour，以及相对罕见的词（含人名）仍按语音拼写，例如 phoebe $\rightarrow$ feeby。

At 100 hours, person names dominate the most frequent errors: phoebe $\rightarrow$ phebe, along with incorrect spacing anyone $\rightarrow$ any one, awhile $\rightarrow$ a while. Finally at 960 hours the word error rate falls to 2% and top errors are mostly articles, incorrect splits, and some very rare words or names such as deucalion or gryce.

到 100 小时，人名主导最频繁错误：phoebe $\rightarrow$ phebe，以及不正确分词 anyone $\rightarrow$ any one、awhile $\rightarrow$ a while。最终到 960 小时，词错误率降到 2%，顶部错误多为冠词、不正确切分，以及一些非常罕见的词或名字如 deucalion 或 gryce。

The “from scratch” 960 hour model has a similar word error rate as the 100 hour pre-trained model and displays a similar pattern of errors.

「从零开始」的 960 小时模型与 100 小时预训练模型有相近的词错误率，并呈现相似的错误模式。

The pre-trained speech representations can be easily adapted to recognize specific sounds while fine-tuning grounds these representations to the actual spelling.

预训练语音表征可轻易适配以识别特定声音，而微调把这些表征锚定到实际拼写。

10m Large LV-60k	1h Large LV-60k	10h Large LV-60k
all $\rightarrow$ al (181)	too $\rightarrow$ to (26)	in $\rightarrow$ and (15)
are $\rightarrow$ ar (115)	until $\rightarrow$ untill (24)	a $\rightarrow$ the (11)
will $\rightarrow$ wil (100)	new $\rightarrow$ knew (22)	o $\rightarrow$ oh (10)
you $\rightarrow$ yo (90)	door $\rightarrow$ dor (18)	and $\rightarrow$ in (9)
one $\rightarrow$ on (89)	says $\rightarrow$ sais (18)	mode $\rightarrow$ mod (9)
two $\rightarrow$ to (81)	soul $\rightarrow$ sol (17)	ursus $\rightarrow$ ersus (9)
well $\rightarrow$ wel (80)	bread $\rightarrow$ bred (16)	tom $\rightarrow$ tome (8)
been $\rightarrow$ ben (73)	poor $\rightarrow$ pore (16)	randal $\rightarrow$ randol (7)
upon $\rightarrow$ apon (73)	a $\rightarrow$ the (13)	the $\rightarrow$ a (7)
good $\rightarrow$ god (67)	either $\rightarrow$ ither (13)	color $\rightarrow$ colour (6)
see $\rightarrow$ se (66)	food $\rightarrow$ fud (13)	flour $\rightarrow$ flower (6)
we $\rightarrow$ whe (60)	doubt $\rightarrow$ dout (12)	phoebe $\rightarrow$ feeby (6)
little $\rightarrow$ litle (54)	earth $\rightarrow$ erth (12)	an $\rightarrow$ and (5)
great $\rightarrow$ grate (53)	led $\rightarrow$ lead (12)	cucumbers $\rightarrow$ cucombers (5)
your $\rightarrow$ yor (53)	sea $\rightarrow$ see (12)	egg $\rightarrow$ eg (5)
could $\rightarrow$ coud (51)	thee $\rightarrow$ the (12)	macklewain $\rightarrow$ macklewaine (5)
here $\rightarrow$ hear (51)	tom $\rightarrow$ tome (12)	magpie $\rightarrow$ magpi (5)
know $\rightarrow$ now (45)	add $\rightarrow$ ad (11)	milner $\rightarrow$ millner (5)
there $\rightarrow$ ther (45)	good $\rightarrow$ god (11)	stacy $\rightarrow$ staci (5)
three $\rightarrow$ thre (45)	heaven $\rightarrow$ heven (11)	trevelyan $\rightarrow$ trevellion (5)
still $\rightarrow$ stil (42)	mary $\rightarrow$ marry (11)	verloc $\rightarrow$ verlock (5)
off $\rightarrow$ of (40)	randal $\rightarrow$ randel (11)	ann $\rightarrow$ an (4)
don’t $\rightarrow$ dont (37)	answered $\rightarrow$ ansered (10)	anyone $\rightarrow$ one (4)
shall $\rightarrow$ shal (36)	blood $\rightarrow$ blod (10)	apartment $\rightarrow$ appartment (4)
little $\rightarrow$ litl (35)	bozzle $\rightarrow$ bosel (10)	basin $\rightarrow$ bason (4)
100h Large LV-60k	960h Large LV-60k	960h Large from scratch
a $\rightarrow$ the (13)	a $\rightarrow$ the (12)	and $\rightarrow$ in (20)
and $\rightarrow$ in (10)	and $\rightarrow$ in (9)	a $\rightarrow$ the (16)
in $\rightarrow$ and (10)	macklewain $\rightarrow$ mackelwaine (7)	in $\rightarrow$ and (13)
o $\rightarrow$ oh (8)	in $\rightarrow$ and (6)	the $\rightarrow$ a (10)
minnetaki $\rightarrow$ minnitaki (7)	o $\rightarrow$ oh (6)	in $\rightarrow$ an (8)
randal $\rightarrow$ randall (7)	bozzle $\rightarrow$ bosell (5)	and $\rightarrow$ an (5)
christie $\rightarrow$ cristy (6)	criss $\rightarrow$ chris (5)	clarke $\rightarrow$ clark (4)
macklewain $\rightarrow$ mackelwane (6)	bozzle $\rightarrow$ bosel (4)	grethel $\rightarrow$ gretel (4)
randal $\rightarrow$ randoll (6)	clarke $\rightarrow$ clark (4)	macklewain $\rightarrow$ mackelwaine (4)
bozzle $\rightarrow$ bosall (5)	colored $\rightarrow$ coloured (4)	this $\rightarrow$ the (4)
kaliko $\rightarrow$ calico (5)	grethel $\rightarrow$ gretel (4)	an $\rightarrow$ and (3)
trevelyan $\rightarrow$ trevelian (5)	lige $\rightarrow$ lyge (4)	anyone $\rightarrow$ one (3)
an $\rightarrow$ and (4)	the $\rightarrow$ a (4)	bozzle $\rightarrow$ basell (3)
and $\rightarrow$ an (4)	and $\rightarrow$ an (3)	buns $\rightarrow$ bunds (3)
anyone $\rightarrow$ one (4)	ann $\rightarrow$ marianne (3)	carrie $\rightarrow$ carry (3)
bozzle $\rightarrow$ bozall (4)	butte $\rightarrow$ bute (3)	criss $\rightarrow$ chris (3)
clarke $\rightarrow$ clark (4)	color $\rightarrow$ colour (3)	he’s $\rightarrow$ is (3)
gryce $\rightarrow$ grice (4)	deucalion $\rightarrow$ ducalion (3)	his $\rightarrow$ is (3)
i’m $\rightarrow$ am (4)	forcemeat $\rightarrow$ meat (3)	honor $\rightarrow$ honour (3)
in $\rightarrow$ ind (4)	gryce $\rightarrow$ grice (3)	lattimer $\rightarrow$ latimer (3)
letty $\rightarrow$ lettie (4)	honor $\rightarrow$ honour (3)	millet $\rightarrow$ mellet (3)
phoebe $\rightarrow$ phebe (4)	kearny $\rightarrow$ kirney (3)	pyncheon $\rightarrow$ pension (3)
the $\rightarrow$ a (4)	nuova $\rightarrow$ noiva (3)	tad $\rightarrow$ ted (3)
ann $\rightarrow$ anne (3)	thing $\rightarrow$ anything (3)	thing $\rightarrow$ anything (3)
awhile $\rightarrow$ while (3)	this $\rightarrow$ the (3)	trevelyan $\rightarrow$ trevelian (3)

Table 11: Top word errors for models trained on 10m, 1h and 10h, 100h, 960h of labeled data and decoded on the Librispeech dev-clean subset without a language model or lexicon (see Table 9 and Table 10 - None). In brackets is the total number of occurrences of each error.

10m Large LV-60k	1h Large LV-60k	10h Large LV-60k
all $\rightarrow$ al (181)	too $\rightarrow$ to (26)	in $\rightarrow$ and (15)
are $\rightarrow$ ar (115)	until $\rightarrow$ untill (24)	a $\rightarrow$ the (11)
will $\rightarrow$ wil (100)	new $\rightarrow$ knew (22)	o $\rightarrow$ oh (10)
you $\rightarrow$ yo (90)	door $\rightarrow$ dor (18)	and $\rightarrow$ in (9)
one $\rightarrow$ on (89)	says $\rightarrow$ sais (18)	mode $\rightarrow$ mod (9)
two $\rightarrow$ to (81)	soul $\rightarrow$ sol (17)	ursus $\rightarrow$ ersus (9)
well $\rightarrow$ wel (80)	bread $\rightarrow$ bred (16)	tom $\rightarrow$ tome (8)
been $\rightarrow$ ben (73)	poor $\rightarrow$ pore (16)	randal $\rightarrow$ randol (7)
upon $\rightarrow$ apon (73)	a $\rightarrow$ the (13)	the $\rightarrow$ a (7)
good $\rightarrow$ god (67)	either $\rightarrow$ ither (13)	color $\rightarrow$ colour (6)
see $\rightarrow$ se (66)	food $\rightarrow$ fud (13)	flour $\rightarrow$ flower (6)
we $\rightarrow$ whe (60)	doubt $\rightarrow$ dout (12)	phoebe $\rightarrow$ feeby (6)
little $\rightarrow$ litle (54)	earth $\rightarrow$ erth (12)	an $\rightarrow$ and (5)
great $\rightarrow$ grate (53)	led $\rightarrow$ lead (12)	cucumbers $\rightarrow$ cucombers (5)
your $\rightarrow$ yor (53)	sea $\rightarrow$ see (12)	egg $\rightarrow$ eg (5)
could $\rightarrow$ coud (51)	thee $\rightarrow$ the (12)	macklewain $\rightarrow$ macklewaine (5)
here $\rightarrow$ hear (51)	tom $\rightarrow$ tome (12)	magpie $\rightarrow$ magpi (5)
know $\rightarrow$ now (45)	add $\rightarrow$ ad (11)	milner $\rightarrow$ millner (5)
there $\rightarrow$ ther (45)	good $\rightarrow$ god (11)	stacy $\rightarrow$ staci (5)
three $\rightarrow$ thre (45)	heaven $\rightarrow$ heven (11)	trevelyan $\rightarrow$ trevellion (5)
still $\rightarrow$ stil (42)	mary $\rightarrow$ marry (11)	verloc $\rightarrow$ verlock (5)
off $\rightarrow$ of (40)	randal $\rightarrow$ randel (11)	ann $\rightarrow$ an (4)
don’t $\rightarrow$ dont (37)	answered $\rightarrow$ ansered (10)	anyone $\rightarrow$ one (4)
shall $\rightarrow$ shal (36)	blood $\rightarrow$ blod (10)	apartment $\rightarrow$ appartment (4)
little $\rightarrow$ litl (35)	bozzle $\rightarrow$ bosel (10)	basin $\rightarrow$ bason (4)
100h Large LV-60k	960h Large LV-60k	960h Large from scratch
a $\rightarrow$ the (13)	a $\rightarrow$ the (12)	and $\rightarrow$ in (20)
and $\rightarrow$ in (10)	and $\rightarrow$ in (9)	a $\rightarrow$ the (16)
in $\rightarrow$ and (10)	macklewain $\rightarrow$ mackelwaine (7)	in $\rightarrow$ and (13)
o $\rightarrow$ oh (8)	in $\rightarrow$ and (6)	the $\rightarrow$ a (10)
minnetaki $\rightarrow$ minnitaki (7)	o $\rightarrow$ oh (6)	in $\rightarrow$ an (8)
randal $\rightarrow$ randall (7)	bozzle $\rightarrow$ bosell (5)	and $\rightarrow$ an (5)
christie $\rightarrow$ cristy (6)	criss $\rightarrow$ chris (5)	clarke $\rightarrow$ clark (4)
macklewain $\rightarrow$ mackelwane (6)	bozzle $\rightarrow$ bosel (4)	grethel $\rightarrow$ gretel (4)
randal $\rightarrow$ randoll (6)	clarke $\rightarrow$ clark (4)	macklewain $\rightarrow$ mackelwaine (4)
bozzle $\rightarrow$ bosall (5)	colored $\rightarrow$ coloured (4)	this $\rightarrow$ the (4)
kaliko $\rightarrow$ calico (5)	grethel $\rightarrow$ gretel (4)	an $\rightarrow$ and (3)
trevelyan $\rightarrow$ trevelian (5)	lige $\rightarrow$ lyge (4)	anyone $\rightarrow$ one (3)
an $\rightarrow$ and (4)	the $\rightarrow$ a (4)	bozzle $\rightarrow$ basell (3)
and $\rightarrow$ an (4)	and $\rightarrow$ an (3)	buns $\rightarrow$ bunds (3)
anyone $\rightarrow$ one (4)	ann $\rightarrow$ marianne (3)	carrie $\rightarrow$ carry (3)
bozzle $\rightarrow$ bozall (4)	butte $\rightarrow$ bute (3)	criss $\rightarrow$ chris (3)
clarke $\rightarrow$ clark (4)	color $\rightarrow$ colour (3)	he’s $\rightarrow$ is (3)
gryce $\rightarrow$ grice (4)	deucalion $\rightarrow$ ducalion (3)	his $\rightarrow$ is (3)
i’m $\rightarrow$ am (4)	forcemeat $\rightarrow$ meat (3)	honor $\rightarrow$ honour (3)
in $\rightarrow$ ind (4)	gryce $\rightarrow$ grice (3)	lattimer $\rightarrow$ latimer (3)
letty $\rightarrow$ lettie (4)	honor $\rightarrow$ honour (3)	millet $\rightarrow$ mellet (3)
phoebe $\rightarrow$ phebe (4)	kearny $\rightarrow$ kirney (3)	pyncheon $\rightarrow$ pension (3)
the $\rightarrow$ a (4)	nuova $\rightarrow$ noiva (3)	tad $\rightarrow$ ted (3)
ann $\rightarrow$ anne (3)	thing $\rightarrow$ anything (3)	thing $\rightarrow$ anything (3)
awhile $\rightarrow$ while (3)	this $\rightarrow$ the (3)	trevelyan $\rightarrow$ trevelian (3)

表 11：在 10 分钟、1 小时、10 小时、100 小时、960 小时标注数据上训练、并在无语言模型或词典条件下于 Librispeech 的 dev-clean 子集解码的模型的顶部词错误（见表 9 与表 10 - None）。括号内为各错误出现总次数。