wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

wav2vec 2.0：语音表征自监督学习框架（wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations）

Alexei Baevski    Henry Zhou    Abdelrahman Mohamed    Michael Auli    {abaevski,henryzhou7,abdo,michaelauli}@fb.com    Facebook AI

Alexei Baevski Henry Zhou Abdelrahman Mohamed Michael Auli
{abaevski,henryzhou7,abdo,michaelauli}@fb.com
Facebook AI

Abstract

摘要

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.[^1]

我们首次表明：仅从语音音频学习强表征，再在有转录的语音上微调，可以超过当时最好的半监督方法，同时概念更简单。wav2vec 2.0 在潜空间掩蔽语音输入，并在联合学习的潜表征量化上求解对比任务。用 Librispeech 全部标注数据时，clean/other 测试集词错误率（WER）为 1.8/3.3。把标注量降到 1 小时后，仍超过此前在 100 小时子集上的最优结果，而标注数据少 100 倍。仅用 10 分钟标注数据、在 5.3 万小时无标注数据上预训练，仍可达 4.8/8.2 WER。这说明在标注稀缺时做语音识别是可行的。

1 Introduction

1 引言

Neural networks benefit from large quantities of labeled training data. However, in many settings labeled data is much harder to come by than unlabeled data: current speech recognition systems require thousands of hours of transcribed speech to reach acceptable performance which is not available for the vast majority of the nearly 7,000 languages spoken worldwide [31]. Learning purely from labeled examples does not resemble language acquisition in humans: infants learn language by listening to adults around them - a process that requires learning good representations of speech.

神经网络受益于大量贴有标签的培训数据。 然而,在许多情况下,贴上标签的数据比没有贴上标签的数据更难获得:目前的语音识别系统需要数千小时的转录语音才能达到可以接受的性能,而全世界使用的近7 000种语言中绝大多数[31]都不具备这种性能。 纯粹从标注的例子中学习并不像人类的语言获得:婴儿通过倾听周围的成年人的心声来学习语言——这个过程需要学习良好的语言表现.

In machine learning, self-supervised learning has emerged as a paradigm to learn general data representations from unlabeled examples and to fine-tune the model on labeled data. This has been particularly successful for natural language processing [43, 45, 9] and is an active research area for computer vision [20, 2, 36, 19, 6].

在机器学习中,自监督的学习已经成为一种范式,从无标签的例子中学习一般数据表达,并在标签数据上微调模型. 这在自然语言处理方面特别成功[43,45,9],是计算机视觉[20,2,36,19,6]的活跃研究领域.

In this paper, we present a framework for self-supervised learning of representations from raw audio data. Our approach encodes speech audio via a multi-layer convolutional neural network and then masks spans of the resulting latent speech representations [26, 56], similar to masked language modeling [9]. The latent representations are fed to a Transformer network to build contextualized representations and the model is trained via a contrastive task where the true latent is to be distinguished from distractors [54, 49, 48, 28]  (§ 2).

在本文中,我们提出了一个框架,供自监督地从原始音频数据中学习表述。 我们的方法通过一个多层次的进化神经网络编码语音音频,然后将由此产生的潜在语音表达的遮罩横跨[26,56],类似于蒙蔽语言模型[9]. 潜在表述被输入Transformer网络,以构建符合背景的表述,模型通过对比性的任务加以培训,将真正的潜在表述与分散表达器区分开来54、49、48、28。

As part of training, we learn discrete speech units [53, 32, 7, 18] via a gumbel softmax [24, 5] to represent the latent representations in the contrastive task (Figure 1) which we find to be more effective than non-quantized targets. After pre-training on unlabeled speech, the model is fine-tuned on labeled data with a Connectionist Temporal Classification (CTC) loss [14, 4] to be used for downstream speech recognition tasks (§ 3)

作为培训的一部分,我们通过口香糖软max[24,5]学习离散的语音单元[53,32,7,18],以代表对比性任务(图1)中的潜在表述,我们发现这个任务比非量化目标更有效. 在对无标签的语音进行预训后,该模型在标注数据上进行了微调,带有连接器时序分类损失[14,4],用于下游语音识别任务(§3).

Figure 1: Illustration of our framework which jointly learns contextualized speech representations and an inventory of discretized speech units.

图1:介绍我们共同学习背景化演讲表述的框架和盘点化演讲单位清单。

Previous work learned a quantization of the data followed by a contextualized representations with a self-attention model [5, 4], whereas our approach solves both problems end-to-end. Masking parts of the input with Transformer networks for speech has been explored  [4, 26], but prior work relies either on a two-step pipeline or their model is trained by reconstructing the filter bank input features. Other related work includes learning representations from auto-encoding the input data [52, 11] or directly predicting future timesteps [8].

以往的工作学到了对数据进行量化,然后是用自觉模型[5,4]进行背景化表述,而我们的方法解决了两个问题。 已经探索了用Transformer网络遮盖输入部分的语音功能[4,26],但之前的工作要么依赖于双步管道,要么通过重建滤波器银行输入功能来训练他们的模型. 其他相关工作包括学习自动编码输入数据[52,11]或直接预测未来时序[8]的表示.

Our results show that jointly learning discrete speech units with contextualized representations achieves substantially better results than fixed units learned in a prior step [4]. We also demonstrate the feasibility of ultra-low resource speech recognition: when using only 10 minutes of labeled data, our approach achieves word error rate (WER) 4.8/8.2 on the clean/other test sets of Librispeech. We set a new state of the art on TIMIT phoneme recognition as well as the 100 hour clean subset of Librispeech. Moreover, when we lower the amount of labeled data to just one hour, we still outperform the previous state of the art self-training method of [42] while using 100 times less labeled data and the same amount of unlabeled data. When we use all 960 hours of labeled data from Librispeech, then our model achieves 1.8/3.3 WER (§ 4, § 5).

我们的结果表明,联合学习具有内在表述的离散语音单元,比在前一个步骤中学习到的固定单元能取得明显更好的结果[4]。 我们还展示了超低资源语音识别的可行性:当使用10分钟的标注数据时,我们的方法在Librispeech的干净/其他测试集上实现了字错误率(WER)4.8/8.2. 我们为TIMIT的电话识别 以及Librispeech 的100小时清洁子集设定了新的技术 此外,当我们把标注数据的数量降到一个小时时,我们仍然比以前[42]的艺术自训练方法要好,同时使用标注数据减少100倍,且未标注数据数量相同. 当我们使用Librispeech所有960小时的标签数据时,我们的模型就实现了1.8/3.3 WER(§4,§5).

2 Model

2 模型

Our model is composed of a multi-layer convolutional feature encoder $f:\mathcal{X}\mapsto\mathcal{Z}$ which takes as input raw audio $\mathcal{X}$ and outputs latent speech representations $\mathbf{z}_{1},\dots,\mathbf{z}_{T}$ for $T$ time-steps. They are then fed to a Transformer $g:\mathcal{Z}\mapsto\mathcal{C}$ to build representations $\mathbf{c}_{1},\dots,\mathbf{c}_{T}$ capturing information from the entire sequence [9, 5, 4]. The output of the feature encoder is discretized to $\mathbf{q}_{t}$ with a quantization module $\mathcal{Z}\mapsto\mathcal{Q}$ to represent the targets (Figure 1) in the self-supervised objective (§ 3.2). Compared to vq-wav2vec [5], our model builds context representations over continuous speech representations and self-attention captures dependencies over the entire sequence of latent representations end-to-end.

我们的模型是由一个多层演化特性编码器$f:\mathcal{X}\mapsto\mathcal{Z}$组成,该编码器以输入原始音频$\mathcal{X}$和输出潜在语音表达$\mathbf{z}_{1},\dots,\mathbf{z}_{T}$为$T$时序. 然后,它们被输入一个Transformer$g:\mathcal{Z}\mapsto\mathcal{C}$,用于构建$\mathbf{c}_{1},\dots,\mathbf{c}_{T}$从整个序列中获取信息的表示力[9,5,4]. 特性编码器的输出被光碟化为$\mathbf{q}_{t}$,并配有量化模块$\mathcal{Z}\mapsto\mathcal{Q}$,以代表自监督的目标(第3.2节)中的目标(图1). 与vq-wav2vec [5] 相比,我们的模型构建了上下文表达方式,而不是连续的语音表达方式,以及自觉捕捉到相对于潜在表达方式端到端的整个序列的依赖性.

Feature encoder.

The encoder consists of several blocks containing a temporal convolution followed by layer normalization [1] and a GELU activation function [21]. The raw waveform input to the encoder is normalized to zero mean and unit variance. The total stride of the encoder determines the number of time-steps $T$ which are input to the Transformer (§ 4.2).

特效编码器

编码器由若干块组成,包含一个时间演化,然后是层态化[1]和一个GELU激活函数[21]. 编码器的原始波形输入常态化为零平均值和单位差异. 编码器的总步数决定了输入Transformer的$T$的时间步数(§4.2).

Contextualized representations with Transformers.

The output of the feature encoder is fed to a context network which follows the Transformer architecture [55, 9, 33]. Instead of fixed positional embeddings which encode absolute positional information, we use a convolutional layer similar to [37, 4, 57] which acts as relative positional embedding. We add the output of the convolution followed by a GELU to the inputs and then apply layer normalization.

QQ 与变形人的背景化表达.

特性编码器的输出被输入一个遵循Transformer架构的上下文网络[55,9,33]. 而不是固定位置嵌入 编码绝对位置信息, 我们使用一个革命层 类似 [37, 4, 57] 作为相对位置嵌入。 在投入中加入一个GELU之后的演化输出,然后实行层态化.

Quantization module.

For self-supervised training we discretize the output of the feature encoder $\mathbf{z}$ to a finite set of speech representations via product quantization [25]. This choice led to good results in prior work which learned discrete units in a first step followed by learning contextualized representations [5]. Product quantization amounts to choosing quantized representations from multiple codebooks and concatenating them. Given $G$ codebooks, or groups, with $V$ entries $e\in\mathbb{R}^{V\times d/G}$, we choose one entry from each codebook and concatenate the resulting vectors $e_{1},\dots,e_{G}$ and apply a linear transformation $\mathbb{R}^{d}\mapsto\mathbb{R}^{f}$ to obtain $\mathbf{q}\in\mathbb{R}^{f}$.

QQ量化模块.

对于自监督的培训,我们将特性编码器$\mathbf{z}$的输出磁盘化为一套通过产品量化的有限语音表示[25]。 这一选择导致先前工作取得良好成果,在第一步学习离散单位,然后学习符合具体情况的表述[5]。 产品数量化等于从多个代码簿中选择数量化的表示并进行调适. 鉴于$G$代码簿,或 组, 带有$V$条目$e\in\mathbb{R}^{V\times d/G}$, 我们从每个代码簿中选择一个条目, 将生成的矢量$e_{1},\dots,e_{G}$连接起来, 并应用线性转换$\mathbb{R}^{d}\mapsto\mathbb{R}^{f}$以获得$\mathbf{q}\in\mathbb{R}^{f}$。

The Gumbel softmax enables choosing discrete codebook entries in a fully differentiable way [16, 24, 35]. We use the straight-through estimator [26] and setup $G$ hard Gumbel softmax operations [24]. The feature encoder output $\mathbf{z}$ is mapped to $\mathbf{l}\in\mathbb{R}^{G\times V}$ logits and the probabilities for choosing the $v$-th codebook entry for group $g$ are

Gumbel softmax能够以完全不同的方式选择离散的代码簿条目[16,24,35]. 我们使用直通估计器[26],并设置$G$硬Gumbel软max操作[24]. 特性编码器输出$\mathbf{z}$被映射到$\mathbf{l}\in\mathbb{R}^{G\times V}$对数, 选择$v$- 第 组$g$代码簿条目的概率是

$$p_{g,v}=\frac{\exp(l_{g,v}+n_{v})/\tau}{\sum_{k=1}^{V}\exp(l_{g,k}+n_{k})/\tau}, \tag{1}$$

where $\tau$ is a non-negative temperature, $n=-\log(-\log(u))$ and $u$ are uniform samples from $\mathcal{U}(0,1)$. During the forward pass, codeword $i$ is chosen by $i=\text{argmax}_{j}p_{g,j}$ and in the backward pass, the true gradient of the Gumbel softmax outputs is used.

其中$\tau$为非负温,$n=-\log(-\log(u))$和$u$为来自$\mathcal{U}(0,1)$的统一样本. 在前传期间,$i$的代号由$i=\text{argmax}_{j}p_{g,j}$选择,在后传时,使用Gumbel软max输出的真实梯度.

3 Training

3 训练

To pre-train the model we mask a certain proportion of time steps in the latent feature encoder space (§ 3.1), similar to masked language modeling in BERT [9]. The training objective requires identifying the correct quantized latent audio representation in a set of distractors for each masked time step (§ 3.2) and the final model is fine-tuned on the labeled data (§ 3.3).

为了对模型进行预训,我们在潜伏的特征编码器空间(§3.1)中掩盖了一定比例的时间步骤,类似于BERT[9]中的遮掩语言模型. 训练已将目标确定为正确的量化的潜在音频表达方式,为每个蒙蔽的时间段(第3.2节)用一组分散注意力器,并在标注数据上对最终模型进行微调(第3.3节)。

3.1 Masking

3.1 掩蔽

We mask a proportion of the feature encoder outputs, or time steps before feeding them to the context network and replace them with a trained feature vector shared between all masked time steps; we do not mask inputs to the quantization module. To mask the latent speech representations output by the encoder, we randomly sample without replacement a certain proportion $p$ of all time steps to be starting indices and then mask the subsequent $M$ consecutive time steps from every sampled index; spans may overlap.

我们掩盖一定比例的特性编码器输出,或时间步骤,然后提供给上下文网络,代之以所有蒙蔽的时间步骤共享的训练有素的特性矢量;我们不掩盖对量化模块的投入. 为了掩盖编码器的潜在语音表达切实有效,我们随机抽样,而不替换所有时间步骤中一定比例的$p$开始指数,然后从每个抽样指数中掩盖随后的$M$连续时间步骤;跨度可能重叠.

3.2 Objective

3.2 目标函数

During pre-training, we learn representations of speech audio by solving a contrastive task $\mathcal{L}_{m}{}$ which requires to identify the true quantized latent speech representation for a masked time step within a set of distractors. This is augmented by a codebook diversity loss $\mathcal{L}_{d}{}$ to encourage the model to use the codebook entries equally often.

在预训期间,我们通过解决一个对比性任务($\mathcal{L}_{m}{}$)来学习语音音频的表达方式,该任务要求在一组分散注意力者中找出真实的被量化的潜在语音表达方式。 此外,还增加了$\mathcal{L}_{d}{}$的代码簿多样性损失,以鼓励该模型同样频繁地使用代码簿条目。

$$\mathcal{L}=\mathcal{L}_{m}{}+\alpha\mathcal{L}_{d}{} \tag{2}$$

where $\alpha$ is a tuned hyperparameter.

其中$\alpha$是一个调制的超参数。

Contrastive Loss.

Given context network output $\mathbf{c}_{t}$ centered over masked time step $t$, the model needs to identify the true quantized latent speech representation $\mathbf{q}_{t}$ in a set of $K+1$ quantized candidate representations $\mathbf{\tilde{q}}\in\mathbf{Q}_{t}$ which includes $\mathbf{q}_{t}$ and $K$ distractors [23, 54]. Distractors are uniformly sampled from other masked time steps of the same utterance. The loss is defined as

相矛盾的损失。

鉴于上下文网络输出$\mathbf{c}_{t}$以遮盖的时步为中心$t$,该模型需要在一套$K+1$量化候选人代表物中确定真正的量化潜在语音代表物$\mathbf{q}_{t}$,其中包含$\mathbf{q}_{t}$和$K$分流器[23,54]. 从同一语音中的其他遮掩的时间步骤中,对干扰器进行统一取样。 损失的定义是:

$$\mathcal{L}_{m}{}=-\log\frac{\exp(sim(\mathbf{c}_{t},\mathbf{q}_{t})/\kappa)}{\sum_{\mathbf{\tilde{q}}\sim\mathbf{Q}_{t}}\exp(sim(\mathbf{c}_{t},\mathbf{\tilde{q}})/\kappa)} \tag{3}$$

where we compute the cosine similarity $sim(\mathbf{a},\mathbf{b})=\mathbf{a}^{T}\mathbf{b}/\|\mathbf{a}\|\|\mathbf{b}\|$ between context representations and quantized latent speech representations [19, 6].

其中计算上下文表述和量化潜在语音表述之间的余弦相似性$sim(\mathbf{a},\mathbf{b})=\mathbf{a}^{T}\mathbf{b}/\|\mathbf{a}\|\|\mathbf{b}\|$[19, 6]。

Diversity Loss.

The contrastive task depends on the codebook to represent both positive and negative examples and the diversity loss $\mathcal{L}_{d}$ is designed to increase the use of the quantized codebook representations [10]. We encourage the equal use of the $V$ entries in each of the $G$ codebooks by maximizing the entropy of the averaged softmax distribution $\mathbf{l}$ over the codebook entries for each codebook $\bar{p}_{g}$ across a batch of utterances; the softmax disribution does not contain the gumbel noise nor a temperature:[^2]

多样性丧失。

对比性任务依赖于代码表来代表正反两个实例,多样性损失$\mathcal{L}_{d}$旨在增加使用量化代码表征[10]. 我们鼓励在每本$G$代码簿中平等使用$V$条目,在每本代码簿的代码目录条目中,通过批次的语句将$\mathbf{l}$平均软马克斯分布的entropy最大化;软马克斯分布不包含口香糖噪音或温度:[^2]

$$\mathcal{L}_{d}{}=\frac{1}{GV}\sum_{g=1}^{G}-H(\bar{p}_{g})=\frac{1}{GV}\sum_{g=1}^{G}\sum_{v=1}^{V}\bar{p}_{g,v}\log{\bar{p}_{g,v}} \tag{4}$$

3.3 Fine-tuning

3.3 微调

Pre-trained models are fine-tuned for speech recognition by adding a randomly initialized linear projection on top of the context network into $C$ classes representing the vocabulary of the task [4]. For Librispeech, we have 29 tokens for character targets plus a word boundary token. Models are optimized by minimizing a CTC loss [14] and we apply a modified version of SpecAugment [41] by masking to time-steps and channels during training which delays overfitting and significantly improves the final error rates, especially on the Libri-light subsets with few labeled examples.

预训模型通过在上下文网络上添加随机初始线性投影到代表任务词汇的$C$类来微调语音识别[4]. 对Librispeech来说,我们有29个字符目标符号加一个字边界符号. 通过尽量减少CTC损失来优化模型[14],我们采用了SpecAugment的修改版[41],在培训期间遮掩到时间步和渠道,从而拖延过度调整并大大改进最终错误率,特别是在贴有标签的例子不多的Libri灯子集上。

4 Experimental Setup

4 实验设置

4.1 Datasets

4.1 数据集

As unlabeled data we consider the Librispeech corpus [40] without transcriptions containing 960 hours of audio (LS-960) or the audio data from LibriVox (LV-60k). For the latter we follow the pre-processing of [27] resulting in 53.2k hours of audio. We fine-tune on five labeled data settings: 960 hours of transcribed Librispeech, the train-clean-100 subset comprising 100 hours (100 hours labeled), as well as the Libri-light limited resource training subsets originally extracted from Librispeech, these are train-10h (10 hours labeled), train-1h (1 hour labeled), train-10min (10 min labeled). We follow the evaluation protocol of Libri-light for these splits and evaluate on the standard Librispech dev-other/clean and test-clean/other sets.

作为无标签的数据,我们认为Librispeech程序[40]没有包含960小时音频(LS-960)或LibriVox音频数据(LV-60k)的抄本。 就后者而言,我们遵循[27]的预处理,导致53.2小时的音频。 我们微调了五个标签数据设置:960小时转写Librispeech,包含100小时(100小时标签)的列车清洁-100子集,以及最初从Librispeech提取的Libri-light有限资源训练子集,这些是10h列车(10小时标签),1h列车(1小时标签),10min列车(10分钟标签). 我们遵循Libri-light对这些拆分的评价协议,并对标准的Librispeech dev-other/clean和测试-clean/其他套件进行评价。

We fine-tune the pre-trained models for phoneme recognition on the TIMIT dataset [13]. It contains five hours of audio recordings with detailed phoneme labels. We use the standard train, dev and test split and follow the standard protocol of collapsing phone labels to 39 classes.

我们微调了TIMIT数据集中电话识别的预训模型[13]. 里面有五个小时的录音带 还有详细的电话标签 我们使用标准列车,dev和测试拆分 并遵循标准协议 崩溃的电话标签39类。

4.2 Pre-training

4.2 预训练

Models are implemented in fairseq [39]. For masking, we sample $p=0.065$ of all time-steps to be starting indices and mask the subsequent $M=10$ time-steps. This results in approximately 49% of all time steps to be masked with a mean span length of 14.7, or 299ms (see Appendix A for more details on masking).

模式在fairseq[39]中实施. 为了遮掩,我们对所有时间步中的$p=0.065$进行抽样,作为起始指数,并遮掩随后的$M=10$时间步. 这导致大约49%的所有时间步骤被遮掩,平均跨度为14.7或299米(关于遮掩的详情见附录A)。

The feature encoder contains seven blocks and the temporal convolutions in each block have 512 channels with strides (5,2,2,2,2,2,2) and kernel widths (10,3,3,3,3,2,2). This results in an encoder output frequency of 49 hz with a stride of about 20ms between each sample, and a receptive field of 400 input samples or 25ms of audio. The convolutional layer modeling relative positional embeddings has kernel size 128 and 16 groups.

特征编码器包含7个块,每个块的时间演化器有512个信道,有跨度(5,2,2,2,2,2)和内核宽度(10,3,3,3,3,2,2). 由此得出编码器输出频率为49hz,每个样本间间隔约20ms,可接受域为400个输入样本或25ms音频. 演化层模型相对位置嵌入具有内核尺寸128和16组.

We experiment with two model configurations which use the same encoder architecture but differ in the Transformer setup: Base contains 12 transformer blocks, model dimension 768, inner dimension (FFN) 3,072 and 8 attention heads. Batches are built by cropping 250k audio samples, or 15.6sec, from each example. Crops are batched together to not exceed 1.4m samples per GPU and we train on a total of 64 V100 GPUs for 1.6 days [38]; the total batch size is 1.6h.

我们试验两种模型配置,它们使用相同的编码器架构,但在Transformer设置上却有所不同:基座包含12个Transformer块,模型尺寸768,内维(FFN)3,072和8个注意头. 从每个例子中抽取250k音频样本,即15.6sec,来构建蝙蝠。 作物按每GPU分批采样不超过1.4m,我们总共训练64个V100GPU,用1.6天[38];总批量尺寸为1.6h.

The Large model contains 24 transformer blocks with model dimension 1,024, inner dimension 4,096 and 16 attention heads. We crop 320k audio samples, or 20sec, with a limit of 1.2m samples per GPU and train on 128 V100 GPUs over 2.3 days for Librispeech and 5.2 days for LibriVox; the total batch size is 2.7h. We use dropout 0.1 in the Transformer, at the output of the feature encoder and the input to the quantization module. Layers are dropped at a rate of 0.05 for Base and 0.2 for Large  [22, 12]; there is no layer drop for LV-60k.

大号型号包含24个变压器块,型号尺寸为1,024,内号尺寸为4,096和16个注意头. 我们收获320k音频样本,即20sec,每GPU限量为1.2m样本,在Librispeech和LibriVox在2.3天内对128V100GPU进行训练;批量总尺寸为2.7h。 我们在Transformer中使用退出0.1,在特性编码器的输出和对量子化模块的输入. 层降速为基数0.05,大数0.2[22,12];LV-60k没有层降.

We optimize with Adam [29], warming up the learning rate for the first 8% of updates to a peak of $5\text{\times}{10}^{-4}$ for Base and $3\text{\times}{10}^{-4}$ for Large, and then linearly decay it. Large trains for 250k updates, Base for 400k updates, and Large on LV-60k for 600k updates. We use weight $\alpha=0.1$ for the diversity loss Equation 2. For the quantization module we use $G=2$ and $V=320$ for both models, resulting in a theoretical maximum of 102.4k codewords. Entries are of size $d/G=128$ for Base amd $d/G=384$ for Large. The Gumbel softmax temperature $\tau$ is annealed from 2 to a minimum of 0.5 for Base and 0.1 for Large by a factor of 0.999995 at every update. The temperature in the contrastive loss (Equation 3) is set to $\kappa=0.1$. For the smaller Librispeech dataset, we regularize the model by applying an L2 penalty to the activations of the final layer of the feature encoder and scale down the gradients for the encoder by a factor of 10. We also use a slightly different encoder architecture where we do not use layer normalization, and instead of normalizing the raw waveform, the output of the first encoder layer is normalized. In the contrastive loss we use $K=100$ distractors. We choose the training checkpoint with the lowest $\mathcal{L}_{m}{}$ on the validation set.

我们优化了亚当[29], 提升了最初8%的更新的学习率, 到顶峰 $5\text{\times}{10}^{-4}$的基, 和$3\text{\times}{10}^{-4}$的大, 然后线性衰变。 大型列车为250k更新,Base为400k更新,LV-60k上大型列车为600k更新. 我们用$\alpha=0.1$来表示多样性的丧失 对于量子化模块,我们使用$G=2$和$V=320$两种模型,导致理论极限为102.4k代码. 条目的大小为: $d/G=128$为Base amd $d/G=384$为大项. Gumbel软max温度$\tau$在每次更新时均以0.99995的系数从2到最小0.5的Base和0.1的Ligh的反射。 对比性损失(方程式3)中的温度设定为$\kappa=0.1$. 对于更小的Librispeech数据集,我们通过对特性编码器最后层的激活应用L2的判罚来规范模型,并将编码器的梯度缩小10倍. 我们还使用一种稍有不同的编码器架构,在这种架构中我们不使用层态化,而不是将原始波形正常化,而是将第一个编码器层的输出正常化. 在对比性损失中,我们使用$K=100$分心器. 我们选择训练检查站,在验证装置上设置最低的$\mathcal{L}_{m}{}$.

4.3 Fine-tuning

4.3 微调

After pre-training we fine-tune the learned representations on labeled data and add a randomly initialized output layer on top of the Transformer to predict characters (Librispeech/Libri-light) or phonemes (TIMIT). For Libri-light, we train three seeds with two different learning rates (2e-5 and 3e-5) for all subsets and choose the configuration with lowest WER on dev-other subset decoded with the official 4-gram language model (LM) with beam 50 and fixed model weights (LM weight 2, word insertion penalty -1). For Base on the labeled 960h subset we use a learning rate of 1e-4.

经过前期训练后,我们微调了标注数据上的学习表现,并在Transformer上方添加随机初始化的输出层,以预测字符(Librispeech/Libri-light)或电话(TIMIT). 对于Libri-light,我们为所有子集训练三个有两种不同学习速率(2e-5和3e-5)的种子,并在dev-other子集解码上选择最小WER的配置,与正式的4克语言模型(LM)用波束50和固定的模型重量(LM重量2,字插入惩罚-1). 对于标注在960h子集上的Base,我们使用1e-4的学习率.

We optimize with Adam and a tri-state rate schedule where the learning rate is warmed up for the first 10% of updates, held constant for the next 40% and then linearly decayed for the remainder. Base uses a batch size of 3.2m samples per GPU and we fine-tune on 8 GPUs, giving a total batch size of 1,600sec. Large batches 1.28m samples on each GPU and we fine-tune on 24 GPUs, resulting in an effective batch size of 1,920sec. For the first 10k updates only the output classifier is trained, after which the Transformer is also updated. The feature encoder is not trained during fine-tuning. We mask the feature encoder representations with a strategy similar to SpecAugment [41] detailed in Appendix B.

我们优化了亚当 和三态的速率表 其中学习速率被热 在最初的10%更新, 保持不变 在接下来的40%, 然后线性衰减 对于其余。 Base使用每个GPU的批量尺寸3.2m样品,我们微调了8个GPU,给出的批量总尺寸为1600sec. 每个GPU上的大批量1.28m样本,我们微调24个GPU,导致有效批量尺寸1,920sec. 对于前10k更新只训练输出分类器,之后Transformer也会更新. 特性编码器在微调时没有经过训练. 我们用类似于SpecAugment[41]的策略来掩盖特征编码器的表示,详见附录B.

4.4 Language Models and Decoding

4.4 语言模型与解码

We consider two types of language models (LM): a 4-gram model and a Transformer [3] trained on the Librispeech LM corpus. The Transformer LM is identical to [51] and contains 20 blocks, model dimension 1,280, inner dimension 6,144 and 16 attention heads. We tune the weights of the language model (interval $[0,5]$) and a word insertion penalty ($[-5,5]$) via Bayesian optimization[^3]: we run 128 trials with beam 500 for the 4-gram LM and beam 50 for the Transformer LM and choose the best set of weights according to performance on dev-other. Test performance is measured with beam 1,500 for the n-gram LM and beam 500 for the Transformer LM. We use the beam search decoder of [44].

我们考虑两种语言模型(LM):一种是4克模型,另一种是Transformer[3],培训内容是Librispeech LMactor. TransformerLM与[51]完全相同,包含20块,型号尺寸1,280,内尺寸6,144和16个注意头. 我们通过贝叶斯优化[^3]调和语言模型的权重(interval $[0,5]$)和一个字插入罚($[-5,5]$):我们运行128个试验,4克LM的束500,TransformerLM的束50,并根据dev-other上的性能选择最好的一套权重. 测试性能为n克LM用1500束测量,TransformerLM用500束测量. 我们使用光束搜索解码器[44].

5 Results

5 结果

Model	Unlabeled	LM	dev		test
data	clean	other		clean	other
10 min labeled
Discrete BERT [4]	LS-960	4-gram	15.7	24.1		16.3	25.2
Base	LS-960	4-gram	8.9	15.7		9.1	15.6
		Transf.	6.6	13.2		6.9	12.9
Large	LS-960	Transf.	6.6	10.6		6.8	10.8
	LV-60k	Transf.	4.6	7.9		4.8	8.2
1h labeled
Discrete BERT [4]	LS-960	4-gram	8.5	16.4		9.0	17.6
Base	LS-960	4-gram	5.0	10.8		5.5	11.3
		Transf.	3.8	9.0		4.0	9.3
Large	LS-960	Transf.	3.8	7.1		3.9	7.6
	LV-60k	Transf.	2.9	5.4		2.9	5.8
10h labeled
Discrete BERT [4]	LS-960	4-gram	5.3	13.2		5.9	14.1
Iter. pseudo-labeling [58]	LS-960	4-gram+Transf.	23.51	25.48		24.37	26.02
	LV-60k	4-gram+Transf.	17.00	19.34		18.03	19.92
Base	LS-960	4-gram	3.8	9.1		4.3	9.5
		Transf.	2.9	7.4		3.2	7.8
Large	LS-960	Transf.	2.9	5.7		3.2	6.1
	LV-60k	Transf.	2.4	4.8		2.6	4.9
100h labeled
Hybrid DNN/HMM [34]	-	4-gram	5.0	19.5		5.8	18.6
TTS data augm. [30]	-	LSTM				4.3	13.5
Discrete BERT [4]	LS-960	4-gram	4.0	10.9		4.5	12.1
Iter. pseudo-labeling [58]	LS-860	4-gram+Transf.	4.98	7.97		5.59	8.95
	LV-60k	4-gram+Transf.	3.19	6.14		3.72	7.11
Noisy student [42]	LS-860	LSTM	3.9	8.8		4.2	8.6
Base	LS-960	4-gram	2.7	7.9		3.4	8.0
		Transf.	2.2	6.3		2.6	6.3
Large	LS-960	Transf.	2.1	4.8		2.3	5.0
	LV-60k	Transf.	1.9	4.0		2.0	4.0

Table 1: WER on the Librispeech dev/test sets when training on the Libri-light low-resource labeled data setups of 10 min, 1 hour, 10 hours and the clean 100h subset of Librispeech. Models use either the audio of Librispeech (LS-960) or the larger LibriVox (LV-60k) as unlabeled data. We consider two model sizes: Base (95m parameters) and Large (317m parameters). Prior work used 860 unlabeled hours (LS-860) but the total with labeled data is 960 hours and comparable to our setup.

型号	无标签	内容	备忘		测试
数据	干净	其他项目		干净	其他项目
** 对吧?
Discrete BERT [4] (英语).	LS-960	4克口径	15.7	24.1		16.3	25.2
基础	LS-960	4克口径	8.9	15.7		9.1	15.6
		传译.	6.6	13.2		6.9	12.9
大块	LS-960	传译.	6.6	10.6		6.8	10.8
	LV-60k (韩语)	传译.	4.6	7.9		4.8	8.2
1小时标签
Discrete BERT [4] (英语).	LS-960	4克口径	8.5	16.4		9.0	17.6
基础	LS-960	4克口径	5.0	10.8		5.5	11.3
		传译.	3.8	9.0		4.0	9.3
大块	LS-960	传译.	3.8	7.1		3.9	7.6
	LV-60k (韩语)	传译.	2.9	5.4		2.9	5.8
10小时标签
Discrete BERT [4] (英语).	LS-960	4克口径	5.3	13.2		5.9	14.1
斜体. 伪标签 [58]	LS-960	4-克+(4-克+) 传译.	23.51	25.48		24.37	26.02
	LV-60k (韩语)	4-克+(4-克+) 传译.	17.00	19.34		18.03	19.92
基础	LS-960	- -	3.8	9.1		4.3	9.5
		传译.	2.9	7.4		3.2	7.8
大型	LS-960	传译.	2.9	5.7		3.2	6.1
	LV-60k (韩语)	传译.	2.4	4.8		2.6	4.9
** 100小时标签**
DNN/HMM混合型 [34]	-	4克口径	5.0	19.5		5.8	18.6
TTS数据ugm. [30]	-	升降机				4.3	13.5
Discrete BERT [4] (英语).	LS-960	4克高压	4.0	10.9		4.5	12.1
斜体. 伪标签 [58]	LS-860	4-克+(4-克+) 传译.	4.98	7.97		5.59	8.95
	LV-60k (韩语)	4-克+(4-克+) 传译.	3.19	6.14		3.72	7.11
噪音学生[42]	LS-860	升降机	3.9	8.8		4.2	8.6
国际义务	LS-960	4克口径	2.7	7.9		3.4	8.0
		传译.	2.2	6.3		2.6	6.3
大块	LS-960	转帐	2.1	4.8		2.3	5.0
	LV-60k (韩语)	传译.	1.9	4.0		2.0	4.0

表1: Librispeech dev/测试集上的WER,在Libri-light低资源标注数据集培训时为10分钟,1小时,10小时,Librispeech的干净100h子集. 模型要么使用Librispeech(LS-960)的音频,要么使用更大的LibriVox(LV-60k)作为无标签数据. 我们考虑两个模型大小:基数(95m参数)和大数(317m参数). 先前的工作使用了860个无标签的小时(LS-860),但有标签数据的总时数是960小时,与我们的设置相当.

5.1 Low-Resource Labeled Data Evaluation

5.1 低资源标注数据评测

We first evaluate our pre-trained models in settings where the amount of labeled data is limited to get a sense of how the representations learned on unlabeled data can improve low resource settings. If a pre-trained model captures the structure of speech, then it should require few labeled examples to fine-tune it for speech recognition. The models are pre-trained on the audio data of either Librispeech (LS-960) or LibriVox (LV-60k) and most results are obtained by decoding with a Transformer language model (Transf.); Appendix C shows results with no language model at all as well as with an n-gram language model.

我们首先在标签数据数量有限的环境下评估我们预先训练过的模型,以便了解在未标签数据上学到的表述如何改善低资源设置。 如果预先训练的模型能捕捉到语音的结构,那么它就应该要求很少有标签的示例来对其进行微调以进行语音识别. 这些模型在Librispeech(LS-960)或LibriVox(LV-60k)的音频数据上经过预先培训,大部分结果是通过Transformer语言模型(Transf)的解码获得的; 附录C显示了完全没有语言模型的结果以及n克语言模型.

The Large model pre-trained on LV-60k and fine-tuned on only 10 minutes of labeled data achieves a word error rate of 5.2/8.6 on the Librispeech clean/other test sets. Ten minutes of labeled data corresponds to just 48 recordings with an average length of 12.5 seconds. This demonstrates that ultra-low resource speech recognition is possible with self-supervised learning on unlabeled data. Our approach of jointly learning discrete units and contextualized representations clearly improves over previous work which learned quantized audio units in a separate step [4], reducing WER by a about a third.

LV-60k上的大型模型预训,仅10分钟标签数据上的微调,在Librispeech清洁/其他测试集上实现了5.2/8.6的词错误率. 10分钟的标签数据只对应48个录音,平均长度为12.5秒. 这表明,通过自监督的对无标签数据的学习,可以对资源语音进行超低的识别。 我们联合学习离散单元和背景化表述的方法,显然比以前在单独步骤[4]中学习量化音频单元的工作有所改进,将WER减少了大约三分之一。

A recent iterative self-training approach [42] represents the state of the art on the clean 100 hour subset of Librispeech but it requires multiple iterations of labeling, filtering, and re-training. Our approach is simpler: we pre-train on the unlabeled data and fine-tune on the labeled data. On the 100 hour subset of Librispeech, their method achieves WER 4.2/8.6 on test-clean/other which compares to WER 2.3/5.0 with the Large model in a like for like setup, a relative WER reduction of 45%/42%.

最近一种迭代自我训练方法[42]代表了Librispeech干净的100小时子集的艺术状态,但它需要标签、过滤和再训练的多重迭代。 我们的方法比较简单:我们先对未贴标签的数据进行预训,并对贴标签的数据进行微调. 在L在其任务范围内的100小时子集上,它们的方法在测试-清洁/其他上实现了WER 4.2/8.6,与大模型的WER 2.3/5.0相比,类似设置,相对WER减少45%/42%。

When the Large model uses an order of magnitude less labeled data (10h labeled), then it still achieves WER 3.2/6.1, an error reduction of 24%/29% relative to iterative self-training. Using only a single hour of labeled data, the same model achieves WER 3.9/7.6 which improves on both test-clean and test-other by 7%/12% - with two orders of magnitude less labeled data. We note that the Libri-light data splits contain both clean and noisy data leading to better accuracy on test-other compared to test-clean. Increasing model size reduces WER on all setups with the largest improvements on test-other (Base vs. Large both on LS-960) and increasing the amount of unlabeled training data also leads to large improvements (Large LS-960 vs. LV-60k).

当大模型使用标注较少的数据(10h标签)的量级时,仍然实现了WER 3.2/6.1,相对于迭代自训,误差减少24%/29%. 同一模型仅使用一个小时的标签数据,就实现了WER 3.9/7.6,在测试-清洁和测试-其他两个方面都提高了7%/12%——两个数量级的标签数据较少。 我们注意到,Libri灯光数据分解包含干净和吵闹的数据,使得其他测试的准确性高于测试清洁。 不断增大的模型尺寸减少了所有安装的WER,对测试-其他的改进最大(LS-960上的Base vs. Large),并且增加了无标签的训练数据数量也导致了巨大的改进(LS-960对LV-60k).

5.2 High-Resource Labeled Data Evaluation on Librispeech

5.2 Librispeech 高资源标注评测

In this section we evaluate the performance when large quantities of labeled speech are available to assess the effectiveness of our approach in a high resource setup. Specifically, we fine-tune the same models as before on the full 960 hours of labeled Librispeech: Base and Large pre-trained on LS-960 as well as Large pre-trained on LV-60k.

在本节中,我们评价在有大量有标签的演讲,以评估我们在高资源结构中做法的有效性时的绩效。 具体地说,我们微调了之前在Librispeech: Base and Large 预训LS-960以及LV-60k上预训的全960小时的相同型号.

Table 2shows that our approach achieves WER 1.8/3.3 on test-clean/other on the full Librispeech benchmark. This is despite a weaker baseline architecture: supervised training of our architecture achieves WER 2.1/4.6 (Large - from scratch) compared to WER 1.9/4.1 for ContextNet [17], the baseline architecture of the state of the art [42]. We use a simple Transformer with CTC which does not perform as well as seq2seq models [51].

表2显示,我们的方法在Librispeech的全部基准上实现了WER 1.8/3.3测试清洁/其他测试。 尽管基线架构较弱:监督的我国建筑培训实现了WER 2.1/4.6(大 -- -- 从零开始),而背景网的WER 1.9/4.1([17])是艺术状态的基准架构[42]。 我们使用一种简单的Transformer,它与CTC不同时使用,以及以下2seq模型[51]。

Note that the vocabulary of our acoustic model (characters) does not match the vocabulary of the LM (words) which delays feedback from the LM and is likely to be detrimental. Most recent work [51, 58, 17, 42] uses the better performing word pieces [50] for both models. Moreover, our result is achieved without any data balancing such as [42]. Finally, self-training is likely complimentary to pre-training and their combination may yield even better results. Appendix E presents a detailed error analysis of our pre-trained models in various labeled data setups.

注意,我们声学模型(特征)的词汇与LM(词)的词汇不匹配,后者延迟了来自LM的反馈,很可能是有害的. 最近作品[51,58,17,42]对两种型号都采用了性能较好的单词作品[50]. 此外,我们取得的成果没有任何数据平衡,例如[42]。 最后,自学培训可能与预训相辅相成,两者结合可能会产生更好的效果。 附录E详细分析了我们在各种标注数据设置中经过预先训练的模型。

Model	Unlabeled	LM	dev		test
data	clean	other		clean	other
Supervised
CTC Transf [51]	-	CLM+Transf.	2.20	4.94		2.47	5.45
S2S Transf.  [51]	-	CLM+Transf.	2.10	4.79		2.33	5.17
Transf. Transducer [60]	-	Transf.	-	-		2.0	4.6
ContextNet [17]	-	LSTM	1.9	3.9		1.9	4.1
Conformer [15]	-	LSTM	2.1	4.3		1.9	3.9
Semi-supervised
CTC Transf. + PL [51]	LV-60k	CLM+Transf.	2.10	4.79		2.33	4.54
S2S Transf. + PL [51]	LV-60k	CLM+Transf.	2.00	3.65		2.09	4.11
Iter. pseudo-labeling [58]	LV-60k	4-gram+Transf.	1.85	3.26		2.10	4.01
Noisy student [42]	LV-60k	LSTM	1.6	3.4		1.7	3.4
This work
Large - from scratch	-	Transf.	1.7	4.3		2.1	4.6
Base	LS-960	Transf.	1.8	4.7		2.1	4.8
Large	LS-960	Transf.	1.7	3.9		2.0	4.1
	LV-60k	Transf.	1.6	3.0		1.8	3.3

Table 2: WER on Librispeech when using all 960 hours of labeled data (cf. Table 1).

型号	无标签	内容	备忘		测试
数据	干净	其他项目		干净	其他项目
** 接受监督**
反恐委员会有权享受以下服务:	-	CLM+ 数据 传译.	2.20	4.94		2.47	5.45
S2S Transf. [51] (英语).	-	CLM+ 数据 传译.	2.10	4.79		2.33	5.17
转译器 [60]	-	传译.	-	-		2.0	4.6
背景网 [17]	-	升降机	1.9	3.9		1.9	4.1
协进会[15]	-	升降机	2.1	4.3		1.9	3.9
** 半监督**
+ PL [51]	LV-60k (韩语)	CLM+ 数据 传译.	2.10	4.79		2.33	4.54
S2S Transf. + PL [51]	LV-60k (韩语)	CLM+ 数据 传译.	2.00	3.65		2.09	4.11
的关键要素:录像带展示了关于依循 我是个力争 你为什么要和平和安全被指定为所针对的 让我看看所给予的支持以和平方式促成了什么均属配套的标志。	除其它外!	4-克+(4-克+) 传译.	1.85	3.26		2.10	4.01
噪音学生[42]	LV-60k (韩语)	升降机	1.6	3.4		1.7	3.4
i 这项工作
大 - 从零开始	-	传译.	1.7	4.3		2.1	4.6
基础	LS-960	传译.	1.8	4.7		2.1	4.8
大块	LS-960	(原始 我没有)	1.7	3.9		2.0	4.1
	LV-60k (韩语 62.	问责	1.6	3.0		1.8	3.3

表2:在使用所有960小时的标签提交的文件时使用Librispeech上的WER(见表1)。

5.3 Phoneme Recognition on TIMIT

5.3 TIMIT 音素识别

Next, we evaluate accuracy on TIMIT phoneme recognition by fine-tuning the pre-trained models on the labeled TIMIT training data. We fine-tune as for the 10 hour subset of Libri-light but do not use a language model. Table 3 shows that our approach can achieve a new state of the art on this dataset, reducing PER by a relative 23%/29% over the next best result on the dev/test sets. Appendix D shows an analysis of how the discrete latent speech representations related to phonemes. Other recent work on pre-training which evaluates on TIMIT includes [47] who solve multiple tasks to learn good representations of speech.

接下来,我们通过微调标定的TIMIT训练数据上预先训练过的模型来评估TIMIT电话识别的准确性. 我们微调了Libri-light的10小时子集,但不使用语言模型。 表3显示,我们的方法可以在这个数据集上实现一个新的技术水平,相对于dev/测试集的下一个最佳结果,PER将相对减少23%/29 %. 附录D分析了离散的潜在语音表述与电话的关系。 最近关于培训前工作的其他工作是对TIMIT进行评估的,其中包括[47] 解决多重任务,学习好语音表征。

	dev PER	test PER
CNN + TD-filterbanks [59]	15.6	18.0
PASE+ [47]	-	17.2
Li-GRU + fMLLR [46]	–	14.9
wav2vec [49]	12.9	14.7
vq-wav2vec [5]	9.6	11.6
This work (no LM)
Large (LS-960)	7.4	8.3

Table 3: TIMIT phoneme recognition accuracy in terms of phoneme error rate (PER).

	dev PER 数据	测试 PER
CNN + TD- filterbanks [59] (美国英语).	15.6	18.0
巴勒斯坦权力机构+[47]	-	17.2
Li-GRU + fMLLR [46] (英语).	- 说吧	14.9
wav2vec [49] (英语).	12.9	14.7
vq-wav2vec [5] (韩语).	9.6	11.6
** 这项工作(无LM)**
大型( LS- 960)	7.4	8.3

表3:TIMIT电话识别精度(电话错误率)(PER).

5.4 Ablations

5.4 消融实验

A difference to previous work [5, 4] is that we quantize the latent audio representations only for the contrastive loss, i.e., when latents are used as targets, but not when the latents are input to the Transformer network. We motivate this choice by an ablating for which we adopt a reduced training setup to increase experimental turn around: we pre-train Base on LS-960 for 250k updates with masking probability $p=0.075$, fine-tune on train-10h for 60k updates on a single GPU with 640k samples per batch, or 40 sec of speech audio. We report the average WER and standard deviation on the concatenation of dev-clean and dev-other (dev PER) for three seeds of fine-tuning.

与之前的作品[5,4]不同的是,我们只将潜在音频表达量量化为对比性损失,即当潜在值被用作目标时,而不是当潜在值被输入Transformer网络时. 我们鼓励这种选择,为此我们采用了一个减少的训练设置来增加实验转弯:我们通过LS-960在250k上进行预训,以遮掩概率$p=0.075$进行更新,在10h号列车上进行微调,以每批640k样本的单一GPU上进行60k更新,或40秒的语音音频. 我们报告三个微调种子的脱氧核糖核酸和脱氧核糖核酸(Dev PER)的结合平均WER和标准偏差。

Table 4 shows that our strategy of continuous inputs with quantized targets (Baseline) performs best. Continuous latent speech representations retain more information to enable better context representations and quantizing the target representations leads to more robust training. Quantizing the latents both in the input and the targets performs least well, and explains the lower performance of prior work [5, 4]. Continuous targets reduce the effectiveness of self-supervised training since targets can capture detailed artifacts of the current sequence, e.g. speaker and background information, which make the task easier and prevent the model from learning general representations beneficial to speech recognition. The training accuracy of identifying the correct latent audio representation increases from 62% to 78.0% when switching from quantized to continuous targets. Continuous inputs and continuous targets perform second best but various attempts to improve it did not lead to better results (see Appendix F for this experiment and other ablations on various hyperparameters).

表4显示,我们以量化目标(基线)持续投入的战略效果最佳。 持续的潜在语音表述保留了更多的信息,以便能够更好地进行背景表述,并量化目标表述,从而导致更强有力的培训。 量化投入和目标中的潜在效果最差,并解释了先前工作绩效较低的原因[5、4]。 持续的目标降低了自监督培训的有效性,因为目标可以捕捉到当前序列的详细文物,如扬声器和背景资料,这使得任务更加容易,使模型无法学习有利于语音识别的一般表述. 在从量化目标转为持续目标时,识别正确潜在音频表现的培训准确度从62%提高到78.0%。 持续的投入和持续的目标表现第二佳,但各种改进努力没有导致更好的结果(关于这一试验和各种超参数的其他拼图,见附录F)。

	avg. WER	std.
Continuous inputs, quantized targets (Baseline)	7.97	0.02
Quantized inputs, quantized targets	12.18	0.41
Quantized inputs, continuous targets	11.18	0.16
Continuous inputs, continuous targets	8.58	0.08

Table 4: Average WER and standard deviation on combined dev-clean/other of Librispeech for three training seeds. We ablate quantizing the context network input and the targets in the contrastive loss.

	v. 微风	师曰.
连续投入,量化目标(基准)	7.97	0.02
量化投入,量化目标.	12.18	0.41
量化投入,持续目标.	11.18	0.16
持续投入、持续目标	8.58	0.08

表4:三种训练种子混合脱氧核糖核酸/其他Librispeech的平均WER和标准偏差。 在对比性损失中,我们放弃对上下文网络输入和目标的量化。

6 Conclusion

6 结论

We presented wav2vec 2.0, a framework for self-supervised learning of speech representations which masks latent representations of the raw waveform and solves a contrastive task over quantized speech representations. Our experiments show the large potential of pre-training on unlabeled data for speech processing: when using only 10 minutes of labeled training data, or 48 recordings of 12.5 seconds on average, we achieve a WER of 4.8/8.2 on test-clean/other of Librispeech.

我们介绍了wav2vec 2.0,这是一个自监督的语音表达学习框架,它掩盖了原始波形的潜在表达方式,并解决了与量化的语音表达方式相比具有对比性的任务。 我们的实验表明,对无标签数据进行语音处理的预训潜力很大:当仅仅使用10分钟的有标签的训练数据,或平均48个12.5秒的录音时,我们在Librispeech的测试-清洁/其他设备上实现了4.8/8.2的WER.

Our model achieves results which achieve a new state of the art on the full Librispeech benchmark for noisy speech. On the clean 100 hour Librispeech setup, wav2vec 2.0 outperforms the previous best result while using 100 times less labeled data. The approach is also effective when large amounts of labeled data are available. We expect performance gains by switching to a seq2seq architecture and a word piece vocabulary.

我们的模式取得了一些成果, 在Librispeech语调的完整基准上实现了一个新的艺术水平。 在干净的100小时Librispeech设置上,wav2vec 2.0在使用100倍少标签数据的同时,超越了之前的最佳结果. 在有大量标签数据可用时,这一方法也有效。 我们期望通过切换到以下2seq架构和一个单词块词汇来提高性能.

Broader Impact

更广泛影响

There are around 7,000 languages in the world and many more dialects. However, for most of them no speech recognition technology exists since current systems require hundreds or thousands of hours of labeled data which is hard to collect for most languages. We have shown that speech recognition models can be built with very small amounts of annotated data at very good accuracy. We hope our work will make speech recognition technology more broadly available to many more languages and dialects.

世界上约有7000种语言,还有更多的方言. 然而无偿服务对大多数语言来说并不存在语音识别技术,因为目前的系统需要数百或数千小时的标签数据,而大多数语言都很难收集这些数据。 我们已经表明,语音识别模型可以非常精确地用极少量的注解数据来构建. 我们希望,我们的工作将使更多的语言和方言更广泛地获得语音识别技术。

Acknowledgments and Disclosure of Funding

致谢与资助披露

We thank Tatiana Likhomanenko and Qiantong Xu for helpful discussion and their help with wav2letter integration.

我们感谢Tatiana Likhomanenko和Xiantong Xu在Wav2letter整合方面进行了有益的讨论和帮助。

References

参考文献

[1] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv , 2016.

[2] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. In Proc. of NeurIPS , 2019.

[3] A. Baevski and M. Auli. Adaptive input representations for neural language modeling. In Proc. of ICLR , 2018.

[4] A. Baevski, M. Auli, and A. Mohamed. Effectiveness of self-supervised pre-training for speech recognition. arXiv , abs/1911.03912, 2019.

[5] A. Baevski, S. Schneider, and M. Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In Proc. of ICLR , 2020.

[6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv , abs/2002.05709, 2020.

[7] J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord. Unsupervised speech representation learning using wavenet autoencoders. arXiv , abs/1901.08810, 2019.

[8] Y. Chung, W. Hsu, H. Tang, and J. R. Glass. An unsupervised autoregressive model for speech representation learning. arXiv , abs/1904.03240, 2019.

[9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv , abs/1810.04805, 2018.

[10] S. Dieleman, A. van den Oord, and K. Simonyan. The challenge of realistic music generation: modelling raw audio at scale. arXiv , 2018.

[11] R. Eloff, A. Nortje, B. van Niekerk, A. Govender, L. Nortje, A. Pretorius, E. Van Biljon, E. van der Westhuizen, L. van Staden, and H. Kamper. Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. arXiv , abs/1904.07556, 2019.

[12] A. Fan, E. Grave, and A. Joulin. Reducing transformer depth on demand with structured dropout. In Proc. of ICLR , 2020.

[13] J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM. Linguistic Data Consortium , 1993.

[14] A. Graves, S. Fernández, and F. Gomez. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proc. of ICML , 2006.

[15] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang. Conformer: Convolution-augmented transformer for speech recognition. arXiv , 2020.

[16] E. J. Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures , volume 33. US Government Printing Office, 1954.

[17] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv , 2020.

[18] D. Harwath, W.-N. Hsu, and J. Glass. Learning hierarchical discrete linguistic units from visually-grounded speech. In Proc. of ICLR , 2020.

[19] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. arXiv , abs/1911.05722, 2019.

[20] O. J. Hénaff, A. Razavi, C. Doersch, S. M. A. Eslami, and A. van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv , abs/1905.09272, 2019.

[21] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv , 2016.

[22] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. arXiv , 2016.

[23] M. G. A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. of AISTATS , 2010.

[24] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv , abs/1611.01144, 2016.

[25] H. Jegou, M. Douze, and C. Schmid. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell. , 33(1):117–128, Jan. 2011.

[26] D. Jiang, X. Lei, W. Li, N. Luo, Y. Hu, W. Zou, and X. Li. Improving transformer-based speech recognition using unsupervised pre-training. arXiv , abs/1910.09932, 2019.

[27] J. Kahn et al. Libri-light: A benchmark for asr with limited or no supervision. In Proc. of ICASSP , 2020.

[28] K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. van den Oord. Learning robust and multilingual speech representations. arXiv , 2020.

[29] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR , 2015.

[30] A. Laptev, R. Korostik, A. Svischev, A. Andrusenko, I. Medennikov, and S. Rybin. You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation. arXiv , abs/2005.07157, 2020.

[31] M. P. Lewis, G. F. Simon, and C. D. Fennig. Ethnologue: Languages of the world, nineteenth edition. Online version: http://www.ethnologue.com , 2016.

[32] A. H. Liu, T. Tu, H. yi Lee, and L. shan Lee. Towards unsupervised speech recognition and synthesis with quantized speech representation learning. arXiv , 2019a.

[33] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019b.

[34] C. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. Ney. Rwth asr systems for librispeech: Hybrid vs attention. In Interspeech 2019 , 2019.

[35] C. J. Maddison, D. Tarlow, and T. Minka. A* sampling. In Advances in Neural Information Processing Systems , pages 3086–3094, 2014.

[36] I. Misra and L. van der Maaten. Self-supervised learning of pretext-invariant representations. arXiv , 2019.

[37] A. Mohamed, D. Okhonko, and L. Zettlemoyer. Transformers with convolutional context for ASR. arXiv , abs/1904.11660, 2019.

[38] M. Ott, S. Edunov, D. Grangier, and M. Auli. Scaling neural machine translation. In Proc. of WMT , 2018.

[39] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations , 2019.

[40] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Proc. of ICASSP , pages 5206–5210. IEEE, 2015.

[41] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Proc. of Interspeech , 2019.

[42] D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le. Improved noisy student training for automatic speech recognition. arXiv , abs/2005.09629, 2020.

[43] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. In Proc. of ACL , 2018.

[44] V. Pratap, A. Hannun, Q. Xu, J. Cai, J. Kahn, G. Synnaeve, V. Liptchinsky, and R. Collobert. Wav2letter++: A fast open-source speech recognition system. In Proc. of ICASSP , 2019.

[45] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf , 2018.

[46] M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio. Light gated recurrent units for speech recognition. IEEE Transactions on Emerging Topics in Computational Intelligence , 2(2):92–102, 2018.

[47] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio. Multi-task self-supervised learning for robust speech recognition. arXiv , 2020.

[48] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux. Unsupervised pretraining transfers well across languages. arXiv , abs/2002.02848, 2020.

[49] S. Schneider, A. Baevski, R. Collobert, and M. Auli. wav2vec: Unsupervised pre-training for speech recognition. In Proc. of Interspeech , 2019.

[50] M. Schuster and K. Nakajima. Japanese and korean voice search. In Proc. of ICASSP , 2012.

[51] G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert. End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures. arXiv , abs/1911.08460, 2020.

[52] A. Tjandra, B. Sisman, M. Zhang, S. Sakti, H. Li, and S. Nakamura. Vqvae unsupervised unit discovery and multi-scale code2spec inverter for zerospeech challenge 2019. arXiv , 1905.11449, 2019.

[53] A. van den Oord, O. Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems , pages 6306–6315, 2017.

[54] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv , abs/1807.03748, 2018.

[55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proc. of NIPS , 2017.

[56] W. Wang, Q. Tang, and K. Livescu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. arXiv , 2020.

[57] F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. In Proc. of ICLR , 2019.

[58] Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert. Iterative pseudo-labeling for speech recognition. arXiv , 2020.

[59] N. Zeghidour, N. Usunier, I. Kokkinos, T. Schaiz, G. Synnaeve, and E. Dupoux. Learning filterbanks from raw speech for phone recognition. In Proc. of ICASSP , 2018.

[60] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss. arXiv , 2020.

Appendices

附录

Appendix A Masking distribution

附录 A 掩蔽分布

When choosing which time-steps to mask, each latent speech representation in an utterance is considered a candidate starting time-step with probability $p$ where $M$ is the length of each masked span starting from the respective time step; both are hyper-parameters. Sampled starting time steps are expanded to length $M$ and spans can overlap.

在选择哪些时间步骤来掩盖时,每个潜在的语音表述都被视为候选人的起始时间步骤,概率为$p$,其中$M$是从相应时间步骤开始的每个蒙面跨度的长度;两者均为超参数。 抽样的起始时间步骤被扩展为$M$长度,跨度可以重叠.

For a 15 sec long audio sample, the average mask length is 14.7 time-steps, corresponding to 299ms of audio, with a median of 10 time-steps, and a maximum of about 100 time steps; about 49% of all time-steps in the sample will be masked. A plot of the corresponding mask length distribution is shown in Figure 2 and an ablation of $M$ and $p$ as well as the effect of other masking strategies is shown in Table 5. Reducing $M$ results in increased prediction accuracy for the self-supervised but the task becomes trivial when spans with length one are masked, leading to poor performance on downstream speech recognition tasks. We also consider other masking strategies: w/o overlap uniform($a$,$b$) samples for each starting index a span length $M^{s}$ from interval $a$ to $b$ and masks the subsequent $M^{s}$ time-steps taking care not to overlap with existing spans; poisson($\lambda$) and normal($\mu$, $\sigma$) sample $M^{s}$ from Poisson and normal distributions.

对于15秒长的音频样本,平均口罩长度为14.7个时步,对应299ms的音频,中位数为10个时步,最大时步约100个;样本中所有时步的约49%会被遮掩. 图2显示了相应遮罩长度分布的图案,表5显示了$M$和$p$的衰减以及其他遮罩策略的效果. 减少$M$会导致自监督者的预测准确性提高,但当一个长度被掩盖时,任务变得微不足道,导致下游语音识别任务表现不佳。 我们还考虑其他遮掩战略:w/o重叠统一($a$,$b$)从$a$到$b$间隔的每个起始指数样本长度为$M^{s}$,并遮掩随后的$M^{s}$时间步,注意不与现有跨度重叠;poisson($\lambda$)和正常($\mu$,$\sigma$)样本来自Poisson和正常分布的$M^{s}$.

Figure 2: Mask length distribution for a 15 second sample with $p=0.065$ and $M=10$.

图2:带有$p=0.065$和$M=10$的15个第二样本的面具长度分布.

	avg WER	std
Baseline ($p=0.075$)	7.97	0.02
Mask length $M=8$	8.33	0.05
Mask length $M=12$	8.19	0.08
Mask length $M=15$	8.43	0.19
Mask probability $p=0.065$	7.95	0.08
Mask probability $p=0.06$	8.14	0.22
Mask w/o overlap, uniform(1,31)	8.39	0.02
Mask w/o overlap, uniform(10,30)	9.17	0.05
Mask w/o overlap, poisson(15)	8.13	0.04
Mask w/o overlap, normal(15, 10)	8.37	0.03
Mask w/o overlap, length 10	9.15	0.02
Mask w/o overlap, length 15	9.43	0.26

Table 5: Ablations on settings for the masking strategy during pre-training. When masking without overlap, we choose starting time steps with $p=0.037$ which results in the total number of masked tokens to match the baseline.

	avg 微风	标准
Baseline ($p=0.075$)	7.97	0.02
遮罩长度 $M=8$	8.33	0.05
遮罩长度 $M=12$	8.19	0.08
遮罩长度 $M=15$	8.43	0.19
遮罩概率$p=0.065$	7.95	0.08
遮罩概率$p=0.06$	8.14	0.22
工作/工作重叠,统一(1,31)	8.39	0.02
工作/工作重叠,统一(10,30)	9.17	0.05
相重叠的遮罩( 15)	8.13	0.04
通常(15、10)	8.37	0.03
口罩重合,长度10	9.15	0.02
口罩重合,长度15	9.43	0.26

表5:在培训前对蒙面策略设置的考虑。 在没有重叠的情况下,我们选择与$p=0.037$的起始时间步骤,从而得出与基准相匹配的蒙面符号总数。

Appendix B Fine-tuning Setup

附录 B 微调设置

During fine-tuning we apply a masking strategy to the feature encoder outputs similar to SpecAugment [41]: we randomly choose a number of starting time steps for which a span of ten subsequent time-steps is replaced with a mask embedding; spans may overlap and we use the same masked time step embedding as during pre-training. We also mask channels by choosing a number of channels as starting indices and then expand each one to cover the subsequent 64 channels. Spans may overlap and the selected channel spans are set to zero value. We use LayerDrop [22, 12] at a rate of 0.05 for Base and 0.1 for Large during fine-tuning.

在勉强调整时,我们对类似于SpecAugment [41] 的特性编码器输出采用掩码策略:我们随机选择若干起始时间步骤,为此,将随后的10个间隔时间步替换为掩码嵌入;间隔时间可能重叠,我们使用与预训时相同的嵌入掩码时间步. 我们还通过选择一些频道作为起始指数来掩盖频道,然后扩大每个频道以覆盖随后的64个频道。 Spans可能重叠,选定频道跨度被设定为零值. 我们使用LyerDrop[22,12],在微调时Base的速率为0.05,大调0.1.

Table 6 summarizes the fine-tuning hyper-parameter settings used for the different labeled data setup. Table 7 shows the decoding parameters used for final evaluations of the various labeled data setups for Librispeech pre-trained models and Table 8 shows decoding parameters for LibriVox.

表6汇总了用于不同标签数据设置的微调超参数设置。 表7显示用于对Librispeech预训模型的各种标注数据集进行最终评价的解码参数,表8显示LibriVox的解码参数.

	timestep mask prob.	channel mask prob.	updates
10 min	0.075	0.008	12k
1 hour	0.075	0.004	13k
10 hours	0.065	0.004	20k
100 hours	0.05	0.008	50k
960 hours	0.05	0.0016	320k
TIMIT	0.065	0.012	40k

Table 6: Fine-tuning hyperparameters

	时间步口罩	频道口罩检测仪.	更新
10分钟	0.075	0.008	12k
1小时	0.075	0.004	13k
10小时	0.065	0.004	20k
100小时	0.05	0.008	50k
960小时	0.05	0.0016	320k
蒂米特	0.065	0.012	40k

表6:微调超参数

	4gram LM weight	4gram word insert.	TransLM weight	TransLM word insert.
10 min	3.23	-0.26	1.20	-1.39
1 hour	2.90	-1.62	1.15	-2.08
10 hours	2.46	-0.59	1.06	-2.32
100 hours	2.15	-0.52	0.87	-1.00
960 hours	1.74	0.52	0.92	-0.86

Table 7: Decoding parameters for Librispeech subsets for models pre-trained on Librispeech

	4克LM重量	4g 单词插入.	跨LM 重量	跨LM字插入.
10分钟	3.23	-0.26	1.20	-1.39
1小时	2.90	-1.62	1.15	-2.08
10小时	2.46	-0.59	1.06	-2.32
100小时	2.15	-0.52	0.87	-1.00
960小时	1.74	0.52	0.92	-0.86

表7: Librispeech 模型子集的解码参数

	4gram LM weight	4gram word insert.	TransLM weight	TransLM word insert.
10 min	3.86	-1.18	1.47	-2.82
1 hour	3.09	-2.33	1.33	-0.69
10 hours	2.12	-0.90	0.94	-1.05
100 hours	2.15	-0.52	0.87	-1.00
960 hours	1.57	-0.64	0.90	-0.31

Table 8: Decoding parameters for Librispeech subsets for models pre-trained on Librivox.

	将成为4克LM重量	4g 单词插入.	跨LM 重量	跨LM字插入.
10分钟	3.86	-1.18	1.47	-2.82
1小时	3.09	-2.33	1.33	-0.69
10小时	2.12	-0.90	0.94	-1.05
100小时	2.15	-0.52	0.87	-1.00
960小时	1.57	-0.64	0.90	-0.31

表8:Librispeech子集在Librivox上预训模型的解码参数.

Appendix C Full results for Libri-light and Librispeech

附录 C Libri-light 与 Librispeech 完整结果

Model	Unlabeled	LM	dev		test
data	clean	other		clean	other
10 min labeled
Base	LS-960	None	46.1	51.5		46.9	50.9
		4-gram	8.9	15.7		9.1	15.6
		Transf.	6.6	13.2		6.9	12.9
Large	LS-960	None	43.0	46.3		43.5	45.3
		4-gram	8.6	12.9		8.9	13.1
		Transf.	6.6	10.6		6.8	10.8
Large	LV-60k	None	38.3	41.0		40.2	38.7
		4-gram	6.3	9.8		6.6	10.3
		Transf.	4.6	7.9		4.8	8.2
1h labeled
Base	LS-960	None	24.1	29.6		24.5	29.7
		4-gram	5.0	10.8		5.5	11.3
		Transf.	3.8	9.0		4.0	9.3
Large	LS-960	None	21.6	25.3		22.1	25.3
		4-gram	4.8	8.5		5.1	9.4
		Transf.	3.8	7.1		3.9	7.6
Large	LV-60k	None	17.3	20.6		17.2	20.3
		4-gram	3.6	6.5		3.8	7.1
		Transf.	2.9	5.4		2.9	5.8
10h labeled
Base	LS-960	None	10.9	17.4		11.1	17.6
		4-gram	3.8	9.1		4.3	9.5
		Transf.	2.9	7.4		3.2	7.8
Large	LS-960	None	8.1	12.0		8.0	12.1
		4-gram	3.4	6.9		3.8	7.3
		Transf.	2.9	5.7		3.2	6.1
Large	LV-60k	None	6.3	9.8		6.3	10.0
		4-gram	2.6	5.5		3.0	5.8
		Transf.	2.4	4.8		2.6	4.9
100h labeled
Base	LS-960	None	6.1	13.5		6.1	13.3
		4-gram	2.7	7.9		3.4	8.0
		Transf.	2.2	6.3		2.6	6.3
Large	LS-960	None	4.6	9.3		4.7	9.0
		4-gram	2.3	5.7		2.8	6.0
		Transf.	2.1	4.8		2.3	5.0
Large	LV-60k	None	3.3	6.5		3.1	6.3
		4-gram	1.8	4.5		2.3	4.6
		Transf.	1.9	4.0		2.0	4.0

Table 9: WER on the Librispeech dev/test sets when training on the Libri-light low-resource labeled data setups (cf. Table 1).

enbergu 型号	无标签	内容	备忘		测试
数据	干净	其他项目		干净	其他项目
10分钟标签
基础	LS-960	无	46.1	51.5		46.9	50.9
		4克口径	8.9	15.7		9.1	15.6
		传译.	6.6	13.2		6.9	12.9
大块	LS-960	无	43.0	46.3		43.5	45.3
		4克口径	8.6	12.9		8.9	13.1
		传译.	6.6	10.6		6.8	10.8
大块	LV-60k (韩语)	无	38.3	41.0		40.2	38.7
		像 4 克一样	6.3	9.8		6.6	10.3
		传译.	4.6	7.9		4.8	8.2
1小时标签
基础	LS-960	无	24.1	29.6		24.5	29.7
		4克口径	5.0	10.8		5.5	11.3
		传译.	3.8	9.0		4.0	9.3
大块	LS-960	无	21.6	25.3		22.1	25.3
		4克口径	4.8	8.5		5.1	9.4
		传译.	3.8	7.1		3.9	7.6
大块	LV-60k (韩语)	无	17.3	20.6		17.2	20.3
		4克口径	3.6	6.5		3.8	7.1
		传译.	2.9	5.4		2.9	5.8
10小时标签
基础	LS-960	无	10.9	17.4		11.1	17.6
		4克口径	3.8	9.1		4.3	9.5
		传译.	2.9	7.4		3.2	7.8
大块	LS-960	无	8.1	12.0		8.0	12.1
		4克口径	3.4	6.9		3.8	7.3
		传译.	2.9	5.7		3.2	6.1
大块	LV-60k (韩语)	无	6.3	9.8		6.3	10.0
		4克口径	2.6	5.5		3.0	5.8
		传译.	2.4	4.8		2.6	4.9
** 100小时标签**
基础	LS-960	无	6.1	13.5		6.1	13.3
		4克口径	2.7	7.9		3.4	8.0
		传译.	2.2	6.3		2.6	6.3
大块	LS-960	无	4.6	9.3		4.7	9.0
		4克口径	2.3	5.7		2.8	6.0
		传译.	2.1	4.8		2.3	5.0
大块	LV-60k (韩语)	无	3.3	6.5		3.1	6.3
		4克口径	1.8	4.5		2.3	4.6
		传译.	1.9	4.0		2.0	4.0

表9: Librispeech dev/测试集上的WER,在Libri-light低资源标签数据集培训时使用WER(见表1)。

Model	Unlabeled	LM	dev		test
data	clean	other		clean	other
Large - from scratch	-	None	2.8	7.6		3.0	7.7
	-	4-gram	1.8	5.4		2.6	5.8
	-	Transf.	1.7	4.3		2.1	4.6
Base	LS-960	None	3.2	8.9		3.4	8.5
		4-gram	2.0	5.9		2.6	6.1
		Transf.	1.8	4.7		2.1	4.8
Large	LS-960	None	2.6	6.5		2.8	6.3
		4-gram	1.7	4.6		2.3	5.0
		Transf.	1.7	3.9		2.0	4.1
Large	LV-60k	None	2.1	4.5		2.2	4.5
		4-gram	1.4	3.5		2.0	3.6
		Transf.	1.6	3.0		1.8	3.3

Table 10: WER on Librispeech when using all 960 hours of Librispeech as labeled data (cf. Table 2).

型号	无标签	内容	备忘		测试
数据	干净	其他项目		干净	其他项目
大 - 从零开始	-	无	2.8	7.6		3.0	7.7
	-	4克口径	1.8	5.4		2.6	5.8
	-	传译.	1.7	4.3		2.1	4.6
之用	LS-960	无	3.2	8.9		3.4	8.5
		4克口径	2.0	5.9		2.6	6.1
		方面取得了重大进展。	1.8	4.7		2.1	4.8
大块	LS-960	无	2.6	6.5		2.8	6.3
		4克口径	1.7	4.6		2.3	5.0
		传译.	1.7	3.9		2.0	4.1
大块	LV-60k (韩语)	无	2.1	4.5		2.2	4.5
		4克口径	1.4	3.5		2.0	3.6
		传译.	1.6	3.0		1.8	3.3

表10:使用所有并予以标记的960小时Librispeech时使用Librispeech上的WER(见表2)。

Appendix D Analysis of Discrete Latent Speech Representations

附录 D 离散潜语音表征分析

Next, we investigate whether the discrete latent speech representations $\mathbf{q}_{t}$ learned by the quantizer relate to phonetic information: Using Large pre-trained on LV-60k and without any fine-tuning, we compute the discrete latents for the training data of TIMIT and compute the co-occurrence between human annotated phonemes and the latents. Ties are broken by choosing the phoneme which is most represented in the receptive field of $\mathbf{q}_{t}$. The training data contains 3696 utterances of average length 13.6 sec, or 563k discrete latents.

使用LV-60k上的大型预训, 不做任何微调, 我们计算TIMIT培训数据的离散潜伏, 选择在$\mathbf{q}_{t}$的可接受领域最能代表的电话, 培训数据包含平均长度为13.6秒的3696句话,即563千个离散潜伏物。

Figure 3plots $P(phoneme|\mathbf{q}_{t})$ and shows that many discrete latents appear to specialize in specific phonetic sounds. The silence phoneme (bcl) represents 22% of all human annotated speech data and is therefore also modeled by many different latents.

图3 pplots $P(phoneme|\mathbf{q}_{t})$,并显示许多离散的潜伏物似乎专门使用特定的语音声音。 并为其配音,

Figure 3: Visualization of the co-occurrence between discrete latent speech representations and phonemes. We plot the conditional probability $P(phoneme|\mathbf{q}_{t})$ on TIMIT train data. The y-axis shows the collapsed 39 classes of phonemes and the x-axis is over the different discrete latents.

图3:将离散的潜在语音表达和电话之间共同发生的现象的可视化。 我们在TIMIT列车数据上绘制了 $P(phoneme|\mathbf{q}_{t})$的有条件概率。 y轴显示崩溃的39类电话,x轴在不同的离散潜伏之上.

Appendix E Speech Recognition Error Analysis

附录 E 语音识别错误分析

In this section we study the most common errors our models make when fine-tuned on different amounts of labeled data (Table 11). We also show transcriptions of a few relatively challenging utterances from the dev-clean subset of Librispeech (Table 12).

在本节中,我们研究了我们模型在对不同数量标签数据进行微调时最常见的错误(表11)。 我们还从Librispeech的纯洁子集(表12)中摘录了一些相对具有挑战性的语音。

We consider models with no lexicon or no language model decoding, marked None in Table 9: Larger capacity decreases error rates: Large on LS-960 improves the word error rate on dev-clean from 46.1 to 43 compared to Base. Increasing the amount of unlabeled training data further decreases the error rate to 33.8 for Large on LS-960.

我们考虑没有词典或语言模型解码的模型,在表9中标注无: 容量更大的减误率:在LS-960上较大,与Base相比,dev-clean上的字错误率从46.1提高到43. 增加未贴标签的培训数据进一步将LS-960上大号的误差率降至33.8。

In the ten minute labeled data setup, the model is still able to recognize basic units of speech: Table 11 shows that most errors are around spelling of words, e.g., omitting silent characters such as could $\rightarrow$ coud, know $\rightarrow$ now, or ignoring repeated letters such as still $\rightarrow$ stil, little $\rightarrow$ litle. The Large LV-60k model achieves WER 38.3 on dev-clean and adding a Transformer language model enables to choose more likely pronunciations during the search and gives a large WER improvement to 5.0.

在10分钟的标注数据设置中,模型仍然能够识别基本的语音单位: 表11显示大多数错误围绕词的拼写,例如省略一些沉默的字符,如*could $\rightarrow$coud,知道$\rightarrow$now,或忽略重复的字母,如still $\rightarrow$stil, little *$\rightarrow$litle. 大型LV-60k模型在dev-clean上实现了WER 38.3,并增加了Transformer语言模型,使得在搜索过程中能够选择更可能的发音,并给5.0以较大的WER改进.

The ten minute models without lexicon and language model tend to spell words phonetically and omit repeated letters, e.g., will $\rightarrow$ wil (Table 11). Spelling errors decrease with more labeled data: with one hour of labeled data, slightly less common words move into the list of the most frequent errors, e.g., heaven and food are spelled phonetically. At ten hours, top errors include articles, e.g., a, the which are a common source of errors in speech recognition in general. There are also alternative spellings, color vs. colour as well as relatively rare words including person names, still spelled phonetically, e.g., phoebe $\rightarrow$ feeby.

没有词典和语言模型的10分钟模型倾向于拼写单词并省略重复字母,如*will $\rightarrow$wil (表11). 拼写错误随着更多的标注数据而减少:在标注数据一个小时后,少一些常见的单词会移动到最常见错误的列表中,例如,heaven和food是拼写. 在10小时,顶部错误包括文章,例如,a,*是一般语音识别错误的常见来源. 还有一些替代的拼法,即color* vs. color,以及相对罕见的单词,包括人名,仍然用拼音拼写,如phoebe *$\rightarrow$*feeby *.

At 100 hours, person names dominate the most frequent errors: phoebe $\rightarrow$ phebe, along with incorrect spacing anyone $\rightarrow$ any one, awhile $\rightarrow$ a while. Finally at 960 hours the word error rate falls to 2% and top errors are mostly articles, incorrect splits, and some very rare words or names such as deucalion or gryce.

在100时,人名在最常见的错误中占主导地位: *phoebe *$\rightarrow$*phebe *, 以及错误的间隔 * 任何人 $\rightarrow$ 任何一个 *, 同时 $\rightarrow$a 同时 . 最后在9时60分,单词错误率下降到2%,最高错误大多是文章,不正确的拆分,一些非常罕见的单词或名称如deucalion或gryce.

The “from scratch” 960 hour model has a similar word error rate as the 100 hour pre-trained model and displays a similar pattern of errors.

“从零开始”的960小时模型与100小时预先训练的模型具有类似的文字错误率,并显示类似的错误模式。

The pre-trained speech representations can be easily adapted to recognize specific sounds while fine-tuning grounds these representations to the actual spelling.

预先训练的语音表达方式可以很容易地调整,以识别具体的声音,同时将这些表达方式与实际拼写方式进行微调.

10m Large LV-60k	1h Large LV-60k	10h Large LV-60k
all $\rightarrow$ al (181)	too $\rightarrow$ to (26)	in $\rightarrow$ and (15)
are $\rightarrow$ ar (115)	until $\rightarrow$ untill (24)	a $\rightarrow$ the (11)
will $\rightarrow$ wil (100)	new $\rightarrow$ knew (22)	o $\rightarrow$ oh (10)
you $\rightarrow$ yo (90)	door $\rightarrow$ dor (18)	and $\rightarrow$ in (9)
one $\rightarrow$ on (89)	says $\rightarrow$ sais (18)	mode $\rightarrow$ mod (9)
two $\rightarrow$ to (81)	soul $\rightarrow$ sol (17)	ursus $\rightarrow$ ersus (9)
well $\rightarrow$ wel (80)	bread $\rightarrow$ bred (16)	tom $\rightarrow$ tome (8)
been $\rightarrow$ ben (73)	poor $\rightarrow$ pore (16)	randal $\rightarrow$ randol (7)
upon $\rightarrow$ apon (73)	a $\rightarrow$ the (13)	the $\rightarrow$ a (7)
good $\rightarrow$ god (67)	either $\rightarrow$ ither (13)	color $\rightarrow$ colour (6)
see $\rightarrow$ se (66)	food $\rightarrow$ fud (13)	flour $\rightarrow$ flower (6)
we $\rightarrow$ whe (60)	doubt $\rightarrow$ dout (12)	phoebe $\rightarrow$ feeby (6)
little $\rightarrow$ litle (54)	earth $\rightarrow$ erth (12)	an $\rightarrow$ and (5)
great $\rightarrow$ grate (53)	led $\rightarrow$ lead (12)	cucumbers $\rightarrow$ cucombers (5)
your $\rightarrow$ yor (53)	sea $\rightarrow$ see (12)	egg $\rightarrow$ eg (5)
could $\rightarrow$ coud (51)	thee $\rightarrow$ the (12)	macklewain $\rightarrow$ macklewaine (5)
here $\rightarrow$ hear (51)	tom $\rightarrow$ tome (12)	magpie $\rightarrow$ magpi (5)
know $\rightarrow$ now (45)	add $\rightarrow$ ad (11)	milner $\rightarrow$ millner (5)
there $\rightarrow$ ther (45)	good $\rightarrow$ god (11)	stacy $\rightarrow$ staci (5)
three $\rightarrow$ thre (45)	heaven $\rightarrow$ heven (11)	trevelyan $\rightarrow$ trevellion (5)
still $\rightarrow$ stil (42)	mary $\rightarrow$ marry (11)	verloc $\rightarrow$ verlock (5)
off $\rightarrow$ of (40)	randal $\rightarrow$ randel (11)	ann $\rightarrow$ an (4)
don’t $\rightarrow$ dont (37)	answered $\rightarrow$ ansered (10)	anyone $\rightarrow$ one (4)
shall $\rightarrow$ shal (36)	blood $\rightarrow$ blod (10)	apartment $\rightarrow$ appartment (4)
little $\rightarrow$ litl (35)	bozzle $\rightarrow$ bosel (10)	basin $\rightarrow$ bason (4)
100h Large LV-60k	960h Large LV-60k	960h Large from scratch
a $\rightarrow$ the (13)	a $\rightarrow$ the (12)	and $\rightarrow$ in (20)
and $\rightarrow$ in (10)	and $\rightarrow$ in (9)	a $\rightarrow$ the (16)
in $\rightarrow$ and (10)	macklewain $\rightarrow$ mackelwaine (7)	in $\rightarrow$ and (13)
o $\rightarrow$ oh (8)	in $\rightarrow$ and (6)	the $\rightarrow$ a (10)
minnetaki $\rightarrow$ minnitaki (7)	o $\rightarrow$ oh (6)	in $\rightarrow$ an (8)
randal $\rightarrow$ randall (7)	bozzle $\rightarrow$ bosell (5)	and $\rightarrow$ an (5)
christie $\rightarrow$ cristy (6)	criss $\rightarrow$ chris (5)	clarke $\rightarrow$ clark (4)
macklewain $\rightarrow$ mackelwane (6)	bozzle $\rightarrow$ bosel (4)	grethel $\rightarrow$ gretel (4)
randal $\rightarrow$ randoll (6)	clarke $\rightarrow$ clark (4)	macklewain $\rightarrow$ mackelwaine (4)
bozzle $\rightarrow$ bosall (5)	colored $\rightarrow$ coloured (4)	this $\rightarrow$ the (4)
kaliko $\rightarrow$ calico (5)	grethel $\rightarrow$ gretel (4)	an $\rightarrow$ and (3)
trevelyan $\rightarrow$ trevelian (5)	lige $\rightarrow$ lyge (4)	anyone $\rightarrow$ one (3)
an $\rightarrow$ and (4)	the $\rightarrow$ a (4)	bozzle $\rightarrow$ basell (3)
and $\rightarrow$ an (4)	and $\rightarrow$ an (3)	buns $\rightarrow$ bunds (3)
anyone $\rightarrow$ one (4)	ann $\rightarrow$ marianne (3)	carrie $\rightarrow$ carry (3)
bozzle $\rightarrow$ bozall (4)	butte $\rightarrow$ bute (3)	criss $\rightarrow$ chris (3)
clarke $\rightarrow$ clark (4)	color $\rightarrow$ colour (3)	he’s $\rightarrow$ is (3)
gryce $\rightarrow$ grice (4)	deucalion $\rightarrow$ ducalion (3)	his $\rightarrow$ is (3)
i’m $\rightarrow$ am (4)	forcemeat $\rightarrow$ meat (3)	honor $\rightarrow$ honour (3)
in $\rightarrow$ ind (4)	gryce $\rightarrow$ grice (3)	lattimer $\rightarrow$ latimer (3)
letty $\rightarrow$ lettie (4)	honor $\rightarrow$ honour (3)	millet $\rightarrow$ mellet (3)
phoebe $\rightarrow$ phebe (4)	kearny $\rightarrow$ kirney (3)	pyncheon $\rightarrow$ pension (3)
the $\rightarrow$ a (4)	nuova $\rightarrow$ noiva (3)	tad $\rightarrow$ ted (3)
ann $\rightarrow$ anne (3)	thing $\rightarrow$ anything (3)	thing $\rightarrow$ anything (3)
awhile $\rightarrow$ while (3)	this $\rightarrow$ the (3)	trevelyan $\rightarrow$ trevelian (3)

Table 11: Top word errors for models trained on 10m, 1h and 10h, 100h, 960h of labeled data and decoded on the Librispeech dev-clean subset without a language model or lexicon (see Table 9 and Table 10 - None). In brackets is the total number of occurrences of each error.

10m 大口径LV-60k	1小时大LV-60k	10小时大型LV-60k
总计$\rightarrow$al(181)	将$\rightarrow$改为(26)	和(促进可持续的)
为$\rightarrow$AR (115)	到$\rightarrow$ 至l(24)为止	a 第二十届(11届)
将$\rightarrow$周期( 100)	新$\rightarrow$知道(22)	o $\rightarrow$ oh (10)
20世纪90年代	门 XX 你没听到吗?	通过以下各项:
(89号)	说$\rightarrow$ sais(18)	模式$\rightarrow$ 模式(9)
二、第$\rightarrow$至第81段	灵魂$\rightarrow$sol(17)	欧松岛$\rightarrow$欧松岛 (9)
20世纪80年代	面包(16)	Tom$\rightarrow$tome (8) (中文(简体) ).
(73个)	贫困$\rightarrow$孔径(16)	Randal$\rightarrow$兰多尔(7)
(73号)	a 第二十届(第13届)	98. 将第的内容 许多方面都与《公约》第2条第1款有关。
优秀的$\rightarrow$神(67)	(13个)	颜色$\rightarrow$颜色 (6)
见$\rightarrow$ sel(66)	food(13)	面粉$\rightarrow$花(6)
我们$\rightarrow$ whe(60)	怀疑 XXMATH0工作报告(12)	phoebe$\rightarrow$收费乘以 (6)
小$\rightarrow$升 (54)	土 土 土 (12)	第$\rightarrow$和第(5)条
伟大的$\rightarrow$栅栏 (53)	牵头的$\rightarrow$ 铅(12)	以利于黄瓜$\rightarrow$昆虫 (5)
您的$\rightarrow$ yor (53) (中文(简体) ).	见(12)	卵$\rightarrow$等 (5)
(51个)	第二十届(12)	马克华因$\rightarrow$马克华因 (5)
在这里,第$\rightarrow$ 听到(51)	Tom Tom$\rightarrow$吨(12)	岩浆$\rightarrow$岩浆 (5)
提供资源,现在知道$\rightarrow$(45)	增加$\rightarrow$ 充分利用(11)	米纳$\rightarrow$磨机(5)
(45) (中文(简体) ).	不错的$\rightarrow$神(11)	stacy$\rightarrow$斯塔西(5)
3个$\rightarrow$(45个)	天堂(11)	Trevilyan$\rightarrow$千兆(5)
仍为$\rightarrow$stil (42)	Mary$\rightarrow$结婚(11)	verloc$\rightarrow$维洛克 (5)
离(40)$\rightarrow$	朗达尔$\rightarrow$朗德尔(11)	ann XXMATH0是否能使用 an (4)
不要$\rightarrow$ 不要(37)	便于使用	任何人
二十(36) 应	血液$\rightarrow$blod (10)	$\rightarrow$公寓(4)
小$\rightarrow$升 (35)	XX XXMA 瞧 XX (10)	$\rightarrow$ 火箭炮(4)
100小时大型LV-60k	960小时大型LV-60k	960h 从零开始较大
a 第二十届(13届)	a 第二十届(12)	在(20)中,
和第二十届特别会议(10)中	第(9)项中的第XX试验性	a 第二十届(16)
20世纪60年代,	马克勒瓦因$\rightarrow$马克勒瓦因 (7)	第$\rightarrow$和第(13)段
o $\rightarrow$ oh (8)	第$\rightarrow$和第(6)段	第二十届特别会议a(10)
minnetaki$\rightarrow$minnitaki (7)	o $\rightarrow$ oh (6)	in $\rightarrow$ an (8)
朗达尔$\rightarrow$朗德尔 (7)	bozle$\rightarrow$波塞尔 (5)	和$\rightarrow$ a (5)
Cristie$\rightarrow$摇篮(6)	criss$\rightarrow$chris (5) ris ris (5) (5)	克拉克$\rightarrow$克拉克 (4)
马克莱万$\rightarrow$马克莱万 (6)	bozle$\rightarrow$硼 (4)	葡萄糖$\rightarrow$葡萄糖 (4)
朗达尔$\rightarrow$朗多尔 (6)	克拉克$\rightarrow$克拉克 (4)	马可瓦因$\rightarrow$马可瓦因 (4)
bo$\rightarrow$ bo (5)	彩色$\rightarrow$彩色 (4)	第二十届执行会议的报告
kaliko$\rightarrow$口径(5)	葡萄糖$\rightarrow$葡萄糖 (4)	a禀第$\rightarrow$和(3)条。
特雷维利安$\rightarrow$特雷维利安 (5)	百合$\rightarrow$百合 (4)	任何人$\rightarrow$一(3)
第$\rightarrow$和第(4)条	第二十届特别会议a(4)	薄膜$\rightarrow$碱 (3)
和$\rightarrow$a (4)	和$\rightarrow$ a (3)	包子$\rightarrow$捆绑(3)
我必须承认,	ann$\rightarrow$马里烷(3)	Carrie$\rightarrow$载体(3)
bozle$\rightarrow$波扎尔 (4)	$\rightarrow$ 字节(3)	criss$\rightarrow$chris (3) 互联网档案馆的存檔,存档日期2013-09-02.
克拉克$\rightarrow$克拉克 (4)	颜色$\rightarrow$颜色 (3)	he’s $\rightarrow$ is (3)
甘油$\rightarrow$甘油 (4)	$\rightarrow$ 倍径(3)	他的$\rightarrow$为(3)
i’m $\rightarrow$ am (4)	强肉$\rightarrow$肉(3)	荣誉$\rightarrow$荣誉 (3)
$\rightarrow$ ind (4) 中的数据	gryce$\rightarrow$苦艾酒(3)	梯子$\rightarrow$梯子 (3)
Litty 知道了吗?	荣誉$\rightarrow$荣誉 (3)	小米$\rightarrow$薄荷 (3)
phebe$\rightarrow$phebe (4)	Kearny$\rightarrow$基尔尼(3)	20MATH0XX养恤金(3)
第二十届特别会议a(4)	nova$\rightarrow$Noiva (3) 维基语录链接:名人名言辞	tad$\rightarrow$ted(3)
ann$\rightarrow$ne (3)	东西$\rightarrow$ 任何东西(3)	东西$\rightarrow$ 任何东西(3)
第$\rightarrow$和(3)条	第二十届特别会议,第(3)段。	特雷维利安$\rightarrow$特雷维利安 (3)

表11:在无语言模型或词汇的Librispeech dev-clean子集上对标注数据进行10米、1小时和10小时、100小时、960小时培训的模型的顶字错误(见表9和表10-无)。 括号中为每次错误发生的总数。

Model	Transcription
Reference	i’m mister christopher from london
10m LV-60k	IM mister CRESTIFER FROME LUNDEN
1h LV-60k	IM mister CRISTIFFHER from LOUNDEN
10h LV-60k	i’m mister CHRYSTEPHER from london
100h LV-60k	i’m mister christopher from london
960h LV-60k	i’m mister christopher from london
960h scratch	I MISSTER christopher from london
Reference	il popolo e una bestia
10m LV-60k	ILPOPULAR ONABESTIA
1h LV-60k	O POPOLAONABASTIA
10h LV-60k	U POPULAONABASTIAR
100h LV-60k	O POPALOON A BASTYA
960h LV-60k	YOU’LL POP A LAWYE ON A BAISTYE
960h scratch	OL POPALOY ON ABESTIA
Reference	he smelt the nutty aroma of the spirit
10m LV-60k	he SMELTD the NUDY aroma of the spirit
1h LV-60k	he SMELTD the NUDDY ARROMA of the spirit
10h LV-60k	he smelt the NUDDY ERROMA of the spirit
100h LV-60k	he smelt the NUDDY aroma of the spirit
960h LV-60k	he smelt the NUTTIE aroma of the spirit
960h scratch	he smelt the nutty EROMA of the spirit
Reference	phoebe merely glanced at it and gave it back
10m LV-60k	FEABY MEARLY glanced at it and gave it BAK
1h LV-60k	FIEABY merely glanced at it and gave it back
10h LV-60k	FEEBY merely glanced at it and gave it back
100h LV-60k	BEBE merely glanced at it and gave it back
960h LV-60k	phoebe merely glanced at it and gave it back
960h scratch	phoebe merely glanced at it and gave it back
Reference	sauterne is a white bordeaux a strong luscious wine the best known varieties being
10m LV-60k	SULTERIN is a white BORDOE a strong LUCHOUS WIN the best NOWN VERIATYS being
1h LV-60k	CLTEREN is a white BORDO a strong LUCHIOUS wine the best known VERIETIES being
10h LV-60k	SOTERN is a white BOURDO a strong LUCIOUS wine the best known VORIETIES being
100h LV-60k	SOTERN is a white BORDAUX a strong LUCIOUS wine the best known varieties being
960h LV-60k	SOTERN is a white bordeaux a strong luscious wine the best known varieties being
960h scratch	SOTERAN is a white bordeaux a strong luscious wine the best known varieties being
Reference	i happen to have mac connell’s box for tonight or there’d be no chance of our getting places
10m LV-60k	i HAPEND to have MECONALES BOXS for TONIT ORE THIRLD be no chance of OR GETING places
1h LV-60k	i happen to have MACCONNEL’S BOCXS for tonight or TE’ELD be no chance of our getting places
10h LV-60k	i HAPPENED to have MUKONNEL’S box for tonight or THERED be no chance of our getting places
100h LV-60k	i HAPPENED to have MC CONNEL’S box for TO NIGHT or there’d be no chance of our getting places
960h LV-60k	i happen to have MC CONALL’S box for TO NIGHT or there’d be no chance of our getting places
960h scratch	i HAPPENE to have MACONEL’S box for TO NIGHT or there’d be no chance of our getting places

Table 12: Examples of transcription of selected utterances from the dev-clean subset by various models without a language model or lexicon. Capitalized words indicate errors.

型号	译名
参考	我是来自伦敦的克里斯托弗先生
10m LV-60k(百万吨)	伦登先生
1 với《公约》第2条	来自伦敦的克里瑟先生
10小时LV-60k	我是伦敦的克里斯·史蒂芬先生
100小时LV-60k	我是伦敦的克里斯托弗先生
960小时LV-60k	我是来自伦敦的克里斯托弗先生
960小时刮伤	我是伦敦的克里斯托弗
参考	(原始内容存档于2018-09-21). il popolo e una bestia
10m LV-60k(百万吨)	萨尔瓦多
1小时LV-60k	巴拿马
10小时LV-60k	萨尔瓦多
100小时LV-60k	欧巴马隆 A BASTYA
960小时LV-60k	你会把一个律师 在巴斯蒂耶
960小时刮伤	澳大利亚了
参考	他闻到精神的坚果香味
10m LV-60k(百万吨)	他闻到NUDY精神的香气
1小时LV-60k	他闻到灵魂的无声阿罗玛
10小时LV-60k	他闻到了灵魂的无声
100小时LV-60k	他闻到了NUDDY的气味
960小时LV-60k	他闻到了NUTTIE的气味
960小时刮伤	他闻到精神的坚果
参考	菲比只是看了一眼,还给我
10m LV-60k(百万吨)	FEABY 轻轻的看着它 并把它BAK
1小时LV-60k	FIEABY只是看了一眼,还给我
10小时LV-60k	菲比只是看了一眼,还给我
100小时LV-60k	只看一眼,还给我
960小时LV-60k	雏鸟只看一眼,还给我
960小时刮伤	菲比只是看了一眼,还给我
参考	苏打酒是白葡萄酒 浓郁的葡萄酒 最著名的品种是
10米LV的平等生产	Sulterin是一个白色的边框 别担心,最棒的就是现在
1小时LV-60k	CLTEREN是一个白色的BORDO 一种浓郁的葡萄酒 最著名的是
10小时LV-60k	SOTERN是一种白色的BOURDO 一种浓郁的葡萄酒 最著名的葡萄酒是
100小时LV-60k	SOTERN是一种白色的BORDAUX 一种浓郁的葡萄酒 最著名的品种是
960小时LV-60k	SOTERN是一种白色的酒 一种浓郁的葡萄酒 最著名的品种是
960小时刮伤	SOTERAN是一种白色 Octoberaux 一种浓郁的葡萄酒إ,最著名的品种是
参考	我碰巧有mac Connell的盒子 今晚或者没有机会 我们得到的地方
10m LV-60k(百万吨)	我希望能有米科纳莱斯瓶 为TONIT ORE三里尔 没有机会或得到的地方
1小时LV-60k	今晚我碰巧有MACONNEL的BOCXS 或者TEELD没有机会 我们得到的地方
10小时LV-60k	{\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}今晚我碰巧有MUKONNEL的盒子 {\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}否则我们不可能找到地方
100小时LV-60k	{\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}我碰巧有MC CONNEL的盒子 {\fn黑体\fs22\bord1\shad0\3aHBE\4aH00\fscx67\fscy66\2cHFFFFFF\3cH808080}或者我们没有机会找到地方
960小时LV-60k	我碰巧有MC CONALL的盒子 晚上或者没有机会 我们得到的地方
960小时刮伤	我碰巧 长官的盒子 晚上或者没有机会 我们得到的地方

表12:没有语言模型或词典的各种模型对从dev-clean子集中选定的语句进行抄录的例子。 大写单词表示错误.

Appendix F Ablations

附录 F 消融实验

Table 13ablates various hyperparameter choices of our architecture. The setup for the baseline model is described in § 5.4. First, we tried to improve the continuous input and continuous target model (§ 5.4) by adding an MLP on top of the continuous target representation and we also tried to use a separate set of encoder parameters for the representations used as input and targets (Separate encoders). Both did not lead to meaningful improvements.

表13列出了我们建筑的各种超参数选择。 基线模型的设置情况见第5.4节。 首先,我们试图改进连续输入和连续目标模型(§5.4),在连续目标表示值之上添加一个MLP,我们还试图对用作输入和目标的表示值(Separate编码器)使用一套单独的编码器参数. 两者都没有导致有意义的改进。

Increasing the receptive field size from 25ms to 30ms had little effect. Setting the diversity penalty weight ($\alpha{}$) too low results in lower codebook usage and lower performance. Setting it too high leads to slight instability. Doubling the number of relative positional embeddings to 256 also did not help. Stopping gradients from the quantizer to the encoder shows that the encoder requires training signal from the quantizer as well.