WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingThanks: * Equal contribution. The work was done at Microsoft during the internship of the first three authors.Thanks: Correspondence to Yu Wu: yuwu1@microsoft.com

Sanyuan Chen*    Chengyi Wang*    Zhengyang Chen*    Yu Wu*    Shujie Liu    Zhuo Chen Affiliation:  Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Affiliation:  Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei

Abstract

摘要

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. The code and pre-trained models are available at https://aka.ms/wavlm.

自监督学习（SSL）在语音识别上取得很大成功，但对其他语音处理任务的探索仍有限。语音信号包含说话人身份、副语言、说话内容等多侧面信息，为所有语音任务学习通用表征具有挑战。为解决该问题，我们提出预训练模型 WavLM，以应对全栈下游语音任务。WavLM 在预训练中联合学习掩码语音预测与去噪：掩码语音预测保留内容建模能力，语音去噪则提升对非 ASR 任务的潜力。此外，WavLM 在 Transformer 结构中采用门控相对位置偏置，以更好捕捉输入语音的序列顺序。我们还将训练数据从 6 万小时扩大到 9.4 万小时。WavLM Large 在 SUPERB 基准上达到当时最优，并在多项代表性基准的语音处理任务上带来显著提升。代码与预训练模型见 https://aka.ms/wavlm。

I Introduction

I 引言

Index Terms:

Over the past few years, self-supervised learning (SSL) has achieved great success in the fields of natural language processing (NLP) [1, 2, 3]. It leverages large amounts of text data to learn universal text representations, which can benefit almost all NLP downstream tasks by fine-tuning. Recently, SSL has also shown prominent results for speech processing, especially on phoneme classification [4] and automatic speech recognition (ASR) [5, 6, 7]. However, in other speech tasks, it is still the standard practice to train models from scratch with task-specific datasets.

索引词：

过去几年，自监督学习（SSL）在自然语言处理（NLP）领域取得巨大成功 [1, 2, 3]。它利用大量文本数据学习通用文本表征，几乎可通过微调惠及所有 NLP 下游任务。近期 SSL 在语音处理上也表现出色，尤其在音素分类 [4] 与自动语音识别（ASR）[5, 6, 7] 上。但在其他语音任务上，仍普遍用任务特定数据集从头训练模型。

Building a general pre-trained model for full stack speech processing tasks is essential to the further development of speech processing, because many tasks are short of supervised data, especially for non-ASR tasks. A model pre-trained on large-scale unlabeled data is able to boost the performance of these tasks, reduce data labeling efforts, and lower entry barriers for individual tasks. Furthermore, it is infeasible to build different pre-trained models for different downstream tasks, as the pre-training stage requires huge computational resources. In the past, it has been infeasible to build such a general model, as different tasks focus on different aspects of speech signals. For instance, speaker verification requires the network to learn the speaker characteristic regardless of the spoken content, while speech recognition demands the network to discard speaker characteristics and focus only on the content information. Meanwhile, unlike verification and recognition tasks, speaker diarization and speech separation involve multiple speakers, which creates additional obstacles to learning general speech representations. Recent advances fueled by large-scale pre-trained models have changed the situation. [8] proves the potential of pre-trained models on full-stack speech tasks by using the weighted sum of embeddings from different layers.[^1] They find different layers containing information useful for different tasks. For instance, the hidden states of the top layers are useful for ASR, while the bottom layers are more effective for speaker verification.

为全栈语音处理任务构建通用预训练模型，对语音处理的进一步发展至关重要：许多任务缺乏有监督数据，尤其是非 ASR 任务。在大规模未标注数据上预训练的模型，可提升这些任务表现、减少标注成本、降低各任务入门门槛。此外，因预训练阶段计算资源巨大，为不同下游任务分别建预训练模型并不现实。过去构建这类通用模型很难，因为不同任务关注语音信号的不同侧面：例如说话人验证要求网络学习说话人特征而忽略内容，语音识别则要求丢弃说话人特征、只关注内容；与验证/识别不同，说话人日志与语音分离涉及多说话人，给学习通用语音表征增添额外障碍。大规模预训练模型推动的进展改变了局面。[8] 通过对不同层嵌入加权求和，证明预训练模型在全栈语音任务上的潜力。[^1] 他们发现不同层包含对不同任务有用的信息：例如顶层隐状态利于 ASR，底层对说话人验证更有效。

While exciting as a proof of concept, there are still some drawbacks in existing pre-trained models: 1) Current pre-trained models are unsatisfactory for multi-speaker tasks, such as speaker diarization and speech separation. Our experiments show that speech separation models trained on top of HuBERT [6], a top-performing speech pre-trained model, achieve only marginal improvement compared with the models trained from scratch. This is mainly because the pre-training methods do not sufficiently enforce speaker discrimination, and the training data contain only single-speaker audios. 2) Speech pre-training crucially relies on high quality and large quantities of unlabeled audios. The existing system utilizes Libri-Light [9] as the main source, but the massive audiobook data mismatches the data in a real scenario and using it exclusively hurts the model performance when the acoustic characteristics of the downstream tasks are different from those of the audiobook [10, 11, 12, 13]. [14] trains wav2vec 2.0 [5] on larger and more diverse datasets, but there are still over 90$\%$ audio data derived from audiobook. To eliminate the audiobook data bias, we try to gather data from different sources as much as possible in our experiments.

尽管作为概念验证令人振奋，现有预训练模型仍有不足：1）对说话人日志、语音分离等多说话人任务表现不佳。实验表明，在顶尖语音预训练模型 HuBERT [6] 之上训练的语音分离模型，相对从头训练仅有边际提升。主因是预训练方法对说话人判别约束不足，且训练数据只含单说话人音频。2）语音预训练严重依赖高质量、大规模未标注音频。现有系统以 Libri-Light [9] 为主源，但海量有声书数据与真实场景不匹配；当下游任务声学特征与有声书不同时，仅用它会损害表现 [10, 11, 12, 13]。[14] 在更大、更多样数据上训练 wav2vec 2.0 [5]，但仍有超过 90% 音频来自有声书。为消除有声书偏置，我们在实验中尽可能从不同来源收集数据。

In this paper, we present WavLM, which learns universal speech representations from massive unlabeled speech data and adapts effectively across various speech processing tasks. We propose a masked speech denoising and prediction framework for WavLM, where some inputs are simulated noisy/overlapped speech with masks and the target is to predict the pseudo-label of the original speech on the masked region like HuBERT. The framework combines the masked speech prediction and denoising in pre-training. Therefore, the WavLM model learns not only the ASR information by the masked speech prediction, but also the knowledge of non-ASR tasks by the speech denoising modeling. For instance, the process of pseudo-label prediction on overlapped speech implicitly improves the model capability on diariazation and separation tasks. The speaker identity information and speech enhancement capability are modeled by the pseudo-label prediction on simulated noisy speech.

本文提出 WavLM：从海量未标注语音学习通用语音表征，并能有效适配多种语音处理任务。我们为 WavLM 提出掩码语音去噪与预测框架：部分输入为带掩码的模拟噪声/重叠语音，目标是像 HuBERT 一样在掩码区域预测原始语音的伪标签。该框架在预训练中结合掩码语音预测与去噪，使 WavLM 既通过掩码预测学习 ASR 信息，又通过去噪建模学习非 ASR 任务知识。例如，在重叠语音上预测伪标签，隐式提升日志与分离能力；在模拟噪声语音上预测伪标签，则建模说话人身份信息与语音增强能力。

In addition, we optimize the model structure and training data of HuBERT and wav2vec 2.0. We add gated relative position bias (grep) [15] to the Transformer structure as the backbone, which improves model performance for ASR and keeps almost the same parameter number and training speed. Compared with the convolutional relative position embedding used in wav2vec 2.0 and HuBERT, the gates allow the relative position bias to be adjusted adaptively by conditioning on the current speech content. To further improve the model robustness and alleviate the data mismatch, we scale up unlabeled pre-training data to 94k hours of public audios. The dataset consists of 60k hours of Libri-Light, 10k hours of GigaSpeech [16], and 24k hours of VoxPopuli [17]. The new dataset consists of training instances from different scenarios, such as podcasts, YouTube, and European Parliament (EP) event recordings.

此外，我们优化 HuBERT 与 wav2vec 2.0 的模型结构与训练数据。在作为骨干的 Transformer 中加入门控相对位置偏置（grep）[15]，提升 ASR 表现，同时参数量与训练速度几乎不变。相对 wav2vec 2.0 与 HuBERT 使用的卷积相对位置嵌入，门控允许相对位置偏置按当前语音内容自适应调整。为进一步提升稳健性并缓解数据不匹配，我们将未标注预训练数据扩大到 9.4 万小时公开音频，包括 6 万小时 Libri-Light、1 万小时 GigaSpeech [16] 与 2.4 万小时 VoxPopuli [17]。新数据覆盖播客、YouTube、欧洲议会（EP）活动录音等不同场景。

We evaluate our models on nineteen subtasks, fifteen of which are from SUPERB, and the other four are classic speech tasks on their representative testsets.

我们在十九项子任务上评估：其中十五项来自 SUPERB，另四项为各代表性测试集上的经典语音任务。

• • WavLM achieves state-of-the-art (SOTA) performance on SUPERB [8, 18]. WavLM Large outperforms HuBERT Large on 14 subtasks, and achieves an absolute 2.4 point improvement in the overall evaluation. Even WavLM Base+, a 3 times smaller model, is better than HuBERT Large owing to our three modifications.

• • WavLM 在 SUPERB [8, 18] 上达到当时最优（SOTA）。WavLM Large 在 14 项子任务上超过 HuBERT Large，总体评估绝对提升 2.4 分。即便参数约小 3 倍的 WavLM Base+，也因我们的三项改动优于 HuBERT Large。

• • Speaker verification is a task to verify the speaker’s identity from the voice characteristics. We select this task to evaluate the model’s capability of extracting speaker-related features. WavLM Large exceeds the well-known SOTA system, ECAPA-TDNN [19], by a large margin and achieves 0.383%, 0.480% and 0.986% EER (Equal Error Rate) on the three official trial lists of VoxCeleb1 [20].

• • 说话人验证从声音特征核实说话人身份。我们用该任务评估模型提取说话人相关特征的能力。WavLM Large 大幅超过知名 SOTA 系统 ECAPA-TDNN [19]，在 VoxCeleb1 [20] 三条官方 trial 列表上分别达到 0.383%、0.480% 与 0.986% EER（等错误率）。

• • Speech separation is a classic multi-speaker task, which is the key to solving the cocktail party problem. The task can evaluate the model’s capability of extracting multiple speech signals from a mixture of sounds. WavLM achieves SOTA performance on the speech separation LibriCSS benchmark [21], and significantly outperforms the previous Conformer model [22] by a 27.7$\%$ relative word error rate (WER) reduction.

• • 语音分离是经典多说话人任务，是解决鸡尾酒会问题的关键。该任务评估模型从混合声音中提取多路语音信号的能力。WavLM 在语音分离 LibriCSS 基准 [21] 上达到 SOTA，相对先前 Conformer 模型 [22] 相对词错误率（WER）显著降低 27.7%。

• • Speaker diarization is a task to recognize “who spoke when” from an input audio stream [23]. WavLM achieves SOTA performance on the CALLHOME speaker diarization benchmark. Compared to the EEND-EDA clustering method [24], our model achieves a 12.6$\%$ diarization error rate reduction.

• • 说话人日志要从输入音频流识别「谁在何时说话」[23]。WavLM 在 CALLHOME 说话人日志基准上达到 SOTA；相对 EEND-EDA 聚类方法 [24]，日志错误率降低 12.6%。

• • Speech recognition requires the model to learn content information, which is the main focus of the previous SSL work. We evaluate our model in the LibriSpeech 960h setting. WavLM shows comparable performance to the wav2vec 2.0 and HuBERT, which achieves 1.8% and 3.2% WER on the test-clean and test-other sets, respectively.

• • 语音识别要求模型学习内容信息，也是先前 SSL 工作的主要焦点。我们在 LibriSpeech 960h 设定下评估：WavLM 与 wav2vec 2.0、HuBERT 相当，在 test-clean 与 test-other 上分别达到 1.8% 与 3.2% WER。

The contribution of the paper can be summarized as follows: 1) WavLM sheds light on a general pre-trained model for full stack speech processing tasks, in contrast to the previous SSL works focusing on a group of similar tasks. 2) We propose simple but effective modifications to the existing pre-trained models, which show general and consistent improvements across downstream tasks. 3) We scale-up self-supervised speech pre-training with more unlabeled data and longer training steps. 4) We achieve SOTA results on the SUPERB benchmark, and significantly boost the performance for various speech processing tasks on their representative benchmarks, including speech separation, speaker verification, and speaker diarization. The models and code are released [^2] to facilitate future research.

本文贡献可概括为：1）WavLM 为全栈语音处理任务揭示通用预训练模型路径，相对先前聚焦一组相似任务的 SSL 工作；2）我们对现有预训练模型提出简单而有效的改动，在下游任务上带来普遍且一致的提升；3）我们用更多未标注数据与更长训练步数放大自监督语音预训练；4）我们在 SUPERB 基准上达到 SOTA，并在语音分离、说话人验证、说话人日志等代表性基准上显著提升表现。模型与代码已发布 [^2] 以促进后续研究。

II Related Work

II 相关工作

SSL methods can be categorized into generative learning, discriminative learning, and multitask learning, based on the training objective. The research line of generative learning can be traced back to the auto-encoding model, which reconstructs the whole speech from latent variables, either continuous [25, 26, 27] or discrete [28]. Recent works propose to predict future frames from the history with an autoregressive model [29, 30, 31, 32], or recover the masked frames from the corrupted speech with a non-autoregressive model [33, 34, 35, 36, 37, 38]. Apart from generative learning, discriminative learning has also gathered interests recently. The well-known examples include CPC [4], wav2vec [39], vq-wav2vec [40], wav2vec 2.0 [5], DiscreteBERT [41], HuBERT [6] and w2v-BERT [42]. CPC and the wav2vec series models use the contrastive InfoNCE loss to discriminate the positive samples from negative samples. Motivated by the masked language model loss in NLP, DiscreteBERT and HuBERT predict discrete targets of masked regions. w2v-BERT further combines the contrastive loss and the masked prediction loss in an end-to-end fashion. Multi-task learning is adopted in PASE [43] and PASE+ [44]. They employ lots of pre-training objectives such as waveform generation, prosody regression, and contrastive objectives. UniSpeech [7] and JUST [45] combine SSL and supervised learning for ASR, and show impressive results on multi-lingual test sets.

按训练目标，SSL 方法可分为生成式学习、判别式学习与多任务学习。生成式学习可追溯到自编码模型：从连续 [25, 26, 27] 或离散 [28] 潜变量重建整段语音。近期工作用自回归模型从历史预测未来帧 [29, 30, 31, 32]，或用非自回归模型从损坏语音恢复掩码帧 [33, 34, 35, 36, 37, 38]。除生成式外，判别式学习也受关注，知名例子包括 CPC [4]、wav2vec [39]、vq-wav2vec [40]、wav2vec 2.0 [5]、DiscreteBERT [41]、HuBERT [6] 与 w2v-BERT [42]。CPC 与 wav2vec 系列用对比 InfoNCE 损失区分正负样本；受 NLP 掩码语言模型损失启发，DiscreteBERT 与 HuBERT 预测掩码区域的离散目标；w2v-BERT 进一步端到端结合对比损失与掩码预测损失。PASE [43] 与 PASE+ [44] 采用多任务学习，使用波形生成、韵律回归与对比目标等多种预训练目标。UniSpeech [7] 与 JUST [45] 为 ASR 结合 SSL 与监督学习，在多语言测试集上表现突出。

Unlike SSL in computer vision (CV) and NLP fields, where one pre-trained model is adapted to various downstream tasks, most speech SSL methods focus on phoneme classification and ASR. Recently, [8] proposed the SUPERB benchmark to evaluate SSL models across different tasks. According to the results, HuBERT enjoys the best generalization ability in the overall evaluation. To better learn speaker characteristics, [46] proposed UniSpeech-SAT, which extends the HuBERT framework with speaker-aware pre-training. It significantly outperforms other pre-trained models on the speaker-related tasks with a slight degradation on the ASR.

与 CV、NLP 中「一个预训练模型适配多种下游」不同，多数语音 SSL 聚焦音素分类与 ASR。近期 [8] 提出 SUPERB 基准，跨任务评估 SSL 模型；结果表明 HuBERT 在总体评估上泛化最好。为更好学习说话人特征，[46] 提出 UniSpeech-SAT，在 HuBERT 框架上扩展说话人感知预训练：说话人相关任务显著超过其他预训练模型，ASR 略有下降。

Compared with existing works, our model is the first to explore SSL for full stack tasks instead of focusing on ASR or other specific tasks. It should be noted that a concurrent work BigSSL [47] also mentions large SSL model could handle various speech tasks. The difference is that our work demonstrates that the full stack tasks can be handled by the careful pre-training and fine-tuning strategy design, even without scaling up the model size to 8 billion parameters.

相对既有工作，我们的模型首次探索面向全栈任务的 SSL，而非聚焦 ASR 或其他特定任务。需指出，同期工作 BigSSL [47] 也提到大型 SSL 模型可处理多种语音任务；差别在于：我们表明即便不把模型放大到 80 亿参数，通过仔细的预训练与微调策略设计也能处理全栈任务。

III Background: HuBERT

III 背景：HuBERT

HuBERT is an SSL method that benefits from an offline clustering step to provide target labels for a BERT-like prediction loss [1]. The backbone is a Transformer encoder [48] with $L$ blocks. During pre-training, the Transformer consumes masked acoustic features $\mathbf{u}$ and outputs hidden states $\mathbf{h}^{L}$. The network is optimized to predict the discrete target sequence $\mathbf{z}$, where each $z_{t}\in[C]$ is a $C$-class categorical variable. The distribution over codewords is parameterized with

HuBERT 是一种 SSL 方法：借助离线聚类为类 BERT 的预测损失提供目标标签 [1]。骨干为含 $L$ 个块的 Transformer 编码器 [48]。预训练时，Transformer 消费掩码声学特征 $\mathbf{u}$，输出隐状态 $\mathbf{h}^{L}$。网络被优化以预测离散目标序列 $\mathbf{z}$，其中每个 $z_{t}\in[C]$ 为 $C$ 类分类变量。码字分布参数化为

$$p(c|\mathbf{h}_{t}^{L})=\frac{{\rm exp}({\rm sim}(\mathbf{h}_{t}^{L}\mathbf{W}^{P},\mathbf{e}_{c})/\tau)}{\sum_{c^{\prime}=1}^{C}{\rm exp}({\rm sim}(\mathbf{h}_{t}^{L}\mathbf{W}^{P},\mathbf{e}_{c^{\prime}})/\tau)} \tag{1}$$

where $\mathbf{W}^{P}$ is a projection matrix, $\mathbf{h}_{t}^{L}$ is the output hidden state for step $t$, $\mathbf{e}_{c}$ is the embedding for codeword $c$, sim($a,b$) computes the cosine similarity and $\tau=0.1$ scales the logit. HuBERT proposes a masked speech prediction task, where the prediction loss is only applied over the masked regions, forcing the model to learn a combined acoustic and language model over the continuous inputs.

其中 $\mathbf{W}^{P}$ 为投影矩阵，$\mathbf{h}_{t}^{L}$ 为第 $t$ 步输出隐状态，$\mathbf{e}_{c}$ 为码字 $c$ 的嵌入，sim($a,b$) 计算余弦相似度，$\tau=0.1$ 缩放 logits。HuBERT 提出掩码语音预测任务：预测损失只作用在掩码区域，迫使模型在连续输入上学习声学与语言模型的结合。

HuBERT adopts an iterative re-clustering and re-training process: For the first iteration, the targets are assigned by clustering the MFCC features of the training data; For the second iteration, a new generation of training targets are created by clustering the latent representations generated by the first iteration trained model.

HuBERT 采用迭代重聚类与重训练：第一轮目标由对训练数据 MFCC 特征聚类得到；第二轮用第一轮训练模型生成的潜表征再聚类，产生新一代训练目标。

Fig. 1: Model Architecture.

图 1：模型结构。

IV WavLM

We propose a masked speech denoising and prediction framework, where some inputs are simulated noisy/overlapped with masks and the target is to predict pseudo-labels of the original speech on the masked region. Unlike existing masked speech modeling (HuBERT), which just focuses on the ASR task, the masked speech denoising allows us to extend pre-trained speech models to non-ASR tasks, since it implicitly models information we need in the speaker identification, separation, and diarization tasks. We further optimize the Transformer backbone and extend pre-training data to 94k public English data.

我们提出掩码语音去噪与预测框架：部分输入为带掩码的模拟噪声/重叠语音，目标是在掩码区域预测原始语音的伪标签。与仅聚焦 ASR 的既有掩码语音建模（HuBERT）不同，掩码语音去噪使我们能把预训练语音模型扩展到非 ASR 任务，因其隐式建模说话人识别、分离与日志所需信息。我们进一步优化 Transformer 骨干，并将预训练数据扩展到 9.4 万小时公开英文数据。

IV-A Model Structure

IV-A 模型结构

Our model architecture uses the Transformer model as the backbone. As shown in Figure 1, it contains a convolutional feature encoder and a Transformer encoder. The convolutional encoder is composed of seven blocks of temporal convolution followed by layer normalization and a GELU activation layer. The temporal convolutions have 512 channels with strides (5,2,2,2,2,2,2) and kernel widths (10,3,3,3,3,2,2), resulting in each output representing about 25ms of audio strode by 20ms. The convolutional output representation $\mathbf{x}$ is masked as the Transformer input. The Transformer is equipped with a convolution-based relative position embedding layer with 128 kernel size and 16 groups at the bottom.

模型结构以 Transformer 为骨干。如图 1 所示，包含卷积特征编码器与 Transformer 编码器。卷积编码器由七个时序卷积块组成，后接层归一化与 GELU 激活；时序卷积 512 通道，步长 (5,2,2,2,2,2,2)、核宽 (10,3,3,3,3,2,2)，使每个输出约代表 25ms 音频、步长 20ms。卷积输出表征 $\mathbf{x}$ 经掩码后作为 Transformer 输入。Transformer 底部配备核大小 128、分组 16 的基于卷积的相对位置嵌入层。

To improve the model, we employ gated relative position bias [15] which is encoded based on the offset between the “key” and “query” in the Transformer self-attention mechanism. Let $\{\mathbf{h}_{i}\}_{i=1}^{T}$ denote the input hidden states for the self-attention module, each $\mathbf{h}_{i}$ is linearly projected to a triple of query, key and value $(\mathbf{q}_{i},\mathbf{k}_{i},\mathbf{v_{i}})$ as:

为改进模型，我们采用门控相对位置偏置 [15]：按 Transformer 自注意力中「key」与「query」的偏移编码。令 $\{\mathbf{h}_{i}\}_{i=1}^{T}$ 为自注意力模块的输入隐状态，每个 $\mathbf{h}_{i}$ 线性投影为 query、key、value 三元组 $(\mathbf{q}_{i},\mathbf{k}_{i},\mathbf{v_{i}})$：

$$\mathbf{q}_{i},\mathbf{k}_{i},\mathbf{v}_{i}=\mathbf{h}_{i}\mathbf{W}^{Q},\mathbf{h}_{i}\mathbf{W}^{K},\mathbf{h}_{i}\mathbf{W}^{V} \tag{2}$$

The self-attention outputs $\{\tilde{\mathbf{h}}_{i}\}_{i=1}^{T}$ are computed via:

自注意力输出 $\{\tilde{\mathbf{h}}_{i}\}_{i=1}^{T}$ 计算为：

$$\displaystyle a_{ij} \\ \displaystyle\propto\exp\{\frac{\mathbf{q}_{i}\cdot\mathbf{k}_{j}}{\sqrt{d_{k}}}+r_{i-j}\} \\ \displaystyle\tilde{\mathbf{h}}_{i} \\ \displaystyle=\sum_{j=1}^{T}{a_{ij}\mathbf{v}_{j}} \tag{3}$$

where $r_{i-j}$ is the gated relative position bias added to the attention logits. It is computed by:

其中 $r_{i-j}$ 为加到注意力 logits 上的门控相对位置偏置，计算为：

$$\displaystyle g_{i}^{\text{(update)}},g_{i}^{\text{(reset)}} \\ \displaystyle=\sigma(\mathbf{q}_{i}\cdot\mathbf{u}),\sigma(\mathbf{q}_{i}\cdot\mathbf{w}) \\ \displaystyle\tilde{r}_{i-j} \\ \displaystyle=wg_{i}^{\text{(reset)}}d_{i-j} \\ \displaystyle r_{i-j}=d_{i-j}+g_{i}^{\text{(update)}} \\ \displaystyle d_{i-j}+(1-g_{i}^{\text{(update)}})\tilde{r}_{i-j}$$

where $d_{i-j}$ is a learnable scalar relative position bias, the vectors $\mathbf{u},\mathbf{w}\in\mathbb{R}^{d_{k}}$ are learnable parameters, $\sigma$ is a sigmoid function, and $w$ is a learnable value.

其中 $d_{i-j}$ 为可学习标量相对位置偏置，向量 $\mathbf{u},\mathbf{w}\in\mathbb{R}^{d_{k}}$ 为可学习参数，$\sigma$ 为 sigmoid，$w$ 为可学习值。

In our work, $d_{i-j}$ is a bucket relative position embedding [3] and the embedding parameters are shared across all layers. We use $n=320$ embeddings and each corresponds to a range of possible $(i-j)$ offsets. The range increased logarithmically up to a maximum offset of $m=800$, beyond which we assign all relative offsets to the same embedding, i.e.,

本文中 $d_{i-j}$ 为桶相对位置嵌入 [3]，嵌入参数跨层共享。我们使用 $n=320$ 个嵌入，各对应一段可能的 $(i-j)$ 偏移；范围对数增长至最大偏移 $m=800$，超出后所有相对偏移共用同一嵌入，即

$$d_{|i-j|}=\begin{cases}|i-j|,&|i-j|<\frac{n}{4}\
\lfloor\frac{n}{4}(\frac{\log(|i-j|)-\log(\frac{n}{4})}{\log(m)-\log(\frac{n}{4})}+1)\rfloor,&\frac{n}{4}\leq|i-j|<m\
\frac{n}{2}-1,&|i-j|\geq m\
\end{cases} \tag{5}$$

$$d_{i-j}=d_{|i-j|}+\frac{n}{2}\cdot\mathds{1}_{\{i-j>0\}} \tag{6}$$

Compared with the convolutional relative position embedding in wav2vec 2.0 and HuBERT, the gates take the content into consideration, and adaptively adjust the relative position bias by conditioning on the current speech content. Intuitively, the same distance offset between two frames tends to play different roles if one frame is the silence while the other belongs to a speech segment.

相对 wav2vec 2.0 与 HuBERT 中的卷积相对位置嵌入，门控会考虑内容，并按当前语音内容自适应调整相对位置偏置。直观上，若一帧为静音而另一帧属于语音段，两帧之间相同距离偏移往往扮演不同角色。

1:  given a batch of speech utterances $\mathbf{U}=\{\mathbf{u}^{i}\}_{i=1}^{B}$ with batch size $B$ and length $L$, mixing probability $p$, a set of DNS noises $\mathbf{N}=\{\mathbf{n}^{i}\}_{i=1}^{M}$ with size $M$, mixing noise probability $p_{n}$
2:  Choose $S$ utterances $\mathbf{U}^{S}\subset\mathbf{U}$ by Bernoulli sampling with probability $p$
3:  for each primary utterance $\mathbf{u}^{\text{pri}}\in\mathbf{U}^{S}$ do
4:    Sample a random value $v$ from the continuous uniform distribution $\mathcal{U}(0,1)$
5:    if $v>p_{n}$ then
6:    Sample a secondary utterance $\mathbf{u}^{\text{sec}}$ from discrete uniform distribution with probability $P(\mathbf{u}^{\text{sec}}=\mathbf{x})=\frac{1}{B},\mathbf{x}\in\mathbf{U}$
7:    Sample the mixing energy ratio $r$ from the continuous uniform distribution $\mathcal{U}(-5,5)$
8:    else
9:    Sample a noise $\mathbf{u}^{\text{sec}}$ from discrete uniform distribution with probability $P(\mathbf{u}^{\text{sec}}=\mathbf{x})=\frac{1}{M},\mathbf{x}\in\mathbf{N}$
10:    Sample the mixing energy ratio $r$ from the continuous uniform distribution $\mathcal{U}(-5,20)$
11:    end if
12:    Sample the mix length $l$ from discrete uniform distribution with probability $P(l=x)=\frac{2}{L},x\in\{1,\cdots,\frac{L}{2}\}$
13:    Sample a start position $s^{\text{pri}}$ of $\mathbf{u}^{\text{pri}}$ from discrete uniform distribution with probability $P(s^{\text{pri}}=x)=\frac{1}{L-l},x\in\{1,\cdots,L-l\}$
14:    Sample a start position $s^{\text{sec}}$ of $\mathbf{u}^{\text{sec}}$ from discrete uniform distribution with probability $P(s^{\text{sec}}=x)=\frac{1}{L-l},x\in\{1,\cdots,L-l\}$
15:    Calculate the energy of the primary utterance $E^{\text{pri}}\leftarrow\frac{\sum\mathbf{u}^{\text{pri}}\cdot\mathbf{u}^{\text{pri}}}{L}$
16:    Calculate the energy of the secondary utterance $E^{\text{sec}}\leftarrow\frac{\sum\mathbf{u}^{\text{sec}}\cdot\mathbf{u}^{\text{sec}}}{L}$
17:    Calculate the mixing scale $scl\leftarrow\sqrt{\frac{E^{\text{pri}}}{10^{\frac{r}{10}}E^{\text{sec}}}}$
18:    $\mathbf{u}^{\text{pri}}[s^{\text{pri}}:s^{\text{pri}}+l]\leftarrow\mathbf{u}^{\text{pri}}[s^{\text{pri}}:s^{\text{pri}}+l]+scl\cdot\mathbf{u}^{\text{sec}}[s^{\text{sec}}:s^{\text{sec}}+l]$
19:  end for
20:  return $\mathbf{U}$

Algorithm 1 Noisy/Overlapped Speech Simulation

1:  给定一批 of speech utterances $\mathbf{U}=\{\mathbf{u}^{i}\}_{i=1}^{B}$ with batch size $B$ and length $L$, 混合概率 $p$, a set of DNS noises $\mathbf{N}=\{\mathbf{n}^{i}\}_{i=1}^{M}$ with size $M$, 混噪概率 $p_{n}$
2:  按伯努利采样以概率 $p$ 选取 $S$ 条语句 $\mathbf{U}^{S}\subset\mathbf{U}$ by Bernoulli sampling with probability $p$
3:  for 每条主语句 $\mathbf{u}^{\text{pri}}\in\mathbf{U}^{S}$ do
4:    采样随机值 $v$ from the continuous uniform distribution $\mathcal{U}(0,1)$
5:    if $v>p_{n}$ then
6:    采样次要语句 $\mathbf{u}^{\text{sec}}$ from discrete uniform distribution with probability $P(\mathbf{u}^{\text{sec}}=\mathbf{x})=\frac{1}{B},\mathbf{x}\in\mathbf{U}$
7:    采样混合能量比 $r$ from the continuous uniform distribution $\mathcal{U}(-5,5)$
8:    else
9:    采样噪声 $\mathbf{u}^{\text{sec}}$ from discrete uniform distribution with probability $P(\mathbf{u}^{\text{sec}}=\mathbf{x})=\frac{1}{M},\mathbf{x}\in\mathbf{N}$
10:    采样混合能量比 $r$ from the continuous uniform distribution $\mathcal{U}(-5,20)$
11:    end if
12:    采样混合长度 $l$ from discrete uniform distribution with probability $P(l=x)=\frac{2}{L},x\in\{1,\cdots,\frac{L}{2}\}$
13:    采样起始位置 $s^{\text{pri}}$ of $\mathbf{u}^{\text{pri}}$ from discrete uniform distribution with probability $P(s^{\text{pri}}=x)=\frac{1}{L-l},x\in\{1,\cdots,L-l\}$
14:    采样起始位置 $s^{\text{sec}}$ of $\mathbf{u}^{\text{sec}}$ from discrete uniform distribution with probability $P(s^{\text{sec}}=x)=\frac{1}{L-l},x\in\{1,\cdots,L-l\}$
15:    计算主语句能量 $E^{\text{pri}}\leftarrow\frac{\sum\mathbf{u}^{\text{pri}}\cdot\mathbf{u}^{\text{pri}}}{L}$
16:    计算次要语句能量 $E^{\text{sec}}\leftarrow\frac{\sum\mathbf{u}^{\text{sec}}\cdot\mathbf{u}^{\text{sec}}}{L}$
17:    计算混合尺度 $scl\leftarrow\sqrt{\frac{E^{\text{pri}}}{10^{\frac{r}{10}}E^{\text{sec}}}}$
18:    $\mathbf{u}^{\text{pri}}[s^{\text{pri}}:s^{\text{pri}}+l]\leftarrow\mathbf{u}^{\text{pri}}[s^{\text{pri}}:s^{\text{pri}}+l]+scl\cdot\mathbf{u}^{\text{sec}}[s^{\text{sec}}:s^{\text{sec}}+l]$
19:  end for
20:  返回 $\mathbf{U}$

算法 1 噪声/重叠语音仿真

IV-B Masked Speech Denoising and Prediction

IV-B 掩码语音去噪与预测

We propose a masked speech denoising and prediction framework to improve model robustness for complex acoustic environments and the preservation of speaker identity. Specifically, we manually simulated noisy/overlapped speech as inputs, and predict the pseudo-labels of original speech on the masked region.

我们提出掩码语音去噪与预测框架，以提升模型对复杂声学环境的稳健性并更好保留说话人身份。具体做法是：人工仿真噪声/重叠语音作为输入，在掩码区域预测原始语音的伪标签。

We simulate the noisy speech with multiple speakers and various background noise for self-supervised pre-training. We randomly select some utterances from each training batch and mix them with a randomly selected noise audio or secondary utterance at a random region. The noise audio and secondary utterance are randomly selected from the same batch, randomly cropped, and scaled by a random source energy ratio. We ensure that the overlap region is less than 50% and take the speaker from the first utterance as the main speaker. With the masked speech denoising and prediction task, the model is trained to identify the main speaker from the noisy/overlapped speech and predict the content information corresponding to the main speaker with the mask prediction loss.

我们用多说话人与多种背景噪声仿真噪声语音，用于自监督预训练。从每个训练 batch 随机选取部分语句，在随机区域与随机选取的噪声音频或次要语句混合。噪声与次要语句从同一 batch 随机选取、随机裁剪，并按随机源能量比缩放。我们保证重叠区域小于 50%，并以第一条语句的说话人为主说话人。借助掩码语音去噪与预测任务，模型被训练从噪声/重叠语音中识别主说话人，并用掩码预测损失预测对应主说话人的内容信息。

IV-B1 Noisy/Overlapped Speech Simulation

The details of our noisy/overlapped speech simulation method are shown in Algorithm 1. Given a batch of speech utterances $\mathbf{U}=\{\mathbf{u}^{i}\}_{i=1}^{B}$ with batch size $B$ and length $L$ and a set of DNS (Deep Noise Suppression) noises [49] $\mathbf{N}=\{\mathbf{n}^{i}\}_{i=1}^{M}$ with size $M$ (line 1), we first randomly choose $S$ utterances to mix $\mathbf{U}^{S}=\{\mathbf{u}^{i}\}_{i=1}^{S}$ from the batch by Bernoulli sampling with probability $p$ (line 2). Then, for each utterance $\mathbf{u}^{\text{pri}}\in\mathbf{U}^{S}$ (line 3), we sample a random value $v$ the continuous uniform distribution $\mathcal{U}(0,1)$ to decide whether to mix noise or a secondary utterance (line 4). If the random value $v$ is greater than the mixing noise probability $p_{n}$ (line 5), we sample a secondary utterance $\mathbf{u}^{\text{sec}}$ from a discrete uniform distribution over the batch $\mathbf{U}$ (line 6), and randomly sample the mixing energy ratio $r$ from the uniform distribution $\mathcal{U}(-5,5)$ (line 7). Otherwise, we sample a noise $\mathbf{u}^{\text{sec}}$ from a discrete uniform distribution over the set of DNS noises $\mathbf{N}$ (line 9), and randomly sample the mixing energy ratio $r$ from the uniform distribution $\mathcal{U}(-5,20)$ (line 10). The sample range of the mixing energy ratio follows the typical training utterance simulation process of speech separation task [50]. Then, we randomly select the mixing regions for both the utterances from the uniform distributions (line 12-14). The mixing length $l$ is uniformly sampled from $\{1,\cdots,\frac{L}{2}\}$ (line 12), and the start positions $s^{\text{pri}}$ and $s^{\text{sec}}$ for utterance $\mathbf{u}^{\text{pri}}$ and $\mathbf{u}^{\text{sec}}$ are both uniformly sampled from $\{1,\cdots,L-l\}$ (line 13 and 14). Note that as the mixing portion in each utterance is constrained to be less than 50%, the primary utterance would always be longer than the secondary utterance, avoiding the problem of the indistinguishable main speaker in the mixed speech signals. Next, given the mixing regions of the primary utterance $\mathbf{u}^{\text{pri}}[s^{\text{pri}}:s^{\text{pri}}+l]$ and the secondary utterance $\mathbf{u}^{\text{sec}}[s^{\text{sec}}:s^{\text{sec}}+l]$, we calculate the corresponding mixing scale of the secondary utterance $scl$ with the energy of the primary utterance $E^{\text{pri}}$, the secondary utterance $E^{\text{sec}}$ (line 15-17). Finally, we mix the selected region of the primary utterance $\mathbf{u}^{\text{pri}}[s^{\text{pri}}:s^{\text{pri}}+l]$ with the secondary utterance $\mathbf{u}^{\text{sec}}[s^{\text{sec}}:s^{\text{sec}}+l]$ scaled by the mixing scale $scl$ (line 18).

IV-B1 噪声/重叠语音仿真

噪声/重叠语音仿真方法细节见算法 1。给定 batch 大小为 $B$、长度为 $L$ 的一批语音语句 $\mathbf{U}=\{\mathbf{u}^{i}\}_{i=1}^{B}$，以及大小为 $M$ 的 DNS（Deep Noise Suppression）噪声集 [49] $\mathbf{N}=\{\mathbf{n}^{i}\}_{i=1}^{M}$（第 1 行），我们先以概率 $p$ 做伯努利采样，从 batch 中随机选取 $S$ 条待混合语句 $\mathbf{U}^{S}=\{\mathbf{u}^{i}\}_{i=1}^{S}$（第 2 行）。对每条 $\mathbf{u}^{\text{pri}}\in\mathbf{U}^{S}$（第 3 行），从连续均匀分布 $\mathcal{U}(0,1)$ 采样随机值 $v$，决定混噪声还是次要语句（第 4 行）。若 $v$ 大于混噪概率 $p_{n}$（第 5 行），则从 batch $\mathbf{U}$ 上离散均匀分布采样次要语句 $\mathbf{u}^{\text{sec}}$（第 6 行），并从 $\mathcal{U}(-5,5)$ 采样混合能量比 $r$（第 7 行）；否则从 DNS 噪声集 $\mathbf{N}$ 采样噪声 $\mathbf{u}^{\text{sec}}$（第 9 行），并从 $\mathcal{U}(-5,20)$ 采样 $r$（第 10 行）。混合能量比采样范围遵循语音分离任务的典型训练语句仿真流程 [50]。随后从均匀分布随机选择两条语句的混合区域（第 12–14 行）：混合长度 $l$ 从 $\{1,\cdots,\frac{L}{2}\}$ 均匀采样（第 12 行），主/次语句起始位置 $s^{\text{pri}}$、$s^{\text{sec}}$ 均从 $\{1,\cdots,L-l\}$ 均匀采样（第 13、14 行）。注意因每条语句混合比例约束小于 50%，主语句总长于次要语句，避免混合信号中主说话人难以区分。接着，给定主语句区域 $\mathbf{u}^{\text{pri}}[s^{\text{pri}}:s^{\text{pri}}+l]$ 与次要语句区域 $\mathbf{u}^{\text{sec}}[s^{\text{sec}}:s^{\text{sec}}+l]$，用主语句能量 $E^{\text{pri}}$ 与次要语句能量 $E^{\text{sec}}$ 计算次要语句混合尺度 $scl$（第 15–17 行）。最后，将所选主语句区域与按 $scl$ 缩放的次要语句区域混合（第 18 行）。

IV-B2 Mask Prediction Loss

Following HuBERT, we use the mask prediction loss to optimize our network. Suppose that we have an utterance $\mathbf{u}$ and its simulated version $\mathbf{u^{\prime}}$, we always generate pseudo-labels $\mathbf{z}$ by feeding $\mathbf{u}$ to the last iteration network. We follow HuBERT using the k-means clustering center on MFCC or latent representations as the pseudo-labels. The details will be introduced in Section V-A. Then, we obtain the hidden state $\mathbf{h}_{t}^{L}$ by feeding $\mathbf{u^{\prime}}$ to the current network, and optimize the objective function:

IV-B2 掩码预测损失

沿用 HuBERT，我们用掩码预测损失优化网络。设有语句 $\mathbf{u}$ 及其仿真版本 $\mathbf{u^{\prime}}$，伪标签 $\mathbf{z}$ 始终由将 $\mathbf{u}$ 送入上一轮网络得到。我们遵循 HuBERT，用 MFCC 或潜表征上的 k-means 聚类中心作为伪标签，细节见第 V-A 节。随后将 $\mathbf{u^{\prime}}$ 送入当前网络得到隐状态 $\mathbf{h}_{t}^{L}$，并优化目标：

$$\mathcal{L}=-\sum_{l\in K}\sum_{t\in M}{\rm log}p(z_{t}|\mathbf{h}_{t}^{L}) \tag{7}$$

where $M$ denotes the set of masked indices in time domain and $\mathbf{h}_{t}^{L}$ is the $L$-layer Transformer output for step $t$. Compared to previous methods, the framework is more beneficial to various non-ASR tasks, since it implicitly models the non-ASR information in pre-training.

其中 $M$ 为时域掩码索引集合，$\mathbf{h}_{t}^{L}$ 为第 $t$ 步的 $L$ 层 Transformer 输出。相对先前方法，该框架因在预训练中隐式建模非 ASR 信息，对多种非 ASR 任务更有利。

IV-C Pre-Training Data

IV-C 预训练数据

We leverage large-scale unsupervised data from diverse domains to improve the robustness of our model. Previous works use LibriSpeech [51] or LibriLight [9] datasets for pre-training, which limits the generalization capability of the pre-trained model since the input data are all extracted from the audiobook. The background acoustics of the speech obtained from the audiobook is different from what is observed in other conditions , since the real captured sounds are usually accompanied by various types of noise.

我们利用来自多样领域的大规模无监督数据以提升模型稳健性。先前工作使用 LibriSpeech [51] 或 LibriLight [9] 预训练，因输入皆来自有声书而限制了预训练模型的泛化能力。

Motivated by this, we extend the training data with two datasets: (1) 10k hours of the GigaSpeech data [16]. It is collected from audiobooks, podcasts and YouTube, covering both read and spontaneous speaking styles, and a variety of topics, such as arts, science, sports, etc. It should be noted that the total data size of GigaSpeech is 40k, but 30k of them are not well processed. For example, there is a large segment of silence at the beginning or at the end of some utterances in the 30k data. More seriously, some utterances just contain background noise without any speech. Thus, we just use the subset of 10k hours of GigaSpeech data, which is well processed and validated with a segmentation pipeline proposed in [16]. (2) VoxPopuli data [17]). It is a large-scale multi-lingual unlabeled audio dataset consisting of over 400k hours of audio in 23 languages, which is collected from 2009-2020 European Parliament (EP) event recordings including plenary sessions, committee meetings, and other events. Since our focus is English-only audio, we use 24k hours of English data in VoxPopuli for pre-training. In total, we collect 94k hours of data, including LibriLight, VoxPopuli, and GigaSpeech. We believe the enriched dataset can improve the model robustness as it contains diverse audio backgrounds, more speakers, and different contents. We call the dataset Mix 94k hr to make the description simple.

受此启发，我们用两套数据扩展训练：(1) 1 万小时 GigaSpeech [16]，采集自有声书、播客与 YouTube，覆盖朗读与自发说话风格，以及艺术、科学、体育等多样主题；需指出该数据含部分弱监督转录，但我们只用不带转录的音频。(2) 约 2.4 万小时 VoxPopuli [17] 英语数据，来自 2009–2020 年欧洲议会活动录音。连同 Libri-Light 的 6 万小时，总计约 9.4 万小时公开音频。相较仅有声书，新数据显著提升场景多样性，有助于缓解预训练与下游之间的声学失配。

IV-D Stabilization of Training

IV-D 训练稳定化

Currently, it is a common practice to use 16-bit float precision (fp16) or mixed precision to pre-train large models for faster computation and less GPU memory consumption. Unfortunately, the training is unstable for large models due to the overflow issue (characterized by NaN losses) [52]. A major reason is the attention score $\frac{\mathbf{q}_{i}\cdot\mathbf{k}_{j}}{\sqrt{d}}$ is larger than the upper bound value of the fp16, resulting in the overflow issue in training.

当前预训练大模型常用 16 位浮点（fp16）或混合精度，以求更快计算与更少 GPU 显存。但对大模型训练不稳定，会出现溢出（表现为 NaN 损失）[52]。一个主要原因是注意力 logits 在加相对位置偏置后数值过大。

We apply a simple trick to alleviate the overflow issue [52]. Given that softmax is invariant under translation by the same value in each coordinate i.e. $\text{softmax}(\mathbf{x}+\alpha)_{k}=\text{softmax}(\mathbf{x})_{k}$, where $\alpha$ denotes a constant number, the equation (3) can be implemented as

我们用一个简单技巧缓解溢出 [52]。鉴于 softmax 对每个坐标平移同一常数不变，即 $\text{softmax}(\mathbf{x}+\alpha)_{k}=\text{softmax}(\mathbf{x})_{k}$，公式 (3) 可实现为

$$\begin{split}\alpha_{i,j}&\propto\exp{\frac{\mathbf{q}{i}\cdot\mathbf{k}{j}}{\sqrt{d}}+r_{i-j}}\
&=\exp{(\frac{\mathbf{q}{i}}{c\sqrt{d}}\cdot\mathbf{k}{j}-\max_{j^{\prime}\leq T}(\frac{\mathbf{q}{i}}{c\sqrt{d}}\cdot\mathbf{k}{j^{\prime}}))\times c+r_{i-j}}.\end{split} \tag{8}$$

where $c$ is a scale hyperparameter and set to 32 in our work. In this way, the overflow issue could be solved, since $\max_{j^{\prime}\leq T}(\frac{\mathbf{q}_{i}}{c\sqrt{d}}\cdot\mathbf{k}_{j^{\prime}})$ could guarantee the max value is smaller than $2^{16}$.

其中 $c$ 为尺度超参数，本文设为 32。如此可解决溢出，因为 $\max_{j^{\prime}\leq T}(\frac{\mathbf{q}_{i}}{c\sqrt{d}}\cdot\mathbf{k}_{j^{\prime}})$ 能保证最大值小于 $2^{16}$。

V Experiment

V 实验

V-A Pre-Training Setup

V-A 预训练设置

The WavLM Base and WavLM Base+ have 12 Transformer encoder layers, 768-dimensional hidden states, and 8 attention heads, resulting in 94.70M parameters. The WavLM Large has 24 Transformer encoder layers, 1024-dimensional hidden states, and 12 attention heads, resulting in 316.62M parameters. The relative position embeddings are shared across all layers, which avoids significantly increasing the number of parameters. We pre-train the WavLM Base model for 400k steps on LibriSpeech 960 hours audio [51] using the label generated by clustering the 6-th transformer layer output of the 1st-iteration HuBERT Base model. The WavLM Base+ and WavLM Large models are pre-trained for 1M and 700k steps on 94k large-scale diverse data using the labels generated by clustering the 9th transformer layer output of the released 2nd-iteration HuBERT Base model [^3]. Even if the 1st iteration WavLM is better than HuBERT, we find using the 1st iteration pseudo-label of HuBERT results in a slightly better 2nd iteration WavLM (0.2 WER reduction on LibriSpeech dev-other). The masked speech denoising modeling is considered in 20$\%$ utterances, where the mixing noise probability $p_{n}$ is set to 0 for WavLM Base model, and 10$\%$ for WavLM Base+ and Large model. For other training configurations, the same hyperparameters are used following [6], which are shown in Appendix A.

WavLM Base 与 WavLM Base+ 有 12 层 Transformer 编码器、768 维隐状态、8 个注意力头，约 94.70M 参数；WavLM Large 有 24 层、1024 维隐状态、12 个注意力头，约 316.62M 参数。门控相对位置偏置的桶嵌入跨层共享。优化与掩码策略大体遵循 HuBERT [6]：Adam 优化器，峰值学习率与预热步数按模型规模设置（详见附录表 VII）。WavLM Base 在 LibriSpeech 960 小时上预训练；WavLM Base+ 与 Large 在 9.4 万小时混合语料上预训练，更新步数更长。目标标签采用迭代聚类：首轮对 MFCC 做 k-means，次轮对上一轮模型中间层表征再聚类。噪声/重叠仿真概率与 DNS 噪声库按第 IV-B 节配置，使模型在预训练阶段即接触多说话人与噪声条件。

Method	#Params	Corpus	Speaker	Content	Semantics	ParaL	Generation	Overall
SID	ASV	SD	PR	ASR	OOD-ASR	KS	QbE	ST	IC	SF	ER	SE	SS	VC
Acc $\uparrow$	EER $\downarrow$	DER $\downarrow$	PER $\downarrow$	WER $\downarrow$	WER $\downarrow$	Acc $\uparrow$	MTWV $\uparrow$	BLEU $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	CER $\downarrow$	Acc $\uparrow$	PESQ $\uparrow$	STOI $\uparrow$	SI-SDRi $\uparrow$	MCD $\downarrow$	WER $\downarrow$	ASV $\uparrow$	Score $\uparrow$
FBANK	0	-	8.5E-4	9.56	10.05	82.01	23.18	63.58	8.63	0.0058	2.32	9.10	69.64	52.94	35.39	2.55	93.6	9.23	8.47	38.3	77.25	43.2
PASE+ [44]	7.83M	LS 50 hr	37.99	11.61	8.68	58.87	25.11	61.56	82.54	0.0072	3.16	29.82	62.14	60.17	57.86	2.56	93.9	9.87	8.66	30.6	63.20	51.5
APC [30]	4.11M	LS 360 hr	60.42	8.56	10.53	41.98	21.28	63.12	91.01	0.0310	5.95	74.69	70.46	50.89	59.33	2.56	93.4	8.92	8.05	27.2	87.25	59.2
VQ-APC [29]	4.63M	LS 360 hr	60.15	8.72	10.45	41.08	21.20	63.56	91.11	0.0251	4.23	74.48	68.53	52.91	59.66	2.56	93.4	8.44	7.84	22.4	94.25	59.5
NPC [33]	19.38M	LS 360 hr	55.92	9.40	9.34	43.81	20.20	61.66	88.96	0.0246	4.32	69.44	72.79	48.44	59.08	2.52	93.1	8.04	7.86	30.4	94.75	59.0
Mockingjay [35]	85.12M	LS 360 hr	32.29	11.66	10.54	70.19	22.82	65.27	83.67	6.6E-04	4.45	34.33	61.59	58.89	50.28	2.53	93.4	9.29	8.29	35.1	79.75	51.0
TERA [34]	21.33M	LS 960 hr	57.57	15.89	9.96	49.17	18.17	58.49	89.48	0.0013	5.66	58.42	67.50	54.17	56.27	2.54	93.6	10.19	8.21	25.1	83.75	57.2
DeCoAR 2.0 [37]	89.84M	LS 960 hr	74.42	7.16	6.59	14.93	13.02	53.62	94.48	0.0406	9.94	90.80	83.28	34.73	62.47	2.47	93.2	8.54	7.83	17.1	90.75	66.3
modified CPC [53]	1.84M	LL 60k hr	39.63	12.86	10.38	42.54	20.18	62.54	91.88	0.0326	4.82	64.09	71.19	49.91	60.96	2.57	93.7	10.40	8.41	26.2	71.00	56.9
wav2vec [39]	32.54M	LS 960 hr	56.56	7.99	9.9	31.58	15.86	55.86	95.59	0.0485	6.61	84.92	76.37	43.71	59.79	2.53	93.8	9.30	7.45	10.1	98.25	63.5
vq-wav2vec [40]	34.15M	LS 960 hr	38.80	10.38	9.93	33.48	17.71	60.66	93.38	0.0410	5.66	85.68	77.68	41.54	58.24	2.48	93.6	8.16	7.08	13.4	100.00	61.8
wav2vec 2.0 Base [5]	95.04M	LS 960 hr	75.18	6.02	6.08	5.74	6.43	46.95	96.23	0.0233	14.81	92.35	88.30	24.77	63.43	2.55	93.9	9.77	7.50	10.5	98.00	69.6
HuBERT Base [6]	94.68M	LS 960 hr	81.42	5.11	5.88	5.41	6.42	46.69	96.30	0.0736	15.53	98.34	88.53	25.20	64.92	2.58	93.9	9.36	7.47	8.0	98.50	70.9
WavLM Base	94.70M	LS 960 hr	84.51	4.69	4.55	4.84	6.21	42.81	96.79	0.0870	20.74	98.63	89.38	22.86	65.94	2.58	94.0	10.37	7.42	8	98.00	72.0
- w/o denoising task	94.70M	LS 960 hr	84.39	4.91	6.03	4.85	6.08	43.61	96.79	0.0799	21.03	98.42	88.69	23.43	65.55	2.56	93.9	9.91	7.43	7.5	97.75	71.7
- w/o structure modification	94.68M	LS 960 hr	84.74	4.61	4.72	5.22	6.80	42.88	96.79	0.0956	20.03	98.31	88.56	24.00	65.60	2.58	94.0	10.29	7.45	8.4	99.00	71.9
WavLM Base+	94.70M	Mix 94k hr	89.42	4.07	3.50	3.92	5.59	38.32	97.37	0.0988	24.25	99.00	90.58	21.20	68.65	2.63	94.3	10.85	7.40	8.1	99.00	73.4
wav2vec 2.0 Large [5]	317.38M	LL 60k hr	86.14	5.65	5.62	4.75	3.75	44.69	96.66	0.0489	12.48	95.28	87.11	27.31	65.64	2.52	94.0	10.02	7.63	15.8	97.25	70.4
HuBERT Large [6]	316.61M	LL 60k hr	90.33	5.98	5.75	3.53	3.62	44.08	95.29	0.0353	20.01	98.76	89.81	21.76	67.62	2.64	94.2	10.45	7.22	9.0	99.25	72.2
WavLM Large	316.62M	Mix 94k hr	95.49	3.77	3.24	3.06	3.44	32.27	97.86	0.0886	26.57	99.31	92.21	18.36	70.62	2.70	94.5	11.19	7.30	9.0	99.00	74.6

TABLE I: Universal speech representation evaluation on SUPERB benchmark. ParaL denote Paralinguistics aspect of speech.

Method	#Params	Corpus	Speaker	Content	Semantics	ParaL	Generation	Overall
SID	ASV	SD	PR	ASR	OOD-ASR	KS	QbE	ST	IC	SF	ER	SE	SS	VC
Acc $\uparrow$	EER $\downarrow$	DER $\downarrow$	PER $\downarrow$	WER $\downarrow$	WER $\downarrow$	Acc $\uparrow$	MTWV $\uparrow$	BLEU $\uparrow$	Acc $\uparrow$	F1 $\uparrow$	CER $\downarrow$	Acc $\uparrow$	PESQ $\uparrow$	STOI $\uparrow$	SI-SDRi $\uparrow$	MCD $\downarrow$	WER $\downarrow$	ASV $\uparrow$	Score $\uparrow$
FBANK	0	-	8.5E-4	9.56	10.05	82.01	23.18	63.58	8.63	0.0058	2.32	9.10	69.64	52.94	35.39	2.55	93.6	9.23	8.47	38.3	77.25	43.2
PASE+ [44]	7.83M	LS 50 hr	37.99	11.61	8.68	58.87	25.11	61.56	82.54	0.0072	3.16	29.82	62.14	60.17	57.86	2.56	93.9	9.87	8.66	30.6	63.20	51.5
APC [30]	4.11M	LS 360 hr	60.42	8.56	10.53	41.98	21.28	63.12	91.01	0.0310	5.95	74.69	70.46	50.89	59.33	2.56	93.4	8.92	8.05	27.2	87.25	59.2
VQ-APC [29]	4.63M	LS 360 hr	60.15	8.72	10.45	41.08	21.20	63.56	91.11	0.0251	4.23	74.48	68.53	52.91	59.66	2.56	93.4	8.44	7.84	22.4	94.25	59.5
NPC [33]	19.38M	LS 360 hr	55.92	9.40	9.34	43.81	20.20	61.66	88.96	0.0246	4.32	69.44	72.79	48.44	59.08	2.52	93.1	8.04	7.86	30.4	94.75	59.0
Mockingjay [35]	85.12M	LS 360 hr	32.29	11.66	10.54	70.19	22.82	65.27	83.67	6.6E-04	4.45	34.33	61.59	58.89	50.28	2.53	93.4	9.29	8.29	35.1	79.75	51.0
TERA [34]	21.33M	LS 960 hr	57.57	15.89	9.96	49.17	18.17	58.49	89.48	0.0013	5.66	58.42	67.50	54.17	56.27	2.54	93.6	10.19	8.21	25.1	83.75	57.2
DeCoAR 2.0 [37]	89.84M	LS 960 hr	74.42	7.16	6.59	14.93	13.02	53.62	94.48	0.0406	9.94	90.80	83.28	34.73	62.47	2.47	93.2	8.54	7.83	17.1	90.75	66.3
modified CPC [53]	1.84M	LL 60k hr	39.63	12.86	10.38	42.54	20.18	62.54	91.88	0.0326	4.82	64.09	71.19	49.91	60.96	2.57	93.7	10.40	8.41	26.2	71.00	56.9
wav2vec [39]	32.54M	LS 960 hr	56.56	7.99	9.9	31.58	15.86	55.86	95.59	0.0485	6.61	84.92	76.37	43.71	59.79	2.53	93.8	9.30	7.45	10.1	98.25	63.5
vq-wav2vec [40]	34.15M	LS 960 hr	38.80	10.38	9.93	33.48	17.71	60.66	93.38	0.0410	5.66	85.68	77.68	41.54	58.24	2.48	93.6	8.16	7.08	13.4	100.00	61.8
wav2vec 2.0 Base [5]	95.04M	LS 960 hr	75.18	6.02	6.08	5.74	6.43	46.95	96.23	0.0233	14.81	92.35	88.30	24.77	63.43	2.55	93.9	9.77	7.50	10.5	98.00	69.6
HuBERT Base [6]	94.68M	LS 960 hr	81.42	5.11	5.88	5.41	6.42	46.69	96.30	0.0736	15.53	98.34	88.53	25.20	64.92	2.58	93.9	9.36	7.47	8.0	98.50	70.9
WavLM Base	94.70M	LS 960 hr	84.51	4.69	4.55	4.84	6.21	42.81	96.79	0.0870	20.74	98.63	89.38	22.86	65.94	2.58	94.0	10.37	7.42	8	98.00	72.0
- w/o denoising task	94.70M	LS 960 hr	84.39	4.91	6.03	4.85	6.08	43.61	96.79	0.0799	21.03	98.42	88.69	23.43	65.55	2.56	93.9	9.91	7.43	7.5	97.75	71.7
- w/o structure modification	94.68M	LS 960 hr	84.74	4.61	4.72	5.22	6.80	42.88	96.79	0.0956	20.03	98.31	88.56	24.00	65.60	2.58	94.0	10.29	7.45	8.4	99.00	71.9
WavLM Base+	94.70M	Mix 94k hr	89.42	4.07	3.50	3.92	5.59	38.32	97.37	0.0988	24.25	99.00	90.58	21.20	68.65	2.63	94.3	10.85	7.40	8.1	99.00	73.4
wav2vec 2.0 Large [5]	317.38M	LL 60k hr	86.14	5.65	5.62	4.75	3.75	44.69	96.66	0.0489	12.48	95.28	87.11	27.31	65.64	2.52	94.0	10.02	7.63	15.8	97.25	70.4
HuBERT Large [6]	316.61M	LL 60k hr	90.33	5.98	5.75	3.53	3.62	44.08	95.29	0.0353	20.01	98.76	89.81	21.76	67.62	2.64	94.2	10.45	7.22	9.0	99.25	72.2
WavLM Large	316.62M	Mix 94k hr	95.49	3.77	3.24	3.06	3.44	32.27	97.86	0.0886	26.57	99.31	92.21	18.36	70.62	2.70	94.5	11.19	7.30	9.0	99.00	74.6

表 I：SUPERB 基准上的通用表征评估。总体分数按官方加权汇总。

V-B Universal Representation Evaluation

V-B 通用表征评估

V-B1 Setup

We first evaluate our models on SUPERB, which is designed to provide a standard and comprehensive testbed for pre-trained models on various speech tasks. It covers fifteen tasks, including Speaker Identification (SID), Automatic Speaker Verification (ASV), Speaker Diarization (SD), Phoneme Recognition (PR), Automatic Speech Recognition (ASR), Out-Of-Domain Automatic Speech Recognition (OOD-ASR), Keyword Spotting (KS), Query by Example Spoken Term Detection (QbE), Speech Translation (ST), Intent Classification (IC), Slot Filling (SF), Emotion Recognition (ER), Speech Enhancement (SE), Speech Separation (SS) and Voice Conversion (VC). These tasks can be grouped into five aspects of speech: content, speaker, semantics, paralinguistics, and generation.

V-B1 设置

我们首先在 SUPERB 上评估模型——该基准旨在为预训练模型在多种语音任务上提供标准、全面的测试平台，覆盖十五项任务，包括说话人识别（SID）、自动说话人验证（ASV）、说话人日志（SD）、音素识别（PR）、ASR、OOD-ASR、关键词检出（KS）、Query-by-Example（QbE）、语音翻译（ST）、意图分类（IC）、槽填充（SF）、情感识别（ER）等。

We follow the settings created by SUPERB. 1) We use the same downstream models as the SUPERB implementations for each downstream task. 2) Pre-trained models are frozen to limit the space of the fine-tuning hyperparameter search. 3) The downstream models consume the weighted sum results of the hidden states extracted from each layer of the pre-trained model. The detailed hyperparameters for fine-tuning our WavLM models on SUPERB downstream tasks are shown in Appendix Hyperparamters for fine-tuning. The overall score is computed by ourselves: we multiply the QbE score with 100, replace each error rate score with (1 - error rate), and average the scores of all tasks.

我们遵循 SUPERB 设定：1）各下游任务使用与 SUPERB 实现相同的下游模型；2）冻结预训练模型，以限制微调超参搜索空间；3）下游模型消费各层隐状态的加权求和结果。

V-B2 Evaluation result

Table I shows the evaluation results. We compare our WavLM with several SSL models which are evaluated by [8, 18]. In general, WavLM is very powerful in universal representation learning. Our WavLM Base+ model has outperformed HuBERT large and wav2vec 2.0 large in the overall score.

V-B2 评估结果

表 I 给出评估结果。我们将 WavLM 与 [8, 18] 评估过的若干 SSL 模型比较。总体而言，WavLM 在通用表征学习上很强：WavLM Base+ 在总体分数上已超过 HuBERT Large 与 wav2vec 2.0 Large。

WavLM Base: From Table I, we can observe that WavLM Base performs better than wav2vec 2.0 Base and HuBERT Base on all downstream tasks. It is a fair comparison as the three models use the same amount of pre-training data and the number of parameters. The results indicate the effectiveness of our structure and the masked speech denoising modeling in universal speech representation learning. We find the most impressive result is speaker diarization, where the WavLM Base outperforms HuBERT Base by 22.6% relatively. Our explanation is that the additional overlapped speech forces the model to deal with multi-speaker signals during pre-training. To verify this assumption, we conduct an ablation study to remove simulated noisy/overlapped speech in pre-training. The performance of the “w/o denoising task” drops significantly for the speaker diarization task. We also evaluate the contribution of the structure change. We can see that, in the “w/o structure modification” setting, performance degradation can be witnessed especially for PR and ASR tasks. It indicates that the gated relative position bias contributes to the performance improvement of the content-related tasks. Meanwhile, we can observe that WavLM performs very well on semantic, paralinguistics, and generation tasks as well, demonstrating our model is general for the full stack speech processing tasks.

WavLM Base：由表 I 可见，WavLM Base 在全部下游任务上优于 wav2vec 2.0 Base 与 HuBERT Base。三者预训练数据量与参数量相同，比较公平；结果说明我们的改动有效。

WavLM Base+: WavLM Base+ shows the contribution from larger and more diverse pre-training data. It consistently improves WavLM Base and even outperforms the wav2vec 2.0 Large and HuBERT Large in the overall score. This indicates that the 960h data are insufficient to fulfill the capacity of the Base model. The combined dataset especially boosts the performance of the testsets which are not extracted from the audiobook, such as ASV, OOD-ASR, IC, SF, and ER.

WavLM Base+： WavLM Base+ 体现更大、更多样预训练数据的贡献：相对 WavLM Base 持续提升，甚至在总体分数上超过 wav2vec 2.0 Large 与 HuBERT Large，表明 960h 数据不足以填满 Base 级模型容量。

WavLM Large: Most tasks benefit from the larger model size, especially for the ASR. We obtain 38$\%$ word error rate reduction on the ASR by model scaling-up. Furthermore, there is 6.07$\%$ absolute improvement on the SID task, indicating the large model size also impacts the speaker-related tasks. Compared to the HuBERT Large model, WavLM Large is consistently better across 14 downstream tasks, demonstrating the modifications are effective for the large-scale models.

WavLM Large：多数任务受益于更大模型，尤其 ASR：模型放大带来约 38% 相对词错误率下降。此外，相对 HuBERT Large 总体分数有约 6.07% 相对提升（绝对约 2.4 分），并在 14 项子任务上更优。

(a) HuBERT Base

V-B3 Analysis

Following the SUPERB policies, we weighted-sum the hidden states of different layers and feed it to the task-specific layers. Figure 2 shows the weights of different layers of HuBERT and WavLM models on the different downstream tasks of the SUPERB benchmark. The larger weight indicates the greater contribution of the corresponding layer. We normalize the weights from different layers based on the hidden state values of their corresponding layers, which eliminates the weight bias to layers with smaller hidden state values.

V-B3 分析

遵循 SUPERB 策略，我们将不同层隐状态加权求和后送入任务特定层。图 2 展示 HuBERT 与 WavLM 在 SUPERB 上各层权重。第 0 层对应输入特征。

As for the Base models, the contribution patterns of different layers are similar between WavLM and HuBERT, as shown in Figure 2a and 2b. We can observe that the bottom layers contribute more to speaker-related tasks, such as speaker identification, automatic speaker verification, and speaker diarization. On the other hand, for automatic speech recognition, phoneme recognition, intent classification, and slot filling tasks, the top layers are more important. It indicates the Base models learn speaker information with the bottom layers while the content and semantic information are encoded in the top layers. The model behavior is similar to Large models. In Figures 2c and 2d, we can see that the top layers contribute most to content and semantic tasks, while the middle layers have a great impact on speaker tasks. The phenomenon indicates how to leverage hidden states of middle layers is the key to the success of speaker-related tasks.

对 Base 模型，WavLM 与 HuBERT 的各层贡献模式相似，见图 2a 与 2b。可观察到底层对说话人相关任务贡献更大，顶层对内容相关任务贡献更大。WavLM 因去噪预训练，在说话人与分离相关任务上对中下层的利用更充分。

Since SUPERB requires the pre-trained model frozen in fine-tuning, it cannot show the power of pre-trained models. To explore the limit of our models, we further select typical speech tasks to evaluate our pre-trained model performance. Four tasks are used to evaluate our model from different perspectives, and the training data amount is not on the same scale for the four tasks. The details of the tasks can be found in Appendix Settings of downstream tasks.

由于 SUPERB 要求微调时冻结预训练模型，它无法充分展示预训练模型的潜力。为此，我们在说话人验证、说话人日志、语音分离与语音识别四项任务上进一步评估：允许按任务微调，并在各任务代表性基准上报告结果。

V-C Speaker Verification

V-C 说话人验证

V-C1 problem formulation

The training dataset for speaker verification contains audio and speaker id pairs as $\mathcal{D}=\{\mathbf{x_{i}},\mathbf{y_{i}}\}$. Given audio clip $\mathbf{x}$ and a reference $\mathbf{x}^{\prime}$, the goal of speaker verification is to determine whether $\mathbf{x}^{\prime}$ is from the same speaker as $\mathbf{x}$.

V-C1 问题表述

说话人验证的训练数据包含音频与说话人标签。给定注册语句与测试语句，系统判断二者是否来自同一说话人。

V-C2 Datasets

VoxCeleb1 [20] and VoxCeleb2 [54] datasets are used in our experiments for speaker verification. For data pre-processing, we apply online data augmentation using the MUSAN [55] noise, DNS noise [49] and the RIR [^4] reverberation with probability 0.6. Voice activity detection (VAD) processing is not adopted. We use all three official trial lists Vox1-O, Vox1-E, and Vox1-H to evaluate the system.

V-C2 数据集

实验使用 VoxCeleb1 [20] 与 VoxCeleb2 [54]。我们在 VoxCeleb2 开发集上训练，并在 VoxCeleb1 的三条官方 trial 列表（Vox1-O、Vox1-E、Vox1-H）上评估。

V-C3 Setup

We choose the ECAPA-TDNN (small) [19] architecture as the downstream model and compare different input speech representations, including handcrafted features and the pre-training features. The model contains a frame encoder to extract speaker information from the input sequence, a statistic pooling layer to transform input to a fixed-dimensional representation, and a fully connected layer to extract speaker embedding. For the handcrafted feature, we compare the reported results in [19] with our re-implemented results, where we extract the 40-dimensional Fbank feature with 25ms window size and 10ms frameshift. For pre-trained representations, we compare WavLM with HuBERT model. Following SUPERB evaluation, we weighted-sum the representations from different transformer layers with learnable weights as the input to the downstream speaker verification task.

V-C3 设置

下游模型选用 ECAPA-TDNN（small）[19]，并与仅用 filter-bank 特征训练的同结构模型比较。预训练表征按 SUPERB 做法对各层做可学习加权求和，再送入 ECAPA-TDNN。我们比较冻结与解冻预训练参数两种设定。

In the training stage, all the recordings are chunked into 3s segments to construct the training batches. We use the additive angular margin (AAM) loss [56] for model optimization and set the margin to 0.2. We also add an Inter-TopK penalty [57] on the 5 easily misclassified centers with a penalty margin of 0.1. We train the ECAPA-TDNN system with Fbank feature for 165 epochs. For systems using pre-trained representations, we first fix the pre-trained model to train ECAPA-TDNN for 20 epochs and then finetune both the pre-trained and ECAPA-TDNN models for another 5 epochs. When we add the large margin fine-tuning strategy [58], we train the systems for an extra 2 epochs, during which we sample 6s training segments and set the AAM margin to 0.4.

训练阶段，所有录音切成 3 秒片段构成训练 batch。优化器与数据增强大体遵循 ECAPA-TDNN 原设定；对解冻设定使用更小学习率更新预训练骨干。

In the evaluation stage, the whole utterance is fed into the system to extract speaker embedding. We use cosine similarity to score the evaluation trial list. We also use the adaptive s-norm [59, 60] to normalize the trial scores. The imposter cohort is estimated from the VoxCeleb2 dev set by speaker-wise averaging all the extracted speaker embeddings. We set the imposter cohort size to 600 in our experiment. To further push the performance, we also introduce the quality-aware score calibration [58] for our best systems, where we randomly generate 30k trials based on the VoxCeleb2 test set to train the calibration model.

评估阶段，整句送入系统提取说话人嵌入。我们用余弦相似度打分，并报告三条 trial 列表上的等错误率（EER）。

Feature	EER (%)
Vox1-O	Vox1-E	Vox1-H
ECAPA-TDNN [19]	1.010	1.240	2.320
ECAPA-TDNN (Ours)	1.080	1.200	2.127
HuBERT Base	0.989	1.068	2.216
HuBERT Large	0.808	0.822	1.678
WavLM Base+	0.84	0.928	1.758
WavLM Large	0.617	0.662	1.318
HuBERT Large∗	0.585	0.654	1.342
WavLM Large∗	0.383	0.480	0.986

TABLE II: Speaker verification results on VoxCeleb1. For the lines with ∗ notation, we add the large margin fine-tuning and quality-aware score calibration [58] to push the limit of the performance.

Feature	EER (%)
Vox1-O	Vox1-E	Vox1-H
ECAPA-TDNN [19]	1.010	1.240	2.320
ECAPA-TDNN (Ours)	1.080	1.200	2.127
HuBERT Base	0.989	1.068	2.216
HuBERT Large	0.808	0.822	1.678
WavLM Base+	0.84	0.928	1.758
WavLM Large	0.617	0.662	1.318
HuBERT Large∗	0.585	0.654	1.342
WavLM Large∗	0.383	0.480	0.986

表 II：VoxCeleb1 说话人验证结果（EER %）。

Method # of speakers in a session 2 3 4 5 6 all x-vector clustering [61] 15.45 18.01 22.68 31.40 34.27 19.43 SC-EEND [62] ‡ 9.57 14.00 21.14 31.07 37.06 15.75 VBx [63] † ‡ 9.44 13.89 16.05 13.87 24.73 13.28 EEND-EDA [64] 8.50 13.24 21.46 33.16 40.29 15.29 EEND-EDA clustering [24] 7.11 11.88 14.37 25.95 21.95 11.84 EEND-vector clustering [65] 7.96 11.93 16.38 21.21 23.10 12.49 EEND-vector clustering (Ours) 7.54 12.42 18.41 26.79 27.40 13.31 HuBERT Base & EEND-vector clustering 7.93 12.07 15.21 19.59 23.32 12.63 HuBERT Large & EEND-vector clustering 7.39 11.97 15.76 19.82 22.10 12.40 WavLM Base+ & EEND-vector clustering 6.99 11.12 15.20 21.61 21.70 11.78 WavLM Large & EEND-vector clustering 6.46 10.69 11.84 12.89 20.70 10.35 • † Oracle speech segments were used. • ‡ Results for these systems are provided in [65].

Method # of speakers in a session 2 3 4 5 6 all x-vector clustering [61] 15.45 18.01 22.68 31.40 34.27 19.43 SC-EEND [62] ‡ 9.57 14.00 21.14 31.07 37.06 15.75 VBx [63] † ‡ 9.44 13.89 16.05 13.87 24.73 13.28 EEND-EDA [64] 8.50 13.24 21.46 33.16 40.29 15.29 EEND-EDA clustering [24] 7.11 11.88 14.37 25.95 21.95 11.84 EEND-vector clustering [65] 7.96 11.93 16.38 21.21 23.10 12.49 EEND-vector clustering (Ours) 7.54 12.42 18.41 26.79 27.40 13.31 HuBERT Base & EEND-vector clustering 7.93 12.07 15.21 19.59 23.32 12.63 HuBERT Large & EEND-vector clustering 7.39 11.97 15.76 19.82 22.10 12.40 WavLM Base+ & EEND-vector clustering 6.99 11.12 15.20 21.61 21.70 11.78 WavLM Large & EEND-vector clustering 6.46 10.69 11.84 12.89 20.70 10.35 • † Oracle speech segments were used. • ‡ Results for these systems are provided in [65].

† Oracle speech segments were used.

‡ Results for these systems are provided in [65].

TABLE III: Diarization error rate (DER %) results on CALLHOME with estimated number of speakers.

Method # of speakers in a session 2 3 4 5 6 all x-vector clustering [61] 15.45 18.01 22.68 31.40 34.27 19.43 SC-EEND [62] ‡ 9.57 14.00 21.14 31.07 37.06 15.75 VBx [63] † ‡ 9.44 13.89 16.05 13.87 24.73 13.28 EEND-EDA [64] 8.50 13.24 21.46 33.16 40.29 15.29 EEND-EDA clustering [24] 7.11 11.88 14.37 25.95 21.95 11.84 EEND-vector clustering [65] 7.96 11.93 16.38 21.21 23.10 12.49 EEND-vector clustering (Ours) 7.54 12.42 18.41 26.79 27.40 13.31 HuBERT Base & EEND-vector clustering 7.93 12.07 15.21 19.59 23.32 12.63 HuBERT Large & EEND-vector clustering 7.39 11.97 15.76 19.82 22.10 12.40 WavLM Base+ & EEND-vector clustering 6.99 11.12 15.20 21.61 21.70 11.78 WavLM Large & EEND-vector clustering 6.46 10.69 11.84 12.89 20.70 10.35 • † Oracle speech segments were used. • ‡ Results for these systems are provided in [65].

Method # of speakers in a session 2 3 4 5 6 all x-vector clustering [61] 15.45 18.01 22.68 31.40 34.27 19.43 SC-EEND [62] ‡ 9.57 14.00 21.14 31.07 37.06 15.75 VBx [63] † ‡ 9.44 13.89 16.05 13.87 24.73 13.28 EEND-EDA [64] 8.50 13.24 21.46 33.16 40.29 15.29 EEND-EDA clustering [24] 7.11 11.88 14.37 25.95 21.95 11.84 EEND-vector clustering [65] 7.96 11.93 16.38 21.21 23.10 12.49 EEND-vector clustering (Ours) 7.54 12.42 18.41 26.79 27.40 13.31 HuBERT Base & EEND-vector clustering 7.93 12.07 15.21 19.59 23.32 12.63 HuBERT Large & EEND-vector clustering 7.39 11.97 15.76 19.82 22.10 12.40 WavLM Base+ & EEND-vector clustering 6.99 11.12 15.20 21.61 21.70 11.78 WavLM Large & EEND-vector clustering 6.46 10.69 11.84 12.89 20.70 10.35 • † Oracle speech segments were used. • ‡ Results for these systems are provided in [65].

† Oracle speech segments were used.

‡ Results for these systems are provided in [65].

表 III：CALLHOME 说话人日志结果（日志错误率）。

V-C4 Results

Table II shows the results for the speaker verification task. From the results, we find that all the systems with pre-trained representations exceed the Fbank baseline system on the Vox1-O and Vox1-E trials. The system with HuBERT Base representations is slightly worse than the Fbank feature on the Vox1-H trial. Interestingly, the representations extracted from our proposed pre-trained models, WavLM Base+ and Large, both outperform the SOTA ECAPA-TDNN system. Compared with the Fbank feature, the representations from WavLM Large achieve over 35% relative EER improvement on all three trials for the VoxCeleb1 evaluation set. To further push the limit of the speaker verification system, we introduce the large margin fine-tuning and quality-aware score calibration strategies [58] into our best systems and the corresponding results are listed at the bottom of Table II. With these two strategies, our best system exceeds the winner system [57] (Vox1-O: 0.461, Vox1-E: 0.634, Vox1-H: 0.993) in VoxSRC challenge 2021[^5] on all the three trials.

V-C4 结果

表 II 给出说话人验证结果。WavLM Large 在三条 trial 上均大幅超过 ECAPA-TDNN 基线，达到 0.383%、0.480%、0.986% EER。即便 Base+ 也显著优于仅用手工特征的系统，说明大规模去噪预训练学到了更强的说话人判别表征。解冻预训练参数通常还能再抬一点，但冻结设定已很强。

V-D Speaker Diarization

V-D 说话人日志

V-D1 problem formulation

Speaker diarization is the task to answer “Who spoke when?”. Given a speech recording $\mathbf{x}=(x_{1},...,x_{T})$, we should assign one or more labels to each $x_{t}$ according to the speaker identity. When we assign more than one label to $x_{t}$, it indicates more than one person is speaking at time $t$, i.e. speaker overlap. Normally, we cannot know the number of speakers of a whole recording in advance. Thus, the built diarization system should have the ability to predict the speaker number for the whole recording and the speaker labels for each frame at the same time.

V-D1 问题表述

说话人日志要回答「谁在何时说话」。给定可能含重叠语音的长录音，系统输出各说话人的时间活动。

V-D2 Datasets

The dataset used in our experiments is split into two parts. The first part is the large-scale simulation training data. The second part is the real data, which is used for evaluation and adaptation. Following the data simulation setup in [65], all the speech data from Switchboard-2 (Phase I & II & III), Switchboard Cellular (Part 1 & 2), the NIST Speaker Recognition Evaluation (2004 & 2005 & 2006 & 2008), the noises from [55] and the simulated room impulse responses used in [66] are leveraged for multi-talker speech simulation. Based on the simulation pipeline introduced in [67], we generate almost 7000 hours simulation data by setting $N_{spk}=3$ and $\beta=10$. We use the telephone conversation dataset CALLHOME [68] for evaluation and adaptation. CALLHOME dataset has 500 sessions of multilingual telephonic speech where each session contains 2 to 6 speakers. Following the data usage in [65], we split the CALLHOME dataset into two parts. The first part is used for adaptation and the second part is used for evaluation.

V-D2 数据集

实验数据分两部分：第一部分为仿真混合数据用于训练；第二部分为 CALLHOME 电话对话用于适应与评估。划分与协议遵循先前 EEND 相关工作。

V-D3 Implementation Details

We leverage the system in [65] as our downstream speaker diarization model. In the system, a long-form recording is first segmented into short blocks, where each short block is assumed to contain at most $S_{Local}$ speakers. As with [65], we set $S_{Local}=3$ in our experiment. Then, the Mel-filterbank-based features extracted from each short block are fed into a Transformer encoder to get the diarization results and $S_{Local}$ speaker embeddings. With the predicted diarization results and estimated speaker embeddings, the whole system is trained by a diarization loss and a speaker loss. During the inference, the diarization results and speaker embeddings are first predicted for each block. A clustering method is then applied to associate the embeddings from the same speaker but in different blocks.

V-D3 实现细节

下游说话人日志系统采用 [65] 的架构。预训练 WavLM 提供帧级表征，再接 Transformer 类日志头。训练时对仿真多说话人混合优化，再在 CALLHOME 上适应。

Following the implementation in [65], we set the block length to 15s, 30s, 30s for the training, adaptation, and evaluation stage respectively. The constrained AHC (Agglomerative Hierarchical Clustering) method is used for embedding clustering during the evaluation stage. When leveraging the pre-trained representations, as with the implementation in section V-C3, we just replace the handcrafted Fbank feature with the pre-trained representation $\mathbf{H}$. Unlike [65], when we feed the diarization system with the pre-trained representations, we do not concatenate the context features for each frame and do not apply the 10 times down-sampling. We find that updating the parameters of the pre-trained model does not improve performance on the CALLHOME dataset. Thus, we freeze the pre-trained model in the fine-tuning stage. One possible explanation is that the test data are real recordings while the training data are simulated recordings, and the model is over-fitted if the pre-trained model is not frozen.

沿用 [65] 的实现，训练、适应与评估的块长分别设为 15s、30s、30s。其余优化超参与基线系统对齐，以便公平比较表征质量。

V-D4 Results

The speaker diarization results on CALLHOME dataset are shown in Table III. In our experiment, we try to reproduce the system in [65] but get slightly worse results. When we replace the handcrafted feature with pre-trained representations, all the systems exceed the performance of our implemented EEND-vector clustering. Compared with the HuBERT, the representations extracted from our proposed WavLM are more useful in speaker diarization. Our proposed WavLM Base+ even outperforms the HuBERT large model. This is because the WavLM models have seen the multi-talker and speaker-overlapped speeches during the training process, and the corresponding training strategy is designed to help WavLM better process this kind of input. Finally, we also list the CALLHOME results from some recently published works. Compared with these results, it is worth noting that our best system has surpassed all the systems evaluated on the CALLHOME dataset and achieved a new SOTA performance.

V-D4 结果

CALLHOME 上的说话人日志结果见表 III。相对 EEND-EDA 聚类等方法，WavLM 显著降低日志错误率，相对降幅约 12.6%。多说话人仿真去噪预训练对重叠场景尤为关键。

System	Overlap ratio in %
0S	0L	10	20	30	40	avg
Conformer [22]	5.4	5.0	7.5	10.7	13.8	17.1	10.6
Conformer (rerun)	4.5	4.4	6.2	8.5	11.0	12.6	8.3
HuBERT Base	4.7	4.6	6.1	7.9	10.6	12.3	8.1
WavLM Base+	4.5	4.4	5.6	7.5	9.4	10.9	7.4
- unfreeze pre-trained parameters	4.5	4.3	5.9	8.3	11.1	12.5	8.2
WavLM Large	4.2	4.1	4.8	5.8	7.4	8.5	6.0

TABLE IV: Separation results on LibriCSS dataset. We freeze the pre-trained parameters by default for the separation task. The results denote %WER score evaluated with E2E Transformer based ASR model [69]. 0S and 0L are utterances with short/long inter-utterance silence. The avg is the weighted averaged WER of different overlapped testsets.

System	Overlap ratio in %
0S	0L	10	20	30	40	avg
Conformer [22]	5.4	5.0	7.5	10.7	13.8	17.1	10.6
Conformer (rerun)	4.5	4.4	6.2	8.5	11.0	12.6	8.3
HuBERT Base	4.7	4.6	6.1	7.9	10.6	12.3	8.1
WavLM Base+	4.5	4.4	5.6	7.5	9.4	10.9	7.4
- unfreeze pre-trained parameters	4.5	4.3	5.9	8.3	11.1	12.5	8.2
WavLM Large	4.2	4.1	4.8	5.8	7.4	8.5	6.0

表 IV：LibriCSS 单通道按语句分离结果（不同重叠比例下的 WER）。

V-E Speech Separation

V-E 语音分离

V-E1 problem formulation

The goal of speech separation is to estimate individual speaker signals from their mixture, where the source signals may be overlapped with each other entirely or partially. Given $S$ source signals $\{\mathbf{x}_{s}=(x_{1},...,x_{T})\}_{s=1}^{S}$, the mixed signal is formulated as $\mathbf{y}=\sum_{s=1}^{S}\mathbf{x}_{s}$. $\mathbf{X}_{s}$ and $\mathbf{Y}$ denote the Short-Time Fourier Transform (STFT) of the source signal and mixed signal, respectively. Following [70] and [71], instead of directly predicting the source STFTs, we firstly estimate a group of masks $\{\mathbf{M}_{s}\}_{s=1}^{S}$ with a deep learning model, and then obtain each source STFT with $\mathbf{X}_{s}=\mathbf{M}_{s}\odot\mathbf{Y}$, where $\odot$ is an elementwise product.

V-E1 问题表述

语音分离目标是从混合中估计各说话人信号，源信号可完全或部分重叠。给定 $S$ 路源信号 $\{\mathbf{x}_{s}=(x_{1},...,x_{T})\}_{s=1}^{S}$，混合信号由它们叠加（可含混响）形成，模型需恢复各源。

V-E2 Datasets

Our training dataset for the separation task consists of 219 hours of artificially reverberated and mixed utterances that are sampled randomly from WSJ1 [72]. Four different mixture types described in [73] are included in the training set. To generate each training mixture, we randomly pick one or two speakers from WSJ1 and convolve each with a room impulse response (RIR) simulated with the image method [74]. The reverberated signals are then rescaled and mixed with a source energy ratio between -5 and 5 dB. In addition, we add simulated isotropic noise [75] with a 0–10 dB signal to noise ratio. The average overlap ratio of the training set is around 50%.

V-E2 数据集

分离任务训练集含 219 小时人工混响与混合语句，随机采样自 WSJ1 [72]。训练集包含 [73] 所述四种混合类型。生成每条训练混合时，我们随机选取源、房间冲激响应与混合配置。

LibriCSS is used for evaluation [21]. The dataset has 10 hours of seven-channel recordings of mixed and concatenated LibriSpeech test utterances. The recordings were made by playing back the mixed audio in a meeting room. We apply the single-channel utterance-wise evaluation schemes of LibriCSS, where the long-form recordings are segmented into individual utterances by using ground-truth time marks to evaluate the pure separation performance.

评估使用 LibriCSS [21]：含 10 小时七通道录音，由在会议室回放混合并拼接的 LibriSpeech 测试语句构成。我们采用 LibriCSS 的单通道按语句评估方案。

V-E3 Implementation details

For the separation task, we use the previous SOTA work [22] as our baseline model, which uses the Conformer-based model for separation, and consists of 16 Conformer encoder layers with 4 attention heads, 256 attention dimensions, and 1024 FFN dimensions. A linear projection layer and sigmoid activation function are attached to the final encoder for the mask prediction. Given the STFT of mixed signal $\mathbf{Y}$ as the input, the separation model estimates masks $\{\mathbf{M}_{s}\}_{s=1}^{S}$, then each source signal can be obtained as $\{\mathbf{X}_{s}=\mathbf{M}_{s}\odot\mathbf{Y}\}_{s=1}^{S}$ for each speaker.

V-E3 实现细节

分离任务以先前 SOTA [22] 为基线：基于 Conformer 的分离模型，含 16 层 Conformer 编码器、4 个注意力头、256 维注意力与 1024 维 FFN。线性投影将特征映射到分离网络输入维。

To fine-tune our pre-trained models on the separation, we use WavLM models as feature extractors and the Conformer-base architecture as the task-specific downstream model. We begin by extracting the pre-trained representation $\mathbf{H}$ as introduced in section V-C3. Secondly, we concatenate the pre-trained representation and STFT representation in the feature dimension. Since the window size and hop length of STFT are typically set to 400 and 160, respectively, the STFT representation $\mathbf{Y}=\{\mathbf{Y}_{t}\}_{t=1}^{T^{\prime}}$ has the half stride size compared to the pre-trained representation $\mathbf{H}=\{\mathbf{H}_{t}\}_{t=1}^{T^{\prime}/2}$. To match the size of the time dimension, we duplicate the pre-trained representation with $\hat{\mathbf{H}}=\{\hat{\mathbf{H}}_{t}=\mathbf{H}_{\lceil\frac{t}{2}\rceil}\}_{t=1}^{T^{\prime}}$, then we can concatenate the two representations $[\mathbf{Y}_{t},\hat{\mathbf{H}}_{t}]$ in the feature dimension for each time step. Finally, we feed the concatenated representations to the downstream model for the mask estimation.

微调时，WavLM 作特征提取器，Conformer-base 作任务特定下游模型。先按第 V-C3 节提取预训练表征 $\mathbf{H}$，再与下游输入拼接或替换，送入分离网络。我们比较冻结与解冻 WavLM 参数。

The separation models are trained with the AdamW optimizer [76], where the weight decay is set to 1e-2. We set the learning rate to 1e-4 and use a warm-up learning schedule with a linear decay, in which the number of the warm-up steps is 10,000 and the total number of the training step is 260,000.

分离模型用 AdamW [76] 训练，权重衰减 1e-2；学习率 1e-4，带线性衰减的预热调度，预热 10,000 步，总训练 260,000 步。

We follow the previous work [22, 77, 78, 50] to evaluate our model with an end-to-end Transformer based ASR models [69], which achieves 2.08% and 4.95% word error rates (WERs) for LibriSpeech test-clean and test-other, respectively.

评估沿用先前工作 [22, 77, 78, 50]，使用端到端 Transformer ASR 模型 [69]：在 LibriSpeech test-clean / test-other 上 WER 分别为 2.08% 与 4.95%。

V-E4 Results

Table IV shows the single-channel utterance-wise separation results on LibriCSS dataset. Our WavLM Base+ and Large models with the frozen pre-trained parameters achieve SOTA results on all the overlap ratio settings, outperforming the baseline results by a large margin.

V-E4 结果

表 IV 给出 LibriCSS 单通道按语句分离结果。冻结预训练参数的 WavLM Base+ 与 Large 在全部重叠比例设定上达到 SOTA，大幅超过基线。

We rerun the previous SOTA work [22] with a modified Conformer-base architecture [79] and a modified training loss [50], which achieve much better baseline results. With the pre-trained representation provided by the HuBERT Base model, the performance is comparable with the baseline results for all the overlap ratios. It is because the HuBERT model is rarely optimized with speaker-overlapped speech and lacks multi-speaker modeling during pre-training.

我们用改动的 Conformer-base 结构 [79] 与改动的训练损失 [50] 重跑先前 SOTA [22]，得到强得多的基线。仅用 HuBERT Base 提供的预训练表征时，各重叠比例表现与该基线相当。

In contrast, our WavLM Base+ with a similar model size can successfully reduce the WER scores, especially for the large overlap ratio audios. We find fine-tuning the parameters of the pre-trained model yields better training accuracy but worse evaluation results than freezing the pre-trained parameters for the separation task. An explanation is that the separation model with pre-trained parameters adaptation would be over-fitted with the artificially mixed training data, and it is evaluated with a real meeting recording dataset. With the pre-trained representation provided by our WavLM Large model, the performance on all the overlap ratio settings can be further improved. It can achieve 32.5% relative WER score reduction for the 40% overlap ratio cases and 27.7% relative WER score reduction on average.

相比之下，相近规模的 WavLM Base+ 能成功降低 WER，尤其在大重叠比例音频上。我们发现：解冻预训练参数会提高训练精度，但评估结果往往不如冻结预训练参数——对分离任务，强而稳的冻结表征更关键。WavLM Large 进一步刷新结果，相对强基线相对 WER 降幅可达约 27.7%。

V-E5 Weight Analysis

For the speaker verification (Section V-C), speech diarization (Section V-D) and speech separation (Section V-E) tasks, we weighted-sum the representations from different layers of the pre-trained models as the input to the task-specific downstream models. Figure 3 shows the weights of different layers of WavLM Base+ and WavLM Large models on these tasks. As with the weight analysis on the SUPERB benchmark in Section V-B3, we can observe that the contribution mostly comes from the bottom layers for all these tasks. It indicates that the shallow layers of WavLM models learn the speaker-related information during the SSL procedure. It is essential to leverage hidden states of intermediate layers for speaker-related tasks to make full use of the pre-trained knowledge of WavLM models.

V-E5 权重分析

对说话人验证（第 V-C 节）、说话人日志（第 V-D 节）与语音分离（第 V-E 节），我们将预训练模型不同层表征加权求和作为下游输入。图 3 展示各任务上学到的层权重分布：说话人相关任务更依赖中下层，分离任务对能区分重叠说话人的层赋予更高权重。

(a) WavLM Base+

Model	Unlabled Data	LM	test-clean	test-other
1-hour labeled
wav2vec 2.0 Base	LS-960	None	24.5	29.7
WavLM Base	LS-960	None	24.5	29.2
WavLM Base+	MIX-94k	None	22.8	26.7
DeCoAR 2.0	LS-960	4-gram	13.8	29.1
DiscreteBERT	LS-960	4-gram	9.0	17.6
wav2vec 2.0 Base	LS-960	4-gram	5.5	11.3
HuBERT Base	LS-960	4-gram	6.1	11.3
WavLM Base	LS-960	4-gram	5.7	10.8
WavLM Base+	MIX-94k	4-gram	5.4	9.8
wav2vec 2.0 Large	LL-60k	4-gram	3.8	7.1
WavLM Large	MIX-94k	4-gram	3.8	6.6
wav2vec2.0 Large	LL-60k	Transformer	2.9	5.8
HuBERT Large	LL-60k	Transformer	2.9	5.4
WavLM Large	MIX-94k	Transformer	2.9	5.1
10-hour labeled
wav2vec 2.0	LS-960	None	11.1	17.6
WavLM Base	LS-960	None	9.8	16.0
WavLM Base+	MIX-94k	None	9.0	14.7
DeCoAR 2.0	LS-960	4-gram	5.4	13.3
DiscreteBERT	LS-960	4-gram	5.9	14.1
wav2vec 2.0	LS-960	4-gram	4.3	9.5
HuBERT Base	LS-960	4-gram	4.3	9.4
WavLM Base	LS-960	4-gram	4.3	9.2
WavLM Base+	MIX-94k	4-gram	4.2	8.8
wav2vec 2.0 Large	LL-60k	4-gram	3.0	5.8
WavLM Large	MIX-94k	4-gram	2.9	5.5
wav2vec 2.0 Large	LL-60k	Transformer	2.6	4.9
HuBERT Large	LL-60k	Transformer	2.4	4.6
WavLM Large	MIX-94k	Transformer	2.4	4.6
100-hour labeled
wav2vec 2.0 Base	LS-960	None	6.1	13.3
WavLM Base	LS-960	None	5.7	12.0
WavLM Base+	MIX-94k	None	4.6	10.1
DeCoAR 2.0	LS-960	4-gram	5.0	12.1
DiscreteBERT	LS-960	4-gram	4.5	12.1
wav2vec 2.0 Base	LS-960	4-gram	3.4	8.0
HuBERT Base	LS-960	4-gram	3.4	8.1
WavLM Base	LS-960	4-gram	3.4	7.7
WavLM Base+	MIX-94k	4-gram	2.9	6.8
wav2vec 2.0 Large	LL-60k	4-gram	2.3	4.6
WavLM Large	MIX-94k	4-gram	2.3	4.6
wav2vec 2.0 Large	LL-60k	Transformer	2.0	4.0
HuBERT Large	LL-60k	Transformer	2.1	3.9
WavLM Large	MIX-94k	Transformer	2.1	4.0

TABLE V: WER on LibriSpeech test sets when trained on the Libri-Light low-resource labeled data setups of 1 hour, 10 hours and the clean 100h subset of LibriSpeech.

Model	Unlabled Data	LM	test-clean	test-other
1-hour labeled
wav2vec 2.0 Base	LS-960	None	24.5	29.7
WavLM Base	LS-960	None	24.5	29.2
WavLM Base+	MIX-94k	None	22.8	26.7
DeCoAR 2.0	LS-960	4-gram	13.8	29.1
DiscreteBERT	LS-960	4-gram	9.0	17.6
wav2vec 2.0 Base	LS-960	4-gram	5.5	11.3
HuBERT Base	LS-960	4-gram	6.1	11.3
WavLM Base	LS-960	4-gram	5.7	10.8
WavLM Base+	MIX-94k	4-gram	5.4	9.8
wav2vec 2.0 Large	LL-60k	4-gram	3.8	7.1
WavLM Large	MIX-94k	4-gram	3.8	6.6
wav2vec2.0 Large	LL-60k	Transformer	2.9	5.8
HuBERT Large	LL-60k	Transformer	2.9	5.4
WavLM Large	MIX-94k	Transformer	2.9	5.1
10-hour labeled
wav2vec 2.0	LS-960	None	11.1	17.6
WavLM Base	LS-960	None	9.8	16.0
WavLM Base+	MIX-94k	None	9.0	14.7
DeCoAR 2.0	LS-960	4-gram	5.4	13.3
DiscreteBERT	LS-960	4-gram	5.9	14.1
wav2vec 2.0	LS-960	4-gram	4.3	9.5
HuBERT Base	LS-960	4-gram	4.3	9.4
WavLM Base	LS-960	4-gram	4.3	9.2
WavLM Base+	MIX-94k	4-gram	4.2	8.8
wav2vec 2.0 Large	LL-60k	4-gram	3.0	5.8
WavLM Large	MIX-94k	4-gram	2.9	5.5
wav2vec 2.0 Large	LL-60k	Transformer	2.6	4.9
HuBERT Large	LL-60k	Transformer	2.4	4.6
WavLM Large	MIX-94k	Transformer	2.4	4.6
100-hour labeled
wav2vec 2.0 Base	LS-960	None	6.1	13.3
WavLM Base	LS-960	None	5.7	12.0
WavLM Base+	MIX-94k	None	4.6	10.1
DeCoAR 2.0	LS-960	4-gram	5.0	12.1
DiscreteBERT	LS-960	4-gram	4.5	12.1
wav2vec 2.0 Base	LS-960	4-gram	3.4	8.0
HuBERT Base	LS-960	4-gram	3.4	8.1
WavLM Base	LS-960	4-gram	3.4	7.7
WavLM Base+	MIX-94k	4-gram	2.9	6.8
wav2vec 2.0 Large	LL-60k	4-gram	2.3	4.6
WavLM Large	MIX-94k	4-gram	2.3	4.6
wav2vec 2.0 Large	LL-60k	Transformer	2.0	4.0
HuBERT Large	LL-60k	Transformer	2.1	3.9
WavLM Large	MIX-94k	Transformer	2.1	4.0

表 V：低资源 ASR 微调结果（1h/10h/100h 标注）。

Model	Unlabled Data	LM	test-clean	test-other
Supervised
CTC Transf [80]	-	CLM+Transf.	2.5	5.5
S2S Transf. [80]	-	CLM+Transf.	2.3	5.2
Transf. Transducer [81]	-	Transf.	2.0	4.6
ContextNet [82]	-	LSTM	1.9	4.1
Conformer Transducer [83]	-	LSTM	1.9	3.9
Pre-training
wav2vec 2.0 Large	LL-60k	Transformer	1.8	3.3
HuBERT Large	LL-60k	Transformer	1.9	3.3
WavLM Large	MIX-94k	Transformer	1.8	3.2

TABLE VI: WER on LibriSpeech when using all 960 hours of labeled data.

Model	Unlabled Data	LM	test-clean	test-other
Supervised
CTC Transf [80]	-	CLM+Transf.	2.5	5.5
S2S Transf. [80]	-	CLM+Transf.	2.3	5.2
Transf. Transducer [81]	-	Transf.	2.0	4.6
ContextNet [82]	-	LSTM	1.9	4.1
Conformer Transducer [83]	-	LSTM	1.9	3.9
Pre-training
wav2vec 2.0 Large	LL-60k	Transformer	1.8	3.3
HuBERT Large	LL-60k	Transformer	1.9	3.3
WavLM Large	MIX-94k	Transformer	1.8	3.2

表 VI：LibriSpeech 960h 全资源 ASR 结果。

V-F Speech Recognition

V-F 语音识别

V-F1 Problem Formulation

Given the input speech signal $\mathbf{x}=(x_{1},...,x_{T})$, the goal of speech recognition is to generate the corresponding transcription $\mathbf{y}=(y_{1},...,y_{L})$, where $T$ and $L$ are the lengths of the speech and transcription, respectively.

V-F1 问题表述

给定输入语音信号 $\mathbf{x}=(x_{1},...,x_{T})$，语音识别目标是生成对应转写 $\mathbf{y}=(y_{1},...,y_{L})$，其中 $T$、$L$ 分别为语音与转写长度。

V-F2 Datasets

We use LibriSpeech for our ASR experiments. For the fine-tuning, we consider four different partitions: 960 hours of transcribed LibriSpeech [51], the train-clean-100 subset (100 hours labeled data), as well as the Libri-Light [9] limited resource training subsets originally extracted from LibriSpeech, including train-10h (10 hours labeled data) and train-1h (1 hour labeled data). We follow the evaluation protocol of Libri-Light for these splits and evaluate on the standard LibriSpeech test-clean/other sets.

V-F2 数据集

ASR 实验使用 LibriSpeech。微调考虑四个划分：960 小时带转写 LibriSpeech [51]、train-clean-100 子集（100 小时标注），以及原先从 Libri-Light [9] 提取的有限资源训练子集（1 小时与 10 小时）。

V-F3 Implementation Details

The pre-trained models are fine-tuned for speech recognition by adding a randomly initialized linear projection layer on top of the Transformer encoder. Models are optimized based on a CTC loss [84] where we have 29 tokens for character targets plus a word boundary token. We apply a modified version of SpecAugment [85] by masking time-steps and channels: we randomly select the starting positions with a predetermined probability and replace a span of ten subsequent time-steps with a mask embedding; different spans may overlap and we use the same masked time step embedding as the one used for pre-training. We also mask channels by choosing a number of channels as starting indices and then expanding to the subsequent 64 channels. Spans may overlap and the selected spans are set to zeros.

V-F3 实现细节

在 Transformer 编码器顶部加随机初始化的线性投影层，对预训练模型做语音识别微调。基于 CTC 损失 [84] 优化，字符目标共 29 个 token，外加词边界标记。

During fine-tuning, the convolutional encoder is always fixed and we freeze the Transformer encoder for the first 10k steps. We optimize with Adam and a tri-stage rate schedule where the learning rate is warmed up for the first 10% of the updates, held constant for the next 40%, and then linearly decayed for the remainder. The Base and Base+ models are fine-tuned on 8 GPUs with a batch size equivalent to 200 seconds of audio for each GPU. The Large model is fine-tuned on 24 GPUs with a batch size equivalent to 80 seconds of audio for each GPU. We also use LayerDrop [86, 87] at a rate of 0.05 for Base/Base+ and 0.1 for LARGE. The summary of the fine-tuning hyperparameter settings used for different labeled data setups can be found in Appendix Hyperparamters for fine-tuning.

微调时卷积编码器始终固定；前 10k 步冻结 Transformer 编码器。用 Adam 与三阶段学习率：前 10% 更新预热，接着 40% 保持，再线性衰减。

For evaluation, we use wav2letter++ [88] beam search decoder with language model (LM) fused decoding as :

评估使用 wav2letter++ [88] 束搜索解码器，并与语言模型（LM）融合解码：

$${\rm log}p_{CTC}(\mathbf{y}|\mathbf{x})+w_{1}{\rm log}p_{LM}(\mathbf{y})+w_{2}|\mathbf{y}| \tag{9}$$

where $w_{1}$ is the language model weight and $w_{2}$ is the word insertion weight. We consider a 4-gram model and a Transformer model, which are identical to [5]. The evaluation hyperparameters are also based on [5].

其中 $w_{1}$ 为语言模型权重，$w_{2}$ 为词插入权重。我们考虑与 [5] 相同的 4-gram 与 Transformer 语言模型；评估超参也基于 [5]。

V-F4 Results

Table V presents the results for the low-resource setup, where the pre-trained models are fine-tuned on the 1 hour, 10 hours or 100 hours of labeled data. We compare our method with several competitive self-supervised approaches in the literature, including DeCoAR 2.0 [37], DiscreteBERT [41], wav2vec 2.0 [5] and HuBERT [6]. Without LM fusion, the WavLM Base model outperforms wav2vec 2.0 by a large margin for all fine-tuning splits, indicating the superiority of our model architecture. Its performance matches or outperforms wav2vec 2.0 and HuBERT with LM. WavLM Base+ improves WavLM Base, especially on the test-other set, indicating increasing the out-of-domain unlabeled data also works for ASR. For the Large model, the observation is consistent that our method achieves comparable or better performance than the baselines. Table VI reports results on the full 960 hours of LibriSpeech data. Overall, the pre-training methods can outperform all supervised models and our model is on par with the two best pre-training results in this setting.

V-F4 结果

表 V 给出低资源设定结果：预训练模型在 1、10 或 100 小时标注数据上微调。我们与文献中若干有竞争力的自监督方法比较，包括 DeCoAR 2.0 [37]、DiscreteBERT、wav2vec 2.0、HuBERT 等。WavLM 与 wav2vec 2.0 / HuBERT 同档或更好；Base+ 因更多样数据在低资源上更稳健。表 VI 显示 960h 全资源设定下，WavLM 在 test-clean / test-other 上达到约 1.8% / 3.2% WER，与顶尖 SSL ASR 系统相当。

VI Conclusion

VI 结论

We present WavLM, a large-scale pre-trained model with 94k hour audio as inputs, to solve full stack speech processing tasks. WavLM extends the HuBERT framework to masked speech prediction and denoising modeling, enabling the pre-trained models to perform well on both ASR and non-ASR tasks. WavLM updates state-of-the-art results on the SUPERB, as well as the representative testsets of speaker verification, speech separation, and speaker diarization. In contrast to previous SSL models, WavLM is not only effective for the ASR task but also has the potential to become the next-generation backbone network for speaker-related tasks.

我们提出 WavLM：以 9.4 万小时音频为输入的大规模预训练模型，用于解决全栈语音处理任务。WavLM 将 HuBERT 框架扩展为掩码语音预测与去噪建模，使预训练模型在 ASR 与非 ASR 任务上均表现出色。WavLM 更新模型结构（门控相对位置偏置）并扩展预训练数据多样性。在 SUPERB 与说话人验证、日志、分离、识别等代表性基准上，WavLM 达到或刷新当时最优。

In the future, we would like to scale up the model size to increase the model capability, as previous work has shown the benefits of more parameters [47]. Meanwhile, the model compression technique is also worth trying due to the time constraint and limited test time resources in real scenarios. It is also a promising direction to jointly learn text and speech representation in a self-supervised pre-training framework [89], as the huge amount of text data might increase the capability of speech content modeling.

未来我们希望放大模型规模以增强能力，先前工作已显示更多参数的好处 [47]。同时，因真实场景的时间约束与有限测试资源，模型压缩也值得尝试。把 WavLM 式去噪预训练与多语言、多任务微调结合，是另一自然方向。

Appendix A Hyperparamters for pre-training

附录 A 预训练超参数

Table VII shows the hyperparameters used for pre-training our WavLM Base, Base+, and Large model, which are adapted from the previous work [6].

表 VII 给出 WavLM Base、Base+ 与 Large 预训练所用超参数，改编自先前工作 [6]。

Model	pre-train data	update steps	learning rate	warmup steps	batch size
WavLM Base	960h	400k	5e-4	32k	350s
WavLM Base+	94kh	1.2M	5e-4	96k	350s
WavLM Large	94kh	700k	1.5e-3	32k	720s

TABLE VII: Hyperparamters for pre-training WavLM models . The unit in batch size computing is second. We use 32 V100 GPUs for base model training, and 64 V100 GPUs for large model.

Model	pre-train data	update steps	learning rate	warmup steps	batch size
WavLM Base	960h	400k	5e-4	32k	350s
WavLM Base+	94kh	1.2M	5e-4	96k	350s
WavLM Large	94kh	700k	1.5e-3	32k	720s

表 VII：预训练超参数。

Settings of downstream tasks

下游任务设定

For the universal representation evaluation, we use the same settings for all the SUPERB tasks in accordance with the SUPERB policies [8].

通用表征评估中，所有 SUPERB 任务设定与 SUPERB 策略一致 [8]。

As for the four additional downstream tasks, including speaker verification, speaker diarization, speech separation, and speech recognition, the implementations are shown in Table VIII, following the previous works [19, 65, 22, 6].

对说话人验证、说话人日志、语音分离与语音识别四项额外下游任务，实现见表 VIII，遵循先前工作 [19, 65, 22, 6]。

Task	dataset	training data duration	downstream model
Speaker Verification	VoxCeleb	2300h	ECAPA-TDNN
Speaker Diarization	CALLHOME	8.7h	Transformer
Speech Separation	LibriCSS	219h	Conformer
Speech Recognition	LibriSpeech	1h/10h/100h/960h	Linear

TABLE VIII: Different settings of the downstream tasks. In the speaker diarization task, it should be noted that the CALLHOME dataset is used for domain adaptation.

Task	dataset	training data duration	downstream model
Speaker Verification	VoxCeleb	2300h	ECAPA-TDNN
Speaker Diarization	CALLHOME	8.7h	Transformer
Speech Separation	LibriCSS	219h	Conformer
Speech Recognition	LibriSpeech	1h/10h/100h/960h	Linear

表 VIII：额外下游任务设定。

Hyperparamters for fine-tuning

微调超参数

As for the universal representation evaluation, Table IX shows the hyperparameters of the learning rate and batch size for fine-tuning our WavLM models in the SUPERB downstream tasks. For the QbE task, which is evaluated by dynamic time warping without fine-tuning, we find that the best results for all the three WavLM models are always from the representations of the last layer. All the other hyperparameters of each downstream task are exactly the same as the official implementation of SUPERB[^6].

通用表征评估方面，表 IX 给出 SUPERB 下游任务微调 WavLM 的学习率与 batch 大小。对不经微调、用动态时间规整评估的 QbE 任务，我们发现 WavLM 的最佳结果来自对中间层表征的特定加权。

As for the speech recognition task fine-tuning, Table X summarizes the hyperparameters used for different labeled data setups.

语音识别任务微调方面，表 X 汇总不同标注数据设定下的超参数。

Task	WavLM Base	WavLM Base+	WavLM Large
learning rate	batch size	learning rate	batch size	learning rate	batch size
Speaker Identification	2e-1	512	1e-1	512	5e-2	512
Automatic Speaker Verification	5e-5	512	5e-5	512	5e-5	512
Speaker Diarization	2e-3	256	5e-4	256	5e-3	256
Phoneme Recognition	5e-4	128	5e-4	128	2e-4	128
Automatic Speech Recognition	5e-4	128	5e-4	128	1e-4	128
Out-of-domain Automatic Speech Recognition	1e-4	16	1e-4	16	1e-4	16
Keyword Spotting	1e-5	512	1e-5	512	1e-5	512
Speech Translation	1e-3	80k	1e-3	80k	1e-3	160k
Intent Classification	5e-5	128	2e-5	128	5e-4	128
Slot Filling	2e-4	128	2e-4	128	1e-4	128
Emotion Recognition	1e-4	32	1e-4	32	1e-5	32
Speech Enhancement	5e-4	64	5e-4	64	5e-4	64
Speech Separation	5e-4	64	1e-3	64	5e-4	64
Voice Conversion	1e-4	6	1e-4	6	1e-4	6

TABLE IX: Hyperparamenters of fine-tuning WavLM models in SUPERB downstream tasks. The batch size of Speech Translation task denotes the number of tokens in each training batch.

Task	WavLM Base	WavLM Base+	WavLM Large
learning rate	batch size	learning rate	batch size	learning rate	batch size
Speaker Identification	2e-1	512	1e-1	512	5e-2	512
Automatic Speaker Verification	5e-5	512	5e-5	512	5e-5	512
Speaker Diarization	2e-3	256	5e-4	256	5e-3	256
Phoneme Recognition	5e-4	128	5e-4	128	2e-4	128
Automatic Speech Recognition	5e-4	128	5e-4	128	1e-4	128
Out-of-domain Automatic Speech Recognition	1e-4	16	1e-4	16	1e-4	16
Keyword Spotting	1e-5	512	1e-5	512	1e-5	512
Speech Translation	1e-3	80k	1e-3	80k	1e-3	160k
Intent Classification	5e-5	128	2e-5	128	5e-4	128
Slot Filling	2e-4	128	2e-4	128	1e-4	128
Emotion Recognition	1e-4	32	1e-4	32	1e-5	32
Speech Enhancement	5e-4	64	5e-4	64	5e-4	64
Speech Separation	5e-4	64	1e-3	64	5e-4	64
Voice Conversion	1e-4	6	1e-4	6	1e-4	6

表 IX：SUPERB 下游任务微调超参数。

Setup	updates	learning rate	timestep mask prob.	channel mask prob.
1 hour (Base/Base+)	13k	5e-5	0.065	0.004
10 hour (Base/Base+)	25k	2e-5	0.075	0.008
100 hour (Base/Base+)	80k	3e-5	0.065	0.008
1 hour (Large)	13k	3e-4	0.075	0.004
10 hour (Large)	20k	1e-4	0.075	0.004
100 hour (Large)	80k	3e-5	0.005	0.008
960 hour (Large)	320k	3e-5	0.005	0.004

TABLE X: Hyperparamenters of fine-tuning WavLM models in speech recognition task.

Setup	updates	learning rate	timestep mask prob.	channel mask prob.
1 hour (Base/Base+)	13k	5e-5	0.065	0.004
10 hour (Base/Base+)	25k	2e-5	0.075	0.008
100 hour (Base/Base+)	80k	3e-5	0.065	0.008
1 hour (Large)	13k	3e-4	0.075	0.004
10 hour (Large)	20k	1e-4	0.075	0.004
100 hour (Large)	80k	3e-5	0.005	0.008
960 hour (Large)	320k	3e-5	0.005	0.004

表 X：ASR 微调超参数。

References

参考文献

[1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in North American Chapter of the Association for Computational Linguistics (NAACL) , 2019, pp. 4171–4186.

[2] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” in Advances in Neural Information Processing Systems (NeurIPS) , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019.

[3] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research (JMLR) , vol. 21, no. 140, pp. 1–67, 2020.

[4] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” CoRR , vol. abs/1807.03748, 2018.

[5] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Advances in Neural Information Processing Systems (NeurIPS) , 2020.

[6] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” arXiv preprint arXiv:2106.07447 , 2021.

[7] C. Wang, Y. Wu, Y. Qian, K. Kumatani, S. Liu, F. Wei, M. Zeng, and X. Huang, “Unispeech: Unified speech representation learning with labeled and unlabeled data,” in International Conference on Machine Learning(ICML) , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 2021, pp. 10 937–10 947. [Online]. Available: http://proceedings.mlr.press/v139/wang21y.html

[8] S. Yang, P. Chi, Y. Chuang, C. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G. Lin, T. Huang, W. Tseng, K. Lee, D. Liu, Z. Huang, S. Dong, S. Li, S. Watanabe, A. Mohamed, and H. Lee, “SUPERB: Speech Processing Universal PERformance Benchmark,” in Interspeech , 2021, pp. 1194–1198.

[9] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen et al. , “Libri-light: A benchmark for asr with limited or no supervision,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7669–7673.

[10] W. Chan, D. Park, C. Lee, Y. Zhang, Q. Le, and M. Norouzi, “Speechstew: Simply mix all available speech recognition data to train one large neural network,” arXiv preprint arXiv:2104.02133 , 2021.

[11] T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve, “Rethinking evaluation in asr: Are our models robust enough?” arXiv preprint arXiv:2010.11745 , 2020.

[12] A. Narayanan, A. Misra, K. C. Sim, G. Pundak, A. Tripathi, M. Elfeky, P. Haghani, T. Strohman, and M. Bacchiani, “Toward domain-invariant speech recognition via large scale training,” in 2018 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 2018, pp. 441–447.

[13] A. Narayanan, R. Prabhavalkar, C.-C. Chiu, D. Rybach, T. N. Sainath, and T. Strohman, “Recognizing long-form speech using streaming end-to-end models,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) . IEEE, 2019, pp. 920–927.

[14] W.-N. Hsu, A. Sriram, A. Baevski, T. Likhomanenko, Q. Xu, V. Pratap, J. Kahn, A. Lee, R. Collobert, G. Synnaeve et al. , “Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training,” arXiv preprint arXiv:2104.01027 , 2021.

[15] Z. Chi, S. Huang, L. Dong, S. Ma, S. Singhal, P. Bajaj, X. Song, and F. Wei, “XLM-E: cross-lingual language model pre-training via ELECTRA,” CoRR , vol. abs/2106.16138, 2021. [Online]. Available: https://arxiv.org/abs/2106.16138

[16] G. Chen, S. Chai, G. Wang, J. Du, W. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang, M. Jin, S. Khudanpur, S. Watanabe, S. Zhao, W. Zou, X. Li, X. Yao, Y. Wang, Y. Wang, Z. You, and Z. Yan, “Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio,” in Interspeech , 2021.

[17] C. Wang, M. Rivière, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” arXiv preprint arXiv:2101.00390 , 2021.

[18] H.-S. Tsai, H.-J. Chang, W.-C. Huang, Z. Huang, K. Lakhotia, S.-w. Yang, S. Dong, A. T. Liu, C.-I. J. Lai, J. Shi et al. , “Superb-sg: Enhanced speech processing universal performance benchmark for semantic and generative capabilities,” arXiv preprint arXiv:2203.06849 , 2022.

[19] B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” arXiv preprint arXiv:2005.07143 , 2020.

[20] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” arXiv preprint arXiv:1706.08612 , 2017.

[21] Z. Chen, T. Yoshioka, L. Lu, T. Zhou, Z. Meng, Y. Luo, J. Wu, X. Xiao, and J. Li, “Continuous speech separation: Dataset and analysis,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7284–7288.

[22] S. Chen, Y. Wu, Z. Chen, J. Wu, J. Li, T. Yoshioka, C. Wang, S. Liu, and M. Zhou, “Continuous speech separation with conformer,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 5749–5753.

[23] T. J. Park, N. Kanda, D. Dimitriadis, K. J. Han, S. Watanabe, and S. Narayanan, “A review of speaker diarization: Recent advances with deep learning,” arXiv preprint arXiv:2101.09624 , 2021.

[24] S. Horiguchi, S. Watanabe, P. Garcia, Y. Xue, Y. Takashima, and Y. Kawaguchi, “Towards neural diarization for unlimited numbers of speakers using global and local attractors,” arXiv preprint arXiv:2107.01545 , 2021.

[25] Y.-C. Chen, S.-F. Huang, H.-y. Lee, Y.-H. Wang, and C.-H. Shen, “Audio word2vec: Sequence-to-sequence autoencoding for unsupervised learning of audio segmentation and representation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) , vol. 27, no. 9, pp. 1481–1493, 2019.

[26] W.-N. Hsu, Y. Zhang, and J. Glass, “Learning latent representations for speech generation and transformation,” in Interspeech , 2017, pp. 1273–1277.

[27] ——, “Unsupervised learning of disentangled and interpretable representations from sequential data,” in Advances in Neural Information Processing Systems (NeurIPS) , 2017.

[28] J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, “Unsupervised speech representation learning using wavenet autoencoders,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) , vol. 27, no. 12, pp. 2041–2053, 2019.

[29] Y.-A. Chung, H. Tang, and J. Glass, “Vector-quantized autoregressive predictive coding,” in Interspeech , 2020, pp. 3760–3764.

[30] Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, “An Unsupervised Autoregressive Model for Speech Representation Learning,” in Interspeech , 2019, pp. 146–150.

[31] Y. Chung and J. Glass, “Generative pre-training for speech with autoregressive predictive coding,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 3497–3501.

[32] Y.-A. Chung and J. Glass, “Improved speech representations with multi-target autoregressive predictive coding,” in Association for Computational Linguistics (ACL) . Online: Association for Computational Linguistics, Jul. 2020, pp. 2353–2358.

[33] A. H. Liu, Y.-A. Chung, and J. Glass, “Non-autoregressive predictive coding for learning speech representations from local dependencies,” arXiv preprint arXiv:2011.00406 , 2020.

[34] A. T. Liu, S.-W. Li, and H.-y. Lee, “Tera: Self-supervised learning of transformer encoder representation for speech,” arXiv preprint arXiv:2007.06028 , 2020.

[35] A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020.

[36] S. Ling, Y. Liu, J. Salazar, and K. Kirchhoff, “Deep contextualized acoustic representations for semi-supervised speech recognition,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 6429–6433.

[37] S. Ling and Y. Liu, “DeCoAR 2.0: Deep contextualized acoustic representations with vector quantization,” arXiv preprint arXiv:2012.06659 , 2020.

[38] Y. Gong, C.-I. J. Lai, Y.-A. Chung, and J. Glass, “Ssast: Self-supervised audio spectrogram transformer,” arXiv preprint arXiv:2110.09784 , 2021.

[39] S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition.” in Interspeech , 2019.

[40] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” in International Conference on Learning Representations (ICLR) , 2020.

[41] A. Baevski and A. Mohamed, “Effectiveness of self-supervised pre-training for asr,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7694–7698.

[42] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” arXiv preprint arXiv:2108.06209 , 2021.

[43] S. Pascual, M. Ravanelli, J. Serrà, A. Bonafonte, and Y. Bengio, “Learning problem-agnostic speech representations from multiple self-supervised tasks,” in Interspeech , 2019, pp. 161–165.

[44] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio, “Multi-task self-supervised learning for robust speech recognition,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 6989–6993.

[45] J. Bai, B. Li, Y. Zhang, A. Bapna, N. Siddhartha, K. C. Sim, and T. N. Sainath, “Joint unsupervised and supervised training for multilingual asr,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 6402–6406.

[46] S. Chen, Y. Wu, C. Wang, Z. Chen, Z. Chen, S. Liu, J. Wu, Y. Qian, F. Wei, J. Li et al. , “Unispeech-sat: Universal speech representation learning with speaker aware pre-training,” arXiv preprint arXiv:2110.05752 , 2021.

[47] Y. Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y. Xu, Y. Huang, S. Wang et al. , “Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,” arXiv preprint arXiv:2109.13226 , 2021.

[48] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS) , 2017, pp. 5998–6008.

[49] C. K. Reddy, H. Dubey, K. Koishida, A. Nair, V. Gopal, R. Cutler, S. Braun, H. Gamper, R. Aichner, and S. Srinivasan, “Interspeech 2021 deep noise suppression challenge,” arXiv preprint arXiv:2101.01902 , 2021.

[50] J. Wu, Z. Chen, S. Chen, Y. Wu, T. Yoshioka, N. Kanda, S. Liu, and J. Li, “Investigation of Practical Aspects of Single Channel Speech Separation for ASR,” in Interspeech , 2021, pp. 3066–3070.

[51] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2015, pp. 5206–5210.

[52] M. Ding, Z. Yang, W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang et al. , “Cogview: Mastering text-to-image generation via transformers,” arXiv preprint arXiv:2105.13290 , 2021.

[53] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7414–7418.

[54] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” arXiv preprint arXiv:1806.05622 , 2018.

[55] D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” arXiv preprint arXiv:1510.08484 , 2015.

[56] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2019, pp. 4690–4699.

[57] M. Zhao, Y. Ma, M. Liu, and M. Xu, “The speakin system for voxceleb speaker recognition challange 2021,” arXiv preprint arXiv:2109.01989 , 2021.

[58] J. Thienpondt, B. Desplanques, and K. Demuynck, “The idlab voxsrc-20 submission: Large margin fine-tuning and quality-aware score calibration in dnn based speaker verification,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 5814–5818.

[59] Z. N. Karam, W. M. Campbell, and N. Dehak, “Towards reduced false-alarms using cohorts,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2011, pp. 4512–4515.

[60] S. Cumani, P. D. Batzu, D. Colibro, C. Vair, P. Laface, and V. Vasilakakis, “Comparison of speaker recognition approaches for real applications.” in Interspeech , 2011, pp. 2365–2368.

[61] S. Horiguchi, Y. Fujita, S. Watanabe, Y. Xue, and K. Nagamatsu, “End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors,” arXiv preprint arXiv:2005.09921 , 2020.

[62] Y. Fujita, S. Watanabe, S. Horiguchi, Y. Xue, J. Shi, and K. Nagamatsu, “Neural speaker diarization with speaker-wise chain rule,” arXiv preprint arXiv:2006.01796 , 2020.

[63] F. Landini, J. Profant, M. Diez, and L. Burget, “Bayesian hmm clustering of x-vector sequences (vbx) in speaker diarization: theory, implementation and analysis on standard tasks,” Computer Speech & Language , vol. 71, p. 101254, 2022.

[64] S. Horiguchi, Y. Fujita, S. Watanabe, Y. Xue, and P. Garcia, “Encoder-decoder based attractor calculation for end-to-end neural diarization,” arXiv preprint arXiv:2106.10654 , 2021.

[65] K. Kinoshita, M. Delcroix, and N. Tawara, “Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech,” arXiv preprint arXiv:2105.09040 , 2021.

[66] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2017, pp. 5220–5224.

[67] Y. Fujita, N. Kanda, S. Horiguchi, K. Nagamatsu, and S. Watanabe, “End-to-end neural speaker diarization with permutation-free objectives,” in Interspeech , 2019, pp. 4300–4304.

[68] M. Przybocki and A. Martin, “2000 nist speaker recognition evaluation (ldc2001s97),” in Philadelphia, New Jersey: Linguistic Data Consortium , 2001.

[69] C. Wang, Y. Wu, Y. Du, J. Li, S. Liu, L. Lu, S. Ren, G. Ye, S. Zhao, and M. Zhou, “Semantic mask for transformer based end-to-end speech recognition,” Interspeech , 2020.

[70] Y. Wang, A. Narayanan, and D. Wang, “On training targets for supervised speech separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP) , vol. 22, no. 12, pp. 1849–1858, 2014.

[71] H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux, “Deep recurrent networks for separation and recognition of single-channel speech in nonstationary background audio,” in New Era for Robust Speech Recognition . Springer, 2017, pp. 165–186.

[72] L. D. C. Philadelphia, “CSR-II (WSJ1) Complete,” 1994, http://catalog.ldc.upenn.edu/LDC94S13A .

[73] T. Yoshioka, H. Erdogan, Z. Chen, and F. Alleva, “Multi-microphone neural speech separation for far-field multi-talker speech recognition,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2018, pp. 5739–5743.

[74] J. Allen and D. Berkley, “Image method for efficiently simulating small-room acoustics,” The Journal of the Acoustical Society of America (JASA) , vol. 65, pp. 943–950, 1979.

[75] E. A. Habets and S. Gannot, “Generating sensor signals in isotropic noise fields,” The Journal of the Acoustical Society of America (JASA) , vol. 122, no. 6, pp. 3464–3470, 2007.

[76] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations (ICLR) , 2018.

[77] S. Chen, Y. Wu, Z. Chen, T. Yoshioka, S. Liu, J. Li, and X. Yu, “Don’t shoot butterfly with rifles: Multi-channel continuous speech separation with early exit transformer,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 6139–6143.

[78] S. Chen, Y. Wu, Z. Chen, J. Wu, T. Yoshioka, S. Liu, J. Li, and X. Yu, “Ultra Fast Speech Separation Model with Teacher Student Learning,” in Interspeech , 2021, pp. 3026–3030.

[79] G. Ye, V. Mazalov, J. Li, and Y. Gong, “Have best of both worlds: two-pass hybrid and e2e cascading framework for speech recognition,” arXiv preprint arXiv:2110.04891 , 2021.

[80] G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert, “End-to-end asr: from supervised to semi-supervised learning with modern architectures,” arXiv preprint arXiv:1911.08460 , 2019.

[81] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7829–7833.

[82] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu, “Contextnet: Improving convolutional neural networks for automatic speech recognition with global context,” Interspeech , pp. 3610–3614, 2020.

[83] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu et al. , “Conformer: Convolution-augmented transformer for speech recognition,” Interspeech , pp. 5036–5040, 2020.

[84] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in International Conference on Machine Learning (ICML) , 2006, pp. 369–376.

[85] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in Interspeech , 2019, pp. 2613–2617.

[86] A. Fan, E. Grave, and A. Joulin, “Reducing transformer depth on demand with structured dropout,” arXiv preprint arXiv:1909.11556 , 2019.

[87] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, “Deep networks with stochastic depth,” in European Conference on Computer Vision (ECCV) . Springer, 2016, pp. 646–661.

[88] V. Pratap et al. , “Wav2letter++: A fast open-source speech recognition system,” in International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 6460–6464.

[89] J. Ao, R. Wang, L. Zhou, S. Liu, S. Ren, Y. Wu, T. Ko, Q. Li, Y. Zhang, Z. Wei et al. , “Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,” arXiv preprint arXiv:2110.07205 , 2021.