X-Vectors: Robust DNN Embeddings for Speaker Recognition
X-VECTORS: ROBUST DNN EMBEDDINGS FOR SPEAKER RECOGNITION
X-Vectors：面向说话人识别的稳健 DNN 嵌入
David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, Sanjeev Khudanpur
Center for Language and Speech Processing & Human Language Technology Center of Excellence
The Johns Hopkins University, Baltimore, MD 21218, USA
ABSTRACT
摘要
In this paper, we use data augmentation to improve performance of deep neural network (DNN) embeddings for speaker recognition. The DNN, which is trained to discriminate between speakers, maps variable-length utterances to fixed-dimensional embeddings that we call x-vectors. Prior studies have found that embeddings leverage large-scale training datasets better than i-vectors. However, it can be challenging to collect substantial quantities of labeled data for training. We use data augmentation, consisting of added noise and reverberation, as an inexpensive method to multiply the amount of training data and improve robustness. The x-vectors are compared with i-vector baselines on Speakers in the Wild and NIST SRE 2016 Cantonese. We find that while augmentation is beneficial in the PLDA classifier, it is not helpful in the i-vector extractor. However, the x-vector DNN effectively exploits data augmentation, due to its supervised training. As a result, the x-vectors achieve superior performance on the evaluation datasets.
本文用数据增强来提升说话人识别中的深度神经网络（DNN）嵌入。该网络被训练来区分说话人，把变长语句映射成固定维嵌入，我们称之为 x-vector。已有研究表明，嵌入比 i-vector 更能利用大规模训练数据，但要收集足够多的带标注数据并不容易。我们把加噪声和加混响当作一种便宜的办法，用来放大训练数据量并提高稳健性。x-vector 与 i-vector 基线在 Speakers in the Wild 以及 NIST SRE 2016 粤语部分上做了比较。我们发现，增强对 PLDA 分类器有帮助，对 i-vector 抽取器却没有帮助。然而，由于是有监督训练，x-vector 网络能够有效利用数据增强。因此，x-vector 在评测集上取得了更好的性能。
Index Terms— speaker recognition, deep neural networks, data augmentation, x-vectors
索引词—说话人识别，深度神经网络，数据增强，x-vector
1. INTRODUCTION
1. 引言
Using deep neural networks (DNN) to capture speaker characteristics is currently a very active research area. In our approach, representations called x-vectors are extracted from a DNN and used like i-vectors. This paper builds on our recent DNN embedding architecture [1]. We show that artificially augmenting the training data with noises and reverberation is a highly effective strategy for improving performance in DNN embedding systems.
用深度神经网络（DNN）刻画说话人特征，眼下是一个非常活跃的研究方向。在我们的方法里，从网络中抽出称为 x-vector 的表征，并像 i-vector 一样使用。本文建立在我们最近的 DNN 嵌入结构 [1] 之上。我们表明，用噪声和混响人为增强训练数据，是提升 DNN 嵌入系统性能的十分有效的策略。
Most speaker recognition systems are based on i-vectors [2]. The standard approach consists of a universal background model (UBM), and a large projection matrix that are learned in an unsupervised way to maximize the data likelihood. The projection maps high-dimensional statistics from the UBM into a low-dimensional representation, known as an i-vector. A probabilistic linear discriminant analysis (PLDA) [3] classifier is used to compare i-vectors, and enable same-or-different speaker decisions [4, 5, 6].
大多数说话人识别系统以 i-vector 为基础 [2]。标准做法包含一个通用背景模型（UBM）和一块很大的投影矩阵 ，二者都以无监督方式学习，以最大化数据似然。该投影把来自 UBM 的高维统计量映射成低维表征，也就是 i-vector。再用概率线性判别分析（PLDA）[3] 分类器比较 i-vector，从而做出是否同一说话人的判决 [4, 5, 6]。
The DNNs most often found in speaker recognition are trained as acoustic models for automatic speech recognition (ASR), and are then used to enhance phonetic modeling in the i-vector UBM: either posteriors from the ASR DNN replace those from a Gaussian mixture model (GMM) [7, 8], or bottleneck features are extracted from the DNN and combined with acoustic features [9]. In either case, if the ASR DNN is trained on in-domain data, the improvement over traditional acoustic i-vectors is substantial [10, 11, 12]. However, this approach introduces the need for transcribed training data and greatly increases computational complexity compared to traditional i-vectors.
说话人识别里最常见的 DNN，是先作为自动语音识别（ASR）的声学模型来训练，再用来加强 i-vector UBM 中的音素建模：要么用 ASR 网络的后验替换高斯混合模型（GMM）的后验 [7, 8]，要么从网络中抽出瓶颈特征，再与声学特征结合 [9]。无论哪一种，只要 ASR 网络是在域内数据上训练的，相对传统声学 i-vector 的提升都相当可观 [10, 11, 12]。不过，这种方法需要带转写的训练数据，计算复杂度也比传统 i-vector 高得多。
Alternatively, neural networks can be directly optimized to discriminate between speakers. This has potential to produce powerful, compact systems [13], that only require speaker labels to train. In early systems, neural networks are trained to separate speakers, and frame-level representations are extracted from the network and used as features for Gaussian speaker models [14, 15, 16]. Heigold et al., introduced an end-to-end system, trained on the phrase “OK Google,” that jointly learns an embedding along with a similarity metric to compare pairs of embeddings [13]. Snyder et al., adapted this approach to a text-independent application and inserted a temporal pooling layer into the network to handle variable-length segments [17]. The work in [1] split the end-to-end approach into two parts: a DNN to produce embeddings and a separately trained classifier to compare them. This facilitates the use of all the accumulated backend technology developed over the years for i-vectors, such as length-normalization, PLDA scoring, and domain adaptation techniques.
另一条路是直接优化神经网络来区分说话人。这有可能做出既强又紧凑的系统 [13]，训练时只需说话人标签。在早期系统里，网络被训练来分开说话人，再从网络中抽出帧级表征，作为高斯说话人模型的特征 [14, 15, 16]。Heigold 等人引入了一套端到端系统，在口令 “OK Google” 上训练，同时学习嵌入以及比较成对嵌入的相似度度量 [13]。Snyder 等人把这一做法改用于与文本无关的场景，并在网络中插入时间池化层，以处理变长片段 [17]。文献 [1] 把端到端方案拆成两部分：一个产生嵌入的 DNN，以及一个单独训练、用来比较嵌入的分类器。这样就可以继续使用这些年为 i-vector 积累下来的后端技术，例如长度归一化、PLDA 打分和域适应方法。
DNN embedding performance appears to be highly scalable with the amount of training data. As a result, these systems have found success leveraging large proprietary datasets [13, 17, 18]. However, recent systems have shown promising performance trained on only publicly available speaker recognition corpora [1, 19, 20]. This paper is based on the work in [1] and applies data augmentation to the DNN training procedure. This increases the amount and diversity of the existing training data, and achieves a significant improvement for the x-vector system. In comparing with x-vectors, we also contribute a study of augmentation in i-vector systems.
DNN 嵌入的性能似乎随训练数据量高度可扩展。因此，这类系统在利用大规模专有数据时取得了成功 [13, 17, 18]。不过，最近也有系统仅在公开的说话人识别语料上训练，就表现出有希望的性能 [1, 19, 20]。本文以 [1] 的工作为基础，把数据增强用到 DNN 的训练过程中。这增加了已有训练数据的数量和多样性，并使 x-vector 系统获得显著提升。在与 x-vector 对照的同时，我们也贡献了一项关于 i-vector 系统中数据增强的研究。
2. SPEAKER RECOGNITION SYSTEMS
2. 说话人识别系统
This section describes the speaker recognition systems developed for this study, which consist of two i-vector baselines and the DNN x-vector system. All systems are built using the Kaldi speech recognition toolkit [21].
本节介绍本研究中搭建的说话人识别系统，包括两个 i-vector 基线和 DNN x-vector 系统。全部系统都用 Kaldi 语音识别工具包构建 [21]。
2.1. Acoustic i-vector
2.1. 声学 i-vector
A traditional i-vector system based on the GMM-UBM recipe described in [11] serves as our acoustic-feature baseline system. The features are 20 MFCCs with a frame-length of 25ms that are mean-normalized over a sliding window of up to 3 seconds. Delta and acceleration are appended to create 60 dimension feature vectors. An energy-based speech activity detection (SAD) system selects features corresponding to speech frames. The UBM is a 2048 component full-covariance GMM. The system uses a 600 dimensional i-vector extractor and PLDA for scoring (see Section 2.4).
我们的声学特征基线，是一套基于 [11] 中 GMM-UBM 配方的传统 i-vector 系统。特征为 20 维 MFCC，帧长 25 ms，并在最长 3 秒的滑动窗口上做均值归一化。再拼接一阶和二阶差分，得到 60 维特征向量。基于能量的语音活动检测（SAD）系统选出对应语音帧的特征。UBM 是一个 2048 个分量的全协方差 GMM。系统使用 600 维 i-vector 抽取器，并用 PLDA 打分（见第 2.4 节）。
2.2. Phonetic bottleneck i-vector
2.2. 音素瓶颈 i-vector
This i-vector system incorporates phonetic bottleneck features (BNF) from an ASR DNN acoustic model and is similar to [9]. The DNN is a time-delay acoustic model with p-norm nonlinearities. The ASR DNN is trained on the Fisher English corpus and uses the same recipe and architecture as the system described in Section 2.2 of [11], except that the penultimate layer is replaced with a 60 dimensional linear bottleneck layer. Excluding the softmax output layer, which is not needed to compute BNFs, the DNN has 9.2 million parameters.
这套 i-vector 系统引入了来自 ASR DNN 声学模型的音素瓶颈特征（BNF），做法与 [9] 类似。该网络是带 p-norm 非线性的时延声学模型。ASR 网络在 Fisher 英语语料上训练，配方和结构与 [11] 第 2.2 节的系统相同，只是把倒数第二层换成了 60 维线性瓶颈层。去掉计算 BNF 时不需要的 softmax 输出层后，该网络有 920 万个参数。
The BNFs are concatenated with the same 20 dimensional MFCCs described in Section 2.1 plus deltas to create 100 dimensional features. The remaining components of the system (feature processing, UBM, i-vector extractor, and PLDA classifier) are identical to the acoustic system in Section 2.1.
BNF 与第 2.1 节中同样的 20 维 MFCC 以及一阶差分拼接，得到 100 维特征。系统的其余部分（特征处理、UBM、i-vector 抽取器和 PLDA 分类器）与第 2.1 节的声学系统完全相同。
2.3. The x-vector system
2.3. x-vector 系统
This section describes the x-vector system. It is based on the DNN embeddings in [1] and described in greater detail there.
本节介绍 x-vector 系统。它以 [1] 中的 DNN 嵌入为基础，更细致的说明见该文。
Our software framework has been made available in the Kaldi toolkit. An example recipe is in the main branch of Kaldi at https://github.com/kaldi-asr/kaldi/tree/master/egs/sre16/v2 and a pretrained x-vector system can be downloaded from http://kaldi-asr.org/models.html. The recipe and model are similar to the x-vector system described in Section 4.4.
我们的软件框架已放进 Kaldi 工具包。示例配方在 Kaldi 主分支 https://github.com/kaldi-asr/kaldi/tree/master/egs/sre16/v2 ，预训练的 x-vector 系统可从 http://kaldi-asr.org/models.html 下载。该配方和模型与第 4.4 节描述的 x-vector 系统相近。
| Layer | Layer context | Total context | Input x output |
|---|---|---|---|
| frame1 | 5 | 120x512 | |
| frame2 | 9 | 1536x512 | |
| frame3 | 15 | 1536x512 | |
| frame4 | 15 | 512x512 | |
| frame5 | 15 | 512x1500 | |
| stats pooling | x3000 | ||
| segment6 | 3000x512 | ||
| segment7 | 512x512 | ||
| softmax | 512x |
Table 1. The embedding DNN architecture. x-vectors are extracted at layer segment6, before the nonlinearity. The in the softmax layer corresponds to the number of training speakers.
| 层 | 层上下文 | 总上下文 | 输入 × 输出 |
|---|---|---|---|
| frame1 | 5 | 120x512 | |
| frame2 | 9 | 1536x512 | |
| frame3 | 15 | 1536x512 | |
| frame4 | 15 | 512x512 | |
| frame5 | 15 | 512x1500 | |
| stats pooling | x3000 | ||
| segment6 | 3000x512 | ||
| segment7 | 512x512 | ||
| softmax | 512x |
表 1. 嵌入 DNN 的结构。x-vector 在 segment6 层、非线性之前取出。softmax 层中的 对应训练说话人的数量。
The features are 24 dimensional filterbanks with a frame-length of 25ms, mean-normalized over a sliding window of up to 3 seconds. The same energy SAD as used in the baseline systems filters out nonspeech frames.
特征是 24 维 filterbank，帧长 25 ms，并在最长 3 秒的滑动窗口上做均值归一化。与基线系统相同的能量 SAD 滤掉非语音帧。
The DNN configuration is outlined in Table 1. Suppose an input segment has frames. The first five layers operate on speech frames, with a small temporal context centered at the current frame . For example, the input to layer frame3 is the spliced output of frame2, at frames , and . This builds on the temporal context of the earlier layers, so that frame3 sees a total context of 15 frames.
DNN 的配置见表 1。设输入片段有 帧。前五层在语音帧上运算，并以当前帧 为中心取一小段时间上下文。例如，frame3 层的输入是 frame2 在第 、 和 帧输出的拼接。这建立在更早层的时间上下文之上，因此 frame3 总共能看到 15 帧的上下文。
The statistics pooling layer aggregates all frame-level outputs from layer frame5 and computes its mean and standard deviation. The statistics are 1500 dimensional vectors, computed once for each input segment. This process aggregates information across the time dimension so that subsequent layers operate on the entire segment. In Table 1, this is denoted by a layer context of and a total context of . The mean and standard deviation are concatenated together and propagated through segment-level layers and finally the softmax output layer. The nonlinearities are all rectified linear units (ReLUs).
统计池化层汇总 frame5 层全部 个帧级输出，并计算其均值和标准差。这些统计量是 1500 维向量，对每个输入片段计算一次。这一过程沿时间维聚合信息，使后续层在整段上运算。在表 1 中，这表示为层上下文 、总上下文 。均值与标准差拼接后，送入段级层，最后到达 softmax 输出层。非线性全部是修正线性单元（ReLU）。
The DNN is trained to classify the speakers in the training data. A training example consists of a chunk of speech features (about 3 seconds average), and the corresponding speaker label. After training, embeddings are extracted from the affine component of layer segment6. Excluding the softmax output layer and segment7 (because they are not needed after training) there is a total of 4.2 million parameters.
该 DNN 被训练来分类训练数据中的 个说话人。一个训练样本由一段语音特征（平均约 3 秒）和对应的说话人标签组成。训练结束后，嵌入从 segment6 层的仿射变换部分取出。去掉 softmax 输出层和 segment7（因为训练结束后不再需要）后，共有 420 万个参数。
2.4. PLDA classifier
2.4. PLDA 分类器
The same type of PLDA [3] classifier is used for the x-vector and i-vector systems. The representations (x-vectors or i-vectors) are centered, and projected using LDA. The LDA dimension was tuned on the SITW development set to 200 for i-vectors and 150 for x-vectors. After dimensionality reduction, the representations are length-normalized and modeled by PLDA. The scores are normalized using adaptive s-norm [22].
x-vector 与 i-vector 系统使用同一类 PLDA [3] 分类器。这些表征（x-vector 或 i-vector）先做中心化，再用 LDA 投影。LDA 维数在 SITW 开发集上调节，i-vector 为 200，x-vector 为 150。降维之后，表征做长度归一化，并由 PLDA 建模。分数用自适应 s-norm 归一化 [22]。
3. EXPERIMENTAL SETUP
3. 实验设置
3.1. Training data
3.1. 训练数据
The training data consists of both telephone and microphone speech, the bulk of which is in English. All wideband audio is downsampled to 8kHz.
训练数据既包含电话语音，也包含麦克风语音，其中大部分是英语。所有宽带音频都降采样到 8 kHz。
The SWBD portion consists of Switchboard 2 Phases 1, 2, and 3 as well as Switchboard Cellular. In total, the SWBD dataset contains about 28k recordings from 2.6k speakers. The SRE portion consists of NIST SREs from 2004 to 2010 along with Mixer 6 and contains about 63k recordings from 4.4k speakers. In the experiments in Sections 4.1–4.4 the extractors (UBM/ or embedding DNN) are trained on SWBD and SRE and the PLDA classifiers are trained on just SRE. Data augmentation is described in Section 3.3 and is applied to these datasets as explained throughout Section 4.
SWBD 部分包括 Switchboard 2 的第 1、2、3 期以及 Switchboard Cellular。合计约有来自 2600 个说话人的 2.8 万条录音。SRE 部分包括 2004 至 2010 年的 NIST SRE 以及 Mixer 6，约有来自 4400 个说话人的 6.3 万条录音。在第 4.1–4.4 节的实验中，抽取器（UBM/ 或嵌入 DNN）在 SWBD 和 SRE 上训练，PLDA 分类器只在 SRE 上训练。数据增强见第 3.3 节，并按第 4 节各小节的说明施加到这些数据上。
In the last experiment in Section 4.5 we incorporate audio from the new VoxCeleb dataset [19] into both extractor and PLDA training lists. The dataset consists of videos from 1,251 celebrity speakers. Although SITW and VoxCeleb were collected independently, we discovered an overlap of 60 speakers between the two datasets. We removed the overlapping speakers from VoxCeleb prior to using it for training. This reduces the size of the dataset to 1,191 speakers and about 20k recordings.
在第 4.5 节的最后一组实验中，我们把新的 VoxCeleb 数据集 [19] 中的音频加入抽取器和 PLDA 的训练列表。该数据集包含 1251 位名人说话人的视频。尽管 SITW 与 VoxCeleb 是独立采集的，我们仍发现二者有 60 个说话人重叠。在用于训练之前，我们从 VoxCeleb 中去掉了这些重叠说话人。数据集规模因此降到 1191 个说话人、约 2 万条录音。
The ASR DNN used in the i-vector (BNF) system was trained on the Fisher English corpus. To achieve a limited form of domain adaptation, the development data from SITW and SRE16 is pooled and used for centering and score normalization. No augmentation is applied to these lists.
i-vector（BNF）系统所用的 ASR 网络在 Fisher 英语语料上训练。为获得有限形式的域适应，SITW 与 SRE16 的开发数据被合并，用于中心化和分数归一化。这些列表不做增强。
3.2. Evaluation
3.2. 评测
Our evaluation consists of two distinct datasets: Speakers in the Wild (SITW) Core [23] and the Cantonese portion of the NIST SRE 2016 evaluation (SRE16) [24]. SITW consists of unconstrained video audio of English speakers, with naturally occurring noises, reverberation, as well as device and codec variability. The SRE16 portion consists of Cantonese conversational telephone speech. Both enroll and test SITW utterances vary in length form 6–240 seconds. For SRE16, the enrollment utterances contain about 60 seconds of speech while the test utterances vary from 10–60 seconds.
评测使用两个不同的数据集：Speakers in the Wild（SITW）Core [23]，以及 NIST SRE 2016 评测的粤语部分（SRE16）[24]。SITW 是英语说话人的无约束视频音频，带有自然出现的噪声、混响，以及设备和编解码差异。SRE16 部分是粤语会话电话语音。SITW 的注册和测试语句长度从 6 秒到 240 秒不等。对 SRE16，注册语句约含 60 秒语音，测试语句则从 10 秒到 60 秒不等。
We report results in terms of equal error-rate (EER) and the minimum of the normalized detection cost function (DCF) at and . Note that the SRE16 results have not been “equalized [24].”
我们以等错误率（EER），以及在 和 处归一化检测代价函数（DCF）的最小值来报告结果。需要说明的是，SRE16 的结果未经 “均衡 [24]”。
3.3. Data augmentation
3.3. 数据增强
Augmentation increases the amount and diversity of the existing training data. Our strategy employs additive noises and reverberation. Reverberation involves convolving room impulse responses (RIR) with audio. We use the simulated RIRs described by Ko et al. in [25], and the reverberation itself is performed with the multi-condition training tools in the Kaldi ASpIRE recipe [21]. For additive noise, we use the MUSAN dataset, which consists of over 900 noises, 42 hours of music from various genres and 60 hours of speech from twelve languages [26]. Both MUSAN and the RIR datasets are freely available from http://www.openslr.org.
增强用来增加已有训练数据的数量和多样性。我们的策略采用加性噪声和混响。混响是把房间脉冲响应（RIR）与音频做卷积。我们使用 Ko 等人在 [25] 中描述的模拟 RIR，混响本身则用 Kaldi ASpIRE 配方中的多条件训练工具完成 [21]。加性噪声使用 MUSAN 数据集，其中包含 900 多种噪声、42 小时不同流派的音乐，以及十二种语言共 60 小时的语音 [26]。MUSAN 和 RIR 数据集都可以从 http://www.openslr.org 免费获得。
We use a 3-fold augmentation that combines the original “clean” training list with two augmented copies. To augment a recording, we choose between one of the following randomly:
我们采用 3 倍增强，把原来的 “清洁” 训练列表与两份增强拷贝合并。要增强一条录音时，我们从下面几种做法中随机选一种：
• babble: Three to seven speakers are randomly picked from MUSAN speech, summed together, then added to the original signal (13-20dB SNR).
• babble（多人叠音）：从 MUSAN 语音中随机挑选三到七个说话人，叠加后再加到原信号上（信噪比 13–20 dB）。
• music: A single music file is randomly selected from MUSAN, trimmed or repeated as necessary to match duration, and added to the original signal (5-15dB SNR).
• music（音乐）：从 MUSAN 中随机选一首音乐，按需要裁切或重复以匹配时长，再加到原信号上（信噪比 5–15 dB）。
• noise: MUSAN noises are added at one second intervals throughout the recording (0-15dB SNR).
• noise（噪声）：把 MUSAN 噪声按一秒间隔加到整条录音中（信噪比 0–15 dB）。
• reverb: The training recording is artificially reverberated via convolution with simulated RIRs.
• reverb（混响）：用模拟 RIR 与训练录音卷积，人为加入混响。
4. RESULTS
4. 结果
The main results are presented in Table 2 and are referenced throughout Sections 4.1–4.5. We compare performance of two i-vector systems, labeled i-vector (acoustic) and i-vector (BNF), with the x-vector system. The systems are described in Sections 2.1, 2.2 and 2.3, respectively. Throughout the following sections, we use the term extractor to refer to either the UBM/ or the embedding DNN.
主要结果列于表 2，并在第 4.1–4.5 节中被反复引用。我们比较两套 i-vector 系统——分别记为 i-vector（acoustic）和 i-vector（BNF）——与 x-vector 系统的性能。这三套系统分别在第 2.1、2.2 和 2.3 节中介绍。在随后各节中，我们用 “抽取器” 一词统称 UBM/ 或嵌入 DNN。
| SITW Core | SRE16 Cantonese | ||||||
|---|---|---|---|---|---|---|---|
| EER(%) | DCF | DCF | EER(%) | DCF | DCF | ||
| 4.1 Original systems | i-vector (acoustic) | 9.29 | 0.621 | 0.785 | 9.23 | 0.568 | 0.741 |
| i-vector (BNF) | 9.10 | 0.558 | 0.719 | 9.68 | 0.574 | 0.765 | |
| x-vector | 9.40 | 0.632 | 0.790 | 8.00 | 0.491 | 0.697 | |
| 4.2 PLDA aug. | i-vector (acoustic) | 8.64 | 0.588 | 0.755 | 8.92 | 0.544 | 0.717 |
| i-vector (BNF) | 8.00 | 0.514 | 0.689 | 8.82 | 0.532 | 0.726 | |
| x-vector | 7.56 | 0.586 | 0.746 | 7.45 | 0.463 | 0.669 | |
| 4.3 Extractor aug. | i-vector (acoustic) | 8.89 | 0.626 | 0.790 | 9.20 | 0.575 | 0.748 |
| i-vector (BNF) | 7.27 | 0.533 | 0.730 | 8.89 | 0.569 | 0.777 | |
| x-vector | 7.19 | 0.535 | 0.719 | 6.29 | 0.428 | 0.626 | |
| 4.4 PLDA and extractor aug. | i-vector (acoustic) | 8.04 | 0.578 | 0.752 | 8.95 | 0.555 | 0.720 |
| i-vector (BNF) | 6.49 | 0.492 | 0.690 | 8.29 | 0.534 | 0.749 | |
| x-vector | 6.00 | 0.488 | 0.677 | 5.86 | 0.410 | 0.593 | |
| 4.5 Incl. VoxCeleb | i-vector (acoustic) | 7.45 | 0.552 | 0.723 | 9.23 | 0.557 | 0.742 |
| i-vector (BNF) | 6.09 | 0.472 | 0.660 | 8.12 | 0.523 | 0.751 | |
| x-vector | 4.16 | 0.393 | 0.606 | 5.71 | 0.399 | 0.569 |
Table 2. Results using data augmentation in various systems. “Extractor” refers to either the UBM/ or the embedding DNN. For each experiment, the best results are boldface.
| SITW Core | SRE16 粤语 | ||||||
|---|---|---|---|---|---|---|---|
| EER(%) | DCF | DCF | EER(%) | DCF | DCF | ||
| 4.1 原始系统 | i-vector (acoustic) | 9.29 | 0.621 | 0.785 | 9.23 | 0.568 | 0.741 |
| i-vector (BNF) | 9.10 | 0.558 | 0.719 | 9.68 | 0.574 | 0.765 | |
| x-vector | 9.40 | 0.632 | 0.790 | 8.00 | 0.491 | 0.697 | |
| 4.2 仅增强 PLDA | i-vector (acoustic) | 8.64 | 0.588 | 0.755 | 8.92 | 0.544 | 0.717 |
| i-vector (BNF) | 8.00 | 0.514 | 0.689 | 8.82 | 0.532 | 0.726 | |
| x-vector | 7.56 | 0.586 | 0.746 | 7.45 | 0.463 | 0.669 | |
| 4.3 仅增强抽取器 | i-vector (acoustic) | 8.89 | 0.626 | 0.790 | 9.20 | 0.575 | 0.748 |
| i-vector (BNF) | 7.27 | 0.533 | 0.730 | 8.89 | 0.569 | 0.777 | |
| x-vector | 7.19 | 0.535 | 0.719 | 6.29 | 0.428 | 0.626 | |
| 4.4 同时增强 PLDA 与抽取器 | i-vector (acoustic) | 8.04 | 0.578 | 0.752 | 8.95 | 0.555 | 0.720 |
| i-vector (BNF) | 6.49 | 0.492 | 0.690 | 8.29 | 0.534 | 0.749 | |
| x-vector | 6.00 | 0.488 | 0.677 | 5.86 | 0.410 | 0.593 | |
| 4.5 加入 VoxCeleb | i-vector (acoustic) | 7.45 | 0.552 | 0.723 | 9.23 | 0.557 | 0.742 |
| i-vector (BNF) | 6.09 | 0.472 | 0.660 | 8.12 | 0.523 | 0.751 | |
| x-vector | 4.16 | 0.393 | 0.606 | 5.71 | 0.399 | 0.569 |
表 2. 在不同系统中使用数据增强的结果。“抽取器”指 UBM/ 或嵌入 DNN。每一组实验中最好的结果以粗体标出。
4.1. Original systems
4.1. 原始系统
In this section, we evaluate systems without data augmentation. The extractors are trained on the SWBD and SRE datasets described in Section 3.1. The PLDA classifiers are trained on just the SRE dataset. Without using augmentation, the best results on SITW are obtained by i-vector (BNF), which is 12% better than the x-vector system at DCF. The acoustic i-vector system also achieves slightly lower error-rates than the x-vector system on SITW. However, even without augmentation, the best results for SRE16 Cantonese are obtained by the x-vectors. In terms of DCF, these embeddings are about 14% better than either i-vector system. We observe that i-vector (BNF) has no advantage over i-vector (acoustic) for this Cantonese speech. This echoes recent studies that have found that the large gains achieved by BNFs in English speech are not necessarily transferable to non-English data [27].
本节在不加数据增强的情况下评测各系统。抽取器在第 3.1 节所述的 SWBD 和 SRE 数据集上训练，PLDA 分类器只在 SRE 上训练。不加增强时，SITW 上最好的结果来自 i-vector（BNF），在 DCF 上比 x-vector 系统好 12%。声学 i-vector 在 SITW 上的错误率也略低于 x-vector。不过，即便没有增强，SRE16 粤语上最好的结果仍来自 x-vector。以 DCF 计，这些嵌入比两套 i-vector 系统都大约好 14%。我们还观察到，对这种粤语语音，i-vector（BNF）并不比 i-vector（acoustic）更有优势。这与近期一些研究相呼应：BNF 在英语语音上取得的大幅提升，并不一定能迁移到非英语数据上 [27]。
4.2. PLDA augmentation
4.2. 增强 PLDA
In this experiment, the augmentation strategy described in Section 3.3 is applied to only the PLDA training list. We use the same extractors as the previous section, which were trained on the original datasets. PLDA augmentation results in a clear improvement for all three systems relative to Section 4.1. However, it appears that the x-vectors may benefit from the PLDA augmentation more than the baseline systems. On SITW, the x-vector system achieves slightly lower error-rates than i-vector (acoustic), but continues to lag behind i-vector (BNF) at most operating points. On SRE16, the x-vectors maintain an advantage over the i-vectors by about 14% in DCF.
本组实验把第 3.3 节的增强策略只加到 PLDA 训练列表上。抽取器与上一节相同，仍在原始数据上训练。相对第 4.1 节，增强 PLDA 使三套系统都有明显提升。不过，x-vector 从 PLDA 增强中获得的收益似乎大于基线系统。在 SITW 上，x-vector 系统的错误率已略低于 i-vector（acoustic），但在多数工作点上仍落后于 i-vector（BNF）。在 SRE16 上，x-vector 相对 i-vector 在 DCF 上仍保持约 14% 的优势。
4.3. Extractor augmentation
4.3. 增强抽取器
We now apply data augmentation to the extractor (UBM/ or embedding DNN) training lists but not the PLDA list. The effect of augmenting the UBM/ is inconsistent in the i-vector system. This observation is supported by prior studies on i-vectors, which have found that augmentation is only effective in the PLDA classifier [28, 29]. On the other hand, augmenting the embedding DNN training list results in a large improvement. In contrast to the i-vector systems, this is considerably more effective than augmenting the PLDA training list. On SITW, the x-vector system achieves lower error-rates than i-vector (acoustic) and has now caught up to the i-vector (BNF) system. On SRE16, the x-vectors are now 25% better than the i-vectors in DCF, which is almost double the improvement the DNN embeddings had with PLDA augmentation alone. The findings in this section indicate that data augmentation is only beneficial for extractors trained with supervision.
我们现在把数据增强加到抽取器（UBM/ 或嵌入 DNN）的训练列表上，但不加到 PLDA 列表上。增强 UBM/ 对 i-vector 系统的效果并不稳定。这一观察得到以往 i-vector 研究的支持：那些研究认为增强只在 PLDA 分类器上有效 [28, 29]。另一方面，增强嵌入 DNN 的训练列表会带来大幅提升。与 i-vector 系统相反，这比只增强 PLDA 训练列表有效得多。在 SITW 上，x-vector 系统的错误率已低于 i-vector（acoustic），并已追上 i-vector（BNF）系统。在 SRE16 上，x-vector 在 DCF 上比 i-vector 好 25%，几乎是仅增强 PLDA 时 DNN 嵌入所获提升的两倍。本节的发现表明，数据增强只对有监督训练的抽取器有益。
4.4. PLDA and extractor augmentation
4.4. 同时增强 PLDA 与抽取器
In the previous sections, we saw that PLDA augmentation was helpful in both i-vector and DNN embedding systems, although extractor augmentation was only clearly beneficial in the embedding system. In this experiment, we apply data augmentation to both the extractor and PLDA training lists. We continue to use SWBD and SRE for extractor training and only SRE for PLDA. On SITW the x-vectors are now 10-25% better than i-vector (acoustic) and are slightly better than i-vector (BNF) at all operating points. On SRE16 Cantonese, the x-vectors continue to maintain the large lead over the i-vector systems established in Section 4.3.
在前面几节中我们看到，增强 PLDA 对 i-vector 和 DNN 嵌入系统都有帮助，但增强抽取器只在嵌入系统上有明确收益。本组实验把数据增强同时加到抽取器和 PLDA 的训练列表上。抽取器仍用 SWBD 和 SRE 训练，PLDA 只用 SRE。在 SITW 上，x-vector 现在比 i-vector（acoustic）好 10–25%，并在所有工作点上略优于 i-vector（BNF）。在 SRE16 粤语上，x-vector 继续保持第 4.3 节中相对 i-vector 系统建立起的大幅领先。
4.5. Including VoxCeleb
4.5. 加入 VoxCeleb
The training data in Sections 4.1–4.4 is dominated by telephone speech. In this experiment, we explore the effect of adding a large amount of microphone speech to the systems in Section 4.4. The VoxCeleb dataset [19] is augmented, and added to both the extractor and PLDA lists. As noted in Section 3.1, we found 60 speakers which overlap with SITW; all speech for these speakers was removed from the training lists.
第 4.1–4.4 节的训练数据以电话语音为主。本组实验考察把大量麦克风语音加入第 4.4 节系统后的效果。VoxCeleb 数据集 [19] 经过增强后，被加入抽取器和 PLDA 列表。如第 3.1 节所述，我们发现有 60 个说话人与 SITW 重叠；这些说话人的全部语音都已从训练列表中去掉。
On SITW, both i-vector and x-vector systems improve significantly. However, the x-vector exploits the large increase in the amount of in-domain data better than the i-vector systems. Compared to i-vector (acoustic), the x-vectors are better by 44% in EER and 29% in DCF. Compared to the i-vector (BNF) system, it is now better by 32% in EER and 17% in DCF. On SRE16, the i-vector systems remain roughly the same compared to Section 4.4, but the x-vectors improve on all operating points by a small amount. These results are illustrated by detection error tradeoff (DET) plots in Figures 1 and 2.
在 SITW 上，i-vector 和 x-vector 系统都有显著提升。不过，x-vector 比 i-vector 系统更好地利用了域内数据量的大幅增加。相对 i-vector（acoustic），x-vector 在 EER 上好 44%，在 DCF 上好 29%。相对 i-vector（BNF）系统，现在 EER 好 32%，DCF 好 17%。在 SRE16 上，i-vector 系统与第 4.4 节大致持平，而 x-vector 在所有工作点上都有小幅提高。这些结果由倒易错误特征（DET）曲线表示，见图 1 和图 2。
Fig. 1. DET curve for the Cantonese portion of NIST SRE16 using Section 4.5 systems.
图 1. 使用第 4.5 节系统时，NIST SRE16 粤语部分上的 DET 曲线。
Fig. 2. DET curve for the SITW Core using Section 4.5 systems.
图 2. 使用第 4.5 节系统时，SITW Core 上的 DET 曲线。
5. CONCLUSIONS
5. 结论
This paper studied DNN embeddings for speaker recognition. We found that data augmentation is an easily implemented and effective strategy for improving their performance. We also made the x-vector system – our implementation of DNN embeddings – available in the Kaldi toolkit. We found that the x-vector system significantly outperformed two standard i-vector baselines on SRE16 Cantonese. After including a large amount of augmented microphone speech, the x-vectors achieved much lower error-rates than our best baseline on Speakers in the Wild. Bottleneck features from an ASR DNN are used in our best i-vector system, and so it requires transcribed data during training. On the other hand, the x-vector DNN needs only speaker labels to train, making it potentially ideal for domains with little transcribed speech. More generally, it appears that x-vectors are now a strong contender for next-generation representations for speaker recognition.
本文研究了用于说话人识别的 DNN 嵌入。我们发现，数据增强是一种容易实现、且能有效提升其性能的策略。我们也把 x-vector 系统——也就是我们对 DNN 嵌入的实现——放进了 Kaldi 工具包。结果表明，x-vector 系统在 SRE16 粤语上显著超过两套标准 i-vector 基线。在加入大量经过增强的麦克风语音之后，x-vector 在 Speakers in the Wild 上的错误率也明显低于我们最好的基线。最好的 i-vector 系统用到了 ASR 网络的瓶颈特征，因此训练时需要转写数据。相比之下，x-vector 网络只需说话人标签即可训练，因而可能特别适合转写很少的领域。更一般地说，x-vector 目前看来已成为下一代说话人表征的有力候选。
6. ACKNOWLEDGMENTS
6. 致谢
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 1232825. This work was partially supported by NSF Grant No CRI-1513128. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation.
本材料基于美国国家科学基金会研究生研究奖学金（批准号 1232825）资助的工作。本研究也部分得到 NSF 资助 CRI-1513128 的支持。文中表达的任何意见、发现以及结论或建议，均为作者个人观点，并不一定反映美国国家科学基金会的立场。
7. REFERENCES
7. 参考文献
[1] D. Snyder, D. Garcia-Romero, D. Povey, and S. Khudanpur, “Deep neural network embeddings for text-independent speaker verification,” Proc. Interspeech, pp. 999–1003, 2017.
[2] N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 4, pp. 788–798, 2011.
[3] S. Ioffe, “Probabilistic linear discriminant analysis,” Computer Vision–ECCV 2006, pp. 531–542, 2006.
[4] P. Kenny, “Bayesian speaker verification with heavy-tailed priors.,” in Odyssey, 2010, p. 14.
[5] N. Brümmer and E. De Villiers, “The speaker partitioning problem.,” in Odyssey, 2010, p. 34.
[6] D. Garcia-Romero and C. Espy-Wilson, “Analysis of i-vector length normalization in speaker recognition systems.,” in Interspeech, 2011, pp. 249–252.
[7] Y. Lei, N. Scheffer, L. Ferrer, and M. McLaren, “A novel scheme for speaker recognition using a phonetically-aware deep neural network,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2014, pp. 1695–1699.
[8] P. Kenny, V. Gupta, T. Stafylakis, P. Ouellet, and J. Alam, “Deep neural networks for extracting Baum-Welch statistics for speaker recognition,” in Proc. Odyssey, 2014.
[9] M. McLaren, Y. Lei, and L. Ferrer, “Advances in deep neural network approaches to speaker recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 4814–4818.
[10] D. Garcia-Romero, X. Zhang, A. McCree, and D. Povey, “Improving speaker recognition performance in the domain adaptation challenge using deep neural networks,” in Spoken Language Technology Workshop (SLT), 2014 IEEE. IEEE, 2014, pp. 378–383.
[11] D. Snyder, D. Garcia-Romero, and D. Povey, “Time delay deep neural network-based universal background models for speaker recognition,” in 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). IEEE, 2015, pp. 92–97.
[12] S. O. Sadjadi, J. Pelecanos, and S. Ganapathy, “The ibm speaker recognition system: Recent advances and error analysis,” Interspeech, pp. 3633–3637, 2016.
[13] G. Heigold, I. Moreno, S. Bengio, and N. Shazeer, “End-to-end text-dependent speaker verification,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 5115–5119.
[14] Y. Konig, L. Heck, M. Weintraub, and K. Sonmez, “Nonlinear discriminant feature extraction for robust text-independent speaker recognition,” in Proc. RLA2C, ESCA workshop on Speaker Recognition and its Commercial and Forensic Applications, 1998.
[15] L. Heck, Y. Konig, K. Sonmez, and M. Weintraub, “Robustness to telephone handset distortion in speaker recognition by discriminative feature design,” in Speech Communication, 2000, vol. 31, pp. 181–192.
[16] A. Salman, Learning speaker-specific characteristics with deep neural architecture, Ph.D. thesis, University of Manchester, 2012.
[17] D. Snyder, P. Ghahremani, D. Povey, D. Garcia-Romero, Y. Carmiel, and S. Khudanpur, “Deep neural network-based speaker embeddings for end-to-end speaker verification,” in Spoken Language Technology Workshop (SLT). IEEE, 2016.
[18] S. Zhang, Z. Chen, Y. Zhao, J. Li, and Y. Gong, “End-to-end attention based text-dependent speaker verification,” in Spoken Language Technology Workshop (SLT), 2016 IEEE. IEEE, 2016, pp. 171–178.
[19] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” in Interspeech, 2017.
[20] C. Zhang and K. Koishida, “End-to-end text-independent speaker verification with triplet loss on short utterances,” Proc. Interspeech, pp. 1487–1491, 2017.
[21] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlíček, Y. Qian, P. Schwarz, et al., “The Kaldi speech recognition toolkit,” in Proceedings of the Automatic Speech Recognition & Understanding (ASRU) Workshop, 2011.
[22] D. Sturim and D. Reynolds, “Speaker adaptive cohort selection for tnorm in text-independent speaker verification,” in Acoustics, Speech, and Signal Processing, 2005. Proceedings.(ICASSP’05). IEEE International Conference on. IEEE, 2005, vol. 1, pp. I–741.
[23] M. McLaren, L. Ferrer, D. Castan, and A. Lawson, “The 2016 speakers in the wild speaker recognition evaluation.,” in Interspeech, 2016, pp. 823–827.
[24] “NIST speaker recognition evaluation 2016,” https://www.nist.gov/itl/iad/mig/speaker-recognition-evaluation-2016/, 2016.
[25] T. Ko, V. Peddinti, D. Povey, M. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 2017, pp. 5220–5224.
[26] D. Snyder, G Chen, and D. Povey, “MUSAN: A Music, Speech, and Noise Corpus,” 2015, arXiv:1510.08484v1.
[27] O. Novotný, P. Matějka, O. Glembeck, O Plchot, F. Grézl, L. Burget, and J. Černocký, “Analysis of the dnn-based sre systems in multi-language conditions,” in Spoken Language Technology Workshop (SLT). IEEE, 2016.
[28] Y. Lei, L. Burget, L. Ferrer, M. Graciarena, and N. Scheffer, “Towards noise-robust speaker recognition using probabilistic linear discriminant analysis,” in Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. IEEE, 2012, pp. 4253–4256.
[29] D. Garcia-Romero, X. Zhou, and C. Espy-Wilson, “Multicondition training of Gaussian plda models in i-vector space for noise and reverberation robust speaker recognition,” in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2012, pp. 4257–4260.