ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表

暂无章节

返回论文列表
目录

ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification

收录于 Interspeech 2020 会议论文;DOI 10.21437/Interspeech.2020-2650;论文集页码 3830–3834被引 2,815 次
打开论文来源

INTERSPEECH 2020
October 25–29, 2020, Shanghai, China

INTERSPEECH 2020
2020 年 10 月 25–29 日,中国上海

ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification

Brecht Desplanques, Jenthe Thienpondt, Kris Demuynck
IDLab, Department of Electronics and Information Systems, imec - Ghent University, Belgium
brecht.desplanques@ugent.be, jenthe.thienpondt@ugent.be

Abstract

摘要

Current speaker verification techniques rely on a neural network to extract speaker representations. The successful x-vector architecture is a Time Delay Neural Network (TDNN) that applies statistics pooling to project variable-length utterances into fixed-length speaker characterizing embeddings. In this paper, we propose multiple enhancements to this architecture based on recent trends in the related fields of face verification and computer vision.

当前的说话人验证方法,主要依靠神经网络来提取说话人表征。效果出色的 x-vector 架构是一种时延神经网络(TDNN),它通过统计池化,把变长话语投影成定长的说话人嵌入。本文根据人脸验证和计算机视觉里的近期做法,对这一架构提出多项改进。

Firstly, the initial frame layers can be restructured into 1-dimensional Res2Net modules with impactful skip connections. Similarly to SE-ResNet, we introduce Squeeze-and-Excitation blocks in these modules to explicitly model channel interdependencies. The SE block expands the temporal context of the frame layer by rescaling the channels according to global properties of the recording. Secondly, neural networks are known to learn hierarchical features, with each layer operating on a different level of complexity. To leverage this complementary information, we aggregate and propagate features of different hierarchical levels. Finally, we improve the statistics pooling module with channel-dependent frame attention. This enables the network to focus on different subsets of frames during each of the channel’s statistics estimation. The proposed ECAPA-TDNN architecture significantly outperforms state-of-the-art TDNN based systems on the VoxCeleb test sets and the 2019 VoxCeleb Speaker Recognition Challenge.

首先,可以把靠前的帧层改写成带有效跳跃连接的一维 Res2Net 模块。与 SE-ResNet 类似,我们在这些模块里引入 Squeeze-and-Excitation 块,以显式建模通道之间的相互依赖。SE 块会按整段录音的全局属性重新缩放各通道,从而扩展帧层的时间上下文。其次,神经网络会学习层级特征,各层对应不同的复杂度。为了利用这些互补信息,我们对不同层级的特征做聚合和传播。最后,我们用通道相关的帧注意力改进统计池化模块,使网络在估计每个通道的统计量时,能够关注不同的帧子集。提出的 ECAPA-TDNN 架构,在 VoxCeleb 测试集和 2019 年 VoxCeleb 说话人识别挑战上,显著超过了当时最先进的基于 TDNN 的系统。

Index Terms: speaker recognition, speaker verification, deep neural networks, x-vectors, channel attention

关键词:说话人识别,说话人验证,深度神经网络,x-vectors,通道注意力

  1. Introduction
  1. 引言

In recent years, x-vectors [1] and their subsequent improvements [2, 3, 4] have consistently provided state-of-the-art results on the task of speaker verification. Improving upon the original Time Delay Neural Network (TDNN) architecture is an active area of research. Usually, the neural networks are trained on the speaker identification task. After convergence, low dimensional speaker embeddings can be extracted from the bottleneck layer preceding the output layer to characterize the speaker in the input recording. Speaker verification can be accomplished by comparing the two embeddings corresponding with an enrollment and a test recording to accept or reject the hypothesis that both recordings contain the same speaker. A simple cosine distance measurement can be used for this comparison. In addition, a more complicated scoring backend can be trained such as Probabilistic Linear Discriminant Analysis (PLDA) [5].

近年来,x-vector [1] 及其后续改进 [2, 3, 4] 在说话人验证任务上一直给出当时最先进的结果。如何改进最初的时延神经网络(TDNN)架构,仍是活跃的研究方向。这类网络通常先在说话人识别任务上训练;收敛之后,从输出层前面的瓶颈层取出低维说话人嵌入,用来刻画输入录音中的说话人。验证时,比较注册录音和测试录音各自对应的嵌入,以接受或拒绝“两段录音是同一说话人”这一假设。比较可以用简单的余弦距离,也可以再训练更复杂的打分后端,例如概率线性判别分析(PLDA)[5]。

The rising popularity of the x-vector system has resulted in significant architectural improvements and optimized training procedures [6] over the original approach. The topology of the system was improved by incorporating elements of the popular ResNet [7] architecture. Adding residual connections between the frame-level layers has been shown to enhance the embeddings [3, 4]. Additionally, residual connections enable the back-propagation algorithm to converge faster and help avoid the vanishing gradient problem [7].

随着 x-vector 系统流行起来,原方法之上已经出现了显著的结构改进和更优化的训练流程 [6]。系统拓扑通过吸收流行的 ResNet [7] 架构中的要素得到了改进。在帧级各层之间加入残差连接,已被证明能够增强嵌入 [3, 4]。此外,残差连接也能让反向传播收敛得更快,并有助于避免梯度消失 [7]。

The statistics pooling layer in the x-vector system projects the variable-length input into a fixed-length representation by gathering simple statistics of hidden node activations across time. The authors in [8, 9] introduce a temporal self-attention system to this pooling layer which allows the network to only focus on frames it deems important. It can also be interpreted as a Voice Activity Detection (VAD) preprocessing step to detect the irrelevant non-speech frames.

x-vector 系统中的统计池化层,通过在时间上汇集隐藏节点激活的简单统计量,把变长输入投影成定长表示。[8, 9] 的作者为这一池化层引入了时序自注意力,使网络可以只关注它认为重要的帧。这也可以理解为一种语音活动检测(VAD)预处理,用来找出无关的非语音帧。

In this work we propose further architectural enhancements to the TDNN architecture and statistics pooling layer. We introduce additional skip connections to propagate and aggregate channels throughout the system. Channel attention that uses a global context is incorporated in the frame layers and statistics pooling layer to improve the results even further.

本文对 TDNN 架构和统计池化层提出进一步的结构改进。我们引入额外的跳跃连接,以便在整个系统中传播和聚合通道。帧层和统计池化层还加入了使用全局上下文的通道注意力,以进一步提高效果。

The paper is organized as follows: Section 2 will describe the current state-of-the-art speaker recognition systems which will be used as a baseline. Section 3 will explain and motivate the novel components of our proposed architecture. Section 4 will explain our experimental setup to test the impact of the individual components in our architecture on the popular VoxCeleb datasets [10, 11, 12]. We discuss the results of these experiments in Section 5. In addition, a comparison between popular state-of-the-art baseline systems will be provided. Section 6 will conclude with a brief overview of our findings.

全文安排如下:第 2 节介绍将作为基线的、当时最先进的说话人识别系统。第 3 节说明所提架构中各新部件的设计及其动机。第 4 节给出实验设置,用于在常用的 VoxCeleb 数据集 [10, 11, 12] 上检验各部件的作用。第 5 节讨论这些实验结果,并与当时流行的先进基线系统进行比较。第 6 节以简要归纳收尾。

  1. DNN speaker recognition systems
  1. 基于 DNN 的说话人识别系统

Two types of DNN-based speaker recognition architectures will serve as strong baselines to measure the impact of our proposed architecture: an x-vector and a ResNet based system which both currently provide state-of-the-art performance on speaker verification tasks such as VoxSRC [12].

我们将用两类基于 DNN 的说话人识别架构作为强基线,来衡量所提架构的效果:一类是 x-vector,一类是基于 ResNet 的系统。二者当时都在 VoxSRC [12] 这类说话人验证任务上提供最先进的表现。

2.1. Extended-TDNN x-vector

2.1. 扩展 TDNN x-vector

The first baseline system is the Extended TDNN x-vector architecture [2, 3, 4] and improves upon the original x-vector system introduced in [1]. The initial frame layers consist of 1-dimensional dilated convolutional layers interleaved with dense layers. Every filter has access to all the features of the previous layer or input layer. The task of the dilated convolutional layers is to gradually build up the temporal context. Residual connections are introduced in all frame-level layers. The frame layers are followed by an attentive statistics pooling layer that calculates the mean and standard deviations of the final frame-level features. The attention system [8] allows the model to select the frames it deems relevant. After the statistics pooling, two fully-connected layers are introduced with the first one acting as a bottleneck layer to generate the low-dimensional speaker characterizing embedding.

第一条基线是扩展 TDNN x-vector 架构 [2, 3, 4],它改进了 [1] 中提出的原始 x-vector 系统。靠前的帧层由一维膨胀卷积层与全连接层交错组成。每个滤波器都能用到上一层或输入层的全部特征。膨胀卷积层的任务,是逐步建立起时间上下文。所有帧级层都加入了残差连接。帧层之后是注意力统计池化层,计算最终帧级特征的均值和标准差。注意力机制 [8] 让模型能够选出它认为相关的帧。统计池化之后接两个全连接层,其中第一层作为瓶颈,用来生成低维的说话人表征嵌入。

2.2. ResNet-based r-vector

2.2. 基于 ResNet 的 r-vector

The second baseline system is the r-vector system proposed in [4]. It is based on the ResNet18 and ResNet34 implementations of the successful ResNet architecture [7]. The convolutional frame layers of this network process the features as a 2-dimensional signal before collecting the mean and standard deviation statistics in the pooling layer. See [4] for more details about the topology.

第二条基线是 [4] 提出的 r-vector 系统。它基于成功的 ResNet 架构 [7] 的 ResNet18 和 ResNet34 实现。该网络的卷积帧层把特征当作二维信号处理,再在池化层中收集均值和标准差统计量。拓扑细节见 [4]。

  1. Proposed ECAPA-TDNN architecture
  1. 所提出的 ECAPA-TDNN 架构

In this section, we examine some of the limitations of the x-vector architecture and incorporate potential solutions in our ECAPA-TDNN architecture. The following subsections will focus on frame-level and pooling-level enhancements. An overview of the complete architecture is given by Figure 2. BN stands for Batch Normalization [13] and the non-linearities are Rectified Linear Units (ReLU) unless specified otherwise.

本节先讨论 x-vector 架构的一些局限,再把可能的解决办法写进我们的 ECAPA-TDNN。下面各小节分别从帧级和池化级改进入手。完整架构总览见图 2。BN 表示批归一化 [13];除非另有说明,非线性均为修正线性单元(ReLU)。

3.1. Channel- and context-dependent statistics pooling

3.1. 通道相关且依赖上下文的统计池化

In recent x-vector architectures, soft self-attention is used for calculating weighted statistics in the temporal pooling layer [8]. Success with multi-headed attention has shown that certain speaker properties can be extracted on different sets of frames [9]. Due to these results, we argue that it might be beneficial to extend this temporal attention mechanism even further to the channel dimension. This enables the network to focus more on speaker characteristics that do not activate on identical or similar time instances, e.g. speaker-specific properties of vowels versus speaker-specific properties of consonants.

在近期的 x-vector 架构中,时间池化层用软自注意力来计算加权统计量 [8]。多头注意力上的成功表明,某些说话人属性可以在不同的帧集合上提取 [9]。据此,我们认为把这一时序注意力再扩展到通道维可能是有益的。这样,网络就能更多地关注那些并非在相同或相近时刻被激活的说话人特征,例如元音上的说话人特性与辅音上的说话人特性。

We implement the attention mechanism as described in [8] and adapt it to be channel-dependent:

我们按 [8] 实现注意力机制,并把它改成通道相关的:

et,c=vcTf(Wht+b)+kc,(1)e_{t,c} = \boldsymbol{v}_c^{T} f(\boldsymbol{W}\boldsymbol{h}_t + \boldsymbol{b}) + k_c, \tag{1}et,c​=vcT​f(Wht​+b)+kc​,(1)

where ht\boldsymbol{h}_tht​ are the activations of the last frame layer at time step ttt. The parameters W∈RR×C\boldsymbol{W} \in \mathbb{R}^{R \times C}W∈RR×C and b∈RR×1\boldsymbol{b} \in \mathbb{R}^{R \times 1}b∈RR×1 project the information for self-attention into a smaller RRR-dimensional representation that is shared across all CCC channels to reduce the parameter count and risk of overfitting. After a non-linearity f(⋅)f(\cdot)f(⋅) this information is transformed to a channel-dependent self-attention score through a linear layer with weights vc∈RR×1\boldsymbol{v}_c \in \mathbb{R}^{R \times 1}vc​∈RR×1 and bias kck_ckc​. This scalar score et,ce_{t,c}et,c​ is then normalized over all frames by applying the softmax function channel-wise across time:

其中 ht\boldsymbol{h}_tht​ 是最后一帧层在时间步 ttt 的激活。参数 W∈RR×C\boldsymbol{W} \in \mathbb{R}^{R \times C}W∈RR×C 和 b∈RR×1\boldsymbol{b} \in \mathbb{R}^{R \times 1}b∈RR×1 把用于自注意力的信息投影到更小的 RRR 维表示上;该表示在全部 CCC 个通道之间共享,以减少参数量并降低过拟合风险。经过非线性 f(⋅)f(\cdot)f(⋅) 之后,再通过带权重 vc∈RR×1\boldsymbol{v}_c \in \mathbb{R}^{R \times 1}vc​∈RR×1 和偏置 kck_ckc​ 的线性层,把这些信息变成通道相关的自注意力分数。随后,对这一标量分数 et,ce_{t,c}et,c​ 在全部帧上按通道做 softmax,完成归一化:

αt,c=exp⁡(et,c)∑τ=1Texp⁡(eτ,c).(2)\alpha_{t,c} = \frac{\exp(e_{t,c})}{\sum_{\tau=1}^{T} \exp(e_{\tau,c})}. \tag{2}αt,c​=∑τ=1T​exp(eτ,c​)exp(et,c​)​.(2)

The self-attention score αt,c\alpha_{t,c}αt,c​ represents the importance of each frame given the channel and is used to calculate the weighted statistics of channel ccc. For each utterance the channel component μ~c\tilde{\mu}_cμ~​c​ of the weighted mean vector μ~\tilde{\boldsymbol{\mu}}μ~​ is estimated as:

自注意力分数 αt,c\alpha_{t,c}αt,c​ 表示在给定通道下各帧的重要性,并用来计算通道 ccc 的加权统计量。对每段话语,加权均值向量 μ~\tilde{\boldsymbol{\mu}}μ~​ 的通道分量 μ~c\tilde{\mu}_cμ~​c​ 估计为:

μ~c=∑t=1Tαt,cht,c.(3)\tilde{\mu}_c = \sum_{t=1}^{T} \alpha_{t,c} h_{t,c}. \tag{3}μ~​c​=t=1∑T​αt,c​ht,c​.(3)

The channel component σ~c\tilde{\sigma}_cσ~c​ of the weighted standard deviation vector σ~\tilde{\boldsymbol{\sigma}}σ~ is constructed as follows:

加权标准差向量 σ~\tilde{\boldsymbol{\sigma}}σ~ 的通道分量 σ~c\tilde{\sigma}_cσ~c​ 按如下方式构造:

σ~c=∑t=1Tαt,cht,c2−μ~c2.(4)\tilde{\sigma}_c = \sqrt{\sum_{t=1}^{T} \alpha_{t,c} h_{t,c}^{2} - \tilde{\mu}_c^{2}}. \tag{4}σ~c​=t=1∑T​αt,c​ht,c2​−μ~​c2​​.(4)

The final output of the pooling layer is given by concatenating the vectors of the weighted mean μ~\tilde{\boldsymbol{\mu}}μ~​ and weighted standard deviation σ~\tilde{\boldsymbol{\sigma}}σ~.

池化层的最终输出,由加权均值 μ~\tilde{\boldsymbol{\mu}}μ~​ 与加权标准差 σ~\tilde{\boldsymbol{\sigma}}σ~ 两个向量拼接得到。

Furthermore, we expand the temporal context of the pooling layer by allowing the self-attention to look at global properties of the utterance. We concatenate the local input ht\boldsymbol{h}_tht​ in (1) with the global non-weighted mean and standard deviation of ht\boldsymbol{h}_tht​ across the time domain. This context vector should allow the attention mechanism to adapt itself to global properties of the utterance such as noise or recording conditions.

此外,我们让自注意力看到整段话语的全局属性,从而扩展池化层的时间上下文。具体做法是,把 (1) 中的局部输入 ht\boldsymbol{h}_tht​,与 ht\boldsymbol{h}_tht​ 在时间维上的全局(非加权)均值和标准差拼接在一起。这一上下文向量应能让注意力机制适应噪声或录音条件等整段话语的全局特性。

3.2. 1-Dimensional Squeeze-Excitation Res2Blocks

3.2. 一维 Squeeze-Excitation Res2Block

The temporal context of frame layers in the original x-vector system is limited to 15 frames. As the network apparently benefits from a wider temporal context [2, 4, 3] we argue it could be beneficial to rescale the frame-level features given global properties of the recording, similar to the global context in the attention module described above. For this purpose we introduce 1-dimensional Squeeze-Excitation (SE) blocks, as this computer vision approach to model global channel interdependencies has been proved successful [14, 15].

原始 x-vector 系统中,帧层的时间上下文限于 15 帧。网络显然能从更宽的时间上下文中获益 [2, 4, 3],因此我们认为,按整段录音的全局属性重新缩放帧级特征可能是有益的,这与上一小节注意力模块中的全局上下文类似。为此我们引入一维 Squeeze-Excitation(SE)块,因为这种用于建模全局通道相互依赖的计算机视觉方法已被证明有效 [14, 15]。

The first component of an SE-block is the squeeze operation which generates a descriptor for each channel. The squeeze operation simply consists of calculating the mean vector z\boldsymbol{z}z of the frame-level features across the time domain:

SE 块的第一个部件是 squeeze 操作,它为每个通道生成一个描述子。该操作很直接,就是在时间维上计算帧级特征的均值向量 z\boldsymbol{z}z:

z=1T∑t=1Tht.(5)\boldsymbol{z} = \frac{1}{T} \sum_{t=1}^{T} \boldsymbol{h}_t. \tag{5}z=T1​t=1∑T​ht​.(5)

The descriptors in oldsymbol{z} are then used in the excitation operation to calculate a weight for each channel. We define the subsequent excitation operation as:

随后,用 z\boldsymbol{z}z 中的描述子做 excitation 操作,为每个通道计算一个权重。我们把接下来的 excitation 定义为:

s=σ(W2f(W1z+b1)+b2)(6)\boldsymbol{s} = \sigma(\boldsymbol{W}_{2} f(\boldsymbol{W}_{1} \boldsymbol{z} + \boldsymbol{b}_{1}) + \boldsymbol{b}_{2}) \tag{6}s=σ(W2​f(W1​z+b1​)+b2​)(6)

with σ(⋅)\sigma(\cdot)σ(⋅) denoting the sigmoid function, f(⋅)f(\cdot)f(⋅) a non-linearity, W1∈RR×C\boldsymbol{W}_{1} \in \mathbb{R}^{R \times C}W1​∈RR×C and W2∈RC×R\boldsymbol{W}_{2} \in \mathbb{R}^{C \times R}W2​∈RC×R. This operation acts as a bottleneck layer with CCC and RRR referring to the number of input channels and reduced dimensionality respectively. The resulting vector s\boldsymbol{s}s contains weights scs_csc​ between zero and one, which are applied to the original input through channel-wise multiplication

其中 σ(⋅)\sigma(\cdot)σ(⋅) 表示 sigmoid 函数,f(⋅)f(\cdot)f(⋅) 为非线性,W1∈RR×C\boldsymbol{W}_{1} \in \mathbb{R}^{R \times C}W1​∈RR×C,W2∈RC×R\boldsymbol{W}_{2} \in \mathbb{R}^{C \times R}W2​∈RC×R。该操作相当于一个瓶颈层,CCC 和 RRR 分别对应输入通道数和降维后的维数。得到的向量 s\boldsymbol{s}s 中,各权重 scs_csc​ 介于 0 和 1 之间,再通过按通道相乘作用到原始输入上

h~c=schc(7)\tilde{\boldsymbol{h}}_{c} = s_{c} \boldsymbol{h}_{c} \tag{7}h~c​=sc​hc​(7)

The 1-dimensional SE-block can be integrated in the x-vector architecture in various ways, with using them after each dilated convolution being the most straightforward one. However, we want to combine them with the benefits of residual connections [7]. Simultaneously, we do not want to increase the total amount of parameters too much compared to the baseline systems. The SE-Res2Block shown in Figure 1 incorporates the requirements mentioned above. We contain the dilated convolutions with a preceding and succeeding dense layer with a context of 1 frame. The first dense layer can be used to reduce the feature dimension, while the second dense layer restores the number of features to the original dimension. This is followed by an SE-block to scale each channel. The whole unit is covered by a skip connection.

一维 SE 块可以用多种方式接入 x-vector 架构,最直接的做法是放在每次膨胀卷积之后。不过,我们还希望同时保留残差连接的好处 [7],又不希望相对基线系统增加太多参数。图 1 所示的 SE-Res2Block 把上述要求结合在一起。我们在膨胀卷积前后各加一个上下文为 1 帧的全连接层:第一层用来降低特征维数,第二层再把特征数恢复到原来的维数。其后接一个 SE 块,用于缩放各通道。整个单元再由一条跳跃连接包起来。

The use of these traditional ResBlocks makes it easy to incorporate advancements concerning this popular computer vision architecture. The recent Res2Net module [16], for example, enhances the central convolutional layer so that it can process multi-scale features by constructing hierarchical residual-like connections within. The integration of this module improves performance, while significantly reducing the number of model parameters.

采用这类传统的 ResBlock,也便于把这一流行计算机视觉架构上的新进展吸收进来。例如最近的 Res2Net 模块 [16],通过在内部构造层级化的类残差连接,增强了中间的卷积层,使之能够处理多尺度特征。接入该模块后,性能得到提升,同时显著减少了模型参数量。

SE-Res2Block of the ECAPA-TDNN architecture

Figure 1: The SE-Res2Block of the ECAPA-TDNN architecture. The standard Conv1D layers have a kernel size of 1. The central Res2Net [16] Conv1D with scale dimension s=8s = 8s=8 expands the temporal context through kernel size kkk and dilation spacing ddd.

图 1:ECAPA-TDNN 架构中的 SE-Res2Block。普通 Conv1D 层的卷积核大小为 1。中间的 Res2Net [16] Conv1D 以尺度维数 s=8s = 8s=8,通过卷积核大小 kkk 和膨胀间隔 ddd 扩展时间上下文。

3.3. Multi-layer feature aggregation and summation

3.3. 多层特征聚合与求和

The original x-vector system only uses the feature map of the last frame-layer for calculating the pooled statistics. Given the hierarchical nature of a TDNN, these deeper level features are the most complex ones and should be strongly correlated with the speaker identities. However, due to evidence in [17, 18] we argue that the more shallow feature maps can also contribute towards more robust speaker embeddings. For each frame, our proposed system concatenates the output feature maps of all the SE-Res2Blocks. After this Multi-layer Feature Aggregation (MFA), a dense layer processes the concatenated information to generate the features for the attentive statistics pooling.

原始 x-vector 系统在计算池化统计量时,只用最后一帧层的特征图。就 TDNN 的层级结构而言,这些更深的特征最复杂,也应当与说话人身份有很强的相关。不过,根据 [17, 18] 中的证据,我们认为更浅的特征图同样有助于得到更稳健的说话人嵌入。对每一帧,所提系统把全部 SE-Res2Block 的输出特征图拼接起来。经过这种多层特征聚合(MFA)之后,再用一个全连接层处理拼接结果,生成供注意力统计池化使用的特征。

Another, complementary way to exploit multi-layer information is to use the output of all preceding SE-Res2Blocks and initial convolutional layer as input for each frame layer block [17, 19]. We implement this by defining the residual connection in each SE-Res2Block as the sum of the outputs of all the previous blocks. We opt for a summation of the feature maps instead of concatenation to restrain the model parameter count. The final architecture without the summed residual connections is shown in Figure 2.

利用多层信息的另一种互补做法,是把此前所有 SE-Res2Block 以及最初卷积层的输出,都作为每个帧层块的输入 [17, 19]。我们的实现是:把每个 SE-Res2Block 中的残差连接,定义为前面所有块输出之和。为了控制模型参数量,我们选择对特征图求和,而不是拼接。不含这种求和残差连接的最终架构见图 2。

  1. Experimental setup
  1. 实验设置

4.1. Training the speaker embedding extractors

4.1. 训练说话人嵌入提取器

We apply the fixed-condition VoxSRC 2019 training restrictions [12] and only use the development part of the VoxCeleb2 dataset [11] with 5994 speakers as training data. A small subset of about 2% of the data is reserved as a validation set for hyperparameter optimization. It is a well known fact that neural networks benefit from data augmentation which generates extra training samples. We generate a total of 6 extra samples for each utterance. The first set of augmentations follow the Kaldi recipe [2] in combination with the publicly available MUSAN dataset (babble, noise) [20] and the RIR dataset (reverb) provided in [21]. The remaining three augmentations are generated with the open-source SoX (tempo up, tempo down) and FFmpeg (alternating opus or aac compression) libraries.

我们遵循 VoxSRC 2019 的固定训练条件 [12],只用 VoxCeleb2 数据集 [11] 开发集中 5994 位说话人的数据来训练。另外留出约 2% 的一小部分作为验证集,用于超参数选择。神经网络能从生成额外训练样本的数据增强中获益,这已是常识。我们对每段话语共生成 6 个额外样本。第一组增强沿用 Kaldi 配方 [2],并结合公开的 MUSAN 数据集(嘈杂人声、噪声)[20] 以及 [21] 提供的 RIR 数据集(混响)。其余三种增强用开源的 SoX(加速、减速)和 FFmpeg(交替使用 opus 或 aac 压缩)库生成。

Network topology of the ECAPA-TDNN

Figure 2: Network topology of the ECAPA-TDNN. We denote kkk for kernel size and ddd for dilation spacing of the Conv1D layers or SE-Res2Blocks. CCC and TTT correspond to the channel and temporal dimension of the intermediate feature-maps respectively. SSS is the number of training speakers.

图 2:ECAPA-TDNN 的网络拓扑。我们用 kkk 表示 Conv1D 层或 SE-Res2Block 的卷积核大小,用 ddd 表示膨胀间隔。CCC 和 TTT 分别对应中间特征图的通道维和时间维。SSS 是训练说话人的数量。

The input features are 80-dimensional MFCCs from a 25 ms window with a 10 ms frame shift. Two second random crops of the MFCCs feature vectors are normalized through cepstral mean subtraction and no voice activity detection is applied. As a final augmentation step, we apply SpecAugment [22] on the log mel spectrogram of the samples. The algorithm randomly masks 0 to 5 frames in the time domain and 0 to 10 channels in the frequency domain. All models are trained with a cyclical learning rate varying between 1e-8 and 1e-3 using the triangular2 policy as described in [23] in conjunction with the Adam optimizer [24]. The duration of one cycle is set to 130k iterations. All systems are trained using AAM-softmax [6, 25] with a margin of 0.2 and softmax prescaling of 30 for 4 cycles. To prevent overfitting, we apply a weight decay on all weights in the model of 2e-5, except for the AAM-softmax weights, which uses 2e-4. The mini-batch size for training is 128.

输入特征是 80 维 MFCC,窗长 25 ms,帧移 10 ms。我们对 MFCC 特征向量做两秒的随机裁剪,并用倒谱均值减做归一化,不使用语音活动检测。作为最后一步增强,我们在样本的对数梅尔谱上使用 SpecAugment [22]。该算法在时间维随机掩盖 0 到 5 帧,在频率维随机掩盖 0 到 10 个通道。所有模型都用周期学习率训练,学习率在 1e-8 与 1e-3 之间变化,采用 [23] 中的 triangular2 策略,并配合 Adam 优化器 [24]。一个周期的长度设为 13 万次迭代。所有系统都用 AAM-softmax [6, 25] 训练 4 个周期,间隔为 0.2,softmax 预缩放为 30。为防止过拟合,模型中全部权重的权重衰减取 2e-5,AAM-softmax 的权重除外,后者取 2e-4。训练的小批量大小为 128。

We study two setups of the proposed ECAPA-TDNN architecture with either 512 or 1024 channels in the convolutional frame layers. The dimension of the bottleneck in the SE-Block and the attention module is set to 128. The scale dimension sss in the Res2Block [16] is set to 8. The number of nodes in the final fully-connected layer is 192. The performance of this system will be compared to the baselines described in Section 2.

我们研究所提 ECAPA-TDNN 架构的两种设置,卷积帧层的通道数分别为 512 或 1024。SE 块和注意力模块中的瓶颈维数设为 128。Res2Block [16] 中的尺度维数 sss 设为 8。最后一层全连接的节点数为 192。该系统的表现将与第 2 节中的基线进行比较。

4.2. Speaker verification

4.2. 说话人验证

Speaker embeddings are extracted from the final fully-connected layer for all systems. Trial scores are produced using the cosine distance between embeddings. Subsequently, all the scores are normalized using adaptive s-norm [26, 27]. The imposter cohort consists of the speaker-wise averages of the length-normalized embeddings of all training utterances. The size of the imposter cohort was set to 1000 for the VoxCeleb test sets and to a more robust value of 50 for the cross-dataset VoxSRC 2019 evaluation.

所有系统都从最后的全连接层提取说话人嵌入。试验分数由嵌入之间的余弦距离给出。随后,全部分数用自适应 s-norm [26, 27] 做归一化。冒名者队列由全部训练话语在长度归一化后、按说话人取平均的嵌入组成。对 VoxCeleb 测试集,冒名者队列大小设为 1000;对跨数据集的 VoxSRC 2019 评测,则取更稳健的 50。

4.3. Evaluation protocol

4.3. 评测协议

The system is evaluated on the popular VoxCeleb1 test sets [10] and VoxSRC 2019 evaluation set [12]. Performance will be measured by providing the Equal Error Rate (EER) and the minimum normalized detection cost MinDCF with Ptarget=10−2P_{target} = 10^{-2}Ptarget​=10−2 and CFA=CMiss=1C_{FA} = C_{Miss} = 1CFA​=CMiss​=1. A concise ablation study is used to gain a deeper understanding how each of the proposed improvements affects the performance.

系统在常用的 VoxCeleb1 测试集 [10] 和 VoxSRC 2019 评测集 [12] 上评估。性能用等错误率(EER)以及最小归一化检测代价 MinDCF 来衡量,其中 Ptarget=10−2P_{target} = 10^{-2}Ptarget​=10−2,CFA=CMiss=1C_{FA} = C_{Miss} = 1CFA​=CMiss​=1。我们还做了一项简明的消融研究,以便更清楚地看出各项改进分别如何影响性能。

  1. Results
  1. 结果

A performance overview of the baseline systems described in Section 2 and our proposed ECAPA-TDNN system is given in Table 1, together with the number of model parameters in the embedding extractor. We implement two setups with the number of filters CCC in the convolutional layers either set to 512 or 1024. Our proposed architecture significantly outperforms all baselines while using fewer model parameters. The larger ECAPA-TDNN system gives an average relative improvement of 18.7% in EER and 12.5% in MinDCF over the best scoring baseline for each test set. We note that the performance of the baselines supersedes the numbers reported in [3, 4] in most cases. We continue with an ablation study of the individual components introduced in Section 3. An overview of these results is given in Table 2.

表 1 给出了第 2 节中各基线系统以及所提 ECAPA-TDNN 系统的性能总览,并同时列出嵌入提取器的模型参数量。我们实现了两种设置,卷积层的滤波器数 CCC 分别取 512 或 1024。所提架构在使用更少模型参数的同时,显著超过了全部基线。更大的 ECAPA-TDNN 系统,相对各测试集上得分最好的基线,EER 平均相对提升 18.7%,MinDCF 平均相对提升 12.5%。我们还注意到,这些基线的表现在多数情况下已经优于 [3, 4] 中报告的数字。接下来我们对第 3 节引入的各个部件做消融研究,结果总览见表 2。

Table 1: EER and MinDCF performance of all systems on the standard VoxCeleb1 and VoxSRC 2019 test sets.

Architecture # Params VoxCeleb1 EER(%) VoxCeleb1 MinDCF VoxCeleb1-E EER(%) VoxCeleb1-E MinDCF VoxCeleb1-H EER(%) VoxCeleb1-H MinDCF VoxSRC19 EER(%)
E-TDNN 6.8M 1.49 0.1604 1.61 0.1712 2.69 0.2419 1.81
E-TDNN (large) 20.4M 1.26 0.1399 1.37 0.1487 2.35 0.2153 1.61
ResNet18 13.8M 1.47 0.1772 1.60 0.1789 2.88 0.2672 1.97
ResNet34 23.9M 1.19 0.1592 1.33 0.1560 2.46 0.2288 1.57
ECAPA-TDNN (C=512) 6.2M 1.01 0.1274 1.24 0.1418 2.32 0.2181 1.32
ECAPA-TDNN (C=1024) 14.7M 0.87 0.1066 1.12 0.1318 2.12 0.2101 1.22

表 1:各系统在标准 VoxCeleb1 和 VoxSRC 2019 测试集上的 EER 与 MinDCF。

架构 参数量 VoxCeleb1 EER(%) VoxCeleb1 MinDCF VoxCeleb1-E EER(%) VoxCeleb1-E MinDCF VoxCeleb1-H EER(%) VoxCeleb1-H MinDCF VoxSRC19 EER(%)
E-TDNN 6.8M 1.49 0.1604 1.61 0.1712 2.69 0.2419 1.81
E-TDNN (large) 20.4M 1.26 0.1399 1.37 0.1487 2.35 0.2153 1.61
ResNet18 13.8M 1.47 0.1772 1.60 0.1789 2.88 0.2672 1.97
ResNet34 23.9M 1.19 0.1592 1.33 0.1560 2.46 0.2288 1.57
ECAPA-TDNN (C=512) 6.2M 1.01 0.1274 1.24 0.1418 2.32 0.2181 1.32
ECAPA-TDNN (C=1024) 14.7M 0.87 0.1066 1.12 0.1318 2.12 0.2101 1.22

Table 2: Ablation study of the ECAPA-TDNN architecture.

Systems EER(%) MinDCF
ECAPA-TDNN (C=512) 1.01 0.1274
A.1 Attentive Statistics [8] 1.12 0.1316
A.2 Channel Att. w/o Context 1.03 0.1288
B.1 No SE-Block 1.27 0.1446
B.2 No Res2Net-Block 1.07 0.1316
C.1 No MFA 1.10 0.1311
C.2 No Res. Connections 1.08 0.1310
C.3 No Sum Res. Connections 1.08 0.1217

表 2:ECAPA-TDNN 架构的消融研究。

系统 EER(%) MinDCF
ECAPA-TDNN (C=512) 1.01 0.1274
A.1 注意力统计 [8] 1.12 0.1316
A.2 通道注意力(无上下文) 1.03 0.1288
B.1 无 SE-Block 1.27 0.1446
B.2 无 Res2Net-Block 1.07 0.1316
C.1 无 MFA 1.10 0.1311
C.2 无残差连接 1.08 0.1310
C.3 无求和残差连接 1.08 0.1217

To measure the impact of our proposed attention module, we run an experiment A.1, that uses the attention module from [8]. We also run a separate experiment A.2 that does not supply the context vector to the proposed attention. The channel- and context-dependent statistics pooling system improves the EER and MinDCF metric with 9.8% and 3.2%, respectively. This confirms the benefits of applying different temporal attention to each channel. Addition of the context vector results in very small performance gains with the system relatively improving about 1.9% in EER and 1.1% in MinDCF. Nonetheless, this strengthens our belief that a TDNN-based architecture should try to exploit global context information.

为衡量所提注意力模块的作用,我们做了实验 A.1,改用 [8] 中的注意力模块。另外还做了实验 A.2,不向所提注意力提供上下文向量。通道相关且依赖上下文的统计池化,使 EER 和 MinDCF 分别相对改进 9.8% 和 3.2%。这证实了给每个通道使用不同时序注意力是有益的。加入上下文向量带来的收益很小,系统在 EER 和 MinDCF 上分别相对改进约 1.9% 和 1.1%。尽管如此,这仍加强了我们的看法:基于 TDNN 的架构应当尝试利用全局上下文信息。

This intuition is confirmed with experiment B.1 that clearly shows the importance of the SE-blocks described in Section 3.2. Incorporating the SE-modules in the Res2Blocks results in relative improvements of 20.5% in EER and 11.9% in the MinDCF metric. This indicates that the limited temporal context of the frame-level features is insufficient and should be complemented with global utterance-based information. In experiment B.2 we replaced the multi-scale features of the Res2Blocks with the standard central dilated 1D convolutional of the ResNet counterpart. Aside from a substantial 30% relative reduction in model parameters, the multi-scale Res2Net approach also leads towards a relative improvement of 5.6% in EER and 3.2% in MinDCF.

实验 B.1 印证了这一直觉,清楚显示了第 3.2 节所述 SE 块的重要性。把 SE 模块纳入 Res2Block 后,EER 相对改进 20.5%,MinDCF 相对改进 11.9%。这说明帧级特征有限的时间上下文并不够,应当用基于整段话语的全局信息来补充。在实验 B.2 中,我们把 Res2Block 的多尺度特征,换成了对应 ResNet 中那种普通的中间膨胀一维卷积。多尺度 Res2Net 方法不但使模型参数量相对减少约 30%,还带来 EER 相对改进 5.6%、MinDCF 相对改进 3.2%。

In experiment C.1, we only use the output of the final SE-Res2Block instead of aggregating the information of all SE-Res2Blocks. Aggregation of the outputs leads to relative improvements of 8.2% in EER and 2.8% in the MinDCF value. Removing all residual connections (experiment C.2) shows a similar rate of degradation. Replacing a standard ResNet skip connection in the SE-Res2Blocks by the sum of the outputs of all previous SE-Res2Blocks improves the EER with 6.5%, while slightly degrading the MinDCF score in experiment C.3. However, experiments during the recently held Short-duration Speaker Verification (SdSV) Challenge 2020 [28] convinced us to incorporate summed residuals in the final ECAPA-TDNN architecture. The strong results in this challenge show the architecture generalizes well to other domains [29].

在实验 C.1 中,我们只使用最后一个 SE-Res2Block 的输出,而不再聚合全部 SE-Res2Block 的信息。对输出做聚合后,EER 相对改进 8.2%,MinDCF 相对改进 2.8%。去掉全部残差连接(实验 C.2)会带来相近幅度的下降。在实验 C.3 中,把 SE-Res2Block 里普通的 ResNet 跳跃连接,换成此前所有 SE-Res2Block 输出之和,EER 改进 6.5%,MinDCF 则略有变差。不过,在当时刚举行的 2020 年短时说话人验证(SdSV)挑战 [28] 中的实验,说服我们在最终的 ECAPA-TDNN 架构里采用求和残差。该挑战中的出色结果说明,这一架构在其他域上也能较好地泛化 [29]。

  1. Conclusion
  1. 结论

In this paper we presented ECAPA-TDNN, a novel TDNN-based speaker embedding extractor for speaker verification. We built further upon the original x-vector architecture and put more Emphasis on Channel Attention, Propagation and Aggregation. The incorporation of Squeeze-Excitation blocks, multi-scale Res2Net features, extra skip connections and channel-dependent attentive statistics pooling, led to significant relative improvements of 19% in EER on average over strong baseline systems on the VoxCeleb and VoxSRC 2019 evaluation sets.

本文提出了 ECAPA-TDNN,一种用于说话人验证的、基于 TDNN 的新型说话人嵌入提取器。我们在原始 x-vector 架构之上继续改进,更强调通道注意力、传播与聚合。加入 Squeeze-Excitation 块、多尺度 Res2Net 特征、额外的跳跃连接,以及通道相关的注意力统计池化之后,在 VoxCeleb 和 VoxSRC 2019 评测集上,相对较强的基线系统,EER 平均相对改进了 19%。

Copyright © 2020 ISCA

http://dx.doi.org/10.21437/Interspeech.2020-2650

  1. References
  1. 参考文献

[1] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. ICASSP, 2018, pp. 5329–5333.

[2] D. Snyder, D. Garcia-Romero, G. Sell, A. McCree, D. Povey, and S. Khudanpur, “Speaker recognition for multi-speaker conversations using x-vectors,” in Proc. ICASSP, 2019, pp. 5796–5800.

[3] D. Garcia-Romero, A. McCree, D. Snyder, and G. Sell, “JHU-HLTCOE system for the VoxSRC speaker recognition challenge,” in Proc. ICASSP, 2020, pp. 7559–7563.

[4] H. Zeinali, S. Wang, A. Silnova, P. Matějka, and O. Plchot, “BUT system description to VoxCeleb speaker recognition challenge 2019,” 2019.

[5] S. Ioffe, “Probabilistic linear discriminant analysis,” in ECCV, 2006, pp. 531–542.

[6] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “ArcFace: Additive angular margin loss for deep face recognition,” in 2019 IEEE/CVF CVPR, 2019, pp. 4685–4694.

[7] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE/CVF CVPR, 2016, pp. 770–778.

[8] K. Okabe, T. Koshinaka, and K. Shinoda, “Attentive statistics pooling for deep speaker embedding,” in Proc. Interspeech, 2018, pp. 2252–2256.

[9] Y. Zhu, T. Ko, D. Snyder, B. K.-W. Mak, and D. Povey, “Self-attentive speaker embeddings for text-independent speaker verification,” in Proc. Interspeech, 2018, pp. 3573–3577.

[10] A. Nagrani, J. S. Chung, and A. Zisserman, “VoxCeleb: A large-scale speaker identification dataset,” in Proc. Interspeech, 2017, pp. 2616–2620.

[11] J. S. Chung, A. Nagrani, and A. Zisserman, “VoxCeleb2: Deep speaker recognition,” in Proc. Interspeech, 2018, pp. 1086–1090.

[12] J. S. Chung, A. Nagrani, E. Coto, W. Xie, M. McLaren, D. A. Reynolds, and A. Zisserman, “VoxSRC 2019: The first VoxCeleb speaker recognition challenge,” 2019.

[13] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. ICML, 2015, pp. 448–456.

[14] J. Hu, L. Shen, and G. Sun, “Squeeze-and-Excitation networks,” in Proc. IEEE/CVF CVPR, 2018, pp. 7132–7141.

[15] J. Zhou, T. Jiang, Z. Li, L. Li, and Q. Hong, “Deep speaker embedding extraction with channel-wise feature responses and additive supervision softmax loss function,” in Proc. Interspeech, 2019, pp. 2883–2887.

[16] S. Gao, M.-M. Cheng, K. Zhao, X. Zhang, M.-H. Yang, and P. H. S. Torr, “Res2Net: A new multi-scale backbone architecture,” IEEE TPAMI, 2019.

[17] J. Lee and J. Nam, “Multi-level and multi-scale feature aggregation using sample-level deep convolutional neural networks for music classification,” 2017.

[18] Z. Gao, Y. Song, I. McLoughlin, P. Li, Y. Jiang, and L.-R. Dai, “Improving Aggregation and Loss Function for Better Embedding Learning in End-to-End Speaker Verification System,” in Proc. Interspeech, 2019, pp. 361–365.

[19] D. Povey, G. Cheng, Y. Wang, K. Li, H. Xu, M. Yarmohammadi, and S. Khudanpur, “Semi-orthogonal low-rank matrix factorization for deep neural networks,” in Proc. Interspeech, 2018, pp. 3743–3747.

[20] D. Snyder, G. Chen, and D. Povey, “MUSAN: A music, speech, and noise corpus,” 2015.

[21] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in Proc. ICASSP, 2017, pp. 5220–5224.

[22] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” in Proc. Interspeech, 2019.

[23] L. N. Smith, “Cyclical learning rates for training neural networks,” in IEEE WACV, 2017, pp. 464–472.

[24] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Proc. ICLR, 2014.

[25] X. Xiang, S. Wang, H. Huang, Y. Qian, and K. Yu, “Margin matters: Towards more discriminative deep neural network embeddings for speaker recognition,” 2019.

[26] Z. N. Karam, W. M. Campbell, and N. Dehak, “Towards reduced false-alarms using cohorts,” in 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011, pp. 4512–4515.

[27] S. Cumani, P. Batzu, D. Colibro, C. Vair, P. Laface, and V. Vasilakakis, “Comparison of speaker recognition approaches for real applications,” in Proc. Interspeech, 2011, pp. 2365–2368.

[28] H. Zeinali, K. A. Lee, J. Alam, and L. Burget, “Short-duration speaker verification (SdSV) challenge 2020: the challenge evaluation plan,” 2019.

[29] J. Thienpondt, B. Desplanques, and K. Demuynck, “Cross-lingual speaker verification with domain-balanced hard prototype mining and language-dependent score normalization,” in Proc. Interspeech, 2020.