Conformer: Convolution-augmented Transformer for Speech Recognition

Conformer：用于语音识别的卷积增强 Transformer

Abstract

摘要

Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs). Transformer models are good at capturing content-based global interactions, while CNNs exploit local features effectively. In this work, we achieve the best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter-efficient way. To this regard, we propose the convolution-augmented transformer for speech recognition, named Conformer. Conformer significantly outperforms the previous Transformer and CNN based models achieving state-of-the-art accuracies. On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.

近来，基于 Transformer 与卷积神经网络（CNN）的模型在自动语音识别（ASR）上表现亮眼，超过了循环神经网络（RNN）。Transformer 擅长捕捉基于内容的全局交互，CNN 则有效利用局部特征。本文研究如何把卷积网络与 Transformer 结合起来，以参数高效的方式同时建模音频序列的局部与全局依赖，从而兼得两者之长。为此，我们提出面向语音识别的卷积增强 Transformer，命名为 Conformer。Conformer 显著优于此前的 Transformer 与 CNN 模型，达到当时最优精度。在广泛使用的 LibriSpeech 基准上，不加语言模型时 WER 为 2.1%/4.3%，加外部语言模型后为 1.9%/3.9%（test/testother）。我们还观察到仅 10M 参数的小模型也能达到有竞争力的 2.7%/6.3%。

Index Terms: speech recognition, attention, convolutional neural networks, transformer, end-to-end

索引词：语音识别、注意力、卷积神经网络、Transformer、端到端

1 Introduction

1 引言

End-to-end automatic speech recognition (ASR) systems based on neural networks have seen large improvements in recent years. Recurrent neural networks (RNNs) have been the de-facto choice for ASR [1, 2, 3, 4] as they can model the temporal dependencies in the audio sequences effectively [5]. Recently, the Transformer architecture based on self-attention [6, 7] has enjoyed widespread adoption for modeling sequences due to its ability to capture long distance interactions and the high training efficiency. Alternatively, convolutions have also been successful for ASR [8, 9, 10, 11, 12], which capture local context progressively via a local receptive field layer by layer.

基于神经网络的端到端自动语音识别（ASR）近年进步很大。循环神经网络（RNN）曾是 ASR 的默认选择 [1, 2, 3, 4]，因为它们能有效建模音频序列中的时间依赖 [5]。近来，基于自注意力的 Transformer 架构 [6, 7] 因能捕捉长距离交互且训练效率高，在序列建模中被广泛采用。另一方面，卷积在 ASR 中同样成功 [8, 9, 10, 11, 12]，它通过逐层局部感受野逐步捕获局部上下文。

However, models with self-attention or convolutions each has its limitations. While Transformers are good at modeling long-range global context, they are less capable to extract fine-grained local feature patterns. Convolution neural networks (CNNs), on the other hand, exploit local information and are used as the de-facto computational block in vision. They learn shared position-based kernels over a local window which maintain translation equivariance and are able to capture features like edges and shapes. One limitation of using local connectivity is that you need many more layers or parameters to capture global information. To combat this issue, contemporary work ContextNet [10] adopts the squeeze-and-excitation module [13] in each residual block to capture longer context. However, it is still limited in capturing dynamic global context as it only applies a global averaging over the entire sequence.

然而，仅用自注意力或仅用卷积的模型各有局限。Transformer 擅长建模长程全局上下文，却不太善于抽取细粒度局部特征模式。卷积神经网络（CNN）则利用局部信息，在视觉里几乎是默认计算块：它们在局部窗口上学习共享的、基于位置的核，保持平移等变性，并能捕获边缘与形状等特征。局部连接的一个限制是：要捕获全局信息，往往需要更多层或更多参数。为缓解这一问题，同期工作 ContextNet [10] 在每个残差块中采用 squeeze-and-excitation 模块 [13] 以捕获更长上下文。不过，由于它只对整段序列做全局平均，在捕获动态全局上下文方面仍然受限。

Recent works have shown that combining convolution and self-attention improves over using them individually [14]. Together, they are able to learn both position-wise local features, and use content-based global interactions. Concurrently, papers like [15, 16] have augmented self-attention with relative position based information that maintains equivariance. Wu et al. [17] proposed a multi-branch architecture with splitting the input into two branches: self-attention and convolution; and concatenating their outputs. Their work targeted mobile applications and showed improvements in machine translation tasks.

近期工作表明，把卷积与自注意力结合起来优于单独使用 [14]。二者一起既能学习按位置的局部特征，又能利用基于内容的全局交互。与此同时，[15, 16] 等论文用相对位置信息增强自注意力以保持等变性。Wu 等人 [17] 提出多分支结构：把输入拆成自注意力与卷积两支，再拼接输出。该工作面向移动端应用，并在机器翻译任务上展示了提升。

Figure 1: Conformer encoder model architecture. Conformer comprises of two macaron-like feed-forward layers with half-step residual connections sandwiching the multi-headed self-attention and convolution modules. This is followed by a post layernorm.

图 1：Conformer 编码器模型架构。 Conformer 由两个类 Macaron 的前馈层（半步残差连接）夹住多头自注意力与卷积模块，之后再接一层后置 LayerNorm。

Figure 2: Convolution module. The convolution module contains a pointwise convolution with an expansion factor of 2 projecting the number of channels with a GLU activation layer, followed by a 1-D Depthwise convolution. The 1-D depthwise conv is followed by a Batchnorm and then a swish activation layer.

图 2：卷积模块。 卷积模块包含扩张因子为 2 的逐点卷积，经 GLU 激活层扩展通道数，随后是一维深度卷积；一维深度卷积后再接 BatchNorm 与 Swish 激活层。

In this work, we study how to organically combine convolutions with self-attention in ASR models. We hypothesize that both global and local interactions are important for being parameter efficient. To achieve this, we propose a novel combination of self-attention and convolution will achieve the best of both worlds – self-attention learns the global interaction whilst the convolutions efficiently capture the relative-offset-based local correlations. Inspired by Wu et al. [17, 18], we introduce a novel combination of self-attention and convolution, sandwiched between a pair feed forward modules, as illustrated in Fig 1.

本文研究如何在 ASR 模型中有机地结合卷积与自注意力。我们假设全局与局部交互对参数效率都很重要。为此，我们提出一种新颖的自注意力与卷积组合，以期兼得两者之长——自注意力学习全局交互，卷积高效捕获基于相对偏移的局部相关。受 Wu 等人 [17, 18] 启发，我们引入把自注意力与卷积夹在一对前馈模块之间的新组合，如图 1 所示。

Our proposed model, named Conformer, achieves state-of-the-art results on LibriSpeech, outperforming the previous best published Transformer Transducer [7] by 15% relative improvement on the testother dataset with an external language model. We present three models based on model parameter limit constraints of 10M , 30M and 118M. Our 10M model shows an improvement when compared to similar sized contemporary work [10] with 2.7%/6.3% on test/testother datasets. Our medium 30M parameters-sized model already outperforms transformer transducer published in [7] which uses 139M model parameters. With the big 118M parameter model, we are able to achieve 2.1%/4.3% without using language models and 1.9%/3.9% with an external language model.

我们提出的模型名为 Conformer，在 LibriSpeech 上取得当时最优结果：相对此前最好的已发表 Transformer Transducer [7]，在加外部语言模型的 testother 上相对提升 15%。我们按 10M、30M、118M 的参数上限给出三档模型。10M 模型相对同规模同期工作 [10] 有提升，test/testother 为 2.7%/6.3%。中档 30M 参数模型已超过 [7] 中使用 139M 参数的 Transformer Transducer。大档 118M 参数模型在不加语言模型时达到 2.1%/4.3%，加外部语言模型后为 1.9%/3.9%。

We further carefully study the effects of the number of attention heads, convolution kernel sizes, activation functions, placement of feed-forward layers, and different strategies of adding convolution modules to a Transformer-based network, and shed light on how each contributes to the accuracy improvements.

我们还仔细研究了注意力头数、卷积核大小、激活函数、前馈层放置位置，以及向基于 Transformer 的网络加入卷积模块的不同策略，并阐明各自如何贡献精度提升。

2 Conformer Encoder

2 Conformer 编码器

Our audio encoder first processes the input with a convolution subsampling layer and then with a number of conformer blocks, as illustrated in Figure 1. The distinctive feature of our model is the use of Conformer blocks in the place of Transformer blocks as in [7, 19].

我们的音频编码器先用卷积下采样层处理输入，再经过若干 Conformer 块，如图 1 所示。模型的显著特征是用 Conformer 块取代 [7, 19] 中的 Transformer 块。

A conformer block is composed of four modules stacked together, i.e, a feed-forward module, a self-attention module, a convolution module, and a second feed-forward module in the end. Sections 2.1, 1, and 2.3 introduce the self-attention, convolution, and feed-forward modules, respectively. Finally, 2.4 describes how these sub blocks are combined.

一个 Conformer 块由四个模块依次堆叠而成：前馈模块、自注意力模块、卷积模块，以及末尾的第二个前馈模块。第 2.1、2.2 与 2.3 节分别介绍自注意力、卷积与前馈模块；最后第 2.4 节说明这些子块如何组合。

2.1 Multi-Headed Self-Attention Module

2.1 多头自注意力模块

We employ multi-headed self-attention (MHSA) while integrating an important technique from Transformer-XL [20], the relative sinusoidal positional encoding scheme. The relative positional encoding allows the self-attention module to generalize better on different input length and the resulting encoder is more robust to the variance of the utterance length. We use pre-norm residual units [21, 22] with dropout which helps training and regularizing deeper models. Figure 3 below illustrates the multi-headed self-attention block.

我们采用多头自注意力（MHSA），并融入 Transformer-XL [20] 的一项重要技术——相对正弦位置编码方案。相对位置编码使自注意力模块在不同输入长度上泛化更好，所得编码器对语句长度变化也更稳健。我们使用带 dropout 的 pre-norm 残差单元 [21, 22]，有助于训练并正则化更深的模型。下图 3 给出多头自注意力块示意。

Figure 3: Multi-Headed self-attention module. We use multi-headed self-attention with relative positional embedding in a pre-norm residual unit.

图 3：多头自注意力模块。 我们在 pre-norm 残差单元中使用带相对位置嵌入的多头自注意力。

2.2 Convolution Module

2.2 卷积模块

Inspired by [17], the convolution module starts with a gating mechanism [23]—a pointwise convolution and a gated linear unit (GLU). This is followed by a single 1-D depthwise convolution layer. Batchnorm is deployed just after the convolution to aid training deep models. Figure 2 illustrates the convolution block.

受 [17] 启发，卷积模块以门控机制开场 [23]——逐点卷积加门控线性单元（GLU），随后是单层一维深度卷积。卷积之后部署 BatchNorm，以辅助训练深层模型。图 2 给出卷积块示意。

2.3 Feed Forward Module

2.3 前馈模块

The Transformer architecture as proposed in [6] deploys a feed forward module after the MHSA layer and is composed of two linear transformations and a nonlinear activation in between. A residual connection is added over the feed-forward layers, followed by layer normalization. This structure is also adopted by Transformer ASR models [7, 24].

[6] 提出的 Transformer 架构在 MHSA 层之后部署前馈模块，由两次线性变换与中间的非线性激活组成。前馈层上加残差连接，再接层归一化。这一结构也被 Transformer ASR 模型采用 [7, 24]。

We follow pre-norm residual units [21, 22] and apply layer normalization within the residual unit and on the input before the first linear layer. We also apply Swish activation [25] and dropout, which helps regularizing the network. Figure 4 illustrates the Feed Forward (FFN) module.

我们遵循 pre-norm 残差单元 [21, 22]，在残差单元内、以及第一个线性层之前的输入上施加层归一化。我们还使用 Swish 激活 [25] 与 dropout，有助于正则化网络。图 4 给出前馈（FFN）模块示意。

Figure 4: Feed forward module. The first linear layer uses an expansion factor of 4 and the second linear layer projects it back to the model dimension. We use swish activation and a pre-norm residual units in feed forward module.

图 4：前馈模块。 第一个线性层使用扩张因子 4，第二个线性层投影回模型维度。前馈模块中使用 Swish 激活与 pre-norm 残差单元。

2.4 Conformer Block

2.4 Conformer 块

Our proposed Conformer block contains two Feed Forward modules sandwiching the Multi-Headed Self-Attention module and the Convolution module, as shown in Figure 1.

我们提出的 Conformer 块包含两个前馈模块，夹住多头自注意力模块与卷积模块，如图 1 所示。

This sandwich structure is inspired by Macaron-Net [18], which proposes replacing the original feed-forward layer in the Transformer block into two half-step feed-forward layers, one before the attention layer and one after. As in Macron-Net, we employ half-step residual weights in our feed-forward (FFN) modules. The second feed-forward module is followed by a final layernorm layer. Mathematically, this means, for input $x_{i}$ to a Conformer block $i$, the output $y_{i}$ of the block is:

这种三明治结构受 Macaron-Net [18] 启发：后者建议把 Transformer 块中原来的前馈层换成两个半步前馈层，一个在注意力之前、一个在之后。与 Macaron-Net 一样，我们在前馈（FFN）模块中采用半步残差权重。第二个前馈模块之后接最终的 LayerNorm 层。数学上，对 Conformer 块 $i$ 的输入 $x_{i}$，块输出 $y_{i}$ 为：

$$\begin{split}\tilde{x_{i}}&=x_{i}+\frac{1}{2}\mathrm{FFN}(x_{i})\
x^{\prime}{i}&=\tilde{x{i}}+\mathrm{MHSA}(\tilde{x_{i}})\
x^{\prime\prime}{i}&=x^{\prime}{i}+\mathrm{Conv}(x^{\prime}{i})\
y{i}&=\mathrm{Layernorm}(x^{\prime\prime}{i}+\frac{1}{2}\mathrm{FFN}(x^{\prime\prime}{i}))\
\end{split} \tag{1}$$

where FFN refers to the Feed forward module, MHSA refers to the Multi-Head Self-Attention module, and Conv refers to the Convolution module as described in the preceding sections.

其中 FFN 指前馈模块，MHSA 指多头自注意力模块，Conv 指前面各节描述的卷积模块。

Our ablation study discussed in Sec 3.4.3 compares the Macaron-style half-step FFNs with the vanilla FFN as used in previous works. We find that having two Macaron-net style feed-forward layers with half-step residual connections sandwiching the attention and convolution modules in between provides a significant improvement over having a single feed-forward module in our Conformer architecture.

第 3.4.3 节的消融把 Macaron 风格的半步 FFN 与此前工作中的普通 FFN 做了对比。我们发现：用两个带半步残差连接的类 Macaron-Net 前馈层夹住中间的注意力与卷积模块，相对在 Conformer 架构中只用单个前馈模块，有显著提升。

The combination of convolution and self-attention has been studied before and one can imagine many ways to achieve that. Different options of augmenting convolutions with self-attention are studied in Sec 3.4.2. We found that convolution module stacked after the self-attention module works best for speech recognition.

卷积与自注意力的组合此前已有研究，可以设想多种实现方式。第 3.4.2 节研究了用卷积增强自注意力的不同选项。我们发现：把卷积模块叠在自注意力模块之后，对语音识别效果最好。

3 Experiments

3 实验

3.1 Data

3.1 数据

We evaluate the proposed model on the LibriSpeech [26] dataset, which consists of 970 hours of labeled speech and an additional 800M word token text-only corpus for building language model. We extracted 80-channel filterbanks features computed from a 25ms window with a stride of 10ms. We use SpecAugment [27, 28] with mask parameter ($F=27$), and ten time masks with maximum time-mask ratio ($p_{S}=0.05$), where the maximum-size of the time mask is set to $p_{S}$ times the length of the utterance.

我们在 LibriSpeech [26] 数据集上评估所提模型；该数据集含 970 小时有标注语音，以及额外约 8 亿词元的纯文本语料用于构建语言模型。我们从 25ms 窗、10ms 步长提取 80 通道滤波器组特征。使用 SpecAugment [27, 28]，掩码参数 ($F=27$)，以及最多十条时间掩码、最大时间掩码比例 ($p_{S}=0.05$)，其中时间掩码的最大长度设为话语长度的 $p_{S}$ 倍。

3.2 Conformer Transducer

We identify three models, small, medium and large, with 10M, 30M, and 118M params, respectively, by sweeping different combinations of network depth, model dimensions, number of attention heads and choosing the best performing one within model parameter size constraints. We use a single-LSTM-layer decoder in all our models. Table  1 describes their architecture hyper-parameters.

我们通过扫描网络深度、模型维度、注意力头数等组合，并在参数量约束内选取表现最好者，确定三档模型：小、中、大，参数量分别为 10M、30M 与 118M。所有模型均使用单层 LSTM 解码器。表 1 给出其架构超参数。

For regularization, we apply dropout [29] in each residual unit of the conformer, i.e, to the output of each module, before it is added to the module input. We use a rate of $P_{drop}=0.1$. Variational noise [5, 30] is introduced to the model as a regularization. A $\ell_{2}$ regularization with $1e-6$ weight is also added to all the trainable weights in the network. We train the models with the Adam optimizer [31] with $\beta_{1}=0.9$, $\beta_{2}=0.98$ and $\epsilon=10^{-9}$ and a transformer learning rate schedule [6], with 10k warm-up steps and peak learning rate $0.05/\sqrt{\bf{d}}$ where $\bf{d}$ is the model dimension in conformer encoder.

正则化方面，我们在 Conformer 的每个残差单元中施加 dropout [29]，即在每个模块输出加回模块输入之前对其做 dropout，比率 $P_{drop}=0.1$。还引入变分噪声 [5, 30] 作为正则。对网络中全部可训练权重施加权重为 $1e-6$ 的 $\ell_{2}$ 正则。用 Adam 优化器 [31] 训练，取 eta_{1}=0.9、eta_{2}=0.98、$\epsilon=10^{-9}$，并采用 Transformer 学习率日程 [6]：1 万步预热，峰值学习率为 $0.05/\sqrt{\bf{d}}$，其中 $\bf{d}$ 为 Conformer 编码器的模型维度。

We use a 3-layer LSTM language model (LM) with width 4096 trained on the LibriSpeech langauge model corpus with the LibriSpeech960h transcripts added, tokenized with the 1k WPM built from LibriSpeech 960h. The LM has word-level perplexity 63.9 on the dev-set transcripts. The LM weight $\lambda$ for shallow fusion is tuned on the dev-set via grid search. All models are implemented with Lingvo toolkit [32].

我们使用宽度为 4096 的 3 层 LSTM 语言模型（LM），在 LibriSpeech 语言模型语料上训练，并加入 LibriSpeech 960h 转写；用由 LibriSpeech 960h 构建的 1k WPM 做分词。该 LM 在开发集转写上的词级困惑度为 63.9。浅融合的 LM 权重 $\lambda$ 在开发集上通过网格搜索调优。全部模型用 Lingvo 工具包 [32] 实现。

Model	Conformer (S)	Conformer (M)	Conformer (L)
Num Params (M)	10.3	30.7	118.8
Encoder Layers	16	16	17
Encoder Dim	144	256	512
Attention Heads	4	4	8
Conv Kernel Size	32	32	32
Decoder Layers	1	1	1
Decoder Dim	320	640	640

Table 1: Model hyper-parameters for Conformer S, M, and L models, found via sweeping different combinations and choosing the best performing models within the parameter limits.

模型	Conformer (S)	Conformer (M)	Conformer (L)
参数量 (M)	10.3	30.7	118.8
编码器层数	16	16	17
编码器维度	144	256	512
注意力头数	4	4	8
卷积核大小	32	32	32
解码器层数	1	1	1
解码器维度	320	640	640

表 1：Conformer S、M、L 模型的超参数；通过扫描不同组合并在参数上限内选取表现最好的模型得到。

3.3 Results on LibriSpeech

3.3 LibriSpeech 结果

Method	#Params (M)	WER Without LM	WER With LM
		testclean	testother	testclean	testother
Hybrid
Transformer [33]	-	-	-	2.26	4.85
CTC
QuartzNet  [9]	19	3.90	11.28	2.69	7.25
LAS
Transformer [34]	270	2.89	6.98	2.33	5.17
Transformer [19]	-	2.2	5.6	2.6	5.7
LSTM	360	2.6	6.0	2.2	5.2
Transducer
Transformer [7]	139	2.4	5.6	2.0	4.6
ContextNet(S) [10]	10.8	2.9	7.0	2.3	5.5
ContextNet(M) [10]	31.4	2.4	5.4	2.0	4.5
ContextNet(L) [10]	112.7	2.1	4.6	1.9	4.1
Conformer (Ours)
Conformer(S)	10.3	2.7	6.3	2.1	5.0
Conformer(M)	30.7	2.3	5.0	2.0	4.3
Conformer(L)	118.8	2.1	4.3	1.9	3.9

Table 2: Comparison of Conformer with recent published models. Our model shows improvements consistently over various model parameter size constraints. At 10.3M parameters, our model is 0.7% better on testother when compared to contemporary work, ContextNet(S) [10]. At 30.7M model parameters our model already significantly outperforms the previous published state of the art results of Transformer Transducer [7] with 139M parameters.

方法	#参数 (M)	无 LM 的 WER	有 LM 的 WER
		testclean	testother	testclean	testother
Hybrid
Transformer [33]	-	-	-	2.26	4.85
CTC
QuartzNet  [9]	19	3.90	11.28	2.69	7.25
LAS
Transformer [34]	270	2.89	6.98	2.33	5.17
Transformer [19]	-	2.2	5.6	2.6	5.7
LSTM	360	2.6	6.0	2.2	5.2
Transducer
Transformer [7]	139	2.4	5.6	2.0	4.6
ContextNet(S) [10]	10.8	2.9	7.0	2.3	5.5
ContextNet(M) [10]	31.4	2.4	5.4	2.0	4.5
ContextNet(L) [10]	112.7	2.1	4.6	1.9	4.1
Conformer（本文）
Conformer(S)	10.3	2.7	6.3	2.1	5.0
Conformer(M)	30.7	2.3	5.0	2.0	4.3
Conformer(L)	118.8	2.1	4.3	1.9	3.9

表 2：Conformer 与近期已发表模型的对比。在多种参数规模约束下，我们的模型均稳定改进。在 10.3M 参数时，相对同期工作 ContextNet(S) [10]，testother 上好 0.7%。在 30.7M 参数时，已显著超过此前发表的、使用 139M 参数的 Transformer Transducer [7] 最优结果。

Table 2 compares the (WER) result of our model on LibriSpeech test-clean/test-other with a few state-of-the-art models include: ContextNet [10], Transformer transducer [7], and QuartzNet [9]. All our evaluation results round up to 1 digit after decimal point.

表 2 把我们模型在 LibriSpeech test-clean/test-other 上的（WER）结果与若干当时最优模型对比，包括：ContextNet [10]、Transformer transducer [7] 与 QuartzNet [9]。所有评测结果保留小数点后 1 位。

Without a language model, the performance of our medium model already achieve competitive results of $2.3$/$5.0$ on test/testother outperforming the best known Transformer, LSTM based model, or a similar sized convolution model. With the language model added, our model achieves the lowest word error rate among all the existing models. This clearly demonstrates the effectiveness of combining Transformer and convolution in a single neural network.

不加语言模型时，中档模型已达到有竞争力的 $2.3$/$5.0$（test/testother），超过已知最好的基于 Transformer、LSTM 的模型，或同规模卷积模型。加入语言模型后，我们的模型在现有模型中取得最低词错误率。这清楚表明：在同一神经网络中结合 Transformer 与卷积是有效的。

3.4 Ablation Studies

3.4 消融实验

3.4.1 Conformer Block vs. Transformer Block

A Conformer block differs from a Transformer block in a number of ways, in particular, the inclusion of a convolution block and having a pair of FFNs surrounding the block in the Macaron-style. Below we study these effects of these differences by mutating a Conformer block towards a Transformer block, while keeping the total number of parameters unchanged. Table 3 shows the impact of each change to the Conformer block. Among all differences, convolution sub-block is the most important feature, while having a Macaron-style FFN pair is also more effective than a single FFN of the same number of parameters. Using swish activations led to faster convergence in the Conformer models.

3.4.1 Conformer 块对比 Transformer 块

Conformer 块与 Transformer 块有多处不同，尤其是加入了卷积块，并以 Macaron 风格用一对 FFN 包围该块。下面我们在总参数量不变的前提下，把 Conformer 块逐步改成 Transformer 块，以研究这些差异的影响。表 3 给出对 Conformer 块每一步改动的影响。在所有差异中，卷积子块是最重要的特征；具有 Macaron 风格的 FFN 对也比同等参数量的单个 FFN 更有效。使用 Swish 激活使 Conformer 模型收敛更快。

Model Architecture	dev clean	dev other	test clean	test other
Conformer Model	1.9	4.4	2.1	4.3
– SWISH + ReLU	1.9	4.4	2.0	4.5
– Convolution Block	2.1	4.8	2.1	4.9
– Macaron FFN	2.1	5.1	2.1	5.0
– Relative Pos. Emb.	2.3	5.8	2.4	5.6

Table 3: Disentangling Conformer. Starting from a Conformer block, we remove its features and move towards a vanilla Transformer block: (1) replacing SWISH with ReLU; (2) removing the convolution sub-block; (3) replacing the Macaron-style FFN pairs with a single FFN; (4) replacing self-attention with relative positional embedding [20] with a vanilla self-attention layer [6]. All ablation study results are evaluated without the external LM.

模型架构	dev clean	dev other	test clean	test other
Conformer 模型	1.9	4.4	2.1	4.3
– SWISH + ReLU	1.9	4.4	2.0	4.5
– 卷积块	2.1	4.8	2.1	4.9
– Macaron FFN	2.1	5.1	2.1	5.0
– 相对位置嵌入	2.3	5.8	2.4	5.6

表 3：拆解 Conformer。 从 Conformer 块出发，逐步去掉特征并逼近普通 Transformer 块：（1）用 ReLU 替换 SWISH；（2）去掉卷积子块；（3）用单个 FFN 替换 Macaron 风格的 FFN 对；（4）用普通自注意力层 [6] 替换带相对位置嵌入的自注意力 [20]。全部消融结果均在不加外部 LM 下评测。

3.4.2 Combinations of Convolution and Transformer Modules

We study the effects of various different ways of combining the multi-headed self-attention (MHSA) module with the convolution module. First, we try replacing the depthwise convolution in the convolution module with a lightweight convolution [35], see a significant drop in the performance especially on the dev-other dataset. Second, we study placing the convolution module before the MHSA module in our Conformer model and find that it degrades the results by 0.1 on dev-other. Another possible way of the architecture is to split the input into parallel branches of multi-headed self attention module and a convolution module with their output concatenated as suggested in [17]. We found that this worsens the performance when compared to our proposed architecture.

3.4.2 卷积与 Transformer 模块的组合方式

我们研究把多头自注意力（MHSA）模块与卷积模块组合的多种方式。首先，尝试用轻量卷积 [35] 替换卷积模块中的深度卷积，性能明显下降，尤其在 dev-other 上。其次，研究在 Conformer 中把卷积模块放在 MHSA 之前，发现 dev-other 上变差 0.1。另一种可能是把输入拆成多头自注意力与卷积的并行分支并拼接输出，如 [17] 所建议；我们发现这相对所提架构会变差。

These results in Table 4 suggest the advantage of placing the convolution module after the self-attention module in the Conformer block.

表 4 的这些结果说明：在 Conformer 块中把卷积模块放在自注意力模块之后更有利。

Model Architecture	dev clean	dev other
Conformer	1.9	4.4
– Depthwise conv + Lightweight convolution	2.0	4.8
Convolution block before MHSA	1.9	4.5
Parallel MHSA and Convolution	2.0	4.9

Table 4: Ablation study of Conformer Attention Convolution Blocks. Varying the combination of the convolution block with the multi-headed self attention: (1) Conformer architecture; (2) Using Lightweight convolutions instead of depthwise convolution in the convolution block in Conformer; (3) Convolution before multi-headed self attention; (4) Convolution and MHSA in parallel with their output concatenated [17].

模型架构	dev clean	dev other
Conformer	1.9	4.4
– 深度卷积 + 轻量卷积	2.0	4.8
卷积块在 MHSA 之前	1.9	4.5
MHSA 与卷积并行	2.0	4.9

表 4：Conformer 注意力—卷积块的消融。 改变卷积块与多头自注意力的组合方式：（1）Conformer 架构；（2）在 Conformer 卷积块中用轻量卷积替代深度卷积；（3）卷积在多头自注意力之前；（4）卷积与 MHSA 并行并拼接输出 [17]。

3.4.3 Macaron Feed Forward Modules

Instead of a single feed-forward module (FFN) post the attention blocks as in the Transformer models, the Conformer block has a pair of macaron-like Feed forward modules sandwiching the self-attention and convolution modules. Further, the Conformer feed forward modules are used with half-step residuals. Table 5 shows the impact of changing the Conformer block to use a single FFN or full-step residuals.

3.4.3 Macaron 前馈模块

不同于 Transformer 模型在注意力块之后只用单个前馈模块（FFN），Conformer 块有一对类 Macaron 的前馈模块夹住自注意力与卷积。此外，Conformer 前馈模块使用半步残差。表 5 给出把 Conformer 块改为单个 FFN 或全步残差时的影响。

Model Architecture	dev clean	dev other	test clean	test other
Conformer	1.9	4.4	2.1	4.3
Single FFN	1.9	4.5	2.1	4.5
Full step residuals	1.9	4.5	2.1	4.5

Table 5: Ablation study of Macaron-net Feed Forward modules. Ablating the differences between the Conformer feed forward module with that of a single FFN used in Transformer models: (1) Conformer; (2) Conformer with full-step residuals in Feed forward modules; (3) replacing the Macaron-style FFN pair with a single FFN.

模型架构	dev clean	dev other	test clean	test other
Conformer	1.9	4.4	2.1	4.3
单个 FFN	1.9	4.5	2.1	4.5
全步残差	1.9	4.5	2.1	4.5

表 5：Macaron-Net 前馈模块消融。 对比 Conformer 前馈模块与 Transformer 中单个 FFN 的差异：（1）Conformer；（2）前馈模块使用全步残差的 Conformer；（3）用单个 FFN 替换 Macaron 风格的 FFN 对。

3.4.4 Number of Attention Heads

In self-attention, each attention head learns to focus on different parts of the input, making it possible to improve predictions beyond the simple weighted average. We perform experiments to study the effect of varying the number of attention heads from $4$ to $32$ in our large model, using the same number of heads in all layers. We find that increasing attention heads up to $16$ improves the accuracy, especially over the devother datasets, as shown in Table 6.

3.4.4 注意力头数

在自注意力中，每个注意力头学习关注输入的不同部分，从而有可能把预测做得比简单加权平均更好。我们做实验，把大模型中的注意力头数从 $4$ 变到 $32$，各层使用相同头数。我们发现把头数增至 $16$ 能提升精度，尤其在 devother 上，如表 6 所示。

Attention Heads	Dim per Head	dev clean	dev other	test clean	test other
4	128	1.9	4.6	2.0	4.5
8	64	1.9	4.4	2.1	4.3
16	32	2.0	4.3	2.2	4.4
32	16	1.9	4.4	2.1	4.5

Table 6: Ablation study on the attention heads in multi-headed self attention.

注意力头数	每头维度	dev clean	dev other	test clean	test other
4	128	1.9	4.6	2.0	4.5
8	64	1.9	4.4	2.1	4.3
16	32	2.0	4.3	2.2	4.4
32	16	1.9	4.4	2.1	4.5

表 6：多头自注意力中注意力头数的消融。

3.4.5 Convolution Kernel Sizes

To study the effect of kernel sizes in the depthwise convolution, we sweep the kernel size in $\{3,7,17,32,65\}$ of the large model, using the same kernel size for all layers. We find that the performance improves with larger kernel sizes till kernel sizes $17$ and $32$ but worsens in the case of kernel size $65$, as show in Table 7. On comparing the second decimal in dev WER, we find kernel size 32 to perform better than rest.

3.4.5 卷积核大小

为研究深度卷积中核大小的影响，我们在大模型上扫描核大小 $\{3,7,17,32,65\}$，各层使用相同核大小。我们发现性能随核变大而提升，直到核大小 $17$ 与 $32$，但核大小 $65$ 时变差，如表 7 所示。比较开发集 WER 的百分位小数后，核大小 32 优于其余。

Kernel size	dev clean	dev other	test clean	test other
3	1.88	4.41	1.99	4.39
7	1.88	4.30	2.02	4.44
17	1.87	4.31	2.04	4.38
32	1.83	4.30	2.03	4.29
65	1.89	4.47	1.98	4.46

Table 7: Ablation study on depthwise convolution kernel sizes.

核大小	dev clean	dev other	test clean	test other
3	1.88	4.41	1.99	4.39
7	1.88	4.30	2.02	4.44
17	1.87	4.31	2.04	4.38
32	1.83	4.30	2.03	4.29
65	1.89	4.47	1.98	4.46

表 7：深度卷积核大小的消融。

4 Conclusion

4 结论

In this work, we introduced Conformer, an architecture that integrates components from CNNs and Transformers for end-to-end speech recognition. We studied the importance of each component, and demonstrated that the inclusion of convolution modules is critical to the performance of the Conformer model. The model exhibits better accuracy with fewer parameters than previous work on the LibriSpeech dataset, and achieves a new state-of-the-art performance at $1.9\%$/$3.9\%$ for test/testother.

本文提出 Conformer：一种把 CNN 与 Transformer 组件整合进端到端语音识别的架构。我们研究了各组件的重要性，并表明加入卷积模块对 Conformer 模型性能至关重要。该模型在 LibriSpeech 上以更少参数取得优于此前工作的精度，并以 test/testother 上 $1.9\%$/$3.9\%$ 刷新当时最优表现。

References

参考文献

[1] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina et al. , “State-of-the-art speech recognition with sequence-to-sequence models,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2018, pp. 4774–4778.

[2] K. Rao, H. Sak, and R. Prabhavalkar, “Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer,” in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) . IEEE, 2017, pp. 193–199.

[3] Y. He, T. N. Sainath, R. Prabhavalkar, I. McGraw, R. Alvarez, D. Zhao, D. Rybach, A. Kannan, Y. Wu, R. Pang, Q. Liang, D. Bhatia, Y. Shangguan, B. Li, G. Pundak, K. C. Sim, T. Bagby, S.-Y. Chang, K. Rao, and A. Gruenstein, “Streaming End-to-end Speech Recognition For Mobile Devices,” in Proc. ICASSP , 2019.

[4] T. N. Sainath, Y. He, B. Li, A. Narayanan, R. Pang, A. Bruguier, S.-y. Chang, W. Li, R. Alvarez, Z. Chen, and et al., “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,” in ICASSP , 2020.

[5] A. Graves, “Sequence transduction with recurrent neural networks,” arXiv preprint arXiv:1211.3711 , 2012.

[6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017.

[7] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7829–7833.

[8] J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. T. Gadde, “Jasper: An end-to-end convolutional neural acoustic model,” arXiv preprint arXiv:1904.03288 , 2019.

[9] S. Kriman, S. Beliaev, B. Ginsburg, J. Huang, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, and Y. Zhang, “Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions,” arXiv preprint arXiv:1910.10261 , 2019.

[10] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu, “Contextnet: Improving convolutional neural networks for automatic speech recognition with global context,” arXiv preprint arXiv:2005.03191 , 2020.

[11] T. N. Sainath, A.-r. Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep convolutional neural networks for lvcsr,” in 2013 IEEE international conference on acoustics, speech and signal processing . IEEE, 2013, pp. 8614–8618.

[12] O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu, “Convolutional neural networks for speech recognition,” IEEE/ACM Transactions on audio, speech, and language processing , vol. 22, no. 10, pp. 1533–1545, 2014.

[13] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 7132–7141.

[14] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le, “Attention augmented convolutional networks,” in Proceedings of the IEEE International Conference on Computer Vision , 2019, pp. 3286–3295.

[15] B. Yang, L. Wang, D. Wong, L. S. Chao, and Z. Tu, “Convolutional self-attention networks,” arXiv preprint arXiv:1904.03107 , 2019.

[16] A. W. Yu, D. Dohan, M.-T. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V. Le, “Qanet: Combining local convolution with global self-attention for reading comprehension,” arXiv preprint arXiv:1804.09541 , 2018.

[17] Z. Wu, Z. Liu, J. Lin, Y. Lin, and S. Han, “Lite transformer with long-short range attention,” arXiv preprint arXiv:2004.11886 , 2020.

[18] Y. Lu, Z. Li, D. He, Z. Sun, B. Dong, T. Qin, L. Wang, and T.-Y. Liu, “Understanding and improving transformer from a multi-particle dynamic system point of view,” arXiv preprint arXiv:1906.02762 , 2019.

[19] S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. E. Y. Soplin, R. Yamamoto, X. Wang et al. , “A comparative study on transformer vs rnn in speech applications,” arXiv preprint arXiv:1909.06317 , 2019.

[20] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” 2019.

[21] Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, Jul. 2019, pp. 1810–1822.

[22] T. Q. Nguyen and J. Salazar, “Transformers without tears: Improving the normalization of self-attention,” arXiv preprint arXiv:1910.05895 , 2019.

[23] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language modeling with gated convolutional networks,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2017, pp. 933–941.

[24] L. Dong, S. Xu, and B. Xu, “Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2018, pp. 5884–5888.

[25] P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for activation functions,” arXiv preprint arXiv:1710.05941 , 2017.

[26] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2015, pp. 5206–5210.

[27] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” arXiv preprint arXiv:1904.08779 , 2019.

[28] D. S. Park, Y. Zhang, C.-C. Chiu, Y. Chen, B. Li, W. Chan, Q. V. Le, and Y. Wu, “Specaugment on large scale datasets,” arXiv preprint arXiv:1912.05533 , 2019.

[29] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research , vol. 15, no. 56, pp. 1929–1958, 2014.

[30] K.-C. Jim, C. L. Giles, and B. G. Horne, “An analysis of noise in recurrent neural networks: convergence and generalization,” IEEE Transactions on neural networks , vol. 7, no. 6, pp. 1424–1438, 1996.

[31] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014.

[32] J. Shen, P. Nguyen, Y. Wu, Z. Chen, and et al., “Lingvo: a modular and scalable framework for sequence-to-sequence modeling,” 2019.

[33] Y. Wang, A. Mohamed, D. Le, C. Liu, A. Xiao, J. Mahadeokar, H. Huang, A. Tjandra, X. Zhang, F. Zhang et al. , “Transformer-based acoustic modeling for hybrid speech recognition,” arXiv preprint arXiv:1910.09799 , 2019.

[34] G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert, “End-to-end asr: from supervised to semi-supervised learning with modern architectures,” 2019.

[35] F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli, “Pay less attention with lightweight and dynamic convolutions,” arXiv preprint arXiv:1901.10430 , 2019.