FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

FastSpeech 2：快速且高质量的端到端文本转语音

Yi Ren Thanks:  Authors contribute equally to this work. Affiliation:  Zhejiang University {rayeren,chenxuhu,zhaozhou}@zju.edu.cn    Chenxu Hu 1 1 footnotemark: 1 Affiliation:  Zhejiang University {rayeren,chenxuhu,zhaozhou}@zju.edu.cn    Xu Tan Affiliation:  Microsoft Research Asia {xuta,taoqin,tyliu}@microsoft.com    Tao Qin Affiliation:  Microsoft Research Asia {xuta,taoqin,tyliu}@microsoft.com    Sheng Zhao Affiliation:  Microsoft Azure Speech Sheng.Zhao@microsoft.com    Zhou Zhao Thanks:  Corresponding author Affiliation:  Zhejiang University {rayeren,chenxuhu,zhaozhou}@zju.edu.cn    Tie-Yan Liu Affiliation:  Microsoft Research Asia {xuta,taoqin,tyliu}@microsoft.com

Abstract

摘要

Non-autoregressive text to speech (TTS) models such as FastSpeech (Ren et al. 2019) can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.

非自回归文本转语音（TTS）模型，例如 FastSpeech（Ren 等，2019），能在音质相当的前提下，比此前的自回归模型快得多地合成语音。FastSpeech 的训练依赖一个自回归教师模型：一方面用它做时长预测（给输入补上更多信息），另一方面做知识蒸馏（简化输出侧的数据分布），以缓解 TTS 中的一对多映射问题（同一段文本对应多种语音变化）。不过 FastSpeech 也有几处不足：1）教师–学生蒸馏流水线复杂且耗时；2）从教师模型抽出的时长不够准，而从教师蒸馏得到的目标梅尔频谱又因数据简化而损失信息，二者都会限制音质。本文提出 FastSpeech 2，针对这些问题，并更好地处理 TTS 中的一对多映射：1）直接用真实目标训练，而不用教师简化后的输出；2）把更多语音变化信息（如音高、能量以及更准确的时长）作为条件输入。具体来说，我们从语音波形中提取时长、音高和能量，训练时直接作为条件输入，推理时改用预测值。我们还设计了 FastSpeech 2s，这是首次尝试从文本并行地直接生成波形，从而享受完全端到端推理的好处。实验表明：1）FastSpeech 2 相对 FastSpeech 训练加速约 3 倍，FastSpeech 2s 的推理更快；2）FastSpeech 2 与 2s 的音质优于 FastSpeech，且 FastSpeech 2 甚至可以超过自回归模型。音频样例见 https://speechresearch.github.io/fastspeech2/。

1 Introduction

1 引言

Neural network based text to speech (TTS) has made rapid progress and attracted a lot of attention in the machine learning and speech community in recent years (Wang et al. 2017; Shen et al. 2018; Ming et al. 2016; Arik et al. 2017; Ping et al. 2018; Ren et al. 2019; Li et al. 2019). Previous neural TTS models (Wang et al. 2017; Shen et al. 2018; Ping et al. 2018; Li et al. 2019) first generate mel-spectrograms autoregressively from text and then synthesize speech from the generated mel-spectrograms using a separately trained vocoder (Van Den Oord et al. 2016; Oord et al. 2017; Prenger et al. 2019; Kim et al. 2018; Yamamoto et al. 2020; Kumar et al. 2019). They usually suffer from slow inference speed and robustness (word skipping and repeating) issues (Ren et al. 2019; Chen et al. 2020). In recent years, non-autoregressive TTS models (Ren et al. 2019; Łańcucki 2020; Kim et al. 2020; Lim et al. 2020; Miao et al. 2020; Peng et al. 2019) are designed to address these issues, which generate mel-spectrograms with extremely fast speed and avoid robustness issues, while achieving comparable voice quality with previous autoregressive models.

基于神经网络的文本转语音（TTS）近年来进展很快，也引起了机器学习和语音社区的广泛关注（Wang 等，2017；Shen 等，2018；Ming 等，2016；Arik 等，2017；Ping 等，2018；Ren 等，2019；Li 等，2019）。此前的神经 TTS 模型（Wang 等，2017；Shen 等，2018；Ping 等，2018；Li 等，2019）先从文本自回归地生成梅尔频谱，再用单独训练的声码器（Van Den Oord 等，2016；Oord 等，2017；Prenger 等，2019；Kim 等，2018；Yamamoto 等，2020；Kumar 等，2019）把频谱合成语音。它们通常受推理速度慢和稳健性（漏词、重复）问题困扰（Ren 等，2019；Chen 等，2020）。近几年出现的非自回归 TTS 模型（Ren 等，2019；Łańcucki，2020；Kim 等，2020；Lim 等，2020；Miao 等，2020；Peng 等，2019）就是为了解决这些问题：它们能以极快的速度生成梅尔频谱、避开稳健性故障，同时音质与此前的自回归模型相当。

Among those non-autoregressive TTS methods, FastSpeech (Ren et al. 2019) is one of the most successful models. FastSpeech designs two ways to alleviate the one-to-many mapping problem: 1) Reducing data variance in the target side by using the generated mel-spectrogram from an autoregressive teacher model as the training target (i.e., knowledge distillation). 2) Introducing the duration information (extracted from the attention map of the teacher model) to expand the text sequence to match the length of the mel-spectrogram sequence. While these designs in FastSpeech ease the learning of the one-to-many mapping problem (see Section 2.1) in TTS, they also bring several disadvantages: 1) The two-stage teacher-student training pipeline makes the training process complicated. 2) The target mel-spectrograms generated from the teacher model have some information loss[^1] compared with the ground-truth ones, since the quality of the audio synthesized from the generated mel-spectrograms is usually worse than that from the ground-truth ones. 3) The duration extracted from the attention map of teacher model is not accurate enough.

在这些非自回归 TTS 方法里，FastSpeech（Ren 等，2019）是最成功的模型之一。FastSpeech 用两种方式缓解一对多映射：1）用自回归教师模型生成的梅尔频谱作为训练目标（即知识蒸馏），从而降低目标侧的数据方差；2）引入时长信息（从教师模型的注意力图中提取），把文本序列展开到与梅尔频谱序列等长。这些设计减轻了 TTS 中一对多映射的学习难度（见第 2.1 节），但也带来几处代价：1）两阶段教师–学生训练使流程变复杂；2）教师生成的目标梅尔频谱相对真实频谱有信息损失[^1]，因为用生成频谱合成的音频通常不如用真实频谱合成的音频；3）从教师注意力图提取的时长不够准确。

In this work, we propose FastSpeech 2 to address the issues in FastSpeech and better handle the one-to-many mapping problem in non-autoregressive TTS. To simplify the training pipeline and avoid the information loss due to data simplification in teacher-student distillation, we directly train the FastSpeech 2 model with ground-truth target instead of the simplified output from a teacher. To reduce the information gap (input does not contain all the information to predict the target) between the input (text sequence) and target output (mel-spectrograms) and alleviate the one-to-many mapping problem for non-autoregressive TTS model training, we introduce some variation information of speech including pitch, energy and more accurate duration into FastSpeech: in training, we extract duration, pitch and energy from the target speech waveform and directly take them as conditional inputs; in inference, we use values predicted by the predictors that are jointly trained with the FastSpeech 2 model. Considering the pitch is important for the prosody of speech and is also difficult to predict due to the large fluctuations along time, we convert the pitch contour into pitch spectrogram using continuous wavelet transform (Tuteur 1988; Grossmann & Morlet 1984) and predict the pitch in the frequency domain, which can improve the accuracy of predicted pitch. To further simplify the speech synthesis pipeline, we introduce FastSpeech 2s, which does not use mel-spectrograms as intermediate output and directly generates speech waveform from text in inference, enjoying low latency in inference. Experiments on the LJSpeech (Ito 2017) dataset show that 1) FastSpeech 2 enjoys much simpler training pipeline (3x training time reduction) than FastSpeech while inherits its advantages of fast, robust and controllable (even more controllable in pitch and energy) speech synthesis, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. We attach audio samples generated by FastSpeech 2 and 2s at https://speechresearch.github.io/fastspeech2/.

本文提出 FastSpeech 2，用来解决 FastSpeech 的上述问题，并更好地处理非自回归 TTS 中的一对多映射。为了简化训练流程、避免教师–学生蒸馏因数据简化造成的信息损失，我们直接用真实目标训练 FastSpeech 2，而不用教师简化后的输出。为了缩小输入（文本序列）与目标输出（梅尔频谱）之间的信息差距（输入并不包含预测目标所需的全部信息），并减轻非自回归 TTS 训练中的一对多映射，我们把音高、能量以及更准确的时长等语音变化信息引入 FastSpeech：训练时从目标语音波形中提取时长、音高和能量，直接作为条件输入；推理时则使用与 FastSpeech 2 联合训练的预测器给出的值。音高对韵律很重要，又因沿时间起伏大而难以预测，于是我们用连续小波变换（Tuteur，1988；Grossmann 与 Morlet，1984）把音高曲线转成音高频谱图，并在频域预测音高，从而提高预测精度。为进一步简化语音合成流水线，我们引入 FastSpeech 2s：它不再把梅尔频谱当作中间输出，推理时直接从文本生成波形，延迟更低。在 LJSpeech（Ito，2017）上的实验表明：1）FastSpeech 2 的训练流程比 FastSpeech 简单得多（训练时间减少约 3 倍），同时继承了快速、稳健、可控（在音高和能量上甚至更可控）的合成优点，FastSpeech 2s 的推理更快；2）FastSpeech 2 与 2s 的音质优于 FastSpeech，且 FastSpeech 2 甚至可以超过自回归模型。我们在 https://speechresearch.github.io/fastspeech2/ 附上了 FastSpeech 2 与 2s 生成的音频样例。

The main contributions of this work are summarized as follows:

本文的主要贡献概括如下：

• • FastSpeech 2 achieves a 3x training speed-up over FastSpeech by simplifying the training pipeline.

• • FastSpeech 2 通过简化训练流程，相对 FastSpeech 实现约 3 倍的训练加速。

• • FastSpeech 2 alleviates the one-to-many mapping problem in TTS and achieves better voice quality.

• • FastSpeech 2 减轻了 TTS 中的一对多映射问题，并获得更好的音质。

• • FastSpeech 2s further simplifies the inference pipeline for speech synthesis while maintaining high voice quality, by directly generating speech waveform from text.

• • FastSpeech 2s 通过直接从文本生成波形，进一步简化了语音合成的推理流程，同时保持较高音质。

2 FastSpeech 2 and 2s

2 FastSpeech 2 与 2s

In this section, we first describe the motivation of the design in FastSpeech 2, and then introduce the architecture of FastSpeech 2, which aims to improve FastSpeech to better handle the one-to-many mapping problem, with simpler training pipeline and higher voice quality. At last, we extend FastSpeech 2 to FastSpeech 2s for fully end-to-end text-to-waveform synthesis[^2].

本节先说明 FastSpeech 2 的设计动机，再介绍其结构：目标是改进 FastSpeech，使其更好地处理一对多映射，同时训练更简单、音质更高。最后，我们把 FastSpeech 2 扩展为 FastSpeech 2s，用于完全端到端的文本至波形合成[^2]。

2.1 Motivation

2.1 动机

TTS is a typical one-to-many mapping problem (Wang et al. 2017; Zhu et al. 2017; Jayne et al. 2012; Gadermayr et al. 2020; Chen et al. 2021), since multiple possible speech sequences can correspond to a text sequence due to variations in speech, such as pitch, duration, sound volume and prosody. In non-autoregressive TTS, the only input information is text which is not enough to fully predict the variance in speech. In this case, the model is prone to overfit to the variations of the target speech in the training set, resulting in poor generalization ability. As mentioned in Section 1, although FastSpeech designs two ways to alleviate the one-to-many mapping problem, they also bring about several issues including 1) the complicated training pipeline; 2) information loss of target mel-spectrogram as analyzed in Table 1; and 3) not accurate enough ground-truth duration as shown in Table 5(a). In the following subsection, we introduce the detailed design of FastSpeech 2 which aims to address these issues.

TTS 是典型的一对多映射问题（Wang 等，2017；Zhu 等，2017；Jayne 等，2012；Gadermayr 等，2020；Chen 等，2021），因为音高、时长、音量和韵律等变化，使同一段文本可以对应多条可能的语音序列。在非自回归 TTS 中，唯一的输入信息是文本，并不足以完全预测语音中的变化。此时模型容易过拟合到训练集目标语音的变化上，泛化变差。如第 1 节所述，FastSpeech 虽然设计了两种方式来缓解一对多映射，但也带来若干问题，包括：1）训练流程复杂；2）目标梅尔频谱的信息损失，见表 1 的分析；3）真实时长不够准确，见表 5(a)。下一小节介绍 FastSpeech 2 的具体设计，正是为了解决这些问题。

Figure 1: The overall architecture for FastSpeech 2 and 2s. LR in subfigure (b) denotes the length regulator proposed in FastSpeech. LN in subfigure (c) denotes layer normalization.

图 1：FastSpeech 2 与 2s 的整体结构。子图 (b) 中的 LR 表示 FastSpeech 提出的长度调节器。子图 (c) 中的 LN 表示层归一化。

2.2 Model Overview

2.2 模型概览

The overall model architecture of FastSpeech 2 is shown in Figure 1(a). The encoder converts the phoneme embedding sequence into the phoneme hidden sequence, and then the variance adaptor adds different variance information such as duration, pitch and energy into the hidden sequence, finally the mel-spectrogram decoder converts the adapted hidden sequence into mel-spectrogram sequence in parallel. We use the feed-forward Transformer block, which is a stack of self-attention (Vaswani et al. 2017) layer and 1D-convolution as in FastSpeech (Ren et al. 2019), as the basic structure for the encoder and mel-spectrogram decoder. Different from FastSpeech that relies on a teacher-student distillation pipeline and the phoneme duration from a teacher model, FastSpeech 2 makes several improvements. First, we remove the teacher-student distillation pipeline, and directly use ground-truth mel-spectrograms as target for model training, which can avoid the information loss in distilled mel-spectrograms and increase the upper bound of the voice quality. Second, our variance adaptor consists of not only duration predictor but also pitch and energy predictors, where 1) the duration predictor uses the phoneme duration obtained by forced alignment (McAuliffe et al. 2017) as training target, which is more accurate than that extracted from the attention map of autoregressive teacher model as verified experimentally in Section 3.2.2; and 2) the additional pitch and energy predictors can provide more variance information, which is important to ease the one-to-many mapping problem in TTS. Third, to further simplify the training pipeline and push it towards a fully end-to-end system, we propose FastSpeech 2s, which directly generates waveform from text, without cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder). In the following subsections, we describe detailed designs of the variance adaptor and direct waveform generation in our method.

FastSpeech 2 的整体结构见图 1(a)。编码器把音素嵌入序列转换成音素隐状态序列，随后方差适配器把时长、音高和能量等不同变化信息加入隐状态，最后梅尔频谱解码器并行地把适配后的隐状态转换成梅尔频谱序列。编码器与梅尔频谱解码器的基本结构沿用 FastSpeech（Ren 等，2019）中的前馈 Transformer 块，即自注意力（Vaswani 等，2017）层与一维卷积的堆叠。与依赖教师–学生蒸馏、并从教师模型获取音素时长的 FastSpeech 不同，FastSpeech 2 做了几处改进。第一，去掉教师–学生蒸馏，直接用真实梅尔频谱作为训练目标，从而避免蒸馏频谱中的信息损失，提高音质上限。第二，方差适配器不仅包含时长预测器，还包含音高和能量预测器：1）时长预测器以强制对齐（McAuliffe 等，2017）得到的音素时长为训练目标，实验上比从自回归教师注意力图提取的时长更准，见第 3.2.2 节；2）额外的音高与能量预测器能提供更多变化信息，这对缓解 TTS 中的一对多映射很重要。第三，为进一步简化训练并把它推向完全端到端系统，我们提出 FastSpeech 2s，直接从文本生成波形，不再串联梅尔频谱生成（声学模型）与波形生成（声码器）。下面几小节分别说明方差适配器以及直接波形生成的设计。

2.3 Variance Adaptor

2.3 方差适配器

The variance adaptor aims to add variance information (e.g., duration, pitch, energy, etc.) to the phoneme hidden sequence, which can provide enough information to predict variant speech for the one-to-many mapping problem in TTS. We briefly introduce the variance information as follows: 1) phoneme duration, which represents how long the speech voice sounds; 2) pitch, which is a key feature to convey emotions and greatly affects the speech prosody; 3) energy, which indicates frame-level magnitude of mel-spectrograms and directly affects the volume and prosody of speech. More variance information can be added in the variance adaptor, such as emotion, style and speaker, and we leave it for future work. Correspondingly, the variance adaptor consists of 1) a duration predictor (i.e., the length regulator, as used in FastSpeech), 2) a pitch predictor, and 3) an energy predictor, as shown in Figure 1(b). In training, we take the ground-truth value of duration, pitch and energy extracted from the recordings as input into the hidden sequence to predict the target speech. At the same time, we use the ground-truth duration, pitch and energy as targets to train the duration, pitch and energy predictors, which are used in inference to synthesize target speech. As shown in Figure 1(c), the duration, pitch and energy predictors share similar model structure (but different model parameters), which consists of a 2-layer 1D-convolutional network with ReLU activation, each followed by the layer normalization and the dropout layer, and an extra linear layer to project the hidden states into the output sequence. In the following paragraphs, we describe the details of the three predictors respectively.

Duration Predictor

The duration predictor takes the phoneme hidden sequence as input and predicts the duration of each phoneme, which represents how many mel frames correspond to this phoneme, and is converted into logarithmic domain for ease of prediction. The duration predictor is optimized with mean square error (MSE) loss, taking the extracted duration as training target. Instead of extracting the phoneme duration using a pre-trained autoregressive TTS model in FastSpeech, we use Montreal forced alignment (MFA) (McAuliffe et al. 2017) tool[^3] to extract the phoneme duration, in order to improve the alignment accuracy and thus reduce the information gap between the model input and output.

时长预测器

时长预测器以音素隐状态序列为输入，预测每个音素的时长，也就是该音素对应多少个梅尔帧；为便于预测，时长会被转换到对数域。预测器以均方误差（MSE）为损失，用提取出的时长作训练目标。与 FastSpeech 用预训练自回归 TTS 模型提取音素时长不同，我们使用 Montreal 强制对齐（MFA）（McAuliffe 等，2017）工具[^3]来提取音素时长，以提高对齐精度，从而缩小模型输入与输出之间的信息差距。

Pitch Predictor

Previous neural network based TTS systems with pitch prediction (Arik et al. 2017; Gibiansky et al. 2017) often predict pitch contour directly. However, due to high variations of ground-truth pitch, the distribution of predicted pitch values is very different from ground-truth distribution, as analyzed in Section 3.2.2. To better predict the variations in pitch contour, we use continuous wavelet transform (CWT) to decompose the continuous pitch series into pitch spectrogram (Suni et al. 2013; Hirose & Tao 2015) and take the pitch spectrogram as the training target for the pitch predictor which is optimized with MSE loss. In inference, the pitch predictor predicts the pitch spectrogram, which is further converted back into pitch contour using inverse continuous wavelet transform (iCWT). We describe the details of pitch extraction, CWT, iCWT and pitch predictor architecture in Appendix D. To take the pitch contour as input in both training and inference, we quantize pitch $F_{0}$ (ground-truth/predicted value for train/inference respectively) of each frame to 256 possible values in log-scale and further convert it into pitch embedding vector $p$ and add it to the expanded hidden sequence.

音高预测器

此前带音高预测的神经 TTS 系统（Arik 等，2017；Gibiansky 等，2017）常常直接预测音高曲线。然而，由于真实音高变化很大，预测值的分布往往与真实分布差得很远，见第 3.2.2 节的分析。为了更好地刻画音高曲线中的变化，我们用连续小波变换（CWT）把连续音高序列分解成音高频谱图（Suni 等，2013；Hirose 与 Tao，2015），并以该频谱图作为音高预测器的训练目标，用 MSE 优化。推理时，预测器先给出音高频谱图，再用逆连续小波变换（iCWT）还原成音高曲线。音高提取、CWT、iCWT 以及预测器结构的细节见附录 D。为了在训练和推理时都能把音高曲线当作输入，我们把每一帧的音高 $F_{0}$（训练用真实值，推理用预测值）在对数尺度上量化成 256 个可能取值，再转换成音高嵌入向量 $p$，加到展开后的隐状态序列上。

Energy Predictor

We compute L2-norm of the amplitude of each short-time Fourier transform (STFT) frame as the energy. Then we quantize energy of each frame to 256 possible values uniformly, encoded it into energy embedding $e$ and add it to the expanded hidden sequence similarly to pitch. We use an energy predictor to predict the original values of energy instead of the quantized values and optimize the energy predictor with MSE loss[^4].

能量预测器

我们把每一帧短时傅里叶变换（STFT）幅度的 L2 范数作为能量，再把每帧能量均匀量化成 256 个可能取值，编码成能量嵌入 $e$，并以与音高类似的方式加到展开后的隐状态上。能量预测器预测的是原始能量值而不是量化值，并用 MSE 优化[^4]。

2.4 FastSpeech 2s

To enable fully end-to-end text-to-waveform generation, in this subsection, we extend FastSpeech 2 to FastSpeech 2s, which directly generates waveform from text, without cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder). As shown in Figure 1(a), FastSpeech 2s generates waveform conditioning on intermediate hidden, which makes it more compact in inference by discarding mel-spectrogram decoder and achieve comparable performance with a cascaded system. We first discuss the challenges in non-autoregressive text-to-waveform generation, then describe details of FastSpeech 2s, including model structure and training and inference processes.

为了实现完全端到端的文本至波形生成，我们在本小节把 FastSpeech 2 扩展为 FastSpeech 2s：它直接从文本生成波形，不再串联梅尔频谱生成（声学模型）与波形生成（声码器）。如图 1(a) 所示，FastSpeech 2s 以中间隐状态为条件生成波形，推理时可以丢掉梅尔频谱解码器，从而更紧凑，并达到与级联系统相当的性能。我们先讨论非自回归文本至波形生成的难点，再说明 FastSpeech 2s 的模型结构以及训练与推理过程。

Challenges in Text-to-Waveform Generation

When pushing TTS pipeline towards fully end-to-end framework, there are several challenges: 1) Since the waveform contains more variance information (e.g., phase) than mel-spectrograms, the information gap between the input and output is larger than that in text-to-spectrogram generation. 2) It is difficult to train on the audio clip that corresponds to the full text sequence due to the extremely long waveform samples and limited GPU memory. As a result, we can only train on a short audio clip that corresponds to a partial text sequence which makes it hard for the model to capture the relationship among phonemes in different partial text sequences and thus harms the text feature extraction.

文本至波形生成的难点

把 TTS 流水线推向完全端到端时，有几处挑战：1）波形比梅尔频谱包含更多变化信息（例如相位），因此输入与输出之间的信息差距比文本至频谱生成更大。2）波形样本极长，GPU 显存有限，很难在对应整段文本的音频片段上训练。于是我们只能在对应部分文本的短音频片段上训练，这使模型难以捕捉不同部分文本中音素之间的关系，从而损害文本特征提取。

Our Method

To tackle the challenges above, we make several designs in the waveform decoder: 1) Considering that the phase information is difficult to predict using a variance predictor (Engel et al. 2020), we introduce adversarial training in the waveform decoder to force it to implicitly recover the phase information by itself (Yamamoto et al. 2020). 2) We leverage the mel-spectrogram decoder of FastSpeech 2, which is trained on the full text sequence to help on the text feature extraction. As shown in Figure 1(d), the waveform decoder is based on the structure of WaveNet (Van Den Oord et al. 2016) including non-causal convolutions and gated activation (Van den Oord et al. 2016). The waveform decoder takes a sliced hidden sequence corresponding to a short audio clip as input and upsamples it with transposed 1D-convolution to match the length of audio clip. The discriminator in the adversarial training adopts the same structure in Parallel WaveGAN (Yamamoto et al. 2020) which consists of ten layers of non-causal dilated 1-D convolutions with leaky ReLU activation function. The waveform decoder is optimized by the multi-resolution STFT loss and the LSGAN discriminator loss following Parallel WaveGAN. In inference, we discard the mel-spectrogram decoder and only use the waveform decoder to synthesize speech audio.

我们的方法

针对上述难点，我们在波形解码器里做了几处设计：1）相位信息很难用方差预测器直接预测（Engel 等，2020），于是我们在波形解码器中引入对抗训练，迫使它自己隐式恢复相位（Yamamoto 等，2020）。2）我们借助 FastSpeech 2 的梅尔频谱解码器——它在完整文本序列上训练——来帮助提取文本特征。如图 1(d) 所示，波形解码器基于 WaveNet（Van Den Oord 等，2016）的结构，包含非因果卷积和门控激活（Van den Oord 等，2016）。它把与短音频片段对应的切片隐状态序列作为输入，用转置一维卷积上采样到与音频片段等长。对抗训练中的判别器采用与 Parallel WaveGAN（Yamamoto 等，2020）相同的结构：十层非因果膨胀一维卷积，激活函数为 leaky ReLU。波形解码器按照 Parallel WaveGAN 用多分辨率 STFT 损失和 LSGAN 判别器损失来优化。推理时，我们丢掉梅尔频谱解码器，只使用波形解码器合成语音。

2.5 Discussions

2.5 讨论

In this subsection, we discuss how FastSpeech 2 and 2s differentiate from previous and concurrent works.

本小节讨论 FastSpeech 2 与 2s 和此前及同期工作的区别。

Compared with Deep Voice (Arik et al. 2017), Deep Voice 2 (Gibiansky et al. 2017) and other methods Fan et al. 2014; Ze et al. 2013 which generate waveform autoregressively and also predict variance information such as duration and pitch, Fastspeech 2 and 2s adopt self-attention based feed-forward network to generate mel-spectrograms or waveform in parallel. While some existing non-autoregressive acoustic models (Zeng et al. 2020; Lim et al. 2020; Kim et al. 2020) mostly focus on improving the duration accuracy, FastSpeech 2 and 2s provide more variation information (duration, pitch and energy) as inputs to reduce the information gap between the input and output. A concurrent work (Łańcucki 2020) employs pitch prediction in phoneme level, while FastSpeech 2 and 2s predict more fine-grained pitch contour in frame level. In addition, to improve the prosody in synthesized speech, FastSpeech 2 and 2s further introduce continuous wavelet transform to model the variations in pitch.

与 Deep Voice（Arik 等，2017）、Deep Voice 2（Gibiansky 等，2017）以及其他自回归生成波形、同时预测时长和音高等变化信息的方法（Fan 等，2014；Ze 等，2013）相比，FastSpeech 2 与 2s 采用基于自注意力的前馈网络，并行生成梅尔频谱或波形。现有的一些非自回归声学模型（Zeng 等，2020；Lim 等，2020；Kim 等，2020）大多把重点放在提高时长精度上，而 FastSpeech 2 与 2s 则把更多变化信息（时长、音高和能量）作为输入，以缩小输入与输出之间的信息差距。一篇同期工作（Łańcucki，2020）在音素级做音高预测，而 FastSpeech 2 与 2s 在帧级预测更细粒度的音高曲线。此外，为改善合成语音的韵律，FastSpeech 2 与 2s 还引入连续小波变换来刻画音高变化。

While some text-to-waveform models such as ClariNet (Ping et al. 2019) jointly train an autoregressive acoustic model and a non-autoregressive vocoder, FastSpeech 2s embraces the fully non-autoregressive architecture for fast inference. A concurrent work called EATS (Donahue et al. 2020) also employs non-autoregressive architecture and adversarial training to convert text to waveform directly and mainly focuses on predicting the duration of each phoneme end-to-end using a differentiable monotonic interpolation scheme. Compared with EATS, FastSpeech 2s additionally provides more variation information to ease the one-to-many mapping problem in TTS.

一些文本至波形模型，例如 ClariNet（Ping 等，2019），会联合训练自回归声学模型和非自回归声码器；FastSpeech 2s 则采用完全非自回归结构，以便快速推理。一篇名为 EATS 的同期工作（Donahue 等，2020）同样用非自回归结构和对抗训练直接把文本转换成波形，重点是用可微的单调插值方案端到端预测每个音素的时长。与 EATS 相比，FastSpeech 2s 额外提供了更多变化信息，以缓解 TTS 中的一对多映射。

Previous non-autoregressive vocoders (Oord et al. 2017; Prenger et al. 2019; Yamamoto et al. 2020; Kumar et al. 2019) are not complete text-to-speech systems, since they convert time aligned linguistic features to waveforms, and require a separate linguistic model to convert input text to linguistic features or an acoustic model to convert input text to acoustic features (e.g., mel-spectrograms). FastSpeech 2s is the first attempt to directly generate waveform from phoneme sequence fully in parallel, instead of linguistic features or mel-spectrograms.

此前的非自回归声码器（Oord 等，2017；Prenger 等，2019；Yamamoto 等，2020；Kumar 等，2019）并不是完整的文本转语音系统：它们把已对齐的语言学特征转换成波形，还需要单独的语言学模型把输入文本转成语言学特征，或用声学模型把输入文本转成声学特征（例如梅尔频谱）。FastSpeech 2s 是首次尝试完全并行地从音素序列直接生成波形，而不是从语言学特征或梅尔频谱出发。

3 Experiments and Results

3 实验与结果

3.1 Experimental Setup

3.1 实验设置

Datasets

We evaluate FastSpeech 2 and 2s on LJSpeech dataset (Ito 2017). LJSpeech contains 13,100 English audio clips (about 24 hours) and corresponding text transcripts. We split the dataset into three sets: 12,228 samples for training, 349 samples (with document title LJ003) for validation and 523 samples (with document title LJ001 and LJ002) for testing. For subjective evaluation, we randomly choose 100 samples in test set. To alleviate the mispronunciation problem, we convert the text sequence into the phoneme sequence (Arik et al. 2017; Wang et al. 2017; Shen et al. 2018; Sun et al. 2019) with an open-source grapheme-to-phoneme tool[^5]. We transform the raw waveform into mel-spectrograms following Shen et al. 2018 and set frame size and hop size to 1024 and 256 with respect to the sample rate 22050.

数据集

我们在 LJSpeech 数据集（Ito，2017）上评估 FastSpeech 2 与 2s。LJSpeech 包含 13,100 条英语音频片段（约 24 小时）及对应文本转写。我们把数据分成三份：12,228 条用于训练，349 条（文档标题为 LJ003）用于验证，523 条（文档标题为 LJ001 和 LJ002）用于测试。主观评测时，我们从测试集中随机抽取 100 条样本。为减轻错误发音，我们用开源字素到音素工具[^5]把文本序列转换成音素序列（Arik 等，2017；Wang 等，2017；Shen 等，2018；Sun 等，2019）。原始波形转成梅尔频谱的方式遵循 Shen 等（2018），在采样率 22050 下将帧长和 hop 分别设为 1024 和 256。

Model Configuration

Our FastSpeech 2 consists of 4 feed-forward Transformer (FFT) blocks (Ren et al. 2019) in the encoder and the mel-spectrogram decoder. The output linear layer in the decoder converts the hidden states into 80-dimensional mel-spectrograms and our model is optimized with mean absolute error (MAE). We add more detailed configurations of FastSpeech 2 and 2s used in our experiments in Appendix A. The details of training and inference are added in Appendix B.

模型配置

我们的 FastSpeech 2 在编码器和梅尔频谱解码器中各使用 4 个前馈 Transformer（FFT）块（Ren 等，2019）。解码器的输出线性层把隐状态转换成 80 维梅尔频谱，模型用平均绝对误差（MAE）优化。更详细的 FastSpeech 2 与 2s 配置见附录 A，训练与推理细节见附录 B。

3.2 Results

3.2 结果

Method	MOS
GT	4.30 $\pm$ 0.07
GT (Mel + PWG)	3.92 $\pm$ 0.08
Tacotron 2 (Shen et al. 2018) (Mel + PWG)	3.70 $\pm$ 0.08
Transformer TTS (Li et al. 2019) (Mel + PWG)	3.72 $\pm$ 0.07
FastSpeech (Ren et al. 2019) (Mel + PWG)	3.68 $\pm$ 0.09
FastSpeech 2 (Mel + PWG)	3.83 $\pm$ 0.08
FastSpeech 2s	3.71 $\pm$ 0.09

Method	CMOS
FastSpeech 2	0.000
FastSpeech	-0.885
Transformer TTS	-0.235

(a) The MOS with 95% confidence intervals.

方法	MOS
GT	4.30 $\pm$ 0.07
GT（Mel + PWG）	3.92 $\pm$ 0.08
Tacotron 2（Shen 等，2018）（Mel + PWG）	3.70 $\pm$ 0.08
Transformer TTS（Li 等，2019）（Mel + PWG）	3.72 $\pm$ 0.07
FastSpeech（Ren 等，2019）（Mel + PWG）	3.68 $\pm$ 0.09
FastSpeech 2（Mel + PWG）	3.83 $\pm$ 0.08
FastSpeech 2s	3.71 $\pm$ 0.09

方法	CMOS
FastSpeech 2	0.000
FastSpeech	-0.885
Transformer TTS	-0.235

(a) 带 95% 置信区间的 MOS。

In this section, we first evaluate the audio quality, training and inference speedup of FastSpeech 2 and 2s. Then we conduct analyses and ablation studies of our method[^6].

本节先评估 FastSpeech 2 与 2s 的音频质量、训练与推理加速，再进行分析和消融实验[^6]。

Audio Quality

To evaluate the perceptual quality, we perform mean opinion score (MOS) (Chu & Peng 2006) evaluation on the test set. Twenty native English speakers are asked to make quality judgments about the synthesized speech samples. The text content keeps consistent among different systems so that all testers only examine the audio quality without other interference factors. We compare the MOS of the audio samples generated by FastSpeech 2 and FastSpeech 2s with other systems, including 1) GT, the ground-truth recordings; 2) GT (Mel + PWG), where we first convert the ground-truth audio into mel-spectrograms, and then convert the mel-spectrograms back to audio using Parallel WaveGAN (Yamamoto et al. 2020) (PWG); 3) Tacotron 2 (Shen et al. 2018) (Mel + PWG); 4) Transformer TTS (Li et al. 2019) (Mel + PWG); 5) FastSpeech (Ren et al. 2019) (Mel + PWG). All the systems in 3), 4) and 5) use Parallel WaveGAN as the vocoder for a fair comparison. The results are shown in Table 1. It can be seen that FastSpeech 2 can surpass and FastSpeech 2s can match the voice quality of autoregressive models Transformer TTS and Tacotron 2. Importantly, FastSpeech 2 outperforms FastSpeech, which demonstrates the effectiveness of providing variance information such as pitch, energy and more accurate duration and directly taking ground-truth speech as training target without using teacher-student distillation pipeline.

音频质量

为评估感知质量，我们在测试集上做平均意见分（MOS）（Chu 与 Peng，2006）评测，请二十位以英语为母语的听者对合成语音样例做质量判断。不同系统的文本内容保持一致，以便听者只考察音质，不受其他因素干扰。我们把 FastSpeech 2 与 FastSpeech 2s 生成样例的 MOS 与下列系统比较：1）GT，真实录音；2）GT（Mel + PWG），先把真实音频转成梅尔频谱，再用 Parallel WaveGAN（Yamamoto 等，2020）（PWG）还原成音频；3）Tacotron 2（Shen 等，2018）（Mel + PWG）；4）Transformer TTS（Li 等，2019）（Mel + PWG）；5）FastSpeech（Ren 等，2019）（Mel + PWG）。3）、4）、5）都用 Parallel WaveGAN 作声码器，以保证公平比较。结果见表 1。可以看到，FastSpeech 2 能够超过、FastSpeech 2s 能够追上自回归模型 Transformer TTS 和 Tacotron 2 的音质。更重要的是，FastSpeech 2 优于 FastSpeech，说明提供音高、能量和更准确时长等变化信息、并直接以真实语音为训练目标、不再走教师–学生蒸馏，是有效的。

Training and Inference Speedup

Method	Training Time (h)	Inference Speed (RTF)	Inference Speedup
Transformer TTS (Li et al. 2019)	38.64	$9.32\times 10^{-1}$	/
FastSpeech (Ren et al. 2019)	53.12	$1.92\times 10^{-2}$	$48.5\times$
FastSpeech 2	17.02	$1.95\times 10^{-2}$	$47.8\times$
FastSpeech 2s	92.18	$\mathbf{1.80\times 10^{-2}}$	$\mathbf{51.8\times}$

Table 2: The comparison of training time and inference latency in waveform synthesis. The training time of FastSpeech includes teacher and student training. RTF denotes the real-time factor, that is the time (in seconds) required for the system to synthesize one second waveform. The training and inference latency tests are conducted on a server with 36 Intel Xeon CPUs, 256GB memory, 1 NVIDIA V100 GPU and batch size of 48 for training and 1 for inference. Besides, we do not include the time of GPU memory garbage collection and transferring input and output data between the CPU and the GPU. The speedup in waveform synthesis for FastSpeech is larger than that reported in Ren et al. 2019 since we use Parallel WaveGAN as the vocoder which is much faster than WaveGlow.

训练与推理加速

方法	训练时间（小时）	推理速度（RTF）	推理加速
Transformer TTS（Li 等，2019）	38.64	$9.32\times 10^{-1}$	/
FastSpeech（Ren 等，2019）	53.12	$1.92\times 10^{-2}$	$48.5\times$
FastSpeech 2	17.02	$1.95\times 10^{-2}$	$47.8\times$
FastSpeech 2s	92.18	$\mathbf{1.80\times 10^{-2}}$	$\mathbf{51.8\times}$

表 2：波形合成中训练时间与推理延迟的比较。FastSpeech 的训练时间包含教师与学生训练。RTF 表示实时率，即系统合成 1 秒波形所需的时间（秒）。训练与推理延迟测试在一台配备 36 颗 Intel Xeon CPU、256GB 内存、1 块 NVIDIA V100 GPU 的服务器上进行，训练批次大小为 48，推理批次大小为 1。此外，我们未计入 GPU 显存垃圾回收以及 CPU 与 GPU 之间传输输入输出数据的时间。FastSpeech 在波形合成上的加速比 Ren 等（2019）报告的更大，因为我们使用的 Parallel WaveGAN 声码器比 WaveGlow 快得多。

FastSpeech 2 simplifies the training pipeline of FastSpeech by removing the teacher-student distillation process, and thus reduces the training time. We list the total training time of Transformer TTS (the autoregressive teacher model), FastSpeech (including the training of Transformer TTS teacher model and FastSpeech student model) and FastSpeech 2 in Table 2. It can be seen that FastSpeech 2 reduces the total training time by $3.12\times$ compared with FastSpeech. Note that training time here only includes acoustic model training, without considering the vocoder training. Therefore, we do not compare the training time of FastSpeech 2s here. We then evaluate the inference latency of FastSpeech 2 and 2s compared with the autoregressive Transformer TTS model, which has the similar number of model parameters with FastSpeech 2 and 2s. We show the inference speedup for waveform generation in Table 2. It can be seen that compared with the Transformer TTS model, FastSpeech 2 and 2s speeds up the audio generation by $47.8\times$ and 51.8$\times$ respectively in waveform synthesis. We can also see that FastSpeech 2s is faster than FastSpeech 2 due to fully end-to-end generation.

FastSpeech 2 去掉了教师–学生蒸馏，从而简化了 FastSpeech 的训练流程并缩短训练时间。表 2 列出了 Transformer TTS（自回归教师模型）、FastSpeech（包含 Transformer TTS 教师模型和 FastSpeech 学生模型的训练）以及 FastSpeech 2 的总训练时间。可以看到，相对 FastSpeech，FastSpeech 2 把总训练时间减少了 $3.12\times$。注意这里的训练时间只包含声学模型训练，不考虑声码器训练，因此此处不比较 FastSpeech 2s 的训练时间。接着，我们评估 FastSpeech 2 与 2s 相对自回归 Transformer TTS 模型的推理延迟；后者的参数量与 FastSpeech 2 和 2s 相近。表 2 给出了波形生成的推理加速。相对 Transformer TTS，FastSpeech 2 与 2s 在波形合成上分别加速 $47.8\times$ 和 $51.8\times$。还可以看到，由于完全端到端生成，FastSpeech 2s 比 FastSpeech 2 更快。

More Accurate Variance Information in Synthesized Speech

Method	$\sigma$	$\gamma$	$\mathcal{K}$	$DTW$
GT	54.4	0.836	0.977	/
Tacotron 2	44.1	1.28	1.311	26.32
TransformerTTS	40.8	0.703	1.419	24.40
FastSpeech	50.8	0.724	-0.041	24.89
FastSpeech 2	54.1	0.881	0.996	24.39
FastSpeech 2 - CWT	42.3	0.771	1.115	25.13
FastSpeech 2s	53.9	0.872	0.998	24.37

Table 3: Standard deviation ($\sigma$), skewness ($\gamma$), kurtosis ($\mathcal{K}$) and average DTW distances (DTW) of pitch in ground-truth and synthesized audio.

合成语音中更准确的变化信息

方法	$\sigma$	$\gamma$	$\mathcal{K}$	$DTW$
GT	54.4	0.836	0.977	/
Tacotron 2	44.1	1.28	1.311	26.32
TransformerTTS	40.8	0.703	1.419	24.40
FastSpeech	50.8	0.724	-0.041	24.89
FastSpeech 2	54.1	0.881	0.996	24.39
FastSpeech 2 - CWT	42.3	0.771	1.115	25.13
FastSpeech 2s	53.9	0.872	0.998	24.37

表 3：真实音频与合成音频中音高的标准差（$\sigma$）、偏度（$\gamma$）、峰度（$\mathcal{K}$）以及平均 DTW 距离（DTW）。

In the paragraph, we measure if providing more variance information (e.g., pitch and energy) as input in FastSpeech 2 and 2s can indeed synthesize speech with more accurate pitch and energy.

本段我们衡量：在 FastSpeech 2 与 2s 中把更多变化信息（如音高和能量）作为输入，是否真的能合成出音高和能量更准确的语音。

For pitch, we compute the moments (standard deviation ($\sigma$), skewness ($\gamma$) and kurtosis ($\mathcal{K}$)) (Andreeva et al. 2014; Niebuhr & Skarnitzl 2019) and average dynamic time warping (DTW) Müller 2007 distance of the pitch distribution for the ground-truth speech and synthesized speech. The results are shown in Table 3. It can be seen that compared with FastSpeech, the moments ($\sigma$, $\gamma$ and $\mathcal{K}$) of generated audio of FastSpeech 2/2s are more close to the ground-truth audio and the average DTW distances to the ground-truth pitch are smaller than other methods, demonstrating that FastSpeech 2/2s can generate speech with more natural pitch contour (which can result in better prosody) than FastSpeech. We also conduct a case study on generated pitch contours in Appendix D.

对音高，我们计算真实语音与合成语音音高分布的矩（标准差（$\sigma$）、偏度（$\gamma$）和峰度（$\mathcal{K}$））（Andreeva 等，2014；Niebuhr 与 Skarnitzl，2019）以及平均动态时间规整（DTW）（Müller，2007）距离。结果见表 3。可以看到，与 FastSpeech 相比，FastSpeech 2/2s 生成音频的矩（$\sigma$、$\gamma$ 和 $\mathcal{K}$）更接近真实音频，到真实音高的平均 DTW 距离也小于其他方法，说明 FastSpeech 2/2s 能生成比 FastSpeech 更自然的音高曲线（从而带来更好的韵律）。我们还在附录 D 对生成的音高曲线做了案例研究。

Method	FastSpeech	FastSpeech 2	FastSpeech 2s
MAE	0.142	0.131	0.133

Table 4: The mean absolute error (MAE) of the energy in synthesized speech audio.

方法	FastSpeech	FastSpeech 2	FastSpeech 2s
MAE	0.142	0.131	0.133

表 4：合成语音中能量的平均绝对误差（MAE）。

For energy, we compute the mean absolute error (MAE) between the frame-wise energy extracted from the generated waveform and the ground-truth speech. To ensure that the numbers of frames in the synthesized and ground-truth speech are the same, we use the ground-truth duration extracted by MFA in both FastSpeech and FastSpeech 2. The results are shown in Table 4. We can see that the MAE of the energy for FastSpeech 2/2s are smaller than that for FastSpeech, indicating that they both synthesize speech audio with more similar energy to the ground-truth audio.

对能量，我们计算从生成波形提取的帧级能量与真实语音之间的平均绝对误差（MAE）。为确保合成语音与真实语音的帧数一致，FastSpeech 和 FastSpeech 2 都使用 MFA 提取的真实时长。结果见表 4。可以看到，FastSpeech 2/2s 的能量 MAE 小于 FastSpeech，说明它们合成的语音在能量上更接近真实音频。

More Accurate Duration for Model Training

We then analyze the accuracy of the provided duration information to train the duration predictor and the effectiveness of more accurate duration for better voice quality based on FastSpeech. We manually align 50 audio generated by the teacher model and the corresponding text in phoneme level and get the ground-truth phoneme-level duration. We compute the average of absolute phoneme boundary differences (McAuliffe et al. 2017) using the duration from the teacher model of FastSpeech and from MFA as used in this paper respectively. The results are shown in Table 5(a). We can see that MFA can generate more accurate duration than the teacher model of FastSpeech. Next, we replace the duration used in FastSpeech (from teacher model) with that extracted by MFA, and conduct the CMOS (Loizou 2011) test to compare the voice quality between the two FastSpeech models trained with different durations[^7]. The results are listed in Table 5(b) and it can be seen that more accurate duration information improves the voice quality of FastSpeech, which verifies the effectiveness of our improved duration from MFA.

用于模型训练的更准时长

接下来，我们分析提供给时长预测器的时长信息有多准确，以及更准时长能否在 FastSpeech 上带来更好音质。我们在音素级手动对齐教师模型生成的 50 条音频与对应文本，得到真实的音素级时长。然后分别用 FastSpeech 教师模型给出的时长和本文使用的 MFA 时长，计算音素边界绝对差的平均值（McAuliffe 等，2017）。结果见表 5(a)。可以看到，MFA 给出的时长比 FastSpeech 教师模型更准。接着，我们把 FastSpeech 使用的时长（来自教师模型）替换成 MFA 提取的时长，并用 CMOS（Loizou，2011）比较用不同时长训练的两个 FastSpeech 模型的音质[^7]。结果见表 5(b)：更准确的时长信息提升了 FastSpeech 的音质，验证了我们从 MFA 得到的改进时长是有效的。

Method	$\Delta$ (ms)
Duration from teacher model	19.68
Duration from MFA	12.47

Setting	CMOS
FastSpeech + Duration from teacher	0
FastSpeech + Duration from MFA	+0.195

(a) Alignment accuracy comparison.

方法	$\Delta$（毫秒）
来自教师模型的时长	19.68
来自 MFA 的时长	12.47

设置	CMOS
FastSpeech + 来自教师的时长	0
FastSpeech + 来自 MFA 的时长	+0.195

(a) 对齐精度比较。

Pitch and Energy Input

We conduct ablation studies to demonstrate the effectiveness of several variance information of FastSpeech 2 and 2s, including pitch and energy[^8]. We conduct CMOS evaluation for these ablation studies. The results are shown in Table 6. We find that removing the energy (Row 3 in both subtables) in FastSpeech 2 and 2s results in performance drop in terms of voice quality (-0.040 and -0.160 CMOS respectively), indicating that energy is effective for FastSpeech 2 in improving the voice quality, and more effective for FastSpeech 2s. We also find that removing the pitch (Row 4 in both subtables) in FastSpeech 2 and 2s results in -0.245 and -1.130 CMOS respectively, which demonstrates the effectiveness of pitch. When we remove both pitch and energy (the last row in both subtables), the voice quality further drops, indicating that both pitch and energy can help improve the performance of FastSpeech 2 and 2s.

音高与能量输入

我们做消融实验，验证 FastSpeech 2 与 2s 中若干变化信息的有效性，包括音高和能量[^8]。这些消融用 CMOS 评估，结果见表 6。我们发现，去掉能量（两个子表的第 3 行）会使 FastSpeech 2 与 2s 的音质下降（CMOS 分别为 -0.040 和 -0.160），说明能量对提升 FastSpeech 2 的音质有效，对 FastSpeech 2s 更有效。去掉音高（两个子表的第 4 行）会使 FastSpeech 2 与 2s 的 CMOS 分别下降 0.245 和 1.130，说明音高有效。当音高和能量都去掉（两个子表的最后一行）时，音质进一步下降，表明二者都能帮助提升 FastSpeech 2 与 2s 的表现。

Predicting Pitch in Frequency Domain

To study the effectiveness of predicting pitch in frequency domain using continuous wavelet transform (CWT) as described in Section 2.3, we directly fit the pitch contour with mean square error like energy in FastSpeech 2 and 2s. We conduct CMOS evaluation and get CMOS drops of 0.185 and 0.201 for FastSpeech 2 and 2s respectively. We also compute the moments of pitch and average DTW distance to the ground-truth pitch as shown in row 6 (denoeted as FastSpeech 2 - CWT) in Table 3. The results demonstrate that CWT can help model the pitch better and improve the prosody of synthesized speech, and thus obtaining better CMOS score.

在频域预测音高

为研究第 2.3 节所述、用连续小波变换（CWT）在频域预测音高是否有效，我们像能量一样直接用均方误差拟合音高曲线，分别在 FastSpeech 2 与 2s 上评估。CMOS 分别下降 0.185 和 0.201。我们还计算了音高的各阶矩以及到真实音高的平均 DTW 距离，见表 3 第 6 行（记为 FastSpeech 2 - CWT）。结果表明，CWT 有助于更好地建模音高、改善合成语音的韵律，从而获得更好的 CMOS。

Mel-Spectrogram Decoder in FastSpeech 2s

To verify the effectiveness of the mel-spectrogram decoder in FastSpeech 2s on text feature extraction as described in Section 2.4, we remove the mel-spectrogram decoder and conduct CMOS evaluation. It causes a 0.285 CMOS drop, which demonstrates that the mel-spectrogram decoder is essential to high-quality waveform generation.

FastSpeech 2s 中的梅尔频谱解码器

为验证第 2.4 节所述、FastSpeech 2s 中梅尔频谱解码器对文本特征提取的作用，我们去掉该解码器并做 CMOS 评估，结果下降 0.285，说明梅尔频谱解码器对高质量波形生成是必要的。

Setting	CMOS
FastSpeech 2	0
FastSpeech 2 - energy	-0.040
FastSpeech 2 - pitch	-0.245
FastSpeech 2 - pitch - energy	-0.370

Setting	CMOS
FastSpeech 2s	0
FastSpeech 2s - energy	-0.160
FastSpeech 2s - pitch	-1.130
FastSpeech 2s - pitch - energy	-1.355

(a) CMOS comparison for FastSpeech 2.

设置	CMOS
FastSpeech 2	0
FastSpeech 2 - energy	-0.040
FastSpeech 2 - pitch	-0.245
FastSpeech 2 - pitch - energy	-0.370

设置	CMOS
FastSpeech 2s	0
FastSpeech 2s - energy	-0.160
FastSpeech 2s - pitch	-1.130
FastSpeech 2s - pitch - energy	-1.355

(a) FastSpeech 2 的 CMOS 比较。

4 Conclusion

4 结论

In this work, we proposed FastSpeech 2, a fast and high-quality end-to-end TTS system, to address the issues in FastSpeech and ease the one-to-many mapping problem: 1) we directly train the model with ground-truth mel-spectrograms to simplify the training pipeline and also avoid information loss compared with FastSpeech; and 2) we improve the duration accuracy and introduce more variance information including pitch and energy to ease the one-to-many mapping problem, and improve pitch prediction by introducing continuous wavelet transform. Moreover, based on FastSpeech 2, we further developed FastSpeech 2s, a non-autoregressive text-to-waveform generation model, which enjoys the benefit of fully end-to-end inference and achieves faster inference speed. Our experimental results show that FastSpeech 2 and 2s outperform FastSpeech, and FastSpeech 2 can even surpass autoregressive models in terms of voice quality, with much simpler training pipeline while inheriting the advantages of fast, robust and controllable speech synthesis of FastSpeech.

本文提出 FastSpeech 2，这是一个快速且高质量的端到端 TTS 系统，用以解决 FastSpeech 中的问题并缓解一对多映射：1）我们直接用真实梅尔频谱训练模型，从而简化训练流程，并避免相对 FastSpeech 的信息损失；2）我们提高时长精度，并引入音高和能量等更多变化信息来缓解一对多映射，同时用连续小波变换改进音高预测。在此基础上，我们进一步发展出 FastSpeech 2s，这是一个非自回归的文本至波形生成模型，能够完全端到端推理，推理速度也更快。实验表明，FastSpeech 2 与 2s 优于 FastSpeech，而且 FastSpeech 2 在音质上甚至可以超过自回归模型；训练流程简单得多，同时继承了 FastSpeech 快速、稳健、可控的合成优点。

High quality, fast and fully end-to-end training without any external libraries is definitely the ultimate goal of neural TTS and also a very challenging problem. To ensure high quality of FastSpeech 2, we use an external high-performance alignment tool and pitch extraction tools, which may seem a little complicated, but are very helpful for high-quality and fast speech synthesis. We believe there will be more simpler solutions to achieve this goal in the future and we will certainly work on fully end-to-end TTS without external alignment models and tools. We will also consider more variance information (Zhang et al. 2021) to further improve the voice quality and speed up the inference with more light-weight model (Luo et al. 2021).

高质量、快速、且完全不依赖外部库的端到端训练，无疑是神经 TTS 的最终目标，也是一个非常困难的问题。为了保证 FastSpeech 2 的高质量，我们使用了外部的高性能对齐工具和音高提取工具；这看起来可能有些复杂，但对高质量、快速的语音合成很有帮助。我们相信未来会有更简单的方案来实现这一目标，我们自己也会继续研究不依赖外部对齐模型与工具的完全端到端 TTS。我们还将考虑引入更多变化信息（Zhang 等，2021）以进一步提升音质，并用更轻量的模型（Luo 等，2021）加快推理。

Acknowledgments

致谢

This work was supported in part by the National Key R&D Program of China (Grant No.2018AAA0100603), National Natural Science Foundation of China (Grant No.62072397), Zhejiang Natural Science Foundation (Grant No.LR19F020006), National Natural Science Foundation of China (Grant No.61836002) and X Lab, the Second Academy of CASIC, Beijing, 100854, China.

本工作部分得到中国国家重点研发计划（批准号 2018AAA0100603）、国家自然科学基金（批准号 62072397）、浙江省自然科学基金（批准号 LR19F020006）、国家自然科学基金（批准号 61836002）以及中国航天科工集团第二研究院 X 实验室（北京，100854）的资助。

License and Agreement

许可与约定

Any organization or individual is prohibited from using any technology mentioned in this paper to synthesize someone’s speech without his/her consent. The use of relevant technologies must not involve child pornography, terrorism, politics or other illegal acts. If you do not comply with this item, you could be in violation of copyright laws and face the risk of legal action.

任何组织或个人不得在未经本人同意的情况下，使用本文提及的任何技术合成其语音。相关技术的使用不得涉及儿童色情、恐怖主义、政治或其他违法行为。若未遵守本条，可能违反版权法并面临法律风险。

References

参考文献

Andreeva et al. (2014) Bistra Andreeva, Grażyna Demenko, Bernd Möbius, Frank Zimmerer, Jeanin Jügler, and Magdalena Oleskowicz-Popiel. Differences of pitch profiles in germanic and slavic languages. In Fifteenth Annual Conference of the International Speech Communication Association , 2014.

Arik et al. (2017) Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. Deep voice: Real-time neural text-to-speech. arXiv preprint arXiv:1702.07825 , 2017.

Chen et al. (2020) Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, and Tao Qin. Multispeech: Multi-speaker text to speech with transformer. In INTERSPEECH , pp. 4024–4028, 2020.

Chen et al. (2021) Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, sheng zhao, and Tie-Yan Liu. Adaspeech: Adaptive text to speech for custom voice. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=Drynvt7gg4L .

Chu & Peng (2006) Min Chu and Hu Peng. Objective measure for estimating mean opinion score of synthesized speech, April 4 2006. US Patent 7,024,362.

Donahue et al. (2020) Jeff Donahue, Sander Dieleman, Mikołaj Bińkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. arXiv preprint arXiv:2006.03575 , 2020.

Engel et al. (2020) Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing. arXiv preprint arXiv:2001.04643 , 2020.

Fan et al. (2014) Yuchen Fan, Yao Qian, Feng-Long Xie, and Frank K Soong. Tts synthesis with bidirectional lstm based recurrent neural networks. In Fifteenth Annual Conference of the International Speech Communication Association , 2014.

Gadermayr et al. (2020) Michael Gadermayr, Maximilian Tschuchnig, Dorit Merhof, Nils Krämer, Daniel Truhn, and Burkhard Gess. An asymetric cycle-consistency loss for dealing with many-to-one mappings in image translation: A study on thigh mr scans. arXiv preprint arXiv:2004.11001 , 2020.

Gibiansky et al. (2017) Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In Advances in neural information processing systems , pp. 2962–2970, 2017.

Grossmann & Morlet (1984) Alexander Grossmann and Jean Morlet. Decomposition of hardy functions into square integrable wavelets of constant shape. SIAM journal on mathematical analysis , 15(4):723–736, 1984.

Hirose & Tao (2015) Keikichi Hirose and Jianhua Tao. Speech Prosody in Speech Synthesis: Modeling and generation of prosody for high quality and flexible speech synthesis . Springer, 2015.

Ito (2017) Keith Ito. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/ , 2017.

Jayne et al. (2012) Chrisina Jayne, Andreas Lanitis, and Chris Christodoulou. One-to-many neural network mapping techniques for face image synthesis. Expert Systems with Applications , 39(10):9778–9787, 2012.

Kim et al. (2020) Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. arXiv preprint arXiv:2005.11129 , 2020.

Kim et al. (2018) Sungwon Kim, Sang-gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon. Flowavenet: A generative flow for raw audio. arXiv preprint arXiv:1811.02155 , 2018.

Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Kumar et al. (2019) Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing Systems , pp. 14881–14892, 2019.

Łańcucki (2020) Adrian Łańcucki. Fastpitch: Parallel text-to-speech with pitch prediction. arXiv preprint arXiv:2006.06873 , 2020.

Li et al. (2019) Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 6706–6713, 2019.

Lim et al. (2020) Dan Lim, Won Jang, Hyeyeong Park, Bongwan Kim, Jesam Yoon, et al. Jdi-t: Jointly trained duration informed transformer for text-to-speech without explicit alignment. arXiv preprint arXiv:2005.07799 , 2020.

Loizou (2011) Philipos C Loizou. Speech quality assessment. In Multimedia analysis, processing and communications , pp. 623–654. Springer, 2011.

Luo et al. (2021) Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Jinzhu Li, Sheng Zhao, Enhong Chen, and Tie-Yan Liu. Lightspeech: Lightweight and fast text to speech with neural architecture search. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021.

McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech , pp. 498–502, 2017.

Miao et al. (2020) Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flow-tts: A non-autoregressive network for text to speech based on flow. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 7209–7213. IEEE, 2020.

Ming et al. (2016) Huaiping Ming, Dongyan Huang, Lei Xie, Jie Wu, Minghui Dong, and Haizhou Li. Deep bidirectional lstm modeling of timbre and prosody for emotional voice conversion. 2016.

Müller (2007) Meinard Müller. Dynamic time warping. Information retrieval for music and motion , pp. 69–84, 2007.

Niebuhr & Skarnitzl (2019) Oliver Niebuhr and Radek Skarnitzl. Measuring a speaker’s acoustic correlates of pitch–but which? a contrastive analysis based on perceived speaker charisma. In Proceedings of 19th International Congress of Phonetic Sciences , 2019.

Oord et al. (2017) Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv:1711.10433 , 2017.

Peng et al. (2019) Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao. Parallel neural text-to-speech. arXiv preprint arXiv:1905.08459 , 2019.

Ping et al. (2018) Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference on Learning Representations , 2018.

Ping et al. (2019) Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to-speech. In International Conference on Learning Representations , 2019.

Prenger et al. (2019) Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 3617–3621. IEEE, 2019.

Ren et al. (2019) Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems , pp. 3165–3174, 2019.

Ryan (1994) Harold Ryan. Ricker, ormsby; klander, bntterwo-a choice of wavelets, 1994.

Shen et al. (2018) Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 4779–4783. IEEE, 2018.

Sun et al. (2019) Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie-Yan Liu. Token-level ensemble distillation for grapheme-to-phoneme conversion. In INTERSPEECH , 2019.

Suni et al. (2013) Antti Santeri Suni, Daniel Aalto, Tuomo Raitio, Paavo Alku, Martti Vainio, et al. Wavelets for intonation modeling in hmm speech synthesis. In 8th ISCA Workshop on Speech Synthesis, Proceedings, Barcelona, August 31-September 2, 2013 . ISCA, 2013.

Tuteur (1988) Franz B Tuteur. Wavelet transformations in signal detection. IFAC Proceedings Volumes , 21(9):1061–1065, 1988.

Van Den Oord et al. (2016) Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. SSW , 125, 2016.

Van den Oord et al. (2016) Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. In Advances in neural information processing systems , pp. 4790–4798, 2016.

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pp. 5998–6008, 2017.

Wang et al. (2017) Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135 , 2017.

Yamamoto et al. (2020) Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6199–6203. IEEE, 2020.

Ze et al. (2013) Heiga Ze, Andrew Senior, and Mike Schuster. Statistical parametric speech synthesis using deep neural networks. In 2013 ieee international conference on acoustics, speech and signal processing , pp. 7962–7966. IEEE, 2013.

Zeng et al. (2020) Zhen Zeng, Jianzong Wang, Ning Cheng, Tian Xia, and Jing Xiao. Aligntts: Efficient feed-forward text-to-speech system without explicit alignment. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 6714–6718. IEEE, 2020.

Zhang et al. (2021) Chen Zhang, Yi Ren, Xu Tan, Jinglin Liu, Kejun Zhang, Tao Qin, Sheng Zhao, and Tie-Yan Liu. Denoispeech: Denoising text to speech with frame-level noise modeling. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021.

Zhu et al. (2017) Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems , pp. 465–476, 2017.

Appendix A Model Configuration

附录 A 模型配置

Our FastSpeech 2 consists of 4 feed-forward Transformer (FFT) blocks (Ren et al. 2019) in the encoder and the mel-spectrogram decoder. In each FFT block, the dimension of phoneme embeddings and the hidden size of the self-attention are set to 256. The number of attention heads is set to 2 and the kernel sizes of the 1D-convolution in the 2-layer convolutional network after the self-attention layer are set to 9 and 1, with input/output size of 256/1024 for the first layer and 1024/256 in the second layer. The size of the phoneme vocabulary is 76, including punctuations. In the variance predictor, the kernel sizes of the 1D-convolution are set to 3, with input/output sizes of 256/256 for both layers and the dropout rate is set to 0.5. Our waveform decoder consists of 1-layer transposed 1D-convolution with filter size 64 and 30 dilated residual convolution blocks, whose skip channel size and kernel size of 1D-convolution are set to 64 and 3. The configurations of the discriminator in FastSpeech 2s are the same as Parallel WaveGAN (Yamamoto et al. 2020). We list hyperparameters and configurations of all models used in our experiments in Table 7.

我们的 FastSpeech 2 在编码器和梅尔频谱解码器中各使用 4 个前馈 Transformer（FFT）块（Ren 等，2019）。每个 FFT 块中，音素嵌入维度和自注意力隐层大小设为 256，注意力头数为 2；自注意力之后两层一维卷积的核大小分别为 9 和 1，第一层输入/输出为 256/1024，第二层为 1024/256。音素词表大小为 76，包含标点。方差预测器中一维卷积的核大小为 3，两层输入/输出均为 256/256，dropout 设为 0.5。波形解码器包含 1 层滤波器大小为 64 的转置一维卷积，以及 30 个膨胀残差卷积块，其跳跃通道数和一维卷积核大小分别为 64 和 3。FastSpeech 2s 中判别器的配置与 Parallel WaveGAN（Yamamoto 等，2020）相同。实验中所有模型的超参数与配置列于表 7。

Hyperparameter	Transformer TTS	FastSpeech/FastSpeech 2/2s
Phoneme Embedding Dimension	256	256
Pre-net Layers	3	/
Pre-net Hidden	256	/
Encoder Layers	4	4
Encoder Hidden	256	256
Encoder Conv1D Kernel	9	9
Encoder Conv1D Filter Size	1024	1024
Encoder Attention Heads	2	2
Mel-Spectrogram Decoder Layers	4	4
Mel-Spectrogram Decoder Hidden	256	256
Mel-Spectrogram Decoder Conv1D Kernel	9	9
Mel-Spectrogram Decoder Conv1D Filter Size	1024	1024
Mel-Spectrogram Decoder Attention Headers	2	2
Encoder/Decoder Dropout	0.1	0.1
Variance Predictor Conv1D Kernel	/	3
Variance Predictor Conv1D Filter Size	/	256
Variance Predictor Dropout	/	0.5
Waveform Decoder Convolution Blocks	/	30
Waveform Decoder Dilated Conv1D Kernel size	/	3
Waveform Decoder Transposed Conv1D Filter Size	/	64
Waveform Decoder Skip Channlel Size	/	64
Batch Size	48	48/48/12
Total Number of Parameters	24M	23M/27M/28M

Table 7: Hyperparameters of Transformer TTS, FastSpeech and FastSpeech 2/2s.

超参数	Transformer TTS	FastSpeech/FastSpeech 2/2s
音素嵌入维度	256	256
Pre-net 层数	3	/
Pre-net 隐层	256	/
编码器层数	4	4
编码器隐层	256	256
编码器 Conv1D 核	9	9
编码器 Conv1D 滤波器大小	1024	1024
编码器注意力头数	2	2
梅尔频谱解码器层数	4	4
梅尔频谱解码器隐层	256	256
梅尔频谱解码器 Conv1D 核	9	9
梅尔频谱解码器 Conv1D 滤波器大小	1024	1024
梅尔频谱解码器注意力头数	2	2
编码器/解码器 Dropout	0.1	0.1
方差预测器 Conv1D 核	/	3
方差预测器 Conv1D 滤波器大小	/	256
方差预测器 Dropout	/	0.5
波形解码器卷积块数	/	30
波形解码器膨胀 Conv1D 核大小	/	3
波形解码器转置 Conv1D 滤波器大小	/	64
波形解码器跳跃通道大小	/	64
批次大小	48	48/48/12
参数总量	24M	23M/27M/28M

表 7：Transformer TTS、FastSpeech 与 FastSpeech 2/2s 的超参数。

Appendix B Training and Inference

附录 B 训练与推理

We train FastSpeech 2 on 1 NVIDIA V100 GPU, with batchsize of 48 sentences. We use the Adam optimizer (Kingma & Ba 2014) with $\beta_{1}=0.9$, $\beta_{2}=0.98$, $\varepsilon=10^{-9}$ and follow the same learning rate schedule in Vaswani et al. 2017. It takes 160k steps for training until convergence. In the inference process, the output mel-spectrograms of our FastSpeech 2 are transformed into audio samples using pre-trained Parallel WaveGAN (Yamamoto et al. 2020)[^9]. For FastSpeech 2s, we train the model on 2 NVIDIA V100 GPUs, with batchsize of 6 sentences on each GPU. The waveform decoder takes the sliced hidden states corresponding to 20,480 waveform sample clips as input. The optimizer and learning rate schedule for FastSpeech 2s are the same as FastSpeech 2. The details of the adversarial training follow Parallel WaveGAN (Yamamoto et al. 2020). It takes 600k steps for training until convergence for FastSpeech 2s.

我们在 1 块 NVIDIA V100 GPU 上训练 FastSpeech 2，批次大小为 48 句。优化器为 Adam（Kingma 与 Ba，2014），$\beta_{1}=0.9$，$\beta_{2}=0.98$，$\varepsilon=10^{-9}$，学习率日程与 Vaswani 等（2017）相同。训练约 160k 步后收敛。推理时，FastSpeech 2 输出的梅尔频谱用预训练的 Parallel WaveGAN（Yamamoto 等，2020）转换成音频样本[^9]。对 FastSpeech 2s，我们在 2 块 NVIDIA V100 GPU 上训练，每卡批次大小为 6 句。波形解码器以对应 20,480 个波形采样点片段的切片隐状态为输入。FastSpeech 2s 的优化器和学习率日程与 FastSpeech 2 相同。对抗训练的细节遵循 Parallel WaveGAN（Yamamoto 等，2020）。FastSpeech 2s 训练约 600k 步后收敛。

Appendix C Modeling Pitch with Continuous Wavelet Transform

附录 C 用连续小波变换建模音高

C.1 Continuous Wavelet Transform

C.1 连续小波变换

Given a continous pitch contour function $F_{0}$, we can convert it to pitch spectrogram $W(\tau,t)$ using continuous wavelet transform (Tuteur 1988; Grossmann & Morlet 1984):

给定连续音高曲线函数 $F_{0}$，可以用连续小波变换（Tuteur，1988；Grossmann 与 Morlet，1984）把它转换成音高频谱图 $W(\tau,t)$：

$$W(\tau,t)=\tau^{-1/2}\int_{-\infty}^{+\infty}F_{0}(x)\psi(\frac{x-t}{\tau})dx$$

where $\psi$ is the Mexican hat mother wavelet (Ryan 1994), $F_{0}(x)$ is the pitch value in position $x$, $\tau$ and $t$ are scale and position of wavelet respectively. The original pitch contour $F_{0}$ can be recovered from the wavelet representation $W(\tau,t)$ by inverse continuous wavelet transform (iCWT) using the following formula:

其中 $\psi$ 是墨西哥帽母小波（Ryan，1994），$F_{0}(x)$ 是位置 $x$ 处的音高值，$\tau$ 和 $t$ 分别是小波的尺度与位置。原始音高曲线 $F_{0}$ 可以用逆连续小波变换（iCWT）从小波表示 $W(\tau,t)$ 恢复，公式如下：

$$F_{0}(t)=\ \int_{-\infty}^{+\infty}\int_{0}^{+\infty}W\left(\tau,t\right)\tau^{-5/2}\psi\left(\frac{x-t}{\tau}\right)dxd\tau$$

Suppose that we decompose the pitch contour $F_{0}$ into 10 scales (Ming et al. 2016), $F_{0}$ can be represented by 10 separate components given by:

若把音高曲线 $F_{0}$ 分解成 10 个尺度（Ming 等，2016），则 $F_{0}$ 可以由以下 10 个独立分量表示：

$$W_{i}(t)=\
W(2^{i+1}\tau_{0},t)(i+2.5)^{-5/2} \tag{1}$$

where $i=1,...,10$ and $\tau_{0}=5ms$, which is originally proposed in Suni et al. 2013. Given 10 wavelet components $\hat{W}_{i}(t)$, we can recompose pitch contour $\hat{F_{0}}$ by the following formula (Ming et al. 2016):

其中 $i=1,...,10$，$\tau_{0}=5ms$，这一设定最初由 Suni 等（2013）提出。给定 10 个小波分量 $\hat{W}_{i}(t)$，可以用以下公式（Ming 等，2016）重建音高曲线 $\hat{F_{0}}$：

$$\hat{F_{0}}(t)=\sum_{i=1}^{10}\hat{W}_{i}(t)(i+2.5)^{-5/2} \tag{2}$$

C.2 Implementation Details

C.2 实现细节

Figure 2: Details in pitch predictor. CWT and iCWT denote continuous wavelet transform and inverse continuous wavelet transform respectively.

图 2：音高预测器的细节。CWT 与 iCWT 分别表示连续小波变换和逆连续小波变换。

First we extract the pitch contour using PyWorldVocoder[^10]. Since CWT is very sensitive to discontinuous signals, we preprocess the pitch contour as follows: 1) we use linear interpolation to fill the unvoiced frame in pitch contour; 2) we transform the resulting pitch contour to logarithmic scale; 3) we normalize it to zero mean and unit variance for each utterance, and we have to save the original utterance-level mean and variance for pitch contour reconstruction; and 4) we convert the normalized pitch contour to pitch spectrogram using continuous wavelet transform following Equation 1.

我们首先用 PyWorldVocoder[^10] 提取音高曲线。由于 CWT 对不连续信号非常敏感，我们对音高曲线做如下预处理：1）用线性插值填补音高曲线中的清音帧；2）把得到的音高曲线转换到对数尺度；3）对每条语句归一化到零均值、单位方差，并保存原始的语句级均值和方差，供重建音高曲线时使用；4）按公式 1，用连续小波变换把归一化后的音高曲线转换成音高频谱图。

As shown in Figure 2, pitch predictor consists of a 2-layer 1D-convolutional network with ReLU activation, each followed by the layer normalization and the dropout layer, and an extra linear layer to project the hidden states into the pitch spectrogram. To predict the mean/variance of recovered pitch contour for each utterance, we average the hidden states output by the 1D-convolutional network on the time dimension to a global vector and project it to mean and variance using a linear layer.

如图 2 所示，音高预测器由带 ReLU 激活的两层一维卷积网络构成，每层之后接层归一化和 dropout，再用一个线性层把隐状态投影到音高频谱图。为了预测每条语句重建音高曲线的均值/方差，我们把一维卷积网络输出的隐状态在时间维上平均成一个全局向量，再用线性层投影到均值和方差。

We train the pitch predictor with ground-truth pitch spectrogram and the mean/variance of pitch contour and optimize it with mean square error. During inference, we predict the pitch spectrogram and the mean/variance of recovered pitch contour using pitch predictor, inverse the pitch spectrogram to pitch contour with inverse continuous wavelet transform (iCWT) following Equation 2, and finally denormalize it with the predicted mean/variance.

我们用真实音高频谱图以及音高曲线的均值/方差训练音高预测器，并以均方误差优化。推理时，用音高预测器预测音高频谱图以及重建音高曲线的均值/方差，再按公式 2 用逆连续小波变换（iCWT）把频谱图还原成音高曲线，最后用预测的均值/方差做反归一化。

Appendix D Case Study on Pitch Contour

附录 D 音高曲线案例研究

In this section, we conduct the case study on pitch contours of the audios generated by different methods. We randomly choose 1 utterance from the test set and plot the pitch countor of ground-truth audio samples and that generated by FastSpeech, FastSpeech 2, FastSpeech 2s in Figure 3. We can see that FastSpeech 2 and 2s can capture the variations in pitch better than FastSpeech thanks to taking pitch information as input.

本节对不同方法生成音频的音高曲线做案例研究。我们从测试集中随机选 1 条语句，画出真实音频以及 FastSpeech、FastSpeech 2、FastSpeech 2s 生成音频的音高曲线，见图 3。可以看到，由于把音高信息作为输入，FastSpeech 2 与 2s 比 FastSpeech 更能捕捉音高变化。

Figure 3: Pitch contours extracted from generated and ground-truth audio samples. We only plot the voiced part of pitch contour. The input text is “The worst, which perhaps was the English, was a terrible falling-off from the work of the earlier presses”.

图 3：从生成音频和真实音频中提取的音高曲线。我们只画出浊音部分的音高。输入文本为 “The worst, which perhaps was the English, was a terrible falling-off from the work of the earlier presses”。

Appendix E Variance Control

附录 E 变化信息控制

FastSpeech 2 and 2s introduce several variance information to ease the one-to-many mapping problem in TTS. As a byproduct, they also make the synthesized speech more controllable and can be used to manually control pitch, duration and energy (volume) of synthesized audio. As a demonstration, we manipulate pitch input to control the pitch of synthesized speech in this subsubsection. We show the mel-spectrograms before and after the pitch manipulation in Figure 4. From the samples, we can see that FastSpeech 2 generates high-quality mel-spectrograms after adjusting the $\hat{F_{0}}$ from 0.75 to 1.50 times. Such manipulation can also be applied to FastSpeech 2s and the results are put in the supplementary materials. We also put the audio samples controlled by other variance information in supplementary materials.

FastSpeech 2 与 2s 引入若干变化信息来缓解 TTS 中的一对多映射。作为副产品，它们也使合成语音更可控，可用于手动控制合成音频的音高、时长和能量（音量）。作为演示，我们在本小节通过操纵音高输入来控制合成语音的音高，并在图 4 中展示调整前后的梅尔频谱。从样例可以看到，把 $\hat{F_{0}}$ 从 0.75 倍调到 1.50 倍后，FastSpeech 2 仍能生成高质量的梅尔频谱。这种操纵也可以用在 FastSpeech 2s 上，结果放在补充材料中。由其他变化信息控制的音频样例也放在补充材料中。

Figure 4: The mel-spectrograms of the voice with different $\hat{F_0}$. $F_0$ is the fundamental frequency of original audio. The red curves denote $\hat{F_0}$ contours. The input text is “They discarded this for a more completely Roman and far less beautiful letter.”

图 4：不同 $\hat{F_0}$ 下的语音梅尔频谱。$F_0$ 是原始音频的基频，红色曲线表示 $\hat{F_0}$ 轮廓。输入文本为 “They discarded this for a more completely Roman and far less beautiful letter.”。