FastSpeech: Fast, Robust and Controllable Text to Speech

FastSpeech：快速、稳健且可控的文本转语音

Yi Ren Thanks:  Equal contribution. Affiliation:  Zhejiang University Email:  rayeren@zju.edu.cn    Yangjun Ruan 1 1 footnotemark: 1 Affiliation:  Zhejiang University Email:  ruanyj3107@zju.edu.cn    Xu Tan Affiliation:  Microsoft Research Email:  xuta@microsoft.com    Tao Qin Affiliation:  Microsoft Research Email:  taoqin@microsoft.com    Sheng Zhao Affiliation:  Microsoft STC Asia Email:  Sheng.Zhao@microsoft.com    Zhou Zhao Thanks:  Corresponding author Affiliation:  Zhejiang University Email:  zhaozhou@zju.edu.cn    Tie-Yan Liu Affiliation:  Microsoft Research Email:  tyliu@microsoft.com

Abstract

摘要

Neural network based end-to-end text to speech (TTS) has significantly improved the quality of synthesized speech. Prominent methods (e.g., Tacotron 2) usually first generate mel-spectrogram from text, and then synthesize speech from the mel-spectrogram using vocoder such as WaveNet. Compared with traditional concatenative and statistical parametric approaches, neural network based end-to-end models suffer from slow inference speed, and the synthesized speech is usually not robust (i.e., some words are skipped or repeated) and lack of controllability (voice speed or prosody control). In this work, we propose a novel feed-forward network based on Transformer to generate mel-spectrogram in parallel for TTS. Specifically, we extract attention alignments from an encoder-decoder based teacher model for phoneme duration prediction, which is used by a length regulator to expand the source phoneme sequence to match the length of the target mel-spectrogram sequence for parallel mel-spectrogram generation. Experiments on the LJSpeech dataset show that our parallel model matches autoregressive models in terms of speech quality, nearly eliminates the problem of word skipping and repeating in particularly hard cases, and can adjust voice speed smoothly. Most importantly, compared with autoregressive Transformer TTS, our model speeds up mel-spectrogram generation by 270x and the end-to-end speech synthesis by 38x. Therefore, we call our model FastSpeech. [^1]

基于神经网络的端到端文本转语音（TTS）显著提升了合成语音质量。代表性方法（如 Tacotron 2）通常先由文本生成梅尔频谱，再用 WaveNet 等声码器合成波形。相对传统拼接与统计参数方法，神经端到端模型推断偏慢，合成语音往往不够稳健（漏词或重复），且缺乏可控性（语速或韵律控制）。本文提出基于 Transformer 的新型前馈网络，并行生成 TTS 用梅尔频谱。具体地，我们从编码器–解码器教师模型提取注意力对齐用于音素时长预测，再由长度调节器把源音素序列扩展到与目标梅尔频谱序列等长，从而实现并行梅尔频谱生成。在 LJSpeech 上的实验表明：并行模型在语音质量上可匹配自回归模型，在特别困难的样本上几乎消除漏词与重复，并能平滑调节语速。尤为重要的是，相对自回归 Transformer TTS，梅尔频谱生成加速约 270 倍，端到端语音合成加速约 38 倍。因此我们称之为 FastSpeech。[^1]

1 Introduction

1 引言

Text to speech (TTS) has attracted a lot of attention in recent years due to the advance in deep learning. Deep neural network based systems have become more and more popular for TTS, such as Tacotron [27], Tacotron 2 [22], Deep Voice 3 [19], and the fully end-to-end ClariNet [18]. Those models usually first generate mel-spectrogram autoregressively from text input and then synthesize speech from the mel-spectrogram using vocoder such as Griffin-Lim [6], WaveNet [24], Parallel WaveNet [16], or WaveGlow [20][^2]. Neural network based TTS has outperformed conventional concatenative and statistical parametric approaches [9, 28] in terms of speech quality.

近年来深度学习推动下，文本转语音（TTS）备受关注。深度神经网络系统日益流行，如 Tacotron [27]、Tacotron 2 [22]、Deep Voice 3 [19] 以及全端到端的 ClariNet [18]。这些模型通常先自回归地由文本生成梅尔频谱，再用 Griffin-Lim [6]、WaveNet [24]、Parallel WaveNet [16] 或 WaveGlow [20] 等声码器合成语音[^2]。神经 TTS 在语音质量上已超过传统拼接与统计参数方法 [9, 28]。

In current neural network based TTS systems, mel-spectrogram is generated autoregressively. Due to the long sequence of the mel-spectrogram and the autoregressive nature, those systems face several challenges:

当前神经 TTS 中梅尔频谱多为自回归生成。由于梅尔频谱序列很长且自回归依赖，系统面临若干挑战：

• • Slow inference speed for mel-spectrogram generation. Although CNN and Transformer based TTS [19, 14] can speed up the training over RNN-based models [22], all models generate a mel-spectrogram conditioned on the previously generated mel-spectrograms and suffer from slow inference speed, given the mel-spectrogram sequence is usually with a length of hundreds or thousands.

• • 梅尔频谱生成推断慢。虽 CNN 与 Transformer TTS [19, 14] 相对 RNN 模型 [22] 可加快训练，但所有模型仍条件于已生成的梅尔频谱逐步生成；梅尔频谱长度常达数百至数千，推断仍然偏慢。

• • Synthesized speech is usually not robust. Due to error propagation [3] and the wrong attention alignments between text and speech in the autoregressive generation, the generated mel-spectrogram is usually deficient with the problem of words skipping and repeating [19].

• • 合成语音通常不够稳健。自回归生成中的误差传播 [3] 以及文本–语音注意力对齐错误，会使梅尔频谱残缺，出现漏词、重复等问题 [19]。

• • Synthesized speech is lack of controllability. Previous autoregressive models generate mel-spectrograms one by one automatically, without explicitly leveraging the alignments between text and speech. As a consequence, it is usually hard to directly control the voice speed and prosody in the autoregressive generation.

• • 合成语音缺乏可控性。先前自回归模型逐帧自动生成梅尔频谱，未显式利用文本–语音对齐，因而难以直接控制语速与韵律。

Considering the monotonous alignment between text and speech, to speed up mel-spectrogram generation, in this work, we propose a novel model, FastSpeech, which takes a text (phoneme) sequence as input and generates mel-spectrograms non-autoregressively. It adopts a feed-forward network based on the self-attention in Transformer [25] and 1D convolution [5, 19, 11]. Since a mel-spectrogram sequence is much longer than its corresponding phoneme sequence, in order to solve the problem of length mismatch between the two sequences, FastSpeech adopts a length regulator that up-samples the phoneme sequence according to the phoneme duration (i.e., the number of mel-spectrograms that each phoneme corresponds to) to match the length of the mel-spectrogram sequence. The regulator is built on a phoneme duration predictor, which predicts the duration of each phoneme.

考虑到文本与语音之间近乎单调的对齐，为加速梅尔频谱生成，我们提出 FastSpeech：以文本（音素）序列为输入，非自回归地生成梅尔频谱。结构基于 Transformer [25] 自注意力与一维卷积 [5, 19, 11] 的前馈网络。梅尔频谱通常远长于对应音素序列，为解决长度不匹配，FastSpeech 用长度调节器按音素时长（每个音素对应的梅尔帧数）上采样音素序列，使之与梅尔频谱等长；调节器建立在音素时长预测器之上。

Our proposed FastSpeech can address the above-mentioned three challenges as follows:

所提 FastSpeech 可如下应对上述三项挑战：

• • Through parallel mel-spectrogram generation, FastSpeech greatly speeds up the synthesis process.

• • 通过并行生成梅尔频谱，大幅加速合成。

• • Phoneme duration predictor ensures hard alignments between a phoneme and its mel-spectrograms, which is very different from soft and automatic attention alignments in the autoregressive models. Thus, FastSpeech avoids the issues of error propagation and wrong attention alignments, consequently reducing the ratio of the skipped words and repeated words.

• • 音素时长预测器保证音素与其梅尔帧之间的硬对齐，迥异于自回归模型中的软自动注意力对齐，从而避免误差传播与错误注意力，降低漏词与重复比例。

• • The length regulator can easily adjust voice speed by lengthening or shortening the phoneme duration to determine the length of the generated mel-spectrograms, and can also control part of the prosody by adding breaks between adjacent phonemes.

• • 长度调节器可通过拉长或缩短音素时长决定生成梅尔频谱长度，从而调节语速；也可在相邻音素间插入停顿，控制部分韵律。

We conduct experiments on the LJSpeech dataset to test FastSpeech. The results show that in terms of speech quality, FastSpeech nearly matches the autoregressive Transformer model. Furthermore, FastSpeech achieves 270x speedup on mel-spectrogram generation and 38x speedup on final speech synthesis compared with the autoregressive Transformer TTS model, almost eliminates the problem of word skipping and repeating, and can adjust voice speed smoothly. We attach some audio files generated by our method in the supplementary materials.

我们在 LJSpeech 上评测 FastSpeech。结果显示：语音质量几乎匹配自回归 Transformer；相对自回归 Transformer TTS，梅尔频谱生成加速 270 倍、最终语音合成加速 38 倍；几乎消除漏词与重复，并能平滑调节语速。补充材料附有若干音频样例。

2 Background

2 背景

In this section, we briefly overview the background of this work, including text to speech, sequence to sequence learning, and non-autoregressive sequence generation.

本节简要回顾工作背景，包括文本转语音、序列到序列学习以及非自回归序列生成。

Text to Speech

TTS [1, 27, 22, 18, 21], which aims to synthesize natural and intelligible speech given text, has long been a hot research topic in the field of artificial intelligence. The research on TTS has shifted from early concatenative synthesis [9], statistical parametric synthesis [28, 13] to neural network based parametric synthesis [1] and end-to-end models [27, 22, 14, 18], and the quality of the synthesized speech by end-to-end models is close to human parity. Neural network based end-to-end TTS models usually first convert the text to acoustic features (e.g., mel-spectrograms) and then transform mel-spectrograms into audio samples. However, most neural TTS systems generate mel-spectrograms autoregressively, which suffers from slow inference speed, and synthesized speech usually lacks of robustness (word skipping and repeating) and controllability (voice speed or prosody control). In this work, we propose FastSpeech to generate mel-spectrograms non-autoregressively, which sufficiently handles the above problems.

文本转语音

TTS [1, 27, 22, 18, 21] 旨在给定文本合成自然、可懂的语音，长期是人工智能热点。研究从早期拼接合成 [9]、统计参数合成 [28, 13] 转向神经参数合成 [1] 与端到端模型 [27, 22, 14, 18]，端到端合成质量已接近真人。神经端到端 TTS 通常先把文本转为声学特征（如梅尔频谱），再把梅尔频谱变成波形。但多数系统仍自回归生成梅尔频谱，推断慢，且语音常缺稳健性（漏词、重复）与可控性（语速或韵律）。本文提出 FastSpeech 非自回归生成梅尔频谱，以充分应对上述问题。

Sequence to Sequence Learning

Sequence to sequence learning [2, 4, 25] is usually built on the encoder-decoder framework: The encoder takes the source sequence as input and generates a set of representations. After that, the decoder estimates the conditional probability of each target element given the source representations and its preceding elements. The attention mechanism [2] is further introduced between the encoder and decoder in order to find which source representations to focus on when predicting the current element, and is an important component for sequence to sequence learning.

序列到序列学习

序列到序列学习 [2, 4, 25] 通常基于编码器–解码器：编码器吃源序列并产生一组表示；解码器在源表示与已生成前缀条件下估计每个目标元素的条件概率。编码器与解码器之间的注意力 [2] 用于决定预测当前元素时关注哪些源表示，是序列到序列学习的重要组件。

In this work, instead of using the conventional encoder-attention-decoder framework for sequence to sequence learning, we propose a feed-forward network to generate a sequence in parallel.

本文不采用传统编码器–注意力–解码器框架做序列到序列学习，而提出前馈网络并行生成序列。

Non-Autoregressive Sequence Generation

Unlike autoregressive sequence generation, non-autoregressive models generate sequence in parallel, without explicitly depending on the previous elements, which can greatly speed up the inference process. Non-autoregressive generation has been studied in some sequence generation tasks such as neural machine translation  [7, 8, 26] and audio synthesis [16, 18, 20]. Our FastSpeech differs from the above works in two aspects: 1) Previous works adopt non-autoregressive generation in neural machine translation or audio synthesis mainly for inference speedup, while FastSpeech focuses on both inference speedup and improving the robustness and controllability of the synthesized speech in TTS. 2) For TTS, although Parallel WaveNet [16], ClariNet [18] and WaveGlow [20] generate audio in parallel, they are conditioned on mel-spectrograms, which are still generated autoregressively. Therefore, they do not address the challenges considered in this work. There is a concurrent work [17] that also generates mel-spectrogram in parallel. However, it still adopts the encoder-decoder framework with attention mechanism, which 1) requires 2$\sim$3x model parameters compared with the teacher model and thus achieves slower inference speedup than FastSpeech; 2) cannot totally solve the problems of word skipping and repeating while FastSpeech nearly eliminates these issues.

非自回归序列生成

与自回归不同，非自回归模型并行生成序列，不显式依赖先前元素，可大幅加速推断。非自回归已在神经机器翻译 [7, 8, 26] 与音频合成 [16, 18, 20] 等任务中研究。FastSpeech 与上述工作有两点不同：1）先前工作多主要为加速推断，而 FastSpeech 同时关注加速以及提升 TTS 合成的稳健性与可控性；2）Parallel WaveNet [16]、ClariNet [18]、WaveGlow [20] 虽并行生成波形，但仍条件于自回归生成的梅尔频谱，因而未解决本文关注的挑战。同期工作 [17] 也并行生成梅尔频谱，但仍用带注意力的编码器–解码器：相对教师模型参数约多 2～3 倍，加速不如 FastSpeech；也不能像 FastSpeech 那样几乎消除漏词与重复。

3 FastSpeech

In this section, we introduce the architecture design of FastSpeech. To generate a target mel-spectrogram sequence in parallel, we design a novel feed-forward structure, instead of using the encoder-attention-decoder based architecture as adopted by most sequence to sequence based autoregressive [25, 22, 14] and non-autoregressive [7, 8, 26] generation. The overall model architecture of FastSpeech is shown in Figure 1. We describe the components in detail in the following subsections.

本节介绍 FastSpeech 结构设计。为并行生成目标梅尔频谱序列，我们设计新型前馈结构，而不采用多数自回归 [25, 22, 14] 与非自回归 [7, 8, 26] 生成所用的编码器–注意力–解码器架构。总体结构见图 1。下文分组件详述。

(a) Feed-Forward Transformer

(a) 前馈 Transformer

3.1 Feed-Forward Transformer

3.1 前馈 Transformer

The architecture for FastSpeech is a feed-forward structure based on self-attention in Transformer [25] and 1D convolution [5, 19]. We call this structure as Feed-Forward Transformer (FFT), as shown in Figure 1(a). Feed-Forward Transformer stacks multiple FFT blocks for phoneme to mel-spectrogram transformation, with $N$ blocks on the phoneme side, and $N$ blocks on the mel-spectrogram side, with a length regulator (which will be described in the next subsection) in between to bridge the length gap between the phoneme and mel-spectrogram sequence. Each FFT block consists of a self-attention and 1D convolutional network, as shown in Figure 1(b). The self-attention network consists of a multi-head attention to extract the cross-position information. Different from the 2-layer dense network in Transformer, we use a 2-layer 1D convolutional network with ReLU activation. The motivation is that the adjacent hidden states are more closely related in the character/phoneme and mel-spectrogram sequence in speech tasks. We evaluate the effectiveness of the 1D convolutional network in the experimental section. Following Transformer [25], residual connections, layer normalization, and dropout are added after the self-attention network and 1D convolutional network respectively.

FastSpeech 的结构是基于 Transformer [25] 自注意力与一维卷积 [5, 19] 的前馈网络，称为 Feed-Forward Transformer（FFT），见图 1(a)。FFT 在音素侧堆叠 $N$ 个 FFT 块、在梅尔频谱侧再堆叠 $N$ 个，中间以长度调节器（下一小节）弥合音素与梅尔频谱长度差。每个 FFT 块含自注意力与一维卷积网络，见图 1(b)。自注意力用多头注意力提取跨位置信息。与 Transformer 中两层全连接不同，我们用带 ReLU 的两层一维卷积：动机是语音任务中字符/音素与梅尔频谱序列的相邻隐状态关系更紧。实验部分评估了一维卷积的有效性。遵循 Transformer [25]，自注意力与一维卷积后分别加残差连接、层归一化与 dropout。

3.2 Length Regulator

3.2 长度调节器

The length regulator (Figure 1(c)) is used to solve the problem of length mismatch between the phoneme and spectrogram sequence in the Feed-Forward Transformer, as well as to control the voice speed and part of prosody. The length of a phoneme sequence is usually smaller than that of its mel-spectrogram sequence, and each phoneme corresponds to several mel-spectrograms. We refer to the length of the mel-spectrograms that corresponds to a phoneme as the phoneme duration (we will describe how to predict phoneme duration in the next subsection). Based on the phoneme duration $d$, the length regulator expands the hidden states of the phoneme sequence $d$ times, and then the total length of the hidden states equals the length of the mel-spectrograms. Denote the hidden states of the phoneme sequence as $\mathcal{H}_{pho}=[h_{1},h_{2},...,h_{n}]$, where $n$ is the length of the sequence. Denote the phoneme duration sequence as $\mathcal{D}=[d_{1},d_{2},...,d_{n}]$, where $\Sigma^{n}_{i=1}d_{i}=m$ and $m$ is the length of the mel-spectrogram sequence. We denote the length regulator $\mathcal{LR}$ as

长度调节器（图 1(c)）用于解决前馈 Transformer 中音素与频谱序列长度不匹配，并控制语速与部分韵律。音素序列通常短于对应梅尔频谱，每个音素对应若干梅尔帧；我们把对应帧数称为音素时长（预测方法见下一小节）。基于时长 $d$，调节器把音素隐状态扩展 $d$ 倍，使总长度等于梅尔频谱长度。记音素隐状态为 $\mathcal{H}_{pho}=[h_{1},h_{2},...,h_{n}]$，$n$ 为序列长；时长序列为 $\mathcal{D}=[d_{1},d_{2},...,d_{n}]$，其中 $\Sigma^{n}_{i=1}d_{i}=m$，$m$ 为梅尔频谱长度。长度调节器 $\mathcal{LR}$ 记为

$$\mathcal{H}_{mel}=\mathcal{LR}(\mathcal{H}_{pho},\mathcal{D},\alpha), \tag{1}$$

where $\alpha$ is a hyperparameter to determine the length of the expanded sequence $\mathcal{H}_{mel}$, thereby controlling the voice speed. For example, given $\mathcal{H}_{pho}=[h_{1},h_{2},h_{3},h_{4}]$ and the corresponding phoneme duration sequence $\mathcal{D}=[2,2,3,1]$, the expanded sequence $\mathcal{H}_{mel}$ based on Equation 1 becomes $[h_{1},h_{1},h_{2},h_{2},h_{3},h_{3},h_{3},h_{4}]$ if $\alpha=1$ (normal speed). When $\alpha=1.3$ (slow speed) and $0.5$ (fast speed), the duration sequences become $\mathcal{D}_{\alpha=1.3}=[2.6,2.6,3.9,1.3]\approx[3,3,4,1]$ and $\mathcal{D}_{\alpha=0.5}=[1,1,1.5,0.5]\approx[1,1,2,1]$, and the expanded sequences become $[h_{1},h_{1},h_{1},h_{2},h_{2},h_{2},h_{3},h_{3},h_{3},h_{3},h_{4}]$ and $[h_{1},h_{2},h_{3},h_{3},h_{4}]$ respectively. We can also control the break between words by adjusting the duration of the space characters in the sentence, so as to adjust part of prosody of the synthesized speech.

其中 $\alpha$ 为超参，决定扩展后序列 $\mathcal{H}_{mel}$ 的长度，从而控制语速。例如给定 $\mathcal{H}_{pho}=[h_{1},h_{2},h_{3},h_{4}]$、$\mathcal{D}=[2,2,3,1]$，当 $\alpha=1$（常速）时按式 (1) 得到 $[h_{1},h_{1},h_{2},h_{2},h_{3},h_{3},h_{3},h_{4}]$；当 $\alpha=1.3$（慢速）与 $0.5$（快速）时，时长变为 $\mathcal{D}_{\alpha=1.3}\approx[3,3,4,1]$、$\mathcal{D}_{\alpha=0.5}\approx[1,1,2,1]$，扩展序列相应变长或变短。也可通过调节句中空格字符的时长控制词间停顿，从而调整部分韵律。

3.3 Duration Predictor

3.3 时长预测器

Phoneme duration prediction is important for the length regulator. As shown in Figure 1(d), the duration predictor consists of a 2-layer 1D convolutional network with ReLU activation, each followed by the layer normalization and the dropout layer, and an extra linear layer to output a scalar, which is exactly the predicted phoneme duration. Note that this module is stacked on top of the FFT blocks on the phoneme side and is jointly trained with the FastSpeech model to predict the length of mel-spectrograms for each phoneme with the mean square error (MSE) loss. We predict the length in the logarithmic domain, which makes them more Gaussian and easier to train. Note that the trained duration predictor is only used in the TTS inference phase, because we can directly use the phoneme duration extracted from an autoregressive teacher model in training (see following discussions).

音素时长预测对长度调节器很关键。如图 1(d)，时长预测器由带 ReLU 的两层一维卷积组成，每层后接层归一化与 dropout，再经线性层输出标量，即预测的音素时长。该模块叠在音素侧 FFT 块之上，与 FastSpeech 联合训练，用均方误差（MSE）预测每个音素对应的梅尔帧数；在对数域预测时长，分布更接近高斯、更易训练。注意训练好的时长预测器只在 TTS 推断阶段使用，训练时可直接用自回归教师模型提取的音素时长（见下文）。

In order to train the duration predictor, we extract the ground-truth phoneme duration from an autoregressive teacher TTS model, as shown in Figure 1d. We describe the detailed steps as follows:

为训练时长预测器，我们从自回归教师 TTS 提取真值音素时长，见图 1d。步骤如下：

• • We first train an autoregressive encoder-attention-decoder based Transformer TTS model following [14].

• • 先按 [14] 训练基于编码器–注意力–解码器的自回归 Transformer TTS。

• • For each training sequence pair, we extract the decoder-to-encoder attention alignments from the trained teacher model. There are multiple attention alignments due to the multi-head self-attention [25], and not all attention heads demonstrate the diagonal property (the phoneme and mel-spectrogram sequence are monotonously aligned). We propose a focus rate $F$ to measure how an attention head is close to diagonal: $F=\frac{1}{S}\sum_{s=1}^{S}{\max_{1\leq t\leq T}a_{s,t}}$, where $S$ and $T$ are the lengths of the ground-truth spectrograms and phonemes, $a_{s,t}$ donates the element in the $s$-th row and $t$-th column of the attention matrix. We compute the focus rate for each head and choose the head with the largest $F$ as the attention alignments.

• • 对每个训练序列对，从训练好的教师模型提取解码器到编码器的注意力对齐。多头自注意力 [25] 会产生多个对齐，并非所有头都呈对角（音素与梅尔频谱单调对齐）。我们提出聚焦率 $F$ 衡量注意力头接近对角的程度：$F=\frac{1}{S}\sum_{s=1}^{S}{\max_{1\leq t\leq T}a_{s,t}}$，其中 $S$、$T$ 为真值频谱与音素长度，$a_{s,t}$ 为注意力矩阵第 $s$ 行第 $t$ 列元素。计算各头聚焦率，取 $F$ 最大的头作为对齐。

• • Finally, we extract the phoneme duration sequence $\mathcal{D}=[d_{1},d_{2},...,d_{n}]$ according to the duration extractor $d_{i}=\sum_{s=1}^{S}[\mathop{\arg\max}_{t}a_{s,t}=i]$. That is, the duration of a phoneme is the number of mel-spectrograms attended to it according to the attention head selected in the above step.

• • 最后按时长提取器 $d_{i}=\sum_{s=1}^{S}[\mathop{\arg\max}_{t}a_{s,t}=i]$ 得到音素时长序列 $\mathcal{D}=[d_{1},d_{2},...,d_{n}]$，即某音素的时长等于按所选注意力头归到它的梅尔帧数。

4 Experimental Setup

4 实验设置

4.1 Datasets

4.1 数据集

We conduct experiments on LJSpeech dataset [10], which contains 13,100 English audio clips and the corresponding text transcripts, with the total audio length of approximate 24 hours. We randomly split the dataset into 3 sets: 12500 samples for training, 300 samples for validation and 300 samples for testing. In order to alleviate the mispronunciation problem, we convert the text sequence into the phoneme sequence with our internal grapheme-to-phoneme conversion tool [23], following [1, 27, 22]. For the speech data, we convert the raw waveform into mel-spectrograms following [22]. Our frame size and hop size are set to 1024 and 256, respectively.

我们在 LJSpeech [10] 上实验：含 13,100 条英语音频及对应文本，总时长约 24 小时。随机划分为训练 12,500、验证 300、测试 300。为减轻错误发音，按 [1, 27, 22] 用内部字素到音素工具 [23] 把文本转为音素序列。语音按 [22] 转成梅尔频谱，帧长 1024、 hop 256。

In order to evaluate the robustness of our proposed FastSpeech, we also choose 50 sentences which are particularly hard for TTS system, following the practice in [19].

为评估稳健性，另选 50 句对 TTS 特别困难的句子，做法参考 [19]。

4.2 Model Configuration

4.2 模型配置

FastSpeech model

Our FastSpeech model consists of 6 FFT blocks on both the phoneme side and the mel-spectrogram side. The size of the phoneme vocabulary is 51, including punctuations. The dimension of phoneme embeddings, the hidden size of the self-attention and 1D convolution in the FFT block are all set to 384. The number of attention heads is set to 2. The kernel sizes of the 1D convolution in the 2-layer convolutional network are both set to 3, with input/output size of 384/1536 for the first layer and 1536/384 in the second layer. The output linear layer converts the 384-dimensional hidden into 80-dimensional mel-spectrogram. In our duration predictor, the kernel sizes of the 1D convolution are set to 3, with input/output sizes of 384/384 for both layers.

FastSpeech 模型

音素侧与梅尔频谱侧各 6 个 FFT 块。音素词表大小 51（含标点）。音素嵌入维、自注意力隐层与 FFT 中一维卷积隐层均为 384；注意力头数 2。两层一维卷积核大小均为 3，第一层输入/输出 384/1536，第二层 1536/384。输出线性层把 384 维隐状态映到 80 维梅尔频谱。时长预测器一维卷积核大小 3，两层输入/输出均为 384/384。

Autoregressive Transformer TTS model

The autoregressive Transformer TTS model serves two purposes in our work: 1) to extract the phoneme duration as the target to train the duration predictor; 2) to generate mel-spectrogram in the sequence-level knowledge distillation (which will be introduced in the next subsection). We refer to [14] for the configurations of this model, which consists of a 6-layer encoder, a 6-layer decoder, except that we use 1D convolution network instead of position-wise FFN. The number of parameters of this teacher model is similar to that of our FastSpeech model.

自回归 Transformer TTS 模型

该模型有两个用途：1）提取音素时长作为时长预测器的训练目标；2）在序列级知识蒸馏中生成梅尔频谱（下一小节介绍）。配置参考 [14]：6 层编码器、6 层解码器，但用一维卷积网络替换位置前馈网络。教师参数量与 FastSpeech 相近。

4.3 Training and Inference

4.3 训练与推断

We first train the autoregressive Transformer TTS model on 4 NVIDIA V100 GPUs, with batchsize of 16 sentences on each GPU. We use the Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.98$, $\varepsilon=10^{-9}$ and follow the same learning rate schedule in [25]. It takes 80k steps for training until convergence. We feed the text and speech pairs in the training set to the model again to obtain the encoder-decoder attention alignments, which are used to train the duration predictor. In addition, we also leverage sequence-level knowledge distillation [12] that has achieved good performance in non-autoregressive machine translation [7, 8, 26] to transfer the knowledge from the teacher model to the student model. For each source text sequence, we generate the mel-spectrograms with the autoregressive Transformer TTS model and take the source text and the generated mel-spectrograms as the paired data for FastSpeech model training.

先在 4 块 NVIDIA V100 上训练自回归 Transformer TTS，每卡 batch 16 句。优化器用 Adam，$\beta_{1}=0.9$，$\beta_{2}=0.98$，$\varepsilon=10^{-9}$，学习率日程同 [25]。约 80k 步收敛。再把训练集文本–语音对送回模型，得到编码器–解码器注意力对齐，用于训练时长预测器。此外采用在非自回归机器翻译中表现良好的序列级知识蒸馏 [12, 7, 8, 26]：对每个源文本，用自回归 Transformer TTS 生成梅尔频谱，以「源文本 + 生成梅尔频谱」作为 FastSpeech 训练配对。

We train the FastSpeech model together with the duration predictor. The optimizer and other hyper-parameters for FastSpeech are the same as the autoregressive Transformer TTS model. The FastSpeech model training takes about 80k steps on 4 NVIDIA V100 GPUs. In the inference process, the output mel-spectrograms of our FastSpeech model are transformed into audio samples using the pretrained WaveGlow [20][^3].

FastSpeech 与时长预测器联合训练；优化器及其他超参与自回归 Transformer TTS 相同。在 4 块 V100 上约 80k 步。推断时，用预训练 WaveGlow [20] 把 FastSpeech 输出的梅尔频谱转为波形[^3]。

5 Results

5 结果

In this section, we evaluate the performance of FastSpeech in terms of audio quality, inference speedup, robustness, and controllability.

本节从音频质量、推断加速、稳健性与可控性评估 FastSpeech。

Audio Quality

We conduct the MOS (mean opinion score) evaluation on the test set to measure the audio quality. We keep the text content consistent among different models so as to exclude other interference factors, only examining the audio quality. Each audio is listened by at least 20 testers, who are all native English speakers. We compare the MOS of the generated audio samples by our FastSpeech model with other systems, which include 1) GT, the ground truth audio; 2) GT (Mel + WaveGlow), where we first convert the ground truth audio into mel-spectrograms, and then convert the mel-spectrograms back to audio using WaveGlow; 3) Tacotron 2 [22] (Mel + WaveGlow); 4) Transformer TTS [14] (Mel + WaveGlow). 5) Merlin [28] (WORLD), a popular parametric TTS system with WORLD [15] as the vocoder. The results are shown in Table 1. It can be seen that our FastSpeech can nearly match the quality of the Transformer TTS model and Tacotron 2 [^4].

音频质量

在测试集上做 MOS（平均意见分）评估音频质量。各模型文本内容保持一致，以排除其他干扰、只考察音频质量。每条音频至少 20 名母语英语听者评分。对比系统包括：1) GT 真值音频；2) GT (Mel + WaveGlow)，真值音频先转梅尔再经 WaveGlow 还原；3) Tacotron 2 [22] (Mel + WaveGlow)；4) Transformer TTS [14] (Mel + WaveGlow)；5) Merlin [28] (WORLD)，流行参数 TTS，声码器为 WORLD [15]。结果见表 1。可见 FastSpeech 质量几乎匹配 Transformer TTS 与 Tacotron 2[^4]。

Method	MOS
GT	4.41 $\pm$ 0.08
GT (Mel + WaveGlow)	4.00 $\pm$ 0.09
Tacotron 2 [22] (Mel + WaveGlow)	3.86 $\pm$ 0.09
Merlin [28] (WORLD)	2.40 $\pm$ 0.13
Transformer TTS [14] (Mel + WaveGlow)	3.88 $\pm$ 0.09
FastSpeech (Mel + WaveGlow)	3.84 $\pm$ 0.08

Table 1: The MOS with 95% confidence intervals.

方法	MOS
GT	4.41 $\pm$ 0.08
GT (Mel + WaveGlow)	4.00 $\pm$ 0.09
Tacotron 2 [22] (Mel + WaveGlow)	3.86 $\pm$ 0.09
Merlin [28] (WORLD)	2.40 $\pm$ 0.13
Transformer TTS [14] (Mel + WaveGlow)	3.88 $\pm$ 0.09
FastSpeech (Mel + WaveGlow)	3.84 $\pm$ 0.08

表 1：带 95% 置信区间的 MOS。

Inference Speedup

We evaluate the inference latency of FastSpeech compared with the autoregressive Transformer TTS model, which has similar number of model parameters with FastSpeech. We first show the inference speedup for mel-spectrogram generation in Table 2. It can be seen that FastSpeech speeds up the mel-spectrogram generation by 269.40x, compared with the Transformer TTS model. We then show the end-to-end speedup when using WaveGlow as the vocoder. It can be seen that FastSpeech can still achieve 38.30x speedup for audio generation.

推断加速

相对参数量相近的自回归 Transformer TTS，我们评估 FastSpeech 推断延迟。表 2 先给出梅尔频谱生成加速：相对 Transformer TTS，FastSpeech 加速 269.40 倍。再用 WaveGlow 作声码器看端到端加速，仍可达 38.30 倍。

Method	Latency (s)	Speedup
Transformer TTS [14] (Mel)	6.735 $\pm$ 3.969	/
FastSpeech (Mel)	0.025 $\pm$ 0.005	269.40$\times$
Transformer TTS [14] (Mel + WaveGlow)	6.895 $\pm$ 3.969	/
FastSpeech (Mel + WaveGlow)	0.180 $\pm$ 0.078	38.30$\times$

Table 2: The comparison of inference latency with 95% confidence intervals. The evaluation is conducted on a server with 12 Intel Xeon CPU, 256GB memory, 1 NVIDIA V100 GPU and batch size of 1. The average length of the generated mel-spectrograms for the two systems are both about 560.

方法	延迟 (s)	加速比
Transformer TTS [14] (Mel)	6.735 $\pm$ 3.969	/
FastSpeech (Mel)	0.025 $\pm$ 0.005	269.40$\times$
Transformer TTS [14] (Mel + WaveGlow)	6.895 $\pm$ 3.969	/
FastSpeech (Mel + WaveGlow)	0.180 $\pm$ 0.078	38.30$\times$

表 2：推断延迟对比（95% 置信区间）。评测服务器含 12 核 Intel Xeon CPU、256GB 内存、1 块 NVIDIA V100，batch size 为 1。两系统生成梅尔频谱平均长度均约 560。

We also visualize the relationship between the inference latency and the length of the predicted mel-spectrogram sequence in the test set. Figure 2 shows that the inference latency barely increases with the length of the predicted mel-spectrogram for FastSpeech, while increases largely in Transformer TTS. This indicates that the inference speed of our method is not sensitive to the length of generated audio due to parallel generation.

我们还可视化推断延迟与预测梅尔频谱序列长度的关系。图 2 显示：FastSpeech 延迟几乎不随预测长度增长，而 Transformer TTS 明显变慢。说明因并行生成，我们方法的推断速度对生成音频长度不敏感。

(a) FastSpeech

Robustness

The encoder-decoder attention mechanism in the autoregressive model may cause wrong attention alignments between phoneme and mel-spectrogram, resulting in instability with word repeating and word skipping. To evaluate the robustness of FastSpeech, we select 50 sentences which are particularly hard for TTS system[^5]. Word error counts are listed in Table 3. It can be seen that Transformer TTS is not robust to these hard cases and gets 34% error rate, while FastSpeech can effectively eliminate word repeating and skipping to improve intelligibility.

稳健性

自回归模型中的编码器–解码器注意力可能导致音素与梅尔频谱对齐错误，出现词重复与漏词等不稳定。为评估稳健性，我们选取 50 句对 TTS 特别困难的句子[^5]。词错误统计见表 3：Transformer TTS 在这些难例上不够稳健，错误率 34%；FastSpeech 可有效消除重复与漏词，提升可懂度。

Method	Repeats	Skips	Error Sentences	Error Rate
Tacotron 2	4	11	12	24%
Transformer TTS	7	15	17	34%
FastSpeech	0	0	0	0%

Table 3: The comparison of robustness between FastSpeech and other systems on the 50 particularly hard sentences. Each kind of word error is counted at most once per sentence.

方法	重复	漏读	出错句数	错误率
Tacotron 2	4	11	12	24%
Transformer TTS	7	15	17	34%
FastSpeech	0	0	0	0%

表 3：在 50 句特别困难句子上，FastSpeech 与其他系统的稳健性对比。每类词错误在每句最多计一次。

Length Control

As mentioned in Section 3.2, FastSpeech can control the voice speed as well as part of the prosody by adjusting the phoneme duration, which cannot be supported by other end-to-end TTS systems. We show the mel-spectrograms before and after the length control, and also put the audio samples in the supplementary material for reference.

长度控制

如第 3.2 节所述，FastSpeech 可通过调节音素时长控制语速与部分韵律，这是其他端到端 TTS 通常做不到的。我们展示长度控制前后的梅尔频谱，并在补充材料中提供音频样例。

Voice Speed The generated mel-spectrograms with different voice speeds by lengthening or shortening the phoneme duration are shown in Figure 3. We also attach several audio samples in the supplementary material for reference. As demonstrated by the samples, FastSpeech can adjust the voice speed from 0.5x to 1.5x smoothly, with stable and almost unchanged pitch.

语速 通过拉长或缩短音素时长得到不同语速的梅尔频谱，见图 3。补充材料另附音频。样例显示 FastSpeech 可在 0.5x 到 1.5x 间平滑调节语速，音高基本稳定。

(a) 1.5x Voice Speed

(a) 1.5 倍语速

Breaks Between Words FastSpeech can add breaks between adjacent words by lengthening the duration of the space characters in the sentence, which can improve the prosody of voice. We show an example in Figure 4, where we add breaks in two positions of the sentence to improve the prosody.

词间停顿 FastSpeech 可通过拉长句中空格字符的时长，在相邻词之间加入停顿，从而改善韵律。图 4 给出一例：在句子两处加入停顿以改善韵律。

(a) Original Mel-spectrograms

(a) 原始梅尔频谱

Ablation Study

We conduct ablation studies to verify the effectiveness of several components in FastSpeech, including 1D Convolution and sequence-level knowledge distillation. We conduct CMOS evaluation for these ablation studies.

消融实验

我们做消融以验证 FastSpeech 若干组件的有效性，包括一维卷积与序列级知识蒸馏，并用 CMOS 评估。

System	CMOS
FastSpeech	0
FastSpeech without 1D convolution in FFT block	-0.113
FastSpeech without sequence-level knowledge distillation	-0.325

Table 4: CMOS comparison in the ablation studies.

系统	CMOS
FastSpeech	0
FastSpeech 去掉 FFT 块中的一维卷积	-0.113
FastSpeech 去掉序列级知识蒸馏	-0.325

表 4：消融实验中的 CMOS 对比。

1D Convolution in FFT Block We propose to replace the original fully connected layer (adopted in Transformer [25]) with 1D convolution in FFT block, as described in Section 3.1. Here we conduct experiments to compare the performance of 1D convolution to the fully connected layer with similar number of parameters. As shown in Table 4, replacing 1D convolution with fully connected layer results in -0.113 CMOS, which demonstrates the effectiveness of 1D convolution.

FFT 块中的一维卷积 如第 3.1 节所述，我们建议用一维卷积替换 Transformer [25] 中原有的全连接层。这里在相近参数量下对比一维卷积与全连接。如表 4，用全连接替换一维卷积导致 CMOS -0.113，说明一维卷积有效。

Sequence-Level Knowledge Distillation As described in Section 4.3, we leverage sequence-level knowledge distillation for FastSpeech. We conduct CMOS evaluation to compare the performance of FastSpeech with and without sequence-level knowledge distillation, as shown in Table 4. We find that removing sequence-level knowledge distillation results in -0.325 CMOS, which demonstrates the effectiveness of sequence-level knowledge distillation.

序列级知识蒸馏 如第 4.3 节，我们对 FastSpeech 使用序列级知识蒸馏。表 4 的 CMOS 对比有无蒸馏的表现：去掉序列级知识蒸馏导致 CMOS -0.325，说明该蒸馏有效。

6 Conclusions

6 结论

In this work, we have proposed FastSpeech: a fast, robust and controllable neural TTS system. FastSpeech has a novel feed-forward network to generate mel-spectrogram in parallel, which consists of several key components including feed-forward Transformer blocks, a length regulator and a duration predictor. Experiments on LJSpeech dataset demonstrate that our proposed FastSpeech can nearly match the autoregressive Transformer TTS model in terms of speech quality, speed up the mel-spectrogram generation by 270x and the end-to-end speech synthesis by 38x, almost eliminate the problem of word skipping and repeating, and can adjust voice speed (0.5x-1.5x) smoothly.

本文提出 FastSpeech：快速、稳健且可控的神经 TTS。它用新型前馈网络并行生成梅尔频谱，关键组件包括前馈 Transformer 块、长度调节器与时长预测器。LJSpeech 实验表明：语音质量几乎匹配自回归 Transformer TTS；梅尔频谱生成加速约 270 倍、端到端合成加速约 38 倍；几乎消除漏词与重复；并能在 0.5x–1.5x 平滑调节语速。

For future work, we will continue to improve the quality of the synthesized speech, and apply FastSpeech to multi-speaker and low-resource settings. We will also train FastSpeech jointly with a parallel neural vocoder to make it fully end-to-end and parallel.

未来工作将继续提升合成质量，并把 FastSpeech 扩展到多说话人与低资源设定；也将与并行神经声码器联合训练，使其成为完全端到端且并行的系统。

Acknowledgments

致谢

This work was supported by the National Natural Science Foundation of China under Grant No.61602405, No.61836002. This work was also supported by the China Knowledge Centre of Engineering Sciences and Technology.

本工作获国家自然科学基金资助（编号 61602405、61836002），并获中国工程科技知识中心支持。

References

参考文献

[1] Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. Deep voice: Real-time neural text-to-speech. arXiv preprint arXiv:1702.07825 , 2017.

[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. ICLR 2015 , 2015.

[3] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems , pages 1171–1179, 2015.

[4] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on , pages 4960–4964. IEEE, 2016.

[5] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1243–1252. JMLR. org, 2017.

[6] Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing , 32(2):236–243, 1984.

[7] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. arXiv preprint arXiv:1711.02281 , 2017.

[8] Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. Non-autoregressive neural machine translation with enhanced decoder input. In AAAI , 2019.

[9] Andrew J Hunt and Alan W Black. Unit selection in a concatenative speech synthesis system using a large speech database. In 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings , volume 1, pages 373–376. IEEE, 1996.

[10] Keith Ito. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/ , 2017.

[11] Zeyu Jin, Adam Finkelstein, Gautham J Mysore, and Jingwan Lu. Fftnet: A real-time speaker-dependent neural vocoder. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2251–2255. IEEE, 2018.

[12] Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947 , 2016.

[13] Hao Li, Yongguo Kang, and Zhenyu Wang. Emphasis: An emotional phoneme-based acoustic model for speech synthesis system. arXiv preprint arXiv:1806.09276 , 2018.

[14] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, Ming Liu, and Ming Zhou. Close to human quality tts with transformer. arXiv preprint arXiv:1809.08895 , 2018.

[15] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: a vocoder-based high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems , 99(7):1877–1884, 2016.

[16] Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv:1711.10433 , 2017.

[17] Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao. Parallel neural text-to-speech. arXiv preprint arXiv:1905.08459 , 2019.

[18] Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to-speech. In International Conference on Learning Representations , 2019.

[19] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference on Learning Representations , 2018.

[20] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3617–3621. IEEE, 2019.

[21] Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Almost unsupervised text to speech and automatic speech recognition. In ICML , 2019.

[22] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4779–4783. IEEE, 2018.

[23] Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie-Yan Liu. Token-level ensemble distillation for grapheme-to-phoneme conversion. In INTERSPEECH , 2019.

[24] Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. SSW , 125, 2016.

[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998–6008, 2017.

[26] Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Non-autoregressive machine translation with auxiliary regularization. In AAAI , 2019.

[27] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135 , 2017.

[28] Zhizheng Wu, Oliver Watts, and Simon King. Merlin: An open source neural network speech synthesis system. In SSW , pages 202–207, 2016.

Appendix A Model Hyperparameters

附录 A 模型超参

Hyperparameter	Transformer TTS	FastSpeech
Phoneme Embedding Dimension	384	384
Pre-net Layers	3	/
Pre-net Hidden	384	/
Encoder/Phoneme-Side FFT Layers	6	6
Encoder/Phoneme-Side FFT Hidden	384	384
Encoder/Phoneme-Side FFT Conv1D Kernel	3	3
Encoder/Phoneme-Side FFT Conv1D Filter Size	1024	1536
Encoder/Phoneme-Side FFT Attention Heads	2	2
Decoder/Mel-Side FFT Layers	6	6
Decoder/Mel-Side FFT Hidden	384	384
Decoder/Mel-Side FFT Conv1D Kernel	3	3
Decoder/Mel-Side FFT Conv1D Filter Size	1024	1536
Decoder/Mel-Side FFT Attention Headers	2	2
Duration Predictor Conv1D Kernel	/	3
Duration Predictor Conv1D Filter Size	/	256
Dropout	0.1	0.1
Batch Size	64 (16 * 4GPUs)	64 (16 * 4GPUs)
Total Number of Parameters	30.7M	30.1M

Table 5: Hyperparameters of Transformer TTS and FastSpeech. Encoder and Decoder are for Transformer TTS, Phoneme-Side and Mel-Side FFT are for FastSpeech.

超参	Transformer TTS	FastSpeech
Phoneme Embedding Dimension	384	384
Pre-net Layers	3	/
Pre-net Hidden	384	/
Encoder/Phoneme-Side FFT Layers	6	6
Encoder/Phoneme-Side FFT Hidden	384	384
Encoder/Phoneme-Side FFT Conv1D Kernel	3	3
Encoder/Phoneme-Side FFT Conv1D Filter Size	1024	1536
Encoder/Phoneme-Side FFT Attention Heads	2	2
Decoder/Mel-Side FFT Layers	6	6
Decoder/Mel-Side FFT Hidden	384	384
Decoder/Mel-Side FFT Conv1D Kernel	3	3
Decoder/Mel-Side FFT Conv1D Filter Size	1024	1536
Decoder/Mel-Side FFT Attention Headers	2	2
Duration Predictor Conv1D Kernel	/	3
Duration Predictor Conv1D Filter Size	/	256
Dropout	0.1	0.1
Batch Size	64 (16 * 4GPUs)	64 (16 * 4GPUs)
Total Number of Parameters	30.7M	30.1M

表 5：Transformer TTS 与 FastSpeech 的超参。Encoder/Decoder 对应 Transformer TTS，Phoneme-Side/Mel-Side FFT 对应 FastSpeech。

Appendix B 50 Particularly Hard Sentences

附录 B 50 句特别困难的句子

The 50 particularly hard sentences mentioned in Section 5 are listed below:

第 5 节提到的 50 句特别困难句子如下：

1. a

2. b

3. c

4. H

5. I

6. J

7. K

8. L

9. 22222222 hello 22222222

10. S D S D Pass zero - zero Fail - zero to zero - zero - zero Cancelled - fifty nine to three - two - sixty four Total - fifty nine to three - two -

11. S D S D Pass - zero - zero - zero - zero Fail - zero - zero - zero - zero Cancelled - four hundred and sixteen - seventy six -

12. zero - one - one - two Cancelled - zero - zero - zero - zero Total - two hundred and eighty six - nineteen - seven -

13. forty one to five three hundred and eleven Fail - one - one to zero two Cancelled - zero - zero to zero zero Total -

14. zero zero one , MS03 - zero twenty five , MS03 - zero thirty two , MS03 - zero thirty nine ,

15. 1b204928 zero zero zero zero zero zero zero zero zero zero zero zero zero zero one seven ole32

16. zero zero zero zero zero zero zero zero two seven nine eight F three forty zero zero zero zero zero six four two eight zero one eight

17. c five eight zero three three nine a zero bf eight FALSE zero zero zero bba3add2 - c229 - 4cdb -

18. Calendaring agent failed with error code 0x80070005 while saving appointment .

19. Exit process - break ld - Load module - output ud - Unload module - ignore ser - System error - ignore ibp - Initial breakpoint -

20. Common DB connectors include the DB - nine , DB - fifteen , DB - nineteen , DB - twenty five , DB - thirty seven , and DB - fifty connectors .

21. To deliver interfaces that are significantly better suited to create and process RFC eight twenty one , RFC eight twenty two , RFC nine seventy seven , and MIME content .

22. int1 , int2 , int3 , int4 , int5 , int6 , int7 , int8 , int9 ,

23. seven _ ctl00 ctl04 ctl01 ctl00 ctl00

24. Http0XX , Http1XX , Http2XX , Http3XX ,

25. config file must contain A , B , C , D , E , F , and G .

26. mondo - debug mondo - ship motif - debug motif - ship sts - debug sts - ship Comparing local files to checkpoint files …

27. Rusbvts . dll Dsaccessbvts . dll Exchmembvt . dll Draino . dll Im trying to deploy a new topology , and I keep getting this error .

28. You can call me directly at four two five seven zero three seven three four four or my cell four two five four four four seven four seven four or send me a meeting request with all the appropriate information .

29. Failed zero point zero zero percent < one zero zero one zero zero zero zero Internal . Exchange . ContentFilter . BVT ContentFilter . BVT_log . xml Error ! Filename not specified .

30. C colon backslash o one two f c p a r t y backslash d e v one two backslash oasys backslash legacy backslash web backslash HELP

31. src backslash mapi backslash t n e f d e c dot c dot o l d backslash backslash m o z a r t f one backslash e x five

32. copy backslash backslash j o h n f a n four backslash scratch backslash M i c r o s o f t dot S h a r e P o i n t dot

33. Take a look at h t t p colon slash slash w w w dot granite dot a b dot c a slash access slash email dot

34. backslash bin backslash premium backslash forms backslash r e g i o n a l o p t i o n s dot a s p x dot c s Raj , DJ ,

35. Anuraag backslash backslash r a d u r five backslash d e b u g dot one eight zero nine underscore P R two h dot s t s contains

36. p l a t f o r m right bracket backslash left bracket f l a v o r right bracket backslash s e t u p dot e x e

37. backslash x eight six backslash Ship backslash zero backslash A d d r e s s B o o k dot C o n t a c t s A d d r e s

38. Mine is here backslash backslash g a b e h a l l hyphen m o t h r a backslash S v r underscore O f f i c e s v r

39. h t t p colon slash slash teams slash sites slash T A G slash default dot aspx As always , any feedback , comments ,

40. two thousand and five h t t p colon slash slash news dot com dot com slash i slash n e slash f d slash two zero zero three slash f d

41. backslash i n t e r n a l dot e x c h a n g e dot m a n a g e m e n t dot s y s t e m m a n a g e

42. I think Rich’s post highlights that we could have been more strategic about how the sum total of XBOX three hundred and sixtys were distributed .

43. 64X64 , 8K , one hundred and eighty four ASSEMBLY , DIGITAL VIDEO DISK DRIVE , INTERNAL , 8X ,

44. So we are back to Extended MAPI and C++ because . Extended MAPI does not have a dual interface VB or VB .Net can read .

45. Thanks , Borge Trongmo Hi gurus , Could you help us E2K ASP guys with the following issue ?

46. Thanks J RGR Are you using the LDDM driver for this system or the in the build XDDM driver ?

47. Btw , you might remember me from our discussion about OWA automation and OWA readiness day a year ago .

48. empidtool . exe creates HKEY_CURRENT_USER Software Microsoft Office Common QMPersNum in the registry , queries AD , and the populate the registry with MS employment ID if available else an error code is logged .

49. Thursday, via a joint press release and Microsoft AI Blog, we will announce Microsoft’s continued partnership with Shell leveraging cloud, AI, and collaboration technology to drive industry innovation and transformation.

50. Actress Fan Bingbing attends the screening of ’Ash Is Purest White (Jiang Hu Er Nv)’ during the 71st annual Cannes Film Festival