Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

Parallel WaveGAN：基于多分辨率频谱图生成对抗网络的快速波形生成模型

Ryuichi Yamamoto    Eunwoo Song    Jae-Min Kim

Abstract

摘要

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves $4.16$ mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.

我们提出 Parallel WaveGAN：一种无需蒸馏、快速且小体积的波形生成方法，基于生成对抗网络。该方法用非自回归 WaveNet，联合优化多分辨率频谱图损失与对抗损失，从而有效刻画真实语音波形的时频分布。由于不需要传统师生框架中的密度蒸馏，整模易于训练；即便结构紧凑，也能生成高保真语音。具体而言，所提 Parallel WaveGAN 仅有 1.44 M 参数，在单 GPU 环境下可以 28.68 倍于实时的速度生成 24 kHz 语音波形。听感测试表明，在基于 Transformer 的文本到语音框架内，该方法达到 $4.16$ 平均意见分，与最佳的基于蒸馏的 Parallel WaveNet 系统相当。

1 Introduction

1 引言

Index Terms:

Deep generative models in text-to-speech (TTS) frameworks have significantly improved the quality of synthetic speech signals [1, 2, 3]. Remarkably, autoregressive generative models such as WaveNet have shown much superior performance over traditional parametric vocoders [4, 5, 6, 7, 8]. However, they suffer from slow inference speed due to their autoregressive nature and thus are limited in their applications to real-time scenarios.

索引词：

深度生成模型在文本到语音（TTS）框架中显著提升了合成语音质量 [1, 2, 3]。尤其是 WaveNet 一类自回归生成模型，表现远超传统参数声码器 [4, 5, 6, 7, 8]。但其自回归特性导致推理缓慢，难以用于实时场景。

One approach to address the limitation is to utilize fast waveform generation methods based on a teacher-student framework [9, 10, 11]. In this framework, a bridge defined as probability density distillation transfers the knowledge of an autoregressive teacher WaveNet to an inverse autoregressive flow (IAF)-based student model [12]. Although the IAF student can achieve real-time generation of speech with reasonable perceptual quality, there remain problems in the training process: it requires not only a well trained teacher model, but also a trial and error methodology to optimize the complicated density distillation process.

一种应对思路是采用基于师生框架的快速波形生成方法 [9, 10, 11]。该框架用概率密度蒸馏作为桥梁，把自回归教师 WaveNet 的知识迁移到基于逆自回归流（IAF）的学生模型 [12]。IAF 学生虽能以合理听感质量实现实时生成，训练过程仍有问题：既需要训好的教师模型，又要对复杂的密度蒸馏反复试错调参。

To overcome the aforementioned problems, we propose a Parallel WaveGAN[^1], a simple and effective parallel waveform generation method based on a generative adversarial network (GAN) [14]. Unlike the conventional distillation-based methods, the Parallel WaveGAN does not require the two-stage, sequential teacher-student training process. In the proposed method, only a non-autoregressive WaveNet model is trained by optimizing the combination of multi-resolution short-time Fourier transform (STFT) and adversarial loss functions that enable the model to effectively capture the time-frequency distribution of the realistic speech waveform. As a result, the entire training process becomes much easier than the conventional methods, as well as the model can produce natural sounding speech waveforms with a small number of model parameters. Our contributions are summarized as follows:

为克服上述问题，我们提出 Parallel WaveGAN[^1]：一种基于生成对抗网络（GAN）[14] 的简单有效并行波形生成方法。与传统基于蒸馏的方法不同，Parallel WaveGAN 不需要两阶段、顺序的师生训练。该方法仅训练一个非自回归 WaveNet，通过联合优化多分辨率短时傅里叶变换（STFT）损失与对抗损失，有效刻画真实语音波形的时频分布。因而整段训练远比传统方法简单，并能以较少参数生成自然语音波形。贡献概括如下：

• • We propose a joint training method of the multi-resolution STFT loss and the waveform-domain adversarial loss. This approach effectively works for the conventional distillation-based Parallel WaveNet (e.g., ClariNet), as well as for the proposed distillation-free Parallel WaveGAN.

• • 我们提出多分辨率 STFT 损失与波形域对抗损失的联合训练。该方法对传统基于蒸馏的 Parallel WaveNet（如 ClariNet）有效，也对所提免蒸馏 Parallel WaveGAN 有效。

• • As the proposed Parallel WaveGAN can be simply trained without any teacher-student framework, our approach significantly reduces both the training and inference time. In particular, the training process becomes 4.82 times faster (from 13.5 days to 2.8 days with two NVIDIA Telsa V100 GPUs) and the inference process becomes 1.96 times faster (from 14.62 to 28.68 real-time[^2] to generate 24 kHz speech waveforms with a single NVIDIA Telsa V100 GPU) compared with the conventional ClariNet model.

• • 由于 Parallel WaveGAN 无需师生框架即可简单训练，训练与推理时间都显著缩短。具体而言，相对传统 ClariNet，训练加快 4.82 倍（两块 NVIDIA Tesla V100 上从 13.5 天降至 2.8 天），推理加快 1.96 倍（单块 NVIDIA Tesla V100 上生成 24 kHz 波形时，从 14.62 倍实时提升到 28.68 倍实时[^2]）。

• • We combined the proposed Parallel WaveGAN with a TTS acoustic model based on a Transformer [15, 16, 17]. The perceptual listening tests verify that the proposed Parallel WaveGAN achieves $4.16$ MOS, which is competitive to the best distillation-based ClariNet model.

• • 我们将 Parallel WaveGAN 与基于 Transformer 的 TTS 声学模型 [15, 16, 17] 结合。听感测试表明，所提 Parallel WaveGAN 达到 $4.16$ MOS，与最佳的基于蒸馏的 ClariNet 相当。

2 Related work

2 相关工作

The idea of using GAN in the Parallel WaveNet framework is not new. In our previous work, the IAF student model was incorporated as a generator and jointly optimized by minimizing the adversarial loss along with the Kullback-Leibler divergence (KLD) and auxiliary losses [11]. As the GAN learns the distribution of realistic speech signals, the method significantly improves the perceptual quality of synthetic signal. However, the complicated training stage based on density distillation limits its utilization.

在 Parallel WaveNet 框架中使用 GAN 并非新想法。我们此前工作把 IAF 学生作为生成器，与对抗损失以及 Kullback-Leibler 散度（KLD）与辅助损失联合优化 [11]。GAN 学习真实语音分布，显著提升合成信号听感；但基于密度蒸馏的复杂训练阶段限制了其可用性。

Our aim is to minimize the effort to train the two-stage pipeline of the conventional teacher-student framework. In other words, we propose a novel method to train the Parallel WaveNet without any distillation process. Juvela et al [18] has proposed a similar approach (e.g., GAN-excited linear prediction; GELP) that generates glottal excitations by using the adversarial training method. However, since GELP requires linear prediction (LP) parameters to convert glottal excitations to speech waveform, quality degradation may occur when the LP parameters contain inevitable errors caused by the TTS acoustic model. To avoid this problem, our method is designed to directly estimate the speech waveform. As it is very difficult to capture the dynamic nature of speech signal including both the vocal cord movement and the vocal tract resonance (represented by glottal excitations and LP parameters in GELP, respectively), we propose a joint optimization method between the adversarial loss and multi-resolution STFT loss in order to capture the time-frequency distributions of the realistic speech signal. As a result, the entire model can be easily trained even with a small number of parameters while effectively reducing the inference time and improving perceptual quality of synthesized speech.

我们的目标是尽量减少传统师生两阶段管线的训练负担，亦即提出一种无需任何蒸馏过程即可训练 Parallel WaveNet 的新方法。Juvela et al [18] 提出过类似思路（如 GAN-excited linear prediction；GELP），用对抗训练生成声门激励。但 GELP 需要线性预测（LP）参数才能把声门激励转为语音波形；当 LP 参数因 TTS 声学模型不可避免地含误差时，质量可能下降。为避免该问题，我们的方法直接估计语音波形。语音同时包含声带运动与声道共振（在 GELP 中分别对应声门激励与 LP 参数），动态性很难刻画，因此我们提出对抗损失与多分辨率 STFT 损失的联合优化，以捕捉真实语音的时频分布。结果是：即便参数很少，整模也易于训练，同时缩短推理时间并提升合成语音听感。

3 Method

3 方法

3.1 Parallel waveform generation based on GAN

3.1 基于 GAN 的并行波形生成

GANs are generative models that are composed of two separate neural networks: a generator ($G$) and a discriminator ($D$) [14]. In our method, a WaveNet-based model conditioned on an auxiliary feature (e.g., mel-spectrogram) is used as the generator, which transforms the input noise to the output waveform in parallel. The generator differs from the original WaveNet in that: (1) we use non-causal convolutions instead of causal convolutions; (2) the input is random noise drawn from a Gaussian distribution; (3) the model is non-autoregressive at both training and inference steps.

GAN 由两个独立神经网络组成：生成器（$G$）与判别器（$D$） [14]。本方法中，以辅助特征（如 mel 频谱图）条件化的 WaveNet 模型作为生成器，并行地把输入噪声变换为输出波形。该生成器与原版 WaveNet 的不同在于：(1) 使用非因果卷积而非因果卷积；(2) 输入为高斯分布采样的随机噪声；(3) 训练与推理均为非自回归。

The generator learns a distribution of realistic waveforms by trying to deceive the discriminator to recognize the generator samples as real. The process is performed by minimizing the adversarial loss[^3] ($L_{\mathrm{adv}}$) as follows:

生成器通过试图欺骗判别器、使其把生成样本判为 真实，来学习真实波形分布。该过程通过最小化对抗损失[^3]（$L_{\mathrm{adv}}$）完成，如下：

$$L_{\mathrm{adv}}(G,D)=\E_{\bz\sim N(0,I)}\left[(1-D(G(\bz)))^{2}\right], \tag{1}$$

where $\bz$ denotes the input white noise. Note that the auxiliary feature for $G$ is omitted for brevity.

其中 $\bz$ 表示输入白噪声。为简洁起见，省略了 $G$ 的辅助特征。

On the other hand, the discriminator is trained to correctly classify the generated sample as fake while classifying the ground truth as real using the following optimization criterion:

另一方面，判别器被训练为正确地把生成样本判为 假、把真值判为 真，优化准则如下：

$$L_{\mathrm{D}}(G,D)=\E_{\bx\sim p_{\mathrm{data}}}[(1-D(\bx))^{2}]+\E_{\bz\sim N(0,I)}[D(G(\bz))^{2}], \tag{2}$$

where $\bx$ and $p_{\mathrm{data}}$ denote the target waveform and its distribution, respectively.

其中 $\bx$ 与 $p_{\mathrm{data}}$ 分别表示目标波形及其分布。

3.2 Multi-resolution STFT auxiliary loss

3.2 多分辨率 STFT 辅助损失

Figure 1: An illustration of our proposed adversarial training framework with the multi-resolution STFT loss.

图 1：所提带多分辨率 STFT 损失的对抗训练框架示意。

To improve the stability and efficiency of the adversarial training process, we propose a multi-resolution STFT auxiliary loss. Fig. 1 shows our framework combining the multi-resolution STFT loss with the adversarial training method as described in section 3.1.

为提升对抗训练的稳定性与效率，我们提出多分辨率 STFT 辅助损失。图 1 展示了将多分辨率 STFT 损失与 3.1 节对抗训练相结合的框架。

Similar to the previous work [11], we define a single STFT loss as follows:

与先前工作 [11] 类似，我们定义单个 STFT 损失如下：

$$L_{\mathrm{s}}(G)=\E_{\bz\sim p(\bz),\bx\sim p_{data}}\left[L_{\mathrm{sc}}(\bx,\bxhat)+L_{\mathrm{mag}}(\bx,\bxhat)\right], \tag{3}$$

where $\bxhat$ denotes the generated sample (i.e., $G(\bz)$), and $L_{\mathrm{sc}}$ and $L_{\mathrm{mag}}$ denote spectral convergence and log STFT magnitude loss, respectively, which are defined as follows [23]:

其中 $\bxhat$ 表示生成样本（即 $G(\bz)$），$L_{\mathrm{sc}}$ 与 $L_{\mathrm{mag}}$ 分别表示 频谱收敛 损失与 对数 STFT 幅度 损失，定义如下 [23]：

$$L_{\mathrm{sc}}(\bx,\bxhat)=\frac{\parallel|\mathrm{STFT}(\bx)|-|\mathrm{STFT}(\bxhat)|\parallel_{F}}{\parallel|\mathrm{STFT}(\bx)|\parallel_{F}}, \tag{4}$$

$$L_{\mathrm{mag}}(\bx,\bxhat)=\frac{1}{N}\parallel\mathrm{log}|\mathrm{STFT}(\bx)|-\mathrm{log}|\mathrm{STFT}(\bxhat)|\parallel_{1}, \tag{5}$$

where $\parallel\cdot\parallel_{F}$ and $\parallel\cdot\parallel_{1}$ denote the Frobenius and $L_{1}$ norms, respectively; $|\mathrm{STFT}(\cdot)|$ and $N$ denote the STFT magnitudes and number of elements in the magnitude, respectively.

其中 $\parallel\cdot\parallel_{F}$ 与 $\parallel\cdot\parallel_{1}$ 分别表示 Frobenius 范数与 $L_{1}$ 范数；$|\mathrm{STFT}(\cdot)|$ 与 $N$ 分别表示 STFT 幅度及其元素个数。

Our multi-resolution STFT loss is the sum of the STFT losses with different analysis parameters (i.e., FFT size, window size, and frame shift). Let $M$ be the number of STFT losses, the multi-resolution STFT auxiliary loss ($L_{\mathrm{aux}}$) is represented as follows:

我们的多分辨率 STFT 损失是具有不同分析参数（即 FFT 大小、窗长与帧移）的多个 STFT 损失之和。设 $M$ 为 STFT 损失个数，则多分辨率 STFT 辅助损失（$L_{\mathrm{aux}}$）表示为：

$$L_{\mathrm{aux}}(G)=\frac{1}{M}\sum_{m=1}^{M}L^{(m)}_{\mathrm{s}}(G). \tag{6}$$

In the STFT-based time-frequency representation of signals, there is a trade-off between time and frequency resolution; e.g., increasing window size gives higher frequency resolution while reducing temporal resolution[24]. By combining multiple STFT losses with different analysis parameters, it greatly helps the generator to learn the time-frequency characteristics of speech [25]. Moreover, it also prevents the generator from being overfit to a fixed STFT representation, which may result in suboptimal performance in the waveform-domain.

在基于 STFT 的时频表示中，时间与频率分辨率存在权衡；例如增大窗长可提高频率分辨率，但降低时间分辨率[24]。组合不同分析参数的多个 STFT 损失，有助于生成器学习语音的时频特性 [25]。此外，也可防止生成器过拟合到固定 STFT 表示，以免在波形域得到次优表现。

Our final loss function for the generator is defined as a linear combination of the multi-resolution STFT loss and the adversarial loss as follows:

生成器的最终损失函数定义为多分辨率 STFT 损失与对抗损失的线性组合：

$$L_{\mathrm{G}}(G,D)=L_{\mathrm{aux}}(G)+\lambda_{\mathrm{adv}}L_{\mathrm{adv}}(G,D), \tag{7}$$

where $\lambda_{\mathrm{adv}}$ denotes the hyperparameter balancing the two loss terms. By jointly optimizing the waveform-domain adversarial loss and the multi-resolution STFT loss, the generator can learn the distribution of the realistic speech waveform effectively.

其中 $\lambda_{\mathrm{adv}}$ 是平衡两项损失的超参数。通过联合优化波形域对抗损失与多分辨率 STFT 损失，生成器能有效学习真实语音波形的分布。

4 Experiments

4 实验

4.1 Experimental setup

4.1 实验设置

4.1.1 Database

In the experiments, we used a phonetically and prosaically balanced speech corpus recorded by a female professional Japanese speaker. The speech signals were sampled at 24 kHz, each sample was quantized by 16 bits. In total, 11,449 utterances (23.09 hours) were used for training, 250 utterances (0.35 hours) were used for validation, and another 250 utterances (0.34 hours) were used for evaluation. The 80-band log-mel spectrograms with band-limited frequency range[^4] (70 to 8000 Hz) were extracted and used as the input auxiliary features for waveform generation models (i.e., local-conditioning [4]). The frame and shift lengths were set to 50 ms and 12.5 ms, respectively. The mel-spectrogram features were normalized to have zero mean and unit variance before training.

4.1.1 数据库

实验使用由一名日语女专业播音员录制、语音与韵律均衡的语料。语音以 24 kHz 采样，每样本 16 bit 量化。共 11,449 句（23.09 小时）用于训练，250 句（0.35 小时）验证，另 250 句（0.34 小时）评测。提取 80 维对数 mel 频谱图，频带受限[^4]（70 至 8000 Hz），作为波形生成模型的输入辅助特征（即局部条件 [4]）。帧长与帧移分别为 50 ms 与 12.5 ms。训练前将 mel 频谱图归一化为零均值单位方差。

4.1.2 Model details

The proposed Parallel WaveGAN consisted of 30 layers of dilated residual convolution blocks with exponentially increasing three dilation cycles [4]. The number of residual and skip channels were set to 64 and the convolution filter size was set to three. The discriminator consisted of ten layers of non-causal dilated 1-D convolutions with leaky ReLU activation function ($\alpha=0.2$). The strides were set to one and linearly increasing dilations were applied for the 1-D convolutions starting from one to eight except for the first and last layers. The number of channels and filter size were the same as the generator. We applied weight normalization to all convolutional layers for both the generator and the discriminator [26].

4.1.2 模型细节

所提 Parallel WaveGAN 由 30 层膨胀残差卷积块组成，膨胀因子按指数增长、共三个膨胀周期 [4]。残差通道与跳跃通道均为 64，卷积核大小为 3。判别器由十层非因果膨胀一维卷积组成，激活为 leaky ReLU（$\alpha=0.2$）。步长为 1，除首末层外，一维卷积的膨胀因子从 1 线性增至 8。通道数与核大小与生成器相同。生成器与判别器的所有卷积层均施加权重归一化 [26]。

At the training stage, the multi-resolution STFT loss was computed by the sum of three different STFT losses as described in Table 1. The discriminator loss was computed by the average of per-time step scalar predictions with the discriminator. The hyperparameter $\lambda_{\mathrm{adv}}$ in equation (7) was chosen to be 4.0 based on our preliminary experiments. Models were trained for 400 K steps with RAdam optimizer ($\epsilon=1e^{-6}$) to stabilize training [27]. Note that the discriminator was fixed for the first 100K steps, and two models were jointly trained afterwards. The minibatch size was set to eight and the length of each audio clip was set to 24 K time samples (1.0 second). The initial learning rate was set to $0.0001$ and $0.00005$ for the generator and discriminator, respectively. The learning rate was reduced by half for every 200 K steps.

训练时，多分辨率 STFT 损失由表 1 所述三个不同 STFT 损失求和得到。判别器损失对判别器逐步标量预测取平均。式 (7) 中超参数 $\lambda_{\mathrm{adv}}$ 据初步实验取 4.0。模型用 RAdam 优化器（$\epsilon=1e^{-6}$）训练 400 K 步以稳定训练 [27]。注意前 100K 步固定判别器，之后两模型联合训练。minibatch 大小为 8，每段音频长度为 24 K 时间采样点（1.0 秒）。生成器与判别器初始学习率分别为 $0.0001$ 与 $0.00005$，每 200 K 步减半。

As baseline systems, we used both the autoregressive Gaussian WaveNet and the parallel one (i.e., ClariNet) [10, 11]. The WaveNet consisted of 24 layers of dilated residual convolution blocks with four dilation cycles. The number of residual and skip channels were set to 128 and the filter size was set to three. The model was trained for 1.5 M steps with RAdam optimizer. The learning rate was set to $0.001$, and it was reduced by half for every 200 K steps. The minibatch size was set to eight and the length of each audio clip was set to 12 K time samples (0.5 second).

基线系统采用自回归高斯 WaveNet 与并行版本（即 ClariNet） [10, 11]。WaveNet 由 24 层膨胀残差卷积块、四个膨胀周期组成。残差与跳跃通道为 128，核大小为 3。用 RAdam 训练 1.5 M 步，学习率 $0.001$，每 200 K 步减半。minibatch 为 8，每段音频 12 K 采样点（0.5 秒）。

To train the baseline ClariNet, the autoregressive WaveNet described above was used as the teacher model. The ClariNet was based on Gaussian IAFs [10], which consisted of six flows. Each flow was parameterized by ten layers of dilated residual convolution blocks with an exponentially increasing dilation cycle. The number of residual and skip channels were set to 64 and the filter size was set to three. The weight coefficients to balance the KLD and STFT auxiliary losses were set to 0.5 and 1.0, respectively. The model was trained for 400 K steps with the same optimizer settings of the Parallel WaveGAN. We also investigated ClariNet with adversarial loss as a hybrid approach of GAN and density distillation [11]. The model structure was the same as the baseline ClariNet, but it was trained with the mixture of KLD, STFT and adversarial losses, where the weight coefficients to balance them were set to 0.05, 1.0 and 4.0, respectively. The model was trained for 200K steps with the fixed discriminator, and the generator and discriminator were jointly optimized for the rest 200 K steps.

训练基线 ClariNet 时，以上述自回归 WaveNet 为教师。ClariNet 基于高斯 IAF [10]，含六个流；每流由十层膨胀残差卷积块与一个指数增长膨胀周期参数化。残差与跳跃通道为 64，核大小为 3。平衡 KLD 与 STFT 辅助损失的权重分别为 0.5 与 1.0。用与 Parallel WaveGAN 相同的优化器设置训练 400 K 步。我们还考察了带对抗损失的 ClariNet，作为 GAN 与密度蒸馏的混合方案 [11]。结构与基线 ClariNet 相同，但用 KLD、STFT 与对抗损失的混合训练，权重分别为 0.05、1.0 与 4.0。前 200K 步固定判别器训练，后 200 K 步生成器与判别器联合优化。

Throughout the waveform generation models, the input auxiliary features were upsampled by nearest neighbor upsampling followed by 2-D convolutions so that the time-resolution of auxiliary features matches the sampling rate of the speech waveform [28, 11]. Note that the auxiliary features were not used for discriminators. All the models were trained using two NVIDIA Tesla V100 GPUs. Experiments were conducted on the NAVER smart machine learning (NSML) platform [29].

各波形生成模型中，输入辅助特征经最近邻上采样再接二维卷积上采样，使其时间分辨率与语音采样率对齐 [28, 11]。注意判别器不使用辅助特征。所有模型在两块 NVIDIA Tesla V100 GPU 上训练。实验在 NAVER smart machine learning（NSML）平台上进行 [29]。

STFT loss	FFT size	Window size	Frame shift
$L_{\mathrm{s}}^{(1)}$	1024	600 (25 ms)	120 (5 ms)
$L_{\mathrm{s}}^{(2)}$	2048	1200 (50 ms)	240 (10 ms)
$L_{\mathrm{s}}^{(3)}$	512	240 (10 ms)	50 ($\approx$ 2 ms)

Table 1: The details of the multi-resolution STFT loss. A Hanning window was applied before the FFT process.

STFT loss	FFT size	Window size	Frame shift
$L_{\mathrm{s}}^{(1)}$	1024	600 (25 ms)	120 (5 ms)
$L_{\mathrm{s}}^{(2)}$	2048	1200 (50 ms)	240 (10 ms)
$L_{\mathrm{s}}^{(3)}$	512	240 (10 ms)	50 ($\approx$ 2 ms)

表 1：多分辨率 STFT 损失细节。FFT 前施加汉宁窗。

System	Model	KLD-based	STFT	Adversarial	Number of	Model	Inference	MOS
index	distillation	loss	loss	layers	size	speed
System 1	WaveNet	-	-	-	24	3.81 M	0.32$\times 10^{-2}$	3.61$\pm$0.12
System 2	ClariNet	Yes	$L_{\mathrm{s}}^{(1)}$	-	60	2.78 M	14.62	3.88$\pm$0.11
System 3	ClariNet	Yes	$L_{\mathrm{s}}^{(1)}+L_{\mathrm{s}}^{(2)}+L_{\mathrm{s}}^{(3)}$	-	60	2.78 M	14.62	4.21$\pm$0.09
System 4	ClariNet	Yes	$L_{\mathrm{s}}^{(1)}+L_{\mathrm{s}}^{(2)}+L_{\mathrm{s}}^{(3)}$	Yes	60	2.78 M	14.62	4.21$\pm$0.09
System 5	Parallel WaveGAN	-	$L_{\mathrm{s}}^{(1)}$	Yes	30	1.44 M	28.68	1.36$\pm$0.07
System 6	Parallel WaveGAN	-	$L_{\mathrm{s}}^{(1)}+L_{\mathrm{s}}^{(2)}+L_{\mathrm{s}}^{(3)}$	Yes	30	1.44 M	28.68	4.06$\pm$0.10
System 7	Recording	-	-	-	-	-	-	4.46$\pm$0.08

Table 2: The inference speed and the MOS results with 95% confidence intervals: Acoustic features extracted from the recorded speech signal were used to compose the input auxiliary features. The evaluation was conducted on a server with a single NVIDIA Tesla V100 GPU. Note that the inference speed ${k}$ means that the system was able to generate waveforms ${k}$ times faster than real-time.

System	Model	KLD-based	STFT	Adversarial	Number of	Model	Inference	MOS
index	distillation	loss	loss	layers	size	speed
System 1	WaveNet	-	-	-	24	3.81 M	0.32$\times 10^{-2}$	3.61$\pm$0.12
System 2	ClariNet	Yes	$L_{\mathrm{s}}^{(1)}$	-	60	2.78 M	14.62	3.88$\pm$0.11
System 3	ClariNet	Yes	$L_{\mathrm{s}}^{(1)}+L_{\mathrm{s}}^{(2)}+L_{\mathrm{s}}^{(3)}$	-	60	2.78 M	14.62	4.21$\pm$0.09
System 4	ClariNet	Yes	$L_{\mathrm{s}}^{(1)}+L_{\mathrm{s}}^{(2)}+L_{\mathrm{s}}^{(3)}$	Yes	60	2.78 M	14.62	4.21$\pm$0.09
System 5	Parallel WaveGAN	-	$L_{\mathrm{s}}^{(1)}$	Yes	30	1.44 M	28.68	1.36$\pm$0.07
System 6	Parallel WaveGAN	-	$L_{\mathrm{s}}^{(1)}+L_{\mathrm{s}}^{(2)}+L_{\mathrm{s}}^{(3)}$	Yes	30	1.44 M	28.68	4.06$\pm$0.10
System 7	Recording	-	-	-	-	-	-	4.46$\pm$0.08

表 2：推理速度与 MOS 结果（含 95% 置信区间）：用录音提取的声学特征作为输入辅助特征。评测在单块 NVIDIA Tesla V100 GPU 服务器上进行。推理速度 ${k}$ 表示系统能以 ${k}$ 倍于实时的速度生成波形。

4.2 Evaluation

4.2 评测

To evaluate the perceptual quality, we performed mean opinion score (MOS)[^5] tests. Eighteen native Japanese speakers were asked to make quality judgments about the synthesized speech samples using the following five possible responses: 1 = Bad; 2 = Poor; 3 = Fair; 4 = Good; and 5 = Excellent. In total, 20 utterances were randomly selected from the evaluation set and were then synthesized using the different models.

为评估听感质量，我们进行平均意见分（MOS）[^5] 测试。十八名日语母语者按五档评分：1 = Bad；2 = Poor；3 = Fair；4 = Good；5 = Excellent。共从评测集随机抽取 20 句，用不同模型合成。

Table 2 shows the inference speed and the MOS test results with respect to different generation models. The findings can be summarized as follows: (1) the systems trained with the STFT loss performed better than ones trained without the STFT loss (i.e., the autoregressive WaveNet). Note that most listeners were unsatisfied with the high-frequency noise caused by the autoregressive WaveNet system. This could be explained by the fact that only the band-limited (70 - 8000 Hz) mel-spectrogram was used for local-conditioning in the WaveNet, while the other systems were able to directly learn full-band frequency information via STFT loss. (2) The proposed multi-resolution STFT loss-based models showed higher perceptual quality than the conventional single STFT losss-based ones (comparing System 3 and 6 with System 2 and 5, respectively). This confirms that the multi-resolution STFT loss effectively captured the time-frequency characteristics of the speech signal, enabling to achieve better performance. (3) The proposed adversarial loss did not work well with the ClariNet. However, its advantage could be found when it was combined with a TTS framework, which will be discussed in the next section. (4) Finally, the proposed Parallel WaveGAN achieved $4.06$ MOS. Although its perceptual quality was relatively worse than the ClariNet’s, the Parallel WaveGAN was able to generate speech signal $1.96$ times faster than the ClariNet. Furthermore, the benefit of the proposed method could be found in its simple training procedure. We measured the total training time for obtaining the optimal models, as described in Table 3. Because the Parallel WaveGAN did not require any complicated density distillation, it only took 2.8 training days to be optimized, which was $2.64$ and $4.82$ times faster than the autoregressive WaveNet and the ClariNet, respectively.

表 2 给出不同生成模型的推理速度与 MOS。发现可概括如下：(1) 用 STFT 损失训练的系统优于不用 STFT 损失者（即自回归 WaveNet）。多数听者不满自回归 WaveNet 的高频噪声；可能因为 WaveNet 局部条件仅用频带受限（70–8000 Hz）mel 频谱，而其他系统可通过 STFT 损失直接学习全带频率信息。(2) 所提多分辨率 STFT 损失模型听感高于传统单 STFT 损失模型（分别比较 System 3、6 与 System 2、5）。这证实多分辨率 STFT 损失有效刻画语音时频特性，从而取得更好表现。(3) 所提对抗损失在 ClariNet 上效果不明显；但其优势在与 TTS 框架结合时显现，见下一节。(4) 最终，所提 Parallel WaveGAN 达到 $4.06$ MOS。虽听感相对低于 ClariNet，但生成速度约为 ClariNet 的 $1.96$ 倍。此外，训练流程简单也是优势。我们按表 3 统计得到最优模型的总训练时间：因无需复杂密度蒸馏，Parallel WaveGAN 仅需 2.8 训练日即可优化，分别比自回归 WaveNet 与 ClariNet 快 $2.64$ 与 $4.82$ 倍。

Model	Training time (days)
WaveNet	7.4
ClariNet	12.7
ClariNet-GAN	13.5
Parallel WaveGAN (ours)	2.8

Table 3: Training time comparison: All the experiments were conducted on a server with two NVIDIA Tesla V100 GPUs. Each vocoder model corresponds to System 1, 3, 4, and 6 described in Table 2, respectively. Note that the times for ClariNets include the training time for the teacher WaveNet.

Model	Training time (days)
WaveNet	7.4
ClariNet	12.7
ClariNet-GAN	13.5
Parallel WaveGAN (ours)	2.8

表 3：训练时间比较：均在两块 NVIDIA Tesla V100 GPU 服务器上进行。各声码器分别对应表 2 中 System 1、3、4、6。注意 ClariNet 系列时间含教师 WaveNet 训练。

4.3 Text-to-speech

4.3 文本到语音

To verify the effectiveness of the proposed method as the vocoder of the TTS framework, we combined the Parallel WaveGAN with the Transformer-based parameter estimator [15, 16, 17].

为验证所提方法作为 TTS 框架声码器的有效性，我们将 Parallel WaveGAN 与基于 Transformer 的参数估计器 [15, 16, 17] 结合。

To train the Transformer, we used the phoneme sequences as input and mel-spectrograms extracted from the recorded speech as output. The model consisted of a six-layer encoder and a six-layer decoder, each was based on multi-head attention (with eight heads). The configuration followed the prior work [17], but the model was modified to accept accent as an external input for pitch accent language (e.g., Japanese) [30]. The model was trained for 1000 epochs using RAdam optimizer with warmup learning rate scheduling [15]. Initial learning rate was set to 1.0 and dynamic batch size (average 64) strategy was used to stabilize training.

训练 Transformer 时，以音素序列为输入、录音提取的 mel 频谱为输出。模型含六层编码器与六层解码器，均基于多头注意力（八头）。配置遵循先前工作 [17]，但修改为接受重音作为外部输入，以适配音高重音语言（如日语） [30]。用 RAdam 与 warmup 学习率调度训练 1000 个 epoch [15]。初始学习率 1.0，采用动态 batch（平均 64）以稳定训练。

In the synthesis step, the input phoneme and accent sequences were converted to the corresponding mel-spectrograms by the Transformer TTS model. By inputting resulting acoustic parameters, vocoder models generated the time-domain speech signal.

合成时，输入音素与重音序列由 Transformer TTS 转为对应 mel 频谱；再以所得声学参数为输入，由声码器生成时域语音信号。

To evaluate the quality of the generated speech samples, we performed MOS tests. The test setups were the same as those described in section 4.2, but we used the autoregressive WaveNet and the parallel generation models trained with the multi-resolution STFT loss in the test (System 1, 3, 4, and 6 described in Table 2, respectively). The results of the MOS tests are shown in Table 4, of which findings can be summarized as follows: (1) the ClariNet trained with the adversarial loss performed better than the system trained without the adversarial loss, although their perceptual qualities were almost same in the analysis/synthesis case (System 3 and 4 shown in Table 2). This implies that the use of adversarial loss was advantageous for improving the model’s robustness to the prediction errors caused by the acoustic model. (2) The merits of the adversarial training were also beneficial to the proposed Parallel WaveGAN system. Consequently, the Parallel WaveGAN with the Transformer TTS model achieved $4.16$ MOS, which was comparable to the best distillation-based Parallel WaveNet system (i.e., ClariNet-GAN).

为评估生成样本质量，我们进行 MOS 测试。设置与 4.2 节相同，但测试使用自回归 WaveNet 以及用多分辨率 STFT 损失训练的并行生成模型（分别对应表 2 的 System 1、3、4、6）。MOS 结果见表 4，可概括为：(1) 带对抗损失训练的 ClariNet 优于不带者，尽管在分析／合成情形下听感几乎相同（表 2 的 System 3 与 4）。说明对抗损失有助于提升模型对声学模型预测误差的鲁棒性。(2) 对抗训练的好处同样惠及所提 Parallel WaveGAN。因而，Parallel WaveGAN 配合 Transformer TTS 达到 $4.16$ MOS，与最佳的基于蒸馏的 Parallel WaveNet 系统（即 ClariNet-GAN）相当。

Model	MOS
Transformer + WaveNet	3.33$\pm$0.11
Transformer + ClariNet	4.00$\pm$0.10
Transformer + ClariNet-GAN	4.14$\pm$0.10
Transformer + Parallel WaveGAN (ours)	4.16$\pm$0.09
Recording	4.46$\pm$0.08

Table 4: MOS results with 95% confidence intervals: Acoustic features generated from the Transformer TTS model were used to compose the input auxiliary features.

Model	MOS
Transformer + WaveNet	3.33$\pm$0.11
Transformer + ClariNet	4.00$\pm$0.10
Transformer + ClariNet-GAN	4.14$\pm$0.10
Transformer + Parallel WaveGAN (ours)	4.16$\pm$0.09
Recording	4.46$\pm$0.08

表 4：MOS 结果（含 95% 置信区间）：用 Transformer TTS 生成的声学特征作为输入辅助特征。

5 Conclusion

5 结论

We proposed Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method based on GAN. By jointly optimizing waveform-domain adversarial loss and multi-resolution STFT loss, our model was able to learn how to generate realistic waveforms without any complicated probability density distillation. Experimental results demonstrated that our proposed method achieved $4.16$ MOS within the Transformer-based TTS framework competitive to the conventional distillation-based approaches, generating 24 kHz speech waveform 28.68 times faster than real-time with only 1.44 M model parameters. Future research includes improving the multi-resolution STFT auxiliary loss to better capture the characteristics of speech (e.g., introducing phase-related loss), and verifying its performance to a variety of corpora, including expressive ones.

我们提出 Parallel WaveGAN：一种免蒸馏、快速且小体积的基于 GAN 的波形生成方法。通过联合优化波形域对抗损失与多分辨率 STFT 损失，模型无需复杂概率密度蒸馏即可学习生成真实波形。实验表明，在基于 Transformer 的 TTS 框架内达到 $4.16$ MOS，与传统基于蒸馏的方法相当，并以仅 1.44 M 参数、28.68 倍实时速度生成 24 kHz 语音。未来工作包括改进多分辨率 STFT 辅助损失以更好刻画语音特性（例如引入相位相关损失），并在更多语料（含表现力语料）上验证表现。

6 Acknowledgements

6 致谢

The work was supported by Clova Voice, NAVER Corp., Seongnam, Korea. The authors would like to thank Adrian Kim, Jung-Woo Ha, Muhammad Ferjad Naeem, and Xiaodong Gu at NAVER Corp., Seongnam, Korea, for their support.

本工作得到韩国城南 NAVER Corp. Clova Voice 支持。作者感谢 NAVER Corp.（韩国城南）的 Adrian Kim、Jung-Woo Ha、Muhammad Ferjad Naeem 与 Xiaodong Gu 的支持。

References

参考文献

[1] H. Zen, A. Senior, and M. Schuster, “Statistical parametric speech synthesis using deep neural networks,” in Proc. ICASSP , 2013, pp. 7962–7966.

[2] E. Song, F. K. Soong, and H.-G. Kang, “Effective spectral and excitation modeling techniques for LSTM-RNN-based speech synthesis systems,” IEEE/ACM Trans. Audio, Speech, and Lang. Process. , vol. 25, no. 11, pp. 2152–2161, 2017.

[3] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan et al. , “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. ICASSP , 2018, pp. 4779–4783.

[4] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499 , 2016.

[5] X. Wang, J. Lorenzo-Trueba, S. Takaki, L. Juvela, and J. Yamagishi, “A comparison of recent waveform generation and acoustic modeling methods for neural-network-based speech synthesis,” in Proc. ICASSP , 2018, pp. 4804–4808.

[6] A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda, and T. Toda, “Speaker-dependent WaveNet vocoder,” in Proc. INTERSPEECH , 2017, pp. 1118–1122.

[7] T. Hayashi, A. Tamamori, K. Kobayashi, K. Takeda, and T. Toda, “An investigation of multi-speaker training for WaveNet vocoder,” in Proc. ASRU , 2017, pp. 712–718.

[8] E. Song, K. Byun, and H.-G. Kang, “Excitnet vocoder: A neural excitation model for parametric speech synthesis systems,” in Proc. EUSIPCO , 2019, pp. 1179–1183.

[9] A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. van den Driessche, E. Lockhart, L. C. Cobo, F. Stimberg et al. , “Parallel WaveNet: Fast high-fidelity speech synthesis,” in Proc. ICML , 2018, pp. 3915–3923.

[10] W. Ping, K. Peng, and J. Chen, “ClariNet: Parallel wave generation in end-to-end text-to-speech,” in Proc. ICLR , 2019.

[11] R. Yamamoto, E. Song, and J.-M. Kim, “Probability density distillation with generative adversarial networks for high-quality parallel waveform generation,” in Proc. INTERSPEECH , 2019, pp. 699–703.

[12] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling, “Improved variational inference with inverse autoregressive flow,” in Proc. NIPS , 2016, pp. 4743–4751.

[13] C. Donahue, J. McAuley, and M. Puckette, “Adversarial audio synthesis,” in Proc. ICLR , 2019.

[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS , 2014, pp. 2672–2680.

[15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proc. NIPS , 2017, pp. 5998–6008.

[16] N. Li, S. Liu, Y. Liu, S. Zhao, M. Liu, and M. T. Zhou, “Neural speech synthesis with Transformer network,” in Proc. AAAI , 2019, pp. 6706–6713.

[17] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Liu, “FastSpeech: Fast, robust and controllable text to speech,” arXiv preprint arXiv:1905.09263 , 2019.

[18] L. Juvela, B. Bollepalli, J. Yamagishi, and P. Alku, “GELP: GAN-excited linear prediction for speech synthesis from mel-spectrogram,” in Proc. INTERSPEECH , 2019, pp. 694–698.

[19] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least squares generative adversarial networks,” in Proc. ICCV , 2017, pp. 2794–2802.

[20] Q. Tian, X. Wan, and S. Liu, “Generative adversarial network based speaker adaptation for high fidelity WaveNet vocoder,” in Proc. SSW , 2019, pp. 19–23.

[21] B. Bollepalli, L. Juvela, and P. Alku, “Generative adversarial network-based glottal waveform model for statistical parametric speech synthesis,” in Proc. INTERSPEECH , 2017, pp. 3394–3398.

[22] S. Pascual, A. Bonafonte, and J. Serrà, “SEGAN: Speech enhancement generative adversarial network,” in Proc. INTERSPEECH , 2017, pp. 3642–3646.

[23] S. Ö. Arık, H. Jun, and G. Diamos, “Fast spectrogram inversion using multi-head convolutional neural networks,” IEEE Signal Procees. Letters , vol. 26, no. 1, pp. 94–98, 2019.

[24] I. Daubechies, “The wavelet transform, time-frequency localization and signal analysis,” IEEE trans. on information theory , vol. 36, no. 5, pp. 961–1005, 1990.

[25] X. Wang, S. Takaki, and J. Yamagishi, “Neural source-filter-based waveform model for statistical parametric speech synthesis,” in Proc. ICASSP , 2019, pp. 5916–5920.

[26] T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” in Proc. NIPS , 2016, pp. 901–909.

[27] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the variance of the adaptive learning rate and beyond,” arXiv preprint arXiv:1908.03265 , 2019.

[28] A. Odena, V. Dumoulin, and C. Olah, “Deconvolution and checkerboard artifacts,” Distill , 2016. [Online]. Available: http://distill.pub/2016/deconv-checkerboard

[29] H. Kim, M. Kim, D. Seo, J. Kim, H. Park, S. Park, H. Jo, K. Kim, Y. Yang, Y. Kim et al. , “NSML: Meet the mlaas platform with a real-world case study,” arXiv preprint arXiv:1810.09957 , 2018.

[30] Y. Yasuda, X. Wang, S. Takaki, and J. Yamagishi, “Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language,” in Proc. ICASSP , 2019, pp. 6905–6909.