StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
摘要1. 引言2. 相关工作3. Star 生成对抗网络3.1. 多域图像到图像转换3.2. 使用多个数据集训练4. 实现5. 实验5.1. 基线模型5.2. 数据集5.3. 训练5.4. 在 CelebA 上的实验结果5.5. 在 RaFD 上的实验结果5.6. 在 CelebA+RaFD 上的实验结果6. 结论参考文献
返回论文列表
目录

StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation

收录于 CVPR 2018 会议论文;Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8789-8797。对照用官方 9 页 CVF PDF(页脚 8789–8797)被引 5,081 次
打开论文来源

StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation

Yunjey Choi1,2^{1,2}1,2 Minje Choi1,2^{1,2}1,2 Munyoung Kim2,3^{2,3}2,3 Jung-Woo Ha2^{2}2 Sunghun Kim2,4^{2,4}2,4 Jaegul Choo1,2^{1,2}1,2

1^{1}1 Korea University 2^{2}2 Clova AI Research, NAVER Corp. 3^{3}3 The College of New Jersey 4^{4}4 Hong Kong University of Science & Technology

Multi-domain translation results on CelebA using knowledge transferred from RaFD

Figure 1. Multi-domain image-to-image translation results on the CelebA dataset via transferring knowledge learned from the RaFD dataset. The first and sixth columns show input images while the remaining columns are images generated by StarGAN. Note that the images are generated by a single generator network, and facial expression labels such as angry, happy, and fearful are from RaFD, not CelebA.

图 1. 在 CelebA 数据集上的多域图像到图像转换结果,其中用到了从 RaFD 数据集学到的知识。第 1 列和第 6 列是输入图像,其余各列是 StarGAN 生成的图像。注意这些图像都由同一个生成器网络生成;angry、happy、fearful 这类表情标签来自 RaFD,而不是 CelebA。

Abstract

摘要

Recent studies have shown remarkable success in image-to-image translation for two domains. However, existing approaches have limited scalability and robustness in handling more than two domains, since different models should be built independently for every pair of image domains. To address this limitation, we propose StarGAN, a novel and scalable approach that can perform image-to-image translations for multiple domains using only a single model. Such a unified model architecture of StarGAN allows simultaneous training of multiple datasets with different domains within a single network. This leads to StarGAN's superior quality of translated images compared to existing models as well as the novel capability of flexibly translating an input image to any desired target domain. We empirically demonstrate the effectiveness of our approach on a facial attribute transfer and a facial expression synthesis tasks.

近来的研究表明,在两个域之间的图像到图像转换上已经取得了显著成功。但现有方法在处理两个以上域时,可扩展性和稳健性都有限,因为每对图像域都要单独建一套模型。为解决这一限制,我们提出 StarGAN:这是一种新的、可扩展的方法,只需单个模型就能在多个域之间做图像到图像转换。StarGAN 这种统一的模型结构,允许在同一个网络里同时训练包含不同域的多个数据集。与现有模型相比,由此得到的转换图像质量更高,并且能够把输入图像灵活转换到任意指定的目标域。我们在人脸属性迁移和面部表情合成任务上,用实验验证了这一方法的效果。

1. Introduction

1. 引言

The task of image-to-image translation is to change a particular aspect of a given image to another, e.g., changing the facial expression of a person from smiling to frowning (see Fig. 1). This task has experienced significant improvements following the introduction of generative adversarial networks (GANs), with results ranging from changing hair color [8], reconstructing photos from edge maps [7], and changing the seasons of scenery images [32].

图像到图像转换的任务,是把给定图像中的某个方面改成另一种样子,例如把一个人的面部表情从微笑改成皱眉(见图 1)。生成对抗网络(GAN)出现以后,这一任务有了显著进展,结果涵盖改发色 [8]、从边缘图重建照片 [7],以及改变风景图像的季节 [32]。

Given training data from two different domains, these models learn to translate images from one domain to the other. We denote the terms attribute as a meaningful feature inherent in an image such as hair color, gender or age, and attribute value as a particular value of an attribute, e.g., black/blond/brown for hair color or male/female for gender. We further denote domain as a set of images sharing the same attribute value. For example, images of women can represent one domain while those of men represent another.

如果训练数据来自两个不同的域,这些模型就学习把图像从一个域转换到另一个域。我们把属性(attribute)定义为图像中固有的有意义特征,例如发色、性别或年龄;把属性值(attribute value)定义为某一属性的具体取值,例如发色的黑/金/棕,或性别的男/女。我们再把域(domain)定义为共享同一属性值的一组图像。例如,女性图像可以代表一个域,男性图像代表另一个域。

Several image datasets come with a number of labeled attributes. For instance, the CelebA [18] dataset contains 40 labels related to facial attributes such as hair color, gender, and age, and the RaFD [12] dataset has 8 labels for facial expressions such as 'happy', 'angry' and 'sad'. These settings enable us to perform more interesting tasks, namely multi-domain image-to-image translation, where we change images according to attributes from multiple domains. The first five columns in Fig. 1 show how a CelebA image can be translated according to any of the four domains, 'blond hair', 'gender', 'aged', and 'pale skin'. We can further extend to training multiple domains from different datasets, such as jointly training CelebA and RaFD images to change a CelebA image's facial expression using features learned by training on RaFD, as in the rightmost columns of Fig. 1.

不少图像数据集都带有多个已标注属性。例如,CelebA [18] 数据集包含 40 个与面部属性相关的标签,如发色、性别和年龄;RaFD [12] 数据集则有 8 个表情标签,如 'happy'、'angry' 和 'sad'。这样的设定让我们可以做更有意思的任务,也就是多域图像到图像转换:按来自多个域的属性来改图像。图 1 的前五列展示了一张 CelebA 图像如何按 'blond hair'、'gender'、'aged' 和 'pale skin' 这四个域中的任意一个进行转换。我们还可以进一步扩展到来自不同数据集的多个域,例如把 CelebA 和 RaFD 的图像联合训练,从而借助在 RaFD 上学到的特征,去改变一张 CelebA 图像的面部表情,如图 1 最右侧几列所示。

However, existing models are both inefficient and ineffective in such multi-domain image translation tasks. Their inefficiency results from the fact that in order to learn all mappings among kkk domains, k(k−1)k(k-1)k(k−1) generators have to be trained. Fig. 2 illustrates how twelve distinct generator networks have to be trained to translate images among four different domains. Meanwhile, they are ineffective that even though there exist global features that can be learned from images of all domains such as face shapes, each generator cannot fully utilize the entire training data and only can learn from two domains out of kkk. Failure to fully utilize training data is likely to limit the quality of generated images. Furthermore, they are incapable of jointly training domains from different datasets because each dataset is partially labeled, which we further discuss in Section 3.2.

然而,现有模型在这类多域图像转换任务上既低效,效果也不好。低效是因为要学习 kkk 个域之间的全部映射,就必须训练 k(k−1)k(k-1)k(k−1) 个生成器。图 2 说明,要在四个不同域之间转换图像,需要训练 12 个彼此不同的生成器网络。与此同时,它们的效果也不好:尽管像脸型这样的全局特征本可以从所有域的图像中学习,但每个生成器无法充分利用全部训练数据,只能从 kkk 个域里的两个域学习。训练数据利用不充分,很可能会限制生成图像的质量。此外,它们也无法联合训练来自不同数据集的域,因为每个数据集都只是部分标注的,这一点我们在第 3.2 节再讨论。

As a solution to such problems we propose StarGAN, a generative adversarial network capable of learning mappings among multiple domains. As demonstrated in Fig. 2(b), our model takes in training data of multiple domains, and learns the mappings between all available domains using only one generator. The idea is simple. Instead of learning a fixed translation (e.g., black-to-blond hair), our model takes in as inputs both image and domain information, and learns to flexibly translate the input image into the corresponding domain. We use a label (e.g., binary or one-hot vector) to represent domain information. During training, we randomly generate a target domain label and train the model to flexibly translate an input image into the target domain. By doing so, we can control the domain label and translate the image into any desired domain at testing phase.

作为对这些问题的解决,我们提出 StarGAN,这是一种能够学习多个域之间映射的生成对抗网络。如图 2(b) 所示,我们的模型接收多个域的训练数据,只用一个生成器学习所有可用域之间的映射。想法很简单。模型不再学习一种固定转换(例如黑发到金发),而是同时把图像和域信息作为输入,学习把输入图像灵活转换到相应的域。我们用标签(例如二进制向量或独热向量)来表示域信息。训练时,我们随机生成目标域标签,训练模型把输入图像灵活转换到该目标域。这样,在测试阶段就可以控制域标签,把图像转换到任意想要的域。

We also introduce a simple but effective approach that enables joint training between domains of different datasets by adding a mask vector to the domain label. Our proposed method ensures that the model can ignore unknown labels and focus on the label provided by a particular dataset. In this manner, our model can perform well on tasks such as synthesizing facial expressions of CelebA images using features learned from RaFD, as shown in the rightmost columns of Fig. 1. As far as our knowledge goes, our work is the first to successfully perform multi-domain image translation across different datasets.

我们还提出一种简单而有效的方法:在域标签上加入掩码向量,从而在来自不同数据集的域之间做联合训练。该方法保证模型可以忽略未知标签,把注意力放在某一数据集提供的标签上。这样,我们的模型就能在这类任务上表现良好,例如借助从 RaFD 学到的特征,为 CelebA 图像合成面部表情,如图 1 最右侧几列所示。据我们所知,这是首次成功在不同数据集之间完成多域图像转换的工作。

Comparison of pairwise cross-domain models with a single StarGAN generator

Figure 2. Comparison between cross-domain models and our proposed model, StarGAN. (a) To handle multiple domains, cross-domain models should be built for every pair of image domains. (b) StarGAN is capable of learning mappings among multiple domains using a single generator. The figure represents a star topology connecting multi-domains.

图 2. 跨域模型与我们提出的模型 StarGAN 的比较。(a) 要处理多个域,跨域模型必须为每一对图像域分别建立。(b) StarGAN 能够用单个生成器学习多个域之间的映射。该图表示的是连接多个域的星形拓扑。

Overall, our contributions are as follows:

总的来说,我们的贡献如下:

  • We propose StarGAN, a novel generative adversarial network that learns the mappings among multiple domains using only a single generator and a discriminator, training effectively from images of all domains.
  • 我们提出 StarGAN,这是一种新的生成对抗网络,只用一个生成器和一个判别器学习多个域之间的映射,并能有效地利用所有域的图像进行训练。
  • We demonstrate how we can successfully learn multi-domain image translation between multiple datasets by utilizing a mask vector method that enables StarGAN to control all available domain labels.
  • 我们展示了如何利用掩码向量方法,成功学习多个数据集之间的多域图像转换,从而使 StarGAN 能够控制所有可用的域标签。
  • We provide both qualitative and quantitative results on facial attribute transfer and facial expression synthesis tasks using StarGAN, showing its superiority over baseline models.
  • 我们在人脸属性迁移和面部表情合成任务上,给出了 StarGAN 的定性和定量结果,表明它优于基线模型。

2. Related Work

2. 相关工作

Generative Adversarial Networks. Generative adversarial networks (GANs) [3] have shown remarkable results in various computer vision tasks such as image generation [1, 6, 23, 31], image translation [7, 8, 32], super-resolution imaging [13], and face image synthesis [9, 15, 25, 30]. A typical GAN model consists of two modules: a discriminator and a generator. The discriminator learns to distinguish between real and fake samples, while the generator learns to generate fake samples that are indistinguishable from real samples. Our approach also leverages the adversarial loss to make the generated images as realistic as possible.

生成对抗网络。 生成对抗网络(GAN)[3] 已在多种计算机视觉任务中取得显著结果,例如图像生成 [1, 6, 23, 31]、图像转换 [7, 8, 32]、超分辨率成像 [13] 以及人脸图像合成 [9, 15, 25, 30]。典型的 GAN 模型由两个模块组成:判别器和生成器。判别器学习区分真样本和假样本,生成器则学习生成与真样本难以区分的假样本。我们的方法也利用对抗损失,使生成图像尽可能逼真。

Conditional GANs. GAN-based conditional image generation has also been actively studied. Prior studies have provided both the discriminator and generator with class information in order to generate samples conditioned on the class [19, 20, 21]. Other recent approaches focused on generating particular images highly relevant to a given text description [24, 29]. The idea of conditional image generation has also been successfully applied to domain transfer [8, 27], super-resolution imaging [13], and photo editing [2, 26]. In this paper, we propose a scalable GAN framework that can flexibly steer the image translation to various target domains, by providing conditional domain information.

条件 GAN。 基于 GAN 的条件图像生成也一直是活跃的研究方向。先前的研究向判别器和生成器同时提供类别信息,以便按类别条件生成样本 [19, 20, 21]。另一些近期方法则关注生成与给定文本描述高度相关的特定图像 [24, 29]。条件图像生成的想法也已成功用于域迁移 [8, 27]、超分辨率成像 [13] 和照片编辑 [2, 26]。在本文中,我们提出一种可扩展的 GAN 框架,通过提供条件域信息,能够灵活地把图像转换引导到各种目标域。

Image-to-Image Translation. Recent work have achieved impressive results in image-to-image translation [7, 8, 16, 32]. For instance, pix2pix [7] learns this task in a supervised manner using cGANs [19]. It combines an adversarial loss with a L1 loss, thus requires paired data samples. To alleviate the problem of obtaining data pairs, unpaired image-to-image translation frameworks [8, 16, 32] have been proposed. UNIT [16] combines variational autoencoders (VAEs) [11] with CoGAN [17], a GAN framework where two generators share weights to learn the joint distribution of images in cross domains. CycleGAN [32] and DiscoGAN [8] preserve key attributes between the input and the translated image by utilizing a cycle consistency loss. However, all these frameworks are only capable of learning the relations between two different domains at a time. Their approaches have limited scalability in handling multiple domains since different models should be trained for each pair of domains. Unlike the aforementioned approaches, our framework can learn the relations among multiple domains using only a single model.

图像到图像转换。 近期工作在图像到图像转换上取得了令人印象深刻的结果 [7, 8, 16, 32]。例如,pix2pix [7] 用条件 GAN(cGAN)[19] 以有监督方式学习这一任务。它把对抗损失与 L1L_1L1​ 损失结合起来,因此需要成对的数据样本。为减轻获取数据对的困难,人们提出了无须配对的图像到图像转换框架 [8, 16, 32]。UNIT [16] 把变分自编码器(VAE)[11] 与 CoGAN [17] 结合起来,后者是一种两个生成器共享权重、以学习跨域图像联合分布的 GAN 框架。CycleGAN [32] 和 DiscoGAN [8] 则利用循环一致性损失,在输入图像和转换后图像之间保留关键属性。不过,这些框架一次都只能学习两个不同域之间的关系。由于每对域都要训练不同的模型,它们在处理多个域时的可扩展性有限。与上述方法不同,我们的框架只用单个模型就能学习多个域之间的关系。

Overview of StarGAN discriminator and generator training

Figure 3. Overview of StarGAN, consisting of two modules, a discriminator DDD and a generator GGG. (a) DDD learns to distinguish between real and fake images and classify the real images to its corresponding domain. (b) GGG takes in as input both the image and target domain label and generates an fake image. The target domain label is spatially replicated and concatenated with the input image. (c) GGG tries to reconstruct the original image from the fake image given the original domain label. (d) GGG tries to generate images indistinguishable from real images and classifiable as target domain by DDD.

图 3. StarGAN 概览,由判别器 DDD 和生成器 GGG 两个模块组成。(a) DDD 学习区分真实图像和伪造图像,并把真实图像分类到其对应的域。(b) GGG 同时把图像和目标域标签作为输入,并生成一张伪造图像。目标域标签在空间上复制后,与输入图像拼接。(c) GGG 在给定原始域标签的情况下,尝试从伪造图像重建原始图像。(d) GGG 尝试生成与真实图像无法区分、并能被 DDD 分类为目标域的图像。

3. Star Generative Adversarial Networks

3. Star 生成对抗网络

We first describe our proposed StarGAN, a framework to address multi-domain image-to-image translation within a single dataset. Then, we discuss how StarGAN incorporates multiple datasets containing different label sets to flexibly perform image translations using any of these labels.

我们先介绍提出的 StarGAN,这是一个在单个数据集内解决多域图像到图像转换的框架。随后再讨论 StarGAN 如何纳入包含不同标签集的多个数据集,从而灵活地使用其中任意标签来做图像转换。

3.1. Multi-Domain Image-to-Image Translation

3.1. 多域图像到图像转换

Our goal is to train a single generator GGG that learns mappings among multiple domains. To achieve this, we train GGG to translate an input image xxx into an output image yyy conditioned on the target domain label ccc, G(x,c)→yG(x, c) \rightarrow yG(x,c)→y. We randomly generate the target domain label ccc so that GGG learns to flexibly translate the input image. We also introduce an auxiliary classifier [21] that allows a single discriminator to control multiple domains. That is, our discriminator produces probability distributions over both sources and domain labels, D:x→{Dsrc(x),Dcls(x)}D: x \rightarrow \{D_{\mathrm{src}}(x), D_{\mathrm{cls}}(x)\}D:x→{Dsrc​(x),Dcls​(x)}. Fig. 3 illustrates the training process of our proposed approach.

我们的目标是训练单个生成器 GGG,使其学习多个域之间的映射。为此,我们训练 GGG 在目标域标签 ccc 的条件下,把输入图像 xxx 转换成输出图像 yyy,即 G(x,c)→yG(x, c) \rightarrow yG(x,c)→y。我们随机生成目标域标签 ccc,以便 GGG 学会灵活地转换输入图像。我们还引入辅助分类器 [21],使单个判别器能够控制多个域。也就是说,我们的判别器同时给出关于来源和域标签的概率分布,D:x→{Dsrc(x),Dcls(x)}D: x \rightarrow \{D_{\mathrm{src}}(x), D_{\mathrm{cls}}(x)\}D:x→{Dsrc​(x),Dcls​(x)}。图 3 展示了所提方法的训练过程。

Adversarial Loss. To make the generated images indistinguishable from real images, we adopt an adversarial loss

对抗损失。 为使生成图像与真实图像难以区分,我们采用对抗损失

Ladv=Ex[log⁡Dsrc(x)]+Ex,c[log⁡(1−Dsrc(G(x,c)))],(1)\mathcal{L}_{\mathrm{adv}} = \mathbb{E}_{x}[\log D_{\mathrm{src}}(x)] + \mathbb{E}_{x,c}[\log (1 - D_{\mathrm{src}}(G(x, c)))], \tag{1}Ladv​=Ex​[logDsrc​(x)]+Ex,c​[log(1−Dsrc​(G(x,c)))],(1)

where GGG generates an image G(x,c)G(x, c)G(x,c) conditioned on both the input image xxx and the target domain label ccc, while DDD tries to distinguish between real and fake images. In this paper, we refer to the term Dsrc(x)D_{\mathrm{src}}(x)Dsrc​(x) as a probability distribution over sources given by DDD. The generator GGG tries to minimize this objective, while the discriminator DDD tries to maximize it.

其中 GGG 在输入图像 xxx 和目标域标签 ccc 的条件下生成图像 G(x,c)G(x, c)G(x,c),而 DDD 则试图区分真实图像和伪造图像。在本文中,我们把 Dsrc(x)D_{\mathrm{src}}(x)Dsrc​(x) 称为 DDD 给出的关于来源的概率分布。生成器 GGG 试图最小化这一目标,判别器 DDD 则试图最大化它。

Domain Classification Loss. For a given input image xxx and a target domain label ccc, our goal is to translate xxx into an output image yyy, which is properly classified to the target domain ccc. To achieve this condition, we add an auxiliary classifier on top of DDD and impose the domain classification loss when optimizing both DDD and GGG. That is, we decompose the objective into two terms: a domain classification loss of real images used to optimize DDD, and a domain classification loss of fake images used to optimize GGG. In detail, the former is defined as

域分类损失。 对给定的输入图像 xxx 和目标域标签 ccc,我们的目标是把 xxx 转换成输出图像 yyy,并且 yyy 被正确分类到目标域 ccc。为满足这一条件,我们在 DDD 之上添加一个辅助分类器,并在优化 DDD 和 GGG 时施加域分类损失。也就是说,我们把目标分解成两项:用于优化 DDD 的真实图像域分类损失,以及用于优化 GGG 的伪造图像域分类损失。具体而言,前者定义为

Lclsr=Ex,c′[−log⁡Dcls(c′∣x)],(2)\mathcal{L}_{\mathrm{cls}}^{r} = \mathbb{E}_{x,c'}[-\log D_{\mathrm{cls}}(c'|x)], \tag{2}Lclsr​=Ex,c′​[−logDcls​(c′∣x)],(2)

where the term Dcls(c′∣x)D_{\mathrm{cls}}(c'|x)Dcls​(c′∣x) represents a probability distribution over domain labels computed by DDD. By minimizing this objective, DDD learns to classify a real image xxx to its corresponding original domain c′c'c′. We assume that the input image and domain label pair (x,c′)(x, c')(x,c′) is given by the training data. On the other hand, the loss function for the domain classification of fake images is defined as

其中 Dcls(c′∣x)D_{\mathrm{cls}}(c'|x)Dcls​(c′∣x) 表示由 DDD 计算出的域标签概率分布。通过最小化这一目标,DDD 学习把真实图像 xxx 分类到其对应的原始域 c′c'c′。我们假定输入图像与域标签对 (x,c′)(x, c')(x,c′) 由训练数据给出。另一方面,伪造图像的域分类损失函数定义为

Lclsf=Ex,c[−log⁡Dcls(c∣G(x,c))].(3)\mathcal{L}_{\mathrm{cls}}^{f} = \mathbb{E}_{x,c}[-\log D_{\mathrm{cls}}(c|G(x, c))]. \tag{3}Lclsf​=Ex,c​[−logDcls​(c∣G(x,c))].(3)

In other words, GGG tries to minimize this objective to generate images that can be classified as the target domain ccc.

换句话说,GGG 试图最小化这一目标,以生成可被分类为目标域 ccc 的图像。

Reconstruction Loss. By minimizing the adversarial and classification losses, GGG is trained to generate images that are realistic and classified to its correct target domain. However, minimizing the losses (Eqs. (1) and (3)) does not guarantee that translated images preserve the content of its input images while changing only the domain-related part of the inputs. To alleviate this problem, we apply a cycle consistency loss [8, 32] to the generator, defined as

重建损失。 通过最小化对抗损失和分类损失,GGG 被训练成生成既逼真、又能被分到正确目标域的图像。然而,仅最小化这些损失(式 (1) 和式 (3))并不能保证转换后的图像在只改变输入中与域相关的部分时,仍保留输入图像的内容。为缓解这一问题,我们对生成器施加循环一致性损失 [8, 32],定义为

Lrec=Ex,c,c′[∥x−G(G(x,c),c′)∥1],(4)\mathcal{L}_{\mathrm{rec}} = \mathbb{E}_{x,c,c'}[\|x - G(G(x, c), c')\|_{1}], \tag{4}Lrec​=Ex,c,c′​[∥x−G(G(x,c),c′)∥1​],(4)

where GGG takes in the translated image G(x,c)G(x, c)G(x,c) and the original domain label c′c'c′ as input and tries to reconstruct the original image xxx. We adopt the L1L1L1 norm as our reconstruction loss. Note that we use a single generator twice, first to translate an original image into an image in the target domain and then to reconstruct the original image from the translated image.

其中 GGG 把转换后的图像 G(x,c)G(x, c)G(x,c) 和原始域标签 c′c'c′ 作为输入,并试图重建原始图像 xxx。我们采用 L1L1L1 范数作为重建损失。注意,我们把同一个生成器使用两次:先把原始图像转换到目标域,再从转换后的图像重建原始图像。

Full Objective. Finally, the objective functions to optimize GGG and DDD are written, respectively, as

总体目标。 最后,用于优化 GGG 和 DDD 的目标函数分别写为

LD=−Ladv+λcls Lclsr,(5)\mathcal{L}_{D} = -\mathcal{L}_{\mathrm{adv}} + \lambda_{\mathrm{cls}}\,\mathcal{L}_{\mathrm{cls}}^{r}, \tag{5}LD​=−Ladv​+λcls​Lclsr​,(5)

LG=Ladv+λcls Lclsf+λrec Lrec,(6)\mathcal{L}_{G} = \mathcal{L}_{\mathrm{adv}} + \lambda_{\mathrm{cls}}\,\mathcal{L}_{\mathrm{cls}}^{f} + \lambda_{\mathrm{rec}}\,\mathcal{L}_{\mathrm{rec}}, \tag{6}LG​=Ladv​+λcls​Lclsf​+λrec​Lrec​,(6)

where λcls\lambda_{\mathrm{cls}}λcls​ and λrec\lambda_{\mathrm{rec}}λrec​ are hyper-parameters that control the relative importance of domain classification and reconstruction losses, respectively, compared to the adversarial loss. We use λcls=1\lambda_{\mathrm{cls}} = 1λcls​=1 and λrec=10\lambda_{\mathrm{rec}} = 10λrec​=10 in all of our experiments.

其中 λcls\lambda_{\mathrm{cls}}λcls​ 和 λrec\lambda_{\mathrm{rec}}λrec​ 是超参数,分别控制域分类损失和重建损失相对于对抗损失的重要程度。在全部实验中,我们取 λcls=1\lambda_{\mathrm{cls}} = 1λcls​=1 和 λrec=10\lambda_{\mathrm{rec}} = 10λrec​=10。

3.2. Training with Multiple Datasets

3.2. 使用多个数据集训练

An important advantage of StarGAN is that it simultaneously incorporates multiple datasets containing different types of labels, so that StarGAN can control all the labels at the test phase. An issue when learning from multiple datasets, however, is that the label information is only partially known to each dataset. In the case of CelebA [18] and RaFD [12], while the former contains labels for attributes such as hair color and gender, it does not have any labels for facial expressions such as 'happy' and 'angry', and vice versa for the latter. This is problematic because the complete information on the label vector c′c'c′ is required when reconstructing the input image xxx from the translated image G(x,c)G(x, c)G(x,c) (See Eq. (4)).

StarGAN 的一个重要优点是,它可以同时纳入包含不同类型标签的多个数据集,从而在测试阶段控制所有标签。不过,从多个数据集学习时有一个问题:每个数据集都只知道部分标签信息。以 CelebA [18] 和 RaFD [12] 为例,前者包含发色、性别一类属性标签,却没有任何 'happy'、'angry' 这类表情标签,后者则正好相反。这会带来困难,因为从转换后的图像 G(x,c)G(x, c)G(x,c) 重建输入图像 xxx 时,需要标签向量 c′c'c′ 的完整信息(见式 (4))。

Mask Vector. To alleviate this problem, we introduce a mask vector mmm that allows StarGAN to ignore unspecified labels and focus on the explicitly known label provided by a particular dataset. In StarGAN, we use an nnn-dimensional one-hot vector to represent mmm, with nnn being the number of datasets. In addition, we define a unified version of the label as a vector

掩码向量。 为缓解这一问题,我们引入掩码向量 mmm,使 StarGAN 能够忽略未指定的标签,把注意力放在某一数据集明确给出的已知标签上。在 StarGAN 中,我们用 nnn 维独热向量表示 mmm,其中 nnn 是数据集的个数。此外,我们把统一后的标签定义为向量

c~=[c1,…,cn,m],(7)\tilde{c} = [c_1, \ldots, c_n, m], \tag{7}c~=[c1​,…,cn​,m],(7)

where [⋅][\cdot][⋅] refers to concatenation, and cic_ici​ represents a vector for the labels of the iii-th dataset. The vector of the known label cic_ici​ can be represented as either a binary vector for binary attributes or a one-hot vector for categorical attributes. For the remaining n−1n-1n−1 unknown labels we simply assign zero values. In our experiments, we utilize the CelebA and RaFD datasets, where nnn is two.

其中 [⋅][\cdot][⋅] 表示拼接,cic_ici​ 表示第 iii 个数据集的标签向量。已知标签的向量 cic_ici​ 既可以表示成二进制属性的二进制向量,也可以表示成类别属性的独热向量。对其余 n−1n-1n−1 个未知标签,我们直接赋零值。在实验中,我们使用 CelebA 和 RaFD 两个数据集,此时 nnn 为 2。

Training Strategy. When training StarGAN with multiple datasets, we use the domain label c~\tilde{c}c~ defined in Eq. (7) as input to the generator. By doing so, the generator learns to ignore the unspecified labels, which are zero vectors, and focus on the explicitly given label. The structure of the generator is exactly the same as in training with a single dataset, except for the dimension of the input label c~\tilde{c}c~. On the other hand, we extend the auxiliary classifier of the discriminator to generate probability distributions over labels for all datasets. Then, we train the model in a multi-task learning setting, where the discriminator tries to minimize only the classification error associated to the known label. For example, when training with images in CelebA, the discriminator minimizes only classification errors for labels related to CelebA attributes, and not facial expressions related to RaFD. Under these settings, by alternating between CelebA and RaFD the discriminator learns all of the discriminative features for both datasets, and the generator learns to control all the labels in both datasets.

训练策略。 当用多个数据集训练 StarGAN 时,我们把式 (7) 中定义的域标签 c~\tilde{c}c~ 作为生成器的输入。这样,生成器就会学习忽略那些作为零向量的未指定标签,把注意力放在明确给出的标签上。除了输入标签 c~\tilde{c}c~ 的维数以外,生成器的结构与在单个数据集上训练时完全相同。另一方面,我们扩展判别器的辅助分类器,使其对所有数据集的标签给出概率分布。随后,我们在多任务学习设定下训练模型:判别器只试图最小化与已知标签相关的分类误差。例如,用 CelebA 中的图像训练时,判别器只最小化与 CelebA 属性相关的分类误差,而不考虑与 RaFD 相关的面部表情。在这些设定下,通过在 CelebA 和 RaFD 之间交替,判别器学习两个数据集的全部判别特征,生成器则学习控制两个数据集中的全部标签。

Facial attribute transfer results on CelebA comparing DIAT, CycleGAN, IcGAN, and StarGAN

Figure 4. Facial attribute transfer results on the CelebA dataset. The first column shows the input image, next four columns show the single attribute transfer results, and rightmost columns show the multi-attribute transfer results. H: Hair color, G: Gender, A: Aged.

图 4. 在 CelebA 数据集上的人脸属性迁移结果。第 1 列是输入图像,接下来四列是单属性迁移结果,最右侧几列是多属性迁移结果。H:发色;G:性别;A:年龄。

4. Implementation

4. 实现

Improved GAN Training. To stabilize the training process and generate higher quality images, we replace Eq. (1) with Wasserstein GAN objective with gradient penalty [1, 4] defined as

改进的 GAN 训练。 为稳定训练过程并生成更高质量的图像,我们用带梯度惩罚的 Wasserstein GAN 目标 [1, 4] 替换式 (1),定义为

Ladv=Ex[Dsrc(x)]−Ex,c[Dsrc(G(x,c))]−λgp Ex^[(∥∇x^Dsrc(x^)∥2−1)2],(8)\mathcal{L}_{\mathrm{adv}} = \mathbb{E}_{x}[D_{\mathrm{src}}(x)] - \mathbb{E}_{x,c}[D_{\mathrm{src}}(G(x, c))] - \lambda_{\mathrm{gp}}\,\mathbb{E}_{\hat{x}}[(\|\nabla_{\hat{x}} D_{\mathrm{src}}(\hat{x})\|_{2} - 1)^{2}], \tag{8}Ladv​=Ex​[Dsrc​(x)]−Ex,c​[Dsrc​(G(x,c))]−λgp​Ex^​[(∥∇x^​Dsrc​(x^)∥2​−1)2],(8)

where x^\hat{x}x^ is sampled uniformly along a straight line between a pair of a real and a generated images. We use λgp=10\lambda_{\mathrm{gp}} = 10λgp​=10 for all experiments.

其中 x^\hat{x}x^ 在一对真实图像与生成图像之间的直线上均匀采样。全部实验中我们取 λgp=10\lambda_{\mathrm{gp}} = 10λgp​=10。

Network Architecture. Adapted from [32], StarGAN has the generator network composed of two convolutional layers with the stride size of two for downsampling, six residual blocks [5], and two transposed convolutional layers with the stride size of two for upsampling. We use instance normalization [28] for the generator but no normalization for the discriminator. We leverage PatchGANs [7, 14, 32] for the discriminator network, which classifies whether local image patches are real or fake. See the appendix (Section ??) for more details about the network architecture.

网络结构。 StarGAN 的生成器网络改编自 [32],由两个步长为 2 的卷积层用于下采样、六个残差块 [5],以及两个步长为 2 的转置卷积层用于上采样组成。生成器使用实例归一化 [28],判别器则不使用归一化。判别器网络采用 PatchGAN [7, 14, 32],用于判断局部图像块是真还是假。网络结构的更多细节见附录(第 ?? 节)。

5. Experiments

5. 实验

In this section, we first compare StarGAN against recent methods on facial attribute transfer by conducting user studies. Next, we perform a classification experiment on facial expression synthesis. Lastly, we demonstrate empirical results that StarGAN can learn image-to-image translation from multiple datasets. All our experiments were conducted by using the model output from unseen images during the training phase.

在这一节中,我们首先通过用户研究,在人脸属性迁移上把 StarGAN 与近期方法进行比较。接下来,我们在面部表情合成上做分类实验。最后,我们给出经验结果,表明 StarGAN 能够从多个数据集学习图像到图像转换。全部实验都使用训练阶段未见过的图像上的模型输出。

5.1. Baseline Models

5.1. 基线模型

As our baseline models, we adopt DIAT [15] and CycleGAN [32], both of which performs image-to-image translation between two different domains. For comparison, we trained these models multiple times for every pair of two different domains. We also adopt IcGAN [22] as a baseline which can perform attribute transfer using a cGAN [21].

作为基线模型,我们采用 DIAT [15] 和 CycleGAN [32],二者都在两个不同域之间做图像到图像转换。为便于比较,我们针对每一对域多次训练这些模型。我们还采用 IcGAN [22] 作为基线,它可以用 cGAN [21] 做属性迁移。

DIAT uses an adversarial loss to learn the mapping from x∈Xx \in Xx∈X to y∈Yy \in Yy∈Y, where xxx and yyy are face images in two different domains XXX and YYY, respectively. This method has a regularization term on the mapping as ∥x−F(G(x))∥1\|x - F(G(x))\|_{1}∥x−F(G(x))∥1​ to preserve identity features of the source image, where FFF is a feature extractor pretrained on a face recognition task.

DIAT 用对抗损失学习从 x∈Xx \in Xx∈X 到 y∈Yy \in Yy∈Y 的映射,其中 xxx 和 yyy 分别是两个不同域 XXX 和 YYY 中的人脸图像。该方法对映射施加正则项 ∥x−F(G(x))∥1\|x - F(G(x))\|_{1}∥x−F(G(x))∥1​,以保留源图像的身份特征,其中 FFF 是在人脸识别任务上预训练的特征提取器。

CycleGAN also uses an adversarial loss to learn the mapping between two different domains XXX and YYY. This method regularizes the mapping via cycle consistency losses, ∥x−(GYX(GXY(x)))∥1\|x - (G_{YX}(G_{XY}(x)))\|_{1}∥x−(GYX​(GXY​(x)))∥1​ and ∥y−(GXY(GYX(y)))∥1\|y - (G_{XY}(G_{YX}(y)))\|_{1}∥y−(GXY​(GYX​(y)))∥1​. This method requires two generators and discriminators for each pair of two different domains.

CycleGAN 同样用对抗损失学习两个不同域 XXX 和 YYY 之间的映射。该方法通过循环一致性损失 ∥x−(GYX(GXY(x)))∥1\|x - (G_{YX}(G_{XY}(x)))\|_{1}∥x−(GYX​(GXY​(x)))∥1​ 和 ∥y−(GXY(GYX(y)))∥1\|y - (G_{XY}(G_{YX}(y)))\|_{1}∥y−(GXY​(GYX​(y)))∥1​ 对映射进行正则。对每一对不同的域,该方法都需要两个生成器和两个判别器。

IcGAN combines an encoder with a cGAN [21] model. cGAN learns the mapping G:{z,c}→xG: \{z, c\} \rightarrow xG:{z,c}→x that generates an image xxx conditioned on both the latent vector zzz and the conditional vector ccc. In addition, IcGAN introduces an encoder to learn the inverse mappings of cGAN, Ez:x→zE_z: x \rightarrow zEz​:x→z and Ec:x→cE_c: x \rightarrow cEc​:x→c. This allows IcGAN to synthesis images by only changing the conditional vector and preserving the latent vector.

IcGAN 把编码器与 cGAN [21] 模型结合起来。cGAN 学习映射 G:{z,c}→xG: \{z, c\} \rightarrow xG:{z,c}→x,在隐向量 zzz 和条件向量 ccc 的条件下生成图像 xxx。此外,IcGAN 引入编码器来学习 cGAN 的逆映射,Ez:x→zE_z: x \rightarrow zEz​:x→z 和 Ec:x→cE_c: x \rightarrow cEc​:x→c。这样,IcGAN 只需改变条件向量、同时保留隐向量,就可以合成图像。

Facial expression synthesis results on RaFD comparing DIAT, CycleGAN, IcGAN, and StarGAN

Figure 5. Facial expression synthesis results on the RaFD dataset.

图 5. 在 RaFD 数据集上的面部表情合成结果。

5.2. Datasets

5.2. 数据集

CelebA. The CelebFaces Attributes (CelebA) dataset [18] contains 202,599 face images of celebrities, each annotated with 40 binary attributes. We crop the initial 178×218178 \times 218178×218 size images to 178×178178\times 178178×178, then resize them as 128×128128\times 128128×128. We randomly select 2,000 images as test set and use all remaining images for training data. We construct seven domains using the following attributes: hair color (black, blond, brown), gender (male/female), and age (young/old).

CelebA。 CelebFaces Attributes(CelebA)数据集 [18] 包含 202,599 张名人面部图像,每张图标注了 40 个二进制属性。我们把原始 178×218178 \times 218178×218 大小的图像裁成 178×178178\times 178178×178,再调整为 128×128128\times 128128×128。我们随机选出 2,000 张图像作为测试集,其余全部用作训练数据。我们用下列属性构造七个域:发色(黑、金、棕)、性别(男/女)以及年龄(年轻/年长)。

RaFD. The Radboud Faces Database (RaFD) [12] consists of 4,824 images collected from 67 participants. Each participant makes eight facial expressions in three different gaze directions, which are captured from three different angles. We crop the images to 256×256256 \times 256256×256, where the faces are centered, and then resize them to 128×128128 \times 128128×128.

RaFD。 Radboud Faces Database(RaFD)[12] 包含从 67 名参与者采集的 4,824 张图像。每位参与者做出八种面部表情,并有三种不同的注视方向,再从三个不同角度拍摄。我们把图像裁成面部居中的 256×256256 \times 256256×256,再调整为 128×128128 \times 128128×128。

5.3. Training

5.3. 训练

All models are trained using Adam [10] with β1=0.5\beta_1 = 0.5β1​=0.5 and β2=0.999\beta_2 = 0.999β2​=0.999. For data augmentation we flip the images horizontally with a probability of 0.5. We perform one generator update after five discriminator updates as in [4]. The batch size is set to 16 for all experiments. For experiments on CelebA, we train all models with a learning rate of 0.0001 for the first 10 epochs and linearly decay the learning rate to 0 over the next 10 epochs. To compensate for the lack of data, when training with RaFD we train all models for 100 epochs with a learning rate of 0.0001 and apply the same decaying strategy over the next 100 epochs. Training takes about one day on a single NVIDIA Tesla M40 GPU.

所有模型都用 Adam [10] 训练,其中 β1=0.5\beta_1 = 0.5β1​=0.5、β2=0.999\beta_2 = 0.999β2​=0.999。在数据增强上,我们以 0.5 的概率对图像做水平翻转。与 [4] 一样,每更新五次判别器,再更新一次生成器。全部实验的批量大小设为 16。在 CelebA 上的实验中,所有模型先以学习率 0.0001 训练 10 个 epoch,再在接下来的 10 个 epoch 里把学习率线性衰减到 0。为弥补数据不足,在 RaFD 上训练时,所有模型先以学习率 0.0001 训练 100 个 epoch,再在接下来的 100 个 epoch 里采用同样的衰减策略。训练在单块 NVIDIA Tesla M40 GPU 上大约需要一天。

5.4. Experimental Results on CelebA

5.4. 在 CelebA 上的实验结果

We first compare our proposed method to the baseline models on a single and multi-attribute transfer tasks. We train the cross-domain models such as DIAT and CycleGAN multiple times considering all possible attribute value pairs. In the case of DIAT and CycleGAN, we perform multi-step translations to synthesize multiple attributes (e.g. transferring a gender attribute after changing a hair color).

我们首先在单属性和多属性迁移任务上,把提出的方法与基线模型进行比较。对 DIAT 和 CycleGAN 这类跨域模型,我们考虑所有可能的属性值对,多次进行训练。对 DIAT 和 CycleGAN,我们用多步转换来合成多个属性(例如先改发色,再迁移性别属性)。

Qualitative evaluation. Fig. 4 shows the facial attribute transfer results on CelebA. We observed that our method provides a higher visual quality of translation results on test data compared to the cross-domain models. One possible reason is the regularization effect of StarGAN through a multi-task learning framework. In other words, rather than training a model to perform a fixed translation (e.g., brown-to-blond hair), which is prone to overfitting, we train our model to flexibly translate images according to the labels of the target domain. This allows our model to learn reliable features universally applicable to multiple domains of images with different facial attribute values.

定性评价。 图 4 给出了在 CelebA 上的人脸属性迁移结果。我们观察到,与跨域模型相比,我们的方法在测试数据上的转换结果视觉质量更高。一个可能的原因是 StarGAN 通过多任务学习框架带来的正则效果。换句话说,我们不是训练模型去做一种容易过拟合的固定转换(例如棕发到金发),而是训练模型按目标域的标签灵活转换图像。这使模型能够学习到可普遍适用于具有不同面部属性值的多个图像域的可靠特征。

Furthermore, compared to IcGAN, our model demonstrates an advantage in preserving the facial identity feature of an input. We conjecture that this is because our method maintains the spatial information by using activation maps from the convolutional layer as latent representation, rather than just a low-dimensional latent vector as in IcGAN.

此外,与 IcGAN 相比,我们的模型在保留输入的面部身份特征方面表现出优势。我们推测,这是因为我们的方法用卷积层的激活图作为隐表示来保持空间信息,而不是像 IcGAN 那样只使用低维隐向量。

Quantitative evaluation protocol. For quantitative evaluations, we performed two user studies in a survey format using Amazon Mechanical Turk (AMT) to assess single and multiple attribute transfer tasks. Given an input image, the Turkers were instructed to choose the best generated image based on perceptual realism, quality of transfer in attribute(s), and preservation of a figure's original identity. The options were four randomly shuffled images generated from four different methods. The generated images in one study have a single attribute transfer in either hair color (black, blond, brown), gender, or age. In another study, the generated images involve a combination of attribute transfers. Each Turker was asked 30 to 40 questions with a few simple yet logical questions for validating human effort. The number of validated Turkers in each user study is 146 and 100 in single and multiple transfer tasks, respectively.

定量评价方案。 为进行定量评价,我们通过 Amazon Mechanical Turk(AMT)以问卷形式做了两项用户研究,分别评估单属性和多属性迁移任务。给定一张输入图像,要求众包工人根据感知真实感、属性迁移质量,以及人物原始身份的保留程度,选出最好的生成图像。选项是由四种不同方法生成、并经随机打乱的四张图像。一项研究中的生成图像只做单属性迁移,属性为发色(黑、金、棕)、性别或年龄之一。另一项研究中的生成图像则涉及属性迁移的组合。每位众包工人要回答 30 到 40 道题,其中包含若干用于验证是否认真作答的简单逻辑题。经校验的众包工人数量在单属性与多属性迁移任务中分别为 146 和 100。

Method Hair color Gender Aged
DIAT 9.3% 31.4% 6.9%
CycleGAN 20.0% 16.6% 13.3%
IcGAN 4.5% 12.9% 9.2%
StarGAN 66.2% 39.1% 70.6%

Table 1. AMT perceptual evaluation for ranking different models on a single attribute transfer task. Each column sums to 100%.

Method Hair color Gender Aged
DIAT 9.3% 31.4% 6.9%
CycleGAN 20.0% 16.6% 13.3%
IcGAN 4.5% 12.9% 9.2%
StarGAN 66.2% 39.1% 70.6%

表 1. 在单属性迁移任务上对各模型排序的 AMT 感知评价。每列求和为 100%。

Method H+G H+A G+A H+G+A
DIAT 20.4% 15.6% 18.7% 15.6%
CycleGAN 14.0% 12.0% 11.2% 11.9%
IcGAN 18.2% 10.9% 20.3% 20.3%
StarGAN 47.4% 61.5% 49.8% 52.2%

Table 2. AMT perceptual evaluation for ranking different models on a multi-attribute transfer task. H: Hair color; G: Gender; A: Aged.

Method H+G H+A G+A H+G+A
DIAT 20.4% 15.6% 18.7% 15.6%
CycleGAN 14.0% 12.0% 11.2% 11.9%
IcGAN 18.2% 10.9% 20.3% 20.3%
StarGAN 47.4% 61.5% 49.8% 52.2%

表 2. 在多属性迁移任务上对各模型排序的 AMT 感知评价。H:发色;G:性别;A:年龄。

Quantitative results. Tables 1 and 2 show the results of our AMT experiment on single- and multi-attribute transfer tasks, respectively. StarGAN obtained the majority of votes for best transferring attributes in all cases. In the case of gender changes in Table 1, the voting difference between our model and other models was marginal, e.g., 39.1% for StarGAN vs. 31.4% for DIAT. However, in multi-attribute changes, e.g., the 'G+A' case in Table 2, the performance difference becomes significant, e.g., 49.8% for StarGAN vs. 20.3% for IcGAN), clearly showing the advantages of StarGAN in more complicated, multi-attribute transfer tasks. This is because unlike the other methods, StarGAN can handle image translation involving multiple attribute changes by randomly generating a target domain label in the training phase.

定量结果。 表 1 和表 2 分别给出了我们在单属性和多属性迁移任务上的 AMT 实验结果。在所有情形中,StarGAN 都获得了“属性迁移最好”的多数票。在表 1 的性别改变这一项上,我们的模型与其他模型的得票差距较小,例如 StarGAN 为 39.1%,DIAT 为 31.4%。但在多属性改变时,例如表 2 中的 'G+A' 情形,性能差距变得明显,例如 StarGAN 为 49.8%,IcGAN 为 20.3%,清楚地显示出 StarGAN 在更复杂的多属性迁移任务上的优势。这是因为与其他方法不同,StarGAN 通过在训练阶段随机生成目标域标签,就能处理涉及多个属性改变的图像转换。

5.5. Experimental Results on RaFD

5.5. 在 RaFD 上的实验结果

We next train our model on the RaFD dataset to learn the task of synthesizing facial expressions. To compare StarGAN and baseline models, we fix the input domain as the 'neutral' expression, but the target domain varies among the seven remaining expressions.

接下来,我们在 RaFD 数据集上训练模型,学习合成面部表情这一任务。为比较 StarGAN 与基线模型,我们把输入域固定为 'neutral' 表情,目标域则在其余七种表情中变化。

Qualitative evaluation. As seen in Fig. 5, StarGAN clearly generates the most natural-looking expressions while properly maintaining the personal identity and facial features of the input. While DIAT and CycleGAN mostly preserve the identity of the input, many of their results are shown blurry and do not maintain the degree of sharpness as seen in the input. IcGAN even fails to preserve the personal identity in the image by generating male images.

定性评价。 如图 5 所示,StarGAN 明显生成了看起来最自然的表情,同时恰当地保留了输入的个人身份和面部特征。DIAT 和 CycleGAN 虽然大多能保留输入的身份,但许多结果显得模糊,未能保持输入中可见的清晰程度。IcGAN 甚至会生成男性图像,从而无法保留图像中的个人身份。

We believe that the superiority of StarGAN in the image quality is due to its implicit data augmentation effect from a multi-task learning setting. RaFD images contain a relatively small size of samples, e.g., 500 images per domain. When trained on two domains, DIAT and CycleGAN can only use 1,000 training images at a time, but StarGAN can use 4,000 images in total from all the available domains for its training. This allows StarGAN to properly learn how to maintain the quality and sharpness of the generated output.

我们认为,StarGAN 在图像质量上的优势,来自多任务学习设定带来的隐式数据增强效果。RaFD 图像的样本量相对较小,例如每个域约 500 张。在两个域上训练时,DIAT 和 CycleGAN 一次只能使用 1,000 张训练图像,而 StarGAN 可以从所有可用域中一共使用 4,000 张图像来训练。这使 StarGAN 能够更好地学习如何保持生成输出的质量和清晰度。

Quantitative evaluation. For a quantitative evaluation, we compute the classification error of a facial expression on synthesized images. We trained a facial expression classifier on the RaFD dataset (90%/10% splitting for training and test sets) using a ResNet-18 architecture [5], resulting in a near-perfect accuracy of 99.55%. We then trained each of image translation models using the same training set and performed image translation on the same, unseen test set. Finally, we classified the expression of these translated images using the above-mentioned classifier. As can be seen in Table 3, our model achieves the lowest classification error, indicating that our model produces the most realistic facial expressions among all the methods compared.

定量评价。 为进行定量评价,我们计算合成图像上的面部表情分类误差。我们用 ResNet-18 结构 [5] 在 RaFD 数据集上训练了一个面部表情分类器(按 90%/10% 划分训练集和测试集),得到接近完美的 99.55% 准确率。随后,我们用同一训练集分别训练各个图像转换模型,并在同一未见过的测试集上做图像转换。最后,用上述分类器对这些转换后图像的表情分类。如表 3 所示,我们的模型取得了最低的分类误差,表明在所有比较的方法中,它生成的面部表情最为真实。

Method Classification error # of parameters
DIAT 4.10 52.6M ×\times× 7
CycleGAN 5.99 52.6M ×\times× 14
IcGAN 8.07 67.8M ×\times× 1
StarGAN 2.12 53.2M ×\times× 1
Real images 0.45 -

Table 3. Classification errors [%] and the number of parameters on the RaFD dataset.

Method Classification error # of parameters
DIAT 4.10 52.6M ×\times× 7
CycleGAN 5.99 52.6M ×\times× 14
IcGAN 8.07 67.8M ×\times× 1
StarGAN 2.12 53.2M ×\times× 1
Real images 0.45 -

表 3. 在 RaFD 数据集上的分类误差 [%] 与参数量。

Another important advantage of our model is the scalability in terms of the number of parameters required. The last column in Table 3 shows that the number of parameters required to learn all translations by StarGAN is seven times smaller than that of DIAT and fourteen times smaller than that of CycleGAN. This is because StarGAN requires only a single generator and discriminator pair, regardless of the number of domains, while in the case of cross-domain models such as CycleGAN, a completely different model should be trained for each source-target domain pair.

我们模型的另一个重要优点,是所需参数量方面的可扩展性。表 3 的最后一列显示,StarGAN 学习全部转换所需的参数量,比 DIAT 小七倍,比 CycleGAN 小十四倍。这是因为无论域的个数是多少,StarGAN 只需要一对生成器和判别器;而在 CycleGAN 这类跨域模型中,每一对源域–目标域都要训练一套完全不同的模型。

5.6. Experimental Results on CelebA+RaFD

5.6. 在 CelebA+RaFD 上的实验结果

Finally, we empirically demonstrate that our model can learn not only from multiple domains within a single dataset, but also from multiple datasets. We train our model jointly on the CelebA and RaFD datasets using the mask vector (see Section 3.2). To distinguish between the model trained only on RaFD and the model trained on both CelebA and RaFD, we denote the former as StarGAN-SNG (single) and the latter as StarGAN-JNT (joint).

最后,我们用实验表明,我们的模型不仅能从单个数据集内的多个域学习,也能从多个数据集学习。我们使用掩码向量,在 CelebA 和 RaFD 数据集上联合训练模型(见第 3.2 节)。为区分只在 RaFD 上训练的模型与在 CelebA 和 RaFD 上共同训练的模型,我们把前者记为 StarGAN-SNG(single,单数据集),把后者记为 StarGAN-JNT(joint,联合)。

Facial expression synthesis on CelebA by StarGAN-SNG versus StarGAN-JNT

Figure 6. Facial expression synthesis results of StarGAN-SNG and StarGAN-JNT on CelebA dataset.

图 6. StarGAN-SNG 与 StarGAN-JNT 在 CelebA 数据集上的面部表情合成结果。

Effects of joint training. Fig. 6 shows qualitative comparisons between StarGAN-SNG and StarGAN-JNT, where the task is to synthesize facial expressions of images in CelebA. StarGAN-JNT exhibits emotional expressions with high visual quality, while StarGAN-SNG generates reasonable but blurry images with gray backgrounds. This difference is due to the fact that StarGAN-JNT learns to translate CelebA images during training but not StarGAN-SNG. In other words, StarGAN-JNT can leverage both datasets to improve shared low-level tasks such facial keypoint detection and segmentation. By utilizing both CelebA and RaFD, StarGAN-JNT can improve these low-level tasks, which is beneficial to learning facial expression synthesis.

联合训练的效果。 图 6 给出了 StarGAN-SNG 与 StarGAN-JNT 的定性比较,任务是为 CelebA 中的图像合成面部表情。StarGAN-JNT 呈现出视觉质量较高的情绪表情,而 StarGAN-SNG 生成的图像虽然合理,但较为模糊,并且带有灰色背景。这一差异是因为 StarGAN-JNT 在训练中学习了转换 CelebA 图像,而 StarGAN-SNG 没有。换句话说,StarGAN-JNT 可以利用两个数据集来改进共享的底层任务,例如面部关键点检测和分割。通过同时使用 CelebA 和 RaFD,StarGAN-JNT 能够改进这些底层任务,从而有利于学习面部表情合成。

Effect of proper versus wrong mask vectors on StarGAN-JNT expression synthesis

Figure 7. Learned role of the mask vector. All images are generated by StarGAN-JNT. The first row shows the result of applying the proper mask vector, and the last row shows the result of applying the wrong mask vector.

图 7. 所学到的掩码向量作用。所有图像均由 StarGAN-JNT 生成。第一行是使用正确掩码向量的结果,最后一行是使用错误掩码向量的结果。

Learned role of mask vector. In this experiment, we gave a one-hot vector ccc by setting the dimension of a particular facial expression (available from the second dataset, RaFD) to one. In this case, since the label associated with the second data set is explicitly given, the proper mask vector would be [0,1][0, 1][0,1]. Fig. 7 shows the case where this proper mask vector was given and the opposite case where a wrong mask vector of [1,0][1, 0][1,0] was given. When the wrong mask vector was used, StarGAN-JNT fails to synthesize facial expressions, and it manipulates the age of the input image. This is because the model ignores the facial expression label as unknown and treats the facial attribute label as valid by the mask vector. Note that since one of the facial attributes is 'young', the model translates the image from young to old when it takes in a zero vector as input. From this behavior, we can confirm that StarGAN properly learned the intended role of a mask vector in image-to-image translations when involving all the labels from multiple datasets altogether.

掩码向量学到的作用。 在这一实验中,我们给出一个独热向量 ccc,把某一特定面部表情(来自第二个数据集 RaFD)对应的维置为 1。此时,由于与第二个数据集相关的标签被明确给出,正确的掩码向量应为 [0,1][0, 1][0,1]。图 7 展示了给出这一正确掩码向量的情形,以及给出错误掩码向量 [1,0][1, 0][1,0] 的相反情形。使用错误掩码向量时,StarGAN-JNT 无法合成面部表情,转而改动输入图像的年龄。这是因为模型按掩码向量把面部表情标签视为未知而忽略,并把面部属性标签视为有效。注意,由于面部属性之一是 'young',当模型接收零向量作为输入时,会把图像从年轻转换到年长。从这一行为可以确认,在同时涉及多个数据集全部标签的图像到图像转换中,StarGAN 正确地学到了掩码向量应有的作用。

6. Conclusion

6. 结论

In this paper, we proposed StarGAN, a scalable image-to-image translation model among multiple domains using a single generator and a discriminator. Besides the advantages in scalability, StarGAN generated images of higher visual quality compared to existing methods [15, 22, 32], owing to the generalization capability behind the multi-task learning setting. In addition, the use of the proposed simple mask vector enables StarGAN to utilize multiple datasets with different sets of domain labels, thus handling all available labels from them. We hope our work to enable users to develop interesting image translation applications across multiple domains.

在本文中,我们提出了 StarGAN,这是一种可扩展的图像到图像转换模型,用单个生成器和单个判别器在多个域之间进行转换。除了可扩展性上的优点,由于多任务学习设定背后的泛化能力,与现有方法 [15, 22, 32] 相比,StarGAN 生成的图像视觉质量也更高。此外,使用我们提出的简单掩码向量,StarGAN 能够利用带有不同域标签集合的多个数据集,从而处理其中所有可用的标签。我们希望这项工作能够帮助使用者开发跨多个域的有趣图像转换应用。

Acknowledgements. This work was mainly done while the first author did a research internship at Clova AI Research, NAVER. We thank all the researchers at NAVER, especially Donghyun Kwak, for insightful discussions. This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (No. NRF2016R1C1B2015924). Jaegul Choo is the corresponding author.

致谢。 本工作主要是在第一作者于 NAVER 的 Clova AI Research 进行研究实习期间完成的。我们感谢 NAVER 的所有研究者,尤其是 Donghyun Kwak,与他们进行了富有启发的讨论。本工作得到了韩国政府(MSIP)资助的韩国国家研究基金会(NRF)项目(编号 NRF2016R1C1B2015924)的部分支持。Jaegul Choo 是通讯作者。

References

参考文献

[1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 214–223, 2017. 2, 5

[2] A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.07093, 2016. 3

[3] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), pages 2672–2680, 2014. 2

[4] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of wasserstein gans. arXiv preprint arXiv:1704.00028, 2017. 5, 6

[5] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 5, 7

[6] X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, and S. Belongie. Stacked generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. 2

[7] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 1, 2, 3, 5

[8] T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim. Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1857–1865, 2017. 1, 2, 3, 4

[9] T. Kim, B. Kim, M. Cha, and J. Kim. Unsupervised visual attribute transfer with reconfigurable generative adversarial networks. arXiv preprint arXiv:1707.09798, 2017. 2

[10] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 6

[11] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), 2014. 3

[12] O. Langner, R. Dotsch, G. Bijlstra, D. H. Wigboldus, S. T. Hawk, and A. Van Knippenberg. Presentation and validation of the radboud faces database. Cognition and Emotion, 24(8):1377–1388, 2010. 2, 4, 6

[13] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2, 3

[14] C. Li and M. Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. In Proceedings of the 14th European Conference on Computer Vision (ECCV), pages 702–716, 2016. 5

[15] M. Li, W. Zuo, and D. Zhang. Deep identity-aware transfer of facial attributes. arXiv preprint arXiv:1610.05586, 2016. 2, 5, 8

[16] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. arXiv preprint arXiv:1703.00848, 2017. 3

[17] M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems (NIPS), pages 469–477, 2016. 3

[18] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015. 2, 4, 6

[19] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014. 3

[20] A. Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01583, 2016. 3

[21] A. Odena, C. Olah, and J. Shlens. Conditional image synthesis with auxiliary classifier gans. arXiv preprint arXiv:1610.09585, 2016. 3, 5

[22] G. Perarnau, J. van de Weijer, B. Raducanu, and J. M. Álvarez. Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355, 2016. 5, 8

[23] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 2

[24] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. arXiv preprint arXiv:1605.05396, 2016. 3

[25] W. Shen and R. Liu. Learning residual images for face attribute manipulation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2

[26] Z. Shu, E. Yumer, S. Hadap, K. Sunkavalli, E. Shechtman, and D. Samaras. Neural face editing with intrinsic image disentangling. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 3

[27] Y. Taigman, A. Polyak, and L. Wolf. Unsupervised cross-domain image generation. In 5th International Conference on Learning Representations (ICLR), 2017. 3

[28] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016. 5

[29] H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. arXiv preprint arXiv:1612.03242, 2016. 3

[30] Z. Zhang, Y. Song, and H. Qi. Age progression/regression by conditional adversarial autoencoder. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. 2

[31] J. Zhao, M. Mathieu, and Y. LeCun. Energy-based generative adversarial network. In 5th International Conference on Learning Representations (ICLR), 2017. 2

[32] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 1, 2, 3, 4, 5, 8