Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表

暂无章节

返回论文列表
目录

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

收录于 ICCV 2017 会议论文;2017 IEEE International Conference on Computer Vision;DOI 10.1109/iccv.2017.167被引 6,384 次
打开论文来源

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Xun Huang Serge Belongie

Department of Computer Science & Cornell Tech, Cornell University

{xh258,sjb344}@cornell.edu

Abstract

摘要

Gatys et al. recently introduced a neural algorithm that renders a content image in the style of another image, achieving so-called style transfer. However, their framework requires a slow iterative optimization process, which limits its practical application. Fast approximations with feed-forward neural networks have been proposed to speed up neural style transfer. Unfortunately, the speed improvement comes at a cost: the network is usually tied to a fixed set of styles and cannot adapt to arbitrary new styles. In this paper, we present a simple yet effective approach that for the first time enables arbitrary style transfer in real-time. At the heart of our method is a novel adaptive instance normalization (AdaIN) layer that aligns the mean and variance of the content features with those of the style features. Our method achieves speed comparable to the fastest existing approach, without the restriction to a pre-defined set of styles. In addition, our approach allows flexible user controls such as content-style trade-off, style interpolation, color & spatial controls, all using a single feed-forward neural network.

Gatys 等人最近提出一种神经算法,能够把一张内容图渲染成另一张图的风格,也就是所谓的风格迁移。不过,他们的框架需要缓慢的迭代优化,实际应用因此受到限制。为了加快神经风格迁移,已有工作用前馈神经网络做快速近似。可惜速度提升是有代价的:网络通常会绑死在一组固定风格上,无法适应任意的新风格。本文提出一种简单而有效的方法,第一次实现了任意风格的实时迁移。方法的核心是一种新的自适应实例归一化(AdaIN)层,它把内容特征的均值和方差对齐到风格特征。我们的方法在速度上可与现有最快的途径相比,同时又不受预先给定风格集合的限制。此外,同一套前馈网络还支持灵活的用户控制,包括内容–风格权衡、风格插值,以及颜色与空间控制。

  1. Introduction

The seminal work of Gatys et al. [16] showed that deep neural networks (DNNs) encode not only the content but also the style information of an image. Moreover, the image style and content are somewhat separable: it is possible to change the style of an image while preserving its content. The style transfer method of [16] is flexible enough to combine content and style of arbitrary images. However, it relies on an optimization process that is prohibitively slow.

Gatys 等人 [16] 的开创性工作表明,深度神经网络(DNN)不仅编码图像的内容,也编码风格信息。而且风格和内容在一定程度上是可分离的:可以在保住内容的同时改变风格。[16] 的风格迁移方法足够灵活,能够组合任意图像的内容与风格。不过,它所依赖的优化过程慢到难以实用。

Significant effort has been devoted to accelerating neural style transfer. [24, 51, 31] attempted to train feed-forward neural networks that perform stylization with a single forward pass. A major limitation of most feed-forward methods is that each network is restricted to a single style. There are some recent works addressing this problem, but they are either still limited to a finite set of styles [11, 32, 55, 5], or much slower than the single-style transfer methods [6].

已有大量工作致力于加快神经风格迁移。[24, 51, 31] 试图训练只需一次前向传播就能完成风格化的前馈神经网络。大多数前馈方法的一个主要限制是,每个网络只能对应一种风格。最近也有工作试图解决这个问题,但它们要么仍局限于有限的一组风格 [11, 32, 55, 5],要么比单风格迁移方法慢得多 [6]。

In this work, we present the first neural style transfer algorithm that resolves this fundamental flexibility-speed dilemma. Our approach can transfer arbitrary new styles in real-time, combining the flexibility of the optimization-based framework [16] and the speed similar to the fastest feed-forward approaches [24, 52]. Our method is inspired by the instance normalization (IN) [52, 11] layer, which is surprisingly effective in feed-forward style transfer. To explain the success of instance normalization, we propose a new interpretation that instance normalization performs style normalization by normalizing feature statistics, which have been found to carry the style information of an image [16, 30, 33]. Motivated by our interpretation, we introduce a simple extension to IN, namely adaptive instance normalization (AdaIN). Given a content input and a style input, AdaIN simply adjusts the mean and variance of the content input to match those of the style input. Through experiments, we find AdaIN effectively combines the content of the former and the style latter by transferring feature statistics. A decoder network is then learned to generate the final stylized image by inverting the AdaIN output back to the image space. Our method is nearly three orders of magnitude faster than [16], without sacrificing the flexibility of transferring inputs to arbitrary new styles. Furthermore, our approach provides abundant user controls at runtime, without any modification to the training process.

本文提出第一种能够解开这一灵活性与速度根本两难的神经风格迁移算法。我们的方法可以实时迁移任意新风格,既保有基于优化的框架 [16] 的灵活性,速度又接近最快的前馈途径 [24, 52]。方法受到实例归一化(IN)[52, 11] 的启发,这一层在前馈风格迁移中出奇地有效。为解释 IN 为何成功,我们提出一种新的理解:实例归一化是通过归一化特征统计量来做风格归一化,而这些统计量已被发现携带着图像的风格信息 [16, 30, 33]。基于这一理解,我们为 IN 引入一个简单的扩展,即自适应实例归一化(AdaIN)。给定内容输入和风格输入,AdaIN 只需调整内容输入的均值和方差,使之与风格输入相一致。实验表明,通过迁移特征统计量,AdaIN 能够有效结合前者的内容和后者的风格。随后再学习一个解码器,把 AdaIN 的输出倒回图像空间,得到最终的风格化图像。与 [16] 相比,我们的方法快了近三个数量级,同时又没有牺牲把输入转到任意新风格的灵活性。此外,该方法在运行时还提供丰富的用户控制,且无需改动训练过程。

  1. Related Work

Style transfer. The problem of style transfer has its origin from non-photo-realistic rendering [28], and is closely related to texture synthesis and transfer [13, 12, 14]. Some early approaches include histogram matching on linear filter responses [19] and non-parametric sampling [12, 15]. These methods typically rely on low-level statistics and often fail to capture semantic structures. Gatys et al. [16] for the first time demonstrated impressive style transfer results by matching feature statistics in convolutional layers of a DNN. Recently, several improvements to [16] have been proposed. Li and Wand [30] introduced a framework based on markov random field (MRF) in the deep feature space to enforce local patterns. Gatys et al. [17] proposed ways to control the color preservation, the spatial location, and the scale of style transfer. Ruder et al. [45] improved the quality of video style transfer by imposing temporal constraints.

风格迁移。 风格迁移问题起源于非真实感绘制 [28],并与纹理合成与迁移密切相关 [13, 12, 14]。早期途径包括在线性滤波器响应上做直方图匹配 [19],以及非参数采样 [12, 15]。这些方法通常依赖低层统计量,往往难以抓住语义结构。Gatys 等人 [16] 第一次通过匹配 DNN 卷积层中的特征统计量,展示了令人印象深刻的风格迁移结果。近来已有若干工作对 [16] 加以改进。Li 和 Wand [30] 在深层特征空间中引入基于马尔可夫随机场(MRF)的框架,以强化局部模式。Gatys 等人 [17] 提出了控制颜色保持、空间位置和风格迁移尺度的方法。Ruder 等人 [45] 通过加入时间约束,提升了视频风格迁移的质量。

The framework of Gatys et al. [16] is based on a slow optimization process that iteratively updates the image to minimize a content loss and a style loss computed by a loss network. It can take minutes to converge even with modern GPUs. On-device processing in mobile applications is therefore too slow to be practical. A common workaround is to replace the optimization process with a feed-forward neural network that is trained to minimize the same objective [24, 51, 31]. These feed-forward style transfer approaches are about three orders of magnitude faster than the optimization-based alternative, opening the door to real-time applications. Wang et al. [53] enhanced the granularity of feed-forward style transfer with a multi-resolution architecture. Ulyanov et al. [52] proposed ways to improve the quality and diversity of the generated samples. However, the above feed-forward methods are limited in the sense that each network is tied to a fixed style. To address this problem, Dumoulin et al. [11] introduced a single network that is able to encode 32 styles and their interpolations. Concurrent to our work, Li et al. [32] proposed a feed-forward architecture that can synthesize up to 300 textures and transfer 16 styles. Still, the two methods above cannot adapt to arbitrary styles that are not observed during training.

Gatys 等人 [16] 的框架建立在缓慢的优化过程上:迭代更新图像,以最小化由损失网络计算出的内容损失和风格损失。即便使用现代 GPU,收敛也可能需要数分钟。因此,在移动应用里做端上处理就慢得不实用。一个常见的变通办法,是用一个训练来最小化同一目标的前馈神经网络来替代优化过程 [24, 51, 31]。这些前馈风格迁移途径比基于优化的方法大约快三个数量级,从而为实时应用打开了大门。Wang 等人 [53] 用多分辨率结构提升了前馈风格迁移的细粒度。Ulyanov 等人 [52] 提出了改进生成样本质量与多样性的方法。不过,上述前馈方法的局限在于,每个网络都绑死在一种固定风格上。为解决这一问题,Dumoulin 等人 [11] 引入了一个能够编码 32 种风格及其插值的单一网络。与本文同期,Li 等人 [32] 提出一种前馈结构,最多可合成 300 种纹理、迁移 16 种风格。尽管如此,这两种方法仍无法适应训练时未曾见过的任意风格。

Very recently, Chen and Schmidt [6] introduced a feed-forward method that can transfer arbitrary styles thanks to a style swap layer. Given feature activations of the content and style images, the style swap layer replaces the content features with the closest-matching style features in a patch-by-patch manner. Nevertheless, their style swap layer creates a new computational bottleneck: more than 95% of the computation is spent on the style swap for 512 × 512 input images. Our approach also permits arbitrary style transfer, while being 1-2 orders of magnitude faster than [6].

就在最近,Chen 和 Schmidt [6] 引入一种借助风格交换层、可以迁移任意风格的前馈方法。给定内容图和风格图的特征激活,风格交换层会按补丁把内容特征替换成最相近的风格特征。不过,这一层又造成了新的计算瓶颈:对 512 × 512 的输入图像,超过 95% 的计算花在风格交换上。我们的方法同样可以进行任意风格迁移,同时比 [6] 快一到两个数量级。

Another central problem in style transfer is which style loss function to use. The original framework of Gatys et al. [16] matches styles by matching the second-order statistics between feature activations, captured by the Gram matrix. Other effective loss functions have been proposed, such as MRF loss [30], adversarial loss [31], histogram loss [54], CORAL loss [41], MMD loss [33], and distance between channel-wise mean and variance [33]. Note that all the above loss functions aim to match some feature statistics between the style image and the synthesized image.

风格迁移里另一个核心问题是使用哪一种风格损失函数。Gatys 等人 [16] 的原始框架通过匹配特征激活的二阶统计量——由 Gram 矩阵刻画——来对齐风格。此后还提出了其他有效的损失函数,例如 MRF 损失 [30]、对抗损失 [31]、直方图损失 [54]、CORAL 损失 [41]、MMD 损失 [33],以及通道均值与方差之间的距离 [33]。需要注意的是,上述损失函数都旨在匹配风格图与合成图之间的某些特征统计量。

Deep generative image modeling. There are several alternative frameworks for image generation, including variational auto-encoders [27], auto-regressive models [40], and generative adversarial networks (GANs) [18]. Remarkably, GANs have achieved the most impressive visual quality. Various improvements to the GAN framework have been proposed, such as conditional generation [43, 23], multi-stage processing [9, 20], and better training objectives [46, 1]. GANs have also been applied to style transfer [31] and cross-domain image generation [50, 3, 23, 38, 37, 25].

深度生成式图像建模。 图像生成还有若干替代框架,包括变分自编码器 [27]、自回归模型 [40],以及生成对抗网络(GAN)[18]。其中 GAN 在视觉质量上最为突出。针对 GAN 框架也已有多种改进,例如条件生成 [43, 23]、多阶段处理 [9, 20],以及更好的训练目标 [46, 1]。GAN 还被用于风格迁移 [31] 和跨域图像生成 [50, 3, 23, 38, 37, 25]。

  1. Background

3.1. Batch Normalization

The seminal work of Ioffe and Szegedy [22] introduced a batch normalization (BN) layer that significantly ease the training of feed-forward networks by normalizing feature statistics. BN layers are originally designed to accelerate training of discriminative networks, but have also been found effective in generative image modeling [42]. Given an input batch x∈RN×C×H×Wx \in \mathbb{R}^{N \times C \times H \times W}x∈RN×C×H×W, BN normalizes the mean and standard deviation for each individual feature channel:

Ioffe 和 Szegedy [22] 的开创性工作引入了批归一化(BN)层,通过归一化特征统计量,显著减轻了前馈网络的训练负担。BN 层最初是为加速判别网络的训练而设计的,后来也被发现对生成式图像建模有效 [42]。给定输入 batch x∈RN×C×H×Wx \in \mathbb{R}^{N \times C \times H \times W}x∈RN×C×H×W,BN 对每一个特征通道分别归一化均值和标准差:

BN(x)=γ(x−μ(x)σ(x))+β(1)\mathrm{BN}(x)=\gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta \tag{1}BN(x)=γ(σ(x)x−μ(x)​)+β(1)

where γ,β∈RC\gamma,\beta\in\mathbb{R}^{C}γ,β∈RC are affine parameters learned from data; μ(x),σ(x)∈RC\mu(x),\sigma(x)\in\mathbb{R}^{C}μ(x),σ(x)∈RC are the mean and standard deviation, computed across batch size and spatial dimensions independently for each feature channel:

其中 γ,β∈RC\gamma,\beta\in\mathbb{R}^{C}γ,β∈RC 是从数据中学到的仿射参数;μ(x),σ(x)∈RC\mu(x),\sigma(x)\in\mathbb{R}^{C}μ(x),σ(x)∈RC 是均值和标准差,在 batch 维和空间维上对每个特征通道独立计算:

μc(x)=1NHW∑n=1N∑w=1W∑h=1Hxnchw(2)\mu_{c}(x)=\frac{1}{NHW}\sum_{n=1}^{N}\sum_{w=1}^{W}\sum_{h=1}^{H} x_{nchw} \tag{2}μc​(x)=NHW1​n=1∑N​w=1∑W​h=1∑H​xnchw​(2)

σc(x)=1NHW∑n=1N∑w=1W∑h=1H(xnchw−μc(x))2+ϵ(3)\sigma_{c}(x)=\sqrt{\frac{1}{NHW}\sum_{n=1}^{N}\sum_{w=1}^{W}\sum_{h=1}^{H}\left(x_{nchw}-\mu_{c}(x)\right)^{2}+\epsilon} \tag{3}σc​(x)=NHW1​n=1∑N​w=1∑W​h=1∑H​(xnchw​−μc​(x))2+ϵ​(3)

BN uses mini-batch statistics during training and replace them with popular statistics during inference, introducing discrepancy between training and inference. Batch renormalization [21] was recently proposed to address this issue by gradually using popular statistics during training. As another interesting application of BN, Li et al. [34] found that BN can alleviate domain shifts by recomputing popular statistics in the target domain. Recently, several alternative normalization schemes have been proposed to extend BN's effectiveness to recurrent architectures [35, 2, 47, 8, 29, 44].

BN 在训练时使用小批量统计量,推理时则换成总体统计量,从而在训练和推理之间引入了不一致。最近提出的批重新归一化 [21] 试图在训练中逐步改用总体统计量,以缓解这一问题。作为 BN 的另一项有趣应用,Li 等人 [34] 发现,在目标域中重新计算总体统计量,BN 可以减轻域偏移。近来还提出了若干替代归一化方案,以把 BN 的效果扩展到循环结构 [35, 2, 47, 8, 29, 44]。

3.2. Instance Normalization

In the original feed-forward stylization method [51], the style transfer network contains a BN layer after each convolutional layer. Surprisingly, Ulyanov et al. [52] found that significant improvement could be achieved simply by replacing BN layers with IN layers:

在最初的前馈风格化方法 [51] 中,风格迁移网络在每个卷积层之后都有一个 BN 层。令人惊讶的是,Ulyanov 等人 [52] 发现,只需把 BN 层换成 IN 层,就能获得显著改进:

IN(x)=γ(x−μ(x)σ(x))+β(4)\mathrm{IN}(x)=\gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta \tag{4}IN(x)=γ(σ(x)x−μ(x)​)+β(4)

Different from BN layers, here μ(x)\mu(x)μ(x) and σ(x)\sigma(x)σ(x) are computed across spatial dimensions independently for each channel and each sample:

与 BN 层不同,这里的 μ(x)\mu(x)μ(x) 和 σ(x)\sigma(x)σ(x) 是在空间维上对每个通道、每个样本独立计算的:

μnc(x)=1HW∑w=1W∑h=1Hxnchw(5)\mu_{nc}(x)=\frac{1}{HW}\sum_{w=1}^{W}\sum_{h=1}^{H} x_{nchw} \tag{5}μnc​(x)=HW1​w=1∑W​h=1∑H​xnchw​(5)

σnc(x)=1HW∑w=1W∑h=1H(xnchw−μnc(x))2+ϵ(6)\sigma_{nc}(x)=\sqrt{\frac{1}{HW}\sum_{w=1}^{W}\sum_{h=1}^{H}\left(x_{nchw}-\mu_{nc}(x)\right)^{2}+\epsilon} \tag{6}σnc​(x)=HW1​w=1∑W​h=1∑H​(xnchw​−μnc​(x))2+ϵ​(6)

Instance normalization versus batch normalization training curves under different contrast conditions

Figure 1. To understand the reason for IN's effectiveness in style transfer, we train an IN model and a BN model with (a) original images in MS-COCO [36], (b) contrast normalized images, and (c) style normalized images using a pre-trained style transfer network [24]. The improvement brought by IN remains significant even when all training images are normalized to the same contrast, but are much smaller when all images are (approximately) normalized to the same style. Our results suggest that IN performs a kind of style normalization.

图 1:为理解 IN 在风格迁移中为何有效,我们分别用 IN 模型和 BN 模型,在 (a) MS-COCO [36] 的原始图像、(b) 对比度归一化图像,以及 (c) 用预训练风格迁移网络 [24] 得到的风格归一化图像上训练。即使把所有训练图像都归一化到相同对比度,IN 带来的改进仍然显著;而当所有图像都被(近似)归一化到同一风格时,这一改进会小得多。结果说明,IN 进行的是一种风格归一化。

Another difference is that IN layers are applied at test time unchanged, whereas BN layers usually replace mini-batch statistics with population statistics.

另一个不同之处是:IN 层在测试时原样使用,而 BN 层通常会把小批量统计量换成总体统计量。

3.3. Conditional Instance Normalization

Instead of learning a single set of affine parameters γ\gammaγ and β\betaβ, Dumoulin et al. [11] proposed a conditional instance normalization (CIN) layer that learns a different set of parameters γs\gamma^{s}γs and βs\beta^{s}βs for each style sss:

Dumoulin 等人 [11] 没有只学习一组仿射参数 γ\gammaγ 和 β\betaβ,而是提出条件实例归一化(CIN)层,为每一种风格 sss 学习不同的参数 γs\gamma^{s}γs 和 βs\beta^{s}βs:

CIN(x;s)=γs(x−μ(x)σ(x))+βs(7)\mathrm{CIN}(x;s)=\gamma^{s}\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta^{s} \tag{7}CIN(x;s)=γs(σ(x)x−μ(x)​)+βs(7)

During training, a style image together with its index sss are randomly chosen from a fixed set of styles s∈{1,2,…,S}s\in\{1,2,\ldots,S\}s∈{1,2,…,S} (S=32S=32S=32 in their experiments). The content image is then processed by a style transfer network in which the corresponding γs\gamma^{s}γs and βs\beta^{s}βs are used in the CIN layers. Surprisingly, the network can generate images in completely different styles by using the same convolutional parameters but different affine parameters in IN layers.

训练时,从固定的风格集合 s∈{1,2,…,S}s\in\{1,2,\ldots,S\}s∈{1,2,…,S} 中随机选取一张风格图及其索引 sss(他们的实验中 S=32S=32S=32)。内容图随后由风格迁移网络处理,其中 CIN 层使用对应的 γs\gamma^{s}γs 和 βs\beta^{s}βs。令人惊讶的是,只需使用相同的卷积参数、在 IN 层换用不同的仿射参数,网络就能生成完全不同风格的图像。

Compared with a network without normalization layers, a network with CIN layers requires 2FS2FS2FS additional parameters, where FFF is the total number of feature maps in the network [11]. Since the number of additional parameters scales linearly with the number of styles, it is challenging to extend their method to model a large number of styles (e.g., tens of thousands). Also, their approach cannot adapt to arbitrary new styles without re-training the network.

与没有归一化层的网络相比,带 CIN 层的网络需要额外的 2FS2FS2FS 个参数,其中 FFF 是网络中特征图的总数 [11]。由于额外参数量随风格数目线性增长,要把他们的方法扩展到大量风格(例如数万种)就很困难。而且,若不重新训练网络,该方法也无法适应任意的新风格。

  1. Interpreting Instance Normalization

Despite the great success of (conditional) instance normalization, the reason why they work particularly well for style transfer remains elusive. Ulyanov et al. [52] attribute the success of IN to its invariance to the contrast of the content image. However, IN takes place in the feature space, therefore it should have more profound impacts than a simple contrast normalization in the pixel space. Perhaps even more surprising is the fact that the affine parameters in IN can completely change the style of the output image.

尽管(条件)实例归一化取得了很大成功,它们为何对风格迁移特别有效,原因仍不清楚。Ulyanov 等人 [52] 把 IN 的成功归因于它对内容图对比度的不变性。然而 IN 发生在特征空间,因此其影响应当比像素空间里一次简单的对比度归一化更为深刻。或许更令人惊讶的是,IN 中的仿射参数可以彻底改变输出图像的风格。

It has been known that the convolutional feature statistics of a DNN can capture the style of an image [16, 30, 33]. While Gatys et al. [16] use the second-order statistics as their optimization objective, Li et al. [33] recently showed that matching many other statistics, including channel-wise mean and variance, are also effective for style transfer. Motivated by these observations, we argue that instance normalization performs a form of style normalization by normalizing feature statistics, namely the mean and variance. Although DNN serves as a image descriptor in [16, 33], we believe that the feature statistics of a generator network can also control the style of the generated image.

已经知道,DNN 的卷积特征统计量能够刻画图像的风格 [16, 30, 33]。Gatys 等人 [16] 把二阶统计量当作优化目标,而 Li 等人 [33] 最近表明,匹配许多其他统计量——包括通道均值和方差——对风格迁移同样有效。受这些观察启发,我们认为实例归一化是通过归一化特征统计量,也就是均值和方差,来进行一种风格归一化。虽然在 [16, 33] 中 DNN 是作为图像描述器来使用的,但我们相信,生成网络的特征统计量同样可以控制生成图像的风格。

We run the code of improved texture networks [52] to perform single-style transfer, with IN or BN layers. As expected, the model with IN converges faster than the BN model (Fig. 1 (a)). To test the explanation in [52], we then normalize all the training images to the same contrast by performing histogram equalization on the luminance channel. As shown in Fig. 1 (b), IN remains effective, suggesting the explanation in [52] to be incomplete. To verify our hypothesis, we normalize all the training images to the same style (different from the target style) using a pre-trained style transfer network provided by [24]. According to Fig. 1 (c), the improvement brought by IN become much smaller when images are already style normalized. The remaining gap can explained by the fact that the style normalization with [24] is not perfect. Also, models with BN trained on style normalized images can converge as fast as models with IN trained on the original images. Our results indicate that IN does perform a kind of style normalization.

我们运行改进版纹理网络 [52] 的代码来做单风格迁移,分别使用 IN 或 BN 层。正如预期,带 IN 的模型比 BN 模型收敛更快(图 1 (a))。为检验 [52] 中的解释,我们接着在亮度通道上做直方图均衡,把所有训练图像归一化到相同对比度。如图 1 (b) 所示,IN 仍然有效,说明 [52] 的解释并不完整。为验证我们的假设,我们使用 [24] 提供的预训练风格迁移网络,把所有训练图像归一化到同一种(不同于目标的)风格。根据图 1 (c),当图像已经做过风格归一化时,IN 带来的改进会小得多。剩下的差距可以用 [24] 的风格归一化并不完美来解释。此外,在风格归一化图像上训练的 BN 模型,可以收敛得和在原始图像上训练的 IN 模型一样快。这些结果表明,IN 的确在进行一种风格归一化。

Since BN normalizes the feature statistics of a batch of samples instead of a single sample, it can be intuitively understood as normalizing a batch of samples to be centered around a single style. Each single sample, however, may still have different styles. This is undesirable when we want to transfer all images to the same style, as is the case in the original feed-forward style transfer algorithm [51]. Although the convolutional layers might learn to compensate the intra-batch style difference, it poses additional challenges for training. On the other hand, IN can normalize the style of each individual sample to the target style. Training is facilitated because the rest of the network can focus on content manipulation while discarding the original style information. The reason behind the success of CIN also becomes clear: different affine parameters can normalize the feature statistics to different values, thereby normalizing the output image to different styles.

由于 BN 归一化的是一批样本而不是单个样本的特征统计量,可以直观地把它理解成:把一批样本归一到围绕同一种风格的中心。但每一个样本仍可能带有不同的风格。当我们希望把所有图像都转到同一种风格时——原始前馈风格迁移算法 [51] 正是如此——这并不可取。卷积层也许能学会补偿 batch 内部的风格差异,但这会给训练带来额外困难。另一方面,IN 可以把每一个样本的风格都归一到目标风格。训练因此更容易,因为网络的其余部分可以专注于内容操作,同时丢掉原来的风格信息。CIN 之所以成功,原因也变得清楚:不同的仿射参数可以把特征统计量归一到不同的值,从而把输出图像归一到不同风格。

  1. Adaptive Instance Normalization

If IN normalizes the input to a single style specified by the affine parameters, is it possible to adapt it to arbitrarily given styles by using adaptive affine transformations? Here, we propose a simple extension to IN, which we call adaptive instance normalization (AdaIN). AdaIN receives a content input xxx and a style input yyy, and simply aligns the channel-wise mean and variance of xxx to match those of yyy. Unlike BN, IN or CIN, AdaIN has no learnable affine parameters. Instead, it adaptively computes the affine parameters from the style input:

如果 IN 是把输入归一到由仿射参数指定的单一风格,那么能否用自适应的仿射变换,使它适应任意给定的风格?这里我们为 IN 提出一个简单的扩展,称为自适应实例归一化(AdaIN)。AdaIN 接收内容输入 xxx 和风格输入 yyy,只需把 xxx 的通道均值和方差对齐到 yyy。与 BN、IN 或 CIN 不同,AdaIN 没有可学习的仿射参数,而是从风格输入中自适应地计算这些参数:

AdaIN(x,y)=σ(y)(x−μ(x)σ(x))+μ(y)(8)\mathrm{AdaIN}(x,y)=\sigma(y)\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\mu(y) \tag{8}AdaIN(x,y)=σ(y)(σ(x)x−μ(x)​)+μ(y)(8)

in which we simply scale the normalized content input with σ(y)\sigma(y)σ(y), and shift it with μ(y)\mu(y)μ(y). Similar to IN, these statistics are computed across spatial locations.

也就是用 σ(y)\sigma(y)σ(y) 缩放已归一化的内容输入,再用 μ(y)\mu(y)μ(y) 做平移。与 IN 类似,这些统计量是在空间位置上计算的。

Intuitively, let us consider a feature channel that detects brushstrokes of a certain style. A style image with this kind of strokes will produce a high average activation for this feature. The output produced by AdaIN will have the same high average activation for this feature, while preserving the spatial structure of the content image. The brushstroke feature can be inverted to the image space with a feed-forward decoder, similar to [10]. The variance of this feature channel can encoder more subtle style information, which is also transferred to the AdaIN output and the final output image.

直观地看,考虑一个检测某种风格笔触的特征通道。带有这类笔触的风格图会在该特征上产生较高的平均激活。AdaIN 的输出在这一特征上也会有同样高的平均激活,同时保住内容图的空间结构。这一笔触特征可以像 [10] 那样,用前馈解码器倒回图像空间。该特征通道的方差还能编码更细微的风格信息,这些信息同样会被传到 AdaIN 的输出以及最终的输出图像。

In short, AdaIN performs style transfer in the feature space by transferring feature statistics, specifically the channel-wise mean and variance. Our AdaIN layer plays a similar role as the style swap layer proposed in [6]. While the style swap operation is very time-consuming and memory-consuming, our AdaIN layer is as simple as an IN layer, adding almost no computational cost.

简言之,AdaIN 通过迁移特征统计量,特别是通道均值和方差,在特征空间中完成风格迁移。我们的 AdaIN 层与 [6] 提出的风格交换层作用相似。风格交换操作非常耗时也非常占内存,而 AdaIN 层和 IN 层一样简单,几乎不增加计算代价。

Overview of the real-time AdaIN style transfer network

Figure 2. An overview of our style transfer algorithm. We use the first few layers of a fixed VGG-19 network to encode the content and style images. An AdaIN layer is used to perform style transfer in the feature space. A decoder is learned to invert the AdaIN output to the image spaces. We use the same VGG encoder to compute a content loss LcL_cLc​ (Equ. 12) and a style loss LsL_sLs​ (Equ. 13).

图 2:我们风格迁移算法的概览。我们使用固定 VGG-19 网络的前若干层来编码内容图和风格图,再用 AdaIN 层在特征空间中完成风格迁移,并学习一个解码器,把 AdaIN 的输出倒回图像空间。我们使用同一个 VGG 编码器计算内容损失 LcL_cLc​(公式 12)和风格损失 LsL_sLs​(公式 13)。

  1. Experimental Setup

Fig. 2 shows an overview of our style transfer network based on the proposed AdaIN layer. Code and pre-trained models (in Torch 7 [7]) are available at: https://github.com/xunhuang1995/AdaIN-style

图 2 给出了基于所提出 AdaIN 层的风格迁移网络概览。代码和预训练模型(基于 Torch 7 [7])见:https://github.com/xunhuang1995/AdaIN-style

6.1. Architecture

Our style transfer network TTT takes a content image ccc and an arbitrary style image sss as inputs, and synthesizes an output image that recombines the content of the former and the style latter. We adopt a simple encoder-decoder architecture, in which the encoder fff is fixed to the first few layers (up to relu4_1) of a pre-trained VGG-19 [48]. After encoding the content and style images in feature space, we feed both feature maps to an AdaIN layer that aligns the mean and variance of the content feature maps to those of the style feature maps, producing the target feature maps ttt:

我们的风格迁移网络 TTT 以内容图 ccc 和任意风格图 sss 为输入,合成一幅重新组合前者内容与后者风格的输出图像。我们采用简单的编码器–解码器结构,其中编码器 fff 固定为预训练 VGG-19 [48] 的前若干层(到 relu4_1 为止)。在特征空间中编码内容图和风格图之后,把两路特征图送入 AdaIN 层,把内容特征图的均值和方差对齐到风格特征图,得到目标特征图 ttt:

t=AdaIN(f(c),f(s))(9)t=\mathrm{AdaIN}(f(c),f(s)) \tag{9}t=AdaIN(f(c),f(s))(9)

A randomly initialized decoder ggg is trained to map ttt back to the image space, generating the stylized image T(c,s)T(c,s)T(c,s):

再训练一个随机初始化的解码器 ggg,把 ttt 映射回图像空间,得到风格化图像 T(c,s)T(c,s)T(c,s):

T(c,s)=g(t)(10)T(c,s)=g(t) \tag{10}T(c,s)=g(t)(10)

The decoder mostly mirrors the encoder, with all pooling layers replaced by nearest up-sampling to reduce checkerboard effects. We use reflection padding in both fff and ggg to avoid border artifacts. Another important architectural choice is whether the decoder should use instance, batch, or no normalization layers. As discussed in Sec. 4, IN normalizes each sample to a single style while BN normalizes a batch of samples to be centered around a single style. Both are undesirable when we want the decoder to generate images in vastly different styles. Thus, we do not use normalization layers in the decoder. In Sec. 7.1 we will show that IN/BN layers in the decoder indeed hurt performance.

解码器大体上与编码器镜像对称,所有池化层都改成最近邻上采样,以减轻棋盘效应。我们在 fff 和 ggg 中都使用反射填充,以避免边界伪影。另一个重要的结构选择是:解码器应该使用实例归一化、批归一化,还是不使用归一化层。如第 4 节所述,IN 会把每个样本归一到单一风格,而 BN 会把一批样本归一到围绕单一风格的中心。当我们希望解码器生成风格迥异的图像时,两者都不可取。因此,我们在解码器中不使用归一化层。第 7.1 节将会表明,解码器中的 IN/BN 层的确会损害性能。

6.2. Training

We train our network using MS-COCO [36] as content images and a dataset of paintings mostly collected from WikiArt [39] as style images, following the setting of [6]. Each dataset contains roughly 80,000 training examples. We use the adam optimizer [26] and a batch size of 8 content-style image pairs. During training, we first resize the smallest dimension of both images to 512 while preserving the aspect ratio, then randomly crop regions of size 256×256256 \times 256256×256. Since our network is fully convolutional, it can be applied to images of any size during testing.

我们按照 [6] 的设定训练网络:用 MS-COCO [36] 作为内容图,用主要收集自 WikiArt [39] 的绘画数据集作为风格图。两个数据集各约有 80,000 个训练样本。我们使用 Adam 优化器 [26],batch 大小为 8 对内容–风格图像。训练时先在保持宽高比的前提下把两张图的短边缩放到 512,再随机裁出 256×256256 \times 256256×256 的区域。由于网络是全卷积的,测试时可以用于任意尺寸的图像。

Similar to [51, 11, 52], we use the pre-trained VGG-19 [48] to compute the loss function to train the decoder:

与 [51, 11, 52] 类似,我们使用预训练的 VGG-19 [48] 来计算训练解码器的损失函数:

L=Lc+λLs(11)L=L_c+\lambda L_s \tag{11}L=Lc​+λLs​(11)

which is a weighted combination of the content loss LcL_cLc​ and the style loss LsL_sLs​ with the style loss weight λ\lambdaλ. The content loss is the Euclidean distance between the target features and the features of the output image. We use the AdaIN output ttt as the content target, instead of the commonly used feature responses of the content image. We find this leads to slightly faster convergence and also aligns with our goal of inverting the AdaIN output ttt.

它是内容损失 LcL_cLc​ 与风格损失 LsL_sLs​ 的加权组合,风格损失权重为 λ\lambdaλ。内容损失是目标特征与输出图像特征之间的欧氏距离。我们把 AdaIN 的输出 ttt 当作内容目标,而不是常用的内容图特征响应。我们发现这样收敛稍快一些,也与把 AdaIN 输出 ttt 倒回图像的目标相一致。

Lc=∥f(g(t))−t∥2(12)L_c=\|f(g(t))-t\|_2 \tag{12}Lc​=∥f(g(t))−t∥2​(12)

Since our AdaIN layer only transfers the mean and standard deviation of the style features, our style loss only matches these statistics. Although we find the commonly used Gram matrix loss can produce similar results, we match the IN statistics because it is conceptually cleaner. This style loss has also been explored by Li et al. [33].

由于 AdaIN 层只迁移风格特征的均值和标准差,我们的风格损失也只匹配这些统计量。虽然我们发现常用的 Gram 矩阵损失也能得到相近的结果,但匹配 IN 统计量在概念上更干净。Li 等人 [33] 也曾探讨过这种风格损失。

Ls=∑i=1L∥μ(ϕi(g(t)))−μ(ϕi(s))∥2+∑i=1L∥σ(ϕi(g(t)))−σ(ϕi(s))∥2(13)L_s=\sum_{i=1}^{L}\|\mu(\phi_i(g(t)))-\mu(\phi_i(s))\|_2+\sum_{i=1}^{L}\|\sigma(\phi_i(g(t)))-\sigma(\phi_i(s))\|_2 \tag{13}Ls​=i=1∑L​∥μ(ϕi​(g(t)))−μ(ϕi​(s))∥2​+i=1∑L​∥σ(ϕi​(g(t)))−σ(ϕi​(s))∥2​(13)

where each ϕi\phi_iϕi​ denotes a layer in VGG-19 used to compute the style loss. In our experiments we use relu1_1, relu2_1, relu3_1, relu4_1 layers with equal weights.

其中每个 ϕi\phi_iϕi​ 表示用来计算风格损失的一个 VGG-19 层。实验中我们使用 relu1_1、relu2_1、relu3_1、relu4_1,并赋予相同权重。

  1. Results

7.1. Comparison with other methods

In this subsection, we compare our approach with three types of style transfer methods: 1) the flexible but slow optimization-based method [16], 2) the fast feed-forward method restricted to a single style [52], and 3) the flexible patch-based method of medium speed [6]. If not mentioned otherwise, the results of compared methods are obtained by running their code with the default configurations.1^11 For [6], we use a pre-trained inverse network provided by the authors. All the test images are of size 512×512512 \times 512512×512.

本小节将我们的方法与三类风格迁移方法进行比较:1)灵活但缓慢的基于优化的方法 [16];2)受限于单一风格的快速前馈方法 [52];3)灵活、速度中等的基于补丁的方法 [6]。除非另有说明,对比方法的结果均通过运行其代码并使用默认配置得到。1^11 对于 [6],我们使用作者提供的预训练逆网络。所有测试图像的尺寸均为 512×512512 \times 512512×512。

1^11We run 500 iterations of [16] using Johnson's public implementation: https://github.com/jcjohnson/neural-style

1^11我们使用 Johnson 的公开实现运行 [16] 共 500 次迭代:https://github.com/jcjohnson/neural-style

Quantitative comparison of style and content loss across methods

Figure 3. Quantitative comparison of different methods in terms of style and content loss. Numbers are averaged over 10 style images and 50 content images randomly chosen from our test set.

图 3:不同方法在风格损失和内容损失上的定量比较。数字是对从测试集中随机选取的 10 张风格图和 50 张内容图取平均得到的。

Qualitative Examples. In Fig. 4 we show example style transfer results generated by compared methods. Note that all the test style images are never observed during the training of our model, while the results of [52] are obtained by fitting one network to each test style. Even so, the quality of our stylized images is quite competitive with [52] and [16] for many images (e.g., row 1, 2, 3). In some other cases (e.g., row 5) our method is slightly behind the quality of [52] and [16]. This is not unexpected, as we believe there is a three-way trade-off between speed, flexibility, and quality. Compared with [6], our method appears to transfer the style more faithfully for most compared images. The last example clearly illustrates a major limitation of [6], which attempts to match each content patch with the closest-matching style patch. However, if most content patches are matched to a few style patches that are not representative of the target style, the style transfer would fail. We thus argue that matching global feature statistics is a more general solution, although in some cases (e.g., row 3) the method of [6] can also produce appealing results.

定性例子。图 4 展示了对比方法生成的风格迁移示例。需要注意的是,所有测试风格图在我们模型的训练中都从未出现过,而 [52] 的结果是为每一种测试风格分别拟合一个网络得到的。即便如此,在许多图像上(例如第 1、2、3 行),我们风格化图像的质量已经可以与 [52] 和 [16] 相竞争。在另一些情形(例如第 5 行),我们的方法在质量上略逊于 [52] 和 [16]。这并不意外,因为我们认为速度、灵活性和质量之间存在三向权衡。与 [6] 相比,我们的方法在多数对比图像上似乎能更忠实地迁移风格。最后一个例子清楚地说明了 [6] 的一个主要局限:它试图把每个内容补丁与最相近的风格补丁对齐。可如果大多数内容补丁都被匹配到少数并不能代表目标风格的风格补丁上,风格迁移就会失败。因此我们认为,匹配全局特征统计量是更一般的解决办法,尽管在某些情形(例如第 3 行)中,[6] 的方法也能产生好看的结果。

Quantitative evaluations. Does our algorithm trade off some quality for higher speed and flexibility, and if so by how much? To answer this question quantitatively, we compare our approach with the optimization-based method [16] and the fast single-style transfer method [52] in terms of the content and style loss. Because our method uses a style loss based on IN statistics, we also modify the loss function in [16] and [52] accordingly for a fair comparison (their results in Fig. 4 are still obtained with the default Gram matrix loss). The content loss shown here is the same as in [52, 16]. The numbers reported are averaged over 10 style images and 50 content images randomly chosen from the test set of the WikiArt dataset [39] and MS-COCO [36].

定量评估。我们的算法是否为了更高的速度和灵活性而牺牲了部分质量,若是,牺牲了多少?为定量回答这个问题,我们在内容损失和风格损失上,把本方法与基于优化的方法 [16] 以及快速单风格迁移方法 [52] 做了比较。由于我们的方法使用基于 IN 统计量的风格损失,为公平比较,我们也相应修改了 [16] 和 [52] 中的损失函数(它们在图 4 中的结果仍由默认的 Gram 矩阵损失得到)。这里给出的内容损失与 [52, 16] 中相同。所报告的数字是对从 WikiArt 数据集 [39] 和 MS-COCO [36] 测试集中随机选取的 10 张风格图和 50 张内容图取平均。

Example arbitrary style transfer results of the AdaIN method

Figure 4. Example style transfer results. All the tested content and style images are never observed by our network during training.

图 4:风格迁移结果示例。所有测试用的内容图和风格图在训练中都未被我们的网络见过。

As shown in Fig. 3, the average content and style loss of our synthesized images are slightly higher but comparable to the single-style transfer method of Ulyanov et al. [52]. In particular, both our method and [52] obtain a style loss similar to that of [16] between 50 and 100 iterations of optimization. This demonstrates the strong generalization ability of our approach, considering that our network has never seen the test styles during training while each network of [52] is specifically trained on a test style. Also, note that our style loss is much smaller than that of the original content image.

如图 3 所示,我们合成图像的平均内容损失和风格损失略高于 Ulyanov 等人 [52] 的单风格迁移方法,但仍与之相当。特别是,我们的方法和 [52] 得到的风格损失,都与 [16] 在优化 50 到 100 次迭代之间的风格损失相近。这说明本方法具有很强的泛化能力——考虑到我们的网络在训练中从未见过测试风格,而 [52] 的每个网络都是专门在某一种测试风格上训练的。另外也请注意,我们的风格损失远小于原始内容图的风格损失。

Speed analysis. Most of our computation is spent on content encoding, style encoding, and decoding, each roughly taking one third of the time. In some application scenarios such as video processing, the style image needs to be encoded only once and AdaIN can use the stored style statistics to process all subsequent images. In some other cases (e.g., transferring the same content to different styles), the computation spent on content encoding can be shared.

速度分析。我们的计算主要花在内容编码、风格编码和解码上,三者大约各占三分之一时间。在视频处理这类应用里,风格图只需编码一次,AdaIN 可以用存储下来的风格统计量处理此后所有图像。在另一些情形(例如把同一内容转到不同风格)中,花在内容编码上的计算也可以共享。

In Tab. 1 we compare the speed of our method with previous ones [16, 52, 11, 6]. Excluding the time for style encoding, our algorithm runs at 56 and 15 FPS for 256×256256 \times 256256×256 and 512×512512 \times 512512×512 images respectively, making it possible to process arbitrary user-uploaded styles in real-time. Among algorithms applicable to arbitrary styles, our method is nearly 3 orders of magnitude faster than [16] and 1-2 orders of magnitude faster than [6]. The speed improvement over [6] is particularly significant for images of higher resolution, since the style swap layer in [6] does not scale well to high resolution style images. Moreover, our approach achieves comparable speed to feed-forward methods limited to a few styles [52, 11]. The slightly longer processing time of our method is mainly due to our larger VGG-based network, instead of methodological limitations. With a more efficient architecture, our speed can be further improved.

表 1 将我们方法的速度与先前工作 [16, 52, 11, 6] 做了比较。不计风格编码的时间,算法在 256×256256 \times 256256×256 和 512×512512 \times 512512×512 图像上分别达到 56 和 15 FPS,从而有可能实时处理用户上传的任意风格。在可用于任意风格的算法中,我们的方法比 [16] 快近 3 个数量级,比 [6] 快 1 到 2 个数量级。相对 [6] 的加速在更高分辨率图像上尤其明显,因为 [6] 中的风格交换层不能很好地扩展到高分辨率风格图。此外,本方法的速度与仅限于少数风格的前馈方法 [52, 11] 相当。我们略长的处理时间,主要是因为基于 VGG 的网络更大,而不是方法本身的限制。如果采用更高效的结构,速度还可以进一步提高。

Method Time (256px) Time (512px) # Styles
Gatys et al. 14.17 (14.19) 46.75 (46.79) ∞\infty∞
Chen and Schmidt 0.171 (0.407) 3.214 (4.144) ∞\infty∞
Ulyanov et al. 0.011 (N/A) 0.038 (N/A) 1
Dumoulin et al. 0.011 (N/A) 0.038 (N/A) 32
Ours 0.018 (0.027) 0.065 (0.098) ∞\infty∞

Table 1. Speed comparison (in seconds) for 256×256256 \times 256256×256 and 512×512512 \times 512512×512 images. Our approach achieves comparable speed to methods limited to a small number styles [52, 11], while being much faster than other existing algorithms applicable to arbitrary styles [16, 6]. We show the processing time both excluding and including (in parenthesis) the style encoding procedure. Results are obtained with a Pascal Titan X GPU and averaged over 100 images.

方法 用时(256px) 用时(512px) 风格数
Gatys et al. 14.17 (14.19) 46.75 (46.79) ∞\infty∞
Chen and Schmidt 0.171 (0.407) 3.214 (4.144) ∞\infty∞
Ulyanov et al. 0.011 (N/A) 0.038 (N/A) 1
Dumoulin et al. 0.011 (N/A) 0.038 (N/A) 32
本文 0.018 (0.027) 0.065 (0.098) ∞\infty∞

表 1:256×256256 \times 256256×256 与 512×512512 \times 512512×512 图像的速度比较(单位为秒)。我们的方法在速度上可与仅限于少量风格的方法 [52, 11] 相比,同时又远快于其他可用于任意风格的现有算法 [16, 6]。我们同时给出了不包括以及包括(括号内)风格编码过程的处理时间。结果在 Pascal Titan X GPU 上测得,并对 100 张图像取平均。

7.2. Additional experiments.

In this subsection, we conduct experiments to justify our important architectural choices. We denote our approach described in Sec. 6 as Enc-AdaIN-Dec. We experiment with a model named Enc-Concat-Dec that replaces AdaIN with concatenation, which is a natural baseline strategy to combine information from the content and style images. In addition, we run models with BN/IN layers in the decoder, denoted as Enc-AdaIN-BNDec and Enc-AdaIN-INDec respectively. Other training settings are kept the same.

本小节通过实验来说明若干重要的结构选择。我们把第 6 节描述的方法记为 Enc-AdaIN-Dec。我们还实验了一种名为 Enc-Concat-Dec 的模型,用拼接代替 AdaIN,这是组合内容图与风格图信息的一种自然基线。此外,我们还运行了在解码器中加入 BN/IN 层的模型,分别记为 Enc-AdaIN-BNDec 和 Enc-AdaIN-INDec。其余训练设置保持不变。

Comparison of AdaIN with concatenation and BN/IN decoder baselines

Figure 5. Comparison with baselines. AdaIN is much more effective than concatenation in fusing the content and style information. Also, it is important not to use BN or IN layers in the decoder.

图 5:与基线的比较。在融合内容与风格信息时,AdaIN 比拼接有效得多。此外,解码器中不宜使用 BN 或 IN 层。

Training curves of style and content loss

Figure 6. Training curves of style and content loss.

图 6:风格损失和内容损失的训练曲线。

In Fig. 5 and 6, we show examples and training curves of the compared methods. In the image generated by the Enc-Concat-Dec baseline (Fig. 5 (d)), the object contours of the style image can be clearly observed, suggesting that the network fails to disentangle the style information from the content of the style image. This is also consistent with Fig. 6, where Enc-Concat-Dec can reach low style loss but fail to decrease the content loss. Models with BN/IN layers also obtain qualitatively worse results and consistently higher losses. The results with IN layers are especially poor. This once again verifies our claim that IN layers tend to normalize the output to a single style and thus should be avoided when we want to generate images in different styles.

图 5 和图 6 给出了对比方法的示例和训练曲线。在 Enc-Concat-Dec 基线生成的图像中(图 5 (d)),可以清楚看到风格图中的物体轮廓,说明网络未能把风格信息从风格图的内容中解开。这与图 6 也相一致:Enc-Concat-Dec 可以把风格损失降得很低,却无法降低内容损失。带 BN/IN 层的模型在定性结果上也更差,损失始终更高。使用 IN 层的结果尤其差。这再次验证了我们的判断:IN 层倾向于把输出归一到单一风格,因此在希望生成不同风格图像时应当避免。

7.3. Runtime controls

To further highlight the flexibility of our method, we show that our style transfer network allows users to control the degree of stylization, interpolate between different styles, transfer styles while preserving colors, and use different styles in different spatial regions. Note that all these controls are only applied at runtime using the same network, without any modification to the training procedure.

为进一步突出本方法的灵活性,我们展示风格迁移网络允许用户控制风格化程度、在不同风格之间插值、在保持颜色的同时迁移风格,以及在不同空间区域使用不同风格。需要注意的是,这些控制都只在运行时作用于同一套网络,无需改动训练流程。

Content-style trade-off. The degree of style transfer can be controlled during training by adjusting the style weight λ\lambdaλ in Eqa. 11. In addition, our method allows content-style trade-off at test time by interpolating between feature maps that are fed to the decoder. Note that this is equivalent to interpolating between the affine parameters of AdaIN.

内容–风格权衡。风格迁移的程度可以在训练时通过调整 Eqa. 11 中的风格权重 λ\lambdaλ 来控制。此外,本方法还允许在测试时通过对送入解码器的特征图做插值,来权衡内容与风格。需要注意,这等价于对 AdaIN 的仿射参数做插值。

T(c,s,α)=g((1−α)f(c)+α AdaIN(f(c),f(s)))(14)T(c,s,\alpha)=g\left((1-\alpha)f(c)+\alpha\,\mathrm{AdaIN}(f(c),f(s))\right) \tag{14}T(c,s,α)=g((1−α)f(c)+αAdaIN(f(c),f(s)))(14)

The network tries to faithfully reconstruct the content image when α=0\alpha=0α=0, and to synthesize the most stylized image when α=1\alpha=1α=1. As shown in Fig. 7, a smooth transition between content-similarity and style-similarity can be observed by changing α\alphaα from 0 to 1.

当 α=0\alpha=0α=0 时,网络试图忠实地重建内容图;当 α=1\alpha=1α=1 时,则合成风格化程度最高的图像。如图 7 所示,把 α\alphaα 从 0 变到 1,可以观察到从内容相似到风格相似的平滑过渡。

Content-style trade-off controlled by interpolation weight alpha

Figure 7. Content-style trade-off. At runtime, we can control the balance between content and style by changing the weight α\alphaα in Equ. 14.

图 7:内容–风格权衡。在运行时,我们可以通过改变公式 14 中的权重 α\alphaα 来控制内容与风格的平衡。

Style interpolation. To interpolate between a set of KKK style images s1,s2,…,sKs_1,s_2,\ldots,s_Ks1​,s2​,…,sK​ with corresponding weights w1,w2,…,wKw_1,w_2,\ldots,w_Kw1​,w2​,…,wK​ such that ∑k=1Kwk=1\sum_{k=1}^{K}w_k=1∑k=1K​wk​=1, we similarly interpolate between feature maps (results shown in Fig. 8):

风格插值。要在一组 KKK 张风格图 s1,s2,…,sKs_1,s_2,\ldots,s_Ks1​,s2​,…,sK​ 之间,按对应权重 w1,w2,…,wKw_1,w_2,\ldots,w_Kw1​,w2​,…,wK​(满足 ∑k=1Kwk=1\sum_{k=1}^{K}w_k=1∑k=1K​wk​=1)做插值,我们同样在特征图之间插值(结果见图 8):

T(c,s1,2,…,K,w1,2,…,K)=g(∑k=1Kwk AdaIN(f(c),f(sk)))(15)T(c,s_{1,2,\ldots,K},w_{1,2,\ldots,K})=g\left(\sum_{k=1}^{K}w_k\,\mathrm{AdaIN}(f(c),f(s_k))\right) \tag{15}T(c,s1,2,…,K​,w1,2,…,K​)=g(k=1∑K​wk​AdaIN(f(c),f(sk​)))(15)

Style interpolation between arbitrary styles via convex combination of AdaIN features

Figure 8. Style interpolation. By feeding the decoder with a convex combination of feature maps transferred to different styles via AdaIN (Equ. 15), we can interpolate between arbitrary new styles.

图 8:风格插值。把经 AdaIN 转到不同风格的特征图做凸组合后再送入解码器(公式 15),就可以在任意新风格之间插值。

Spatial and color control. Gatys et al. [17] recently introduced user controls over color information and spatial locations of style transfer, which can be easily incorporated into our framework. To preserve the color of the content image, we first match the color distribution of the style image to that of the content image (similar to [17]), then perform a normal style transfer using the color-aligned style image as the style input. Examples results are shown in Fig. 9.

空间与颜色控制。Gatys 等人 [17] 最近引入了对风格迁移中颜色信息和空间位置的用户控制,这些控制很容易纳入我们的框架。为保持内容图的颜色,我们先把风格图的颜色分布对齐到内容图(与 [17] 类似),再以颜色对齐后的风格图作为风格输入,做一次常规的风格迁移。示例结果见图 9。

Color-preserved style transfer using color control

Figure 9. Color control. Left: content and style images. Right: color-preserved style transfer result.

图 9:颜色控制。左:内容图与风格图。右:保持颜色的风格迁移结果。

In Fig. 10 we demonstrate that our method can transfer different regions of the content image to different styles. This is achieved by performing AdaIN separately to different regions in the content feature maps using statistics from different style inputs, similar to [4, 17] but in a completely feed-forward manner. While our decoder is only trained on inputs with homogeneous styles, it generalizes naturally to inputs in which different regions have different styles.

图 10 表明,我们的方法可以把内容图的不同区域转到不同风格。做法是:对内容特征图的不同区域,分别使用来自不同风格输入的统计量做 AdaIN,这与 [4, 17] 类似,但完全是前馈的。尽管解码器只在风格均匀的输入上训练过,它也能自然地泛化到不同区域具有不同风格的输入。

Spatial control of style transfer with two style images and masks

Figure 10. Spatial control. Left: content image. Middle: two style images with corresponding masks. Right: style transfer result.

图 10:空间控制。左:内容图。中:两张风格图及对应掩膜。右:风格迁移结果。

  1. Discussion and Conclusion

In this paper, we present a simple adaptive instance normalization (AdaIN) layer that for the first time enables arbitrary style transfer in real-time. Beyond the fascinating applications, we believe this work also sheds light on our understanding of deep image representations in general.

本文提出一种简单的自适应实例归一化(AdaIN)层,第一次实现了任意风格的实时迁移。除了这些引人入胜的应用之外,我们相信这项工作也对一般意义上的深度图像表示有所启发。

It is interesting to consider the conceptual differences between our approach and previous neural style transfer methods based on feature statistics. Gatys et al. [16] employ an optimization process to manipulate pixel values to match feature statistics. The optimization process is replaced by feed-forward neural networks in [24, 51, 52]. Still, the network is trained to modify pixel values to indirectly match feature statistics. We adopt a very different approach that directly aligns statistics in the feature space in one shot, then inverts the features back to the pixel space.

把我们的方法与以往基于特征统计量的神经风格迁移方法做概念上的比较,是一件有意思的事。Gatys 等人 [16] 用优化过程去改像素值,以匹配特征统计量。[24, 51, 52] 用前馈神经网络替代了这一优化过程。尽管如此,网络仍是被训练去修改像素值,从而间接匹配特征统计量。我们采用了一种很不一样的途径:在特征空间中一次性地直接对齐统计量,再把特征倒回像素空间。

Given the simplicity of our approach, we believe there is still substantial room for improvement. In future works we plan to explore more advanced network architectures such as the residual architecture [24] or an architecture with additional skip connections from the encoder [23]. We also plan to investigate more complicated training schemes like the incremental training [32]. Moreover, our AdaIN layer only aligns the most basic feature statistics (mean and variance). It is possible that replacing AdaIN with correlation alignment [49] or histogram matching [54] could further improve quality by transferring higher-order statistics. Another interesting direction is to apply AdaIN to texture synthesis.

鉴于本方法的简单性,我们相信仍有很大的改进空间。在未来工作中,我们计划探索更先进的网络结构,例如残差结构 [24],或带有从编码器出发的额外跳连的结构 [23]。我们也计划研究更复杂的训练方案,例如增量训练 [32]。此外,我们的 AdaIN 层只对齐最基本的特征统计量(均值和方差)。用相关对齐 [49] 或直方图匹配 [54] 替换 AdaIN,有可能通过迁移更高阶统计量进一步提升质量。另一个有意思的方向是把 AdaIN 用于纹理合成。

Acknowledgments

致谢

We would like to thank Andreas Veit for helpful discussions. This work was supported in part by a Google Focused Research Award, AWS Cloud Credits for Research and a Facebook equipment donation.

我们感谢 Andreas Veit 的有益讨论。本工作部分得到 Google Focused Research Award、AWS Cloud Credits for Research 以及 Facebook 设备捐赠的支持。

References

参考文献

[1] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017. 2

[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 2

[3] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. arXiv preprint arXiv:1612.05424, 2016. 2

[4] A. J. Champandard. Semantic style transfer and turning two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768, 2016. 8

[5] D. Chen, L. Yuan, J. Liao, N. Yu, and G. Hua. Stylebank: An explicit representation for neural image style transfer. In CVPR, 2017. 1

[6] T. Q. Chen and M. Schmidt. Fast patch-based style transfer of arbitrary style. arXiv preprint arXiv:1612.04337, 2016. 1, 2, 4, 5, 6, 7

[7] R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for machine learning. In NIPS Workshop, 2011. 4

[8] T. Cooijmans, N. Ballas, C. Laurent, C¸. G¨ulc¸ehre, and A. Courville. Recurrent batch normalization. In ICLR, 2017. 2

[9] E. L. Denton, S. Chintala, R. Fergus, et al. Deep genera- tive image models using a laplacian pyramid of adversarial networks. In NIPS, 2015. 2

[10] A. Dosovitskiy and T. Brox. Inverting visual representations with convolutional networks. In CVPR, 2016. 4

[11] V. Dumoulin, J. Shlens, and M. Kudlur. A learned represen- tation for artistic style. In ICLR, 2017. 1, 2, 3, 5, 6, 7

[12] A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. In SIGGRAPH, 2001. 1

[13] A. A. Efros and T. K. Leung. Texture synthesis by nonparametric sampling. In ICCV, 1999. 1

[14] M. Elad and P. Milanfar. Style-transfer via texture-synthesis. arXiv preprint arXiv:1609.03057, 2016. 1

[15] O. Frigo, N. Sabater, J. Delon, and P. Hellier. Split and match: example-based adaptive patch sampling for unsupervised style transfer. In CVPR, 2016. 1

[16] L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In CVPR, 2016. 1, 2, 3, 5, 6, 7, 8

[17] L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman. Controlling perceptual factors in neural style transfer. In CVPR, 2017. 1, 7, 8

[18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014. 2

[19] D. J. Heeger and J. R. Bergen. Pyramid-based texture analy- sis/synthesis. In SIGGRAPH, 1995. 1

[20] X. Huang, Y. Li, O. Poursaeed, J. Hopcroft, and S. Belongie. Stacked generative adversarial networks. In CVPR, 2017. 2

[21] S. Ioffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. arXiv preprint arXiv:1702.03275, 2017. 2

[22] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JMLR, 2015. 2

[23] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017. 2, 8

[24] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 1, 2, 3, 8

[25] T. Kim, M. Cha, H. Kim, J. Lee, and J. Kim. Learning to discover cross-domain relations with generative adversarial networks. arXiv preprint arXiv:1703.05192, 2017. 2

[26] D. Kingma and J. Ba. Adam: A method for stochastic opti- mization. In ICLR, 2015. 5

[27] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In ICLR, 2014. 2

[28] J. E. Kyprianidis, J. Collomosse, T. Wang, and T. Isenberg. State of the” art: A taxonomy of artistic stylization techniques for images and video. TVCG, 2013. 1

[29] C. Laurent, G. Pereyra, P. Brakel, Y. Zhang, and Y. Ben- gio. Batch normalized recurrent neural networks. In ICASSP, 2016. 2

[30] C. Li and M. Wand. Combining markov random fields and convolutional neural networks for image synthesis. In CVPR, 2016. 1, 2, 3

[31] C. Li and M. Wand. Precomputed real-time texture synthesis with markovian generative adversarial networks. In ECCV, 2016. 1, 2

[32] Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M.-H. Yang. Diversified texture synthesis with feed-forward networks. In CVPR, 2017. 1, 2, 8

[33] Y. Li, N. Wang, J. Liu, and X. Hou. Demystifying neural style transfer. arXiv preprint arXiv:1701.01036, 2017. 1, 2, 3, 5

[34] Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779, 2016. 2

[35] Q. Liao, K. Kawaguchi, and T. Poggio. Streaming normal- ization: Towards simpler and more biologically-plausible normalizations for online and recurrent learning. arXiv preprint arXiv:1610.06160, 2016. 2

[36] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Doll´ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 3, 5

[37] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. arXiv preprint arXiv:1703.00848, 2017. 2

[38] M.-Y. Liu and O. Tuzel. Coupled generative adversarial net- works. In NIPS, 2016. 2

[39] K. Nichol. Painter by numbers, wikiart. https://www. kaggle.com/c/painter-by-numbers, 2016. 5

[40] A. v. d. Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016. 2

[41] X. Peng and K. Saenko. Synthetic to real adaptation with deep generative correlation alignment networks. arXiv preprint arXiv:1701.05524, 2017. 2

[42] A. Radford, L. Metz, and S. Chintala. Unsupervised repre- sentation learning with deep convolutional generative adversarial networks. In ICLR, 2016. 2

[43] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. In ICML, 2016. 2

[44] M. Ren, R. Liao, R. Urtasun, F. H. Sinz, and R. S. Zemel. Normalizing the normalizers: Comparing and extending network normalization schemes. In ICLR, 2017. 2

[45] M. Ruder, A. Dosovitskiy, and T. Brox. Artistic style transfer for videos. In GCPR, 2016. 1

[46] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Rad- ford, and X. Chen. Improved techniques for training gans. In NIPS, 2016. 2

[47] T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In NIPS, 2016. 2

[48] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 4, 5

[49] B. Sun, J. Feng, and K. Saenko. Return of frustratingly easy domain adaptation. In AAAI, 2016. 8

[50] Y. Taigman, A. Polyak, and L. Wolf. Unsupervised cross- domain image generation. In ICLR, 2017. 2

[51] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky. Tex- ture networks: Feed-forward synthesis of textures and stylized images. In ICML, 2016. 1, 2, 4, 5, 8

[52] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In CVPR, 2017. 1, 2, 3, 5, 6, 7, 8

[53] X. Wang, G. Oxholm, D. Zhang, and Y.-F. Wang. Multimodal transfer: A hierarchical deep convolutional neural network for fast artistic style transfer. arXiv preprint arXiv:1612.01895, 2016. 2

[54] P. Wilmot, E. Risser, and C. Barnes. Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv:1701.08893, 2017. 2, 8

[55] H. Zhang and K. Dana. Multi-style generative network for real-time transfer. arXiv preprint arXiv:1703.06953, 2017. 1