Image Style Transfer Using Convolutional Neural Networks
Image Style Transfer Using Convolutional Neural Networks
Leon A. Gatys
Centre for Integrative Neuroscience, University of Tübingen, Germany
Bernstein Center for Computational Neuroscience, Tübingen, Germany
Graduate School of Neural Information Processing, University of Tübingen, Germany
leon.gatys@bethgelab.org
Alexander S. Ecker
Centre for Integrative Neuroscience, University of Tübingen, Germany
Bernstein Center for Computational Neuroscience, Tübingen, Germany
Max Planck Institute for Biological Cybernetics, Tübingen, Germany
Baylor College of Medicine, Houston, TX, USA
Matthias Bethge
Centre for Integrative Neuroscience, University of Tübingen, Germany
Bernstein Center for Computational Neuroscience, Tübingen, Germany
Max Planck Institute for Biological Cybernetics, Tübingen, Germany
Abstract
摘要
Rendering the semantic content of an image in different styles is a difficult image processing task. Arguably, a major limiting factor for previous approaches has been the lack of image representations that explicitly represent semantic information and, thus, allow to separate image content from style. Here we use image representations derived from Convolutional Neural Networks optimised for object recognition, which make high level image information explicit. We introduce A Neural Algorithm of Artistic Style that can separate and recombine the image content and style of natural images. The algorithm allows us to produce new images of high perceptual quality that combine the content of an arbitrary photograph with the appearance of numerous wellknown artworks. Our results provide new insights into the deep image representations learned by Convolutional Neural Networks and demonstrate their potential for high level image synthesis and manipulation.
把一幅图像的语义内容用不同风格来呈现,是一项困难的图像处理任务。可以说,先前方法的一个主要限制是,缺少能够显式表示语义信息、从而把图像内容与风格分开的图像表示。这里我们使用从为物体识别而优化的卷积神经网络中得到的图像表示,这些表示把高层图像信息显式化。我们提出一种神经艺术风格算法(A Neural Algorithm of Artistic Style),可以把自然图像的内容与风格分开并重新组合。该算法使我们能够生成感知质量很高的新图像,把任意照片的内容与众多著名艺术作品的外观结合起来。我们的结果为卷积神经网络学到的深度图像表示提供了新的认识,并展示了它们在高层图像合成与处理上的潜力。
1. Introduction
1. 引言
Transferring the style from one image onto another can be considered a problem of texture transfer. In texture transfer the goal is to synthesise a texture from a source image while constraining the texture synthesis in order to preserve the semantic content of a target image. For texture synthesis there exist a large range of powerful non-parametric algorithms that can synthesise photorealistic natural textures by resampling the pixels of a given source texture [7, 30, 8, 20]. Most previous texture transfer algorithms rely on these nonparametric methods for texture synthesis while using different ways to preserve the structure of the target image. For instance, Efros and Freeman introduce a correspondence map that includes features of the target image such as image intensity to constrain the texture synthesis procedure [8]. Hertzman et al. use image analogies to transfer the texture from an already stylised image onto a target image[13]. Ashikhmin focuses on transferring the high-frequency texture information while preserving the coarse scale of the target image [1]. Lee et al. improve this algorithm by additionally informing the texture transfer with edge orientation information [22].
把一幅图像的风格转到另一幅图像上,可以看成纹理迁移问题。在纹理迁移中,目标是从源图像合成一种纹理,同时约束纹理合成,以保住目标图像的语义内容。对纹理合成,已有大量强大的非参数算法,可以通过对给定源纹理的像素重采样,合成出照片般真实的自然纹理 [7, 30, 8, 20]。先前大多数纹理迁移算法依赖这些非参数方法来做纹理合成,同时用不同方式保住目标图像的结构。例如,Efros 和 Freeman 引入一张对应图,把目标图像的特征(如图像强度)包括进来,以约束纹理合成过程 [8]。Hertzman 等人用图像类比,把已经风格化的图像上的纹理转到目标图像上 [13]。Ashikhmin 着重迁移高频纹理信息,同时保住目标图像的粗尺度 [1]。Lee 等人通过额外用边缘方向信息来指导纹理迁移,改进了这一算法 [22]。
Although these algorithms achieve remarkable results, they all suffer from the same fundamental limitation: they use only low-level image features of the target image to inform the texture transfer. Ideally, however, a style transfer algorithm should be able to extract the semantic image content from the target image (e.g. the objects and the general scenery) and then inform a texture transfer procedure to render the semantic content of the target image in the style of the source image. Therefore, a fundamental prerequisite is to find image representations that independently model variations in the semantic image content and the style in which it is presented. Such factorised representations were previously achieved only for controlled subsets of natural images such as faces under different illumination conditions and characters in different font styles [29] or handwritten digits and house numbers [17]. To generally separate content from style in natural images is still an extremely difficult problem.
尽管这些算法取得了可观的结果,它们都受制于同一个根本限制:它们只用目标图像的低层图像特征来指导纹理迁移。然而,理想情况下,风格迁移算法应该能够从目标图像中提取语义内容(例如物体和整体场景),再指导纹理迁移过程,把目标图像的语义内容按源图像的风格呈现出来。因此,一个根本前提是找到能够独立建模语义内容变化与呈现风格变化的图像表示。这类分解表示此前只在受控的自然图像子集上实现过,例如不同光照条件下的人脸和不同字体风格的字符 [29],或手写数字与门牌号 [17]。一般地把自然图像中的内容与风格分开,仍然是一个极其困难的问题。
However, the recent advance of Deep Convolutional Neural Networks [18] has produced powerful computer vision systems that learn to extract high-level semantic information from natural images. It was shown that Convolutional Neural Networks trained with sufficient labeled data on specific tasks such as object recognition learn to extract high-level image content in generic feature representations that generalise across datasets [6] and even to other visual information processing tasks [19, 4, 2, 9, 23], including texture recognition [5] and artistic style classification [15]. In this work we show how the generic feature representations learned by high-performing Convolutional Neural Networks can be used to independently process and manipulate the content and the style of natural images. We introduce A Neural Algorithm of Artistic Style, a new algorithm to perform image style transfer. Conceptually, it is a texture transfer algorithm that constrains a texture synthesis method by feature representations from state-of-the-art Convolutional Neural Networks. Since the texture model is also based on deep image representations, the style transfer method elegantly reduces to an optimisation problem within a single neural network. New images are generated by performing a pre-image search to match feature representations of example images. This general approach has been used before in the context of texture synthesis [12, 25, 10] and to improve the understanding of deep image representations [27, 24]. In fact, our style transfer algorithm combines a parametric texture model based on Convolutional Neural Networks [10] with a method to invert their image representations [24].
不过,深度卷积神经网络的近期进展 [18] 已经产生了强大的计算机视觉系统,它们学会从自然图像中提取高层语义信息。已经表明,在物体识别这类特定任务上用足够标注数据训练的卷积神经网络,会在通用特征表示中学会提取高层图像内容,这些表示可以跨数据集泛化 [6],甚至泛化到其他视觉信息处理任务 [19, 4, 2, 9, 23],包括纹理识别 [5] 和艺术风格分类 [15]。在本工作中我们表明,高性能卷积神经网络学到的通用特征表示,可以用来独立地处理和操纵自然图像的内容与风格。我们提出一种神经艺术风格算法,这是一种做图像风格迁移的新算法。从概念上说,它是一种纹理迁移算法,用先进卷积神经网络的特征表示来约束纹理合成方法。由于纹理模型也基于深度图像表示,风格迁移方法便优雅地归结为单个神经网络内的优化问题。新图像通过对示例图像的特征表示做原像搜索来生成。这一一般做法此前已用于纹理合成 [12, 25, 10],以及增进对深度图像表示的理解 [27, 24]。事实上,我们的风格迁移算法把基于卷积神经网络的参数纹理模型 [10] 与反演其图像表示的方法 [24] 结合了起来。
2. Deep image representations
2. 深度图像表示
The results presented below were generated on the basis of the VGG network [28], which was trained to perform object recognition and localisation [26] and is described extensively in the original work [28]. We used the feature space provided by a normalised version of the 16 convolutional and 5 pooling layers of the 19-layer VGG network. We normalized the network by scaling the weights such that the mean activation of each convolutional filter over images and positions is equal to one. Such re-scaling can be done for the VGG network without changing its output, because it contains only rectifying linear activation functions and no normalization or pooling over feature maps. We do not use any of the fully connected layers. The model is publicly available and can be explored in the caffe-framework [14]. For image synthesis we found that replacing the maximum pooling operation by average pooling yields slightly more appealing results, which is why the images shown were generated with average pooling.
下面给出的结果是在 VGG 网络 [28] 的基础上生成的,该网络被训练来做物体识别与定位 [26],并在原工作中有详尽描述 [28]。我们使用的是 19 层 VGG 网络中 16 个卷积层和 5 个池化层的归一化版本所提供的特征空间。我们通过对权重缩放来归一化网络,使每个卷积滤波器在图像和位置上的平均激活等于 1。对 VGG 网络可以这样重新缩放而不改变其输出,因为它只包含线性整流激活函数,并且没有在特征图上做归一化或池化。我们不使用任何全连接层。该模型是公开可得的,可以在 caffe 框架中探索 [14]。对图像合成,我们发现用平均池化替换最大池化会得到略更悦目的结果,因此所示图像都是用平均池化生成的。
2.1. Content representation
2.1. 内容表示
Generally each layer in the network defines a non-linear filter bank whose complexity increases with the position of the layer in the network. Hence a given input image is encoded in each layer of the Convolutional Neural Network by the filter responses to that image. A layer with distinct filters has feature maps each of size , where is the height times the width of the feature map. So the responses in a layer can be stored in a matrix where is the activation of the th filter at position in layer . To visualise the image information that is encoded at different layers of the hierarchy one can perform gradient descent on a white noise image to find another image that matches the feature responses of the original image (Fig 1, content reconstructions) [24]. Let and be the original image and the image that is generated, and and their respective feature representation in layer . We then define the squared-error loss between the two feature representations
一般而言,网络中的每一层都定义了一个非线性滤波器组,其复杂性随该层在网络中的位置而增加。因此,给定输入图像 在卷积神经网络的每一层中,由对该图像的滤波器响应来编码。一个具有 个不同滤波器的层有 张特征图,每张大小为 ,其中 是特征图高度乘以宽度。于是第 层的响应可以存成矩阵 ,其中 是第 个滤波器在第 层位置 上的激活。要可视化层次结构中不同层所编码的图像信息,可以对一张白噪声图像做梯度下降,找出另一张与原图特征响应相匹配的图像(图 1,内容重建)[24]。设 和 分别是原图和生成的图像, 和 是它们在第 层各自的特征表示。然后我们定义这两个特征表示之间的平方误差损失
The derivative of this loss with respect to the activations in layer equals
该损失对第 层激活的导数等于
from which the gradient with respect to the image can be computed using standard error back-propagation (Fig 2, right). Thus we can change the initially random image until it generates the same response in a certain layer of the Convolutional Neural Network as the original image . When Convolutional Neural Networks are trained on object recognition, they develop a representation of the image that makes object information increasingly explicit along the processing hierarchy [10]. Therefore, along the processing hierarchy of the network, the input image is transformed into representations that are increasingly sensitive to the actual content of the image, but become relatively invariant to its precise appearance. Thus, higher layers in the network capture the high-level content in terms of objects and their arrangement in the input image but do not constrain the exact pixel values of the reconstruction very much (Fig 1, content reconstructions d, e). In contrast, reconstructions from the lower layers simply reproduce the exact pixel values of the original image (Fig 1, content reconstructions a–c). We therefore refer to the feature responses in higher layers of the network as the content representation.
由此可以用标准误差反向传播计算对图像 的梯度(图 2,右)。于是我们可以改变最初随机的图像 ,直到它在卷积神经网络的某一层产生与原图 相同的响应。当卷积神经网络在物体识别上训练时,它们会沿着处理层次发展出一种使物体信息越来越显式的图像表示 [10]。因此,沿着网络的处理层次,输入图像被变换成对图像实际内容越来越敏感、而对精确外观相对不变的表示。于是,网络中较高的层捕捉的是输入图像中以物体及其布置而言的高层内容,但对重建的精确像素值约束并不强(图 1,内容重建 d、e)。相反,从较低层重建则只是复现原图的精确像素值(图 1,内容重建 a–c)。因此我们把网络较高层中的特征响应称为内容表示。
Figure 1. Image representations in a Convolutional Neural Network (CNN). A given input image is represented as a set of filtered images at each processing stage in the CNN. While the number of different filters increases along the processing hierarchy, the size of the filtered images is reduced by some downsampling mechanism (e.g. max-pooling) leading to a decrease in the total number of units per layer of the network. Content Reconstructions. We can visualise the information at different processing stages in the CNN by reconstructing the input image from only knowing the network's responses in a particular layer. We reconstruct the input image from from layers 'conv1_2' (a), 'conv2_2' (b), 'conv3_2' (c), 'conv4_2' (d) and 'conv5_2' (e) of the original VGG-Network. We find that reconstruction from lower layers is almost perfect (a–c). In higher layers of the network, detailed pixel information is lost while the high-level content of the image is preserved (d,e). Style Reconstructions. On top of the original CNN activations we use a feature space that captures the texture information of an input image. The style representation computes correlations between the different features in different layers of the CNN. We reconstruct the style of the input image from a style representation built on different subsets of CNN layers ('conv1_1' (a), 'conv1_1' and 'conv2_1' (b), 'conv1_1', 'conv2_1' and 'conv3_1' (c), 'conv1_1', 'conv2_1', 'conv3_1' and 'conv4_1' (d), 'conv1_1', 'conv2_1', 'conv3_1', 'conv4_1' and 'conv5_1' (e). This creates images that match the style of a given image on an increasing scale while discarding information of the global arrangement of the scene.
图 1. 卷积神经网络(CNN)中的图像表示。给定输入图像在 CNN 的每个处理阶段被表示为一组经过滤波的图像。沿着处理层次,不同滤波器的数目增加,而滤波后图像的大小则通过某种下采样机制(例如最大池化)减小,从而导致网络每一层的单元总数下降。内容重建。我们可以只根据网络在某一层的响应来重建输入图像,从而可视化 CNN 不同处理阶段上的信息。我们从原始 VGG 网络的 'conv1_2'(a)、'conv2_2'(b)、'conv3_2'(c)、'conv4_2'(d)和 'conv5_2'(e)层重建输入图像(原文作 from from)。我们发现,从较低层重建几乎是完美的(a–c)。在网络的较高层,细致的像素信息会丢失,而图像的高层内容得以保留(d、e)。风格重建。在原始 CNN 激活之上,我们使用一个捕捉输入图像纹理信息的特征空间。风格表示计算 CNN 不同层中不同特征之间的相关。我们从建立在 CNN 层不同子集上的风格表示来重建输入图像的风格('conv1_1'(a),'conv1_1' 与 'conv2_1'(b),'conv1_1'、'conv2_1' 与 'conv3_1'(c),'conv1_1'、'conv2_1'、'conv3_1' 与 'conv4_1'(d),'conv1_1'、'conv2_1'、'conv3_1'、'conv4_1' 与 'conv5_1'(e))。这样会生成在越来越大的尺度上与给定图像风格相匹配、同时丢掉场景全局布置信息的图像。
2.2. Style representation
2.2. 风格表示
To obtain a representation of the style of an input image, we use a feature space designed to capture texture information [10]. This feature space can be built on top of the filter responses in any layer of the network. It consists of the correlations between the different filter responses, where the expectation is taken over the spatial extent of the feature maps. These feature correlations are given by the Gram matrix , where is the inner product between the vectorised feature maps and in layer :
为得到输入图像风格的表示,我们使用一个旨在捕捉纹理信息的特征空间 [10]。这一特征空间可以建立在网络任意一层的滤波器响应之上。它由不同滤波器响应之间的相关组成,期望是在特征图的空间范围上取的。这些特征相关由 Gram 矩阵 给出,其中 是第 层中向量化特征图 与 的内积:
By including the feature correlations of multiple layers, we obtain a stationary, multi-scale representation of the input image, which captures its texture information but not the global arrangement. Again, we can visualise the information captured by these style feature spaces built on different layers of the network by constructing an image that matches the style representation of a given input image (Fig 1, style reconstructions). This is done by using gradient descent from a white noise image to minimise the mean-squared distance between the entries of the Gram matrices from the original image and the Gram matrices of the image to be generated [10, 25]. Let and be the original image and the image that is generated, and and their respective style representation in layer . The contribution of layer to the total loss is then
通过纳入多层的特征相关,我们得到输入图像的一种平稳、多尺度表示,它捕捉其纹理信息,但不捕捉全局布置。同样,我们可以通过构造一张与给定输入图像的风格表示相匹配的图像,来可视化建立在网络不同层上的这些风格特征空间所捕捉的信息(图 1,风格重建)。做法是从一张白噪声图像出发做梯度下降,最小化原图 Gram 矩阵的条目与待生成图像 Gram 矩阵之间的均方距离 [10, 25]。设 和 分别是原图和生成的图像, 和 是它们在第 层各自的风格表示。第 层对总损失的贡献则为
and the total style loss is
而总的风格损失为
where are weighting factors of the contribution of each layer to the total loss (see below for specific values of in our results). The derivative of with respect to the activations in layer can be computed analytically:
其中 是每一层对总损失贡献的加权系数(下面给出我们结果中 的具体取值)。 对第 层激活的导数可以解析计算:
The gradients of with respect to the pixel values can be readily computed using standard error back-propagation (Fig 2, left).
对像素值 的梯度可以用标准误差反向传播直接算出(图 2,左)。
2.3. Style transfer
2.3. 风格迁移
To transfer the style of an artwork onto a photograph we synthesise a new image that simultaneously matches the content representation of and the style representation of (Fig 2). Thus we jointly minimise the distance of the feature representations of a white noise image from the content representation of the photograph in one layer and the style representation of the painting defined on a number of layers of the Convolutional Neural Network. The loss function we minimise is
要把一件艺术品 的风格转到一张照片 上,我们合成一张新图像,使其同时匹配 的内容表示和 的风格表示(图 2)。于是我们联合最小化一张白噪声图像的特征表示,与照片在某一层的内容表示、以及在卷积神经网络若干层上定义的绘画风格表示之间的距离。我们最小化的损失函数是
where and are the weighting factors for content and style reconstruction, respectively. The gradient with respect to the pixel values can be used as input for some numerical optimisation strategy. Here we use L-BFGS [32], which we found to work best for image synthesis. To extract image information on comparable scales, we always resized the style image to the same size as the content image before computing its feature representations. Finally, note that in difference to [24] we do not regularise our synthesis results with image priors. It could be argued, though, that the texture features from lower layers in the network act as a specific image prior for the style image. Additionally some differences in the image synthesis are expected due to the different network architecture and optimisation algorithm we use.
其中 和 分别是内容和风格重建的加权系数。对像素值的梯度 可以用作某种数值优化策略的输入。这里我们使用 L-BFGS [32],我们发现它对图像合成效果最好。为在可比尺度上提取图像信息,我们在计算风格图像的特征表示之前,总是先把它缩放到与内容图像相同的大小。最后要注意,与 [24] 不同,我们不用图像先验来正则化合成结果。不过也可以说,网络较低层的纹理特征对风格图像起到了一种特定图像先验的作用。此外,由于我们使用了不同的网络结构和优化算法,图像合成中也会预期出现一些差异。
Figure 2. Style transfer algorithm. First content and style features are extracted and stored. The style image is passed through the network and its style representation on all layers included are computed and stored (left). The content image is passed through the network and the content representation in one layer is stored (right). Then a random white noise image is passed through the network and its style features and content features are computed. On each layer included in the style representation, the element-wise mean squared difference between and is computed to give the style loss (left). Also the mean squared difference between and is computed to give the content loss (right). The total loss is then a linear combination between the content and the style loss. Its derivative with respect to the pixel values can be computed using error back-propagation (middle). This gradient is used to iteratively update the image until it simultaneously matches the style features of the style image and the content features of the content image (middle, bottom).
图 2. 风格迁移算法。首先提取并存储内容和风格特征。风格图像 通过网络,其在所有被纳入层上的风格表示 被计算并存储(左)。内容图像 通过网络,其在某一层的内容表示 被存储(右)。然后一张随机白噪声图像 通过网络,计算其风格特征 和内容特征 。在风格表示所包含的每一层上,计算 与 的逐元素均方误差,得到风格损失 (左)。同时计算 与 的均方误差,得到内容损失 (右)。总损失 则是内容损失与风格损失的线性组合。它对像素值的导数可以用误差反向传播计算(中)。该梯度被用来迭代更新图像 ,直到它同时匹配风格图像 的风格特征和内容图像 的内容特征(中,下)。
3. Results
3. 结果
The key finding of this paper is that the representations of content and style in the Convolutional Neural Network are well separable. That is, we can manipulate both representations independently to produce new, perceptually meaningful images. To demonstrate this finding, we generate images that mix the content and style representation from two different source images. In particular, we match the content representation of a photograph depicting the riverfront of the Neckar river in Tübingen, Germany and the style representations of several well-known artworks taken from different periods of art (Fig 3). The images shown in Fig 3 were synthesised by matching the content representation on layer 'conv4_2' and the style representation on layers 'conv1_1', 'conv2_1', 'conv3_1', 'conv4_1' and 'conv5_1' ( in those layers, in all other layers). The ratio was either (Fig 3 B), (Fig 3 C), (Fig 3 D), or (Fig 3 E, F).
本文的关键发现是,卷积神经网络中的内容表示与风格表示是可以很好地分开的。也就是说,我们可以独立地操纵这两种表示,以产生新的、在感知上有意义的图像。为证明这一发现,我们生成把两张不同源图像的内容表示与风格表示混合起来的图像。具体而言,我们匹配一张描绘德国图宾根内卡河畔的照片的内容表示,以及取自不同艺术时期的若干著名作品的风格表示(图 3)。图 3 中所示图像是通过在 'conv4_2' 层匹配内容表示,并在 'conv1_1'、'conv2_1'、'conv3_1'、'conv4_1' 和 'conv5_1' 层匹配风格表示而合成的(在这些层上 ,在所有其他层上 )。比值 分别为 (图 3 B)、(图 3 C)、(图 3 D)或 (图 3 E、F)。
Figure 3. Images that combine the content of a photograph with the style of several well-known artworks. The images were created by finding an image that simultaneously matches the content representation of the photograph and the style representation of the artwork. The original photograph depicting the Neckarfront in Tübingen, Germany, is shown in A (Photo: Andreas Praefcke). The painting that provided the style for the respective generated image is shown in the bottom left corner of each panel. B The Shipwreck of the Minotaur by J.M.W. Turner, 1805. C The Starry Night by Vincent van Gogh, 1889. D Der Schrei by Edvard Munch, 1893. E Femme nue assise by Pablo Picasso, 1910. F Composition VII by Wassily Kandinsky, 1913.
图 3. 把一张照片的内容与若干著名艺术作品的风格结合起来的图像。这些图像是通过找到一张同时匹配照片内容表示和艺术品风格表示的图像而创建的。描绘德国图宾根内卡河畔(Neckarfront)的原始照片示于 A(照片:Andreas Praefcke)。为各张生成图像提供风格的绘画示于每幅图的左下角。B 是 J.M.W. Turner 的 The Shipwreck of the Minotaur,1805。C 是 Vincent van Gogh 的 The Starry Night,1889。D 是 Edvard Munch 的 Der Schrei,1893。E 是 Pablo Picasso 的 Femme nue assise,1910。F 是 Wassily Kandinsky 的 Composition VII,1913。
3.1. Trade-off between content and style matching
3.1. 内容匹配与风格匹配之间的权衡
Of course, image content and style cannot be completely disentangled. When synthesising an image that combines the content of one image with the style of another, there usually does not exist an image that perfectly matches both constraints at the same time. However, since the loss function we minimise during image synthesis is a linear combination between the loss functions for content and style respectively, we can smoothly regulate the emphasis on either reconstructing the content or the style (Fig 4). A strong emphasis on style will result in images that match the appearance of the artwork, effectively giving a texturised version of it, but show hardly any of the photograph's content (, Fig 4, top left). When placing strong emphasis on content, one can clearly identify the photograph, but the style of the painting is not as well-matched (, Fig 4, bottom right). For a specific pair of content and style images one can adjust the trade-off between content and style to create visually appealing images.
当然,图像内容与风格并不能完全解开。当合成一张把一幅图像的内容与另一幅图像的风格结合起来的图像时,通常并不存在同时完美满足这两个约束的图像。不过,由于我们在图像合成中最小化的损失函数是内容损失与风格损失的线性组合,我们可以平滑地调节对重建内容或重建风格的强调(图 4)。强烈强调风格,会得到与艺术品外观相匹配的图像,实际上给出它的纹理化版本,但几乎看不出照片的内容(,图 4,左上)。当强烈强调内容时,可以清楚地认出照片,但绘画的风格匹配得没有那么好(,图 4,右下)。对一对特定的内容图与风格图,可以调节内容与风格之间的权衡,以创建视觉上悦目的图像。
Figure 4. Relative weighting of matching content and style of the respective source images. The ratio between matching the content and matching the style increases from top left to bottom right. A high emphasis on the style effectively produces a texturised version of the style image (top left). A high emphasis on the content produces an image with only little stylisation (bottom right). In practice one can smoothly interpolate between the two extremes.
图 4. 对各自源图像的内容匹配与风格匹配的相对加权。内容匹配与风格匹配之间的比值 从左上到右下递增。强烈强调风格,实际上会得到风格图像的纹理化版本(左上)。强烈强调内容,会得到只有很少风格化的图像(右下)。实践中可以在这两个极端之间平滑插值。
3.2. Effect of different layers of the Convolutional Neural Network
3.2. 卷积神经网络不同层的影响
Another important factor in the image synthesis process is the choice of layers to match the content and style representation on. As outlined above, the style representation is a multi-scale representation that includes multiple layers of the neural network. The number and position of these layers determines the local scale on which the style is matched, leading to different visual experiences (Fig 1, style reconstructions). We find that matching the style representations up to higher layers in the network preserves local images structures an increasingly large scale, leading to a smoother and more continuous visual experience. Thus, the visually most appealing images are usually created by matching the style representation up to high layers in the network, which is why for all images shown we match the style features in layers 'conv1_1', 'conv2_1', 'conv3_1', 'conv4_1' and 'conv5_1' of the network. To analyse the effect of using different layers to match the content features, we present a style transfer result obtained by stylising a photograph with the same artwork and parameter configuration (), but in one matching the content features on layer 'conv2_2' and in the other on layer 'conv4_2' (Fig 5). When matching the content on a lower layer of the network, the algorithm matches much of the detailed pixel information in the photograph and the generated image appears as if the texture of the artwork is merely blended over the photograph (Fig 5, middle). In contrast, when matching the content features on a higher layer of the network, detailed pixel information of the photograph is not as strongly constraint and the texture of the artwork and the content of the photograph are properly merged. That is, the fine structure of the image, for example the edges and colour map, is altered such that it agrees with the style of the artwork while displaying the content of the photograph (Fig 5, bottom).
图像合成过程中另一个重要因素,是选择在哪些层上匹配内容表示和风格表示。如上所述,风格表示是一种包含神经网络多个层的多尺度表示。这些层的数目和位置决定了风格被匹配的局部尺度,从而带来不同的视觉体验(图 1,风格重建)。我们发现,把风格表示匹配到网络中更高的层,会在越来越大的尺度上保住局部图像结构(原文作 images structures an increasingly large scale),从而带来更平滑、更连续的视觉体验。因此,视觉上最悦目的图像通常是通过把风格表示匹配到网络中的高层来创建的,这就是为什么对所有展示的图像,我们都在网络的 'conv1_1'、'conv2_1'、'conv3_1'、'conv4_1' 和 'conv5_1' 层匹配风格特征。为分析使用不同层来匹配内容特征的效果,我们给出用同一件艺术品和同一参数配置()对一张照片做风格化的结果,但一次在 'conv2_2' 层匹配内容特征,另一次在 'conv4_2' 层匹配(图 5)。当在网络较低层匹配内容时,算法会匹配照片中大量细致的像素信息,生成的图像看起来就像只是把艺术品的纹理叠在照片上(图 5,中)。相反,当在网络较高层匹配内容特征时,照片的细致像素信息不会被那么强地约束(原文作 constraint),艺术品的纹理与照片的内容会适当地融合。也就是说,图像的精细结构,例如边缘和色彩映射,会被改变,使之与艺术品的风格相符,同时显示照片的内容(图 5,下)。
Figure 5. The effect of matching the content representation in different layers of the network. Matching the content on layer 'conv2_2' preserves much of the fine structure of the original photograph and the synthesised image looks as if the texture of the painting is simply blended over the photograph (middle). When matching the content on layer 'conv4_2' the texture of the painting and the content of the photograph merge together such that the content of the photograph is displayed in the style of the painting (bottom). Both images were generated with the same choice of parameters (). The painting that served as the style image is shown in the bottom left corner and is named Jesuiten III by Lyonel Feininger, 1915.
图 5. 在网络不同层匹配内容表示的效果。在 'conv2_2' 层匹配内容,会保住原始照片的许多精细结构,合成图像看起来就像只是把绘画的纹理叠在照片上(中)。在 'conv4_2' 层匹配内容时,绘画的纹理与照片的内容融合在一起,使照片的内容以该绘画的风格显示出来(下)。两张图像都用同一组参数生成()。作为风格图像的绘画示于左下角,是 Lyonel Feininger 1915 年的 Jesuiten III。
3.3. Initialisation of gradient descent
3.3. 梯度下降的初始化
We have initialised all images shown so far with white noise. However, one could also initialise the image synthesis with either the content image or the style image. We explored these two alternatives (Fig 6 A, B): although they bias the final image somewhat towards the spatial structure of the initialisation, the different initialisations do not seem to have a strong effect on the outcome of the synthesis procedure. It should be noted that only initialising with noise allows to generate an arbitrary number of new images (Fig 6 C). Initialising with a fixed image always deterministically leads to the same outcome (up to stochasticity in the gradient descent procedure).
到目前为止展示的所有图像,我们都是用白噪声初始化的。不过,也可以用内容图像或风格图像来初始化图像合成。我们探索了这两种替代做法(图 6 A、B):尽管它们会使最终图像多少偏向初始化的空间结构,不同的初始化对合成过程的结果似乎没有很强的影响。应当指出,只有用噪声初始化,才允许生成任意数量的新图像(图 6 C)。用一张固定图像初始化,总是确定地得到同一结果(直到梯度下降过程中的随机性)。
图 6. 梯度下降的初始化。A 从内容图像初始化。B 从风格图像初始化。C 从不同白噪声图像初始化得到的四个样本。对所有图像,比值 都等于 。
3.4. Photorealistic style transfer
3.4. 照片般真实的风格迁移
Thus far the focus of this paper was on artistic style transfer. In general though, the algorithm can transfer the style between arbitrary images. As an example we transfer the style of a photograph of New York by night onto an image of London in daytime (Fig 7). Although photorealism is not fully preserved, the synthesised image resembles much of the colours and lightning of the style image and to some extent displays an image of London by night.
到目前为止,本文的重点是艺术风格迁移。不过一般而言,该算法可以在任意图像之间迁移风格。作为例子,我们把一张纽约夜景照片的风格转到一张伦敦日景图像上(图 7)。尽管照片真实感并未完全保住,合成图像在很大程度上类似风格图像的色彩和光线(原文作 lightning),并且在某种程度上呈现出一幅伦敦夜景。
图 7. 照片般真实的风格迁移。风格从一张纽约夜景照片转到一张伦敦日景图片上。图像合成从内容图像初始化,比值 等于 。
4. Discussion
4. 讨论
Here we demonstrate how to use feature representations from high-performing Convolutional Neural Networks to transfer image style between arbitrary images. While we are able to show results of high perceptual quality, there are still some technical limitations to the algorithm. Probably the most limiting factor is the resolution of the synthesised images. Both, the dimensionality of the optimisation problem as well as the number of units in the Convolutional Neural Network grow linearly with the number of pixels. Therefore the speed of the synthesis procedure depends heavily on image resolution. The images presented in this paper were synthesised in a resolution of about pixels and the synthesis procedure could take up to an hour on a Nvidia K40 GPU (depending on the exact image size and the stopping criteria for the gradient descent). While this performance currently prohibits online and interactive applications of our style transfer algorithm, it is likely that future improvements in deep learning will also increase the performance of this method. Another issue is that synthesised images are sometimes subject to some low-level noise. While this is less of an issue in the artistic style transfer, the problem becomes more apparent when both, content and style images, are photographs and the photorealism of the synthesised image is affected. However, the noise is very characteristic and appears to resemble the filters of units in the network. Thus it could be possible to construct efficient de-noising techniques to post-process the images after the optimisation procedure.
这里我们展示了如何使用高性能卷积神经网络中的特征表示,在任意图像之间迁移图像风格。尽管我们能够给出感知质量很高的结果,该算法仍有一些技术上的限制。最主要的限制因素大概是合成图像的分辨率。优化问题的维数以及卷积神经网络中的单元数,都随像素数线性增长。因此合成过程的速度严重依赖于图像分辨率。本文展示的图像是在大约 像素的分辨率下合成的,在一块 Nvidia K40 GPU 上合成过程可能需要一个小时(取决于精确的图像大小和梯度下降的停止准则)。虽然这一性能目前妨碍了我们的风格迁移算法的在线与交互应用,但深度学习未来的改进也很可能会提高这一方法的性能。另一个问题是,合成图像有时会带有一些低层噪声。这在艺术风格迁移中不太成问题,但当内容图和风格图都是照片、从而影响合成图像的照片真实感时,问题会更加明显。不过,这种噪声很有特点,看起来类似网络中单元的滤波器。因此有可能构造有效的去噪技术,在优化过程之后对图像做后处理。
Artistic stylisation of images is traditionally studied in computer graphics under the label of non-photorealistic rendering. Apart from the work on texture transfer, common approaches are conceptually quite different to our work in that they give specialised algorithms to render a source image in one specific style. For a recent review of the field we refer the reader to [21].
图像的艺术风格化传统上在计算机图形学中以非真实感绘制这一名称来研究。除了纹理迁移方面的工作外,常见方法在概念上与我们的工作相当不同,因为它们给出的是把源图像绘制成某一种特定风格的专门算法。关于该领域的近期综述,我们请读者参阅 [21]。
The separation of image content from style is not necessarily a well defined problem. This is mostly because it is not clear what exactly defines the style of an image. It might be the brush strokes in a painting, the colour map, certain dominant forms and shapes, but also the composition of a scene and the choice of the subject of the image – and probably it is a mixture of all of them and many more. Therefore, it is generally not clear if image content and style can be completely separated at all – and if so, how. For example it is not possible to render an image in the style of van Gogh's "Starry Night" without having image structures that resemble the stars. In our work we consider style transfer to be successful if the generated image 'looks like' the style image but shows the objects and scenery of the content image. We are fully aware though that this evaluation criterion is neither mathematically precise nor universally agreed upon. Nevertheless, we find it truly fascinating that a neural system, which is trained to perform one of the core computational tasks of biological vision, automatically learns image representations that allow – at least to some extent – the separation of image content from style. One explanation may be that when learning object recognition, the network has to become invariant to all image variation that preserves object identity. Representations that factorise the variation in the content of an image and the variation in its appearance would be extremely practical for this task. In light of the striking similarities between performance-optimised artificial neural networks and biological vision [11, 31, 3, 19, 16], it is thus tempting to speculate that the human ability to abstract content from style – and therefore our ability to create and enjoy art – might also be primarily a preeminent signature of the powerful inference capabilities of our visual system.
把图像内容与风格分开,未必是一个定义良好的问题。这主要是因为并不清楚究竟是什么定义了一幅图像的风格。它可能是绘画中的笔触、色彩映射、某些主导的形式与形状,但也可以是场景的构图和图像题材的选择,而且很可能是所有这些以及更多因素的混合。因此,一般并不清楚图像内容与风格究竟能否完全分开,以及如果能分开,该如何分开。例如,不可能在没有类似星星的图像结构的情况下,把一幅图像画成梵高《星夜》的风格。在我们的工作中,如果生成的图像“看起来像”风格图像,却显示内容图像中的物体和景色,我们就认为风格迁移是成功的。不过我们完全清楚,这一评价准则既不是数学上精确的,也不是普遍公认的。尽管如此,我们仍然觉得真正令人着迷的是:一个被训练来执行生物视觉核心计算任务之一的神经系统,会自动学会至少在某种程度上允许把图像内容与风格分开的图像表示。一种解释可能是,在学习物体识别时,网络必须对所有保住物体身份的图像变化变得不变。把图像内容上的变化与外观上的变化分解开来的表示,对这一任务会极其实用。鉴于性能优化的人工神经网络与生物视觉之间显著的相似性 [11, 31, 3, 19, 16],因此忍不住会推测:人类从风格中抽象出内容的能力,进而我们创造和欣赏艺术的能力,主要也可能是我们视觉系统强大推理能力的一个突出标志。
References
参考文献
[1] N. Ashikhmin. Fast texture transfer. IEEE Computer Graphics and Applications, 23(4):38–43, July 2003. 1
[2] M. Berning, K. M. Boergens, and M. Helmstaedter. SegEM: Efficient Image Analysis for High-Resolution Connectomics. Neuron, 87(6):1193–1206, Sept. 2015. 2
[3] C. F. Cadieu, H. Hong, D. L. K. Yamins, N. Pinto, D. Ardila, E. A. Solomon, N. J. Majaj, and J. J. DiCarlo. Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition. PLoS Comput Biol, 10(12):e1003963, Dec. 2014. 8
[4] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. arXiv:1412.7062 [cs], Dec. 2014. arXiv: 1412.7062. 2
[5] M. Cimpoi, S. Maji, and A. Vedaldi. Deep filter banks for texture recognition and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3828–3836, 2015. 2
[6] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. arXiv:1310.1531 [cs], Oct. 2013. arXiv: 1310.1531. 2
[7] A. Efros and T. K. Leung. Texture synthesis by nonparametric sampling. In Computer Vision, 1999. The Proceedings of the Seventh IEEE International Conference on, volume 2, pages 1033–1038. IEEE, 1999. 1
[8] A. A. Efros and W. T. Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 341–346. ACM, 2001. 1
[9] D. Eigen and R. Fergus. Predicting Depth, Surface Normals and Semantic Labels With a Common Multi-Scale Convolutional Architecture. pages 2650–2658, 2015. 2
[10] L. A. Gatys, A. S. Ecker, and M. Bethge. Texture Synthesis Using Convolutional Neural Networks. In Advances in Neural Information Processing Systems 28, 2015. 3, 4
[11] U. Güçlü and M. A. J. v. Gerven. Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream. The Journal of Neuroscience, 35(27):10005–10014, July 2015. 8
[12] D. J. Heeger and J. R. Bergen. Pyramid-based Texture Analysis/Synthesis. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’95, pages 229–238, New York, NY, USA, 1995. ACM. 3
[13] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin. Image analogies. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 327–340. ACM, 2001. 1
[14] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia, pages 675–678. ACM, 2014. 3
[15] S. Karayev, M. Trentacoste, H. Han, A. Agarwala, T. Darrell, A. Hertzmann, and H. Winnemoeller. Recognizing image style. arXiv preprint arXiv:1311.3715, 2013. 2
[16] S.-M. Khaligh-Razavi and N. Kriegeskorte. Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation. PLoS Comput Biol, 10(11):e1003915, Nov. 2014. 8
[17] D. P. Kingma, S. Mohamed, D. Jimenez Rezende, and M. Welling. Semi-supervised Learning with Deep Generative Models. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3581–3589. Curran Associates, Inc., 2014. 2
[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097–1105, 2012. 2
[19] M. Kümmerer, L. Theis, and M. Bethge. Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet. In ICLR Workshop, 2015. 2, 8
[20] V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick. Graphcut textures: image and video synthesis using graph cuts. In ACM Transactions on Graphics (ToG), volume 22, pages 277–286. ACM, 2003. 1
[21] J. E. Kyprianidis, J. Collomosse, T. Wang, and T. Isenberg. State of the "Art": A Taxonomy of Artistic Stylization Techniques for Images and Video. Visualization and Computer Graphics, IEEE Transactions on, 19(5):866–885, 2013. 8
[22] H. Lee, S. Seo, S. Ryoo, and K. Yoon. Directional Texture Transfer. In Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering, NPAR ’10, pages 43–48, New York, NY, USA, 2010. ACM. 1
[23] J. Long, E. Shelhamer, and T. Darrell. Fully Convolutional Networks for Semantic Segmentation. pages 3431–3440, 2015. 2
[24] A. Mahendran and A. Vedaldi. Understanding Deep Image Representations by Inverting Them. arXiv:1412.0035 [cs], Nov. 2014. arXiv: 1412.0035. 3, 6
[25] J. Portilla and E. P. Simoncelli. A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients. International Journal of Computer Vision, 40(1):49–70, Oct. 2000. 3, 4
[26] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575 [cs], Sept. 2014. arXiv: 1409.0575. 3
[27] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. arXiv:1312.6034 [cs], Dec. 2013. 3
[28] K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556 [cs], Sept. 2014. arXiv: 1409.1556. 3
[29] J. B. Tenenbaum and W. T. Freeman. Separating style and content with bilinear models. Neural computation, 12(6):1247–1283, 2000. 2
[30] L. Wei and M. Levoy. Fast texture synthesis using tree-structured vector quantization. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 479–488. ACM Press/Addison-Wesley Publishing Co., 2000. 1
[31] D. L. K. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, page 201403112, May 2014. 8
[32] C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal. Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on Mathematical Software (TOMS), 23(4):550–560, 1997. 6






