Adding Conditional Control to Text-to-Image Diffusion Models

为文本到图像扩散模型加入条件控制

Lvmin Zhang    Anyi Rao    Maneesh Agrawala Affiliation:  Stanford University Affiliation:  {lvmin, anyirao, maneesh}@cs.stanford.edu

Abstract

摘要

We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with “zero convolutions” (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, e.g., edges, depth, segmentation, human pose, etc., with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small ($<$50k) and large ($>$1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.

我们提出 ControlNet：一种神经网络结构，可为大型预训练文生图扩散模型加入空间条件控制。ControlNet 锁定已可上线的大规模扩散模型，并复用其在数十亿图像上预训练得到的深层、稳健编码层，作为学习多样条件控制的强骨干。结构通过「零卷积」（权重初始化为零的卷积层）相连，参数从零逐步长大，确保微调阶段不会被有害噪声破坏。我们在 Stable Diffusion 上测试了多种条件控制，例如边缘、深度、分割、人体姿态等，覆盖单条件与多条件、有提示与无提示等多种设定。结果表明，ControlNet 在小规模（少于 5 万）与大规模（超过 100 万）数据上训练都足够稳健。大量实验说明，ControlNet 有望把条件控制扩展到更广的图像扩散应用。

Figure 1: Controlling Stable Diffusion with learned conditions. ControlNet allows users to add conditions like Canny edges (top), human pose (bottom), etc., to control the image generation of large pretrained diffusion models. The default results use the prompt “a high-quality, detailed, and professional image”. Users can optionally give prompts like the “chef in kitchen”.

图 1：用学到的条件控制 Stable Diffusion。ControlNet 允许用户加入 Canny 边缘（上）、人体姿态（下）等条件，以控制大型预训练扩散模型的图像生成。默认结果使用提示「a high-quality, detailed, and professional image」。用户也可另给提示，例如「chef in kitchen」。

1 Introduction

1 引言

Many of us have experienced flashes of visual inspiration that we wish to capture in a unique image. With the advent of text-to-image diffusion models [54, 62, 72], we can now create visually stunning images by typing in a text prompt. Yet, text-to-image models are limited in the control they provide over the spatial composition of the image; precisely expressing complex layouts, poses, shapes and forms can be difficult via text prompts alone. Generating an image that accurately matches our mental imagery often requires numerous trial-and-error cycles of editing a prompt, inspecting the resulting images and then re-editing the prompt.

许多人脑中会闪过想用一张独特图像留住的视觉灵感。随着文生图扩散模型[54, 62, 72]出现，我们只需键入文本提示，就能生成视觉上惊艳的图像。然而，文生图模型对图像空间构图的控制仍然有限：单靠文本很难精确表达复杂布局、姿态、形状与形体。要让生成图贴近心中画面，往往要反复改提示、看结果、再改提示。

Can we enable finer grained spatial control by letting users provide additional images that directly specify their desired image composition? In computer vision and machine learning, these additional images (e.g., edge maps, human pose skeletons, segmentation maps, depth, normals, etc.) are often treated as conditioning on the image generation process. Image-to-image translation models [34, 98] learn the mapping from conditioning images to target images. The research community has also taken steps to control text-to-image models with spatial masks [6, 20], image editing instructions [10], personalization via finetuning [21, 75], etc. While a few problems (e.g., generating image variations, inpainting) can be resolved with training-free techniques like constraining the denoising diffusion process or editing attention layer activations, a wider variety of problems like depth-to-image, pose-to-image, etc., require end-to-end learning and data-driven solutions.

能否让用户再提供若干直接指定构图的附加图像，从而实现更细粒度的空间控制？在计算机视觉与机器学习中，这类附加图像（例如边缘图、人体骨架、分割图、深度、法向等）常被当作图像生成过程的条件。图像到图像翻译模型[34, 98]学习从条件图到目标图的映射。学界也尝试用空间掩码[6, 20]、图像编辑指令[10]、微调式个性化[21, 75]等方式控制文生图模型。有些问题（例如生成图像变体、修复）可用无训练技巧解决，例如约束去噪扩散过程或编辑注意力层激活；但深度到图像、姿态到图像等更广一类问题，仍需要端到端学习与数据驱动方案。

Learning conditional controls for large text-to-image diffusion models in an end-to-end way is challenging. The amount of training data for a specific condition may be significantly smaller than the data available for general text-to-image training. For instance, the largest datasets for various specific problems (e.g., object shape/normal, human pose extraction, etc.) are usually about 100K in size, which is 50,000 times smaller than the LAION-5B [79] dataset that was used to train Stable Diffusion [82]. The direct finetuning or continued training of a large pretrained model with limited data may cause overfitting and catastrophic forgetting [31, 75]. Researchers have shown that such forgetting can be alleviated by restricting the number or rank of trainable parameters [14, 25, 31, 92]. For our problem, designing deeper or more customized neural architectures might be necessary for handling in-the-wild conditioning images with complex shapes and diverse high-level semantics.

端到端地为大型文生图扩散模型学习条件控制并不容易。特定条件的训练数据量往往远小于通用文生图训练数据。例如，物体形状/法向、人体姿态提取等问题上最大的数据集通常约 10 万量级，比训练 Stable Diffusion[82] 所用的 LAION-5B[79] 小约 5 万倍。在有限数据上直接微调或继续训练大型预训练模型，容易过拟合与灾难性遗忘[31, 75]。已有研究显示，限制可训练参数数量或秩可以缓解遗忘[14, 25, 31, 92]。对我们的问题而言，要处理形状复杂、高层语义多样的开放域条件图像，或许还需要更深或更定制的网络结构。

This paper presents ControlNet, an end-to-end neural network architecture that learns conditional controls for large pretrained text-to-image diffusion models (Stable Diffusion in our implementation). ControlNet preserves the quality and capabilities of the large model by locking its parameters, and also making a trainable copy of its encoding layers. This architecture treats the large pretrained model as a strong backbone for learning diverse conditional controls. The trainable copy and the original, locked model are connected with zero convolution layers, with weights initialized to zeros so that they progressively grow during the training. This architecture ensures that harmful noise is not added to the deep features of the large diffusion model at the beginning of training, and protects the large-scale pretrained backbone in the trainable copy from being damaged by such noise.

本文提出 ControlNet：一种端到端神经网络结构，可为大型预训练文生图扩散模型（实现上以 Stable Diffusion 为例）学习条件控制。ControlNet 通过锁定大模型参数来保留其质量与能力，同时对其编码层制作一份可训练副本。该结构把大型预训练模型当作学习多样条件控制的强骨干。可训练副本与原始锁定模型之间用零卷积相连，权重初始化为零，训练中逐步长大。这样可保证训练初期不会把有害噪声注入大扩散模型的深层特征，也保护可训练副本里那份大规模预训练骨干不被噪声破坏。

Our experiments show that ControlNet can control Stable Diffusion with various conditioning inputs, including Canny edges, Hough lines, user scribbles, human key points, segmentation maps, shape normals, depths, etc. (Figure 1). We test our approach using a single conditioning image, with or without text prompts, and we demonstrate how our approach supports the composition of multiple conditions. Additionally, we report that the training of ControlNet is robust and scalable on datasets of different sizes, and that for some tasks like depth-to-image conditioning, training ControlNets on a single NVIDIA RTX 3090Ti GPU can achieve results competitive with industrial models trained on large computation clusters. Finally, we conduct ablative studies to investigate the contribution of each component of our model, and compare our models to several strong conditional image generation baselines with user studies.

实验表明，ControlNet 可用多种条件输入控制 Stable Diffusion，包括 Canny 边缘、Hough 直线、用户涂鸦、人体关键点、分割图、形状法向、深度等（图 1）。我们测试了单张条件图、有/无文本提示，并展示如何组合多种条件。此外，我们报告 ControlNet 在不同规模数据集上训练稳健且可扩展；在深度到图像等任务上，仅用单卡 NVIDIA RTX 3090Ti 训练的 ControlNet，即可达到与大规模计算集群上训练的工业模型相当的结果。最后，我们做消融实验分析各组件贡献，并以用户研究对照若干强条件图像生成基线。

In summary, (1) we propose ControlNet, a neural network architecture that can add spatially localized input conditions to a pretrained text-to-image diffusion model via efficient finetuning, (2) we present pretrained ControlNets to control Stable Diffusion, conditioned on Canny edges, Hough lines, user scribbles, human key points, segmentation maps, shape normals, depths, and cartoon line drawings, and (3) we validate the method with ablative experiments comparing to several alternative architectures, and conduct user studies focused on several previous baselines across different tasks.

总结如下：（1）我们提出 ControlNet，一种通过高效微调为预训练文生图扩散模型加入空间局部输入条件的网络结构；（2）我们给出若干预训练 ControlNet，可按 Canny 边缘、Hough 直线、用户涂鸦、人体关键点、分割图、形状法向、深度与卡通线稿控制 Stable Diffusion；（3）我们用消融实验对照多种替代结构，并在不同任务上对先前基线做用户研究以验证方法。

2 Related Work

2 相关工作

2.1 Finetuning Neural Networks

2.1 神经网络微调

One way to finetune a neural network is to directly continue training it with the additional training data. But this approach can lead to overfitting, mode collapse, and catastrophic forgetting. Extensive research has focused on developing finetuning strategies that avoid such issues.

微调神经网络的一种做法是用额外训练数据直接继续训练。但这容易导致过拟合、模式崩塌与灾难性遗忘。大量研究致力于发展能规避这些问题的微调策略。

HyperNetwork is an approach that originated in the Natural Language Processing (NLP) community [25], with the aim of training a small recurrent neural network to influence the weights of a larger one. It has been applied to image generation with generative adversarial networks (GANs) [4, 18]. Heathen et al.  [26] and Kurumuz [43] implement HyperNetworks for Stable Diffusion [72] to change the artistic style of its output images.

超网络（HyperNetwork） 源于自然语言处理（NLP）社区[25]，目标是训练一个小的循环网络去影响更大网络的权重。它已被用于生成对抗网络（GAN）的图像生成[4, 18]。Heathen 等[26]与 Kurumuz[43]为 Stable Diffusion[72]实现了超网络，以改变输出图像的艺术风格。

Adapter methods are widely used in NLP for customizing a pretrained transformer model to other tasks by embedding new module layers into it [30, 84]. In computer vision, adapters are used for incremental learning [74] and domain adaptation [70]. This technique is often used with CLIP [66] for transferring pretrained backbone models to different tasks [23, 66, 85, 94]. More recently, adapters have yielded successful results in vision transformers [49, 50] and ViT-Adapter [14]. In concurrent work with ours, T2I-Adapter [56] adapts Stable Diffusion to external conditions.

适配器（Adapter） 方法在 NLP 中广泛用于：通过嵌入新模块层，把预训练 Transformer 定制到其他任务[30, 84]。在计算机视觉里，适配器用于增量学习[74]与域适应[70]。该技术常与 CLIP[66]结合，把预训练骨干迁移到不同任务[23, 66, 85, 94]。近期，适配器在视觉 Transformer[49, 50]与 ViT-Adapter[14]上也取得成功。与我们同期的工作 T2I-Adapter[56]则把 Stable Diffusion 适配到外部条件。

Additive Learning circumvents forgetting by freezing the original model weights and adding a small number of new parameters using learned weight masks [51, 74], pruning [52], or hard attention [80]. Side-Tuning [92] uses a side branch model to learn extra functionality by linearly blending the outputs of a frozen model and an added network, with a predefined blending weight schedule.

加性学习（Additive Learning） 通过冻结原模型权重、再用学到的权重掩码[51, 74]、剪枝[52]或硬注意力[80]加入少量新参数，来规避遗忘。Side-Tuning[92]用侧分支模型学习额外功能，按预定义混合权重日程线性混合冻结模型与新增网络的输出。

Low-Rank Adaptation (LoRA) prevents catastrophic forgetting [31] by learning the offset of parameters with low-rank matrices, based on the observation that many over-parameterized models reside in a low intrinsic dimension subspace [2, 47].

低秩适应（LoRA） 基于许多过参数化模型落在低本征维子空间的观察[2, 47]，用低秩矩阵学习参数偏移，从而防止灾难性遗忘[31]。

Zero-Initialized Layers are used by ControlNet for connecting network blocks. Research on neural networks has extensively discussed the initialization and manipulation of network weights [36, 37, 44, 45, 46, 76, 83, 95]. For example, Gaussian initialization of weights can be less risky than initializing with zeros [1]. More recently, Nichol et al. [59] discussed how to scale the initial weight of convolution layers in a diffusion model to improve the training, and their implementation of “zero_module” is an extreme case to scale weights to zero. Stability’s model cards [83] also mention the use of zero weights in neural layers. Manipulating the initial convolution weights is also discussed in ProGAN [36], StyleGAN [37], and Noise2Noise [46].

零初始化层 被 ControlNet 用来连接网络块。神经网络研究广泛讨论了权重的初始化与操控[36, 37, 44, 45, 46, 76, 83, 95]。例如，高斯初始化往往比全零初始化风险更低[1]。近期，Nichol 等[59]讨论如何缩放扩散模型中卷积层的初始权重以改善训练，其「zero_module」实现是把权重缩放到零的极端情形。Stability 的模型卡[83]也提到在神经层中使用零权重。操控初始卷积权重在 ProGAN[36]、StyleGAN[37]与 Noise2Noise[46]中亦有讨论。

2.2 Image Diffusion

2.2 图像扩散

Image Diffusion Models were first introduced by Sohl-Dickstein et al.  [81] and have been recently applied to image generation [17, 42]. The Latent Diffusion Models (LDM) [72] performs the diffusion steps in the latent image space [19], which reduces the computation cost. Text-to-image diffusion models achieve state-of-the-art image generation results by encoding text inputs into latent vectors via pretrained language models like CLIP [66]. Glide [58] is a text-guided diffusion model supporting image generation and editing. Disco Diffusion [5] processes text prompts with clip guidance. Stable Diffusion [82] is a large-scale implementation of latent diffusion [72]. Imagen [78] directly diffuses pixels using a pyramid structure without using latent images. Commercial products include DALL-E2 [62] and Midjourney [54].

图像扩散模型 最早由 Sohl-Dickstein 等[81]提出，近期被用于图像生成[17, 42]。潜空间扩散模型（LDM）[72]在潜图像空间[19]中执行扩散步骤，从而降低计算成本。文生图扩散模型借助 CLIP[66]等预训练语言模型把文本编码为潜向量，达到当前最优的图像生成效果。Glide[58]是支持生成与编辑的文本引导扩散模型。Disco Diffusion[5]用 CLIP 引导处理文本提示。Stable Diffusion[82]是潜空间扩散[72]的大规模实现。Imagen[78]采用金字塔结构直接在像素上扩散，不使用潜图像。商业产品包括 DALL-E2[62]与 Midjourney[54]。

Controlling Image Diffusion Models facilitate personalization, customization, or task-specific image generation. The image diffusion process directly provides some control over color variation [53] and inpainting [67, 7]. Text-guided control methods focus on adjusting prompts, manipulating CLIP features, and modifying cross-attention [7, 10, 20, 27, 40, 41, 58, 64, 67]. MakeAScene [20] encodes segmentation masks into tokens to control image generation. SpaText [6] maps segmentation masks into localized token embeddings. GLIGEN [48] learns new parameters in attention layers of diffusion models for grounded generating. Textual Inversion [21] and DreamBooth [75] can personalize content in the generated image by finetuning the image diffusion model using a small set of user-provided example images. Prompt-based image editing [10, 33, 86] provides practical tools to manipulate images with prompts. Voynov et al.  [88] propose an optimization method that fits the diffusion process with sketches. Concurrent works [8, 9, 32, 56] examine a wide variety of ways to control diffusion models.

控制图像扩散模型 有助于个性化、定制或任务专用图像生成。图像扩散过程本身可对颜色变化[53]与修复[67, 7]提供一定控制。文本引导控制方法侧重调整提示、操控 CLIP 特征与修改交叉注意力[7, 10, 20, 27, 40, 41, 58, 64, 67]。MakeAScene[20]把分割掩码编码为词元以控制生成。SpaText[6]把分割掩码映射为局部化词元嵌入。GLIGEN[48]在扩散模型注意力层中学习新参数以做接地生成。Textual Inversion[21]与 DreamBooth[75]可用用户提供的少量示例图像微调扩散模型，从而个性化生成内容。基于提示的图像编辑[10, 33, 86]提供了用提示操控图像的实用工具。Voynov 等[88]提出用优化使扩散过程拟合草图。同期工作[8, 9, 32, 56]考察了控制扩散模型的多种路径。

2.3 Image-to-Image Translation

2.3 图像到图像翻译

Conditional GANs [15, 34, 63, 90, 93, 97, 98, 99] and transformers [13, 19, 68] can learn the mapping between different image domains, e.g., Taming Transformer [19] is a vision transformer approach; Palette [77] is a conditional diffusion model trained from scratch; PITI [89] is a pretraining-based conditional diffusion model for image-to-image translation. Manipulating pretrained GANs can handle specific image-to-image tasks, e.g., StyleGANs can be controlled by extra encoders [71], with more applications studied in [3, 22, 38, 39, 55, 60, 65, 71].

条件 GAN[15, 34, 63, 90, 93, 97, 98, 99]与 Transformer[13, 19, 68]可以学习不同图像域之间的映射，例如 Taming Transformer[19]是一种视觉 Transformer 方法；Palette[77]是从零训练的条件扩散模型；PITI[89]是面向图像到图像翻译的预训练条件扩散模型。操控预训练 GAN 也能处理特定的图像到图像任务，例如 StyleGAN 可用额外编码器控制[71]，更多应用见[3, 22, 38, 39, 55, 60, 65, 71]。

3 Method

3 方法

Figure 2: A neural block takes a feature map $x$ as input and outputs another feature map $y$, as shown in (a). To add a ControlNet to such a block we lock the original block and create a trainable copy and connect them together using zero convolution layers, i.e., $1\times 1$ convolution with both weight and bias initialized to zero. Here $c$ is a conditioning vector that we wish to add to the network, as shown in (b).

图 2：一个神经块以特征图 $x$ 为输入并输出另一特征图 $y$，如 (a) 所示。要为该块加入 ControlNet，我们锁定原块、创建可训练副本，并用零卷积层相连，即权重与偏置均初始化为零的 $1\times 1$ 卷积。此处 $c$ 是希望加入网络的条件向量，如 (b) 所示。

ControlNet is a neural network architecture that can enhance large pretrained text-to-image diffusion models with spatially localized, task-specific image conditions. We first introduce the basic structure of a ControlNet in Section 3.1 and then describe how we apply a ControlNet to the image diffusion model Stable Diffusion [72] in Section 3.2. We elaborate on our training in Section 3.3 and detail several extra considerations during inference such as composing multiple ControlNets in Section 3.4.

ControlNet 是一种神经网络结构，可用空间局部、任务专用的图像条件增强大型预训练文生图扩散模型。我们先在第 3.1 节介绍 ControlNet 的基本结构，再在第 3.2 节说明如何把它接到图像扩散模型 Stable Diffusion[72] 上。第 3.3 节详述训练，第 3.4 节说明推理时的若干额外考虑，例如组合多个 ControlNet。

3.1 ControlNet

ControlNet injects additional conditions into the blocks of a neural network (Figure 2). Herein, we use the term network block to refer to a set of neural layers that are commonly put together to form a single unit of a neural network, e.g., resnet block, conv-bn-relu block, multi-head attention block, transformer block, etc. Suppose $\mathcal{F}(\cdot;\Theta)$ is such a trained neural block, with parameters $\Theta$, that transforms an input feature map $\bm{x}$, into another feature map $\bm{y}$ as

ControlNet 把额外条件注入神经网络的各个块（图 2）。此处我们用网络块指常被组合为网络单一单元的一组神经层，例如 resnet 块、conv-bn-relu 块、多头注意力块、Transformer 块等。设 $\mathcal{F}(\cdot;\Theta)$ 为这样一个已训练神经块，参数为 $\Theta$，它把输入特征图 $\bm{x}$ 变换为另一特征图 $\bm{y}$：

$$\vskip-3.0pt\bm{y}=\mathcal{F}(\bm{x};\Theta). \tag{1}$$

In our setting, $\bm{x}$ and $\bm{y}$ are usually 2D feature maps, i.e., $\bm{x}\in\mathbb{R}^{h\times w\times c}$ with $\{h,w,c\}$ as the height, width, and number of channels in the map, respectively (Figure 2a).

在我们的设定中，$\bm{x}$ 与 $\bm{y}$ 通常是二维特征图，即 $\bm{x}\in\mathbb{R}^{h\times w\times c}$，其中 $\{h,w,c\}$ 分别为高度、宽度与通道数（图 2a）。

To add a ControlNet to such a pre-trained neural block, we lock (freeze) the parameters $\Theta$ of the original block and simultaneously clone the block to a trainable copy with parameters $\Theta_{\text{c}}$ (Figure 2b). The trainable copy takes an external conditioning vector $\bm{c}$ as input. When this structure is applied to large models like Stable Diffusion, the locked parameters preserve the production-ready model trained with billions of images, while the trainable copy reuses such large-scale pretrained model to establish a deep, robust, and strong backbone for handling diverse input conditions.

要为这样的预训练神经块加入 ControlNet，我们锁定（冻结）原块参数 $\Theta$，同时把该块克隆为一份参数为 $\Theta_{\text{c}}$ 的可训练副本（图 2b）。可训练副本以外部分条件向量 $\bm{c}$ 为输入。当该结构接到 Stable Diffusion 这类大模型上时，锁定参数保留了用数十亿图像训练好的可上线模型，而可训练副本则复用这份大规模预训练模型，建立起处理多样输入条件的深层、稳健、强力骨干。

The trainable copy is connected to the locked model with zero convolution layers, denoted $\mathcal{Z}(\cdot;\cdot)$. Specifically, $\mathcal{Z}(\cdot;\cdot)$ is a $1\times 1$ convolution layer with both weight and bias initialized to zeros. To build up a ControlNet, we use two instances of zero convolutions with parameters $\Theta_{\text{z1}}$ and $\Theta_{\text{z2}}$ respectively. The complete ControlNet then computes

可训练副本通过记为 $\mathcal{Z}(\cdot;\cdot)$ 的零卷积层接到锁定模型。具体地，$\mathcal{Z}(\cdot;\cdot)$ 是权重与偏置均初始化为零的 $1\times 1$ 卷积层。构建 ControlNet 时，我们使用两份零卷积，参数分别为 $\Theta_{\text{z1}}$ 与 $\Theta_{\text{z2}}$。完整 ControlNet 计算为

$$\vskip-3.0pt\bm{y}_{\text{c}}=\mathcal{F}(\bm{x};\Theta)+\mathcal{Z}(\mathcal{F}(\bm{x}+\mathcal{Z}(\bm{c};\Theta_{\text{z1}});\Theta_{\text{c}});\Theta_{\text{z2}}), \tag{2}$$

where $\bm{y}_{\text{c}}$ is the output of the ControlNet block. In the first training step, since both the weight and bias parameters of a zero convolution layer are initialized to zero, both of the $\mathcal{Z}(\cdot;\cdot)$ terms in Equation (2) evaluate to zero, and

其中 $\bm{y}_{\text{c}}$ 是 ControlNet 块的输出。在训练第一步，由于零卷积的权重与偏置均初始化为零，式 (2) 中两个 $\mathcal{Z}(\cdot;\cdot)$ 项都等于零，于是

$$\vskip-3.0pt\bm{y}_{\text{c}}=\bm{y}. \tag{3}$$

In this way, harmful noise cannot influence the hidden states of the neural network layers in the trainable copy when the training starts. Moreover, since $\mathcal{Z}(\bm{c};\Theta_{\text{z1}})=\bm{0}$ and the trainable copy also receives the input image $\bm{x}$, the trainable copy is fully functional and retains the capabilities of the large, pretrained model allowing it to serve as a strong backbone for further learning. Zero convolutions protect this backbone by eliminating random noise as gradients in the initial training steps. We detail the gradient calculation for zero convolutions in supplementary materials.

这样，训练开始时有害噪声不会影响可训练副本中各神经层的隐状态。此外，由于 $\mathcal{Z}(\bm{c};\Theta_{\text{z1}})=\bm{0}$，且可训练副本同样接收输入图像 $\bm{x}$，可训练副本功能完整，并保留大预训练模型的能力，可作为进一步学习的强骨干。零卷积通过在初始训练步中消除随机噪声梯度来保护该骨干。零卷积的梯度计算详见补充材料。

Figure 3: Stable Diffusion’s U-net architecture connected with a ControlNet on the encoder blocks and middle block. The locked, gray blocks show the structure of Stable Diffusion V1.5 (or V2.1, as they use the same U-net architecture). The trainable blue blocks and the white zero convolution layers are added to build a ControlNet.

图 3：Stable Diffusion 的 U-Net 结构在编码器块与中间块上接入 ControlNet。锁定的灰色块展示 Stable Diffusion V1.5（或 V2.1，二者使用相同 U-Net 结构）的结构。蓝色可训练块与白色零卷积层被加入以构建 ControlNet。

3.2 ControlNet for Text-to-Image Diffusion

3.2 面向文生图扩散的 ControlNet

We use Stable Diffusion [72] as an example to show how ControlNet can add conditional control to a large pretrained diffusion model. Stable Diffusion is essentially a U-Net [73] with an encoder, a middle block, and a skip-connected decoder. Both the encoder and decoder contain 12 blocks, and the full model contains 25 blocks, including the middle block. Of the 25 blocks, 8 blocks are down-sampling or up-sampling convolution layers, while the other 17 blocks are main blocks that each contain 4 resnet layers and 2 Vision Transformers (ViTs). Each ViT contains several cross-attention and self-attention mechanisms. For example, in Figure 3a, the “SD Encoder Block A” contains 4 resnet layers and 2 ViTs, while the “$\times 3$” indicates that this block is repeated three times. Text prompts are encoded using the CLIP text encoder [66], and diffusion timesteps are encoded with a time encoder using positional encoding.

我们以 Stable Diffusion[72] 为例，说明 ControlNet 如何为大型预训练扩散模型加入条件控制。Stable Diffusion 本质上是一个含编码器、中间块与跳跃连接解码器的 U-Net[73]。编码器与解码器各含 12 个块，连同中间块共 25 个块。其中 8 个是下采样或上采样卷积层，其余 17 个是主块，每个含 4 个 resnet 层与 2 个视觉 Transformer（ViT）。每个 ViT 含若干交叉注意力与自注意力机制。例如在图 3a 中，「SD Encoder Block A」含 4 个 resnet 层与 2 个 ViT，而「$\times 3$」表示该块重复三次。文本提示用 CLIP 文本编码器[66]编码，扩散时间步用位置编码的时间编码器编码。

The ControlNet structure is applied to each encoder level of the U-net (Figure 3b). In particular, we use ControlNet to create a trainable copy of the 12 encoding blocks and 1 middle block of Stable Diffusion. The 12 encoding blocks are in 4 resolutions ($64\times 64,32\times 32,16\times 16,8\times 8$) with each one replicated 3 times. The outputs are added to the 12 skip-connections and 1 middle block of the U-net. Since Stable Diffusion is a typical U-net structure, this ControlNet architecture is likely to be applicable with other models.

ControlNet 结构施加于 U-Net 的每一级编码器（图 3b）。具体地，我们用 ControlNet 为 Stable Diffusion 的 12 个编码块与 1 个中间块创建可训练副本。12 个编码块对应 4 种分辨率（$64\times 64,32\times 32,16\times 16,8\times 8$），每种重复 3 次。输出被加到 U-Net 的 12 条跳跃连接与 1 个中间块上。由于 Stable Diffusion 是典型的 U-Net 结构，该 ControlNet 架构很可能也适用于其他模型。

The way we connect the ControlNet is computationally efficient — since the locked copy parameters are frozen, no gradient computation is required in the originally locked encoder for the finetuning. This approach speeds up training and saves GPU memory. As tested on a single NVIDIA A100 PCIE 40GB, optimizing Stable Diffusion with ControlNet requires only about 23% more GPU memory and 34% more time in each training iteration, compared to optimizing Stable Diffusion without ControlNet.

我们连接 ControlNet 的方式在计算上高效——锁定副本参数被冻结，微调时原锁定编码器无需计算梯度。这加快训练并节省 GPU 显存。在单卡 NVIDIA A100 PCIE 40GB 上测试，相对不带 ControlNet 优化 Stable Diffusion，带 ControlNet 时每次训练迭代约多占 23% GPU 显存、多花 34% 时间。

Image diffusion models learn to progressively denoise images and generate samples from the training domain. The denoising process can occur in pixel space or in a latent space encoded from training data. Stable Diffusion uses latent images as the training domain as working in this space has been shown to stabilize the training process [72]. Specifically, Stable Diffusion uses a pre-processing method similar to VQ-GAN [19] to convert $512\times 512$ pixel-space images into smaller $64\times 64$ latent images. To add ControlNet to Stable Diffusion, we first convert each input conditioning image (e.g., edge, pose, depth, etc.) from an input size of $512\times 512$ into a $64\times 64$ feature space vector that matches the size of Stable Diffusion. In particular, we use a tiny network $\mathcal{E}(\cdot)$ of four convolution layers with $4\times 4$ kernels and $2\times 2$ strides (activated by ReLU, using 16, 32, 64, 128, channels respectively, initialized with Gaussian weights and trained jointly with the full model) to encode an image-space condition $\bm{c}_{\text{i}}$ into a feature space conditioning vector $\bm{c}_{\text{f}}$ as,

图像扩散模型学习逐步去噪图像，并从训练域生成样本。去噪可发生在像素空间，也可发生在由训练数据编码得到的潜空间。Stable Diffusion 以潜图像为训练域，因为在该空间工作已被证明能稳定训练过程[72]。具体地，Stable Diffusion 采用类似 VQ-GAN[19] 的预处理，把 $512\times 512$ 像素空间图像转换为更小的 $64\times 64$潜图像。要为 Stable Diffusion 加入 ControlNet，我们先把每张输入条件图像（例如边缘、姿态、深度等）从 $512\times 512$ 转为与 Stable Diffusion 尺寸匹配的 $64\times 64$ 特征空间向量。具体地，我们用一个由四层卷积组成的小网络 $\mathcal{E}(\cdot)$（核 $4\times 4$、步长 $2\times 2$，ReLU 激活，通道分别为 16、32、64、128，高斯初始化并与全模型联合训练）把图像空间条件 $\bm{c}_{\text{i}}$ 编码为特征空间条件向量 $\bm{c}_{\text{f}}$：

$$\vskip-3.0pt\bm{c}_{\text{f}}=\mathcal{E}(\bm{c}_{\text{i}}). \tag{4}$$

The conditioning vector $\bm{c}_{\text{f}}$ is passed into the ControlNet.

条件向量 $\bm{c}_{\text{f}}$ 被送入 ControlNet。

3.3 Training

3.3 训练

Given an input image $\bm{z}_{0}$, image diffusion algorithms progressively add noise to the image and produce a noisy image $\bm{z}_{t}$, where $t$ represents the number of times noise is added. Given a set of conditions including time step $\bm{t}$, text prompts $\bm{c}_{t}$, as well as a task-specific condition $\bm{c}_{\text{f}}$, image diffusion algorithms learn a network $\epsilon_{\theta}$ to predict the noise added to the noisy image $\bm{z}_{t}$ with

给定输入图像 $\bm{z}_{0}$，图像扩散算法逐步向图像加噪并得到噪声图像 $\bm{z}_{t}$，其中 $t$ 表示加噪次数。给定时间步 $\bm{t}$、文本提示 $\bm{c}_{t}$ 以及任务专用条件 $\bm{c}_{\text{f}}$ 等条件集合，图像扩散算法学习网络 $\epsilon_{\theta}$，以预测加到噪声图像 $\bm{z}_{t}$ 上的噪声：

$$\vskip-3.0pt\mathcal{L}=\mathbb{E}_{\bm{z}_{0},\bm{t},\bm{c}_{t},\bm{c}_{\text{f}},\epsilon\sim\mathcal{N}(0,1)}\Big[\|\epsilon-\epsilon_{\theta}(\bm{z}_{t},\bm{t},\bm{c}_{t},\bm{c}_{\text{f}}))\|_{2}^{2}\Big], \tag{5}$$

where $\mathcal{L}$ is the overall learning objective of the entire diffusion model. This learning objective is directly used in finetuning diffusion models with ControlNet.

其中 $\mathcal{L}$ 是整个扩散模型的总体学习目标。用 ControlNet 微调扩散模型时直接使用该目标。

In the training process, we randomly replace 50% text prompts $\bm{c}_{t}$ with empty strings. This approach increases ControlNet’s ability to directly recognize semantics in the input conditioning images (e.g., edges, poses, depth, etc.) as a replacement for the prompt.

训练中，我们以 50% 概率把文本提示 $\bm{c}_{t}$ 随机替换为空字符串。这样可增强 ControlNet 直接从输入条件图像（例如边缘、姿态、深度等）中识别语义的能力，从而替代提示。

Figure 4: The sudden convergence phenomenon. Due to the zero convolutions, ControlNet always predicts high-quality images during the entire training. At a certain step in the training process (e.g., the 6133 steps marked in bold), the model suddenly learns to follow the input condition.

图 4：「突然收敛」现象。由于零卷积，ControlNet 在整个训练过程中始终能预测高质量图像。在训练的某一步（例如加粗标出的第 6133 步），模型突然学会跟随输入条件。

Figure 5: Effect of Classifier-Free Guidance (CFG) and the proposed CFG Resolution Weighting (CFG-RW).

图 5：无分类器引导（CFG）与所提 CFG 分辨率加权（CFG-RW）的效果。

Figure 6: Composition of multiple conditions. We present the application to use depth and pose simultaneously.

图 6：多条件组合。我们展示同时使用深度与姿态的应用。

During the training process, since zero convolutions do not add noise to the network, the model should always be able to predict high-quality images. We observe that the model does not gradually learn the control conditions but abruptly succeeds in following the input conditioning image; usually in less than 10K optimization steps. As shown in Figure 4, we call this the “sudden convergence phenomenon”.

训练过程中，由于零卷积不会向网络加入噪声，模型应始终能预测高质量图像。我们观察到，模型并非逐渐学会控制条件，而是突然成功跟随输入条件图像；通常发生在不到 1 万次优化步之内。如图 4 所示，我们称之为「突然收敛现象」。

Sketch	Normal map	Depth map	Canny[11] edge	M-LSD[24] line	HED[91] edge	ADE20k[96] seg.	Human pose

Figure 7: Controlling Stable Diffusion with various conditions without prompts. The top row is input conditions, while all other rows are outputs. We use the empty string as input prompts. All models are trained with general-domain data. The model has to recognize semantic contents in the input condition images to generate images.

Sketch	Normal map	Depth map	Canny[11] edge	M-LSD[24] line	HED[91] edge	ADE20k[96] seg.	Human pose

图 7：在无提示情况下用多种条件控制 Stable Diffusion。顶行是输入条件，其余行是输出。输入提示使用空字符串。所有模型均在通用域数据上训练。模型必须识别输入条件图像中的语义内容才能生成图像。

3.4 Inference

3.4 推理

We can further control how the extra conditions of ControlNet affect the denoising diffusion process in several ways.

我们还可以用若干方式进一步控制 ControlNet 的额外条件如何影响去噪扩散过程。

Classifier-free guidance resolution weighting. Stable Diffusion depends on a technique called Classifier-Free Guidance (CFG) [29] to generate high-quality images. CFG is formulated as $\epsilon_{\text{prd}}=\epsilon_{\text{uc}}+\beta_{\text{cfg}}(\epsilon_{\text{c}}-\epsilon_{\text{uc}})$ where $\epsilon_{\text{prd}}$, $\epsilon_{\text{uc}}$, $\epsilon_{\text{c}}$, $\beta_{\text{cfg}}$ are the model’s final output, unconditional output, conditional output, and a user-specified weight respectively. When a conditioning image is added via ControlNet, it can be added to both $\epsilon_{\text{uc}}$ and $\epsilon_{\text{c}}$, or only to the $\epsilon_{\text{c}}$. In challenging cases, e.g., when no prompts are given, adding it to both $\epsilon_{\text{uc}}$ and $\epsilon_{\text{c}}$ will completely remove CFG guidance (Figure 5b); using only $\epsilon_{\text{c}}$ will make the guidance very strong (Figure 5c). Our solution is to first add the conditioning image to $\epsilon_{\text{c}}$ and then multiply a weight $w_{i}$ to each connection between Stable Diffusion and ControlNet according to the resolution of each block $w_{i}=64/h_{i}$, where $h_{i}$ is the size of $i^{\text{th}}$ block, e.g., $h_{1}=8,h_{2}=16,...,h_{13}=64$. By reducing the CFG guidance strength , we can achieve the result shown in Figure 5d, and we call this CFG Resolution Weighting.

无分类器引导分辨率加权。 Stable Diffusion 依赖称为无分类器引导（CFG）[29] 的技术生成高质量图像。CFG 写作 $\epsilon_{\text{prd}}=\epsilon_{\text{uc}}+\beta_{\text{cfg}}(\epsilon_{\text{c}}-\epsilon_{\text{uc}})$，其中 $\epsilon_{\text{prd}}$、$\epsilon_{\text{uc}}$、$\epsilon_{\text{c}}$、$\beta_{\text{cfg}}$ 分别为模型最终输出、无条件输出、条件输出与用户指定权重。当通过 ControlNet 加入条件图像时，既可把它同时加到 $\epsilon_{\text{uc}}$ 与 $\epsilon_{\text{c}}$，也可只加到 $\epsilon_{\text{c}}$。在困难情形下，例如不给提示时，同时加到两者会完全去掉 CFG 引导（图 5b）；只使用 $\epsilon_{\text{c}}$ 则会使引导过强（图 5c）。我们的做法是先把条件图像加到 $\epsilon_{\text{c}}$，再按各块分辨率给 Stable Diffusion 与 ControlNet 之间的每条连接乘以权重 $w_{i}=64/h_{i}$，其中 $h_{i}$ 是第 $i$ 个块的尺寸，例如 $h_{1}=8,h_{2}=16,...,h_{13}=64$。通过降低 CFG 引导强度，可以得到图 5d 的结果，我们称之为 CFG 分辨率加权。

Method	Result Quality $\uparrow$	Condition Fidelity $\uparrow$
PITI 89	1.10 $\pm$ 0.05	1.02 $\pm$ 0.01
Sketch-Guided [88] ($\beta=1.6$)	3.21 $\pm$ 0.62	2.31 $\pm$ 0.57
Sketch-Guided [88] ($\beta=3.2$)	2.52 $\pm$ 0.44	3.28 $\pm$ 0.72
ControlNet-lite	3.93 $\pm$ 0.59	4.09 $\pm$ 0.46
ControlNet	4.22 $\pm$ 0.43	4.28 $\pm$ 0.45

Table 1: Average User Ranking (AUR) of result quality and condition fidelity. We report the user preference ranking (1 to 5 indicates worst to best) of different methods.

Method	Result Quality $\uparrow$	Condition Fidelity $\uparrow$
PITI 89	1.10 $\pm$ 0.05	1.02 $\pm$ 0.01
Sketch-Guided [88] ($\beta=1.6$)	3.21 $\pm$ 0.62	2.31 $\pm$ 0.57
Sketch-Guided [88] ($\beta=3.2$)	2.52 $\pm$ 0.44	3.28 $\pm$ 0.72
ControlNet-lite	3.93 $\pm$ 0.59	4.09 $\pm$ 0.46
ControlNet	4.22 $\pm$ 0.43	4.28 $\pm$ 0.45

表 1：结果质量与条件保真度的平均用户排名（AUR）。我们报告不同方法的用户偏好排名（1 到 5 表示从差到好）。

Composing multiple ControlNets. To apply multiple conditioning images (e.g., Canny edges, and pose) to a single instance of Stable Diffusion, we can directly add the outputs of the corresponding ControlNets to the Stable Diffusion model (Figure 6). No extra weighting or linear interpolation is necessary for such composition.

组合多个 ControlNet。 要把多张条件图像（例如 Canny 边缘与姿态）施加到同一份 Stable Diffusion 实例，可以直接把对应 ControlNet 的输出加到 Stable Diffusion 模型上（图 6）。这种组合无需额外加权或线性插值。

Figure 8: Ablative study of different architectures on a sketch condition and different prompt settings. For each setting, we show a random batch of 6 samples without cherry-picking. Images are at 512 $\times$ 512 and best viewed when zoomed in. The green “conv” blocks on the left are standard convolution layers initialized with Gaussian weights.

图 8：在草图条件与不同提示设定下对不同结构的消融研究。每种设定展示未经挑选的随机 6 张样本。图像分辨率为 512 $\times$ 512，放大后观看效果更佳。左侧绿色「conv」块是用高斯权重初始化的标准卷积层。

4 Experiments

4 实验

We implement ControlNets with Stable Diffusion to test various conditions, including Canny Edge [11], Depth Map [69], Normal Map [87], M-LSD lines [24], HED soft edge [91], ADE20K segmentation [96], Openpose [12], and user sketches. See also the supplementary material for examples of each conditioning along with detailed training and inference parameters.

我们在 Stable Diffusion 上实现 ControlNet，测试多种条件，包括 Canny 边缘[11]、深度图[69]、法向图[87]、M-LSD 直线[24]、HED 软边缘[91]、ADE20K 分割[96]、Openpose[12] 以及用户草图。各类条件示例与详细训练、推理参数见补充材料。

4.1 Qualitative Results

4.1 定性结果

Figure 1 shows the generated images in several prompt settings. Figure 7 shows our results with various conditions without prompts, where the ControlNet robustly interprets content semantics in diverse input conditioning images.

图 1 展示了若干提示设定下的生成图像。图 7 展示无提示时多种条件的结果，ControlNet 能稳健地解释多样输入条件图像中的内容语义。

4.2 Ablative Study

4.2 消融研究

Figure 9: Comparison to previous methods.We present the qualitative comparisons to PITI [89], Sketch-Guided Diffusion [88], and Taming Transformers [19].

图 9：与先前方法的对比。我们给出相对 PITI[89]、Sketch-Guided Diffusion[88] 与 Taming Transformers[19] 的定性比较。

We study alternative structures of ControlNets by (1) replacing the zero convolutions with standard convolution layers initialized with Gaussian weights, and (2) replacing each block’s trainable copy with one single convolution layer, which we call ControlNet-lite. See also the supplementary material for the full details of these ablative structures.

我们通过以下方式研究 ControlNet 的替代结构：（1）用高斯权重初始化的标准卷积层替换零卷积；（2）把每个块的可训练副本替换为单个卷积层，称为 ControlNet-lite。这些消融结构的完整细节见补充材料。

We present 4 prompt settings to test with possible behaviors of real-world users: (1) no prompt; (2) insufficient prompts that do not fully cover objects in conditioning images, e.g., the default prompt of this paper “a high-quality, detailed, and professional image”; (3) conflicting prompts that change the semantics of conditioning images; (4) perfect prompts that describe necessary content semantics, e.g., “a nice house”. Figure 8a shows that ControlNet succeeds in all 4 settings. The lightweight ControlNet-lite (Figure 8c) is not strong enough to interpret the conditioning images and fails in the insufficient and no prompt conditions. When zero convolutions are replaced, the performance of ControlNet drops to about the same as ControlNet-lite, indicating that the pretrained backbone of the trainable copy is destroyed during finetuning (Figure 8b).

我们给出 4 种提示设定，以测试真实用户可能出现的行为：（1）无提示；（2）不足以完全覆盖条件图像中物体的提示，例如本文默认提示「a high-quality, detailed, and professional image」；（3）改变条件图像语义的冲突提示；（4）描述必要内容语义的完美提示，例如「a nice house」。图 8a 显示 ControlNet 在全部 4 种设定下都成功。轻量的 ControlNet-lite（图 8c）不足以解释条件图像，在提示不足与无提示条件下失败。当替换掉零卷积后，ControlNet 的表现降到与 ControlNet-lite 大致相当，说明可训练副本中的预训练骨干在微调中被破坏（图 8b）。

ADE20K (GT)	VQGAN [19]	LDM [72]	PITI [89]	ControlNet-lite	ControlNet
0.58 $\pm$ 0.10	0.21 $\pm$ 0.15	0.31 $\pm$ 0.09	0.26 $\pm$ 0.16	0.32 $\pm$ 0.12	0.35 $\pm$ 0.14

Table 2: Evaluation of semantic segmentation label reconstruction (ADE20K) with Intersection over Union (IoU $\uparrow$).

ADE20K (GT)	VQGAN [19]	LDM [72]	PITI [89]	ControlNet-lite	ControlNet
0.58 $\pm$ 0.10	0.21 $\pm$ 0.15	0.31 $\pm$ 0.09	0.26 $\pm$ 0.16	0.32 $\pm$ 0.12	0.35 $\pm$ 0.14

表 2：用交并比（IoU $\uparrow$）评估语义分割标签重建（ADE20K）。

4.3 Quantitative Evaluation

4.3 定量评估

User study. We sample 20 unseen hand-drawn sketches, and then assign each sketch to 5 methods: PITI [89]’s sketch model, Sketch-Guided Diffusion (SGD) [88] with default edge-guidance scale ($\beta=1.6$), SGD [88] with relatively high edge-guidance scale ($\beta=3.2$), the aforementioned ControlNet-lite, and ControlNet. We invited 12 users to rank these 20 groups of 5 results individually in terms of “the quality of displayed images” and “the fidelity to the sketch”. In this way, we obtain 100 rankings for result quality and 100 for condition fidelity. We use the Average Human Ranking (AHR) as a preference metric where users rank each result on a scale of 1 to 5 (lower is worse). The average rankings are shown in Table 1.

用户研究。 我们采样 20 张未见过的手绘草图，再把每张草图分给 5 种方法：PITI[89] 的草图模型、默认边缘引导尺度（$\beta=1.6$）的 Sketch-Guided Diffusion（SGD）[88]、较高边缘引导尺度（$\beta=3.2$）的 SGD[88]、前述 ControlNet-lite，以及 ControlNet。我们邀请 12 位用户分别按*「所显示图像的质量」与「对草图的保真度」*给这 20 组、每组 5 个结果排名。由此得到 100 次结果质量排名与 100 次条件保真度排名。我们用平均人类排名（AHR）作为偏好指标，用户按 1 到 5 分给每个结果打分（越低越差）。平均排名见表 1。

Comparison to industrial models. Stable Diffusion V2 Depth-to-Image (SDv2-D2I) [83] is trained with a large-scale NVIDIA A100 cluster, thousands of GPU hours, and more than 12M training images. We train a ControlNet for the SD V2 with the same depth conditioning but only use 200k training samples, one single NVIDIA RTX 3090Ti, and 5 days of training. We use 100 images generated by each SDv2-D2I and ControlNet to teach 12 users to distinguish the two methods. Afterwards, we generate 200 images and ask the users to tell which model generated each image. The average precision of the users is $0.52\pm 0.17$, indicating that the two method yields almost indistinguishable results.

与工业模型对比。 Stable Diffusion V2 Depth-to-Image（SDv2-D2I）[83] 在大规模 NVIDIA A100 集群上训练，耗费数千 GPU 小时与超过 1200 万训练图像。我们为 SD V2 训练相同深度条件的 ControlNet，但仅用 20 万训练样本、单卡 NVIDIA RTX 3090Ti 与 5 天训练。我们用 SDv2-D2I 与 ControlNet 各生成的 100 张图像教 12 位用户区分两种方法；随后再生成 200 张图像，请用户判断每张由哪个模型生成。用户平均精度为 $0.52\pm 0.17$，表明两种方法几乎难以区分。

Method	FID $\downarrow$	CLIP-score $\uparrow$	CLIP-aes. $\uparrow$
Stable Diffusion	6.09	0.26	6.32
VQGAN 19*	26.28	0.17	5.14
LDM 72*	25.35	0.18	5.15
PITI 89	19.74	0.20	5.77
ControlNet-lite	17.92	0.26	6.30
ControlNet	15.27	0.26	6.31

Table 3: Evaluation for image generation conditioned by semantic segmentation. We report FID, CLIP text-image score, and CLIP aesthetic scores for our method and other baselines. We also report the performance of Stable Diffusion without segmentation conditions. Methods marked with “*” are trained from scratch.

Method	FID $\downarrow$	CLIP-score $\uparrow$	CLIP-aes. $\uparrow$
Stable Diffusion	6.09	0.26	6.32
VQGAN 19*	26.28	0.17	5.14
LDM 72*	25.35	0.18	5.15
PITI 89	19.74	0.20	5.77
ControlNet-lite	17.92	0.26	6.30
ControlNet	15.27	0.26	6.31

表 3：语义分割条件图像生成的评估。我们报告本方法与其他基线的 FID、CLIP 文图分数与 CLIP 美学分数，并报告无分割条件时 Stable Diffusion 的表现。标「*」的方法为从零训练。

Condition reconstruction and FID score. We use the test set of ADE20K [96] to evaluate the conditioning fidelity. The state-of-the-art segmentation method OneFormer [35] achieves an Intersection-over-Union (IoU) with 0.58 on the ground-truth set. We use different methods to generate images with ADE20K segmentations and then apply OneFormer to detect the segmentations again to compute the reconstructed IoUs (Table 2). Besides, we use Frechet Inception Distance (FID) [28] to measure the distribution distance over randomly generated 512$\times$512 image sets using different segmentation-conditioned methods, as well as text-image CLIP scores [66] and CLIP aesthetic score [79] in Table 3. See also the supplementary material for detailed settings.

条件重建与 FID 分数。 我们用 ADE20K[96] 测试集评估条件保真度。当前最优分割方法 OneFormer[35] 在真值集上交并比（IoU）为 0.58。我们用不同方法按 ADE20K 分割生成图像，再用 OneFormer 检测分割并计算重建 IoU（表 2）。此外，我们用 Fréchet Inception Distance（FID）[28] 衡量不同分割条件方法随机生成的 512$\times$512 图像集之间的分布距离，并在表 3 中报告文图 CLIP 分数[66]与 CLIP 美学分数[79]。详细设定见补充材料。

4.4 Comparison to Previous Methods

4.4 与先前方法的对比

Figure 9 presents a visual comparison of baselines and our method (Stable Diffusion + ControlNet). Specifically, we show the results of PITI [89], Sketch-Guided Diffusion [88], and Taming Transformers [19]. (Note that the backbone of PITI is OpenAI GLIDE [57] that have different visual quality and performance.) We observe that ControlNet can robustly handle diverse conditioning images and achieves sharp and clean results.

图 9 给出基线与本方法（Stable Diffusion + ControlNet）的视觉对比。具体展示 PITI[89]、Sketch-Guided Diffusion[88] 与 Taming Transformers[19] 的结果。（注意 PITI 的骨干是 OpenAI GLIDE[57]，视觉质量与性能不同。）我们观察到 ControlNet 能稳健处理多样条件图像，并得到清晰干净的结果。

Figure 10: The influence of different training dataset sizes. See also the supplementary material for extended examples.

图 10：不同训练数据集规模的影响。扩展示例见补充材料。

Figure 11: Interpreting contents. If the input is ambiguous and the user does not mention object contents in prompts, the results look like the model tries to interpret input shapes.

图 11：解释内容。若输入含糊且用户未在提示中提及物体内容，结果看起来像模型在试图解释输入形状。

Figure 12: Transfer pretrained ControlNets to community models [16, 61] without training the neural networks again.

图 12：把预训练 ControlNet 迁移到社区模型[16, 61]，无需再次训练神经网络。

4.5 Discussion

4.5 讨论

Influence of training dataset sizes. We demonstrate the robustness of the ControlNet training in Figure 10. The training does not collapse with limited 1k images, and allows the model to generate a recognizable lion. The learning is scalable when more data is provided.

训练数据集规模的影响。 我们在图 10 中展示 ControlNet 训练的稳健性。即便仅有 1 千张图像，训练也不会崩塌，模型仍能生成可辨认的狮子。提供更多数据时，学习可扩展。

Capability to interpret contents. We showcase ControlNet’s capability to capture the semantics from input conditioning images in Figure 11.

解释内容的能力。 我们在图 11 中展示 ControlNet 从输入条件图像中捕捉语义的能力。

Transferring to community models. Since ControlNets do not change the network topology of pretrained SD models, it can be directly applied to various models in the stable diffusion community, such as Comic Diffusion [61] and Protogen 3.4 [16], in Figure 12.

迁移到社区模型。 由于 ControlNet 不改变预训练 SD 模型的网络拓扑，它可直接施加到 Stable Diffusion 社区中的多种模型，例如 Comic Diffusion[61] 与 Protogen 3.4[16]，见图 12。

5 Conclusion

5 结论

ControlNet is a neural network structure that learns conditional control for large pretrained text-to-image diffusion models. It reuses the large-scale pretrained layers of source models to build a deep and strong encoder to learn specific conditions. The original model and trainable copy are connected via “zero convolution” layers that eliminate harmful noise during training. Extensive experiments verify that ControlNet can effectively control Stable Diffusion with single or multiple conditions, with or without prompts. Results on diverse conditioning datasets show that the ControlNet structure is likely to be applicable to a wider range of conditions, and facilitate relevant applications.

ControlNet 是一种为大型预训练文生图扩散模型学习条件控制的神经网络结构。它复用源模型的大规模预训练层，构建深层、强力的编码器以学习特定条件。原模型与可训练副本通过「零卷积」层相连，在训练中消除有害噪声。大量实验验证，ControlNet 能在单条件或多条件、有提示或无提示下有效控制 Stable Diffusion。在多样条件数据集上的结果表明，ControlNet 结构很可能适用于更广的条件范围，并推动相关应用。

Acknowledgment

致谢

This work was partially supported by the Stanford Institute for Human-Centered AI and the Brown Institute for Media Innovation.

本工作部分得到 Stanford Institute for Human-Centered AI 与 Brown Institute for Media Innovation 的支持。

References

参考文献

[1] Sadia Afrin. Weight initialization in neural network, inspired by andrew ng, https://medium.com/@safrin1128/weight-initialization-in-neural-network-inspired-by-andrew-ng-e0066dc4a566, 2020.

[2] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing , pages 7319–7328, Online, Aug. 2021. Association for Computational Linguistics.

[3] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or. Only a matter of style: Age transformation using a style-based regression model. ACM Transactions on Graphics (TOG) , 40(4), 2021.

[4] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18511–18521, 2022.

[5] Alembics. Disco diffusion, https://github.com/alembics/disco-diffusion, 2022.

[6] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. arXiv preprint arXiv:2211.14305 , 2022.

[7] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18208–18218, 2022.

[8] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08113 , 2023.

[9] Dina Bashkirova, Jose Lezama, Kihyuk Sohn, Kate Saenko, and Irfan Essa. Masksketch: Unpaired structure-guided masked image generation. arXiv preprint arXiv:2302.05496 , 2023.

[10] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022.

[11] John Canny. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , (6):679–698, 1986.

[12] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2019.

[13] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12299–12310, 2021.

[14] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. International Conference on Learning Representations , 2023.

[15] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8789–8797, 2018.

[16] darkstorm2150. Protogen x3.4 (photorealism) official release, https://civitai.com/models/3666/protogen-x34-photorealism-official-release, 2022.

[17] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780–8794, 2021.

[18] Tan M. Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua. Hyperinverter: Improving stylegan inversion via hypernetwork. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11389–11398, 2022.

[19] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12873–12883, 2021.

[20] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision (ECCV) , pages 89–106. Springer, 2022.

[21] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

[22] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG) , 41(4):1–13, 2022.

[23] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 , 2021.

[24] Geonmo Gu, Byungsoo Ko, SeoungHyun Go, Sung-Hyun Lee, Jingeun Lee, and Minchul Shin. Towards light-weight and real-time line segment detection. In Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

[25] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations , 2017.

[26] Heathen. Hypernetwork style training, a tiny guide, stable-diffusion-webui, https://github.com/automatic1111/stable-diffusion-webui/discussions/2670, 2022.

[27] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

[28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

[29] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022.

[30] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790–2799, 2019.

[31] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

[32] Lianghua Huang, Di Chen, Yu Liu, Shen Yujun, Deli Zhao, and Zhou Jingren. Composer: Creative and controllable image synthesis with composable conditions. 2023.

[33] Nisha Huang, Fan Tang, Weiming Dong, Tong-Yee Lee, and Changsheng Xu. Region-aware diffusion for zero-shot text-driven image editing. arXiv preprint arXiv:2302.11797 , 2023.

[34] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1125–1134, 2017.

[35] Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer: One Transformer to Rule Universal Image Segmentation. 2023.

[36] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. International Conference on Learning Representations , 2018.

[37] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4401–4410, 2019.

[38] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. IEEE Transactions on Pattern Analysis , 2021.

[39] Oren Katzir, Vicky Perepelook, Dani Lischinski, and Daniel Cohen-Or. Multi-level latent space structuring for generative control. arXiv preprint arXiv:2202.05910 , 2022.

[40] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 , 2022.

[41] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2426–2435, 2022.

[42] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in Neural Information Processing Systems , 34:21696–21707, 2021.

[43] Kurumuz. Novelai improvements on stable diffusion, https://blog.novelai.net/novelai-improvements-on-stable-diffusion-e10d38db82ac, 2022.

[44] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436–444, May 2015.

[45] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278–2324, 1998.

[46] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. Proceedings of the 35th International Conference on Machine Learning , 2018.

[47] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. International Conference on Learning Representations , 2018.

[48] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. 2023.

[49] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. arXiv preprint arXiv:2203.16527 , 2022.

[50] Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429 , 2021.

[51] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In European Conference on Computer Vision (ECCV) , pages 67–82, 2018.

[52] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7765–7773, 2018.

[53] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2021.

[54] Midjourney. https://www.midjourney.com/, 2023.

[55] Ron Mokady, Omer Tov, Michal Yarom, Oran Lang, Inbar Mosseri, Tali Dekel, Daniel Cohen-Or, and Michal Irani. Self-distilled stylegan: Towards generation from internet photos. In ACM SIGGRAPH 2022 Conference Proceedings , pages 1–9, 2022.

[56] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023.

[57] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. CoRR , 2021.

[58] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. 2022.

[59] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162–8171. PMLR, 2021.

[60] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. Mystyle: A personalized generative prior. arXiv preprint arXiv:2203.17272 , 2022.

[61] ogkalu. Comic-diffusion v2, trained on 6 styles at once, https://huggingface.co/ogkalu/comic-diffusion, 2022.

[62] OpenAI. Dall-e-2, https://openai.com/product/dall-e-2, 2023.

[63] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2337–2346, 2019.

[64] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. arXiv preprint arXiv:2302.03027 , 2023.

[65] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2085–2094, October 2021.

[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748–8763. PMLR, 2021.

[67] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

[68] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821–8831. PMLR, 2021.

[69] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(3):1623–1637, 2020.

[70] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi-domain deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8119–8127, 2018.

[71] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

[72] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684–10695, 2022.

[73] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention MICCAI International Conference , pages 234–241, 2015.

[74] Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 42(3):651–663, 2018.

[75] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242 , 2022.

[76] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature , 323(6088):533–536, Oct. 1986.

[77] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings , SIGGRAPH ’22, New York, NY, USA, 2022. Association for Computing Machinery.

[78] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

[79] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

[80] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning , pages 4548–4557. PMLR, 2018.

[81] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256–2265. PMLR, 2015.

[82] Stability. Stable diffusion v1.5 model card, https://huggingface.co/runwayml/stable-diffusion-v1-5, 2022.

[83] Stability. Stable diffusion v2 model card, stable-diffusion-2-depth, https://huggingface.co/stabilityai/stable-diffusion-2-depth, 2022.

[84] Asa Cooper Stickland and Iain Murray. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In International Conference on Machine Learning , pages 5986–5995, 2019.

[85] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. arXiv preprint arXiv:2112.06825 , 2021.

[86] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572 , 2022.

[87] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv:1908.00463 , 2019.

[88] Andrey Voynov, Kfir Abernan, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. 2022.

[89] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. 2022.

[90] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8798–8807, 2018.

[91] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 1395–1403, 2015.

[92] Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, and Jitendra Malik. Side-tuning: Network adaptation via additive side networks. In European Conference on Computer Vision (ECCV) , pages 698–714. Springer, 2020.

[93] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar-based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5143–5153, 2020.

[94] Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930 , 2021.

[95] Jiawei Zhao, Florian Schäfer, and Anima Anandkumar. Zero initialization: Initializing residual networks with only zeros and ones. arXiv , 2021.

[96] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 633–641, 2017.

[97] Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full-resolution correspondence learning for image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11465–11475, 2021.

[98] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on , 2017.

[99] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances in Neural Information Processing Systems , 30, 2017.