Denoising Diffusion Implicit Models

Jiaming Song    Chenlin Meng & Stefano Ermon Affiliation:  Stanford University Email:  {tsong,chenlin,ermon}@cs.stanford.edu

Abstract

摘要

Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps in order to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a particular Markovian diffusion process. We generalize DDPMs via a class of non-Markovian diffusion processes that lead to the same training objective. These non-Markovian processes can correspond to generative processes that are deterministic, giving rise to implicit models that produce high quality samples much faster. We empirically demonstrate that DDIMs can produce high quality samples $10\times$ to $50\times$ faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, perform semantically meaningful image interpolation directly in the latent space, and reconstruct observations with very low error.

去噪扩散概率模型（DDPM）无需对抗训练即可生成高质量图像，但采样需模拟很长的马尔可夫链。为加速采样，我们提出去噪扩散隐式模型（DDIM）：一类更高效的迭代隐式概率模型，训练过程与 DDPM 相同。在 DDPM 中，生成过程定义为某特定马尔可夫扩散过程的逆。我们通过一类非马尔可夫扩散过程推广 DDPM，它们导向相同的训练目标。这些非马尔可夫过程可对应确定性的生成过程，从而得到能更快产出高质量样本的隐式模型。经验表明，相对 DDPM，DDIM 在墙钟时间上可快 $10\times$ 到 $50\times$ 产出高质量样本，允许在计算与样本质量之间权衡，可在潜空间直接做语义有意义的图像插值，并能以很低误差重建观测。

1 Introduction

1 引言

Deep generative models have demonstrated the ability to produce high quality samples in many domains (Karras et al. 2020; van den Oord et al. 2016a). In terms of image generation, generative adversarial networks (GANs, Goodfellow et al. 2014) currently exhibits higher sample quality than likelihood-based methods such as variational autoencoders (Kingma & Welling 2013), autoregressive models (van den Oord et al. 2016b) and normalizing flows (Rezende & Mohamed 2015; Dinh et al. 2016). However, GANs require very specific choices in optimization and architectures in order to stabilize training (Arjovsky et al. 2017; Gulrajani et al. 2017; Karras et al. 2018; Brock et al. 2018), and could fail to cover modes of the data distribution (Zhao et al. 2018).

深度生成模型已在许多领域展示高质量采样能力 (Karras et al. 2020; van den Oord et al. 2016a)。图像生成方面，生成对抗网络（GAN, Goodfellow et al. 2014）目前样本质量通常高于变分自编码器 (Kingma & Welling 2013)、自回归模型 (van den Oord et al. 2016b) 与归一化流 (Rezende & Mohamed 2015; Dinh et al. 2016) 等基于似然的方法。但 GAN 需要非常特定的优化与结构选择才能稳住训练 (Arjovsky et al. 2017; Gulrajani et al. 2017; Karras et al. 2018; Brock et al. 2018)，也可能覆盖不到数据分布的全部模态 (Zhao et al. 2018)。

Recent works on iterative generative models (Bengio et al. 2014), such as denoising diffusion probabilistic models (DDPM, Ho et al. 2020) and noise conditional score networks (NCSN, Song & Ermon 2019) have demonstrated the ability to produce samples comparable to that of GANs, without having to perform adversarial training. To achieve this, many denoising autoencoding models are trained to denoise samples corrupted by various levels of Gaussian noise. Samples are then produced by a Markov chain which, starting from white noise, progressively denoises it into an image. This generative Markov Chain process is either based on Langevin dynamics (Song & Ermon 2019) or obtained by reversing a forward diffusion process that progressively turns an image into noise (Sohl-Dickstein et al. 2015).

关于迭代生成模型的近期工作 (Bengio et al. 2014)，例如去噪扩散概率模型（DDPM, Ho et al. 2020）与噪声条件分数网络（NCSN, Song & Ermon 2019），已表明无需对抗训练也能产出可比 GAN 的样本。做法是训练许多去噪自编码模型，对被不同程度高斯噪声腐蚀的样本去噪；再从白噪声出发，用一条马尔可夫链逐步去噪成图像。该生成马尔可夫链过程要么基于 Langevin 动力学 (Song & Ermon 2019)，要么通过对逐步把图像变成噪声的前向扩散过程取逆得到 (Sohl-Dickstein et al. 2015)。

A critical drawback of these models is that they require many iterations to produce a high quality sample. For DDPMs, this is because that the generative process (from noise to data) approximates the reverse of the forward diffusion process (from data to noise), which could have thousands of steps; iterating over all the steps is required to produce a single sample, which is much slower compared to GANs, which only needs one pass through a network. For example, it takes around 20 hours to sample 50k images of size $32\times 32$ from a DDPM, but less than a minute to do so from a GAN on a Nvidia 2080 Ti GPU. This becomes more problematic for larger images as sampling 50k images of size $256\times 256$ could take nearly $1000$ hours on the same GPU.

这类模型的关键缺点是：要产出高质量样本需要很多次迭代。对 DDPM 而言，是因为生成过程（噪声到数据）近似前向扩散过程（数据到噪声）的逆，后者可能有数千步；要得到单个样本就必须遍历全部步骤，远慢于只需一次网络前向的 GAN。例如，在 Nvidia 2080 Ti 上，从 DDPM 采样 5 万张 $32\times 32$ 图像约需 20 小时，而 GAN 不到一分钟。图像更大时更糟：同 GPU 上采 5 万张 $256\times 256$ 可能接近 $1000$ 小时。

To close this efficiency gap between DDPMs and GANs, we present denoising diffusion implicit models (DDIMs). DDIMs are implicit probabilistic models (Mohamed & Lakshminarayanan 2016) and are closely related to DDPMs, in the sense that they are trained with the same objective function. In Section 3, we generalize the forward diffusion process used by DDPMs, which is Markovian, to non-Markovian ones, for which we are still able to design suitable reverse generative Markov chains. We show that the resulting variational training objectives have a shared surrogate objective, which is exactly the objective used to train DDPM. Therefore, we can freely choose from a large family of generative models using the same neural network simply by choosing a different, non-Markovian diffusion process (Section 4.1) and the corresponding reverse generative Markov Chain. In particular, we are able to use non-Markovian diffusion processes which lead to ”short” generative Markov chains (Section 4.2) that can be simulated in a small number of steps. This can massively increase sample efficiency only at a minor cost in sample quality.

为缩小 DDPM 与 GAN 之间的效率差距，我们提出去噪扩散隐式模型（DDIM）。DDIM 是隐式概率模型 (Mohamed & Lakshminarayanan 2016)，与 DDPM 密切相关：训练目标函数相同。第 3 节把 DDPM 使用的马尔可夫前向扩散过程推广到非马尔可夫过程，并仍能设计合适的反向生成马尔可夫链。我们证明所得变分训练目标有一个共享替代目标，恰好是训练 DDPM 所用目标。因此，只需选择不同的非马尔可夫扩散过程（第 4.1 节）及相应反向生成马尔可夫链，即可用同一神经网络自由选用一大族生成模型。特别地，我们能使用导向「短」生成马尔可夫链的非马尔可夫扩散过程（第 4.2 节），用很少步数即可模拟。这可在样本质量仅小幅下降的代价下大幅提高采样效率。

In Section 5, we demonstrate several empirical benefits of DDIMs over DDPMs. First, DDIMs have superior sample generation quality compared to DDPMs, when we accelerate sampling by $10\times$ to $100\times$ using our proposed method. Second, DDIM samples have the following “consistency” property, which does not hold for DDPMs: if we start with the same initial latent variable and generate several samples with Markov chains of various lengths, these samples would have similar high-level features. Third, because of “consistency” in DDIMs, we can perform semantically meaningful image interpolation by manipulating the initial latent variable in DDIMs, unlike DDPMs which interpolates near the image space due to the stochastic generative process.

第 5 节展示 DDIM 相对 DDPM 的若干经验优势。第一，用所提方法把采样加速 $10\times$ 到 $100\times$ 时，DDIM 样本质量优于 DDPM。第二，DDIM 样本具有 DDPM 所不具备的「一致性」：从同一初始潜变量出发、用不同长度的马尔可夫链生成多个样本时，这些样本会有相似的高层特征。第三，因 DDIM 有「一致性」，可通过操纵初始潜变量做语义有意义的图像插值；而 DDPM 因生成过程随机，插值更靠近图像空间。

2 Background

2 背景

Figure 1: Graphical models for diffusion (left) and non-Markovian (right) inference models.

图 1：扩散（左）与非马尔可夫（右）推断模型的图模型。

Given samples from a data distribution $q({\bm{x}}_{0})$, we are interested in learning a model distribution $p_{\theta}({\bm{x}}_{0})$ that approximates $q({\bm{x}}_{0})$ and is easy to sample from. Denoising diffusion probabilistic models (DDPMs, Sohl-Dickstein et al. 2015; Ho et al. 2020) are latent variable models of the form

给定数据分布 $q({\bm{x}}_{0})$ 的样本，我们希望学习近似 $q({\bm{x}}_{0})$ 且易于采样的模型分布 $p_{\theta}({\bm{x}}_{0})$。去噪扩散概率模型（DDPM, Sohl-Dickstein et al. 2015; Ho et al. 2020）是如下形式的潜变量模型

$$\displaystyle p_{\theta}({\bm{x}}_{0})=\int p_{\theta}({\bm{x}}_{0:T})\mathrm{d}{\bm{x}}_{1:T},\quad\text{where}\quad p_{\theta}({\bm{x}}_{0:T}):=p_{\theta}({\bm{x}}_{T})\prod_{t=1}^{T}p^{(t)}_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t}) \tag{1}$$

where ${\bm{x}}_{1},\ldots,{\bm{x}}_{T}$ are latent variables in the same sample space as ${\bm{x}}_{0}$ (denoted as ${\mathcal{X}}$). The parameters $\theta$ are learned to fit the data distribution $q({\bm{x}}_{0})$ by maximizing a variational lower bound:

其中 ${\bm{x}}_{1},\ldots,{\bm{x}}_{T}$ 是与 ${\bm{x}}_{0}$ 同一样本空间（记为 ${\mathcal{X}}$）中的潜变量。参数 $\theta$ 通过最大化变分下界来拟合数据分布 $q({\bm{x}}_{0})$：

$$\displaystyle\max_{\theta}{\mathbb{E}}_{q({\bm{x}}_{0})}[\log p_{\theta}({\bm{x}}_{0})]\leq\max_{\theta}{\mathbb{E}}_{q({\bm{x}}_{0},{\bm{x}}_{1},\ldots,{\bm{x}}_{T})}\left[\log p_{\theta}({\bm{x}}_{0:T})-\log q({\bm{x}}_{1:T}|{\bm{x}}_{0})\right] \tag{2}$$

where $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$ is some inference distribution over the latent variables. Unlike typical latent variable models (such as the variational autoencoder (Rezende et al. 2014)), DDPMs are learned with a fixed (rather than trainable) inference procedure $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$, and latent variables are relatively high dimensional. For example, Ho et al. 2020 considered the following Markov chain with Gaussian transitions parameterized by a decreasing sequence $\alpha_{1:T}\in(0,1]^{T}$:

其中 $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$ 是潜变量上的某推断分布。与典型潜变量模型（如变分自编码器 (Rezende et al. 2014)）不同，DDPM 用固定（而非可训练）的推断程序 $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$ 学习，且潜变量维数相对较高。例如，Ho et al. 2020 考虑了由递减序列 $\alpha_{1:T}\in(0,1]^{T}$ 参数化的如下高斯转移马尔可夫链：

$$\displaystyle q({\bm{x}}_{1:T}|{\bm{x}}_{0}):=\prod_{t=1}^{T}q({\bm{x}}_{t}|{\bm{x}}_{t-1}),\text{where}\ q({\bm{x}}_{t}|{\bm{x}}_{t-1}):={\mathcal{N}}\left(\sqrt{\frac{\alpha_{t}}{\alpha_{t-1}}}{\bm{x}}_{t-1},\left(1-\frac{\alpha_{t}}{\alpha_{t-1}}\right){\bm{I}}\right) \tag{3}$$

where the covariance matrix is ensured to have positive terms on its diagonal. This is called the forward process due to the autoregressive nature of the sampling procedure (from ${\bm{x}}_{0}$ to ${\bm{x}}_{T}$). We call the latent variable model $p_{\theta}({\bm{x}}_{0:T})$, which is a Markov chain that samples from ${\bm{x}}_{T}$ to ${\bm{x}}_{0}$, the generative process, since it approximates the intractable reverse process $q({\bm{x}}_{t-1}|{\bm{x}}_{t})$. Intuitively, the forward process progressively adds noise to the observation ${\bm{x}}_{0}$, whereas the generative process progressively denoises a noisy observation (Figure 1, left).

其中协方差矩阵对角项保证为正。因其采样过程的自回归性质（从 ${\bm{x}}_{0}$ 到 ${\bm{x}}_{T}$），这称为前向过程。我们把从 ${\bm{x}}_{T}$ 采到 ${\bm{x}}_{0}$ 的潜变量模型 $p_{\theta}({\bm{x}}_{0:T})$ 称为生成过程，因为它近似难处理的反向过程 $q({\bm{x}}_{t-1}|{\bm{x}}_{t})$。直观上，前向过程逐步给观测 ${\bm{x}}_{0}$ 加噪，生成过程则逐步给带噪观测去噪（图 1，左）。

A special property of the forward process is that

前向过程的一个特殊性质是

$$q({\bm{x}}_{t}|{\bm{x}}_{0}):=\int q({\bm{x}}_{1:t}|{\bm{x}}_{0})\mathrm{d}{\bm{x}}_{1:(t-1)}={\mathcal{N}}({\bm{x}}_{t};\sqrt{\alpha_{t}}{\bm{x}}_{0},(1-\alpha_{t}){\bm{I}});$$

so we can express ${\bm{x}}_{t}$ as a linear combination of ${\bm{x}}_{0}$ and a noise variable $\epsilon$:

因此可将 ${\bm{x}}_{t}$ 写成 ${\bm{x}}_{0}$ 与噪声变量 $\epsilon$ 的线性组合：

$$\displaystyle{\bm{x}}_{t}=\sqrt{\alpha_{t}}{\bm{x}}_{0}+\sqrt{1-\alpha_{t}}\epsilon,\quad\text{where}\quad\epsilon\sim{\mathcal{N}}({\bm{0}},{\bm{I}}). \tag{4}$$

When we set $\alpha_{T}$ sufficiently close to $0$, $q({\bm{x}}_{T}|{\bm{x}}_{0})$ converges to a standard Gaussian for all ${\bm{x}}_{0}$, so it is natural to set $p_{\theta}({\bm{x}}_{T}):={\mathcal{N}}({\bm{0}},{\bm{I}})$. If all the conditionals are modeled as Gaussians with trainable mean functions and fixed variances, the objective in Eq. (2) can be simplified to[^1]:

当把 $\alpha_{T}$ 设得充分接近 $0$ 时，对一切 ${\bm{x}}_{0}$，$q({\bm{x}}_{T}|{\bm{x}}_{0})$ 收敛到标准高斯，因而自然取 $p_{\theta}({\bm{x}}_{T}):={\mathcal{N}}({\bm{0}},{\bm{I}})$。若所有条件分布都建模为具有可训练均值函数与固定方差的高斯，则式 (2) 中的目标可简化为[^1]：

$$\displaystyle L_{\gamma}(\epsilon_{\theta}) \\ \displaystyle:=\sum_{t=1}^{T}\gamma_{t}{\mathbb{E}}_{{\bm{x}}_{0}\sim q({\bm{x}}_{0}),\epsilon_{t}\sim{\mathcal{N}}({\bm{0}},{\bm{I}})}\left[{\lVert{\epsilon_{\theta}^{(t)}(\sqrt{\alpha_{t}}{\bm{x}}_{0}+\sqrt{1-\alpha_{t}}\epsilon_{t})-\epsilon_{t}}\rVert}_{2}^{2}\right] \tag{5}$$

where $\epsilon_{\theta}:=\{\epsilon_{\theta}^{(t)}\}_{t=1}^{T}$ is a set of $T$ functions, each $\epsilon_{\theta}^{(t)}:{\mathcal{X}}\to{\mathcal{X}}$ (indexed by $t$) is a function with trainable parameters $\theta^{(t)}$, and $\gamma:=[\gamma_{1},\ldots,\gamma_{T}]$ is a vector of positive coefficients in the objective that depends on $\alpha_{1:T}$. In Ho et al. 2020, the objective with $\gamma={\bm{1}}$ is optimized instead to maximize generation performance of the trained model; this is also the same objective used in noise conditional score networks (Song & Ermon 2019) based on score matching (Hyvärinen 2005; Vincent 2011). From a trained model, ${\bm{x}}_{0}$ is sampled by first sampling ${\bm{x}}_{T}$ from the prior $p_{\theta}({\bm{x}}_{T})$, and then sampling ${\bm{x}}_{t-1}$ from the generative processes iteratively.

其中 $\epsilon_{\theta}:=\{\epsilon_{\theta}^{(t)}\}_{t=1}^{T}$ 是 $T$ 个函数的集合，每个 $\epsilon_{\theta}^{(t)}:{\mathcal{X}}\to{\mathcal{X}}$（由 $t$ 索引）带可训练参数 $\theta^{(t)}$，$\gamma:=[\gamma_{1},\ldots,\gamma_{T}]$ 是目标中依赖 $\alpha_{1:T}$ 的正系数向量。Ho et al. 2020 优化 $\gamma={\bm{1}}$ 的目标以提升训练模型的生成表现；这也与基于分数匹配 (Hyvärinen 2005; Vincent 2011) 的噪声条件分数网络 (Song & Ermon 2019) 所用目标相同。从训练好的模型采样 ${\bm{x}}_{0}$ 时，先从先验 $p_{\theta}({\bm{x}}_{T})$ 采 ${\bm{x}}_{T}$，再迭代地从生成过程采 ${\bm{x}}_{t-1}$。

The length $T$ of the forward process is an important hyperparameter in DDPMs. From a variational perspective, a large $T$ allows the reverse process to be close to a Gaussian (Sohl-Dickstein et al. 2015), so that the generative process modeled with Gaussian conditional distributions becomes a good approximation; this motivates the choice of large $T$ values, such as $T=1000$ in Ho et al. 2020. However, as all $T$ iterations have to be performed sequentially, instead of in parallel, to obtain a sample ${\bm{x}}_{0}$, sampling from DDPMs is much slower than sampling from other deep generative models, which makes them impractical for tasks where compute is limited and latency is critical.

前向过程长度 $T$ 是 DDPM 的重要超参数。从变分视角看，大 $T$ 使反向过程接近高斯 (Sohl-Dickstein et al. 2015)，从而用高斯条件分布建模的生成过程成为好近似；这促使选择如 Ho et al. 2020 中 $T=1000$ 的大 $T$。但要得到样本 ${\bm{x}}_{0}$，全部 $T$ 次迭代必须串行而非并行，因而从 DDPM 采样远慢于其他深度生成模型，在算力有限、延迟关键的任务上不实用。

3 Variational Inference for non-Markovian Forward Processes

3 非马尔可夫前向过程的变分推断

Because the generative model approximates the reverse of the inference process, we need to rethink the inference process in order to reduce the number of iterations required by the generative model. Our key observation is that the DDPM objective in the form of $L_{\gamma}$ only depends on the marginals[^2] $q({\bm{x}}_{t}|{\bm{x}}_{0})$, but not directly on the joint $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$. Since there are many inference distributions (joints) with the same marginals, we explore alternative inference processes that are non-Markovian, which leads to new generative processes (Figure 1, right). These non-Markovian inference process lead to the same surrogate objective function as DDPM, as we will show below. In Appendix A, we show that the non-Markovian perspective also applies beyond the Gaussian case.

因为生成模型近似推断过程的逆，要减少生成模型所需迭代次数，就需要重新思考推断过程。关键观察是：形如 $L_{\gamma}$ 的 DDPM 目标只依赖边缘[^2] $q({\bm{x}}_{t}|{\bm{x}}_{0})$，而不直接依赖联合 $q({\bm{x}}_{1:T}|{\bm{x}}_{0})$。由于有许多具有相同边缘的推断分布（联合），我们探索非马尔可夫的替代推断过程，从而得到新的生成过程（图 1，右）。如下所示，这些非马尔可夫推断过程导向与 DDPM 相同的替代目标函数。附录 A 表明非马尔可夫视角也适用于高斯情形之外。

3.1 Non-Markovian forward processes

3.1 非马尔可夫前向过程

Let us consider a family ${\mathcal{Q}}$ of inference distributions, indexed by a real vector $\sigma\in\mathbb{R}_{\geq 0}^{T}$:

考虑由实向量 $\sigma\in\mathbb{R}_{\geq 0}^{T}$ 索引的一族推断分布 ${\mathcal{Q}}$：

$$\displaystyle q_{\sigma}({\bm{x}}_{1:T}|{\bm{x}}_{0}):=q_{\sigma}({\bm{x}}_{T}|{\bm{x}}_{0})\prod_{t=2}^{T}q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0}) \tag{6}$$

where $q_{\sigma}({\bm{x}}_{T}|{\bm{x}}_{0})={\mathcal{N}}(\sqrt{\alpha_{T}}{\bm{x}}_{0},(1-\alpha_{T}){\bm{I}})$ and for all $t>1$,

其中 $q_{\sigma}({\bm{x}}_{T}|{\bm{x}}_{0})={\mathcal{N}}(\sqrt{\alpha_{T}}{\bm{x}}_{0},(1-\alpha_{T}){\bm{I}})$，且对一切 $t>1$,

$$\displaystyle q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0})={\mathcal{N}}\left(\sqrt{\alpha_{t-1}}{\bm{x}}_{0}+\sqrt{1-\alpha_{t-1}-\sigma^{2}_{t}}\cdot{\frac{{\bm{x}}_{t}-\sqrt{\alpha_{t}}{\bm{x}}_{0}}{\sqrt{1-\alpha_{t}}}},\sigma_{t}^{2}{\bm{I}}\right). \tag{7}$$

The mean function is chosen to order to ensure that $q_{\sigma}({\bm{x}}_{t}|{\bm{x}}_{0})={\mathcal{N}}(\sqrt{\alpha_{t}}{\bm{x}}_{0},(1-\alpha_{t}){\bm{I}})$ for all $t$ (see Lemma 1 of Appendix B), so that it defines a joint inference distribution that matches the ‘‘marginals’’ as desired. The forward process[^3] can be derived from Bayes’ rule:

均值函数的选取是为了保证对一切 $t$ 有 $q_{\sigma}({\bm{x}}_{t}|{\bm{x}}_{0})={\mathcal{N}}(\sqrt{\alpha_{t}}{\bm{x}}_{0},(1-\alpha_{t}){\bm{I}})$（见附录 B 引理 1），从而定义匹配所需「边缘」的联合推断分布。前向过程[^3]可由贝叶斯法则导出：

$$\displaystyle q_{\sigma}({\bm{x}}_{t}|{\bm{x}}_{t-1},{\bm{x}}_{0})=\frac{q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0})q_{\sigma}({\bm{x}}_{t}|{\bm{x}}_{0})}{q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{0})}, \tag{8}$$

which is also Gaussian (although we do not use this fact for the remainder of this paper). Unlike the diffusion process in Eq. (3), the forward process here is no longer Markovian, since each ${\bm{x}}_{t}$ could depend on both ${\bm{x}}_{t-1}$ and ${\bm{x}}_{0}$. The magnitude of $\sigma$ controls the how stochastic the forward process is; when $\sigma\to{\bm{0}}$, we reach an extreme case where as long as we observe ${\bm{x}}_{0}$ and ${\bm{x}}_{t}$ for some $t$, then ${\bm{x}}_{t-1}$ become known and fixed.

它也是高斯的（后文虽不用这一事实）。与式 (3) 中的扩散过程不同，这里的前向过程不再是马尔可夫的，因为每个 ${\bm{x}}_{t}$ 可能同时依赖 ${\bm{x}}_{t-1}$ 与 ${\bm{x}}_{0}$。$\sigma$ 的大小控制前向过程有多随机；当 $\sigma\to{\bm{0}}$ 时，达到极端情形：只要观察到某 $t$ 的 ${\bm{x}}_{0}$ 与 ${\bm{x}}_{t}$，${\bm{x}}_{t-1}$ 便已知且固定。

3.2 Generative process and unified variational inference objective

3.2 生成过程与统一变分推断目标

Next, we define a trainable generative process $p_{\theta}({\bm{x}}_{0:T})$ where each $p_{\theta}^{(t)}({\bm{x}}_{t-1}|{\bm{x}}_{t})$ leverages knowledge of $q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0})$. Intuitively, given a noisy observation ${\bm{x}}_{t}$, we first make a prediction[^4] of the corresponding ${\bm{x}}_{0}$, and then use it to obtain a sample ${\bm{x}}_{t-1}$ through the reverse conditional distribution $q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0})$, which we have defined.

接下来定义可训练生成过程 $p_{\theta}({\bm{x}}_{0:T})$，其中每个 $p_{\theta}^{(t)}({\bm{x}}_{t-1}|{\bm{x}}_{t})$ 利用 $q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0})$ 的知识。直观上，给定带噪观测 ${\bm{x}}_{t}$，我们先对其对应的 ${\bm{x}}_{0}$ 做预测[^4]，再通过已定义的反向条件分布 $q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0})$ 得到样本 ${\bm{x}}_{t-1}$。

For some ${\bm{x}}_{0}\sim q({\bm{x}}_{0})$ and $\epsilon_{t}\sim{\mathcal{N}}({\bm{0}},{\bm{I}})$, ${\bm{x}}_{t}$ can be obtained using Eq. (4). The model $\epsilon_{\theta}^{(t)}({\bm{x}}_{t})$ then attempts to predict $\epsilon_{t}$ from ${\bm{x}}_{t}$, without knowledge of ${\bm{x}}_{0}$. By rewriting Eq. (4), one can then predict the denoised observation, which is a prediction of ${\bm{x}}_{0}$ given ${\bm{x}}_{t}$:

对某 ${\bm{x}}_{0}\sim q({\bm{x}}_{0})$ 与 $\epsilon_{t}\sim{\mathcal{N}}({\bm{0}},{\bm{I}})$，可用式 (4) 得到 ${\bm{x}}_{t}$。模型 $\epsilon_{\theta}^{(t)}({\bm{x}}_{t})$ 试图在不知 ${\bm{x}}_{0}$ 的情况下从 ${\bm{x}}_{t}$ 预测 $\epsilon_{t}$。改写式 (4)，即可预测去噪观测，亦即给定 ${\bm{x}}_{t}$ 对 ${\bm{x}}_{0}$ 的预测：

$$\displaystyle f_{\theta}^{(t)}({\bm{x}}_{t}):=({\bm{x}}_{t}-\sqrt{1-\alpha_{t}}\cdot\epsilon_{\theta}^{(t)}({\bm{x}}_{t}))/\sqrt{\alpha_{t}}. \tag{9}$$

We can then define the generative process with a fixed prior $p_{\theta}({\bm{x}}_{T})={\mathcal{N}}({\bm{0}},{\bm{I}})$ and

于是可用固定先验 $p_{\theta}({\bm{x}}_{T})={\mathcal{N}}({\bm{0}},{\bm{I}})$ 定义生成过程，并取

$$\displaystyle p_{\theta}^{(t)}({\bm{x}}{t-1}|{\bm{x}}{t})=\begin{cases}{\mathcal{N}}(f_{\theta}^{(1)}({\bm{x}}{1}),\sigma{1}^{2}{\bm{I}})&\text{if}\ t=1\
q_{\sigma}({\bm{x}}{t-1}|{\bm{x}}{t},f_{\theta}^{(t)}({\bm{x}}_{t}))&\text{otherwise,}\end{cases} \tag{10}$$

where $q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},f_{\theta}^{(t)}({\bm{x}}_{t}))$ is defined as in Eq. (7) with ${\bm{x}}_{0}$ replaced by $f_{\theta}^{(t)}({\bm{x}}_{t})$. We add some Gaussian noise (with covariance $\sigma_{1}^{2}{\bm{I}}$) for the case of $t=1$ to ensure that the generative process is supported everywhere.

其中 $q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},f_{\theta}^{(t)}({\bm{x}}_{t}))$ 按式 (7) 定义，只是把 ${\bm{x}}_{0}$ 换成 $f_{\theta}^{(t)}({\bm{x}}_{t})$。对 $t=1$ 的情形，我们加入协方差为 $\sigma_{1}^{2}{\bm{I}}$ 的高斯噪声，以保证生成过程在处处有支撑。

We optimize $\theta$ via the following variational inference objective (which is a functional over $\epsilon_{\theta}$):

我们通过如下变分推断目标（作为 $\epsilon_{\theta}$ 上的泛函）优化 $\theta$：

$$\displaystyle J_{\sigma}(\epsilon_{\theta}):={\mathbb{E}}_{{\bm{x}}_{0:T}\sim q_{\sigma}({\bm{x}}_{0:T})}[\log q_{\sigma}({\bm{x}}_{1:T}|{\bm{x}}_{0})-\log p_{\theta}({\bm{x}}_{0:T})] \\ \displaystyle= \\ \displaystyle\ {\mathbb{E}}_{{\bm{x}}_{0:T}\sim q_{\sigma}({\bm{x}}_{0:T})}\left[\log q_{\sigma}({\bm{x}}_{T}|{\bm{x}}_{0})+\sum_{t=2}^{T}\log q_{\sigma}({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0})-\sum_{t=1}^{T}\log p_{\theta}^{(t)}({\bm{x}}_{t-1}|{\bm{x}}_{t})-\log p_{\theta}({\bm{x}}_{T})\right] \tag{11}$$

where we factorize $q_{\sigma}({\bm{x}}_{1:T}|{\bm{x}}_{0})$ according to Eq. (6) and $p_{\theta}({\bm{x}}_{0:T})$ according to Eq. (1).

其中我们按式 (6) 分解 $q_{\sigma}({\bm{x}}_{1:T}|{\bm{x}}_{0})$，按式 (1) 分解 $p_{\theta}({\bm{x}}_{0:T})$。

From the definition of $J_{\sigma}$, it would appear that a different model has to be trained for every choice of $\sigma$, since it corresponds to a different variational objective (and a different generative process). However, $J_{\sigma}$ is equivalent to $L_{\gamma}$ for certain weights $\gamma$, as we show below.

从 $J_{\sigma}$ 的定义看，似乎对每个 $\sigma$ 都要训练不同模型，因为它对应不同变分目标（以及不同生成过程）。但如下所示，对某些权重 $\gamma$，$J_{\sigma}$ 等价于 $L_{\gamma}$。

Theorem 1.

For all $\sigma>{\bm{0}}$, there exists $\gamma\in\mathbb{R}_{>0}^{T}$ and $C\in\mathbb{R}$, such that $J_{\sigma}=L_{\gamma}+C$.

定理 1。

对一切 $\sigma>{\bm{0}}$，存在 $\gamma\in\mathbb{R}_{>0}^{T}$ 与 $C\in\mathbb{R}$，使得 $J_{\sigma}=L_{\gamma}+C$.

The variational objective $L_{\gamma}$ is special in the sense that if parameters $\theta$ of the models $\epsilon_{\theta}^{(t)}$ are not shared across different $t$, then the optimal solution for $\epsilon_{\theta}$ will not depend on the weights $\gamma$ (as global optimum is achieved by separately maximizing each term in the sum). This property of $L_{\gamma}$ has two implications. On the one hand, this justified the use of $L_{\bm{1}}$ as a surrogate objective function for the variational lower bound in DDPMs; on the other hand, since $J_{\sigma}$ is equivalent to some $L_{\gamma}$ from Theorem 1, the optimal solution of $J_{\sigma}$ is also the same as that of $L_{\bm{1}}$. Therefore, if parameters are not shared across $t$ in the model $\epsilon_{\theta}$, then the $L_{\bm{1}}$ objective used by Ho et al. 2020 can be used as a surrogate objective for the variational objective $J_{\sigma}$ as well.

变分目标 $L_{\gamma}$ 的特殊之处在于：若模型 $\epsilon_{\theta}^{(t)}$ 的参数 $\theta$ 在不同 $t$ 之间不共享，则 $\epsilon_{\theta}$ 的最优解不依赖权重 $\gamma$（因为全局最优可通过分别最大化求和中的每一项达到）。$L_{\gamma}$ 的这一性质有两重含义。一方面，它为在 DDPM 中用 $L_{\bm{1}}$ 作为变分下界的替代目标提供了理由；另一方面，由定理 1，$J_{\sigma}$ 等价于某 $L_{\gamma}$，因而 $J_{\sigma}$ 的最优解也与 $L_{\bm{1}}$ 相同。因此，若模型 $\epsilon_{\theta}$ 中参数不跨 $t$ 共享，则 Ho et al. 2020 使用的 $L_{\bm{1}}$ 目标也可作为变分目标 $J_{\sigma}$ 的替代目标。

4 Sampling from Generalized Generative Processes

4 从广义生成过程采样

With $L_{\bm{1}}$ as the objective, we are not only learning a generative process for the Markovian inference process considered in Sohl-Dickstein et al. 2015 and Ho et al. 2020, but also generative processes for many non-Markovian forward processes parametrized by $\sigma$ that we have described. Therefore, we can essentially use pretrained DDPM models as the solutions to the new objectives, and focus on finding a generative process that is better at producing samples subject to our needs by changing $\sigma$.

以 $L_{\bm{1}}$ 为目标时，我们不仅在为 Sohl-Dickstein et al. 2015 与 Ho et al. 2020 所考虑的马尔可夫推断过程学习生成过程，也为由 $\sigma$ 参数化的许多非马尔可夫前向过程学习生成过程。因此，本质上可以把预训练 DDPM 模型当作新目标的解，并通过改变 $\sigma$ 去寻找更符合需求的生成过程，而无需重训。

4.1 Denoising Diffusion Implicit Models

From $p_{\theta}({\bm{x}}_{1:T})$ in Eq. (10), one can generate a sample ${\bm{x}}_{t-1}$ from a sample ${\bm{x}}_{t}$ via:

由式 (10) 中的 $p_{\theta}({\bm{x}}_{1:T})$，可从样本 ${\bm{x}}_{t}$ 生成样本 ${\bm{x}}_{t-1}$：

$$\displaystyle{\bm{x}}_{t-1} \\ \displaystyle=\sqrt{\alpha_{t-1}}\underbrace{\left(\frac{{\bm{x}}_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}^{(t)}({\bm{x}}_{t})}{\sqrt{\alpha_{t}}}\right)}_{\text{`` predicted }{\bm{x}}_{0}\text{''}}+\underbrace{\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2}}\cdot\epsilon_{\theta}^{(t)}({\bm{x}}_{t})}_{\text{``direction pointing to }{\bm{x}}_{t}\text{''}}+\underbrace{\sigma_{t}\epsilon_{t}}_{\text{random noise}} \tag{12}$$

where $\epsilon_{t}\sim{\mathcal{N}}({\bm{0}},{\bm{I}})$ is standard Gaussian noise independent of ${\bm{x}}_{t}$, and we define $\alpha_{0}:=1$. Different choices of $\sigma$ values results in different generative processes, all while using the same model $\epsilon_{\theta}$, so re-training the model is unnecessary. When $\sigma_{t}=\sqrt{(1-\alpha_{t-1})/(1-\alpha_{t})}\sqrt{1-\alpha_{t}/\alpha_{t-1}}$ for all $t$, the forward process becomes Markovian, and the generative process becomes a DDPM.

其中 $\epsilon_{t}\sim{\mathcal{N}}({\bm{0}},{\bm{I}})$ 是与 ${\bm{x}}_{t}$ 独立的标准高斯噪声，并定义 $\alpha_{0}:=1$。不同 $\sigma$ 取值给出不同生成过程，却共用同一模型 $\epsilon_{\theta}$，因而无需重训。当对一切 $t$ 取 $\sigma_{t}=\sqrt{(1-\alpha_{t-1})/(1-\alpha_{t})}\sqrt{1-\alpha_{t}/\alpha_{t-1}}$ 时，前向过程变为马尔可夫，生成过程变为 DDPM。

We note another special case when $\sigma_{t}=0$ for all $t$[^5]; the forward process becomes deterministic given ${\bm{x}}_{t-1}$ and ${\bm{x}}_{0}$, except for $t=1$; in the generative process, the coefficient before the random noise $\epsilon_{t}$ becomes zero. The resulting model becomes an implicit probabilistic model (Mohamed & Lakshminarayanan 2016), where samples are generated from latent variables with a fixed procedure (from ${\bm{x}}_{T}$ to ${\bm{x}}_{0}$). We name this the denoising diffusion implicit model (DDIM, pronounced /d:Im/), because it is an implicit probabilistic model trained with the DDPM objective (despite the forward process no longer being a diffusion).

另一特殊情形是对一切 $t$ 取 $\sigma_{t}=0$[^5]：给定 ${\bm{x}}_{t-1}$ 与 ${\bm{x}}_{0}$，前向过程变为确定性的（$t=1$ 除外）；生成过程中随机噪声 $\epsilon_{t}$ 前的系数变为零。所得模型成为隐式概率模型 (Mohamed & Lakshminarayanan 2016)，样本由潜变量经固定程序（从 ${\bm{x}}_{T}$ 到 ${\bm{x}}_{0}$）生成。我们称之为去噪扩散隐式模型（DDIM，读作 /d:Im/），因为它虽用 DDPM 目标训练，却是隐式概率模型（尽管前向过程已不再是扩散）。

4.2 Accelerated generation processes

4.2 加速生成过程

In the previous sections, the generative process is considered as the approximation to the reverse process; since of the forward process has $T$ steps, the generative process is also forced to sample $T$ steps. However, as the denoising objective $L_{\bm{1}}$ does not depend on the specific forward procedure as long as $q_{\sigma}({\bm{x}}_{t}|{\bm{x}}_{0})$ is fixed, we may also consider forward processes with lengths smaller than $T$, which accelerates the corresponding generative processes without having to train a different model.

前几节把生成过程视为反向过程的近似；由于前向过程有 $T$ 步，生成过程也被强制采样 $T$ 步。但只要 $q_{\sigma}({\bm{x}}_{t}|{\bm{x}}_{0})$ 固定，去噪目标 $L_{\bm{1}}$ 就不依赖具体前向程序，因而也可考虑长度小于 $T$ 的前向过程，从而在无需另训模型的情况下加速对应生成过程。

Figure 2: Graphical model for accelerated generation, where $\tau=[1,3]$.

图 2：加速生成的图模型，其中 $\tau=[1,3]$。

Let us consider the forward process as defined not on all the latent variables ${\bm{x}}_{1:T}$, but on a subset $\{{\bm{x}}_{\tau_{1}},\ldots,{\bm{x}}_{\tau_{S}}\}$, where $\tau$ is an increasing sub-sequence of $[1,\ldots,T]$ of length $S$. In particular, we define the sequential forward process over ${\bm{x}}_{\tau_{1}},\ldots,{\bm{x}}_{\tau_{S}}$ such that $q({\bm{x}}_{\tau_{i}}|{\bm{x}}_{0})={\mathcal{N}}(\sqrt{\alpha_{\tau_{i}}}{\bm{x}}_{0},(1-\alpha_{\tau_{i}}){\bm{I}})$ matches the “marginals” (see Figure 2 for an illustration). The generative process now samples latent variables according to $\text{reversed}(\tau)$, which we term (sampling) trajectory. When the length of the sampling trajectory is much smaller than $T$, we may achieve significant increases in computational efficiency due to the iterative nature of the sampling process.

考虑前向过程不是定义在全部潜变量 ${\bm{x}}_{1:T}$ 上，而是定义在子集 $\{{\bm{x}}_{\tau_{1}},\ldots,{\bm{x}}_{\tau_{S}}\}$ 上，其中 $\tau$ 是 $[1,\ldots,T]$ 的长度为 $S$ 的递增子序列。特别地，我们在 ${\bm{x}}_{\tau_{1}},\ldots,{\bm{x}}_{\tau_{S}}$ 上定义序列前向过程，使 $q({\bm{x}}_{\tau_{i}}|{\bm{x}}_{0})={\mathcal{N}}(\sqrt{\alpha_{\tau_{i}}}{\bm{x}}_{0},(1-\alpha_{\tau_{i}}){\bm{I}})$ 匹配「边缘」（示意见图 2）。生成过程现在按 $\text{reversed}(\tau)$ 采样潜变量，我们称之为*（采样）轨迹*。当采样轨迹长度远小于 $T$ 时，因采样过程的迭代本质，可显著提高计算效率。

Using a similar argument as in Section 3, we can justify using the model trained with the $L_{\bm{1}}$ objective, so no changes are needed in training. We show that only slight changes to the updates in Eq. (12) are needed to obtain the new, faster generative processes, which applies to DDPM, DDIM, as well as all generative processes considered in Eq. (10). We include these details in Appendix C.1.

用与第 3 节类似的论证，可以说明使用以 $L_{\bm{1}}$ 目标训练的模型是合理的，因而训练无需改动。我们证明：对式 (12) 的更新只需轻微修改，即可得到新的、更快的生成过程，适用于 DDPM、DDIM 以及式 (10) 中考虑的全部生成过程。细节见附录 C.1。

In principle, this means that we can train a model with an arbitrary number of forward steps but only sample from some of them in the generative process. Therefore, the trained model could consider many more steps than what is considered in (Ho et al. 2020) or even a continuous time variable $t$ (Chen et al. 2020). We leave empirical investigations of this aspect as future work.

原则上，这意味着可用任意数量的前向步训练模型，但在生成过程中只从其中一部分采样。因此，训练模型可考虑远多于 (Ho et al. 2020) 的步数，甚至连续时间变量 $t$ (Chen et al. 2020)。经验研究留作未来工作。

4.3 Relevance to Neural ODEs

4.3 与 Neural ODE 的关系

Moreover, we can rewrite the DDIM iterate according to Eq. (12), and its similarity to Euler integration for solving ordinary differential equations (ODEs) becomes more apparent:

此外，可按式 (12) 改写 DDIM 迭代，其与求解常微分方程（ODE）的欧拉积分的相似性更加明显：

$$\displaystyle\frac{{\bm{x}}_{t-\Delta t}}{\sqrt{\alpha_{t-\Delta t}}}=\frac{{\bm{x}}_{t}}{\sqrt{\alpha_{t}}}+\left(\sqrt{\frac{1-\alpha_{t-\Delta t}}{\alpha_{t-\Delta t}}}-\sqrt{\frac{1-\alpha_{t}}{\alpha_{t}}}\right)\epsilon_{\theta}^{(t)}({\bm{x}}_{t}) \tag{13}$$

To derive the corresponding ODE, we can reparameterize $(\sqrt{1-\alpha}/\sqrt{\alpha})$ with $\sigma$ and $({\bm{x}}/\sqrt{\alpha})$ with $\bar{{\bm{x}}}$. In the continuous case, $\sigma$ and ${\bm{x}}$ are functions of $t$, where $\sigma:{\mathbb{R}}_{\geq 0}\to{\mathbb{R}}_{\geq 0}$ is continous, increasing with $\sigma(0)=0$. Equation (13) with can be treated as a Euler method over the following ODE:

为导出对应 ODE，可用 $\sigma$ 重参数化 $(\sqrt{1-\alpha}/\sqrt{\alpha})$，用 $\bar{{\bm{x}}}$ 重参数化 $({\bm{x}}/\sqrt{\alpha})$。连续情形下，$\sigma$ 与 ${\bm{x}}$ 是 $t$ 的函数，其中 $\sigma:{\mathbb{R}}_{\geq 0}\to{\mathbb{R}}_{\geq 0}$ 连续、递增且 $\sigma(0)=0$。带式 (13) 的更新可视为如下 ODE 上的欧拉法：

$$\displaystyle\mathrm{d}\bar{{\bm{x}}}(t)=\epsilon_{\theta}^{(t)}\left(\frac{\bar{{\bm{x}}}(t)}{\sqrt{\sigma^{2}+1}}\right)\mathrm{d}\sigma(t), \tag{14}$$

where the initial conditions is ${\bm{x}}(T)\sim{\mathcal{N}}(0,\sigma(T))$ for a very large $\sigma(T)$ (which corresponds to the case of $\alpha\approx 0$). This suggests that with enough discretization steps, the we can also reverse the generation process (going from $t=0$ to $T$), which encodes ${\bm{x}}_{0}$ to ${\bm{x}}_{T}$ and simulates the reverse of the ODE in Eq. (14). This suggests that unlike DDPM, we can use DDIM to obtain encodings of the observations (as the form of ${\bm{x}}_{T}$), which might be useful for other downstream applications that requires latent representations of a model.

初值为 ${\bm{x}}(T)\sim{\mathcal{N}}(0,\sigma(T))$，且 $\sigma(T)$ 很大（对应 $\alpha\approx 0$）。这表明在足够多的离散化步下，我们也可反转生成过程（从 $t=0$ 走到 $T$），把 ${\bm{x}}_{0}$ 编码到 ${\bm{x}}_{T}$，并模拟式 (14) 中 ODE 的逆。这表明与 DDPM 不同，我们可用 DDIM 得到观测的编码（形式为 ${\bm{x}}_{T}$），这可能对需要模型潜表示的其他下游应用有用。

In a concurrent work, (Song et al. 2020) proposed a “probability flow ODE” that aims to recover the marginal densities of a stochastic differential equation (SDE) based on scores, from which a similar sampling schedule can be obtained. Here, we state that the our ODE is equivalent to a special case of theirs (which corresponds to a continuous-time analog of DDPM).

同期工作 (Song et al. 2020) 提出「概率流 ODE」，旨在恢复基于分数的随机微分方程（SDE）的边缘密度，并可得到类似采样日程。此处我们指出：我们的 ODE 等价于他们的一个特例（对应 DDPM 的连续时间类比）。

Proposition 1.

The ODE in Eq. (14) with the optimal model $\epsilon_{\theta}^{(t)}$ has an equivalent probability flow ODE corresponding to the “Variance-Exploding” SDE in Song et al. 2020.

命题 1。

式 (14) 中带最优模型 $\epsilon_{\theta}^{(t)}$ 的 ODE，有等价的概率流 ODE，对应于 Song et al. 2020 中的「方差爆炸」SDE。

We include the proof in Appendix B. While the ODEs are equivalent, the sampling procedures are not, since the Euler method for the probability flow ODE will make the following update:

证明见附录 B。尽管 ODE 等价，采样程序并不相同，因为概率流 ODE 的欧拉法将做如下更新：

$$\displaystyle\frac{{\bm{x}}_{t-\Delta t}}{\sqrt{\alpha_{t-\Delta t}}}=\frac{{\bm{x}}_{t}}{\sqrt{\alpha_{t}}}+\frac{1}{2}\left(\frac{1-\alpha_{t-\Delta t}}{\alpha_{t-\Delta t}}-\frac{1-\alpha_{t}}{\alpha_{t}}\right)\cdot\sqrt{\frac{\alpha_{t}}{1-\alpha_{t}}}\cdot\epsilon_{\theta}^{(t)}({\bm{x}}_{t}) \tag{15}$$

which is equivalent to ours if $\alpha_{t}$ and $\alpha_{t-\Delta t}$ are close enough. In fewer sampling steps, however, these choices will make a difference; we take Euler steps with respect to $\mathrm{d}\sigma(t)$ (which depends less directly on the scaling of “time” $t$) whereas Song et al. 2020 take Euler steps with respect to $\mathrm{d}t$.

若 $\alpha_{t}$ 与 $\alpha_{t-\Delta t}$ 足够接近，则与我们的更新等价。但在更少采样步时，这些选择会有差别：我们相对 $\mathrm{d}\sigma(t)$ 走欧拉步（对「时间」$t$ 的缩放依赖更间接），而 Song et al. 2020 相对 $\mathrm{d}t$ 走欧拉步。

5 Experiments

5 实验

In this section, we show that DDIMs outperform DDPMs in terms of image generation when fewer iterations are considered, giving speed ups of $10\times$ to $100\times$ over the original DDPM generation process. Moreover, unlike DDPMs, once the initial latent variables ${\bm{x}}_{T}$ are fixed, DDIMs retain high-level image features regardless of the generation trajectory, so they are able to perform interpolation directly from the latent space. DDIMs can also be used to encode samples that reconstruct them from the latent code, which DDPMs cannot do due to the stochastic sampling process.

本节表明：在考虑更少迭代时，DDIM 在图像生成上优于 DDPM，相对原始 DDPM 生成过程可加速 $10\times$ 到 $100\times$。此外，与 DDPM 不同，一旦固定初始潜变量 ${\bm{x}}_{T}$，无论生成轨迹如何，DDIM 都保留高层图像特征，因而能直接从潜空间插值。DDIM 还可用于编码样本并从潜码重建，而 DDPM 因采样随机无法做到。

For each dataset, we use the same trained model with $T=1000$ and the objective being $L_{\gamma}$ from Eq. (5) with $\gamma={\bm{1}}$; as we argued in Section 3, no changes are needed with regards to the training procedure. The only changes that we make is how we produce samples from the model; we achieve this by controlling $\tau$ (which controls how fast the samples are obtained) and $\sigma$ (which interpolates between the deterministic DDIM and the stochastic DDPM).

对每个数据集，我们使用同一以 $T=1000$、目标为式 (5) 中 $\gamma={\bm{1}}$ 的 $L_{\gamma}$ 训练的模型；如第 3 节所论，训练过程无需改动。唯一改动是如何从模型产出样本；我们通过控制 $\tau$（控制采样有多快）与 $\sigma$（在确定性 DDIM 与随机 DDPM 之间插值）实现。

We consider different sub-sequences $\tau$ of $[1,\ldots,T]$ and different variance hyperparameters $\sigma$ indexed by elements of $\tau$. To simplify comparisons, we consider $\sigma$ with the form:

我们考虑 $[1,\ldots,T]$ 的不同子序列 $\tau$，以及由 $\tau$ 元素索引的不同方差超参数 $\sigma$。为简化比较，考虑如下形式的 $\sigma$：

$$\displaystyle\sigma_{\tau_{i}}(\eta)=\eta\sqrt{(1-\alpha_{\tau_{i-1}})/(1-\alpha_{\tau_{i}})}\sqrt{1-{\alpha_{\tau_{i}}}/{\alpha_{\tau_{i-1}}}}, \tag{16}$$

where $\eta\in\mathbb{R}_{\geq 0}$ is a hyperparameter that we can directly control. This includes an original DDPM generative process when $\eta=1$ and DDIM when $\eta=0$. We also consider DDPM where the random noise has a larger standard deviation than $\sigma(1)$, which we denote as $\hat{\sigma}$: $\hat{\sigma}_{\tau_{i}}=\sqrt{1-{\alpha_{\tau_{i}}}/{\alpha_{\tau_{i-1}}}}$ . This is used by the implementation in Ho et al. 2020 only to obtain the CIFAR10 samples, but not samples of the other datasets. We include more details in Appendix D.

其中 $\eta\in\mathbb{R}_{\geq 0}$ 是可直接控制的超参数。这包含 $\eta=1$ 时的原始 DDPM 生成过程与 $\eta=0$ 时的 DDIM。我们也考虑随机噪声标准差大于 $\sigma(1)$ 的 DDPM，记为 $\hat{\sigma}$：$\hat{\sigma}_{\tau_{i}}=\sqrt{1-{\alpha_{\tau_{i}}}/{\alpha_{\tau_{i-1}}}}$。Ho et al. 2020 的实现仅在获取 CIFAR10 样本时使用它，其他数据集样本则不用。更多细节见附录 D。

5.1 Sample quality and efficiency

5.1 样本质量与效率

In Table 1, we report the quality of the generated samples with models trained on CIFAR10 and CelebA, as measured by Frechet Inception Distance (FID (Heusel et al. 2017)), where we vary the number of timesteps used to generate a sample ($\dim(\tau)$) and the stochasticity of the process ($\eta$). As expected, the sample quality becomes higher as we increase $\dim(\tau)$, presenting a trade-off between sample quality and computational costs. We observe that DDIM ($\eta=0$) achieves the best sample quality when $\dim(\tau)$ is small, and DDPM ($\eta=1$ and $\hat{\sigma}$) typically has worse sample quality compared to its less stochastic counterparts with the same $\dim(\tau)$, except for the case for $\dim(\tau)=1000$ and $\hat{\sigma}$ reported by Ho et al. 2020 where DDIM is marginally worse. However, the sample quality of $\hat{\sigma}$ becomes much worse for smaller $\dim(\tau)$, which suggests that it is ill-suited for shorter trajectories. DDIM, on the other hand, achieves high sample quality much more consistently.

表 1 报告在 CIFAR10 与 CelebA 上训练的模型生成样本质量，以 Frechet Inception Distance（FID (Heusel et al. 2017)）度量；我们改变生成样本所用时间步数（$\dim(\tau)$）与过程随机性（$\eta$）。一如预期，增大 $\dim(\tau)$ 时样本质量变高，呈现样本质量与计算成本的权衡。我们观察到：当 $\dim(\tau)$ 较小时，DDIM（$\eta=0$）取得最佳样本质量；在相同 $\dim(\tau)$ 下，DDPM（$\eta=1$ 与 $\hat{\sigma}$）通常比更不随机的对应物更差，唯一例外是 Ho et al. 2020 报告的 $\dim(\tau)=1000$ 且用 $\hat{\sigma}$ 时 DDIM 略差。但 $\hat{\sigma}$ 在更小 $\dim(\tau)$ 时质量急剧变差，说明它不适合更短轨迹。相较之下，DDIM 能更稳定地达到高质量。

In Figure 3, we show CIFAR10 and CelebA samples with the same number of sampling steps and varying $\sigma$. For the DDPM, the sample quality deteriorates rapidly when the sampling trajectory has 10 steps. For the case of $\hat{\sigma}$, the generated images seem to have more noisy perturbations under short trajectories; this explains why the FID scores are much worse than other methods, as FID is very sensitive to such perturbations (as discussed in Jolicoeur-Martineau et al. 2020).

图 3 展示相同采样步数、不同 $\sigma$ 下的 CIFAR10 与 CelebA 样本。对 DDPM，采样轨迹只有 10 步时质量迅速恶化。对 $\hat{\sigma}$，短轨迹下生成图像似乎带有更多噪声扰动；这解释了为何 FID 远差于其他方法，因为 FID 对此类扰动非常敏感（见 Jolicoeur-Martineau et al. 2020 讨论）。

In Figure 4, we show that the amount of time needed to produce a sample scales linearly with the length of the sample trajectory. This suggests that DDIM is useful for producing samples more efficiently, as samples can be generated in much fewer steps. Notably, DDIM is able to produce samples with quality comparable to 1000 step models within $20$ to $100$ steps, which is a $10\times$ to $50\times$ speed up compared to the original DDPM. Even though DDPM could also achieve reasonable sample quality with $100\times$ steps, DDIM requires much fewer steps to achieve this; on CelebA, the FID score of the 100 step DDPM is similar to that of the 20 step DDIM.

图 4 表明产出一个样本所需时间随采样轨迹长度线性增长。这说明 DDIM 有助于更高效产出样本，因为可用少得多的步数生成。值得注意的是，DDIM 能在 $20$ 到 $100$ 步内产出与 1000 步模型可比的质量，相对原始 DDPM 加速 $10\times$ 到 $50\times$。尽管 DDPM 用 $100\times$ 步也能达到合理质量，DDIM 达到该质量所需步数少得多；在 CelebA 上，100 步 DDPM 的 FID 与 20 步 DDIM 相近。

		CIFAR10 ($32\times 32$)	CelebA ($64\times 64$)
$S$	10	20	50	100	1000	10	20	50	100	1000
$\eta$	$0.0$	13.36	6.84	4.67	4.16	4.04	17.33	13.73	9.17	6.53	3.51
0.2	14.04	7.11	4.77	4.25	4.09	17.66	14.11	9.51	6.79	3.64
0.5	16.66	8.35	5.25	4.46	4.29	19.86	16.06	11.01	8.09	4.28
1.0	41.07	18.36	8.01	5.78	4.73	33.12	26.03	18.48	13.93	5.98
$\hat{\sigma}$	367.43	133.37	32.72	9.99	3.17	299.71	183.83	71.71	45.20	3.26

Table 1: CIFAR10 and CelebA image generation measured in FID. $\eta=1.0$ and $\hat{\sigma}$ are cases of DDPM (although Ho et al. 2020 only considered $T=1000$ steps, and $S<T$ can be seen as simulating DDPMs trained with $S$ steps), and $\eta=0.0$ indicates DDIM.

		CIFAR10 ($32\times 32$)	CelebA ($64\times 64$)
$S$	10	20	50	100	1000	10	20	50	100	1000
$\eta$	$0.0$	13.36	6.84	4.67	4.16	4.04	17.33	13.73	9.17	6.53	3.51
0.2	14.04	7.11	4.77	4.25	4.09	17.66	14.11	9.51	6.79	3.64
0.5	16.66	8.35	5.25	4.46	4.29	19.86	16.06	11.01	8.09	4.28
1.0	41.07	18.36	8.01	5.78	4.73	33.12	26.03	18.48	13.93	5.98
$\hat{\sigma}$	367.43	133.37	32.72	9.99	3.17	299.71	183.83	71.71	45.20	3.26

表 1：CIFAR10 与 CelebA 图像生成的 FID。$\eta=1.0$ 与 $\hat{\sigma}$ 为 DDPM 情形（尽管 Ho et al. 2020 只考虑了 $T=1000$ 步，$S<T$ 可视为模拟以 $S$ 步训练的 DDPM），$\eta=0.0$ 表示 DDIM。

Figure 3: CIFAR10 and CelebA samples with $\dim(\tau)=10$ and $\dim(\tau)=100$.

图 3：$\dim(\tau)=10$ 与 $\dim(\tau)=100$ 时的 CIFAR10 与 CelebA 样本。

Figure 4: Hours to sample 50k images with one Nvidia 2080 Ti GPU and samples at different steps.

图 4：在一块 Nvidia 2080 Ti GPU 上采样 5 万张图像所需小时数，以及不同步数下的样本。

5.2 Sample consistency in DDIMs

5.2 DDIM 中的样本一致性

For DDIM, the generative process is deterministic, and ${\bm{x}}_{0}$ would depend only on the initial state ${\bm{x}}_{T}$. In Figure 5, we observe the generated images under different generative trajectories (i.e. different $\tau$) while starting with the same initial ${\bm{x}}_{T}$. Interestingly, for the generated images with the same initial ${\bm{x}}_{T}$, most high-level features are similar, regardless of the generative trajectory. In many cases, samples generated with only 20 steps are already very similar to ones generated with 1000 steps in terms of high-level features, with only minor differences in details. Therefore, it would appear that ${\bm{x}}_{T}$ alone would be an informative latent encoding of the image; and minor details that affects sample quality are encoded in the parameters, as longer sample trajectories gives better quality samples but do not significantly affect the high-level features. We show more samples in Appendix D.4.

对 DDIM，生成过程是确定性的，${\bm{x}}_{0}$ 只依赖初始状态 ${\bm{x}}_{T}$。图 5 中，我们观察从同一初始 ${\bm{x}}_{T}$ 出发、不同生成轨迹（即不同 $\tau$）下的生成图像。有趣的是，对相同初始 ${\bm{x}}_{T}$ 的生成图像，无论生成轨迹如何，多数高层特征相似。许多情形下，仅 20 步生成的样本在高层特征上已与 1000 步非常接近，仅细节有小差别。因此，${\bm{x}}_{T}$ 本身似乎就是图像的信息性潜编码；而影响样本质量的次要细节编码在参数中，因为更长采样轨迹给出更好质量，却不显著改变高层特征。更多样本见附录 D.4。

Figure 5: Samples from DDIM with the same random ${\bm{x}}_{T}$ and different number of steps.

图 5：同一随机 ${\bm{x}}_{T}$、不同步数下的 DDIM 样本。

5.3 Interpolation in deterministic generative processes

5.3 确定性生成过程中的插值

Figure 6: Interpolation of samples from DDIM with $\dim(\tau)=50$.

图 6：$\dim(\tau)=50$ 时 DDIM 样本的插值。

Since the high level features of the DDIM sample is encoded by ${\bm{x}}_{T}$, we are interested to see whether it would exhibit the semantic interpolation effect similar to that observed in other implicit probabilistic models, such as GANs (Goodfellow et al. 2014). This is different from the interpolation procedure in Ho et al. 2020, since in DDPM the same ${\bm{x}}_{T}$ would lead to highly diverse ${\bm{x}}_{0}$ due to the stochastic generative process[^6]. In Figure 6, we show that simple interpolations in ${\bm{x}}_{T}$ can lead to semantically meaningful interpolations between two samples. We include more details and samples in Appendix D.5. This allows DDIM to control the generated images on a high level directly through the latent variables, which DDPMs cannot.

由于 DDIM 样本的高层特征由 ${\bm{x}}_{T}$ 编码，我们关心它是否会表现出与 GAN (Goodfellow et al. 2014) 等其他隐式概率模型类似的语义插值效应。这不同于 Ho et al. 2020 中的插值程序：在 DDPM 中，因生成过程随机，同一 ${\bm{x}}_{T}$ 会导致高度多样的 ${\bm{x}}_{0}$[^6]。图 6 表明，在 ${\bm{x}}_{T}$ 上做简单插值即可得到两样本之间语义有意义的插值。更多细节与样本见附录 D.5。这使得 DDIM 能通过潜变量在高层直接控制生成图像，而 DDPM 做不到。

5.4 Reconstruction from Latent Space

5.4 从潜空间重建

As DDIM is the Euler integration for a particular ODE, it would be interesting to see whether it can encode from ${\bm{x}}_{0}$ to ${\bm{x}}_{T}$ (reverse of Eq. (14)) and reconstruct ${\bm{x}}_{0}$ from the resulting ${\bm{x}}_{T}$ (forward of Eq. (14))[^7]. We consider encoding and decoding on the CIFAR-10 test set with the CIFAR-10 model with $S$ steps for both encoding and decoding; we report the per-dimension mean squared error (scaled to $[0,1]$) in Table 2. Our results show that DDIMs have lower reconstruction error for larger $S$ values and have properties similar to Neural ODEs and normalizing flows. The same cannot be said for DDPMs due to their stochastic nature.

由于 DDIM 是某 ODE 的欧拉积分，值得看它能否从 ${\bm{x}}_{0}$ 编码到 ${\bm{x}}_{T}$（式 (14) 的逆）并从所得 ${\bm{x}}_{T}$ 重建 ${\bm{x}}_{0}$（式 (14) 的正演）[^7]。我们在 CIFAR-10 测试集上，用 CIFAR-10 模型以 $S$ 步做编码与解码；表 2 报告缩放到 $[0,1]$ 的逐维均方误差。结果表明：更大的 $S$ 下 DDIM 重建误差更低，并具有类似 Neural ODE 与归一化流的性质。DDPM 因随机性不能如是说。

$S$	10	20	50	100	200	500	1000
Error	0.014	0.0065	0.0023	0.0009	0.0004	0.0001	$0.0001$

Table 2: Reconstruction error with DDIM on CIFAR-10 test set, rounded to $10^{-4}$.