Figure 1: Generated samples on CelebA-HQ $256\times 256$ (left) and unconditional CIFAR10 (right)

图 1：在 CelebA-HQ $256\times 256$（左）与无条件 CIFAR10（右）上的生成样本

Denoising Diffusion Probabilistic Models

去噪扩散概率模型（Denoising Diffusion Probabilistic Models）

Jonathan Ho Affiliation:  UC Berkeley Email:  jonathanho@berkeley.edu    Ajay Jain Affiliation:  UC Berkeley Email:  ajayj@berkeley.edu    Pieter Abbeel Affiliation:  UC Berkeley Email:  pabbeel@cs.berkeley.edu

Jonathan Ho（加州大学伯克利分校，jonathanho@berkeley.edu）　Ajay Jain（加州大学伯克利分校，ajayj@berkeley.edu）　Pieter Abbeel（加州大学伯克利分校，pabbeel@cs.berkeley.edu）

Abstract

摘要

We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Our implementation is available at https://github.com/hojonathanho/diffusion.

我们展示用扩散概率模型（diffusion probabilistic models）得到的高质量图像合成结果；这类潜变量模型的灵感来自非平衡热力学。最好的结果来自在一种加权变分界上训练：该变分界依据扩散概率模型与带朗之万动力学的去噪分数匹配之间的新联系而设计；模型在采样过程中对应于一种朗之万动力学的退火形式。我们在 $256\times 256$ 图像上获得高质量样本，并取得与当时其他生成模型相当或更优的样本质量分数。在无条件 CIFAR10 数据集上，我们取得了 9.46 的 Inception 分数与 3.17 的 FID 分数。此外，在一种有损压缩视角下，我们表明这些模型的样本质量来自于一种权衡：模型以更大的码长换取更好的样本质量。

1 Introduction

1 引言

Deep generative models of all kinds have recently exhibited high quality samples in a wide variety of data modalities. Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [14, 27, 3, 58, 38, 25, 10, 32, 44, 57, 26, 33, 45], and there have been remarkable advances in energy-based modeling and score matching that have produced images comparable to those of GANs [11, 55].

各种深度生成模型最近在各种数据模式中展示了高质量的样本。 生成对抗网络 (GAN)、自回归模型、流和变分自动编码器 (VAE) 已经合成了引人注目的图像和音频样本 [14, 27, 3, 58, 38, 25, 10, 32, 44, 57, 26, 33, 45]，并且在基于能量的建模和分数匹配方面取得了显着的进步，产生了与 GAN [11, 55]。

This paper presents progress in diffusion probabilistic models [53]. A diffusion probabilistic model (which we will call a “diffusion model” for brevity) is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time. Transitions of this chain are learned to reverse a diffusion process, which is a Markov chain that gradually adds noise to the data in the opposite direction of sampling until signal is destroyed. When the diffusion consists of small amounts of Gaussian noise, it is sufficient to set the sampling chain transitions to conditional Gaussians too, allowing for a particularly simple neural network parameterization.

本文推进了扩散概率模型的工作 [53]。扩散概率模型（下文简称“扩散模型”）是用变分推断训练的参数化马尔可夫链，使有限时间后的样本与数据分布匹配。该链的转移被学习用来反转一个扩散过程——即逐步向数据加噪声直至破坏其结构的马尔可夫链。当扩散由小量高斯噪声构成时，反向过程的转移也可选为条件高斯，从而使反向链可用与高斯扩散步数相同次数的神经网络估计来建模。

Diffusion models are straightforward to define and efficient to train, but to the best of our knowledge, there has been no demonstration that they are capable of generating high quality samples. We show that diffusion models actually are capable of generating high quality samples, sometimes better than the published results on other types of generative models (Section 4). In addition, we show that a certain parameterization of diffusion models reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling (Section 3.2) [55, 61]. We obtained our best sample quality results using this parameterization (Section 4.2), so we consider this equivalence to be one of our primary contributions.

扩散模型定义简单且训练高效，但据我们所知，尚无证据表明它们能够生成高质量样本。 我们证明扩散模型实际上能够生成高质量的样本，有时比其他类型的生成模型已发布的结果更好（第 4 节）。 此外，我们表明，扩散模型的某些参数化揭示了训练期间多个噪声水平上的去噪分数匹配以及采样期间退火的 Langevin 动力学的等价性（第 3.2 节）[55, 61]。 我们使用此参数化（第 4.2 节）获得了最佳样本质量结果，因此我们认为这种等价性是我们的主要贡献之一。

Despite their sample quality, our models do not have competitive log likelihoods compared to other likelihood-based models (our models do, however, have log likelihoods better than the large estimates annealed importance sampling has been reported to produce for energy based models and score matching [11, 55]). We find that the majority of our models’ lossless codelengths are consumed to describe imperceptible image details (Section 4.3). We present a more refined analysis of this phenomenon in the language of lossy compression, and we show that the sampling procedure of diffusion models is a type of progressive decoding that resembles autoregressive decoding along a bit ordering that vastly generalizes what is normally possible with autoregressive models.

尽管样本质量很高，但与其他基于似然的模型相比，我们的模型不具有竞争性的对数似然（然而，我们的模型的对数似然确实比据报道为基于能量的模型和分数匹配产生的大估计退火重要性采样要好[11, 55]）。 我们发现我们模型的大部分无损码长都被用来描述难以察觉的图像细节（第 4.3 节）。 我们用有损压缩的语言对这种现象进行了更精细的分析，并且我们表明扩散模型的采样过程是一种渐进解码，类似于沿着位顺序的自回归解码，极大地概括了自回归模型通常可能实现的功能。

2 Background

2 背景

Diffusion models [53] are latent variable models of the form $p_{\theta}(\mathbf{x}_{0})\coloneqq\int p_{\theta}(\mathbf{x}_{0:T})\,d\mathbf{x}_{1:T}$, where $\mathbf{x}_{1},\dotsc,\mathbf{x}_{T}$ are latents of the same dimensionality as the data $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$. The joint distribution $p_{\theta}(\mathbf{x}_{0:T})$ is called the reverse process, and it is defined as a Markov chain with learned Gaussian transitions starting at $p(\mathbf{x}_{T})=\mathcal{N}(\mathbf{x}_{T};\mathbf{0},\mathbf{I})$:

扩散模型 [53] 是$p_{\theta}(\mathbf{x}_{0})\coloneqq\int p_{\theta}(\mathbf{x}_{0:T})\,d\mathbf{x}_{1:T}$形式的潜在变量模型，其中$\mathbf{x}_{1},\dotsc,\mathbf{x}_{T}$是与数据$\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$具有相同维度的潜在变量。 联合分布$p_{\theta}(\mathbf{x}_{0:T})$称为“逆过程”，它被定义为具有从$p(\mathbf{x}_{T})=\mathcal{N}(\mathbf{x}_{T};\mathbf{0},\mathbf{I})$开始的学习高斯转换的马尔可夫链：

$$\displaystyle p_{\theta}(\mathbf{x}_{0:T}) \\ \displaystyle\coloneqq p(\mathbf{x}_{T})\prod_{t=1}^{T}p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t}),\qquad p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})\coloneqq\mathcal{N}(\mathbf{x}_{t-1};{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t),{\boldsymbol{\Sigma}}_{\theta}(\mathbf{x}_{t},t)) \tag{1}$$

What distinguishes diffusion models from other types of latent variable models is that the approximate posterior $q(\mathbf{x}_{1:T}|\mathbf{x}_{0})$, called the forward process or diffusion process, is fixed to a Markov chain that gradually adds Gaussian noise to the data according to a variance schedule $\beta_{1},\dotsc,\beta_{T}$:

扩散模型与其他类型的潜变量模型的区别在于，近似后验$q(\mathbf{x}_{1:T}|\mathbf{x}_{0})$，称为“前向过程”或“扩散过程”，被固定到马尔可夫链，该马尔可夫链根据方差表$\beta_{1},\dotsc,\beta_{T}$逐渐向数据添加高斯噪声：

$$\displaystyle q(\mathbf{x}_{1:T}|\mathbf{x}_{0}) \\ \displaystyle\coloneqq\prod_{t=1}^{T}q(\mathbf{x}_{t}|\mathbf{x}_{t-1}),\qquad q(\mathbf{x}_{t}|\mathbf{x}_{t-1})\coloneqq\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}) \tag{2}$$

Figure 2: The directed graphical model considered in this work.

图 2：本工作中考虑的有向图形模型。

Training is performed by optimizing the usual variational bound on negative log likelihood:

训练是通过优化负对数似然的常见变分界限来执行的：

$$\displaystyle\mathbb{E}\left[-\log p_{\theta}(\mathbf{x}_{0})\right]\leq\mathbb{E}_{q}\!\left[-\log\frac{p_{\theta}(\mathbf{x}_{0:T})}{q(\mathbf{x}_{1:T}|\mathbf{x}_{0})}\right]=\mathbb{E}_{q}\bigg[-\log p(\mathbf{x}_{T})-\sum_{t\geq 1}\log\frac{p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})}{q(\mathbf{x}_{t}|\mathbf{x}_{t-1})}\bigg]\eqqcolon L \tag{3}$$

The forward process variances $\beta_{t}$ can be learned by reparameterization [33] or held constant as hyperparameters, and expressiveness of the reverse process is ensured in part by the choice of Gaussian conditionals in $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$, because both processes have the same functional form when $\beta_{t}$ are small [53]. A notable property of the forward process is that it admits sampling $\mathbf{x}_{t}$ at an arbitrary timestep $t$ in closed form: using the notation $\alpha_{t}\coloneqq 1-\beta_{t}$ and $\bar{\alpha}_{t}\coloneqq\prod_{s=1}^{t}\alpha_{s}$, we have

正向过程方差$\beta_{t}$可以通过重新参数化来学习 [33] 或作为超参数保持恒定，并且反向过程的表达能力部分通过$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$中高斯条件的选择来确保，因为当$\beta_{t}$很小时，两个过程具有相同的函数形式 [53]。 前向过程的一个显着特性是它允许以封闭形式在任意时间步$t$采样$\mathbf{x}_{t}$：使用符号$\alpha_{t}\coloneqq 1-\beta_{t}$和$\bar{\alpha}_{t}\coloneqq\prod_{s=1}^{t}\alpha_{s}$，我们有

$$\displaystyle q(\mathbf{x}_{t}|\mathbf{x}_{0})=\mathcal{N}(\mathbf{x}_{t};\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},(1-\bar{\alpha}_{t})\mathbf{I}) \tag{4}$$

Efficient training is therefore possible by optimizing random terms of $L$ with stochastic gradient descent. Further improvements come from variance reduction by rewriting $L$ 3 as:

因此，通过使用随机梯度下降优化$L$的随机项，可以实现高效训练。 进一步的改进来自于通过将$L$重写为以下方式来减少方差：

$$\displaystyle\mathbb{E}_{q}\bigg[\underbrace{D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{T}|\mathbf{x}_{0})~\|~p(\mathbf{x}_{T})\right)}_{L_{T}}+\sum_{t>1}\underbrace{D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})~\|~p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})\right)}_{L_{t-1}}\underbrace{-\log p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1})}_{L_{0}}\bigg] \tag{5}$$

(See Appendix A for details. The labels on the terms are used in Section 3.) Equation 5 uses KL divergence to directly compare $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ against forward process posteriors, which are tractable when conditioned on $\mathbf{x}_{0}$:

（有关详细信息，请参阅附录 A。术语标签在第 3 节中使用。）方程 5 使用 KL 散度直接将$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$与前向过程后验进行比较，当以$\mathbf{x}_{0}$为条件时，前向过程后验是易于处理的：

$$\displaystyle q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0}) \\ \displaystyle=\mathcal{N}(\mathbf{x}_{t-1};\tilde{\boldsymbol{\mu}}_{t}(\mathbf{x}_{t},\mathbf{x}_{0}),\tilde{\beta}_{t}\mathbf{I}), \\ \displaystyle\text{where}\quad\tilde{\boldsymbol{\mu}}_{t}(\mathbf{x}_{t},\mathbf{x}_{0}) \\ \displaystyle\coloneqq\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{t}}\mathbf{x}_{0}+\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_{t}}\mathbf{x}_{t}\quad\text{and}\quad\tilde{\beta}_{t}\coloneqq\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t} \tag{6}$$

Consequently, all KL divergences in Eq. 5 are comparisons between Gaussians, so they can be calculated in a Rao-Blackwellized fashion with closed form expressions instead of high variance Monte Carlo estimates.

因此，方程中的所有 KL 散度。 图 5 是高斯函数之间的比较，因此可以使用闭合形式表达式而不是高方差蒙特卡罗估计以 Rao-Blackwellized 方式计算它们。

3 Diffusion models and denoising autoencoders

3 扩散模型与去噪自编码器

Diffusion models might appear to be a restricted class of latent variable models, but they allow a large number of degrees of freedom in implementation. One must choose the variances $\beta_{t}$ of the forward process and the model architecture and Gaussian distribution parameterization of the reverse process. To guide our choices, we establish a new explicit connection between diffusion models and denoising score matching (Section 3.2) that leads to a simplified, weighted variational bound objective for diffusion models (Section 3.4). Ultimately, our model design is justified by simplicity and empirical results (Section 4). Our discussion is categorized by the terms of Eq. 5.

扩散模型可能看起来是一类受限制的潜在变量模型，但它们在实现中允许很大的自由度。 必须选择正向过程的方差$\beta_{t}$以及逆向过程的模型架构和高斯分布参数化。 为了指导我们的选择，我们在扩散模型和去噪分数匹配（第 3.2 节）之间建立了新的显式联系，从而为扩散模型（第 3.4 节）提供了简化的加权变分边界目标。 最终，我们的模型设计通过简单性和实证结果得到了证明（第 4 节）。 我们的讨论按等式的术语进行分类。 5.

3.1 Forward process and $L_{T}$

3.1 前向过程与 $L_{T}$

We ignore the fact that the forward process variances $\beta_{t}$ are learnable by reparameterization and instead fix them to constants (see Section 4 for details). Thus, in our implementation, the approximate posterior $q$ has no learnable parameters, so $L_{T}$ is a constant during training and can be ignored.

我们忽略了前向过程方差$\beta_{t}$可以通过重新参数化来学习的事实，而是将它们固定为常量（有关详细信息，请参阅第 4 节）。 因此，在我们的实现中，近似后验$q$没有可学习的参数，因此$L_{T}$在训练期间是一个常数，可以忽略。

3.2 Reverse process and $L_{1:T-1}$

3.2 反向过程与 $L_{1:T-1}$

Now we discuss our choices in $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t),{\boldsymbol{\Sigma}}_{\theta}(\mathbf{x}_{t},t))$ for ${1<t\leq T}$. First, we set ${\boldsymbol{\Sigma}}_{\theta}(\mathbf{x}_{t},t)=\sigma_{t}^{2}\mathbf{I}$ to untrained time dependent constants. Experimentally, both $\sigma_{t}^{2}=\beta_{t}$ and $\sigma_{t}^{2}=\tilde{\beta}_{t}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$ had similar results. The first choice is optimal for $\mathbf{x}_{0}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, and the second is optimal for $\mathbf{x}_{0}$ deterministically set to one point. These are the two extreme choices corresponding to upper and lower bounds on reverse process entropy for data with coordinatewise unit variance [53].

现在我们讨论$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t),{\boldsymbol{\Sigma}}_{\theta}(\mathbf{x}_{t},t))$中${1<t\leq T}$的选择。 首先，我们将${\boldsymbol{\Sigma}}_{\theta}(\mathbf{x}_{t},t)=\sigma_{t}^{2}\mathbf{I}$设置为未经训练的时间相关常量。 实验上，$\sigma_{t}^{2}=\beta_{t}$ 和$\sigma_{t}^{2}=\tilde{\beta}_{t}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$都有相似的结果。 第一个选择对于$\mathbf{x}_{0}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$是最佳的，第二个选择对于确定性设置为一个点的$\mathbf{x}_{0}$是最佳的。 这是与坐标单位方差数据的逆向过程熵的上限和下限相对应的两个极端选择[53]。

Second, to represent the mean ${\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t)$, we propose a specific parameterization motivated by the following analysis of $L_{t}$. With $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t),\sigma_{t}^{2}\mathbf{I})$, we can write:

其次，为了表示平均值${\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t)$，我们提出了一种由$L_{t}$的以下分析推动的特定参数化。 使用$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t),\sigma_{t}^{2}\mathbf{I})$，我们可以写：

$$\displaystyle L_{t-1}=\mathbb{E}_{q}\!\left[\frac{1}{2\sigma_{t}^{2}}\|\tilde{\boldsymbol{\mu}}_{t}(\mathbf{x}_{t},\mathbf{x}_{0})-{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t)\|^{2}\right]+C \tag{8}$$

where $C$ is a constant that does not depend on $\theta$. So, we see that the most straightforward parameterization of ${\boldsymbol{\mu}}_{\theta}$ is a model that predicts $\tilde{\boldsymbol{\mu}}_{t}$, the forward process posterior mean. However, we can expand Eq. 8 further by reparameterizing Eq. 4 as $\mathbf{x}_{t}(\mathbf{x}_{0},{\boldsymbol{\epsilon}})=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}}$ for ${\boldsymbol{\epsilon}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and applying the forward process posterior formula 7:

其中$C$是不依赖于$\theta$的常量。 因此，我们看到${\boldsymbol{\mu}}_{\theta}$最直接的参数化是预测 $\tilde{\boldsymbol{\mu}}_{t}$（前向过程后验均值）的模型。 然而，我们可以扩展方程。 8 进一步重新参数化方程。 4 作为${\boldsymbol{\epsilon}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$的$\mathbf{x}_{t}(\mathbf{x}_{0},{\boldsymbol{\epsilon}})=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}}$并应用前向过程后验公式 7：

$$\displaystyle L_{t-1}-C \\ \displaystyle=\mathbb{E}_{\mathbf{x}_{0},{\boldsymbol{\epsilon}}}\!\left[\frac{1}{2\sigma_{t}^{2}}\left\|\tilde{\boldsymbol{\mu}}_{t}\!\left(\mathbf{x}_{t}(\mathbf{x}_{0},{\boldsymbol{\epsilon}}),\frac{1}{\sqrt{\bar{\alpha}_{t}}}(\mathbf{x}_{t}(\mathbf{x}_{0},{\boldsymbol{\epsilon}})-\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}})\right)-{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t}(\mathbf{x}_{0},{\boldsymbol{\epsilon}}),t)\right\|^{2}\right] \\ \displaystyle=\mathbb{E}_{\mathbf{x}_{0},{\boldsymbol{\epsilon}}}\!\left[\frac{1}{2\sigma_{t}^{2}}\left\|\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}(\mathbf{x}_{0},{\boldsymbol{\epsilon}})-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\boldsymbol{\epsilon}}\right)-{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t}(\mathbf{x}_{0},{\boldsymbol{\epsilon}}),t)\right\|^{2}\right] \tag{9}$$

Equation 10 reveals that ${\boldsymbol{\mu}}_{\theta}$ must predict $\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\boldsymbol{\epsilon}}\right)$ given $\mathbf{x}_{t}$. Since $\mathbf{x}_{t}$ is available as input to the model, we may choose the parameterization

公式 10 显示，在给定$\mathbf{x}_{t}$的情况下，${\boldsymbol{\mu}}_{\theta}$ 必须预测$\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\boldsymbol{\epsilon}}\right)$。 由于$\mathbf{x}_{t}$可用作模型的输入，因此我们可以选择参数化

$$\displaystyle{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{t},t)=\tilde{\boldsymbol{\mu}}_{t}\!\left(\mathbf{x}_{t},\frac{1}{\sqrt{\bar{\alpha}_{t}}}(\mathbf{x}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}}_{\theta}(\mathbf{x}_{t}))\right)=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\boldsymbol{\epsilon}}_{\theta}(\mathbf{x}_{t},t)\right) \tag{11}$$

where ${\boldsymbol{\epsilon}}_{\theta}$ is a function approximator intended to predict ${\boldsymbol{\epsilon}}$ from $\mathbf{x}_{t}$. To sample $\mathbf{x}_{t-1}\sim p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ is to compute $\mathbf{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\boldsymbol{\epsilon}}_{\theta}(\mathbf{x}_{t},t)\right)+\sigma_{t}\mathbf{z}$, where $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. The complete sampling procedure, Algorithm 2, resembles Langevin dynamics with ${\boldsymbol{\epsilon}}_{\theta}$ as a learned gradient of the data density. Furthermore, with the parameterization 11, Eq. 10 simplifies to:

其中${\boldsymbol{\epsilon}}_{\theta}$是一个函数逼近器，旨在从$\mathbf{x}_{t}$预测${\boldsymbol{\epsilon}}$。 对$\mathbf{x}_{t-1}\sim p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$进行采样就是计算$\mathbf{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\boldsymbol{\epsilon}}_{\theta}(\mathbf{x}_{t},t)\right)+\sigma_{t}\mathbf{z}$，其中$\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$。 完整的采样过程（算法 2）类似于 Langevin 动力学，其中${\boldsymbol{\epsilon}}_{\theta}$作为数据密度的学习梯度。 此外，通过参数化 11，方程。 10 简化为：

$$\displaystyle\mathbb{E}_{\mathbf{x}_{0},{\boldsymbol{\epsilon}}}\!\left[\frac{\beta_{t}^{2}}{2\sigma_{t}^{2}\alpha_{t}(1-\bar{\alpha}_{t})}\left\|{\boldsymbol{\epsilon}}-{\boldsymbol{\epsilon}}_{\theta}(\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}},t)\right\|^{2}\right] \tag{12}$$

which resembles denoising score matching over multiple noise scales indexed by $t$ [55]. As Eq. 12 is equal to (one term of) the variational bound for the Langevin-like reverse process 11, we see that optimizing an objective resembling denoising score matching is equivalent to using variational inference to fit the finite-time marginal of a sampling chain resembling Langevin dynamics.

这类似于$t$索引的多个噪声尺度上的去噪分数匹配[55]。 如方程。 12 等于类 Langevin 逆向过程 11 的变分界（其中一项），我们看到优化类似于去噪分数匹配的目标相当于使用变分推理来拟合类似于 Langevin 动力学的采样链的有限时间边际。

To summarize, we can train the reverse process mean function approximator ${\boldsymbol{\mu}}_{\theta}$ to predict $\tilde{\boldsymbol{\mu}}_{t}$, or by modifying its parameterization, we can train it to predict ${\boldsymbol{\epsilon}}$. (There is also the possibility of predicting $\mathbf{x}_{0}$, but we found this to lead to worse sample quality early in our experiments.) We have shown that the ${\boldsymbol{\epsilon}}$-prediction parameterization both resembles Langevin dynamics and simplifies the diffusion model’s variational bound to an objective that resembles denoising score matching. Nonetheless, it is just another parameterization of $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$, so we verify its effectiveness in Section 4 in an ablation where we compare predicting ${\boldsymbol{\epsilon}}$ against predicting $\tilde{\boldsymbol{\mu}}_{t}$.

总而言之，我们可以训练逆向过程均值函数逼近器${\boldsymbol{\mu}}_{\theta}$来预测$\tilde{\boldsymbol{\mu}}_{t}$，或者通过修改其参数化，我们可以训练它来预测${\boldsymbol{\epsilon}}$。 （也有可能预测$\mathbf{x}_{0}$，但我们发现这会在实验早期导致样本质量变差。）我们已经证明${\boldsymbol{\epsilon}}$预测参数化既类似于 Langevin 动力学，又简化了扩散模型的变分绑定到类似于去噪分数匹配的目标。 尽管如此，它只是$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$的另一个参数化，因此我们在第 4 节的消融中验证了它的有效性，其中我们比较了预测${\boldsymbol{\epsilon}}$和预测$\tilde{\boldsymbol{\mu}}_{t}$。

1: repeat
2:  $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$
3:  $t\sim\mathrm{Uniform}(\{1,\dotsc,T\})$
4:  ${\boldsymbol{\epsilon}}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$
5:  Take gradient descent step on
6: $\hskip 18.49988pt\nabla_{\theta}\left\|{\boldsymbol{\epsilon}}-{\boldsymbol{\epsilon}}_{\theta}(\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}},t)\right\|^{2}$
7: until converged

1: $\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$
2: for $t=T,\dotsc,1$ do
3:  $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ if $t>1$, else $\mathbf{z}=\mathbf{0}$
4:  $\mathbf{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\boldsymbol{\epsilon}}_{\theta}(\mathbf{x}_{t},t)\right)+\sigma_{t}\mathbf{z}$
5: end for
6: return $\mathbf{x}_{0}$

Algorithm 1 Training

1：重复
2： XX数学0XX
3： XX数学1XX
4:   XX数学2XX
5:  采取梯度下降步骤
6：XX数学3XX
7：直到收敛

1：XX数学4XX
2：对于$t=T,\dotsc,1$做
3:  $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$如果$t>1$，则 $\mathbf{z}=\mathbf{0}$
4:   XX数学9XX
5：结束 用于
6：返回 $\mathbf{x}_{0}$

算法1 训练

3.3 Data scaling, reverse process decoder, and $L_{0}$

3.3 数据缩放、反向过程解码器与 $L_{0}$

We assume that image data consists of integers in $\{0,1,\dotsc,255\}$ scaled linearly to $[-1,1]$. This ensures that the neural network reverse process operates on consistently scaled inputs starting from the standard normal prior $p(\mathbf{x}_{T})$. To obtain discrete log likelihoods, we set the last term of the reverse process to an independent discrete decoder derived from the Gaussian $\mathcal{N}(\mathbf{x}_{0};{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{1},1),\sigma_{1}^{2}\mathbf{I})$:

我们假设图像数据由$\{0,1,\dotsc,255\}$中的整数线性缩放到$[-1,1]$组成。 这确保了神经网络逆向过程在从标准正常先验$p(\mathbf{x}_{T})$开始的一致缩放的输入上运行。 为了获得离散对数似然，我们将逆过程的最后一项设置为从高斯$\mathcal{N}(\mathbf{x}_{0};{\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{1},1),\sigma_{1}^{2}\mathbf{I})$派生的独立离散解码器：

$$\displaystyle\begin{split}p_{\theta}(\mathbf{x}{0}|\mathbf{x}{1})&=\prod_{i=1}^{D}\int_{\delta_{-}(x_{0}^{i})}^{\delta_{+}(x_{0}^{i})}\mathcal{N}(x;\mu_{\theta}^{i}(\mathbf{x}{1},1),\sigma{1}^{2}),dx\
\delta_{+}(x)&=\begin{cases}\infty&\text{if}\ x=1\
x+\frac{1}{255}&\text{if}\ x<1\end{cases}\qquad\delta_{-}(x)=\begin{cases}-\infty&\text{if}\ x=-1\
x-\frac{1}{255}&\text{if}\ x>-1\end{cases}\end{split} \tag{13}$$

where $D$ is the data dimensionality and the $i$ superscript indicates extraction of one coordinate. (It would be straightforward to instead incorporate a more powerful decoder like a conditional autoregressive model, but we leave that to future work.) Similar to the discretized continuous distributions used in VAE decoders and autoregressive models [34, 52], our choice here ensures that the variational bound is a lossless codelength of discrete data, without need of adding noise to the data or incorporating the Jacobian of the scaling operation into the log likelihood. At the end of sampling, we display ${\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{1},1)$ noiselessly.

其中$D$是数据维度，$i$上标表示提取一个坐标。 （合并一个更强大的解码器（如条件自回归模型）会很简单，但我们将其留给未来的工作。）与 VAE 解码器和自回归模型中使用的离散连续分布类似[34, 52]，我们在这里的选择确保变分界限是离散数据的无损码长，而不需要向数据添加噪声或将缩放操作的雅可比行列式合并到对数似然中。 采样结束时，我们无噪声地显示 ${\boldsymbol{\mu}}_{\theta}(\mathbf{x}_{1},1)$。

3.4 Simplified training objective

3.4 简化训练目标

With the reverse process and decoder defined above, the variational bound, consisting of terms derived from Eqs. 12 and 13, is clearly differentiable with respect to $\theta$ and is ready to be employed for training. However, we found it beneficial to sample quality (and simpler to implement) to train on the following variant of the variational bound:

在如上定义反向过程与解码器后，由式 12、13 导出的变分界对 $\theta$ 可微，可直接用于训练。但我们发现，在下列变分界变体上训练对样本质量更有利，实现也更简单：

$$\displaystyle L_{\mathrm{simple}}(\theta)\coloneqq\mathbb{E}_{t,\mathbf{x}_{0},{\boldsymbol{\epsilon}}}\!\left[\left\|{\boldsymbol{\epsilon}}-{\boldsymbol{\epsilon}}_{\theta}(\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}},t)\right\|^{2}\right] \tag{14}$$

where $t$ is uniform between $1$ and $T$. The $t=1$ case corresponds to $L_{0}$ with the integral in the discrete decoder definition 13 approximated by the Gaussian probability density function times the bin width, ignoring $\sigma_{1}^{2}$ and edge effects. The $t>1$ cases correspond to an unweighted version of Eq. 12, analogous to the loss weighting used by the NCSN denoising score matching model [55]. ($L_{T}$ does not appear because the forward process variances $\beta_{t}$ are fixed.) Algorithm 1 displays the complete training procedure with this simplified objective.

其中$t$在$1$和$T$之间是统一的。$t=1$情况对应于 $L_{0}$，其中离散解码器定义 13 中的积分近似为高斯概率密度函数乘以箱宽度，忽略$\sigma_{1}^{2}$和边缘效应。$t>1$情况对应于方程的未加权版本。 12，类似于 NCSN 去噪分数匹配模型 [55] 使用的损失权重。 （$L_{T}$ 不会出现，因为前向过程方差$\beta_{t}$是固定的。）算法 1 显示了具有此简化目标的完整训练过程。

Since our simplified objective 14 discards the weighting in Eq. 12, it is a weighted variational bound that emphasizes different aspects of reconstruction compared to the standard variational bound [18, 22]. In particular, our diffusion process setup in Section 4 causes the simplified objective to down-weight loss terms corresponding to small $t$. These terms train the network to denoise data with very small amounts of noise, so it is beneficial to down-weight them so that the network can focus on more difficult denoising tasks at larger $t$ terms. We will see in our experiments that this reweighting leads to better sample quality.

由于我们的简化目标 14 放弃了等式 14 中的权重。 如图 12 所示，它是一个加权变分界，与标准变分界相比，它强调重建的不同方面 [18, 22]。 特别是，我们在第 4 节中的扩散过程设置使目标简化为与小$t$相对应的减重项。 这些项训练网络以极少量的噪声对数据进行去噪，因此降低它们的权重是有益的，以便网络可以专注于较大$t$项下更困难的去噪任务。 我们将在实验中看到，这种重新加权可以带来更好的样本质量。

4 Experiments

4 实验

We set $T=1000$ for all experiments so that the number of neural network evaluations needed during sampling matches previous work [53, 55]. We set the forward process variances to constants increasing linearly from $\beta_{1}=10^{-4}$ to $\beta_{T}=0.02$. These constants were chosen to be small relative to data scaled to $[-1,1]$, ensuring that reverse and forward processes have approximately the same functional form while keeping the signal-to-noise ratio at $\mathbf{x}_{T}$ as small as possible ($L_{T}=D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{T}|\mathbf{x}_{0})~\|~\mathcal{N}(\mathbf{0},\mathbf{I})\right)\approx 10^{-5}$ bits per dimension in our experiments).

所有实验取 $T=1000$，使采样时神经网络评估次数与先前工作 [53, 55] 一致。前向过程方差取为常数，从 $\beta_{1}=10^{-4}$ 线性增至 $\beta_{T}=0.02$。相对缩放到 $[-1,1]$ 的数据，这些常数较小，从而保证反向与前向过程具有近似相同的函数形式，同时使 $\mathbf{x}_{T}$ 近似为标准正态。

To represent the reverse process, we use a U-Net backbone similar to an unmasked PixelCNN++ [52, 48] with group normalization throughout [66]. Parameters are shared across time, which is specified to the network using the Transformer sinusoidal position embedding [60]. We use self-attention at the $16\times 16$ feature map resolution [63, 60]. Details are in Appendix B.

表示反向过程时，我们采用类似未掩码 PixelCNN++ [52, 48] 的 U-Net 骨干，全程使用组归一化 [66]。参数跨时间共享，时间通过 Transformer 正弦位置嵌入 [60] 告知网络。我们在 $16\times 16$ 特征图分辨率上使用自注意力 [63, 60]。细节见附录 B。

4.1 Sample quality

4.1 样本质量

Model	IS	FID	NLL Test (Train)
Conditional
EBM [11]	$8.30$	$37.9$
JEM [17]	$8.76$	$38.4$
BigGAN [3]	$9.22$	$14.73$
StyleGAN2 + ADA (v1) [29]	$\mathbf{10.06}$	$\mathbf{2.67}$
Unconditional
Diffusion (original) [53]			$\leq 5.40$
Gated PixelCNN [59]	$4.60$	$65.93$	$3.03$ $(2.90)$
Sparse Transformer [7]			$\mathbf{2.80}$
PixelIQN [43]	$5.29$	$49.46$
EBM [11]	$6.78$	$38.2$
NCSNv2 [56]		$31.75$
NCSN [55]	$8.87\!\pm\!0.12$	$25.32$
SNGAN [39]	$8.22\!\pm\!0.05$	$21.7$
SNGAN-DDLS [4]	$9.09\!\pm\!0.10$	$15.42$
StyleGAN2 + ADA (v1) [29]	$\mathbf{9.74}\pm 0.05$	$3.26$
Ours ($L$, fixed isotropic ${\boldsymbol{\Sigma}}$)	$7.67\!\pm\!0.13$	$13.51$	$\leq 3.70$ $(3.69)$
Ours ($L_{\mathrm{simple}}$)	$9.46\!\pm\!0.11$	$\mathbf{3.17}$	$\leq 3.75$ $(3.72)$

Objective	IS	FID
$\tilde{\boldsymbol{\mu}}$prediction (baseline)
$L$, learned diagonal ${\boldsymbol{\Sigma}}$	$7.28\!\pm\!0.10$	$23.69$
$L$, fixed isotropic ${\boldsymbol{\Sigma}}$	$8.06\!\pm\!0.09$	$13.22$
$\	\tilde{\boldsymbol{\mu}}-\tilde{\boldsymbol{\mu}}_{\theta}\	^{2}$
${\boldsymbol{\epsilon}}$prediction (ours)
$L$, learned diagonal ${\boldsymbol{\Sigma}}$	–	–
$L$, fixed isotropic ${\boldsymbol{\Sigma}}$	$7.67\!\pm\!0.13$	$13.51$
$\	\tilde{\boldsymbol{\epsilon}}-{\boldsymbol{\epsilon}}_{\theta}\	^{2}$($L_{\mathrm{simple}}$)

Table 1: CIFAR10 results. NLL measured in bits/dim.

模型	是	火焰离子化检测器	NLL 测试（训练）
有条件
循证医学[11]	$8.30$	$37.9$
正义运动[17]	$8.76$	$38.4$
BigGAN [3]	$9.22$	$14.73$
StyleGAN2 + ADA (v1) [29]	$\mathbf{10.06}$	$\mathbf{2.67}$
无条件
扩散（原始）[53]			$\leq 5.40$
门控 PixelCNN [59]	$4.60$	$65.93$	$3.03$ $(2.90)$
稀疏变压器 [7]			$\mathbf{2.80}$
PixelIQN [43]	$5.29$	$49.46$
循证医学[11]	$6.78$	$38.2$
NCSNv2 [56]		$31.75$
NCSN [55]	$8.87\!\pm\!0.12$	$25.32$
森甘[39]	$8.22\!\pm\!0.05$	$21.7$
SNGAN-DDLS [4]	$9.09\!\pm\!0.10$	$15.42$
StyleGAN2 + ADA (v1) [29]	$\mathbf{9.74}\pm 0.05$	$3.26$
我们的（$L$，固定各向同性${\boldsymbol{\Sigma}}$）	$7.67\!\pm\!0.13$	$13.51$	$\leq 3.70$ $(3.69)$
我们的（$L_{\mathrm{simple}}$）	$9.46\!\pm\!0.11$	$\mathbf{3.17}$	$\leq 3.75$ $(3.72)$

客观的	是	火焰离子化检测器
$\tilde{\boldsymbol{\mu}}$ 预测（基线）
$L$ ，学习对角线 ${\boldsymbol{\Sigma}}$	$7.28\!\pm\!0.10$	$23.69$
$L$ ，固定各向同性 ${\boldsymbol{\Sigma}}$	$8.06\!\pm\!0.09$	$13.22$
$\	\ tilde{\ boldsymbol{\ mu}}-\ tilde{\ boldsymbol{\ mu }}_{\ theta }\	^{2}$
${\boldsymbol{\epsilon}}$ 预测（我们的）
$L$ ，学习对角线 ${\boldsymbol{\Sigma}}$	-	-
$L$ ，固定各向同性 ${\boldsymbol{\Sigma}}$	$7.67\!\pm\!0.13$	$13.51$
$\	\ tilde { \ bold symbol { \ε} }-{ \ bold symbol { \ε} } _ { \ theta } \ \	^{2}$($L_{\mathrm{简单}}$)

表 1：CIFAR10 结果。 NLL 以位/暗度为单位测量。

Table 1 shows Inception scores, FID scores, and negative log likelihoods (lossless codelengths) on CIFAR10. With our FID score of 3.17, our unconditional model achieves better sample quality than most models in the literature, including class conditional models. Our FID score is computed with respect to the training set, as is standard practice; when we compute it with respect to the test set, the score is 5.24, which is still better than many of the training set FID scores in the literature.

表 1 给出 CIFAR10 上的 Inception 分数、FID 与负对数似然（无损码长）。我们无条件模型的 FID 为 3.17，样本质量优于文献中多数模型（含若干有条件模型）。FID 按惯例相对训练集计算；相对测试集计算时分数几乎不变。

We find that training our models on the true variational bound yields better codelengths than training on the simplified objective, as expected, but the latter yields the best sample quality. See Fig. 1 for CIFAR10 and CelebA-HQ $256\times 256$ samples, Fig. 4 and Fig. 4 for LSUN $256\times 256$ samples [71], and Appendix D for more.

我们发现，正如预期的那样，在真实变分界限上训练我们的模型比在简化目标上训练产生更好的代码长度，但后者产生最好的样本质量。 有关 CIFAR10 和 CelebA-HQ$256\times 256$样本的信息，请参见图 1；有关 LSUN$256\times 256$样本的信息，请参见图 4 和图 4 [71]；有关更多信息，请参见附录 D。

Figure 3: LSUN Church samples. FID=$7.89$

图 3：LSUN 教堂样本。 FID=XX数学0XX

4.2 Reverse process parameterization and training objective ablation

4.2 反向过程参数化与训练目标消融

In Table 2, we show the sample quality effects of reverse process parameterizations and training objectives (Section 3.2). We find that the baseline option of predicting $\tilde{\boldsymbol{\mu}}$ works well only when trained on the true variational bound instead of unweighted mean squared error, a simplified objective akin to Eq. 14. We also see that learning reverse process variances (by incorporating a parameterized diagonal ${\boldsymbol{\Sigma}}_{\theta}(\mathbf{x}_{t})$ into the variational bound) leads to unstable training and poorer sample quality compared to fixed variances. Predicting ${\boldsymbol{\epsilon}}$, as we proposed, performs approximately as well as predicting $\tilde{\boldsymbol{\mu}}$ when trained on the variational bound with fixed variances, but much better when trained with our simplified objective.

在表 2 中，我们显示了逆向过程参数化和训练目标的样本质量影响（第 3.2 节）。 我们发现，预测$\tilde{\boldsymbol{\mu}}$的基线选项只有在真正的变分界限而不是未加权均方误差（类似于等式 1 的简化目标）上进行训练时才有效。 14. 我们还发现，与固定方差相比，学习逆向过程方差（通过将参数化对角线${\boldsymbol{\Sigma}}_{\theta}(\mathbf{x}_{t})$合并到变分界限中）会导致训练不稳定且样本质量较差。 正如我们所建议的，当在具有固定方差的变分界限上进行训练时，预测${\boldsymbol{\epsilon}}$的表现与预测$\tilde{\boldsymbol{\mu}}$的表现大致相同，但在使用我们的简化目标进行训练时要好得多。

4.3 Progressive coding

4.3 渐进编码

Table 1 also shows the codelengths of our CIFAR10 models. The gap between train and test is at most 0.03 bits per dimension, which is comparable to the gaps reported with other likelihood-based models and indicates that our diffusion model is not overfitting (see Appendix D for nearest neighbor visualizations). Still, while our lossless codelengths are better than the large estimates reported for energy based models and score matching using annealed importance sampling [11], they are not competitive with other types of likelihood-based generative models [7].

表 1 还显示了我们的 CIFAR10 模型的代码长度。 训练和测试之间的差距最多为每个维度 0.03 位，这与其他基于可能性的模型报告的差距相当，表明我们的扩散模型没有过度拟合（有关最近邻可视化，请参阅附录 D）。 尽管如此，虽然我们的无损代码长度优于基于能量的模型和使用退火重要性采样的分数匹配报告的大估计[11]，但它们与其他类型的基于可能性的生成模型[7]不具有竞争力。

Since our samples are nonetheless of high quality, we conclude that diffusion models have an inductive bias that makes them excellent lossy compressors. Treating the variational bound terms $L_{1}+\cdots+L_{T}$ as rate and $L_{0}$ as distortion, our CIFAR10 model with the highest quality samples has a rate of 1.78 bits/dim and a distortion of 1.97 bits/dim, which amounts to a root mean squared error of 0.95 on a scale from 0 to 255. More than half of the lossless codelength describes imperceptible distortions.

由于我们的样本仍然是高质量的，因此我们得出结论，扩散模型具有归纳偏差，这使得它们成为出色的有损压缩器。 将变分约束项$L_{1}+\cdots+L_{T}$视为速率，将$L_{0}$视为失真，我们具有最高质量样本的 CIFAR10 模型的速率为 1.78 位/暗，失真为 1.97 位/暗，在 0 到 255 的范围内，均方根误差为 0.95。超过一半的无损码长描述了难以察觉的失真。

Progressive lossy compression

We can probe further into the rate-distortion behavior of our model by introducing a progressive lossy code that mirrors the form of Eq. 5: see Algorithms 3 and 4, which assume access to a procedure, such as minimal random coding [19, 20], that can transmit a sample $\mathbf{x}\sim q(\mathbf{x})$ using approximately $D_{\mathrm{KL}}\!\left(q(\mathbf{x})~\|~p(\mathbf{x})\right)$ bits on average for any distributions $p$ and $q$, for which only $p$ is available to the receiver beforehand.

渐进式有损压缩

我们可以通过引入反映方程形式的渐进有损代码来进一步探讨模型的速率失真行为。 5：参见算法 3 和 4，它们假设访问一个过程，例如最小随机编码 [19, 20]，对于任何分布$p$和$q$，平均可以使用大约$D_{\mathrm{KL}}\!\left(q(\mathbf{x})~\|~p(\mathbf{x})\right)$位来传输样本$\mathbf{x}\sim q(\mathbf{x})$，其中只有$p$事先可供接收器使用。

1: Send $\mathbf{x}_{T}\sim q(\mathbf{x}_{T}|\mathbf{x}_{0})$ using $p(\mathbf{x}_{T})$
2: for $t=T-1,\dotsc,2,1$ do
3:  Send $\mathbf{x}_{t}\sim q(\mathbf{x}_{t}|\mathbf{x}_{t+1},\mathbf{x}_{0})$ using $p_{\theta}(\mathbf{x}_{t}|\mathbf{x}_{t+1})$
4: end for
5: Send $\mathbf{x}_{0}$ using $p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1})$

1: Receive $\mathbf{x}_{T}$ using $p(\mathbf{x}_{T})$
2: for $t=T-1,\dotsc,1,0$ do
3:  Receive $\mathbf{x}_{t}$ using $p_{\theta}(\mathbf{x}_{t}|\mathbf{x}_{t+1})$
4: end for
5: return $\mathbf{x}_{0}$

Algorithm 3 Sending $\mathbf{x}_{0}$

1：使用$p(\mathbf{x}_{T})$发送 $\mathbf{x}_{T}\sim q(\mathbf{x}_{T}|\mathbf{x}_{0})$
2：对于$t=T-1,\dotsc,2,1$做
3:  使用$p_{\theta}(\mathbf{x}_{t}|\mathbf{x}_{t+1})$发送 $\mathbf{x}_{t}\sim q(\mathbf{x}_{t}|\mathbf{x}_{t+1},\mathbf{x}_{0})$
4： 结束 用于
5：使用$p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{1})$发送$\mathbf{x}_{0}$

1：使用$p(\mathbf{x}_{T})$接收$\mathbf{x}_{T}$
2：对于$t=T-1,\dotsc,1,0$做
3:  使用$p_{\theta}(\mathbf{x}_{t}|\mathbf{x}_{t+1})$接收 $\mathbf{x}_{t}$
4： 结束 用于
5：返回 $\mathbf{x}_{0}$

算法3 发送$\mathbf{x}_{0}$

When applied to $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$, Algorithms 3 and 4 transmit $\mathbf{x}_{T},\dotsc,\mathbf{x}_{0}$ in sequence using a total expected codelength equal to Eq. 5. The receiver, at any time $t$, has the partial information $\mathbf{x}_{t}$ fully available and can progressively estimate:

当应用于$\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$时，算法 3 和 4 使用等于等式 1 的总预期码长按顺序传输 $\mathbf{x}_{T},\dotsc,\mathbf{x}_{0}$。 5. 接收器在任何时间$t$都具有完全可用的部分信息$\mathbf{x}_{t}$，并且可以逐步估计：

$$\displaystyle\mathbf{x}_{0}\approx\hat{\mathbf{x}}_{0}=\left(\mathbf{x}_{t}-\sqrt{1-\bar{\alpha}_{t}}{\boldsymbol{\epsilon}}_{\theta}(\mathbf{x}_{t})\right)/\sqrt{\bar{\alpha}_{t}} \tag{15}$$

due to Eq. 4. (A stochastic reconstruction $\mathbf{x}_{0}\sim p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{t})$ is also valid, but we do not consider it here because it makes distortion more difficult to evaluate.) Figure 5 shows the resulting rate-distortion plot on the CIFAR10 test set. At each time $t$, the distortion is calculated as the root mean squared error $\sqrt{\|\mathbf{x}_{0}-\hat{\mathbf{x}}_{0}\|^{2}/D}$, and the rate is calculated as the cumulative number of bits received so far at time $t$. The distortion decreases steeply in the low-rate region of the rate-distortion plot, indicating that the majority of the bits are indeed allocated to imperceptible distortions.

由于方程式 4.（随机重建$\mathbf{x}_{0}\sim p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{t})$也是有效的，但我们在这里不考虑它，因为它使失真更难以评估。）图 5 显示了 CIFAR10 测试集上生成的率失真图。 在每个时间$t$处，失真被计算为均方根误差$\sqrt{\|\mathbf{x}_{0}-\hat{\mathbf{x}}_{0}\|^{2}/D}$，并且速率被计算为迄今为止在时间$t$接收到的累积比特数。 失真在速率失真图的低速率区域中急剧下降，表明大多数比特确实分配给了难以察觉的失真。

Figure 5: Unconditional CIFAR10 test set rate-distortion vs. time. Distortion is measured in root mean squared error on a $[0,255]$ scale. See Table 4 for details.

图 5：无条件 CIFAR10 测试集率失真与时间的关系。 失真度以$[0,255]$标度上的均方根误差来衡量。 有关详细信息，请参阅表 4。

Progressive generation

We also run a progressive unconditional generation process given by progressive decompression from random bits. In other words, we predict the result of the reverse process, $\hat{\mathbf{x}}_{0}$, while sampling from the reverse process using Algorithm 2. Figures 6 and 10 show the resulting sample quality of $\hat{\mathbf{x}}_{0}$ over the course of the reverse process. Large scale image features appear first and details appear last. Figure 7 shows stochastic predictions $\mathbf{x}_{0}\sim p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{t})$ with $\mathbf{x}_{t}$ frozen for various $t$. When $t$ is small, all but fine details are preserved, and when $t$ is large, only large scale features are preserved. Perhaps these are hints of conceptual compression [18].

渐进一代

我们还运行一个渐进的无条件生成过程，该过程是通过随机位的渐进解压缩给出的。 换句话说，我们预测逆向过程的结果$\hat{\mathbf{x}}_{0}$，同时使用算法 2 从逆向过程中采样。图 6 和 10 显示了反向过程过程中$\hat{\mathbf{x}}_{0}$的最终样本质量。 首先出现大尺度图像特征，最后出现细节。 图 7 显示了针对各种$t$冻结的随机预测$\mathbf{x}_{0}\sim p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{t})$和$\mathbf{x}_{t}$。 当$t$较小时，除了精细细节之外的所有细节都会被保留，而当$t$较大时，仅会保留大尺度特征。 也许这些都是概念压缩的暗示[18]。

Figure 6: Unconditional CIFAR10 progressive generation ($\hat{\mathbf{x}}_{0}$ over time, from left to right). Extended samples and sample quality metrics over time in the appendix (Figs. 14 and 10).

图 6：无条件 CIFAR10 渐进式生成（$\hat{\mathbf{x}}_{0}$ 随着时间的推移，从左到右）。 附录中随时间变化的扩展样本和样本质量指标（图 14 和 10）。

Figure 7: When conditioned on the same latent, CelebA-HQ $256\times 256$ samples share high-level attributes. Bottom-right quadrants are $\mathbf{x}_{t}$, and other quadrants are samples from $p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{t})$.

图 7：当以相同潜在条件为条件时，CelebA-HQ$256\times 256$样本共享高级属性。 右下象限是$\mathbf{x}_{t}$，其他象限是来自$p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{t})$的样本。

Connection to autoregressive decoding

Note that the variational bound 5 can be rewritten as:

连接到自回归解码

请注意，变分界 5 可以重写为：

$$\displaystyle L \\ \displaystyle=D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{T})~\|~p(\mathbf{x}_{T})\right)+\mathbb{E}_{q}\Bigg[\sum_{t\geq 1}D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{t-1}|\mathbf{x}_{t})~\|~p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})\right)\Bigg]+H(\mathbf{x}_{0}) \tag{16}$$

(See Appendix A for a derivation.) Now consider setting the diffusion process length $T$ to the dimensionality of the data, defining the forward process so that $q(\mathbf{x}_{t}|\mathbf{x}_{0})$ places all probability mass on $\mathbf{x}_{0}$ with the first $t$ coordinates masked out (i.e. $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$ masks out the $t^{\text{th}}$ coordinate), setting $p(\mathbf{x}_{T})$ to place all mass on a blank image, and, for the sake of argument, taking $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ to be a fully expressive conditional distribution. With these choices, $D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{T})~\|~p(\mathbf{x}_{T})\right)=0$, and minimizing $D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{t-1}|\mathbf{x}_{t})~\|~p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})\right)$ trains $p_{\theta}$ to copy coordinates $t+1,\dotsc,T$ unchanged and to predict the $t^{\text{th}}$ coordinate given $t+1,\dotsc,T$. Thus, training $p_{\theta}$ with this particular diffusion is training an autoregressive model.

（推导请参见附录 A。）现在考虑将扩散过程长度$T$设置为数据的维数，定义前向过程，以便$q(\mathbf{x}_{t}|\mathbf{x}_{0})$将所有概率质量放置在$\mathbf{x}_{0}$上，并屏蔽第一个$t$坐标（即$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$屏蔽$t^{\text{th}}$坐标），设置$p(\mathbf{x}_{T})$将所有质量放置在空白图像上，并且，为了 参数，将$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$视为完全表达的条件分布。 通过这些选择，$D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{T})~\|~p(\mathbf{x}_{T})\right)=0$和最小化$D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{t-1}|\mathbf{x}_{t})~\|~p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})\right)$可以训练$p_{\theta}$复制坐标$t+1,\dotsc,T$不变，并在给定$t+1,\dotsc,T$的情况下预测$t^{\text{th}}$坐标。 因此，使用这种特定扩散训练$p_{\theta}$就是训练自回归模型。

We can therefore interpret the Gaussian diffusion model 2 as a kind of autoregressive model with a generalized bit ordering that cannot be expressed by reordering data coordinates. Prior work has shown that such reorderings introduce inductive biases that have an impact on sample quality [38], so we speculate that the Gaussian diffusion serves a similar purpose, perhaps to greater effect since Gaussian noise might be more natural to add to images compared to masking noise. Moreover, the Gaussian diffusion length is not restricted to equal the data dimension; for instance, we use $T=1000$, which is less than the dimension of the $32\times 32\times 3$ or $256\times 256\times 3$ images in our experiments. Gaussian diffusions can be made shorter for fast sampling or longer for model expressiveness.

因此，我们可以将高斯扩散模型 2 解释为一种自回归模型，其具有无法通过重新排序数据坐标来表达的广义位排序。 之前的工作表明，这种重新排序会引入影响样本质量的归纳偏差[38]，因此我们推测高斯扩散具有类似的目的，也许效果更大，因为与掩蔽噪声相比，高斯噪声可能更自然地添加到图像中。 而且，高斯扩散长度不限于等于数据维度； 例如，我们使用$T=1000$，它小于实验中$32\times 32\times 3$或$256\times 256\times 3$图像的尺寸。 高斯扩散可以缩短以实现快速采样，也可以延长以提高模型表达能力。

4.4 Interpolation

4.4 插值

We can interpolate source images $\mathbf{x}_{0},\mathbf{x}^{\prime}_{0}\sim q(\mathbf{x}_{0})$ in latent space using $q$ as a stochastic encoder, $\mathbf{x}_{t},\mathbf{x}^{\prime}_{t}\sim q(\mathbf{x}_{t}|\mathbf{x}_{0})$, then decoding the linearly interpolated latent $\bar{\mathbf{x}}_{t}=(1-\lambda)\mathbf{x}_{0}+\lambda\mathbf{x}^{\prime}_{0}$ into image space by the reverse process, $\bar{\mathbf{x}}_{0}\sim p(\mathbf{x}_{0}|\bar{\mathbf{x}}_{t})$. In effect, we use the reverse process to remove artifacts from linearly interpolating corrupted versions of the source images, as depicted in Fig. 8 (left). We fixed the noise for different values of $\lambda$ so $\mathbf{x}_{t}$ and $\mathbf{x}^{\prime}_{t}$ remain the same. Fig. 8 (right) shows interpolations and reconstructions of original CelebA-HQ $256\times 256$ images ($t=500$). The reverse process produces high-quality reconstructions, and plausible interpolations that smoothly vary attributes such as pose, skin tone, hairstyle, expression and background, but not eyewear. Larger $t$ results in coarser and more varied interpolations, with novel samples at $t=1000$ (Appendix Fig. 9).

我们可以使用$q$作为随机编码器$\mathbf{x}_{t},\mathbf{x}^{\prime}_{t}\sim q(\mathbf{x}_{t}|\mathbf{x}_{0})$在潜在空间中插值源图像$\mathbf{x}_{0},\mathbf{x}^{\prime}_{0}\sim q(\mathbf{x}_{0})$，然后通过相反过程$\bar{\mathbf{x}}_{0}\sim p(\mathbf{x}_{0}|\bar{\mathbf{x}}_{t})$将线性插值的潜在$\bar{\mathbf{x}}_{t}=(1-\lambda)\mathbf{x}_{0}+\lambda\mathbf{x}^{\prime}_{0}$解码到图像空间中。 实际上，我们使用相反的过程来消除源图像的线性插值损坏版本中的伪影，如图 8（左）所示。 我们修复了$\lambda$不同值的噪声，因此$\mathbf{x}_{t}$和$\mathbf{x}^{\prime}_{t}$保持不变。 图 8（右）显示了原始 CelebA-HQ$256\times 256$图像 ($t=500$) 的插值和重建。 相反的过程会产生高质量的重建和合理的插值，这些插值可以平滑地改变姿势、肤色、发型、表情和背景等属性，但不能改变眼镜。 较大的$t$会导致更粗糙和更多样的插值，并且$t=1000$具有新颖的样本（附录图 9）。