Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

面向高分辨率图像合成的缩放整流流 Transformer

Patrick Esser    Sumith Kulal    Andreas Blattmann    Rahim Entezari    Jonas Müller    Harry Saini    Yam Levi    Dominik Lorenz    Axel Sauer    Frederic Boesel    Dustin Podell    Tim Dockhorn    Zion English    Kyle Lacey    Alex Goodwin    Yannik Marek    Robin Rombach Correspondence to: <first.last>@stability.ai

Abstract

摘要

Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension, typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations. Our largest models outperform state-of-the-art models, and we will make our experimental data, code, and model weights publicly available.

扩散模型通过反转数据走向噪声的前向路径，从噪声生成数据，已成为图像与视频等高维感知数据上强有力的生成建模技术。整流流（rectified flow）是近期提出的生成模型形式，把数据与噪声连成直线。尽管理论性质更好、概念更简洁，它尚未成为标准实践。本文改进整流流训练中的噪声采样，使其更偏向感知相关尺度；大规模研究表明，相对既有扩散公式，该做法在高分辨率文生图上表现更优。此外，我们提出一种面向文生图的新型 Transformer 架构：两模态使用独立权重，并在图像与文本 token 之间做双向信息流，从而提升文本理解、排版与人类偏好评分。我们证明该架构遵循可预测的缩放趋势，验证损失降低与按多种指标及人类评估衡量的文生图改进相关。我们最大的模型超过现有最优；实验数据、代码与模型权重将公开。

Keywords:

Stability AI

关键词：

Stability AI

Figure 1: High-resolution samples from our 8B rectified flow model, showcasing its capabilities in typography, precise prompt following and spatial reasoning, attention to fine details, and high image quality across a wide variety of styles.

图 1：我们 8B 整流流模型的高分辨率样本，展示其在排版、精确遵循提示与空间推理、关注细节以及跨多样风格的高图像质量等方面的能力。

1 Introduction

1 引言

Diffusion models create data from noise (Song et al., 2020). They are trained to invert forward paths of data towards random noise and, thus, in conjunction with approximation and generalization properties of neural networks, can be used to generate new data points that are not present in the training data but follow the distribution of the training data (Sohl-Dickstein et al., 2015; Song & Ermon, 2020). This generative modeling technique has proven to be very effective for modeling high-dimensional, perceptual data such as images (Ho et al., 2020). In recent years, diffusion models have become the de-facto approach for generating high-resolution images and videos from natural language inputs with impressive generalization capabilities (Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al., 2022; Podell et al., 2023; Dai et al., 2023; Esser et al., 2023; Blattmann et al., 2023b; Betker et al., 2023; Blattmann et al., 2023a; Singer et al., 2022). Due to their iterative nature and the associated computational costs, as well as the long sampling times during inference, research on formulations for more efficient training and/or faster sampling of these models has increased (Karras et al., 2023; Liu et al., 2022).

扩散模型从噪声生成数据 (Song et al., 2020)。它们被训练以反转数据走向随机噪声的前向路径，因而结合神经网络的逼近与泛化性质，可用于生成训练数据中不存在、但遵循训练数据分布的新数据点 (Sohl-Dickstein et al., 2015; Song & Ermon, 2020)。这一生成建模技术对图像等高维感知数据尤为有效 (Ho et al., 2020)。近年来，扩散模型已成为从自然语言生成高分辨率图像与视频的默认路径，并展现出可观的泛化能力 (Saharia et al., 2022b; Ramesh et al., 2022; Rombach et al., 2022; Podell et al., 2023; Dai et al., 2023; Esser et al., 2023; Blattmann et al., 2023b; Betker et al., 2023; Blattmann et al., 2023a; Singer et al., 2022)。由于其迭代特性与相关计算开销，以及推理时采样时间较长，围绕更高效训练与/或更快采样的公式研究也在增多 (Karras et al., 2023; Liu et al., 2022)。

While specifying a forward path from data to noise leads to efficient training, it also raises the question of which path to choose. This choice can have important implications for sampling. For example, a forward process that fails to remove all noise from the data can lead to a discrepancy in training and test distribution and result in artifacts such as gray image samples (Lin et al., 2024). Importantly, the choice of the forward process also influences the learned backward process and, thus, the sampling efficiency. While curved paths require many integration steps to simulate the process, a straight path could be simulated with a single step and is less prone to error accumulation. Since each step corresponds to an evaluation of the neural network, this has a direct impact on the sampling speed.

指定从数据到噪声的前向路径虽使训练高效，却引出「选哪条路径」的问题。这一选择对采样有重要影响。例如，若前向过程未能从数据中去掉全部噪声，就会造成训练与测试分布不一致，并出现灰色图像样本等伪影 (Lin et al., 2024)。更关键的是，前向过程的选择也会影响学到的反向过程，进而影响采样效率。弯曲路径需要许多积分步才能模拟，而直线路径原则上只需一步，且更不易累积误差。每一步对应一次神经网络求值，因而直接关系到采样速度。

A particular choice for the forward path is a so-called Rectified Flow (Liu et al., 2022; Albergo & Vanden-Eijnden, 2022; Lipman et al., 2023), which connects data and noise on a straight line. Although this model class has better theoretical properties, it has not yet become decisively established in practice. So far, some advantages have been empirically demonstrated in small and medium-sized experiments (Ma et al., 2024), but these are mostly limited to class-conditional models. In this work, we change this by introducing a re-weighting of the noise scales in rectified flow models, similar to noise-predictive diffusion models (Ho et al., 2020). Through a large-scale study, we compare our new formulation to existing diffusion formulations and demonstrate its benefits.

前向路径的一种特定选择是所谓的整流流（Rectified Flow） (Liu et al., 2022; Albergo & Vanden-Eijnden, 2022; Lipman et al., 2023)，它把数据与噪声连成直线。尽管该类模型理论性质更好，实践中却尚未被广泛确立。迄今在中小规模实验中已实证出若干优势 (Ma et al., 2024)，但多限于类别条件模型。本文改变这一点：我们对照噪声预测扩散模型 (Ho et al., 2020)，为整流流引入噪声尺度的重加权。通过大规模研究，我们把新公式与既有扩散公式对比，并展示其收益。

We show that the widely used approach for text-to-image synthesis, where a fixed text representation is fed directly into the model (e.g., via cross-attention (Vaswani et al., 2017; Rombach et al., 2022)), is not ideal, and present a new architecture that incorporates learnable streams for both image and text tokens, which enables a two-way flow of information between them. We combine this with our improved rectified flow formulation and investigate its scalability. We demonstrate a predictable scaling trend in the validation loss and show that a lower validation loss correlates strongly with improved automatic and human evaluations.

我们表明：当前文生图的主流做法——把固定文本表示直接送入模型（例如经交叉注意力 (Vaswani et al., 2017; Rombach et al., 2022)）——并不理想；并提出一种新架构，为图像与文本 token 各设可学习流，使二者双向交换信息。我们把它与改进后的整流流公式结合并考察可扩展性。我们在验证损失上观察到可预测的缩放趋势，并表明更低的验证损失与自动指标及人类评估的改进强相关。

Our largest models outperform state-of-the art open models such as SDXL (Podell et al., 2023), SDXL-Turbo (Sauer et al., 2023), Pixart-$\alpha$ (Chen et al., 2023), and closed-source models such as DALL-E 3 (Betker et al., 2023) both in quantitative evaluation (Ghosh et al., 2023) of prompt understanding and human preference ratings.

我们最大的模型在提示理解的定量评估 (Ghosh et al., 2023) 与人类偏好评分上，均超过开源最优如 SDXL (Podell et al., 2023)、SDXL-Turbo (Sauer et al., 2023)、Pixart-$\alpha$ (Chen et al., 2023)，以及闭源如 DALL-E 3 (Betker et al., 2023)。

The core contributions of our work are: (i) We conduct a large-scale, systematic study on different diffusion model and rectified flow formulations to identify the best setting. For this purpose, we introduce new noise samplers for rectified flow models that improve performance over previously known samplers. (ii) We devise a novel, scalable architecture for text-to-image synthesis that allows bi-directional mixing between text and image token streams within the network. We show its benefits compared to established backbones such as UViT (Hoogeboom et al., 2023) and DiT (Peebles & Xie, 2023). Finally, we (iii) perform a scaling study of our model and demonstrate that it follows predictable scaling trends. We show that a lower validation loss correlates strongly with improved text-to-image performance assessed via metrics such as T2I-CompBench (Huang et al., 2023), GenEval (Ghosh et al., 2023) and human ratings. We make results, code, and model weights publicly available.

本文核心贡献如下：(i) 我们对多种扩散与整流流公式做大规模系统研究，以确定最佳设定；为此引入新的整流流噪声采样器，相对既有采样器提升表现。(ii) 我们设计一种可扩展的文生图架构，在网络内部允许文本与图像 token 流双向混合；相对 UViT (Hoogeboom et al., 2023) 与 DiT (Peebles & Xie, 2023) 等既有骨干，我们展示其优势。最后，(iii) 我们对模型做缩放研究，表明其遵循可预测的缩放趋势；验证损失降低与经 T2I-CompBench (Huang et al., 2023)、GenEval (Ghosh et al., 2023) 及人类评分衡量的文生图表现强相关。结果、代码与模型权重将公开。

2 Simulation-Free Training of Flows

2 无需仿真的流训练

We consider generative models that define a mapping between samples $x_{1}$ from a noise distribution $p_{1}$ to samples $x_{0}$ from a data distribution $p_{0}$ in terms of an ordinary differential equation (ODE),

我们考虑这样一类生成模型：用常微分方程（ODE）定义从噪声分布 $p_{1}$ 的样本 $x_{1}$ 到数据分布 $p_{0}$ 的样本 $x_{0}$ 的映射，

$$dy_{t}=v_{\Theta}(y_{t},t)\,dt\;, \tag{1}$$

where the velocity $v$ is parameterized by the weights $\Theta$ of a neural network. Prior work by Chen et al. (2018) suggested to directly solve Equation 1 via differentiable ODE solvers. However, this process is computationally expensive, especially for large network architectures that parameterize $v_{\Theta}(y_{t},t)$. A more efficient alternative is to directly regress a vector field $u_{t}$ that generates a probability path between $p_{0}$ and $p_{1}$. To construct such a $u_{t}$, we define a forward process, corresponding to a probability path $p_{t}$ between $p_{0}$ and $p_{1}=\mathcal{N}(0,1)$, as

其中速度 $v$ 由神经网络权重 $\Theta$ 参数化。Chen et al. (2018) 曾建议用可微 ODE 求解器直接求解式 1，但对参数化 $v_{\Theta}(y_{t},t)$ 的大型网络而言计算昂贵。更高效的做法是直接回归一个在 $p_{0}$ 与 $p_{1}$ 之间生成概率路径的向量场 $u_{t}$。为构造这样的 $u_{t}$，我们定义前向过程，对应 $p_{0}$ 与 $p_{1}=\mathcal{N}(0,1)$ 之间的概率路径 $p_{t}$：

$$z_{t}=a_{t}x_{0}+b_{t}\epsilon\quad\text{where}\;\epsilon\sim\mathcal{N}(0,I)\;. \tag{2}$$

For $a_{0}=1,b_{0}=0,a_{1}=0$ and $b_{1}=1$, the marginals,

当 $a_{0}=1,b_{0}=0,a_{1}=0$ 且 $b_{1}=1$ 时，边缘分布

$$\displaystyle p_{t}(z_{t}) \\ \displaystyle=\mathbb{E}_{\epsilon\sim\mathcal{N}(0,I)}p_{t}(z_{t}|\epsilon)\;, \tag{3}$$

are consistent with the data and noise distribution.

与数据及噪声分布一致。

To express the relationship between $z_{t},x_{0}$ and $\epsilon$, we introduce $\psi_{t}$ and $u_{t}$ as

为表达 $z_{t}$、$x_{0}$ 与 $\epsilon$ 的关系，我们引入 $\psi_{t}$ 与 $u_{t}$ 如下

$$\displaystyle\psi_{t}(\cdot|\epsilon) \\ \displaystyle:x_{0}\mapsto a_{t}x_{0}+b_{t}\epsilon \\ \displaystyle u_{t}(z|\epsilon) \\ \displaystyle\coloneqq\psi^{\prime}_{t}(\psi_{t}^{-1}(z|\epsilon)|\epsilon) \tag{4}$$

Since $z_{t}$ can be written as solution to the ODE $z_{t}^{\prime}=u_{t}(z_{t}|\epsilon)$, with initial value $z_{0}=x_{0}$, $u_{t}(\cdot|\epsilon)$ generates $p_{t}(\cdot|\epsilon)$. Remarkably, one can construct a marginal vector field $u_{t}$ which generates the marginal probability paths $p_{t}$ (Lipman et al., 2023) (see B.1), using the conditional vector fields $u_{t}(\cdot|\epsilon)$:

由于 $z_{t}$ 可写成 ODE $z_{t}^{\prime}=u_{t}(z_{t}|\epsilon)$、初值 $z_{0}=x_{0}$ 的解，$u_{t}(\cdot|\epsilon)$ 生成 $p_{t}(\cdot|\epsilon)$。值得注意的是，可用条件向量场 $u_{t}(\cdot|\epsilon)$ 构造生成边缘概率路径 $p_{t}$ 的边缘向量场 $u_{t}$（Lipman et al., 2023）（见 B.1）：

$$\displaystyle u_{t}(z)=\mathbb{E}_{\epsilon\sim\mathcal{N}(0,I)}u_{t}(z|\epsilon)\frac{p_{t}(z|\epsilon)}{p_{t}(z)} \tag{6}$$

While regressing $u_{t}$ with the Flow Matching objective

若用流匹配目标直接回归 $u_{t}$

$$\displaystyle\mathcal{L}_{FM}=\mathbb{E}_{t,p_{t}(z)}||v_{\Theta}(z,t)-u_{t}(z)||_{2}^{2}. \tag{7}$$

directly is intractable due to the marginalization in Equation 6, Conditional Flow Matching (see B.1),

因式 6 中的边缘化而不可行；条件流匹配（见 B.1）

$$\displaystyle\mathcal{L}_{CFM}=\mathbb{E}_{t,p_{t}(z|\epsilon),p(\epsilon)}||v_{\Theta}(z,t)-u_{t}(z|\epsilon)||_{2}^{2}\;, \tag{8}$$

with the conditional vector fields $u_{t}(z|\epsilon)$ provides an equivalent yet tractable objective.

借助条件向量场 $u_{t}(z|\epsilon)$，给出等价且可计算的目标。

To convert the loss into an explicit form we insert $\psi_{t}^{\prime}(x_{0}|\epsilon)=a_{t}^{\prime}x_{0}+b_{t}^{\prime}\epsilon$ and $\psi_{t}^{-1}(z|\epsilon)=\frac{z-b_{t}\epsilon}{a_{t}}$ into (5)

为把损失写成显式形式，将 $\psi_{t}^{\prime}(x_{0}|\epsilon)=a_{t}^{\prime}x_{0}+b_{t}^{\prime}\epsilon$ 与 $\psi_{t}^{-1}(z|\epsilon)=\frac{z-b_{t}\epsilon}{a_{t}}$ 代入 (5)

$$z_{t}^{\prime}=u_{t}(z_{t}|\epsilon)=\frac{a_{t}^{\prime}}{a_{t}}z_{t}-\epsilon b_{t}(\frac{a_{t}^{\prime}}{a_{t}}-\frac{b_{t}^{\prime}}{b_{t}})\;. \tag{9}$$

Now, consider the signal-to-noise ratio $\lambda_{t}:=\log\frac{a_{t}^{2}}{b_{t}^{2}}$. With $\lambda_{t}^{\prime}=2(\frac{a_{t}^{\prime}}{a_{t}}-\frac{b_{t}^{\prime}}{b_{t}})$, we can rewrite Equation 9 as

现考虑信噪比 $\lambda_{t}:=\log\frac{a_{t}^{2}}{b_{t}^{2}}$。由 $\lambda_{t}^{\prime}=2(\frac{a_{t}^{\prime}}{a_{t}}-\frac{b_{t}^{\prime}}{b_{t}})$，可把式 9 改写为

$$u_{t}(z_{t}|\epsilon)=\frac{a_{t}^{\prime}}{a_{t}}z_{t}-\frac{b_{t}}{2}\lambda_{t}^{\prime}\epsilon \tag{10}$$

Next, we use Equation 10 to reparameterize Equation 8 as a noise-prediction objective:

接着用式 10 把式 8 重参数化为噪声预测目标：

$$\displaystyle\mathcal{L}_{CFM} \\ \displaystyle=\mathbb{E}_{t,p_{t}(z|\epsilon),p(\epsilon)}||v_{\Theta}(z,t)-\frac{a_{t}^{\prime}}{a_{t}}z+\frac{b_{t}}{2}\lambda_{t}^{\prime}\epsilon||_{2}^{2} \\ \displaystyle=\mathbb{E}_{t,p_{t}(z|\epsilon),p(\epsilon)}\left(-\frac{b_{t}}{2}\lambda_{t}^{\prime}\right)^{2}||\epsilon_{\Theta}(z,t)-\epsilon||_{2}^{2} \tag{11}$$

where we defined $\epsilon_{\Theta}\coloneqq\frac{-2}{\lambda_{t}^{\prime}b_{t}}(v_{\Theta}-\frac{a_{t}^{\prime}}{a_{t}}z)$.

其中定义 $\epsilon_{\Theta}\coloneqq\frac{-2}{\lambda_{t}^{\prime}b_{t}}(v_{\Theta}-\frac{a_{t}^{\prime}}{a_{t}}z)$。

Note that the optimum of the above objective does not change when introducing a time-dependent weighting. Thus, one can derive various weighted loss functions that provide a signal towards the desired solution but might affect the optimization trajectory. For a unified analysis of different approaches, including classic diffusion formulations, we can write the objective in the following form (following Kingma & Gao (2023)):

注意：引入依赖时间的权重并不改变上述目标的最优解。因此可导出多种加权损失，它们都指向同一期望解，但可能影响优化轨迹。为统一分析包括经典扩散在内的不同做法，我们把目标写成如下形式（依 Kingma & Gao (2023)）：

$$\mathcal{L}_{w}(x_{0})=-\frac{1}{2}\mathbb{E}_{t\sim\mathcal{U}(t),\epsilon\sim\mathcal{N}(0,I)}\left[w_{t}\lambda_{t}^{\prime}\|\epsilon_{\Theta}(z_{t},t)-\epsilon\|^{2}\right]\;,$$

where $w_{t}=-\frac{1}{2}\lambda_{t}^{\prime}b_{t}^{2}$ corresponds to $\mathcal{L}_{CFM}$.

其中 $w_{t}=-\frac{1}{2}\lambda_{t}^{\prime}b_{t}^{2}$ 对应 $\mathcal{L}_{CFM}$。

3 Flow Trajectories

3 流轨迹

In this work, we consider different variants of the above formalism that we briefly describe in the following.

本文考虑上述形式体系的若干变体，简述如下。

Rectified Flow

Rectified Flows (RFs) (Liu et al., 2022; Albergo & Vanden-Eijnden, 2022; Lipman et al., 2023) define the forward process as straight paths between the data distribution and a standard normal distribution, i.e.

整流流

整流流（RF）(Liu et al., 2022; Albergo & Vanden-Eijnden, 2022; Lipman et al., 2023) 把前向过程定义为数据分布与标准正态之间的直线路径，即

$$z_{t}=(1-t)x_{0}+t\epsilon\;, \tag{13}$$

and uses $\mathcal{L}_{CFM}$ which then corresponds to $w_{t}^{\text{RF}}=\frac{t}{1-t}$. The network output directly parameterizes the velocity $v_{\Theta}$.

并使用 $\mathcal{L}_{CFM}$，此时对应 $w_{t}^{\text{RF}}=\frac{t}{1-t}$。网络输出直接参数化速度 $v_{\Theta}$。

EDM

EDM (Karras et al., 2022) uses a forward process of the form

EDM

EDM (Karras et al., 2022) 使用如下形式的前向过程

$$z_{t}=x_{0}+b_{t}\epsilon \tag{14}$$

where (Kingma & Gao, 2023) $b_{t}=\exp{F_{\mathcal{N}}^{-1}(t|P_{m},P_{s}^{2})}$ with $F_{\mathcal{N}}^{-1}$ being the quantile function of the normal distribution with mean $P_{m}$ and variance $P_{s}^{2}$. Note that this choice results in

其中 (Kingma & Gao, 2023) $b_{t}=\exp{F_{\mathcal{N}}^{-1}(t|P_{m},P_{s}^{2})}$，$F_{\mathcal{N}}^{-1}$ 为均值 $P_{m}$、方差 $P_{s}^{2}$ 的正态分位数函数。该选择导致

$$\lambda_{t}\sim\mathcal{N}(-2P_{m},(2P_{s})^{2})\quad\text{for}\;t\sim\mathcal{U}(0,1) \tag{15}$$

The network is parameterized through an F-prediction (Kingma & Gao, 2023; Karras et al., 2022) and the loss can be written as $\mathcal{L}_{w_{t}^{\text{EDM}}}$ with

网络经 F-预测参数化 (Kingma & Gao, 2023; Karras et al., 2022)，损失可写成 $\mathcal{L}_{w_{t}^{\text{EDM}}}$，其中

$$w_{t}^{\text{EDM}}=\mathcal{N}(\lambda_{t}|-2P_{m},(2P_{s})^{2})(e^{-\lambda_{t}}+0.5^{2}) \tag{16}$$

Cosine

(Nichol & Dhariwal, 2021) proposed a forward process of the form

余弦

(Nichol & Dhariwal, 2021) 提出如下形式的前向过程

$$z_{t}=\cos\bigl(\frac{\pi}{2}t\bigr)x_{0}+\sin\bigl(\frac{\pi}{2}t\bigr)\epsilon\;. \tag{17}$$

In combination with an $\epsilon$-parameterization and loss, this corresponds to a weighting $w_{t}=\operatorname{sech}(\lambda_{t}/2)$. When combined with a v-prediction loss (Kingma & Gao, 2023), the weighting is given by $w_{t}=e^{-\lambda_{t}/2}$.

与 $\epsilon$-参数化及损失结合时，对应权重 $w_{t}=\operatorname{sech}(\lambda_{t}/2)$。与 v-预测损失结合时 (Kingma & Gao, 2023)，权重为 $w_{t}=e^{-\lambda_{t}/2}$。

(LDM-)Linear

LDM (Rombach et al., 2022) uses a modification of the DDPM schedule (Ho et al., 2020). Both are variance preserving schedules, i.e. $b_{t}=\sqrt{1-a_{t}^{2}}$, and define $a_{t}$ for discrete timesteps $t=0,\dots,T-1$ in terms of diffusion coefficients $\beta_{t}$ as $a_{t}=(\prod_{s=0}^{t}(1-\beta_{s}))^{\frac{1}{2}}$. For given boundary values $\beta_{0}$ and $\beta_{T-1}$, DDPM uses $\beta_{t}=\beta_{0}+\frac{t}{T-1}(\beta_{T-1}-\beta_{0})$ and LDM uses $\beta_{t}=\left(\sqrt{\beta_{0}\vphantom{\beta_{T-1}}}+\frac{t}{T-1}(\sqrt{\beta_{T-1}}-\sqrt{\beta_{0}\vphantom{\beta_{T-1}}})\right)^{2}$.

(LDM-)线性

LDM (Rombach et al., 2022) 使用 DDPM 日程 (Ho et al., 2020) 的变体。二者皆为方差保持日程，即 $b_{t}=\sqrt{1-a_{t}^{2}}$，并对离散时间步 $t=0,\dots,T-1$ 用扩散系数 $\beta_{t}$ 定义 $a_{t}=(\prod_{s=0}^{t}(1-\beta_{s}))^{\frac{1}{2}}$。给定边界 $\beta_{0}$ 与 $\beta_{T-1}$，DDPM 取 $\beta_{t}=\beta_{0}+\frac{t}{T-1}(\beta_{T-1}-\beta_{0})$，LDM 取 $\beta_{t}=\left(\sqrt{\beta_{0}}+\frac{t}{T-1}(\sqrt{\beta_{T-1}}-\sqrt{\beta_{0}})\right)^{2}$。

3.1 Tailored SNR Samplers for RF models

3.1 面向 RF 模型的定制 SNR 采样器

The RF loss trains the velocity $v_{\Theta}$ uniformly on all timesteps in $[0,1]$. Intuitively, however, the resulting velocity prediction target $\epsilon-x_{0}$ is more difficult for $t$ in the middle of $[0,1]$, since for $t=0$, the optimal prediction is the mean of $p_{1}$, and for $t=1$ the optimal prediction is the mean of $p_{0}$. In general, changing the distribution over $t$ from the commonly used uniform distribution $\mathcal{U}(t)$ to a distribution with density $\pi(t)$ is equivalent to a weighted loss $\mathcal{L}_{w_{t}^{\pi}}$ with

RF 损失在 $[0,1]$ 上均匀训练速度 $v_{\Theta}$。直观上，所得速度预测目标 $\epsilon-x_{0}$ 在区间中部更难：在 $t=0$ 时最优预测是 $p_{1}$ 的均值，在 $t=1$ 时是 $p_{0}$ 的均值。一般而言，把 $t$ 上常用的均匀分布 $\mathcal{U}(t)$ 换成密度为 $\pi(t)$ 的分布，等价于加权损失 $\mathcal{L}_{w_{t}^{\pi}}$，其中

$$w_{t}^{\pi}=\frac{t}{1-t}\pi(t) \tag{18}$$

Thus, we aim to give more weight to intermediate timesteps by sampling them more frequently. Next, we describe the timestep densities $\pi(t)$ that we use to train our models.

因此我们希望更频繁地采样中间时间步，以给予更大权重。下面描述用于训练的时间步密度 $\pi(t)$。

Logit-Normal Sampling

One option for a distribution that puts more weight on intermediate steps is the logit-normal distribution (Atchison & Shen, 1980). Its density,

Logit-正态采样

一种给中间步更大权重的分布是 logit-正态分布 (Atchison & Shen, 1980)。其密度

$$\pi_{\text{ln}}(t;m,s)=\frac{1}{s\sqrt{2\pi}}\frac{1}{t(1-t)}\exp\Bigl(-\frac{(\text{logit}(t)-m)^{2}}{2s^{2}}\Bigr), \tag{19}$$

where $\text{logit}(t)=\log\frac{t}{1-t}$, has a location parameter, $m$, and a scale parameter, $s$. The location parameter enables us to bias the training timesteps towards either data $p_{0}$ (negative $m$) or noise $p_{1}$ (positive $m$). As shown in Figure 11, the scale parameters controls how wide the distribution is.

其中 $\text{logit}(t)=\log\frac{t}{1-t}$，具有位置参数 $m$ 与尺度参数 $s$。位置参数可将训练时间步偏向数据 $p_{0}$（负 $m$）或噪声 $p_{1}$（正 $m$）。如图 11 所示，尺度参数控制分布宽度。

In practice, we sample the random variable $u$ from a normal distribution $u\sim\mathcal{N}(u;m,s)$ and map it through the standard logistic function.

实践中，我们从正态 $u\sim\mathcal{N}(u;m,s)$ 采样，再经标准 logistic 函数映射。

Mode Sampling with Heavy Tails

The logit-normal density always vanishes at the endpoints $0$ and $1$. To study whether this has adverse effects on the performance, we also use a timestep sampling distribution with strictly positive density on $[0,1]$. For a scale parameter $s$, we define

带重尾的众数采样

logit-正态密度在端点 $0$ 与 $1$ 处总为零。为考察这是否损害表现，我们也使用在 $[0,1]$ 上严格正密度的时间步采样分布。对尺度参数 $s$，定义

$$f_{\text{mode}}(u;s)=1-u-s\cdot\Bigl(\cos^{2}\bigl(\frac{\pi}{2}u\bigr)-1+u\Bigr). \tag{20}$$

For $-1\leq s\leq\frac{2}{\pi-2}$, this function is monotonic, and we can use it to sample from the implied density $\pi_{\text{mode}}(t;s)=\left|\frac{d}{dt}f_{\text{mode}}^{-1}(t)\right|$. As seen in Figure 11, the scale parameter controls the degree to which either the midpoint (positive $s$) or the endpoints (negative $s$) are favored during sampling. This formulation also includes a uniform weighting $\pi_{\text{mode}}(t;s=0)=\mathcal{U}(t)$ for $s=0$, which has been used widely in previous works on Rectified Flows (Liu et al., 2022; Ma et al., 2024).

对 $-1\leq s\leq\frac{2}{\pi-2}$，该函数单调，可用其从隐含密度 $\pi_{\text{mode}}(t;s)=\left|\frac{d}{dt}f_{\text{mode}}^{-1}(t)\right|$ 采样。如图 11，尺度参数控制采样时更偏向中点（正 $s$）还是端点（负 $s$）。该形式在 $s=0$ 时包含均匀加权 $\pi_{\text{mode}}(t;s=0)=\mathcal{U}(t)$，此前整流流工作中广泛使用 (Liu et al., 2022; Ma et al., 2024)。

CosMap

Finally, we also consider the cosine schedule (Nichol & Dhariwal, 2021) from Section 3 in the RF setting. In particular, we are looking for a mapping $f:u\mapsto f(u)=t,\;u\in[0,1]$, such that the log-snr matches that of the cosine schedule: $2\log\frac{\cos(\frac{\pi}{2}u)}{\sin(\frac{\pi}{2}u)}=2\log\frac{1-f(u)}{f(u)}$. Solving for $f$, we obtain for $u\sim\mathcal{U}(u)$

CosMap

最后，我们也在 RF 设定下考虑第 3 节的余弦日程 (Nichol & Dhariwal, 2021)。具体地，我们寻找映射 $f:u\mapsto f(u)=t,\;u\in[0,1]$，使 log-snr 与余弦日程匹配：$2\log\frac{\cos(\frac{\pi}{2}u)}{\sin(\frac{\pi}{2}u)}=2\log\frac{1-f(u)}{f(u)}$。求解 $f$，对 $u\sim\mathcal{U}(u)$ 得到

$$t=f(u)=1-\frac{1}{\tan(\frac{\pi}{2}u)+1}, \tag{21}$$

from which we obtain the density

由此得到密度

$$\pi_{\text{CosMap}}(t)=\left|\frac{d}{dt}f^{-1}(t)\right|=\frac{2}{\pi-2\pi t+2\pi t^{2}}. \tag{22}$$

4 Text-to-Image Architecture

4 文生图架构

(a) Overview of all components.

(a) 全部组件总览。

For text-conditional sampling of images, our model has to take both modalities, text and images, into account. We use pretrained models to derive suitable representations and then describe the architecture of our diffusion backbone. An overview of this is presented in Figure 2.

为做文本条件的图像采样，模型须同时考虑文本与图像两种模态。我们用预训练模型得到合适表示，再描述扩散骨干架构。总览见图 2。

Our general setup follows LDM (Rombach et al., 2022) for training text-to-image models in the latent space of a pretrained autoencoder. Similar to the encoding of images to latent representations, we also follow previous approaches (Saharia et al., 2022b; Balaji et al., 2022) and encode the text conditioning $c$ using pretrained, frozen text models. Details can be found in Section B.2.

总体设定遵循 LDM (Rombach et al., 2022)，在预训练自编码器的潜空间中训练文生图模型。与把图像编成潜表示类似，我们也沿用既有做法 (Saharia et al., 2022b; Balaji et al., 2022)，用预训练、冻结的文本模型编码文本条件 $c$。细节见第 B.2 节。

Multimodal Diffusion Backbone Our architecture builds upon the DiT (Peebles & Xie, 2023) architecture. DiT only considers class conditional image generation and uses a modulation mechanism to condition the network on both the timestep of the diffusion process and the class label. Similarly, we use embeddings of the timestep $t$ and $c_{\text{vec}}$ as inputs to the modulation mechanism. However, as the pooled text representation retains only coarse-grained information about the text input (Podell et al., 2023), the network also requires information from the sequence representation $c_{\text{ctxt}}$.

多模态扩散骨干 我们的架构建立在 DiT (Peebles & Xie, 2023) 之上。DiT 只考虑类别条件图像生成，并用调制机制把网络条件到扩散时间步与类别标签。类似地，我们用时间步 $t$ 与 $c_{\text{vec}}$ 的嵌入作为调制输入。但由于池化后的文本表示只保留粗粒度信息 (Podell et al., 2023)，网络还需要序列表示 $c_{\text{ctxt}}$ 中的信息。

We construct a sequence consisting of embeddings of the text and image inputs. Specifically, we add positional encodings and flatten $2\times 2$ patches of the latent pixel representation $x\in\mathbb{R}^{h\times w\times c}$ to a patch encoding sequence of length $\frac{1}{2}\cdot h\cdot\frac{1}{2}\cdot w$. After embedding this patch encoding and the text encoding $c_{\text{ctxt}}$ to a common dimensionality, we concatenate the two sequences. We then follow DiT and apply a sequence of modulated attention and MLPs.

我们构造由文本与图像嵌入组成的序列。具体地，为潜像素表示 $x\in\mathbb{R}^{h\times w\times c}$ 的 $2\times 2$ patch 加入位置编码并展平，得到长度为 $\frac{1}{2}\cdot h\cdot\frac{1}{2}\cdot w$ 的 patch 编码序列。将该 patch 编码与文本编码 $c_{\text{ctxt}}$ 嵌入到共同维度后拼接。随后按 DiT，施加一系列经调制的注意力与 MLP。

Since text and image embeddings are conceptually quite different, we use two separate sets of weights for the two modalities. As shown in Figure 2(b), this is equivalent to having two independent transformers for each modality, but joining the sequences of the two modalities for the attention operation, such that both representations can work in their own space yet take the other one into account.

由于文本与图像嵌入在概念上差异较大，我们为两模态使用两套独立权重。如图 2(b) 所示，这等价于每模态各有一个独立 Transformer，但在注意力运算时把两模态序列拼在一起，使二者各自在自己的空间工作，又能互相参照。

For our scaling experiments, we parameterize the size of the model in terms of the model’s depth $d$, i.e. the number of attention blocks, by setting the hidden size to $64\cdot d$ (expanded to $4\cdot 64\cdot d$ channels in the MLP blocks), and the number of attention heads equal to $d$.

在缩放实验中，我们用模型深度 $d$（即注意力块数）参数化规模：隐层大小取 $64\cdot d$（MLP 块中扩展到 $4\cdot 64\cdot d$ 通道），注意力头数等于 $d$。

5 Experiments

5 实验

	rank averaged over
variant	all	5 steps	50 steps
rf/lognorm(0.00, 1.00)	01.54	01.25	01.50
rf/lognorm(1.00, 0.60)	02.08	03.50	02.00
rf/lognorm(0.50, 0.60)	02.71	08.50	01.00
rf/mode(1.29)	02.75	03.25	03.00
rf/lognorm(0.50, 1.00)	02.83	01.50	02.50
eps/linear	02.88	04.25	02.75
rf/mode(1.75)	03.33	02.75	02.75
rf/cosmap	04.13	03.75	04.00
edm(0.00, 0.60)	05.63	13.25	03.25
rf	05.67	06.50	05.75
v/linear	06.83	05.75	07.75
edm(0.60, 1.20)	09.00	13.00	09.00
v/cos	09.17	12.25	08.75
edm/cos	11.04	14.25	11.25
edm/rf	13.04	15.25	13.25
edm(-1.20, 1.20)	15.58	20.25	15.00

Table 1: Global ranking of variants. For this ranking, we apply non-dominated sorting averaged over EMA and non-EMA weights, two datasets and different sampling settings.

	平均排序
变体	全部	5 步	50 步
rf/lognorm(0.00, 1.00)	01.54	01.25	01.50
rf/lognorm(1.00, 0.60)	02.08	03.50	02.00
rf/lognorm(0.50, 0.60)	02.71	08.50	01.00
rf/mode(1.29)	02.75	03.25	03.00
rf/lognorm(0.50, 1.00)	02.83	01.50	02.50
eps/linear	02.88	04.25	02.75
rf/mode(1.75)	03.33	02.75	02.75
rf/cosmap	04.13	03.75	04.00
edm(0.00, 0.60)	05.63	13.25	03.25
rf	05.67	06.50	05.75
v/linear	06.83	05.75	07.75
edm(0.60, 1.20)	09.00	13.00	09.00
v/cos	09.17	12.25	08.75
edm/cos	11.04	14.25	11.25
edm/rf	13.04	15.25	13.25
edm(-1.20, 1.20)	15.58	20.25	15.00

表 1：各变体的全局排序。 该排序对 EMA 与非 EMA 权重、两个数据集及不同采样设定，做非支配排序后取平均。

	ImageNet	CC12M
variant	CLIP	FID	CLIP	FID
rf	0.247	049.70	0.217	094.90
edm(-1.20, 1.20)	0.236	063.12	0.200	116.60
eps/linear	0.245	048.42	0.222	0**90.34
v/cos	0.244	050.74	0.209	097.87
v/linear	0.246	051.68	0.217	100.76
rf/lognorm(0.50, 0.60)	0.256	080.41	0.233	120.84
rf/mode(1.75)	0.253	0****44.39	0.218	094.06
rf/lognorm(1.00, 0.60)	0.254	114.26	0.234	147.69
rf/lognorm(-0.50, 1.00)	0.248	045.64	0.219	089.70
rf/lognorm(0.00, 1.00)	0.250	0**45.78	0.224	089.91

Table 2: Metrics for different variants. FID and CLIP scores of different variants with 25 sampling steps. We highlight the best, second best, and third best entries.

	ImageNet	CC12M
变体	CLIP	FID	CLIP	FID
rf	0.247	049.70	0.217	094.90
edm(-1.20, 1.20)	0.236	063.12	0.200	116.60
eps/linear	0.245	048.42	0.222	0**90.34
v/cos	0.244	050.74	0.209	097.87
v/linear	0.246	051.68	0.217	100.76
rf/lognorm(0.50, 0.60)	0.256	080.41	0.233	120.84
rf/mode(1.75)	0.253	0****44.39	0.218	094.06
rf/lognorm(1.00, 0.60)	0.254	114.26	0.234	147.69
rf/lognorm(-0.50, 1.00)	0.248	045.64	0.219	089.70
rf/lognorm(0.00, 1.00)	0.250	0**45.78	0.224	089.91

表 2：不同变体的指标。 25 采样步下的 FID 与 CLIP。我们标出最优、次优与第三优条目。

5.1 Improving Rectified Flows

5.1 改进整流流

We aim to understand which of the approaches for simulation-free training of normalizing flows as in Equation 1 is the most efficient. To enable comparisons across different approaches, we control for the optimization algorithm, the model architecture, the dataset and samplers. In addition, the losses of different approaches are incomparable and also do not necessarily correlate with the quality of output samples; hence we need evaluation metrics that allow for a comparison between approaches. We train models on ImageNet (Russakovsky et al., 2014) and CC12M (Changpinyo et al., 2021), and evaluate both the training and the EMA weights of the models during training using validation losses, CLIP scores (Radford et al., 2021; Hessel et al., 2021), and FID (Heusel et al., 2017) under different sampler settings (different guidance scales and sampling steps). We calculate the FID on CLIP features as proposed by (Sauer et al., 2021). All metrics are evaluated on the COCO-2014 validation split (Lin et al., 2014). Full details on the training and sampling hyperparameters are provided in Section B.3.

我们希望弄清：式 1 这类无需仿真的归一化流训练，哪种做法最有效。为跨方法可比，我们固定优化算法、模型架构、数据集与采样器。此外，不同做法的损失不可直接比较，也不一定与样本质量相关，因而需要能跨方法比较的评估指标。我们在 ImageNet (Russakovsky et al., 2014) 与 CC12M (Changpinyo et al., 2021) 上训练，并在训练过程中用验证损失、CLIP 分数 (Radford et al., 2021; Hessel et al., 2021) 与 FID (Heusel et al., 2017)，在不同采样设定（不同引导尺度与步数）下评估训练权重与 EMA 权重。FID 按 (Sauer et al., 2021) 在 CLIP 特征上计算。全部指标在 COCO-2014 验证集 (Lin et al., 2014) 上评估。训练与采样超参见第 B.3 节。

5.1.1 Results

We train each of 61 different formulations on the two datasets. We include the following variants from Section 3:

5.1.1 结果

我们在两个数据集上各训练 61 种公式。纳入第 3 节的如下变体：

• • Both $\epsilon$- and v-prediction loss with linear (eps/linear, v/linear) and cosine (eps/cos, v/cos) schedule.
• • RF loss with $\pi_{\text{mode}}(t;s)$ (rf/mode(s)) with 7 values for $s$ chosen uniformly between $-1$ and $1.75$, and additionally for $s=1.0$ and $s=0$ which corresponds to uniform timestep sampling (rf/mode).
• • RF loss with $\pi_{\text{ln}}(t;m,s)$ (rf/lognorm(m, s)) with 30 values for $(m,s)$ in the grid with $m$ uniform between $-1$ and $1$, and $s$ uniform between $0.2$ and $2.2$.
• • RF loss with $\pi_{\text{CosMap}}(t)$ (rf/cosmap).
• • EDM (edm($P_{m},P_{s}$)) with 15 values for $P_{m}$ chosen uniformly between $-1.2$ and $1.2$ and $P_{s}$ uniform between $0.6$ and $1.8$. Note that $P_{m},P_{s}=(-1.2,1.2)$ corresponds to the parameters in (Karras et al., 2022).
• • EDM with a schedule such that it matches the log-SNR weighting of rf (edm/rf) and one that matches the log-SNR weighting of v/cos (edm/cos).

• • 线性（eps/linear、v/linear）与余弦（eps/cos、v/cos）日程下的 $\epsilon$- 与 v-预测损失。
• • 带 $\pi_{\text{mode}}(t;s)$ 的 RF 损失（rf/mode(s)）：$s$ 在 $-1$ 与 $1.75$ 间均匀取 7 个值，另加 $s=1.0$ 与 $s=0$（均匀时间步，记 rf/mode）。
• • 带 $\pi_{\text{ln}}(t;m,s)$ 的 RF 损失（rf/lognorm(m, s)）：$(m,s)$ 在网格上取 30 组，$m$ 在 $-1$ 与 $1$ 间均匀，$s$ 在 $0.2$ 与 $2.2$ 间均匀。
• • 带 $\pi_{\text{CosMap}}(t)$ 的 RF 损失（rf/cosmap）。
• • EDM（edm($P_{m},P_{s}$)）：$P_{m}$ 在 $-1.2$ 与 $1.2$ 间均匀取 15 个值，$P_{s}$ 在 $0.6$ 与 $1.8$ 间均匀。注意 $P_{m},P_{s}=(-1.2,1.2)$ 对应 (Karras et al., 2022) 的参数。
• • 使 log-SNR 加权分别匹配 rf（edm/rf）与 v/cos（edm/cos）的 EDM 日程。

For each run, we select the step with minimal validation loss when evaluated with EMA weights and then collect CLIP scores and FID obtained with 6 different sampler settings both with and without EMA weights.

对每次运行，我们选取 EMA 权重下验证损失最低的步数，再收集 6 种采样设定下、带或不带 EMA 的 CLIP 与 FID。

For all 24 combinations of sampler settings, EMA weights, and dataset choice, we rank the different formulations using a non-dominated sorting algorithm. For this, we repeatedly compute the variants that are Pareto optimal according to CLIP and FID scores, assign those variants the current iteration index, remove those variants, and continue with the remaining ones until all variants get ranked. Finally, we average those ranks over the 24 different control settings.

对采样设定、EMA 权重与数据集选择的全部 24 种组合，我们用非支配排序算法给各公式打分：反复按 CLIP 与 FID 找出帕累托最优变体，赋当前迭代序号后剔除，直至全部排完；最后对 24 种控制设定取平均秩。

We present the results in Tab. 1, where we only show the two best-performing variants for those variants that were evaluated with different hyperparameters. We also show ranks where we restrict the averaging over sampler settings with 5 steps and with 50 steps.

结果见表 1；对扫过多种超参的变体，我们只展示表现最好的两种。我们也给出仅在 5 步与仅在 50 步采样设定上平均的秩。

We observe that rf/lognorm(0.00, 1.00) consistently achieves a good rank. It outperforms a rectified flow formulation with uniform timestep sampling (rf) and thus confirms our hypothesis that intermediate timesteps are more important. Among all the variants, only rectified flow formulations with modified timestep sampling perform better than the LDM-Linear (Rombach et al., 2022) formulation (eps/linear) used previously.

我们观察到 rf/lognorm(0.00, 1.00) 持续取得较好名次。它超过均匀时间步采样的整流流（rf），从而印证「中间时间步更重要」的假设。在全部变体中，只有改过时间步采样的整流流公式超过此前使用的 LDM-Linear (Rombach et al., 2022)（eps/linear）。

We also observe that some variants perform well in some settings but worse in others, e.g. rf/lognorm(0.50, 0.60) is the best-performing variant with 50 sampling steps but much worse (average rank 8.5) with 5 sampling steps. We observe a similar behavior with respect to the two metrics in Tab. 2. The first group shows representative variants and their metrics on both datasets with 25 sampling steps. The next group shows the variants that achieve the best CLIP and FID scores. With the exception of rf/mode(1.75), these variants typically perform very well in one metric but relatively badly in the other. In contrast, we once again observe that rf/lognorm(0.00, 1.00) achieves good performance across metrics and datasets, where it obtains the third-best scores two out of four times and once the second-best performance.

我们也观察到：有些变体在某些设定下很好、在另一些下较差，例如 rf/lognorm(0.50, 0.60) 在 50 采样步时最优，但在 5 步时差很多（平均秩 8.5）。表 2 中两指标亦有类似现象。第一组展示代表性变体在两数据集、25 采样步下的指标；下一组展示取得最佳 CLIP 与 FID 的变体。除 rf/mode(1.75) 外，这些变体往往在一个指标上很好、在另一个上相对较差。相比之下，rf/lognorm(0.00, 1.00) 再次在跨指标与跨数据集上表现稳健：四次中两次第三、一次第二。

Finally, we illustrate the qualitative behavior of different formulations in Figure 3, where we use different colors for different groups of formulations (edm, rf, eps and v). Rectified flow formulations generally perform well and, compared to other formulations, their performance degrades less when reducing the number of sampling steps.

最后，图 3 用不同颜色标出不同公式族（edm、rf、eps 与 v）的定性行为。整流流整体表现较好，且相对其他公式，减少采样步数时性能下降更少。

Figure 3: Rectified flows are sample efficient. Rectified Flows perform better then other formulations when sampling fewer steps. For 25 and more steps, only rf/lognorm(0.00, 1.00) remains competitive to eps/linear.

图 3：整流流的采样效率高。 采样步数较少时，整流流优于其他公式。达到 25 步及以上时，仅 rf/lognorm(0.00, 1.00) 仍能与 eps/linear 抗衡。

5.2 Improving Modality Specific Representations

5.2 改进模态特定表示

Having found a formulation in the previous section that allows rectified flow models to not only compete with established diffusion formulations such as LDM-Linear (Rombach et al., 2022) or EDM (Karras et al., 2022), but even outperforms them, we now turn to the application of our formulation to high-resolution text-to-image synthesis. Accordingly, the final performance of our algorithm depends not only on the training formulation, but also on the parameterization via a neural network and the quality of the image and text representations we use. In the following sections, we describe how we improve all these components before scaling our final method in Section 5.3.

上一节已找到一种公式，使整流流不仅能与 LDM-Linear (Rombach et al., 2022) 或 EDM (Karras et al., 2022) 等既有扩散公式竞争，甚至超过它们；我们转而把它用于高分辨率文生图。因而最终表现不仅取决于训练公式，也取决于神经网络参数化以及所用图像与文本表示的质量。以下各节说明我们如何改进这些组件，再在第 5.3 节缩放最终方法。

5.2.1 Improved Autoencoders

Latent diffusion models achieve high efficiency by operating in the latent space of a pretrained autoencoder (Rombach et al., 2022), which maps an input RGB $X\in\mathbb{R}^{H\times W\times 3}$ into a lower-dimensional space $x=E(X)\in\mathbb{R}^{h\times w\times d}$. The reconstruction quality of this autoencoder provides an upper bound on the achievable image quality after latent diffusion training. Similar to Dai et al. (2023), we find that increasing the number of latent channels $d$ significantly boosts reconstruction performance, see Table 3. Intuitively, predicting latents with higher $d$ is a more difficult task, and thus models with increased capacity should be able to perform better for larger $d$, ultimately achieving higher image quality. We confirm this hypothesis in Figure 10, where we see that the $d=16$ autoencoder exhibits better scaling performance in terms of sample FID. For the remainder of this paper, we thus choose $d=16$.

5.2.1 改进的自编码器

潜扩散通过在预训练自编码器的潜空间中运算获得高效性 (Rombach et al., 2022)：把 RGB 输入 $X\in\mathbb{R}^{H\times W\times 3}$ 映到更低维空间 $x=E(X)\in\mathbb{R}^{h\times w\times d}$。该自编码器的重建质量为潜扩散训练后可达的图像质量设下上界。与 Dai et al. (2023) 类似，我们发现增大潜通道数 $d$ 显著提升重建表现，见表 3。直观上，预测更高 $d$ 的潜表示更难，因而更大容量的模型应在更大 $d$ 上表现更好，最终得到更高图像质量。图 10 印证了这一点：$d=16$ 的自编码器在样本 FID 上展现更好的缩放。后文因此取 $d=16$。

Metric	4 chn	8 chn	16 chn
FID ($\downarrow$)	02.41	01.56	0****1.06
Perceptual Similarity ($\downarrow$)	00.85	00.68	0****0.45
SSIM ($\uparrow$)	00.75	00.79	0****0.86
PSNR ($\uparrow$)	25.12	26.40	28.62

Table 3: Improved Autoencoders. Reconstruction performance metrics for different channel configurations. The downsampling factor for all models is $f=8$.

指标	4 通道	8 通道	16 通道
FID ($\downarrow$)	02.41	01.56	0****1.06
感知相似度 ($\downarrow$)	00.85	00.68	0****0.45
SSIM ($\uparrow$)	00.75	00.79	0****0.86
PSNR ($\uparrow$)	25.12	26.40	28.62

表 3：改进的自编码器。 不同通道配置下的重建指标。全部模型下采样因子 $f=8$。

a space elevator, cinematic scifi art

A cheeseburger with juicy beef patties and melted cheese sits on top of a toilet that looks like a throne and stands in the middle of the royal chamber.

a hole in the floor of my bathroom with small gremlins living in it

a small office made out of car parts

This dreamlike digital art captures a vibrant, kaleidoscopic bird in a lush rainforest.

human life depicted entirely out of fractals

an origami pig on fire in the middle of a dark room with a pentagram on the floor

an old rusted robot wearing pants and a jacket riding skis in a supermarket.

smiling cartoon dog sits at a table, coffee mug on hand, as a room goes up in flames. “This is fine,” the dog assures himself.

A whimsical and creative image depicting a hybrid creature that is a mix of a waffle and a hippopotamus. This imaginative creature features the distinctive, bulky body of a hippo, but with a texture and appearance resembling a golden-brown, crispy waffle. The creature might have elements like waffle squares across its skin and a syrup-like sheen. It’s set in a surreal environment that playfully combines a natural water habitat of a hippo with elements of a breakfast table setting, possibly including oversized utensils or plates in the background. The image should evoke a sense of playful absurdity and culinary fantasy.

（图中提示示例，原文保留）

a space elevator, cinematic scifi art

A cheeseburger with juicy beef patties and melted cheese sits on top of a toilet that looks like a throne and stands in the middle of the royal chamber.

a hole in the floor of my bathroom with small gremlins living in it

a small office made out of car parts

This dreamlike digital art captures a vibrant, kaleidoscopic bird in a lush rainforest.

human life depicted entirely out of fractals

an origami pig on fire in the middle of a dark room with a pentagram on the floor

an old rusted robot wearing pants and a jacket riding skis in a supermarket.

smiling cartoon dog sits at a table, coffee mug on hand, as a room goes up in flames. “This is fine,” the dog assures himself.

A whimsical and creative image depicting a hybrid creature that is a mix of a waffle and a hippopotamus. This imaginative creature features the distinctive, bulky body of a hippo, but with a texture and appearance resembling a golden-brown, crispy waffle. The creature might have elements like waffle squares across its skin and a syrup-like sheen. It’s set in a surreal environment that playfully combines a natural water habitat of a hippo with elements of a breakfast table setting, possibly including oversized utensils or plates in the background. The image should evoke a sense of playful absurdity and culinary fantasy.

5.2.2 Improved Captions

Betker et al. (2023) demonstrated that synthetically generated captions can greatly improve text-to-image models trained at scale. This is due to the oftentimes simplistic nature of the human-generated captions that come with large-scale image datasets, which overly focus on the image subject and usually omit details describing the background or composition of the scene, or, if applicable, displayed text (Betker et al., 2023). We follow their approach and use an off-the-shelf, state-of-the-art vision-language model, CogVLM (Wang et al., 2023), to create synthetic annotations for our large-scale image dataset. As synthetic captions may cause a text-to-image model to forget about certain concepts not present in the VLM’s knowledge corpus, we use a ratio of 50 % original and 50 % synthetic captions.

5.2.2 改进的标题

Betker et al. (2023) 表明：合成生成的标题可大幅提升大规模训练的文生图模型。原因在于大规模图像数据自带的人工标题往往过于简略，过度聚焦主体，常省略背景、构图，以及（如有）图中文字的细节 (Betker et al., 2023)。我们沿用该思路，用现成的先进视觉–语言模型 CogVLM (Wang et al., 2023) 为大规模图像数据生成合成标注。由于合成标题可能使模型遗忘 VLM 知识库中未覆盖的概念，我们采用 50% 原始 + 50% 合成的比例。

To assess the effect of training on this caption mix, we train two $d=15$ MM-DiT models for 250k steps, one on only original captions and the other on the 50/50 mix. We evaluate the trained models using the GenEval benchmark (Ghosh et al., 2023) in Table 4. The results demonstrate that the model trained with the addition of synthetic captions clearly outperforms the model that only utilizes original captions. We thus use the 50/50 synthetic/original caption mix for the remainder of this work.

为评估该标题混合的效果，我们训练两个 $d=15$ 的 MM-DiT 各 250k 步：其一仅用原始标题，其二用 50/50 混合。用 GenEval 基准 (Ghosh et al., 2023) 评估，见表 4。结果表明：加入合成标题的模型明显超过仅用原始标题者。后文因此沿用 50/50 合成/原始混合。

	Original Captions	50/50 Mix
	success rate [%]	success rate [%]
Color Attribution	11.75	24.75
Colors	71.54	68.09
Position	06.50	18.00
Counting	33.44	41.56
Single Object	95.00	93.75
Two Objects	41.41	52.53
Overall score	43.27	49.78

Table 4: Improved Captions. Using a 50/50 mixing ratio of synthetic (via CogVLM (Wang et al., 2023)) and original captions improves text-to-image performance. Assessed via the GenEval (Ghosh et al., 2023) benchmark.

	原始标题	50/50 混合
	成功率 [%]	成功率 [%]
颜色归属	11.75	24.75
颜色	71.54	68.09
位置	06.50	18.00
计数	33.44	41.56
单物体	95.00	93.75
两物体	41.41	52.53
总分	43.27	49.78

表 4：改进的标题。 以 50/50 混合合成（经 CogVLM (Wang et al., 2023)）与原始标题可提升文生图表现；经 GenEval (Ghosh et al., 2023) 评估。

5.2.3 Improved Text-to-Image Backbones

In this section, we compare the performance of existing transformer-based diffusion backbones with our novel multimodal transformer-based diffusion backbone, MM-DiT, as introduced in Section 4. MM-DiT is specifically designed to handle different domains, here text and image tokens, using (two) different sets of trainable model weights. More specifically, we follow the experimental setup from Section 5.1 and compare text-to-image performance on CC12M of DiT, CrossDiT (DiT but with cross-attending to the text tokens instead of sequence-wise concatenation (Chen et al., 2023)) and our MM-DiT. For MM-DiT, we compare models with two sets of weights and three sets of weights, where the latter handles the CLIP (Radford et al., 2021) and T5 (Raffel et al., 2019) tokens (c.f. Section 4) separately. Note that DiT (w/ concatenation of text and image tokens as in Section 4) can be interpreted as a special case of MM-DiT with one shared set of weights for all modalities. Finally, we consider the UViT (Hoogeboom et al., 2023) architecture as a hybrid between the widely used UNets and transformer variants.

5.2.3 改进的文生图骨干

本节把既有基于 Transformer 的扩散骨干与第 4 节引入的新型多模态扩散骨干 MM-DiT 对比。MM-DiT 专为处理不同域（此处为文本与图像 token）设计，使用（两）套不同的可训练权重。更具体地，我们沿用第 5.1 节设定，在 CC12M 上比较 DiT、CrossDiT（DiT 但对文本 token 做交叉注意力而非序列拼接 (Chen et al., 2023)）与我们的 MM-DiT。对 MM-DiT，我们比较两套权重与三套权重：后者把 CLIP (Radford et al., 2021) 与 T5 (Raffel et al., 2019) token（参见第 4 节）分开处理。注意：按第 4 节拼接文本与图像 token 的 DiT，可视为全部模态共享一套权重的 MM-DiT 特例。最后，我们把 UViT (Hoogeboom et al., 2023) 视为广泛使用的 UNet 与 Transformer 变体之间的混合。

We analyze the convergence behavior of these architectures in Figure 4: Vanilla DiT underperforms UViT. The cross-attention DiT variant CrossDiT achieves better performance than UViT, although UViT seems to learn much faster initially. Our MM-DiT variant significantly outperforms the cross-attention and vanilla variants. We observe only a small gain when using three parameter sets instead of two (at the cost of increased parameter count and VRAM usage), and thus opt for the former option for the remainder of this work.

图 4 分析这些架构的收敛：原版 DiT 不及 UViT。交叉注意力变体 CrossDiT 超过 UViT，尽管 UViT 初期学得更快。我们的 MM-DiT 显著超过交叉注意力与原版变体。用三套参数相对两套仅有小幅收益（代价是参数量与显存上升），因而后文采用两套权重方案。

Figure 4: Training dynamics of model architectures. Comparative analysis of DiT, CrossDiT, UViT, and MM-DiT on CC12M, focusing on validation loss, CLIP score, and FID. Our proposed MM-DiT performs favorably across all metrics.

图 4：模型架构的训练动态。 在 CC12M 上比较 DiT、CrossDiT、UViT 与 MM-DiT 的验证损失、CLIP 分数与 FID。所提 MM-DiT 在各项指标上均表现更好。

5.3 Training at Scale

5.3 规模化训练

Before scaling up, we filter and preencode our data to ensure safe and efficient pretraining. Then, all previous considerations of diffusion formulations, architectures, and data culminate in the last section, where we scale our models up to 8B parameters.

在放大之前，我们过滤并预编码数据，以保证预训练安全且高效。随后，此前关于扩散公式、架构与数据的全部考量汇于本节：把模型扩到 8B 参数。

Pre-Training Mitigations

Training data significantly impacts a generative model’s abilities. Consequently, data filtering is effective at constraining undesirable capabilities (Nichol, 2022). Before training at sale, we filter our data for the following categories: (i) Sexual content: We use NSFW-detection models to filter for explicit content. (ii) Aesthetics: We remove images for which our rating systems predict a low score. (iii) Regurgitation: We use a cluster-based deduplication method to remove perceptual and semantic duplicates from the training data; see Section E.2.

预训练缓解措施

训练数据显著影响生成模型能力。因此，数据过滤能有效约束不良能力 (Nichol, 2022)。放大训练前，我们按以下类别过滤：(i) 色情内容：用 NSFW 检测模型过滤露骨内容。(ii) 美学：去掉评分系统预测分低的图像。(iii) 复读/回吐：用基于聚类的去重去掉感知与语义重复；见第 E.2 节。

Precomputing Image and Text Embeddings

Our model uses the output of multiple pretrained, frozen networks as inputs (autoencoder latents and text encoder representations). Since these outputs are constant during training, we precompute them once for the entire dataset. We provide a detailed discussion of our approach in Section E.1.

预计算图像与文本嵌入

模型把多个预训练、冻结网络的输出（自编码器潜表示与文本编码器表示）当作输入。这些输出在训练中恒定，我们因而对整个数据集预计算一次。详细讨论见第 E.1 节。

5.3.2 Finetuning on High Resolutions

Figure 5: Effects of QK-normalization. Normalizing the Q- and K-embeddings before calculating the attention matrix prevents the attention-logit growth instability (left), which causes the attention entropy to collapse (right) and has been previously reported in the discriminative ViT literature (Dehghani et al., 2023; Wortsman et al., 2023). In contrast with these previous works, we observe this instability in the last transformer blocks of our networks. Maximum attention logits and attention entropies are shown averaged over the last 5 blocks of a 2B (d=24) model.

5.3.2 高分辨率微调

图 5：QK 归一化的效果。 在计算注意力矩阵前对 Q、K 嵌入归一化，可防止注意力 logit 增长导致的不稳定（左），该不稳定会使注意力熵塌缩（右），此前已在判别式 ViT 文献中报告 (Dehghani et al., 2023; Wortsman et al., 2023)。与这些工作不同，我们在网络最后若干 Transformer 块中观察到该不稳定。图中最大注意力 logit 与注意力熵对 2B（d=24）模型最后 5 块取平均。

QK-Normalization

In general, we pretrain all of our models on low-resolution images of size $256^{2}$ pixels. Next, we finetune our models on higher resolutions with mixed aspect ratios (see next paragraph for details). We find that, when moving to high resolutions, mixed precision training can become unstable and the loss diverges. This can be remedied by switching to full precision training — but comes with a $\sim 2\times$ performance drop compared to mixed-precision training. A more efficient alternative is reported in the (discriminative) ViT literature: Dehghani et al. (2023) observe that the training of large vision transformer models diverges because the attention entropy grows uncontrollably. To avoid this, Dehghani et al. (2023) propose to normalize Q and K before the attention operation. We follow this approach and use RMSNorm (Zhang & Sennrich, 2019) with learnable scale in both streams of our MMDiT architecture for our models, see Figure 2. As demonstrated in Figure 5, the additional normalization prevents the attention logit growth instability, confirming findings by Dehghani et al. (2023) and Wortsman et al. (2023) and enables efficient training at bf16-mixed (Chen et al., 2019) precision when combined with $\epsilon=10^{-15}$ in the AdamW (Loshchilov & Hutter, 2017) optimizer. This technique can also be applied on pretrained models that have not used qk-normalization during pretraining: The model quickly adapts to the additional normalization layers and trains more stably. Finally, we would like to point out that although this method can generally help to stabilize the training of large models, it is not a universal recipe and may need to be adapted depending on the exact training setup.

QK 归一化

一般地，我们先在 $256^{2}$ 低分辨率图像上预训练全部模型，再在混合宽高比的更高分辨率上微调（细节见下段）。我们发现：进入高分辨率后，混合精度训练可能不稳、损失发散。切到全精度可缓解，但相对混合精度大约慢 $\sim 2\times$。更高效的替代方案见于（判别式）ViT 文献：Dehghani et al. (2023) 观察到大视觉 Transformer 训练因注意力熵失控增长而发散；为避免这一点，他们建议在注意力前对 Q、K 归一化。我们沿用该做法，在 MMDiT 两流中使用带可学习尺度的 RMSNorm (Zhang & Sennrich, 2019)，见图 2。如图 5 所示，额外归一化防止了注意力 logit 增长不稳定，印证 Dehghani et al. (2023) 与 Wortsman et al. (2023) 的发现，并在与 AdamW (Loshchilov & Hutter, 2017) 中 $\epsilon=10^{-15}$ 结合时，支持 bf16-mixed (Chen et al., 2019) 高效训练。该技术也可用于预训练时未使用 qk-归一化的模型：模型能很快适应额外归一化层并更稳地训练。最后需指出：尽管该方法通常有助于稳定大模型训练，它并非万能配方，仍须按具体训练设定调整。

Positional Encodings for Varying Aspect Ratios

After training on a fixed $256\times 256$ resolution we aim to (i) increase the resolution and resolution and (ii) enable inference with flexible aspect ratios. Since we use 2d positional frequency embeddings we have to adapt them based on the resolution. In the multi-aspect ratio setting, a direct interpolation of the embeddings as in (Dosovitskiy et al., 2020) would not reflect the side lengths correctly. Instead we use a combination of extended and interpolated position grids which are subsequently frequency embedded.

可变宽高比的位置编码

在固定 $256\times 256$ 分辨率训练后，我们希望 (i) 提高分辨率，(ii) 支持灵活宽高比推理。由于使用二维位置频率嵌入，须按分辨率适配。在多宽高比设定下，像 (Dosovitskiy et al., 2020) 那样直接插值嵌入无法正确反映边长。我们改用扩展与插值相结合的位置网格，再做频率嵌入。

For a target resolution of $S^{2}$ pixels, we use bucketed sampling (NovelAI, 2022; Podell et al., 2023) such that that each batch consists of images of a homogeneous size $H\times W$, where $H\cdot W\approx S^{2}$. For the maximum and minimum training aspect ratios, this results in the maximum values for width, $W_{\text{max}}$, and height, $H_{\text{max}}$, that will be encountered. Let $h_{\text{max}}=H_{\text{max}}/16,w_{\text{max}}=W_{\text{max}}/16$ and $s=S/16$ be the corresponding sizes in latent space (a factor 8) after patching (a factor 2). Based on these values, we construct a vertical position grid with the values $((p-\frac{h_{\text{max}}-s}{2})\cdot\frac{256}{S})_{p=0}^{{h_{\text{max}}-1}}$ and correspondingly for the horizontal positions. We then center-crop from the resulting positional 2d grid before embedding it.

对目标分辨率 $S^{2}$ 像素，我们用分桶采样 (NovelAI, 2022; Podell et al., 2023)，使每个 batch 内图像尺寸齐整为 $H\times W$，且 $H\cdot W\approx S^{2}$。按训练时最大与最小宽高比，得到会遇到的最大宽 $W_{\text{max}}$ 与高 $H_{\text{max}}$。令 $h_{\text{max}}=H_{\text{max}}/16,w_{\text{max}}=W_{\text{max}}/16$，$s=S/16$ 为潜空间（下采样因子 8）再经 patch（因子 2）后的对应尺寸。据此构造竖直位置网格，取值为 $((p-\frac{h_{\text{max}}-s}{2})\cdot\frac{256}{S})_{p=0}^{{h_{\text{max}}-1}}$，水平方向同理。再从所得二维位置网格中心裁剪后嵌入。

Figure 6: Timestep shifting at higher resolutions. Top right: Human quality preference rating when applying the shifting based on Equation 23. Bottom row: A $512^{2}$ model trained and sampled with $\sqrt{m/n}=1.0$ (top) and $\sqrt{m/n}=3.0$ (bottom). See Section 5.3.2.

图 6：更高分辨率下的时间步平移。 右上： 按式 23 施加平移时的人类质量偏好。底行： 在 $512^{2}$ 上训练并采样、$\sqrt{m/n}=1.0$（上）与 $\sqrt{m/n}=3.0$（下）的对比。见第 5.3.2 节。

Resolution-dependent shifting of timestep schedules

Intuitively, since higher resolutions have more pixels, we need more noise to destroy their signal. Assume we are working in a resolution with $n=H\cdot W$ pixels. Now, consider a ”constant” image, i.e. one where every pixel has the value $c$. The forward process produces $z_{t}=(1-t)c\mathbbm{1}+t\epsilon$, where both $\mathbbm{1}$ and $\epsilon\in\mathbb{R}^{n}$. Thus, $z_{t}$ provides $n$ observations of the random variable $Y=(1-t)c+t\eta$ with $c$ and $\eta$ in $\mathbb{R}$, and $\eta$ follows a standard normal distribution. Thus, $\mathbb{E}(Y)=(1-t)c$ and $\sigma(Y)=t$. We can therefore recover $c$ via $c=\frac{1}{1-t}\mathbb{E}(Y)$, and the error between $c$ and its sample estimate $\hat{c}=\frac{1}{1-t}\sum_{i=1}^{n}z_{t,i}$ has a standard deviation of $\sigma(t,n)=\frac{t}{1-t}\sqrt{\frac{1}{n}}$ (because the standard error of the mean for $Y$ has deviation $\frac{t}{\sqrt{n}}$). So if one already knows that the image $z_{0}$ was constant across its pixels, $\sigma(t,n)$ represents the degree of uncertainty about $z_{0}$. For example, we immediately see that doubling the width and height leads to half the uncertainty at any given time $0<t<1$. But, we can now map a timestep $t_{n}$ at resolution $n$ to a timestep $t_{m}$ at resolution $m$ that results in the same degree of uncertainty via the ansatz $\sigma(t_{n},n)=\sigma(t_{m},m)$. Solving for $t_{m}$ gives

随分辨率平移时间步日程

直观上，更高分辨率像素更多，需要更多噪声才能破坏信号。设分辨率为 $n=H\cdot W$ 像素。考虑「常数」图像，即每个像素取值 $c$。前向过程得到 $z_{t}=(1-t)c\mathbbm{1}+t\epsilon$，其中 $\mathbbm{1}$ 与 $\epsilon\in\mathbb{R}^{n}$。于是 $z_{t}$ 提供随机变量 $Y=(1-t)c+t\eta$（$c,\eta\in\mathbb{R}$，$\eta$ 标准正态）的 $n$ 次观测。故 $\mathbb{E}(Y)=(1-t)c$，$\sigma(Y)=t$。因而可用 $c=\frac{1}{1-t}\mathbb{E}(Y)$ 恢复 $c$，而 $c$ 与样本估计 $\hat{c}=\frac{1}{1-t}\sum_{i=1}^{n}z_{t,i}$ 之间误差的标准差为 $\sigma(t,n)=\frac{t}{1-t}\sqrt{\frac{1}{n}}$（因 $Y$ 均值标准误为 $\frac{t}{\sqrt{n}}$）。因此，若已知 $z_{0}$ 在像素上恒定，$\sigma(t,n)$ 表示对 $z_{0}$ 的不确定程度。例如立即可见：宽高各加倍，则任意 $0<t<1$ 上的不确定度减半。我们可用 ansatz $\sigma(t_{n},n)=\sigma(t_{m},m)$，把分辨率 $n$ 上的时间步 $t_{n}$ 映到分辨率 $m$ 上具有相同不确定度的 $t_{m}$。求解 $t_{m}$ 得

$$t_{m}=\frac{\sqrt{\frac{m}{n}}t_{n}}{1+(\sqrt{\frac{m}{n}}-1)t_{n}} \tag{23}$$

Figure 7: Human Preference Evaluation against currrent closed and open SOTA generative image models. Our 8B model compares favorable against current state-of-the-art text-to-image models when evaluated on the parti-prompts (Yu et al., 2022) across the categories visual quality, prompt following and typography generation.

图 7：相对当前闭源与开源文生图最优模型的人类偏好评估。 我们的 8B 模型在 parti-prompts (Yu et al., 2022) 上，于视觉质量、遵循提示与排版生成等类别中，与当前最优文生图模型相比表现更优。

We visualize this shifting function in Figure 6. Note that the assumption of constant images is not realistic. To find good values for the shift value $\alpha\coloneq\sqrt{\frac{m}{n}}$ during inference, we apply them to the sampling steps of a model trained at resolution $1024\times 1024$ and run a human preference study. The results in Figure 6 show a strong preference for samples with shifts greater than $1.5$ but less drastic differences among the higher shift values. In our subsequent experiments, we thus use a shift value of $\alpha=3.0$ both during training and sampling at resolution $1024\times 1024$. A qualitative comparison between samples after 8k training steps with and without such a shift can be found in Figure 6. Finally, note that Equation 23 implies a log-SNR shift of $\log\frac{n}{m}$ similar to (Hoogeboom et al., 2023):

图 6 可视化该平移函数。注意「常数图像」假设并不现实。为在推理时找合适的平移值 $\alpha\coloneq\sqrt{\frac{m}{n}}$，我们把它施加到在 $1024\times 1024$ 上训练的模型的采样步上，并做人类偏好研究。图 6 显示：平移大于 $1.5$ 的样本明显更受偏好，更高平移值之间差异不那么剧烈。后续实验因此在 $1024\times 1024$ 训练与采样时取 $\alpha=3.0$。有无该平移、训练 8k 步后的定性对比亦见图 6。最后，式 23 蕴含与 (Hoogeboom et al., 2023) 类似的 log-SNR 平移 $\log\frac{n}{m}$：

$$\displaystyle\lambda_{t_{m}} \\ \displaystyle=2\log\frac{1-t_{n}}{\sqrt{\frac{m}{n}}t_{n}} \\ \displaystyle=\lambda_{t_{n}}-2\log\alpha=\lambda_{t_{n}}-\log\frac{m}{n}\;. \tag{24}$$

After the shifted training at resolution $1024\times 1024$, we align the model using Direct Preference Optimization (DPO) as described in Appendix C.

在 $1024\times 1024$ 上完成平移训练后，我们按附录 C 用直接偏好优化（DPO）对齐模型。

5.3.3 Results

Figure 8: Quantitative effects of scaling. We analyze the impact of model size on performance, maintaining consistent training hyperparameters throughout. An exception is depth=38, where learning rate adjustments at $3\times 10^{5}$ steps were necessary to prevent divergence. (Top) Validation loss smoothly decreases as a function of both model size and training steps for both image (columns 1 and 2) and video models (columns 3 and 4). (Bottom) Validation loss is a strong predictor of overall model performance. There is a marked correlation between validation loss and holistic image evaluation metrics, including GenEval (Ghosh et al., 2023), column 1, human preference, column 2, and T2I-CompBench (Huang et al., 2023), column 3. For video models we observe a similar correlation between validation loss and human preference, column 4. .

5.3.3 结果

图 8：缩放的定量效果。 我们分析模型规模对表现的影响，训练超参保持一致；例外是 depth=38，在 $3\times 10^{5}$ 步时调整学习率以防发散。(上) 图像（第 1–2 列）与视频模型（第 3–4 列）的验证损失均随模型规模与训练步数平滑下降。(下) 验证损失是整体模型表现的强预测因子：与 GenEval (Ghosh et al., 2023)（第 1 列）、人类偏好（第 2 列）及 T2I-CompBench (Huang et al., 2023)（第 3 列）等整体图像评估强相关。视频模型上，验证损失与人类偏好（第 4 列）亦有类似相关。

In Figure 8, we examine the effect of training our MM-DiT at scale. For images, we conduct a large scaling study and train models with different numbers of parameters for 500k steps on $256^{2}$ pixels resolution using preencoded data, c.f. Section E.1, with a batch size of 4096. We train on $2\times 2$ patches (Peebles & Xie, 2023), and report validation losses on the CoCo dataset (Lin et al., 2014) every 50k steps. In particular, to reduce noise in the validation loss signal, we sample loss levels equidistant in $t\in(0,1)$ and compute validation loss for each level separately. We then average the loss across all but the last ($t=1$) levels.

图 8 考察规模化训练 MM-DiT 的效果。对图像，我们做大规模缩放研究：用不同参数量的模型在 $256^{2}$ 上训练 500k 步，使用预编码数据（参见第 E.1 节），batch 大小 4096。我们在 $2\times 2$ patch 上训练 (Peebles & Xie, 2023)，每 50k 步在 CoCo (Lin et al., 2014) 上报验证损失。为降低验证损失噪声，我们在 $t\in(0,1)$ 上等距采样损失水平，分别计算，再对除最后一层（$t=1$）外全部水平取平均。

Similarly, we conduct a preliminary scaling study of our MM-DiT on videos. To this end we start from the pretrained image weights and additionally use a 2x temporal patching. We follow Blattmann et al. (2023b) and feed data to the pretrained model by collapsing the temporal into the batch axis. In each attention layer we rearrange the representation in the visual stream and add a full attention over all spatio-temporal tokens after the spatial attention operation before the final feedforward layer. Our video models are trained for 140k steps with a batch size of 512 on videos comprising 16 frames with $256^{2}$ pixels. We report validation losses on the Kinetics dataset (Carreira & Zisserman, 2018) every 5k steps. Note that our reported FLOPs for video training in Figure 8 are only FLOPs from video training and do not include the FLOPs from image pretraining.

类似地，我们对 MM-DiT 做初步视频缩放研究。从预训练图像权重出发，额外使用 2× 时间 patch。我们沿用 Blattmann et al. (2023b)：把时间维塌到 batch 轴再喂入预训练模型；在每个注意力层，重排视觉流表示，并在空间注意力之后、最终前馈之前，对全部时空 token 做一次全注意力。视频模型训练 140k 步，batch 512，视频为 16 帧、$256^{2}$ 像素。每 5k 步在 Kinetics (Carreira & Zisserman, 2018) 上报验证损失。注意图 8 中视频训练的 FLOPs 仅含视频阶段，不含图像预训练。

For both the image and video domains, we observe a smooth decrease in the validation loss when increasing model size and training steps. We find the validation loss to be highly correlated to comprehensive evaluation metrics (CompBench (Huang et al., 2023), GenEval (Ghosh et al., 2023)) and to human preference. These results support the validation loss as a simple and general measure of model performance. Our results do not show saturation neither for image not for video models.

在图像与视频域，我们都观察到：增大模型与训练步数时，验证损失平滑下降。验证损失与综合评估指标（CompBench (Huang et al., 2023)、GenEval (Ghosh et al., 2023)）及人类偏好高度相关。这些结果支持把验证损失当作简单且通用的表现度量。图像与视频模型均未见饱和迹象。

Figure 12 illustrates how training a larger model for longer impacts sample quality. Tab. 5 shows the results of GenEval in full. When applying the methods presented in Section 5.3.2 and increasing training image resolution, our biggest model excels in most categories and outperforms DALLE 3 (Betker et al., 2023), the current state of the art in prompt comprehension, in overall score.

图 12 展示更长时间训练更大模型如何影响样本质量。表 5 给出完整 GenEval 结果。应用第 5.3.2 节方法并提高训练分辨率后，我们最大的模型在多数类别领先，并在总分上超过当前提示理解最优的 DALL·E 3 (Betker et al., 2023)。

All text-encoders	w/o T5 (Raffel et al., 2019)
“A burger patty, with the bottom bun and lettuce and tomatoes. ”COFFEE” written on it in mustard”
“A monkey holding a sign reading ”Scaling transformer models is awesome!”
“A mischievous ferret with a playful grin squeezes itself into a large glass jar, surrounded by colorful candy. The jar sits on a wooden table in a cozy kitchen, and warm sunlight filters through a nearby window”

Figure 9: Impact of T5. We observe T5 to be important for complex prompts e.g. such involving a high degree of detail or longer spelled text (rows 2 and 3). For most prompts, however, we find that removing T5 at inference time still achieves competitive performance.

全部文本编码器	去掉 T5 (Raffel et al., 2019)
“A burger patty, with the bottom bun and lettuce and tomatoes. ”COFFEE” written on it in mustard”
“A monkey holding a sign reading ”Scaling transformer models is awesome!”
“A mischievous ferret with a playful grin squeezes itself into a large glass jar, surrounded by colorful candy. The jar sits on a wooden table in a cozy kitchen, and warm sunlight filters through a nearby window”

图 9：T5 的影响。 我们观察到 T5 对复杂提示（例如细节很多或较长拼写文字，第 2–3 行）很重要。但对多数提示，推理时去掉 T5 仍可达到有竞争力的表现。

Our $d=38$ model outperforms current proprietary (Betker et al., 2023; ide, 2024) and open (Sauer et al., 2023; pla, 2024; Chen et al., 2023; Pernias et al., 2023) SOTA generative image models in human preference evaluation on the Parti-prompts benchmark (Yu et al., 2022) in the categories visual aesthetics, prompt following and typography generation, c.f. Figure 7. For evaluating human preference in these categories, raters were shown pairwise outputs from two models, and asked to answer the following questions: Prompt following: Which image looks more representative to the text shown above and faithfully follows it? Visual aesthetics: Given the prompt, which image is of higher-quality and aesthetically more pleasing? Typography: Which image more accurately shows/displays the text specified in the above description? More accurate spelling is preferred! Ignore other aspects.

我们的 $d=38$ 模型在 Parti-prompts 基准 (Yu et al., 2022) 的人类偏好评估中，于视觉美学、遵循提示与排版生成等类别超过当前专有 (Betker et al., 2023; ide, 2024) 与开源 (Sauer et al., 2023; pla, 2024; Chen et al., 2023; Pernias et al., 2023) 最优图像生成模型，见图 7。评估时向评分者展示两模型成对输出，并询问：遵循提示：哪张图对上方文本更具代表性且忠实遵循？视觉美学：给定提示，哪张图质量更高、更美观？排版：哪张图更准确显示上述描述中指定的文字？拼写更准确者优先！忽略其他方面。

Lastly, Table 6 highlights an intriguing result: not only do bigger models perform better, they also require fewer steps to reach their peak performance.

最后，表 6 给出一个有意思的结果：更大的模型不仅表现更好，达到峰值表现所需的步数也更少。

Model	Overall	Objects	Counting	Colors	Position	Color
Single	Two	Attribution
minDALL-E	0.23	0.73	0.11	0.12	0.37	0.02	0.01
SD v1.5	0.43	0.97	0.38	0.35	0.76	0.04	0.06
PixArt-alpha	0.48	0.98	0.50	0.44	0.80	0.08	0.07
SD v2.1	0.50	0.98	0.51	0.44	0.85	0.07	0.17
DALL-E 2	0.52	0.94	0.66	0.49	0.77	0.10	0.19
SDXL	0.55	0.98	0.74	0.39	0.85	0.15	0.23
SDXL Turbo	0.55	1.00	0.72	0.49	0.80	0.10	0.18
IF-XL	0.61	0.97	0.74	0.66	0.81	0.13	0.35
DALL-E 3	0.67	0.96	0.87	0.47	0.83	0.43	0.45
Ours (depth=18), $512^{2}$	0.58	0.97	0.72	0.52	0.78	0.16	0.34
Ours (depth=24), $512^{2}$	0.62	0.98	0.74	0.63	0.67	0.34	0.36
Ours (depth=30), $512^{2}$	0.64	0.96	0.80	0.65	0.73	0.33	0.37
Ours (depth=38), $512^{2}$	0.68	0.98	0.84	0.66	0.74	0.40	0.43
Ours (depth=38), $512^{2}$ w/DPO	0.71	0.98	0.89	0.73	0.83	0.34	0.47
Ours (depth=38), $1024^{2}$ w/DPO	0.74	0.99	0.94	0.72	0.89	0.33	0.60

Table 5: GenEval comparisons. Our largest model (depth=38) outperforms all current open models and DALLE-3 (Betker et al., 2023) on GenEval (Ghosh et al., 2023). We highlight the best, second best, and third best entries. For DPO, see Appendix C.