AGI Daily
资讯日报词条论文人物新创
返回论文列表
摘要1. 引言2. 相关工作3. 预备知识3.1. Rectified Flow 模型3.2. 用 ODE inversion 做图像编辑4. 对 inversion 编辑的再解释5. FlowEdit5.1. 实践中的考虑5.2. 与基于优化的方法的关系5.3. 与(精确)inversion 编辑的比较6. 实验7. 结论与局限致谢参考文献
返回论文列表
目录

FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models

收录于 ICCV 2025 Best Student Paper;CVF 论文集第 19721–19730 页;DOI 10.1109/iccv51701.2025.01834;arXiv:2412.08629被引 288 次
打开论文来源

FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models

FlowEdit:用预训练流模型做无需 inversion 的文本图像编辑

Vladimir Kulikov  Matan Kleiner  Inbar Huberman-Spiegelglas  Tomer Michaeli
Technion – Israel Institute of Technology

FlowEdit teaser: pairs of real and edited images under prompt changes such as this to home, cat to raccoon, and steak to bread

Figure 1. FlowEdit. We present an inversion-free, optimization-free and model agnostic method for text-based image editing using pre-trained flow models. As opposed to the editing-by-inversion paradigm, FlowEdit constructs an ODE that directly maps the source image distribution to the target image distribution (corresponding to the source and target prompts). This ODE achieves a lower transport cost and thus leads to better structure preservation, achieving state of the art results on complex editing tasks. From left to right, top to bottom, the first five images were obtained with FLUX and the rest with Stable Diffusion 3. Text indicates changes in the prompts.

图 1. FlowEdit。我们提出一种无需 inversion、无需优化、且不依赖特定模型的方法,用预训练流模型做基于文本的图像编辑。与「先 inversion 再编辑」的范式不同,FlowEdit 构造一条常微分方程(ODE),把源图像分布直接映射到目标图像分布(分别对应源提示和目标提示)。这条 ODE 的传输代价更低,因而能更好地保持结构,并在复杂编辑任务上达到目前最佳水平。按从左到右、从上到下的顺序,前五张结果由 FLUX 得到,其余由 Stable Diffusion 3 得到。图中文字标出了提示里被改动的部分。

Abstract

摘要

Editing real images using a pre-trained text-to-image (T2I) diffusion/flow model often involves inverting the image into its corresponding noise map. However, inversion by itself is typically insufficient for obtaining satisfactory results, and therefore many methods additionally intervene in the sampling process. Such methods achieve improved results but are not seamlessly transferable between model architectures. Here, we introduce FlowEdit, a text-based editing method for pre-trained T2I flow models, which is inversion-free, optimization-free and model agnostic. Our method constructs an ODE that directly maps between the source and target distributions (corresponding to the source and target text prompts) and achieves a lower transport cost than the inversion approach. This leads to state-of-the-art results, as we illustrate with Stable Diffusion 3 and FLUX. Code and examples are available on the project's webpage.

用预训练的文生图(T2I)扩散/流模型编辑真实图像时,通常要先把图像 inversion 成对应的噪声图。可是单靠 inversion 往往得不到满意结果,许多方法因此还会再干预采样过程。这些方法效果更好,但不能在不同模型架构之间顺畅迁移。本文提出 FlowEdit,这是一种面向预训练 T2I 流模型的文本编辑方法,无需 inversion,无需优化,也不依赖特定模型。我们构造一条 ODE,在源分布和目标分布(分别对应源文本提示和目标文本提示)之间直接映射,传输代价低于 inversion 路线。这带来了目前最佳的结果,我们在 Stable Diffusion 3 和 FLUX 上作了展示。代码和示例见项目主页(https://matankleiner.github.io/flowedit/)。

1. Introduction

1. 引言

Diffusion and flow models generate samples through an iterative process, which starts from pure noise and employs denoising-like steps. Beyond data generation, many methods employ pre-trained diffusion/flow models for editing real signals [5, 7, 12, 14–16, 19, 20, 23, 29, 30, 37, 44, 48]. While some techniques rely on test-time optimization [15, 20], many methods aim for optimization-free approaches. A common first step in the latter category of methods is inversion, i.e. extracting the initial noise vector that presumably generates the signal one wishes to edit. This noise vector is then used to generate the edited signal by injecting a different condition to the model, such as a modified text prompt [19, 32, 45, 47].

扩散模型和流模型通过迭代过程生成样本:从纯噪声出发,再反复做类似去噪的步骤。除了生成数据,许多方法还用预训练的扩散/流模型来编辑真实信号 [5, 7, 12, 14–16, 19, 20, 23, 29, 30, 37, 44, 48]。有些技术依赖测试时优化 [15, 20],也有不少方法追求无需优化。后一类里,常见的第一步就是 inversion,也就是抽出那份据信能生成待编辑信号的初始噪声向量。然后把这份噪声送回模型,并换上不同的条件,例如修改后的文本提示,从而生成编辑结果 [19, 32, 45, 47]。

In the context of image editing, many works noted that this editing-by-inversion paradigm often leads to insufficient fidelity to the source image (see e.g. [19]). To overcome this, some methods attempt to reduce the errors incurred in the inversion process [11, 32]. However, this rarely eliminates the problem as even when editing generated images, for which the initial noise is precisely known, editing-by-inversion commonly fails to preserve structure. Other methods intervene in the sampling process by injecting internal model representations (e.g. attention maps) recorded during the inversion process [6, 16, 44]. These methods achieve improved fidelity but are not easily transferable across model architectures and sampling schemes.

在图像编辑里,许多工作都注意到,这种「先 inversion 再编辑」的范式常常对源图像保真不足(例如见 [19])。为了缓解这一点,有些方法试图减小 inversion 过程中的误差 [11, 32]。但这很少能把问题彻底去掉:即便去编辑生成图像,初始噪声已知且精确,这条路线也常常保不住结构。另一些方法会干预采样过程,把 inversion 阶段记录下来的模型内部表征(例如注意力图)再注入进去 [6, 16, 44]。这些方法能提高保真度,但不容易迁移到新的模型架构和采样方案上。

In this work, we present FlowEdit, a text-based editing method for pre-trained text-to-image (T2I) flow models, which breaks away from the editing-by-inversion paradigm. FlowEdit is optimization-free and model agnostic, making it easily transferable between models. Rather than mapping the image to noise and the noise back to an edited image, it constructs a direct path between the source and target distributions (corresponding to the source and target text prompts). This path is shorter than that achieved by inversion, and thus maintains better fidelity to the source image. We evaluate FlowEdit on diverse edits using FLUX [24] and Stable Diffusion 3 (SD3) [10], and show that it achieves state-of-the-art results across the board (see Fig. 1).

本文提出 FlowEdit,这是一种面向预训练文生图(T2I)流模型的文本编辑方法,不再走「先 inversion 再编辑」的老路。FlowEdit 无需优化,也不依赖特定模型,因而容易在不同模型之间迁移。它不是先把图像映射到噪声、再从噪声映射回编辑图像,而是在源分布和目标分布(分别对应源文本提示和目标文本提示)之间构造一条直接路径。这条路径比 inversion 更短,因此对源图像的保真更好。我们在 FLUX [24] 和 Stable Diffusion 3(SD3)[10] 上评估了多样的编辑,并表明 FlowEdit 在各项设置中都达到了目前最佳水平(见图 1)。

Our method is motivated by a novel reinterpretation of editing-by-inversion as a direct path between the source and target distributions. Armed with this observation, we derive an alternative direct path that has a lower transport cost, as we show on synthetic examples.

我们的方法,建立在对「inversion 编辑」的一种新解读上:它可以看成源分布和目标分布之间的一条直接路径。有了这一观察,我们再导出另一条直接路径,其传输代价更低,并在合成例子上作了展示。

2. Related work

2. 相关工作

To edit a real image using a pre-trained diffusion model, some methods use optimization over the image itself [15, 21, 22, 33]. These methods utilize the generative prior of a T2I diffusion model as a loss, which they optimize to push the image to comply with the user-provided prompt. Recently, Yang et al. [49] suggested a similar method for flow models. These methods are resource intensive at test-time.

要用预训练扩散模型编辑真实图像,有些方法直接在图像上做优化 [15, 21, 22, 33]。它们把 T2I 扩散模型的生成先验当成损失,再优化图像,使其符合用户给出的提示。最近,Yang 等人 [49] 为流模型提出了类似方法。这类方法在测试时消耗资源较多。

Many zero-shot editing methods do not require optimization. The first step of most of these methods is image-to-noise inversion [5, 6, 8, 16, 19, 40, 43–45, 47]. However, the noise map obtained through naive inversion [41, 47] is generally unsuitable for effective editing [16, 19, 32]. This is often attributed to the inaccurate inversion process, leading to attempts to improve the inversion accuracy [13, 31, 32, 45]. Yet, even exact inversion on synthetic data leads to unsatisfactory editing [19]. To address this, many methods extract during the inversion stage structural information implicitly encoded within the model architecture. They then inject it in the sampling process to achieve better structure preservation [5, 6, 16, 34, 44]. These methods are typically tailored for specific model architectures and sampling methods, limiting their transferability to new settings, such as to flow models [10, 24]. Several recent methods were proposed for editing with flow models [3, 39, 46]. However, these methods rely on inversion and in some cases also on feature injection, and thus suffer from the same limitations as their diffusion-model counterparts.

许多零样本编辑方法并不需要优化。其中大多数的第一步,仍是把图像 inversion 到噪声 [5, 6, 8, 16, 19, 40, 43–45, 47]。可是,朴素 inversion [41, 47] 得到的噪声图,通常并不适合做有效编辑 [16, 19, 32]。这常被归咎于 inversion 过程不够准确,于是有人试图提高 inversion 精度 [13, 31, 32, 45]。然而,即便在合成数据上做精确 inversion,编辑结果往往仍不理想 [19]。为应对这一点,许多方法会在 inversion 阶段抽出隐含在模型架构里的结构信息,再在采样过程中注入,以更好地保持结构 [5, 6, 16, 34, 44]。这些方法通常针对特定的模型架构和采样方案来设计,迁移到新设置(例如流模型 [10, 24])时会受到限制。最近也有若干方法专门做流模型上的编辑 [3, 39, 46]。但它们仍然依赖 inversion,有的还要做特征注入,因而和对应的扩散模型方法有着同样的局限。

Unlike previous works, our method does not rely on inversion. It maps from source to target distributions without traversing through the Gaussian distribution. It also avoids optimization and does not intervene in the model internals, making it easily adaptable to new models.

与以往工作不同,我们的方法不依赖 inversion。它从源分布映射到目标分布时,并不经过高斯分布。它也不做优化,不干预模型内部,因而容易适配到新模型上。

3. Preliminaries

3. 预备知识

3.1. Rectified Flow models

3.1. Rectified Flow 模型

Generative flow models attempt to construct a transportation between the distributions of two random vectors, X0X_0X0​ and X1X_1X1​, defined by an ordinary differential equation (ODE) over time t∈[0,1]t\in[0,1]t∈[0,1],

dZt=V(Zt,t) dt.(1)dZ_{t}=V(Z_{t},t)\,dt. \tag{1}dZt​=V(Zt​,t)dt.(1)

Here, VVV is a time-dependent velocity field, usually parameterized by a neural network, which has the property that if the boundary condition at t=1t=1t=1 is Z1=X1Z_1=X_1Z1​=X1​, then Z0Z_0Z0​ is distributed like X0X_0X0​. It is common to choose X1∼N(0,I)X_1\sim\mathcal{N}(0,I)X1​∼N(0,I), which allows to easily draw samples from the distribution of X0X_0X0​. This is done by initializing the ODE at t=1t=1t=1 with a sample of a standard Gaussian vector, and numerically solving the ODE backwards in time to obtain a sample Z0Z_0Z0​ from the distribution of X0X_0X0​ at time t=0t=0t=0.

生成式流模型试图在两个随机向量 X0X_0X0​ 和 X1X_1X1​ 的分布之间构造一种传输,它由时间 t∈[0,1]t\in[0,1]t∈[0,1] 上的常微分方程(ODE)给出:

dZt=V(Zt,t) dt.(1)dZ_{t}=V(Z_{t},t)\,dt. \tag{1}dZt​=V(Zt​,t)dt.(1)

其中 VVV 是随时间变化的速度场,通常由神经网络参数化。它有这样的性质:如果 t=1t=1t=1 时的边界条件是 Z1=X1Z_1=X_1Z1​=X1​,那么 Z0Z_0Z0​ 的分布就与 X0X_0X0​ 相同。实践中常取 X1∼N(0,I)X_1\sim\mathcal{N}(0,I)X1​∼N(0,I),以便容易从 X0X_0X0​ 的分布中抽样。具体做法是:在 t=1t=1t=1 用一个标准高斯向量的样本初始化这条 ODE,再沿时间反向数值求解,得到 t=0t=0t=0 时来自 X0X_0X0​ 分布的样本 Z0Z_0Z0​。

Rectified flows [2, 26, 28] are a particular type of flow models, which are trained such that the marginal at time ttt corresponds to a linear interpolation between X0X_0X0​ and X1X_1X1​,

Zt∼(1−t)X0+tX1.(2)Z_{t}\sim(1-t)X_{0}+tX_{1}. \tag{2}Zt​∼(1−t)X0​+tX1​.(2)

Rectified flows have the desired property that their sampling paths are relatively straight. This allows using a small number of discretization steps when solving the ODE.

Rectified flow [2, 26, 28] 是一类特殊的流模型,训练时让时刻 ttt 的边际分布对应于 X0X_0X0​ 与 X1X_1X1​ 之间的线性插值:

Zt∼(1−t)X0+tX1.(2)Z_{t}\sim(1-t)X_{0}+tX_{1}. \tag{2}Zt​∼(1−t)X0​+tX1​.(2)

Rectified flow 有一个理想性质:采样路径相对较直。因此在求解 ODE 时,可以用较少的离散步数。

Text-to-image flow models employ a velocity field V(Xt,t,C)V(X_t,t,C)V(Xt​,t,C) that depends on a text prompt CCC. Such models are trained on pairs of text and image data, (C,X0)(C,X_0)(C,X0​), so as to allow sampling from the conditional distribution of X0∣CX_0|CX0​∣C.

文生图流模型使用依赖于文本提示 CCC 的速度场 V(Xt,t,C)V(X_t,t,C)V(Xt​,t,C)。这类模型在文本–图像数据对 (C,X0)(C,X_0)(C,X0​) 上训练,从而可以从条件分布 X0∣CX_0|CX0​∣C 中抽样。

3.2. Image editing using ODE inversion

3.2. 用 ODE inversion 做图像编辑

Suppose we are given a real image, XsrcX^{\text{src}}Xsrc, which we want to edit by providing a text prompt csrcc_{\text{src}}csrc​ describing the image and a text prompt ctarc_{\text{tar}}ctar​ describing the desired edit. A common approach to do so is by using a pre-trained text-conditioned diffusion/flow model and employing inversion. Specifically, let us denote the text conditioned velocities as Vsrc(Xt,t)≜V(Xt,t,csrc)V^{\text{src}}(X_t,t)\triangleq V(X_t,t,c_{\text{src}})Vsrc(Xt​,t)≜V(Xt​,t,csrc​) and Vtar(Xt,t)≜V(Xt,t,ctar)V^{\text{tar}}(X_t,t)\triangleq V(X_t,t,c_{\text{tar}})Vtar(Xt​,t)≜V(Xt​,t,ctar​). Methods relying on inversion start by extracting the initial noise map corresponding to the source image. This is done by traversing the forward process defined by the ODE

dZtsrc=Vsrc(Ztsrc,t) dt,(3)dZ^{\text{src}}_{t}=V^{\text{src}}(Z^{\text{src}}_{t},t)\,dt, \tag{3}dZtsrc​=Vsrc(Ztsrc​,t)dt,(3)

starting at t=0t=0t=0 from the source image, Z0src=XsrcZ^{\text{src}}_{0}=X^{\text{src}}Z0src​=Xsrc, and reaching a noise map Z1srcZ^{\text{src}}_{1}Z1src​ at t=1t=1t=1. The process ZtsrcZ^{\text{src}}_{t}Ztsrc​ is the path shown on the left of Fig. 2a. Then, the reverse ODE,

dZttar=Vtar(Zttar,t) dt,(4)dZ^{\text{tar}}_{t}=V^{\text{tar}}(Z^{\text{tar}}_{t},t)\,dt, \tag{4}dZttar​=Vtar(Zttar​,t)dt,(4)

is solved backward in time, starting from t=1t=1t=1 with the extracted noise, Z1tar=Z1srcZ^{\text{tar}}_{1}=Z^{\text{src}}_{1}Z1tar​=Z1src​, and reaching an edited image Z0tarZ^{\text{tar}}_{0}Z0tar​ at t=0t=0t=0. This is the path shown on the right of Fig. 2a.

假设给定一张真实图像 XsrcX^{\text{src}}Xsrc,我们希望用一句描述该图像的文本提示 csrcc_{\text{src}}csrc​,以及一句描述目标编辑的文本提示 ctarc_{\text{tar}}ctar​ 来改它。常见做法是使用预训练的文本条件扩散/流模型,并借助 inversion。具体地,把文本条件下的速度记为 Vsrc(Xt,t)≜V(Xt,t,csrc)V^{\text{src}}(X_t,t)\triangleq V(X_t,t,c_{\text{src}})Vsrc(Xt​,t)≜V(Xt​,t,csrc​) 和 Vtar(Xt,t)≜V(Xt,t,ctar)V^{\text{tar}}(X_t,t)\triangleq V(X_t,t,c_{\text{tar}})Vtar(Xt​,t)≜V(Xt​,t,ctar​)。依赖 inversion 的方法,首先抽出与源图像对应的初始噪声图。这是通过沿如下 ODE 定义的前向过程走一遍完成的:

dZtsrc=Vsrc(Ztsrc,t) dt,(3)dZ^{\text{src}}_{t}=V^{\text{src}}(Z^{\text{src}}_{t},t)\,dt, \tag{3}dZtsrc​=Vsrc(Ztsrc​,t)dt,(3)

在 t=0t=0t=0 从源图像 Z0src=XsrcZ^{\text{src}}_{0}=X^{\text{src}}Z0src​=Xsrc 出发,到 t=1t=1t=1 得到噪声图 Z1srcZ^{\text{src}}_{1}Z1src​。过程 ZtsrcZ^{\text{src}}_{t}Ztsrc​ 就是图 2a 左侧的那条路径。然后,反向 ODE

dZttar=Vtar(Zttar,t) dt,(4)dZ^{\text{tar}}_{t}=V^{\text{tar}}(Z^{\text{tar}}_{t},t)\,dt, \tag{4}dZttar​=Vtar(Zttar​,t)dt,(4)

沿时间反向求解:在 t=1t=1t=1 从抽出的噪声 Z1tar=Z1srcZ^{\text{tar}}_{1}=Z^{\text{src}}_{1}Z1tar​=Z1src​ 出发,到 t=0t=0t=0 得到编辑图像 Z0tarZ^{\text{tar}}_{0}Z0tar​。这就是图 2a 右侧的路径。

Three-panel diagram comparing editing by ODE inversion, a reinterpretation of inversion, and FlowEdit on a tiger-to-cat example

Figure 2. Editing by inversion vs. FlowEdit. (a) In inversion based editing, the source image Z0srcZ^{\text{src}}_{0}Z0src​ is first mapped to the noise space by solving the forward ODE conditioned on the source prompt (left path). Then, the extracted noise is used to solve the reverse ODE conditioned on the target prompt to obtain Z0tarZ^{\text{tar}}_{0}Z0tar​ (right path). The images at the bottom visualize this transition. (b) We reinterpret inversion as a direct path between the source and target distributions (bottom path). This is done by using the velocities calculated during the inversion and sampling (green and red arrows) to calculate an editing direction (orange arrow) that drives the evolution of the direct path ZtinvZ^{\text{inv}}_{t}Ztinv​ through an ODE. The resulting path is noise-free, as demonstrated by the images at the bottom. (c) FlowEdit traverses a shorter direct path, ZtFEZ^{\text{FE}}_{t}ZtFE​, without relying on inversion. At each timestep, we directly add random noise to Z^0src\hat{Z}^{\text{src}}_{0}Z^0src​ to obtain Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ and use that direction to create Z^ttar\hat{Z}^{\text{tar}}_{t}Z^ttar​ from ZtFEZ^{\text{FE}}_{t}ZtFE​ (gray parallelogram). We then calculate the corresponding velocities and average over multiple realizations (not shown in the figure) to obtain the next ODE step (orange arrow). The images at the bottom demonstrate our noise-free path.

图 2. inversion 编辑与 FlowEdit 的对比。(a) 在基于 inversion 的编辑中,源图像 Z0srcZ^{\text{src}}_{0}Z0src​ 先通过在源提示条件下求解前向 ODE,被映射到噪声空间(左路)。然后用抽出的噪声,在目标提示条件下求解反向 ODE,得到 Z0tarZ^{\text{tar}}_{0}Z0tar​(右路)。底部的图像把这一过渡可视化出来。(b) 我们把 inversion 重新解释为源分布与目标分布之间的一条直接路径(底部路径)。具体做法是:利用 inversion 和采样过程中算出的速度(绿色和红色箭头),得到一个编辑方向(橙色箭头),再通过一条 ODE 驱动直接路径 ZtinvZ^{\text{inv}}_{t}Ztinv​ 演化。得到的路径是无噪声的,底部图像说明了这一点。(c) FlowEdit 走一条更短的直接路径 ZtFEZ^{\text{FE}}_{t}ZtFE​,并不依赖 inversion。在每个时间步,我们直接给 Z^0src\hat{Z}^{\text{src}}_{0}Z^0src​ 加上随机噪声得到 Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​,再用这个方向从 ZtFEZ^{\text{FE}}_{t}ZtFE​ 构造 Z^ttar\hat{Z}^{\text{tar}}_{t}Z^ttar​(灰色平行四边形)。然后计算对应的速度,并对多次实现取平均(图中未画出),得到下一步 ODE 更新(橙色箭头)。底部图像展示了我们这条无噪声路径。

Editing by inversion tends to produce unsatisfactory results, and is thus usually accompanied by feature map injection. While the sub-optimal results are commonly blamed on the inaccurate discretization of the ODE, they are also encountered when editing generated images and using their ground-truth initial noise maps [19]. Our method retains the simplicity of editing by inversion and achieves state of the art results without any intervention in the model internals.

inversion 编辑往往效果不理想,因此通常还要配合特征图注入。次优结果常被归咎于 ODE 离散化不够准确,可是在编辑生成图像、并使用其真实初始噪声图时,同样的问题也会出现 [19]。我们的方法保留了 inversion 编辑的简洁性,而且在不干预模型内部的前提下,达到了目前最佳水平。

4. Reinterpretation of editing by inversion

4. 对 inversion 编辑的再解释

Before presenting FlowEdit, we reinterpret the editing-by-inversion approach. This will serve to motivate our method.

在介绍 FlowEdit 之前,我们先重新解释 inversion 编辑。这将成为后文方法的动机。

Inversion based editing transports between the source and target distributions, while passing through the distribution of Gaussian noise. However, this approach can also be expressed as a direct path between the source and target distributions. Namely, we can construct a path ZtinvZ^{\text{inv}}_{t}Ztinv​ that starts at t=1t=1t=1 in the source distribution and reaches the target distribution at t=0t=0t=0. Specifically, given the forward and reverse flow trajectories, ZtsrcZ^{\text{src}}_{t}Ztsrc​ and ZttarZ^{\text{tar}}_{t}Zttar​, we define

Ztinv=Z0src+Zttar−Ztsrc.(5)Z^{\text{inv}}_{t}=Z^{\text{src}}_{0}+Z^{\text{tar}}_{t}-Z^{\text{src}}_{t}. \tag{5}Ztinv​=Z0src​+Zttar​−Ztsrc​.(5)

Note that when going from t=1t=1t=1 to t=0t=0t=0 we transition from the source image, Z1inv=Z0src=XsrcZ^{\text{inv}}_{1}=Z^{\text{src}}_{0}=X^{\text{src}}Z1inv​=Z0src​=Xsrc (because Z1tar=Z1srcZ^{\text{tar}}_{1}=Z^{\text{src}}_{1}Z1tar​=Z1src​), to the edited image, Z0inv=Z0tarZ^{\text{inv}}_{0}=Z^{\text{tar}}_{0}Z0inv​=Z0tar​. This path is depicted at the bottom of Fig. 2b, in which the gray parallelogram visualizes the relation (5).

基于 inversion 的编辑在源分布和目标分布之间传输,途中会经过高斯噪声分布。不过,这一做法也可以写成源分布与目标分布之间的一条直接路径。也就是说,我们可以构造一条路径 ZtinvZ^{\text{inv}}_{t}Ztinv​,它在 t=1t=1t=1 时从源分布出发,在 t=0t=0t=0 时到达目标分布。具体地,给定前向和反向流轨迹 ZtsrcZ^{\text{src}}_{t}Ztsrc​ 与 ZttarZ^{\text{tar}}_{t}Zttar​,我们定义

Ztinv=Z0src+Zttar−Ztsrc.(5)Z^{\text{inv}}_{t}=Z^{\text{src}}_{0}+Z^{\text{tar}}_{t}-Z^{\text{src}}_{t}. \tag{5}Ztinv​=Z0src​+Zttar​−Ztsrc​.(5)

注意,当时间从 t=1t=1t=1 走到 t=0t=0t=0 时,我们从源图像 Z1inv=Z0src=XsrcZ^{\text{inv}}_{1}=Z^{\text{src}}_{0}=X^{\text{src}}Z1inv​=Z0src​=Xsrc(因为 Z1tar=Z1srcZ^{\text{tar}}_{1}=Z^{\text{src}}_{1}Z1tar​=Z1src​)过渡到编辑图像 Z0inv=Z0tarZ^{\text{inv}}_{0}=Z^{\text{tar}}_{0}Z0inv​=Z0tar​。这条路径画在图 2b 底部,其中灰色平行四边形可视化了关系 (5)。

Let us express (5) as an ODE. Differentiating both sides of (5) and substituting (3) and (4), this relation becomes

dZtinv=VtΔ(Ztsrc,Zttar) dt,(6)dZ^{\text{inv}}_{t}=V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{tar}}_{t})\,dt, \tag{6}dZtinv​=VtΔ​(Ztsrc​,Zttar​)dt,(6)

where VtΔ(Ztsrc,Zttar)=Vtar(Zttar,t)−Vsrc(Ztsrc,t)V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{tar}}_{t})=V^{\text{tar}}(Z^{\text{tar}}_{t},t)-V^{\text{src}}(Z^{\text{src}}_{t},t)VtΔ​(Ztsrc​,Zttar​)=Vtar(Zttar​,t)−Vsrc(Ztsrc​,t). This is visualized by the green, red, and orange arrows in Fig. 2b. Now, isolating ZttarZ^{\text{tar}}_{t}Zttar​ from (5) and substituting it into (6), we get that the direct path is the solution of the ODE

dZtinv=VtΔ(Ztsrc,Ztinv+Ztsrc−Z0src) dt(7)dZ^{\text{inv}}_{t}=V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{inv}}_{t}+Z^{\text{src}}_{t}-Z^{\text{src}}_{0})\,dt \tag{7}dZtinv​=VtΔ​(Ztsrc​,Ztinv​+Ztsrc​−Z0src​)dt(7)

with boundary condition Z1inv=Z0srcZ^{\text{inv}}_{1}=Z^{\text{src}}_{0}Z1inv​=Z0src​ at t=1t=1t=1.

让我们把 (5) 写成一条 ODE。对 (5) 两边求导,再代入 (3) 和 (4),这一关系变成

dZtinv=VtΔ(Ztsrc,Zttar) dt,(6)dZ^{\text{inv}}_{t}=V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{tar}}_{t})\,dt, \tag{6}dZtinv​=VtΔ​(Ztsrc​,Zttar​)dt,(6)

其中 VtΔ(Ztsrc,Zttar)=Vtar(Zttar,t)−Vsrc(Ztsrc,t)V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{tar}}_{t})=V^{\text{tar}}(Z^{\text{tar}}_{t},t)-V^{\text{src}}(Z^{\text{src}}_{t},t)VtΔ​(Ztsrc​,Zttar​)=Vtar(Zttar​,t)−Vsrc(Ztsrc​,t)。图 2b 里的绿色、红色和橙色箭头把这一点画了出来。现在从 (5) 解出 ZttarZ^{\text{tar}}_{t}Zttar​ 并代入 (6),可知这条直接路径是如下 ODE 的解:

dZtinv=VtΔ(Ztsrc,Ztinv+Ztsrc−Z0src) dt(7)dZ^{\text{inv}}_{t}=V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{inv}}_{t}+Z^{\text{src}}_{t}-Z^{\text{src}}_{0})\,dt \tag{7}dZtinv​=VtΔ​(Ztsrc​,Ztinv​+Ztsrc​−Z0src​)dt(7)

边界条件为:t=1t=1t=1 时 Z1inv=Z0srcZ^{\text{inv}}_{1}=Z^{\text{src}}_{0}Z1inv​=Z0src​。

There exist, of course, many paths connecting Z0srcZ^{\text{src}}_{0}Z0src​ and Z0tarZ^{\text{tar}}_{0}Z0tar​. What is so special about the path defined by (7)? It turns out that images along this path are noise-free, as shown at the bottom of Fig. 2b. The reason this happens is that the noisy images ZttarZ^{\text{tar}}_{t}Zttar​ and ZtsrcZ^{\text{src}}_{t}Ztsrc​ contain roughly the same noise constituent. Therefore, the vectors Vtar(Zttar,t)V^{\text{tar}}(Z^{\text{tar}}_{t},t)Vtar(Zttar​,t) and Vsrc(Ztsrc,t)V^{\text{src}}(Z^{\text{src}}_{t},t)Vsrc(Ztsrc​,t), which point at the source and target distributions, respectively, remove roughly the same noise component, so that the difference vector VtΔ(Ztsrc,Zttar)V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{tar}}_{t})VtΔ​(Ztsrc​,Zttar​) (orange arrow in Fig. 2b), encompasses the difference only between the clean image predictions.

当然,连接 Z0srcZ^{\text{src}}_{0}Z0src​ 和 Z0tarZ^{\text{tar}}_{0}Z0tar​ 的路径有很多。由 (7) 定义的这条路径有什么特别之处?事实证明,这条路径上的图像是无噪声的,如图 2b 底部所示。原因在于,带噪图像 ZttarZ^{\text{tar}}_{t}Zttar​ 和 ZtsrcZ^{\text{src}}_{t}Ztsrc​ 大致含有相同的噪声成分。因此,分别指向源分布和目标分布的向量 Vtar(Zttar,t)V^{\text{tar}}(Z^{\text{tar}}_{t},t)Vtar(Zttar​,t) 与 Vsrc(Ztsrc,t)V^{\text{src}}(Z^{\text{src}}_{t},t)Vsrc(Ztsrc​,t),会去掉大致相同的噪声分量,于是差分向量 VtΔ(Ztsrc,Zttar)V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{tar}}_{t})VtΔ​(Ztsrc​,Zttar​)(图 2b 中的橙色箭头)只包含干净图像预测之间的差异。

How do images evolve along this direct path? At the beginning of the path (ttt close to 111), the noise level in ZtsrcZ^{\text{src}}_{t}Ztsrc​ and ZttarZ^{\text{tar}}_{t}Zttar​ is substantial, and therefore the vector VtΔ(Ztsrc,Zttar)V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{tar}}_{t})VtΔ​(Ztsrc​,Zttar​) captures only differences in coarse image structures. As ttt gets smaller, the noise level drops and higher frequency contents are unveiled. In other words, the path (7) constitutes a sort of autoregressive coarse-to-fine evolution from Z0srcZ^{\text{src}}_{0}Z0src​ to Z0tarZ^{\text{tar}}_{0}Z0tar​. This can be seen in the images at the bottom of Fig. 2b, where the first features that get modified are the coarse structures, and the last features to get updated are the fine textures. See App. J for additional illustrations.

沿这条直接路径,图像是如何演化的?在路径开头(ttt 接近 111)时,ZtsrcZ^{\text{src}}_{t}Ztsrc​ 和 ZttarZ^{\text{tar}}_{t}Zttar​ 中的噪声水平很高,因此向量 VtΔ(Ztsrc,Zttar)V^{\Delta}_{t}(Z^{\text{src}}_{t},Z^{\text{tar}}_{t})VtΔ​(Ztsrc​,Zttar​) 只捕捉到图像粗结构上的差异。随着 ttt 变小,噪声水平下降,更高频的内容才显露出来。换句话说,路径 (7) 构成了一种从 Z0srcZ^{\text{src}}_{0}Z0src​ 到 Z0tarZ^{\text{tar}}_{0}Z0tar​ 的自回归式由粗到细演化。这可以在图 2b 底部的图像中看到:最先被修改的是粗结构,最后被更新的才是细纹理。更多示意图见附录 J。

Scatter-plot diagram showing source-to-target pairings: FlowEdit keeps modes separated while inversion mixes them

Figure 3. Source to target pairings for editing-by-inversion and FlowEdit. Samples from the two modes in the source distribution are colored in blue and red (top left). As can be seen in the right panes, FlowEdit results in separated blue and red modes, as opposed to editing-by-inversion, where the modes are intermixed, indicating higher transport cost in terms of MSE. This can be explained by the Gaussian in the bottom left, through which the editing-by-inversion path must traverse before reaching the target. Please see the supplementary video for animated visualizations.

图 3. inversion 编辑与 FlowEdit 的源–目标配对。源分布两个模态中的样本分别标成蓝色和红色(左上)。从右侧面板可以看出,FlowEdit 得到的是分开的蓝色和红色模态;而 inversion 编辑会把两个模态混在一起,这意味着以均方误差衡量的传输代价更高。这一点可以由左下角的高斯分布来解释:inversion 编辑的路径在到达目标之前,必须先经过它。动画可视化请见补充视频。

5. FlowEdit

The fact that editing-by-inversion can be expressed as a direct ODE does not imply that it induces a desirable pairing between source and target samples. Figure 3 shows a simple example, where the source and target distributions are Gaussian mixtures with modes centered at {(−152,−152),(152,152)}\bigl\{\bigl(-\frac{15}{\sqrt{2}},-\frac{15}{\sqrt{2}}\bigr),\bigl(\frac{15}{\sqrt{2}},\frac{15}{\sqrt{2}}\bigr)\bigr\}{(−2​15​,−2​15​),(2​15​,2​15​)} and {(−15,0),(0,15)}\{(-15,0),(0,15)\}{(−15,0),(0,15)}, respectively. A good editing method should map each mode in the source distribution to its closest mode in the target distribution. In the context of image editing, this means that source images are modified as little as possible, while transporting to the target distribution. However, as seen on the bottom right of Fig. 3, this is not what happens in inversion. The reason the induced pairings look the way they look can be understood by inspecting how samples are mapped to their “initial” Gaussian noise component (bottom left).

inversion 编辑可以写成一条直接 ODE,并不意味着它会在源样本和目标样本之间诱导出理想的配对。图 3 给出了一个简单例子:源分布和目标分布都是高斯混合,模态中心分别在 {(−152,−152),(152,152)}\bigl\{\bigl(-\frac{15}{\sqrt{2}},-\frac{15}{\sqrt{2}}\bigr),\bigl(\frac{15}{\sqrt{2}},\frac{15}{\sqrt{2}}\bigr)\bigr\}{(−2​15​,−2​15​),(2​15​,2​15​)} 和 {(−15,0),(0,15)}\{(-15,0),(0,15)\}{(−15,0),(0,15)}。一种好的编辑方法,应当把源分布中的每个模态映射到目标分布中最近的模态。在图像编辑里,这意味着在把图像送到目标分布的同时,对源图像的改动尽可能小。然而,如图 3 右下所示,inversion 并不是这样做的。之所以会出现这样的配对,可以通过观察样本如何被映射到它们的「初始」高斯噪声分量来理解(左下)。

Our goal is to construct an alternative mapping, which leads to lower distances between the source and target samples. Our method, which we coin FlowEdit, is illustrated on the top right of Fig. 3. As can be seen, it maps each mode in the source distribution to its nearest mode in the target distribution. Compared to editing-by-inversion, FlowEdit achieves less than twice as low a transportation cost, measured by the average squared distance between source samples and their paired target samples. We emphasize that our method is based on a heuristic, which is not guaranteed to precisely map to the target distribution. In particular, it does not necessarily coincide with the optimal transport mapping. Yet, as we will see in the context of images, it achieves state-of-the-art results in terms of structure preservation (i.e. small transportation cost) while adhering to the target text prompt (generating samples in the target distribution).

我们的目标是构造另一种映射,使源样本和目标样本之间的距离更小。我们把这种方法称为 FlowEdit,图 3 右上给出了示意。可以看到,它把源分布中的每个模态映射到目标分布中最近的模态。与 inversion 编辑相比,以源样本与其配对目标样本之间的平均平方距离来衡量,FlowEdit 的传输代价不到其一半。需要强调的是,我们的方法基于启发式,并不能保证精确映射到目标分布。尤其是,它不一定与最优传输映射重合。不过,正如我们在图像情境中将看到的,它在保持结构(也就是传输代价小)的同时,又能遵循目标文本提示(在目标分布中生成样本),从而达到目前最佳水平。

How can we depart from the pairings dictated by inversion? Our key idea is to use multiple random pairings, and average their associated velocity fields. More concretely, we propose to use the same ODE as (7), but to replace the inversion path ZtinvZ^{\text{inv}}_{t}Ztinv​ by a different random process with the same marginals. Then, at each timestep, we average the directions corresponding to different realizations of that process. Specifically, consider the alternative forward process

Z^tsrc=(1−t)Z0src+tNt,(8)\hat{Z}^{\text{src}}_{t}=(1-t)Z^{\text{src}}_{0}+tN_{t}, \tag{8}Z^tsrc​=(1−t)Z0src​+tNt​,(8)

where NtN_tNt​ is a Gaussian process that is statistically independent of Z0srcZ^{\text{src}}_{0}Z0src​ and has marginals Nt∼N(0,1)N_t\sim\mathcal{N}(0,1)Nt​∼N(0,1) for every t∈[0,1]t\in[0,1]t∈[0,1]. We construct a path ZtFEZ^{\text{FE}}_{t}ZtFE​ by solving the ODE

dZtFE=E[VtΔ(Z^tsrc, ZtFE+Z^tsrc−Z0src)∣Z0src] dt(9)dZ^{\text{FE}}_{t}=\mathbb{E}\Bigl[V^{\Delta}_{t}\bigl(\hat{Z}^{\text{src}}_{t},\,Z^{\text{FE}}_{t}+\hat{Z}^{\text{src}}_{t}-Z^{\text{src}}_{0}\bigr)\Bigm|Z^{\text{src}}_{0}\Bigr]\,dt \tag{9}dZtFE​=E[VtΔ​(Z^tsrc​,ZtFE​+Z^tsrc​−Z0src​)​Z0src​]dt(9)

with boundary condition Z1FE=Z0srcZ^{\text{FE}}_{1}=Z^{\text{src}}_{0}Z1FE​=Z0src​ at t=1t=1t=1. Note that the expectation here is w.r.t. Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ (equivalently NtN_tNt​). Also note that we need not specify the covariance function of NtN_tNt​ as (9) depends only on the marginals of the process Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ and not on its entire distribution law.

怎样才能摆脱 inversion 所决定的配对?我们的关键想法是:使用多种随机配对,并对它们对应的速度场取平均。更具体地说,我们仍使用与 (7) 相同的 ODE,但把 inversion 路径 ZtinvZ^{\text{inv}}_{t}Ztinv​ 换成另一个具有相同边际的随机过程。然后在每个时间步,对这一过程不同实现所对应的方向取平均。具体考虑如下替代前向过程

Z^tsrc=(1−t)Z0src+tNt,(8)\hat{Z}^{\text{src}}_{t}=(1-t)Z^{\text{src}}_{0}+tN_{t}, \tag{8}Z^tsrc​=(1−t)Z0src​+tNt​,(8)

其中 NtN_tNt​ 是一个高斯过程,它与 Z0srcZ^{\text{src}}_{0}Z0src​ 统计独立,并且对每个 t∈[0,1]t\in[0,1]t∈[0,1] 都有边际 Nt∼N(0,1)N_t\sim\mathcal{N}(0,1)Nt​∼N(0,1)。我们通过求解 ODE

dZtFE=E[VtΔ(Z^tsrc, ZtFE+Z^tsrc−Z0src)∣Z0src] dt(9)dZ^{\text{FE}}_{t}=\mathbb{E}\Bigl[V^{\Delta}_{t}\bigl(\hat{Z}^{\text{src}}_{t},\,Z^{\text{FE}}_{t}+\hat{Z}^{\text{src}}_{t}-Z^{\text{src}}_{0}\bigr)\Bigm|Z^{\text{src}}_{0}\Bigr]\,dt \tag{9}dZtFE​=E[VtΔ​(Z^tsrc​,ZtFE​+Z^tsrc​−Z0src​)​Z0src​]dt(9)

来构造路径 ZtFEZ^{\text{FE}}_{t}ZtFE​,边界条件为 t=1t=1t=1 时 Z1FE=Z0srcZ^{\text{FE}}_{1}=Z^{\text{src}}_{0}Z1FE​=Z0src​。注意,这里的期望是关于 Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​(等价地,关于 NtN_tNt​)来取的。还要注意,我们不必指定 NtN_tNt​ 的协方差函数,因为 (9) 只依赖于过程 Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ 的边际,而不依赖于它的整个分布律。

A schematic illustration is shown in Fig. 2c. The point Z^0src\hat{Z}^{\text{src}}_{0}Z^0src​ on the left corresponds to a single draw of NtN_tNt​. The distribution of Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ at time ttt is shown in cyan. For each such draw, we calculate Z^ttar=ZtFE+Z^tsrc−Z0src\hat{Z}^{\text{tar}}_{t}=Z^{\text{FE}}_{t}+\hat{Z}^{\text{src}}_{t}-Z^{\text{src}}_{0}Z^ttar​=ZtFE​+Z^tsrc​−Z0src​, as in the editing-by-inversion ODE. This is indicated by the gray parallelogram. Now, given Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ and Z^ttar\hat{Z}^{\text{tar}}_{t}Z^ttar​, we compute the velocity field VtΔ(Z^tsrc,Z^ttar)=Vtar(Z^ttar,t)−Vsrc(Z^tsrc,t)V^{\Delta}_{t}(\hat{Z}^{\text{src}}_{t},\hat{Z}^{\text{tar}}_{t})=V^{\text{tar}}(\hat{Z}^{\text{tar}}_{t},t)-V^{\text{src}}(\hat{Z}^{\text{src}}_{t},t)VtΔ​(Z^tsrc​,Z^ttar​)=Vtar(Z^ttar​,t)−Vsrc(Z^tsrc​,t) to obtain an update direction. This is illustrated by the green, red, and orange arrows. We repeat this multiple times and average the resulting orange arrows (not shown in the figure) to obtain the final update direction.

图 2c 给出了示意。左侧的点 Z^0src\hat{Z}^{\text{src}}_{0}Z^0src​ 对应于 NtN_tNt​ 的一次抽样。时刻 ttt 时 Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ 的分布用青色表示。对每一次这样的抽样,我们像 inversion 编辑的 ODE 那样计算 Z^ttar=ZtFE+Z^tsrc−Z0src\hat{Z}^{\text{tar}}_{t}=Z^{\text{FE}}_{t}+\hat{Z}^{\text{src}}_{t}-Z^{\text{src}}_{0}Z^ttar​=ZtFE​+Z^tsrc​−Z0src​。灰色平行四边形标出了这一步。然后,给定 Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ 和 Z^ttar\hat{Z}^{\text{tar}}_{t}Z^ttar​,我们计算速度场 VtΔ(Z^tsrc,Z^ttar)=Vtar(Z^ttar,t)−Vsrc(Z^tsrc,t)V^{\Delta}_{t}(\hat{Z}^{\text{src}}_{t},\hat{Z}^{\text{tar}}_{t})=V^{\text{tar}}(\hat{Z}^{\text{tar}}_{t},t)-V^{\text{src}}(\hat{Z}^{\text{src}}_{t},t)VtΔ​(Z^tsrc​,Z^ttar​)=Vtar(Z^ttar​,t)−Vsrc(Z^tsrc​,t),得到更新方向。绿色、红色和橙色箭头说明了这一点。我们把这一过程重复多次,并对得到的橙色箭头取平均(图中未画出),作为最终的更新方向。

Algorithm 1 Simplified algorithm for FlowEdit

Input: real image XsrcX^{\text{src}}Xsrc, {ti}i=0T\{t_i\}_{i=0}^{T}{ti​}i=0T​, nmax⁡n_{\max}nmax​, navgn_{\text{avg}}navg​

Output: edited image XtarX^{\text{tar}}Xtar

Init: Ztmax⁡FE=XsrcZ^{\text{FE}}_{t_{\max}}=X^{\text{src}}Ztmax​FE​=Xsrc

for i=nmax⁡i=n_{\max}i=nmax​ to 111 do

  • Nti∼N(0,1)N_{t_i}\sim\mathcal{N}(0,1)Nti​​∼N(0,1)
  • Ztisrc←(1−ti)Xsrc+tiNtiZ^{\text{src}}_{t_i}\leftarrow(1-t_i)X^{\text{src}}+t_i N_{t_i}Zti​src​←(1−ti​)Xsrc+ti​Nti​​
  • Ztitar←ZtiFE+Ztisrc−XsrcZ^{\text{tar}}_{t_i}\leftarrow Z^{\text{FE}}_{t_i}+Z^{\text{src}}_{t_i}-X^{\text{src}}Zti​tar​←Zti​FE​+Zti​src​−Xsrc
  • VtiΔ←Vtar(Ztitar,ti)−Vsrc(Ztisrc,ti)V^{\Delta}_{t_i}\leftarrow V^{\text{tar}}(Z^{\text{tar}}_{t_i},t_i)-V^{\text{src}}(Z^{\text{src}}_{t_i},t_i)Vti​Δ​←Vtar(Zti​tar​,ti​)−Vsrc(Zti​src​,ti​)
  • Zti−1FE←ZtiFE+(ti−1−ti)VtiΔZ^{\text{FE}}_{t_{i-1}}\leftarrow Z^{\text{FE}}_{t_i}+(t_{i-1}-t_i)V^{\Delta}_{t_i}Zti−1​FE​←Zti​FE​+(ti−1​−ti​)Vti​Δ​

end for

Return: Z0FE=X0tarZ^{\text{FE}}_{0}=X^{\text{tar}}_{0}Z0FE​=X0tar​

(The inner assignments may optionally be averaged over navgn_{\text{avg}}navg​ samples.)

算法 1 FlowEdit 的简化算法

输入: 真实图像 XsrcX^{\text{src}}Xsrc,{ti}i=0T\{t_i\}_{i=0}^{T}{ti​}i=0T​,nmax⁡n_{\max}nmax​,navgn_{\text{avg}}navg​

输出: 编辑后的图像 XtarX^{\text{tar}}Xtar

初始化: Ztmax⁡FE=XsrcZ^{\text{FE}}_{t_{\max}}=X^{\text{src}}Ztmax​FE​=Xsrc

for i=nmax⁡i=n_{\max}i=nmax​ to 111 do

  • Nti∼N(0,1)N_{t_i}\sim\mathcal{N}(0,1)Nti​​∼N(0,1)
  • Ztisrc←(1−ti)Xsrc+tiNtiZ^{\text{src}}_{t_i}\leftarrow(1-t_i)X^{\text{src}}+t_i N_{t_i}Zti​src​←(1−ti​)Xsrc+ti​Nti​​
  • Ztitar←ZtiFE+Ztisrc−XsrcZ^{\text{tar}}_{t_i}\leftarrow Z^{\text{FE}}_{t_i}+Z^{\text{src}}_{t_i}-X^{\text{src}}Zti​tar​←Zti​FE​+Zti​src​−Xsrc
  • VtiΔ←Vtar(Ztitar,ti)−Vsrc(Ztisrc,ti)V^{\Delta}_{t_i}\leftarrow V^{\text{tar}}(Z^{\text{tar}}_{t_i},t_i)-V^{\text{src}}(Z^{\text{src}}_{t_i},t_i)Vti​Δ​←Vtar(Zti​tar​,ti​)−Vsrc(Zti​src​,ti​)
  • Zti−1FE←ZtiFE+(ti−1−ti)VtiΔZ^{\text{FE}}_{t_{i-1}}\leftarrow Z^{\text{FE}}_{t_i}+(t_{i-1}-t_i)V^{\Delta}_{t_i}Zti−1​FE​←Zti​FE​+(ti−1​−ti​)Vti​Δ​

end for

返回: Z0FE=X0tarZ^{\text{FE}}_{0}=X^{\text{tar}}_{0}Z0FE​=X0tar​

(内部赋值可以选择对 navgn_{\text{avg}}navg​ 个样本取平均。)

Generated cat images compared with dog images produced by FlowEdit and by inversion

Figure 4. Cats and dogs experiment. We generated 1000 cat images using diverse prompts and edited them to dog images using both FlowEdit and inversion. FlowEdit outperforms editing by inversion, achieving a lower transport costs and better FID and KID scores (computed against 1000 generated dog images).

图 4. 猫与狗实验。我们用多样的提示生成了 1000 张猫图,再分别用 FlowEdit 和 inversion 把它们编辑成狗图。FlowEdit 优于 inversion 编辑,传输代价更低,FID 和 KID 也更好(对照 1000 张生成的狗图计算)。

5.1. Practical considerations

5.1. 实践中的考虑

In practice, a discrete set of timesteps {ti}i=0T\{t_i\}_{i=0}^{T}{ti​}i=0T​ is used to drive the editing process, where TTT is the number of discretization steps. Additionally, the expectation in (9) is approximated by averaging navgn_{\text{avg}}navg​ model predictions at each timestep. As opposed to the theoretical expectation operator, when taking navgn_{\text{avg}}navg​ to be small, the covariance function of NtN_tNt​ starts playing a role. To obtain a good approximation, it is possible to exploit the averaging that naturally occurs across timesteps. We do so by choosing the covariance function of NtN_tNt​ to satisfy E[NtNs]=0\mathbb{E}[N_t N_s]=0E[Nt​Ns​]=0 for every ∣t−s∣>δ|t-s|>\delta∣t−s∣>δ, where δ\deltaδ is the ODE discretization step, so that the noise becomes independent across timesteps.

实践中,编辑过程由一组离散时间步 {ti}i=0T\{t_i\}_{i=0}^{T}{ti​}i=0T​ 驱动,其中 TTT 是离散步数。此外,(9) 中的期望在每个时间步用 navgn_{\text{avg}}navg​ 次模型预测的平均来近似。与理论上的期望算子不同,当 navgn_{\text{avg}}navg​ 取得较小时,NtN_tNt​ 的协方差函数开始起作用。为了得到较好的近似,可以利用时间步之间自然发生的平均。我们的做法是:选择 NtN_tNt​ 的协方差函数,使得对每个 ∣t−s∣>δ|t-s|>\delta∣t−s∣>δ 都有 E[NtNs]=0\mathbb{E}[N_t N_s]=0E[Nt​Ns​]=0,其中 δ\deltaδ 是 ODE 的离散步长,从而使噪声在不同时间步之间相互独立。

Note that the averaged velocity term in (9) corresponds to the difference E[Vtar(Z^ttar,t) ∣ Z0src]−E[Vsrc(Z^tsrc,t) ∣ Z0src]\mathbb{E}[V^{\text{tar}}(\hat{Z}^{\text{tar}}_{t},t)\,|\,Z^{\text{src}}_{0}]-\mathbb{E}[V^{\text{src}}(\hat{Z}^{\text{src}}_{t},t)\,|\,Z^{\text{src}}_{0}]E[Vtar(Z^ttar​,t)∣Z0src​]−E[Vsrc(Z^tsrc​,t)∣Z0src​], which could be computed by sampling Z^ttar\hat{Z}^{\text{tar}}_{t}Z^ttar​ and Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ independently. However, we choose to compute them with the same noise instances, which further improves robustness to small values of navgn_{\text{avg}}navg​. This is aligned with the observation of Hertz et al. [15] in the context of diffusion models, that calculating a difference between correlated noisy marginals reduces artifacts. See Sec. 5.2 for further discussion.

注意,(9) 中平均后的速度项,对应于差值 E[Vtar(Z^ttar,t) ∣ Z0src]−E[Vsrc(Z^tsrc,t) ∣ Z0src]\mathbb{E}[V^{\text{tar}}(\hat{Z}^{\text{tar}}_{t},t)\,|\,Z^{\text{src}}_{0}]-\mathbb{E}[V^{\text{src}}(\hat{Z}^{\text{src}}_{t},t)\,|\,Z^{\text{src}}_{0}]E[Vtar(Z^ttar​,t)∣Z0src​]−E[Vsrc(Z^tsrc​,t)∣Z0src​],本来可以通过独立抽样 Z^ttar\hat{Z}^{\text{tar}}_{t}Z^ttar​ 和 Z^tsrc\hat{Z}^{\text{src}}_{t}Z^tsrc​ 来计算。但我们选择用同一组噪声实例来计算它们,这能进一步提高对较小 navgn_{\text{avg}}navg​ 的鲁棒性。这与 Hertz 等人 [15] 在扩散模型中的观察一致:在相关的带噪边际之间做差分,可以减少伪影。进一步讨论见第 5.2 节。

Similarly to [19, 30], we define an integer that determines the starting timestep for the process 0≤nmax⁡≤T0\leq n_{\max}\leq T0≤nmax​≤T, meaning that the process is initialized with Ztnmax⁡FE=XsrcZ^{\text{FE}}_{t_{n_{\max}}}=X^{\text{src}}Ztnmax​​FE​=Xsrc. When nmax⁡=Tn_{\max}=Tnmax​=T, the full edit path is traversed and the strongest edit is obtained. When nmax⁡<Tn_{\max}<Tnmax​<T, the first (T−nmax⁡)(T-n_{\max})(T−nmax​) timesteps are skipped, effectively shortening the edit path. This is equivalent to inversion, where weaker edits are obtained by inverting up to timestep nmax⁡n_{\max}nmax​, and sampling from there. We illustrate the effect of nmax⁡n_{\max}nmax​ in App. C.

与 [19, 30] 类似,我们定义一个整数来决定过程的起始时间步,0≤nmax⁡≤T0\leq n_{\max}\leq T0≤nmax​≤T,也就是用 Ztnmax⁡FE=XsrcZ^{\text{FE}}_{t_{n_{\max}}}=X^{\text{src}}Ztnmax​​FE​=Xsrc 来初始化。当 nmax⁡=Tn_{\max}=Tnmax​=T 时,会走完整条编辑路径,得到最强的编辑。当 nmax⁡<Tn_{\max}<Tnmax​<T 时,会跳过前 (T−nmax⁡)(T-n_{\max})(T−nmax​) 个时间步,相当于把编辑路径缩短。这与 inversion 里的做法相当:只 inversion 到时间步 nmax⁡n_{\max}nmax​,再从那里开始采样,就会得到较弱的编辑。我们在附录 C 中展示了 nmax⁡n_{\max}nmax​ 的影响。

Algorithm 1 provides a simplified overview of our algorithm. For a more detailed version, please see Alg. S1.

算法 1 给出了我们算法的简化概览。更详细的版本请见算法 S1。

5.2. Relation to optimization based methods

5.2. 与基于优化的方法的关系

Superficially, FlowEdit may seem similar to optimization based methods like DDS [15] and PDS [22]. These techniques, introduced for diffusion rather than flow, attempt to minimize a loss that measures the alignment between the model's noise predictions with the source and target prompts. They approximate the gradient of the loss using only forward-passes through the model. For example, in DDS, the update step is the difference between the two noise predictions, which can be considered similar to our VtΔ(Z^tsrc,Z^ttar)V^{\Delta}_{t}(\hat{Z}^{\text{src}}_{t},\hat{Z}^{\text{tar}}_{t})VtΔ​(Z^tsrc​,Z^ttar​) direction. However, as we show in App. L, writing VtΔ(Z^tsrc,Z^ttar)V^{\Delta}_{t}(\hat{Z}^{\text{src}}_{t},\hat{Z}^{\text{tar}}_{t})VtΔ​(Z^tsrc​,Z^ttar​) in terms of noise predictions reveals that it is different from both the DDS and the PDS updates.

表面上,FlowEdit 可能看起来类似 DDS [15] 和 PDS [22] 这类基于优化的方法。这些技术是为扩散模型而非流模型提出的,试图最小化一个损失,该损失衡量模型噪声预测与源提示、目标提示的对齐程度。它们只用模型的前向计算来近似损失的梯度。例如在 DDS 中,更新步就是两次噪声预测之差,这可以看成与我们的 VtΔ(Z^tsrc,Z^ttar)V^{\Delta}_{t}(\hat{Z}^{\text{src}}_{t},\hat{Z}^{\text{tar}}_{t})VtΔ​(Z^tsrc​,Z^ttar​) 方向相似。然而,正如我们在附录 L 中表明的,如果把 VtΔ(Z^tsrc,Z^ttar)V^{\Delta}_{t}(\hat{Z}^{\text{src}}_{t},\hat{Z}^{\text{tar}}_{t})VtΔ​(Z^tsrc​,Z^ttar​) 写成噪声预测的形式,会发现它既不同于 DDS 的更新,也不同于 PDS 的更新。

One may still wonder whether FlowEdit can be viewed as an optimization process, only for a different loss. Specifically, similarly to DDS, which attempts to minimize the norm of the difference between noise predictions in diffusion models, maybe FlowEdit attempts to minimize the norm of the difference between velocity predictions in flow models? In App. L, we show that this optimization viewpoint is unjustified even for DDS. Namely, the DDS iterations do not decrease the DDS loss; they rather tend to increase it. Moreover, if allowed to continue beyond the default number of iterations, the quality of the edited image deteriorates. The optimization viewpoint is unnatural also because unlike standard optimizers, FlowEdit does not choose timesteps at random but rather uses a monotonically decreasing schedule {ti}i=0nmax⁡\{t_i\}_{i=0}^{n_{\max}}{ti​}i=0nmax​​. Additionally, it must use a learning rate of exactly dt=ti−1−tidt=t_{i-1}-t_idt=ti−1​−ti​ at iteration iii (namely, the sum of all step-sizes must be 1). Any slight deviation from this learning rate causes the results to deteriorate significantly (see App. L).

人们仍可能问:FlowEdit 是否也可以看成一个优化过程,只是损失不同?具体来说,DDS 试图最小化扩散模型中噪声预测之差的范数,那么 FlowEdit 是不是在试图最小化流模型中速度预测之差的范数?我们在附录 L 中表明,即便对 DDS 而言,这种优化视角也站不住脚。也就是说,DDS 的迭代并不会降低 DDS 损失,反而往往会把它抬高。而且,如果允许迭代次数超过默认值,编辑图像的质量还会变差。这个优化视角之所以不自然,还因为 FlowEdit 并不像标准优化器那样随机选择时间步,而是使用单调递减的时间表 {ti}i=0nmax⁡\{t_i\}_{i=0}^{n_{\max}}{ti​}i=0nmax​​。此外,它在第 iii 次迭代必须使用恰好为 dt=ti−1−tidt=t_{i-1}-t_idt=ti−1​−ti​ 的学习率(也就是说,所有步长之和必须为 1)。学习率只要稍有偏离,结果就会明显变差(见附录 L)。

5.3. Comparison to editing by (exact) inversion

5.3. 与(精确)inversion 编辑的比较

To demonstrate the reduced transport cost of FlowEdit compared to editing-by-inversion, we evaluate both methods on a synthetic dataset of model-generated images. This way, the initial noise maps are known, ensuring the inversion is exact and eliminating potential issues of approximate inversion. The dataset consists of 1000 cat images generated by SD3 using variations of the prompt “a photo of cat” generated by Llama3 [9]. We edit these images using both FlowEdit and inversion, with the target prompt identical to the source prompt, except for replacing “cat” with “dog”. We calculate the transport cost for both methods by measuring the MSE between source and edited images in the model's latent space, as well as LPIPS [50] on the decoded images. As expected, FlowEdit achieves a lower transport cost (1376 vs. 2239 for MSE, 0.15 vs. 0.25 for LPIPS), indicating superior preservation of the structure and semantics of the source image. Figure 4 shows a small qualitative comparison for the cat-to-dog edits.

为了说明 FlowEdit 相对 inversion 编辑降低了传输代价,我们在由模型生成的合成图像数据集上评估两种方法。这样,初始噪声图是已知的,inversion 是精确的,从而排除了近似 inversion 可能带来的问题。该数据集包含 1000 张由 SD3 生成的猫图,提示是 Llama3 [9] 生成的 “a photo of cat” 的多种变体。我们分别用 FlowEdit 和 inversion 编辑这些图像,目标提示与源提示相同,只是把 “cat” 换成 “dog”。两种方法的传输代价,用源图像与编辑图像在模型潜在空间中的均方误差(MSE)来衡量,并在解码图像上计算 LPIPS [50]。如所预期,FlowEdit 的传输代价更低(MSE 为 1376 对 2239,LPIPS 为 0.15 对 0.25),表明它对源图像结构和语义的保持更好。图 4 给出了猫改狗编辑的一小部分定性比较。

To assess the alignment of our edits with the target distribution, we generated 1000 dog images with SD3, using the same target prompts but with “dog” instead of “cat”. We followed by calculating FID [17] and KID [4] between the generated dog images and the edited dog images, for both methods. Our method achieves lower FID (51.14 vs. 55.88) and KID (0.017 vs. 0.023), indicating that similarly to inversion, our ODE path is able to produce images from the target distribution. See App. M for more details and results.

为评估编辑结果与目标分布的对齐程度,我们用 SD3 又生成了 1000 张狗图,提示与前面的目标提示相同,只是把 “cat” 换成 “dog”。然后对两种方法,分别计算这些生成狗图与编辑得到的狗图之间的 FID [17] 和 KID [4]。我们的方法取得了更低的 FID(51.14 对 55.88)和 KID(0.017 对 0.023),说明与 inversion 类似,我们的 ODE 路径也能够生成来自目标分布的图像。更多细节和结果见附录 M。

Grid of FlowEdit results on diverse real images, including bike to vespa, rabbit to puppy, and coconut to baseball

Figure 5. FlowEdit results. FlowEdit successfully edits diverse images using various editing prompts. The edits preserve the structure of the original image, changing only the specified region. FLUX was used for the first and third rows and SD3 for the second and fourth rows.

图 5. FlowEdit 的结果。FlowEdit 能用各种编辑提示,成功地改写多样的图像。编辑保持了原图结构,只改变指定区域。第一、三行使用 FLUX,第二、四行使用 SD3。

6. Experiments

6. 实验

Implementation details. In our experiments we use the official weights of SD3 medium [42] and FLUX.1 dev [25], available at HuggingFace, as the base T2I flow models. For SD3 we use T=50T=50T=50 steps, with nmax⁡=33n_{\max}=33nmax​=33. SD3 employs CFG [18] for their text conditioning. We set the source and target scales to 3.5 and 13.5, respectively. For FLUX we use T=28T=28T=28 steps, with nmax⁡=24n_{\max}=24nmax​=24. FLUX takes CFG as an input conditioning, which we set to the values of 1.5 for the source conditioning, and 5.5 for the target. For both methods we use navg=1n_{\text{avg}}=1navg​=1 (see Sec. 5.1 and App. K). We use these hyperparameters for all the results in Figs. 5,6,7.

实现细节。 实验中,我们使用 HuggingFace 上提供的 SD3 medium [42] 和 FLUX.1 dev [25] 官方权重,作为基础 T2I 流模型。对 SD3,我们使用 T=50T=50T=50 步,并取 nmax⁡=33n_{\max}=33nmax​=33。SD3 用 CFG [18] 做文本条件,我们把源尺度和目标尺度分别设为 3.5 和 13.5。对 FLUX,我们使用 T=28T=28T=28 步,并取 nmax⁡=24n_{\max}=24nmax​=24。FLUX 把 CFG 作为输入条件,我们把源条件设为 1.5,目标条件设为 5.5。两种方法都取 navg=1n_{\text{avg}}=1navg​=1(见第 5.1 节和附录 K)。图 5、6、7 中的全部结果都使用这些超参数。

Dataset. Our dataset consists of a diverse set of over 70 real images of dimension 102421024^{2}10242 from the DIV2K dataset [1] and from royalty free online sources [35, 36]. Each image has a source prompt, which was obtained from LLaVA-1.5 [27] and manually refined. The source prompts have little effect on FlowEdit's results. In particular, they can be omitted (App. I). For each image, several handcrafted target prompts are provided to facilitate diverse edits. Overall, the dataset consists of over 250 text-image pairs, and is used to evaluate our and the competing methods. The dataset, including source and target prompt pairs, is available on our official github repository.

数据集。 我们的数据集包含 70 多张多样的真实图像,分辨率为 102421024^{2}10242,来自 DIV2K 数据集 [1] 以及免版税的在线来源 [35, 36]。每张图都有一句源提示,先由 LLaVA-1.5 [27] 得到,再经人工润色。源提示对 FlowEdit 的结果影响很小,甚至可以省略(附录 I)。对每张图,我们还手写了若干目标提示,以便做多样化编辑。整个数据集共有 250 多对文本–图像,用于评估我们的方法以及对照方法。包含源提示和目标提示对的数据集,发布在我们的官方 GitHub 仓库(https://github.com/fallenshock/FlowEdit)。

Competing methods. We compare our method against several competing text-based real image editing methods that use flow models. The first is editing by ODE inversion, which we apply with the same hyperparameters as our method. The second is SDEdit [30], which is easily applied to flow models by adding noise to the source image up to a specified nmax⁡n_{\max}nmax​ step and then performing regular sampling conditioned on a target prompt to obtain the edit. The parameter nmax⁡n_{\max}nmax​ controls the strength of the edit. Additionally, we compare to iRFDS [49], which is an SDS-based editing method for flow models. We use their official implementation and hyperparameters, which are available only for SD3. Furthermore, we compare to RF-Inversion [39]. As an official implementation is not available at the time of writing, we implement it based on the provided pseudo-code and the hyperparameters reported in the SM, which are available only for FLUX. Finally, we compare to RF Edit [46] using their official implementation, which is also available only for FLUX. Whenever possible, we performed a hyperparameters search for each method to identify the optimal settings for editing. The final parameters used to create the edited images in the paper are detailed in App. B. We do not compare FlowEdit to text-based image editing techniques for diffusion models [6, 19, 44], as these are not easily adapted to flow models, and a direct comparison would not be fair due to model differences.

对照方法。 我们把方法与若干同样使用流模型、基于文本的真实图像编辑方法作比较。第一种是 ODE inversion 编辑,超参数与我们的方法相同。第二种是 SDEdit [30],它很容易用到流模型上:把源图像加噪到指定的 nmax⁡n_{\max}nmax​ 步,再在目标提示条件下做常规采样,得到编辑结果。参数 nmax⁡n_{\max}nmax​ 控制编辑强度。此外,我们还与 iRFDS [49] 比较,这是一种面向流模型、基于 SDS 的编辑方法。我们使用其官方实现和超参数,而它们只提供了 SD3 版本。我们还与 RF-Inversion [39] 比较。撰写本文时官方实现尚未发布,我们根据其给出的伪代码以及补充材料中报告的超参数自行实现,而这些超参数也只针对 FLUX。最后,我们用 RF Edit [46] 的官方实现与之比较,该实现同样只提供 FLUX 版本。只要条件允许,我们都会为每种方法做超参数搜索,以找到适合编辑的设置。本文中生成编辑图像所用的最终参数详见附录 B。我们没有把 FlowEdit 与面向扩散模型的文本图像编辑技术 [6, 19, 44] 直接比较,因为这些方法不容易适配到流模型,而且由于底层模型不同,直接比较也不公平。

Qualitative comparisons of FlowEdit against SDEdit, ODE inversion, iRFDS, RF inversion and RF Edit on SD3 and FLUX

Figure 6. Qualitative comparisons. With both SD3 (left) and FLUX (right), FlowEdit better adheres to the target prompt while simultaneously preserving the structure of the original image. The value next to SDEdit indicates the strength.

图 6. 定性比较。在 SD3(左)和 FLUX(右)上,FlowEdit 都能更好地符合目标提示,同时保持原图结构。SDEdit 旁的数值表示编辑强度。

Qualitative evaluation. Figures 1, 5, S1 show editing results obtained with FlowEdit across a diverse set of images and target prompts, including localized edits to text and objects, modifications of multiple objects at once, pose changes, etc. Our edits exhibit good structural preservation of the source image, and simultaneously maintain good adherence to the target text. Figure 6 shows comparisons between FlowEdit and the competing methods for both SD3 and FLUX. FlowEdit is the only method that consistently adheres to the text prompt while preserving the structure of the original image. For example, FlowEdit is the only method that can both change the large text on the gas station sign (third row) and preserve the cars and background of the source image. Some methods, like RF-Inversion, are able to change the text on the sign but fail to maintain the structure of the original image, while others, such as iRFDS, somewhat preserve the structure but fail to fully change the sign. In the last row, while all methods perform the required edit (changing the decoration to strawberries), only FlowEdit (for both SD3 and FLUX) preserves the background accurately. Other methods introduce additional, unintended elements such as furniture and flowers. See App. B for more qualitative comparisons.

定性评估。 图 1、图 5 和图 S1 展示了 FlowEdit 在多样图像和目标提示上的编辑结果,包括对文字和物体的局部修改、同时改多个物体、姿态变化等。我们的编辑对源图像有较好的结构保持,同时对目标文本也有较好的符合度。图 6 给出了 FlowEdit 与对照方法在 SD3 和 FLUX 上的比较。FlowEdit 是唯一能持续既符合文本提示、又保持原图结构的方法。例如,只有 FlowEdit 既能改掉加油站招牌上的大字(第三行),又能保住源图像中的汽车和背景。有的方法,如 RF-Inversion,能改招牌上的字,却保不住原图结构;另一些方法,如 iRFDS,结构保持得还行,却不能把招牌彻底改掉。在最后一行,虽然所有方法都完成了指定编辑(把装饰改成草莓),但只有 FlowEdit(在 SD3 和 FLUX 上皆如此)准确地保住了背景。其他方法会额外引入家具、花卉等并非预期的元素。更多定性比较见附录 B。

CLIP versus LPIPS scatter plots comparing FlowEdit with competing methods on Stable Diffusion 3 and FLUX

Figure 7. Quantitative comparisons. FlowEdit achieves a favorable balance between text adherence (CLIP) and structure preservation (LPIPS) compared to other methods. Connected markers represent different hyperparameters (see App. B).

图 7. 定量比较。与其他方法相比,FlowEdit 在文本符合度(CLIP)和结构保持(LPIPS)之间取得了更有利的平衡。相连的标记表示不同的超参数(见附录 B)。

Quantitative evaluation. We numerically evaluate the results of FlowEdit and the other methods using LPIPS [50] to measure the semantic structure preservation (lower is better) and CLIP [38] to assess text adherence (higher is better). Figure 7 displays these results for SD3 (left) and FLUX (right), with varied hyperparameters. The plots show that FlowEdit achieves a favorable balance between structure preservation and text adherence. The other methods either maintain the structure of original image at the cost of a weak edit or modify the image with little regard to its original semantics. Additional quantitative evaluations and details, including additional similarity metrics beyond LPIPS, are provided in App. B.

定量评估。 我们用 LPIPS [50] 衡量语义结构保持(越低越好),用 CLIP [38] 评估文本符合度(越高越好),对 FlowEdit 和其他方法的结果做数值评估。图 7 展示了 SD3(左)和 FLUX(右)在不同超参数下的结果。图中可见,FlowEdit 在结构保持和文本符合度之间取得了更有利的平衡。其他方法要么以编辑偏弱为代价保住原图结构,要么改动图像时几乎不顾原有语义。更多定量评估和细节,包括 LPIPS 之外的相似性指标,见附录 B。

Style-editing examples turning landscapes and photos into Minecraft, painting, anime and Disney styles

Figure 8. Text-based style editing. FlowEdit changes the style of an image at the cost of slightly deviating from the original image structure. SD3 was used for the first row and FLUX for the second. Hyperparameters used for these results are reported in App. D.

图 8. 基于文本的风格编辑。FlowEdit 能改变图像风格,代价是会略微偏离原图结构。第一行使用 SD3,第二行使用 FLUX。这些结果所用的超参数见附录 D。

Text-based style editing. The hyperparameters used for the previous experiments lead to strong structure preservation and are thus not optimal for style editing. Figure 8 demonstrates the results of FlowEdit for text-based style editing, where by allowing some deviation from the original structure, we gain stylistic flexibility. To achieve these results, we simply remove the dependence on the source image in the final generation steps, as discussed in App. D.

基于文本的风格编辑。 前面实验所用的超参数会带来很强的结构保持,因而并不适合风格编辑。图 8 展示了 FlowEdit 做文本风格编辑的结果:通过允许对原有结构有一定偏离,我们获得了风格上的灵活性。为得到这些结果,我们只是在最后几步生成中去掉对源图像的依赖,详见附录 D。

7. Conclusion and limitations

7. 结论与局限

We introduced FlowEdit – an inversion-free, optimization-free and model agnostic method for text-based image editing using pre-trained flow models. Our approach constructs a direct ODE between the source and target distributions (corresponding to source and target text prompts), without passing through the standard Gaussian distribution as in inversion-based editing. Evaluations on synthetic datasets show that FlowEdit achieves lower transport costs, and thus stronger structure preservation. This translates to state-of-the-art performance across various editing tasks, as we illustrated with FLUX and SD3. While FlowEdit's strong structure preservation is beneficial for precise editing tasks, it can become a limitation when substantial modifications to large regions of the image are desired. This is illustrated in Fig. S6 in the context of pose and background editing.

我们提出了 FlowEdit:一种无需 inversion、无需优化、也不依赖特定模型的方法,用预训练流模型做基于文本的图像编辑。我们的做法是在源分布和目标分布(分别对应源文本提示和目标文本提示)之间直接构造一条 ODE,而不像 inversion 编辑那样途经标准高斯分布。在合成数据集上的评估表明,FlowEdit 的传输代价更低,因而结构保持更强。这一点转化为各种编辑任务上的目前最佳表现,我们在 FLUX 和 SD3 上作了展示。FlowEdit 很强的结构保持对精确编辑有利,但若希望对图像的大片区域做大幅度修改,它也可能成为限制。图 S6 在姿态和背景编辑的情境中说明了这一点。

Acknowledgements

致谢

This research was supported by the Israel Science Foundation (grant no. 2318/22) and by the Ollendorff Minerva Center, ECE faculty, Technion.

本研究得到以色列科学基金会(资助号 2318/22)以及 Technion 电气与计算机工程学院 Ollendorff Minerva Center 的资助。

References

参考文献

[1] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 126–135, 2017.

[2] Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023.

[3] Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemchinov, Kfir Aberman, Dani Lischinski, and Daniel Cohen-Or. Stable flow: Vital layers for training-free image editing. arXiv preprint arXiv:2411.14430, 2024.

[4] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.

[5] Manuel Brack, Felix Friedrich, Katharia Kornmeier, Linoy Tsaban, Patrick Schramowski, Kristian Kersting, and Apolinário Passos. Ledits++: Limitless image editing using text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8861–8870, 2024.

[6] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560–22570, 2023.

[7] Nathaniel Cohen, Vladimir Kulikov, Matan Kleiner, Inbar Huberman-Spiegelglas, and Tomer Michaeli. Slicedit: Zero-shot video editing with text-to-image diffusion models using spatio-temporal slices. In Proceedings of the 41st International Conference on Machine Learning, pages 9109–9137. PMLR, 2024.

[8] Gilad Deutch, Rinon Gal, Daniel Garibi, Or Patashnik, and Daniel Cohen-Or. Turboedit: Text-based image editing using few-step diffusion models. In SIGGRAPH Asia 2024 Conference Papers, pages 1–12, 2024.

[9] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.

[10] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.

[11] Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Elor, and Daniel Cohen-Or. Renoise: Real image inversion through iterative noising. In Computer Vision – ECCV 2024, pages 395–413, 2024.

[12] René Haas, Inbar Huberman-Spiegelglas, Rotem Mulayoff, Stella Graßhof, Sami S Brandt, and Tomer Michaeli. Discovering interpretable directions in the semantic latent space of diffusion models. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pages 1–9. IEEE, 2024.

[13] Ligong Han, Song Wen, Qi Chen, Zhixing Zhang, Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et al. Proxedit: Improving tuning-free real image editing with proximal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4291–4301, 2024.

[14] Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19740–19750, 2023.

[15] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2328–2337, 2023.

[16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to-prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2023.

[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.

[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.

[19] Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12469–12478, 2024.

[20] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023.

[21] Jeongsol Kim, Geon Yeong Park, and Jong Chul Ye. Dreamsampler: Unifying diffusion sampling and score distillation for image manipulation. In Computer Vision – ECCV 2024, pages 398–414. Springer Nature Switzerland, 2024.

[22] Juil Koo, Chanho Park, and Minhyuk Sung. Posterior distillation sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13352–13361, 2024.

[23] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In The Eleventh International Conference on Learning Representations, 2023.

[24] Black Forest Labs. Flux. https://github.com/black-forest-labs/flux, 2024. Accessed: 2024-11-14.

[25] Black Forest Labs. Official weights of FLUX.1 dev. https://huggingface.co/black-forest-labs/FLUX.1-dev, 2024. Accessed: 2024-11-14.

[26] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.

[27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024.

[28] Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.

[29] Hila Manor and Tomer Michaeli. Zero-shot unsupervised and text-based audio editing using DDPM inversion. In Proceedings of the 41st International Conference on Machine Learning, pages 34603–34629. PMLR, 2024.

[30] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.

[31] Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 2063–2072, 2025.

[32] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023.

[33] Hyelin Nam, Gihyun Kwon, Geon Yeong Park, and Jong Chul Ye. Contrastive denoising score for text-guided latent diffusion image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9192–9201, 2024.

[34] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.

[35] Pexels. Pexels - free stock photos & videos. https://www.pexels.com, 2024. Accessed: 2024-11-14.

[36] PxHere. Pxhere - free images & free stock photos. https://pxhere.com/, 2024. Accessed: 2024-11-14.

[37] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15932–15942, 2023.

[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.

[39] Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. Semantic image inversion and editing using rectified stochastic differential equations. In The Thirteenth International Conference on Learning Representations, 2025.

[40] Dvir Samuel, Barak Meiri, Haggai Maron, Yoad Tewel, Nir Darshan, Shai Avidan, Gal Chechik, and Rami Ben-Ari. Lightning-fast image inversion and editing for text-to-image diffusion models. In The Thirteenth International Conference on Learning Representations, 2025.

[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.

[42] StablityAI. Official weights of SD3 medium diffusers. https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers, 2024. Accessed: 2024-11-14.

[43] Linoy Tsaban and Apolinário Passos. Ledits: Real image editing with ddpm inversion and semantic guidance. arXiv preprint arXiv:2307.00522, 2023.

[44] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921–1930, 2023.

[45] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22532–22541, 2023.

[46] Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, and Ying Shan. Taming rectified flow for inversion and editing. arXiv preprint arXiv:2411.04746, 2024.

[47] Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero-shot image editing and guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7378–7387, 2023.

[48] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623–7633, 2023.

[49] Xiaofeng Yang, Chen Cheng, Xulei Yang, Fayao Liu, and Guosheng Lin. Text-to-image rectified flow as plug-and-play priors. In The Thirteenth International Conference on Learning Representations, 2025.

[50] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.

FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Models | AGIDaily