Generative Image Dynamics

生成式图像动力学

Zhengqi Li    Richard Tucker    Noah Snavely    Aleksander Holynski Affiliation:  Google Research

Abstract

摘要

We present an approach to modeling an image-space prior on scene motion. Our prior is learned from a collection of motion trajectories extracted from real video sequences depicting natural, oscillatory dynamics of objects such as trees, flowers, candles, and clothes swaying in the wind. We model dense, long-term motion in the Fourier domain as spectral volumes, which we find are well-suited to prediction with diffusion models. Given a single image, our trained model uses a frequency-coordinated diffusion sampling process to predict a spectral volume, which can be converted into a motion texture that spans an entire video. Along with an image-based rendering module, the predicted motion representation can be used for a number of downstream applications, such as turning still images into seamlessly looping videos, or allowing users to interact with objects in real images, producing realistic simulated dynamics (by interpreting the spectral volumes as image-space modal bases). See our project page for more results: generative-dynamics.github.io.

我们提出一种对场景运动建模图像空间先验的方法。该先验从真实视频序列中提取的运动轨迹集合学习而来；这些视频刻画树木、花朵、蜡烛与衣物等物体在风中摇曳的自然振荡动力学。我们在傅里叶域将稠密、长时程运动建模为谱体积（spectral volumes），并发现其非常适合用扩散模型做预测。给定单张图像，训练好的模型用频率协调的扩散采样过程预测谱体积，再将其转换为跨越整段视频的运动纹理。配合基于图像的渲染模块，预测的运动表示可用于多项下游应用：例如把静图变成无缝循环视频，或让用户与真实图像中的物体交互、产生逼真的模拟动力学（将谱体积解读为图像空间模态基）。更多结果见项目页：generative-dynamics.github.io。

1 Introduction

1 引言

The natural world is always in motion, with even seemingly static scenes containing subtle oscillations as a result of wind, water currents, respiration, or other natural rhythms. Emulating this motion is crucial in visual content synthesis—human sensitivity to motion can cause imagery without motion (or with slightly unrealistic motion) to seem uncanny or unreal.

自然世界始终在运动——即便看似静止的场景，也会因风、水流、呼吸或其他自然节律而含有细微振荡。在视觉内容合成中，模拟这种运动至关重要：人类对运动高度敏感，缺少运动（或运动略显不真实）的画面会显得诡异或不真实。

While it is easy for humans to interpret or imagine motion in scenes, training a model to learn or produce realistic scene motion is far from trivial. The motion we observe in the world is the result of a scene’s underlying physical dynamics, i.e., forces applied to objects that respond according to their unique physical properties—their mass, elasticity, etc—quantities that are hard to measure and capture at scale. Fortunately, measuring them is not necessary for certain applications: e.g., one can simulate plausible dynamics in a scene by simply analyzing some observed 2D motion [23].

人类解读或想象场景中的运动相对容易，但训练模型去学习或产生逼真场景运动远非易事。我们在世界中观察到的运动，是场景底层物理动力学的结果——即施加于物体上的力，物体按其独特物理属性（质量、弹性等）作出响应；这些量难以大规模测量与采集。幸运的是，对某些应用不必测量它们：例如，仅通过分析观察到的二维运动，就可在场景中模拟可信动力学 [23]。

This same observed motion can also serve as a supervisory signal in learning dynamics across scenes—because although observed motion is multi-modal and grounded in complex physical effects, it is nevertheless often predictable: candles will flicker in certain ways, trees will sway, and their leaves will rustle. For humans, this predictability is ingrained in our systems of perception: by viewing a still image, we can imagine plausible motions— or, since there might have been many possible such motions, a distribution of natural motions conditioned on that image. Given the facility with which humans are able to model these distributions, a natural research problem is to model them computationally.

同一套观测运动也可作为跨场景学习动力学的监督信号——因为尽管观测运动是多模态的、植根于复杂物理效应，它往往仍可预测：蜡烛会以特定方式闪烁，树木会摇摆，树叶会沙沙作响。对人类而言，这种可预测性已内化于感知系统：看着一张静图，我们能想象可信运动——或者，由于可能存在多种此类运动，想象以该图像为条件的自然运动分布。鉴于人类建模这些分布如此熟练，用计算方式建模它们便成为自然的研究问题。

Recent advances in generative models, in particular conditional diffusion models [85, 87, 44], have enabled us to model rich distributions, including distributions of real images conditioned on text [75, 73, 74]. This capability has enabled several new applications, such as text-conditioned generation of diverse and realistic image content. Following the success of these image models, recent work has extended these models to other domains, such as videos [43, 7] and 3D geometry [101, 103, 100, 77].

生成模型、尤其是条件扩散模型 [85, 87, 44] 的近期进展，使我们能建模丰富分布，包括以文本为条件的真实图像分布 [75, 73, 74]。这一能力催生了多项新应用，例如以文本为条件生成多样且逼真的图像内容。继图像模型成功之后，近期工作将这些模型扩展到其他领域，例如视频 [43, 7] 与三维几何 [101, 103, 100, 77]。

In this paper, we model a generative prior for image-space scene motion, i.e., the motion of all pixels in a single image. This model is trained on motion trajectories automatically extracted from a large collection of real video sequences. In particular, from each training video we compute motion in the form of a spectral volume [22, 23], a frequency-domain representation of dense, long-range pixel trajectories. Spectral volumes are well-suited to scenes that exhibit oscillatory dynamics, e.g., trees and flowers moving in the wind. We find that this representation is also highly effective as an output of a diffusion model for modeling scene motion. We train a generative model that, conditioned on a single image, can sample spectral volumes from its learned distribution. A predicted spectral volume can then be directly transformed into a motion texture—a set of long-range, per-pixel motion trajectories—that can be used to animate the image. The spectral volume can also be interpreted as an image-space modal basis for use in simulating interactive dynamics [22].

本文为图像空间场景运动——即单张图像中所有像素的运动——建模生成先验。该模型在从大规模真实视频序列自动提取的运动轨迹上训练。具体地，对每段训练视频，我们以谱体积 [22, 23] 的形式计算运动：这是稠密、长程像素轨迹的频域表示。谱体积适合刻画呈现振荡动力学的场景，例如风中的树木与花朵。我们发现，这一表示作为扩散模型输出、用于建模场景运动也非常有效。我们训练一个以单张图像为条件的生成模型，可从其学得分布中采样谱体积。预测的谱体积可直接变换为运动纹理——一组长程、逐像素运动轨迹——用以驱动图像动画。谱体积也可解读为图像空间模态基，用于模拟交互动力学 [22]。

We predict spectral volumes from input images using a diffusion model that generates coefficients one frequency at a time, but coordinates these predictions across frequency bands through a shared attention module. The predicted motions can be used to synthesize future frames (via an image-based rendering model)—turning still images into realistic animations, as illustrated in Fig. .

我们用扩散模型从输入图像预测谱体积：逐频率生成系数，但通过共享注意力模块跨频带协调这些预测。预测的运动可用于合成未来帧（经由基于图像的渲染模型）——把静图变成逼真动画，如图所示。

Compared with priors over raw RGB pixels, priors over motion capture more fundamental, lower-dimensional structure that efficiently explains long-range variations in pixel values. Hence, generating intermediate motion leads to more coherent long-term generation and more fine-grained control over animations. We demonstrate the use of our trained model in several downstream applications, such as creating seamless looping videos, editing the generated motions, and enabling interactive dynamic images via image-space modal bases, i.e., simulating the response of object dynamics to user-applied forces [22].

相对原始 RGB 像素上的先验，运动上的先验捕捉更本质、更低维的结构，能高效解释像素值的长程变化。因此，生成中间运动可带来更连贯的长时程生成，以及对动画更细粒度的控制。我们展示训练好的模型在若干下游应用中的用法，例如创建无缝循环视频、编辑生成运动，以及经由图像空间模态基实现交互式动态图像——即模拟物体动力学对用户施加力的响应 [22]。

2 Related Work

2 相关工作

Generative synthesis.

Recent advances in generative models have enabled photorealistic synthesis of images conditioned on text prompts [24, 75, 73, 74, 17, 16]. These text-to-image models can be augmented to synthesize video sequences by extending the generated image tensors along a time dimension [43, 84, 7, 106, 111, 62, 106, 9]. While these methods can produce video sequences that capture the spatiotemporal statistics of real footage, these videos often suffer from artifacts like incoherent motion, unrealistic temporal variation in textures, and violations of physical constraints like preservation of mass.

生成式合成。

生成模型的近期进展已能以文本提示为条件做照片级真实的图像合成 [24, 75, 73, 74, 17, 16]。这些文生图模型可通过沿时间维扩展生成的图像张量来合成视频序列 [43, 84, 7, 106, 111, 62, 106, 9]。尽管这些方法能产生捕捉真实素材时空统计的视频序列，视频往往仍受伪影困扰，例如运动不连贯、纹理的时间变化不真实，以及违反质量守恒等物理约束。

Animating images.

Instead of generating videos entirely from text, other techniques take as input a still picture and animate it. Many recent deep learning methods adopt a 3D-Unet architecture to produce video volumes directly [53, 40, 36, 27, 93, 47]. These models are effectively the same video generation models (but conditioned on image information instead of text), and exhibit similar artifacts to those mentioned above. One way to overcome these limitations is to not directly generate the video content itself, but instead animate an input source image through image-based rendering, i.e., moving the image content around according to motion derived from external sources such as a driving video [81, 82, 80, 99, 51], motion or 3D geometry priors [97, 104, 102, 109, 8, 65, 67, 46, 63, 64, 29, 101, 90], or user annotations [20, 38, 6, 33, 105, 98, 108, 18]. Animating images according to motion fields yields greater temporal coherence and realism, but these prior methods either require additional guidance signals or user input, or utilize limited motion representations.

为图像做动画。

另一些技术不以文本从头生成视频，而以静图为输入并为其做动画。许多近期深度学习方法采用 3D-Unet 架构直接产生视频体积 [53, 40, 36, 27, 93, 47]。这些模型实质上仍是视频生成模型（只是以图像信息而非文本为条件），并呈现与上文类似的伪影。克服这些局限的一种方式，不是直接生成视频内容本身，而是通过基于图像的渲染为输入源图像做动画——即按来自外部源的运动移动图像内容，例如驱动视频 [81, 82, 80, 99, 51]、运动或三维几何先验 [97, 104, 102, 109, 8, 65, 67, 46, 63, 64, 29, 101, 90]，或用户标注 [20, 38, 6, 33, 105, 98, 108, 18]。按运动场为图像做动画可带来更强的时间连贯性与真实感，但这些先前方法要么需要额外引导信号或用户输入，要么使用受限的运动表示。

Motion models and motion priors.

In computer graphics, natural, oscillatory 3D motion (e.g., water rippling or trees waving in the wind) can be modeled with noise that is shaped in the Fourier domain and then converted to time-domain motion fields [88, 79]. Some of these methods rely on a modal analysis of the underlying dynamics of the system being simulated [89, 25, 22]. These spectral techniques were adapted to animate plants, water, and clouds from single 2D pictures by Chuang et al. [20], given user annotations. Our work is especially inspired by Davis [23], who connected modal analysis of a scene with the motions observed in a video of that scene, and used this analysis to simulate interactive dynamics from a video. We adopt the frequency-space spectral volume motion representation from Davis et al., extract this representation from a large set of training videos, and show that spectral volumes are suitable for predicting motion from single images with diffusion models.

运动模型与运动先验。

在计算机图形学中，自然、振荡的三维运动（例如水面涟漪或风中树木）可用在傅里叶域整形后再转换为时域运动场的噪声来建模 [88, 79]。其中一些方法依赖对被模拟系统底层动力学的模态分析 [89, 25, 22]。这些谱技术经 Chuang et al. [20] 改造后，在给定用户标注的情况下，可从单张二维图片为植物、水与云做动画。我们的工作尤其受 Davis [23] 启发：他将场景的模态分析与该场景视频中观测到的运动联系起来，并用此分析从视频模拟交互动力学。我们采用 Davis et al. 的频域谱体积运动表示，从大规模训练视频中提取该表示，并表明谱体积适合用扩散模型从单张图像预测运动。

Other methods have used various motion representations in prediction tasks, where an image or video is used to inform a deterministic future motion estimate [71, 34], or a more rich distribution of possible motions [94, 96, 104]. However, many of these methods predict an optical flow motion estimate (i.e., the instantaneous motion of each pixel), not full motion trajectories. In addition, much of this prior work is focused on tasks like activity recognition, not on synthesis tasks. More recent work has demonstrated the advantages of modeling and predicting motion using generative models in a number of closed-domain settings such as humans and animals [91, 19, 28, 72, 2, 107].

另一些方法在预测任务中使用各种运动表示：以图像或视频告知确定性的未来运动估计 [71, 34]，或更丰富的可能运动分布 [94, 96, 104]。但这些方法许多预测的是光流运动估计（即每个像素的瞬时运动），而非完整运动轨迹。此外，大量先前工作聚焦于活动识别等任务，而非合成任务。更近的工作在人类与动物等若干封闭领域设定中，展示了用生成模型建模与预测运动的优势 [91, 19, 28, 72, 2, 107]。

Videos as textures.

Certain moving scenes can be thought of as a kind of texture—termed dynamic textures [26]—that model videos as space-time samples of a stochastic process. Dynamic textures can represent smooth, natural motions like waves, flames, or moving trees, and have been widely used for video classification, segmentation or encoding [12, 14, 15, 76, 13]. A related kind of texture, called a video texture, represents a moving scene as a set of input video frames along with transition probabilities between any pair of frames [78, 66]. A number of methods estimate dynamic or video textures through analysis of scene motion and pixel statistics, with the aim of generating seamlessly looping or infinitely varying output videos [59, 58, 1, 78, 21, 32]. In contrast to much of this work, our method learns priors in advance that can then be applied to single images.

作为纹理的视频。

某些运动场景可被视为一种纹理——称为动态纹理 [26]——将视频建模为随机过程的时空采样。动态纹理可表示波浪、火焰或摇曳树木等平滑自然运动，并广泛用于视频分类、分割或编码 [12, 14, 15, 76, 13]。一种相关纹理称为视频纹理，将运动场景表示为输入视频帧集合，以及任意帧对之间的转移概率 [78, 66]。若干方法通过分析场景运动与像素统计来估计动态或视频纹理，目标是生成无缝循环或无限变化的输出视频 [59, 58, 1, 78, 21, 32]。与这些工作中的多数不同，我们的方法预先学习先验，再应用到单张图像上。

3 Overview

3 概览

Given a single picture $I_{0}$, our goal is to generate a video $\{\hat{I}_{1},\hat{I}_{2}.,...,\hat{I}_{T}\}$ featuring oscillatory motions such as those of trees, flowers, or candle flames swaying in the breeze. Our system consists of two modules: a motion prediction module and an image-based rendering module. Our pipeline begins by using a latent diffusion model (LDM) to predict a spectral volume $\mathcal{S}=\left(S_{f_{0}},S_{f_{1}},...,S_{f_{K-1}}\right)$ for the input $I_{0}$. The predicted spectral volume is then transformed to a motion texture $\mathcal{F}=\left(F_{1},F_{2},...,F_{T}\right)$ through an inverse discrete Fourier transform. This motion determines the position of each input pixel at every future time step.

给定单张图片 $I_{0}$，我们的目标是生成视频 $\{\hat{I}_{1},\hat{I}_{2},...,\hat{I}_{T}\}$，其中呈现树木、花朵或微风中烛焰等振荡运动。系统由两个模块组成：运动预测模块与基于图像的渲染模块。流水线首先用潜扩散模型（LDM）为输入 $I_{0}$ 预测谱体积 $\mathcal{S}=\left(S_{f_{0}},S_{f_{1}},...,S_{f_{K-1}}\right)$。预测的谱体积再经逆离散傅里叶变换转换为运动纹理 $\mathcal{F}=\left(F_{1},F_{2},...,F_{T}\right)$。该运动决定每个输入像素在每个未来时间步的位置。

Given a predicted motion texture, we then animate the input RGB image using a neural image-based rendering technique (Sec. 5). We explore applications of this method, including producing seamless looping animations and simulating interactive dynamics, in Sec. 6.

给定预测的运动纹理，我们再用神经基于图像的渲染技术为输入 RGB 图像做动画（第 5 节）。我们在第 6 节探索该方法的应用，包括产生无缝循环动画与模拟交互动力学。

4 Predicting motion

4 预测运动

Figure 2: Left: We visualize the average power spectrum for the $X$ and $Y$ motion components extracted from real videos, shown as the blue and green curves. Natural oscillation motions are composed primarily of low-frequency components, and so we use the first $K=16$ terms, marked with red dots. Right: we show a histogram of the amplitude of Fourier terms at $3.0$ Hz after (1) scaling amplitude by image width and height (blue), or (2) frequency adaptive normalization (red). Our adaptive normalization prevents the coefficients from concentrating at extreme values.

图 2：左：我们可视化从真实视频提取的 $X$ 与 $Y$ 运动分量的平均功率谱，分别以蓝、绿曲线标出。自然振荡运动主要由低频分量构成，因此我们使用前 $K=16$ 项，以红点标出。右：我们展示在 $3.0$ Hz 处傅里叶项幅度的直方图，分别对应 (1) 按图像宽高缩放幅度（蓝），或 (2) 频率自适应归一化（红）。我们的自适应归一化防止系数集中在极值。

4.1 Motion representation

4.1 运动表示

Figure 3: Motion prediction module. We predict a spectral volume $\mathcal{S}$ through a frequency-coordinated denoising model. Each block of the diffusion network $\epsilon_{\theta}$ interleaves 2D spatial layers with attention layers (red box, right), and iteratively denoises latent features $z^{n}$. The denoised features are fed to a decoder $\mathcal{D}$ to produce $\mathcal{S}$. During training, we concatenate the downsampled input $I_{0}$ with noisy latent features encoded from a real motion texture via an encoder $\mathcal{E}$, and replace the noisy features with Gaussian noise $z^{N}$ during inference (left).

图 3：运动预测模块。我们通过频率协调去噪模型预测谱体积 $\mathcal{S}$。扩散网络 $\epsilon_{\theta}$ 的每个块将二维空间层与注意力层交错（右侧红框），并迭代去噪潜特征 $z^{n}$。去噪后的特征送入解码器 $\mathcal{D}$ 以产生 $\mathcal{S}$。训练时，我们将下采样的输入 $I_{0}$ 与经编码器 $\mathcal{E}$ 从真实运动纹理编码得到的噪声潜特征拼接；推断时用高斯噪声 $z^{N}$ 替换噪声特征（左）。

Formally, a motion texture is a sequence of time-varying 2D displacement maps $\mathcal{F}=\{F_{t}|t=1,...,T\}$, where the 2D displacement vector $F_{t}(\mathbf{p})$ at each pixel coordinate $\mathbf{p}$ from input image $I_{0}$ defines the position of that pixel at a future time $t$ [20]. To generate a future frame at time $t$, one can splat pixels from $I_{0}$ using the corresponding displacement map $D_{t}$, resulting in a forward-warped image $I^{\prime}_{t}$:

形式上，运动纹理是时变二维位移图序列 $\mathcal{F}=\{F_{t}|t=1,...,T\}$，其中输入图像 $I_{0}$ 每个像素坐标 $\mathbf{p}$ 处的二维位移向量 $F_{t}(\mathbf{p})$ 定义该像素在未来时刻 $t$ 的位置 [20]。要生成时刻 $t$ 的未来帧，可用对应位移图 $D_{t}$ 对 $I_{0}$ 的像素做 splat，得到前向扭曲图像 $I^{\prime}_{t}$：

$$\displaystyle I^{\prime}_{t}(\mathbf{p}+F_{t}(\mathbf{p}))=I_{0}(\mathbf{p}). \tag{1}$$

If our goal is to produce a video via a motion texture, then one choice would be to predict a time-domain motion texture directly from an input image. However, the size of the motion texture would need to scale with the length of the video: generating $T$ output frames implies predicting $T$ displacement fields. To avoid predicting such a large output representation for long videos, many prior animation methods either generate video frames autoregressively [57, 60, 29, 93, 7], or predict each future output frame independently via an extra time embedding [4]. However, neither strategy ensures long-term temporal consistency of generated videos.

若目标是经由运动纹理产生视频，一种选择是直接从输入图像预测时域运动纹理。但运动纹理的大小需随视频长度缩放：生成 $T$ 帧输出意味着预测 $T$ 个位移场。为避免为长视频预测如此大的输出表示，许多先前动画方法要么自回归生成视频帧 [57, 60, 29, 93, 7]，要么通过额外时间嵌入独立预测每一未来输出帧 [4]。但这两种策略都无法保证生成视频的长时程时间一致性。

Fortunately, many natural motions can be described as a superposition of a small number of harmonic oscillators represented with different frequencies, amplitude and phases [20, 25, 69, 50, 23]. Because these underlying motions are quasi-periodic, it is natural to model them in the frequency domain. Hence, we adopt from Davis et al. [23] an efficient frequency space representation of motion in a video called a spectral volume, visualized in Fig. 3. A spectral volume is the temporal Fourier transform of per-pixel trajectories extracted from a video.

幸运的是，许多自然运动可描述为少量谐波振荡器的叠加，各振荡器以不同频率、幅度与相位表示 [20, 25, 69, 50, 23]。由于这些底层运动是准周期的，在频域建模它们是自然的。因此，我们采用 Davis et al. [23] 提出的视频运动高效频域表示——谱体积，见图 3。谱体积是从视频提取的逐像素轨迹的时间傅里叶变换。

Given this motion representation, we formulate the motion prediction problem as a multi-modal image-to-image translation task: from an input image to an output motion spectral volume. We adopt latent diffusion models (LDMs) to generate spectral volumes comprised of a $4K$-channel 2D motion spectrum map, where $K<<T$ is the number of frequencies modeled, and where at each frequency we need four scalars to represent the complex Fourier coefficients for the $x$- and $y$-dimensions. Note that the motion trajectory of a pixel at future time steps $\mathcal{F}(\mathbf{p})=\{F_{t}(\mathbf{p})|t=1,2,...T\}$ and its representation as a spectral volume $\mathcal{S}(\mathbf{p})=\{S_{f_{k}}(\mathbf{p})|k=0,1,..\frac{T}{2}-1\}$ are related by the Fast Fourier transform (FFT):

给定该运动表示，我们将运动预测问题表述为多模态图像到图像翻译任务：从输入图像到输出运动谱体积。我们采用潜扩散模型（LDM）生成由 $4K$ 通道二维运动谱图构成的谱体积，其中 $K<<T$ 为所建模的频率数，且在每个频率上需要四个标量来表示 $x$ 与 $y$ 维的复傅里叶系数。注意，像素在未来时间步的运动轨迹 $\mathcal{F}(\mathbf{p})=\{F_{t}(\mathbf{p})|t=1,2,...T\}$ 与其作为谱体积的表示 $\mathcal{S}(\mathbf{p})=\{S_{f_{k}}(\mathbf{p})|k=0,1,..\frac{T}{2}-1\}$ 由快速傅里叶变换（FFT）关联：

$$\displaystyle\mathcal{S}(\mathbf{p})=\text{FFT}(\mathcal{F}(\mathbf{p})). \tag{2}$$

How should we select the $K$ output frequencies? Prior work in real-time animation has observed that most natural oscillation motions are composed primarily of low-frequency components [25, 69]. To validate this observation, we computed the average power spectrum of the motion extracted from 1,000 randomly sampled 5-second real video clips. As shown in the left plot of Fig. 2, the power spectrum of the motion decreases exponentially with increasing frequency. This suggests that most natural oscillation motions can indeed be well represented by low-frequency terms. In practice, we found that the first $K=16$ Fourier coefficients are sufficient to realistically reproduce the original natural motion in a range of real videos and scenes.

应如何选择 $K$ 个输出频率？实时动画领域的先前工作观察到，大多数自然振荡运动主要由低频分量构成 [25, 69]。为验证该观察，我们计算了从 1,000 段随机采样的 5 秒真实视频片段提取的运动的平均功率谱。如图 2 左图所示，运动功率谱随频率升高近似指数衰减。这表明大多数自然振荡运动确实可由低频项良好表示。实践中，我们发现前 $K=16$ 个傅里叶系数足以在一系列真实视频与场景中逼真复现原始自然运动。

4.2 Predicting motion with a diffusion model

4.2 用扩散模型预测运动

We choose a latent diffusion model (LDM) [74] as the backbone for our motion prediction module, as LDMs are more computationally efficient than pixel-space diffusion models, while preserving synthesis quality. A standard LDM consists of two main modules: (1) a variational autoencoder (VAE) that compresses the input image to a latent space through an encoder $z=E(I)$, then reconstructs the input from the latent features via a decoder $I=D(z)$, and (2) a U-Net based diffusion model that learns to iteratively denoise features starting from Gaussian noise. Our training applies this process not to RGB images but to spectral volumes from real video sequences, which are encoded and then diffused for $n$ steps with a pre-defined variance schedule to produce noisy latents $z^{n}$. The 2D U-Nets are trained to denoise the noisy latents by iteratively estimating the noise $\epsilon_{\theta}(z^{n};n,c)$ used to update the latent feature at each step $n\in(1,2,...,N)$. The training loss for the LDM is written as

我们选择潜扩散模型（LDM） [74] 作为运动预测模块的骨干：相对像素空间扩散模型，LDM 计算更高效，同时保持合成质量。标准 LDM 由两个主要模块组成：(1) 变分自编码器（VAE），经编码器 $z=E(I)$ 将输入图像压缩到潜空间，再经解码器 $I=D(z)$ 从潜特征重建输入；(2) 基于 U-Net 的扩散模型，学习从高斯噪声出发迭代去噪特征。我们的训练不将该过程应用于 RGB 图像，而是应用于来自真实视频序列的谱体积：谱体积被编码，再按预定义方差日程扩散 $n$ 步，得到噪声潜变量 $z^{n}$。二维 U-Net 被训练为通过迭代估计噪声 $\epsilon_{\theta}(z^{n};n,c)$ 来去噪噪声潜变量，并在每一步 $n\in(1,2,...,N)$ 更新潜特征。LDM 的训练损失写为

$$\displaystyle\mathcal{L}_{\text{LDM}}=\mathbb{E}_{n\in\mathcal{U}[1,N],\epsilon^{n}\in\mathcal{N}(0,1)}\left[||\epsilon^{n}-\epsilon_{\theta}(z^{n};n,c)||^{2}\right] \tag{3}$$

where $c$ is the embedding of any conditional signal, such as text, or, in our case, the first frame of the training video sequence, $I_{0}$. The clean latent features $z^{0}$ are then passed through the decoder to recover the spectral volume.

其中 $c$ 是任意条件信号的嵌入，例如文本，或在我们的情形下为训练视频序列的第一帧 $I_{0}$。干净潜特征 $z^{0}$ 再经解码器恢复谱体积。

Frequency adaptive normalization.

One issue we observed is that motion textures have particular distribution characteristics across frequencies. As visualized in the left plot of Fig. 2, the amplitude of the spectral volumes spans a range of 0 to 100 and decays approximately exponentially with increasing frequency. As diffusion models require that the absolute values of the output are between -1 and 1 for stable training and denoising [44], we must normalize the coefficients of $\mathcal{S}$ extracted from real videos before using them for training. If we scale the magnitudes of these coefficients to [0,1] based on the image dimensions as in prior work [29, 77], nearly all the coefficients at higher frequencies will end up close to zero, as shown in the right plot of Fig. 2. Models trained on such data can produce inaccurate motions, since during inference, even small prediction errors can cause large relative errors after denormalization.

频率自适应归一化。

我们观察到的一个问题是：运动纹理在各频率上具有特定的分布特征。如图 2 左图所示，谱体积幅度跨越 0 到 100，并随频率升高近似指数衰减。由于扩散模型要求输出绝对值落在 -1 到 1 之间以稳定训练与去噪 [44]，我们必须在用于训练前归一化从真实视频提取的 $\mathcal{S}$ 系数。若按先前工作 [29, 77] 基于图像尺寸将系数幅度缩放到 [0,1]，几乎所有较高频率处的系数都会接近零，如图 2 右图所示。在此类数据上训练的模型可能产生不准确运动，因为推断时即便很小的预测误差，在反归一化后也会造成很大的相对误差。

To address this issue, we employ a simple but effective frequency adaptive normalization method: First, we independently normalize Fourier coefficients at each frequency based on statistics computed from the training set. Namely, for each individual frequency $f_{j}$, we compute the $95^{\textrm{th}}$ percentile of Fourier coefficient magnitudes over all input samples and use that value as a per-frequency scaling factor $s_{f_{j}}$. We then apply a power transformation to each scaled Fourier coefficient to pull it away from extreme values. In practice, we observe that a square root performs better than other nonlinear transformations such as log or reciprocal. In summary, the final coefficient values of spectral volume $\mathcal{S}(\mathbf{p})$ at frequency $f_{j}$ (used for training our LDM) are computed as

为应对该问题，我们采用简单而有效的频率自适应归一化方法：首先，我们基于训练集统计量，对每个频率独立归一化傅里叶系数。具体地，对每个频率 $f_{j}$，我们计算所有输入样本上傅里叶系数幅度的第 $95$ 百分位，并以该值作为逐频率缩放因子 $s_{f_{j}}$。随后对每个缩放后的傅里叶系数施加幂变换，使其远离极值。实践中，我们观察到平方根优于对数或倒数等其他非线性变换。综上，用于训练我们 LDM 的谱体积 $\mathcal{S}(\mathbf{p})$ 在频率 $f_{j}$ 处的最终系数值计算为

$$\displaystyle S^{\prime}_{f_{j}}(\mathbf{p})=\text{sign}(S_{f_{j}})\sqrt{\left|\frac{S_{f_{j}}(\mathbf{p})}{s_{f_{j}}}\right|}. \tag{4}$$

As shown on the right plot of Fig. 2, after applying frequency adaptive normalization, the spectral volume coefficients distribute more evenly.

如图 2 右图所示，施加频率自适应归一化后，谱体积系数分布更均匀。

Frequency-coordinated denoising.

The straightforward way to predict a spectral volume $\mathcal{S}$ with $K$ frequency bands is to output a tensor of $4K$ channels from a single diffusion U-Net. However, as in prior work [7], we observe that training a model to produce a large number of channels can yield over-smoothed, inaccurate outputs. An alternative would be to independently predict each individual frequency slice by injecting an extra frequency embedding into the LDM [4], but this design choice would result in uncorrelated predictions in the frequency domain, leading to unrealistic motion.

频率协调去噪。

用单一扩散 U-Net 预测含 $K$ 个频带的谱体积 $\mathcal{S}$ 的直接做法，是输出 $4K$ 通道张量。但与先前工作 [7] 一样，我们观察到训练模型产生大量通道会导致过度平滑、不准确的输出。另一种选择是通过向 LDM 注入额外频率嵌入 [4] 独立预测每个频率切片，但该设计选择会造成频域预测互不相关，从而导致不真实运动。

Therefore, inspired by recent video diffusion work [7], we propose a frequency-coordinated denoising strategy, illustrated in Fig. 3. In particular, given an input image $I_{0}$, we first train an LDM $\epsilon_{\theta}$ to predict a single 4-channel frequency slice of spectral volume $S_{f_{j}}$, where we inject an extra frequency embedding along with the time-step embedding into the LDM. We then freeze the parameters of this LDM $\epsilon_{\theta}$, introduce attention layers interleaved with the 2D spatial layers of $\epsilon_{\theta}$ across the $K$ frequency bands, and fine-tune. Specifically, for a batch size $B$, the 2D spatial layers of $\epsilon_{\theta}$ treat the corresponding $B\cdot K$ noisy latent features of channel size $C$ as independent samples with shape $\mathcal{R}^{(B\cdot K)\times C\times H\times W}$. The attention layer then interprets these as consecutive features spanning the frequency axis, and we reshape the latent features from previous 2D spatial layers to $\mathcal{R}^{B\times K\times C\times H\times W}$ before feeding them to the attention layers. In other words, the frequency attention layers are fine-tuned to coordinate all frequency slices so as to produce coherent spectral volumes. In our experiments, we see that the average VAE reconstruction error improves from $0.024$ to $0.018$ when we switch from a single 2D U-Net to a frequency-coordinated denoising module, suggesting an improved upper bound on LDM prediction accuracy; in Sec. 7.3, we also show that this design choice improves video generation quality.

因此，受近期视频扩散工作 [7] 启发，我们提出频率协调去噪策略，见图 3。具体地，给定输入图像 $I_{0}$，我们首先训练 LDM $\epsilon_{\theta}$ 预测谱体积的单个 4 通道频率切片 $S_{f_{j}}$，同时向 LDM 注入额外频率嵌入与时间步嵌入。随后冻结该 LDM $\epsilon_{\theta}$ 的参数，引入与 $\epsilon_{\theta}$ 的二维空间层交错、跨 $K$ 个频带的注意力层，并微调。具体地，对批大小 $B$，$\epsilon_{\theta}$ 的二维空间层将对应的、通道数为 $C$ 的 $B\cdot K$ 个噪声潜特征视为独立样本，形状为 $\mathcal{R}^{(B\cdot K)\times C\times H\times W}$。注意力层再将这些解读为沿频率轴连续的特征，我们在送入注意力层前将先前二维空间层的潜特征重塑为 $\mathcal{R}^{B\times K\times C\times H\times W}$。换言之，频率注意力层经微调以协调所有频率切片，从而产生连贯的谱体积。实验中，从单一二维 U-Net 切换到频率协调去噪模块后，平均 VAE 重建误差从 $0.024$ 改善到 $0.018$，表明 LDM 预测精度的上界有所提升；第 7.3 节我们也表明该设计选择改善了视频生成质量。

5 Image-based rendering

5 基于图像的渲染

We now describe how we take a spectral volume $\mathcal{S}$ predicted for a given input image $I_{0}$ and render a future frame $\hat{I}_{t}$ at time $t$. We first derive a motion texture in the time domain using the inverse temporal FFT applied at each pixel $\mathcal{F}(\mathbf{p})=\text{FFT}^{-1}(\mathcal{S}(\mathbf{p}))$. To produce a future frame $\hat{I}_{t}$, we adopt a deep image-based rendering technique and perform splatting with the predicted motion field $F_{t}$ to forward-warp the encoded $I_{0}$, as shown in Fig. 4. Since forward warping can lead to holes, and multiple source pixels can map to the same output 2D location, we adopt the feature pyramid softmax splatting strategy proposed in prior work on frame interpolation [68].

我们现在描述如何取给定输入图像 $I_{0}$ 预测的谱体积 $\mathcal{S}$，并渲染时刻 $t$ 的未来帧 $\hat{I}_{t}$。我们首先在每个像素上对 $\mathcal{S}$ 施加时间逆 FFT，得到时域运动纹理 $\mathcal{F}(\mathbf{p})=\text{FFT}^{-1}(\mathcal{S}(\mathbf{p}))$。为产生未来帧 $\hat{I}_{t}$，我们采用深度基于图像的渲染技术，并用预测运动场 $F_{t}$ 做 splat，以前向扭曲编码后的 $I_{0}$，见图 4。由于前向扭曲可能导致空洞，且多个源像素可映射到同一输出二维位置，我们采用先前帧插值工作 [68] 提出的特征金字塔 softmax splatting 策略。

Figure 4: Rendering module. We fill in missing content and refine the warped input image using a deep image-based rendering module, where multi-scale features are extracted from the input image $I_{0}$. Softmax splatting is then applied over the features with a motion field $F_{t}$ from time $0$ to $t$ (subject to the weights $W$). The warped features are fed to an image synthesis network to produce the rendered image $\hat{I}_{t}$.

图 4：渲染模块。我们用深度基于图像的渲染模块填补缺失内容并细化扭曲后的输入图像：从输入图像 $I_{0}$ 提取多尺度特征。随后用从时刻 0 到 $t$ 的运动场 $F_{t}$（受权重 $W$ 约束）对特征施加 softmax splatting。扭曲后的特征送入图像合成网络，产生渲染图像 $\hat{I}_{t}$。

Specifically, we encode $I_{0}$ through a feature extractor network to produce a multi-scale feature map. For each individual feature map at scale $j$, we resize and scale the predicted 2D motion field $F_{t}$ according to the resolution. As in Davis et al. [22], we use predicted flow magnitude as a proxy for depth to determine the contributing weight of each source pixel mapped to its destination location. In particular, we compute a per-pixel weight, $W(\mathbf{p})=\frac{1}{T}\sum_{t}||F_{t}(\mathbf{p})||_{2}$ as the average magnitude of the predicted motion texture. In other words, we assume large motions correspond to moving foreground objects, and small or zero motions correspond to background. We use motion-derived weights instead of learnable ones as [46] because we observe that in the single-view case, learnable weights are not effective for addressing disocclusion ambiguities.

具体地，我们经特征提取网络编码 $I_{0}$，产生多尺度特征图。对尺度 $j$ 处的每个特征图，我们按分辨率缩放预测的二维运动场 $F_{t}$。与 Davis et al. [22] 一样，我们用预测光流幅度作为深度代理，以确定映射到目标位置的每个源像素的贡献权重。具体地，我们计算逐像素权重 $W(\mathbf{p})=\frac{1}{T}\sum_{t}||F_{t}(\mathbf{p})||_{2}$，即预测运动纹理的平均幅度。换言之，我们假设大幅运动对应运动的前景物体，小或零运动对应背景。我们使用由运动导出的权重而非如 [46] 的可学习权重，因为我们观察到在单视图情形下，可学习权重对解决去遮挡歧义并不有效。

With the motion field $F_{t}$ and weights $W$, we apply softmax splatting to warp the feature map at each scale to produce a warped feature. The warped features are then injected into the corresponding blocks of an image synthesis decoder to produce a final rendered image $\hat{I}_{t}$.

有了运动场 $F_{t}$ 与权重 $W$，我们对每一尺度的特征图施加 softmax splatting，得到扭曲特征。扭曲特征再注入图像合成解码器的对应块，以产生最终渲染图像 $\hat{I}_{t}$。

We jointly train the feature extractor and synthesis networks with start and target frames $(I_{0},I_{t})$ randomly sampled from real videos, using the estimated flow field from $I_{0}$ to $I_{t}$ to warp encoded features from $I_{0}$, and supervising predictions $\hat{I}_{t}$ against $I_{t}$ with a VGG perceptual loss [49].

我们用从真实视频随机采样的起止帧 $(I_{0},I_{t})$ 联合训练特征提取与合成网络：用从 $I_{0}$ 到 $I_{t}$ 的估计流场扭曲来自 $I_{0}$ 的编码特征，并以 VGG 感知损失 [49] 将预测 $\hat{I}_{t}$ 对照 $I_{t}$ 做监督。

6 Applications

6 应用

Image-to-video.

Our system enables the animation of a single still picture by first predicting a motion spectral volume from the input image and generating an animation by applying our image-based rendering module to the motion texture transformed from the spectral volume. Since we explicitly model scene motion, this allows us to produce slow-motion videos by linearly interpolating the motion texture, or to magnify (or minify) animated motions by adjusting the amplitude of the predicted spectral volume coefficients.

图像到视频。

我们的系统可先从输入图像预测运动谱体积，再对由谱体积变换得到的运动纹理施加基于图像的渲染模块，从而为单张静图做动画。由于我们显式建模场景运动，可通过线性插值运动纹理产生慢动作视频，或通过调整预测谱体积系数的幅度来放大（或缩小）动画运动。

Seamless looping.

Many applications require videos that loop seamlessly, where there is no discontinuity between the start and end of the video. Unfortunately, it is hard to find a large collection of seamlessly looping videos for training. Instead, we devise a method to use our motion diffusion model, trained on regular non-looping video clips, to produce seamless looping video. Inspired by recent work on guidance for image editing [30, 3], our method is a motion self-guidance technique that guides the motion denoising sampling processing using explicit looping constraints. In particular, at each iterative denoising step during inference, we incorporate an additional motion guidance signal alongside standard classifier-free guidance [45], where we enforce each pixel’s position and velocity at the start and end frames to be as similar as possible:

无缝循环。

许多应用需要无缝循环的视频——视频起止之间没有不连续。遗憾的是，很难找到大规模无缝循环视频用于训练。我们转而设计一种方法：用在常规非循环视频片段上训练的运动扩散模型产生无缝循环视频。受图像编辑引导近期工作 [30, 3] 启发，我们的方法是一种运动自引导技术，用显式循环约束引导运动去噪采样过程。具体地，在推断的每次迭代去噪步，我们在标准无分类器引导 [45] 之外纳入额外运动引导信号，强制每个像素在起止帧的位置与速度尽可能相似：

$$\hat{\epsilon}^{n}=(1+w)\epsilon_{\theta}(z^{n};n,c)-w\epsilon_{\theta}(z^{n};n,\emptyset)+u\sigma^{n}\nabla_{z^{n}}\mathcal{L}^{n}{g}\
\mathcal{L}^{n}{g}=||F^{n}{T}-F^{n}{1}||{1}+||\nabla F^{n}{T}-\nabla F^{n}{1}||{1} \tag{5}$$

where $F^{n}_{t}$ is the predicted 2D displacement field at time $t$ and denoising step $n$. $w$ is the classifier-free guidance weight, and $u$ is the motion self-guidance weight. In the supplemental video, we apply a baseline appearance-based looping algorithm [58] to generate a looping video from our non-looping output, and show that our motion self-guidance technique produces seamless looping videos with less distortion and fewer artifacts.

其中 $F^{n}_{t}$ 是时刻 $t$、去噪步 $n$ 处预测的二维位移场。$w$ 是无分类器引导权重，$u$ 是运动自引导权重。在补充视频中，我们将基于外观的循环基线算法 [58] 应用于我们的非循环输出以生成循环视频，并表明我们的运动自引导技术能以更少扭曲与伪影产生无缝循环视频。

Interactive dynamics from a single image.

Davis et al. [22] show that the spectral volume, evaluated at certain resonant frequencies, can approximate an image-space modal basis that is a projection of the vibration modes of the underlying scene (or, more generally, captures spatial and temporal correlations in oscillatory dynamics), and can be used to simulate the object’s response to a user-defined force. We adopt this modal analysis method [22, 70], allowing us to write the image-space 2D motion displacement field for the object’s physical response as a weighted sum of motion spectrum coefficients $S_{f_{j}}$ modulated by the state of complex modal coordinates $\mathbf{q}_{f_{j}}(t)$ at each simulated time step $t$:

来自单张图像的交互动力学。

Davis et al. [22] 表明，在某些共振频率处求值的谱体积可近似图像空间模态基——它是底层场景振动模态的投影（或更一般地，捕捉振荡动力学中的时空相关），并可用于模拟物体对用户定义力的响应。我们采用该模态分析方法 [22, 70]，从而可将物体物理响应的图像空间二维运动位移场写成运动谱系数 $S_{f_{j}}$ 的加权和，并由每个模拟时间步 $t$ 处复模态坐标 $\mathbf{q}_{f_{j}}(t)$ 的状态调制：

$$\displaystyle F_{t}({\mathbf{p}})=\sum_{f_{j}}S_{f_{j}}(\mathbf{p})\mathbf{q}_{f_{j}}(t) \tag{6}$$

We simulate the state of the modal coordinates $\mathbf{q}_{f_{j}}(t)$ via an explicit Euler method applied to the equations of motion for a decoupled mass-spring-damper system represented in modal space [22, 23, 70]. We refer readers to supplementary material and original work for a full derivation. Note that our method produces an interactive scene from a single picture, whereas these prior methods required a video as input.

我们通过显式欧拉法求解模态空间中解耦质量—弹簧—阻尼器系统的运动方程，来模拟模态坐标 $\mathbf{q}_{f_{j}}(t)$ 的状态 [22, 23, 70]。完整推导请参阅补充材料与原作。注意，我们的方法从单张图片产生交互场景，而先前方法需要视频作为输入。

7 Experiments

7 实验

	Image Synthesis	Video Synthesis
Method	FID	KID	FVD	$\text{FVD}_{32}$	DTFVD	$\text{DTFVD}_{32}$
TATS [35]	65.8	1.67	265.6	419.6	22.6	40.7
Stochastic I2V [27]	68.3	3.12	253.5	320.9	16.7	41.7
MCVD [93]	63.4	2.97	208.6	270.4	19.5	53.9
LFDM [67]	47.6	1.70	187.5	254.3	13.0	45.6
DMVFN [48]	37.9	1.09	206.5	316.3	11.2	54.5
Endo et al. [29]	10.4	0.19	166.0	231.6	5.35	65.1
Holynski et al. [46]	11.2	0.20	179.0	253.7	7.23	46.8
Ours	4.03	0.08	47.1	62.9	2.53	6.75

Table 1: Quantitative comparisons on the test set. We report both image synthesis and video synthesis quality. Here, KID is scaled by 100. Lower is better for all error. See Sec. 7.1 for descriptions of baselines and error metrics.

	图像合成	视频合成
方法	FID	KID	FVD	$\text{FVD}_{32}$	DTFVD	$\text{DTFVD}_{32}$
TATS [35]	65.8	1.67	265.6	419.6	22.6	40.7
Stochastic I2V [27]	68.3	3.12	253.5	320.9	16.7	41.7
MCVD [93]	63.4	2.97	208.6	270.4	19.5	53.9
LFDM [67]	47.6	1.70	187.5	254.3	13.0	45.6
DMVFN [48]	37.9	1.09	206.5	316.3	11.2	54.5
Endo et al. [29]	10.4	0.19	166.0	231.6	5.35	65.1
Holynski et al. [46]	11.2	0.20	179.0	253.7	7.23	46.8
Ours	4.03	0.08	47.1	62.9	2.53	6.75

表 1：测试集上的定量比较。我们报告图像合成与视频合成质量。此处 KID 已乘以 100。所有误差越低越好。基线与误差指标说明见第 7.1 节。

I Input image	I Reference	I Stochastic-I2V [27]	I MCVD [93]	I Endo et al. [29]	I Holynski et al. [46]	I Ours

Figure 5: $X$-$t$ slices of videos generated by different approaches. From left to right: input image and corresponding $X$-$t$ video slices from the ground truth video, from videos generated by three baselines [27, 93, 29, 46], and finally videos generated by our approach.

I 输入图像	I 参考	I Stochastic-I2V [27]	I MCVD [93]	I Endo et al. [29]	I Holynski et al. [46]	I 本文

图 5：不同方法生成视频的 $X$-$t$ 切片。从左到右：输入图像与对应真实视频的 $X$-$t$ 切片，三种基线 [27, 93, 29, 46] 生成视频的切片，以及我们方法生成视频的切片。

Implementation details.

We use an LDM [74] as the backbone for predicting spectral volumes, for which we use a VAE with a continuous latent space of dimension $4$. We train the VAE with an $L_{1}$ reconstruction loss, a multi-scale gradient consistency loss [55, 54, 56], and a KL-divergence loss with respective weights of $1,0.2,10^{-6}$. We train the same 2D U-Net used in the original LDM work to perform iterative denosing with a simple MSE loss [44], and adopt the attention layers from [41] for frequency-coordinated denoising. For quantitative evaluation, we train both VAE and LDM on images of size $256\times 160$ from scratch for fair comparisons, and it takes around 6 days to converge using 16 Nvidia A100 GPUs. For our main quantitative and qualitative results, we run the motion diffusion model with DDIM [86] for 250 steps. We also show generated videos of up to a resolution of $512\times 288$, created by fine-tuning a pre-trained image inpainting LDM model [74] on our dataset.

实现细节。

我们用 LDM [74] 作为预测谱体积的骨干，其中 VAE 使用维度为 $4$ 的连续潜空间。我们以 $L_{1}$ 重建损失、多尺度梯度一致性损失 [55, 54, 56] 与 KL 散度损失训练 VAE，权重分别为 $1,0.2,10^{-6}$。我们训练原始 LDM 工作中使用的同一二维 U-Net，以简单 MSE 损失 [44] 做迭代去噪，并采用 [41] 的注意力层做频率协调去噪。定量评估时，我们为公平比较从零训练图像尺寸为 $256\times 160$ 的 VAE 与 LDM，在 16 块 Nvidia A100 GPU 上约 6 天收敛。对主要定量与定性结果，我们以 DDIM [86] 运行运动扩散模型 250 步。我们也展示最高 $512\times 288$ 分辨率的生成视频，通过在我们的数据集上微调预训练图像修复 LDM 模型 [74] 得到。

We adopt ResNet-34 [39] for the feature extractor in our IBR module. Our image synthesis network is based on an architecture for conditional image inpainting [110, 57]. Our rendering module runs in real-time at 25FPS on a Nvidia V100 GPU during inference. We adopt universal guidance [3] to produce seamless looping videos, where we set weights $w=1.75,u=200$, and use 500 DDIM steps with 2 self-recurrence iterations.

我们的 IBR 模块采用 ResNet-34 [39] 作为特征提取器。图像合成网络基于条件图像修复架构 [110, 57]。渲染模块在 Nvidia V100 GPU 上推断时以 25FPS 实时运行。我们采用通用引导 [3] 产生无缝循环视频，权重设为 $w=1.75,u=200$，并用 500 步 DDIM 与 2 次自循环迭代。

Data.

We collect and process a set of 3,015 videos of natural scenes exhibiting oscillatory motions from online sources our own captures. We withhold 10% of videos for testing and use the remainder for training. To extract ground truth motion trajectories, we apply a coarse-to-fine flow method [10, 61] between each selected starting image and every future frame of the video. As training data, we take every 10th video frame as input images and derive the corresponding ground truth spectral volumes using the computed motion trajectories across the following 149 frames. In total, our data consists of over 150K image-motion pairs.

数据。

我们从在线来源与自行采集中收集并处理了 3,015 段呈现振荡运动的自然场景视频。留出 10% 视频用于测试，其余用于训练。为提取真实运动轨迹，我们在每张选定起始图像与视频每个未来帧之间施加由粗到细的光流方法 [10, 61]。作为训练数据，我们每隔 10 帧取一帧作为输入图像，并用随后 149 帧上计算的运动轨迹导出对应真实谱体积。总计数据超过 15 万对图像—运动样本。

Figure 6: Sliding window FID and DTFVD. We show sliding window FID with window size 30 frames, and DTFVD with size 16 frames, for videos generated by different methods.

图 6：滑动窗口 FID 与 DTFVD。我们展示不同方法生成视频的滑动窗口 FID（窗口大小 30 帧）与 DTFVD（窗口大小 16 帧）。

Baselines.

We compare our approach to recent single-image animation and video prediction methods. Endo et al. [29] and DMVFN [48] predict instantaneous 2D motion fields and render future frames auto-regressively. Holynski et al. [46] instead simulates motion through a single static Eulerian motion description. Other recent work such as Stochastic Image-to-Video (Stochastic-I2V) [27], TATS [35], and MCVD [93] adopt VAEs, transformers, or diffusion models to directly predict raw video frames; LFDM [67] generates future frames by predicting flow volumes and warping latents in a diffusion model. We train all the above methods on our data using their respective open-source implementations.[^1]

基线。

我们将方法与近期单图动画与视频预测方法比较。Endo et al. [29] 与 DMVFN [48] 预测瞬时二维运动场并自回归渲染未来帧。Holynski et al. [46] 则通过单一静态欧拉运动描述模拟运动。其他近期工作如 Stochastic Image-to-Video（Stochastic-I2V） [27]、TATS [35] 与 MCVD [93] 采用 VAE、Transformer 或扩散模型直接预测原始视频帧；LFDM [67] 在扩散模型中通过预测流体积并扭曲潜变量来生成未来帧。我们用各自开源实现在我们的数据上训练上述所有方法。[^1]

We evaluate the quality of the videos generated by our approach and by prior baselines in two ways. First, we evaluate the quality of individual synthesized frames using metrics designed for image synthesis tasks. We adopt the Fréchet Inception Distance (FID) [42] and Kernel Inception Distance (KID) [5] to measure the average distance between the distributions of generated frames and ground truth frames.

我们从两方面评估我们方法与先前基线生成视频的质量。首先，用面向图像合成任务的指标评估单帧合成质量。我们采用 Fréchet Inception Distance（FID） [42] 与 Kernel Inception Distance（KID） [5]，度量生成帧分布与真实帧分布之间的平均距离。

Second, to evaluate the quality and temporal coherence of synthesized videos, we adopt the Fréchet Video Distance [92] with window size 16 (FVD) and 32 ($\text{FVD}_{32}$), based on an I3D model [11] trained on the Human Kinetics datasets [52]. To more faithfully reflect synthesis quality for the natural oscillation motions we seek to generate, we also adopt the Dynamic Texture Frechet Video Distance [27], which measures distance from videos with window size 16 (DTFVD) and size 32 ($\text{DTFVD}_{32}$), using a I3D model trained on the Dynamic Textures Database [37], a dataset consisting primarily of natural motion textures.

其次，为评估合成视频的质量与时间连贯性，我们采用基于在 Human Kinetics 数据集 [52] 上训练的 I3D 模型 [11] 的 Fréchet Video Distance [92]，窗口大小分别为 16（FVD）与 32（$\text{FVD}_{32}$）。为更忠实地反映我们希望生成的自然振荡运动的合成质量，我们还采用 Dynamic Texture Fréchet Video Distance [27]，用在 Dynamic Textures Database [37]（主要由自然运动纹理构成）上训练的 I3D 模型，度量窗口大小 16（DTFVD）与 32（$\text{DTFVD}_{32}$）的视频距离。

We further use a sliding window FID of window size 30 frames, and a sliding window DTFVD with window size 16 frames, as in [60, 57], to measure how generated video quality degrades over time. For all methods, we evaluate metrics at $256\times 128$ resolution by center-cropping.

我们进一步使用窗口大小 30 帧的滑动窗口 FID，以及窗口大小 16 帧的滑动窗口 DTFVD，如 [60, 57]，以度量生成视频质量随时间如何退化。对所有方法，我们通过中心裁剪在 $256\times 128$ 分辨率上评估指标。

	Image Synthesis	Video Synthesis
Method	FID	KID	FVD	$\text{FVD}_{32}$	DTFVD	$\text{DTFVD}_{32}$
Repeat $I_{0}$	-	-	237.5	316.7	5.30	45.6
$K=4$	3.92	0.07	60.3	78.4	3.12	8.59
$K=8$	3.95	0.07	52.1	68.7	2.71	7.37
$K=24$	4.09	0.08	48.2	65.1	2.50	6.94
w/o adaptive norm.	4.53	0.09	62.7	80.1	3.16	8.19
Independent pred.	4.00	0.08	52.5	71.3	2.70	7.40
Volume pred.	4.74	0.09	53.7	71.1	2.83	7.79
Baseline splat [46]	4.25	0.09	49.5	66.8	2.83	7.27
Full ($K=16$)	4.03	0.08	47.1	62.9	2.53	6.75

Table 2: Ablation study. Sec. 7.3 describes each configuration.

	图像合成	视频合成
方法	FID	KID	FVD	$\text{FVD}_{32}$	DTFVD	$\text{DTFVD}_{32}$
Repeat $I_{0}$	-	-	237.5	316.7	5.30	45.6
$K=4$	3.92	0.07	60.3	78.4	3.12	8.59
$K=8$	3.95	0.07	52.1	68.7	2.71	7.37
$K=24$	4.09	0.08	48.2	65.1	2.50	6.94
w/o adaptive norm.	4.53	0.09	62.7	80.1	3.16	8.19
Independent pred.	4.00	0.08	52.5	71.3	2.70	7.40
Volume pred.	4.74	0.09	53.7	71.1	2.83	7.79
Baseline splat [46]	4.25	0.09	49.5	66.8	2.83	7.27
Full ($K=16$)	4.03	0.08	47.1	62.9	2.53	6.75

表 2：消融研究。第 7.3 节描述各配置。

7.1 Quantitative results

7.1 定量结果

Table 1 shows quantitative comparisons between our approach and baselines on our test set. Our approach significantly outperforms prior single-image animation baselines in terms of both image and video synthesis quality. Specifically, our much lower FVD and DT-FVD distances suggest that the videos generated by our approach are more realistic and more temporally coherent. Further, Fig. 6 shows the sliding window FID and sliding window DT-FVD distances of generated videos from different methods. Thanks to the global spectral volume representation, videos generated by our approach do not suffer from degradation over time.

表 1 展示我们方法与基线在测试集上的定量比较。在图像与视频合成质量上，我们的方法显著优于先前单图动画基线。具体地，我们低得多的 FVD 与 DT-FVD 距离表明，我们方法生成的视频更逼真、时间上更连贯。此外，图 6 展示不同方法生成视频的滑动窗口 FID 与滑动窗口 DT-FVD 距离。得益于全局谱体积表示，我们方法生成的视频不会随时间退化。

7.2 Qualitative results

7.2 定性结果

We visualize qualitative comparisons between videos as spatio-temporal $X$-$t$ slices of the generated videos, a standard way of visualizing small motions in a video [95]. As shown in Fig. 5, our generated video dynamics more strongly resemble the motion patterns observed in the corresponding real reference videos (second column), compared to other methods. Baselines such as Stochastic I2V [27] and MCVD [93] fail to model both appearance and motion realistically over time. Endo et al. [29] and Holynski et al. [46] produce video frames with fewer artifacts but exhibits over-smooth or non-oscillatory motion over time. We refer readers to supplementary material to assess the quality of generated video frames and estimated motion across different methods.

我们以生成视频的时空 $X$-$t$ 切片可视化定性比较，这是可视化视频中小运动的标准方式 [95]。如图 5 所示，相对其他方法，我们生成的视频动力学更强烈地类似于对应真实参考视频（第二列）中观测到的运动模式。Stochastic I2V [27] 与 MCVD [93] 等基线无法随时间同时逼真地建模外观与运动。Endo et al. [29] 与 Holynski et al. [46] 产生伪影更少的视频帧，但随时间呈现过度平滑或非振荡运动。请参阅补充材料以评估不同方法生成视频帧与估计运动的质量。

I Input	I AnimateDiff	I ModelScope	I GEN-2

Figure 7: We show generated future frames from three recent large video diffusion models [36, 98, 31].

I 输入	I AnimateDiff	I ModelScope	I GEN-2

图 7：我们展示三种近期大型视频扩散模型 [36, 98, 31] 生成的未来帧。

7.3 Ablation study

7.3 消融研究

We conduct an ablation study to validate the major design choices in our motion prediction and rendering modules, comparing our full configuration with different variants. Specifically, we evaluate results using different numbers of frequency bands $K=4,8,16,24$. We observe that increasing the number of frequency bands improves video prediction quality, but the improvement is marginal with more than 16 frequencies. Next, we remove adaptive frequency normalization from the ground truth spectral volumes, and instead just scale them based on input image width and height (w/o adaptive norm.). Additionally, we remove the frequency coordinated-denoising module (Independent pred.), or replace it with a simpler DM where a tensor volume of $4K$ channel spectral volumes are predicted jointly via a single 2D U-net diffusion model (Volume pred.). Finally, we compare results where we use a baseline rendering method that applies softmax splatting over single-scale features subject to learnable weights as used by [46] (Baseline splat). We also add a baseline where the generated video is a volume by repeating input image $N$ times (Repeat $I_{0}$). From Table 2, we observe that all simpler or alternative configurations lead to worse performance compared with our full model.

我们进行消融研究，以验证运动预测与渲染模块中的主要设计选择，将完整配置与不同变体比较。具体地，我们用不同频带数 $K=4,8,16,24$ 评估结果。我们观察到增加频带数可改善视频预测质量，但超过 16 个频率后改善变得边际。接着，我们从真实谱体积中去掉自适应频率归一化，仅按输入图像宽高缩放（w/o adaptive norm.）。此外，我们去掉频率协调去噪模块（Independent pred.），或以更简单的 DM 替换——经由单一二维 U-Net 扩散模型联合预测 $4K$ 通道谱体积张量（Volume pred.）。最后，我们比较采用基线渲染方法的结果：对单尺度特征按可学习权重施加 softmax splatting，如 [46]（Baseline splat）。我们也加入将输入图像重复 $N$ 次构成体积的基线（Repeat $I_{0}$）。由表 2 可见，所有更简单或替代配置相对我们完整模型均导致更差表现。

7.4 Comparing to large video models

7.4 与大型视频模型比较

We further perform a user study and compare our generated animations with ones from recent large video diffusion models: AnimateDiff [36], ModelScope [98] and Gen-2 [31], which predict video volumes directly. On a randomly selected 30 videos from the test set, we ask users “which video is more realistic?”. Users report a 80.9% preference for our approach over others. Moreover, as shown in Fig. 7, we observe that the generated videos from these baselines are either unable to adhere to the input image content, or exhibit gradual color drift and distortion over time. We refer readers to the supplementary material for full comparisons.

我们进一步做用户研究，将生成动画与近期大型视频扩散模型比较：AnimateDiff [36]、ModelScope [98] 与 Gen-2 [31]，它们直接预测视频体积。在测试集随机选取的 30 段视频上，我们询问用户「哪段视频更逼真？」。用户报告相对其他方法偏好我们方法的比例为 80.9%。此外，如图 7 所示，我们观察到这些基线生成的视频要么无法贴合输入图像内容，要么随时间呈现逐渐的颜色漂移与扭曲。完整比较请参阅补充材料。

8 Discussion and conclusion

8 讨论与结论

Figure 8: Limitations. We show examples of rendered future frames (even), and overlay of input and rendered images (odd). Our method can produce artifacts in regions of thin objects or large motions, and regions requiring filling large amount of new contents.

图 8：局限。我们展示渲染未来帧（偶数列）以及输入与渲染图像的叠加（奇数列）示例。我们的方法可能在细薄物体或大幅运动区域，以及需要填补大量新内容的区域产生伪影。

Limitations.

Since our approach only predicts lower frequencies of a spectral volume, it can fail to model non-oscillating motions or high-frequency vibrations—this may be resolved by using learned motion bases. Furthermore, the quality of generated videos relies on the quality of underlying motion trajectories, which may degrade in scenes with thin moving objects or objects with large displacements. Even if correct, motions that require generating large amounts of new unseen content may also cause degradations (Fig. 8).

局限。

由于我们的方法仅预测谱体积的较低频率，可能无法建模非振荡运动或高频振动——这或许可通过使用学习的运动基来解决。此外，生成视频的质量依赖底层运动轨迹的质量，在含细薄运动物体或大幅位移物体的场景中可能退化。即便运动正确，需要生成大量未见新内容的运动也可能造成退化（图 8）。

Conclusion.

We present a new approach for modeling natural oscillation dynamics from a single still picture. Our image-space motion prior is represented with spectral volumes, a frequency representation of per-pixel motion trajectories, which we find to be efficient and effective for prediction with diffusion models, and which we learn from collections of real world videos. Spectral volumes are predicted using frequency-coordinated latent diffusion model and are used to animate future video frames through an image-based rendering module. We show that our approach produces photo-realistic animations from a single picture and significantly outperforms prior baselines, and that it can enable several downstream applications such as creating seamlessly looping or interactive image dynamics.

结论。

我们提出从单张静图建模自然振荡动力学的新方法。我们的图像空间运动先验以谱体积表示——逐像素运动轨迹的频率表示；我们发现其对用扩散模型做预测高效且有效，并从真实世界视频集合中学习。谱体积用频率协调的潜扩散模型预测，并经由基于图像的渲染模块驱动未来视频帧动画。我们表明该方法从单张图片产生照片级真实动画，显著优于先前基线，并能支持多项下游应用，例如创建无缝循环或交互式图像动力学。

Acknowledgements.

We thank Abe Davis, Rick Szeliski, Andrew Liu, Boyang Deng, Qianqian Wang, Xuan Luo, and Lucy Chai for fruitful discussions and helpful comments.

致谢。

我们感谢 Abe Davis、Rick Szeliski、Andrew Liu、Boyang Deng、Qianqian Wang、Xuan Luo 与 Lucy Chai 富有成果的讨论与有益意见。

References

参考文献

[1] Aseem Agarwala, Ke Colin Zheng, Chris Pal, Maneesh Agrawala, Michael Cohen, Brian Curless, David Salesin, and Richard Szeliski. Panoramic video textures. In ACM Trans. Graphics (SIGGRAPH) , pages 821–827. 2005.

[2] Hyemin Ahn, Esteve Valls Mascaro, and Dongheui Lee. Can we use diffusion probabilistic models for 3d motion prediction? arXiv preprint arXiv:2302.14503 , 2023.

[3] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 843–852, 2023.

[4] Hugo Bertiche, Niloy J Mitra, Kuldeep Kulkarni, Chun-Hao P Huang, Tuanfeng Y Wang, Meysam Madadi, Sergio Escalera, and Duygu Ceylan. Blowing in the wind: Cyclenet for human cinemagraphs from still images. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 459–468, 2023.

[5] Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. arXiv preprint arXiv:1801.01401 , 2018.

[6] Andreas Blattmann, Timo Milbich, Michael Dorkenwald, and Björn Ommer. ipoke: Poking a still image for controlled stochastic video synthesis. In Proc. Int. Conf. on Computer Vision (ICCV) , pages 14707–14717, 2021.

[7] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 22563–22575, 2023.

[8] Richard Strong Bowen, Richard Tucker, Ramin Zabih, and Noah Snavely. Dimensions of motion: Monocular prediction through flow subspaces. In International Conference on 3D Vision (3DV) , pages 454–464, 2022.

[9] Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei Efros, and Tero Karras. Generating long videos of dynamic scenes. Neural Information Processing Systems , 35:31769–31781, 2022.

[10] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. High accuracy optical flow estimation based on a theory for warping. In Proc. European Conf. on Computer Vision (ECCV) , pages 25–36, 2004.

[11] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 6299–6308, 2017.

[12] Dan Casas, Marco Volino, John Collomosse, and Adrian Hilton. 4d video textures for interactive character appearance. In Computer Graphics Forum , volume 33, pages 371–380. Wiley Online Library, 2014.

[13] Antoni B Chan and Nuno Vasconcelos. Mixtures of dynamic textures. In Proc. Int. Conf. on Computer Vision (ICCV) , pages 641–647, 2005.

[14] Antoni B Chan and Nuno Vasconcelos. Classifying video with kernel dynamic textures. In Proc. Computer Vision and Pattern Recognition (CVPR) , 2007.

[15] Antoni B Chan and Nuno Vasconcelos. Modeling, clustering, and segmenting video with mixtures of dynamic textures. Trans. Pattern Analysis and Machine Intelligence , 30(5):909–926, 2008.

[16] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023.

[17] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 11315–11325, 2022.

[18] Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. Motion-conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304.14404 , 2023.

[19] Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 18000–18010, 2023.

[20] Yung-Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David H Salesin, and Richard Szeliski. Animating pictures with stochastic motion textures. In ACM Trans. Graphics (SIGGRAPH) , pages 853–860, 2005.

[21] Vincent C Couture, Michael S Langer, and Sebastien Roy. Omnistereo video textures without ghosting. In International Conference on 3D Vision , pages 64–70. IEEE, 2013.

[22] Abe Davis, Justin G Chen, and Frédo Durand. Image-space modal bases for plausible manipulation of objects in video. ACM Trans. Graphics (SIGGRAPH) , 34(6):1–7, 2015.

[23] Myers Abraham Davis. Visual vibration analysis . PhD thesis, Massachusetts Institute of Technology, 2016.

[24] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Neural Information Processing Systems , 34:8780–8794, 2021.

[25] Julien Diener, Mathieu Rodriguez, Lionel Baboud, and Lionel Reveret. Wind projection basis for real-time animation of trees. In Computer graphics forum , volume 28, pages 533–540. Wiley Online Library, 2009.

[26] Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, and Stefano Soatto. Dynamic textures. 51:91–109, 2003.

[27] Michael Dorkenwald, Timo Milbich, Andreas Blattmann, Robin Rombach, Konstantinos G. Derpanis, and Bjorn Ommer. Stochastic image-to-video synthesis using cinns. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 3742–3753, June 2021.

[28] Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, and Artsiom Sanakoyeu. Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 481–490, 2023.

[29] Yuki Endo, Yoshihiro Kanamori, and Shigeru Kuriyama. Animating landscape: Self-supervised learning of decoupled motion and appearance for single-image video synthesis. ACM Trans. Graphics (SIGGRAPH Asia) , 38(6):175:1–175:19, 2019.

[30] Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. arXiv preprint arXiv:2306.00986 , 2023.

[31] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proc. Int. Conf. on Computer Vision (ICCV) , pages 7346–7356, 2023.

[32] Matthew Flagg, Atsushi Nakazawa, Qiushuang Zhang, Sing Bing Kang, Young Kee Ryu, Irfan Essa, and James M Rehg. Human video textures. In Proceedings of the 2009 symposium on Interactive 3D graphics and games , pages 199–206, 2009.

[33] Jean-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning , pages 3233–3246. PMLR, 2020.

[34] Ruohan Gao, Bo Xiong, and Kristen Grauman. Im2Flow: Motion hallucination from static images for action recognition. In Proc. Computer Vision and Pattern Recognition (CVPR) , 2018.

[35] Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.03638 , 2022.

[36] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023.

[37] Isma Hadji and Richard P Wildes. A new large scale dynamic texture dataset with application to convnet understanding. In Proc. European Conf. on Computer Vision (ECCV) , pages 320–335, 2018.

[38] Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajectories. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 7854–7863, 2018.

[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 770–778, 2016.

[40] Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, et al. Animate-a-story: Storytelling with retrieval-augmented video generation. arXiv preprint arXiv:2307.06940 , 2023.

[41] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221 , 2022.

[42] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Neural Information Processing Systems , 30, 2017.

[43] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

[44] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems , 33:6840–6851, 2020.

[45] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

[46] Aleksander Holynski, Brian L Curless, Steven M Seitz, and Richard Szeliski. Animating pictures with Eulerian motion fields. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 5810–5819, 2021.

[47] Tobias Hoppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. Trans. Mach. Learn. Res. , 2022, 2022.

[48] Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, and Shuchang Zhou. A dynamic multi-scale voxel flow network for video prediction. ArXiv , abs/2303.09875, 2023.

[49] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proc. European Conf. on Computer Vision (ECCV) , pages 694–711, 2016.

[50] Hitoshi Kanda and Jun Ohya. Efficient, realistic method for animating dynamic behaviors of 3d botanical trees. In International Conference on Multimedia and Expo , volume 2, pages II–89. IEEE, 2003.

[51] Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman. Dreampose: Fashion image-to-video synthesis via stable diffusion. arXiv preprint arXiv:2304.06025 , 2023.

[52] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

[53] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523 , 2018.

[54] Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T Freeman. Learning the depths of moving people by watching frozen people. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 4521–4530, 2019.

[55] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 2041–2050, 2018.

[56] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 4273–4284, 2023.

[57] Zhengqi Li, Qianqian Wang, Noah Snavely, and Angjoo Kanazawa. Infinitenature-zero: Learning perpetual view generation of natural scenes from single images. In Proc. European Conf. on Computer Vision (ECCV) , pages 515–534. Springer, 2022.

[58] Jing Liao, Mark Finch, and Hugues Hoppe. Fast computation of seamless video loops. ACM Trans. Graphics (SIGGRAPH) , 34(6):1–10, 2015.

[59] Zicheng Liao, Neel Joshi, and Hugues Hoppe. Automated video looping with progressive dynamism. ACM Transactions on Graphics (TOG) , 32(4):1–10, 2013.

[60] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proc. Int. Conf. on Computer Vision (ICCV) , pages 14458–14467, 2021.

[61] Ce Liu. Beyond pixels: exploring new representations and applications for motion analysis . PhD thesis, Massachusetts Institute of Technology, 2009.

[62] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 10209–10218, 2023.

[63] Aniruddha Mahapatra and Kuldeep Kulkarni. Controllable animation of fluid elements in still images. In Proc. Computer Vision and Pattern Recognition (CVPR) , 2022.

[64] Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Lee, Sergey Tulyakov, and Jun-Yan Zhu. Text-guided synthesis of eulerian cinemagraphs. 2023.

[65] Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu. Implicit warping for animation with image sets. Neural Information Processing Systems , 35:22438–22450, 2022.

[66] Medhini Narasimhan, Shiry Ginosar, Andrew Owens, Alexei A Efros, and Trevor Darrell. Strumming to the beat: Audio-conditioned contrastive video textures. In Proc. Winter Conference on Applications of Computer Vision , pages 3761–3770, 2022.

[67] Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 18444–18455, 2023.

[68] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 5437–5446, 2020.

[69] Shin Ota, Machiko Tamura, Kunihiko Fujita, T Fujimoto, K Muraoka, and Norishige Chiba. 1/f/sup/spl beta//noise-based real-time animation of trees swaying in wind fields. In Proceedings Computer Graphics International , pages 52–59. IEEE, 2003.

[70] Automne Petitjean, Yohan Poirier-Ginter, Ayush Tewari, Guillaume Cordonnier, and George Drettakis. Modalnerf: Neural modal analysis and synthesis for free-viewpoint navigation in dynamically vibrating scenes. In Computer Graphics Forum , volume 42, 2023.

[71] Silvia L. Pintea, Jan C. van Gemert, and Arnold W. M. Smeulders. Déjà vu: Motion prediction in static images. In Proc. European Conf. on Computer Vision (ECCV) , 2014.

[72] Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H Bermano, and Daniel Cohen-Or. Single motion diffusion. arXiv preprint arXiv:2302.05905 , 2023.

[73] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

[74] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 10684–10695, 2022.

[75] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Neural Information Processing Systems , 35:36479–36494, 2022.

[76] Payam Saisan, Gianfranco Doretto, Ying Nian Wu, and Stefano Soatto. Dynamic texture recognition. In Proc. Computer Vision and Pattern Recognition (CVPR) , 2001.

[77] Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J. Fleet. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation, 2023.

[78] Arno Schödl, Richard Szeliski, David H Salesin, and Irfan Essa. Video textures. In ACM Trans. Graphics (SIGGRAPH) , pages 489–498, 2000.

[79] Mikio Shinya and Alain Fournier. Stochastic motion—motion under the influence of wind. Computer Graphics Forum , 11(3), 1992.

[80] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 2377–2386, 2019.

[81] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Neural Information Processing Systems , 32, 2019.

[82] Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 13653–13662, 2021.

[83] Chen Qian Kwan-Yee Lin Hongsheng Li Siming Fan, Jingtan Piao. Simulating fluids in real-world still images. arXiv preprint , arXiv:2204.11335, 2022.

[84] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 3626–3636, 2022.

[85] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256–2265. PMLR, 2015.

[86] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502 , October 2020.

[87] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

[88] Jos Stam. Multi-scale stochastic modelling of complex natural phenomena . PhD thesis, 1995.

[89] Jos Stam. Stochastic dynamics: Simulating the effects of turbulence on flexible structures. Computer Graphics Forum , 16(3), 1997.

[90] Ryusuke Sugimoto, Mingming He, Jing Liao, and Pedro V Sander. Water simulation and rendering from a still photograph. In ACM Trans. Graphics (SIGGRAPH Asia) , pages 1–9, 2022.

[91] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916 , 2022.

[92] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 , 2018.

[93] Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal. Mcvd: Masked conditional video diffusion for prediction, generation, and interpolation. In Neural Information Processing Systems , 2022.

[94] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Neural Information Processing Systems , 2016.

[95] Neal Wadhwa, Michael Rubinstein, Frédo Durand, and William T Freeman. Phase-based video motion processing. ACM Trans. Graphics (SIGGRAPH) , 32(4):1–10, 2013.

[96] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In Proc. European Conf. on Computer Vision (ECCV) , 2016.

[97] Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense optical flow prediction from a static image. In Proc. Int. Conf. on Computer Vision (ICCV) , pages 2443–2451, 2015.

[98] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02018 , 2023.

[99] Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. arXiv preprint arXiv:2203.09043 , 2022.

[100] Frederik Warburg, Ethan Weber, Matthew Tancik, Aleksander Holynski, and Angjoo Kanazawa. Nerfbusters: Removing ghostly artifacts from casually captured nerfs. arXiv preprint arXiv:2304.10532 , 2023.

[101] Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628 , 2022.

[102] Chung-Yi Weng, Brian Curless, and Ira Kemelmacher-Shlizerman. Photo wake-up: 3d character animation from a single photo. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 5908–5917, 2019.

[103] Jamie Wynn and Daniyar Turmukhambetov. DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models. In Proc. Computer Vision and Pattern Recognition (CVPR) , 2023.

[104] Tianfan Xue, Jiajun Wu, Katherine L Bouman, and William T Freeman. Visual dynamics: Stochastic future generation via layered cross convolutional networks. Trans. Pattern Analysis and Machine Intelligence , 41(9):2236–2250, 2019.

[105] Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089 , 2023.

[106] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 18456–18466, 2023.

[107] Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmented motion diffusion model. arXiv preprint arXiv:2304.01116 , 2023.

[108] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077 , 2023.

[109] Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proc. Computer Vision and Pattern Recognition (CVPR) , pages 3657–3666, 2022.

[110] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In International Conference on Learning Representations (ICLR) , 2021.

[111] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 , 2022.