VideoPoet: A Large Language Model for Zero-Shot Video Generation

VideoPoet：面向零样本视频生成的大语言模型

Dan Kondratyuk Affiliation:  Google    Lijun Yu Affiliation:  Google Affiliation:  Carnegie Mellon University Correspondence to: lijuny@google.com    Xiuye Gu Affiliation:  Google    José Lezama Affiliation:  Google    Jonathan Huang Affiliation:  Google Correspondence to: jonathanhuang@google.com    Grant Schindler Affiliation:  Google    Rachel Hornung Affiliation:  Google    Vighnesh Birodkar Affiliation:  Google    Jimmy Yan Affiliation:  Google    Ming-Chang Chiu Affiliation:  Google    Krishna Somandepalli Affiliation:  Google    Hassan Akbari Affiliation:  Google    Yair Alon Affiliation:  Google    Yong Cheng Affiliation:  Google    Josh Dillon Affiliation:  Google    Agrim Gupta Affiliation:  Google    Meera Hahn Affiliation:  Google    Anja Hauth Affiliation:  Google    David Hendon Affiliation:  Google    Alonso Martinez Affiliation:  Google    David Minnen Affiliation:  Google    Mikhail Sirotenko Affiliation:  Google    Kihyuk Sohn Affiliation:  Google    Xuan Yang Affiliation:  Google    Hartwig Adam Affiliation:  Google    Ming-Hsuan Yang Affiliation:  Google    Irfan Essa Affiliation:  Google    Huisheng Wang Affiliation:  Google    David A. Ross Affiliation:  Google    Bryan Seybold Affiliation:  Google Correspondence to: seybold@google.com    Lu Jiang Affiliation:  Google Affiliation:  Carnegie Mellon University Correspondence to: roadjiang@gmail.com Correspondence to: dross@google.com

Abstract

摘要

We present VideoPoet, a model for synthesizing high-quality videos from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs – including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves as a foundation that is adapted to a range of video generation tasks. We present results demonstrating the model’s state-of-the-art capabilities in zero-shot video generation, specifically highlighting the generation of high-fidelity motions. Project page: https://sites.research.google/videopoet/.

我们提出 VideoPoet：一个能从多种条件信号合成高质量视频的模型。它采用仅解码器 Transformer 架构，处理图像、视频、文本与音频等多模态输入。训练流程遵循大语言模型（LLM）惯例，分为预训练与任务适配两阶段。预训练阶段，VideoPoet 在自回归 Transformer 框架内混合多种多模态生成目标；预训练得到的 LLM 作为基础模型，再适配到一系列视频生成任务。实验表明，该模型在零样本视频生成上达到先进水平，尤其擅长生成高保真运动。项目页：https://sites.research.google/videopoet/。

Keywords:

Figure 1: VideoPoet Overview: a versatile video generator that conditions on multiple types of inputs and performs a variety of video generation tasks.

关键词：

图 1：VideoPoet 概览：一个可接受多种输入条件、完成多种视频生成任务的通用视频生成器。

1 Introduction

1 引言

Recently, there has been a surge of generative video models capable of a variety of video creation tasks. These include text-to-video (Zhang et al., 2023a; Singer et al., 2022), image-to-video (Yu et al., 2023c), video-to-video stylization (Chen et al., 2023b; Chai et al., 2023; Voleti et al., 2022), and video editing (Ceylan et al., 2023; Wang et al., 2023b; Geyer et al., 2023) among other video applications. Most existing models employ diffusion-based methods for video generation. These video models typically start with a pretrained image model, such as Stable Diffusion (Rombach et al., 2022; Podell et al., 2023), that produces high-fidelity images for individual frames, and then fine-tune the model to improve temporal consistency across video frames.

近来涌现了一批能完成多种视频创作任务的生成式视频模型，包括文生视频 (Zhang et al., 2023a; Singer et al., 2022)、图生视频 (Yu et al., 2023c)、视频到视频风格化 (Chen et al., 2023b; Chai et al., 2023; Voleti et al., 2022) 与视频编辑 (Ceylan et al., 2023; Wang et al., 2023b; Geyer et al., 2023) 等。多数现有模型采用基于扩散的视频生成方法：通常从能产出高保真单帧的预训练图像模型出发，如 Stable Diffusion (Rombach et al., 2022; Podell et al., 2023)，再微调以改善跨帧时间一致性。

While Large Language Models (LLMs) are commonly used as foundation models across various modalities including language (Brown et al., 2020), code (Li et al., 2023; OpenAI, 2023), audio (Rubenstein et al., 2023), speech (Agostinelli et al., 2023), and robotics (Driess et al., 2023; Zitkovich et al., 2023), the diffusion model remains the predominant approach for video generation. Although early research has demonstrated the effectiveness of LLMs in text-to-image generation, e.g., DALL-E (Ramesh et al., 2022), Parti (Yu et al., 2022) and (Ding et al., 2021), and text-to-video, e.g., CogVideo (Hong et al., 2022)), language models have not reached a level of quality on par with video diffusion models in tasks like text-to-video generation as shown in previous studies (Nash et al., 2022; Villegas et al., 2022). In contrast to training exclusively for text-to-video tasks, the generative model of LLMs in the language domain emphasizes a large pretraining stage to learn a foundation (Bommasani et al., 2021) by examining pretraining tasks that extend beyond text-to-video generation.

大语言模型（LLM）已广泛用作语言 (Brown et al., 2020)、代码 (Li et al., 2023; OpenAI, 2023)、音频 (Rubenstein et al., 2023)、语音 (Agostinelli et al., 2023) 与机器人 (Driess et al., 2023; Zitkovich et al., 2023) 等多模态领域的基础模型，但视频生成仍以扩散模型为主。早期研究已表明 LLM 在文生图上有效，例如 DALL-E (Ramesh et al., 2022)、Parti (Yu et al., 2022) 与 (Ding et al., 2021)，以及文生视频如 CogVideo (Hong et al., 2022)；然而如既往研究所示 (Nash et al., 2022; Villegas et al., 2022)，语言模型在文生视频等任务上的质量尚未与视频扩散模型持平。与只为文生视频训练不同，语言领域的 LLM 生成范式强调大规模预训练，通过审视超出文生视频的预训练任务来学习基础能力 (Bommasani et al., 2021)。

A notable advantage of employing LLMs in video generation lies in the ease of integrating existing LLM frameworks. This integration allows for reusing LLM infrastructure and leverages the optimizations our community has developed over many years for LLMs, including optimizations in learning recipes for model scaling (Brown et al., 2020; Chowdhery et al., 2022), training and inference infrastructure (Du et al., 2022), hardware, among other advancements. This couples with their flexibility in encoding many diverse tasks in the same model (Raffel et al., 2020), which stands in contrast to most diffusion models where architectural changes and adapter modules are the dominant approach used to adapt the model to more diverse tasks (Zhang et al., 2023b).

在视频生成中使用 LLM 的一个显著优点，是易于接入现有 LLM 框架：可复用社区多年积累的基础设施与优化，包括模型缩放的学习配方 (Brown et al., 2020; Chowdhery et al., 2022)、训练与推理基础设施 (Du et al., 2022)、硬件等。这与它们能在同一模型中编码多样任务的灵活性 (Raffel et al., 2020) 相结合；相对地，多数扩散模型仍主要靠改架构与适配器模块来扩展任务 (Zhang et al., 2023b)。

In this paper, we exploit language models for video generation, following the canonical training protocols of LLMs in the language domain. We introduce VideoPoet, a language model for video generation. VideoPoet employs a decoder-only LLM architecture (Anil et al., 2023; OpenAI, 2023) that admits image, video, and audio modalities as discrete tokens, each produced by their respective tokenizer.

本文沿语言领域 LLM 的标准训练协议，探索用语言模型做视频生成。我们提出 VideoPoet：面向视频生成的语言模型。它采用仅解码器 LLM 架构 (Anil et al., 2023; OpenAI, 2023)，将图像、视频与音频模态接纳为各自分词器产出的离散词元。

The training process of VideoPoet consists of two stages: (1) pretraining and (2) task-adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal pretraining objectives within an autoregressive transformer framework. After pretraining, the model functions as a versatile multi-task video generation model such as text-to-video, image-to-video, video editing and video-to-video stylization. These capabilities are inherently integrated into a single LLM, rather than relying on a separate generative model controlled by text prompts (Tang et al., 2023). During subsequent task-adaptation, the pretrained model can be further fine-tuned either to enhance its generation quality on the training tasks or to perform new tasks.

VideoPoet 的训练分两阶段：(1) 预训练；(2) 任务适配。预训练阶段，在自回归 Transformer 框架内混合多种多模态预训练目标。预训练后，模型即可作为通用多任务视频生成器，覆盖文生视频、图生视频、视频编辑与视频到视频风格化等；这些能力内嵌于单一 LLM，而非依赖由文本提示控制的独立生成模型 (Tang et al., 2023)。随后的任务适配阶段，可进一步微调预训练模型以提升训练任务上的生成质量，或学习新任务。

Experiments show VideoPoet’s state-of-the-art capabilities in generating videos with large and high-fidelity motions. With the powerful capabilities of the transformer architecture, VideoPoet can be straightforwardly trained on a multi-task, multimodal generative objective, allowing for generating consistent and realistic motion driven by text or other prompts. Furthermore, VideoPoet can synthesize coherent long videos of up to 10 seconds by autoregressively extending the content, conditioned on the last second of the generated video.

实验表明，VideoPoet 在生成大幅度、高保真运动的视频上达到先进水平。凭借 Transformer 架构的强大能力，它可直接在多任务、多模态生成目标上训练，从而在文本或其他提示驱动下生成一致且真实的运动。此外，以已生成视频的最后一秒为条件自回归延展内容，VideoPoet 可合成长达约 10 秒的连贯长视频。

We also demonstrate that VideoPoet is capable of zero-shot video generation. We use the term “zero-shot video generation” as VideoPoet processes new text, image, or video inputs that diverge from the training data distribution. Furthermore, VideoPoet handles new tasks not included in its training. For example, VideoPoet is able to perform new editing tasks by sequentially chaining tasks together.

我们还展示 VideoPoet 具备零样本视频生成能力。此处「零样本视频生成」指：处理偏离训练数据分布的新文本、图像或视频输入；以及处理训练中未包含的新任务。例如，可通过顺序串联任务完成新的编辑任务。

Our contribution is a proof of concept demonstrating an understudied approach to high-quality video generation with LLMs, distinct from the dominant diffusion-based methods. Specifically, the main contributions include:

本文贡献是一个概念验证：展示一条与主流扩散方法不同、用 LLM 做高质量视频生成的、研究尚少的路径。具体而言，主要贡献包括：

• • A method for training a Large Language Model (LLM) specifically for video generation, utilizing tokenized data that incorporates both text-paired and unpaired videos.
• • A video super-resolution method that increases spatial resolution within the latent token space using a bidirectional transformer with efficient windowed local attention.
• • Evaluations and demonstrations to highlight VideoPoet’s competitive and state-of-the-art performance, especially in generating realistic and interesting videos with motion.

• • 一种专为视频生成训练大语言模型（LLM）的方法，使用同时包含有文本配对与无配对视频的分词数据。
• • 一种在潜在词元空间内提升空间分辨率的视频超分方法，采用带高效窗口局部注意力的双向 Transformer。
• • 评估与演示，突出 VideoPoet 的竞争力与先进表现，尤其是在生成真实有趣的运动视频方面。

Figure 2: Sequence layout for VideoPoet. We encode all modalities into the discrete token space, so that we can directly use large language model architectures for video generation. We denote special tokens in $<>$ (see Table 4 for definitions). The modality agnostic tokens are in darker red; the text related components are in blue; the vision related components are in yellow; the audio related components are in green. The left portion of the layout on light yellow represents the bidirectional prefix inputs. The right portion on darker red represents the autoregressively generated outputs with causal attention.

图 2：VideoPoet 的序列布局。 我们将所有模态编码到离散词元空间，从而可直接用大语言模型架构做视频生成。尖括号 $<>$ 中为特殊词元（定义见表 4）。模态无关词元为深红；文本相关为蓝；视觉相关为黄；音频相关为绿。浅黄左侧为双向前缀输入；深红右侧为带因果注意力的自回归生成输出。

2 Related Work

2 相关工作

Video diffusion models.

Recently, numerous video generation methods use diffusion-based methods for text-to-video (Ho et al., 2022a; Blattmann et al., 2023b; Zhang et al., 2023a; Blattmann et al., 2023a; He et al., 2023; Zhou et al., 2022; Wang et al., 2023a; Ge et al., 2023; Wang et al., 2023d; Wang et al., 2023c; Singer et al., 2022; Zhang et al., 2023a; Zeng et al., 2023) and video-to-video editing (Liew et al., 2023; Feng et al., 2023; Esser et al., 2023; Chen et al., 2023b). As video diffusion models are usually derived from text-to-image diffusion models (Ramesh et al., 2021; Saharia et al., 2022), additional tasks and modalities are added via inference tricks (Meng et al., 2021), architectural changes (Esser et al., 2023; Liew et al., 2023) and adapter layers (Zhang et al., 2023b; Guo et al., 2023). Although these models are composable after training, they are not trained end-to-end in a unified framework. Our multitask pretraining strategy in a single model improves performance and provides zero-shot video generation capabilities.

视频扩散模型。

近来大量视频生成方法用扩散做文生视频 (Ho et al., 2022a; Blattmann et al., 2023b; Zhang et al., 2023a; Blattmann et al., 2023a; He et al., 2023; Zhou et al., 2022; Wang et al., 2023a; Ge et al., 2023; Wang et al., 2023d; Wang et al., 2023c; Singer et al., 2022; Zhang et al., 2023a; Zeng et al., 2023) 与视频到视频编辑 (Liew et al., 2023; Feng et al., 2023; Esser et al., 2023; Chen et al., 2023b)。视频扩散模型通常派生自文生图扩散模型 (Ramesh et al., 2021; Saharia et al., 2022)，额外任务与模态则通过推理技巧 (Meng et al., 2021)、架构改动 (Esser et al., 2023; Liew et al., 2023) 与适配器层 (Zhang et al., 2023b; Guo et al., 2023) 加入。这些模型训练后虽可组合，但并非在统一框架端到端训练。我们在单一模型中的多任务预训练策略既提升表现，也带来零样本视频生成能力。

Language models for video and image generation.

Video language models are typically derived from the general family of transformer-based language models (Vaswani et al., 2017; Raffel et al., 2020) that easily combine multiple tasks in pretraining and demonstrate powerful zero-shot capabilities. Image generation language models can generate images autoregressively (Yu et al., 2022) or via masked prediction (Chang et al., 2022; Chang et al., 2023). Both families have been extended to text-to-video (Hong et al., 2022; Villegas et al., 2022; Hu et al., 2023; Yan et al., 2021) using paired data. While other text-to-video work with transformers only leverages video-text pairs for training, we also leverage unpaired videos (without text) and the same video for different tasks. Since video language models can flexibly incorporate numerous tasks (Yu et al., 2023a; Nash et al., 2022), including video-to-video, we extend this family of work to text- and multimodal-conditioned tasks in this work with a synergistic pretraining strategy across various tasks.

面向视频与图像生成的语言模型。

视频语言模型通常源自易于在预训练中组合多任务、并展现强零样本能力的、基于 Transformer 的语言模型族 (Vaswani et al., 2017; Raffel et al., 2020)。图像生成语言模型可自回归生成 (Yu et al., 2022)，也可做掩码预测 (Chang et al., 2022; Chang et al., 2023)。两族方法都已扩展到用配对数据做文生视频 (Hong et al., 2022; Villegas et al., 2022; Hu et al., 2023; Yan et al., 2021)。其他基于 Transformer 的文生视频工作只利用视频–文本对训练；我们还利用无文本配对的视频，并让同一视频服务不同任务。由于视频语言模型可灵活纳入众多任务 (Yu et al., 2023a; Nash et al., 2022)，包括视频到视频，本文把这一族工作扩展到文本与多模态条件任务，并采用跨任务协同的预训练策略。

Pretraining task design in LLMs.

As language models can easily incorporate multiple training tasks, task selection is an important area of research. GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022) demonstrate that training LLMs on diverse tasks leads to positive scaling effects on zero- and few-shot tasks. Other approaches show that masking approaches are a valuable learning target (Hoffmann et al., 2022; Yu et al., 2023a; Yu et al., 2024). As the model size grows, training data must grow as well (Hoffmann et al., 2022) to maintain similar performance. Our pretraining strategy enables using the same video for multiple training tasks even without paired text. This design facilitates training on a large quantity of video-only examples, thereby decreasing the demand for video-text pairs.

LLM 中的预训练任务设计。

语言模型易于纳入多种训练任务，任务选择因此成为重要研究方向。GPT-3 (Brown et al., 2020) 与 PaLM (Chowdhery et al., 2022) 表明，在多样任务上训练 LLM 对零样本与少样本任务有正向缩放效应。其他工作显示掩码类目标是有价值的学习靶 (Hoffmann et al., 2022; Yu et al., 2023a; Yu et al., 2024)。模型变大时，训练数据也需增长 (Hoffmann et al., 2022) 以维持相近表现。我们的预训练策略使同一视频即使没有配对文本也能服务多个训练任务，从而便于在大量仅视频样本上训练，降低对视频–文本对的需求。

3 Model Overview

3 模型概览

We propose an effective method for video generation and related tasks from different input signals by leveraging large language models. Our model consists of three components: (1) modality-specific tokenizers, (2) a language model backbone (Fig. 2), and (3) a super-resolution module (Fig. 3). The tokenizers map input data – i.e. image pixels, video frames, and audio waveforms – into discrete tokens in a unified vocabulary. The visual and audio tokens are flattened into a sequence of integers. Next, the LLM accepts these tokens as input along with text embeddings, and is responsible for generative multi-task and multimodal modeling. As illustrated in Fig. 2, VideoPoet conditions on text embeddings, visual tokens, and audio tokens, and autoregressively predicts visual and audio tokens. Subsequently, the super-resolution module increases the resolution of the video outputs while refining visual details for higher quality.

我们提出一种借助大语言模型、从不同输入信号完成视频生成及相关任务的有效方法。模型含三部分：(1) 模态专用分词器；(2) 语言模型骨干（图 2）；(3) 超分模块（图 3）。分词器把输入数据——即图像像素、视频帧与音频波形——映射为统一词表中的离散词元。视觉与音频词元展平为整数序列。随后 LLM 接受这些词元与文本嵌入，负责生成式多任务与多模态建模。如图 2，VideoPoet 以文本嵌入、视觉词元与音频词元为条件，自回归预测视觉与音频词元。之后超分模块提升视频输出分辨率，并细化视觉细节以获得更高质量。

3.1 Tokenization

3.1 分词

We employ the MAGVIT-v2 (Yu et al., 2024) tokenizer for joint image and video tokenization, and the SoundStream (Zeghidour et al., 2021) tokenizer for audio. Visual and audio vocabularies are concatenated into a unified vocabulary. The text modality is represented by embeddings.

图像与视频联合分词采用 MAGVIT-v2 (Yu et al., 2024)，音频采用 SoundStream (Zeghidour et al., 2021)。视觉与音频词表拼接为统一词表。文本模态由嵌入表示。

Image and video tokenizer.

Visual tokenizers are key to generating high-quality video content, often determining the upper limit of achievable video generation quality (Yu et al., 2024). Among existing tokenizers (Esser et al., 2020; Villegas et al., 2022; Yu et al., 2023a; Yu et al., 2023b), we choose the MAGVIT-v2 (Yu et al., 2024) tokenizer due to its performance in visual quality and high compression capabilities, which effectively reduce the sequence length required by the LLM, thereby facilitating more efficient and effective learning. Specifically, a video clip is encoded and quantized into an integer sequence integers, with a decoder mapping back to pixel space. MAGVIT-v2 tokenizes 17-frame 2.125-second 128$\times$128 resolution videos sampled at 8 fps to produce a latent shape of $(5,16,16)$, which is then flattened into 1280 tokens, with a vocabulary size of $2^{18}$. We also tokenize videos into portrait aspect ratio at 128$\times$224 resolution, producing a latent shape of $(5,28,16)$, or 2240 tokens.

图像与视频分词器。

视觉分词器是生成高质量视频内容的关键，往往决定可达到的视频生成质量上限 (Yu et al., 2024)。在现有分词器中 (Esser et al., 2020; Villegas et al., 2022; Yu et al., 2023a; Yu et al., 2023b)，我们选择 MAGVIT-v2 (Yu et al., 2024)，因其视觉质量与高压缩能力突出，能有效缩短 LLM 所需序列长度，从而更高效、更有效地学习。具体地，视频片段被编码并量化为整数序列，再由解码器映回像素空间。MAGVIT-v2 将以 8 fps 采样的 17 帧、2.125 秒、$128\times 128$ 分辨率视频分词为形状 $(5,16,16)$ 的潜变量，展平为 1280 个词元，词表大小 $2^{18}$。我们也按竖屏比例 $128\times 224$ 分词，得到潜变量形状 $(5,28,16)$，即 2240 个词元。

We enforce causal temporal dependency, which facilitates the generation of longer videos. To jointly represent images and videos, we encode the initial frame of a video or a static image into tokens with a consistent shape of $(1,16,16)$. We use the COMMIT (Yu et al., 2023a) encoding scheme to tokenize the inpainting and outpainting tasks.

我们施加因果时间依赖，便于生成更长视频。为联合表示图像与视频，将视频首帧或静态图像编码为形状一致的 $(1,16,16)$ 词元。修复与外扩任务使用 COMMIT (Yu et al., 2023a) 编码方案。

Audio tokenizer.

We tokenize audio clips with a pretrained SoundStream (Zeghidour et al., 2021) tokenizer. We embed 2.125 seconds of audio to produce 106 latent frames with a residual vector quantizer (RVQ) of four levels. Two possible choices exist for predicting the audio tokens with the RVQ representation: 1) sequentially predicting the entire audio clip from a lower to a higher RVQ level, or 2) simultaneously predicting all RVQ levels for a single audio token. Our results suggest that the former method demonstrates a slight advantage over the latter approach. Finally, each RVQ level has a disjoint vocabulary with each level containing 1,024 codes. This results in a combined audio vocabulary size of 4,096 codes.

音频分词器。

我们用预训练 SoundStream (Zeghidour et al., 2021) 对音频片段分词。2.125 秒音频嵌入为 106 个潜在帧，残差向量量化（RVQ）共四级。用 RVQ 表示预测音频词元有两种选择：1) 从低到高 RVQ 层级顺序预测整段音频；2) 对单个音频词元同时预测全部 RVQ 层级。结果表明前者略优于后者。每一 RVQ 层级使用互不相交的词表，每级 1,024 个码，合计音频词表大小 4,096。

Text embedding as input.

Pretrained text representations, in general, outperform training our model by learning text tokens from scratch. We use pretrained language embeddings from a frozen T5 XL encoder (Raffel et al., 2020). For tasks with text guidance, such as text-to-video, T5 XL embeddings are projected into the transformer’s embedding space with a linear layer.

以文本嵌入为输入。

一般而言，预训练文本表示优于从零学习文本词元。我们使用冻结 T5 XL 编码器 (Raffel et al., 2020) 的预训练语言嵌入。对文生视频等有文本引导的任务，用线性层把 T5 XL 嵌入投影到 Transformer 嵌入空间。

3.2 Language Model Backbone

3.2 语言模型骨干

After converting the image, video, and audio modalities into discrete tokens within a shared vocabulary, we can directly leverage a language model to generate videos and audios in the token space. We use a prefix language model with a decoder-only architecture as the backbone. By constructing different patterns of input tokens to output tokens during training, we can control the tasks the model is able to perform as explained in Section 4.

将图像、视频与音频模态转为共享词表中的离散词元后，即可直接用语言模型在词元空间生成视频与音频。骨干采用仅解码器架构的前缀语言模型。通过在训练时构造不同的输入词元到输出词元模式，可控制模型能完成的任务，详见第 4 节。

3.3 Super-Resolution

3.3 超分辨率

Figure 3: Custom transformer architecture for video super-resolution.

图 3：用于视频超分的定制 Transformer 架构。

Generating high-resolution (HR) videos autoregressively entails heavy computational costs due to the increase in sequence length. To illustrate this with an example, the video tokenizer of Section 3.1 operating on a $17\times 896\times 512$ video produces a sequence of $35,840$ tokens, making autoregressive sampling highly impractical. Aiming at efficient and high-quality generative video upsampling, we develop a custom spatial super-resolution (SR) non-autoregressive video transformer (Yu et al., 2023a) to operate in token space on top of the language model output. To mitigate the computational requirements of the very long sequences involved, and in particular the quadratic memory of the self-attention layers, our design incorporates windowed local attention (Gupta et al., 2022). Specifically, our SR transformer is composed of blocks of three transformer layers, each of which performs self-attention in a local window aligned with one of three axes (Tu et al., 2022): spatial vertical, spatial horizontal and temporal. The cross-attention layers attend to the low-resolution (LR) token sequence and are also divided into local windows, isomorphic to those of the self-attention layers. All blocks also include cross-attention to T5 XL text embeddings. See Fig. 3 for a schematic representation of the custom transformer architecture.

自回归生成高分辨率（HR）视频会因序列变长而计算代价沉重。举例：第 3.1 节分词器作用于 $17\times 896\times 512$ 视频时产生 $35,840$ 个词元，自回归采样极不现实。为高效且高质量地做生成式视频上采样，我们设计定制的空间超分（SR）非自回归视频 Transformer (Yu et al., 2023a)，在语言模型输出之上的词元空间操作。为缓解超长序列的计算需求，尤其是自注意力层的二次内存，设计纳入窗口局部注意力 (Gupta et al., 2022)。具体地，SR Transformer 由三层一组的块构成，每层在对齐三个轴之一的局部窗口内做自注意力 (Tu et al., 2022)：空间竖直、空间水平与时间。交叉注意力层关注低分辨率（LR）词元序列，并同样划入与自注意力同构的局部窗口。所有块还对 T5 XL 文本嵌入做交叉注意力。定制架构示意见图 3。

To account for the larger vocabulary size, we follow (Yu et al., 2024), and train the SR transformer using token factorization with $k=2$ factors, which converts a $262,144$-way classification problem into two $512$-way classification problems. The LR token sequences are obtained by tokenizing bicubic-downsampled versions of the ground truth videos and applying noise augmentation (Ho et al., 2022a) in the discrete latent space, to mitigate the distribution mismatch between real and generated videos. Specifically, we randomly resample the value of a random subset of the LR tokens and independently drop the LR condition and text embeddings for 10% of the training samples. During inference, we use non-autoregressive sampling (Chang et al., 2022; Yu et al., 2023a) with classifier-free guidance independently on both the LR condition and the text embeddings (Brooks et al., 2023). We use a cascade of two $2\times$ stages to generate videos of $896\times 512$ resolution from the $224\times 128$ base output of VideoPoet. More implementaiton details can be found in the appendix.

为应对更大词表，我们遵循 (Yu et al., 2024)，用 $k=2$ 的词元因子化训练 SR Transformer，把 $262,144$ 路分类转为两个 $512$ 路分类。LR 词元序列通过对真值视频双三次下采样后分词得到，并在离散潜空间施加噪声增强 (Ho et al., 2022a)，以缓解真实与生成视频之间的分布错配。具体地，我们随机重采样 LR 词元随机子集的取值，并对 10% 训练样本独立丢弃 LR 条件与文本嵌入。推理时使用非自回归采样 (Chang et al., 2022; Yu et al., 2023a)，并对 LR 条件与文本嵌入分别做无分类器引导 (Brooks et al., 2023)。我们级联两个 $2\times$ 阶段，把 VideoPoet 基座输出的 $224\times 128$ 提升到 $896\times 512$。更多实现细节见附录。

4 LLM Pretraining for Generation

4 面向生成的 LLM 预训练

4.1 Task Prompt Design

4.1 任务提示设计

We design a pretraining task mixture, each with a defined prefix input and output. The model conditions on the prefix, applying the loss solely to the output. Fig. 2 shows a typical input-output sequence layout. For each task, the input sequence may include three types of values: text embeddings (T5), visual tokens (MAGVIT-v2), and audio tokens (SoundStream). The model outputs two types of tokens: visual and audio tokens. To facilitate training, VideoPoet employs special tokens, as listed in Appendix Table 4. In the following, we describe key designs for the task prompts.

我们设计预训练任务混合，每个任务有明确的前缀输入与输出。模型以前缀为条件，损失仅施加在输出上。图 2 给出典型输入–输出序列布局。对每个任务，输入序列可含三类取值：文本嵌入（T5）、视觉词元（MAGVIT-v2）与音频词元（SoundStream）。模型输出两类词元：视觉与音频。为便于训练，VideoPoet 使用附录表 4 所列特殊词元。下文描述任务提示的关键设计。

Pretraining tasks.

We consider the following tasks. Unconditioned video generation: Generate video frames without conditioning on an input. Text-to-video (T2V): Generate video from a text prompt. Video future prediction (FP): Given an input video of variable length, predict future frames. Image-to-video (I2V): Given the first frame of a video as an input image, predict the future frames. Video inpainting/outpainting (Painting): Given a masked video, predict the video with the masked contents filled in. Video stylization: Given text, optical flow, and depth, predict the video frames (Section 4.1). Audio-to-video: Given an input audio waveform, predict the corresponding video. Video-to-audio: Given an input video, predict the corresponding audio waveform. Audio-video continuation (AVCont) given an input frame and its audio, predict the rest of the video and audio. In principle, the model can generate text, but we have not explicitly evaluated this ability.

预训练任务。

我们考虑如下任务。无条件视频生成：不依赖输入生成视频帧。文生视频（T2V）：由文本提示生成视频。视频未来预测（FP）：给定可变长度输入视频，预测未来帧。图生视频（I2V）：给定视频首帧作为输入图像，预测后续帧。视频修复/外扩（Painting）：给定带掩码视频，预测填补掩码内容后的视频。视频风格化：给定文本、光流与深度，预测视频帧（第 4.1 节）。音频到视频：给定输入音频波形，预测对应视频。视频到音频：给定输入视频，预测对应音频波形。音视频续写（AVCont）：给定一帧及其音频，预测其余视频与音频。原则上模型也可生成文本，但我们未显式评估该能力。

To indicate the type of task, we condition on the token, which has a unique value for each unique output. We note that not all input variations need a new ; the model adapts to different context signals for identical outputs. For instance, text-to-video, image-to-video, and unconditioned video generation share the same . If a modality is absent in a task, related input/output tokens and special tokens are excluded, shortening the sequence.

为指示任务类型，我们以 词元为条件，每种独特输出对应唯一取值。并非所有输入变体都需要新的 ；对相同输出，模型可适应不同上下文信号。例如文生视频、图生视频与无条件视频生成共享同一 。若某任务缺少某模态，则排除相关输入/输出词元与特殊词元，序列随之缩短。

Representing an image as a video.

In text-to-image pretraining, we omit the and tokens from the input sequence, enabling continuous token generation for inference of longer videos. This approach blurs the boundary between video and image generation tasks, enhancing cross-modality information sharing. This design leads to the prediction of higher-quality initial frames and reduces errors and artifacts in subsequent frames.

把图像表示为视频。

在文生图预训练中，我们从输入序列省略 与 词元，使推理时可连续生成词元以得到更长视频。这一做法模糊视频与图像生成任务的边界，增强跨模态信息共享，从而有助于预测更高质量的初始帧，并减少后续帧中的误差与伪影。

Video token format.

We generate video tokens at two resolutions, 128$\times$128 and 128$\times$224, each available in two lengths: 17 frames and 41 frames, both encoded at 8 frames per second. Special conditioning tokens are used to signal the desired resolutions and durations for video generation. Images are a special case of a 1-frame video, which we tokenize at 128$\times$128 resolution.

视频词元格式。

我们在两种分辨率生成视频词元：$128\times 128$ 与 $128\times 224$，各有两种长度：17 帧与 41 帧，均以每秒 8 帧编码。用特殊条件词元指示期望的分辨率与时长。图像视为 1 帧视频的特例，按 $128\times 128$ 分词。

Video stylization.

For video stylization, we adopt a method motivated by (Zhang et al., 2023b; Chen et al., 2023b; Esser et al., 2023), predicting videos from text, optical flow, and depth signals. The training task for stylization is to reconstruct the ground truth video from the given optical flow, depth, and text information, but during inference, we apply optical flow and depth estimation on an input video but then vary the text prompt to generate a new style, e.g. “cartoon.” Similar to (Esser et al., 2023), text dictates the output “content” or appearance, while optical flow and depth guide its “structure.”

视频风格化。

风格化采用受 (Zhang et al., 2023b; Chen et al., 2023b; Esser et al., 2023) 启发的方法：由文本、光流与深度信号预测视频。训练任务是从给定光流、深度与文本重建真值视频；推理时对输入视频估计光流与深度，再更换文本提示以生成新风格，例如 “cartoon”。与 (Esser et al., 2023) 类似，文本决定输出的「内容」或外观，光流与深度引导其「结构」。

4.2 Training Strategy

4.2 训练策略

For multi-task training, we use the Alternating Gradient Descent (AGD) method (Akbari et al., 2023) to train videos of varying lengths. We design the tasks in the AGD format resulting in a near 0% padding ratio, lower than that of the packing approach (Raffel et al., 2020). This is accomplished by grouping tasks by sequence length and alternately sampling one group at each iteration. Since sequence lengths are fixed and vary significantly across tasks, e.g., first frame and long video generation, we achieve efficient training with minimal padding.

多任务训练采用交替梯度下降（AGD）方法 (Akbari et al., 2023) 以处理不同长度的视频。我们把任务设计成 AGD 格式，填充率接近 0%，低于打包方法 (Raffel et al., 2020)。做法是按序列长度分组任务，每次迭代交替采样一组。由于序列长度固定且跨任务差异大（例如首帧与长视频生成），我们以极少填充实现高效训练。

We find that sampling from image and video datasets uniformly across time can lead to suboptimal results, as training on images can enhance the model’s understanding of objects but does not capture any motions that are represented in video data. Thus, we devise a two-stage pretraining strategy, where we augment our sampling weights to sample image data 90% of the time and video data 10% of the time for the first 25% iterations of training. We then switch to training on video 90% and image 10% for the remaining iterations.

我们发现，若在时间上均匀采样图像与视频数据，结果往往次优：图像训练能增强对物体的理解，却捕捉不到视频中的运动。因此采用两阶段预训练：前 25% 迭代把采样权重设为图像 90%、视频 10%；之后切换为视频 90%、图像 10%。

We fine-tune our pretrained model for enhanced performance on specific tasks or for new task adaptation, such as text-to-video and image-to-video tasks, using a data subset of higher quality. These videos are sourced from broad internal sources with millions of videos that contain simpler clips, and are not manually tailored or selected. This results in improved generation quality, consistent with Zhou et al. (2023), and addresses decoding collapse issues, characterized by repetitive token predictions. Such fine-tuning not only diversifies outputs but also allows for a higher classifier-free guidance scale (Ho & Salimans, 2022), boosting overall quality.

我们在更高质量的数据子集上微调预训练模型，以增强特定任务表现或适配新任务，如文生视频与图生视频。这些视频来自涵盖数百万视频的广泛内部来源，片段更简单，并非人工精选。结果与 Zhou et al. (2023) 一致，生成质量提升，并缓解以重复词元预测为特征的解码崩溃。此类微调不仅使输出更多样，也允许使用更高的无分类器引导尺度 (Ho & Salimans, 2022)，从而提升整体质量。

5 Experiments

5 实验

5.1 Experimental Setup

5.1 实验设置

Training tasks.

We train the model on a mixture of pretraining tasks as detailed in Section 4.1. We finetune a model on a high-quality training subset for text-to-video evaluations, as discussed in Section 4.2. Unless explicitly stated, we do not finetune on specific tasks for evaluations.

训练任务。

我们按第 4.1 节所述的预训练任务混合训练模型。如第 4.2 节讨论，在高质量训练子集上微调模型以做文生视频评估。除非另有说明，评估时不对特定任务单独微调。

Datasets.

We train on a total of 1B image-text pairs and $\sim$270M videos ($\sim$100M with paired text, of which $\sim$50M are used for high-quality finetuning, and $\sim$170M with paired audio) from the public internet and other sources, i.e. around 2 trillion tokens across all modalities. The data has been filtered to remove egregious content and sampled to improve contextual and demographic diversity.

数据集。

训练数据共计 10 亿图文对与约 2.7 亿视频（约 1 亿带配对文本，其中约 5,000 万用于高质量微调；约 1.7 亿带配对音频），来自公开互联网及其他来源，即跨所有模态约 2 万亿词元。数据经过滤以去除不当内容，并经采样以改善情境与人口统计多样性。

Method	Pretraining Tasks	Zero-shot Evaluation Benchmark
T2I	T2V	Uncond	FP	Painting	AVCont	T2V	FP	Inpainting	Outpainting
MSR-VTT	UCF101	K600	SSv2	SSv2
CLIPSIM $\uparrow$	FVD $\downarrow$	FVD $\downarrow$	FVD $\downarrow$	FVD $\downarrow$
T2V		✓					0.244	822	759	2,333	2,310
T2V+I	✓	✓					0.247	1,025	794	2,118	1,916
SSL			✓	✓	✓	✓	0.226	1,742	700	1,093	1,500
NO T2I		✓	✓	✓	✓	✓	0.235	1,008	755	95	389
ALL	✓	✓	✓	✓	✓	✓	0.240	1,085	729	127	636
ALL (8B)	✓	✓	✓	✓	✓	✓	0.305	355	687	4.7	13.76

Table 1: Pretraining task analysis on 300M models. The top rows list models with 300M parameters, trained on a subset of the data, and are comparable to each other. The last row shows an 8B model trained on the entire dataset. T2I (text-to-image), T2V (text-to-video), FP (frame prediction), Painting (inpainting/outpainting), Uncond (unconditional generation), AVCont (audio-video continuation), and SSL (self-supervised learning).

| 方法 | 预训练任务 | 零样本评估基准 | | | | | | | | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| T2I | T2V | Uncond | FP | Painting | AVCont | T2V | FP | Inpainting | Outpainting | | |
| MSR-VTT | UCF101 | K600 | SSv2 | SSv2 | | | | | | | |
| CLIPSIM↑(MSR) | FVD↓(UCF) | FVD↓(K600) | FVD↓(SSv2修) | FVD↓(SSv2外) | | | | | | | |
| T2V | | ✓ | | | | | 0.244 | 822 | 759 | 2,333 | 2,310 |
| T2V+I | ✓ | ✓ | | | | | 0.247 | 1,025 | 794 | 2,118 | 1,916 |
| SSL | | | ✓ | ✓ | ✓ | ✓ | 0.226 | 1,742 | 700 | 1,093 | 1,500 |
| NO T2I | | ✓ | ✓ | ✓ | ✓ | ✓ | 0.235 | 1,008 | 755 | 95 | 389 |
| ALL | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 0.240 | 1,085 | 729 | 127 | 636 |
| ALL (8B) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 0.305 | 355 | 687 | 4.7 | 13.76 |

表 1：300M 模型上的预训练任务分析。 上半部分为参数量 3 亿、在数据子集上训练、彼此可比较的模型；最后一行是在全数据上训练的 80 亿参数模型。T2I（文生图）、T2V（文生视频）、FP（帧预测）、Painting（修复/外扩）、Uncond（无条件生成）、AVCont（音视频续写）、SSL（自监督学习）。

Evaluation protocol.

We employ a zero-shot generation evaluation protocol, as the model has not been trained on the training data of target benchmarks. Specifically, the evaluation benchmark includes two text-to-video generation datasets, MSR-VTT (Xu et al., 2016) and UCF-101 (Soomro et al., 2012), as well as the frame prediction task on Kinetics 600 (K600) (Carreira et al., 2018), in which the first 5 frames are provided as the condition to predict the next 11 frames. We also include inpainting and outpainting tasks (Yu et al., 2023a) on Something-Something V2 (SSv2) (Goyal et al., 2017).

评估协议。

我们采用零样本生成评估协议：模型未在目标基准的训练集上训练。具体地，评估基准包括两个文生视频数据集 MSR-VTT (Xu et al., 2016) 与 UCF-101 (Soomro et al., 2012)，以及 Kinetics 600 (K600) (Carreira et al., 2018) 上的帧预测任务（以前 5 帧为条件预测后 11 帧）。还包括 Something-Something V2 (SSv2) (Goyal et al., 2017) 上的修复与外扩任务 (Yu et al., 2023a)。

We employ widely used metrics such as Fréchet Video Distance (FVD) (Unterthiner et al., 2018), CLIP similarity score (Wu et al., 2021), and Inception Score (IS) (Saito et al., 2020) for evaluation. Note that the specific metrics and evaluation methods vary across different datasets. Detailed information on these variations can be found in Section A.5.5. We include examples of the generated videos in the supplementary materials.

我们采用广泛使用的指标：Fréchet 视频距离（FVD） (Unterthiner et al., 2018)、CLIP 相似度 (Wu et al., 2021) 与 Inception Score（IS） (Saito et al., 2020)。注意不同数据集的具体指标与评估方法有所不同，细节见第 A.5.5 节。生成视频示例见补充材料。

5.2 Pretraining Task Analysis

5.2 预训练任务分析

We investigate the learning capabilities of different combinations of pretraining tasks using a model with 300 million parameters. All task combinations are trained using a learning rate of $10^{-3}$ for the same number of steps (300k) with a batch size of 1024.

我们用 3 亿参数模型考察不同预训练任务组合的学习能力。所有任务组合以学习率 $10^{-3}$、相同步数（30 万）、批大小 1024 训练。

For the analysis of pretraining tasks, we consider text-to-video (T2V), text-to-image (T2I), and four self-supervised learning (SSL) tasks: frame prediction (FP), central inpainting and central outpainting (Painting) (Yu et al., 2023a) and audio-video continuation (AVCont) where the model is provided with the first frame and its corresponding audio to predict the subsequent 16 frames and matching audio. For each video task, we uniformly select 20% of training samples from a random subset of 50 million videos. For the text-to-image task, we randomly sample 50 million text-image pairs from our training dataset. For tasks involving audio, our sampling is exclusive to videos that contain an audio track.

预训练任务分析中，我们考虑文生视频（T2V）、文生图（T2I），以及四项自监督（SSL）任务：帧预测（FP）、中心修复与中心外扩（Painting） (Yu et al., 2023a)，以及音视频续写（AVCont）——给定首帧及其对应音频，预测随后 16 帧与匹配音频。对每个视频任务，从随机抽取的 5,000 万视频子集中均匀选取 20% 训练样本；文生图任务从训练数据中随机采样 5,000 万图文对；涉及音频的任务仅采样含音轨的视频。

The evaluation results are presented in Table 1. We assess a model across the four tasks within the zero-shot evaluation benchmark: the T2V task on MSR-VTT (Xu et al., 2016) and UCF 101 (Soomro et al., 2012), the FP on K600 (Carreira et al., 2018), and central inpainting and outpainting on SSv2 (Goyal et al., 2017). The audio generation results (FAD) are reported in Fig. 5(b) of the Appendix. In these experiments, we employ a single model to perform all the tasks. The model is not trained on the training data of these evaluation datasets, and thus it is a zero-shot evaluation.

评估结果见表 1。我们在零样本评估基准的四项任务上评估：MSR-VTT (Xu et al., 2016) 与 UCF 101 (Soomro et al., 2012) 上的 T2V、K600 (Carreira et al., 2018) 上的 FP，以及 SSv2 (Goyal et al., 2017) 上的中心修复与外扩。音频生成结果（FAD）见附录图 5(b)。这些实验用单一模型完成全部任务；模型未在这些评估数据集的训练集上训练，因此为零样本评估。

The top rows of Table 1 depict each pretraining task configuration of the 300 million parameter model, which are comparable in their setups. Our evaluation benchmarks span diverse visual domains, posing a challenge to achieving consistent improvement across all of them. Nevertheless, incorporating all pretraining tasks results in the best overall performance, on average, across all evaluated tasks. Additionally, the significant disparity observed in the “SSL” row suggests the limitations of self-supervised training and underscores the necessity for text-paired data during training. Both single-task and multi-task models are trained for the same number of steps. The minor decrease in performance of multi-task training in Table 1 might be due to the insufficient training of each task. The last row, “ALL (8B)”, is the model with 8 billion parameters, trained on the pretraining tasks as discussed in Section 3 and utilized significantly more compute.

表 1 上半部分给出 3 亿参数模型各预训练任务配置，设定彼此可比较。评估基准横跨多样视觉领域，要在所有领域上一致改进并不容易。不过，纳入全部预训练任务在所有评估任务上平均表现最佳。「SSL」行的显著落差则提示纯自监督训练的局限，以及训练中需要文本配对数据。单任务与多任务模型训练步数相同；表 1 中多任务训练的小幅下降可能因各任务训练不足。最后一行「ALL (8B)」为 80 亿参数模型，按第 3 节所述预训练任务训练，并使用显著更多算力。

Figure 4: Human evaluation results on text-to-video (T2V) generation. Green and pink bars represent the proportion of trials where VideoPoet was preferred over or less preferred to an alternative, respectively.

图 4：文生视频（T2V）上的人工评估结果。 绿色与粉色条分别表示 VideoPoet 相对对照方法更受偏好或更不受偏好的试验比例。

5.3 Comparison with the State-of-the-Art

5.3 与先进方法比较

Model	MSR-VTT	UCF-101
	CLIPSIM	FVD	FVD	IS
CogVideo (EN) (2022)	0.2631	1294	702	25.27
MagicVideo (2022)	-	998	655	-
Video LDM (2023b)	0.2929	-	551	33.45
ModelScopeT2V (2023a)	0.2930	550	-	-
InternVid (2023d)	0.2951	-	617	21.04
VideoFactory (2023c)	0.3005	-	410	-
Make-A-Video (2022)	0.3049	-	367	33.00
Show-1 (2023a)	0.3072	538	394	35.42
VideoPoet (Pretrain)	0.3049	213	355	38.44
VideoPoet (Task adapt)	0.3123	-	-	-

Table 2: Comparison on zero-shot text-to-video benchmarks. See Appendix A.5.5 for evaluation details.

模型	MSR-VTT	UCF-101
	CLIPSIM	FVD	FVD	IS
CogVideo (EN) (2022)	0.2631	1294	702	25.27
MagicVideo (2022)	-	998	655	-
Video LDM (2023b)	0.2929	-	551	33.45
ModelScopeT2V (2023a)	0.2930	550	-	-
InternVid (2023d)	0.2951	-	617	21.04
VideoFactory (2023c)	0.3005	-	410	-
Make-A-Video (2022)	0.3049	-	367	33.00
Show-1 (2023a)	0.3072	538	394	35.42
VideoPoet (Pretrain)	0.3049	213	355	38.44
VideoPoet (Task adapt)	0.3123	-	-	-

表 2：零样本文生视频基准上的比较。 评估细节见附录 A.5.5。

Text-to-Video (T2V).

Table 2 shows zero-shot text-to-video evaluation results on the common MSR-VTT (Xu et al., 2016) and UCF-101 (Soomro et al., 2012) datasets. Our model performs favorably in terms of CLIP similarity and FVD scores on MSR-VTT and UCF-101. The pretrained foundation model already achieves competitive performance on all metrics. After finetuned on high-quality subset of text-video pairs, VideoPoet achieves even better CLIPSIM on MSR-VTT. More details on the evaluation settings can be found in Section A.5.5.

文生视频（T2V）。

表 2 给出常见 MSR-VTT (Xu et al., 2016) 与 UCF-101 (Soomro et al., 2012) 数据集上的零样本文生视频评估。我们的模型在 CLIP 相似度与 FVD 上表现有利。预训练基础模型已在全部指标上达到有竞争力的水平；在高质量文视频对子集上微调后，VideoPoet 在 MSR-VTT 上取得更好的 CLIPSIM。更多评估设定细节见第 A.5.5 节。

Human Evaluations with Text-to-Video (T2V).

We analyze VideoPoet using human raters and compare with other recent models: Show-1 (Zhang et al., 2023a), VideoCrafter (Chen et al., 2023a), Phenaki (Villegas et al., 2022), Pika (Pika, 2023), Gen2 (Runway, 2023), WALT (Gupta et al., 2023) and Lumiere (Bar-Tal et al., 2024). Show-1, VideoCrafter, Pika, Gen2, WALT and Lumiere are video diffusion models while Phenaki is a token-based model using masked token modeling (Chang et al., 2022). We ran the most up-to-date model versions as of January 2024 and note that WALT and Lumiere were concurrently developed and not available during initial submission of this paper.

文生视频（T2V）人工评估。

我们用人工评分分析 VideoPoet，并与近期模型比较：Show-1 (Zhang et al., 2023a)、VideoCrafter (Chen et al., 2023a)、Phenaki (Villegas et al., 2022)、Pika (Pika, 2023)、Gen2 (Runway, 2023)、WALT (Gupta et al., 2023) 与 Lumiere (Bar-Tal et al., 2024)。Show-1、VideoCrafter、Pika、Gen2、WALT 与 Lumiere 为视频扩散模型，Phenaki 为使用掩码词元建模 (Chang et al., 2022) 的基于词元的模型。我们运行截至 2024 年 1 月的最新模型版本；注意 WALT 与 Lumiere 与本文同期开发，初稿提交时尚未可用。

We first develop a unified evaluation prompt bank consisting of $\sim 250$ selected prompts from a variety of categories and styles. Our prompts are sourced from published prompt sets (e.g., Show-1, Video LDM (Blattmann et al., 2023b)). We select the prompts prior to generating videos and fix these choices after initial selection. We also select preferentially for prompts that contain an explicit mention of motion so that the evaluation would not be biased for models that generate high quality videos that are almost still (e.g., “person jumping off of a chair” over “person standing on a chair”). Note that due to time constraints, our experiments for Pika and Gen2 were run on a subset of 50 prompts due to having to submit these manually via their web interface. These 50 prompts were pre-selected (before any evaluations were run) so as to be representative of the entire set.

我们首先构建统一评估提示库，从多种类别与风格中选出约 $250$ 条提示。提示来自已发表提示集（例如 Show-1、Video LDM (Blattmann et al., 2023b)）。在生成视频之前选定提示，并在初次选定后固定。我们还优先选择明确提及运动的提示，以免评估偏向生成近乎静止高质量视频的模型（例如偏好「人从椅子上跳下」而非「人站在椅子上」）。因时间限制，Pika 与 Gen2 的实验在 50 条提示子集上通过其网页界面手工提交完成；这 50 条在任何评估开始前预先选定，以代表全集。

For this user study we use the fine-tuned version of VideoPoet as discussed in Section 4.2. As part of our system we also use a fixed negative prompt for all inference calls as well as lightweight prompt rewriting. We then compare VideoPoet against alternative models in side-by-side fashion for each prompt. Raters are shown videos generated by two models at a time (in randomized order so as to not bias raters). We refer readers to Appendix A.5.4 for more details of our methodology.

本用户研究使用第 4.2 节讨论的微调版 VideoPoet。系统对所有推理调用使用固定负面提示，并做轻量提示改写。随后按提示与对照模型做并排比较。评分者每次看到两个模型生成的视频（顺序随机以防偏差）。方法细节见附录 A.5.4。

Not all methods generate videos at the same size, aspect ratio, or framerate, and we thus resize and resample each video to a fixed area while maintaining its original aspect ratio as well as common framerate. Raters are then asked to compare the videos along 5 dimensions and for each dimension to report which video is better. The 5 dimensions are: (1) text fidelity (which video follows the text prompt most faithfully), (2) video quality, (3) motion “interestingness”, (4) motion realism, and (5) temporal consistency. Raters are required to go through a collection of training examples for each of these 5 dimensions before they begin.

各方法生成视频的尺寸、宽高比或帧率并不相同，因此我们将每个视频缩放到固定面积并重采样，同时保持原始宽高比与共同帧率。评分者沿 5 个维度比较视频，并报告每个维度上哪条更好。五个维度为：(1) 文本保真（哪条更忠实遵循文本提示）；(2) 视频质量；(3) 运动「趣味性」；(4) 运动真实感；(5) 时间一致性。评分者开始前须完成这 5 个维度的训练示例。

Our findings are summarized in Fig. 4, where green and pink bars represent the proportion of trials where VideoPoet was preferred or less preferred over an alternative, respectively. We observe that VideoPoet is competitive with state of the art video diffusion models despite taking a dramatically different approach, even outperforming these baseline models along most dimensions. More specifically, VideoPoet achieves significant wins along motion interestingness and realism and Lumiere (Bar-Tal et al., 2024) which is diffusion based and concurrent to our work, is the only model that outperforms VideoPoet on Video Quality.

发现汇总于图 4：绿/粉条分别表示 VideoPoet 相对对照更受偏好或更不受偏好的比例。我们观察到，尽管路径截然不同，VideoPoet 仍可与先进视频扩散模型竞争，并在多数维度上胜过这些基线。更具体地，它在运动趣味性与真实感上取得显著优势；与本文同期、基于扩散的 Lumiere (Bar-Tal et al., 2024) 是唯一在视频质量上超过 VideoPoet 的模型。

5.4 Runtime

5.4 运行时间

For a batch size of 4 videos and generating 17 frames at 8fps using TPUv5p (4 chips) accelerators, our base model runs in 34s, the detokenizer (converting tokens to pixels) requires 1.3s and super-resolution is 6.8s — thus, amortized run time is about 5 seconds per second of output video. Note that our model has not been optimized, and any acceleration techniques applicable to standard LLMs could be applied here as well.

批大小为 4、生成 17 帧（8 fps）、使用 TPUv5p（4 芯片）时，基座模型耗时 34 秒，去分词（词元转像素）1.3 秒，超分 6.8 秒——摊销后约每秒输出视频需 5 秒运行时间。注意模型尚未优化，适用于标准 LLM 的加速技术同样可用。

5.5 LLM’s Diverse Capabilities in Video Generation

5.5 LLM 在视频生成中的多样能力

This subsection presents several capabilities we discover from the pretrained VideoPoet, shedding light on the LLM’s promising potential in video generation. By combining the flexibility of our autoregressive language model to perform diverse tasks such as extending video in time, inpainting, outpainting, and stylization, VideoPoet accomplishes multiple tasks in a unified model.

本节展示从预训练 VideoPoet 中发现的若干能力，揭示 LLM 在视频生成中的潜力。通过把自回归语言模型在时间延展、修复、外扩与风格化等多样任务上的灵活性组合起来，VideoPoet 在统一模型中完成多项任务。

Coherent long video generation and image-to-video.

Figure 4: 10-Second long video generation example. By predicting 1-second video segments from an initial 1-second clip, VideoPoet can iteratively generate videos of extended lengths.

连贯长视频生成与图生视频。

图 4：10 秒长视频生成示例。 从初始 1 秒片段出发，逐段预测 1 秒视频，VideoPoet 可迭代生成更长视频。

Animated from historical photo

Animated from painting

Figure 4: Examples of videos animated from still images plus text prompts tailored to each initial image.

由历史照片驱动动画

由画作驱动动画

图 4：由静帧加针对各初始图像定制的文本提示驱动的视频示例。

A benefit of an decoder-based language model is that it pairs well with autoregressively extending generation in time. We present two different variants: Generating longer videos and converting images to videos. Encoding the first frame independently allows us to convert any image into the initial frame of a video without padding. Subsequent frames are generated by predicting remaining tokens, transforming the image into a video as shown in Fig. 4[^1].

仅解码器语言模型的一个好处，是与沿时间自回归延展生成相契合。我们展示两种变体：生成更长视频，以及把图像转为视频。独立编码首帧后，无需填充即可把任意图像变成视频的初始帧；再预测其余词元，即可如图 4 所示把图像变成视频[^1]。

This results in the capability to generate videos longer than 10 seconds or to allow users to iteratively extend video clips based on previously generated video, and produces temporally consistent videos without significant distortion. Such capabilities are rarely observed in contemporary diffusion models.

由此可生成超过 10 秒的视频，或允许用户基于已生成片段迭代延展，并得到无明显畸变的时间一致视频。当代扩散模型中很少见到这类能力。

Zero-shot video editing and task chaining.

Animated from still image

Stylized video Prompt: An oil painting of a snowman with a red hat opening their mouth to yawn

Figure 4: Example of zero-shot video editing via task chaining (text conditioned image-to-video and stylization)

零样本视频编辑与任务串联。

由静帧驱动动画

风格化视频 提示： An oil painting of a snowman with a red hat opening their mouth to yawn

图 4：通过任务串联实现零样本视频编辑的示例（文本条件图生视频与风格化）

With the multi-task pretraining, VideoPoet exhibits task generalization that can be chained together to perform novel tasks. We show the model can apply image-to-video animation followed by video-to-video stylization in Fig. 4. In the Appendix, Fig. 6 shows another example applying video-to-video outpainting, followed by editing them with additional video-to-video effects. At each stage, the quality of the output is sufficient to remain in-distribution (i.e. teacher forcing) for the next stage without noticeable artifacts. These capabilities can be attributed to our multimodal task design within an LLM transformer framework that allows for modeling multimodal content using a single transformer architecture over a unified vocabulary.

借助多任务预训练，VideoPoet 展现可串联以完成新任务的任务泛化。图 4 展示先做图生视频动画、再做视频到视频风格化。附录图 6 另有一例：先视频到视频外扩，再叠加其他视频到视频特效。每一阶段输出质量足以保持在下一阶段的分布内（即教师强制），无明显伪影。这些能力可归因于在 LLM Transformer 框架内的多模态任务设计，使单一 Transformer 架构能在统一词表上建模多模态内容。

Zero-shot video stylization.

Stylization results are presented in Section A.4 where the structure and text are used as prefixes to guide the language model. Unlike other stylization methods that employ adapter modules such as cross-attention networks (Zhang et al., 2023b) or latent blending (Meng et al., 2021), our approach stylizes videos within an LLM as one of several generative tasks.

零样本视频风格化。

风格化结果见第 A.4 节：结构与文本作为前缀引导语言模型。与使用交叉注意力网络 (Zhang et al., 2023b) 或潜空间混合 (Meng et al., 2021) 等适配器模块的风格化方法不同，我们的方法把视频风格化作为 LLM 内若干生成任务之一。

3D structure, camera motion, visual styles.

Even though we do not add specific training data or losses to encourage 3D consistency, our model can rotate around objects and predict reasonable visualizations of the backside of objects. Additionally, with only a small proportion of input videos and texts describing camera motion, our model can use short text prompts to apply a range of camera motions to image-to-video and text-to-video generations (see Fig. 6).

三维结构、相机运动、视觉风格。

即使未加入专门鼓励三维一致性的训练数据或损失，模型也能绕物体旋转并合理可视化物体背面。此外，仅用少量描述相机运动的输入视频与文本，模型就能用短文本提示在图生视频与文生视频上施加多种相机运动（见图 6）。

5.6 Limitations

5.6 局限

Despite VideoPoet demonstrating highly competitive performance of LLMs relative to state-of-the-art models, certain limitations are still observed. For example, the RGB frame reconstruction from compressed and quantized tokens place an upper bound on the generative model’s visual fidelity. Second, the per-frame aesthetic biases in static scenes does not match the best baseline. This difference is largely due to a choice of training data, where we focus our training on more natural aesthetics and excluded some sources containing copyrighted images, such as LAION (Schuhmann et al., 2022), which is commonly used in other work. Finally, small objects and fine-grained details, especially when coupled with significant motions, remains difficult within the token-based modeling.

尽管 VideoPoet 相对先进模型展现出 LLM 的高度竞争力，仍有若干局限。例如，从压缩量化词元重建 RGB 帧，给生成模型的视觉保真度设了上限。其次，静态场景的逐帧审美偏差不及最佳基线；这很大程度上来自训练数据选择——我们更侧重自然审美，并排除了含版权图像的部分来源，如其他工作常用的 LAION (Schuhmann et al., 2022)。最后，小物体与细粒度细节，尤其在伴随大幅运动时，在基于词元的建模中仍然困难。

6 Conclusion

6 结论

VideoPoet demonstrates the potential of a large language model that is trained on discrete visual, text and audio tokens, in generating videos of compelling state-of-the-art quality. A particular strength of our model lies in its ability to generate high-fidelity, large, and complex motions. Our large language model formulation benefits from training across a variety of multimodal tasks with a unified architecture and vocabulary. Consequently, the pretrained model is adept at multi-task video creation, and serves as a foundation for a diverse variety of video generation related capabilities, including multiple forms of editing.

VideoPoet 表明：在离散视觉、文本与音频词元上训练的大语言模型，有潜力生成具有竞争力乃至先进质量的视频。模型的一个突出优势是生成高保真、大幅度且复杂的运动。LLM 表述受益于在统一架构与词表上跨多种多模态任务训练；因此预训练模型擅长多任务视频创作，并可作为多样视频生成相关能力——包括多种编辑形式——的基础。

Acknowledgements

致谢

We give special thanks to Alex Siegman, Victor Gomes, and Brendan Jou for managing computing resources. We also give thanks to Aren Jansen, Marco Tagliasacchi, Neil Zeghidour, John Hershey for audio tokenization and processing, Angad Singh for storyboarding in “Rookie the Raccoon”, Cordelia Schmid for research discussions, David Salesin, Tomas Izo, and Rahul Sukthankar for their support, and Jay Yagnik for the initial concept.

特别感谢 Alex Siegman、Victor Gomes 与 Brendan Jou 管理计算资源。亦感谢 Aren Jansen、Marco Tagliasacchi、Neil Zeghidour、John Hershey 在音频分词与处理上的帮助，Angad Singh 为 “Rookie the Raccoon” 分镜，Cordelia Schmid 的研究讨论，David Salesin、Tomas Izo 与 Rahul Sukthankar 的支持，以及 Jay Yagnik 的初始构想。

Impact Statement

影响声明

This paper introduces research aimed at advancing the field of video generation and introduces a new tool to enhance human creativity. In considering the societal impact of our video generation model, VideoPoet, several concerns emerge, including the potential for misuse and ethical considerations. Misuse may entail the creation of deceptive or harmful content, such as misinformation or deepfakes. Ethical considerations encompass ensuring that generated content adheres to ethical standards, avoids perpetuating harmful stereotypes, and respects cultural sensitivities. To mitigate these risks, we adhere to evolving strategies developed within our community. This includes using digital watermarking in generated videos to enable traceability and accountability, and maintain transparency in our model design to foster trust.

本文介绍旨在推进视频生成领域的研究，并引入增强人类创造力的新工具。审视 VideoPoet 的社会影响时，若干关切浮现，包括潜在误用与伦理考量。误用可能包括制造误导或有害内容，如虚假信息或深度伪造。伦理考量涵盖确保生成内容符合伦理标准、避免固化有害刻板印象，并尊重文化敏感性。为缓解这些风险，我们遵循社区内不断演进的策略，包括对生成视频使用数字水印以便追溯与问责，并保持模型设计透明以增进信任。

References

参考文献

Agostinelli et al. (2023) Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325 , 2023.

Akbari et al. (2023) Akbari, H., Kondratyuk, D., Cui, Y., Hornung, R., Wang, H., and Adam, H. Alternating gradient descent and mixture-of-experts for integrated multimodal perception. arXiv preprint arXiv:2305.06324 , 2023.

Anil et al. (2023) Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 , 2023.

Bar-Tal et al. (2024) Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Li, Y., Michaeli, T., et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945 , 2024.

Blattmann et al. (2023a) Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023a.

Blattmann et al. (2023b) Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR , pp. 22563–22575, 2023b.

Bommasani et al. (2021) Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Brooks et al. (2023) Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In CVPR , pp. 18392–18402, 2023.

Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurIPS , 33:1877–1901, 2020.

Carreira et al. (2018) Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., and Zisserman, A. A short note about kinetics-600. arXiv preprint arXiv:1808.01340 , 2018.

Ceylan et al. (2023) Ceylan, D., Huang, C.-H. P., and Mitra, N. J. Pix2video: Video editing using image diffusion. In CVPR , pp. 23206–23217, 2023.

Chai et al. (2023) Chai, W., Guo, X., Wang, G., and Lu, Y. Stablevideo: Text-driven consistency-aware diffusion video editing. In CVPR , pp. 23040–23050, 2023.

Chang et al. (2022) Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In CVPR , pp. 11315–11325, 2022.

Chang et al. (2023) Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023.

Chen et al. (2023a) Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q., Wang, X., et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512 , 2023a.

Chen et al. (2023b) Chen, W., Wu, J., Xie, P., Wu, H., Li, J., Xia, X., Xiao, X., and Lin, L. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840 , 2023b.

Chiu et al. (2023) Chiu, M.-C., Chen, P.-Y., and Ma, X. Better may not be fairer: A study on subgroup discrepancy in image classification. In ICCV , pp. 4956–4966, 2023.

Chowdhery et al. (2022) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. PaLM: Scaling language modeling with pathways. arXiv:2204.02311 , 2022.

Ding et al. (2021) Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. Cogview: Mastering text-to-image generation via transformers. NeurIPS , pp. 19822–19835, 2021.

Driess et al. (2023) Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

Du et al. (2022) Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. GLaMs: Efficient scaling of language models with mixture-of-experts. In ICML , 2022.

Esser et al. (2020) Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In CVPR , pp. 12868–12878, 2020.

Esser et al. (2023) Esser, P., Chiu, J., Atighehchian, P., Granskog, J., and Germanidis, A. Structure and content-guided video synthesis with diffusion models. In CVPR , pp. 7346–7356, 2023.

Feng et al. (2023) Feng, R., Weng, W., Wang, Y., Yuan, Y., Bao, J., Luo, C., Chen, Z., and Guo, B. Ccedit: Creative and controllable video editing via diffusion models. arXiv preprint arXiv:2309.16496 , 2023.

Ge et al. (2023) Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.-B., Liu, M.-Y., and Balaji, Y. Preserve your own correlation: A noise prior for video diffusion models. In CVPR , pp. 22930–22941, 2023.

Geyer et al. (2023) Geyer, M., Bar-Tal, O., Bagon, S., and Dekel, T. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373 , 2023.

Goyal et al. (2017) Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al. The “something something” video database for learning and evaluating visual common sense. In ICCV , 2017.

Guo et al. (2023) Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023.

Gupta et al. (2022) Gupta, A., Tian, S., Zhang, Y., Wu, J., Martín-Martín, R., and Fei-Fei, L. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206.11894 , 2022.

Gupta et al. (2023) Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Fei-Fei, L., Essa, I., Jiang, L., and Lezama, J. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 , 2023.

He et al. (2023) He, Y., Yang, T., Zhang, Y., Shan, Y., and Chen, Q. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 , 2(3):4, 2023.

Hershey et al. (2017) Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., Slaney, M., Weiss, R., and Wilson, K. Cnn architectures for large-scale audio classification. In ICASSP , 2017. URL https://arxiv.org/abs/1609.09430 .

Ho & Salimans (2022) Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Ho et al. (2022a) Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022a.

Ho et al. (2022b) Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. arXiv:2204.03458 , 2022b.

Hoffmann et al. (2022) Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

Hong et al. (2022) Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 , 2022.

Hu et al. (2023) Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., and Corrado, G. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080 , 2023.

Li et al. (2023) Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. StarCoder: may the source be with you! arXiv:2305.06161 , 2023.

Liew et al. (2023) Liew, J. H., Yan, H., Zhang, J., Xu, Z., and Feng, J. Magicedit: High-fidelity and temporally coherent video editing. arXiv preprint arXiv:2308.14749 , 2023.

Meng et al. (2021) Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

Nash et al. (2022) Nash, C., Carreira, J., Walker, J., Barr, I., Jaegle, A., Malinowski, M., and Battaglia, P. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.09494 , 2022.

OpenAI (2023) OpenAI. GPT-4 technical report. arXiv:2303.08774 , 2023.

Perazzi et al. (2016) Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., and Sorkine-Hornung, A. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR , 2016.

Pika (2023) Pika. Pika 1.0, 2023. URL https://pika.art/launch .

Podell et al. (2023) Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Raffel et al. (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(1):5485–5551, 2020.

Ramesh et al. (2021) Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 , 2021.

Ramesh et al. (2022) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

Ranftl et al. (2020) Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., and Koltun, V. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI , 44(3):1623–1637, 2020.

Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR , pp. 10684–10695, 2022.

Rubenstein et al. (2023) Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., Quitry, F. d. C., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925 , 2023.

Runway (2023) Runway. Gen2, 2023. URL https://runwayml.com/ .

Saharia et al. (2022) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS , 35:36479–36494, 2022.

Saito et al. (2020) Saito, M., Saito, S., Koyama, M., and Kobayashi, S. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. IJCV , 128(10):2586–2606, 2020.

Schuhmann et al. (2022) Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278–25294, 2022.

Schumann et al. (2021) Schumann, C., Ricco, S., Prabhu, U., Ferrari, V., and Pantofaru, C. A step toward more inclusive people annotations for fairness. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pp. 916–925, 2021.

Schumann et al. (2023) Schumann, C., Olanubi, G. O., Wright, A., Monk, E., Heldreth, C., and Ricco, S. Consensus and subjectivity of skin tone annotation for ML fairness. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. URL https://openreview.net/forum?id=L9I9FhHfS3 .

Singer et al. (2022) Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

Soomro et al. (2012) Soomro, K., Zamir, A. R., and Shah, M. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Sun et al. (2022) Sun, D., Herrmann, C., Reda, F., Rubinstein, M., Fleet, D. J., and Freeman, W. T. Disentangling architecture and training for optical flow. In ECCV , 2022.

Tang et al. (2023) Tang, Z., Yang, Z., Zhu, C., Zeng, M., and Bansal, M. Any-to-any generation via composable diffusion. arXiv preprint arXiv:2305.11846 , 2023.

Tu et al. (2022) Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., and Li, Y. Maxvit: Multi-axis vision transformer. In ECCV , pp. 459–479, 2022.

Unterthiner et al. (2018) Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 , 2018.

Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. NeurIPS , 30, 2017.

Villegas et al. (2022) Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210.02399 , 2022.

Voleti et al. (2022) Voleti, V., Jolicoeur-Martineau, A., and Pal, C. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. NeurIPS , 35:23371–23385, 2022.

Wang et al. (2023a) Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., and Zhang, S. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 , 2023a.

Wang et al. (2023b) Wang, W., Xie, K., Liu, Z., Chen, H., Cao, Y., Wang, X., and Shen, C. Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.17599 , 2023b.

Wang et al. (2023c) Wang, W., Yang, H., Tuo, Z., He, H., Zhu, J., Fu, J., and Liu, J. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305.10874 , 2023c.

Wang et al. (2023d) Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Chen, X., Wang, Y., Luo, P., Liu, Z., et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 , 2023d.

Wu et al. (2021) Wu, C., Huang, L., Zhang, Q., Li, B., Ji, L., Yang, F., Sapiro, G., and Duan, N. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806 , 2021.

Xu et al. (2016) Xu, J., Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. In CVPR , pp. 5288–5296, 2016.

Yan et al. (2021) Yan, W., Zhang, Y., Abbeel, P., and Srinivas, A. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157 , 2021.

Yu et al. (2022) Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 , 2022.

Yu et al. (2023a) Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I., et al. MAGVIT: Masked generative video transformer. In CVPR , 2023a.

Yu et al. (2023b) Yu, L., Cheng, Y., Wang, Z., Kumar, V., Macherey, W., Huang, Y., Ross, D. A., Essa, I., Bisk, Y., Yang, M.-H., et al. SPAE: Semantic pyramid autoencoder for multimodal generation with frozen llms. In NeurIPS , 2023b.

Yu et al. (2024) Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Hauptmann, A. G., et al. Language model beats diffusion–tokenizer is key to visual generation. In ICLR , 2024.

Yu et al. (2023c) Yu, S., Sohn, K., Kim, S., and Shin, J. Video probabilistic diffusion models in projected latent space. In CVPR , pp. 18456–18466, 2023c.

Zeghidour et al. (2021) Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30:495–507, 2021.

Zeng et al. (2023) Zeng, Y., Wei, G., Zheng, J., Zou, J., Wei, Y., Zhang, Y., and Li, H. Make pixels dance: High-dynamic video generation. arXiv preprint arXiv:2311.10982 , 2023.

Zhang et al. (2023a) Zhang, D. J., Wu, J. Z., Liu, J.-W., Zhao, R., Ran, L., Gu, Y., Gao, D., and Shou, M. Z. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309.15818 , 2023a.

Zhang et al. (2023b) Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In CVPR , pp. 3836–3847, 2023b.

Zhang et al. (2023c) Zhang, Y., Jiang, L., Turk, G., and Yang, D. Auditing gender presentation differences in text-to-image models. arXiv preprint arXiv:2302.03675 , 2023c.

Zhou et al. (2023) Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 , 2023.

Zhou et al. (2022) Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., and Feng, J. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 , 2022.

Zitkovich et al. (2023) Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In CoRL , 2023.

Appendix A Appendix

附录 A

A.1 Responsible AI and Fairness Analysis

A.1 负责任 AI 与公平性分析

We evaluate whether the generated outputs of our model are fair regarding protected attributes such as (1) Perceived Age (2) Perceived Gender Expression (3) Perceived Skin Tone. We construct 306 prompts with template — “a {profession or people descriptor} looking {adverb} at the camera” with “profession” being crawled from the US Bureau of Labor and Statistics and “people descriptors” including emotion state, socioeconomic class, etc. The “adverb” is used to generate semantically unchanged prompt templates such as “straightly” or “directly”. We generate 8 videos for each prompt and for each generated video we infer an approximation of the expressed attribute regarding the 3 protected attributes. Across 10 prompts that have the same semantic meaning but different “adverbs”, we observe our outputs generally introduced a stronger distribution shift toward “Young Adults” (age 18-35), “Male” and “Light Skin Tone”. However, we observe changing the “adverb” in the prompt template can significantly alter the output distributions. Therefore, our model can be prompted to produce outputs with non-uniform distributions across these groups, but also possess the ability of being prompted to enhance uniformity, though prompts are semantically unchanged. While research has been conducted in the image generation and recognition domain (Zhang et al., 2023c; Schumann et al., 2021; Schumann et al., 2023; Chiu et al., 2023), this finding highlights the importance of continued research to develop strategies to mitigate issues and improve fairness for video generation.

我们评估模型生成输出在受保护属性上是否公平，包括：(1) 感知年龄；(2) 感知性别表达；(3) 感知肤色。构造 306 条模板提示——「a {profession or people descriptor} looking {adverb} at the camera」，其中 profession 来自美国劳工统计局爬取，people descriptors 含情绪状态、社会经济阶层等；adverb 用于生成语义不变的提示模板，如 “straightly” 或 “directly”。每条提示生成 8 条视频，并对每条视频推断上述 3 个受保护属性的近似表达。在 10 条语义相同但 adverb 不同的提示上，我们观察到输出总体更偏向「青年成人」（18–35 岁）、「男性」与「浅肤色」。但更换提示模板中的 adverb 会显著改变输出分布。因此，模型既可被提示产生跨群体非均匀分布，也可在提示语义不变的情况下被提示增强均匀性。尽管图像生成与识别领域已有相关研究 (Zhang et al., 2023c; Schumann et al., 2021; Schumann et al., 2023; Chiu et al., 2023)，这一发现凸显继续研究以缓解问题、提升视频生成公平性的重要性。

A.2 Model Scale and Performance

A.2 模型规模与表现

To analyze model performance versus model scale, we use a subset of the training set without text-paired data and a slightly different task prompt design. We evaluate the video generation quality using FVD (Unterthiner et al., 2018) and audio generation quality using the Fréchet Audio Distance (FAD), which uses the VGGish model as the embedding function (Hershey et al., 2017). Both FVD and FAD metrics are calculated using a held-out subset of 25 thousand videos.

为分析模型表现相对规模的变化，我们使用不含文本配对数据的训练子集，以及略有不同的任务提示设计。用 FVD (Unterthiner et al., 2018) 评估视频生成质量，用 Fréchet 音频距离（FAD，以 VGGish 为嵌入函数） (Hershey et al., 2017) 评估音频生成质量。FVD 与 FAD 均在留出的 2.5 万视频子集上计算。

Fig. 5 shows that as the model size grows and the amount of training data increases, performance improves across visual and audiovisual tasks. After obtaining the above results, we retrain our 1B and 8B models using the task design and text-paired training data discussed in Section 3. Section A.2.1 shows a qualitative comparison of the 1B and 8B pretrained models. Increasing the model size improved temporal consistency, prompt fidelity, and motion dynamics while adding capabilities for limited text rendering, spatial understanding, and counting.

图 5 显示，随着模型变大、训练数据增多，视觉与视听任务表现均提升。得到上述结果后，我们按第 3 节讨论的任务设计与文本配对训练数据重新训练 10 亿与 80 亿参数模型。第 A.2.1 节给出 10 亿与 80 亿预训练模型的定性比较。增大模型改善了时间一致性、提示保真与运动动态，并增加了有限文字渲染、空间理解与计数等能力。

(a) Video generation quality in FVD ($\downarrow$).

(a) 以 FVD（↓）衡量的视频生成质量。

A.2.1 Qualitative Comparison of 1B and 8B models

prompt: A portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House holding a sign on the chest that says Welcome Friends!

prompt: A kangaroo holding a sign with the letter A on it

prompt: A photo of an astronaut riding a horse in the forest. There is a river in front of them with water lilies

prompt: A zoomed out map of the United States made out of sushi. It is on a table next to a glass of red wine. Pieces of sushi disappear one by one

prompt: Rotating around a vase holding a dozen roses

Figure 5: A comparison of a 1B (left) and 8B (right) parameter models on the same prompt and settings.

A.2.1 10 亿与 80 亿参数模型的定性比较

提示： A portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House holding a sign on the chest that says Welcome Friends!

提示： A kangaroo holding a sign with the letter A on it

提示： A photo of an astronaut riding a horse in the forest. There is a river in front of them with water lilies

提示： A zoomed out map of the United States made out of sushi. It is on a table next to a glass of red wine. Pieces of sushi disappear one by one

提示： Rotating around a vase holding a dozen roses

图 5：同一提示与设定下 10 亿（左）与 80 亿（右）参数模型的比较。

In Figure 5, we show outputs of 1B and 8B parameter models on the same prompts. Four frames from the best video output of each model in a batch of four text-to-video samples were selected to represent the model. In the first row, the 1B model is unstable with large changes to the subject over time and misses elements from the complex prompt. This prompt was originally used for scaling comparisons in (Yu et al., 2022), and compared to a dedicated image-only model, our model does not preserve text as well given the training data used. In the second row, we use a simpler text task and show that the 8B model can represent a single letter clearly, but the 1B model still produces artifacts. In the third row, we show that the 8B model learns spatial positioning such as the river being in front of the astronaut and horse. In the fourth row, we show that the 8B parameter model learned a stop motion style to have items disappear “one by one” and can follow a complicated layout from a long prompt. In contrast, the 1B model includes all of the nouns, but is unstable over time and does not follow the layout indicated in the prompt. In the bottom row, we show that the 8B model understands counts of objects in that it displays a full bouquet (though 12 roses are not explicitly in frame) and smooth consistent motion as opposed to the 5 roses and distorting objects produced by the 1B model. Overall, scaling the model improved temporal consistency, prompt fidelity, and motion dynamics while adding capabilities for limited text rendering, spatial understanding, and counting.

图 5 展示同一提示下 10 亿与 80 亿参数模型的输出。从每模型一批四条文生视频样本中选取最佳视频的四帧为代表。第一行中，10 亿模型主体随时间大幅变化而不稳定，并遗漏复杂提示中的元素。该提示原用于 (Yu et al., 2022) 的缩放比较；相对专用图像模型，在我们所用训练数据下，本模型对文字的保持较弱。第二行使用更简单的文本任务：80 亿模型能清晰表示单个字母，10 亿模型仍有伪影。第三行显示 80 亿模型学到空间位置，例如河在宇航员与马前方。第四行显示 80 亿模型学到定格动画式「逐个消失」，并能遵循长提示中的复杂布局；相对地，10 亿模型包含全部名词，但时间上不稳定且不遵循提示布局。最下行显示 80 亿模型理解物体计数：呈现完整花束（虽未显式拍到 12 朵玫瑰）与平滑一致运动，而 10 亿模型只有约 5 朵玫瑰且物体扭曲。总体而言，放大模型改善了时间一致性、提示保真与运动动态，并增加了有限文字渲染、空间理解与计数能力。

A.3 Additional Generated Examples

A.3 更多生成示例

We include most generated videos in the supplementary materials for an enhanced visualization of motion and visual quality, in addition to Fig. 6 and Fig. 6.

除图 6 外，多数生成视频放在补充材料中，以便更好观察运动与视觉质量。

Original Video

Outpainted Video

Stylized Video Prompt: A gingerbread and candy train on a track

Figure 6: Example of zero-shot video editing via task chaining (outpainting and stylization) – the original video is first outpainted and then stylized via a text prompt.

原始视频

外扩后视频

风格化视频 提示： A gingerbread and candy train on a track

图 6：通过任务串联实现零样本视频编辑的示例（外扩与风格化）——原始视频先外扩，再按文本提示风格化。

Camera Motion: Arc shot

Camera Motion: FPV drone shot

Figure 6: Examples of directed camera movement from the same initial frame.

相机运动：Arc shot

相机运动：FPV drone shot

图 6：同一初始帧上受控相机运动的示例。

A.4 Video Stylization

A.4 视频风格化

To perform video stylization, we follow an approach inspired by (Zhang et al., 2023b; Chen et al., 2023b; Esser et al., 2023) to predict videos from the combination of text, optical flow, and depth signals. On a subset of steps, we also condition on the first video frame. As described in (Esser et al., 2023), the text will generally define the “content” or appearance of the output and the optical flow and depth control the “structure.” In contrast to the diffusion-based approaches that usually use external cross-attention networks (Zhang et al., 2023b) or latent blending (Meng et al., 2021) for stylization, our approach is more closely related to machine translation using large language models in that we only need to provide the structure and text as a prefix to a language model.

做视频风格化时，我们遵循受 (Zhang et al., 2023b; Chen et al., 2023b; Esser et al., 2023) 启发的思路，由文本、光流与深度信号的组合预测视频。在部分步骤上还以视频首帧为条件。如 (Esser et al., 2023) 所述，文本通常定义输出的「内容」或外观，光流与深度控制「结构」。与通常用外部交叉注意力网络 (Zhang et al., 2023b) 或潜空间混合 (Meng et al., 2021) 做风格化的扩散方法不同，我们的方法更接近用大语言模型做机器翻译：只需把结构与文本作为语言模型的前缀。

To perform the task, we estimate optical flow from RAFT (Sun et al., 2022) and produce monocular depth maps from MIDAS (Ranftl et al., 2020), and then normalize and concatenate on the channel dimension. This conveniently produces the same number of channels as the RGB ground truth and so can be tokenized in the same fashion as RGB videos with the MAGVIT-v2 tokenizer without retraining the tokenizer. The task of stylization is to reconstruct the ground truth video from the given optical flow, depth, and text information. During inference, we apply optical flow and depth estimation on an input video but then vary the text prompt to generate a new style, e.g. “cartoon”.

执行该任务时，用 RAFT (Sun et al., 2022) 估计光流，用 MIDAS (Ranftl et al., 2020) 生成单目深度图，再归一化并在通道维拼接。这恰好得到与 RGB 真值相同的通道数，因而可用 MAGVIT-v2 分词器按与 RGB 视频相同的方式分词，无需重训分词器。风格化任务是从给定光流、深度与文本重建真值视频。推理时对输入视频做光流与深度估计，再更换文本提示以生成新风格，例如 “cartoon”。

Model	CLIPSIM
Control-A-Video (Chen et al., 2023b)[depth]	0.3246
VideoPoet (Ours)	0.3417

Table 3: Comparison on video stylization. VideoPoet outperforms Control-A-Video by a large margin.

模型	CLIPSIM
Control-A-Video (Chen et al., 2023b)[depth]	0.3246
VideoPoet (Ours)	0.3417

表 3：视频风格化比较。 VideoPoet 大幅优于 Control-A-Video。

To evaluate stylization capabilities, we choose 20 videos from the public DAVIS 2016[^2] (Perazzi et al., 2016) dataset and provide 2 style prompts for each video. For more details, please refer to Section A.5.7. Following (Esser et al., 2023), we evaluated the CLIP-embedding consistency between each frame and the text prompt to determine if the stylization results matches the text. As shown in Table 3, VideoPoet outperforms Control-A-Video conditioned on depth by a large margin. We also conduct human evaluations as discussed above comparing with Control-A-Video (Chen et al., 2023b). Human raters consistently prefer our text fidelity and video quality as shown in Fig. 7.

为评估风格化能力，我们从公开 DAVIS 2016[^2] (Perazzi et al., 2016) 数据集选取 20 条视频，每条提供 2 个风格提示。更多细节见第 A.5.7 节。遵循 (Esser et al., 2023)，我们评估每帧与文本提示之间的 CLIP 嵌入一致性，以判断风格化结果是否匹配文本。如表 3，VideoPoet 大幅优于以深度为条件的 Control-A-Video。我们还按上文所述与 Control-A-Video (Chen et al., 2023b) 做人工评估；如图 7，评分者在文本保真与视频质量上一致偏好我们的结果。

Figure 7: Human side-by-side evaluations comparing VideoPoet with the video stylization model Control-a-video (Chen et al., 2023b). Raters prefer VideoPoet on both text fidelity and video quality. Green and pink bars represent the proportion of trials where VideoPoet was preferred over an alternative, or preferred less than an alternative, respectively.

图 7：VideoPoet 与视频风格化模型 Control-a-video (Chen et al., 2023b) 的人工并排评估。 评分者在文本保真与视频质量上均偏好 VideoPoet。绿/粉条分别表示 VideoPoet 相对对照更受偏好或更不受偏好的试验比例。

A.5 Additional Implementation and Evaluation Details

A.5 额外实现与评估细节

A.5.1 Additional Implementation Details

The unified vocabulary is constructed as follows: the initial 256 codes are reserved for special tokens and task prompts. Table 4 lists some examples of special tokens. Subsequently, the next 262,144 codes are allocated for image and video tokenization. This is followed by 4,096 audio codes. We also include a small text vocabulary of English words. Overall, this produces a total vocabulary size of approximately 300,000.

A.5.1 额外实现细节

统一词表构造如下：前 256 个码保留给特殊词元与任务提示。表 4 列出部分特殊词元示例。随后分配 262,144 个码给图像与视频分词，再接 4,096 个音频码。我们还纳入一个小型英文单词文本词表。总体词表大小约 300,000。

Special Token	Usage
	Beginning of sequence
	Task to perform for this sequence
	Beginning of the text input.
	End of the text input.
	Beginning of the visual input.
	End of the video input.
	Beginning of the audio input.
	End of the audio input.
	The source of the video to generate.
	Output resolution for the video.
	Beginning of the video output.
	End of the video output.
	Beginning of the audio output.
	End of the audio output.
	End of the entire sequence.

Table 4: List of representative special tokens used in training and inference.

特殊词元	用途
	序列开始
	本序列要执行的任务
	文本输入开始
	文本输入结束
	视觉输入开始
	视频输入结束
	音频输入开始
	音频输入结束
	待生成视频的来源
	视频输出分辨率
	视频输出开始
	视频输出结束
	音频输出开始
	音频输出结束
	整个序列结束

表 4：训练与推理中使用的代表性特殊词元列表。

Since the first frame is tokenized separately, MAGVIT-v2 allows images to be represented in the same vocabulary as video. In addition to being more compact, images provide many learnable characteristics that are not typically represented in videos, such as strong visual styles (e.g., art paintings), objects which are infrequently seen in video, rich captions, and significantly more text-image paired training data. When training on images, we resize the images to 128$\times$128 which are then tokenized to a latent shape of $(1,16,16)$, or 256 tokens. We scale the MAGVIT-v2 model’s size and train it on the datasets discussed in Section 5.1. The training follows two steps: image training, inflation (Yu et al., 2024) and video training. Due to images requiring fewer tokens, we can include roughly 5$\times$ more images per batch than videos, i.e. 256 image tokens vs. 1280 video tokens. We use up to a maximum of 64 text tokens for all of our experiments. For the token, the resolution is only specified for $128\times 224$ output, $128\times 128$ resolution is assumed otherwise.

由于首帧单独分词，MAGVIT-v2 允许图像与视频共享同一词表。除更紧凑外，图像还提供视频中通常缺少的可学习特性，如强烈视觉风格（例如画作）、视频中少见的物体、丰富描述，以及显著更多的图文配对训练数据。训练图像时缩放到 $128\times 128$，分词为形状 $(1,16,16)$ 即 256 个词元。我们放大 MAGVIT-v2 模型规模，并在第 5.1 节所述数据集上训练。训练分两步：图像训练、膨胀 (Yu et al., 2024) 与视频训练。因图像所需词元更少，每批可纳入约 $5\times$ 于视频的图像，即 256 个图像词元对 1280 个视频词元。所有实验文本词元上限为 64。 词元仅在输出为 $128\times 224$ 时指定，否则默认 $128\times 128$。

The video-to-video tasks use the COMMIT encoding (Yu et al., 2023a) to obtain the tokens for the tasks such as inpainting and outpainting. Text is encoded as T5 XL embeddings (Raffel et al., 2020) and are inserted into reserved sequence positions right after the token as shown in Fig. 2.

视频到视频任务使用 COMMIT 编码 (Yu et al., 2023a) 获取修复与外扩等任务的词元。文本编码为 T5 XL 嵌入 (Raffel et al., 2020)，并如图 2 插入 词元后的保留序列位置。

A.5.2 Super-resolution implementation details

We use a 1B model for the first $2\times$ spatial super-resolution stage and a 500M model for the second $2\times$ stage. The first super-resolution stage models videos of $17\times 448\times 256$ pixels with a token sequence of shape $(5,56,32)$. The second stage models videos of $17\times 896\times 512$ pixels with a token sequence of shape $(5,112,64)$. The token sequences are obtained with the same MAGVIT-v2 (Yu et al., 2024) tokenizer used for the base language model. The custom super-resolution transformer has local self-attention windows for vertical, horizontal and temporal layers of shape $(1,56,4),(1,8,32),(5,8,8)$ in the first stage and $(1,112,2),(1,4,64),(5,8,8)$ in the second stage, respectively (Fig. 3). The cross-attention layers attend to local windows in the low-resolution sequence isomorphic to self-attention windows but with half the spatial size.

A.5.2 超分实现细节

第一级 $2\times$ 空间超分使用 10 亿参数模型，第二级 $2\times$ 使用 5 亿参数模型。第一级建模 $17\times 448\times 256$ 像素视频，词元序列形状 $(5,56,32)$；第二级建模 $17\times 896\times 512$ 像素，序列形状 $(5,112,64)$。词元序列由与基座语言模型相同的 MAGVIT-v2 (Yu et al., 2024) 分词器得到。定制超分 Transformer 在第一级的竖直、水平与时间局部自注意力窗口形状分别为 $(1,56,4)$、$(1,8,32)$、$(5,8,8)$，第二级为 $(1,112,2)$、$(1,4,64)$、$(5,8,8)$（图 3）。交叉注意力层关注低分辨率序列中与自注意力窗口同构但空间尺寸减半的局部窗口。

We train the super-resolution stages on a dataset of 64M high-quality text-video pairs using the masked modeling objective of MAGVIT (Yu et al., 2023a), with token factorization into $k=2$ groups (Yu et al., 2024). During inference, we use the sampling algorithm of MAGVIT-v2 (Yu et al., 2024) with 24 sampling steps for each stage and classifier-free guidance scale (Ho & Salimans, 2022; Brooks et al., 2023) of $4.0/8.0$ for the text condition and $1.0/2.0$ for the low-resolution condition, in the first/second stage.

我们在 6,400 万高质量文视频对数据集上，用 MAGVIT (Yu et al., 2023a) 的掩码建模目标训练超分阶段，词元因子化为 $k=2$ 组 (Yu et al., 2024)。推理时采用 MAGVIT-v2 (Yu et al., 2024) 的采样算法，每级 24 步，无分类器引导尺度 (Ho & Salimans, 2022; Brooks et al., 2023) 在第一/第二级对文本条件为 $4.0/8.0$，对低分辨率条件为 $1.0/2.0$。

A.5.3 Additional Evaluation details

We measure CLIP similarity scores (Wu et al., 2021) following an implementation given by Villegas et al. (2022), measure FVD (Unterthiner et al., 2018) following Yu et al. (2023a) on UCF101 dataset and following Zhang et al. (2023a) on MSR-VTT, and measure Inception Score (IS) (Saito et al., 2020). When the evaluation protocol is on 16 frames, we discard the generated last frame to make a 16-frame video.

A.5.3 额外评估细节

我们按 Villegas et al. (2022) 的实现测量 CLIP 相似度 (Wu et al., 2021)，在 UCF101 上按 Yu et al. (2023a)、在 MSR-VTT 上按 Zhang et al. (2023a) 测量 FVD (Unterthiner et al., 2018)，并测量 Inception Score（IS） (Saito et al., 2020)。当评估协议针对 16 帧时，丢弃生成的最后一帧以得到 16 帧视频。

A.5.4 Additional Human Evaluation details

Figure 8: Example screenshot of the user interface for human side-by-side comparisons.

A.5.4 额外人工评估细节

图 8：人工并排比较用户界面截图示例。

Figure 8 shows an example screenshot of our side-by-side UI for comparing models. We used a team of 7 human raters to complete all ratings. To achieve best results, we call VideoPoet using the negative prompt “a still shot of an ugly cartoon, slideshow of an empty scene, low resolution, distorted and disfigured” and rewrite the given prompt by appending the string highly detailed, cinematic, arc shot, high contrast, soft lighting, 8k.

图 8 展示用于比较模型的并排 UI 截图示例。由 7 名人工评分者完成全部评分。为获得最佳结果，调用 VideoPoet 时使用负面提示 “a still shot of an ugly cartoon, slideshow of an empty scene, low resolution, distorted and disfigured”，并在给定提示后追加 highly detailed, cinematic, arc shot, high contrast, soft lighting, 8k。

A.5.5 Zero-shot Text-to-Video Evaluation Settings

We report the details of our zero-shot text-to-video settings here. We note that some details are missing in previous papers and different papers use different settings. Hence, we provide all the details and hope this evaluation setting can serve as a standard text-to-video generation benchmark. Our results are reported on the 8B model and we adopt classifier-free guidance (Ho & Salimans, 2022).

A.5.5 零样本文生视频评估设定

此处报告零样本文生视频设定细节。我们注意到既往论文常缺失部分细节，且设定各异。因此我们给出全部细节，并希望该评估设定能成为文生视频生成的标准基准。结果基于 80 亿参数模型，并采用无分类器引导 (Ho & Salimans, 2022)。

All metrics are evaluated on generated videos containing 16 frames with a resolution of 256 x 256. We first generate videos of 128 x 128 resolution and then resize to 256 x 256 via bicubic upsampling.

所有指标在含 16 帧、分辨率为 $256\times 256$ 的生成视频上评估。我们先生成 $128\times 128$ 视频，再经双三次上采样到 $256\times 256$。

Zero-shot MSR-VTT. For CLIP score, we used all 59,794 captions from the MSR-VTT test set. We use CLIP ViT-B/16 model following Phenaki (Villegas et al., 2022). We note that some papers use other CLIP models, e.g., VideoLDM (Blattmann et al., 2023b) uses ViT-B/32. Our CLIP score evaluated on the ViT-B/32 backbone for MSR-VTT is 30.01. For the FVD metric, to evaluate on a wide range of captions as well as to be comparable with previous papers that evaluate on 2,048 videos, we evaluate on the first 40,960 captions in the MSR-VTT test set. More specifically, we report the FVD metrics on 2048 videos with 20 repeats. The FVD real features are extracted from 2,048 videos sampled from the MSR-VTT test set. We sample the central 16 frames of each real video, without any temporal downsampling, i.e., we use the original fps in the MSR-VTT dataset (30 fps as reported in Xu et al. (2016)). The FVD is evaluated with an I3D model trained on Kinetics-400.

零样本 MSR-VTT。CLIP 分数使用 MSR-VTT 测试集全部 59,794 条描述。按 Phenaki (Villegas et al., 2022) 使用 CLIP ViT-B/16；部分论文使用其他 CLIP，例如 VideoLDM (Blattmann et al., 2023b) 用 ViT-B/32。我们在 ViT-B/32 骨干上测得的 MSR-VTT CLIP 分数为 30.01。FVD 方面，为覆盖广泛描述并与评估 2,048 条视频的既往论文可比，我们在 MSR-VTT 测试集前 40,960 条描述上评估：报告 2048 条视频、重复 20 次的 FVD。FVD 真实特征从 MSR-VTT 测试集采样的 2,048 条视频提取；每条真实视频取中心 16 帧，不做时间下采样，即使用 MSR-VTT 原始帧率（Xu et al. (2016) 报告为 30 fps）。FVD 用在 Kinetics-400 上训练的 I3D 评估。

Zero-shot UCF-101. Following VDM (Ho et al., 2022b), we sample 10,000 videos from the UCF-101 test set and use their categories as the text prompts to generate 10,000 videos. We use the class text prompts provided in PYoCo (Ge et al., 2023) to represent the 101 categories. To compute the FVD real features, we sample 10K videos from the training set, following TGAN2 (Saito et al., 2020). We sample the central 16 frames for each real video , without any temporal downsampling, i.e., we use the original fps in the UCF-101 dataset (25 fps as reported in (Soomro et al., 2012)). The FVD metric is evaluated with an I3D model trained on Kinetics-400 and the IS metric is evaluated with a C3D model trained on UCF-101.

零样本 UCF-101。遵循 VDM (Ho et al., 2022b)，从 UCF-101 测试集采样 10,000 条视频，以其类别为文本提示生成 10,000 条视频。用 PYoCo (Ge et al., 2023) 提供的类别文本提示表示 101 类。计算 FVD 真实特征时，按 TGAN2 (Saito et al., 2020) 从训练集采样 1 万条视频。每条真实视频取中心 16 帧，不做时间下采样，即使用 UCF-101 原始帧率（(Soomro et al., 2012) 报告为 25 fps）。FVD 用 Kinetics-400 上训练的 I3D，IS 用 UCF-101 上训练的 C3D。

A.5.6 Self-Supervised Tasks Evaluation Settings

Self-supervised learning tasks include frame prediction on K600 with 5 frames as condition, as well as inpainting and outpainting on SSv2. FVD (Unterthiner et al., 2018) is used as the primary metric, calculated with 16 frames at 128$\times$128 resolution. We follow MAGVIT (Yu et al., 2023a) in evaluating these tasks against the respective real distribution, using 50000$\times$4 samples for K600 and 50000 samples for SSv2.

A.5.6 自监督任务评估设定

自监督任务包括 K600 上以 5 帧为条件的帧预测，以及 SSv2 上的修复与外扩。主指标为 FVD (Unterthiner et al., 2018)，在 $128\times 128$ 分辨率、16 帧上计算。我们遵循 MAGVIT (Yu et al., 2023a)，相对各自真实分布评估这些任务：K600 用 $50000\times 4$ 样本，SSv2 用 50000 样本。

A.5.7 Stylization Evaluation on DAVIS

To evaluate the CLIP similarity score and human preference on video stylization, we use the following set of videos and prompts. We select 20 videos from DAVIS 2016 (Perazzi et al., 2016), and for each video we take 16 frames starting from the initial frame specified below and evaluate stylization on the two text prompts specified below. To be easily reproducible, we use a central square crop at the height of the video and evaluate the output videos at 256x256 resolution. We use CLIP-B/16 for the similarity score. Several prompts below are used in or inspired by previous work (Esser et al., 2023; Chen et al., 2023b; Liew et al., 2023).

A.5.7 DAVIS 上的风格化评估

为评估风格化的 CLIP 相似度与人工偏好，我们使用如下视频与提示集合。从 DAVIS 2016 (Perazzi et al., 2016) 选取 20 条视频；对每条视频，从下方指定的起始帧起取 16 帧，并在下方指定的两条文本提示上评估风格化。为便于复现，按视频高度做中心方形裁剪，并在 $256\times 256$ 分辨率评估输出。相似度使用 CLIP-B/16。若干提示用于或受既往工作启发 (Esser et al., 2023; Chen et al., 2023b; Liew et al., 2023)。

video name	starting frame	first text prompt
elephant	10	oil painting of an elephant walking away
elephant	10	cartoon animation of an elephant walking through dirt surrounded by boulders
car-turn	40	car on a snowcovered road in the countryside
car-turn	40	8-bit pixelated car driving down the road
dog-agility	0	a dog in the style of a comic book
dog-agility	0	a dog running through a field of poles in the style of cyberpunk
bmx-bumps	10	riding a bicycle on a rainbow track in space with stars and planets in the background
bmx-bumps	10	riding a bicycle on a dirt track in the style of a graphic novel
train	0	a gingerbread steam train made of candy
train	0	a train in lava
bus	0	a black and white drawing of a bus
bus	0	a bus in cyberpunk style
lucia	0	an astronaut walking on mars
lucia	0	a claymation animation of a woman walking
tennis	15	a robot throwing a laser ball
tennis	15	astronaut playing tennis on the surface of the moon
bear	60	a polar bear exploring on an iceberg
bear	60	a space bear walking beneath the stars
flamingo	0	2D vector animation of a group of flamingos standing near some rocks and water
flamingo	0	oil painting of pink flamingos wading
hike	0	a green alien explorer hiking in the mountains
hike	0	paper cut-out mountains with a paper cut-out hiker
goat	59	a tiger prowling along the ridge above a jungle
goat	59	a dragon prowling over a crater on the moon
parkour	60	a man jumping over rocks in a red sandstone canyon
parkour	60	a robot dodging through an obstacle course
cows	10	a pig standing in the mud
cows	10	a robotic cow walking along a muddy road
camel	10	a camel robot on a snowy day
camel	10	toy camel standing on dirt near a fence
blackswan	0	a watercolor painting of a white swan
blackswan	0	a crochet black swan swims in a pond with rocks and vegetation
dog	20	a cat walking
dog	20	a dalmatian dog walking
kite-surf	10	a sand surfer kicking up sand in the desert
kite-surf	10	kite surfer in the ocean at sunset
libby	0	chinese ink painting of a dog running
libby	0	3D animation of a small dog running through grass
horsejump-high	0	a cartoon of a magical flying horse jumping over an obstacle
horsejump-high	0	person rides on a horse while jumping over an obstacle with an aurora borealis in the background

Table 5: DAVIS stylization evaluation settings.

视频名	起始帧	第一条文本提示
elephant	10	oil painting of an elephant walking away
elephant	10	cartoon animation of an elephant walking through dirt surrounded by boulders
car-turn	40	car on a snowcovered road in the countryside
car-turn	40	8-bit pixelated car driving down the road
dog-agility	0	a dog in the style of a comic book
dog-agility	0	a dog running through a field of poles in the style of cyberpunk
bmx-bumps	10	riding a bicycle on a rainbow track in space with stars and planets in the background
bmx-bumps	10	riding a bicycle on a dirt track in the style of a graphic novel
train	0	a gingerbread steam train made of candy
train	0	a train in lava
bus	0	a black and white drawing of a bus
bus	0	a bus in cyberpunk style
lucia	0	an astronaut walking on mars
lucia	0	a claymation animation of a woman walking
tennis	15	a robot throwing a laser ball
tennis	15	astronaut playing tennis on the surface of the moon
bear	60	a polar bear exploring on an iceberg
bear	60	a space bear walking beneath the stars
flamingo	0	2D vector animation of a group of flamingos standing near some rocks and water
flamingo	0	oil painting of pink flamingos wading
hike	0	a green alien explorer hiking in the mountains
hike	0	paper cut-out mountains with a paper cut-out hiker
goat	59	a tiger prowling along the ridge above a jungle
goat	59	a dragon prowling over a crater on the moon
parkour	60	a man jumping over rocks in a red sandstone canyon
parkour	60	a robot dodging through an obstacle course
cows	10	a pig standing in the mud
cows	10	a robotic cow walking along a muddy road
camel	10	a camel robot on a snowy day
camel	10	toy camel standing on dirt near a fence
blackswan	0	a watercolor painting of a white swan
blackswan	0	a crochet black swan swims in a pond with rocks and vegetation
dog	20	a cat walking
dog	20	a dalmatian dog walking
kite-surf	10	a sand surfer kicking up sand in the desert
kite-surf	10	kite surfer in the ocean at sunset
libby	0	chinese ink painting of a dog running
libby	0	3D animation of a small dog running through grass
horsejump-high	0	a cartoon of a magical flying horse jumping over an obstacle
horsejump-high	0	person rides on a horse while jumping over an obstacle with an aurora borealis in the background

表 5：DAVIS 风格化评估设定。