Visual Autoregressive Modeling:  Scalable Image Generation via Next-Scale Prediction

视觉自回归建模：通过下一尺度预测实现可扩展图像生成

Keyu Tian,  Yi Jiang,  Zehuan Yuan,  Bingyue Peng,  Liwei Wang † † thanks: Corresponding authors:   wanglw@pku.edu.cn , yuanzehuan@bytedance.com ;  $†$: project lead Email:  yuanzehuan@bytedance.com, bingyue.peng@bytedance.com, wanglw@pku.edu.cn Try and explore our online demo at:˜  https://var.vision Codes and models:˜  https://github.com/FoundationVision/VAR Affiliation:  Peking University    Affiliation:  Bytedance Inc Email:  keyutian@stu.pku.edu.cn Email:  jiangyi.enjoy@bytedance.com

Abstract

摘要

We present Visual AutoRegressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine “next-scale prediction” or “next-resolution prediction”, diverging from the standard raster-scan “next-token prediction”. This simple, intuitive methodology allows autoregressive (AR) transformers to learn visual distributions fast and can generalize well: VAR, for the first time, makes GPT-style AR models surpass diffusion transformers in image generation. On ImageNet 256$\times$256 benchmark, VAR significantly improve AR baseline by improving Fréchet inception distance (FID) from 18.65 to 1.73, inception score (IS) from 80.4 to 350.2, with 20$\times$ faster inference speed. It is also empirically verified that VAR outperforms the Diffusion Transformer (DiT) in multiple dimensions including image quality, inference speed, data efficiency, and scalability. Scaling up VAR models exhibits clear power-law scaling laws similar to those observed in LLMs, with linear correlation coefficients near $-0.998$ as solid evidence. VAR further showcases zero-shot generalization ability in downstream tasks including image in-painting, out-painting, and editing. These results suggest VAR has initially emulated the two important properties of LLMs: Scaling Laws and zero-shot generalization. We have released all models and codes to promote the exploration of AR/VAR models for visual generation and unified learning.

我们提出视觉自回归建模（Visual AutoRegressive modeling，VAR）：把图像上的自回归学习改写成由粗到细的「下一尺度预测」或「下一分辨率预测」，而不再沿用光栅扫描式的「下一词元预测」。这一简明思路让自回归（AR）Transformer 能更快学会视觉分布，并具有良好泛化。VAR 首次使 GPT 风格的 AR 模型在图像生成上超过扩散 Transformer。在 ImageNet 256$\times$256 基准上，VAR 把 AR 基线的 Fréchet inception distance（FID）从 18.65 提到 1.73、inception score（IS）从 80.4 提到 350.2，推理速度约快 20 倍。经验上，VAR 在图像质量、推理速度、数据效率与可扩展性等多个维度也优于 Diffusion Transformer（DiT）。放大 VAR 模型可观察到与大语言模型类似的幂律缩放律，线性相关系数接近 $-0.998$。VAR 还在图像修补、外扩与编辑等下游任务上展现零样本泛化。这些结果说明 VAR 初步复现了 LLM 的两个重要性质：缩放律与零样本泛化。我们已开源全部模型与代码，以推动 AR/VAR 在视觉生成与统一学习上的探索。

Figure 1: Generated samples from Visual AutoRegressive (VAR) transformers trained on ImageNet. We show 512$\times$512 samples (top), 256$\times$256 samples (middle), and zero-shot image editing results (bottom).

图 1：在 ImageNet 上训练的视觉自回归（VAR）Transformer 生成样本。上行为 512$\times$512，中行为 256$\times$256，下行为零样本图像编辑结果。

Figure 2: Standard autoregressive modeling (AR) vs. our proposed visual autoregressive modeling (VAR). (a) AR applied to language: sequential text token generation from left to right, word by word; (b) AR applied to images: sequential visual token generation in a raster-scan order, from left to right, top to bottom; (c) VAR for images: multi-scale token maps are autoregressively generated from coarse to fine scales (lower to higher resolutions), with parallel token generation within each scale. VAR requires a multi-scale VQVAE to work.

图 2：标准自回归建模（AR）与本文视觉自回归建模（VAR）的对比。（a）AR 用于语言：自左向右、逐词生成文本词元；（b）AR 用于图像：按光栅扫描顺序、自左向右、自上而下逐个生成视觉词元；（c）VAR 用于图像：按由粗到细（由低到高分辨率）自回归生成多尺度词元图，每一尺度内并行生成词元。VAR 需要多尺度 VQVAE 配合工作。

1 Introduction

1 引言

The advent of GPT series [65, 66, 15, 62, 1] and more autoregressive (AR) large language models (LLMs) [22, 4, 38, 82, 83, 90, 78, 5, 79] has heralded a new epoch in the field of artificial intelligence. These models exhibit promising intelligence in generality and versatility that, despite issues like hallucinations [39], are still considered to take a solid step toward the general artificial intelligence (AGI). At the core of these models is a self-supervised learning strategy – predicting the next token in a sequence, a simple yet profound approach. Studies into the success of these large AR models have highlighted their scalability and generalizabilty: the former, as exemplified by scaling laws [43, 35], allows us to predict large model’s performance from smaller ones and thus guides better resource allocation, while the latter, as evidenced by zero-shot and few-shot learning [66, 15], underscores the unsupervised-trained models’ adaptability to diverse, unseen tasks. These properties reveal AR models’ potential in learning from vast unlabeled data, encapsulating the essence of “AGI”.

GPT 系列 [65, 66, 15, 62, 1] 以及更多自回归（AR）大语言模型（LLM） [22, 4, 38, 82, 83, 90, 78, 5, 79] 的出现，标志着人工智能进入新阶段。这些模型在通用性与多任务能力上展现出令人瞩目的智能；尽管仍有幻觉等问题 [39]，仍被视为迈向通用人工智能（AGI）的坚实一步。其核心是一种自监督学习策略——在序列中预测下一个词元，简单却深刻。对这些大型 AR 模型成功的研究，突出了它们的可扩展性与泛化性：前者以缩放律 [43, 35] 为代表，使我们能用小模型表现预测大模型，从而更好分配资源；后者以零样本与少样本学习 [66, 15] 为证，强调无监督训练模型对多样、未见任务的适应力。这些性质揭示了 AR 模型从海量无标注数据中学习的潜力，也正触及「AGI」的要旨。

In parallel, the field of computer vision has been striving to develop large autoregressive or world models [58, 57, 6], aiming to emulate their impressive scalability and generalizability. Trailblazing efforts like VQGAN and DALL-E [30, 67] along with their successors [68, 92, 50, 99] have showcased the potential of AR models in image generation. These models utilize a visual tokenizer to discretize continuous images into grids of 2D tokens, which are then flattened to a 1D sequence for AR learning (Fig. 2 b), mirroring the process of sequential language modeling (Fig. 2 a). However, the scaling laws of these models remain underexplored, and more frustratingly, their performance significantly lags behind diffusion models [63, 3, 51], as shown in Fig. 3. In contrast to the remarkable achievements of LLMs, the power of autoregressive models in computer vision appears to be somewhat locked.

与此同时，计算机视觉领域也在努力发展大型自回归或世界模型 [58, 57, 6]，以期复现其可扩展性与泛化性。VQGAN 与 DALL-E [30, 67] 及其后续工作 [68, 92, 50, 99] 展示了 AR 模型在图像生成上的潜力。它们用视觉分词器把连续图像离散成二维词元网格，再展平为一维序列做 AR 学习（图 2 b），类比语言建模中的顺序过程（图 2 a）。然而，这些模型的缩放律仍缺乏深入探讨；更令人沮丧的是，其性能明显落后于扩散模型 [63, 3, 51]，如图 3 所示。相较 LLM 的巨大成就，自回归模型在计算机视觉中的潜力似乎仍被锁住。

Figure 3: Scaling behavior of different model families on ImageNet 256$\times$256 generation benchmark. The FID of the validation set serves as a reference lower bound (1.78). VAR with 2B parameters reaches an FID of 1.73, surpassing L-DiT with 3B or 7B parameters.

图 3：不同模型族在 ImageNet 256$\times$256 生成基准上的缩放行为。验证集 FID 作为参考下界（1.78）。2B 参数的 VAR 达到 FID 1.73，超过 3B 或 7B 参数的 L-DiT。

Autoregressive modeling requires defining the order of data. Our work reconsiders how to “order” an image: Humans typically perceive or create images in a hierachical manner, first capturing the global structure and then local details. This multi-scale, coarse-to-fine nature suggests an “order” for images. Also inspired by the widespread multi-scale designs [54, 52, 81, 44], we define autoregressive learning for images as “next-scale prediction” in Fig. 2 (c), diverging from the conventional “next-token prediction” in Fig. 2 (b). Our approach begins by encoding an image into multi-scale token maps. The autoregressive process is then started from the 1$\times$1 token map, and progressively expands in resolution: at each step, the transformer predicts the next higher-resolution token map conditioned on all previous ones. We refer to this methodology as Visual AutoRegressive (VAR) modeling.

自回归建模需要定义数据的顺序。我们重新思考如何给图像「排序」：人通常以层次方式感知或创作图像——先把握全局结构，再补局部细节。这种多尺度、由粗到细的特点提示了图像的一种「顺序」。亦受广泛多尺度设计 [54, 52, 81, 44] 启发，我们在图 2（c）中把图像自回归学习定义为「下一尺度预测」，而非图 2（b）中的常规「下一词元预测」。做法是先把图像编码成多尺度词元图；自回归从 1$\times$1 词元图起步，分辨率逐步升高：每一步 Transformer 在全部已有词元图的条件下，预测下一张更高分辨率的词元图。我们称这一方法为视觉自回归（VAR）建模。

VAR directly leverages GPT-2-like transformer architecture [66] for visual autoregressive learning. On the ImageNet 256$\times$256 benchmark, VAR significantly improves its AR baseline, achieving a Fréchet inception distance (FID) of 1.73 and an inception score (IS) of 350.2, with inference speed 20$\times$ faster (see Sec. 7 for details). Notably, VAR surpasses the Diffusion Transformer (DiT) – the foundation of leading diffusion systems like Stable Diffusion 3.0 and SORA [29, 14] – in FID/IS, data efficiency, inference speed, and scalability. VAR models also exhibit scaling laws akin to those witnessed in LLMs. Lastly, we showcase VAR’s zero-shot generalization capabilities in tasks like image in-painting, out-painting, and editing. In summary, our contributions to the community include:

VAR 直接采用类 GPT-2 的 Transformer 架构 [66] 做视觉自回归学习。在 ImageNet 256$\times$256 基准上，VAR 相对其 AR 基线大幅提升，达到 FID 1.73、IS 350.2，推理速度约快 20 倍（详见第 7 节）。尤其是，VAR 在 FID/IS、数据效率、推理速度与可扩展性上超过 Diffusion Transformer（DiT）——后者是 Stable Diffusion 3.0 与 SORA [29, 14] 等领先扩散系统的基础。VAR 模型也呈现与 LLM 类似的缩放律。最后，我们展示了 VAR 在图像修补、外扩与编辑等任务上的零样本泛化。总结本文对社区的贡献如下：

1. A new visual generative framework using a multi-scale autoregressive paradigm with next-scale prediction, offering new insights in autoregressive algorithm design for computer vision.
2. An empirical validation of VAR models’ Scaling Laws and zero-shot generalization potential, which initially emulates the appealing properties of large language models (LLMs).
3. A breakthrough in visual autoregressive model performance, making GPT-style autoregressive methods surpass strong diffusion models in image synthesis for the first time[^1].
4. A comprehensive open-source code suite, including both VQ tokenizer and autoregressive model training pipelines, to help propel the advancement of visual autoregressive learning.

1. 一套采用下一尺度预测的多尺度自回归视觉生成框架，为计算机视觉中的自回归算法设计提供新视角。
2. 对 VAR 模型缩放律与零样本泛化潜力的经验验证，初步复现大语言模型（LLM）的诱人性质。
3. 视觉自回归模型性能上的突破，使 GPT 风格自回归方法首次在图像合成上超过强扩散模型[^1]。
4. 一套完整开源代码，涵盖 VQ 分词器与自回归模型训练管线，助力推进视觉自回归学习。

2 Related Work

2 相关工作

2.1 Properties of large autoregressive language models

2.1 大型自回归语言模型的性质

Scaling laws  are found and studied in autoregressive language models [43, 35], which describe a power-law relationship between the scale of model (or dataset, computation, etc.) and the cross-entropy loss value on the test set. Scaling laws allow us to directly predict the performance of a larger model from smaller ones [1], thus guiding better resource allocation. More pleasingly, they show that the performance of LLMs can scale well with the growth of model, data, and computation and never saturate, which is considered a key factor in the success of  [15, 82, 83, 98, 90, 38]. The success brought by scaling laws has inspired the vision community to explore more similar methods for multimodality understanding and generation [53, 2, 88, 27, 96, 77, 21, 23, 41, 31, 32, 80, 87].

缩放律 在自回归语言模型中被发现并研究 [43, 35]，描述模型（或数据、算力等）规模与测试集交叉熵损失之间的幂律关系。缩放律使我们能由小模型直接预测更大模型的表现 [1]，从而更好分配资源。更令人振奋的是，它们表明 LLM 的性能可随模型、数据与算力增长而良好扩展且不易饱和，这被视为  [15, 82, 83, 98, 90, 38] 成功的关键因素。缩放律带来的成功也激励视觉社区探索更多类似方法，用于多模态理解与生成 [53, 2, 88, 27, 96, 77, 21, 23, 41, 31, 32, 80, 87]。

Zero-shot generalization.  Zero-shot generalization [72] refers to the ability of a model, particularly a Large Language Model, to perform tasks that it has not been explicitly trained on. Within the realm of the computer vision, there is a burgeoning interest in the zero-shot and in-context learning abilities of foundation models, CLIP [64], SAM [48], Dinov2 [61]. Innovations like Painter [89] and LVM [6] extend visual prompters [40, 11] to achieve in-context learning in vision.

零样本泛化。 零样本泛化 [72] 指模型——尤其是大语言模型——完成其未显式训练过的任务的能力。在计算机视觉领域，基础模型的零样本与上下文学习能力正受到广泛关注，如 CLIP [64]、SAM [48]、Dinov2 [61]。Painter [89] 与 LVM [6] 等创新把视觉提示器 [40, 11] 扩展到视觉中的上下文学习。

2.2 Visual generation

2.2 视觉生成

Raster-scan autoregressive models  for visual generation necessitate the encoding of 2D images into 1D token sequences. Early endeavors [20, 84] have shown the ability to generate RGB (or grouped) pixels in the standard row-by-row, raster-scan manner. [69] extends [84] by using multiple independent trainable networks to do super-resolution repeatedly. VQGAN [30] advances [20, 84] by doing autoregressive learning in the latent space of VQVAE [85]. It employs GPT-2 decoder-only transformer to generate tokens in the raster-scan order, like how ViT [28] serializes 2D images into 1D patches. VQVAE-2 [68] and RQ-Transformer [50] also follow this raster-scan manner but use extra scales or stacked codes. Parti [93], based on the architecture of ViT-VQGAN [92], scales the transformer to 20B parameters and works well in text-to-image synthesis.

光栅扫描自回归模型 做视觉生成时，需要把二维图像编码为一维词元序列。早期工作 [20, 84] 已展示按标准逐行光栅扫描生成 RGB（或分组）像素的能力。[69] 在 [84] 基础上用多个独立可训网络反复做超分。VQGAN [30] 在 VQVAE [85] 潜空间做自回归学习，进一步推进 [20, 84]：它用 GPT-2 式仅解码器 Transformer，按光栅扫描顺序生成词元，类似 ViT [28] 把二维图像串成一维 patch。VQVAE-2 [68] 与 RQ-Transformer [50] 同样沿用光栅扫描，但引入额外尺度或堆叠码。Parti [93] 基于 ViT-VQGAN [92] 架构，把 Transformer 扩到 20B 参数，在文生图合成上表现良好。

Masked-prediction model.  MaskGIT [17] employs a VQ autoencoder and a masked prediction transformer similar to BERT [25, 10, 34] to generate VQ tokens through a greedy algorithm. MagViT [94] adapts this approach to videos, and MagViT-2 [95] enhances [17, 94] by introducing an improved VQVAE for both images and videos. MUSE [16] further scales MaskGIT to 3B parameters.

掩码预测模型。 MaskGIT [17] 采用 VQ 自编码器与类似 BERT [25, 10, 34] 的掩码预测 Transformer，用贪心算法生成 VQ 词元。MagViT [94] 将该思路扩展到视频；MagViT-2 [95] 通过改进的 VQVAE 增强 [17, 94]，同时覆盖图像与视频。MUSE [16] 进一步把 MaskGIT 扩到 3B 参数。

Diffusion models ’ progress has centered around improved learning or sampling [76, 75, 55, 56, 7], guidance [37, 60], latent learning [70], and architectures [36, 63, 71, 91]. DiT and U-ViT [63, 8] replaces or integrates the U-Net with transformer, and inspires recent image [19, 18] or video synthesis systems [12, 33] including Stable Diffusion 3.0 [29], SORA [14], and Vidu [9].

扩散模型 的进展围绕更好的学习或采样 [76, 75, 55, 56, 7]、引导 [37, 60]、潜空间学习 [70] 与架构 [36, 63, 71, 91]。DiT 与 U-ViT [63, 8] 用 Transformer 替换或融合 U-Net，并启发了近期图像 [19, 18] 与视频合成系统 [12, 33]，包括 Stable Diffusion 3.0 [29]、SORA [14] 与 Vidu [9]。

3 Method

3 方法

3.1 Preliminary: autoregressive modeling via next-token prediction

3.1 预备：基于下一词元预测的自回归建模

Formulation.  Consider a sequence of discrete tokens $x=(x_{1},x_{2},\dots,x_{T})$, where $x_{t}\in[V]$ is an integer from a vocabulary of size $V$. The next-token autoregressive posits the probability of observing the current token $x_{t}$ depends only on its prefix $(x_{1},x_{2},\dots,x_{t-1})$. This unidirectional token dependency assumption allows for the factorization of the sequence $x$’s likelihood:

形式化。 考虑离散词元序列 $x=(x_{1},x_{2},\dots,x_{T})$，其中 $x_{t}\in[V]$ 是大小为 $V$ 的词表中的整数。下一词元自回归假定观测当前词元 $x_{t}$ 的概率只依赖于其前缀 $(x_{1},x_{2},\dots,x_{t-1})$。这一单向词元依赖假设允许把序列 $x$ 的似然分解为：

$$\displaystyle p(x_{1},x_{2},\dots,x_{T})=\prod_{t=1}^{T}p(x_{t}\mid x_{1},x_{2},\dots,x_{t-1}). \tag{1}$$

Training an autoregressive model $p_{\theta}$ involves optimizing $p_{\theta}(x_{t}\mid x_{1},x_{2},\dots,x_{t-1})$ over a dataset. This is known as the “next-token prediction”, and the trained $p_{\theta}$ can generate new sequences.

训练自回归模型 $p_{\theta}$ 即在数据集上优化 $p_{\theta}(x_{t}\mid x_{1},x_{2},\dots,x_{t-1})$。这就是「下一词元预测」；训好的 $p_{\theta}$ 可生成新序列。

Tokenization.  Images are inherently 2D continuous signals. To apply autoregressive modeling to images via next-token prediction, we must:  1) tokenize an image into several discrete tokens, and  2) define a 1D order of tokens for unidirectional modeling.  For 1), a quantized autoencoder such as [30] is often used to convert the image feature map $f\in\mathbb{R}^{h\times w\times C}$ to discrete tokens $q\in[V]^{h\times w}$:

分词。 图像本质上是二维连续信号。要通过下一词元预测对图像做自回归建模，必须： 1）把图像分词成若干离散词元； 2）为单向建模定义词元的一维顺序。 对 1），常用如 [30] 一类量化自编码器，把图像特征图 $f\in\mathbb{R}^{h\times w\times C}$ 转为离散词元 $q\in[V]^{h\times w}$：

$$\displaystyle f=\mathcal{E}(im),\quad~~q=\mathcal{Q}(f), \tag{2}$$

where $im$ denotes the raw image, $\mathcal{E}(\cdot)$ a encoder, and $\mathcal{Q}(\cdot)$ a quantizer. The quantizer typically includes a learnable codebook $Z\in\mathbb{R}^{V\times C}$ containing $V$ vectors. The quantization process $q=\mathcal{Q}(f)$ will map each feature vector $f^{(i,j)}$ to the code index $q^{(i,j)}$ of its nearest code in the Euclidean sense:

其中 $im$ 表示原始图像，$\mathcal{E}(\cdot)$ 为编码器，$\mathcal{Q}(\cdot)$ 为量化器。量化器通常包含可学习码本 $Z\in\mathbb{R}^{V\times C}$，含 $V$ 个向量。量化过程 $q=\mathcal{Q}(f)$ 把每个特征向量 $f^{(i,j)}$ 映射到欧氏意义下最近码的索引 $q^{(i,j)}$：

$$\displaystyle q^{(i,j)}=\left(\argmin_{v\in[V]}\|\text{lookup}(Z,v)-f^{(i,j)}\|_{2}\right)\in[V], \tag{3}$$

where $\text{lookup}(Z,v)$ means taking the $v$-th vector in codebook $Z$. To train the quantized autoencoder, $Z$ is looked up by every $q^{(i,j)}$ to get $\hat{f}$, the approximation of original $f$. Then a new image $\hat{im}$ is reconstructed using the decoder $\mathcal{D}(\cdot)$ given $\hat{f}$, and a compound loss $\mathcal{L}$ is minimized:

其中 $\text{lookup}(Z,v)$ 表示取码本 $Z$ 中第 $v$ 个向量。训练量化自编码器时，用每个 $q^{(i,j)}$ 查表得到 $\hat{f}$，作为原始 $f$ 的近似。再用解码器 $\mathcal{D}(\cdot)$ 在 $\hat{f}$ 上重建新图像 $\hat{im}$，并最小化复合损失 $\mathcal{L}$：

$$\displaystyle\hat{f} \\ \displaystyle=\text{lookup}(Z,q),\quad\quad\quad~~~\hat{im}=\mathcal{D}(\hat{f}), \\ \displaystyle\mathcal{L} \\ \displaystyle=\|im-\hat{im}\|_{2}+\|f-\hat{f}\|_{2}+\lambda_{\text{P}}\mathcal{L}_{\text{P}}(\hat{im})+\lambda_{\text{G}}\mathcal{L}_{\text{G}}(\hat{im}), \tag{4}$$

where $\mathcal{L}_{\text{P}}(\cdot)$ is a perceptual loss such as LPIPS [97], $\mathcal{L}_{\text{G}}(\cdot)$ a discriminative loss like StyleGAN’s discriminator loss [46], and $\lambda_{\text{P}}$, $\lambda_{\text{G}}$ are loss weights. Once the autoencoder $\{\mathcal{E},\mathcal{Q},\mathcal{D}\}$ is fully trained, it will be used to tokenize images for subsequent training of a unidirectional autoregressive model.

其中 $\mathcal{L}_{\text{P}}(\cdot)$ 为感知损失如 LPIPS [97]，$\mathcal{L}_{\text{G}}(\cdot)$ 为判别损失如 StyleGAN 的判别器损失 [46]，$\lambda_{\text{P}}$、$\lambda_{\text{G}}$ 为损失权重。量化自编码器 $\{\mathcal{E},\mathcal{Q},\mathcal{D}\}$ 训好后，用于后续单向自回归模型训练时的图像分词。

The image tokens in $q\in[V]^{h\times w}$ are arranged in a 2D grid. Unlike natural language sentences with an inherent left-to-right ordering, the order of image tokens must be explicitly defined for unidirectional autoregressive learning. Previous AR methods [30, 92, 50] flatten the 2D grid of $q$ into a 1D sequence $x=(x_{1},\dots,x_{h\times w})$ using some strategy such as row-major raster scan, spiral, or z-curve order. Once flattened, they can extract a set of sequences $x$ from the dataset, and then train an autoregressive model to maximize the likelihood in (1) via next-token prediction.

$q\in[V]^{h\times w}$ 中的图像词元排成二维网格。自然语言句子有固有的从左到右顺序，而图像词元的顺序必须为单向自回归学习显式定义。先前 AR 方法 [30, 92, 50] 用行优先光栅扫描、螺旋或 z 曲线等策略，把 $q$ 的二维网格展平为一维序列 $x=(x_{1},\dots,x_{h\times w})$。展平后即可从数据集抽取一组序列 $x$，再通过下一词元预测训练自回归模型以最大化式 (1) 的似然。

Discussion on the weakness of vanilla autoregressive models.  The above approach of tokenizing and flattening enable next-token autoregressive learning on images, but introduces several issues:

关于朴素自回归模型弱点的讨论。 上述分词与展平使图像上的下一词元自回归学习成为可能，但也引入若干问题：

1. Mathematical premise violation.  In quantized autoencoders (VQVAEs), the encoder typically produces an image feature map $f$ with inter-dependent feature vectors $f^{(i,j)}$ for all $i,j$. So after quantization and flattening, the token sequence $(x_{1},x_{2},\dots,x_{h\times w})$ retains bidirectional correlations. This contradicts the unidirectional dependency assumption of autoregressive models, which dictates that each token $x_{t}$ should only depend on its prefix $(x_{1},x_{2},\dots,x_{t-1})$.
2. Inability to perform some zero-shot generalization.  Similar to issue 1), The unidirectional nature of image autoregressive modeling restricts their generalizability in tasks requiring bidirectional reasoning. E.g., it cannot predict the top part of an image given the bottom part.
3. Structural degradation.  The flattening disrupts the spatial locality inherent in image feature maps. For example, the token $q^{(i,j)}$ and its 4 immediate neighbors $q^{(i\pm 1,j)}$, $q^{(i,j\pm 1)}$ are closely correlated due to their proximity. This spatial relationship is compromised in the linear sequence $x$, where unidirectional constraints diminish these correlations.
4. Inefficiency.  Generating an image token sequence $x=(x_{1},x_{2},\dots,x_{n\times n})$ with a conventional self-attention transformer incurs $\mathcal{O}(n^{2})$ autoregressive steps and $\mathcal{O}(n^{6})$ computational cost.

1. 数学前提被破坏。 在量化自编码器（VQVAE）中，编码器通常产生特征图 $f$，其中各特征向量 $f^{(i,j)}$ 彼此依赖。量化并展平后，词元序列 $(x_{1},x_{2},\dots,x_{h\times w})$ 仍保留双向相关。这与自回归模型的单向依赖假设相矛盾——后者要求每个词元 $x_{t}$ 只依赖其前缀 $(x_{1},x_{2},\dots,x_{t-1})$。
2. 难以做某些零样本泛化。 与问题 1) 类似，图像自回归建模的单向性限制了需要双向推理的任务上的泛化。例如，给定图像下半部分时，无法预测上半部分。
3. 结构退化。 展平破坏了图像特征图固有的空间局部性。例如，词元 $q^{(i,j)}$ 与其四个近邻 $q^{(i\pm 1,j)}$、$q^{(i,j\pm 1)}$ 因邻近而高度相关。在线性序列 $x$ 中，这种空间关系被削弱；单向约束进一步削弱这些相关性。
4. 效率低。 用常规自注意力 Transformer 生成图像词元序列 $x=(x_{1},x_{2},\dots,x_{n\times n})$，需要 $\mathcal{O}(n^{2})$ 次自回归步与 $\mathcal{O}(n^{6})$ 计算量。

Issues 2) and 3) are evident (see examples above). Regarding issue 1), we present empirical evidence in Appendix A. The proof of issue 3) is detailed in Appendix B. These theoretical and practical limitations call for a rethinking of autoregressive models in the context of image generation.

问题 2) 与 3) 显而易见（见上文示例）。关于问题 1)，我们在附录 A 给出经验证据。问题 3) 的证明详见附录 B。这些理论与实践局限，要求我们在图像生成语境下重新思考自回归模型。

Figure 4: VAR involves two separated training stages. Stage 1: a multi-scale VQ autoencoder encodes an image into $K$ token maps $R=(r_{1},r_{2},\dots,r_{K})$ and is trained by a compound loss (5). For details on “Multi-scale quantization” and “Embedding”, check Algorithm 1 and 2.  Stage 2: a VAR transformer is trained via next-scale prediction (6): it takes $(\texttt{[s]},r_{1},r_{2},\dots,r_{K-1})$ as input to predict $(r_{1},r_{2},r_{3},\dots,r_{K})$. The attention mask is used in training to ensure each $r_{k}$ can only attend to $r_{\leq k}$. Standard cross-entropy loss is used.

图 4：VAR 分两个独立训练阶段。阶段 1：多尺度 VQ 自编码器把图像编码为 $K$ 张词元图 $R=(r_{1},r_{2},\dots,r_{K})$，用复合损失 (5) 训练。「多尺度量化」与「嵌入」细节见算法 1 与 2。 阶段 2：VAR Transformer 通过下一尺度预测 (6) 训练：输入 $(\texttt{[s]},r_{1},r_{2},\dots,r_{K-1})$，预测 $(r_{1},r_{2},r_{3},\dots,r_{K})$。训练时使用注意力掩码，确保每个 $r_{k}$ 只能关注 $r_{\leq k}$。损失为标准交叉熵。

3.2 Visual autoregressive modeling via next-scale prediction

3.2 基于下一尺度预测的视觉自回归建模

Reformulation.  We reconceptualize the autoregressive modeling on images by shifting from “next-token prediction” to “next-scale prediction” strategy. Here, the autoregressive unit is an entire token map, rather than a single token. We start by quantizing a feature map $f\in\mathbb{R}^{h\times w\times C}$ into $K$ multi-scale token maps $(r_{1},r_{2},\dots,r_{K})$, each at a increasingly higher resolution $h_{k}\times w_{k}$, culminating in $r_{K}$ matches the original feature map’s resolution $h\times w$. The autoregressive likelihood is formulated as:

改写。 我们把图像上的自回归建模从「下一词元预测」改写为「下一尺度预测」策略。此处自回归单元是整张词元图，而非单个词元。先把特征图 $f\in\mathbb{R}^{h\times w\times C}$ 量化为 $K$ 张多尺度词元图 $(r_{1},r_{2},\dots,r_{K})$，分辨率 $h_{k}\times w_{k}$ 逐步升高，最终 $r_{K}$ 与原始特征图分辨率 $h\times w$ 一致。自回归似然写为：

$$\displaystyle p(r_{1},r_{2},\dots,r_{K})=\prod_{k=1}^{K}p(r_{k}\mid r_{1},r_{2},\dots,r_{k-1}), \tag{6}$$

where each autoregressive unit $r_{k}\in[V]^{h_{k}\times w_{k}}$ is the token map at scale $k$ containing $h_{k}\times w_{k}$ tokens, and the sequence $(r_{1},r_{2},\dots,r_{k-1})$ serves as the the “prefix” for $r_{k}$. During the $k$-th autoregressive step, all distributions over the $h_{k}\times w_{k}$ tokens in $r_{k}$ will be generated in parallel, conditioned on $r_{k}$’s prefix and associated $k$-th position embedding map. This “next-scale prediction” methodology is what we define as visual autoregressive modeling (VAR), depicted on the right side of Fig. 4. Note that in the training of VAR, a block-wise causal attention mask is used to ensure that each $r_{k}$ can only attend to its prefix $r_{\leq k}$. During inference, kv-caching can be used and no mask is needed.

其中每个自回归单元 $r_{k}\in[V]^{h_{k}\times w_{k}}$ 是尺度 $k$ 上含 $h_{k}\times w_{k}$ 个词元的词元图，序列 $(r_{1},r_{2},\dots,r_{k-1})$ 作为 $r_{k}$ 的「前缀」。在第 $k$ 个自回归步，$r_{k}$ 中全部 $h_{k}\times w_{k}$ 个词元的分布在其前缀与第 $k$ 张位置嵌入图条件下并行生成。这一「下一尺度预测」方法即我们定义的视觉自回归建模（VAR），见图 4 右侧。注意训练 VAR 时使用分块因果注意力掩码，确保每个 $r_{k}$ 只能关注其前缀 $r_{\leq k}$。推理时可使用 kv-caching，无需掩码。

Discussion.  VAR addresses the previously mentioned three issues as follows:

讨论。 VAR 对前述三个问题的回应如下：

1. The mathematical premise is satisfied if we constrain each $r_{k}$ to depend only on its prefix, that is, the process of getting $r_{k}$ is solely related to $r_{\leq k}$. This constraint is acceptable as it aligns with the natural, coarse-to-fine progression characteristics like human visual perception and artistic drawing (as we discussed in Sec. 1). Further details are provided in the Tokenization below.
2. The spatial locality is preserved as (i) there is no flattening operation in VAR, and (ii) tokens in each $r_{k}$ are fully correlated. The multi-scale design additionally reinforces the spatial structure.
3. The complexity for generating an image with $n\times n$ latent is significantly reduced to $\mathcal{O}(n^{4})$, see Appendix for proof. This efficiency gain arises from the parallel token generation in each $r_{k}$.

1. 若约束每个 $r_{k}$ 只依赖其前缀——即得到 $r_{k}$ 的过程仅与 $r_{\leq k}$ 有关——则数学前提得以满足。该约束可接受，因为它符合人类视觉感知与艺术绘制这类自然的由粗到细进程（见第 1 节讨论）。更多细节见下文分词。
2. 空间局部性得以保留，因为 (i) VAR 不做展平，(ii) 每张 $r_{k}$ 内词元完全相关。多尺度设计还进一步强化空间结构。
3. 生成 $n\times n$ 潜变量图像的复杂度显著降为 $\mathcal{O}(n^{4})$，证明见附录。效率提升来自每张 $r_{k}$ 内的并行词元生成。

Tokenization.  We develope a new multi-scale quantization autoencoder to encode an image to $K$ multi-scale discrete token maps $R=(r_{1},r_{2},\dots,r_{K})$ necessary for VAR learning (6). We employ the same architecture as VQGAN [30] but with a modified multi-scale quantization layer. The encoding and decoding procedures with residual design on $f$ or $\hat{f}$ are detailed in algorithms 1 and 2. We empirically find this residual-style design, akin to [50], can perform better than independent interpolation. Algorithm 1 shows that each $r_{k}$ would depend only on its prefix $(r_{1},r_{2},\dots,r_{k-1})$. Note that a shared codebook $Z$ is utilized across all scales, ensuring that each $r_{k}$’s tokens belong to the same vocabulary $[V]$. To address the information loss in upscaling $z_{k}$ to $h_{K}\times w_{K}$, we use $K$ extra convolution layers $\{\phi_{k}\}_{k=1}^{K}$. No convolution is used after downsampling $f$ to $h_{k}\times w_{k}$.

分词。 我们设计了新的多尺度量化自编码器，把图像编码为 VAR 学习 (6) 所需的 $K$ 张多尺度离散词元图 $R=(r_{1},r_{2},\dots,r_{K})$。架构与 VQGAN [30] 相同，但改用多尺度量化层。在 $f$ 或 $\hat{f}$ 上带残差的编码与解码流程见算法 1 与 2。经验上，这种类似 [50] 的残差式设计优于独立插值。算法 1 表明每个 $r_{k}$ 只依赖其前缀 $(r_{1},r_{2},\dots,r_{k-1})$。注意所有尺度共用码本 $Z$，保证每个 $r_{k}$ 的词元属于同一词表 $[V]$。为弥补把 $z_{k}$ 上采样到 $h_{K}\times w_{K}$ 时的信息损失，我们使用 $K$ 个额外卷积层 $\{\phi_{k}\}_{k=1}^{K}$。把 $f$ 下采样到 $h_{k}\times w_{k}$ 之后不再使用卷积。

Inputs: raw image $im$;
1 Hyperparameters: steps $K$, resolutions $(h_{k},w_{k})_{k=1}^{K}$;
2 $f=\mathcal{E}(im)$, $R=[]$;
3 for $k=1,\cdots,K$ do
4 $r_{k}=\mathcal{Q}(\text{interpolate}(f,h_{k},w_{k}))$;
5 $R=\text{queue\_push}(R,r_{k})$;
6 $z_{k}=\text{lookup}(Z,r_{k})$;
7 $z_{k}=\text{interpolate}(z_{k},h_{K},w_{K})$;
8 $f=f-\phi_{k}(z_{k})$;
9 Return: multi-scale tokens $R$;

Algorithm 1  Multi-scale VQVAE Encoding

输入：原始图像 $im$；
1 超参数：步数 $K$，分辨率 $(h_{k},w_{k})_{k=1}^{K}$；
2 $f=\mathcal{E}(im)$，$R=[]$；
3 for $k=1,\cdots,K$ do
4 $r_{k}=\mathcal{Q}(\text{interpolate}(f,h_{k},w_{k}))$；
5 $R=\text{queue\_push}(R,r_{k})$；
6 $z_{k}=\text{lookup}(Z,r_{k})$；
7 $z_{k}=\text{interpolate}(z_{k},h_{K},w_{K})$；
8 $f=f-\phi_{k}(z_{k})$；
9 返回：多尺度词元 $R$；

算法 1  多尺度 VQVAE 编码

Inputs: multi-scale token maps $R$;
1 Hyperparameters: steps $K$, resolutions $(h_{k},w_{k})_{k=1}^{K}$;
2 $\hat{f}=0$;
3 for $k=1,\cdots,K$ do
4 $r_{k}=\text{queue\_pop}(R)$;
5 $z_{k}=\text{lookup}(Z,r_{k})$;
6 $z_{k}=\text{interpolate}(z_{k},h_{K},w_{K})$;
7 $\hat{f}=\hat{f}+\phi_{k}(z_{k})$;
8 $\hat{im}=\mathcal{D}(\hat{f})$;
9 Return: reconstructed image $\hat{im}$;

Algorithm 2  Multi-scale VQVAE Reconstruction

输入：多尺度词元图 $R$；
1 超参数：步数 $K$，分辨率 $(h_{k},w_{k})_{k=1}^{K}$；
2 $\hat{f}=0$；
3 for $k=1,\cdots,K$ do
4 $r_{k}=\text{queue\_pop}(R)$；
5 $z_{k}=\text{lookup}(Z,r_{k})$；
6 $z_{k}=\text{interpolate}(z_{k},h_{K},w_{K})$；
7 $\hat{f}=\hat{f}+\phi_{k}(z_{k})$；
8 $\hat{im}=\mathcal{D}(\hat{f})$；
9 返回：重建图像 $\hat{im}$；

算法 2  多尺度 VQVAE 重建

4 Implementation details

4 实现细节

VAR tokenizer.  As aforementioned, we use the vanilla VQVAE architecture [30] and a multi-scale quantization scheme with $K$ extra convolutions (0.03M extra parameters). We use a shared codebook for all scales with $V=4096$. Following the baseline [30], our tokenizer is also trained on OpenImages [49] with the compound loss (5) and a spatial downsample ratio of $16\times$.

VAR 分词器。 如前所述，我们采用原版 VQVAE 架构 [30] 与带 $K$ 个额外卷积（约 0.03M 额外参数）的多尺度量化方案。所有尺度共用码本，$V=4096$。与基线 [30] 一致，分词器也在 OpenImages [49] 上用复合损失 (5) 训练，空间下采样比为 $16\times$。

VAR transformer.  Our main focus is on VAR algorithm so we keep a simple model architecture design. We adopt the architecture of standard decoder-only transformers akin to GPT-2 and VQGAN [66, 30] with adaptive normalization (AdaLN), which has widespread adoption and proven effectiveness in many visual generative models [46, 47, 45, 74, 73, 42, 63, 19]. For class-conditional synthesis, we use the class embedding as the start token [s] and also the condition of AdaLN. We found normalizing $queries$ and $keys$ to unit vectors before attention can stablize the training. We do not use advanced techniques in large language models, such as rotary position embedding (RoPE), SwiGLU MLP, or RMS Norm [82, 83]. Our model shape follows a simple rule like [43] that the width $w$, head counts $h$, and drop rate $dr$ are linearly scaled with the depth $d$ as follows:

VAR Transformer。 本文重点在 VAR 算法，故保持简单模型架构。我们采用类似 GPT-2 与 VQGAN [66, 30] 的标准仅解码器 Transformer，并加入自适应归一化（AdaLN）；该设计在许多视觉生成模型中被广泛采用并证明有效 [46, 47, 45, 74, 73, 42, 63, 19]。对类别条件合成，用类别嵌入作为起始词元 [s]，同时作为 AdaLN 的条件。我们发现在注意力前把 $queries$ 与 $keys$ 归一化为单位向量可稳定训练。我们未使用大语言模型中的先进技术，如旋转位置嵌入（RoPE）、SwiGLU MLP 或 RMS Norm [82, 83]。模型形状遵循类似 [43] 的简单规则：宽度 $w$、头数 $h$ 与丢弃率 $dr$ 随深度 $d$ 线性缩放：

$$\displaystyle w=64d,\quad\quad h=d,\quad\quad dr=0.1\cdot d/24. \tag{7}$$

Consequently, the main parameter count $N$ of a VAR transformer with depth $d$ is given by[^2]:

因此，深度为 $d$ 的 VAR Transformer 的主要参数量 $N$ 为[^2]：

$$\displaystyle N(d)=\underbrace{d\cdot 4w^{2}}_{\text{self-attention}}+\underbrace{d\cdot 8w^{2}}_{\text{feed-forward}}+\underbrace{d\cdot 6w^{2}}_{\text{adaptive layernorm}}=18\,dw^{2}=73728\,d^{3}. \tag{8}$$

All models are trained with the similar settings: a base learning rate of $10^{-4}$ per 256 batch size, an AdamW optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.95$, $\text{decay}=0.05$, a batch size from 768 to 1024 and training epochs from 200 to 350 (depends on model size). The evaluations in Sec. 5 suggest that such a simple model design are capable of scaling and generalizing well.

所有模型采用相近设置训练：按每 256 batch size 计基础学习率 $10^{-4}$，AdamW 优化器 $\beta_{1}=0.9$、$\beta_{2}=0.95$、$\text{decay}=0.05$，batch size 768 到 1024，训练轮数 200 到 350（视模型大小而定）。第 5 节评估表明，如此简单的模型设计已能良好扩展与泛化。

5 Empirical Results

5 实验结果

This section first compares VAR with other image generative model families in Sec. 5.1. Evaluations on the scalability and generalizability of VAR models are presented in Sec. 5.2 and Appendix 6. For implementation details and ablation study, please see Appendix 4 and Appendix 7.

本节先在第 5.1 节把 VAR 与其他图像生成模型族比较。可扩展性与泛化性评估见第 5.2 节与附录 6。实现细节与消融见附录 4 与附录 7。

Type	Model	FID$\downarrow$	IS$\uparrow$	Pre$\uparrow$	Rec$\uparrow$	#Para	#Step	Time
GAN	BigGAN [13]	6.95	224.5	0.89	0.38	112M	1	$-$
GAN	GigaGAN [42]	3.45	225.5	0.84	0.61	569M	1	$-$
GAN	StyleGan-XL [74]	2.30	265.1	0.78	0.53	166M	1	0.3 [74]
Diff.	ADM [26]	10.94	101.0	0.69	0.63	554M	250	168 [74]
Diff.	CDM [36]	4.88	158.7	$-$	$-$	$-$	8100	$-$
Diff.	LDM-4-G [70]	3.60	247.7	$-$	$-$	400M	250	$-$
Diff.	DiT-L/2 [63]	5.02	167.2	0.75	0.57	458M	250	31
Diff.	DiT-XL/2 [63]	2.27	278.2	0.83	0.57	675M	250	45
Diff.	L-DiT-3B [3]	2.10	304.4	0.82	0.60	3.0B	250	$>$45
Diff.	L-DiT-7B [3]	2.28	316.2	0.83	0.58	7.0B	250	$>$45
Mask.	MaskGIT [17]	6.18	182.1	0.80	0.51	227M	8	0.5 [17]
Mask.	RCG (cond.) [51]	3.49	215.5	$-$	$-$	502M	20	1.9 [51]
AR	VQVAE-2*†* [68]	31.11	$\sim$45	0.36	0.57	13.5B	5120	$-$
AR	VQGAN*†* [30]	18.65	80.4	0.78	0.26	227M	256	19 [17]
AR	VQGAN [30]	15.78	74.3	$-$	$-$	1.4B	256	24
AR	VQGAN-re [30]	5.20	280.3	$-$	$-$	1.4B	256	24
AR	ViTVQ [92]	4.17	175.1	$-$	$-$	1.7B	1024	$>$24
AR	ViTVQ-re [92]	3.04	227.4	$-$	$-$	1.7B	1024	$>$24
AR	RQTran. [50]	7.55	134.0	$-$	$-$	3.8B	68	21
AR	RQTran.-re [50]	3.80	323.7	$-$	$-$	3.8B	68	21
VAR	VAR-$d16$	3.30	274.4	0.84	0.51	310M	10	0.4
VAR	VAR-$d20$	2.57	302.6	0.83	0.56	600M	10	0.5
VAR	VAR-$d24$	2.09	312.9	0.82	0.59	1.0B	10	0.6
VAR	VAR-$d30$	1.92	323.1	0.82	0.59	2.0B	10	1
VAR	VAR-$d30$-re	1.73	350.2	0.82	0.60	2.0B	10	1
	(validation data)	1.78	236.9	0.75	0.67

Table 1: Generative model family comparison on class-conditional ImageNet 256$\times$256. “$\downarrow$” or “$\uparrow$” indicate lower or higher values are better. Metrics include Fréchet inception distance (FID), inception score (IS), precision (Pre) and recall (rec). “#Step”: the number of model runs needed to generate an image. Wall-clock inference time relative to VAR is reported. Models with the suffix “-re” used rejection sampling. ${\dagger}$: taken from MaskGIT [17].

类型	模型	FID$\downarrow$	IS$\uparrow$	Pre$\uparrow$	Rec$\uparrow$	#Para	#Step	时间
GAN	BigGAN [13]	6.95	224.5	0.89	0.38	112M	1	$-$
GAN	GigaGAN [42]	3.45	225.5	0.84	0.61	569M	1	$-$
GAN	StyleGan-XL [74]	2.30	265.1	0.78	0.53	166M	1	0.3 [74]
Diff.	ADM [26]	10.94	101.0	0.69	0.63	554M	250	168 [74]
Diff.	CDM [36]	4.88	158.7	$-$	$-$	$-$	8100	$-$
Diff.	LDM-4-G [70]	3.60	247.7	$-$	$-$	400M	250	$-$
Diff.	DiT-L/2 [63]	5.02	167.2	0.75	0.57	458M	250	31
Diff.	DiT-XL/2 [63]	2.27	278.2	0.83	0.57	675M	250	45
Diff.	L-DiT-3B [3]	2.10	304.4	0.82	0.60	3.0B	250	$>$45
Diff.	L-DiT-7B [3]	2.28	316.2	0.83	0.58	7.0B	250	$>$45
Mask.	MaskGIT [17]	6.18	182.1	0.80	0.51	227M	8	0.5 [17]
Mask.	RCG (cond.) [51]	3.49	215.5	$-$	$-$	502M	20	1.9 [51]
AR	VQVAE-2*†* [68]	31.11	$\sim$45	0.36	0.57	13.5B	5120	$-$
AR	VQGAN*†* [30]	18.65	80.4	0.78	0.26	227M	256	19 [17]
AR	VQGAN [30]	15.78	74.3	$-$	$-$	1.4B	256	24
AR	VQGAN-re [30]	5.20	280.3	$-$	$-$	1.4B	256	24
AR	ViTVQ [92]	4.17	175.1	$-$	$-$	1.7B	1024	$>$24
AR	ViTVQ-re [92]	3.04	227.4	$-$	$-$	1.7B	1024	$>$24
AR	RQTran. [50]	7.55	134.0	$-$	$-$	3.8B	68	21
AR	RQTran.-re [50]	3.80	323.7	$-$	$-$	3.8B	68	21
VAR	VAR-$d16$	3.30	274.4	0.84	0.51	310M	10	0.4
VAR	VAR-$d20$	2.57	302.6	0.83	0.56	600M	10	0.5
VAR	VAR-$d24$	2.09	312.9	0.82	0.59	1.0B	10	0.6
VAR	VAR-$d30$	1.92	323.1	0.82	0.59	2.0B	10	1
VAR	VAR-$d30$-re	1.73	350.2	0.82	0.60	2.0B	10	1
	（验证数据）	1.78	236.9	0.75	0.67

表 1：ImageNet 256$\times$256 条件生成。「$\downarrow$」或「$\uparrow$」表示越低或越高越好。指标包括 Fréchet inception distance（FID）、inception score（IS）、精确率（Pre）与召回率（rec）。「#Step」：生成一张图像所需的模型运行次数。墙钟推理时间相对 VAR 报告。后缀「-re」表示使用拒绝采样。${\dagger}$：引自 MaskGIT [17]。

5.1 State-of-the-art image generation

5.1 领先的图像生成

Setup.  We test VAR models with depths 16, 20, 24, and 30 on ImageNet 256$\times$256 and 512$\times$512 conditional generation benchmarks and compare them with the state-of-the-art image generation model families. Among all VQVAE-based AR or VAR models, VQGAN [30] and ours use the same architecture (CNN) and training data (OpenImages [49]) for VQVAE, while ViT-VQGAN [92] uses a ViT autoencoder, and both it and RQTransformer [50] trains the VQVAE directly on ImageNet. The results are summaried in Tab. 1 and Tab. 2.

设置。 我们在 ImageNet 256$\times$256 与 512$\times$512 条件生成基准上测试深度 16、20、24、30 的 VAR 模型，并与领先图像生成模型族比较。在所有基于 VQVAE 的 AR 或 VAR 模型中，VQGAN [30] 与我们的 VQVAE 使用相同架构（CNN）与训练数据（OpenImages [49]），而 ViT-VQGAN [92] 使用 ViT 自编码器，它与 RQTransformer [50] 都直接在 ImageNet 上训练 VQVAE。结果汇总于表 1 与表 2。

Overall comparison.  In comparison with existing generative approaches including generative adversarial networks (GAN), diffusion models (Diff.), BERT-style masked-prediction models (Mask.), and GPT-style autoregressive models (AR), our visual autoregressive (VAR) establishes a new model class. As shown in Tab. 1, VAR not only achieves the best FID/IS but also demonstrates remarkable speed in image generation. VAR also maintains decent precision and recall, confirming its semantic consistency. These advantages hold true on the 512$\times$512 synthesis benchmark, as detailed in Tab. 2. Notably, VAR significantly advances traditional AR capabilities. To our knowledge, this is the first time of autoregressive models outperforming Diffusion transformers, a milestone made possible by VAR’s resolution of AR limitations discussed in Section 3.

总体比较。 相对现有生成路径——包括生成对抗网络（GAN）、扩散模型（Diff.）、BERT 风格掩码预测模型（Mask.）与 GPT 风格自回归模型（AR）——我们的视觉自回归（VAR）确立了一类新模型。如表 1 所示，VAR 不仅取得最佳 FID/IS，在图像生成速度上也十分突出，并保持不错的精确率与召回率，说明语义一致性良好。这些优势在 512$\times$512 合成基准上同样成立，见表 2。尤其是，VAR 显著推进了传统 AR 能力。据我们所知，这是自回归模型首次超过扩散 Transformer，这一里程碑得益于 VAR 化解了第 3 节讨论的 AR 局限。

Type	Model	FID$\downarrow$	IS$\uparrow$	Time
GAN	BigGAN [13]	8.43	177.9	$-$
Diff.	ADM [26]	23.24	101.0	$-$
Diff.	DiT-XL/2 [63]	3.04	240.8	81
Mask.	MaskGIT [17]	7.32	156.0	0.5*†*
AR	VQGAN [30]	26.52	66.8	25*†*
VAR	VAR-$d36$-s	2.63	303.2	1

Table 2: ImageNet 512$\times$512 conditional generation. ${\dagger}$: quoted from MaskGIT [17]. “-s”: a single shared AdaLN layer is used due to resource limitation.

类型	模型	FID$\downarrow$	IS$\uparrow$	时间
GAN	BigGAN [13]	8.43	177.9	$-$
Diff.	ADM [26]	23.24	101.0	$-$
Diff.	DiT-XL/2 [63]	3.04	240.8	81
Mask.	MaskGIT [17]	7.32	156.0	0.5*†*
AR	VQGAN [30]	26.52	66.8	25*†*
VAR	VAR-$d36$-s	2.63	303.2	1

表 2：ImageNet 512$\times$512 条件生成。 ${\dagger}$：引自 MaskGIT [17]。「-s」：因资源限制使用单个共享 AdaLN 层。

Efficiency comparison.  Conventional autoregressive (AR) models [30, 68, 92, 50] suffer a lot from the high computational cost, as the number of image tokens is quadratic to the image resolution. A full autoregressive generation of $n^{2}$ tokens requires $\mathcal{O}(n^{2})$ decoding iterations and $\mathcal{O}(n^{6})$ total computations. In contrast, VAR only requires $\mathcal{O}(\log(n))$ iterations and $\mathcal{O}(n^{4})$ total computations. The wall-clock time reported in Tab. 1 also provides empirical evidence that VAR is around 20 times faster than VQGAN and ViT-VQGAN even with more model parameters, reaching the speed of efficient GAN models which only require 1 step to generate an image.

效率比较。 传统自回归（AR）模型 [30, 68, 92, 50] 计算代价很高，因为图像词元数与分辨率成平方关系。完整自回归生成 $n^{2}$ 个词元需要 $\mathcal{O}(n^{2})$ 次解码迭代与 $\mathcal{O}(n^{6})$ 总计算量。相比之下，VAR 只需 $\mathcal{O}(\log(n))$ 次迭代与 $\mathcal{O}(n^{4})$ 总计算量。表 1 报告的墙钟时间也提供经验证据：即使参数更多，VAR 仍比 VQGAN 与 ViT-VQGAN 大约快 20 倍，接近只需 1 步生成图像的高效 GAN 模型速度。

Compared with popular diffusion transformer.  The VAR model surpasses the recently popular diffusion models Diffusion Transformer (DiT), which serves as the precursor to the latest Stable-Diffusion 3 [29] and SORA [14], in multiple dimensions: 1) In image generation diversity and quality (FID and IS), VAR with 2B parameters consistently performs better than DiT-XL/2 [63], L-DiT-3B, and L-DiT-7B [3]. VAR also maintains comparable precision and recall. 2) For inference speed, the DiT-XL/2 requires 45$\times$ the wall-clock time compared to VAR, while 3B and 7B models [3] would cost much more. 3) VAR is considered more data-efficient, as it requires only 350 training epochs compared to DiT-XL/2’s 1400. 4) For scalability, Fig. 3 and Tab. 1 show that DiT only obtains marginal or even negative gains beyond 675M parameters. In contrast, the FID and IS of VAR are consistently improved, aligning with the scaling law study in Sec. 5.2. These results establish VAR as potentially a more efficient and scalable model for image generation than models like DiT.

与流行扩散 Transformer 的比较。 VAR 在多个维度超过近期流行的扩散模型 Diffusion Transformer（DiT）——后者是最新 Stable-Diffusion 3 [29] 与 SORA [14] 的前身：1）在图像生成多样性与质量（FID 与 IS）上，2B 参数的 VAR 持续优于 DiT-XL/2 [63]、L-DiT-3B 与 L-DiT-7B [3]，精确率与召回率也保持可比。2）推理速度上，DiT-XL/2 的墙钟时间约为 VAR 的 45 倍，而 3B 与 7B 模型 [3] 代价更高。3）VAR 更数据高效：只需 350 个训练 epoch，而 DiT-XL/2 为 1400。4）可扩展性上，图 3 与表 1 显示 DiT 在超过 675M 参数后增益微弱甚至为负；相比之下，VAR 的 FID 与 IS 持续改进，与第 5.2 节缩放律研究一致。这些结果说明 VAR 有望成为比 DiT 一类模型更高效、更可扩展的图像生成模型。

Figure 5: Scaling laws with VAR transformer size $N$, with power-law fits (dashed) and equations (in legend). Small, near-zero exponents $\alpha$ suggest a smooth decline in both test loss $L$ and token error rate $Err$ when scaling up VAR transformer. Axes are all on a logarithmic scale. The Pearson correlation coefficients near $-0.998$ signify a strong linear relationship between $log(N)$ vs. $log(L)$ or $log(N)$ vs. $log(Err)$.

图 5：随 VAR Transformer 规模 $N$ 的缩放律，含幂律拟合（虚线）与方程（图例）。很小、接近零的指数 $\alpha$ 表明放大 VAR Transformer 时，测试损失 $L$ 与词元错误率 $Err$ 均平滑下降。各轴均为对数尺度。Pearson 相关系数接近 $-0.998$，表明 $\log(N)$ 与 $\log(L)$ 或 $\log(N)$ 与 $\log(Err)$ 之间存在强线性关系。

5.2 Power-law scaling laws

5.2 幂律缩放律

Background.  Prior research [43, 35, 38, 1] have established that scaling up autoregressive (AR) large language models (LLMs) leads to a predictable decrease in test loss $L$. This trend correlates with parameter counts $N$, training tokens $T$, and optimal training compute $C_{\text{min}}$, following a power-law:

背景。 先前研究 [43, 35, 38, 1] 表明，放大自回归（AR）大语言模型（LLM）会带来可预测的测试损失 $L$ 下降。该趋势与参数量 $N$、训练词元数 $T$ 以及最优训练算力 $C_{\text{min}}$ 相关，遵循幂律：

$$\displaystyle L=(\beta\cdot X)^{\alpha}, \tag{9}$$

where $X$ can be any of $N$, $T$, or $C_{\text{min}}$. The exponent $\alpha$ reflects the smoothness of power-law, and $L$ denotes the reducible loss normalized by irreducible loss $L_{\infty}$ [35][^3]. A logarithmic transformation to $L$ and $X$ will reveal a linear relation between $\log(L)$ and $\log(X)$:

其中 $X$ 可以是 $N$、$T$ 或 $C_{\text{min}}$。指数 $\alpha$ 反映幂律平滑程度；$L$ 表示相对不可约损失 $L_{\infty}$ [35] 归一化后的可约损失[^3]。对 $L$ 与 $X$ 取对数后，$\log(L)$ 与 $\log(X)$ 呈线性关系：

$$\displaystyle\log(L)=\alpha\log(X)+\alpha\log\beta. \tag{10}$$

An appealing phenomenon is that both [43] and [35] never observed deviation from these linear relationships at the higher end of $X$, although flattening is inevitable as the loss approaches zero.

一个有趣的现象是，[43] 与 [35] 在 $X$ 的高端从未观察到偏离这些线性关系，尽管损失趋近零时变平不可避免。

These observed scaling laws [43, 35, 38, 1] not only validate the scalability of LLMs but also serve as a predictive tool for AR modeling, which facilitates the estimation of performance for larger AR models based on their smaller counterparts, thereby saving resource usage by large model performance forecasting. Given these appealing properties of scaling laws brought by LLMs, their replication in computer vision is therefore of significant interest.

这些观测到的缩放律 [43, 35, 38, 1] 不仅验证了 LLM 的可扩展性，也为 AR 建模提供预测工具：可用较小 AR 模型估计更大模型的表现，从而通过性能预报节省资源。鉴于 LLM 带来的这些诱人性质，在计算机视觉中复现它们因此具有重要意义。

Setup of scaling VAR models.  Following the protocols from [43, 35, 38, 1], we examine whether our VAR model complies with similar scaling laws. We trained models across 12 different sizes, from 18M to 2B parameters, on the ImageNet training set [24] containing 1.28M images (or 870B image tokens under our VQVAE) per epoch. For models of different sizes, training spanned 200 to 350 epochs, with a maximum number of tokens reaching 305 billion. Below we focus on the scaling laws with model parameters $N$ and optimal training compute $C_{\text{min}}$ given sufficient token count $T$.

缩放 VAR 模型的设置。 遵循 [43, 35, 38, 1] 的协议，我们检验 VAR 是否服从类似缩放律。我们在含 128 万张图像的 ImageNet 训练集 [24] 上训练了 12 种不同规模模型（18M 到 2B 参数；按我们的 VQVAE，每 epoch 约 870B 图像词元）。不同规模训练 200 到 350 个 epoch，最大词元数达 3050 亿。下文聚焦在词元数 $T$ 充足时，相对模型参数 $N$ 与最优训练算力 $C_{\text{min}}$ 的缩放律。

Scaling laws with model parameters $N$.  We first investigate the test loss trend as the VAR model size increases. The number of parameters $N(d)=73728\,d^{3}$ for a VAR transformer with depth $d$ is specified in (8). We varied $d$ from $6$ to $30$, yielding 12 models with 18.5M to 2.0B parameters. We assessed the final test cross-entropy loss $L$ and token prediction error rates $Err$ on the ImageNet validation set of 50,000 images [24]. We computed $L$ and $Err$ for both the last scale (at the last next-scale autoregressive step), as well as the global average. Results are plotted in Fig. 5, where we

相对模型参数 $N$ 的缩放律。 我们首先考察 VAR 模型变大时测试损失的趋势。深度 $d$ 的 VAR Transformer 参数量 $N(d)=73728\,d^{3}$ 由式 (8) 给出。我们令 $d$ 从 $6$ 到 $30$，得到 18.5M 到 2.0B 参数的 12 个模型。在含 5 万张图像的 ImageNet 验证集 [24] 上评估最终测试交叉熵损失 $L$ 与词元预测错误率 $Err$。我们分别计算最后一尺度（最后一次下一尺度自回归步）以及全局平均的 $L$ 与 $Err$。结果见图 5，我们

Figure 6: Scaling laws with optimal training compute $C_{\text{min}}$. Line color denotes different model sizes. Red dashed lines are power-law fits with equations in legend. Axes are on a logarithmic scale. Pearson coefficients near $-0.99$ indicate strong linear relationships between $\log(C_{\text{min}})$ vs. $\log(L)$ or $\log(C_{\text{min}})$ vs. $\log(Err)$.

图 6：相对最优训练算力 $C_{\text{min}}$ 的缩放律。线颜色表示不同模型规模。红色虚线为幂律拟合，方程见图例。坐标轴为对数尺度。Pearson 系数接近 $-0.99$，表明 $\log(C_{\text{min}})$ 与 $\log(L)$ 或 $\log(C_{\text{min}})$ 与 $\log(Err)$ 之间存在强线性关系。

observed a clear power-law scaling trend for $L$ as a function of $N$, as consistent with [43, 35, 38, 1]. The power-law scaling laws can be expressed as:

观测到 $L$ 相对 $N$ 清晰的幂律缩放趋势，与 [43, 35, 38, 1] 一致。幂律缩放律可写为：

$$\displaystyle L_{\text{last}}=(2.0\cdot N)^{-0.23}\quad\text{and}\quad L_{\text{avg}}=(2.5\cdot N)^{-0.20}. \tag{11}$$

Although the scaling laws are mainly studied on the test loss, we also empirically observed similar power-law trends for the token error rate $Err$:

尽管缩放律主要研究测试损失，我们也经验上观测到词元错误率 $Err$ 的类似幂律趋势：

$$\displaystyle Err_{\text{last}}=(4.9\cdot 10^{2}N)^{-0.016}\quad\text{and}\quad Err_{\text{avg}}=(6.5\cdot 10^{2}N)^{-0.010}. \tag{12}$$

These results verify the strong scalability of VAR, by which scaling up VAR transformers can continuously improve the model’s test performance.

这些结果验证了 VAR 的强可扩展性：放大 VAR Transformer 可持续改进模型测试表现。

Scaling laws with optimal training compute $C_{\text{min}}$.  We then examine the scaling behavior of VAR transformers when increasing training compute $C$. For each of the 12 models, we traced the test loss $L$ and token error rate $Err$ as a function of $C$ during training quoted in PFlops ($10^{15}$ floating-point operations per second). The results are plotted in Fig. 6. Here, we draw the Pareto frontier of $L$ and $Err$ to highlight the optimal training compute $C_{\text{min}}$ required to reach a certain value of loss or error.

相对最优训练算力 $C_{\text{min}}$ 的缩放律。 我们接着考察增大训练算力 $C$ 时 VAR Transformer 的缩放行为。对 12 个模型中的每一个，我们在训练过程中追踪测试损失 $L$ 与词元错误率 $Err$ 相对 $C$ 的函数，算力以 PFlops（每秒 $10^{15}$ 次浮点运算）计。结果见图 6。这里我们画出 $L$ 与 $Err$ 的 Pareto 前沿，以突出达到某一损失或错误水平所需的最优训练算力 $C_{\text{min}}$。

The fitted power-law scaling laws for $L$ and $Err$ as a function of $C_{\text{min}}$ are:

拟合得到的 $L$ 与 $Err$ 相对 $C_{\text{min}}$ 的幂律缩放律为：

$$\displaystyle L_{\text{last}}=(2.2\cdot 10^{-5}C_{\text{min}})^{-0.13} \\ \displaystyle L_{\text{avg}}=(1.5\cdot 10^{-5}C_{\text{min}})^{-0.16}, \\ \displaystyle Err_{\text{last}}=(8.1\cdot 10^{-2}C_{\text{min}})^{-0.0067} \\ \displaystyle Err_{\text{avg}}=(4.4\cdot 10^{-2}C_{\text{min}})^{-0.011}. \tag{13}$$

These relations (14, 16) hold across 6 orders of magnitude in $C_{\text{min}}$, and our findings are consistent with those in [43, 35]: when trained with sufficient data, larger VAR transformers are more compute-efficient because they can reach the same level of performance with less computation.

这些关系（14, 16）在 $C_{\text{min}}$ 跨越 6 个数量级时成立，我们的发现与 [43, 35] 一致：在数据充足时训练，更大的 VAR Transformer 更算力高效，因为它们可用更少计算达到同等表现。

5.3 Visualization of scaling effect

5.3 缩放效应可视化

To better understand how VAR models are learning when scaled up, we compare some generated $256\times 256$ samples from VAR models of 4 different sizes (depth 6, 16, 26, 30) and 3 different training stages (20%, 60%, 100% of total training tokens) in Fig. 7. To keep the content consistent, a same random seed and teacher-forced initial tokens are used. The observed improvements in visual fidelity and soundness are consistent with the scaling laws, as larger transformers are thought able to learn more complex and fine-grained image distributions.

为更好理解放大时 VAR 模型在学什么，我们在图 7 比较了 4 种规模（深度 6、16、26、30）与 3 个训练阶段（总训练词元的 20%、60%、100%）的 VAR 模型生成的部分 $256\times 256$ 样本。为保持内容一致，使用相同随机种子与教师强制的初始词元。观测到的视觉保真度与合理性提升与缩放律一致：更大 Transformer 被认为能学习更复杂、更细粒度的图像分布。

Figure 7: Scaling model size $N$ and training compute $C$ improves visual fidelity and soundness. Zoom in for a better view. Samples are drawn from VAR models of 4 different sizes and 3 different training stages. 9 class labels (from left to right, top to bottom) are: flamingo 130, arctic wolf 270, macaw 88, Siamese cat 284, oscilloscope 688, husky 250, mollymawk 146, volcano 980, and catamaran 484.

图 7：放大模型规模 $N$ 与训练算力 $C$ 可提升视觉保真度与合理性。建议放大查看。样本来自 4 种规模、3 个训练阶段的 VAR 模型。9 个类别标签（从左到右、从上到下）为：flamingo 130、arctic wolf 270、macaw 88、Siamese cat 284、oscilloscope 688、husky 250、mollymawk 146、volcano 980、catamaran 484。

Figure 8: Zero-shot evaluation in downstream tasks containing in-painting, out-painting, and class-conditional editing. The results show that VAR can generalize to novel downstream tasks without special design and finetuning. Zoom in for a better view.

图 8：下游任务上的零样本评估，含修补、外扩与类别条件编辑。结果表明 VAR 无需特殊设计与微调即可泛化到新下游任务。建议放大查看。

6 Zero-shot task generalization

6 零样本任务泛化

Image in-painting and out-painting.  VAR-$d$30 is tested. For in- and out-painting, we teacher-force ground truth tokens outside the mask and let the model only generate tokens within the mask. No class label information is injected into the model. The results are visualized in Fig. 8. Without modifications to the network architecture or tuning parameters, VAR has achieved decent results on these downstream tasks, substantiating the generalization ability of VAR.

图像修补与外扩。 测试 VAR-$d$30。对修补与外扩，我们在掩码外教师强制真实词元，仅让模型生成掩码内词元。不向模型注入类别标签信息。结果见图 8。无需改网络架构或调参，VAR 在这些下游任务上取得不错结果，印证了其泛化能力。

Class-conditional image editing.  Following MaskGIT [17] we also tested VAR on the class-conditional image editing task. Similar to the case of in-painting, the model is forced to generate tokens only in the bounding box conditional on some class label. Fig. 8 shows the model can produce plausible content that fuses well into the surrounding contexts, again verifying the generality of VAR.

类别条件图像编辑。 遵循 MaskGIT [17]，我们也在类别条件图像编辑任务上测试 VAR。与修补类似，模型在给定类别标签条件下，仅在边界框内生成词元。图 8 显示模型能产生与周围上下文融合良好的合理内容，再次验证 VAR 的通用性。

	Description	Para.	Model	AdaLN	Top-$k$	CFG	Cost	FID$\downarrow$	$\Delta$
1	AR [30]	227M	AR	✗	✗	✗	1	18.65	0.00
2	AR to VAR	207M	VAR-$d$16	✗	✗	✗	0.013	5.22	$-$13.43
3	$+$AdaLN	310M	VAR-$d$16	✓	✗	✗	0.016	4.95	$-$13.70
4	$+$Top-$k$	310M	VAR-$d$16	✓	600	✗	0.016	4.64	$-$14.01
5	$+$CFG	310M	VAR-$d$16	✓	600	2.0	0.022	3.60	$-$15.05
5	$+$Attn. Norm.	310M	VAR-$d$16	✓	600	2.0	0.022	3.30	$-$15.35
6	$+$Scale up	2.0B	VAR-$d$30	✓	600	2.0	0.052	1.73	$-$16.85

Table 3: Ablation study of VAR. The first two rows compare GPT-2-style transformers trained under AR or VAR algorithm without any bells and whistles. Subsequent lines show the influence of VAR enhancements. “AdaLN”: adaptive layernorm. “CFG”: classifier-free guidance. “Attn. Norm.”: normalizing $q$ and $k$ to unit vectors before attention. “Cost”: inference cost relative to the baseline. “$\Delta$”: FID reduction to the baseline.

	说明	参数	模型	AdaLN	Top-$k$	CFG	代价	FID$\downarrow$	$\Delta$
1	AR [30]	227M	AR	✗	✗	✗	1	18.65	0.00
2	AR 到 VAR	207M	VAR-$d$16	✗	✗	✗	0.013	5.22	$-$13.43
3	$+$AdaLN	310M	VAR-$d$16	✓	✗	✗	0.016	4.95	$-$13.70
4	$+$Top-$k$	310M	VAR-$d$16	✓	600	✗	0.016	4.64	$-$14.01
5	$+$CFG	310M	VAR-$d$16	✓	600	2.0	0.022	3.60	$-$15.05
5	$+$Attn. Norm.	310M	VAR-$d$16	✓	600	2.0	0.022	3.30	$-$15.35
6	$+$放大	2.0B	VAR-$d$30	✓	600	2.0	0.052	1.73	$-$16.85

表 3：VAR 消融研究。前两行比较在无额外技巧时，按 AR 或 VAR 算法训练的 GPT-2 风格 Transformer。后续行展示 VAR 增强的影响。「AdaLN」：自适应层归一化。「CFG」：无分类器引导。「Attn. Norm.」：注意力前将 $q$ 与 $k$ 归一化为单位向量。「Cost」：相对基线的推理代价。「$\Delta$」：相对基线的 FID 降幅。