Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens

用离散扩散时间步 Token 做生成式多模态预训练

Kaihang Pan 1 1 1 Equal Contribution. Affiliation:  Zhejiang University    Wang Lin 1 1 1 Equal Contribution. Affiliation:  Zhejiang University    Zhongqi Yue 1 1 1 Equal Contribution. Affiliation:  Nanyang Technological University    Tenglong Ao Affiliation:  Peking University    Liyu Jia Affiliation:  Nanyang Technological University    Wei Zhao Affiliation:  Huawei Singapore Research Center {kaihangpan, linwanglw, junchengli, siliang}@zju.edu.cn , aubrey.tenglong.ao@gmail.com{zhongqi.yue, hanwangzhang}@ntu.edu.sg, liyu002@e.ntu.edu.sg, zhaowei82@huawei.com    Juncheng Li 2 2 2 Juncheng Li is the corresponding author. Affiliation:  Zhejiang University    Siliang Tang Affiliation:  Zhejiang University    Hanwang Zhang Affiliation:  Nanyang Technological University

Abstract

摘要

Recent endeavors in Multimodal Large Language Models (MLLMs) aim to unify visual comprehension and generation by combining LLM and diffusion models, the state-of-the-art in each task, respectively. Existing approaches rely on spatial visual tokens, where image patches are encoded and arranged according to a spatial order (e.g., raster scan). However, we show that spatial tokens lack the recursive structure inherent to languages, hence form an impossible language for LLM to master. In this paper, we build a proper visual language by leveraging diffusion timesteps to learn discrete, recursive visual tokens. Our proposed tokens recursively compensate for the progressive attribute loss in noisy images as timesteps increase, enabling the diffusion model to reconstruct the original image at any timestep. This approach allows us to effectively integrate the strengths of LLMs in autoregressive reasoning and diffusion models in precise image generation, achieving seamless multimodal comprehension and generation within a unified framework. Extensive experiments show that we achieve superior performance for multimodal comprehension and generation simultaneously compared with other MLLMs. Project Page: https://DDT-LLaMA.github.io/.

近期多模态大语言模型（MLLM）试图把 LLM 与扩散模型各自在理解与生成上的优势合在一起，统一视觉理解与生成。现有做法多依赖空间视觉 token：把图像块按空间顺序（如光栅扫描）编码排列。我们指出，空间 token 缺少语言固有的递归结构，对 LLM 而言实质是一种「不可能语言」。本文借助扩散时间步，学习离散、递归的视觉 token，从而构造合适的视觉语言。所提 token 会随时间步增大、噪声图属性逐步丢失而递归补偿，使扩散模型在任意时间步都能重建原图。由此可以把 LLM 的自回归推理能力与扩散模型的精细图像生成能力有效结合，在统一框架内完成多模态理解与生成。大量实验表明，相较其他 MLLM，我们在理解与生成上同时取得更优表现。项目页：https://DDT-LLaMA.github.io/。

1 Introduction

1 引言

Multimodal Large Language Models (MLLMs) [24, 77] strive to unify the comprehension and generation of data across various modalities within the same next-token prediction paradigm of LLM [56, 75]. Specifically, given a user query about comprehension—“What kind of dog is in this picture [IMG]”, or generation—“Turning the picture [IMG] into a sketch”, the model can complete the task by sequentially predicting the appropriate text or image tokens.

多模态大语言模型（MLLM）[24, 77] 希望在与 LLM [56, 75] 相同的下一 token 预测范式下，统一跨模态数据的理解与生成。具体而言，面对理解类提问——「这张图 [IMG] 里是什么狗」，或生成类请求——「把图 [IMG] 改成素描」，模型应依次预测合适的文本或图像 token 完成任务。

The challenge lies in the conflicting objectives of the two tasks. Comprehension pursues a many-to-one mapping that abstracts visual details (e.g., many photos of corgi dogs result in recognition of “corgi”). On the other hand, generation finds a one-to-one mapping that preserves visual details (e.g., the sketched image specific to the query image). To bridge these conflicting objectives, the most straightforward way is to combine LLMs and diffusion models (DMs), which excel in comprehension and generation respectively. There are two main integration strategies. One approach is cascading [53], where the LLM understands user queries and plans instructions, which are then passed to the DM for generation. The other approach involves quantizing [18, 97, 36, 84, 94] or mapping [15] the continuous visual modalities into discrete tokens for LLM processing, followed by decoding these tokens into images using the DM. Recently, Transfusion [95] aims to combine the LLM and DM into a single model, creating a unified framework for both understanding and generating multimodal content.

难点在于两项任务目标冲突。理解追求多对一映射，抽象视觉细节（例如多张柯基照片都识别为「corgi」）；生成则需要一对一映射，保留视觉细节（例如与查询图对应的素描）。为弥合冲突，最直接的做法是组合分别擅长理解与生成的 LLM 与扩散模型（DM）。主要有两类整合策略：一是级联 [53]，由 LLM 理解用户意图并规划指令，再交给 DM 生成；二是把连续视觉模态量化 [18, 97, 36, 84, 94] 或映射 [15] 成离散 token 供 LLM 处理，再用 DM 把 token 解码成图。最近 Transfusion [95] 试图把 LLM 与 DM 并入同一模型，形成统一的多模态理解与生成框架。

Figure 1: Auto-regressive training curves of diffusion timestep tokens (left) and spatial tokens (right) under different degrees of sequence perturbation.

图 1：在不同程度的序列扰动下，扩散时间步 token（左）与空间 token（右）的自回归训练曲线。

Notably, all existing methods are based on spatial visual tokens, which are extracted from image patches and arranged in a spatial order (e.g., row by row in raster scan order). However, following the evaluation in [34], we find that spatial token sequences lack the traits of human language, and essentially form an impossible language [34] for LLM to master. Specifically, the performance of LLMs on language tokens should degrade rapidly when their order is disrupted. However, when we train an LLM to model spatial token sequences in Figure 1 right, the order of tokens has minimal impact on the convergence of the LLM, deviating from phenomenons observed in human language. This finding is perhaps non-surprising, as spatial tokens lack the recursive structure that characterizes languages. For example, one can extend “a man” by recursively embedding phrases to get “a man [[walking] in the park]”.

值得注意的是，现有方法几乎都基于空间视觉 token：从图像块提取并按空间顺序排列（如按行光栅扫描）。但按 [34] 的评测，我们发现空间 token 序列并不具备人类语言的性状，实质构成 LLM 难以掌握的不可能语言 [34]。具体而言，打乱语言 token 顺序时，LLM 表现应迅速下降；而在图 1 右侧用 LLM 建模空间 token 序列时，token 顺序对收敛影响极小，与人类语言现象相悖。这并不意外——空间 token 缺少语言特有的递归结构。例如可以把「a man」递归嵌入短语扩展成「a man [[walking] in the park]」。

To address the limitation of spatial tokens, we propose to learn tokens with recursive structure by Discrete Diffusion Timestep (DDT) tokenization. Specifically, the forward process of a Diffusion Model (DM) incrementally adds Gaussian noise to images at each time-step $t$, which collapses different images into similar ones by gradually losing the visual attributes that differentiate them. We aim to learn an expanding sequence of discrete tokens to make up for the incremental attribute loss as $t$ increases. For example, given a noise-free image $\mathbf{x}_{0}$, we obtain its $t$-specific token sequence $f_{t}(\mathbf{x}_{0})$ by an encoder $f$, with an initial condition $f_{0}(\mathbf{x}_{0})=\emptyset$ (i.e., no attribute loss). At $t$, the noised image is given by $\mathbf{x}_{t}$. We train the DM to reconstruct $\mathbf{x}_{0}$ given $\mathbf{x}_{t}$ and $f_{t}(\mathbf{x}_{0})$, so that the tokens compensate for the attribute loss in $\mathbf{x}_{t}$. As $t$ increases, we learn a recursive sequence given by $f_{t+1}(\mathbf{x}_{0})=\left(f_{t}(\mathbf{x}_{0}\right),\textrm{V}_{t+1})$, where the appended visual token $\textrm{V}_{t+1}$ accounts for the increased attribute loss in $\mathbf{x}_{t+1}$ over $\mathbf{x}_{t}$. Until at $t=T$, $x_{T}$ becomes pure noise where all attributes are lost. We output $f_{T}(\mathbf{x}_{0})$ as the final token sequence for the image $\mathbf{x}_{0}$, which encodes all its visual attributes. We do the same language verification on our DDT tokens in Figure 1 left. With our recursive tokens, we observe findings similar to those on languages.

为克服空间 token 的局限，我们提出用离散扩散时间步（Discrete Diffusion Timestep，DDT）分词来学习具有递归结构的 token。扩散模型前向过程在每个时间步 $t$ 向图像逐步加入高斯噪声，不同图像因区分性视觉属性逐渐丢失而趋于相似。我们希望学习一段不断扩展的离散 token 序列，以弥补随 $t$ 增大而递增的属性损失。例如对无噪声图 $\mathbf{x}_{0}$，用编码器 $f$ 得到与 $t$ 相关的 token 序列 $f_{t}(\mathbf{x}_{0})$，初值 $f_{0}(\mathbf{x}_{0})=\emptyset$（即无属性损失）。在时刻 $t$，噪声图为 $\mathbf{x}_{t}$。我们训练 DM 在给定 $\mathbf{x}_{t}$ 与 $f_{t}(\mathbf{x}_{0})$ 时重建 $\mathbf{x}_{0}$，使这些 token 补偿 $\mathbf{x}_{t}$ 中的属性损失。随 $t$ 增大，学习递归序列 $f_{t+1}(\mathbf{x}_{0})=\left(f_{t}(\mathbf{x}_{0}\right),\textrm{V}_{t+1})$，其中追加的视觉 token $\textrm{V}_{t+1}$ 对应 $\mathbf{x}_{t+1}$ 相对 $\mathbf{x}_{t}$ 多出的属性损失。直至 $t=T$，$x_{T}$ 成为纯噪声、属性尽失，输出 $f_{T}(\mathbf{x}_{0})$ 作为图像 $\mathbf{x}_{0}$ 的最终 token 序列，编码全部视觉属性。我们在图 1 左侧对 DDT token 做同样的语言验证，观察到与语言类似的现象。

Using a DDT tokenizer trained on ImageNet [14], we build an MLLM that simultaneously unseals the potential of LLM in language modeling and DM in image generation. First, we transform images into DDT tokens. Then we train an MLLM with a unified next-token-prediction objective on a vast corpus of image-text pairs (see Section 3), where the model essentially learns to translate between human language and the visual language using DDT tokens. Finally in inference, when MLLM outputs visual tokens, they are mapped to images using our trained DM as decoder. This leads to a unified framework for comprehension and generation. Our main contributions are three-fold:

用在 ImageNet [14] 上训练的 DDT 分词器，我们构建同时释放 LLM 语言建模与 DM 图像生成潜力的 MLLM。首先把图像转成 DDT token；再在大规模图文对上以统一的下一 token 预测目标训练 MLLM（见第 3 节），实质是学习人类语言与 DDT 视觉语言之间的互译；推理时，当 MLLM 输出视觉 token，再用训练好的 DM 作解码器映射回图像。由此形成理解与生成统一的框架。主要贡献有三点：

• • We propose DDT to learn discrete, recursive visual tokens by recovering visual attributes lost in the diffusion process that can subsequently drive an MLLM.

• • 提出 DDT：通过恢复扩散过程中丢失的视觉属性来学习离散、递归的视觉 token，进而驱动 MLLM。

• • We propose a novel way to integrate LLMs and diffusion models with our DDT tokens, producing an effective MLLM for both visual comprehension and generation.

• • 提出用 DDT token 整合 LLM 与扩散模型的新方式，得到兼顾视觉理解与生成的有效 MLLM。

• • Despite using a tokenizer trained on just ImageNet, we achieve a new SOTA for multimodal comprehension and generation (i.e., image generation, image editing, visual comprehension), surpassing MLLMs that use tokenizers trained on large-scale data (e.g., Laion [68] and Coyo [7]).

• • 即便分词器只在 ImageNet 上训练，我们仍在多模态理解与生成（文生图、图像编辑、视觉理解）上取得新 SOTA，超过使用在大规模数据（如 Laion [68]、Coyo [7]）上训练分词器的 MLLM。

2 Related Work

2 相关工作

Research on achieving unified understanding and generation in multimodal models has primarily focused on two main strategies: cascading architectures and tokenization-based methods. Cascading architectures [23, 71, 33, 24] integrate separate modality-specific encoders and decoders, each pre-trained independently, and then fuse their representations through projection layers to create combined models for multimodal tasks. Notable examples include models such as EMU2 [71], which uses pre-trained language models [75] augmented with EVA-02-CLIP-E-plus [70] for comprehension tasks, and cascade an SDXL-initialized [62] diffusion model for visual generation tasks. In contrast, tokenization-based methods [73, 60, 80, 81, 49, 100, 11, 77] aim to create a unified framework by converting visual and textual inputs into a discrete space, and then jointly training a single transformer based solely on next-token prediction, thereby eliminating the need for diffusion or compositional approaches entirely. Moreover, recent advances, such as TransFusion [95] and Show-o [82], explore a blend of diffusion and autoregressive models within a single transformer for enhanced performance. However, although these methods provide a step towards unification, most of them focus on spatial tokens for vision, which are extracted from image patches and arranged in a spatial order (e.g., row by row in raster scan order). These spatial tokens lack the traits of human language, resulting in an inability to seamlessly integrate with human natural language within an MLLM to mutually enhance each other. Consequently, existing MLLMs still lag behind specialized architectures like SDXL [62] in visual generation tasks and LLaVA-1.6 [51] in visual comprehension tasks. These indicate the need for further exploration into more holistic tokenization methods that go beyond spatial representations to achieve more comprehensive multimodal comprehension and generation.

实现多模态统一理解与生成的研究大致沿两条线：级联架构与基于分词的方法。级联架构 [23, 71, 33, 24] 把各自预训练的模态编解码器通过投影层融合，例如 EMU2 [71] 用预训练语言模型 [75] 加 EVA-02-CLIP-E-plus [70] 做理解，再级联以 SDXL [62] 初始化的扩散模型做视觉生成。分词方法 [73, 60, 80, 81, 49, 100, 11, 77] 则把视觉与文本都映射到离散空间，仅用下一 token 预测联合训练单个 Transformer，从而摆脱扩散或组合式流水线。近期 TransFusion [95]、Show-o [82] 等在同一 Transformer 中混合扩散与自回归以提升表现。但这些工作多数仍聚焦从图像块按空间顺序提取的空间 token；它们缺少人类语言性状，难以与自然语言在 MLLM 内无缝融合、相互增强。因此现有 MLLM 在视觉生成上仍落后于 SDXL [62] 一类专用架构，在视觉理解上落后于 LLaVA-1.6 [51]。这表明需要超越空间表示、更整体的分词方法，才能更全面地实现多模态理解与生成。

Figure 2: The overview of our methods. (a): The architecture of diffusion timestep tokenizer encodes an image to a recursive sequence of discrete tokens. (b): An MLLM architecture that unifies comprehension and generation based on next token prediction.

图 2：方法总览。(a)：扩散时间步分词器把图像编码为递归的离散 token 序列。(b)：基于下一 token 预测、统一理解与生成的 MLLM 架构。

3 Diffusion Timestep Tokenizer

3 扩散时间步分词器

Our goal is to train an image tokenizer that encodes an image to a recursive sequence of discrete tokens—to effectively train an MLLM—and decode the tokens back to the image—to render the image tokens predicted by the MLLM during inference. The model consists of 1) an encoder $f$ that maps a noise-free image $\mathbf{x}_{0}$ to a sequence of continuous features $(\hat{\mathrm{V}}_{1},\ldots,\hat{\mathrm{V}}_{T})$; 2) a quantizer that assigns each continuous feature $\hat{\mathrm{V}}_{i}$ to its token embedding $\mathrm{V}_{i}\in\mathcal{C}$ in a fixed size dictionary $\mathcal{C}$; 3) a Diffusion Model (DM) $d$ as a decoder that reconstructs $\hat{\mathbf{x}}_{0}=d\left(\mathbf{x}_{t},t,(\mathrm{V}_{1},\ldots,\mathrm{V}_{t})\right)$, where $\mathbf{x}_{t}$ is a sampled noised image at timestep $t$ and the tuple denotes the first $t$ tokens from the quantizer. The model is trained end-to-end with (primarily) a reconstruction loss. And then we provide details on each part below.

目标是训练图像分词器：把图像编码成递归的离散 token 序列以便训练 MLLM，并在推理时把 MLLM 预测的图像 token 解码回图像。模型包含：1）编码器 $f$，把无噪声图 $\mathbf{x}_{0}$ 映射为连续特征序列 $(\hat{\mathrm{V}}_{1},\ldots,\hat{\mathrm{V}}_{T})$；2）量化器，把每个连续特征 $\hat{\mathrm{V}}_{i}$ 赋值为固定词典 $\mathcal{C}$ 中的 token 嵌入 $\mathrm{V}_{i}\in\mathcal{C}$；3）以扩散模型（DM）$d$ 作为解码器，重建 $\hat{\mathbf{x}}_{0}=d\left(\mathbf{x}_{t},t,(\mathrm{V}_{1},\ldots,\mathrm{V}_{t})\right)$，其中 $\mathbf{x}_{t}$ 为时间步 $t$ 采样的噪声图，元组为量化器给出的前 $t$ 个 token。模型端到端训练，主要以重建损失为主。下文分别说明各部分。

Encoder. As shown in Figure 2(a) left, we use a transformer-based model with learnable query tokens as our encoder. The input to the encoder is a noise-free image (patchified and flattened into image tokens similar to ViT [16]) and the set of $T$ learnable query tokens. Note that the input query tokens have the same size $T$ as the desired output tokens. We first apply 1D and 2D sinusoidial position embedding on the query and image tokens, respectively. Following SD3 [19], we then use two independent transformers to process the query and image tokens, and join them for the attention operation. Finally, only the transformed query tokens are kept as the output $(\hat{\mathrm{V}}_{1},\ldots,\hat{\mathrm{V}}_{T})$, where $\hat{\mathrm{V}}_{i}\in\mathbb{R}^{n}$ is an $n$-dimensional feature vector.

编码器。如图 2(a) 左，我们用带可学习 query token 的 Transformer 编码器。输入为无噪声图（类似 ViT [16] 切块展平为图像 token）以及 $T$ 个可学习 query token；query 数量与期望输出 token 数同为 $T$。先分别对 query 与图像 token 施加 1D / 2D 正弦位置编码。遵循 SD3 [19]，用两个独立 Transformer 分别处理 query 与图像 token，再在注意力中拼接；最终只保留变换后的 query token 作为输出 $(\hat{\mathrm{V}}_{1},\ldots,\hat{\mathrm{V}}_{T})$，其中 $\hat{\mathrm{V}}_{i}\in\mathbb{R}^{n}$ 为 $n$ 维特征向量。

Quantizer. We feed the encoder output to a standard Vector Quantization (VQ) module [76]. Specifically, the VQ has a fixed size dictionary $\mathcal{C}$ ($|\mathcal{C}|=65536$ in our experiments), where each entry is a vector of dimension $m$. We first use a linear layer to project the encoder output from $\mathbb{R}^{n}$ to $\mathbb{R}^{m}$, and then find $\mathrm{V}_{i}$ in $\mathcal{C}$ with the largest cosine similarity with each $\hat{\mathrm{V}}_{i}$. Note that we use an EMA-variant of VQ to improve training stability, i.e., updating $\mathcal{C}$ by gradually moving each entry towards the cluster center of its matched encoder outputs. To improve codebook usage, we use two tricks: 1) We use a small $m=16$ based on the empirical results in [84]; 2) Inspired from [89], we monitor dead entries in $\mathcal{C}$ at each training step (i.e., rarely matched with any $\mathrm{V}_{i}$) and setting them to random $\mathrm{V}_{i}$ in a training batch.

量化器。把编码器输出送入标准向量量化（VQ）模块 [76]。词典 $\mathcal{C}$ 大小固定（实验中 $|\mathcal{C}|=65536$），每项为 $m$ 维向量。先用线性层把编码器输出从 $\mathbb{R}^{n}$ 投到 $\mathbb{R}^{m}$，再按余弦相似度最大选取 $\mathrm{V}_{i}\in\mathcal{C}$。采用 EMA 版 VQ 以提升训练稳定性，即逐步把各码本项移向其匹配编码器输出的簇心。为提高码本利用率有两招：1）按 [84] 经验取较小 $m=16$；2）受 [89] 启发，每步监控 $\mathcal{C}$ 中几乎不被匹配的「死码」，重置为当前 batch 中随机 $\mathrm{V}_{i}$。

Decoder. Our decoder is a DM. As shown in Figure 2(a) right, we use the same MMDiT [19] architecture for our decoder $d$ with slight modifications: 1) Instead of inputting text tokens, we input a sequence of quantized tokens; 2) We use a linear layer to project the $m$-dimensional vector to the latent dimension of the MMDiT; 3) We do not use pooled token embedding as in SD3. Given a timestep $t$, a sampled noisy image $\mathbf{x}_{t}$ from $\mathbf{x}_{0}$, a sequence of tokens from the quantizer, $d$ is trained to output a reconstructed $\mathbf{x}_{0}$ as shown below.

解码器。解码器是 DM。如图 2(a) 右，采用与 SD3 相同的 MMDiT [19] 架构并做少量修改：1）输入改为量化 token 序列而非文本 token；2）用线性层把 $m$ 维向量投到 MMDiT 潜空间维；3）不使用 SD3 中的 pooled token 嵌入。给定时间步 $t$、由 $\mathbf{x}_{0}$ 采样的噪声图 $\mathbf{x}_{t}$ 以及量化器给出的 token 序列，训练 $d$ 输出重建的 $\mathbf{x}_{0}$，如下所述。

Training. Recall that we aim to learn a recursive sequence in the form $f_{t}(\mathbf{x}_{0})=\left(f_{t-1}(\mathbf{x}_{0}\right),\textrm{V}_{t})$ at each $t$, such that $f_{t}(\mathbf{x}_{0})$ makes up for the attribute loss in the noisy image $\mathbf{x}_{t}$. Hence we use an expanding set of tokens $({\mathrm{V}}_{1},\ldots,{\mathrm{V}}_{t})$ as the input to the decoder $d$, and train everything end-to-end with the reconstruction loss:

训练。我们要在每个 $t$ 学到形如 $f_{t}(\mathbf{x}_{0})=\left(f_{t-1}(\mathbf{x}_{0}\right),\textrm{V}_{t})$ 的递归序列，使 $f_{t}(\mathbf{x}_{0})$ 弥补噪声图 $\mathbf{x}_{t}$ 中的属性损失。因此向解码器 $d$ 输入不断扩展的 token 集合 $({\mathrm{V}}_{1},\ldots,{\mathrm{V}}_{t})$，并以重建损失端到端训练：

$$\small\mathcal{L}=\mathop{\mathbb{E}}_{t,\mathbf{x}_{0},\epsilon}\;\underbrace{\left[\lVert d\left(t\epsilon+(1-t)\mathbf{x}_{0},t,({\mathrm{V}}_{1},\ldots,{\mathrm{V}}_{t})\right)-\mathbf{x}_{0}\rVert^{2}\right]}_{\text{$\mathcal{L}_{t}$}}, \tag{1}$$

where we follow Rectified Flow [54] to sample the noisy $\mathbf{x}_{t}=t\epsilon+(1-t)\mathbf{x}_{0}$ with the Gaussian noise $\epsilon$. We also use a standard commitment loss $\sum_{i=1}^{T}\lVert\hat{\mathrm{V}}_{i}-\mathrm{V}_{i}\rVert^{2}$ to regularize the encoder output (continuous vector) to be similar with the matched quantized vector.

其中按 Rectified Flow [54] 用高斯噪声 $\epsilon$ 采样噪声图 $\mathbf{x}_{t}=t\epsilon+(1-t)\mathbf{x}_{0}$。另加标准 commitment 损失 $\sum_{i=1}^{T}\lVert\hat{\mathrm{V}}_{i}-\mathrm{V}_{i}\rVert^{2}$，约束编码器连续输出接近匹配的量化向量。

	GenEval	T2I-CompBench	DrawBench
Method	Overall$\uparrow$	SingObj$\uparrow$	TwoObj$\uparrow$	Counting$\uparrow$	Colors$\uparrow$	Position$\uparrow$	ColorAttri$\uparrow$	Color$\uparrow$	Shape$\uparrow$	Texture$\uparrow$	Spatial$\uparrow$	NonSpatial$\uparrow$	Complex$\uparrow$	Clip-T$\uparrow$
Diffusion-based T2I Specialists
DALL-E 2  [65]	0.52	0.94	0.66	0.49	0.77	0.10	0.19	0.575	0.546	0.637	0.128	0.304	0.370	-
SDv1.5 [66]	0.43	0.97	0.38	0.35	0.76	0.04	0.06	0.373	0.365	0.422	0.131	-	-	-
SDv2.1 [66]	0.50	0.98	0.51	0.44	0.85	0.07	0.17	0.569	0.450	0.498	0.174	-	-	-
SDXL [62]	0.55	0.98	0.74	0.39	0.85	0.15	0.23	0.637	0.541	0.564	0.203	0.311	0.409	-
PixArt-alpha [9]	0.48	0.98	0.50	0.44	0.80	0.08	0.07	0.689	0.558	0.704	0.208	0.318	0.412	-
DALL-E 3  [4]	0.67	0.96	0.87	0.47	0.83	0.43	0.45	0.811	0.675	0.807	-	-	-	-
SD3 [19]	0.74	0.99	0.94	0.72	0.89	0.33	0.60	-	-	-	-	-	-	-
MLLM-based Generalist
SEED-LLaMA [23]	0.33	0.89	0.17	0.25	0.59	0.02	0.03	0.176	0.184	0.202	0.068	0.289	0.227	0.220
LaVIT [33]	0.51	0.93	0.70	0.43	0.67	0.17	0.18	0.439	0.457	0.575	0.151	0.311	0.320	0.259
Emu2-Gen [71]	0.35	0.93	0.28	0.14	0.70	0.05	0.02	0.338	0.359	0.400	0.099	0.299	0.285	0.235
SEED-X [24]	0.49	0.96	0.57	0.29	0.82	0.14	0.15	0.657	0.492	0.603	0.172	0.306	0.361	0.265
VILA-U [81]	0.40	0.88	0.42	0.25	0.69	0.08	0.09	0.568	0.433	0.501	0.154	0.307	0.334	0.262
Lumina-mGPT [49]	0.52	0.99	0.70	0.34	0.85	0.08	0.17	0.556	0.465	0.588	0.176	0.312	0.343	0.267
Transfusion [95]	0.63	-	-	-	-	-	-	-	-	-	-	-	-	-
Emu3 [77]	0.54	0.98	0.71	0.34	0.81	0.17	0.21	0.611	0.473	0.618	0.137	0.300	0.345	0.265
Ours	0.66	0.99	0.64	0.56	0.87	0.39	0.48	0.728	0.514	0.642	0.176	0.314	0.345	0.267

Table 1: Comparison with state-of-the-art models on GenEval, T2I CompBench and DrawBench on zero-shot text-to-image generation. The best results among MLLM-based Generalists are in bold fonts. And the best results among all methods are underline.

	GenEval	T2I-CompBench	DrawBench
Method	Overall$\uparrow$	SingObj$\uparrow$	TwoObj$\uparrow$	Counting$\uparrow$	Colors$\uparrow$	Position$\uparrow$	ColorAttri$\uparrow$	Color$\uparrow$	Shape$\uparrow$	Texture$\uparrow$	Spatial$\uparrow$	NonSpatial$\uparrow$	Complex$\uparrow$	Clip-T$\uparrow$
基于扩散的文生图专用模型
DALL-E 2  [65]	0.52	0.94	0.66	0.49	0.77	0.10	0.19	0.575	0.546	0.637	0.128	0.304	0.370	-
SDv1.5 [66]	0.43	0.97	0.38	0.35	0.76	0.04	0.06	0.373	0.365	0.422	0.131	-	-	-
SDv2.1 [66]	0.50	0.98	0.51	0.44	0.85	0.07	0.17	0.569	0.450	0.498	0.174	-	-	-
SDXL [62]	0.55	0.98	0.74	0.39	0.85	0.15	0.23	0.637	0.541	0.564	0.203	0.311	0.409	-
PixArt-alpha [9]	0.48	0.98	0.50	0.44	0.80	0.08	0.07	0.689	0.558	0.704	0.208	0.318	0.412	-
DALL-E 3  [4]	0.67	0.96	0.87	0.47	0.83	0.43	0.45	0.811	0.675	0.807	-	-	-	-
SD3 [19]	0.74	0.99	0.94	0.72	0.89	0.33	0.60	-	-	-	-	-	-	-
基于 MLLM 的通才模型
SEED-LLaMA [23]	0.33	0.89	0.17	0.25	0.59	0.02	0.03	0.176	0.184	0.202	0.068	0.289	0.227	0.220
LaVIT [33]	0.51	0.93	0.70	0.43	0.67	0.17	0.18	0.439	0.457	0.575	0.151	0.311	0.320	0.259
Emu2-Gen [71]	0.35	0.93	0.28	0.14	0.70	0.05	0.02	0.338	0.359	0.400	0.099	0.299	0.285	0.235
SEED-X [24]	0.49	0.96	0.57	0.29	0.82	0.14	0.15	0.657	0.492	0.603	0.172	0.306	0.361	0.265
VILA-U [81]	0.40	0.88	0.42	0.25	0.69	0.08	0.09	0.568	0.433	0.501	0.154	0.307	0.334	0.262
Lumina-mGPT [49]	0.52	0.99	0.70	0.34	0.85	0.08	0.17	0.556	0.465	0.588	0.176	0.312	0.343	0.267
Transfusion [95]	0.63	-	-	-	-	-	-	-	-	-	-	-	-	-
Emu3 [77]	0.54	0.98	0.71	0.34	0.81	0.17	0.21	0.611	0.473	0.618	0.137	0.300	0.345	0.265
Ours	0.66	0.99	0.64	0.56	0.87	0.39	0.48	0.728	0.514	0.642	0.176	0.314	0.345	0.267

表 1：零样本文生图在 GenEval、T2I-CompBench 与 DrawBench 上与现有方法对比。MLLM 通才中最优结果加粗；全体方法最优结果加下划线。

4 DDT-LLaMA

The process begins by converting images into a 1D recursive discrete sequence like a foreign language that LLM can read. This transformation allows us to pursue a unified next-token-prediction training objective across a vast array of image-text data. To achieve this, visual tokens are concatenated with text tokens to form a multi-modal sequence. Special tokens, [BOV] and [EOV], are strategically placed at the beginning and end of the visual sequence to distinguish between modalities. We Initialize the MLLM from a pre-trained LLM, specifically the LLaMA3-8B [17] model, with an expansion of its vocabulary by 65,536 visual codes. Then MLLM adopts the general Language Modeling (LM) objective to directly maximize the likelihood of each multi-modal sequence in an auto-regressive manner:

流程首先把图像转成一维递归离散序列，像一门 LLM 可读的「外语」。这使得我们能在海量图文数据上采用统一的下一 token 预测训练目标。视觉 token 与文本 token 拼接成多模态序列；在视觉段首尾放置特殊 token [BOV]、[EOV] 以区分模态。MLLM 以预训练 LLaMA3-8B [17] 初始化，词表扩展 65,536 个视觉码。随后以通用语言建模（LM）目标，自回归最大化每条多模态序列的似然：

$$p(y)=\sum_{y\in\mathcal{D}}\sum_{i=1}^{S}\log P_{\theta}(y_{i}|y_{<i}). \tag{2}$$

where $y$ is the text or visual token with $S$ as the sequence length. Since both image and text are represented as discrete token IDs, we can use the cross-entropy to supervise the token prediction at each position for both modalities with a shared prediction head. The training regimen is composed of two stages, during which all parameters of the MLLM are meticulously fine-tuned.

其中 $y$ 为文本或视觉 token，$S$ 为序列长度。由于图像与文本都表示为离散 token ID，可用同一预测头、以交叉熵在各位置监督两种模态的 token 预测。训练分两阶段，期间精细微调 MLLM 全部参数。

Stage1: Pre-training.

In the first stage, we aim to enhance the alignment between DDT tokens and text tokens, transitioning from LLM to MLLM. Our pre-training dataset, sourced from Laion [68] and Coyo [7], consists of 200M text-to-image pairs. Each pair is structured in the format “[BOS] [BOV] [EOV] [EOS]”, where [BOS] and [EOS] are special tokens from the text tokenizer. Moreover, to prevent the forgetting of textual capabilities, we also incorporate pure text data at a fusion ratio of 10%. The pre-training is conducted on 512 Ascend 910B NPUs, lasting for nearly two weaks.

阶段 1：预训练

第一阶段目标是增强 DDT token 与文本 token 的对齐，完成从 LLM 到 MLLM 的过渡。预训练数据来自 Laion [68] 与 Coyo [7]，共 2 亿图文对，格式为「[BOS] [BOV] [EOV] [EOS]」，其中 [BOS]、[EOS] 为文本分词器特殊符号。为防止文本能力遗忘，另以 10% 融合比混入纯文本。预训练在 512 块 Ascend 910B NPU 上进行，历时约两周。

Stage2: Instruction Tuning.

In the second stage, we perform multimodal instruction tuning to align our MLLM with human instructions through supervised fine-tuning on public image-text datasets. To enhance the MLLM’s comprehension and generation capabilities in complex scenarios, we employ supervised training data from visual comprehension and generation tasks. We use the instruction format as “USER: ASSISTANT: ” and only the content of is accounted for loss. See Appendix A and B for more details.

阶段 2：指令微调

第二阶段做多模态指令微调，用公开图文数据的监督微调使 MLLM 对齐人类指令。为提升复杂场景下的理解与生成，我们采用来自视觉理解与生成任务的监督数据。指令格式为「USER: ASSISTANT: 」，仅对 计算损失。更多细节见附录 A、B。

Inference.

After DDT-LLaMA predicts [BOV] token, it consecutively performs $T$ next-token-predictions to generate a set of discrete DDT tokens $\{V_{1},...,V_{T}\}$. We then feed these tokens into the decoder to guide the denoising of random noise into an image over $T$ timesteps through a DDPM process. Similar to the training procedure, at timestep $t$ during inference, we provide only the first $t$ tokens as the guidance, while the remaining $(T-t)$ tokens are masked as $0$.

推理

DDT-LLaMA 预测出 [BOV] 后，连续做 $T$ 次下一 token 预测，得到离散 DDT token 集合 $\{V_{1},...,V_{T}\}$。再把这些 token 送入解码器，经 $T$ 步 DDPM 过程把随机噪声去噪成图。与训练类似，推理时步 $t$ 只提供前 $t$ 个 token 作引导，其余 $(T-t)$ 个 mask 为 0。

5 Experiment

5 实验

Figure 3: Qualitative results of DDT-LLaMA text-to-image generation.

图 3：DDT-LLaMA 文生图定性结果。

EVR MA5K MagicBrush Method L1$\downarrow$ CVS$\uparrow$ L1$\downarrow$ LPIPS$\downarrow$ L1$\downarrow$ CVS$\uparrow$ Image editing Specialist InsPix2Pix [6] 18.9 81.4 17.6 35.9 10.1 85.2 MGIE [21] 16.3 81.7 13.3 29.8 8.2 91.1 UltraEdit [93] 14.4 81.6 16.5 35.2 6.6 88.4 MLLM-based Generalist GILL [35] 31.8 65.0 27.4 44.3 28.3 75.2 Emu2-Gen [71] 22.8 80.3 20.5 36.4 19.9 85.7 SEED-LLaMA [23] 28.4 72.3 24.6 39.0 24.5 80.9 LaVIT [33] 26.8 73.8 25.1 36.9 25.3 81.1 SEED-X-Edit [24] 16.7 81.9 14.6 32.7 18.4 86.1 Ours 14.5 85.5 14.2 29.6 7.1 92.4

Table 2: Zero-shot image editing results.

EVR MA5K MagicBrush Method L1$\downarrow$ CVS$\uparrow$ L1$\downarrow$ LPIPS$\downarrow$ L1$\downarrow$ CVS$\uparrow$ 图像编辑专用模型 InsPix2Pix [6] 18.9 81.4 17.6 35.9 10.1 85.2 MGIE [21] 16.3 81.7 13.3 29.8 8.2 91.1 UltraEdit [93] 14.4 81.6 16.5 35.2 6.6 88.4 基于 MLLM 的通才模型 GILL [35] 31.8 65.0 27.4 44.3 28.3 75.2 Emu2-Gen [71] 22.8 80.3 20.5 36.4 19.9 85.7 SEED-LLaMA [23] 28.4 72.3 24.6 39.0 24.5 80.9 LaVIT [33] 26.8 73.8 25.1 36.9 25.3 81.1 SEED-X-Edit [24] 16.7 81.9 14.6 32.7 18.4 86.1 Ours 14.5 85.5 14.2 29.6 7.1 92.4

表 2：零样本图像编辑结果。

Figure 4: Qualitative comparison with EMU3 on T2I generation. DDT-LLaMA better responses to prompts related to counting or position.

图 4：与 EMU3 的文生图定性对比。DDT-LLaMA 对计数、位置类提示响应更好。

5.1 Text-to-Image Generation

5.1 文生图

Automated Metric Evaluation.

To evaluate our model’s capabilities in zero-shot image synthesis, we first conduct an automated metric evaluation on 3 popular text-to-image benchmarks: GenEval [25], T2I-CompBench [30], and DrawBench [67]. The comparison results of DDT-LLaMA against Diffusion-based T2I Specialists, and MLLM-based generalists are shown in Table 1. In most settings, our method surpasses other MLLMs and is even competitive with several diffusion-based models. For example, in the overall GenEval score, DDT-LLaMA significantly outperforms SEED-X [24], Emu3 [77], and the specialist method, SDXL [62]. This highlights that our model facilitates better vision-text alignment.

自动指标评测

为评估零样本图像合成能力，我们先在三个常用文生图基准上做自动评测：GenEval [25]、T2I-CompBench [30] 与 DrawBench [67]。DDT-LLaMA 与基于扩散的文生图专用模型、基于 MLLM 的通才模型对比见表 1。多数设定下，我们的方法超过其他 MLLM，甚至可与部分扩散模型竞争。例如 GenEval 总分上，DDT-LLaMA 显著优于 SEED-X [24]、Emu3 [77] 以及专用模型 SDXL [62]，说明模型促进了更好的视觉—文本对齐。

Furthermore, through detailed analysis of the results, we find that compared to other MLLMs, DDT-LLaMA achieves higher scores in tasks related to color, counting, and position. It excels in understanding object attributes such as color, quantity, and the spatial relationships between objects. However, it underperforms in tasks of ’TwoObj’, which simply requires generating two objects with no additional constraints. This is largely due to the fact that the image-text captions in our training data rarely include this straightforward prompt format.

进一步细分结果发现，相较其他 MLLM，DDT-LLaMA 在颜色、计数、位置相关任务得分更高，更能理解物体颜色、数量及物体间空间关系。但在「TwoObj」（只需生成两个无额外约束物体）上偏弱，很大程度上是因为训练图文标题很少出现这种直白提示格式。

Figure 5: Qualitative comparison on image editing.

图 5：图像编辑定性对比。

Qualitative Examples.

In Figure 10, we present some qualitative examples of DDT-LLaMA on image synthesis tasks. It can be observed that DDT-LLaMA effectively follows various types of instructions, including complex ones such as generating surreal images and multi-condition combined prompts, to generate semantically consistent images. Of course, we acknowledge that there is room for improvement in the aesthetic quality of the images. This limitation stems from the fact that our current tokenizer was only trained on ImageNet with a resolution of 256x256. So its performance in reconstructing open-domain images is inferior to that of baseline methods (e.g., the EMU3 tokenizer MOVQ), which restricts our capability in T2I generation. More qualitative examples are in Appendix C.

定性样例

图 10 给出 DDT-LLaMA 在图像合成上的部分定性样例。可见它能跟随多种指令——包括生成超现实图像、多条件组合提示——产出语义一致的图像。当然，美学质量仍有提升空间：当前分词器仅在 256×256 的 ImageNet 上训练，开放域重建不如基线（如 EMU3 的 MOVQ 分词器），从而限制了文生图能力。更多样例见附录 C。

Moreover, Figure 4 presents a direct comparison between DDT-LLaMA and Emu3 on prompts involving counting, color, and position. Though Emu3 generates images with higher aesthetic quality, it struggles to accurately respond to such prompts. In contrast, DDT-LLaMA generates images that better align with the object attributes and the spatial relationships specified in the prompt.

此外图 4 直接对比 DDT-LLaMA 与 Emu3 在计数、颜色、位置类提示上的表现。Emu3 图像更美观，但难以准确响应此类提示；相较之下，DDT-LLaMA 生成的图像更贴合提示中指定的物体属性与空间关系。

5.2 Image Editing

5.2 图像编辑

Automated Metric Evaluation.

We evaluate our model in zero-shot instruction-based image editing across three datasets: EVR [72], MA5k [69], and MagicBrush [90]. Following [21], for EVR and MagicBrush, we treat L1 and CVS (Clip-Visual-Similarity) as the evaluation metrics. For MA5K, we utilize L1 and LPIPS [91]) for evaluation. The results are shown in Table 2.

自动指标评测

我们在 EVR [72]、MA5k [69]、MagicBrush [90] 三个数据集上做零样本指令图像编辑评测。依 [21]，EVR 与 MagicBrush 用 L1 与 CVS（Clip-Visual-Similarity）；MA5K 用 L1 与 LPIPS [91]。结果见表 2。

We find that DDT-LLaMA demonstrates significant superiority over existing MLLMs across all test datasets and metrics. It not only effectively comprehends the instruction to execute editing, but also excels at preserving image fidelity, which is rarely seen in prior work. Furthermore, when compared with specialist models, DDT-LLaMA still shows stronger performance. It consistently outperforms InsPix2Pix [6] , also showing greater effectiveness than MGIE [21] and UltraEdit [93] across most datasets.

DDT-LLaMA 在全部测试集与指标上显著优于现有 MLLM：既能理解指令完成编辑，又擅长保持图像保真——这在先前工作中少见。与专用模型相比仍更强：持续超过 InsPix2Pix [6]，并在多数数据集上优于 MGIE [21] 与 UltraEdit [93]。

Qualitative Examples.

Figure 13 shows some qualitative image editing examples of DDT-LLaMA and baselines[24, 93]. DDT-LLaMA supports a wide spectrum of editing operations including both local change (e.g., removal, replacement) and global change (change time, manipulation). For each case, our output images not only resemble the source image to a high degree but also are coherent with the instruction, demonstrating that DDT-LLaMA achieves a great trade-off between fidelity and editability.

定性样例

图 13 展示 DDT-LLaMA 与基线 [24, 93] 的部分图像编辑样例。DDT-LLaMA 支持从局部（移除、替换）到全局（改时间、操纵）的广泛编辑操作。各例输出既高度像源图，又与指令一致，说明在保真与可编辑性之间取得良好折中。

Models Size Image Gen Image caption VQA MLLM Bench NoCaps Flickr30K VQA GQA OKVQA VizWiz MME SEEDBench POPE Specialized Visual Comprehension MLLMs InstructBlip [13] 7B ✗ 123.1 82.4 - 49.2 - 34.5 1212.8 58.8 - QWenVL-Chat [3] 7B ✗ 120.2 81.0 78.2 57.5 56.6 38.9 1487.57 58.2 - LLaVA-1.5 [50] 7B ✗ 105.5 - 78.5 62.0 53.4 50.0 1510.7 64.3 85.9 mPLUG-Owl2 [83] 7B ✗ - 85.1 79.4 56.1 57.7 54.5 1450.3 57.8 86.2 ShareGPT4V [10] 7B ✗ - - 80.6 63.3 - 57.2 1567.4 69.7 - LLaVA-1.6(HD) [51] 7B ✗ 88.3 - 80.6 64.2 44.3 60.6 - 64.7 86.5 VILA [44] 7B ✗ - - 79.9 62.3 - 57.9 1533.0 61.1 85.5 MLLMs for both Visual Comprehension and Generation Emu2-Chat [71] 37B - 119.8 86.0 84.9 65.1 64.8 54.9 1315.7 62.8 - SEED-LLaMA [23] 8B $\checkmark$ 90.4 66.7 66.2 34.8 45.9 55.1 736.7 51.5 - VILA-U [81] 7B $\checkmark$ 95.4 70.3 75.3 58.3 59.6 35.3 1336.2 56.3 83.9 LaVIT [33] 7B $\checkmark$ 114.2 83.0 66.0 46.8 54.6 38.5 997.9 57.3 82.2 Emu3 [77] 8B - 117.5 82.5 75.1 60.3 61.2 50.5 - 68.2 85.2 Ours 8B $\checkmark$ 124.2 88.5 76.4 66.3 65.9 57.3 1378.9 67.9 86.9

Table 3: Comparison for multimodal comprehension on vision-language benchmarks.

Models Size Image Gen Image caption VQA MLLM Bench NoCaps Flickr30K VQA GQA OKVQA VizWiz MME SEEDBench POPE 专用视觉理解 MLLM InstructBlip [13] 7B ✗ 123.1 82.4 - 49.2 - 34.5 1212.8 58.8 - QWenVL-Chat [3] 7B ✗ 120.2 81.0 78.2 57.5 56.6 38.9 1487.57 58.2 - LLaVA-1.5 [50] 7B ✗ 105.5 - 78.5 62.0 53.4 50.0 1510.7 64.3 85.9 mPLUG-Owl2 [83] 7B ✗ - 85.1 79.4 56.1 57.7 54.5 1450.3 57.8 86.2 ShareGPT4V [10] 7B ✗ - - 80.6 63.3 - 57.2 1567.4 69.7 - LLaVA-1.6(HD) [51] 7B ✗ 88.3 - 80.6 64.2 44.3 60.6 - 64.7 86.5 VILA [44] 7B ✗ - - 79.9 62.3 - 57.9 1533.0 61.1 85.5 同时支持视觉理解与生成的 MLLM Emu2-Chat [71] 37B - 119.8 86.0 84.9 65.1 64.8 54.9 1315.7 62.8 - SEED-LLaMA [23] 8B $\checkmark$ 90.4 66.7 66.2 34.8 45.9 55.1 736.7 51.5 - VILA-U [81] 7B $\checkmark$ 95.4 70.3 75.3 58.3 59.6 35.3 1336.2 56.3 83.9 LaVIT [33] 7B $\checkmark$ 114.2 83.0 66.0 46.8 54.6 38.5 997.9 57.3 82.2 Emu3 [77] 8B - 117.5 82.5 75.1 60.3 61.2 50.5 - 68.2 85.2 Ours 8B $\checkmark$ 124.2 88.5 76.4 66.3 65.9 57.3 1378.9 67.9 86.9

表 3：视觉—语言基准上的多模态理解对比。

5.3 Vision-Language Comprehension

5.3 视觉—语言理解

We also evaluate the vision-language comprehension capabilities across various public benchmarks [1, 61, 2, 32, 55, 27, 20, 38, 42]. The primary results, detailed in Table 3, compare two categories of methods: specialized visual comprehension MLLMs [13, 3, 53, 83, 10, 51, 44] and MLLMs designed for both visual comprehension and generation [71, 24, 81, 33, 77]. Notably, DDT-LLaMA is an encoder-free MLLM without pretrained encoders such as CLIP [64]. Compared to other MLLMs, it significantly surpasses its counterparts across multiple benchmarks, even without relying on a specialized pretrained CLIP. It also outperforms many MLLMs that focus solely on comprehension tasks, highlighting the inherent capabilities and promising potential of DDT tokens in multimodal understanding.

我们还在多个公开基准 [1, 61, 2, 32, 55, 27, 20, 38, 42] 上评估视觉—语言理解能力。主要结果见表 3，对比两类方法：专用视觉理解 MLLM [13, 3, 53, 83, 10, 51, 44]，以及同时面向理解与生成的 MLLM [71, 24, 81, 33, 77]。值得注意的是，DDT-LLaMA 是无 CLIP [64] 等预训练编码器的无编码器 MLLM。相较其他 MLLM，它在多个基准上显著领先，甚至不依赖专用预训练 CLIP；也超过许多只做理解的 MLLM，凸显 DDT token 在多模态理解上的内在能力与潜力。

5.4 In-Depth Analysis

5.4 深入分析

Method Reconstruction (PSNR) $\uparrow$ Class-cond (FID) $\downarrow$ VQGAN [18] 18.9 15.8 VQGAN-LC [97] 20.8 15.4 RQ-Transformer [36] - 7.6 MoVQ [94] 21.5 7.1 Ours 21.7 6.1

Table 4: Reconstruction quality on ImageNet validation split and FID score comparison for autoregressive class-conditional synthesis on 256 $\times$ 256 ImageNet.

Method Reconstruction (PSNR) $\uparrow$ Class-cond (FID) $\downarrow$ VQGAN [18] 18.9 15.8 VQGAN-LC [97] 20.8 15.4 RQ-Transformer [36] - 7.6 MoVQ [94] 21.5 7.1 Ours 21.7 6.1

表 4：ImageNet 验证集重建质量，以及 256$\times$256 ImageNet 自回归类别条件合成的 FID 对比。

5.4.1 Class-conditional Generation of DDT

Our tokenizer is trained on 256×256 images from the ImageNet train split. The PSNR between the reconstructed and original images from the validation split reaches 21.7, outperforming spatial tokenizers trained on ImageNet (using a comparable number of tokens).

5.4.1 DDT 的类别条件生成

分词器在 ImageNet 训练划分的 256×256 图像上训练。验证集重建相对原图的 PSNR 达 21.7，优于在 ImageNet 上以相近 token 数训练的空间分词器。

To directly compare the performance of DDT tokens with other spatial tokens for image generation, we also train an autoregressive GPT model for class-conditional image generation, following  [18]. We compare our model with several previous SOTA spatial tokenizers [18, 36, 94] for autoregressive modeling of class-conditional image synthesis on ImageNet. As shown in Table 4, under the same autoregressive paradigm, our method achieves a superior FID score, demonstrating stronger performance than tokenizers that generate spatially-based visual tokens.

为直接比较 DDT token 与其他空间 token 在图像生成上的表现，我们还按 [18] 训练自回归 GPT 做类别条件图像生成，并与若干先前 SOTA 空间分词器 [18, 36, 94] 在 ImageNet 自回归类别条件合成上对比。如表 4，在相同自回归范式下，我们的方法取得更优 FID，强于生成空间视觉 token 的分词器。

Figure 6: Results of Counterfactual Interpolation with DDT tokens and VQGAN tokens.

图 6：DDT token 与 VQGAN token 的反事实插值结果。

Figure 7: Scaling MLLM size (2B, 8B) and training compute (50%, 75%, 100% of total tokens) improves visual quality.

图 7：扩大 MLLM 规模（2B、8B）与训练算力（总 token 的 50%、75%、100%）可提升视觉质量。

Counterfactual Interpolation.

To directly show that DDT tokens are recursive and decouple visual attributes at various granularities, we first conduct counterfactual interpolation with both DDT tokens and VQGAN tokens. Specifically, for the token sequence derived from an image, we replace a subset of tokens with those from another image, while keeping the remaining tokens fixed, with the resulting sequence fed into the decoder for image generation.

反事实插值

为直接表明 DDT token 具有递归性并在不同粒度解耦视觉属性，我们先对 DDT token 与 VQGAN token 做反事实插值：对一幅图得到的 token 序列，用另一幅图的部分 token 替换子集、其余固定，再送入解码器生成图像。

As shown in Figure 16, for VQGAN tokens, counterfactual interpolation actually performs CutMix [88], where regions from two images are concatenated. In contrast, DDT tokens, with their disentangled representation, ensure that only the attributes captured by the substituted tokens change in the generated counterfactuals, which allows for a seamless semantic blending of the two images.

如图 16，VQGAN token 的反事实插值实质是 CutMix [88]，把两图区域拼接；相较之下，DDT token 以解耦表示保证生成反事实中仅被替换 token 所刻画的属性变化，从而实现两图的平滑语义融合。

Decoding Image with an expanding subset of DDT tokens.

To further illustrate the recursive nature of DDT tokens, we only input a subset of the first $t$ timestep tokens that are autoregressively sampled by DDT-LLaMA, into the decoder for image synthesis ($t$ ranges from 1 to 480). As shown in Figure 17, with the number of tokens increasing, the image’s attributes are progressively recovered. Initially, the decoder reconstructs fine details, with contours and color information gradually completed until the full structure of the image is outlined. This finding further demonstrates that DDT tokens are recursive, and DDT-LLaMA effectively disentangles visual attributes, progressively inferring visual features at varying granularities based on textual instructions. This behavior aligns with our expectations for the DDT token design.

用扩展的 DDT token 子集解码图像

为进一步说明 DDT token 的递归性，我们只把 DDT-LLaMA 自回归采样得到的前 $t$ 个时间步 token 子集送入解码器合成图像（$t$ 从 1 到 480）。如图 17，随 token 增多，图像属性逐步恢复：起初重建细节，轮廓与颜色信息逐渐补全，直至勾勒完整结构。这进一步表明 DDT token 是递归的，且 DDT-LLaMA 能有效解耦视觉属性，按文本指令逐步推断不同粒度的视觉特征，符合我们对 DDT token 设计的预期。

Figure 8: Decoding images with an expanding subset of autoregressive-sampled DDT tokens.

图 8：用自回归采样得到的扩展 DDT token 子集解码图像。

5.4.3 Perplexity of Text Generation

To further demonstrate that our recursive DDT tokens can facilitate the synergism between comprehension and generation, we record the perplexity of text prediction during auto-regressive multimodal pre-training, with VQGAN tokens and DDT tokens, respectively. As shown in Figure 9(a), training with DDT tokens results in a decrease in perplexity for text generation, whereas training with VQGAN tokens yields the opposite conclusion. This highlights that our DDT tokens are better suited for unified autoregressive modeling with text tokens, much like a foreign language that LLMs can read.

5.4.3 文本生成困惑度

为进一步证明递归 DDT token 能促进理解与生成协同，我们分别在 VQGAN token 与 DDT token 的自回归多模态预训练中记录文本预测困惑度。如图 9(a)，用 DDT token 训练时文本生成困惑度下降，而用 VQGAN token 则相反。说明 DDT token 更适合与文本 token 做统一自回归建模，宛如一门 LLM 可读的外语。

5.4.4 A/B Test on Image Editing

By disentangling visual attributes, DDT tokens hold an advantage over spatial tokens for tasks that require attribute modifying, such as image editing. To demonstrate this, we first use Gemma2-2B [74] as the backbone and leverage both DDT tokens and MoVQ tokens on the same 200M image-text dataset for pre-training, respectively. We then fine-tuned both models on the same image editing training set for consistent training steps. For evaluation, we manually collect a set of image editing test data and conduct an A/B test for human assessment. As shown in Figure 9(b), DDT-Gemma outperforms MoVQ-Gemma in 65 editing cases in total, showing a notable advantage in both global and local editing instructions. While MoVQ-Gemma surpasses DDT-Gemma in only 10 cases. It further underscores the superiority of DDT tokens.

5.4.4 图像编辑 A/B 测试

通过解耦视觉属性，DDT token 在需要修改属性的任务（如图像编辑）上优于空间 token。为此，我们先以 Gemma2-2B [74] 为骨干，分别在同一 2 亿图文数据上用 DDT token 与 MoVQ token 预训练，再在同一图像编辑训练集上微调相同步数。评测时人工收集一组编辑测试数据做人评 A/B。如图 9(b)，DDT-Gemma 在共 65 个编辑案例中胜出，全局与局部指令均占优；MoVQ-Gemma 仅在 10 例中胜出，进一步凸显 DDT token 优势。

5.4.5 Scaling Laws of DDT-based MLLM

We have also observed preliminary indications of scaling laws in our DDT-based MLLM. As mentioned in Sec 5.4.4, we also leverage Gemma2-2B for pre-training. And we further compare the performance of T2I generation with two model sizes (2B,8B) at three different training stages (50%, 75%, and 100% of total training tokens), as shown in Figure 14. The observed improvements in visual quality align with scaling laws, which suggest that larger transformers trained on more extensive datasets can learn more detailed and fine-grained image distributions.

5.4.5 基于 DDT 的 MLLM 缩放规律

我们还观察到基于 DDT 的 MLLM 呈现初步缩放规律迹象。如第 5.4.4 节所述，我们也用 Gemma2-2B 预训练，并比较两种模型规模（2B、8B）在三个训练阶段（总训练 token 的 50%、75%、100%）的文生图表现，见图 14。视觉质量提升与缩放规律一致：更大 Transformer、更充分数据能学到更细粒度的图像分布。

Figure 9: (a) Perplexity of MLLM in text generation during image-generation training. (b) A/B test on image editing.

图 9：(a) 图像生成训练期间 MLLM 文本生成困惑度。(b) 图像编辑 A/B 测试。

6 Conclusion

6 结论

In this work, we propose Discrete Diffusion Timestep (DDT) tokenizer to learn discrete, recursive visual tokens, which recursively compensate for the progressive attribute loss in noisy images as timesteps increase. With image decoding as DDT tokens, we train DDT-LLaMA on a vast corpus of image-text pairs for vision-language alignment. Extensive experiments showcase the immense potential of DDT-LLaMA across various tasks, e.g., T2I generation, image editing, and vision-language understanding. We are currently working on scaling up the training of our DDT tokenizer and the MLLM. In the near future, we will release a more powerful version of DDT-LLaMA, along with a more detailed technical report. Stay tuned!

本文提出离散扩散时间步（DDT）分词器，学习离散、递归的视觉 token，随时间步增大递归补偿噪声图中递进的属性损失。将图像解码为 DDT token 后，我们在大规模图文语料上训练 DDT-LLaMA 以对齐视觉与语言。大量实验展示了它在文生图、图像编辑、视觉—语言理解等任务上的巨大潜力。我们正扩大 DDT 分词器与 MLLM 的训练规模；近期将发布更强版 DDT-LLaMA 及更详细技术报告，敬请期待！

Acknowledgements

致谢

This work was supported by the NSFC (62272411), the Key R&D Projects in Zhejiang Province (No. 2024C01106, 2025C01030), the Zhejiang NSF (LRG25F020001).

本工作获国家自然科学基金（62272411）、浙江省重点研发计划（No. 2024C01106、2025C01030）以及浙江省自然科学基金（LRG25F020001）资助。

References

参考文献

Supplementary Material

补充材料

Overview

In this supplementary material, we present the following content:

概述

本补充材料包含以下内容：

[1] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8948–8957, 2019.

[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425–2433, 2015.

[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 , 2023.

[4] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , 2(3):8, 2023.

[5] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology , pages 333–342, 2010.

[6] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18392–18402, 2023.

[7] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset , 2022.

[8] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684 , 2024.

[9] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- α \alpha : Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023a.

[10] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 , 2023b.

[11] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 , 2025.

[12] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1–113, 2023.

[13] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.

[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee, 2009.

[15] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.11499 , 2023.

[16] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

[17] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

[18] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873–12883, 2021.

[19] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning , 2024.

[20] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.

[21] Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction-based image editing via multimodal large language models. arXiv preprint arXiv:2309.17102 , 2023.

[22] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

[23] Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218 , 2023.

[24] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396 , 2024.

[25] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems , 36, 2024.

[26] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904–6913, 2017.

[27] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3608–3617, 2018.

[28] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.

[29] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, 2022.

[30] Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. arXiv preprint arXiv:2307.06350 , 2023a.

[31] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models, 2023b.

[32] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6700–6709, 2019.

[33] Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Yadong Mu, et al. Unified language-vision pretraining in llm with dynamic discrete visual tokenization. In International Conference on Learning Representations , 2024.

[34] Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. Mission: Impossible language models. arXiv preprint arXiv:2401.06416 , 2024.

[35] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems , 36, 2024.

[36] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11523–11532, 2022.

[37] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 , 2023a.

[38] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13299–13308, 2024.

[39] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022.

[40] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal llms to follow zero-shot demonstrative instructions. arXiv preprint arXiv:2308.04152 , 2023b.

[41] Juncheng Li, Siliang Tang, Linchao Zhu, Wenqiao Zhang, Yi Yang, Tat-Seng Chua, Fei Wu, and Yueting Zhuang. Variational cross-graph reasoning and adaptive structured semantics learning for compositional temporal grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(10):12601–12617, 2023c.

[42] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 , 2023d.

[43] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, 2023e.

[44] Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023a.

[45] Wang Lin, Jingyuan Chen, Jiaxin Shi, Zirun Guo, Yichen Zhu, Zehan Wang, Tao Jin, Zhou Zhao, Fei Wu, YAN Shuicheng, et al. Action imitation in common action space for customized action image synthesis. In The Thirty-eighth Annual Conference on Neural Information Processing Systems .

[46] Wang Lin, Tao Jin, Wenwen Pan, Linjun Li, Xize Cheng, Ye Wang, and Zhou Zhao. Tavt: Towards transferable audio-visual text generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 14983–14999, 2023b.

[47] Wang Lin, Tao Jin, Ye Wang, Wenwen Pan, Linjun Li, Xize Cheng, and Zhou Zhao. Exploring group video captioning with efficient relational approximation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15281–15290, 2023c.

[48] Wang Lin, Jingyuan Chen, Jiaxin Shi, Yichen Zhu, Chen Liang, Junzhong Miao, Tao Jin, Zhou Zhao, Fei Wu, Shuicheng Yan, et al. Non-confusing generation of customized concepts in diffusion models. arXiv preprint arXiv:2405.06914 , 2024.

[49] Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408.02657 , 2024a.

[50] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26296–26306, 2024b.

[51] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024c.

[52] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv e-prints , pages arXiv–2402, 2024d.

[53] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437 , 2023.

[54] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 , 2022.

[55] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition , pages 3195–3204, 2019.

[56] OpenAI. Chatgpt. https://chat.openai.com , 2023.

[57] Kaihang Pan, Juncheng Li, Hongye Song, Jun Lin, Xiaozhong Liu, and Siliang Tang. Self-supervised meta-prompt learning with meta-gradient regularization for few-shot generalization. arXiv preprint arXiv:2303.12314 , 2023.

[58] Kaihang Pan, Zhaoyu Fan, Juncheng Li, Qifan Yu, Hao Fei, Siliang Tang, Richang Hong, Hanwang Zhang, and Qianru Sun. Towards unified multimodal editing with enhanced knowledge collaboration. Advances in Neural Information Processing Systems , 37:110290–110314, 2024a.

[59] Kaihang Pan, Juncheng Li, Wenjie Wang, Hao Fei, Hongye Song, Wei Ji, Jun Lin, Xiaozhong Liu, Tat-Seng Chua, and Siliang Tang. I3: I ntent-i ntrospective retrieval conditioned on i nstructions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1839–1849, 2024b.

[60] Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang. Auto-encoding morph-tokens for multimodal llm. arXiv preprint arXiv:2405.01926 , 2024c.

[61] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pages 2641–2649, 2015.

[62] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

[63] Haiyi Qiu, Minghe Gao, Long Qian, Kaihang Pan, Qifan Yu, Juncheng Li, Wenjie Wang, Siliang Tang, Yueting Zhuang, and Tat-Seng Chua. Step: Enhancing video-llms’ compositional reasoning by spatio-temporal graph-guided self-training. arXiv preprint arXiv:2412.00161 , 2024.

[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748–8763. PMLR, 2021.

[65] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

[66] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684–10695, 2022.

[67] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems , 35:36479–36494, 2022.

[68] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278–25294, 2022.

[69] Jing Shi, Ning Xu, Yihang Xu, Trung Bui, Franck Dernoncourt, and Chenliang Xu. Learning by planning: Language-guided global image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13590–13599, 2021.

[70] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

[71] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14398–14409, 2024.

[72] Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing visual relationships via language. arXiv preprint arXiv:1906.07689 , 2019.

[73] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818 , 2024.

[74] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 , 2024.

[75] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

[76] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

[77] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869 , 2024a.

[78] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Conghui He, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation, 2024b.

[79] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024c.

[80] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848 , 2024a.

[81] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.04429 , 2024b.

[82] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528 , 2024.

[83] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13040–13051, 2024.

[84] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627 , 2021.

[85] Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. Visually-prompted language model for fine-grained scene graph generation in an open world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 21560–21571, 2023.

[86] Qifan Yu, Wei Chow, Zhongqi Yue, Kaihang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unified high-quality image editing for any idea. arXiv preprint arXiv:2411.15738 , 2024a.

[87] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12944–12953, 2024b.

[88] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6023–6032, 2019.

[89] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30:495–507, 2021.

[90] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems , 36:31428–31449, 2023.

[91] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586–595, 2018.

[92] Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al. Hive: Harnessing human feedback for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9026–9036, 2024.

[93] Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-based fine-grained image editing at scale. arXiv preprint arXiv:2407.05282 , 2024.

[94] Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems , 35:23412–23425, 2022.

[95] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039 , 2024.

[96] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection, 2022.

[97] Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406.11837 , 2024a.

[98] Yun Zhu, Jianhao Guo, Fei Wu, and Siliang Tang. Rosa: A robust self-aligned framework for node-node graph contrastive learning. arXiv preprint arXiv:2204.13846 , 2022.

[99] Yun Zhu, Haizhou Shi, Xiaotang Wang, Yongchao Liu, Yaoke Wang, Boci Peng, Chuntao Hong, and Siliang Tang. Graphclip: Enhancing transferability in graph foundation models for text-attributed graphs. arXiv preprint arXiv:2410.10329 , 2024b.

[100] Xianwei Zhuang, Yuxin Xie, Yufan Deng, Liming Liang, Jinghan Ru, Yuguo Yin, and Yuexian Zou. Vargpt: Unified understanding and generation in a visual autoregressive multimodal large language model. arXiv preprint arXiv:2501.12327 , 2025.

Appendix A Implementation Details

附录 A 实现细节

A.1 Tokenizer

A.1 分词器

Encoder.

In the encoder, we set $T=480$ (i.e., the number of query tokens). The dimensions of the query tokens and the noise-free image are $R^{480\times 256}$ and $R^{1024\times 256}$, respectively. The encoder contains two independent transformers, each comprising $20$ layers with latent dimension of $256$. Following SD3 [19], despite the noise-free images and query tokens being input into separate transformers, we join the sequences of the two for the attention operation. This allows both representations to operate independently while considering the influence of the other. The encoder output retains only the transformed query tokens, serving as the image’s latent representations.

编码器

编码器中设 $T=480$（即 query token 数）。query token 与无噪声图的维度分别为 $R^{480\times 256}$ 与 $R^{1024\times 256}$。编码器含两个独立 Transformer，各 20 层、潜维 256。遵循 SD3 [19]，尽管无噪声图与 query 分别进入不同 Transformer，注意力时仍拼接两序列，使两者独立运算同时考虑彼此影响。编码器输出仅保留变换后的 query token，作为图像潜表示。

Quantizer.

The quantizer is an EMA-variant of vector quantization. Following [84], we leverage a linear projection from the encoder output to low-dimensional variable space for code index lookup (i.e., reduced from a $256$-d vector to a $16$-d vector per code). We also apply L2 normalization on the encoded latent features and codebook latent variables. Moreover, at each training step, we reset the dead entries in the codebook $\mathcal{C}$ (i.e., rarely matched with any tokens) to random tokens in the training batch.

量化器

量化器为 EMA 版向量量化。按 [84]，用线性投影把编码器输出投到低维再查码（每码由 256 维降至 16 维），并对编码潜特征与码本潜变量做 L2 归一化。另在每步把码本 $\mathcal{C}$ 中几乎不被匹配的死码重置为当前训练 batch 中的随机 token。

Decoder.

we use the same MMDiT architecture proposed in SD3 [19] for our decoder with minor modifications. Each transformer in the MMDiT comprises $24$ layers with latent dimension of $1536$. The sequence of quantized tokens replaces the text tokens as input, with a linear layer to project the $16$-dimensional quantized vector to the latent dimension (i.e., $1536$) of the MMDiT. Additionally, we also removed the pooled token embedding introduced in SD3.

解码器

解码器采用 SD3 [19] 提出的 MMDiT 架构并做少量修改。MMDiT 中每个 Transformer 为 24 层、潜维 1536。量化 token 序列替代文本 token 作为输入，用线性层把 16 维量化向量投到 MMDiT 潜维（1536），并去掉 SD3 中的 pooled token 嵌入。

Furthermore, the hyper-parameters of training the tokenizer are detailed in Table 5.

分词器训练超参详见表 5。

Hyper-parameters MLLM-pt MLLM-ft Tokenizer LLM init LLama3-8B MLLM-pt - Optimizer AdamW AdamW AdamW Optimizer param. $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1\mathrm{e}{-6}$ $\beta_{1}=0.9,\beta_{2}=0.99,\epsilon=1\mathrm{e}{-6}$ Peak LR 1e-4 1e-5 1e-4 LR scheduler Cosine Cosine Linear+Cosine Batch size 1280 256/128 1024 Training Steps 360K 160K 140K Warmup Steps 5K 2K 5K Weight decay 0.05 0.05 0.0 Gradient clipping 1.0 1.0 - Numerical precision bfloat16 bfloat16 bfloat16 Resource Usage 512 Ascend 910B 256 Ascend 910B 32 NVIDIA A800 Framework Megatron(TP=8) Megatron(TP=8) DDP

Table 5: The detailed training hyper-parameters. “MLLM-pt” denotes the pretraining of DDT-LLaMA, “MLLM-ft” denotes the instruction tuning of DDT-LLaMA, while “Tokenizer” denotes the training of DDT.

Hyper-parameters MLLM-pt MLLM-ft Tokenizer LLM init LLama3-8B MLLM-pt - Optimizer AdamW AdamW AdamW Optimizer param. $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1\mathrm{e}{-6}$ $\beta_{1}=0.9,\beta_{2}=0.99,\epsilon=1\mathrm{e}{-6}$ Peak LR 1e-4 1e-5 1e-4 LR scheduler Cosine Cosine Linear+Cosine Batch size 1280 256/128 1024 Training Steps 360K 160K 140K Warmup Steps 5K 2K 5K Weight decay 0.05 0.05 0.0 Gradient clipping 1.0 1.0 - Numerical precision bfloat16 bfloat16 bfloat16 Resource Usage 512 Ascend 910B 256 Ascend 910B 32 NVIDIA A800 Framework Megatron(TP=8) Megatron(TP=8) DDP

表 5：训练超参细则。「MLLM-pt」为 DDT-LLaMA 预训练，「MLLM-ft」为指令微调，「Tokenizer」为 DDT 分词器训练。

Figure 10: More qualitative results of DDT-LLaMA text-to-image generation. (the supplement to Figure 3 in the main paper)

图 10：DDT-LLaMA 文生图更多定性结果。（正文图 3 的补充）

Figure 11: More qualitative comparison with EMU3 on T2I generation (PART-1). DDT-LLaMA can better respond to prompts related to counting, color, and position. (the supplement to Figure 4 in the main paper)

图 11：与 EMU3 文生图更多定性对比（第 1 部分）。DDT-LLaMA 对计数、颜色、位置类提示响应更好。（正文图 4 的补充）

Figure 12: More qualitative comparison with EMU3 on T2I generation (PART-2). DDT-LLaMA can better respond to prompts related to counting, color, and position. (the supplement to Figure 4 in the main paper)

图 12：与 EMU3 文生图更多定性对比（第 2 部分）。DDT-LLaMA 对计数、颜色、位置类提示响应更好。（正文图 4 的补充）

Figure 13: Qualitative comparison of MOVQ-Gemma(2B), DDT-Gemma(2B), and DDT-LLaMA(8B) in the image editing task. In most instances, DDT-Gemma outperforms MOVQ-Gemma. Furthermore, DDT-LLaMA not only effectively comprehends and executes editing instructions accurately but also excels in preserving image fidelity.

图 13：MOVQ-Gemma(2B)、DDT-Gemma(2B) 与 DDT-LLaMA(8B) 在图像编辑任务上的定性对比。多数情形下 DDT-Gemma 优于 MOVQ-Gemma；DDT-LLaMA 不仅准确理解并执行编辑指令，还擅长保持图像保真。

Figure 14: (More examples of text-to-image generation with different MLLM size (2B, 8B) and training compute (50%, 75%, 100% of total tokens). the supplement to Figure 7 in Section 5.4.5 of the main paper)

图 14：不同 MLLM 规模（2B、8B）与训练算力（总 token 的 50%、75%、100%）下文生图更多样例。（正文第 5.4.5 节图 7 的补充）

Figure 15: Qualitative results of DDT-LLaMA visual comprehension.

图 15：DDT-LLaMA 视觉理解定性结果。

Figure 16: More results of counterfactual interpolation with DDT tokens and VQGAN tokens. (the supplement to Figure 6 in Section 5.4.2 of the main paper)

图 16：DDT token 与 VQGAN token 反事实插值更多结果。（正文第 5.4.2 节图 6 的补充）

Figure 17: More results of decoding images with an expanding subset of autoregressive-sampled DDT tokens (from 1 to $T=480$). $y_{k}$ denotes the number of sampled DDT tokens that are fed into the decoder for image generation, $1=y_{0}<y_{1}<y_{2}<...<y_{t-1}<y_{t}=480$. (the supplement to Figure 8 in Section 5.4.2 of the main paper)

图 17：用自回归采样得到的扩展 DDT token 子集（从 1 到 $T=480$）解码图像的更多结果。$y_{k}$ 表示送入解码器生成图像的采样 DDT token 数，$1=y_{0}<y_{1}<y_{2}<...<y_{t-1}<y_{t}=480$。（正文第 5.4.2 节图 8 的补充）

A.2 MLLM

We initialize our MLLM from a pretrained LLM, specifically using the Llama3-8b model [17], which has only undergone pretraining without instruction tuning. Additionally, we expand its vocabulary by adding $|\mathcal{C}|=65,536$ visual codes and two extra special tokens ([BOV] and [EOV]). Since both image and text are represented as discrete token IDs, we can use the cross-entropy to supervise the token prediction at each position for both modalities with a shared prediction head. A shared head has proven more effective than using separate heads for each modality in training, enhancing the upper capabilities of the MLLM. During inference, when generating content for a specific modality, tokens that do not fall in that modality’s space should be masked. For example, when generating text, we mask the logits of the $65,536$ visual codes before sampling the text tokens; Similarly, when the [BOV] token indicates the beginning of image output, we mask the logits for $128,256$ text words before sampling the visual tokens. Moreover, we set $topk=50,topp=1.0$ for text token sampling and $topk=4096,topp=0.9$ for visual token sampling. Besides, for text-to-image generation, during inference we use classifier-free guidance on the logits for autoregressive sampling in a manner similar to [52, 77]. We set the guidance scale to $8.0$.

MLLM 以仅完成预训练、未经指令微调 的 Llama3-8b [17] 初始化，并扩展词表加入 $|\mathcal{C}|=65{,}536$ 个视觉码以及 [BOV]、[EOV] 两个特殊 token。图像与文本均为离散 token ID，可用共享预测头以交叉熵在各位置监督两种模态；共享头比分模态头更有效，能抬高 MLLM 能力上限。推理时生成某模态内容需 mask 非该模态 logit：生成文本时先 mask 65,536 个视觉码再采样；出现 [BOV] 表示开始输出图像时，先 mask 128,256 个文本词再采样视觉 token。文本采样设 $topk=50,topp=1.0$，视觉采样设 $topk=4096,topp=0.9$。文生图推理时，按类似 [52, 77] 的方式对自回归采样 logit 做无分类器引导，引导尺度设为 $8.0$。

During both pretraining and instruction-tuning, all parameters of the MLLM are fully fine-tuned. The hyper-parameters of both training stages for DDT-LLaMA are shown in Table 5. The overall training is stable; we observed only one minor spike in the loss curve during pretraining. Following [12], we resume training from a checkpoint approximately 500 steps before the onset of the spike.

预训练与指令微调阶段均对 MLLM 全参数微调。两阶段超参见表 5。整体训练稳定；预训练损失曲线仅出现一次轻微尖峰，按 [12] 从尖峰出现前约 500 步的检查点恢复训练。

Appendix B Training Data & Evaluation Details

附录 B 训练数据与评测细节

B.1 Training Data

B.1 训练数据

Tokenizer Training.

Our diffusion timestep tokenizer is trained on the training split of ImageNet [14], which comprises about 1.28 million images. Besides, each training image is center-cropped to a size of $256\times 256$. The training was conducted on 32 NVIDIA A800 GPUs and lasted for nearly one weak.

分词器训练

扩散时间步分词器在 ImageNet [14] 训练划分上训练，约 128 万张图；每张训练图中心裁剪为 $256\times 256$。训练在 32 块 NVIDIA A800 上进行，历时约一周。

MLLM Pretraining.

Our pretraining dataset, sourced from Laion [68], consists of 200 million text-to-image pairs. Each pair includes an image accompanied by a brief, coarse-grained native caption and a detailed, fine-grained generated caption. For the generation of detailed captions, we employ ShareGPT-4V [10] to annotate over 400 million images sourced from Laion [68] and Coyo [coyo]. Considering the propensity of ShareGPT-4V to generate captions that may not align with the images due to hallucinations, we utilize CLIP scores [64] to filter all generated image-text pairs, retaining only those with the highest CLIP scores, totaling 200 million images. For pretraining, we structure each pair in the format: “[BOS] [BOV] [EOV] [EOS]” for pretraining, where [BOS] and [EOS] are the original special tokens from the text tokenizer, [BOV] and [EOV] marking the start and the end of the vision input. Each image has a 60% chance of being paired with the long caption and a 40% probability of being paired with the short caption during training. Besides, each sample’s caption has a 10% probability of being dropped out. Furthermore, to preserve the textual capabilities of MLLM, we supplement our dataset with purely textual data from Wikipedia and Pile [22], at a ratio of 10%. The pre-training was conducted on 512 Ascend 910B NPUs and lasted for nearly two weaks.

MLLM 预训练

预训练数据来自 Laion [68]，共 2 亿图文对。每对含简短粗粒度原生标题与细粒度生成标题。细粒度标题用 ShareGPT-4V [10] 标注来自 Laion [68] 与 Coyo 的逾 4 亿图像；因 ShareGPT-4V 易幻觉，用 CLIP 分数 [64] 过滤，只保留分数最高的 2 亿对。预训练格式为「[BOS] [BOV] [EOV] [EOS]」，[BOS]/[EOS] 为文本分词器原有特殊符，[BOV]/[EOV] 标记视觉输入起止。训练时每图以 60% 概率配长标题、40% 配短标题；标题有 10% 概率被丢弃。为保留文本能力，另以 10% 比例混入 Wikipedia 与 Pile [22] 纯文本。预训练在 512 块 Ascend 910B NPU 上进行，历时约两周。

MLLM Instruction Tuning.

During instruction tuning, we incorporate a variety of tasks, outlined as follows: (1) Text-to-Image Generation: We employ datasets including ShareGPT4V caption [10], ALLaVA [8] and GRIT [8], utilizing a prompt template formatted as:“[BOS] USER: Please Generate an image. ASSISTANT: [BOV] [EOV] [EOS]”.

MLLM 指令微调

指令微调纳入多种任务，如下：(1) 文生图：使用 ShareGPT4V caption [10]、ALLaVA [8] 与 GRIT [8] 等，提示模板为「[BOS] USER: Please Generate an image. ASSISTANT: [BOV] [EOV] [EOS]」。

(2) Image editing: We employ datasets such as InstructPix2Pix [6] and Hive [92], utilizing a prompt template formatted as:“[BOS] USER: [BOV] [EOV] Please Generate an image. ASSISTANT: [BOV] [EOV] [EOS]”. Besides, considering the generally mediocre quality of existing image editing datasets, we also construct a batch of higher quality image editing data for integration into the training.

(2) 图像编辑：使用 InstructPix2Pix [6]、Hive [92] 等，模板为「[BOS] USER: [BOV] [EOV] Please Generate an image. ASSISTANT: [BOV] [EOV] [EOS]」。鉴于现有编辑数据质量整体一般，我们还自建一批更高质量编辑数据并入训练。

(3) Image caption & VQA: We mainly leverage ShareGPT4V(-instruct) [10], ALLaVA(-instruct) [8] and select part of the held-in training datasets in InstructBlip [13] for instruction-tuning. We follow the [13] to design the instruction templates.

(3) 图像描述与 VQA：主要使用 ShareGPT4V(-instruct) [10]、ALLaVA(-instruct) [8]，并选用 InstructBlip [13] 中部分 held-in 训练数据；指令模板设计遵循 [13]。

B.2 Evaluation Details

B.2 评测细节

Baseline Methods.

For text-to-image generation tasks, we compare DDT-LLaMA with both diffusion-based T2I specialists and MLLM-based generalists. The diffusion-based T2I specialists include DALL-E 2 [65], SDv1.5 [66], SDv2.1 [66], SDXL [62], PixArt-alpha [9], DALL-E 3 [4], and SD3 [19]. For the MLLM-based generalists, we include comparisons with SEED-LLaMA [23], LaVIT [33], Emu2-Gen [71], SEED-X [24], VILA-U [81], Lumina-mGPT [49], and Emu3 [77]. (For EMU3, we report its results without prompt rewriting for fair comparison.)

基线方法

文生图任务上，DDT-LLaMA 与基于扩散的文生图专用模型及基于 MLLM 的通才对比。扩散专用模型包括 DALL-E 2 [65]、SDv1.5 [66]、SDv2.1 [66]、SDXL [62]、PixArt-alpha [9]、DALL-E 3 [4]、SD3 [19]；MLLM 通才包括 SEED-LLaMA [23]、LaVIT [33]、Emu2-Gen [71]、SEED-X [24]、VILA-U [81]、Lumina-mGPT [49]、Emu3 [77]。（EMU3 报告未做提示改写的结果以便公平对比。）

For image editing tasks, we compare DDT-LLaMA with both specialized image editing models and generalist MLLM-based models. The image editing specialists we evaluate include InsPix2Pix [6], MGIE [21], and UltraEdit [93]. Among the MLLM-based generalist models, we compare against GILL [35], Emu2-Gen [71], SEED-LLLaMA [23], LaVIT [33], and SEED-X-Edit [24]. (We exclude MLLMs such as EMU3 because they lack image editing capabilities.)

图像编辑任务上，对比专用编辑模型与通才 MLLM。专用模型含 InsPix2Pix [6]、MGIE [21]、UltraEdit [93]；通才含 GILL [35]、Emu2-Gen [71]、SEED-LLaMA [23]、LaVIT [33]、SEED-X-Edit [24]。（排除 EMU3 等不具备图像编辑能力的 MLLM。）

For visual comprehension and generation tasks, we compare DDT-LLaMA with specialized visual comprehension MLLMs, and MLLMs capable of both visual comprehension and generation. The specialized visual comprehension models include InstructBlip[13], QWenVL-Chat [3], LLaVA-1.5 [50], mPLUG-Owl2 [83], ShareGPT4V [10], LLaVA-1.6(HD) [51], and VILA [44]. For models supporting both visual comprehension and generation, we compare DDT-LLaMA against Emu2-Chat [71], SEED-LLLaMA [23], VILA-U [81], LaVIT [33], and Emu3 [77].

视觉理解与生成任务上，对比专用视觉理解 MLLM 以及同时支持理解与生成的 MLLM。专用理解模型含 InstructBlip [13]、QWenVL-Chat [3]、LLaVA-1.5 [50]、mPLUG-Owl2 [83]、ShareGPT4V [10]、LLaVA-1.6(HD) [51]、VILA [44]；双能力模型对比 Emu2-Chat [71]、SEED-LLaMA [23]、VILA-U [81]、LaVIT [33]、Emu3 [77]。

Evaluation Dataset.

For text-to-image generation tasks, we conduct zero-shot evaluation on 3 benchmarks: GenEval [25], T2I-CompBench [30], and DrawBench [67]. GenEval contains 6 different subtasks of varying difficulty requiring various compositional skills, including single object (SingObj), single object (TwoObj), counting, colors, position, color binding (ColorAttri). And we adopt the metric proposed by  [25] for evaluation. Each subtask is scored independently, and the overall score is calculated as the average of all six subtask scores. The T2I-CompBench suite encompasses six subtasks: color, shape, texture, spatial, non-spatial, and complex (complex compositions). Building on prior research, we employ the Blip-VQA score [39] to assess the color, shape, and texture subtasks. For spatial evaluation, we use the UniDet score [96]; for non-spatial evaluations, the CLIP score [64, 29]; and for complex compositions, the 3-in-1 Metric [30]. In terms of DrawBench, we leverage Clip text-visual feature similarity [64] as the evaluation metric.

评测数据集

文生图零样本评测在三个基准上：GenEval [25]、T2I-CompBench [30]、DrawBench [67]。GenEval 含六个难度不等、需不同组合能力的子任务：单物体（SingObj）、双物体（TwoObj）、计数、颜色、位置、颜色绑定（ColorAttri）；采用 [25] 指标，各子任务独立打分，总分为六项平均。T2I-CompBench 含颜色、形状、纹理、空间、非空间、复杂组合六子任务；延续先前工作，颜色/形状/纹理用 Blip-VQA 分 [39]，空间用 UniDet 分 [96]，非空间用 CLIP 分 [64, 29]，复杂组合用 3-in-1 指标 [30]。DrawBench 以 CLIP 文本—视觉特征相似度 [64] 为指标。

We also conduct zero-shot instruction-based image editing across three datasets: EVR [72], MA5k [69], and MagicBrush [90]. Following [21], for EVR and MagicBrush, we treat the standard pixel difference (L1) and visual feature similarity from the CLIP visual encoder (CVS) between generated images and ground-truth goals as the evaluation metrics. For MA5K, we utilize L1 and LPIPS [91] as the evaluation metrics.

还在 EVR [72]、MA5k [69]、MagicBrush [90] 上做零样本指令图像编辑。依 [21]，EVR 与 MagicBrush 以生成图与真值目标的标准像素差（L1）以及 CLIP 视觉编码器特征相似度（CVS）为指标；MA5K 用 L1 与 LPIPS [91]。

For visual comprehension tasks, we conduct zero-shot evaluation on a wide range of academic benchmarks, including image caption (NoCaps [1], Flickr30K [61]), VQA (VQAv2 [26], GQA [32], OKVQA [55], VizWiz [5]), MLLM-oriented Comprehension Benchmarks (MME [20], SEEDBench [37], POPE [43]). (Note: In Table3 of the main paper, we use “VQA” to denote VQAv2). We employ CIDEr as the metric for image caption tasks, and VQA accuracy for VQA tasks. We employ the CIDEr metric to evaluate performance on image captioning tasks, while using VQA accuracy for the VQA datasets. Moreover, each MLLM-oriented benchmark is evaluated according to its specific prescribed methodologies, where we report the perception score for MME, MCQ accuracy for SEEDBench, and the F1 score for POPE.

视觉理解零样本评测覆盖广泛学术基准：图像描述（NoCaps [1]、Flickr30K [61]）、VQA（VQAv2 [26]、GQA [32]、OKVQA [55]、VizWiz [5]）、面向 MLLM 的理解基准（MME [20]、SEEDBench [37]、POPE [43]）。（注：正文表 3 中「VQA」指 VQAv2。）图像描述用 CIDEr，VQA 用准确率；各 MLLM 向基准按其规定方法评测，MME 报感知分、SEEDBench 报选择题准确率、POPE 报 F1。

Appendix C Additional Examples of T2I Generation

附录 C 文生图补充样例

In Figure 10, we present more qualitative examples of DDT-LLaMA on text-to-image generation tasks. DDT-LLaMA adeptly handles various types of instructions, including complex ones such as generating surreal images (e.g., “A panda drinking coffee”, “ sloth with pink hat”) and multi-condition combined prompts (e.g., “An emoji of a baby panda wearing a red hat, blue gloves, green shirt, and blue pants”), to generate semantically-consistent images.

在图 10中给出更多 DDT-LLaMA 文生图定性样例。它能处理多种指令，包括生成超现实图像（如「喝咖啡的熊猫」「戴粉帽的树懒」）以及多条件组合提示（如「戴红帽、蓝手套、绿衫、蓝裤的宝宝熊猫表情」），产出语义一致的图像。

Furthermore, in Figure 11 and Figure 12, we present a direct comparison between DDT-LLaMA and Emu3 across 54 prompts involving counting, color, and positioning. It is evident that Emu3 falls short in these areas: (1) For counting-related prompts, EMU3 often generates images with an incorrect number of objects. (2) For prompts related to positioning, Emu3 frequently misplaces objects, and sometimes only one of the objects is generated. (3) For color-related prompts, EMU3 often incorrectly assigns colors to the objects, and it may also generate images where the arrangement or presence of objects is disordered (“A photo of a purple suitcase and an orange pizza” in Figure 12). In contrast, DDT-LLaMA generates images that more accurately reflect the desired object attributes (number and color) and adhere to the spatial specifications outlined in the prompts.

此外在图 11与图 12中，直接对比 DDT-LLaMA 与 Emu3 在 54 条涉及计数、颜色、位置的提示上的表现。可见 Emu3 在这些方面不足：(1) 计数类提示常生成错误数量的物体；(2) 位置类提示常放错位置，有时只生成其中一个物体；(3) 颜色类提示常给物体上错色，也可能物体排布或出现混乱（如图 12「紫色行李箱与橙色披萨」）。相较之下，DDT-LLaMA 生成的图像更准确反映期望的物体属性（数量与颜色），并遵循提示中的空间规定。

Appendix D Additional Comparison on Image Editing

附录 D 图像编辑补充对比

As discussed in Section 5.4.4, we also employ Gemma2-2b [74] as the initial LLM and leverage both DDT tokens and MoVQ tokens for pretraining and instruction tuning, which we refer to as DDT-Gemma and MOVQ-Gemma, respectively. In Figure 13, we showcase a series of qualitative examples that compare MOVQ-Gemma (2B), DDT-Gemma (2B), and DDT-LLaMA (8B). First, when comparing MOVQ-Gemma and DDT-Gemma, it is evident that in many editing cases,MOVQ-Gemma often gives up editing, typically returning the original image as the output. In contrast, DDT-Gemma exhibits a more robust comprehension of the editing instructions and delivers superior results in A/B tests, which indicates that our recursive DDT tokens outperform spatial tokens in image editing tasks.

如第 5.4.4 节所述，我们还以 Gemma2-2b [74] 为初始 LLM，分别用 DDT token 与 MoVQ token 做预训练与指令微调，记为 DDT-Gemma 与 MOVQ-Gemma。图 13展示 MOVQ-Gemma(2B)、DDT-Gemma(2B) 与 DDT-LLaMA(8B) 的一系列定性对比。先看 MOVQ-Gemma 与 DDT-Gemma：许多编辑案例中 MOVQ-Gemma 常常放弃编辑，直接返回原图；DDT-Gemma 对编辑指令理解更稳，A/B 测试结果更好，表明递归 DDT token 在图像编辑上优于空间 token。

Furthermore, DDT-Gemma sometimes faces problems with incomplete modifications or fails to maintain image fidelity in areas not targeted by the edits. For example, in the 7th case of Figure 13 (“remove the fog”), only the red exhaust behind the car is eliminated by DDT-Gemma. In the 12th case (“change the yellow roses to red roses”), the shape of the roses is also inadvertently changed by DDT-Gemma. In contrast,scaling up the backbone model from 2B to 8B significantly improves the editing performance. We can see that DDT-LLaMA not only effectively comprehends the instructions to accurately execute the editing, but also excels at preserving image fidelity. This serves as evidence of the scaling-law properties of DDT tokens.

此外，DDT-Gemma 有时修改不完整，或在非编辑区域保真不足。例如图 13 第 7 例（「去掉雾」）只去掉了车后红色尾气；第 12 例（「黄玫瑰改红玫瑰」）玫瑰形状也被意外改变。相较之下，把骨干从 2B 扩到 8B 显著提升编辑表现：DDT-LLaMA 既能准确执行编辑，又擅长保持图像保真，可作为 DDT token 具备缩放规律的证据。

Appendix E Examples of Visual Comprehension

附录 E 视觉理解样例

In Figure 15, we show some qualitative examples of visual comprehension. Although lack of pretrained encoders like CLIP [64], DDT-LLaMA can still effectively understand the visual semantics in the images and accurately infer the answer based on the given textual instruction.

在图 15给出部分视觉理解定性样例。虽无 CLIP [64] 等预训练编码器，DDT-LLaMA 仍能有效理解图像视觉语义，并按文本指令准确推断答案。

Appendix F Additional Examples of In-depth Analysis

附录 F 深入分析补充样例

F.1 Counterfactual Interpolation

F.1 反事实插值

Figure 16 shows more results of counterfactual interpolation of VQGAN tokens [18] and DDT tokens. DDT tokens employ a disentangled representation to ensure that only the attributes represented by the substituted tokens vary in the generated counterfactuals, which allows for a seamless semantic integration of the two images.

图 16给出 VQGAN token [18] 与 DDT token 反事实插值的更多结果。DDT token 以解耦表示保证生成反事实中仅被替换 token 所代表的属性变化，从而实现两图的平滑语义融合。

F.2 Decoding with a subset of DDT tokens

F.2 用 DDT token 子集解码

In Figure 17, we show more results demonstrating how autoregressive-sampled DDT tokens can be decoded into images in order. As the number of sampled tokens increases (from 1 to $T=480$), the image attributes are progressively reconstructed – from fine details to the completion of coarse-grained contours and color information. This confirms thatour DDT token sequence successfully decouples image attributes and possesses recursive properties.

在图 17给出更多结果，展示自回归采样的 DDT token 如何按序解码成图。随采样 token 数增加（从 1 到 $T=480$），图像属性逐步重建——从细节到粗粒度轮廓与颜色信息补全。这确认了DDT token 序列成功解耦图像属性并具备递归性质。

F.3 Scaling Laws of DDT-based MLLM

F.3 基于 DDT 的 MLLM 缩放规律

In Figure 14, we show more examples of text-to-image generation examples using two model sizes (Gemma 2B, LLama 8B [17]) at three different training stages (50%, 75%, and 100% of total training tokens). The enhancements in visual quality observed correspond with scaling laws, which indicate that larger transformers trained on more comprehensive datasets tend to yield superior text-to-image performance.

在图 14给出两种模型规模（Gemma 2B、LLaMA 8B [17]）在三个训练阶段（总训练 token 的 50%、75%、100%）的更多文生图样例。观察到的视觉质量提升与缩放规律一致：更大 Transformer、在更充分数据上训练，往往带来更优文生图表现。

Appendix G Limitation and Future Work

附录 G 局限与未来工作

Our current tokenizer is trained solely on the ImageNet dataset [14] at a resolution of 256x256 pixels, and it faces limitations in reconstructing open-domain images compared to baseline methods. For example, the EMU3 tokenizer MOVQ [94], which is trained on a significantly larger dataset (from Laion and InternVID [78]), achieves superior reconstruction performance than ours. As our 200M pretraining dataset filtered from an open-domain image dataset (i.e., Laion), the inadequate reconstructive capability of DDT restricts DDT-LLaMA’s ability in text-to-image generation. This particularly impacts the aesthetic quality of the images generated by DDT-LLaMA, as illustrated in Figure 10, Figure 11, and Figure 12.

当前分词器仅在 256×256 的 ImageNet [14] 上训练，相较基线，开放域重建能力有限。例如 EMU3 的 MOVQ 分词器 [94] 在显著更大数据（来自 Laion 与 InternVID [78]）上训练，重建优于我们。由于我们从开放域（Laion）过滤得到的 2 亿预训练数据受 DDT 重建能力制约，DDT-LLaMA 的文生图能力——尤其美学质量——受限，见图 10、图 11、图 12。

We are currently working on improving and scaling up the training of our DDT-tokenizer and the MLLM on a significantly larger dataset (about 500M images). In the near future, we will release a more powerful version of DDT-LLaMA, along with a detailed technical report. Stay tuned! Building on this foundation, we aim to further demonstrate that DDT-LLaMA is a significant approach for addressing visual-language tasks [87, 85, 86, 58, 40] and collaborative NLP tasks [28, 98, 99, 59, 57, 79]. We also seek to extend the capabilities of DDT-LLaMA to support more vision-language tasks [46, 47, 48, 45, 31, 63, 41] such as video comprehension and video generation.

我们正改进并扩大 DDT 分词器与 MLLM 在更大数据集（约 5 亿图）上的训练。*近期将发布更强版 DDT-LLaMA 及详细技术报告，敬请期待！*在此基础上，我们希望进一步表明 DDT-LLaMA 是应对视觉—语言任务 [87, 85, 86, 58, 40] 与协作式 NLP 任务 [28, 98, 99, 59, 57, 79] 的重要路径，并扩展其能力以支持更多视觉—语言任务 [46, 47, 48, 45, 31, 63, 41]，如视频理解与视频生成。