ViLL-E: Video LLM Embeddings for Retrieval
ViLL-E: Video LLM Embeddings for Retrieval
ViLL-E:用于检索的视频大语言模型嵌入
Rohit Gupta and Jayakrishnan Unnikrishnan and Fan Fei and Sheng Liu
{rohitgpt, jayunn, feiff, shenlu}@amazon.com
Son Tran
sontran@amazon.com
Mubarak Shah
shah@crcv.ucf.edu
Abstract
摘要
Video Large Language Models (VideoLLMs) excel at video understanding tasks where outputs are textual, such as Video Question Answering and Video Captioning. However, they underperform specialized embedding-based models in Retrieval tasks, such as Text-to-Video Retrieval and Moment Retrieval. We introduce ViLL-E (Video-LLM-Embed), a unified VideoLLM architecture endowed with a novel embedding generation mechanism that allows the model to “think longer” for complex videos and stop early for easy ones. We train this model with a three-stage training methodology combining generative and contrastive learning: initial large-scale pre-training with video-caption pairs; followed by continual training on a smaller, detailed-caption dataset; and concluding with task-specific fine-tuning on a novel multi-task dataset covering Video QA, Temporal Localization, Video Retrieval, and Video-Text Matching. Our model significantly improves temporal localization (on avg. % over other VideoLLMs) and video retrieval (up to 4% over dual encoder models), achieving performance comparable to state-of-the-art specialized embedding models while remaining competitive on VideoQA tasks. Furthermore, our joint contrastive-generative training unlocks new zero-shot capabilities, significantly outperforming state-of-the-art methods in composed video retrieval (+5% over SotA) and retrieval from long text (+2% over SotA).
视频大语言模型(VideoLLM)在输出为文本的视频理解任务上表现突出,例如视频问答和视频字幕。但在检索类任务上,它们往往落后于专门的嵌入模型,例如文本到视频检索和时刻检索。我们提出 ViLL-E(Video-LLM-Embed),这是一套统一的 VideoLLM 架构,并配备了一种新的嵌入生成机制,使模型能对复杂视频“多想一会儿”,对简单视频则提前结束。我们用结合生成学习与对比学习的三阶段方法训练该模型:先在大规模视频-字幕对上预训练;再在一个更小、字幕更细的数据集上继续训练;最后在一个新的多任务数据集上做任务微调,覆盖视频问答、时序定位、视频检索和视频-文本匹配。该模型显著提升了时序定位(平均比其他 VideoLLM 高 7%)和视频检索(最高比双编码器模型高 4%),达到与最先进专用嵌入模型相当的水平,同时在视频问答上仍具竞争力。此外,联合的对比-生成训练还解锁了新的零样本能力,在组合视频检索上显著超过现有最优方法(高出 5 个百分点),在长文本检索上也高出 2 个百分点。
1 Introduction
1 引言
Figure 1: Left & Bottom Right: VideoLLMs lag expert models on some retrieval-based tasks, e.g. Temporal Localization (in green [17, 10, 15]), and are incapable of others, e.g. Text-to-Video Retrieval (in red [49, 1, 44]). A key difference between existing VideoLLMs and state of the art expert models in these tasks is the use of embeddings (Top Right). Our approach, ViLL-E (VideoLLM-Embed, pronounced willy) equips VideoLLMs with embedding generation. ViLL-E matches SotA on retrieval tasks while remaining competitive on generation tasks such as VideoQA (in orange [47, 22]). ◆ indicates limited capabilities.
图 1:左图与右下:VideoLLM 在部分检索类任务上落后于专用模型,例如时序定位(绿色 [17, 10, 15]),并且无法完成另一些任务,例如文本到视频检索(红色 [49, 1, 44])。现有 VideoLLM 与这些任务上的先进专用模型之间,一个关键差别是是否使用嵌入(右上)。我们的方法 ViLL-E(VideoLLM-Embed,读作 willy)为 VideoLLM 补上了嵌入生成能力。ViLL-E 在检索任务上与最先进方法持平,同时在视频问答等生成任务上仍具竞争力(橙色 [47, 22])。◆ 表示能力有限。
Large Language Models (LLMs) have demonstrated significant capabilities in a “single model multi-task approach” setting, effectively solving various natural language understanding tasks. Inspired by the success of LLMs [7, 41, 13, 34], numerous studies have integrated additional modalities like images [19, 26, 3] and audio [38, 56] into these models. These Multimodal LLMs models excel at most vision tasks that can be formulated as text generation problems. Of specific interest are Video Large Language Models (VideoLLMs) [29, 24, 57, 21], which incorporate videos into LLMs by encoding video frames and aligning visual features with LLM feature spaces via projection layers, enabling video understanding through tasks like captioning and QA.
大语言模型(LLM)已在“单模型多任务”设定中展现出很强的能力,能够有效完成各类自然语言理解任务。受 LLM 成功的启发 [7, 41, 13, 34],许多研究把图像 [19, 26, 3] 和音频 [38, 56] 等更多模态接入这些模型。这类多模态大语言模型,在大多数可以写成文本生成问题的视觉任务上表现很好。其中尤其值得关注的是视频大语言模型(VideoLLM)[29, 24, 57, 21]:它们对视频帧编码,再经投影层把视觉特征对齐到 LLM 的特征空间,从而通过字幕、问答等任务完成视频理解。
However, this generative approach has limitations. Many video tasks, like Text-to-Video Retrieval (T2V), require embedding-based matching rather than text generation. Such tasks typically use specialized models with aligned video-text embeddings, fine-tuned per dataset. Currently these specialized methods lead performance on standard benchmarks. Even VideoLLM variants that are specialized to attempt temporal reasoning (e.g. LLaVA-ST [18]) trail expert detectors such as QD-DETR [30] by double-digit margins on ActivityNet [15] and Charades-STA [10]. Figure 1 underscores the point: every top performer in video retrieval, temporal localisation or cross-modal search (VidLA [37], QD-DETR [30], SigLIP [53]) achieves its performance with embeddings, while Video-LLMs dominate the Video-QA column. Also, recent NLP research has demonstrated that LLMs can be transformed into strong retrieval models with contrastive finetuning on surprisingly little data (see, e.g., GRIT [31] or E5 [43]). This raises the question: Instead of maintaining two separate stacks, can a single model handle both generative tasks and extract discriminative video/text embeddings, addressing the retrieval and localization gaps?
不过,这种生成式路线有其局限。许多视频任务,例如文本到视频检索(T2V),需要的是基于嵌入的匹配,而不是生成文本。这类任务通常使用带有对齐视频-文本嵌入的专用模型,并按数据集分别微调。目前,这些专用方法在标准基准上仍领先。即便是专门做时序推理的 VideoLLM 变体(如 LLaVA-ST [18]),在 ActivityNet [15] 和 Charades-STA [10] 上,也比 QD-DETR [30] 这类专用检测器落后两位数。图 1 把这一点说得很清楚:视频检索、时序定位或跨模态搜索中的领先方法(VidLA [37]、QD-DETR [30]、SigLIP [53])都靠嵌入取得成绩,而 VideoLLM 则主导视频问答这一列。与此同时,近期自然语言处理研究表明,只需相对不多的对比微调,就能把 LLM 变成很强的检索模型(例如 GRIT [31] 或 E5 [43])。这就提出一个问题:与其维护两套独立系统,能不能用同一个模型既做生成任务,又抽出可判别的视频/文本嵌入,从而补上检索和定位上的差距?
Motivated by these observations, we propose an approach that combines generative and embedding tasks during Video-LLM training, aiming for strong performance on generative tasks like VideoQA, while also enabling embedding generation for tasks such as video retrieval—previously unsupported by VideoLLMs. Our approach builds on PaliGemma [6], a standard multi-modal LLM and adds a learnable attentive-pooling “embedding head” after the LLM decoder. The embedding head is designed to adapt to video complexity during inference. This allows the model to “think longer” for hard videos but return quickly for easy ones. We propose three-stage training, which begins with large-scale pre-training on video-caption pairs using joint supervision: next-word prediction for generation and a multi-modal contrastive loss for embedding. In the second stage, we continue pre-training on a smaller dataset of videos with high-quality generated captions. The final stage involves instruction finetuning on a multi-task dataset covering VideoQA, captioning, localization, and retrieval. Our proposed model, ViLL-E (pronounced willy), is competitive with VideoLLMs on VideoQA and outperforms them on localization. Our embeddings narrow the gap with non-LLM localization expert methods and enable SoTA performance on video retrieval.
基于这些观察,我们提出在 VideoLLM 训练中同时纳入生成任务和嵌入任务,目标是在视频问答等生成任务上保持强劲表现,同时使模型能够生成嵌入,以支持视频检索这类以往 VideoLLM 做不了的任务。方法建立在标准多模态大语言模型 PaliGemma [6] 之上,并在 LLM 解码器之后增加一个可学习的注意力池化“嵌入头”。嵌入头在推理时会按视频复杂度自适应调整,从而对难视频“多想一会儿”,对简单视频则尽快返回。我们采用三阶段训练:第一阶段在视频-字幕对上大规模预训练,并用联合监督——生成用下一词预测,嵌入用多模态对比损失。第二阶段在一个带高质量生成字幕的较小数据集上继续预训练。最后一阶段在覆盖视频问答、字幕、定位和检索的多任务数据集上做指令微调。我们提出的模型 ViLL-E(读作 willy)在视频问答上与 VideoLLM 相当,在定位上超过它们。我们的嵌入缩小了与非 LLM 定位专用方法的差距,并在视频检索上达到最先进水平。
In summary, the key contributions of our work are:
概括起来,本文的主要贡献如下:
- • We present ViLL-E, an unified solution to diverse video-understanding tasks, which is the first VideoLLM that can generate text responses or video & text embeddings with a single model.
- • 我们提出 ViLL-E,作为多样视频理解任务的统一方案;它是首个能用同一套模型既生成文本回复、又生成视频与文本嵌入的 VideoLLM。
- • An effective three-stage training pipeline: (1) large-scale joint captioning + contrastive learning, (2) continued pre-training on high quality captions, and (3) multitask fine-tuning.
- • 一条有效的三阶段训练流程:(1)大规模联合字幕生成 + 对比学习,(2)在高质量字幕上继续预训练,(3)多任务微调。
- • Our single model ViLL-E, outperforms VideoLLMs by more than 7% across three moment retrieval benchmarks, rivals them on VideoQA, and is competitive with expert retrieval models.
- • 单一模型 ViLL-E 在三个时刻检索基准上比 VideoLLM 高出 7% 以上,在视频问答上与它们相当,并与专用检索模型具有竞争力。
- • ViLL-E also solves new tasks in zero-shot setting, including long-caption retrieval, compositional video search, and two-stage retrieve-and-match pipelines using a single model.
- • ViLL-E 还能在零样本设定下完成新任务,包括长字幕检索、组合视频搜索,以及用同一模型实现的两阶段“检索再匹配”流程。
2 Related Work
2 相关工作
Video Expert Models for Retrieval and Localization: Prior works on text-video and video-text retrieval [50, 28, 11, 37, 45] typically extend dual-encoder image-text models such as CLIP [33] or BLIP [20] to video by aggregating the CLIP vision encoder outputs from multiple frames with architecture adaptations to model temporal relations [5, 8]. For retrieval they use dense vector search in the embedding space while a few other methods use an additional reranking step with cross-modality fusion layers [23, 9]. Prior works on temporal localization such as Moment-DETR [17], QD-DETR [30], TR-DETR [39] and UnLoc [51] build on ideas from retrieval. Typically Video-Text Fusion is also employed to score the match between a given text and video segment for compatibility. Specific unique ideas include additional modules like time span prediction in QD-DETR and learnable moment queries in Moment-DETR. Vid2Seq [52] utilizes dense captioning trained on large video-text datasets instead of retrieval for localization. These expert models are close to the state of the art frontier for their respective tasks, however they are specialized for one task, and are typically finetuned individually for each dataset.
用于检索与定位的视频专用模型: 先前关于文本-视频和视频-文本检索的工作 [50, 28, 11, 37, 45],通常把 CLIP [33] 或 BLIP [20] 这类双编码器图像-文本模型扩展到视频:对多帧的 CLIP 视觉编码器输出做聚合,并调整结构以建模时序关系 [5, 8]。检索时,它们在嵌入空间中做稠密向量搜索;也有少数方法再加一步跨模态融合层做重排 [23, 9]。时刻定位方面的先前工作,如 Moment-DETR [17]、QD-DETR [30]、TR-DETR [39] 和 UnLoc [51],也借鉴了检索中的思路。通常还会用视频-文本融合,为给定文本与视频片段的匹配程度打分。一些独特设计包括 QD-DETR 中的时间跨度预测模块,以及 Moment-DETR 中可学习的时刻查询。Vid2Seq [52] 则不用检索来做定位,而是在大规模视频-文本数据上训练稠密字幕。这些专用模型在各自任务上接近最先进水平,但往往只针对单一任务,并且通常要按数据集分别微调。
Video LLMs: Many recent works have extended ImageLLMs such as LLaVA [26] to videos. VideoLLaVA [24] is a straightforward extension that utilizes LanguageBind [59] encoder instead of CLIP because its trained on both images and videos. Some works have tried to extend a BLIP2 style architecture to Videos, namely VideoChat [21] and Video LLaMA [57]. In addition, some works have focused on creation of strong IFT datasets specific to video tasks, VideoChatGPT [29] created IFT data using ActivityNet dense captions and semi-automated process with questions generated by GPT3.5 using dense text captions and open vocabulary attributes. In VideoChat2 [21], another diverse IFT dataset was created by combining many (30+) datasets together. These models excel at video question answering and other text generation tasks, however they often struggle at temporal localization tasks and don’t generate embeddings, which are necessary for video retrieval.
视频大语言模型: 许多近期工作把 LLaVA [26] 这类图像 LLM 扩展到了视频。VideoLLaVA [24] 是较直接的扩展,用 LanguageBind [59] 编码器替换 CLIP,因为它同时在图像和视频上训练过。也有工作尝试把 BLIP-2 风格的架构扩展到视频,例如 VideoChat [21] 和 Video LLaMA [57]。此外,一些工作专注于为视频任务构建更强的指令微调(IFT)数据:VideoChatGPT [29] 利用 ActivityNet 稠密字幕,并以半自动流程、由 GPT-3.5 根据稠密文本字幕和开放词汇属性生成问题。VideoChat2 [21] 则把 30 多个数据集合并,得到另一套多样的 IFT 数据。这些模型在视频问答和其他文本生成任务上表现出色,但在时序定位上往往吃力,也不能生成视频检索所必需的嵌入。
Video LLMs with Localization IFT: Recent works have sought to instill temporal awareness and localization skills in video LLMs. Broadly, the architectures of these approaches can be categorized into LLAVA-like or BLIP2-like [19] depending on whether or not they use a Q-Former for token reduction before input to LLM. TimeChat [36] uses sliding window Q-Formers as a projection layer, VTimeLLM [12] uses MLP, HawkEye [46] uses simple Q-Former and Momentor [32] proposes a parallel temporal perception module as part of its projection layers. LLaVA-ST [18] inserts special spatio-temporal tokens into the language stream, and aligns them with vision features through a language-aligned position embedding. To impart temporal awareness to the models, these works use IFT datasets with specialized tasks such as dense captioning, timestamp generation for localization, repetition counting and more. On localization tasks, these models outperform general VideoLLMs, however they are unable to outperform expert models which do not use LLMs.
带定位指令微调的视频大语言模型: 近期工作试图让视频 LLM 具备时间意识和定位能力。按架构,这类方法大致可分成 LLaVA 风格和 BLIP-2 风格 [19],差别在于是否在送入 LLM 前用 Q-Former 压缩 token。TimeChat [36] 用滑窗 Q-Former 作投影层,VTimeLLM [12] 用 MLP,HawkEye [46] 用简单的 Q-Former,Momentor [32] 则在投影层中加入并行的时间感知模块。LLaVA-ST [18] 把特殊的时空 token 插入语言流,再通过与语言对齐的位置编码把它们和视觉特征对齐。为了赋予模型时间意识,这些工作使用带有专门任务的 IFT 数据,例如稠密字幕、用于定位的时间戳生成、重复计数等。在定位任务上,它们好过通用 VideoLLM,但仍然赶不上不使用 LLM 的专用模型。
LLMs for Embedding generation: Some works in the NLP domain [43, 16] have demonstrated that generative LLMs can be finetuned with a limited amount of data (relative to the scale of generative pre-training) to generate strong embeddings for a variety of downstream tasks including retrieval. GRIT [31] has demonstrated that a pre-trained LLM can be jointly finetuned to improve on both generative instruction following and embedding generation, maintaining strong performance on both sets of tasks. Some recent concurrent works such as VLM2Vec [14], GME [58] and MM-Embed [25] have sought to use LLMs for multi-modal embeddings, but are limited to images.
用大语言模型生成嵌入: 自然语言处理领域的一些工作 [43, 16] 表明,生成式 LLM 只需相对生成预训练而言较少的数据做微调,就能为检索等各类下游任务生成很强的嵌入。GRIT [31] 进一步表明,可以对预训练 LLM 做联合微调,同时提升生成式指令遵循和嵌入生成,并在两类任务上都保持强劲表现。近期一些并行工作,如 VLM2Vec [14]、GME [58] 和 MM-Embed [25],也尝试用 LLM 做多模态嵌入,但目前仍限于图像。
Our approach unifies text-generative and embedding-generative capabilities into a single VideoLLM model, which is competitive with task-specific and dataset-specific expert models on temporal localization and retrieval tasks that require embedding generation, as well as with Video LLMs on text generation like QA and captioning tasks.
我们的方法把文本生成与嵌入生成统一到同一个 VideoLLM 中。它在需要生成嵌入的时序定位和检索任务上,能与按任务、按数据集定制的专用模型竞争;在问答和字幕等文本生成任务上,也能与 VideoLLM 竞争。
Figure 2: Our model ViLL-E is a multi-modal LLM which has been equipped with additional embedding generation capabilities. Individual video frames are first encoded by a pretrained vision encoder, with extracted features projected into embedding vectors. These visual embeddings are combined with textual embeddings of an input prompt and jointly processed autoregressively through a pre-trained Multi-Modal LLM’s transformer layers. After the model autoregressively generates an end-of-sequence (|EOS|) token, all the generated tokens are collected and passed to our embedding head. The embedding head employs an attention-based learnable pooling mechanism to extract and aggregate relevant information across the tokens; The generated tokens are further passed through projection layers and mean pooled to yield a compact, informative video embedding. The ability to dynamically generate different number of tokens to feed the embedding head allows the model to handle videos of different complexity levels and is key for its success at varied retrieval tasks.
图 2:我们的模型 ViLL-E 是一个被赋予了额外嵌入生成能力的多模态大语言模型。各视频帧先由预训练视觉编码器编码,提取的特征再投影为嵌入向量。这些视觉嵌入与输入提示的文本嵌入合并,并经由预训练多模态 LLM 的 Transformer 层做自回归联合处理。模型自回归生成结束符(|EOS|)之后,把全部已生成 token 收集起来,送入我们的嵌入头。嵌入头用基于注意力的可学习池化机制,从这些 token 中提取并聚合相关信息;再经投影层和均值池化,得到一条紧凑而信息丰富的视频嵌入。能够动态生成不同数量的 token 再送给嵌入头,使模型可以应对不同复杂度的视频,这也是它能在多种检索任务上取得成功的关键。
3 Method
3 方法
In this section, we present the architecture and training methodology of our VideoLLM, designed to leverage multi-modal video data to perform both generative and embedding tasks. Our approach involves a multi-stage and multi-task process that includes continued multi-modal pre-training to improve video language alignment, and subsequent multi-task finetuning.
本节介绍我们这套 VideoLLM 的架构与训练方法,旨在利用多模态视频数据同时完成生成任务和嵌入任务。方法是一个多阶段、多任务过程,包括继续做多模态预训练以加强视频-语言对齐,以及随后的多任务微调。
3.1 Model Architecture
3.1 模型架构
Our model (see Figure 2) features an LLM backbone, shared by both its generative and embedding tasks, integrated with a vision encoder module designed to effectively process video inputs. We initialize the model’s weights using a pretrained multimodal LLM (PaliGemma-3B [6]). Following PrefixLM (Raffel et al., 2020) [34], visual tokens and input prompts are processed with bidirectional attention, whereas causal attention is applied to the suffix autoregressively generated by the model. For text generation tasks (e.g. VideoQA), our model follows the same approach as PaliGemma.
如图 2 所示,我们的模型以一个 LLM 骨干为核心,生成任务和嵌入任务共享该骨干,并配有一个用于有效处理视频输入的视觉编码器模块。模型权重用预训练多模态大语言模型 PaliGemma-3B [6] 初始化。遵循 PrefixLM(Raffel et al., 2020)[34],视觉 token 和输入提示使用双向注意力,模型自回归生成的后缀则使用因果注意力。对于视频问答等文本生成任务,模型的做法与 PaliGemma 相同。
3.2 Embedding Head Design
3.2 嵌入头设计
We introduce a learnable embedding head to support retrieval-oriented tasks with four design goals: (1) dedicated modeling capacity separate from generative tasks, (2) a bottleneck that yields dense embeddings without overly restricting representation power, (3) adaptability to variable token lengths for videos of different complexity, and (4) parameter efficiency. We evaluated several approaches: 1. Attention-Free pooling: Pools penultimate layer LLM tokens into a fixed-size embedding, lightweight but dependent on heuristic pooling (cf. Sentence-BERT [35]). 2. Self-Attention: Adds an extra transformer layer, offering some task-specific capacity but redundant with prior LLM layers and lacking a bottleneck. 3. Q-Former: Learned queries cross-attend to LLM outputs to produce fixed-length embeddings (e.g., BLIP-2 [19]); effective but inflexible for variable content length, as output size always equals number of learnable queries. 4. ‘K-Former’: Learns only keys, enabling variable-length outputs. 5. Our ‘KV-Former’: Uses LLM tokens as queries with learnable key and value “pooling tokens,” each. Enabling variable-length outputs while acting as a bottleneck akin to dictionary learning.
我们引入一个可学习的嵌入头,以支持面向检索的任务,并设定四个设计目标:(1)与生成任务分开的专用建模容量,(2)在不过度限制表征能力的前提下得到稠密嵌入的瓶颈,(3)能适应不同复杂度视频的可变 token 长度,(4)参数效率。我们比较了几种方案:1. 无注意力池化:把倒数第二层的 LLM token 池化成固定大小的嵌入,轻量但依赖启发式池化(参见 Sentence-BERT [35])。2. 自注意力:额外加一层 Transformer,能提供一定的任务专用容量,但与前面的 LLM 层重复,且没有瓶颈。3. Q-Former:用可学习查询对 LLM 输出做交叉注意力,得到固定长度嵌入(如 BLIP-2 [19]);效果不错,但对可变内容长度不够灵活,因为输出大小始终等于可学习查询的个数。4. “K-Former”:只学习键,从而允许变长输出。5. 我们的 “KV-Former”:用 LLM token 作为查询,并各设 个可学习的键和值“池化 token”。这样既能变长输出,又能像字典学习那样形成一个瓶颈。
Figure 3: Three stages of training. Stage 1 (§ 3.4), the large scale joint contrastive and generative pre-training, which utilizes (a) Video-Text retrieval task and (b) Video captioning on the large scale Shutterstock video dataset. Stage 2 (§ 3.5), continues a similar joint pre-training approach (retrieval + captioning tasks as shown in (a) and (b)) on a smaller high quality dataset created using Claude-3-Sonnet. Stage 3 (§ 3.6), our multi-task fine-tuning stage, in addition to the prior tasks (retrieval + captioning) integrates (b) Video QA, (c) Video-Text matching and (d) Temporal Localization to improve all-around retrieval capabilities.
图 3:三阶段训练。第一阶段(第 3.4 节) 是大规模联合对比与生成预训练,在大规模 Shutterstock 视频数据上使用(a)视频-文本检索任务和(b)视频字幕。第二阶段(第 3.5 节) 在用 Claude-3-Sonnet 构建的较小高质量数据集上,继续类似的联合预训练(如图中(a)(b)所示的检索 + 字幕任务)。第三阶段(第 3.6 节) 是多任务微调:在原有任务(检索 + 字幕)之外,再加入(b)视频问答、(c)视频-文本匹配和(d)时序定位,以全面提升检索能力。
Our embedding head (on the right in Fig. 2) receives tokens from the final LLM transformer layer and employs a learnable attentive pooling mechanism. Specifically, a transformer-based self-attention layer adaptively weighs tokens, using the LLM’s tokens as queries and learnable keys and values (“pooling tokens”). After this attention operation, the weighted representations are projected into the embedding space through a multi-layer perceptron (MLP) and subsequently mean-pooled.
我们的嵌入头(图 2 右侧)接收来自 LLM 最后一层 Transformer 的 token,并采用可学习的注意力池化。具体来说,一层基于 Transformer 的自注意力会对 token 做自适应加权,用 LLM 的 token 作为查询,用 个可学习的键和值(“池化 token”)。注意力之后,加权表征再经多层感知机(MLP)投影到嵌入空间,并做均值池化。
For the video embedding generation, our model uniquely leverages an end-of-sequence ()-triggered embedding generation mechanism. This mechanism allows the model to utilize variable inference steps tailored to each video’s complexity - more steps for complex videos needing deeper analysis, fewer for simple ones. This adaptive computation approach substantially improves performance over existing fixed-step LLM-based embedding methods, offering both efficiency and improved representational quality. The proposed design is flexible and can be leveraged for various tasks, e.g. text and video embeddings for temporal localization, composed video retrieval etc. For text embedding generation, we rely on standard inference similar to prior works [31], with entire text embeddings being generated in a single step.
在生成视频嵌入时,我们的模型独特地采用由结束符()触发的嵌入生成机制。该机制允许模型按每段视频的复杂度使用可变的推理步数——复杂、需要更深分析的视频用更多步,简单视频则用更少步。这种自适应计算显著优于现有固定步数的基于 LLM 的嵌入方法,既更高效,表征质量也更好。这一设计很灵活,可用于多种任务,例如时序定位中的文本与视频嵌入、组合视频检索等。文本嵌入的生成则沿用与先前工作 [31] 类似的标准推理,整个文本嵌入一步算出。
3.3 Training Objectives
3.3 训练目标
We follow a three-stage training strategy: (1) pre-training on a large, diverse, and noisy dataset to jointly improve video captioning and embedding capabilities; (2) continued pre-training on a smaller, higher-quality dataset; and (3) multi-task finetuning to unlock the model’s performance across diverse downstream tasks. To ensure parameter efficiency during fine-tuning, we adopt Low-Rank Adapters (LoRA). The vision projection module and embedding head are fully trained to maximize representation learning. Our model is designed to effectively balance multimodal generative capabilities with state-of-the-art performance on embedding-oriented retrieval tasks. Across our training stages, a total of four different tasks are used, which are described below.
我们采用三阶段训练策略:(1)在一个大规模、多样且带噪声的数据集上预训练,同时提升视频字幕和嵌入能力;(2)在一个更小、质量更高的数据集上继续预训练;(3)多任务微调,以释放模型在多样下游任务上的表现。为了在微调时保持参数效率,我们使用低秩适配器(LoRA)。视觉投影模块和嵌入头则全量训练,以充分学习表征。模型的设计目标是在多模态生成能力与面向嵌入的检索任务的先进性能之间取得平衡。各个训练阶段一共使用下面四类任务。
Text–Video Retrieval (Fig. 3a). This task aligns video and text embeddings using CLIP-like [33] contrastive loss, encouraging each paired example to outrank in-batch negatives.
文本-视频检索(图 3a)。 该任务用类似 CLIP [33] 的对比损失对齐视频嵌入和文本嵌入,使每个成对样本在批次内负例中排到前面。
Video Captioning & Question–Answering (Fig. 3b). The goal of captioning is for the model to generate accurate descriptions of videos. The model learns to predict each token in the caption, given the previous tokens and the video. For QA, the model generates answers to questions about the video; the question serves as part of the prompt, guiding the model’s attention to answer accurately. We use the next-token prediction loss.
视频字幕与问答(图 3b)。 字幕任务的目标是让模型生成对视频的准确描述。模型在给定已生成词和视频的条件下,学习预测字幕中的每一个 token。问答任务中,模型为关于视频的问题生成答案;问题作为提示的一部分,引导模型把注意力放到准确作答上。我们使用下一词预测损失。
Text–Video Matching (Fig. 3c). As a complement to the retrieval task we also formulate a matching task where the model is trained to identify if a given video and caption form a matching pair. Binary decisions are supervised by labels.
文本-视频匹配(图 3c)。 作为检索任务的补充,我们还设置了匹配任务:训练模型判断给定视频和字幕是否构成匹配对。二分类决策由标签监督。
Temporal Localization (Fig. 3d). Localization requires the model to match specific video segments to text annotations while distinguishing similar but incorrect segments. Sliding window hard negative mining provides challenging non-matching segments with limited temporal overlap (IoU ), helping the model learn to differentiate similar content. We use a standard contrastive formulation with one positive and two hard negatives per query.
时序定位(图 3d)。 定位要求模型把特定视频片段与文本标注匹配起来,同时区分相近但不正确的片段。我们用滑窗挖掘困难负例,提供时间重叠有限(IoU )的非匹配片段,帮助模型学会区分相近内容。我们采用标准对比公式,每个查询对应一个正例和两个困难负例。
Complete notation and full equations are provided in the Appendix B. The following subsections discuss our multi-stage training strategy in detail.
完整记号和全部公式见附录 B。 下面几小节详细讨论我们的多阶段训练策略。
3.4 Training Stage 1: Generative-Contrastive Pre-Training
3.4 训练阶段 1:生成-对比预训练
In stage 1, we utilize a large video-caption dataset and jointly train the network using generative and contrastive supervision. In the first forward pass, for each video caption pair, video input tokens are passed along with a prompt asking the model to describe the video. Captioning output for the generative part and the video embedding are generated simultaneously in this pass. This is followed by a second computationally lighter forward pass with text-only input to generate the text embedding. The next token prediction loss is applied to the caption generation task, and the CLIP multi-modal contrastive loss is applied for the embedding generation task. The generative component aims to improve the model’s ability to create coherent and contextually relevant textual descriptions of the video content, while the contrastive component enhances its discriminative power by teaching it to differentiate between relevant and irrelevant text-video pairs. This dual-objective training strategy ensures that the model not only learns to generate high-quality textual content but also develops a better alignment between modalities.
第一阶段,我们使用大规模视频-字幕数据,并用生成监督和对比监督联合训练网络。第一次前向传播中,对每个视频-字幕对,把视频输入 token 连同一条要求描述该视频的提示一起送入模型。这一趟会同时得到生成支路的字幕输出,以及视频嵌入。随后进行第二次计算量更轻的、仅含文本输入的前向传播,以生成文本嵌入。下一词预测损失加在字幕生成任务上,CLIP 多模态对比损失加在嵌入生成任务上。生成目标旨在提高模型为视频内容写出连贯、符合语境的文本描述的能力;对比目标则通过让模型区分相关与不相关的文本-视频对,增强其判别力。这种双目标训练使模型不仅学会生成高质量文本,也能更好地对齐不同模态。
3.5 Training Stage 2: Continued Pre-Training
3.5 训练阶段 2:继续预训练
While our pre-training data has a diverse collection of videos, the captions tend to be short summaries of the video content, often missing many details in the video. Since preserving important details from the video in our VideoLLM’s feature space is beneficial for downstream tasks, we supplement our pre-training with a small intermediate pre-training phase using higher quality captions for a subset of videos from our pre-training set. The dataset for intermediate pre-training with high quality descriptive captions is discussed in detail in Section 4.
预训练数据中的视频虽然多样,字幕却往往只是对视频内容的短摘要,常常漏掉许多细节。把视频中的重要细节保留在 VideoLLM 的特征空间里,对下游任务是有益的,因此我们补充了一个较小的中间预训练阶段,对预训练集的一个子集使用更高质量的字幕。用于这一阶段的高质量描述性字幕数据,将在第 4 节详细讨论。
3.6 Training Stage 3: Multi-Task Finetuning
3.6 训练阶段 3:多任务微调
In this stage, a high-quality finetuning dataset is used combining supervision across four tasks: captioning or question answering (QA), retrieval, matching and temporal localization. The fine-tuning process integrates these tasks to ensure that the model develops well-rounded capabilities, enabling it to handle diverse real-world scenarios that involve multi-modal understanding and generation.
这一阶段使用高质量微调数据,把四类任务的监督合在一起:字幕或问答(QA)、检索、匹配,以及时序定位。微调把这些任务结合起来,使模型形成更均衡的能力,从而应对涉及多模态理解与生成的多样现实场景。
4 Training Data
4 训练数据
In this section, we provide details about the datasets used in the different training stages of our model.
本节介绍模型各个训练阶段所用的数据集。
4.1 Stage 1: Large Scale Pre-Training Data
4.1 阶段 1:大规模预训练数据
For pre-training, we use video–caption pairs from the licensed Shutterstock stock videos dataset[^1], which contains 10M unique captions. Retaining one random clip per caption yields 10M video–caption pairs, equivalent to the WebVid-10M dataset [4] previously used in academic research.
预训练使用来自授权 Shutterstock 素材视频数据集的视频-字幕对[^1],其中包含 1000 万条不重复字幕。每个字幕保留一个随机片段,便得到 1000 万对视频-字幕,规模与此前学术研究常用的 WebVid-10M 数据集 [4] 相当。
图 4:把原始字幕与 Claude-3-Sonnet 生成的高质量字幕加以对照。
4.2 Stage 2: High Quality Intermediate Data
4.2 阶段 2:高质量中间数据
We use a re-captioned subset of the Shutterstock dataset, whose extensively labeled keywords we utilize for the balancing operation. We exclude keywords with 30 occurrences. Starting with the most frequent remaining keywords, we add a maximum of 500 videos per keyword to the candidate pool. Further details are in Supplementary L. In order to generate this data within a limited budget, we generate captions for a diverse subset of 200,000 videos. We follow the insight from prior works (e.g., LLaVA [26], Meta-CLIP [48], Cambrian [40]) that under-sampling common concepts is important for large scale vision-language training to achieve better generalization on rare concepts.
我们使用 Shutterstock 数据集中经过重新配字幕的一个子集;该数据集带有大量关键词标注,我们用它们做平衡抽样。出现次数少于 30 的关键词被排除。从剩余关键词中频率最高的开始,每个关键词最多向候选池加入 500 条视频。更多细节见补充材料 L。为了在有限预算内生成这些数据,我们为一组多样的 20 万条视频生成字幕。我们沿用先前工作(如 LLaVA [26]、Meta-CLIP [48]、Cambrian [40])的看法:在大规模视觉-语言训练中,对常见概念做降采样,有助于在罕见概念上获得更好的泛化。
We use the Claude-3-Sonnet model prompted with 20 frames from the video, each resized to 480 pixels height, along with the following text prompt: “Provide detailed narrative caption for the video, covering its overall theme, visuals, characters, actions, scene, and key moments comprehensively". The original caption and generated high quality caption for a sample video are shown in Fig. 4.
我们使用 Claude-3-Sonnet,从视频中取 20 帧、每帧高度缩放到 480 像素,并配上如下文本提示:“为该视频提供详细的叙事性字幕,全面覆盖整体主题、画面、人物、动作、场景和关键时刻”。一条样例视频的原始字幕和生成的高质量字幕见图 4。
4.3 Stage 3: Multi-Task Fine-Tuning Data
4.3 阶段 3:多任务微调数据
| Dataset | # | Cap | QA | Loc |
|---|---|---|---|---|
| MSR-VTT | 10K | ✓ | ✓ | ✗ |
| ActivityNet | 30K | ✓ | ✓ | ✓ |
| DiDeMo | 10K | ✓ | ✗ | ✓ |
| Shutterstock | 50K | ✓ | ✗ | ✗ |
Table 2: Multi-Task Dataset.
| 数据集 | 数量 | 字幕 | 问答 | 定位 |
|---|---|---|---|---|
| MSR-VTT | 10K | ✓ | ✓ | ✗ |
| ActivityNet | 30K | ✓ | ✓ | ✓ |
| DiDeMo | 10K | ✓ | ✗ | ✓ |
| Shutterstock | 50K | ✓ | ✗ | ✗ |
表 2:多任务数据集。
Our fine-tuning stage is analogous to Instruction FineTuning (IFT) in LLMs. We design our fine-tuning dataset to include multi-task supervision covering these tasks as well as video captioning. We select 100,000 samples from a small number of video-text datasets and obtain supervision from each as summarized in Table 2. All these datasets contribute to captioning, retrieval and matching supervision. MSR-VTT provides short video captions and question answering supervision. ActivityNet and DiDeMo consist of longer videos with distinct temporally localizable captions for chunks of the video. Hence, these datasets can provide localization supervision. ActivityNet has also previously been annotated with question answering data. We add a random subset of Shutterstock video-caption pairs (original captions) to the mix to maintain long tail diversity.
我们的微调阶段类似于大语言模型中的指令微调(Instruction FineTuning, IFT)。微调数据集被设计为包含覆盖这些任务以及视频字幕的多任务监督。我们从少数几个视频-文本数据集中选取 10 万条样本,并按表 2 汇总的方式从每个数据集获取监督。所有这些数据集都提供字幕、检索和匹配监督。MSR-VTT 提供短视频字幕和问答监督。ActivityNet 和 DiDeMo 包含较长视频,其字幕可以按片段做时序定位,因此能够提供定位监督。ActivityNet 此前也标注过问答数据。我们还加入一部分随机抽取的、带原始字幕的 Shutterstock 视频-字幕对,以保持长尾多样性。
5 Results
5 实验结果
First we demonstrate the unified capabilities of our model on 8 benchmark datasets. Next up, we demonstrate the unique capabilities of our model by solving novel tasks that are hard for any single model prior to this work. Finally, we show ablations demonstrating the effectiveness of each component in our modeling and training approach.
我们首先在 8 个基准数据集上展示模型的统一能力。接下来,我们展示模型在若干新任务上的独特能力,这些任务在本文之前很难由单一模型完成。最后,我们用消融实验说明建模和训练中各部分的作用。
5.1 Retrieval Tasks
5.1 检索任务
We evaluate our model on video retrieval, and moment retrieval/temporal localization tasks. These benchmarks together provide a holistic evaluation of video understanding capabilities. We use standard evaluation protocols following prior works [36, 37, 24].
我们在视频检索,以及时刻检索/时序定位任务上评估模型。这些基准合在一起,可以对视频理解能力做较完整的评价。我们采用与先前工作 [36, 37, 24] 一致的标准评估协议。
Moment Retrieval: We test on ActivityNet-Captions, Charades-STA and QVHighlights. Contextualized text embeddings are extracted by passing {sparse video frames} <|txt_embed|> {text}. Video clips are extracted using a sliding window operation and clip embeddings are generated by passing {clip frames} <|vid_embed|>. Text and Video Clip embedding similarities are post-processed using standard methods to get the matching segment.
时刻检索:我们在 ActivityNet-Captions、Charades-STA 和 QVHighlights 上测试。上下文相关的文本嵌入,通过输入{稀疏视频帧} <|txt_embed|> {文本} 得到。视频片段由滑窗切出,片段嵌入通过输入{片段帧} <|vid_embed|> 得到。文本嵌入与视频片段嵌入的相似度再按标准方法后处理,以得到匹配片段。
Video Retrieval: We evaluate on MSR-VTT, DiDeMo and VATEX. Video embeddings are extracted by passing following input to the model: {video} <|vid_embed|>, while text embeddings are extracted by passing <|txt_embed|> {text}. Best pair is matched using nearest neighbor retrieval.
视频检索:我们在 MSR-VTT、DiDeMo 和 VATEX 上评估。视频嵌入通过向模型输入{视频} <|vid_embed|> 得到,文本嵌入通过输入 <|txt_embed|> {文本} 得到。最佳配对用最近邻检索匹配。
| Text to Moment Retrieval | Text to Video Retrieval | |||||
|---|---|---|---|---|---|---|
| Method | ActivityNet | Charades | QVHighlights | MSR-VTT | DiDeMo | VATEX |
| R@1,IoU=0.5 | R@1,IoU=0.5 | mAP | R@1 | R@1 | R@1 | |
| Expert Models | ||||||
| M-DETR NeurIPS 21 | - | 53.6FT | 35.7FT | ✗ | ✗ | ✗ |
| QD-DETR CVPR 23 | 33.2FT | 57.3FT | 38.9FT | ✗ | ✗ | ✗ |
| TR-DETR AAAI 24 | - | 57.6FT | 39.9FT | ✗ | ✗ | ✗ |
| SigLIP ICCV 23 | ✗ | ✗ | ✗ | 51.7FT | 55.4FT | 40.8 |
| VidLA CVPR 24 | ✗ | ✗ | ✗ | 58.0FT | 61.1FT | - |
| InternVideo2-CLIP-1B ECCV 24 | ✗ | ✗ | ✗ | 50.0 | 47.7 | 63.2 |
| Video LLMs | ||||||
| Momentor-7B ICML 24 | 23.0 | 27.5 | 7.6 | ✗ | ✗ | ✗ |
| VTimeLLM-7B CVPR 24 | 27.8 | 27.5 | - | ✗ | ✗ | ✗ |
| TimeChat-7B CVPR 24 | 16.3 | 32.2 | 21.7 | ✗ | ✗ | ✗ |
| Qwen2.5-VL-3B arXiv | 28.6 | 38.1 | - | ✗ | ✗ | ✗ |
| LLaVA-ST-7B CVPR 25 | 31.2 | 44.8 | - | ✗ | ✗ | ✗ |
| ViLL-E-2.5B (Ours) | 39.4 | 51.5 | 39.0 | 62.5 | 61.4 | 63.5 |
Table 1: Results on retrieval tasks. In moment retrieval, we outperform specialized Video LLMs by over 8%, and approach the performance of SotA dataset-specific expert models. In video retrieval we beat finetuned SotA models. FT models fine-tuned on train splits. ✗- model is incapable of task
| 文本到时刻检索 | 文本到视频检索 | |||||
|---|---|---|---|---|---|---|
| 方法 | ActivityNet | Charades | QVHighlights | MSR-VTT | DiDeMo | VATEX |
| R@1,IoU=0.5 | R@1,IoU=0.5 | mAP | R@1 | R@1 | R@1 | |
| 专用模型 | ||||||
| M-DETR NeurIPS 21 | - | 53.6FT | 35.7FT | ✗ | ✗ | ✗ |
| QD-DETR CVPR 23 | 33.2FT | 57.3FT | 38.9FT | ✗ | ✗ | ✗ |
| TR-DETR AAAI 24 | - | 57.6FT | 39.9FT | ✗ | ✗ | ✗ |
| SigLIP ICCV 23 | ✗ | ✗ | ✗ | 51.7FT | 55.4FT | 40.8 |
| VidLA CVPR 24 | ✗ | ✗ | ✗ | 58.0FT | 61.1FT | - |
| InternVideo2-CLIP-1B ECCV 24 | ✗ | ✗ | ✗ | 50.0 | 47.7 | 63.2 |
| Video LLM | ||||||
| Momentor-7B ICML 24 | 23.0 | 27.5 | 7.6 | ✗ | ✗ | ✗ |
| VTimeLLM-7B CVPR 24 | 27.8 | 27.5 | - | ✗ | ✗ | ✗ |
| TimeChat-7B CVPR 24 | 16.3 | 32.2 | 21.7 | ✗ | ✗ | ✗ |
| Qwen2.5-VL-3B arXiv | 28.6 | 38.1 | - | ✗ | ✗ | ✗ |
| LLaVA-ST-7B CVPR 25 | 31.2 | 44.8 | - | ✗ | ✗ | ✗ |
| ViLL-E-2.5B(本文) | 39.4 | 51.5 | 39.0 | 62.5 | 61.4 | 63.5 |
表 1:检索任务结果。在时刻检索上,我们比专门的 Video LLM 高出 8% 以上,并接近按数据集微调的最先进专用模型。在视频检索上,我们超过经过微调的最先进模型。FT 表示在训练集上微调的模型。✗ 表示该模型无法完成该任务。
In moment retrieval (Table 1), ViLL-E outperforms localization specialized video-LLMs by nearly 10%, and approaches or surpasses the per-dataset fine-tuned performance of non-LLM methods. In text-video retrieval, ViLL-E matches image (SigLIP) and video (VidLA) retrieval models.
在时刻检索上(表 1),ViLL-E 比专做定位的 VideoLLM 高出近 10%,并接近或超过按数据集微调的非 LLM 方法。在文本-视频检索上,ViLL-E 与图像检索模型 SigLIP 和视频检索模型 VidLA 相当。
5.2 QA Tasks
5.2 问答任务
Among VideoLLMs specialized for temporal localization, our method achieves top performance in several categories (MSR, VCG, MVBench) indicated by green upward arrows showing improvements over previous methods. It slightly under-performs compared to LLaVA-ST on the MSVD dataset, indicated by a small red downward arrow. While SotA general VideoLLMs Qwen-2.5VL and LLaVA-Video outperform ViLL-E on MVBench & VideoMME, our approach remains competitive.
在专做时序定位的 VideoLLM 中,我们的方法在若干类别(MSR、VCG、MVBench)上取得最高成绩,图中用绿色向上箭头标出相对先前方法的提升。它在 MSVD 数据集上略低于 LLaVA-ST,图中用一个较小的红色向下箭头标出。最先进的通用 VideoLLM,即 Qwen-2.5VL 和 LLaVA-Video,在 MVBench 和 VideoMME 上超过 ViLL-E,但我们的方法仍然具有竞争力。
| Method | MSR | VCG | MSVD | MVBench | VidMME |
|---|---|---|---|---|---|
| General Video LLMs | |||||
| Qwen-2.5VL | - | - | - | 67.0 | 61.5 |
| LLaVA-Video | - | 3.5 | - | 58.6 | 63.3 |
| Temporally-Specialized Video LLMs | |||||
| ST-LLM | 63.2 | 3.15 | 74.6 | 54.9 | 37.9 |
| LLaVA-ST | 59.0 | 3.3 | 75.9 | 64.2 | |
| Momentor | 55.6 | 3.0 | 68.9 | - | - |
| VTimeLLM | 50.2 | 2.9 | - | - | - |
| TimeChat | 45.0 | 2.3 | - | 38.5 | 34.7 |
| ViLL-E | 65.2 | 3.7 | 75.2 | 64.7 | 45.0 |
Table 3: ViLL-E is competitive with SotA VideoLLMs on QA tasks in both simple VideoQA (MSR-VTT QA, VideoChatGPT Benchmark and MSVD-QA) and Comprehensive Benchmarks (MVBench and VideoMME).
| 方法 | MSR | VCG | MSVD | MVBench | VidMME |
|---|---|---|---|---|---|
| 通用 Video LLM | |||||
| Qwen-2.5VL | - | - | - | 67.0 | 61.5 |
| LLaVA-Video | - | 3.5 | - | 58.6 | 63.3 |
| 时序专用 Video LLM | |||||
| ST-LLM | 63.2 | 3.15 | 74.6 | 54.9 | 37.9 |
| LLaVA-ST | 59.0 | 3.3 | 75.9 | 64.2 | |
| Momentor | 55.6 | 3.0 | 68.9 | - | - |
| VTimeLLM | 50.2 | 2.9 | - | - | - |
| TimeChat | 45.0 | 2.3 | - | 38.5 | 34.7 |
| ViLL-E | 65.2 | 3.7 | 75.2 | 64.7 | 45.0 |
表 3:ViLL-E 在问答任务上与最先进的 VideoLLM 具有竞争力,既包括简单视频问答(MSR-VTT QA、VideoChatGPT 基准和 MSVD-QA),也包括综合基准(MVBench 和 VideoMME)。
5.3 Additional Tasks
5.3 额外任务
In this section we demonstrate ViLL-E’s ability to solve some unique tasks that are made possible by having unified generative and embedding capabilities in one VideoLLM.
本节展示 ViLL-E 在若干独特任务上的能力,这些任务之所以可行,正是因为生成能力与嵌入能力被统一在同一个 VideoLLM 中。
Two Stage Retrieval using Re-Ranking: This task leverages the VideoLLM’s reasoning capability to augment retrieval performance. In the first stage, Top- candidate videos matching a caption are retrieved using generated embeddings. During second stage matching, each candidate video-caption pair is input to the VideoLLM and the matching loss is used to re-rank the Top- candidate videos. The results in Table 4 () show that our two stage matching process results in an improvement of about 2% in R@1 accuracy.
利用重排的两阶段检索: 该任务借助 VideoLLM 的推理能力来增强检索表现。第一阶段,用生成的嵌入检索与字幕匹配的前 个候选视频。第二阶段匹配时,把每个候选视频-字幕对输入 VideoLLM,并用匹配损失对这前 个候选做重排。表 4()的结果表明,这一两阶段匹配过程使 R@1 大约提升 2%。
| MSR-VTT | T V | V T | ||
|---|---|---|---|---|
| Method | R@1 | R@5 | R@1 | R@5 |
| VidLA | 58.0 | 81.1 | 56.1 | 80.5 |
| Ours (1 Stage) | 62.5 | 78.1 | 55.3 | 74.8 |
| Ours (2 Stage) | 62.8 | 80.1 | 57.3 | 83.5 |
Table 4: Two-stage retrieval + matching.
| MSR-VTT | T V | V T | ||
|---|---|---|---|---|
| 方法 | R@1 | R@5 | R@1 | R@5 |
| VidLA | 58.0 | 81.1 | 56.1 | 80.5 |
| 本文(1 阶段) | 62.5 | 78.1 | 55.3 | 74.8 |
| 本文(2 阶段) | 62.8 | 80.1 | 57.3 | 83.5 |
表 4:两阶段检索 + 匹配。
Composed Video Retrieval [42] is the task of retrieving a video matching a given query video along with a modifier text. The retrieved video must closely match a hypothetical video containing the contents of the original video further modified by the instructions in the change text. This is a relatively new task in the field of video retrieval and forms a challenging zero-shot benchmark for our model. To solve this, we extract multi-modal query embeddings by passing {source video frames} <|txt_embed|> {change text}, and retrieve videos by matching with target video embeddings extracted as we do for retrieval. The results in Table 5 demonstrate that in the zero-shot setting our method surpasses recent baselines by more than 5%.
组合视频检索 [42] 的任务是:给定一条查询视频和一句修改文本,检索与之匹配的视频。检索到的视频应尽可能接近这样一条假想视频——它包含原视频的内容,并按修改文本中的指令加以改变。这在视频检索中还是一项较新的任务,也构成对我们模型颇具挑战的零样本基准。为完成该任务,我们通过输入{源视频帧} <|txt_embed|> {修改文本} 提取多模态查询嵌入,再与按检索任务同样方式得到的目标视频嵌入做匹配。表 5 表明,在零样本设定下,我们的方法比近期基线高出 5 个点以上。
| Method | R@1 | R@5 | R@10 |
|---|---|---|---|
| COVR-BLIP AAAI 24 | 45.46 | 70.46 | 79.54 |
| Thawakar et al. CVPR 24 | 47.52 | 72.18 | 82.37 |
| Ours | 53.13 | 74.80 | 85.95 |
Table 5: Zero-Shot Composed Video Retrieval.
| 方法 | R@1 | R@5 | R@10 |
|---|---|---|---|
| COVR-BLIP AAAI 24 | 45.46 | 70.46 | 79.54 |
| Thawakar et al. CVPR 24 | 47.52 | 72.18 | 82.37 |
| 本文 | 53.13 | 74.80 | 85.95 |
表 5:零样本组合视频检索。
Detailed Video Caption Retrieval: Dual encoder video-text retrieval models, like CLIP, are limited by short context windows (e.g., 77 tokens). Our LLM approach supports much longer texts:
详细视频字幕检索: CLIP 这类双编码器视频-文本检索模型受限于较短的上下文窗口(例如 77 个 token)。我们基于 LLM 的方法可以支持长得多的文本:
| Method | Short Caption | Long Caption | ||||||
|---|---|---|---|---|---|---|---|---|
| Avg. = 27 words | Avg. = 500 words | |||||||
| T V | V T | T V | V T | |||||
| R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | |
| SigLIP ICCV23 | 62.5 | 83.2 | 65.4 | 85.1 | 51.7 | 72.7 | 60.4 | 81.0 |
| VidLA CVPR24 | 61.2 | 85.7 | 65.9 | 85.0 | 45.3 | 65.2 | 55.9 | 72.7 |
| LongCLIP ECCV24 | 63.0 | 85.4 | 61.5 | 83.6 | 73.5 | 91.7 | 74.5 | 92.8 |
| ViLL-E | 64.3 | 85.7 | 65.5 | 85.4 | 75.7 | 92.4 | 75.1 | 93.3 |
Table 6: Zero-Shot Video Detailed Caption Retrieval.
| 方法 | 短字幕 | 长字幕 | ||||||
|---|---|---|---|---|---|---|---|---|
| 平均 = 27 词 | 平均 = 500 词 | |||||||
| T V | V T | T V | V T | |||||
| R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | R@1 | R@5 | |
| SigLIP ICCV23 | 62.5 | 83.2 | 65.4 | 85.1 | 51.7 | 72.7 | 60.4 | 81.0 |
| VidLA CVPR24 | 61.2 | 85.7 | 65.9 | 85.0 | 45.3 | 65.2 | 55.9 | 72.7 |
| LongCLIP ECCV24 | 63.0 | 85.4 | 61.5 | 83.6 | 73.5 | 91.7 | 74.5 | 92.8 |
| ViLL-E | 64.3 | 85.7 | 65.5 | 85.4 | 75.7 | 92.4 | 75.1 | 93.3 |
表 6:零样本视频详细字幕检索。
thousands of tokens, enabling effective zero-shot retrieval using detailed captions. Existing benchmarks (e.g., YouCook2) are either too domain-specific or overlap with our training data. We use the AuroraCap-VideoDetailCaption dataset [2], which includes both long and short captions for 1,027 videos. As shown in Table 6, our model benefits from longer captions, improving R@1 by 11.4% (text-to-video) and 9.6% (video-to-text). In contrast, models like SigLIP [53] and VidLA [37] are constrained by short text encoder contexts (64 tokens). LongCLIP [54], with a 248-token limit, improves on long captions but lags our approach.
可达数千个 token,从而能够用详细字幕做有效的零样本检索。现有基准(如 YouCook2)要么过于限域,要么与我们的训练数据重叠。我们使用 AuroraCap-VideoDetailCaption 数据集 [2],它为 1,027 条视频同时提供长字幕和短字幕。如表 6 所示,我们的模型能从更长的字幕中获益,文本到视频的 R@1 提高 11.4%,视频到文本提高 9.6%。相比之下,SigLIP [53] 和 VidLA [37] 受限于较短的文本编码器上下文(64 个 token)。LongCLIP [54] 的上限是 248 个 token,在长字幕上有所提升,但仍落后于我们的方法。
5.4 Ablations
5.4 消融实验
We ablate the design choices for key components of our approach. For computational efficiency, some experiments are performed without pre-training.
我们对方法中关键部分的设计选择做了消融。出于计算效率,部分实验在不做预训练的条件下进行。
| G | C | M | MSR | VCG | MSR | DDM | VTX | ANet | QVH | Ch. |
|---|---|---|---|---|---|---|---|---|---|---|
| ✓ | ✓ | ✓ | 65.1 | 3.7 | 62.8 | 61.5 | 63.7 | 39.4 | 38.9 | 51.7 |
| ✓ | ✓ | × | 63.9 | 3.7 | 60.3 | 60.2 | 62.2 | 39.1 | 38.5 | 51.5 |
| ✓ | × | × | 61.3 | 3.0 | 25.1 | 23.5 | 11.9 | 28.7 | 23.8 | 42.1 |
| × | ✓ | × | 45.5 | 2.1 | 54.7 | 53.1 | 55.2 | 29.3 | 30.3 | 48.8 |
| Head | MSR | VCG | MSR | DDM | VTX | ANet | QVH | Ch. |
|---|---|---|---|---|---|---|---|---|
| Ours | 55.9 | 3.1 | 49.3 | 45.5 | 45.3 | 32.3 | 32.6 | 49.3 |
| × | 40.8 | 2.8 | 30.7 | 29.1 | 32.1 | 19.8 | 20.4 | 30.7 |
| Linear | 42.9 | 2.8 | 32.4 | 29.3 | 33.2 | 23.6 | 25.4 | 42.7 |
| MLP | 51.7 | 3.0 | 43.5 | 39.8 | 39.5 | 30.2 | 29.7 | 47.1 |
| Self-Att | 52.5 | 3.2 | 43.8 | 40.9 | 40.9 | 28.2 | 45.7 | |
| Q-Former | 52.1 | 3.0 | 47.5 | 43.2 | 42.8 | 27.9 | 44.2 | |
| K-Former | 52.7 | 3.1 | 49.0 | 44.1 | 44.7 | 32.8 | 47.4 |
| Pre | HQ | MSR | VCG | MSR | DDM | VTX | ANet | QVH | Ch. |
|---|---|---|---|---|---|---|---|---|---|
| ✓ | ✓ | 65.1 | 3.7 | 62.8 | 61.5 | 63.7 | 39.4 | 38.9 | 51.7 |
| ✓ | × | 63.2 | 3.3 | 58.6 | 59.6 | 60.5 | 36.8 | 38.8 | 51.9 |
| × | × | 55.9 | 3.1 | 49.3 | 45.5 | 45.3 | 32.3 | 32.6 | 49.3 |
(a) Supervision Type (During Finetuning)
Table 7: Ablation Experiments. Benchmarks (left to right). QA: MSR-VTT-QA, VideoChatGPT. Retrieval: MSR-VTT, DiDeMo, VATEX. Localization: ActivityNet, QVHighlights, Charades-STA. Legend: (a) G = Generative, C = Contrastive, M = Matching. (c) Pre. = 10M scale; HQ = High-Quality 200k.
| G | C | M | MSR | VCG | MSR | DDM | VTX | ANet | QVH | Ch. |
|---|---|---|---|---|---|---|---|---|---|---|
| ✓ | ✓ | ✓ | 65.1 | 3.7 | 62.8 | 61.5 | 63.7 | 39.4 | 38.9 | 51.7 |
| ✓ | ✓ | × | 63.9 | 3.7 | 60.3 | 60.2 | 62.2 | 39.1 | 38.5 | 51.5 |
| ✓ | × | × | 61.3 | 3.0 | 25.1 | 23.5 | 11.9 | 28.7 | 23.8 | 42.1 |
| × | ✓ | × | 45.5 | 2.1 | 54.7 | 53.1 | 55.2 | 29.3 | 30.3 | 48.8 |
| 嵌入头 | MSR | VCG | MSR | DDM | VTX | ANet | QVH | Ch. |
|---|---|---|---|---|---|---|---|---|
| 本文 | 55.9 | 3.1 | 49.3 | 45.5 | 45.3 | 32.3 | 32.6 | 49.3 |
| × | 40.8 | 2.8 | 30.7 | 29.1 | 32.1 | 19.8 | 20.4 | 30.7 |
| Linear | 42.9 | 2.8 | 32.4 | 29.3 | 33.2 | 23.6 | 25.4 | 42.7 |
| MLP | 51.7 | 3.0 | 43.5 | 39.8 | 39.5 | 30.2 | 29.7 | 47.1 |
| Self-Att | 52.5 | 3.2 | 43.8 | 40.9 | 40.9 | 28.2 | 45.7 | |
| Q-Former | 52.1 | 3.0 | 47.5 | 43.2 | 42.8 | 27.9 | 44.2 | |
| K-Former | 52.7 | 3.1 | 49.0 | 44.1 | 44.7 | 32.8 | 47.4 |
| Pre | HQ | MSR | VCG | MSR | DDM | VTX | ANet | QVH | Ch. |
|---|---|---|---|---|---|---|---|---|---|
| ✓ | ✓ | 65.1 | 3.7 | 62.8 | 61.5 | 63.7 | 39.4 | 38.9 | 51.7 |
| ✓ | × | 63.2 | 3.3 | 58.6 | 59.6 | 60.5 | 36.8 | 38.8 | 51.9 |
| × | × | 55.9 | 3.1 | 49.3 | 45.5 | 45.3 | 32.3 | 32.6 | 49.3 |
(a)监督类型(微调阶段)
表 7:消融实验。基准从左到右。问答:MSR-VTT-QA、VideoChatGPT。检索:MSR-VTT、DiDeMo、VATEX。定位:ActivityNet、QVHighlights、Charades-STA。图例:(a)G = 生成,C = 对比,M = 匹配。(c)Pre. = 千万级规模;HQ = 20 万条高质量数据。
Supervision Types: (Table 7a) We find that combining generative and contrastive objectives during fine-tuning complements each other, resulting in improvement on both generative and embeddings tasks. Retrieval and localization performances drop moderately on removing matching loss, and significantly on removing contrastive training, indicating the importance of learning good embeddings.
监督类型:(表 7a)我们发现,微调时把生成目标和对比目标结合使用,二者可以互补,从而同时提升生成任务和嵌入任务。去掉匹配损失后,检索和定位成绩中等程度下降;去掉对比训练后下降更为显著,说明学好嵌入非常重要。
Embedding Head (Table 7b) We find that our attentive pooling head outperforms other potential approaches discussed previously in Section 3.2.
嵌入头(表 7b)我们发现,本文的注意力池化头优于第 3.2 节讨论过的其他候选方案。
Pre-Training (Table 7c) The large scale pre-training stage is very important for retrieval tasks, whereas the second pre-training stage with longer higher quality captions helps particularly in longer video datasets and generative tasks.
预训练(表 7c)大规模预训练阶段对检索任务非常重要;而使用更长、质量更高字幕的第二阶段预训练,则尤其有助于较长视频的数据集以及生成任务。
Additional experiments and analysis of our adaptive embedding mechanism: including inference latency, per-duration-bin retrieval accuracy, and distribution of generated token counts, are provided in Appendix A (Tables 10 and 11).
关于自适应嵌入机制的更多实验和分析,包括推理延迟、按视频时长分箱的检索准确率,以及生成 token 数量的分布,见附录 A(表 10 和表 11)。
6 Conclusion
6 结论
We introduced ViLL-E, a unified VideoLLM that combines text generation and embedding-based retrieval within one model. Through a multi-stage generative–contrastive training strategy and an adaptive embedding head, ViLL-E achieves strong results across retrieval, localization, and QA, while enabling new zero-shot tasks like composed and long-caption retrieval. This work highlights the potential of merging generative and discriminative learning for unified multimodal understanding.
我们提出了 ViLL-E,一个在同一模型中结合文本生成与基于嵌入的检索的统一 VideoLLM。通过多阶段的生成-对比训练策略和自适应嵌入头,ViLL-E 在检索、定位和问答上取得了强劲结果,并能够完成组合检索、长字幕检索等新的零样本任务。这项工作表明,把生成学习与判别学习合并起来,有望实现统一的多模态理解。
7 Limitations
7 局限
Our model inherits some limitations of the base Paligemma LLM, e.g. it lacks general multi-turn conversation abilities. Secondly, because our training dataset is primarily in English, we expect that our model would lose some of Paligemma’s multi-lingual capabilities. Our work serves as a proof of concept for the unification of generative and embedding capabilties in VideoLLMs, and any practical model would need to address the issues of multi-turn conversations and multi-linguality.
我们的模型继承了作为底座的 PaliGemma LLM 的一些局限,例如尚不具备通用的多轮对话能力。其次,由于训练数据以英语为主,我们预计模型会失去 PaliGemma 的部分多语能力。这项工作是对 VideoLLM 中生成与嵌入能力合一的概念验证,真正可实用的模型还需要解决多轮对话和多语问题。
References
参考文献
[1] L. Anne Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. Russell (2017) Localizing moments in video with natural language . In Proceedings of the IEEE international conference on computer vision , pp. 5803–5812 . Cited by: Figure 1 .
[2] Anonymous (2024) AuroraCap: efficient, performant video detailed captioning and a new benchmark . In Submitted to The Thirteenth International Conference on Learning Representations , Note: under review External Links: Link Cited by: §5.3 .
[3] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou (2023) Qwen-vl: a frontier large vision-language model with versatile abilities . arXiv preprint arXiv:2308.12966 . Cited by: §1 .
[4] M. Bain, A. Nagrani, G. Varol, and A. Zisserman (2021) Frozen in time: a joint video and image encoder for end-to-end retrieval . In IEEE International Conference on Computer Vision , Cited by: §4.1 .
[5] G. Bertasius, H. Wang, and L. Torresani (2021) Is space-time attention all you need for video understanding? . In ICML , Vol. 2 , pp. 4 . Cited by: §2 .
[6] L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, et al. (2024) Paligemma: a versatile 3b vlm for transfer . arXiv preprint arXiv:2407.07726 . Cited by: §1 , §3.1 .
[7] T. B. Brown (2020) Language models are few-shot learners . arXiv preprint arXiv:2005.14165 . Cited by: §1 .
[8] D. Chen, C. Tao, L. Hou, L. Shang, X. Jiang, and Q. Liu (2022) Litevl: efficient video-language learning with enhanced spatial-temporal modeling . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 7985–7997 . Cited by: §2 .
[9] F. Cheng, X. Wang, J. Lei, D. Crandall, M. Bansal, and G. Bertasius (2023) Vindlu: a recipe for effective video-and-language pretraining . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10739–10750 . Cited by: §2 .
[10] J. Gao, C. Sun, Z. Yang, and R. Nevatia (2017) Tall: temporal activity localization via language query . In Proceedings of the IEEE international conference on computer vision , pp. 5267–5275 . Cited by: Figure 1 , §1 .
[11] S. K. Gorti, N. Vouitsis, J. Ma, K. Golestan, M. Volkovs, A. Garg, and G. Yu (2022) X-pool: cross-modal language-video attention for text-video retrieval . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 5006–5015 . Cited by: §2 .
[12] B. Huang, X. Wang, H. Chen, Z. Song, and W. Zhu (2024) Vtimellm: empower llm to grasp video moments . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14271–14280 . Cited by: §2 .
[13] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. (2023) Mistral 7b . arXiv preprint arXiv:2310.06825 . Cited by: §1 .
[14] Z. Jiang, R. Meng, X. Yang, S. Yavuz, Y. Zhou, and W. Chen (2025) VLM2vec: training vision-language models for massive multimodal embedding tasks . In The Thirteenth International Conference on Learning Representations , External Links: Link Cited by: §2 .
[15] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles (2017) Dense-captioning events in videos . In Proceedings of the IEEE international conference on computer vision , pp. 706–715 . Cited by: Figure 1 , §1 .
[16] C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping (2024) NV-embed: improved techniques for training llms as generalist embedding models . arXiv preprint arXiv:2405.17428 . Cited by: §2 .
[17] J. Lei, T. L. Berg, and M. Bansal (2021) Detecting moments and highlights in videos via natural language queries . Advances in Neural Information Processing Systems 34 , pp. 11846–11858 . Cited by: Figure 1 , §2 .
[18] H. Li, J. Chen, Z. Wei, S. Huang, T. Hui, J. Gao, X. Wei, and S. Liu (2025) LLaVA-st: a multimodal large language model for fine-grained spatial-temporal understanding . arXiv preprint arXiv:2501.08282 . Cited by: §1 , §2 .
[19] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models . In International conference on machine learning , pp. 19730–19742 . Cited by: §1 , §2 , §3.2 .
[20] J. Li, D. Li, C. Xiong, and S. Hoi (2022) Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation . In International conference on machine learning , pp. 12888–12900 . Cited by: §2 .
[21] K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao (2023) Videochat: chat-centric video understanding . arXiv preprint arXiv:2305.06355 . Cited by: §1 , §2 .
[22] K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, et al. (2024) Mvbench: a comprehensive multi-modal video understanding benchmark . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 22195–22206 . Cited by: Figure 1 .
[23] K. Li, Y. Wang, Y. Li, Y. Wang, Y. He, L. Wang, and Y. Qiao (2023) Unmasked teacher: towards training-efficient video foundation models . 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 19891–19903 . External Links: Link Cited by: §2 .
[24] B. Lin, B. Zhu, Y. Ye, M. Ning, P. Jin, and L. Yuan (2023) Video-llava: learning united visual representation by alignment before projection . arXiv preprint arXiv:2311.10122 . Cited by: §1 , §2 , §5.1 .
[25] S. Lin, C. Lee, M. Shoeybi, J. Lin, B. Catanzaro, and W. Ping (2024) Mm-embed: universal multimodal retrieval with multimodal llms . arXiv preprint arXiv:2411.02571 . Cited by: §2 .
[26] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2024) Visual instruction tuning . Advances in neural information processing systems 36 . Cited by: §1 , §2 , §4.2 .
[27] X. Liu, Q. Wang, Y. Hu, X. Tang, S. Zhang, S. Bai, and X. Bai (2022) End-to-end temporal action detection with transformer . IEEE Transactions on Image Processing 31 , pp. 5427–5441 . Cited by: Appendix G .
[28] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li (2022) Clip4clip: an empirical study of clip for end to end video clip retrieval and captioning . Neurocomputing 508 , pp. 293–304 . Cited by: §2 .
[29] M. Maaz, H. Rasheed, S. Khan, and F. S. Khan (2023) Video-chatgpt: towards detailed video understanding via large vision and language models . arXiv preprint arXiv:2306.05424 . Cited by: §1 , §2 .
[30] W. Moon, S. Hyun, S. Park, D. Park, and J. Heo (2023) Query-dependent video representation for moment retrieval and highlight detection . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 23023–23033 . Cited by: §1 , §2 .
[31] N. Muennighoff, H. Su, L. Wang, N. Yang, F. Wei, T. Yu, A. Singh, and D. Kiela (2024) Generative representational instruction tuning . arXiv preprint arXiv:2402.09906 . Cited by: Appendix A , §1 , §2 , §3.2 .
[32] L. Qian, J. Li, Y. Wu, Y. Ye, H. Fei, T. Chua, Y. Zhuang, and S. Tang (2024) Momentor: advancing video large language model with fine-grained temporal reasoning . In Forty-first International Conference on Machine Learning , External Links: Link Cited by: §2 .
[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision . In International conference on machine learning , pp. 8748–8763 . Cited by: Appendix B , §2 , §3.3 .
[34] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020) Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of machine learning research 21 ( 140 ), pp. 1–67 . Cited by: §1 , §3.1 .
[35] N. Reimers and I. Gurevych (2019) Sentence-bert: sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 3980–3990 . External Links: Document , Link Cited by: §3.2 .
[36] S. Ren, L. Yao, S. Li, X. Sun, and L. Hou (2024) Timechat: a time-sensitive multimodal large language model for long video understanding . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14313–14323 . Cited by: §2 , §5.1 .
[37] M. N. Rizve, F. Fei, J. Unnikrishnan, S. Tran, B. Z. Yao, B. Zeng, M. Shah, and T. Chilimbi (2024) VidLA: video-language alignment at scale . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14043–14055 . Cited by: §1 , §2 , §5.1 , §5.3 .
[38] P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, et al. (2023) Audiopalm: a large language model that can speak and listen . arXiv preprint arXiv:2306.12925 . Cited by: §1 .
[39] H. Sun, M. Zhou, W. Chen, and W. Xie (2024) Tr-detr: task-reciprocal transformer for joint moment retrieval and highlight detection . In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38 , pp. 4998–5007 . Cited by: §2 .
[40] S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, A. Wang, R. Fergus, Y. LeCun, and S. Xie (2024) Cambrian-1: a fully open, vision-centric exploration of multimodal llms . External Links: 2406.16860 , Link Cited by: §4.2 .
[41] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023) Llama: open and efficient foundation language models . arXiv preprint arXiv:2302.13971 . Cited by: §1 .
[42] L. Ventura, A. Yang, C. Schmid, and G. Varol (2024) Covr: learning composed video retrieval from web video captions . In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38 , pp. 5270–5279 . Cited by: §5.3 .
[43] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei (2022) Text embeddings by weakly-supervised contrastive pre-training . arXiv preprint arXiv:2212.03533 . Cited by: §1 , §2 .
[44] X. Wang, J. Wu, J. Chen, L. Li, Y. Wang, and W. Y. Wang (2019) Vatex: a large-scale, high-quality multilingual dataset for video-and-language research . In Proceedings of the IEEE/CVF international conference on computer vision , pp. 4581–4591 . Cited by: Figure 1 .
[45] Y. Wang, K. Li, X. Li, J. Yu, Y. He, G. Chen, B. Pei, R. Zheng, Z. Wang, Y. Shi, et al. (2024) Internvideo2: scaling foundation models for multimodal video understanding . In European Conference on Computer Vision , pp. 396–416 . Cited by: §2 .
[46] Y. Wang, X. Meng, J. Liang, Y. Wang, Q. Liu, and D. Zhao (2024) Hawkeye: training video-text llms for grounding text in videos . arXiv preprint arXiv:2403.10228 . Cited by: §2 .
[47] D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang (2017) Video question answering via gradually refined attention over appearance and motion . In Proceedings of the 25th ACM International Conference on Multimedia , MM ’17 , New York, NY, USA , pp. 1645–1653 . External Links: ISBN 9781450349062 , Link , Document Cited by: Figure 1 .
[48] H. Xu, S. Xie, X. Tan, P. Huang, R. Howes, V. Sharma, S. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer (2024) Demystifying CLIP data . In The Twelfth International Conference on Learning Representations , External Links: Link Cited by: §4.2 .
[49] J. Xu, T. Mei, T. Yao, and Y. Rui (2016) Msr-vtt: a large video description dataset for bridging video and language . In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 5288–5296 . Cited by: Figure 1 .
[50] H. Xue, Y. Sun, B. Liu, J. Fu, R. Song, H. Li, and J. Luo (2023) Clip-vip: adapting pre-trained image-text model to video-language alignment . In The Eleventh International Conference on Learning Representations , Cited by: §2 .
[51] S. Yan, X. Xiong, A. Nagrani, A. Arnab, Z. Wang, W. Ge, D. Ross, and C. Schmid (2023) Unloc: a unified framework for video localization tasks . In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 13623–13633 . Cited by: §2 .
[52] A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid (2023) Vid2seq: large-scale pretraining of a visual language model for dense video captioning . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10714–10726 . Cited by: §2 .
[53] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer (2023) Sigmoid loss for language image pre-training . In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 11975–11986 . Cited by: §1 , §5.3 .
[54] B. Zhang, P. Zhang, X. Dong, Y. Zang, and J. Wang (2025) Long-clip: unlocking the long-text capability of clip . In Computer Vision – ECCV 2024 , A. Leonardis, E. Ricci, S. Roth, O. Russakovsky, T. Sattler, and G. Varol (Eds.) , Cham , pp. 310–325 . External Links: ISBN 978-3-031-72983-6 Cited by: §5.3 .
[55] C. Zhang, J. Wu, and Y. Li (2022) ActionFormer: localizing moments of actions with transformers . In European Conference on Computer Vision (ECCV) , Cited by: Appendix G .
[56] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu (2023) Speechgpt: empowering large language models with intrinsic cross-modal conversational abilities . arXiv preprint arXiv:2305.11000 . Cited by: §1 .
[57] H. Zhang, X. Li, and L. Bing (2023) Video-llama: an instruction-tuned audio-visual language model for video understanding . arXiv preprint arXiv:2306.02858 . Cited by: §1 , §2 .
[58] X. Zhang, Y. Zhang, W. Xie, M. Li, Z. Dai, D. Long, P. Xie, M. Zhang, W. Li, and M. Zhang (2024) GME: improving universal multimodal retrieval by multimodal llms . arXiv preprint arXiv:2412.16855 . Cited by: §2 .
[59] B. Zhu, B. Lin, M. Ning, Y. Yan, J. Cui, W. HongFa, Y. Pang, W. Jiang, J. Zhang, Z. Li, C. W. Zhang, Z. Li, W. Liu, and L. Yuan (2024) LanguageBind: extending video-language pretraining to n-modality by language-based semantic alignment . In The Twelfth International Conference on Learning Representations , External Links: Link Cited by: §2 .
Overview of Supplementary Material
补充材料概览
A. Additional Ablations
A. 补充消融实验
B. Loss Notation and Equations
B. 损失记号与公式
C. Implementation Details
C. 实现细节
D. Broader Impacts
D. 更广泛的影响
E. Privacy Safeguards
E. 隐私保护措施
F. Licensing Information
F. 许可信息
G. Additional Details About Localization Inference
G. 定位推理的补充细节
H. Additional Details About Two-stage retrieval Inference
H. 两阶段检索推理的补充细节
I. Additional Details About Composed Video Retrieval Inference
I. 组合视频检索推理的补充细节
J. Additional Details About Stage 1 joint pre-training
J. 第一阶段联合预训练的补充细节
K. Additional Details About Stage 3 Temporal Localization training
K. 第三阶段时序定位训练的补充细节
L. Details About High Quality Pretraining Dataset Construction: Balancing
L. 高质量预训练数据构建细节:平衡抽样
M. Visualizing High Quality Dataset Statistics
M. 高质量数据统计的可视化
N. Qualitative Captioning Results
N. 定性字幕结果
O. Visualization of Video Embeddings
O. 视频嵌入的可视化
Appendix A Additional Ablations
附录 A 补充消融实验
Number of Pooling Tokens (Table 8) We find that in practice, a relatively high value of (we use 256) is optimal (compare to Q-Former, where usually 32 queries are used). Going beyond 256 to higher values, gains seem minimal.
池化 token 数量(表 8)我们发现,在实践中取一个相对较大的值(我们使用 256)是较优的(可与 Q-Former 通常只用 32 个查询相对照)。再增加到高于 256 时,收益似乎已经很小。
| # | Pre. | VCG | MSR | VTX | QVH |
|---|---|---|---|---|---|
| Ours (256) | × | 3.1 | 49.3 | 45.3 | 32.6 |
| 32 | × | 2.7 | 34.8 | 35.5 | 25.7 |
| 64 | × | 3.0 | 42.5 | 39.2 | 29.5 |
| 128 | × | 3.1 | 48.6 | 44.9 | 32.0 |
| 512 | × | 3.1 | 49.5 | 45.0 | 32.4 |
Table 8: Number of Pooling Tokens
| # | Pre. | VCG | MSR | VTX | QVH |
|---|---|---|---|---|---|
| 本文(256) | × | 3.1 | 49.3 | 45.3 | 32.6 |
| 32 | × | 2.7 | 34.8 | 35.5 | 25.7 |
| 64 | × | 3.0 | 42.5 | 39.2 | 29.5 |
| 128 | × | 3.1 | 48.6 | 44.9 | 32.0 |
| 512 | × | 3.1 | 49.5 | 45.0 | 32.4 |
表 8:池化 token 数量
Number of Embedding Tokens (Table 9) Our video embedding generation outputs a variable number of tokens (until ) to send to the embedding head. Whereas prior encoder approaches (e.g. GRIT [31]) generate a fixed number of tokens, to match this setting and measure the benefit of our dynamic approach we also train the model to use just 1 (or 5) token for embedding generation.
嵌入 token 数量(表 9)我们的视频嵌入生成会输出可变数量的 token(直到 ),再送给嵌入头。先前的编码器方法(如 GRIT [31])则生成固定数量的 token。为了与这一设定对齐,并衡量动态方法的收益,我们也训练模型只用 1 个(或 5 个)token 来生成嵌入。
| # | Pre. | MSR | VCG | MSR | DDM | VTX | ANet | QVH | Ch. |
|---|---|---|---|---|---|---|---|---|---|
| Ours | × | 55.9 | 3.1 | 49.3 | 45.5 | 45.3 | 32.3 | 32.6 | 49.3 |
| 1 | × | 50.2 | 2.6 | 45.8 | 37.6 | 38.9 | 23.7 | 24.1 | 36.8 |
| 5 | × | 54.1 | 2.8 | 47.9 | 40.3 | 41.2 | 27.3 | 27.8 | 42.3 |
Table 9: Number of Embedding Tokens
| # | Pre. | MSR | VCG | MSR | DDM | VTX | ANet | QVH | Ch. |
|---|---|---|---|---|---|---|---|---|---|
| 本文 | × | 55.9 | 3.1 | 49.3 | 45.5 | 45.3 | 32.3 | 32.6 | 49.3 |
| 1 | × | 50.2 | 2.6 | 45.8 | 37.6 | 38.9 | 23.7 | 24.1 | 36.8 |
| 5 | × | 54.1 | 2.8 | 47.9 | 40.3 | 41.2 | 27.3 | 27.8 | 42.3 |
表 9:嵌入 token 数量
Inference Latency vs. Number of Embedding Tokens (Table 10) We measure inference latency on an AWS g6e.8xlarge instance equipped with an NVIDIA L40S GPU. Our adaptive method adds only marginal overhead compared to a fixed 5-token baseline, while achieving substantially better accuracy (Table 9).
推理延迟与嵌入 token 数量(表 10)我们在配备 NVIDIA L40S GPU 的 AWS g6e.8xlarge 实例上测量推理延迟。与固定 5 个 token 的基线相比,我们的自适应方法只增加很少的开销,同时准确率明显更高(表 9)。
| Tokens | Latency (ms) |
|---|---|
| Fixed – 1 | 238 |
| Fixed – 5 | 257 |
| Fixed – 10 | 326 |
| Adaptive (Ours) | 262 |
Table 10: Inference latency vs. number of embedding tokens for 12-frames (batch size 16) on NVIDIA L40S.
| Token 数 | 延迟(毫秒) |
|---|---|
| 固定 – 1 | 238 |
| 固定 – 5 | 257 |
| 固定 – 10 | 326 |
| 自适应(本文) | 262 |
表 10:在 NVIDIA L40S 上,12 帧、批量大小为 16 时,推理延迟与嵌入 token 数量的关系。
Adaptive Tokens Across Video Durations (Table 11) To further analyze the adaptive token mechanism, we break down the ActivityNet Captions dataset by video duration. This dataset has a wide and nearly uniform distribution of videos up to 5 minutes, providing balanced bins. Our adaptive method generalizes better to variable video lengths, significantly outperforming fixed-token approaches across all duration bins.
不同视频时长上的自适应 token(表 11)为进一步分析自适应 token 机制,我们按视频时长对 ActivityNet Captions 数据集做了拆分。该数据集中最长约 5 分钟的视频分布较宽且近乎均匀,因而各时长区间较为均衡。我们的自适应方法对可变视频长度的泛化更好,在所有时长区间上都明显超过固定 token 方法。
| Method | 0–60s | 60–120s | 120–180s | 180s+ | Overall |
|---|---|---|---|---|---|
| VidLA Baseline | 65.7 | 66.5 | 65.0 | 63.5 | 65.2 |
| Fixed Token = 1 | 66.2 | 64.7 | 63.5 | 62.2 | 64.1 |
| Fixed Tokens = 5 | 66.6 | 67.4 | 66.4 | 65.4 | 66.5 |
| Adaptive (Ours) | 67.9 | 68.5 | 67.8 | 67.1 | 67.9 |
Table 11: ActivityNet Captions Text-to-Video Retrieval (R@1) broken down by video duration.
| 方法 | 0–60秒 | 60–120秒 | 120–180秒 | 180秒以上 | 总体 |
|---|---|---|---|---|---|
| VidLA 基线 | 65.7 | 66.5 | 65.0 | 63.5 | 65.2 |
| 固定 token = 1 | 66.2 | 64.7 | 63.5 | 62.2 | 64.1 |
| 固定 token = 5 | 66.6 | 67.4 | 66.4 | 65.4 | 66.5 |
| 自适应(本文) | 67.9 | 68.5 | 67.8 | 67.1 | 67.9 |
表 11:ActivityNet Captions 文本到视频检索(R@1)按视频时长分箱。
To verify that the model actually leverages variable-length generation, we measure the average number of tokens produced by the adaptive method per duration bin: 0–60s: 8.6 tokens; 60–120s: 10.1 tokens; 120–180s: 11.5 tokens; 180s+: 14.8 tokens. This confirms that the model naturally “thinks longer” for more complex, longer videos.
为确认模型确实在利用变长生成,我们测量了自适应方法在各时长区间上产生的平均 token 数:0–60秒:8.6 个;60–120秒:10.1 个;120–180秒:11.5 个;180秒以上:14.8 个。这证实模型会对更复杂、更长的视频自然地“多想一会儿”。
Appendix B Shared Notation
附录 B 共用记号
For clarity we reserve a single symbol for each recurring concept. An entire video is denoted by a capital letter, , whereas a temporal segment cut from that video is represented in lowercase, . Hard-negative segments for localization are denoted by , where indexes the negatives for sample . Any natural-language sequence—be it a caption, a question, or an answer—is denoted by ; its -th token is , and the total length is for captions or for answers. The LLM maps raw inputs to embeddings, producing , and , representing our model and represents the mode token, e.g. |VID_EMB| is the video embedding token. All contrastive objectives use the same similarity function (cosine by default) scaled by a temperature parameter . We use a mini-batch of size .
为清楚起见,我们对每个反复出现的概念只保留一个符号。整段视频用大写字母 表示,从该视频中切出的时间片段用小写 表示。用于定位的困难负例片段记为 ,其中 为样本 的负例编号。任何自然语言序列——不论是字幕、问题还是答案——都记为 ;它的第 个 token 是 ,总长度对字幕为 ,对答案为 。LLM 把原始输入映射为嵌入,得到 、 和 ,其中 表示我们的模型, 表示模式 token,例如 |VID_EMB| 是视频嵌入 token。所有对比目标使用相同的相似度函数 (默认是余弦),并由温度参数 缩放。我们使用大小为 的小批量。
Text–Video Retrieval (Fig. 3a). This task aligns video and text embeddings using CLIP-like [33] contrastive loss.
文本-视频检索(图 3a)。 该任务用类似 CLIP [33] 的对比损失对齐视频嵌入和文本嵌入。
Video Captioning & Question–Answering (Fig. 3b). The goal of captioning is for the model to generate accurate descriptions of videos. The model learns to predict each token in the caption, given the previous tokens and the video. Whereas for QA, the model generates answers to questions about the video. The question serves as part of the prompt, guiding the model’s attention to answer accurately.The next-token prediction loss is used.
视频字幕与问答(图 3b)。 字幕任务的目标是让模型生成对视频的准确描述。模型在给定已生成词和视频的条件下,学习预测字幕中的每一个 token。问答任务中,模型为关于视频的问题生成答案。问题作为提示的一部分,引导模型把注意力放到准确作答上。使用的是下一词预测损失。
Text–Video Matching (Fig. 3c). As a complement to the retrieval task we also formulate a matching task where the model is trained to identify if a given video and caption form a matching pair. Binary decisions are supervised by the label .
文本-视频匹配(图 3c)。 作为检索任务的补充,我们还设置了匹配任务:训练模型判断给定视频和字幕是否构成匹配对。二分类决策由标签 监督。
Temporal Localization (Fig. 3d). Localization requires the model to match specific video segments to text annotations while distinguishing similar but incorrect segments. Sliding window hard negative mining provides challenging non-matching segments with limited temporal overlap (IoU 0.2), helping the model learn to differentiate closely related content. Let (positive) and (j-th hard negative for sample i, ; IoU ). The localization loss, , can be written as
时序定位(图 3d)。 定位要求模型把特定视频片段与文本标注匹配起来,同时区分相近但不正确的片段。我们用滑窗挖掘困难负例,提供时间重叠有限(IoU 0.2)的非匹配片段,帮助模型学会区分相近内容。记 (正例),(样本 i 的第 j 个困难负例,;IoU )。定位损失 可以写成
Appendix C Implementation Details and Hyperparameters
附录 C 实现细节与超参数
We use PaliGemma as our primary MLLM backbone. PaliGemma is openly licensed (Apache 2.0) and is based on the Gemma-2B language model and uses SigLIP-SO-400M ViT as its vision encoder. The single layer embedding head has the same dimensions as a Gemma-2B layer (hidden_dim = 2048, mlp_dim = 16384, and n_heads = 8). All video frames are resized to 224 224 before feeding into the model. LoRA adapters of rank 128 were used for finetuning and are applied to all linear layers of the LLM transformer (i.e., excluding the vision encoder, embedding layers, and output heads). PaliGemma’s and tokens are leveraged to initialize and . A modified Huggingface Trainer is used for training, with FSDP mode used across GPUs to reduce usage. Training hyperparameters for each stage are provided in Table 12. The key reason behind these choices is improving the efficiency of the large scale Stage 1 training. The learning rates are chosen on the basis of small training runs on the MSR-VTT dataset. Inference latency is benchmarked on an AWS g6e.8xlarge instance with an NVIDIA L40S GPU. For temporal localization, we use a sliding window of 10 seconds with a stride of 5 seconds (see Appendix G).
我们以 PaliGemma 作为主要的多模态大语言模型骨干。PaliGemma 采用开放许可(Apache 2.0),语言部分基于 Gemma-2B,视觉编码器使用 SigLIP-SO-400M ViT。单层嵌入头的维度与 Gemma-2B 的一层相同(hidden_dim = 2048,mlp_dim = 16384,n_heads = 8)。所有视频帧在送入模型前被缩放到 224 224。微调使用秩为 128 的 LoRA 适配器,并加在 LLM Transformer 的全部线性层上(即不包括视觉编码器、嵌入层和输出头)。PaliGemma 的 和 token 被用来初始化 和 。训练使用经过修改的 Huggingface Trainer,并在各 GPU 上启用 FSDP 模式以降低 占用。各阶段的训练超参数见表 12。这些选择的主要目的,是提高大规模第一阶段训练的效率。学习率是根据在 MSR-VTT 上的小规模试跑选定的。推理延迟在配备 NVIDIA L40S GPU 的 AWS g6e.8xlarge 实例上测定。时序定位使用 10 秒滑窗、步长 5 秒(见附录 G)。
Multi-Task Finetuning Batch Creation: Following Table 2, each video in our dataset has 4 boolean flags indicating which training tasks are supported for the video. When a random batch of videos is sampled, the flags are sampled with them. The flags are used to filter a partial task specific minibatch, with one minibatch per task. Gradients are accumulated across all training tasks before updating the model weights.
多任务微调的批次构造:按表 2,数据集中每条视频有 4 个布尔标记,表明该视频支持哪些训练任务。随机抽取一批视频时,这些标记一并被抽到。标记用于过滤出按任务划分的部分小批量,每个任务一个小批量。各训练任务的梯度先累积,再更新模型权重。
| Hyperparameter | Stage 1/2 | Stage 3 |
|---|---|---|
| Number of GPUs | 32 | 8 |
| Batch Size (per GPU) | 12 | 8 |
| Gradient Accumulation | 2 | 4 |
| Optimizer | Lion | AdamW |
| Base Learning Rate | 1e-5 | 5e-6 |
| Warmup Steps | 2000 | 1000 |
| Weight Decay∗ | 0.00001 | - |
| LoRA Rank | 128 | 32 |
| Training Steps | 25,000 | 4,000 |
| Max Token Length | 2304/4096 | 4096 |
Table 12: Training hyperparameters across different stages. ∗- only applied to new embedding head.
| 超参数 | 阶段 1/2 | 阶段 3 |
|---|---|---|
| GPU 数量 | 32 | 8 |
| 批量大小(每 GPU) | 12 | 8 |
| 梯度累积 | 2 | 4 |
| 优化器 | Lion | AdamW |
| 基础学习率 | 1e-5 | 5e-6 |
| 预热步数 | 2000 | 1000 |
| 权重衰减∗ | 0.00001 | - |
| LoRA 秩 | 128 | 32 |
| 训练步数 | 25,000 | 4,000 |
| 最大 token 长度 | 2304/4096 | 4096 |
表 12:各阶段的训练超参数。∗ 仅作用于新的嵌入头。
Appendix D Broader Impacts
附录 D 更广泛的影响
A key finding of our work is the ability to unify generative and embedding models, which could potentially reduce the amount of energy used and environmental impact from maintaining two different model pipelines for inference in practical applications. In terms of potential negative impact, our model is very general, and hence shares some of the potential for misuse inherent to generalist computer vision models.
本文的一个关键发现是可以把生成模型与嵌入模型统一起来,这在实际应用的推理中有可能减少维护两套模型流水线所带来的能耗和环境影响。在潜在的负面影响方面,我们的模型相当通用,因此也带有通用计算机视觉模型固有的一些被滥用风险。
Appendix E Privacy Safeguards
附录 E 隐私保护措施
All videos are stored and used with the faces of humans blurred using the deface python package.
所有视频在存储和使用时,都用 deface 这个 Python 软件包对人脸做了模糊处理。
Appendix F Licenses
附录 F 许可
WebVid-10M (Shutterstock clips). A proprietary corpus governed by Shutterstock’s Terms of Service; URLs and captions may be downloaded for internal, non-commercial research only, and redistribution is forbidden.
WebVid-10M(Shutterstock 片段)。 受 Shutterstock 服务条款约束的专有语料;URL 和字幕仅可下载用于内部、非商业研究,禁止再分发。
MSR-VTT. Released by Microsoft Research solely for academic research; no explicit open-source license is attached, so usage is restricted to the terms on the dataset website/publication.
MSR-VTT。 由微软研究院仅供学术研究发布;没有附加明确的开源许可,因此使用须遵守数据集网站/论文中的条款。
ActivityNet. Distributed under the MIT License, permitting commercial and non-commercial use provided that copyright and license notices are retained.
ActivityNet。 以 MIT 许可证分发,允许商业和非商业使用,但须保留版权与许可声明。
DiDeMo. Available under the BSD 2-Clause license, a permissive license that allows modification and redistribution with minimal obligations.
DiDeMo。 采用 BSD 2-Clause 许可证,这是一种宽松许可,允许修改和再分发,义务较少。
VATEX. Shared under Creative Commons Attribution 4.0 International (CC BY 4.0); free use with attribution.
VATEX。 以知识共享署名 4.0 国际许可(CC BY 4.0)共享;可自由使用,但须署名。
QVHighlights. Released with a Creative Commons Attribution–NonCommercial–ShareAlike 4.0 license (CC BY-NC-SA 4.0); commercial use is prohibited and derivatives must adopt the same license.
QVHighlights。 以知识共享署名-非商业性使用-相同方式共享 4.0 许可(CC BY-NC-SA 4.0)发布;禁止商业使用,演绎作品须采用相同许可。
Charades & Charades-STA. Allen AI provides these datasets under a non-commercial research license; any for-profit use requires separate permission.
Charades 与 Charades-STA。 Allen AI 以非商业研究许可提供这些数据集;任何营利性使用都需要另行许可。
SigLIP-SO-400M visual encoder. Model weights and code are released under the Apache License 2.0.
SigLIP-SO-400M 视觉编码器。 模型权重和代码以 Apache License 2.0 发布。
PaliGemma-3B multimodal LLM. Weights are covered by Google’s Gemma Open Model License 1.0; accompanying reference code is Apache 2.0.
PaliGemma-3B 多模态大语言模型。 权重受 Google 的 Gemma Open Model License 1.0 约束;随附的参考代码为 Apache 2.0。
Claude-3 Sonnet (caption generator). A proprietary model whose use is governed by Anthropic’s Terms of Service.
Claude-3 Sonnet(字幕生成器)。 专有模型,使用受 Anthropic 服务条款约束。
Appendix G Localization Inference
附录 G 定位推理
We apply post-processing inspired from prior embedding-based localization approaches to merge segments.
我们采用受先前基于嵌入的定位方法启发的后处理,来合并片段。
We split the video into non-overlapping windows and compute a text-video similarity score for each segment. We take the highest-scoring window (with score ) as the seed, then merge adjacent windows to the left and right as long as each neighbor’s score exceeds a merge threshold () or is at least a fixed ratio (i.e., ) of the seed’s score. Formally, a neighboring window is merged if
我们把视频切成互不重叠的窗口,并为每个片段计算文本-视频相似度分数。取分数最高的窗口(分数为 )作为种子,然后向左右合并相邻窗口,只要每个邻居的分数超过合并阈值(),或至少达到种子分数的一个固定比例(即 )。形式上,相邻窗口 在满足下式时被合并
Expansion stops at the first neighbor on either side that fails both conditions. The final prediction is the union of all accepted windows. Center correction is applied to the start and end clips if they are not the seed clip.
扩展在左右两侧第一个同时不满足两个条件的邻居处停止。最终预测是所有被接受窗口的并集。若起始和结束片段不是种子片段,则对其做中心校正。
Some prior works that we referred to in order to develop our post-processing strategy include [55, 27].
我们在设计后处理策略时参考的部分先前工作包括 [55, 27]。
We ablate some of the choices related to our localization inference protocol here:
这里对定位推理协议中的若干选择做了消融:
| Window | Stride | IoU | R@1, IoU=0.5 |
|---|---|---|---|
| 5s | 5s | 0.3 | 39.0 |
| 5s | 5s | 0.4 | 39.4 |
| 5s | 5s | 0.5 | 35.7 |
| 10s | 10s | 0.4 | 39.2 |
| 10s | 5s | 0.4 | 41.7 |
| 15s | 10s | 0.4 | 39.9 |
Table 13: Ablation of localization inference protocol choices. Score on ActivityNet Dataset.
| 窗口 | 步长 | IoU | R@1, IoU=0.5 |
|---|---|---|---|
| 5s | 5s | 0.3 | 39.0 |
| 5s | 5s | 0.4 | 39.4 |
| 5s | 5s | 0.5 | 35.7 |
| 10s | 10s | 0.4 | 39.2 |
| 10s | 5s | 0.4 | 41.7 |
| 15s | 10s | 0.4 | 39.9 |
表 13:定位推理协议选择的消融。分数来自 ActivityNet 数据集。
Appendix H Matching Based Two Stage Retrieval
附录 H 基于匹配的两阶段检索
We provide more details of how we enhance video-caption retrieval performance by combining retrieval techniques with advanced matching capabilities of a VideoLLM. Below, we describe the steps for each stage:
我们更详细地说明,如何把检索技术与 VideoLLM 更强的匹配能力结合起来,以提升视频-字幕检索表现。下面分阶段描述各步骤:
Stage 1: Embedding-Based Retrieval
第一阶段:基于嵌入的检索
- Embedding Generation: • Text embeddings for the input caption are generated using ViLL-E in text embedding mode. • Video embeddings for all videos in the database are pre-computed using ViLL-E in video embedding mode.
- 嵌入生成: • 输入字幕的文本嵌入由 ViLL-E 在文本嵌入模式下生成。• 数据库中全部视频的视频嵌入预先由 ViLL-E 在视频嵌入模式下算好。
- Similarity Computation: • The similarity between the caption embedding and each video embedding is computed, typically using cosine similarity.
- 相似度计算: • 计算字幕嵌入与每个视频嵌入之间的相似度,通常使用余弦相似度。
- Top- Retrieval: • The most similar video candidates are selected based on the similarity scores. This reduces the search space for the more computationally intensive second stage.
- 取回前 个: • 根据相似度分数选出最相似的 个候选视频。这样可以为计算更重的第二阶段缩小搜索范围。
图 5:两步检索推理。
Stage 2: Matching-Based Re-Ranking
第二阶段:基于匹配的重排
- Video-Caption Pair Processing: • Each video in the Top- set is paired with the input caption and fed into the VideoLLM. • A carefully designed prompt is used to guide the VideoLLM in determining whether the pair is a match.
- 视频-字幕对处理: • 把前 个集合中的每个视频与输入字幕配对,送入 VideoLLM。• 使用精心设计的提示,引导 VideoLLM 判断该对是否匹配。
- Prediction and Scoring: • For each video-caption pair, the VideoLLM predicts a likelihood score for both Yes (match) and No (no match) tokens. • A matching score is computed from the logits of these predictions (e.g., softmax probabilities or logit differences).
- 预测与计分: • 对每个视频-字幕对,VideoLLM 分别预测 Yes(匹配)和 No(不匹配)两个 token 的似然分数。• 再由这些预测的 logit 计算匹配分数(例如 softmax 概率或 logit 之差)。
- Re-ranking: • The Top- () videos are re-ranked based on their matching scores, with higher scores indicating stronger matches.
- 重排: • 按匹配分数对前 个()视频重新排序,分数越高表示匹配越强。
Benefits of the Two-Stage Approach
两阶段方法的好处
- • Efficiency: By leveraging computationally efficient embedding-based retrieval in the first stage, the system avoids evaluating all database videos
- • 效率: 第一阶段利用计算上更高效的基于嵌入的检索,系统不必评估数据库中的全部视频
- • Accuracy: The VideoLLM’s reasoning capabilities in the second stage allow it to resolve ambiguities and refine rankings, leading to a significant improvement in retrieval accuracy (e.g., increase in R@1 accuracy).
- • 准确率: 第二阶段借助 VideoLLM 的推理能力消解歧义并细化排序,从而使检索准确率有明显提升(例如 R@1 大约提高 )。
- • Scalability: The division into two stages ensures that the system can scale to large databases without compromising on retrieval quality.
- • 可扩展性: 分成两个阶段,使系统能够扩展到大型数据库,同时不牺牲检索质量。
Prompt for Matching
用于匹配的提示
The prompt provided to the VideoLLM for second-stage matching is structured as follows:
提供给 VideoLLM 做第二阶段匹配的提示结构如下:
- • Input: Caption: [Retrieved first stage caption]. Question: Does the above video match the caption? Answer "Yes" or "No".
- • 输入: Caption: [Retrieved first stage caption]. Question: Does the above video match the caption? Answer "Yes" or "No".
- • Output: – Token logits for Yes and No, which are further used to compute the matching score.
- • 输出: – Yes 和 No 的 token logit,再用来计算匹配分数。
Appendix I Composed Video Retrieval
附录 I 组合视频检索
As illustrated in Fig. 6, inference in the composed-video-retrieval (CoVR) pipeline begins by encoding the source clip with a spatio-temporal vision encoder whose output tokens are linearly projected into the hidden dimension of the downstream large-language-model (LLM) transformer. A single text-edit token, |TXT_EMB|, embeds the user’s natural-language instruction and is concatenated to the projected visual tokens, enabling early fusion of visual context and linguistic modification. The resulting sequence is processed by several frozen LLM transformer layers, whose final hidden states are pooled to yield a compact query representation that jointly reflects the appearance of the source clip and the requested transformation. In parallel, every clip in the retrieval gallery is independently passed through the same vision-encoder projection LLM steps—without the text token—to obtain target embeddings via a lightweight embedding head. At retrieval time, cosine (or dot-product) similarity is computed between the composed query embedding and all target embeddings, and the gallery video with maximal similarity is returned as the predicted target.
如图 6 所示,组合视频检索(CoVR)流程的推理首先用时空视觉编码器对源片段编码,再将其输出 token 线性投影到下游大语言模型(LLM)Transformer 的隐藏维度。一个单独的文本编辑 token |TXT_EMB| 嵌入用户的自然语言指令,并与投影后的视觉 token 拼接,从而实现视觉语境与语言修改的早期融合。得到的序列经过若干冻结的 LLM Transformer 层,其最后隐状态再被池化,形成一个紧凑的查询表征,同时反映源片段的外观和所请求的变换。与此并行,检索库中的每个片段都独立经过相同的视觉编码器 投影 LLM 步骤——但不加文本 token——再经一个轻量嵌入头得到目标嵌入。检索时,在组合查询嵌入与所有目标嵌入之间计算余弦(或点积)相似度,并把相似度最高的库内视频作为预测目标返回。
图 6:组合视频检索推理。
Appendix J Detailed Overview of Stage 1 Pretraining
附录 J 第一阶段预训练的详细说明
During Stage 1 joint pre-training (Fig. 9) the model is optimized with both a causal-language-model (LM) objective and a video–text contrastive objective, each applied in a separate forward pass that nevertheless shares parameters. First, a batch of videos is fed through a frozen spatio-temporal vision encoder; its output tokens are linearly projected into the hidden dimension of a large language model (LLM) that has been lightweight-fine-tuned via LoRA adapters (modules marked with fire symbol). These visual tokens are prepended to the textual prompt “ … Describe this video.” and processed by the LLM. Two heads branch from the final hidden states: (i) a language-modeling head generates a caption autoregressively, whose cross-entropy with the ground-truth caption constitutes the language-modeling loss; and (ii) an embedding head pools the hidden states to yield a fixed-length video embedding.
在第一阶段联合预训练中(图 9),模型同时用因果语言模型(LM)目标和视频-文本对比目标进行优化,两者分别在两次前向传播中施加,但共享参数。首先,一批视频送入冻结的时空视觉编码器;其输出 token 被线性投影到已用 LoRA 适配器(图中带火焰标记的模块)做轻量微调的大语言模型(LLM)的隐藏维度。这些视觉 token 被放在文本提示 “ … Describe this video.” 之前,再由 LLM 处理。最后隐状态分出两个头:(i)语言模型头自回归生成字幕,其与真实字幕的交叉熵构成语言模型损失;(ii)嵌入头对隐状态做池化,得到固定长度的视频嵌入。
In the second forward pass, only caption strings are supplied—again through the same LLM + LoRA stack, ensuring weight sharing between modalities—and their hidden states are mapped by the embedding head to caption embeddings. The embedding corresponding to the correct caption serves as the positive partner for the video embedding, while captions of the remaining videos in the batch act as negatives. A standard InfoNCE objective then enforces contrastive alignment between the video embedding and its positive caption while repelling mismatched pairs.
第二次前向传播只提供字幕字符串——仍然经过同一套 LLM + LoRA,以保证模态之间共享权重——其隐状态再由嵌入头映射为字幕嵌入。与视频对应的正确字幕嵌入作为正例伙伴,批次中其余视频的字幕则作为负例。随后用标准 InfoNCE 目标,在视频嵌入与其正例字幕之间做对比对齐,并推开不匹配的配对。
The two losses are summed, so the model simultaneously learns to generate fluent video descriptions and to discriminate between semantically consistent versus inconsistent video–text pairs, yielding representations that are effective for both captioning and downstream retrieval tasks.
两项损失相加,因此模型同时学会生成流畅的视频描述,并区分语义一致与不一致的视频-文本对,从而得到对字幕生成和下游检索都有效的表征。
Appendix K Detailed Overview of Stage 3: Temporal localization training
附录 K 第三阶段时序定位训练的详细说明
As depicted in Fig. 10, Stage 3 (Temporal Localization Task) fine-tunes the model for temporal localization by contrasting text queries with candidate video clips extracted from long-form footage**. Given an annotated clip-level caption (e.g., “playing with glow slime in the dark”), a small set of sparsely sampled frames from the entire video is first prepended to the caption prompt to furnish coarse visual context, after which the sequence is routed through the LoRA-adapted LLM and pooled by the embedding head to yield a contextualized text embedding. In parallel, the full video is partitioned with a fixed-stride sliding window; each windowed segment is passed through the same vision-encoder projection LLM embedding-head stack to obtain video-clip embeddings. The clip whose temporal bounds coincide with the ground-truth annotation is designated the positive match, while all other windowed segments within the batch serve as hard negatives. A contrastive InfoNCE loss is then applied between the text embedding and the set of clip embeddings, driving the model to maximise similarity with the positive segment and to minimise similarity with temporally misaligned negatives. This procedure equips the shared embedding space with fine-grained temporal sensitivity, enabling the model to accurately localise textual events within extended, untrimmed videos without requiring explicit frame-level supervision.
如图 10 所示,第三阶段(时序定位任务)通过对文本查询与从长视频中切出的候选片段做对比,来微调模型的时序定位能力。给定一条片段级标注字幕(例如 “playing with glow slime in the dark”),先从整段视频中稀疏采样少量帧,附加到字幕提示前,以提供粗略的视觉上下文,再把该序列送入经过 LoRA 适配的 LLM,并由嵌入头池化,得到上下文相关的文本嵌入。与此并行,整段视频按固定步长的滑窗划分;每个窗口片段都经过同一套视觉编码器 投影 LLM 嵌入头,得到视频片段嵌入。时间范围与真实标注重合的片段被指定为正例匹配,批次内其他窗口片段则作为困难负例。随后在文本嵌入与一组片段嵌入之间施加对比 InfoNCE 损失,推动模型提高与正例片段的相似度、降低与时间不对齐负例的相似度。这一过程使共享嵌入空间具备细粒度的时间敏感性,从而能在加长、未经剪辑的视频中准确定位文本事件,而无需显式的帧级监督。
Appendix L Intermediate Pre-training Data Balancing
附录 L 中间预训练数据的平衡
The Shutterstock dataset is extensively labelled with keywords, which we use to select a balanced set of videos for our intermediate pre-training dataset. Keywords with fewer than 30 occurrences are excluded, and up to 500 videos per remaining keyword are added to the candidate pool, starting with the most frequent keywords, to ensure a balanced and representative selection. The original frequency distribution of keywords, and the final achieved distribution are shown in Fig. 7. Note that as each video has multiple keywords, this is not a proper probability distribution.
Shutterstock 数据集带有大量关键词标注,我们用它们为中间预训练集选出一组平衡的视频。出现次数少于 30 的关键词被排除;从剩余关键词中频率最高的开始,每个关键词最多向候选池加入 500 条视频,以保证选择既平衡又有代表性。关键词的原始频率分布以及最终得到的分布见图 7。需要注意的是,由于每条视频有多个关键词,这并不是一个严格的概率分布。
图 7:我们通过对原数据集中的视频做子抽样、下调常见概念的比例,来保证数据集中的概念平衡。纵轴为对数坐标。
Appendix M High Quality Data Statistics
附录 M 高质量数据统计
Our high quality recaptioning of a subset of shutterstock videos significantly increases the level of detail in the captions from having <10 words per caption on average to having 130+ words on average per caption.
我们对 Shutterstock 视频子集做高质量重新配字幕后,字幕的细致程度显著提高:平均每条由不足 10 个词增加到 130 个词以上。
图 8:我们的字幕显著提高了细致程度。蓝色为原始字幕,红色为生成字幕。
Figure 9: Stage 1 Training. Joint Generative and Contrastive Pre-Training. This stage requires 2 forward passes of the model per step. In the first step, the videos along with prompt are passed to the model and the video caption and video embedding are generated simultaneously. In the second forward pass, only the captions are passed to generate caption embeddings. The ground-truth caption for the video serves as the positive match, whereas the captions for the other videos in the batch are negatives for the contrastive loss.
图 9:第一阶段训练。联合生成与对比预训练。该阶段每一步需要对模型做两次前向传播。第一步把视频连同提示送入模型,同时生成视频字幕和视频嵌入。第二次前向传播只送入字幕,以生成字幕嵌入。该视频的真实字幕作为正例匹配,批次中其他视频的字幕则作为对比损失的负例。
Figure 10: Stage 3: Temporal Localization Task Training. Contextualized Text Embeddings (Left Half): Text embedding are generated based on text input such as “Playing with glow slime in the dark.” A selection of sparse frames from the long video is sampled and added to the prompt to provide context. Sliding Window Video Clip Embeddings (Right Half): The long video is divided into clips via a sliding window mechanism, which extracts several short clips. Video embeddings for the clips are used to create the positive and hard negative matches for the text, based on the annotations from the dataset. Training is done using the contrastive loss.
图 10:第三阶段:时序定位任务训练。上下文相关的文本嵌入(左半):文本嵌入根据诸如 “Playing with glow slime in the dark.” 这样的文本输入生成。从长视频中采样一组稀疏帧并加入提示,以提供上下文。滑窗视频片段嵌入(右半):长视频通过滑窗机制被切成若干短片段。片段的视频嵌入根据数据集中的标注,为文本构造正例和困难负例匹配。训练使用对比损失。
Appendix N Qualitative Video Captioning Results
附录 N 定性视频字幕结果
ViLL-E is able to generate rich video captions exceeding even the quality of some ground-truth captions in datasets. Some examples of this on the MSR-VTT dataset are provided in Figure 11. The figures compare ground-truth descriptions with those generated by the ViLL-E system, highlighting its ability to enrich scene interpretations. For a rocket launch, the ground-truth describes a simple ascent with smoke at the base, while ViLL-E adds detail about a vapor trail over a forested area. In a depiction of miniature donkeys, the ground-truth focuses on basic movement and noise, whereas ViLL-E elaborates on their interactions, feeding, and social behavior on a farm. Similarly, a scene of cartoon birds flying is expanded by ViLL-E into a vivid portrayal of a large flock creating intricate patterns over a mountainous landscape. These comparisons underscore ViLL-E’s capacity to provide more nuanced and context-rich scene descriptions.
ViLL-E 能够生成内容丰富的视频字幕,细致程度甚至超过数据集中的部分真实字幕。图 11 给出了在 MSR-VTT 数据集上的若干例子。这些图把真实描述与 ViLL-E 生成的字幕加以对照,突出了它丰富场景解释的能力。对于火箭发射,真实字幕只写了底部带烟的简单升空,而 ViLL-E 补充了林地上空的蒸汽尾迹。在微型驴的画面中,真实字幕主要写动作和叫声,ViLL-E 则进一步写到它们在农场上的互动、进食和社交行为。类似地,卡通鸟飞翔的场景被 ViLL-E 扩展为一大群鸟在山地景观上形成复杂图案的生动描写。这些对照表明,ViLL-E 能够提供更细致、更有语境的场景描述。
图 11:MSR-VTT 上的定性视频字幕结果
Appendix O Visualization of Video Embeddings
附录 O 视频嵌入的可视化
Text and Video embeddings for video from a held out val set for ShutterStock are visualized after being reduced to two dimensions with Barnes-Hut t-SNE (perplexity = 30, init = PCA), producing two coordinates for every video–caption pair while approximately maintaining local neighbourhood structure. To aid qualitative inspection, we cluster the caption embeddings with K-means (k = 20). The dominant, non-stop-word within each cluster’s captions is used as an interpretable label (e.g., “beach”, “office”, “smiling”). The visualization is rendered as a 4 by 5 grid: each panel isolates one cluster, plotting video embeddings as filled circles and caption embeddings as crosses. Grey lines connect every video point to its paired caption point, revealing intra-cluster fine-grained alignment patterns. The resulting figure offers an intuitive map for understanding the retrieval performance of our model across various domains; a high number of intersecting lines indicates poor performance in that cluster and vice versa. Our model exhibits strong performance on videos with concepts like “couple”, “flying”, “summer” etc.
我们对 Shutterstock 一个留出验证集中的视频,将其文本嵌入和视频嵌入用 Barnes-Hut t-SNE(perplexity = 30,init = PCA)降到二维后加以可视化,为每一对视频-字幕得到两个坐标,同时大致保持局部邻域结构。为便于定性观察,我们用 K-means(k = 20)对字幕嵌入聚类。每个簇的字幕中占主导、且非停用词的词被用作可解释标签(例如 “beach”“office”“smiling”)。可视化排成 4×5 的网格:每个子图单独展示一个簇,视频嵌入画成实心圆,字幕嵌入画成叉号。灰色线段把每个视频点与其配对的字幕点连起来,以揭示簇内更细的对齐模式。得到的图可以直观地反映模型在不同领域上的检索表现:交叉线段越多,说明该簇上的表现越差,反之亦然。我们的模型在带有 “couple”“flying”“summer” 这类概念的视频上表现较强。
图 12:展示我们模型的检索嵌入,以及它们在不同视频子领域中的对齐情况











