LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation
LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation
LLM2CLIP：强大的语言模型释放更丰富的跨模态表征
Abstract
摘要
CLIP is a seminal multimodal model that maps images and text into a shared representation space by contrastive learning on billions of image–caption pairs. Inspired by the rapid progress of large language models (LLMs), we investigate how the superior linguistic understanding and broad world knowledge of LLMs can further strengthen CLIP—particularly in handling long, complex captions. We introduce an efficient fine-tuning framework that embeds an LLM into a pretrained CLIP while incurring almost the same training cost as regular CLIP fine-tuning. Our method first “embedding-izes” the LLM for the CLIP setting, then couples it to the pretrained CLIP vision encoder through a lightweight adaptor trained on only a few million image–caption pairs. With this strategy we achieve large performance gains—without large-scale retraining—over state-of-the-art CLIP variants such as EVA02 and SigLIP-2. The LLM-enhanced CLIP delivers consistent improvements across a wide spectrum of downstream tasks, including linear-probe classification, zero-shot image–text retrieval with both short and long captions (in English and other languages), zero-shot/supervised image segmentation, object detection, and used as tokenizer for multimodal large-model benchmarks. Code & Models: https://aka.ms/llm2clip.
CLIP 是一项奠基性的多模态工作：它在数十亿图文对上做对比学习，把图像和文本映射到同一表征空间。受大语言模型（LLM）快速进展的启发，我们考察如何把 LLM 更强的语言理解和更广的世界知识注入 CLIP，尤其是提升它对长而复杂标题的处理能力。我们提出一套高效微调框架，把 LLM 接入已经预训练好的 CLIP，训练开销几乎与普通 CLIP 微调相当。方法先按 CLIP 的设定把 LLM「嵌入化」，再通过一个轻量适配器，在仅数百万图文对上把它接到预训练视觉编码器。不必做大规模重训，就能在 EVA02、SigLIP-2 等先进 CLIP 变体上取得明显提升。增强后的 CLIP 在一系列下游任务上表现更稳，包括线性探测分类、英文及其他语言的短标题与长标题零样本图文检索、零样本及监督式图像分割、目标检测，以及作为多模态大模型评测中的分词器（tokenizer）。代码与模型： https://aka.ms/llm2clip。
1 Introduction
1 引言
Figure 1: LLM2CLIP Overview. After applying caption contrastive fine-tuning to the LLM, the increased textual discriminability enables more effective CLIP training. We leverage the open-world knowledge and general capabilities of the LLM to better process dense captions, addressing the previous limitations of the pretrained CLIP visual encoder and providing richer textual supervision.
图 1：LLM2CLIP 方法总览。对大语言模型做标题对比微调后，文本可分性提高，有助于更有效地训练 CLIP。我们借助 LLM 的开放世界知识和通用能力，更好地处理稠密标题，弥补此前预训练 CLIP 视觉编码器的不足，并提供更丰富的文本监督。
CLIP (Radford et al. 2021; Tschannen et al. 2025) has emerged as one of the most influential cross-modal foundation models in recent years. Trained on hundreds of millions to tens of billions of image–text pairs via contrastive pre-training, it embeds vision and language into a shared representation space. As a retriever, CLIP underpins zero-shot classification, detection, segmentation, and—most notably—image–text retrieval. As a feature extractor, it supports a broad spectrum of cross-modal applications, from image/video understanding to text-to-image and text-to-video generation. For instance, Multi-modality large language models such as LLaVA (Li et al. 2024a) and Qwen-VL (Bai et al. 2023) rely on CLIP’s visual features, while image/video generative model like Stable-Diffusion-3 (Esser et al. 2024) and Wan (Kong et al. 2024) leverage its text encoder. Nevertheless, as we push toward broader generality and higher task complexity, the representational capacity inherited from CLIP’s original paradigm is gradually becoming inadequate.
CLIP（Radford et al. 2021；Tschannen et al. 2025）已成为近年来影响最大的跨模态基础模型之一。它在数亿到数百亿图文对上做对比预训练，把视觉和语言嵌入同一表征空间。作为检索器，CLIP 支撑零样本分类、检测、分割，以及尤为重要的图文检索；作为特征提取器，它覆盖从图像/视频理解到文生图、文生视频的广泛跨模态应用。例如，LLaVA（Li et al. 2024a）、Qwen-VL（Bai et al. 2023）等多模态大语言模型依赖 CLIP 的视觉特征，而 Stable-Diffusion-3（Esser et al. 2024）、Wan（Kong et al. 2024）等图像/视频生成模型则使用它的文本编码器。不过，随着应用更广、任务更复杂，沿用 CLIP 原有范式所继承的表征能力，已逐渐显得不够。
The rapid progress of large language models (LLMs) has dramatically advanced textual understanding and generation, motivating us to inject stronger LLMs into the CLIP training pipeline so as to bolster multimodal capability. Prior work such as LLM2Vec (BehnamGhader et al. 2024) and NV-Embed-v2 (Lee et al. 2024) has shown that LLMs can be transformed into competitive text–embedding models, now topping leaderboards like MTEB (Muennighoff et al. 2022). Follow-up efforts (e.g. MM-E5 (Chen et al. 2025), VLM2Vec (Jiang et al. 2024)) have extended this idea to multi-modality representations. Yet CLIP’s light dual-tower architecture remains the de-facto embedding model in many scenarios. For instance, in standard zero-shot image–text retrieval, a 400 M-parameter SigLIP-2 So/14 reaches 85.7 (TI) and 94.9 (IT) on Flickr30K at resolution, while VLM2Vec—built on a 7B-parameter LLaVA-1.6 and trained at —achieves 79.8 and 91.6, respectively. CLIP’s lightweight design also scales more readily to large datasets and batch sizes (SigLIP-2 (Tschannen et al. 2025) is trained on 40 B image–text pairs). Consequently, mainstream multimodal retrieval systems—and the multimodal encoders that serve VL-LMs and diffusion models—are increasingly built on CLIP-based architectures. This raises a key question: how can we efficiently harness the strengths of modern LLMs to augment these pretrained CLIP models and further elevate their capabilities?
大语言模型在文本理解与生成上的快速进展，促使我们把更强的 LLM 引入 CLIP 训练流程，以增强多模态能力。已有工作如 LLM2Vec（BehnamGhader et al. 2024）和 NV-Embed-v2（Lee et al. 2024）表明，LLM 可以改造成有竞争力的文本嵌入模型，并在 MTEB（Muennighoff et al. 2022）等排行榜上名列前茅。后续工作（如 MM-E5（Chen et al. 2025）、VLM2Vec（Jiang et al. 2024））又把这一思路延伸到多模态表征。即便如此，CLIP 轻量的双塔结构在许多场景里仍是事实上的嵌入模型。以标准零样本图文检索为例，参数约 4 亿的 SigLIP-2 So/14 在 分辨率的 Flickr30K 上，文本到图像（TI）达到 85.7，图像到文本（IT）达到 94.9；而基于 70 亿参数 LLaVA-1.6、在 上训练的 VLM2Vec，相应成绩为 79.8 和 91.6。CLIP 的轻量设计也更适合放大到大数据集和大批量（SigLIP-2（Tschannen et al. 2025）在约 400 亿图文对上训练）。因此，主流多模态检索系统，以及服务于视觉语言模型和扩散模型的多模态编码器，越来越多地建立在 CLIP 类架构上。这就提出一个关键问题：怎样才能高效借助现代 LLM 的长处，增强这些已经预训练好的 CLIP 模型，并进一步提高它们的能力？
The potential benefits of incorporating LLMs into CLIP are clear. LLMs’ strong textual understanding can fundamentally improve CLIP’s ability to handle image captions, drastically enhancing its ability to process long and complex texts—a well-known limitation of vanilla CLIP. Moreover, LLMs are trained on a vast corpus of text, possessing open-world knowledge. This allows them to expand on caption information during training, increasing the efficiency of the learning process.
把 LLM 引入 CLIP 的潜在好处很清楚。LLM 较强的文本理解可以从根本上改善 CLIP 处理图像标题的能力，显著提升它对长而复杂文本的处理——这正是原始 CLIP 为人熟知的短板。此外，LLM 在海量文本上训练，具备开放世界知识，训练时可以扩充标题中的信息，从而提高学习效率。
Bringing LLMs into the CLIP framework, however, raises two key challenges: 1. Feature separability. Vanilla LLM embeddings are not sufficiently discriminative for contrastive training; existing “LLM-as-embedding” recipes target pure-text tasks. We therefore devise a CLIP-specific embeddingization strategy for LLMs that yields far more separable caption features. 2. Training cost. CLIP pre-training is already expensive; naively fine-tuning an LLM jointly would be prohibitive. We propose a lightweight fine-tuning procedure that injects LLM power into a pretrained CLIP at almost no extra cost.
不过，把 LLM 纳入 CLIP 框架会带来两个关键困难。1. 特征可分性。 原始 LLM 嵌入对对比训练来说不够可分；现有的「把 LLM 当嵌入模型」做法主要面向纯文本任务。为此，我们针对 CLIP 设计了一套 LLM 的嵌入化策略，使标题特征的可分性明显提高。2. 训练成本。 CLIP 预训练本身已经很昂贵，若再与 LLM 联合微调，开销难以承受。我们提出一套轻量微调流程，几乎不加额外成本，就能把 LLM 的能力注入已经预训练好的 CLIP。
This paper introduces LLM2CLIP, an efficient fine–tuning framework that augments the feature space of a pre-trained CLIP with an embedding-tuned LLM, thereby importing LLM capability at very low cost. Injecting a vanilla LLM into CLIP is problematic: as shown by the cases in Figure 2, a COCO test reveals that raw LLM embeddings have poor separability for image captions. To remedy this, we perform caption-contrastive(CC) fine-tuning on the LLM with a set of high-quality caption datasets, revisiting several design choices—e.g. employing average pooling to aggregate token features, enabling bidirectional attention, and training with a supervised SimCSE contrastive loss.
本文提出 LLM2CLIP，这是一套高效微调框架：用经过嵌入调优的 LLM 来扩充已预训练 CLIP 的特征空间，从而以极低成本引入 LLM 能力。把未经改造的 LLM 直接接入 CLIP 会有问题：如图 2 中的例子所示，在 COCO 测试里，原始 LLM 嵌入对图像标题的可分性很差。为弥补这一点，我们在一批高质量标题数据上对 LLM 做标题对比（CC）微调，并重新考察若干设计选择，例如用平均池化汇聚词元特征、启用双向注意力，以及用监督式 SimCSE 对比损失来训练。
Another key challenge when introducing LLMs into CLIP is the presence of two text encoders—the original CLIP text encoder and the newly added LLM—which were never aligned during pre-training. We conduct experiments to explore efficient integration of the LLM into the CLIP architecture. As shown in Figure 1, we adopt a cost-effective solution: freeze all LLM gradients, treat its sentence embeddings as fixed features, and append a small learnable adaptor trained with the CLIP visual encoder. The original CLIP text encoder is discarded during both training and inference. Ablation studies indicate that alternative strategies—combining CLIP and LLM embeddings via triplet contrastive losses or concatenating features—provide marginal or negative gains compared to simply replacing the CLIP text encoder. This minimal design yields an LLM-enhanced cross-modal space while keeping compute cost nearly identical to standard CLIP fine-tuning.
把 LLM 引入 CLIP 的另一关键困难是：系统里会出现两个文本编码器——原来的 CLIP 文本编码器和新加入的 LLM——它们在预训练阶段从未对齐过。我们做了实验，探寻把 LLM 高效接入 CLIP 架构的方式。如图 1 所示，我们采用成本较低的方案：冻结 LLM 的全部梯度，把它的句嵌入当作固定特征，再接一个可学习的小型适配器，与 CLIP 视觉编码器一起训练。原来的 CLIP 文本编码器在训练和推理时都不再使用。消融表明，用三重对比损失结合 CLIP 与 LLM 嵌入，或把特征拼接起来，相对「直接替换 CLIP 文本编码器」只有很小的增益，甚至会变差。这一最简设计可以得到由 LLM 增强的跨模态空间，计算开销几乎与标准 CLIP 微调相当。
Empirical results confirm that LLM2CLIP yields substantial improvements—even with only a few million training examples—boosting the performance of the original CLIP across a variety of downstream tasks. Our contributions are as follows:
实验证实，即便只有数百万训练样本，LLM2CLIP 也能带来明显提升，并在多种下游任务上增强原来的 CLIP。本文贡献如下：
- • We empirically demonstrate that injecting LLM capability into CLIP brings significant performance gains.
- • 我们用实验表明，把 LLM 能力注入 CLIP 可以带来显著的性能提升。
- • We design (i) a caption-contrastive fine-tuning recipe that turns an LLM into a effective embedding model for CLIP, and (ii) a efficient fine-tuning method that couples this embedding with a pretrained CLIP.
- • 我们设计了：（i）一套标题对比微调方案，把 LLM 变成适用于 CLIP 的有效嵌入模型；（ii）一种高效微调方法，把该嵌入与已预训练的 CLIP 结合起来。
- • Extensive experiments reveal that the resulting LLM-enhanced CLIP markedly improves several state-of-the-art models—including EVA02 and SigLIP-2—on a wide spectrum of multimodal benchmarks: short/long-text and cross-lingual image retrieval, zero-shot classification, detection, segmentation, and even as the visual encoder inside LLaVA-1.5. Remarkably, LLM2CLIP lifts SigLIP-2’s short-caption retrieval by +1.0/+1.9, long-caption retrieval by +14.8/+15.8, and multilingual tasks by +11.9/+15.2.
- • 大量实验表明，得到的 LLM 增强 CLIP 明显提升了 EVA02、SigLIP-2 等多个先进模型在一系列多模态基准上的表现，涵盖短/长文本及跨语图像检索、零样本分类、检测、分割，甚至作为 LLaVA-1.5 内部的视觉编码器。LLM2CLIP 使 SigLIP-2 的短标题检索提高 +1.0/+1.9，长标题检索提高 +14.8/+15.8，多语任务提高 +11.9/+15.2。
Figure 2: Real examples of top-1 results from the caption-to-caption retrieval experiment in MS COCO 5K test set. Before fine-tuning, Llama3’s results were often unrelated.
图 2：MS COCO 5K 测试集上标题到标题检索的 top-1 真实例子。微调前，Llama3 的结果常常与查询无关。
2 Related Works
2 相关工作
CLIP meets Stronger Language Models. Several works have explored the integration of LLMs into CLIP. JinaCLIP (Koukounas et al. 2024a) employed Jina-embeddings-v2 (Günther et al. 2023) as the text encoder, which is a BERT variant with 137M parameters, supporting longer texts. Though achieving similar visual performance to EVA-CLIP (Sun et al. 2023a), its text encoder is far behind ours, limiting the potential benefits from LLMs. MATE (Jang et al. 2024) designed a learnable adaptor to bridge the gap between CLIP’s text encoder and LLMs. They trained the CLIP visual encoder using LoRA on a small dataset focused on long-text image retrieval tasks. However, they did not recognize the critical issue we propose: the poor separability of LLM feature space, which is insufficient for direct support of CLIP training. This paper presents the first comprehensive study of an efficient strategy for incorporating LLMs into CLIP fine-tuning, thereby boosting performance.
CLIP 与更强的语言模型。 已有若干工作探索把 LLM 接入 CLIP。JinaCLIP（Koukounas et al. 2024a）用 Jina-embeddings-v2（Günther et al. 2023）作文本编码器，这是一个 1.37 亿参数的 BERT 变体，可以处理更长文本。它的视觉表现与 EVA-CLIP（Sun et al. 2023a）相近，但文本编码器远弱于我们所用的模型，限制了从 LLM 中获益的空间。MATE（Jang et al. 2024）设计了可学习适配器，用于弥合 CLIP 文本编码器与 LLM 之间的差距，并在面向长文本图像检索的小数据集上，用 LoRA 训练 CLIP 视觉编码器。不过，他们并未意识到我们指出的关键问题：LLM 特征空间的可分性较差，不足以直接支撑 CLIP 训练。本文首次较为完整地研究了把 LLM 高效纳入 CLIP 微调、从而提升性能的策略。
CLIP meets Longer Captions. CLIP’s text embedding is widely recognized as coarse and limited to 77 tokens. Many works have attempted to extend caption length and retrain CLIP accordingly, including DCI (Urbanek et al. 2024) with human annotation, LaCLIP (Fan et al. 2024) using ChatGPT and Bard, DreamLIP (Zheng et al. 2024) leveraging ShareCaptioner (Chen et al. 2023), and Recap-DataComp-1B (Li et al. 2024b) using Llama3-trained LLAVA1.5. To handle longer captions, these methods employ workarounds such as summarization (Urbanek et al. 2024), segmentation (Fan et al. 2024; Zheng et al. 2024), or positional encoding finetuning (Zhang et al. 2024a). In comparison, LLM2CLIP leverages LLMs as the text encoder, enabling comprehensive understanding of long and dense captions while utilizing LLMs’ open-world knowledge.
CLIP 与更长的标题。 CLIP 的文本嵌入通常被认为较粗，并且受限于 77 个词元。许多工作试图加长标题并据此重训 CLIP，包括依靠人工标注的 DCI（Urbanek et al. 2024）、使用 ChatGPT 和 Bard 的 LaCLIP（Fan et al. 2024）、借助 ShareCaptioner（Chen et al. 2023）的 DreamLIP（Zheng et al. 2024），以及用经 Llama3 训练的 LLAVA1.5 的 Recap-DataComp-1B（Li et al. 2024b）。为处理更长标题，这些方法采用摘要（Urbanek et al. 2024）、切分（Fan et al. 2024；Zheng et al. 2024）或位置编码微调（Zhang et al. 2024a）等权宜办法。相比之下，LLM2CLIP 直接把 LLM 当作文本编码器，既能够较完整地理解长而稠密的标题，又能利用 LLM 的开放世界知识。
| Method | Res | Flickr | COCO | SG4V | Urban | DOCCI | Avg | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||
| ViT-B/16 (86M) | |||||||||||||
| ALIGN (Jia et al. 2021) | 224 | 80.6 | 62.2 | 52.0 | 43.2 | 75.9 | 80.6 | 62.2 | 59.1 | 59.7 | 62.1 | 66.1 | 61.4 |
| BLIP (Li et al. 2022a) | 224 | 80.6 | 74.1 | 61.7 | 48.5 | 65.8 | 74.3 | 45.5 | 48.5 | 50.5 | 53.5 | 60.8 | 59.8 |
| Long-CLIP (Zhang et al. 2024a) | 224 | 85.8 | 70.6 | 56.9 | 40.9 | 94.8 | 93.5 | 79.1 | 79.1 | 63.1 | 71.4 | 75.9 | 71.1 |
| jina-clip-v2 (Koukounas et al. 2024b) | 224 | 84.4 | 69.8 | 57.1 | 41.9 | 94.5 | 91.1 | 80.4 | 78.0 | 77.6 | 78.2 | 78.8 | 71.8 |
| SigLIP (Zhai et al. 2023) | 224 | 89.1 | 74.7 | 65.8 | 47.8 | 85.8 | 83.5 | 62.8 | 62.4 | 70.1 | 70.6 | 74.7 | 67.8 |
| MetaCLIP (Xu et al. 2023) | 224 | 85.6 | 70.8 | 59.3 | 41.3 | 90.4 | 86.6 | 68.9 | 63.3 | 70.9 | 71.5 | 75.0 | 66.7 |
| CLIP (Radford et al. 2021) | 224 | 82.3 | 62.2 | 52.4 | 33.1 | 84.5 | 79.8 | 67.5 | 53.1 | 60.7 | 57.1 | 69.5 | 57.1 |
| +LLM2CLIP-15M | 224 | 91.8 | 80.7 | 64.8 | 52.4 | 97.8 | 97.6 | 90.1 | 92.1 | 86.1 | 86.9 | 86.1 | 81.9 |
| EVA02 (Sun et al. 2023b) | 224 | 86.2 | 71.5 | 58.7 | 42.1 | 90.5 | 85.5 | 67.0 | 60.8 | 67.7 | 68.0 | 74.0 | 65.6 |
| +LLM2CLIP-15M | 224 | 91.2 | 81.4 | 66.0 | 53.9 | 98.5 | 98.5 | 88.2 | 91.3 | 86.8 | 88.5 | 86.1 | 82.7 |
| ViT-L/14 (307M) | |||||||||||||
| Long-CLIP (Zhang et al. 2024a) | 224 | 90.0 | 76.2 | 62.8 | 46.3 | 97.2 | 97.3 | 82.5 | 86.1 | 66.5 | 78.6 | 79.8 | 76.9 |
| MetaCLIP (Xu et al. 2023) | 224 | 90.1 | 76.4 | 64.5 | 47.1 | 86.4 | 79.5 | 73.4 | 70.0 | 76.5 | 76.7 | 78.2 | 69.9 |
| L-NV2 (Zhang et al. 2025) | 224 | 87.9 | 75.8 | 62.4 | 48.7 | 94.8 | 94.9 | 81.5 | 80.2 | 76.5 | 78.9 | 80.6 | 75.7 |
| CLIP (Radford et al. 2021) | 224 | 85.2 | 65.0 | 56.3 | 36.5 | 84.2 | 83.6 | 68.3 | 55.6 | 63.1 | 65.8 | 71.4 | 61.3 |
| +LLM2CLIP-15M | 224 | 93.0 | 83.5 | 67.4 | 56.4 | 98.7 | 98.6 | 91.5 | 94.1 | 88.3 | 90.4 | 87.8 | 84.6 |
| EVA02 (Sun et al. 2023b) | 224 | 89.7 | 77.3 | 63.7 | 47.5 | 91.9 | 89.3 | 73.3 | 68.5 | 73.5 | 75.0 | 78.4 | 71.5 |
| +LLM2CLIP-3M | 224 | 94.3 | 84.3 | 65.2 | 56.0 | 98.0 | 98.0 | 91.0 | 94.7 | 87.8 | 90.7 | 87.3 | 84.7 |
| +LLM2CLIP-15M | 224 | 94.6 | 85.0 | 69.5 | 58.3 | 98.9 | 99.1 | 93.6 | 95.7 | 89.8 | 91.2 | 89.3 | 85.8 |
| +LLM2CLIP-60M | 224 | 95.9 | 85.1 | 71.7 | 58.5 | 99.2 | 99.2 | 95.2 | 97.0 | 90.1 | 92.0 | 90.4 | 86.3 |
| CLIP (Radford et al. 2021) | 336 | 87.7 | 67.0 | 58.0 | 37.1 | 86.2 | 84.0 | 72.8 | 57.0 | 67.4 | 65.7 | 74.4 | 62.2 |
| +LLM2CLIP-15M | 336 | 91.2 | 82.1 | 65.5 | 53.6 | 98.1 | 98.4 | 90.3 | 93.2 | 87.7 | 89.0 | 86.6 | 83.3 |
| +LLM2CLIP-60M | 336 | 93.9 | 84.3 | 68.5 | 54.8 | 98.9 | 99.1 | 96.6 | 97.9 | 89.6 | 91.6 | 91.1 | 84.5 |
| EVA02 (Sun et al. 2023b) | 336 | 89.6 | 78.0 | 64.2 | 47.9 | 91.5 | 89.4 | 76.6 | 70.0 | 74.7 | 76.4 | 79.3 | 72.3 |
| +LLM2CLIP-15M | 336 | 94.7 | 85.1 | 69.7 | 58.5 | 98.8 | 99.2 | 93.5 | 96.2 | 89.2 | 91.3 | 88.0 | 84.8 |
| +LLM2CLIP-60M | 336 | 95.9 | 85.4 | 72.4 | 58.8 | 98.8 | 99.2 | 95.5 | 97.3 | 90.7 | 92.6 | 91.0 | 86.8 |
| SigLIP (Zhai et al. 2023) | 384 | 93.7 | 81.4 | 72.0 | 53.9 | 90.1 | 90.4 | 76.3 | 74.4 | 77.8 | 79.5 | 82.0 | 75.9 |
| SO/14 (428M) | |||||||||||||
| SigLIP (Zhai et al. 2023) | 224 | 91.0 | 75.2 | 69.8 | 51.8 | 56.1 | 50.0 | 31.5 | 29.1 | 42.3 | 44.8 | 58.1 | 50.2 |
| SigLIP2 (Tschannen et al. 2025) | 224 | 93.9 | 82.9 | 72.0 | 55.5 | 90.2 | 87.2 | 75.7 | 74.5 | 77.0 | 78.9 | 81.8 | 75.8 |
| +LLM2CLIP-60M | 224 | 95.2 | 84.6 | 73.7 | 57.6 | 99.4 | 99.4 | 96.2 | 97.3 | 91.5 | 92.9 | 91.2 | 86.4 |
| SigLIP (Zhai et al. 2023) | 384 | 94.3 | 83.0 | 72.4 | 54.3 | 91.6 | 89.4 | 74.5 | 73.6 | 74.5 | 75.8 | 81.5 | 75.2 |
| InternVL (6B) | |||||||||||||
| InternVL (Chen et al. 2024) | 224 | 94.2 | 81.6 | 69.4 | 53.2 | 92.4 | 90.5 | 79.6 | 77.7 | 79.2 | 81.6 | 83.0 | 76.9 |
| VLM2Vec (7B) | |||||||||||||
| VLM2Vec* (Jiang et al. 2024) | 1344 | 91.6 | 79.8 | 61.3 | 51.5 | 93.9 | 91.0 | 90.9 | 92.0 | 80.5 | 86.0 | 83.6 | 80.1 |
Table 1: Systematic comparison of model performance on multiple datasets. * means reproduced by ourselves.
| 方法 | 分辨率 | Flickr | COCO | SG4V | Urban | DOCCI | 平均 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||
| ViT-B/16 (86M) | |||||||||||||
| ALIGN (Jia et al. 2021) | 224 | 80.6 | 62.2 | 52.0 | 43.2 | 75.9 | 80.6 | 62.2 | 59.1 | 59.7 | 62.1 | 66.1 | 61.4 |
| BLIP (Li et al. 2022a) | 224 | 80.6 | 74.1 | 61.7 | 48.5 | 65.8 | 74.3 | 45.5 | 48.5 | 50.5 | 53.5 | 60.8 | 59.8 |
| Long-CLIP (Zhang et al. 2024a) | 224 | 85.8 | 70.6 | 56.9 | 40.9 | 94.8 | 93.5 | 79.1 | 79.1 | 63.1 | 71.4 | 75.9 | 71.1 |
| jina-clip-v2 (Koukounas et al. 2024b) | 224 | 84.4 | 69.8 | 57.1 | 41.9 | 94.5 | 91.1 | 80.4 | 78.0 | 77.6 | 78.2 | 78.8 | 71.8 |
| SigLIP (Zhai et al. 2023) | 224 | 89.1 | 74.7 | 65.8 | 47.8 | 85.8 | 83.5 | 62.8 | 62.4 | 70.1 | 70.6 | 74.7 | 67.8 |
| MetaCLIP (Xu et al. 2023) | 224 | 85.6 | 70.8 | 59.3 | 41.3 | 90.4 | 86.6 | 68.9 | 63.3 | 70.9 | 71.5 | 75.0 | 66.7 |
| CLIP (Radford et al. 2021) | 224 | 82.3 | 62.2 | 52.4 | 33.1 | 84.5 | 79.8 | 67.5 | 53.1 | 60.7 | 57.1 | 69.5 | 57.1 |
| +LLM2CLIP-15M | 224 | 91.8 | 80.7 | 64.8 | 52.4 | 97.8 | 97.6 | 90.1 | 92.1 | 86.1 | 86.9 | 86.1 | 81.9 |
| EVA02 (Sun et al. 2023b) | 224 | 86.2 | 71.5 | 58.7 | 42.1 | 90.5 | 85.5 | 67.0 | 60.8 | 67.7 | 68.0 | 74.0 | 65.6 |
| +LLM2CLIP-15M | 224 | 91.2 | 81.4 | 66.0 | 53.9 | 98.5 | 98.5 | 88.2 | 91.3 | 86.8 | 88.5 | 86.1 | 82.7 |
| ViT-L/14 (307M) | |||||||||||||
| Long-CLIP (Zhang et al. 2024a) | 224 | 90.0 | 76.2 | 62.8 | 46.3 | 97.2 | 97.3 | 82.5 | 86.1 | 66.5 | 78.6 | 79.8 | 76.9 |
| MetaCLIP (Xu et al. 2023) | 224 | 90.1 | 76.4 | 64.5 | 47.1 | 86.4 | 79.5 | 73.4 | 70.0 | 76.5 | 76.7 | 78.2 | 69.9 |
| L-NV2 (Zhang et al. 2025) | 224 | 87.9 | 75.8 | 62.4 | 48.7 | 94.8 | 94.9 | 81.5 | 80.2 | 76.5 | 78.9 | 80.6 | 75.7 |
| CLIP (Radford et al. 2021) | 224 | 85.2 | 65.0 | 56.3 | 36.5 | 84.2 | 83.6 | 68.3 | 55.6 | 63.1 | 65.8 | 71.4 | 61.3 |
| +LLM2CLIP-15M | 224 | 93.0 | 83.5 | 67.4 | 56.4 | 98.7 | 98.6 | 91.5 | 94.1 | 88.3 | 90.4 | 87.8 | 84.6 |
| EVA02 (Sun et al. 2023b) | 224 | 89.7 | 77.3 | 63.7 | 47.5 | 91.9 | 89.3 | 73.3 | 68.5 | 73.5 | 75.0 | 78.4 | 71.5 |
| +LLM2CLIP-3M | 224 | 94.3 | 84.3 | 65.2 | 56.0 | 98.0 | 98.0 | 91.0 | 94.7 | 87.8 | 90.7 | 87.3 | 84.7 |
| +LLM2CLIP-15M | 224 | 94.6 | 85.0 | 69.5 | 58.3 | 98.9 | 99.1 | 93.6 | 95.7 | 89.8 | 91.2 | 89.3 | 85.8 |
| +LLM2CLIP-60M | 224 | 95.9 | 85.1 | 71.7 | 58.5 | 99.2 | 99.2 | 95.2 | 97.0 | 90.1 | 92.0 | 90.4 | 86.3 |
| CLIP (Radford et al. 2021) | 336 | 87.7 | 67.0 | 58.0 | 37.1 | 86.2 | 84.0 | 72.8 | 57.0 | 67.4 | 65.7 | 74.4 | 62.2 |
| +LLM2CLIP-15M | 336 | 91.2 | 82.1 | 65.5 | 53.6 | 98.1 | 98.4 | 90.3 | 93.2 | 87.7 | 89.0 | 86.6 | 83.3 |
| +LLM2CLIP-60M | 336 | 93.9 | 84.3 | 68.5 | 54.8 | 98.9 | 99.1 | 96.6 | 97.9 | 89.6 | 91.6 | 91.1 | 84.5 |
| EVA02 (Sun et al. 2023b) | 336 | 89.6 | 78.0 | 64.2 | 47.9 | 91.5 | 89.4 | 76.6 | 70.0 | 74.7 | 76.4 | 79.3 | 72.3 |
| +LLM2CLIP-15M | 336 | 94.7 | 85.1 | 69.7 | 58.5 | 98.8 | 99.2 | 93.5 | 96.2 | 89.2 | 91.3 | 88.0 | 84.8 |
| +LLM2CLIP-60M | 336 | 95.9 | 85.4 | 72.4 | 58.8 | 98.8 | 99.2 | 95.5 | 97.3 | 90.7 | 92.6 | 91.0 | 86.8 |
| SigLIP (Zhai et al. 2023) | 384 | 93.7 | 81.4 | 72.0 | 53.9 | 90.1 | 90.4 | 76.3 | 74.4 | 77.8 | 79.5 | 82.0 | 75.9 |
| SO/14 (428M) | |||||||||||||
| SigLIP (Zhai et al. 2023) | 224 | 91.0 | 75.2 | 69.8 | 51.8 | 56.1 | 50.0 | 31.5 | 29.1 | 42.3 | 44.8 | 58.1 | 50.2 |
| SigLIP2 (Tschannen et al. 2025) | 224 | 93.9 | 82.9 | 72.0 | 55.5 | 90.2 | 87.2 | 75.7 | 74.5 | 77.0 | 78.9 | 81.8 | 75.8 |
| +LLM2CLIP-60M | 224 | 95.2 | 84.6 | 73.7 | 57.6 | 99.4 | 99.4 | 96.2 | 97.3 | 91.5 | 92.9 | 91.2 | 86.4 |
| SigLIP (Zhai et al. 2023) | 384 | 94.3 | 83.0 | 72.4 | 54.3 | 91.6 | 89.4 | 74.5 | 73.6 | 74.5 | 75.8 | 81.5 | 75.2 |
| InternVL (6B) | |||||||||||||
| InternVL (Chen et al. 2024) | 224 | 94.2 | 81.6 | 69.4 | 53.2 | 92.4 | 90.5 | 79.6 | 77.7 | 79.2 | 81.6 | 83.0 | 76.9 |
| VLM2Vec (7B) | |||||||||||||
| VLM2Vec* (Jiang et al. 2024) | 1344 | 91.6 | 79.8 | 61.3 | 51.5 | 93.9 | 91.0 | 90.9 | 92.0 | 80.5 | 86.0 | 83.6 | 80.1 |
表 1：多个数据集上的系统性能比较。* 表示由我们自行复现。
3 Methods
3 方法
3.1 Stage 1: LLM Caption Contrastive Fine-tuning
3.1 第一阶段：LLM 标题对比微调
Traditionally, an LLM’s output layer functions as a classification head that produces discrete text tokens. Recent studies—such as LLM2Vec (BehnamGhader et al. 2024) and NV-EMBED-v2 (Lee et al. 2024)—have shown that, with additional design tweaks, an LLM can be repurposed as a text-embedding model. Yet these efforts do not account for CLIP’s cross-modal pre-training regime. Here, we advance the discussion from three fronts—model architecture, training methods and training data—explicitly targeting the seamless integration of LLMs with CLIP.
传统上，LLM 的输出层是一个分类头，用来生成离散文本词元。近期研究，如 LLM2Vec（BehnamGhader et al. 2024）和 NV-EMBED-v2（Lee et al. 2024）表明，再辅以一些设计调整，就可以把 LLM 改作文本嵌入模型。不过，这些工作并未考虑 CLIP 的跨模态预训练设定。这里我们从模型结构、训练方法和训练数据三方面推进讨论，目标是让 LLM 能够顺畅地与 CLIP 结合。
Model Architecture:
模型结构：
- • Sentence Token Representation: We map the output layer features of the LLM to represent whole sentences. We considered two strategies: using the [EOS] token, which can attend to the entire sentence during pretraining, or employing average pooling across all output tokens. Empirically, average pooling performed better and is our default setting.
- • 句子词元表示： 我们把 LLM 输出层特征映射为整句表示。考虑了两种策略：使用预训练时能够看到全句的 [EOS] 词元，或对全部输出词元做平均池化。实验中平均池化更好，因此作为默认设置。
- • Bidirectional Attention: Given that the generative capability is unnecessary for our encoder-focused application, we remove the attention mask from the LLM, enabling bidirectional textual relationship modeling for enhanced comprehension.
- • 双向注意力： 我们的应用以编码为主，并不需要生成能力，因此去掉 LLM 的注意力掩码，使模型能够双向建模文本关系，从而加强理解。
- • Fine-tuning Parameters: To efficiently activate stronger textual comprehension capabilities within the output features of LLMs, we apply parameter-efficient fine-tuning using Low-Rank Adaptation (LoRA).
- • 微调参数： 为了高效激活 LLM 输出特征中更强的文本理解能力，我们用低秩适配（LoRA）做参数高效微调。
- • Adaptor Design: Adding an adaptor following the LLM can potentially enhance feature separability. We experimented with three adaptor variants: (1) a Transformer-based adaptor composed of several latent cross-attention transformer layers following NV-EMBED-v2, (2) a Linear adaptor consisting of multiple linear layers directly attached after extracting sentence tokens from LLM outputs and (3) without using adaptor.
- • 适配器设计： 在 LLM 之后加一个适配器，有可能提高特征可分性。我们试验了三种变体：（1）基于 Transformer 的适配器，由若干潜在交叉注意力 Transformer 层组成，做法参考 NV-EMBED-v2；（2）线性适配器，在从 LLM 输出中取出句子词元后直接接多层线性层；（3）不使用适配器。
Training Methods:
训练方法：
- • Masked Next Token Prediction (MNTP): Inspired by masked language modeling in BERT, MNTP improves feature quality by masking and predicting specific tokens. Unlike BERT, predictions are always made at the position preceding the masked token, aligning with LLM’s next-token prediction convention. While effective in pure text tasks (as shown by LLM2Vec), we find out that MNTP alone or combined with contrastive learning underperformed relative to direct contrastive fine-tuning in multimodal scenarios.
- • 掩码下一词预测（MNTP）： 受 BERT 掩码语言建模启发，MNTP 通过掩码并预测特定词元来提升特征质量。与 BERT 不同，预测始终在被掩码词元的前一个位置进行，以贴合 LLM 的下一词预测习惯。在纯文本任务上这种方法有效（如 LLM2Vec 所示），但我们发现：在多模态场景里，单独使用 MNTP，或把它与对比学习组合，都不如直接做对比微调。
- • Textual Contrastive Learning: Following SimCSE, we employ textual contrastive learning to enhance feature separability, pulling positive samples closer and pushing negative samples apart in feature space. Two variants are possible: unsupervised (using dropout augmentation) and supervised (leveraging semantic pairs from annotated data). We default to supervised contrastive learning due to superior performance. Specifically, we generate positive pairs using two distinct captions from the same image, framed by a system prompt: ”Given a caption, retrieve a similar relevant caption.”
- • 文本对比学习： 我们沿用 SimCSE，用文本对比学习提高特征可分性，在特征空间中拉近正样本、推开负样本。可以有两种变体：无监督（用 dropout 增强）和监督（利用标注数据中的语义对）。监督式对比学习效果更好，因此作为默认。具体做法是：取同一张图像的两条不同标题作为正对，并加上系统提示：“Given a caption, retrieve a similar relevant caption.”
Training Data:
训练数据：
To leverage LLM embeddings for CLIP fine-tuning, we fine-tune the LLM on caption corpora extracted from image–text datasets, aligning its embedding space with downstream multimodal tasks. Experiments use the Dreamlip (Zheng et al. 2024) captions, which provide multiple captions per image, suitable for supervised textual contrastive learning. To preserve general language understanding, we also incorporate 1.5M pure-text pairs from Echo Embeddings (Springer et al. 2024) into a mixed contrastive training setup.
为了把 LLM 嵌入用于 CLIP 微调，我们在从图文数据集中抽出的标题语料上微调 LLM，使其嵌入空间与下游多模态任务对齐。实验使用 Dreamlip（Zheng et al. 2024）的标题，每张图像有多条标题，适合做监督式文本对比学习。为保留通用语言理解能力，我们还把 Echo Embeddings（Springer et al. 2024）中的 150 万纯文本对纳入混合对比训练。
3.2 Stage 2: LLM2CLIP Post Fine-tuning
3.2 第二阶段：LLM2CLIP 后期微调
We aim to perform post fine-tuning on pretrained CLIP since we believe the fundamental visual-language mapping has been well-established through CLIP’s extensive pretraining. By introducing an LLM, we expect to further enrich this representation space, potentially improving CLIP’s capabilities through a highly cost-effective fine-tuning step. CLIP inherently consists of a Vision Encoder (typically a ViT) and a relatively small Text Encoder (roughly one-third the parameters of the Vision Encoder), structured as a compact autoregressive model. Integrating a caption-contrastive fine-tuned LLM as an additional text encoder and finding an effective approach to enhance CLIP performance is critical. Here, we also elaborate on model architecture, training methods and training data.
我们希望在已预训练的 CLIP 上做后期微调，因为认为视觉–语言的基本映射已经通过 CLIP 的大规模预训练建立起来。引入 LLM 后，我们期望进一步丰富这一表征空间，并借助成本很低的一步微调提升 CLIP 的能力。CLIP 本身由视觉编码器（通常是 ViT）和相对较小的文本编码器（参数大约是视觉编码器的三分之一）组成，文本一侧是紧凑的自回归模型。把经过标题对比微调的 LLM 作为额外的文本编码器接入，并找到能有效提升 CLIP 的办法，这一点至关重要。下面同样从模型结构、训练方法和训练数据展开。
Model Architecture: Training CLIP typically requires large-scale datasets and substantial batch sizes, resulting in considerable computational costs. Therefore, we employ efficient Parameter-Efficient Fine-Tuning (PEFT) methods for this stage. By default, we enable full gradients for the visual encoder to facilitate learning from LLM knowledge. Given the large size of the LLM, we compare two strategies: (1) fine-tuning the LLM with LoRA, and (2) freezing LLM gradients and appending previously introduced Transformer or Linear adaptors after its output layer as learnable modules to obtain sentence token features. Method (2) offers clear advantages: it completely eliminates LLM gradient updates, significantly reducing GPU memory usage. Moreover, since the adaptor is placed after the LLM, text features can be precomputed and stored via offline inference, reducing inference overhead from multiple epochs to a single pass. As a result, this approach avoids loading the LLM into GPU memory during training, accelerating experimentation and reducing training time and memory footprint. By default, we adopt method (2) with an adaptor of four linear blocks. Detailed efficiency analysis is provided in Section 4.4.
模型结构： 训练 CLIP 通常需要大规模数据和很大的批量，计算成本很高，因此这一阶段我们采用参数高效微调（PEFT）。默认情况下，我们打开视觉编码器的全部梯度，以便从 LLM 的知识中学习。鉴于 LLM 体积很大，我们比较了两种策略：（1）用 LoRA 微调 LLM；（2）冻结 LLM 梯度，在其输出层后接前文所述的 Transformer 或线性适配器，作为可学习模块来得到句子词元特征。策略（2）的好处很清楚：完全不必更新 LLM 梯度，显存占用显著下降。而且适配器接在 LLM 之后，文本特征可以离线推断并预先存好，把多轮训练中的反复推断变成一次前向计算。这样，训练时不必把 LLM 装进 GPU 显存，实验更快，训练时间和显存占用也更低。默认采用策略（2），适配器由四个线性模块组成。更细致的效率分析见第 4.4 节。
Training Methods: After introducing an LLM into CLIP, we are left with two text encoders. How should they be used most effectively? We tested several approaches: (a) Removing the original CLIP Text Encoder and using LLM with CLIP Vision Encoder directly for cross-modal contrastive learning. (b) Retaining both LLM and original Text Encoder, applying separate CLIP losses to both encoders. (c) Adding an additional contrastive learning task between LLM and the original Text Encoder to method (b). (d) Concatenating LLM and Text Encoder output features in method (b), then performing contrastive learning with CLIP Vision Encoder. Our extensive experiments led us to adopt method (a) for its simplicity and effectiveness.
训练方法： 把 LLM 引入 CLIP 之后，系统里剩下两个文本编码器，应当怎样用最合适？我们试验了几种做法：（a）去掉原来的 CLIP 文本编码器，直接用 LLM 与 CLIP 视觉编码器做跨模态对比学习；（b）同时保留 LLM 和原文本编码器，对两个编码器分别施加 CLIP 损失；（c）在（b）的基础上，再在 LLM 与原文本编码器之间增加一项对比学习；（d）在（b）中把 LLM 与原文本编码器的输出特征拼接，再与 CLIP 视觉编码器做对比学习。经过大量实验，我们采用方法（a），因为它更简单，效果也足够好。
Training Data: With the integration of LLM, our method inherently excels at processing dense and detailed image captions. Datasets such as DreamLIP and Recaption, which leverage Share-Captioner (Chen et al. 2023), InstructBLIP (Dai et al. 2023) and LLaVA-1.5 (Liu et al. 2023a) to generate multiple dense captions per image, can be effectively utilized. We explored multiple caption scenarios, employing multiple positive caption examples during contrastive training and blending real and MLLM-generated dense captions. Ultimately, we adopted a strategy blending real captions and dense MLLM captions at a 50% ratio.
训练数据： 接入 LLM 后，我们的方法天然更擅长处理稠密、细致的图像标题。DreamLIP、Recaption 等数据集借助 Share-Captioner（Chen et al. 2023）、InstructBLIP（Dai et al. 2023）和 LLaVA-1.5（Liu et al. 2023a），为每张图像生成多条稠密标题，可以有效利用。我们尝试了多种标题设定：在对比训练中使用多条正标题，并把真实标题与多模态大模型生成的稠密标题混合。最终采用真实标题与稠密 MLLM 标题按 50% 比例混合的策略。
4 Experiments
4 实验
4.1 Default Experimental Settings
4.1 默认实验设置
Default Settings for Stage-1 We choose to use LLaMA 3.1 8B as the LLM integrated into CLIP pre-training, removing the causal attention mask to enable bidirectional attention. No additional adapters are used in caption contrastive finetune, and average pooling is employed to obtain the sentence token embedding. MNTP is not used by default.
第一阶段的默认设置 我们选择把 LLaMA 3.1 8B 作为接入 CLIP 预训练的 LLM，去掉因果注意力掩码以启用双向注意力。标题对比微调不加额外适配器，用平均池化得到句子词元嵌入。默认不使用 MNTP。
We utilize 30M DreamLIP caption data to perform caption contrastive fine-tuning on the LLM. Randomly sampling any two captions of an image from DreamLIP as positive pairs, we conduct supervised SimCSE training.
我们用 3000 万条 DreamLIP 标题数据对 LLM 做标题对比微调。从 DreamLIP 中随机抽取同一图像的任意两条标题作为正对，进行监督式 SimCSE 训练。
Default Settings for Stage-2 After caption contrastive fine-tuning, we replace the original CLIP text encoder with the LLaMA-3.1-8B to perform cross-modal contrastive learning fine-tuning with the vision encoder. By default, we open the ViT gradients and freeze the LLM gradients, then add an adapter layer composed of four linear layers after the LLM. The design of the Adaptor is the same as FuseMix (Vouitsis et al. 2023), using a simple inverted bottleneck MLP architecture. By default, we use 15M DreamLIP annotated subsets of CC 3M and CC 12M. For the 60M data setting, we use DreamLIP re-annotated CC 3M and CC 12M, YFCC 15M, and a 30M subset of the LAION dataset. For the 3M setting, we only use the CC 3M subset.
第二阶段的默认设置 完成标题对比微调后，我们用 LLaMA-3.1-8B 替换原来的 CLIP 文本编码器，与视觉编码器一起做跨模态对比微调。默认打开 ViT 的梯度、冻结 LLM 的梯度，并在 LLM 之后加由四层线性层组成的适配器。适配器设计与 FuseMix（Vouitsis et al. 2023）相同，采用简单的倒瓶颈 MLP 结构。默认使用 CC 3M 与 CC 12M 上经 DreamLIP 标注的 1500 万子集。在 6000 万数据设定中，使用 DreamLIP 重新标注的 CC 3M、CC 12M、YFCC 15M，以及 LAION 的 3000 万子集。3M 设定则只用 CC 3M 子集。
4.2 System Comparison
4.2 系统比较
In the system comparison, we used the default settings from Section 4.1. However, we also conducted experiments on different pretrained SOTA CLIP models to verify that applying LLM2CLIP post training can significantly enhance their performance.
系统比较采用第 4.1 节的默认设置。我们还在多种已预训练的先进 CLIP 模型上做了实验，以验证施加 LLM2CLIP 后期训练能够明显提升它们的性能。
Zero-Shot English Text-Image Retrieval For short-text retrieval, we used the MSCOCO (Lin et al. 2014) 5K test set and the Flickr (Young et al. 2014) 1K test set. For long-text retrieval, we employed a 1K subset of ShareGPT4V (Chen et al. 2023) and the Urban1K (Zhang et al. 2024a) dataset from LongCLIP (Zhang et al. 2024a), along with the DOCCI (Onoe et al. 2024) dataset.
零样本英文图文检索 短文本检索使用 MSCOCO（Lin et al. 2014）5K 测试集和 Flickr（Young et al. 2014）1K 测试集。长文本检索采用 ShareGPT4V（Chen et al. 2023）的 1K 子集、来自 LongCLIP（Zhang et al. 2024a）的 Urban1K（Zhang et al. 2024a）数据集，以及 DOCCI（Onoe et al. 2024）数据集。
The ShareGPT4V-1M dataset consists of captions generated using GPT-4V and ShareCaptioner, covering images from LAION, CC, SBU (Ordonez et al. 2011), and MS COCO. Urban1K includes captions for 1,000 urban scene images, each richly annotated with detailed descriptions. DOCCI contains 1.5K high-resolution images with human-annotated captions and was used for retrieval evaluation. To facilitate observation, we additionally report the average Top-1 image-to-text (I2T) and text-to-image (T2I) accuracy across these five datasets.
ShareGPT4V-1M 数据集由 GPT-4V 和 ShareCaptioner 生成的标题组成，图像来自 LAION、CC、SBU（Ordonez et al. 2011）和 MS COCO。Urban1K 包含 1000 张城市场景图像的标题，每张都配有细致描述。DOCCI 含有 1500 张带人工标注标题的高分辨率图像，用于检索评测。为便于观察，我们还报告这五个数据集上平均的 Top-1 图像到文本（I2T）和文本到图像（T2I）准确率。
LLM2CLIP Improves CLIP for Long- and Short-Text Retrieval. In Table 1, we compared OpenAI’s CLIP, EVA02, and SigLIP2, and conducted experiments using three types of visual models: ViT-B/16, ViT-L/14, and SoViT/14. Regardless of the model size or resolution shown in the table, LLM2CLIP yields a significant performance boost for both CLIP and EVA02. For example, under EVA02 at 224 resolution, the LLM2CLIP-60M design achieves an average Top-1 retrieval accuracy improvement of +12 and +14.8. Even for SigLIP2—which was pretrained on 40B data—our method still brings an average performance improvement of +9.4 and +10.6; even on the short-caption datasets Flickr and COCO, where SigLIP2 is already very familiar, improvements of +1 and +1.9 are observed. Comparatively, the improvements for long captions are even larger, as the inherent large window of LLMs fully leverages their advantage in understanding long and complex texts. Notably, LLM2CLIP even surpasses InternVL—which contains 6 B trainable parameters—and VLM2Vec, whose pipeline fine-tunes the entire LLM with LoRA.
LLM2CLIP 提升 CLIP 的长、短文本检索。 在表 1 中，我们比较了 OpenAI 的 CLIP、EVA02 和 SigLIP2，并在 ViT-B/16、ViT-L/14、SoViT/14 三类视觉模型上做了实验。无论表中所示的模型规模或分辨率如何，LLM2CLIP 都能给 CLIP 和 EVA02 带来明显提升。例如，在分辨率为 224 的 EVA02 上，LLM2CLIP-60M 设计使平均 Top-1 检索准确率分别提高 12 和 14.8 个百分点。即便是在 400 亿数据上预训练的 SigLIP2，我们的方法仍带来平均 +9.4 和 +10.6 的提升；即使在 SigLIP2 已经相当熟悉的短标题数据集 Flickr 和 COCO 上，也分别看到 +1 和 +1.9 的改进。相对而言，长标题上的提升更大，因为 LLM 固有的大窗口充分发挥了它理解长而复杂文本的优势。LLM2CLIP 甚至超过了拥有 60 亿可训练参数的 InternVL，以及用 LoRA 微调整个 LLM 的 VLM2Vec 流程。
| Models | Flickr-CN | COCO-CN | XM3600 | |||
|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | |
| CN-CLIP | 80.2 | 68.0 | 63.4 | 64.0 | – | – |
| EVA-L-224 | 4.4 | 0.9 | 2.6 | 1.0 | 14.0 | 8.0 |
| +LLM2CLIP | 90.6 | 75.6 | 72.0 | 70.1 | 68.3 | 56.0 |
| SigLIP2 | 79.2 | 56.9 | 55.3 | 51.7 | 59.7 | 48.2 |
| +LLM2CLIP | 90.0 | 76.1 | 70.8 | 70.2 | 69.1 | 56.3 |
Table 2: Multi-lingua retrieval results on different datasets.
| 模型 | Flickr-CN | COCO-CN | XM3600 | |||
|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | |
| CN-CLIP | 80.2 | 68.0 | 63.4 | 64.0 | – | – |
| EVA-L-224 | 4.4 | 0.9 | 2.6 | 1.0 | 14.0 | 8.0 |
| +LLM2CLIP | 90.6 | 75.6 | 72.0 | 70.1 | 68.3 | 56.0 |
| SigLIP2 | 79.2 | 56.9 | 55.3 | 51.7 | 59.7 | 48.2 |
| +LLM2CLIP | 90.0 | 76.1 | 70.8 | 70.2 | 69.1 | 56.3 |
表 2：不同数据集上的多语检索结果。
| Methods | Imagenet | ||
|---|---|---|---|
| 0-shot* | 0-shot | Linear | |
| CLIP L/14-336 | 74.9 | 76.6 | 84.8 |
| +LLM2CLIP | 74.6 | 75.8 | 85.2 |
| CLIP L/14 | 73.7 | 75.5 | 83.9 |
| +LLM2CLIP | 73.2 | 74.3 | 84.4 |
Table 3: Zero-shot classification and linear probe performance on ImageNet. *0-shot uses the class template ‘a photo of the {classname}’ only.
| 方法 | ImageNet | ||
|---|---|---|---|
| 零样本* | 零样本 | 线性探测 | |
| CLIP L/14-336 | 74.9 | 76.6 | 84.8 |
| +LLM2CLIP | 74.6 | 75.8 | 85.2 |
| CLIP L/14 | 73.7 | 75.5 | 83.9 |
| +LLM2CLIP | 73.2 | 74.3 | 84.4 |
表 3：ImageNet 上的零样本分类与线性探测表现。*零样本仅使用类别模板 ‘a photo of the {classname}’。
| Method | Zero-shot Seg. mIOU | OV-COCO Det. | COCO val2017 | |||||
|---|---|---|---|---|---|---|---|---|
| COCO-S | ADE | VOC | City | Novel | Base | All | APbb/APseg | |
| EVA02 | 12.9 | 11.5 | 21.0 | 13.5 | 24.7 | 53.6 | 46.0 | 45.0/38.2 |
| +LLM2CLIP | 15.3 | 15.8 | 29.1 | 20.1 | 28.9 | 54.7 | 48.0 | 45.6/38.7 |
Table 4: Zero-shot/supervised segmentation, open-vocabulary detection benchmarks. COCO val2017 is supervised results using CLIP’s visual encoder.
| 方法 | 零样本分割 mIoU | OV-COCO 检测 | COCO val2017 | |||||
|---|---|---|---|---|---|---|---|---|
| COCO-S | ADE | VOC | City | Novel | Base | All | APbb/APseg | |
| EVA02 | 12.9 | 11.5 | 21.0 | 13.5 | 24.7 | 53.6 | 46.0 | 45.0/38.2 |
| +LLM2CLIP | 15.3 | 15.8 | 29.1 | 20.1 | 28.9 | 54.7 | 48.0 | 45.6/38.7 |
表 4：零样本/监督分割与开放词汇检测基准。COCO val2017 是使用 CLIP 视觉编码器的监督结果。
| Model | VQA | Pope | MM | Seed | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| V2 | GQA | Vz | SQA | TV | R | A | P | MME | MB | MC | LB | MV | All | I | V | |
| Llava (Paper) | 78.5 | 62.0 | 50.0 | 66.8 | 58.2 | 87.3 | 86.1 | 84.2 | 1510.7 | 64.3 | 58.3 | 65.4 | 31.1 | 58.6 | 66.1 | 37.3 |
| Llava (Rep.) | 79.04 | 62.86 | 50.57 | 67.97 | 57.48 | 87.7 | 84.85 | 86.3 | 1476.69 | 66.66 | 60.39 | 58.0 | 34.3 | 59.86 | 66.95 | 39.71 |
| +LLM2CLIP | 79.80 | 63.15 | 52.37 | 69.92 | 58.35 | 88.55 | 82.76 | 87.75 | 1505.82 | 68.29 | 60.40 | 62.7 | 34.8 | 60.96 | 68.80 | 38.96 |
Table 5: Performance of Llava 1.5 benchmarks. For + LLM2CLIP we replace Llava’s CLIP Vit-L/14 with our finetuned version.
| 模型 | VQA | Pope | MM | Seed | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| V2 | GQA | Vz | SQA | TV | R | A | P | MME | MB | MC | LB | MV | All | I | V | |
| Llava (Paper) | 78.5 | 62.0 | 50.0 | 66.8 | 58.2 | 87.3 | 86.1 | 84.2 | 1510.7 | 64.3 | 58.3 | 65.4 | 31.1 | 58.6 | 66.1 | 37.3 |
| Llava (Rep.) | 79.04 | 62.86 | 50.57 | 67.97 | 57.48 | 87.7 | 84.85 | 86.3 | 1476.69 | 66.66 | 60.39 | 58.0 | 34.3 | 59.86 | 66.95 | 39.71 |
| +LLM2CLIP | 79.80 | 63.15 | 52.37 | 69.92 | 58.35 | 88.55 | 82.76 | 87.75 | 1505.82 | 68.29 | 60.40 | 62.7 | 34.8 | 60.96 | 68.80 | 38.96 |
表 5：LLaVA 1.5 各基准上的表现。对于 + LLM2CLIP，我们用微调后的版本替换 LLaVA 的 CLIP ViT-L/14。
Fine-tuning Data Volume Analysis. In Table 1, we experimented with 3M, 15M, and 60M training data on CLIP and EVA02 using ViT-L/14. More data consistently improves retrieval performance for both long and short text tasks at 224 and 336 resolutions. With 3M data, EVA02 shows noticeable improvement in long text retrieval but minimal gain in short text retrieval, suggesting the model first compensates for disrupting the original cross-modal space before achieving broader improvements.
微调数据量分析。 在表 1 中，我们在 CLIP 和采用 ViT-L/14 的 EVA02 上试验了 300 万、1500 万和 6000 万训练数据。无论在 224 还是 336 分辨率，更多数据都持续提升长、短文本检索。在 300 万数据下，EVA02 的长文本检索有明显提升，短文本检索增益很小，说明模型会先弥补对原有跨模态空间的扰动，然后再取得更全面的改进。
Zero-Shot Multilingual Text-Image Retrieval As in Table 2, We conducted cross-lingual retrieval experiments where CLIP vision encoders were trained exclusively on English text. We successfully endowed EVA02 with multilingual capabilities, which it previously lacked, and significantly enhanced the already robust multilingual performance of SigLIP2.
零样本多语图文检索 如表 2 所示，我们做了跨语检索实验，其中 CLIP 视觉编码器只在英文文本上训练。我们成功给原本几乎没有多语能力的 EVA02 补上了多语能力，并明显提高了 SigLIP2 本来已经较强的多语表现。
Zero-shot & Linear Probe ImageNet Classification We evaluate our model on ImageNet using two standard protocols: (1) Zero-shot classification, where we report accuracy using the prompt template “a photo of the {classname}”, following the CLIP methodology. We also evaluate the average performance over 80 handcrafted prompt variants, as commonly practiced in prior work. (2) Linear probing, where we freeze the visual encoder and train a linear classifier using fixed hyperparameters across all experiments (batch size 1024, learning rate 0.1, momentum 0.9, weight decay 0, 50 epochs with SGD).
零样本与线性探测 ImageNet 分类 我们按两种标准协议在 ImageNet 上评测。（1）零样本分类：按 CLIP 的做法，报告使用提示模板 “a photo of the {classname}” 的准确率；也按以往常见做法，报告 80 种手工提示变体上的平均成绩。（2）线性探测：冻结视觉编码器，用固定超参数训练线性分类器（批量 1024，学习率 0.1，动量 0.9，权重衰减 0，SGD 训练 50 个周期）。
As shown in Table 3, we conduct experiments using CLIP ViT-L/14 at both 224 and 336 resolutions. We observe that after applying LLM2CLIP, zero-shot performance on ImageNet shows a modest drop. However, the accuracy from linear probing improves over the original CLIP baseline. This suggests that while the quality of the learned visual features remains strong (as linear probing only tests CLIP visual encoder’s representation power using a supervised head), the overall multimodal alignment in the shared space may slightly deteriorate since zero-shot classification drops, especially for fine-grained category separation.
如表 3 所示，我们在 224 和 336 两种分辨率上用 CLIP ViT-L/14 做了实验。施加 LLM2CLIP 后，ImageNet 零样本成绩略有下降，但线性探测准确率高于原来的 CLIP 基线。这说明所学视觉特征本身仍然更强（线性探测只用带监督的分类头来检验 CLIP 视觉编码器的表征能力），但共享空间里的整体多模态对齐可能略有减弱——因为零样本分类下降了，在细粒度类别区分上尤其如此。
We hypothesize that the drop in zero-shot performance could be mitigated by increasing the amount of LLM2CLIP fine-tuning data, which remains to be verified in future work. Nonetheless, this trade-off echoes a common pattern observed in recent CLIP fine-tuning studies, such as LongCLIP (Zhang et al. 2024a) and CLIP-MoE (Zhang et al. 2024b), where performance on head classes improves while long-tail categories suffer due to distribution imbalance—particularly under limited-scale fine-tuning. The contrast with retrieval results is instructive: retrieval benefits greatly because its vocabulary is broad and frequent, whereas classification demands uniform discriminability across all (often obscure) class nouns.
我们推测，零样本成绩的回落或许可以通过增加 LLM2CLIP 微调数据来缓解，这一点有待后续工作验证。不过，这种取舍也常见于近期的 CLIP 微调研究，例如 LongCLIP（Zhang et al. 2024a）和 CLIP-MoE（Zhang et al. 2024b）：头部类别变好，长尾类别则因分布不均而受损，在小规模微调时尤为明显。与检索结果的对照很能说明问题：检索受益很大，是因为它的词汇面广、出现频率高；分类则要求在全部类别名词上（其中不少很生僻）都有均匀的可分性。
Zero-Shot / Supervised Segmentation and Object Detection As our linear-probe results already suggest, the visual features produced by the encoder become noticeably stronger after LLM2CLIP training. Table 4 further confirms this on both zero-shot and fully supervised object detection and segmentation benchmarks. In the zero-shot setting—which relies on the text encoder—LLM2CLIP improves CLIP’s multimodal representations; in the supervised setting—where only the vision encoder is finetuned—it also enhances the purely visual, low-level feature extractor. We conjecture that these gains stem from the LLM’s ability to parse dense captions more accurately, especially spatial terms and object-to-object relations, and to inject that knowledge into the joint vision–language space.
零样本 / 监督分割与目标检测 线性探测结果已经表明，经过 LLM2CLIP 训练后，编码器给出的视觉特征明显变强。表 4 在零样本和全监督的目标检测与分割基准上进一步证实了这一点。零样本设定依赖文本编码器，LLM2CLIP 改善了 CLIP 的多模态表征；监督设定只微调视觉编码器，它同样增强了纯视觉的底层特征提取器。我们猜想，这些增益来自 LLM 更能准确解析稠密标题，尤其是空间用语和物体之间的关系，并把这些知识注入联合的视觉–语言空间。
Multimodal Large Language Models Performance Following LLAVA1.5 (Liu et al. 2023b), we used OpenAI’s CLIP-ViT-L-336 encoder with a simple MLP head to connect to Vicuna-7B. Pretraining included 558K image-caption pairs and 665K visual instruction samples. We fine-tuned the LLAVA1.5 encoder using LLM2CLIP to assess improvements in visual feature extraction. As shown in Table 5, LLM2CLIP fine-tuning enhanced multimodal model performance on over 87.5% of benchmarks, with minor losses in only two tasks. This highlights the potential of LLM2CLIP for improving CLIP visual encoder’s abilities for complex image reasoning and understanding.
多模态大语言模型表现 我们沿用 LLaVA-1.5（Liu et al. 2023b），用 OpenAI 的 CLIP-ViT-L-336 编码器加一个简单 MLP 头接到 Vicuna-7B。预训练包含 55.8 万图文对和 66.5 万视觉指令样本。我们用 LLM2CLIP 微调 LLaVA-1.5 的编码器，以评估视觉特征提取是否改善。如表 5 所示，LLM2CLIP 微调在超过 87.5% 的基准上提升了多模态模型表现，仅有两项任务出现小幅下降。这说明 LLM2CLIP 有潜力增强 CLIP 视觉编码器在复杂图像推理与理解上的能力。
| Stage1 | Avg | |
|---|---|---|
| I2T | T2I | |
| Lora, AvgPool, Bidirectional, Supervise Simcse | 80.4 | 77.9 |
| Lora, AvgPool, Bidirectional, Un-supervise Simcse | 59.2 | 57.7 |
| Lora, AvgPool, Bidirectional, MNTP | 70.1 | 67.0 |
| Lora, AvgPool, Bidirectional, MNTP, Supervise Simcse | 79.7 | 77.2 |
| Lora, AvgPool, Casual, Supervise Simcse | 80.0 | 77.5 |
| Lora, EOS, Bidirectional, Supervise Simcse | 80.0 | 77.3 |
| Frozen, Linear Adaptor, AvgPool, Bidirectional, Supervise Simcse | 74.1 | 71.3 |
Table 6: Ablation study on the training methods of LLM caption contrastive finetuning in Stage 1. red text indicates content that differs from the default setting.
| 第一阶段 | 平均 | |
|---|---|---|
| I2T | T2I | |
| Lora, AvgPool, Bidirectional, Supervise Simcse | 80.4 | 77.9 |
| Lora, AvgPool, Bidirectional, Un-supervise Simcse | 59.2 | 57.7 |
| Lora, AvgPool, Bidirectional, MNTP | 70.1 | 67.0 |
| Lora, AvgPool, Bidirectional, MNTP, Supervise Simcse | 79.7 | 77.2 |
| Lora, AvgPool, Casual, Supervise Simcse | 80.0 | 77.5 |
| Lora, EOS, Bidirectional, Supervise Simcse | 80.0 | 77.3 |
| Frozen, Linear Adaptor, AvgPool, Bidirectional, Supervise Simcse | 74.1 | 71.3 |
表 6：第一阶段 LLM 标题对比微调训练方法的消融。红色文字标出与默认设置不同的内容。
| Method | Avg | |
|---|---|---|
| I2T | T2I | |
| CLIP | 74.4 | 72.0 |
| Directly Finetune (50%) | 74.5 | 72.3 |
| bge-en-icl | 78.9 | 78.2 |
| LLM2Vec-Llama-3-8B | 81.4 | 80.2 |
| NV-Embed-v2 | 81.4 | 79.9 |
| VLM2Vec | 78.2 | 77.1 |
| bge-m3-XLM-R | 65.0 | 63.6 |
| jina-v3-XLM-R | 73.6 | 71.0 |
| e5 (XLM-R) | 74.0 | 71.7 |
| Qwen2.5-0.5B-CC | 75.6 | 73.0 |
| Llama-3.2-1B-CC | 80.4 | 77.9 |
| Llama-3-8B-CC | 83.4 | 80.9 |
| DeepSeek-R1-Distill-Llama-8B-CC | 83.5 | 80.5 |
| Llama-3.1-8B-CC | 84.8 | 81.0 |
| Llama3.1-8B | 66.5 | 62.5 |
Table 7: Ablation for using different text encoder. ”-CC” indicates encoders with caption contrastive fine-tuning.
| 方法 | 平均 | |
|---|---|---|
| I2T | T2I | |
| CLIP | 74.4 | 72.0 |
| 直接微调 (50%) | 74.5 | 72.3 |
| bge-en-icl | 78.9 | 78.2 |
| LLM2Vec-Llama-3-8B | 81.4 | 80.2 |
| NV-Embed-v2 | 81.4 | 79.9 |
| VLM2Vec | 78.2 | 77.1 |
| bge-m3-XLM-R | 65.0 | 63.6 |
| jina-v3-XLM-R | 73.6 | 71.0 |
| e5 (XLM-R) | 74.0 | 71.7 |
| Qwen2.5-0.5B-CC | 75.6 | 73.0 |
| Llama-3.2-1B-CC | 80.4 | 77.9 |
| Llama-3-8B-CC | 83.4 | 80.9 |
| DeepSeek-R1-Distill-Llama-8B-CC | 83.5 | 80.5 |
| Llama-3.1-8B-CC | 84.8 | 81.0 |
| Llama3.1-8B | 66.5 | 62.5 |
表 7：使用不同文本编码器的消融。“-CC” 表示经过标题对比微调的编码器。
4.3 Ablation Study: Stage-1 Caption Contrastive Fine-tuning
4.3 消融实验：第一阶段标题对比微调
For simplicity, we select Llama 3.1 1B as the default LLM to investigate different design choices for caption contrastive fine-tuning in this section. We use a subset of the CC 3M dataset for caption contrastive fine-tuning. Additionally, we explore augmenting the training data with the Wikitext-103 dataset (Merity et al. 2016) for MNTP training and the E5 dataset (Springer et al. 2024) as supplementary pure-text data for caption contrastive fine-tuning.
为简化起见，本节默认用 Llama 3.1 1B 作为 LLM，考察标题对比微调的不同设计选择。标题对比微调使用 CC 3M 的一个子集。此外，我们还尝试用 Wikitext-103（Merity et al. 2016）扩充 MNTP 训练数据，并用 E5 数据集（Springer et al. 2024）作为标题对比微调的补充纯文本数据。
| Stage1 | Stage2 | Avg | |
|---|---|---|---|
| I2T | T2I | ||
| – | – | 78.3 | 75.5 |
| – | Linear() | 79.2 | 76.7 |
| – | Linear() | 80.1 | 76.8 |
| – | Linear() | 80.4 | 77.9 |
| Linear() | Linear() | 80.5 | 77.7 |
| – | Transformer() | 80.2 | 77.3 |
| Transformer() | Transformer() | 80.5 | 77.3 |
Table 8: Ablation experiments for adaptor design on Stage 1 caption contrastive finetune and Stage 2 LLM2CLIP post training. The 4-layer Linear Adaptor has 67.1M parameters, while the single layer Transformer Adaptor has 67.6M.
| 第一阶段 | 第二阶段 | 平均 | |
|---|---|---|---|
| I2T | T2I | ||
| – | – | 78.3 | 75.5 |
| – | Linear() | 79.2 | 76.7 |
| – | Linear() | 80.1 | 76.8 |
| – | Linear() | 80.4 | 77.9 |
| Linear() | Linear() | 80.5 | 77.7 |
| – | Transformer() | 80.2 | 77.3 |
| Transformer() | Transformer() | 80.5 | 77.3 |
表 8：第一阶段标题对比微调与第二阶段 LLM2CLIP 后期训练中适配器设计的消融。四层线性适配器有 6710 万参数，单层 Transformer 适配器有 6760 万参数。
Architecture Design Ablation. We examine various adaptor architecture in Table A7. Comparing rows 4 and 5, as well as rows 6 and 7, we observe that neither Linear nor Transformer adaptors significantly impact the performance of subsequent CLIP fine-tuning. Therefore, we default to using no adaptor in Stage 1.
结构设计消融。 我们在表 A7 中考察了多种适配器结构。比较第 4、5 行以及第 6、7 行可以看到，无论是线性适配器还是 Transformer 适配器，对后续 CLIP 微调的性能都没有显著影响。因此，第一阶段默认不使用适配器。
| Methods | Training Loss | Testing Text Encoder | Average | ||
|---|---|---|---|---|---|
| I2T | T2I | ||||
| CLIP | CL(CLIP-T, CLIP-V) | CLIP-T | 74.4 | 72.0 | |
| Directly Finetune | CL(CLIP-T, CLIP-V) | CLIP-T | 74.5 | 72.3 | |
| +LLM2CLIP | a) | CL(LLM, CLIP) | LLM | 83.9 | 82.1 |
| b) | CL(LLM, CLIP-V)+ CL(CLIP-T, CLIP-V) | CLIP-T | 74.4 | 72.0 | |
| LLM | 83.6 | 81.8 | |||
| c) | CL(LLM, CLIP-V) +CL(CLIP-T, CLIP-V) +CL(CLIP-T, LLM) | CLIP-T | 74.0 | 72.1 | |
| LLM | 83.7 | 81.4 | |||
| d) | CL(LLM, CLIP-V) +CL(CLIP-T, CLIP-V)+ CL(Cat(CLIP-T, LLM), CLIP-V) | CLIP-T | 74.8 | 71.3 | |
| LLM | 83.8 | 82.3 | |||
| Cat(CLIP-T, LLM) | Cat(CLIP-T, | LLM) | 84.7 | 82.8 | |
| Cat(CLIP-T, | |||||
| LLM) |
| Cat(CLIP-T, |
|---|
| LLM) |
Table 9: LLM2CLIP Training Method Analysis. We experimented with various possibilities for fine-tuning between the LLM and the pretrained CLIP model’s Vision Encoder (CLIP-V) and Text Encoder (CLIP-T). ”CL” denotes the contrastive learning loss, and ”Cat” means concatenation. Items a–d correspond to the a–d in Section 3.2.
| 方法 | 训练损失 | 测试用文本编码器 | 平均 I2T | 平均 T2I |
|---|---|---|---|---|
| CLIP | CL(CLIP-T, CLIP-V) | CLIP-T | 74.4 | 72.0 |
| 直接微调 | CL(CLIP-T, CLIP-V) | CLIP-T | 74.5 | 72.3 |
| +LLM2CLIP a) | CL(LLM, CLIP) | LLM | 83.9 | 82.1 |
| b) | CL(LLM, CLIP-V)+ CL(CLIP-T, CLIP-V) | CLIP-T | 74.4 | 72.0 |
| LLM | 83.6 | 81.8 | ||
| c) | CL(LLM, CLIP-V) +CL(CLIP-T, CLIP-V) +CL(CLIP-T, LLM) | CLIP-T | 74.0 | 72.1 |
| LLM | 83.7 | 81.4 | ||
| d) | CL(LLM, CLIP-V) +CL(CLIP-T, CLIP-V)+ CL(Cat(CLIP-T, LLM), CLIP-V) | CLIP-T | 74.8 | 71.3 |
| LLM | 83.8 | 82.3 | ||
| Cat(CLIP-T, LLM) | 84.7 | 82.8 |
表 9：LLM2CLIP 训练方法分析。我们试验了在 LLM 与已预训练 CLIP 的视觉编码器（CLIP-V）、文本编码器（CLIP-T）之间微调的多种可能。“CL” 表示对比学习损失，“Cat” 表示拼接。条目 a–d 对应第 3.2 节中的 a–d。
Training Method Ablation. We conduct an ablation study of Stage-1 LLM caption contrastive fine-tuning methods in Table A5, observing the following key insights: 1. Freezing LLM gradients and relying solely on adaptors yields poor performance, indicating that enabling LoRA is essential. 2. SimCSE is identified as the most critical loss function. MNTP does not significantly enhance SimCSE, and relying solely on MNTP without SimCSE severely deteriorates performance. 3. Supervised SimCSE substantially outperforms unsupervised SimCSE. 4. The performance difference between causal and bidirectional attention is negligible.
训练方法消融。 我们在表 A5 中对第一阶段 LLM 标题对比微调方法做了消融，主要观察如下：1. 冻结 LLM 梯度、只靠适配器，效果很差，说明启用 LoRA 是必要的。2. SimCSE 是最关键的损失函数。MNTP 并不能明显增强 SimCSE；若只用 MNTP、不用 SimCSE，性能会严重下降。3. 监督式 SimCSE 明显优于无监督 SimCSE。4. 因果注意力与双向注意力之间的性能差异可以忽略。
Ablation across different LLM backbones. Table A6 compares a variety of text-embedding models plugged into our pipeline. Our approach delivers substantial gains over all contenders. The line highlighted in red shows that plain Llama 3.1-8B—without our caption-contrastive (CC) fine-tuning—performs very poorly and even harms the original CLIP, underscoring the necessity of the CC stage. We also evaluated several state-of-the-art embedding models, including bge, jina-v3, and the VLM2Vec embedding obtained from a multimodal LLM fine-tune. In every case, the LLM2CLIP variant still achieves the best results, confirming the effectiveness of our training recipe.
不同 LLM 主干的消融。 表 A6 比较了接入我们流程的多种文本嵌入模型。该方法相对所有对照都有明显增益。标红的一行显示：未经我们标题对比（CC）微调的普通 Llama 3.1-8B 表现很差，甚至会损害原来的 CLIP，这说明 CC 阶段是必要的。我们也评估了若干先进嵌入模型，包括 bge、jina-v3，以及由多模态 LLM 微调得到的 VLM2Vec 嵌入。在所有情形中，LLM2CLIP 变体仍然最好，确认了我们训练方案的有效性。
4.4 Ablation Study: Stage-2 LLM2CLIP
4.4 消融实验：第二阶段 LLM2CLIP
Adaptor Design. As shown in Table A7, we explore adaptor structures in the Stage-2 LLM2CLIP cross-modal pre-training. Comparing Linear () and Transformer (), their performances are relatively similar, and thus, we select the simpler Linear adaptor structure. Furthermore, in our layer-wise ablation of Linear adaptors, we find that increasing the adaptor size indeed improves performance. Specifically, adaptor performance improves progressively when transitioning from no adaptor to 1, 2 and 4-layer.
适配器设计。 如表 A7 所示，我们在第二阶段 LLM2CLIP 的跨模态预训练中考察了适配器结构。比较 Linear（）与 Transformer（），二者表现较为接近，因此我们选择更简单的线性适配器。在线性适配器的层数消融中，增大适配器确实能提高性能。具体而言，从不用适配器到 1、2、4 层，效果逐步提升。
Training-method ablation. We explored multiple ways to integrate an LLM during CLIP fine-tuning (implementation details appear in Table A8. Comparing experiments (a)–(b) and (b)–(c) shows that neither (i) optimising two separate losses nor (ii) explicitly aligning the two text encoders brings any meaningful benefit. This observation motivates our choice to replace, rather than reuse, CLIP’s original text encoder in the main pipeline. Experiment (d)—which performs contrastive learning on the concatenated outputs of both encoders—does deliver a noticeable uplift; for the sake of simplicity, we do not adopt it as our default, but regard it as a promising future work.
训练方法消融。 我们探索了在 CLIP 微调中接入 LLM 的多种方式（实现细节见表 A8）。比较实验（a）与（b）、以及（b）与（c）可以看到：分别优化两项损失，或显式对齐两个文本编码器，都没有带来有意义的收益。这一观察促使我们在主流程中选择替换、而不是复用 CLIP 原来的文本编码器。实验（d）对两个编码器的拼接输出做对比学习，确实带来了明显提升；为保持简单，我们没有把它作为默认，但认为这是有前景的后续方向。
5 Conclusion
5 结论
We have revisited the problem of upgrading CLIP with large-language-model knowledge and proposed a lightweight pipeline that first turns an LLM into a discriminative caption-embedding module and then inserts it into CLIP via a cost-effective fine-tuning stage. With only million-scale training pairs and a compute budget essentially identical to standard CLIP fine-tuning, our method delivers sizeable gains—on top of already SOTA baselines—in image–text retrieval, image classification (zero-shot & linear probe), zero-shot/supervised detection/segmentation, and even as the visual encoder inside a multimodal LLM.
我们重新审视了用大语言模型知识升级 CLIP 的问题，并提出一条轻量流程：先把 LLM 变成可分辨的标题嵌入模块，再通过成本较低的微调阶段把它插入 CLIP。只需百万量级的训练对，计算预算也与标准 CLIP 微调基本相当，该方法就能在已经很强的基线之上，在图文检索、图像分类（零样本与线性探测）、零样本/监督检测与分割，乃至作为多模态大模型内部的视觉编码器等任务上，带来可观提升。
Acknowledgments
致谢
This work is partially supported by the National Natural Science Foundation of China (NSFC Granted No. 62276190).
本工作部分得到国家自然科学基金（批准号 62276190）的资助。
References
参考文献
Bai et al. (2023) J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou Qwen-vl: a frontier large vision-language model with versatile abilities . arXiv preprint arXiv:2308.12966 . Cited by: §1 .
BehnamGhader et al. (2024) P. BehnamGhader, V. Adlakha, M. Mosbach, D. Bahdanau, N. Chapados, and S. Reddy LLM2Vec: Large language models are secretly powerful text encoders . arXiv preprint . External Links: Link Cited by: §1 , §3.1 .
Caesar et al. (2018) H. Caesar, J. Uijlings, and V. Ferrari Coco-stuff: thing and stuff classes in context . In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1209–1218 . Cited by: §C.2 .
Chen et al. (2025) H. Chen, L. Wang, N. Yang, Y. Zhu, Z. Zhao, F. Wei, and Z. Dou Mme5: improving multimodal multilingual embeddings via high-quality synthetic data . arXiv preprint arXiv:2502.08468 . Cited by: §1 .
Chen et al. (2023) L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin Sharegpt4v: improving large multi-modal models with better captions . arXiv preprint arXiv:2311.12793 . Cited by: §2 , §3.2 , §4.2 .
Chen et al. (2024) Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 24185–24198 . Cited by: Table 1 .
Cordts et al. (2016) M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele The cityscapes dataset for semantic urban scene understanding . In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3213–3223 . Cited by: §C.2 .
Dai et al. (2023) W. Dai, J. Li, D. Li, A. Meng Huat Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning . arXiv e-prints , pp. arXiv:2305.06500 . External Links: Document , 2305.06500 Cited by: §3.2 .
Esser et al. (2024) P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis . In Forty-first international conference on machine learning , Cited by: §1 .
Everingham et al. (2015) M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman The pascal visual object classes challenge: a retrospective . International journal of computer vision 111 ( 1 ), pp. 98–136 . Cited by: §C.2 .
Fan et al. (2024) L. Fan, D. Krishnan, P. Isola, D. Katabi, and Y. Tian Improving clip training with language rewrites . Advances in Neural Information Processing Systems 36 . Cited by: §2 .
Goyal et al. (2017) Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh Making the v in vqa matter: elevating the role of image understanding in visual question answering . In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 6904–6913 . Cited by: §C.2 .
Günther et al. (2023) M. Günther, J. Ong, I. Mohr, A. Abdessalem, T. Abel, M. K. Akram, S. Guzman, G. Mastrapas, S. Sturua, B. Wang, et al. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents . arXiv preprint arXiv:2310.19923 . Cited by: §2 .
Gurari et al. (2018) D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham Vizwiz grand challenge: answering visual questions from blind people . In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 3608–3617 . Cited by: §C.2 .
He et al. (2017) K. He, G. Gkioxari, P. Dollár, and R. Girshick Mask r-cnn . In Proceedings of the IEEE international conference on computer vision , pp. 2961–2969 . Cited by: §C.2 .
Hudson and Manning (2019) D. A. Hudson and C. D. Manning Gqa: a new dataset for real-world visual reasoning and compositional question answering . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 6700–6709 . Cited by: §C.2 .
Jang et al. (2024) Y. K. Jang, J. Kang, Y. J. Lee, and D. Kim MATE: meet at the embedding–connecting images with long texts . arXiv preprint arXiv:2407.09541 . Cited by: §2 .
Jia et al. (2021) C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. Le, Y. Sung, Z. Li, and T. Duerig Scaling up visual and vision-language representation learning with noisy text supervision . In International conference on machine learning , pp. 4904–4916 . Cited by: Table 1 .
Jiang et al. (2024) Z. Jiang, R. Meng, X. Yang, S. Yavuz, Y. Zhou, and W. Chen VLM2Vec: training vision-language models for massive multimodal embedding tasks . ArXiv abs/2410.05160 . External Links: Link Cited by: §1 , Table 1 .
Kong et al. (2024) W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: a systematic framework for large video generative models . arXiv preprint arXiv:2412.03603 . Cited by: §1 .
Koukounas et al. (2024a) A. Koukounas, G. Mastrapas, M. Günther, B. Wang, S. Martens, I. Mohr, S. Sturua, M. K. Akram, J. F. Martínez, S. Ognawala, et al. Jina clip: your clip model is also your text retriever . arXiv preprint arXiv:2405.20204 . Cited by: §2 .
Koukounas et al. (2024b) A. Koukounas, G. Mastrapas, B. Wang, M. K. Akram, S. Eslami, M. Günther, I. Mohr, S. Sturua, S. Martens, N. Wang, et al. Jina-clip-v2: multilingual multimodal embeddings for text and images . arXiv preprint arXiv:2412.08802 . Cited by: Table 1 .
Lan et al. (2017) W. Lan, X. Li, and J. Dong Fluency-guided cross-lingual image captioning . In Proceedings of the 25th ACM international conference on Multimedia , pp. 1549–1557 . Cited by: §C.2 .
Lee et al. (2024) C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping Nv-embed: improved techniques for training llms as generalist embedding models . arXiv preprint arXiv:2405.17428 . Cited by: §1 , §3.1 .
Li et al. (2024a) B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li LLaVA-onevision: easy visual task transfer . arXiv preprint arXiv:2408.03326 . Cited by: §1 .
Li et al. (2023a) B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan Seed-bench: benchmarking multimodal llms with generative comprehension . arXiv preprint arXiv:2307.16125 . Cited by: §C.2 .
Li et al. (2022a) J. Li, D. Li, C. Xiong, and S. Hoi Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation . In International conference on machine learning , pp. 12888–12900 . Cited by: Table 1 .
Li et al. (2024b) X. Li, H. Tu, M. Hui, Z. Wang, B. Zhao, J. Xiao, S. Ren, J. Mei, Q. Liu, H. Zheng, et al. What if we recaption billions of web images with llama-3? . arXiv preprint arXiv:2406.08478 . Cited by: §2 .
Li et al. (2019) X. Li, C. Xu, X. Wang, W. Lan, Z. Jia, G. Yang, and J. Xu COCO-cn for cross-lingual image tagging, captioning, and retrieval . IEEE Transactions on Multimedia 21 ( 9 ), pp. 2347–2360 . Cited by: §C.2 .
Li et al. (2022b) Y. Li, H. Mao, R. Girshick, and K. He Exploring plain vision transformer backbones for object detection . In European conference on computer vision , pp. 280–296 . Cited by: §C.2 .
Li et al. (2023b) Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen Evaluating object hallucination in large vision-language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pp. 292–305 . Cited by: §C.2 .
Lin et al. (2014) T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick Microsoft coco: common objects in context . In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pp. 740–755 . Cited by: Table A1 , §4.2 .
Liu et al. (2023a) H. Liu, C. Li, Y. Li, and Y. J. Lee Improved baselines with visual instruction tuning . arXiv:2310.03744 . Cited by: §3.2 .
Liu et al. (2023b) H. Liu, C. Li, Q. Wu, and Y. J. Lee Visual instruction tuning . NeurIPS . Cited by: §4.2 .
Liu et al. (2024) Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: is your multi-modal model an all-around player? . In European conference on computer vision , pp. 216–233 . Cited by: §C.2 .
Lu et al. (2022) P. Lu, S. Mishra, T. Xia, L. Qiu, K. Chang, S. Zhu, O. Tafjord, P. Clark, and A. Kalyan Learn to explain: multimodal reasoning via thought chains for science question answering . Advances in Neural Information Processing Systems 35 , pp. 2507–2521 . Cited by: §C.2 .
Merity et al. (2016) S. Merity, C. Xiong, J. Bradbury, and R. Socher Pointer sentinel mixture models . arXiv: Computation and Language,arXiv: Computation and Language ( en-US ). Cited by: §4.3 .
Muennighoff et al. (2022) N. Muennighoff, N. Tazi, L. Magne, and N. Reimers Mteb: massive text embedding benchmark . arXiv preprint arXiv:2210.07316 . Cited by: §1 .
Onoe et al. (2024) Y. Onoe, S. Rane, Z. Berger, Y. Bitton, J. Cho, R. Garg, A. Ku, Z. Parekh, J. Pont-Tuset, G. Tanzer, S. Wang, and J. Baldridge DOCCI: descriptions of connected and contrasting images . External Links: 2404.19753 , Link Cited by: §4.2 .
Ordonez et al. (2011) V. Ordonez, G. Kulkarni, and T. Berg Im2Text: describing images using 1 million captioned photographs . In Advances in Neural Information Processing Systems , J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.) , Vol. 24 , pp. . External Links: Link Cited by: §4.2 .
Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision . In International conference on machine learning , pp. 8748–8763 . Cited by: §1 , Table 1 , Table 1 , Table 1 .
Singh et al. (2019) A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach Towards vqa models that can read . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 8317–8326 . Cited by: §C.2 .
Springer et al. (2024) J. M. Springer, S. Kotha, D. Fried, G. Neubig, and A. Raghunathan Repetition improves language model embeddings . arXiv preprint arXiv:2402.15449 . Cited by: §C.1 , §3.1 , §4.3 .
Sun et al. (2023a) Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao EVA-CLIP: Improved Training Techniques for CLIP at Scale . arXiv e-prints , pp. arXiv:2303.15389 . External Links: Document , 2303.15389 Cited by: §2 .
Sun et al. (2023b) Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao Eva-clip: improved training techniques for clip at scale . arXiv preprint arXiv:2303.15389 . Cited by: Table 1 , Table 1 , Table 1 .
Thapliyal et al. (2022) A. V. Thapliyal, J. P. Tuset, X. Chen, and R. Soricut Crossmodal-3600: a massively multilingual multimodal evaluation dataset . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 715–729 . Cited by: §C.2 .
Tschannen et al. (2025) M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. Siglip 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features . arXiv preprint arXiv:2502.14786 . Cited by: §1 , §1 , Table 1 .
Urbanek et al. (2024) J. Urbanek, F. Bordes, P. Astolfi, M. Williamson, V. Sharma, and A. Romero-Soriano A picture is worth more than 77 text tokens: evaluating clip-style models on dense captions . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 26700–26709 . Cited by: §2 .
Vouitsis et al. (2023) N. Vouitsis, Z. Liu, S. K. Gorti, V. Villecroze, J. C. Cresswell, G. Yu, G. Loaiza-Ganem, and M. Volkovs Data-efficient multimodal fusion on a single gpu . 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 27229–27241 . External Links: Link Cited by: §4.1 .
Wu et al. (2023) S. Wu, W. Zhang, L. Xu, S. Jin, X. Li, W. Liu, and C. C. Loy Clipself: vision transformer distills itself for open-vocabulary dense prediction . arXiv preprint arXiv:2310.01403 . Cited by: §C.2 .
Xu et al. (2023) H. Xu, S. Xie, X. E. Tan, P. Huang, R. Howes, V. Sharma, S. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer Demystifying clip data . arXiv preprint arXiv:2309.16671 . Cited by: Table 1 , Table 1 .
Yao et al. (2021) L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu Filip: fine-grained interactive language-image pre-training . arXiv preprint arXiv:2111.07783 . Cited by: Appendix D .
Young et al. (2014) P. Young, A. Lai, M. Hodosh, and J. Hockenmaier From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions . Transactions of the Association for Computational Linguistics 2 , pp. 67–78 . Cited by: §4.2 .
Zhai et al. (2023) X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer Sigmoid loss for language image pre-training . In Proceedings of the IEEE/CVF international conference on computer vision , pp. 11975–11986 . Cited by: Table 1 , Table 1 , Table 1 , Table 1 .
Zhang et al. (2024a) B. Zhang, P. Zhang, X. Dong, Y. Zang, and J. Wang Long-clip: unlocking the long-text capability of clip . arXiv preprint arXiv:2403.15378 . Cited by: Table 1 , Table 1 , §2 , §4.2 , §4.2 .
Zhang et al. (2024b) J. Zhang, X. Qu, T. Zhu, and Y. Cheng Clip-moe: towards building mixture of experts for clip with diversified multiplet upcycling . arXiv preprint arXiv:2409.19291 . Cited by: §4.2 .
Zhang et al. (2025) L. Zhang, Q. Yang, and A. Agrawal Assessing and learning alignment of unimodal vision and language models . In Proceedings of the Computer Vision and Pattern Recognition Conference , pp. 14604–14614 . Cited by: Table 1 .
Zheng et al. (2024) K. Zheng, Y. Zhang, W. Wu, F. Lu, S. Ma, X. Jin, W. Chen, and Y. Shen DreamLIP: language-image pre-training with long captions . ArXiv abs/2403.17007 . External Links: Link Cited by: §2 , §3.1 .
Zhou et al. (2017) B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba Scene parsing through ade20k dataset . In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 633–641 . Cited by: §C.2 .
Zhou et al. (2022) C. Zhou, C. C. Loy, and B. Dai Extract free dense labels from clip . In European conference on computer vision , pp. 696–712 . Cited by: §C.2 .
Appendix A Reproducibility Statement
附录 A 可复现性声明
To ensure the reproducibility of our work, we are committed to making all the training and testing code, along with the datasets used in our experiments, publicly available in our code repository. Additionally, we will open-source our LLM2CLIP-enhanced versions of SigLip2, EVA02, and OpenAI’s CLIP to facilitate applications within the open-source community requiring high-quality embedding models.
为确保工作可复现，我们承诺在代码仓库中公开全部训练与测试代码，以及实验所用数据集。此外，我们将开源经 LLM2CLIP 增强的 SigLip2、EVA02 和 OpenAI CLIP 版本，以便开源社区中需要高质量嵌入模型的应用使用。
We will provide comprehensive documentation alongside the codebase to facilitate straightforward replication of our experiments. All hyperparameter settings, training configurations, and preprocessing steps will be thoroughly documented and made accessible to the research community.
我们还将随代码库提供较为完整的文档，便于直接复现实验。全部超参数设置、训练配置和预处理步骤都会详细记录，并向研究社区开放。
Additionally, we will include detailed environment setup instructions, dependency requirements, and step-by-step execution guidelines to minimize barriers for reproduction. Our commitment to open science ensures that researchers can easily validate, extend, and build upon our contributions.
另外，我们将提供详细的环境配置说明、依赖要求和逐步执行指南，以降低复现门槛。我们对开放科学的承诺，是为了让研究者能够较容易地验证、扩展并建立在我们的工作之上。
Appendix B Models and Checkpoints
附录 B 模型与检查点
Table A2 lists all the models and checkpoints used in our experiments, ensuring transparency and facilitating reproducibility. It includes both vision and text encoders, with corresponding links to their publicly available implementations or weights.
表 A2 列出了实验中用到的全部模型和检查点，以便保证透明并利于复现。其中既包括视觉编码器，也包括文本编码器，并给出对应的公开实现或权重链接。
Appendix C Experiment Details
附录 C 实验细节
C.1 Training Setup
C.1 训练设置
Settings for Stage-1
We choose to use LLaMA 3.1 8B as the LLM integrated into CLIP pre-training, removing the causal attention mask to enable bidirectional attention. No additional adapters are used in caption contrastive finetune, and average pooling is employed to obtain the sentence token embedding. MNTP is not used by default. Additionally, to ensure the generalizability of LLM textual differentiation capabilities, we incorporated a 1.5M paired pure-text dataset curated by Echo Embeddings (Springer et al. 2024) into our contrastive training process.Our models are trained with LoRA (, ), bfloat16 quantization, gradient checkpointing, and FlashAttention-2 for memory efficiency. Training uses the AdamW optimizer (learning rate , 300-step linear warm-up), sequence length 512, effective batch size 2048, and 1 epoch over 30M(1.5m paired pure-text and captions dataset) samples on 32 NVIDIA A100 GPUs.
第一阶段设置
我们选择把 LLaMA 3.1 8B 作为接入 CLIP 预训练的 LLM，去掉因果注意力掩码以启用双向注意力。标题对比微调不加额外适配器，用平均池化得到句子词元嵌入。默认不使用 MNTP。此外，为使 LLM 的文本区分能力更具普遍性，我们把 Echo Embeddings（Springer et al. 2024）整理的 150 万对纯文本数据纳入对比训练。模型训练使用 LoRA（，）、bfloat16 量化、梯度检查点，以及 FlashAttention-2 以节省显存。优化器为 AdamW（学习率 ，300 步线性预热），序列长度 512，有效批量 2048，在 3000 万样本（150 万对纯文本与标题数据）上训练 1 个周期，使用 32 块 NVIDIA A100 GPU。
Settings for Stage-2
We employed the AdamW optimizer, utilizing a total batch size of 4096, and set the learning rate to 1e-5 with a cosine decay rate of 0.05. For every training step we randomly sampled a single caption from the DreamLIP caption data. All models were trained for 4 epochs by default. The embedding dimension for both modalities was set to 1280.
第二阶段设置
我们使用 AdamW 优化器，总批量为 4096，学习率设为 1e-5，余弦衰减率为 0.05。每一步训练都从 DreamLIP 标题数据中随机抽取一条标题。所有模型默认训练 4 个周期。两个模态的嵌入维度都设为 1280。
C.2 Evaluation
C.2 评测
Multilingual Retrieval
We evaluate our models on three cross-modal retrieval benchmarks that support multilingual and cross-lingual evaluation: Flickr-CN, COCO-CN, and XM3600. The results are shown in Table 2 of the main text..
多语检索
我们在三个支持多语和跨语评测的跨模态检索基准上评估模型：Flickr-CN、COCO-CN 和 XM3600。结果见正文表 2。
Flickr-CN (Lan et al. 2017) and COCO-CN (Li et al. 2019) are Chinese adaptations of the popular Flickr30K and MS-COCO datasets, respectively. These datasets contain images paired with Chinese captions, enabling evaluation of cross-modal retrieval performance in Chinese.
Flickr-CN（Lan et al. 2017）和 COCO-CN（Li et al. 2019）分别是常用 Flickr30K 与 MS-COCO 数据集的中文改编版。这些数据集把图像与中文标题配对，用于评估中文跨模态检索表现。
Crossmodal-3600(XM-3600)
(Thapliyal et al. 2022) is a geographically-diverse multilingual dataset comprising 3,600 images annotated with human-generated reference captions in 36 languages . The images were carefully selected from across the world, covering regions where the 36 languages are natively spoken.
Crossmodal-3600（XM-3600）
该数据集（Thapliyal et al. 2022）是一个地理分布多样的多语数据集，包含 3600 张图像，配有 36 种语言的人工参考标题。图像从世界各地仔细选取，覆盖这 36 种语言的母语使用地区。
Open-Vocabulary Semantic Segmentation
CLIP-like models enable patch-level semantic matching by aligning visual and textual representations, facilitating open-vocabulary semantic segmentation. Following MaskCLIP (Zhou et al. 2022), we compute cosine similarity between patch embeddings and sentence embeddings to generate segmentation masks.
开放词汇语义分割
CLIP 类模型通过对齐视觉与文本表征，实现补丁级语义匹配，从而支持开放词汇语义分割。我们沿用 MaskCLIP（Zhou et al. 2022），计算补丁嵌入与句子嵌入的余弦相似度，以生成分割掩码。
We evaluated our models on four benchmark datasets: ADE20K (Zhou et al. 2017), COCO-Stuff164k (Caesar et al. 2018), VOC20 (Everingham et al. 2015), and Cityscapes (Cordts et al. 2016) using mean Intersection over Union (mIoU) to measure segmentation accuracy. The results are shown in Table 4 of the main text.
我们在四个基准数据集上评估模型：ADE20K（Zhou et al. 2017）、COCO-Stuff164k（Caesar et al. 2018）、VOC20（Everingham et al. 2015）和 Cityscapes（Cordts et al. 2016），用平均交并比（mIoU）衡量分割准确率。结果见正文表 4。
Open-Vocabulary Object Detection
To assess the fine-grained localization capabilities of our approach, we integrate LLM2CLIP as the backbone network for open-vocabulary detection tasks. Following established protocols from prior work (Wu et al. 2023), we adopt a two-stage detection framework for evaluation. We conduct experiments on the OV-COCO benchmark and report box Average Precision (AP) at IoU threshold 0.5 across three categories: base classes (seen during training), novel classes (unseen during training), and all classes combined. The novel class performance particularly demonstrates the model’s generalization ability to previously unseen object categories. The strong results on novel classes, as shown in Table 4 of the main paper, demonstrate our method’s effectiveness in recognizing and localizing previously unseen objects, highlighting its robust generalization capabilities for open-vocabulary detection tasks.
开放词汇目标检测
为评估方法的细粒度定位能力，我们把 LLM2CLIP 作为开放词汇检测任务的骨干网络。按照已有工作（Wu et al. 2023）的既定协议，采用两阶段检测框架进行评测。我们在 OV-COCO 基准上做实验，并报告 IoU 阈值为 0.5 时三类框平均精度（AP）：训练中见过的基类、训练中未见的新类，以及全部类别。新类上的表现尤其能够说明模型对未见物体类别的泛化能力。如正文表 4 所示，新类上的较强结果说明该方法能够识别并定位此前未见的物体，体现了其在开放词汇检测任务上较为稳健的泛化能力。
Supervised Instance Segmentation.
To evaluate the dense prediction capabilities of our method on supervised instance segmentation tasks, we employ LLM2CLIP as the backbone for ViTDet (Li et al. 2022b) within the Mask R-CNN (He et al. 2017) framework. The models are trained on the COCO dataset using 4 A100 machines with 2 images per GPU, resulting in a total batch size of 64. We train for 25 epochs using the AdamW optimizer with a learning rate of 0.0001, , , and a weight decay of 0.1.
监督实例分割。
为评估方法在监督实例分割任务上的稠密预测能力，我们在 Mask R-CNN（He et al. 2017）框架中，把 LLM2CLIP 用作 ViTDet（Li et al. 2022b）的骨干。模型在 COCO 上训练，使用 4 台 A100 机器，每块 GPU 2 张图像，总批量为 64。训练 25 个周期，优化器为 AdamW，学习率 0.0001，，，权重衰减 0.1。
Our approach achieves results with 45.6 mAP for object detection and 38.7 mAP for instance segmentation, as shown in Table 4 of the main text, demonstrating the effectiveness of our cross-modal pre-training in improving dense prediction tasks. These results highlight the model’s ability to transfer learned representations to complex downstream tasks requiring both object localization and precise pixel-level segmentation.
如正文表 4 所示，我们的方法在目标检测上达到 45.6 mAP，在实例分割上达到 38.7 mAP，说明跨模态预训练有助于改进稠密预测任务。这些结果也表明，所学表征能够迁移到既需要物体定位、又需要精确像素级分割的复杂下游任务。
Comparisons on General Multimodal Benchmarks.
We evaluate LLM2CLIP (EVA02-L-336) as a visual feature extractor for multimodal large language models, comparing it against the standard CLIP baseline. Our experiments are conducted using LLaVA-v1.5-7B, which originally employs CLIP as its visual encoder. To ensure fair and rigorous comparison, we maintain all parameter configurations identical to the original LLaVA setup and utilize the same training data provided by the LLaVA framework. The comprehensive evaluation results on various multimodal understanding benchmarks((i.e., VQA (Goyal et al. 2017) (Hudson and Manning 2019) (Gurari et al. 2018) (Lu et al. 2022) (Singh et al. 2019), POPE (Li et al. 2023b), MMBench (Liu et al. 2024), SEED (Li et al. 2023a)) are presented in Table 5 of the main text.
通用多模态基准上的比较。
我们把 LLM2CLIP（EVA02-L-336）作为多模态大语言模型的视觉特征提取器，与标准 CLIP 基线比较。实验使用原本以 CLIP 为视觉编码器的 LLaVA-v1.5-7B。为使比较公平、严格，我们保持全部参数配置与原版 LLaVA 相同，并使用 LLaVA 框架提供的同一份训练数据。在各类多模态理解基准（即 VQA（Goyal et al. 2017）（Hudson and Manning 2019）（Gurari et al. 2018）（Lu et al. 2022）（Singh et al. 2019）、POPE（Li et al. 2023b）、MMBench（Liu et al. 2024）、SEED（Li et al. 2023a））上的综合评测结果，见正文表 5。
Figure A1: Comprehensive evaluation results on cross-modal retrieval tasks. The radar plot illustrates the performance of different models across multiple benchmarks. Our methods consistently achieve higher scores across all evaluation metrics compared to baseline models.
图 A1：跨模态检索任务的综合评测结果。雷达图展示了不同模型在多个基准上的表现。与基线模型相比，我们的方法在全部评测指标上始终取得更高分数。
Appendix D More Analysis
附录 D 更多分析
Quantitative Analysis of the Original LLM’s Feature Separability.
As shown in Table A1, we conducted caption-to-caption retrieval experiments on the COCO 5K test set (each image has five captions; we used the first two captions as positive pairs). We observed that large language models, such as Llama 3 8B, achieve very low accuracy, struggling to distinguish effectively between different captions of the same image. This result highlights the necessity of the embedding-oriented training (caption contrastive fine-tuning) explored in this work for adapting LLMs specifically for the CLIP scenario.
原始 LLM 特征可分性的定量分析。
如表 A1 所示，我们在 COCO 5K 测试集上做了标题到标题检索实验（每张图像有五条标题；我们用前两条作为正对）。可以看到，Llama 3 8B 这类大语言模型的准确率很低，难以有效区分同一图像的不同标题。这一结果说明，本工作所探索的面向嵌入的训练（标题对比微调），对于把 LLM 专门适配到 CLIP 场景是必要的。
| Language Model | Top 1 Acc. |
|---|---|
| CLIP-L/14 | 25.2 |
| EVA02-L/14 | 27.1 |
| Llama3-8B | 5.2 |
| Llama3.2-1B | 5.6 |
| Llama3-8B-CC | 29.5 |
| Llama3.2-1B-CC | 29.4 |
Table A1: Comparison of top-1 retrieval accuracy for various language models on the MS COCO (Lin et al. 2014) 5K test set.
| 语言模型 | Top-1 准确率 |
|---|---|
| CLIP-L/14 | 25.2 |
| EVA02-L/14 | 27.1 |
| Llama3-8B | 5.2 |
| Llama3.2-1B | 5.6 |
| Llama3-8B-CC | 29.5 |
| Llama3.2-1B-CC | 29.4 |
表 A1：各语言模型在 MS COCO（Lin et al. 2014）5K 测试集上的 top-1 检索准确率比较。
Table A2: Models and Checkpoints used in our experiments, including both vision and text encoders.
表 A2：实验所用模型与检查点，包括视觉编码器和文本编码器。
Figure A2: Multilingual zero-shot retrieval results on Crossmodal-3600. (a) Multilingual zero-shot image-to-text retrieval (Recall@1). (b) Multilingual zero-shot text-to-image retrieval (Recall@1).
图 A2：Crossmodal-3600 上的多语零样本检索结果。（a）多语零样本图像到文本检索（Recall@1）。（b）多语零样本文本到图像检索（Recall@1）。
| Methods(MLLM’s Caption Ratio) | Flickr | COCO | ShareGPT4V | Urban-1k | DOCCI | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | |
| CLIP | 89.6 | 77.8 | 59.4 | 48.6 | 88.1 | 87.7 | 68.0 | 74.8 | 67.0 | 71.3 | 74.4 | 72.0 |
| Directly Finetune(50%) | 89.3 | 77.8 | 59.3 | 48.5 | 88.3 | 88.2 | 68.5 | 76.0 | 67.2 | 71.2 | 74.5 | 72.3 |
| + LLM2CLIP (100%) | 89.2 | 77.2 | 55.6 | 47.6 | 97.6 | 98.0 | 88.5 | 92.5 | 82.6 | 86.9 | 82.7 | 80.4 |
| + LLM2CLIP (75%) | 92.0 | 81.3 | 61.5 | 52.0 | 97.7 | 97.8 | 85.1 | 92.2 | 84.0 | 87.7 | 84.1 | 82.2 |
| + LLM2CLIP (50%) | 91.9 | 81.7 | 62.0 | 52.7 | 97.2 | 97.6 | 84.6 | 91.3 | 83.7 | 87.3 | 83.9 | 82.1 |
| + LLM2CLIP (25%) | 91.1 | 81.8 | 61.8 | 53.0 | 97.3 | 97.2 | 83.4 | 90.6 | 83.9 | 86.9 | 83.5 | 81.9 |
| + LLM2CLIP (0%) | 91.1 | 81.5 | 60.9 | 53.0 | 96.0 | 94.4 | 72.0 | 85.6 | 82.6 | 85.6 | 80.5 | 80.0 |
Table A3: Ablation study on the proportion of dense captions from different MLLMs. Here, we explore the impact of both original short image captions and synthetic dense captions generated by MLLMs on the performance of LLM2CLIP.
| 方法（MLLM 标题比例） | Flickr | COCO | ShareGPT4V | Urban-1k | DOCCI | 平均 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | |
| CLIP | 89.6 | 77.8 | 59.4 | 48.6 | 88.1 | 87.7 | 68.0 | 74.8 | 67.0 | 71.3 | 74.4 | 72.0 |
| 直接微调(50%) | 89.3 | 77.8 | 59.3 | 48.5 | 88.3 | 88.2 | 68.5 | 76.0 | 67.2 | 71.2 | 74.5 | 72.3 |
| + LLM2CLIP (100%) | 89.2 | 77.2 | 55.6 | 47.6 | 97.6 | 98.0 | 88.5 | 92.5 | 82.6 | 86.9 | 82.7 | 80.4 |
| + LLM2CLIP (75%) | 92.0 | 81.3 | 61.5 | 52.0 | 97.7 | 97.8 | 85.1 | 92.2 | 84.0 | 87.7 | 84.1 | 82.2 |
| + LLM2CLIP (50%) | 91.9 | 81.7 | 62.0 | 52.7 | 97.2 | 97.6 | 84.6 | 91.3 | 83.7 | 87.3 | 83.9 | 82.1 |
| + LLM2CLIP (25%) | 91.1 | 81.8 | 61.8 | 53.0 | 97.3 | 97.2 | 83.4 | 90.6 | 83.9 | 86.9 | 83.5 | 81.9 |
| + LLM2CLIP (0%) | 91.1 | 81.5 | 60.9 | 53.0 | 96.0 | 94.4 | 72.0 | 85.6 | 82.6 | 85.6 | 80.5 | 80.0 |
表 A3：来自不同 MLLM 的稠密标题比例消融。这里考察原始短图像标题与 MLLM 生成的合成稠密标题对 LLM2CLIP 性能的影响。
| Model | Strategy | Training Hour | Batchsize | Flickr | COCO | ShareGPT4V | Urban1k | DOCCI | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | i2t | t2i | ||||
| Llama-3.1-8B | LLM LoRA | 17 | 768 | 90.0 | 80.9 | 60.9 | 50.6 | 97.5 | 96.9 | 88.9 | 90.8 | 82.0 | 84.3 | 83.9 | 80.7 |
| LLM Frozen + Linear Adaptor + Offline-loading | 1.3 | 16384 | 85.2 | 74.6 | 52.3 | 40.8 | 87.7 | 89.6 | 57.5 | 59.2 | 49.9 | 48.3 | 66.5 | 62.5 | |
| Llama-3.1-8B-CC | LLM LoRA | 17 | 704 | 90.5 | 81.8 | 61.5 | 52.4 | 98.1 | 98.2 | 91.0 | 92.8 | 85.7 | 87.4 | 85.4 | 82.5 |
| LLM Frozen + Linear Adaptor | 5.5 | 4096 | 91.9 | 81.7 | 62.0 | 52.7 | 97.2 | 97.6 | 84.6 | 91.3 | 83.7 | 87.3 | 83.9 | 82.1 | |
| LLM Frozen + Linear Adaptor + Offline-loading | 1.3 | 16384 | 93.2 | 82.2 | 64.2 | 54.0 | 98.2 | 98.3 | 87.8 | 93.1 | 86.2 | 88.8 | 85.9 | 83.3 |
Table A4: Efficiency study of LLM2CLIP. We use two nodes, each equipped with 8 A100 40GB GPUs, and conduct experiments by maximizing GPU memory utilization to achieve the largest possible batch size. We report training hours, batch size, and retrieval accuracy for each experiment. During offloading, we precompute text captions offline into embeddings using the LLM. This approach is a one-time preprocessing step that effectively reduces the experimental overhead.
| 模型 | 策略 | 训练小时 | 批量 | Flickr | COCO | ShareGPT4V | Urban1k | DOCCI | 平均 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | i2t | t2i | ||||
| Llama-3.1-8B | LLM LoRA | 17 | 768 | 90.0 | 80.9 | 60.9 | 50.6 | 97.5 | 96.9 | 88.9 | 90.8 | 82.0 | 84.3 | 83.9 | 80.7 |
| LLM 冻结 + 线性适配器 + 离线加载 | 1.3 | 16384 | 85.2 | 74.6 | 52.3 | 40.8 | 87.7 | 89.6 | 57.5 | 59.2 | 49.9 | 48.3 | 66.5 | 62.5 | |
| Llama-3.1-8B-CC | LLM LoRA | 17 | 704 | 90.5 | 81.8 | 61.5 | 52.4 | 98.1 | 98.2 | 91.0 | 92.8 | 85.7 | 87.4 | 85.4 | 82.5 |
| LLM 冻结 + 线性适配器 | 5.5 | 4096 | 91.9 | 81.7 | 62.0 | 52.7 | 97.2 | 97.6 | 84.6 | 91.3 | 83.7 | 87.3 | 83.9 | 82.1 | |
| LLM 冻结 + 线性适配器 + 离线加载 | 1.3 | 16384 | 93.2 | 82.2 | 64.2 | 54.0 | 98.2 | 98.3 | 87.8 | 93.1 | 86.2 | 88.8 | 85.9 | 83.3 |
表 A4：LLM2CLIP 的效率研究。我们使用两个节点，每个节点配备 8 块 A100 40GB GPU，并在尽量占满显存的前提下做到最大批量。对每个实验报告训练小时数、批量和检索准确率。在卸载（offloading）时，我们用 LLM 把文本标题离线预计算为嵌入。这是一次性预处理，可以有效降低实验开销。
| Stage1 | Flickr | COCO | ShareGPT4V | Urban-1k | DOCCI | Avg | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | |
| Lora, AvgPool, Bidirectional, Supervise Simcse | 88.9 | 78.8 | 59.8 | 49.1 | 96.3 | 95.6 | 80.1 | 85.1 | 77.0 | 80.8 | 80.4 | 77.9 |
| Lora, AvgPool, Bidirectional, Un-supervise Simcse | 77.2 | 66.0 | 48.8 | 38.2 | 80.2 | 86.2 | 47.7 | 53.5 | 41.9 | 44.7 | 59.2 | 57.7 |
| Lora, AvgPool, Bidirectional, MNTP | 82.0 | 71.7 | 51.2 | 43.1 | 92.0 | 91.9 | 67.0 | 70.3 | 58.4 | 58.2 | 70.1 | 67.0 |
| Lora, AvgPool, Bidirectional, MNTP, Supervise Simcse | 88.2 | 78.3 | 59.6 | 48.9 | 95.5 | 95.1 | 79.7 | 86.1 | 75.3 | 77.7 | 79.7 | 77.2 |
| Lora, AvgPool, Casual, Supervise Simcse | 89.4 | 78.7 | 59.9 | 49.0 | 95.4 | 95.7 | 79.9 | 84.9 | 75.3 | 79.1 | 80.0 | 77.5 |
| Lora, EOS, Bidirectional, Supervise Simcse | 89.6 | 78.6 | 59.4 | 48.5 | 95.3 | 95.4 | 79.8 | 83.8 | 76.0 | 80.0 | 80.0 | 77.3 |
| Frozen, Linear Adaptor, AvgPool, Bidirectional, Supervise Simcse | 85.5 | 75.5 | 54.1 | 45.3 | 92.0 | 93.5 | 72.3 | 75.5 | 66.4 | 66.9 | 74.1 | 71.3 |
Table A5: Ablation study on the training methods of LLM caption contrastive finetuning in Stage 1. Rows with a light blue background represent our default setting, and red text indicates content that differs from the default setting.
| 第一阶段 | Flickr | COCO | ShareGPT4V | Urban-1k | DOCCI | 平均 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | |
| Lora, AvgPool, Bidirectional, Supervise Simcse | 88.9 | 78.8 | 59.8 | 49.1 | 96.3 | 95.6 | 80.1 | 85.1 | 77.0 | 80.8 | 80.4 | 77.9 |
| Lora, AvgPool, Bidirectional, Un-supervise Simcse | 77.2 | 66.0 | 48.8 | 38.2 | 80.2 | 86.2 | 47.7 | 53.5 | 41.9 | 44.7 | 59.2 | 57.7 |
| Lora, AvgPool, Bidirectional, MNTP | 82.0 | 71.7 | 51.2 | 43.1 | 92.0 | 91.9 | 67.0 | 70.3 | 58.4 | 58.2 | 70.1 | 67.0 |
| Lora, AvgPool, Bidirectional, MNTP, Supervise Simcse | 88.2 | 78.3 | 59.6 | 48.9 | 95.5 | 95.1 | 79.7 | 86.1 | 75.3 | 77.7 | 79.7 | 77.2 |
| Lora, AvgPool, Casual, Supervise Simcse | 89.4 | 78.7 | 59.9 | 49.0 | 95.4 | 95.7 | 79.9 | 84.9 | 75.3 | 79.1 | 80.0 | 77.5 |
| Lora, EOS, Bidirectional, Supervise Simcse | 89.6 | 78.6 | 59.4 | 48.5 | 95.3 | 95.4 | 79.8 | 83.8 | 76.0 | 80.0 | 80.0 | 77.3 |
| Frozen, Linear Adaptor, AvgPool, Bidirectional, Supervise Simcse | 85.5 | 75.5 | 54.1 | 45.3 | 92.0 | 93.5 | 72.3 | 75.5 | 66.4 | 66.9 | 74.1 | 71.3 |
表 A5：第一阶段 LLM 标题对比微调训练方法的消融。浅蓝色背景行为默认设置，红色文字标出与默认设置不同的内容。
| Method | Flickr | COCO | ShareGPT4V | Urban-1k | DOCCI | Avg | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | |
| CLIP | 89.6 | 77.8 | 59.4 | 48.6 | 88.1 | 87.7 | 68.0 | 74.8 | 67.0 | 71.3 | 74.4 | 72.0 |
| Directly Finetune (50%) | 89.3 | 77.8 | 59.3 | 48.5 | 88.3 | 88.2 | 68.5 | 76.0 | 67.2 | 71.2 | 74.5 | 72.3 |
| bge-en-icl | 89.1 | 78.5 | 58.8 | 49.7 | 95.0 | 95.7 | 77.9 | 87.7 | 73.5 | 79.4 | 78.9 | 78.2 |
| LLM2Vec-Llama-3-8B | 91.1 | 81.4 | 61.5 | 51.9 | 94.5 | 96.4 | 82.1 | 88.6 | 77.7 | 82.6 | 81.4 | 80.2 |
| NV-Embed-v2 | 90.4 | 80.0 | 60.5 | 51.6 | 94.5 | 95.7 | 83.3 | 90.0 | 78.3 | 82.1 | 81.4 | 79.9 |
| bge-m3-XLM-R | 80.7 | 70.3 | 51.4 | 42.0 | 84.5 | 86.5 | 56.8 | 63.4 | 51.7 | 55.9 | 65.0 | 63.6 |
| jina-v3-XLM-R | 84.4 | 73.7 | 56.3 | 45.6 | 90.0 | 90.9 | 70.8 | 74.1 | 66.7 | 70.6 | 73.6 | 71.0 |
| e5 (XLM-R) | 86.4 | 75.3 | 56.4 | 46.2 | 88.4 | 88.5 | 71.1 | 77.3 | 67.5 | 71.2 | 74.0 | 71.7 |
| VLM2VEC | 91.6 | 79.8 | 61.3 | 51.5 | 93.9 | 91.0 | 90.9 | 92.0 | 80.5 | 86.0 | 83.6 | 80.1 |
| VLM2VEC (finetune) | 90.2 | 79.3 | 60.0 | 50.1 | 89.8 | 91.4 | 76.9 | 85.8 | 74.1 | 78.7 | 78.2 | 77.1 |
| Qwen2.5-0.5B-CC | 86.2 | 74.9 | 56.0 | 45.1 | 92.6 | 93.2 | 73.7 | 79.0 | 69.5 | 73.0 | 75.6 | 73.0 |
| Llama-3.2-1B-CC | 88.9 | 78.8 | 59.8 | 49.1 | 96.3 | 95.6 | 80.1 | 85.1 | 77.0 | 80.8 | 80.4 | 77.9 |
| Llama-3-8B-CC | 90.4 | 80.7 | 62.7 | 51.9 | 96.5 | 96.2 | 84.2 | 89.5 | 83.3 | 86.4 | 83.4 | 80.9 |
| DeepSeek-R1-Distill-Llama-8B-CC | 91.7 | 80.9 | 62.2 | 51.9 | 96.7 | 96.3 | 84.3 | 88.3 | 82.5 | 85.2 | 83.5 | 80.5 |
| Llama-3.1-8B-CC | 92.2 | 81.5 | 63.5 | 52.3 | 97.1 | 96.2 | 86.5 | 89.3 | 84.7 | 85.9 | 84.8 | 81.0 |
| Llama3.1-8B | 85.2 | 74.6 | 52.3 | 40.8 | 87.7 | 89.6 | 57.5 | 59.2 | 49.9 | 48.3 | 66.5 | 62.5 |
Table A6: Ablation for using different text encoder. BGE-EN-ICL, LLM2Vec-Llama-3-8B, and NV-Embed-v2 are community-released text encoders derived from LLMs. The suffix ”-CC” indicates encoders that have undergone our caption contrastive fine-tuning method based on the respective original LLMs.
| 方法 | Flickr | COCO | ShareGPT4V | Urban-1k | DOCCI | 平均 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | |
| CLIP | 89.6 | 77.8 | 59.4 | 48.6 | 88.1 | 87.7 | 68.0 | 74.8 | 67.0 | 71.3 | 74.4 | 72.0 |
| 直接微调 (50%) | 89.3 | 77.8 | 59.3 | 48.5 | 88.3 | 88.2 | 68.5 | 76.0 | 67.2 | 71.2 | 74.5 | 72.3 |
| bge-en-icl | 89.1 | 78.5 | 58.8 | 49.7 | 95.0 | 95.7 | 77.9 | 87.7 | 73.5 | 79.4 | 78.9 | 78.2 |
| LLM2Vec-Llama-3-8B | 91.1 | 81.4 | 61.5 | 51.9 | 94.5 | 96.4 | 82.1 | 88.6 | 77.7 | 82.6 | 81.4 | 80.2 |
| NV-Embed-v2 | 90.4 | 80.0 | 60.5 | 51.6 | 94.5 | 95.7 | 83.3 | 90.0 | 78.3 | 82.1 | 81.4 | 79.9 |
| bge-m3-XLM-R | 80.7 | 70.3 | 51.4 | 42.0 | 84.5 | 86.5 | 56.8 | 63.4 | 51.7 | 55.9 | 65.0 | 63.6 |
| jina-v3-XLM-R | 84.4 | 73.7 | 56.3 | 45.6 | 90.0 | 90.9 | 70.8 | 74.1 | 66.7 | 70.6 | 73.6 | 71.0 |
| e5 (XLM-R) | 86.4 | 75.3 | 56.4 | 46.2 | 88.4 | 88.5 | 71.1 | 77.3 | 67.5 | 71.2 | 74.0 | 71.7 |
| VLM2VEC | 91.6 | 79.8 | 61.3 | 51.5 | 93.9 | 91.0 | 90.9 | 92.0 | 80.5 | 86.0 | 83.6 | 80.1 |
| VLM2VEC（微调） | 90.2 | 79.3 | 60.0 | 50.1 | 89.8 | 91.4 | 76.9 | 85.8 | 74.1 | 78.7 | 78.2 | 77.1 |
| Qwen2.5-0.5B-CC | 86.2 | 74.9 | 56.0 | 45.1 | 92.6 | 93.2 | 73.7 | 79.0 | 69.5 | 73.0 | 75.6 | 73.0 |
| Llama-3.2-1B-CC | 88.9 | 78.8 | 59.8 | 49.1 | 96.3 | 95.6 | 80.1 | 85.1 | 77.0 | 80.8 | 80.4 | 77.9 |
| Llama-3-8B-CC | 90.4 | 80.7 | 62.7 | 51.9 | 96.5 | 96.2 | 84.2 | 89.5 | 83.3 | 86.4 | 83.4 | 80.9 |
| DeepSeek-R1-Distill-Llama-8B-CC | 91.7 | 80.9 | 62.2 | 51.9 | 96.7 | 96.3 | 84.3 | 88.3 | 82.5 | 85.2 | 83.5 | 80.5 |
| Llama-3.1-8B-CC | 92.2 | 81.5 | 63.5 | 52.3 | 97.1 | 96.2 | 86.5 | 89.3 | 84.7 | 85.9 | 84.8 | 81.0 |
| Llama3.1-8B | 85.2 | 74.6 | 52.3 | 40.8 | 87.7 | 89.6 | 57.5 | 59.2 | 49.9 | 48.3 | 66.5 | 62.5 |
表 A6：使用不同文本编码器的消融。BGE-EN-ICL、LLM2Vec-Llama-3-8B 和 NV-Embed-v2 是社区发布的、由 LLM 得到的文本编码器。后缀 “-CC” 表示在相应原始 LLM 上经过我们的标题对比微调。
| Stage 1 | Stage 2 | Flickr | COCO | ShareGPT4V | Urban-1k | DOCCI | Average | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||
| – | – | 89.2 | 77.7 | 59.0 | 47.3 | 94.3 | 94.3 | 75.0 | 80.5 | 74.0 | 77.5 | 78.3 | 75.5 |
| – | Linear() | 88.5 | 77.5 | 59.8 | 48.4 | 95.1 | 94.5 | 77.2 | 84.4 | 75.3 | 78.7 | 79.2 | 76.7 |
| – | Linear() | 89.6 | 78.1 | 59.6 | 48.5 | 95.5 | 94.7 | 79.7 | 84.0 | 76.1 | 79.0 | 80.1 | 76.8 |
| – | Linear() | 88.9 | 78.8 | 59.8 | 49.1 | 96.3 | 95.6 | 80.1 | 85.1 | 77.0 | 80.8 | 80.4 | 77.9 |
| Linear() | Linear() | 89.9 | 78.8 | 59.7 | 48.9 | 95.1 | 95.6 | 79.9 | 84.9 | 78.0 | 80.4 | 80.5 | 77.7 |
| – | Transformer() | 88.6 | 78.7 | 59.9 | 48.6 | 95.8 | 95.1 | 79.5 | 84.5 | 77.0 | 79.8 | 80.2 | 77.3 |
| Transformer() | Transformer() | 89.8 | 78.6 | 58.1 | 49.0 | 96.1 | 95.4 | 81.9 | 85.6 | 76.5 | 78.1 | 80.5 | 77.3 |
Table A7: Ablation experiments for adaptor design on Stage 1 caption contrastive finetune and Stage 2 LLM2CLIP post training. In parentheses, () denotes an adaptor with layers. The 4-layer Linear Adaptor has 67.1M parameters, while the single layer Transformer Adaptor has 67.6M parameters. This experiment is conducted using llama 3.1 1B and OpenAI CLIP ViT-L 224, with CC 3M data utilized in both Stage 1 and Stage 2. The light blue row indicate our default setting.
| 第一阶段 | 第二阶段 | Flickr | COCO | ShareGPT4V | Urban-1k | DOCCI | 平均 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||
| – | – | 89.2 | 77.7 | 59.0 | 47.3 | 94.3 | 94.3 | 75.0 | 80.5 | 74.0 | 77.5 | 78.3 | 75.5 |
| – | Linear() | 88.5 | 77.5 | 59.8 | 48.4 | 95.1 | 94.5 | 77.2 | 84.4 | 75.3 | 78.7 | 79.2 | 76.7 |
| – | Linear() | 89.6 | 78.1 | 59.6 | 48.5 | 95.5 | 94.7 | 79.7 | 84.0 | 76.1 | 79.0 | 80.1 | 76.8 |
| – | Linear() | 88.9 | 78.8 | 59.8 | 49.1 | 96.3 | 95.6 | 80.1 | 85.1 | 77.0 | 80.8 | 80.4 | 77.9 |
| Linear() | Linear() | 89.9 | 78.8 | 59.7 | 48.9 | 95.1 | 95.6 | 79.9 | 84.9 | 78.0 | 80.4 | 80.5 | 77.7 |
| – | Transformer() | 88.6 | 78.7 | 59.9 | 48.6 | 95.8 | 95.1 | 79.5 | 84.5 | 77.0 | 79.8 | 80.2 | 77.3 |
| Transformer() | Transformer() | 89.8 | 78.6 | 58.1 | 49.0 | 96.1 | 95.4 | 81.9 | 85.6 | 76.5 | 78.1 | 80.5 | 77.3 |
表 A7：第一阶段标题对比微调与第二阶段 LLM2CLIP 后期训练中适配器设计的消融。括号中（）表示具有 层的适配器。四层线性适配器有 6710 万参数，单层 Transformer 适配器有 6760 万参数。本实验使用 Llama 3.1 1B 与 OpenAI CLIP ViT-L 224，第一、二阶段都使用 CC 3M 数据。浅蓝色行表示我们的默认设置。
| Methods | Training Loss | Testing Text Encoder | Flickr | COCO | ShareGPT4V | Urban1k | DOCCI | Average | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||||
| CLIP | CL(CLIP-T, CLIP-V) | CLIP-T | 89.6 | 77.8 | 59.4 | 48.6 | 88.1 | 87.7 | 68.0 | 74.8 | 67.0 | 71.3 | 74.4 | 72.0 | |
| Directly Finetune | CL(CLIP-T, CLIP-V) | CLIP-T | 89.3 | 77.8 | 59.3 | 48.5 | 88.3 | 88.2 | 68.5 | 76.0 | 67.2 | 71.2 | 74.5 | 72.3 | |
| +LLM2CLIP | a) | CL(LLM, CLIP) | LLM | 91.9 | 81.7 | 62.0 | 52.7 | 97.2 | 97.6 | 84.6 | 91.3 | 83.7 | 87.3 | 83.9 | 82.1 |
| b) | CL(LLM, CLIP-V)+ CL(CLIP-T, CLIP-V) | CLIP-T | 89.6 | 77.8 | 59.4 | 48.6 | 88.1 | 87.7 | 68.0 | 74.8 | 67.0 | 71.3 | 74.4 | 72.0 | |
| LLM | 90.1 | 81.3 | 61.3 | 52.4 | 97.1 | 97.4 | 85.6 | 91.1 | 83.8 | 86.9 | 83.6 | 81.8 | |||
| c) | CL(LLM, CLIP-V)+CL(CLIP-T, CLIP-V)+ CL(CLIP-T, LLM) | CLIP-T | 89.0 | 77.9 | 58.5 | 47.7 | 89.1 | 88.8 | 67.0 | 75.5 | 66.3 | 70.7 | 74.0 | 72.1 | |
| LLM | 90.8 | 80.5 | 61.5 | 51.8 | 97.1 | 97.7 | 85.9 | 90.3 | 83.4 | 86.7 | 83.7 | 81.4 | |||
| d) | CL(LLM, CLIP-V)+CL(CLIP-T, CLIP-V)+ CL(Cat(CLIP-T, LLM), CLIP-V) | CLIP-T | 89.6 | 77.9 | 59.1 | 48.1 | 89.4 | 87.5 | 69.1 | 72.5 | 67.0 | 70.6 | 74.8 | 71.3 | |
| LLM | 90.7 | 81.4 | 61.3 | 52.3 | 97.5 | 97.9 | 85.5 | 92.5 | 83.9 | 87.5 | 83.8 | 82.3 | |||
| Cat(CLIP-T,LLM) | 91.3 | 82.4 | 62.9 | 53.3 | 97.8 | 97.4 | 86.6 | 92.9 | 84.9 | 88.2 | 84.7 | 82.8 |
Table A8: LLM2CLIP Training Method Analysis. We experimented with various possibilities for fine-tuning between the LLM and the pretrained CLIP model’s Vision Encoder (CLIP-V) and Text Encoder (CLIP-T). In the figure, ”CL” denotes the contrastive learning loss, and ”Cat” stands for concatenation. Items a–d correspond to the training methods section in Section 3.2. In this experiment, we performed zero-shot image-text retrieval tests on each text encoder, including the new feature projection space formed after concatenation.
| 方法 | 训练损失 | 测试用文本编码器 | Flickr | COCO | ShareGPT4V | Urban1k | DOCCI | 平均 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | |||
| CLIP | CL(CLIP-T, CLIP-V) | CLIP-T | 89.6 | 77.8 | 59.4 | 48.6 | 88.1 | 87.7 | 68.0 | 74.8 | 67.0 | 71.3 | 74.4 | 72.0 |
| 直接微调 | CL(CLIP-T, CLIP-V) | CLIP-T | 89.3 | 77.8 | 59.3 | 48.5 | 88.3 | 88.2 | 68.5 | 76.0 | 67.2 | 71.2 | 74.5 | 72.3 |
| +LLM2CLIP a) | CL(LLM, CLIP) | LLM | 91.9 | 81.7 | 62.0 | 52.7 | 97.2 | 97.6 | 84.6 | 91.3 | 83.7 | 87.3 | 83.9 | 82.1 |
| b) | CL(LLM, CLIP-V)+ CL(CLIP-T, CLIP-V) | CLIP-T | 89.6 | 77.8 | 59.4 | 48.6 | 88.1 | 87.7 | 68.0 | 74.8 | 67.0 | 71.3 | 74.4 | 72.0 |
| LLM | 90.1 | 81.3 | 61.3 | 52.4 | 97.1 | 97.4 | 85.6 | 91.1 | 83.8 | 86.9 | 83.6 | 81.8 | ||
| c) | CL(LLM, CLIP-V)+CL(CLIP-T, CLIP-V)+ CL(CLIP-T, LLM) | CLIP-T | 89.0 | 77.9 | 58.5 | 47.7 | 89.1 | 88.8 | 67.0 | 75.5 | 66.3 | 70.7 | 74.0 | 72.1 |
| LLM | 90.8 | 80.5 | 61.5 | 51.8 | 97.1 | 97.7 | 85.9 | 90.3 | 83.4 | 86.7 | 83.7 | 81.4 | ||
| d) | CL(LLM, CLIP-V)+CL(CLIP-T, CLIP-V)+ CL(Cat(CLIP-T, LLM), CLIP-V) | CLIP-T | 89.6 | 77.9 | 59.1 | 48.1 | 89.4 | 87.5 | 69.1 | 72.5 | 67.0 | 70.6 | 74.8 | 71.3 |
| LLM | 90.7 | 81.4 | 61.3 | 52.3 | 97.5 | 97.9 | 85.5 | 92.5 | 83.9 | 87.5 | 83.8 | 82.3 | ||
| Cat(CLIP-T,LLM) | 91.3 | 82.4 | 62.9 | 53.3 | 97.8 | 97.4 | 86.6 | 92.9 | 84.9 | 88.2 | 84.7 | 82.8 |
表 A8：LLM2CLIP 训练方法分析。我们试验了在 LLM 与已预训练 CLIP 的视觉编码器（CLIP-V）、文本编码器（CLIP-T）之间微调的多种可能。表中 “CL” 表示对比学习损失，“Cat” 表示拼接。条目 a–d 对应第 3.2 节训练方法部分。本实验对每个文本编码器做了零样本图文检索测试，也包括拼接后形成的新特征投影空间。
Detailed Results for X3600 Multilingual Task.
In Figure 2(a) and Figure 2(b), we visualize the I2T and T2I results for 36 different languages. It can be observed that our method outperforms the original SigLip2 text encoder, despite the latter being trained with 10% of its training samples from 12 billion alt-texts covering 109 languages. In contrast, our method LLM2CLIP was trained exclusively on 15 million English-only samples. This clearly demonstrates the inherent multilingual advantage gained by incorporating an LLM.
X3600 多语任务的详细结果。
在图 2（a）和 图 2（b）中，我们可视化了 36 种语言上的 I2T 与 T2I 结果。可以看到，我们的方法超过了原来的 SigLip2 文本编码器，尽管后者有 10% 的训练样本来自覆盖 109 种语言的 120 亿条替代文本（alt-text）。相比之下，LLM2CLIP 只在 1500 万条纯英文样本上训练。这清楚说明，引入 LLM 本身就能带来多语优势。
Data Ratio Ablation.
In Table A3, we experiment with various dataset compositions. The results show a clear trend: higher proportions of dense captions enhance performance in long-text image retrieval tasks, whereas lower dense caption ratios improve short-text retrieval performance. We select a 1:1 ratio by default, as it provides the most balanced improvement for both short and long texts.
数据比例消融。
在表 A3 中，我们试验了多种数据构成。结果有清楚的趋势：稠密标题比例更高，会长文本图像检索更好；稠密标题比例更低，则短文本检索更好。我们默认选择 1:1 的比例，因为它对短文本和长文本都能给出最均衡的提升。
Interestingly, even training exclusively on real short captions (100%), our model still significantly boosts performance for both short and long-text retrieval tasks. This improvement can be attributed to the incorporation of LLM, given that real captions share the same distribution as the CLIP pre-training data, yet LLM integration still yields substantial gains.
*有意思的是，即便只在真实短标题上训练（100%），我们的模型仍然显著提升了短文本和长文本检索。*这一改进可以归因于引入了 LLM：真实标题与 CLIP 预训练数据同分布，但接入 LLM 仍然带来可观增益。
However, when training exclusively on synthesized long texts, although retrieval performance on long-text tasks can be significantly improved, performance on short-text retrieval tasks tends to decline. We hypothesize two possible reasons for this phenomenon: First, relying entirely on synthesized data might introduce significant biases in the training samples. Second, when the learned feature space is constructed exclusively from long texts, the model may prioritize global semantic matching and overlook fine-grained information such as specific words, potentially negatively affecting short-text retrieval. Therefore, we believe that for contrastive learning tasks focusing solely on long texts, it is important to consider including both global and local levels of text granularity during training like FLIP (Yao et al. 2021), or introducing sufficient hard examples to encourage the model to attend to low-level semantic details.
不过，若只在合成长文本上训练，长文本检索可以明显提高，短文本检索却往往下降。我们对这一现象提出两种可能原因：第一，完全依赖合成数据可能给训练样本带来明显偏差；第二，当所学特征空间完全由长文本构成时，模型可能更侧重全局语义匹配，而忽略特定词等细粒度信息，从而对短文本检索产生不利影响。因此我们认为，若对比学习任务只关注长文本，训练时应当同时考虑全局和局部两个层次的文本粒度，例如 FLIP（Yao et al. 2021）的做法，或引入足够的困难样本，促使模型关注底层语义细节。
Optimization Strategy Ablation and Efficiency Analysis of LLM2CLIP.
As shown in Table A4, we conducted experiments comparing methods such as LoRA training, frozen LLM online training, and offloading on fixed hardware. Experimental results demonstrate that, with identical computational resources, our proposed offloading method can increase the batch size by tenfold and extend training time by fourfold, ultimately achieving higher performance.
LLM2CLIP 的优化策略消融与效率分析。
如表 A4 所示，我们在固定硬件上比较了 LoRA 训练、冻结 LLM 的在线训练，以及卸载（offloading）等方法。实验结果表明，在相同计算资源下，我们提出的卸载方法可以把批量扩大十倍，并将训练时间延长四倍，最终取得更高的性能。
Complete Tables and Additional Visualizations.
In Tables Table A5, Table A6, Table A7, and Table A8, we provide additional detailed results corresponding to Table 6, 7, 8, and 9 in the main text, facilitating a clear analysis of the differences in retrieval performance between long and short texts. The conclusions remain consistent with those presented in the main text. Additionally, Figure A1 illustrates radar plots highlighting the performance improvements achieved by LLM2CLIP on SigLip2 and EVA02.
完整表格与补充可视化。
在表 A5、表 A6、表 A7 和表 A8 中，我们提供了与正文表 6、7、8、9 相对应的更详细结果，便于清楚分析长、短文本检索表现的差异。结论与正文一致。此外，图 A1 用雷达图展示了 LLM2CLIP 在 SigLip2 和 EVA02 上带来的性能提升。