Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
Molmo 与 PixMo：面向顶尖视觉–语言模型的开放权重与开放数据
Matt DeitkeChristopher Clark *† Sangho Lee † Rohun Tripathi † Yue Yang † Jae Sung Park Affiliation: University of Washington Mohammadreza SalehiNiklas Muennighoff † Kyle Lo † Luca Soldaini † Jiasen Lu † Taira Anderson † Erin Bransom † Kiana Ehsani † Huong Ngo † YenSung Chen † Ajay Patel † Mark Yatskar † Chris Callison-Burch † Andrew Head † Rose Hendrix † Favyen Bastani † Eli VanderBilt † Nathan Lambert † Yvonne Chou † Arnavi Chheda † Jenna Sparks † Sam Skjonsberg † Michael Schmitz † Aaron Sarnat † Byron Bischoff † Pete Walsh † Chris Newell † Piper Wolters † Tanmay Gupta † Kuo-Hao Zeng † Jon Borchardt † Dirk Groeneveld † Crystal Nam † Sophie Lebrecht † Caitlin Wittlif † Carissa Schoenick † Oscar Michel † Affiliation: University of Washington Ranjay KrishnaLuca Weihs † Noah A. Smith Hannaneh Hajishirzi Ross Girshick Ali Farhadi Aniruddha Kembhavi † Allen Institute for AI
Abstract
摘要
Today’s most advanced vision-language models (VLMs) remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed VLMs into open ones. As a result, the community has been missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key contribution is a collection of new datasets called PixMo, including a dataset of highly detailed image captions for pre-training, a free-form image Q&A dataset for fine-tuning, and an innovative 2D pointing dataset, all collected without the use of external VLMs. The success of our approach relies on careful modeling choices, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets. Our best-in-class 72B model not only outperforms others in the class of open weight and data models, but also outperforms larger proprietary models including Claude 3.5 Sonnet, and Gemini 1.5 Pro and Flash, second only to GPT-4o based on both academic benchmarks and on a large human evaluation. Our model weights, new datasets, and source code are available at https://molmo.allenai.org/blog.
当今最强的视觉–语言模型（VLM）仍大多闭源。表现最好的开源权重模型，往往高度依赖来自闭源 VLM 的合成数据才能刷出分数，实质是在把闭源模型蒸馏成开源版本。于是，学界一直缺少「如何从零做出强 VLM」的基础经验。本文提出 Molmo：在同等开放度档位里达到顶尖水平的一族 VLM。核心贡献是一套新数据 PixMo，包括用于预训练的高细节图像描述、用于微调的自由形式图像问答，以及创新的二维指点数据——全部不借助外部 VLM 采集。方法上的成功来自审慎的建模选择、调通的训练流程，以及更关键的新数据质量。我们最优的 72B 模型不仅压过同档「权重+数据开放」模型，也在学术基准与大规模人工评测上超过 Claude 3.5 Sonnet、Gemini 1.5 Pro/Flash 等更大闭源系统，仅次于 GPT-4o。模型权重、新数据与代码见 https://molmo.allenai.org/blog。
1 Introduction
1 引言
Figure 1: Datasets in PixMo (left) and the capabilities they enable in Molmo (right). PixMo consists of three annotated datasets and four synthetic datasets, all constructed without the use of VLMs. The annotated datasets include: dense captions (for pre-training), instruction following (for fine-tuning), and pointing (for fine-tuning, to support grounding and counting). The four synthetic datasets augment these datasets by targeting additional skills (e.g., clock reading, document understanding).
图 1：PixMo 数据集（左）及其为 Molmo 带来的能力（右）。PixMo 含三个人工标注集与四个合成集，均不依赖 VLM 构建。标注集包括：稠密描述（预训练）、指令跟随（微调）、指点（微调，支撑 grounding 与计数）。四个合成集针对读钟、文档理解等额外技能补强。
Large multimodal models are used ubiquitously today. Proprietary models—GPT-4o, Gemini-1.5 Pro, Claude 3.5 Sonnet—produce comprehensive image descriptions and accurately answer complex visual questions. Unfortunately, the most performant of these vision-language models (VLMs) remain proprietary with neither model weights, data, nor code being publicly released.
大规模多模态模型已无处不在。GPT-4o、Gemini-1.5 Pro、Claude 3.5 Sonnet 等闭源系统能给出完整图像描述，也能答对复杂视觉问题。遗憾的是，表现最好的视觉–语言模型（VLM）仍不公开权重、数据与代码。
To foster scientific exploration, numerous research efforts have attempted to reproduce similar capabilities in open models. Early works, exemplified by LLaVA [69], produced fully open weights and training data but now lag significantly behind the state-of-the-art. More recent, stronger open-weight models have trended towards less open data: the training data may either be proprietary (e.g., [10]) or, in cases where it is released, there is a heavy reliance on synthetic data generated by proprietary systems, e.g., models are trained on datasets like ShareGPT4V [15] which uses GPT-4V [88] to generate a large set of detailed image captions. The resulting VLMs, therefore, are effectively distillations of proprietary VLMs. As it stands, the scientific community is missing foundational knowledge about how to build performant VLMs from scratch (more discussion about this and related work are in the Appendix).
为推动科学探索，大量工作试图在开放模型上复现相近能力。以 LLaVA [69] 为代表的早期工作同时开放权重与训练数据，但已明显落后于前沿。更近、更强的开源权重模型则越来越不开放数据：训练数据要么本身闭源（如 [10]），要么即便公开，也严重依赖闭源系统生成的合成数据——例如 ShareGPT4V [15] 用 GPT-4V [88] 大批量生成细粒度描述。由此得到的 VLM，实质是闭源 VLM 的蒸馏版。目前学界仍缺少「如何从零构建强 VLM」的基础认知（附录有更多讨论与相关工作）。
In this work, we present the Molmo (Multimodal Open Language Model) family of state-of-the-art open VLMs with released model weights and released vision-language training data without any reliance on synthetic data from other VLMs, including proprietary ones. The success of our approach relies on careful model design choices, a well-tuned training pipeline, and most critically, the quality of our new datasets, collectively named PixMo (Pixels for Molmo), which are fully open.
本文提出 Molmo（Multimodal Open Language Model）一族顶尖开放 VLM：既公开模型权重，也公开视觉–语言训练数据，且不依赖任何其他 VLM（含闭源）的合成数据。成功取决于审慎的模型设计、调通的训练流水线，以及最关键的新数据质量——我们统称这批数据为 PixMo（Pixels for Molmo），全部开放。
High-quality multimodal data, both for pre-training and fine-tuning, is a key missing piece for training open VLMs that are competitive with closed ones. The academic community has struggled to collect such datasets due to high costs and the difficulty of obtaining high-quality annotations from crowd sourcing platforms. To build PixMo, we introduce several key data collection innovations that allow us to quickly collect high-quality data from untrained annotators, see Figure 1.
高质量多模态数据——无论预训练还是微调——正是开放 VLM 追赶闭源系统的关键缺口。学界难做这类数据，一是贵，二是众包平台很难拿到高质量标注。为构建 PixMo，我们引入若干采集上的关键创新，使未受专门训练的标注者也能较快产出高质量数据，见图 1。
PixMo includes a dataset of 712k images with very long (200+ word) detailed captions. Collecting this data was difficult because directly asking annotators to write such captions produces poor results: they tend to focus on a few salient visual elements [17], typing long paragraphs is time-consuming, and annotators can potentially copy-and-paste responses from proprietary VLMs, circumventing our goal of avoiding distillation. Instead, we ask annotators to describe images in speech for 60 to 90 seconds. Empirically, we found that with this modality switching “trick” annotators provide far more detailed descriptions in less time, and for each description we collect an audio receipt (i.e., the annotator’s recording) proving that a VLM was not used.
PixMo 含 71.2 万张图，配以很长（200+ 词）的细粒度描述。直接让人写这种描述效果差：人往往只盯住少数显著区域 [17]，长文打字费时，还可能从闭源 VLM 复制粘贴，违背「避免蒸馏」的目标。我们改让标注者用语音描述图像 60–90 秒。经验上，这一「换模态」技巧带来更细、更快的描述，且每条都有音频回执（标注者录音），可证明未使用 VLM。
PixMo also includes an array of fine-tuning datasets. To collect instruction-following data, we have users interactively edit responses with a language-only LLM to obtain high-quality and accurate free-form responses. We gather 162k annotations on 73k images in this way. We also collect a unique new data source that grounds language in images with 2D points. Using points enables us to collect grounding data much faster than would be possible using bounding boxes or segmentation masks since it is much easier to annotate, and we take advantage of this by collecting over 2.3 million grounding annotations for a diverse range of objects, expressions, and scenes. This novel pointing data enables our models to answer some questions more naturally by pointing to the pixels that support the answer, improves counting accuracy (the model counts by pointing), and we believe it will open up an important future direction in which VLMs enable agents (e.g., robots, web agents) to act by pointing in their environments, e.g., to a navigation waypoint, to an object to pick up, or to a user interface button to press. Finally, we introduce several novel synthetic datasets (meaning with no or minimal human annotations, but still not using a VLM) with data targeting particular skills (clock reading, chart understanding, table understanding, etc.) that complements existing open-source datasets.
PixMo 还有一系列微调数据。采集指令跟随时，让用户与纯语言模型交互修订回答，得到高质量、准确的自由形式回复；共在 7.3 万张图上收集 16.2 万条标注。我们还新增一种用二维点把语言锚定到图像的数据源。点比框或分割掩码好标得多，因而 grounding 数据可大幅加速；我们借此收集逾 230 万条 grounding 标注，覆盖多样物体、表述与场景。这类指点数据让模型能更自然地用支撑答案的像素作答，也能提升计数（模型靠指点来数），并有望打开重要方向：让 VLM 驱动的智能体（如机器人、网页智能体）在环境中用指点行动——指向导航路点、待抓取物体或待点击的界面按钮。最后，我们还引入若干新合成数据集（无人或极少人工标注，但仍不用 VLM），针对读钟、图表理解、表格理解等技能，补强已有开源数据。
We train models on these datasets following a mostly standard design using a pre-trained LLM and vision encoder, but with some new improvements including a simplified two-stage training pipeline, a novel overlapping multi-crop strategy, an efficient method of training on images with multiple annotations, and some new insights in how to set up the optimizer and vision/language connector. We evaluate the Molmo family of models on 11 academic benchmarks and with a human evaluation that allows us to rank models by user preference. Our most efficient model, MolmoE-1B, based on the OLMoE-1B-7B [87] mixture-of-experts LLM, nearly matches the performance of GPT-4V on both our academic benchmarks and user preference. Molmo-7B-O and Molmo-7B-D, based on OLMo-7B [37] and Qwen2 7B [120], respectively, perform comfortably between GPT-4V and GPT-4o [90] on both academic benchmarks and user preference. Our best-in-class Molmo-72B model, based on Qwen2 72B, achieves the highest academic benchmark score and ranks second by human preference, just behind GPT-4o. Our best model outperforms many state-of-the-art proprietary systems, including Gemini 1.5 Pro and Flash [103], and Claude 3.5 Sonnet [7]. We will also release a 100% fully open Molmo model, based on a MetaCLIP [118] vision encoder and OLMo LLM, for which every bit of training data is publicly available. In addition, we perform an expansive set of ablations to better inform the scientific community of how various model and data design choices affect VLMs.
训练上我们大体沿用「预训练 LLM + 视觉编码器」的标准设计，但加入简化两阶段流程、新型重叠多裁剪、多标注图像的高效训练，以及对优化器与视觉/语言连接器的新设置。我们在 11 项学术基准与按用户偏好排序的人工评测上评估 Molmo 全家桶。最省算力的 MolmoE-1B 基于 OLMoE-1B-7B [87] 混合专家 LLM，学术基准与用户偏好均接近 GPT-4V。基于 OLMo-7B [37] 与 Qwen2 7B [120] 的 Molmo-7B-O、Molmo-7B-D，在两类评测上都稳居 GPT-4V 与 GPT-4o [90] 之间。最优的 Molmo-72B（Qwen2 72B）学术均分最高，人工偏好仅次于 GPT-4o。它还超过 Gemini 1.5 Pro/Flash [103]、Claude 3.5 Sonnet [7] 等多家顶尖闭源系统。我们还将发布 100% 全开放的 Molmo：视觉编码器为 MetaCLIP [118]，语言模型为 OLMo，训练数据每一比特均可公开获取。此外，我们做了大量消融，帮助学界理清模型与数据设计如何影响 VLM。
Figure 2: Molmo follows the simple and standard design of connecting a vision encoder and a language model.
图 2：Molmo 沿用「视觉编码器 + 语言模型」这一简洁标准结构。
2 Architecture
2 架构
Our model architecture (Figure 2) follows a standard design, combining pre-trained language and vision models (e.g., [69]). It has four components: (1) a pre-processor that converts the input image into multiscale, multi-crop images, (2) a ViT image encoder [31] that computes per-patch features for each image independently, (3) a connector that pools and projects patch features into the LLM’s embedding space, and (4) a decoder-only LLM [109, 95].
模型结构（图 2）沿用标准设计，组合预训练语言与视觉模型（如 [69]）。共四部分：(1) 把输入图像变成多尺度、多裁剪图像的预处理器；(2) 对每张图独立计算 patch 特征的 ViT 图像编码器 [31]；(3) 将 patch 特征池化并投影到 LLM 嵌入空间的连接器；(4) 仅解码器 LLM [109, 95]。
From this template, we build a family of models by selecting a vision encoder and LLM, keeping the training data and recipe consistent across choices (except for learning rates). We primarily use OpenAI’s ViT-L/14 336px CLIP model [96] due to strong performance in initial experiments, but similar results are achievable with SigLIP [130] and the fully open MetaCLIP [118] (see Section 6). For the LLM, we experiment across scales and openness levels: fully open OLMo-7B-1024-preview, fully open OLMoE-1B-7B (our most efficient model), open-weight Qwen2 7B [120], and open-weight Qwen2 72B (our best-performing model).
在此模板上，我们通过更换视觉编码器与 LLM 组成一族模型，训练数据与配方保持一致（学习率除外）。主用 OpenAI 的 ViT-L/14 336px CLIP [96]，因前期实验表现强；SigLIP [130] 与完全开放的 MetaCLIP [118] 也能得到相近结果（见第 6 节）。LLM 则横跨规模与开放度：全开放 OLMo-7B-1024-preview、全开放 OLMoE-1B-7B（最省算力）、开源权重 Qwen2 7B [120]，以及开源权重 Qwen2 72B（表现最好）。
Figure 3: An image cropped without (left) and with (right) overlap. Highlighted regions show areas used by the LLM. Overlapping crops ensure that central patches are encoded with neighboring context; for example, the patches containing the bike’s brand name are always part of a crop where the entire name is visible.
图 3：无重叠（左）与有重叠（右）的图像裁剪。高亮区域为送入 LLM 的部分。重叠裁剪保证中心 patch 编码时带有邻域上下文；例如含自行车品牌名的 patch，总会落在能看见完整名字的裁剪里。
Cropping.
Most ViTs only support square images at a fixed resolution that is generally too low for fine-grained tasks such as OCR or detailed captioning. To address this issue, we follow recent works [124, 30, 19, 70, 85] by dividing the image into multiple square crops that tile the image. Additionally, the full image, resized to the ViT’s resolution, provides a low-resolution overview. Each crop is processed independently by the ViT. See the Appendix for details.
裁剪。
多数 ViT 只接受固定分辨率的方形图，对 OCR 或细粒度描述往往偏低。我们沿用近期做法 [124, 30, 19, 70, 85]，把图像切成铺满画面的多个方形裁剪；同时把整图缩到 ViT 分辨率，提供低分辨率总览。每个裁剪由 ViT 独立处理。细节见附录。
One limitation of cropping is that border patches lack context from adjacent patches (see Figure 3). To mitigate this, we allow crops to overlap so each patch has context from at least some neighboring patches. Patch features from the overlap are not passed to the connector or LLM so that the passed patch features exactly tile the high-resolution image. Overlapping slightly reduces the tiled image resolution, but this can be offset by using more crops. Overlapping significantly improves results, as shown in Section 6.
裁剪的局限是边界 patch 缺少相邻 patch 的上下文（见图 3）。为此我们允许裁剪重叠，使每个 patch 至少看到部分邻居。重叠区的 patch 特征不送入连接器或 LLM，从而交给下游的特征仍精确铺满高分辨率图。重叠会略降铺贴分辨率，可用更多裁剪补偿。第 6 节显示重叠显著提升结果。
Vision-language connector.
Once crops are encoded by the vision encoder, we build patch features by concatenating features from the third-to-last and tenth-from-last ViT layers, which improves performance slightly over using a single layer. Each 22 patch window is then pooled into a single vector using a multi-headed attention layer, where the mean of the patches serves as the query. This attention pooling outperforms simple feature concatenation (see Section 6). Finally, pooled features are mapped to the LLM’s embedding space via an MLP.
视觉–语言连接器。
裁剪经视觉编码器后，我们把倒数第 3 层与倒数第 10 层的特征拼接成 patch 特征，略优于单层。再用多头注意力把每个 patch 窗池成一个向量，查询为这些 patch 的均值。注意力池化优于简单特征拼接（见第 6 节）。最后经 MLP 映到 LLM 嵌入空间。
Arranging vision tokens.
Pooled patch features (vision tokens) are sequenced left-to-right, top-to-bottom, starting with patches from the low-resolution full image, followed by high-resolution crop patches arranged in row-major order. Special tokens are inserted to mark the start and end of both low- and high-resolution patch sequences, with row-end tokens added between rows to indicate row transitions.
视觉 token 排布。
池化后的 patch 特征（视觉 token）按从左到右、从上到下排列：先是低分辨率整图的 patch，再是按行优先排布的高分辨率裁剪 patch。用特殊 token 标出低/高分辨率序列的起止，并在行末插入 token 标记换行。
Dropout.
Residual dropout is applied to the LLM, but not the image encoder and vision-language connector. During pre-training on dense captions, dropout is applied only to text tokens to encourage reliance on the encoded image rather than language priors. This is not used in fine-tuning, as shorter target responses result in too little dropout. Text-only dropout during pre-training enhances captioning and downstream performance, as shown in Section 6.
Dropout。
残差 dropout 只加在 LLM 上，不加在图像编码器与视觉–语言连接器。稠密描述预训练时，dropout 仅作用于文本 token，迫使模型更依赖编码后的图像而非语言先验。微调目标较短，dropout 过弱，故不再使用。第 6 节显示预训练阶段仅文本 dropout 能提升描述与下游表现。
Multi-annotated images.
Our multimodal data often includes multiple annotations per image (e.g., VQA v2.0 has multiple question-answer pairs). To train efficiently, we arrange all of text annotation tokens for an image in one long sequence, masking attention so tokens for each annotation attend to the image tokens, each other, but not to tokens from different annotations. This setup is equivalent to training on individual image-text pairs but avoids redundant image encoding, reducing the number of processed images by two-thirds and shortening training time by over half, with only a 25% increase in sequence length for our data mix.
多标注图像。
多模态数据常对同一图像有多条标注（如 VQA v2.0 多组问答）。为提效，我们把同一图像的全部文本标注 token 排进一条长序列，并用注意力掩码：各标注内部与图像 token 互看，但不看其他标注。这与逐条图文对训练等价，却避免重复编码图像，处理图像数约减三分之二，训练时间缩短一半以上，而本数据混合下序列长度仅增约 25%。
3 Data
3 数据
PixMo contains seven datasets, three with human annotations and four created with synthetic data generation pipelines (see Figure 1). Below, we describe these datasets and data collection methods; additional details and examples are in the Appendix.
PixMo 含七个数据集：三个人工标注、四个合成流水线生成（见图 1）。下文介绍各集与采集方法；更多细节与样例见附录。
PixMo-Cap.
We collected PixMo-Cap as a source of high-quality pre-training data, featuring a diverse set of images paired with highly detailed dense captions. We began by sourcing web images across 70 diverse topics (e.g., street signs, memes, food, drawings, websites, blurry photos, etc.). For each image, three annotators initially provided detailed descriptions by speaking for at least 60 seconds. In later stages, we used one annotator per image with a 90-second minimum, which improved efficiency without sacrificing quality. We prompted the annotators with seven questions to answer, detailed in the Appendix.
PixMo-Cap。
PixMo-Cap 用作高质量预训练源：多样图像配以极细的稠密描述。我们从约 70 个主题（路牌、梗图、食物、绘画、网页、模糊照片等）抓取网络图。早期每张图由三名标注者各口述至少 60 秒；后期改为每图一人、至少 90 秒，效率升而质量不降。提示中含七个需回答的问题，详见附录。
The annotators’ audio was transcribed using a standard speech-to-text system, yielding raw transcripts. A final high-quality image caption was then created by prompting a language-only LLM to summarize multiple raw transcripts per image or, for single transcripts, to enhance its quality (e.g., removing spoken artifacts, normalizing style). In total, we collected 712k distinct images with 1.3M transcripts and captions. Our captions average 196 words, compared to 11 words in COCO captions [17] and 37 words in localized narratives [93], highlighting their greater detail.
音频经标准语音识别转成原始转写。再让纯语言模型对多条转写做摘要，或对单条转写润色（去口语痕迹、统一风格），得到最终高质量描述。共 71.2 万张不重复图、130 万条转写与描述。平均 196 词，远高于 COCO 描述的 11 词 [17] 与 Localized Narratives 的 37 词 [93]。
PixMo-AskModelAnything.
We collected this data to enable the model to answer diverse questions it might encounter in real-world use. To create image-question-answer triplets, annotators worked with a language-only LLM. An annotator selected an image from a large pool and wrote a question about it. Then, we ran a standard non-VLM OCR model and a PixMo-Cap-trained model on the image. The language-only LLM answered the question from the OCR data and dense caption. The annotator could accept or reject the answer and if rejected, they specified the issue and requested a revision until the answer was satisfactory. We collected 162k question-answer pairs in 73k images.
PixMo-AskModelAnything。
为让模型能答真实使用中的多样问题而采集。构造图–问–答三元组时，标注者与纯语言模型协作：从大图池选题并写问题；再对图像跑非 VLM 的 OCR 与 PixMo-Cap 训练模型；纯语言模型据 OCR 与稠密描述作答。标注者可接受或拒绝；拒绝则说明问题并要求改写，直至满意。共在 7.3 万张图上得到 16.2 万问答对。
PixMo-Points.
We collected pointing data to achieve three goals: (1) enable the model to point to items described by text, (2) enable the model to count by pointing, and (3) use pointing as a form of visual explanation when answering questions. For the first two goals, annotators were asked to point at something in an image, describe it, and then point to each instance of it in the image, ensuring exhaustive coverage. We also collected “not present” data so models can learn to handle cases where an item is not in the image. Pointing data also naturally supports answering counting questions with a chain-of-thought formed by the sequence of points. This resulted in 2.3M question-points pairs from 223k images. To enable points as explanations, we adapted the PixMo-AskModelAnything pipeline to let annotators pass the LLM a list of text-annotated points, prompting the LLM to use them in its answer when relevant. We collected 79k point-explanation annotations on 14k images.
PixMo-Points。
指点数据服务三个目标：(1) 按文本描述指到目标；(2) 靠指点计数；(3) 回答时用指点作视觉解释。前两者让标注者先指并描述某物，再穷尽标出图中每一实例；另采集「不存在」样本，使模型学会处理目标不在图中的情形。指点序列天然构成计数的思维链。最终在 22.3 万张图上得到 230 万问–点对。为把点当解释，我们改造 AskModelAnything 流程，允许标注者把带文本标注的点列表交给 LLM，并提示在相关时写入回答；在 1.4 万张图上收集 7.9 万条点解释标注。
PixMo-CapQA.
We generated 214k question-answer pairs, covering diverse topics and styles, from 165k images by prompting a language-only LLM to ask and answer questions given only the ground-truth caption for an image.
PixMo-CapQA。
仅凭图像的真值描述，提示纯语言模型自问自答，从 16.5 万张图生成覆盖多样主题与风格的 21.4 万问答对。
PixMo-Docs.
We used an extensive and carefully tuned prompting framework to prompt an LLM to generate code for 255k text and figure-heavy images, including charts, documents, tables, and diagrams. We then prompted the LLM to generate 2.3M question-answer pairs based on privileged access to the code (the images were not used).
PixMo-Docs。
用精心调过的大量提示，让 LLM 生成 25.5 万张偏文本与图表的图像代码（图表、文档、表格、示意图等）；再凭对代码的特权访问（不看图）生成 230 万问答对。
PixMo-Clocks.
We rendered synthetic clocks matched with a time-telling question-answer pair. The images use 50 different watch bodies and 160k realistic diverse watch faces set to random times. We collected 826k examples.
PixMo-Clocks。
渲染合成时钟并配对报时问答。约 50 种表体与约 16 万张真实多样表盘，时间随机。共 82.6 万例。
PixMo-Count.
We used a standard non-VLM object detector [136] on web images to create image and counting QA pairs. For each image, we selected the class with the most detections after strict confidence thresholding. Following CountBenchQA [10], we manually verified 120 samples per count from 2 to 10, creating validation and test sets of 540 images each. These diverse images form a more challenging counting QA set than CountBenchQA, which has reported limitations [10]. The remaining samples with counts between 0 and 10 form a training set of 36k images, each annotated with points (object centers) and a QA pair.
PixMo-Count。
用标准非 VLM 目标检测器 [136] 在网络图上生成计数问答。每图在严格置信度阈值后取检测最多的类别。参照 CountBenchQA [10]，对计数 2–10 每档人工核验 120 例，得到各 540 张的验证/测试集；图像更杂，比已有局限的 CountBenchQA [10] 更难。其余计数 0–10 的样本构成 3.6 万张训练集，每张带物体中心点与问答对。
4 Training
4 训练
Pre-training.
We pre-train all model parameters on PixMo-Cap to generate either the caption or one of the audio transcripts for a given image. A prompt specifies which style to generate and, 90% of the time, includes a length hint to guide the model’s output length. This hint improves caption and pre-training quality, as shown in Section 6.
预训练。
在 PixMo-Cap 上预训练全部参数，让模型为给定图像生成描述或某条音频转写。提示指定生成风格，并在 90% 情形加入长度提示以约束输出长度。第 6 节显示这能提升描述与预训练质量。
Previous work has often included a separate training stage to tune only the vision-language connector [59, 106, 27, 69, 10]. We find this step unnecessary when pre-training on PixMo-Cap (see Section 6), also explored in [48]. Instead, we apply a higher learning rate with a shorter warmup for the connector parameters, allowing them to adjust more quickly at the start of training. Skipping this stage reduces training time and complexity, and eliminates the need for the noisy web-scale data typically used in this phase.
以往工作常另设阶段只调视觉–语言连接器 [59, 106, 27, 69, 10]。在 PixMo-Cap 上预训练时我们发现这一步不必（见第 6 节；[48] 也有类似探讨）。改为给连接器更高学习率与更短预热，使其开训更快适应。跳过该阶段可减时降复杂度，也免去这一阶段常用的嘈杂网络级数据。
We train for four epochs using AdamW [50, 73] with a cosine learning rate decaying to 10% of its peak. Learning rates are set to 2e-4 (connector), 6e-6 (ViT), and 2e-5 (LM), with a 200-step warmup for the connector and 2000 steps for the ViT and LM. Gradient clipping is applied separately to the LM, image encoder, and connector parameters. Full hyper-parameters are provided in the Appendix.
用 AdamW [50, 73] 训练四个 epoch，余弦学习率衰减到峰值的 10%。学习率：连接器 2e-4、ViT 6e-6、LM 2e-5；连接器预热 200 步，ViT 与 LM 预热 2000 步。对 LM、图像编码器、连接器分别做梯度裁剪。完整超参见附录。
Figure 4: Datasets used for fine-tuning, shown in proportion to their sampling rates. Green denotes human-annotated data we collected, blue denotes synthetic data we generated, and purple represents pre-existing academic datasets. PixMo-Docs has been subdivided into charts, tables, diagrams, and other.
图 4：微调所用数据集，面积按采样率比例。绿色为我们采集的人工标注，蓝色为我们生成的合成数据，紫色为已有学术集。PixMo-Docs 再拆成图表、表格、示意图与其他。
Fine-tuning.
We fine-tune the model on a mix of PixMo datasets and open-source training datasets, including: VQA v2.0 (COCO 2014 subset) [36], TextVQA [100], OK-VQA [81], ChartQA (re-weighted to balance human and augmented examples) [82], DocVQA [83], InfographicVQA [84], AI2D (transparent and opaque label boxes) [49], A-OKVQA [99], AndroidControl [62], ScienceQA [76], TabMWP [77], ST-VQA [11], TallyQA [2], DVQA [46], FigureQA [47], and PlotQA [86].
微调。
在 PixMo 与开源训练集的混合上微调，包括：VQA v2.0（COCO 2014 子集） [36]、TextVQA [100]、OK-VQA [81]、ChartQA（重加权以平衡人工与增强样本） [82]、DocVQA [83]、InfographicVQA [84]、AI2D（透明与不透明标签框） [49]、A-OKVQA [99]、AndroidControl [62]、ScienceQA [76]、TabMWP [77]、ST-VQA [11]、TallyQA [2]、DVQA [46]、FigureQA [47]、PlotQA [86]。
We sample datasets at rates proportional to the square root of their size, with manual down weighting of some very large synthetic datasets (PlotQA, FigureQA, DVQA, and PixMo-Clocks). We observe that pointing tasks learn more slowly than QA tasks, so we significantly up-weight the pointing data. Final mixture rates are shown in Figure 4, with full details in the Appendix.
采样率与数据集规模的平方根成正比，并对 PlotQA、FigureQA、DVQA、PixMo-Clocks 等超大合成集人工降权。指点任务学得比问答慢，故显著上调指点数据。最终混合比例见图 4，细节见附录。
The academic datasets in this mixture teach specific skills and help the model perform well on corresponding benchmark test sets. However, these datasets often have answer styles that are not ideal for user interactions, as answers are usually very short and may reflect unique stylistic quirks from data collection (e.g., DocQA requires verbatim text from documents, while ChartQA specifies digits without commas). To prevent these styles from affecting user-facing responses, we prompt the model with a task-specific style tag (e.g., prefixing VQA v2.0 questions with “vqa2:”). The model learns to use these styles only when requested.
混合中的学术集教具体技能，并帮助对应基准测试集得分。但其答案风格往往不宜直接面向用户：答案通常极短，还带采集时的奇特约定（如 DocQA 要求 verbatim 抄文档，ChartQA 要求数字不加逗号）。为避免污染用户侧回答，我们给模型加任务风格标签（如 VQA v2.0 问题前缀 “vqa2:”）。模型只在被请求时才用这些风格。
We use style tags for all datasets except PixMo-AskModelAnything, -CapQA, -Points, -Count and -Cap. For PixMo-Cap, we create 30 prompts for caption generation. For pointing data, we create 100 question templates that ask for the location or count of the target expression. The model then returns a list of points and, for counting questions, the total count. These prompts and templates are randomly sampled during training. We still use a style tag for pointing-as-an-explanation data since we find performance in this mode can be less reliable, so it should only be used when users request it.
除 PixMo-AskModelAnything、-CapQA、-Points、-Count 与 -Cap 外，其余数据集都用风格标签。PixMo-Cap 约准备 30 条描述生成提示；指点数据约 100 条问位置或计数的模板，模型返回点列表，计数题再给总数。训练时随机抽提示/模板。点作解释的数据仍加风格标签，因该模式不够稳，宜仅在用户明确要求时启用。
For pointing, the model outputs points as plain-text coordinates normalized between 0 and 100. When pointing to multiple items, points are ordered top-down, left-to-right, with each point numbered (see Figure 2 and details in the Appendix). Pointing enables a unique chain-of-thought approach to counting where the model counts by sequentially pointing to each occurrence of the target object, improving performance (see Section 6).
指点时，模型以纯文本输出归一化到 0–100 的坐标。多目标时按从上到下、从左到右编号（见图 2 与附录）。指点还带来独特的计数思维链：依次指向每个目标再给出总数，可提升表现（见第 6 节）。
5 Evaluation
5 评测
| model | AI2D test [49] | ChartQA test [82] | VQA v2.0 testdev [36] | DocVQA test [83] | InfoQA test [84] | TextVQA val [100] | RealWorldQA [116] | MMMU val [129] | MathVista testmini [78] | CountBenchQA [10] | PixMo-Count test | Average | Elo score | Elo rank |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| API call only | ||||||||||||||
| GPT-4V [88] | 89.4 | 78.1 | 77.2 | 87.2 | 75.1 | 78.0 | 61.4 | 63.1 | 58.1 | 69.9 | 45.0 | 71.1 | 1041 | 10 |
| GPT-4o-0513 [90] | 94.2 | 85.7 | 78.7 | 92.8 | 79.2 | 77.4 | 75.4 | 69.1 | 63.8 | 87.9 | 59.6 | 78.5 | 1079 | 1 |
| Gemini 1.5 Flash [103] | 91.7 | 85.4 | 80.1 | 89.9 | 75.3 | 78.7 | 67.5 | 56.1 | 58.4 | 81.6 | 61.1 | 75.1 | 1054 | 7 |
| Gemini 1.5 Pro [103] | 94.4 | 87.2 | 80.2 | 93.1 | 81.0 | 78.7 | 70.4 | 62.2 | 63.9 | 85.8 | 64.3 | 78.3 | 1074 | 3 |
| Claude-3 Haiku [7] | 86.7 | 81.7 | 68.4 | 88.8 | 56.1 | 67.3 | 45.5 | 50.2 | 46.4 | 83.0 | 43.9 | 65.3 | 999 | 18 |
| Claude-3 Opus [7] | 88.1 | 80.8 | 66.3 | 89.3 | 55.6 | 67.5 | 49.8 | 59.4 | 50.5 | 83.6 | 43.3 | 66.7 | 971 | 21 |
| Claude-3.5 Sonnet [7] | 94.7 | 90.8 | 70.7 | 95.2 | 74.3 | 74.1 | 60.1 | 68.3 | 67.7 | 89.7 | 58.3 | 76.7 | 1069 | 4 |
| Open weights only | ||||||||||||||
| PaliGemma-mix-3B [10] | 72.3 | 33.7 | 76.3 | 31.3 | 21.4 | 56.0 | 55.2 | 34.9 | 28.7 | 80.6 | 60.0 | 50.0 | 937 | 27 |
| Phi3.5-Vision-4B [1] | 78.1 | 81.8 | 75.7 | 69.3 | 36.6 | 72.0 | 53.6 | 43.0 | 43.9 | 64.6 | 38.3 | 59.7 | 982 | 19 |
| Qwen2-VL-7B [111] | 83.0 | 83.0 | 82.9 | 94.5 | 76.5 | 84.3 | 70.1 | 54.1 | 58.2 | 76.5 | 48.0 | 73.7 | 1025 | 14 |
| Qwen2-VL-72B [111] | 88.1 | 88.3 | 81.9 | 96.5 | 84.5 | 85.5 | 77.8 | 64.5 | 70.5 | 80.4 | 55.7 | 79.4 | 1037 | 12 |
| InternVL2-8B [104] | 83.8 | 83.3 | 76.7 | 91.6 | 74.8 | 77.4 | 64.2 | 51.2 | 58.3 | 57.8 | 43.9 | 69.4 | 953 | 23 |
| InternVL2-Llama-3-76B [104] | 87.6 | 88.4 | 85.6 | 94.1 | 82.0 | 84.4 | 72.7 | 58.2 | 65.5 | 74.7 | 54.6 | 77.1 | 1018 | 16 |
| Pixtral-12B [3] | 79.0 | 81.8 | 80.2 | 90.7 | 50.8 | 75.7 | 65.4 | 52.5 | 58.0 | 78.8 | 51.7 | 69.5 | 1016 | 17 |
| Llama-3.2V-11B-Instruct [5] | 91.1 | 83.4 | 75.2 | 88.4 | 63.6 | 79.7 | 64.1 | 50.7 | 51.5 | 73.1 | 47.4 | 69.8 | 1040 | 11 |
| Llama-3.2V-90B-Instruct [5] | 92.3 | 85.5 | 78.1 | 90.1 | 67.2 | 82.3 | 69.8 | 60.3 | 57.3 | 78.5 | 58.5 | 74.5 | 1063 | 5 |
| Open weights + data ( distilled) | ||||||||||||||
| LLaVA-1.5-7B [69] | 55.5 | 17.8 | 78.5 | 28.1 | 25.8 | 58.2 | 54.8 | 35.7 | 25.6 | 40.1 | 27.6 | 40.7 | 951 | 26 |
| LLaVA-1.5-13B [69] | 61.1 | 18.2 | 80.0 | 30.3 | 29.4 | 61.3 | 55.3 | 37.0 | 27.7 | 47.1 | 35.2 | 43.9 | 960 | 22 |
| xGen-MM-interleave-4B [119] | 74.2 | 60.0 | 81.5 | 61.4 | 31.5 | 71.0 | 61.2 | 41.1 | 40.5 | 81.9 | 50.2 | 59.5 | 979 | 20 |
| Cambrian-1-8B [106] | 73.0 | 73.3 | 81.2 | 77.8 | 41.6 | 71.7 | 64.2 | 42.7 | 49.0 | 76.4 | 46.6 | 63.4 | 952 | 25 |
| Cambrian-1-34B [106] | 79.7 | 75.6 | 83.8 | 75.5 | 46.0 | 76.7 | 67.8 | 49.7 | 53.2 | 75.6 | 50.7 | 66.8 | 953 | 24 |
| LLaVA OneVision-7B [59] | 81.4 | 80.0 | 84.0 | 87.5 | 68.8 | 78.3 | 66.3 | 48.8 | 63.2 | 78.8 | 54.4 | 72.0 | 1024 | 15 |
| LLaVA OneVision-72B [59] | 85.6 | 83.7 | 85.2 | 91.3 | 74.9 | 80.5 | 71.9 | 56.8 | 67.5 | 84.3 | 60.7 | 76.6 | 1051 | 8 |
| The Molmo family: Open weights, Open data, Open training code, Open evaluations | ||||||||||||||
| MolmoE-1B | 86.4 | 78.0 | 83.9 | 77.7 | 53.9 | 78.8 | 60.4 | 34.9 | 34.0 | 87.2 | 79.6 | 68.6 | 1032 | 13 |
| Molmo-7B-O | 90.7 | 80.4 | 85.3 | 90.8 | 70.0 | 80.4 | 67.5 | 39.3 | 44.5 | 89.0 | 83.3 | 74.6 | 1051 | 9 |
| Molmo-7B-D | 93.2 | 84.1 | 85.6 | 92.2 | 72.6 | 81.7 | 70.7 | 45.3 | 51.6 | 88.5 | 84.8 | 77.3 | 1056 | 6 |
| Molmo-72B | 96.3 | 87.3 | 86.5 | 93.5 | 81.9 | 83.1 | 75.2 | 54.1 | 58.6 | 91.2 | 85.2 | 81.2 | 1077 | 2 |
Table 1: We present academic benchmark results for 10 common datasets, plus a new counting benchmark, PixMo-Count, which features more challenging natural images than CountBenchQA. We categorize models into four groups: (top) proprietary models accessible only via API calls, (upper middle) models with released weights but closed data, (lower middle) models with released weights and training data (noting some of these use distillation () from proprietary VLMs via synthetic data), and (bottom) the Molmo family of models.
| 模型 | AI2D test [49] | ChartQA test [82] | VQA v2.0 testdev [36] | DocVQA test [83] | InfoQA test [84] | TextVQA val [100] | RealWorldQA [116] | MMMU val [129] | MathVista testmini [78] | CountBenchQA [10] | PixMo-Count test | 平均 | Elo 分 | Elo 名次 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 仅 API 调用 | ||||||||||||||
| GPT-4V [88] | 89.4 | 78.1 | 77.2 | 87.2 | 75.1 | 78.0 | 61.4 | 63.1 | 58.1 | 69.9 | 45.0 | 71.1 | 1041 | 10 |
| GPT-4o-0513 [90] | 94.2 | 85.7 | 78.7 | 92.8 | 79.2 | 77.4 | 75.4 | 69.1 | 63.8 | 87.9 | 59.6 | 78.5 | 1079 | 1 |
| Gemini 1.5 Flash [103] | 91.7 | 85.4 | 80.1 | 89.9 | 75.3 | 78.7 | 67.5 | 56.1 | 58.4 | 81.6 | 61.1 | 75.1 | 1054 | 7 |
| Gemini 1.5 Pro [103] | 94.4 | 87.2 | 80.2 | 93.1 | 81.0 | 78.7 | 70.4 | 62.2 | 63.9 | 85.8 | 64.3 | 78.3 | 1074 | 3 |
| Claude-3 Haiku [7] | 86.7 | 81.7 | 68.4 | 88.8 | 56.1 | 67.3 | 45.5 | 50.2 | 46.4 | 83.0 | 43.9 | 65.3 | 999 | 18 |
| Claude-3 Opus [7] | 88.1 | 80.8 | 66.3 | 89.3 | 55.6 | 67.5 | 49.8 | 59.4 | 50.5 | 83.6 | 43.3 | 66.7 | 971 | 21 |
| Claude-3.5 Sonnet [7] | 94.7 | 90.8 | 70.7 | 95.2 | 74.3 | 74.1 | 60.1 | 68.3 | 67.7 | 89.7 | 58.3 | 76.7 | 1069 | 4 |
| 仅开放权重 | ||||||||||||||
| PaliGemma-mix-3B [10] | 72.3 | 33.7 | 76.3 | 31.3 | 21.4 | 56.0 | 55.2 | 34.9 | 28.7 | 80.6 | 60.0 | 50.0 | 937 | 27 |
| Phi3.5-Vision-4B [1] | 78.1 | 81.8 | 75.7 | 69.3 | 36.6 | 72.0 | 53.6 | 43.0 | 43.9 | 64.6 | 38.3 | 59.7 | 982 | 19 |
| Qwen2-VL-7B [111] | 83.0 | 83.0 | 82.9 | 94.5 | 76.5 | 84.3 | 70.1 | 54.1 | 58.2 | 76.5 | 48.0 | 73.7 | 1025 | 14 |
| Qwen2-VL-72B [111] | 88.1 | 88.3 | 81.9 | 96.5 | 84.5 | 85.5 | 77.8 | 64.5 | 70.5 | 80.4 | 55.7 | 79.4 | 1037 | 12 |
| InternVL2-8B [104] | 83.8 | 83.3 | 76.7 | 91.6 | 74.8 | 77.4 | 64.2 | 51.2 | 58.3 | 57.8 | 43.9 | 69.4 | 953 | 23 |
| InternVL2-Llama-3-76B [104] | 87.6 | 88.4 | 85.6 | 94.1 | 82.0 | 84.4 | 72.7 | 58.2 | 65.5 | 74.7 | 54.6 | 77.1 | 1018 | 16 |
| Pixtral-12B [3] | 79.0 | 81.8 | 80.2 | 90.7 | 50.8 | 75.7 | 65.4 | 52.5 | 58.0 | 78.8 | 51.7 | 69.5 | 1016 | 17 |
| Llama-3.2V-11B-Instruct [5] | 91.1 | 83.4 | 75.2 | 88.4 | 63.6 | 79.7 | 64.1 | 50.7 | 51.5 | 73.1 | 47.4 | 69.8 | 1040 | 11 |
| Llama-3.2V-90B-Instruct [5] | 92.3 | 85.5 | 78.1 | 90.1 | 67.2 | 82.3 | 69.8 | 60.3 | 57.3 | 78.5 | 58.5 | 74.5 | 1063 | 5 |
| 开放权重 + 数据 （ 蒸馏） | ||||||||||||||
| LLaVA-1.5-7B [69] | 55.5 | 17.8 | 78.5 | 28.1 | 25.8 | 58.2 | 54.8 | 35.7 | 25.6 | 40.1 | 27.6 | 40.7 | 951 | 26 |
| LLaVA-1.5-13B [69] | 61.1 | 18.2 | 80.0 | 30.3 | 29.4 | 61.3 | 55.3 | 37.0 | 27.7 | 47.1 | 35.2 | 43.9 | 960 | 22 |
| xGen-MM-interleave-4B [119] | 74.2 | 60.0 | 81.5 | 61.4 | 31.5 | 71.0 | 61.2 | 41.1 | 40.5 | 81.9 | 50.2 | 59.5 | 979 | 20 |
| Cambrian-1-8B [106] | 73.0 | 73.3 | 81.2 | 77.8 | 41.6 | 71.7 | 64.2 | 42.7 | 49.0 | 76.4 | 46.6 | 63.4 | 952 | 25 |
| Cambrian-1-34B [106] | 79.7 | 75.6 | 83.8 | 75.5 | 46.0 | 76.7 | 67.8 | 49.7 | 53.2 | 75.6 | 50.7 | 66.8 | 953 | 24 |
| LLaVA OneVision-7B [59] | 81.4 | 80.0 | 84.0 | 87.5 | 68.8 | 78.3 | 66.3 | 48.8 | 63.2 | 78.8 | 54.4 | 72.0 | 1024 | 15 |
| LLaVA OneVision-72B [59] | 85.6 | 83.7 | 85.2 | 91.3 | 74.9 | 80.5 | 71.9 | 56.8 | 67.5 | 84.3 | 60.7 | 76.6 | 1051 | 8 |
| Molmo 系列：开放权重、开放数据、开放训练代码、开放评测 | ||||||||||||||
| MolmoE-1B | 86.4 | 78.0 | 83.9 | 77.7 | 53.9 | 78.8 | 60.4 | 34.9 | 34.0 | 87.2 | 79.6 | 68.6 | 1032 | 13 |
| Molmo-7B-O | 90.7 | 80.4 | 85.3 | 90.8 | 70.0 | 80.4 | 67.5 | 39.3 | 44.5 | 89.0 | 83.3 | 74.6 | 1051 | 9 |
| Molmo-7B-D | 93.2 | 84.1 | 85.6 | 92.2 | 72.6 | 81.7 | 70.7 | 45.3 | 51.6 | 88.5 | 84.8 | 77.3 | 1056 | 6 |
| Molmo-72B | 96.3 | 87.3 | 86.5 | 93.5 | 81.9 | 83.1 | 75.2 | 54.1 | 58.6 | 91.2 | 85.2 | 81.2 | 1077 | 2 |
表 1：10 个常用学术基准结果，外加新的计数基准 PixMo-Count（自然图像比 CountBenchQA 更难）。模型分四组：（上）仅 API 可达的闭源模型；（中上）开放权重但数据封闭；（中下）开放权重与训练数据（部分经闭源 VLM 合成数据蒸馏，标 ）；（下）Molmo 系列。
| ViT-L/14 | cap | 11-avg |
|---|---|---|
| OpenAI CLIP 336px | 54.1 | 76.9 |
| MetaCLIP 336px | 54.1 | 77.2 |
| SigLIP-So400m 384px | 54.4 | 77.1 |
| DINOv2 336px | 53.2 | 75.6 |
| # crops train, test | cap | 11-avg |
|---|---|---|
| 14, 4 | 52.0 | 71.0 |
| 14, 12 | 52.0 | 74.1 |
| 14, 36 | 52.0 | 74.2 |
| 12, 12 | 54.1 | 74.9 |
| 12, 36 | 54.1 | 76.9 |
| 36, 36 | 54.0 | 77.2 |
| pre-train, fine-tune | cap | 11-avg |
|---|---|---|
| off, off | 53.1 | 74.6 |
| off, on | 53.1 | 76.6 |
| on, on | 53.7 | 77.0 |
| on (text only), on | 54.1 | 76.9 |
| cropping | cap | 11-avg |
|---|---|---|
| single | 46.7 | 62.8 |
| multi, no overlap | 53.4 | 75.7 |
| multi, overlap | 54.1 | 76.9 |
| setting | cap | 11-avg |
|---|---|---|
| off | 53.0 | 76.2 |
| on | 54.1 | 76.9 |
| 22 pooling | cap | 11-avg |
|---|---|---|
| stacking | 53.7 | 76.1 |
| attention | 54.1 | 76.9 |
(a)
| ViT-L/14 | cap | 11-avg |
|---|---|---|
| OpenAI CLIP 336px | 54.1 | 76.9 |
| MetaCLIP 336px | 54.1 | 77.2 |
| SigLIP-So400m 384px | 54.4 | 77.1 |
| DINOv2 336px | 53.2 | 75.6 |
| 训练/测试裁剪数 | cap | 11-avg |
|---|---|---|
| 14, 4 | 52.0 | 71.0 |
| 14, 12 | 52.0 | 74.1 |
| 14, 36 | 52.0 | 74.2 |
| 12, 12 | 54.1 | 74.9 |
| 12, 36 | 54.1 | 76.9 |
| 36, 36 | 54.0 | 77.2 |
| 预训练、微调 | cap | 11-avg |
|---|---|---|
| 关, 关 | 53.1 | 74.6 |
| 关, 开 | 53.1 | 76.6 |
| 开, 开 | 53.7 | 77.0 |
| 开（仅文本）, 开 | 54.1 | 76.9 |
| 裁剪 | cap | 11-avg |
|---|---|---|
| 单裁剪 | 46.7 | 62.8 |
| 多裁剪，无重叠 | 53.4 | 75.7 |
| 多裁剪，有重叠 | 54.1 | 76.9 |
| 设置 | cap | 11-avg |
|---|---|---|
| 关 | 53.0 | 76.2 |
| 开 | 54.1 | 76.9 |
| 池化 | cap | 11-avg |
|---|---|---|
| 堆叠 | 53.7 | 76.1 |
| 注意力 | 54.1 | 76.9 |
(a)
We evaluate on academic benchmarks, noting that comparisons require care, as prompting, alignment with benchmark-specific answer styles, and use of benchmark training data can significantly affect performance. To complement this, we conduct a human evaluation to rank models based on user preference.
我们在学术基准上评测，但比较需谨慎：提示、是否对齐基准答题风格、是否使用基准训练集，都会显著影响分数。为此我们另做人工评测，按用户偏好给模型排序。
For academic benchmarking, we gather or compute results for all models on 10 common datasets and the PixMo-Count test set, which we include due to its higher difficulty compared to existing counting benchmarks. We prioritize author-published results but fill in missing results with the best previously reported values from technical reports or sources like the OpenVLM Leaderboard. If data is still missing, we compute it ourselves. Notably, computing results is challenging, as performance can vary significantly (e.g., by 10%) based on evaluation details. Additionally, critical information such as prompts or data processing steps is often unavailable, making it hard to reproduce results, highlighting the need for evaluation openness.
学术评测上，我们汇总或自行计算各模型在 10 个常用集与 PixMo-Count 测试集上的结果；纳入后者是因其难度高于既有计数基准。优先采用作者公布成绩，缺失则用技术报告或 OpenVLM Leaderboard 等处已报最佳值补齐；仍缺则自行计算。自行复现并不容易：评测细节可让分数差出约 10%。提示与数据处理步骤也常不公开，难复现，凸显评测开放的必要。
In our human evaluation, we collect 15k diverse image-text prompt pairs and queried the VLMs for responses. We sample and present the resulting image-text-response triplets for all VLM pairings to a group of 870 human annotators, who provide pairwise preference rankings. Across all model pairs, we gather over 325k ratings (450 per model pair). From this data, we calculate an Elo ranking using the Bradley-Terry model, following the methodology of Chatbot Arena [21].
人工评测中，我们收集 1.5 万组多样图文提示，向各 VLM 要回答；再对所有模型配对抽样，把图–文–答三元组交给约 870 名标注者做成对偏好。全模型对合计逾 32.5 万次评分（每对约 450）。据此用 Bradley-Terry 模型算 Elo，方法对齐 Chatbot Arena [21]。
For Molmo, we evaluate all academic datasets with 36 crops (up from 12 used in training), except for counting tasks, as pointing capabilities do not generalize well with different numbers of test crops. A small amount of high-res post-training can resolve this issue, see the Appendix.
对 Molmo，除计数任务外，学术集评测一律用 36 个裁剪（训练时为 12）；因指点能力对测试裁剪数变化泛化不佳。少量高分辨率后训练可缓解，见附录。
When possible, we use relevant style prompts[^1] (e.g., “vqa2:”). For evaluation-only datasets, we use the VQA v2.0 (for short answer) or A-OKVQA (for multiple choice) style tags to elicit the often expected short answer style. For human evaluation, we omit style tags and use 12 crops, as some counting questions use pointing. Evaluators are only shown the output text, not the points.
尽量使用相应风格提示[^1]（如 “vqa2:”）。仅评测集则用 VQA v2.0（短答）或 A-OKVQA（多选）风格标签，以引出常期望的短答风格。人工评测不加风格标签，并用 12 裁剪（部分计数题依赖指点）。评测者只看文本输出，不看点。
Broadly speaking, the academic benchmark results and human evaluation agree, with the exception of Qwen2-VL [111], which performs strongly on the academic benchmarks and comparatively underperforms in the human evaluation. We highlight a few key results from Table 1:
总体而言，学术基准与人工评测一致；例外是 Qwen2-VL [111]：学术很强，人工偏好相对偏弱。表 1 中若干要点如下：
- • MolmoE-1B, our most efficient model based on the OLMoE-1B-7B mixture-of-experts LLM, nearly matches GPT-4V on academic benchmarks and Elo.
- • 基于 OLMoE-1B-7B 混合专家 LLM 的最省算力模型 MolmoE-1B，学术基准与 Elo 均接近 GPT-4V。
- • Molmo models based on OLMo-7B-1024-preview and Qwen2 7B LLMs perform between GPT-4V and GPT-4o on academic benchmarks and Elo.
- • 基于 OLMo-7B-1024-preview 与 Qwen2 7B 的 Molmo，学术与 Elo 均介于 GPT-4V 与 GPT-4o 之间。
- • Our best-in-class Qwen2 72B based model achieves the highest academic benchmark score and ranks second in Elo, just behind GPT-4o.
- • 最优的 Qwen2 72B 版学术均分最高，Elo 仅次于 GPT-4o。
- • Our best model also outperforms many state-of-the-art proprietary systems, including Gemini 1.5 Pro and Flash and Claude 3.5 Sonnet.
- • 最优模型还超过 Gemini 1.5 Pro/Flash、Claude 3.5 Sonnet 等多家顶尖闭源系统。
Molmo-72B also underwent an independent Elo evaluation via Chatbot Arena, where it outperforms all open models but ranks lower than several proprietary models (e.g., GPT-4o and Claude 3.5 Sonnet).[^2] The full results table is in the Appendix. The difference likely stems from the types of questions evaluated. While we cannot perform a full analysis since the questions are not public, we do note our data includes many counting and image-description questions which are particular strengths of Molmo.
Molmo-72B 还在 Chatbot Arena 做了独立 Elo：超过全部开源模型，但仍低于若干闭源（如 GPT-4o、Claude 3.5 Sonnet）。[^2] 完整表见附录。差距很可能来自题目类型。题目未公开，无法完整分析；我们注意到自有数据含大量计数与图像描述题，而这正是 Molmo 的强项。
Molmo excels at answering questions about natural images, matching or outperforming all models on the zero-shot RealWorldQA benchmark and achieving state-of-the-art results on the highly competitive VQA v2.0. On OCR-centric benchmarks (ChartQA, DocQA, InfoQA, TextVQA), Molmo surpasses other open models and some proprietary ones but trails slightly behind Qwen2-VL. On counting tasks (CountBenchQA and PixMo-Count), Molmo leads all models due to our new pointing data and chain-of-thought point-and-count abilities. However, on reasoning tasks (MMMU, MathVista) Molmo lags, likely because its training mix lacks data focused on advanced reasoning.
Molmo 擅长自然图像问答：零样本 RealWorldQA 持平或超过所有模型，竞争激烈的 VQA v2.0 上亦达顶尖。OCR 向基准（ChartQA、DocQA、InfoQA、TextVQA）上超过其他开源及部分闭源，但略逊于 Qwen2-VL。计数（CountBenchQA、PixMo-Count）因新指点数据与「先指后数」思维链领先全部模型。推理向（MMMU、MathVista）则偏弱，可能因训练混合缺少高阶推理数据。
We conduct several additional skill-specific evaluations, summarized here with details in the Appendix. On a clock-reading benchmark [121], Molmo at all scales dramatically outperforms other VLMs including proprietary ones, but trails specialized non-VLM models [121]. To assess Molmo’s potential for action, we tested Molmo-72B on AndroidControl [62], achieving 88.7% low-level and 69.0% high-level accuracy, comparable to the reported 83.2% and 70.8% in [62]. On NLP benchmarks, Molmo shows a slight performance drop versus its component LLM, which can be offset by additional text-only data. We also introduce a new pointing benchmark using SAM [51], where Molmo models at all scales demonstrate strong performance.
另做若干技能专项评测，此处摘要，细节见附录。读钟基准 [121] 上，各规模 Molmo 大幅超过含闭源在内的其他 VLM，但仍不及专用非 VLM 模型 [121]。为评估行动潜力，Molmo-72B 在 AndroidControl [62] 上达到低层 88.7%、高层 69.0%，接近原文报告的 83.2% 与 70.8%。NLP 基准上相对其成分 LLM 略有掉点，可用额外纯文本数据补回。我们还用 SAM [51] 建了新指点基准，各规模 Molmo 均表现强。
| # PixMo-Cap images | cap | 11-avg |
|---|---|---|
| 000k (0.0%) | - | 74.9 |
| 089k (12.5%) | 49.6 | 75.5 |
| 178k (25.0%) | 51.6 | 76.3 |
| 356k (50.0%) | 52.6 | 76.2 |
| 712k (100.0%) | 54.1 | 76.9 |
| data | cap | 11-avg |
|---|---|---|
| stage 0.5 LAION | 53.9 | 76.9 |
| ShareGPT4V+o (158k images) | 36.3 | 74.9 |
| PixMo-Cap images: | ||
| our raw transcripts only | 45.2 | 76.4 |
| our cleaned transcripts only | 53.0 | 76.5 |
| our raw & cleaned transcripts | 54.1 | 76.9 |
| captioned by GPT-4o | 52.9 | 77.5 |
| data | 11-avg |
|---|---|
| academic only | 72.5 |
| plus PixMo-Docs | 74.0 |
| PixMo- plus academic | 76.9 |
| remove PixMo-AMA | 76.8 |
| remove PixMo-CapQA | 77.0 |
| remove PixMo-Docs | 75.8 |
| remove PixMo-Clocks | 76.9 |
| remove pointing task | 76.2 |
(a)
| PixMo-Cap 图像数 | cap | 11-avg |
|---|---|---|
| 000k (0.0%) | - | 74.9 |
| 089k (12.5%) | 49.6 | 75.5 |
| 178k (25.0%) | 51.6 | 76.3 |
| 356k (50.0%) | 52.6 | 76.2 |
| 712k (100.0%) | 54.1 | 76.9 |
| 数据 | cap | 11-avg |
|---|---|---|
| stage 0.5 LAION | 53.9 | 76.9 |
| ShareGPT4V+o（15.8 万图） | 36.3 | 74.9 |
| PixMo-Cap 图像： | ||
| 仅原始转写 | 45.2 | 76.4 |
| 仅清洗转写 | 53.0 | 76.5 |
| 原始+清洗转写 | 54.1 | 76.9 |
| 由 GPT-4o 描述 | 52.9 | 77.5 |
| 数据 | 11-avg |
|---|---|
| 仅学术集 | 72.5 |
| 加上 PixMo-Docs | 74.0 |
| PixMo- + 学术 | 76.9 |
| 去掉 PixMo-AMA | 76.8 |
| 去掉 PixMo-CapQA | 77.0 |
| 去掉 PixMo-Docs | 75.8 |
| 去掉 PixMo-Clocks | 76.9 |
| 去掉指点任务 | 76.2 |
(a)
| strategy | CBQA | PCQA |
|---|---|---|
| count | 87.9 | 80.2 |
| point then count | 89.4 | 86.3 |
| count then point | 81.5 | 77.6 |
| pointing + regex | 88.4 | 85.4 |
| order | CBQA | PCQA |
|---|---|---|
| on | 89.4 | 86.3 |
| off | 85.4 | 74.1 |
| points, length | CBQA | PCQA |
|---|---|---|
| actual, correct | 89.4 | 86.3 |
| random, correct | 85.9 | 76.3 |
| random, random | 76.3 | 75.7 |
| tokens | CBQA | PCQA |
|---|---|---|
| plain-text | 89.4 | 86.3 |
| special | 85.8 | 80.9 |
(a)
| 策略 | CBQA | PCQA |
|---|---|---|
| 只计数 | 87.9 | 80.2 |
| 先指再数 | 89.4 | 86.3 |
| 先数再指 | 81.5 | 77.6 |
| 指点 + 正则计数 | 88.4 | 85.4 |
| 顺序 | CBQA | PCQA |
|---|---|---|
| 开 | 89.4 | 86.3 |
| 关 | 85.4 | 74.1 |
| 点、长度 | CBQA | PCQA |
|---|---|---|
| 真实位置、正确个数 | 89.4 | 86.3 |
| 随机位置、正确个数 | 85.9 | 76.3 |
| 随机位置、随机个数 | 76.3 | 75.7 |
| token | CBQA | PCQA |
|---|---|---|
| 纯文本 | 89.4 | 86.3 |
| 特殊位置 token | 85.8 | 80.9 |
(a)
6 Ablations
6 消融
We performed extensive ablations on model design (Table 2) and training data (Table 3), reporting two metrics: an metric (“cap ”) we developed to measure the precision and recall of captions generated by the model after pre-training (details in the Appendix), and the average accuracy on our 11 benchmark suite (“11-avg”), using validation sets when available. We report because we believe it reflects broad-range image understanding learned during pre-training, and many of our modeling design choices were based on this evaluation since it does not require running the more costly fine-tuning stage. While performing these ablation we observed captioning improvements generally, but not always, correspond to benchmark suite improvements. Our ablations test modifications to the Molmo-7B-D model configuration with key findings summarized below and further details in Table 2 and 3 captions and the Appendix.
我们对模型设计（表 2）与训练数据（表 3）做了大量消融，报告两项指标：自研的 （“cap ”），衡量预训练后描述的精确率与召回（附录详述）；以及 11 项基准套件平均准确率（“11-avg”，有验证集则用验证集）。报告 是因它反映预训练学到的宽域图像理解，且许多建模选择据此做出——不必跑更贵的微调。消融中可见描述提升大体、但不总是对应基准提升。消融在 Molmo-7B-D 配置上改动，要点如下，细节见表 2、表 3 图注与附录。
Model ablations.
We vary several design choices, finding:
模型消融。
我们变动多项设计，发现：
- • Vision encoders trained on noisy web-scale data (CLIP, SigLIP, MetaCLIP) all work roughly the same, including the fully open MetaCLIP (Table ). Every model component and every bit of data in a Molmo model equipped with MetaCLIP and OLMo is open.
- • 在嘈杂网络级数据上训的视觉编码器（CLIP、SigLIP、MetaCLIP）效果大致相当，含完全开放的 MetaCLIP（表 ）。配备 MetaCLIP 与 OLMo 的 Molmo，每个组件与每一比特数据都开放。
- • Excluding pointing and captioning, increasing the image resolution by increasing the number of crops generally improves performance, and tuning at a high resolution yields a slight gain (Table ).
- • 除指点与描述外，靠增加裁剪数提高分辨率通常能提升表现；在高分辨率上微调还有小幅收益（表 ）。
- • Our novel text-only dropout used in pre-training improves captioning performance (Table ).
- • 预训练中我们提出的仅文本 dropout 能提升描述表现（表 ）。
- • Using multiple crops, instead of a single low-resolution input, is extremely important. Our novel overlapping crop design yields significant improvements (Table ).
- • 用多裁剪而非单一低分辨率输入极为重要；重叠裁剪设计带来显著提升（表 ）。
- • Our novel length-conditioned captioning is a strong pre-training task, improving downstream results (Table ).
- • 带长度条件的描述是强预训练任务，能改善下游（表 ）。
- • Attention pooling yields improvements to both metrics over the baseline feature stacking approach (Table ).
- • 注意力池化相对基线特征堆叠，两项指标都更好（表 ）。
Data ablations.
We train with various data choices, finding:
数据消融。
我们尝试多种数据选择，发现：
- • Scaling PixMo-Cap from 0 to 712k images significantly improves captioning and benchmark metrics (Table ).
- • 将 PixMo-Cap 从 0 扩到 71.2 万图，显著提升描述与基准指标（表 ）。
- • Adding noisy web-scale data does not improve our pre-training, our PixMo-Cap data yields better results than a similar amount of data from ShareGPT4v/o, and our human annotated captioning data works roughly as well as captioning data generated by GPT-4o on the PixMo-Cap images (Table ).
- • 加入嘈杂网络级数据无助预训练；同规模下 PixMo-Cap 优于 ShareGPT4v/o；我们的人工描述与在同一批 PixMo-Cap 图上用 GPT-4o 生成的描述效果大致相当（表 ）。
- • The PixMo supervised fine-tuning datasets improve benchmark task performance beyond the academic datasets, mainly by improving document-heavy and counting tasks (Table ).
- • PixMo 有监督微调集在学术集之上进一步抬高基准，主要来自偏文档与计数任务（表 ）。
Counting.
We ablate several details of counting using models fine-tuned on just PixMo-Points and PixMo-Count data:
计数。
仅在 PixMo-Points 与 PixMo-Count 上微调的模型上，我们消融若干计数细节：
- • Chain-of-thought point-then-count performs significantly better than generating a count only or generating a count followed by points. Pointing on its own, counted by regular expression, is only slightly worse (Table ).
- • 思维链「先指再数」显著优于只出计数或「先数再指」；仅指点再用正则数点数，只略差（表 ）。
- • Training with points in a predicable spatial order (top-down, left-to-right) works significantly better than training with unordered points (Table ).
- • 按可预期空间顺序（上→下、左→右）训练，显著优于无序点（表 ）。
- • Pointing uses extra inference computation that may improve results regardless of the generated text. However, we find that using correct point locations and counts works significantly better than using random locations and/or numbers of points (Table ).
- • 指点会多耗推理算力，或与生成文本无关也能抬分；但正确点位与个数显著优于随机位置和/或随机点数（表 ）。
- • Representing point coordinates in plain-text works better than introducing special location tokens (Table ).
- • 用纯文本表示点坐标优于引入特殊位置 token（表 ）。
| model | Elo score | win % vs. default |
|---|---|---|
| Claude-3.5 Sonnet | 1047 | 65% |
| PixMo-Cap w/ GPT-4o captions (Table ) | 1018 | 55% |
| PixMo-, remove PixMo-CapQA (Table ) | 1015 | 50% |
| Molmo-7B-D default | 1014 | n/a |
| PixMo-, no academic datasets (Table ) | 1013 | 42% |
| GPT-4V | 1010 | 47% |
| DINOv2 vision encoder (Table ) | 1999 | 45% |
| PixMo-, remove PixMo-AMA (Table ) | 1995 | 40% |
| no PixMo-Cap data (Table ) | 1990 | 35% |
| academic only (Table ) | 1897 | 17% |
Table 5: Elo scores and win rates (excluding ties) for select ablations of Molmo-7B-D and two API-only models for context.
| 模型 | Elo 分 | 相对默认胜率 |
|---|---|---|
| Claude-3.5 Sonnet | 1047 | 65% |
| PixMo-Cap + GPT-4o 描述（表 ） | 1018 | 55% |
| PixMo-，去掉 PixMo-CapQA（表 ） | 1015 | 50% |
| Molmo-7B-D 默认 | 1014 | 不适用 |
| PixMo-，无学术集（表 ） | 1013 | 42% |
| GPT-4V | 1010 | 47% |
| DINOv2 视觉编码器（表 ） | 1999 | 45% |
| PixMo-，去掉 PixMo-AMA（表 ） | 1995 | 40% |
| 无 PixMo-Cap（表 ） | 1990 | 35% |
| 仅学术集（表 ） | 1897 | 17% |
表 5：Molmo-7B-D 部分消融与两家仅 API模型的 Elo 与胜率（不含平局），供对照。
Human evaluation.
A human evaluation of select ablation models in Table 5 shows that PixMo data, especially PixMo-Cap and PixMo-AskModelAnything, is important for generating responses that users like. Academic datasets improve human scores, but are extremely poor if used on their own. GPT-4o captions on our images also perform well, which we believe is due to recent advances in GPT and the diversity of our image collection (e.g., the ShareGPT datasets significantly underperform our data even at the same scale, see Tables and ). While distilling from proprietary models might be effective, we emphasize that it is critical for the scientific community to understand how to train competitive VLMs without doing so. Molmo and PixMo take an important step towards this understanding.
人工评测。
表 5 中部分消融的人工评测显示：PixMo 数据——尤其 PixMo-Cap 与 PixMo-AskModelAnything——对用户喜欢的回答很重要。学术集能抬人工分，但单独使用极差。在我们图像上用 GPT-4o 描述也好，我们归因于 GPT 近期进步与图像集合多样性（如同规模下 ShareGPT 数据明显弱于我们的数据，见表 ）。从闭源蒸馏或许有效，但学界更需弄清不蒸馏也能训出有竞争力 VLM 的路径。Molmo 与 PixMo 朝这一理解迈出重要一步。
Appendix
附录
The appendix includes the following sections:
附录包含以下各节：
- • §A - Model Details
- • §A - 模型细节
- • §B - Training Details
- • §B - 训练细节
- • §C - Evaluation Details
- • §C - 评测细节
- • §D - Result Details
- • §D - 结果细节
- • §E - Ablation Details
- • §E - 消融细节
- • §F - Data Details
- • §F - 数据细节
- • §G - Dataset Examples
- • §G - 数据样例
- • §H - Related Work
- • §H - 相关工作
Appendix A Model Details
附录 A 模型细节
We present additional details about image encoding, hyperparameters, and implementation choices.
下文补充图像编码、超参数与实现选择。
A.1 Image Encoding
A.1 图像编码
Figure 7: Converting an image into tokens. The image (top left) is turned into a single low-res and several overlapping high-res crops (bottom left). Padding (the black borders) is used so each crop is square and the aspect ratio of the image is preserved. The final token sequence for the image (right, arranged top-down left-to-right with line breaks for clarity) is built by extracting patch-level features from the crops, shown here using images of the patches, and special tokens. An image start and image end token are placed before/after the high-res and low-res patches, and column tokens are inserted after each row of patches. This example uses 4 high-res crops and extracts features from 36 (66) patches per crop, in practice Molmo typically uses 12 high-res crops and extracts features from 144 (1212) patches per crop.
图 7：把图像变成 token。左上原图变为一个低分辨率裁剪与若干重叠高分辨率裁剪（左下）。黑色边框为填充，使每个裁剪为正方形并保持宽高比。最终图像 token 序列（右，为清晰自上而下、自左而右排并换行）由各裁剪的 patch 级特征（此处用 patch 小图示意）与特殊 token 组成：高低分辨率 patch 前后放图像起止 token，每行 patch 后插入列/行末 token。示例用 4 个高分辨率裁剪、每裁剪 36（）个 patch；实际 Molmo 常用 12 个高分辨率裁剪、每裁剪 144（）个 patch。
Our method of encoding images is shown in Figure 7. Cropping is done by first choosing a rectangular grid (e.g., a 22, 31, etc.) where each square of the grid matches the ViT’s input size. When using overlapping crops, these squares are moved closer together so that they overlap by a fixed margin (we use a margin of 4 patches or 56 pixels), which reduces the overall size of the grid.
图像编码方式见图 7。裁剪时先选一矩形网格（如 、 等），格中每格对齐 ViT 输入尺寸。使用重叠裁剪时，把这些方格拉近，按固定边距重叠（我们用 4 个 patch 即 56 像素），从而缩小网格总尺寸。
Then the image is up-scaled to fit within that grid as well as possible while preserving its aspect ratio by making either its height or width the same size as the grid. The grid is chosen to require the least amount of up-scaling, and in the event of ties, to minimize its size. We also set a maximum number of crops, and if the image cannot be covered by that many crops, the image will instead be down-scaled to fit the grid, and the grid is chosen to minimize the amount of down-scaling required while not exceeding the maximum number of crops. In either case, the re-scaled image is padded with black borders so that it exactly fits the grid, and then crops are extracted from this padded image. The low-resolution crop is built by resizing and padding the image so it matches image ViT’s supported resolution.
随后在保持宽高比前提下尽量放大图像以落入该网格：使高或宽之一与网格对齐。网格选取原则是上采样最少，平局时取更小网格。我们还设最大裁剪数；若该数盖不住图像，则改为缩小图像适配网格，并选使下采样最少且不超最大裁剪数的网格。两种情形下，缩放后的图像都用黑边填充以精确贴合网格，再从该填充图切裁剪。低分辨率裁剪则是把原图缩放并填充到 ViT 支持的分辨率。
Each crop is processed independently by the ViT and connector to get visual embeddings of each patch. A learned embedding is added to the patch features from each crop (before the connector is applied) depending on whether that patch includes no padding, some padding, or is all padding, so the model can distinguish padding from images that naturally have black borders. These embeddings are arranged with special tokens as described in Section 2, also shown in Figure 7 right. For image/text inputs we encode the input image first, followed by any text.
每个裁剪经 ViT 与连接器独立处理，得到各 patch 的视觉嵌入。在连接器之前，按该 patch「无填充 / 部分填充 / 全填充」加入可学习嵌入，以便区分填充与本身带黑边的图像。这些嵌入按第 2 节所述与特殊 token 排布，亦见图 7 右。图文输入时先编码图像，再接文本。
A.2 Hyper-Parameters
A.2 超参数
| 1B-E | 7B-D | 7B-O | 72B-D | ||
|---|---|---|---|---|---|
| Image Encoder | Params | 290m | |||
| Dim | 1024 | ||||
| MLP Dim | 4096 | ||||
| Act. | GELU | ||||
| Heads | 16 | ||||
| KV Heads | 16 | ||||
| Layers | 23 | ||||
| Image Size | 336336 | ||||
| Patch Size | 14 | ||||
| Dropout | 0.0 | ||||
| V/L Connector | Params | 12m | 110m | 74m | 310m |
| Pool Size | 22 | ||||
| Pool Dim | 1024 | ||||
| Pool Heads | 16 | ||||
| MLP Dim | 1024 | 37888 | 22016 | 59136 | |
| Act. | SwiGLU | ||||
| Dropout | 0.0 | ||||
| LLM | Params | 1.2b (6.9b) | 7.6b | 7.3b | 72b |
| Embed | 50304 | 152064 | 100352 | 152064 | |
| Dim | 2048 | 3584 | 4096 | 8192 | |
| MLP Dim | 204864 | 37888 | 22016 | 59136 | |
| Act. | SwiGLU | ||||
| Heads | 16 | 28 | 32 | 80 | |
| KV Heads | 16 | 4 | 32 | 8 | |
| Layers | 16 | 28 | 32 | 64 | |
| Theta | 10k | 1m | 0.5m | 1m | |
| Dropout | 0.1 | ||||
| Pre-Train | Warmup ViT | 2000 | |||
| Warmup Con. | 200 | ||||
| Warmup LLM | 2000 | ||||
| LR ViT | 6e-6 | ||||
| LR Con. | 2e-4 | ||||
| LR LLM | 2e-5 | 2e-5 | 2e-5 | 1e-5 | |
| Cosine Decay | 10% | ||||
| Eps. | 1e-6 | ||||
| Betas | 0.9, 0.95 | ||||
| Batch Size | 128 | ||||
| Steps | 22.3k | ||||
| Fine-Tune | Warmup ViT | 200 | |||
| Warmup Con. | 200 | ||||
| Warmup LLM | 200 | ||||
| LR ViT | 5e-6 | 5e-6 | 5e-6 | 3e-6 | |
| LR Con. | 5e-6 | 5e-6 | 5e-6 | 3e-6 | |
| LR LLM | 2e-5 | 1e-5 | 1e-5 | 5e-6 | |
| Cosine Decay | 10% | ||||
| Eps. | 1e-6 | ||||
| Betas | 0.9, 0.95 | ||||
| Batch Size | 256 | ||||
| Steps | 30k | 30k | 32k | 20k |
Table 6: Model and training hyper-parameters. Molmo-1B-E has 1.2b active parameters, but 6.9b total. Its LLM MLP layers have 64 experts with 8 active at once.
| 1B-E | 7B-D | 7B-O | 72B-D | ||
|---|---|---|---|---|---|
| 图像编码器 | 参数量 | 290m | |||
| Dim | 1024 | ||||
| MLP Dim | 4096 | ||||
| Act. | GELU | ||||
| Heads | 16 | ||||
| KV Heads | 16 | ||||
| Layers | 23 | ||||
| 图像尺寸 | 336336 | ||||
| Patch 尺寸 | 14 | ||||
| Dropout | 0.0 | ||||
| 视觉/语言连接器 | 参数量 | 12m | 110m | 74m | 310m |
| Pool Size | 22 | ||||
| Pool Dim | 1024 | ||||
| Pool Heads | 16 | ||||
| MLP Dim | 1024 | 37888 | 22016 | 59136 | |
| Act. | SwiGLU | ||||
| Dropout | 0.0 | ||||
| LLM | 参数量 | 1.2b (6.9b) | 7.6b | 7.3b | 72b |
| Embed | 50304 | 152064 | 100352 | 152064 | |
| Dim | 2048 | 3584 | 4096 | 8192 | |
| MLP Dim | 204864 | 37888 | 22016 | 59136 | |
| Act. | SwiGLU | ||||
| Heads | 16 | 28 | 32 | 80 | |
| KV Heads | 16 | 4 | 32 | 8 | |
| Layers | 16 | 28 | 32 | 64 | |
| Theta | 10k | 1m | 0.5m | 1m | |
| Dropout | 0.1 | ||||
| 预训练 | Warmup ViT | 2000 | |||
| Warmup Con. | 200 | ||||
| Warmup LLM | 2000 | ||||
| LR ViT | 6e-6 | ||||
| LR Con. | 2e-4 | ||||
| LR LLM | 2e-5 | 2e-5 | 2e-5 | 1e-5 | |
| Cosine Decay | 10% | ||||
| Eps. | 1e-6 | ||||
| Betas | 0.9, 0.95 | ||||
| Batch Size | 128 | ||||
| Steps | 22.3k | ||||
| 微调 | Warmup ViT | 200 | |||
| Warmup Con. | 200 | ||||
| Warmup LLM | 200 | ||||
| LR ViT | 5e-6 | 5e-6 | 5e-6 | 3e-6 | |
| LR Con. | 5e-6 | 5e-6 | 5e-6 | 3e-6 | |
| LR LLM | 2e-5 | 1e-5 | 1e-5 | 5e-6 | |
| Cosine Decay | 10% | ||||
| Eps. | 1e-6 | ||||
| Betas | 0.9, 0.95 | ||||
| Batch Size | 256 | ||||
| Steps | 30k | 30k | 32k | 20k |
表 6：模型与训练超参数。Molmo-1B-E 活跃参数 1.2b，总参数 6.9b；其 LLM MLP 有 64 个专家，每次激活 8 个。
Hyper-parameters for the Molmo models and the AdamW [50, 73] optimizers are shown in Table 6. The connector MLP uses the same intermediate dimension as the LLM, so its size depends on the LLM. The connector pooling layer and ViT architecture are the same between all models. All runs used a cosine learning rate schedule ending at 10% of the peak learning rate [72].
Molmo 各模型与 AdamW [50, 73] 优化器的超参数见表 6。连接器 MLP 中间维与 LLM 相同，故大小随 LLM 而变。连接器池化层与 ViT 结构在各模型间一致。所有运行均用余弦学习率，终点为峰值的 10% [72]。
Learning rates are similar between the models, except we find it helpful to reduce the learning rate for Molmo-72B. We also find Molmo-72B learns faster than the other models and can therefore be trained for fewer steps. Molmo-7B-O was trained for slightly longer due to a minor configuration difference, but we do not think it affected performance.
各模型学习率相近，但对 Molmo-72B 降低学习率更有益。72B 学得更快，因而可少训若干步。Molmo-7B-O 因配置小差异略多训了一点，我们认为对性能无实质影响。
Figure 8: Training loss curves for Molmo-7B-D with model weights and gradient reduction in bfloat16 (blue) and float32 (pink). Float32 is our default configuration.
图 8：Molmo-7B-D 在权重与梯度归约用 bfloat16（蓝）与 float32（粉）时的训练损失曲线。默认配置为 float32。
A.3 Implementation
A.3 实现
Our implementation uses PyTorch with Fully Sharded Data Parallel (FSDP) [135] based on the OLMo codebase [37]. We do not use FlashAttention [29, 28] since it does not support the more complex masks that are required for multi-annotated images, but we find using PyTorch’s Scaled Dot Product Attention (SDPA) achieves close to the same speed.
实现基于 OLMo 代码库 [37]，用 PyTorch 与 Fully Sharded Data Parallel（FSDP） [135]。因多标注图像需要更复杂的注意力掩码，我们未用 FlashAttention [29, 28]；改用 PyTorch 的 Scaled Dot Product Attention（SDPA），速度接近。
To improve throughput, we utilize PyTorch’s Automatic Mixed Precision (AMP) module[^3], which enables most operations to run in half-precision with bfloat16 numbers. However, as shown in Figure 8, keeping model weights and performing gradient reduction in half-precision degrades training loss, so these are retained in full precision. Additionally, computations for layer normalization [8] and Rotary Position Embedding (RoPE) [101] are explicitly carried out in full precision.
为提高吞吐，使用 PyTorch 自动混合精度（AMP）[^3]，使多数运算以 bfloat16 半精度运行。但如图 8，权重与梯度归约若也用半精度会恶化训练损失，故二者保持全精度。层归一化 [8] 与旋转位置编码（RoPE） [101] 的计算亦显式用全精度。
When computing gradients with FSDP, each GPU computes a gradient on a small mini-batch of examples, after which gradients are averaged across all devices. We always compute the per-device gradient by dividing the total loss on that device by the average number of loss-tokens across all devices, not the number of loss tokens on that particular device. This avoids a subtle bias that effectively up-weights examples with a small number of loss tokens (e.g., with short responses) since those examples tend to be paired with a smaller divisor if using the device-local number of loss tokens. Using the average number of loss tokens across all devices largely resolves this issue since our global batches are much larger than the device-local batches. This issue has been discussed in other places[^4] [40] and is known to have affected many codebases.[^5] We observe that captioning performance can drop by 0.5-1 points without this fix.
FSDP 求梯度时，各 GPU 在小 mini-batch 上算梯度再跨设备平均。设备上梯度一律用该设备总损失除以全部设备上 loss-token 的平均数，而非本设备 token 数。这避免微妙偏差：短回答等少 loss-token 样本若用本设备除数，会有效被上权。全局 batch 远大于设备 batch 时，用全局平均可基本消除该问题。文献与社区亦有讨论[^4] [40]，且影响过许多代码库。[^5] 我们观察到不做此修正时描述指标可掉 0.5–1 分。
During fine-tuning, mixing is done within each batch so batches contain examples from a variety of tasks. We set a maximum sequence length of 2304 for both pre-training and fine-tuning, and truncate examples longer than that (in practice, truncation only happens for certain synthetic datasets like DVQA [46] which contains many annotations per image, or for the occasional outlier example in other datasets).
微调时在每个 batch 内混合，使 batch 含多种任务。预训练与微调最大序列长均为 2304，超长则截断（实际主要见于 DVQA [46] 等每图标注极多的合成集，或其他集偶发离群样本）。
We find training to be stable, without loss spikes or NaNs, likely in part because we use pre-trained models.
训练稳定，无损失尖峰或 NaN，部分因使用了预训练模型。
Appendix B Training Details
附录 B 训练细节
Here we discuss the training mixture and how tasks are formatted during pre-training and fine-tuning.
本节讨论训练混合，以及预训练与微调时任务如何格式化。
B.1 Pre-Training Task Details
B.1 预训练任务细节
During pre-training, we train on each image paired with its caption and one of its audio transcripts. For images with multiple transcripts, we select one randomly each epoch. We use multi-annotation training (see Section C) to train on both the caption and the transcript jointly.
预训练时，每张图与其描述及一条音频转写配对训练。多转写时每 epoch 随机选一条。用多标注训练（见第 C 节）同时学描述与转写。
We prompt the model with either “long_caption:” or “transcript:” for captions and transcript respectively (a natural language prompt is used instead during instruction fine-tuning). We also add a length hint: an integer providing a noisy hint as to the correct output length. This hint is computed as the length of the transcript/caption in characters, plus a noise factor drawn from a random normal with a standard deviation of 25. The hint is then divided by 15 and rounded down to keep the hint in roughly the range of 0 to 100. This noise is added so that the length functions more like a guideline than a hard constraint, leaving the model some flexibility to adjust the caption as appropriate for the image. For example, even with a long length hint, its preferable that a caption for a very plain image be short instead of becoming repetitive or inane due to lack of content to describe.
提示分别为 “long_caption:” 或 “transcript:”（指令微调改用自然语言提示）。另加长度提示：一个带噪整数，暗示正确输出长度。算法为转写/描述的字符长度加标准差 25 的正态噪声，再除以 15 并向下取整，使提示大致落在 0–100。噪声意在使其更像指引而非硬约束，便于模型按图像内容调整长度——例如即便长度提示很长，极朴素的图仍宜短描述，以免因内容不足而重复或空洞。
We add the hint to the prompt 90% of the time, for example: “long_caption_83:” for a length hint of 83, and 10% of the time no length hint is used to maintain the ability to output a default caption.
90% 的提示带长度提示，如 “long_caption_83:” 表示提示 83；另 10% 不加，以保留输出默认描述的能力。
Figure 9: Captioning precision and recall with different length hints for Molmo-7B-D after pre-training. A short hint reduces recall since the model describes fewer things, but can boost accuracy since the description tends to focus on the more salient, easier-to-understand parts of the image.
图 9：Molmo-7B-D 预训练后，不同长度提示下的描述精确率与召回。短提示会降低召回（描述更少），但可能提高准确率，因描述更聚焦显著、易理解的区域。
Adjusting the length hint allows a trade off between precision and recall in captioning, see Figure 9 (see Section C for captioning metric details). In all of these settings, the average caption length when using a length hint is within 10 characters of the expected length, showing the models follow the length hints well. For our ablations, we report scores with a length hint of 65, which performs similarly or slightly better than using no length hint.
调节长度提示可在描述的精确率与召回间权衡，见图 9（指标细节见第 C 节）。各设置下，带长度提示时平均描述长度与期望相差在 10 字符内，说明模型能较好跟随。消融报告用长度提示 65，表现与不加提示相近或略好。
Preliminary experiments with mixing in other sources of captions (COCO Captions [17], Localized Narratives [93], or captions derived from Visual Genome annotations [52] ) did not improve scores on our captioning metric, so we use PixMo-Cap alone for pre-training.
初步尝试混入其他描述源（COCO Captions [17]、Localized Narratives [93]，或由 Visual Genome 标注派生 [52]）未能提升我们的描述指标，故预训练只用 PixMo-Cap。
B.2 Fine-Tuning Task Details
B.2 微调任务细节
| name | rate | images | anno. | tokens | avg. crops |
|---|---|---|---|---|---|
| PixMo (Annotated) | 38.1 | 1m | 3.3m | 350m | 10.6 |
| Points | 28.8 | 220k | 2.3m | 160m | 10.1 |
| AskModelAnything | 3.8 | 71k | 160k | 17m | 10.0 |
| Cap | 3.2 | 712k | 712k | 160m | 10.9 |
| PointQA | 2.4 | 14k | 76k | 11m | 11.0 |
| PixMo (Synthetic) | 31.6 | 1.3m | 3.3m | 120m | 11.0 |
| Count | 6.2 | 36k | 37k | 3m | 11.9 |
| CapQA | 5.7 | 160k | 210k | 38m | 10.8 |
| Clocks | 5.3 | 800k | 800k | 20m | 10.5 |
| Docs-Charts | 5.2 | 120k | 1.1m | 34m | 12.8 |
| Docs-Other | 4.0 | 71k | 610k | 15m | 12.8 |
| Docs-Tables | 3.3 | 47k | 420k | 12m | 12.1 |
| Docs-Diagrams | 1.9 | 16k | 140k | 3.6m | 12.5 |
| Academic | 30.3 | 880k | 25m | 1b | 8.0 |
| TallyQA | 3.9 | 130k | 250k | 4.6m | 6.1 |
| VQA v2.0 | 3.1 | 83k | 440k | 7.9m | 6.7 |
| AndroidControl | 2.9 | 74k | 300k | 13m | 11.0 |
| A-OKVQA | 2.8 | 17k | 17k | 380k | 6.8 |
| DocVQA | 2.1 | 10k | 39k | 1m | 12.9 |
| TextVQA | 2.0 | 22k | 35k | 700k | 12.7 |
| ChartQA | 1.8 | 18k | 28k | 850k | 9.3 |
| ST-VQA | 1.7 | 18k | 25k | 530k | 4.6 |
| InfographicVQA | 1.6 | 4.4k | 24k | 670k | 12.0 |
| TabWMP | 1.6 | 23k | 23k | 930k | 2.3 |
| PlotQA | 1.5 | 160k | 20m | 930m | 12.4 |
| AI2D | 1.3 | 6.2k | 15k | 630k | 6.4 |
| DVQA | 1.1 | 200k | 2.3m | 51m | 5.0 |
| FigureQA | 1.1 | 100k | 1.3m | 25m | 6.1 |
| OK-VQA | 1.0 | 9k | 9k | 180k | 6.8 |
| ScienceQA | 0.8 | 5k | 6.2k | 460k | 4.3 |
Table 7: Full list of instruction fine-tuning tasks. Columns show the sampling rate, the total number of images and annotations (i.e., the number of question/answer pairs), the number of text tokens using the Qwen2 tokenizer, and the average number of crops. The number of crops per an image can be at most 13 (one low-res, and 12 high-res), but can be lower for datasets with smaller images. Shaded rows show the total counts for all datasets in the category.
| 名称 | 采样率 | 图像 | 标注 | token | 平均裁剪 |
|---|---|---|---|---|---|
| PixMo（标注） | 38.1 | 1m | 3.3m | 350m | 10.6 |
| Points | 28.8 | 220k | 2.3m | 160m | 10.1 |
| AskModelAnything | 3.8 | 71k | 160k | 17m | 10.0 |
| Cap | 3.2 | 712k | 712k | 160m | 10.9 |
| PointQA | 2.4 | 14k | 76k | 11m | 11.0 |
| PixMo（合成） | 31.6 | 1.3m | 3.3m | 120m | 11.0 |
| Count | 6.2 | 36k | 37k | 3m | 11.9 |
| CapQA | 5.7 | 160k | 210k | 38m | 10.8 |
| Clocks | 5.3 | 800k | 800k | 20m | 10.5 |
| Docs-Charts | 5.2 | 120k | 1.1m | 34m | 12.8 |
| Docs-Other | 4.0 | 71k | 610k | 15m | 12.8 |
| Docs-Tables | 3.3 | 47k | 420k | 12m | 12.1 |
| Docs-Diagrams | 1.9 | 16k | 140k | 3.6m | 12.5 |
| 学术集 | 30.3 | 880k | 25m | 1b | 8.0 |
| TallyQA | 3.9 | 130k | 250k | 4.6m | 6.1 |
| VQA v2.0 | 3.1 | 83k | 440k | 7.9m | 6.7 |
| AndroidControl | 2.9 | 74k | 300k | 13m | 11.0 |
| A-OKVQA | 2.8 | 17k | 17k | 380k | 6.8 |
| DocVQA | 2.1 | 10k | 39k | 1m | 12.9 |
| TextVQA | 2.0 | 22k | 35k | 700k | 12.7 |
| ChartQA | 1.8 | 18k | 28k | 850k | 9.3 |
| ST-VQA | 1.7 | 18k | 25k | 530k | 4.6 |
| InfographicVQA | 1.6 | 4.4k | 24k | 670k | 12.0 |
| TabWMP | 1.6 | 23k | 23k | 930k | 2.3 |
| PlotQA | 1.5 | 160k | 20m | 930m | 12.4 |
| AI2D | 1.3 | 6.2k | 15k | 630k | 6.4 |
| DVQA | 1.1 | 200k | 2.3m | 51m | 5.0 |
| FigureQA | 1.1 | 100k | 1.3m | 25m | 6.1 |
| OK-VQA | 1.0 | 9k | 9k | 180k | 6.8 |
| ScienceQA | 0.8 | 5k | 6.2k | 460k | 4.3 |
表 7：指令微调任务全表。列依次为采样率、图像与标注总数（即问答对数）、用 Qwen2 分词器计的文本 token 数、平均裁剪数。每图裁剪最多 13（1 低分辨率 + 12 高分辨率），小图数据集可更少。阴影行为该类别合计。
Table 7 shows a list of our fine-tuning tasks. We only train on the train sets. Formatting and task-specific details are described below.
表 7 列出微调任务。我们只在训练集上训练。格式与任务细节如下。
Multiple choice questions.
For multiple choice questions in academic datasets (AI2D, A-OKVQA, ScienceQA), we append “Choices:”, a newline, and then new-line separated options with capital letter answer labels. The model predicts the answer label only. Note some multiple-choice questions appear in other, more diverse formats in PixMo-CapQA and PixMo-AskModelAnything.
选择题。
学术集中的选择题（AI2D、A-OKVQA、ScienceQA）后接 “Choices:”、换行，再换行分隔的大写字母选项。模型只预测选项字母。注意 PixMo-CapQA 与 PixMo-AskModelAnything 中也有更多样格式的选择题。
Multiple answers.
For datasets with multiple answers per question (e.g., VQA v2.0), we only use the most common answer for training. If there are multiple answers that are equally common, we randomly select from among them each epoch.
多答案。
对每题多答案的数据集（如 VQA v2.0），训练只用最常见答案；并列最常见则每 epoch 随机选一。
Pointing.
Pointing uses an HTML-like format. (x,y) coordinates are scaled to 0-100. For a single point, the format is: Inline text For multiple points the format is: <points x1="10.0" y1="10.0" x2="20.0" y1="20.0" ... alt="alt text">Inline text Numbering the points makes counting easier because the total count is always the number of the last point.
指点。
指点用类 HTML 格式。(x,y) 缩放到 0–100。单点形如：内联文本；多点形如：<points x1="10.0" y1="10.0" x2="20.0" y1="20.0" ... alt="alt text">内联文本。给点编号便于计数，因为总数就是最后一个点的编号。
When interacting with users, we generally replace the point(s) text with the inline text, and show the image with the points using the alt text as hover text. For pointing and counting, the inline and alt text are both the name of what is being pointed at. For pointing-as-an-explanation these fields can be different.
与用户交互时，一般用内联文本替换点标记，并在图上显示点，悬停显示 alt。指点与计数时，内联与 alt 都是所指对象名；点作解释时可不同。
PixMo-Points.
Counting or pointing with a very large number of objects can lead to very long sequence lengths. To avoid memory errors we do not train on data with more than 40 counts; we expect to remove this limitation in future iterations of Molmo.
PixMo-Points。
对极多目标计数或指点会使序列很长。为免显存错误，我们不训练超过 40 个计数的数据；预期后续版本去掉这一限制。
PixMo-AskModelAnything.
“How many” questions are common in PixMo-AskModelAnything, but are not accompanied with pointing data. We observe that this can lead to the model failing to point when asked counting questions. To resolve this, we heuristically detect such questions and prefix them with an instruction to not point (e.g., “Answer without points.”), randomly selected from a pool of 20 such instructions.
PixMo-AskModelAnything。
其中常见 “How many” 类问题但不带指点。我们观察到这会导致计数题时模型不去指点。于是启发式识别此类问题，并从前缀池（约 20 条）随机选一条「勿指点」指令（如 “Answer without points.”）加在前面。
AI2D.
AI2 Diagrams requires labeling regions of the images with letters, and then training the model to answer questions by predicting the correct region by returning its letter. Evaluations in the literature have been mixed between labeling the regions with opaque boxes (e.g., [5, 89]) and transparent boxes (e.g., [106, 71, 10]). We train in both settings and present our main results with transparent boxes. Results with opaque boxes are in Section D.
AI2D。
AI2 Diagrams 需用字母标出图中区域，再训练模型用字母回答指向正确区域。文献评测混用了不透明框（如 [5, 89]）与透明框（如 [106, 71, 10]）。两种设定我们都训，主结果用透明框；不透明框结果见第 D 节。
For AI2D questions where the answers are just letters, we list the multiple-choice options without a letter prefix.
对答案本身只是字母的 AI2D 题，列出多选选项时不加字母前缀。
AI2D does not have a validation set, so we built our own custom validation set by separating out 384 images (with roughly 2000 questions-answer pairs). None of our models are trained on this set.
AI2D 无官方验证集，我们自建：分出 384 张图（约 2000 问答对）。任何模型都未在其上训练。
ChartQA.
The ChartQA train set contains many synthetic questions (21k synthetic vs. 7k non-synthetic), which we observe can be noisy and lower quality. To reduce the weight of these examples we re-weight ChartQA so the total weight of the synthetic and non-synthetic examples are equal. This also means the training data better matches the validation and test data which are evenly split between synthetic and non-synthetic questions.
ChartQA。
训练集含大量合成题（约 2.1 万合成 vs 0.7 万非合成），我们观察其噪声大、质量偏低。于是重加权使合成与非合成总权重相等，也更贴近验证/测试上各半的构成。
A-OKVQA.
We train on the multiple choice questions and, for questions not marked as difficult direct answer, also use them as direct answer questions by not using the answer options. We use different style tags for direct answer and multiple choice versions of the questions.
A-OKVQA。
训练多选题；对未标为困难直接回答的题，也去掉选项当直接回答题用。直接回答与多选版本使用不同风格标签。
TabWMP.
For TabWMP we treat the task as short answer and do not show the multiple-choice options.
TabWMP。
当作短答任务，不展示多选选项。
AndroidControl.
We train on four input-output configurations: low-level instruction to action, high-level goal to action, low-level and high-level inputs to action, and high-level goal to action with chain-of-thought reasoning. Only the instruction and screenshot are provided to the model as input; accessibility trees, action history, and a prompt with details like available actions are omitted. Target actions are represented as text output strings and (x,y) coordinates are scaled to 0-100 just as with regular pointing.
AndroidControl。
训练四种输入–输出配置：低层指令→动作、高层目标→动作、低层+高层→动作，以及带思维链的高层目标→动作。输入只给指令与截图；无障碍树、动作历史，也不给可用动作清单类提示。目标动作表示为文本串，(x,y) 缩放到 0–100。
B.3 Training Time
B.3 训练时间
| pre-train | fine-tune | ||||||
|---|---|---|---|---|---|---|---|
| GPUs | time | GPU hr. | GPUs | time | GPU hr. | ||
| 1B-E | 8 | 33.3 | 264 | 64 | 13.3 | 850 | |
| 7B-D | 64 | 8.6 | 550 | 128 | 11.2 | 1.4k | |
| 7B-O | 64 | 8.9 | 570 | 128 | 13.5 | 1.7k | |
| 72B | 128 | 33.3 | 4.2k | 256 | 32.4 | 8.3k |
Table 8: Training times for the Molmo models using H100 GPUs.
| pre-train | fine-tune | ||||||
|---|---|---|---|---|---|---|---|
| GPUs | time | GPU hr. | GPUs | time | GPU hr. | ||
| 1B-E | 8 | 33.3 | 264 | 64 | 13.3 | 850 | |
| 7B-D | 64 | 8.6 | 550 | 128 | 11.2 | 1.4k | |
| 7B-O | 64 | 8.9 | 570 | 128 | 13.5 | 1.7k | |
| 72B | 128 | 33.3 | 4.2k | 256 | 32.4 | 8.3k |
表 8：Molmo 各模型在 H100 GPU 上的训练时间。
Training time and number of GPUs used are shown in Table 8. All models were trained with H100 GPUs with infiniband connectivity.
训练时间与 GPU 数见表 8。全部使用带 InfiniBand 互联的 H100。
Appendix C Evaluation Details
附录 C 评测细节
Captioning metric (cap ).
We measure captioning quality, relative to an evaluation set of 1500[^6] images, using the harmonic mean of captioning precision and recall, i.e. the score. The evaluation set was gathered through a similar protocol as PixMo-Cap (selecting a small number of images matching a diverse set of categories), but the images were selected manually and are disjoint from images in PixMo-Cap. Each evaluation image has up to six audio transcripts associated with it. To define the precision and recall of a caption for an image, let be the generated caption and be the set of ground-truth transcripts for the image. We prompt GPT-4o to enumerate a list of all distinct atomic statements contained in and, separately, the transcripts in . We then prompt GPT-4o to match each item in the list of atomic statements from to items in the list of atomic statements from . To compute recall, we consider matches as true positives and unmatched items from ’s list as false negatives. To compute precision, we prompt GPT-4o with the raw transcripts and the list of statements from and ask it to say if each statement is consistent (a true positive) or inconsistent (a false positive) with the transcripts. (We avoid using the atomic statements from when computing precision because it’s a potentially noisy processing step that is not necessary.) We average precision and recall over all images in the evaluation set and compute the score of the averaged precision and recall values to produce our final summary metric: cap .
描述指标（cap ）。
我们相对由 1500[^6] 张图组成的评测集，用描述精确率与召回的调和平均（即 ）衡量描述质量。评测集采集协议类似 PixMo-Cap（按多样类别选少量图），但图像人工挑选且与 PixMo-Cap 无交。每张评测图最多关联六条音频转写。设 为生成描述， 为该图真值转写集合。我们提示 GPT-4o 分别枚举 与 中所有互不相同的原子陈述；再提示其把 的原子陈述与 的原子陈述做匹配。算召回时，匹配计为真正例， 列表中未匹配项计为假负例。算精确率时，把原始转写与 的陈述列表交给 GPT-4o，判断每条陈述与转写是否一致（真正例）或不一致（假正例）。（精确率不用 的原子陈述，因那一步可能嘈杂且非必需。）在评测集上对精确率与召回取平均，再对平均精确率与平均召回算 ，得到最终汇总指标 cap 。
While this metric is imperfect (e.g., GPT-4o makes mistakes, the transcripts do not contain all true statements about the image, etc.) we found that improvements to cap corresponded to improvements in our subjective impressions of caption quality, and thus it was a useful internal metric for guiding model and data design. Most of our model design and pre-training data decisions were based on improving captioning quality, see Section D and Figure 11 for more discussion.
该指标并不完美（如 GPT-4o 会出错，转写也未穷尽图中全部真实陈述等），但我们发现 cap 的提升与主观描述质量改善一致，因而成为指导模型与数据设计的有用内部指标。多数模型设计与预训练数据决策都围绕提升描述质量，更多讨论见第 D 节与图 11。
Human evaluation.
We defined 10 question categories and crowd sourced image-question pairs, using the same workers as for other annotation tasks. This resulted in the following categories and image-question pair counts:
人工评测。
我们定义 10 类问题，并由与其他标注任务相同的众包工人采集图–问对，各类数量如下：
- • Output formatting: 1525
- • 输出格式：1525
- • Fine-grained detailed QA: 1510
- • 细粒度细节问答：1510
- • General: 1504
- • 综合：1504
- • Documents: 1499
- • 文档：1499
- • Captioning: 1493
- • 描述：1493
- • Counting: 1490
- • 计数：1490
- • Homework: 1489
- • 作业：1489
- • Charts: 1473
- • 图表：1473
- • Named entities: 1448
- • 命名实体：1448
- • Creativity: 1420
- • 创意：1420
We performed two human evaluations, the first for a large set of models presented in our main results Table 1 and a second for a smaller set of ablation models in Table 5. To collect feedback, we presented an annotator with an image, a question, and the output of “model A” and “model B”, without revealing the model identities. The annotator had five options: tie (both bad), tie (both good), model A is better, model B is better, or I don’t know. The last option was for cases where the annotator did not know the correct answer (e.g., a math problem they do not know how to solve). For the first study, we sampled image-question pairs randomly from the 10 categories, until we had collected 450 feedback responses per model pair. For the second study, we used a refined methodology in which we first manually verified the quality of the question-image pairs, resulting in a fixed set of 500 questions (exactly 50 per category), all of which were used for each pair of models. After collecting the feedback, we removed the I don’t know responses and computed an Elo ranking using the Bradley-Terry model, following the methodology of Chatbot Arena [21].
我们做了两次人工评测：第一次覆盖主文表 1 中的大批模型，第二次覆盖表 5 中较小消融集。采集反馈时，向标注者展示图像、问题以及「模型 A」「模型 B」的输出（不透露身份）。可选五项：平局（都差）、平局（都好）、A 更好、B 更好，或我不知道。最后一项用于标注者不懂正确答案的情形（如不会做的数学题）。第一次研究从 10 类中随机抽样图–问，直至每对模型约 450 条反馈；第二次先人工核验图–问质量，固定 500 题（每类恰 50），每对模型都用同一套。去掉「我不知道」后，用 Bradley-Terry 按 Chatbot Arena [21] 方法算 Elo。
AndroidControl.
To evaluate Molmo on AndroidControl, we provided only the task instruction (high-level or low-level) and the current screenshot as input. The results are reported on the in-domain data test set (IDD) and the metric is step-wise accuracy.
AndroidControl。
评测 Molmo 时，输入仅给任务指令（高层或低层）与当前截图。结果报告于域内测试集（IDD），指标为逐步准确率。
Appendix D Result Details
附录 D 结果细节
Chatbot Arena.
In Table 9 we report a summarized version of the vision leaderboard for queries in English from the Chatbot Arena [21], an independent third-party VLM evaluation. Molmo-72B outperforms all the fully open and open weight models but lags behind some of the propriety VLMs. As noted in the paper, we did our own Elo evaluation (see Section 5), in which Molmo-72B ranks higher (2nd place). The difference in rating likely stems from the types of questions evaluated. We do note our data includes many counting and image-description questions which are particular strengths of Molmo.
Chatbot Arena。
表 9 汇总 Chatbot Arena [21] 英文视觉排行榜（独立第三方 VLM 评测）。Molmo-72B 超过全部全开放与开源权重模型，但仍落后于部分闭源 VLM。如正文所述，我们自做的 Elo（见第 5 节）中 Molmo-72B 排名更高（第 2）。评分差异很可能来自题目类型。我们注意到自有数据含大量计数与图像描述题，而这正是 Molmo 的强项。
| model | score | 95% CI | opennness |
|---|---|---|---|
| Gemini-Exp-114 [103] | 1278 | +28/-27 | API only |
| ChatGPT-4o-latest (20240903) [90] | 1256 | +13/-13 | API only |
| Gemini-1.5-Pro-002 [103] | 1220 | +15/-14 | API only |
| Gemini-1.5-Flash-002 [103] | 1219 | +15/-17 | API only |
| GPT-4o-2024-05-13 [90] | 1213 | +9/-9 | API only |
| Claude 3.5 Sonnet (20240620) [7] | 1187 | +9/-7 | API only |
| Claude 3.5 Sonnet (20241022) [7] | 1184 | +15/-15 | API only |
| Gemini-1.5-Pro-001 [103] | 1158 | +9/-8 | API only |
| GPT-4-Turbo-2024-04-09 [88] | 1157 | +7/-10 | API only |
| Gemini-1.5-Flash-8B-Exp-0827 [103] | 1137 | +15/-13 | API only |
| GPT-4o-2024-08-06 [90] | 1131 | +18/-20 | API only |
| Gemini-1.5-Flash-8B-001 [103] | 1133 | +10/-15 | API only |
| GPT-4o-mini-2024-07-18 [89] | 1124 | +7/-9 | API only |
| Molmo-72B | 1115 | +18/-17 | Fully Open |
| Qwen2-VL-72B [111] | 1113 | +15/-17 | Open Weight |
| InternVL2-26B [104] | 1096 | +11/-10 | Open Weight |
| Pixtral-12B-2409 [3] | 1085 | +13/-14 | Open Weight |
| Llama-3.2V-90B-Instruct [5] | 1085 | +12/-14 | Open Weight |
| Gemini-1.5-Flash-001 [103] | 1087 | +8/-8 | API only |
| Molmo-7B-D | 1076 | +15/-18 | Fully Open |
| Yi-Vision [4] | 1070 | +21/-26 | Distilled |
| Claude 3 Opus [7] | 1073 | +6/-8 | API only |
| Qwen2-VL-7B [111] | 1068 | +15/-14 | Open Weight |
| Llama-3.2V-11B-Instruct [5] | 1061 | +14/-14 | Open Weight |
Table 9: Chatbot Arena’s vision leaderboard for English queries. The table is up to date as of Nov. 13, 2024. We show up to 20 rows for clarity.
| 模型 | 分数 | 95% CI | 开放度 |
|---|---|---|---|
| Gemini-Exp-114 [103] | 1278 | +28/-27 | 仅 API |
| ChatGPT-4o-latest (20240903) [90] | 1256 | +13/-13 | 仅 API |
| Gemini-1.5-Pro-002 [103] | 1220 | +15/-14 | 仅 API |
| Gemini-1.5-Flash-002 [103] | 1219 | +15/-17 | 仅 API |
| GPT-4o-2024-05-13 [90] | 1213 | +9/-9 | 仅 API |
| Claude 3.5 Sonnet (20240620) [7] | 1187 | +9/-7 | 仅 API |
| Claude 3.5 Sonnet (20241022) [7] | 1184 | +15/-15 | 仅 API |
| Gemini-1.5-Pro-001 [103] | 1158 | +9/-8 | 仅 API |
| GPT-4-Turbo-2024-04-09 [88] | 1157 | +7/-10 | 仅 API |
| Gemini-1.5-Flash-8B-Exp-0827 [103] | 1137 | +15/-13 | 仅 API |
| GPT-4o-2024-08-06 [90] | 1131 | +18/-20 | 仅 API |
| Gemini-1.5-Flash-8B-001 [103] | 1133 | +10/-15 | 仅 API |
| GPT-4o-mini-2024-07-18 [89] | 1124 | +7/-9 | 仅 API |
| Molmo-72B | 1115 | +18/-17 | 全开放 |
| Qwen2-VL-72B [111] | 1113 | +15/-17 | 开放权重 |
| InternVL2-26B [104] | 1096 | +11/-10 | 开放权重 |
| Pixtral-12B-2409 [3] | 1085 | +13/-14 | 开放权重 |
| Llama-3.2V-90B-Instruct [5] | 1085 | +12/-14 | 开放权重 |
| Gemini-1.5-Flash-001 [103] | 1087 | +8/-8 | 仅 API |
| Molmo-7B-D | 1076 | +15/-18 | 全开放 |
| Yi-Vision [4] | 1070 | +21/-26 | 蒸馏 |
| Claude 3 Opus [7] | 1073 | +6/-8 | 仅 API |
| Qwen2-VL-7B [111] | 1068 | +15/-14 | 开放权重 |
| Llama-3.2V-11B-Instruct [5] | 1061 | +14/-14 | 开放权重 |
表 9：Chatbot Arena 英文查询视觉排行榜。数据截至 2024-11-13。为清晰最多展示 20 行。
Clock reading.
PixMo-Clocks is a novel source of clock reading data, a data type that is missing from most VLM training data (for which data information is published; we cannot know for models without published data details, such as API-only models and many open-weight models such as Pixtral and Llama 3). PixMo-Clocks are entirely synthetic and show a variety of watch bodies and faces against plain backgrounds (see Figure 19 for examples).
读钟。
PixMo-Clocks 是新的读钟数据源；多数公开了数据信息的 VLM 训练集缺少此类（对未公布数据细节的模型——仅 API 与许多开源权重如 Pixtral、Llama 3——无从得知）。PixMo-Clocks 全合成，表体与表盘多样、背景朴素（样例见图 19）。
We tested how well Molmo trained on this data performs on the in-the-wild clock reading benchmark introduced in [121]. The benchmark sources clock images from three different datasets, COCO [65], OpenImages [53] and Clock Movies, a newly collected dataset based on the film The Clock (2010).[^7] They are highly out-of-distribution relative to the PixMo-Clocks training data. We also benchmarked several API-only models and open-weight (+ open-data) models for comparison. We compare all of these VLMs against the model presented from [121] that is specialized at the single task of clock reading.
我们在 [121] 提出的野外读钟基准上测试用该数据训练的 Molmo。基准时钟图来自 COCO [65]、OpenImages [53] 以及基于电影 The Clock（2010）新采的 Clock Movies。[^7] 相对 PixMo-Clocks 训练数据高度分布外。我们还评测了若干仅 API 与开源权重（+开放数据）模型，并与 [121] 的专用非 VLM 读钟模型对照。
We used the same query for all the VLMs: ‘‘What time is being shown? Please respond only with the time as hours and minutes in HH:MM format.’’, and followed the official evaluation protocol.[^8] Table 10 highlights that all the VLMs including proprietary models struggle to read clocks, with the exception of Molmo; see the notable performance gaps between Molmo and other VLMs. Molmo-72B surprisingly underperforms Molmo-7B-D and MolmoE-1B. This might be partially due to that PixMo-Clocks accounts only for 5.3% of the fine-tuning data mixture and we trained Molmo-72B for fewer steps than the others. Augmenting PixMo-Clocks with real-world clock images could potentially increase performance, closing the gap between Molmo and the specialized clock reading model.
所有 VLM 使用同一查询：「现在显示的是几点？请仅用 HH:MM 格式的小时与分钟作答。」并遵循官方评测协议。[^8] 表 10 显示，除 Molmo 外，含闭源在内的各 VLM 读钟都很吃力；Molmo 与其他 VLM 差距显著。Molmo-72B 意外弱于 Molmo-7B-D 与 MolmoE-1B，可能部分因为 PixMo-Clocks 仅占微调混合的 5.3%，且 72B 训练步数更少。若用真实世界时钟图增强 PixMo-Clocks，或可再抬表现，缩小与专用读钟模型的差距。
Despite training on synthetic data, we qualitatively observe the clocking-reading capabilities generalize effectively to more complex questions and to captioning. An example is in Figure 1 lower right of the main paper.
尽管训在合成数据上，我们定性观察到读钟能力能泛化到更复杂问题与描述。主文图 1 右下有一例。
| model | acc. | hour acc. | min. acc. |
|---|---|---|---|
| GPT-4o-0513 [90] | 2.7 | 14.2 | 8.6 |
| Gemini 1.5 Pro [103] | 0.9 | 11.6 | 5.1 |
| Claude-3.5 Sonnet [7] | 6.6 | 22.3 | 17.5 |
| PaliGemma-mix-3B [10] | 6.1 | 21.0 | 15.8 |
| Phi3.5-Vision-4B [1] | 1.9 | 12.0 | 7.6 |
| Qwen2-VL-72B [111] | 9.1 | 24.9 | 18.4 |
| InternVL2-Llama-3-76B [104] | 3.3 | 16.3 | 9.9 |
| Pixtral-12B [3] | 1.7 | 11.9 | 6.7 |
| Llama-3.2V-90B-Instruct [5] | 3.4 | 17.9 | 10.1 |
| LLaVA-1.5-13B [69] | 0.8 | 11.6 | 5.7 |
| xGen-MM-interleave-4B [119] | 2.0 | 11.9 | 8.0 |
| Cambrian-1-34B [106] | 1.8 | 11.1 | 7.2 |
| LLaVA OneVision-72B [59] | 5.7 | 17.9 | 15.4 |
| MolmoE-1B | 65.8 | 77.9 | 74.1 |
| Molmo-7B-O | 64.2 | 76.3 | 73.8 |
| Molmo-7B-D | 68.2 | 78.6 | 76.0 |
| Molmo-72B | 65.6 | 77.1 | 73.7 |
| Specialized single-task model [121] | 78.9 | 84.2 | 82.9 |
Table 10: Clock reading benchmark results. We report the averages of overall, hour and minute accuracies, each evaluated on three different test sets based on COCO, OpenImages and Clock Movies, respectively. Bold numbers represent the highest VLM scores while the best numbers, excluding Molmo, are underlined. We categorize models into five groups: (first) API-only, (second) open-weight, (third) open-weight and open-data, (four) the Molmo family and (five) the specialized clock reading model.
| 模型 | 准确率 | 小时准确率 | 分钟准确率 |
|---|---|---|---|
| GPT-4o-0513 [90] | 2.7 | 14.2 | 8.6 |
| Gemini 1.5 Pro [103] | 0.9 | 11.6 | 5.1 |
| Claude-3.5 Sonnet [7] | 6.6 | 22.3 | 17.5 |
| PaliGemma-mix-3B [10] | 6.1 | 21.0 | 15.8 |
| Phi3.5-Vision-4B [1] | 1.9 | 12.0 | 7.6 |
| Qwen2-VL-72B [111] | 9.1 | 24.9 | 18.4 |
| InternVL2-Llama-3-76B [104] | 3.3 | 16.3 | 9.9 |
| Pixtral-12B [3] | 1.7 | 11.9 | 6.7 |
| Llama-3.2V-90B-Instruct [5] | 3.4 | 17.9 | 10.1 |
| LLaVA-1.5-13B [69] | 0.8 | 11.6 | 5.7 |
| xGen-MM-interleave-4B [119] | 2.0 | 11.9 | 8.0 |
| Cambrian-1-34B [106] | 1.8 | 11.1 | 7.2 |
| LLaVA OneVision-72B [59] | 5.7 | 17.9 | 15.4 |
| MolmoE-1B | 65.8 | 77.9 | 74.1 |
| Molmo-7B-O | 64.2 | 76.3 | 73.8 |
| Molmo-7B-D | 68.2 | 78.6 | 76.0 |
| Molmo-72B | 65.6 | 77.1 | 73.7 |
| Specialized single-task model [121] | 78.9 | 84.2 | 82.9 |
表 10：野外读钟基准结果。报告总体准确率以及小时、分钟准确率；分别在基于 COCO、OpenImages 与 Clock Movies 的三个测试集上评测。粗体为最高 VLM 分数，下划线为除 Molmo 外最佳。模型分五组：（一）仅 API；（二）开放权重；（三）开放权重+开放数据；（四）Molmo 系列；（五）专用读钟模型。
Pointing.
To evaluate the model’s pointing performance, we constructed an evaluation set of 493 image and pointing question pairs. Each example was manually verified to ensure that either there is no target object or each target object instance is annotated with a single point and an accurate segmentation mask. The segmentation masks are generated by SAM [51] using each ground-truth point as a prompt.
指点。
为评测指点，我们构建 493 组图–指点问题。每例经人工核验：要么无目标，要么每个目标实例标有单点与准确分割掩码。掩码由 SAM [51] 以真值点为提示生成。
For cases with no target object, precision and recall are calculated as 1 if the model responds correctly (e.g. outputs “This isn’t in the image.”) and 0 otherwise. When a target object is present, we first compute the pairwise distances between the predicted points and the ground-truth points to serve as the cost in the Jonker-Volgenant algorithm [45, 25], which we then use to assign each predicted point to one of the ground-truth points. We then use the verified segmentation masks to determine if each predicted point with an assignment is a true positive or false positive. Specifically, we calculate precision as the fraction of predicted points located within the segmentation mask of their assigned ground-truth point, and recall as the fraction of segmentation masks covered by predicted points.
无目标时：模型答对（如输出 “This isn’t in the image.”）则精确率与召回为 1，否则为 0。有目标时：先算预测点与真值点的两两距离，作为 Jonker-Volgenant 算法 [45, 25] 的代价，把每个预测点匹配到一个真值点；再用核验过的分割掩码判定每个已匹配预测点是真正例还是假正例。具体地，精确率为落在其匹配真值点分割掩码内的预测点比例，召回为被预测点覆盖到的分割掩码比例。
Table 11 demonstrates Molmo’s superior pointing capability. Similar to captioning and counting, pointing performance declines when the number of crops are unequal at training and test time.
表 11 显示 Molmo 指点能力领先。与描述、计数类似，训练与测试裁剪数不一致时指点会下降。
| model | precision | recall | |
|---|---|---|---|
| MolmoE-1B | 73.0 | 72.9 | 72.2 |
| Molmo-7B-O | 75.7 | 75.5 | 75.1 |
| Molmo-7B-D | 75.0 | 74.6 | 74.3 |
| Molmo-7B-D (36 crops) | 58.4 | 58.7 | 58.1 |
| Molmo-72B | 75.8 | 75.4 | 75.2 |
Table 11: Pointing evaluation results. Pointing can perform poorly when of crops are unequal at training and test time.
| 模型 | 精确率 | 召回 | |
|---|---|---|---|
| MolmoE-1B | 73.0 | 72.9 | 72.2 |
| Molmo-7B-O | 75.7 | 75.5 | 75.1 |
| Molmo-7B-D | 75.0 | 74.6 | 74.3 |
| Molmo-7B-D (36 crops) | 58.4 | 58.7 | 58.1 |
| Molmo-72B | 75.8 | 75.4 | 75.2 |
表 11：指点评测结果。训练与测试裁剪数不一致时，指点可能明显变差。
| # crops train, test | CountBenchQA | Pixmo-Count | 11-avg |
|---|---|---|---|
| val | |||
| 12, 36∗ | 87.7 | 73.9 | 75.8 |
| 12, 36 | 88.5 | 85.2 | 76.9 |
| 36, 36 | 88.9 | 87.4 | 77.2 |
| 12 → 36, 36 | 88.9 | 87.4 | 77.2 |
Table 12: High-resolution fine-tuning results. Result on counting datasets (CountBenchQA and PixMo-Count val set) and the overall average (11-avg) using different numbers of crops at train and test. Note: 12, 36∗ uses a higher number of crops (36 crops) at test time for counting datasets, which leads to a much worse accuracy. Our default setting (highlighted in gray) uses the same number of crops (12 crops) during training and inference for counting datasets. We experiment with fine-tuning the 12-crop model at higher resolution and evaluating with 36 crops (12 36, 36).
| 训练/测试裁剪数 | CountBenchQA | Pixmo-Count | 11-avg |
|---|---|---|---|
| val | |||
| 12, 36∗ | 87.7 | 73.9 | 75.8 |
| 12, 36 | 88.5 | 85.2 | 76.9 |
| 36, 36 | 88.9 | 87.4 | 77.2 |
| 12 → 36, 36 | 88.9 | 87.4 | 77.2 |
表 12：高分辨率微调结果。不同训练/测试裁剪数下，计数集（CountBenchQA 与 PixMo-Count 验证集）与总体平均（11-avg）。注：12, 36∗ 表示测试用更多裁剪（36）但未做高分辨率微调。
High-resolution fine-tuning.
In Table of the main paper, we showed that training the model with higher resolution (i.e., more image crops) yields slight improvements on the 11-avg metric (from 76.9 to 77.2 when increasing the number of crops used in training from 12 to 36). Rather than directly training at a higher resolution, we explore fine-tuning the model initially trained with 12 crops using a higher resolution. Specifically, we continue training the 12-crop model for 3000 additional steps (10% of the fine-tuning steps) with 36 crops, roughly halving the learning rates of the vision encoder (lr=2e-6), connector (lr=2e-6), and language model (lr=5e-6). We keep the global batch size at 256 and use a warmup of 200 steps for all modules.
高分辨率微调。
主文表中显示，用更高分辨率（即更多图像裁剪）训练可略提 11-avg（训练裁剪从 12 增到 36 时，76.9→77.2）。我们未直接从高分辨率开训，而是探索：先以 12 裁剪训好的模型，再在更高分辨率上继续微调。具体地，对 12 裁剪模型再训 3000 步（约微调总步数的 10%），裁剪数升至 36，并把视觉编码器、连接器、语言模型的学习率大致减半（分别为 2e-6、2e-6、5e-6）。全局 batch 仍为 256，各模块预热 200 步。
Table 12 presents the results. Note that simply increasing the number of crops at inference time (first row) leads to degraded performance on counting tasks (88.5 → 87.7 for CountBenchQA and 85.2 → 73.9 for PixMo-Count). This suggests that a mismatch between training and testing resolutions adversely affects counting performance. As a result, our default model (second row) uses the same number of crops (12 crops) for counting datasets.
表 12 给出结果。仅在推理时增加裁剪（第一行）会损害计数（CountBenchQA 88.5→87.7，PixMo-Count 85.2→73.9），说明训练与测试分辨率不匹配会伤计数。因此默认模型（第二行）对计数集仍用相同裁剪数（12）。
After fine-tuning the model with higher resolution (fourth row), we observe that its counting performance can recover when evaluated with 36 crops, matching that of the model trained directly with 36 crops (third row), without sacrificing the overall 11-avg performance. This demonstrates that a brief period of high-resolution fine-tuning can effectively restore counting capabilities without affecting the average performance.
经更高分辨率微调后（第四行），在 36 裁剪评测时计数可恢复到与直接用 36 裁剪训练的模型（第三行）相当，且不牺牲总体 11-avg。说明短暂高分辨率微调能有效恢复计数能力，而不影响平均表现。
Text-only benchmarks.
PixMo consists exclusively of multimodal image-text data, without any text-only data. To investigate the potential impact of training solely on multimodal data on performance in text-only tasks, we report the results on common text benchmarks which assess a wide range of capabilities. We carefully follow the setup used by Llama 3 [5] for each task, ensuring that we can reproduce their numbers within the reported confidence intervals. As shown in Table 13, the Qwen2 language model employed in Molmo-7B-D appears to lose some knowledge across various tasks as a result of multimodal fine-tuning.
纯文本基准。
PixMo 全是多模态图文数据，不含纯文本。为考察只训多模态是否冲击纯文本任务，我们报告覆盖多种能力的常用文本基准。各任务严格对齐 Llama 3 [5] 设定，并在其报告置信区间内复现其分数。如表 13，Molmo-7B-D 所用的 Qwen2 语言模型在多模态微调后，多项任务上似乎损失了部分知识。
We run a small experiment at the 7B scale to test whether adding text-only data from Tulu 3[^9] [113] to our fine-tuning data mixture can address this issue. We use two different ratios: the entire dataset and a version with 10% down-sampling. Incorporating this text data enhances model performance on text-only tasks, particularly those involving mathematical reasoning and programming. Interestingly, down-sampling to 10% of the data leads to better results on most text-only tasks and improves the average performance across the 11 multimodal academic benchmarks.
我们在 7B 规模做小实验：向微调混合加入来自 Tulu 3[^9] [113] 的纯文本数据，试两种比例：全集与 10% 降采样。加入文本数据可提升纯文本任务，尤其数学推理与编程。有趣的是，降采样到 10% 在多数纯文本任务上更好，还提升了 11 项多模态学术基准的平均表现。
| model | MMLU [38] | MMLU-Pro [114] | GSM-8k [24] | MATH [39] | ARC-C [23] | HumanEval [16] | 11-avg |
|---|---|---|---|---|---|---|---|
| Qwen2-7B (language model) | 70.2 | 42.1 | 71.8 | 40.3 | 87.5 | 47.6 | - |
| Molmo-7B-D | 64.6 | 32.2 | 58.8 | 11.5 | 81.5 | 36.6 | 76.9 |
| Molmo-7B-D + Tulu 3 | 64.9 | 38.6 | 67.7 | 8.3 | 84.5 | 51.2 | 76.9 |
| Molmo-7B-D + Tulu 3 0.1 down-sample | 65.4 | 37.3 | 71.2 | 27.5 | 84.9 | 55.5 | 77.1 |
Table 13: Text-only benchmark results. 11-avg denotes the average performance on 11 academic benchmarks.
| 模型 | MMLU [38] | MMLU-Pro [114] | GSM-8k [24] | MATH [39] | ARC-C [23] | HumanEval [16] | 11-avg |
|---|---|---|---|---|---|---|---|
| Qwen2-7B (language model) | 70.2 | 42.1 | 71.8 | 40.3 | 87.5 | 47.6 | - |
| Molmo-7B-D | 64.6 | 32.2 | 58.8 | 11.5 | 81.5 | 36.6 | 76.9 |
| Molmo-7B-D + Tulu 3 | 64.9 | 38.6 | 67.7 | 8.3 | 84.5 | 51.2 | 76.9 |
| Molmo-7B-D + Tulu 3 0.1 down-sample | 65.4 | 37.3 | 71.2 | 27.5 | 84.9 | 55.5 | 77.1 |
表 13：纯文本基准。加入降采样的 Tulu 3 数据可部分挽回多模态训练带来的文本能力损失，并略提 11-avg。
Human evaluation.
Figure 10: Human evaluation outcomes for matches between various models vs. Molmo-7B-D. We expand upon the win rate (excluding ties) shown in Table 5 to report the full breakdown of wins, losses, ties (both good), and ties (both bad). We removed the I don’t know responses, which accounted for 2.9% of all human feedback, before calculating the outcome rates.
人工评测。
图 10：各模型对阵 Molmo-7B-D 的人工评测结果。在表 5 所示胜率（不含平局）之外，给出胜、负、平局（都好）、平局（都差）的完整分解。计算比率前去掉占全部人工反馈 2.9% 的「我不知道」。
Table 5 of the main paper reports the win rates of several ablation and API-only models vs. Molmo-7B-D when ties are excluded (a standard metric reported in the LMSYS Chatbot Arena). Ties make up a significant portion of the matches, so we report the full breakdown of match outcomes in Figure 10 to better characterize the human evaluation. For example, when paired against Claude-3.5 Sonnet 45.5% of matches resulted in a tie where both responses were good, 14.1% in a tie where both responses were bad, Claude won 26.1% of the time, and Molmo-7B-D won 14.3% of the time. As a second example, compared to Molmo fine-tuned only on academic data the breakdown is 28.4% (ties, both good), 22.1% (ties, both bad), 8.4% (it wins), 41.1% (Molmo-7B-D wins).
主文表 5 报告若干消融与仅 API 模型相对 Molmo-7B-D 的胜率（不含平局，LMSYS Chatbot Arena 常用指标）。平局占对局很大比例，故在图 10 给出完整结果分解。例如对阵 Claude-3.5 Sonnet：45.5% 双方都好的平局，14.1% 双方都差的平局，Claude 胜 26.1%，Molmo-7B-D 胜 14.3%。再如对阵仅学术数据微调的 Molmo：28.4%（都好平局）、22.1%（都差平局）、对方胜 8.4%、Molmo-7B-D 胜 41.1%。
AI2D with opaque boxes.
Table 14 shows result with and without opaque boxes on AI2D. The two options are described and discussed in Section B.2.
带不透明框的 AI2D。
表 14 给出 AI2D 上透明与不透明框结果。两种选项见第 B.2 节讨论。
| model | opaque | transparent |
|---|---|---|
| MolmoE-1B | 75.7 | 86.4 |
| Molmo-7B-O | 79.8 | 90.7 |
| Molmo-7B-D | 82.4 | 93.2 |
| Molmo-72B | 86.4 | 96.3 |
Table 14: AI2D test scores with transparent and opaque boxes.
| 模型 | 不透明 | 透明 |
|---|---|---|
| MolmoE-1B | 75.7 | 86.4 |
| Molmo-7B-O | 79.8 | 90.7 |
| Molmo-7B-D | 82.4 | 93.2 |
| Molmo-72B | 86.4 | 96.3 |
表 14：AI2D 测试集上透明与不透明框分数。
Cap and 11-avg correlation.
Figure 11: Relationship between cap and 11-avg. Our model development was driven by increasing cap . Here, we show a scatter plot of cap vs. the 11 benchmark average (11-avg) from 22 ablation experiments, including all ablations that: (1) affect pre-training and (2) use PixMo-Cap. The Pearson correlation () is 0.82 and a least-squares regression line is shown in red.
Cap 与 11-avg 的相关性。
图 11：cap 与 11-avg 的关系。模型开发以抬高 cap 为主线。图中为 22 次消融实验的散点：均满足 (1) 影响预训练且 (2) 使用 PixMo-Cap。Pearson 相关 ，红线为最小二乘回归。
For the majority of the project we did not look at downstream tasks[^10] and instead made most modeling decisions to maximize our captioning metric. At the conclusion of the project, we used our ablation experiments to analyze the relationship between cap and the 11 benchmark average (11-avg), shown in Figure 11. The scatter plot includes results from the 22 experiments that meet two conditions: (1) the experiment affects pre-training and (2) the experiment uses PixMo-Cap. We exclude a small number of experiments that use different pre-training data, e.g. ShareGPT4o/v, because cap becomes an out-of-domain evaluation that is not directly comparable to in-domain results. We observe a strong correlation (Pearson 0.82), suggesting that optimizing for dense captioning may be a reasonable proxy for a broad range of downstream tasks—though we have not established a causal relationship and this suggestion should be taken with that caveat in mind.
项目大部分时间我们几乎不看下游任务[^10]，建模决策主要最大化描述指标。收尾时用消融分析 cap 与 11 基准平均（11-avg）的关系，见图 11。散点含满足两条件的 22 次实验：(1) 影响预训练；(2) 使用 PixMo-Cap。排除少量换用不同预训练数据的实验（如 ShareGPT4o/v），因彼时 cap 成域外评测、不可与域内直接比。观察到强相关（Pearson 0.82），提示优化稠密描述或可作为宽域下游任务的合理代理——但未建立因果，需谨慎解读。
Leaderboards.
We submitted Molmo-72B-D to several leaderboards. Molmo-72B-D achieves first on the VQA v2.0 leaderboard on the A-OKVQA leaderboard, achieves third on DocQA and InfoQA, behind QwenVL-72B, InternVL2-Pro. [^11]
排行榜。
我们向若干排行榜提交了 Molmo-72B-D：在 VQA v2.0 与 A-OKVQA 排行榜上排名第一；DocQA 与 InfoQA 上第三，落后于 QwenVL-72B、InternVL2-Pro。[^11]
Appendix E Ablations Details
附录 E 消融细节
E.1 Discussion of Main Paper Ablations
E.1 主文消融讨论
Vision encoder.
We adopted OpenAI’s CLIP early on and used it for our main results and as the default in our ablations. Later, we evaluated the three alternative choices in Table . All encoders are ViT-L/14 with 336336 pixel inputs, except for SigLIP which uses 384384 pixels. For MetaCLIP, we started with the weights of the 224224 model and resized the positional embeddings to 336336 before using it in Molmo. To equalize computation, we slightly reduced the maximum number of crops for SigLIP so that the average vision token count is similar for all models. Overall, the three encoders that were trained on web-scale noisy image-text data perform very similar to each other on both metrics. Of significant note, this includes MetaCLIP which is a fully open model (data and weights) meaning that every model component and every bit of data in a Molmo model equipped with MetaCLIP and OLMo is open. In retrospect, we should have used MetaCLIP as our default vision encoder, but we evaluated it too late in the process to retrain all Molmo models and ablations that were already based on OpenAI’s CLIP vision encoder.
视觉编码器。
我们很早就采用 OpenAI 的 CLIP，主结果与消融默认均用它。之后才评测表中三种替代。除 SigLIP 用 外，其余均为 ViT-L/14、 输入。MetaCLIP 从 权重起步，把位置嵌入调到 再接入 Molmo。为对齐算力，对 SigLIP 略减最大裁剪数，使平均视觉 token 数相近。总体而言，三个在网络级嘈杂图文上训的编码器在两项指标上十分接近。尤其值得注意的是其中含完全开放（数据与权重）的 MetaCLIP——这意味着配备 MetaCLIP 与 OLMo 的 Molmo，每个组件与每一比特数据都开放。事后看本应以 MetaCLIP 为默认视觉编码器，但评估太晚，已无法重训所有基于 OpenAI CLIP 的 Molmo 与消融。
Also, surprisingly, when using the DINOv2 backbone—which is trained on images only (no text, no label supervision)—Molmo performs only slightly worse than the vision-language supervised vision encoders. DINOv2 also performs well in our user study (Table 5), with a win-rate (excluding ties) of 45% compared to our standard Molmo-7B-D configuration (i.e., Molmo-7B-D wins 55% of all non-tie matches against its DINOv2-based variant).
此外令人意外的是：改用仅在图像上训练、无文本与标签监督的 DINOv2 骨干时，Molmo 只略弱于视觉–语言监督的编码器。DINOv2 在用户研究（表 5）中也不错：相对标准 Molmo-7B-D，不含平局的胜率为 45%（即 Molmo-7B-D 在非平局对局中胜 55%）。
Image resolution.
Table shows that using more crops (and thus higher image resolution) for training and testing generally improves results. We found that some tasks, like document-heavy ones, even benefit from using more crops in inference than the number used for training. However, for captioning and pointing (and thus counting), results degrade when the number of test crops does not equal the number of training crops. Therefore, for captioning and counting tasks we always force these values to be the same. As shown in Table 12 this awkward detail can be remediated by a small amount of high-resolution fine-tuning and then always using that same number of crops during inference for all tasks.
图像分辨率。
表中显示训练与测试用更多裁剪（因而更高分辨率）通常更好。文档向任务甚至受益于推理裁剪数多于训练。但描述与指点（因而计数）在测试裁剪数≠训练裁剪数时会变差，故这两类任务我们强制二者相同。如表 12，这一别扭细节可通过少量高分辨率微调缓解，之后所有任务推理都用同一裁剪数。
Dropout.
Dropout in the LLM generally improves both pre-training and fine-tuning (Table ). We also find that our novel text-token-only dropout, in which dropout is only applied to the text tokens of the caption, not to the vision or prompt tokens, improves the captioning metric. We hypothesize that this restricted dropout encourages the model to rely more on the vision tokens, rather than guess based on the previous text tokens, when generating tokens which may reduce hallucinations.
Dropout。
LLM 中的 dropout 通常同时改善预训练与微调（表 ）。我们还发现仅对描述文本 token 做 dropout、不对视觉或提示 token 做的新做法能提升描述指标。我们假设这种受限 dropout 促使模型生成时更依赖视觉 token，而非凭前文猜测，从而可能减少幻觉。
Length conditioning.
Our captioning pre-training task includes a length hint. In Table we ablate this design choice and find that it significantly impacts the captioning metric, but also improves the downstream tasks. Note that length conditioning only changes the pre-training task; the fact that it improves the downstream metrics indicates that captioning with length conditioning is a better pre-training task than just captioning.
长度条件。
描述预训练任务含长度提示。表中消融显示它显著影响描述指标，也改善下游。注意长度条件只改预训练任务；下游变好说明「带长度条件的描述」比单纯描述是更好的预训练任务。
PixMo-Cap scaling.
In Table we show the scaling effects of PixMo-Cap data by training with smaller fractions of the data in both pre-training and as part of the fine-tuning data mixture. Both metrics clearly improve as the amount of captioning data varies from none at all to the full set of 712k images. We also tested the model with no PixMo-Cap data in our user study (Table 5), where it had a win-rate (excluding ties) of only 35% compared to our standard Molmo-7B-D configuration. Removing PixMo-Cap data has a severe negative impact on its user preference score.
PixMo-Cap 规模。
表中展示在预训练与微调混合中使用更小比例 PixMo-Cap 的规模效应。描述数据从无到满额 71.2 万图，两项指标都明显上升。用户研究（表 5）中去掉 PixMo-Cap 的模型相对标准 Molmo-7B-D 不含平局胜率仅 35%，对用户偏好打击严重。
Pre-training data.
We consider different choices of pre-training data in Table . Pre-training VLMs, not just the vision encoder, with web-scale noisy image-text data is a popular data choice in contemporary methods (e.g., [10, 5]). We test if this has any advantage using data from LAION [98]. To do this we add a preliminary training stage that tunes the model for 50k steps and a batch size of 1024 on image/text pairs from LAION 2B. In this stage only the V/L connector is tuned, the LLM and image encoder are frozen. This pre-trained model is then trained on the dense captions and then our instruction tuning mixture as normal. We find no improvement in metrics using this strategy, allowing us to keep the training pipeline simple.
预训练数据。
表中比较不同预训练数据。当代方法常在网络级嘈杂图文上预训练整棵 VLM（不只视觉编码器）（如 [10, 5]）。我们用 LAION [98] 检验是否有收益：先加一阶段，在 LAION 2B 图文对上训 5 万步、batch 1024，只调视觉/语言连接器，冻结 LLM 与图像编码器；再照常训稠密描述与指令混合。两项指标均无提升，故可保持训练流水线简洁。
Another popular choice is to use ShareGPT4V/o [15], which involves distilling from GPT-4 through captions. Using this data instead of PixMo-Cap performs worse on both metrics even when approximately controlling for the data scale (compare to 178k PixMo-Cap images in Table . In contrast, if we caption all PixMo-Cap images with GPT-4o and train on those captions, both metrics perform strongly. We think this is likely because PixMo-Cap has a more diverse image distribution, and due to captioning improvements in GPT-4o. Finally, we compare our default setting to either using only the raw audio transcripts or only using the LLM cleaned transcripts, both of which perform slightly worse than our default strategy of using both.
另一流行选择是 ShareGPT4V/o [15]，经描述蒸馏自 GPT-4。用它替代 PixMo-Cap，即便大致控制数据规模（对照表中 17.8 万 PixMo-Cap 图），两项指标都更差。反之，若用 GPT-4o 描述全部 PixMo-Cap 图像并据此训练，两项指标都强——我们归因于 PixMo-Cap 图像分布更多样，以及 GPT-4o 描述质量提升。最后，默认设定（原始转写与 LLM 清洗转写都用）略优于只用原始或只用清洗转写。
Supervised fine-tuning data.
We explore choices of fine-tuning data in Table . Using only academic data sets (specifically the ones in Fig. 4, but excluding AndroidControl) performs significantly worse than our full mixture (72.2% vs. 76.8%). The gap is primarily explained by PixMo-Docs, which improves results on document-heavy tasks, and the counting data from PixMo-Points and PixMo-Count. The other fine-tuning PixMo datasets have a small, and sometimes slightly negative, impact on the 11 benchmarks; they primarily add new skills to the model and improve user experience when chatting with it, as shown by user preference scores in Table 5.
有监督微调数据。
表中探索微调数据选择。仅用学术集（具体为图 4 中集合，不含 AndroidControl）显著弱于完整混合（72.2% 对 76.8%）。差距主要来自提升文档向任务的 PixMo-Docs，以及 PixMo-Points / PixMo-Count 的计数数据。其余 PixMo 微调集对 11 项基准影响很小、有时略负；它们主要给模型加新技能、改善聊天体验，见表 5 用户偏好。
Counting.
In Table we compare encoding points in plain-text as numbers between 0.0 and 100.0 with one significant digit of precision (our default) vs. adding 1000 special point tokens to the model’s tokenizer, maintaining the same spatial precision. We find that using special point tokens performs substantially worse than the simple plain-text representation.
计数。
表中比较两种点编码：默认把坐标写成 0.0–100.0、一位小数的纯文本数字；或在分词器中加入 1000 个特殊点 token 且保持相同空间精度。特殊点 token 明显弱于简单纯文本表示。
| layers | cap | 11-avg |
|---|---|---|
| 3rd-to-last & 10th-to-last | 54.1 | 76.9 |
| only 3rd-to-last | 53.7 | 76.6 |
| only 10th-to-last | 52.5 | 76.3 |
| steps (ViT / con. / LLM) | cap | 11-avg |
|---|---|---|
| 2000 / 200 / 2000 | 54.1 | 76.9 |
| 200 / 200 / 200 | 53.7 | 76.9 |
| grad norm | cap | 11-avg |
|---|---|---|
| component-wise | 54.1 | 76.9 |
| global | 53.6 | 76.9 |
| global, fine-tune only | 54.1 | 76.9 |
(a)
| 层 | cap | 11-avg |
|---|---|---|
| 倒数第 3 与倒数第 10 | 54.1 | 76.9 |
| 仅倒数第 3 | 53.7 | 76.6 |
| 仅倒数第 10 | 52.5 | 76.3 |
| 步数（ViT / 连接器 / LLM） | cap | 11-avg |
|---|---|---|
| 2000 / 200 / 2000 | 54.1 | 76.9 |
| 200 / 200 / 200 | 53.7 | 76.9 |
| 梯度范数 | cap | 11-avg |
|---|---|---|
| 按组件 | 54.1 | 76.9 |
| 全局 | 53.6 | 76.9 |
| 全局，仅微调 | 54.1 | 76.9 |
(a)
E.2 Additional Ablations
E.2 补充消融
Additional model ablations are presented in Table 15 for vision encoder layers, learning rate warmup, and gradient normalization. See table captions for more details.
表 15 给出视觉编码器层、学习率预热与梯度归一化的补充模型消融，细节见表注。
Appendix F Data Details
附录 F 数据细节
Figure 12: PixMo-Points distribution of counts. We show the number of pointing questions (on a log scale) with answers in different ranges (e.g., 1 to 10, 11 to 20, etc.).
图 12：PixMo-Points 计数分布。纵轴为对数刻度，展示答案落在不同区间（如 1–10、11–20 等）的指点问题数量。
PixMo-Points.
The PixMo-Points dataset has a total of 229k unique images and a total of 1.98M referring expressions. It has an average of 8.7 distinct expressions per image with an average 5.5 points per expression, and an average of 47.7 total points per image. Additionally, there are 359k instances with no target object (no points). Figure 12 shows the distribution of number of points for expressions with non-zero points. PixMo-Points is a much larger and more diverse dataset than previous works such as gRefCOCO [66] (which contains a total of 20k images, 60k distinct instances, 278k expressions, of which 80k are multi-target and 32k are no-target expressions) and also much larger than RefCOCO, RefCOCOg and RefCOCO+ [126], each with about 86k, 142k and 141k unique referring expressions respectively and no multi-target references. Additionally, PixMo-Points focuses on referring to points and not segmentation masks, making it significantly more efficient to collect.
PixMo-Points。
共 22.9 万张不重复图、198 万条指称表达；平均每图 8.7 条不同表达、每表达 5.5 个点、每图合计 47.7 个点。另有 35.9 万条无目标（无点）实例。图 12 给出非零点数表达的点数分布。规模与多样性远超 gRefCOCO [66]（2 万图、6 万实例、27.8 万表达，其中 8 万多目标、3.2 万无目标）以及 RefCOCO / RefCOCOg / RefCOCO+ [126]（各约 8.6 万、14.2 万、14.1 万唯一指称、且无多目标）。此外 PixMo-Points 锚定的是点而非分割掩码，采集效率高得多。
PixMo-Cap.
We prompted our annotators with the following questions to answer in their spoken image descriptions.
PixMo-Cap。
我们提示标注者在口述图像描述时回答下列问题。
- What is the image at first glance?
- 第一眼这张图是什么？
- What are the objects and their counts?
- 有哪些物体，各有多少？
- What does the text say?
- 图上文字写了什么？
- What are the positions of the objects?
- 物体分别在什么位置？
- What subtle details are noticeable?
- 有哪些细微可见细节？
- What is in the background?
- 背景里有什么？
- What is the style and color?
- 风格与色彩如何？
PixMo-Docs.
We developed a generation framework for synthesizing text- and figure-heavy images. The core idea is to harness the coding capabilities of a text-only LLM to generate programs that render image data. These programs are then used as context for another LLM to construct instruction-tuning datasets.
PixMo-Docs。
我们搭建用于合成偏文本与图表图像的生成框架。核心是利用纯文本 LLM 的编码能力生成渲染图像的程序，再把这些程序当作上下文，让另一 LLM 构造指令微调数据。
Our framework supports seven programming languages/rendering libraries, including Matplotlib, Plotly, LaTeX, HTML, Vega-Lite, Mermaid, and Graphviz. Using these tools, we designed specialized pipelines to generate charts, tables, diagrams, and various types of documents.
框架支持七种编程语言/渲染库：Matplotlib、Plotly、LaTeX、HTML、Vega-Lite、Mermaid、Graphviz。据此设计专用流水线，生成图表、表格、示意图及各类文档。
The framework accepts text input to control the generation process. For instance, given the input “restaurant menu”, the system selects the appropriate tools to generate relevant data. To diversify the final datasets, we use a comprehensive set of input queries. Additionally, we enhance data diversity by incorporating personas [35], which control the content and style of the synthetic data. For example, when generating “restaurant menu” data with the persona “A barbecue enthusiast known for their amazing grilled food at every Tennessee Vols game”, the framework produces a data point featuring a “Southern fusion menu combining traditional BBQ with international flavors, presented on a wooden board background”. This approach allows us to enrich the variety within each category of synthetic data.
框架接受文本输入控制生成。例如输入 “restaurant menu”，系统选用合适工具生成相关数据。为多样化最终数据集，我们使用全面的查询集合，并引入 persona [35] 控制合成内容与风格。例如以 persona「一位以田纳西志愿军每场比赛上惊艳烧烤闻名的烧烤爱好者」生成餐厅菜单时，框架会产出「木板背景上呈现、融合传统 BBQ 与国际风味的南方融合菜单」一类数据点，从而丰富各类合成数据的内部多样性。
We use Claude-3.5 Sonnet [7] for code generation and GPT-4o-mini [89] during the instruction-tuning data generation stage, prioritizing cost efficiency.
代码生成用 Claude-3.5 Sonnet [7]，指令微调数据生成阶段用 GPT-4o-mini [89]，以兼顾成本。
Figure 13: VLM Openness Comparison. We characterize the openness of VLMs based on two attributes (open weights, open data and code) across three model components (the VLM and its two pre-trained components, the LLM backbone and the vision encoder). In addition to open vs. closed, we use the ”distilled” label to indicate that the data used to train the VLM includes images and text generated by a different, proprietary VLM, meaning that the model cannot be reproduced without a dependency on the proprietary VLM.
图 13：VLM 开放度比较。我们用两个属性（开放权重；开放数据与代码）刻画三类组件（VLM 本身及其两个预训练组件：LLM 骨干与视觉编码器）的开放度。除开放/封闭外，用「蒸馏」标签表示训练数据含由另一闭源 VLM 生成的图文，因而无法在不依赖该闭源 VLM 的情况下复现模型。
Appendix G Dataset Examples
附录 G 数据样例
We include randomly selected examples from the PixMo- datasets. Prompts are shown in bold, and points are shown with pink dots.
我们收录 PixMo- 数据集的随机样例。提示以粗体显示，点以粉色圆点标出。
- • PixMo-Cap - Figure 14
- • PixMo-Cap - 图 14
- • PixMo-AskModelAnything- Figure 15
- • PixMo-AskModelAnything - 图 15
- • PixMo-Points - Figure 16
- • PixMo-Points - 图 16
- • PixMo-Points with explanations - Figure 17
- • 带解释的 PixMo-Points - 图 17
- • PixMo-CapQA - Figure 18
- • PixMo-CapQA - 图 18
- • PixMo-Clocks - Figure 19
- • PixMo-Clocks - 图 19
- • PixMo-Count - Figure 20
- • PixMo-Count - 图 20
- • PixMo-Docs (charts) - Figure 21
- • PixMo-Docs（图表） - 图 21
- • PixMo-Docs (tables) - Figure 22
- • PixMo-Docs（表格） - 图 22
- • PixMo-Docs (diagrams) - Figure 23
- • PixMo-Docs（示意图） - 图 23
- • PixMo-Docs (other) - Figure 24
- • PixMo-Docs（其他） - 图 24
Appendix H Related Work
附录 H 相关工作
Vision-language contrastive models.
Vision-language models have become popular in the last few years. Models such as CLIP [96] and ALIGN [43] that are trained on noisy web data provide strong language-aligned image encoders and perform well on downstream classification and image-text retrieval tasks, without any task specific tuning. Previous works [115, 33] proposed similar ideas before transformers [109] became popular. Since CLIP was released, other works have focused on making the CLIP pipeline fully open [118, 20]. However, vision encoders trained with noisy web data have limitations in discerning details, as discussed in [107].
视觉–语言对比模型。
近年视觉–语言模型流行起来。CLIP [96]、ALIGN [43] 等在嘈杂网络数据上训练，提供强语言对齐的图像编码器，无需任务微调即可在分类与图文检索上表现良好。Transformer [109] 流行前已有类似思路 [115, 33]。CLIP 发布后，工作重心之一是把 CLIP 流水线完全开放 [118, 20]。但嘈杂网络数据训出的视觉编码器在分辨细节上有局限，见 [107]。
Multimodal LLMs.
Multimodal LLMs often use CLIP-style image encoders and align image embeddings with the LLM input space via a connector module [69, 59, 71, 92, 26, 112, 41, 105, 85, 131]. Some works have also explored using multiple image encoders in tandem with CLIP-style encoders [106, 74], such as using self-supervised learning (SSL) encoders [91]. Many works use a pre-training stage for just the connector weights [69, 106, 12, 34] while others do not have an explicit connector training stage [48, 10, 67, 18]. In contrast, two other common architecture strategies are (1) directly connecting the image embedding to different LLM layer embeddings via cross-attention [6, 5, 58, 134] and (2) removing the image encoder and directly inputting the pixels [9, 56]. The cross-attention design naturally allows for the introduction of a large number of new parameters, which enables freezing the LLM while still training an effective VLM. This approach has the advantage of maintaining text-only task performance (cf. Table 13). Due to the compute constraints for training and inference of these models, there has also been a rise in efficient multimodal LLMs [128, 79, 14, 64, 22, 123, 137].
多模态 LLM。
多模态 LLM 常用 CLIP 风格图像编码器，经连接模块把图像嵌入对齐到 LLM 输入空间 [69, 59, 71, 92, 26, 112, 41, 105, 85, 131]。也有工作把多个编码器与 CLIP 风格编码器并用 [106, 74]，例如自监督（SSL）编码器 [91]。许多工作先只训连接器 [69, 106, 12, 34]，也有不做显式连接器阶段的 [48, 10, 67, 18]。另两类常见结构是：(1) 经交叉注意力把图像嵌入直接接到不同 LLM 层 [6, 5, 58, 134]；(2) 去掉图像编码器、直接吃像素 [9, 56]。交叉注意力天然可引入大量新参数，从而冻结 LLM 仍训出有效 VLM，并利于保住纯文本任务表现（对照表 13）。因训练与推理算力约束，高效多模态 LLM 也在增多 [128, 79, 14, 64, 22, 123, 137]。
The best performing multimodal LLMs [90, 7, 103] are proprietary closed source models. While they are very capable, not much is known about how these models are trained and what data they use. In contrast, many works release their model weights [3, 111, 1, 10, 5] but don’t release their training recipes or don’t disclose all the data used. Other works provide all the training details and data [59, 106, 119, 54, 133], but use data generated by proprietary VLMs such as [15]. Hence, there is a need for a fully open SoTA training pipeline that does not use previously trained multimodal LLMs to generate data.
表现最好的多模态 LLM [90, 7, 103] 多为闭源。能力虽强，训练方式与数据所知甚少。相对地，许多工作开放权重 [3, 111, 1, 10, 5] 却不公开配方或不披露全部数据；另一些给出完整训练细节与数据 [59, 106, 119, 54, 133]，却使用闭源 VLM 生成的数据（如 [15]）。因此需要一条不依赖既有多模态 LLM 生成数据的、全开放且达到 SoTA 的训练流水线。
Vision-language instruction tuning datasets.
The rise in popularity of VLMs has also led to a rise of methods to build visual instruction-tuning data. A common approach is to annotate an image with vision models (or use ground-truth annotations), and then use a LLM to generate QA pairs [69, 137, 57, 125] from those annotations. However, these approaches are limited since the automatically generated annotations can be noisy, and even ground-truth annotations often do not comprehensively describe all the details in the image. PixMo-CapQA takes a similar approach but uses the detailed captions from PixMo-Cap which provide more comprehensive image descriptions. Many recent methods have used proprietary VLMs to annotate images directly [15, 13, 71, 110, 68], which is effective but makes the training pipeline dependent on a closed source VLM.
视觉–语言指令微调数据。
VLM 流行也催生了构建视觉指令微调数据的方法。常见做法是用视觉模型（或真值）标注图像，再让 LLM 据此生成问答 [69, 137, 57, 125]。局限在于自动标注可能嘈杂，即便真值标注也往往无法覆盖图中全部细节。PixMo-CapQA 思路类似，但依托 PixMo-Cap 的细粒度描述，覆盖更全。近期许多方法直接用闭源 VLM 标注图像 [15, 13, 71, 110, 68]，有效却使训练流水线依赖闭源 VLM。
It is also very common to pair templated instructions with existing annotated datasets to build instruction tuning data (e.g., [57, 75, 44, 5]). While Molmo also uses academic datasets, we prefer style tags over natural language instructions since we believe our data, and in particular PixMo-AskModelAnything, provides better training for conversational user interactions.
也很常见的是把模板化指令与已有标注集合配对做成指令数据（如 [57, 75, 44, 5]）。Molmo 也用学术集，但更偏好风格标签而非自然语言指令，因我们认为自有数据——尤其 PixMo-AskModelAnything——更能训练对话式用户交互。
Our approach to having annotators work with a LLM when generating QA pairs is similar to the approach in [80], but we extend this idea to image/language data.
生成问答时让标注者与 LLM 协作的做法类似 [80]，我们把它扩展到图/语言数据。
Synthetic vision-language datasets.
Prior approaches to synthetic chart generation typically only support one or two types of charts [46, 86, 47], often with a heavy focus on bar charts or line plots. PixMo-Docs uses code as the text-only representation for the LLM which lets us support much more diverse formats, including heat-maps, violin plots, chord diagrams, geographic plots, tree maps among many others. Our use of HTML for document generation is also similar to [55], however we consider many additional approaches to representing documents besides HTML.
合成视觉–语言数据。
既往合成图表生成通常只支持一两类图 [46, 86, 47]，且偏重柱状图或折线。PixMo-Docs 用代码作为给 LLM 的纯文本表示，因而能覆盖热图、小提琴图、弦图、地理图、树图等远更丰富的格式。用 HTML 生成文档也类似 [55]，但我们还考虑了 HTML 以外的多种文档表示。
Synthetic clock data has been considered [121]. Our approach uses real watch faces instead of rendering clocks purely from a simulator, which gives our synthetic data more diversity (e.g., watches with no second hand, stylized decorations or coloring, background images, a separate inner piece to show seconds). Combining these two datasets might yield additional improvements.
合成时钟数据亦有先例 [121]。我们用真实表盘而非纯模拟器渲染，多样性更高（如无秒针、风格化装饰或配色、背景图、独立秒盘等）。两套数据结合或可再提升。
VLM grounding.
Multimodal LLMs that support grounding language in an image are becoming more common [94, 132, 92, 97, 117, 61, 127]. These works commonly use automated object detectors and/or existing referring expression datasets [126, 66, 52] for training data. Of these datasets, GRES [66] is most similar to PixMo-Points in that it is human-annotated and includes arbitrary expressions (not just object categories), none-present annotations, and allows expressions to refer to multiple object in the image. However, it only grounds a limited category of objects (e.g., only COCO categories), and rarely grounds expressions in a large numbers of objects in a single image. For PixMo-Points we source a diverse set of images and collect points from human annotators. We annotate points instead of segmentation masks, which enables us to collect 1.98M unique referring expression instances with an average of 5.5 points per expression.
VLM grounding。
支持把语言锚定到图像的多模态 LLM 越来越多 [94, 132, 92, 97, 117, 61, 127]。训练数据常用自动检测器和/或既有指称表达数据集 [126, 66, 52]。其中 GRES [66] 与 PixMo-Points 最接近：人工标注、含任意表达（不限物体类别）、无目标标注，且允许一条表达指多个目标。但它只锚定有限类别（如仅 COCO 类），且很少在单图锚定大量目标。PixMo-Points 则采集多样图像并由人标点；标点而非分割掩码，使我们能收集 198 万唯一指称实例、平均每表达 5.5 个点。
Bootstrapping from LLMs.
Closed text-only LLMs are commonly used for data generation and curation [69]. We also made use of closed text-only LLMs when building several of the PixMo datasets. Given our stance against using VLMs for building datasets, it’s worth justifying the use of closed LLMs. It is true using closed LLMs means the current data pipeline is not entirely open. However, once open LLMs (e.g., [37]) become sufficiently good they can be used in place of closed ones to build a dataset functionally equivalent to PixMo. Our philosophy is that we should not wait for open LLM research to achieve this goal and instead we should pursue research on building open VLMs in parallel. We note that using one VLM to build another VLM is entirely different than using an LLM, because the dependency is circular and therefore cannot result in a fully open system at a later point in time.
从 LLM 引导。
闭源纯文本 LLM 常用于数据生成与整理 [69]。构建若干 PixMo 数据集时我们也用了闭源纯文本 LLM。既然反对用VLM 建数据，有必要说明为何仍用闭源 LLM。确实，用闭源 LLM 意味着当前数据流水线并非完全开放。但一旦开放 LLM（如 [37]）足够强，即可替换闭源 LLM，构建功能上等价于 PixMo 的数据。我们的立场是：不必等开放 LLM 研究达成后再动手，而应并行推进开放 VLM 研究。用一个 VLM 去建另一个 VLM 则完全不同：依赖是环形的，因而日后也无法变成全开放系统。
Figure 14: Randomly selected examples from PixMo-Cap with our prompt templates.
图 14：从 PixMo-Cap 随机抽取的样例，配以我们的提示模板。
Figure 15: Randomly selected examples from PixMo-AskModelAnything.
图 15：从 PixMo-AskModelAnything 随机抽取的样例。
Figure 16: Randomly selected examples from PixMo-Points. Even when text has been cut off, all points still appear in the image. Our templated prompts can be ungrammatical for some of these options, but we find they are still sufficient to let the model respond correctly to natural language instructions.
图 16：从 PixMo-Points 随机抽取的样例。即便文字被裁切，点仍全部落在图内。部分模板提示在语法上不完美，但仍足以让模型正确响应自然语言指令。
Figure 17: Randomly selected examples from the experimental PixMo-Points data that includes points with explanations.
图 17：实验性 PixMo-Points（含带解释的点）随机样例。
Figure 18: Randomly selected examples from the synthetic PixMo-CapQA data generated from PixMo-Cap.
图 18：由 PixMo-Cap 生成的合成 PixMo-CapQA 随机样例。
Figure 19: Randomly selected examples from the synthetic PixMo-Clocks data after our data augmentation.
图 19：数据增强后的合成 PixMo-Clocks 随机样例。
Figure 20: Randomly selected examples from the synthetic PixMo-Count data.
图 20：合成 PixMo-Count 随机样例。
Figure 21: Randomly selected chart examples from the synthetic PixMo-Docs data.
图 21：合成 PixMo-Docs 中的随机图表样例。
Figure 22: Randomly selected table examples from the synthetic PixMo-Docs data.
图 22：合成 PixMo-Docs 中的随机表格样例。
Figure 23: Randomly selected diagram examples from the synthetic PixMo-Docs data.
图 23：合成 PixMo-Docs 中的随机示意图样例。
Figure 24: Other randomly selected documents from the synthetic PixMo-Docs data.
图 24：合成 PixMo-Docs 中其他随机文档样例。
References
参考文献
20660108922671411210567581348513161741125413356183463102421229413292971171275755443212879146422123137
（参考文献条目见原文。）
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 , 2024.
[2] Manoj Acharya, Kushal Kafle, and Christopher Kanan. TallyQA: Answering complex counting questions. In AAAI , 2019.
[3] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073 , 2024.
[4] 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, and more. Yi: Open foundation models by 01.ai. arXiv preprint arXiv:2403.04652 , 2024.
[5] Meta AI. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.
[6] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022.
[7] Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024.
[8] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In NeurIPS Deep Learning Symposium , 2016.
[9] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Fuyu-8b: A multimodal architecture for ai agents, 2023.
[10] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07726 , 2024.
[11] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV , 2019.
[12] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal llm. In CVPR , 2024.
[13] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684 , 2024a.
[14] Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, Changyi Liu, Dewen Fan, Huihui Xiao, Jiahong Wu, Fan Yang, Size Li, and Di Zhang. Evlm: An efficient vision-language model for visual understanding. arXiv preprint arXiv:2407.14177 , 2024b.
[15] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. ShareGPT4V: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 , 2023a.
[16] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.
[17] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015.
[18] Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199 , 2023b.
[19] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to GPT-4V? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821 , 2024c.
[20] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR , 2023.
[21] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In ICML , 2024.
[22] Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and Chunhua Shen. Mobilevlm : A fast, strong and open vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886 , 2023.
[23] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.
[24] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.
[25] David F Crouse. On implementing 2d rectangular assignment algorithms. IEEE Transactions on Aerospace and Electronic Systems , 2016.
[26] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS , 2023.
[27] Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: Open frontier-class multimodal LLMs. arXiv preprint arXiv:2409.11402 , 2024.
[28] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In ICLR , 2024.
[29] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In NeurIPS , 2022.
[30] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD. arXiv preprint arXiv:2404.06512 , 2024.
[31] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.
[32] Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jan Kautz, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. Vila 2 : Vila augmented vila. arXiv preprint arXiv:2407.17453 , 2024.
[33] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc’Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. NeurIPS , 2013.
[34] Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211 , 2024.
[35] Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.20094 , 2024.
[36] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR , 2017.
[37] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. OLMo: Accelerating the science of language models. In ACL , 2024.
[38] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR , 2021a.
[39] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS Track on Datasets and Benchmarks , 2021b.
[40] Joeri R Hermans, Gerasimos Spanakis, and Rico Möckel. Accumulated gradient normalization. In Asian Conference on Machine Learning , pages 439–454. PMLR, 2017.
[41] Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500 , 2024.
[42] Anwen Hu, Haiyang Xu, Jiabo Ye, Mingshi Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. In Findings of EMNLP , 2024.
[43] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.
[44] Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483 , 2024.
[45] Roy Jonker and Ton Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing , 1987.
[46] Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. DVQA: Understanding data visualizations via question answering. In CVPR , 2018.
[47] Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300 , 2017.
[48] Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865 , 2024.
[49] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV , 2016.
[50] Diederik P Kingma. Adam: A method for stochastic optimization. In ICLR , 2015.
[51] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV , 2023.
[52] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision , 123:32 – 73, 2016.
[53] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV , 2020.
[54] Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. arXiv preprint arXiv:2408.12637 , 2024a.
[55] Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into HTML code with the websight dataset. arXiv preprint arXiv:2403.09029 , 2024b.
[56] Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi-modality model. arXiv preprint arXiv:2311.04219 , 2023a.
[57] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, C. Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425 , 2023b.
[58] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726 , 2023c.
[59] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-OneVision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 , 2024a.
[60] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.
[61] Junyan Li, Delin Chen, Yining Hong, Zhenfang Chen, Peihao Chen, Yikang Shen, and Chuang Gan. Covlm: Composing visual entities and relationships in large language models via communicative decoding. In ICLR , 2024b.
[62] Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.03679 , 2024c.
[63] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. CVPR , 2024d.
[64] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947 , 2024.
[65] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.
[66] Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Generalized referring expression segmentation. In CVPR , 2023a.
[67] Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. SPHINX-x: Scaling data and parameters for a family of multi-modal large language models. In ICML , 2024a.
[68] Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311.10774 , 2023b.
[69] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS , 2023c.
[70] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR , 2024b.
[71] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024c.
[72] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.
[73] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019.
[74] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525 , 2024a.
[75] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR , 2024b.
[76] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS , 2022.
[77] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In ICLR , 2023.
[78] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR , 2024c.
[79] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instruction tuning for large language models. In NeurIPS , 2023.
[80] Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. ExpertQA: Expert-curated questions and attributed answers. arXiv preprint arXiv:2309.07852 , 2023.
[81] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In CVPR , 2019.
[82] Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL , 2022.
[83] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. DocVQA: A dataset for VQA on document images. In WACV , 2021.
[84] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. InfographicVQA. In WACV , 2022.
[85] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu He, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, and Yinfei Yang. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611 , 2024.
[86] Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. PlotQA: Reasoning over scientific plots. In WACV , 2020.
[87] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. OLMoE: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060 , 2024.
[88] OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.
[89] OpenAI. GPT-4o mini system card, 2024a.
[90] OpenAI. GPT-4o system card. arXiv preprint arXiv:2410.21276 , 2024b.
[91] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.
[92] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 , 2023.
[93] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In ECCV , 2020.
[94] Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chellappa, and Amjad Almahairi. Jack of all tasks, master of many: Designing general-purpose coarse-to-fine vision-language model. In CVPR , 2023.
[95] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. OpenAI Blog , 2018.
[96] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.
[97] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. GLaMM: Pixel grounding large multimodal model. In CVPR , 2024.
[98] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.
[99] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. In ECCV , 2022.
[100] Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR , 2019.
[101] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 2024.
[102] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In CVPR , 2024.
[103] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024a.
[104] OpenGVLab Team. Internvl2: Better than the best—expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024b.
[105] Rahul Thapa, Kezhen Chen, Ian Covert, Rahul Chalamala, Ben Athiwaratkun, Shuaiwen Leon Song, and James Zou. Dragonfly: Multi-resolution zoom-in encoding enhances vision-language models. arXiv preprint arXiv:2406.00977 , 2024.
[106] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. In NeurIPS , 2024a.
[107] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In CVPR , 2024b.
[108] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill, and Zacharias Janssen. Multimodal few-shot learning with frozen language models. In NeurIPS , 2021.
[109] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.
[110] Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574 , 2023a.
[111] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024a.
[112] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. In NeurIPS , 2024b.
[113] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. In NeurIPS Track on Datasets and Benchmarks , 2023b.
[114] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In NeurIPS Track on Datasets and Benchmarks , 2024c.
[115] Jason Weston, Samy Bengio, and Nicolas Usunier. Wsabie: Scaling up to large vocabulary image annotation. In IJCAI , 2011.
[116] xAI. RealWorldQA. https://huggingface.co/datasets/xai-org/RealworldQA , 2024. Accessed: 2024-09-24.
[117] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In CVPR , 2024.
[118] Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In ICLR , 2024.
[119] Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xGen-MM (BLIP-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872 , 2024.
[120] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671 , 2024.
[121] Charig Yang, Weidi Xie, and Andrew Zisserman. It’s about time: Analog clock reading in the wild. In CVPR , 2022.
[122] Huanjin Yao, Wenhao Wu, Taojiannan Yang, Yuxin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for mllms. In NeurIPS , 2024a.
[123] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 , 2024b.
[124] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. UReader: Universal OCR-free visually-situated language understanding with multimodal large language model. In Findings of EMNLP , 2023.
[125] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704 , 2023.
[126] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV , 2016.
[127] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In CVPR , 2024a.
[128] Zhengqing Yuan, Zhaoxu Li, Weiran Huang, Yanfang Ye, and Lichao Sun. Tinygpt-v: Efficient multimodal large language model via small backbones. In ICML Workshop on Advancing Neural Network Training , 2024b.
[129] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR , 2024.
[130] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV , 2023.
[131] Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, Afshin Dehghan, Peter Grasch, and Yinfei Yang. Mm1.5: Methods, analysis & insights from multimodal llm fine-tuning. arXiv preprint arXiv:2409.20566 , 2024a.
[132] Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong-You Chen, Tsu-Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, and Yinfei Yang. Ferret-v2: An improved baseline for referring and grounding with large language models. In COLM , 2024b.
[133] Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320 , 2024c.
[134] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In ICLR , 2024d.
[135] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: Experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277 , 2023.
[136] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV , 2022.
[137] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In ICLR , 2024.