SAM 3D: 3Dfy Anything in Images

SAM 3D: 3Dfy Anything in Images

SAM 3D：把图像里的任意物体三维化

SAM 3D Team Affiliation: Meta Superintelligence Labs Xingyu Chen Affiliation: Meta Superintelligence Labs Core Contributor (Alphabetical, Equal Contribution) Fu-Jen Chu Affiliation: Meta Superintelligence Labs Core Contributor (Alphabetical, Equal Contribution) Pierre Gleize Affiliation: Meta Superintelligence Labs Core Contributor (Alphabetical, Equal Contribution) Kevin J Liang Affiliation: Meta Superintelligence Labs Core Contributor (Alphabetical, Equal Contribution) Alexander Sax Affiliation: Meta Superintelligence Labs Core Contributor (Alphabetical, Equal Contribution) Hao Tang Affiliation: Meta Superintelligence Labs Core Contributor (Alphabetical, Equal Contribution) Weiyao Wang Affiliation: Meta Superintelligence Labs Core Contributor (Alphabetical, Equal Contribution) Michelle Guo Affiliation: Meta Superintelligence Labs Thibaut Hardin Affiliation: Meta Superintelligence Labs Xiang Li Affiliation: Meta Superintelligence Labs Intern Aohan Lin Affiliation: Meta Superintelligence Labs Jiawei Liu Affiliation: Meta Superintelligence Labs Ziqi Ma Affiliation: Meta Superintelligence Labs Intern Anushka Sagar Affiliation: Meta Superintelligence Labs Bowen Song Affiliation: Meta Superintelligence Labs Intern Xiaodong Wang Affiliation: Meta Superintelligence Labs Jianing Yang Affiliation: Meta Superintelligence Labs Intern Bowen Zhang Affiliation: Meta Superintelligence Labs Intern Piotr Dollár Affiliation: Meta Superintelligence Labs Project Lead Georgia Gkioxari Affiliation: Meta Superintelligence Labs Project Lead Matt Feiszli Affiliation: Meta Superintelligence Labs Jitendra Malik Affiliation: Meta Superintelligence Labs

Abstract

摘要

5:1

5:1

Figure 1: SAM 3D converts a single image into a composable 3D scene made of individual objects. Our method predicts per-object geometry, texture, and layout, enabling full scene reconstruction. Bottom: high-quality 3D assets recovered for each object.

图 1： SAM 3D 把单张图像转换成由各个物体组成、可组合的三维场景 。 我们的方法预测每个物体的几何、纹理和布局，从而实现完整的场景重建。底部：为每个物体恢复出的高质量三维资产。

1 Introduction

1 引言

In this paper (see Figure 1) we present SAM 3D, a generative neural network for 3D reconstruction from a single image. The model can reconstruct 3D shape and texture for any object, as well as its layout with respect to the camera, even in complex scenes with significant clutter and occlusion. As the reconstruction is of full 3D shape, not just of the visible 2.5D surface, one can then re-render the object from any desired viewpoint.

在本文中（见图 1），我们提出 SAM 3D，这是一个用于从单张图像做三维重建的生成式神经网络。该模型可以为任意物体重建三维形状和纹理，以及它相对于相机的布局，即便是在杂乱和遮挡都很严重的复杂场景中也是如此。由于重建的是完整的三维形状，而不只是可见的 2.5D 表面，随后可以从任意指定视点重新渲染该物体。

Computer vision has traditionally focused on multi-view geometry as providing the primary signal for 3D shape. However psychologists (and artists before them) have long known that humans can perceive depth and shape from a single image, e.g . Koenderink et al. (1992) demonstrated this elegantly by showing that humans can estimate surface normals at probe points on an object’s image, which can then be integrated to a full surface. In psychology textbooks these single image cues to 3D shape are called “pictorial cues”, and include information such as in shading and texture patterns, but also recognition - the “familiar object” cue. In computer vision, this line of research dates back to Roberts (1963), who showed that once an image pattern was recognized as a known object, its 3D shape and pose could be recovered. The central insight is that recognition enables 3D reconstruction, an idea that has since resurfaced in different technical instantiations (Debevec et al., 2023; Cashman and Fitzgibbon, 2012; Kar et al., 2015; Gkioxari et al., 2019; Xiang et al., 2025; Wu et al., 2024a). Note that this permits generalization to novel objects, because even if a specific object has not been seen before, it is made up of parts seen before.

计算机视觉传统上把多视图几何当作三维形状的主要信号。不过心理学家（以及更早的艺术家）早就知道， 人 可以从单张图像感知深度和形状。例如，Koenderink et al. (1992) 漂亮地证明：人能够估计物体图像上探测点处的表面法向，再把这些法向积分成完整表面。在心理学教科书里，这些来自单张图像的三维形状线索被称为“图画线索”（pictorial cues），既包括阴影和纹理模式中的信息，也包括识别，也就是“熟悉物体”线索。在计算机视觉中，这条研究线索可以追溯到 Roberts (1963)：他表明，一旦某个图像模式被识别为已知物体，就可以恢复它的三维形状和姿态。核心洞见是：识别使三维重建成为可能。此后这一想法以不同的技术形式反复出现（Debevec et al., 2023；Cashman and Fitzgibbon, 2012；Kar et al., 2015；Gkioxari et al., 2019；Xiang et al., 2025；Wu et al., 2024a）。需要指出的是，这允许推广到未见过的新物体，因为即便某个具体物体以前没见过，它也是由见过的部件组成的。

A fundamental challenge for learning such models is the lack of data: specifically, natural images paired with 3D ground truth are difficult to obtain at scale. Recent work (Yang et al., 2024b; Xiang et al., 2025) has shown strong reconstruction from single images. However, these models are trained on isolated objects and struggle with objects in natural scenes, where they may be distant or heavily occluded. To add such images to the training set, we need to find a way to associate specific objects in such images with 3D shape models, acknowledging that generalist human annotators find it hard to do so (unlike, say, attaching a label like “cat” or marking its boundary). Two insights made this possible:

学习这类模型的一个根本挑战是缺少数据：具体而言，与三维真值成对的自然图像很难大规模获得。近期工作（Yang et al., 2024b；Xiang et al., 2025）已经展示了从单张图像出发的较强重建。不过，这些模型是在孤立物体上训练的，面对自然场景中的物体往往会吃力，因为它们可能距离很远，或被严重遮挡。要把这类图像加入训练集，我们需要找到一种办法，把图像中的特定物体与三维形状模型对应起来，同时也要承认：非专业的人类标注员很难完成这件事（这和贴上“猫”这样的标签、或标出它的边界不同）。两点认识使这件事成为可能：

• We can create synthetic scenes where 3D object models are rendered and pasted into images (inspired by Dosovitskiy et al. (2015)).

• 我们可以创建合成场景：把三维物体模型渲染后贴进图像（受 Dosovitskiy et al. (2015) 启发）。

• While humans can’t easily generate 3D shape models for objects, they can select the likely best 3D model from a set of proffered choices and align its pose to the image (or declare that none of the choices is good).

• 人虽然不容易直接 生成 物体的三维形状模型，但可以从一组给定候选中 选出 很可能最好的三维模型，并把它的姿态对齐到图像上（或者声明这些候选里没有合适的）。

We design a training pipeline and data engine by adapting modern, multistage training recipes pioneered by LLMs (Minaee et al., 2025; Mo et al., 2025). As in recent works, we first train on a large collection of rendered synthetic objects. This is supervised pretraining: our model learns a rich vocabulary for object shape and texture, preparing it for real-world reconstruction. Next is mid-training with semi-synthetic data produced by pasting rendered models into natural images. Finally, post-training adapts the model to real images, using both a novel model-in-the-loop (MITL) pipeline and human 3D artists, and aligns it to human preference. We find that synthetic pretraining generalizes, given adequate post-training on natural images.

我们通过改造由大语言模型开创的现代多阶段训练配方（Minaee et al., 2025；Mo et al., 2025），设计了训练流程和数据引擎。与近期工作一样，我们首先在大量渲染的合成物体上训练。这是有监督预训练：模型学到一套丰富的物体形状与纹理词汇，为真实世界重建做好准备。接下来是中期训练，使用把渲染模型贴进自然图像后得到的半合成数据。最后，后训练把模型适配到真实图像上，同时使用一套新的模型在环（model-in-the-loop，MITL）流程和人类三维艺术家，并把它对齐到人的偏好。我们发现，只要在自然图像上有充分的后训练，合成预训练就可以泛化。

Our post-training data, obtained from our MITL data pipeline, is key to obtaining good performance in natural images. Generalist human annotators aren’t capable of producing 3D shape ground truth; hence our annotators select and align 3D models to objects in images from the output of modules - computational and retrieval-based - that produce multiple initial 3D shape proposals. Human annotators select from these proposals, or route them to human artists for a subset of hard instances. The vetted annotations feed back into model training, and the improved model is reintegrated into the data engine to further boost annotation quality. This virtuous cycle steadily improves the quality of 3D annotations, labeling rates, and model performance.

我们从 MITL 数据流程中得到的后训练数据，是在自然图像上取得良好性能的关键。非专业的人类标注员无法直接给出三维形状真值；因此，我们的标注员是从若干模块（计算式和检索式）产生的多个初始三维形状提案中，选择并对齐到图像里的物体。标注员从这些提案中选择，或把其中一部分困难实例转给人类艺术家。经过核验的标注再反馈到模型训练中，改进后的模型再被重新接入数据引擎，以进一步提高标注质量。这一良性循环会稳步提升三维标注的质量、标注速度和模型性能。

1,000

1,000

We summarize our contributions as follows:

我们将贡献概括如下：

• We introduce SAM 3D, a new foundation model for 3D that predicts object shape, texture, and pose from a single image. By releasing code, model weights, and a demo, we hope to stimulate further advancements in 3D reconstruction and downstream applications of 3D.

• 我们提出 SAM 3D ，一个新的三维基础模型，能够从单张图像预测物体的形状、纹理和姿态。通过发布代码、模型权重和演示，我们希望推动三维重建及其下游应用的进一步发展。

• We build a MITL pipeline for annotating shape, texture, and pose data, providing visually grounded 3D reconstruction data at unprecedented scale.

• 我们构建了一套用于标注形状、纹理和姿态数据的 MITL 流程，从而在前所未有的规模上提供视觉锚定的三维重建数据。

• We exploit this data via LLM-style pretraining and post-training in a novel framework for 3D reconstruction, combining synthetic pretraining with real-world alignment to overcome the orders of magnitude data gap between 3D and domains such as text, images, or video.

• 我们在一套新的三维重建框架中，以大语言模型式的预训练和后训练来利用这些数据，把合成预训练与真实世界对齐结合起来，以克服三维与文本、图像或视频等领域之间数量级的数据差距。

• We release a challenging benchmark for real-world 3D object reconstruction, SA-3DAO. Experiments show SAM 3D’s significant gains via metrics and large-scale human preference.

• 我们发布一份有挑战性的真实世界三维物体重建基准 SA-3DAO。实验通过指标和大规模人人偏好表明，SAM 3D 取得了显著提升。

2 The SAM 3D Model

2 SAM 3D 模型

Figure 2: SAM 3D architecture. (top) SAM 3D first predicts coarse shape and layout with the Geometry model; (right) the mixture of transformers architecture apply a two-stream approach with information sharing in the multi-modal self-attention layer. (bottom) The voxels predicted by the Geometry model are passed to the Texture & Refinement model, which adds higher resolution detail and textures.

图 2： SAM 3D 的结构 。（ 上 ）SAM 3D 首先用几何模型预测粗形状和布局；（ 右 ）混合 Transformer 结构采用双流方式，并在多模态自注意力层中共享信息。（ 下 ）几何模型预测的体素被送入纹理与细化模型，以加入更高分辨率的细节和纹理。

2.1 Problem Formulation

2.1 问题形式化

M

I

2.2 Architecture

2.2 结构

We build upon recent SOTA two-stage latent flow matching architectures (Xiang et al., 2025). SAM 3D first jointly predicts object pose and coarse shape, then refines the shapes by integrating pictorial cues (see Figure 2). Unlike Xiang et al. (2025) that reconstructs isolated objects, SAM 3D predicts object layout, creating coherent multi-object scenes.

我们建立在近期最先进的两阶段潜在流匹配结构之上（Xiang et al., 2025）。SAM 3D 首先联合预测物体姿态和粗形状，再通过整合图画线索来细化形状（见图 2）。与重建孤立物体的 Xiang et al. (2025) 不同，SAM 3D 还预测物体布局，从而生成协调的多物体场景。

Input encoding. We use DINOv2 (Oquab et al., 2023) as an encoder to extract features from two pairs of images, resulting in 4 sets of conditioning tokens:

输入编码。 我们用 DINOv2（Oquab et al., 2023）作为编码器，从两对图像中提取特征，得到 4 组条件词元：

I

M

I

I

P

P

Figure 3: SAM 3D data, with a green outline around the target object, and the ground truth mesh shown in the bottom right. Samples are divided into four rows, based on type. Art-3DO meshes are untextured, while the rest may be textured or not, depending on the underlying asset (Iso-3DO, RP-3DO) or if the mesh was annotated for texture (MITL-3DO).

图 3： SAM 3D 的数据 ，目标物体用绿色轮廓标出，右下角为真值网格。样本按类型分成四行。Art-3DO 的网格没有纹理，其余则可能有纹理或没有纹理，取决于底层资产（Iso-3DO、RP-3DO），或该网格是否被标注了纹理（MITL-3DO）。

p(O,R,t,s|I,M)

p(O,R,t,s|I,M)

p(S,T|I,M,O)

p(S,T|I,M,O)

\mathcal{D}_{m}

\mathcal{D}_{m}

3 Training SAM 3D

3 训练 SAM 3D

SAM 3D breaks the 3D data barrier using a recipe that progresses from synthetic pretraining to natural post-training, adapting the playbook from LLMs, robotics, and other large generative models. We build capabilities by stacking different training strategies in pre- and mid-training, and then align the model to real data and human-preferred behaviors through a post-training data flywheel. SAM 3D uses the following approach:

SAM 3D 用一套从合成预训练推进到自然图像后训练的配方来打破三维数据壁垒，改造了来自大语言模型、机器人以及其他大规模生成模型的方法手册。我们在预训练和中期训练中叠放不同的训练策略以积累能力，再通过后训练的数据飞轮，把模型对齐到真实数据和人所偏好的行为。SAM 3D 采用如下做法：

Step 1: Pretraining. This phase builds foundational capabilities, such as shape generation, into a base model.

第 1 步：预训练 。 这一阶段把形状生成等基础能力写入一个基座模型。

Step 1.5: Mid-Training. Sometimes called continued pretraining, mid-training imparts general skills such as occlusion robustness, mask-following, and using visual cues.

第 1.5 步：中期训练 。 有时也称为继续预训练，中期训练赋予遮挡鲁棒性、掩码跟随以及使用视觉线索等通用技能。

(I,M)\rightarrow(S,T,R,t,s)

(I,M)\rightarrow(S,T,R,t,s)

This alignment (step 2) can be repeated, first collecting data with the current model and then improving the model with the new data. This creates a virtuous cycle with humans providing the supervision. Figure 12 shows that as we run the data engine longer, model performance steadily improves; dataset generation emerges as a byproduct of this alignment.

这种对齐（第 2 步）可以反复进行：先用当前模型收集数据，再用新数据改进模型。这就形成了一个由人提供监督的良性循环。图 12 表明，数据引擎运行得越久，模型性能就越稳步提高；数据集的生成则是这种对齐的副产品。

The following sections detail the training objectives and data sources used in SAM 3D. We focus on the Geometry model; Texture & Refinement is trained similarly (details in Section C.5). Training hyper-parameters are in Section C.7.

以下几节详述 SAM 3D 所用的训练目标和数据来源。我们把重点放在几何模型上；纹理与细化模型的训练方式类似（细节见第 C.5 节）。训练超参数见第 C.7 节。

Figure 4: SAM 3D training paradigm. We employ a multi-stage pipeline incrementally exposing the model to increasingly complex data and modalities.

图 4： SAM 3D 的训练范式 。 我们采用多阶段流程，逐步让模型接触越来越复杂的数据和模态。

S,R

Training stage	Modalities	Datasets	Condition input
Stage 1 Geometry model
Pre-training	$S,R$	Iso-3DO	object-centric crop
Mid-training	$S,R$	RP-3DO†	full image
$S,R,t,s$	ProcThor, RP-3DO‡	full image, pointmap∗
SFT	$S,R,t,s$	MITL, Art-3DO	full image, pointmap∗
Alignment	$S$	MITL preference	full image, pointmap∗
Stage 2 Texture & Refinement model
Pre-training	$T$	Iso-3DO-500K	object-centric crop
Mid-training	$T$	RP-3DO†,§	full image
SFT	$T$	MITL	full image
Alignment	$T$	MITL preference	full image

S,R

训练阶段	模态	数据集	条件输入
第 1 阶段几何模型
预训练	$S,R$	Iso-3DO	以物体为中心的裁剪
中期训练	$S,R$	RP-3DO†	整图
	$S,R,t,s$	ProcThor, RP-3DO‡	整图，点图∗
SFT	$S,R,t,s$	MITL, Art-3DO	整图，点图∗
对齐	$S$	MITL 偏好	整图，点图∗
第 2 阶段纹理与细化模型
预训练	$T$	Iso-3DO-500K	以物体为中心的裁剪
中期训练	$T$	RP-3DO†,§	整图
SFT	$T$	MITL	整图
对齐	$T$	MITL 偏好	整图

3.1 Pre & Mid-Training: Building a Base Model

3.1 预训练与中期训练：构建基座模型

Training begins with synthetic pretraining and mid-training, leveraging available large-scale datasets to learn strong priors for shape and texture, and skills such as mask-following, occlusion handling, and pose estimation. The rich features learned here drastically reduce the number of labeled real-world samples required in post-training (Hernandez et al., 2021), which generally incur acquisition costs. In pre- and mid-training, models are trained with rectified conditional flow matching (Liu et al., 2022) to generate multiple 3D modalities (see Section C.2).

训练从合成预训练和中期训练开始，利用已有的大规模数据集来学习形状和纹理的强先验，以及掩码跟随、遮挡处理和姿态估计等技能。这里学到的丰富特征会大幅减少后训练所需的带标注真实世界样本数量（Hernandez et al., 2021），而这类样本通常获取成本较高。在预训练和中期训练中，模型用矫正条件流匹配（rectified conditional flow matching）（Liu et al., 2022）来生成多种三维模态（见第 C.2 节）。

I

I

3.1.2 Mid-Training: Semi-Synthetic Capabilities Next, mid-training builds up foundational skills that will enable the model to handle objects in real-world images:

3.1.2 中期训练：半合成能力 接下来，中期训练积累基础技能，使模型能够处理真实世界图像中的物体：

• Mask-following : We train the model to reconstruct a target object, defined by a binary mask on the input image.

• 掩码跟随 ：我们训练模型重建由输入图像上的二值掩码所定义的目标物体。

• Occlusion robustness : The artificial occluders in our dataset incentivize learning shape completion.

• 遮挡鲁棒性 ：数据集中的人工遮挡物会激励模型学习形状补全。

• Layout estimation : We train the model to produce translation and scale in normalized camera coordinates.

• 布局估计 ：我们训练模型在归一化相机坐标中给出平移和尺度。

61

6100

After mid-training (2.7 trillion training tokens), the model has now been trained with all input and output modalities for visually grounded 3D reconstruction. However, all data used has been (semi-)synthetic; to both close the domain gap and fully leverage real-world cues, we need real images.

中期训练（2.7 万亿训练词元）之后，模型已经在视觉锚定三维重建的全部输入和输出模态上受过训练。不过，所用数据都是（半）合成的；要缩小域差距并充分利用真实世界线索，我们还需要真实图像。

Figure 5: Life of an example going through the data collection pipeline. We streamline annotation by breaking it into subtasks: annotators first choose target objects (Stage 1); rank and select 3D model candidates (Stage 2); then pose these models within a 2.5D scene (Stage 3). Stages 2 and 3 use model-in-the-loop.

图 5： 一个样本走过数据收集流程的一生 。 我们把标注拆成子任务来简化流程：标注员先选择目标物体（第 1 阶段）；再对三维模型候选排序并选择（第 2 阶段）；然后把这些模型摆放到 2.5D 场景中（第 3 阶段）。第 2、第 3 阶段使用模型在环。

3.2 Post-Training: Real-World Alignment

3.2 后训练：真实世界对齐

In post-training, we have two goals. The first is to close the domain gap between (semi-)synthetic data and natural images. The second is to align with human preference for shape quality. We adapt the model by using our data engine iteratively; we first (i) collect training data with the current model, and then (ii) update our model using multi-stage post-training on this collected data. We then repeat.

在后训练中，我们有两个目标。第一是缩小（半）合成数据与自然图像之间的域差距。第二是对齐人对形状质量的偏好。我们用数据引擎迭代地适配模型：先 （i）用当前模型收集训练数据 ，再 （ii）在这些收集到的数据上通过多阶段后训练更新模型 。然后重复这一过程。

3.2.1 Post-Training: Collection Step The core challenge with collecting data for 3D visual grounding is that most people cannot create meshes directly; this requires skilled 3D artists, who even then can take multiple hours. This is different from the segmentation masks collected in SAM (Kirillov et al., 2023). However, given options, most people can choose which mesh resembles an object in the image most accurately. This fact forms the foundation of our data collection for SAM 3D. We convert preferences into training data as follows: sample from our post-trained model, ask annotators to choose the best candidate and then grade its overall quality according to a rubric which we define and update. If the quality meets the (evolving) bar, the candidate becomes a training sample.

3.2.1 后训练：收集步骤 为三维视觉锚定收集数据的核心困难是：大多数人无法直接创建网格；这需要熟练的三维艺术家，即便如此也可能花费数小时。这与 SAM（Kirillov et al., 2023）中收集的分割掩码不同。不过，在给出选项时，大多数人 能够 选出哪一个网格与图像中的物体最相像。这一事实构成了我们为 SAM 3D 收集数据的基础。我们把偏好转成训练数据的做法如下：从经过后训练的模型中采样，请标注员选出最好的候选，再按我们定义并更新的细则给总体质量打分。如果质量达到（不断演进的）门槛，该候选就成为训练样本。

80\%

80\%

S

S

(I,M)

(I,M)

Figure 6: Qualitative comparison to competing image-to-3D asset methods. We compare to the recent Trellis (Xiang et al., 2025), Hunyuan3D-2.1 (Hunyuan3D et al., 2025), Direct3D-S2 (Wu et al., 2025a) and Hi3DGen (Ye et al., 2025) on the artist-generated SA-3DAO for single shape reconstruction; we provide the 3D artist-created ground truth mesh as reference.

图 6： 与相互竞争的图像到三维资产方法的定性比较 。 我们在艺术家生成的 SA-3DAO 上，与近期的 Trellis（Xiang et al., 2025）、Hunyuan3D-2.1（Hunyuan3D et al., 2025）、Direct3D-S2（Wu et al., 2025a）和 Hi3DGen（Ye et al., 2025）比较单形状重建；并提供三维艺术家创建的真值网格作为参考。

(S,T)

(S,T)

r>\alpha

N=8

Stage 2.5: Hard example triage (Artists). When no model produces a reasonable object shape, our non-specialist annotators cannot correct the meshes, resulting in a lack of data precisely where the model needs it most. We route a small percentage of these hardest cases to professional 3D artists for direct annotation, and we denote this set Art-3DO .

第 2.5 阶段：困难样本分流（艺术家）。 当没有任何模型能给出合理的物体形状时，非专业标注员也无法修正这些网格，于是恰恰在模型最需要数据的地方出现了缺口。我们把一小部分最困难的情形转给专业三维艺术家直接标注，并把这批数据记为 Art-3DO 。

Figure 7: Qualitative comparison to competing scene reconstruction methods. We show SAM 3D’s full 3D scene reconstructions versus alternatives (Wen et al., 2024; Huang et al., 2025).

图 7： 与相互竞争的场景重建方法的定性比较 。 我们展示 SAM 3D 的完整三维场景重建，并与其他方法对照（Wen et al., 2024；Huang et al., 2025）。

(R,t,s)

(R,t,s)

q(S,T,R,t,s~|~I,M)

q(S,T,R,t,s~|~I,M)

D^{+}

D^{+}

Supervised Fine-Tuning (SFT). When post-training begins, the base model has only seen synthetic data. Due to the large domain gap between synthetic and real-world data, we begin by finetuning on our aligned meshes from Stage 3.

监督微调（SFT）。 后训练开始时，基座模型只见过合成数据。由于合成数据与真实世界数据之间的域差距很大，我们首先在来自第 3 阶段的已对齐网格上微调。

We begin SFT with the noisier non-expert labels (MITL-3DO), followed by the smaller, high-quality set from 3D artists (Art-3DO). The high quality Art-3DO data enhances model quality by aligning outputs with artists’ aesthetic preferences. We find this helps suppress common failures, e.g . floaters, bottomless meshes, and missing symmetry.

我们先从噪声较大的非专家标注（MITL-3DO）开始做 SFT，再接着使用来自三维艺术家的较小但高质量集合（Art-3DO）。高质量的 Art-3DO 数据通过把输出对齐到艺术家的审美偏好来提升模型质量。我们发现这有助于抑制常见失败，例如漂浮物、没有底部的网格，以及缺失的对称性。

D^{+}

D^{+}

25\rightarrow 4

25\rightarrow 4

4 Experiments

4 实验

\uparrow

	SA-3DAO	ISO3D Eval Set
Model	F1@0.01 ( $\uparrow$ )	vIoU ( $\uparrow$ )	Chamfer ( $\downarrow$ )	EMD ( $\downarrow$ )	ULIP ( $\uparrow$ )	Uni3D ( $\uparrow$ )
Trellis	0.1475	0.1392	0.0902	0.2131	0.1473	0.3698
HY3D-2.1	0.1399	0.1266	0.1126	0.2432	0.1293	0.3546
HY3D-2.0	0.1574	0.1504	0.0866	0.2049	0.1484	0.3662
Direct3D-S2	0.1513	0.1465	0.0962	0.2160	0.1405	0.3653
TripoSG	0.1533	0.1445	0.0844	0.2057	0.1529	0.3687
Hi3DGen	0.1629	0.1531	0.0937	0.2134	0.1419	0.3594
SAM 3D	0.2344	0.2311	0.0400	0.1211	0.1488	0.3707

\uparrow

	SA-3DAO				ISO3D 评测集
模型	F1@0.01 ( $\uparrow$ )	vIoU ( $\uparrow$ )	Chamfer ( $\downarrow$ )	EMD ( $\downarrow$ )	ULIP ( $\uparrow$ )	Uni3D ( $\uparrow$ )
Trellis	0.1475	0.1392	0.0902	0.2131	0.1473	0.3698
HY3D-2.1	0.1399	0.1266	0.1126	0.2432	0.1293	0.3546
HY3D-2.0	0.1574	0.1504	0.0866	0.2049	0.1484	0.3662
Direct3D-S2	0.1513	0.1465	0.0962	0.2160	0.1405	0.3653
TripoSG	0.1533	0.1445	0.0844	0.2057	0.1529	0.3687
Hi3DGen	0.1629	0.1531	0.0937	0.2134	0.1419	0.3594
SAM 3D	0.2344	0.2311	0.0400	0.1211	0.1488	0.3707

Figure 8: Preference comparison on scene-level and object-level reconstruction. Numbers indicate human preference rates. Objects comparisons are done on textured meshes. SAM 3D is significantly preferred over others on all fronts.

图 8： 场景级和物体级重建上的偏好比较 。 数字表示人人偏好率。物体比较在带纹理的网格上进行。SAM 3D 在所有方面都明显更受偏好。

Dataset. To comprehensively evaluate the model capability under real-world scenarios, we carefully build a new benchmark SA-3DAO[^1], consisting of 1K 3D artist-created meshes created from natural images. We also include ISO3D from 3D Arena (Ebert, 2025) for quantitatively evaluating shape and texture, and Aria Digital Twin (ADT) (Pan et al., 2023) for layout. We further conduct human preference evaluation on two curated sets for both scene-level and object-level reconstruction. The Pref Set uses real-world images from MetaCLIP (Xu et al., 2024a) and SA-1B (Kirillov et al., 2023), as well as a set based on LVIS (Gupta et al., 2019). Refer to Appendix D for details on evaluation sets.

数据集。 为了全面评估模型在真实世界场景下的能力，我们精心构建了新基准 SA-3DAO[^1] ，包含 1000 个由三维艺术家根据自然图像创建的网格。我们还纳入 3D Arena（Ebert, 2025）中的 ISO3D ，用于定量评估形状和纹理，以及 Aria Digital Twin（ ADT ）（Pan et al., 2023），用于评估布局。我们进一步在两份整理好的集合上，分别对场景级和物体级重建做人人偏好评估。 Pref Set 使用来自 MetaCLIP（Xu et al., 2024a）和 SA-1B（Kirillov et al., 2023）的真实世界图像，以及一份基于 LVIS（Gupta et al., 2019）的集合。评测集的细节见附录 D。

Settings. We conduct experiments with a Geometry model trained to condition on pointmaps. For datasets where pointmaps are unavailable, we estimate them with Wang et al. (2025a). We found that shape and texture quality do not depend on whether the model is trained with pointmap conditioning (see Section E.5), but layout (translation/scale) evaluation in Table 3 requires ground-truth depth/pointmap as reference.

设置。 我们用一个被训练为以点图为条件的几何模型做实验。对于没有点图的数据集，我们用 Wang et al. (2025a) 来估计点图。我们发现，形状和纹理质量并不依赖于模型是否在点图条件下训练（见第 E.5 节），但表 3 中的布局（平移/尺度）评测需要真值深度/点图作为参考。

4.1 Comparison with SOTA

4.1 与当时最好方法的比较

\uparrow

		SA-3DAO	Aria Digital Twin
Generation	Model	3D IoU ( $\uparrow$ )	ICP-Rot. ( $\downarrow$ )	ADD-S ( $\downarrow$ )	ADD-S @ 0.1 ( $\uparrow$ )	3D IoU ( $\uparrow$ )	ICP-Rot. ( $\downarrow$ )	ADD-S ( $\downarrow$ )	ADD-S @ 0.1 ( $\uparrow$ )
Pipeline	Trellis + Megapose	0.2449	39.3866	0.5391	0.2831	0.2531	33.6114	0.4358	0.1971
Pipeline	HY3D-2.0 + Megapose	0.2518	33.8307	0.7146	0.3647	0.3794	29.0066	0.1457	0.4211
Pipeline	HY3D-2.0 + FoundationPose	0.2937	32.9444	0.3705	0.5396	0.3864	25.1435	0.1026	0.5992
Pipeline	HY3D-2.1 + FoundationPose	0.2395	39.8357	0.4186	0.4177	0.2795	33.1197	0.2135	0.4129
Pipeline	SAM 3D + FoundationPose	0.2837	32.9168	0.3848	0.5079	0.3661	18.9102	0.0930	0.6495
Joint	MIDI	-	-	-	-	0.0336	44.2353	2.5278	0.0175
Joint	SAM 3D	0.4254	20.7667	0.2661	0.7232	0.4970	15.2515	0.0765	0.7673

\uparrow

		SA-3DAO				Aria Digital Twin
生成方式	模型	3D IoU ( $\uparrow$ )	ICP-Rot. ( $\downarrow$ )	ADD-S ( $\downarrow$ )	ADD-S @ 0.1 ( $\uparrow$ )	3D IoU ( $\uparrow$ )	ICP-Rot. ( $\downarrow$ )	ADD-S ( $\downarrow$ )	ADD-S @ 0.1 ( $\uparrow$ )
流水线	Trellis + Megapose	0.2449	39.3866	0.5391	0.2831	0.2531	33.6114	0.4358	0.1971
流水线	HY3D-2.0 + Megapose	0.2518	33.8307	0.7146	0.3647	0.3794	29.0066	0.1457	0.4211
流水线	HY3D-2.0 + FoundationPose	0.2937	32.9444	0.3705	0.5396	0.3864	25.1435	0.1026	0.5992
流水线	HY3D-2.1 + FoundationPose	0.2395	39.8357	0.4186	0.4177	0.2795	33.1197	0.2135	0.4129
流水线	SAM 3D + FoundationPose	0.2837	32.9168	0.3848	0.5079	0.3661	18.9102	0.0930	0.6495
联合	MIDI	-	-	-	-	0.0336	44.2353	2.5278	0.0175
联合	SAM 3D	0.4254	20.7667	0.2661	0.7232	0.4970	15.2515	0.0765	0.7673

5:1

5:1

6:1

6:1

Figure 9: Preference comparison on texture . Since SAM 3D provides higher quality shape, we use the geometry results from SAM 3D and only perform texture generations for all methods. SAM 3D significantly outperforms others.

图 9： 纹理上的偏好比较 。由于 SAM 3D 提供了更高质量的形状，我们对所有方法都使用 SAM 3D 的几何结果，只比较纹理生成。SAM 3D 明显优于其他方法。

4.2 Analysis Studies

4.2 分析研究

Post-training iterations steadily improve performance. We observed steady improvements as we ran the data engine for longer, with near-linear Elo scaling shown in the historical comparisons from Stage 2 of our data engine (Figure 12). We found it important to scale all stages simultaneously. The cumulatively linear effect results from more data engine iterations, along with scaling up pretraining, mid-training, and adding new post-training stages. Figure 12 shows that iterating MITL-3DO data alone yields consistent improvements but with decreasing marginal impact.

后训练迭代会稳步提升性能。 我们观察到，数据引擎运行得越久，性能就越稳步提高；数据引擎第 2 阶段的历史比较呈现出近乎线性的 Elo 扩展（图 12）。我们发现，同时扩大所有阶段很重要。这种累积的近线性效果，来自更多的数据引擎迭代，以及扩大预训练、中期训练，并加入新的后训练阶段。图 12 还表明，仅迭代 MITL-3DO 数据也会带来持续改进，但边际影响在下降。

\uparrow

	SA-3DAO	Preference set
Training Stage	F1 @ 0.01 ( $\uparrow$ )	vIoU ( $\uparrow$ )	Chamfer ( $\downarrow$ )	EMD ( $\downarrow$ )	Texture WR ( $\uparrow$ )
Pre-training (Iso-3DO)	0.1349	0.1202	0.1036	0.2396	-
+ Mid-training (RP-3DO)	0.1705	0.1683	0.0760	0.1821	60.7
+ SFT (MITL-3DO)	0.2027	0.2025	0.0578	0.1510	66.9
+ DPO (MITL-3DO)	0.2156	0.2156	0.0498	0.1367	66.4
+ SFT (Art-3DO)	0.2331	0.2337	0.0445	0.1257	-
+ DPO (Art-3DO)	0.2344	0.2311	0.0400	0.1211	-

\uparrow

	SA-3DAO				偏好集
训练阶段	F1 @ 0.01 ( $\uparrow$ )	vIoU ( $\uparrow$ )	Chamfer ( $\downarrow$ )	EMD ( $\downarrow$ )	纹理胜率 ( $\uparrow$ )
预训练（Iso-3DO）	0.1349	0.1202	0.1036	0.2396	-
+ 中期训练（RP-3DO）	0.1705	0.1683	0.0760	0.1821	60.7
+ SFT（MITL-3DO）	0.2027	0.2025	0.0578	0.1510	66.9
+ DPO（MITL-3DO）	0.2156	0.2156	0.0498	0.1367	66.4
+ SFT（Art-3DO）	0.2331	0.2337	0.0445	0.1257	-
+ DPO（Art-3DO）	0.2344	0.2311	0.0400	0.1211	-

Figure 10: Historical Elo from data engine Figure 11: Impact of expanding training data Figure 12: Data engine with additional iterations. The plots show Elo scores of different models; a 400 point Elo difference corresponds to 10:1 odds in a preference test. Models were (a) checkpoints around 3 weeks apart, indicating cumulative improvements as we scale and add different stages and (b) post-trained (SFT) using expanded training data.

图 10：数据引擎的历史 Elo 图 11：扩大训练数据的影响 图 12： 带有额外迭代的数据引擎 。 图中给出不同模型的 Elo 分数；400 分的 Elo 差距对应于偏好测试中 10:1 的胜算。这些模型分别是 （a） 大约每隔 3 周的检查点，表明随着我们扩大规模并加入不同阶段，改进是累积的；以及 （b） 使用扩大后的训练数据做后训练（SFT）。

Multi-stage training improves performance. SAM 3D’s real-world performance emerges through multi-stage training. Table 4 reveals near-monotonic 3D shape improvements as each training stage is added, validating the approach that leads to the final model (last row). In the appendix, Figure 19 shows similar results for texture and Table 7 shows the contribution of each individual real-world data stage, by knocking out the MITL-3DO, Art-3DO data, or DPO stages.

多阶段训练会提升性能。 SAM 3D 的真实世界性能是通过多阶段训练涌现出来的。表 4 显示，每增加一个训练阶段，三维形状几乎都单调改进，从而验证了通向最终模型（最后一行）的这条路径。在附录中，图 19 给出了纹理上的类似结果，表 7 则通过去掉 MITL-3DO、Art-3DO 数据或 DPO 阶段，展示各个真实世界数据阶段各自的贡献。

N

N

5 Related Work

5 相关工作

3D reconstruction has been a longstanding challenge in computer vision. Classical methods leverage multiple views and include binocular stereopsis (Wheatstone, 1838), structure-from-motion (Hartley and Zisserman, 2003; Szeliski, 2022; Scharstein and Szeliski, 2002; Torresani et al., 2008; Tomasi and Kanade, 1992), and SLAM (Smith et al., 1990; Castellanos et al., 1999). Other strategies reconstruct by analysis (e.g., silhouettes (Esteban and Schmitt, 2004)) or by synthesis via volume rendering (Kajiya and Von Herzen, 1984), using either implicit representations (Mildenhall et al., 2020) or explicit ones (Sitzmann et al., 2019; Liu et al., 2020). Supervised deep learning methods predict voxels (Xie et al., 2019; Wang et al., 2021), point clouds (Van Hoorick et al., 2022), or meshes (Worchel et al., 2022; Wen et al., 2019), or optimize implicit representations (Liu et al., 2024), e.g., signed distance functions (SDFs), often with high-quality output but requiring multiple views at inference. In contrast, we focus on the more restrictive setting of a single RGB image at test time.

三维重建 一直是计算机视觉中的长期挑战。经典方法利用多个视图，包括双眼立体视觉（Wheatstone, 1838）、运动恢复结构（Hartley and Zisserman, 2003；Szeliski, 2022；Scharstein and Szeliski, 2002；Torresani et al., 2008；Tomasi and Kanade, 1992），以及 SLAM（Smith et al., 1990；Castellanos et al., 1999）。其他策略通过分析来重建（例如轮廓（Esteban and Schmitt, 2004）），或通过体积渲染以合成的方式重建（Kajiya and Von Herzen, 1984），既使用隐式表示（Mildenhall et al., 2020），也使用显式表示（Sitzmann et al., 2019；Liu et al., 2020）。有监督的深度学习方法预测体素（Xie et al., 2019；Wang et al., 2021）、点云（Van Hoorick et al., 2022）或网格（Worchel et al., 2022；Wen et al., 2019），或优化隐式表示（Liu et al., 2024），例如有向距离函数（SDF）；它们常常能给出高质量输出，但在推理时需要多个视图。与之不同，我们关注的是测试时只有单张 RGB 图像这一更受限制的设定。

Single-view 3D reconstruction is considerably more difficult. A large body of work trains reconstruction models with direct 3D supervision, predicting meshes (Xu et al., 2019; Kulkarni et al., 2022), voxels (Girdhar et al., 2016; Wu et al., 2017; Popov et al., 2020), point clouds (Fan et al., 2017; Mescheder et al., 2019), or CAD-aligned geometry (Wang et al., 2018; Gkioxari et al., 2019). More recently, large internet corpora of 3D object assets have enabled a line of generative approaches trained using VAE (Kingma and Welling, 2013) latent representations (Zhang et al., 2023; Xiang et al., 2025; Ren et al., 2024). A related challenge is recovering the full shape of occluded objects from a partial mask (Xu et al., 2024b; Wu et al., 2025b); SAM 3D handles this through data rather than architecture. However, existing generative methods are typically evaluated on simplified synthetic single-object benchmarks such as ShapeNet (Chang et al., 2015), Pix3D (Sun et al., 2018), or Objaverse (Deitke et al., 2023).

单视图三维重建 要困难得多。大量工作用直接的三维监督来训练重建模型，预测网格（Xu et al., 2019；Kulkarni et al., 2022）、体素（Girdhar et al., 2016；Wu et al., 2017；Popov et al., 2020）、点云（Fan et al., 2017；Mescheder et al., 2019），或与 CAD 对齐的几何（Wang et al., 2018；Gkioxari et al., 2019）。最近，互联网上的大规模三维物体资产使得一类生成式方法成为可能，它们用 VAE（Kingma and Welling, 2013）的潜在表示来训练（Zhang et al., 2023；Xiang et al., 2025；Ren et al., 2024）。一个相关挑战是从部分掩码恢复被遮挡物体的完整形状（Xu et al., 2024b；Wu et al., 2025b）；SAM 3D 是通过数据而不是结构来处理这一点的。不过，已有生成方法通常是在简化的合成单物体基准上评测的，例如 ShapeNet（Chang et al., 2015）、Pix3D（Sun et al., 2018）或 Objaverse（Deitke et al., 2023）。

Layout estimation extends single-instance estimation to full scenes. Instead of predicting a single mesh for the entire scene, SAM 3D estimates scene layout by using mask prompts to indicate which objects to reconstruct, and then estimates both shape and pose for each mask instance. Approaches can be decomposed into so-called model-based or model-free methods. Model-based methods decompose the problem into independent parts, first estimating shape (or pulling it from a CAD database) and then posing the objects in the scene (Labbé et al., 2022; Wen et al., 2024; Maninis et al., 2022; Tyszkiewicz et al., 2022; Maninis et al., 2023; Rozumnyi et al., 2023). Alternatively, model-free methods estimate both shape and pose jointly, as in SAM 3D (Brazil et al., 2023; Huang et al., 2025; Yao et al., 2025; Ardelean et al., 2025; Geng et al., 2025; Shi et al., 2025). However, prior approaches are typically restricted to tabletop robotics, streets, or indoor scenes where objects rest on a supporting surface. In contrast, our approach estimates both shape and pose for a broad range of object types across diverse scenes.

布局估计 把单实例估计扩展到完整场景。SAM 3D 并不为整个场景预测一个网格，而是用 掩码提示 指明要重建哪些物体，再为每个掩码实例估计形状和姿态。这类方法可以分成所谓的基于模型的方法和无模型方法。基于模型的方法把问题拆成相互独立的部分：先估计形状（或从 CAD 库中取出），再把物体摆放到场景中（Labbé et al., 2022；Wen et al., 2024；Maninis et al., 2022；Tyszkiewicz et al., 2022；Maninis et al., 2023；Rozumnyi et al., 2023）。另一类无模型方法则像 SAM 3D 一样联合估计形状和姿态（Brazil et al., 2023；Huang et al., 2025；Yao et al., 2025；Ardelean et al., 2025；Geng et al., 2025；Shi et al., 2025）。不过，先前方法通常局限于桌面机器人、街道，或物体搁在支撑面上的室内场景。与之不同，我们的方法能在多样场景中，为更广泛的物体类型估计形状和姿态。

3D datasets. Sourcing 3D annotations is challenging: the modality itself is complex, and the specialized tools required are hard to master. Anecdotally, modeling a 3D mesh from a reference image can take an experienced artist hours (Appendix D). Instead, existing 3D datasets (e.g ., ShapeNet (Chang et al., 2015), Objaverse-XL (Deitke et al., 2023)) primarily consist of single synthetic objects; without paired real-world images, models can only learn from rendered views. In the real-world domain, existing datasets are small and mostly indoors (Reizenstein et al., 2021; Khanna et al., 2024; Fu et al., 2021; Szot et al., 2021; Pan et al., 2023), or procedurally generated (Dai et al., 2024; Raistrick et al., 2024; Deitke et al., 2022). Models trained on such constrained data struggle to generalize.

三维数据集。 获取三维标注很困难：这一模态本身就复杂，所需的专业工具也很难掌握。根据经验，有经验的艺术家根据参考图像建模一个三维网格也可能花费数小时（附录 D）。因此，已有三维数据集（例如 ShapeNet（Chang et al., 2015）、Objaverse-XL（Deitke et al., 2023））主要由单个合成物体组成；没有成对的真实世界图像，模型就只能从渲染视图中学习。在真实世界领域，已有数据集规模较小，且大多是室内的（Reizenstein et al., 2021；Khanna et al., 2024；Fu et al., 2021；Szot et al., 2021；Pan et al., 2023），或是程序生成的（Dai et al., 2024；Raistrick et al., 2024；Deitke et al., 2022）。在这类受限数据上训练的模型难以泛化。

\rightarrow

\rightarrow

Multi-stage pretraining. Modern pretraining increasingly employs multiple training stages. Early work on curriculum learning (Bengio et al., 2009) provided a basis for staged data mixing in pretraining, with higher-quality data coming later (Grattafiori and others, 2024; OLMo et al., 2025). Li et al. (2023b); Abdin and others (2024) show that mixing synthetic/web curricula can achieve strong performance at smaller scales. Increasingly, additional mid-training stages are used for capability injection, such as context extension (Grattafiori and others, 2024) or coding (Rozière et al., 2024), and recent work finds that mid-training significantly improves post-training effectiveness (Lambert, 2025; Wang et al., 2025b). SAM 3D introduces pretraining and mid-training that can generalize for 3D.

多阶段预训练。 现代预训练越来越多地采用多个训练阶段。关于课程学习的早期工作（Bengio et al., 2009）为预训练中的分阶段数据混合提供了基础，质量更高的数据会放在后面（Grattafiori and others, 2024；OLMo et al., 2025）。Li et al. (2023b)；Abdin and others (2024) 表明，混合合成/网络课程可以在较小规模上取得较强性能。越来越多的做法是再加中期训练阶段来注入能力，例如扩展上下文（Grattafiori and others, 2024）或编程（Rozière et al., 2024）；近期工作还发现，中期训练会显著提高后训练的效果（Lambert, 2025；Wang et al., 2025b）。SAM 3D 引入了能够为三维泛化的预训练和中期训练。

6 Conclusion

6 结论

We share SAM 3D: a new foundation model for full reconstruction of 3D shape, texture, and layout of objects from natural images. SAM 3D’s robustness on in-the-wild images, made possible by an innovative data engine and modern training recipe, represents a step change for 3D and an advance towards real-world 3D perception. With the release of our model, we expect SAM 3D to unlock new capabilities across diverse applications, such as robotics, AR/VR, gaming, film, and interactive media.

我们分享 SAM 3D：一个新的基础模型，用于从自然图像完整重建物体的三维形状、纹理和布局。凭借创新的数据引擎和现代训练配方，SAM 3D 在野外图像上的鲁棒性，代表着三维领域的一次跃迁，也是迈向真实世界三维感知的一步。随着模型的发布，我们期待 SAM 3D 能在机器人、AR/VR、游戏、影视和交互媒体等多样应用中解锁新的能力。

Acknowledgements

致谢

We thank the following individuals for their contributions to this work:

我们感谢下列人士对本工作的贡献：

For their contributions to SAM Playground Engineering we thank: Robbie Adkins, Rene de la Fuente, Facundo Figueroa, Alex He, Dex Honsa, Alex Lende, Jonny Li, Peter Park, Don Pinkus, Roman Radle, Phillip Thomas, and Meng Wang. We thank our excellent XFN team for leadership and support: Kris Kitani, Vivian Lee, Sasha Mitts, George Orlin, Nikhila Ravi, and Andrew Westbury. Thanks to Helen Klein, Mallika Malhotra, and Azita Shokrpour for support with Legal, Privacy, and Integrity. We thank Michelle Chan, Kei Koyama, William Ngan, Yael Yungster for all the design support throughout the project. Thanks to Arpit Kalla for work on model efficiency. We thank Faye Ma and Kehan Lyu for data engineering support and tooling, and Emmanuel Hernandez, Robert Kuo for pipeline development. We thank Nan Yang for support with egocentric video data efforts. Thanks to our two interns Cem Gokmen, Jasmine Shone for their work on 3D and Lea Wilken for feedback on the manuscript. Thanks to our fantastic data operations team: Paris Baptiste, Karen Bergan, Kai Brown, Ida Cheng, Khadijat Durojaiye, Patrick Edwards, Daniella Factor, Eva Galper, Leonna Jones, Zayida Suber, Tatum Turner, Joseph Walker, and Claudette Ward.

感谢为 SAM Playground 工程作出贡献的：Robbie Adkins、Rene de la Fuente、Facundo Figueroa、Alex He、Dex Honsa、Alex Lende、Jonny Li、Peter Park、Don Pinkus、Roman Radle、Phillip Thomas 和 Meng Wang。感谢出色的跨职能团队在领导和支持上的帮助：Kris Kitani、Vivian Lee、Sasha Mitts、George Orlin、Nikhila Ravi 和 Andrew Westbury。感谢 Helen Klein、Mallika Malhotra 和 Azita Shokrpour 在法律、隐私和诚信方面的支持。感谢 Michelle Chan、Kei Koyama、William Ngan、Yael Yungster 在整个项目中提供的设计支持。感谢 Arpit Kalla 在模型效率上的工作。感谢 Faye Ma 和 Kehan Lyu 在数据工程支持和工具上的帮助，以及 Emmanuel Hernandez、Robert Kuo 在流程开发上的帮助。感谢 Nan Yang 对第一人称视频数据工作的支持。感谢两位实习生 Cem Gokmen、Jasmine Shone 在三维方面的工作，以及 Lea Wilken 对文稿的反馈。感谢出色的数据运营团队：Paris Baptiste、Karen Bergan、Kai Brown、Ida Cheng、Khadijat Durojaiye、Patrick Edwards、Daniella Factor、Eva Galper、Leonna Jones、Zayida Suber、Tatum Turner、Joseph Walker 和 Claudette Ward。

References

参考文献

Yao et al. (2025) K. Yao, L. Zhang, X. Yan, Y. Zeng, Q. Zhang, L. Xu, W. Yang, J. Gu, and J. Yu Cast: component-aligned 3d scene reconstruction from an rgb image . ACM Transactions on Graphics (TOG) . Cited by: §5 . Ye et al. (2025) C. Ye, Y. Wu, Z. Lu, J. Chang, X. Guo, J. Zhou, H. Zhao, and X. Han Hi3dgen: high-fidelity 3d geometry generation from images via normal bridging . arXiv preprint arXiv:2503.22236 3 , pp. 2 . Cited by: Figure 6 , Table 2 . Yu et al. (2020) F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell BDD100K: a diverse driving video database with scalable annotation tooling . In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Cited by: §A.1 . Yuan et al. (2023) Z. Yuan, H. Yuan, C. Li, G. Dong, K. Lu, C. Tan, C. Zhou, and J. Zhou Scaling relationship on learning mathematical reasoning with large language models . External Links: 2308.01825 , Link Cited by: §A.8 , §5 . Zhang et al. (2023) B. Zhang, J. Tang, M. Niessner, and P. Wonka 3dshape2vecset: a 3d shape representation for neural fields and generative diffusion models . ACM Transactions On Graphics (TOG) 42 ( 4 ), pp. 1-16 . Cited by: §5 . Zhou et al. (2023) J. Zhou, J. Wang, B. Ma, Y. Liu, T. Huang, and X. Wang Uni3d: exploring unified 3d representation at scale . arXiv preprint arXiv:2310.06773 . Cited by: §D.3.1 , Table 2 . Zhou et al. (2019) Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li On the continuity of rotation representations in neural networks . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 5745-5753 . Cited by: §E.4 , §2.2 .

Outline

附录提纲

The appendix provides additional context to the main paper; it contains additional details about the method and the implementation in SAM 3D, as well as ablations.

附录为正文提供补充背景；其中包含关于方法和 SAM 3D 实现的更多细节，以及消融实验。

The structure of the appendix is as follows:

附录结构如下：

(i) Data Engine details: A more detailed description of the data collection used in the collection step in Section 3.2.1.

（i） 数据引擎细节 ： 对第 3.2.1 节 收集步骤 中所用数据收集的更详细描述。

(ii) Pretraining and Mid-Training Data: How we collected and filtered the data used for pretraining and mid-training the Geometry and Texture & Refinement models

（ii） 预训练与中期训练数据 ： 我们如何收集和过滤用于预训练及中期训练几何模型、纹理与细化模型的数据

(iii) Training details: Architectural details about MoT and the VAEs. Definitions used for objectives used in each stage. Details on the Geometry and Texture & Refinement models.

（iii） 训练细节 ： 关于 MoT 和 VAE 的结构细节。各阶段所用目标的定义。关于几何模型以及纹理与细化模型的细节。

(iv) Evaluations: Introducing the details of the new SA-3DAO benchmark, and evaluation protocols of preference tests and quantitative metrics

（iv） 评测 ： 介绍新基准 SA-3DAO 的细节，以及偏好测试和定量指标的评测协议

(v) Additional experiments and qualitative examples: Providing additional analysis and insights into the model’s performance.

（v） 补充实验和定性例子 ： 对模型性能提供更多分析和见解。

(vi) Limitations: An analysis of common failure modes, and future work

（vi） 局限 ： 对常见失败模式的分析，以及未来工作

Appendix A Data Annotation Engine Details

附录 A 数据标注引擎细节

A.1 Stage 1: Image and Object Candidate Sourcing

A.1 第 1 阶段：图像与物体候选的来源

Image sources. To promote generalization across diverse real-world scenes, we expanded our domain coverage by sourcing images from multiple datasets. These include large-scale web-sourced imagery (SA-1B (Kirillov et al., 2023), MetaCLIP (Xu et al., 2024a)), video data capturing everyday environments (SA-VI (Li et al., 2023a)), egocentric video datasets (Ego4D (Grauman et al., 2022), Ego-Exo4D (Grauman and others, 2024), AEA (Lv et al., 2024), AEO (Straub et al., 2024), Nymeria (Ma et al., 2024)), and domain-specific collections such as food (Food Recognition (Bossard et al., 2014)) and driving scenes (BDD100k (Yu et al., 2020)).

图像来源。 为了促进在多样真实世界场景上的泛化，我们从多个数据集中获取图像，以扩大领域覆盖。这些数据包括大规模网络图像（SA-1B（Kirillov et al., 2023）、MetaCLIP（Xu et al., 2024a））、捕捉日常环境的视频数据（SA-VI（Li et al., 2023a））、第一人称视频数据集（Ego4D（Grauman et al., 2022）、Ego-Exo4D（Grauman and others, 2024）、AEA（Lv et al., 2024）、AEO（Straub et al., 2024）、Nymeria（Ma et al., 2024）），以及食物（Food Recognition（Bossard et al., 2014））和驾驶场景（BDD100k（Yu et al., 2020））等特定领域集合。

We first filter out images with low resolution, severe blurriness, low contrast, or noticeable artifacts to ensure high-quality visual inputs that are representative of real-world scenarios. Next, we employ visual-language models for object recognition to generate object-level annotations for each image. Images containing only uninformative backgrounds (e.g ., ground, sky, ocean) without salient 3D objects are subsequently removed from the dataset.

我们首先过滤掉分辨率低、模糊严重、对比度低或有明显伪影的图像，以确保高质量、能够代表真实世界情形的视觉输入。随后，我们用视觉-语言模型做物体识别，为每张图像生成物体级标注。只包含无信息背景（例如地面、天空、海洋）而没有显著三维物体的图像，会随后从数据集中去掉。

For each object description, we employ a referral segmentation model to visually ground the object, followed by human annotator verification or refinement of object masks. We discard low-quality masks, masks covering multiple objects, or partial masks that do not capture a distinct object part. This ensures that each retained mask corresponds to a clearly indexable single object instance with sufficient granularity.

对每个物体描述，我们用指称分割模型把该物体在视觉上锚定下来，再由人类标注员核验或修正物体掩码。我们丢弃低质量掩码、覆盖多个物体的掩码，或未能捕捉某个明确物体部件的部分掩码。这保证每个保留下来的掩码都对应于一个可以清楚索引的单个物体实例，并具有足够的粒度。

1,200

1,200

We retain object category labels and continuously monitor the distribution of objects passing through our data engine. To balance throughput and efficiency, we employ a curriculum-inspired sampling strategy, progressing from simple to increasingly complex geometries. Specifically, we begin with rigid objects of simple shapes (e.g ., balls, cylinders), transition to more structurally complex objects (e.g ., tools, buildings) and ultimately include non-rigid and highly deformable objects (e.g ., animals, humans, clothing). The sampling distribution is adaptively adjusted to reflect the evolving dataset composition, with particular emphasis on gradually expanding coverage of long-tail object categories. Through this strategy, we’re able to source 850,000 unique object instances from 360,000 images, with annotations covering a wide range of object categories.

我们保留物体类别标签，并持续监测穿过数据引擎的物体分布。为了在吞吐量和效率之间取得平衡，我们采用受课程学习启发的采样策略，从简单几何逐步过渡到越来越复杂的几何。具体而言，我们从形状简单的刚性物体开始（例如球、圆柱），再过渡到结构更复杂的物体（例如工具、建筑），最终纳入非刚性和高度可变形的物体（例如动物、人、服装）。采样分布会根据不断变化的数据集构成自适应调整，并特别强调逐步扩大对长尾物体类别的覆盖。通过这一策略，我们能够从 36 万张图像中得到 85 万个独特物体实例，标注覆盖了广泛的物体类别。

Figure 13: Category distribution of SAM 3D training data . The plot above shows the distribution of the top 80 object categories, which includes a long tail.

图 13： SAM 3D 训练数据的类别分布 。上图给出前 80 个物体类别的分布，其中包含长尾。

Texture MITL-3DO. The MITL-3DO dataset for texture is separate from the dataset for shape and layout, but is collected in a similar fashion. The images are sourced from SA-1B (Kirillov et al., 2023), and we additionally sample a dataset of examples with higher aesthetics - objects with minimal occlusion and high brightness, contrast, colorfulness, sharpness, and aesthetic score - to seed the model with higher-quality texture annotations. We found the high-aesthetics dataset to further improve human preference rate (see “AES” preference win rate in Figure 19).

纹理 MITL-3DO。 用于纹理的 MITL-3DO 数据集与用于形状和布局的数据集是分开的，但收集方式类似。图像来自 SA-1B（Kirillov et al., 2023），我们还额外采样了一份审美更高的样本集合——遮挡很少，并且亮度、对比度、色彩丰富度、锐度和审美分数都较高的物体——以便用更高质量的纹理标注为模型奠基。我们发现，这份高审美数据还能进一步提高人人偏好率（见图 19 中的“AES”偏好胜率）。

A.2 Stage 2: 3D Model-in-the-Loop Suite

A.2 第 2 阶段：三维模型在环套件

N

N

• Retrieval : The nearest 3D object is retrieved from a shape object library (pretraining data) using both image- and text-based similarity. For text similarity, we compare visual object descriptions; for image similarity, we compute the distance between CLIP embeddings. While this retrieval approach is nearly guaranteed not to provide an exact 3D reconstruction, it can provide a high quality mesh with matching semantics, particularly when model-generated 3D shapes fail entirely.

• 检索 ：用基于图像和基于文本的相似度，从形状物体库（预训练数据）中检索最近的三维物体。文本相似度比较的是视觉物体描述；图像相似度计算的是 CLIP 嵌入之间的距离。这种检索方法几乎可以肯定无法给出精确的三维重建，但可以提供语义匹配的高质量网格，尤其是在模型生成的三维形状完全失败时。

• Text-to-3D generation : A text-to-3D generative method produces 3D object meshes based on a textual descriptions. This approach can be helpful when image-conditioning is challenging due to clutter or occlusion, but human recognition can still identify the object.

• 文本到三维生成 ：一种文本到三维的生成方法根据文本描述产生三维物体网格。当由于杂乱或遮挡而使图像条件变得困难、但人的识别仍能辨认出该物体时，这一做法会有帮助。

• Image-to-3D generation : Image-to-3D methods, including our own SAM 3D checkpoint, generate 3D objects in the form of point clouds or meshes, conditioned on the image input. When successful, this tends to produce examples that go beyond semantic matches and better respect the object’s physical appearance in the image. However, lack of robustness to occlusion or clutter can negatively impact the results.

• 图像到三维生成 ：包括我们自己的 SAM 3D 检查点在内的图像到三维方法，以图像输入为条件，生成点云或网格形式的三维物体。一旦成功，这类例子往往不仅在语义上匹配，还能更好地符合物体在图像中的物理外观。不过，对遮挡或杂乱缺少鲁棒性，可能对结果造成负面影响。

3D texture model suite. For texture generation, we utilize image-to-3D models, multi-view texture generation models, and our own SAM 3D checkpoints. All texture candidates are generated using shapes produced by the SAM 3D Geometry model, ensuring that texture models have the best chance of success, even in cases of heavy occlusion.

三维纹理模型套件。 对于纹理生成，我们使用图像到三维模型、多视图纹理生成模型，以及我们自己的 SAM 3D 检查点。所有纹理候选都用 SAM 3D 几何模型产生的形状来生成，从而确保即便在严重遮挡的情形下，纹理模型也有最好的成功机会。

Figure 14: Stage 2 UI sketch. Annotators can only choose between options; they cannot directly edit the meshes or textures.

图 14： 第 2 阶段界面示意 。 标注员只能在选项之间选择；他们不能直接编辑网格或纹理。

N

N

\alpha

\alpha

A.3 Stage 2.5: 3D Artist Mesh Details

A.3 第 2.5 阶段：三维艺术家网格的细节

When the 3D model-in-the-loop suite fails to generate an acceptable mesh for a particular sample, the aforementioned preference-based annotation approach is unable to provide the data needed to improve the model for such objects. To overcome this data distribution blind spot, we work with a team of 3D artists to build meshes for such hard meshes. Given the high cost of specialized 3D artists, we seek to maximize their value by ensuring each object sent to the 3D artists represents a genuine failure case that cannot be resolved by the data engine alone. To maximize the value of this investment, we develop a refined labeling framework that categorizes failures into common types: e.g ., complex geometry, occlusion, transparency, and small object size. We balance sampling across these categories. In addition, we employ clustering techniques over images, 3D latents, and object semantics to deduplicate candidates, ensuring that one or a few representative samples per group suffices for effective coverage in data sampling.

当三维模型在环套件无法为某个样本生成可接受的网格时，前述基于偏好的标注方法就无法提供改进这类物体所需的数据。为了克服这一数据分布上的盲区，我们与一组三维艺术家合作，为这些困难网格建模。鉴于专业三维艺术家成本很高，我们力求让送到他们那里的每个物体都真正代表数据引擎自身无法解决的失败情形，从而最大化其价值。为了让这项投入尽可能值当，我们设计了一套更细致的标注框架，把失败归入常见类型：例如复杂几何、遮挡、透明，以及物体尺寸很小。我们在这些类别之间做平衡采样。此外，我们在图像、三维潜在表征和物体语义上使用聚类技术来去除重复候选，以确保每个组只需一两个代表性样本，就足以在数据采样中形成有效覆盖。

Additional details on the data collection process for meshes created by 3D artists can be found in Section D.1, which employed a similar mesh creation process by the artists, but with more intentional curation of inputs.

关于三维艺术家创建网格的数据收集过程的更多细节，可见第 D.1 节；那里的艺术家采用了类似的网格创建流程，但对输入做了更有意识的筛选。

A.4 Stage 3: 3D Mesh Alignment

A.4 第 3 阶段：三维网格对齐

We collect object pose annotations by aligning meshes from prior stages to a scene point cloud derived from the input image. To make this accessible to generalist annotators, we designed and implemented an annotation tool which allows the annotator to manipulate 3D meshes to align to a 2.5D point cloud pre-computed by an off-the-shelf depth estimator. Annotators can use either keyboard or mouse to rotate, translate, and scale the meshes so that the mesh is accurately anchored to the 2.5D point cloud. We also provide additional functions including (a) mesh visibility toggle, (b) target indicator toggle, (c) point cloud size adjustment, (d) control visibility toggle, (e) undo, (f) camera view reset and pre-defined view, and (g) mesh IOU indicator as shown in Figure 15.

我们通过把前几阶段得到的网格对齐到由输入图像得到的场景点云上，来收集物体姿态标注。为了让非专业标注员也能完成这项工作，我们设计并实现了一个标注工具，允许标注员操作三维网格，使其对齐到由现成深度估计器预先计算的 2.5D 点云。标注员可以用键盘或鼠标旋转、平移和缩放网格，使网格准确地锚定到 2.5D 点云上。我们还提供了额外功能，包括（a）网格可见性开关，（b）目标指示器开关，（c）点云大小调节，（d）控制可见性开关，（e）撤销，（f）相机视图重置和预定义视图，以及（g）网格 IoU 指示器，如图 15 所示。

Figure 15: Stage 3 UI sketch. The UI supports annotators in directly placing the object in the 2.5D pointcloud.

图 15： 第 3 阶段界面示意 。 该界面支持标注员把物体直接放置到 2.5D 点云中。

A.5 Annotation Statistics

A.5 标注统计

• Stage 1: Annotators on average spend 10 seconds to segment a single interesting object. We utilize SAM (Kirillov et al., 2023) as a tool to assist in segmentation.

• 第 1 阶段：标注员平均花费 10 秒分割一个感兴趣的单个物体。我们用 SAM（Kirillov et al., 2023）作为辅助分割工具。

• Stage 2: Annotators on average spend 80 seconds to select the best candidate shape/texture from 6-10 candidate meshes from variable sources.

• 第 2 阶段：标注员平均花费 80 秒，从来源不一的 6-10 个候选网格中选出最好的形状/纹理候选。

• Stage 3: Annotators on average spend 150 seconds to anchor and orient the matched 3D shape to the 2.5D point cloud.

• 第 3 阶段：标注员平均花费 150 秒，把匹配到的三维形状锚定并对准到 2.5D 点云。

• Over the lifetime of the project (including development), our MITL data engine yields 3.14 million trainable shapes, 1.23 million samples of layout data, 100K trainable textures, and over 7 million pairwise preferences.

• 在项目生命周期内（包括开发阶段），我们的 MITL 数据引擎产生了 314 万个可训练形状、123 万个布局数据样本、约 10 万个可训练纹理，以及超过 700 万对成对偏好。

A.6 Core Alignment Algorithm

A.6 核心对齐算法

\pi_{0}

\pi_{0}

A.6.1 Basic Algorithm Algorithm 1 shows the core alignment algorithm, used for all texture annotations and most shape annotations (MITL-3DO). During each collection step, we generate a set of predictions from the current model, and ask annotators to rank and verify these predictions. Generalist annotators can only choose between model outputs and accept/reject; they cannot edit. We maximize the probability of a successful annotation at each iteration by ensembling multiple models and combining multiple models with human preferences into an expert annotator.

A.6.1 基本算法 算法 1 给出了核心对齐算法，用于全部纹理标注和大多数形状标注（MITL-3DO）。在每个收集步骤中，我们从当前模型生成一组预测，并请标注员对这些预测排序和核验。非专业标注员只能在模型输出之间选择并接受/拒绝；他们不能编辑。我们通过集成多个模型，并把多个模型与人的偏好组合成一个 专家 标注者，来最大化每次迭代中成功标注的概率。

The learning efficiency of the alignment, or the “speed of the data flywheel”, is controlled by two factors:

对齐的学习效率，或者说“数据飞轮的速度”，由两个因素控制：

• Amplification factor : The size of the performance gap between current model and the expert annotations at each iteration

• 放大因子 ：在每次迭代中，当前模型与专家标注之间的性能差距大小

• Stepwise efficiency : How closely the new model approximates the previous expert from the previous iteration

• 逐步效率 ：新模型在多大程度上逼近上一轮迭代中的先前专家

The former induces an upper bound on the new policy’s performance at each iteration, while the latter describes how closely we approach that upper bound - similar to Expert Iteration (Anthony et al., 2017).

前者对每次迭代中新策略的性能给出了上界，后者则描述我们在多大程度上接近这个上界——这类似于专家迭代（Expert Iteration）（Anthony et al., 2017）。

A.6.2 Training Intuition Our goal in post-training (see Algorithm 1) is to align the model to match human preference on the distribution of all possible real-world objects.

A.6.2 训练直觉 我们在后训练中的目标（见算法 1）是：在 所有 可能的真实世界物体的分布上，把模型对齐到人的偏好。

The core algorithm in our data engine generates samples by asking humans to select viable samples from a set of candidate generations. Challenging inputs often result in no viable candidate generations and thus never get selected by humans. However, at any current time our model is usually good on some parts of the data distribution, but not on other parts, as shown in the cartoon below:

我们数据引擎中的核心算法，通过请人从一组候选生成中选出可用样本来产生样本。有挑战性的输入常常导致没有任何可用的候选生成，因而永远不会被人选中。不过，在任一当前时刻，我们的模型通常只在数据分布的某些部分上表现较好，在其他部分上则不然，正如下面这幅示意图所示：

Figure 16: Simplified cartoon depiction of data engine improvement. The diagram depicts model sample quality (color) across the real-world distribution of images and masks. During training, the model begins by doing well (teal) on common categories and simple objects (chairs, bottles, signs, cars). Our goal is to both improve accuracy and robustness on these easy examples (teal), and then push the model to improve performance on less common objects (yellow) in the tail of the probability distribution. While the amplification stage of MITL generally leads to the slow expansion of existing regions of success, using 3D artists to create data for the hardest samples allows us to shortcut the process by “seeding” new regions of the data distribution, which may have taken us longer to reach through MITL verification alone.

图 16： 数据引擎改进的简化示意 。 图中用颜色表示模型在真实世界图像与掩码分布上的样本质量。训练开始时，模型在常见类别和简单物体（椅子、瓶子、标志、汽车）上表现较好（青绿色）。我们的目标是既提高这些容易样本（ 青绿色 ）上的精度和鲁棒性，再推动模型改进概率分布尾部那些不太常见的物体（ 黄色 ）上的性能。MITL 的放大阶段通常只会让已有的成功区域缓慢扩展，而让三维艺术家为最困难的样本创建数据，则可以通过在数据分布的新区域“播种”，来缩短这一过程；若只靠 MITL 核验，到达这些区域可能需要更长时间。

The intuition behind the data engine in SAM 3D is that these green islands of reliably good performance correspond to high-density parts of the training data (O’Neill and others, 2024; NVIDIA et al., 2025), and the approach in SAM 3D is that we want to push out from these islands of reliably good generations into the “tail” of the distribution, demarcated by yellow and white background in the cartoon above. The yellow parts of the distribution are challenging for the model, but near enough to the blue islands, that we can occasionally generate satisfactory annotations, but it requires humans to go through many samples.

SAM 3D 数据引擎背后的直觉是：这些能够可靠给出较好结果的绿色岛屿，对应于训练数据中的高密度部分（O’Neill and others, 2024；NVIDIA et al., 2025）；SAM 3D 的做法是，我们希望从这些可靠生成的岛屿向外推进，进入上图中用黄色和白色背景标出的分布“尾部”。分布中的黄色部分对模型有挑战，但足够靠近蓝色岛屿，以至于我们 偶尔 仍能生成令人满意的标注，不过这需要人查看大量样本。

This can create a chicken-and-egg problem where, for the model to become good, it must already be capable of produce a good generation; at least some of the time. For examples that are so challenging that the probability of success is extremely low (white), the model has no hope and we ask human 3D artists (Section A.3) to provide supervision in this part of the data distribution, in order to seed new islands.

这可能造成一个先有鸡还是先有蛋的问题：模型要变好，就必须已经有能力产生好的生成，至少有时如此。对于成功概率极低的那些极具挑战性的例子（白色），模型没有希望，于是我们请人类三维艺术家（第 A.3 节）在数据分布的这一部分提供监督，以便播种新的岛屿。

A.7 Increasing Amplification Factor with Search

A.7 用搜索提高放大因子

N

N

\sim 50

\sim 50

N

N

N

N

A.8 Relationship to Self-Training Approaches

A.8 与自训练方法的关系

q

q

This reframing helps make the relationship to existing work more clear. The most similar learning algorithm to our data engine is Expert Iteration (ExIt) (Anthony et al., 2017). As in ExIt, each iteration starts with a current policy, that we amplify using additional information into an expert policy, and we use this expert policy to generate supervision for imitation learning. Unlike ExIt, which uses purely imitation learning, we use humans-as-verifiers to select which samples to train on, and we make use of additional preference signals as reward signal (Section 3.2.2). However, there are also notable differences in type of supervision that can be used and the amplification steps. Our expert policy amplification step uses a model ensemble instead of only tree search with a value function, and we use preferences in the update step to better align to the task.

这种重新表述有助于把与已有工作的关系说得更清楚。与我们的数据引擎最相似的学习算法是专家迭代（Expert Iteration，ExIt）（Anthony et al., 2017）。与 ExIt 一样，每次迭代都从当前策略开始，我们用额外信息把它放大成专家策略，再用这个专家策略为模仿学习生成监督。与只使用纯模仿学习的 ExIt 不同，我们用人作为核验者来选择训练哪些样本，并在更新步骤中利用额外的偏好信号作为奖励信号（第 3.2.2 节）。不过，在可用监督的类型以及放大步骤上也有明显差异。我们的专家策略放大步骤使用的是模型集成，而不仅仅是带价值函数的树搜索，并且我们在更新步骤中使用偏好，以便更好地对齐到任务。

Algorithm 1 uses reward ranking, similar to RAFT (Dong et al., 2023) and RFT (Yuan et al., 2023), although the alignment algorithm in SAM 3D adds explicit expert policies/ensembles and leverages preference supervision.

算法 1 使用奖励排序，类似于 RAFT（Dong et al., 2023）和 RFT（Yuan et al., 2023），不过 SAM 3D 中的对齐算法还加入了显式的专家策略/集成，并利用了偏好监督。

Appendix B Pretraining and Mid-Training Data Details

附录 B 预训练与中期训练数据的细节

B.1 Iso-3DO Data Filtering

B.1 Iso-3DO 数据过滤

For the Iso-3DO data used for pretraining, the quality of the 3D meshes can vary substantially, and not all samples exhibit high-fidelity geometry. Such examples can ultimately prove harmful to model pretraining, even at scale. One way to filter data is by an aesthetic score filter, as employed by Xiang et al. (2025), which primarily emphasizes visual and textural appeal. We employ a similar filter process for the Texture & Refinement model.

对于预训练所用的 Iso-3DO 数据，三维网格的质量可能相差很大，并非所有样本都具有高保真几何。即便在大规模下，这类样本最终也可能对模型预训练有害。一种过滤数据的方法是审美分数过滤器，正如 Xiang et al. (2025) 所采用的，它主要强调视觉和纹理上的吸引力。我们对纹理与细化模型采用了类似的过滤流程。

However, this filter does not necessarily capture the geometric quality of a training data. Therefore, we develop a rule-based filtering strategy on shape to curate the pretraining data for the Geometry model, removing data with following characteristics:

不过，这一过滤器并不一定能捕捉训练数据的几何质量。因此，我们为几何模型的预训练数据开发了一套基于规则的形状过滤策略，去掉具有以下特征的数据：

• Overly simplistic geometry, characterized by extremely small volumes (e.g ., near-degenerate point-like structures) or minimal normal direction variation (e.g ., flat, sheet-like surfaces).

• 过于简单的几何 ，其特征是体积极小（例如近乎退化的点状结构），或法向变化极小（例如平坦的片状表面）。

• Structural outliers, which includes meshes containing spatial outliers: isolated points or fragments that deviate significantly from the primary 3D structure.

• 结构异常值 ，包括含有空间异常的网格：明显偏离主要三维结构的孤立点或碎片。

B.2 Render-and-Paste Data Pipeline

B.2 渲染并粘贴的数据流程

We define the Render-Paste approach as follows: Given a natural image and an object instance defined by its mask segment, we replace the object in the image with a synthetic 3D object drawn from the same synthetic sources used in Iso-3DO. The size and position of the 3D object are determined using the 2D object mask together with a pointmap produced by a single-image depth estimator, which also guides the object’s visibility and occlusion to obtain a natural appearance in the final rendering. By nature of starting from a synthetic 3D object first, the resulting data (which we refer to as RP-3DO) has excellent 3D ground truth precision and pixel-alignment compared to subsequent data sources in our training pipeline, which much try to reconstruct 3D from partial 2D information as part of the data annotation process.

我们把 Render-Paste 方法定义如下：给定一张自然图像，以及由其掩码片段定义的一个物体实例，我们用从 Iso-3DO 相同合成来源中取出的合成三维物体，替换图像中的该物体。三维物体的大小和位置由二维物体掩码以及单图深度估计器产生的点图共同决定，点图也指导物体的可见性与遮挡，以使最终渲染具有自然外观。由于一开始用的就是合成三维物体，得到的数据（我们称为 RP-3DO ）具有很好的三维真值精度和像素对齐，相比之下，我们训练流程中后续的数据来源必须在标注过程中从部分二维信息去重建三维。

In the following sections, we introduce three variants of Render-Paste that differ along two axes: pose information and semantic relevance.

在以下几节中，我们介绍 Render-Paste 的三种变体，它们沿两个轴有所不同：姿态信息和语义相关性。

• Section B.2.1: Flying Occlusions (FO) inserts randomly oriented synthetic objects without pose information, resulting in pose-unaware but semantically loose composites.

• 第 B.2.1 节： 飞行遮挡 （FO） 插入随机朝向、没有姿态信息的合成物体，得到对姿态不敏感、语义也较松的合成图。

• Section B.2.2: Object Swap - Random (OS-R) determines object scale and translation from masks and pointmaps, while using a random rotation and object. Beyond simple replacement, the incorporation of depth ordering provides meaningful visual cues for object size and spatial placement, yielding pose-aware but not fully aligned insertions with moderate semantic relevance, higher than in the Flying Occlusions setting.

• 第 B.2.2 节： 物体替换-随机 （OS-R） 根据掩码和点图确定物体的尺度和平移，同时使用随机旋转和随机物体。除了简单替换之外，引入深度顺序还能为物体大小和空间放置提供有意义的视觉线索，从而得到对姿态有意识、但并未完全对齐、语义相关性中等（高于飞行遮挡设定）的插入。

• Section B.2.3: Object Swap - Annotated (OS-A) replaces the original object using the annotator-provided ground-truth scale, translation, and rotation, producing fully pose-aligned and semantically matched renderings.

• 第 B.2.3 节： 物体替换-已标注 （OS-A） 使用标注员提供的真值尺度、平移和旋转来替换原始物体，从而产生完全姿态对齐、且语义匹配的渲染。

B.2.1 Flying Occlusions (FO) The aim of this dataset is to build invariance to occlusion and size variations that commonly occur in real-world scenarios—and to enable the model to leverage full image context instead of only object-centered crops—we construct a dataset of natural images with blended synthetic 3D objects. Inspired by Flying Chairs (Dosovitskiy et al., 2015) and FlyingThings3D (Mayer et al., 2016), we name our first variant Flying Occlusions, reflecting its use of freely inserted synthetic objects.

B.2.1 飞行遮挡（FO） 这一数据集的目标是建立对真实世界中常见的遮挡和尺寸变化的不变性，并使模型能够利用整图上下文，而不只是以物体为中心的裁剪。为此，我们构造了一份把合成三维物体混合进自然图像的数据集。受 Flying Chairs（Dosovitskiy et al., 2015）和 FlyingThings3D（Mayer et al., 2016）启发，我们把第一种变体命名为飞行遮挡，以反映它自由插入合成物体的做法。

M_{\mathrm{obj}}

M_{\mathrm{obj}}

Finally, to prevent the model from always predicting the occluded object, in one third of the samples, we treat the selected mesh as the occluder. In these cases, the mask of the selected mesh is complete. In total, we have 55.1M sample with 2.87M unique meshes and 11.17M unique images.

最后，为了防止模型总是去预测被遮挡的物体，在三分之一的样本中，我们把选定网格当作遮挡物。在这些情形下，选定网格的掩码是完整的。我们总共有 5510 万个样本，包含 287 万个独特网格和 1117 万张独特图像。

B.2.2 Object Swap - Random (OS-R) To enhance robustness to variations in object location and scale, we propose Object Swap - Random (OS-R), a depth-aware render-paste strategy that replaces an object in a natural image with a randomly selected synthetic mesh.

B.2.2 物体替换-随机（OS-R） 为了增强对物体位置和尺度变化的鲁棒性，我们提出物体替换-随机（OS-R），这是一种有深度意识的渲染-粘贴策略，用随机选取的合成网格替换自然图像中的一个物体。

I

I

I^{\prime}

M_{\mathrm{vis}}

To ensure the dataset provides sufficient visual cues for estimating translation and scale, we use heuristics to replace only objects that are partially occluded or supported along the bottom, which provides depth ordering and T-junction cues, respectively. We verify these cues by trying to find occlusion boundaries: we sample points on opposite sides of the mask border, and if the outer pixel is significantly closer to the camera than the inner pixel, we consider this part of the boundary occluded. A sample is retained if it meets one of two conditions: (1) physical support, where the background is closer to the camera along the bottom 10% of the object (indicating it rests on a surface), or (2) partial occlusion, where foreground elements occlude at least 10% of the total object perimeter. This process yields 5.95M training samples composed of 2.38M unique meshes and 1.20M unique images.

为了确保数据集为估计平移和尺度提供足够的视觉线索，我们用启发式方法，只替换被部分遮挡或沿底部受到支撑的物体，它们分别提供深度顺序和 T 形交线索。我们通过尝试寻找遮挡边界来核验这些线索：我们在掩码边界两侧采样点，如果外侧像素明显比内侧像素更靠近相机，就把这部分边界视为被遮挡。样本在满足以下两个条件之一时被保留：（1） 物理支撑 ，即沿物体底部 10% 处背景更靠近相机（表明它搁在某个表面上）；或（2） 部分遮挡 ，即前景元素遮挡了物体周长至少 10%。这一过程得到 595 万个训练样本，由 238 万个独特网格和 120 万张独特图像组成。

B.2.3 Object Swap - Annotated (OS-A) In addition to the Object Swap - Random variant, we construct a complementary render-and-paste setting, which we refer to as Object Swap - Annotated (OS-A), which performs an in-place replacement of a real image with a rendered human-annotated object. The motivation for this dataset is to enable Texture & Refinement training that faithfully preserves pixel-aligned correspondence between the rendered mesh and the visual appearance of the target object in the image.

B.2.3 物体替换-已标注（OS-A） 除了 物体替换-随机 这一变体之外，我们还构造了一个互补的渲染并粘贴设定，称为 物体替换-已标注 （OS-A），它对真实图像做原地替换，换成渲染后的人类标注物体。这份数据的动机是：使纹理与细化训练能够忠实地保持渲染网格与图像中目标物体视觉外观之间的像素对齐对应。

S

S

B.3 Lighting for Texture Data

B.3 纹理数据的光照

For Iso-3DO and RP-3DO (FO), we randomize the lighting (i.e ., direction and intensity) applied on the input images, and use ambient lighting when rendering views used to computing the target latents. Qualitatively, such data processing encourages the model to predict “de-lighted” textures without baking in strong directional shading or specular highlights from the input render. We verify that this is preferred by humans through preference tests (see “Lighting Aug” preference rate in Figure 19).

对于 Iso-3DO 和 RP-3DO（FO），我们随机化施加在输入图像上的光照（也就是方向和强度），并在渲染用于计算目标潜在表征的视图时使用环境光。从定性上看，这样的数据处理会鼓励模型预测“去光照”的纹理，而不是把输入渲染中强烈的方向性阴影或镜面高光烘焙进去。我们通过偏好测试验证了这更受人偏好（见图 19 中的“Lighting Aug”偏好率）。

Appendix C Details on Model Training

附录 C 模型训练的细节

The following sections outline the details of the Geometry and Texture & Refinement models, including architecture, training objective, and training hyperparameters.

以下几节概述几何模型以及纹理与细化模型的细节，包括结构、训练目标和训练超参数。

C.1 Architecture Details on Geometry Model

C.1 几何模型的结构细节

64^{3}

16^{3}\times 8

The MoT design allows independently training of some modalities while maintaining performance on others (e.g ., fine-tune shape or layout only), thanks to the structured attention mask illustrated in Figure 2. This proves helpful when training on datasets that contain labels for only one modality (e.g . shape-only), and when freezing shape capabilities and finetuning just for layout. At the same time, MoT still allows for information sharing during the forward pass, through the joint self-attention layers for cross-modal interaction. This shared context is critical for self-consistency: notably, rotation is only meaningful when anchored to the predicted shape.

MoT 的设计允许对某些模态独立训练，同时保持其他模态上的性能（例如只微调形状或只微调布局），这得益于图 2 中所示的结构化注意力掩码。当训练数据只包含一种模态的标签（例如只有形状），以及在冻结形状能力、只微调布局时，这一点很有帮助。与此同时，MoT 在前向过程中仍允许信息共享，通过联合自注意力层进行跨模态交互。这种共享上下文对自一致性至关重要：尤其是，旋转只有锚定到所预测的形状时才有意义。

C.2 Pretraining & SFT Objective: Conditional Rectified Flow Matching

C.2 预训练与 SFT 目标：条件整流流匹配

I

I

\mathcal{L}_{\text{CFM}}=\sum_{m\in\mathcal{M}}\lambda_{m}\underset{\raisebox{-3.91806pt}{$\scriptstyle\tau,\mathbf{x}_{0}^{m}$}}{\text{\Large$\mathbb{E}$}}\left[\|\mathbf{v}^{m}-\mathbf{v}_{\theta}^{m}(\mathbf{x}_{\tau}^{m},c,\tau)\|^{2}\right] \tag{1}

\mathcal{M}=\{S,R,t,s\}

\mathcal{M}=\{S,R,t,s\}

\{\mathbf{x}_{1}^{m}\}_{m\in\mathcal{M}}\sim p(\mathcal{M}|c)

\{\mathbf{x}_{1}^{m}\}_{m\in\mathcal{M}}\sim p(\mathcal{M}|c)

The Texture & Refinement model optimizes analogous flow-matching objectives using SLAT features. We train both models using AdamW (without weight decay), and training hyperparameters such as sampling and learning rate schedules, EMA weights are in Section C.7.

纹理与细化模型用 SLAT 特征优化类似的流匹配目标。两个模型都用 AdamW（不加权重衰减）训练，采样和学习率日程、EMA 权重等训练超参数见第 C.7 节。

We find it noteworthy that flow matching objective is sufficient for SAM 3D to learn the task of 3D reconstruction, without explicitly enforcing geometric constraints through loss objectives. Interestingly, we observed empirically that while such losses can be helpful in low-data and low-compute regimes, their value declines as data and compute increase, suggesting these geometric priors can be learned implicitly with scale.

我们觉得值得指出的是：流匹配目标就足以让 SAM 3D 学会三维重建这一任务，而不必通过损失目标显式施加几何约束。有意思的是，我们从经验上观察到，这类损失在低数据和低算力设定下可能有帮助，但随着数据和算力增加，它们的价值会下降，这表明这些几何先验可以随规模被隐式学到。

C.3 Preference Alignment Objective: DPO

C.3 偏好对齐目标：DPO

c

c

\displaystyle\mathcal{L}{\mathrm{DPO}} \ \displaystyle=-\mathbb{E}{\begin{array}[]{rl}I&\sim~\mathcal{I},\(x_{0}^{w},x_{0}^{l})&\sim~\mathcal{X}{I}^{2}\\tau&\sim~\mathcal{U}(0,T)\x{\tau}^{w}&\simq(x_{\tau}^{w}\mid x_{0}^{w})\x_{\tau}^{l}&\simq(x_{\tau}^{l}\mid x_{0}^{l})\end{array}}\big[\operatorname{\log\sigma}\big(-\beta Tw(\tau)\cdot\Delta\big)\big] \ \displaystyle\text{where}\quad\Delta \ \displaystyle=|\mathbf{v}^{w}-\mathbf{v}{\theta}(x{\tau}^{w},c,\tau)|{2}^{2}-|\mathbf{v}^{w}-\mathbf{v}{\mathrm{ref}}(x_{\tau}^{w},c,\tau)|{2}^{2} \ \displaystyle\quad-\big(|\mathbf{v}^{l}-\mathbf{v}{\theta}(x_{\tau}^{l},c,\tau)|{2}^{2}-|\mathbf{v}^{l}-\mathbf{v}{\mathrm{ref}}(x_{\tau}^{l},c,\tau)|_{2}^{2}\big)

\mathbf{v}^{w}

\mathbf{v}^{w}

Implementation details. We apply DPO on shape prediction in the Geometry model and the predictions of the Texture & Refinement model. We use the preference data collected in Stage 2, where we remove the negatives from non-SAM 3D generations (e.g . retrieval-based methods or multi-view diffusion texture generations), since they are out of the distribution for SAM 3D.

实现细节。 我们把 DPO 用在几何模型的形状预测上，以及纹理与细化模型的预测上。我们使用第 2 阶段收集的偏好数据，并从中去掉非 SAM 3D 生成的负例（例如基于检索的方法，或多视图扩散纹理生成），因为它们对 SAM 3D 来说是分布外的。

C.4 Model Distillation Objective: Shortcut Models

C.4 模型蒸馏目标：捷径模型

For applications needing online 3D perception capabilities (e.g . robotics), model inference time is an essential consideration. In diffusion and flow matching models, the most straightforward way to improve inference speed is by reducing the number of function evaluations (NFE). However, naïvely decreasing the number of steps can significantly degrade performance. Instead, we employ flow matching distillation techniques to reduce the number of inference steps while minimizing impact to quality. Specifically, we adopt the diffusion shortcut formulation from Frans et al. (2024), which offers several advantages over previous consistency distillation approaches: (1) it is simple, avoiding multi-stage training and instability; and (2) the model supports two modes, allowing seamless switching back to the original flow matching inference, so a single model can serve both purposes. Unlike the original formulation, we do not train shortcut models from scratch. Instead, we initialize from fully trained checkpoints and further finetune them with the shortcut objective.

对于需要在线三维感知能力的应用（例如机器人），模型推理时间是一个重要考虑。在扩散和流匹配模型中，提升推理速度最直接的办法是减少函数求值次数（NFE）。不过，简单地减少步数会显著降低性能。相反，我们采用流匹配蒸馏技术，在减少推理步数的同时尽量减小对质量的影响。具体而言，我们采用 Frans et al. (2024) 的扩散捷径公式，它相对先前的一致性蒸馏方法有几项优点：（1）它很简单，避免了多阶段训练和不稳定；（2）模型支持两种模式，可以无缝切回原来的流匹配推理，从而让一个模型同时服务两种用途。与原始公式不同，我们并不从零开始训练捷径模型。相反，我们从充分训练的检查点初始化，再用捷径目标进一步微调它们。

\displaystyle\mathcal{L}{S}(\theta) \ \displaystyle=\mathbb{E}{\begin{subarray}{c}x_{0}\sim\mathcal{N}(0,I),\x_{1}\sim p(x),\\tau,d\sim p(\tau,d)\end{subarray}}\Big[\qquad\underbrace{\left|\mathbf{v}-\mathbf{v}{\theta}(x{\tau},,c,,\tau,,d{=}0)\right|^{2}}{\text{Flow-Matching}}\qquad+\qquad\underbrace{\left|\mathbf{v}{\mathrm{consistency}}-\mathbf{v}{\theta}(x{\tau},,c,,\tau,,2d)\right|^{2}}_{\text{Self-Consistency}}\qquad\Big]. \tag{8}

where:

其中：

x_{0}\sim\mathcal{N}(0,I)

x_{0}\sim\mathcal{N}(0,I)

x_{1}\sim p(x)

x_{1}\sim p(x)

x_{\tau}

x_{\tau}

\tau

\tau

d

d

c

c

p(\tau,d)

p(\tau,d)

\mathbf{v}_{\theta}(x_{\tau},\,c,\,\tau,\,d)

\mathbf{v}_{\theta}(x_{\tau},\,c,\,\tau,\,d)

\mathbf{v}

\mathbf{v}

\mathbf{v}_{\mathrm{consistency}}

\mathbf{v}_{\mathrm{consistency}}

x_{\tau}

x_{\tau}

w_{\mathrm{CFG}}=2

w_{\mathrm{CFG}}=2

C.5 Texture & Refinement Training Details

C.5 纹理与细化的训练细节

We train the Texture & Refinement model following a multi-stage training paradigm analogous to that of the Geometry model (described in Section 3). Below we provide implementation details for the texture training stages.

我们按照与几何模型（见第 3 节）类似的多阶段训练范式来训练纹理与细化模型。下面给出纹理训练各阶段的实现细节。

VAE Training. Learning the inverse problem of image to texture map requires a strong alignment in training data between them. However, in previous work (Xiang et al., 2025), renderings of meshes create artifacts like reflections and shadows; objects consistently having extremely dark bottoms is one such common example. To curate a cleaner set of ground truth data, we create our latent SLAT with a “de-lighted rendering” by using ambient lighting to minimize such artifacts. We use this to process all stages of data. We select a uniform lighting strength, based on computing the similarities between SAM 3D predictions and the original image using RP-3DO (FO).

VAE 训练。 学习从图像到纹理图这一逆问题，要求训练数据在二者之间有很强的对齐。不过，在先前工作（Xiang et al., 2025）中，网格渲染会产生反射和阴影一类伪影；物体底部经常极暗，就是一个常见例子。为了整理出更干净的真值数据，我们用环境光做“去光照渲染”，以尽量减少这类伪影，并据此创建潜在 SLAT。我们用这一做法处理所有阶段的数据。我们基于用 RP-3DO（FO）计算 SAM 3D 预测与原始图像之间的相似度，选定一个均匀的光照强度。

Pretraining. We start with pretraining the model on Iso-3DO-500K, a partition of Iso-3DO data with high aesthetics. Training on such data allows the model to predict plausible, high-quality texture that is characteristic of many 3D asset generation models. To ensure robustness of the texture model on real-world, complex images, we must further train on increasingly challenging data, via additional training stages described below.

预训练。 我们首先在 Iso-3DO-500K 上预训练模型，这是 Iso-3DO 数据中审美较高的一个分区。在这类数据上训练，能让模型预测许多三维资产生成模型所特有的、合理且高质量的纹理。为了保证纹理模型在真实世界、复杂图像上的稳健性，我们必须在日益有挑战的数据上继续训练，也就是通过下面描述的额外训练阶段。

For pretraining data in Iso-3DO, we render conditional images from the 3D mesh. We introduce a random lighting augmentation, where for each view, we apply a random lighting setting in the rendering engine. We hope the model can learn to remove the lighting effects when generating textures.

对于 Iso-3DO 中的预训练数据，我们从三维网格渲染条件图像。我们引入随机光照增强：对每个视图，在渲染引擎中施加一种随机光照设定。我们希望模型在生成纹理时能够去掉光照效应。

Mid-training. During mid-training, we train on RP-3DO (FO and OS-A). It is starting at this stage where we additionally provide full image conditioning to the model, as we believe contextual cues can help the model predict plausible textures, especially when the object is heavily occluded. We show the effect of training on RP-3DO (FO), as well as the effect of further adding RP-3DO (OS-A) training data to this stage, in Figure 19.

中期训练。 在中期训练中，我们在 RP-3DO（FO 和 OS-A）上训练。正是从这一阶段开始，我们额外向模型提供整图条件，因为我们相信上下文线索能帮助模型预测合理的纹理，尤其是在物体被严重遮挡时。我们在图 19 中展示了在 RP-3DO（FO）上训练的效果，以及进一步向这一阶段加入 RP-3DO（OS-A）训练数据的效果。

We also introduce image data augmentation for RP-3DO (FO): Mask augmentation and Blur augmentation. The Mask augmentation randomly erode or dilate the input mask. This is designed to handle noise in mask at inference time (e.g . segmentation predictions from a model). The Blur augmentation applies a downsample operation on the image followed by an upsample operation. This is especially important to handle cases for motion blur and small objects in an image. These augmentation is carefully studied in Section E.2.

我们还为 RP-3DO（FO）引入图像数据增强： 掩码 增强和 模糊 增强。掩码增强会随机腐蚀或膨胀输入掩码。这是为了应对推理时掩码中的噪声（例如来自模型的分割预测）。模糊增强对图像先做下采样再做上采样。这对于处理运动模糊和图像中的小物体尤其重要。这些增强在第 E.2 节中做了仔细研究。

Supervised fine-tuning (SFT). In the SFT stage, the model is trained on MITL-3DO texture annotations, which includes the “aesthetic” samples described in Section A.1. We show the effect of scaling the MITL-3DO texture annotations by 2x in Figure 19, which improves human preference rate by 14.2%.

有监督微调（SFT）。 在 SFT 阶段，模型在 MITL-3DO 纹理标注上训练，其中包括第 A.1 节所述的“审美”样本。我们在图 19 中展示了把 MITL-3DO 纹理标注扩大 2 倍的效果，人人偏好率提高了 14.2%。

Preference optimization. Like the Geometry model, we run a final DPO stage to align the model with human preferences collected from the texture data engine. The effect of DPO on texture performance is shown in both Table 4 and Figure 19. We follow the same training objective described in Equation (C.3).

偏好优化。 与几何模型一样，我们再跑一个最终的 DPO 阶段，把模型对齐到从纹理数据引擎收集的人的偏好。DPO 对纹理性能的影响在表 4 和图 19 中都有展示。我们沿用公式（C.3）中描述的同一训练目标。

C.6 Texture & Refinement VAE

C.6 纹理与细化的 VAE

We make improvements over the original SLAT VAE design in Xiang et al. (2025), where features are back-projected to all voxels, including those that are not visible (i.e ., occluded) from the current image. This original design choice leads to reduced sharpness in the reconstructed images. To address this issue, we back-project features only to voxels that are visible from each image, utilizing the depth information from that specific view. We call this VAE variant Depth-VAE. During training, we normalize the Kullback-Leibler (KL) regularization term by the active voxel count to prevent large objects from dominating the training loss. We also fine-tune the decoder for downstream needs, such as reducing the number of decoded Gaussians for faster inference.

我们相对 Xiang et al. (2025) 中原来的 SLAT VAE 设计做了改进：原先特征会被反投影到所有体素，包括从当前图像不可见（也就是被遮挡）的体素。这一原始设计选择会降低重建图像的锐度。为了解决这个问题，我们只把特征反投影到从每张图像可见的体素，并利用该特定视图的深度信息。我们把这一 VAE 变体称为 Depth-VAE。在训练中，我们按活跃体素数对 Kullback-Leibler（KL）正则项做归一化，以防止大物体主导训练损失。我们还为下游需求微调解码器，例如减少解码出的高斯数量以加快推理。

\mathbf{F}\in\mathbb{R}^{B\times C\times H\times W}

\mathbf{F}\in\mathbb{R}^{B\times C\times H\times W}

\mathbf{u}_{i}\in\mathbf{U}

\mathbf{u}_{i}\in\mathbf{U}

\mathbf{U}

\mathbf{U}

\mathbf{D}_{\text{surf}}(x,y)=\min_{i:(x_{i},y_{i})=(x,y)}\hat{d}_{i}. \tag{9}

\mathbf{U}

\mathbf{U}

\mathbf{M}

\mathbf{M}

\mathbf{M}_{i}=\mathbb{I}\left[\mathbf{d}_{\text{ref},i}>\hat{d}_{i}-\tau\right]. \tag{10}

\tilde{\mathbf{M}}

\tilde{\mathbf{M}}

Weighted Aggregation. The final depth-aware representation is the weighted sum of visible patch features:

加权聚合。 最终的有深度意识表征，是可见块特征的加权和：

\mathbf{F}_{\text{depth}}=\sum_{b}\tilde{\mathbf{M}}_{b}\odot\mathbf{f}_{b}. \tag{11}

C.7 Training Hyperparameters

C.7 训练超参数

10^{-4}

Training stage	Datasets	Condition input	Learning rate	Modality weights	# Meshes	# Images	# Samples	# Tokens
Pre-training	Iso-3DO	object-centric crop	$10^{-4}$	$S{=}1.0,R{=}0.1$	2.7M	64.8M	64.8M	2.5T
Mid-training	RP-3DO (FO)	full image	$10^{-4}$	$S{=}1.0,R{=}0.1$	2.87M	11.17M	55.1M	2.4T
RP-3DO (OS-R), ProcThor	full image, pointmap	$10^{-4}$	$S{=}1.0,R{=}0.1,t{=}1.0,s{=}0.1$	2.38M	1.20M	5.95M	0.3T
SFT	MITL-3DO, Art-3DO	full image, pointmap	$10^{-5}$	$S{=}1.0,R{=}0.1,t{=}1.0,s{=}0.1$	0.6M	0.5M	0.6M	0.9T
Alignment	MITL preference	full image, pointmap	$2.5\times 10^{-6}$	$S{=}1.0$	88K	31K	44K	-

10^{-4}

训练阶段	数据集	条件输入	学习率	模态权重	网格数	图像数	样本数	token 数
预训练	Iso-3DO	以物体为中心的裁剪	$10^{-4}$	$S{=}1.0,R{=}0.1$	2.7M	64.8M	64.8M	2.5T
中期训练	RP-3DO（FO）	整图	$10^{-4}$	$S{=}1.0,R{=}0.1$	2.87M	11.17M	55.1M	2.4T
	RP-3DO（OS-R），ProcThor	整图，点图	$10^{-4}$	$S{=}1.0,R{=}0.1,t{=}1.0,s{=}0.1$	2.38M	1.20M	5.95M	0.3T
SFT	MITL-3DO，Art-3DO	整图，点图	$10^{-5}$	$S{=}1.0,R{=}0.1,t{=}1.0,s{=}0.1$	0.6M	0.5M	0.6M	0.9T
对齐	MITL 偏好	整图，点图	$2.5\times 10^{-6}$	$S{=}1.0$	88K	31K	44K	-

10^{-4}

Training stage	Datasets	Condition input	Learning rate	EMA	# Meshes	# Images	# Samples	# Tokens
Pre-training	Trellis500K	object-centric crop	$10^{-4}$	0.9999	350K	9M	10M	1.1T
Mid-training	RP-3DO (FO,OS-A)	full image	$10^{-4}$	0.9999	800K	2.4M	2.4M	1T
SFT	MITL	full image	$10^{-5}$	0.999	$\sim$ 100K	100K	100K	115B
Alignment	MITL preference	full image	$10^{-6}$	0.99	146K	73K	73K	-

10^{-4}

训练阶段	数据集	条件输入	学习率	EMA	网格数	图像数	样本数	token 数
预训练	Trellis500K	以物体为中心的裁剪	$10^{-4}$	0.9999	350K	9M	10M	1.1T
中期训练	RP-3DO（FO，OS-A）	整图	$10^{-4}$	0.9999	800K	2.4M	2.4M	1T
SFT	MITL	整图	$10^{-5}$	0.999	$\sim$ 100K	100K	100K	115B
对齐	MITL 偏好	整图	$10^{-6}$	0.99	146K	73K	73K	-

We summarize training parameters in details in Table 5 for Geometry model and Table 6 for Texture & Refinement model.

我们在表 5 中详细汇总几何模型的训练参数，在表 6 中汇总纹理与细化模型的训练参数。

\alpha_{k}

\alpha_{k}

For the Texture & Refinement model, we perform pretraining on 256 A100s for 245 epochs with a batch size of 4, followed by mid-training on 256 A100s for 80 epochs with a batch size of 4. For SFT, we use 192 A100s for 89 epochs and batch size of 4. Finally, DPO is conducted on 128 A100s for 2 epochs with a batch size of 3.

对于纹理与细化模型，我们先在 256 块 A100 上以批次大小 4 预训练 245 个 epoch，再在 256 块 A100 上以批次大小 4 做 80 个 epoch 的中期训练。SFT 使用 192 块 A100，做 89 个 epoch，批次大小为 4。最后，DPO 在 128 块 A100 上进行 2 个 epoch，批次大小为 3。

Appendix D Evaluation

附录 D 评测

Current evaluation benchmarks for visually grounded 3D object reconstruction fall short of capturing the complexity of the real world. Many rely on synthetic datasets (Deitke et al., 2023; Chang et al., 2015) where single objects are rendered in isolation, centered against a white background. This introduces a large visual gap with real-world evaluation conditions and the rich variation of real-world imagery. Efforts to move to real data mostly focus on indoor environments (Khanna et al., 2024; Sun et al., 2018; Pan et al., 2023), but these benchmarks heavily skew toward furniture categories such as chairs and tables, limiting the diversity of objects that models must handle in practice. As a result, evaluations on these datasets do not reflect the challenges of natural environments, where objects are occluded, scenes are cluttered, scales vary, lighting conditions complicate appearance, and domains span far beyond indoor or synthetic scenes.

目前针对有视觉依据的三维物体重建的评测基准，还不足以捕捉真实世界的复杂性。许多基准依赖合成数据集（Deitke et al., 2023；Chang et al., 2015），其中单个物体被孤立渲染，居中放在白色背景前。这与真实世界的评测条件、以及真实世界图像的丰富变化之间，引入了很大的视觉差距。转向真实数据的努力大多聚焦于室内环境（Khanna et al., 2024；Sun et al., 2018；Pan et al., 2023），但这些基准严重偏向椅子、桌子一类家具类别，限制了模型在实践中必须处理的物体多样性。因此，在这些数据集上的评测并不能反映自然环境中的挑战：物体会被遮挡，场景会杂乱，尺度会变化，光照条件会让外观更复杂，而且所跨领域远不止室内或合成场景。

D.1 SA-3DAO: A New Benchmark for Real-World 3D Object Reconstruction

D.1 SA-3DAO：一个面向真实世界三维物体重建的新基准

We introduce SAM 3D Artist Objects (SA-3DAO), a new benchmark designed to capture the diversity and complexity of real-world 3D perception. SA-3DAO consists of 1,000 untextured 3D objects, created from and carefully aligned to selected natural images capturing scenes spanning both indoor and outdoor environments, including parks, ski resorts, flea markets, parades and more. The benchmark covers a wide spectrum of object types: objects range from large, structured entities such as ski lifts and escalators, to everyday items like clothing, to rare and culturally specific objects such as tribal face masks. Crucially, the 3D ground truth are of high-fidelity created by professional 3D artists, who are tasked with producing accurate 3D shapes for the objects depicted in the input images. This combination of visual diversity, real-world context, and professionally crafted 3D ground truth makes SA-3DAO a comprehensive testbed for evaluating 3D object reconstruction models.

我们引入 SAM 3D Artist Objects（SA-3DAO），这是一个旨在捕捉真实世界三维感知的多样性和复杂性的新基准。SA-3DAO 包含 1,000 个无纹理三维物体，它们根据所选自然图像创建并仔细对齐，这些图像捕捉的场景既包括室内也包括室外环境，如公园、滑雪场、跳蚤市场、游行等。该基准覆盖了广泛的物体类型：从滑雪缆车和自动扶梯这类大型、有结构的实体，到衣物这类日常物品，再到部落面具这类稀有、具有文化特殊性的物体。至关重要的是，三维真值具有高保真度，由专业三维艺术家制作，他们的任务是为输入图像中所描绘的物体给出准确的三维形状。视觉多样性、真实世界上下文，以及专业制作的三维真值结合在一起，使 SA-3DAO 成为评测三维物体重建模型的一个全面试验台。

Figure 18: Distribution of number of objects per image in SA-3DAO. The number of objects follows a roughly power-law distribution.

图 18： SA-3DAO 中每张图像的物体数量分布 。 物体数量大致遵循幂律分布。

Collection details. We task professional 3D artists with recovering the shape of a target object from a single image, mirroring our model’s goal of reversing the photographic transform (as described in Section 2.1). In other words, the 3D artists must create a whole 3D mesh that precisely aligns with the object’s visible pixels in the image. Even under ideal settings, this requires contending with only partial information as the back side of the object is typically unseen, but many of the objects in SA-3DAO additionally have natural occlusions, or are small in size within the image; disentangling depth versus scale is often also challenging for a single image. To fill these information gaps, artists rely on recognition and context, using common-sense priors, physical plausibility, and assumptions of symmetry (when appropriate) to complete the meshes. The requirements imposed by this task are atypical to the normal 3D asset creation process that artists are more accustomed to, and efficient annotation requires learning a different mode of operation. After acclimation to the task, completion time per object mesh can vary considerably, ranging from up to 5 minutes for obvious objects with simple geometries to over 5 hours for more challenging cases; the median mesh in our dataset has 4751 vertices. Many of the images provided multiple objects with meshes from the 3D artists; we show the frequency distribution in Figure 18.

收集细节。 我们请专业三维艺术家从单张图像恢复目标物体的形状，这与我们模型的目标相一致：反转摄影变换（如第 2.1 节所述）。换句话说，三维艺术家必须创建一个完整的三维网格，使其与图像中该物体的可见像素精确对齐。即便在理想设定下，这也需要应对只有部分信息的情况，因为物体背面通常看不见；而 SA-3DAO 中的许多物体还有自然遮挡，或在图像中尺寸很小；对单张图像而言，把深度与尺度分开也常常很有挑战。为了填补这些信息缺口，艺术家依赖识别和上下文，使用常识先验、物理合理性，以及（在适当时候）对称性假设来补全网格。这一任务所施加的要求，不同于艺术家更习惯的常规三维资产创建流程，高效标注需要学会另一种工作方式。适应任务之后，每个物体网格的完成时间仍可能相差很大，从几何简单、一目了然的物体最多 5 分钟，到更有挑战的情形超过 5 小时；我们数据集中网格顶点数的中位数是 4751。许多图像提供了多个由三维艺术家给出网格的物体；我们在图 18 中展示频次分布。

D.2 Human Preference Set

D.2 人人偏好集

We further expand our evaluation suite to support more rigorous and domain-targeted assessments. While SA-3DAO provides a general and standardized way to measure progress, we want to also capture the challenges of settings where 3D perception is most critical, such as robotic manipulation and egocentric vision. To address this, we design a human preference set composed of images drawn from these domains of interest. This set enables evaluation through direct human judgment, providing insights that go beyond numerical metrics and capturing aspects of 3D perception that are important in embodied and real-world applications.

我们进一步扩展评测套件，以支持更严格、更针对特定领域的评估。虽然 SA-3DAO 提供了一种通用且标准化的进度衡量方式，我们也希望捕捉三维感知最为关键的那些设定中的挑战，例如机器人操作和第一人称视觉。为此，我们设计了一个人人偏好集，由从这些关注领域抽取的图像组成。这一集合通过直接的人的判断来做评测，提供超越数值指标的洞察，并捕捉在具身和真实世界应用中重要的三维感知方面。

Domains. We design four human preference test datasets to comprehensively evaluate model capabilities across different scenarios.

领域。 我们设计了四份人人偏好测试数据集，以全面评估模型在不同场景下的能力。

• SA-1B (Kirillov et al., 2023): We uniformly sample 1,000 image and object mask pairs, covering a diverse range of object categories. This set is intended to assess the model’s generalization ability across varied object distributions, with particular emphasis on long-tail categories.

• SA-1B （Kirillov et al., 2023）：我们均匀采样 1,000 对图像和物体掩码，覆盖多样的物体类别。这一集合旨在评估模型在不同物体分布上的泛化能力，尤其强调长尾类别。

• MetaCLIP (Xu et al., 2024a): We select 1,000 samples where the object masks are of median or heavily occluded. This set evaluates the model’s performance in reconstructing occluded objects, a common challenge in cluttered scenes.

• MetaCLIP （Xu et al., 2024a）：我们选出 1,000 个物体掩码为中等或严重遮挡的样本。这一集合评估模型在重建被遮挡物体上的表现，这是杂乱场景中的常见挑战。

• LVIS (Gupta et al., 2019): We densely sample 1,000 images containing between 10 and 30 objects per scene, and is designed to evaluate the model’s transferability to out-of-domain data and to demonstrate its ability to capture physical properties within dense scene layouts.

• LVIS （Gupta et al., 2019）：我们密集采样 1,000 张每场景含 10 到 30 个物体的图像，旨在评估模型向领域外数据的可迁移性，并展示它在密集场景布局中捕捉物理属性的能力。

• Aria Digital Twin (Pan et al., 2023): We sample a smaller set of 40 video frames, with around 30 objects per scene. This dataset is intended to compare against baselines on scenes with highly accurate pointmaps, similar to those on which the baselines were trained.

• Aria Digital Twin （Pan et al., 2023）：我们采样一个较小的约 40 个视频帧集合，每场景大约 30 个物体。这一数据集旨在在具有高度准确点图的场景上与基线比较，这些点图类似于基线所训练的那些。

Setup. Human preference evaluations are conducted through a structured sequence of pairwise comparisons. For each image and a masked object, annotators are first presented with two reconstructions (model “A” vs. “B”) and asked to select the one that most accurately matches the object in the image. The chosen reconstruction is then compared against the output of a third model (model “C”), and this process continues iteratively until all candidate models have been compared. Through this series of binary decisions, the most accurate reconstruction is identified as the preference for that particular image. To ensure fairness and avoid bias, the order of comparisons is randomized and the identity of models are anonymized.

设置。 人人偏好评测通过一组结构化的成对比较来进行。对每张图像和一个被掩码的物体，标注员首先看到两次重建（模型“A”对“B”），并被要求选出与图像中物体最准确匹配的那一次。被选中的重建再与第三个模型（模型“C”）的输出比较，这一过程迭代进行，直到所有候选模型都已被比较。通过这一系列二选一的决定，最准确的重建被认定为对该特定图像的偏好。为了保证公平并避免偏差，比较顺序是随机的，模型身份是匿名的。

D.3 Evaluation Metrics

D.3 评测指标

[-1,1]

[-1,1]

0.01

0.01

64^{3}

64^{3}

• Chamfer Distance (CD): Measures bidirectional nearest-neighbor distance between reconstructed and ground-truth point sets, highlighting fine-grained geometric deviation and penalizing missing or distorted regions.

• 倒角距离（CD） ： 衡量重建点集与真值点集之间的双向最近邻距离，突出细粒度几何偏差，并对缺失或变形区域加以惩罚。

• Earth Mover’s Distance (EMD): Quantifies the minimal cost required to transport one point distribution to match the other. EMD is more stringent than CD, capturing global structural differences and enforcing bijective correspondence between distributions.

• 推土机距离（EMD） ： 量化把一个点分布搬运到与另一个相匹配所需的最小代价。EMD 比 CD 更严格，捕捉全局结构差异，并强制分布之间的双射对应。

Together, these metrics provide a comprehensive view of reconstruction fidelity, from local accuracy to global shape consistency.

这些指标合在一起，从局部精度到全局形状一致性，为重建保真度提供了一个全面视角。

101

101

D.3.2 Layout Metrics Definitions To evaluate single-object pose and compare with existing methods, we employ standard 6D pose estimation metrics, and then define ICP rotation error below.

D.3.2 布局指标的定义 为了评估单物体姿态并与已有方法比较，我们采用标准的 6D 姿态估计指标，并在下面定义 ICP 旋转误差。

• 3D IoU: Measures the overlap of 3D axis-aligned bounding boxes between predicted and ground-truth bounding boxes, using the intersection-over-union. Values range from 0 (no overlap) to 1 (perfect overlap).

• 3D IoU ： 用交并比衡量预测边界框与真值边界框之间三维轴对齐边界框的重叠。取值从 0（无重叠）到 1（完全重叠）。

{R}_{\text{pred}}

{R}_{\text{pred}}

\displaystyle\text{ADD}(\mathcal{A},\mathcal{B})

\displaystyle\text{ADD}(\mathcal{A},\mathcal{B})

• ADD-S @ 0.1: A binary value per-sample indicating whether the ADD-S distance is less than 10% of the object’s diameter.

• ADD-S @ 0.1 ： 每个样本的一个二值量，表示 ADD-S 距离是否小于物体直径的 10%。

Appendix E Additional Ablations

附录 E 更多消融

E.1 Intermediate Training Stage Knockout

E.1 去掉中间训练阶段

While Table 4 in the main paper shows the cumulative effect of adding different stages during training, Table 7 shows the impact of real-world data as intermediate stages. Knocking out any of these stages results in a notable drop in shape performance.

正文中的表 4 展示了训练过程中逐步加入不同阶段的累积效果，而表 7 展示的是作为中间阶段的真实世界数据的影响。去掉这些阶段中的任何一个，都会使形状性能明显下降。

\uparrow

		SA-3DAO
Model	Training Setup	F1 @ 0.01 ( $\uparrow$ )	vIoU ( $\uparrow$ )	Chamfer ( $\downarrow$ )	EMD ( $\downarrow$ )
SAM 3D	Full	0.2344	0.2311	0.0400	0.1211
	w/o training on MITL-3DO	0.2211	0.2220	0.0486	0.1338
	w/o training on Art-3DO	0.2027	0.2025	0.0578	0.1510
	w/o DPO on MITL-3DO	0.2156	0.2156	0.0498	0.1367

\uparrow

		SA-3DAO
模型	训练设定	F1 @ 0.01 ( $\uparrow$ )	vIoU ( $\uparrow$ )	Chamfer ( $\downarrow$ )	EMD ( $\downarrow$ )
SAM 3D	完整	0.2344	0.2311	0.0400	0.1211
	不在 MITL-3DO 上训练	0.2211	0.2220	0.0486	0.1338
	不在 Art-3DO 上训练	0.2027	0.2025	0.0578	0.1510
	不在 MITL-3DO 上做 DPO	0.2156	0.2156	0.0498	0.1367

E.2 Texture Evaluations

E.2 纹理评测

Comparison to SOTA. We compare SAM 3D with existing methods on a holistic level (combined geometry and texture prediction) in Table 8. We compare against existing image-to-3D methods that predict Gaussians or textured meshes, including Trellis (Xiang et al., 2025) and Hunyuan3D-2.1 (Hunyuan3D et al., 2025). We also conduct a texture-only comparison by providing SAM 3D geometry as input to the texture modules of the aforementioned baselines, with the addition of Unitex (Liang et al., 2025b), a model that performs texture prediction given paired image and shape input.

与最好方法的比较。 我们在表 8 中从整体层面（几何与纹理的联合预测）把 SAM 3D 与已有方法比较。我们比较已有的、预测高斯或带纹理网格的图像到三维方法，包括 Trellis（Xiang et al., 2025）和 Hunyuan3D-2.1（Hunyuan3D et al., 2025）。我们还做了仅纹理的比较：把 SAM 3D 的几何作为输入，交给上述基线的纹理模块，并另外加入 Unitex（Liang et al., 2025b），这是一个在成对的图像和形状输入下做纹理预测的模型。

SAM 3D WR over baselines, SAM 3D shape Model ISO3D Preference Set SA-3DAO LVIS Trellis 81.1 87.0 86.2 89.1 Hunyuan3D-2.1 63.8 87.0 86.2 89.1 Hunyuan3D-2.0 70.1 77.5 77.4 85.7 Unitex 83.3 84.7 84.5 88.3 Table 8: 3D texture. Preference results comparing SAM 3D to competing image-to-3D methods on ISO3D (Ebert, 2025), Preference Set, and SA-3DAO. We compare to the recent Trellis (Xiang et al., 2025), Hunyuan3D-2.1 (Hunyuan3D et al., 2025), and Unitex (Liang et al., 2025b), with the same shape of SAM 3D. The human preference rates represent preference for SAM 3D over each baseline approach.

	SAM 3D WR over baselines, SAM 3D shape
Model	ISO3D	Preference Set	SA-3DAO	LVIS
Trellis	81.1	87.0	86.2	89.1
Hunyuan3D-2.1	63.8	87.0	86.2	89.1
Hunyuan3D-2.0	70.1	77.5	77.4	85.7
Unitex	83.3	84.7	84.5	88.3

给定 SAM 3D 形状时，SAM 3D 相对基线的胜率 模型 ISO3D 偏好集 SA-3DAO LVIS Trellis 81.1 87.0 86.2 89.1 Hunyuan3D-2.1 63.8 87.0 86.2 89.1 Hunyuan3D-2.0 70.1 77.5 77.4 85.7 Unitex 83.3 84.7 84.5 88.3 表 8： 三维纹理 。 在 ISO3D（Ebert, 2025）、偏好集和 SA-3DAO 上，把 SAM 3D 与竞品图像到三维方法比较的偏好结果。我们在使用同一份 SAM 3D 形状的条件下，与近期的 Trellis（Xiang et al., 2025）、Hunyuan3D-2.1（Hunyuan3D et al., 2025）和 Unitex（Liang et al., 2025b）比较。人人偏好率表示相对每种基线方法对 SAM 3D 的偏好。

	给定 SAM 3D 形状时，SAM 3D 相对基线的胜率
模型	ISO3D	偏好集	SA-3DAO	LVIS
Trellis	81.1	87.0	86.2	89.1
Hunyuan3D-2.1	63.8	87.0	86.2	89.1
Hunyuan3D-2.0	70.1	77.5	77.4	85.7
Unitex	83.3	84.7	84.5	88.3

We report human preference for SAM 3D over each baseline method on multiple datasets: ISO3D (Ebert, 2025), Preference Set, SA-3DAO, and LVIS (Gupta et al., 2019). Results indicate that SAM 3D outperforms existing methods on the holistic image-to-3D task as well as texture estimation - this is due to the fact that the evaluated datasets often contain occlusion and clutter, which is a setting prior works struggle on.

我们在多个数据集上报告相对每种基线方法对 SAM 3D 的人人偏好：ISO3D（Ebert, 2025）、偏好集、SA-3DAO，以及 LVIS（Gupta et al., 2019）。结果表明，SAM 3D 在整体的图像到三维任务以及纹理估计上都优于已有方法——这是因为所评测的数据集常常包含遮挡和杂乱，而这正是先前工作难以应对的设定。

Ablations for Texture Training. We conduct comprehensive studies on design choices for Texture & Refinement model (Figure 19) using annotator preferences on the Pref Set. We benchmark each component to the alternative model without the change. We remark a few themes here:

纹理训练的消融。 我们用 Pref Set 上标注员的偏好，对纹理与细化模型的设计选择做了全面研究（图 19）。我们把每个组件与去掉该改动的对照模型做基准比较。这里我们指出几个主题：

Figure 19: Ablations for the Texture & Refinement model, grouped by training stage. Percentages denote human preference rate for each ablation, over the model without the ablation.

图 19： 按训练阶段分组的纹理与细化模型消融 。 百分比表示相对去掉该消融后的模型，对各消融的人人偏好率。

• Augmentation is very important, with lighting augmentation to be the most critical here. This is expected, given the Mask and Blur augmentations primarily focus on specific challenging cases (poor mask quality and low resolution inputs), so their effects get diluted in a holistic evaluation.

• 增强非常重要，其中光照增强在这里最为关键。这是可以预期的，因为掩码增强和模糊增强主要针对特定的有挑战情形（掩码质量差和低分辨率输入），所以它们的效果在整体评测中会被稀释。

• RP-3DO data are critical and helps the model adapt to real world.

• RP-3DO 数据至关重要，能帮助模型适应真实世界。

• Post-training data are critical, with significant gains coming from it. It demonstrates the effectiveness of our Data Engine, and DPO further amplifies the gains. In addition, sourcing specific type of data (AES) and scaling the data both show significant improvements.

• 后训练数据至关重要，其中有显著收益来自它。这展示了我们数据引擎的有效性，而 DPO 进一步放大了这些收益。此外，引入特定类型的数据（AES）以及扩大数据规模都显示出显著改进。

E.3 Layout Test-Time Optimization

E.3 布局的测试时优化

Render-and-compare is a longstanding popular approach for pose estimation (Labbé et al., 2022; Wen et al., 2024): iteratively render the object’s shape according to the most recently predicted pose, directly compare with the input pixels, and then adjust the pose prediction accordingly. This heuristic-based search can lead to fairly accurate results, even with weak initial pose samples (“proposals” from a base model). By contrast, SAM 3D operates in a feedforward manner, directly diffusing the object pose (rotation, translation, and scale) conditioned on the image features and, optionally, the scene pointmap. Notably, we do not include any sort of pixel-based loss objective in this process. However, SAM 3D’s pose estimation can very naturally be used as a proposal for render-and-compare optimization. We include experiments showing the impact of this post-optimization process in Table 9, demonstrating that we can achieve further gains in pose metrics on ADT (Pan et al., 2023).

渲染并比较是姿态估计中长期流行的做法（Labbé et al., 2022；Wen et al., 2024）：按最近一次预测的姿态迭代渲染物体的形状，直接与输入像素比较，再据此调整姿态预测。这种基于启发式的搜索可以给出相当准确的结果，即便初始姿态样本较弱（来自基座模型的“提案”）。相比之下，SAM 3D 以前馈方式工作，在图像特征以及（可选的）场景点图的条件下，直接扩散物体姿态（旋转、平移和尺度）。值得注意的是，我们在这一过程中不包含任何基于像素的损失目标。不过，SAM 3D 的姿态估计可以很自然地用作渲染并比较优化的提案。我们在表 9 中给出展示这一后优化过程影响的实验，表明我们可以在 ADT（Pan et al., 2023）的姿态指标上取得进一步提升。

\uparrow

Model	3D IoU ( $\uparrow$ )	ICP-Rot. ( $\downarrow$ )	ADD-S ( $\downarrow$ )	ADD-S @ 0.1 ( $\uparrow$ )	2D IoU ( $\uparrow$ )
SAM 3D	0.4837	14.4702	0.08265	0.7545	0.5143
SAM 3D (post-optim)	0.5258	14.1460	0.07932	0.7617	0.6487

\uparrow

模型	3D IoU ( $\uparrow$ )	ICP-Rot. ( $\downarrow$ )	ADD-S ( $\downarrow$ )	ADD-S @ 0.1 ( $\uparrow$ )	2D IoU ( $\uparrow$ )
SAM 3D	0.4837	14.4702	0.08265	0.7545	0.5143
SAM 3D（后优化）	0.5258	14.1460	0.07932	0.7617	0.6487

554

1027

E.4 Rotation Representation

E.4 旋转表征

We compare different rotation representations while keeping all other architecture and training settings identical. Each model is trained on the pretraining dataset and evaluated on a held-out Objaverse test split of 216 samples. As shown in Table 10, switching from a quaternion representation to the 6D continuous rotation parameterization (Zhou et al., 2019) yields a notable reduction in oriented rotation error, confirming that the 6D formulation provides a smoother optimization landscape more suitable for generative modeling. Further applying normalization to the 6D rotation vectors using the statistics over the training datasets leads to an additional improvement when training the flow matching models.

我们在保持所有其他结构和训练设定相同的条件下，比较不同的旋转表征。每个模型都在预训练数据集上训练，并在一份留出的、含 216 个样本的 Objaverse 测试划分上评测。如表 10 所示，从四元数表征改到 6D 连续旋转参数化（Zhou et al., 2019），会明显降低有朝向的旋转误差，这确认了 6D 公式提供了更平滑、更适合生成建模的优化地形。再进一步用训练数据集上的统计量对 6D 旋转向量做归一化，在训练流匹配模型时还会带来额外改进。

\downarrow

Representation	Chamfer ( $\downarrow$ )	ICP-Rot. ( $\downarrow$ )
Quaternion	0.0061	17.9585
6D Rotation	0.0074	15.5399
Normalized 6D Rotation	0.0049	14.5946

\downarrow

表征	Chamfer ( $\downarrow$ )	ICP-Rot. ( $\downarrow$ )
四元数	0.0061	17.9585
6D 旋转	0.0074	15.5399
归一化 6D 旋转	0.0049	14.5946

E.5 Pointmap Minimally Affects Shape

E.5 点图对形状的影响很小

SAM 3D can condition on a 2.5D pointmap derived from sensor measurements (see Section 2.2) or from an off-the-shelf monocular depth estimation derived from the image itself, as is primarily used throughout this work. The latter is notable, as it also means that the current model can continue to benefit from future improvements of depth estimation methods. We observe that the pointmap minimally affects the shape performance: in a head-to-head preference test for shape on LVIS, the version of SAM 3D conditioned on pointmaps and the version without pointmaps are each selected 48% of the time.

SAM 3D 可以以从传感器测量得到的 2.5D 点图为条件（见第 2.2 节），也可以以由图像本身得到的现成单目深度估计为条件，而本文主要使用后者。后者值得注意，因为它也意味着当前模型可以继续受益于未来深度估计方法的改进。我们观察到，点图对形状性能的影响很小：在 LVIS 上针对形状的两两偏好测试中，以点图为条件的 SAM 3D 版本和不以点图为条件的版本各自被选中 48% 的次数。

E.6 Texture & Refinement Depth-VAE Comparison

E.6 纹理与细化 Depth-VAE 的比较

Table 11 shows the result of the improvements made to the VAE used in the Texture & Refinement model; see Section C.6. We found that the depth feature significantly improves the perceptual quality of reconstruction, while scaling the training data further improves the reconstruction performance. We also notice that the enhancement primarily arises from the difficult scenarios for the non-depth VAE (when it has a bad performance).

表 11 给出对纹理与细化模型所用 VAE 所做改进的结果；见第 C.6 节。我们发现，深度特征显著提高了重建的感知质量，而扩大训练数据则进一步改进重建性能。我们还注意到，提升主要来自对非深度 VAE 困难的那些情形（当它表现较差时）。

\uparrow

Method	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )
Non-Depth VAE	30.65	0.9470	0.04776
Depth-VAE	30.87	0.9500	0.04579
Depth-VAE + scaling	31.60	0.9547	0.04093

\uparrow

方法	PSNR ( $\uparrow$ )	SSIM ( $\uparrow$ )	LPIPS ( $\downarrow$ )
非深度 VAE	30.65	0.9470	0.04776
Depth-VAE	30.87	0.9500	0.04579
Depth-VAE + 扩大规模	31.60	0.9547	0.04093

N

N

\downarrow

	Tail Holdout	Epic Kitchens	SA-3DAO
	Chamfer $\downarrow$	F1 $\uparrow$	Chamfer $\downarrow$	F1 $\uparrow$	Chamfer $\downarrow$	F1 $\uparrow$
SFT with $N=2$	0.0059	0.39	0.0094	0.30	0.0083	0.26
SFT with $N=50$ recovery	0.0053	0.41	0.0090	0.32	0.0081	0.26

\downarrow

	尾部留出集	Epic Kitchens	SA-3DAO
	Chamfer $\downarrow$	F1 $\uparrow$	Chamfer $\downarrow$	F1 $\uparrow$	Chamfer $\downarrow$	F1 $\uparrow$
用 $N=2$ 做 SFT	0.0059	0.39	0.0094	0.30	0.0083	0.26
用 $N=50$ 恢复后做 SFT	0.0053	0.41	0.0090	0.32	0.0081	0.26

N

N

68.9\%

68.9\%

E.8 Model Distillation Results

E.8 模型蒸馏结果

Figure 20: Model distillation. Geometry model shortcut versus flow matching. Flow matching distillation enables the model to perform significantly better during the early iterations, nearly on par with 25 steps performance.

图 20： 模型蒸馏 。 几何模型的捷径与流匹配对照。流匹配蒸馏使模型在早期迭代中表现显著更好，几乎与 25 步的性能相当。

Figure 21: Qualitative examples after distillation. Visualization using difference number of steps in flow matching mode and shortcut mode. The black background displays the mesh rendering without texture, while the grey background shows the rendering of the Gaussian splattings.

图 21： 蒸馏后的定性例子 。 在流匹配模式和捷径模式下使用不同步数的可视化。黑色背景展示无纹理的网格渲染，灰色背景展示高斯溅射的渲染。

\times

图 20 展示了有无蒸馏时 SAM 3D 的性能，按几何模型的流匹配步数绘制。具体而言，与带 CFG 的 25 步流匹配相比，1 步和 4 步方法分别带来 38 倍和 10 倍的推理速度提升。在流匹配模式中，CFG 施加在前一半步上，因此总 NFE 是步数的 1.5 倍。相比之下，捷径模式的 NFE 等于步数，因为 CFG 被直接蒸馏进模型。对于纹理与细化模型，我们在最终发布中选择不施加蒸馏，因为最终模型在较少步数下就已经表现很好。这是因为整体几何主要由几何模型的体素输出决定，增加步数并不会显著改变几何。不过，如图 21 所示——我们展示了在几何模型以及纹理与细化模型都开启捷径模式时、使用不同推理步数的模型输出可视化——在使用较少步数时，捷径模型蒸馏会改进纹理质量。

E.9 Input Mask Robustness

E.9 输入掩码的稳健性

Figure 22: SAM 3D robustness to input mask precision. Top : Over-segmentation; Bottom : Under-segmentation.

图 22： SAM 3D 对输入掩码精度的稳健性 。 上 ：过度分割； 下 ：分割不足。

SAM 3D assumes a segmentation mask as input to designate to the model which object within a scene should be reconstructed, as well as isolate the object’s pixels for visual feature extraction. Accurate segmentation by hand can be time-consuming, but the ready availability of segmentation models like the SAM series (Kirillov et al., 2023; Ravi et al., 2025; Carion et al., 2026) offer a fast, promptable solution. On the other hand, automatic methods are not perfect, and as a preceding step to SAM 3D, there is risk of compounding errors. Yet, we observe empirically that SAM 3D has fair robustness to errors in mask precision (see Figure 22), demonstrating the ability to recognize the intended target object and reconstruct it. We also note that as an input to the model, SAM 3D is agnostic to the source of mask, and thus is able to take advantage of ground truth masks or advancements in segmentation by future models.

SAM 3D 假定以一个分割掩码作为输入，向模型指定场景中应重建哪一个物体，并把该物体的像素隔离开来以提取视觉特征。手工做准确分割可能很耗时，但像 SAM 系列（Kirillov et al., 2023；Ravi et al., 2025；Carion et al., 2026）这样现成的分割模型，提供了一种快速、可提示的解决方案。另一方面，自动方法并不完美，而作为 SAM 3D 的前序步骤，存在误差叠加的风险。不过，我们从经验上观察到，SAM 3D 对掩码精度上的误差具有相当的稳健性（见图 22），表现出识别预定目标物体并重建它的能力。我们还指出，作为模型的输入，SAM 3D 对掩码来源并不敏感，因此能够利用真值掩码，或未来模型在分割上的进展。

Appendix F Limitations

附录 F 局限

O\in\mathbb{R}^{64^{3}}

O\in\mathbb{R}^{64^{3}}

Object layouts are another area where improvements can be made. SAM 3D predicts objects one at a time, and isn’t trained to reason about physical interactions, such as contact, physical stability, interpenetration, or co-alignment (e.g . on the same ground plane). Multi-object prediction combined with appropriate losses would allow joint reasoning about multiple objects in a scene. Similarly, while SAM 3D’s single-object inference is embarrassingly parallel across multiple GPUs, scene-based inference may also present opportunities to more efficiently accelerate the reconstruction of entire scenes. Finally, SAM 3D’s texture predictions are made without knowledge of the predicted object’s pose; as a result, for objects with rotational symmetry, it can occasionally predict textures that effectively rotate the object to the incorrect orientation.

物体布局是另一个可以改进的方面。SAM 3D 一次预测一个物体，并且没有被训练去推理物理交互，例如接触、物理稳定性、相互穿透，或共同对齐（例如在同一地平面上）。多物体预测再配以适当的损失，将允许对场景中的多个物体做联合推理。类似地，虽然 SAM 3D 的单物体推理在多块 GPU 上是完全可并行的（embarrassingly parallel），基于场景的推理也可能提供更有效地加速整场景重建的机会。最后，SAM 3D 的纹理预测是在不知道所预测物体姿态的情况下做出的；因此，对于具有旋转对称性的物体，它偶尔会预测出实际上把物体转到错误朝向的纹理。

Figure 23: Additional qualitative shape and texture results of our model on the SA-3DAO eval set. For models that include texture, we show the untextured mesh (left) and textured mesh (right) separately.

图 23： 我们的模型在 SA-3DAO 评测集上的更多定性形状与纹理结果 。 对于包含纹理的模型，我们分别展示无纹理网格（左）和有纹理网格（右）。

Figure 24: Qualitative examples for scene reconstruction. Showing examples of SAM 3D and alternative scene reconstruction methods.

图 24： 场景重建的定性例子 。 展示 SAM 3D 与其他场景重建方法的例子。

SAM 3D: 3Dfy Anything in Images

SAM 3D：把图像里的任意物体三维化

Abstract

摘要

1 Introduction

1 引言

2 The SAM 3D Model

2 SAM 3D 模型

2.1 Problem Formulation

2.1 问题形式化

2.2 Architecture

2.2 结构

Input encoding.

输入编码。

The Geometry Model

几何模型

The Texture & Refinement Model

纹理与细化模型

3D Decoders.

三维解码器。

3 Training SAM 3D

3 训练 SAM 3D

3.1 Pre & Mid-Training: Building a Base Model

3.1 预训练与中期训练：构建基座模型

3.1.1 Pretraining: Single Isolated 3D Assets

3.1.1 预训练：单个孤立的三维资产

3.1.2 Mid-Training: Semi-Synthetic Capabilities

3.1.2 中期训练：半合成能力

3.2 Post-Training: Real-World Alignment

3.2 后训练：真实世界对齐

3.2.1 Post-Training: Collection Step

3.2.1 后训练：收集步骤

Stage 1: Choosing target objects (I,M)(I,M)(I,M).

第 1 阶段：选择目标物体 (I,M)(I,M)(I,M)。

Stage 2: Object model ranking and selection (S,T)(S,T)(S,T).

第 2 阶段：物体模型的排序与选择 (S,T)(S,T)(S,T)。

Stage 2.5: Hard example triage (Artists).

第 2.5 阶段：困难样本分流（艺术家）。

Stage 3: Aligning objects to 2.5D scene (R,t,s)(R,t,s)(R,t,s).

第 3 阶段：把物体对齐到 2.5D 场景 (R,t,s)(R,t,s)(R,t,s)。

3.2.2 Post-Training: Model Improvement Step

3.2.2 后训练：模型改进步骤

Supervised Fine-Tuning (SFT).

监督微调（SFT）。

Preference optimization (alignment).

偏好优化（对齐）。

Distillation.

蒸馏。

4 Experiments

4 实验

Dataset.

数据集。

Settings.

设置。

4.1 Comparison with SOTA

4.1 与当时最好方法的比较

3D shape and texture.

三维形状与纹理。

3D scene reconstruction.

三维场景重建。

4.2 Analysis Studies

4.2 分析研究

Post-training iterations steadily improve performance.

后训练迭代会稳步提升性能。

Multi-stage training improves performance.

多阶段训练会提升性能。

Other ablations.

其他消融。

5 Related Work

5 相关工作

3D reconstruction

三维重建

Single-view 3D reconstruction

单视图三维重建

Layout estimation

布局估计

3D datasets.

三维数据集。

Post-training.

后训练。

Stage 1: Choosing target objects $(I,M)$ .

第 1 阶段：选择目标物体 $(I,M)$ 。

Stage 2: Object model ranking and selection $(S,T)$ .

第 2 阶段：物体模型的排序与选择 $(S,T)$ 。

Stage 3: Aligning objects to 2.5D scene $(R,t,s)$ .

第 3 阶段：把物体对齐到 2.5D 场景 $(R,t,s)$ 。

A.7.1 Best-of- $N$ Search with Reward Models

A.7.1 用奖励模型做 Best-of- $N$ 搜索