CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image

CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image

CAST：从一张 RGB 图像做部件对齐的三维场景重建 CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image

Kaixin Yao Note: Equal contributions. Affiliation: ShanghaiTech University , Shanghai , China Affiliation: Deemos Technology Co., Ltd. , Shanghai , China email: yaokx2023@shanghaitech.edu.cn , Longwen Zhang Affiliation: ShanghaiTech University , Shanghai , China Affiliation: Deemos Technology Co., Ltd. , Shanghai , China email: zhanglw2@shanghaitech.edu.cn , Xinhao Yan Affiliation: ShanghaiTech University , Shanghai , China Affiliation: Deemos Technology Co., Ltd. , Shanghai , China email: yanxh@shanghaitech.edu.cn , Yan Zeng Affiliation: ShanghaiTech University , Shanghai , China Affiliation: Deemos Technology Co., Ltd. , Shanghai , China email: zengyan2024@shanghaitech.edu.cn , Qixuan Zhang Note: Project leader. Affiliation: ShanghaiTech University , Shanghai , China Affiliation: Deemos Technology Co., Ltd. , Shanghai , China email: zhangqx1@shanghaitech.edu.cn , Wei Yang Affiliation: Huazhong University of Science and Technology , China , Shanghai email: weiyangcs@hust.edu.cn , Lan Xu Note: Corresponding author. Affiliation: ShanghaiTech University , China , Shanghai email: xulan1@shanghaitech.edu.cn , Jiayuan Gu Affiliation: ShanghaiTech University , China , Shanghai email: gujy1@shanghaitech.edu.cn and Jingyi Yu Affiliation: ShanghaiTech University , China , Shanghai email: yujingyi@shanghaitech.edu.cn

Figure 1. CAST brings diverse 3D scenes to life from a single image, where the relations between objects shaped by their physical roles and interactions come together to form a cohesive and immersive virtual environment.

图 1. CAST 从单张图像把多样的三维场景做活。物体之间由物理角色和相互作用塑造出的关系汇合在一起，形成一个连贯、可沉浸的虚拟环境。

Abstract

摘要

从单张 RGB 图像恢复高质量三维场景，在计算机图形学里仍然很难。现有方法常常受限于特定领域，或生成的物体质量偏低。针对这些问题，我们提出 CAST（Component-Aligned 3D Scene Reconstruction from a Single RGB Image），一种新的三维场景重建方法。CAST 先从输入图像中提取物体级二维分割和相对深度，再用基于 GPT 的模型分析物体之间的空间关系。这样就能理解场景里物体如何彼此关联，使重建更加连贯。随后，CAST 使用遮挡感知的大规模三维生成模型，为每个物体独立生成完整几何；借助掩码自编码器（Masked Auto Encoder，MAE）和点云条件，减轻遮挡与物体信息不完整的影响，并使几何和纹理与源图像对齐。为了把每个物体放进场景，对齐生成模型会计算所需变换，使生成网格能够准确放置，并融入场景点云。最后，CAST 施加物理感知校正：由细粒度关系图生成约束图，引导优化物体位姿，保证物理一致性和空间连贯性。借助符号距离场（Signed Distance Fields，SDF），模型能够处理遮挡、物体穿透和悬浮，使生成场景更符合真实世界中的物理交互。实验表明，CAST 显著提升了单图像三维场景重建的质量，在场景理解和重建任务中都更真实、更准确。CAST 可用于虚拟内容创作，例如沉浸式游戏环境和影视制作，把现实布置无缝接入虚拟景观。它也可以用于机器人，提高 real-to-simulation 流程的效率，为机器人系统提供真实、可扩展的仿真环境。

1. Introduction

1. 引言

Keywords: Humans exist within clear networks of relations—family, friends, coworkers—that guide our decisions and behaviors. These connections shape our world and give it structure. Similarly, objects in a space also function within their own networks (Latour, 2005), but less noticed. They do not just exist in isolation; their placement, design, and material arise from physical constraints, functional roles, and human design intentions and influence how we move, interact, and perceive space. For example, a chair leans against a table for support, a cup rests on a saucer, and a lamp’s light interacts with surrounding surfaces, casting shadows that shape the overall scene. Recognizing these relations is critical for accurate scene parsing, modeling, and, more recently, 3D generation, ensuring virtual environments feel as realistic and coherent as the real world.

关键词： 人生活在清晰的关系网络里——家人、朋友、同事——这些关系引导我们的决策和行为。这些联结塑造世界，并给它结构。空间中的物体同样运行在自己的网络里（Latour, 2005），只是更少被注意到。它们并不孤立存在；其摆放、设计和材料来自物理约束、功能角色和人的设计意图，并影响我们如何移动、交互和感知空间。例如，椅子靠着桌子获得支撑，杯子搁在茶托上，灯的光线与周围表面相互作用，投下的影子塑造整个场景。识别这些关系，对准确的场景解析、建模，以及近年的三维生成都至关重要，才能让虚拟环境像真实世界一样可信、连贯。

Significant progress has been made in generating single objects from text or image prompts. Neural rendering approaches (Poole et al., 2022; Wang et al., 2024a) optimize implicit representations, while native 3D generators (Zhang et al., 2023; Zhang et al., 2024a; Xiang et al., 2024) directly create 3D shapes and textures via end-to-end learning. While these methods show promise for individual objects, applying them to generate entire scenes by assembling objects sequentially faces with notable shortcomings. A key challenge is accurate pose estimation. Existing methods often assume objects are view-aligned, which is rarely the case in real-world scenes. Objects may appear in diverse orientations, constrained by design, physics, or partial occlusion. Yet, most existing methods prioritize geometric fidelity over pose alignment, leaving this critical aspect underexplored.

从文本或图像提示生成单个物体，已经取得显著进展。神经渲染方法（Poole et al., 2022; Wang et al., 2024a）优化隐式表示，原生三维生成器（Zhang et al., 2023; Zhang et al., 2024a; Xiang et al., 2024）则通过端到端学习直接创建三维形状和纹理。这些方法在单个物体上表现出潜力，但若按顺序组装物体来生成整幅场景，仍有明显不足。一个关键困难是准确的姿态估计。现有方法常常假设物体已经与视角对齐，这在真实场景里很少成立。物体可能呈现多样朝向，并受到设计、物理或部分遮挡的约束。多数已有方法更重视几何保真，而把姿态对齐这一关键问题留给后面。

An even more fundamental issue arises from the lack of inter-object spatial relations. Even with accurate poses, generated scenes often suffer from physically implausible artifacts: objects penetrate one another, float, or fail to make contact where necessary. These errors stem from the absence of spatial and physical constraints that naturally bind objects together, much as human relations structure our social world. While some recent methods (Liu et al., 2022; Zhang et al., 2024b) encode spatial relations implicitly using encoder-decoder architectures, they remain limited to specific domains such as indoor scenes. Other scene-level generators (Dogaru et al., 2024) position objects in a global coordinate system but neglect their relative poses and dependencies, further compromising realism and usability for downstream applications like editing, animation, and simulation.

更根本的问题来自缺少物体之间的空间关系。即便姿态准确，生成场景也常常出现物理上不合理的伪影：物体互相穿透、悬空，或该接触的地方没有接触。这些错误源于缺少把物体自然绑在一起的空间和物理约束，就像人际关系会结构化我们的社会世界。一些近期方法（Liu et al., 2022; Zhang et al., 2024b）用编码器-解码器架构隐式编码空间关系，但仍限于室内场景等特定领域。另一些场景级生成器（Dogaru et al., 2024）把物体放进全局坐标系，却忽略相对姿态和相互依赖，进一步损害真实感，也降低了编辑、动画和仿真等下游用途。

To this end, we propose CAST, a C omponent- A ligned 3D S cene reconstruction method for compositional reconstruction of a 3D scene from a single RGB image. CAST generates high-quality 3D meshes for individual objects, along with their similarity transformations (rotation, translation, scale), ensuring alignment with the reference image and enforcing physically sound interdependencies. CAST starts by processing an unstructured RGB image using 2D foundation models (e.g., Florence-2 (Xiao et al., 2024), GroundingDINO (Liu et al., 2025), SAM (Ravi et al., 2024), Grounded-SAM (Ren et al., 2024)) to recognize, localize, and segment objects in an open-vocabulary manner. Off-the-shelf monocular depth estimators (Wang et al., 2024b) provide partial 3D point clouds and initial estimates of inter-object spatial relations, including relative transformations and scales.

为此，我们提出 CAST，一种面向单张 RGB 图像的部件对齐三维场景重建方法，用来组合式地重建三维场景。CAST 为各个物体生成高质量三维网格，并给出相似变换（旋转、平移、缩放），使其与参考图像对齐，同时施加物理上合理的相互依赖。CAST 先用二维基础模型（例如 Florence-2（Xiao et al., 2024）、GroundingDINO（Liu et al., 2025）、SAM（Ravi et al., 2024）、Grounded-SAM（Ren et al., 2024））以开放词汇方式识别、定位并分割图像中的物体。现成的单目深度估计器（Wang et al., 2024b）提供局部三维点云，以及对物体间空间关系的初步估计，包括相对变换和尺度。

The first core component of CAST is our perceptive 3D instance generator with two modules: an occlusion-aware object generation module and a pose alignment generation module. The object generation module employs a latent diffusion-based generative model to produce high-fidelity object meshes conditioned on partial image segments and optional point clouds. This module incorporates an occlusion-aware 2D image encoder capable of inferring occluded regions, ensuring robust feature extraction for image conditions. To improve robustness to real-world point cloud conditioning, we simulate partial point clouds with occluded regions during training, enabling the model to handle occlusion effectively. The pose alignment module features an alignment generative model that produces a transformed partial point cloud, aligning with the complete geometry implicitly represented in the latent space. The similarity transformation is derived from the generated transformed point cloud and the partial point cloud estimated from the camera. Unlike direct pose regression methods (Kehl et al., 2017; Labbé et al., 2020), our method estimates transformations through generation, capturing the multi-modal nature of pose alignment.

CAST 的第一个核心部件是感知式三维实例生成器，它包含两个模块：遮挡感知的物体生成模块，以及姿态对齐生成模块。物体生成模块采用基于潜扩散的生成模型，以局部图像片段和可选点云为条件，生成高保真物体网格。该模块包含一个遮挡感知的二维图像编码器，能够推断被遮挡区域，从而为图像条件提供稳健的特征。为了提高对真实世界点云条件的鲁棒性，我们在训练时模拟带遮挡区域的残缺点云，使模型能够有效处理遮挡。姿态对齐模块包含一个对齐生成模型，它生成变换后的残缺点云，并与潜空间中隐式表示的完整几何对齐。相似变换由生成的变换点云和从相机估计的残缺点云导出。与直接回归姿态的方法（Kehl et al., 2017; Labbé et al., 2020）不同，我们通过对齐生成来估计变换，以刻画姿态对齐的多峰性。

The second core component of CAST addresses inter-object spatial relations. Despite accurate pixel alignment, physically implausible artifacts such as penetration or floating can occur without explicit modeling of physical constraints. CAST introduces a physics-aware correction process to ensure spatial and physical coherence. GPT-4v (Achiam et al., 2023) is utilized to identifies commonsense physical relations grounded in the input image, which are then used to optimize object poses based on these constraints. This process ensures that reconstructed scenes exhibit realistic physical interdependencies, making them suitable for applications like simulation, editing, and rendering.

CAST 的第二个核心部件处理物体之间的空间关系。即便像素对齐准确，如果不显式建模物理约束，仍可能出现穿透或悬浮等物理上不合理的伪影。CAST 引入物理感知校正过程，以保证空间和物理上的连贯。GPT-4v（Achiam et al., 2023）被用来识别扎根于输入图像的常识物理关系，随后根据这些约束优化物体位姿。这一过程使重建场景展现出真实的物理相互依赖，适合仿真、编辑和渲染等应用。

Remarkably, CAST excels at generating perceptually realistic 3D scenes from a wide range of images, whether they are sourced from indoor or outdoor settings, real-world captured, or AI-generated. Unlike previous approaches (Liu et al., 2022; Dai et al., 2024), CAST supports open-vocabulary reconstruction, even for challenging, in-the-wild images, thanks to our deliberate pipeline design. Quantitatively, CAST surpasses strong baselines in the indoor dataset, 3D-Front (Fu et al., 2021), regarding object- and scene-level geometry quality. It also outperforms on perceptual and physical realism across a diverse set of images, including in-the-wild scenarios, as verified by visual-language models and user studies.

值得注意的是，CAST 能够从各类图像中生成感知上真实的三维场景，无论图像来自室内还是室外、真实拍摄还是人工智能生成。与先前方法（Liu et al., 2022; Dai et al., 2024）不同，得益于我们有意设计的流程，CAST 支持开放词汇重建，即便面对具有挑战性的野外图像也是如此。在定量上，CAST 在室内数据集 3D-Front（Fu et al., 2021）上，就物体级和场景级几何质量而言超过强基线。在涵盖野外场景的多样图像上，经视觉-语言模型和用户研究验证，它在感知真实性和物理合理性上也表现更好。

Given only a single image, CAST can faithfully reconstruct the scene, with detailed geometry, vivid textures of the objects, and more importantly, the spatial and physical interdependencies between them. This capability democratizes virtual creation: a single snapshot of a room or outdoor space becomes a fully realized 3D environment, where objects are precisely posed, interact naturally, and account for occlusions. Game developers can integrate real-world setups into immersive landscapes, and filmmakers can effortlessly generate intricate virtual sets—unlocking creative potentials. Beyond entertainment, CAST paves the way for smarter robots. It can facilitate the real-to-simulation pipeline (Li et al., 2024a; Torne et al., 2024) by enabling robotics researchers to construct digital replicas from real-world demonstration datasets with more efficient and scalable simulation workflows.

仅凭单张图像，CAST 就能忠实重建场景：物体有细致的几何和生动的纹理，更重要的是，物体之间还有空间和物理上的相互依赖。这一能力降低了虚拟创作的门槛：对房间或室外空间拍一张快照，就能得到完整实现的三维环境，其中物体姿态准确、交互自然，并考虑了遮挡。游戏开发者可以把现实布置接入沉浸式景观，电影制作者也能更轻松地生成复杂的虚拟布景，从而打开新的创作空间。在娱乐之外，CAST 也为更聪明的机器人铺路。它可以通过更高效、可扩展的仿真流程，帮助机器人研究者从真实世界演示数据构建数字副本，从而推进 real-to-simulation 管线（Li et al., 2024a; Torne et al., 2024）。

2. Related Work

2. 相关工作

Transforming real-world scenes into the digital realm enhances our ability to understand, recreate, and interact with the 3D world around us. This practice is widely embraced in industries such as animation, film, gaming, architecture, and manufacturing. It enables the creation of immersive movie experiences, the digital preservation of historical relics, and the development of interactive environments for gaming. For example, James Cameron employed groundbreaking 3D scanning technology in Avatar (2009) to bring the lush, realistic world of Pandora to life. Similarly, in the gaming industry, The Witcher 3: Wild Hunt incorporated lifelike terrain and architectures inspired by real-world locations in Poland, blending authentic cultural and natural elements with imaginative, open-world exploration.

把真实世界场景转入数字领域，能增强我们理解、再现并与周围三维世界交互的能力。这一做法已广泛应用于动画、电影、游戏、建筑和制造业。它使沉浸式电影体验、历史遗迹的数字化保存，以及游戏中的交互环境成为可能。例如，James Cameron 在《阿凡达》（2009）中采用突破性的三维扫描技术，把潘多拉那片繁茂而真实的世界搬上银幕。类似地，在游戏产业中，《巫师 3：狂猎》融入了受波兰真实地点启发的逼真地形和建筑，把真实的文化和自然元素与富于想象的开放世界探索结合在一起。

Photogrammetry is a widely used method to capture the physical world in high detail and translate it into digital form (Goesele et al., 2007; Chen et al., 2018; Mildenhall et al., 2020; Barron et al., 2021; Barron et al., 2022; Müller et al., 2022; Kerbl et al., 2023), but it requires tens to hundreds of images from multiple viewpoints, making it time-consuming, resource-intensive, and hard to scale. In contrast, single-image-based approaches are more efficient and scalable, requiring only one image that can be easily obtained from online repositories, eliminating the need for expensive scanning devices or multi-view setups.

摄影测量是把物理世界高细节地捕捉并转成数字形式的常用方法（Goesele et al., 2007; Chen et al., 2018; Mildenhall et al., 2020; Barron et al., 2021; Barron et al., 2022; Müller et al., 2022; Kerbl et al., 2023），但它需要从多个视角拍摄数十到上百张图像，费时、耗费资源，也难以扩展。相比之下，基于单张图像的方法更高效、也更可扩展，只需要一张容易从线上仓库获得的图像，而不必依赖昂贵的扫描设备或多视角采集。

2.1. Single Image Scene Reconstruction

2.1. 单图像场景重建

Scene-level reconstruction from a single image presents challenges due to object diversity, occlusions, and the need to preserve spatial relations. A starting point is monocular depth estimation, where depth is inferred from a single image, typically generating a depth point cloud (Yin et al., 2023; Bhat et al., 2023; Piccinelli et al., 2024; Wang et al., 2024b; Yang et al., 2024b). While it provides valuable information, it struggles with occlusions and hidden portions of the scene. To address this, novel view synthesis methods use representations like radiance fields (Yu et al., 2021; Tian et al., 2023; Yu et al., 2022) and 3D Gaussian (Szymanowicz et al., 2024b; Szymanowicz et al., 2024a), learning occlusion priors from 3D datasets (Geiger et al., 2013; Dai et al., 2017; Chang et al., 2015; Sun et al., 2018). Despite these advances, monocular reconstruction methods still often struggle to provide detailed and precise scene representations.

从单张图像做场景级重建很困难，原因在于物体多样、存在遮挡，还需要保持空间关系。一个起点是单目深度估计：从单张图像推断深度，通常生成深度点云（Yin et al., 2023; Bhat et al., 2023; Piccinelli et al., 2024; Wang et al., 2024b; Yang et al., 2024b）。它能提供有价值的信息，但难以处理遮挡和场景中被挡住的部分。为解决这一点，新视角合成方法使用辐射场（Yu et al., 2021; Tian et al., 2023; Yu et al., 2022）和三维高斯（Szymanowicz et al., 2024b; Szymanowicz et al., 2024a）等表示，从三维数据集中学习遮挡先验（Geiger et al., 2013; Dai et al., 2017; Chang et al., 2015; Sun et al., 2018）。尽管有这些进展，单目重建方法往往仍难以给出细致而精确的场景表示。

Some methods focus on directly regressing geometries along with their semantic labels in the scene (Dahnert et al., 2021; Gkioxari et al., 2022; Chu et al., 2023; Chen et al., 2024a). These approaches typically rely on scene datasets with ground truth object annotations, such as Matterport3D (Fu et al., 2021) and 3DFront (Fu et al., 2021), which are often small in scale and limited to indoor room environments. However, the feed-forward nature of these methods leads to the generation of geometries that often lack sufficient detail and quality.

一些方法专注于直接回归场景中的几何及其语义标签（Dahnert et al., 2021; Gkioxari et al., 2022; Chu et al., 2023; Chen et al., 2024a）。这些方法通常依赖带有物体真值标注的场景数据集，例如 Matterport3D（Fu et al., 2021）和 3DFront（Fu et al., 2021）；这类数据规模往往较小，且限于室内房间环境。由于采用前馈方式，生成的几何常常缺少足够的细节和质量。

To better film real world to digital, other methods turn to retrieval-based approaches (Langer et al., 2022; Gümeli et al., 2022; Kuo et al., 2021; Gao et al., 2024b; Dai et al., 2024), which enhance scene quality by searching for and replacing objects in a scene with similar objects from a pre-existing dataset. These methods incorporate advanced tools such as GPT-4 (Achiam et al., 2023), SAM (Kirillov et al., 2023; Ren et al., 2024), and depth priors to decompose scenes. While these methods improve scene realism by integrating real-world objects, they are constrained by the richness and scope of the datasets they rely on. For scenes outside the dataset’s domain, retrieval-based methods either produce erroneous results or fail to find suitable replacements, significantly degrading the quality of the reconstructed scene.

为了更好地把现实世界搬进数字世界，另一些方法转向基于检索的路线（Langer et al., 2022; Gümeli et al., 2022; Kuo et al., 2021; Gao et al., 2024b; Dai et al., 2024）：通过在已有数据集中搜索相似物体并替换场景中的物体，来提升场景质量。这些方法结合 GPT-4（Achiam et al., 2023）、SAM（Kirillov et al., 2023; Ren et al., 2024）和深度先验等工具来分解场景。虽然借助真实物体能提高场景真实感，它们仍受所依赖数据集的丰富程度和覆盖范围限制。对数据集领域之外的场景，检索方法要么给出错误结果，要么找不到合适替换，从而显著降低重建质量。

Figure 2. Overview of the proposed pipeline. The input RGB image is processed through scene analysis to extract key information, followed by pose-aware generation to create initial 3D models. Physical constraint refinement ensures realistic interactions and spatial relations, yielding a high-quality, mesh-based 3D scene.

图 2. 所提流程概览。输入 RGB 图像先经过场景分析提取关键信息，再通过姿态感知生成得到初始三维模型。物理约束 refinement 保证交互和空间关系真实，最终得到基于网格的高质量三维场景。

2.2. Reconstruction as Generation

2.2. 把重建写成生成

With the continuous advancements in the field, the ability to create high-quality 3D digital assets from various types of open-vocabulary images or text prompts has significantly improved. This advancement has precipitated a paradigm shift where the single-view reconstruction problem evolves into a generative 3D synthesis framework. This paradigm change allows for the generation of 3D assets without being confined to a fixed dataset, enabling more flexible and scalable scene reconstruction.

随着该领域持续进展，从各类开放词汇图像或文本提示创建高质量三维数字资产的能力已显著提高。这一进展带来了范式转变：单视图重建问题演进为生成式三维合成框架。这一转变使三维资产的生成不必再困于固定数据集，从而支持更灵活、可扩展的场景重建。

当前三维资产生成研究有很大一部分关注从二维图像生成模型中蒸馏三维几何（Poole et al., 2022; Tang et al., 2023; Wang et al., 2024a）。更新的工作通过引入多视图图像监督来扩展这一路线（Liu et al., 2023a; Liu et al., 2023c; Liu et al., 2024; Voleti et al., 2025; Long et al., 2024; Wu et al., 2024a），这些模型常常在 Objaverse（Deitke et al., 2023）这类大规模物体数据集的渲染图像上训练，以增强生成时的视图一致性。一些方法根据输入图像直接回归单个物体的形状和外观（Hong et al., 2023; Tang et al., 2025）。这些方法能获得令人满意的视觉效果，却常常无法再现精细的几何细节。为了提高三维几何质量，越来越多工作完全不再使用二维监督，而是直接在三维资产上训练（Deitke et al., 2023; Deitke et al., 2024）。这些方法结合先进处理技术，能生成高质量的物体级几何（Zhang et al., 2023; Zhang et al., 2024a; Xiang et al., 2024）。不过，这类方法关注孤立物体，无法处理场景级挑战，例如建模空间层次、物体间关系以及环境光照。由于建模物体关系、光照和材质的计算与表示复杂度很高，场景生成仍然发展不足。尽管有进展，当前方法仍难以产出可充分实现、可编辑的三维场景。已有范式要么用视频扩散模型（Ho et al., 2022b; Ho et al., 2022a; Blattmann et al., 2023）生成可漫游的二维投影（Bruce et al., 2024; Yu et al., 2024），要么借助扩散先验，通过三维高斯溅射得到体场景近似（Wu et al., 2024b; Gao et al., 2024a; Liang et al., 2024）。这些方法能给出有吸引力的视觉效果，但与传统制作流程不兼容，缺少可编辑网格、UV 映射和可分解的 PBR 材质。

A more feasible paradigm decomposes scenes into modular components—objects, backgrounds, and environmental generating and reassembling them into an editable scene graph for greater flexibility and precision. For example, Gen3DSR (Dogaru et al., 2024) uses DreamGaussian (Tang et al., 2023) for open-vocabulary reconstruction. However, it struggles with occlusions, pose estimation, and editing individual objects, while relying on 2D models leads to poor geometric details and low-fidelity representations. Another recent work, Midi (Huang et al., 2024), learns spatial relations between objects in a scene but requires training on datasets with ground truth 3D meshes and annotations. This reliance on specific datasets limits its scalability and generalization to arbitrary scenes.

更可行的范式是把场景分解成模块化部件——物体、背景和环境——分别生成后再组装成可编辑的场景图，以获得更大灵活性和精度。例如，Gen3DSR（Dogaru et al., 2024）使用 DreamGaussian（Tang et al., 2023）做开放词汇重建。不过，它在遮挡、姿态估计和单独编辑物体上仍有困难，且依赖二维模型会导致几何细节差、表示保真度低。另一项近期工作 Midi（Huang et al., 2024）学习场景中物体之间的空间关系，但需要在带有三维网格真值和标注的数据集上训练。这种对特定数据集的依赖限制了它向任意场景的扩展和泛化。

Our approach shares conceptual foundations with classical analysis-by-synthesis methods (Yuille and Kersten, 2006), as both aim to infer 3D structure by generating explanations for observed imagery. However, while analysis-by-synthesis relies on iterative rendering and pixel-level optimization, our method leverages pre-trained generative models and learned priors to synthesize plausible 3D scenes directly, often bypassing explicit rendering and optimization loops, thereby improving scalability, efficiency, and adaptability to open-world scenarios.

我们的方法与经典的分析-综合方法（Yuille and Kersten, 2006）共享概念基础，两者都通过为观测影像生成解释来推断三维结构。不过，分析-综合依赖迭代渲染和像素级优化，而我们的方法借助预训练生成模型和学到的先验直接合成合理的三维场景，常常绕过显式渲染和优化循环，从而提高可扩展性、效率，以及对开放世界场景的适应性。

Building on this foundation, we present a novel scene reconstruction pipeline that generates each object independently and aligns them into a cohesive scene. Unlike existing methods, our approach preserves accurate geometry, textures, and consistent spatial relations, resulting in more realistic, reliable, and editable reconstructions with improved quality and flexibility.

在此基础上，我们提出一种新的场景重建流程：独立生成每个物体，再把它们对齐成连贯场景。与已有方法不同，我们的方法保留准确的几何、纹理和一致的空间关系，从而得到更真实、可靠、可编辑的重建，质量和灵活性都有提高。

2.3. Physics-Aware 3D Modeling

2.3. 物理感知的三维建模

Generating physically plausible 3D assets is crucial for ensuring realism and functionality in applications such as animation, gaming, and robotics. While recent 3D generative models excel at creating visually realistic objects, they often fall short in achieving physical plausibility. To address this limitation, physics-aware 3D generative models have been developed to integrate physical principles into the generation process. Some methods use soft-body simulation to animate 3D Gaussians (Xie et al., 2024; Zhong et al., 2025), or generate articulated objects with physics-based penalties (Liu et al., 2023b), while others ensure self-supporting structures through rigid-body simulation (Mezghanni et al., 2021; Mezghanni et al., 2022; Chen et al., 2024b) or FEM (Guo et al., 2024; Xu et al., 2024). These methods leverage offline or online physical simulations to check the physical validity of generated shapes and in turn guide generation. However, these approaches are typically confined to individual objects, overlooking the mutual influences between multiple objects within a scene.

生成物理上合理的三维资产，对保证动画、游戏和机器人等应用中的真实感和可用性至关重要。近期三维生成模型擅长创造视觉上真实的物体，但往往达不到物理合理性。为弥补这一不足，物理感知的三维生成模型被用来把物理原则纳入生成过程。一些方法用软体仿真驱动三维高斯动画（Xie et al., 2024; Zhong et al., 2025），或用基于物理的惩罚生成关节物体（Liu et al., 2023b）；另一些则通过刚体仿真（Mezghanni et al., 2021; Mezghanni et al., 2022; Chen et al., 2024b）或有限元方法（Guo et al., 2024; Xu et al., 2024）保证自支撑结构。这些方法借助离线或在线物理仿真检查生成形状的物理有效性，并反过来引导生成。不过，它们通常限于单个物体，忽略了场景中多个物体之间的相互影响。

Incorporating physical constraints into scene synthesis is much more challenging due to the inclusion of more complex relations, e.g., inter-object contact. Yang et al. (2024a) integrates constraints like object collisions, room layout, and object reachability into their scene-level generation pipeline. However, it is limited to indoor scene synthesis and relies on a closed-vocabulary database to perform shape retrieval. Ni et al. (2024) addresses the issue of physical implausibility in multi-view neural reconstruction. It leverages both differentiable rendering and physical simulation to learn implicit representations. However, it requires multi-view images as input, focuses on individual objects, and primarily addresses only stability (simulating the dropping of objects). In contrast, our method operates in an open-vocabulary setting and requires only a single input image. Furthermore, it accounts for more complex inter-object relations, particularly support and contact, making it more versatile and applicable across diverse scenarios.

把物理约束纳入场景合成要困难得多，因为其中包含更复杂的关系，例如物体之间的接触。Yang et al.（2024a）把物体碰撞、房间布局和物体可达性等约束融入场景级生成流程。不过，它限于室内场景合成，并依赖封闭词汇数据库做形状检索。Ni et al.（2024）处理多视图神经重建中的物理不合理问题，同时利用可微渲染和物理仿真来学习隐式表示。但它需要多视图图像作为输入，关注单个物体，并且主要只处理稳定性（模拟物体下落）。相比之下，我们的方法在开放词汇设定下工作，并且只需要单张输入图像。它还考虑更复杂的物体间关系，尤其是支撑和接触，因而在多样场景中更通用。

3. Overview

3. 概述

Scene-level reconstruction from a single image is a fundamental challenge in computer graphics, with broad applications in animation, virtual reality, and interactive gaming. Unlike object-level reconstruction, which focuses on isolated objects, scene-level reconstruction emphasizes the arrangement and relations of multiple entities under realistic (or stylized) physics. By capturing per-object structures, spatial relations, and contextual cues, this holistic approach enables more immersive experiences, compelling narratives, and efficient workflows—benefits that surpass those of single-object reconstructions. Although previous methodologies have explored feed-forward pipelines or retrieval-based approaches using fixed 3D templates (Liu et al., 2022; Dai et al., 2024), these methods often struggle to capture nuanced scene semantics and complex object relations. To address these limitations, we propose a generation-driven scene reconstruction approach with emphasized object relations to construct high-fidelity, contextually consistent 3D environments from a single, unannotated RGB image whether sourced from real-world photography or synthetic data (see Fig. 2).

从单张图像做场景级重建是计算机图形学中的基本挑战，在动画、虚拟现实和交互游戏中有广泛应用。与关注孤立物体的物体级重建不同，场景级重建强调多个实体在真实（或风格化）物理下的排布和关系。通过捕捉每个物体的结构、空间关系和上下文线索，这种整体方法能够带来更沉浸的体验、更有说服力的叙事，以及更高效的工作流——这些好处超出了单物体重建。先前方法探索了前馈流程，或使用固定三维模板的检索方法（Liu et al., 2022; Dai et al., 2024），但它们往往难以捕捉细腻的场景语义和复杂的物体关系。为解决这些限制，我们提出一种以生成为驱动、强调物体关系的场景重建方法，从单张未标注的 RGB 图像构建高保真、上下文一致的三维环境；图像既可以来自真实摄影，也可以来自合成数据（见图 2）。

A key insight of our method is the thorough object relation analysis of scene contextual information. First, we perform object segmentation to identify and localize constituent objects within the image. We then obtain preliminary geometric information, i.e., point clouds, and explore semantic and spatial relations among objects. This contextual backbone informs our subsequent object-wise generation pipeline, ensuring that each reconstructed object retains not only its geometric fidelity but also its correct placement in the broader scene. Finally, we synthesize a coherent 3D environment that respects physical plausibility—achieving structurally sound layouts and realistic interactions among scene elements.

我们方法的一个关键认识，是对场景上下文信息做充分的物体关系分析。首先，我们进行物体分割，识别并定位图像中的组成物体。随后获得初步几何信息，也就是点云，并探索物体之间的语义和空间关系。这一上下文骨架会指导后续按物体生成的流程，确保每个重建物体不仅保持几何保真，也能在更大场景中被正确放置。最后，我们合成一个尊重物理合理性的连贯三维环境，得到结构可靠的布局和场景元素之间真实的交互。

Our research focuses on two primary objectives: to explore how generative models can effectively capture complex inter-object relations in order to produce realistic, scene-level reconstructions from a single image; and to identify strategies for integrating geometric cues and contextual information that maximize accuracy and plausibility in 3D reconstructions. Through this investigation, we demonstrate that generative methods provide a more flexible and robust alternative to traditional feed-forward and retrieval-based techniques. These methods allow for fine-grained control over both object-level details and global scene composition, thus streamlining content creation pipelines for animation, game development, and other fields requiring accurate, visually compelling 3D models. This work highlights the advantages of a generation-centric framework and lays the groundwork for future advancements in scene-level 3D reconstruction. It also underscores the growing importance of context-driven approaches in bridging the gap between 2D imagery and immersive, interactive virtual environments.

我们的研究关注两个主要目标：探索生成模型如何有效捕捉复杂的物体间关系，从而从单张图像产生真实的场景级重建；以及找出整合几何线索和上下文信息的策略，以最大化三维重建的准确性和合理性。通过这一研究，我们表明生成方法为传统前馈和检索技术提供了更灵活、更稳健的替代。这些方法能够对物体级细节和全局场景构图进行细粒度控制，从而简化动画、游戏开发以及其他需要准确、视觉上有吸引力的三维模型的内容创作流程。这项工作突出了以生成为中心的框架的优势，并为场景级三维重建的后续进展奠定基础。它也强调了上下文驱动方法在弥合二维影像与沉浸、交互的虚拟环境之间差距时日益重要。

\{\bm{M}_{i}\}

预处理 为了从单张图像完成全面的场景重建，我们首先进行广泛的语义提取，为后续处理提供坚实基础。具体而言，我们用 Florence-2（Xiao et al., 2024）识别物体、生成描述，并用边界框定位每个物体。随后借助 GPT-4v（Achiam et al., 2023）过滤虚假检测，分离出有意义的组成物体，从而实现不受预定义类别限制的开放词汇物体识别。接下来，我们使用 GroundedSAM-v2（Ren et al., 2024）为每个标注物体 \{m{o}_{i}\} 生成精细分割掩码 \{m{M}_{i}\}，从而同时获得精确的物体边界和相应的遮挡掩码；这些掩码在物体生成阶段起着关键的辅助作用。除语义线索外，我们还通过提取场景级点云来融入几何信息。我们使用 MoGe（Wang et al., 2024b），为每个物体 \{m{o}_{i}\},i\in\{1,\dots,N\} 生成与像素对齐的点云 \{m{q}_{i}\}，以及场景坐标系中的全局相机参数。这些额外的几何数据随后与每个物体的分割掩码匹配，为最终的三维场景重建提供可靠的结构参考。

Figure 3. Network design of our alignment generation model (Sec. 4.2), occlusion-aware object generation model (Sec. 4.1), and an illustrative figure of the texture generation model.

图 3. 我们对齐生成模型（第 4.2 节）、遮挡感知物体生成模型（第 4.1 节）的网络设计，以及纹理生成模型的示意。

4. Perceptive 3D Instance Generation

4. 感知式三维实例生成

In the endeavor to reconstruct high-fidelity 3D scenes from single RGB images, a brute-force approach involves generating the entire scene mesh directly using techniques such as single-image depth estimation or diffusion priors. However, this method inherently struggles to manage occlusions, render invisible components, and accurately represent object relations due to the complex and intertwined nature of real-world scenes. Instead of generating an entire scene mesh directly, our approach focuses on individual object generation and then arranges the objects via precise relational alignment, as illustrated in 3. This strategy offers several advantages: 1. focusing on individual objects ensures higher geometric fidelity and allows for detailed modeling, resulting in more accurate and visually appealing scene components. 2. operating within a canonical space ensures that generated assets adhere to standardized orientations and scales, seamlessly integrating with artist-defined coordinate systems and promoting consistency across digital content creation tools. 3. the modular approach supports various applications such as editing, rendering, and simulation, enabling independent manipulation of objects for greater flexibility and efficiency. By decomposing scene reconstruction into object-wise generation and alignment, our method improves asset quality and manageability while enhancing the overall coherence and functionality of the 3D environment. This approach addresses challenges like geometric precision and efficient post-processing, advancing single-image 3D scene generation.

要从单张 RGB 图像重建高保真三维场景，一种蛮力做法是直接用单图像深度估计或扩散先验生成整幅场景网格。不过，由于真实场景复杂且相互缠绕，这种方法天生难以处理遮挡、画出不可见部件，并准确表示物体关系。我们不直接生成整幅场景网格，而是先逐个生成物体，再通过精确的关系对齐来排布它们，如图 3 所示。这一策略有几项好处：1. 聚焦单个物体能保证更高的几何保真，并允许细致建模，从而得到更准确、视觉上更有吸引力的场景部件。2. 在规范空间中操作，使生成资产遵守标准化的朝向和尺度，能够无缝接入艺术家定义的坐标系，并促进数字内容创作工具之间的一致性。3. 模块化方法支持编辑、渲染和仿真等多种应用，可以独立操作物体，灵活性和效率更高。把场景重建分解为按物体生成与对齐，既能提高资产质量与可管理性，也能增强三维环境整体的连贯性和可用性。这一做法回应了几何精度和高效后处理等挑战，推进了单图像三维场景生成。

Object-wise generation presents significant challenges, primarily due to partial observations of objects within a scene caused by occlusions and limited sensor coverage. Additionally, existing generation methods often fail to coordinate multiple objects cohesively, resulting in inconsistent and unrealistic scenes. To overcome these limitations, we propose an Occlusion-Aware 3D Object Generation framework that integrates partial observations with comprehensive scene understanding. Specifically, given an image and its point cloud, our framework generates a high-quality 3D asset that not only resembles the input image but also aligns accurately with the partial point cloud represented in its corresponding canonical space. Furthermore, we compute a transformation matrix that maps the generated object from its canonical space back to the original scene space, ensuring spatial consistency within the scene.

按物体生成面临显著挑战，主要是因为遮挡和有限的传感器覆盖，使场景中的物体往往只有部分观测。此外，已有生成方法常常无法让多个物体协调一致，导致场景不一致、不真实。为克服这些限制，我们提出遮挡感知的三维物体生成框架，把局部观测与完整的场景理解结合起来。具体而言，给定图像及其点云，我们的框架生成高质量三维资产：它不仅与输入图像相似，还能准确对齐到相应规范空间中表示的残缺点云。此外，我们计算一个变换矩阵，把生成物体从规范空间映射回原始场景空间，以保证场景内的空间一致性。

A critical aspect of our object generation process is the utilization of a large generative model to generate holistic and high-fidelity object meshes from partial image and point cloud observations. To do so, we first follow state-of-art native 3D generative models (Xiang et al., 2024; Zhang et al., 2023; Zhang et al., 2024a) to pre-train a large-scale 3D generative model conditioning on textual and image inputs.

物体生成过程的一个关键方面，是利用大规模生成模型，从残缺的图像和点云观测中生成整体、高保真的物体网格。为此，我们首先沿循先进的原生三维生成模型（Xiang et al., 2024; Zhang et al., 2023; Zhang et al., 2024a），预训练一个以文本和图像为条件的大规模三维生成模型。

\mathcal{E}

\mathcal{E}

\bm{Z}=\mathcal{E}(\bm{X}),\ \mathcal{D}(\bm{Z},\bm{p})=\text{SDF}(\bm{p}), \tag{1}

\bm{X}

\bm{X}

\epsilon_{\text{obj}}(\bm{Z}_{t};t,\bm{c})\rightarrow\bm{Z}, \tag{2}

\epsilon

\epsilon

4.1. Occlusion-aware 3D Object Generation

4.1. 遮挡感知的三维物体生成

\bm{M}

\bm{I}

\bm{c}_{m}=\mathcal{E}_{\text{DINOv2}}(\bm{I}\odot\bm{M}), \tag{3}

\bm{M}

\bm{M}

\bm{c}

\bm{c}

\bm{p}_{\text{gt}}

\bm{p}_{\text{gt}}

\epsilon(\bm{Z}_{t};t,\bm{c},\bm{p}_{\text{disturb}})\rightarrow\bm{Z}, \tag{4}

where the conditioning adaptation scheme is based on attention mechanism similar to Zhang et al. (2023); Zhang et al. (2024a). Additionally, to mimic real-world occlusions and missing data, we randomly mask sets of basic primitives—such as circles and rectangles—in the depth maps from various camera views. This results in partial point clouds with occluded and incomplete regions, further enhancing the model’s ability to handle imperfect inputs. A critical design choice in our approach is to maintain the alignment of partial point clouds with the geometry in our training data set. Unlike methods that apply random scaling, translation, or rotation to augmented point clouds, our aligned partial point clouds ensure that the generative model can more effectively conform to the input point clouds’s inherent structure. This alignment restricts the model to adhere closely to the actual shapes and scales of objects, thereby facilitating more precise and coherent 3D reconstructions. By conditioning on these well-aligned partial point clouds, our model achieves superior alignment both in overall size and local geometric details, resulting in high-quality and reliable 3D geometry generation.

其中条件适配方案基于与 Zhang et al.（2023）；Zhang et al.（2024a）类似的注意力机制。此外，为了模拟真实世界中的遮挡和缺失数据，我们从不同相机视角在深度图中随机掩蔽圆形和矩形等基本图元。这会得到带有遮挡和残缺区域的局部点云，进一步增强模型处理不完美输入的能力。我们方法中的一个关键设计选择，是保持残缺点云与训练数据集中几何的对齐。与对增强点云施加随机缩放、平移或旋转的方法不同，对齐后的残缺点云使生成模型能更有效地符合输入点云固有的结构。这种对齐限制模型紧密遵循物体的实际形状和尺度，从而促成更精确、更连贯的三维重建。通过对这些对齐良好的残缺点云施加条件，我们的模型在整体尺寸和局部几何细节上都能取得更好的对齐，从而生成高质量、可靠的三维几何。

4.2. Generative Alignment

4.2. 生成式对齐

\bm{q}\in\mathbb{R}^{N\times 3}

\bm{q}\in\mathbb{R}^{N\times 3}

\epsilon_{\text{align}}(\bm{p}_{t};t,\bm{q},\bm{Z})\rightarrow\bm{p}, \tag{5}

\epsilon_{\text{align}}

\epsilon_{\text{align}}

\bm{q}

\bm{q}

4.3. Iterative Generation Procedure

4.3. 迭代生成流程

k

k

\bm{z}^{(k)}

\bm{c}

\bm{z}^{(k)}=\text{ObjectGen}(\bm{c},\bm{p}^{(k)}\otimes\beta^{(k)}). \tag{6}

\bm{z}^{(k)}

\bm{z}^{(k)}

\bm{z}^{(k)}

\bm{z}^{(k)}

\bm{p}^{(k+1)}=\text{AlignGen}(\bm{q},\bm{z}^{(k)}). \tag{7}

\bm{p}^{(k+1)}

\bm{p}^{(k+1)}

\bm{p}^{(k+1)}

\bm{p}^{(k+1)}

This iterative loop—alternating between geometry generation and transformation estimation—continues until convergence criteria are met. Convergence is achieved when the changes in transformation parameters fall below a predefined threshold or when a maximum number of iterations is reached. The result is a high-fidelity 3D object that is both visually accurate and geometrically aligned with the input data. By tightly integrating the Object Generation and Alignment Generation modules within an iterative framework, our approach effectively balances aesthetic fidelity with geometric precision. This joint generation process leverages both visual and depth information, ensuring that each 3D asset is of high quality and accurately positioned. Consequently, the pipeline lays a robust foundation for constructing physically correct and visually coherent 3D scenes, facilitating a wide range of downstream applications such as editing, rendering, and animation.

这一在几何生成和变换估计之间交替的迭代循环会持续到满足收敛准则。当变换参数的变化低于预设阈值，或达到最大迭代次数时，即视为收敛。结果是既在视觉上准确、又在几何上与输入数据对齐的高保真三维物体。通过在迭代框架内紧密整合 物体生成 和 对齐生成 模块，我们的方法能够在外观保真与几何精度之间取得平衡。这一联合生成过程同时利用视觉和深度信息，确保每个三维资产质量高、位置准。因此，该流程为构建物理正确、视觉连贯的三维场景打下坚实基础，便于支持编辑、渲染和动画等广泛的下游应用。

Once the object geometry is determined, we apply a state-of-the-art texture generation module to create photo-realistic surface details. Following established texture synthesis pipelines (Zhang et al., 2023; Zhang et al., 2024a), we assign UV mappings and train a generative network to paint detailed textures onto the 3D meshes. This module is designed to robustly handle images under various augmentation, ensuring that the final textures match the input appearance even under occlusion or limited visibility.

一旦确定物体几何，我们使用先进的纹理生成模块来创建照片级的表面细节。沿循已有的纹理合成流程（Zhang et al., 2023; Zhang et al., 2024a），我们指定 UV 映射，并训练生成网络把细致纹理绘制到三维网格上。该模块被设计成能够稳健处理各种增强下的图像，确保最终纹理即使在遮挡或可见范围有限时，也与输入外观匹配。

Figure 4. Physics-aware correction via constraint graph mapped from fine-grained relation graph. Top: Floating surfboard grounded on the van. Bottom: Penetrating guitar and cooler separated.

图 4. 由细粒度关系图映射得到的约束图所引导的物理感知校正。上：悬浮的冲浪板被落到面包车上。下：互相穿透的吉他和保温箱被分开。

Figure 5. Bringing the vibrant diversity of the real world into the virtual realm, this collection reimagines open-vocabulary scenes as immersive digital environments, capturing the richness and depth of each unique setting. For each scene, the images display as follows: the top-left shows the input image, the top-center displays the rendered geometry, and the right presents the rendered image with realistic textures.

图 5. 把现实世界的丰富多样性带进虚拟领域：这一组结果把开放词汇场景重想象为可沉浸的数字环境，捕捉每个独特场景的丰富性和深度。对每个场景，图像展示如下：左上为输入图像，上中为渲染几何，右侧为带真实纹理的渲染图像。

5. Physics-Aware Correction

5. 物理感知校正

The pipeline detailed in Sec. 4 individually generates each 3D object instance and estimates its similarity transformation (scaling, rotation, and translation) based on a single input image. While our proposed modules achieve high accuracy, the resulting scenes are sometimes not physically plausible. For instance, as illustrated in Fig. 4, one object (e.g., a guitar) may intersect with another (e.g., a cooler), or an object (e.g., a surfboard) may appear to float unnaturally without any support (e.g., from a van).

第 4 节详述的流程会单独生成每个三维物体实例，并根据单张输入图像估计其相似变换（缩放、旋转和平移）。尽管所提模块能达到较高精度，得到的场景有时在物理上并不合理。例如，如图 4 所示，一个物体（如吉他）可能与另一个物体（如保温箱）相交，或者一个物体（如冲浪板）在没有任何支撑（如来自面包车）的情况下不自然地悬浮。

To address these issues, we introduce a physics-aware correction process that optimizes the rotation and translation of objects, ensuring the scene adheres to physical constraints consistent with common sense. The correction process is motivated by physical simulation (Sec. 5.1) and formulated as an optimization problem (Sec. 5.2) based on inter-object relations represented by a scene graph (Sec. 5.3) extracted from the image.

为解决这些问题，我们引入物理感知校正过程，优化物体的旋转和平移，使场景遵守与常识一致的物理约束。校正过程受到物理仿真的启发（第 5.1 节），并形式化为一个优化问题（第 5.2 节），其依据是从图像中提取、由场景图表示的物体间关系（第 5.3 节）。

5.1. A Quick Primer to Rigid-Body Simulation

5.1. 刚体仿真速览

We introduce the fundamental principles of physical (rigid-body) simulation, which inspire our problem formulation and make our framework more accessible to downstream applications such as gaming and robotics. For a thorough survey, we refer the readers to Bender et al. (2012).

我们介绍物理（刚体）仿真的基本原理，它们启发了我们的问题形式化，也使我们的框架更易于接入游戏和机器人等下游应用。更全面的综述见 Bender et al.（2012）。

In rigid-body simulations, the world is modeled as an ordinary differential equation (ODE) process. In each simulation step, it begins with the Newton-Euler (differential) equations, which describe the dynamic motion of rigid bodies in the absence of contact. Collision detection is conducted to find the contact points between rigid bodies, which are needed to determine contact forces. For contact handling and collision resolution, there are usually several conditions: non-penetration constraints to prevent bodies from overlapping, a friction model ensuring contact forces remain within their friction cones, and complementarity constraints that enforce specific disjunctive relations among variables. Solvers are used to resolve the system comprising equations and inequalities, subsequently updating the velocity and position of each rigid body.

在刚体仿真中，世界被建模为常微分方程（ODE）过程。在每个仿真步中，它从牛顿-欧拉（微分）方程开始，这些方程描述没有接触时刚体的动态运动。随后进行 碰撞检测 ，找出刚体之间的接触点，这些点被用来确定接触力。对于接触处理和碰撞化解，通常有若干条件：防止物体重叠的非穿透约束，保证接触力留在摩擦锥内的摩擦模型，以及在变量之间施加特定析取关系的互补约束。求解器用来化解由方程和不等式组成的系统，随后更新每个刚体的速度和位置。

A straightforward approach to enhance physical plausibility is to utilize an off-the-shelf rigid-body simulator to process the scene, starting from the initial state estimated by the pipeline previously described and obtaining the rest state after simulation. However, this method presents several challenges. 1) Partial Scene : Some objects may be missing due to the limitations of 2D foundation models, and thus not reconstructed. Simulating a partial scene under full physical rules can lead to suboptimal results (see Fig. 10). 2) Imperfect Geometries : While our 3D generative model produces high-quality geometries, minor imperfections may still occur. Rigid-body simulators typically require convex decomposition (Mamou and Ghorbel, 2009; Mamou et al., 2016; Wei et al., 2022) of objects, which introduces additional complexity and hyperparameters. Overly fine-grained decomposition can result in non-flat, complex surfaces, causing objects to fall or move unexpectedly during simulation. Conversely, coarse decomposition may lead to visually floating objects due to discrepancies between the visual and collision geometries. 3) Initial Penetrations : Despite the high accuracy of pose estimation, significant inter-object penetrations may exist in the initial state. These penetrations create instability for standard rigid-body solvers and, in some cases, lead to unsolvable scenarios if the solver is not customized for those cases.

增强物理合理性的一种直接做法，是用现成的刚体仿真器处理场景：从前面流程估计的初始状态出发，得到仿真后的静止状态。不过，这种方法有几项挑战。1） 残缺场景 ：由于二维基础模型的限制，一些物体可能缺失，因而没有被重建。在完整物理规则下仿真残缺场景，可能导致不佳结果（见图 10）。2） 不完美的几何 ：虽然我们的三维生成模型能产生高质量几何，仍可能出现轻微缺陷。刚体仿真器通常需要对物体做凸分解（Mamou and Ghorbel, 2009; Mamou et al., 2016; Wei et al., 2022），这会引入额外的复杂度和超参数。分解过细会导致非平坦、复杂的表面，使物体在仿真中意外下落或移动。分解过粗则可能因视觉几何与碰撞几何不一致，而看起来物体在悬浮。3） 初始穿透 ：尽管姿态估计精度较高，初始状态中仍可能存在明显的物体间穿透。这些穿透会使标准刚体求解器不稳定；若求解器没有针对这些情形定制，有时还会导致无法求解。

Thus, we propose a customized and simplified “physical simulation” to optimize the object poses, ensuring that the scene adheres to common-sense physical principles derived from the single image. Note that our approach does not model full dynamics. For example, an object may not remain stable in its current pose over time. However, it should be physically plausible at the current time step. We argue that our optimized results can serve as a reliable initialization for subsequent physical simulations.

因此，我们提出一种定制且简化的“物理仿真”来优化物体位姿，使场景遵守从单张图像得出的常识物理原则。 需要注意的是，我们的方法并不建模完整动力学。例如，一个物体在当前姿态下未必能随时间保持稳定。不过，它在当前时刻应当是物理上合理的。我们认为，优化后的结果可以作为后续物理仿真的可靠初值。

5.2. Problem Formulation and Physical Constraints

5.2. 问题形式化与物理约束

We formulate the physics-aware correction process as an optimization problem, aiming to minimize the total cost that represents pairwise constraints on objects.

我们把物理感知校正过程形式化为一个优化问题，目标是最小化表示物体成对约束的总代价。

\min_{\mathcal{T}=\{T_{1},T_{2},\dots,T_{N}\}}\sum_{i,j}C(T_{i},T_{j};\bm{o}_{i},\bm{o}_{j}) \tag{8}

N

N

Motivated by physical simulation, we categorize the relations into two types: contact and support . The relations are identified with the assistance of a VLM, as detailed in Sec. 5.3.

受物理仿真启发，我们把关系分成两类： 接触 和 支撑 。这些关系借助视觉语言模型识别，详见第 5.3 节。

\bm{o}_{i}

\bm{o}_{i}

\begin{split}C(T_{i},T_{j};\bm{o}{i}\to\bm{o}{j})=-\frac{\sum_{p\in\partial\bm{o}{j}}D{i}(p(T_{j}))\mathbb{I}(D_{i}(p(T_{j}))<0)}{\sum_{p\in\partial\bm{o}{j}}\mathbb{I}(D{i}(p(T_{j}))<0)}\+\max(\min_{p\in\partial\bm{o}{j}}D{i}(p(T_{j})),0)\C(T_{i},T_{j};\bm{o}{j}\to\bm{o}{i})=-\frac{\sum_{p\in\partial\bm{o}{i}}D{j}(p(T_{i}))\mathbb{I}(D_{j}(p(T_{i}))<0)}{\sum_{p\in\partial\bm{o}{i}}\mathbb{I}(D{j}(p(T_{i}))<0)}\+\max(\min_{p\in\partial\bm{o}{i}}D{j}(p(T_{i})),0)\C(T_{i},T_{j})=C(T_{i},T_{j};\bm{o}{i}\to\bm{o}{j})+C(T_{i},T_{j};\bm{o}{j}\to\bm{o}{i})\\text{if }\bm{o}{i}\text{ and }\bm{o}{j}\text{ are in contact}\end{split} \tag{9}

\partial\bm{o}_{i}

\partial\bm{o}_{i}

\bm{o}_{i}

\bm{o}_{i}

C(T_{i},T_{j})=|\min_{p\in\partial\bm{o}_{j}}D_{i}(p(T_{j}))|,\ \text{if }\bm{o}_{i}\text{ supports }\bm{o}_{j} \tag{10}

Furthermore, for flat supporting surfaces like the ground or walls, we regularize the SDF values near the contact region, to ensure that objects make close contact with these surfaces. This regularization handles scenarios where objects are partially reconstructed, such as a van with only two wheels, as illustrated in Fig. 4.

此外，对于地面或墙壁这类平坦支撑面，我们正则化接触区域附近的 SDF 值，以确保物体与这些表面紧密接触。这一正则化用来处理物体只被部分重建的情形，例如图 4 中只重建出两个轮子的面包车。

C(T_{i},T_{j})=\frac{\sum_{p\in\partial\bm{o}_{j}}D_{i}(p(T_{j})\mathbb{I}(0<D_{i}(p)<\sigma)}{\sum_{p\in\partial\bm{o}_{j}}\mathbb{I}(0<D_{i}(p)<\sigma)} \tag{11}

\mathbb{I}

\mathbb{I}

5.3. Scene Relation Graph

5.3. 场景关系图

Physical cues, particularly inter-object relations, are visually present in the image. We leverage the strong common-sense reasoning capabilities (Rana et al., 2023; Li et al., 2024b; Cheng et al., 2024) of visual-language models, specifically GPT-4v (Achiam et al., 2023), to identify pairwise physical constraints as defined in Sec. 5.2. Given an image, we employ the Set of Mark (Yang et al., 2023) (SoM) technique to visually prompt GPT-4v to describe the inter-object relations, and subsequently extract a scene relation graph from the answers. To address the sampling uncertainty inherent in VLMs, we adopt an ensemble strategy, combining results from multiple trials. We define relations as correct if they appear in more than half of the samples to produce a robust inferred graph. To be more specific, we apply the Set-of-Mark method with random colorization and numerical ordering multiple times,enabling more reliable and consistent outputs for further GPT-based question-answering tasks.

物理线索，尤其是物体之间的关系，在图像中是可见的。我们利用视觉语言模型强大的常识推理能力（Rana et al., 2023; Li et al., 2024b; Cheng et al., 2024），具体是 GPT-4v（Achiam et al., 2023），来识别第 5.2 节定义的成对物理约束。给定图像，我们采用 Set of Mark（Yang et al., 2023）（SoM）技术，以视觉方式提示 GPT-4v 描述物体间关系，随后从回答中提取 场景关系图 。为了应对视觉语言模型固有的采样不确定性，我们采用集成策略，合并多次试验的结果。我们把出现在超过一半样本中的关系定义为正确，以得到稳健的推断图。更具体地说，我们多次应用带随机着色和数字编号的 Set-of-Mark 方法，使后续基于 GPT 的问答输出更可靠、更一致。

Instead of directly asking GPT-4v to identify Support and Contact relations, we first provide it with more fine-grained physical relations, such as Stack (Object 2 supports Object 1), Lean (Object 1 leans against Object 2), and Hang (Object 2 supports Object 1 from above). We instruct GPT-4v to analyze numbered objects from the Set-of-Mark method and output all contact-based relations, covering six types: Stack, Lean, Hang, Clamped, Contained, and Edge/Point. The prompt specifies that only contacting objects have relations and defaults to Stack for ambiguous cases.

我们并不直接让 GPT-4v 识别 支撑 和 接触 关系，而是先向它提供更细粒度的物理关系，例如 堆叠 （物体 2 从下方支撑物体 1）、 倚靠 （物体 1 靠在物体 2 上）和 悬挂 （物体 2 从上方支撑物体 1）。我们指示 GPT-4v 分析 Set-of-Mark 方法编号后的物体，并输出所有基于接触的关系，覆盖六种类型：堆叠、倚靠、悬挂、夹持、包含，以及边或点接触。提示词规定只有相互接触的物体才有关系，对难以判断的情形默认使用堆叠。

We then map these detailed relations to the predefined categories of Support and Contact for further optimization. Specifically, if there are edges pointing toward each other between two nodes, the edge is categorized as Contact; otherwise, it is categorized as Support . Prompting GPT-4v with these nuanced relations helps eliminate potential ambiguity in binary relation classification and facilitates more accurate reasoning by GPT-4v. An example of the resulting graph is illustrated in Fig. 4.

随后，我们把这些细致关系映射到预定义的 支撑 和 接触 类别，供进一步优化。具体而言，如果两个节点之间存在彼此指向的边，该边被归为 接触 ；否则归为 支撑 。用这些细腻关系提示 GPT-4v，有助于消除二元关系分类中的潜在歧义，并便于 GPT-4v 做更准确的推理。图 4 展示了得到的图的一个例子。

The mapped scene constraint graph is a directed graph where nodes represent object instances and edges denote physical relations between objects. A Contact relation is represented by a bidirectional edge, while a Support relation is depicted as a directed edge. This graph serves as the foundation for defining the cost functions used in Eq. 8.

映射得到的场景约束图是有向图，节点表示物体实例，边表示物体之间的物理关系。 接触 关系用双向边表示， 支撑 关系用有向边表示。该图是定义公式 8 中代价函数的基础。

5.4. Optimization with Physics-Aware Relation Graph

5.4. 基于物理感知关系图的优化

Given the physical constraints defined by the inferred relation graph, we can instantiate our cost functions as described in Eq. 8. The graph allows us to reduce the number of pairwise constraints that need to be optimized, in contrast to a full physical simulation.

给定由推断关系图定义的物理约束，我们可以按公式 8 实例化代价函数。与完整物理仿真相比，该图使我们能够减少需要优化的成对约束数量。

For the implementation, we uniformly sample a fixed number of points from the surface of each object at its rest pose. These points are then transformed according to the current object’s pose parameters and used to query the SDF values with respect to another object (and its pose). SDF computation is handled by Open3D, and Pytorch is used to auto-differentiate the loss function.

实现上，我们在每个物体的静止姿态下，从其表面均匀采样固定数量的点。这些点随后按当前物体的位姿参数变换，并用来查询相对于另一个物体（及其位姿）的 SDF 值。SDF 计算由 Open3D 处理，PyTorch 用来对损失函数自动微分。

Figure 6. Qualitative comparisons of CAST with state-of-the-art single-image scene reconstruction methods. From left to right: Input image, CAST, ACDC, and Gen3DSR. Top to bottom: random open vocabulary dataset (rows 1-3), Gen3DSR input (rows 4-5), ACDC input (rows 6-7).

图 6. CAST 与先进单图像场景重建方法的定性比较。从左到右：输入图像、CAST、ACDC 和 Gen3DSR。从上到下：随机开放词汇数据集（第 1-3 行）、Gen3DSR 输入（第 4-5 行）、ACDC 输入（第 6-7 行）。

6. Result

6. 结果

Fig. 5 showcases a range of 3D scenes generated by our method from single-view inputs across a diverse range of open-vocabulary scenarios, featuring detailed indoor environments, close-up captures of objects, and AI-generated imagery. These examples highlight the versatility and robustness of our approach, exhibiting high-fidelity geometry, realistic textures, and convincing scene compositions.

图 5 展示了我们的方法从单视图输入生成的一系列三维场景，覆盖多样的开放词汇情形，包括细致的室内环境、物体特写，以及人工智能生成的图像。这些例子突出了方法的通用性和稳健性，展现出高保真几何、真实纹理和有说服力的场景构图。

6.1. Implementation Details

6.1. 实现细节

ObjectGen The ObjectGen (Sec. 4.1) model’s pretraining follows the methodology outlined in 3DShape2VecSet (Zhang et al., 2023) and CLAY (Zhang et al., 2024a), where we leverage both a Variational Autoencoder (VAE) and a Latent Diffusion Model (LDM) to generate 3D object geometries. Both the VAE and LDM modules are implemented using a 24-layer transformer, comprising a total of 1.5 billion parameters. The model is trained on the Objaverse (Deitke et al., 2023) dataset, which consists of approximately 500,000 3D assets after filtering. The partial point cloud conditioning follows a similar approach to CLAY’s adaptation framework. We encode the canonical-space partial point cloud as positional embeddings with a feature dimension of 512, which are injected into the main LDM transformer using cross-attention mechanisms. For each 3D asset, we render 32 views and precompute depth maps using MoGe (Wang et al., 2024b) and Metric3D (Yin et al., 2023). These depth maps are then lifted into point clouds during training, with random masks applied to simulate occlusions. We sample 2048 points from the unprojected point cloud using Farthest Point Sampling (FPS), which serves as conditioning input for the LDM. To enhance the model’s robustness, we randomly interpolate between the ground truth and predicted partial point clouds, allowing the system to handle data of varying quality. The conditioning module is trained on 200K curated data from Objaverse over 3000 epochs with 64 Nvidia A800 GPUs required approximately one week. The AdamW optimizer is used with a learning rate of 1e-5. For inferencing a single object, object generation takes approximately 7 seconds, and texture generation takes approximately 10 seconds per object on an NVIDIA A6000 GPU.

ObjectGen ObjectGen（第 4.1 节）模型的预训练沿循 3DShape2VecSet（Zhang et al., 2023）和 CLAY（Zhang et al., 2024a）中概述的方法：我们同时利用变分自编码器（VAE）和潜扩散模型（LDM）生成三维物体几何。VAE 和 LDM 模块都用 24 层 Transformer 实现，总计 15 亿参数。模型在 Objaverse（Deitke et al., 2023）数据集上训练；过滤后约有 50 万个三维资产。残缺点云条件采用与 CLAY 适配框架类似的做法。我们把规范空间残缺点云编码为特征维 512 的位置嵌入，再用交叉注意力注入主 LDM Transformer。对每个三维资产，我们渲染 32 个视图，并用 MoGe（Wang et al., 2024b）和 Metric3D（Yin et al., 2023）预计算深度图。这些深度图在训练时被提升为点云，并施加随机掩码以模拟遮挡。我们用最远点采样（FPS）从未投影点云中采样 2048 个点，作为 LDM 的条件输入。为增强模型鲁棒性，我们在真值和预测的残缺点云之间随机插值，使系统能够处理不同质量的数据。条件模块在从 Objaverse 精选的 20 万条数据上训练 3000 个 epoch，使用 64 张 Nvidia A800 GPU，大约需要一周。优化器为 AdamW，学习率为 1e-5。推理单个物体时，在 NVIDIA A6000 GPU 上，物体生成大约需要 7 秒，纹理生成大约需要 10 秒。

\bm{z}

\bm{z}

Method CLIP↑ GPT-4↓ VQ↑ PP↑ ACDC 69.77 2.7 5.58% 22.86% Gen3DSR 79.84 2.175 6.35% 5.72% ours 85.77 1.125 88.07% 71.42% Table 1. Quantitative comparison of scene reconstruction methods across four metrics with CLIP score, GPT-4 ranking, user study of visual quality (VQ), and physical plausibility (PP).

Method	CLIP↑	GPT-4↓	VQ↑	PP↑
ACDC	69.77	2.7	5.58%	22.86%
Gen3DSR	79.84	2.175	6.35%	5.72%
ours	85.77	1.125	88.07%	71.42%

方法 CLIP↑ GPT-4↓ VQ↑ PP↑ ACDC 69.77 2.7 5.58% 22.86% Gen3DSR 79.84 2.175 6.35% 5.72% 本文 85.77 1.125 88.07% 71.42% 表 1. 场景重建方法在四项指标上的定量比较：CLIP 分数、GPT-4 排序、视觉质量用户研究（VQ）以及物理合理性（PP）。

方法	CLIP↑	GPT-4↓	VQ↑	PP↑
ACDC	69.77	2.7	5.58%	22.86%
Gen3DSR	79.84	2.175	6.35%	5.72%
本文	85.77	1.125	88.07%	71.42%

6.2. Comparison

6.2. 比较

Method CD-S↓ FS-S↑ CD-O↓ FS-O↑ IoU-B↑ ACDC 0.104 39.46 0.072 41.99 0.541 InstPIFU 0.092 39.12 0.103 38.29 0.436 Gen3DSR 0.083 38.95 0.071 39.13 0.459 ours 0.052 56.18 0.057 56.50 0.603 Table 2. Quantitative comparison of scene reconstruction performance on the 3D-Front indoor dataset. We evaluate different methods based on Chamfer Distance (CD) for shape accuracy, F-Score (FS) for object-level reconstruction quality, and Intersection over Union (IoU) for scene-level overlap.

Method	CD-S↓	FS-S↑	CD-O↓	FS-O↑	IoU-B↑
ACDC	0.104	39.46	0.072	41.99	0.541
InstPIFU	0.092	39.12	0.103	38.29	0.436
Gen3DSR	0.083	38.95	0.071	39.13	0.459
ours	0.052	56.18	0.057	56.50	0.603

方法 CD-S↓ FS-S↑ CD-O↓ FS-O↑ IoU-B↑ ACDC 0.104 39.46 0.072 41.99 0.541 InstPIFU 0.092 39.12 0.103 38.29 0.436 Gen3DSR 0.083 38.95 0.071 39.13 0.459 本文 0.052 56.18 0.057 56.50 0.603 表 2. 在 3D-Front 室内数据集上的场景重建性能定量比较。我们按形状精度的 Chamfer 距离（CD）、物体级重建质量的 F-Score（FS），以及场景级重叠的交并比（IoU）评估不同方法。

方法	CD-S↓	FS-S↑	CD-O↓	FS-O↑	IoU-B↑
ACDC	0.104	39.46	0.072	41.99	0.541
InstPIFU	0.092	39.12	0.103	38.29	0.436
Gen3DSR	0.083	38.95	0.071	39.13	0.459
本文	0.052	56.18	0.057	56.50	0.603

Qualitative Comparisons We first evaluate our method, CAST, against state-of-the-art single-image scene reconstruction techniques on open-vocabulary scenarios. We also included images used by ACDC and Gen3DSR to further demonstrate the scene reconstruction results of different methods. Fig. 6 illustrates the performance of three methods—(1) the retrieval-based approach ACDC (Dai et al., 2024), (2) the generation-based method Gen3DSR (Dogaru et al., 2024), and (3) our proposed CAST—across both reference and novel views. Our results highlight CAST’s superior ability to accurately reconstruct scenes in diverse settings, including indoor and outdoor environments, close-up perspectives, and AI-generated imagery.

定性比较 我们首先在开放词汇情形下，把 CAST 与先进的单图像场景重建技术进行比较。我们也纳入了 ACDC 和 Gen3DSR 使用的图像，以进一步展示不同方法的场景重建结果。图 6 展示了三种方法在参考视图和新视图上的表现：（1）基于检索的 ACDC（Dai et al., 2024），（2）基于生成的 Gen3DSR（Dogaru et al., 2024），以及（3）我们提出的 CAST。结果突出了 CAST 在多样设定中准确重建场景的能力，包括室内外环境、特写视角以及人工智能生成的图像。

As shown in Fig.6, CAST distinguishes itself from both ACDC and Gen3DSR through innovative advancements. Unlike ACDC, which is limited to indoor scenes and relies on large datasets for object retrieval, often producing objects similar to those in the scene rather than the objects themselves, CAST supports open-vocabulary generalization. This allows CAST to accurately reconstruct objects in varied and complex environments. While ACDC uses simple bounding boxes as proxies, CAST combines image-based physical priors with mesh optimization to effectively manage complex scenes. In contrast to Gen3DSR, CAST employs direct 3D generation via a Masked Autoencoder, eliminating the error-prone 2D inpainting step. This results in smoother meshes, significantly outperforming Gen3DSR in single-object generation quality, particularly in challenging scenes. Moreover, Gen3DSR’s lack of simulation often leads to issues such as interpenetration or floating objects, causing scenes to appear consistent only from the input viewpoint and degrading novel view rendering. CAST, by contrast, ensures robust scene consistency across perspectives. CAST demonstrates robust scene reconstructions under varying conditions, underscoring its versatility for a broad range of real-world and generated scenarios.

如图 6 所示，CAST 通过若干新进展与 ACDC 和 Gen3DSR 区分开来。ACDC 限于室内场景，并依赖大型数据集做物体检索，常常生成与场景中物体相似、却并非物体本身的结果；CAST 则支持开放词汇泛化，从而能够在多变而复杂的环境中准确重建物体。ACDC 用简单边界框作为代理，CAST 则把基于图像的物理先验与网格优化结合起来，以有效管理复杂场景。与 Gen3DSR 相比，CAST 通过掩码自编码器直接进行三维生成，去掉了容易出错的二维补全步骤。这会得到更平滑的网格，在单物体生成质量上明显超过 Gen3DSR，在困难场景中尤其如此。此外，Gen3DSR 缺少仿真，常常导致穿透或悬浮，使场景仅从输入视点看起来一致，而新视角渲染则会变差。CAST 则能保证跨视角的稳健场景一致性。CAST 在多变条件下展现出稳健的场景重建，说明它适用于广泛的真实与生成情形。

To assess both the visual fidelity and semantic accuracy of the generated scenes, we employ two complementary evaluation methods including CLIP Score (Zhengwentai, 2023) and GPT-4 Reasoning. We compute the CLIP score between the rendered scene and the input image to measure overall reconstruction quality and visual similarity. To minimize environmental distractions, we remove backgrounds from both the rendered and reference images before computing the score. We additionally leverage GPT-4 to rank the generated scenes based on various semantic aspects, including object arrangement, physical relations, and scene realism. This semantic feedback helps identify alignment or contextual errors that might not be apparent through pixel-based scores alone.

为了评估生成场景的视觉保真度和语义准确性，我们采用两种互补的评价方法：CLIP 分数（Zhengwentai, 2023）和 GPT-4 推理。我们计算渲染场景与输入图像之间的 CLIP 分数，以衡量整体重建质量和视觉相似度。为减少环境干扰，计算分数前会去掉渲染图像和参考图像的背景。我们还利用 GPT-4 根据物体排布、物理关系和场景真实感等语义方面对生成场景排序。这种语义反馈有助于发现仅靠像素分数可能看不出的对齐或上下文错误。

Beyond automated metrics, we also conducted a user study focusing on two key aspects including Visual Quality (VQ) and Physical Plausibility (PP). We randomly selected paired reference, novel, and target views, asking participants to choose which method’s output best matched the input image in terms of both similarity and overall aesthetics. To reduce potential biases introduced by visual resemblance, participants in a separate session only viewed rendered results—without the original input images—and judged which scene appeared more realistic based on physical constraints and common sense (e.g., preventing floating objects or improbable contacts).

在自动指标之外，我们还进行了用户研究，关注两个关键方面：视觉质量（VQ）和物理合理性（PP）。我们随机选择成对的参考视图、新视图和目标视图，请参与者判断哪种方法的输出在相似度和整体美感上最匹配输入图像。为减少视觉相似带来的潜在偏差，参与者在另一场次中只观看渲染结果——不看原始输入图像——并根据物理约束和常识判断哪个场景看起来更真实（例如，避免悬浮物体或不可能的接触）。

As shown in Tab. 1, CAST outperforms both ACDC and Gen3DSR in all four evaluated metrics, confirming its effectiveness in producing scenes that are visually coherent and physically plausible.

如表 1 所示，CAST 在全部四项评价指标上都超过 ACDC 和 Gen3DSR，证实它能够有效生成视觉连贯、物理合理的场景。

Figure 7. We evaluate the generation performance with and without the occlusion-aware generation module. The RGB and normal renderings of the object highlight the significance of this module in ensuring the completeness and high quality of the generated object.

图 7. 我们评估有无遮挡感知生成模块时的生成表现。物体的 RGB 和法向渲染突出了该模块对保证生成物体完整性和高质量的重要性。

Quantitative Comparisons Although CAST is designed to handle open-vocabulary scenes, many such scenes lack mesh ground truth, which complicates direct quantitative comparisons. To address this, we perform additional evaluations on the 3DFront dataset (Fu et al., 2021). This dataset offers ground-truth meshes alongside corresponding rendered images, enabling a more precise assessment of both object-level and scene-level reconstructions. We compare our method with InstPIFu (Liu et al., 2022), ACDC (Dai et al., 2024), and Gen3DSR (Dogaru et al., 2024). We compute Chamfer Distance and F-Score at the object level, as well as IoU, Chamfer Distance, and F-Score at the scene level, to assess both the fidelity of individual object geometries and the accuracy of their spatial layout. To ensure fairness, we replace the segmentation modules in other methods with ground-truth (GT) masks so that any differences stem purely from reconstruction ability rather than object partitioning.

定量比较 尽管 CAST 被设计用来处理开放词汇场景，许多这类场景缺少网格真值，这使直接定量比较变得复杂。为此，我们在 3DFront 数据集（Fu et al., 2021）上做额外评估。该数据集提供真值网格及相应渲染图像，从而能够更精确地评估物体级和场景级重建。我们把方法与 InstPIFu（Liu et al., 2022）、ACDC（Dai et al., 2024）和 Gen3DSR（Dogaru et al., 2024）比较。我们在物体级计算 Chamfer 距离和 F-Score，在场景级计算 IoU、Chamfer 距离和 F-Score，以同时评估单个物体几何的保真度及其空间布局的准确性。为了保证公平，我们把其他方法中的分割模块替换为真值（GT）掩码，使差异纯粹来自重建能力，而不是物体划分。

As summarized in Tab. 2, CAST not only achieves higher object-level generation quality but also surpasses existing approaches in scene layout accuracy. Even within the constraints of an indoor dataset, our method demonstrates robust performance and consistent improvements over competing baselines.

如表 2 所总结，CAST 不仅在物体级生成质量上更高，在场景布局准确性上也超过已有方法。即便在室内数据集的约束下，我们的方法也展现出稳健表现，并相对竞争基线持续改进。

6.3. Evaluations

6.3. 评估

Figure 8. A stack of books with varying lengths and widths directly generated as a single complex object, demonstrating how point cloud conditioning enhances the preservation of scale, dimensions, and local details compared to traditional methods.

图 8. 一摞长度和宽度各不相同的书被直接生成为一个复杂物体，说明与传统方法相比，点云条件如何更好地保持尺度、尺寸和局部细节。

Figure 9. Comparative evaluation of pose estimation methods. Our pose alignment module demonstrates superior alignment accuracy compared to Iterative Closest Point (ICP) and differentiable rendering (DR).

图 9. 姿态估计方法的比较评估。与迭代最近点（ICP）和可微渲染（DR）相比，我们的姿态对齐模块展现出更高的对齐精度。

To elucidate the individual contributions of key components in CAST, we conducted a series of ablation studies. These experiments systematically removed or altered specific components to assess their impact on overall performance. The ablation studies focus on several key design choices: occlusion-aware object generation, point-cloud conditioning, generative alignment, and the physics-aware correction process.

为阐明 CAST 中关键部件各自的贡献，我们进行了一系列消融研究。这些实验系统地移除或改动特定部件，以评估它们对整体性能的影响。消融研究关注几项关键设计选择：遮挡感知的物体生成、点云条件、生成式对齐，以及物理感知校正过程。

Ablation on Occlusion-Aware Generation Occlusions are a significant challenge in scenes with complex objects. To assess the effectiveness of the Masked Autoencoder (MAE) in handling occlusions, we performed an ablation study comparing generation results with and without MAE components. As shown in Fig. 7, the results highlight the importance of the occlusion-aware module. Without MAE, the generated objects for partially occluded regions exhibit significant degradation. For example, the spaceship appears fragmented and incomplete, while the cup is depicted as broken with missing parts. In contrast, when MAE conditioning is applied, the model successfully infers and fills the occluded regions, resulting in more accurate and visually coherent generations that align better with the input image. This demonstrates the critical role of the occlusion-aware module in ensuring that occluded objects are reconstructed accurately, improving both the completeness and realism of the final 3D scene.

遮挡感知生成的消融 在包含复杂物体的场景中，遮挡是一项显著挑战。为评估掩码自编码器（MAE）处理遮挡的效果，我们进行了有无 MAE 部件的生成结果比较。如图 7 所示，结果突出了遮挡感知模块的重要性。没有 MAE 时，部分被遮挡区域的生成物体会出现明显退化。例如，飞船显得破碎而不完整，杯子则被画成缺损、缺少部件。相比之下，施加 MAE 条件后，模型能够成功推断并填补被遮挡区域，得到更准确、视觉上更连贯、也更符合输入图像的生成结果。这说明遮挡感知模块对于准确重建被遮挡物体、提高最终三维场景的完整性和真实感至关重要。

Ablation on Partial Point Cloud Conditioning We conducted an ablation study to investigate the role of canonical space partial point cloud conditioning in our generation process. Although directly generating from the input image can produce visually plausible results. In the absence of pixel-level alignment, the model struggles to maintain correct object quantity and scale, resulting in unsatisfactory generation. To more effectively showcase the importance of point cloud conditioning in generating a single instance, we opted to directly generate a more complex instance structure: a stack of six books with varying lengths and widths. As depicted in Fig. 8. When the generation process relies solely on the input image, without the benefit of point cloud conditioning, the results frequently exhibit inaccuracies in both the number and dimensions of the generated objects. By contrast, integrating point cloud conditioning introduces a robust geometric prior that significantly improves the precision of the generated scene. This enhancement ensures that objects with intricate shapes and varying dimensions are reconstructed more accurately, closely resembling their real-world counterparts depicted in the input image. This demonstrates the critical role of geometric priors in enhancing the fidelity of 3D scene generation by preserving the true dimensions and shapes.

残缺点云条件的消融 我们进行了消融研究，考察规范空间残缺点云条件在生成过程中的作用。虽然直接从输入图像生成可以产生视觉上合理的结果，但在缺少像素级对齐时，模型难以保持正确的物体数量和尺度，生成往往不能令人满意。为了更有效地展示点云条件在生成单个实例时的重要性，我们选择直接生成更复杂的实例结构：一摞长度和宽度各不相同的六本书。如图 8 所示。当生成过程只依赖输入图像、没有点云条件时，结果常常在生成物体的数量和尺寸上出现偏差。相比之下，加入点云条件会引入稳健的几何先验，显著提高生成场景的精度。这一增强确保形状复杂、尺寸多变的物体被更准确地重建，更接近输入图像中描绘的真实对应物。这说明几何先验通过保持真实尺寸和形状，对提高三维场景生成的保真度至关重要。

Figure 10. Comparison of scene reconstruction with and without relational graph constraints. By integrating relational graph constraints, our method ensures both physical plausibility and accurate alignment with the intended scene, maintaining correct spatial relations.

图 10. 有无关系图约束时的场景重建比较。通过融入关系图约束，我们的方法既保证物理合理性，又与预期场景准确对齐，并保持正确的空间关系。

Effectiveness of Alignment Generation To assess the effectiveness of our pose alignment module, we compared it with common pose estimation methods such as Iterative Closest Point (ICP) (Arun et al., 1987; Best, 1992) and differentiable rendering (Laine et al., 2020). The generated mesh was provided to different pose estimation methods to align it with the reference RGB image and its corresponding depth prediction. For the ICP method, we uniformly sampled a point cloud from the generated mesh and normalized both the sampled and estimated point clouds by their bounding boxes to address scale differences. We used the ICP implementation in Open3D (Zhou et al., 2018) to register these two normalized point clouds. For differentiable rendering, we optimized the rotation and translation parameters to transform the generated mesh so that the rendered image aligned with the reference RGB image. As shown in Fig. 9, our method surpasses both ICP and differentiable rendering in alignment accuracy. ICP often struggles with accurate pose estimation due to outliers in point clouds, unknown object scales, and symmetrical or repetitive geometries, which can lead to local minima. Differentiable rendering, on the other hand, is significantly impacted by occlusions in the RGB input, disrupting the optimization of object poses and preventing precise alignment with the input image. Our results show that our pose alignment module outperforms traditional ICP and differentiable rendering methods, demonstrating its robustness in accurately estimating object poses from generated meshes and improving alignment with input images.

对齐生成的有效性 为评估姿态对齐模块的效果，我们把它与迭代最近点（ICP）（Arun et al., 1987; Best, 1992）和可微渲染（Laine et al., 2020）等常见姿态估计方法进行比较。生成网格被交给不同姿态估计方法，以与参考 RGB 图像及其相应深度预测对齐。对 ICP 方法，我们从生成网格均匀采样点云，并用边界框对采样点云和估计点云做归一化，以处理尺度差异。我们使用 Open3D（Zhou et al., 2018）中的 ICP 实现来配准这两个归一化点云。对可微渲染，我们优化旋转和平移参数，以变换生成网格，使渲染图像与参考 RGB 图像对齐。如图 9 所示，我们的方法在对齐精度上超过 ICP 和可微渲染。ICP 常常因为点云中的离群点、未知物体尺度，以及对称或重复几何而难以准确估计姿态，容易陷入局部最优。可微渲染则明显受到 RGB 输入中遮挡的影响，这会破坏物体姿态的优化，使其无法与输入图像精确对齐。结果表明，我们的姿态对齐模块超过传统 ICP 和可微渲染方法，说明它能够从生成网格稳健地估计物体姿态，并改善与输入图像的对齐。

Effect of Physical Consistency Enforcement In CAST, physical constraints are essential for achieving realistic object interactions and maintaining spatial coherence within a scene. While we address common challenges such as occlusions and incomplete views, issues like floating objects, penetration, and misaligned spatial relations still occur. As shown in Fig. 10, scenes generated without relational constraints may appear physically inconsistent, when only physical simulation is applied, objects adhere to physical laws, but their relative positioning and overall arrangement can differ significantly from the intended scene (e.g., an onion might fall off a surface, disrupting the original composition). By incorporating relational graph constraints, our method ensures that the objects not only comply with physical feasibility but also align with the intended scene layout, preserving both the physical plausibility and the desired spatial relations.

施加物理一致性的效果 在 CAST 中，物理约束对于实现真实的物体交互并保持场景内的空间连贯至关重要。虽然我们处理了遮挡和不完整视图等常见挑战，悬浮、穿透和空间关系错位等问题仍可能出现。如图 10 所示，没有关系约束时生成的场景可能看起来物理不一致；若只施加物理仿真，物体会遵守物理定律，但其相对位置和整体排布可能与预期场景明显不同（例如，洋葱可能从表面掉下去，破坏原来的构图）。通过纳入关系图约束，我们的方法确保物体既符合物理可行性，又与预期场景布局对齐，同时保留物理合理性和所需的空间关系。

Method CD-S↓ FS-S↑ CD-O↓ FS-O↑ IoU-B↑ Vanilla 0.079 53.38 0.069 52.83 0.515 + MAE 0.064 53.79 0.066 54.32 0.548 + PCD 0.056 53.91 0.060 54.60 0.582 + iter. 0.052 56.18 0.057 56.50 0.603 Table 3. Quantitative ablation study of the MAE module, point cloud conditioning (PCD), and the iterative refinement strategy (iter.). For simplicity, we only display the added key component in each row.

Method	CD-S↓	FS-S↑	CD-O↓	FS-O↑	IoU-B↑
Vanilla	0.079	53.38	0.069	52.83	0.515
+ MAE	0.064	53.79	0.066	54.32	0.548
+ PCD	0.056	53.91	0.060	54.60	0.582
+ iter.	0.052	56.18	0.057	56.50	0.603

方法 CD-S↓ FS-S↑ CD-O↓ FS-O↑ IoU-B↑ 基础版 0.079 53.38 0.069 52.83 0.515 + MAE 0.064 53.79 0.066 54.32 0.548 + PCD 0.056 53.91 0.060 54.60 0.582 + 迭代 0.052 56.18 0.057 56.50 0.603 表 3. 对 MAE 模块、点云条件（PCD）和迭代 refinement 策略（iter.）的定量消融。为简洁起见，每一行只显示新加入的关键部件。

方法	CD-S↓	FS-S↑	CD-O↓	FS-O↑	IoU-B↑
基础版	0.079	53.38	0.069	52.83	0.515
+ MAE	0.064	53.79	0.066	54.32	0.548
+ PCD	0.056	53.91	0.060	54.60	0.582
+ 迭代	0.052	56.18	0.057	56.50	0.603

Quantitative Ablation Study of Different Modules To quantitatively evaluate the contribution of each module to overall performance, we conducted a comprehensive ablation study. As shown in Tab. 3, we assessed the impact of removing or altering key components on the final scene quality. The results indicate that each component contributes significantly to the overall performance of our method. The quantitative analysis further highlights the importance of each module in achieving high-quality, physically consistent, and realistic scene reconstructions.

不同模块的定量消融研究 为定量评估每个模块对整体性能的贡献，我们进行了全面的消融研究。如表 3 所示，我们评估了移除或改动关键部件对最终场景质量的影响。结果表明，每个部件都对我们方法的整体性能有显著贡献。定量分析进一步突出了每个模块在实现高质量、物理一致且真实的场景重建中的重要性。

Applications As shown in Fig. 11, CAST transforms a single image into a fully realized 3D scene, enabling a wide range of applications. This ability to reconstruct detailed environments powers physics-based animation by ensuring realistic object interactions. It also supports real-to-simulation workflows in robotics, allowing for accurate scene replication from real-world datasets. In game development, CAST facilitates the creation of immersive environments, where faithfully reconstructed scenes are seamlessly integrated into interactive worlds using Unreal Engine.

应用 如图 11 所示，CAST 把单张图像转变成完整实现的三维场景，从而支持广泛的应用。这种重建细致环境的能力通过保证真实的物体交互，为基于物理的动画提供动力。它也支持机器人中的 real-to-simulation 工作流，使人们能够从真实世界数据准确复制场景。在游戏开发中，CAST 便于创建沉浸式环境，忠实重建的场景可以用 Unreal Engine 无缝接入交互世界。

Figure 11. CAST enables realistic physics-based animations, immersive game environments, and efficient real-to-simulation transitions, driving innovation across various fields.

图 11. CAST 能够支持真实的物理动画、沉浸式游戏环境，以及高效的 real-to-simulation 转换，推动多个领域的创新。

7. Conclusions

7. 结论

In this paper, we introduced CAST, a novel single-image 3D scene reconstruction method that combines geometric fidelity, pixel-level alignment, and physically grounded constraints. By integrating scene decomposition, a perceptive 3D instance generation framework, and physical correction techniques, CAST addresses key challenges such as pose misalignment, object interdependencies, and partial occlusions. This structured pipeline results in 3D scenes that are both visually accurate and physically consistent, pushing beyond the limitations of traditional object-centric approaches. We validated CAST through extensive experiments and user studies, demonstrating significant performance improvements over state-of-the-art methods in terms of visual quality and physical plausibility. We anticipate that CAST will serve as a strong foundation for future developments in 3D generation, scene reconstruction, and immersive content creation.

本文介绍了 CAST，一种新的单图像三维场景重建方法，它把几何保真、像素级对齐和有物理根据的约束结合起来。通过整合场景分解、感知式三维实例生成框架和物理校正技术，CAST 回应了姿态错位、物体相互依赖和局部遮挡等关键挑战。这一结构化流程得到既视觉准确、又物理一致的三维场景，超越了传统以物体为中心的方法的限制。我们通过大量实验和用户研究验证了 CAST，表明它在视觉质量和物理合理性上相对先进方法有显著提升。我们预期 CAST 将成为三维生成、场景重建和沉浸式内容创作后续发展的坚实基础。

Figure 12. In some scenes, transparent glass, textiles, and fabrics are difficult to express, as the mesh struggles to represent them realistically.

图 12. 在一些场景中，透明玻璃、纺织物和织物难以表达，因为网格很难真实地表示它们。

Limitations and Future Work The quality of scene generation in CAST is heavily dependent on the underlying object generation model. At present, the model still lacks sufficient detail and precision, this limitation leads to noticeable inconsistencies in the generated objects, affecting their alignment and spatial relations in the scene. Additionally, the current mesh representation struggles with materials like textiles, glasses or fabrics, often appearing unnatural, and fails to accurately depict transparent materials, as shown in Fig. 12. Although additional modules have been incorporated to enhance object robustness and similarity, the need for more advanced and robust generation models remains. A more detailed and accurate object generator could significantly improve the overall scene quality and enhance its real-world applicability.

局限与未来工作 CAST 中场景生成的质量在很大程度上依赖于底层物体生成模型。目前，该模型仍缺少足够的细节和精度，这一限制会在生成物体中造成明显不一致，影响它们在场景中的对齐和空间关系。此外，当前网格表示难以处理纺织物、玻璃或织物一类材质，常常显得不自然，也无法准确描绘透明材质，如图 12 所示。尽管已经加入额外模块来增强物体的鲁棒性和相似度，仍需要更先进、更稳健的生成模型。更细致、更准确的物体生成器可以显著提高整体场景质量，并增强其现实适用性。

A notable limitation of the current method is the absence of lighting estimation and background modeling. Without realistic lighting, the interactions between objects and their surroundings may lack natural shading and illumination effects, impacting the visual realism and immersion of the generated 3D environments. For enhanced visual realism, we employ an off-the-shelf panoramic HDR generation tool (Hyper3D, 2025) combined with preset lighting conditions in blender manually. Future enhancements in CAST could benefit from integrating advanced techniques for lighting estimation and background modeling, which would significantly enrich the contextual depth and visual fidelity of the scenes.

当前方法的一个显著局限是缺少光照估计和背景建模。没有真实光照，物体与周围环境的交互可能缺少自然的阴影和照明效果，从而影响生成三维环境的视觉真实感和沉浸感。为了增强视觉真实感，我们使用现成的全景 HDR 生成工具（Hyper3D, 2025），并在 Blender 中手工结合预设光照条件。CAST 的后续增强可以受益于融入先进的光照估计和背景建模技术，这将显著丰富场景的上下文深度和视觉保真度。

In more complex scenes, the performance of the current method may experience slight degradation. Challenges such as intricate spatial layouts and dense object configurations could affect the accuracy of scene reconstruction to some extent. While CAST currently excels at reconstructing individual scenes, there is significant potential to utilize its outputs to build large-scale datasets, facilitating advanced research on fully learned scene or video generation pipelines. Expanding the variety and realism of generated scenes in this manner could further improve the robustness and applicability of 3D generative models in areas such as film production, simulation, and immersive media.

在更复杂的场景中，当前方法的表现可能略有下降。错综的空间布局和密集的物体配置等挑战，可能在一定程度上影响场景重建的准确性。虽然 CAST 目前擅长重建单个场景，它的输出仍有很大潜力被用来构建大规模数据集，从而推动对完全学习的场景或视频生成流程的进一步研究。以这种方式扩展生成场景的多样性和真实感，可以进一步提高三维生成模型在影视制作、仿真和沉浸式媒体等领域的鲁棒性和适用性。

References

参考文献

Yuille and Kersten (2006) Alan Yuille and Daniel Kersten. 2006. Vision as Bayesian inference: analysis by synthesis? Trends in cognitive sciences 10, 7 (2006), 301-308. Zhang et al . (2023) Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 2023. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1-16. Zhang et al . (2024b) Jiancheng Zhang, Haijin Zeng, Yongyong Chen, Dengxiu Yu, and Yin-Ping Zhao. 2024b. Improving Spectral Snapshot Reconstruction with Spectral-Spatial Rectification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 25817-25826. Zhang et al . (2024a) Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. 2024a. CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets. ACM Transactions on Graphics (TOG) 43, 4 (2024), 1-20. Zhengwentai (2023) SUN Zhengwentai. 2023. clip-score: CLIP Score for PyTorch. https://github.com/taited/clip-score . Version 0.1.1. Zhong et al . (2025) Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. 2025. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In European Conference on Computer Vision . Springer, 407-423. Zhou et al . (2018) Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. 2018. Open3D: A Modern Library for 3D Data Processing. ArXiv abs/1801.09847 (2018).

Appendix A GPT-4v prompt

附录 A GPT-4v 提示词

Acknowledgements. The following prompt instructs GPT-4V to act as an object-relationship analyst for a numbered scene image. It defines six relationship types (Stack, Lean, Hang, Clamped, Contained, Edge/Point), sets strict formatting rules (JSON objects in a list). For full details, see the appendix prompt as below.

致谢。 下面的提示词指示 GPT-4V 作为编号场景图像的物体-关系分析器。它定义了六种关系类型（堆叠、倚靠、悬挂、夹持、包含、边或点接触），并设定严格的格式规则（列表中的 JSON 对象）。完整细节见下面的附录提示词。

prompting_text_system = """You␣are␣an␣expert␣in␣object␣recognition␣and␣spatial␣reasoning. ###␣Task␣Description␣###Analyze␣an␣image␣with␣numbered␣objects␣and␣determine␣their␣relationships.For␣each␣pair␣of␣related␣objects,␣output␣a␣JSON␣object␣containing␣the␣relationship␣details.Ensure␣you␣output␣all␣possible␣relationships,␣even␣those␣that␣may␣be␣difficult␣to␣judge␣or␣less␣obvious. ###␣Relationship␣Definitions␣###1.␣Stack:␣Object␣1␣is␣on␣top␣of␣Object␣2␣(Object␣2␣supports␣Object␣1␣from␣below)2.␣Lean:␣Object␣1␣is␣leaning␣against␣Object␣2␣(Object␣2␣supports␣Object␣1␣laterally)3.␣Hang:␣Object␣1␣is␣hanging␣from␣Object␣2␣(Object␣2␣supports␣Object␣1␣from␣above)4.␣Clamped:␣Object␣1␣is␣clamped␣by␣Object␣2␣(Object␣2␣grips␣Object␣1␣on␣multiple␣sides)5.␣Contained:␣Object␣1␣is␣inside␣Object␣2␣(Object␣2␣encloses␣Object␣1)6.␣Edge/Point:␣Object␣1␣is␣touching␣Object␣2␣at␣an␣edge␣or␣point␣(minimal␣contact,␣no␣significant␣support) ###␣Important␣Note␣###Only␣objects␣that␣are␣in␣contact␣with␣each␣other␣should␣have␣a␣relationship.For␣each␣relationship,␣always␣ensure␣the␣following:1.␣Use␣the␣correct␣relationship␣type.2.␣Provide␣a␣clear␣explanation␣of␣the␣relationship.3.␣For␣cases␣that␣are␣in␣contact␣but␣hard␣to␣choose␣which␣type,␣use␣"Stack".###␣Output␣Format␣###{␣␣’pair’:␣[obj1_num,␣obj2_num],␣␣’relationship’:␣’Stack’/’Lean’/’Hang’/’Clamped’/’Contained’/’Edge/Point’,␣␣’reason’:␣’explanation’} ###␣Examples␣###{␣’pair’:␣[1,␣2],␣␣’relationship’:␣’Stack’,␣␣␣␣␣␣’reason’:␣’Book␣(1)␣is␣stacked␣on␣top␣of␣Table␣(2)’␣}{␣’pair’:␣[3,␣4],␣␣’relationship’:␣’Lean’,␣␣␣␣␣␣␣’reason’:␣’A␣chair␣(3)␣is␣leaning␣against␣a␣wall␣(4)’␣}{␣’pair’:␣[5,␣6],␣␣’relationship’:␣’Hang’,␣␣␣␣␣␣␣’reason’:␣’A␣lamp␣(5)␣is␣hanging␣from␣the␣ceiling␣(6)’␣}{␣’pair’:␣[7,␣8],␣␣’relationship’:␣’Clamped’,␣␣␣␣’reason’:␣’A␣pipe␣(7)␣is␣clamped␣by␣a␣bracket␣(8)’␣}{␣’pair’:␣[9,␣10],␣’relationship’:␣’Contained’,␣␣’reason’:␣’A␣pencil␣(9)␣is␣inside␣a␣pencil␣case␣(10)’␣}{␣’pair’:␣[11,12],␣’relationship’:␣’Edge/Point’,␣’reason’:␣’A␣book␣(11)␣and␣a␣pen␣(12)␣are␣touching␣at␣the␣edge’␣}Only␣objects␣that␣are␣in␣contact␣with␣each␣other␣should␣have␣a␣relationship.""" content_user = [{"type": "text","text": (’’’␣␣␣␣␣␣␣␣␣␣␣␣"###␣Object␣Details␣###"␣␣␣␣␣␣␣␣␣␣␣␣"I␣have␣labeled␣a␣bright␣numeric␣ID␣at␣the␣center␣for␣each␣visual␣object␣in␣the␣image."␣␣␣␣␣␣␣␣␣␣␣␣"Please␣analyze␣all␣relationships␣between␣the␣numbered␣objects␣and␣output␣JSON␣objects␣following␣the␣specified␣format.␣"␣␣␣␣␣␣␣␣␣␣␣␣"Ensure␣each␣relationship␣includes:"␣␣␣␣␣␣␣␣␣␣␣␣"1.␣The␣correct␣relationship␣type"␣␣␣␣␣␣␣␣␣␣␣␣"2.␣A␣clear␣reason␣for␣the␣relationship"␣␣␣␣␣␣␣␣␣␣␣␣"List␣all␣relationships␣as␣a␣JSON␣array."␣␣␣␣␣␣␣␣’’’),},{"type": "image_url","image_url": {"url": f"data:image/png;base64,{base64_image}"},},] Listing 1: System prompt for GPT-4V

prompting_text_system = """You are an expert in object recognition and spatial reasoning. 任务描述 分析一幅带有编号物体的图像，并确定它们的关系。对每一对相关物体，输出一个包含关系细节的 JSON 对象。确保输出所有可能的关系，即使那些可能难以判断或不那么明显的关系。 关系定义 Stack（堆叠）：物体 1 在物体 2 上面（物体 2 从下方支撑物体 1） Lean（倚靠）：物体 1 靠在物体 2 上（物体 2 从侧面支撑物体 1） Hang（悬挂）：物体 1 悬挂在物体 2 上（物体 2 从上方支撑物体 1） Clamped（夹持）：物体 1 被物体 2 夹住（物体 2 从多个侧面握住物体 1） Contained（包含）：物体 1 在物体 2 内部（物体 2 包围物体 1） Edge/Point（边或点接触）：物体 1 在边缘或点处触及物体 2（接触很少，没有显著支撑） 重要说明 只有相互接触的物体才应具有关系。对每条关系，始终确保： 使用正确的关系类型。 提供对该关系的清楚解释。 对于已经接触但难以选择类型的情形，使用 "Stack"。 输出格式 { 'pair': [obj1_num, obj2_num], 'relationship': 'Stack'/'Lean'/'Hang'/'Clamped'/'Contained'/'Edge/Point', 'reason': 'explanation'} 示例 { 'pair': [1, 2], 'relationship': 'Stack', 'reason': 'Book (1) is stacked on top of Table (2)' }{ 'pair': [3, 4], 'relationship': 'Lean', 'reason': 'A chair (3) is leaning against a wall (4)' }{ 'pair': [5, 6], 'relationship': 'Hang', 'reason': 'A lamp (5) is hanging from the ceiling (6)' }{ 'pair': [7, 8], 'relationship': 'Clamped', 'reason': 'A pipe (7) is clamped by a bracket (8)' }{ 'pair': [9, 10], 'relationship': 'Contained', 'reason': 'A pencil (9) is inside a pencil case (10)' }{ 'pair': [11,12], 'relationship': 'Edge/Point', 'reason': 'A book (11) and a pen (12) are touching at the edge' }只有相互接触的物体才应具有关系。""" content_user = [{"type": "text","text": (''' "### 物体细节 ###" "我已经在图像中每个可见物体的中心标注了醒目的数字编号。" "请分析编号物体之间的所有关系，并按指定格式输出 JSON 对象。" "确保每条关系包括：" "1. 正确的关系类型" "2. 对该关系的清楚理由" "把所有关系列成一个 JSON 数组。" '''),},{"type": "image_url","image_url": {"url": f"data:image/png;base64,{base64_image}"},},] 列表 1：GPT-4V 的系统提示词

CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image

CAST：从一张 RGB 图像做部件对齐的三维场景重建

Abstract

摘要

1. Introduction

1. 引言

Keywords:

关键词：

2. Related Work

2. 相关工作

2.1. Single Image Scene Reconstruction

2.1. 单图像场景重建

2.2. Reconstruction as Generation

2.2. 把重建写成生成

2.3. Physics-Aware 3D Modeling

2.3. 物理感知的三维建模

3. Overview

3. 概述

Preprocessing

预处理

4. Perceptive 3D Instance Generation

4. 感知式三维实例生成

4.1. Occlusion-aware 3D Object Generation

4.1. 遮挡感知的三维物体生成

Canonical Point Cloud Conditioning

规范点云条件

4.2. Generative Alignment

4.2. 生成式对齐

4.3. Iterative Generation Procedure

4.3. 迭代生成流程

Step 1: Object Generation

步骤 1：物体生成

Step 2: Alignment

步骤 2：对齐

Step 3: Refinement

步骤 3：Refinement

5. Physics-Aware Correction

5. 物理感知校正

5.1. A Quick Primer to Rigid-Body Simulation

5.1. 刚体仿真速览

5.2. Problem Formulation and Physical Constraints

5.2. 问题形式化与物理约束

5.3. Scene Relation Graph

5.3. 场景关系图

5.4. Optimization with Physics-Aware Relation Graph

5.4. 基于物理感知关系图的优化

6. Result

6. 结果

6.1. Implementation Details

6.1. 实现细节

ObjectGen

ObjectGen

AlignGen

AlignGen

6.2. Comparison

6.2. 比较

Qualitative Comparisons

定性比较

Quantitative Comparisons

定量比较

6.3. Evaluations

6.3. 评估

Ablation on Occlusion-Aware Generation

遮挡感知生成的消融

Ablation on Partial Point Cloud Conditioning

残缺点云条件的消融

Effectiveness of Alignment Generation

对齐生成的有效性

Effect of Physical Consistency Enforcement

施加物理一致性的效果

Quantitative Ablation Study of Different Modules

不同模块的定量消融研究

Applications

应用

7. Conclusions

7. 结论

Limitations and Future Work

局限与未来工作

References

参考文献