RayZer: A Self-supervised Large View Synthesis Model
RayZer: A Self-supervised Large View Synthesis Model
RayZer：一种自监督的大规模新视角合成模型
Hanwen Jiang Hao Tan Peng Wang Haian Jin Yue Zhao Sai Bi Affiliation: The University of Texas at Austin Adobe Research Cornell University Kai Zhang Fujun Luan Kalyan Sunkavalli Qixing Huang Georgios Pavlakos Affiliation:
Abstract
摘要
We present RayZer, a self-supervised multi-view 3D Vision model trained without any 3D supervision, i.e., camera poses and scene geometry, while exhibiting emerging 3D awareness. Concretely, RayZer takes unposed and uncalibrated images as input, recovers camera parameters, reconstructs a scene representation, and synthesizes novel views. During training, RayZer relies solely on its self-predicted camera poses to render target views, eliminating the need for any ground-truth camera annotations and allowing RayZer to be trained with 2D image supervision. The emerging 3D awareness of RayZer is attributed to two key factors. First, we design a self-supervised framework, which achieves 3D-aware auto-encoding of input images by disentangling camera and scene representations. Second, we design a transformer-based model in which the only 3D prior is the ray structure, connecting camera, pixel, and scene simultaneously. RayZer demonstrates comparable or even superior novel view synthesis performance than “oracle” methods that rely on pose annotations in both training and testing. Project: https://hwjiang1510.github.io/RayZer/
我们提出 RayZer，这是一个自监督的多视图三维视觉模型。它在训练时不使用任何三维监督，也就是不使用相机位姿和场景几何，却仍能表现出涌现出的三维感知能力。具体而言，RayZer 以未给定位姿、也未经标定的图像为输入，恢复相机参数，重建场景表征，并合成新视角。训练阶段，RayZer 只依靠自己预测的相机位姿来渲染目标视图，因此不再需要任何真值相机标注，只用二维图像监督即可训练。RayZer 之所以能出现三维感知，主要有两个原因。第一，我们设计了一套自监督框架，通过把相机表征和场景表征拆开，实现对输入图像的三维感知自编码。第二，我们设计了一个基于 Transformer 的模型，其中唯一的三维先验是射线结构，它同时把相机、像素和场景联系起来。在新视角合成上，RayZer 可以达到甚至超过在训练和测试时都依赖位姿标注的“oracle”方法。项目页：https://hwjiang1510.github.io/RayZer/
Figure 1: We propose RayZer, a self-supervised multi-view 3D Vision model trained on unlabeled data without any annotations, e.g., camera pose labels. At inference, RayZer supports feed-forward novel view synthesis from unposed & uncalibrated images. RayZer achieves novel view synthesis performance comparable to that of supervised “oracle” methods (GS-LRM and LVSM), which require camera labels in both training and inference, and even outperforms them when they rely on (potentially noisy) COLMAP camera annotations. We show two examples on the right, where COLMAP camera annotations lead to consistent failures of GS-LRM and LVSM during inference.
图 1：我们提出 RayZer，这是一个在无标注数据上训练的自监督多视图三维视觉模型，训练时不使用任何标注，例如相机位姿标签。推理时，RayZer 支持从未给定位姿且未标定的图像出发，做前馈式新视角合成。RayZer 的新视角合成性能可以与受监督的“oracle”方法（GS-LRM 和 LVSM）相当，而这些方法在训练和推理时都需要相机标注；当它们依赖（可能带噪声的）COLMAP 相机标注时，RayZer 甚至还能超过它们。右侧给出两个例子：COLMAP 相机标注使 GS-LRM 和 LVSM 在推理时持续失败。
1 Introduction
1 引言
Self-supervised learning has driven the rise of foundation models, enabling training on vast amounts of unlabeled data and fueled by the scaling law [38]. This paradigm has proven highly effective for LLMs [60], VLMs [2], and visual generation [56]. In contrast, 3D Vision models still rely heavily on ground-truth 3D geometry and camera pose labels [77, 28], which are usually estimated from time-consuming optimization methods, e.g., COLMAP [65], and are not always perfect. This reliance limits both learning scalability and effectiveness. To break free from this constraint, we move beyond the supervised paradigm and ask: how far can we push a 3D Vision model without any 3D supervision?
自监督学习推动了基础模型的兴起，使模型可以在大量无标注数据上训练，并受益于扩展定律（scaling law）[38]。这一范式在大语言模型[60]、视觉语言模型[2]和视觉生成[56]中都已证明非常有效。相比之下，三维视觉模型仍然严重依赖真值三维几何和相机位姿标签[77, 28]。这些标签通常由耗时的优化方法估计得到，例如 COLMAP[65]，而且并不总是准确。这种依赖既限制了学习的可扩展性，也限制了效果。为了摆脱这一约束，我们走出受监督范式，并追问：如果完全不用三维监督，三维视觉模型还能走多远？
In this paper, we present RayZer, a large multi-view 3D model trained with self-supervision and exhibiting emerging 3D awareness. The input of RayZer is unposed and uncalibrated multi-view images, sampled from continuous video frames or unordered multi-view captures. RayZer first recovers the camera parameters, then reconstructs the scene representation, and finally renders novel views. The key insight of our self-supervised training is to use the camera poses predicted by RayZer itself to render views that provide photometric supervision, rather than following the standard protocol of using ground truth poses for rendering [74, 29, 92]. Thus, RayZer can be trained with zero 3D supervision, i.e., no 3D geometry or camera pose supervision. During inference, RayZer predicts camera and scene representations in a feed-forward manner, without requiring per-scene optimization. We show inference results in Fig. 1.
本文提出 RayZer，这是一个以自监督方式训练、并表现出涌现三维感知的大规模多视图三维模型。RayZer 的输入是未给定位姿、也未经标定的多视图图像，可从连续视频帧或无序的多视图采集中采样。RayZer 先恢复相机参数，再重建场景表征，最后渲染新视角。我们自监督训练的关键想法是：用 RayZer 自己预测的相机位姿去渲染视图，从而提供光度监督，而不是沿用使用真值位姿来渲染的常规做法[74, 29, 92]。因此，RayZer 可以在零三维监督下训练，也就是没有三维几何监督，也没有相机位姿监督。推理时，RayZer 以前馈方式预测相机和场景表征，不需要按场景再做优化。推理结果见图 1。
As RayZer uses the camera poses predicted by itself for training, this self-supervised task can be interpreted as 3D-aware image auto-encoding [61, 95, 41]. Initially, RayZer disentangles input images into camera parameters and scene representations (reconstruction). It then re-entangles these predicted representations back into images (rendering). To facilitate this disentanglement, we control the information flow. As shown in Fig. 2, we divide all images into two parts: one set predicts the scene representation (input views), while the other offers photometric self-supervision (target views). This is achieved by using estimated poses of the second set to render the scene representation predicted from the first set, thereby preventing trivial solutions that are not 3D-aware.
由于 RayZer 在训练时使用自己预测的相机位姿，这一自监督任务也可以理解为三维感知的图像自编码[61, 95, 41]。首先，RayZer 把输入图像拆开为相机参数和场景表征（重建）；然后，再把这些预测出的表征重新组合成图像（渲染）。为了促成这种拆分，我们控制信息流动。如图 2 所示，我们把全部图像分成两部分：一组用来预测场景表征（输入视图），另一组提供光度上的自监督（目标视图）。具体做法是：用第二组的估计位姿，去渲染由第一组预测得到的场景表征，从而避免出现并不具备三维感知的平凡解。
To facilitate self-supervised learning, RayZer is built only with transformers – no 3D representation, hand-crafted rendering equation, or 3D-informed architectures. This design is motivated by self-supervised large models in other modalities [6, 2, 56], enabling RayZer to flexibly and effectively learn domain-specific knowledge. The only 3D prior incorporated in RayZer is the ray structure, which simultaneously models the relationship between camera, pixels (image), and scene. Concretely, RayZer first predicts camera poses, which are then converted into pixel-aligned Plücker ray maps [57] to guide the scene reconstruction that follows. This ray-based representation serves as a strong prior for addressing the chicken-and-egg problem of structure and motion [68], effectively allowing the camera and scene representations to regularize each other during training.
为了便于自监督学习，RayZer 只由 Transformer 构成，不使用三维表征、手写渲染方程，或带有三维先验的结构。这一设计受到其他模态中自监督大模型的启发[6, 2, 56]，使 RayZer 能够灵活而有效地学习领域知识。RayZer 中写入的唯一三维先验是射线结构，它同时刻画相机、像素（图像）和场景之间的关系。具体来说，RayZer 先预测相机位姿，再把它们转换成与像素对齐的 Plücker 射线图[57]，用来引导随后的场景重建。这种基于射线的表征，为结构与运动这一先有鸡还是先有蛋的问题[68]提供了强先验，使相机表征和场景表征在训练中能够互相约束。
We evaluate RayZer on three datasets, including both scene-level and object-level data with different camera configurations. We observe that RayZer demonstrates comparable or even better novel view synthesis performance than “oracle” methods [91, 33] that use pose labels in both training and testing. Interestingly, we identify that potentially noisy pose annotations from COLMAP can limit the performance of “oracle” models. The results not only demonstrate the effectiveness of RayZer, but also shows the potential of 3D Vision models to break free from supervised learning.
我们在三个数据集上评估 RayZer，其中既有场景级数据，也有物体级数据，相机配置各不相同。我们看到，RayZer 的新视角合成性能可以达到甚至超过那些在训练和测试时都使用位姿标签的“oracle”方法[91, 33]。有意思的是，我们还发现，COLMAP 可能带噪声的位姿标注会限制“oracle”模型的表现。这些结果不仅说明了 RayZer 的有效性，也表明三维视觉模型有可能摆脱对监督学习的依赖。
2 Related Work
2 相关工作
Large-scale 3D Vision Models. 3D Vision models learn 3D representations and priors from data [15, 58, 23, 39, 59, 71, 72, 94, 93]. Recently, researchers have developed large-scale models to acquire general 3D knowledge. One research direction focuses on designing improved model architectures that incorporate the inductive biases of multi-view stereo [86, 75, 10, 14] and epipolar geometry [25, 19, 9, 13]. Another line of work leverages full transformer models that intentionally omit architectural 3D inductive biases [62, 54, 29]. For example, LEAP [29], LRMs [28, 74, 91, 78, 98], and DUSt3R [77, 43, 20, 83, 76] are the first works employing transformers to convert 2D images into 3D representations. SRT [62] and LVSM [33] further replace 3D representations and physical rendering equations with latent representations and learned rendering functions, improving performance and scalability. However, they still require ground-truth camera poses for supervised training and/or accurate camera annotations during inference. To achieve scalable supervised learning, MegaSynth [32] and Stereo4D [34] leverage synthetic data and stereo videos to expand the data scale, however, curating data for different tasks can be laborious. In contrast, RayZer explores self-supervised training to break free from supervised learning.
大规模三维视觉模型。三维视觉模型从数据中学习三维表征和先验[15, 58, 23, 39, 59, 71, 72, 94, 93]。近来，研究者开始构建大规模模型，以获取更一般的三维知识。一条路线是改进模型结构，把多视图立体[86, 75, 10, 14]和极几何[25, 19, 9, 13]的归纳偏置写进网络。另一条路线则使用纯 Transformer 模型，有意去掉结构上的三维归纳偏置[62, 54, 29]。例如，LEAP[29]、各类 LRM[28, 74, 91, 78, 98] 以及 DUSt3R[77, 43, 20, 83, 76]，是较早用 Transformer 把二维图像转换成三维表征的工作。SRT[62] 和 LVSM[33] 进一步用潜在表征和学习到的渲染函数，替代三维表征和物理渲染方程，从而提升性能和可扩展性。不过，它们在受监督训练时仍需要真值相机位姿，并且/或者在推理时仍需要准确的相机标注。为了把监督学习做大规模，MegaSynth[32] 和 Stereo4D[34] 借助合成数据和立体视频来扩大数据规模，但为不同任务整理数据仍然费力。相比之下，RayZer 探索用自监督训练来摆脱对监督学习的依赖。
Figure 2: Our proposed self-supervised training framework. This is an abstract design that we later operationalize with our RayZer model (illustrated in Fig. 3 and Sec. 4). We divide the input images into two sets and . We predict the scene representation from , and use the predicted cameras of (shown in orange) to render the scene. We leverage photometric loss between raw input and its prediction for training.
图 2：我们提出的自监督训练框架。这是一个抽象设计，随后会用 RayZer 模型具体实现（见图 3 和第 4 节）。我们把输入图像分成两组 和 。场景表征由 预测，再用 的预测相机（以橙色标出）去渲染场景。训练时，我们在原始输入 与其预测 之间使用光度损失。
Self-supervised 3D Representation Learning. Learning 3D-aware representations from unlabeled image data is a long-standing problem in 3D Vision. One line of work leverages single-view images. However, they either only work for a specific category [82, 37, 53, 7, 47, 52] or can only recover partial observations [8, 64, 81]. Some works explore semi-supervised learning and achieve better scalability [84, 30], but performance is still highly restricted to the model weights, which are initialized by fully supervised training [85]. The most relevant work is self-supervised learning from multi-view images [69, 80, 79]. For example, Zhou et al. [95], Lai et al. [41], and their following works [21, 87] use camera motion as 2D or 3D warping operations to regularize learning. However, this strong inductive bias limits the learning effectiveness. RUST [63] is a pioneering work in learning latent scene representations from unposed imagery. RayZer is different in three aspects. First, RayZer initially estimates camera poses and uses poses to condition the following latent reconstruction. In contrast, RUST operates in an inverse pipeline – it first reconstructs the scene and then estimates the camera poses. Second, RayZer employs different explicit pose representations to improve information disentanglement and 3D awareness, enabling novel view synthesis by geometrically interpolating predicted poses. Instead, RUST uses a latent pose representation, which makes scene-pose disentanglement challenging and they are not explicitly 3D-aware. Third, RayZer follows the model architecture of LVSM [33] using pure self-attention in transformers, which is different from RUST that follows SRT using convolution and cross-attention.
自监督的三维表征学习。从无标注图像数据中学习具备三维感知的表征，是三维视觉中的一个长期问题。一条路线使用单视图图像。不过，它们要么只适用于特定类别[82, 37, 53, 7, 47, 52]，要么只能恢复部分观测[8, 64, 81]。也有工作探索半监督学习，可扩展性更好[84, 30]，但性能仍然高度受限于由全监督训练初始化的模型权重[85]。最相关的工作是从多视图图像出发的自监督学习[69, 80, 79]。例如，Zhou 等人[95]、Lai 等人[41]及其后续工作[21, 87]把相机运动当作二维或三维的扭曲操作，用来约束学习。但这种很强的归纳偏置会限制学习效果。RUST[63] 是从无位姿图像中学习潜在场景表征的开创性工作。RayZer 与它有三点不同。第一，RayZer 先估计相机位姿，再用位姿作为条件去做随后的潜在重建；RUST 则是相反的流程，先重建场景，再估计相机位姿。第二，RayZer 使用不同的显式位姿表征，以改善信息拆分和三维感知，并能够通过几何插值预测位姿来合成新视角。RUST 使用的是潜在位姿表征，场景与位姿更难拆开，而且并不显式具备三维感知。第三，RayZer 沿用 LVSM[33] 的模型结构，在 Transformer 中只使用纯自注意力；RUST 则沿用 SRT，使用卷积和交叉注意力。
Figure 3: RayZer self-supervised learning framework. RayZer takes in unposed and uncalibrated multi-view images and predicts per-view camera parameters and a scene representation, which supports novel view rendering. (Left) RayZer first estimates camera parameters, where one view is selected as the canonical reference view (in blue box). RayZer predicts the intrinsics and the relative camera poses of all views. The predicted cameras are then converted into pixel-aligned Plücker ray maps . (Middle) RayZer uses a subset of input images, , as well as their previously predicted camera Plücker ray maps, , to predict a latent scene representation. Here, the Plücker ray maps, , serve as an effective condition for scene reconstruction. (Right) RayZer can render a target image given the scene representation and a target camera. During training, we use , which is the previously predicted cameras Plücker ray maps of , to render . This allows training RayZer end-to-end with self-supervised photometric losses between inputs and their renderings .
图 3：RayZer 的自监督学习框架。RayZer 接收未给定位姿、也未经标定的多视图图像 ，预测每张视图的相机参数以及一个支持新视角渲染的场景表征。（左）RayZer 先估计相机参数，其中一张视图被选为规范参考视图（蓝色框）。RayZer 预测所有视图的内参和相对相机位姿 。预测出的相机再被转换成与像素对齐的 Plücker 射线图 。（中）RayZer 用输入图像的一个子集 ，以及先前预测得到的相机 Plücker 射线图 ，来预测潜在场景表征。这里，Plücker 射线图 是场景重建的有效条件。（右）给定场景表征 和目标相机，RayZer 可以渲染目标图像。训练时，我们使用 ，也就是 先前预测得到的相机 Plücker 射线图，来渲染 。这样就可以用输入 与其渲染结果 之间的自监督光度损失，对 RayZer 做端到端训练。
Optimization-based Unsupervised SfM, SLAM, and NVS. Although these methods are not directly comparable to RayZer, we discuss them due to the similar input-output formulations. In detail, these methods optimize target predictions on a per-scene basis [67, 65], while RayZer is a feed-forward parametrized model, learning priors by training on large data. The traditional SfM, SLAM, and NVS methods are unsupervised [65, 26]. Although generally performing well, they are restricted by the complicated hand-crafted workflow, leading to requirements of dense-view inputs [89], slow speed [65], and sensitivity to hyper-parameters [70]. Recent optimization-based NeRF and 3DGS works can also perform NVS from unposed images [48, 4, 22, 66]. However, they do not have learnable model parameters to encode priors, thus requiring off-the-shelf models trained with 3D supervision as regularization or providing initialization.
基于优化的无监督 SfM、SLAM 与新视角合成。这些方法并不能与 RayZer 直接比较，但由于输入输出形式相近，我们仍加以讨论。具体而言，这些方法按场景优化目标预测[67, 65]，而 RayZer 是前馈的参数化模型，通过在大规模数据上训练来学习先验。传统的 SfM、SLAM 和新视角合成方法是无监督的[65, 26]。它们通常效果不错，但受制于复杂的手工流程，往往需要稠密视图输入[89]，速度较慢[65]，并且对超参数敏感[70]。近来基于优化的 NeRF 和 3DGS 工作，也可以从未给定位姿的图像出发做新视角合成[48, 4, 22, 66]。不过，它们没有可学习的模型参数来编码先验，因而需要借助在三维监督下训练好的现成模型来做正则，或提供初始化。
3 Preliminaries
3 预备知识
We introduce two important building blocks of RayZer, i.e., the latent set scene representation and how to render it.
我们介绍 RayZer 的两个重要构件，即潜在集合场景表征，以及如何对它进行渲染。
Latent set scene representation. Compressing data into tokens in latent space is a common practice in text, image, video, etc. Recently, this representation has also been extended to 3D research [62, 88, 63, 33]. In contrast to classical explicit (e.g., meshes and point clouds), implicit (e.g., NeRF [51] and SDF [55]), and hybrid (e.g., triplane [8] and 3DGS [40]) representations that are 3D-aware, the latent set representation is not explicitly 3D-aware. It serves as a compression of scene information, where the 3D-awareness properties are fully learned. The latent set scene representation can be denoted as , where is the number of tokens in the set and is the latent feature dimension.
潜在集合场景表征。把数据压缩成潜在空间中的 token，在文本、图像、视频等模态中已很常见。近来，这种表征也被扩展到三维研究中[62, 88, 63, 33]。与经典的显式表征（例如网格和点云）、隐式表征（例如 NeRF[51] 和 SDF[55]）以及混合表征（例如 triplane[8] 和 3DGS[40]）这些本身具备三维感知的形式不同，潜在集合表征并不显式具备三维感知。它是对场景信息的一种压缩，其中的三维感知性质是完全学出来的。潜在集合场景表征可以记为 ，其中 是集合中的 token 数， 是潜在特征维数。
Rendering latent set scene representation requires a network, say , as introduced by SRT [62] and LVSM [33]. We formulate it as , where is a ray and is the rendered property, e.g., RGB values, of the corresponding pixel[^1]. This formulation is the same as traditional Graphics rendering techniques [1, 36], as , where is a pre-defined and handcrafted rendering equation, e.g., alpha-blending ray marching in NeRF. Differently, our “rendering equation” is a learned model parameterized with weights , and our scene representation is a latent token set as discussed previously. We omit the model parametrization, e.g., weight , in the following description for clarity.
渲染潜在集合场景表征需要一个网络，记为 ，这一设定由 SRT[62] 和 LVSM[33] 引入。我们把它写成 ，其中 是一条射线， 是对应像素被渲染出的属性，例如 RGB 值[^1]。这一形式与传统图形学渲染技术[1, 36]相同，后者写作 ，其中 是预先定义、手工设计的渲染方程，例如 NeRF 中的 alpha 混合射线行进。不同之处在于，我们的“渲染方程”是一个以权重 参数化的学习模型，而我们的场景表征如前所述是一个潜在 token 集合。为表述清晰，后文将省略模型参数化，例如权重 。
4 RayZer
In this section, we first introduce RayZer’s self-supervised learning framework (Sec. 4.1). Then, we present the details of the RayZer model architecture (Sec. 4.2).
在这一节中，我们先介绍 RayZer 的自监督学习框架（第 4.1 节），再给出 RayZer 模型结构的细节（第 4.2 节）。
4.1 RayZer’s Self-supervised Learning
4.1 RayZer 的自监督学习
We first formulate the input and output of RayZer. We then introduce the self-supervised learning framework.
我们先形式化 RayZer 的输入和输出，再介绍自监督学习框架。
We focus on the standard setting of modeling static scenes [65]. The input of RayZer is a set of unposed and uncalibrated multi-view images , which can come from unlabeled video frames or image sets. The output is a parametrization of the inputs, i.e., camera intrinsics, per-view camera poses, and scene representation, enabling novel view synthesis. To predict these representations, we build the RayZer model and train it with self-supervised learning – no 3D supervision, i.e., 3D geometry, and camera pose annotations during training.
我们关注对静态场景建模的标准设定[65]。RayZer 的输入是一组未给定位姿、也未经标定的多视图图像 ，它们可以来自无标注视频帧或图像集合。输出是对输入的一种参数化，即相机内参、每张视图的相机位姿，以及场景表征，从而支持新视角合成。为了预测这些表征，我们构建 RayZer 模型，并用自监督学习来训练——训练时没有三维监督，也就是没有三维几何和相机位姿标注。
To train RayZer with self-supervision, we control the data information flow. We split the input images into two non-overlapping subsets and , where and . RayZer uses to predict the scene representation, and use for providing supervision. Thus, RayZer renders images that correspond to , denoted as , and we apply photometric losses:
为了用自监督训练 RayZer，我们控制数据的信息流动。我们把输入图像 分成两个互不重叠的子集 和 ，满足 且 。RayZer 用 预测场景表征，并用 提供监督。因此，RayZer 渲染与 相对应的图像，记为 ，并施加光度损失：
where is the size (number of images) of , is the image that corresponds to a predicted image , and is the weight for perceptual loss [35, 46]. The two sets are randomly sampled during training.
其中 是 的大小（图像数量）， 是与预测图像 相对应的图像， 是感知损失的权重[35, 46]。这两个集合在训练时随机采样。
4.2 RayZer Model
4.2 RayZer 模型
Overview. As introduced in Sec. 4.1, RayZer recovers both camera parameters and the scene representation from unposed, uncalibrated input images. A key design element of RayZer is its cascaded prediction of camera and scene representations. This is motivated by the fact that even noisy cameras can be a strong condition for better scene reconstruction [65, 31, 92], which is analogous to traditional structure-from-motion methods [65] and is in contrast with recent reconstruction-first methods [77, 74, 63]. This design can provide mutual regularization of predicting pose and scene during training, facilitating self-supervised learning.
总览。如第 4.1 节所述，RayZer 从未给定位姿、也未经标定的输入图像中，同时恢复相机参数和场景表征。RayZer 的一个关键设计，是对相机和场景表征做级联预测。这一设计的动机是：即便是带噪声的相机，也可以成为更好场景重建的强条件[65, 31, 92]，这与传统的运动恢复结构方法[65]类似，而与近来“先重建”的方法[77, 74, 63]相对。这一设计可以在训练中对位姿预测和场景预测形成相互约束，从而促进自监督学习。
RayZer builds a pure transformer-based model, benefiting from its scalability and flexibility. As shown in Fig. 3, RayZer first tokenizes input images and uses a transformer-based encoder to predict camera parameters of all views. In this step, the cameras are represented by their intrinsics and SE(3) camera poses. This low-dimensional, geometrically well-defined parametrization helps disentangle image information from the camera representation.
RayZer 构建了一个纯 Transformer 模型，以利用其可扩展性和灵活性。如图 3 所示，RayZer 先把输入图像分词，再用基于 Transformer 的编码器预测所有视图的相机参数。在这一步中，相机由其内参和 SE(3) 相机位姿表示。这种低维、几何含义明确的参数化，有助于把图像信息从相机表征中拆开。
RayZer then transforms the SE(3) camera poses and intrinsics into Plücker ray maps [57], representing the predicted cameras as pixel-aligned rays. This ray-based representation captures both the 2D ray-pixel alignment and the 3D ray geometry, providing fine-grained, ray-level details that encapsulate the physical properties of the camera model. The ray maps serve as a condition for improving the reconstruction stage that follows.
随后，RayZer 把 SE(3) 相机位姿和内参转换成 Plücker 射线图[57]，把预测出的相机表示成与像素对齐的射线。这种基于射线的表征既刻画了二维的射线—像素对齐，也刻画了三维的射线几何，能够提供细粒度的、射线级的细节，并包含相机模型的物理属性。射线图随后作为条件，用于改进接下来的重建阶段。
From the image and predicted Plücker rays of , RayZer uses another transformer-based encoder to predict the latent set scene representation (introduced in Sec. 3 and detailed later). Then, RayZer uses the previously estimated cameras of to predict , providing photometric self-supervision (Eq. 1). We now formally introduce the RayZer model.
从 的图像及其预测的 Plücker 射线出发，RayZer 再用另一个基于 Transformer 的编码器，预测潜在集合场景表征（第 3 节已介绍，后文再详述）。然后，RayZer 用先前估计得到的 相机去预测 ，从而提供光度上的自监督（公式 1）。下面我们形式化地介绍 RayZer 模型。
Image Tokenization. For all input images , we patchify them into non-overlapping patches following ViT [18]. Each patch is in , where is the patch size. We use a linear layer to encode each patch into a token in , leading to a patch-aligned token map for each image, where , , and is the latent dimension.
图像分词。对全部 张输入图像 ，我们沿用 ViT[18]，把它们切成互不重叠的图像块。每个块属于 ，其中 是块大小。我们用一层线性层把每个块编码成 中的一个 token，从而为每张图像得到与块对齐的 token 图 ，其中 ，， 是潜在维数。
We then add positional embeddings (p.e.) to the tokens, enabling the following model to be aware of the spatial location and the corresponding image index of each token. Specifically, we combine the sinusoidal spatial p.e. [18] and the sinusoidal image index p.e. [3] using a linear layer; note that the image index p.e. is shared among all tokens from the same image. When training on continuous video frames, these image index embeddings also encode sequential priors, which benefits pose estimation. Finally, we reshape the token maps of all images into a set, denoted as (recall that the transformer is invariant to the permutation of tokens). For brevity, we will use this notation for latent token sets throughout the rest of the paper.
然后，我们为这些 token 加上位置编码（p.e.），使后续模型能够感知每个 token 的空间位置及其所属图像编号。具体来说，我们用一层线性层，把正弦空间位置编码[18]和正弦图像编号位置编码[3]结合起来；注意，来自同一张图像的所有 token 共享同一组图像编号位置编码。当在连续视频帧上训练时，这些图像编号嵌入还会编码序列先验，有利于位姿估计。最后，我们把所有图像的 token 图重排成一个集合，记为 （回忆一下，Transformer 对 token 的排列是不变的）。为简明起见，后文都将用这一记号表示潜在 token 集合。
Camera Estimator. The camera estimator predicts camera parameters, i.e., camera poses and intrinsics, for all input images. We use a learnable camera token in as the initial feature for this prediction for all views. We repeat the token times and add them with image index p.e. such that they correspond to the images. We denote this camera feature initialization as . We then use the camera estimator composed of full self-attention transformer layers to update the camera tokens, as:
相机估计器。相机估计器 为所有输入图像预测相机参数，即相机位姿和内参。我们使用一个可学习的、属于 的相机 token，作为所有视图做这一预测的初始特征。我们把该 token 重复 次，并加上图像编号位置编码，使它们分别对应这 张图像。我们将这一相机特征初始化记为 。随后，我们用由全自注意力 Transformer 层组成的相机估计器来更新这些相机 token，写作：
where denotes concatenation along the token dimension (the union set of two token sets), and and are the updated tokens. We note that is not used for the following computation – it is only used as context to update in the transformer layers. For clarity, we formulate the transformer layers as follows:
其中 表示沿 token 维拼接（两个 token 集合的并）， 和 是更新后的 token。需要说明的是， 不会用于后续计算，它只是在 Transformer 层中作为更新 的上下文。为清晰起见，我们将 Transformer 层表述如下：
where is the number of layers, and the split operation recovers the two token sets, inverting Eq. 3. This notation remains consistent throughout the rest of the paper.
其中 是层数，split 操作恢复出两个 token 集合，相当于公式 3 的逆操作。后文将沿用这一记法。
We then predict the camera parameters for each image independently. For camera pose prediction, we follow prior works of using relative camera poses to resolve ambiguity [89, 31]. We select one view as the canonical reference (e.g., with identity rotation and zero translation), while for every non-canonical view, we predict its relative pose with respect to the canonical view. We parametrize the SO(3) rotation using a continuous 6D representation [97], and we predict the relative pose with a two-layer MLP as follows:
然后，我们为每张图像独立预测相机参数。在位姿预测上，我们沿用已有工作，使用相对相机位姿来消除歧义[89, 31]。我们选一张视图作为规范参考（例如旋转为单位阵、平移为零）；对每一张非规范视图，我们预测它相对于规范视图的相对位姿。我们用连续的 6 维表示[97]来参数化 SO(3) 旋转，并用一个两层 MLP 预测相对位姿，如下：
where denotes concatenation along the feature dimension, and (all in ) are the camera tokens for image and the canonical view, respectively. The output represents the predicted pose parameters, which are then transformed into an SE(3) pose for image .
其中 表示沿特征维拼接， 和 （都属于 ）分别是图像 和规范视图的相机 token。输出 表示预测的位姿参数，随后被转换成图像 的 SE(3) 位姿 。
For intrinsics prediction, following prior works [24, 41], we parameterize intrinsics using a single focal length value, under the assumptions that i) the focal lengths along the x and y axes are identical, ii) all views share the same intrinsics, and iii) the principal point is at the image center. We predict the focal length using a two-layer MLP:
在内参预测上，我们沿用已有工作[24, 41]，用单个焦距值来参数化内参，并作如下假定：i) 轴和 轴上的焦距相同；ii) 所有视图共享同一组内参；iii) 主点位于图像中心。我们用一个两层 MLP 预测焦距：
The predicted focal length is then converted into the intrinsics matrix .
预测得到的焦距再被转换成内参矩阵 。
Figure 4: Visualization results on RealEstate and DL3DV. We compare RayZer with “oracle” methods GS-LRM and LVSM, which use COLMAP pose annotations in both training and testing. Our self-supervised RayZer model does not use any pose annotations. Generally, RayZer performs on par with “oracle” methods (first row), and can outperform them on cases that COLMAP usually struggles to handle, e.g., glasses and white walls (highlighted with red boxes). The results verify our analysis on the problems of using COLMAP in Sec. 5.2.
图 4：在 RealEstate 和 DL3DV 上的可视化结果。我们将 RayZer 与“oracle”方法 GS-LRM 和 LVSM 比较，后两者在训练和测试时都使用 COLMAP 位姿标注。我们的自监督 RayZer 模型不使用任何位姿标注。总体上看，RayZer 与“oracle”方法相当（第一行），并且在 COLMAP 通常难以处理的情形上可以超过它们，例如玻璃和白墙（用红框标出）。这些结果验证了我们在第 5.2 节中关于使用 COLMAP 所存在问题的分析。
Scene Reconstructor. As discussed in Sec. 4.1, we predict the scene representation from image set and additionally condition it on the previously predicted camera parameters . We first convert to pixel-aligned Plücker rays [57] for each image, denoted as . Similar to image inputs, we also tokenize the Plücker rays into patch-level tokens using a linear layer, yielding . We index the image and Plücker rays tokens corresponding to the image set , denoted as and (each in , respectively). We fuse these tokens along the feature dimension with a two-layer MLP:
场景重建器。如第 4.1 节所述，我们从图像集合 预测场景表征，并额外以先前预测得到的相机参数 为条件。我们先把 转换成每张图像上与像素对齐的 Plücker 射线[57]，记为 。与图像输入类似，我们也用一层线性层把 Plücker 射线分词成块级 token，得到 。我们取出对应于图像集合 的图像 token 和 Plücker 射线 token，分别记为 和 （各属于 ）。我们用一个两层 MLP 沿特征维融合这些 token：
where represents the fused tokens. Importantly, we use the raw image tokens rather than the pose transformer output for this fusion. This design choice prevents leakage of information from the image set , since the camera estimator transformer producing has access to a global context that includes tokens from .
其中 表示融合后的 token。重要的是，这次融合使用的是原始图像 token ，而不是位姿 Transformer 的输出 。这一设计是为了避免来自图像集合 的信息泄漏，因为产生 的相机估计器 Transformer 能看到包含 token 的全局上下文。
We then employ a scene reconstructor consisting of full self-attention transformer layers to predict the latent scene representation. To initialize this representation, we use a set of learnable tokens , where denotes the number of tokens. We formulate the process as follows:
随后，我们使用由全自注意力 Transformer 层组成的场景重建器 来预测潜在场景表征。为了初始化这一表征，我们使用一组可学习 token ，其中 表示 token 数量。该过程表述如下：
The update rule is identical to the transformer layers in the camera estimator . Here, represents the final latent scene representation predicted from . Meanwhile, is discarded.
更新规则与相机估计器 中的 Transformer 层相同。这里， 表示从 预测得到的最终潜在场景表征。同时， 被丢弃。
Rendering Decoder. We first define the rendering decoder and then describe its training usage.
渲染解码器。我们先定义渲染解码器，再说明它在训练中的用法。
We use a transformer-based decoder with full self-attention for rendering, following LVSM [33]. For a target image, we begin by representing it as pixel-aligned Plücker rays and tokenize these rays using a linear layer to obtain target tokens . Next, we fuse the scene information by updating the tokens with a decoder comprising transformer layers:
我们沿用 LVSM[33]，使用带全自注意力的 Transformer 解码器来做渲染。对一张目标图像，我们先把它表示成与像素对齐的 Plücker 射线，再用一层线性层对这些射线分词，得到目标 token 。接着，我们用由 Transformer 层组成的解码器 更新这些 token，以融入场景信息：
where is subsequently discarded, while the update rule of is the same as previously introduced modules. Finally, we decode the RGB values at the patch level with an MLP:
其中 随后被丢弃， 的更新规则与前面介绍的模块相同。最后，我们用一个 MLP 在块级解码 RGB 值：
where . We reshape to recover the 2D spatial structure, yielding a final rendered image in .
其中 。我们将 重塑以恢复二维空间结构，得到最终渲染图像，形状为 。
During training, we use the predicted Plücker ray maps , which correspond to , to render images of and then compute the self-supervised loss as defined in Eq. 1.
训练时，我们使用预测得到的、与 相对应的 Plücker 射线图 ，来渲染 的图像，再按公式 1 计算自监督损失。
5 Experiments
5 实验
In this section, we introduce the experimental setting and present the evaluation results. For the implementation, RayZer employs 24 transformer layers, with 8 layers for each of the camera estimator, scene encoder, and rendering decoder. We train RayZer with a learning rate of with a cosine scheduler for 50,000 iterations and a batch size of 256. The weight of perceptual loss is . For all experiments, we used a resolution of 256 with a patch size of 16. More details are in the Appendix.
在这一节中，我们介绍实验设置并给出评估结果。实现上，RayZer 使用 24 层 Transformer，相机估计器、场景编码器和渲染解码器各占 8 层。我们用 的学习率和余弦调度训练 50,000 步，批大小为 256。感知损失的权重为 。所有实验都使用分辨率 256、块大小 16。更多细节见附录。
5.1 Experimental Setup
5.1 实验设置
We introduce our experimental setup, including datasets, evaluation protocol and metrics, as well as baseline methods.
我们介绍实验设置，包括数据集、评估协议与指标，以及基线方法。
Datasets. We use three datasets to evaluate RayZer, including two scene-level datasets, DL3DV [49] and RealEstate [96], and an object-level dataset Objaverse [17] (rendered as videos). We train and test on each dataset separately. The numbers of input views () and target views () are set to 16 and 8 for DL3DV, 5 and 5 for RealEstate, and 12 and 8 for Objaverse, respectively. We sample input images with the index ranges of 64-96, 128-192, and 50-65 on DL3DV, RealEstate, and Objaverse, respectively. These values are chosen based on data difficulty, especially camera baseline, following prior works [9, 91, 98]. We use the offical DL3DV train-test split, and split RealEstate following [9]. More details can be found in the Appendix.
数据集。我们用三个数据集评估 RayZer，其中包括两个场景级数据集 DL3DV[49] 和 RealEstate[96]，以及一个被渲染成视频的物体级数据集 Objaverse[17]。每个数据集上单独训练和测试。输入视图（）和目标视图（）的数量，在 DL3DV 上设为 16 和 8，在 RealEstate 上设为 5 和 5，在 Objaverse 上设为 12 和 8。我们分别在 DL3DV、RealEstate 和 Objaverse 上，按索引范围 64–96、128–192 和 50–65 采样输入图像。这些取值依据数据难度、尤其是相机基线来选定，并沿用已有工作[9, 91, 98]。我们使用官方的 DL3DV 训练—测试划分，并按[9]划分 RealEstate。更多细节见附录。
Evaluation Protocol and Metrics. We evaluate novel view synthesis quality. Specifically, the evaluation protocol of RayZer is different from the “oracle” and supervised methods, which use ground-truth poses to render images. Instead, we use predicted poses to render novel views, thereby assessing the compatibility between the predicted poses and the scene representation. Since the model is trained without explicit pose annotations, the learned poses exist in a different space, and their direct correspondence to standard pose annotations is unknown. This evaluation protocol follows RUST [63]. We note that the target views are used only for pose estimation and not for scene representation prediction, ensuring that no information leakage occurs.
评估协议与指标。我们评估新视角合成质量。具体来说，RayZer 的评估协议不同于“oracle”方法和受监督方法，后两者用真值位姿来渲染图像。我们则用预测位姿来渲染新视角，从而评估预测位姿与场景表征是否兼容。由于模型在训练时没有显式的位姿标注，学到的位姿处在另一个空间中，它们与标准位姿标注的直接对应关系是未知的。这一评估协议沿用 RUST[63]。需要说明的是，目标视图只用于位姿估计，不用于场景表征预测，从而保证不会发生信息泄漏。
Baselines. We compare RayZer with two types of methods, including 1) “oracle” methods, i.e., GS-LRM [91] and LVSM [33] (encoder-decoder version), that use ground-truth camera poses during both training (as supervision) and inference (as pre-requisite). LVSM also uses latent set scene representation. Thus, it serves as the main comparison for the “oracle” methods; 2) supervised method, i.e., PF-LRM [74], which requires camera supervision to learn pose estimation and reconstruction; thus, it is pose-free during inference. For fair comparisons, we use 16 transformer layers in total for GS-LRM and LVSM. Thus, their number of parameters is the same as RayZer, except that RayZer has another camera estimator to handle unposed images. We use 24 transformer layers for PF-LRM. We also consider the self-supervised method RUST [63], but since it does not have an official public implementation, we ablate the key design differences between RUST and RayZer in Table 7 instead.
基线。我们将 RayZer 与两类方法比较，包括：1）“oracle”方法，即 GS-LRM[91] 和 LVSM[33]（编码器—解码器版本），它们在训练（作为监督）和推理（作为前提）时都使用真值相机位姿。LVSM 同样使用潜在集合场景表征，因此是“oracle”方法中的主要对照；2）受监督方法，即 PF-LRM[74]，它需要相机监督来学习位姿估计和重建，因而在推理时是无位姿的。为公平比较，GS-LRM 和 LVSM 一共使用 16 层 Transformer。这样，它们的参数量与 RayZer 相同，只是 RayZer 另外还有一个相机估计器，用来处理未给定位姿的图像。PF-LRM 使用 24 层 Transformer。我们也考虑了自监督方法 RUST[63]，但由于它没有官方公开实现，我们改为在表 7 中消融 RUST 与 RayZer 的关键设计差异。
| Training | Inference w. | Even Sample | Random Sample | |||||
|---|---|---|---|---|---|---|---|---|
| Supervision | COLMAP Cam. | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| “Oracle” methods (assume inputs are posed & use pose annotations during training) | ||||||||
| GS-LRM | 2D + Camera | Yes | 23.49 | 0.712 | 0.252 | 23.02 | 0.705 | 0.266 |
| LVSM | 2D + Camera | Yes | 23.69 | 0.723 | 0.242 | 23.10 | 0.703 | 0.257 |
| Unsupervised methods (inputs are un-posed & no pose annotations used during training) | ||||||||
| RayZer | 2D | No | 24.36 | 0.757 | 0.209 | 23.72 | 0.733 | 0.222 |
Table 1: Evaluation results on DL3DV. The camera annotations used by the “oracle” models come from COLMAP. The results are reported with continuous video frames (ordered) as the input. The results for the unordered image set input are in Table. 4. The input and target views can be evenly or randomly sampled from video frames. We bold our result if it is better than the “oracle” models.
| 训练 | 推理时使用 | 均匀采样 | 随机采样 | |||||
|---|---|---|---|---|---|---|---|---|
| 监督信号 | COLMAP 相机 | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| “Oracle”方法（假定输入已有位姿，训练时使用位姿标注） | ||||||||
| GS-LRM | 二维 + 相机 | 是 | 23.49 | 0.712 | 0.252 | 23.02 | 0.705 | 0.266 |
| LVSM | 二维 + 相机 | 是 | 23.69 | 0.723 | 0.242 | 23.10 | 0.703 | 0.257 |
| 无监督方法（输入无位姿，训练时也不使用位姿标注） | ||||||||
| RayZer | 二维 | 否 | 24.36 | 0.757 | 0.209 | 23.72 | 0.733 | 0.222 |
表 1：在 DL3DV 上的评估结果。“oracle”模型使用的相机标注来自 COLMAP。结果以连续视频帧（有序）作为输入报告。无序图像集合输入的结果见表 4。输入视图和目标视图可以从视频帧中均匀或随机采样。若我们的结果优于“oracle”模型，则以粗体标出。
5.2 Results
5.2 结果
Main results. Table 1-3 summarizes the results on the three datasets. Remarkably, without any 3D labels (e.g., camera pose annotations) during training, RayZer achieves performance comparable to the best “oracle” model, LVSM. In fact, RayZer even outperforms LVSM on DL3DV and RealEstate10k while performing slightly worse on Objaverse. We conjecture that this is because the camera poses in DL3DV and RealEstate are annotated by COLMAP, which can be imperfect and set an upper bound for “oracle” methods that are supervised by COLMAP annotations. In contrast, our self-supervised approach enables the model to learn a pose space that optimally benefits latent reconstruction and novel view synthesis. This hypothesis is further supported by the results on Objaverse – a synthetic dataset with perfect pose annotations from the rendering tool – where LVSM, acting as a true oracle, outperforms RayZer. Nonetheless, the small performance gap showcases the effectiveness of our self-supervised training. Visualizations in Fig. 4 further support our conjecture regarding COLMAP’s noisy poses, as both LVSM and GS-LRM consistently underperform on challenging cases that COLMAP usually fails. These results not only validate our self-supervised learning approach but also demonstrate its potential to break free from the limitations of supervised learning.
主要结果。表 1–3 汇总了三个数据集上的结果。值得注意的是，训练时没有任何三维标签（例如相机位姿标注），RayZer 仍能达到最强“oracle”模型 LVSM 的相近水平。事实上，RayZer 在 DL3DV 和 RealEstate10k 上甚至超过了 LVSM，只在 Objaverse 上略差。我们猜想，这是因为 DL3DV 和 RealEstate 中的相机位姿由 COLMAP 标注，可能并不完美，从而给依赖这些标注做监督的“oracle”方法设下了上限。相比之下，我们的自监督方法使模型能够学到一套更有利于潜在重建和新视角合成的位姿空间。这一假设也得到 Objaverse 结果的支持——这是一套带有渲染工具给出的精确位姿标注的合成数据，此时真正充当 oracle 的 LVSM 超过了 RayZer。不过，性能差距很小，说明自监督训练是有效的。图 4 的可视化进一步支持了我们关于 COLMAP 位姿带噪声的判断：在 COLMAP 通常会失败的困难情形上，LVSM 和 GS-LRM 都持续表现较差。这些结果不仅验证了我们的自监督学习方法，也表明它有潜力摆脱监督学习的限制。
Figure 5: Visualization results on Objaverse. RayZer performs on par with LVSM and outperforms the supervised method PF-LRM.
图 5：在 Objaverse 上的可视化结果。RayZer 与 LVSM 相当，并超过受监督方法 PF-LRM。
| Training | Inference w. | Even Sample | Random Sample | |||||
|---|---|---|---|---|---|---|---|---|
| Supervision | COLMAP Cam. | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| “Oracle” methods (assume inputs are posed & use pose annotations during training) | ||||||||
| GS-LRM | 2D + Camera | Yes | 24.25 | 0.770 | 0.227 | 23.21 | 0.748 | 0.251 |
| LVSM | 2D + Camera | Yes | 27.00 | 0.851 | 0.157 | 25.88 | 0.828 | 0.175 |
| Unsupervised methods (inputs are un-posed & no pose annotations used during training) | ||||||||
| RayZer | 2D | No | 27.48 | 0.861 | 0.146 | 26.32 | 0.835 | 0.164 |
Table 2: Evaluation results on RealEstate with continuous video frames inputs. The camera annotations come from COLMAP.
| 训练 | 推理时使用 | 均匀采样 | 随机采样 | |||||
|---|---|---|---|---|---|---|---|---|
| 监督信号 | COLMAP 相机 | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| “Oracle”方法（假定输入已有位姿，训练时使用位姿标注） | ||||||||
| GS-LRM | 二维 + 相机 | 是 | 24.25 | 0.770 | 0.227 | 23.21 | 0.748 | 0.251 |
| LVSM | 二维 + 相机 | 是 | 27.00 | 0.851 | 0.157 | 25.88 | 0.828 | 0.175 |
| 无监督方法（输入无位姿，训练时也不使用位姿标注） | ||||||||
| RayZer | 二维 | 否 | 27.48 | 0.861 | 0.146 | 26.32 | 0.835 | 0.164 |
表 2：在 RealEstate 上以连续视频帧为输入的评估结果。相机标注来自 COLMAP。
Using unordered image sets for training. RayZer can be trained on continuous video frames (Table 1-3) or unordered image sets (Table 4). Note that these two training settings are applied separately. As shown in Table 4, we observe that the model trained with unordered image sets performs worse than the one trained with continuous video frames. We notice that the difference is at the pose estimation stage – specifically, the image index positional embedding encourages local pose smoothness that benefits the learning of pose estimation on continuous frames. This finding suggests that scaling training data using video resources, which are plentiful online, could be more advantageous than relying on unordered image sets that are often limited in scale and contain noisy content [73, 45].
用无序图像集合训练。RayZer 既可以在连续视频帧上训练（表 1–3），也可以在无序图像集合上训练（表 4）。需要注意，这两种训练设定是分开进行的。如表 4 所示，我们看到，用无序图像集合训练的模型，表现不如用连续视频帧训练的模型。我们注意到，差异出在位姿估计阶段——具体来说，图像编号位置编码会鼓励局部位姿平滑，从而有利于在连续帧上学习位姿估计。这一发现表明，用网上大量存在的视频资源来扩大训练数据，可能比依赖规模往往有限、内容也常带噪声的无序图像集合更有利[73, 45]。
| Training | Inference w. | Even Sample | Random Sample | |||||
|---|---|---|---|---|---|---|---|---|
| Supervision | GT Cam. | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| “Oracle” methods (assume inputs are posed & use pose annotations during training) | ||||||||
| LVSM | 2D + GT Cam. | Yes | 32.34 | 0.950 | 0.050 | 32.34 | 0.949 | 0.051 |
| Supervised methods (inputs are un-posed & use pose annotations during training) | ||||||||
| PF-LRM | 2D + GT Cam. | Yes (render) | 25.48 | 0.882 | 0.110 | 25.43 | 0.881 | 0.111 |
| Unsupervised methods (inputs are un-posed & no pose annotations used during training) | ||||||||
| RayZer | 2D | No | 31.52 | 0.945 | 0.052 | 31.42 | 0.943 | 0.053 |
Table 3: Evaluation results on Objaverse with continuous video frames inputs. The camera annotations are Blender ground-truth. PF-LRM uses ground-truth poses to render novel views, same with oracle methods, and we evaluate its predicted pose in Table 5.
| 训练 | 推理时使用 | 均匀采样 | 随机采样 | |||||
|---|---|---|---|---|---|---|---|---|
| 监督信号 | 真值相机 | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| “Oracle”方法（假定输入已有位姿，训练时使用位姿标注） | ||||||||
| LVSM | 二维 + 真值相机 | 是 | 32.34 | 0.950 | 0.050 | 32.34 | 0.949 | 0.051 |
| 受监督方法（输入无位姿，但训练时使用位姿标注） | ||||||||
| PF-LRM | 二维 + 真值相机 | 是（渲染） | 25.48 | 0.882 | 0.110 | 25.43 | 0.881 | 0.111 |
| 无监督方法（输入无位姿，训练时也不使用位姿标注） | ||||||||
| RayZer | 二维 | 否 | 31.52 | 0.945 | 0.052 | 31.42 | 0.943 | 0.053 |
表 3：在 Objaverse 上以连续视频帧为输入的评估结果。相机标注是 Blender 真值。PF-LRM 与 oracle 方法一样，用真值位姿渲染新视角；我们在表 5 中评估它预测的位姿。
| Training | Inf. w. | Continuous | Even Sample | Random Sample | |||||
|---|---|---|---|---|---|---|---|---|---|
| Supervision | GT Pose | Inputs | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| (1) | 2D | No | ✓ | 24.36 | 0.757 | 0.209 | 23.72 | 0.733 | 0.222 |
| (2) | 2D | No | ✗ | 20.56 | 0.576 | 0.334 | 20.02 | 0.566 | 0.356 |
Table 4: Evaluating RayZer performance when using continuous or unordered images for training on DL3DV. In evaluations, the input frames are sampled from continuous video frames. (1) keeps their temporal continuity (encoded by the image index p.e.) during training. (2) randomly shuffles the images during training.
| 训练 | 推理时使用 | 连续 | 均匀采样 | 随机采样 | |||||
|---|---|---|---|---|---|---|---|---|---|
| 监督信号 | 真值位姿 | 输入 | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| (1) | 二维 | 否 | ✓ | 24.36 | 0.757 | 0.209 | 23.72 | 0.733 | 0.222 |
| (2) | 二维 | 否 | ✗ | 20.56 | 0.576 | 0.334 | 20.02 | 0.566 | 0.356 |
表 4：在 DL3DV 上评估用连续图像或无序图像训练时的 RayZer 表现。评估时，输入帧从连续视频帧中采样。(1) 在训练时保留它们的时间连续性（由图像编号位置编码来表示）。(2) 在训练时随机打乱图像。
5.3 Analysis of Camera Poses
5.3 相机位姿分析
RayZer’s learned camera pose space. We visualize some camera poses predicted by RayZer in Fig. 6. Although RayZer predicts SE(3) camera poses, we observe that these poses do not exactly match the real-world pose space. This result indicates that the SE(3) poses, which are later converted into Plücker ray maps, offer a degree of flexibility. Since both the rendering decoder and the scene representation operate in latent space, RayZer remains robust to any warping between the learned pose space and the actual real-world poses, as long as the poses are compatible with the scene representation and the decoder.
RayZer 学到的相机位姿空间。我们在图 6 中可视化了 RayZer 预测的一些相机位姿。尽管 RayZer 预测的是 SE(3) 相机位姿，我们看到这些位姿并不与真实世界的位姿空间完全重合。这一结果表明，随后被转换成 Plücker 射线图的 SE(3) 位姿具有一定灵活性。由于渲染解码器和场景表征都在潜在空间中运作，只要位姿与场景表征、解码器相兼容，RayZer 就能对学到位姿空间与真实世界位姿之间的任何扭曲保持稳健。
3D Awareness of predicted camera poses. We further investigate whether the pose space learned by RayZer is 3D aware. To this end, we interpolate the predicted poses of input views to synthesize more novel views, where the camera pose of a novel view is interpolated from two neighboring input views. We use ground-truth camera poses to calculate the interpolation coefficients, checking whether predicted poses follow the same geometric interpolation rules. We include the details of the interpolation method in Appendix. As shown in Table 5, RayZer demonstrates significantly better performance than PF-LRM and the naive baseline of copying the nearest rendered input view. These results verify that poses predicted by RayZer are interpolatable and 3D-aware.
预测相机位姿的三维感知。我们进一步考察 RayZer 学到的位姿空间是否具备三维感知。为此，我们对输入视图的预测位姿做插值，以合成更多新视角，其中新视角的相机位姿由两个相邻输入视图插值得到。我们用真值相机位姿计算插值系数，以检查预测位姿是否遵循同样的几何插值规则。插值方法的细节见附录。如表 5 所示，RayZer 明显优于 PF-LRM，也优于直接复制最近已渲染输入视图的朴素基线。这些结果验证了：RayZer 预测的位姿是可插值的，并且具备三维感知。
Probing the learned camera pose space. To probe how much actual pose information is learned by RayZer, we follow RUST [63] to fit a lightweight 2-layer MLP head on the pose features. We freeze the camera estimator’s transformers and train the MLP under camera supervision. As shown in Table 6, our probing outperforms the supervised baseline (which has the same model architecture and uses transformers trained from scratch), indicating that RayZer’s novel view synthesis self-supervision facilitates a better latent pose space. In contrast, supervised learning struggles due to the challenges of low-dimensional pose representation [90, 97, 44, 5, 11].
探测学到的相机位姿空间。 为了探测 RayZer 到底学到了多少真正的位姿信息，我们沿用 RUST[63]，在位姿特征上拟合一个轻量的两层 MLP 头。我们冻住相机估计器中的 Transformer，并在相机监督下训练该 MLP。如表 6 所示，我们的探测优于受监督基线（该基线具有相同的模型结构，并且 Transformer 是从头训练的），表明 RayZer 的新视角合成自监督有助于得到更好的潜在位姿空间。相比之下，监督学习会因低维位姿表征带来的困难而表现不佳[90, 97, 44, 5, 11]。
5.4 Ablation Study
5.4 消融实验
We ablate the main design choices of RayZer from three aspects, including scene representation, 3D prior, and the overall model paradigm. As shown in Table 7 (1), when using the 3DGS representation rather than the latent set representation, the training does not converge. This verifies the optimization difficulty of explicit 3D representation [40, 91] and demonstrates the flexibility of the latent representation with its learned rendering decoder.
我们从场景表征、三维先验和整体模型范式三个方面，消融 RayZer 的主要设计选择。如表 7(1) 所示，当使用 3DGS 表征而不是潜在集合表征时，训练无法收敛。这验证了显式三维表征的优化困难[40, 91]，也说明了潜在表征及其学习到的渲染解码器具有灵活性。
Table 7 (2) and (3) ablate the prior of camera representation. Without Plücker ray maps, we observe a degraded performance in (2), showing the effectiveness of using Plücker ray maps to regularize the solution of structure-and-motion problem. Besides, we observe a slightly better performance of (3), which directly uses camera tokens , compared to (2). The reason is that the camera tokens can leak target image information, while SE(3) poses used in (2) serve as an information bottleneck to enforce this disentanglement. Moreover, SE(3) poses are geometrically well-defined, allowing us to interpolate them and generate novel views along the interpolated camera trajectory, while the latent camera representation is not directly interpolable.
表 7(2) 和 (3) 消融了相机表征中的先验。去掉 Plücker 射线图后，我们在 (2) 中看到性能下降，说明用 Plücker 射线图来约束结构与运动问题的解是有效的。此外，我们看到直接使用相机 token 的 (3)，表现略好于 (2)。原因是相机 token 可能泄漏目标图像信息，而 (2) 中使用的 SE(3) 位姿则充当信息瓶颈，强制完成这种拆分。而且，SE(3) 位姿在几何上含义明确，我们可以对它们插值，并沿插值后的相机轨迹生成新视角；潜在相机表征则不能直接插值。
Table 7 (4) ablates the overall paradigm. When the model first predicts the latent scene and then estimates poses, we observe a degraded performance. In detail, the pose estimator takes the scene representation and target image feature tokens as inputs. The result verifies our insight that pose estimation can be a strong condition for scene reconstruction, championing traditional pose-first methods in the context of self-supervised learning. Note that combining (3) and (4) will be a model that is similar to RUST conceptually.
表 7(4) 消融了整体范式。当模型先预测潜在场景、再估计位姿时，我们看到性能下降。具体来说，位姿估计器以场景表征和目标图像特征 token 为输入。这一结果验证了我们的判断：位姿估计可以成为场景重建的强条件，并在自监督学习的语境下支持传统的“先估计位姿”方法。需要注意，把 (3) 和 (4) 结合起来，在概念上会得到一个类似于 RUST 的模型。
Figure 6: Visualization of RayZer predicted cameras learned with self-supervision. We visualize 3 out of 5 rendered views due to space limit, where the image index is highlighted by its color.
图 6：对 RayZer 在自监督下学到的预测相机做可视化。受篇幅限制，我们可视化 5 个渲染视图中的 3 个，图像编号用颜色标出。
| Training | Inference w. | Even Sample | Random Sample | |||||
|---|---|---|---|---|---|---|---|---|
| Supervision | GT Pose | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| Supervised methods (inputs are un-posed & use pose annotations during training) | ||||||||
| PF-LRM | 2D + GT Pose | No | 20.63 | 0.819 | 0.160 | 21.27 | 0.827 | 0.154 |
| Unsupervised methods (inputs are un-posed & no pose annotations used during training) | ||||||||
| RayZer-copy | 2D | No | 19.56 | 0.812 | 0.159 | 20.17 | 0.820 | 0.150 |
| RayZer | 2D | No | 27.01 | 0.900 | 0.075 | 26.87 | 0.896 | 0.078 |
Table 5: Evaluating 3D awareness of predicted camera poses on Objaverse. Unlike Table 3, here we render novel views by interpolating predicted poses of input views, where the interpolation coefficients are calculated from GT poses. This experiment tests whether the learned SE(3) poses are geometrically well-defined and 3D-aware. We also compare against a naive baseline “RayZer-copy” that simply copies the nearest rendered input view.
| 训练 | 推理时使用 | 均匀采样 | 随机采样 | |||||
|---|---|---|---|---|---|---|---|---|
| 监督信号 | 真值位姿 | PSNR↑ | SSIM↑ | LPIPS↓ | PSNR↑ | SSIM↑ | LPIPS↓ | |
| 受监督方法（输入无位姿，但训练时使用位姿标注） | ||||||||
| PF-LRM | 二维 + 真值位姿 | 否 | 20.63 | 0.819 | 0.160 | 21.27 | 0.827 | 0.154 |
| 无监督方法（输入无位姿，训练时也不使用位姿标注） | ||||||||
| RayZer-copy | 二维 | 否 | 19.56 | 0.812 | 0.159 | 20.17 | 0.820 | 0.150 |
| RayZer | 二维 | 否 | 27.01 | 0.900 | 0.075 | 26.87 | 0.896 | 0.078 |
表 5：在 Objaverse 上评估预测相机位姿的三维感知。与表 3 不同，这里通过对输入视图预测位姿做插值来渲染新视角，插值系数由真值位姿计算得到。该实验检验学到的 SE(3) 位姿在几何上是否含义明确，以及是否具备三维感知。我们还与一个朴素基线“RayZer-copy”比较，它只是复制最近的已渲染输入视图。
| Pose Encoder | Rotation Acc. (%) | Translation Acc. (%) | |||||
|---|---|---|---|---|---|---|---|
| () | R@10*∘* | R@20*∘* | R@30*∘* | t@0.1 | t@0.2 | t@0.3 | |
| DL3DV | supervised | 39.3 | 63.0 | 77.8 | 15.7 | 33.1 | 44.4 |
| self-supervised | 47.6 | 72.5 | 84.0 | 20.8 | 44.0 | 60.5 | |
| RealEstate | supervised | 87.0 | 96.4 | 99.6 | 44.6 | 59.3 | 82.5 |
| self-supervised | 99.6 | 99.9 | 100 | 61.2 | 84.2 | 92.8 | |
| Objaverse | supervised | 19.8 | 46.7 | 66.0 | 15.1 | 37.2 | 53.8 |
| self-supervised | 33.6 | 69.2 | 86.8 | 20.1 | 52.7 | 75.5 |
Table 6: Effectiveness of self-supervised pre-training for pose estimation. We train a two-layer MLP (with supervised learning) to read out latent camera tokens predicted by the pose encoder , where the backbone is frozen. At the same time, we also compare with the baseline where both encoder and the pose prediction MLP are trained with supervised learning from scratch.
| 位姿编码器 | 旋转精度（%） | 平移精度（%） | |||||
|---|---|---|---|---|---|---|---|
| () | R@10*∘* | R@20*∘* | R@30*∘* | t@0.1 | t@0.2 | t@0.3 | |
| DL3DV | 受监督 | 39.3 | 63.0 | 77.8 | 15.7 | 33.1 | 44.4 |
| 自监督 | 47.6 | 72.5 | 84.0 | 20.8 | 44.0 | 60.5 | |
| RealEstate | 受监督 | 87.0 | 96.4 | 99.6 | 44.6 | 59.3 | 82.5 |
| 自监督 | 99.6 | 99.9 | 100 | 61.2 | 84.2 | 92.8 | |
| Objaverse | 受监督 | 19.8 | 46.7 | 66.0 | 15.1 | 37.2 | 53.8 |
| 自监督 | 33.6 | 69.2 | 86.8 | 20.1 | 52.7 | 75.5 |
表 6：自监督预训练对位姿估计的作用。我们训练一个两层 MLP（以监督学习方式）来读出由位姿编码器 预测的潜在相机 token ，其中主干网络被冻住。同时，我们也与从零开始、同时用监督学习训练编码器 和位姿预测 MLP 的基线进行比较。
| Even Sample | Random Sample | ||||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | ||
| (0) | RayZer | 24.36 | 0.757 | 0.209 | 23.72 | 0.733 | 0.222 |
| (1) | Representation - 3DGS + rasterization | – | – | failed | – | – | |
| (2) | Prior - no Plücker ray, use SE(3) pose | 22.73 | 0.687 | 0.249 | 21.88 | 0.647 | 0.274 |
| (3) | Prior - no explicit pose, use latent camera | 23.13 | 0.700 | 0.251 | 22.36 | 0.668 | 0.272 |
| (4) | Paradigm - scene first, not pose first | 13.31 | 0.338 | 0.732 | 13.12 | 0.337 | 0.729 |
Table 7: Ablation study of RayZer designs on DL3DV with continuous inputs. (1) is a variant uses the 3D Gaussian representation rather than latent scene representation with its learned rendering decoder used by RayZer; (2) does not use Plücker ray maps for conditioning latent reconstruction. Instead, it encodes the SE(3) poses and intrinsics into tokens as condition; (3) directly uses the latent camera tokens , rather than converting it to any explicit forms of cameras, to condition the latent scene reconstruction; (4) first reconstructs latent scene and then estimates pose as Plücker ray maps, contrasting our pose-first paradigm.
| 均匀采样 | 随机采样 | ||||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | ||
| (0) | RayZer | 24.36 | 0.757 | 0.209 | 23.72 | 0.733 | 0.222 |
| (1) | 表征 — 3DGS + 光栅化 | – | – | 失败 | – | – | |
| (2) | 先验 — 不用 Plücker 射线，使用 SE(3) 位姿 | 22.73 | 0.687 | 0.249 | 21.88 | 0.647 | 0.274 |
| (3) | 先验 — 不用显式位姿，使用潜在相机 | 23.13 | 0.700 | 0.251 | 22.36 | 0.668 | 0.272 |
| (4) | 范式 — 先场景，而不是先位姿 | 13.31 | 0.338 | 0.732 | 13.12 | 0.337 | 0.729 |
表 7：在以连续输入的 DL3DV 上，对 RayZer 设计的消融。(1) 是使用三维高斯表征的变体，而不是 RayZer 所用的、配有学习渲染解码器的潜在场景表征；(2) 不用 Plücker 射线图 来条件化潜在重建，而是把 SE(3) 位姿 和内参 编码成 token 作为条件；(3) 直接使用潜在相机 token 来条件化潜在场景重建，而不把它转换成任何显式的相机形式；(4) 先重建潜在场景，再把位姿估计为 Plücker 射线图，与我们“先估计位姿”的范式相对。
6 Conclusion
6 结论
We introduce RayZer, a self-supervised large multi-view 3D Vision model trained with zero 3D supervision, i.e., no 3D geometry and camera annotations. RayZer achieves comparable or even better novel view synthesis performance than prior works that use pose labels in both training and inference, verifying the feasibility of breaking free from supervised learning.
我们提出 RayZer，这是一个在零三维监督下训练的自监督大规模多视图三维视觉模型，也就是不使用三维几何和相机标注。RayZer 的新视角合成性能可以达到甚至超过那些在训练和推理时都使用位姿标签的已有工作，从而验证了摆脱监督学习是可行的。
References
参考文献
[1] Arthur Appel. Some techniques for shading machine renderings of solids. In Proceedings of the April 30–May 2, 1968, spring joint computer conference , pages 37–45, 1968.
[2] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In CVPR , pages 22861–22872, 2024.
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML , page 4, 2021.
[4] Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. In CVPR , pages 4160–4169, 2023.
[5] Romain Brégier. Deep regression on manifolds: a 3d rotation case study. In 2021 International Conference on 3D Vision (3DV) , pages 166–174. IEEE, 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS , 33:1877–1901, 2020.
[7] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR , pages 5799–5809, 2021.
[8] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR , pages 16123–16133, 2022.
[9] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In CVPR , pages 19457–19467, 2024.
[10] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV , pages 14124–14133, 2021.
[11] Jiayi Chen, Yingda Yin, Tolga Birdal, Baoquan Chen, Leonidas J Guibas, and He Wang. Projective manifold gradient layer for deep rotation regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6646–6655, 2022.
[12] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016.
[13] Yuedong Chen, Haofei Xu, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Explicit correspondence matching for generalizable neural radiance fields. arXiv preprint arXiv:2304.12294 , 2023.
[14] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In ECCV , pages 370–386. Springer, 2024.
[15] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV , pages 628–644. Springer, 2016.
[16] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 , 2023.
[17] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR , pages 13142–13153, 2023.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.
[19] Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. In CVPR , pages 4970–4980, 2023.
[20] Bardienus Duisterhof, Lojze Zust, Philippe Weinzaepfel, Vincent Leroy, Yohann Cabon, and Jerome Revaud. Mast3r-sfm: a fully-integrated solution for unconstrained structure-from-motion. arXiv preprint arXiv:2409.19152 , 2024.
[21] Yang Fu, Ishan Misra, and Xiaolong Wang. Mononerf: Learning generalizable nerfs from monocular videos without camera poses. In International Conference on Machine Learning , pages 10392–10404. PMLR, 2023.
[22] Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A Efros, and Xiaolong Wang. Colmap-free 3d gaussian splatting. In CVPR , pages 20796–20805, 2024.
[23] Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning a predictable and generative vector representation for objects. In ECCV , pages 484–499. Springer, 2016.
[24] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2485–2494, 2020.
[25] Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu. Epipolar transformers. In CVPR , pages 7779–7788, 2020.
[26] Benno Heigl, Reinhard Koch, Marc Pollefeys, Joachim Denzler, and Luc Van Gool. Plenoptic modeling and rendering from image sequences taken by a hand-held camera. In Mustererkennung 1999: 21. DAGM-Symposium Bonn, 15.–17. September 1999 , pages 94–101. Springer, 1999.
[27] Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXiv:2010.04245 , 2020.
[28] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 , 2023.
[29] Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. Leap: Liberate sparse-view 3d modeling from camera poses. arXiv preprint arXiv:2310.01410 , 2023.
[30] Hanwen Jiang, Qixing Huang, and Georgios Pavlakos. Real3d: Scaling up large reconstruction models with real-world images. arXiv preprint arXiv:2406.08479 , 2024a.
[31] Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. In 2024 International Conference on 3D Vision (3DV) , pages 31–41. IEEE, 2024b.
[32] Hanwen Jiang, Zexiang Xu, Desai Xie, Ziwen Chen, Haian Jin, Fujun Luan, Zhixin Shu, Kai Zhang, Sai Bi, Xin Sun, et al. Megasynth: Scaling up 3d scene reconstruction with synthesized data. arXiv preprint arXiv:2412.14166 , 2024c.
[33] Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, and Zexiang Xu. Lvsm: A large view synthesis model with minimal 3d inductive bias. arXiv preprint arXiv:2410.17242 , 2024a.
[34] Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey, Noah Snavely, and Aleksander Holynski. Stereo4d: Learning how things move in 3d from internet stereo videos. arXiv preprint arXiv:2412.09621 , 2024b.
[35] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV , pages 694–711. Springer, 2016.
[36] James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques , pages 143–150, 1986.
[37] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specific mesh reconstruction from image collections. In ECCV , pages 371–386, 2018.
[38] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.
[39] Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. NIPS , 30, 2017.
[40] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. , 42(4):139–1, 2023.
[41] Zihang Lai, Sifei Liu, Alexei A Efros, and Xiaolong Wang. Video autoencoder: self-supervised disentanglement of static 3d structure and motion. In ICCV , pages 9730–9740, 2021.
[42] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https://github.com/facebookresearch/xformers , 2022.
[43] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In ECCV , pages 71–91. Springer, 2024.
[44] Jake Levinson, Carlos Esteves, Kefan Chen, Noah Snavely, Angjoo Kanazawa, Afshin Rostamizadeh, and Ameesh Makadia. An analysis of svd for deep rotation estimation. Advances in Neural Information Processing Systems , 33:22554–22565, 2020.
[45] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2041–2050, 2018.
[46] Zhengqi Li, Wenqi Xian, Abe Davis, and Noah Snavely. Crowdsampling the plenoptic function. In ECCV , pages 178–196. Springer, 2020.
[47] Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey. Sdf-srn: Learning signed distance 3d object reconstruction from static images. NeurIPS , 33:11453–11464, 2020.
[48] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In ICCV , pages 5741–5751, 2021.
[49] Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In CVPR , pages 22160–22169, 2024.
[50] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations , 2018.
[51] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99–106, 2021.
[52] Siva Karthik Mustikovela, Varun Jampani, Shalini De Mello, Sifei Liu, Umar Iqbal, Carsten Rother, and Jan Kautz. Self-supervised viewpoint learning from image collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3971–3981, 2020.
[53] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In ICCV , pages 7588–7597, 2019.
[54] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751 , 2022.
[55] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR , pages 165–174, 2019.
[56] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV , pages 4195–4205, 2023.
[57] Julius Plucker. Xvii. on a new geometry of space. Philosophical Transactions of the Royal Society of London , (155):725–791, 1865.
[58] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view cnns for object classification on 3d data. In CVPR , pages 5648–5656, 2016.
[59] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In CVPR , pages 918–927, 2018.
[60] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.
[61] David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. Learning internal representations by error propagation, 1985.
[62] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In CVPR , pages 6229–6238, 2022.
[63] Mehdi SM Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučić, and Klaus Greff. Rust: Latent neural scene representations from unposed imagery. In CVPR , pages 17297–17306, 2023.
[64] Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, and Deqing Sun. Vq3d: Learning a 3d-aware generative model on imagenet. In ICCV , pages 4240–4250, 2023.
[65] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR , pages 4104–4113, 2016.
[66] Cameron Smith, David Charatan, Ayush Tewari, and Vincent Sitzmann. Flowmap: High-quality camera poses, intrinsics, and depth via gradient descent. arXiv preprint arXiv:2404.15259 , 2024.
[67] Randall C Smith and Peter Cheeseman. On the representation and estimation of spatial uncertainty. The International Journal of Robotics Research , 5(4):56–68, 1986.
[68] Noah Snavely. Lecture 15: Structure from motion. https://www.cs.cornell.edu/courses/cs5670/2017sp/lectures/lec15_sfm.pdf , 2017.
[69] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV , pages 776–794. Springer, 2020.
[70] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In CVPR , pages 4190–4200, 2023.
[71] Shubham Tulsiani, Saurabh Gupta, David F Fouhey, Alexei A Efros, and Jitendra Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In CVPR , pages 302–310, 2018.
[72] Hsiao-Yu Fish Tung, Ricson Cheng, and Katerina Fragkiadaki. Learning spatial common sense with geometry-aware recurrent networks. In CVPR , pages 2595–2603, 2019.
[73] Joseph Tung, Gene Chou, Ruojin Cai, Guandao Yang, Kai Zhang, Gordon Wetzstein, Bharath Hariharan, and Noah Snavely. Megascenes: Scene-level view synthesis at scale. In European Conference on Computer Vision , pages 197–214. Springer, 2024.
[74] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311.12024 , 2023.
[75] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR , pages 4690–4699, 2021.
[76] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. arXiv preprint arXiv:2501.12387 , 2025.
[77] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR , pages 20697–20709, 2024.
[78] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality meshes. arXiv preprint arXiv:2404.12385 , 2024.
[79] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. Advances in Neural Information Processing Systems , 35:3502–3516, 2022.
[80] Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari. Multiview compressive coding for 3d reconstruction. In CVPR , pages 9065–9075, 2023.
[81] Jianfeng Xiang, Jiaolong Yang, Binbin Huang, and Xin Tong. 3d-aware image generation using 2d diffusion models. In ICCV , pages 2383–2393, 2023.
[82] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. NIPS , 29, 2016.
[83] Jianing Yang, Alexander Sax, Kevin J Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. arXiv preprint arXiv:2501.13928 , 2025a.
[84] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR , pages 10371–10381, 2024.
[85] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. NeurIPS , 37:21875–21911, 2025b.
[86] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR , pages 4578–4587, 2021.
[87] Yanjie Ze, Nicklas Hansen, Yinbo Chen, Mohit Jain, and Xiaolong Wang. Visual reinforcement learning with self-supervised 3d representations. IEEE Robotics and Automation Letters , 8(5):2890–2897, 2023.
[88] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions On Graphics (TOG) , 42(4):1–16, 2023.
[89] Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In ECCV , pages 592–611. Springer, 2022a.
[90] Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Pose estimation via ray diffusion. arXiv preprint arXiv:2402.14817 , 2024a.
[91] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-lrm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404.19702 , 2024b.
[92] Shangzhan Zhang, Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yujun Shen, and Gordon Wetzstein. Flare: Feed-forward geometry, appearance and camera estimation from uncalibrated sparse views. arXiv preprint arXiv:2502.12138 , 2025.
[93] Tianyuan Zhang, Zhengfei Kuang, Haian Jin, Zexiang Xu, Sai Bi, Hao Tan, He Zhang, Yiwei Hu, Milos Hasan, William T. Freeman, Kai Zhang, and Fujun Luan. Relitlrm: Generative relightable radiance for large reconstruction models, 2024c.
[94] Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T Freeman. Structure and motion from casual videos. In ECCV , pages 20–37. Springer, 2022b.
[95] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR , pages 1851–1858, 2017.
[96] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817 , 2018.
[97] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5745–5753, 2019.
[98] Chen Ziwen, Hao Tan, Kai Zhang, Sai Bi, Fujun Luan, Yicong Hong, Li Fuxin, and Zexiang Xu. Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats. arXiv preprint 2410.12781 , 2024.
A Experimental Details
A 实验细节
In this section, we introduce more details of RayZer.
在这一节中，我们补充介绍 RayZer 的更多细节。
Objaverse Data Details. We render Objaverse as continuous videos for training and evaluation. The frames are rendered with corresponding cameras on a unit sphere with a constant distance to the object center. Specifically, we render about 70 frames for azimuth 0*∘* to 360*∘, where the elevation is randomly sampled between -20∘* to 60*∘* for each shape instance. We sample frames with the distance between the first frame and the last frame being 50 to 65, covering the camera azimuth rotation for about one cycle.
Objaverse 数据细节。我们将 Objaverse 渲染成连续视频，用于训练和评估。这些帧由位于单位球面上、与物体中心保持固定距离的相应相机渲染得到。具体来说，我们大约按方位角 到 渲染 70 帧，每个形状实例的仰角在 到 之间随机采样。我们采样帧时，使第一帧与最后一帧之间的间隔为 50 到 65，大约覆盖相机方位角旋转一周。
Camera Interpolation Details. For the experiment of interpolating predicted cameras, we use Spherical Linear Interpolation (Slerp) for interpolating the camera pose rotation. This is based on the fact that the camera of Objaverse is moving at a constant speed. Thus, Slerp ensures the correct rotation interpolation. We then find the location on the unit sphere that corresponds to this interpolated rotation angle. Thus, we ensure the interpolated cameras are still on the unit sphere, which matches the camera sampling rule for rendering. In conclusion, this interpolation assumes that 1) the camera is moving in a constant speed, and 2) the rule of sampling camera location is known. Thus, this interpolation is only applicable to the synthetic Objaverse data, and does not apply to DL3DV and RealEstate.
相机插值细节。在对预测相机做插值的实验中，我们用球面线性插值（Slerp）来插值相机位姿的旋转。这是因为 Objaverse 的相机以恒定速度运动，因此 Slerp 能够保证正确的旋转插值。随后，我们找到与该插值旋转角相对应的单位球面上的位置。这样可以保证插值后的相机仍在单位球面上，与渲染时的相机采样规则一致。总之，这一插值假定：1) 相机以恒定速度运动；2) 采样相机位置的规则已知。因此，这一插值只适用于合成的 Objaverse 数据，并不适用于 DL3DV 和 RealEstate。
More Training Details. For all transformer layers in RayZer, we apply QK-Norm [27] to stabilize the training. We use a latent dimension of 768 for RayZer and all baselines methods. RayZer and LVSM both use a latent set scene representation with 3072 tokens. We use mixed precision training [50] with BF16, further accelerated by FlashAttention-V2 [16] of xFormers [42] and gradient checkpointing [12].
更多训练细节。对 RayZer 中的所有 Transformer 层，我们施加 QK-Norm[27] 以稳定训练。RayZer 和所有基线方法都使用 768 维的潜在维数。RayZer 和 LVSM 都使用带有 3072 个 token 的潜在集合场景表征。我们使用 BF16 混合精度训练[50]，并进一步用 xFormers[42] 中的 FlashAttention-V2[16] 以及梯度检查点[12] 加速。
We train RayZer and all baselines with the same training protocol. We use 32 A100 GPUs with a total batch size of 256. During training, we warm up with 3000 iterations, using a linearly increased learning rate from 0 to . We apply a cosine learning rate decay, while the final learning rate is . We train all baselines with steps. We clip the gradient with norm larger than 1.0. We follow all other hyper-parameters of LVSM.
我们用同一套训练协议训练 RayZer 和所有基线。我们使用 32 张 A100 GPU，总批大小为 256。训练时先预热 3000 步，学习率从 0 线性增加到 。我们再使用余弦学习率衰减，最终学习率为 。所有基线都训练 步。我们对范数大于 1.0 的梯度做裁剪。其余超参数均沿用 LVSM。
More Model Details. Following LVSM, we do not use bias terms in linear and normalization layers. We also apply the depth-wise initialization for transformer layers.
更多模型细节。沿用 LVSM，我们在线性层和归一化层中不使用偏置项。我们还对 Transformer 层施加按深度的初始化。
Ablation details. In Table 7 (2), we use a two-layer MLP to encode the camera pose and intrinsics back to a latent pose representation in . In detail, for the predicted pose of each image (in 6D representation [97]), and the camera intrinsics (as the 4-dimensional focal length and principal points of x-axis and y-axis), we first concatenate them, getting a 10-dimensional pose representation. Then, we use the MLP to map it as a high-dimensional pose feature token. To predict the target views, we use a set of learnable patch-aligned spatial tokens shared across all target images as the initialization. Thus, the rendering decoder takes in the spatial tokens, the scene tokens, and the pose token. After using transformer for updating, we use the updated spatial tokens to regress the pixel values.
消融细节。在表 7(2) 中，我们用一个两层 MLP，把相机位姿和内参重新编码成 中的潜在位姿表征。具体来说，对每张图像预测得到的位姿（采用 6 维表示[97]）以及相机内参（作为 轴和 轴上的 4 维焦距与主点），我们先把它们拼接，得到一个 10 维的位姿表征。然后，用 MLP 把它映射成高维的位姿特征 token。为了预测目标视图，我们使用一组在所有目标图像之间共享的、可学习且与块对齐的空间 token 作为初始化。因此，渲染解码器接收空间 token、场景 token 和位姿 token。用 Transformer 更新之后，我们用更新后的空间 token 回归像素值。
B RayZer Training with Continuous Inputs
B 使用连续输入训练 RayZer
RayZer takes in multi-view image inputs, which can be sampled from either continuous video frames or an unordered image set. In this section, we present two design choices to improve self-supervised learning on video frames input.
RayZer 接收多视图图像输入，这些输入既可以从连续视频帧中采样，也可以来自无序图像集合。在这一节中，我们给出两项设计，用于改进在视频帧输入上的自监督学习。
Canonical View Selection. Prior works [74, 29] usually select the first image in an image sequence as the canonical view. In contrast, we select the frame at the middle time-step as canonical. In this context, the pose prediction initialized with a zero mean for its weights will have a small pose data variance. Otherwise, when using the first frame as canonical, the variance can be much larger. Note that this difference in pose variance can be easily handled with ground-truth camera supervision, thus, prior works choose the first image as the canonical view. However, this is more important for unsupervised methods, like RayZer.
规范视图选择。已有工作[74, 29]通常把图像序列中的第一张图选为规范视图。相比之下，我们把中间时刻的那一帧选为规范视图。在这一设定下，权重以零均值初始化的位姿预测 会面对较小的位姿数据方差。反之，若把第一帧作为规范视图，方差会大得多。需要注意，这种位姿方差上的差异，在有真值相机监督时很容易处理，因此已有工作会选择第一张图作为规范视图。但对像 RayZer 这样的无监督方法来说，这一点更为重要。
Curriculum. We gradually increase the training difficulty by sampling video frames with an increasing distance range. With proper initialization of the model for camera pose estimation, it first learns from images with small camera baselines, benefiting the training with larger camera baselines, that follows. In detail, we use a curriculum with a frame sampling range of 48-64, 96-128, and 24-32 at the beginning of training for DL3DV, RealEstate, and Objaverse, respectively. The frame sampling range is linearly increased to 64-96, 128-192, and 48-65 at the end of training for DL3DV, RealEstate, and Objaverse, respectively. The final frame sampling range is also used for the evaluation. The sampling ranges are set based on the difficulty (camera baseline) of each dataset, following prior works [9, 91, 98, 32, 74].
课程学习。我们通过逐步增大采样视频帧的间距范围，来逐渐提高训练难度。在相机位姿估计的模型得到合适初始化后，它先从相机基线较小的图像中学习，这有利于随后在更大相机基线上的训练。具体来说，训练开始时，我们在 DL3DV、RealEstate 和 Objaverse 上分别使用 48–64、96–128 和 24–32 的帧采样范围作为课程。到训练结束时，这些范围再线性增加到 64–96、128–192 和 48–65。最终的帧采样范围也用于评估。采样范围依据每个数据集的难度（相机基线）设定，并沿用已有工作[9, 91, 98, 32, 74]。
Experiments. We include ablations in Table 8, where removing any of the previously discussed techniques leads to a degraded performance. This demonstrates the effectiveness of our designs of selecting canonical view and using frame sampling curriculum during training.
实验。我们在表 8 中给出消融，去掉前面讨论的任何一项技术都会导致性能下降。这说明，我们在训练中选择规范视图以及使用帧采样课程的设计是有效的。
| Even Sample | Random Sample | ||||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | ||
| (0) | RayZer | 24.36 | 0.757 | 0.209 | 23.72 | 0.733 | 0.222 |
| (1) | first frame as canonical | 23.86 | 0.736 | 0.224 | 23.78 | 0.737 | 0.225 |
| (2) | no curriculum | 23.87 | 0.734 | 0.226 | 23.87 | 0.735 | 0.226 |
Table 8: Ablation study of RayZer techniques to train on continuous video frames. (1) is a variant choosing the first image in the sequence as the canonical view, rather than choosing the middle frame. (2) does not use the frame sampling curriculum.
| 均匀采样 | 随机采样 | ||||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | ||
| (0) | RayZer | 24.36 | 0.757 | 0.209 | 23.72 | 0.733 | 0.222 |
| (1) | 以第一帧为规范视图 | 23.86 | 0.736 | 0.224 | 23.78 | 0.737 | 0.225 |
| (2) | 不使用课程 | 23.87 | 0.734 | 0.226 | 23.87 | 0.735 | 0.226 |
表 8：对 RayZer 在连续视频帧上训练所用技术的消融。(1) 是把序列中的第一张图选为规范视图的变体，而不是选择中间帧。(2) 不使用帧采样课程。
Figure 7: Visualization of RayZer failure cases on DL3DV.
图 7：RayZer 在 DL3DV 上失败情形的可视化。
C More Results
C 更多结果
In this section, we present more results for discussing RayZer’s failure cases and show more visualizations.
在这一节中，我们给出更多结果，讨论 RayZer 的失败情形，并展示更多可视化。
Failure Case Pattern. We observe that RayZer can fail when dealing with fine-grained geometry, complicated materials, and occlusions. We present the visualization in Fig. 7. In detail, RayZer fails to handle complicated plant geometry (first row). This failure is not specific to RayZer – GS-LRM and LVSM also can not handle it. In the second and last row, RayZer fails to handle multiple stacked glasses and is not perfect on the specular reflection of the silver teapots. GS-LRM and LVSM also demonstrate imperfect results. In the third and fourth rows, all methods, including RayZer, fail to handle occlusions, where the side view of the exhibition stand is not observed in input views (third row), and the chairs in the fourth row have self-occlusion.
失败情形的模式。我们观察到，当面对细粒度几何、复杂材质和遮挡时，RayZer 可能失败。可视化见图 7。具体来说，RayZer 无法处理好复杂的植物几何（第一行）。这一失败并非 RayZer 所独有——GS-LRM 和 LVSM 也同样处理不好。在第二行和最后一行，RayZer 无法处理好多层堆叠的玻璃，在银色茶壶的镜面反射上也不完美。GS-LRM 和 LVSM 同样给出不完美的结果。在第三行和第四行，包括 RayZer 在内的所有方法都无法处理好遮挡：展台的侧面在输入视图中没有被观察到（第三行），第四行中的椅子则存在自遮挡。
More Comparisons. We present more visualization results, comparing with GS-LRM and LVSM in Fig. 8. RayZer generally performs on par, while being a self-supervised method that does not require any camera pose annotations.
更多比较。我们在图 8 中给出更多与 GS-LRM 和 LVSM 比较的可视化结果。总体上看，RayZer 与它们相当，同时作为自监督方法，并不需要任何相机位姿标注。
More Visualization. We present more visualization results comparing with ground-truth novel views in Fig. 9-11.
更多可视化。我们在图 9–11 中给出更多与真值新视角进行比较的可视化结果。
D More Discussion
D 进一步讨论
Why does RayZer demonstrates strong novel view synthesis quality while the fine-tuned pose estimation is not perfect (Table 7 in the main manuscript)? We conjecture RayZer’s pose space jointly learns the actual pose information and 3D-aware video frame interpolation at the same time. On datasets with small camera baselines (RealEstate), which is easy to learn, RayZer mainly focuses on learning actual pose estimation. This is supported by the accurate pose estimation performance on RealEstate. On datasets that have large camera baselines (DL3DV and Objaverse), where pose estimation is harder to learn with only self-supervision, RayZer also leverages video interpolation cues together with pose estimation to perform novel view synthesis.
为什么 RayZer 能表现出很强的新视角合成质量，而微调后的位姿估计却并不完美（主文表 7）？我们猜想，RayZer 的位姿空间同时学习了真正的位姿信息，以及具备三维感知的视频帧插值。在相机基线较小、较容易学习的数据集（RealEstate）上，RayZer 主要专注于学习真正的位姿估计。RealEstate 上准确的位姿估计表现支持这一点。在相机基线较大、仅靠自监督更难学习位姿估计的数据集（DL3DV 和 Objaverse）上，RayZer 还会把视频插值线索与位姿估计结合起来，以完成新视角合成。
Thus, the method to further enhance disentanglement of interpolation and pose estimation would be an important future direction. In RayZer, using unordered image sets for training and using continuous video frames for training can be two extreme cases in the spectrum for learning this disentanglement. In detail, learning on continuous video frames with using image index positional embeddings strongly encourages the camera pose local smoothness to enhance training performance; while training on unordered image sets fully discards this prior. Finding a balance between the two and designing a better method to encourage the camera pose local smoothness is a promising avenue to solve the structure-and-motion problem with learning SE(3) camera poses in the real-world space.
因此，进一步增强插值与位姿估计之间的拆分，将是一个重要的未来方向。在 RayZer 中，用无序图像集合训练和用连续视频帧训练，可以看作学习这种拆分的两个极端。具体来说，在连续视频帧上学习并使用图像编号位置编码，会强烈鼓励相机位姿的局部平滑，从而提升训练表现；而在无序图像集合上训练则完全丢掉这一先验。在两者之间找到平衡，并设计更好的方法来鼓励相机位姿的局部平滑，是用在真实世界空间中学习 SE(3) 相机位姿来解决结构与运动问题的一条有前景的路径。
Figure 8: Visual compression of RayZer and “oracle” methods on DL3DV.
图 8：RayZer 与“oracle”方法在 DL3DV 上的视觉压缩对比。
Figure 9: Visual compression with ground-truth novel views on DL3DV. The first row of each sample is the target novel views, and the second row are images rendered by RayZer.
图 9：在 DL3DV 上，与真值新视角的视觉压缩对比。每个样本的第一行是目标新视角，第二行是 RayZer 渲染出的图像。
Figure 10: Visual compression with ground-truth novel views on RealEstate. The first row of each sample is the target novel views, and the second row are images rendered by RayZer.
图 10：在 RealEstate 上，与真值新视角的视觉压缩对比。每个样本的第一行是目标新视角，第二行是 RayZer 渲染出的图像。
Figure 11: Visual compression with ground-truth novel views on Objaverse. The first row of each sample is the target novel views, and the second row are images rendered by RayZer.
图 11：在 Objaverse 上，与真值新视角的视觉压缩对比。每个样本的第一行是目标新视角，第二行是 RayZer 渲染出的图像。