Neural Inverse Rendering from Propagating Light

从传播光做神经逆渲染

Anagh Malik Benjamin Attal Andrew Xie Matthew O’Toole David B. Lindell Affiliation:  University of Toronto Vector Institute Carnegie Mellon University    * joint first authors equal contribution https://anaghmalik.com/InvProp

Abstract

摘要

We present the first system for physically based, neural inverse rendering from multi-viewpoint videos of propagating light. Our approach relies on a time-resolved extension of neural radiance caching — a technique that accelerates inverse rendering by storing infinite-bounce radiance arriving at any point from any direction. The resulting model accurately accounts for direct and indirect light transport effects and, when applied to captured measurements from a flash lidar system, enables state-of-the-art 3D reconstruction in the presence of strong indirect light. Further, we demonstrate view synthesis of propagating light, automatic decomposition of captured measurements into direct and indirect components, as well as novel capabilities such as multi-view time-resolved relighting of captured scenes.

我们提出首个面向传播光多视角视频、基于物理的神经逆渲染系统。方法依赖神经辐射缓存的时间分辨扩展——通过存储任意点、任意方向到达的无限次反弹辐射来加速逆渲染。所得模型能准确计入直接与间接光传输效应；用于闪光激光雷达实采测量时，能在强间接光下实现当时最优的三维重建。我们还展示传播光新视角合成、将测量自动分解为直接/间接分量，以及多视角时间分辨重光照等新能力。

Figure 1: We introduce a method to model and invert multi-view, time-resolved measurements of propagating light from a flash lidar system. (row 1) Our method accurately recovers the geometry of this scene and enables rendering of time-resolved lidar measurements that reveal light propagation from novel views. (row 2) Physically-based modeling enables novel applications, such as time-resolved relighting and automatic decomposition of light transport into direct and indirect components.

图 1：我们提出一种方法，对闪光激光雷达系统采集的传播光多视角、时间分辨测量建模并求逆。（第 1 行）方法准确恢复该场景几何，并能渲染揭示新视角光传播的时间分辨激光雷达测量。（第 2 行）基于物理的建模带来新应用，如时间分辨重光照，以及将光传输自动分解为直接与间接分量。

1 Introduction

1 引言

Ultrafast imaging systems such as lidar illuminate a scene with a pulse of light and capture the backscattered “echoes” from the propagating wavefront [75]. Precisely measuring the speed-of-light time delay of backscattered light enables 3D reconstruction, and so lidar systems based on this principle are popular in applications from autonomous driving [36] to augmented reality [2] and remote sensing [13].

激光雷达等超快成像系统用一束光脉冲照亮场景，并捕获传播波前的后向散射「回波」 [75]。精确测量后向散射光的光速时间延迟可做三维重建，因此基于该原理的激光雷达广泛用于自动驾驶 [36]、增强现实 [2] 与遥感 [13] 等。

Lidar relies on time-resolved measurements of direct light transport, or light that reflects directly from a surface back to the sensor. Measurements of indirect light transport (i.e., light that scatters multiple times before reaching the sensor) are typically ignored or discarded because modeling indirect light requires computationally expensive (and often intractable) inverse rendering using procedures such as recursive path tracing [65, 25]. Still, measurements of indirect light are a rich source of information about material properties, appearance, and geometry [56, 88, 99, 62, 32]. Our work seeks to model and invert multi-viewpoint, time-resolved measurements of propagating light from a lidar system to recover scene geometry and to render videos of propagating light from novel viewpoints and under novel illumination conditions (see Figure 1).

激光雷达依赖对直接光传输的时间分辨测量，即从表面直接反射回传感器的光。间接光传输（到达传感器前多次散射的光）的测量通常被忽略或丢弃，因为建模间接光需要计算昂贵（往往难以处理）的逆渲染，例如递归路径追踪 [65, 25]。然而间接光测量是材质、外观与几何的丰富信息源 [56, 88, 99, 62, 32]。本文希望对激光雷达的多视角、时间分辨传播光测量建模并求逆，以恢复场景几何，并在新视角与新光照条件下渲染传播光视频（见图 1）。

Conventional lidar systems pre-process captured time-resolved measurements into a 3D point cloud, which represents an estimate of scene geometry based on direct light transport. While recent work leverages lidar measurements captured from multiple viewpoints to perform 3D reconstruction and novel view synthesis, existing methods use a point cloud representation [71, 12, 96, 21] or direct-only time-resolved measurements [47, 45, 67, 54], and thus ignore indirect light. Other methods for multi-viewpoint reconstruction with active imaging use time-of-flight sensors [3, 59], or structured light [77, 23], but similarly fail to explicitly model indirect light. Our work is close to that of Malik et al. [48], which uses lidar measurements and a representation based on neural radiance fields (NeRFs) [51] to render videos of propagating light from novel viewpoints. However, while their representation is effective for view synthesis, it does not use a physically-based model, and so cannot reconstruct accurate geometry or render the scene under novel illumination conditions.

传统激光雷达把时间分辨测量预处理成三维点云，表示基于直接光传输的几何估计。虽有工作利用多视角激光雷达做三维重建与新视角合成，但现有方法使用点云表示 [71, 12, 96, 21] 或仅直接分量的时间分辨测量 [47, 45, 67, 54]，因而忽略间接光。其他主动成像多视角重建用飞行时间传感器 [3, 59] 或结构光 [77, 23]，同样未显式建模间接光。与我们最接近的是 Malik et al. [48]：用激光雷达测量与基于神经辐射场（NeRF） [51] 的表示，从新视角渲染传播光视频。但其表示虽利于视角合成，并非基于物理模型，因而无法重建准确几何，也不能在新光照下渲染场景。

Accurately modeling indirect light transport requires solving the rendering equation [29]. Many renderers use path tracing algorithms based on Monte Carlo sampling, and differentiable versions of these methods have shown promise for inverse rendering of conventional (i.e., steady state) images [35, 44, 58] and time-resolved measurements [91, 94], such as those from a lidar system. However, while these methods work in constrained problem settings, such as non-line-of-sight imaging [82, 24], they are difficult to deploy on captured multi-viewpoint experiments due to their computational expense and sensitivity to noise and local minima. To address this issue, recent methods approximate the rendering equation in a hybrid fashion using a volumetric representation of geometry and physics-based appearance models [98, 79]. This type of approach has proven especially effective for inverse rendering of appearance, geometry, and material parameters from conventional intensity images [99, 28, 42, 4]. Still, no previous technique performs inverse rendering of time-resolved direct and indirect light transport in the same fashion.

准确建模间接光传输需要求解渲染方程 [29]。许多渲染器使用基于蒙特卡洛采样的路径追踪；其可微版本在常规（稳态）图像 [35, 44, 58] 与激光雷达一类时间分辨测量 [91, 94] 的逆渲染上已有前景。但这些方法多在受限设定（如非视距成像 [82, 24]）有效，因计算代价高、对噪声与局部极小敏感，难以部署到多视角实采实验。为此，近期方法用几何的体表示与基于物理的外观模型混合近似渲染方程 [98, 79]。这类做法在从常规强度图像逆渲染外观、几何与材质参数上尤其有效 [99, 28, 42, 4]。仍尚无技术以同样方式对时间分辨的直接与间接光传输做逆渲染。

Here, we propose a method for inverse rendering from multi-viewpoint, time-resolved measurements of propagating light from a flash lidar system — a type of lidar that flood-illuminates the entire scene with a pulse of light. Our approach is based on a hybrid neural representation, where we model geometry using volume rendering, and appearance using a physically-based model that simulates global illumination effects using a radiance cache [87]. Specifically, instead of integrating light paths using path tracing, our radiance cache stores a representation of time-resolved radiance arriving at any point in a volume from any direction. The representation is optimized in an amortized fashion, thereby removing the need to evaluate the rendering integral recursively; instead, we need only render direct lighting from the lidar and query the radiance cache to evaluate indirect light at any surface point. Our model builds on previous neural rendering frameworks for multi-viewpoint, time-resolved rendering of the direct [47] and indirect components [48] of propagating light, and grounds them in a framework for physically-based rendering. Overall, we make the following contributions.

本文提出从闪光激光雷达（用一束脉冲洪泛照亮整个场景的激光雷达）的多视角、时间分辨传播光测量做逆渲染的方法。我们采用混合神经表示：用体渲染建模几何，用基于物理、经辐射缓存 [87] 模拟全局光照的外观模型。具体地，不以路径追踪积分光路，而是用辐射缓存存储体中任意点、任意方向到达的时间分辨辐射。该表示以摊销方式优化，从而无需递归求渲染积分；只需渲染来自激光雷达的直接光照，并查询辐射缓存以评估任意表面点的间接光。模型建立在先前面向传播光直接 [47] 与间接 [48] 分量多视角时间分辨渲染的神经框架之上，并将它们纳入基于物理渲染的框架。贡献如下。

• • We propose a method for neural inverse rendering of propagating light using a physically-based model with a time-resolved radiance cache.

• • 我们提出用带时间分辨辐射缓存的基于物理模型，对传播光做神经逆渲染的方法。

• • We capture a new dataset of multi-viewpoint, time-resolved flash lidar measurements with calibrated light source and camera positions.

• • 我们采集带标定光源与相机位姿的多视角、时间分辨闪光激光雷达新数据集。

• • We demonstrate our method in simulation and on captured data; we show state-of-the-art results in geometry reconstruction under strong indirect light transport, and we render videos of propagating light from novel viewpoints and under novel illumination conditions in scenes with varying reflectance properties and significant indirect light transport effects.

• • 我们在仿真与实采数据上验证方法；在强间接光下几何重建达到当时最优，并在反射特性各异、间接光显著的场景中，从新视角与新光照渲染传播光视频。

2 Related Work

2 相关工作

Our approach brings together the areas of time-resolved imaging and rendering, as well as inverse rendering.

我们的方法汇合时间分辨成像与渲染，以及逆渲染等领域。

Time-resolved imaging.

Time-of-flight systems such as lidar measure the time of flight by marking the arrival time of a backscattered pulse of light [33]. These systems usually combine nanosecond or picosecond pulsed lasers with fast photodiodes [22] or single-photon avalanche diodes (SPADs) [70, 31, 78] to measure ultrafast variations in incident light. The resulting time-resolved measurements capture direct and indirect light transport, and can be used to record videos of propagating light at ultrafast timescales [84, 60, 16, 38]. While continuous-wave time-of-flight systems [34, 18] can also be used to capture light propagation, their temporal resolution is more limited (e.g., nanoseconds rather than picoseconds) [20, 60].

时间分辨成像。

激光雷达等飞行时间系统通过标记后向散射光脉冲的到达时间测量飞行时间 [33]。通常将纳秒或皮秒脉冲激光与快速光电二极管 [22] 或单光子雪崩二极管（SPAD） [70, 31, 78] 结合，测量入射光的超快变化。所得时间分辨测量同时捕获直接与间接光传输，可用于以超快时间尺度记录传播光视频 [84, 60, 16, 38]。连续波飞行时间系统 [34, 18] 也可捕获光传播，但时间分辨率更有限（如纳秒级而非皮秒级） [20, 60]。

Our approach uses a lidar system with a picosecond pulsed laser and a SPAD to capture multi-view videos of propagating light. Similar to previous work [10, 62, 69, 92, 15, 47, 48], we repeatedly illuminate the scene with pulses of light from the laser. The SPAD detects the arrival times of individual photons with picosecond-level accuracy and outputs a photon count histogram that approximates the time-resolved waveform of incident light [61]. Our approach is the first to capture a multi-view dataset of photon count histograms where both the flash lidar light source and sensor vary in position. Moreover, we develop the first technique for physically-based time-resolved inverse rendering using multi-viewpoint videos of propagating light.

我们使用带皮秒脉冲激光与 SPAD 的激光雷达采集传播光多视角视频。与先前工作 [10, 62, 69, 92, 15, 47, 48] 类似，用激光脉冲反复照亮场景。SPAD 以皮秒级精度检测单光子到达时间，输出近似入射光时间分辨波形的光子计数直方图 [61]。我们首次采集闪光激光雷达光源与传感器位姿均变化的多视角光子计数直方图数据集，并首次提出基于物理的、利用传播光多视角视频的时间分辨逆渲染技术。

Time-resolved rendering.

Time-resolved path-tracing renderers simulate wavefronts of propagating light [27] and account for effects such as birefringence [58], refraction [64], and volumetric scattering [26, 63]. Recently, differentiable versions of these time-resolved renderers [94, 91] have been developed; however, robust analysis-by-synthesis scene reconstruction using these methods is an open problem due to computational complexity and sensitivity to initialization and noise. Other renderers approximate the time-resolved light transport matrix [60, 68] for specific imaging problems such as non-line-of-sight imaging. In this area, techniques have been developed to model two-bounce [81, 32], three-bounce [41, 39, 92, 69, 76], or higher-order scattering events [37, 73] to recover occluded geometry. In the non-line-of-sight setting, renderers usually assume that light reflects off of planar surfaces with known (usually diffuse) reflectance properties, or else model specific scattering paths rather than arbitrary light transport [15]. Although several works consider indirect light transport in the context of continuous-wave time-of-flight sensors, they usually treat it as a residual to be removed when estimating depth [18, 57, 1]. We perform physically-based modeling of multiply scattered light without restrictive assumptions on scene geometry or material properties and integrate our approach into an inverse rendering framework for scene reconstruction from captured measurements.

时间分辨渲染。

时间分辨路径追踪渲染器模拟传播光波前 [27]，并计入双折射 [58]、折射 [64] 与体散射 [26, 63] 等效应。近来已有可微时间分辨渲染器 [94, 91]；但因计算复杂且对初始化与噪声敏感，用其做稳健的分析—合成场景重建仍是开放问题。其他渲染器为非视距成像等特定问题近似时间分辨光传输矩阵 [60, 68]。该领域已有建模两跳 [81, 32]、三跳 [41, 39, 92, 69, 76] 或更高阶散射 [37, 73] 以恢复被遮挡几何的技术。非视距设定中，渲染器通常假定光从具有已知（多为漫反射）反射特性的平面反射，或只建模特定散射路径而非任意光传输 [15]。若干工作在连续波飞行时间传感器语境下考虑间接光，但通常将其当作估计深度时要去除的残差 [18, 57, 1]。我们在不对场景几何或材质做苛刻假定的前提下，对多次散射光做基于物理的建模，并纳入从实采测量重建场景的逆渲染框架。

Physically-based inverse rendering.

Inverse rendering aims to recover scene attributes, like materials, lighting, and geometry, from a set of images [66, 95, 11]. Existing techniques use a physically-based model of light transport [65, 83, 29] and differentiable rendering with gradient-based optimization to decompose the scene’s appearance into its constituent attributes [10, 35, 25]. Recent techniques for physically-based rendering using NeRFs [51] have made inverse rendering considerably more robust, but either only consider direct illumination [79, 98, 86, 7, 8], or require explicitly simulating multiple light bounces to model indirect light [46], which is computationally expensive. Another approach is to use radiance caches — data structures that store the hemisphere of incoming radiance at every point [87]. This hemisphere is then integrated against the local bidirectional reflectance distribution function (BRDF) [65] to yield outgoing radiance, which is optimized to match the observed image pixels. Combining the radiance cache with NeRFs leads to more efficient modeling of indirect light [93, 42, 99, 4, 89].

基于物理的逆渲染。

逆渲染旨在从一组图像恢复材质、光照与几何等场景属性 [66, 95, 11]。现有技术使用基于物理的光传输模型 [65, 83, 29] 与可微渲染、基于梯度的优化，将外观分解为组成属性 [10, 35, 25]。近期基于 NeRF [51] 的基于物理渲染使逆渲染更稳健，但要么只考虑直接光照 [79, 98, 86, 7, 8]，要么需显式模拟多次反弹以建模间接光 [46]，计算昂贵。另一路线是辐射缓存——在每点存储入射辐射半球的数据结构 [87]。将该半球与局部双向反射分布函数（BRDF） [65] 积分得到出射辐射，并优化以匹配观测像素。辐射缓存与 NeRF 结合能更高效建模间接光 [93, 42, 99, 4, 89]。

Finally, we note that the concurrent work of Wu et al. [90] makes use of radiance-caching-based inverse rendering with a collocated point light source and color camera. However, no previous technique performs physically-based inverse rendering from multi-viewpoint time-resolved measurements. To this end, we develop a new, time-resolved radiance cache, enabling neural inverse rendering from videos of propagating light.

最后，同期工作 Wu et al. [90] 在共位点光源与彩色相机上使用基于辐射缓存的逆渲染。但尚无技术从多视角时间分辨测量做基于物理的逆渲染。为此我们提出新的时间分辨辐射缓存，实现从传播光视频的神经逆渲染。

Figure 2: Method overview. (a) Our time-resolved renderer combines physically-based rendering for primary rays (left inset), and neural rendering for an indirect radiance cache along secondary rays (right inset). (b) The incident radiance $L_{\mathrm{i}}$ at a sensor pixel is a function of the outgoing radiance $L_{\mathrm{o}}$ from each point $\mathbf{x}\mathopen{}\left(t\mathclose{}\right)$ along a sensor ray, which integrates incident direct light $L_{\text{i}}^{\textit{dir}}$ and indirect light $L_{\mathrm{i}}^{\textit{cache}}$ from a pulsed laser source. (b, left) The rendering equation computes the outgoing radiance as an integral over the positive hemisphere $\Omega$ (with respect to the normal $\mathbf{n}$) of the incident radiance. (b, right) Applying volume rendering to the outgoing radiance yields the incident radiance at a sensor pixel. (c) The radiance cache is used to evaluate indirect radiance $L_{\mathrm{i}}^{\textit{cache}}$ by casting secondary rays and querying neural networks at points $\mathbf{x}^{\prime}(t)$. (c, left) The networks output the reflectance $f^{\textit{dir}}$ of direct light $L_{\text{i}}^{\textit{dir}}$, indirect reflectance $f_{\Omega}^{\textit{indir}}$, and indirect incident radiance $L_{\text{i},\Omega}^{\textit{indir}}$. These quantities are used to calculate the time-resolved direct ($L_{\mathrm{o}}^{\textit{cache,dir}}$) and indirect ($L_{\mathrm{o}}^{\textit{cache,indir}}$) outgoing radiance in the direction $\boldsymbol{\omega}_{\mathrm{o}}^{\prime}$. (c, right) Volume rendering the outgoing radiance along a secondary ray yields $L_{\mathrm{i}}^{\textit{cache}}(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}},\tau)$. We optimize the scene appearance and geometry to enforce consistency between rendered and captured measurements.

图 2：方法概览。（a）我们的时间分辨渲染器将主光线的基于物理渲染（左插图）与次级光线沿间接辐射缓存的神经渲染（右插图）结合。（b）传感器像素处入射辐射 $L_{\mathrm{i}}$ 是沿传感器光线各点 $\mathbf{x}\mathopen{}\left(t\mathclose{}
ight)$出射辐射$L_{\mathrm{o}}$的函数，积分来自脉冲激光的入射直接光$L_{ ext{i}}^{ extit{dir}}$与间接光$L_{\mathrm{i}}^{ extit{cache}}$。（**b**，*左*）渲染方程将出射辐射算为正半球$\Omega$（相对法向$\mathbf{n}$）上入射辐射的积分。（**b**，*右*）对出射辐射做体渲染得到传感器像素入射辐射。（**c**）辐射缓存通过投射次级光线并在点$\mathbf{x}^{\prime}(t)$查询神经网络，评估间接辐射$L_{\mathrm{i}}^{ extit{cache}}$。（**c**，*左*）网络输出直接光反射率$f^{ extit{dir}}$、间接反射率$f_{\Omega}^{ extit{indir}}$与间接入射辐射$L_{ ext{i},\Omega}^{ extit{indir}}$，用于计算沿$oldsymbol{\omega}{\mathrm{o}}^{\prime}$的时间分辨直接（$L{\mathrm{o}}^{ extit{cache,dir}}$）与间接（$L_{\mathrm{o}}^{ extit{cache,indir}}$）出射辐射。（**c**，*右*）沿次级光线体渲染出射辐射得到$L_{\mathrm{i}}^{ extit{cache}}(\mathbf{x},oldsymbol{\omega}_{\mathrm{i}}, au)$。我们优化场景外观与几何，使渲染与实采测量一致。

3 Background: Radiance Caching with NeRFs

3 背景：带 NeRF 的辐射缓存

The rendering equation [29], models the outgoing radiance of light in direction $\boldsymbol{\omega}_{\mathrm{o}}$ at a point $\mathbf{x}$ along a ray $\mathbf{x}\mathopen{}\left(t\mathclose{}\right)=\mathbf{o}-t\boldsymbol{\omega}_{\mathrm{o}}$ with origin $\mathbf{o}$ and ray parameter $t$:

渲染方程 [29] 建模沿光线 $\mathbf{x}\mathopen{}\left(t\mathclose{}
ight)=\mathbf{o}-toldsymbol{\omega}{\mathrm{o}}$（原点$\mathbf{o}$、参数$t$）上点$\mathbf{x}$、方向$oldsymbol{\omega}{\mathrm{o}}$ 的出射辐射：

\displaystyle\resizebox{29143103}{}{$L_{\mathrm{o}}(\mathbf{x}\mathopen{}\left(t\mathclose{}\right),\boldsymbol{\omega}_{\mathrm{o}})=\mathop{\text{\Large$\int_{\text{\normalsize$\scriptstyle\mathrm{\Omega}$}}$}}\nolimits f\mathopen{}\left(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}},\boldsymbol{\omega}_{\mathrm{o}}\mathclose{}\right)L_{\mathrm{i}}\mathopen{}\left(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}}\mathclose{}\right)\mathopen{}\left(\mathbf{n}\cdot\boldsymbol{\omega}_{\mathrm{i}}\mathclose{}\right)\,d\boldsymbol{\omega}_{\mathrm{i}}\,$}. \tag{1}

The equation integrates the incident radiance $L_{\mathrm{i}}$ arriving to $\mathbf{x}$ from direction $\boldsymbol{\omega}_{\mathrm{i}}$ weighted by the BRDF $f$. The integral is over the positive hemisphere with respect to the normal $\mathbf{n}$: $\mathrm{\Omega}=\{\boldsymbol{\omega}_{\mathrm{i}}:\mathbf{n}\cdot\boldsymbol{\omega}_{\mathrm{i}}>0\}$. Naive evaluation of the rendering equation leads to an exponential increase in computation since the equation must be evaluated recursively to compute the incident radiance.

该式对从方向 oldsymbol{\omega}_{\mathrm{i}} 到达 $\mathbf{x}$ 的入射辐射 $L_{\mathrm{i}}$ 按 BRDF $f$ 加权积分。积分域为相对法向 $\mathbf{n}$ 的正半球：$\mathrm{\Omega}=\{\boldsymbol{\omega}_{\mathrm{i}}:\mathbf{n}\cdot\boldsymbol{\omega}_{\mathrm{i}}>0\}$。朴素求渲染方程会导致计算指数增长，因为求入射辐射需递归求方程。

To avoid this computational penalty, we leverage radiance caching, which removes the problematic recursion by replacing incident radiance $L_{\mathrm{i}}$ in the rendering equation with a look-up into a cache $L_{\mathrm{i}}^{\textit{cache}}$:

为避免该代价，我们采用辐射缓存：用缓存查找 $L_{\mathrm{i}}^{\textit{cache}}$ 替换渲染方程中的入射辐射 $L_{\mathrm{i}}$，从而去掉有问题的递归：

\displaystyle\resizebox{29143103}{}{$L_{\mathrm{o}}(\mathbf{x}\mathopen{}\left(t\mathclose{}\right),\boldsymbol{\omega}_{\mathrm{o}})=\mathop{\text{\Large$\int_{\text{\normalsize$\scriptstyle\mathrm{\Omega}$}}$}}\nolimits f\mathopen{}\left(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}},\boldsymbol{\omega}_{\mathrm{o}}\mathclose{}\right)L_{\mathrm{i}}^{\textit{cache}}\mathopen{}\left(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}}\mathclose{}\right)\mathopen{}\left(\mathbf{n}\cdot\boldsymbol{\omega}_{\mathrm{i}}\mathclose{}\right)\,d\boldsymbol{\omega}_{\mathrm{i}}\,$}. \tag{2}

The integral can be efficiently approximated, e.g., by sampling the cache and the BRDF using multiple importance sampling [65].

该积分可高效近似，例如用多重重要性采样对缓存与 BRDF 采样 [65]。

Recent work demonstrates that NeRFs provide accurate modeling of the radiance cache [28, 40, 4]. Specifically, we can compute $L_{\mathrm{i}}^{\textit{cache}}\mathopen{}\left(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}}\mathclose{}\right)$ by volume rendering the NeRF along a secondary ray $\mathbf{x}^{\prime}(t)=\mathbf{o}^{\prime}-t\boldsymbol{\omega}_{\mathrm{o}}^{\prime}$, where $\mathbf{o}^{\prime}=\mathbf{x}$ and $\boldsymbol{\omega}_{\mathrm{o}}^{\prime}=\boldsymbol{\omega}_{\mathrm{i}}$ [14, 43, 51]:

近期工作表明 NeRF 能准确建模辐射缓存 [28, 40, 4]。具体地，可沿次级光线 $\mathbf{x}^{\prime}(t)=\mathbf{o}^{\prime}-t\boldsymbol{\omega}_{\mathrm{o}}^{\prime}$（其中 $\mathbf{o}^{\prime}=\mathbf{x}$，$\boldsymbol{\omega}_{\mathrm{o}}^{\prime}=\boldsymbol{\omega}_{\mathrm{i}}$）对 NeRF 做体渲染，计算 $L_{\mathrm{i}}^{\textit{cache}}\mathopen{}\left(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}}\mathclose{}\right)$ [14, 43, 51]：

$$\displaystyle L_{\mathrm{i}}^{\textit{cache}}(\mathbf{o}^{\prime},\boldsymbol{\omega}_{\mathrm{i}}) \\ \displaystyle=\sum_{k=1}^{\mathrm{N}}w_{k}\,L_{\mathrm{o}}^{\textit{cache}}\mathopen{}\left(\mathbf{x}^{\prime}(t_{k}),\boldsymbol{\omega}_{\mathrm{o}}^{\prime}\mathclose{}\right). \tag{3}$$

Here, $L_{\mathrm{o}}^{\textit{cache}}$ is the outgoing radiance at each point along the secondary ray predicted by the NeRF. The values $w_{k}$ are quadrature weights that account for the transmittance and absorption along the ray, calculated as a function of the density $\sigma$ at each sample point $\mathbf{x}^{\prime}(t_{k})$ and the ray interval $(\Delta t)_{k}$ [50, 80]:

此处 $L_{\mathrm{o}}^{\textit{cache}}$ 为 NeRF 预测的次级光线上各点出射辐射。$w_{k}$ 为计入透射与吸收的求积权重，由各采样点 $\mathbf{x}^{\prime}(t_{k})$ 的密度 $\sigma$ 与光线区间 $(\Delta t)_{k}$ 计算 [50, 80]：

\displaystyle\resizebox{29143103}{}{$w_{k}=\mathopen{}\left(1-e^{-\sigma(\mathbf{x}^{\prime}(t_{k}))(\Delta t)_{k}}\mathclose{}\right)e^{\mathopen{}\left(-\sum_{j=1}^{k-1}\sigma(\mathbf{x}^{\prime}(t_{j}))(\Delta t)_{j}\mathclose{}\right)}\,$}. \tag{4}

Our work adapts this observation to time-resolved rendering based on lidar measurements.

本文将该观察推广到基于激光雷达测量的时间分辨渲染。

4 Method

4 方法

We model and invert time-resolved light transport, including direct and indirect effects, to recover scene geometry and material properties. Our method uses a physically-based time-resolved renderer with a time-resolved radiance cache parameterized by neural networks, as shown in Figure 2. We perform inverse rendering by optimizing the representation using measurements of propagating light from a flash lidar system.

我们对时间分辨光传输（含直接与间接效应）建模并求逆，以恢复场景几何与材质。方法使用带神经网络参数化时间分辨辐射缓存的、基于物理的时间分辨渲染器，见图 2。我们用闪光激光雷达的传播光测量优化该表示，完成逆渲染。

4.1 Physically-Based Time-Resolved Rendering

4.1 基于物理的时间分辨渲染

We model a lidar measurement by casting a primary ray $\mathbf{x}\mathopen{}\left(t\mathclose{}\right)=\mathbf{o}-t\boldsymbol{\omega}_{\mathrm{o}}$ into the scene. For each point along that ray, we render the outgoing radiance, $L_{\mathrm{o}}$, in the direction of the sensor. Our time-resolved rendering equation is a modified version of Equation 1, where we add the time of flight $\tau$ as

我们向场景投射主光线 $\mathbf{x}\mathopen{}\left(t\mathclose{}\right)=\mathbf{o}-t\boldsymbol{\omega}_{\mathrm{o}}$ 来建模激光雷达测量。对光线上每一点，渲染朝向传感器方向的出射辐射 $L_{\mathrm{o}}$。时间分辨渲染方程是式 1 的修改版，加入飞行时间 $\tau$：

$$\displaystyle L_{\mathrm{o}}(\mathbf{x}\mathopen{}\left(t\mathclose{}\right),\boldsymbol{\omega}_{\mathrm{o}},\tau) \\ \displaystyle=\int_{\mathrm{\Omega}}f\mathopen{}\left(\mathbf{x}\mathopen{}\left(t\mathclose{}\right),\boldsymbol{\omega}_{\mathrm{i}},\boldsymbol{\omega}_{\mathrm{o}}\mathclose{}\right)L_{\mathrm{i}}\mathopen{}\left(\mathbf{x}\mathopen{}\left(t\mathclose{}\right),\boldsymbol{\omega}_{\mathrm{i}},\tau\mathclose{}\right)\mathopen{}\left(\mathbf{n}\cdot\boldsymbol{\omega}_{\mathrm{i}}\mathclose{}\right)\,d\boldsymbol{\omega}_{\mathrm{i}}. \tag{5}$$

The reflectance $f$, is modeled using the Disney–GGX BRDF [9], which depends on the material properties of the scene as described in the appendix.

反射率 $f$ 用 Disney–GGX BRDF [9] 建模，依赖附录所述场景材质属性。

We further decompose the incident radiance into two components $L_{\mathrm{i}}=L_{\text{i}}^{\textit{dir}}+L_{\mathrm{i}}^{\textit{cache}}$: a direct component $L_{\text{i}}^{\textit{dir}}$, and an indirect component $L_{\mathrm{i}}^{\textit{cache}}$ evaluated using the radiance cache. The direct component is given as

我们进一步把入射辐射分解为两项 $L_{\mathrm{i}}=L_{\text{i}}^{\textit{dir}}+L_{\mathrm{i}}^{\textit{cache}}$：直接分量 $L_{\text{i}}^{\textit{dir}}$，以及用辐射缓存评估的间接分量 $L_{\mathrm{i}}^{\textit{cache}}$。直接分量为

$$\displaystyle L_{\text{i}}^{\textit{dir}}(\mathbf{x}\mathopen{}\left(t\mathclose{}\right),\boldsymbol{\omega}_{\mathrm{i}},\tau)=\frac{\delta(\boldsymbol{\omega}_{\ell}-\boldsymbol{\omega}_{\mathrm{i}})L_{\text{i}}^{\ell}\mathopen{}\left(\boldsymbol{\omega}_{\ell},\tau-\frac{||\mathbf{x}(t)-\mathbf{x}_{\ell}||}{c}\mathclose{}\right)}{||\mathbf{x}\mathopen{}\left(t\mathclose{}\right)-\mathbf{x}_{\ell}||^{2}}, \tag{6}$$

where $\mathbf{x}_{\ell}$ is the position of the light source, $\boldsymbol{\omega}_{\ell}$ is the direction from the light source, $L_{\mathrm{i}}^{\ell}$ is the light source intensity, and $\delta(\boldsymbol{\omega}_{\ell}-\boldsymbol{\omega}_{\mathrm{i}})$ is a Dirac delta function. We model the inverse-square law intensity falloff and the time delay to position $\mathbf{x}\mathopen{}\left(t\mathclose{}\right)$ based on the speed of light $c$.

其中 $\mathbf{x}_{\ell}$ 为光源位置，$\boldsymbol{\omega}_{\ell}$ 为来自光源的方向，$L_{\mathrm{i}}^{\ell}$ 为光源强度，$\delta(\boldsymbol{\omega}_{\ell}-\boldsymbol{\omega}_{\mathrm{i}})$ 为狄拉克函数。我们按光速 $c$ 建模到位置 $\mathbf{x}\mathopen{}\left(t\mathclose{}\right)$ 的平方反比衰减与时间延迟。

Similar to the steady-state case (Equation 3), the time-resolved radiance cache $L_{\mathrm{i}}^{\textit{cache}}\mathopen{}\left(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}},\tau\mathclose{}\right)$ is evaluated using secondary rays $\mathbf{x}^{\prime}(t)=\mathbf{o}^{\prime}-t\boldsymbol{\omega}_{\mathrm{o}}^{\prime}$ cast from points on the primary ray $\mathbf{o}^{\prime}=\mathbf{x}$ with $\boldsymbol{\omega}_{\mathrm{o}}^{\prime}=\boldsymbol{\omega}_{\mathrm{i}}$. We use time-resolved volume rendering [3, 47, 17] to render the radiance cache as

与稳态情形（式 3）类似，时间分辨辐射缓存 $L_{\mathrm{i}}^{\textit{cache}}\mathopen{}\left(\mathbf{x},\boldsymbol{\omega}_{\mathrm{i}},\tau\mathclose{}\right)$ 用自主光线上点出发的次级光线 $\mathbf{x}^{\prime}(t)=\mathbf{o}^{\prime}-t\boldsymbol{\omega}_{\mathrm{o}}^{\prime}$（$\mathbf{o}^{\prime}=\mathbf{x}$，$\boldsymbol{\omega}_{\mathrm{o}}^{\prime}=\boldsymbol{\omega}_{\mathrm{i}}$）评估。我们用时间分辨体渲染 [3, 47, 17] 渲染缓存：

$$\displaystyle L_{\mathrm{i}}^{\textit{cache}}(\mathbf{o}^{\prime},\boldsymbol{\omega}_{\mathrm{i}},\tau)=\sum_{k=1}^{\mathrm{N}}w_{k}\,L_{\mathrm{o}}^{\textit{cache}}\mathopen{}\left(\mathbf{x}^{\prime}(t_{k}),\mathbf{x}_{\ell},\boldsymbol{\omega}_{\mathrm{o}}^{\prime},\tau-\frac{t_{k}}{c}\mathclose{}\right), \tag{7}$$

where $L_{\mathrm{o}}^{\textit{cache}}$ is the outgoing radiance predicted by a neural representation. The above states that the light incident at $\mathbf{o}^{\prime}$ in direction $\boldsymbol{\omega}_{\mathrm{i}}$ at time $\tau$ is the sum of delayed copies of the light leaving each point $\mathbf{x}^{\prime}$ along $\boldsymbol{\omega}_{\mathrm{o}}^{\prime}$, and the delay depends on the distance to $\mathbf{o}^{\prime}$ (given by the ray parameter $t_{k}$).

其中 $L_{\mathrm{o}}^{\textit{cache}}$ 为神经表示预测的出射辐射。上式表明：在时间 $\tau$、方向 $\boldsymbol{\omega}_{\mathrm{i}}$ 到达 $\mathbf{o}^{\prime}$ 的光，是沿 $\boldsymbol{\omega}_{\mathrm{o}}^{\prime}$ 离开各点 $\mathbf{x}^{\prime}$ 的光的延迟副本之和，延迟取决于到 $\mathbf{o}^{\prime}$ 的距离（由光线参数 $t_{k}$ 给出）。

After evaluating the time-resolved rendering equation (Equation 5) for each point on the primary ray, the lidar measurement is computed by volume rendering the outgoing radiance in the same way as for the cache:

对主光线上每一点求完时间分辨渲染方程（式 5）后，激光雷达测量同样对出射辐射做体渲染，方式与缓存相同：

$$\displaystyle L_{\mathrm{i}}(\mathbf{o},\boldsymbol{\omega}_{\mathrm{o}},\tau) \\ \displaystyle=\sum_{k=1}^{\mathrm{N}}w_{k}\,L_{\mathrm{o}}\mathopen{}\left(\mathbf{x}\mathopen{}\left(t_{k}\mathclose{}\right),\boldsymbol{\omega}_{\mathrm{o}},\tau-\frac{t_{k}}{c}\mathclose{}\right). \tag{8}$$

4.2 Time-Resolved Radiance Cache

4.2 时间分辨辐射缓存

Representation.

The cache is parameterized using a multi-resolution hash encoding $\mathcal{H}^{\textit{app}}$ [55, 5] to learn a position-dependent appearance feature $\mathbf{f}^{\textit{app}}$. Similarly, a hash encoding-based neural network $\mathcal{N}^{\textit{geom}}$ represents scene geometry through density and normals. That is,

表示。

缓存用多分辨率哈希编码 $\mathcal{H}^{\textit{app}}$ [55, 5] 参数化，学习位置相关外观特征 $\mathbf{f}^{\textit{app}}$。同样，基于哈希编码的网络 $\mathcal{N}^{\textit{geom}}$ 通过密度与法向表示场景几何。即

$$\mathbf{f}^{\textit{app}}=\mathcal{H}^{\textit{app}}(\mathbf{x}),\quad\sigma,\mathbf{n}=\mathcal{N}^{\textit{geom}}(\mathbf{x}). \tag{9}$$

The density values used for volume rendering are shared across both the physically-based model and the radiance cache (i.e., Equations 7 and 8). The appearance features are used to compute the radiance cache, which we decompose into direct and indirect components as

用于体渲染的密度在基于物理的模型与辐射缓存间共享（即式 7 与 8）。外观特征用于计算辐射缓存，我们将其分解为直接与间接分量：

$$\displaystyle L_{\mathrm{o}}^{\textit{cache}}=L_{\mathrm{o}}^{\textit{cache,dir}}+L_{\mathrm{o}}^{\textit{cache,indir}}. \tag{10}$$

We describe each of these components as follows.

两项如下。

Direct light.

The direct component is due to light that is emitted from the lidar source at $\mathbf{x}_{\ell}$, propagates to a point $\mathbf{x}^{\prime}$ in the scene, and scatters directly to the ray origin $\mathbf{o}^{\prime}$:

直接光。

直接分量来自激光雷达源 $\mathbf{x}_{\ell}$ 发出、传到场景点 $\mathbf{x}^{\prime}$ 并直接散射到光线原点 $\mathbf{o}^{\prime}$ 的光：

$$\displaystyle L_{\mathrm{o}}^{\textit{cache,dir}}(\mathbf{x}^{\prime}(t),\mathbf{x}_{\ell},\boldsymbol{\omega}_{\mathrm{o}}^{\prime},\tau) \\ \displaystyle=f^{\textit{dir}}\mathopen{}\left(\mathbf{f}^{\textit{app}},\mathbf{n},\boldsymbol{\omega}_{\ell},\boldsymbol{\omega}_{\mathrm{o}}^{\prime}\mathclose{}\right)\,L_{\text{i}}^{\textit{dir}}(\mathbf{x}^{\prime}(t),\boldsymbol{\omega}_{\ell},\tau)\mathopen{}\left(\mathbf{n}\cdot\boldsymbol{\omega}_{\ell}\mathclose{}\right). \tag{11}$$

Here, $f^{\textit{dir}}$ is a neural network that learns the BRDF (see the appendix for a detailed description). In practice, we discretize the equation and compute a vector that represents radiance at each time interval.

此处 $f^{\textit{dir}}$ 为学习 BRDF 的神经网络（细节见附录）。实践中离散化该式，计算表示各时间区间辐射的向量。

Indirect light.

We leverage a split-sum approximation [30] to efficiently cache outgoing indirect light:

间接光。

我们用 split-sum 近似 [30] 高效缓存出射间接光：

$$\displaystyle L_{\mathrm{o}}^{\textit{cache,indir}}(\mathbf{x}^{\prime}(t),\mathbf{x}_{\ell},\boldsymbol{\omega}_{\mathrm{o}}^{\prime},\tau) \\ \displaystyle\approx\int_{\mathrm{\Omega}}f\mathopen{}\left(\mathbf{x}^{\prime},\boldsymbol{\omega}_{\mathrm{i}}^{\prime},\boldsymbol{\omega}_{\mathrm{o}}^{\prime}\mathclose{}\right)\mathopen{}\left(\mathbf{n}\cdot\boldsymbol{\omega}_{\mathrm{i}}^{\prime}\mathclose{}\right)\,d\boldsymbol{\omega}_{\mathrm{i}}^{\prime}\cdot\int_{\mathrm{\Omega}}L_{\mathrm{i}}\mathopen{}\left(\mathbf{x}^{\prime},\mathbf{x}_{\ell},\boldsymbol{\omega}_{\mathrm{i}}^{\prime},\tau\mathclose{}\right)d\boldsymbol{\omega}_{\mathrm{i}}^{\prime} \\ \displaystyle=f_{\Omega}^{\textit{indir}}(\mathbf{f}^{\textit{app}},\mathbf{n},\boldsymbol{\omega}_{\mathrm{o}}^{\prime})\cdot L_{\text{i},\Omega}^{\textit{indir}}(\mathbf{f}^{\textit{app}},\mathbf{x}_{\ell},\mathbf{n},\boldsymbol{\omega}_{\mathrm{o}}^{\prime}), \tag{12}$$

where $f_{\Omega}^{\textit{indir}}$ and $L_{\text{i},\Omega}^{\textit{indir}}$ are neural networks that predict the integrated BRDF and integrated incident radiance, respectively. We include conditioning on $\mathbf{x}_{\ell}$, as the indirect light depends on light source position. Following Malik et al. [48], $L_{\text{i},\Omega}^{\textit{indir}}$ predicts a vector that represents radiance over discretized time intervals.

其中 $f_{\Omega}^{\textit{indir}}$ 与 $L_{\text{i},\Omega}^{\textit{indir}}$ 分别为预测积分 BRDF 与积分入射辐射的神经网络。因间接光依赖光源位置，我们对 $\mathbf{x}_{\ell}$ 做条件化。遵循 Malik et al. [48]，$L_{\text{i},\Omega}^{\textit{indir}}$ 预测表示离散时间区间上辐射的向量。

4.3 Inverse Rendering from Propagating Light

4.3 从传播光做逆渲染

The lidar system captures the time-resolved incident radiance at the sensor $L_{\text{i}}^{\textit{meas}}$. We optimize the representation by minimizing the difference between the lidar measurements and the output $L_{\mathrm{i}}$ of the physically-based renderer (we omit their dependence on $\mathbf{o}$, $\boldsymbol{\omega}_{\mathrm{o}}$, and $\tau$ for brevity):

激光雷达系统捕获传感器处时间分辨入射辐射 $L_{\text{i}}^{\textit{meas}}$。我们通过最小化激光雷达测量与基于物理渲染器输出 $L_{\mathrm{i}}$ 的差异来优化表示（为简洁省略对 $\mathbf{o}$、$\boldsymbol{\omega}_{\mathrm{o}}$、$\tau$ 的依赖）：

$$\displaystyle\mathcal{L}_{\textit{data}}=\sum_{\mathbf{o},\boldsymbol{\omega}_{\mathrm{o}},\tau}\alpha(L_{\mathrm{i}}^{\textit{cache}})\lVert L_{\mathrm{i}}-L_{\text{i}}^{\textit{meas}}\rVert^{2}. \tag{13}$$

As the cache can also be used to render the time-resolved incident radiance at the sensor, we supervise it in the same fashion by minimizing $\mathcal{L}_{\textit{cache}}$, which replaces $L_{\mathrm{i}}$ with $L_{\mathrm{i}}^{\textit{cache}}$ in the above. The function $\alpha$ is chosen to more strongly penalize errors in darker regions, which improves perceptual quality similar to applying a tonemapping curve  [52]:

缓存也可用于渲染传感器处时间分辨入射辐射，我们以同样方式最小化 $\mathcal{L}_{\textit{cache}}$（上式中用 $L_{\mathrm{i}}^{\textit{cache}}$ 替换 $L_{\mathrm{i}}$）监督之。函数 $\alpha$ 用于更重惩罚暗区误差，类似施加色调映射曲线以提升感知质量 [52]：

$$\displaystyle\alpha(L)=\Big|\sum_{\tau}L\Big|^{-\beta}, \tag{14}$$

where $\beta$ is a hyperparameter. Note that in Equation 13, we compute this weight using the incoming radiance from the cache ($L_{\mathrm{i}}^{\textit{cache}}$ instead of $L_{\mathrm{i}}$), as it is not affected by Monte Carlo render noise.

其中 $\beta$ 为超参。注意式 13 中该权重用缓存入射辐射（$L_{\mathrm{i}}^{\textit{cache}}$ 而非 $L_{\mathrm{i}}$）计算，因其不受蒙特卡洛渲染噪声影响。

Following Hadadan et al. [19], we leverage a radiometric prior that constrains the direct and indirect light rendered using the cache to be consistent with the full physically-based model. Specifically, we render the direct and indirect components of radiance at sample points $\mathbf{x}$ along the primary ray, using the cache ($L_{\mathrm{o}}^{\textit{cache,dir/indir}}$) and the physically-based model (i.e., $L_{\mathrm{o}}^{\textit{dir/indir}}$, given by evaluating Equation 5 using only $L_{\text{i}}^{\textit{dir}}$ or $L_{\mathrm{i}}^{\textit{cache}}$, respectively). We constrain the cache using the loss:

遵循 Hadadan et al. [19]，我们使用辐射度量先验，约束缓存渲染的直接与间接光与完整基于物理模型一致。具体地，在主光线采样点 $\mathbf{x}$ 上，分别用缓存（$L_{\mathrm{o}}^{\textit{cache,dir/indir}}$）与基于物理模型（即仅用 $L_{\text{i}}^{\textit{dir}}$ 或 $L_{\mathrm{i}}^{\textit{cache}}$ 求式 5 得到的 $L_{\mathrm{o}}^{\textit{dir/indir}}$）渲染直接/间接分量，并以损失约束缓存：

\displaystyle\resizebox{32561971}{}{$\mathcal{L}_{\textit{dir/indir}}=\sum_{\mathbf{x},\boldsymbol{\omega}_{\mathrm{o}},\tau}\alpha(L_{\mathrm{o}}^{\textit{cache,dir/indir}})\lVert L_{\mathrm{o}}^{\textit{cache,dir/indir}}-L_{\mathrm{o}}^{\textit{dir/indir}}\rVert^{2}$}. \tag{15}

Finally, the complete photometric loss function is

最终完整光度损失函数为

$$\displaystyle\mathcal{L}_{\textit{data}}+\lambda_{\textit{cache}}\mathcal{L}_{\textit{cache}}+\lambda_{\textit{dir}}\mathcal{L}_{\textit{dir}}+\lambda_{\textit{indir}}\mathcal{L}_{\textit{indir}}, \tag{16}$$

where $\lambda_{\textit{cache}}$, $\lambda_{\textit{dir}}$, and $\lambda_{\textit{indir}}$ are hyperparameters that weigh each loss component. By minimizing this loss function, the method recovers a material model for the scene (parameterized using the Disney–GGX model) and the scene geometry, normals, and appearance parameters. In addition to the above photometric loss, we include a regularizer $\mathcal{L}_{\textit{normals}}$ that ties predicted normals to analytic normals from the density field [85]; a smoothness penalty $\mathcal{L}_{\textit{geom}}$ on the analytic normals; a smoothness penalty $\mathcal{L}_{\textit{mat}}$ on the predicted BRDF parameters; proposal resampling and distortion losses $\mathcal{L}_{\textit{interlevel}}$ and $\mathcal{L}_{\textit{distortion}}$ as in Zip-NeRF [5]; and a mask loss $\mathcal{L}_{\textit{mask}}$.

其中 $\lambda_{\textit{cache}}$、$\lambda_{\textit{dir}}$、$\lambda_{\textit{indir}}$ 为各损失分量权重。最小化该损失可恢复场景的材质模型（Disney–GGX 参数化）以及几何、法向与外观参数。除上述光度损失外，还包括将预测法向与密度场解析法向对齐的正则 $\mathcal{L}_{\textit{normals}}$ [85]；解析法向光滑罚 $\mathcal{L}_{\textit{geom}}$；预测 BRDF 参数光滑罚 $\mathcal{L}_{\textit{mat}}$；Zip-NeRF [5] 式的提案重采样与畸变损失 $\mathcal{L}_{\textit{interlevel}}$、$\mathcal{L}_{\textit{distortion}}$；以及掩码损失 $\mathcal{L}_{\textit{mask}}$。

We use multiple importance sampling for secondary rays based on the BRDF and a learnable importance sampler for incident illumination as in Attal et al. [4]. The learnable importance sampler is supervised with a loss $\mathcal{L}_{\textit{vMF}}$. We represent the time-resolved direct outgoing light as a one-hot vector, where each bin corresponds to a discrete time interval, and indirect light as a dense vector of the same size (where the size depends on the dataset). A complete description is provided in the appendix.

次级光线采用基于 BRDF 与可学习入射光照重要性采样器的多重重要性采样，如 Attal et al. [4]。可学习重要性采样器用损失 $\mathcal{L}_{\textit{vMF}}$ 监督。时间分辨直接出射光表示为 one-hot 向量（每维对应离散时间区间），间接光为同尺寸稠密向量（尺寸依赖数据集）。完整描述见附录。

Figure 3: Multi-view capture setup. An elevation arm controls the elevation angle and a rotation stage controls the azimuth angle of the scanning SPAD. Laser light is out-coupled from an optical fiber through a collimating lens and diffusers onto the scene.

图 3：多视角采集装置。俯仰臂控制扫描 SPAD 的俯仰角，旋转台控制方位角。激光经光纤出射，通过准直透镜与散射片照射场景。

Figure 4: Simulated results. Compared to the baselines our method recovers more accurate normals and similar or improved intensity images due to physically-based modeling of time-resolved indirect light transport (see arrows in the lidar frame insets).

图 4：仿真结果。相对基线，我们的方法因对时间分辨间接光传输做基于物理的建模，恢复更准确的法向，强度图相近或更好（见激光雷达帧插图中箭头）。

5 Results

5 结果

Figure 5: Results on the captured dataset. Our method recovers more accurate normals compared to FWP++ (cols. 3, 5) due to its physically-based modeling of indirect light transport effects (visible in the individual lidar frames; cols. 4, 6). Areas where FWP++ predicts incorrect normals usually correspond to regions with indirect light (arrows).

图 5：实采数据集结果。因对间接光传输做基于物理建模，我们的方法相对 FWP++（第 3、5 列）恢复更准确法向（在各激光雷达帧中可见；第 4、6 列）。FWP++ 法向错误区域通常对应间接光区域（箭头）。

Figure 6: Relighting results using three different light sources (color-coded in the RGB channels) for the captured house scene (top) and the simulated kitchen scene (bottom).

图 6：对实采 house 场景（上）与仿真 kitchen 场景（下），用三个不同光源（在 RGB 通道中着色编码）的重光照结果。

We evaluate our system on three tasks: (1) view synthesis of time-resolved lidar measurements, (2) view synthesis of integrated (steady-state) lidar images, and (3) geometry reconstruction.

我们在三项任务上评估系统：(1) 时间分辨激光雷达测量的视角合成，(2) 积分（稳态）激光雷达图像的视角合成，(3) 几何重建。

Evaluation metrics.

To assess the rendered integrated lidar images, we use PSNR, SSIM, and LPIPS [97]. To evaluate the accuracy of the recovered time-resolved measurements, we use the transient intersection-over-union (transient IOU) introduced by Malik et al. [48]. We use mean absolute error (MAE) and L1 error to measure the accuracy of the recovered normals and depth, respectively. We report quantitative results for both simulated and captured scenes in Table 1.

评测指标。

对渲染的积分激光雷达图像，使用 PSNR、SSIM 与 LPIPS [97]。对恢复的时间分辨测量，使用 Malik et al. [48] 提出的瞬态交并比（transient IOU）。用法向平均绝对误差（MAE）与深度 L1 误差分别衡量法向与深度精度。表 1 报告仿真与实采场景的定量结果。

Baselines.

We compare our method to the following state-of-the-art time-resolved neural rendering techniques: T-NeRF [47], which uses a neural representation and a rendering model for time-resolved lidar measurements (but which only accounts for the direct component of light), and Flying with Photons (FWP) [48], which predicts time-resolved radiance at every point in space. Since the original version of FWP does not model light sources or movement of light sources, we use a modified version (FWP++) equivalent to our full, time-resolved radiance cache. We implement both baselines using the same hash-encoding-based neural representation [55], and we use the same regularizers and hyperparameters for fairness.

基线。

我们与以下当时最优时间分辨神经渲染技术比较：T-NeRF [47]（用神经表示与时间分辨激光雷达渲染模型，但只计入光的直接分量），以及 Flying with Photons（FWP） [48]（预测空间每点的时间分辨辐射）。因原版 FWP 不建模光源或光源运动，我们使用等价于完整时间分辨辐射缓存的修改版（FWP++）。两基线均用同一基于哈希编码的神经表示 [55] 实现，并使用相同正则与超参以保证公平。

5.1 Simulated Results

5.1 仿真结果

Dataset.

We use a modification [72] of the Mitsuba 2 renderer [58] to render multi-view transient data for three small-scale, object-centric synthetic scenes, Cornell box, pots, peppers, as well as one room scale scene, kitchen [6].

数据集。

我们用 Mitsuba 2 渲染器 [58] 的修改版 [72]，为三个小尺度、以物体为中心的合成场景 Cornell box、pots、peppers，以及一个房间尺度场景 kitchen [6] 渲染多视角瞬态数据。

Comparison.

In Figure 4, we show integrated lidar scans from novel viewpoints, recovered normals, and individual lidar frames for the peppers and pots scenes. Since T-NeRF [47] only models direct light, it fails to recover accurate geometry under strong indirect light from specular reflections and diffuse inter-reflections. In particular, it introduces floating artifacts to explain these effects and hence fails to predict novel views accurately.

比较。

图 4 展示 peppers 与 pots 场景的新视角积分激光雷达扫描、恢复法向与单帧激光雷达。因 T-NeRF [47] 只建模直接光，在镜面反射与漫反射互反射造成的强间接光下无法恢复准确几何，尤其会引入漂浮伪影解释这些效应，从而新视角预测失败。

On the other hand, FWP++ [48] models both direct and indirect radiance and recovers accurate integrated novel views and lidar frames. However, because it does not use a physically-accurate rendering model, it overfits to the data. Specifically, it is free to use a mirror copy of the scene to explain specular reflections (e.g., on the floor of the peppers scene or the partially specular blue walls in the pots scene). Further, it uses incorrect depths to explain diffuse inter-reflections (e.g., at the wall corners in the peppers scene or along the fluting of the column in the pots scene).

另一方面，FWP++ [48] 同时建模直接与间接辐射，能恢复准确的积分新视角与激光雷达帧。但因其未使用物理准确的渲染模型，会过拟合数据。具体地，它可用场景的镜像副本解释镜面反射（如 peppers 地板或 pots 部分镜面蓝墙）；也用错误深度解释漫反射互反射（如 peppers 墙角或 pots 柱身凹槽）。

	method	PSNR (dB)$\,\uparrow$	LPIPS$\,\downarrow$	SSIM$\,\uparrow$	MAE$\,\downarrow$	L1 depth$\,\downarrow$	T-IOU$\,\uparrow$
sim	T-NeRF [47]	22.44	0.40	0.71	28.00	0.59	0.58
FWP++ [48]	29.00	0.30	0.87	22.80	0.47	0.73
ours	30.99	0.31	0.89	8.45	0.21	0.76
cap	T-NeRF [47]	14.67	0.53	0.35	—	—	0.23
FWP++ [48]	28.45	0.32	0.81	—	—	0.55
ours	27.39	0.33	0.80	—	—	0.54

Table 1: Evaluation of lidar rendering from novel viewpoints and geometry recovery. Each dataset (sim and real) contains 4 scenes.

	方法	PSNR (dB)$\,\uparrow$	LPIPS$\,\downarrow$	SSIM$\,\uparrow$	MAE$\,\downarrow$	L1 depth$\,\downarrow$	T-IOU$\,\uparrow$
sim	T-NeRF [47]	22.44	0.40	0.71	28.00	0.59	0.58
	FWP++ [48]	29.00	0.30	0.87	22.80	0.47	0.73
	ours	30.99	0.31	0.89	8.45	0.21	0.76
cap	T-NeRF [47]	14.67	0.53	0.35	—	—	0.23
	FWP++ [48]	28.45	0.32	0.81	—	—	0.55
	ours	27.39	0.33	0.80	—	—	0.54

表 1：新视角激光雷达渲染与几何恢复评测。各数据集（sim 与 real）含 4 个场景。

5.2 Captured Results

5.2 实采结果

Dataset.

We use a hardware setup (Figure 3) similar to that of Malik et al. [48] to capture a multi-viewpoint flash lidar dataset. Unlike Malik et al. [48], our light source position moves with the camera’s viewpoint rather than being stationary with respect to the scene. We capture three scenes: globe, house, and spheres, and we use the statue scene from Malik et al. [48], which shows that our method can handle stationary light sources. We provide more details about capture and calibration in the appendix.

数据集。

我们使用与 Malik et al. [48] 类似的硬件装置（图 3）采集多视角闪光激光雷达数据。与 Malik et al. [48] 不同，我们的光源随相机视角移动，而非相对场景固定。采集三个场景：globe、house 与 spheres，并使用 Malik et al. [48] 的 statue 场景以表明方法可处理固定光源。采集与标定细节见附录。

Comparison.

Visually, the captured results follow a similar trend to the simulated ones. Figure 5 shows that we recover more accurate geometry than FWP++, especially for areas where indirect light is present, such as the corners of the walls in the globe scene or the bottom of the candles in the statue scene.

比较。

视觉上，实采结果趋势与仿真类似。图 5 显示我们比 FWP++ 恢复更准确几何，尤其在有间接光的区域，如 globe 墙角或 statue 蜡烛底部。

Table 1 provides quantitative results for novel view synthesis of lidar measurements. While we find that FWP++ shows slight improvements over our approach for view synthesis, we hypothesize that this is because it uses a far less constrained model—our approach is physically grounded and could thus be more sensitive to the calibration of the physical system and model mismatch. Note that we omit depth and normal metrics for the captured experiments, as there is no ground truth reference.

表 1 给出激光雷达测量新视角合成的定量结果。虽发现 FWP++ 在视角合成上略优于我们，我们推测因其模型约束更少——我们的方法有物理根基，因而可能对物理系统标定与模型失配更敏感。注意实采实验省略深度与法向指标，因无真值参考。

5.3 Additional Results

5.3 补充结果

Relighting.

Our approach enables time-resolved view synthesis and relighting, which, to our knowledge, has not been demonstrated before from captured multi-view, time-resolved measurements. Figure 1 shows relighting results on a novel view from the captured house scene with a simulated pulsed source that projects a pattern with the letters “C-V-P-R”. In Figure 6, we show two additional examples of time-resolved relighting: (1) the same house scene with three different light sources that converge on the house, and (2) the simulated kitchen scene, relit with three point sources that emit pulse trains.

重光照。

我们的方法支持时间分辨视角合成与重光照；据我们所知，这尚未在实采多视角、时间分辨测量上展示过。图 1 展示在实采 house 场景新视角上，用投影「C-V-P-R」字母图案的仿真脉冲源做重光照。图 6 另给两个时间分辨重光照例子：(1) 同一 house 场景上三个汇聚到房屋的不同光源；(2) 仿真 kitchen 用三个发射脉冲串的点光源重光照。

Note that the indirect component of the radiance cache is predicted using a neural network conditioned on the light source position (Equation 12). So, if the intensity profile of the light source used for relighting differs from the training data (e.g., a projector instead of a uniform point source as in Figure 1), we use fine-tuning with the radiometric loss $\mathcal{L}_{\textit{dir/indir}}$ of Equation 15 and the desired light source profile (additional details are provided in the appendix).

注意辐射缓存的间接分量由以光源位置为条件的神经网络预测（式 12）。因此，若重光照所用光源强度轮廓与训练数据不同（如图 1 中用投影机而非均匀点源），我们用式 15 的辐射度量损失 $\mathcal{L}_{\textit{dir/indir}}$ 与目标光源轮廓做微调（细节见附录）。

Time-resolved imaging without lidar.

Although our system recovers time-resolved light transport, it does not necessarily require supervision with lidar measurements. Notably, we can train our model using continuous-wave time-of-flight (CW-ToF) measurements or even intensity images, as both can be derived from time-resolved data. We demonstrate this capability by generating time-resolved videos of propagating light based on each input type. Figure 7 shows the rendered time-resolved measurements after training with either CW-ToF measurements (emulated by convolving the lidar measurements with the CW-ToF illumination waveform) or intensity images (emulated by integrating the lidar measurements over time). We also demonstrate recovery of direct and indirect light transport effects from the ToF measurements and the steady-state images. For example, in house, we recover diffuse inter-reflections under the mushroom, and in spheres, we recover reflections from the ground to the specular sphere (arrows in Figure 7).

无激光雷达的时间分辨成像。

尽管系统恢复时间分辨光传输，监督未必需要激光雷达测量。值得注意的是，可用连续波飞行时间（CW-ToF）测量甚至强度图像训练模型，二者均可由时间分辨数据导出。我们按各输入类型生成传播光时间分辨视频以展示该能力。图 7 显示分别用 CW-ToF 测量（将激光雷达测量与 CW-ToF 照明波形卷积来仿真）或强度图像（对激光雷达测量时间积分来仿真）训练后渲染的时间分辨测量。我们也展示从 ToF 测量与稳态图像恢复直接与间接光传输效应。例如在 house 中恢复蘑菇下方的漫反射互反射，在 spheres 中恢复从地面到镜面球的反射（图 7 箭头）。

For this application, we only modify the supervision of the model. Specifically, we convert the time-resolved predictions to emulate the CW-ToF or steady-state measurements before calculating the loss. After optimization, we directly render the time-resolved video frames.

对该应用，我们只修改模型监督方式：先把时间分辨预测转换为仿真 CW-ToF 或稳态测量再算损失。优化后直接渲染时间分辨视频帧。

Material decomposition.

Our method also recovers the material parameters for the Disney–GGX BRDF (albedo, roughness, and metalness). We visualize these parameters for synthetic and captured scenes in the appendix.

材质分解。

方法还恢复 Disney–GGX BRDF 的材质参数（反照率、粗糙度与金属度）。附录可视化了仿真与实采场景上的这些参数。

Figure 7: Our approach recovers time-resolved videos of propagating light after training on continuous-wave time-of-flight (CW-ToF) measurements or intensity images. The video frames show direct and indirect light transport effects (arrows).

图 7：在连续波飞行时间（CW-ToF）测量或强度图像上训练后，我们的方法恢复传播光时间分辨视频。视频帧显示直接与间接光传输效应（箭头）。

6 Discussion

6 讨论

Limitations.

As noted in the results section, our method relies on a more constrained physical model than other approaches, including the FWP++ baseline. As such, we notice some performance degradation compared to baselines on captured data, where model mismatch is a potential issue. This problem could perhaps be mitigated through better calibration of the physical setup. Additionally, our method requires more than one day of optimization on a single GPU due to the time-consuming physical light transport simulation, I/O penalties from loading large time-resolved measurement vectors, and GPU memory bandwidth requirements. To address this, it may be possible to use a different neural representation that does not predict the entire time-resolved vector, but instead predicts and supervises the signal at a single time instant. Using faster neural representations, like 3D Gaussian Ray Tracing [53] or EVER [46], is also an interesting direction.

局限。

如结果节所述，相对包括 FWP++ 在内的其他方法，我们的方法依赖约束更强的物理模型。因而在可能存在模型失配的实采数据上，相对基线会有一定性能下降。更好的物理装置标定或许能缓解。此外，因物理光传输仿真耗时、加载大型时间分辨测量向量的 I/O 开销以及 GPU 内存带宽需求，单卡优化需一天以上。对此，或可采用不预测整段时间分辨向量、而在单一时刻预测并监督信号的神经表示。使用更快表示如 3D Gaussian Ray Tracing [53] 或 EVER [46] 也是有趣方向。

Impact and future applications.

Although we do not tackle the problem of non-line-of-sight imaging in this work, it is in principle possible to extend our framework for this application in unconstrained conditions, such as non-line-of-sight imaging with non-planar relay surfaces [39]. Due to the widespread use of lidar technologies, we believe our work has potential for impact in areas such as autonomous navigation or remote sensing — especially in scenarios with strong indirect lighting effects.

影响与未来应用。

虽未在本文处理非视距成像，原则上可将框架扩展到非平面中继表面非视距成像等非受限条件 [39]。鉴于激光雷达技术广泛使用，我们相信本工作在自动驾驶或遥感等领域——尤其在强间接光照场景——具有潜在影响。

Acknowledgments.

DBL acknowledges support from NSERC under the RGPIN program, the Canada Foundation for Innovation, and the Ontario Research Fund. BA is supported by a Meta Research PhD Fellowship. MO acknowledges support from NSF CAREER 2238485.

致谢。

DBL 感谢 NSERC RGPIN、加拿大创新基金会与安大略研究基金支持。BA 获 Meta Research PhD Fellowship 资助。MO 获 NSF CAREER 2238485 支持。

References

参考文献

[1] Amit Adam, Christoph Dann, Omer Yair, Shai Mazor, and Sebastian Nowozin. Bayesian time-of-flight for realtime shape, illumination and albedo. IEEE Trans. Pattern Anal. Mach. Intell. , 39(5):851–864, 2016.

[2] Apple Inc. Apple vision pro, 2024. Accessed: 2024-11-04.

[3] Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthew O’Toole. TöRF: Time-of-flight radiance fields for dynamic scene view synthesis. Proc. NeurIPS , 2021.

[4] Benjamin Attal, Dor Verbin, Ben Mildenhall, Peter Hedman, Jonathan T Barron, Matthew O’Toole, and Pratul P Srinivasan. Flash cache: Reducing bias in radiance cache based inverse rendering. In Proc. ECCV , 2024.

[5] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-NeRF: Anti-aliased grid-based neural radiance fields. In Proc. ICCV , 2023.

[6] Benedikt Bitterli. Rendering resources, 2016. https://benedikt-bitterli.me/resources/.

[7] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P.A. Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. In Proc. NeurIPS , 2021.

[8] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan T. Barron, Hendrik P.A. Lensch, and Varun Jampani. SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections. In Proc. NeurIPS , 2022.

[9] Brent Burley and Walt Disney Animation Studios. Physically-based shading at Disney. In ACM SIGGRAPH Courses , 2012.

[10] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proc. CVPR , 2019.

[11] Paul Debevec. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. SIGGRAPH , 1998.

[12] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proc. CVPR , 2022.

[13] Pinliang Dong and Qi Chen. LiDAR remote sensing and applications . CRC Press, 2017.

[14] Robert A Drebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. ACM SIGGRAPH , 22(4):65–74, 1988.

[15] Daniele Faccio, Andreas Velten, and Gordon Wetzstein. Non-line-of-sight imaging. Nat. Rev. Phys. , 2(6):318–327, 2020.

[16] Genevieve Gariepy, Nikola Krstajić, Robert Henderson, Chunyong Li, Robert R Thomson, Gerald S Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. Single-photon sensitive light-in-fight imaging. Nat. Commun. , 6(1):6021, 2015.

[17] Ioannis Gkioulekas, Anat Levin, and Todd Zickler. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In Proc. ECCV , 2016.

[18] Mohit Gupta, Shree K Nayar, Matthias B Hullin, and Jaime Martin. Phasor imaging: A generalization of correlation-based time-of-flight imaging. ACM Trans. Graph. , 34(5):1–18, 2015.

[19] Saeed Hadadan, Geng Lin, Jan Novák, Fabrice Rousselle, and Matthias Zwicker. Inverse global illumination using a neural radiometric prior. In Proc. SIGGRAPH , pages 1–11, 2023.

[20] Felix Heide, Matthias B Hullin, James Gregson, and Wolfgang Heidrich. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph. , 32(4):1–10, 2013.

[21] Shengyu Huang, Zan Gojcic, Zian Wang, Francis Williams, Yoni Kasten, Sanja Fidler, Konrad Schindler, and Or Litany. Neural lidar fields for novel view synthesis. In Proc. ICCV , 2023.

[22] Andrew S Huntington. InGaAs avalanche photodiodes for ranging and Lidar . Woodhead Publishing, 2020.

[23] Kazuto Ichimaru, Diego Thomas, Takafumi Iwaguchi, and Hiroshi Kawasaki. Neural active structure-from-motion in dark and textureless environment. Proc. ACCV , 2024.

[24] Julian Iseringhausen and Matthias B Hullin. Non-line-of-sight reconstruction using efficient transient rendering. ACM Trans. Graph. , 39(1):1–14, 2020.

[25] Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vicini. Dr. jit: A just-in-time compiler for differentiable rendering. ACM Trans. Graph. , 41(4):1–19, 2022.

[26] Adrian Jarabo and Victor Arellano. Bidirectional rendering of vector light transport. In Computer Graphics Forum , pages 96–105. Wiley Online Library, 2018.

[27] Adrian Jarabo, Julio Marco, Adolfo Munoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. A framework for transient rendering. ACM Trans. Graph. , 33(6):1–10, 2014.

[28] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. TensoIR: Tensorial inverse rendering. In Proc. CVPR , 2023.

[29] James T Kajiya. The rendering equation. In Proc. SIGGRAPH , 1986.

[30] Brian Karis. Real shading in Unreal Engine 4. ACM SIGGRAPH Courses , 2013.

[31] Ahmed Kirmani, Dheera Venkatraman, Dongeek Shin, Andrea Colaço, Franco NC Wong, Jeffrey H Shapiro, and Vivek K Goyal. First-photon imaging. Science , 343(6166):58–61, 2014.

[32] Tzofi Klinghoffer, Xiaoyu Xiang, Siddharth Somasundaram, Yuchen Fan, Christian Richardt, Ramesh Raskar, and Rakesh Ranjan. PlatoNeRF: 3D reconstruction in Plato’s cave via single-view two-bounce lidar. In Proc. CVPR , 2024.

[33] Walter Koechner. Optical ranging system employing a high power injection laser diode. IEEE Trans. Aerosp. Electron. Syst. , AES-4(1):81–91, 1968.

[34] Larry Li et al. Time-of-flight camera—an introduction. Technical white paper , 2014.

[35] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. , 37(6):1–11, 2018.

[36] You Li and Javier Ibanez-Guzman. Lidar for autonomous driving: The principles, challenges, and trends for automotive lidar and perception systems. IEEE Signal Process. Mag. , 37(4):50–61, 2020.

[37] David B Lindell and Gordon Wetzstein. Three-dimensional imaging through scattering media based on confocal diffuse tomography. Nat. Commun. , 11(1):4517, 2020.

[38] David B Lindell, Matthew O’Toole, and Gordon Wetzstein. Towards transient imaging at interactive rates with single-photon detectors. In Proc. ICCP , 2018.

[39] David B Lindell, Gordon Wetzstein, and Matthew O’Toole. Wave-based non-line-of-sight imaging using fast fk migration. ACM Trans. Graph. , 38(4):1–13, 2019.

[40] Jingwang Ling, Ruihan Yu, Feng Xu, Chun Du, and Shuang Zhao. NeRF as a non-distant environment emitter in physics-based inverse rendering. In Proc. SIGGRAPH , 2024.

[41] Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature , 572(7771):620–623, 2019.

[42] Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. NERO: Neural geometry and BRDF reconstruction of reflective objects from multiview images. ACM Trans. Graph. , 42(4):1–22, 2023.

[43] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Trans. Graph. , 38(4):1–14, 2019.

[44] Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. , 38(6):1–14, 2019.

[45] Weihan Luo, Anagh Malik, and David B Lindell. Transientangelo: Few-viewpoint surface reconstruction using single-photon lidar. Proc. WACV , 2025.

[46] Alexander Mai, Dor Verbin, Falko Kuester, and Sara Fridovich-Keil. Neural microfacet fields for inverse rendering. In Proc. ICCV , 2023.

[47] Anagh Malik, Parsa Mirdehghan, Sotiris Nousias, Kiriakos N Kutulakos, and David B Lindell. Transient neural radiance fields for lidar view synthesis and 3D reconstruction. In Proc. NeurIPS , 2023.

[48] Anagh Malik, Noah Juravsky, Ryan Po, Gordon Wetzstein, Kiriakos N. Kutulakos, and David B. Lindell. Flying with photons: Rendering novel views of propagating light. In Proc. ECCV , 2024.

[49] Mathworks. Camera calibrator app. https://www.mathworks.com/help/vision/ref/cameracalibrator-app.html , 2020.

[50] Nelson Max. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph. , 1(2):99–108, 1995.

[51] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Commun. ACM , 65(1):99–106, 2021.

[52] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan, and Jonathan T. Barron. NeRF in the dark: High dynamic range view synthesis from noisy raw images. In Proc. CVPR , 2022.

[53] Nicolas Moenne-Loccoz, Ashkan Mirzaei, Or Perel, Riccardo de Lutio, Janick Martinez Esturo, Gavriel State, Sanja Fidler, Nicholas Sharp, and Zan Gojcic. 3D Gaussian ray tracing: Fast tracing of particle scenes. Proc. SIGGRAPH Asia , 2024.

[54] Fangzhou Mu, Carter Sifferman, Sacha Jungerman, Yiquan Li, Mark Han, Michael Gleicher, Mohit Gupta, and Yin Li. Towards 3D vision with low-cost single-photon cameras. In Proc. CVPR , 2024.

[55] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. (SIGGRAPH) , 41(4):1–15, 2022.

[56] Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. , 30(6):1–10, 2011.

[57] Nikhil Naik, Achuta Kadambi, Christoph Rhemann, Shahram Izadi, Ramesh Raskar, and Sing Bing Kang. A light transport model for mitigating multipath interference in time-of-flight sensors. In Proc. CVPR , pages 73–81, 2015.

[58] Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. , 38(6):1–17, 2019.

[59] Mikhail Okunev, Marc Mapeke, Benjamin Attal, Christian Richardt, Matthew O’Toole, and James Tompkin. Flowed time of flight radiance fields. In Proc. ECCV , 2024.

[60] Matthew O’Toole, Felix Heide, Lei Xiao, Matthias B Hullin, Wolfgang Heidrich, and Kiriakos N Kutulakos. Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans. Graph. , 33(4):1–11, 2014.

[61] Matthew O’Toole, Felix Heide, David B Lindell, Kai Zang, Steven Diamond, and Gordon Wetzstein. Reconstructing transient images from single-photon sensors. In Proc. CVPR , 2017.

[62] Matthew O’Toole, David B Lindell, and Gordon Wetzstein. Confocal non-line-of-sight imaging based on the light-cone transform. Nature , 555(7696):338–341, 2018.

[63] Adithya Pediredla, Ashok Veeraraghavan, and Ioannis Gkioulekas. Ellipsoidal path connections for time-gated rendering. ACM Trans. Graph. , 38(4):1–12, 2019.

[64] Adithya Pediredla, Yasin Karimi Chalmiani, Matteo Giuseppe Scopelliti, Maysamreza Chamanzar, Srinivasa Narasimhan, and Ioannis Gkioulekas. Path tracing estimators for refractive radiative transfer. ACM Trans. Graph. , 39(6):1–15, 2020.

[65] Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physically based rendering: From theory to implementation . MIT Press, 2023.

[66] Ravi Ramamoorthi and Pat Hanrahan. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH , 2001.

[67] Andrea Ramazzina, Stefanie Walz, Pragyan Dahal, Mario Bijelic, and Felix Heide. Gated fields: Learning scene reconstruction from gated videos. In Proc. CVPR , 2024.

[68] Raskar Ramesh and James Davis. 5D time-light transport matrix: What can we reason about scene properties? Technical report, MIT, 2008.

[69] Joshua Rapp, Charles Saunders, Julián Tachella, John Murray-Bruce, Yoann Altmann, Jean-Yves Tourneret, Stephen McLaughlin, Robin MA Dawson, Franco NC Wong, and Vivek K Goyal. Seeing around corners with edge-resolved transient imaging. Nat. Commun. , 11(1):5929, 2020a.

[70] Joshua Rapp, Julian Tachella, Yoann Altmann, Stephen McLaughlin, and Vivek K Goyal. Advances in single-photon lidar for autonomous vehicles: Working principles, challenges, and recent advances. IEEE Signal Process. Mag. , 37(4):62–71, 2020b.

[71] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proc. CVPR , 2022.

[72] Diego Royo, Jorge García, Adolfo Muñoz, and Adrian Jarabo. Non-line-of-sight transient rendering. Comput. Graph. , 107:84–92, 2022.

[73] Diego Royo, Talha Sultan, Adolfo Muñoz, Khadijeh Masumnia-Bisheh, Eric Brandt, Diego Gutierrez, Andreas Velten, and Julio Marco. Virtual mirrors: Non-line-of-sight imaging beyond the third bounce. ACM Trans. Graph. , 42(4):1–15, 2023.

[74] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proc. CVPR , 2016.

[75] Brent Schwarz. Mapping the world in 3D. Nat. Photonics , 4(7):429–430, 2010.

[76] Sheila Seidel, Hoover Rueda-Chacón, Iris Cusini, Federica Villa, Franco Zappa, Christopher Yu, and Vivek K Goyal. Non-line-of-sight snapshots and background mapping with an active corner camera. Nat. Commun. , 14(1):3677, 2023.

[77] Aarrushi Shandilya, Benjamin Attal, Christian Richardt, James Tompkin, and Matthew O’Toole. Neural fields for structured lighting. In Proc. ICCV , 2023.

[78] Dongeek Shin, Ahmed Kirmani, Vivek K Goyal, and Jeffrey H Shapiro. Photon-efficient computational 3-d and reflectivity imaging with single-photon detectors. IEEE Trans. Comput. Imaging , 1(2):112–125, 2015.

[79] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In Proc. CVPR , 2021.

[80] Andrea Tagliasacchi and Ben Mildenhall. Volume rendering digest (for NeRF). arXiv preprint , 2022.

[81] Chia-Yin Tsai, Ashok Veeraraghavan, and Aswin C Sankaranarayanan. Shape and reflectance from two-bounce light transients. In Proc. ICCP , pages 1–10. IEEE, 2016.

[82] Chia-Yin Tsai, Aswin C Sankaranarayanan, and Ioannis Gkioulekas. Beyond volumetric albedo–a surface optimization framework for non-line-of-sight imaging. In Proc. CVPR , 2019.

[83] Eric Veach. Robust Monte Carlo methods for light transport simulation . Stanford University, 1998.

[84] Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. , 32(4):1–8, 2013.

[85] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. In Proc. CVPR , 2022.

[86] Dor Verbin, Ben Mildenhall, Peter Hedman, Jonathan T Barron, Todd Zickler, and Pratul P Srinivasan. Eclipse: Disambiguating illumination and materials using unintended shadows. In Proc. CVPR , 2024.

[87] Gregory J Ward, Francis M Rubinstein, and Robert D Clear. A ray tracing solution for diffuse interreflection. In Proc. SIGGRAPH , 1988.

[88] Di Wu, Andreas Velten, Matthew O’Toole, Belen Masia, Amit Agrawal, Qionghai Dai, and Ramesh Raskar. Decomposing global light transport using time of flight imaging. Int. J. Comput. Vis. , 107:123–138, 2014.

[89] Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjie Fan, and Xin Yu. Nefii: Inverse rendering for reflectance decomposition with near-field indirect illumination. In Proc. CVPR , pages 4295–4304, 2023.

[90] Jiaye Wu, Saeed Hadadan, Geng Lin, Matthias Zwicker, David Jacobs, and Roni Sengupta. GaNI: Global and near field illumination aware neural inverse rendering. arXiv , 2024.

[91] Lifan Wu, Guangyan Cai, Ravi Ramamoorthi, and Shuang Zhao. Differentiable time-gated rendering. ACM Trans. Graph. , 40(6):1–16, 2021.

[92] Shumian Xin, Sotiris Nousias, Kiriakos N Kutulakos, Aswin C Sankaranarayanan, Srinivasa G Narasimhan, and Ioannis Gkioulekas. A theory of Fermat paths for non-line-of-sight shape reconstruction. In Proc. CVPR , 2019.

[93] Yao Yao, Jingyang Zhang, Jingbo Liu, Yihang Qu, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan. NEILF: Neural incident light field for physically-based material estimation. In Proc. ECCV , 2022.

[94] Shinyoung Yi, Donggun Kim, Kiseok Choi, Adrian Jarabo, Diego Gutierrez, and Min H. Kim. Differentiable transient rendering. ACM Trans. Graph. , 40(6), 2021.

[95] Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In Proc. SIGGRAPH , 1999.

[96] Junge Zhang, Feihu Zhang, Shaochen Kuang, and Li Zhang. NeRF-lidar: Generating realistic lidar point clouds with neural radiance fields. In Proc. AAAI , 2024.

[97] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. CVPR , 2018.

[98] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Trans. Graph. , 40(6):1–18, 2021.

[99] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination for inverse rendering. In Proc. CVPR , 2022.

Appendix A Appendix Overview

附录 A 附录概览

We provide additional implementation details related to the architecture of our model, optimization procedure, and experimental settings. We also include supplemental results and details about the captured dataset. Code and data are available from our project webpage. Please refer to the webpage and video for animated visualizations of results, including lidar view synthesis, reconstructed geometry, time-resolved relighting, and separation of direct and indirect light.

我们提供与模型架构、优化流程与实验设置相关的额外实现细节，以及补充结果与实采数据集说明。代码与数据见项目页。请参阅网页与视频获取结果动画可视化，包括激光雷达视角合成、重建几何、时间分辨重光照以及直接/间接光分离。

Appendix B Implementation Details

附录 B 实现细节

B.1 Architecture Details

B.1 架构细节

Geometry.

We use Zip-NeRF’s [5] proposal sampling architecture to represent scene geometry and for volume rendering. Specifically, we use two hash-encoding-based “proposal” networks that output density, which is used for hierarchical sampling, and one final network that outputs the density used in Equation 4, as well as normals $\mathbf{n}$ used for the cache and physically-based rendering. The hash-encoding based proposal networks have spatial resolutions of 512 and 1024 along all axes, while the final density network has a resolution of 2048. Each network has a multi-layer perception (MLP) head with 2 layers and 64 hidden units.

几何。

我们用 Zip-NeRF [5] 的提案采样架构表示场景几何并做体渲染。具体使用两个基于哈希编码的「提案」网络输出密度用于层次采样，以及一个最终网络输出式 4 所用密度，以及缓存与基于物理渲染用法向 $\mathbf{n}$。提案网络空间分辨率各轴为 512 与 1024，最终密度网络为 2048。各网络带 2 层、64 隐单元的多层感知机（MLP）头。

We use 64 samples for the first proposal network, 64 samples for the second proposal network, and 32 samples for the final geometry network to volume render the cache geometry. In order to render the physically-based model, we leverage a single sample quadrature estimator for both primary and secondary rays, as in Attal et al. [4].

体渲染缓存几何时，第一提案网络用 64 个样本、第二提案网络 64 个、最终几何网络 32 个。渲染基于物理模型时，遵循 Attal et al. [4]，对主光线与次级光线使用单样本求积估计。

Cache.

The position-dependent appearance feature $\mathbf{f}^{\textit{app}}$ used for the cache has dimension 128 and is predicted with a hash encoding that has a spatial resolution of 2048. The learned BRDF for the direct component of the cache $f^{\textit{dir}}$ in Equation 11 is a sum of diffuse BRDF $f^{\textit{dir,diff}}(\mathbf{f}^{\textit{app}})$, and specular BRDF $f^{\textit{dir,spec}}\mathopen{}\left(\mathbf{f}^{\textit{app}},\mathbf{n},\boldsymbol{\omega}_{\ell},\boldsymbol{\omega}_{\mathrm{o}}^{\prime}\mathclose{}\right)$. We predict the diffuse BRDF as a function of $\mathbf{f}^{\textit{app}}$ alone, with a 2-layer, 64-hidden-unit MLP. We predict the specular BRDF as a function of $\mathbf{f}^{\textit{app}}$ as well as the dot product between the normal $\mathbf{n}$ and normalized half vector $\frac{\boldsymbol{\omega}_{\ell}+\boldsymbol{\omega}_{\mathrm{o}}^{\prime}}{||\boldsymbol{\omega}_{\ell}+\boldsymbol{\omega}_{\mathrm{o}}^{\prime}||}$ with a 2-layer, 64-hidden-unit MLP.

缓存。

缓存用的位置相关外观特征 $\mathbf{f}^{\textit{app}}$ 维数为 128，由空间分辨率 2048 的哈希编码预测。式 11 中缓存直接分量的学习 BRDF $f^{\textit{dir}}$ 为漫反射 BRDF $f^{\textit{dir,diff}}(\mathbf{f}^{\textit{app}})$ 与镜面 BRDF $f^{\textit{dir,spec}}$ 之和。漫反射 BRDF 仅作为 $\mathbf{f}^{\textit{app}}$ 的函数，用 2 层 64 隐单元 MLP 预测；镜面 BRDF 作为 $\mathbf{f}^{\textit{app}}$ 以及法向与归一化半角向量点积的函数，用同样规模 MLP 预测。

The specular indirect component of the cache, as described in Equation 12, uses a split-sum approximation. We predict $f_{\Omega}^{\textit{indir}}(\mathbf{f}^{\textit{app}},\mathbf{n},\boldsymbol{\omega}_{\mathrm{o}}^{\prime})$ as a function of the appearance feature and the dot product between normals $\mathbf{n}$ and outgoing direction $\boldsymbol{\omega}_{\mathrm{o}}^{\prime}$. We predict $L_{\text{i},\Omega}^{\textit{indir}}(\mathbf{f}^{\textit{app}},\mathbf{x}_{\ell},\mathbf{n},\boldsymbol{\omega}_{\mathrm{o}}^{\prime})$ as a function of the appearance feature, the reflected direction $\textit{reflect}(\boldsymbol{\omega}_{\mathrm{o}}^{\prime},\mathbf{n})$, and the light source position $\mathbf{x}_{\ell}$. Again, both use 2-layer, 64 hidden unit MLPs. We also predict a purely diffuse indirect component $L_{\mathrm{o}}^{\textit{indir},\textit{diff}}$ that is a function of the appearance feature and is conditioned on light source position, with a 2-layer 64-hidden-unit MLP.

式 12 所述缓存镜面间接分量使用 split-sum 近似。我们预测积分 BRDF 为外观特征与法向—出射方向点积的函数；预测积分入射辐射为外观特征、反射方向与光源位置的函数。二者均为 2 层 64 隐单元 MLP。我们还预测纯漫反射间接分量，作为外观特征函数并以光源位置为条件。

Materials.

We leverage the Disney–GGX [9] BRDF parameterization, with parameters albedo $\mathbf{a}(\mathbf{x})$, metalness $m(\mathbf{x})$, and roughness $r(\mathbf{x})$. This BRDF can be written as:

材质。

我们采用 Disney–GGX [9] BRDF 参数化，参数为反照率、金属度与粗糙度。该 BRDF 可写为：

$$\displaystyle f(\boldsymbol{\omega}_{\mathrm{i}},\boldsymbol{\omega}_{\mathrm{o}},\mathbf{x})=f_{\textit{diffuse}}(\mathbf{x})+f_{\textit{specular}}(\boldsymbol{\omega}_{\mathrm{i}},\boldsymbol{\omega}_{\mathrm{o}},\mathbf{x}) \\ \displaystyle f_{\textit{diffuse}}(\mathbf{x})=\frac{(1-m(\mathbf{x}))\mathbf{a}(\mathbf{x})}{\pi} \\ \displaystyle f_{\textit{specular}}(\boldsymbol{\omega}_{\mathrm{i}},\boldsymbol{\omega}_{\mathrm{o}},\mathbf{x})=\frac{DFG}{4(\mathbf{n}\cdot\boldsymbol{\omega}_{\mathrm{i}})(\mathbf{n}\cdot\boldsymbol{\omega}_{\mathrm{o}})} \tag{17}$$

We refer to Burley [9] and Liu et al. [42] for definitions of $(D,F,G)$. We use the Trowbridge-Reitz distribution function [65] for the normal distribution function $D$.

$(D,F,G)$ 的定义参见 Burley [9] 与 Liu et al. [42]。法向分布函数 $D$ 使用 Trowbridge-Reitz 分布 [65]。

We predict a material feature $\mathbf{f}_{mat}$ using a hash-encoding-based network with a resolution of 2048, and decode all of the above parameters using a linear layer from this feature.

我们用分辨率 2048 的基于哈希编码网络预测材质特征，再用线性层解码上述全部参数。

Importance sampling.

We leverage multiple importance sampling (MIS) [65], using the distribution function of the GGX BRDF, and a learned von Mises-Fisher-based importance sampler with an architecture similar to that of Attal et al. [4]. We supervise the importance sampler using the integrated intensity along secondary rays.

重要性采样。

我们采用多重重要性采样（MIS） [65]，使用 GGX BRDF 的分布函数，以及架构类似 Attal et al. [4] 的、基于可学习 von Mises–Fisher 的重要性采样器。用次级光线沿程积分强度监督该采样器。

Figure B.1: Rendered views and materials for simulated (rows 1–2) and captured scenes (rows 3–4). See the text for a detailed description.

图 B.1：仿真（第 1–2 行）与实采场景（第 3–4 行）的渲染视图与材质。详细说明见正文。

B.2 Loss Details

B.2 损失细节

Mask loss.

The mask loss for a particular ray is defined as:

掩码损失。

某条光线的掩码损失定义为：

$$\displaystyle\mathcal{L}_{\textit{mask}}=\textit{mask}\cdot\mathopen{}\left|1-\textit{acc}\mathclose{}\right|+\mathopen{}\left(1-\textit{mask}\mathclose{}\right)\cdot\mathopen{}\left|\textit{acc}\mathclose{}\right|, \tag{20}$$

where acc is the accumulated transmittance (or sum of the render weights) along a particular ray.

其中 acc 为该光线的累积透射率（或渲染权重之和）。

Predicted normal loss.

As discussed, we output the predicted normals using the density hash-encoding-based network. Similar to Ref-NeRF [85] and TensoIR [28], we constrain the predicted normals to match the negative gradient of the density field with an L2 loss:

预测法向损失。

如前所述，我们用基于密度哈希编码的网络输出预测法向。类似 Ref-NeRF [85] 与 TensoIR [28]，用 L2 损失约束预测法向匹配密度场负梯度：

$$\displaystyle\mathcal{L}_{\textit{normals}}=\sum_{k}w_{k}\mathopen{}\left\lVert\mathbf{n}_{k}^{\textit{pred}}-\mathbf{n}_{k}^{\textit{derived}}\mathclose{}\right\rVert^{2}, \tag{21}$$

where $w_{k}$ are the render weights for a given ray, and

其中 $w_{k}$ 为给定光线的渲染权重，且

$$\displaystyle\mathbf{n}_{k}^{\textit{derived}}=-\frac{\nabla\sigma(\mathbf{x}_{k})}{\lVert\nabla\sigma(\mathbf{x}_{k})\rVert}. \tag{22}$$

The loss weight $\lambda_{\textit{normals}}$ varies per-dataset.

损失权重 $\lambda_{\textit{normals}}$ 随数据集变化。

Material smoothness loss.

For the smoothness los $\mathcal{L}_{\textit{mat}}$, we leverage the implementation of TensoIR [28] for synthetic datasets, and a standard L2 smoothness loss for captured datasets.

材质光滑损失。

对光滑损失 $\mathcal{L}_{\textit{mat}}$，合成数据采用 TensoIR [28] 实现，实采数据采用标准 L2 光滑损失。

RawNeRF loss.

For the photometric losses (Equations 13 and 14 of the main paper), we use $\beta=1$ for synthetic scenes, $\beta=2$ for the cache in captured scenes, and $\beta=1$ for the physically-based model in captured scenes.

RawNeRF 损失。

对光度损失（正文式 13 与 14），合成场景取 $\beta=1$，实采场景缓存取 $\beta=2$，实采场景基于物理模型取 $\beta=1$。

Other loss hyperparameters.

For our photometric losses, we set $\lambda_{\textit{cache}}=10$, $\lambda_{\textit{dir}}=1$, $\lambda_{\textit{indir}}=1$. For the additional losses, we set $\lambda_{\textit{interlevel}}=0.01$ for all scenes. For the simulated scenes, we set $\lambda_{\textit{geom}}=0.0008$, $\lambda_{\textit{disortion}}=0.0001$, and $\lambda_{\textit{mask}}=0.1$. For the captured scenes, we set $\mathcal{L}_{\textit{geom}}=0.00025$ and $\lambda_{\textit{disortion}}=0.001$. We assume that the scene mask is all ones (i.e., all opaque) for captured scenes, and we set the mask loss to $\lambda_{\textit{mask}}=0.001$.

其他损失超参。

光度损失设 $\lambda_{\textit{cache}}=10$，$\lambda_{\textit{dir}}=1$，$\lambda_{\textit{indir}}=1$。额外损失中，所有场景 $\lambda_{\textit{interlevel}}=0.01$。仿真场景设 $\lambda_{\textit{geom}}=0.0008$，$\lambda_{\textit{disortion}}=0.0001$，$\lambda_{\textit{mask}}=0.1$。实采场景设几何光滑为 $0.00025$，畸变为 $0.001$。实采假定场景掩码全为 1（即全不透明），掩码损失 $\lambda_{\textit{mask}}=0.001$。

B.3 Time-Resolved Imaging Without Lidar

B.3 无激光雷达的时间分辨成像

Section 5.3 of the paper discusses how our model can recover time-resolved videos of propagating light by training on indirect time-of-flight or intensity images. In both cases, we write the loss as

正文第 5.3 节讨论如何在间接飞行时间或强度图像上训练，以恢复传播光时间分辨视频。两种情形下，损失写为

$$\displaystyle\mathcal{L}_{\textit{data}}=\sum_{\mathbf{o},\boldsymbol{\omega}_{\mathrm{o}}}\alpha(L_{\mathrm{i}}^{\textit{cache}})\sum_{k}\Big|\Big|\sum_{\tau}g_{k}(\tau)\mathopen{}\left(L_{\mathrm{i}}-L_{\text{i}}^{\textit{meas}}\mathclose{}\right)\Big|\Big|^{2}. \tag{23}$$

Here, $\{g_{k}(\cdot)\}_{k}$ defines a set of path length importance functions induced by the indirect time-of-flight or intensity sensor [3]. For indirect time-of-flight, we have:

此处 $\{g_{k}(\cdot)\}_{k}$ 定义由间接飞行时间或强度传感器诱导的一组路径长度重要性函数 [3]。对间接飞行时间有：

$$\displaystyle g_{k}(\tau)=\cos(2\pi f_{k}\cdot\tau+\theta_{k})+1, \tag{24}$$

where $f_{k}$ are frequencies and $\theta_{k}$ are phase shifts. We use $(f_{1},\theta_{1})=(30\times 10^{6},0)$, $(f_{2},\theta_{2})=(30\times 10^{6},\pi)$, $(f_{3},\theta_{3})=(170\times 10^{6},0)$, $(f_{4},\theta_{4})=(170\times 10^{6},\pi)$. For intensity images, we use $g_{1}=1$. We apply the same consistency loss as in Equation 15 of the main paper without adjustments.

其中 $f_{k}$ 为频率，$\theta_{k}$ 为相位。我们取 $(f_{1},\theta_{1})=(30\times 10^{6},0)$，$(f_{2},\theta_{2})=(30\times 10^{6},\pi)$，$(f_{3},\theta_{3})=(170\times 10^{6},0)$，$(f_{4},\theta_{4})=(170\times 10^{6},\pi)$。对强度图像取 $g_{1}=1$。一致性损失与正文式 15 相同，不做调整。

B.4 Finetuning for Relighting

B.4 重光照微调

As discussed in Section 5.3 of the paper, we leverage finetuning for relighting whenever the intensity profile of the light source differs from the training data (e.g. a projector as in Fig. 1 of the paper). In order to do this, we freeze all model parameters, apart from those that define the cache direct and indirect appearance (Equation 11 and Equation 12). We then train these parameters in order to minimize the radiometric prior (Equation 15).

如正文第 5.3 节所述，当光源强度轮廓与训练数据不同（如正文图 1 的投影机）时，我们用微调做重光照。做法是冻结除定义缓存直接与间接外观（式 11 与式 12）以外的全部模型参数，再训练这些参数以最小化辐射度量先验（式 15）。

Appendix C Additional Results

附录 C 补充结果

C.1 Material Decomposition

C.1 材质分解

In Fig. B.1, we show the recovered albedo, roughness, and metalness for simulated and captured scenes from a novel view. Qualitatively, the results align with expectations in several respects. The recovered albedo factors out variations in shading and illumination; the roughness is low/dark for specular objects (floor, ball, peppers in row 1; pot in row 2; chrome balls in row 3); and the metalness is bright/high for the pot in row 2 and chrome balls in row 3. Generally, the materials are harder to interpret for the captured results—though we expect that improvements to the system calibration would likely improve the results.

图 B.1 展示仿真与实采场景在新视角下恢复的反照率、粗糙度与金属度。定性上多方面符合预期：恢复的反照率剔出明暗与光照变化；镜面物体粗糙度低/暗；金属物体金属度亮/高。实采结果材质一般更难解释——我们预期改进系统标定可能改善结果。

We note that for Fig. B.1, we leverage an additional loss applied to the integrated time-resolved measurements — specifically the loss in Equation 23 for intensity images. We find that this slightly improves the convergence of the recovered materials.

对图 B.1，我们对积分时间分辨测量额外施加损失——具体即强度图像情形下的式 23。发现这略微改善恢复材质的收敛。

C.2 Additional Baselines

C.2 额外基线

We include another T-NeRF [47] baseline, which applies a matched filter to the time-resolved measurement to find the direct peak—similar to a conventional lidar—before supervision. We include this result in Table C.1 (see T-NeRF w/ filtering).

我们另加入一个 T-NeRF [47] 基线：在监督前对时间分辨测量做匹配滤波找直接峰——类似传统激光雷达。结果见表 C.1（见 T-NeRF w/ filtering）。

The baseline improves upon T-NeRF and even outperforms FWP++ for geometry modeling. This is expected since one of the main reasons T-NeRF fails in geometry recovery is the presence of the indirect component of light in the lidar scans. However, our method still outperforms this new baseline since the matched filter does not always accurately localize the time of the direct surface reflection, especially under strong indirect light. The new baseline also struggles with novel view synthesis since it does not model indirect light transport effects.

该基线优于 T-NeRF，甚至在几何建模上超过 FWP++。这在预期之中，因 T-NeRF 几何失败的主因之一正是激光雷达扫描中的间接光分量。但我们的方法仍优于该新基线，因匹配滤波在强间接光下并不总能准确定位直接表面反射时刻。新基线也不建模间接光传输，新视角合成仍吃力。

	method	PSNR (dB)$\,\uparrow$	LPIPS$\,\downarrow$	SSIM$\,\uparrow$	MAE$\,\downarrow$	L1 depth$\,\downarrow$	T-IOU$\,\uparrow$
sim	T-NeRF [47]	22.44	0.40	0.71	28.00	0.59	0.58
T-NeRF w/ filtering	24.52	0.34	0.78	22.54	0.40	0.70
FWP++ [48]	29.00	0.30	0.87	22.80	0.47	0.73
	ours	30.99	0.31	0.89	8.45	0.21	0.76

Table C.1: Evaluation of lidar rendering from novel viewpoints and geometry recovery.

	方法	PSNR (dB)$\,\uparrow$	LPIPS$\,\downarrow$	SSIM$\,\uparrow$	MAE$\,\downarrow$	L1 depth$\,\downarrow$	T-IOU$\,\uparrow$
sim	T-NeRF [47]	22.44	0.40	0.71	28.00	0.59	0.58
	T-NeRF w/ filtering	24.52	0.34	0.78	22.54	0.40	0.70
	FWP++ [48]	29.00	0.30	0.87	22.80	0.47	0.73
	ours	30.99	0.31	0.89	8.45	0.21	0.76

表 C.1：新视角激光雷达渲染与几何恢复评测。

C.3 Quantitative Results

C.3 定量结果

We provide a per-scene breakdown of quantitative results for simulated scenes in Table C.2 and captured scenes in Table C.3. We see similar trends for all scenes as described in the main text.

表 C.2 与表 C.3 分别给出仿真与实采场景的分场景定量结果。各场景趋势与正文描述一致。

C.4 Qualitative Results

C.4 定性结果

We provide additional qualitative results on novel views in Figure C.2 and in the supplemental web page, which includes novel view flythroughs, time-resolved relighting, and separation of direct and indirect light. We emphasize that our method recovers more accurate geometry, particularly in scenarios involving strong indirect lighting from specular reflections or diffuse inter-reflections, outperforming previous approaches.

图 C.2 与补充网页提供更多新视角定性结果，含新视角漫游、时间分辨重光照以及直接/间接光分离。我们强调：在镜面反射或漫反射互反射造成强间接光照的情形下，方法恢复更准确几何，优于先前方法。

		Pots	Cornell	Peppers	Kitchen
PSNR	T-NeRF	23.78	23.90	19.07	23.00
FWP	28.64	31.75	33.01	22.61
ours	30.44	32.38	37.46	23.68
LPIPS	T-NeRF	0.36	0.32	0.44	0.49
FWP	0.26	0.30	0.26	0.39
ours	0.35	0.31	0.27	0.30
SSIM	T-NeRF	0.73	0.82	0.72	0.56
FWP	0.86	0.87	0.94	0.79
ours	0.90	0.89	0.93	0.84
MAE	T-NeRF	36.09	18.33	13.03	44.56
FWP	37.41	10.86	7.20	35.75
ours	7.81	10.25	2.65	13.08
L1	T-NeRF	0.18	0.10	0.42	1.66
FWP	0.29	0.10	0.28	1.20
ours	0.04	0.09	0.19	0.53
T-IOU	T-NeRF	0.66	0.69	0.76	0.20
FWP	0.82	0.82	0.88	0.41
ours	0.88	0.78	0.94	0.46

Table C.2: Breakdown of results on the simulated scenes for PSNR, LPIPS, SSIM, MAE, L1 Depth (L1) and Transient IOU (T-IOU).

		Pots	Cornell	Peppers	Kitchen
PSNR	T-NeRF	23.78	23.90	19.07	23.00
	FWP	28.64	31.75	33.01	22.61
	ours	30.44	32.38	37.46	23.68
LPIPS	T-NeRF	0.36	0.32	0.44	0.49
	FWP	0.26	0.30	0.26	0.39
	ours	0.35	0.31	0.27	0.30
SSIM	T-NeRF	0.73	0.82	0.72	0.56
	FWP	0.86	0.87	0.94	0.79
	ours	0.90	0.89	0.93	0.84
MAE	T-NeRF	36.09	18.33	13.03	44.56
	FWP	37.41	10.86	7.20	35.75
	ours	7.81	10.25	2.65	13.08
L1	T-NeRF	0.18	0.10	0.42	1.66
	FWP	0.29	0.10	0.28	1.20
	ours	0.04	0.09	0.19	0.53
T-IOU	T-NeRF	0.66	0.69	0.76	0.20
	FWP	0.82	0.82	0.88	0.41
	ours	0.88	0.78	0.94	0.46

表 C.2：仿真场景上 PSNR、LPIPS、SSIM、MAE、L1 Depth（L1）与 Transient IOU（T-IOU）分项结果。

		House	Globe	Spheres	Statue
PSNR	T-NeRF	15.94	11.44	13.25	18.05
FWP	27.40	26.00	28.51	31.89
ours	27.47	25.97	26.07	30.04
LPIPS	T-NeRF	0.46	0.56	0.53	0.58
FWP	0.30	0.34	0.38	0.26
ours	0.32	0.34	0.39	0.28
SSIM	T-NeRF	0.36	0.19	0.35	0.51
FWP	0.78	0.75	0.81	0.92
ours	0.79	0.75	0.75	0.90
T-IOU	T-NeRF	0.34	0.10	0.13	0.34
FWP	0.62	0.54	0.43	0.60
ours	0.60	0.53	0.44	0.60

Table C.3: Breakdown of results on the captured scenes for PSNR, LPIPS, SSIM, MAE and Transient IOU (T-IOU).

		House	Globe	Spheres	Statue
PSNR	T-NeRF	15.94	11.44	13.25	18.05
	FWP	27.40	26.00	28.51	31.89
	ours	27.47	25.97	26.07	30.04
LPIPS	T-NeRF	0.46	0.56	0.53	0.58
	FWP	0.30	0.34	0.38	0.26
	ours	0.32	0.34	0.39	0.28
SSIM	T-NeRF	0.36	0.19	0.35	0.51
	FWP	0.78	0.75	0.81	0.92
	ours	0.79	0.75	0.75	0.90
T-IOU	T-NeRF	0.34	0.10	0.13	0.34
	FWP	0.62	0.54	0.43	0.60
	ours	0.60	0.53	0.44	0.60

表 C.3：实采场景上 PSNR、LPIPS、SSIM 与 Transient IOU（T-IOU）分项结果。

Figure C.1: Additional captured results comparing reconstructed normals from the proposed method to those of T-NeRF [47] and FWP++ [48].

图 C.1：更多实采结果，比较所提方法与 T-NeRF [47]、FWP++ [48] 重建的法向。

Figure C.2: Additional simulated results comparing rendered novel views and reconstructed normals from the proposed method to those of T-NeRF [47] and FWP++ [48].

图 C.2：更多仿真结果，比较渲染新视角与重建法向。

Appendix D Dataset

附录 D 数据集

D.1 Calibration

D.1 标定

To capture our real multi-viewpoint dataset, we use a hardware setup similar to the one used by Malik et al. [48], with a 532 nm laser emitting 35 ps pulses at a 10 MHz synced with a single pixel scanning SPAD at 512×512 resolution. We capture multiple viewpoints with the same rotation table and elevation arm setup. Specifically, our light source position is fixed for all viewpoints with respect to the camera rather than to the scene. Camera intrinsics are calibrated with a checkerboard and the MATLAB Camera Calibration Toolbox [49], and extrinsics are calibrated using COLMAP [74] with a scene including a checkerboard so that radial camera pose translation can be scaled by matching the reconstruction to the board’s known geometry.

为采集真实多视角数据集，我们使用与 Malik et al. [48] 类似的硬件：532 nm 激光以 10 MHz 发射 35 ps 脉冲，与分辨率 512×512 的单像素扫描 SPAD 同步。我们用同一旋转台与俯仰臂采集多视角，但光源随传感器移动。

For our scenes, we assume our light sources are point sources, calibrated so that their location is known with respect to the scene. We simulate point light sources by passing our free-space laser light, coupled through multi-mode fiber, through a collimating lens, and multiple high-power diffusers. To address any residual imperfections in our non-ideal point source, we image a uniformly reflective, diffuse surface with a pre-calibrated pose, using a checkerboard pattern for alignment. This process enables us to compute a directional intensity profile for the light source, which we model during inverse rendering.

对各场景，我们假定光源为点源，并标定其相对场景的位置。将自由空间激光经多模光纤、准直透镜与多个高功率散射片，以仿真点光源洪泛照明。

The light source position is calibrated using the following procedure. We (1) capture a checkerboard and compute corner poses, (2) use the corresponding time-resolved measurement for each corner to measure total ToF and thus distance from the light source to the camera, (3) subtract the calibrated corner pose to camera distance, and (4) trilaterate to locate the unknown light source position.

光源位置标定步骤如下：(1) 拍摄棋盘格并计算角点位姿；(2) 用各角点对应时间分辨测量得到总飞行时间，从而得到光源到相机的距离；(3) 减去已标定的相机到角点距离，得到光源到角点距离；(4) 用多点距离约束三角化光源位置。

D.2 Scene Descriptions

D.2 场景描述

We provide a description of each captured scene in Table D.1.

表 D.1 给出各实采场景的描述。

Scene Description	Description	Training Views	Test Views	Azimuth Span	Normalization Scale
House	A diffused pulsed laser source rotates with the lidar sensor and illuminates a ceramic house, mushroom, and pumpkin with a plate in the background.	81	13	240°	600
Globe	A diffused pulsed laser source rotates with the lidar sensor and illuminates a globe and a lightbulb. Our model reconstructs the fine details of the wires of the lightbulb stand.	55	11	132°	600
Spheres	A diffused pulsed laser source rotates with the lidar sensor and illuminates two specular spheres.	56	11	132°	600
Statue	From the Flying with Photons Dataset [48]: a stationary diffused pulsed laser source illuminates a statue of David and two candles from the side.	60	15	150°	600

Table D.1: Descriptions of the captured scenes. All scenes have a calibrated bin width of 0.0105 m and span 15 degrees in elevation angle.

场景	描述	训练视角	测试视角	方位跨度	归一化尺度
House	散射脉冲激光源随激光雷达传感器旋转，照亮陶瓷房屋、蘑菇与南瓜，背景有盘子。	81	13	240°	600
Globe	散射脉冲激光源随激光雷达传感器旋转，照亮地球仪与周围物体。	81	13	240°	600
Spheres	散射脉冲激光源随激光雷达传感器旋转，照亮镜面与漫反射球体。	81	13	240°	600
Statue	固定散射脉冲激光源照亮雕像与蜡烛（来自 Malik et al.）。	—	—	—	—

表 D.1：实采场景描述。各场景均有标定的双臂几何与已知光源位置。