Score Matching with Missing Data

Josh Givens Affiliation:  School of Mathematics, University of Bristol, Bristol, UK Correspondence to: josh.givens@bristol.ac.uk    Song Liu Affiliation:  School of Mathematics, University of Bristol, Bristol, UK    Henry W J Reeve Affiliation:  School of Artificial Intelligence, Nanjing University, China

Abstract

摘要

Score matching is a vital tool for learning the distribution of data with applications across many areas including diffusion processes, energy based modelling, and graphical model estimation. Despite all these applications, little work explores its use when data is incomplete. We address this by adapting score matching (and its major extensions) to work with missing data in a flexible setting where data can be partially missing over any subset of the coordinates. We provide two separate score matching variations for general use, an importance weighting (IW) approach, and a variational approach. We provide finite sample bounds for our IW approach in finite domain settings and show it to have especially strong performance in small sample lower dimensional cases. Complementing this, we show our variational approach to be strongest in more complex high-dimensional settings which we demonstrate on graphical model estimation tasks on both real and simulated data.

Score matching 是学习数据分布的重要工具，已广泛用于扩散过程、基于能量的建模以及图模型估计等领域。尽管应用很多，在数据不完整时如何使用它却少有研究。我们针对坐标任意子集均可部分缺失的灵活设定，把 score matching（及其主要扩展）改造成能处理缺失数据的形式。我们给出两种通用变体：重要性加权（IW）方法与变分方法。对 IW 方法，我们在有限支撑设定下给出有限样本界，并表明它在小样本、较低维情形表现尤为突出；与之互补，变分方法在更复杂的高维设定更强，我们在真实与模拟数据上的图模型估计任务中验证了这一点。

1 Introduction

1 引言

Keywords:

Over the last decade, score matching has established itself as a powerful tool with downstream use in many areas of machine learning. Examples include: energy based modelling (Swersky et al., 2011; Bao et al., 2020; Li et al., 2019b), mode-seeking clustering (Sasaki et al., 2014), and perhaps most prominently of all Diffusion processes (Song & Ermon, 2019; Song et al., 2021b; Tashiro et al., 2021; Song et al., 2021a; Huang et al., 2021). Score matching aims to learn the score of a distribution which is the gradient of the log of the probability density function (PDF) ($\bm{s}(x)=\nabla_{\bm{x}}\log p(\bm{x})$). In contrast to modelling the density directly, the score does not need to integrate to one meaning there is no need to calculate a normalising constant. This allows it to be much more more flexibly modelled than the density itself. Furthermore, the validity of the score matching objective itself requires only very mild assumptions of the family of proposed scores further ensuring this flexibility. Alongside the classical method (Hyvärinen, 2005), various adaptations of score matching have arisen to improve performance, decrease computational cost, and extend the approach to a wider range of settings (Hyvärinen, 2007; Vincent, 2011; Song et al., 2020; Liu et al., 2022).

关键词：

过去十年，score matching 已成为机器学习诸多下游任务的有力工具。例子包括：基于能量的建模（Swersky et al., 2011；Bao et al., 2020；Li et al., 2019b）、寻模聚类（Sasaki et al., 2014），以及最为突出的扩散过程（Song & Ermon, 2019；Song et al., 2021b；Tashiro et al., 2021；Song et al., 2021a；Huang et al., 2021）。Score matching 的目标是学习分布的 score，即概率密度函数（PDF）对数的梯度（$\bm{s}(x)=\nabla_{\bm{x}}\log p(\bm{x})$）。与直接建模密度不同，score 不必积分为一，因而无需计算归一化常数，建模比密度本身更灵活。此外，score matching 目标本身对提议 score 族只需很弱的假设，进一步保证了这种灵活性。除经典方法（Hyvärinen, 2005）外，已有多种改编用于提升性能、降低计算量，并把方法拓展到更广设定（Hyvärinen, 2007；Vincent, 2011；Song et al., 2020；Liu et al., 2022）。

In this work, we extend the score matching framework to handle missing data at training time. Specifically, we learn the full score function from partially missing multidimensional input data, a paradigm we term missing score matching. Crucially, our approach is compatible with any parameterised score model, enabling its application to both explicit score formulations and more general approaches such as neural networks (NNs). We propose two methods to adapt the original score matching method as well as its popular adaptations, truncated, sliced, and denoising score matching (Hyvärinen, 2007; Vincent, 2011; Liu & Wang, 2017; Song et al., 2020). These two distinct but closely related methods complement each other allowing for a wide range of problems to be tackled. The first method is a simpler importance weighting (IW) approach which we refer to as marginal IW score matching. For this method we obtain finite sample bounds in the bounded domain setting under certain conditions. We also provide experimental results demonstrating its efficacy in lower dimensional settings and where less data is available. Our second approach is a more computationally sophisticated variational approach which we refer to as marginal variational score matching. We demonstrate the efficacy of this approach in more complex, high dimensional settings by applying it to the problem of graphical model estimation with both real and synthetic datasets.

本文把 score matching 框架扩展到训练期存在缺失数据的情形。具体地，我们从部分缺失的多维输入中学习完整 score 函数，并称之为 缺失 score matching。关键是：方法兼容任意参数化 score 模型，既可用于显式 score 形式，也可用于神经网络（NN）等更一般做法。我们提出两种方法，用以改造原始 score matching 及其流行变体——截断、切片与去噪 score matching（Hyvärinen, 2007；Vincent, 2011；Liu & Wang, 2017；Song et al., 2020）。这两种彼此相关却不同的方法互为补充，可覆盖广泛问题。第一种较简单，是重要性加权（IW）方法，称为边际 IW score matching；在有界支撑设定下我们给出有限样本界，并在较低维、数据更少时用实验展示其效力。第二种计算上更精细，是变分方法，称为边际变分 score matching；我们把它用于真实与合成数据上的图模型估计，展示其在更复杂高维设定中的表现。

In section 2 we discuss relevant works for score matching and related fields. In section 3 we will introduce our problem more formally including score matching and any notation used. Section 4 will be used to introduce our methods. Section 5 will present results on some real and simulated datasets. In Section 6 we give our conclusion.

第 2 节讨论 score matching 及相关领域的已有工作。第 3 节更形式地引入问题设定、score matching 与所用记号。第 4 节介绍我们的方法。第 5 节给出真实与模拟数据上的结果。第 6 节总结。

2 Related Works

2 相关工作

While there has been some work which utilises score matching with missing data, these approaches mostly do so exclusively through the lens of diffusion models. Specifically works such as MissDiff (Ouyang et al., 2023) and Ambient Diffusion (Daras et al., 2023) require the score function itself to take the form of a neural network (NN) which learns the scores of the fully-observed and corrupted scores simultaneously. This prohibits their use in situations where our model for the score is some explicit parameterisation whose parameters we want to learn as is the case in settings such as energy based modelling Li et al. (2023); Bao et al. (2020); Salimans & Ho (2021) and Gaussian graphical models (Lin et al., 2016; Yu et al., 2018). Ambient Diffusion also requires the data to be further artificially corrupted in order to create a pseudo-supervised learning paradigm making both Ambient Diffusion and MissDiff subject to various levels of out of sample learning without specific adjustments for this phenomenon.

已有一些工作在缺失数据下使用 score matching，但大多只从扩散模型视角切入。例如 MissDiff（Ouyang et al., 2023）与 Ambient Diffusion（Daras et al., 2023）要求 score 本身是神经网络，同时学习完全观测与损坏观测的 score。这使其无法用于我们希望学习显式参数化 score 的场景，例如基于能量的建模（Li et al., 2023；Bao et al., 2020；Salimans & Ho, 2021）以及高斯图模型（Lin et al., 2016；Yu et al., 2018）。Ambient Diffusion 还要求对数据再施加人工损坏以构造伪监督，从而使 Ambient Diffusion 与 MissDiff 在不同程度上依赖样本外学习，除非为此做专门调整。

Looking more generally at distribution estimation with missing data, multiple works in the field of generative modelling have looked to tackle the problem of providing a generative model for a distribution given corrupted samples from it. Prominent among these are MisGAN (Li et al., 2019a), which presents a marginalised GAN framework and MCFlow (Richardson et al., 2020) , which presents a EM like normalising flow framework. Neither of these approaches allow for flexible specification of a parametric density estimate however with MCFlow requiring the density to be a normalising flow and MisGAN having no model for the density whatsoever.

更一般地看缺失数据下的分布估计，生成建模领域已有多篇工作试图在只有损坏样本时给出生成模型。突出者包括 MisGAN（Li et al., 2019a）的边际化 GAN 框架，以及 MCFlow（Richardson et al., 2020）这类类 EM 的归一化流框架。二者都难以灵活指定参数化密度估计：MCFlow 要求密度是归一化流，MisGAN 则根本没有密度模型。

To our knowledge, the only approach which seems to adapt score matching to missing data in a parameter preserving manner is presented in (Uehara et al., 2020) using an iterative EM-like procedure. However they themselves admit that there is little intuitive understanding of when this approach will converge. Additionally, due to the nature of the score matching objective, the expectation step cannot be directly approximated using Monte Carlo estimation and instead requires fractional importance weighting, a method which employs nested Monte Carlo estimates introducing bias into the training objective.

据我们所知，以保持参数形式的方式把 score matching 适配到缺失数据的，似乎只有 Uehara et al.（2020）的迭代类 EM 流程。但作者也承认，对该流程何时收敛缺少直观理解。此外，由于 score matching 目标的性质，期望步无法直接用蒙特卡洛近似，而需要分数重要性加权；该方法使用嵌套蒙特卡洛估计，会给训练目标引入偏差。

Parallel to this, some papers have looked to extend score matching to the latent variable setting, an area with much commonality to missing data (Vértes & Sahani, 2016; Bao et al., 2020; Bao et al., 2021). Latent variable modelling differs in two crucial aspects from missing score matching. Firstly the components which are unobserved (the latent variables) remain constant between samples, and secondly there is not necessarily a notion of a ground truth for the unobserved components in when data is corrupted. Additionally each of these works has limitations; Vértes & Sahani (2016) only applies to exponential families, Bao et al. (2020) requires a gradient unrolling step in its optimisation which is computationally expensive and can lead to errors in the optimisation procedure (as acknowledged in their follow on work), and Bao et al. (2021) is only given for denoising score matching, not for classical or sliced score matching.

与此并行，一些工作把 score matching 扩展到隐变量设定，这与缺失数据有许多共通之处（Vértes & Sahani, 2016；Bao et al., 2020；Bao et al., 2021）。隐变量建模与缺失 score matching 有两点关键不同：其一，未观测分量（隐变量）在各样本间保持同一角色；其二，数据损坏时，未观测分量未必存在可对照的真值。这些工作也各有局限：Vértes & Sahani（2016）只适用于指数族；Bao et al.（2020）在优化中需要梯度展开，计算昂贵且可能导致优化误差（其后续工作也承认这一点）；Bao et al.（2021）只给出了去噪 score matching，未覆盖经典或切片 score matching。

3 Setting

3 设定

3.1 Notation

3.1 记号

For $n\in\mathbb{N}$ let $[n]\coloneqq\{1,\dotsc,n\}$. For a random variable $Z$ we use $\supp(Z)$ for the support of $Z$. For $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ we write $\partial_{j}f(\bm{x})\coloneqq\frac{\partial f}{\partial x_{j}}$ where $\bm{x}=\left(x_{1},\dotsc,x_{d}\right)^{\top}$ and $\nabla_{\bm{x}}f(\bm{x})\coloneqq\left(\partial_{j}f(\bm{x}),\dotsc,\partial_{d}f(\bm{x})\right)^{\top}\!\!,$ the gradient of $f$. For $\bm{f}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ take $\bm{f}(\bm{x})_{j}$ as the $j$th component of $\bm{f}(\bm{x})$ and write $\nabla_{\bm{x}}\cdot\bm{f}(\bm{x})\coloneqq\partial_{1}\bm{f}(\bm{x})_{1}+\cdots+\partial_{d}\bm{f}(\bm{x})_{d}$. Finally for $\bm{a},\bm{b}\in\mathbb{R}^{d}$, take $\bm{a}\circ\bm{b}$ to be the Hadamard product.

对 $n\in\mathbb{N}$，令 $[n]\coloneqq\{1,\dotsc,n\}$。对随机变量 $Z$，用 $\supp(Z)$ 表示其支撑。对 $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$，记 $\partial_{j}f(\bm{x})\coloneqq\frac{\partial f}{\partial x_{j}}$，其中 $\bm{x}=\left(x_{1},\dotsc,x_{d}\right)^{\top}$，并记 $\nabla_{\bm{x}}f(\bm{x})\coloneqq\left(\partial_{j}f(\bm{x}),\dotsc,\partial_{d}f(\bm{x})\right)^{\top}\!\!,$ 为 $f$ 的梯度。对 $\bm{f}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$，取 $\bm{f}(\bm{x})_{j}$ 为 $\bm{f}(\bm{x})$ 的第 $j$ 个分量，并记 $\nabla_{\bm{x}}\cdot\bm{f}(\bm{x})\coloneqq\partial_{1}\bm{f}(\bm{x})_{1}+\cdots+\partial_{d}\bm{f}(\bm{x})_{d}$。最后，对 $\bm{a},\bm{b}\in\mathbb{R}^{d}$，取 $\bm{a}\circ\bm{b}$ 为 Hadamard 积。

We now introduce some indexing notation which we will be using for RVs and functions throughout. This will prove useful when identifying the missing non-missing components of our data. Let $Z$ be a random variable taking values in $\mathbb{R}^{d}$. We use $Z_{j}$ to refer to the $j$th component $Z$ and for $\lambda\subseteq[d]$ take $Z_{\lambda}=\{Z_{j}\}_{j\in\lambda}$. We use negation in indexing to mean the complementing coordinates. More precisely we let $-j$ denote $[d]\setminus\{j\}$ and let $-\lambda$ denote $[d]\setminus\lambda$. We typically use $Z^{(i)}$ to denote an independent copy of $Z$. For a function $f:\mathcal{X}\rightarrow\mathcal{Y}$ and $\bm{x}_{\lambda}\in\mathcal{X}_{\lambda},\bm{x}^{\prime}_{-\lambda}\in\mathcal{X}_{-\lambda}$, we take $f(\bm{x}_{\lambda},\bm{x}^{\prime}_{-\lambda})$ to be $f(\bm{z})$ where

下面引入贯穿全文的随机变量与函数的指标记号，便于区分数据中缺失与未缺失的分量。设 $Z$ 为取值于 $\mathbb{R}^{d}$ 的随机变量。用 $Z_{j}$ 表示 $Z$ 的第 $j$ 个分量；对 $\lambda\subseteq[d]$，取 $Z_{\lambda}=\{Z_{j}\}_{j\in\lambda}$。指标上的取反表示互补坐标：更精确地，令 $-j$ 表示 $[d]\setminus\{j\}$，令 $-\lambda$ 表示 $[d]\setminus\lambda$。通常用 $Z^{(i)}$ 表示 $Z$ 的独立副本。对函数 $f:\mathcal{X}\rightarrow\mathcal{Y}$ 以及 $\bm{x}_{\lambda}\in\mathcal{X}_{\lambda},\bm{x}^{\prime}_{-\lambda}\in\mathcal{X}_{-\lambda}$，取 $f(\bm{x}_{\lambda},\bm{x}^{\prime}_{-\lambda})$ 为 $f(\bm{z})$，其中

$$\displaystyle z_{j}\coloneqq\begin{cases}x_{j}\quad\text{if }j\in\lambda\
x^{\prime}_{j}\quad\text{if }j\in-\lambda\end{cases}.$$

We will take $X$ to be a RV taking values in $\mathcal{X}\subseteq\mathbb{R}^{d}$ representing our original dataset and $X^{\prime}$ to be a RV representing some generative/variational/importance weighting distribution. i.e., the “artificial distributions” we will utilise in our method. Similarly, we take $\mathbb{E},\mathbb{E}^{\prime}$ to be expectations with respect to (w.r.t.) $X,X^{\prime}$ respectively.

我们取 $X$ 为取值于 $\mathcal{X}\subseteq\mathbb{R}^{d}$ 的随机变量，代表原始数据；$X^{\prime}$ 为代表某种生成／变分／重要性加权分布的随机变量，即方法中用到的「人工分布」。类似地，分别用 $\mathbb{E},\mathbb{E}^{\prime}$ 表示关于 $X,X^{\prime}$ 的期望。

Throughout we take $p$ to be the pdf of the RV, $X$, and $p_{\theta}$ to be a model therein. We let $q$ represent an unnormalised density (i.e. $N^{-1}\cdot q=p$ for some normalising constant $N>0$.) We will write marginalisations/conditionings for both true and model densities implicitly with $p(\bm{x}_{\lambda})\coloneqq\int_{\mathcal{X}}p(\bm{x})\mathrm{d}\bm{x}_{-\lambda}$ and $p(\bm{x}_{\lambda}|\bm{x}_{-}\lambda)$ being the conditional density of $X_{\lambda}|X_{-\lambda}=\bm{x}_{-\lambda}$ for example.

全文用 $p$ 表示随机变量 $X$ 的 pdf，$p_{\theta}$ 表示相应模型。用 $q$ 表示未归一化密度（即存在归一化常数 $N>0$ 使得 $N^{-1}\cdot q=p$）。对真密度与模型密度，边际／条件都用隐式写法，例如 $p(\bm{x}_{\lambda})\coloneqq\int_{\mathcal{X}}p(\bm{x})\mathrm{d}\bm{x}_{-\lambda}$，以及 $p(\bm{x}_{\lambda}|\bm{x}_{-\lambda})$ 表示 $X_{\lambda}|X_{-\lambda}=\bm{x}_{-\lambda}$ 的条件密度。

Now that we have introduced our notation we can move onto the key area of focus for our work, score matching.

记号就绪后，我们转入本文的核心：score matching。

3.2 Score Matching

First proposed by (Hyvärinen, 2005), score matching aims to learn the gradient of the log-density (score). The advantage of this framework over full density approaches such as maximum likelihood estimation (MLE) is that we are not restricted to parametric models which integrate to $1$. This allows us to be much more flexible in how we parameterise in turn making high dimensional distribution modelling more feasible. We now introduce the approach.

Score matching 最早由 Hyvärinen（2005）提出，目标是学习对数密度的梯度（score）。相对最大似然估计（MLE）等完整密度方法，其优势在于不必把参数模型限制为积分为 $1$ 的族，从而参数化更灵活，高维分布建模也更可行。下面介绍该方法。

Let $X$ be a RV over $\mathbb{R}^{d}$ with PDF $p$. We say that $q$ is the unnormalised density of $X$ if $N^{-1}\cdot q(\bm{x})=p(\bm{x})$ where $p$ is the PDF of $X$ and $N$ is the normalising constant of $q$. Define the score, of $X$ to be

设 $X$ 为 $\mathbb{R}^{d}$ 上的随机变量，pdf 为 $p$。若 $N^{-1}\cdot q(\bm{x})=p(\bm{x})$（$N$ 为 $q$ 的归一化常数），则称 $q$ 为 $X$ 的未归一化密度。定义 $X$ 的 score 为

$$\displaystyle\bm{s}(\bm{x})\coloneqq\nabla_{\bm{x}}\log p(\bm{x})=\nabla_{\bm{x}}\log q(x).$$

The aim of score matching is to learn $\bm{s}$ from a collection of IID copies of $X$ which we denote $\mathcal{D}\coloneqq\{X^{(i)}\}_{i=1}^{n}$. Following Hyvärinen (2005), we introduce a generic parameterised proposal score $\bm{s}_{\theta}$ for $\theta\in\Theta\subseteq\mathbb{R}^{p}$ and aim to minimise the Fisher Divergence between the true distribution and our proposal distribution which is given by

Score matching 的目标是从 $X$ 的 IID 样本集合 $\mathcal{D}\coloneqq\{X^{(i)}\}_{i=1}^{n}$ 中学习 $\bm{s}$。依 Hyvärinen（2005），引入泛型参数化提议 score $\bm{s}_{\theta}$（$\theta\in\Theta\subseteq\mathbb{R}^{p}$），并最小化真分布与提议分布之间的 Fisher 散度，其定义为

$$\displaystyle F(\theta)\coloneqq\mathbb{E}[\|\bm{s}(X)-\bm{s}_{\theta}(X)\|^{2}].$$

The key result from Hyvärinen (2005) which enables us to practically implement score matching is that under certain (fairly minimal) regularity conditions, which we provide in Appendix D.1, we have

Hyvärinen（2005）的关键结果是：在某些（相当弱的）正则条件下——详见附录 D.1——有

$$\displaystyle L(\theta) \\ \displaystyle\coloneqq\mathbb{E}\left[2\nabla_{X}\cdot\bm{s}_{\theta}(X)+\|\bm{s}_{\theta}(X)\|^{2}\right]=F(\theta)-C \tag{1}$$

where here and throughout, we take $C$ to represent any constant which does not depend upon $\theta$. Crucially, $L(\theta)$ is now an expectation of observable random variables. Hence we can now approximate this with our data and take $\hat{\theta}$ as

此处及全文，$C$ 表示不依赖于 $\theta$ 的任意常数。关键是，$L(\theta)$ 已是可观测量的期望，因而可用数据近似，并取 $\hat{\theta}$ 为

$$\displaystyle\hat{\theta}\coloneqq\argmin_{\theta}\frac{1}{n}\sum_{i=1}^{n}\left[2\nabla_{X^{(i)}}\cdot\bm{s}_{\theta}(X^{(i)})+\|\bm{s}_{\theta}(X^{(i)})\|^{2}\right].$$

Truncated Score Matching

A limitation of standard score matching is that it requires $\lim_{x_{i}\rightarrow\infty}p(\bm{x})=0$ for all $x_{i}\in\mathbb{R}$. Thus it cannot be used for many distributions with compact support if the density does not converge to zero at the (topological) boundary. Initial work to adapt score matching to truncated distributions was presented in (Hyvärinen, 2007) for distributions on $[0,\infty)$ then further expanded in (Liu et al., 2022; Yu et al., 2022) to general compact spaces $\mathcal{X}$. For our compact space $\mathcal{X}\subseteq\mathbb{R}^{d}$ we use $\partial\mathcal{X}$ to denote the (topological) boundary. We now minimise some weighted version of the Fisher divergence whose weights go to zero at the boundary. Specifically let $\bm{g}:\mathcal{X}\rightarrow\mathbb{R}$ be a function satisfying $\lim_{\bm{x}\rightarrow\bm{x}^{\prime}}\bm{g}(\bm{x})_{j}=0$ for any $\bm{x}^{\prime}\in\partial\mathcal{X},~j\in[d]$. Our objective is then

截断 Score Matching

标准 score matching 的一个局限是要求对所有 $x_{i}\in\mathbb{R}$ 有 $\lim_{x_{i}\rightarrow\infty}p(\bm{x})=0$。因此，若密度在（拓扑）边界处不趋于零，许多紧支撑分布无法直接使用。Hyvärinen（2007）率先把 score matching 适配到 $[0,\infty)$ 上的截断分布，随后 Liu et al.（2022）；Yu et al.（2022）推广到一般紧空间 $\mathcal{X}$。对紧空间 $\mathcal{X}\subseteq\mathbb{R}^{d}$，用 $\partial\mathcal{X}$ 表示其（拓扑）边界。我们转而最小化某种加权 Fisher 散度，权重在边界趋于零。具体地，设 $\bm{g}:\mathcal{X}\rightarrow\mathbb{R}$ 满足：对任意 $\bm{x}^{\prime}\in\partial\mathcal{X},~j\in[d]$，有 $\lim_{\bm{x}\rightarrow\bm{x}^{\prime}}\bm{g}(\bm{x})_{j}=0$。目标函数为

$$\displaystyle F_{\trunc}(\theta)\coloneqq\mathbb{E}\left[\left\|\bm{g}^{\frac{1}{2}}(X)\circ\left(\bm{s}_{\theta}(X)-\bm{s}(X)\right)\right\|^{2}\right].$$

Just as in classical score matching we obtain an equivalence (though this time via Green’s theorem rather than simple integration by parts) giving us that under certain regularity conditions on $\bm{g},\bm{s}$, and $\mathcal{X}$,

与经典 score matching 类似，我们仍可得到等价形式（这次借助 Green 定理而非简单分部积分）：在 $\bm{g},\bm{s}$ 与 $\mathcal{X}$ 的适当正则条件下，

$$\displaystyle L_{\trunc}(\theta)\coloneqq \\ \displaystyle\mathbb{E}\left[\sum_{j\in d}\bm{g}(X)_{j}\left(2\partial_{j}\bm{s}_{\theta}(X)_{j}+\bm{s}_{\theta}(X)_{j}^{2}\right)\right] \\ \displaystyle+~\mathbb{E}\left[\sum_{j\in d}\partial_{j}\bm{g}(X)_{j}\bm{s}_{\theta}(X)_{j}\right]=F_{\trunc}(\theta)-C.$$

This can again be approximated via data using standard Monte Carlo approximation. Full details on the conditions required for this approach alongside the proof can be found in (Liu et al., 2022). Two other key extensions of score matching are sliced score matching (Song et al., 2020) and denoising score matching (Vincent, 2011). We introduce these extensions in Appendix D with our corresponding adaptations to missing data given in Appendix A.1. Now, we give our missing data scenario.

同样可用标准蒙特卡洛用数据近似。所需条件与证明详见 Liu et al.（2022）。Score matching 的另外两个重要扩展是切片 score matching（Song et al., 2020）与去噪 score matching（Vincent, 2011）。我们在附录 D 介绍这些扩展，并在附录 A.1 给出对应的缺失数据改编。下面给出缺失数据场景。

3.3 Missing Data Scenario

3.3 缺失数据场景

Instead of observing samples from $X$ we assume that we observe samples from the corrupted version of the RV given by $\tilde{X}$. To define $\tilde{X}$ we introduce a mask RV $M$ over $\{0,1\}^{d}$ and then define $\tilde{X}$ by

我们不再直接观测 $X$ 的样本，而假设观测的是其损坏版本 $\tilde{X}$。为定义 $\tilde{X}$，引入取值于 $\{0,1\}^{d}$ 的掩码随机变量 $M$，并定义 $\tilde{X}$ 为

$$\displaystyle\tilde{X}{j}=\begin{cases}X{j}&\text{if }M_{j}=1\
\varnothing&\text{if }M_{j}=0\end{cases}$$

where $\tilde{X}_{j}=\varnothing$ represents that coordinate being missing. We will be focussing on the missing completely at random scenario where $M\!\!\perp\!\!X$. However, we do provide an extension to missing not at random data in Appendix A.1.4. We introduce the RV $\Lambda$ on $\mathcal{P}([d])$ defined by $\Lambda\coloneqq\{i\in[d]|M_{i}=1\}$ so that $\Lambda$ gives the non-corrupted coordinates of $\tilde{X}$ and take $\lambda$ to be a sample of $\Lambda$. Crucially given samples from $\tilde{X}$, we also have samples from $X_{\Lambda}$.

其中 $\tilde{X}_{j}=\varnothing$ 表示该坐标缺失。我们聚焦完全随机缺失（MCAR）情形，即 $M\!\!\perp\!\!X$；附录 A.1.4 也给出了非随机缺失的扩展。再引入取值于 $\mathcal{P}([d])$ 的随机变量 $\Lambda\coloneqq\{i\in[d]|M_{i}=1\}$，于是 $\Lambda$ 给出 $\tilde{X}$ 中未损坏的坐标，并取 $\lambda$ 为 $\Lambda$ 的一个样本。关键是：给定 $\tilde{X}$ 的样本，我们也得到了 $X_{\Lambda}$ 的样本。

Our aim is to adapt the score matching objective to estimate the full score $\bm{s}$ by a parameterised score $\bm{s}_{\theta}$ using samples from the corrupted data $\tilde{D}\coloneqq\{\tilde{X}^{(i)}\}_{i=1}^{n}\equiv\{X^{(i)}_{\Lambda_{i}}\}_{i=1}^{n}$.

我们的目标是：用损坏数据样本 $\tilde{D}\coloneqq\{\tilde{X}^{(i)}\}_{i=1}^{n}\equiv\{X^{(i)}_{\Lambda_{i}}\}_{i=1}^{n}$，把 score matching 目标改造成用参数化 score $\bm{s}_{\theta}$ 估计完整 score $\bm{s}$。

4 Marginal Score Matching

4 边际 Score Matching

To motivate our approach we look at how we might use MLE in the case where the normalising constant and conditional normalising constants were calculable. For $p_{\theta}$ our parametric model of the density, we would choose $\hat{\theta}$ to be

为动机我们的做法，先看在归一化常数与条件归一化常数都可计算时如何做 MLE。对密度的参数模型 $p_{\theta}$，会取 $\hat{\theta}$ 为

$$\displaystyle\hat{\theta}\coloneqq\argmax_{\theta}\sum_{i=1}^{n}\log\tilde{p}_{\theta}(\tilde{X}^{(i)})$$

where $\tilde{p}_{\theta}$ is the associated corrupted data density when $X\sim p_{\theta}$. As our data is missing completely at random this is actually equivalent to maximising

其中 $\tilde{p}_{\theta}$ 是当 $X\sim p_{\theta}$ 时对应的损坏数据密度。由于数据完全随机缺失，这实际上等价于最大化

$$\displaystyle\sum_{i=1}^{n}\log p_{\theta;\Lambda_{i}}(X_{\Lambda_{i}}^{(i)})\text{, where }p_{\theta;\lambda}(\bm{x}_{\lambda})\coloneqq\int_{\mathcal{X}_{-\lambda}}p_{\theta}(\bm{x})\mathrm{d}\bm{x}_{-\lambda}.$$

For notational simplicity we will thus reframe our problem as working with marginal samples $\{X^{(i)}_{\Lambda_{i}}\}_{i=1}^{n}$.

为记号简洁，我们把问题改写成处理边际样本 $\{X^{(i)}_{\Lambda_{i}}\}_{i=1}^{n}$。

4.1 Marginal Score Matching

4.1 边际 Score Matching

Our approach is to directly alter the score matching objective similarly. Just as densities have associated marginal densities so do scores have associated marginal scores.

做法是类似地直接改写 score matching 目标。正如密度有对应的 边际密度，score 也有对应的 边际 score。

Definition 4.1(Marginal Score function).

Let $\bm{s}$ be a score function with $\bm{s}(\bm{x})=\nabla_{\bm{x}}\log q(\bm{x})$ for $q$ an unnormalised PDF. Then the associated marginal score function is

定义 4.1（边际 Score 函数）。

设 $\bm{s}$ 为 score 函数，且对未归一化 pdf $q$ 有 $\bm{s}(\bm{x})=\nabla_{\bm{x}}\log q(\bm{x})$。则相应的 边际 score 函数为

$$\displaystyle\bm{s}_{\lambda}(\bm{x}_{\lambda})\coloneqq\nabla_{\bm{x}_{\lambda}}\log\int_{\mathbb{R}^{d-|\Lambda|}}q(\bm{x})\mathrm{d}\bm{x}_{-\lambda}. \tag{2}$$

This definition of marginal scores restricts $\bm{s}$ to a genuine score function. For this reason we will also want $\bm{s}_{\theta}$ to always be a genuine score function or at least to have an anti-derivative. The simplest way to achieve this is to work with $q_{\theta}:\mathcal{X}\rightarrow(0,\infty)$ as our baseline and define $\bm{s}_{\theta}(\bm{x})\coloneqq\nabla_{\bm{x}}\log q_{\theta}(\bm{x})$. We will also take $p_{\theta}(\bm{x})\coloneqq\left(\int_{\mathcal{X}}q_{\theta}(\bm{x})\mathrm{d}\bm{x}\right)^{-1}q_{\theta}(\bm{x})$ which we assume to be unknown.

该边际 score 定义要求 $\bm{s}$ 本身是真正的 score 函数。因此我们也希望 $\bm{s}_{\theta}$ 始终是真正的 score，或至少有原函数。最简单的做法是以 $q_{\theta}:\mathcal{X}\rightarrow(0,\infty)$ 为基底，定义 $\bm{s}_{\theta}(\bm{x})\coloneqq\nabla_{\bm{x}}\log q_{\theta}(\bm{x})$。同时取 $p_{\theta}(\bm{x})\coloneqq\left(\int_{\mathcal{X}}q_{\theta}(\bm{x})\mathrm{d}\bm{x}\right)^{-1}q_{\theta}(\bm{x})$，并假定其未知。

With this notion of a marginal score we can define our marginal Fisher divergence to be

有了边际 score 的概念，可将边际 Fisher 散度定义为

$$\displaystyle F_{\marg}(\theta) \\ \displaystyle\coloneqq\mathbb{E}[\|\bm{s}_{\Lambda}(X_{\Lambda})-\bm{s}_{\Lambda;\theta}(X_{\Lambda})\|^{2}] \tag{3}$$

where $\bm{s}_{\lambda;\theta}$ is defined analogously to $\bm{s}_{\lambda}$. As with normal score matching can relate this objective to one involving no terms of $\bm{s}_{\lambda}$. We first need the following assumptions.

其中 $\bm{s}_{\lambda;\theta}$ 的定义与 $\bm{s}_{\lambda}$ 类似。与普通 score matching 一样，可以把该目标改写成不含 $\bm{s}_{\lambda}$ 的形式。为此先给出以下假设。

Assumption 4.2.

For any $\theta>0,\lambda\in\supp(\Lambda)$:

假设 4.2。

对任意 $\theta>0,\lambda\in\supp(\Lambda)$：

(a) $p_{\theta}$ is well defined, i.e. $\int_{\mathcal{X}}q_{\theta}(x)\mathrm{d}x<\infty$;

(a) $p_{\theta}$ 良定义，即 $\int_{\mathcal{X}}q_{\theta}(x)\mathrm{d}x<\infty$；

(b) $\mathbb{E}[\|\bm{s}_{\lambda}(X_{\lambda})\|^{2}],~\mathbb{E}[\|\bm{s}_{\lambda;\theta}(X_{\lambda})\|^{2}]<\infty$;

(b) $\mathbb{E}[\|\bm{s}_{\lambda}(X_{\lambda})\|^{2}],~\mathbb{E}[\|\bm{s}_{\lambda;\theta}(X_{\lambda})\|^{2}]<\infty$；

(c) $p_{\lambda}(\bm{x})$ is differentiable and $q_{\lambda;\theta}$ is twice differentiable;

(c) $p_{\lambda}(\bm{x})$ 可微，且 $q_{\lambda;\theta}$ 二阶可微；

(d) $p_{\lambda}(\bm{x}_{\lambda})\bm{s}_{\lambda;\theta}(\bm{x}_{\lambda})\overset{}{\longrightarrow}0$ as $\|\bm{x}\|\overset{}{\longrightarrow}\infty$;

(d) 当 $\|\bm{x}\|\overset{}{\longrightarrow}\infty$ 时，$p_{\lambda}(\bm{x}_{\lambda})\bm{s}_{\lambda;\theta}(\bm{x}_{\lambda})\overset{}{\longrightarrow}0$；

(e) $p_{\lambda;\theta}(X_{\lambda})=p_{\lambda}(X_{\lambda})$ almost surely (a.s.) for all $\lambda\in\supp(\Lambda)$, implies that $p_{\theta}(X)=p(X)$ a.s..

(e) 若对所有 $\lambda\in\supp(\Lambda)$ 几乎必然（a.s.）有 $p_{\lambda;\theta}(X_{\lambda})=p_{\lambda}(X_{\lambda})$，则几乎必然有 $p_{\theta}(X)=p(X)$。

Assumption (a) ensures that our proposal unnormalised density is always a genuine unnormalised density. Assumptions (b)-(d) are similar to the standard assumptions given for standard score matching. Assumption (e) is an identifiability assumption which is required to be feasibly able to learn the true data distribution from our corrupted data.

假设 (a) 保证提议的未归一化密度始终是真正的未归一化密度。(b)–(d) 与标准 score matching 的常见假设类似。(e) 是可辨识性假设：要能从损坏数据学到真数据分布，这一条是必要的。

Proposition 4.3.

Given Assumptions 4.2(a)-(d) hold

命题 4.3。

在假设 4.2(a)–(d) 成立时

$$\displaystyle L_{\marg}(\theta)\coloneqq \\ \displaystyle\mathbb{E}[2\nabla_{X_{\Lambda}}\cdot\bm{s}_{\Lambda;\theta}(X_{\Lambda})+\|\bm{s}_{\Lambda;\theta}(X_{\Lambda})\|^{2}] \\ \displaystyle= \\ \displaystyle F_{\marg}(\theta)-C. \tag{4}$$

If (e) also holds and there exists some $\theta^{}$such that$\bm{s}{\theta^{*}}(X)=\bm{s}(X)$a.s.. Then if$\tilde{\theta}$is a minimiser of$L{\marg}(\theta)$we have that$q_{\tilde{\theta}}(X)=Np(X)$a.s. for some constant$N$, i.e. the minimiser is the true unnormalised density.*

若 (e) 也成立，且存在某个 $\theta^{}$使得几乎必然有$\bm{s}{\theta^{*}}(X)=\bm{s}(X)$。则若$\tilde{\theta}$是$L{\mathrm{marg}}(\theta)$的极小点，几乎必然有$q_{\tilde{\theta}}(X)=Np(X)$（$N$ 为某常数），即极小点对应真未归一化密度。*

Through this result we have shown, much like with standard score matching, that under certain regularity conditions our objective is uniquely minimised by the true unnormalised density. We then approximate this objective by

由此我们证明：与标准 score matching 类似，在一定正则条件下，目标由真未归一化密度唯一最小化。随后用下式近似该目标

$$\displaystyle\hat{L}_{\marg;n}(\theta)\coloneqq\frac{1}{n}\sum_{i=1}^{n}\nabla_{X_{\Lambda_{i}}^{(i)}}\cdot\bm{s}_{\Lambda_{i},\theta}(X^{(i)}_{\Lambda_{i}})+\|\bm{s}_{\Lambda_{i};\theta}(X^{(i)}_{\Lambda_{i}})\|^{2}$$

and choose $\hat{\theta}=\argmin_{\theta}\hat{L}_{\marg;n}(\theta)$.

并取 $\hat{\theta}=\argmin_{\theta}\hat{L}_{\mathrm{marg};n}(\theta)$。

Unfortunately this approach in its current state is practically infeasible as the integrals involved in deriving the marginal scores for any non-trivial problem will be intractable. Hence, we must devise a way to estimate the marginal scores without having to compute the integrals. We tackle this issue in Section 4.2, but first we provide a similar result for the case of truncated score matching.

遗憾的是，当前形式在实践中几乎不可行：对任何非平凡问题，推导边际 score 所需积分通常不可计算。因此必须设法在不显式积分的情况下估计边际 score。第 4.2 节处理这一问题；在此之前，我们先给出截断 score matching 的类似结果。

4.1.1 Truncated Score Matching

Truncated score matching can be adapted similarly to standard score matching by simply having marginal weighting functions $\bm{g}_{\lambda}:\mathcal{X}_{\lambda}\rightarrow[0,\infty)$ for each subset $\lambda\in\supp(\Lambda)$ and taking the marginal truncated Fisher divergence to be

4.1.1 截断 Score Matching

截断 score matching 可类似改编：对每个子集 $\lambda\in\supp(\Lambda)$ 引入边际权重函数 $\bm{g}_{\lambda}:\mathcal{X}_{\lambda}\rightarrow[0,\infty)$，并将边际截断 Fisher 散度取为

$$\displaystyle F_{\truncmarg}(\theta)\!\!\coloneqq\mathbb{E}\left[\left\|\bm{g}_{\Lambda}(X_{\Lambda})^{\frac{1}{2}}\circ\left(\bm{s}_{\Lambda}(X_{\Lambda})-\bm{s}_{\Lambda;\theta}(X_{\Lambda})\right)\right\|^{2}\right].$$

using integration by parts gives the following equivalence

经分部积分得到如下等价形式

$$\displaystyle L_{\truncmarg}(\theta)\!\!\coloneqq \\ \displaystyle\mathbb{E}\!\!\left[\sum_{j\in\Lambda}\bm{g}_{\Lambda}(X_{\Lambda})_{j}\!\left(\bm{s}_{\Lambda;\theta}(X_{\Lambda})^{2}_{j}\!+\!2\partial_{j}\bm{s}_{\Lambda;\theta}(X_{\Lambda})_{j}\right)\right] \\ \displaystyle+~\mathbb{E}\!\!\left[\sum_{j\in\Lambda}2\partial_{j}\bm{g}_{\Lambda}(X_{\Lambda})_{j}\bm{s}_{\Lambda;\theta}(X_{\Lambda})_{j}\right] \\ \displaystyle= \\ \displaystyle F_{\truncmarg}(\theta)-C. \tag{5}$$

Proof given in Appendix C.1.1. We then take $\hat{L}_{\truncmarg;n}$ as the Monte-Carlo estimate of $L_{\truncmarg}$. We also construct similar objectives from sliced and denoising score matching as well as a similar result for missing not at random data in Appendix A.1. We now move to the task of estimating the marginal scores in these objectives.

证明见附录 C.1.1。再取 $\hat{L}_{\mathrm{truncmarg};n}$ 为 $L_{\mathrm{truncmarg}}$ 的蒙特卡洛估计。我们也在附录 A.1 中从切片与去噪 score matching 构造了类似目标，并给出非随机缺失数据的相应结果。接下来转向估计这些目标中的边际 score。

4.2 Importance Weighting

4.2 重要性加权

Our first proposal is an importance weighting approach. Let $p^{\prime}$ be a density over $R^{d-|\lambda|}$ which we can both evaluate and sample from then

第一种方案是重要性加权。设 $p^{\prime}$ 为 $\mathbb{R}^{d-|\lambda|}$ 上既可求值又可采样的密度，则

$$\displaystyle\int_{\mathbb{R}^{d-|\lambda|}}q_{\theta}(\bm{x})\mathrm{d}\bm{x}_{-\lambda} \\ \displaystyle=\mathbb{E}_{X^{\prime}_{\lambda}\sim p^{\prime}}\left[\frac{q_{\theta}(\bm{x}_{\lambda},X^{\prime}_{-\lambda})}{p^{\prime}(X^{\prime}_{-\lambda})}\right]. \tag{6}$$

This allows us to define our marginal score estimate.

由此可定义我们的 边际 score 估计。

Definition 4.4(Marginal Score Estimate).

For a given $\lambda\in\supp(\Lambda)~,\bm{x}_{\lambda}\in\mathcal{X}_{\lambda}$, score model, $\bm{s}_{\theta}$, and $r\in\mathbb{N}$ we take our estimate of $\bm{s}_{\theta;\lambda,r}(\bm{x}_{\lambda})$ to be

定义 4.4（边际 Score 估计）。

对给定 $\lambda\in\supp(\Lambda)~,\bm{x}_{\lambda}\in\mathcal{X}_{\lambda}$、score 模型 $\bm{s}_{\theta}$ 以及 $r\in\mathbb{N}$，取 $\bm{s}_{\theta;\lambda,r}(\bm{x}_{\lambda})$ 的估计为

$$\displaystyle\hat{\bm{s}}_{\lambda,r;\theta}\coloneqq\nabla_{\bm{x}_{\lambda}}\log\left(\frac{1}{r}\sum_{k=1}^{r}\frac{q_{\theta}(\bm{x}_{\lambda},X^{\prime(k)}_{-\lambda})}{p^{\prime}(X^{{}^{\prime}(k)}_{-\lambda})}\right) \tag{7}$$

where $X^{\prime(1)}_{-\lambda},\dotsc,X^{\prime(r)}_{-\lambda}$ are IID copies of $X^{\prime}_{-\lambda}\sim p^{\prime}$.

其中 $X^{\prime(1)}_{-\lambda},\dotsc,X^{\prime(r)}_{-\lambda}$ 为 $X^{\prime}_{-\lambda}\sim p^{\prime}$ 的 IID 副本。

4.2.1 IW Sample Objective

We can now plug these marginal score estimates into our sample objective for either normal or truncated score matching. We use $\marg/\truncmarg$ to denote analogous definitions and results for both marginal and truncated marginal score matching. Let $\{X^{(i)}_{\Lambda_{i}}\}_{i=1}^{n}$ be our samples from $X_{\Lambda}$. We then take our IW sample objective to be as $\hat{L}_{\marg/\truncmarg;n}(\theta)$ but with $\hat{\bm{s}}_{\Lambda_{i},r;\theta}(X_{\Lambda_{i}}^{(i)})$ replacing $\bm{s}_{\Lambda_{i};\theta}(X_{\Lambda_{i}}^{(i)})$. The full objective is given in Appendix E.1.1 We refer to this sample objective as $\hat{L}_{\marg/\truncmarg;n,r}(\theta)$ and take our estimate to be

4.2.1 IW 样本目标

现在可以把这些边际 score 估计代入普通或截断 score matching 的样本目标。用 $\mathrm{marg}/\mathrm{truncmarg}$ 表示边际与截断边际两种情形下的对应定义与结果。设 $\{X^{(i)}_{\Lambda_{i}}\}_{i=1}^{n}$ 为我们从 $X_{\Lambda}$ 得到的样本。IW 样本目标取为与 $\hat{L}_{\mathrm{marg}/\mathrm{truncmarg};n}(\theta)$ 相同，但用 $\hat{\bm{s}}_{\Lambda_{i},r;\theta}(X_{\Lambda_{i}}^{(i)})$ 替换 $\bm{s}_{\Lambda_{i};\theta}(X_{\Lambda_{i}}^{(i)})$。完整目标见附录 E.1.1。我们称该样本目标为 $\hat{L}_{\mathrm{marg}/\mathrm{truncmarg};n,r}(\theta)$，并取估计为

$$\displaystyle\hat{\theta}\coloneqq\argmin_{\theta}\hat{L}_{\marg/\truncmarg;n,r}(\theta).$$

Algorithm 1 gives our high level estimation algorithm.

算法 1 给出高层估计流程。

Input: $\{X^{(i)}_{\Lambda_{i}}\}_{i\in[n]}$, $q_{\theta}$, $p^{\prime}$, $\theta_{0}$, $r\in\mathbb{N}$.
Set $\theta=\theta_{0}$.
repeat
for i=1 to n do
Sample $\{X^{\prime(i,k)}\}_{k\in r}$ from $p^{\prime}(.|X_{\Lambda_{i}}^{(i)})$.
Use $X^{(i)}_{\Lambda_{i}},\{X^{\prime(i,k)}_{-\Lambda_{i}}\}_{k\in[r]}$ to get Monte-Carlo estimates, $\hat{\bm{s}}_{\Lambda_{i},r;\theta}(X^{(i)}_{\Lambda_{i}})$, of the marginal scores by (7).
end for
Use $\hat{\bm{s}}_{\Lambda_{i},r;\theta}(X^{(i)}_{\Lambda_{i}})$ to obtain $\hat{L}_{\marg/\truncmarg;n,r}(\theta)$ by (4).
Compute $\nabla_{\theta}\hat{L}_{\marg/\truncmarg;n,r}(\theta)$ and update the value of $\theta$.
until Maximum iteration reached.

Algorithm 1 Marginal IW Score Matching

输入： $\{X^{(i)}_{\Lambda_{i}}\}_{i\in[n]}$，$q_{\theta}$，$p^{\prime}$，$\theta_{0}$，$r\in\mathbb{N}$。
置 $\theta=\theta_{0}$。
repeat
for i=1 to n do
从 $p^{\prime}(.|X_{\Lambda_{i}}^{(i)})$ 采样 $\{X^{\prime(i,k)}\}_{k\in r}$。
用 $X^{(i)}_{\Lambda_{i}},\{X^{\prime(i,k)}_{-\Lambda_{i}}\}_{k\in[r]}$ 按 (7) 得到边际 score 的蒙特卡洛估计 $\hat{\bm{s}}_{\Lambda_{i},r;\theta}(X^{(i)}_{\Lambda_{i}})$。
end for
用 $\hat{\bm{s}}_{\Lambda_{i},r;\theta}(X^{(i)}_{\Lambda_{i}})$ 按 (4) 得到 $\hat{L}_{\mathrm{marg}/\mathrm{truncmarg};n,r}(\theta)$。
计算 $\nabla_{\theta}\hat{L}_{\mathrm{marg}/\mathrm{truncmarg};n,r}(\theta)$ 并更新 $\theta$。
until 达到最大迭代次数。

算法 1 边际 IW Score Matching

Remark 4.5.

Algorithm 1 can directly be applied to both sliced and denoised score matching by replacing equation (4) by equations (13) and (15) respectively.

注 4.5。

算法 1 可直接用于切片与去噪 score matching：分别用式 (13)、(15) 替换式 (4) 即可。

4.2.2 Finite Sample Bounds

A benefit of truncated score matching is that it allows us to work on distributions with densities bounded below which enables us to give finite sample bounds for the error of our estimated score w.r.t. our marginal objective. We briefly present these now with more detail given in Appendix A.2.

4.2.2 有限样本界

截断 score matching 的一个好处是可处理密度有下界的分布，从而能对估计 score 相对边际目标的误差给出有限样本界。下面简要给出；更多细节见附录 A.2。

Theorem 4.6.

Suppose assumption 4.2 alongside assumptions A.1, A.11, A.13 from the Appendix hold and let $\theta_{n,r}\in\Theta$ be the minimiser of $\hat{L}_{\truncmarg;n,r}(\theta)$. If $\Theta\subseteq\mathbb{R}^{p}$ with $\diam(\Theta)=A$ then for sufficiently large $n,r$

定理 4.6。

设假设 4.2 以及附录中的假设 A.1、A.11、A.13 成立，并令 $\theta_{n,r}\in\Theta$ 为 $\hat{L}_{\mathrm{truncmarg};n,r}(\theta)$ 的极小点。若 $\Theta\subseteq\mathbb{R}^{p}$ 且 $\mathrm{diam}(\Theta)=A$，则当 $n,r$ 充分大时

$$\displaystyle\mathbb{P}\Bigg(F_{\truncmarg}(\theta_{n,r})\geq \\ \displaystyle\beta_{1}\sqrt{\frac{p\log(dnrA/\delta)}{\min\{r,n\}}}\Bigg)<\delta.$$

Note that $r$ is the number of importance weighting samples for each data sample and therefore is something we can choose ourself. This means that with this approach we can achieve approximately $\sqrt{n}$ convergence rates. A downside however is that to achieve this we need $r$ to be of order at least $n$ which would lead to an $O(n^{2})$ computational cost. In practice we find relatively strong performance choosing $r$ small. Setting it at $r=10$ in our experiments.

注意 $r$ 是每个数据样本的重要性加权样本数，因而可由我们自行选择。这意味着该方法大约可达到 $\sqrt{n}$ 收敛速率。代价是：要达到该速率，$r$ 至少需为 $n$ 量级，从而带来 $O(n^{2})$ 计算成本。实践中取较小的 $r$ 仍表现不错；实验中我们设 $r=10$。

Remark 4.7.

The error presented is measured with respect to our Marginal Fisher Divergence, rather than the full Fisher Divergence (which would be the preferred accuracy metric). Relating these two quantities requires connecting the fully observed distribution to its marginals, a task that depends on the specific form of the distribution. Investigating the assumptions and conditions under which this connection can be made offers an interesting and valuable direction for future research.

注 4.7。

这里给出的误差是相对边际 Fisher 散度度量的，而非完整 Fisher 散度（后者通常是更理想的精度指标）。把二者联系起来需要把完全观测分布与其边际关联起来，而这依赖于分布的具体形式。研究在何种假设与条件下可建立这种联系，是有价值的后续方向。

4.3 Gradient First Approach

4.3 先求梯度的做法

A key limitation with an IW approach is that it will struggle in higher dimensional scenarios. Additionally the importance weighting is embedded inside other functions which leads to the same nested expectation issue as the EM approach of Uehara et al. (2020), causing bias in our estimator. As an alternative to this we build upon a variational approach initially discussed in the context of latent variable models in Vértes & Sahani (2016); Bao et al. (2020); Bao et al. (2021).

IW 方法的关键局限是在高维场景会吃力。此外，重要性加权嵌在其他函数内部，会带来与 Uehara et al.（2020）EM 方法相同的嵌套期望问题，从而使估计有偏。作为替代，我们建立在 Vértes & Sahani（2016）；Bao et al.（2020）；Bao et al.（2021）于隐变量语境下讨论过的变分思路之上。

The core idea is to start with $L_{\marg}$ as before and then take gradients w.r.t. our parameters before then writing our objective in terms of expectations over $X_{-\lambda|\lambda;\theta}$. As we don’t then need to take gradients of these expectations w.r.t. $\theta$, we can estimate them with any black-box method we desire, opening the door for variational approximation to be used. This approach has been explored for exponential family distributions (Vértes & Sahani, 2016) and for denoising score matching (Bao et al., 2021) however we provide the most general version of this result which can be applied to any of the score matching methods and any model class. We first introduce the following key Lemma.

核心想法是：仍从 $L_{\mathrm{marg}}$ 出发，先对参数求梯度，再把目标写成关于 $X_{-\lambda|\lambda;\theta}$ 的期望。由于随后不必再对这些期望关于 $\theta$ 求导，可用任意黑盒方法估计它们，从而为变分近似打开大门。指数族（Vértes & Sahani, 2016）与去噪 score matching（Bao et al., 2021）已有探索；我们给出最一般形式，可套用到任意 score matching 方法与任意模型类。先引入下述关键引理。

Lemma 4.8.

Fix $\lambda\subseteq[d],\bm{x}_{\lambda}\in\mathcal{X}_{\lambda}$. We have that for any function $h_{\theta}:\mathcal{X}\rightarrow\mathbb{R}$.

引理 4.8。

固定 $\lambda\subseteq[d],\bm{x}_{\lambda}\in\mathcal{X}_{\lambda}$。对任意函数 $h_{\theta}:\mathcal{X}\rightarrow\mathbb{R}$，有

$$\displaystyle\bm{s}_{\theta;\lambda}(\bm{x}_{\lambda})\!= \\ \displaystyle\mathbb{E}^{\prime}[\bm{s}_{\theta}(\bm{x}_{\lambda},X^{\prime}_{-\lambda})_{\lambda}] \\ \displaystyle\nabla\mathbb{E}^{\prime}[h_{\theta}(\bm{x}_{\lambda},X^{\prime}_{-\lambda})]\!= \\ \displaystyle\mathbb{E}^{\prime}[\nabla h_{\theta}(\bm{x}_{\lambda},X^{\prime}_{-\lambda})] \\ \displaystyle\!+\!\cov^{\prime}\!(\bm{s}_{\theta}(\bm{x}_{\lambda},X^{\prime}_{-\lambda}),h_{\theta}(\bm{x}_{\lambda},X^{\prime}_{-\lambda})) \tag{8}$$

where $\nabla$ represents the gradient w.r.t. either $\bm{x}_{\lambda}$ or $\theta$ and here $\mathbb{E}^{\prime},~\cov^{\prime}\!$ are w.r.t. ${X^{\prime}_{-\lambda}|X_{\lambda}=\bm{x}_{\lambda}\sim p_{\theta}(.|\bm{x}_{\lambda}})$.

其中 $\nabla$ 表示关于 $\bm{x}_{\lambda}$ 或 $\theta$ 的梯度；此处 $\mathbb{E}^{\prime},~\mathrm{cov}^{\prime}\!$ 取关于 ${X^{\prime}_{-\lambda}|X_{\lambda}=\bm{x}_{\lambda}\sim p_{\theta}(.|\bm{x}_{\lambda}})$。

This results allows us to obtain our alternative objective.

该结果使我们得到替代目标。

Corollary 4.9.

Let $L_{\marg}$ be defined as in (4). We have that

推论 4.9。

设 $L_{\mathrm{marg}}$ 如 (4) 所定义。则

$$\displaystyle\nabla_{\theta}L_{\marg}(\theta)= \\ \displaystyle\mathbb{E}\bigg[2\sum_{j\in\Lambda}\Big(\Psi_{\Lambda}\big(\bm{s}_{\theta}(.)_{j}^{2}+\partial_{j}\bm{s}_{\theta}(.)_{j}\big) \\ \displaystyle\quad~~-\mathbb{E}^{\prime}[\bm{s}_{\theta}(X_{\Lambda},X^{\prime}_{-\Lambda})_{j}]\Psi_{\Lambda}(\bm{s}_{\theta}(.)_{j})\Big)\bigg] \tag{10}$$

where for any function $h_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}$, $\lambda\subseteq[d]$,

其中对任意函数 $h_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}$、$\lambda\subseteq[d]$，

$$\displaystyle\Psi_{\Lambda}(h_{\theta})= \\ \displaystyle\mathbb{E}^{\prime}[\nabla_{\theta}h_{\theta}(X_{\Lambda},X^{\prime}_{-\Lambda})] \\ \displaystyle+\cov^{\prime}\!\left(\nabla_{\theta}\log q_{\theta}(X_{\Lambda},X^{\prime}_{-\Lambda}),h_{\theta}(X_{\Lambda},X^{\prime}_{-\Lambda})\right)$$

and $\mathbb{E}^{\prime},\cov^{\prime}\!$ are w.r.t. $X^{\prime}_{-\Lambda}|X_{\Lambda}\sim p_{\theta}(.|X_{\Lambda})$ with $\mathbb{E}$ being w.r.t. $X_{\Lambda}\sim p$.

且 $\mathbb{E}^{\prime},\mathrm{cov}^{\prime}\!$ 取关于 $X^{\prime}_{-\Lambda}|X_{\Lambda}\sim p_{\theta}(.|X_{\Lambda})$，$\mathbb{E}$ 取关于 $X_{\Lambda}\sim p$。

Proofs for both results are given in Appendix C.3.

两个结果的证明见附录 C.3。

Crucially $\mathbb{E}^{\prime},\cov^{\prime}\!$ can be estimated freely. This allows us to use variational inference to approximate $p_{\theta}(\bm{x}_{-\lambda}|\bm{x}_{\lambda})$ and in turn the expectations and covariances in (10).

关键是 $\mathbb{E}^{\prime},\mathrm{cov}^{\prime}\!$ 可自由估计。因此可用变分推断近似 $p_{\theta}(\bm{x}_{-\lambda}|\bm{x}_{\lambda})$，进而估计 (10) 中的期望与协方差。

Remark 4.10.

We provide additional implementation details for computing this gradient estimate in Appendix A.5. We also discuss equivalences between this objective and our marginal IW objective in A.3.

注 4.10。

附录 A.5 给出该梯度估计的更多实现细节；A.3 讨论该目标与边际 IW 目标之间的等价性。

We explore estimation of $\mathbb{E}^{\prime},\cov^{\prime}\!$ in Section 4.3.2 but first we provide a similar result for truncated score matching.

第 4.3.2 节讨论 $\mathbb{E}^{\prime},\mathrm{cov}^{\prime}\!$ 的估计；此前先给出截断 score matching 的类似结果。

4.3.1 Truncated Score Matching

We define a similar objective for truncated score matching.

4.3.1 截断 Score Matching

我们对截断 score matching 定义类似目标。

Corollary 4.11.

With $L_{\truncmarg}$ defined as in (5) we have that