Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Arthur Jacot
École Polytechnique Fédérale de Lausanne
arthur.jacot@netopera.net

Franck Gabriel
Imperial College London and École Polytechnique Fédérale de Lausanne
franckrgabriel@gmail.com

Clément Hongler
École Polytechnique Fédérale de Lausanne
clement.hongler@gmail.com

32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.

Abstract

摘要

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit (12; 9), thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_{\theta}$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK.

在初始化时，人工神经网络（ANN）在无限宽极限下等价于高斯过程（12; 9），从而与核方法联系起来。我们证明，ANN 在训练中的演化也可以用一个核来描述：在网络参数上做梯度下降时，网络函数 $f_{\theta}$（把输入向量映到输出向量）会沿着泛函代价（它是凸的，与参数代价不同）相对于一个新核——神经正切核（Neural Tangent Kernel，NTK）——的核梯度前进。这一核对于刻画 ANN 的泛化性质是中心性的。NTK 在初始化时是随机的，并会在训练中变化；但在无限宽极限下，它收敛到一个可以显式写出的极限核，并且在训练过程中保持不变。于是我们可以在函数空间而不是参数空间里研究 ANN 的训练。训练能否收敛，也可以与极限 NTK 的正定性联系起来。

We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function $f_{\theta}$ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping.

接着我们聚焦于最小二乘回归，并证明在无限宽极限下，网络函数 $f_{\theta}$ 在训练中服从一个线性微分方程。收敛沿相对于 NTK 的最大核主成分最快，从而为早停提供了一种理论上的动机。

Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

最后我们从数值上研究 NTK，观察它在宽网络上的行为，并与无限宽极限加以比较。

1 Introduction

1 引言

Artificial neural networks (ANNs) have achieved impressive results in numerous areas of machine learning. While it has long been known that ANNs can approximate any function with sufficiently many hidden neurons (7; 10), it is not known what the optimization of ANNs converges to. Indeed the loss surface of neural networks optimization problems is highly non-convex: it has a high number of saddle points which may slow down the convergence (4). A number of results (3; 13; 14) suggest that for wide enough networks, there are very few “bad” local minima, i.e. local minima with much higher cost than the global minimum. More recently, the investigation of the geometry of the loss landscape at initialization has been the subject of a precise study (8). The analysis of the dynamics of training in the large-width limit for shallow networks has seen recent progress as well (11). To the best of the authors knowledge, the dynamics of deep networks has however remained an open problem until the present paper: see the contributions section below.

人工神经网络（ANN）已在机器学习的许多领域取得令人瞩目的结果。虽然人们早已知道，只要隐神经元足够多，ANN 可以逼近任意函数（7; 10），但 ANN 的优化会收敛到什么，仍不清楚。神经网络优化问题的损失曲面高度非凸：其中有大量鞍点，可能拖慢收敛（4）。一些结果（3; 13; 14）表明，对足够宽的网络而言，“很差”的局部极小很少，也就是代价远高于全局最小的局部极小很少。更近一些，初始化处损失地貌的几何已有精确研究（8）。浅层网络在大宽度极限下的训练动力学也有新的进展（11）。据作者所知，直到本文之前，深层网络的动力学仍是未解决问题：见下文的贡献一节。

A particularly mysterious feature of ANNs is their good generalization properties in spite of their usual over-parametrization (16). It seems paradoxical that a reasonably large neural network can fit random labels, while still obtaining good test accuracy when trained on real data (19). It can be noted that in this case, kernel methods have the same properties (1).

ANN 一个特别费解的特点是：尽管通常过参数化，它们仍有很好的泛化（16）。一个规模合理的网络既能拟合随机标签，在真实数据上训练时又能得到不错的测试准确率，这看起来自相矛盾（19）。可以指出，在这种情形下，核方法也有同样的性质（1）。

In the infinite-width limit, ANNs have a Gaussian distribution described by a kernel (12; 9). These kernels are used in Bayesian inference or Support Vector Machines, yielding results comparable to ANNs trained with gradient descent (9; 2). We will see that in the same limit, the behavior of ANNs during training is described by a related kernel, which we call the neural tangent network (NTK).

在无限宽极限下，ANN 具有由一个核描述的高斯分布（12; 9）。这些核被用于贝叶斯推断或支持向量机，得到的结果可与用梯度下降训练的 ANN 相比（9; 2）。我们将看到，在同一极限下，ANN 在训练中的行为由一个相关的核描述，我们称之为 neural tangent network（NTK）。

1.1 Contribution

1.1 贡献

We study the network function $f_{\theta}$ of an ANN, which maps an input vector to an output vector, where $\theta$ is the vector of the parameters of the ANN. In the limit as the widths of the hidden layers tend to infinity, the network function at initialization, $f_{\theta}$ converges to a Gaussian distribution (12; 9).

我们研究 ANN 的网络函数 $f_{\theta}$：它把输入向量映到输出向量，其中 $\theta$ 是 ANN 的参数向量。当隐层宽度趋于无穷时，初始化处的网络函数 $f_{\theta}$ 收敛到高斯分布（12; 9）。

In this paper, we investigate fully connected networks in this infinite-width limit, and describe the dynamics of the network function $f_{\theta}$ during training:

本文在这一无限宽极限下研究全连接网络，并描述网络函数 $f_{\theta}$ 在训练中的动力学：

• During gradient descent, we show that the dynamics of $f_{\theta}$ follows that of the so-called kernel gradient descent in function space with respect to a limiting kernel, which only depends on the depth of the network, the choice of nonlinearity and the initialization variance.

• 在梯度下降中，我们证明 $f_{\theta}$ 的动力学与函数空间中相对于某个极限核的所谓核梯度下降一致；该极限核只依赖于网络深度、非线性的选取以及初始化方差。

• The convergence properties of ANNs during training can then be related to the positive-definiteness of the infinite-width limit NTK. The values of the network function $f_{\theta}$ outside the training set is described by the NTK, which is crucial to understand how ANN generalize.

• 于是，ANN 在训练中的收敛性质可以与无限宽极限 NTK 的正定性联系起来。训练集之外的网络函数 $f_{\theta}$ 取值由 NTK 描述，这对理解 ANN 如何泛化至关重要。

• For a least-squares regression loss, the network function $f_{\theta}$ follows a linear differential equation in the infinite-width limit, and the eigenfunctions of the Jacobian are the kernel principal components of the input data. This shows a direct connection to kernel methods and motivates the use of early stopping to reduce overfitting in the training of ANNs.

• 对最小二乘回归损失，网络函数 $f_{\theta}$ 在无限宽极限下服从线性微分方程，而雅可比的特征函数正是输入数据的核主成分。这显示了与核方法的直接联系，并为在 ANN 训练中用早停减轻过拟合提供了动机。

• Finally we investigate these theoretical results numerically for an artificial dataset (of points on the unit circle) and for the MNIST dataset. In particular we observe that the behavior of wide ANNs is close to the theoretical limit.

• 最后，我们在一个人造数据集（单位圆上的点）以及 MNIST 数据集上，从数值上考察这些理论结果。特别地，我们观察到宽 ANN 的行为接近理论极限。

2 Neural networks

2 神经网络

In this article, we consider fully-connected ANNs with layers numbered from $0$ (input) to $L$ (output), each containing $n_{0},\ldots,n_{L}$ neurons, and with a Lipschitz, twice differentiable nonlinearity function $\sigma:\mathbb{R}\to\mathbb{R}$, with bounded second derivative.$^{1}$

本文考虑全连接 ANN，层从 $0$（输入）编号到 $L$（输出），各层分别有 $n_{0},\ldots,n_{L}$ 个神经元，并带有一个 Lipschitz、二阶可微且二阶导数有界的非线性函数 $\sigma:\mathbb{R}\to\mathbb{R}$。$^{1}$

$^{1}$While these smoothness assumptions greatly simplify the proofs of our results, they do not seem to be strictly needed for the results to hold true.

$^{1}$这些光滑性假设大大简化了我们结果的证明，但结果成立似乎并不严格需要它们。

This paper focuses on the ANN realization function $F^{(L)}:\mathbb{R}^{P}\to\mathcal{F}$, mapping parameters $\theta$ to functions $f_{\theta}$ in a space $\mathcal{F}$. The dimension of the parameter space is $P=\sum_{\ell=0}^{L-1}(n_{\ell}+1)n_{\ell+1}$: the parameters consist of the connection matrices $W^{(\ell)}\in\mathbb{R}^{n_{\ell}\times n_{\ell+1}}$ and bias vectors $b^{(\ell)}\in\mathbb{R}^{n_{\ell+1}}$ for $\ell=0,\ldots,L-1$. In our setup, the parameters are initialized as iid Gaussians $\mathcal{N}(0,1)$.

本文关注 ANN 的实现函数 $F^{(L)}:\mathbb{R}^{P}\to\mathcal{F}$，它把参数 $\theta$ 映到函数空间 $\mathcal{F}$ 中的函数 $f_{\theta}$。参数空间的维数是 $P=\sum_{\ell=0}^{L-1}(n_{\ell}+1)n_{\ell+1}$：参数由连接矩阵 $W^{(\ell)}\in\mathbb{R}^{n_{\ell}\times n_{\ell+1}}$ 和偏置向量 $b^{(\ell)}\in\mathbb{R}^{n_{\ell+1}}$ 组成，其中 $\ell=0,\ldots,L-1$。在我们的设定里，参数初始化为独立同分布的高斯 $\mathcal{N}(0,1)$。

For a fixed distribution $p^{\mathrm{in}}$ on the input space $\mathbb{R}^{n_{0}}$, the function space $\mathcal{F}$ is defined as $\{f:\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{L}}\}$. On this space, we consider the seminorm $\|\cdot\|_{p^{\mathrm{in}}}$, defined in terms of the bilinear form

对输入空间 $\mathbb{R}^{n_{0}}$ 上的固定分布 $p^{\mathrm{in}}$，函数空间 $\mathcal{F}$ 定义为 $\{f:\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{L}}\}$。在这个空间上，我们考虑半范数 $\|\cdot\|_{p^{\mathrm{in}}}$，它由下述双线性形式给出：

$$\langle f,g\rangle_{p^{\mathrm{in}}}=\mathbb{E}_{x\sim p^{\mathrm{in}}}\big[f(x)^{T}g(x)\big].$$

In this paper, we assume that the input distribution $p^{\mathrm{in}}$ is the empirical distribution on a finite dataset $x_{1},\ldots,x_{N}$, i.e the sum of Dirac measures $\frac{1}{N}\sum_{i=0}^{N}\delta_{x_{i}}$.

本文假定输入分布 $p^{\mathrm{in}}$ 是有限数据集 $x_{1},\ldots,x_{N}$ 上的经验分布，即 Dirac 测度之和 $\frac{1}{N}\sum_{i=0}^{N}\delta_{x_{i}}$。

We define the network function by $f_{\theta}(x):=\tilde{\alpha}^{(L)}(x;\theta)$, where the functions $\tilde{\alpha}^{(\ell)}(\cdot;\theta):\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{\ell}}$ (called preactivations) and $\alpha^{(\ell)}(\cdot;\theta):\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{\ell}}$ (called activations) are defined from the $0$-th to the $L$-th layer by:

我们把网络函数定义为 $f_{\theta}(x):=\tilde{\alpha}^{(L)}(x;\theta)$，其中函数 $\tilde{\alpha}^{(\ell)}(\cdot;\theta):\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{\ell}}$（称为预激活）和 $\alpha^{(\ell)}(\cdot;\theta):\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{\ell}}$（称为激活）从第 $0$ 层到第 $L$ 层由下式递归给出：

$$\begin{aligned}
\alpha^{(0)}(x;\theta) &= x, \
\tilde{\alpha}^{(\ell+1)}(x;\theta) &= \frac{1}{\sqrt{n_{\ell}}}W^{(\ell)}\alpha^{(\ell)}(x;\theta)+\beta b^{(\ell)}, \
\alpha^{(\ell)}(x;\theta) &= \sigma(\tilde{\alpha}^{(\ell)}(x;\theta)),
\end{aligned}$$

where the nonlinearity $\sigma$ is applied entrywise. The scalar $\beta>0$ is a parameter which allows us to tune the influence of the bias on the training.

其中非线性 $\sigma$ 按分量作用。标量 $\beta>0$ 是一个参数，用来调节偏置对训练的影响。

Remark 1. Our definition of the realization function $F^{(L)}$ slightly differs from the classical one. Usually, the factors $\frac{1}{\sqrt{n_{\ell}}}$ and the parameter $\beta$ are absent and the parameters are initialized using what is sometimes called LeCun initialization, taking $W^{(\ell)}_{ij}\sim\mathcal{N}(0,\frac{1}{n_{\ell}})$ and $b^{(\ell)}_{j}\sim\mathcal{N}(0,1)$ (or sometimes $b^{(\ell)}_{j}=0$) to compensate. While the set of representable functions $F^{(L)}(\mathbb{R}^{P})$ is the same for both parametrizations (with or without the factors $\frac{1}{\sqrt{n_{\ell}}}$ and $\beta$), the derivatives of the realization function with respect to the connections $\partial_{W^{(\ell)}_{ij}}F^{(L)}$ and bias $\partial_{b^{(\ell)}_{j}}F^{(L)}$ are scaled by $\frac{1}{\sqrt{n_{\ell}}}$ and $\beta$ respectively in comparison to the classical parametrization.

注 1. 我们对实现函数 $F^{(L)}$ 的定义与经典定义略有不同。通常没有因子 $\frac{1}{\sqrt{n_{\ell}}}$ 和参数 $\beta$，参数则用有时称为 LeCun 初始化的方式初始化，取 $W^{(\ell)}_{ij}\sim\mathcal{N}(0,\frac{1}{n_{\ell}})$，$b^{(\ell)}_{j}\sim\mathcal{N}(0,1)$（有时也取 $b^{(\ell)}_{j}=0$）来补偿。两种参数化（有或没有因子 $\frac{1}{\sqrt{n_{\ell}}}$ 和 $\beta$）所能表示的函数集合 $F^{(L)}(\mathbb{R}^{P})$ 是相同的，但实现函数对连接的导数 $\partial_{W^{(\ell)}_{ij}}F^{(L)}$ 和对偏置的导数 $\partial_{b^{(\ell)}_{j}}F^{(L)}$，相对于经典参数化分别被缩放了 $\frac{1}{\sqrt{n_{\ell}}}$ 和 $\beta$。

The factors $\frac{1}{\sqrt{n_{\ell}}}$ are key to obtaining a consistent asymptotic behavior of neural networks as the widths of the hidden layers $n_{1},\ldots,n_{L-1}$ grow to infinity. However a side-effect of these factors is that they reduce greatly the influence of the connection weights during training when $n_{\ell}$ is large: the factor $\beta$ is introduced to balance the influence of the bias and connection weights. In our numerical experiments, we take $\beta=0.1$ and use a learning rate of $1.0$, which is larger than usual, see Section 6. This gives a behaviour similar to that of a classical network of width $100$ with a learning rate of $0.01$.

因子 $\frac{1}{\sqrt{n_{\ell}}}$ 是使神经网络在隐层宽度 $n_{1},\ldots,n_{L-1}$ 趋于无穷时具有一致渐近行为的关键。但这些因子的一个副作用是：当 $n_{\ell}$ 很大时，它们会大大削弱连接权重在训练中的影响；引入因子 $\beta$，正是为了平衡偏置与连接权重的影响。在数值实验中，我们取 $\beta=0.1$，并用 $1.0$ 的学习率，这比通常更大，见第 6 节。这给出的行为，类似于宽度为 $100$、学习率为 $0.01$ 的经典网络。

3 Kernel gradient

3 核梯度

The training of an ANN consists in optimizing $f_{\theta}$ in the function space $\mathcal{F}$ with respect to a functional cost $C:\mathcal{F}\to\mathbb{R}$, such as a regression or cross-entropy cost. Even for a convex functional cost $C$, the composite cost $C\circ F^{(L)}:\mathbb{R}^{P}\to\mathbb{R}$ is in general highly non-convex (3). We will show that during training, the network function $f_{\theta}$ follows a descent along the kernel gradient with respect to the Neural Tangent Kernel (NTK) which we introduce in Section 4. This makes it possible to study the training of ANNs in the function space $\mathcal{F}$, on which the cost $C$ is convex.

训练 ANN，就是在函数空间 $\mathcal{F}$ 中相对某个泛函代价 $C:\mathcal{F}\to\mathbb{R}$（例如回归或交叉熵代价）优化 $f_{\theta}$。即便泛函代价 $C$ 是凸的，复合代价 $C\circ F^{(L)}:\mathbb{R}^{P}\to\mathbb{R}$ 一般仍高度非凸（3）。我们将证明，在训练中，网络函数 $f_{\theta}$ 沿着第 4 节引入的神经正切核（NTK）的核梯度下降。于是可以在函数空间 $\mathcal{F}$ 中研究 ANN 的训练，而在这个空间上代价 $C$ 是凸的。

A multi-dimensional kernel $K$ is a function $\mathbb{R}^{n_{0}}\times\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{L}\times n_{L}}$, which maps any pair $(x,x')$ to an $n_{L}\times n_{L}$-matrix such that $K(x,x')=K(x',x)^{T}$ (equivalently $K$ is a symmetric tensor in $\mathcal{F}\otimes\mathcal{F}$). Such a kernel defines a bilinear map on $\mathcal{F}$, taking the expectation over independent $x,x'\sim p^{\mathrm{in}}$:

一个多维核 $K$ 是函数 $\mathbb{R}^{n_{0}}\times\mathbb{R}^{n_{0}}\to\mathbb{R}^{n_{L}\times n_{L}}$，它把任意一对 $(x,x')$ 映到一个 $n_{L}\times n_{L}$ 矩阵，并满足 $K(x,x')=K(x',x)^{T}$（等价地说，$K$ 是 $\mathcal{F}\otimes\mathcal{F}$ 中的对称张量）。这样的核在 $\mathcal{F}$ 上定义一个双线性映射，期望对独立的 $x,x'\sim p^{\mathrm{in}}$ 来取：

$$\langle f,g\rangle_{K}:=\mathbb{E}_{x,x'\sim p^{\mathrm{in}}}\big[f(x)^{T}K(x,x')g(x')\big].$$

The kernel $K$ is positive definite with respect to $\|\cdot\|_{p^{\mathrm{in}}}$ if $\|f\|_{p^{\mathrm{in}}}>0\implies\|f\|_{K}>0$.

若 $\|f\|_{p^{\mathrm{in}}}>0\implies\|f\|_{K}>0$，则称核 $K$ 相对于 $\|\cdot\|_{p^{\mathrm{in}}}$ 是正定的。

We denote by $\mathcal{F}^{*}$ the dual of $\mathcal{F}$ with respect to $p^{\mathrm{in}}$, i.e. the set of linear forms $\mu:\mathcal{F}\to\mathbb{R}$ of the form $\mu=\langle d,\cdot\rangle_{p^{\mathrm{in}}}$ for some $d\in\mathcal{F}$. Two elements of $\mathcal{F}$ define the same linear form if and only if they are equal on the data. The constructions in the paper do not depend on the element $d\in\mathcal{F}$ chosen in order to represent $\mu$ as $\langle d,\cdot\rangle_{p^{\mathrm{in}}}$. Using the fact that the partial application of the kernel $K_{i,\cdot}(x,\cdot)$ is a function in $\mathcal{F}$, we can define a map $\Phi_{K}:\mathcal{F}^{*}\to\mathcal{F}$ mapping a dual element $\mu=\langle d,\cdot\rangle_{p^{\mathrm{in}}}$ to the function $f_{\mu}=\Phi_{K}(\mu)$ with values:

我们把 $\mathcal{F}$ 相对于 $p^{\mathrm{in}}$ 的对偶记作 $\mathcal{F}^{*}$，也就是形如 $\mu=\langle d,\cdot\rangle_{p^{\mathrm{in}}}$（对某个 $d\in\mathcal{F}$）的线性型 $\mu:\mathcal{F}\to\mathbb{R}$ 的集合。$\mathcal{F}$ 中两个元素定义同一个线性型，当且仅当它们在数据上相等。文中的构造不依赖于为把 $\mu$ 写成 $\langle d,\cdot\rangle_{p^{\mathrm{in}}}$ 而选取的那个 $d\in\mathcal{F}$。利用核的偏应用 $K_{i,\cdot}(x,\cdot)$ 是 $\mathcal{F}$ 中的函数，我们可以定义映射 $\Phi_{K}:\mathcal{F}^{*}\to\mathcal{F}$，它把对偶元 $\mu=\langle d,\cdot\rangle_{p^{\mathrm{in}}}$ 映到函数 $f_{\mu}=\Phi_{K}(\mu)$，其取值为：

$$f_{\mu,i}(x)=\mu\,K_{i,\cdot}(x,\cdot)=\langle d,K_{i,\cdot}(x,\cdot)\rangle_{p^{\mathrm{in}}}.$$

For our setup, which is that of a finite dataset $x_{1},\ldots,x_{N}\in\mathbb{R}^{n_{0}}$, the cost functional $C$ only depends on the values of $f\in\mathcal{F}$ at the data points. As a result, the (functional) derivative of the cost $C$ at a point $f_{0}\in\mathcal{F}$ can be viewed as an element of $\mathcal{F}^{*}$, which we write $\partial^{\mathrm{in}}_{f}C|_{f_{0}}$. We denote by $d|_{f_{0}}\in\mathcal{F}$, a corresponding dual element, such that $\partial^{\mathrm{in}}_{f}C|_{f_{0}}=\langle d|_{f_{0}},\cdot\rangle_{p^{\mathrm{in}}}$.

在我们的设定中，数据集 $x_{1},\ldots,x_{N}\in\mathbb{R}^{n_{0}}$ 是有限的，代价泛函 $C$ 只依赖于 $f\in\mathcal{F}$ 在数据点上的取值。因此，代价 $C$ 在点 $f_{0}\in\mathcal{F}$ 处的（泛函）导数可以看成 $\mathcal{F}^{*}$ 中的一个元素，记作 $\partial^{\mathrm{in}}_{f}C|_{f_{0}}$。我们把对应的对偶元记作 $d|_{f_{0}}\in\mathcal{F}$，使得 $\partial^{\mathrm{in}}_{f}C|_{f_{0}}=\langle d|_{f_{0}},\cdot\rangle_{p^{\mathrm{in}}}$。

The kernel gradient $\nabla_{K}C|_{f_{0}}\in\mathcal{F}$ is defined as $\Phi_{K}\big(\partial^{\mathrm{in}}_{f}C|_{f_{0}}\big)$. In contrast to $\partial^{\mathrm{in}}_{f}C$ which is only defined on the dataset, the kernel gradient generalizes to values $x$ outside the dataset thanks to the kernel $K$:

核梯度 $\nabla_{K}C|_{f_{0}}\in\mathcal{F}$ 定义为 $\Phi_{K}\big(\partial^{\mathrm{in}}_{f}C|_{f_{0}}\big)$。与只在数据集上有定义的 $\partial^{\mathrm{in}}_{f}C$ 不同，核梯度借助核 $K$，可以把取值推广到数据集之外的 $x$：

$$\nabla_{K}C|_{f_{0}}(x)=\frac{1}{N}\sum_{j=1}^{N}K(x,x_{j})\,d|_{f_{0}}(x_{j}).$$

A time-dependent function $f(t)$ follows the kernel gradient descent with respect to $K$ if it satisfies the differential equation

若时变函数 $f(t)$ 满足下述微分方程，则称它相对于 $K$ 做核梯度下降：

$$\partial_{t}f(t)=-\nabla_{K}C|_{f(t)}.$$

During kernel gradient descent, the cost $C(f(t))$ evolves as

在核梯度下降中，代价 $C(f(t))$ 的演化是

$$\partial_{t}C|_{f(t)}=-\big\langle d|_{f(t)},\nabla_{K}C|_{f(t)}\big\rangle_{p^{\mathrm{in}}}=-\big\|d|_{f(t)}\big\|_{K}^{2}.$$

Convergence to a critical point of $C$ is hence guaranteed if the kernel $K$ is positive definite with respect to $\|\cdot\|_{p^{\mathrm{in}}}$: the cost is then strictly decreasing except at points such that $\|d|_{f(t)}\|_{p^{\mathrm{in}}}=0$. If the cost is convex and bounded from below, the function $f(t)$ therefore converges to a global minimum as $t\to\infty$.

因此，若核 $K$ 相对于 $\|\cdot\|_{p^{\mathrm{in}}}$ 正定，就可以保证收敛到 $C$ 的临界点：此时除了满足 $\|d|_{f(t)}\|_{p^{\mathrm{in}}}=0$ 的点之外，代价严格下降。若代价还是凸的并且有下界，则函数 $f(t)$ 在 $t\to\infty$ 时收敛到全局最小。

3.1 Random functions approximation

3.1 随机函数逼近

As a starting point to understand the convergence of ANN gradient descent to kernel gradient descent in the infinite-width limit, we introduce a simple model, inspired by the approach of (15).

作为理解 ANN 梯度下降在无限宽极限下收敛到核梯度下降的起点，我们引入一个简单模型，灵感来自（15）的方法。

A kernel $K$ can be approximated by a choice of $P$ random functions $f^{(p)}$ sampled independently from any distribution on $\mathcal{F}$ whose (non-centered) covariance is given by the kernel $K$:

核 $K$ 可以用 $P$ 个随机函数 $f^{(p)}$ 来逼近：这些函数独立地从 $\mathcal{F}$ 上任意一个（非中心）协方差由核 $K$ 给出的分布中抽样：

$$\mathbb{E}\big[f^{(p)}_{k}(x)f^{(p)}_{k'}(x')\big]=K_{kk'}(x,x').$$

These functions define a random linear parametrization $F^{\mathrm{lin}}:\mathbb{R}^{P}\to\mathcal{F}$

这些函数定义了一个随机线性参数化 $F^{\mathrm{lin}}:\mathbb{R}^{P}\to\mathcal{F}$

$$\theta\mapsto f^{\mathrm{lin}}_{\theta}=\frac{1}{\sqrt{P}}\sum_{p=1}^{P}\theta_{p}f^{(p)}.$$

The partial derivatives of the parametrization are given by

该参数化的偏导数由下式给出：

$$\partial_{\theta_{p}}F^{\mathrm{lin}}(\theta)=\frac{1}{\sqrt{P}}f^{(p)}.$$

Optimizing the cost $C\circ F^{\mathrm{lin}}$ through gradient descent, the parameters follow the ODE:

通过梯度下降优化代价 $C\circ F^{\mathrm{lin}}$ 时，参数服从常微分方程：

$$\partial_{t}\theta_{p}(t)=-\partial_{\theta_{p}}(C\circ F^{\mathrm{lin}})(\theta(t))=-\frac{1}{\sqrt{P}}\partial^{\mathrm{in}}_{f}C|_{f^{\mathrm{lin}}_{\theta(t)}}f^{(p)}=-\frac{1}{\sqrt{P}}\big\langle d|_{f^{\mathrm{lin}}_{\theta(t)}},f^{(p)}\big\rangle_{p^{\mathrm{in}}}.$$

As a result the function $f^{\mathrm{lin}}_{\theta(t)}$ evolves according to

于是函数 $f^{\mathrm{lin}}_{\theta(t)}$ 按下面的方式演化：

$$\partial_{t}f^{\mathrm{lin}}_{\theta(t)}=\frac{1}{\sqrt{P}}\sum_{p=1}^{P}\partial_{t}\theta_{p}(t)f^{(p)}=-\frac{1}{P}\sum_{p=1}^{P}\big\langle d|_{f^{\mathrm{lin}}_{\theta(t)}},f^{(p)}\big\rangle_{p^{\mathrm{in}}}f^{(p)},$$

where the right-hand side is equal to the kernel gradient $-\nabla_{\tilde{K}}C$ with respect to the tangent kernel

其中右边等于相对于切核 $\tilde{K}$ 的核梯度 $-\nabla_{\tilde{K}}C$，而

$$\tilde{K}=\sum_{p=1}^{P}\partial_{\theta_{p}}F^{\mathrm{lin}}(\theta)\otimes\partial_{\theta_{p}}F^{\mathrm{lin}}(\theta)=\frac{1}{P}\sum_{p=1}^{P}f^{(p)}\otimes f^{(p)}.$$

This is a random $n_{L}$-dimensional kernel with values $\tilde{K}_{ii'}(x,x')=\frac{1}{P}\sum_{p=1}^{P}f^{(p)}_{i}(x)f^{(p)}_{i'}(x')$.

这是一个随机的 $n_{L}$ 维核，取值为 $\tilde{K}_{ii'}(x,x')=\frac{1}{P}\sum_{p=1}^{P}f^{(p)}_{i}(x)f^{(p)}_{i'}(x')$。

Performing gradient descent on the cost $C\circ F^{\mathrm{lin}}$ is therefore equivalent to performing kernel gradient descent with the tangent kernel $\tilde{K}$ in the function space. In the limit as $P\to\infty$, by the law of large numbers, the (random) tangent kernel $\tilde{K}$ tends to the fixed kernel $K$, which makes this method an approximation of kernel gradient descent with respect to the limiting kernel $K$.

因此，对代价 $C\circ F^{\mathrm{lin}}$ 做梯度下降，等价于在函数空间中用切核 $\tilde{K}$ 做核梯度下降。在 $P\to\infty$ 的极限下，由大数定律，（随机）切核 $\tilde{K}$ 趋于固定核 $K$，于是这一方法成为相对于极限核 $K$ 的核梯度下降的一种逼近。

4 Neural tangent kernel

4 神经正切核

For ANNs trained using gradient descent on the composition $C\circ F^{(L)}$, the situation is very similar to that studied in the Section 3.1. During training, the network function $f_{\theta}$ evolves along the (negative) kernel gradient

对用梯度下降训练复合代价 $C\circ F^{(L)}$ 的 ANN 而言，情形与第 3.1 节所研究的非常相似。在训练中，网络函数 $f_{\theta}$ 沿着（负的）核梯度演化：

$$\partial_{t}f_{\theta}(t)=-\nabla_{\Theta^{(L)}}C|_{f_{\theta}(t)}$$

with respect to the neural tangent kernel (NTK)

这里相对于的是神经正切核（NTK）

$$\Theta^{(L)}(\theta)=\sum_{p=1}^{P}\partial_{\theta_{p}}F^{(L)}(\theta)\otimes\partial_{\theta_{p}}F^{(L)}(\theta).$$

However, in contrast to $F^{\mathrm{lin}}$, the realization function $F^{(L)}$ of ANNs is not linear. As a consequence, the derivatives $\partial_{\theta_{p}}F^{(L)}(\theta)$ and the neural tangent kernel depend on the parameters $\theta$. The NTK is therefore random at initialization and varies during training, which makes the analysis of the convergence of $f_{\theta}$ more delicate.

然而，与 $F^{\mathrm{lin}}$ 不同，ANN 的实现函数 $F^{(L)}$ 不是线性的。因此，导数 $\partial_{\theta_{p}}F^{(L)}(\theta)$ 以及神经正切核都依赖于参数 $\theta$。于是 NTK 在初始化时是随机的，并会在训练中变化，这使得对 $f_{\theta}$ 收敛的分析更为细致。

In the next subsections, we show that, in the infinite-width limit, the NTK becomes deterministic at initialization and stays constant during training. Since $f_{\theta}$ at initialization is Gaussian in the limit, the asymptotic behavior of $f_{\theta}$ during training can be explicited in the function space $\mathcal{F}$.

在接下来的小节中，我们证明：在无限宽极限下，NTK 在初始化时变为确定的，并在训练中保持不变。由于 $f_{\theta}$ 在初始化时极限下是高斯的，训练中 $f_{\theta}$ 的渐近行为可以在函数空间 $\mathcal{F}$ 中显式写出。

4.1 Initialization

4.1 初始化

As observed in (12; 9), the output functions $f_{\theta,i}$ for $i=1,\ldots,n_{L}$ tend to iid Gaussian processes in the infinite-width limit (a proof in our setup is given in the appendix):

正如（12; 9）所观察到的，输出函数 $f_{\theta,i}$（$i=1,\ldots,n_{L}$）在无限宽极限下趋于独立同分布的高斯过程（我们设定下的证明见附录）：

Proposition 1. For a network of depth $L$ at initialization, with a Lipschitz nonlinearity $\sigma$, and in the limit as $n_{1},\ldots,n_{L-1}\to\infty$, the output functions $f_{\theta,k}$, for $k=1,\ldots,n_{L}$, tend (in law) to iid centered Gaussian processes of covariance $\Sigma^{(L)}$, where $\Sigma^{(L)}$ is defined recursively by:

命题 1. 对深度为 $L$、非线性 $\sigma$ 为 Lipschitz 的网络，在初始化时，当 $n_{1},\ldots,n_{L-1}\to\infty$ 时，输出函数 $f_{\theta,k}$（$k=1,\ldots,n_{L}$）依分布趋于协方差为 $\Sigma^{(L)}$ 的独立同分布中心高斯过程，其中 $\Sigma^{(L)}$ 由下述递归定义：

$$\Sigma^{(1)}(x,x')=\frac{1}{n_{0}}x^{T}x'+\beta^{2},\qquad \Sigma^{(L+1)}(x,x')=\mathbb{E}_{f\sim\mathcal{N}(0,\Sigma^{(L)})}\big[\sigma(f(x))\sigma(f(x'))\big]+\beta^{2},$$

taking the expectation with respect to a centered Gaussian process $f$ of covariance $\Sigma^{(L)}$.

期望是对协方差为 $\Sigma^{(L)}$ 的中心高斯过程 $f$ 来取的。

Remark 2. Strictly speaking, the existence of a suitable Gaussian measure with covariance $\Sigma^{(L)}$ is not needed: we only deal with the values of $f$ at $x,x'$ (the joint measure on $f(x),f(x')$ is simply a Gaussian vector in 2D). For the same reasons, in the proof of Proposition 1 and Theorem 1, we will freely speak of Gaussian processes without discussing their existence.

注 2. 严格来说，并不需要存在一个协方差为 $\Sigma^{(L)}$ 的合适高斯测度：我们只处理 $f$ 在 $x,x'$ 处的取值（$f(x),f(x')$ 上的联合测度不过是二维高斯向量）。出于同样的原因，在命题 1 和定理 1 的证明中，我们将直接谈论高斯过程，而不讨论其存在性。

The first key result of our paper (proven in the appendix) is the following: in the same limit, the Neural Tangent Kernel (NTK) converges in probability to an explicit deterministic limit.

本文的第一个关键结果（证明见附录）如下：在同一极限下，神经正切核（NTK）依概率收敛到一个可以显式写出的确定性极限。

Theorem 1. For a network of depth $L$ at initialization, with a Lipschitz nonlinearity $\sigma$, and in the limit as the layers width $n_{1},\ldots,n_{L-1}\to\infty$, the NTK $\Theta^{(L)}$ converges in probability to a deterministic limiting kernel:

定理 1. 对深度为 $L$、非线性 $\sigma$ 为 Lipschitz 的网络，在初始化时，当各层宽度 $n_{1},\ldots,n_{L-1}\to\infty$ 时，NTK $\Theta^{(L)}$ 依概率收敛到一个确定性的极限核：

$$\Theta^{(L)}\to\Theta^{(L)}_{\infty}\otimes\mathrm{Id}_{n_{L}}.$$

The scalar kernel $\Theta^{(L)}_{\infty}:\mathbb{R}^{n_{0}}\times\mathbb{R}^{n_{0}}\to\mathbb{R}$ is defined recursively by

标量核 $\Theta^{(L)}_{\infty}:\mathbb{R}^{n_{0}}\times\mathbb{R}^{n_{0}}\to\mathbb{R}$ 由下述递归定义：

$$\Theta^{(1)}_{\infty}(x,x')=\Sigma^{(1)}(x,x'),\qquad \Theta^{(L+1)}_{\infty}(x,x')=\Theta^{(L)}_{\infty}(x,x')\,\dot{\Sigma}^{(L+1)}(x,x')+\Sigma^{(L+1)}(x,x'),$$

where

其中

$$\dot{\Sigma}^{(L+1)}(x,x')=\mathbb{E}_{f\sim\mathcal{N}(0,\Sigma^{(L)})}\big[\dot{\sigma}(f(x))\dot{\sigma}(f(x'))\big],$$

taking the expectation with respect to a centered Gaussian process $f$ of covariance $\Sigma^{(L)}$, and where $\dot{\sigma}$ denotes the derivative of $\sigma$.

期望是对协方差为 $\Sigma^{(L)}$ 的中心高斯过程 $f$ 来取的，而 $\dot{\sigma}$ 表示 $\sigma$ 的导数。

Remark 3. By Rademacher’s theorem, $\dot{\sigma}$ is defined everywhere, except perhaps on a set of zero Lebesgue measure.

注 3. 由 Rademacher 定理，$\dot{\sigma}$ 几乎处处有定义，至多在一个 Lebesgue 测度为零的集合上例外。

Note that the limiting $\Theta^{(L)}_{\infty}$ only depends on the choice of $\sigma$, the depth of the network and the variance of the parameters at initialization (which is equal to $1$ in our setting).

注意，极限核 $\Theta^{(L)}_{\infty}$ 只依赖于 $\sigma$ 的选取、网络深度，以及参数在初始化时的方差（在我们的设定中等于 $1$）。

4.2 Training

4.2 训练

Our second key result is that the NTK stays asymptotically constant during training. This applies for a slightly more general definition of training: the parameters are updated according to a training direction $d_{t}\in\mathcal{F}$:

我们的第二个关键结果是：NTK 在训练中渐近保持不变。这对稍更一般的训练定义也成立：参数按训练方向 $d_{t}\in\mathcal{F}$ 更新：

$$\partial_{t}\theta_{p}(t)=\big\langle\partial_{\theta_{p}}F^{(L)}(\theta(t)),d_{t}\big\rangle_{p^{\mathrm{in}}}.$$

In the case of gradient descent, $d_{t}=-d|_{f_{\theta}(t)}$ (see Section 3), but the direction may depend on another network, as is the case for e.g. Generative Adversarial Networks (6). We only assume that the integral $\int_{0}^{T}\|d_{t}\|_{p^{\mathrm{in}}}\,dt$ stays stochastically bounded as the width tends to infinity, which is verified for e.g. least-squares regression, see Section 5.

在梯度下降的情形，$d_{t}=-d|_{f_{\theta}(t)}$（见第 3 节），但该方向也可以依赖于另一个网络，例如生成对抗网络（6）。我们只假定，当宽度趋于无穷时，积分 $\int_{0}^{T}\|d_{t}\|_{p^{\mathrm{in}}}\,dt$ 保持随机有界；这对最小二乘回归等情形是成立的，见第 5 节。

Theorem 2. Assume that $\sigma$ is a Lipschitz, twice differentiable nonlinearity function, with bounded second derivative. For any $T$ such that the integral $\int_{0}^{T}\|d_{t}\|_{p^{\mathrm{in}}}\,dt$ stays stochastically bounded, as $n_{1},\ldots,n_{L-1}\to\infty$, we have, uniformly for $t\in[0,T]$,

定理 2. 设 $\sigma$ 是 Lipschitz、二阶可微且二阶导数有界的非线性函数。对任意使积分 $\int_{0}^{T}\|d_{t}\|_{p^{\mathrm{in}}}\,dt$ 保持随机有界的 $T$，当 $n_{1},\ldots,n_{L-1}\to\infty$ 时，我们有：对 $t\in[0,T]$ 一致地，

$$\Theta^{(L)}(t)\to\Theta^{(L)}_{\infty}\otimes\mathrm{Id}_{n_{L}}.$$

As a consequence, in this limit, the dynamics of $f_{\theta}$ is described by the differential equation

因此，在这一极限下，$f_{\theta}$ 的动力学由下述微分方程描述：

$$\partial_{t}f_{\theta}(t)=\Phi_{\Theta^{(L)}_{\infty}\otimes\mathrm{Id}_{n_{L}}}\big(\langle d_{t},\cdot\rangle_{p^{\mathrm{in}}}\big).$$

Remark 4. As the proof of the theorem (in the appendix) shows, the variation during training of the individual activations in the hidden layers shrinks as their width grows. However their collective variation is significant, which allows the parameters of the lower layers to learn: in the formula of the limiting NTK $\Theta^{(L+1)}_{\infty}(x,x')$ in Theorem 1, the second summand $\Sigma^{(L+1)}$ represents the learning due to the last layer, while the first summand represents the learning performed by the lower layers.

注 4. 正如定理证明（见附录）所示，隐层中单个激活在训练中的变化会随着宽度增大而缩小。但它们的集体变化仍是显著的，这使较低层的参数能够学习：在定理 1 的极限 NTK 公式 $\Theta^{(L+1)}_{\infty}(x,x')$ 中，第二项 $\Sigma^{(L+1)}$ 表示最后一层所做的学习，而第一项表示较低层所做的学习。

As discussed in Section 3, the convergence of kernel gradient descent to a critical point of the cost $C$ is guaranteed for positive definite kernels. The limiting NTK is positive definite if the span of the derivatives $\partial_{\theta_{p}}F^{(L)}$, $p=1,\ldots,P$ becomes dense in $\mathcal{F}$ w.r.t. the $p^{\mathrm{in}}$-norm as the width grows to infinity. It seems natural to postulate that the span of the preactivations of the last layer (which themselves appear in $\partial_{\theta_{p}}F^{(L)}$, corresponding to the connection weights of the last layer) becomes dense in $\mathcal{F}$, for a large family of measures $p^{\mathrm{in}}$ and nonlinearities (see e.g. (7; 10) for classical theorems about ANNs and approximation).

如第 3 节所讨论的，对正定核而言，核梯度下降保证收敛到代价 $C$ 的临界点。若当宽度趋于无穷时，导数 $\partial_{\theta_{p}}F^{(L)}$（$p=1,\ldots,P$）的张成在 $p^{\mathrm{in}}$-范数下在 $\mathcal{F}$ 中变得稠密，则极限 NTK 是正定的。一个自然的假定是：最后一层预激活的张成（它们本身出现在对应于最后一层连接权重的 $\partial_{\theta_{p}}F^{(L)}$ 中）对很大一类测度 $p^{\mathrm{in}}$ 和非线性，会在 $\mathcal{F}$ 中变得稠密（关于 ANN 与逼近的经典定理，例如见（7; 10））。

5 Least-squares regression

5 最小二乘回归

Given a goal function $f^{*}$ and input distribution $p^{\mathrm{in}}$, the least-squares regression cost is

给定目标函数 $f^{*}$ 和输入分布 $p^{\mathrm{in}}$，最小二乘回归代价是

$$C(f)=\frac{1}{2}\|f-f^{*}\|_{p^{\mathrm{in}}}^{2}=\frac{1}{2}\mathbb{E}_{x\sim p^{\mathrm{in}}}\big[\|f(x)-f^{*}(x)\|^{2}\big].$$

Theorems 1 and 2 apply to an ANN trained on such a cost. Indeed the norm of the training direction $\|d(f)\|_{p^{\mathrm{in}}}=\|f^{*}-f\|_{p^{\mathrm{in}}}$ is strictly decreasing during training, bounding the integral. We are therefore interested in the behavior of a function $f_{t}$ during kernel gradient descent with a kernel $K$ (we are of course especially interested in the case $K=\Theta^{(L)}_{\infty}\otimes\mathrm{Id}_{n_{L}}$):

定理 1 和定理 2 适用于在这种代价上训练的 ANN。事实上，训练方向的范数 $\|d(f)\|_{p^{\mathrm{in}}}=\|f^{*}-f\|_{p^{\mathrm{in}}}$ 在训练中严格下降，从而给出积分的界。因此我们关心函数 $f_{t}$ 在以核 $K$ 做核梯度下降时的行为（我们当然尤其关心 $K=\Theta^{(L)}_{\infty}\otimes\mathrm{Id}_{n_{L}}$ 的情形）：

$$\partial_{t}f_{t}=\Phi_{K}\big(\langle f^{*}-f,\cdot\rangle_{p^{\mathrm{in}}}\big).$$

The solution of this differential equation can be expressed in terms of the map $\Pi:f\mapsto\Phi_{K}\big(\langle f,\cdot\rangle_{p^{\mathrm{in}}}\big)$:

这一微分方程的解可以用映射 $\Pi:f\mapsto\Phi_{K}\big(\langle f,\cdot\rangle_{p^{\mathrm{in}}}\big)$ 来表达：

$$f_{t}=f^{*}+e^{-t\Pi}(f_{0}-f^{*})$$

where $e^{-t\Pi}=\sum_{k=0}^{\infty}\frac{(-t)^{k}}{k!}\Pi^{k}$ is the exponential of $-t\Pi$. If $\Pi$ can be diagonalized by eigenfunctions $f^{(i)}$ with eigenvalues $\lambda_{i}$, the exponential $e^{-t\Pi}$ has the same eigenfunctions with eigenvalues $e^{-t\lambda_{i}}$.

其中 $e^{-t\Pi}=\sum_{k=0}^{\infty}\frac{(-t)^{k}}{k!}\Pi^{k}$ 是 $-t\Pi$ 的指数。若 $\Pi$ 可以被特征函数 $f^{(i)}$（特征值为 $\lambda_{i}$）对角化，则指数 $e^{-t\Pi}$ 有相同的特征函数，对应特征值为 $e^{-t\lambda_{i}}$。

For a finite dataset $x_{1},\ldots,x_{N}$ of size $N$, the map $\Pi$ takes the form

对大小为 $N$ 的有限数据集 $x_{1},\ldots,x_{N}$，映射 $\Pi$ 取如下形式：

$$\Pi(f)_{k}(x)=\frac{1}{N}\sum_{i=1}^{N}\sum_{k'=1}^{n_{L}}f_{k'}(x_{i})K_{kk'}(x_{i},x).$$

The map $\Pi$ has at most $Nn_{L}$ positive eigenfunctions, and they are the kernel principal components $f^{(1)},\ldots,f^{(Nn_{L})}$ of the data with respect to to the kernel $K$ (17; 18). The corresponding eigenvalues $\lambda_{i}$ is the variance captured by the component.

映射 $\Pi$ 至多有 $Nn_{L}$ 个正的特征函数，它们正是数据相对于核 $K$ 的核主成分 $f^{(1)},\ldots,f^{(Nn_{L})}$（17; 18）。相应的特征值 $\lambda_{i}$ 是该成分所捕获的方差。

Decomposing the difference $(f^{*}-f_{0})=\Delta^{0}_{f}+\Delta^{1}_{f}+\cdots+\Delta^{Nn_{L}}_{f}$ along the eigenspaces of $\Pi$, the trajectory of the function $f_{t}$ reads

把差 $(f^{*}-f_{0})=\Delta^{0}_{f}+\Delta^{1}_{f}+\cdots+\Delta^{Nn_{L}}_{f}$ 沿 $\Pi$ 的特征空间分解后，函数 $f_{t}$ 的轨迹为

$$f_{t}=f^{*}+\Delta^{0}_{f}+\sum_{i=1}^{Nn_{L}}e^{-t\lambda_{i}}\Delta^{i}_{f},$$

where $\Delta^{0}_{f}$ is in the kernel (null-space) of $\Pi$ and $\Delta^{i}_{f}\propto f^{(i)}$.

其中 $\Delta^{0}_{f}$ 在 $\Pi$ 的核（零空间）中，且 $\Delta^{i}_{f}\propto f^{(i)}$。

The above decomposition can be seen as a motivation for the use of early stopping. The convergence is indeed faster along the eigenspaces corresponding to larger eigenvalues $\lambda_{i}$. Early stopping hence focuses the convergence on the most relevant kernel principal components, while avoiding to fit the ones in eigenspaces with lower eigenvalues (such directions are typically the ‘noisier’ ones: for instance, in the case of the RBF kernel, lower eigenvalues correspond to high frequency functions).

上述分解可以看成使用早停的一种动机。沿对应于较大特征值 $\lambda_{i}$ 的特征空间，收敛确实更快。因此早停把收敛集中在最相关的核主成分上，同时避免去拟合特征值较低的特征空间中的成分（这些方向通常更“噪”：例如对 RBF 核，较小的特征值对应于高频函数）。

Note that by the linearity of the map $e^{-t\Pi}$, if $f_{0}$ is initialized with a Gaussian distribution (as is the case for ANNs in the infinite-width limit), then $f_{t}$ is Gaussian for all times $t$. Assuming that the kernel is positive definite on the data (implying that the $Nn_{L}\times Nn_{L}$ Gram marix $\tilde{K}=(K_{kk'}(x_{i},x_{j}))_{ik,jk'}$ is invertible), as $t\to\infty$ limit, we get that $f_{\infty}=f^{*}+\Delta^{0}_{f}=f_{0}-\sum_{i}\Delta^{i}_{f}$ takes the form