Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training

扩散模型为何不记忆：训练中的隐式动力学正则

Tony Bonnaire † Affiliation:  LPENS Affiliation:  Université PSL, Paris Email:  tony.bonnaire@phys.ens.fr    Raphaël Urfin † Affiliation:  LPENS Affiliation:  Université PSL, Paris Email:  raphael.urfin@phys.ens.fr    Giulio Biroli Affiliation:  LPENS Affiliation:  Université PSL, Paris Email:  giulio.biroli@phys.ens.fr    Marc Mézard Affiliation:  Department of Computing Sciences Affiliation:  Bocconi University, Milano Email:  marc.mezard@unibocconi.it

Tony Bonnaire †（LPENS / Université PSL, Paris）   Raphaël Urfin †（LPENS / Université PSL, Paris）   Giulio Biroli（LPENS / Université PSL, Paris）   Marc Mézard（Bocconi University, Milano）

Abstract

摘要

Diffusion models have achieved remarkable success across a wide range of generative tasks. A key challenge is understanding the mechanisms that prevent their memorization of training data and allow generalization. In this work, we investigate the role of the training dynamics in the transition from generalization to memorization. Through extensive experiments and theoretical analysis, we identify two distinct timescales: an early time $\tau_{\mathrm{gen}}$ at which models begin to generate high-quality samples, and a later time $\tau_{\mathrm{mem}}$ beyond which memorization emerges. Crucially, we find that $\tau_{\mathrm{mem}}$ increases linearly with the training set size $n$, while $\tau_{\mathrm{gen}}$ remains constant. This creates a growing window of training times with $n$ where models generalize effectively, despite showing strong memorization if training continues beyond it. It is only when $n$ becomes larger than a model-dependent threshold that overfitting disappears at infinite training times. These findings reveal a form of implicit dynamical regularization in the training dynamics, which allow to avoid memorization even in highly overparameterized settings. Our results are supported by numerical experiments with standard U-Net architectures on realistic and synthetic datasets, and by a theoretical analysis using a tractable random features model studied in the high-dimensional limit.

扩散模型在各类生成任务上取得了显著成功。关键挑战之一是理解是什么机制阻止它们记忆训练数据、并允许泛化。本文研究训练动力学在从泛化转向记忆中的作用。通过大量实验与理论分析，我们识别出两个不同的时间尺度：较早的 $\tau_{\mathrm{gen}}$，模型开始生成高质量样本；较晚的 $\tau_{\mathrm{mem}}$，此后记忆开始出现。关键地，$\tau_{\mathrm{mem}}$ 随训练集大小 $n$ 线性增长，而 $\tau_{\mathrm{gen}}$ 基本保持不变。这使随 $n$ 增大出现一段不断扩大的训练时间窗口，模型在其中有效泛化；若继续训练超过该窗口，则会表现出强烈记忆。它源于梯度下降训练神经网络得分函数时的隐式动力学正则。我们在 CelebA 上的 U-Net、随机特征模型与高斯混合等设定中验证了这一图像，并给出解析结果支持。

1 Introduction

1 引言

Diffusion Models [49, 20, 54, 55] achieve state-of-the-art performance in a wide variety of AI tasks such as the generation of images [45], audios [64], videos [33], and scientific data [31, 40]. This class of generative models, inspired by out-of-equilibrium thermodynamics [49], corresponds to a two-stage process: the first one, called forward, gradually adds noise to a data, whereas the second one, called backward, generates new data by denoising Gaussian white noise samples. In DMs, the reverse process typically involves solving a stochastic differential equation (SDE) with a force field called score. However, it is also possible to define a deterministic transport through an ordinary differential equation (ODE), treating the score as a velocity field, an approach that is for instance followed in flow matching [32].

扩散模型（Diffusion Models）[49, 20, 54, 55] 在图像[45]、音频[64]、视频[33] 与科学数据[31, 40] 等广泛 AI 任务上达到最先进水平。这类生成模型受非平衡热力学启发[49]，对应两阶段过程：前向过程逐步向数据加噪，后向过程通过对高斯白噪声样本去噪来生成新数据。在扩散模型中，反向动力学由得分函数（score）$\mathbf{s}(\mathbf{x},t)=\nabla_{\mathbf{x}}\log P_t(\mathbf{x})$ 驱动。

Understanding the generalization properties of score-based generative methods is a central issue in machine learning, and a particularly important question is how memorization of the training set is avoided in practice. A model without regularization achieving zero training loss only learns the empirical score, and is bound to reproduce samples of the training dataset at the end of the backward process. This memorization regime [30, 6] is empirically observed when the training set is small and disappears when it increases beyond a model-dependent threshold [24]. Understanding the mechanisms controlling this change of regimes from memorization to generalization is a central challenge for both theory and applications. Model regularization and inductive biases imposed by the network architecture were shown to play a role [25, 47], as well as a dynamical regularization due to the finiteness of the learning rate [62]. However, the regime shift described above is consistently observed even in models where all these regularization mechanisms are present. This suggests that the core mechanism behind the transition from memorization to generalization lies elsewhere. In this work, we demonstrate – first through numerical experiments, and then via the theoretical analysis of a simplified model – that this transition is driven by an implicit dynamical bias towards generalizing solutions emerging in the training, which allows to avoid the memorization phase.

理解基于得分的生成方法的泛化性质是机器学习的中心问题之一，尤其重要的是：实践中如何避免对训练集的记忆。一个没有正则、把训练损失降到零的模型只会学到经验得分，并在后向过程结束时复现训练样本。这种记忆机制[30, 6] 在训练集较小、维度较高时会被经验观察到；但在大规模设定中，记忆往往在 $n$ 远未指数级增长于维度 $d$ 之前就消失了。

Figure 1: Qualitative summary of our contributions. (Left) Illustration of the training dynamics of a diffusion model. Depending on the training time $\tau$, we identify three regimes measured by the inverse quality of the generated samples (blue curve) and their memorization fraction (red curve). The generalization regime extends over a large window of training times which increases with the training set size $n$. On top, we show a one dimensional example of the learned score function during training (orange). The gray line gives the exact empirical score, at a given noise level, while the black dashed line corresponds to the true (population) score. (Right) Phase diagram in the $(n,p)$ plane illustrating three regimes of diffusion models: Memorization when $n$ is sufficiently small at fixed $p$, Architectural Regularization for $n>n^{\star}(p)$ (which is model and dataset dependent, as discussed in [15, 25]), and Dynamical Regularization, corresponding to a large intermediate generalization regime obtained when the training dynamics is stopped early, i.e. $\tau\in\left[\tau_{\mathrm{gen}},\tau_{\mathrm{mem}}\right]$.

图1:我们的贡献的定性摘要。 **(左)* 介绍推广模式的培训动态。 根据训练时间$\tau$,我们确定了三种由生成样品(蓝色曲线)的反向质量及其记忆分数(红色曲线)测量的制度。 通用制度覆盖了培训时间的大窗口,随着培训设置的大小$n$的增加。 在顶部,我们傀儡在训练期间(橙色)展示一个驯服时学习得分功能的一雌性例子畸形.侬. 灰色线给出精确的经验正在进行得分 这就是 天呐 当一个噪声级别上 黑色破 我来对应真实的(人口)分数 (权利) XXMAdong2XX平面中的相位都有权显示有代表性的三种扩散模式:XXMA的人民币在固定的$p$时足够小的纪念物;$n>n^{\star}(p)$的建筑规范化(这是模型在上述[15,25]中所讨论的橱窗和数据集的依赖物);动态规范化,相当于早期停止培训动力学时获得的大型中间泛化制度,即XX参与了$\tau\in\left[\tau_{\mathrm{gen}},\tau_{\mathrm{mem}}\right]$.

Contributions and theoretical picture.

We investigate the dynamics of score learning using gradient descent, both numerically and analytically, and study the generation properties of the score depending on the time $\tau$ at which the training is stopped. The theoretical picture built from our results and combining several findings from the recent literature is illustrated in Fig. 1. The two main parameters are the size of the training set $n$ and the expressivity of the class of score functions on which one trains the model, characterized by a number of parameters $p$; when both $n$ and $p$ are large one can identify three main regimes. Given $p$, if $n$ is larger than $n^{*}(p)$ (which depends on the training set and on the class of scores), the score model is not expressive enough to represent the empirical score associated to $n$ data, and instead provides a smooth interpolation, approximately independent of the training set. In this regime, even with a very large training time $\tau\to\infty$, memorization does not occur because the model is regularized by its architecture and the finite number of parameters. When $n<n^{*}(p)$ the model is expressive enough to memorize, and two timescales emerge during training: one, $\tau_{\mathrm{gen}}$, is the minimum training time required to achieve high-quality data generation; the second, $\tau_{\mathrm{mem}}>\tau_{\mathrm{gen}}$, signals when further training induces memorization, and causes the model to increasingly reproduce the training samples (left panel). The first timescale, $\tau_{\mathrm{gen}}$, is found independent of $n$, whereas the second, $\tau_{\mathrm{mem}}$, grows approximately linearly with $n$, thus opening a large window of training times during which the model generalizes if early stopped when $\tau\in[\tau_{\mathrm{gen}},\tau_{\mathrm{mem}}]$. Our results shows that implicit dynamical regularization in training plays a crucial role in score-based generative models, substantially enlarging the generalization regime (see right panel of Fig. 1), and hence allowing to avoid memorization even in highly overparameterized settings. We find that the key mechanism behind the widening gap between $\tau_{\mathrm{gen}}$ and $\tau_{\mathrm{mem}}$ is the irregularity of the empirical score at low noise level and large $n$. In this regime the models used to approximate the score provide a smooth interpolation that remains stable for a long period of training times and closely approximates the population score, a behavior likely rooted in the spectral bias of neural networks [42]. Only at very long training times do the dynamics converge to the low lying minimum corresponding to the empirical score, leading to memorization (as illustrated in the one-dimensional examples in the left panel of Fig. 1).

贡献与理论图景。

我们用梯度下降从数值与解析两方面研究得分学习动力学，并考察训练停止时刻 $\tau$ 对生成性质的影响。综合本文结果与近期文献若干发现所形成的理论图景见图 1。两个主要参数是训练集大小 $n$，以及所训练得分函数类的表达能力（以参数量 $p$ 刻画）；当 $n$ 与 $p$ 都很大时，可以区分三种主要机制。给定 $p$，若 $n$ 大于 $n^{*}(p)$（依赖于训练集与得分函数类），得分模型不足以表示与 $n$ 个数据对应的经验得分，而会给出近似不依赖训练集的平滑插值。在此机制下，即便训练时间 $\tau\to\infty$ 很长，记忆也不会发生，因为架构与有限参数量本身构成正则。当 $n<n^{*}(p)$ 时模型足以记忆，训练中出现两个时间尺度：$\tau_{\mathrm{gen}}$ 是达到高质量数据生成所需的最短训练时间；$\tau_{\mathrm{mem}}>\tau_{\mathrm{gen}}$ 标志着继续训练会诱发记忆，使模型越来越多地复现训练样本（左图）。第一个时间尺度 $\tau_{\mathrm{gen}}$ 与 $n$ 无关，第二个 $\tau_{\mathrm{mem}}$ 近似随 $n$ 线性增长，从而在早停于 $\tau\in[\tau_{\mathrm{gen}},\tau_{\mathrm{mem}}]$ 时打开一大段泛化窗口。结果表明，训练中的隐式动力学正则对基于得分的生成模型至关重要，显著扩大了泛化区间（见图 1 右），即便在高度过参数化设定下也能避免记忆。我们发现 $\tau_{\mathrm{gen}}$ 与 $\tau_{\mathrm{mem}}$ 间距拉大的关键机制，是低噪声、大 $n$ 下经验得分的不规则性。此时用于近似得分的模型给出平滑插值，在很长一段训练时间内保持稳定，并紧密逼近总体得分，这一行为很可能源于神经网络的频谱偏置 [42]。只有在极长训练时间后，动力学才收敛到对应经验得分的低洼极小，从而导致记忆（如图 1 左的一维示例所示）。

The theoretical picture described above is based on our numerical and analytical results, and builds up on previous works, in particular numerical analysis characterizing the memorization–generalization transition [18, 63], analytical works on memorization of DMs [15, 25, 24], and studies on the spectral bias of deep neural networks [42]. Our numerical experiments use a class of scores based on a realistic U-Net [46] trained on downscaled images of the CelebA dataset [34]. By varying $n$ and $p$, we measure the evolution of the sample quality (through FID) and the fraction of memorization during learning, which support the theoretical scenario presented in Fig. 1. Additional experimental results on synthetic data are provided in Supplemental Material (SM, Sects. A and B). On the analytical side, we focus on a class of scores constructed from random features and simplified models of data, following [15]. In this setting, the timescales of training dynamics correspond directly to the inverse eigenvalues of the random feature correlation matrix. Leveraging tools from random matrix theory, we compute the spectrum in the limit of large datasets, high-dimensional data, and overparameterized models. This analysis reveals, in a fully tractable way, how the theoretical picture of Fig. 1 emerges within the random feature framework.

以上描述的理论画面以我们的数值和分析结果为基础,并借鉴了以往的著作,尤其是描述记忆-泛化过渡的数值分析[18,63],关于DMs的记忆分析作品[15,25,24],关于深神经网络光谱偏差的研究[42]. 我们的数字实验使用基于一个现实的U-Net的分数等级[46],训练了CelebA数据集的降级图像[34]. 通过不同的$n$和$p$,我们测量样本质量的演变(通过FID)和学习过程中的记忆的一小部分,这支持了Fig中提出的理论情景. 1. 关于合成数据的其他实验结果载于补充材料(SM、Sects)。 页:1 在分析方面,我们侧重于从随机特征和简化的数据模型中构建的分数类别,遵循[15]. 在此设定中,训练动力学的时间尺度直接对应随机特征相关性矩阵的反义值. 从随机矩阵理论中利用工具,我们计算出大数据集,高维数据和超参数模型的极限光谱. 这一分析以完全可拉动的方式揭示了图1的理论画面是如何在随机特征框架内出现的.

Related works.

• • The memorization transition in DMs has been the subject of several recent empirical investigations [9, 50, 51] which have demonstrated that state-of-the-art image DMs – including Stable Diffusion and DALL·E – can reproduce a non-negligible portion of their training data, indicating a form of memorization. Several additional works [18, 63] examined how this phenomenon is influenced by factors such as data distribution, model configuration, and training procedure, and provide a strong basis for the numerical part of our work.
• • A series of theoretical studies in the high-dimensional regime have analyzed the memorization–generalization transition during the generative dynamics under the empirical score assumption [6, 1, 57], showing how trajectories are attracted to the training samples. Within this high-dimensional framework, [11, 12, 60, 15] study the score learning for various model classes. In particular, [15] uses a Random Feature Neural Network [43]. The authors compute the asymptotic training and test losses for $\tau\rightarrow\infty$ and relate it to memorization. The theoretical part of our work generalizes this approach to study the role of training dynamics and early stopping in the memorization–generalization transition.
• • Recent works have also uncovered complementary sources of implicit regularization explaining how DMs avoid memorization. Architectural biases and limited network capacity were for instance shown to constrain memorization in [25, 24], and finiteness of the learning rate prevents the model from learning the empirical score in [62]. Also related to our analysis, [29] provides general bounds showing the beneficial role of early stopping the training dynamics to enhance generalization for finitely supported target distributions, as well as a study of its effect for one-dimensional gaussian mixtures.
• • Finally, previous studies on supervised learning [42, 65], and more recently on DMs [59], have shown that deep neural networks display a frequency-dependent learning speed, and hence a learning bias towards low frequency functions. This fact plays an important role in the results we present since the empirical score contains a low frequency part that is close to the population score, and a high-frequency part that is dataset-dependent. To the best of our knowledge, the training time to learn the high-frequency part and hence memorize, that we find to scale with $n$, has not been studied from this perspective in the context of score-based generative methods.

相关作品。

• 模式草案的记忆过渡是最近几次经验调查[9、50、51]的主题,这些调查表明,最先进的模式草案 --灿烂的图像,包括稳定扩散和DALL-E -- -- 可以复制其培训数据中不可忽略的部分,表明一种记忆形式。 另有几部著作[18,63]研究了这一现象如何受到数据分布、模型配置和培训程序等因素的影响,并为我们工作的数字部分提供了坚实的基础。
• 高维度制度的一系列理论研究分析了经验分数假设[6,1,57]下基因动力学期间的记忆-通用过渡,显示了轨迹如何被训练样本所吸引. 在这个高维度框架内,[11,12,60,15]为各种模范班学习分数学习. 特别是[15]使用随机特征神经网络[43]. 作者计算$\tau\rightarrow\infty$的不对称培训和测试损失,并将其与记忆联系起来。 我们工作的理论部分概括了这一方法,以研究培训动态和早期停止记忆-普遍化过渡的作用。
• 最近的工作还发现了间接正规化的补充来源,解释模式草案如何避免记忆。 例如,在[25、24]中,建筑偏见和有限的网络能力被证明制约了记忆,学习率的有限性使模型无法学习[62]中的经验分数。 与我们的分析同样相关的是,[29] 提供了一般性的界限,说明及早停止培训动态对加强有限支持的目标分布的概括性的益处,以及研究其对一维加乌西混合物的影响。
• 最后,以前关于监督学习的研究表明[42、65],最近关于DMs[59]的研究表明,深神经网络显示一个依赖频率的学习速度,因此对低频率功能的学习偏向。 这一事实在我们目前的成果中发挥着重要作用,因为经验分数包含一个接近人口分数的低频部分,以及一个依赖数据集的高频部分. 就我们所知,我们发现使用$n$进行高频部分并因此记住的训练时间,还没有从这一角度在基于分数的基因方法中加以研究。

Setting: generative diffusion and score learning.

Standard DMs define a transport from a target distribution $P_{0}$ in $\mathbb{R}^{d}$ to a Gaussian white noise $\mathcal{N}(0,\bm{I}_{d})$ through a forward process defined as an Ornstein-Uhlenbeck (OU) stochastic differential equation (SDE):

• 设定:基因传播和分数学习姻缘。

标准DMs定义从$\mathbb{R}^{d}$中的目标分布$P_{0}$到高斯白噪 $\mathcal{N}(0,\bm{I}_{d})$,通过一个向前过程定义为Ornstein-Uhlenbeck(OU)斜差方程(SDE):

$$\displaystyle\mathrm{d}\bm{\mathrm{x}}=-\bm{\mathrm{x}}(t)\mathrm{d}t+\mathrm{d}\bm{\mathrm{B}}(t), \tag{1}$$

where $\mathrm{d}\bm{\mathrm{B}}(t)$ is square root of two times a Wiener process. Generation is performed by time-reversing the SDE (1) using the score function $\bm{\mathrm{s}}(\bm{\mathrm{x}},t)=\nabla_{\bm{\mathrm{x}}}\log P_{t}(\bm{\mathrm{x}})$,

其中$\mathrm{d}\bm{\mathrm{B}}(t)$是双倍维纳过程的平方根. 生成是通过使用分数函数$\bm{\mathrm{s}}(\bm{\mathrm{x}},t)=\nabla_{\bm{\mathrm{x}}}\log P_{t}(\bm{\mathrm{x}})$的SDE (1) 时间反转实现的,

$$\displaystyle-\mathrm{d}\bm{\mathrm{x}}=\left[\bm{\mathrm{x}}(t)+2\bm{\mathrm{s}}(\bm{\mathrm{x}},t)\right]\mathrm{d}t+\mathrm{d}\bm{\mathrm{B}}(t), \tag{2}$$

where $P_{t}(\bm{\mathrm{x}})$ is the probability density at time $t$ along the forward process, and the noise $\mathrm{d}\bm{\mathrm{B}}(t)$ is also the square root of two times a Wiener process. As shown in the seminal works [23, 58], $\bm{\mathrm{s}}(\bm{\mathrm{x}},t)$ can be obtained by minimizing the score matching loss

其中$P_{t}(\bm{\mathrm{x}})$是沿前进过程的$t$时间的概率密度,噪声$\mathrm{d}\bm{\mathrm{B}}(t)$也是双倍维纳过程的平方根. 如开创性作品[23,58]所示,$\bm{\mathrm{s}}(\bm{\mathrm{x}},t)$可以通过最小化比分损失获得.

$$\displaystyle\hat{\bm{\mathrm{s}}}(\bm{\mathrm{x}},t)=\arg\min_{\bm{\mathrm{s}}}\mathbb{E}_{\bm{\mathrm{x}}\sim P_{0},\bm{\xi}\sim\mathcal{N}(0,\bm{I}_{d})}\left[\lVert\sqrt{\Delta_{t}}\bm{\mathrm{s}}(\bm{\mathrm{x}}(t),t)+\bm{\xi}\rVert^{2}\right], \tag{3}$$

where $\Delta_{t}=1-e^{-2t}$. In practice, the optimization problem is restricted to a parametrized class of functions $\bm{\mathrm{s}}_{\bm{\theta}}(\bm{\mathrm{x}}(t),t)$ defined, for example, by a neural network with parameters $\bm{\theta}$. The expectation over $\bm{\mathrm{x}}$ is replaced by the empirical average over the training set ($n$ iid samples $\bm{\mathrm{x}}^{\nu}$ drawn from $P_{0}$),

第$\Delta_{t}=1-e^{-2t}$号 在实际操作中,优化问题被限制在被参数$\bm{\theta}$的神经网络所定义的$\bm{\mathrm{s}}_{\bm{\theta}}(\bm{\mathrm{x}}(t),t)$功能的准化类. 对XXMA 这方面的期望被培训组的经验平均值所取代($n$ iid样本 $\bm{\mathrm{x}}^{\nu}$来自$P_{0}$),

$$\displaystyle\mathcal{L}_{t}(\bm{\theta},\{\bm{\mathrm{x}}^{\nu}\}_{\nu=1}^{n})=\frac{1}{n}\sum_{\nu=1}^{n}\mathbb{E}_{\bm{\xi}\sim\mathcal{N}(0,\bm{I}_{d})}\left[\lVert\sqrt{\Delta_{t}}\bm{\mathrm{s}}_{\bm{\theta}}(\bm{\mathrm{x}}^{\nu}(t))+\bm{\xi}\rVert^{2}\right], \tag{4}$$

where $\bm{\mathrm{x}}^{\nu}_{t}(\bm{\xi})=e^{-t}\bm{\mathrm{x}}^{\nu}+\sqrt{\Delta_{t}}\bm{\xi}$. The loss in (4) can be minimized with standard optimizers, such as stochastic gradient descent [44] or Adam [28]. In practice, a single model conditioned on the diffusion time $t$ is trained by integrating (4) over time [26]. The solution of the minimization of (4) is the so-called empirical score (e.g. [6, 30]), defined as $\bm{\mathrm{s}}_{\mathrm{emp}}(\bm{\mathrm{x}},t)=\nabla_{\bm{\mathrm{x}}}\log P_{t}^{\mathrm{emp}}(\bm{\mathrm{x}})$, with

第$\bm{\mathrm{x}}^{\nu}_{t}(\bm{\xi})=e^{-t}\bm{\mathrm{x}}^{\nu}+\sqrt{\Delta_{t}}\bm{\xi}$号 (4)中的损失可以用标准优化器最小化,如斜坡梯度下降[44]或亚当[28]. 在实践中,一个以传播时间为条件的单一模型$t$通过集成(4)在一段时间[26]进行训练. 最小化(4)的解决方案是所谓的经验得分(如[6,30]),定义为$\bm{\mathrm{s}}_{\mathrm{emp}}(\bm{\mathrm{x}},t)=\nabla_{\bm{\mathrm{x}}}\log P_{t}^{\mathrm{emp}}(\bm{\mathrm{x}})$,与.

$$P_{t}^{\mathrm{emp}}(\bm{\mathrm{x}})=\frac{1}{n\left(2\pi\Delta_{t}\right)^{d/2}}\sum_{\nu=1}^{n}e^{-\frac{1}{2\Delta_{t}}\lVert\bm{\mathrm{x}}-\bm{\mathrm{x}}^{\nu}e^{-t}\rVert^{2}_{2}}. \tag{5}$$

This solution is known to inevitably recreate samples of the training set at the end of the generative process (i.e., it perfectly memorizes), unless $n$ grows exponentially with the dimension $d$ [6]. However, this is not the case in many practical applications where memorization is only observed for relatively small values of $n$, and disappears well before $n$ becomes exponentially large in $d$. The empirical minimization performed in practice, within a given class of models and a given minimization procedure, does not drive the optimization to the global minimum of (4), but instead to a smoother estimate of the score that is independent of the training set with good generalization properties [24], as the global minimum of (3) would do. Understanding how it is possible, and in particular the role played by the training dynamics to avoid memorization, is the central aim of the present work.

众所周知,除非$n$与维度$d$[6]成指数增长,否则这种解决方案将不可避免地重现基因过程末端设定的训练样本(即完全记忆). 然而,在许多实际应用中,情况并非如此,因为人们只观察到纪念值相对较小的$n$,而且早在$n$在$d$成指数大之前就已消失。 在实践中,在一定的模型类别和特定最小化程序内进行的经验最小化并不* 推动优化到全球最低(4),而是像全球最低(3)那样,更平稳地估计独立于具有良好一般化特性的培训集的得分[24]。 了解如何做到这一点,特别是了解培训动态在避免记忆方面的作用,是目前工作的核心目标。

2 Generalization and memorization during training of diffusion models

2 扩散模型训练中的泛化与记忆

Figure 2: Memorization transition as a function of the training set size $n$ for U-Net score models on CelebA. (Left) FID (solid lines, left axis) and memorization fraction $f_{\mathrm{mem}}$ (dashed lines, right axis) against training time $\tau$ for various $n$. Inset: normalized memorization fraction $f_{\mathrm{mem}}(\tau)/f_{\mathrm{mem}}(\tau_{\mathrm{max}})$ with the rescaled time $\tau/n$. (Middle) Training (solid lines) and test (dashed lines) loss with $\tau$ for several $n$ at fixed $t=0.01$. Inset: both losses plotted against $\tau/n$. Error bars on the losses are imperceptible. (Right) Generated samples from the model trained with $n=1024$ for $\tau=100$K or $\tau=1.62$M steps, along with their nearest neighbors in the training set.

图2:** 记忆过渡作为培训设置大小$n$用于CelebA上的U-Net分数模型 *(Left) * FID(固态线,左轴)和记忆分数 $f_{\mathrm{mem}}$(dashed line,右轴) 相对于培训时间$\tau$用于各种 $n$. 套期: 普通的记分$f_{\mathrm{mem}}(\tau)/f_{\mathrm{mem}}(\tau_{\mathrm{max}})$与重定时间 $\tau/n$. *(中) * 在固定的$t=0.01$中,若干$n$的培训(固线)和试验(干线)损失为$\tau$。 套期:两项损失均与$\tau/n$相对应。 损失上的错误栏是无法察觉的。 (权利) 用$n=1024$为$\tau=100$ K或$\tau=1.62$ M级台阶培训的模型生成样本,以及培训组中最近的邻居。 **

Data & architecture.

We conduct our experiments on the CelebA face dataset [34], which we convert to grayscale downsampled images of size $d=32\times 32$, and vary the training set size $n$ from 128 up to 32768. Our score model has a U-Net architecture [46] with three resolution levels and a base channel width of $W$ with multipliers 1, 2 and 3 respectively. All our networks are DDPMs [20] trained to predict the injected noise at diffusion time $t$ using SGD with momentum at fixed batch size $\min(n,512)$. The models are all conditioned on $t$, i.e. a single model approximates the score at all times, and make use of a standard sinusoidal position embedding [56] that is added to the features of each resolution. More details about the numerical setup can be found in SM (Sect. A).

QQ 数据架构( A).

我们在CelebA面部数据集上进行实验[34],我们将其转换成灰度下取样的大小为$d=32\times 32$的图像,并将训练集大小从128到32768. 我们的分数模型有一个U-Net架构[46],分辨度分为三个级别,基信道宽度为$W$,乘数分别为1,2和3. 我们所有的网络都是DDPMs [20] ,在扩散时间使用具有固定批量大小的$\min(n,512)$的SGD来预测注入的噪音。 这些模型都以$t$为条件,即一个单一模型在任何时候都大致比分,并使用嵌入每个分辨率特征的标准sinoidal位置[56]. 关于数字设置的更多细节,请参见SM(A节).

Evaluation metrics.

To study the transition from generalization to memorization during training, we monitor the loss (4) during training using a fixed diffusion time $t=0.01$. At various numbers of SGD updates $\tau$, we compute the loss on all $n$ training examples (training loss) and on a held-out test set of 2048 images (test loss). To characterize the score obtained after a training time $\tau$, we assess the originality and quality of samples by generating 10K samples using a DDIM accelerated sampling [52]. We compute (i) the Fréchet-Inception Distance [19] against 10K test samples which we use to identify the generalization time $\tau_{\mathrm{gen}}$; and (ii) the fraction of memorized generated samples $f_{\mathrm{mem}}(\tau)$ granting access to $\tau_{\mathrm{mem}}$, the memorization time. Following previous numerical studies [63, 18], a generated sample $\bm{\mathrm{x}}_{\tau}$ is considered memorized if

• 评价尺度。

为了研究培训期间从一般化向记忆化的过渡,我们利用$t=0.01$固定传播时间监测培训期间的损失(4)。 在各种编号的SGD更新$\tau$中,我们计算所有$n$培训实例的损失(培训损失)和由2048个图像(测试损失)组成的搁置测试集的损失。 为了描述训练时间$\tau$之后获得的分数,我们通过使用DDIM加速采样生成10K样品,评估样品的原生性和质量[52]. 我们计算(一) Fréchet-Inception District [19] 相对于10K测试样本,我们用来确定通用时间$\tau_{\mathrm{gen}}$;和(二) 记忆生成的$f_{\mathrm{mem}}(\tau)$样本的一小部分,允许访问$\tau_{\mathrm{mem}}$,即纪念时间。 经过以前的数值研究[63, 18],生成的$\bm{\mathrm{x}}_{\tau}$样本如果

$$\mathbb{E}_{\bm{\mathrm{x}}_{\tau}}\left[\frac{\lVert\bm{\mathrm{x}}_{\tau}-\bm{\mathrm{a}}^{\mu_{1}}\rVert_{2}}{\lVert\bm{\mathrm{x}}_{\tau}-\bm{\mathrm{a}}^{\mu_{2}}\rVert_{2}}\right]<k, \tag{6}$$

where $\bm{\mathrm{a}}^{\mu_{1}}$ and $\bm{\mathrm{a}}^{\mu_{2}}$ are the nearest and second nearest neighbors of $\bm{\mathrm{x}}_{\tau}$ in the training set in the $L_{2}$ sense. In what follows, we choose to work with $k=1/3$ [63, 18], but we checked that varying $k$ to $1/2$ or $1/4$ does not impact the claims about the scaling. Error bars in the figures correspond to twice the standard deviation over 5 different test sets for FIDs, and 5 noise realizations for $\mathcal{L}_{\mathrm{train}}$ and $\mathcal{L}_{\mathrm{test}}$. For $f_{\mathrm{mem}}$, we report the 95% CIs on the mean evaluated with 1,000 bootstrap samples.

其中$\bm{\mathrm{a}}^{\mu_{1}}$和$\bm{\mathrm{a}}^{\mu_{2}}$是$\bm{\mathrm{x}}_{\tau}$在$L_{2}$意义上的训练集中距离最近和次近的邻居. 在下文中,我们选择与$k=1/3$ [63, 18] 合作,但我们检查了不同的$k$至$1/2$或$1/4$不会影响关于缩放的索赔. 数字中的错误栏对应于5个不同测试集上的标准偏差的两倍,对$\mathcal{L}_{\mathrm{train}}$和$\mathcal{L}_{\mathrm{test}}$的5个噪声实现. 对于 $f_{\mathrm{mem}}$,我们报告95%的CI 在平均评估 1,000靴子样本。

Role of training set size on the learning dynamics.

At fixed model capacity ($p=4\times 10^{6}$, base width $W=32$), we investigate how the training set size $n$ impacts the previous metrics. In the left panel of Fig. 2, we first report the FID (solid lines) and $f_{\mathrm{mem}}(\tau)$ (dashed lines) for various $n$. All trainings dynamics exhibit two phases. First, the FID quickly decreases to reach a minimum value on a timescale $\tau_{\mathrm{gen}}$ ($\approx 100$K) that does not depend on $n$. In the right panel, the generated samples at $\tau=100$K clearly differ from their nearest neighbors in the training set, indicating that the model generalizes correctly. Beyond this time, the FID remains flat. $f_{\mathrm{mem}}(\tau)$ is zero until a later time $\tau_{\mathrm{mem}}$ after which it increases, clearly signaling the entrance into a memorization regime, as illustrated by the generated samples in the right-most panel of Fig. 2, very close to their nearest neighbors. Both the transition time $\tau_{\mathrm{mem}}$ and the value of the final fraction $f_{\mathrm{mem}}(\tau_{\mathrm{max}})$ (with $\tau_{\mathrm{max}}$ being one to four million SGD steps) vary with $n$. The inset plot shows the normalized memorization fraction $f_{\mathrm{mem}}(\tau)/f_{\mathrm{mem}}(\tau_{\mathrm{max}})$ against the rescaled time $\tau/n$, making all curves collapse and increase at around $\tau/n\approx 300$, showing that $\tau_{\mathrm{mem}}\propto n$, and demonstrating the existence of a generalization window for $\tau\in\left[\tau_{\mathrm{gen}},\tau_{\mathrm{mem}}\right]$ that widens linearly with $n$, as illustrated in the left panel of Fig. 1.

训练集大小对学习动力学的影响。

在固定模型容量（$p=4\times 10^{6}$，基宽度 $W=32$）下，我们考察训练集大小 $n$ 如何影响上述指标。图 2 左图给出不同 $n$ 下的 FID（实线）与 $f_{\mathrm{mem}}(\tau)$（虚线）。所有训练动力学都呈现两阶段：首先 FID 迅速下降，在与 $n$ 无关的时间尺度 $\tau_{\mathrm{gen}}$（约 $100$K）上达到最小值。右图显示，$\tau=100$K 时生成样本与训练集最近邻明显不同，说明模型正确泛化。此后 FID 保持平台；$f_{\mathrm{mem}}(\tau)$ 在更晚的 $\tau_{\mathrm{mem}}$ 之前为零，之后上升，明确进入记忆机制。

As highlighted in the introduction, memorization in DMs is ultimately driven by the overfitting of the empirical score $\bm{\mathrm{s}}_{\mathrm{mem}}(\bm{\mathrm{x}},t)$. The evolution of $\mathcal{L}_{\mathrm{train}}(\tau)$ and $\mathcal{L}_{\mathrm{test}}(\tau)$ at fixed $t=0.01$ are shown in the middle panel of Fig. 2 for $n$ ranging from 512 to 32768. Initially, the two losses remain nearly indistinguishable, indicating that the learned score $\bm{\mathrm{s}}_{\bm{\theta}}(\bm{\mathrm{x}},t)$ does not depend on the training set. Beyond a critical time, $\mathcal{L}_{\mathrm{train}}$ continues to decrease while $\mathcal{L}_{\mathrm{test}}$ increases, leading to a nonzero generalization loss whose magnitude depends on $n$. As $n$ increases, this critical time also increases and, eventually, the training and test loss gap shrinks: for $n=32768$, the test loss remains close to the training loss, even after 11 million SGD steps. The inset shows the evolution of both losses with $\tau/n$, demonstrating that the overfitting time scales linearly with the training set size $n$, just like $\tau_{\mathrm{mem}}$ identified in the left panel. Moreover, there is a consistent lag between the overfitting time and $\tau_{\mathrm{mem}}$ at fixed $n$, reflecting the additional training required for the model to overfit the empirical score sufficiently to reproduce the training samples, and therefore to impact the memorization fraction.

正如导言中所强调的那样,模式草案中的纪念最终是由经验得分$\bm{\mathrm{s}}_{\mathrm{mem}}(\bm{\mathrm{x}},t)$的过度匹配所推动的。 $\mathcal{L}_{\mathrm{train}}(\tau)$和$\mathcal{L}_{\mathrm{test}}(\tau)$在固定的$t=0.01$的演化过程在图2的中间面板中显示,$n$的演化范围从512到32768. 最初,这两种损失仍然几乎无法区分,表明所学得分$\bm{\mathrm{s}}_{\bm{\theta}}(\bm{\mathrm{x}},t)$并不取决于训练集. 在关键时刻之后,$\mathcal{L}_{\mathrm{train}}$继续减少,而$\mathcal{L}_{\mathrm{test}}$则增加,导致非零一般化损失,其程度取决于$n$。 随着$n$的增加,这一关键时间也增加了,最终培训和测试损失差距缩小:对于$n=32768$来说,即使经过1100万SGD步骤,测试损失仍然接近培训损失. 套件显示这两种损失与$\tau/n$的演化,表明时间尺度与训练设置大小的$n$线性比过大,与左面面板确定的$\tau_{\mathrm{mem}}$一样. 此外,在固定的$n$中,过度适应的时间和$\tau_{\mathrm{mem}}$之间一直存在滞后,这反映了模型需要额外培训,以过度适应经验得分,足以复制培训样本,从而影响记忆分数。

Memorization is not due to data repetition.

We must stress that this delayed memorization with $n$ is not due to the mere repetition of training samples, as a first intuition could suggest. In SM Sects. A and B, we show that full-batch updates still yield $\tau_{\mathrm{mem}}\propto n$. In other words, even if at fixed $\tau$ all models have processed each sample equally often, larger $n$ consistently postpone memorization. This confirms that memorization in DMs is driven by a fundamental $n$-dependent change in the loss landscape – not by a sample repetition during training.

记忆并非来自数据重复。

必须强调：随 $n$ 推迟的记忆不是仅因训练样本被看到的次数更少——尽管直觉上容易这么想。附录 A、B 表明，即便采用全 batch 更新，$\tau_{\mathrm{mem}}\propto n$ 仍然成立。换言之，在固定 $\tau$ 下各模型已同等次数地处理每个样本时，更大的 $n$ 仍一致地推迟记忆。这说明扩散模型中的记忆由损失地貌随 $n$ 的根本变化驱动，而非训练中的样本重复。

Figure 3: Effect of the number of parameters in the U-Net architecture on the timescales of the training dynamics. (Left) FID (panels A, B) and normalized memorization fraction $f_{\mathrm{mem}}(\tau)/f_{\mathrm{mem}}(\tau_{\mathrm{max}})$ (panels C, D) for various $n$ and $W$ during training. In panels B and D, time is rescaled such that all curves collapse. (Right) $(n,p)$ phase diagram of generalization vs memorization for U-Nets trained on CelebA. Curves show, for $\tau\in\{\tau_{\mathrm{gen}},3\tau_{\mathrm{gen}},8\tau_{\mathrm{gen}}\}$, the minimal dataset size $n(p)$ satisfying $f_{\mathrm{mem}}(\tau)=0$. The shaded background indicates the memorization–generalization boundary for $\tau=\tau_{\mathrm{gen}}$.

图3:** U-Net架构参数数对培训动态时限的影响。 *(左) * FID(A、B小组)和在培训期间用于各种$n$和$W$的普通纪念分数$f_{\mathrm{mem}}(\tau)/f_{\mathrm{mem}}(\tau_{\mathrm{max}})$(C、D小组)。 在B面板和D面板中,时间被调整到所有曲线都崩溃。 *(右) *$(n,p)$相位图 泛化图与关于CelebA培训的U-Net的纪念图。 曲线显示,对于$\tau\in\{\tau_{\mathrm{gen}},3\tau_{\mathrm{gen}},8\tau_{\mathrm{gen}}\}$,最小数据集大小$n(p)$满足$f_{\mathrm{mem}}(\tau)=0$. 阴影背景表示$\tau=\tau_{\mathrm{gen}}$的纪念-通用边界。 **

Effect of the model capacity.

To study more precisely the role of the model capacity on the memorization–generalization transition, we vary the number of parameters $p$ by changing the U-Nets base width $W\in\{8,16,32,48,64\}$, resulting in a total of $p\in\{0.26,1,4,9,16\}\times 10^{6}$ parameters. In the left panel of Fig. 3, we plot both the FID (top row) and the normalized memorization fraction (bottom row) as functions of $\tau$ for several width $W$ and training set sizes $n$. Panels A and C demonstrate that higher-capacity networks (larger $W$) achieve high-quality generation and begin to memorize earlier than smaller ones. Panels B and D show that the two characteristic timescales simply scale as $\tau_{\mathrm{gen}}\propto W^{-1}$ and $\tau_{\mathrm{mem}}\propto nW^{-1}$. In particular, this implies that, for $W>8$, the critical training set size $n_{\mathrm{gm}}(p)$ at which $\tau_{\mathrm{mem}}=\tau_{\mathrm{gen}}$ is approximately independent of $p$ (at least on the limited values of $p$ we focused on). When $n>n_{\mathrm{gm}}(p)$, the interval $\left[\tau_{\mathrm{gen}},\tau_{\mathrm{mem}}\right]$ opens up, so that early stopping within this window yields high quality samples without memorization. In the right panel of Fig. 3, we display this boundary (solid line) in the $(n,p)$ plane by fixing the training time to $\tau=\tau_{\mathrm{gen}}$, that we identify numerically using the collapse of all FIDs at around $W\tau_{\mathrm{gen}}\approx 3\times 10^{6}$ (see panel B), and computing the smallest $n$ such that $f_{\mathrm{mem}}(\tau)=0$. The resulting solid curve delineates two regimes: below the curve, memorization already starts at $\tau_{\mathrm{gen}}$; above the curve, the models generalize perfectly under early stopping. We repeat this experiment for $\tau=3\tau_{\mathrm{gen}}$ and $\tau=8\tau_{\mathrm{gen}}$, showing saturation to larger and larger $p$ as $\tau$ increases. Eventually, for $\tau\to\infty$, we expect these successive boundaries to converge to the architectural regularization threshold $n^{\star}(p)$, i.e. the point beyond which the network avoids memorization because it is not expressive enough, as found in [15] and highlighted in the right panel of Fig. 1. In order to estimate $n^{\star}(p)$, we measure for a given $\tau$ the largest $n(\tau)$ yielding $f_{\mathrm{mem}}\approx 0$. The curve $n(\tau)$ approaches $n^{\star}(p)$ for large $\tau$. We therefore estimate $n^{\star}(p)$ by measuring the asymptotic values of $n(\tau)$, which in practice is reached already at $\tau=\tau_{\mathrm{max}}=2$M updates for the values of $W$ we focus on.

模型容量的影响。

为更精确研究容量对记忆–泛化转变的作用，我们通过改变 U-Net 基宽度 $W\in\{8,16,32,48,64\}$ 来改变参数量 $p$，得到 $p\in\{0.26,1,4,9,16\}\times 10^{6}$。图 3 左图给出若干 $W$ 与 $n$ 下 FID（上行）与归一化记忆比例（下行）随 $\tau$ 的变化。面板 A、C 表明更高容量网络（更大 $W$）更早达到高质量生成，也更早开始记忆。面板 B、D 显示两个特征时间尺度简单标度为 $\tau_{\mathrm{gen}}\propto W^{-1}$、$\tau_{\mathrm{mem}}\propto n W^{-1}$。

3 Training dynamics of a Random Features Network

3 随机特征网络的训练动力学

Notations.

We use bold symbols for vectors and matrices. The $L^{2}$ norm of a vector $\bm{\mathrm{x}}$ is denoted by $\lVert\bm{\mathrm{x}}\rVert=(\sum_{i}\bm{\mathrm{x}}_{i}^{2})^{1/2}$. We write $f=\mathcal{O}(g)$ to mean that in the limit $n,p\to\infty$, there exists a constant $C$ such that $\lvert f\rvert\leq C\lvert g\rvert$.

注释。

我们在向量和矩阵中使用粗体符号. 矢量$\bm{\mathrm{x}}$的$L^{2}$规范由$\lVert\bm{\mathrm{x}}\rVert=(\sum_{i}\bm{\mathrm{x}}_{i}^{2})^{1/2}$表示. 我们将$f=\mathcal{O}(g)$写为:在极限$n,p\to\infty$中,存在一个恒定的$C$,即$\lvert f\rvert\leq C\lvert g\rvert$.

Setting.

We study analytically a model introduced in [15], where the data lie in $d$ dimensions. We parametrize the score with a Random Features Neural Network [43]

准备

我们从分析角度研究了[15]中引入的模型,其中的数据是$d$维度。 我们用随机功能神经网络来压平得分 [43]

$$\displaystyle\bm{\mathrm{s}}_{\bm{\mathrm{A}}}(\bm{\mathrm{x}})=\frac{\bm{\mathrm{A}}}{\sqrt{p}}\sigma\left(\frac{\bm{\mathrm{W}}\bm{\mathrm{x}}}{\sqrt{d}}\right). \tag{7}$$

An RFNN, illustrated in Fig. 4 (left), is a two-layer neural-network whose first layer weights ($\bm{\mathrm{W}}\in\mathbb{R}^{p\times d}$) are drawn from a Gaussian distribution and remain frozen while the second layer weights ($\bm{\mathrm{A}}\in\mathbb{R}^{d\times p}$) are learned during training. This model has already served as theoretical framework for studying several behaviors of deep neural network such as the double descent phenomenon [35, 13]. $\sigma$ is an element-wise non-linear activation function. We consider a training set of $n$ iid samples $\bm{\mathrm{x}}^{\nu}\sim P_{\bm{\mathrm{x}}}$ for $\nu=1,\ldots,n$ and we focus on the high-dimensional limit $d,p,n\rightarrow\infty$ with the ratios $\psi_{p}=p/d,\psi_{n}=n/d$ kept fixed. We study the training dynamics associated to the minimization of the empirical score matching loss defined in (4) at a fixed diffusion time $t$. This is a simplification compared to practical methods, which use a single model for all $t$. It has been already studied in previous theoretical works [11, 15]. The loss (4) is rescaled by a factor $1/d$ in order to ensure a finite limit at large $d$. We also study the evolution of the test loss evaluated on test points and the distance to the exact score $\bm{\mathrm{s}}(\bm{\mathrm{x}})=\nabla\log P_{\bm{\mathrm{x}}}$,

图4(左)所示的RFNN是一种双层神经网络,其第一层重量($\bm{\mathrm{W}}\in\mathbb{R}^{p\times d}$)来自高斯分布,在训练期间学习到第二层重量($\bm{\mathrm{A}}\in\mathbb{R}^{d\times p}$)时保持冷冻. 这个模型已经作为研究深神经网络的几种行为,如双向下降现象的理论框架[35,13]. $\sigma$是一种元素明智的非线性激活功能. 我们考虑为$\nu=1,\ldots,n$提供一套由$n$ iid样本组成的训练集,我们侧重于高维限的$d,p,n\rightarrow\infty$,保持$\psi_{p}=p/d,\psi_{n}=n/d$的比例不变。 我们研究了与在$t$的固定传播时间尽量减少(4)中定义的实证分数匹配损失有关的培训动态。 与实用方法相比,这是一种简化,对所有$t$使用单一模式。 在之前的理论著作中已经研究过[11,15]. 损失(4)按$1/d$因数调整,以确保在$d$大范围内有限度。 我们还研究测试点评估的测试损失的演变情况,以及准确得分$\bm{\mathrm{s}}(\bm{\mathrm{x}})=\nabla\log P_{\bm{\mathrm{x}}}$的距离。

$$\displaystyle\mathcal{L}_{\mathrm{test}}=\frac{1}{d}\mathbb{E}_{\bm{\mathrm{x}},\bm{\xi}}\left[\lVert\sqrt{\Delta_{t}}\bm{\mathrm{s}}_{\bm{\mathrm{A}}}(\bm{\mathrm{x}}_{t}(\bm{\xi}))+\bm{\xi}\rVert^{2}\right],\quad\mathcal{E}_{\mathrm{score}}=\frac{1}{d}\mathbb{E}_{\bm{\mathrm{x}}}\left[\lVert\bm{\mathrm{s}}_{\bm{\mathrm{A}}}(\bm{\mathrm{x}})-\nabla\log P_{\bm{\mathrm{x}}}\rVert^{2}\right], \tag{8}$$

where the expectations $\mathbb{E}_{\bm{\mathrm{x}},\bm{\xi}}$ are computed over $\bm{\mathrm{x}}\sim P_{\bm{\mathrm{x}}}$ and $\bm{\xi}\sim\mathcal{N}(0,\bm{I}_{d})$. The generalization loss, defined as $\mathcal{L}_{\mathrm{gen}}=\mathcal{L}_{\mathrm{test}}-\mathcal{L}_{\mathrm{train}}$, indicates the degree of overfitting in the model while the distance to the exact score $\mathcal{E}_{\mathrm{score}}$ measures the quality of the generation as it is an upper bound on the Kullback–Leibler divergence between the target and generated distributions [53, 8]. The weights $\bm{\mathrm{A}}$ are updated via gradient descent

第$\mathbb{E}_{\bm{\mathrm{x}},\bm{\xi}}$和$\bm{\xi}\sim\mathcal{N}(0,\bm{I}_{d})$之间计算的期望值。 泛化损失被定义为$\mathcal{L}_{\mathrm{gen}}=\mathcal{L}_{\mathrm{test}}-\mathcal{L}_{\mathrm{train}}$,表示模型中过度适应的程度,而与精确分数$\mathcal{E}_{\mathrm{score}}$的距离则测量生成质量,因为它是Kullback - Leibleer在目标与生成分布之间的上方界限[53,8].$\bm{\mathrm{A}}$通过梯度下降更新加权数

$$\displaystyle\bm{\mathrm{A}}^{(k+1)}=\bm{\mathrm{A}}^{(k)}-\eta\nabla_{\bm{\mathrm{A}}}\mathcal{L}_{\mathrm{train}}(\bm{\mathrm{A}}^{(k)}), \tag{9}$$

where $\eta$ is the learning rate. In the high-dimensional limit, as the learning rate $\eta\to 0$, and after rescaling time as $\tau=k\eta/d^{2}$, the discrete-time dynamics converges to the following continuous-time gradient flow:

其中$\eta$为学习率. 在高维限中,作为学习速率$\eta\to 0$,在将时间调整为$\tau=k\eta/d^{2}$后,离散时间动态会趋同到以下连续时间梯度流: 1.

$$\displaystyle\dot{\bm{\mathrm{A}}}(\tau)=-d^{2}\nabla_{\bm{\mathrm{A}}}\mathcal{L}_{\mathrm{train}}(\bm{\mathrm{A}}(\tau))=-2\Delta_{t}\frac{d}{p}\bm{\mathrm{A}}\bm{\mathrm{U}}-\frac{2d\sqrt{\Delta_{t}}}{\sqrt{p}}\bm{\mathrm{V}}^{T}, \tag{10}$$

with

与

$$\displaystyle\bm{\mathrm{U}}=\frac{1}{n}\sum_{\nu=1}^{n}\mathbb{E}_{\bm{\xi}}\left[\sigma\left(\frac{\bm{\mathrm{W}}\bm{\mathrm{x}}_{t}^{\nu}(\bm{\xi})}{\sqrt{d}}\right)\sigma\left(\frac{\bm{\mathrm{W}}\bm{\mathrm{x}}_{t}^{\nu}(\bm{\xi})}{\sqrt{d}}\right)^{T}\right],\quad\bm{\mathrm{V}}=\frac{1}{n}\sum_{\nu=1}^{n}\mathbb{E}_{\bm{\xi}}\left[\sigma\left(\frac{\bm{\mathrm{W}}\bm{\mathrm{x}}_{t}^{\nu}(\bm{\xi})}{\sqrt{d}}\right)\bm{\xi}^{T}\right]. \tag{11}$$

Figure 4: (Left) Illustration of an RFNN. (Middle/Right) Spectrum of $\bm{\mathrm{U}}$. Density $\rho(\lambda)$ from Theorem 3.1 in the overparameterized Regime I described in Theorem 3.2, with $\psi_{p}=64$, $\psi_{n}=8$, $t=0.01$, and $\rho_{\bm{\Sigma}}(\lambda)=\delta(\lambda-1)$. The bulk of the spectrum (orange) is between $\lambda\approx 10$ and $\lambda\approx 45$. The histogram shows the eigenvalues from a single realization of $\bm{\mathrm{U}}$ at $d=100$. Inset: zoom near $\lambda=0$ (in blue) showing the first bulk $\rho_{1}$ and the delta peak at $\lambda=s_{t}^{2}$. (Right) Same as (Middle), but with $\rho_{\bm{\Sigma}}(\lambda)=\frac{1}{2}\delta(\lambda-0.5)+\frac{1}{2}\delta(\lambda-1.5)$. The first bulk in blue remains unchanged, as it depends only on $\sigma_{\bm{\mathrm{x}}}^{2}=\Tr(\bm{\Sigma})/d=1$ in both cases, while the second bulk varies with $\bm{\Sigma}$.

图4:(左)** RFNN说明**(中/右) ** 第$\bm{\mathrm{U}}$号决议。 ** 密度 $\rho(\lambda)$来自定理3.2所述超参数化制度I中的定理3.1,其中$\psi_{p}=64$,$\psi_{n}=8$,$t=0.01$,和$\rho_{\bm{\Sigma}}(\lambda)=\delta(\lambda-1)$. 大部分频谱(橙色)介于$\lambda\approx 10$和$\lambda\approx 45$之间. 直方图显示了$\bm{\mathrm{U}}$在$d=100$的单一实现中得出的eigen值. inset:在$\lambda=0$附近放大(蓝色)显示第一批$\rho_{1}$和位于$\lambda=s_{t}^{2}$的三角洲峰. (*Right *) 与(中)相同,但与$\rho_{\bm{\Sigma}}(\lambda)=\frac{1}{2}\delta(\lambda-0.5)+\frac{1}{2}\delta(\lambda-1.5)$相同。 以蓝色为单位的第一批没有变化,因为这两种情况只取决于$\sigma_{\bm{\mathrm{x}}}^{2}=\Tr(\bm{\Sigma})/d=1$,而第二批则与$\bm{\Sigma}$不同。

Assumptions.

For our analytical results to hold, we make the following mathematical assumptions which are standard when studying Random Features [41, 17, 22] namely (i) the activation function $\sigma$ admits a Hermite polynomial expansion $\sigma(x)=\sum_{s=0}^{\infty}\frac{\alpha_{s}}{s!}He_{s}(x)$; and (ii) the data distribution $P_{\bm{\mathrm{x}}}$ has sub-Gaussian tails and a covariance $\bm{\Sigma}=\mathbb{E}_{P_{\bm{\mathrm{x}}}}[\bm{\mathrm{x}}\bm{\mathrm{x}}^{T}]$ with bounded spectrum. We assume that the empirical distribution of eigenvalues of $\bm{\Sigma}$ converges weakly in the high dimensional limit to a deterministic density $\rho_{\bm{\Sigma}}(\lambda)$ and that $\Tr(\bm{\Sigma})/d$ converges to a finite limit (for a more precise mathematical statement see SM Sect. C.3). Moreover, we make additional assumptions that are not essential to the proofs but which simplify the analysis: (iii) the activation function $\sigma$ verifies $\mu_{0}=\mathbb{E}_{z}[\sigma(z)]=0$; and (iv) the second layer $\bm{\mathrm{A}}$ is initialized with zero weights $\bm{\mathrm{A}}(\tau=0)=0$. In numerical applications, unless specified, we use $\sigma(z)=\tanh(z)$ and $P_{\bm{\mathrm{x}}}=\mathcal{N}(0,\bm{I}_{d})$.

• 假设。

为了维持分析结果,我们提出下列数学假设,这些假设在研究随机特征[41、17、22]时是标准的,即(一) 激活函数$\sigma$ 承认一种赫米特聚体膨胀$\sigma(x)=\sum_{s=0}^{\infty}\frac{\alpha_{s}}{s!}He_{s}(x)$ ;和(二) 数据分布$P_{\bm{\mathrm{x}}}$ 具有亚古西尾部和具有定频谱的共变$\bm{\Sigma}=\mathbb{E}_{P_{\bm{\mathrm{x}}}}[\bm{\mathrm{x}}\bm{\mathrm{x}}^{T}]$。 我们假设,$\bm{\Sigma}$的基值经验分布在高维度限度中弱化为确定密度$\rho_{\bm{\Sigma}}(\lambda)$,$\Tr(\bm{\Sigma})/d$趋同为有限限度(更精确的数学说明见SM Sect. C.3). 此外,我们还作出一些对证明并非必不可少的、但可简化分析的其他假设:(三) 激活函数$\sigma$验证$\mu_{0}=\mathbb{E}_{z}[\sigma(z)]=0$;和(四) 第二层$\bm{\mathrm{A}}$初始化为零重量$\bm{\mathrm{A}}(\tau=0)=0$。 在数字应用中,除非具体说明,我们使用$\sigma(z)=\tanh(z)$和$P_{\bm{\mathrm{x}}}=\mathcal{N}(0,\bm{I}_{d})$。

A

A级

Emergence of the two timescales during training.

We first show in Fig. 5 that the behavior of training and test losses in the RF model mirrors the one found in realistic cases in Sect. 2, with a separation of timescales $\tau_{\mathrm{gen}}$ and $\tau_{\mathrm{mem}}$ which increases with $n$. Equation (10) is linear in $\bm{\mathrm{A}}$ and hence it can be solved exactly (see SM). The timescales of the training dynamics are given by the inverse eigenvalues of the $p\times p$ matrix $\Delta_{t}\bm{\mathrm{U}}/\psi_{p}$. Building on the Gaussian Equivalence Principle [16, 17, 36] and the theory of linear pencils [7], [15] (2025) derive a coupled system of equations characterizing the Stieltjes transform of the eigenvalue density $\rho(\lambda)$ of $\bm{\mathrm{U}}$ for isotropic Gaussian data that lie in a $D$-dimensional subspace with $D\leq d$ and $D=\mathcal{O}(d)$. We offer an alternative derivation presented in SM for general variance using the replica method [37] – a heuristic method from the statistical physics of disordered systems – yielding the more compact formulation for obtaining the spectrum stated in Theorem 3.1. Before stating the theorem, we introduce

训练期间两个时标的出现.

我们首先在图5中显示,RF模型中的培训和测试损失行为与第2节中现实案例中发现的行为相仿,时间尺度为$\tau_{\mathrm{gen}}$和$\tau_{\mathrm{mem}}$,它们随着$n$的增加而增加。 方程(10)在$\bm{\mathrm{A}}$中是线性的,因此可以精确解决(见SM)。 训练动力学的时间尺度由$p\times p$矩阵$\Delta_{t}\bm{\mathrm{U}}/\psi_{p}$的反正值给出. 基于高斯等值原理[16,17,36]和线性铅笔理论[7],15,衍生出一个结合的方程系统,其特征是$\bm{\mathrm{U}}$的基值密度$\rho(\lambda)$的Stieltjes变换,用于位于$D$的异地高斯数据 - 带有$D\leq d$和$D=\mathcal{O}(d)$的维次空间. 我们用复制法[37],即无序系统统计物理学中的一种热度法,在SM中为一般差异提出一种备选推论,从而得出更紧凑的公式,以获得定理3.1所述频谱。 在陈述定理之前,我们先介绍一下

$$\displaystyle b_{t}=\mathbb{E}_{u,v}[v\sigma(e^{-t}\sigma_{\bm{\mathrm{x}}}u+\sqrt{\Delta_{t}}v)],\quad a_{t}=\mathbb{E}_{u,v}[\sigma(e^{-t}\sigma_{\bm{\mathrm{x}}}u+\sqrt{\Delta_{t}}v)\frac{u}{e^{-t}\sigma_{\bm{\mathrm{x}}}}], \\ \displaystyle v_{t}^{2}=\mathbb{E}_{u,v,w}[\sigma(e^{-t}\sigma_{\bm{\mathrm{x}}}u+\sqrt{\Delta_{t}}v)\sigma(e^{-t}\sigma_{\bm{\mathrm{x}}}u+\sqrt{\Delta_{t}}w)]-a_{t}^{2}e^{-2t}\sigma_{\bm{\mathrm{x}}}^{2}, \\ \displaystyle s_{t}^{2}=\mathbb{E}_{u}[\sigma(\Gamma_{t}u)^{2}]-a_{t}^{2}e^{-2t}\sigma_{\bm{\mathrm{x}}}^{2}-v_{t}^{2}-b_{t}^{2}, \tag{12}$$

where $\sigma_{\bm{\mathrm{x}}}^{2}=\frac{\Tr(\bm{\Sigma})}{d}$, $\Gamma_{t}=e^{-2t}\sigma_{\bm{\mathrm{x}}}^{2}+\Delta_{t}=1+e^{-2t}(\sigma_{\bm{\mathrm{x}}}^{2}-1)$ and the expectation is over the $u,v,w$ random variables which are independent standard Gaussian $\mathcal{N}(0,1)$.

其中$\sigma_{\bm{\mathrm{x}}}^{2}=\frac{\Tr(\bm{\Sigma})}{d}$,$\Gamma_{t}=e^{-2t}\sigma_{\bm{\mathrm{x}}}^{2}+\Delta_{t}=1+e^{-2t}(\sigma_{\bm{\mathrm{x}}}^{2}-1)$和期望值大于$u,v,w$随机变量,这些变量是独立的标准高斯$\mathcal{N}(0,1)$.

Theorem 3.1.

Let $q(z)=\frac{1}{p}\Tr(\bm{\mathrm{U}}-z\bm{I}_{p})^{-1}$, $r(z)=\frac{1}{p}\Tr(\bm{\Sigma}^{1/2}\bm{\mathrm{W}}^{T}(\bm{\mathrm{U}}-z\bm{I}_{p})^{-1}\bm{\mathrm{W}}\bm{\Sigma}^{1/2})$ and $s(z)=\frac{1}{p}\Tr(\bm{\mathrm{W}}^{T}(\bm{\mathrm{U}}-z\bm{I}_{p})^{-1}\bm{\mathrm{W}})$, with $z\in\mathbb{C}$. Let

** 定 义 3.1****。 **

• 让$q(z)=\frac{1}{p}\Tr(\bm{\mathrm{U}}-z\bm{I}_{p})^{-1}$、$r(z)=\frac{1}{p}\Tr(\bm{\Sigma}^{1/2}\bm{\mathrm{W}}^{T}(\bm{\mathrm{U}}-z\bm{I}_{p})^{-1}\bm{\mathrm{W}}\bm{\Sigma}^{1/2})$和$s(z)=\frac{1}{p}\Tr(\bm{\mathrm{W}}^{T}(\bm{\mathrm{U}}-z\bm{I}_{p})^{-1}\bm{\mathrm{W}})$与$z\in\mathbb{C}$。 让*

$$\displaystyle\hat{s}(q)=b_{t}^{2}\psi_{p}+\frac{1}{q}, \\ \displaystyle\hat{r}(r,q)=\frac{\psi_{p}a_{t}^{2}e^{-2t}}{1+\frac{a_{t}^{2}e^{-2t}\psi_{p}}{\psi_{n}}r+\frac{\psi_{p}v_{t}^{2}}{\psi_{n}}q}. \tag{15}$$

Then $q(z),r(z)$ and $s(z)$ satisfy the following set of three equations:

• 然后$q(z),r(z)$和$s(z)$满足以下一组三个方程:*

$$\displaystyle s=\int\mathrm{d}\rho_{\bm{\Sigma}}(\lambda)\frac{1}{\hat{s}(q)+\lambda\hat{r}(r,q)}, \\ \displaystyle r=\int\mathrm{d}\rho_{\bm{\Sigma}}(\lambda)\frac{\lambda}{\hat{s}(q)+\lambda\hat{r}(r,q)}, \\ \displaystyle\psi_{p}(s_{t}^{2}-z)+\frac{\psi_{p}v_{t}^{2}}{1+\frac{a_{t}^{2}e^{-2t}\psi_{p}}{\psi_{n}}r+\frac{\psi_{p}v_{t}^{2}}{\psi_{n}}q}+\frac{1-\psi_{p}}{q}-\frac{s}{q^{2}}=0, \tag{17}$$

The eigenvalue distribution of $\bm{\mathrm{U}}$, $\rho(\lambda)$, can then be obtained using the Sokhotski–Plemelj inversion formula $\rho(\lambda)=\underset{\varepsilon\rightarrow 0^{+}}{\lim}\frac{1}{\pi}\operatorname{Im}q(\lambda+i\varepsilon)$.

• $\bm{\mathrm{U}}$,$\rho(\lambda)$的等值分布,然后可以使用Sokhotski-Plemelj反演公式$\rho(\lambda)=\underset{\varepsilon\rightarrow 0^{+}}{\lim}\frac{1}{\pi}\operatorname{Im}q(\lambda+i\varepsilon)$获得.

We now focus on the asymptotic regime $\psi_{p},\psi_{n}\gg 1$, typical for strongly over-parameterized models trained on large data sets. In this limit, the spectrum of $\bm{\mathrm{U}}$ can be described analytically by the following Theorem 3.2.

峦 我们现在着重注意 Glentimotic system $\psi_{p},\psi_{n}\gg 1$,这是在大型数据集方面训练的高度超参数化模型的典型。 在这个限制中,$\bm{\mathrm{U}}$的频谱可以通过以下定理3.2来分析描述.

Theorem 3.2(Informal).

Let $\rho$ denote the spectral density of $\bm{\mathrm{U}}$.

** 定理3.2**(非正式)**。 **

• 让$\rho$表示$\bm{\mathrm{U}}$的光谱密度。

Regime I (overparametrized): $\psi_{p}>\psi_{n}\gg 1$. $$\rho(\lambda)=\Bigl(1-\frac{1+\psi_{n}}{\psi_{p}}\Bigr)\delta(\lambda-{s_{t}^{2}})+\frac{\psi_{n}}{\psi_{p}}\,\rho_{1}(\lambda)+\frac{1}{\psi_{p}}\,\rho_{2}(\lambda).$$
Regime II (underparametrized): $\psi_{n}>\psi_{p}\gg 1$. $$\rho(\lambda)=\Bigl(1-\frac{1}{\psi_{p}}\Bigr)\rho_{1}(\lambda)+\frac{1}{\psi_{p}}\,\rho_{2}(\lambda).$$

• 制度一(超标):$\psi_{p}>\psi_{n}\gg 1$。
• II号制度(不足):$\psi_{n}>\psi_{p}\gg 1$。 *$$\rho(\lambda)=\Bigl(1-\frac{1}{\psi_{p}}\Bigr)\rho_{1}(\lambda)+\frac{1}{\psi_{p}}\,\rho_{2}(\lambda).$$

where $\rho_{1}$ is an atomless measure with support

$\rho_{1}$是一种有支持的无原子措施