Learning Dynamics of LLM Finetuning

大语言模型微调的学习动力学

Yi Ren Affiliation: University of British Columbia Email: renyi.joshua@gmail.com    Danica J. Sutherland Affiliation: University of British Columbia & Amii Email: dsuth@cs.ubc.ca

Abstract

摘要

Learning dynamics, which describes how the learning of specific training examples influences the model’s predictions on other examples, gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique “squeezing effect” to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM’s finetuning but also inspires a simple, effective method to improve alignment performance. Code for experiments is available at https://github.com/Joshua-Ren/Learning_dynamics_LLM.

学习动力学刻画特定训练样本的学习如何影响模型对其它样本的预测，是理解深度学习系统行为的有力工具。我们分析大语言模型在各类微调中的学习动力学，逐步分解不同潜在回复之间影响力的累积。该框架可对指令微调与偏好微调中许多常见算法的训练现象给出统一解释。尤其是，我们提出一种假说，解释为何某些幻觉在微调后反而加重：例如模型会用问题 B 的回复中的措辞或事实去回答问题 A，或在生成时反复使用相似的简单短语。我们还扩展该框架，强调一种独特的「挤压效应」，用以解释 off-policy 直接偏好优化（DPO）中先前观察到的现象——DPO 跑得过久，连期望输出的概率也会下降。该框架亦有助于理解 on-policy DPO 及其变体收益来自何处。分析不仅为 LLM 微调提供新视角，还启发了一种简单有效、可提升对齐表现的方法。实验代码见 https://github.com/Joshua-Ren/Learning_dynamics_LLM。

1 Introduction

1 引言

Deep neural networks usually acquire new knowledge by updating their parameters via gradient descent (GD). This procedure can be described by learning dynamics, which links changes in the model’s predictions to the gradients generated by learning specific examples. With the help of learning dynamics, researchers have not only explained many interesting phenomena during training, e.g., the “zig-zag” learning path (Ren et al. 2022a) and the formation of compositional concept space (Park et al. 2024a), but used these insights to propose novel, improved algorithms in different problems (e.g. Ren et al. 2023b; Pruthi et al. 2020a; Xia et al. 2024a).

深度神经网络通常通过梯度下降（GD）更新参数来获取新知识。该过程可用学习动力学描述：它把模型预测的变化与学习特定样本所产生的梯度联系起来。借助学习动力学，研究者不仅解释了训练中的诸多现象，例如「之」字形学习路径（Ren et al. 2022a）以及组合概念空间的形成（Park et al. 2024a），还据此提出了面向不同问题的新算法（如 Ren et al. 2023b；Pruthi et al. 2020a；Xia et al. 2024a）。

The study of large language models (LLM) is gaining popularity due to their surprising capabilities on various tasks. To ensure the LLMs follow human instructions and align well with human preferences, finetuning has attracted much recent attention. Practitioners often start with instruction tuning, where the model learns extra knowledge necessary for the downstream task, and then preference tuning, where the model aligns its outputs to human preference (Ouyang et al. 2022a). Various finetuning algorithms have been proposed to fit into this pipeline, with differing explanations as to why they improve the model’s performance.

大语言模型（LLM）因在多种任务上的出色能力而受到广泛关注。为使其遵循人类指令并与人类偏好对齐，微调成为近期热点。实践中通常先做指令微调，让模型学习下游任务所需的额外知识，再做偏好微调，使输出与人类偏好对齐（Ouyang et al. 2022a）。围绕该流程已有多种微调算法，对其有效性的解释也不尽相同。

Contrary to most existing analyses of LLM finetuning, which use the perspective of their training targets, their status at the end of training, or their relationships to reinforcement learning (e.g. Ji et al. 2024a; Rafailov et al. 2024a; Tajwar et al. 2024a), this paper tries to understand LLMs’ evolution from a dynamical perspective. Specifically, we formalize the learning dynamics of LLM finetuning by decomposing the change of the model’s prediction into three terms which play different roles. This framework can be easily adapted to various finetuning algorithms with different goals, including supervised finetuning (SFT, Wei et al. 2022a), direct preference optimization (DPO, Rafailov et al. 2023a, and its variants) and even reinforcement learning based methods (e.g., PPO, Schulman et al. 2017a). This framework helps explain several interesting and counter-intuitive observations during training – including the “repeater” phenomenon after preference tuning (Holtzman et al. 2020a), hallucination (Huang et al. 2023a), the decay in confidence of all responses during off-policy DPO (Rafailov et al. 2024a), and more.

与多数从训练目标、训练终点状态或与强化学习关系出发分析 LLM 微调的工作不同（如 Ji et al. 2024a；Rafailov et al. 2024a；Tajwar et al. 2024a），本文试图从动力学视角理解 LLM 的演化。具体而言，我们将 LLM 微调的学习动力学形式化，把模型预测的变化分解为起不同作用的三项。该框架可轻松适配目标各异的多种微调算法，包括监督微调（SFT，Wei et al. 2022a）、直接偏好优化（DPO，Rafailov et al. 2023a 及其变体），乃至基于强化学习的方法（如 PPO，Schulman et al. 2017a）。它有助于解释训练中若干有趣且反直觉的现象——包括偏好微调后的「复读机」现象（Holtzman et al. 2020a）、幻觉（Huang et al. 2023a）、off-policy DPO 中所有回复置信度下降（Rafailov et al. 2024a）等。

Moreover, we also provide a new perspective on understanding why off-policy DPO and other variants underperform their on-policy counterparts (Guo et al. 2024c). Our explanation starts by observing an interesting “squeezing effect,” which we demonstrate is a consequence of gradient ascent (as in DPO and similar algorithms) on models with cross-entropy loss following a softmax layer. In short, for each token’s prediction, the negative gradient will push down the model’s predictions on (almost) all possible output labels, moving this probability mass to the most-likely labels. This can be detrimental to the alignment we are trying to achieve. This effect is most serious when the negative gradient is imposed on an already-unlikely label, which is why the confidence of almost all responses decreases during off-policy DPO. Inspired by this, we propose a simple, counter-intuitive, but very effective method to further improve alignment performance.

此外，我们还给出理解为何 off-policy DPO 及其变体逊于 on-policy 对照的新视角（Guo et al. 2024c）。解释始于一种有趣的「挤压效应」：我们证明，它是带 Softmax 层的交叉熵模型在做梯度上升（如 DPO 及同类算法）时的必然后果。简言之，对每个 token 的预测，负梯度会压低模型在（几乎）所有可能输出标签上的预测，并把概率质量推向当前最可能的标签。这对我们试图达成的对齐可能有害。当负梯度作用在本已不太可能的标签上时，该效应尤为严重，这也解释了为何 off-policy DPO 中几乎所有回复的置信度都会下降。受此启发，我们提出一种简单、看似反直觉却非常有效的方法，以进一步提升对齐表现。

2 Definition of Learning Dynamics and an MNIST Example

2 学习动力学的定义与一个 MNIST 例子

Learning dynamics is usually an umbrella term describing how the change of a specific factor influences the model’s prediction. In this paper, we narrow down it to describe “how the change in model’s parameter $\theta$ influences the corresponding change in $f_{\theta}$”, i.e., the relationship between $\Delta\theta$ and $\Delta f_{\theta}$. When the model updates its parameters using gradient descent (GD), we have

学习动力学通常是一个统称，描述某个因素的变化如何影响模型预测。本文将范围收窄为「参数 $\theta$ 的变化如何引起 $f_{\theta}$ 的变化」，即 $\Delta\theta$ 与 $\Delta f_{\theta}$ 的关系。当模型用梯度下降（GD）更新参数时，有

\quad\Delta\theta\triangleq\theta^{t+1}-\theta^{t}=-\eta\cdot\nabla\mathcal{L}\left(f_{\theta}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}}),{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}}\right);\quad\Delta f({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})\triangleq f_{\theta^{t+1}}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})-f_{\theta^{t}}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}}), \tag{1}

where the update of $\theta$ during step $t\rightarrow t+1$ is given by one gradient update on the sample pair ({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}}) with learning rate $\eta$. In short, the learning dynamics in this paper address the question:

其中 $\theta$ 在步 $t\rightarrow t+1$ 的更新由对样本对 ({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}})、学习率 $\eta$ 的一次梯度更新给出。简言之，本文的学习动力学要回答的问题是：

After an GD update on {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}}, how does the model’s prediction on {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} change?

在 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 上做一次 GD 更新后，模型对 {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 的预测如何变化？

Studying the learning dynamics can shed light on many important problems in deep learning and also help to understand various counter-intuitive phenomena. Appendix A further discusses related work.

研究学习动力学可以澄清深度学习中许多重要问题，也有助于理解各种反直觉现象。附录 A 进一步讨论相关工作。

As a warm-up, we first consider a standard supervised learning problem, where the model learns to map $\bm{\mathsf{x}}$ to predictions $\bm{\mathsf{y}}=\{y_{1},\dots,y_{L}\}\in\mathcal{V}^{L}$, where $\mathcal{V}$ is the vocabulary of size $V$. The model usually outputs a probability distribution by first generating a matrix of logits $\bm{\mathsf{z}}=h_{\theta}(\bm{\mathsf{x}})\in\mathbb{R}^{V\times L}$ and then takes the $\Softmax$ of each column. We can track the change in the model’s confidence by observing $\log\pi_{\theta}(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})$.

作为热身，先考虑标准监督学习：模型学习把 $\bm{\mathsf{x}}$ 映射到预测 $\bm{\mathsf{y}}=\{y_{1},\dots,y_{L}\}\in\mathcal{V}^{L}$，其中 $\mathcal{V}$ 为大小 $V$ 的词表。模型通常先生成 logits 矩阵 $\bm{\mathsf{z}}=h_{\theta}(\bm{\mathsf{x}})\in\mathbb{R}^{V\times L}$，再对每列做 Softmax 得到概率分布。我们可通过观察 $\log\pi_{\theta}(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})$ 来跟踪模型置信度的变化。

Per-step influence decomposition.

The learning dynamics of (1) become,

逐步影响力分解。

式 (1) 的学习动力学变为

\displaystyle\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})\triangleq\log\pi_{\theta^{t+1}}(\bm{\mathsf{y}}\mid{\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})-\log\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}}),. \tag{2}

For simplicity, we start from the $L=1$ scenario, where the $\Delta\theta$ and $\Delta\log\pi$ can be linked by the following result, a version of a result of Ren et al. 2022a proved and further discussed in Appendix B. For multi-label classification ($L>1$), the updates separate; we can calculate $L$ different $\Delta\log\pi^{t}$ and stack them together.

为简单起见，先从 $L=1$ 情形出发：$\Delta\theta$ 与 $\Delta\log\pi$ 可由下述结果联系起来，这是 Ren et al. 2022a 结果的一个版本，证明与进一步讨论见附录 B。对多标签分类（$L>1$），更新可分离；我们可计算 $L$ 个不同的 $\Delta\log\pi^{t}$ 再堆叠。

Proposition 1.

Let $\pi=\Softmax(\bm{\mathsf{z}})$ and $\bm{\mathsf{z}}=h_{\theta}(\bm{\mathsf{x}})$. The one-step learning dynamics decompose as

命题 1。

设 $\pi=\Softmax(\bm{\mathsf{z}})$ 且 $\bm{\mathsf{z}}=h_{\theta}(\bm{\mathsf{x}})$。一步学习动力学可分解为

\underbrace{\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid{\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})}_{V\times 1}=-\eta\underbrace{\mathcal{A}^{t}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})}_{V\times V}\underbrace{\mathcal{K}^{t}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}})}_{V\times V}\underbrace{\mathcal{G}^{t}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}})}_{V\times 1}{}+\bigO(\eta^{2}\|\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}})\|^{2}_{\mathrm{op}}), \tag{3}

where \mathcal{A}^{t}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})=\nabla_{\bm{\mathsf{z}}}\log\pi_{\theta^{t}}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})=I-\bm{\mathsf{1}}\pi_{\theta^{t}}^{\top}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}}), \mathcal{K}^{t}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}})=(\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}})(\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}})|_{\theta^{t}})^{\top} is the empirical neural tangent kernel of the logit network $\bm{\mathsf{z}}$, and \mathcal{G}^{t}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}})=\nabla_{\bm{\mathsf{z}}}\mathcal{L}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}})|_{\bm{\mathsf{z}}^{t}}.

其中 \mathcal{A}^{t}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})=\nabla_{\bm{\mathsf{z}}}\log\pi_{\theta^{t}}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})=I-\bm{\mathsf{1}}\pi_{\theta^{t}}^{\top}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})，\mathcal{K}^{t}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}},{\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}})=(\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})|_{\theta^{t}})(\nabla_{\theta}\bm{\mathsf{z}}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}})|_{\theta^{t}})^{\top} 是 logits 网络 $\bm{\mathsf{z}}$ 的经验神经正切核，\mathcal{G}^{t}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}})=\nabla_{\bm{\mathsf{z}}}\mathcal{L}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}})|_{\bm{\mathsf{z}}^{t}}。

\mathcal{A}^{t}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})=I-\bm{\mathsf{1}}\pi_{\theta^{t}}^{\top}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}}) only depends on the model’s current predicted probability. The matrix $\mathcal{K}^{t}$ is the empirical neural tangent kernel (eNTK, Jacot et al. 2018a) of the model, the product of the model’s gradients with respect to {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} and {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}}. The analysis in this paper relies on the following assumption:

\mathcal{A}^{t}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}})=I-\bm{\mathsf{1}}\pi_{\theta^{t}}^{\top}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}}) 只取决于模型当前预测概率。矩阵 $\mathcal{K}^{t}$ 是模型的经验神经正切核（eNTK，Jacot et al. 2018a），即模型分别对 {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 与 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 的梯度之积。本文分析依赖下述假设：

During the training, the relative influence of learning $x_{u}$ on all other different $x_{o}$ is relatively stable.

训练过程中，学习 $x_{u}$ 对其它各个 $x_{o}$ 的相对影响力保持相对稳定。

The common “lazy eNTK” assumption discussed in Arora et al. 2019a is a sufficient but not necessary condition for this paper. Appendix C provides a more detailed discussion and experimental verification for both MNIST and LLM settings. We can then think of $\mathcal{K}^{t}$ as a model-specific similarity measurement between different input samples: larger $\|\mathcal{K}^{t}\|_{F}$ means the update of $\bm{\mathsf{x}}_{u}$ likely influences the model’s prediction on $\bm{\mathsf{x}}_{o}$ more. Finally, $\mathcal{G}^{t}$ is determined by the loss function $\mathcal{L}$, which provides the energy and direction for the model’s adaptation. For example, for cross-entropy loss $\mathcal{L}_{\text{CE}}\triangleq-\bm{\mathsf{y}}_{u}^{\top}\log\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}}_{u})$, we have $\mathcal{G}_{\text{CE}}^{t}=\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{x}}_{u})-\bm{\mathsf{y}}_{u}$, a length-$V$ vector that points from the model’s current predictive distribution to the desired supervisory distribution. For typical “hard” labels, $\bm{\mathsf{y}}_{u}$ is a one-hot vector $\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}}$.

Arora et al. 2019a 讨论的常见「惰性 eNTK」假设是本文的充分非必要条件。附录 C 对 MNIST 与 LLM 设定给出更细讨论与实验验证。于是可将 $\mathcal{K}^{t}$ 视为模型特异的样本相似度度量：$\|\mathcal{K}^{t}\|_{F}$ 越大，更新 $\bm{\mathsf{x}}_{u}$ 越可能更强地影响模型对 $\bm{\mathsf{x}}_{o}$ 的预测。最后，$\mathcal{G}^{t}$ 由损失函数 $\mathcal{L}$ 决定，为模型适应提供能量与方向。例如对交叉熵损失 $\mathcal{L}_{\text{CE}}\triangleq-\bm{\mathsf{y}}_{u}^{\top}\log\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}}_{u})$，有 $\mathcal{G}_{\text{CE}}^{t}=\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{x}}_{u})-\bm{\mathsf{y}}_{u}$，这是从当前预测分布指向期望监督分布的长度为 $V$ 的向量。对典型「硬」标签，$\bm{\mathsf{y}}_{u}$ 是 one-hot 向量 $\bm{\mathsf{e}}_{\bm{\mathsf{y}}_{u}}$。

Accumulated influence and a demonstration on MNIST.

Proposition 1 describes how the update of {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} changes the model’s prediction on {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} for each learning step. Since a real model updates its parameters for many steps, it is important to ask about accumulation of these per-step influences over time. We start by analyzing a simple example of training a LeNet on the MNIST dataset (LeCun et al. 1998a).

累积影响力与 MNIST 上的演示。

命题 1 描述每一步学习中更新 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 如何改变模型对 {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 的预测。真实模型会更新许多步，因此需要关心这些逐步影响力如何随时间累积。我们先分析在 MNIST 上训练 LeNet 的简单例子（LeCun et al. 1998a）。

See Figure 1-(a), where the network $\pi_{\theta^{t}}$ is updating its parameters using the loss calculated on one training example ({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}}=\bm{\mathsf{e}}_{\texttt{4}}). The residual term \mathcal{G}^{t}_{\text{CE}}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}}) is then represented by the red arrows, which all start from \pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}}) and point to $\bm{\mathsf{e}}_{\texttt{4}}$. We can then ask how the model’s predictions on different {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} change after this update. As in Figure 1-(b), for an {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} in the same class with {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} (i.e., the identical case), the predicted probability of this correct label is “pulled up” by this update, as expected. On the other hand, if this {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} is similar to {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} (i.e., $\|\mathcal{K}^{t}\|_{F}$ is reasonably large) but comes from another class, then the predicted probability on {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}}’s class (not the correct label of {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}}) would be “pulled up,” as in the second panel of Figure 1-(b). Last, for examples that look dissimilar to {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} (small $\|\mathcal{K}^{t}\|_{F}$), this update will not change the model’s prediction on {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} much, as in the bottom panel in Figure 1-(b).

见图 1-(a)：网络 $\pi_{\theta^{t}}$ 用单个训练样本 ({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}}=\bm{\mathsf{e}}_{\texttt{4}}) 上的损失更新参数。残差项 \mathcal{G}^{t}_{\text{CE}}({\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}},{\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}}) 用红色箭头表示，均从 \pi_{\theta^{t}}(\bm{\mathsf{y}}\mid{\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}}) 指向 $\bm{\mathsf{e}}_{\texttt{4}}$。接着可问：这次更新后，模型对不同 {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 的预测如何变化。如图 1-(b)，若 {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 与 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 同类（即相同类情形），正确标签的预测概率会如预期被「拉高」。另一方面，若该 {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 与 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 相似（即 $\|\mathcal{K}^{t}\|_{F}$ 足够大）却来自另一类，则 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 所在类（而非 {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 的正确标签）的预测概率会被「拉高」，见图 1-(b) 第二幅。最后，对与 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 外观不相似的样本（$\|\mathcal{K}^{t}\|_{F}$ 小），这次更新几乎不改变模型对 {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 的预测，见图 1-(b) 底幅。

Figure 1: The per-step learning dynamics and the accumulated influence in an MNIST experiment.

图 1：MNIST 实验中的逐步学习动力学与累积影响力。

The interactions among the updates of different inputs then form an interesting pattern for the learned predictions. As illustrated in Figure 1-(c), when making predictions on images coming from class 4, the model tends to assign higher confidence on class 9. That is because the examples in class 9 on average look more similar to class 4 than examples in other classes. Hence the update of examples in classes 4 and 9 will reinforce their mutual influence and lead to a bump in their predictions. To verify this, we plot the average of $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})$ for $\bm{\mathsf{x}}$ from each of the classes in Figure 1-(d). The values of some off-diagonal patches are significantly higher than others, which means the examples in those classes look more similar, like 4 and 9, 5 and 3, 8 and 5, etc.

不同输入更新之间的相互作用，会在学到的预测上形成有趣图案。如图 1-(c) 所示，对来自类别 4 的图像做预测时，模型往往对类别 9 给出更高置信度。因为类别 9 的样本平均而言比其它类更像类别 4，于是类别 4 与 9 样本的更新会相互强化影响力，在彼此预测上形成鼓包。为验证这一点，我们在图 1-(d) 画出各类 $\bm{\mathsf{x}}$ 上 $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})$ 的平均。部分非对角块的值显著高于其它，说明这些类的样本更相似，例如 4 与 9、5 与 3、8 与 5 等。

3 Learning Dynamics of LLM’s Finetuning

3 LLM 微调的学习动力学

Although learning dynamics have been applied to many deep learning systems, extending this framework to LLM finetuning is non-trivial. The first problem is the high dimensionality and the sequence nature of both the input and output signals. The high-dimensional property makes it hard to observe the model’s output, and the sequential nature makes the distributions on different tokens mutually dependent, which is more complicated than a standard multi-label classification problem considered by most previous work. Furthermore, as there are many different algorithms for LLM finetuning – SFT (Wei et al. 2022a), RLHF (Ouyang et al. 2022a), DPO (Rafailov et al. 2023a), etc. – analyzing them under a uniform framework is challenging. Finally, compared with the training-from-scratch scenario, where a roughly uniform distribution over all possible outputs is usually assumed, LLMs’ finetuning dynamics heavily rely on the pretrained base model, which could make the analysis harder. For example, the pretrained model usually assigns little probability mass to unlikely tokens, which is good for most applications but leads to risk of the “squeezing effect” we show later. We now tackle these problems and propose a unified framework for different finetuning methods.

尽管学习动力学已用于许多深度学习系统，将其扩展到 LLM 微调并非易事。首要问题是输入与输出信号都具有高维性与序列性。高维使观察模型输出困难，序列性又使不同 token 上的分布彼此依赖，比多数先前工作考虑的标准多标签分类更复杂。此外，LLM 微调算法众多——SFT（Wei et al. 2022a）、RLHF（Ouyang et al. 2022a）、DPO（Rafailov et al. 2023a）等——要在统一框架下分析它们颇具挑战。最后，与通常假设输出近似均匀的从零训练不同，LLM 微调动力学严重依赖预训练基座，分析更难。例如，预训练模型通常把很少概率质量分给不太可能的 token，这对多数应用有利，却会带来后文所示「挤压效应」的风险。下文我们着手解决这些问题，并提出面向不同微调方法的统一框架。

3.1 Per-step Decomposition of the SFT Loss

3.1 SFT 损失的逐步分解

The typical loss function used during supervised finetuning is the negative log-likelihood (NLL) of a given completion $\bm{\mathsf{y}}^{+}_{u}=[y^{+}_{1},\dots,y^{+}_{L}]\in\mathcal{V}^{L}$, conditioned on the prompt $\bm{\mathsf{x}}_{u}$:

监督微调中常用的损失是给定补全 $\bm{\mathsf{y}}^{+}_{u}=[y^{+}_{1},\dots,y^{+}_{L}]\in\mathcal{V}^{L}$、以提示 $\bm{\mathsf{x}}_{u}$ 为条件的负对数似然（NLL）：

$$\mathcal{L}_{\text{SFT}}(\bm{\mathsf{x}}_{u},\bm{\mathsf{y}}_{u}^{+})\triangleq-\sum_{l=1}^{L}\log\pi(y=y_{l}^{+}\mid\bm{\mathsf{y}}^{+}_{<l},\bm{\mathsf{x}}_{u})=-\sum_{l=1}^{L}\bm{\mathsf{e}}_{y_{l}^{+}}\cdot\log\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}}_{u},\bm{\mathsf{y}}^{+}_{<l}). \tag{4}$$

Note that compared with the multi-label classification problem discussed before, where the joint distribution of all labels can be factorized as $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})=\prod_{l}\pi(y_{l}\mid\bm{\mathsf{x}})$, the sequential nature of language modeling makes the analysis more complicated, because we must have $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})=\prod_{l}\pi(y_{l}\mid\bm{\mathsf{x}},\bm{\mathsf{y}}_{<l})$. To solve this problem, we can merge this factorization into the definition of the backbone $h_{\theta}$ while keeping the format of Proposition 1. Specifically, letting $\bm{\mathsf{\chi}}$ be the concatenation of $\bm{\mathsf{x}}$ and $\bm{\mathsf{y}}$, the prediction of all tokens of $\bm{\mathsf{y}}$ is

注意，与前文多标签分类中联合分布可分解为 $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})=\prod_{l}\pi(y_{l}\mid\bm{\mathsf{x}})$ 不同，语言建模的序列性使分析更复杂，因为必须有 $\pi(\bm{\mathsf{y}}\mid\bm{\mathsf{x}})=\prod_{l}\pi(y_{l}\mid\bm{\mathsf{x}},\bm{\mathsf{y}}_{<l})$。为处理这一点，可将该分解并入骨干 $h_{\theta}$ 的定义，同时保持命题 1 的形式。具体地，令 $\bm{\mathsf{\chi}}$ 为 $\bm{\mathsf{x}}$ 与 $\bm{\mathsf{y}}$ 的拼接，则 $\bm{\mathsf{y}}$ 全部 token 的预测为

$$\bm{\mathsf{z}}=h_{\theta}\left(\bm{\mathsf{\chi}}\right);\quad\pi\left(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}\right)=\Softmaxcol\left(\bm{\mathsf{z}}\right).$$

Here $\bm{\mathsf{z}}$ is a $V\times L$ matrix where each column contains the logits of the prediction of the $l$th token. Our $h_{\theta}$, even though it takes the entire sequence ${\bm{\mathsf{\chi}}}$ as its input, will force the model not to refer to the future tokens $\bm{\mathsf{y}}_{>l}$ when making predictions on the $l$-th token, commonly implemented via “causal masking” (proposed in Vaswani et al. 2017a, details in Figure 10(a) of Appendix D). Then, we can calculate $(\nabla_{\theta}\bm{\mathsf{z}}_{l}(\bm{\mathsf{\chi}}_{o})|_{\theta^{t}})(\nabla_{\theta}\bm{\mathsf{z}}_{l}(\bm{\mathsf{\chi}}_{u})|_{\theta^{t}})^{\top}$ on each column of $\bm{\mathsf{z}}$ and stack them to form a $V\times V\times L$ tensor $\mathcal{K}^{t}(\bm{\mathsf{\chi}}_{o},\bm{\mathsf{\chi}}_{u})$. The calculation of $\mathcal{G}^{t}$ and $\mathcal{A}^{t}$ also follows a similar procedure. Thanks to the causal mask implemented in $h_{\theta}$, the resulting decomposition is almost identical to that in a multi-label classification problem. Assuming have a response $\bm{\mathsf{y}}_{u}$ of length $L$ associated with $\bm{\mathsf{x}}_{u}$, stacked into $\bm{\mathsf{\chi}}_{u}$, and $\bm{\mathsf{y}}_{o}$ of length $M$ associated with $\bm{\mathsf{x}}_{o}$, stacked into $\bm{\mathsf{\chi}}_{o}$. The change of the model’s prediction on the $m$-th token of $\bm{\mathsf{y}}_{o}$ can be represented as, when $\bm{\mathsf{z}}$ gradients have bounded norm,

这里 $\bm{\mathsf{z}}$ 是 $V\times L$ 矩阵，每列含第 $l$ 个 token 预测的 logits。我们的 $h_{\theta}$ 虽以整段序列 ${\bm{\mathsf{\chi}}}$ 为输入，但会强制模型在预测第 $l$ 个 token 时不参考未来 token $\bm{\mathsf{y}}_{>l}$，通常由「因果掩码」实现（Vaswani et al. 2017a 提出，细节见附录 D 图 10(a)）。于是可对 $\bm{\mathsf{z}}$ 的每列计算 $(\nabla_{\theta}\bm{\mathsf{z}}_{l}(\bm{\mathsf{\chi}}_{o})|_{\theta^{t}})(\nabla_{\theta}\bm{\mathsf{z}}_{l}(\bm{\mathsf{\chi}}_{u})|_{\theta^{t}})^{\top}$，再堆成 $V\times V\times L$ 张量 $\mathcal{K}^{t}(\bm{\mathsf{\chi}}_{o},\bm{\mathsf{\chi}}_{u})$。$\mathcal{G}^{t}$ 与 $\mathcal{A}^{t}$ 的计算类似。得益于 $h_{\theta}$ 中的因果掩码，所得分解几乎与多标签分类相同。假设与 $\bm{\mathsf{x}}_{u}$ 关联的回复 $\bm{\mathsf{y}}_{u}$ 长度为 $L$，拼成 $\bm{\mathsf{\chi}}_{u}$；与 $\bm{\mathsf{x}}_{o}$ 关联的 $\bm{\mathsf{y}}_{o}$ 长度为 $M$，拼成 $\bm{\mathsf{\chi}}_{o}$。在 $\bm{\mathsf{z}}$ 梯度范数有界时，$\bm{\mathsf{y}}_{o}$ 第 $m$ 个 token 上模型预测的变化可表示为

$$[\underbrace{\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{o})}_{V\times M}]_{m}=-\sum_{l=1}^{L}\eta[\underbrace{\mathcal{A}^{t}(\bm{\mathsf{\chi}}_{o})}_{V\times V\times M}]_{m}[\underbrace{\mathcal{K}^{t}(\bm{\mathsf{\chi}}_{o},\bm{\mathsf{\chi}}_{u})}_{V\times V\times L}]_{l}[\underbrace{\mathcal{G}^{t}(\bm{\mathsf{\chi}}_{u})}_{V\times L}]_{l}+\bigO(\eta^{2}), \tag{5}$$

where $\mathcal{G}^{t}_{\text{SFT}}\left(\bm{\mathsf{\chi}}_{u}\right)=\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{u})-\bm{\mathsf{y}}_{u}$. Compared with Proposition 1, the main difference is that the eNTK term also depends on the responses $\bm{\mathsf{y}}_{u}$ and $\bm{\mathsf{y}}_{o}$, which allows us to answer questions like

其中 $\mathcal{G}^{t}_{\text{SFT}}\left(\bm{\mathsf{\chi}}_{u}\right)=\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{u})-\bm{\mathsf{y}}_{u}$。与命题 1 相比，主要差别是 eNTK 项也依赖于回复 $\bm{\mathsf{y}}_{u}$ 与 $\bm{\mathsf{y}}_{o}$，从而可回答如下问题

For a prompt $\bm{\mathsf{x}}_{u}$, how does learning the response {\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}^{+}} influence the model’s belief about a response {\color[rgb]{1,0.5,0}\bm{\mathsf{y}}_{u}^{\prime}}?

对提示 $\bm{\mathsf{x}}_{u}$，学习回复 {\color[rgb]{0,1,1}\bm{\mathsf{y}}_{u}^{+}} 如何影响模型对回复 {\color[rgb]{1,0.5,0}\bm{\mathsf{y}}_{u}^{\prime}} 的信念？

When tracking the model’s confidence on different responses given the question $\bm{\mathsf{x}}_{u}$, learning from $\bm{\mathsf{y}}_{u}^{+}$ will impose a strong “upwards” pressure on $\bm{\mathsf{y}}_{u}^{+}$, as illustrated in the first panel of Figure 2. At the same time, the confidence of “similar responses” will also be slightly pulled up, like how learning a 4 influences the prediction on 9 in the MNIST example. We will discuss how to understand the “similarity” between two sequences of responses in the next section.

在跟踪给定问题 $\bm{\mathsf{x}}_{u}$ 下模型对不同回复的置信度时，从 $\bm{\mathsf{y}}_{u}^{+}$ 学习会对 $\bm{\mathsf{y}}_{u}^{+}$ 施加强烈「向上」压力，如图 2 第一幅所示。与此同时，「相似回复」的置信度也会被略微拉高，类似 MNIST 中学习 4 会影响对 9 的预测。下一节讨论如何理解两段回复之间的「相似性」。

3.2 Per-step Decomposition of the DPO Loss

3.2 DPO 损失的逐步分解

Preference finetuning, which teaches the model to provide responses that align better with human preferences, is also an important phase in LLM finetuning pipelines. Different from the SFT stage above, where the training tells the model “what to do”, many preference finetuning methods also teach the model “what not to do,” which makes the learning dynamics more complex. For intuition, we start by analyzing a typical method, i.e., DPO (direct preference optimization, an RL-free method), under a similar framework. Following Rafailov et al. 2023a, the loss function of off-policy DPO is

偏好微调教模型给出更符合人类偏好的回复，同样是 LLM 微调流程中的重要阶段。与上文 SFT「告诉模型该做什么」不同，许多偏好微调方法还会教模型「不该做什么」，学习动力学因此更复杂。为建立直觉，我们先在相似框架下分析典型方法 DPO（直接偏好优化，一种免 RL 方法）。按 Rafailov et al. 2023a，off-policy DPO 的损失函数为

$$\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{(\bm{\mathsf{x}}_{u},\bm{\mathsf{y}}_{u}^{+},\bm{\mathsf{y}}_{u}^{-})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta^{t}}({\bm{\mathsf{y}}_{u}^{+}}\mid\bm{\mathsf{\chi}}_{u}^{+})}{\pi_{\text{ref}}({\bm{\mathsf{y}}_{u}^{+}}\mid\bm{\mathsf{\chi}}_{u}^{+})}-\beta\log\frac{\pi_{\theta^{t}}({\bm{\mathsf{y}}_{u}^{-}}\mid\bm{\mathsf{\chi}}_{u}^{-})}{\pi_{\text{ref}}({\bm{\mathsf{y}}_{u}^{-}}\mid\bm{\mathsf{\chi}}_{u}^{-})}\right)\right], \tag{6}$$

where ${\bm{\mathsf{y}}_{u}^{+}}$ and ${\bm{\mathsf{y}}_{u}^{-}}$ are pre-generated responses, and $\pi_{\text{ref}}$ is the reference model, typically the result of SFT. In the loss function, the $\pi_{\theta^{t}}$ terms are also calculated using teacher forcing. Hence we can decompose the learning dynamics for DPO similarly to Equation 5,

其中 ${\bm{\mathsf{y}}_{u}^{+}}$ 与 ${\bm{\mathsf{y}}_{u}^{-}}$ 是预先生成的回复，$\pi_{\text{ref}}$ 为参考模型，通常是 SFT 结果。损失中的 $\pi_{\theta^{t}}$ 项同样用 teacher forcing 计算，故可将 DPO 的学习动力学按与式 5 类似的方式分解：

$$\displaystyle[\Delta\log\pi^{t}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{o})]_{m}=-\sum_{l=1}^{L}\eta[\mathcal{A}^{t}(\bm{\mathsf{\chi}}_{o})]_{m}\left([\mathcal{K}^{t}(\bm{\mathsf{\chi}}_{o},{\bm{\mathsf{\chi}}}_{u}^{+})]_{l}[\mathcal{G}_{\text{DPO+}}^{t}]_{l}-[\mathcal{K}^{t}(\bm{\mathsf{\chi}}_{o},{\bm{\mathsf{\chi}}}_{u}^{-})]_{l}[\mathcal{G}_{\text{DPO-}}^{t}]_{l}\right)+\bigO(\eta^{2}) \\ \displaystyle\mathcal{G}^{t}_{\text{DPO+}}=\beta(1-a)\left(\pi_{\theta^{t}}({\bm{\mathsf{y}}}\mid\bm{\mathsf{\chi}}_{u}^{+})-{\bm{\mathsf{y}}_{u}^{+}}\right);\qquad\mathcal{G}^{t}_{\text{DPO-}}=\beta(1-a)\left(\pi_{\theta^{t}}({\bm{\mathsf{y}}}\mid\bm{\mathsf{\chi}}_{u}^{-})-{\bm{\mathsf{y}}_{u}^{-}}\right), \tag{7}$$

where $a$ is the margin (i.e., the $\sigma(\cdot)$ value) for the $l$-th token, which represents how well the current policy separates $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ compared with the reference policy. Due to the monotonicity of $\sigma(\cdot)$, a larger margin leads to larger $a$, which in turn restrains the strength of $\mathcal{G}^{t}_{\text{DPO+/-}}$. In other words, $\mathcal{G}^{t}_{\text{DPO+/-}}$ automatically provides less energy on the examples that are already well separated. We then check the role of $\beta$, which controls the regularizing effect on the KL distance between $\pi_{\theta^{t}}$ and $\pi_{\text{ref}}$ in the original RL loss (Rafailov et al. 2023a). When the margin is negative, larger $\beta$ leads to a smaller $a$ and hence provides stronger $\mathcal{G}^{t}_{\text{DPO+/-}}$ for the model to “catch up” the separating ability of the reference model faster. But when the model is better and has a positive margin, increasing $\beta$ will increase $a$ and hence create a negative influence on $\beta(1-a)$, which makes the model update less. This aligns well with the claims of Rafailov et al. 2023a: the stronger regularizing effect tends to “drag $\pi_{\theta}$ back towards $\pi_{\text{ref}}$” when its predictions deviate from $\pi_{\text{ref}}$ too much. The derivation and the $\mathcal{G}^{t}$ functions for other RL-free methods are given in Section B.2.2.

其中 $a$ 是第 $l$ 个 token 的间隔（即 $\sigma(\cdot)$ 的值），表示当前策略相对参考策略区分 $\bm{\mathsf{y}}_{u}^{+}$ 与 $\bm{\mathsf{y}}_{u}^{-}$ 的好坏。由于 $\sigma(\cdot)$ 单调，间隔越大则 $a$ 越大，从而抑制 $\mathcal{G}^{t}_{\text{DPO+/-}}$ 的强度。换言之，对已分得很开的样本，$\mathcal{G}^{t}_{\text{DPO+/-}}$ 会自动提供更少能量。再看 $\beta$ 的作用：它在原始 RL 损失中控制 $\pi_{\theta^{t}}$ 与 $\pi_{\text{ref}}$ 之间 KL 距离的正则强度（Rafailov et al. 2023a）。当间隔为负时，更大的 $\beta$ 使 $a$ 更小，从而给出更强的 $\mathcal{G}^{t}_{\text{DPO+/-}}$，让模型更快「追上」参考模型的区分能力。但当模型更好、间隔为正时，增大 $\beta$ 会增大 $a$，进而对 $\beta(1-a)$ 产生负向影响，使更新变弱。这与 Rafailov et al. 2023a 的论断一致：更强的正则倾向于在预测偏离 $\pi_{\text{ref}}$ 过多时「把 $\pi_{\theta}$ 拉回 $\pi_{\text{ref}}$」。推导及其它免 RL 方法的 $\mathcal{G}^{t}$ 见第 B.2.2 节。

These analyses make no assumptions on where $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ come from. Hence our framework can be directly extended to on-policy RL-free methods, which often perform better than their off-policy counterparts (Tajwar et al. 2024a; Guo et al. 2024c). The main difference between these algorithms is how the supervisory responses are generated. Off-policy methods typically use a fixed pre-collected dataset, with $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ are generated by another LLM or humans. In other words, it is likely that both the chosen and rejected responses come from the “less likely” region of the model’s prediction. On-policy responses, though, are more likely to have higher predicted probabilities under this model, as they were sampled from it. We will show next that imposing large negative pressure on an unlikely prediction will lead to unexpected behavior, giving a new explanation for why on-policy sampling is important for algorithms with large negative gradients.

上述分析不对 $\bm{\mathsf{y}}_{u}^{+}$ 与 $\bm{\mathsf{y}}_{u}^{-}$ 的来源做假设，因此框架可直接扩展到往往优于 off-policy 对照的 on-policy 免 RL 方法（Tajwar et al. 2024a；Guo et al. 2024c）。算法间的主要差别在于监督回复如何生成。Off-policy 方法通常使用固定预采集数据集，$\bm{\mathsf{y}}_{u}^{+}$ 与 $\bm{\mathsf{y}}_{u}^{-}$ 由另一 LLM 或人类生成；换言之，偏好与拒绝回复都可能落在模型预测的「不太可能」区域。On-policy 回复则更可能在该模型下具有更高预测概率，因为它们是从该模型采样得到的。下一节将表明：对不太可能的预测施加很大的负压会导致意外行为，从而为「为何带大负梯度的算法需要 on-policy 采样」提供新解释。

Figure 2: The updating vector provided by the residual term $\mathcal{G}^{t}$ of different algorithms. The gray $\bm{\mathsf{y}}$ are responses sampled from $\pi$ in an on-policy way. In the second panel, we demonstrate the “squeezing effect” caused by imposing a big negative gradient on a “valley” region of a distribution. For more details about this counter-intuitive effect, please refer to Section 3.3 and Appendix E. Other panels demonstrate on-policy DPO (and IPO), SPIN (Chen et al. 2024a), SPPO (Wu et al. 2024a), and SLiC (Zhao et al. 2023a).

图 2：不同算法残差项 $\mathcal{G}^{t}$ 提供的更新向量。灰色 $\bm{\mathsf{y}}$ 是以 on-policy 方式从 $\pi$ 采样的回复。第二幅展示在分布的「谷底」区域施加大负梯度引起的「挤压效应」。该反直觉效应的更多细节见第 3.3 节与附录 E。其余各幅展示 on-policy DPO（及 IPO）、SPIN（Chen et al. 2024a）、SPPO（Wu et al. 2024a）与 SLiC（Zhao et al. 2023a）。

3.3 The Squeezing Effect Caused by Negative Gradient

3.3 负梯度引起的挤压效应

As demonstrated by the first two panels in Figure 2, the use of large negative gradients is the main difference between the learning dynamics of SFT and DPO. We will show later that this difference is the key to understanding why the learning curves of SFT and DPO behave so differently. For example, Rafailov et al. 2024a; Tajwar et al. 2024a; Pal et al. 2024a (and our Figure 4) reported that the confidence of both $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ gradually decreases when conducting DPO, while the confidence of $\bm{\mathsf{y}}_{u}^{+}$ rarely drops during SFT. This trend becomes more serious if $\pi_{\theta^{0}}$ is finetuned for longer before conducting DPO (reported in Figure 3 of Rafailov et al. 2024a and verified by our Figure 17). Furthermore, we also find that for all the $\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{u})$ we track (various responses similar to $\bm{\mathsf{y}}_{u}^{+}$ or $\bm{\mathsf{y}}_{u}^{-}$; details in the next section), none of them increase during the DPO phase. This is different from SFT and quite counter-intuitive:

如图 2 前两幅所示，使用大负梯度是 SFT 与 DPO 学习动力学的主要差别。后文将表明，正是这一差别是理解为何 SFT 与 DPO 学习曲线表现如此不同的关键。例如，Rafailov et al. 2024a、Tajwar et al. 2024a、Pal et al. 2024a（以及我们的图 4）报告：做 DPO 时 $\bm{\mathsf{y}}_{u}^{+}$ 与 $\bm{\mathsf{y}}_{u}^{-}$ 的置信度都会逐渐下降，而 SFT 中 $\bm{\mathsf{y}}_{u}^{+}$ 的置信度很少下降。若在做 DPO 前对 $\pi_{\theta^{0}}$ 微调更久，该趋势更严重（见 Rafailov et al. 2024a 图 3，并由我们的图 17 验证）。此外我们还发现，所跟踪的所有 $\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{u})$（与 $\bm{\mathsf{y}}_{u}^{+}$ 或 $\bm{\mathsf{y}}_{u}^{-}$ 相似的各类回复；细节见下节）在 DPO 阶段无一上升。这与 SFT 不同，且相当反直觉：

If everything we observe is becoming less confident, where has the probability mass gone?

若我们观察到的一切都变得更不自信，概率质量去哪了？

To answer this question, we first identify a phenomenon we call the squeezing effect, which occurs when using negative gradients from any model outputting a distribution with $\Softmax$ output heads, even in a simple multi-class logistic regression task. Specifically, in the $L=1$ case when we impose a negative gradient on label $\bm{\mathsf{y}}_{u}^{-}$, we can describe the changing of model’s predictive distribution $\pi_{\theta^{t+1}}$ as follows:

为回答该问题，我们先指出一种称为挤压效应的现象：只要模型用 Softmax 输出头给出分布，即便在简单的多类逻辑回归任务中，使用负梯度也会出现该现象。具体地，在 $L=1$ 情形对标签 $\bm{\mathsf{y}}_{u}^{-}$ 施加负梯度时，可将预测分布 $\pi_{\theta^{t+1}}$ 的变化描述如下：

• • Guarantee: the confidence of $\bm{\mathsf{y}}_{u}^{-}$, i.e., $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}_{u}^{-})$ will decrease.
• • Guarantee: the decreased probability mass is largely “squeezed” into the output which was most confident before the update: if $\bm{\mathsf{y}}^{*}=\argmax_{i\in[V]\setminus\{\bm{\mathsf{y}}_{u}^{-}\}}\pi_{\theta^{t}}(\bm{\mathsf{y}}=i)$, then $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}=\bm{\mathsf{y}}^{*})$ will increase.
• • Trend: the rich get richer and the poor get poorer: generally, dimensions with high $\pi_{\theta^{t}}$ tend to increase, and those with low $\pi_{\theta^{t}}$ tend to decrease.
• • Trend: peakier $\pi_{\theta^{t}}$ squeezes more. If the probability mass concentrates on few dimensions in $\pi_{\theta^{t}}$, which is common for a pretrained model, all $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}\neq\bm{\mathsf{y}}^{*})$ decrease (only $\bm{\mathsf{y}}^{*}$ is “rich”).
• • Trend: smaller $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$ exacerbate the squeezing effect: if $\bm{\mathsf{y}}_{u}^{-}$ is unlikely under $\pi_{\theta^{t}}$, the probability mass of all other $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}\neq\bm{\mathsf{y}}^{*})$ will be more seriously decreased, and the $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}=\bm{\mathsf{y}}^{*})$ increases more.

• • 保证：$\bm{\mathsf{y}}_{u}^{-}$ 的置信度，即 $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}_{u}^{-})$ 会下降。
• • 保证：下降的概率质量很大程度上被「挤压」到更新前最自信的输出上：若 $\bm{\mathsf{y}}^{*}=\argmax_{i\in[V]\setminus\{\bm{\mathsf{y}}_{u}^{-}\}}\pi_{\theta^{t}}(\bm{\mathsf{y}}=i)$，则 $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}=\bm{\mathsf{y}}^{*})$ 会上升。
• • 趋势：富者愈富、贫者愈贫：一般地，$\pi_{\theta^{t}}$ 高的维倾向于上升，低的维倾向于下降。
• • 趋势：更尖峰的 $\pi_{\theta^{t}}$ 挤压更强。若 $\pi_{\theta^{t}}$ 的概率质量集中在少数维上（预训练模型常见），则所有 $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}\neq\bm{\mathsf{y}}^{*})$ 都下降（只有 $\bm{\mathsf{y}}^{*}$「富」）。
• • 趋势：更小的 $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-})$ 会加剧挤压：若 $\bm{\mathsf{y}}_{u}^{-}$ 在 $\pi_{\theta^{t}}$ 下不太可能，则其它所有 $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}\neq\bm{\mathsf{y}}^{*})$ 下降更严重，且 $\pi_{\theta^{t+1}}(\bm{\mathsf{y}}=\bm{\mathsf{y}}^{*})$ 上升更多。

Appendix E illustrates the squeezing effect and analytically proves its existence for logistic regression models, by directly computing $\pi_{\theta^{t+1}}/\pi_{\theta^{t}}$ in different situations. Section 4.2 also experimentally verifies the analysis above in real LLM experiments. Note that in practical settings, where both positive and negative pressures and the auto-regressive nature are strictly considered, the squeezing effect can become more complicated. The differences between the two eNTK terms in Equation 7 also influence the relative strength and direction of these two pressures. Razin et al. 2025a analyze a similar problem in a token-level setting, and their conclusions align with ours well. We left a more detailed analysis to our future work.

附录 E 说明挤压效应，并通过直接计算不同情形下的 $\pi_{\theta^{t+1}}/\pi_{\theta^{t}}$，对逻辑回归模型给出存在性的解析证明。第 4.2 节亦在真实 LLM 实验中验证上述分析。注意在同时严格考虑正负压力与自回归特性的实际设定中，挤压效应会更复杂。式 7 中两个 eNTK 项的差异也会影响这两种压力的相对强度与方向。Razin et al. 2025a 在 token 级设定下分析了类似问题，结论与我们一致。更细致的分析留待未来工作。

We can now get a high-level overview of the learning dynamics of a typical off-policy DPO algorithm. Since both $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ are not sampled from the model’s distribution, $\bm{\mathsf{y}}^{*}$ sometimes can be dissimilar to $\bm{\mathsf{y}}_{u}^{+}$, and the $\bm{\mathsf{y}}_{u}^{-}$ are likely located in a valley region of the model’s prediction. Then its learning dynamics would look like the sketch in the second panel of Figure 2: the confidence on almost all $\bm{\mathsf{y}}$ are pushed down. At the same time, all the decreased probability mass is squeezed to $\bm{\mathsf{y}}^{*}$, which might make the model keep generating repeated phrases, as reported by Holtzman et al. 2020a. Variants of DPO algorithms often unintentionally mitigate this squeezing effect by constraining the strength of the negative gradient or the positions of $\bm{\mathsf{y}}_{u}^{-}$, which partially explains their benefits. The last four panels in Figure 2 and Section B.2.2 have more details.

至此可对典型 off-policy DPO 算法的学习动力学有一高层概览。由于 $\bm{\mathsf{y}}_{u}^{+}$ 与 $\bm{\mathsf{y}}_{u}^{-}$ 都不是从模型分布采样，$\bm{\mathsf{y}}^{*}$ 有时会与 $\bm{\mathsf{y}}_{u}^{+}$ 不相似，而 $\bm{\mathsf{y}}_{u}^{-}$ 很可能落在模型预测的谷底区域。于是其学习动力学会如图 2 第二幅草图：几乎所有 $\bm{\mathsf{y}}$ 上的置信度都被压低；同时下降的概率质量被挤压到 $\bm{\mathsf{y}}^{*}$，可能使模型不断生成重复短语，正如 Holtzman et al. 2020a 所报告。各类 DPO 变体往往通过约束负梯度强度或 $\bm{\mathsf{y}}_{u}^{-}$ 的位置无意中缓解挤压效应，这部分解释了它们的收益。图 2 后四幅与第 B.2.2 节有更多细节。

4 Experimental Verifications

4 实验验证

We now verify our analysis in practical settings. We first create the training set $\mathcal{D}_{\text{train}}$ by randomly selecting 5000 examples from the training split of the dataset. We consider two common datasets, Antropic-HH (Bai et al. 2022a) and UltraFeedback (Cui et al. 2023a), in all experiments. Each example in $\mathcal{D}_{\text{train}}$ contains three components: the prompt (or question) $\bm{\mathsf{x}}$, the preferred response $\bm{\mathsf{y}}^{+}$, and the less preferred response $\bm{\mathsf{y}}^{-}$. SFT finetunes with $\bm{\mathsf{x}}$ and $\bm{\mathsf{y}}^{+}$, while DPO uses all three (subscripts of $\bm{\mathsf{x}}$ and $\bm{\mathsf{y}}$ are removed for conciseness). We repeat the experiments on two series of models: pythia-410M/1B/1.4B/2.8B (Biderman et al. 2023a) and Qwen1.5-0.5B/1.8B (Bai et al. 2023a).

我们现在在实际设定中验证分析。首先从数据集训练划分随机选取 5000 条构成训练集 $\mathcal{D}_{\text{train}}$。全部实验考虑两个常用数据集：Antropic-HH（Bai et al. 2022a）与 UltraFeedback（Cui et al. 2023a）。$\mathcal{D}_{\text{train}}$ 中每条含三部分：提示（或问题）$\bm{\mathsf{x}}$、偏好回复 $\bm{\mathsf{y}}^{+}$、较不偏好回复 $\bm{\mathsf{y}}^{-}$。SFT 用 $\bm{\mathsf{x}}$ 与 $\bm{\mathsf{y}}^{+}$ 微调，DPO 三者皆用（为简洁省略 $\bm{\mathsf{x}}$、$\bm{\mathsf{y}}$ 下标）。实验在两系列模型上重复：pythia-410M/1B/1.4B/2.8B（Biderman et al. 2023a）与 Qwen1.5-0.5B/1.8B（Bai et al. 2023a）。

To get a more detailed observation of the learning dynamics, we further create a probing dataset $\mathcal{D}_{\text{prob}}$ by randomly selecting 500 examples from $\mathcal{D}_{\text{train}}$, and generate several typical responses based on the corresponding $\bm{\mathsf{x}}$, $\bm{\mathsf{y}}^{+}$, or $\bm{\mathsf{y}}^{-}$. (We also study another probing dataset where all $\bm{\mathsf{x}}$ come from the test set in an ablation study in the appendix.) Then for each $\bm{\mathsf{x}}$ in $\mathcal{D}_{\text{prob}}$, we can observe how $\log\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}})$ gradually changes on different types of $\bm{\mathsf{y}}$. For example, one extended response type can be a rephrase of $\bm{\mathsf{y}}^{+}$, an irrelevant response answering another question $\bm{\mathsf{x}}^{\prime}$, or just a randomly generated English sentence with the same number of words with $\bm{\mathsf{y}}^{+}$. We explain why we need these extended responses and how they are generated in detail in Section D.1. In short, $\mathcal{D}_{\text{prob}}$ helps us to get a more fine-grind inspection of the learning dynamics, which can not only support our analysis above, but also shed more light on how the model’s prediction evolves on the entire $\mathcal{Y}\in\mathbb{R}^{V\times L}$, a very sparse and huge space.

为更细致观察学习动力学，我们再从 $\mathcal{D}_{\text{train}}$ 随机选 500 条构成探测集 $\mathcal{D}_{\text{prob}}$，并基于对应的 $\bm{\mathsf{x}}$、$\bm{\mathsf{y}}^{+}$ 或 $\bm{\mathsf{y}}^{-}$ 生成若干典型回复。（附录消融中还研究了另一探测集，其中所有 $\bm{\mathsf{x}}$ 来自测试集。）于是对 $\mathcal{D}_{\text{prob}}$ 中每个 $\bm{\mathsf{x}}$，可观察 $\log\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}})$ 在不同类型 $\bm{\mathsf{y}}$ 上如何逐渐变化。例如，扩展回复类型可以是 $\bm{\mathsf{y}}^{+}$ 的改写、回答另一问题 $\bm{\mathsf{x}}^{\prime}$ 的无关回复，或与 $\bm{\mathsf{y}}^{+}$ 词数相同的随机英文句。第 D.1 节详细说明为何需要这些扩展回复以及如何生成。简言之，$\mathcal{D}_{\text{prob}}$ 有助于更细粒度检视学习动力学，既支撑上文分析，也揭示模型预测在整个 $\mathcal{Y}\in\mathbb{R}^{V\times L}$（极其稀疏且巨大的空间）上如何演化。

4.1 Learning dynamics of SFT

4.1 SFT 的学习动力学

The main lesson we learn from the analysis in Section 3.1 is that learning from $\bm{\mathsf{y}}_{u}^{+}$ not only increases the model’s confidence on $\bm{\mathsf{y}}_{u}^{+}$, but also indirectly “pulls up” responses similar to $\bm{\mathsf{y}}_{u}^{+}$ with a smaller strength (scaled roughly by $\|\mathcal{K}^{t}\|_{F}$), similar to how learning a “4” influences the prediction of a “9” in the MNIST example. At the same time, the increase of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}|\bm{\mathsf{\chi}}_{u})$ naturally “pushes down” all $\bm{\mathsf{y}}\neq\bm{\mathsf{y}}_{u}^{+}$, because the model’s predicted probability to all responses in $\mathcal{Y}$-space must sum to one. The model’s behavior on different $\bm{\mathsf{y}}$ is mostly a trade-off among these pressures. To verify this claim, we finetune the model for several epochs and evaluate the model’s prediction on all responses in $\mathcal{D}_{\text{prob}}$ every 25 updates (with a training batch size of 4, the probing occurs every 100 examples). For each type of response, we average the model’s confidence on all 500 examples and report the mean value of their log-likelihood.

第 3.1 节分析的主要启示是：从 $\bm{\mathsf{y}}_{u}^{+}$ 学习不仅提高模型对 $\bm{\mathsf{y}}_{u}^{+}$ 的置信度，还会以较小强度（大致按 $\|\mathcal{K}^{t}\|_{F}$ 缩放）间接「拉高」与 $\bm{\mathsf{y}}_{u}^{+}$ 相似的回复，类似 MNIST 中学习「4」会影响对「9」的预测。同时，$\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid\bm{\mathsf{\chi}}_{u})$ 的上升自然会「压低」所有 $\bm{\mathsf{y}}\neq\bm{\mathsf{y}}_{u}^{+}$，因为 $\mathcal{Y}$ 空间中所有回复的预测概率之和为 1。模型在不同 $\bm{\mathsf{y}}$ 上的行为主要是这些压力之间的权衡。为验证该论断，我们微调若干 epoch，每 25 次更新评估模型在 $\mathcal{D}_{\text{prob}}$ 全部回复上的预测（训练批大小为 4，即每 100 条样本探测一次）。对每类回复，在全部 500 例上平均模型置信度，并报告对数似然均值。

As demonstrated in the first panel of Figure 3, the model’s confidence on $\bm{\mathsf{y}}_{u}^{+}$ keeps increasing throughout the whole learning process, which is straightforward because the main “pull-up” pressure is imposed directly on $\bm{\mathsf{y}}_{u}^{+}$. However, the behavior of some responses similar to $\bm{\mathsf{y}}_{u}^{+}$ is non-trivial. For example, we draw the following types of responses in the same panel, i.e., the less preferred response for the same question ($\bm{\mathsf{y}}_{u}^{-}$), two types of rephrases of $\bm{\mathsf{y}}_{u}^{+}$ generated by ChatGPT ($\bm{\mathsf{y}}_{\text{gpts}}^{+}$ and $\bm{\mathsf{y}}_{\text{gptf}}^{+}$), another preferred response randomly selected from the test set ($\bm{\mathsf{y}}_{\text{test}}^{+}$), or even a randomly generated English sentence ($\bm{\mathsf{y}}_{\text{hum}}$). The model’s confidence in these responses are all slightly increased at the beginning of training, and then gradually decrease as the training goes on, even though the model never sees them during SFT. This counter-intuitive behavior can be well explained by the learning dynamics we discussed before. Since all these examples are “similar” to $\bm{\mathsf{y}}_{u}^{+}$ to some extent (at least, they are all common “standard English” sentences), their $\|\mathcal{K}^{t}\|_{F}$ are reasonably large. Then learning $\bm{\mathsf{y}}_{u}^{+}$ will indirectly increase the model’s confidence of these similar $\bm{\mathsf{y}}$. That is why the corresponding $\pi_{\theta^{t}}(\bm{\mathsf{y}}|\bm{\mathsf{\chi}}_{u})$ are slightly increased at the beginning of training. However, as the training goes on, the model’s confidence on $\bm{\mathsf{y}}_{u}^{+}$ keeps increasing and the update energy, the norm of $\mathcal{G}_{\text{SFT}}^{t}$ in Equation 5, gradually decreases. That means the indirect “pull-up” pressures are also diminished accordingly. Then, the “push-down” pressure on all $\bm{\mathsf{y}}\neq\bm{\mathsf{y}}_{u}^{+}$ becomes dominant and all the related curves start going down.

如图 3 第一幅所示，模型对 $\bm{\mathsf{y}}_{u}^{+}$ 的置信度在整个学习过程中持续上升，这很直接，因为主要「拉高」压力直接作用在 $\bm{\mathsf{y}}_{u}^{+}$ 上。但一些与 $\bm{\mathsf{y}}_{u}^{+}$ 相似的回复行为并不平凡。同幅中我们画出：同一问题的较不偏好回复（$\bm{\mathsf{y}}_{u}^{-}$）、由 ChatGPT 生成的两类 $\bm{\mathsf{y}}_{u}^{+}$ 改写（$\bm{\mathsf{y}}_{\text{gpts}}^{+}$ 与 $\bm{\mathsf{y}}_{\text{gptf}}^{+}$）、从测试集随机选取的另一偏好回复（$\bm{\mathsf{y}}_{\text{test}}^{+}$），乃至随机生成的英文句（$\bm{\mathsf{y}}_{\text{hum}}$）。这些回复上的置信度在训练初期都略有上升，随后随训练逐渐下降，尽管 SFT 中从未见过它们。该反直觉行为可由前述学习动力学很好解释。这些样本在某种程度上都与 $\bm{\mathsf{y}}_{u}^{+}$「相似」（至少都是常见「标准英文」句子），其 $\|\mathcal{K}^{t}\|_{F}$ 足够大，于是学习 $\bm{\mathsf{y}}_{u}^{+}$ 会间接提高这些相似 $\bm{\mathsf{y}}$ 的置信度，故训练初期对应 $\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{u})$ 略升。但随训练推进，对 $\bm{\mathsf{y}}_{u}^{+}$ 的置信度持续上升，式 5 中更新能量 $\|\mathcal{G}_{\text{SFT}}^{t}\|$ 逐渐减小，间接「拉高」压力随之减弱，于是对所有 $\bm{\mathsf{y}}\neq\bm{\mathsf{y}}_{u}^{+}$ 的「压低」压力占主导，相关曲线开始下降。

To verify the existence of this global “push-down” pressure, we observe two types of responses; both have the same number of words as their $\bm{\mathsf{y}}_{u}^{+}$. One is a purely random English word sequence $\bm{\mathsf{y}}^{\prime}_{\text{rnd}}$. Another is a random permutation of all the words in $\bm{\mathsf{y}}_{u}^{+}$, which is called $\bm{\mathsf{y}}_{\text{urnd}}^{+}$. Since both are not natural language, we expect the $\|\mathcal{K}^{t}\|_{F}$ between them and $\bm{\mathsf{y}}_{u}^{+}$ to be very small, which means learning from $\bm{\mathsf{y}}_{u}^{+}$ imposes almost no “pull-up” pressure on them; thus the “push-down” pressure will dominate through the whole training procedure. These analyses are well supported by the second panel in Figure 3, in which we see these $\pi_{\theta^{t}}(\bm{\mathsf{y}}|\bm{\mathsf{\chi}}_{u})$ all start from a very small value, and keep decreasing throughout the training.

为验证这种全局「压低」压力的存在，我们观察两类与各自 $\bm{\mathsf{y}}_{u}^{+}$ 词数相同的回复：一是纯随机英文词序列 $\bm{\mathsf{y}}^{\prime}_{\text{rnd}}$；二是 $\bm{\mathsf{y}}_{u}^{+}$ 全部词的随机置换，记为 $\bm{\mathsf{y}}_{\text{urnd}}^{+}$。二者都不是自然语言，我们预期它们与 $\bm{\mathsf{y}}_{u}^{+}$ 之间的 $\|\mathcal{K}^{t}\|_{F}$ 很小，即从 $\bm{\mathsf{y}}_{u}^{+}$ 学习几乎不对它们施加「拉高」压力，因而「压低」压力会贯穿整个训练。图 3 第二幅很好支持这些分析：这些 $\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{u})$ 均从很小的值起步，并在训练全程持续下降。

Another interesting type of responses is $\bm{\mathsf{y}}_{j\neq u}^{+}$, a preferred response for another question $\bm{\mathsf{x}}_{j\neq u}$ in the training set. For these responses, the model’s prediction on $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{j\neq u}^{+}|\bm{\mathsf{\chi}}_{u})$ will be kept influenced by two “pull-up” pressures: one is from learning $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+}]$, another is from learning $[\bm{\mathsf{x}}_{j\neq u};\bm{\mathsf{y}}_{j\neq u}^{+}]$, where the latter might be even stronger as the gradient is directly calculated by observing $\bm{\mathsf{y}}_{j\neq u}^{+}$. That explains why we see the confidence on $\bm{\mathsf{y}}_{j\neq u}^{+}$ keeps increasing with a smaller rate compared with $\bm{\mathsf{y}}_{u}^{+}$ in the third panel. Because the “pull-up” pressure is always strong enough to counter the “push-down” one. These observations provide us with a unique explanation of why specific types of hallucinations are amplified after SFT. Specifically, the increase of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{j\neq u}^{+}|\bm{\mathsf{\chi}}_{u})$ means if we ask the model to answer a question $\bm{\mathsf{x}}_{u}$, it might provide a response from (or partially from) another unrelated question $\bm{\mathsf{x}}_{j\neq u}$ in the training set.

另一类有趣的回复是 $\bm{\mathsf{y}}_{j\neq u}^{+}$，即训练集中另一问题 $\bm{\mathsf{x}}_{j\neq u}$ 的偏好回复。对这些回复，模型对 $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{j\neq u}^{+}\mid\bm{\mathsf{\chi}}_{u})$ 的预测会持续受到两股「拉高」压力影响：一来自学习 $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+}]$，二来自学习 $[\bm{\mathsf{x}}_{j\neq u};\bm{\mathsf{y}}_{j\neq u}^{+}]$，后者可能更强，因为梯度是通过直接观察 $\bm{\mathsf{y}}_{j\neq u}^{+}$ 算得的。这解释了为何第三幅中对 $\bm{\mathsf{y}}_{j\neq u}^{+}$ 的置信度以小于 $\bm{\mathsf{y}}_{u}^{+}$ 的速率持续上升——「拉高」压力始终足以对抗「压低」。这些观察为「为何某些幻觉在 SFT 后被放大」提供了独特解释：具体地，$\pi_{\theta^{t}}(\bm{\mathsf{y}}_{j\neq u}^{+}\mid\bm{\mathsf{\chi}}_{u})$ 上升意味着若问模型 $\bm{\mathsf{x}}_{u}$，它可能给出（或部分给出）训练集中另一无关问题 $\bm{\mathsf{x}}_{j\neq u}$ 的回复。

Figure 3: First three: learning dynamics of SFT on different response types. Fourth: SFT 10 epochs then DPO. Last: the accumulated influence when SFT using different $\bm{\mathsf{y}}$ (full results in Appendix C and D).

图 3：前三幅：SFT 在不同回复类型上的学习动力学。第四幅：SFT 10 个 epoch 再做 DPO。末幅：用不同 $\bm{\mathsf{y}}$ 做 SFT 时的累积影响力（完整结果见附录 C 与 D）。

Last, to further explore the “similarity” between different responses from the model’s perspective. we SFT the model using more types of responses and observe how $\pi_{\theta}(\bm{\mathsf{y}}^{\prime}\mid\bm{\mathsf{\chi}}_{u})$ changes accordingly. The results are demonstrated in Figure 3, where the blue and orange colors represent the positive and negative influence respectively. The x-axis is the updating response while the y-axis denotes the observing response. Hence the first column resembles how different $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}^{\prime}]$ changes when we SFT the model using $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+}]$. One interesting finding is that all responses generated by ChatGPT are considered very similar to each other, regardless of how semantically different they are. Probably, LLM has its preferred idioms or phrases, which could be considered as a type of “fingerprint”. We left this interesting problem for our future work.

最后，为进一步从模型视角探索不同回复之间的「相似性」，我们用更多类型的回复做 SFT，并观察 $\pi_{\theta}(\bm{\mathsf{y}}^{\prime}\mid\bm{\mathsf{\chi}}_{u})$ 如何相应变化。结果见图 3，蓝、橙分别表示正向与负向影响。横轴为更新所用回复，纵轴为观察回复。因此第一列类似：用 $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+}]$ 做 SFT 时，不同 $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}^{\prime}]$ 如何变化。一个有趣发现是：ChatGPT 生成的所有回复彼此都被视为非常相似，无论语义差别多大。或许 LLM 有其偏好的惯用表达或短语，可视为一种「指纹」。该有趣问题留待未来工作。

4.2 Learning dynamics of off-policy DPO

4.2 Off-policy DPO 的学习动力学

Figure 4: Learning dynamics of off-policy DPO. The last panel verifies the existence of the squeezing effect.

图 4：Off-policy DPO 的学习动力学。末幅验证挤压效应的存在。

To verify our framework also explains the model’s behavior in preference tuning, we conduct similar experiments for DPO. Recall the residual term $\mathcal{G}^{t}_{\text{DPO}}$ introduces a pair of arrows on both $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$, with different directions. To show how these two pressures influence the model, we check two types of rephrases of $\bm{\mathsf{y}}_{u}^{+}$ or $\bm{\mathsf{y}}_{u}^{-}$ ($\bm{\mathsf{y}}_{\text{gpts}}^{+}$, $\bm{\mathsf{y}}_{\text{gptf}}^{+}$, $\bm{\mathsf{y}}_{\text{gpts}}^{-}$, and $\bm{\mathsf{y}}_{\text{gptf}}^{-}$, used in the previous experiment). See the three curves in the first panel in Figure 4, where the two rephrases decrease at a similar speed, faster than the decay of $\bm{\mathsf{y}}_{u}^{+}$. That is because the upward pressure is directly imposed on $\bm{\mathsf{y}}_{u}^{+}$ rather than these rephrases. Similarly, in the second panel, we observe that $\bm{\mathsf{y}}_{u}^{-}$ decays faster than its rephrases, because $\mathcal{G}^{t}_{\text{DPO}}$ directly imposes a negative pressure on $\bm{\mathsf{y}}_{u}^{-}$. Then in the third panel, we find the rephrases of $\bm{\mathsf{y}}_{u}^{+}$ consistently decay slower than those of $\bm{\mathsf{y}}_{u}^{-}$, although none of them ever occur during training. That is because these responses are close to $\bm{\mathsf{y}}_{u}^{+}$ or $\bm{\mathsf{y}}_{u}^{-}$ in $\mathcal{Y}$, which means their $\|\mathcal{K}^{t}\|_{F}$ is relatively large. Hence the pressures imposed on $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ also introduce a non-negligible influence on them. Last, in the fourth panel, the margin $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}|\bm{\mathsf{\chi}}_{u})-\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}|\bm{\mathsf{\chi}}_{u})$ keeps increasing, which means the model is gaining the ability to separate $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ as the training goes on.

为验证框架同样能解释偏好微调中的模型行为，我们对 DPO 做类似实验。回忆 $\mathcal{G}^{t}_{\text{DPO}}$ 残差项在 $\bm{\mathsf{y}}_{u}^{+}$ 与 $\bm{\mathsf{y}}_{u}^{-}$ 上引入方向不同的一对箭头。为展示这两种压力如何影响模型，我们检查 $\bm{\mathsf{y}}_{u}^{+}$ 或 $\bm{\mathsf{y}}_{u}^{-}$ 的两类改写（前一实验用过的 $\bm{\mathsf{y}}_{\text{gpts}}^{+}$、$\bm{\mathsf{y}}_{\text{gptf}}^{+}$、$\bm{\mathsf{y}}_{\text{gpts}}^{-}$、$\bm{\mathsf{y}}_{\text{gptf}}^{-}$）。见图 4 第一幅三条曲线：两类改写以相近速度下降，快于 $\bm{\mathsf{y}}_{u}^{+}$ 的衰减——因为向上压力直接作用在 $\bm{\mathsf{y}}_{u}^{+}$ 而非这些改写上。类似地，第二幅中 $\bm{\mathsf{y}}_{u}^{-}$ 比其改写衰减更快，因为 $\mathcal{G}^{t}_{\text{DPO}}$ 直接对 $\bm{\mathsf{y}}_{u}^{-}$ 施加负压。第三幅中，$\bm{\mathsf{y}}_{u}^{+}$ 的改写始终比 $\bm{\mathsf{y}}_{u}^{-}$ 的改写衰减更慢，尽管训练中从未出现它们——因为这些回复在 $\mathcal{Y}$ 中靠近 $\bm{\mathsf{y}}_{u}^{+}$ 或 $\bm{\mathsf{y}}_{u}^{-}$，即 $\|\mathcal{K}^{t}\|_{F}$ 相对较大，故作用在 $\bm{\mathsf{y}}_{u}^{+}$、$\bm{\mathsf{y}}_{u}^{-}$ 上的压力也会对它们产生不可忽视的影响。最后，第四幅中间隔 $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid\bm{\mathsf{\chi}}_{u})-\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{-}\mid\bm{\mathsf{\chi}}_{u})$ 持续增大，说明随训练推进模型区分 $\bm{\mathsf{y}}_{u}^{+}$ 与 $\bm{\mathsf{y}}_{u}^{-}$ 的能力在增强。

Although $\mathcal{G}^{t}_{\text{DPO}}$ directly imposes a “pull-up” pressure on $\bm{\mathsf{y}}_{u}^{+}$, the value of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}|\bm{\mathsf{\chi}}_{u})$ does not increase a lot as it does in SFT. The downward arrow on $\bm{\mathsf{y}}_{u}^{-}$ indeed introduces a “push-down” pressure on responses that are similar to $\bm{\mathsf{y}}_{u}^{-}$, but the influence is unlikely to be that strong (it will be weakened by $\|\mathcal{K}^{t}\|_{F}$) to make the confidence on almost every observing responses decrease so fast, as demonstrated in the last panel of Figure 3 where we use similar $\eta$ for both SFT and DPO. Then, where has the probability mass gone during DPO? The key to answering this question is the squeezing effect discussed in Section 3.3: since the big negative gradient is imposed on $\bm{\mathsf{y}}_{u}^{-}$, which is at this point probably in a region of low $\pi_{\theta^{t}}(\bm{\mathsf{y}}|\bm{\mathsf{\chi}}_{u})$, the confidence of most $\bm{\mathsf{y}}$ will be decreased and $\pi_{\theta^{t}}(\bm{\mathsf{y}}^{*}|\bm{\mathsf{\chi}}_{u})$ will increase very fast.

尽管 $\mathcal{G}^{t}_{\text{DPO}}$ 直接对 $\bm{\mathsf{y}}_{u}^{+}$ 施加「拉高」压力，$\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid\bm{\mathsf{\chi}}_{u})$ 并未像 SFT 中那样大幅上升。$\bm{\mathsf{y}}_{u}^{-}$ 上的向下箭头确实会对与 $\bm{\mathsf{y}}_{u}^{-}$ 相似的回复引入「压低」压力，但该影响不太可能强到（会被 $\|\mathcal{K}^{t}\|_{F}$ 削弱）使几乎所有观测回复的置信度下降如此之快——见图 3 末幅，我们对 SFT 与 DPO 使用相近的 $\eta$。那么，DPO 期间概率质量去哪了？ 回答关键是第 3.3 节的挤压效应：大负梯度作用在此时很可能处于低 $\pi_{\theta^{t}}(\bm{\mathsf{y}}\mid\bm{\mathsf{\chi}}_{u})$ 区域的 $\bm{\mathsf{y}}_{u}^{-}$ 上，于是多数 $\bm{\mathsf{y}}$ 的置信度下降，而 $\pi_{\theta^{t}}(\bm{\mathsf{y}}^{*}\mid\bm{\mathsf{\chi}}_{u})$ 快速上升。

To verify this, we report the log-likelihood of $\bm{\mathsf{y}}$ chosen by greedy decoding: each token is chosen by maximizing the conditional probability given $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{<l}^{+}]$ in real-time, where $\bm{\mathsf{y}}_{<l}^{+}$ is a sub-sequence of $\bm{\mathsf{y}}_{u}^{+}$. As illustrated by the last panel of Figure 4, the confidence of this “teacher forcing” greedy $\bm{\mathsf{y}}$ increases very fast (from -113 to -63), which is even faster than the increase of $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}|\bm{\mathsf{\chi}}_{u})$ during SFT (from -130 to -90), within 8 epochs. However, the tokens with the highest confidence do not necessarily form a preferred response: it will reinforce the prior bias in $\theta^{0}$. This could be a reasonable explanation of the “degeneration” reported in recent work (e.g. Holtzman et al. 2020a): as $\pi_{\theta^{t}}$ becomes more peaky at its most confident predictions, it is easier to sample sequences with repeated phrases. Note that such behavior could also be understood as a special type of self-bias amplifying (Ren et al. 2024a), which would bring more serious consequences if it is combined with a multiple-generation self-improving algorithm, e.g., self-reward (Yuan et al. 2024a), iterative DPO (Xiong et al. 2024a), etc.

为验证这一点，我们报告贪心解码所选 $\bm{\mathsf{y}}$ 的对数似然：每个 token 在给定 $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{<l}^{+}]$ 时实时最大化条件概率，其中 $\bm{\mathsf{y}}_{<l}^{+}$ 是 $\bm{\mathsf{y}}_{u}^{+}$ 的子序列。如图 4 末幅，这种「teacher forcing」贪心 $\bm{\mathsf{y}}$ 的置信度上升极快（从 -113 到 -63），甚至快于 SFT 中 $\pi_{\theta^{t}}(\bm{\mathsf{y}}_{u}^{+}\mid\bm{\mathsf{\chi}}_{u})$ 的上升（从 -130 到 -90），且仅用 8 个 epoch。然而，置信度最高的 token 未必构成偏好回复：它会强化 $\theta^{0}$ 中的先验偏差。这可合理解释近期工作报告的「退化」（如 Holtzman et al. 2020a）：当 $\pi_{\theta^{t}}$ 在最自信预测上变得更尖峰时，更容易采样出含重复短语的序列。注意，此类行为也可理解为一种特殊的自我偏差放大（Ren et al. 2024a）；若再与多轮生成式自我改进算法结合，例如 self-reward（Yuan et al. 2024a）、迭代 DPO（Xiong et al. 2024a）等，后果会更严重。

In summary, the behaviors of different types of responses all match our analyses well. More subtle trends of different responses support our story well (both for SFT and DPO).Due to space constraints, we explain these (and the full results on other models and datasets) in Appendix D.

总之，各类回复的行为都与我们的分析吻合。不同回复更细微的趋势也支持上述故事（对 SFT 与 DPO 皆然）。篇幅所限，这些内容以及其它模型与数据集上的完整结果放在附录 D。

4.3 Mitigating the squeezing effect by augmenting the training set for SFT

4.3 通过扩充 SFT 训练集缓解挤压效应

Since the “squeezing effect” caused by the big negative gradient on unlikely predictions can damage the model’s performance during DPO, we can first train the model on both $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+}]$ and $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{-}]$ during the SFT stage (making the negative response more likely), and then run the usual DPO. Following the analysis above, we can expect during this new SFT stage, the region of those responses similar to $\bm{\mathsf{y}}_{u}^{+}$ or $\bm{\mathsf{y}}_{u}^{-}$ will be “pulled up” simultaneously. This is what we want because in many cases, both $\bm{\mathsf{y}}_{u}^{+}$ and $\bm{\mathsf{y}}_{u}^{-}$ are reasonably good responses for the question $\bm{\mathsf{x}}_{u}$; the new SFT design hence helps to pull up a larger region that contains more suitable responses compared with the baseline SFT. After that, the “push-down” pressure imposed during DPO can efficiently decrease the model’s confidence on $\bm{\mathsf{y}}_{u}^{-}$ and its similar responses. Since $\bm{\mathsf{y}}_{u}^{-}$ is no longer so unlikely before DPO, the squeezing effect should not be as strong as in the baseline procedure.

由于对不太可能预测施加大负梯度引起的「挤压效应」会损害 DPO 中的模型表现，我们可以先在 SFT 阶段同时在 $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{+}]$ 与 $[\bm{\mathsf{x}}_{u};\bm{\mathsf{y}}_{u}^{-}]$ 上训练（使负回复更可能），再按常規跑 DPO。按上文分析，可预期在这个新的 SFT 阶段，与 $\bm{\mathsf{y}}_{u}^{+}$ 或 $\bm{\mathsf{y}}_{u}^{-}$ 相似的回复区域会被同时「拉高」。这正是我们想要的：许多情形下 $\bm{\mathsf{y}}_{u}^{+}$ 与 $\bm{\mathsf{y}}_{u}^{-}$ 都是对问题 $\bm{\mathsf{x}}_{u}$ 相当不错的回复；新的 SFT 设计因此有助于拉高比基线 SFT 更大、含更多合适回复的区域。之后，DPO 中施加的「压低」压力可有效降低模型对 $\bm{\mathsf{y}}_{u}^{-}$ 及其相似回复的置信度。由于 DPO 前 $\bm{\mathsf{y}}_{u}^{-}$ 不再那么不可能，挤压效应应不如基线流程严重。

DPO Ep.	ChatGPT	Claude
0	0.4729	0.4679
2	0.6518	0.5151
4	0.6928	0.6045
6	0.6667	0.5432

Table 1: Win-rate against baseline.

DPO Ep.	ChatGPT	Claude
0	0.4729	0.4679
2	0.6518	0.5151
4	0.6928	0.6045
6	0.6667	0.5432

表 1：相对基线的胜率。

We call our training pipeline “extend” and compare its learning dynamics with the baseline setting in Figure 5. It is clear that the squeezing effect is mitigated, because the confidence of other responses all decays slower during DPO, and we also observe a big drop in the greedy-decoding response when DPO starts. To further show that mitigating the squeezing effect indeed brings benefits, we compare the responses generated by models trained using different methods by feeding them to ChatGPT and Claude3. Specifically, we first SFT the model for two epochs using two methods discussed above and call the resulting policy network $\pi_{\text{base}}$ and $\pi_{\text{extend}}$. Then, we conduct identical DPO training on both $\pi_{\text{base}}$ and $\pi_{\text{extend}}$ for several epochs. The win rate of the proposed method against the baseline one is provided in Table 1. It is clear that before DPO, $\pi_{\text{base}}$ is better, because $\pi_{\text{extend}}$ is explicitly trained on those $\bm{\mathsf{y}}^{-}$. However, the $\pi_{\text{extend}}$ performs better after DPO several epochs since the squeezing effect is efficiently mitigated. Please refer to Appendix F for more details. In the future, this simple method inspired by our analysis could be further improved by introducing more responses, e.g., rephrases of $\bm{\mathsf{y}}_{u}^{+}$, etc., during both stages, and also by combining with many existing RL-free methods we mentioned before.

我们将该训练流程称为「extend」，并在图 5 中与基线设定比较学习动力学。显然挤压效应得到缓解：其它回复在 DPO 中衰减更慢，且 DPO 开始时贪心解码回复出现明显下降。为进一步表明缓解挤压确实带来收益，我们将不同方法训得模型生成的回复送入 ChatGPT 与 Claude3 比较。具体地，先用上述两种方法各 SFT 两个 epoch，得到策略网络 $\pi_{\text{base}}$ 与 $\pi_{\text{extend}}$；再对二者做相同的若干 epoch DPO。所提方法相对基线的胜率见表 1。显然，DPO 前 $\pi_{\text{base}}$ 更好，因为 $\pi_{\text{extend}}$ 显式在那些 $\bm{\mathsf{y}}^{-}$ 上训练过；但 DPO 若干 epoch 后 $\pi_{\text{extend}}$ 表现更好，因挤压效应被有效缓解。更多细节见附录 F。未来，这一由分析启发的简单方法还可通过在两阶段引入更多回复（如 $\bm{\mathsf{y}}_{u}^{+}$ 的改写等），并与前述多种免 RL 方法结合而进一步改进。

Figure 5: Learning dynamics of the baseline and the proposed method with training data extension. Key trends to observe: 1.) Baseline and the extend method have similar behavior on $\bm{\mathsf{y}}^{+}_{u}$ during SFT; 2.) The extend method considerably increase $\bm{\mathsf{y}}^{-}_{u}$ during SFT; 3.) The squeezing effect of the extend method is weaker (all other responses decay slower and the confidence on the “greedy-decoding” response increases slower).

图 5：基线与所提训练数据扩充方法的学习动力学。需关注的关键趋势：1）基线与 extend 在 SFT 阶段对 $\bm{\mathsf{y}}^{+}_{u}$ 行为相似；2）extend 在 SFT 中显著提高 $\bm{\mathsf{y}}^{-}_{u}$；3）extend 的挤压效应更弱（其它回复衰减更慢，「贪心解码」回复置信度上升更慢）。

5 Conclusion

5 结论

Learning dynamics, which depict how the model’s prediction changes when it learns new examples, provide a powerful tool to analyze the behavior of models trained with gradient descent. To better utilize this tool in the context of LLM finetuning, we first derive the step-wise decomposition of LLM finetuning for various common algorithms. Then, we propose a unified framework for understanding LLM predictions’ behaviors across different finetuning methods. The proposed analysis successfully explains various phenomena during LLM’s instruction tuning and preference tuning, some of them are quite counter-intuitive. We also shed light on how specific hallucinations are introduced in the SFT stage, as previously observed (Gekhman et al. 2024a), and where the improvements of some new RL-free algorithms come from compared with the vanilla off-policy DPO. The analysis of the squeezing effect also has the potential to be applied to other deep learning systems which apply big negative gradients to already-unlikely outcomes. Finally, inspired by this analysis, we propose a simple (but counter-intuitive) method that is effective in improving the alignment of models.

学习动力学刻画模型在学习新样本时预测如何变化，是分析梯度下降训练模型行为的有力工具。为在 LLM 微调场景更好利用该工具，我们首先导出多种常见算法的逐步分解，再提出统一框架，理解不同微调方法下 LLM 预测行为。所提分析成功解释了指令微调与偏好微调中的多种现象，其中一些相当反直觉。我们还阐明了 SFT 阶段如何引入特定类型幻觉（此前由 Gekhman et al. 2024a 观察到），以及一些新的免 RL 算法相对朴素 off-policy DPO 的改进从何而来。对挤压效应的分析也有望推广到其它对已不太可能结果施加大负梯度的深度学习系统。最后，受该分析启发，我们提出一种简单（但反直觉）且能有效提升模型对齐的方法。

Acknowledgements

致谢

This research was enabled in part by support provided by the Canada CIFAR AI Chairs program, WestGrid, and Compute Canada. We thank Shangmin Guo, Noam Razin, Wonho Bae, and Hamed Shirzad for their valuable discussions and feedback. We also appreciate the constructive comments from the anonymous reviewers, which helped improve this work.

本研究部分得益于 Canada CIFAR AI Chairs 项目、WestGrid 与 Compute Canada 的支持。感谢 Shangmin Guo、Noam Razin、Wonho Bae 与 Hamed Shirzad 的宝贵讨论与反馈。亦感谢匿名审稿人的建设性意见，帮助改进本文。

References

参考文献

Arora et al. (2019) Sanjeev Arora, Simon. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov and Ruosong Wang “On Exact Computation with an Infinitely Wide Neural Net” In NeurIPS , 2019 arXiv: 1904.11955

Azar et al. (2024) Mohammad Azar, Zhaohan Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko and Daniele Calandriello “A general theoretical paradigm to understand learning from human preferences” In International Conference on Artificial Intelligence and Statistics , 2024, pp. 4447–4455 PMLR

Bai et al. (2023) Jinze Bai, Shuai Bai and et.al “Qwen Technical Report”, 2023 arXiv: 2309.16609

Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli and Tom Henighan “Training a helpful and harmless assistant with reinforcement learning from human feedback”, 2022 arXiv: 2204.05862

Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Khan, Shivanshu Purohit, USVSN Prashanth and Edward Raff “Pythia: A suite for analyzing large language models across training and scaling” In ICML , 2023, pp. 2397–2430 PMLR

Caccia et al. (2020) Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau and Laurent Charlin “Language GANs Falling Short” In International Conference on Learning Representations , 2020

Chen et al. (2024) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji and Quanquan Gu “Self-play fine-tuning converts weak language models to strong language models”, 2024 arXiv: 2401.01335

Choshen et al. (2020) Leshem Choshen, Lior Fox, Zohar Aizenbud and Omri Abend “On the Weaknesses of Reinforcement Learning for Neural Machine Translation” In International Conference on Learning Representations , 2020

Christiano et al. (2017) Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei “Deep reinforcement learning from human preferences” In NeurIPS , 2017

Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu and Maosong Sun “UltraFeedback: Boosting Language Models with High-quality Feedback”, 2023 arXiv: 2310.01377 [cs.CL]

Feldman (2020) Dan Feldman “Introduction to core-sets: an updated survey”, 2020 arXiv: 2011.09384

Gekhman et al. (2024) Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart and Jonathan Herzig “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?”, 2024 arXiv: 2405.05904

Guo et al. (2025) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang and Xiao Bi “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning” In arXiv preprint arXiv:2501.12948 , 2025

Guo et al. (2024) Shangmin Guo, Yi Ren, Stefano Albrecht and Kenny Smith “lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning” In ICLR , 2024

Guo et al. (2024a) Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao and Bilal Piot “Direct language model alignment from online AI feedback”, 2024 arXiv: 2402.04792

Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals and Jeff Dean “Distilling the knowledge in a neural network”, 2015 arXiv: 1503.02531

Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi “The Curious Case of Neural Text Degeneration” In ICLR , 2020

Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng and Bing Qin “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions”, 2023 arXiv: 2311.05232

Jacot et al. (2018) Arthur Jacot, Franck Gabriel and Clément Hongler “Neural tangent kernel: Convergence and generalization in neural networks” In NeurIPS , 2018

Ji et al. (2024) Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang and Minlie Huang “Towards Efficient and Exact Optimization of Language Model Alignment”, 2024 arXiv: 2402.00856

Kumar et al. (2022) Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma and Percy Liang “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution” In ICLR , 2022

LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner “Gradient-based learning applied to document recognition” In Proceedings of the IEEE 86.11 Ieee, 1998, pp. 2278–2324

Liu et al. (2020) Sheng Liu, Jonathan Niles-Weed, Narges Razavian and Carlos Fernandez-Granda “Early-learning regularization prevents memorization of noisy labels” In NeurIPS , 2020

Mohamadi et al. (2022) Mohamad Mohamadi, Wonho Bae and Danica. Sutherland “Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels” In NeurIPS , 2022 arXiv: 2206.12569

Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama and Alex Ray “Training language models to follow instructions with human feedback” In NeurIPS , 2022, pp. 27730–27744

Pal et al. (2024) Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu and Colin White “Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive”, 2024 arXiv: 2402.13228

Park et al. (2024) Core Park, Maya Okawa, Andrew Lee, Ekdeep Lubana and Hidenori Tanaka “Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space” In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

Pruthi et al. (2020) Garima Pruthi, Frederick Liu, Satyen Kale and Mukund Sundararajan “Estimating training data influence by tracing gradient descent” In NeurIPS , 2020

Rafailov et al. (2024) Rafael Rafailov, Joey Hejna, Ryan Park and Chelsea Finn “From r r to Q ∗ Q^{*} : Your Language Model is Secretly a Q-Function”, 2024 arXiv: 2404.12358

Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon and Chelsea Finn “Direct preference optimization: Your language model is secretly a reward model” In NeurIPS , 2023

Razin et al. (2025) Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora and Boris Hanin “Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization” In The Thirteenth International Conference on Learning Representations , 2025

Ren et al. (2023) Yi Ren, Shangmin Guo, Wonho Bae and Danica. Sutherland “How to prepare your task head for finetuning” In ICLR , 2023

Ren et al. (2020) Yi Ren, Shangmin Guo, Matthieu Labeau, Shay. Cohen and Simon Kirby “Compositional languages emerge in a neural iterated learning model” In ICLR , 2020

Ren et al. (2024) Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang and Danica Sutherland “Bias Amplification in Language Model Evolution: An Iterated Learning Perspective” In NeurIPS , 2024

Ren et al. (2022) Yi Ren, Shangmin Guo and Danica. Sutherland “Better Supervisory Signals by Observing Learning Paths” In ICLR , 2022

Ren et al. (2023a) Yi Ren, Samuel Lavoie, Michael Galkin, Danica Sutherland and Aaron Courville “Improving compositional generalization using iterated learning and simplicial embeddings” In NeurIPS , 2023

Ren & Sutherland (2024) Yi Ren and Danica. Sutherland “Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics” In NeurIPS 2024 Workshop on Compositional Learning: Perspectives, Methods, and Paths Forward , 2024

Schaeffer et al. (2024) Rylan Schaeffer, Brando Miranda and Sanmi Koyejo “Are emergent abilities of large language models a mirage?” In Advances in Neural Information Processing Systems 36 , 2024

Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov “Proximal policy optimization algorithms” In arXiv preprint arXiv:1707.06347 , 2017

Settles (2009) Burr Settles “Active learning literature survey” University of Wisconsin-Madison Department of Computer Sciences, 2009

Tajwar et al. (2024) Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn and Aviral Kumar “Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data”, 2024 arXiv: 2404.14367 [cs.LG]

Tang et al. (2024) Yunhao Tang, Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, BernardoÁvila Pires, Michal Valko and Yong Cheng “Understanding the performance gap between online and offline alignment algorithms” In arXiv preprint arXiv:2405.08448 , 2024

Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan. Gomez, Lukasz Kaiser and Illia Polosukhin “Attention Is All You Need” In NeurIPS , 2017 arXiv: 1706.03762

Wang et al. (2018) Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba and Alexei Efros “Dataset distillation”, 2018 arXiv: 1811.10959

Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Yu, Brian Lester, Nan Du, Andrew. Dai and Quoc Le “Finetuned Language Models are Zero-Shot Learners” In ICLR , 2022

Wu et al. (2024) Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang and Quanquan Gu “Self-play preference optimization for language model alignment”, 2024 arXiv: 2405.00675

Xia et al. (2024) Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora and Danqi Chen “Less: Selecting influential data for targeted instruction tuning”, 2024 arXiv: 2402.04333

Xiong et al. (2024) Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang and Tong Zhang “Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint” In ICML , 2024

Yang et al. (2024) Zhaorui Yang, Qian Liu, Tianyu Pang, Han Wang, Haozhe Feng, Minfeng Zhu and Wei Chen “Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning”, 2024 arXiv: 2402.13669

Yuan et al. (2024) Weizhe Yuan, Richard Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu and Jason Weston “Self-rewarding language models”, 2024 arXiv: 2401.10020

Zhang et al. (2024) Ruiqi Zhang, Licong Lin, Yu Bai and Song Mei “Negative preference optimization: From catastrophic collapse to effective unlearning” In arXiv preprint arXiv:2404.05868 , 2024

Zhang et al. (2023) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang and Yulong Chen “Siren’s song in the AI ocean: a survey on hallucination in large language models” In arXiv preprint arXiv:2309.01219 , 2023

Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh and Peter Liu “Slic-hf: Sequence likelihood calibration with human feedback”, 2023 arXiv: 2305.10425

References

参考文献

LeCun et al. (1998a) Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner “Gradient-based learning applied to document recognition” In Proceedings of the IEEE 86.11 Ieee, 1998, pp. 2278–2324

Settles (2009a) Burr Settles “Active learning literature survey” University of Wisconsin-Madison Department of Computer Sciences, 2009

Hinton et al. (2015a) Geoffrey Hinton, Oriol Vinyals and Jeff Dean “Distilling the knowledge in a neural network”, 2015 arXiv: 1503.02531

Christiano et al. (2017a) Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei “Deep reinforcement learning from human preferences” In NeurIPS , 2017

Schulman et al. (2017a) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov “Proximal policy optimization algorithms” In arXiv preprint arXiv:1707.06347 , 2017

Vaswani et al. (2017a) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan. Gomez, Lukasz Kaiser and Illia Polosukhin “Attention Is All You Need” In NeurIPS , 2017 arXiv: 1706.03762

Jacot et al. (2018a) Arthur Jacot, Franck Gabriel and Clément Hongler “Neural tangent kernel: Convergence and generalization in neural networks” In NeurIPS , 2018

Wang et al. (2018a) Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba and Alexei Efros “Dataset distillation”, 2018 arXiv: 1811.10959

Arora et al. (2019a) Sanjeev Arora, Simon. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov and Ruosong Wang “On Exact Computation with an Infinitely Wide Neural Net” In NeurIPS , 2019 arXiv: 1904.11955

Caccia et al. (2020a) Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau and Laurent Charlin “Language GANs Falling Short” In International Conference on Learning Representations , 2020

Choshen et al. (2020a) Leshem Choshen, Lior Fox, Zohar Aizenbud and Omri Abend “On the Weaknesses of Reinforcement Learning for Neural Machine Translation” In International Conference on Learning Representations , 2020

Feldman (2020a) Dan Feldman “Introduction to core-sets: an updated survey”, 2020 arXiv: 2011.09384

Holtzman et al. (2020a) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi “The Curious Case of Neural Text Degeneration” In ICLR , 2020

Liu et al. (2020a) Sheng Liu, Jonathan Niles-Weed, Narges Razavian and Carlos Fernandez-Granda “Early-learning regularization prevents memorization of noisy labels” In NeurIPS , 2020

Pruthi et al. (2020a) Garima Pruthi, Frederick Liu, Satyen Kale and Mukund Sundararajan “Estimating training data influence by tracing gradient descent” In NeurIPS , 2020

Ren et al. (2020a) Yi Ren, Shangmin Guo, Matthieu Labeau, Shay. Cohen and Simon Kirby “Compositional languages emerge in a neural iterated learning model” In ICLR , 2020

Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli and Tom Henighan “Training a helpful and harmless assistant with reinforcement learning from human feedback”, 2022 arXiv: 2204.05862

Kumar et al. (2022a) Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma and Percy Liang “Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution” In ICLR , 2022

Mohamadi et al. (2022a) Mohamad Mohamadi, Wonho Bae and Danica. Sutherland “Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels” In NeurIPS , 2022 arXiv: 2206.12569

Ouyang et al. (2022a) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama and Alex Ray “Training language models to follow instructions with human feedback” In NeurIPS , 2022, pp. 27730–27744

Ren et al. (2022a) Yi Ren, Shangmin Guo and Danica. Sutherland “Better Supervisory Signals by Observing Learning Paths” In ICLR , 2022

Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Yu, Brian Lester, Nan Du, Andrew. Dai and Quoc Le “Finetuned Language Models are Zero-Shot Learners” In ICLR , 2022

Bai et al. (2023a) Jinze Bai, Shuai Bai and et.al “Qwen Technical Report”, 2023 arXiv: 2309.16609

Biderman et al. (2023a) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Khan, Shivanshu Purohit, USVSN Prashanth and Edward Raff “Pythia: A suite for analyzing large language models across training and scaling” In ICML , 2023, pp. 2397–2430 PMLR

Cui et al. (2023a) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu and Maosong Sun “UltraFeedback: Boosting Language Models with High-quality Feedback”, 2023 arXiv: 2310.01377 [cs.CL]

Huang et al. (2023a) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng and Bing Qin “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions”, 2023 arXiv: 2311.05232

Rafailov et al. (2023a) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon and Chelsea Finn “Direct preference optimization: Your language model is secretly a reward model” In NeurIPS , 2023

Ren et al. (2023b) Yi Ren, Shangmin Guo, Wonho Bae and Danica. Sutherland “How to prepare your task head for finetuning” In ICLR , 2023

Ren et al. (2023c) Yi Ren, Samuel Lavoie, Michael Galkin, Danica Sutherland and Aaron Courville “Improving compositional generalization using iterated learning and simplicial embeddings” In NeurIPS , 2023

Zhang et al. (2023a) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang and Yulong Chen “Siren’s song in the AI ocean: a survey on hallucination in large language models” In arXiv preprint arXiv:2309.01219 , 2023

Zhao et al. (2023a) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh and Peter Liu “Slic-hf: Sequence likelihood calibration with human feedback”, 2023 arXiv: 2305.10425

Azar et al. (2024a) Mohammad Azar, Zhaohan Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko and Daniele Calandriello “A general theoretical paradigm to understand learning from human preferences” In International Conference on Artificial Intelligence and Statistics , 2024, pp. 4447–4455 PMLR

Chen et al. (2024a) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji and Quanquan Gu “Self-play fine-tuning converts weak language models to strong language models”, 2024 arXiv: 2401.01335

Gekhman et al. (2024a) Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart and Jonathan Herzig “Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?”, 2024 arXiv: 2405.05904

Guo et al. (2024b) Shangmin Guo, Yi Ren, Stefano Albrecht and Kenny Smith “lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning” In ICLR , 2024

Guo et al. (2024c) Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao and Bilal Piot “Direct language model alignment from online AI feedback”, 2024 arXiv: 2402.04792

Ji et al. (2024a) Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang and Minlie Huang “Towards Efficient and Exact Optimization of Language Model Alignment”, 2024 arXiv: 2402.00856

Pal et al. (2024a) Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu and Colin White “Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive”, 2024 arXiv: 2402.13228

Park et al. (2024a) Core Park, Maya Okawa, Andrew Lee, Ekdeep Lubana and Hidenori Tanaka “Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space” In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

Rafailov et al. (2024a) Rafael Rafailov, Joey Hejna, Ryan Park and Chelsea Finn “From r r to Q ∗ Q^{*} : Your Language Model is Secretly a Q-Function”, 2024 arXiv: 2404.12358

Ren et al. (2024a) Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang and Danica Sutherland “Bias Amplification in Language Model Evolution: An Iterated Learning Perspective” In NeurIPS , 2024

Ren & Sutherland (2024a) Yi Ren and Danica. Sutherland “Understanding Simplicity Bias towards Compositional Mappings via Learning Dynamics” In NeurIPS 2024 Workshop on Compositional Learning: Perspectives, Methods, and Paths Forward , 2024

Schaeffer et al. (2024a) Rylan Schaeffer, Brando Miranda and Sanmi Koyejo “Are emergent abilities of large language models a mirage?” In Advances in Neural Information Processing Systems 36 , 2024

Tajwar et al. (2024a) Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn and Aviral Kumar “Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data”, 2024 arXiv: 2404.14367 [cs.LG]

Tang et al. (2024a) Yunhao Tang, Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, BernardoÁvila Pires, Michal Valko and Yong Cheng “Understanding the performance gap between online and offline alignment algorithms” In arXiv preprint arXiv:2405.08448 , 2024

Wu et al. (2024a) Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang and Quanquan Gu “Self-play preference optimization for language model alignment”, 2024 arXiv: 2405.00675

Xia et al. (2024a) Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora and Danqi Chen “Less: Selecting influential data for targeted instruction tuning”, 2024 arXiv: 2402.04333

Xiong et al. (2024a) Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang and Tong Zhang “Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint” In ICML , 2024

Yang et al. (2024a) Zhaorui Yang, Qian Liu, Tianyu Pang, Han Wang, Haozhe Feng, Minfeng Zhu and Wei Chen “Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning”, 2024 arXiv: 2402.13669

Yuan et al. (2024a) Weizhe Yuan, Richard Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu and Jason Weston “Self-rewarding language models”, 2024 arXiv: 2401.10020

Zhang et al. (2024a) Ruiqi Zhang, Licong Lin, Yu Bai and Song Mei “Negative preference optimization: From catastrophic collapse to effective unlearning” In arXiv preprint arXiv:2404.05868 , 2024

Guo et al. (2025a) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang and Xiao Bi “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning” In arXiv preprint arXiv:2501.12948 , 2025

Razin et al. (2025a) Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora and Boris Hanin “Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization” In The Thirteenth International Conference on Learning Representations , 2025

Appendix A More Related Works

附录 A 更多相关工作

A.1 More about learning dynamics

A.1 关于学习动力学的更多讨论

Instead of emphasizing the model’s convergence status and guarantees, learning dynamics focus more on its relative behavior during training. Since many LLMs exhibit “emergent ability” during training in terms of the amount of training data or inference time (Schaeffer et al. 2024a), like the “aha moment” mentioned in Guo et al. 2025a, we hence believe the analysis of learning dynamics can provide a novel perspective on understanding the LLM system.

学习动力学不强调模型的收敛状态与保证，而更关注训练过程中的相对行为。由于许多 LLM 在训练数据量或推理时间维度上会表现出「涌现能力」（Schaeffer et al. 2024a），例如 Guo et al. 2025a 提到的「aha moment」，我们相信学习动力学分析可为理解 LLM 系统提供新视角。

Beyond their application to LLMs, learning dynamics are widely utilized in analyzing various machine learning problems. For example, if we consider {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} from the training set, and {\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} from the test set, this form of learning dynamics provides a new perspective on generalization: the model generalizes better if the loss of f_{\theta}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}}) keeps decreasing when it learns from {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}}. By studying the influence of different {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} at different stages during supervised learning, Ren et al. 2022a explain a “zigzag” pattern of the learning path, which sheds light on why the model can spontaneously pursue better supervisory signals and correct noisy labels in the early stage of training (see also Liu et al. 2020a). Kumar et al. 2022a; Ren et al. 2023b apply learning dynamics to explain why directly finetuning a well-trained backbone with a randomly initialized task head might harm the out-of-distribution generalization ability. Ren et al. 2020a; Ren et al. 2023c; Ren & Sutherland 2024a also explains where the simplicity bias favoring compositional representations comes from during knowledge distillation (Hinton et al. 2015a), providing a new perspective of understanding why successive knowledge transferring can improve the model’s systematic generalization ability.

除应用于 LLM 外，学习动力学也广泛用于分析各类机器学习问题。例如，若 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 来自训练集、{\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}} 来自测试集，这种形式的学习动力学为泛化提供新视角：当模型从 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 学习时，若 f_{\theta}({\color[rgb]{1,0.5,0}\bm{\mathsf{x}}_{o}}) 的损失持续下降，则泛化更好。通过研究监督学习不同阶段不同 {\color[rgb]{0,1,1}\bm{\mathsf{x}}_{u}} 的影响，Ren et al. 2022a 解释了学习路径的「之」字形图案，从而说明为何模型能自发追求更好的监督信号并在训练早期纠正噪声标签（亦见 Liu et al. 2020a）。Kumar et al. 2022a、Ren et al. 2023b 用学习动力学解释：为何用随机初始化的任务头直接微调训好的骨干可能损害分布外泛化。Ren et al. 2020a、Ren et al. 2023c、Ren & Sutherland 2024a 亦解释了知识蒸馏（Hinton et al. 2015a）中偏好组合表示的简单性偏置从何而来，为理解连续知识迁移为何能提升系统化泛化能力提供新视角。

Besides explaining the model’s behavior, learning dynamics is also helpful for evaluating the quality or the effectiveness of different training samples. For example, Pruthi et al. 2020a propose a quantitative metric called TracIn to compute the influence of a training example on the predictions made by the model. This metric is then applied by Xia et al. 2024a to search for the most influential examples in LLM instruction finetuning. By expanding Equation 1 in the neural tangent kernel (NTK) regime, Guo et al. 2024b propose a metric called lpNTK to measure the relative difficulty among different training samples. These metrics and analyses inspired by learning dynamics are expected to be helpful in many related fields, like coreset selection (Feldman 2020a), active learning (Settles 2009a) (see, e.g., Mohamadi et al. 2022a), and dataset distillation (Wang et al. 2018a).

除解释模型行为外，学习动力学也有助于评估不同训练样本的质量或有效性。例如，Pruthi et al. 2020a 提出定量指标 TracIn，计算训练样本对模型预测的影响；Xia et al. 2024a 再用该指标在 LLM 指令微调中搜索最具影响力的样本。通过在神经正切核（NTK）机制下展开式 1，Guo et al. 2024b 提出 lpNTK 指标，度量不同训练样本的相对难度。这些由学习动力学启发的指标与分析有望惠及许多相关领域，如核心集选择（Feldman 2020a）、主动学习（Settles 2009a；亦见 Mohamadi et al. 2022a）以及数据集蒸馏（Wang et al. 2018a）。

A.2 More about LLM’s finetuning

A.2 关于 LLM 微调的更多讨论

In this paper, we broadly define finetuning as any in-weight learning on top of a pretrained base model, including supervised finetuning (SFT), direct policy optimization (DPO, Rafailov et al. 2023a) and its variants, etc. Since the analysis throughout this paper relies on the “teacher forcing” mechanism and the relatively stable eNTK assumption, our framework cannot be directly applied to algorithms with token-wise supervision like reinforcement learning with human feedback (RLHF, Ouyang et al. 2022a) and proximal policy optimization (PPO, Schulman et al. 2017a)[^1]. We leave the study of the token-wise learning dynamics, which aligns better with the “squeezing effect” in real settings, to future work.

本文广义地将微调定义为在预训练基座之上的任何权重内学习，包括监督微调（SFT）、直接偏好优化（DPO，Rafailov et al. 2023a）及其变体等。由于全文分析依赖「teacher forcing」机制与相对稳定的 eNTK 假设，我们的框架不能直接应用于带 token 级监督的算法，如基于人类反馈的强化学习（RLHF，Ouyang et al. 2022a）与近端策略优化（PPO，Schulman et al. 2017a）[^1]。与真实设定中「挤压效应」更吻合的 token 级学习动力学，留待未来工作。

We also identify several related works that report similar observations on the phenomena discussed in this paper. For example, Zhang et al. 2023a; Gekhman et al. 2024a mentioned that learning new facts during SFT tends to make the model hallucinate more, which aligns with our finding that the model tends to use $\bm{\mathsf{y}}_{j\neq i}^{+}$ when answering question $i$. Holtzman et al. 2020a related the peakiness of the model’s distribution to LLM’s “repeater phenomena”, which also indirectly supports our claims well: more DPO leads to a more serious squeezing effect, hence the model’s prediction becomes peakier on most tokens, which makes the aforementioned phenomena more common.