Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov   2 2 footnotemark: 2 † † thanks: Equal contribution; more junior authors listed earlier.    Archit Sharma 1 1 footnotemark: 1    2 2 footnotemark: 2    Eric Mitchell 1 1 footnotemark: 1    2 2 footnotemark: 2    Stefano Ermon 2 2 footnotemark: 2    3 3 footnotemark: 3    Christopher D. Manning 2 2 footnotemark: 2    Chelsea Finn 2 2 footnotemark: 2    2 2 footnotemark: 2   Stanford University 3 3 footnotemark: 3   CZ Biohub Email:  {rafailov,architsh,eric.mitchell}@cs.stanford.edu

Abstract

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

大规模无监督语言模型（LM）能习得广博世界知识与一定推理技能，但其训练完全无监督，精确控制行为仍然困难。现有获得可引导性的方法会收集模型生成相对质量的人类标注，再微调无监督 LM 以对齐这些偏好，常用手段是基于人类反馈的强化学习（RLHF）。然而 RLHF 复杂且常不稳定：先拟合反映人类偏好的奖励模型，再用强化学习微调大型无监督 LM，在尽量不偏离原模型的前提下最大化该估计奖励。本文为 RLHF 中的奖励模型引入新参数化，使对应最优策略可闭式提取，从而仅用简单分类损失求解标准 RLHF 问题。所得算法称为直接偏好优化（Direct Preference Optimization，DPO）：稳定、有效且计算轻量，微调时无需从 LM 采样，也几乎不必大幅调超参。实验表明，DPO 可将 LM 微调到与现有方法相当或更好的人类偏好对齐水平。尤其在控制生成情感方面，DPO 超过基于 PPO 的 RLHF；在摘要与单轮对话的回复质量上匹配或更优，同时实现与训练都简单得多。

1 Introduction

Large unsupervised language models (LMs) trained on very large datasets acquire surprising capabilities [11, 7, 42, 8]. However, these models are trained on data generated by humans with a wide variety of goals, priorities, and skillsets. Some of these goals and skillsets may not be desirable to imitate; for example, while we may want our AI coding assistant to understand common programming mistakes in order to correct them, nevertheless, when generating code, we would like to bias our model toward the (potentially rare) high-quality coding ability present in its training data. Similarly, we might want our language model to be aware of a common misconception believed by 50% of people, but we certainly do not want the model to claim this misconception to be true in 50% of queries about it! In other words, selecting the model’s desired responses and behavior from its very wide knowledge and abilities is crucial to building AI systems that are safe, performant, and controllable [28]. While existing methods typically steer LMs to match human preferences using reinforcement learning (RL), we will show that the RL-based objective used by existing methods can be optimized exactly with a simple binary cross-entropy objective, greatly simplifying the preference learning pipeline.

在超大数据集上训练的大规模无监督语言模型（LM）会获得令人惊讶的能力 [11, 7, 42, 8]。然而这些模型的训练数据来自目标、优先级与技能水平差异极大的人类。其中一些目标与技能未必值得模仿：例如我们希望 AI 编程助手理解常见编程错误以便纠正，但在生成代码时，仍希望模型偏向训练数据中（可能稀少的）高质量编程能力。类似地，我们可能希望语言模型知道有半数人相信的常见误解，却绝不希望它在半数相关查询中把该误解说成事实。换言之，从模型极宽的知识与能力中选出期望的回复与行为，对构建安全、有效、可控的 AI 系统至关重要 [28]。现有方法通常用强化学习（RL）引导 LM 匹配人类偏好；我们将证明，现有方法所用的基于 RL 的目标，可用简单的二元交叉熵目标精确优化，从而大幅简化偏好学习流程。

Figure 1: DPO optimizes for human preferences while avoiding reinforcement learning. Existing methods for fine-tuning language models with human feedback first fit a reward model to a dataset of prompts and human preferences over pairs of responses, and then use RL to find a policy that maximizes the learned reward. In contrast, DPO directly optimizes for the policy best satisfying the preferences with a simple classification objective, fitting an implicit reward model whose corresponding optimal policy can be extracted in closed form.

图 1：DPO 在优化人类偏好的同时避开强化学习。用人类反馈微调语言模型的现有方法，先在提示与成对回复偏好数据集上拟合奖励模型，再用 RL 寻找最大化所学奖励的策略。相对地，DPO 用简单分类目标直接优化最能满足偏好的策略，拟合一个隐式奖励模型，其对应最优策略可闭式提取。

At a high level, existing methods instill the desired behaviors into a language model using curated sets of human preferences representing the types of behaviors that humans find safe and helpful. This preference learning stage occurs after an initial stage of large-scale unsupervised pre-training on a large text dataset. While the most straightforward approach to preference learning is supervised fine-tuning on human demonstrations of high quality responses, the most successful class of methods is reinforcement learning from human (or AI) feedback (RLHF/RLAIF; [12, 2]). RLHF methods fit a reward model to a dataset of human preferences and then use RL to optimize a language model policy to produce responses assigned high reward without drifting excessively far from the original model. While RLHF produces models with impressive conversational and coding abilities, the RLHF pipeline is considerably more complex than supervised learning, involving training multiple LMs and sampling from the LM policy in the loop of training, incurring significant computational costs.

宏观上，现有方法用策展过的人类偏好集把期望行为注入语言模型，这些偏好代表人类认为安全、有帮助的行为类型。该偏好学习阶段发生在大规模无监督预训练之后。偏好学习最直接的做法是对高质量回复的人类演示做监督微调，但最成功的一类方法是基于人类（或 AI）反馈的强化学习（RLHF/RLAIF；[12, 2]）。RLHF 先在人类偏好数据上拟合奖励模型，再用 RL 优化语言模型策略，使其生成高奖励回复且不过度偏离原模型。尽管 RLHF 能产出对话与编程能力出色的模型，其管线远比监督学习复杂：需训练多个 LM，并在训练环中从 LM 策略采样，计算代价显著。

In this paper, we show how to directly optimize a language model to adhere to human preferences, without explicit reward modeling or reinforcement learning. We propose Direct Preference Optimization (DPO), an algorithm that implicitly optimizes the same objective as existing RLHF algorithms (reward maximization with a KL-divergence constraint) but is simple to implement and straightforward to train. Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective. Like existing algorithms, DPO relies on a theoretical preference model (such as the Bradley-Terry model; [5]) that measures how well a given reward function aligns with empirical preference data. However, while existing methods use the preference model to define a preference loss to train a reward model and then train a policy that optimizes the learned reward model, DPO uses a change of variables to define the preference loss as a function of the policy directly. Given a dataset of human preferences over model responses, DPO can therefore optimize a policy using a simple binary cross entropy objective, producing the optimal policy to an implicit reward function fit to the preference data.

本文展示如何在不做显式奖励建模、也不做强化学习的情况下，直接优化语言模型以遵循人类偏好。我们提出直接偏好优化（DPO）：隐式优化与现有 RLHF 算法相同的目标（带 KL 散度约束的奖励最大化），但实现简单、训练直接。直观上，DPO 更新提高偏好回复相对非偏好回复的对数概率，同时引入动态的逐例重要性权重，以避免我们在朴素概率比目标上观察到的模型退化。与现有算法一样，DPO 依赖理论偏好模型（如 Bradley-Terry 模型 [5]）度量给定奖励函数与经验偏好数据的契合程度。不同的是：现有方法用偏好模型定义损失来训练奖励模型，再训练优化该奖励的策略；DPO 则通过变量替换，把偏好损失直接写成策略的函数。给定关于模型回复的人类偏好数据集，DPO 因而可用简单二元交叉熵目标优化策略，得到拟合偏好数据的隐式奖励函数所对应的最优策略。

Our main contribution is Direct Preference Optimization (DPO), a simple RL-free algorithm for training language models from preferences. Our experiments show that DPO is at least as effective as existing methods, including PPO-based RLHF, for learning from preferences in tasks such as sentiment modulation, summarization, and dialogue, using language models with up to 6B parameters.

我们的主要贡献是直接偏好优化（DPO）——一种从偏好训练语言模型的简单、无 RL 算法。实验表明，在情感调制、摘要与对话等任务上，用至多 6B 参数的语言模型从偏好中学习时，DPO 至少与包括基于 PPO 的 RLHF 在内的现有方法同样有效。

2 Related Work

Self-supervised language models of increasing scale learn to complete some tasks zero-shot [33] or with few-shot prompts [6, 27, 11]. However, their performance on downstream tasks and alignment with user intent can be significantly improved by fine-tuning on datasets of instructions and human-written completions [25, 38, 13, 41]. This ‘instruction-tuning’ procedure enables LLMs to generalize to instructions outside of the instruction-tuning set and generally increase their usability [13]. Despite the success of instruction tuning, relative human judgments of response quality are often easier to collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of human preferences, improving proficiency in translation [20], summarization [40, 51], story-telling [51], and instruction-following [28, 34]. These methods first optimize a neural network reward function for compatibility with the dataset of preferences under a preference model such as the Bradley-Terry model [5], then fine-tune a language model to maximize the given reward using reinforcement learning algorithms, commonly REINFORCE [47], proximal policy optimization (PPO; [39]), or variants [34]. A closely-related line of work leverages LLMs fine-tuned for instruction following with human feedback to generate additional synthetic preference data for targeted attributes such as safety or harmlessness [2], using only weak supervision from humans in the form of a text rubric for the LLM’s annotations. These methods represent a convergence of two bodies of work: one body of work on training language models with reinforcement learning for a variety of objectives [35, 29, 48] and another body of work on general methods for learning from human preferences [12, 21]. Despite the appeal of using relative human preferences, fine-tuning large language models with reinforcement learning remains a major practical challenge; this work provides a theoretically-justified approach to optimizing relative preferences without RL.

规模不断增大的自监督语言模型能以零样本 [33] 或少样本提示 [6, 27, 11] 完成部分任务。但在指令与人类撰写补全的数据集上微调后，它们在下游任务上的表现及与用户意图的对齐可显著提升 [25, 38, 13, 41]。这一「指令微调」使 LLM 能泛化到指令微调集之外的指令，并整体提高可用性 [13]。尽管指令微调成功，对回复质量的相对人类判断往往比专家演示更易收集，因而后续工作用人类偏好数据集微调 LLM，提升了翻译 [20]、摘要 [40, 51]、讲故事 [51] 与指令遵循 [28, 34] 等能力。这些方法先在 Bradley-Terry 等偏好模型 [5] 下优化与偏好数据兼容的神经网络奖励函数，再用强化学习算法——常见为 REINFORCE [47]、近端策略优化（PPO；[39]）或其变体 [34]——微调语言模型以最大化该奖励。密切相关的一条线利用经人类反馈指令微调的 LLM，仅为安全或无害等目标属性生成额外合成偏好数据 [2]，人类侧只需以文本评分标准形式提供弱监督。这些方法代表两股工作的汇合：一股用强化学习按多种目标训练语言模型 [35, 29, 48]，另一股是从人类偏好学习的一般方法 [12, 21]。尽管相对人类偏好很有吸引力，用强化学习微调大语言模型仍是重大实践挑战；本文给出在无 RL 条件下优化相对偏好的有理论依据的方法。

Outside of the context of language, learning policies from preferences has been studied in both bandit and reinforcement learning settings, and several approaches have been proposed. Contextual bandit learning using preferences or rankings of actions, rather than rewards, is known as a contextual dueling bandit (CDB; [50, 14]). In the absence of absolute rewards, theoretical analysis of CDBs substitutes the notion of an optimal policy with a von Neumann winner, a policy whose expected win rate against any other policy is at least 50% [14]. However, in the CDB setting, preference labels are given online, while in learning from human preferences, we typically learn from a fixed batch of offline preference-annotated action pairs [49]. Similarly, preference-based RL (PbRL) learns from binary preferences generated by an unknown ‘scoring’ function rather than rewards [9, 37]. Various algorithms for PbRL exist, including methods that can reuse off-policy preference data, but generally involve first explicitly estimating the latent scoring function (i.e. the reward model) and subsequently optimizing it [16, 9, 12, 36, 21]. We instead present a single stage policy learning approach that directly optimizes a policy to satisfy preferences.

在语言之外，从偏好学习策略已在老虎机与强化学习设定中研究，并提出多种方法。用动作偏好或排序而非奖励的上下文老虎机学习，称为上下文对决老虎机（CDB；[50, 14]）。在缺乏绝对奖励时，CDB 的理论分析用冯·诺依曼赢家替代最优策略概念：即对任意其他策略期望胜率至少 50% 的策略 [14]。但在 CDB 设定中偏好标签在线给出；而从人类偏好学习时，我们通常从一批离线、带偏好标注的动作对学习 [49]。类似地，基于偏好的 RL（PbRL）从未知「打分」函数生成的二元偏好而非奖励中学习 [9, 37]。PbRL 有多种算法，包括可复用离策略偏好数据的方法，但一般仍先显式估计潜在打分函数（即奖励模型）再优化它 [16, 9, 12, 36, 21]。我们则提出单阶段策略学习：直接优化策略以满足偏好。

3 Preliminaries

We review the RLHF pipeline in [51] (and later [40, 1, 28]). It usually includes three phases: 1) supervised fine-tuning (SFT); 2) preference sampling and reward learning and 3) RL optimization.

我们回顾 [51]（以及后续 [40, 1, 28]）中的 RLHF 管线。它通常包含三个阶段：1）监督微调（SFT）；2）偏好采样与奖励学习；3）RL 优化。

SFT: RLHF typically begins by fine-tuning a pre-trained LM with supervised learning on high-quality data for the downstream task(s) of interest (dialogue, summarization, etc.), to obtain a model $\pi^{\text{SFT}}$.

SFT：RLHF 通常先在感兴趣下游任务（对话、摘要等）的高质量数据上，用监督学习微调预训练 LM，得到模型 $\pi^{\text{SFT}}$。

Reward Modelling Phase: In the second phase the SFT model is prompted with prompts $x$ to produce pairs of answers $(y_{1},y_{2})\sim\pi^{\text{SFT}}(y\mid x)$. These are then presented to human labelers who express preferences for one answer, denoted as $y_{w}\succ y_{l}\mid x$ where $y_{w}$ and $y_{l}$ denotes the preferred and dispreferred completion amongst $(y_{1},y_{2})$ respectively. The preferences are assumed to be generated by some latent reward model $r^{*}(y,x)$, which we do not have access to. There are a number of approaches used to model preferences, the Bradley-Terry (BT) [5] model being a popular choice (although more general Plackett-Luce ranking models [32, 23] are also compatible with the framework if we have access to several ranked answers). The BT model stipulates that the human preference distribution $p^{*}$ can be written as:

奖励建模阶段：第二阶段用提示 $x$ 询问 SFT 模型，生成成对答案 $(y_{1},y_{2})\sim\pi^{\text{SFT}}(y\mid x)$。再交给人类标注者表达偏好，记为 $y_{w}\succ y_{l}\mid x$，其中 $y_{w}$、$y_{l}$ 分别是 $(y_{1},y_{2})$ 中更受偏好与更不受偏好的补全。偏好被假定由某个我们无法访问的潜在奖励模型 $r^{*}(y,x)$ 生成。建模偏好的方法有多种，Bradley-Terry（BT）[5] 模型是常见选择（若能获得若干排序答案，更一般的 Plackett-Luce 排序模型 [32, 23] 也与本框架兼容）。BT 模型规定人类偏好分布 $p^{*}$ 可写成：

$$p^{*}(y_{1}\succ y_{2}\mid x)=\frac{\exp\left(r^{*}(x,y_{1})\right)}{\exp\left(r^{*}(x,y_{1})\right)+\exp\left(r^{*}(x,y_{2})\right)}. \tag{1}$$

Assuming access to a static dataset of comparisons $\mathcal{D}=\bigl\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\bigr\}_{i=1}^{N}$ sampled from $p^{*}$, we can parametrize a reward model $r_{\phi}(x,y)$ and estimate the parameters via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss:

假定可访问从 $p^{*}$ 采样的静态比较数据集 $\mathcal{D}=\bigl\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\bigr\}_{i=1}^{N}$，可参数化奖励模型 $r_{\phi}(x,y)$ 并用极大似然估计参数。将该问题写成二元分类，得到负对数似然损失：

$$\mathcal{L}_{R}(r_{\phi},\mathcal{D})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\bigl[\log\sigma(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l}))\bigr] \tag{2}$$

where $\sigma$ is the logistic function. In the context of LMs, the network $r_{\phi}(x,y)$ is often initialized from the SFT model $\pi^{\text{SFT}}(y\mid x)$ with the addition of a linear layer on top of the final transformer layer that produces a single scalar prediction for the reward value [51]. To ensure a reward function with lower variance, prior works normalize the rewards, such that $\mathbb{E}_{x,y\sim\mathcal{D}}\left[r_{\phi}(x,y)\right]=0$ for all $x$.

其中 $\sigma$ 为 logistic 函数。在 LM 语境下，网络 $r_{\phi}(x,y)$ 常从 SFT 模型 $\pi^{\text{SFT}}(y\mid x)$ 初始化，并在最终 transformer 层之上加线性层，产出标量奖励预测 [51]。为降低奖励方差，先前工作会归一化奖励，使得对所有 $x$ 有 $\mathbb{E}_{x,y\sim\mathcal{D}}\left[r_{\phi}(x,y)\right]=0$。

RL Fine-Tuning Phase: During the RL phase, the learned reward function is used to provide feedback to the language model. Following prior works [17, 18], the optimization is formulated as

RL 微调阶段：RL 阶段用学到的奖励函数向语言模型提供反馈。遵循先前工作 [17, 18]，优化被表述为

$$\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(y\mid x)}\bigl[r_{\phi}(x,y)\bigr]-\beta\mathbb{D}_{\textrm{KL}}\bigl[\pi_{\theta}(y\mid x)\mid\mid\pi_{\text{ref}}(y\mid x)\bigr], \tag{3}$$

where $\beta$ is a parameter controlling the deviation from the base reference policy $\pi_{\text{ref}}$, namely the initial SFT model $\pi^{\text{SFT}}$. In practice, the language model policy $\pi_{\theta}$ is also initialized to $\pi^{\text{SFT}}$. The added constraint is important, as it prevents the model from deviating too far from the distribution on which the reward model is accurate, as well as maintaining the generation diversity and preventing mode-collapse to single high-reward answers. Due to the discrete nature of language generation, this objective is not differentiable and is typically optimized with reinforcement learning. The standard approach [51, 40, 1, 28] has been to construct the reward function ${r(x,y)=r_{\phi}(x,y)-\beta(\log\pi_{\theta}(y\mid x)-\log\pi_{\text{ref}}(y\mid x))}$, and maximize using PPO [39].

其中 $\beta$ 控制相对基础参考策略 $\pi_{\text{ref}}$（即初始 SFT 模型 $\pi^{\text{SFT}}$）的偏离程度。实践中，语言模型策略 $\pi_{\theta}$ 也初始化为 $\pi^{\text{SFT}}$。附加约束很重要：它防止模型偏离奖励模型仍准确的分布过远，并维持生成多样性、避免塌缩到单一高奖励答案。由于语言生成离散，该目标不可微，通常用强化学习优化。标准做法 [51, 40, 1, 28] 是构造奖励函数 ${r(x,y)=r_{\phi}(x,y)-\beta(\log\pi_{\theta}(y\mid x)-\log\pi_{\text{ref}}(y\mid x))}$，再用 PPO [39] 最大化。

4 Direct Preference Optimization

Motivated by the challenges of applying reinforcement learning algorithms on large-scale problems such as fine-tuning language models, our goal is to derive a simple approach for policy optimization using preferences directly. Unlike prior RLHF methods, which learn a reward and then optimize it via RL, our approach leverages a particular choice of reward model parameterization that enables extraction of its optimal policy in closed form, without an RL training loop. As we will describe next in detail, our key insight is to leverage an analytical mapping from reward functions to optimal policies, which enables us to transform a loss function over reward functions into a loss function over policies. This change-of-variables approach avoids fitting an explicit, standalone reward model, while still optimizing under existing models of human preferences, such as the Bradley-Terry model. In essence, the policy network represents both the language model and the (implicit) reward.

受大规模问题上应用强化学习算法（如微调语言模型）之挑战的驱动，我们的目标是导出直接用偏好做策略优化的简单方法。与先前先学奖励再用 RL 优化的 RLHF 方法不同，我们的方法利用特定奖励模型参数化，使其最优策略可闭式提取，无需 RL 训练环。如下详述，关键洞见是利用从奖励函数到最优策略的解析映射，把关于奖励函数的损失函数变换为关于策略的损失函数。这一变量替换避免拟合显式、独立的奖励模型，同时仍在 Bradley-Terry 等现有人类偏好模型下优化。本质上，策略网络同时表示语言模型与（隐式）奖励。

Deriving the DPO objective. We start with the same RL objective as prior work, Eq. 3, under a general reward function $r$. Following prior work [31, 30, 19, 15], it is straightforward to show that the optimal solution to the KL-constrained reward maximization objective in Eq. 3 takes the form:

导出 DPO 目标。 我们从与先前工作相同的 RL 目标（式 3）出发，奖励函数取一般形式 $r$。遵循先前工作 [31, 30, 19, 15]，容易证明式 3 中带 KL 约束的奖励最大化目标的最优解具有形式：

$$\pi_{r}(y\mid x)=\frac{1}{Z(x)}\pi_{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right), \tag{4}$$

where $Z(x)=\sum_{y}\pi_{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right)$ is the partition function. See Appendix A.1 for a complete derivation. Even if we use the MLE estimate $r_{\phi}$ of the ground-truth reward function $r^{*}$, it is still expensive to estimate the partition function $Z(x)$ [19, 15], which makes this representation hard to utilize in practice. However, we can rearrange Eq. 4 to express the reward function in terms of its corresponding optimal policy $\pi_{r}$, the reference policy $\pi_{\text{ref}}$, and the unknown partition function $Z(\cdot)$. Specifically, we first take the logarithm of both sides of Eq. 4 and then with some algebra we obtain:

其中 $Z(x)=\sum_{y}\pi_{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right)$ 为配分函数。完整推导见附录 A.1。即便使用真实奖励 $r^{*}$ 的 MLE 估计 $r_{\phi}$，估计配分函数 $Z(x)$ 仍昂贵 [19, 15]，使该表示难以实用。但我们可重排式 4，用对应最优策略 $\pi_{r}$、参考策略 $\pi_{\text{ref}}$ 与未知配分函数 $Z(\cdot)$ 表达奖励函数。具体地，先对式 4 两边取对数，再经代数运算得到：

$$r(x,y)=\beta\log\frac{\pi_{r}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}+\beta\log Z(x). \tag{5}$$

We can apply this reparameterization to the ground-truth reward $r^{*}$ and corresponding optimal model $\pi^{*}$. Fortunately, the Bradley-Terry model depends only on the difference of rewards between two completions, i.e., ${p^{*}(y_{1}\succ y_{2}\mid x)=\sigma(r^{*}(x,y_{1})-r^{*}(x,y_{2}))}$. Substituting the reparameterization in Eq. 5 for $r^{*}(x,y)$ into the preference model Eq. 1, the partition function cancels, and we can express the human preference probability in terms of only the optimal policy $\pi^{*}$ and reference policy $\pi_{\text{ref}}$. Thus, the optimal RLHF policy $\pi^{*}$ under the Bradley-Terry model satisfies the preference model:

可将该重参数化应用于真实奖励 $r^{*}$ 与对应最优模型 $\pi^{*}$。幸运的是，Bradley-Terry 模型只依赖两个补全的奖励之差，即 ${p^{*}(y_{1}\succ y_{2}\mid x)=\sigma(r^{*}(x,y_{1})-r^{*}(x,y_{2}))}$。把式 5 中对 $r^{*}(x,y)$ 的重参数化代入偏好模型式 1，配分函数消去，人类偏好概率便可仅用最优策略 $\pi^{*}$ 与参考策略 $\pi_{\text{ref}}$ 表达。因此，Bradley-Terry 模型下的最优 RLHF 策略 $\pi^{*}$ 满足偏好模型：

$$p^{*}(y_{1}\succ y_{2}\mid x)=\frac{1}{1+\exp\left(\beta\log\frac{\pi^{*}(y_{2}\mid x)}{\pi_{\text{ref}}(y_{2}\mid x)}-\beta\log\frac{\pi^{*}(y_{1}\mid x)}{\pi_{\text{ref}}(y_{1}\mid x)}\right)} \tag{6}$$

The derivation is in Appendix A.2. While Eq. 6 uses the Bradley-Terry model, we can similarly derive expressions under the more general Plackett-Luce models [32, 23], shown in Appendix A.3.

推导见附录 A.2。式 6 使用 Bradley-Terry 模型；在更一般的 Plackett-Luce 模型 [32, 23] 下可类似导出表达式，见附录 A.3。

Now that we have the probability of human preference data in terms of the optimal policy rather than the reward model, we can formulate a maximum likelihood objective for a parametrized policy $\pi_{\theta}$. Analogous to the reward modeling approach (i.e. Eq. 2), our policy objective becomes:

既然已把人类偏好数据的概率写成最优策略而非奖励模型的函数，便可对参数化策略 $\pi_{\theta}$ 建立极大似然目标。类比奖励建模方法（即式 2），策略目标变为：

$$\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\text{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\text{ref}}(y_{l}\mid x)}\right)\right]. \tag{7}$$

This way, we fit an implicit reward using an alternative parameterization, whose optimal policy is simply $\pi_{\theta}$. Moreover, since our procedure is equivalent to fitting a reparametrized Bradley-Terry model, it enjoys certain theoretical properties, such as consistencies under suitable assumption of the preference data distribution [4]. In Section 5, we further discuss theoretical properties of DPO in relation to other works.

这样，我们用替代参数化拟合隐式奖励，其最优策略正是 $\pi_{\theta}$。此外，由于该过程等价于拟合重参数化的 Bradley-Terry 模型，在偏好数据分布的合适假设下享有一致性等理论性质 [4]。第 5 节将进一步讨论 DPO 相对其他工作的理论性质。

What does the DPO update do? For a mechanistic understanding of DPO, it is useful to analyze the gradient of the loss function $\mathcal{L}_{\text{DPO}}$. The gradient with respect to the parameters $\theta$ can be written as:

DPO 更新在做什么？ 为从机制上理解 DPO，分析损失函数 $\mathcal{L}_{\text{DPO}}$ 的梯度很有用。对参数 $\theta$ 的梯度可写成：

$$\nabla_{\theta}\mathcal{L}{\text{DPO}}(\pi{\theta};\pi_{\text{ref}})=\
-\beta\mathbb{E}{(x,y{w},y_{l})\sim\mathcal{D}}\bigg[\underbrace{\sigma(\hat{r}{\theta}(x,y{l})-\hat{r}{\theta}(x,y{w}))}{\text{higher weight when reward estimate is wrong}}\bigg[\underbrace{\nabla{\theta}\log\pi(y_{w}\mid x)}{\text{increase likelihood of $y{w}}}-\underbrace{\nabla_{\theta}\log\pi(y_{l}\mid x)}_{\text{decrease likelihood ofy_{l}}}\bigg]\bigg],$

where $\hat{r}_{\theta}(x,y)=\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ is the reward implicitly defined by the language model $\pi_{\theta}$ and reference model $\pi_{\text{ref}}$ (more in Section 5). Intuitively, the gradient of the loss function $\mathcal{L}_{\text{DPO}}$ increases the likelihood of the preferred completions $y_{w}$ and decreases the likelihood of dispreferred completions $y_{l}$. Importantly, the examples are weighed by how much higher the implicit reward model $\hat{r}_{\theta}$ rates the dispreferred completions, scaled by $\beta$, i.e, how incorrectly the implicit reward model orders the completions, accounting for the strength of the KL constraint. Our experiments suggest the importance of this weighting, as a naïve version of this method without the weighting coefficient can cause the language model to degenerate (Appendix Table 3).

其中 $\hat{r}_{\theta}(x,y)=\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ 是由语言模型 $\pi_{\theta}$ 与参考模型 $\pi_{\text{ref}}$ 隐式定义的奖励（详见第 5 节）。直观上，$\mathcal{L}_{\text{DPO}}$ 的梯度提高偏好补全 $y_{w}$ 的似然、降低非偏好补全 $y_{l}$ 的似然。重要的是，样本按隐式奖励模型 $\hat{r}_{\theta}$ 给非偏好补全打分高出多少加权，并经 $\beta$ 缩放——即隐式奖励模型对补全排序错到什么程度，同时计入 KL 约束强度。实验表明该加权很关键：去掉加权系数的朴素版本会导致语言模型退化（附录表 3）。

DPO outline. The general DPO pipeline is as follows: 1) Sample completions $y_{1},y_{2}\sim\pi_{\text{ref}}(\cdot\mid x)$ for every prompt $x$, label with human preferences to construct the offline dataset of preferences $\mathcal{D}=\{x^{(i)},y_{w}^{(i)},y_{l})^{(i)}\}_{i=1}^{N}$ and 2) optimize the language model $\pi_{\theta}$ to minimize $\mathcal{L}_{\text{DPO}}$ for the given $\pi_{\text{ref}}$ and $\mathcal{D}$ and desired $\beta$. In practice, one would like to reuse preference datasets publicly available, rather than generating samples and gathering human preferences. Since the preference datasets are sampled using $\pi^{\text{SFT}}$, we initialize $\pi_{\text{ref}}=\pi^{\text{SFT}}$ whenever available. However, when $\pi^{\text{SFT}}$ is not available, we initialize $\pi_{\text{ref}}$ by maximizing likelihood of preferred completions ${(x,y_{w})}$, that is, ${\pi_{\text{ref}}=\argmax_{\pi}\mathbb{E}_{x,y_{w}\sim\mathcal{D}}\left[\log\pi(y_{w}\mid x)\right]}$. This procedure helps mitigate the distribution shift between the true reference distribution which is unavailable, and $\pi_{\text{ref}}$ used by DPO. Further details related to the implementation and hyperparameters can be found in Appendix B.

DPO 流程概要。 一般 DPO 管线如下：1）对每个提示 $x$ 采样补全 $y_{1},y_{2}\sim\pi_{\text{ref}}(\cdot\mid x)$，用人类偏好标注，构造离线偏好数据集 $\mathcal{D}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\}_{i=1}^{N}$；2）在给定 $\pi_{\text{ref}}$、$\mathcal{D}$ 与期望 $\beta$ 下，优化语言模型 $\pi_{\theta}$ 以最小化 $\mathcal{L}_{\text{DPO}}$。实践中人们更愿复用公开偏好数据集，而非自行采样并收集人类偏好。由于偏好数据集用 $\pi^{\text{SFT}}$ 采样，只要可得便初始化 $\pi_{\text{ref}}=\pi^{\text{SFT}}$。若 $\pi^{\text{SFT}}$ 不可得，则通过对偏好补全 ${(x,y_{w})}$ 极大似然来初始化 $\pi_{\text{ref}}$，即 ${\pi_{\text{ref}}=\argmax_{\pi}\mathbb{E}_{x,y_{w}\sim\mathcal{D}}\left[\log\pi(y_{w}\mid x)\right]}$。该步骤有助于缓解不可得的真实参考分布与 DPO 所用 $\pi_{\text{ref}}$ 之间的分布偏移。实现与超参细节见附录 B。

5 Theoretical Analysis of DPO

In this section, we give further interpretation of the DPO method, provide theoretical backing, and relate advantages of DPO to issues with actor critic algorithms used for RLHF (such as PPO [39]).

本节进一步解释 DPO 方法，提供理论支撑，并把 DPO 的优势与用于 RLHF 的演员–评论家算法（如 PPO [39]）的问题联系起来。

5.1 Your Language Model Is Secretly a Reward Model

DPO is able to bypass both fitting an explicit reward and performing RL to learn the policy using a single maximum likelihood objective. Note the optimization objective Eq. 5 is equivalent to a Bradley-Terry model with a reward parameterization $r^{*}(x,y)=\beta\log\frac{\pi^{*}_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ and we optimize our parametric model $\pi_{\theta}$, equivalently to the reward model optimization in Eq. 2 under the change of variables. In this section we will build the theory behind this reparameterization, show that it does not constrain the class of learned reward models, and allows for the exact recovery of the optimal policy. We begin with by defining an equivalence relation between reward functions.

DPO 能用单一极大似然目标绕过显式拟合奖励以及用 RL 学习策略。注意优化目标式 5 等价于奖励参数化为 $r^{*}(x,y)=\beta\log\frac{\pi^{*}_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ 的 Bradley-Terry 模型，我们在变量替换下优化参数模型 $\pi_{\theta}$，等价于式 2 中的奖励模型优化。本节建立该重参数化背后的理论，证明它不约束可学奖励模型的类别，并允许精确恢复最优策略。我们先定义奖励函数之间的等价关系。

Definition 1.

We say that two reward functions $r(x,y)$ and $r^{\prime}(x,y)$ are equivalent iff ${r(x,y)-r^{\prime}(x,y)=f(x)}$ for some function $f$.

定义 1。

称两个奖励函数 $r(x,y)$ 与 $r^{\prime}(x,y)$ 等价，当且仅当存在函数 $f$ 使 ${r(x,y)-r^{\prime}(x,y)=f(x)}$。

It is easy to see that this is indeed an equivalence relation, which partitions the set of reward functions into classes. We can state the following two lemmas:

容易验证这确为等价关系，并把奖励函数集合划分为若干类。可陈述以下两条引理：

Lemma 1.

Under the Plackett-Luce, and in particular the Bradley-Terry, preference framework, two reward functions from the same class induce the same preference distribution.

引理 1。

在 Plackett-Luce、特别是 Bradley-Terry 偏好框架下，同一类中的两个奖励函数诱导相同的偏好分布。

Lemma 2.

Two reward functions from the same equivalence class induce the same optimal policy under the constrained RL problem.

引理 2。

同一等价类中的两个奖励函数，在带约束的 RL 问题下诱导相同的最优策略。

The proofs are straightforward and we defer them to Appendix A.5. The first lemma is a well-known under-specification issue with the Plackett-Luce family of models [32]. Due to this under-specification, we usually have to impose additional identifiability constraints to achieve any guarantees on the MLE estimates from Eq. 2 [4]. The second lemma states that all reward functions from the same class yield the same optimal policy, hence for our final objective, we are only interested in recovering an arbitrary reward function from the optimal class. We prove the following Theorem in Appendix A.6:

证明直接，推迟到附录 A.5。第一条引理是 Plackett-Luce 族模型的已知欠定问题 [32]。由于该欠定，通常需施加额外可识别性约束，才能对式 2 的 MLE 估计给出任何保证 [4]。第二条引理表明同一类中所有奖励函数给出相同最优策略，因此对最终目标，我们只关心恢复最优类中的任一奖励函数。我们在附录 A.6 证明下述定理：

Theorem 1.

Under mild assumptions, all reward classes consistent with the Plackett-Luce (and Bradley-Terry in particular) models can be represented with the reparameterization ${r(x,y)=\beta\log\frac{\pi(y\mid x)}{\pi_{\text{ref}}(y\mid x)}}$ for some model $\pi(y\mid x)$ and a given reference model $\pi_{\text{ref}}(y\mid x)$.

定理 1。

在温和假设下，与 Plackett-Luce（特别是 Bradley-Terry）模型一致的所有奖励类，都可用重参数化 ${r(x,y)=\beta\log\frac{\pi(y\mid x)}{\pi_{\text{ref}}(y\mid x)}}$ 表示，其中 $\pi(y\mid x)$ 为某模型，$\pi_{\text{ref}}(y\mid x)$ 为给定参考模型。

Proof Sketch.

Consider any reward function $r(x,y)$, which induces a corresponding optimal model $\pi_{r}(y\mid x)$, specified by Eq. 4. We will show that a reward function from the equivalence class of $r$ can be represented using the reparameterization given above. We define the projection $f$ as

证明概要。

考虑任意奖励函数 $r(x,y)$，它由式 4 诱导对应最优模型 $\pi_{r}(y\mid x)$。我们将证明 $r$ 的等价类中某个奖励函数可用上述重参数化表示。定义投影 $f$ 为

$$f(r;\pi_{\text{ref}},\beta)(x,y)=r(x,y)-\beta\log\sum_{y}\pi_{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right) \tag{8}$$

The operator $f$ simply normalizes the reward function with the logarithm of the partition function of $\pi_{r}$. Since the added normalization term is only a function of the prefix $x$, $f(r;\pi_{\text{ref}},\beta)(x,y)$ is a reward function in the equivalence class of $r(x,y)$. Finally, replacing $r$ with the RHS of Eq. 5 (which holds for any reward function), we have $f(r;\pi_{\text{ref}},\beta)(x,y)=\beta\log\frac{\pi_{r}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$. That is, the projection $f$ produces a member of the equivalence class of $r$ with the desired form, and we do not lose any generality in our reward model from the proposed reparameterization. ∎

算子 $f$ 仅用 $\pi_{r}$ 配分函数的对数归一化奖励函数。由于所加归一化项只是前缀 $x$ 的函数，$f(r;\pi_{\text{ref}},\beta)(x,y)$ 属于 $r(x,y)$ 的等价类。最后，用式 5 右侧替换 $r$（该式对任意奖励函数成立），得到 $f(r;\pi_{\text{ref}},\beta)(x,y)=\beta\log\frac{\pi_{r}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$。亦即投影 $f$ 产出具有期望形式的 $r$ 等价类成员，所提重参数化并不损失奖励模型的一般性。∎

We can alternatively view Theorem 1 as specifying exactly which reward function within each equivalence class the DPO reparameterization selects, that is, the reward function satisfying:

也可把定理 1 看作精确指明 DPO 重参数化在每个等价类中选取哪个奖励函数，即满足下式的奖励函数：

$$\sum_{y}\underbrace{\pi_{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r(x,y)\right)}_{=\pi(y\mid x)\text{, using Thm.~\ref{thm:main} reparam.}}=1, \tag{9}$$

i.e., $\pi(y\mid x)$ is a valid distribution (probabilities are positive and sum to 1). However, following Eq. 4, we can see that Eq. 9 is the partition function of the optimal policy induced by the reward function $r(x,y)$. The key insight of the DPO algorithm is that we can impose certain constraints on the under-constrained Plackett-Luce (and Bradley-Terry in particular) family of preference models, such that we preserve the class of representable reward models, but explicitly make the optimal policy in Eq. 4 analytically tractable for all prompts $x$.

亦即 $\pi(y\mid x)$ 是合法分布（概率非负且和为 1）。但由式 4 可见，式 9 正是奖励函数 $r(x,y)$ 所诱导最优策略的配分函数。DPO 算法的关键洞见是：可对欠定的 Plackett-Luce（特别是 Bradley-Terry）偏好模型族施加某些约束，在保持可表示奖励模型类别的同时，显式使式 4 中的最优策略对所有提示 $x$ 解析可求。

5.2 Instability of Actor-Critic Algorithms

We can also use our framework to diagnose instabilities with standard actor-critic algorithms used for the RLHF, such as PPO. We follow the RLHF pipeline and focus on the RL fine-tuning step outlined in Section 3. We can draw connections to the control as inference framework [22] for the constrained RL problem outlined in 3. We assume a parameterized model $\pi_{\theta}(y\mid x)$ and minimize $\mathbb{D}_{\text{KL}}[\pi_{\theta}(y|x)\mid\mid\pi^{*}(y\mid x)]$ where $\pi^{*}$ is the optimal policy from Eq. 7 induced by the reward function $r_{\phi}(y,x)$. With some algebra this leads to the optimization objective:

也可用本框架诊断用于 RLHF 的标准演员–评论家算法（如 PPO）的不稳定性。我们遵循 RLHF 管线，聚焦第 3 节概述的 RL 微调步骤。可把第 3 节带约束 RL 问题与「控制即推断」框架 [22] 联系起来。假定参数化模型 $\pi_{\theta}(y\mid x)$，最小化 $\mathbb{D}_{\text{KL}}[\pi_{\theta}(y|x)\mid\mid\pi^{*}(y\mid x)]$，其中 $\pi^{*}$ 是由奖励函数 $r_{\phi}(y,x)$ 经式 7 诱导的最优策略。经代数运算得到优化目标：

$$\max_{\pi_{\theta}}\mathbb{E}_{\pi_{\theta}(y\mid x)}\bigg[\underbrace{r_{\phi}(x,y)-\beta\log\sum_{y}\pi_{\text{ref}}(y\mid x)\exp\left(\frac{1}{\beta}r_{\phi}(x,y)\right)}_{f(r_{\phi},\pi_{\text{ref}},\beta)}-\underbrace{\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}}_{\text{KL}}\bigg] \tag{10}$$

This is the same objective optimized in prior works [51, 40, 1, 28] using the DPO-equivalent reward for the reward class of $r_{\phi}$. In this setting, we can interpret the normalization term in $f(r_{\phi},\pi_{\text{ref}},\beta)$ as the soft value function of the reference policy $\pi_{\text{ref}}$. While this term does not affect the optimal solution, without it, the policy gradient of the objective could have high variance, making learning unstable. We can accommodate for the normalization term using a learned value function, but that can also be difficult to optimize. Alternatively, prior works have normalized rewards using a human completion baseline, essentially a single sample Monte-Carlo estimate of the normalizing term. In contrast the DPO reparameterization yields a reward function that does not require any baselines.

这正是先前工作 [51, 40, 1, 28] 用与 $r_{\phi}$ 奖励类的 DPO 等价奖励所优化的同一目标。在此设定下，可将 $f(r_{\phi},\pi_{\text{ref}},\beta)$ 中的归一化项解释为参考策略 $\pi_{\text{ref}}$ 的软价值函数。该项虽不影响最优解，但若缺少它，目标的策略梯度可能方差很大，使学习不稳定。可用学到的价值函数容纳归一化项，但价值函数也可能难优化。或者，先前工作用人类补全基线归一化奖励，本质上是对归一化项的单样本蒙特卡洛估计。相对地，DPO 重参数化给出的奖励函数不需要任何基线。

6 Experiments

In this section, we empirically evaluate DPO’s ability to train policies directly from preferences. First, in a well-controlled text-generation setting, we ask: how efficiently does DPO trade off maximizing reward and minimizing KL-divergence with the reference policy, compared to common preference learning algorithms such as PPO? Next, we evaluate DPO’s performance on larger models and more difficult RLHF tasks, including summarization and dialogue. We find that with almost no tuning of hyperparameters, DPO tends to perform as well or better than strong baselines like RLHF with PPO as well as returning the best of $N$ sampled trajectories under a learned reward function. Before presenting these results, we describe the experimental set-up; additional details are in Appendix C.

本节经验评估 DPO 直接从偏好训练策略的能力。首先，在受控文本生成设定中询问：相对 PPO 等常见偏好学习算法，DPO 在最大化奖励与最小化相对参考策略的 KL 散度之间权衡效率如何？接着评估 DPO 在更大模型与更难 RLHF 任务（含摘要与对话）上的表现。我们发现：几乎无需调超参，DPO 往往与基于 PPO 的 RLHF、以及在学到的奖励函数下返回 $N$ 条采样轨迹中最优者等强基线相当或更好。在展示结果前，先说明实验设置；更多细节见附录 C。

Tasks. Our experiments explore three different open-ended text generation tasks. For all experiments, algorithms learn a policy from a dataset of preferences $\mathcal{D}=\bigl\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\bigr\}_{i=1}^{N}$. In controlled sentiment generation, $x$ is a prefix of a movie review from the IMDb dataset [24], and the policy must generate $y$ with positive sentiment. In order to perform a controlled evaluation, for this experiment we generate preference pairs over generations using a pre-trained sentiment classifier, where $p(\text{positive}\mid x,y_{w})>p(\text{positive}\mid x,y_{l})$. For SFT, we fine-tune GPT-2-large until convergence on reviews from the train split of the IMDB dataset (further details in App C.1). In summarization, $x$ is a forum post from Reddit; the policy must generate a summary $y$ of the main points in the post. Following prior work, we use the Reddit TL;DR summarization dataset [43] along with human preferences gathered by [40]. We use an SFT model fine-tuned on human-written forum post summaries[^1] with the TRLX [44] framework for RLHF. The human preference dataset was gathered by [40] on samples from a different, but similarly-trained, SFT model. Finally, in single-turn dialogue, $x$ is a human query, which may be anything from a question about astrophysics to a request for relationship advice. A policy must produce an engaging and helpful response $y$ to a user’s query; we use the Anthropic Helpful and Harmless dialogue dataset [1], containing 170k dialogues between a human and an automated assistant. Each transcript ends with a pair of responses generated by a large (although unknown) language model along with a preference label denoting the human-preferred response. In this setting, no pre-trained SFT model is available; we therefore fine-tune an off-the-shelf language model on only the preferred completions to form the SFT model.

任务。 实验覆盖三项开放式文本生成任务。所有实验中，算法从偏好数据集 $\mathcal{D}=\bigl\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\bigr\}_{i=1}^{N}$ 学习策略。在受控情感生成中，$x$ 是 IMDb 数据集 [24] 影评前缀，策略须生成带正面情感的 $y$。为做受控评估，本实验用预训练情感分类器生成成对偏好，满足 $p(\text{positive}\mid x,y_{w})>p(\text{positive}\mid x,y_{l})$。SFT 阶段在 IMDB 训练集影评上微调 GPT-2-large 至收敛（详见附录 C.1）。在摘要中，$x$ 是 Reddit 论坛帖；策略须生成抓住帖子要点的摘要 $y$。遵循先前工作，使用 Reddit TL;DR 摘要数据集 [43] 以及 [40] 收集的人类偏好。SFT 模型在人类撰写的论坛帖摘要上微调[^1]，RLHF 使用 TRLX [44] 框架。人类偏好数据集由 [40] 在另一台但训练方式相近的 SFT 模型样本上收集。最后，在单轮对话中，$x$ 是人类查询，从天体物理问题到情感建议请求不等。策略须对用户查询给出吸引人且有帮助的回复 $y$；我们使用 Anthropic Helpful and Harmless 对话数据集 [1]，含 17 万段人类与自动助手的对话。每段转录以某大型（虽未知）语言模型生成的一对回复结尾，并带标明人类更偏好哪条的标签。此设定无预训练 SFT 模型可用；因此我们仅在偏好补全上微调现成语言模型，形成 SFT 模型。

Figure 2: Left. The frontier of expected reward vs KL to the reference policy. DPO provides the highest expected reward for all KL values, demonstrating the quality of the optimization. Right. TL;DR summarization win rates vs. human-written summaries, using GPT-4 as evaluator. DPO exceeds PPO’s best-case performance on summarization, while being more robust to changes in the sampling temperature.

图 2：左。 期望奖励相对参考策略 KL 的前沿。对所有 KL 取值，DPO 给出最高期望奖励，显示优化质量。右。 以 GPT-4 为评判的 TL;DR 摘要相对人类撰写摘要的胜率。DPO 在摘要上超过 PPO 的最佳表现，且对采样温度变化更稳健。

Evaluation. Our experiments use two different approaches to evaluation. In order to analyze the effectiveness of each algorithm in optimizing the constrained reward maximization objective, in the controlled sentiment generation setting we evaluate each algorithm by its frontier of achieved reward and KL-divergence from the reference policy; this frontier is computable because we have acccess to the ground-truth reward function (a sentiment classifier). However, in the real world, the ground truth reward function is not known; therefore, we evaluate algorithms with their win rate against a baseline policy, using GPT-4 as a proxy for human evaluation of summary quality and response helpfulness in the summarization and single-turn dialogue settings, respectively. For summarization, we use reference summaries in the test set as the baseline; for dialogue, we use the preferred response in the test dataset as the baseline. While existing studies suggest LMs can be better automated evaluators than existing metrics [10], we conduct a human study to justify our usage of GPT-4 for evaluation in Sec. 6.4. We find GPT-4 judgments correlate strongly with humans, with human agreement with GPT-4 typically similar or higher than inter-human annotator agreement.

评估。 实验采用两种评估方式。为分析各算法优化带约束奖励最大化目标的有效性，在受控情感生成设定中，用各算法达到的奖励与相对参考策略 KL 散度的前沿评估；该前沿可计算，因我们可访问真实奖励函数（情感分类器）。但现实中真实奖励未知；因此在摘要与单轮对话设定中，用相对基线策略的胜率评估算法，分别以 GPT-4 作为摘要质量与回复有用性的人类评估代理。摘要以测试集参考摘要为基线；对话以测试集中的偏好回复为基线。虽有研究提示 LM 可比现有指标更好充当自动评判 [10]，我们仍在第 6.4 节做人研究以证明用 GPT-4 评估的合理性。发现 GPT-4 判断与人类高度相关，人类与 GPT-4 的一致率通常与人类互评一致率相当或更高。

Methods. In addition to DPO, we evaluate several existing approaches to training language models to adhere to human preferences. Most simply, we explore zero-shot prompting with GPT-J [45] in the summarization task and 2-shot prompting with Pythia-2.8B [3] in the dialogue task. In addition, we evaluate the SFT model as well as Preferred-FT, which is a model fine-tuned with supervised learning on the chosen completion $y_{w}$ from either the SFT model (in controlled sentiment and summarization) or a generic LM (in single-turn dialogue). Another pseudo-supervised method is Unlikelihood [46], which simply optimizes the policy to maximize the probability assigned to $y_{w}$ and minimize the probability assigned to $y_{l}$; we use an optional coefficient $\alpha\in[0,1]$ on the ‘unlikelihood’ term. We also consider PPO [39] using a reward function learned from the preference data and PPO-GT, which is an oracle that learns from the ground truth reward function available in the controlled sentiment setting. In our sentiment experiments, we use two implementations of PPO-GT, one of-the-shelf version [44] as well as a modified version that normalizes rewards and further tunes hyperparameters to improve performance (we also use these modifications when running ‘normal’ PPO with learned rewards). Finally, we consider the Best of $N$ baseline, sampling $N$ responses from the SFT model (or Preferred-FT in dialogue) and returning the highest-scoring response according to a reward function learned from the preference dataset. This high-performing method decouples the quality of the reward model from the PPO optimization, but is computationally impractical even for moderate $N$ as it requires sampling $N$ completions for every query at test time.

方法。 除 DPO 外，我们评估若干训练语言模型以遵循人类偏好的现有方法。最简单的是摘要任务上用 GPT-J [45] 零样本提示，以及对话任务上用 Pythia-2.8B [3] 做 2-shot 提示。此外评估 SFT 模型与 Preferred-FT：后者在所选补全 $y_{w}$ 上做监督学习微调，$y_{w}$ 来自 SFT 模型（受控情感与摘要）或通用 LM（单轮对话）。另一种伪监督方法是 Unlikelihood [46]：优化策略以最大化赋给 $y_{w}$ 的概率并最小化赋给 $y_{l}$ 的概率；我们在「反似然」项上使用可选系数 $\alpha\in[0,1]$。还考虑用从偏好数据学到的奖励函数的 PPO [39]，以及可在受控情感设定访问真实奖励的预言机 PPO-GT。情感实验中使用两个 PPO-GT 实现：现成版本 [44]，以及归一化奖励并进一步调超参以提升表现的修改版（对用学到奖励的「普通」PPO 也使用这些修改）。最后考虑 Best of $N$ 基线：从 SFT 模型（对话中为 Preferred-FT）采样 $N$ 条回复，按从偏好数据学到的奖励函数返回最高分回复。该方法把奖励模型质量与 PPO 优化解耦，表现强，但即使对中等 $N$，测试时也需对每个查询采样 $N$ 条补全，计算上不实用。

6.1 How well can DPO optimize the RLHF objective?

Figure 3: Left. Win rates computed by GPT-4 for Anthropic-HH one-step dialogue; DPO is the only method that improves over chosen summaries in the Anthropic-HH test set. Right. Win rates for different sampling temperatures over the course of training. DPO’s improvement over the dataset labels is fairly stable over the course of training for different sampling temperatures.

图 3：左。 Anthropic-HH 单步对话上由 GPT-4 计算的胜率；DPO 是唯一相对 Anthropic-HH 测试集所选摘要有提升的方法。右。 训练过程中不同采样温度下的胜率。相对数据集标签，DPO 的改进在不同采样温度下于训练过程中相当稳定。

The KL-constrained reward maximization objective used in typical RLHF algorithms balances exploitation of reward while restricting the policy from deviating far from the reference policy. Therefore, when comparing algorithms, we must take into account both reward achieved as well as the KL discrepancy; achieving slightly higher reward but with much higher KL is not necessarily desirable. Figure 2 shows the reward-KL frontier for various algorithms in the sentiment setting. We execute multiple training runs for each algorithm, using a different hyperparameter for policy conservativeness in each run (target KL $\in\{3,6,9,12\}$ for PPO, $\beta\in\{0.05,0.1,1,5\}$, $\alpha\in\{0.05,0.1,0.5,1\}$ for unlikelihood, random seeds for preferred-FT). This sweep includes 22 runs in total. After each 100 training steps until convergence, we evaluate each policy on a set of test prompts, computing the average reward under the true reward function as well as the average sequence-level KL[^2] with the reference policy $\text{KL}\left(\pi\mid\mid\pi_{\text{ref}}\right)$. We find that DPO produces by far the most efficient frontier, achieving the highest reward while still achieving low KL. This result is particularly notable for multiple reasons. First, DPO and PPO optimize the same objective, but DPO is notably more efficient; DPO’s reward/KL tradeoff strictly dominates PPO. Second, DPO achieves a better frontier than PPO, even when PPO can access ground truth rewards (PPO-GT).

典型 RLHF 算法所用的带 KL 约束奖励最大化目标，在利用奖励的同时限制策略偏离参考策略过远。因此比较算法时，必须同时考虑达到的奖励与 KL 差异；略高奖励却伴随高得多的 KL 未必可取。图 2 展示情感设定中各算法的奖励–KL 前沿。我们对每个算法执行多次训练，每次使用不同的策略保守性超参（PPO 的目标 KL $\in\{3,6,9,12\}$，DPO 的 $\beta\in\{0.05,0.1,1,5\}$，反似然的 $\alpha\in\{0.05,0.1,0.5,1\}$，Preferred-FT 的随机种子）。扫参共 22 次运行。每训练 100 步直至收敛后，在一组测试提示上评估各策略，计算真实奖励函数下的平均奖励，以及相对参考策略的平均序列级 KL[^2] $\text{KL}\left(\pi\mid\mid\pi_{\text{ref}}\right)$。我们发现 DPO 产出远最高效的前沿：在保持低 KL 的同时达到最高奖励。该结果因多重原因尤为值得注意。第一，DPO 与 PPO 优化同一目标，但 DPO 明显更高效；DPO 的奖励/KL 权衡严格支配 PPO。第二，即便 PPO 可访问真实奖励（PPO-GT），DPO 仍达到更好的前沿。

6.2 Can DPO scale to real preference datasets?

Next, we evaluate fine-tuning performance of DPO on summarization and single-turn dialogue. For summarization, automatic evaluation metrics such as ROUGE can be poorly correlated with human preferences [40], and prior work has found that fine-tuning LMs using PPO on human preferences to provide more effective summaries. We evaluate different methods by sampling completions on the test split of TL;DR summarization dataset, and computing the average win rate against reference completions in the test set. The completions for all methods are sampled at temperatures varying from 0.0 to 1.0, and the win rates are shown in Figure 2 (right). DPO, PPO and Preferred-FT all fine-tune the same GPT-J SFT model[^3]. We find that DPO has a win rate of approximately 61% at a temperature of 0.0, exceeding the performance of PPO at  57% at its optimal sampling temperature of 0.0. DPO also achieves a higher maximum win rate compared to the best of $N$ baseline. We note that we did not meaningfully tune DPO’s $\beta$ hyperparameter, so these results may underestimate DPO’s potential. Moreover, we find DPO to be much more robust to the sampling temperature than PPO, the performance of which can degrade to that of the base GPT-J model at high temperatures. Preferred-FT does not improve significantly over the SFT model. We also compare DPO and PPO head-to-head in human evaluations in Section 6.4, where DPO samples at temperature 0.25 were preferred 58% times over PPO samples at temperature 0.

接着评估 DPO 在摘要与单轮对话上的微调表现。对摘要而言，ROUGE 等自动指标可能与人类偏好相关性差 [40]，先前工作发现用 PPO 在人类偏好上微调 LM 能提供更有效摘要。我们在 TL;DR 摘要数据集测试划分上采样补全，计算相对测试集参考补全的平均胜率。所有方法的补全在温度 0.0 到 1.0 下采样，胜率见图 2（右）。DPO、PPO 与 Preferred-FT 都微调同一 GPT-J SFT 模型[^3]。发现温度 0.0 时 DPO 胜率约 61%，超过 PPO 在其最优采样温度 0.0 时约 57% 的表现。DPO 的最大胜率也高于 Best of $N$ 基线。需注意我们未有意义地调节 DPO 的 $\beta$ 超参，因此结果可能低估 DPO 潜力。此外，DPO 对采样温度远比 PPO 稳健，后者在高温下可能退化到接近基础 GPT-J 的水平。Preferred-FT 相对 SFT 模型无显著提升。第 6.4 节还在人类评估中头对头比较 DPO 与 PPO：温度 0.25 的 DPO 样本相对温度 0 的 PPO 样本胜率 58%。

On single-turn dialogue, we evaluate the different methods on the subset of the test split of the Anthropic HH dataset [1] with one step of human-assistant interaction. GPT-4 evaluations use the preferred completions on the test as the reference to compute the win rate for different methods. As there is no standard SFT model for this task, we start with a pre-trained Pythia-2.8B, use Preferred-FT to train a reference model on the chosen completions such that completions are within distribution of the model, and then train using DPO. We also compare against the best of 128 Preferred-FT completions (we found the Best of $N$ baseline plateaus at 128 completions for this task; see Appendix Figure 4) and a 2-shot prompted version of the Pythia-2.8B base model, finding DPO performs as well or better for the best-performing temperatures for each method. We also evaluate an RLHF model trained with PPO on the Anthropic HH dataset [^4] from a well-known source [^5], but are unable to find a prompt or sampling temperature that gives performance better than the base Pythia-2.8B model. Based on our results from TL;DR and the fact that both methods optimize the same reward function, we consider Best of 128 a rough proxy for PPO-level performance. Overall, DPO is the only computationally efficient method that improves over the preferred completions in the Anthropic HH dataset, and provides similar or better performance to the computationally demanding Best of 128 baseline. Finally, Figure 3 shows that DPO converges to its best performance relatively quickly.

在单轮对话上，我们在 Anthropic HH 数据集 [1] 测试划分中仅含一步人类–助手交互的子集上评估各方法。GPT-4 评估以测试集偏好补全为参考计算各方法胜率。该任务无标准 SFT 模型，故从预训练 Pythia-2.8B 出发，用 Preferred-FT 在所选补全上训练参考模型，使补全落在模型分布内，再用 DPO 训练。还与 Best of 128 Preferred-FT 补全比较（发现该任务上 Best of $N$ 在 128 条后趋于平台；见附录图 4），以及 Pythia-2.8B 基座的 2-shot 提示版本；在各方法最佳温度下，DPO 表现相当或更好。我们也评估了来自知名来源[^5]、在 Anthropic HH 上用 PPO 训练的 RLHF 模型[^4]，但未能找到优于基座 Pythia-2.8B 的提示或采样温度。基于 TL;DR 结果以及两方法优化同一奖励函数，我们将 Best of 128 视为 PPO 级表现的粗略代理。总体而言，DPO 是唯一相对 Anthropic HH 数据集偏好补全有提升且计算高效的方法，并提供与计算昂贵的 Best of 128 基线相似或更好的表现。最后，图 3 显示 DPO 相对较快收敛到最佳表现。

6.3 Generalization to a new input distribution

	Win rate vs. ground truth
Alg.	Temp $0$	Temp $0.25$
DPO	0.36	0.31
PPO	0.26	0.23

Table 1: GPT-4 win rates vs. ground truth summaries for out-of-distribution CNN/DailyMail input articles.

	相对真实摘要胜率
算法	温度 $0$	温度 $0.25$
DPO	0.36	0.31
PPO	0.26	0.23

表 1：分布外 CNN/DailyMail 输入文章上相对真实摘要的 GPT-4 胜率。

To further compare the performance of PPO and DPO under distribution shifts, we evaluate the PPO and DPO policies from our Reddit TL;DR summarization experiment on a different distribution, news articles in the test split of the CNN/DailyMail dataset [26], using the best sampling temperatures from TL;DR (0 and 0.25). The results are presented in Table 1. We computed the GPT-4 win rate against the ground-truth summaries in the datasets, using the same GPT-4 (C) prompt we used for Reddit TL;DR, but replacing the words “forum post” with “news article”. For this new distribution, DPO continues to outperform the PPO policy by a significant margin. This experiment provides initial evidence that DPO policies can generalize similarly well to PPO policies, even though DPO does not use the additional unlabeled Reddit TL;DR prompts that PPO uses.

为进一步比较分布偏移下 PPO 与 DPO 的表现，我们把 Reddit TL;DR 摘要实验中的 PPO 与 DPO 策略放到另一分布——CNN/DailyMail 数据集 [26] 测试划分中的新闻文章——上评估，采用 TL;DR 上最佳采样温度（0 与 0.25）。结果见表 1。我们计算相对数据集真实摘要的 GPT-4 胜率，使用与 Reddit TL;DR 相同的 GPT-4（C）提示，但把「论坛帖」换成「新闻文章」。在该新分布上，DPO 继续以显著优势超过 PPO 策略。本实验提供初步证据：即便 DPO 未使用 PPO 所用的额外未标注 Reddit TL;DR 提示，DPO 策略仍可与 PPO 策略类似地泛化。

6.4 Validating GPT-4 judgments with human judgments

We conduct a human study to verify the reliability of GPT-4’s judgments, using the results of the TL;DR summarization experiment and two different GPT-4 prompts. The GPT-4 (S) (simple) prompt simply asks for which summary better-summarizes the important information in the post. The GPT-4 (C) (concise) prompt also asks for which summary is more concise; we evaluate this prompt because we find that GPT-4 prefers longer, more repetitive summaries than humans do with the GPT-4 (S) prompt. See Appendix C.2 for the complete prompts. We perform three comparisons, using the highest (DPO, temp. 0.25), the lowest (PPO, temp. 1.0), and a

我们开展人研究以验证 GPT-4 判断的可靠性，使用 TL;DR 摘要实验结果与两种 GPT-4 提示。GPT-4（S）（简单）提示只问哪条摘要更好总结了帖子中的重要信息。GPT-4（C）（简洁）提示还要求哪条摘要更简洁；评估该提示是因为我们发现用 GPT-4（S） 时，GPT-4 比人类更偏好更长、更重复的摘要。完整提示见附录 C.2。我们做三组比较，覆盖最高（DPO，温度 0.25）、最低（PPO，温度 1.0）与

	DPO	SFT	PPO-1
N respondents	272	122	199
GPT-4 (S) win %	47	27	13
GPT-4 (C) win %	54	32	12
Human win %	58	43	17
GPT-4 (S)-H agree	70	77	86
GPT-4 (C)-H agree	67	79	85
H-H agree	65	-	87

Table 2: Comparing human and GPT-4 win rates and per-judgment agreement on TL;DR summarization samples. Humans agree with GPT-4 about as much as they agree with each other. Each experiment compares a summary from the stated method with a summary from PPO with temperature 0.

	DPO	SFT	PPO-1
应答人数	272	122	199
GPT-4（S）胜率 %	47	27	13
GPT-4（C）胜率 %	54	32	12
人类胜率 %	58	43	17
GPT-4（S）–人类一致	70	77	86
GPT-4（C）–人类一致	67	79	85
人类–人类一致	65	-	87

表 2：在 TL;DR 摘要样本上比较人类与 GPT-4 胜率及逐条判断一致率。人类与 GPT-4 的一致程度，约等于人类彼此之间的一致程度。每组实验将所述方法的摘要与温度 0 的 PPO 摘要比较。

middle-performing (SFT, temp. 0.25) method with the aim of covering a diversity of sample qualities; all three methods are compared against greedily-sampled PPO (its best-performing temperature). We find that with both prompts, GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for human evaluations (due to limited human raters, we only collect multiple human judgments for the DPO and PPO-1 comparisons). Overall, the GPT-4 (C) prompt generally provides win rates more representative of humans; we therefore use this prompt for the main results in Section 6.2. For additional details about the human study, including the web interface presented to raters and the list of human volunteers, see Appendix D.3.

中等表现（SFT，温度 0.25）方法，旨在覆盖多样本质量；三者均与贪心采样的 PPO（其最佳温度）比较。发现两种提示下，GPT-4 与人类的一致频率通常与人类互评相当，表明 GPT-4 是合理的人类评估代理（因人类标注者有限，仅对 DPO 与 PPO-1 比较收集多人判断）。总体而言，GPT-4（C） 提示给出的胜率通常更代表人类；因此第 6.2 节主结果使用该提示。人研究更多细节（含展示给标注者的网页界面与志愿者名单）见附录 D.3。

7 Discussion

Learning from preferences is a powerful, scalable framework for training capable, aligned language models. We have introduced DPO, a simple training paradigm for training language models from preferences without reinforcement learning. Rather than coercing the preference learning problem into a standard RL setting in order to use off-the-shelf RL algorithms, DPO identifies a mapping between language model policies and reward functions that enables training a language model to satisfy human preferences directly, with a simple cross-entropy loss, without reinforcement learning or loss of generality. With virtually no tuning of hyperparameters, DPO performs similarly or better than existing RLHF algorithms, including those based on PPO; DPO thus meaningfully reduces the barrier to training more language models from human preferences.

从偏好学习是训练有能力、对齐的语言模型的强大可扩展框架。我们提出 DPO：无需强化学习、从偏好训练语言模型的简单范式。不是把偏好学习硬塞进标准 RL 设定以便套用现成 RL 算法，DPO 找出语言模型策略与奖励函数之间的映射，从而可用简单交叉熵损失直接训练语言模型以满足人类偏好，既不做强化学习，也不损失一般性。几乎无需调超参，DPO 表现与包括基于 PPO 的方法在内的现有 RLHF 算法相当或更好；因而切实降低了从人类偏好训练更多语言模型的门槛。

Limitations & Future Work. Our results raise several important questions for future work. How does the DPO policy generalize out of distribution, compared with learning from an explicit reward function? Our initial results suggest that DPO policies can generalize similarly to PPO-based models, but more comprehensive study is needed. For example, can training with self-labeling from the DPO policy similarly make effective use of unlabeled prompts? On another front, how does reward over-optimization manifest in the direct preference optimization setting, and is the slight decrease in performance in Figure 3-right an instance of it? Additionally, while we evaluate models up to 6B parameters, exploration of scaling DPO to state-of-the-art models orders of magnitude larger is an exciting direction for future work. Regarding evaluations, we find that the win rates computed by GPT-4 are impacted by the prompt; future work may study the best way to elicit high-quality judgments from automated systems. Finally, many possible applications of DPO exist beyond training language models from human preferences, including training generative models in other modalities.

局限与未来工作。 我们的结果提出若干重要后续问题。相对从显式奖励函数学习，DPO 策略在分布外如何泛化？初步结果提示 DPO 策略可与基于 PPO 的模型类似地泛化，但仍需更全面研究。例如，用 DPO 策略自标注能否同样有效利用未标注提示？另一方面，奖励过优化在直接偏好优化设定中如何表现，图 3 右轻微性能下降是否其实例？此外，我们评估至多 6B 参数的模型，把 DPO 扩展到量级大得多的前沿模型是令人兴奋的方向。评估方面，发现 GPT-4 计算的胜率受提示影响；未来工作可研究如何最好地从自动系统引出高质量判断。最后，DPO 在从人类偏好训练语言模型之外还有许多可能应用，包括其他模态的生成模型训练。

Acknowledgements

EM gratefully acknowledges funding from a Knight-Hennessy Graduate Fellowship. CF and CM are CIFAR Fellows. This work was supported in part by the Stanford Accelerator for Learning (SAL) and Stanford Institute for Human-Centered Artificial Intelligence (HAI) Generative AI for the Future of Learning seed grant program. The Stanford Center for Research on Foundation Models (CRFM) provided part of the compute resources used for the experiments in this work. This work was supported in part by ONR grant N00014-20-1-2675.

EM 感谢 Knight-Hennessy 研究生奖学金资助。CF 与 CM 为 CIFAR Fellows。本工作部分得到 Stanford Accelerator for Learning（SAL）与 Stanford Institute for Human-Centered Artificial Intelligence（HAI）Generative AI for the Future of Learning 种子资助计划支持。Stanford Center for Research on Foundation Models（CRFM）为本工作实验提供部分算力。本工作部分得到 ONR 拨款 N00014-20-1-2675 支持。

References

[1] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022a.

[2] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b.

[3] S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O’Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.

[4] H. Bong and A. Rinaldo. Generalized results for the existence and consistency of the MLE in the Bradley-Terry-Luce model. International Conference on Machine Learning , 2022. arXiv:2110.11487.

[5] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324–345, 1952. doi: https://doi.org/10.2307/2334029 .

[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877–1901. Curran Associates, Inc., 2020a. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .

[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020b.

[8] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. arXiv preprint arXiv:2303.12712.

[9] R. Busa-Fekete, B. Szörényi, P. Weng, W. Cheng, and E. Hüllermeier. Preference-based reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine Learning , 97(3):327–351, July 2014. doi: 10.1007/s10994-014-5458-8 . URL https://doi.org/10.1007/s10994-014-5458-8 .

[10] Y. Chen, R. Wang, H. Jiang, S. Shi, and R.-L. Xu. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. ArXiv , abs/2304.00723, 2023.

[11] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

[12] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/d5e2c0adad503c91f91df240d0cd4e49-Paper.pdf .

[13] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling instruction-finetuned language models, 2022.

[14] M. Dudík, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi. Contextual dueling bandits. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pages 563–587, Paris, France, 03–06 Jul 2015. PMLR. URL https://proceedings.mlr.press/v40/Dudik15.html .

[15] D. Go, T. Korbak, G. Kruszewski, J. Rozen, N. Ryu, and M. Dymetman. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning , ICML’23. JMLR.org, 2023.

[16] A. Jain, B. Wojcik, T. Joachims, and A. Saxena. Learning trajectory preferences for manipulators via iterative improvement. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/file/c058f544c737782deacefa532d9add4c-Paper.pdf .

[17] N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning , pages 1645–1654. PMLR, 2017.

[18] N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. S. Gu, and R. Picard. Human-centric dialog training via offline reinforcement learning. arXiv preprint arXiv:2010.05848 , 2020.

[19] T. Korbak, H. Elsahar, G. Kruszewski, and M. Dymetman. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 16203–16220. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/67496dfa96afddab795530cc7c69b57a-Paper-Conference.pdf .

[20] J. Kreutzer, J. Uyheng, and S. Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1777–1788, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1165 . URL https://aclanthology.org/P18-1165 .

[21] A. Kupcsik, D. Hsu, and W. S. Lee. Learning Dynamic Robot-to-Human Object Handover from Human Feedback , pages 161–176. Springer International Publishing, 01 2018. ISBN 978-3-319-51531-1. doi: 10.1007/978-3-319-51532-8_10 .

[22] S. Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018.

[23] R. D. Luce. Individual choice behavior: A theoretical analysis. Courier Corporation , 2012.

[24] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/P11-1015 .

[25] S. Mishra, D. Khashabi, C. Baral, and H. Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3470–3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.244 . URL https://aclanthology.org/2022.acl-long.244 .

[26] R. Nallapati, B. Zhou, C. dos Santos, Ç. Gulçehre, and B. Xiang. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning , pages 280–290, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. doi: 10.18653/v1/K16-1028 . URL https://aclanthology.org/K16-1028 .

[27] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , SC ’21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384421. doi: 10.1145/3458817.3476209 . URL https://doi.org/10.1145/3458817.3476209 .

[28] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 27730–27744. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf .

[29] R. Paulus, C. Xiong, and R. Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=HkAClQgA- .

[30] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

[31] J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning , pages 745–750, 2007.

[32] R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 24(2):193–202, 1975. doi: https://doi.org/10.2307/2346567 .

[33] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners, 2019. Ms., OpenAI.

[34] R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=8aHzds2uUyB .

[35] M. Ranzato, S. Chopra, M. Auli, and W. Zaremba. Sequence level training with recurrent neural networks. CoRR , abs/1511.06732, 2015.

[36] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems (RSS) , 2017.

[37] A. Saha, A. Pacchiano, and J. Lee. Dueling rl: Reinforcement learning with trajectory preferences. In F. Ruiz, J. Dy, and J.-W. van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 6263–6289. PMLR, 25–27 Apr 2023. URL https://proceedings.mlr.press/v206/saha23a.html .

[38] V. Sanh, A. Webson, C. Raffel, S. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. S. Bari, C. Xu, U. Thakker, S. S. Sharma, E. Szczechla, T. Kim, G. Chhablani, N. Nayak, D. Datta, J. Chang, M. T.-J. Jiang, H. Wang, M. Manica, S. Shen, Z. X. Yong, H. Pandey, R. Bawden, T. Wang, T. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. A. Fries, R. Teehan, T. L. Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=9Vrb9D0WI4 .

[39] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.

[40] N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning to summarize from human feedback, 2022.

[41] R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou, C.-C. Chang, I. Krivokon, W. Rusch, M. Pickett, P. Srinivasan, L. Man, K. Meier-Hellstern, M. R. Morris, T. Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera-Arcas, C. Cui, M. Croak, E. Chi, and Q. Le. Lamda: Language models for dialog applications, 2022.

[42] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

[43] M. Völske, M. Potthast, S. Syed, and B. Stein. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization , pages 59–63, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4508 . URL https://aclanthology.org/W17-4508 .

[44] L. von Werra, J. Tow, reciprocated, S. Matiana, A. Havrilla, cat state, L. Castricato, Alan, D. V. Phung, A. Thakur, A. Bukhtiyarov, aaronrmm, F. Milo, Daniel, D. King, D. Shin, E. Kim, J. Wei, M. Romero, N. Pochinkov, O. Sanseviero, R. Adithyan, S. Siu, T. Simonini, V. Blagojevic, X. Song, Z. Witten, alexandremuzio, and crumb. CarperAI/trlx: v0.6.0: LLaMa (Alpaca), Benchmark Util, T5 ILQL, Tests, Mar. 2023. URL https://doi.org/10.5281/zenodo.7790115 .

[45] B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax , May 2021.

[46] S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319 , 2019.

[47] R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. , 8(3–4):229–256, may 1992. ISSN 0885-6125. doi: 10.1007/BF00992696 . URL https://doi.org/10.1007/BF00992696 .

[48] Y. Wu and B. Hu. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018. ISBN 978-1-57735-800-8.

[49] X. Yan, C. Luo, C. L. A. Clarke, N. Craswell, E. M. Voorhees, and P. Castells. Human preferences as dueling bandits. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR ’22, page 567–577, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323. doi: 10.1145/3477495.3531991 . URL https://doi.org/10.1145/3477495.3531991 .

[50] Y. Yue, J. Broder, R. Kleinberg, and T. Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences , 78(5):1538–1556, 2012. ISSN 0022-0000. doi: https://doi.org/10.1016/j.jcss.2011.12.028 . URL https://www.sciencedirect.com/science/article/pii/S0022000012000281 . JCSS Special Issue: Cloud Computing 2011.

[51] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences, 2020.

Author Contributions

All authors provided valuable contributions to designing, analyzing, and iterating on experiments, writing and editing the paper, and generally managing the project’s progress.

全体作者在设计、分析与迭代实验，撰写与编辑论文，以及整体推进项目进度方面均作出宝贵贡献。

RR proposed using autoregressive reward models in discussions with EM; derived the DPO objective; proved the theoretical properties of the algorithm and wrote the relevant sections and appendices. He also suggested and helped with organizing experiments and contributed some of the PPO and reward learning baselines.

RR 在与 EM 的讨论中提出使用自回归奖励模型；导出 DPO 目标；证明算法理论性质并撰写相关章节与附录。他还建议并协助组织实验，贡献部分 PPO 与奖励学习基线。

AS initiated the discussion on using weighted regression methods as an alternative to PPO; initiated project-related organization, wrote initial analysis connecting DPO with weighted regression and unlikelihood; design and iterations of DPO + baseline implementations, initial exploratory experiments for DPO; substantial experiment organization and design (datasets, baselines, evaluation); led model training and evaluation for controlled sentiment generation and summarization; design iterations for GPT-4 evaluation (particularly summarization); substantial writing contributions to abstract, prelims/method and experiments; editing contributions to other sections.

AS 发起用加权回归方法作为 PPO 替代的讨论；发起项目相关组织工作，撰写连接 DPO 与加权回归及反似然的初步分析；设计并迭代 DPO 与基线实现、DPO 的初步探索实验；大量实验组织与设计（数据集、基线、评估）；主导受控情感生成与摘要的模型训练与评估；GPT-4 评估的设计迭代（尤其摘要）；对摘要、预备知识/方法与实验有大量写作贡献；对其他章节有编辑贡献。

EM provided input on early discussions on learning autoregressive reward functions; wrote the first implementation of DPO and ran the first DPO experiments; trained the large-scale (summarization and dialogue) DPO models used in paper experiments; conducted initial GPT-4 win rate evaluations and set up related infrastructure; recruited participants for, conducted, and analyzed results from the human study; wrote the abstract, introduction, related work, discussion, and most of experiments; and assisted with editing the rest of the paper.

EM 在学习自回归奖励函数的早期讨论中提供意见；写出 DPO 的首个实现并跑出首批 DPO 实验；训练论文实验所用的大规模（摘要与对话）DPO 模型；开展初始 GPT-4 胜率评估并搭建相关基础设施；招募人研究参与者、实施并分析结果；撰写摘要、引言、相关工作、讨论及实验的大部分；并协助编辑其余部分。

CF, CM, & SE supervised the research, suggested ideas and experiments, and assisted in writing the paper.

CF、CM 与 SE 指导研究，提出想法与实验，并协助撰写论文。

Appendix A Mathematical Derivations

A.1 Deriving the Optimum of the KL-Constrained Reward Maximization Objective

In this appendix, we will derive Eq. 4. Analogously to Eq. 3, we optimize the following objective:

本附录导出式 4。类比式 3，我们优化如下目标：

$$\max_{\pi}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi}\bigl[r(x,y)\bigr]-\beta\mathbb{D}_{\textrm{KL}}\bigl[\pi(y|x)||\pi_{\text{ref}}(y|x)\bigr] \tag{11}$$

under any reward function $r(x,y)$, reference model $\pi_{\text{ref}}$ and a general non-parametric policy class. We now have:

其中奖励函数 $r(x,y)$、参考模型 $\pi_{\text{ref}}$ 与一般非参数策略类任意。于是有：

$$\displaystyle\max_{\pi}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi} \\ \displaystyle\bigl[r(x,y)\bigr]-\beta\mathbb{D}_{\textrm{KL}}\bigl[\pi(y|x)\mid\mid\pi_{\text{ref}}(y|x)\bigr] \\ \displaystyle=\max_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi(y|x)}\left[r(x,y)-\beta\log\frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}\right] \\ \displaystyle=\min_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi(y|x)}\left[\log\frac{\pi(y|x)}{\pi_{\text{ref}}(y|x)}-\frac{1}{\beta}r(x,y)\right] \\ \displaystyle=\min_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{y\sim\pi(y|x)}\left[\log\frac{\pi(y|x)}{\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right)}-\log Z(x)\right] \tag{12}$$

where we have partition function:

其中配分函数为：

$$Z(x)=\sum_{y}\pi_{\text{ref}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right).$$

Note that the partition function is a function of only $x$ and the reference policy $\pi_{\text{ref}}$, but does not depend on the policy $\pi$. We can now define

注意配分函数只是 $x$ 与参考策略 $\pi_{\text{ref}}$ 的函数，不依赖策略 $\pi$。现可定义

$$\pi^{*}(y|x)=\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right),$$

which is a valid probability distribution as $\pi^{*}(y|x)\geq 0$ for all $y$ and $\sum_{y}\pi^{*}(y|x)=1$. Since $Z(x)$ is not a function of $y$, we can then re-organize the final objective in Eq 12 as:

这是合法概率分布，因为对所有 $y$ 有 $\pi^{*}(y|x)\geq 0$ 且 $\sum_{y}\pi^{*}(y|x)=1$。由于 $Z(x)$ 不是 $y$ 的函数，可将式 12 的最终目标重排为：

$$\displaystyle\min_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{y\sim\pi(y|x)}\left[\log\frac{\pi(y|x)}{\pi^{*}(y|x)}\right]-\log Z(x)\right]= \\ \displaystyle\min_{\pi}\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{D}_{\text{KL}}(\pi(y|x)\mid\mid\pi^{*}(y|x))-\log Z(x)\right] \tag{13}$$

Now, since $Z(x)$ does not depend on $\pi$, the minimum is achieved by the policy that minimizes the first KL term. Gibbs’ inequality tells us that the KL-divergence is minimized at 0 if and only if the two distributions are identical. Hence we have the optimal solution:

由于 $Z(x)$ 不依赖 $\pi$，最小值由最小化第一个 KL 项的策略达到。Gibbs 不等式告诉我们：当且仅当两分布相同，KL 散度取最小值 0。因此最优解为：

$$\pi(y|x)=\pi^{*}(y|x)=\frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\left(\frac{1}{\beta}r(x,y)\right) \tag{15}$$

for all $x\in\mathcal{D}$. This completes the derivation.

对所有 $x\in\mathcal{D}$ 成立。推导完毕。

A.2 Deriving the DPO Objective Under the Bradley-Terry Model

It is straightforward to derive the DPO objective under the Bradley-Terry preference model as we have

在 Bradley-Terry 偏好模型下导出 DPO 目标是直接的，因为我们有

$$p^{*}(y_{1}\succ y_{2}|x)=\frac{\exp\left(r^{*}(x,y_{1})\right)}{\exp\left(r^{*}(x,y_{1})\right)+\exp\left(r^{*}(x,y_{2})\right)} \tag{16}$$

In Section 4 we showed that we can express the (unavailable) ground-truth reward through its corresponding optimal policy:

第 4 节已说明，可通过对应最优策略表达（不可得的）真实奖励：

$$r^{*}(x,y)=\beta\log\frac{\pi^{*}(y|x)}{\pi_{\text{ref}}(y|x)}+\beta\log Z(x) \tag{17}$$

Substituting Eq. 17 into Eq. 16 we obtain:

把式 17 代入式 16 得到：

$$\displaystyle p^{*}(y_{1}\succ y_{2}|x) \\ \displaystyle=\frac{\exp\left(\beta\log\frac{\pi^{*}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}+\beta\log Z(x)\right)}{\exp\left(\beta\log\frac{\pi^{*}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}+\beta\log Z(x)\right)+\exp\left(\beta\log\frac{\pi^{*}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}+\beta\log Z(x)\right)} \\ \displaystyle=\frac{1}{1+\exp\left(\beta\log\frac{\pi^{*}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}-\beta\log\frac{\pi^{*}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}\right)} \\ \displaystyle=\sigma\left(\beta\log\frac{\pi^{*}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}-\beta\log\frac{\pi^{*}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}\right).$$

The last line is the per-instance loss in Equation 7.

最后一行即式 7 中的逐例损失。

A.3 Deriving the DPO Objective Under the Plackett-Luce Model

The Plackett-Luce model [32, 23] is a generalization of the Bradley-Terry model over rankings (rather than just pair-wise comparisons). Similar to to the Bradley-Terry model, it stipulates that when presented with a set of possible choices, people prefer a choice with probability proportional to the value of some latent reward function for that choice. In our context, when presented with a prompt $x$ and a set of $K$ answers $y_{1},\ldots,y_{K}$ a user would output a permutation $\tau:[K]\to[K]$, giving their ranking of the answers. The Plackett-Luce model stipulates that

Plackett-Luce 模型 [32, 23] 是 Bradley-Terry 模型在排序（而非仅成对比较）上的推广。与 Bradley-Terry 类似，它规定：面对一组可能选择时，人们偏好某选择的概率与该选择的某潜在奖励函数值成比例。在本语境中，面对提示 $x$ 与 $K$ 个答案 $y_{1},\ldots,y_{K}$ 时，用户会输出排列 $\tau:[K]\to[K]$，给出其对答案的排序。Plackett-Luce 模型规定

$$p^{*}(\tau|y_{1},\ldots,y_{K},x)=\prod_{k=1}^{K}\frac{\exp(r^{*}(x,y_{\tau(k)}))}{\sum_{j=k}^{K}\exp(r^{*}(x,y_{\tau(j)}))} \tag{18}$$

Notice that when $K=2$, Equation 18 reduces to the Bradley-Terry model. However, for the general Plackett-Luce model, we can still utilize the results of Eq. 5 and substitute the reward function parameterized by its optimal policy. Similarly to Appendix A.2, the normalization constant $Z(x)$ cancels out and we’re left with:

注意当 $K=2$ 时，式 18 退化为 Bradley-Terry 模型。但对一般 Plackett-Luce 模型，仍可利用式 5 的结果，代入由最优策略参数化的奖励函数。与附录 A.2 类似，归一化常数 $Z(x)$ 消去，得到：

$$p^{*}(\tau|y_{1},\ldots,y_{K},x)=\prod_{k=1}^{K}\frac{\exp\left(\beta\log\frac{\pi^{*}(y_{\tau(k)}|x)}{\pi_{\text{ref}}(y_{\tau(k)}|x)}\right)}{\sum_{j=k}^{K}\exp\left(\beta\log\frac{\pi^{*}(y_{\tau(j)}|x)}{\pi_{\text{ref}}(y_{\tau(j)}|x)}\right)} \tag{19}$$

Similarly to the approach of Section 4, if we have access to a dataset $\mathcal{D}=\{\tau^{(i)},y_{1}^{(i)},\ldots,y_{K}^{(i)},x^{(i)}\}_{i=1}^{N}$ of prompts and user-specified rankings, we can use a parameterized model and optimize this objective with maximum-likelihood.:

与第 4 节方法类似，若可访问提示与用户指定排序的数据集 $\mathcal{D}=\{\tau^{(i)},y_{1}^{(i)},\ldots,y_{K}^{(i)},x^{(i)}\}_{i=1}^{N}$，可用参数化模型并以极大似然优化该目标：

$$\mathcal{L}_{\text{DPO}}(\pi_{\theta},\pi_{\text{ref}})=-\mathbb{E}_{\tau,y_{1},\ldots,y_{K},x\sim\mathcal{D}}\left[\log\prod_{k=1}^{K}\frac{\exp\left(\beta\log\frac{\pi_{\theta}(y_{\tau(k)}|x)}{\pi_{\text{ref}}(y_{\tau(k)}|x)}\right)}{\sum_{j=k}^{K}\exp\left(\beta\log\frac{\pi_{\theta}(y_{\tau(j)}|x)}{\pi_{\text{ref}}(y_{\tau(j)}|x)}\right)}\right] \tag{20}$$

A.4 Deriving the Gradient of the DPO Objective

In this section we derive the gradient of the DPO objective:

本节导出 DPO 目标的梯度：

$$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\nabla_{\theta}\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}-\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}\right)\right] \tag{21}$$

We can rewrite the RHS of Equation 21 as

可将式 21 右侧改写为

$$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\frac{\sigma^{\prime}\left(u\right)}{\sigma\left(u\right)}\nabla_{\theta}\left(u\right)\right], \tag{22}$$

where $u=\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}-\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}$.

其中 $u=\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}-\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}$。

Using the properties of sigmoid function $\sigma^{\prime}(x)=\sigma(x)(1-\sigma(x))$ and $\sigma(-x)=1-\sigma(x)$, we obtain the final gradient

利用 sigmoid 性质 $\sigma^{\prime}(x)=\sigma(x)(1-\sigma(x))$ 与 $\sigma(-x)=1-\sigma(x)$，得到最终梯度

$$\nabla_{\theta}\mathcal{L}{\text{DPO}}(\pi{\theta};\pi_{\text{ref}})=\
-\mathbb{E}{(x,y{w},y_{l})\sim\mathcal{D}}\bigg[\beta\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)\bigg[\nabla_{\theta}\log\pi(y_{w}\mid x)-\nabla_{\theta}\log\pi(y_{l}\mid x)\bigg]\bigg],$$

After using the reward substitution of $\hat{r}_{\theta}(x,y)=\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ we obtain the final form of the gradient from Section 4.

再用奖励替换 $\hat{r}_{\theta}(x,y)=\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$，即得第 4 节梯度的最终形式。

A.5 Proof of Lemma 1 and 2

In this section, we will prove the two lemmas from Section 5.

本节证明第 5 节的两条引理。

**Lemma 1 Restated.**Under the Plackett-Luce preference framework, and in particular the Bradley-Terry framework, two reward functions from the same equivalence class induce the same preference distribution.

引理 1（重述）。在 Plackett-Luce 偏好框架、特别是 Bradley-Terry 框架下，同一等价类中的两个奖励函数诱导相同的偏好分布。

Proof.

We say that two reward functions $r(x,y)$ and $r^{\prime}(x,y)$ are from the same equivalence class if $r^{\prime}(x,y)=r(x,y)+f(x)$ for some function $f$. We consider the general Plackett-Luce (with the Bradley-Terry model a special case for $K=2$) and denote the probability distribution over rankings induced by a particular reward function $r(x,y)$ as $p_{r}$. For any prompt $x$, answers $y_{1},\ldots,y_{K}$ and ranking $\tau$ we have:

证明。

称两个奖励函数 $r(x,y)$ 与 $r^{\prime}(x,y)$ 属于同一等价类，若存在函数 $f$ 使 $r^{\prime}(x,y)=r(x,y)+f(x)$。我们考虑一般 Plackett-Luce（Bradley-Terry 为 $K=2$ 的特例），并把特定奖励函数 $r(x,y)$ 诱导的排序概率分布记为 $p_{r}$。对任意提示 $x$、答案 $y_{1},\ldots,y_{K}$ 与排序 $\tau$ 有：

$$\displaystyle p_{r^{\prime}}(\tau|y_{1},\ldots,y_{K},x) \\ \displaystyle=\prod_{k=1}^{K}\frac{\exp(r^{\prime}(x,y_{\tau(k)}))}{\sum_{j=k}^{K}\exp(r^{\prime}(x,y_{\tau(j)}))} \\ \displaystyle=\prod_{k=1}^{K}\frac{\exp(r(x,y_{\tau(k)})+f(x))}{\sum_{j=k}^{K}\exp(r(x,y_{\tau(j)})+f(x))} \\ \displaystyle=\prod_{k=1}^{K}\frac{\exp(f(x))\exp(r(x,y_{\tau(k)}))}{\exp(f(x))\sum_{j=k}^{K}\exp(r(x,y_{\tau(j)}))} \\ \displaystyle=\prod_{k=1}^{K}\frac{\exp(r(x,y_{\tau(k)}))}{\sum_{j=k}^{K}\exp(r(x,y_{\tau(j)}))} \\ \displaystyle=p_{r}(\tau|y_{1},\ldots,y_{K},x),$$