Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps
Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps
通过遗忘推理步骤来衡量思维链忠实性
Technion - Israel Institute of Technology, University of Utah
martin.tutek@gmail.com
{fateme.hashemi, ana.marasovic}@utah.edu
belinkov@technion.ac.il
Abstract
摘要
When prompted to think step-by-step, language models (LMs) produce a chain of thought (CoT), a sequence of reasoning steps that the model supposedly used to produce its prediction. Despite much work on CoT prompting it is unclear if reasoning verbalized in a CoT is faithful to the models’ parameteric beliefs. We introduce a framework for measuring parametric faithfulness of generated reasoning, and propose Faithfulness by Unlearning Reasoning steps (FUR), an instance of this framework. FUR erases information contained in reasoning steps from model parameters, and measures faithfulness as the resulting effect of the model’s prediction. Our experiments with four LMs and five multi-hop multi-choice question answering (MCQA) datasets show that FUR is frequently able to precisely change the underlying models’ prediction for a given instance by unlearning key steps, indicating when a CoT is parametrically faithful. Further analysis shows that CoTs generated by models post-unlearning support different answers, hinting at a deeper effect of unlearning.
当被提示要逐步思考时,语言模型会生成思维链(CoT),也就是一串据称被模型用来得出预测的推理步骤。尽管关于 CoT 提示的研究已经很多,口头说出来的推理是否忠实于模型的参数化信念,仍然不清楚。我们提出一套衡量生成推理参数忠实性的框架,并给出该框架的一个实例:通过遗忘推理步骤衡量忠实性(FUR)。FUR 从模型参数中抹去推理步骤所包含的信息,再用这一干预对模型预测的影响来衡量忠实性。我们在四个语言模型和五个多跳多项选择问答(MCQA)数据集上的实验表明,通过遗忘关键步骤,FUR 常常能够精确改变模型对该实例的预测,从而标明一条 CoT 何时在参数上是忠实的。进一步分析还显示,遗忘之后模型新生成的 CoT 会支持不同的答案,暗示遗忘的影响更深层。
Code available at https://github.com/technion-cs-nlp/parametric-faithfulness.
1 Introduction
1 引言
Figure 1: An illustration of PFF and FUR. In order to produce a parameter intervention, we first prompt the model to produce an answer and reasoning chain (CoT). We then segment the reasoning chain and unlearn content tokens from a single reasoning step from the model. The unlearned model is then prompted to produce an answer. We measure faithfulness as the adverse effect of unlearning onto the models’ initial prediction.
图 1:PFF 与 FUR 的示意。为了做参数干预,我们先提示模型生成答案和推理链(CoT),再把推理链切成步骤,并从模型中遗忘单个推理步骤里的内容词。随后再提示被遗忘后的模型给出答案。我们把遗忘对模型初始预测的不利影响,作为忠实性的度量。
Language models (LMs) can perform various tasks accurately and verbalize some reasoning via a so-called chain of thought (CoT) (Kojima et al., 2022; Wei et al., 2022), even without specialized supervised training. CoT reasoning is emerging as a powerful technique for improving the performance of LMs in complex tasks (OpenAI, 2024; DeepSeek-AI et al., 2025). It is not clear, however, whether the reasoning encoded in the CoT is a faithful representation of the internal reasoning process of the model, casting doubts about the reliability of CoT as a window onto the model’s ‘thought process’.
语言模型(LM)能在多种任务上给出正确结果,并通过所谓的思维链(CoT)把一部分推理说出来(Kojima et al., 2022; Wei et al., 2022),即使没有专门的监督训练也是如此。CoT 推理正在成为提高语言模型在复杂任务上表现的有力手段(OpenAI, 2024; DeepSeek-AI et al., 2025)。不过,编码在 CoT 里的推理,是否忠实地代表了模型内部的推理过程,并不清楚,这让人怀疑:把 CoT 当作观察模型“思考过程”的窗口,究竟是否可靠。
Various works set out to explore CoT faithfulness by perturbing tokens within the CoT and observing whether the contextual corruptions affect model prediction (Lanham et al., 2023; Bentham et al., 2024; Chen et al., 2024b; Madsen et al., 2024). This setup is inherently imprecise, as erasing steps from context does not remove knowledge from parameters, and the model may still be able to reconstruct corrupted information when generating a prediction. Such approaches of context perturbation actually measure self-consistency or contextual faithfulness rather than parametric faithfulness, for which one would need to erase knowledge from parameters (Parcalabescu and Frank, 2024).
已有不少工作通过扰动 CoT 中的词元,观察上下文损坏会不会影响模型预测,来探究 CoT 忠实性(Lanham et al., 2023; Bentham et al., 2024; Chen et al., 2024b; Madsen et al., 2024)。这种设定本质上并不精确:从上下文里删去步骤,并没有把知识从参数中去掉,模型在生成预测时仍可能把被破坏的信息重构回来。这类上下文扰动,实际衡量的是自一致性或上下文忠实性,而不是参数忠实性;后者需要从参数中抹去知识(Parcalabescu and Frank, 2024)。
We begin by introducing the Parametric Faithfulness Framework (PFF), a novel approach to measuring faithfulness of verbalized reasoning. We define necessary components of instances of such a framework in two stages: (1) an intervention on the model parameters, which aims to erase information in the CoT from model parameters; and (2) evaluating parametric faithfulness, i.e. whether the intervention affected the models’ prediction. See components in Figure 1. PFF is a general framework that can be instantiated with various interventions and applied to different types of CoT and other free-text explanations.
我们先提出参数忠实性框架(PFF),用来衡量口头推理的忠实性。这一框架的实例需要两阶段的必要组件:(1)对模型参数做干预,目的是把 CoT 中的信息从参数里抹掉;(2)评估参数忠实性,也就是看干预是否影响了模型的预测。各组件见图 1。PFF 是一个通用框架,可以用不同的干预来实例化,并适用于不同类型的 CoT 以及其他自由文本解释。
In this work, we propose an instance of PFF we call Faithfulness by Unlearning Reasoning steps (FUR), a machine unlearning-based (Cao and Yang, 2015) approach to assessing CoT faithfulness. We use NPO (Zhang et al., 2024b), a preference-optimization-based unlearning method for PFF stage 1, the intervention on the model. We propose two metrics of quantifying faithfulness of reasoning steps: FF-HARD quantifies whether the CoT as a whole is faithful, while FF-SOFT identifies the most salient reasoning steps within the CoT. Concretely, we (a) generate a CoT, (b) segment it into steps, (c) independently unlearn knowledge encoded within each step from model parameters and (d) measure the effect of erased knowledge on the models’ prediction (Figure 1). If the target step was successfully and precisely unlearned, and the models’ prediction changed, the step faithfully explains the models’ underlying reasoning process.
在本文中,我们提出 PFF 的一个实例,称为通过遗忘推理步骤衡量忠实性(FUR),这是一种基于机器遗忘(Cao and Yang, 2015)的 CoT 忠实性评估方法。在 PFF 的第一阶段,也就是对模型的干预,我们使用基于偏好优化的遗忘方法 NPO(Zhang et al., 2024b)。我们还提出两种量化推理步骤忠实性的指标:FF-HARD 衡量整条 CoT 是否忠实,FF-SOFT 则用来找出 CoT 中最关键的推理步骤。具体做法是:(a)生成一条 CoT,(b)把它切成步骤,(c)把每一步所编码的知识从模型参数中独立遗忘,(d)衡量被抹去的知识对模型预测的影响(图 1)。如果目标步骤被成功且精确地遗忘,并且模型的预测发生了改变,那么这一步就忠实地解释了模型底层的推理过程。
Through experimental evaluation on four LMs and five MCQA multi-hop reasoning datasets, we show we are able to perform valid interventions that affect model predictions while retaining models’ general capabilities. In subsequent analyses we show unlearning has a profound effect on the model, modifying the answer supported by verbalized reasoning post-unlearning. We also compare parametric faithfulness to plausibility via a human study, finding that humans do not consider steps identified as important by FUR plausible. This finding highlights a need for specialized alignment to obtain CoTs that are both plausible and faithful.
我们在四个语言模型和五个多跳 MCQA 推理数据集上实验,表明可以做出有效的干预:它们能够影响模型预测,同时又保住模型的通用能力。后续分析显示,遗忘对模型的影响很深,会改变遗忘之后口头推理所支持的答案。我们还通过一项人类研究,把参数忠实性与合理性加以比较,发现人类并不认为被 FUR 标为重要的步骤是合理的。这一发现说明,若要得到既合理又忠实的 CoT,还需要专门的对齐。
The contributions of this work are as follows:
本文的贡献如下:
- We introduce PFF, a framework for measuring parametric faithfulness of LM reasoning.
- 我们提出 PFF,一套衡量语言模型推理参数忠实性的框架。
- We instantiate PFF with FUR using NPO, a model unlearning method, and demonstrate its effectiveness on unlearning fine-grained reasoning steps.
- 我们用模型遗忘方法 NPO 把 PFF 实例化为 FUR,并表明它能有效遗忘细粒度的推理步骤。
- We introduce FF-HARD and FF-SOFT, metrics evaluating reasoning faithfulness, which can be applied to full chains or individual steps.
- 我们提出 FF-HARD 和 FF-SOFT 两种衡量推理忠实性的指标,既可以用于整条推理链,也可以用于单个步骤。
- We perform detailed analyses, including human and LLM-as-a-judge annotations, evaluating whether unlearning fundamentally changes the verbalized reasoning, and if steps identified as faithful are also plausible.
- 我们做了细致分析,包括人类标注和以大模型为评委的标注,考查遗忘是否从根本上改变了口头推理,以及被识别为忠实的步骤是否同时也是合理的。
2 Background and Related Work
2 背景与相关工作
When CoT prompted, models exhibit better performance on complex multi-hop and arithmetic reasoning tasks (Zhou et al., 2023; Fu et al., 2023b; Sprague et al., 2025) compared to being prompted directly (no-CoT). Chains of thought can be used as additional context where models can store results of intermediate hops, but they also provide additional compute irrespective of content (Pfau et al., 2024; Biran et al., 2024). Verbalized reasoning steps are frequently hypothesized to be an accurate depiction of the models’ internal reasoning process (Kojima et al., 2022; Fu et al., 2023a; Sun et al., 2023). However, faithfulness of CoTs should not be assumed despite how plausible they might seem (Jacovi and Goldberg, 2020; Bao et al., 2025).
在使用 CoT 提示时,模型在复杂的多跳和算术推理任务上,表现要好于直接提示(no-CoT)(Zhou et al., 2023; Fu et al., 2023b; Sprague et al., 2025)。思维链既可以作为额外上下文,让模型存放中间跳的结果,也能提供与内容无关的额外计算(Pfau et al., 2024; Biran et al., 2024)。口头推理步骤常常被假定为模型内部推理过程的准确刻画(Kojima et al., 2022; Fu et al., 2023a; Sun et al., 2023)。然而,无论 CoT 看起来多么合理,其忠实性都不应当被假定(Jacovi and Goldberg, 2020; Bao et al., 2025)。
Issues with CoTs.
CoT 的问题。
Natural language explanations such as CoTs exhibit a number of issues. They are frequently unreliable, yielding inconsistent answers after supposedly inconsequential perturbations (Camburu et al., 2020; Lanham et al., 2023; Madsen et al., 2024; Sedova et al., 2024). CoTs have been shown to not align with generated answers (Bao et al., 2025), they are often not useful to humans (Joshi et al., 2023) and can contain factually incorrect or hallucinated information (Kim et al., 2021; Kim et al., 2023; Zheng et al., 2023b; Peng et al., 2023; Zhang et al., 2024a). Most importantly, CoTs can obfuscate the true reasoning process of the LM (Turpin et al., 2023; Roger and Greenblatt, 2023).
自然语言解释,包括 CoT,存在不少问题。它们常常不可靠:在看起来无关紧要的扰动之后,就会给出不一致的答案(Camburu et al., 2020; Lanham et al., 2023; Madsen et al., 2024; Sedova et al., 2024)。已有研究显示,CoT 与生成的答案并不对齐(Bao et al., 2025),对人类往往也没有帮助(Joshi et al., 2023),还可能包含事实错误或幻觉信息(Kim et al., 2021; Kim et al., 2023; Zheng et al., 2023b; Peng et al., 2023; Zhang et al., 2024a)。最重要的是,CoT 可能把语言模型真正的推理过程掩盖起来(Turpin et al., 2023; Roger and Greenblatt, 2023)。
Contextual vs. Parameteric Influence.
上下文影响与参数影响。
Prior work has recognized the discord between contextual and parametric influence on the outputs of LMs (Neeman et al., 2023; Bao et al., 2025). Prompting models with hypothetical or factually incorrect information causes them to change their otherwise consistently correct predictions (Kim et al., 2021; Kim et al., 2023; Simhi et al., 2024; Minder et al., 2025), highlighting their high sensitivity to context tokens and confounding any conclusions drawn from contextual perturbations applied to reasoning steps. The main issue with work investigating self-consistency is the possibility of the LM reconstructing information obfuscated by the contextual perturbation—despite the verbalized knowledge missing, this reasoning could still be retrieved from the latent space (Yang et al., 2024; Deng et al., 2024). To account such confounders, we only use information from generated CoTs to guide unlearning, while we generate predictions directly without CoTs, thus disentangling contextual influence from the prediction.
先前工作已经注意到,上下文和参数对语言模型输出的影响并不一致(Neeman et al., 2023; Bao et al., 2025)。若用假设性或事实错误的信息去提示模型,它们往往会改变本来一贯正确的预测(Kim et al., 2021; Kim et al., 2023; Simhi et al., 2024; Minder et al., 2025),说明模型对上下文词元非常敏感,这会干扰任何从对推理步骤做上下文扰动中得出的结论。自一致性研究的主要问题在于:语言模型仍可能把被上下文扰动掩盖的信息重构出来——口头知识虽然缺失,这类推理仍可能从潜在空间中被取回(Yang et al., 2024; Deng et al., 2024)。为了控制这类混杂,我们只用生成的 CoT 来指导遗忘,而预测则直接生成、不再使用 CoT,从而把上下文影响从预测中剥离出去。
Measuring Faithfulness.
衡量忠实性。
Various tests and metrics for quantifying faithfulness of free-text explanations in LMs have previously been proposed (Lanham et al., 2023; Bentham et al., 2024; Atanasova et al., 2023; Siegel et al., 2024). By measuring properties such as sufficiency through simulatability or counterfactual interventions (Atanasova et al., 2023; Lanham et al., 2023), these studies quantify susceptibility of the models’ predictions to changes in context or input. Such approaches are valid only if there is no direct causal link between the input and prediction that bypasses the explanation, which is rare in LMs (Bao et al., 2025). In our work, we analyze whether parametric perturbations that affect the generated CoT also affect the prediction. The closest to ours are the contemporaneous works of Yeo et al. (2024) who use activation patching to measure causal effect of corrupting certain hidden states, and Zaman and Srivastava (2025) who use knowledge editing to evaluate existing (un)faithfulness metrics.
已有多种测试和指标,用来量化语言模型自由文本解释的忠实性(Lanham et al., 2023; Bentham et al., 2024; Atanasova et al., 2023; Siegel et al., 2024)。这些研究通过可模拟性或反事实干预来衡量充分性等性质(Atanasova et al., 2023; Lanham et al., 2023),从而量化模型预测对上下文或输入变化的敏感程度。这类方法只有在输入与预测之间不存在绕过解释的直接因果路径时才成立,而在语言模型里这种情况很少见(Bao et al., 2025)。在我们的工作中,我们分析的是:影响生成 CoT 的参数扰动,是否同样影响预测。与我们最接近的同期工作,包括 Yeo et al.(2024)用激活修补来衡量损坏某些隐状态的因果效应,以及 Zaman and Srivastava(2025)用知识编辑来评估已有的(不)忠实性指标。
Background on Machine Unlearning.
机器遗忘的背景。
Machine unlearning aims to remove some and only some undesired knowledge or behavior so as not to be regurgitated by LMs (Cao and Yang, 2015; Harding et al., 2019; Ippolito et al., 2023). There are multiple approaches to unlearning for LMs, overviewed in Geng et al. (2025) and Appendix A. They typically reduce the capability of the underlying LM on target data, while retaining performance on retain data and general capabilities. In this paper, we unlearn reasoning steps by finetuning using the negative preference optimization (NPO) loss on the forget data (Zhang et al., 2024b) that discourages the preference for forget sequences. We add it to the KL divergence between the original and “unlearned” model’s predictions on the retain set (Chen and Yang, 2023; Yao et al., 2024). We chose NPO+KL as it can be applied to unstructured text and outperforms alternatives. More details in §4.1.
机器遗忘的目标是去掉一部分、并且只去掉那一部分不受欢迎的知识或行为,使语言模型不再把它们复述出来(Cao and Yang, 2015; Harding et al., 2019; Ippolito et al., 2023)。面向语言模型的遗忘有多种做法,Geng et al.(2025)和附录 A 做了概述。这些方法通常降低底层模型在目标数据上的能力,同时保持保留数据和通用能力上的表现。本文中,我们通过在遗忘数据上用负面偏好优化(NPO)损失做微调(Zhang et al., 2024b),来遗忘推理步骤;该损失抑制模型对遗忘序列的偏好。我们再把它与原模型与“已遗忘”模型在保留集上预测之间的 KL 散度相加(Chen and Yang, 2023; Yao et al., 2024)。我们选择 NPO+KL,是因为它可以用于非结构化文本,并且优于其他替代方案。更多细节见第 4.1 节。
3 PFF: A Framework for Measuring Parametric Faithfulness
3 PFF:衡量参数忠实性的框架
We introduce a framework for measuring the faithfulness of generated reasoning, which we call parametric faithfulness. This framework supports multiple ways to measure parametric faithfulness, and in §4, we propose one such way.
我们提出一套衡量生成推理忠实性的框架,称为参数忠实性。该框架支持多种衡量参数忠实性的方式;在第 4 节中,我们提出其中一种。
图 2:PFF 两个阶段的概览:(1)参数干预;(2)评估。我们用 NPO+KL 把 PFF 实例化为 FUR,并用对照组来保证遗忘的精确性,再配合忠实性指标。
Motivation.
动机。
A line of work has analyzed the sensitivity of models to perturbations applied to reasoning steps (Lanham et al., 2023; Bentham et al., 2024; Chen et al., 2024b; Madsen et al., 2024, inter alia) under the guise of faithfulness. While perturbations applied to generated reasoning remove information from context, the model could still retrieve such information from its parameters (Neeman et al., 2023). Perturbing the reasoning chain while maintaining model parameters fixed measures self-consistency (Parcalabescu and Frank, 2024). Self-consistency can be viewed as faithfulness of the model output with respect to the reasoning chain (contextual faithfulness), but it does not reflect faithfulness of the reasoning chain with respect to model parameters, which we call parametric faithfulness. Between the two, parametric faithfulness provides stronger guarantees. Models could recover information erased only from context, and introduced mistakes might make the model prioritize erroneous context. While these confounders need not always dictate the models’ output, in contextual faithfulness they can never be explained away without quantifying the effect of parameters. In other words, to measure parametric faithfulness, we have to intervene on parameters.
已有一系列工作,在忠实性的名义下,分析模型对推理步骤扰动的敏感程度(Lanham et al., 2023; Bentham et al., 2024; Chen et al., 2024b; Madsen et al., 2024, inter alia)。对生成推理施加扰动,虽然从上下文中去掉了信息,模型仍可能从自己的参数中把这些信息取回来(Neeman et al., 2023)。在保持模型参数不变的前提下扰动推理链,衡量的是自一致性(Parcalabescu and Frank, 2024)。自一致性可以看作模型输出相对于推理链的忠实性(上下文忠实性),但它并不反映推理链相对于模型参数的忠实性,也就是我们所说的参数忠实性。二者之中,参数忠实性提供更强的保证。只从上下文中抹去的信息,模型仍可能恢复;引入的错误还可能让模型更优先采信错误的上下文。这些混杂因素未必总会主导模型输出,但在上下文忠实性里,如果不量化参数的作用,它们就永远无法被排除。换句话说,要衡量参数忠实性,就必须对参数做干预。
Framework.
框架。
The proposed framework involves two multi-step stages: (1) performing a valid reasoning-based intervention on the model’s parameters, and (2) evaluating parametric faithfulness. We outline our framework in Figure 2.
所提出的框架包含两个多步骤阶段:(1)对模型参数做一次有效的、基于推理的干预;(2)评估参数忠实性。框架概览见图 2。
The first stage begins by instructing the model to generate reasoning, which we will evaluate for faithfulness. The reasoning is broken into reasoning steps of a chosen granularity. Each individual reasoning step is used to guide an intervention on ’s parameters, targeting those where a step’s information is stored. This produces a modified model, . Moving to the next stage makes sense only if the intervention is successful. Thus, our framework requires defining and implementing controls that verify that the change in behavior between and stems from the intended intervention rather than extraneous factors.
第一阶段先指示模型 生成推理,以便评估其忠实性。再把推理按选定的粒度切成推理步骤。每一个推理步骤都用来指导对 参数的干预,目标是该步骤信息所存放的那些参数。由此得到修改后的模型 。只有当干预成功时,进入下一阶段才有意义。因此,我们的框架要求定义并实现对照,用来验证 与 之间的行为变化来自预定的干预,而不是其他因素。
In the second stage, faithfulness is assessed with at least one of two evaluation protocols: (1) Instruct both and to directly give answers, then compute how often and how strongly their answers differ; (2) Instruct and to reason-then-answer, then compute how often they present different reasoning. In both cases, the more faithful the reasoning is to internal computations, the greater the difference in answers and reasoning between and should be.
在第二阶段,至少用两种评估协议中的一种来衡量忠实性:(1)让 和 都直接给答案,再计算它们的答案有多频繁、以及在多大程度上不同;(2)让 和 先推理再作答,再计算它们给出不同推理的频率。无论哪种情形,推理对内部计算越忠实, 与 在答案和推理上的差异就应当越大。
The first protocol uses direct answers rather than those obtained via CoT prompting because the reasoning steps are expected to change after the intervention. This shift in context makes it unclear whether changes in the answers come from the intended effect of the intervention or from the new reasoning context post-intervention. When comparing direct answers, we hypothesize that if the model generates the same answer using direct and CoT prompting, then the internal reasoning of the model is also the same.
第一种协议使用直接答案,而不是通过 CoT 提示得到的答案,因为干预之后推理步骤预期会改变。上下文一变,答案的变化究竟来自干预的预定效果,还是来自干预后新的推理上下文,就说不清了。在比较直接答案时,我们的假设是:如果模型用直接提示和 CoT 提示给出相同答案,那么模型的内部推理也是相同的。
4 FUR: Unlearning Reasoning Steps
4 FUR:遗忘推理步骤
We instantiate the parametric faithfulness framework (§3) by specifying its three elements: unlearning reasoning steps as the parameter intervention method (§4.1), controls to assess unlearning validity (§4.2), and faithfulness measurements (§4.3).
我们通过指定三个要素,来实例化参数忠实性框架(第 3 节):把遗忘推理步骤作为参数干预方法(第 4.1 节),用对照评估遗忘是否有效(第 4.2 节),以及忠实性度量(第 4.3 节)。
4.1 Parameter Intervention
4.1 参数干预
The idea behind unlearning reasoning steps as the intervention is that once the information contained in generated reasoning is successfully erased from the model ’s parameters, its modified version should not produce the same predictions or reasoning that did if that reasoning is indeed associated with ’s internal computations.
把遗忘推理步骤当作干预,其想法是:一旦生成推理中所含的信息被成功从模型 的参数中抹去,如果这些推理确实与 的内部计算相关,那么修改后的 就不应当再给出与 相同的预测或推理。
We erase knowledge contained in the verbalized reasoning steps using a preference-optimization unlearning method, NPO (Zhang et al., 2024b). Specifically, the KL-regularized variant of it, which also minimizes the divergence between the base and unlearned model outputs on a retain set to preserve fluency. We provide a detailed overview in Appendix A for readers unfamiliar with NPO+KL.
我们用基于偏好优化的遗忘方法 NPO(Zhang et al., 2024b),来抹去口头推理步骤中包含的知识。具体使用的是其带 KL 正则的变体:它还会在保留集上最小化原模型与已遗忘模型输出之间的散度,以保持流畅性。若不熟悉 NPO+KL,附录 A 提供了较详细的概述。
We experimented with NPO+grad-diff, but results were always slightly worse than NPO+KL. We explored ROME and MEMIT (Meng et al., 2022; Meng et al., 2023), but they require a structured format, and do not perform well under paraphrases.
NPO+KL requires defining the forget set, , and the retain set, , which we construct as follows. First, we set reasoning steps to be sentences with at least two content words. For each step, we construct its of input-output pairs formed by taking, for each content word in the step, the prefix up to that word as input and the tokens of the content word as output. NPO updates the model’s parameters to discourage it from predicting content words given prior context. We similarly construct for a given step as content words from four randomly selected CoT steps from other instances. Concretely, we sample other instances from the same dataset and randomly select a CoT step with at least two content words from each instance. We then minimize the KL divergence between the outputs of the original model and the model post-unlearning, using content tokens from these steps as the targets. The KL divergence preserves the model’s original completions for these non-target contexts.
NPO+KL 需要定义遗忘集 和保留集 ,我们按如下方式构造。首先,把推理步骤定义为至少包含两个内容词的句子。对每一步,我们为其构造 :对步骤中的每个内容词,把该词之前的前缀当作输入、该内容词的词元当作输出,形成输入–输出对。NPO 更新模型参数,抑制它在给定先前上下文时预测 中的内容词。对给定步骤,我们类似地构造 :从其他实例中随机选出四个 CoT 步骤的内容词。具体来说,我们从同一数据集中抽样 4 个其他实例,并从每个实例中随机选一个至少含两个内容词的 CoT 步骤。然后,以这些步骤中的内容词元为目标,最小化原模型与遗忘后模型输出之间的 KL 散度。KL 散度用来保持模型在这些非目标上下文上的原有补全。
Unlearning tokens beyond content words was detrimental to the model’s fluency in our early exploration.
We unlearn each reasoning step individually, for a total of iterations, and refer to the model obtained after unlearning the -th reasoning step alone as . One unlearning iteration refers to a pass over . We only update the second FF2 matrix of the Transformer MLPs, as this layer was found to act as a memory store (Geva et al., 2021b; Meng et al., 2022) and model editing methods frequently target it to update information (Meng et al., 2022; Meng et al., 2023; Hong et al., 2024). We only vary the learning rate while keeping the remainder of method-specific hyperparameters fixed to values found by original works. We report them in Appendix E.
我们把每个推理步骤单独遗忘,共进行 次迭代,并把只遗忘第 个推理步骤后得到的模型记为 。一次遗忘迭代指对 的一遍遍历。我们只更新 Transformer MLP 中的第二个 FF2 矩阵,因为这一层被认为起着记忆存储的作用(Geva et al., 2021b; Meng et al., 2022),模型编辑方法也常以它为更新信息的目标(Meng et al., 2022; Meng et al., 2023; Hong et al., 2024)。除学习率外,其余方法特定超参数都固定为原论文中的取值。我们在附录 E 中报告这些设置。
4.2 Controls
4.2 对照
Unlearning is deemed successful if the target information is removed (high efficacy), but the model retains its general capabilities, fluency, and performance on non-forgotten in-domain data (high specificity) (Gandikota et al., 2024). We adapt these criteria for unlearning methods within FUR.
如果目标信息被去掉(高效力),同时模型仍保持通用能力、流畅性以及在未被遗忘的域内数据上的表现(高特异性),遗忘就被认为是成功的(Gandikota et al., 2024)。我们把这些标准适配到 FUR 中的遗忘方法上。
Efficacy.
效力。
We measure efficacy of unlearning as the reduction in the length-normalized sequence probability of the unlearned CoT step. Concretely, for a reasoning step , consisting of tokens , the length-normalized probability of that reasoning step with prefix under model is:
我们用被遗忘 CoT 步骤的长度归一化序列概率的下降,来衡量遗忘效力。具体来说,对由 个词元 组成的推理步骤 ,该步骤在前缀 下、由模型 给出的长度归一化概率为:
where consists of the query for the given instance (comprising the question and answer choices) and the previous reasoning steps . Then, efficacy is the normalized difference in reasoning step probabilities of the initial model and the model post-unlearning the i-th step, :
其中 由该实例的查询 (包含问题和选项)以及此前的推理步骤 组成。于是,效力 是初始模型 与遗忘第 步之后的模型 在该推理步骤概率上的归一化差:
Note that when computing , we use the original prefix generated by . Throughout our experiments, we report average efficacy across unlearned steps and instances.
注意,在计算 时,我们使用由 生成的原始前缀 。在全部实验中,我们报告跨被遗忘步骤和实例的平均效力。
Specificity.
特异性。
We measure specificity of unlearning on unrelated, but in-domain data to account for the adverse effect of model unlearning. To this end, we randomly select instances from the same dataset as a held-out set , and measure specificity as the proportion of unchanged labels on this held-out set after unlearning. Therefore, for predicted labels under the initial model and produced by the unlearned model :
为了衡量遗忘的不利影响,我们在无关但同域的数据上评估遗忘的特异性。为此,我们从同一数据集中随机选出 个实例作为留出集 ,并把特异性定义为遗忘后该留出集上标签保持不变的比例。因此,对初始模型 的预测标签 以及遗忘后模型 给出的 ,有:
We choose once and use it to evaluate every unlearned model . Note that this approach might be overly strict as some instances from sometimes require information from the target step, which we unlearn. This effect is noticeable in Sports (§6.2). We leave this consideration for future work.
We compute the specificity score after each iteration of unlearning for the target reasoning step . Unless stated otherwise, we report averages of specificity across unlearning iterations, reasoning steps, and instances.
我们对目标推理步骤 的每一次遗忘迭代都计算特异性分数。除非另有说明,我们报告跨遗忘迭代、推理步骤和实例的平均特异性。
General Capabilities.
通用能力。
In order to measure whether unlearning affects general model capabilities, we compare the performance on MMLU (Hendrycks et al., 2021) before and after unlearning. Due to prohibitive costs of evaluating few-shot MMLU for each instance and unlearned CoT step, we (1) opt for zero-shot evaluation as the instruction-tuned models report good performance in this setup, and (2) MMLU score of the model after unlearning each step from randomly selected CoTs ( unlearning steps).
为了衡量遗忘是否影响模型的通用能力,我们比较遗忘前后在 MMLU(Hendrycks et al., 2021)上的表现。由于对每个实例和每个被遗忘 CoT 步骤都做少样本 MMLU 成本过高,我们(1)改为零样本评估,因为指令微调模型在这一设定下也有较好表现;(2)只在随机选出的 条 CoT 的每一步都被遗忘之后评估 MMLU(大约 次遗忘步骤)。
Remark.
说明。
Note that we do not aim for efficacy to reach , as that would imply that the unlearned step has probability (Eq. 2), which in turn would likely adversely affect the fluency of the model. Rather, we want the original CoT step to become a less likely reasoning pathway, but still a possible sequence of tokens. The core tension between efficacy, specificity, and general capabilities is delicate, and presents one major hurdle in model unlearning.
注意,我们并不追求效力达到 ,因为那意味着被遗忘步骤的概率为 (式 2),这很可能损害模型的流畅性。我们希望的是,原来的 CoT 步骤变成一条更不太可能的推理路径,但仍然是可能的词元序列。效力、特异性和通用能力之间的核心张力很细,也是模型遗忘中的一大难点。
4.3 Faithfulness Measurements
4.3 忠实性度量
We deploy the faithfulness evaluation protocol described in §3, where we prompt and to answer directly, without reasoning, and then compute how often their answers differ. If ’s verbalized reasoning is generally faithful to its internal computations, the answer will change frequently.
我们采用第 3 节所述的忠实性评估协议:提示 和 直接作答、不再推理,然后计算它们的答案有多频繁地不同。如果 的口头推理总体上忠实于其内部计算,答案就会经常改变。
We propose hard and soft versions of estimating faithfulness (ff) of full reasoning chains and segmented steps, respectively. The hard version (FF-HARD) provides a binary answer to whether an explanation is faithful or not, by measuring whether unlearning any step causes the model to output a different label as the most likely one:
我们提出衡量完整推理链以及切分后步骤忠实性(FF)的硬版本和软版本。硬版本(FF-HARD)给出一条解释是否忠实的二元答案:只要遗忘任一步骤会使模型把另一个标签变为最可能的标签,就判定为忠实:
where is the -th reasoning step and the prediction made by (after the -th reasoning step is unlearned). The use-case for FF-HARD is answering the question: Is the reasoning chain produced by the LM faithful?
其中 是第 个推理步骤, 是 (遗忘第 个推理步骤之后)给出的预测。FF-HARD 的用例是回答:语言模型生成的推理链是否忠实?
A single faithful step is sufficient to show that the gist of the model’s internal reasoning is captured by the verbalized reasoning. In this sense, the entire chain can be considered faithful.
The soft version (FF-SOFT) assigns a value to a reasoning step, indicating how much probability mass has unlearning that step shifted from the initial answer.
软版本(FF-SOFT)给一个推理步骤赋予取值 的分数,表示遗忘该步骤把多少概率质量从初始答案上移走。
The use-case for FF-SOFT is answering: Which are the most salient steps of the reasoning chain?
FF-SOFT 的用例是回答:推理链中哪些步骤最为关键?
Perfectly determining whether a reasoning chain constitutes a faithful explanation is difficult. Due to the existence of alternative explanations (Wang et al., 2023), it is possible that a faithful explanation, even when unlearned from model parameters, will not tangibly affect the models’ prediction. Therefore, we do not expect FF-HARD to have perfect recall. However, when an unlearned step notably changes the model’s prediction, without adversely affecting the general capabilities of the model, we can confidently claim that step to be faithful. For the remaining FF-HARD instances, there are three possibilities: (1) FUR failed to uncover and unlearn the true reasoning path, (2) the model used multiple valid reasoning paths, and unlearning one did not significantly affect its prediction, or (3) the model was genuinely unfaithful in its explanation. In this sense, FF-HARD represents a lower bound on the model’s true faithfulness — it is the rate at which we can successfully uncover faithful reasoning (assuming that the flip happened due to a valid intervention).
要完美判断一条推理链是否构成忠实解释,并不容易。由于存在替代解释(Wang et al., 2023),一个忠实的解释即使从模型参数中被遗忘,也不一定会对模型预测产生可感知的影响。因此,我们并不指望 FF-HARD 有完美的召回。不过,当被遗忘的步骤明显改变了模型预测,同时又没有损害模型的通用能力时,我们可以有把握地宣称该步骤是忠实的。对其余 FF-HARD 的实例,有三种可能:(1)FUR 没能找出并遗忘真正的推理路径;(2)模型使用了多条有效推理路径,遗忘其中一条并未显著影响预测;(3)模型的解释本身确实不忠实。在这个意义上,FF-HARD 是模型真实忠实性的下界——它是我们能够成功揭示忠实推理的比率(假定预测翻转来自一次有效干预)。
Table 1: Unlearning results. Efficacy (Eff) is the percentage reduction in the probability of the unlearned CoT step (Eq. 2). Specificity (Spec) is the agreement of with on the held-out set (Eq. 3). General capabilities (Gen) measures accuracy of models on MMLU post-unlearning. The second column shows the base MMLU accuracy of each model. Scores reported are averages across CoTs & all steps (Eff, Spec) or CoTs & all steps (Gen).
| Base | ARC-Challenge | OpenbookQA | Sports | StrategyQA | TruthfulQA | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Gen | Eff | Spec | Gen | Eff | Spec | Gen | Eff | Spec | Gen | Eff | Spec | Gen | Eff | Spec | Gen |
| LLaMA-8B | ||||||||||||||||
| LLaMA-3B | ||||||||||||||||
| Mistral-2 | ||||||||||||||||
| Phi-3 |
表 1:遗忘结果。效力(Eff)是被遗忘 CoT 步骤概率的下降百分比(式 2)。特异性(Spec)是 与 在留出集上的一致率(式 3)。通用能力(Gen)衡量遗忘后模型在 MMLU 上的准确率。第二列给出每个模型的基础 MMLU 准确率。所报分数是 条 CoT 及全部步骤的平均值(Eff、Spec),或 条 CoT 及全部步骤的平均值(Gen)。
| 基础 | ARC-Challenge | OpenbookQA | Sports | StrategyQA | TruthfulQA | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 模型 | Gen | Eff | Spec | Gen | Eff | Spec | Gen | Eff | Spec | Gen | Eff | Spec | Gen | Eff | Spec | Gen |
| LLaMA-8B | ||||||||||||||||
| LLaMA-3B | ||||||||||||||||
| Mistral-2 | ||||||||||||||||
| Phi-3 |
5 Experimental Setup
5 实验设置
We conduct all of our experiments zero-shot on multi-choice question answering (MCQA) datasets.
我们的全部实验都在多项选择问答(MCQA)数据集上以零样本方式进行。
Models.
模型。
We use four representative instruction-tuned models from three families: LLaMA-3-8B-Instruct and Llama-3.2-3B-Instruct (Touvron et al., 2023), Mistral-7B-Instruct-v0.2 (Jiang et al., 2023), and Phi-3-mini-4k-Instruct (Abdin et al., 2024).
我们使用来自三个模型族的四个有代表性的指令微调模型:LLaMA-3-8B-Instruct 和 Llama-3.2-3B-Instruct(Touvron et al., 2023)、Mistral-7B-Instruct-v0.2(Jiang et al., 2023),以及 Phi-3-mini-4k-Instruct(Abdin et al., 2024)。
Datasets.
数据集。
We employ five diverse multi-hop datasets: OpenbookQA (Mihaylov et al., 2018, Book;), ARC-Challenge (Clark et al., 2018, Arc-ch;), StrategyQA (Geva et al., 2021a, SQA;), TruthfulQA (Lin et al., 2022, TQA;) and the Sports understanding subtask of BigBench-Hard (Srivastava et al., 2023). These datasets span a variety of domains, necessitating knowledge of science, sports, geography, health, law, finance and logic. We choose MCQA as the target task as it simplifies analysis of how the models’ predictive distribution shifts after unlearning due to availability of alternative answers. To retain comparable sizes, and due to expensive runtime of unlearning each CoT step, we select a subset of instances from the test split of each dataset to balance the question sources. Details of datasets and models are in Appendix C.
我们采用五个多样的多跳数据集:OpenbookQA(Mihaylov et al., 2018,简称 Book)、ARC-Challenge(Clark et al., 2018,简称 Arc-ch)、StrategyQA(Geva et al., 2021a,简称 SQA)、TruthfulQA(Lin et al., 2022,简称 TQA),以及 BigBench-Hard 中的 Sports understanding 子任务(Srivastava et al., 2023)。这些数据集覆盖科学、体育、地理、健康、法律、金融和逻辑等多个领域。我们选择 MCQA 作为目标任务,是因为它提供了替代答案,便于分析遗忘后模型预测分布如何移动。为了保持规模可比,也由于对每个 CoT 步骤做遗忘的运行时间很贵,我们从每个数据集的测试集中选出 个实例的子集,以平衡问题来源。数据集与模型的细节见附录 C。
For SQA, we use instances from the validation split due to the availability of labels. Sports has a total of instances.
Generating CoTs.
生成 CoT。
We use a two-step prompting approach (Bowman et al., 2022; Lanham et al., 2023), where the model is first prompted to generate the CoT based on the question and answer options, and subsequently prompted to complete the answer letter based on the question, answer choices, and the CoT. We use greedy decoding when generating, producing a single CoT for each instance. For the prompts used, see Appendix D.
我们采用两步提示(Bowman et al., 2022; Lanham et al., 2023):先根据问题和选项提示模型生成 CoT,再根据问题、选项和该 CoT 提示模型补全答案字母。生成时使用贪心解码,每个实例只生成一条 CoT。所用提示见附录 D。
Preprocessing CoTs.
预处理 CoT。
To obtain fine-grained information on faithfulness of individual steps, we segment each CoT into sentences using NLTK (Bird, 2006). When unlearning, we target only tokens that are constituents of content words. We opt for this approach so as to not unlearn the capability to verbalize reasoning from the models, but only knowledge within the steps, which we frequently observed prior to making this modification.
为了获得单个步骤忠实性的细粒度信息,我们用 NLTK(Bird, 2006)把每条 CoT 切成句子。遗忘时,只针对作为内容词组成成分的词元。我们采用这一做法,是为了不把模型口头推理的能力忘掉,而只忘掉步骤中的知识;在做这一修改之前,我们经常观察到前一种情况。
Concretely, we select noun, proper noun, verb, adjective, and number tokens, after running part-of-speech tagging with SpaCy en_core_web_sm (https://spacy.io/).
Table 2: % of instances where adding mistakes or unlearning a reasoning step changes the model’s answer. Measured only on instances where no-CoT and CoT model predictions agree. Scores over higher in bold.
| ARC-challenge | OpenbookQA | Sports | StrategyQA | TruthfulQA | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | FUR | +mistake | FUR | +mistake | FUR | +mistake | FUR | +mistake | FUR | +mistake |
| LLaMA-8B | ||||||||||
| LLaMA-3B | ||||||||||
| Mistral-2 | ||||||||||
| Phi-3 |
表 2:加入错误或遗忘一个推理步骤后,模型答案发生变化的实例百分比。只在无 CoT 与有 CoT 的模型预测一致的实例上衡量。高出 以上的分数用加粗标出。
| ARC-challenge | OpenbookQA | Sports | StrategyQA | TruthfulQA | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 模型 | FUR | +mistake | FUR | +mistake | FUR | +mistake | FUR | +mistake | FUR | +mistake |
| LLaMA-8B | ||||||||||
| LLaMA-3B | ||||||||||
| Mistral-2 | ||||||||||
| Phi-3 |
6 Results
6 结果
We first report results of control measurements validating our intervention (§6.1). Subsequently, we demonstrate the effectiveness of FUR in detecting faithful reasoning (§6.2). We then investigate the effect of unlearning on generated CoTs (§6.3). Finally, we use FUR to identify key reasoning steps and assess their plausibility in a user study (§6.4).
我们先报告用来验证干预有效性的对照测量结果(第 6.1 节)。随后展示 FUR 在检测忠实推理上的效果(第 6.2 节)。接着考察遗忘对生成 CoT 的影响(第 6.3 节)。最后,我们用 FUR 找出关键推理步骤,并在用户研究中评估其合理性(第 6.4 节)。
6.1 Effectiveness of Unlearning
6.1 遗忘的有效性
We report the results of unlearning when using the best hyperparameters for each method and dataset in Table 1. We measure each model’s efficacy, specificity, and MMLU performance before and after unlearning. The specificity and general capabilities of these models are largely unchanged while reporting good efficacy, indicating that the information from the target CoT step has been unlearned without affecting the model adversely. We report the results of various learning rates and discuss methodological choices in Appendix E.
表 1 报告了各方法、各数据集在最佳超参数下的遗忘结果。我们衡量了每个模型在遗忘前后的效力、特异性以及 MMLU 表现。这些模型的特异性和通用能力大体保持不变,同时效力较好,说明目标 CoT 步骤中的信息已被遗忘,且没有对模型造成不利影响。不同学习率的结果以及方法选择的讨论见附录 E。
6.2 Does Unlearning Change Predictions?
6.2 遗忘是否会改变预测?
In the previous section, we show that we can precisely unlearn information encoded in a reasoning step. We now focus on how frequently unlearning information from reasoning steps causes the model predictions to change through FF-HARD (Eq. 4), and contrast our method to Add-mistake, a contextual faithfulness method (Lanham et al., 2023). Add-mistake prompts another LM (in our case, gpt-4o-mini-2024-07-18) to introduce a mistake to a single step of a CoT generated by the target model. The target model is then prompted with a perturbed CoT containing the mistake. If the prediction of the model changes, the CoT is considered faithful. We report the results of instance-level faithfulness for FUR and Add-mistake in Table 2. We compare to this popular baseline, as ground-truth faithfulness is not available and cannot be directly labeled since the model’s internal process is unknown. Thus, we analyze whether FUR reveals different trends compared to an existing metric.
上一节表明,我们可以精确地遗忘编码在一个推理步骤中的信息。现在我们用 FF-HARD(式 4)关注:遗忘推理步骤中的信息会使模型预测改变的频率有多高,并把我们的方法与 Add-mistake 这一上下文忠实性方法(Lanham et al., 2023)对照。Add-mistake 提示另一个语言模型(本文中为 gpt-4o-mini-2024-07-18)在目标模型生成的 CoT 的某一步中引入一个错误,再用含有该错误的扰动 CoT 去提示目标模型。若模型预测改变,该 CoT 就被认为是忠实的。我们在表 2 中报告 FUR 与 Add-mistake 的实例级忠实性结果。我们与这一常用基线比较,是因为不存在可直接标注的忠实性真值——模型的内部过程是未知的。因此,我们分析 FUR 所揭示的趋势是否不同于已有指标。
We explore and comment other baselines in Appendix B.
Both methods report reasonably high percentages of changing predictions across all models and datasets, but in general parametric faithfulness through FUR identifies a larger proportion of faithful CoTs than contextual faithfulness. This result suggests that contextual faithfulness may underestimate CoT faithfulness. Notably, Sports, the dataset where Add-mistake reports strong results, has a high degree of knowledge overlap between instances. This causes the specificity scores (Eq. 3) to sometimes decrease even if the intervention is precise, and a more precise specificity criterion would likely yield better parametric faithfulness.
两种方法在全部模型和数据集上都报告了相当高的预测改变比例,但总体而言,通过 FUR 衡量的参数忠实性,比上下文忠实性识别出的忠实 CoT 比例更高。这一结果说明,上下文忠实性可能低估了 CoT 的忠实性。值得注意的是,Add-mistake 表现较强的 Sports 数据集,实例之间知识重叠程度很高。这会使得即使干预本身精确,特异性分数(式 3)有时仍会下降;若采用更精确的特异性标准,参数忠实性可能会更高。
We find that unlearning efficacy is highly indicative of faithfulness. The Pearson correlation between average efficacy and FF-HARD is high: with . We interpret this as indication that reasoning chains generated by the models are generally faithful, as the stronger we unlearn, the more frequent the change in prediction. The limiting factor is that stronger unlearning damages model integrity. Nevertheless, development of more precise unlearning techniques will remove this limitation. We discuss this further, along with step-level faithfulness Appendix I.
我们发现,遗忘效力对忠实性有很强的指示作用。平均效力与 FF-HARD 之间的 Pearson 相关系数很高:为 ,。我们把这解释为:模型生成的推理链总体上是忠实的,因为遗忘越强,预测改变就越频繁。限制因素在于,更强的遗忘会损害模型完整性。不过,更精确的遗忘技术一旦出现,这一限制就会被消除。我们在附录 I 中连同步骤级忠实性一并讨论这一点。
Figure 3: A sample result of unlearning applied to a CoT step generated by LLaMA-3-8B on an instance from OpenbookQA. The bar charts represent no-CoT probability assigned to each answer option in that unlearning iteration. Model CoTs pre- and post-unlearning are displayed below. We omit CoTs from other unlearning iterations for space as they change very little after the 2nd iteration. One step is slightly shortened for presentation purposes.
图 3:将遗忘应用于 LLaMA-3-8B 在 OpenbookQA 一个实例上生成的某个 CoT 步骤的示例结果。柱状图表示该次遗忘迭代中、无 CoT 设定下各答案选项的概率。图下方展示遗忘前后的模型 CoT。其他遗忘迭代的 CoT 因篇幅省略,因为它们在第 2 次迭代之后变化很小。有一步为便于展示略作缩短。
Table 3: LLM-as-a-judge results assessing if CoTs support different answers after unlearning. Numbers are percentages of how frequently GPT-4o states that the CoT supports a different answer post-unlearning.
| Model | Arc-ch | Book | Sports | SQA | TQA |
|---|---|---|---|---|---|
| LLaMA-8B | |||||
| LLaMA-3B | |||||
| Mistral-2 | |||||
| Phi-3 |
表 3:用大模型充当评判者,评估遗忘后的 CoT 是否支持不同答案。数字是 GPT-4o 判定遗忘后 CoT 支持不同答案的百分比。
| 模型 | Arc-ch | Book | Sports | SQA | TQA |
|---|---|---|---|---|---|
| LLaMA-8B | |||||
| LLaMA-3B | |||||
| Mistral-2 | |||||
| Phi-3 |
6.3 Does Unlearning Change Reasoning?
6.3 遗忘是否会改变推理?
Thus far, we focused on one of the two PFF faithfulness measurement protocols, where we directly prompt models pre- and post-unlearning. In this section we analyze the other protocol by examining whether reasoning within CoTs also changes post-unlearning. To illustrate this, Figure 3 visualizes how prediction probabilities of the no-CoT-prompted model change through unlearning iterations, along with the CoTs of the unlearned model. ‘Base’ refers to the model pre-unlearning. We see that even after a single unlearning iteration, all of the probability mass is reassigned from the initial prediction onto two alternatives. The CoT follows the prediction of the no-CoT model, now arguing against the initial prediction post-unlearning.
到目前为止,我们关注的是 PFF 两种忠实性衡量协议中的一种:在遗忘前后直接提示模型。本节分析另一种协议,考察 CoT 中的推理在遗忘后是否也会改变。为说明这一点,图 3 可视化了无 CoT 提示下模型的预测概率如何随遗忘迭代变化,并给出已遗忘模型的 CoT。“Base”指遗忘前的模型。可以看到,即使只做一次遗忘迭代,全部概率质量就从初始预测转移到两个替代选项上。CoT 跟随无 CoT 模型的预测,在遗忘后转而论证反对最初的预测。
To quantitatively assess how frequently the verbalized reasoning of the model changes post-unlearning, we employ an LLM-as-a-judge (Zheng et al., 2023a) to verify if unlearning caused the generated CoT to support a different answer, indicating deeper unlearning, or if the change in model prediction is not reflected in reasoning (Cohen et al., 2024). We first select instances where CoT and no-CoT models agree in their changed predictions. From these cases, we select reasoning steps from the last iteration of unlearning. We prompt gpt-4o-mini-2024-07-18 to judge whether the CoTs generated by the model before and after unlearning support different answers. We report results in Table 3 and detail our setup in Appendix H.
为了定量评估模型口头推理在遗忘后改变的频率,我们采用大模型充当评判者(Zheng et al., 2023a),检验遗忘是否使生成的 CoT 转而支持另一个答案——这表示更深层的遗忘——还是模型预测的变化并未反映在推理中(Cohen et al., 2024)。我们先选出 CoT 与无 CoT 模型在改变后的预测上一致的实例,再从这些情形中取出最后一次遗忘迭代的推理步骤,并提示 gpt-4o-mini-2024-07-18 判断模型在遗忘前后生成的 CoT 是否支持不同答案。结果见表 3,设置细节见附录 H。
Overall, post-unlearning CoTs largely support different answers compared to the base LM, indicating the unlearning-based intervention fundamentally changes the models’ verbalized reasoning.
总体来看,遗忘后的 CoT 大多支持与基础语言模型不同的答案,说明基于遗忘的干预从根本上改变了模型的口头推理。
图 4:通过遗忘推理步骤得到的热图。 表示 FF-SOFT:初始答案概率的变化。正变化表示概率从初始预测上被去掉,负变化表示概率被加到初始预测上。
6.4 Quantifying Step Level Faithfulness
6.4 量化步骤级忠实性
In this section, we showcase how FF-SOFT (Eq. 5) can be used to identify which reasoning steps in a given instance contribute the most toward the prediction. For one example in Figure 4 we plot heatmaps for each reasoning step, which indicate how much probability mass has been shifted to (red) or from (green) the models’ initial answer when that step was unlearned. We can see in the example that steps that verbalize background information (1, 3) and directly state the models’ prediction (4) decrease the probability that the model assigns to its initial prediction, while unlearning the background step (2) actually increases probability of the initial answer.
本节展示如何用 FF-SOFT(式 5)找出给定实例中哪些推理步骤对预测贡献最大。在图 4 的一个例子中,我们为每个推理步骤画出热图,表示当该步骤被遗忘时,有多少概率质量被移向(红色)或移离(绿色)模型的初始答案。从该例可以看到,口头给出背景信息的步骤(1、3)以及直接陈述模型预测的步骤(4),会降低模型赋给初始预测的概率;而遗忘背景步骤(2)反而提高了初始答案的概率。
To quantitatively assess whether FF-SOFT identifies plausible steps as relevant, we conduct a user study on a random sample of instances. We show each participant a question, answer choices, and CoT steps, highlighting the answer predicted by the model and the target CoT step. We prompt the participants to annotate whether the step in question supports the predicted answer in context of the given CoT on a 1–5 Likert scale (Likert, 1932). We provide more details of the user study, data selection and the protocol in Appendix J.
为了定量评估 FF-SOFT 是否把合理的步骤识别为相关步骤,我们在随机抽样的 个实例上做了用户研究。每位参与者会看到问题、选项和 CoT 步骤,其中模型预测的答案以及目标 CoT 步骤被高亮标出。我们请参与者在 1–5 的 Likert 量表上标注:在给定 CoT 的语境中,该步骤是否支持所预测的答案(Likert, 1932)。用户研究、数据选择和协议的更多细节见附录 J。
We find a weak Pearson correlation of between FF-SOFT and human ratings of supportiveness. This result provides further evidence that faithfulness, in general, does not correlate with plausibility (Agarwal et al., 2024). In order to improve correspondence between these two notions, one might need to specifically align LMs for reasoning plausibility (Ouyang et al., 2022).
我们发现 FF-SOFT 与人类对支持程度的评分之间只有弱的 Pearson 相关,系数为 。这一结果进一步说明,忠实性总体上并不与合理性相关(Agarwal et al., 2024)。若要提高这两种概念之间的对应,或许需要专门针对推理合理性来对齐语言模型(Ouyang et al., 2022)。
7 Discussion and Future Outlook
7 讨论与展望
Recent works have turned to exploring test-time scaling via reasoning language models such as OpenAI o1 (Jaech et al., 2024) and DeepSeek R1 (DeepSeek-AI et al., 2025). Such models are trained to generate comprehensive reasoning chains spanning thousands of tokens, which incurs an additional layer of complexity for applying FUR due to a large number of constituent steps. While a body of work strives to reduce overthinking in CoTs (Jin et al., 2024; Hassid et al., 2025; Xu et al., 2025; Amiri et al., 2025), the problem of length still persists in cases where long CoTs are necessary. The issue here is twofold: (1) fully fine-tuning all second feed-forward layers of the model involves updating a large number of parameters, and (2) intervening on each CoT step can be time-consuming.
近来的工作开始探索用推理语言模型做测试时扩展,例如 OpenAI o1(Jaech et al., 2024)和 DeepSeek R1(DeepSeek-AI et al., 2025)。这类模型被训练去生成长达数千词元的完整推理链,组成步骤很多,给应用 FUR 增加了一层复杂性。虽然已有不少工作致力于减少 CoT 中的过度思考(Jin et al., 2024; Hassid et al., 2025; Xu et al., 2025; Amiri et al., 2025),但在必须使用长 CoT 的情形里,长度问题仍然存在。这里的难点有两方面:(1)对模型全部第二个前馈层做全量微调,需要更新的参数量很大;(2)对每个 CoT 步骤做干预可能很耗时。
图 5:LLaMA 系列模型在 Sports understanding 和 StrategyQA 上的 FF-HARD 分数。LoRA 微调的模型能够赶上、甚至超过全量微调得到的分数。
Reducing time complexity.
降低时间复杂度。
FUR can innately be applied to individual CoT steps in parallel. However, not all CoT steps are equally salient for the prediction. We envision that models similar to verifiers (Lightman et al., 2024; Chen et al., 2024a; Jacovi et al., 2024) can detect, and rank, CoT steps most important for the models’ reasoning (Bogdan et al., 2025, thought anchors;), which can then be prioritized for erasure. Recent work supporting our vision delves into analyzing the importance of, as well as controlling LMs chains-of-thought (Lee et al., 2025; Xiao et al., 2025; Yu et al., 2025). An interesting avenue for further work is to train models ranking CoT step importance on the signal produced by unlearning success, which has the potential to yield further insights into LM internals.
FUR 本来就可以并行地应用于各个 CoT 步骤。不过,并非所有 CoT 步骤对预测都同样关键。我们设想,类似于验证器的模型(Lightman et al., 2024; Chen et al., 2024a; Jacovi et al., 2024)可以检测并排列对模型推理最重要的 CoT 步骤(Bogdan et al., 2025,思想锚点),再优先把这些步骤抹去。支持这一设想的近期工作,则深入分析并控制语言模型的思维链(Lee et al., 2025; Xiao et al., 2025; Yu et al., 2025)。一个有趣的后续方向,是用遗忘成功所产生的信号,训练对 CoT 步骤重要性打分的模型,这有可能进一步揭示语言模型的内部机制。
Reducing space complexity.
降低空间复杂度。
To address the issue of space, we explore using LoRA (Hu et al., 2022) to reduce the number of active parameters. We apply LoRA tuning with ranks , and to the FF2 matrix of two LLaMA models (3B and 8B) and instances from Sports and StrategyQA. As seen in Figure 5, LoRA offers a potential alternative to full fine-tuning. Interestingly, applying FUR to LLaMA-3B changes predictions on the exact same set of instances across all variants, indicating that targeted knowledge might reside in a low rank within that model. We detail the experimental setup and provide additional results in Appendix F.
为应对空间问题,我们探索用 LoRA(Hu et al., 2022)减少有效参数数量。我们对两个 LLaMA 模型(3B 和 8B)的 FF2 矩阵,以及 Sports 和 StrategyQA 上的 个实例,分别用秩为 、 和 的 LoRA 做微调。如图 5 所示,LoRA 有望成为全量微调的替代。有趣的是,把 FUR 应用于 LLaMA-3B 时,所有变体都在完全相同的实例集合上改变了预测,说明目标知识在该模型中可能处于低秩位置。实验设置和更多结果见附录 F。
8 Conclusion
8 结论
We introduced a novel parametric faithfulness framework (PFF) for precisely measuring faithfulness of chains of thought. We instantiated the framework by proposing faithfulness through unlearning reasoning steps (FUR) and introduced two metrics for quantifying faithfulness of CoTs. The hard metric FF-HARD answers the question “Is the CoT generated by the model faithful?”, while the soft metric FF-SOFT answers the question “Which CoT steps are most relevant for the models’ prediction?”. We then conducted detailed qualitative and quantitative analyses confirming the validity of our proposed approach, and demonstrating its benefits compared to perturbation-based contextual faithfulness approaches. We showed that unlearning certain steps causes the model to verbalize a reasoning pathway arguing for a different answer, confirming that the unlearned steps were internally used to generate the prediction. We also found that CoT steps identified as highly relevant are not considered plausible by humans, higlighting the need for specialized alignment.
我们提出了一个新的参数忠实性框架(PFF),用于精确衡量思维链的忠实性。我们通过提出“借助遗忘推理步骤衡量忠实性”(FUR)来实例化该框架,并引入了两种量化 CoT 忠实性的指标。硬指标 FF-HARD 回答“模型生成的 CoT 是否忠实?”,软指标 FF-SOFT 回答“哪些 CoT 步骤对模型预测最为相关?”。随后我们做了细致的定性和定量分析,确认所提方法的有效性,并展示它相对于基于扰动的上下文忠实性方法的优势。我们表明,遗忘某些步骤会使模型口头给出一条论证不同答案的推理路径,从而确认被遗忘的步骤在内部被用来生成预测。我们还发现,被识别为高度相关的 CoT 步骤,在人类看来并不合理,这凸显了专门对齐的必要性。
Limitations
局限
The implementation of our proposed framework has a number of limitations, both in design as well as implementation. By eliminating the contextual confounder, we limit ourselves to studying cases in which the CoT and no-CoT predictions of the models agree — as these are the only cases where one can hypothesize both instances of the model use the same reasoning. This limitation can be bypassed in future work by measuring sensitivity of the CoT prompted model post-unlearning to surface level changes in the CoT, denoting consistency under semantically equivalent context rather than sensitivity to surface level cues.
我们所提框架的实现有若干局限,既有设计上的,也有实现上的。通过消除上下文这一混杂因素,我们只能研究模型的 CoT 预测与无 CoT 预测相一致的情形——因为只有在这些情形里,才能假设模型的两种实例使用相同的推理。这一局限可以在后续工作中绕过:衡量遗忘后、经 CoT 提示的模型对 CoT 表层变化的敏感程度,关注的是语义等价上下文下的一致性,而不是对表层线索的敏感。
Secondly, our approach relies on machine unlearning techniques, which are imperfect. It is possible that either localization of information within parameters or their erasure are imprecise or inefficient for some target reasoning steps. We rely on the rapid development of the field of machine unlearning and model editing to produce better and more precise methods such as CRISP (Ashuach et al., 2025a) and PISCES (Gur-Arieh et al., 2025), which can seamlessly be integrated into our framework. While our method identifies faithful explanations with high precision, its recall cannot be guaranteed due to either unsuccessful unlearning, unfaithful explanation or the existence of alternative explanations. Furthermore, applying machine unlearning requires the capability to fine-tune the target model, which makes FUR not applicable to closed API-based models. Despite this limitation, we strongly believe that credible faithfulness of natural language explanations such as CoTs requires parameter access and interventions.
其次,我们的方法依赖机器遗忘技术,而这类技术并不完美。对某些目标推理步骤,信息在参数中的定位或其抹除可能并不精确或并不高效。我们依赖机器遗忘与模型编辑领域的快速发展,产生更好、更精确的方法,例如 CRISP(Ashuach et al., 2025a)和 PISCES(Gur-Arieh et al., 2025),它们可以无缝接入我们的框架。虽然我们的方法能以高精度识别忠实解释,但由于遗忘不成功、解释本身不忠实,或存在替代解释,其召回并不能得到保证。此外,应用机器遗忘需要能够微调目标模型,因此 FUR 不适用于封闭的、基于 API 的模型。尽管有这一限制,我们仍然坚信:要对 CoT 这类自然语言解释给出可信的忠实性结论,就需要参数访问和干预。
Lastly, our experimental setup is limited to English language MCQA tasks. We opt for MCQA as it simplifies the analyses we perform in the paper, by allowing us to visualize probability distribution shifts over answer options without producing answer options ourselves. Both faithfulness metrics in FUR only take into account the probability, or whether the answer is the decoding, and are thus applicable beyond the MCQA scenario. Applying our method to other tasks such as long-form generation can be done by assessing whether the direct answer changes after unlearning a reasoning step. We opt for natural language tasks as factual information is conceptually easier to unlearn compared to, e.g., procedural information driving arithmetic reasoning (Ruis et al., 2025).
最后,我们的实验设置仅限于英语的 MCQA 任务。选择 MCQA,是因为它简化了文中的分析:我们可以可视化答案选项上的概率分布变化,而不必自己生成选项。FUR 中的两项忠实性指标都只考虑概率,或答案是否为 解码,因此也可以用到 MCQA 之外的场景。要把我们的方法应用到长文生成等其他任务,可以评估直接答案在遗忘一个推理步骤后是否改变。我们选择自然语言任务,是因为事实性信息在概念上比驱动算术推理的程序性信息更容易遗忘(Ruis et al., 2025)。
Acknowledgments
致谢
This research was supported by the Israel Science Foundation (grant 448/20), an Azrieli Foundation Early Career Faculty Fellowship, and an AI Alignment grant from Open Philanthropy. This research was funded by the European Union (ERC, Control-LM, 101165402). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
本研究得到以色列科学基金会(资助号 448/20)、Azrieli Foundation 早期职业教师基金,以及 Open Philanthropy 的 AI Alignment 资助。本研究由欧盟资助(ERC,Control-LM,101165402)。文中观点仅代表作者,并不必然反映欧盟或欧洲研究理事会执行机构的立场。欧盟及资助机构对此不承担责任。
References
参考文献
Abdin et al. (2024) M. I. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. S. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, C. C. T. Mendes, W. Chen, V. Chaudhary, P. Chopra, A. D. Giorno, G. de Rosa, M. Dixon, R. Eldan, D. Iter, A. Garg, A. Goswami, S. Gunasekar, E. Haider, J. Hao, R. J. Hewett, J. Huynh, M. Javaheripi, X. Jin, P. Kauffmann, N. Karampatziakis, D. Kim, M. Khademi, L. Kurilenko, J. R. Lee, Y. T. Lee, Y. Li, C. Liang, W. Liu, E. Lin, Z. Lin, P. Madan, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, X. Song, M. Tanaka, X. Wang, R. Ward, G. Wang, P. A. Witte, M. Wyatt, C. Xu, J. Xu, S. Yadav, F. Yang, Z. Yang, D. Yu, C. Zhang, C. Zhang, J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou Phi-3 technical report: A highly capable language model locally on your phone. CoRR abs/2404.14219.
Agarwal et al. (2024) C. Agarwal, S. H. Tanneru, and H. Lakkaraju Faithfulness vs. plausibility: on the (un)reliability of explanations from large language models. CoRR abs/2402.04614.
Amiri et al. (2025) A. Amiri, X. Huang, M. Rofin, and M. Hahn Lower bounds for chain-of-thought reasoning in hard-attention transformers. CoRR abs/2502.02393.
Ashuach et al. (2025a) T. Ashuach, D. Arad, A. Mueller, M. Tutek, and Y. Belinkov CRISP: persistent concept unlearning via sparse autoencoders. arXiv preprint arXiv:2508.13650.
Ashuach et al. (2025b) T. Ashuach, M. Tutek, and Y. Belinkov REVS: unlearning sensitive information in language models via rank editing in the vocabulary space. In Findings of the Association for Computational Linguistics: ACL 2025 , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.) , Vienna, Austria , pp. 14774–14797.
Atanasova et al. (2023) P. Atanasova, O. Camburu, C. Lioma, T. Lukasiewicz, J. G. Simonsen, and I. Augenstein Faithfulness tests for natural language explanations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , A. Rogers, J. L. Boyd-Graber, and N. Okazaki (Eds.) , pp. 283–294.
Bao et al. (2025) G. Bao, H. Zhang, C. Wang, L. Yang, and Y. Zhang How likely do llms with cot mimic human reasoning?. COLING , pp. 7831–7850.
Bentham et al. (2024) O. Bentham, N. Stringham, and A. Marasovic Chain-of-thought unfaithfulness as disguised accuracy. Trans. Mach. Learn. Res. 2024.
Biran et al. (2024) E. Biran, D. Gottesman, S. Yang, M. Geva, and A. Globerson Hopping too late: exploring the limitations of large language models on multi-hop queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 , Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.) , pp. 14113–14130.
Bird (2006) S. Bird NLTK: the natural language toolkit. In ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Sydney, Australia, 17-21 July 2006 , N. Calzolari, C. Cardie, and P. Isabelle (Eds.) ,
Bogdan et al. (2025) P. C. Bogdan, U. Macar, N. Nanda, and A. Conmy Thought anchors: which LLM reasoning steps matter?. CoRR abs/2506.19143.
Bowman et al. (2022) S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukosiute, A. Askell, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Olah, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, J. Kernion, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, L. Lovitt, N. Elhage, N. Schiefer, N. Joseph, N. Mercado, N. DasSarma, R. Larson, S. McCandlish, S. Kundu, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Telleen-Lawton, T. Brown, T. Henighan, T. Hume, Y. Bai, Z. Hatfield-Dodds, B. Mann, and J. Kaplan Measuring progress on scalable oversight for large language models. CoRR abs/2211.03540.
Camburu et al. (2020) O. Camburu, B. Shillingford, P. Minervini, T. Lukasiewicz, and P. Blunsom Make up your mind! adversarial generation of inconsistent natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault (Eds.) , pp. 4157–4165.
Cao and Yang (2015) Y. Cao and J. Yang Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, SP 2015, San Jose, CA, USA, May 17-21, 2015 , pp. 463–480.
Chen et al. (2024a) G. Chen, M. Liao, C. Li, and K. Fan AlphaMath almost zero: process supervision without process. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024 , A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang (Eds.) ,
Chen and Yang (2023) J. Chen and D. Yang Unlearn what you want to forget: efficient unlearning for LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , H. Bouamor, J. Pino, and K. Bali (Eds.) , Singapore , pp. 12041–12052.
Chen et al. (2024b) Y. Chen, R. Zhong, N. Ri, C. Zhao, H. He, J. Steinhardt, Z. Yu, and K. R. McKeown Do models explain themselves? counterfactual simulatability of natural language explanations. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 ,
Clark et al. (2018) P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR abs/1803.05457.
Cohen et al. (2024) R. Cohen, E. Biran, O. Yoran, A. Globerson, and M. Geva Evaluating the ripple effects of knowledge editing in language models. Trans. Assoc. Comput. Linguistics 12 , pp. 283–298.
DeepSeek-AI et al. (2025) DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, and S. S. Li DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning. CoRR abs/2501.12948.
Deng et al. (2024) Y. Deng, Y. Choi, and S. M. Shieber From explicit cot to implicit cot: learning to internalize cot step by step. CoRR abs/2405.14838.
Eldan and Russinovich (2023) R. Eldan and M. Russinovich Who’s harry potter? approximate unlearning in llms. CoRR abs/2310.02238.
Fu et al. (2023a) Y. Fu, H. Peng, L. Ou, A. Sabharwal, and T. Khot Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.) , Proceedings of Machine Learning Research , Vol. 202 , pp. 10421–10430.
Fu et al. (2023b) Y. Fu, H. Peng, A. Sabharwal, P. Clark, and T. Khot Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 ,
Gandikota et al. (2024) R. Gandikota, S. Feucht, S. Marks, and D. Bau Erasing conceptual knowledge from language models. CoRR abs/2410.02760.
Geng et al. (2025) J. Geng, Q. Li, H. Woisetschlaeger, Z. Chen, Y. Wang, P. Nakov, H. Jacobsen, and F. Karray A comprehensive survey of machine unlearning techniques for large language models. CoRR abs/2503.01854.
Geva et al. (2021a) M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics 9 , pp. 346–361.
Geva et al. (2021b) M. Geva, R. Schuster, J. Berant, and O. Levy Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.) , Online and Punta Cana, Dominican Republic , pp. 5484–5495.
Gur-Arieh et al. (2025) Y. Gur-Arieh, C. Suslik, Y. Hong, F. Barez, and M. Geva Precise in-parameter concept erasure in large language models. arXiv preprint arXiv:2505.22586.
Harding et al. (2019) E. L. Harding, J. J. Vanto, R. Clark, L. Hannah Ji, and S. C. Ainsworth Understanding the scope and impact of the california consumer privacy act of 2018. Journal of Data Protection & Privacy 2 ( 3 ), pp. 234–253.
Hassid et al. (2025) M. Hassid, G. Synnaeve, Y. Adi, and R. Schwartz Don’t overthink it. preferring shorter thinking chains for improved LLM reasoning. CoRR abs/2505.17813.
Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt Measuring massive multitask language understanding. 9th International Conference on Learning Representations.
Hong et al. (2024) Y. Hong, L. Yu, S. Ravfogel, H. Yang, and M. Geva Intrinsic evaluation of unlearning using parametric knowledge traces. CoRR abs/2406.11614.
Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 ,
Ippolito et al. (2023) D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini Preventing generation of verbatim memorization in language models gives a false sense of privacy. In Proceedings of the 16th International Natural Language Generation Conference, INLG 2023, Prague, Czechia, September 11 - 15, 2023 , C. M. Keet, H. Lee, and S. Zarrieß (Eds.) , pp. 28–53.
Jacovi et al. (2024) A. Jacovi, Y. Bitton, B. Bohnet, J. Herzig, O. Honovich, M. Tseng, M. Collins, R. Aharoni, and M. Geva A chain-of-thought is as strong as its weakest link: A benchmark for verifiers of reasoning chains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 , L. Ku, A. Martins, and V. Srikumar (Eds.) , pp. 4615–4634.
Jacovi and Goldberg (2020) A. Jacovi and Y. Goldberg Towards faithfully interpretable NLP systems: how should we define and evaluate faithfulness?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.) , Online , pp. 4198–4205.
Jaech et al. (2024) A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, A. Iftimie, A. Karpenko, A. T. Passos, A. Neitz, A. Prokofiev, A. Wei, A. Tam, A. Bennett, A. Kumar, A. Saraiva, A. Vallone, A. Duberstein, A. Kondrich, A. Mishchenko, A. Applebaum, A. Jiang, A. Nair, B. Zoph, B. Ghorbani, B. Rossen, B. Sokolowsky, B. Barak, B. McGrew, B. Minaiev, B. Hao, B. Baker, B. Houghton, B. McKinzie, B. Eastman, C. Lugaresi, C. Bassin, C. Hudson, C. M. Li, C. de Bourcy, C. Voss, C. Shen, C. Zhang, C. Koch, C. Orsinger, C. Hesse, C. Fischer, C. Chan, D. Roberts, D. Kappler, D. Levy, D. Selsam, D. Dohan, D. Farhi, D. Mely, D. Robinson, D. Tsipras, D. Li, D. Oprica, E. Freeman, E. Zhang, E. Wong, E. Proehl, E. Cheung, E. Mitchell, E. Wallace, E. Ritter, E. Mays, F. Wang, F. P. Such, F. Raso, F. Leoni, F. Tsimpourlas, F. Song, F. von Lohmann, F. Sulit, G. Salmon, G. Parascandolo, G. Chabot, G. Zhao, G. Brockman, G. Leclerc, H. Salman, H. Bao, H. Sheng, H. Andrin, H. Bagherinezhad, H. Ren, H. Lightman, H. W. Chung, I. Kivlichan, I. O’Connell, I. Osband, I. C. Gilaberte, and I. Akkaya OpenAI o1 system card. CoRR abs/2412.16720.
Jang et al. (2023) J. Jang, D. Yoon, S. Yang, S. Cha, M. Lee, L. Logeswaran, and M. Seo Knowledge unlearning for mitigating privacy risks in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , A. Rogers, J. L. Boyd-Graber, and N. Okazaki (Eds.) , pp. 14389–14408.
Jiang et al. (2023) A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed Mistral 7b. CoRR abs/2310.06825.
Jin et al. (2024) M. Jin, Q. Yu, D. Shu, H. Zhao, W. Hua, Y. Meng, Y. Zhang, and M. Du The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 , L. Ku, A. Martins, and V. Srikumar (Eds.) , pp. 1830–1842.
Joshi et al. (2023) B. Joshi, Z. Liu, S. Ramnath, A. Chan, Z. Tong, S. Nie, Q. Wang, Y. Choi, and X. Ren Are machine rationales (not) useful to humans? measuring and improving human utility of free-text rationales. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , A. Rogers, J. L. Boyd-Graber, and N. Okazaki (Eds.) , pp. 7103–7128.
Kim et al. (2023) N. Kim, P. M. Htut, S. R. Bowman, and J. Petty (QA) 2 {{}^{2}} : question answering with questionable assumptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , A. Rogers, J. L. Boyd-Graber, and N. Okazaki (Eds.) , pp. 8466–8487.
Kim et al. (2021) N. Kim, E. Pavlick, B. K. Ayan, and D. Ramachandran Which linguist invented the lightbulb? presupposition verification for question-answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021 , C. Zong, F. Xia, W. Li, and R. Navigli (Eds.) , pp. 3932–3945.
Kojima et al. (2022) T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.) ,
Lanham et al. (2023) T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion, K. Lukosiute, K. Nguyen, N. Cheng, N. Joseph, N. Schiefer, O. Rausch, R. Larson, S. McCandlish, S. Kundu, S. Kadavath, S. Yang, T. Henighan, T. Maxwell, T. Telleen-Lawton, T. Hume, Z. Hatfield-Dodds, J. Kaplan, J. Brauner, S. R. Bowman, and E. Perez Measuring faithfulness in chain-of-thought reasoning. CoRR abs/2307.13702.
Lee et al. (2025) S. Lee, S. Kim, M. Seo, Y. Jo, D. Go, H. Hwang, J. Park, X. Yue, S. Welleck, G. Neubig, M. Lee, and M. Seo The cot encyclopedia: analyzing, predicting, and controlling how a reasoning model will think. CoRR abs/2505.10185.
Li et al. (2024) N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A. Dombrowski, S. Goel, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Herbert-Voss, C. B. Breuer, A. Zou, M. Mazeika, Z. Wang, P. Oswal, W. Lin, A. A. Hunt, J. Tienken-Harder, K. Y. Shih, K. Talley, J. Guan, I. Steneker, D. Campbell, B. Jokubaitis, S. Basart, S. Fitz, P. Kumaraguru, K. K. Karmakar, U. K. Tupakula, V. Varadharajan, Y. Shoshitaishvili, J. Ba, K. M. Esvelt, A. Wang, and D. Hendrycks The WMDP benchmark: measuring and reducing malicious use with unlearning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 ,
Lightman et al. (2024) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 ,
Likert (1932) R. Likert A technique for the measurement of attitudes. Archives of Psychology.
Lin et al. (2022) S. Lin, J. Hilton, and O. Evans TruthfulQA: measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , S. Muresan, P. Nakov, and A. Villavicencio (Eds.) , pp. 3214–3252.
Liu et al. (2024) S. Liu, Y. Yao, J. Jia, S. Casper, N. Baracaldo, P. Hase, X. Xu, Y. Yao, H. Li, K. R. Varshney, M. Bansal, S. Koyejo, and Y. Liu Rethinking machine unlearning for large language models. CoRR abs/2402.08787.
Madsen et al. (2024) A. Madsen, S. Chandar, and S. Reddy Are self-explanations from large language models faithful?. In Findings of the Association for Computational Linguistics: ACL 2024 , L. Ku, A. Martins, and V. Srikumar (Eds.) , Bangkok, Thailand , pp. 295–337.
Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.) ,
Meng et al. (2023) K. Meng, A. S. Sharma, A. J. Andonian, Y. Belinkov, and D. Bau Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 ,
Mihaylov et al. (2018) T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii (Eds.) , Brussels, Belgium , pp. 2381–2391.
Minder et al. (2025) J. Minder, K. Du, N. Stoehr, G. Monea, C. Wendler, R. West, and R. Cotterell Controllable context sensitivity and the knob behind it. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 ,
Neel et al. (2021) S. Neel, A. Roth, and S. Sharifi-Malvajerdi Descent-to-delete: gradient-based methods for machine unlearning. In Algorithmic Learning Theory, 16-19 March 2021, Virtual Conference, Worldwide , V. Feldman, K. Ligett, and S. Sabato (Eds.) , Proceedings of Machine Learning Research , Vol. 132 , pp. 931–962.
Neeman et al. (2023) E. Neeman, R. Aharoni, O. Honovich, L. Choshen, I. Szpektor, and O. Abend DisentQA: disentangling parametric and contextual knowledge with counterfactual question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.) , Toronto, Canada , pp. 10056–10070.
OpenAI (2024) OpenAI Introducing openai o1.
Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.) ,
Parcalabescu and Frank (2024) L. Parcalabescu and A. Frank On measuring faithfulness or self-consistency of natural language explanations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 6048–6089.
Peng et al. (2023) B. Peng, M. Galley, P. He, H. Cheng, Y. Xie, Y. Hu, Q. Huang, L. Liden, Z. Yu, W. Chen, and J. Gao Check your facts and try again: improving large language models with external knowledge and automated feedback. CoRR abs/2302.12813.
Pfau et al. (2024) J. Pfau, W. Merrill, and S. R. Bowman Let’s think dot by dot: hidden computation in transformer language models. CoRR abs/2404.15758.
Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.) ,
Roger and Greenblatt (2023) F. Roger and R. Greenblatt Preventing language models from hiding their reasoning. CoRR abs/2310.18512.
Ruis et al. (2025) L. Ruis, M. Mozes, J. Bae, S. R. Kamalakara, D. Gnaneshwar, A. Locatelli, R. Kirk, T. Rocktäschel, E. Grefenstette, and M. Bartolo Procedural knowledge in pretraining drives reasoning in large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 ,
Sedova et al. (2024) A. Sedova, R. Litschko, D. Frassinelli, B. Roth, and B. Plank To know or not to know? analyzing self-consistency of large language models under ambiguity. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024 , Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.) , pp. 17203–17217.
Siegel et al. (2024) N. Y. Siegel, O. Camburu, N. Heess, and M. Pérez-Ortiz The probabilities also matter: A more faithful metric for faithfulness of free-text explanations in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024 - Short Papers, Bangkok, Thailand, August 11-16, 2024 , L. Ku, A. Martins, and V. Srikumar (Eds.) , pp. 530–546.
Simhi et al. (2024) A. Simhi, J. Herzig, I. Szpektor, and Y. Belinkov Distinguishing ignorance from error in LLM hallucinations. CoRR abs/2410.22071.
Sprague et al. (2025) Z. R. Sprague, F. Yin, J. D. Rodriguez, D. Jiang, M. Wadhwa, P. Singhal, X. Zhao, X. Ye, K. Mahowald, and G. Durrett To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 ,
Srivastava et al. (2023) A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slone, A. Rahane, A. S. Iyer, A. Andreassen, A. Madotto, A. Santilli, A. Stuhlmüller, A. M. Dai, A. La, A. K. Lampinen, A. Zou, A. Jiang, A. Chen, A. Vuong, A. Gupta, A. Gottardi, A. Norelli, A. Venkatesh, A. Gholamidavoodi, A. Tabassum, A. Menezes, A. Kirubarajan, A. Mullokandov, A. Sabharwal, A. Herrick, A. Efrat, A. Erdem, A. Karakas, B. R. Roberts, B. S. Loe, B. Zoph, B. Bojanowski, B. Özyurt, B. Hedayatnia, B. Neyshabur, B. Inden, B. Stein, B. Ekmekci, B. Y. Lin, B. Howald, B. Orinion, C. Diao, C. Dour, C. Stinson, C. Argueta, C. F. Ramírez, C. Singh, C. Rathkopf, C. Meng, C. Baral, C. Wu, C. Callison-Burch, C. Waites, C. Voigt, C. D. Manning, C. Potts, C. Ramirez, C. E. Rivera, C. Siro, C. Raffel, C. Ashcraft, C. Garbacea, D. Sileo, D. Garrette, D. Hendrycks, D. Kilman, D. Roth, D. Freeman, D. Khashabi, D. Levy, D. M. González, D. Perszyk, D. Hernandez, D. Chen, D. Ippolito, D. Gilboa, D. Dohan, D. Drakard, D. Jurgens, D. Datta, D. Ganguli, D. Emelin, D. Kleyko, D. Yuret, D. Chen, D. Tam, D. Hupkes, D. Misra, D. Buzan, D. C. Mollo, D. Yang, D. Lee, D. Schrader, E. Shutova, E. D. Cubuk, E. Segal, E. Hagerman, E. Barnes, E. Donoway, E. Pavlick, E. Rodolà, E. Lam, E. Chu, E. Tang, E. Erdem, E. Chang, E. A. Chi, E. Dyer, E. J. Jerzak, E. Kim, E. E. Manyasi, E. Zheltonozhskii, F. Xia, F. Siar, F. Martínez-Plumed, F. Happé, F. Chollet, F. Rong, G. Mishra, G. I. Winata, G. de Melo, G. Kruszewski, G. Parascandolo, G. Mariani, G. Wang, G. Jaimovitch-López, G. Betz, G. Gur-Ari, H. Galijasevic, H. Kim, H. Rashkin, H. Hajishirzi, H. Mehta, H. Bogar, H. Shevlin, H. Schütze, H. Yakura, H. Zhang, H. M. Wong, I. Ng, I. Noble, J. Jumelet, J. Geissinger, J. Kernion, J. Hilton, J. Lee, J. F. Fisac, J. B. Simon, J. Koppel, J. Zheng, J. Zou, J. Kocon, J. Thompson, J. Wingfield, J. Kaplan, J. Radom, J. Sohl-Dickstein, J. Phang, J. Wei, J. Yosinski, J. Novikova, J. Bosscher, J. Marsh, J. Kim, J. Taal, J. H. Engel, J. Alabi, J. Xu, J. Song, J. Tang, J. Waweru, J. Burden, J. Miller, J. U. Balis, J. Batchelder, J. Berant, J. Frohberg, J. Rozen, J. Hernández-Orallo, J. Boudeman, J. Guerr, J. Jones, J. B. Tenenbaum, J. S. Rule, J. Chua, K. Kanclerz, K. Livescu, K. Krauth, K. Gopalakrishnan, K. Ignatyeva, K. Markert, K. D. Dhole, K. Gimpel, K. Omondi, K. W. Mathewson, K. Chiafullo, K. Shkaruta, K. Shridhar, K. McDonell, K. Richardson, L. Reynolds, L. Gao, L. Zhang, L. Dugan, L. Qin, L. C. Ochando, L. Morency, L. Moschella, L. Lam, L. Noble, L. Schmidt, L. He, L. O. Colón, L. Metz, L. K. Senel, M. Bosma, M. Sap, M. ter Hoeve, M. Farooqi, M. Faruqui, M. Mazeika, M. Baturan, M. Marelli, M. Maru, M. J. Ramírez-Quintana, M. Tolkiehn, M. Giulianelli, M. Lewis, M. Potthast, M. L. Leavitt, M. Hagen, M. Schubert, M. Baitemirova, M. Arnaud, M. McElrath, M. A. Yee, M. Cohen, M. Gu, M. I. Ivanitskiy, M. Starritt, M. Strube, M. Swedrowski, M. Bevilacqua, M. Yasunaga, M. Kale, M. Cain, M. Xu, M. Suzgun, M. Walker, M. Tiwari, M. Bansal, M. Aminnaseri, M. Geva, M. Gheini, M. V. T., N. Peng, N. A. Chi, N. Lee, N. G. Krakover, N. Cameron, N. Roberts, N. Doiron, N. Martinez, N. Nangia, N. Deckers, N. Muennighoff, N. S. Keskar, N. Iyer, N. Constant, N. Fiedel, N. Wen, O. Zhang, O. Agha, O. Elbaghdadi, O. Levy, O. Evans, P. A. M. Casares, P. Doshi, P. Fung, P. P. Liang, P. Vicol, P. Alipoormolabashi, P. Liao, P. Liang, P. Chang, P. Eckersley, P. M. Htut, P. Hwang, P. Milkowski, P. Patil, P. Pezeshkpour, P. Oli, Q. Mei, Q. Lyu, Q. Chen, R. Banjade, R. E. Rudolph, R. Gabriel, R. Habacker, R. Risco, R. Millière, R. Garg, R. Barnes, R. A. Saurous, R. Arakawa, R. Raymaekers, R. Frank, R. Sikand, R. Novak, R. Sitelew, R. LeBras, R. Liu, R. Jacobs, R. Zhang, R. Salakhutdinov, R. Chi, R. Lee, R. Stovall, R. Teehan, R. Yang, S. Singh, S. M. Mohammad, S. Anand, S. Dillavou, S. Shleifer, S. Wiseman, S. Gruetter, S. R. Bowman, S. S. Schoenholz, S. Han, S. Kwatra, S. A. Rous, S. Ghazarian, S. Ghosh, S. Casey, S. Bischoff, S. Gehrmann, S. Schuster, S. Sadeghi, S. Hamdan, S. Zhou, S. Srivastava, S. Shi, S. Singh, S. Asaadi, S. S. Gu, S. Pachchigar, S. Toshniwal, S. Upadhyay, S. (. Debnath, S. Shakeri, S. Thormeyer, S. Melzi, S. Reddy, S. P. Makini, S. Lee, S. Torene, S. Hatwar, S. Dehaene, S. Divic, S. Ermon, S. Biderman, S. Lin, S. Prasad, S. T. Piantadosi, S. M. Shieber, S. Misherghi, S. Kiritchenko, S. Mishra, T. Linzen, T. Schuster, T. Li, T. Yu, T. Ali, T. Hashimoto, T. Wu, T. Desbordes, T. Rothschild, T. Phan, T. Wang, T. Nkinyili, T. Schick, T. Kornev, T. Tunduny, T. Gerstenberg, T. Chang, T. Neeraj, T. Khot, T. Shultz, U. Shaham, V. Misra, V. Demberg, V. Nyamai, V. Raunak, V. V. Ramasesh, V. U. Prabhu, V. Padmakumar, V. Srikumar, W. Fedus, W. Saunders, W. Zhang, W. Vossen, X. Ren, X. Tong, X. Zhao, X. Wu, X. Shen, Y. Yaghoobzadeh, Y. Lakretz, Y. Song, Y. Bahri, Y. Choi, Y. Yang, Y. Hao, Y. Chen, Y. Belinkov, Y. Hou, Y. Hou, Y. Bai, Z. Seid, Z. Zhao, Z. Wang, Z. J. Wang, Z. Wang, and Z. Wu Beyond the imitation game: quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res. 2023.
Sun et al. (2023) Z. Sun, X. Wang, Y. Tay, Y. Yang, and D. Zhou Recitation-augmented language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 ,
Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom Llama 2: open foundation and fine-tuned chat models. CoRR abs/2307.09288.
Turpin et al. (2023) M. Turpin, J. Michael, E. Perez, and S. R. Bowman Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.) ,
Wang et al. (2024) X. Wang, B. Ma, C. Hu, L. Weber-Genzel, P. Röttger, F. Kreuter, D. Hovy, and B. Plank "My answer is c": first-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 , L. Ku, A. Martins, and V. Srikumar (Eds.) , pp. 7407–7416.
Wang et al. (2023) X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou Self-consistency improves chain of thought reasoning in language models. ICLR.
Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.) ,
Wu et al. (2023) X. Wu, J. Li, M. Xu, W. Dong, S. Wu, C. Bian, and D. Xiong DEPN: detecting and editing privacy neurons in pretrained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , H. Bouamor, J. Pino, and K. Bali (Eds.) , pp. 2875–2886.
Xiao et al. (2025) Y. Xiao, J. Wang, R. Yuan, C. Xu, K. Xu, W. Li, and P. Liu LIMOPro: reasoning refinement for efficient and effective test-time scaling. CoRR abs/2505.19187.
Xu et al. (2025) S. Xu, W. Xie, L. Zhao, and P. He Chain of draft: thinking faster by writing less. CoRR abs/2502.18600.
Yang et al. (2024) S. Yang, E. Gribovskaya, N. Kassner, M. Geva, and S. Riedel Do large language models latently perform multi-hop reasoning?. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 , L. Ku, A. Martins, and V. Srikumar (Eds.) , pp. 10210–10229.
Yao et al. (2024) Y. Yao, X. Xu, and Y. Liu Large language model unlearning. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024 , A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang (Eds.) ,
Yeo et al. (2024) W. J. Yeo, R. Satapathy, and E. Cambria Towards faithful natural language explanations: A study using activation patching in large language models. CoRR abs/2410.14155.
Yu et al. (2025) S. Yu, Y. Xiong, J. Wu, X. Li, T. Yu, X. Chen, R. Sinha, J. Shang, and J. McAuley Explainable chain-of-thought reasoning: an empirical analysis on state-aware reasoning dynamics. arXiv preprint arXiv:2509.00190.
Zaman and Srivastava (2025) K. Zaman and S. Srivastava A causal lens for evaluating faithfulness metrics. CoRR abs/2502.18848.
Zhang et al. (2024a) M. Zhang, O. Press, W. Merrill, A. Liu, and N. A. Smith How language model hallucinations can snowball. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 ,
Zhang et al. (2024b) R. Zhang, L. Lin, Y. Bai, and S. Mei Negative preference optimization: from catastrophic collapse to effective unlearning. CoRR abs/2404.05868.
Zheng et al. (2023a) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.) ,
Zheng et al. (2023b) S. Zheng, J. Huang, and K. C. Chang Why does chatgpt fall short in providing truthful answers?. CoRR abs/2304.10513.
Zhou et al. (2023) D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q. V. Le, and E. H. Chi Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 ,
Appendix A Background on Machine Unlearning
附录 A 机器遗忘的背景
Motivated by the need to erase sensitive information from machine learning models (Cao and Yang, 2015; Harding et al., 2019; Ippolito et al., 2023), the field of machine unlearning emerged as an efficient alternative to filter-then-retrain-based approaches (Neel et al., 2021; Jang et al., 2023; Eldan and Russinovich, 2023; Liu et al., 2024, inter alia). Machine unlearning methods update parameters of the model in a way that reduces the competency of the model on unwanted data (henceforth, forget), while retaining general capabilities through regularization on retain data. Such methods decrease token probabilities on the forget data by gradient-based approaches (Jang et al., 2023; Eldan and Russinovich, 2023; Gandikota et al., 2024; Li et al., 2024; Zhang et al., 2024b) or directly updating parameters (Meng et al., 2022; Meng et al., 2023; Wu et al., 2023; Ashuach et al., 2025b).
出于从机器学习模型中抹去敏感信息的需要(Cao and Yang, 2015; Harding et al., 2019; Ippolito et al., 2023),机器遗忘作为一个高效替代出现,用来取代“先过滤再重新训练”的做法(Neel et al., 2021; Jang et al., 2023; Eldan and Russinovich, 2023; Liu et al., 2024, inter alia)。机器遗忘方法更新模型参数,降低模型在不受欢迎的数据(下文称遗忘数据)上的能力,同时通过对保留数据正则化来保持通用能力。这类方法通过基于梯度的方式降低遗忘数据上的词元概率(Jang et al., 2023; Eldan and Russinovich, 2023; Gandikota et al., 2024; Li et al., 2024; Zhang et al., 2024b),或直接更新参数(Meng et al., 2022; Meng et al., 2023; Wu et al., 2023; Ashuach et al., 2025b)。
In order to ensure that unlearning does not adversely affect the model to the point it is unusable, LMs need to satisfy the following desiderata post-unlearning: (1) efficacy, controlling whether the forget data was erased from the model; (2) specificity, controlling that the edit is localized to the target information, often by probing the model on closely related data; (3) general capabilities, measuring whether the model retains fluency and performance on unrelated data.
为了确保遗忘不会把模型损害到无法使用的程度,语言模型在遗忘后需要满足以下要求:(1)效力,控制遗忘数据是否已从模型中被抹去;(2)特异性,控制编辑是否局限在目标信息上,通常通过在密切相关的数据上探测模型来衡量;(3)通用能力,衡量模型是否仍保持流畅性以及在无关数据上的表现。
Negative Preference Optimization.
负面偏好优化。
In this work, we use a preference-optimization based unlearning method: negative preference optimization (Zhang et al., 2024b, NPO;). The core idea underpinning NPO is rooted in gradient ascent. The initial model is trained on a mixture of wanted and unwanted data . Our goal is to eliminate unwanted information from the model. Therefore, applying the reverse language modeling objective on the forget data:
在本文中,我们使用一种基于偏好优化的遗忘方法:负面偏好优化(Zhang et al., 2024b,简称 NPO)。NPO 的核心想法源于梯度上升。初始模型 在想要的与不想要的数据的混合 上训练。我们的目标是从模型中消除不想要的信息 。因此,在遗忘数据上施加反向的语言建模目标:
would approximately revert the optimization, producing .
将近似地逆转优化,得到 。
Applying gradient ascent in this way runs into two practical issues. Firstly, we often do not have access to the training dataset, and therefore neither to the unwanted data . Secondly, the gradient ascent objective is unbounded by virtue of maximizing the next-token prediction loss, frequently resulting in catastrophic collapse (Zhang et al., 2024b). Machine unlearning approaches resolve the first issue by approximating with a forget set containing samples of unwanted data. NPO resolves the second issue by constraining that the policy (predictive distribution) of the unlearned model should not diverge too far from a reference model . In practice, the frozen base model is used as the reference ().
以这种方式做梯度上升会遇到两个实际问题。第一,我们往往无法获得训练数据集,因而也无法获得不想要的数据 。第二,梯度上升目标通过最大化下一词预测损失而没有上界,常常导致灾难性崩溃(Zhang et al., 2024b)。机器遗忘方法用包含不受欢迎数据样本的遗忘集来近似 ,从而解决第一个问题。NPO 则通过约束已遗忘模型的策略(预测分布) 不要偏离参考模型 过远,来解决第二个问题。实践中,冻结的基础模型被用作参考()。
The NPO loss is then defined as:
于是,NPO 损失定义为:
where is the inverse temperature. This loss can be viewed as Direct Preference Optimization (Rafailov et al., 2023, DPO;) without the positive samples. In practice, along with the loss term in Eq. (7), NPO also constrains the KL divergence between the unlearned and reference models on retain data in order to guarantee fluency, similar to other works (Li et al., 2024; Gandikota et al., 2024):
其中 是逆温度。这一损失可以看作没有正样本的直接偏好优化(Rafailov et al., 2023,简称 DPO)。实践中,除式(7)中的损失项外,NPO 还会在保留数据上约束已遗忘模型与参考模型之间的 KL 散度,以保证流畅性,这与其他工作类似(Li et al., 2024; Gandikota et al., 2024):
As the KL divergence regularizer maintains that the unlearned model does not diverge from the reference, it does not introduce new information to the model.
由于 KL 散度正则项保证已遗忘模型不会偏离参考模型,它不会向模型引入新的信息。
In our work, the forget set consists of all tokens of a given content word from a CoT step as the output to be forgotten, paired with the word’s preceding context as the input . The retain data is constructed similarly out of CoT steps from unrelated instances. We alter a subset of the full model’s parameters by minimizing:
在我们的工作中,遗忘集由某个 CoT 步骤中一个内容词的全部词元作为待遗忘的输出 ,并与该词之前的上下文作为输入 配对而成。保留数据则用无关实例中的 CoT 步骤按同样方式构造。我们通过最小化下式,来改动完整模型参数的一个子集 :
Appendix B Alternatives to Measuring (Un)Faithfulness
附录 B 衡量(不)忠实性的其他方法
A number of methods have been proposed with the goal of measuring faithfulness of model reasoning (Lanham et al., 2023; Atanasova et al., 2023; Bentham et al., 2024; Siegel et al., 2024; Chen et al., 2024b; Madsen et al., 2024, inter alia). However, not all of them are applicable to our setup. Some of the methods are designed for NLI tasks (Atanasova et al., 2023; Siegel et al., 2024; Parcalabescu and Frank, 2024) while others aim at multi-hop QA tasks (Parcalabescu and Frank, 2024; Lanham et al., 2023; Chen et al., 2024b; Bentham et al., 2024). In our work, we focus on QA tasks as datasets requiring multi-hop reasoning are more prominent in this task (Jacovi et al., 2024), allowing us a broader domain coverage. We further choose for MCQA, as alternative answers in these datasets are often by design plausible, and this allows for a more in-depth analysis of how unlearning affects the underlying reasoning of the model, by e.g. making it opt for plausible alternatives. On the contrary, in NLI, the model is either right or wrong – there are few “alternative explanations” and the analysis one can do is limited.
已有多种方法被提出,用来衡量模型推理的忠实性(Lanham et al., 2023; Atanasova et al., 2023; Bentham et al., 2024; Siegel et al., 2024; Chen et al., 2024b; Madsen et al., 2024, inter alia)。不过,它们并不都能用到我们的设定中。有些方法是为 NLI 任务设计的(Atanasova et al., 2023; Siegel et al., 2024; Parcalabescu and Frank, 2024),另一些则面向多跳问答任务(Parcalabescu and Frank, 2024; Lanham et al., 2023; Chen et al., 2024b; Bentham et al., 2024)。本文聚焦问答任务,因为需要多跳推理的数据集在这类任务中更为常见(Jacovi et al., 2024),从而使我们能够覆盖更广的领域。我们进一步选择 MCQA,是因为这些数据集中的替代答案往往在设计上就是合理的,这便于更深入地分析遗忘如何影响模型的底层推理,例如使它转而选择合理的替代项。相比之下,在 NLI 中模型要么对要么错——几乎没有“替代解释”,可做的分析也有限。
Other Baselines.
其他基线。
We have considered other baselines applicable to CoT reasoning (Zaman and Srivastava, 2025, cf. Table 1. in). Namely, we explore Early Answering, Filler Tokens, Adding Mistakes and Paraphrasing (Lanham et al., 2023) as well as CC-SHAP (Parcalabescu and Frank, 2024). Adding Mistakes is a simple contextual faithfulness method that works well, and we compare to its results in Table 2. We replicate the Paraphrasing setup and find that it is able to identify a small proportion (%, on average) of instances as unfaithful. See full results in Table 4. Importantly, this does not imply that the remaining instances are faithful, which is the goal of our work.
我们也考虑了其他可应用于 CoT 推理的基线(参见 Zaman and Srivastava, 2025, 表 1)。具体来说,我们考察了 Early Answering、Filler Tokens、Adding Mistakes 和 Paraphrasing(Lanham et al., 2023),以及 CC-SHAP(Parcalabescu and Frank, 2024)。Adding Mistakes 是一种简单且效果不错的上下文忠实性方法,我们在表 2 中与它的结果做了比较。我们复现了 Paraphrasing 的设定,发现它只能把一小部分实例(平均 )识别为不忠实。完整结果见表 4。重要的是,这并不意味着其余实例就是忠实的,而识别忠实实例正是我们工作的目标。
Table 4: The percentage of CoTs identified as unfaithful by the Paraphrase baseline (Lanham et al., 2023).
| Model | Arc-Ch | Book | Sports | SQA |
|---|---|---|---|---|
| LLaMA-8B | ||||
| LLaMA-3B | ||||
| Mistral-2 | ||||
| Phi-3 |
表 4:Paraphrase 基线(Lanham et al., 2023)识别为不忠实的 CoT 百分比。
| 模型 | Arc-Ch | Book | Sports | SQA |
|---|---|---|---|---|
| LLaMA-8B | ||||
| LLaMA-3B | ||||
| Mistral-2 | ||||
| Phi-3 |
The remaining methods from Lanham et al. (2023) aim to identify whether reasoning is produced post-hoc, or truly necessary to produce the prediction. Early Answering truncates the CoT, while Filler Tokens substitutes the CoT with ellipsis tokens. Then, if the answer did not change, the CoT is deemed unnecessary (post-hoc reasoning). These measures do not aim to determine faithfulness of CoTs. Post-hoc reasoning can still be a true verbalization of latent reasoning, which is what both FUR and Add-Mistake find Table 2 since we only evaluate faithfulness cases where CoT and no-CoT predictions of models agree. Such cases would be identified as post-hoc reasoning according to Early Answering and Filler Tokens, but should not be discarded as unfaithful.
Lanham et al.(2023)中其余方法,旨在识别推理是事后生成的,还是真正为产生预测所必需。Early Answering 截断 CoT,Filler Tokens 则用省略号词元替换 CoT。若答案没有改变,该 CoT 就被认为是不必要的(事后推理)。这些度量并不以判定 CoT 的忠实性为目标。事后推理仍然可以是对潜在推理的真实口头化,而这正是 FUR 和 Add-Mistake 在表 2 中所发现的——因为我们只在模型的 CoT 预测与无 CoT 预测一致的情形下评估忠实性。按 Early Answering 和 Filler Tokens 的标准,这类情形会被识别为事后推理,但不应当因此被当作不忠实而丢弃。
Finally, we experiment with CC-SHAP (Parcalabescu and Frank, 2024), a self-consistency measure based on Shapley values, which measures the convergence between input tokens salient for the prediction and explanation. We use the official implementation from the authors, but when applying SHAP to instances from our dataset, relative importances of tokens from input are frequently exactly zero (importances for reasoning do not behave in this manner), which results in NaN CC-SHAP scores in % of instances across datasets and models. We believe such low scores for model predictions are caused by the fact that the inputs are only the question and answer options, while the evidence (reasoning) is intrinsic to the model.
最后,我们试验了 CC-SHAP(Parcalabescu and Frank, 2024),这是一种基于 Shapley 值的自一致性度量,衡量对预测和对解释都显著的输入词元之间的一致性。我们使用作者提供的官方实现,但在把 SHAP 应用到我们数据集中的实例时,输入词元的相对重要性经常恰好为零(推理上的重要性并不会这样),从而导致跨数据集和模型有 的实例得到 NaN 的 CC-SHAP 分数。我们认为,模型预测上出现如此低的分数,是因为输入只有问题和选项,而证据(推理)是模型内在的。
Appendix C Dataset and Model Statistics
附录 C 数据集与模型统计
We report the base performance of the analyzed models on the datasets we selected, with and without CoT in Table 5. Statistics on the total, and average counts of CoT steps can be seen in Table 6. We describe and exemplify the prompting setup in Appendix D.
我们在表 5 中报告所分析模型在所选数据集上、有 CoT 与无 CoT 时的基础表现。CoT 步骤的总数和平均数见表 6。提示设置的说明与示例见附录 D。
To compute model predictions, we use letter completion. We evaluate the probability each model assigns to the first letters of the answer choices (i.e. A, B, C, D, E) and then normalize the probabilities so that they sum to to obtain model predictions over the answer set. We account for the verbosity issues raised by Wang et al. (2024) by directly prompting the model with the prefix “My answer is (”, making it to choose from the answer choices.
为了计算模型预测,我们使用字母补全。我们评估每个模型赋给选项首字母(即 A、B、C、D、E)的概率,再把这些概率归一化使它们之和为 ,从而得到模型在答案集合上的预测。为应对 Wang et al.(2024)提出的冗长问题,我们直接用前缀“My answer is (”提示模型,迫使它从选项中选择。
Table 5: Results of analyzed models on the datasets when prompted with and without CoTs. Results better by at least one percentage point underlined. In general, the tasks are difficult for the models, and using CoT improves over no-CoT.
| Model | CoT | Arc-Ch | Book | Sports | SQA | TQA |
|---|---|---|---|---|---|---|
| LLaMA-8B | ✗ | |||||
| LLaMA-8B | ✓ | |||||
| LLaMA-3B | ✗ | |||||
| LLaMA-3B | ✓ | |||||
| Mistral-2 | ✗ | |||||
| Mistral-2 | ✓ | |||||
| Phi-3 | ✗ | |||||
| Phi-3 | ✓ |
表 5:所分析模型在有 CoT 与无 CoT 提示下的数据集结果。至少高出一个百分点的结果用下划线标出。总体而言,这些任务对模型较难,使用 CoT 优于无 CoT。
| 模型 | CoT | Arc-Ch | Book | Sports | SQA | TQA |
|---|---|---|---|---|---|---|
| LLaMA-8B | ✗ | |||||
| LLaMA-8B | ✓ | |||||
| LLaMA-3B | ✗ | |||||
| LLaMA-3B | ✓ | |||||
| Mistral-2 | ✗ | |||||
| Mistral-2 | ✓ | |||||
| Phi-3 | ✗ | |||||
| Phi-3 | ✓ |
Table 6: Average number of CoT steps per model and dataset, measured on the full instances from each dataset ( for Sports).
| Model | Arc-Ch | Book | Sports | SQA | TQA |
|---|---|---|---|---|---|
| LLaMA-8B | |||||
| LLaMA-3B | |||||
| Mistral-2 | |||||
| Phi-3 |
表 6:各模型和数据集上每条 CoT 的平均步骤数,在每个数据集全部 个实例上衡量(Sports 为 个)。
| 模型 | Arc-Ch | Book | Sports | SQA | TQA |
|---|---|---|---|---|---|
| LLaMA-8B | |||||
| LLaMA-3B | |||||
| Mistral-2 | |||||
| Phi-3 |
Appendix D MCQA Task Prompts
附录 D MCQA 任务提示
We use two flavors of prompts when producing model predictions and the CoT for the evaluated tasks. In the first, direct prompting setup, we directly prompt the model to generate the answer based on the question and answer options. The second, two-step setup first prompts the model to generate a CoT, then concatenates the CoT to the question and answer options, and prompts the model to produce the answer. Prompts adapted from (Bowman et al., 2022; Lanham et al., 2023; Bentham et al., 2024). We conduct both prompting setups in zero-shot manner.
在为所评估任务生成模型预测和 CoT 时,我们使用两种提示。第一种是直接提示:根据问题和选项直接提示模型生成答案。第二种是两步设定:先提示模型生成一条 CoT,再把该 CoT 拼接到问题和选项上,提示模型给出答案。提示改编自 Bowman et al.(2022)、Lanham et al.(2023)和 Bentham et al.(2024)。两种提示都在零样本设定下进行。
Direct Answer Prompt
直接答案提示
Human: Question: [Question]
Choices:
[Answer_choices]
Assistant: The single, most likely answer is (
直接答案提示:
Human: Question: [Question]
Choices:
[Answer_choices]
Assistant: The single, most likely answer is (
CoT Prompt
CoT 提示
Human: Question: [Question]
Choices:
[Answer_choices]
Assistant: Let's think step by step:
CoT 提示:
Human: Question: [Question]
Choices:
[Answer_choices]
Assistant: Let's think step by step:
CoT Answer Prompt
CoT 答案提示
Human: Question: [Question]
Choices:
[Answer_choices]
[Chain_of_thought]
Human: Given all of the above, what's the single, most likely answer?"
Assistant: The single, most likely answer is (
CoT 答案提示:
Human: Question: [Question]
Choices:
[Answer_choices]
[Chain_of_thought]
Human: Given all of the above, what's the single, most likely answer?"
Assistant: The single, most likely answer is (
Appendix E Unlearning Setup & Hyperparameters
附录 E 遗忘设置与超参数
We adapt the implementation of NPO+KL from the official repository. We use the best hyperparameters found by the original paper (Zhang et al., 2024b) except for the values which we highlight in bold. See Table 7 for values.
我们从官方代码库改编了 NPO+KL 的实现。除我们用加粗标出的取值外,我们采用原论文(Zhang et al., 2024b)找到的最佳超参数。取值见表 7。
Table 7: Hyperparameters used in the implementation of NPO+KL. Bold values deviate from the original paper.
| Hyperparameter | Value |
|---|---|
| beta | |
| npo_coeff | |
| KL_coeff | |
| ref_policy | fine_tuned |
| epochs | |
| warmup | no |
表 7:NPO+KL 实现中使用的超参数。加粗的取值与原论文不同。
| 超参数 | 取值 |
|---|---|
| beta | |
| npo_coeff | |
| KL_coeff | |
| ref_policy | fine_tuned |
| epochs | |
| warmup | no |
We deviate in our choice of epochs since we are unlearning a single sentence, and in our preliminary experiments, epochs (iterations) of unlearning always sufficed. We deviate in our choice of warmup as each epoch is a single unlearning step – there is a total of one instance, thus the warmup simply skips a step as the learning rate in the first iteration of the schedule corresponds to .
我们在轮数上做了改动,因为我们只遗忘一个句子;在初步实验中, 轮(迭代)遗忘总是足够。我们在预热上做了改动,因为每一轮就是一次遗忘步骤——总共只有一个实例,因而预热实际上会跳过一步,因为调度中第一次迭代的学习率对应于 。
Unlearning Setup.
遗忘设置。
When performing unlearning, we backpropagate only on target tokens which are constituents of content words, namely nouns, proper nouns, adjectives, verbs and numbers. We filter out and don’t unlearn all CoT steps which do not have at least two target tokens. This usually corresponds to the index in the CoT step enumeration which plenty of models produce (e.g. *1.*This is a CoT step), where “1.” is sentencized as a standalone sentence by SpaCy.
做遗忘时,我们只在作为内容词组成成分的目标词元上反向传播,即名词、专有名词、形容词、动词和数字。我们过滤掉、并且不去遗忘所有目标词元少于两个的 CoT 步骤。这通常对应于不少模型会生成的 CoT 步骤编号(例如 *1.*This is a CoT step),其中“1.”会被 SpaCy 单独切成一句。
When unlearning, NPO+KL uses KL regularization to control updates to model parameters, which could otherwise be unbounded (Zhang et al., 2024b). During optimization, the model is regularized not to deviate from its initial version with respect to KL divergence of the predictive distribution on a retain set. For the retain set, we select a random sample of other CoT steps from the same dataset. We perform the same filtering in the retain set, keeping only steps which contain more than two tokens which are constituents of content words, and only target those words for KL regularization.
遗忘时,NPO+KL 用 KL 正则来控制对模型参数的更新,否则更新可能没有上界(Zhang et al., 2024b)。在优化过程中,模型被正则化,使其在保留集上的预测分布的 KL 散度不要偏离初始版本。对保留集,我们从同一数据集中随机抽取 个其他 CoT 步骤。我们对保留集做同样的过滤,只保留包含两个以上内容词组成词元的步骤,并且只对这些词做 KL 正则。
E.1 Learning Rate Selection
E.1 学习率选择
For each model and dataset, we perform a hyperparameter sweep on the learning rate values, as we find different models respond differently to varying unlearning strength. We report the results in graphical Figure 6 and tabular format below Table 8. We selected the best learning rate as the one with highest efficacy while maintaining , i.e., allowing for a single prediction to differ from the base model on the held-out set , on average.
对每个模型和数据集,我们都对学习率做超参数扫描,因为我们发现不同模型对遗忘强度的反应不同。结果以图 6 的图形形式以及下面表 8 的表格形式报告。我们把最佳学习率选为在保持 的前提下效力最高的那一个,也就是平均而言,允许留出集 上有一个预测与基础模型不同。
Figure 6: Learning rate selection results for NPO+KL. Experiments ran on instances for all datasets. Size of the marker depicts faithfulness, only for information purposes — faithfulness was not used as the selection criterion. Learning rates omitted for clarity, but as a rule, the higher the learning rate, the higher the efficacy, and the lower the specificity. Figure presented for glance-value, scores are also reported in tabular format in Table 8.
图 6:NPO+KL 的学习率选择结果。实验在全部数据集的 个实例上运行。标记大小表示忠实性,仅供参考——忠实性并未用作选择标准。为清晰起见省略了学习率,但一般规律是:学习率越高,效力越高,特异性越低。本图供快速浏览,分数也以表格形式列在表 8 中。
Table 8: Learning rate selection results for NPO+KL. Experiments ran on instances for all datasets. Faithfulness was not used as the selection criterion, but is here only for informativeness. Best learning rates per model & dataset highlighted. Criterion was s.t. .
| Arc-Challenge | OpenbookQA | Sports | StrategyQA | TruthfulQA | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LR | Eff | Spec | FF | Eff | Spec | FF | Eff | Spec | FF | Eff | Spec | FF | Eff | Spec | FF | |
| LLaMA-8B | — | — | — | — | — | — | ||||||||||
| LLaMA-8B | — | — | — | |||||||||||||
| LLaMA-8B | ||||||||||||||||
| LLaMA-8B | ||||||||||||||||
| LLaMA-8B | — | — | — | |||||||||||||
| LLaMA-8B | — | — | — | |||||||||||||
| LLaMA-8B | — | — | — | — | — | — | — | — | — | — | — | — | ||||
| LLaMA-3B | — | — | — | |||||||||||||
| LLaMA-3B | ||||||||||||||||
| LLaMA-3B | ||||||||||||||||
| LLaMA-3B | — | — | — | |||||||||||||
| LLaMA-3B | ||||||||||||||||
| Mistral-2 | — | — | — | — | — | — | ||||||||||
| Mistral-2 | ||||||||||||||||
| Mistral-2 | ||||||||||||||||
| Mistral-2 | ||||||||||||||||
| Mistral-2 | — | — | — | — | — | — | ||||||||||
| Phi-3 | ||||||||||||||||
| Phi-3 | — | — | — | |||||||||||||
| Phi-3 | ||||||||||||||||
| Phi-3 | — | — | — | — | — | — | — | — | — | |||||||
| Phi-3 | — | — | — | |||||||||||||
| Phi-3 | — | — | — | — | — | — |
表 8:NPO+KL 的学习率选择结果。实验在全部数据集的 个实例上运行。忠实性并未用作选择标准,此处仅供参考。各模型与数据集的最佳学习率已标出。选择标准为 ,约束为 。
| Arc-Challenge | OpenbookQA | Sports | StrategyQA | TruthfulQA | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 学习率 | Eff | Spec | FF | Eff | Spec | FF | Eff | Spec | FF | Eff | Spec | FF | Eff | Spec | FF | |
| LLaMA-8B | — | — | — | — | — | — | ||||||||||
| LLaMA-8B | — | — | — | |||||||||||||
| LLaMA-8B | ||||||||||||||||
| LLaMA-8B | ||||||||||||||||
| LLaMA-8B | — | — | — | |||||||||||||
| LLaMA-8B | — | — | — | |||||||||||||
| LLaMA-8B | — | — | — | — | — | — | — | — | — | — | — | — | ||||
| LLaMA-3B | — | — | — | |||||||||||||
| LLaMA-3B | ||||||||||||||||
| LLaMA-3B | ||||||||||||||||
| LLaMA-3B | — | — | — | |||||||||||||
| LLaMA-3B | ||||||||||||||||
| Mistral-2 | — | — | — | — | — | — | ||||||||||
| Mistral-2 | ||||||||||||||||
| Mistral-2 | ||||||||||||||||
| Mistral-2 | ||||||||||||||||
| Mistral-2 | — | — | — | — | — | — | ||||||||||
| Phi-3 | ||||||||||||||||
| Phi-3 | — | — | — | |||||||||||||
| Phi-3 | ||||||||||||||||
| Phi-3 | — | — | — | — | — | — | — | — | — | |||||||
| Phi-3 | — | — | — | |||||||||||||
| Phi-3 | — | — | — | — | — | — |
Appendix F LoRA Setup & Hyperparameters
附录 F LoRA 设置与超参数
In this section, we outline the hyperparameters, experimental setup used to LoRA-tune the models in §7 as well present additional results.
本节概述第 7 节中用 LoRA 微调模型时所采用的超参数、实验设置,并给出补充结果。
LoRA tuning setup.
LoRA 微调设置。
We perform the feasibility analysis of LoRA tuning on two models from the LLaMA family and two datasets which proved most difficult for the models: Sports understanding and StrategyQA. LoRA tuning is less invasive for the base model compared to full (FF2) fine-tuning – made evident by the fact that the learning rate can be stronger without affecting the models adversely. For brevity, we omit a full table (akin to Table 8) and only report the used LoRA hyperparameters and learning rate ranges in Table 9. The best learning rates were for LLaMA-3-8B, and for LLaMA-3-3B. Note that these are a -fold increase compared to best values found for full tuning.
我们在 LLaMA 系列的两个模型上,以及模型表现最困难的两个数据集——Sports understanding 和 StrategyQA——上分析 LoRA 微调的可行性。与全量(FF2)微调相比,LoRA 微调对基础模型的侵入更小——这体现在学习率可以更强,却不会对模型造成不利影响。为简洁起见,我们省略类似表 8 的完整表格,只在表 9 中报告所用的 LoRA 超参数和学习率范围。最佳学习率为:LLaMA-3-8B 是 ,LLaMA-3-3B 是 。注意,这大约是全量微调最佳取值的 倍。
Table 9: Hyperparameters used to LoRA-tune the LLaMA models on Sports and StrategyQA.
| Hyperparameter | Value(s) |
|---|---|
| learning_rate | |
| rank | |
| lora_alpha | |
| lora_dropout | |
| target_module | down_proj (FF2) |
表 9:在 Sports 和 StrategyQA 上对 LLaMA 模型做 LoRA 微调所用的超参数。
| 超参数 | 取值 |
|---|---|
| learning_rate | |
| rank | |
| lora_alpha | |
| lora_dropout | |
| target_module | down_proj (FF2) |
Does low-rank unlearning fundamentally affect model reasoning?
低秩遗忘是否会从根本上影响模型推理?
Due to the need to perform another learning rate sweep over the models and datasets, we did not perform experiments on all instances from the datasets but rather reported those on a instance sample which we used for the learning rate selection. As seen in Section 7, FF-HARD scores obtained by LoRA-tuned models are comparable to ones of full fine-tuning. We now ask the question: Do low-rank updates have the same effect on the model reasoning post-unlearning as the ones we observed in Section 6.3?
由于还需要对模型和数据集再做一轮学习率扫描,我们没有在全部实例上做实验,而是报告用于学习率选择的 个实例样本上的结果。如第 7 节所示,LoRA 微调模型得到的 FF-HARD 分数与全量微调相当。我们现在要问:低秩更新在遗忘后,是否会对模型推理产生与第 6.3 节中观察到的相同效果?
We conduct the same LLM-as-a-judge experiment previously ran for full fine-tuning (cf. Table 3 and Appendix H) on the instance sample, and report results in Table 10. We find that GPT-4o largely agrees that the models argue for a different answer option post-unlearning, indicating that even low rank adaptation has a profound effect on the verbalized reasoning. We believe LoRA presents a viable alternative to full fine-tuning. However, in order to fully purge unwanted information from the model, full fine-tuning is necessary.
我们在这 个实例样本上,重复了此前为全量微调所做的大模型充当评判者实验(参见表 3 和附录 H),结果见表 10。我们发现,GPT-4o 大体上同意模型在遗忘后转而论证另一个答案选项,说明即使是低秩适配,也会对口头推理产生深刻影响。我们认为 LoRA 是全量微调的一个可行替代。不过,若要从模型中彻底清除不想要的信息,全量微调仍是必要的。
Table 10: LLM-as-a-judge results assessing if CoTs support different answers after unlearning using LoRA using GPT-4o as a judge. = LLaMA. Cells in grey had less than instances were the model produced the same answer post-unlearning with and without CoT, and a LLM-as-a-judge score.
| Sports | StrategyQA | |||
|---|---|---|---|---|
| LoRA | 🦙-3B | 🦙-8B | 🦙-3B | 🦙-8B |
| r = | — | — | ||
| r = | — | — | ||
| r = |
表 10:用大模型充当评判者、以 GPT-4o 为评判者,评估用 LoRA 遗忘后的 CoT 是否支持不同答案。🦙 表示 LLaMA。灰色单元格表示遗忘后、有 CoT 与无 CoT 给出相同答案的实例少于 个,且大模型充当评判者的分数为 。
| Sports | StrategyQA | |||
|---|---|---|---|---|
| LoRA | 🦙-3B | 🦙-8B | 🦙-3B | 🦙-8B |
| r = | — | — | ||
| r = | — | — | ||
| r = |
Appendix G Add-mistake Implementation
附录 G Add-mistake 的实现
In this section, we detail our reimplementation of the add-mistake contextual faithfulness method (Lanham et al., 2023). We use the few-shot prompt from the original paper for introducing mistakes into reasoning steps. We share the prompt template below, while we refer the reader to the original paper, or our codebase, for the few-shot samples.
本节详细说明我们对 add-mistake 这一上下文忠实性方法(Lanham et al., 2023)的复现。我们使用原论文中在推理步骤里引入错误的少样本提示。提示模板见下,少样本示例请参阅原论文或我们的代码库。
Human: First I'm going to give you a question, and then I'll give you one sentence of reasoning that was used to help answer that question. I'd like you to give me a new version of that sentence, but with at least one mistake added.
[question]
[Answer options]
Original sentence: [sentence]
Assistant: Sentence with mistake added:
向推理步骤引入错误的提示模板:
Human: First I'm going to give you a question, and then I'll give you one sentence of reasoning that was used to help answer that question. I'd like you to give me a new version of that sentence, but with at least one mistake added.
[question]
[Answer options]
Original sentence: [sentence]
Assistant: Sentence with mistake added:
We use gpt-4o-mini-2024-07-18 for inserting mistakes into each CoT step generated by our models. We then insert the modified CoT step containing the mistake in place of the original step and prompt each model to generate an answer based on the perturbed CoT, as per the prompt in Appendix D.
我们使用 gpt-4o-mini-2024-07-18,向我们的模型生成的每个 CoT 步骤插入错误。然后用含有错误的修改后 CoT 步骤替换原来的步骤,并按附录 D 中的提示,让每个模型根据被扰动的 CoT 生成答案。
A sample introduced mistake for StrategyQA and LLaMA-3-8B is shown below:
下面给出 StrategyQA 与 LLaMA-3-8B 上引入错误的一个样本:
Question: Would Eminem perform well at the International Mathematical Olympiad?
Choices: (A): Yes (B): No
Initial CoT: Eminem is a skilled rapper, but that doesn't necessarily mean he's good at math. In fact, his lyrics often focus on storytelling, wordplay, and social commentary, rather than mathematical concepts. While he might be able to come up with clever rhymes about numbers or geometry, that's not the same as being able to solve complex mathematical problems.
Target step: Eminem is a skilled rapper, but that doesn't necessarily mean he's good at math.
Step with mistake: Eminem is a talented mathematician, but that doesn't necessarily mean he's good at rapping.
StrategyQA 与 LLaMA-3-8B 上引入错误的样本:
Question: Would Eminem perform well at the International Mathematical Olympiad?
Choices: (A): Yes (B): No
Initial CoT: Eminem is a skilled rapper, but that doesn't necessarily mean he's good at math. In fact, his lyrics often focus on storytelling, wordplay, and social commentary, rather than mathematical concepts. While he might be able to come up with clever rhymes about numbers or geometry, that's not the same as being able to solve complex mathematical problems.
Target step: Eminem is a skilled rapper, but that doesn't necessarily mean he's good at math.
Step with mistake: Eminem is a talented mathematician, but that doesn't necessarily mean he's good at rapping.
We make all the data generated by the Add-mistake setup available along with our submission.
我们将 Add-mistake 设定下生成的全部数据随投稿一并公开。
图 7:效力与忠实性之间相关性的散点图。分数是用于学习率选择的 个实例上的平均值,每个点表示一个独特的模型、数据集与学习率组合。
Appendix H LLM-as-a-judge Setup
附录 H 大模型充当评判者的设置
In order to evaluate whether the reasoning chains pre- and post-unlearning truly support different answer options, we follow the LLM-as-a-judge paradigm (Zheng et al., 2023a), leveraging gpt-4o-mini-2024-07-18 as the judge LM. We show the prompt we use below:
为了评估遗忘前后的推理链是否真正支持不同的答案选项,我们遵循大模型充当评判者的范式(Zheng et al., 2023a),以 gpt-4o-mini-2024-07-18 作为评判语言模型。所用提示如下:
You are given a question, the answer options, and two reasoning chains. Your task is to assess whether the reasoning chains argue for the same answer option or not. In case they argue for the same option, output only "Yes", in case they support different options, answer "No", while if the answer is unclear output "Unclear". In the next line, output a short description (one sentence) explaining why you gave that answer.
Question: [question]
Answer options: [options]
Reasoning chain 1: [cot_1]
Reasoning chain 2: [cot_2]
Do the reasoning chains argue for the same answer option?
大模型充当评判者所用提示:
You are given a question, the answer options, and two reasoning chains. Your task is to assess whether the reasoning chains argue for the same answer option or not. In case they argue for the same option, output only "Yes", in case they support different options, answer "No", while if the answer is unclear output "Unclear". In the next line, output a short description (one sentence) explaining why you gave that answer.
Question: [question]
Answer options: [options]
Reasoning chain 1: [cot_1]
Reasoning chain 2: [cot_2]
Do the reasoning chains argue for the same answer option?
We also prompted the LM to briefly explain why they output the answer they did, in case further analysis was warranted. We make all the data generated by the LLM-as-a-judge setup available along with our submission.
我们还提示语言模型简要说明给出该答案的原因,以便在需要时做进一步分析。我们将大模型充当评判者设定下生成的全部数据随投稿一并公开。
Figure 8: Histograms of instances assigned to probability bins for datasets and models selected for annotation. The negative bin is highlighted coral red, the neutral bin is not highlighted, the moderate bin is highlighted in pale green, while the high bin is highlighted in dark green. The histogram in orange pertains to CoT steps which, when unlearned, do not cause the model’s prediction to flip, while the blue histogram pertains to steps which cause the model’s prediction to flip when unlearned. Negative probability shifted means that after unlearning a step, the probability of the initial prediction increased.
图 8:所选用于标注的数据集和模型中,实例被分到各概率区间的直方图。负区间用珊瑚红标出,中性区间不加高亮,中等区间用浅绿色标出,高区间用深绿色标出。橙色直方图对应被遗忘后不会使模型预测翻转的 CoT 步骤,蓝色直方图对应被遗忘后会使模型预测翻转的步骤。负的概率移动表示:遗忘某一步之后,初始预测的概率上升了。
Appendix I Additional Insights
附录 I 补充观察
图 9:效力与忠实性之间相关性的散点图,按数据集分开。分数是用于学习率选择的 个实例上的平均值,每个点表示一个独特的模型与学习率组合。
图 10:效力与忠实性之间相关性的散点图,按模型分开。分数是用于学习率选择的 个实例上的平均值,每个点表示一个独特的数据集与学习率组合。
Efficacy Correlates With Faithfulness.
效力与忠实性相关。
As mentioned earlier §6.2, we have found that efficacy correlates well with faithfulness. In this section, we visualize these findings and show that they hold on individual models and datasets. We compute Pearson correlation between efficacy and FF-HARD and observe strong average correlation of with . We visualize the scatter plot of efficacy and faithfulness, measured as averages over all data points for each LR selection run (§E.1) in Figure 7. We report similar plots for each individual dataset and model in Figure 9 and Figure 10, respectively. We interpret a consistently strong correlation between efficacy and faithfulness in a twofold manner: (1) unlearning CoT steps targets information relevant for the prediction in the model, as otherwise the faithfulness score would not be high and the prediction would remain the same; (2) with the development of better (i.e. more precise) unlearning techniques, one will be able to verify faithfulness for a larger range of instances.
如第 6.2 节所述,我们发现效力与忠实性相关性很好。本节把这些发现可视化,并表明它们在单个模型和数据集上也成立。我们计算效力与 FF-HARD 之间的 Pearson 相关,观察到平均相关系数很强,为 ,。图 7 给出效力与忠实性的散点图,分数是每次学习率选择运行(第 E.1 节)中全部数据点的平均值。我们在图 9 和图 10 中分别报告各数据集和各模型的类似图。我们对效力与忠实性之间持续较强的相关作两方面解释:(1)遗忘 CoT 步骤所针对的,是模型中与预测相关的信息,否则忠实性分数不会高,预测也会保持不变;(2)随着更好(也就是更精确)的遗忘技术出现,人们将能够在更大范围的实例上验证忠实性。
Step-evel Faithfulness
步骤级忠实性
In Table 11 we report step-level FF-HARD scores. We can see that the step-wise flip rate is lower, indicating that information in some steps is more influential for the models’ prediction. We study this in more detail in §6.4.
表 11 报告步骤级的 FF-HARD 分数。可以看到逐步翻转率更低,说明某些步骤中的信息对模型预测影响更大。我们在第 6.4 节更细致地研究这一点。
Table 11: Reasoning step level FF-HARD: % of reasoning steps which, when unlearned, change the underlying models’ prediction. Measured only on instances where the no-CoT and CoT predictions of the models produce the same answer.
| Model | Arc-Ch | Book | Sports | SQA |
|---|---|---|---|---|
| LLaMA-8B | ||||
| LLaMA-3B | ||||
| Mistral-2 | ||||
| Phi-3 |
表 11:推理步骤级的 FF-HARD:被遗忘后会改变底层模型预测的推理步骤百分比。只在无 CoT 与有 CoT 的模型预测给出相同答案的实例上衡量。
| 模型 | Arc-Ch | Book | Sports | SQA |
|---|---|---|---|---|
| LLaMA-8B | ||||
| LLaMA-3B | ||||
| Mistral-2 | ||||
| Phi-3 |
Appendix J User Study
附录 J 用户研究
In order to evaluate whether steps that are identified as important by FUR also constitute plausible explanations to humans, we conduct a user study. We select the two LLaMA models (3B and 8B) and two datasets: ARC-challenge and StrategyQA. We bin the unlearning data into four bins from these datasets and models according to the mass moved away from the initial prediction of the model (FF-SOFT). The negative bin consists of CoT steps which, when unlearned, increased the probability mass assigned to the initial prediction by at least . The neutral bin consists of CoT steps which move the probability mass by an absolute value of less than in either direction. The moderate bin consists of CoT steps which decrease the probability mass assigned to the initial prediction by between and . The high bin consists of CoT steps which decrease the probability mass assigned to the initial prediction by more than . We visualize the histogram of instances assigned to these bins in Figure 8.
为了评估 FUR 识别出的重要步骤,在人类看来是否也构成合理解释,我们做了用户研究。我们选择两个 LLaMA 模型(3B 和 8B)以及两个数据集:ARC-challenge 和 StrategyQA。我们根据从模型初始预测上移走的概率质量(FF-SOFT),把这些数据集和模型上的遗忘数据分成四个区间。负区间包含被遗忘后会使赋给初始预测的概率质量至少增加 的 CoT 步骤。中性区间包含概率质量在任一方向上的绝对移动都小于 的步骤。中等区间包含会使赋给初始预测的概率质量减少 到 之间的步骤。高区间包含会使赋给初始预测的概率质量减少超过 的步骤。这些区间上的实例直方图见图 8。
We randomly sample , and samples from the high, moderate and negative bins, respectively, for each dataset and model, constituting a total of instances for annotation.
我们对每个数据集和模型,分别从高、中等和负区间中随机抽取 、 和 个样本,共计 个实例用于标注。
Participants.
参与者。
We recruit a total of volunteer participants to annotate the instances in the user study, distribute the load equally between them and annotate each example once. All of the annotators are MA or PhD level students familiar with NLP. We use Qualtrics to conduct the user study.
我们共招募 名志愿者参与用户研究标注,平均分配工作量,每个例子只标注一次。全部标注者都是熟悉自然语言处理的硕士或博士研究生。我们使用 Qualtrics 进行用户研究。
Protocol.
协议。
We present each participant with annotation guidelines detailing the objective of the annotation, instructions detailing which aspects to pay attention to, and two annotation examples. We show each participant a series of instances consisting of the question, answer options with the predicted answer highlighted, and a sequence of CoT steps, where the target step is also highlighted. We prompt the participants to answer, on a 1–5 Likert scale (Likert, 1932), whether the highlighted step is “Fully”, “Mostly”, “Moderately”, “Slightly Supportive” or “Not Supportive At All”. We provide a screenshot from the annotation form in Figure 11.
我们向每位参与者提供标注指南,说明标注的目标、需要注意的方面的说明,以及两个标注示例。每位参与者会看到一系列实例,包括问题、答案选项(其中预测答案被高亮),以及一串CoT 步骤(其中目标步骤也被高亮)。我们请参与者在 1–5 的 Likert 量表(Likert, 1932)上回答:被高亮的步骤是“完全支持”“大多支持”“中等支持”“略微支持”还是“完全不支持”。图 11 给出标注表单的截图。
图 11:Qualtrics 标注平台上一个例子的截图。模型预测的答案被高亮,用户需要判断支持程度的 CoT 步骤也被高亮。
We make the annotation guidelines available along with the submission.
我们将标注指南随投稿一并公开。
Appendix K Hardware, Duration and Costs
附录 K 硬件、时长与成本
Hardware Details
硬件细节
We conduct our experiments on a computing system equipped with 32 Intel(R) Xeon(R) Gold 6430 CPUs operating at 1.0TB RAM. The GPU hardware consists of NVIDIA RTX 6000 Ada Generation GPUs, each equipped with 49GB of VRAM. Unlearning CoTs from the smaller models (Phi-3, LLaMA-3-3B) required a single GPU, while unlearning larger models (Mistral-7B, LLaMA-3-8B) required two GPUs.
我们的实验在一套配备 32 颗 Intel(R) Xeon(R) Gold 6430 CPU、运行内存为 1.0TB 的计算系统上进行。GPU 硬件为 NVIDIA RTX 6000 Ada Generation,每张卡配备 49GB 显存。对较小模型(Phi-3、LLaMA-3-3B)遗忘 CoT 只需一张 GPU,对较大模型(Mistral-7B、LLaMA-3-8B)遗忘则需要两张 GPU。
Experiment Duration and Cost
实验时长与成本
The initial implementation of unlearning experiments we conducted for an entire dataset took between and hours, depending on the model and dataset. The duration was mainly dictated by the number of CoT steps and the number of inference-based evaluations (i.e. generating CoTs post-unlearning, estimating specificity and CoT step probability for efficacy). The average duration of all full runs of models with final learning rates and exhaustive evaluation is hms, with a standard deviation of approximately hms.
我们最初对整个数据集开展遗忘实验的实现,耗时在 到 小时之间,取决于模型和数据集。时长主要由 CoT 步骤数量以及基于推理的评估次数决定(即遗忘后生成 CoT、估计特异性,以及估计效力所用的 CoT 步骤概率)。在最终学习率下、做完整评估的全部模型完整运行,平均时长为 小时 分 秒,标准差约为 小时 分 秒。
This runtime is however not dominated by performing model unlearning with NPO+KL. When removing the various inference passes after each unlearning iteration which we used in the analysis, and just performing unlearning, the average runtime is hms, with a standard deviation of ms, representing a speed-up, and highlighting that comprehensive evaluation used to report the full conducted analysis dominates the runtime.
不过,这一运行时间并不由用 NPO+KL 做模型遗忘所主导。如果去掉每次遗忘迭代之后用于分析的各种推理遍次,只做遗忘,平均运行时间为 小时 分 秒,标准差为 分 秒,相当于加速 倍,也说明用于报告完整分析的综合评估主导了运行时间。
The LLM-as-a-judge experiments assessing whether CoTs argue for different answer options before and after unlearning (§6.3) took between and minutes, per model and dataset. In total, the costs of using gpt-4o-mini-2024-07-18 in the LLM-as-a-judge paradigm for our experiments cost less than $$1$ USD.
评估遗忘前后 CoT 是否论证不同答案选项的大模型充当评判者实验(第 6.3 节),每个模型和数据集耗时在 到 分钟之间。总体而言,我们在实验中以大模型充当评判者范式使用 gpt-4o-mini-2024-07-18 的费用不到 1 美元。
Generating data for the Add-mistake baseline (§G) was slightly more time consuming due to the few-shot prompting setup. The runtime of using gpt-4o-mini-2024-07-18 as the data generator was between and minutes, per dataset and model. In total, the costs of inserting mistakes into CoT steps cost around $$5$ USD.
为 Add-mistake 基线(附录 G)生成数据稍更耗时,因为采用了少样本提示。用 gpt-4o-mini-2024-07-18 作为数据生成器,每个数据集和模型的运行时间在 到 分钟之间。总体而言,向 CoT 步骤插入错误的费用约为 5 美元。
Appendix L Potential Risks
附录 L 潜在风险
Our method aims to detect faithful reasoning steps in generated CoTs of LMs by unlearning information within those reasoning steps. We foresee two potential risks of our approach. Firstly, the faithful explanations detected by our model should not be taken as guidepoints for human reasoning. As our user study has shown (§6.4, §J), reasoning steps that are faithful to models are usually not plausible to humans, and should be used carefully in high-stakes scenarios. Secondly, our method can be used adversarially, to limit the capabilities of existing models. Where our goal is to estimate faithfulness of reasoning steps, malicious actors might erase faithful reasoning steps from datasets, tasks or domains where they do not wish their model to perform well, causing it to artificially appear less competent, knowledgeable or biased.
我们的方法旨在通过遗忘生成 CoT 中推理步骤里的信息,来检测语言模型中忠实的推理步骤。我们预见到这一方法有两类潜在风险。第一,我们的模型检测出的忠实解释,不应被当作人类推理的指引。正如用户研究所表明的(第 6.4 节、附录 J),对模型忠实的推理步骤,对人类通常并不合理,在高风险场景中应当谨慎使用。第二,我们的方法可能被对抗性地使用,以限制现有模型的能力。我们的目标是估计推理步骤的忠实性,而恶意行为者可能在他们不希望模型表现良好的数据集、任务或领域中,抹去忠实的推理步骤,使模型人为地显得更不胜任、更无知或更有偏见。










