AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models

AlphaEdit：零空间约束的语言模型知识编辑

Junfeng Fang Thanks:  Equal contribution. $ˆ †$Corresponding author: {xiangwang1223, xiangnanhe}@gmail.com . Affiliation:  University of Science and Technology of China, National University of Singapore fangjf1997@gmail.com, jianghc@mail.ustc.edu.cn    Houcheng Jiang    Kun Wang    Yunshan Ma Affiliation:  University of Science and Technology of China, National University of Singapore fangjf1997@gmail.com, jianghc@mail.ustc.edu.cn    Jie Shi Affiliation:  University of Science and Technology of China, National University of Singapore fangjf1997@gmail.com, jianghc@mail.ustc.edu.cn    Xiang Wang    Xiangnan He    Tat-Seng Chua Affiliation:  University of Science and Technology of China, National University of Singapore fangjf1997@gmail.com, jianghc@mail.ustc.edu.cn

Abstract

摘要

Large language models (LLMs) often exhibit hallucinations, producing incorrect or outdated knowledge. Hence, model editing methods have emerged to enable targeted knowledge updates. To achieve this, a prevailing paradigm is the locating-then-editing approach, which first locates influential parameters and then edits them by introducing a perturbation. While effective, current studies have demonstrated that this perturbation inevitably disrupt the originally preserved knowledge within LLMs, especially in sequential editing scenarios. To address this, we introduce AlphaEdit, a novel solution that projects perturbation onto the null space of the preserved knowledge before applying it to the parameters. We theoretically prove that this projection ensures the output of post-edited LLMs remains unchanged when queried about the preserved knowledge, thereby mitigating the issue of disruption. Extensive experiments on various LLMs, including LLaMA3, GPT2-XL, and GPT-J, show that AlphaEdit boosts the performance of most locating-then-editing methods by an average of $36.7\%$ with a single line of additional code for projection solely. Our code is available at: https://github.com/jianghoucheng/AlphaEdit.

大语言模型（LLM）常出现幻觉，输出错误或过时的知识。因此，模型编辑方法应运而生，用于对知识做定向更新。主流范式是「先定位再编辑」：先找到有影响力的参数，再通过引入扰动完成编辑。尽管有效，现有研究已表明，这类扰动不可避免地会破坏 LLM 中原本应保留的知识，在序列编辑场景下尤为严重。为此，我们提出 AlphaEdit：在把扰动作用到参数之前，先将其投影到保留知识的零空间上。我们从理论上证明，这一投影保证编辑后的 LLM 在被问及保留知识时输出保持不变，从而缓解破坏问题。在 LLaMA3、GPT2-XL、GPT-J 等多种 LLM 上的大量实验表明，AlphaEdit 仅需额外一行投影代码，就能把多数「先定位再编辑」方法的性能平均提升 $36.7\%$。代码见：https://github.com/jianghoucheng/AlphaEdit。

1 Introduction

1 引言

Large language models (LLMs) have demonstrated the capability to store extensive knowledge during pre-training and recall it during inference (Brown et al. 2020; Petroni et al. 2019; Roberts et al. 2020; Liu et al. 2024). Despite this, they frequently exhibit hallucinations, producing incorrect or outdated information (Cao et al. 2021; Mitchell et al. 2022a). While fine-tuning with updated knowledge offers a straightforward solution, it is often prohibitively time-consuming (Mitchell et al. 2022b). In sight of this, model editing methods have emerged, enabling updating the target knowledge while preserving other knowledge (Yao et al. 2023; Gupta et al. 2024). Broadly, model editing approaches fall into two categories: (1) parameter-modifying methods, which directly adjust a small subset of parameters (Meng et al. 2023; Jiang et al. 2025), and (2) parameter-preserving methods that integrate additional modules without altering the original parameters (Huang et al. 2023; Yu et al. 2024; Hartvigsen et al. 2023; Zheng et al. 2023).

大语言模型（LLM）在预训练阶段已展现出存储大量知识、并在推理时调用的能力（Brown et al. 2020；Petroni et al. 2019；Roberts et al. 2020；Liu et al. 2024）。尽管如此，它们仍频繁出现幻觉，产出错误或过时信息（Cao et al. 2021；Mitchell et al. 2022a）。用更新后的知识做微调是直观方案，但往往耗时过长（Mitchell et al. 2022b）。鉴于此，模型编辑方法兴起：在更新目标知识的同时尽量保留其他知识（Yao et al. 2023；Gupta et al. 2024）。广义上，模型编辑分为两类：（1）改参方法，直接调整一小部分参数（Meng et al. 2023；Jiang et al. 2025）；（2）保参方法，在不改原参数的前提下接入额外模块（Huang et al. 2023；Yu et al. 2024；Hartvigsen et al. 2023；Zheng et al. 2023）。

In this paper, we aim to explore the parameter-modifying methods for model editing. Concretely, current parameter-modifying methods typically follow the locate-then-edit paradigm (Meng et al. 2022). The basic idea is to first locate influential parameters ${\bm{W}}$ through causal tracing, and then edit them by introducing a perturbation $\bm{\Delta}$ (Li et al. 2024b). The common objective for solving $\bm{\Delta}$ is to minimize the output error on the to-be-updated knowledge, denoted as $e_{1}$. Additionally, the output error on the to-be-preserved knowledge, $e_{0}$, is typically incorporated into the objective function, acting as a constraint to ensure the model’s accuracy on the preserved knowledge.

本文聚焦改参类模型编辑。具体而言，当前改参方法多遵循「先定位再编辑」范式（Meng et al. 2022）：先用因果追踪定位有影响力的参数 ${\bm{W}}$，再引入扰动 $\bm{\Delta}$ 完成编辑（Li et al. 2024b）。求解 $\bm{\Delta}$ 的常见目标是最小化待更新知识上的输出误差 $e_{1}$；同时，待保留知识上的输出误差 $e_{0}$ 通常也会写进目标函数，作为约束以保证模型在保留知识上的正确性。

Figure 1: Comparison between the current methods and AlphaEdit. (a) and (d) exhibit the objectives, where $\lambda$ is the coefficient to keep balance between $e_{0}$ and $e_{1}$ in the objective; (b) and (e) show the distributions of hidden representations after dimensionality reduction within the pre-edited and post-edited LLaMA3, respectively; (c) depicts the output of the post-edited LLaMA3. Best viewed in color.

图 1：现有方法与 AlphaEdit 的对比。(a)(d) 展示目标函数，$\lambda$ 用于在目标中平衡 $e_{0}$ 与 $e_{1}$；(b)(e) 展示编辑前、后 LLaMA3 隐表示降维后的分布；(c) 展示编辑后 LLaMA3 的输出。建议彩色阅读。

Despite their success, the current paradigm faces a critical limitation: it struggles to maintain a balance between knowledge-update error $e_{1}$ and knowledge-preservation error $e_{0}$. Specifically, to prioritize the success of update, prior studies focus more on minimizing $e_{1}$ by assigning a larger weight, while taking insufficient control over $e_{0}$. This could make the LLMs after editing (*i.e.,*the post-edited LLMs) overfit to the updated knowledge. This overfitting would introduce a distribution shift of the hidden representations within LLMs. Figure 2 (b) showcases this shift, where the hidden representations in the post-edited LLaMA-3 (8B) (Meta 2024) diverge from their distribution in the original LLM (*i.e.,*the pre-edited LLM). Worse still, in sequential editing scenarios (Gupta & Anumanchipalli 2024) where the LLM undergoes multiple sequential edits, the accumulation of overfitting gradually erodes the model’s ability to preserve knowledge and generate coherent sentences, eventually leading to model forgetting and model collapse, as depicted in Figure 2 (c).

尽管取得成功，当前范式仍有关键局限：难以在知识更新误差 $e_{1}$ 与知识保留误差 $e_{0}$ 之间维持平衡。具体来说，为优先保证更新成功，既有工作往往给 $e_{1}$ 更大权重，对 $e_{0}$ 控制不足。这会使编辑后的 LLM（即 post-edited LLM）过拟合到更新知识。过拟合会带来 LLM 内部隐表示的分布偏移。图 2 (b) 展示了这一偏移：编辑后 LLaMA-3（8B）（Meta 2024）的隐表示相对原始 LLM（即编辑前 LLM）发生偏离。更糟的是，在序列编辑场景（Gupta & Anumanchipalli 2024）——模型经历多次连续编辑——过拟合的累积会逐渐侵蚀保留知识与连贯造句的能力，最终导致模型遗忘与崩溃，如图 2 (c) 所示。

To address these flaws, we instead remove $e_{0}$ from the current objective, allowing the model to focus solely on minimizing $e_{1}$ without trade-offs. To avoid overfitting to the to-be-updated knowledge, we project the solution of this new objective onto the null space (Wang et al. 2021) of the preserved knowledge before applying it to the model parameters, as shown in Figure 2 (d). By leveraging the mathematical properties of matrix projection and null space, our new objective ensures that the distribution of hidden representations within LLMs remains invariant post-edited, as shown in Figure 2 (e). This invariance enables the post-edited LLMs to reduce $e_{1}$ while keeping $e_{0}$ close to zero, thereby alleviating the issues of model forgetting and model collapse. A detailed theoretical proof is provided in Section 3. In a nutshell, we term the method as AlphaEdit, a simple yet effective editing approach with a null-space constraint for LLMs.

为解决上述缺陷，我们从当前目标中去掉 $e_{0}$，让模型只专注最小化 $e_{1}$、不再做权衡。为避免过拟合待更新知识，我们在把该新目标的解应用到模型参数之前，先将其投影到保留知识的零空间（Wang et al. 2021）上，如图 2 (d)。借助矩阵投影与零空间的数学性质，新目标保证编辑后 LLM 内部隐表示分布保持不变，如图 2 (e)。这种不变性使编辑后模型在压低 $e_{1}$ 的同时把 $e_{0}$ 维持在接近零，从而缓解遗忘与崩溃。详细理论证明见第 3 节。简言之，我们称该方法为 AlphaEdit：一种带零空间约束、简单却有效的 LLM 编辑方法。

To validate the effectiveness of our method, we conducted extensive experiments on multiple representative LLMs, such as GPT-2 XL (Radford et al. 2019) and LLaMA-3 (8B). The results show that, compared to the best-performing baseline methods, AlphaEdit can achieve over a 36.7% performance improvement on average by adding just one line of code to the conventional model editing method, MEMIT (Meng et al. 2023), as illustrated in Figure 2. Furthermore, we empirically verified that this simple optimization can be easily integrated into most existing model editing methods (Ma et al. 2024; Gu et al. 2024), functioning as a plug-and-play enhancement that significantly boosts their performance. This highlights AlphaEdit’s crucial role in efficient knowledge updates for LLMs, enabling broader applications and future advancements in the field.

为验证有效性，我们在 GPT-2 XL（Radford et al. 2019）、LLaMA-3（8B）等代表性 LLM 上做了大量实验。结果显示，相对表现最好的基线，AlphaEdit 仅在传统模型编辑方法 MEMIT（Meng et al. 2023）上额外加一行代码，平均就能带来超过 $36.7\%$ 的性能提升，如图 2。我们还实证验证：这一简单优化可轻松接入多数现有模型编辑方法（Ma et al. 2024；Gu et al. 2024），作为即插即用增强显著抬升其表现。这凸显了 AlphaEdit 对 LLM 高效知识更新 的关键作用，有助于更广应用与后续进展。

2 Preliminary

2 预备知识

2.1 Autoregressive Language Model

2.1 自回归语言模型

An autoregressive large language model (LLM) predicts the next token ${\bm{x}}$ in a sequence based on the preceding tokens. Specifically, the hidden state of ${\bm{x}}$ at layer $l$ within the model, denoted as ${\bm{h}}^{l}$, can be calculated as:

自回归大语言模型（LLM）根据前文预测序列中的下一个 token ${\bm{x}}$。具体地，模型第 $l$ 层中 ${\bm{x}}$ 的隐状态记为 ${\bm{h}}^{l}$，可按下式计算：

$$\displaystyle{\bm{h}}^{l}={\bm{h}}^{l-1}+{\bm{a}}^{l}+{\bm{m}}^{l},\quad{\bm{m}}^{l}={\bm{W}}_{\text{out}}^{l}\,\sigma({\bm{W}}_{\text{in}}^{l}\,\gamma({\bm{h}}^{l-1}+{\bm{a}}^{l})\,),$$

where ${\bm{a}}^{l}$ and ${\bm{m}}^{l}$ represent the outputs of the attention block and the feed-forward network (FFN) layer, respectively; ${\bm{W}}_{\text{in}}^{l}$ and ${\bm{W}}_{\text{out}}^{l}$ are the weight matrices of the FFN layers; $\sigma$ is the non-linear activation function, and $\gamma$ denotes the layer normalization. Following Meng et al. 2022, we express the attention and FFN modules in parallel here.

其中 ${\bm{a}}^{l}$ 与 ${\bm{m}}^{l}$ 分别表示注意力块与前馈网络（FFN）层的输出；${\bm{W}}_{\text{in}}^{l}$ 与 ${\bm{W}}_{\text{out}}^{l}$ 为 FFN 层的权重矩阵；$\sigma$ 为非线性激活，$\gamma$ 表示层归一化。遵循 Meng et al. 2022，此处将注意力与 FFN 写成并联形式。

It is worth noting that ${\bm{W}}_{\text{out}}^{l}$ within FFN layers is often interpreted as a linear associative memory, functioning as key-value storage for information retrieval (Geva et al. 2021). Specifically, if the knowledge stored in LLMs is formalized as $(s,r,o)$ — representing subject $s$, relation $r$, and object $o$ (e.g., $s=\text{``The latest Olympic Game''}$, $r=\text{``was held in''}$, $o=\text{``Paris''}$) — ${\bm{W}}_{\text{out}}^{l}$ associates a set of input keys ${\bm{k}}$ encoding $(s,r)$ with corresponding values ${\bm{v}}$ encoding $(o)$. That is,

值得注意的是，FFN 层中的 ${\bm{W}}_{\text{out}}^{l}$ 常被解读为线性联想记忆，充当信息检索的键值存储（Geva et al. 2021）。具体地，若将 LLM 中存储的知识形式化为 $(s,r,o)$——分别表示主语 $s$、关系 $r$ 与宾语 $o$（例如 $s=\text{``The latest Olympic Game''}$，$r=\text{``was held in''}$，$o=\text{``Paris''}$）——则 ${\bm{W}}_{\text{out}}^{l}$ 把编码 $(s,r)$ 的一组输入键 ${\bm{k}}$ 与编码 $(o)$ 的对应值 ${\bm{v}}$ 关联起来。即：

$$\displaystyle\underbrace{{\bm{m}}^{l}}_{\begin{subarray}{c}{\bm{v}}\end{subarray}}={\bm{W}}_{\text{out}}^{l}\,\underbrace{\sigma({\bm{W}}_{\text{in}}^{l}\,\gamma({\bm{h}}^{l-1}+{\bm{a}}^{l})\,)}_{\begin{subarray}{c}{\bm{k}}\end{subarray}}.$$

This interpretation has inspired most model editing methods to modify the FFN layers for knowledge updates (Hase et al. 2023; Li et al. 2024a; Hu et al. 2024). For simplicity, we use ${\bm{W}}$ to refer to ${\bm{W}}_{\text{out}}^{l}$ in the following sections.

这一解读启发了多数模型编辑方法去修改 FFN 层以更新知识（Hase et al. 2023；Li et al. 2024a；Hu et al. 2024）。为简洁起见，后文用 ${\bm{W}}$ 指代 ${\bm{W}}_{\text{out}}^{l}$。

2.2 Model Editing in LLMs

2.2 LLM 中的模型编辑

Model editing aims to update knowledge stored in LLMs through a single edit or multiple edits (i.e., sequential editing). Each edit modifies the model parameters ${\bm{W}}$ by adding a perturbation $\bm{\Delta}$ in locate-then-edit paradigm. Specifically, suppose each edit needs to update $u$ pieces of knowledge in the form of $(s,r,o)$. The perturbed ${\bm{W}}$ is expected to associate $u$ new ${\bm{k}}$-${\bm{v}}$ pairs, where ${\bm{k}}$ and ${\bm{v}}$ encode $(s,r)$ and $(o)$ of the new knowledge, respectively. Let ${\bm{W}}\in\mathbb{R}^{d_{1}\times d_{0}}$, where $d_{0}$ and $d_{1}$ represent the dimensions of the FFN’s intermediate and output layers. Then, we can stack these keys and values into matrices following:

模型编辑旨在通过单次或多次编辑（即序列编辑）更新 LLM 中存储的知识。在「先定位再编辑」范式下，每次编辑通过对模型参数 ${\bm{W}}$ 加扰动 $\bm{\Delta}$ 来完成。具体地，设每次编辑需更新 $u$ 条形如 $(s,r,o)$ 的知识。扰动后的 ${\bm{W}}$ 期望关联 $u$ 对新的 ${\bm{k}}$-${\bm{v}}$，其中 ${\bm{k}}$、${\bm{v}}$ 分别编码新知识的 $(s,r)$ 与 $(o)$。令 ${\bm{W}}\in\mathbb{R}^{d_{1}\times d_{0}}$，其中 $d_{0}$、$d_{1}$ 分别为 FFN 中间层与输出层维度。可将这些键与值按如下方式堆叠成矩阵：

$${\bm{K}}_{1}=\left[{\bm{k}}_{1}\left|{\bm{k}}_{2}\right|\ldots\mid{\bm{k}}_{u}\right]\in\mathbb{R}^{d_{0}\times u},\quad{\bm{V}}_{1}=\left[{\bm{v}}_{1}\left|{\bm{v}}_{2}\right|\ldots\mid{\bm{v}}_{u}\right]\in\mathbb{R}^{d_{1}\times u}, \tag{3}$$

where the subscripts of ${\bm{k}}$ and ${\bm{v}}$ represent the index of the to-be-updated knowledge. Based on these, the objective can be expressed as:

其中 ${\bm{k}}$、${\bm{v}}$ 的下标表示待更新知识的索引。据此，目标可写为：

$$\begin{gathered}\bm{\Delta}=\argmin_{\bm{\tilde{\Delta}}}\left\|({{\bm{W}}}+\bm{\tilde{\Delta}}){\bm{K}}_{1}-{\bm{V}}_{1}\right\|^{2},\end{gathered} \tag{4}$$

where $\left\|\cdot\right\|^{2}$ denotes the sum of the squared elements in the matrix.

其中 $\left\|\cdot\right\|^{2}$ 表示矩阵元素平方和。

Additionally, let ${\bm{K}}_{0}$ and ${\bm{V}}_{0}$ represent the matrices formed by stacking the ${\bm{k}}$ and ${\bm{v}}$ of the preserved knowledge. Current methods (Meng et al. 2023; Gu et al. 2024) typically incorporate the error involving ${\bm{K}}_{0}$ and ${\bm{V}}_{0}$ to preserve it, as follows:

此外，令 ${\bm{K}}_{0}$、${\bm{V}}_{0}$ 表示由保留知识的 ${\bm{k}}$、${\bm{v}}$ 堆叠而成的矩阵。当前方法（Meng et al. 2023；Gu et al. 2024）通常把涉及 ${\bm{K}}_{0}$、${\bm{V}}_{0}$ 的误差纳入目标以保留这些知识，如下：

$$\begin{gathered}\bm{\Delta}=\argmin_{\bm{\tilde{\Delta}}}(\left\|({{\bm{W}}}+\bm{\tilde{\Delta}}){\bm{K}}_{1}-{\bm{V}}_{1}\right\|^{2}+\left\|({{\bm{W}}}+\bm{\tilde{\Delta}}){\bm{K}}_{0}-{\bm{V}}_{0}\right\|^{2}).\end{gathered} \tag{5}$$

Since ${\bm{K}}_{0}$ and ${\bm{V}}_{0}$ encode the preserved knowledge in LLMs, we have ${\bm{W}}{\bm{K}}_{0}={\bm{V}}_{0}$ (cf. Eqn. 2). Thus, by applying the normal equation (Lang 2012), if the closed-form solution of Eqn. 5 exists, it can be written as:

由于 ${\bm{K}}_{0}$、${\bm{V}}_{0}$ 编码 LLM 中的保留知识，有 ${\bm{W}}{\bm{K}}_{0}={\bm{V}}_{0}$（参见式 2）。于是，由正规方程（Lang 2012），若式 5 存在闭式解，则可写为：

$$\bm{\Delta}=\left({\bm{V}}_{1}-{{\bm{W}}}{\bm{K}}_{1}\right){\bm{K}}_{1}^{T}\left({\bm{K}}_{0}{\bm{K}}_{0}^{T}+{\bm{K}}_{1}{\bm{K}}_{1}^{T}\right)^{-1}. \tag{6}$$

Although ${\bm{K}}_{0}$ is difficult to obtain directly since we hardly have access to the LLM’s full extent of knowledge, it can be estimated using abundant text input (Meng et al. 2023). In practical applications, $100,000$ $(s,r,o)$ triplets from Wikipedia are typically randomly selected to encode ${\bm{K}}_{0}$ (Meng et al. 2023), making ${\bm{K}}_{0}$ a high-dimensional matrix with $100,000$ columns (i.e., ${\bm{K}}_{0}\in\mathbb{R}^{d_{0}\times 100,000}$). See Appendix B.1 for detailed implementation steps.

尽管 ${\bm{K}}_{0}$ 难以直接获得——我们几乎无法访问 LLM 的全部知识——但可用大量文本输入估计（Meng et al. 2023）。实践中通常从 Wikipedia 随机选取 $100{,}000$ 条 $(s,r,o)$ 三元组编码 ${\bm{K}}_{0}$（Meng et al. 2023），使 ${\bm{K}}_{0}$ 成为有 $100{,}000$ 列的高维矩阵（即 ${\bm{K}}_{0}\in\mathbb{R}^{d_{0}\times 100{,}000}$）。详细实现见附录 B.1。

3 Method

3 方法

In this section, we first introduce the concept of the null space and its relationship to model editing (Section 3.1). Based on this, we present the method for projecting a given perturbation $\bm{\Delta}$ onto the null space of the matrix ${\bm{K}}_{0}$, which encodes the persevered knowledge (Section 3.2). Following that, a new editing objective that incorporates the aforementioned projection method is introduced in Section 3.3.

本节先介绍零空间概念及其与模型编辑的关系（第 3.1 节）。据此给出如何将给定扰动 $\bm{\Delta}$ 投影到编码保留知识的矩阵 ${\bm{K}}_{0}$ 的零空间（第 3.2 节）。随后在第 3.3 节引入融入上述投影的新编辑目标。

3.1 Null Space

3.1 零空间

Null space is at the core of our work. Here we first introduce the definition of the left null space (hereafter referred to simply as null space). Specifically, given two matrices ${\bm{A}}$ and ${\bm{B}}$, ${\bm{B}}$ is in the null space of ${\bm{A}}$ if and only if ${\bm{B}}{\bm{A}}=\bm{0}$. See Adam-NSCL (Wang et al. 2021) for more details.

零空间是本文核心。这里先给出左零空间的定义（后文简称为零空间）。具体地，给定矩阵 ${\bm{A}}$、${\bm{B}}$，当且仅当 ${\bm{B}}{\bm{A}}=\bm{0}$ 时，称 ${\bm{B}}$ 落在 ${\bm{A}}$ 的零空间中。更多细节见 Adam-NSCL（Wang et al. 2021）。

In the context of model editing, if the perturbation $\bm{\Delta}$ is projected into the null space of ${\bm{K}}_{0}$ (i.e., ${\bm{\Delta}^{\prime}}{\bm{K}}_{0}=\bm{0}$, where $\bm{\Delta}\textquoteright$ denotes the projected perturbation), adding it to the parameters ${\bm{W}}$ results in:

在模型编辑语境下，若将扰动 $\bm{\Delta}$ 投影到 ${\bm{K}}_{0}$ 的零空间（即 ${\bm{\Delta}^{\prime}}{\bm{K}}_{0}=\bm{0}$，其中 $\bm{\Delta}'$ 为投影后的扰动），再加到参数 ${\bm{W}}$ 上，则有：

$$({\bm{W}}+{\bm{\Delta}\textquoteright}){\bm{K}}_{0}={\bm{W}}{\bm{K}}_{0}={\bm{V}}_{0}. \tag{7}$$

This implies that the projected $\bm{\Delta}$ will not disrupt the key-value associations of the preserved knowledge (i.e., $\{{\bm{K}}_{0},{\bm{V}}_{0}\}$), ensuring that the storage of the preserved knowledge remains intact.

这意味着：投影后的 $\bm{\Delta}$ 不会破坏保留知识的键值关联（即 $\{{\bm{K}}_{0},{\bm{V}}_{0}\}$），从而保证保留知识的存储完好。

Therefore, in this work, before adding perturbation $\bm{\Delta}$ to the model parameters ${\bm{W}}$, we project it onto the null space of ${\bm{K}}_{0}$ to protect the preserved knowledge. This protection allows us to remove the first term — the term focusing on protecting the preserved knowledge — from the objective in Eqn. 5.

因此，本文在把扰动 $\bm{\Delta}$ 加到模型参数 ${\bm{W}}$ 之前，先将其投影到 ${\bm{K}}_{0}$ 的零空间以保护保留知识。这一保护使我们得以从式 5 的目标中去掉第一项——专注于保护保留知识的那一项。

3.2 Null Space Projecting

3.2 零空间投影

In Section 3.1, we briefly explained why $\bm{\Delta}$ should be projected into the null space of ${\bm{K}}_{0}$. In this part, we focus on how to implement this projection.

第 3.1 节简要说明了为何要把 $\bm{\Delta}$ 投影到 ${\bm{K}}_{0}$ 的零空间。本小节聚焦如何实现该投影。

As introduced at the end of Section 2.2, the matrix ${\bm{K}}_{0}\in\mathbb{R}^{d_{0}\times 100,000}$ has a high dimensionality with $100,000$ columns. Hence, directly projecting the given perturbation $\bm{\Delta}$ onto the null space of ${\bm{K}}_{0}$ presents significant computational and storage challenges. In sight of this, we adopt the null space of the non-central covariance matrix ${\bm{K}}_{0}{\bm{K}}_{0}^{T}\in\mathbb{R}^{d_{0}\times d_{0}}$ as a substitute to reduce computational complexity, as $d_{0}$ is typically much smaller than $100,000$. This matrix’s null space is equal to that of ${\bm{K}}_{0}$ (please see Appendix B.2 for detailed proof).

如第 2.2 节末所述，矩阵 ${\bm{K}}_{0}\in\mathbb{R}^{d_{0}\times 100{,}000}$ 列数高达 $100{,}000$。因此直接把给定扰动 $\bm{\Delta}$ 投影到 ${\bm{K}}_{0}$ 的零空间会带来显著的计算与存储负担。有鉴于此，我们改用非中心协方差矩阵 ${\bm{K}}_{0}{\bm{K}}_{0}^{T}\in\mathbb{R}^{d_{0}\times d_{0}}$ 的零空间作为替代以降低复杂度，因为 $d_{0}$ 通常远小于 $100{,}000$。该矩阵与 ${\bm{K}}_{0}$ 的零空间相同（详细证明见附录 B.2）。

Following the existing methods for conducting null space projection (Wang et al. 2021), we first apply a Singular Value Decomposition (SVD) to ${\bm{K}}_{0}({\bm{K}}_{0})^{T}$:

遵循既有零空间投影做法（Wang et al. 2021），我们先对 ${\bm{K}}_{0}({\bm{K}}_{0})^{T}$ 做奇异值分解（SVD）：

$$\left\{{\bm{U}},{\bm{\Lambda}},({\bm{U}})^{T}\right\}=\text{SVD}\left({\bm{K}}_{0}({\bm{K}}_{0})^{T}\right), \tag{8}$$

where each column in ${\bm{U}}$ is an eigenvector of ${\bm{K}}_{0}({\bm{K}}_{0})^{T}$. Then, we remove the eigenvectors in ${\bm{U}}$ that correspond to non-zero eigenvalues[^1], and define the remaining submatrix as $\hat{{\bm{U}}}$. Based on this, the projection matrix ${\bm{P}}$ can be defined as follows:

其中 ${\bm{U}}$ 的每一列是 ${\bm{K}}_{0}({\bm{K}}_{0})^{T}$ 的特征向量。随后去掉 ${\bm{U}}$ 中对应非零特征值的特征向量[^1]，将其余子矩阵记为 $\hat{{\bm{U}}}$。据此，投影矩阵 ${\bm{P}}$ 可定义为：

$${\bm{P}}=\hat{{\bm{U}}}(\hat{{\bm{U}}})^{T}. \tag{9}$$

This projection matrix can map the column vectors of $\bm{\Delta}$ into the null space of ${\bm{K}}_{0}({\bm{K}}_{0})^{T}$, as it satisfies the condition $\bm{\Delta}{\bm{P}}\cdot{\bm{K}}_{0}({\bm{K}}_{0})^{T}=\bm{0}$. The detailed derivation is exhibited in Appendix B.3.

该投影矩阵可将 $\bm{\Delta}$ 的列向量映射到 ${\bm{K}}_{0}({\bm{K}}_{0})^{T}$ 的零空间，因为它满足 $\bm{\Delta}{\bm{P}}\cdot{\bm{K}}_{0}({\bm{K}}_{0})^{T}=\bm{0}$。详细推导见附录 B.3。

Since ${\bm{K}}_{0}$ and ${\bm{K}}_{0}({\bm{K}}_{0})^{T}$ share the same null space, we can derive $\bm{\Delta}{\bm{P}}\cdot{\bm{K}}_{0}=\bm{0}$. Hence, we have:

由于 ${\bm{K}}_{0}$ 与 ${\bm{K}}_{0}({\bm{K}}_{0})^{T}$ 共享同一零空间，可得 $\bm{\Delta}{\bm{P}}\cdot{\bm{K}}_{0}=\bm{0}$。于是：

$$({\bm{W}}+\bm{\Delta}{\bm{P}}){\bm{K}}_{0}={\bm{W}}{\bm{K}}_{0}={\bm{V}}_{0}. \tag{10}$$

This shows the projection matrix ${\bm{P}}$ ensures that the model edits occur without interference with the preserved knowledge in LLMs.

这表明投影矩阵 ${\bm{P}}$ 保证模型编辑不会干扰 LLM 中的保留知识。

3.3 Null-Space Constrained Model Editing

3.3 零空间约束的模型编辑

Section 3.2 has provided how to apply projection to ensure that preserved knowledge is not disrupted. Here, we introduce how to leverage this projection to optimize the current model editing objective.

第 3.2 节给出了如何用投影保证保留知识不被破坏。这里介绍如何借助该投影优化当前模型编辑目标。

Figure 3: Comparison between the paradigms of AlphaEdit and current method. Best viewed in color.

图 3：AlphaEdit 与现有方法范式的对比。建议彩色阅读。

Starting with the single-edit objective in Eqn. 5, the optimization follows three steps: (1) Replace $\bm{\Delta}$ with the projected perturbation $\bm{\Delta}{\bm{P}}$, ensuring that the perturbation would not disrupt the preserved knowledge; (2) Remove the first term involving ${\bm{K}}_{0}$, as Step (1) has already protected the preserved knowledge from being disrupted; (3) Add a regularization term $||\bm{\Delta}{\bm{P}}||^{2}$ to guarantee stable convergence. With these optimizations, Eqn. 5 becomes:

从式 5 的单次编辑目标出发，优化分三步：(1) 用投影扰动 $\bm{\Delta}{\bm{P}}$ 替换 $\bm{\Delta}$，确保扰动不破坏保留知识；(2) 去掉涉及 ${\bm{K}}_{0}$ 的第一项，因为步骤 (1) 已保护保留知识；(3) 加入正则项 $||\bm{\Delta}{\bm{P}}||^{2}$ 以保证稳定收敛。经此优化，式 5 变为：

$$\bm{\Delta}=\argmin_{\bm{\tilde{\Delta}}}\left(\,||({\bm{W}}+\bm{\tilde{\Delta}}{\bm{P}}){\bm{K}}_{1}-{\bm{V}}_{1}||^{2}+\|\bm{\tilde{{\Delta}}}{\bm{P}}\|^{2}\right), \tag{11}$$

where ${\bm{K}}_{1}$ and ${\bm{V}}_{1}$ denote the key and value matrices of to-be-updated knowledge defined in Eqn. 3.

其中 ${\bm{K}}_{1}$、${\bm{V}}_{1}$ 表示式 3 中定义的待更新知识的键、值矩阵。

In sequential editing tasks, during the current edit, we also need to add a term to the objective (cf. Eqn. 11) to prevent the perturbation from disrupting the updated knowledge in previous edits. Let ${\bm{K}}_{p}$ and ${\bm{V}}_{p}$ present the key and value matrices of the previously updated knowledge, analogous to the earlier definitions of ${\bm{K}}_{1}$ and ${\bm{V}}_{1}$. This term should minimize $||({\bm{W}}+\bm{\tilde{\Delta}}{\bm{P}}){\bm{K}}_{p}-{\bm{V}}_{p}||^{2}$ to protect the association. Since the related knowledge has been updated in previous edits, we have ${\bm{W}}{\bm{K}}_{p}={\bm{V}}_{p}$. Hence, this term can be simplified to $||\bm{\tilde{\Delta}}{\bm{P}}{\bm{K}}_{p}||^{2}$, and adding it to Eqn. 11 gives the new objective:

在序列编辑任务中，当前编辑还需在目标中加入一项（参见式 11），防止扰动破坏此前各次编辑已更新的知识。令 ${\bm{K}}_{p}$、${\bm{V}}_{p}$ 表示此前已更新知识的键、值矩阵，定义方式类比 ${\bm{K}}_{1}$、${\bm{V}}_{1}$。该项应最小化 $||({\bm{W}}+\bm{\tilde{\Delta}}{\bm{P}}){\bm{K}}_{p}-{\bm{V}}_{p}||^{2}$ 以保护关联。由于相关知识已在先前编辑中更新，有 ${\bm{W}}{\bm{K}}_{p}={\bm{V}}_{p}$。故该项可简化为 $||\bm{\tilde{\Delta}}{\bm{P}}{\bm{K}}_{p}||^{2}$，将其加入式 11 得到新目标：

$$\bm{\Delta}=\argmin_{\bm{\tilde{\Delta}}}\left(\,||({\bm{W}}+\bm{\tilde{\Delta}}{\bm{P}}){\bm{K}}_{1}-{\bm{V}}_{1}||^{2}+\|\bm{\tilde{\Delta}}{\bm{P}}\|^{2}+||\bm{\tilde{\Delta}}{\bm{P}}{\bm{K}}_{p}||^{2}\right). \tag{12}$$

To facilitate expression, we define the residual vector of the current edit as ${\bm{R}}={\bm{V}}_{1}-{\bm{W}}{\bm{K}}_{1}$. Based on this, Eqn. 12 can be solved using the normal equation (Lang 2012):

为便于表述，将当前编辑的残差向量定义为 ${\bm{R}}={\bm{V}}_{1}-{\bm{W}}{\bm{K}}_{1}$。据此，可用正规方程（Lang 2012）求解式 12：

$$(\bm{\Delta}{\bm{P}}{\bm{K}}_{1}-{\bm{R}}){\bm{K}}_{1}^{T}{\bm{P}}+\bm{\Delta}{\bm{P}}+\bm{\Delta}{\bm{P}}{\bm{K}}_{p}{\bm{K}}_{p}^{T}{\bm{P}}=\mathbf{0}. \tag{13}$$

Solving Eqn. 13 yields the final perturbation $\bm{{\Delta}}_{\textbf{AlphaEdit}}=\bm{\Delta}{\bm{P}}$ which will be added to the model parameters ${\bm{W}}$:

求解式 13 得到最终扰动 $\bm{{\Delta}}_{\textbf{AlphaEdit}}=\bm{\Delta}{\bm{P}}$，将其加到模型参数 ${\bm{W}}$：

$$\bm{{\Delta}}_{\textbf{AlphaEdit}}={\bm{R}}{\bm{K}}_{1}^{T}{\bm{P}}({\bm{K}}_{p}{\bm{K}}_{p}^{T}{\bm{P}}+{\bm{K}}_{1}{\bm{K}}_{1}^{T}{\bm{P}}+{\bm{I}})^{-1}. \tag{14}$$

The detailed derivation process and the invertibility of the term within the brackets are provided in Appendices B.4 and B.5 respectively. This solution $\bm{{\Delta}}_{\textbf{AlphaEdit}}$ could not only store the to-be-updated knowledge in the current edit, but also ensure that both the preserved knowledge and the previously updated knowledge remain unaffected. Furthermore, for better comparison, we also present the commonly used solution in existing methods like MEMIT (Meng et al. 2023) as follows[^2]:

详细推导过程及括号内项的可逆性分别见附录 B.4 与 B.5。解 $\bm{{\Delta}}_{\textbf{AlphaEdit}}$ 不仅能在当前编辑中存入待更新知识，还能保证保留知识与此前已更新知识均不受影响。为便于对比，我们也给出 MEMIT（Meng et al. 2023）等现有方法常用的解[^2]：

$$\bm{{\Delta}}_{\textbf{MEMIT}}={\bm{R}}{\bm{K}}_{1}^{T}({\bm{K}}_{p}{\bm{K}}_{p}^{T}+{\bm{K}}_{1}{\bm{K}}_{1}^{T}+{\bm{K}}_{0}{\bm{K}}_{0}^{T})^{-1}. \tag{15}$$

By comparing Eqn. 14 and 15, it is evident that our approach requires only a minor modification to the standard solution by incorporating the projection matrix ${\bm{P}}$. This makes our method easily integrable into existing model editing algorithms. Figure 3 summarizes this modification from the perspective of convergence objectives. We emphasize that by adding just a single line of code for this modification, the performance of most editing methods could be significantly enhanced, as shown in Figure 2. More detailed experimental results are exhibited in Section 4.

对比式 14 与 15 可见，我们的方法只需在标准解上乘入投影矩阵 ${\bm{P}}$，改动很小，因而易于接入现有模型编辑算法。图 3 从收敛目标角度概括了这一修改。我们强调：仅额外加一行代码完成该修改，多数编辑方法的性能就能显著提升，如图 2。更详细实验结果见第 4 节。

Furthermore, since the projection matrix ${\bm{P}}$ is entirely independent of the to-be-updated knowledge, it only needs to be computed once and can then be directly applied to any downstream editing tasks. Consequently, AlphaEdit introduces negligible additional time consumption compared to baselines, making it both efficient and effective.

此外，由于投影矩阵 ${\bm{P}}$ 完全独立于待更新知识，只需计算一次即可直接用于任意下游编辑任务。因此相对基线，AlphaEdit 引入的额外时间开销可忽略，兼具高效与有效。

4 Experiment

4 实验

In this section, we conduct experiments to address the following research questions:

本节通过实验回答以下研究问题：

• • RQ1: How does AlphaEdit perform on sequential editing tasks compared to baseline methods? Can it mitigate the issues of model forgetting and model collapse exhibited in Figure 2?
• • RQ2: How does AlphaEdit-edited LLM perform on general ability evaluations? Does the post-edited LLM successfully retain its inherent capabilities?
• • RQ3: Can AlphaEdit effectively prevent the model from overfitting to updated knowledge? Specifically, can the post-edited LLM avoid shifts in the distribution of hidden representations?
• • RQ4: Can the performance of baseline methods be significantly improved with a single line of code in AlphaEdit (i.e., the code for projection)?

• • RQ1： 相对基线，AlphaEdit 在序列编辑任务上表现如何？能否缓解图 2 所示的模型遗忘与崩溃？
• • RQ2： 经 AlphaEdit 编辑后的 LLM 在通用能力评测上表现如何？编辑后模型是否成功保留固有能力？
• • RQ3： AlphaEdit 能否有效防止模型过拟合更新知识？具体地，编辑后 LLM 能否避免隐表示分布发生偏移？
• • RQ4： 仅加入 AlphaEdit 中一行投影代码，基线方法的性能能否显著提升？

Method	Model	Counterfact	ZsRE
		Eff.$\uparrow$	Gen.$\uparrow$	Spe.$\uparrow$	Flu.$\uparrow$	Consis.$\uparrow$	Eff.$\uparrow$	Gen.$\uparrow$	Spe.$\uparrow$
Pre-edited	LLaMA3	7.85$\pm$0.26	10.58$\pm$0.26	89.48$\pm$0.18	635.23$\pm$0.11	24.14$\pm$0.08	36.99$\pm$0.30	36.34$\pm$0.30	31.89$\pm$0.22
FT	83.33$\pm$0.37	67.79$\pm$0.40	46.63$\pm$0.37	233.72$\pm$0.22	8.77$\pm$0.05	30.48$\pm$0.26	30.22$\pm$0.32	15.49$\pm$0.17
MEND	63.24$\pm$0.31	61.17$\pm$0.36	45.37$\pm$0.38	372.16$\pm$0.80	4.21$\pm$0.05	0.91$\pm$0.05	1.09$\pm$0.05	0.53$\pm$0.02
InstructEdit	66.58$\pm$0.24	64.18$\pm$0.35	47.14$\pm$0.37	443.85$\pm$0.78	7.28$\pm$0.04	1.58$\pm$0.04	1.36$\pm$0.08	1.01$\pm$0.05
ROME	64.40$\pm$0.41	61.42$\pm$0.42	49.44$\pm$0.38	449.06$\pm$0.26	3.31$\pm$0.02	2.01$\pm$0.07	1.80$\pm$0.07	0.69$\pm$0.03
MEMIT	65.65$\pm$0.47	64.65$\pm$0.42	51.56$\pm$0.38	437.43$\pm$1.67	6.58$\pm$0.11	34.62$\pm$0.36	31.28$\pm$0.34	18.49$\pm$0.19
PRUNE	68.25$\pm$0.46	64.75$\pm$0.41	49.82$\pm$0.36	418.03$\pm$1.52	5.90$\pm$0.10	24.77$\pm$0.27	23.87$\pm$0.27	20.69$\pm$0.23
RECT	66.05$\pm$0.47	63.62$\pm$0.43	61.41$\pm$0.37	526.62$\pm$0.44	20.54$\pm$0.09	86.05$\pm$0.23	80.54$\pm$0.27	31.67$\pm$0.22
AlphaEdit	98.90$\pm$0.10	94.22$\pm$0.19	67.88$\pm$0.29	622.49$\pm$0.16	32.40$\pm$0.11	94.47$\pm$0.13	91.13$\pm$0.19	32.55$\pm$0.22
Pre-edited	GPT-J	16.22$\pm$0.31	18.56$\pm$0.45	83.11$\pm$0.13	621.81$\pm$0.67	29.74$\pm$0.51	26.32$\pm$037	25.79$\pm$0.25	27.42$\pm$0.53
FT	92.15$\pm$0.27	72.38$\pm$0.38	43.35$\pm$0.37	297.92$\pm$0.77	6.65$\pm$0.10	72.37$\pm$0.29	68.91$\pm$0.32	19.66$\pm$0.23
MEND	46.15$\pm$0.50	46.22$\pm$0.51	53.90$\pm$0.48	242.41$\pm$0.41	3.94$\pm$0.03	0.71$\pm$0.04	0.71$\pm$0.04	0.52$\pm$0.03
InstructEdit	50.62$\pm$0.58	51.73$\pm$0.42	56.28$\pm$0.50	245.89$\pm$0.44	4.21$\pm$0.04	0.92$\pm$0.07	0.88$\pm$0.03	0.65$\pm$0.06
ROME	57.50$\pm$0.48	54.20$\pm$0.40	52.05$\pm$0.31	589.42$\pm$0.08	3.22$\pm$0.02	56.42$\pm$0.42	54.65$\pm$0.42	9.86$\pm$0.16
MEMIT	98.55$\pm$0.11	95.50$\pm$0.16	63.64$\pm$0.31	546.28$\pm$0.88	34.89$\pm$0.15	94.91$\pm$0.16	90.22$\pm$0.23	30.39$\pm$0.27
PRUNE	86.15$\pm$0.34	86.85$\pm$0.29	53.87$\pm$0.35	427.14$\pm$0.53	14.78$\pm$0.11	0.15$\pm$0.02	0.15$\pm$0.02	0.00$\pm$0.00
RECT	98.80$\pm$0.10	86.58$\pm$0.28	72.22$\pm$0.28	617.31$\pm$0.19	41.39$\pm$0.12	96.38$\pm$0.14	91.21$\pm$0.21	27.79$\pm$0.26
AlphaEdit	99.75$\pm$0.08	96.38$\pm$0.23	75.48$\pm$0.21	618.50$\pm$0.17	42.08$\pm$0.15	99.79$\pm$0.14	96.00$\pm$0.22	28.29$\pm$0.25
Pre-edited	GPT2-XL	22.23$\pm$0.73	24.34$\pm$0.62	78.53$\pm$0.33	626.64$\pm$0.31	31.88$\pm$0.20	22.19$\pm$0.24	31.30$\pm$0.27	24.15$\pm$0.32
FT	63.55$\pm$0.48	42.20$\pm$0.41	57.06$\pm$0.30	519.35$\pm$0.27	10.56$\pm$0.05	37.11$\pm$0.39	33.30$\pm$0.37	10.36$\pm$0.17
MEND	50.80$\pm$0.50	50.80$\pm$0.48	49.20$\pm$0.51	407.21$\pm$0.08	1.01$\pm$0.00	0.00$\pm$0.00	0.00$\pm$0.00	0.00$\pm$0.00
InstructEdit	55.32$\pm$0.58	53.63$\pm$0.42	53.25$\pm$0.62	412.57$\pm$0.15	1.08$\pm$0.03	3.54$\pm$0.03	4.25$\pm$0.02	3.23$\pm$0.04
ROME	54.60$\pm$0.48	51.18$\pm$0.40	52.68$\pm$0.33	366.13$\pm$1.40	0.72$\pm$0.02	47.50$\pm$0.43	43.56$\pm$0.42	14.27$\pm$0.19
MEMIT	94.70$\pm$0.22	85.82$\pm$0.28	60.50$\pm$0.32	477.26$\pm$0.54	22.72$\pm$0.15	79.17$\pm$0.32	71.44$\pm$0.36	26.42$\pm$0.25
PRUNE	82.05$\pm$0.38	78.55$\pm$0.34	53.02$\pm$0.35	530.47$\pm$0.39	15.93$\pm$0.11	21.62$\pm$0.30	19.27$\pm$0.28	13.19$\pm$0.18
RECT	92.15$\pm$0.26	81.15$\pm$0.33	65.13$\pm$0.31	480.83$\pm$0.62	21.05$\pm$0.16	81.02$\pm$0.31	73.08$\pm$0.35	24.85$\pm$0.25
AlphaEdit	99.50$\pm$0.24	93.95$\pm$0.34	66.39$\pm$0.31	597.88$\pm$0.18	39.38$\pm$0.15	94.81$\pm$0.30	86.11$\pm$0.29	25.88$\pm$0.21

Table 1: Comparison of AlphaEdit with existing methods on the sequential model editing task. Eff., Gen., Spe., Flu. and Consis. denote Efficacy, Generalization, Specificity, Fluency and Consistency, respectively. The best results are highlighted in bold, while the second-best results are underlined.

方法	模型	Counterfact	ZsRE
		Eff.$\uparrow$	Gen.$\uparrow$	Spe.$\uparrow$	Flu.$\uparrow$	Consis.$\uparrow$	Eff.$\uparrow$	Gen.$\uparrow$	Spe.$\uparrow$
Pre-edited	LLaMA3	7.85$\pm$0.26	10.58$\pm$0.26	89.48$\pm$0.18	635.23$\pm$0.11	24.14$\pm$0.08	36.99$\pm$0.30	36.34$\pm$0.30	31.89$\pm$0.22
FT	83.33$\pm$0.37	67.79$\pm$0.40	46.63$\pm$0.37	233.72$\pm$0.22	8.77$\pm$0.05	30.48$\pm$0.26	30.22$\pm$0.32	15.49$\pm$0.17
MEND	63.24$\pm$0.31	61.17$\pm$0.36	45.37$\pm$0.38	372.16$\pm$0.80	4.21$\pm$0.05	0.91$\pm$0.05	1.09$\pm$0.05	0.53$\pm$0.02
InstructEdit	66.58$\pm$0.24	64.18$\pm$0.35	47.14$\pm$0.37	443.85$\pm$0.78	7.28$\pm$0.04	1.58$\pm$0.04	1.36$\pm$0.08	1.01$\pm$0.05
ROME	64.40$\pm$0.41	61.42$\pm$0.42	49.44$\pm$0.38	449.06$\pm$0.26	3.31$\pm$0.02	2.01$\pm$0.07	1.80$\pm$0.07	0.69$\pm$0.03
MEMIT	65.65$\pm$0.47	64.65$\pm$0.42	51.56$\pm$0.38	437.43$\pm$1.67	6.58$\pm$0.11	34.62$\pm$0.36	31.28$\pm$0.34	18.49$\pm$0.19
PRUNE	68.25$\pm$0.46	64.75$\pm$0.41	49.82$\pm$0.36	418.03$\pm$1.52	5.90$\pm$0.10	24.77$\pm$0.27	23.87$\pm$0.27	20.69$\pm$0.23
RECT	66.05$\pm$0.47	63.62$\pm$0.43	61.41$\pm$0.37	526.62$\pm$0.44	20.54$\pm$0.09	86.05$\pm$0.23	80.54$\pm$0.27	31.67$\pm$0.22
AlphaEdit	98.90$\pm$0.10	94.22$\pm$0.19	67.88$\pm$0.29	622.49$\pm$0.16	32.40$\pm$0.11	94.47$\pm$0.13	91.13$\pm$0.19	32.55$\pm$0.22
Pre-edited	GPT-J	16.22$\pm$0.31	18.56$\pm$0.45	83.11$\pm$0.13	621.81$\pm$0.67	29.74$\pm$0.51	26.32$\pm$037	25.79$\pm$0.25	27.42$\pm$0.53
FT	92.15$\pm$0.27	72.38$\pm$0.38	43.35$\pm$0.37	297.92$\pm$0.77	6.65$\pm$0.10	72.37$\pm$0.29	68.91$\pm$0.32	19.66$\pm$0.23
MEND	46.15$\pm$0.50	46.22$\pm$0.51	53.90$\pm$0.48	242.41$\pm$0.41	3.94$\pm$0.03	0.71$\pm$0.04	0.71$\pm$0.04	0.52$\pm$0.03
InstructEdit	50.62$\pm$0.58	51.73$\pm$0.42	56.28$\pm$0.50	245.89$\pm$0.44	4.21$\pm$0.04	0.92$\pm$0.07	0.88$\pm$0.03	0.65$\pm$0.06
ROME	57.50$\pm$0.48	54.20$\pm$0.40	52.05$\pm$0.31	589.42$\pm$0.08	3.22$\pm$0.02	56.42$\pm$0.42	54.65$\pm$0.42	9.86$\pm$0.16
MEMIT	98.55$\pm$0.11	95.50$\pm$0.16	63.64$\pm$0.31	546.28$\pm$0.88	34.89$\pm$0.15	94.91$\pm$0.16	90.22$\pm$0.23	30.39$\pm$0.27
PRUNE	86.15$\pm$0.34	86.85$\pm$0.29	53.87$\pm$0.35	427.14$\pm$0.53	14.78$\pm$0.11	0.15$\pm$0.02	0.15$\pm$0.02	0.00$\pm$0.00
RECT	98.80$\pm$0.10	86.58$\pm$0.28	72.22$\pm$0.28	617.31$\pm$0.19	41.39$\pm$0.12	96.38$\pm$0.14	91.21$\pm$0.21	27.79$\pm$0.26
AlphaEdit	99.75$\pm$0.08	96.38$\pm$0.23	75.48$\pm$0.21	618.50$\pm$0.17	42.08$\pm$0.15	99.79$\pm$0.14	96.00$\pm$0.22	28.29$\pm$0.25
Pre-edited	GPT2-XL	22.23$\pm$0.73	24.34$\pm$0.62	78.53$\pm$0.33	626.64$\pm$0.31	31.88$\pm$0.20	22.19$\pm$0.24	31.30$\pm$0.27	24.15$\pm$0.32
FT	63.55$\pm$0.48	42.20$\pm$0.41	57.06$\pm$0.30	519.35$\pm$0.27	10.56$\pm$0.05	37.11$\pm$0.39	33.30$\pm$0.37	10.36$\pm$0.17
MEND	50.80$\pm$0.50	50.80$\pm$0.48	49.20$\pm$0.51	407.21$\pm$0.08	1.01$\pm$0.00	0.00$\pm$0.00	0.00$\pm$0.00	0.00$\pm$0.00
InstructEdit	55.32$\pm$0.58	53.63$\pm$0.42	53.25$\pm$0.62	412.57$\pm$0.15	1.08$\pm$0.03	3.54$\pm$0.03	4.25$\pm$0.02	3.23$\pm$0.04
ROME	54.60$\pm$0.48	51.18$\pm$0.40	52.68$\pm$0.33	366.13$\pm$1.40	0.72$\pm$0.02	47.50$\pm$0.43	43.56$\pm$0.42	14.27$\pm$0.19
MEMIT	94.70$\pm$0.22	85.82$\pm$0.28	60.50$\pm$0.32	477.26$\pm$0.54	22.72$\pm$0.15	79.17$\pm$0.32	71.44$\pm$0.36	26.42$\pm$0.25
PRUNE	82.05$\pm$0.38	78.55$\pm$0.34	53.02$\pm$0.35	530.47$\pm$0.39	15.93$\pm$0.11	21.62$\pm$0.30	19.27$\pm$0.28	13.19$\pm$0.18
RECT	92.15$\pm$0.26	81.15$\pm$0.33	65.13$\pm$0.31	480.83$\pm$0.62	21.05$\pm$0.16	81.02$\pm$0.31	73.08$\pm$0.35	24.85$\pm$0.25
AlphaEdit	99.50$\pm$0.24	93.95$\pm$0.34	66.39$\pm$0.31	597.88$\pm$0.18	39.38$\pm$0.15	94.81$\pm$0.30	86.11$\pm$0.29	25.88$\pm$0.21

表 1：序列模型编辑任务上 AlphaEdit 与现有方法的对比。Eff.、Gen.、Spe.、Flu.、Consis. 分别表示 Efficacy、Generalization、Specificity、Fluency、Consistency。最优加粗，次优下划线。

4.1 Experimental Setup

4.1 实验设置

We begin by briefly outlining the evaluation metrics, datasets, and baseline methods. For more detailed descriptions of the experimental settings, please refer to Appendix A.

我们先简要概述评价指标、数据集与基线方法。更细致的实验设定见附录 A。

Base LLMs & Baseline Methods. Our experiments are conducted on three LLMs: GPT2-XL (1.5B), GPT-J (6B), and LLaMA3 (8B). We compare our method against several model editing baselines, including Fine-Tuning (FT) (Zhu et al. 2020), MEND (Mitchell et al. 2022a), InstructEdit (Zhang et al. 2024b), ROME (Meng et al. 2022), MEMIT (Meng et al. 2023), PRUNE (Ma et al. 2024), and RECT (Gu et al. 2024). Furthermore, to further validate the generalizability of AlphaEdit, we include three memory-based editing methods (i.e., SERAC (Mitchell et al. 2022b), GRACE (Hartvigsen et al. 2023) and MELO (Yu et al. 2024)) as baselines in Appendix C.5, along with two additional base LLMs: Gemma (Mesnard et al. 2024) and phi-1.5 (Li et al. 2023) in Appendix C.6.

基座 LLM 与基线方法。 实验在三个 LLM 上进行：GPT2-XL（1.5B）、GPT-J（6B）与 LLaMA3（8B）。对比基线包括 Fine-Tuning（FT）（Zhu et al. 2020）、MEND（Mitchell et al. 2022a）、InstructEdit（Zhang et al. 2024b）、ROME（Meng et al. 2022）、MEMIT（Meng et al. 2023）、PRUNE（Ma et al. 2024）与 RECT（Gu et al. 2024）。为进一步验证 AlphaEdit 的泛化性，附录 C.5 另纳入三种基于记忆的编辑方法（即 SERAC（Mitchell et al. 2022b）、GRACE（Hartvigsen et al. 2023）与 MELO（Yu et al. 2024））作基线，附录 C.6 再补充两个基座 LLM：Gemma（Mesnard et al. 2024）与 phi-1.5（Li et al. 2023）。

Datasets & Evaluation Metrics. We evaluate AlphaEdit using two widely adopted benchmarks: the Counterfact dataset (Meng et al. 2022) and the ZsRE dataset (Levy et al. 2017). In line with prior works (Meng et al. 2022), we employ Efficacy (efficiency success), Generalization (paraphrase success), Specificity (neighborhood success), Fluency (generation entropy), and Consistency (reference score) as evaluation metrics. In addition, for comprehensive evaluation, Appendix C.7 presents experiments conducted on three additional datasets: LongformEvaluation (Rosati et al. 2024), MQUAKE (Zhong et al. 2023), and KnowEdit (Zhang et al. 2024d). We encourage interested readers to refer to Appendix C.7 for further details.

数据集与评价指标。 我们在两个广泛使用的基准上评估 AlphaEdit：Counterfact（Meng et al. 2022）与 ZsRE（Levy et al. 2017）。与既有工作（Meng et al. 2022）一致，采用 Efficacy（编辑成功率）、Generalization（改写成功率）、Specificity（邻域成功率）、Fluency（生成熵）与 Consistency（参考分数）作为指标。为更全面评估，附录 C.7 还在 LongformEvaluation（Rosati et al. 2024）、MQUAKE（Zhong et al. 2023）与 KnowEdit（Zhang et al. 2024d）三个额外数据集上报告实验。感兴趣的读者可参阅附录 C.7。

4.2 Performance on Knowledge Update and Preservation (RQ1)

4.2 知识更新与保留表现（RQ1）

To evaluate the performance of different editing methods in terms of knowledge update and retention, we conducted sequential editing on three base LLMs using AlphaEdit and the baseline methods. Table 1 presents the results under a commonly used configuration for the sequential editing task, where 2,000 samples are randomly drawn from the dataset for updates, with 100 samples per edit (i.e., a batch size of 100). For additional experimental results, such as case studies of model outputs after editing, please refer to Appendix C. Based on Table 1, we can draw the following observations:

为评估不同编辑方法在知识更新与保留上的表现，我们在三个基座 LLM 上用 AlphaEdit 与各基线做序列编辑。表 1 给出序列编辑任务常用配置下的结果：从数据集随机抽取 2,000 条样本做更新，每次编辑 100 条（即 batch size 为 100）。更多结果——如编辑后模型输出的案例——见附录 C。基于表 1，我们有如下观察：

• • Obs 1: AlphaEdit achieves superior performance across nearly all metrics and base models. Specifically, in terms of Efficacy and Generalization metrics, AlphaEdit provides an average improvement of 12.54% and 16.78%, respectively, over the best baseline. On LLaMA3, these performance boosts are even more remarkable (i.e., $32.85\%\uparrow$ and $30.60\%\uparrow$). These gains arise from AlphaEdit’s ability to mitigate the trade-off between updating and preserving knowledge.
• • Obs 2: AlphaEdit enhances text generation fluency and coherence. In addition to editing capabilities, AlphaEdit also exhibits substantial improvements in Fluency and Coherence. For instance, on GPT2-XL, AlphaEdit achieves an $18.33\%$ improvement over the strongest baseline, demonstrating that it can preserve both the knowledge and the ability to generate fluent text.

• • 观察 1：AlphaEdit 在几乎所有指标与基座模型上均取得更优表现。 具体地，在 Efficacy 与 Generalization 上，AlphaEdit 相对最强基线平均分别提升 $12.54\%$ 与 $16.78\%$；在 LLaMA3 上提升更为显著（即 $32.85\%\uparrow$ 与 $30.60\%\uparrow$）。这些收益来自 AlphaEdit 缓解了更新与保留知识之间的权衡。
• • 观察 2：AlphaEdit 提升文本生成的流畅性与连贯性。 除编辑能力外，AlphaEdit 在 Fluency 与 Coherence 上也有可观提升。例如在 GPT2-XL 上，相对最强基线提升 $18.33\%$，表明它既能保留知识，也能维持流畅造句能力。

4.3 General Capability Tests (RQ2)

4.3 通用能力测试（RQ2）

Figure 4: F1 scores of the post-edited LLaMA3 (8B) on six tasks (i.e., SST, MRPC, CoLA, RTE, MMLU and NLI) used for general capability testing. Best viewed in color.

图 4：编辑后 LLaMA3（8B）在通用能力测试六项任务（即 SST、MRPC、CoLA、RTE、MMLU 与 NLI）上的 F1。建议彩色阅读。