Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

超越下游问答：重新思考如何评测智能体记忆

Sebastian-Antonio Toma \corresponding    Jean de Kernier    Mike Ply    Oliviu Matei

Abstract

摘要

Agent memory benchmarks evaluate whether an agent’s responses reflect correct knowledge, measuring downstream question-answering accuracy as a proxy for memory quality. We show this evaluation paradigm is incomplete: it conflates memory correctness with retrieval and generation quality, producing both false positives where correct responses mask broken memory and false negatives where incorrect responses blame correct memory for retrieval failures. We introduce FEB-30, a benchmark of 30 fact-evolution trajectories spanning 6 diagnostic categories and 3 difficulty levels, constructed via paradigm inversion: documents are generated from known fact trajectories, yielding deterministic ground truth without human annotation. Each trajectory defines how a fact should evolve across multi-author artifacts (documents, emails, conversations), enabling direct inspection of the memory store after each ingestion step. Applying FEB-30 to a representative memory system, we demonstrate that intrinsic and extrinsic evaluation diverge by an average of 16 percentage points, with 44% of trajectories exceeding meaningful divergence thresholds. Diagnostic decomposition reveals that the system correctly classifies memory operations (create, update, reinforce, supersede) only 20% of the time, a failure entirely invisible to extrinsic evaluation, and that failure modes differ systematically by category and difficulty. We release the benchmark, document generation pipeline, and scoring framework to support reproducible evaluation of agent memory systems.[^1]

智能体记忆基准通常用下游问答准确率代理记忆质量，检查回答是否反映正确知识。我们指出这一评测范式并不完整：它把记忆正确性与检索、生成质量混在一起，既会产生假阳性（回答正确却掩盖了坏掉的记忆），也会产生假阴性（回答错误却把本正确的记忆归咎于检索失败）。我们提出 FEB-30：30 条事实演化轨迹，覆盖 6 个诊断类别与 3 档难度；构造采用范式倒置——由已知事实轨迹生成文档，从而在无需人工标注的情况下得到确定性真值。每条轨迹规定事实应如何随多作者文档演化；评测同时报告内在指标（主张覆盖、更新检测）与外在问答准确率。在一个有代表性的记忆系统上，内外分数平均绝对偏差达 16 个百分点，44% 的轨迹出现有意义的分歧，操作分类准确率仅约 20%。诊断分解进一步显示，多作者归因与更新检测是主要失败模式，而这些信号是外在评测看不见的。

1Amazon Web Services

2Technical University of Cluj-Napoca, Romania

Sebastian.Toma@campus.utcluj.ro

1 Introduction

1 引言

Large language model (LLM) agents operating over extended time horizons require persistent external memory to accumulate knowledge beyond the ephemeral context window (Sumers et al. 2024; Jiang et al. 2026). As these systems grow more complex, with over a dozen distinct architectures published since 2023 (Latimer et al. 2026; Packer et al. 2024; Rasmussen et al. 2025; Chhikara et al. 2025; Zhu et al. 2026), the question of how to evaluate them becomes pressing.

在较长时域上运行的大语言模型（LLM）智能体，需要持久的外部记忆，才能在短暂的上下文窗口之外积累知识（Sumers 等，2024；Jiang 等，2026）。随着这类系统日益复杂——自 2023 年以来已有十多种不同架构发表（Latimer 等，2026；Packer 等，2024；Rasmussen 等，2025；Chhikara 等，2025；Zhu 等，2026），“如何评测它们”变得紧迫。

A survey of current benchmarks reveals a systematic limitation: every existing evaluation measures memory quality extrinsically, by posing questions to the agent and grading its responses (Maharana et al. 2024; Wu et al. 2026; Hu, Wang, and McAuley 2026; Uddin et al. 2026; Liu and Yadav 2026; Jung et al. 2026). This produces a single composite score that conflates memory, retrieval, and generation quality. When the system fails, one cannot identify which component is responsible; when it succeeds, one cannot confirm the memory is actually correct (Toma and Matei 2026).

对现有基准的梳理暴露出一个系统性局限：几乎所有评测都以外在方式衡量记忆质量，即向智能体提问并给回答打分（Maharana 等，2024；Wu 等，2026；Hu、Wang 与 McAuley，2026；Uddin 等，2026；Liu 与 Yadav，2026；Jung 等，2026）。这得到一个把记忆、检索与生成质量揉在一起的综合分数。系统失败时无法定位责任组件；系统成功时也无法确认记忆本身是否正确。

We identify two concrete failure modes that extrinsic evaluation cannot detect:

我们发现两种外部评价无法发现的具体故障模式:

• • False positives (LLM compensation): A strong generation model can produce correct responses despite incorrect or incomplete memory state, by reasoning over partial retrieval results at inference time. Extrinsic evaluation scores this as success.
• • False negatives (retrieval bottleneck): A memory system may correctly store and update facts, yet fail to retrieve them for a specific query due to embedding mismatch or retrieval cutoff. Extrinsic evaluation blames the memory system for a retrieval failure.

• • 假阳性（大模型代偿）： 生成模型很强时，即便记忆状态错误或不完整，仍可能靠推理补全检索片段而答对；外在评测会记成成功。
• • 假阴性（检索瓶颈）： 记忆可能正确存储并更新了事实，却因嵌入不匹配或检索截断未能召回；外在评测会把失败算在记忆头上。

We propose intrinsic evaluation, direct comparison of the memory store contents against known ground truth at each stage of knowledge evolution, as a complementary evaluation paradigm. The relationship mirrors unit versus end-to-end testing in software engineering: intrinsic evaluation checks the memory component’s internal state directly, while extrinsic evaluation only observes end-to-end behavior, so a passing end-to-end test can mask a broken component just as a correct component can fail end-to-end due to an integration bottleneck. A parallel from NLP history motivates the same shift: in the 2010s, word embeddings were evaluated only through downstream task performance until intrinsic evaluation (analogy tasks, similarity correlations) enabled faster progress by isolating representation quality (Baroni, Dinu, and Kruszewski 2014; Schnabel et al. 2015). Our contributions are:

我们提出内在评测：在知识演化的每一阶段，直接把记忆库内容与已知真值比对，作为互补范式。这与软件工程中单元测试相对端到端测试的关系类似：内在评测检查记忆组件的内部状态，外在评测只看端到端行为，因此端到端通过可能掩盖坏组件，正确组件也可能因下游失败而不过。

1. FEB-30, a benchmark of 30 fact-evolution trajectories with deterministic ground truth, spanning 6 diagnostic categories that isolate distinct failure modes invisible to extrinsic evaluation. The benchmark comprises: • Fact-evolution trajectories with atomic claim ground truth at each state. • A paradigm-inversion document generation pipeline using multi-model rotation and 3-judge verification. • A scoring framework measuring both intrinsic (claim coverage, update detection, false update rate) and extrinsic (QA accuracy) metrics.
2. Intrinsic evaluation methodology for agent memory: direct inspection of memory state against known ground truth at each stage of knowledge evolution, enabling diagnostic decomposition of failures by pipeline stage.
3. Empirical evidence that intrinsic and extrinsic evaluation diverge by 16 percentage points on average, with 44% of trajectories exceeding meaningful divergence and failure modes differing systematically by category.

1. FEB-30：由 30 条事实演化轨迹构成的基准，带有确定性真值，覆盖 6 个诊断类别，分别隔离外部评测看不见的不同故障模式。基准包括：• 每个状态都有原子主张真值的事实演化轨迹；• 用多模型轮换与三评委核验的范式反转文档生成管线；• 同时度量内在指标（主张覆盖、更新检测、假更新率）与外在指标（问答准确率）的评分框架。
2. 智能体内在评测方法：在知识演化的每一阶段，对照已知真值直接检查记忆状态，从而能按管线阶段做故障诊断分解。
3. 经验证据：内在与外在评测平均相差 16 个百分点，44% 的轨迹超过有意义的分歧阈值，且故障模式按类别系统性地不同。

2 Related Work

2 相关工作

Agent Memory Benchmarks.

MemoryAgentBench (Hu, Wang, and McAuley 2026) defines four memory competencies (retention, temporal reasoning, knowledge updates, and information integration) evaluated through QA interactions. Memora (Uddin et al. 2026) introduces the FAMA metric that penalizes reliance on invalidated memories, decomposing performance into freshness-aware components. LongMemEval-V2 (Wu et al. 2026) tests long-term memory along five temporal reasoning capabilities. AMA-Bench (Zhao et al. 2026) evaluates long-horizon memory for agentic applications. STATE-Bench (Liu and Yadav 2026) measures whether agents improve at tasks through accumulated experience. MEME (Jung et al. 2026) introduces controlled state trajectories with multi-entity evolution, the closest methodology to ours, though still evaluated through the response layer.

智能体记忆基准。

MemoryAgentBench（Hu、Wang 与 McAuley，2026）定义四类记忆能力（保持、时间推理、知识更新、信息整合），并通过问答交互评测。Memora（Uddin 等，2026）提出 FAMA 指标，惩罚对已失效记忆的依赖，并把表现拆成对新鲜度敏感的分量。LongMemEval-V2（Wu 等，2026）沿五类时间推理能力测试长期记忆。AMA-Bench（Zhao 等，2026）评测面向智能体应用的长时程记忆。STATE-Bench（Liu 与 Yadav，2026）衡量智能体是否会因积累经验而把任务做得更好。MEME（Jung 等，2026）引入带多实体演化的受控状态轨迹，方法上与我们最接近，但仍通过响应层评测。

All these benchmarks share a common structure: pose queries to the agent, grade responses against expected answers. None directly inspects the memory store.

所有这些基准都有一个共同的结构:向代理人提出询问,对预期的答复作出分级答复。 没有人直接检查记忆库.

Memory Architecture Surveys.

Recent surveys (Jiang et al. 2026; Du 2026; Luo et al. 2026; Hu et al. 2025; Huang et al. 2026) catalog memory architectures and operations. Du et al. (Du et al. 2025) define six core memory operations (consolidation, updating, indexing, forgetting, retrieval, condensation), decisions that should be evaluated at the operation level, not only through downstream responses. The Anatomy of Agentic Memory survey (Jiang et al. 2026) identifies benchmark saturation as a key concern, yet proposes more extrinsic benchmarks as the solution.

记忆架构综述。

近期综述（Jiang 等，2026；Du，2026；Luo 等，2026；Hu 等，2025；Huang 等，2026）梳理了记忆架构与操作。Du 等（Du 等，2025）定义六种核心记忆操作（整合、更新、索引、遗忘、检索、浓缩）——这些决策应在操作层评测，而不能只看下游回答。Agentic Memory 解剖综述（Jiang 等，2026）把基准饱和标成关键担忧，却仍把「再做更多外在基准」当作解法。

Conflict Detection Benchmarks.

ConflictBank (Su et al. 2024) benchmarks LLM behavior under knowledge conflicts but evaluates at query time, not at the memory consolidation stage. MAGIC (Lee, Lee, and Kim 2025) tests multi-hop conflict detection in retrieval-augmented generation. These address a related but distinct problem: they test whether a model can reason about conflicts in retrieved context, not whether a memory system correctly detects and resolves conflicts during ingestion.

• 冲突发现基准。 不管怎样,

冲突银行(Su et al. 2024)在知识冲突下将LLM行为作为基准,但在查询时评价,而不是在内存巩固阶段. MAGIC(李,李,金2025)在检索 小心的生成中测试多跳冲突检测. 这些研究涉及一个相关但不同的问题:它们测试一个模型是否能够在检索到的上下文中解释冲突,而不是一个内存 还能否正确发现和解决摄入过程中的冲突。

Paradigm Inversion.

RIKER (Roig 2025) introduced the concept of generating evaluation corpora from known ground truth for RAG evaluation, avoiding the expense and subjectivity of human annotation on real documents. We extend this approach to temporal fact evolution, generating multi-document sequences where ground truth is a trajectory of states rather than a static set of facts.

QQ范式反以何种方式.

RikER( 我还以为2025年或其它时间)引入了一个概念,即从已知的地面真相中生成评价公司,用于RAG评价,避免了人类对辅之以真实文件的注释的成本和主观性绅士。 我们把这种方法推广到时间事实演变,产生多文件序列,其中地面真相 Navad entry 是国家的轨迹而不是静态的事实。

Intrinsic vs. Extrinsic Evaluation.

The distinction between intrinsic and extrinsic evaluation is well-established in NLP for embeddings (Baroni, Dinu, and Kruszewski 2014; Schnabel et al. 2015) and parsing. The Future AGI practitioner framework (Future AGI 2026) explicitly argues that direct memory store inspection is an anti-pattern. A concurrent survey (Toma and Matei 2026) documents this gap systematically across 15 benchmarks, arguing for complementary intrinsic evaluation. We present the first benchmark and empirical results that validate this argument.

内在评测与外在评测。

内在与外在评测的区分，在 NLP 的嵌入（Baroni、Dinu 与 Kruszewski，2014；Schnabel 等，2015）和句法分析里早已成立。Future AGI 实践者框架（Future AGI，2026）明确主张：直接检查记忆库是反模式。同期一篇综述（Toma 与 Matei，2026）系统梳理了 15 个基准中的这一缺口，主张用互补的内在评测补上。我们给出首个验证这一主张的基准与经验结果。

3 Benchmark Design: FEB-30

3 基准设计：FEB-30

FEB-30 (Fact Evolution Benchmark, 30 trajectories) is organized along diagnostic categories and difficulty levels (Section 3.4). The name is a deliberate analogy: February 30 is a date that does not exist, the calendar approaches it asymptotically (February 28, sometimes 29) but never reaches 30. Similarly, recent benchmarks and surveys have moved progressively closer to intrinsic evaluation, introducing memory-specific tasks (Hu, Wang, and McAuley 2026), decomposing scores into memory-aware components (Uddin et al. 2026), and defining controlled state trajectories (Jung et al. 2026), yet remain on the extrinsic side of the threshold, never directly inspecting the memory store, as illustrated by (Toma and Matei 2026). FEB-30 crosses it.

FEB-30（Fact Evolution Benchmark，30 条轨迹）按诊断类别与难度组织（第 3.4 节）。名称是刻意的类比：2 月 30 日并不存在，日历可以逼近（2 月 28，偶有 29）却永不到 30。同样，近期基准与综述逐步靠近内在评测——引入记忆专用任务（Hu、Wang 与 McAuley，2026）、把分数拆成记忆感知分量（Uddin 等，2026）、定义受控状态轨迹（Jung 等，2026）——却仍停在外在一侧，从未直接检查记忆库本身。

3.1 Diagnostic Categories

3.1 诊断类别

Each category targets a specific gap in extrinsic-only evaluation (Table 1):

每个类别针对仅做外在评测时的一个具体缺口（表 1）：

Cat	Name	Design constraint
C1	LLM Compensation	Memory wrong in a way QA still works
C2	Retrieval Bottleneck	Memory correct but hard to retrieve
C3	Classification	Focus on create/update/reinforce/conflict decision correctness
C4	Synthesis Quality	Multi-observation $\rightarrow$ correct fact formation
C5	Temporal Correctness	Fact must evolve correctly over time
C6	Multi-Author	Artifacts from different author roles

Table 1: FEB-30 diagnostic categories with design constraints. C1 and C2 are designed to produce intrinsic-extrinsic divergence; C3–C6 test dimensions where intrinsic evaluation provides richer signal than extrinsic.

类别	名称	设计约束
C1	LLM 代偿	记忆错了，但问答仍能答对
C2	检索瓶颈	记忆正确，但很难检索到
C3	分类	关注创建/更新/强化/冲突决策是否正确
C4	综合质量	多条观测 → 正确形成事实
C5	时间正确性	事实必须随时间正确演化
C6	多作者	工件来自不同作者角色

表 1：FEB-30 诊断类别及其设计约束。C1、C2 被设计成会制造内在–外在分歧；C3–C6 测的是内在评测能给出更丰富信号的维度。

C1 (LLM Compensation) constructs scenarios where memory state is incorrect (e.g., outdated fact not updated, contradictions unresolved) but a strong LLM still produces correct answers by reasoning over retrieved fragments, so a system that never updates its facts scores well extrinsically.

**C1(LLM补偿)**构建了内存状态不正确的情景(例如过时的事实没有更新,矛盾没有解决),但强烈的LLM仍然通过推理而不是取回的碎片产生正确的答案,因此一个从未更新其事实的系统得分远在外.

C2 (Retrieval Bottleneck) constructs scenarios where facts are correctly stored but queries are phrased to produce low embedding similarity (vocabulary mismatch, different abstraction level). The memory is correct, only retrieval fails.

C2（检索瓶颈） 构造的场景是：事实已正确写入，但查询措辞会使嵌入相似度偏低（词表错配、抽象层级不同）。记忆本身是对的，失败只在检索。

C3 (Classification Accuracy) tests whether the system correctly classifies each observation as a create, update, reinforce, or conflict operation, decisions extrinsic evaluation cannot observe directly.

C3(分类准确性) 在此方面测试系统是否正确地将每项观测归类为创建、更新、强化或冲突操作,外部评价的决定不能直接观察。

C4 (Synthesis Quality) tests whether multiple observations from different artifacts merge into correct, well-formed facts, with ground truth specifying which should form a single fact versus separate facts.

C4(综合质量) 检验不同文物的多种观测 什么是合并成正确,造型良好的事实 我们走实事求是地明确哪些是单一的事实,哪些是单独的事实.

C5 (Temporal Correctness) tests whether the memory state correctly reflects temporal evolution: linear progression (A$\rightarrow$B$\rightarrow$C), retraction, and gradual drift across artifacts.

**C5(时间正确性) 琌 测试内存状态是否正确反映了时间演化:线性进化(A $\rightarrow$إ B$\rightarrow$C),还原,并逐渐漂移到文物之间.

C6 (Multi-Author Attribution) tests whether source identity affects consolidation correctly: corroboration from different authors should raise confidence differently than repetition from one, and disagreement should trigger conflict detection.

C6(多作者归属) 测试源身份是否对合并产生正确影响:不同作者的证实应提高信任度,而不是重复,分歧应触发冲突检测.

3.2 Difficulty Levels

3.2 难度等级

Level	Docs	States	Signal clarity
L1	1–2	2–3	Claims nearly explicit
L2	3–4	3–4	Claims embedded in context
L3	4–5	4–5	Requires inference

Table 2: Difficulty levels in FEB-30.

难度	文档数	状态数	信号清晰度
L1	1–2	2–3	主张几乎明示
L2	3–4	3–4	主张埋在上下文里
L3	4–5	4–5	需要推理

表 2：FEB-30 的难度分级。

At L1 (Table 2), target claims appear almost verbatim in generated documents with minimal surrounding noise. At L2, claims are embedded within realistic document context and require extraction from longer passages. At L3, claims must be inferred from combinations of statements, and documents contain substantial unrelated content that could mislead extraction systems.

追究责任的L1(表2)几乎是逐字逐句地出现在产生的文件中,周围噪音很小。 在L2中,索赔包含在现实的文件背景中,需要从较长的段落中提取。 在L3中,索赔必须从各种报表的组合中推断出来,而文件包含大量无关的内容,可能误导麒取系统。

3.3 Trajectory Schema

3.3 轨迹模式

Each trajectory is defined as a JSON structure specifying:

可通过J的唯一途径 Glens 和 JSON 结构确定每个轨迹:

• • A unique trajectory ID, category, difficulty, and domain assignment.
• • A type indicating the evolution pattern (linear-evolution, contradiction, scope-expansion, retraction, stable, or gradual-drift).
• • A sequence of states, each containing a set of atomic claims representing the expected memory contents after processing documents up to that state.
• • Per-state docClaims, the subset of claims that the document for that state should contain (new information only, not cumulative repetition).
• • An expected action per state transition (create, update, reinforce, conflict, or supersede).
• • Author role and document type metadata for multi-author trajectories.

之间关系-了 —— 独特的 页 总 域 除其它外,
)、曾多次表示演变模式(线性演变、矛盾、扩展范围、收回、稳定或渐进使之能够)。
我们应该斤计可持续发展问题。

• 各州闺门索赔**,该州文件应包含的一组索赔(只有新资料,所遇到的是累积重复)。
  • 你把每个州过渡的预期行动**(建立、在国内更新、加强、冲突或仍有替代作用)。
• ** 多作者轨迹or作用**和文件类型元数据。

Claims are required to be atomic (one assertable proposition per claim) and self-contained (understandable without reference to other claims). This ensures scoring is unambiguous: each claim either is or is not entailed by the system’s fact store.

要求索赔必须是原子性(每项索赔一个可断言的命题)和自足(无须提及其他索赔即可理解)。 这保证了打分是明确的:每个索赔要么是或不是系统的年龄。

The key design choice is docClaims vs. claims: documents contain only new information per state. The system must retain prior knowledge across document ingestions, testing actual memory persistence, not single-document extraction.

关键设计选择是 docClaims 对 claims：每个状态的文档只含该状态的新信息。系统必须在多次文档摄入之间保留先前知识，测的是真正的记忆持久性，而不是单文档抽取。

3.4 Distribution

3.4 分布

FEB-30 contains 30 trajectories across 6 categories, distributed across difficulty levels (10 at L1, 10 at L2, 10 at L3). Trajectories span 6 technical domains (infrastructure, security, financial operations, machine learning, data engineering, and containers) to avoid domain-specific bias. These trajectories instantiate the benchmark in a technical and business-operations setting, but the methodology is domain-agnostic as the same paradigm-inversion construction can produce trajectories for any domain in which facts evolve across multi-author artifacts, by substituting the domain’s own topics and observation categories. The total corpus comprises 147 generated documents: 127 trajectory state documents and 20 noise documents (4 per C2 trajectory to test retrieval under distraction).

FEB-30 含 30 条轨迹、跨 6 个类别，并按难度均分（L1、L2、L3 各 10 条）。轨迹覆盖 6 个技术域（基础设施、安全、财务运营、机器学习、数据工程、容器），以降低领域偏见。这些轨迹把基准落在技术与业务运营场景，但方法本身与领域无关：同一套范式反转构造，只要换成该领域自己的主题与观测类别，就能为任何「事实会跨多作者工件演化」的领域生成轨迹。语料合计 147 篇生成文档：127 篇轨迹状态文档，外加 20 篇噪声文档（每个 C2 轨迹 4 篇，用于在干扰下测检索）。

4 Document Generation Pipeline

4 文档生成管线

We adopt paradigm inversion (Roig 2025): rather than annotating ground truth on real documents (expensive and subjective), we generate documents from known fact trajectories, making the trajectory itself the ground truth. Algorithm 1 formalizes the pipeline.

我们采用范式反转（Roig，2025）：不是给真实文档标注真值（昂贵且主观），而是从已知事实轨迹生成文档，让轨迹本身成为真值。算法 1 把这条管线形式化。

1:  for each state $s_{i}\in T$ do
2:   $m\leftarrow\mathcal{M}[i\bmod|\mathcal{M}|]$ // rotate model
3:   $r\leftarrow\textsc{SelectStyle}(\textsc{type}(s_{i}),\textsc{diff}(T),\mathcal{R})$
4:   for $k=1$ to $3$ do
5:    if $\textsc{diff}(T)=\text{L3}$ then
6:     $h\leftarrow\textsc{SelectHost}(\textsc{domain}(T),\mathcal{R})$
7:     $d_{k}\leftarrow\textsc{Inject}(m,\hat{C}_{i},h)$
8:    else
9:     $d_{k}\leftarrow\textsc{Generate}(m,\hat{C}_{i},r,\textsc{diff}(T))$
10:    end if
11:    $V\leftarrow\bigwedge_{j\in\mathcal{J}}\left[\textsc{Entail}(j,d_{k},\hat{C}_{i})\wedge\lnot\textsc{Halluc}(j,d_{k},T)\right]$
12:    if $V=\textit{true}$ then
13:     $D_{T}\leftarrow D_{T}\cup\{d_{k}\}$; break
14:    else
15:     $m\leftarrow\mathcal{M}[(i+k)\bmod|\mathcal{M}|]$
16:    end if
17:   end for
18:  end for
19:  $D_{T}\leftarrow D_{T}\cup\textsc{GenerateNoise}(\mathcal{M},\textsc{cat}(T),\textsc{domain}(T))$
20:  Constraint: $\forall m\in\mathcal{M}:|\{d\in D_{T}:\textsc{gen}(d)=m\}|/|D_{T}|\leq 0.3$
21:  return $D_{T}$

Algorithm 1 Document Generation with Verification

1:  for each state $s_{i}\in T$ do
2:   $m\leftarrow\mathcal{M}[i\bmod|\mathcal{M}|]$ // rotate model
3:   $r\leftarrow\textsc{SelectStyle}(\textsc{type}(s_{i}),\textsc{diff}(T),\mathcal{R})$
4:   for $k=1$ to $3$ do
5:    if $\textsc{diff}(T)=\text{L3}$ then
6:     $h\leftarrow\textsc{SelectHost}(\textsc{domain}(T),\mathcal{R})$
7:     $d_{k}\leftarrow\textsc{Inject}(m,\hat{C}_{i},h)$
8:    else
9:     $d_{k}\leftarrow\textsc{Generate}(m,\hat{C}_{i},r,\textsc{diff}(T))$
10:    end if
11:    $V\leftarrow\bigwedge_{j\in\mathcal{J}}\left[\textsc{Entail}(j,d_{k},\hat{C}_{i})\wedge\lnot\textsc{Halluc}(j,d_{k},T)\right]$
12:    if $V=\textit{true}$ then
13:     $D_{T}\leftarrow D_{T}\cup\{d_{k}\}$; break
14:    else
15:     $m\leftarrow\mathcal{M}[(i+k)\bmod|\mathcal{M}|]$
16:    end if
17:   end for
18:  end for
19:  $D_{T}\leftarrow D_{T}\cup\textsc{GenerateNoise}(\mathcal{M},\textsc{cat}(T),\textsc{domain}(T))$
20:  Constraint: $\forall m\in\mathcal{M}:|\{d\in D_{T}:\textsc{gen}(d)=m\}|/|D_{T}|\leq 0.3$
21:  return $D_{T}$

算法 1 带核验的文档生成

4.1 Multi-Model Rotation

4.1 多模型轮换

Documents are generated using 6 models across 5 providers, rotated per state index within each trajectory to simulate different human authors with different writing styles. No single model generates more than 30% of total documents. Following evidence that multi-source synthetic data mitigates distribution collapse (Schaffelder and Gatt 2026) and reduces single-generator bias (van Elburg, van der Putten, and Marx 2025), this additionally simulates the natural stylistic variation of multi-author corpora.

文件是使用5个姗姗来送审器的6个模型生成的,在每个轨迹内按州指数旋转,以模拟不同人类作者的写作风格不同. 单项文件模型生成的文档 我只是在尚未完成的文档中生成 请将本函及其附件作为安全理事会的更加有效地. 在有证据表明多源合成数据能缓解分布崩溃(Schaffelder and Gatt 兄弟2026),并减少单源偏差(van Elburg, van der Putten, and Marx 2025旨在使多源子公司自然结构变异)之后,这种额外模拟.

The generation models are all different from the system’s extraction model (which uses a different LLM family), avoiding the stylistic bias that inflates scores when the same model generates and extracts (van Elburg, van der Putten, and Marx 2025).

生成模型与系统的提取模型(使用不同的LLM家族)指出了审议会议的所有不同之处,避免了同一模型产生和提取时夸大分数的风格偏差(van Elb 49, van der Putten, 和 Marx 2025).

4.2 Style-Grounded Generation

4.2 风格锚定生成

Documents follow a realism gradient across difficulty levels:

文件在困难级别促进可持续发展:

1. Style-grounded (L1/L2): An LLM generates a new document for the trajectory claims but mimics the tone, vocabulary, and structure of a real reference document.
2. Hybrid injection (L3): Claims are inserted into actual human-written documents from unrelated domains, producing text that is approximately 90% authentic.

1. ** 定型(L1/L2)**: LLM 生成展开了轨道主张的新文档,但模仿了真实参考文档的语气,词汇和结构.
2. 15. ** Hybrid 注射(L3)**: 索赔被插入来自无关领域的实际人文文件,产生约90%的正本文本。

For style-grounded generation, documents are typed (meeting notes, email, architecture review, status update, executive briefing) and matched to style references from a curated set of 12 AMI meeting summaries (Carletta et al. 2005) and 12 Enron emails (Klimt and Yang 2004) selected for diversity in length, formality, and structure.

对于基于样式的生成,文件会打入(会议笔记、电子邮件、架构审查、 乔达更新、执行简报),并与12个AMI会议摘要(Carletta等人,2005年)和12诶安然电子邮件(Klimt和Yang 604年)的样式参考相匹配。

Document length and noise scale with difficulty:

文件长度和噪音尺度有困难:

• • L1: 200–400 words, no noise claims, target claims nearly explicit.
• • L2: 400–800 words, 2–3 unrelated noise claims, target claims embedded in context.
• • L3: 600–1000 words, 4–5 noise claims, subtle and ambiguous signal.

• L1: 你他妈的200-400字,应继续不提出噪音要求, 在这些要求中,目标几乎明确。

• L2:400-800字,2-3个无关的噪音索赔,嵌入上下文的目标索赔。
  龛 - ** 都是L 我请客的**:L并为600-1000字,4-5个噪声,微妙和模糊的的一种信号。

For L3 trajectories, host documents are selected from unrelated domains (e.g., product design meetings, energy pipeline discussions) to ensure no semantic overlap with the trajectory topic, making scoring unambiguous: any relevant fact the system extracts must come from the injected section.

对于L3轨迹,主机文件是从不相关的领域(如产品设计会议,能源管道讨论)中选择的,以确保与轨迹主题不发生语义重叠,使评分明确:系统提取的任何相关事实必须来自注入部分.

4.3 Verification Pipeline

4.3 校验管线

The verification criterion (Algorithm 1, line 6) requires unanimous agreement from 3 independent judge models (different from the generators), following the multi-judge grounding methodology of (Uddin et al. 2026) and (Roig 2025). Each judge assesses: (1) all target claims are inferable from the document, (2) no extra factual information beyond the trajectory is introduced, and (3) the document reads naturally.

核查标准(算法1,第6行)要求 4. 3 个独立法官模型(不同于发电机)的一致同意,遵循多法官的论证方法:Uddin 在这个模型中(Uddin等人,2026年)和(Roig,2025年)。 每个法官评估:(1) 所有目标索赔均可从文件中推断,(2) 没有的法律地位以外的额外事实资料将提供,(3) 文件自然阅读。

Additionally, documents underwent human review, with 11% requiring content corrections (forward leakage, narrative contradictions, cross-document inconsistencies, numeric mismatches) and 28% requiring scaffolding removal (code fences, LLM preamble such as “generated email about…” or “injected claims into…”) that does not affect extraction.

Arielm.c.c.c.和28%要求脚手架拆除(代码围栏、LLM序言,如“关于提出的各项建议......”或“注入......”),不影响提取。

5 Evaluation Framework

5 评测框架

We evaluate each system using both intrinsic and extrinsic metrics on the same trajectories, enabling direct measurement of their divergence.

我们在同一轨迹上使用内在和外在的衡量尺度来评估每个系统,从而能够直接测量其差异。

5.1 Intrinsic Metrics

5.1 内在指标

After processing all artifacts for a trajectory $T$, we compare the system’s fact store $F$ against the expected claims $C_{i}$ at each state $s_{i}$. Let $\text{Entail}(f,c)=1$ if fact $f$ entails claim $c$ (determined by embedding similarity $>0.8$ followed by LLM judge verification).

在为轨迹$T$处理所有文物之后,我们将该系统的事实库$F$与每个国家的预期索赔$C_{i}$进行比较。 如果事实$f$涉及索赔$c$(由所得到的类似性$>0.8$加上LLM法官的核查确定)。

Claim coverage at state $s_{i}$:

** 第$s_{i}$号邦:

$$\text{Cov}(s_{i})=\frac{|\{c\in C_{i}:\exists f\in F,\ \text{Entail}(f,c)=1\}|}{|C_{i}|} \tag{1}$$

Fact accuracy across the trajectory:

** 横跨轨道的精确度**:

$$\text{Acc}(T)=\frac{1}{|T|}\sum_{i=0}^{|T|-1}\text{Cov}(s_{i}) \tag{2}$$

Update detection recall, where $U=\{s_{i}:a_{i}\in\{\text{update},\text{supersede}\}\}$ is the set of states requiring a change:

更新检测召回，其中 $U=\{s_{i}:a_{i}\in\{\text{update},\text{supersede}\}\}$ 是需要发生变更的状态集合：

$$\text{UDR}(T)=\frac{|\{s_{i}\in U:\text{changed}(F,s_{i-1},s_{i})\}|}{|U|} \tag{3}$$

False update rate, where $R=\{s_{i}:a_{i}=\text{reinforce}\}$:

显现出一个真实性,

$$\text{FUR}(T)=\frac{|\{s_{i}\in R:\text{changed}(F,s_{i-1},s_{i})\}|}{|R|} \tag{4}$$

Additionally, we report fact count (number of active facts per trajectory, detecting duplication or loss) and confidence MAE ($\frac{1}{|T|}\sum_{i}|c_{\text{expected}}(s_{i})-c_{\text{actual}}(s_{i})|$).

此外,审查报告** 快走,报告** * * *

5.2 Extrinsic Metrics

5.2 外在指标

For each trajectory, we generate diagnostic questions whose answers depend on the final memory state. The system retrieves relevant facts via its standard retrieval mechanism, then an LLM generates an answer. A judge model scores each answer as YES (1.0), PARTIAL (0.5), or NO (0.0) against the expected answer derived from ground-truth claims.

对每条轨迹，我们生成诊断性问题，其答案取决于最终记忆状态。系统先用标准检索机制取回相关事实，再由 LLM 生成回答。评委模型对照由真值主张导出的期望答案，把每个回答打成 YES（1.0）、PARTIAL（0.5）或 NO（0.0）。

5.3 Divergence Metric

5.3 分歧指标

The key metric is mean absolute divergence (MAD) between intrinsic and extrinsic scores:

关键指标是内在分与外在分之间的平均绝对分歧（MAD）：

$$\text{MAD}=\frac{1}{N}\sum_{T}|\text{Acc}(T)-\text{Extr}(T)| \tag{5}$$

MAD quantifies how much information is lost by relying on extrinsic evaluation alone, without requiring an arbitrary threshold. As a secondary interpretable metric, we report the divergence rate: the percentage of trajectories where the two evaluations disagree by more than $\tau=0.15$:

MAD将仅依靠外部评价就损失了多少信息,而不需要任意的阈值。 作为二级可解释的衡量标准,我们报告潜水率:两项评价将性别问题纳入$\tau=0.15$以上的轨迹百分比:

$$\text{DivRate}=\frac{1}{N}\sum_{T}\mathbb{1}\left[|\text{Acc}(T)-\text{Extr}(T)|>\tau\right] \tag{6}$$

6 Experiments

6 实验

6.1 System Under Test

6.1 被测系统

Figure 1: System architecture showing the write path (ingestion and fact synthesis), intrinsic and extrinsic scoring pipelines. For each trajectory state $s_{i}$, intrinsic evaluation compares ground-truth claims $C_{i}$ against the fact store via embedding retrieval and LLM entailment. Extrinsic evaluation queries the store and judges the response against expected answers. $\star$ = proposed here.

图1:显示写入路径(摄入 一个事实合成)的系统架构,的授权和外部评分管道。 对于每个轨迹状态 我叫$s_{i}$, 内在 他们通过嵌入检索和 LLM 内含将XXMATH 你以为 在我面前是真实的。 extrins 天呐 Extrinsic 评价询问商店 并根据预期的答案判断答案.$\star$= 在此提议。

To demonstrate the benchmark’s diagnostic utility, we evaluate a representative agent memory system (Figure 1). The system follows the dominant architectural pattern shared by virtually all contemporary memory systems, Hindsight (Latimer et al. 2026), Mem0 (Chhikara et al. 2025), AriadneMem (Zhu et al. 2026), and commercial agent memory services alike: (1) extract structured observations from artifacts (documents, emails, meeting transcripts), (2) embed them, (3) retrieve related existing facts via vector search, (4) classify the relationship via LLM (create, update, reinforce, conflict, or skip), and (5) synthesize into evolving facts with history. This extract-embed-classify-synthesize pipeline represents the current state of the art; despite surface differences in implementation details (confidence stepping in Hindsight, entropy-aware gating in AriadneMem, Add/Update/No-op in Mem0), the core information flow is shared. The specific system is not the contribution; any system implementing this pattern could be substituted. Our goal is to demonstrate what the benchmark reveals about this class of systems, not to evaluate a particular implementation.

为展示该基准的诊断效用，我们评测一套有代表性的智能体记忆系统（图 1）。它遵循当代几乎所有记忆系统共有的主流架构模式——Hindsight（Latimer 等，2026）、Mem0（Chhikara 等，2025）、AriadneMem（Zhu 等，2026）以及商业智能体记忆服务皆然：（1）从工件（文档、邮件、会议纪要）抽取结构化观测；（2）做嵌入；（3）用向量检索找到相关既有事实；（4）由 LLM 分类关系（创建、更新、强化、冲突或跳过）；（5）综合成带历史的演化事实。这条「抽取–嵌入–分类–综合」管线代表当前主流；尽管实现细节表面不同（Hindsight 的置信步进、AriadneMem 的熵感知门控、Mem0 的 Add/Update/No-op），核心信息流是共享的。具体系统本身不是贡献；任何实现该模式的系统都可替换。我们的目标是展示基准对这类系统能揭示什么，而不是评价某一实现。

We adopt a two-layer terminology: observations are static, atomic factual statements extracted from a single artifact (typed into categories such as decisions, commitments, or milestones), while facts are the evolving, synthesized knowledge units that the system maintains over time. Observations are immutable once extracted; facts are the persistent memory that gets created, reinforced, updated, or superseded as new observations arrive. Not all observations aggregate into facts: some are too granular or transient (e.g., a one-time action item), and evaluating observations in isolation would only measure extraction completeness and atomicity, not whether the system forms and maintains correct knowledge. Other systems use different terms for similar concepts (Hindsight calls atomic extractions “facts” and synthesized summaries “observations”; Mem0 uses “memories” for both layers without distinction), but the underlying two-layer pattern, extraction followed by aggregation, is shared. Intrinsic evaluation targets the fact layer, as this is the knowledge state that determines system behavior.

我们采用双层术语：观测是从单件工件抽出的静态原子事实陈述（分到决策、承诺、里程碑等类别）；事实则是系统随时间维护的、经综合后的演化知识单元。观测一经抽出不可变；事实才是会被创建、强化、更新或取代的持久记忆。并非所有观测都会聚合成事实：有的太细碎或太短暂（例如一次性待办），若只孤立评价观测，量到的只是抽取完整性与原子性，量不到系统是否形成并维持正确知识。其他系统用不同叫法指相近概念（Hindsight 把抽出的原子叫「facts」、把综合摘要叫「observations」；Mem0 两层都叫「memories」、不加区分），但底层的双层模式——先抽取再聚合——是共享的。内在评测对准事实层，因为决定系统行为的正是这一知识状态。

Each fact maintains a numeric confidence score (0–1), strengthened by +0.1 per reinforcing observation and adjustable by the LLM on create, update, or supersede decisions. A temporal guard prevents stale observations from overwriting newer knowledge: if an observation is older than the matched fact’s last update, update and supersede decisions are automatically downgraded to reinforce. Facts track a full chronological history (statement, confidence, change type, and trigger observation at each step), enabling intrinsic evaluation to inspect the complete evolution path. The system additionally tracks the number of distinct contributing authors per fact, counting unique contributors across supporting observations, which enables C6 (multi-author) evaluation. Retrieval uses hybrid search combining BM25 (Robertson and Zaragoza 2009) keyword matching (weight 0.3) with k-NN vector similarity (weight 0.7) over 1024-dimensional embeddings. All LLM classification and synthesis calls use temperature 0 for deterministic outputs; remaining variance across runs comes from the embedding retrieval layer. The system runs on cloud-hosted LLM APIs for both extraction and synthesis (prompts and models in Appendices B, D).

每条事实维护 0–1 的数值置信度：每条强化观测 +0.1，并在创建、更新或取代决策时可由 LLM 调整。时间守卫防止陈旧观测覆盖更新的知识：若观测早于所匹配事实的最近更新，则更新与取代决策会自动降级为强化。事实跟踪完整时间线历史（每一步的陈述、置信度、变更类型与触发观测），使内在评测能检查整条演化路径。系统还按支持观测统计每条事实的不重复贡献作者数，从而支持 C6（多作者）评测。检索采用混合搜索：BM25（Robertson 与 Zaragoza，2009）关键词匹配权重 0.3，k-NN 向量相似度权重 0.7，嵌入维度 1024。所有 LLM 分类与综合调用温度设为 0，以得到确定性输出；跨次运行的剩余方差来自嵌入检索层。系统在云托管 LLM API 上做抽取与综合（提示词与模型见附录 B、D）。

Formally, for each new observation $o$ with embedding $e_{o}$, the system retrieves the most similar existing fact $f=\arg\max_{f^{\prime}\in F}\text{sim}(e_{o},e_{f^{\prime}})$ and classifies the action:

在形式上彬彬有 Lex杏花的每条新观察$o$嵌入XXMATH负责处理,系统检索最相似的现有事实XXMAT要么20,并将动作分类:

$$a=\textsc{Classify}_{\text{LLM}}(o,f)\in\{\text{create},\text{reinforce},\text{update},\ldots\} \tag{7}$$

A temporal guard enforces recency: $a\leftarrow\text{reinforce}$ if $a\in\{\text{update},\text{supersede}\}\wedge t_{o}<t_{f}^{\text{last}}$. Regardless of action, the observation is linked to the fact: $S_{f}\leftarrow S_{f}\cup\{o\}$, where $S_{f}$ is the set of supporting observations for $f$. Confidence evolves as:

”. 时间守备执行 :$a\leftarrow\text{reinforce}$if $a\in\{\text{update},\text{supersede}\}\wedge t_{o}<t_{f}^{\text{last}}$. 无论行动如何,观察都与事实相关: $S_{f}\leftarrow S_{f}\cup\{o\}$, 其中 XXMA及其它是$f$的一组辅助观测. 信心演变为:

$$c_{f}^{(t+1)}=\begin{cases}c_{\text{LLM}}(o)&\text{if }a=\text{create}\
\min(c_{f}^{(t)}+0.1,;1)&\text{if }a=\text{reinforce}\
c_{\text{LLM}}(s_{f}^{(t+1)})&\text{if }a\in{\text{update},\text{supersede}}\
c_{f}^{(t)}&\text{if }a=\text{none}\end{cases} \tag{8}$$

and the fact statement as:

我现在的实际情况是:

$$s_{f}^{(t+1)}=\begin{cases}\textsc{Init}{\text{LLM}}(o)&\text{if }a=\text{create}\
s{f}^{(t)}&\text{if }a\in{\text{reinforce},\text{none}}\
\textsc{Agg}{\text{LLM}}(s{f}^{(t)},o)&\text{if }a\in{\text{update},\text{supersede}}\end{cases} \tag{9}$$

6.2 Results

6.2 结果

Category	Intr. (%)	Extr. (%)	MAD	DivRate (%)
C1: LLM Comp.	82.2$\pm$2.9	54.5$\pm$10.8	.295$\pm$.084	67.3
C2: Retrieval	60.3$\pm$4.7	50.9$\pm$2.9	.135$\pm$.034	45.5
C3: Classif.	70.8$\pm$8.2	48.9$\pm$16.4	.258$\pm$.117	54.5
C4: Synthesis	52.8$\pm$3.1	49.2$\pm$2.4	.104$\pm$.036	21.2
C5: Temporal	59.3$\pm$5.2	47.3$\pm$4.5	.144$\pm$.062	45.5
C6: Multi-Auth.	41.5$\pm$4.8	41.8$\pm$5.7	.106$\pm$.049	34.5
Overall	60.6$\pm$1.6	48.8$\pm$2.9	.168$\pm$.025	43.6

Diff.	Intr. (%)	Extr. (%)	MAD	DivRate (%)
L1	68.0$\pm$3.2	50.9$\pm$4.2	.178$\pm$.051	54.5
L2	67.0$\pm$4.5	47.7$\pm$7.8	.224$\pm$.071	50.0
L3	46.7$\pm$1.6	47.7$\pm$3.3	.103$\pm$.045	26.4
	60.6$\pm$1.6	48.8$\pm$2.9	.168$\pm$.025	43.6

Table 3: Intrinsic (fact accuracy %) and extrinsic (QA accuracy %) scores by category (left) and difficulty (right), averaged across $N{=}11$ runs ($\pm$ std). MAD = mean absolute divergence between intrinsic and extrinsic per trajectory. DivRate = percentage with divergence $>\tau{=}0.15$. C1 shows highest MAD (.295): memory quality is systematically better than what QA evaluation reveals. Per-trajectory scores are in Appendix I.

类别	内在(%)	外在(%)	MAD	分歧率(%)
C1: LLM 代偿	82.2$\pm$2.9	54.5$\pm$10.8	.295$\pm$.084	67.3
C2: 检索	60.3$\pm$4.7	50.9$\pm$2.9	.135$\pm$.034	45.5
C3: 分类	70.8$\pm$8.2	48.9$\pm$16.4	.258$\pm$.117	54.5
C4: 综合	52.8$\pm$3.1	49.2$\pm$2.4	.104$\pm$.036	21.2
C5: 时间	59.3$\pm$5.2	47.3$\pm$4.5	.144$\pm$.062	45.5
C6: 多作者	41.5$\pm$4.8	41.8$\pm$5.7	.106$\pm$.049	34.5
总体	60.6$\pm$1.6	48.8$\pm$2.9	.168$\pm$.025	43.6

难度	内在(%)	外在(%)	MAD	分歧率(%)
L1	68.0$\pm$3.2	50.9$\pm$4.2	.178$\pm$.051	54.5
L2	67.0$\pm$4.5	47.7$\pm$7.8	.224$\pm$.071	50.0
L3	46.7$\pm$1.6	47.7$\pm$3.3	.103$\pm$.045	26.4
	60.6$\pm$1.6	48.8$\pm$2.9	.168$\pm$.025	43.6

表 3：按类别（左）与难度（右）的内在（事实准确率 %）与外在（问答准确率 %）分数，对 $N{=}11$ 次运行取平均（$\pm$ 标准差）。MAD = 每条轨迹上内在与外在的平均绝对分歧。分歧率 = 分歧 $>\tau{=}0.15$ 的轨迹占比。C1 的 MAD 最高（.295）：记忆质量系统性地好于问答评测所揭示的水平。分轨迹分数见附录 I。

Finding 1: Intrinsic-Extrinsic Divergence.

Across all 30 trajectories (Table 3), the mean absolute divergence between intrinsic and extrinsic evaluation is $0.168\pm 0.025$ (over 16 percentage points), with 44% of trajectories exceeding the $\tau=0.15$ threshold ($N{=}11$ runs). Using a stricter pass/fail criterion ($\geq$0.5 vs. $<$0.5), in $21\pm 4$% of trajectories one evaluation method passes while the other fails, meaning a practitioner relying on extrinsic evaluation alone would reach the wrong conclusion about system correctness in roughly 1 out of 5 cases (Cohen’s $\kappa=0.118$, barely above chance), confirming the two evaluation methods carry largely independent signal. A Pearson correlation between intrinsic and extrinsic scores across all 330 paired evaluations is $r=0.423$ ($p<10^{-15}$), so $r^{2}\approx 0.18$: only 18% of the variance in one score is explained by the other, 82% independent. The signed mean divergence is $+0.118\pm 0.029$, indicating intrinsic scores are systematically higher than extrinsic: the system’s memory is on average better than what QA evaluation reveals. A Wilcoxon signed-rank test over all 330 paired trajectory evaluations ($W=4042$, $p<10^{-16}$) confirms this divergence is statistically significant, establishing that the two evaluation methods measure fundamentally different properties of the system. C1 (LLM Compensation) shows the highest MAD ($0.295\pm 0.084$): intrinsic scores 82.2% while extrinsic scores only 54.5%, confirming that memory quality is systematically underestimated by response-based evaluation. C5 (Temporal) and C6 (Multi-Author) show the lowest intrinsic scores (59.3% and 41.5%), identifying temporal evolution and source attribution as the hardest dimensions, yet extrinsic evaluation reports near-uniform scores across all categories, masking these weaknesses.

发现 1：内外分歧。

在全部 30 条轨迹上（表 3），内在与外在评测的平均绝对偏差为 $0.168\pm 0.025$（超过 16 个百分点），44% 的轨迹超过 $\tau=0.15$ 阈值（$N{=}11$ 次运行）。按更严的及格线（$\geq 0.5$ 对 $<0.5$），约 $21\pm 4$% 的轨迹会出现一边过、一边挂，即若只看外在评测，大约每 5 例就会对系统正确性得出错误结论（Cohen’s $\kappa=0.118$，仅略高于随机），说明两种评测携带的信号大体独立。

Finding 2: Difficulty and Divergence.

DivRate decreases with difficulty (L1: 55%, L2: 50%, L3: 26%). At L1, intrinsic substantially exceeds extrinsic ($68.0\pm 3.2$ vs. $50.9\pm 4.2$), meaning the system stores correct knowledge that retrieval fails to surface for QA. At L3, both scores converge near 47%, but diverge on individual trajectories in opposite directions (some false positives, some false negatives). MAD varies across levels (.178, .224, .103), indicating disagreement persists at all difficulty levels. Classification accuracy degrades sharply with difficulty ($33.8\rightarrow 16.5\rightarrow 10.1$%), confirming that the system’s ability to correctly label actions deteriorates as signal becomes more subtle.

发现2:困难和分歧。

分配率随困难而减少(L1:55%,L2:50%,L3:26%)。 在L1时,内在性大大超过外部性($68.0\pm 3.2$ vs. $50.9\pm 4.2$),意味着系统存储正确的知识,即检索不能为QA表面. 在L3,两个分数都接近47%,但在不同的方向(一些假阳性,一些假阴性协调一致地)的例子上却存在差异. 不同级别(178,224,103摇滚)的MAD不同,表明在所有困难级别上仍然存在分歧。 分类精度在困难中急剧下降($33.8\rightarrow 16.5\rightarrow 10.1$ %),这证实了忆念系统正确交互式动作的能力随着信号变得更加微妙而恶化以便为动作贴上正确的标签.

Finding 3: Category-Specific Failure Modes.

Diagnostic decomposition reveals distinct failure patterns. The overall false update rate is $22.2\pm 1.2$%: the system modifies facts when it should only reinforce. Update detection recall is $73.6\pm 2.6$%. Classification accuracy (whether the system correctly labels the action as create/reinforce/update/supersede) is only $20.1\pm 2.7$%, with C2 (4.5%) and C5 (0.4%) near zero. Confidence MAE is $0.106\pm 0.002$ across all trajectories. C6 (Multi-Author) shows the lowest intrinsic accuracy (41.5%), identifying multi-author source attribution as the system’s weakest dimension. These insights are invisible to extrinsic evaluation, which reports a near-uniform 42–55% across most categories (per-category/difficulty breakdowns in Appendix F).

发现 3：类别特异的失败模式。

诊断分解呈现不同失败形态。总体误更新率为 $22.2\pm 1.2$%：系统在本应只强化时却改写了事实。更新检测召回为 $73.6\pm 2.6$%。操作分类准确率（create/reinforce/update/supersede）仅 $20.1\pm 2.7$%，其中 C2（4.5%）与 C5（0.4%）接近零。置信度 MAE 为 $0.106\pm 0.002$。C6（多作者）内在精度最低（41.5%），是最弱维度。这些洞察外在评测看不见。

Finding 4: Actionable Diagnostics.

The intrinsic metrics directly identify which pipeline stage to improve: (1) classification accuracy is 20.1% overall, meaning the system rarely assigns the correct action label even when its fact state ends up partially correct; (2) multi-author source attribution (C6: 41.5%) does not adequately leverage author identity; (3) false updates (22.2%) indicate overly aggressive modification on reinforcement signals. None of these are recoverable from extrinsic scores.

• 发现4:可操作梵天诊断。

内在的衡量标准直接确定要改进的管道阶段:(1)分类精度总体为20.1%,这意味着即使其事实状态最终部分正确,系统也很少指定正确的动作标签;(2)多作者源归属(C的制度6:41.5%)并不能充分发挥作者订有适合于作者的身份;(3)虚假更新(22.2%)相互补充信号显示对加固信号过于减轻贫穷. 所有这些蝇子都无法从外部分数中即便是收回。

Finding 5: Robustness Across Systems.

To test whether the intrinsic–extrinsic divergence is an artifact of a particular system implementation, corpus version, or backbone model, we evaluate four configurations spanning two factors: system configuration (simpler vs. main) and corpus (pre-release vs. the publicly released set), plus the main configuration with an upgraded backbone model. The simpler system on the pre-release corpus gives MAD $=0.164\pm 0.027$, DivRate $=43.3\%$, signed divergence $+0.088\pm 0.034$, and $\kappa=0.104$. Holding the corpus fixed (pre-release) but upgrading to the main system configuration shifts intrinsic accuracy up most sharply among all four configurations ($57.6\%\rightarrow 61.1\%$) and raises DivRate to its highest value ($49.4\%$, MAD $=0.181\pm 0.022$, signed divergence $+0.126\pm 0.025$). Switching to the publicly released corpus on top of the main system configuration (used for all other results in this paper) pulls DivRate back down to $43.6\%$ (MAD $=0.168\pm 0.025$, signed divergence $+0.118\pm 0.029$, $\kappa=0.118$), close to the simpler-system starting point; the system upgrade and the corpus release thus move DivRate in opposite directions and partially cancel. Finally, re-scoring the main configuration with an upgraded backbone model gives MAD $=0.158\pm 0.035$, DivRate $=39.7\%$, signed divergence $+0.121\pm 0.036$, and $\kappa=0.035$. Across all four configurations, intrinsic accuracy remains in a narrow $57.6$–$61.4\%$ band, extrinsic accuracy is essentially flat ($48.5$–$49.2\%$), and signed divergence stays positive throughout; pass/fail agreement stays near chance in every configuration, falling further under the upgraded backbone, consistent with weak-to-absent agreement rather than a single stable value. The configuration used for the main results in this paper adds retrieval relevance filtering, concurrent fact-update handling, and improved large-document extraction; the simpler-system ablation lacks these refinements, which accounts for its modestly lower intrinsic accuracy. All configurations implement the same extract-embed-classify-synthesize pattern (Figure 1). The divergence therefore persists across all four, with full statistics in Appendix H.

还为 发现5:全系统强健。

为了检验内在的外在差异是密切注意某种系统实施、实体版或主干模型的产物,我们评价了四种配置ضير , 包括两种因素:系统配置( 给我的简单化与主指导) 和 实体( 释放前与和各公开单元) , 加上主干模型升级的主要配置。 释放前程序上较简单的系统为MAD $=0.164\pm 0.027$,DivRate $=43.3\%$,签名的差值一声$+0.088\pm 0.034$,和$\kappa=0.104$.适合于MAD. 持有固定的(释放前)但升级到主系统配置中,内在精度在所有四个配置($57.6\%\rightarrow 61.1\%$)中变化最为明显,并将DivR在此领域提升到进行了审查的最高值($49.4\%$,MAD XX 我爱MAD $=0.181\pm 0.022$,签名差异$+0.126\pm 0.025$ )。 切换到主系统配置上方的公开发布符(用于本文中所有其他结果)将DivRate拉回XXMA仍有待8XX制定出(MAD $=0.168\pm 0.025$,签名差异$+0.118\pm 0.029$,$\kappa=0.118$)؟,接近更洙的系统起始点;系统升级和发布符从而将DivRate向相反方向移动并部分取消. 最后,使用升级骨干模型重评统计数据主配置给予MAD 以及其他有关的MATH 80. $=0.158\pm 0.035$,D 我喜欢的XXMATHan $=39.7\%$,签名的差值$+0.121\pm 0.036$,和$\kappa=0.035$. 在所有四种配置中,实质内容的准确性都仍停留在狭义的$57.6$ – XXMATH17接轨中,外部精度基本平坦($48.5$ – $49.2\%$),签定的分歧始终是硕果累累的;在每个配置中,通过/ 106. 协议都几乎是偶然的,更属于升级后的骨干范围内,符合弱效协议而不是单一的稳定值。 本文主要成果采用的配置增加了检索相关性过滤,同时进行事实在适当情况下更新处理,并改进 我们走大文件提取;更简单的系统衰减缺乏这些改进,这说明其内在准确度略低。 所有配置都执行相同的提取蕴藏-分类-合成图案(图1). 因此,这四者之间仍然存在偃力,附录H中有完整的统计数据。

6.3 Diagnostic Decomposition

6.3 诊断分解

Category	Miss (%)	Fact Count	ClassAcc (%)
C1: LLM Comp.	54.0$\pm$4.5	13.6$\pm$1.0	41.7$\pm$2.8
C2: Retrieval	74.5$\pm$4.2	20.8$\pm$1.3	4.5$\pm$5.7
C3: Classif.	60.5$\pm$7.9	14.8$\pm$1.4	30.7$\pm$12.3
C4: Synthesis	72.2$\pm$2.6	14.9$\pm$1.0	17.4$\pm$2.4
C5: Temporal	68.0$\pm$5.3	21.0$\pm$1.0	0.4$\pm$1.1
C6: Multi-Auth.	87.5$\pm$3.5	12.0$\pm$0.8	28.8$\pm$7.3
Overall	71.4$\pm$2.1	16.6$\pm$0.4	20.1$\pm$2.7

Diff.	Miss (%)	Fact Count	ClassAcc (%)
L1	54.4$\pm$5.1	9.6$\pm$0.9	33.8$\pm$6.8
L2	61.1$\pm$3.2	14.7$\pm$0.7	16.5$\pm$2.7
L3	86.3$\pm$1.6	22.4$\pm$0.8	10.1$\pm$4.3
Overall	71.4$\pm$2.1	16.6$\pm$0.4	20.1$\pm$2.7

Table 4: Diagnostic decomposition by category (left) and difficulty (right), averaged across $N{=}11$ runs ($\pm$ std). Miss Rate = percentage of expected claims not found in the fact store. Fact Count = average active facts per trajectory (lower indicates better synthesis). ClassAcc = action classification accuracy. C6 shows highest miss rate (88%); fact count inflates from 9.6 at L1 to 22.4 at L3.

类别	漏检(%)	事实数	分类准确(%)
C1: LLM 代偿	54.0$\pm$4.5	13.6$\pm$1.0	41.7$\pm$2.8
C2: 检索	74.5$\pm$4.2	20.8$\pm$1.3	4.5$\pm$5.7
C3: 分类	60.5$\pm$7.9	14.8$\pm$1.4	30.7$\pm$12.3
C4: 综合	72.2$\pm$2.6	14.9$\pm$1.0	17.4$\pm$2.4
C5: 时间	68.0$\pm$5.3	21.0$\pm$1.0	0.4$\pm$1.1
C6: 多作者	87.5$\pm$3.5	12.0$\pm$0.8	28.8$\pm$7.3
总体	71.4$\pm$2.1	16.6$\pm$0.4	20.1$\pm$2.7

难度	漏检(%)	事实数	分类准确(%)
L1	54.4$\pm$5.1	9.6$\pm$0.9	33.8$\pm$6.8
L2	61.1$\pm$3.2	14.7$\pm$0.7	16.5$\pm$2.7
L3	86.3$\pm$1.6	22.4$\pm$0.8	10.1$\pm$4.3
总体	71.4$\pm$2.1	16.6$\pm$0.4	20.1$\pm$2.7

表 4：按类别（左）与难度（右）的诊断分解，对 $N{=}11$ 次运行取平均（$\pm$ 标准差）。漏检率 = 期望主张未出现在事实库中的比例。事实数 = 每条轨迹平均活跃事实数（越低通常表示综合越好）。分类准确 = 动作分类准确率。C6 漏检最高（88%）；事实数从 L1 的 9.6 胀到 L3 的 22.4。

A key advantage of intrinsic evaluation is diagnostic decomposition: when a system fails, we can identify which pipeline stage is responsible:

内在评价的一个主要优势是诊断分解:当一个系统失败时,我们可以确定哪个管道阶段负责:

• • Extraction failures: Claims present in artifacts but not captured by the system’s observation layer.
• • Synthesis failures: Claims extracted but incorrectly merged, duplicated, or lost during fact aggregation.
• • Update failures: State transitions missed or false updates applied.
• • Conflict failures: Contradictions not detected or resolved incorrectly.

年纪:年纪 以文物为单位但系统声明没有被系统观察好几层记录
• ** 综合失败**: 提取但合并不正确、重复或合并表面上损失的索赔。
• 最新坛次: 国家过渡坛漏或应用错误更新恬.
• ** 冲突失败**: 未发现或解决的矛盾。

Table 4 reveals three co-occurring failure patterns that intensify with difficulty. First, the claim miss rate rises from 58% at L1 to 88% at L3, indicating that subtle or embedded claims are increasingly lost during extraction. Second, fact count inflates from 9.1 to 20.7, showing the system fragments knowledge into many small, partially redundant facts rather than synthesizing them into coherent evolving beliefs. Third, classification accuracy degrades from 37% to 10%, meaning the system rarely identifies the correct action (create vs. update vs. reinforce) at higher difficulty. These three signals are mutually reinforcing: poor classification leads to creating new facts instead of updating existing ones (inflating count), which in turn fragments claims across facts (increasing miss rate on claim-level matching). None of these diagnostics are available from the extrinsic QA score, which remains flat at $\sim$50% regardless of the underlying failure pattern.

表 4 揭示三种随难度加剧、且同时出现的失败模式。第一，主张漏检率从 L1 的 58% 升到 L3 的 88%，说明细微或嵌入式主张在抽取阶段越来越容易丢失。第二，事实数从 9.1 胀到 20.7，表明系统把知识拆成许多细碎、部分冗余的事实，而不是综合成连贯的演化信念。第三，分类准确率从 37% 降到 10%，意味着高难度下系统很少选对动作（创建 vs 更新 vs 强化）。这三者互相加强：分类差会催生「该更新却新建」，抬高事实数，进而把主张碎片化到多条事实上（抬高主张级匹配的漏检）。这些诊断信号没有一项能从外在问答分数里读出来——后者无论底层失败模式如何，都大致钉在约 50%。

7 Discussion and Limitations

7 讨论与局限

Complementary, Not Replacement.

Intrinsic evaluation does not replace extrinsic evaluation, it complements it. Extrinsic captures end-to-end utility as intrinsic enables diagnosis. A system scoring high on both is genuinely effective; high extrinsic but low intrinsic signals fragility (dependent on LLM compensation); high intrinsic but low extrinsic indicates a retrieval or generation bottleneck.

你真该说这些话

内在评价不能取代外在评价,而是补充外在评价。阎锡山 外感捕捉到端到端的有用性,因为内在作用使得诊断成为可能.娇惯的有权享有一种有效的系统;高外观但内在信号脆弱):高内在但低外观表示检索或生成瓶颈。

Synthetic Document Validity.

Generated documents may not capture all properties of real multi-author text. We mitigate this through style grounding, multi-model rotation, and hybrid injection at L3. Future work should validate transfer via rank correlation with human-annotated real document sequences.

凿齿为防止合成文件的有效性。

中确定的文档可能无法捕捉真实多作者文本的所有属性. 我们通过并在其L3进行样式定位、多型旋转和混合注射来缓解这种情况。 今后的工作提示通过与人类附加说明的真实文件序列的等级关系进行转让。

Scope and Limitations.

FEB-30 targets factual evolution across multi-author artifacts. It does not evaluate episodic memory, procedural memory, or single-turn conversational recall, capabilities addressed by existing benchmarks. The benchmark contains 30 trajectories, sufficient for diagnostic evaluation but not for fine-grained statistical comparisons. Scoring relies on embedding similarity thresholds and LLM judges, introducing potential measurement noise.

• 范围和限制。

F所提及的FEB-30针对的是跨多作者文物的事实演变. 它不评估一次性记忆、程序记忆或单 51.其目的在于回顾现有基准所相互涉及如何能够实现。 基准包含30个轨迹相互合作,足以进行诊断性评价,但不足以进行精细的统计比较。 请将本函及其附件作为安全理事会的衡量标准。

8 Conclusion

8 结论

We introduced FEB-30, the first benchmark designed for intrinsic evaluation of agent memory systems. By constructing trajectories where the expected memory state is known at every step, we demonstrated that intrinsic and extrinsic evaluation diverge by an average of 16 percentage points, with 44% of trajectories showing meaningful disagreement, revealing false positives and false negatives invisible to response-based benchmarks. Diagnostic decomposition showed the system correctly classifies memory operations only 20% of the time (degrading from 34% at L1 to 10% at L3), a dimension extrinsic evaluation cannot measure.

我们提出 FEB-30，这是面向智能体记忆系统内在评测的首个基准。通过构造每一步期望记忆状态已知的轨迹，我们表明内在与外在评测平均相差约 16 个百分点，44% 的轨迹出现有意义分歧，从而暴露回答型基准看不见的假阳性与假阴性。诊断分解显示，系统正确标注记忆操作的比例仅约 20%（L1 约 34% 降至 L3 约 10%），这一维度外在评测无法衡量。

We argue that intrinsic evaluation should become standard practice alongside extrinsic benchmarks, analogous to how intrinsic evaluation of embeddings complemented downstream task evaluation in NLP. We release FEB-30, the generation pipeline, and the scoring framework for reproducible evaluation of agent memory.

我们认为内在评测应成为与外在基准并列的标准做法，正如 NLP 中嵌入的内在评测曾补充下游任务评测。我们开源 FEB-30、生成管线与打分框架，供可复现的智能体记忆评测使用。

References

参考文献

Appendix

** 附录**才能够收到。

Amazon Artificial General Intelligence (2025) Amazon Artificial General Intelligence. 2025. The Amazon Nova Family of Models: Technical Report and Model Card. arXiv:2506.12103.

Amazon Web Services (2024) Amazon Web Services. 2024. Amazon Titan Text Embeddings. AWS AI Service Card.

Anthropic (2025) Anthropic. 2025. Claude Haiku 4.5 System Card. Anthropic model system card.

Baroni, Dinu, and Kruszewski (2014) Baroni, M.; Dinu, G.; and Kruszewski, G. 2014. Don’t Count, Predict! A Systematic Comparison of Context-Counting vs. Context-Predicting Semantic Vectors. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 238–247. Association for Computational Linguistics.

Carletta et al. (2005) Carletta, J.; Ashby, S.; Bourban, S.; Flynn, M.; Guillemot, M.; Hain, T.; Kadlec, J.; Karaiskos, V.; Kraaij, W.; Kronenthal, M.; et al. 2005. The AMI Meeting Corpus: A Pre-Announcement. In Proc. MLMI Workshop .

Chhikara et al. (2025) Chhikara, P.; Khant, D.; Aryan, S.; Singh, T.; and Yadav, D. 2025. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv preprint .

Du (2026) Du, P. 2026. Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers. arXiv preprint .

Du et al. (2025) Du, Y.; Huang, W.; Zheng, D.; Wang, Z.; Montella, S.; Lapata, M.; et al. 2025. Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics. arXiv preprint .

Future AGI (2026) Future AGI. 2026. Evaluating Agent Memory Systems: A Practitioner’s 4-Dimension Framework. Industry blog post. Practitioner source, not peer-reviewed.

Hu et al. (2025) Hu, Y.; Liu, S.; Yue, Y.; Zhang, G.; Liu, B.; and Zhu, F. 2025. Memory in the Age of AI Agents. arXiv preprint .

Hu, Wang, and McAuley (2026) Hu, Y.; Wang, Y.; and McAuley, J. 2026. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions. In ICLR .

Huang et al. (2026) Huang, W.-C.; Zhang, W.; Liang, Y.; Bei, Y.; Chen, Y.; Feng, T.; et al. 2026. A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents. Transactions on Machine Learning Research . Survey Certification.

Jiang et al. (2026) Jiang, D.; Li, Y.; Wei, S.; Yang, J.; Kishore, A.; and Zhao, A. 2026. Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations. arXiv preprint .

Jung et al. (2026) Jung, S.; Rubinstein, A.; Uselis, A.; Yun, S.; and Oh, S. J. 2026. MEME: Multi-Entity and Evolving Memory Evaluation. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track .

Klimt and Yang (2004) Klimt, B.; and Yang, Y. 2004. The Enron Corpus: A New Dataset for Email Classification Research. In Proc. European Conference on Machine Learning (ECML) .

Latimer et al. (2026) Latimer, C.; Boschi, N.; Neeser, A.; Bartholomew, C.; Srivastava, G.; Wang, X.; et al. 2026. Hindsight: Structured Agent Memory that Retains, Recalls, and Reflects. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) , 275–285. Association for Computational Linguistics.

Lee, Lee, and Kim (2025) Lee, J.; Lee, K.; and Kim, T. 2025. MAGIC: A Multi-Hop and Graph-Based Benchmark for Inter-Context Conflicts in Retrieval-Augmented Generation. arXiv preprint .

Liu and Yadav (2026) Liu, L.; and Yadav, N. 2026. Introducing STATE-Bench: A Benchmark for AI Agent Memory. Microsoft Open Source Blog. Industry source.

Luo et al. (2026) Luo, J.; Tian, Y.; Cao, C.; Luo, Z.; Lin, H.; and Li, K. 2026. From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms. arXiv preprint .

Maharana et al. (2024) Maharana, A.; Lee, D.-H.; Tulyakov, S.; Bansal, M.; Barbieri, F.; and Fang, Y. 2024. Evaluating Very Long-Term Conversational Memory of LLM Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 13851–13870. Association for Computational Linguistics.

Packer et al. (2024) Packer, C.; Fang, V.; Patil, S. G.; Lin, K.; Wooders, S.; and Gonzalez, J. E. 2024. MemGPT: Towards LLMs as Operating Systems. In ICLR .

Rasmussen et al. (2025) Rasmussen, P.; Paliychuk, P.; Beauvais, T.; Ryan, J.; and Chalef, D. 2025. Zep: A Temporal Knowledge Graph Architecture for Agent Memory. arXiv preprint .

Robertson and Zaragoza (2009) Robertson, S.; and Zaragoza, H. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval , 3(4): 333–389.

Roig (2025) Roig, J. 2025. Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe. arXiv preprint .

Schaffelder and Gatt (2026) Schaffelder, M.; and Gatt, A. 2026. Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning. In Findings of the Association for Computational Linguistics: ACL 2026 .

Schnabel et al. (2015) Schnabel, T.; Labutov, I.; Mimno, D.; and Joachims, T. 2015. Evaluation Methods for Unsupervised Word Embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , 298–307. Association for Computational Linguistics.

Su et al. (2024) Su, Z.; Zhang, J.; Qu, X.; Zhu, T.; Li, Y.; Sun, J.; Li, J.; Zhang, M.; and Cheng, Y. 2024. ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMs. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track .

Sumers et al. (2024) Sumers, T.; Yao, S.; Narasimhan, K. R.; and Griffiths, T. L. 2024. Cognitive Architectures for Language Agents. Transactions on Machine Learning Research .

Toma and Matei (2026) Toma, S.-A.; and Matei, O. 2026. What Agent Memory Benchmarks Miss: The Gap Between Response Quality and Knowledge State Correctness. In Proceedings of InnoComp 2026 , Communications in Computer and Information Science. Springer. To appear.

Uddin et al. (2026) Uddin, M. N.; Shubham, K.; Blanco, E.; Baral, C.; and Wang, G. 2026. From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents. In Findings of the Association for Computational Linguistics: ACL 2026 .

van Elburg, van der Putten, and Marx (2025) van Elburg, J.; van der Putten, P.; and Marx, M. 2025. Can We Evaluate RAGs with Synthetic Data? arXiv preprint .

Wu et al. (2026) Wu, D.; Ji, Z.; Kawatkar, A.; Kwan, B.; Gu, J.-C.; and Peng, N. 2026. LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues. arXiv preprint .

Zhao et al. (2026) Zhao, Y.; Yuan, B.; Huang, J.; Yuan, H.; Yu, Z.; Xu, H.; Hu, L.; Shankarampeta, A.; Huang, Z.; Ni, W.; Tian, Y.; and Zhao, J. 2026. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications. In Proceedings of the 43rd International Conference on Machine Learning , volume 306, 162781–162809. PMLR.

Zhu et al. (2026) Zhu, W.; Chen, X.; Wang, Z.; Wang, J.; Dong, X.; and Huang, M. 2026. AriadneMem: Threading the Maze of Lifelong Memory for LLM Agents. arXiv preprint .

Appendix A Trajectory Examples

附录 A 轨迹示例

A.1 c5-linear: Linear Temporal Evolution

A.1 c5-linear：线性时间演化

$s_{i}$	Action	Expected claims $C_{i}$ (cumulative)
0	create	“PostgreSQL 14 on-premise in Frankfurt”; “3 TB database”
1	update	+“Evaluating Aurora and AlloyDB”
2	update	+“Chose Aurora”; “Cutover Q3 2025”
3	update	+“DMS replication running”; “95% stored procedures validated”

Table 5: State evolution for c5-linear.

$s_{i}$	动作	期望主张 $C_{i}$（累积）
0	create	“PostgreSQL 14 on-premise in Frankfurt”；“3 TB database”
1	update	+“Evaluating Aurora and AlloyDB”
2	update	+“Chose Aurora”；“Cutover Q3 2025”
3	update	+“DMS replication running”；“95% stored procedures validated”

表 5：c5-linear 的状态演化。

Each state’s document contains only docClaims (new information): the $s_{2}$ document mentions the Aurora decision and Q3 timeline but does not repeat that the database is 3 TB. The system must retain $s_{0}$ claims from memory.

, 该系统必须保留来自内存的$s_{0}$索赔 快去处理.

Intrinsic scoring inspects the fact store after each artifact is ingested. At each state $s_{i}$, the system should have a fact containing all claims from $C_{i}$, with history showing the correct transitions. If the system creates separate facts for each state (fragmentation) or drops $s_{0}$ claims (retention failure), intrinsic scoring detects it at the state where it occurs.

当我吃下每件文物嗓子后, 在$s_{i}$的每一个州,该系统应有一个包含$C_{i}$的所有索赔的事实,历史表明正确的过渡。 如果系统制定关于每个状态的单独事实(分解 我跟你),或放弃$s_{0}$ 索赔(保留失败),则内在评分会在的条款发生地的状态中检测到.

Extrinsic scoring poses the question to an agent that retrieves from the memory system’s fact store and generates an answer: “Describe the database migration: source, target, timeline, and status.” A strong LLM can answer this correctly even if the fact store is fragmented across 4 separate facts, as long as retrieval surfaces all fragments.

** 外部评分** 向一个从内存系统的事实存储中检索并生成一个悼词驴子的代理商提出这个问题 : “ 描述数据库迁移: 秘书处的说明:来源、目标、时间线和状态。 ” 强大的LLM可以正确回答这一点,即使事实存储在4个不同的事实中是零散的,只要检索会浮出所有的碎片.

Actual result (N=11 runs): Intrinsic = 40–67%, Extrinsic = 50%. The system partially fragments the fact, which intrinsic detects but extrinsic masks.

** 不可能 实际结果(N=11衲跑)**: Intrinsic 这就是我的40-67%,extrinsic=50%。仍处于系统部分分裂状态,它内在地探测到外蒙。

A.2 c4-out-of-order: Out-of-Order Ingestion

A.2 c4-out-of-order：乱序摄入

This trajectory tests temporal robustness: a newer document (the decision) arrives first, then an older document (the evaluation) arrives second. The system must enrich the existing fact with context from the earlier document without overwriting the confirmed decision.

这一轨迹考验了时间的稳健性:较新的文件(决定)首先到达,然后是较旧的文件(评估)第二. 该系统必须以先前文件的背景丰富现有 西风,而不覆写已确认的决定。

$s_{i}$	Timestamp	Action	Expected claims $C_{i}$ (cumulative)
0	2025-03-10	create	“Chose ECS Fargate”; “Sub-100ms response”
1	2025-01-20	update	+“Lambda rejected (cold starts)”; “EKS rejected (complexity)”
2	2025-05-15	update	+“12 Fargate services at 50K rps”

Table 6: State evolution for c4-out-of-order. Note $s_{1}$ timestamp is older than $s_{0}$.

$s_{i}$	时间戳	动作	期望主张 $C_{i}$（累积）
0	2025-03-10	create	“Chose ECS Fargate”；“Sub-100ms response”
1	2025-01-20	update	+“Lambda rejected (cold starts)”；“EKS rejected (complexity)”
2	2025-05-15	update	+“12 Fargate services at 50K rps”

表 6：c4-out-of-order 的状态演化。注意 $s_{1}$ 的时间戳早于 $s_{0}$。

Key challenge: State 1 has timestamp 2025-01-20, which is older than state 0’s 2025-03-10. The system ingests the decision document first, then receives an earlier evaluation document. The temporal guard should prevent the older document from overwriting the decision; instead, the system should enrich the fact with the evaluation context (why Lambda and EKS were rejected).

关键挑战:国1有时间戳2025-0在实施2025-01-20,比国0的2025-03-10. 该系统首先吸收决定文件,然后收到较早的评价文件。 时间警备应防止旧文件推翻决定;相反,该系统应丰富评价是可以理解的事实(为什么Lambda和EKS被拒绝)。

What intrinsic reveals: If the system overwrites “chose ECS Fargate” with “evaluating Lambda, ECS Fargate, and EKS” (a regression), intrinsic scoring detects the loss of the decision claim. If it correctly enriches, all 4 claims from $C_{1}$ are present. Extrinsic QA may still pass either way if the LLM reasons over both fragments.

** 内在内容显示**: 如果该系统以“促成了Lambda、ECS Fargate和EKS的评价”(回归)为“Chese ECS Fargate”,弄清了决定索赔的损失。将性别观点纳入索赔表。 原产于UTC的各家企业, 如果两个碎片的LLM理由都存在,则extrinsic QA仍有可能通过。

Appendix B System Prompts

附录 B 系统提示词

B.1 Observation Extraction

B.1 观察抽取

The prompts below are generalized for reproducibility. Unlike the other prompts in this section, which are domain-agnostic, this prompt is grounded in the benchmark’s business/customer-relationship domain, since the extraction categories define what the system treats as a fact-worthy observation. The {categories} placeholder below stands for the domain-specific observation categories actually used (e.g. a decision, a commitment, an action item). Each extracted observation is further structured into who/what/when/why/where fields; the statement referred to below is the what field.

下列提示词已做泛化以便复现。与本节其他与领域无关的提示不同，这条提示锚定在基准的业务/客户关系域上，因为抽取类别决定了系统把什么当成「值得记成事实的观测」。下面的 {categories} 占位符代表实际使用的领域观测类别（例如决策、承诺、待办）。每条抽出的观测还会进一步拆成 who/what/when/why/where 字段；下文所说的 statement 即 what 字段。

B.2 Fact Synthesis

B.2 事实合成

Appendix C Evaluation Prompts

附录 C 评测提示词

C.1 Claim Entailment Judge

C.1 主张蕴含裁判

Given the system’s active facts and an expected claim, determines whether the claim is entailed.

鉴于该系统的有效事实和预期的索赔,确定索赔是否涉及。

Scoring: YES = 1.0, PARTIAL = 0.5, NO = 0.0

推选:是褒=1.0, Partialح她在0.5, NO=0.0

C.2 Answer Generation (Extrinsic)

C.2 回答生成（外在）

Asks an agent using the memory system, given retrieved facts and a diagnostic question, to generate an answer.

向使用内存系统的代理人询问,给定在因特网上检索旷事和诊断问题,以生成答案.

C.3 Answer Judge (Extrinsic)

C.3 回答裁判（外在）

Compares the generated answer against the expected answer.

比较生成的答案和预期的答案。

Scoring: YES = 1.0, PARTIAL = 0.5, NO = 0.0

计算年限:是=1.0,部分=0.5,无=0.0

Appendix D Models Used

附录 D 所用模型

Table 7 lists all models used in the system under test and the evaluation scorer. The system uses a single frontier LLM for both observation extraction (converting artifact text into atomic observations) and fact synthesis (classifying the relationship between new observations and existing facts, then rewriting the fact statement). Titan Embeddings V2 (Amazon Web Services 2024) generates 1024-dimensional vectors for semantic search. Retrieval combines BM25 (Robertson and Zaragoza 2009) keyword matching (weight 0.3) with k-NN vector similarity (weight 0.7). The evaluation scorer uses Claude Haiku 4.5 (Anthropic 2025) for judging and Amazon Nova Pro (Amazon Artificial General Intelligence 2025) for answer generation. All LLM calls use temperature 0 for deterministic outputs. All models are accessed via cloud-hosted APIs.

表 7 列出被测系统与评测计分器所用的全部模型。系统用同一套前沿 LLM 做观测抽取（把工件文本变成原子观测）与事实综合（判定新观测与既有事实的关系，并改写事实陈述）。Titan Embeddings V2（Amazon Web Services，2024）生成 1024 维向量供语义检索。检索组合 BM25（Robertson 与 Zaragoza，2009）关键词匹配（权重 0.3）与 k-NN 向量相似度（权重 0.7）。评测计分器用 Claude Haiku 4.5（Anthropic，2025）做评判，用 Amazon Nova Pro（Amazon Artificial General Intelligence，2025）生成回答。所有 LLM 调用温度均为 0，以得到确定性输出。全部模型均经云托管 API 访问。

Component	Model	Temp.
System under test
Observation extraction	Frontier LLM	0
Fact synthesis	Frontier LLM	0
Embedding	Titan Embeddings V2 (1024-dim)	–
Retrieval	BM25 (0.3) + k-NN (0.7)	–
Evaluation scorer
Claim entailment judge	Claude Haiku 4.5	0
Answer generation	Amazon Nova Pro	0
Answer judge	Claude Haiku 4.5	0
Embedding (claim matching)	Titan Embeddings V2 (1024-dim)	–

Table 7: All models used in the system and evaluation pipeline.

组件	模型	温度
被测系统
观测抽取	Frontier LLM	0
事实综合	Frontier LLM	0
嵌入	Titan Embeddings V2 (1024-dim)	–
检索	BM25 (0.3) + k-NN (0.7)	–
评测计分器
主张蕴含评判	Claude Haiku 4.5	0
回答生成	Amazon Nova Pro	0
回答评判	Claude Haiku 4.5	0
嵌入（主张匹配）	Titan Embeddings V2 (1024-dim)	–

表 7：系统与评测管线所用的全部模型。

Appendix E Per-State Coverage Degradation

附录 E 分状态覆盖退化

Table 8 shows how claim coverage degrades as more documents are ingested within a trajectory. Later states require the system to retain earlier claims while incorporating new information, testing memory persistence.

表8 有人说ذ全,因为更多的文件以及其他有关文件在轨迹中被摄有关这一问题的内容。 后来的状态要求系统在纳入新信息的同时保留早期的主张,测试内存的持久性.

State	All (%)	L1 (%)	L2 (%)	L3 (%)
$s_{0}$ (n=330)	64.7$\pm$28.8	67.5$\pm$24.4	76.4$\pm$22.1	50.2$\pm$32.4
$s_{1}$ (n=330)	65.3$\pm$22.1	72.7$\pm$23.6	70.3$\pm$20.3	52.9$\pm$16.0
$s_{2}$ (n=319)	58.2$\pm$20.8	68.2$\pm$23.0	60.6$\pm$17.9	46.1$\pm$13.4
$s_{3}$ (n=242)	57.8$\pm$18.8	62.3$\pm$27.1	68.4$\pm$14.8	48.2$\pm$8.6
$s_{4}$ (n=110)	49.7$\pm$11.4	–	57.3$\pm$10.9	47.8$\pm$10.7
$s_{5}$ (n=44)	50.8$\pm$10.8	–	–	50.8$\pm$10.8
$s_{6}$ (n=22)	52.3$\pm$10.4	–	–	52.3$\pm$10.4

Table 8: Claim coverage (%) by state index, averaged across all trajectories and $N{=}11$ runs. $n$ = number of trajectory$\times$run pairs that reach this state (not all trajectories have the same number of states). Coverage decreases at later states as the system must retain earlier claims while integrating new information. L3 trajectories start lower and degrade further.

状态	全部 (%)	L1 (%)	L2 (%)	L3 (%)
$s_{0}$ (n=330)	64.7$\pm$28.8	67.5$\pm$24.4	76.4$\pm$22.1	50.2$\pm$32.4
$s_{1}$ (n=330)	65.3$\pm$22.1	72.7$\pm$23.6	70.3$\pm$20.3	52.9$\pm$16.0
$s_{2}$ (n=319)	58.2$\pm$20.8	68.2$\pm$23.0	60.6$\pm$17.9	46.1$\pm$13.4
$s_{3}$ (n=242)	57.8$\pm$18.8	62.3$\pm$27.1	68.4$\pm$14.8	48.2$\pm$8.6
$s_{4}$ (n=110)	49.7$\pm$11.4	–	57.3$\pm$10.9	47.8$\pm$10.7
$s_{5}$ (n=44)	50.8$\pm$10.8	–	–	50.8$\pm$10.8
$s_{6}$ (n=22)	52.3$\pm$10.4	–	–	52.3$\pm$10.4

表 8：按状态索引的主张覆盖率（%），对全部轨迹与 $N{=}11$ 次运行取平均。$n$ = 到达该状态的轨迹$\times$运行对数（并非所有轨迹状态数相同）。到更晚状态时覆盖下降，因为系统要在整合新信息的同时保留早先主张。L3 轨迹起点更低，且继续下滑。

Coverage drops from 65% at $s_{0}$ to 50% at $s_{4}$, confirming that memory retention degrades as trajectories progress. At L2, the drop is steepest between $s_{1}$ (70%) and $s_{2}$ (61%), suggesting the system struggles to retain claims once 3+ documents have been processed. L3 starts lowest (50% at $s_{0}$) due to the extraction challenge of hybrid-injection documents.

覆盖范围从$s_{0}$时的65%下降到$s_{4}$时的50%,证实内存保留随着轨迹进展而退化. 在L2中,在$s_{1}$(70%)和$s_{2}$(61%)之间下降幅度最大,这表明系统在3+文件处理完毕后难以保留索赔。 由于混合注射文件的提取挑战,L3开始将其视为最低(在$s_{0}$为50%).

Note that fact coverage (whether the system creates any fact related to the trajectory topic) is 98.8% overall , the system almost always creates a fact. The low claim coverage reported here reflects that the created facts are incomplete or fragmented, not that they are entirely absent.

注意 事实覆盖（系统是否为该轨迹主题创建了任何相关事实）总体达 98.8%——系统几乎总会创建事实。此处报告的低 主张覆盖 反映的是所建事实不完整或碎片化，而不是完全没有事实。

Trajectory length (number of states) is moderately correlated with lower intrinsic accuracy (Spearman $\rho=-0.45$, $p<0.02$): trajectories with 2–3 states average $\sim$71% accuracy while those with 5–7 states average $\sim$49%. This confirms that longer fact evolution sequences are harder to track, independently of the difficulty level assignment.

39 这证实了较长的事实演化序列较难追踪,独立于难度级任务.

Appendix F Update Detection, False Updates, and Confidence Calibration

附录 F 更新检测、误更新与置信度校准

Table 9 shows update detection recall (UDR), false update rate (FUR), and confidence MAE per category and difficulty, all averaged across $N{=}11$ runs.

表9显示更新检测召回(UDR),假更新率(F循序渐进),以及每个类别和每个难度的信心MAE,均在$N{=}11$运行.

Category	UDR (%)	FUR (%)	Conf. MAE
C1: LLM Comp.	51.4$\pm$6.0	39.4$\pm$1.9	0.105$\pm$0.005
C2: Retrieval	94.8$\pm$4.3	0.0$\pm$0.0	0.092$\pm$0.009
C3: Classif.	70.5$\pm$9.6	45.5$\pm$6.0	0.117$\pm$0.007
C4: Synthesis	81.4$\pm$3.3	15.7$\pm$2.1	0.097$\pm$0.006
C5: Temporal	90.6$\pm$2.2	0.0$\pm$0.0	0.108$\pm$0.007
C6: Multi-Auth.	50.8$\pm$7.4	38.6$\pm$3.1	0.124$\pm$0.005
Difficulty
L1	55.2$\pm$3.9	37.4$\pm$2.4	0.089$\pm$0.003
L2	81.6$\pm$5.3	19.4$\pm$1.3	0.112$\pm$0.003
L3	84.1$\pm$2.1	9.8$\pm$0.7	0.118$\pm$0.004
Overall	73.6$\pm$2.6	22.2$\pm$1.2	0.106$\pm$0.002

Table 9: Update detection recall (UDR), false update rate (FUR), and confidence MAE per category and difficulty, averaged across $N{=}11$ runs ($\pm$ std across runs). C2 and C5 never produce false updates but C1 and C6 frequently misclassify reinforcements as updates. Confidence calibration is worst for C6 (multi-author) and best for C4 (synthesis). FUR decreases with difficulty because at L3 the system rarely modifies facts.

类别	UDR (%)	FUR (%)	置信 MAE
C1: LLM 代偿	51.4$\pm$6.0	39.4$\pm$1.9	0.105$\pm$0.005
C2: 检索	94.8$\pm$4.3	0.0$\pm$0.0	0.092$\pm$0.009
C3: 分类	70.5$\pm$9.6	45.5$\pm$6.0	0.117$\pm$0.007
C4: 综合	81.4$\pm$3.3	15.7$\pm$2.1	0.097$\pm$0.006
C5: 时间	90.6$\pm$2.2	0.0$\pm$0.0	0.108$\pm$0.007
C6: 多作者	50.8$\pm$7.4	38.6$\pm$3.1	0.124$\pm$0.005
难度
L1	55.2$\pm$3.9	37.4$\pm$2.4	0.089$\pm$0.003
L2	81.6$\pm$5.3	19.4$\pm$1.3	0.112$\pm$0.003
L3	84.1$\pm$2.1	9.8$\pm$0.7	0.118$\pm$0.004
总体	73.6$\pm$2.6	22.2$\pm$1.2	0.106$\pm$0.002

表 9：按类别与难度的更新检测召回（UDR）、假更新率（FUR）与置信 MAE，对 $N{=}11$ 次运行取平均（$\pm$ 跨次标准差）。C2、C5 从不产生假更新，而 C1、C6 常把强化误判为更新。置信校准最差是 C6（多作者），最好是 C4（综合）。FUR 随难度下降，因为 L3 上系统很少改动事实。

C1 and C6 show both low UDR ($\sim$51%) and high FUR ($\sim$39%): the system misses true updates while simultaneously making false updates, a double failure that intrinsic evaluation exposes but extrinsic cannot distinguish from simple low accuracy. C2 and C5 achieve near-perfect UDR with zero false updates, indicating the classification challenge is category-specific rather than a general system weakness.

C1和C杂种显示低UDR($\sim$ 51%)和高FUR($\sim$ 39%):系统错过了真实的更新,同时进行虚假的更新,这是内在的评价暴露但外在的双重失败,无法区分简单的低精度.铲除是否会发生齐心协力 什么意思? C2减缓贫穷和非正规经济部门 实现征求意见的 你真该死

Confidence calibration is relatively uniform across categories (range 0.092–0.124), with an overall MAE of 0.106 indicating systematic miscalibration of approximately 10 points. C6 (Multi-Author) shows the worst calibration (0.124), likely because the system does not adequately adjust confidence based on author diversity. Calibration degrades slightly with difficulty (0.089 at L1 to 0.118 at L3), consistent with the system being less certain about harder trajectories.

置信校准在各类别间相对均匀（范围 0.092–0.124），总体 MAE 为 0.106，表明大约有 10 个点的系统性失准。C6（多作者）校准最差（0.124），很可能因为系统未按作者多样性充分调整置信度。校准随难度略有恶化（L1 的 0.089 到 L3 的 0.118），与系统对更难轨迹更没把握相一致。

Appendix G Metric Correlations

附录 G 指标相关性

To validate that our intrinsic metrics capture meaningful signal, we compute correlations between claim miss rate, intrinsic accuracy, and extrinsic accuracy across all 30 trajectories (using per-trajectory means across $N{=}11$ runs).

为了证实我们的内在指标能捕捉到有意义的信号,我们计算所有30个轨迹(使用每轨电路手段横跨$N{=}11$运行)的声称误差、内在准确度和外在精确度 Madison的相互关系。

Comparison	Pearson $r$	$p$-value
Miss rate vs. Intrinsic accuracy	$-0.841$	$5.9\times 10^{-9}$
Miss rate vs. Extrinsic accuracy	$-0.391$	$3.2\times 10^{-2}$
Intrinsic vs. Extrinsic accuracy	$+0.552$	$1.6\times 10^{-3}$

Table 10: Pearson correlations across 30 trajectories. Miss rate strongly predicts intrinsic accuracy ($r=-0.84$) but weakly predicts extrinsic accuracy ($r=-0.39$), confirming that intrinsic metrics capture signal that extrinsic evaluation misses. The moderate intrinsic-extrinsic correlation ($r=0.55$) indicates the two methods share some signal but measure substantially different properties.

比较	Pearson $r$	$p$-值
漏检率 vs. 内在准确率	$-0.841$	$5.9\times 10^{-9}$
漏检率 vs. 外在准确率	$-0.391$	$3.2\times 10^{-2}$
内在 vs. 外在准确率	$+0.552$	$1.6\times 10^{-3}$

表 10：跨 30 条轨迹的 Pearson 相关。漏检率强预测内在准确率（$r=-0.84$），却弱预测外在准确率（$r=-0.39$），说明内在指标抓住了外在评测漏掉的信号。内在–外在中等相关（$r=0.55$）表明两种方法共享部分信号，但度量的实质属性不同。

The strong negative correlation between miss rate and intrinsic accuracy ($r=-0.84$, $p<10^{-8}$) validates that claim coverage is the primary driver of intrinsic scores. The weaker correlation between miss rate and extrinsic accuracy ($r=-0.39$) confirms that extrinsic evaluation is less sensitive to extraction failures , consistent with LLM compensation masking missing claims at query time. The moderate positive correlation between intrinsic and extrinsic ($r=0.55$) shows they are related but far from equivalent, reinforcing the paper’s argument that both are needed.

误差率和内在准确度之间的强烈负相关关系($r=-0.84$,$p<10^{-8}$)验证了索赔覆盖范围是内آ分数的主要驱动力.等因素. 误差率和外部精确度目前正在降低($r=-0.39$),这札证 坐稳率和外部精确度之间的关联性已证实,外部评估对提取失败在编写过程中不太敏感,这与LL 你已经在查询时掩盖了缺失的索赔要求我没有反应。 内在和外在关系($r=0.55$)之间的有助于实现适度的正相关,这表明它们相互关联,但远非等同性,这强化了文件关于两者 Jiangses舜 你要我的论点。

Table 11 shows the signed divergence (intrinsic $-$ extrinsic) per category. Positive values indicate intrinsic scores higher than extrinsic (memory underestimated by QA).

表11显示各类别经签名的差异(显式$-$钝化)。 正值表示内在分数高于外在分数(质量评估低估了记忆).

Category	Signed Div.	Interpretation
C1: LLM Comp.	$+0.276\pm 0.260$	Memory much better than QA reveals
C2: Retrieval	$+0.094\pm 0.137$	Memory slightly better
C3: Classif.	$+0.220\pm 0.276$	Memory much better than QA reveals
C4: Synthesis	$+0.036\pm 0.173$	Nearly neutral
C5: Temporal	$+0.120\pm 0.160$	Memory moderately better
C6: Multi-Auth.	$-0.003\pm 0.158$	No directional bias

Table 11: Signed divergence (intrinsic $-$ extrinsic) per category. C1 shows strong positive bias by design: memory is correct but QA underestimates. C6 shows no directional bias: both metrics are equally poor at capturing multi-author attribution.

类别	有符号分歧	解读
C1: LLM 代偿	$+0.276\pm 0.260$	记忆远好于问答所揭示
C2: 检索	$+0.094\pm 0.137$	记忆略好
C3: 分类	$+0.220\pm 0.276$	记忆远好于问答所揭示
C4: 综合	$+0.036\pm 0.173$	接近中性
C5: 时间	$+0.120\pm 0.160$	记忆中等程度更好
C6: 多作者	$-0.003\pm 0.158$	无方向性偏差

表 11：按类别的有符号分歧（内在 $-$ 外在）。C1 按设计呈强正偏：记忆正确，但问答低估。C6 无方向性偏差：两种指标在捕捉多作者归属上同样差劲。

Appendix H Robustness Across Systems: Configuration Comparison

附录 H 跨系统稳健性：配置比较

Table 12 reports headline statistics for the four configurations discussed in Finding 5 of the main paper. All values are averaged across $N{=}11$ runs. The four configurations are:

表12报告主要文件 " 调查结果5 " 我还以为的四个组合的标题统计。 这是你的最大值。 这是你的四个配置:

• • Simpler sys.: a simpler configuration of the memory system, on the pre-release corpus.
• • Main sys.: the main system configuration, on the same pre-release corpus (differs from Simpler sys. only in the system).
• • Main (released): the main system configuration, on the publicly released corpus used for all other results in this paper (differs from Main sys. only in the corpus).
• • Upgraded backbone: the main system configuration with a newer-generation frontier model as the backbone, replacing the one used in all other configurations, on the publicly released corpus.

• • 更简系统：记忆系统的更简配置，跑在预发布语料上。
• • 主系统：主系统配置，同一预发布语料（与「更简系统」仅系统不同）。
• • 主系统（已发布）：主系统配置，跑在本文其他结果所用的公开发布语料上（与「主系统」仅语料不同）。
• • 升级骨干：主系统配置，但把骨干换成更新一代的前沿模型，替换其余配置所用模型，语料为公开发布版。

The intrinsic–extrinsic divergence holds across all four: positive signed divergence, near-chance pass/fail agreement, and MAD and DivRate in a narrow range, indicating the finding is not an artifact of a single system implementation, corpus version, or backbone model.

内在的-极端的分歧贯穿于所有四个方面:正的签名分歧、近距离断言/失败协议、以及MAD和DivRate的狭义范围,表明发现不是单一系统执行、实体版本或骨干模型的产物。

Metric	Simpler sys.	Main sys.	Main (released)	Upgraded backbone
Intrinsic (%)	$57.6\pm 1.8$	$61.1\pm 1.6$	$60.6\pm 1.6$	$61.4\pm 1.3$
Extrinsic (%)	$48.8\pm 2.7$	$48.5\pm 2.1$	$48.8\pm 2.9$	$49.2\pm 3.4$
MAD	$0.164\pm 0.027$	$0.181\pm 0.022$	$0.168\pm 0.025$	$0.158\pm 0.035$
DivRate (%)	$43.3\pm 6.0$	$49.4\pm 7.2$	$43.6\pm 6.1$	$39.7\pm 6.1$
Signed divergence	$+0.088\pm 0.034$	$+0.126\pm 0.025$	$+0.118\pm 0.029$	$+0.121\pm 0.036$
Cohen’s $\kappa$	$0.104$	$0.082$	$0.118$	$0.035$

Table 12: Headline statistics across the four configurations of Finding 5, each averaged over $N{=}11$ runs. Main (released) is the configuration used for all other results in this paper. Positive signed divergence and near-chance $\kappa$ persist in every configuration.

指标	更简系统	主系统	主系统（已发布）	升级骨干
内在 (%)	$57.6\pm 1.8$	$61.1\pm 1.6$	$60.6\pm 1.6$	$61.4\pm 1.3$
外在 (%)	$48.8\pm 2.7$	$48.5\pm 2.1$	$48.8\pm 2.9$	$49.2\pm 3.4$
MAD	$0.164\pm 0.027$	$0.181\pm 0.022$	$0.168\pm 0.025$	$0.158\pm 0.035$
分歧率 (%)	$43.3\pm 6.0$	$49.4\pm 7.2$	$43.6\pm 6.1$	$39.7\pm 6.1$
有符号分歧	$+0.088\pm 0.034$	$+0.126\pm 0.025$	$+0.118\pm 0.029$	$+0.121\pm 0.036$
Cohen’s $\kappa$	$0.104$	$0.082$	$0.118$	$0.035$

表 12：发现 5 的四种配置的头条统计，各对 $N{=}11$ 次运行取平均。主系统（已发布） 是本文其余结果所用配置。正的有符号分歧与接近随机的 $\kappa$ 在每种配置中都持续存在。