ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models
ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models
ImplicitMemBench:度量大语言模型中的不自觉行为适应
Chonghan Qin¹, Xiachong Feng¹*, Weitao Ma², Xiaocheng Feng², Lingpeng Kong¹
¹The University of Hong Kong
²Harbin Institute of Technology
qinch@connect.hku.hk, fengxc@hku.hk
- Corresponding author. ¹Code and data are available at ImplicitMemBench.
Abstract
摘要
Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval. This gap is critical: effective assistants must automatically apply learned procedures or avoid failed actions without explicit reminders. We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three cognitively grounded constructs drawn from standard cognitive-science accounts of non-declarative memory: Procedural Memory (one-shot skill acquisition after interference), Priming (theme-driven bias via paired experimental/control instances), and Classical Conditioning (Conditioned Stimulus–Unconditioned Stimulus (CS–US) associations shaping first decisions). Our 300-item suite employs a unified Learning/Priming–Interfere–Test protocol with first-attempt scoring. Evaluation of 17 models reveals severe limitations: no model exceeds 66% overall, with top performers DeepSeek-R1 (65.3%), Qwen3-32B (64.1%), and GPT-5 (63.0%) far below human baselines. Analysis uncovers dramatic asymmetries (inhibition 17.6% vs. preference 75.0%) and universal bottlenecks requiring architectural innovations beyond parameter scaling. ImplicitMemBench reframes evaluation from ‘‘what agents recall’’ to ‘‘what they automatically enact’’ [^1].
现有面向大模型智能体的记忆基准,多半在测对事实的外显回忆,却很少触及内隐记忆——经验在无需有意识提取的情况下,自行变成自动化行为。这一缺口很要紧:真正好用的助手,应当能自动套用已经学会的程序,或避开曾经失败的动作,而不必每次再被明确提醒。我们提出 ImplicitMemBench,这是首个系统评测内隐记忆的基准。它从认知科学中关于非陈述性记忆的标准叙述出发,落地为三个有认知依据的构念:程序记忆(在干扰之后仍能一次习得技能)、启动效应(通过成对的实验/对照实例,测量主题带来的偏向),以及经典条件作用(条件刺激—无条件刺激(CS–US)联结如何塑造第一次决策)。全部 300 道题目共用一套学习/启动—干扰—测试协议,并按首次作答计分。对 17 个模型的评测暴露出严重局限:没有模型总分超过 66%,表现最好的 DeepSeek-R1(65.3%)、Qwen3-32B(64.1%)和 GPT-5(63.0%)也远低于人类基线。进一步分析还发现巨大的不对称(抑制 17.6%,偏好 75.0%),以及需要超越参数规模扩张才能克服的普遍瓶颈。ImplicitMemBench 把评测从「智能体记住了什么」转向「它们会自动做出什么」[^1]。
1 Introduction
1 引言
Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains (Zhao et al., 2023; Wu et al., 2024), maturing into assistants for daily productivity and specialized workflows (DeepSeek-AI et al., 2025; Zeng et al., 2025; OpenAI, 2025; Anthropic, 2025; Qwen et al., 2024; Yang et al., 2025). To serve users reliably, these systems require memory: the ability to accumulate and leverage experience across interactions. Recent work establishes this need (He et al., 2024; Zhang et al., 2025a; Zhang et al., 2025b), with production systems deploying memory features (OpenAI, 2024). Benchmarks such as LoCoMo, LongMemEval, MADial-Bench, MemBench, MemoryAgentBench, MEMTRACK, and GoodAI LTM now evaluate multi-session QA, retrieval, dialogue, state tracking, and conversational integration under explicit, actively triggered settings spanning contexts from 400 tokens to 1.5M tokens, or variable long-horizon environments (Maharana et al., 2024; Wu et al., 2025; He et al., 2025; Tan et al., 2025; Hu et al., 2026; Deshpande et al., 2025; Castillo-Bolado et al., 2024).
大语言模型已在诸多领域展现出可观能力(Zhao et al., 2023; Wu et al., 2024),并逐渐成为日常办公与专业流程中的助手(DeepSeek-AI et al., 2025; Zeng et al., 2025; OpenAI, 2025; Anthropic, 2025; Qwen et al., 2024; Yang et al., 2025)。要稳定服务用户,这些系统需要记忆:在多次交互中积累并调用经验。近期工作已经说明这一点(He et al., 2024; Zhang et al., 2025a; Zhang et al., 2025b),生产系统也开始部署记忆功能(OpenAI, 2024)。LoCoMo、LongMemEval、MADial-Bench、MemBench、MemoryAgentBench、MEMTRACK 和 GoodAI LTM 等基准,分别在多轮问答、检索、对话、状态跟踪和会话整合上做评测;它们都采用外显、主动触发的设定,上下文从 400 个 token 到 150 万个 token 不等,也有长度不固定的长程环境(Maharana et al., 2024; Wu et al., 2025; He et al., 2025; Tan et al., 2025; Hu et al., 2026; Deshpande et al., 2025; Castillo-Bolado et al., 2024)。
Despite this progress, existing benchmarks predominantly evaluate explicit memory: conscious retrieval of factual information. Table 1 shows prior work uniformly adopts query-response protocols where models are explicitly prompted to recall facts, limiting evaluation to deliberate retrieval rather than unconscious behavioral adaptation. Yet critical failures arise from missing implicit memory, experience that becomes automated behavior rather than explicit recollection. Effective assistants should automatically apply learned procedures after distractions or avoid repeatedly-failed tools without explicit reminders. Existing benchmarks fail to diagnose this because they (i) employ QA formats explicitly cueing target information, (ii) stress storage capacity over first-attempt triggers after interference, and (iii) utilize costly pipelines hindering reproducibility.
尽管已有这些进展,现有基准主要仍在评测外显记忆:对事实信息的有意识提取。表 1 显示,先前工作几乎一律采用问答协议,明确提示模型去回忆事实,于是评测停在刻意检索,而不是不自觉的行为适应。许多关键失败,其实来自缺失内隐记忆,也就是经验变成自动化行为,而不是明确回想。有效的助手应能在受到干扰后自动执行已学会的程序,或在没有明示提醒时避开反复失败的工具。现有基准难以诊断这一点,是因为它们(i)采用会明确提示目标信息的问答格式,(ii)更强调存储容量,而不是干扰之后的首次作答触发,(iii)流水线成本高,不利于复现。
| Benchmark | Memory Type | Evaluation Trigger | Context Scale (tokens) | Evaluation Size | Task Focus |
|---|---|---|---|---|---|
| LoCoMo (Maharana et al., 2024) | Explicit | Active (Explicit query) | 9K | 300 conversations | Multi-session QA |
| LongMemEval (Wu et al., 2025) | Explicit | Active (Explicit query) | 115K–1.5M | 500 questions | Information retrieval |
| MADial-Bench (He et al., 2025) | Explicit | Active (Emotion cued) | 400 | 160 dialogues | Emotional dialogue |
| MemBench (Tan et al., 2025) | Explicit | Active (Explicit query) | 1K–100K | 53,000 QAs | Reflective memory during observation |
| MemoryAgentBench (Hu et al., 2026) | Explicit | Active (Explicit query) | 100K–1.4M | 14 datasets / 2,071 QAs | 4 Competencies (Retrieval, Test-time Learning, Long Range Understanding, Selective Forgetting) |
| MEMTRACK (Deshpande et al., 2025) | Explicit | Active (Explicit task) | Variable | 210 instances | Multi-platform State Tracking |
| GoodAI LTM (Castillo-Bolado et al., 2024) | Explicit | Active (Explicit query) | 2K–500K | Variable | Conversational / Info Integration |
| ImplicitMemBench (Ours) | Implicit | Passive (Scenario) | 500 | 300 instances | Unconscious adaptation |
Table 1: Comparison of ImplicitMemBench with existing memory benchmarks. Most prior benchmarks target explicit memory through active retrieval triggers, whereas ImplicitMemBench evaluates implicit memory through passive, scenario-driven behavioral adaptation.
| 基准 | 记忆类型 | 评测触发 | 上下文规模(token) | 评测量 | 任务重心 |
|---|---|---|---|---|---|
| LoCoMo (Maharana et al., 2024) | 外显 | 主动(明示询问) | 9K | 300 段对话 | 多会话问答 |
| LongMemEval (Wu et al., 2025) | 外显 | 主动(明示询问) | 115K–1.5M | 500 道问题 | 信息检索 |
| MADial-Bench (He et al., 2025) | 外显 | 主动(情绪提示) | 400 | 160 段对话 | 情感对话 |
| MemBench (Tan et al., 2025) | 外显 | 主动(明示询问) | 1K–100K | 53,000 道问答 | 观察过程中的反思性记忆 |
| MemoryAgentBench (Hu et al., 2026) | 外显 | 主动(明示询问) | 100K–1.4M | 14 个数据集 / 2,071 道问答 | 四项能力(检索、测试时学习、长程理解、选择性遗忘) |
| MEMTRACK (Deshpande et al., 2025) | 外显 | 主动(明示任务) | 不固定 | 210 个实例 | 多平台状态跟踪 |
| GoodAI LTM (Castillo-Bolado et al., 2024) | 外显 | 主动(明示询问) | 2K–500K | 不固定 | 对话 / 信息整合 |
| ImplicitMemBench(本文) | 内隐 | 被动(情境驱动) | 500 | 300 个实例 | 不自觉适应 |
表 1:ImplicitMemBench 与现有记忆基准的对比。先前基准大多针对通过主动检索触发来考查的外显记忆,而 ImplicitMemBench 通过被动、由情境驱动的行为适应来评测内隐记忆。
We introduce ImplicitMemBench, a cognitively grounded benchmark evaluating implicit memory in LLM agents. We operationalize three constructs: Procedural Memory assesses one-shot skill acquisition persisting after interference, Priming measures theme-driven biases via paired experimental/control instances, and Classical Conditioning evaluates whether Conditioned Stimulus–Unconditioned Stimulus (CS–US) exposure shapes first decisions. Our paradigm selection is guided by the classical taxonomy of non-declarative memory (Squire, 2004). We focus on three mechanisms that are especially relevant to LLM agents: procedural memory for internalizing new routines, priming for context-driven adaptation without explicit instruction, and classical conditioning for forming automatic protective responses from experience. We exclude non-associative learning in this version because it is less directly aligned with the high-level semantic decision-making emphasized in current agentic systems. This grounding lets us map established cognitive constructs to text-based agent evaluation through functional isomorphism rather than surface analogy. All 300 items follow a unified Learning/Priming–Interfere–Test protocol with first-attempt scoring isolating automatized behavior from explicit recall. Figure 1 illustrates our framework: procedural tasks use rule-based validators, priming employs LLM judges comparing experimental versus control conditions, and classical conditioning tracks binary first-attempt avoidance. This design enables lightweight, reproducible evaluation revealing that implicit memory formation poses profound challenges with no model achieving human-like automaticity.
我们提出有认知依据的基准 ImplicitMemBench,用来评测大模型智能体的内隐记忆。三个构念的操作化如下:程序记忆考查一次习得的技能在干扰之后能否保持;启动效应通过成对的实验/对照实例,测量主题带来的偏向;经典条件作用则看条件刺激—无条件刺激(CS–US)的暴露,会不会塑造第一次决策。范式选择遵循非陈述性记忆的经典分类(Squire, 2004)。我们聚焦三类对大模型智能体尤其要紧的机制:程序记忆用来内化新惯例,启动效应用来在没有明示指令时做情境适应,经典条件作用用来从经验中形成自动的保护性反应。本版不包括非联结学习,因为它与当前智能体系统所强调的高层语义决策关系不那么直接。这样处理,是把既有认知构念通过功能同构映射到基于文本的智能体评测上,而不是做表面类比。全部 300 道题目共用一套学习/启动—干扰—测试协议,并以首次作答计分,从而把自动化行为与外显回忆分开。图 1 给出整体框架:程序任务使用基于规则的校验器,启动效应由大模型评委比较实验条件与对照条件,经典条件作用跟踪首次作答时的二元回避。这一设计轻量、可复现,结果说明内隐记忆的形成相当困难,没有模型达到接近人类的自动化水平。
Figure 1: Overall framework. (a) Dataset generation pipeline using LLM-generated candidates refined via fine-tuning and human editing. (b) Evaluation metrics: hybrid automatic/human validation for Procedural Memory (FTA), LLM-as-Judge for Priming (Priming Influence Score) and Classical Conditioning (FTA).
图 1:整体框架。(a)数据集生成流水线:先由大模型生成候选,再经微调与人工编辑加以精炼。(b)评测指标:程序记忆采用自动校验与人工核验相结合的首次作答正确率(FTA);启动效应由大模型评委给出启动影响分;经典条件作用同样以 FTA 衡量。
We evaluate 17 models spanning proprietary (GPT-5, Claude-4.1-opus, Gemini-2.5-pro) and open-source systems (DeepSeek-R1, Qwen3-32B, LLaMA-3.3-70B). As Figure 2 shows, results reveal fundamental limitations. First, a severe ceiling effect emerges: top performers remain far below human baselines, with no model exceeding two-thirds overall accuracy. Second, paradigm asymmetry shows dramatic variance: procedural memory proves most tractable while classical conditioning creates substantial bottlenecks, and priming clusters in a narrow moderate range. Third, capability dissociation reveals that excellence in one paradigm fails to predict success in others, with the strongest procedural learner suffering dramatic drops on classical conditioning. Fine-grained analysis uncovers systematic failures: models struggle profoundly with inhibition versus preference-based learning, and multiple categories remain universally challenging across all architectures.
我们评测了 17 个模型,既包括闭源系统(GPT-5、Claude-4.1-opus、Gemini-2.5-pro),也包括开源系统(DeepSeek-R1、Qwen3-32B、LLaMA-3.3-70B)。如图 2 所示,结果暴露出根本性的局限。第一,存在严重的天花板效应:领先模型仍远低于人类基线,没有模型总分超过三分之二。第二,范式不对称十分显著:程序记忆相对最好做,经典条件作用构成明显瓶颈,启动效应则挤在一段不宽的中等区间。第三,能力分离表明,某一范式上的出色表现并不能预示其他范式同样成功——程序记忆最强的模型,在经典条件作用上也会大幅下滑。细粒度分析还发现系统性失败:模型在抑制相对偏好学习上尤其吃力,而且有多类任务对所有架构都普遍困难。
图 2:全部受评模型在 ImplicitMemBench 上的成绩排序。模型按总分排列,闭源模型为红珊瑚色,开源模型为青绿色。水平虚线标出全部模型的平均成绩。
Our main contributions are: (1) We present the first benchmark of implicit memory in LLMs, operationalizing procedural learning, priming, and classical conditioning under a unified protocol isolating automatized behavior through first-attempt scoring. (2) We design an evaluation framework combining rule-based validators and LLM judges, enabling reproducible assessment via a compact 300-item suite. (3) We evaluate 17 models revealing critical weaknesses: severe behavioral asymmetries and universal bottlenecks requiring architectural innovations beyond parameter scaling. (4) We further show that representative memory-augmented agents do not consistently improve performance on ImplicitMemBench, suggesting that implicit memory cannot be reduced to explicit storage and retrieval alone.
主要贡献如下:(1)给出面向大模型内隐记忆的首个基准,在统一协议下把程序学习、启动效应和经典条件作用操作化,并用首次作答计分把自动化行为分离出来。(2)设计结合规则校验器与大模型评委的评测框架,以紧凑的 300 题套件支持可复现评估。(3)评测 17 个模型,揭示严重的行为不对称和普遍瓶颈,说明需要超越参数规模扩张的架构创新。(4)进一步表明,有代表性的记忆增强智能体并不能在 ImplicitMemBench 上稳定提高成绩,内隐记忆不能被还原成外显的存储与检索。
2 Related Work
2 相关工作
Existing memory benchmarks for LLM agents predominantly evaluate explicit memory through active retrieval triggers. LoCoMo (Maharana et al., 2024) studies multi-session dialogue continuity via QA and event summarization over 300 conversations of roughly 9k tokens each. LongMemEval (Wu et al., 2025) examines long-context memory in 115k–1.5M-token settings across retrieval and multi-session reasoning tasks. MADial-Bench (He et al., 2025) focuses on emotion-support dialogue in 160 dialogues. MemBench (Tan et al., 2025) expands evaluation to factual and reflective memory across participation/observation scenarios with contexts ranging from 1K to 100K tokens. More recent benchmarks broaden task coverage while retaining the same explicit framing: MemoryAgentBench (Hu et al., 2026) evaluates four competencies—accurate retrieval, test-time learning, long-range understanding, and selective forgetting—over 14 datasets / 2,071 QAs in 100K–1.4M-token settings; MEMTRACK (Deshpande et al., 2025) studies multi-platform state tracking in variable long-horizon environments; and GoodAI LTM (Castillo-Bolado et al., 2024) emphasizes conversational information integration in 2K–500K-token contexts. As summarized in Table 1, prior work covers increasingly diverse tasks and scales, but still targets explicit memory through active triggers, leaving implicit memory unaddressed. ImplicitMemBench fills this gap as the first systematic evaluation of implicit memory via procedural learning, priming, and classical conditioning, using efficient ~500-token protocols to assess unconscious behavioral adaptations. Beyond benchmark design, another line of work augments LLM agents with external memory modules or retrieval-based long-term memory systems. These methods primarily target explicit storage and recall. As we show in Section D, however, such mechanisms do not consistently improve performance on ImplicitMemBench, indicating that implicit memory is not reducible to explicit retrieval alone.
现有面向大模型智能体的记忆基准,主要通过主动检索触发来评测外显记忆。LoCoMo(Maharana et al., 2024)用问答和事件摘要研究多会话对话连续性,共 300 段对话,每段大约 9k 个 token。LongMemEval(Wu et al., 2025)在 115k–1.5M 个 token 的设定下考查长上下文记忆,覆盖检索与多会话推理。MADial-Bench(He et al., 2025)关注 160 段情绪支持对话。MemBench(Tan et al., 2025)把评测扩展到事实记忆与反思性记忆,覆盖参与/观察两类情景,上下文从 1K 到 100K 个 token。更新的基准扩大了任务覆盖,但框架仍是外显的:MemoryAgentBench(Hu et al., 2026)在 100K–1.4M 个 token 的设定下,用 14 个数据集、2,071 道问答评测四项能力——准确检索、测试时学习、长程理解和选择性遗忘;MEMTRACK(Deshpande et al., 2025)研究可变长程环境中的多平台状态跟踪;GoodAI LTM(Castillo-Bolado et al., 2024)则强调 2K–500K 个 token 上下文中的对话信息整合。如表 1 所总结,先前工作在任务和规模上越来越多样,但仍然通过主动触发考查外显记忆,内隐记忆仍是空白。ImplicitMemBench 补上这一缺口,以程序学习、启动效应和经典条件作用系统评测内隐记忆,并用约 500 个 token 的高效协议衡量不自觉的行为适应。在基准设计之外,另有工作为智能体加装外部记忆模块或基于检索的长期记忆。这些方法主要针对外显存储与回忆。我们在附录 D 中表明,这类机制并不能在 ImplicitMemBench 上稳定提高成绩,说明内隐记忆不能被还原成外显检索。
3 ImplicitMemBench
3.1 Cognitive Grounding and Paradigm Selection
3.1 认知依据与范式选择
Our benchmark is grounded in the taxonomy of non-declarative memory (Squire, 2004). We focus on procedural memory, priming, and classical conditioning because they capture complementary forms of implicit adaptation that are especially relevant to LLM agents. We do not include non-associative learning in the current version, as it is less directly connected to the semantic, decision-oriented behaviors emphasized in modern agent workflows. Our design principle is functional isomorphism: we translate standard cognitive mechanisms into text-based agentic scenarios while preserving their core causal structure.
本基准以非陈述性记忆的分类为根据(Squire, 2004)。我们聚焦程序记忆、启动效应和经典条件作用,因为它们刻画了彼此补充、又对大模型智能体特别要紧的内隐适应形式。当前版本不包括非联结学习,因为它与现代智能体工作流所强调的语义决策行为关系较远。设计原则是功能同构:把标准认知机制转写为基于文本的智能体情景,同时保留其核心因果结构。
3.2 Tasks
3.2 任务
We operationalize these three paradigms as follows: Procedural Memory (§ 3.2.1) tests acquisition of new behavioral patterns from minimal exposure; Priming (§ 3.2.2) measures unconscious transfer of thematic elements from prior context; and Classical Conditioning (§ 3.2.3) evaluates formation of automatic stimulus-response associations through repeated pairing.
三个范式的操作化如下:程序记忆(§ 3.2.1)考查能否从极少暴露中习得新的行为模式;启动效应(§ 3.2.2)测量先前上下文中主题要素的不自觉迁移;经典条件作用(§ 3.2.3)评测通过反复配对能否形成自动的刺激—反应联结。
3.2.1 Procedural Memory
3.2.1 程序记忆
Motivation
动机
Procedural memory enables automatic execution of learned skills without conscious recall. For AI agents, this means internalizing novel operational protocols from minimal demonstrations and executing them flawlessly despite distractions. Current LLMs excel at declarative knowledge but often revert to pre-trained patterns when new rules contradict their priors. We evaluate whether models can truly proceduralize routines, transforming explicit instructions into automatic behaviors that persist through interference.
程序记忆使人能够自动执行已学会的技能,而不必有意识地回想。对人工智能智能体而言,这意味着从极少示范中内化新的操作协议,并在受到干扰后仍能准确执行。当前大模型擅长陈述性知识,可一旦新规则与预训练模式冲突,往往又退回到旧习惯。我们要评测的是:模型能否真正把惯例程序化,把明示指令变成能够穿过干扰仍保持的自动行为。
Task Design
任务设计
We structure procedural memory evaluation across five complementary domains, each targeting different aspects of rule internalization and automatic execution, shown in Table 2.
程序记忆评测覆盖五个互补领域,分别针对规则内化与自动执行的不同侧面,见表 2。
| Domain | Core Challenge | Representative Tasks |
|---|---|---|
| Tool & API Usage | Override ingrained calling conventions | Reversed Parameters (dstsrc), Session Prefix (auth fusion), Alien Filesystem (custom separators) |
| Linguistic Formats | Internalize arbitrary templates | Scribe’s Signature (fixed wrapper), Corporate Etiquette (mandated greeting/closing) |
| Logical Operations | Apply non-standard operators | Omega: , Modified Fibonacci: |
| Abstract Rules | Form habits in micro-worlds | Forbidden Square (board constraint), Triple Knock (ritual sequence) |
| Creative Constraints | Maintain style without reminders | Voice Consistency (no first-person), Botanical Similes (nature metaphors) |
Table 2: Five procedural memory domains. Each forces models to suppress pre-trained behaviors for new procedures.
| 领域 | 核心难点 | 代表性任务 |
|---|---|---|
| 工具与 API 使用 | 覆盖已经固化的调用惯例 | 参数对调(dstsrc)、会话前缀(认证拼接)、外星文件系统(自定义分隔符) |
| 语言格式 | 内化任意模板 | 抄写员署名(固定包装)、公司礼仪(规定的问候/结尾) |
| 逻辑运算 | 套用非标准算符 | Omega:;改进斐波那契: |
| 抽象规则 | 在微世界中形成习惯 | 禁入方格(棋盘约束)、三击(仪式序列) |
| 创造性约束 | 无需提醒也能维持风格 | 声音一致性(不用第一人称)、植物比喻(自然隐喻) |
表 2:程序记忆的五个领域。每一领域都迫使模型抑制预训练行为,改用新的程序。
These tasks emphasize proceduralization over memorization: models must internalize rules from minimal exposure and execute them despite extensive interference and format variations. The design systematically varies exemplar clarity, interference intensity, and the presence of misleading alternatives to probe the depth of behavioral automation.
这些任务强调程序化,而不是死记:模型必须从极少暴露中内化规则,并在大量干扰和格式变化之后仍能执行。设计还系统变化示例清晰度、干扰强度,以及是否出现误导性备选,以探测行为自动化的深度。
Evaluation Framework: Learning-Interference-Test Protocol
评测框架:学习—干扰—测试协议
Our three-phase protocol provides minimal specification (rule + 1-3 examples), extensive interference (15 misleading turns), then novel test probes requiring first-attempt success. This isolates procedural memory from explicit recall by testing whether routines automatize despite interference. Validation uses deterministic parsing for structured outputs and LLM judgment for semantic adherence.
三阶段协议只给出很少的说明(规则加 1–3 个例子),随后是大量干扰(15 轮误导性对话),再用新颖的测试探针要求首次作答即成功。这样可以把程序记忆与外显回忆分开:看惯例是否在干扰之后已经自动化。校验方面,结构化输出用确定性解析,语义合规则由大模型评委判断。
3.2.2 Priming
3.2.2 启动效应
Motivation
动机
Priming demonstrates how prior exposure unconsciously influences subsequent behavior. For AI agents, this implicit contextual sensitivity is crucial: absorbing environmental cues to shape responses without explicit instruction. Current LLMs exhibit shallow keyword repetition but lack true thematic priming; they struggle to let abstract patterns from prior context subtly influence creative outputs. We evaluate whether models can internalize thematic schemas and transfer them implicitly through intervening distractions.
启动效应表明,先前暴露会在不知不觉中影响随后的行为。对人工智能智能体来说,这种内隐的情境敏感很关键:吸收环境线索来塑造回应,而不需要明示指令。当前大模型往往只会做浅层的关键词重复,缺乏真正的主题启动;它们很难让先前上下文中的抽象模式,细致地影响创造性输出。我们要评测的是:模型能否内化主题图式,并在中间插入干扰之后,仍把它们不自觉地迁移出去。
Task Design
任务设计
We structure priming evaluation through matched experimental-control pairs that isolate the causal effect of thematic exposure, as shown in Table 3.
启动效应评测采用成对的实验—对照设计,以分离主题暴露的因果效应,见表 3。
| Component | Experimental Condition | Control Condition |
|---|---|---|
| Priming Text | Rich thematic paragraph (e.g., Abyssal Deep-Sea: bioluminescence, crushing pressure, ancient creatures) | Technical specification (e.g., ISO Container Standards: dimensions, load capacity, stacking protocols) |
| Interference | Identical neutral technical content (2 turns) | |
| Test Probe | Identical creative task (naming, tagline, concept generation) | |
| Expected Result | Thematic bias toward primed domain | Neutral, generic responses |
Table 3: Priming experimental design with matched controls. Each row is a complete trial pair.
| 组成 | 实验条件 | 对照条件 |
|---|---|---|
| 启动文本 | 主题丰富的段落(例如 深海深渊:生物荧光、高压、远古生物) | 技术说明(例如 ISO 集装箱标准:尺寸、载重、堆叠规范) |
| 干扰 | 相同的中性技术内容(2 轮) | |
| 测试探针 | 相同的创造性任务(命名、标语、概念生成) | |
| 预期结果 | 向被启动领域的主题偏向 | 中性、泛化的回答 |
表 3:带匹配对照的启动实验设计。每一行都是完整的一对试次。
Themes span diverse conceptual territories: Arctic Expedition, Volcanic Eruption, Renaissance Alchemy, Ancient Oracle, each with distinct sensory-emotional signatures. This design isolates implicit transfer: differences between experimental and control responses reveal pure priming effects.
主题覆盖不同的概念领域:北极探险、火山喷发、文艺复兴炼金术、古代神谕,各自带有不同的感官—情绪特征。这样的设计用来分离内隐迁移:实验条件与对照条件回答之间的差异,就是纯粹的启动效应。
Evaluation Framework: Priming-Interference-Test Protocol
评测框架:启动—干扰—测试协议
Our three-phase design provides thematic exposure (evocative vs neutral), neutral interference (cognitive buffer), then creative generation tasks. This measures unconscious thematic transfer via systematic bias in outputs despite no explicit reminders. LLM-based evaluation detects thematic alignment beyond surface keywords.
三阶段设计先给出主题暴露(唤起性文本对中性文本),再插入中性干扰(作为认知缓冲),最后是创造性生成任务。它测量的是不自觉的主题迁移:即使没有明示提醒,输出中是否仍系统出现偏向。基于大模型的评测用于检测超越表层关键词的主题对齐。
3.2.3 Classical Conditioning
3.2.3 经典条件作用
Motivation
动机
Classical conditioning enables organisms to form automatic stimulus-response associations. This unconscious learning prevents harm through rapid, automatic responses. For AI agents, such adaptive safety mechanisms are essential: learning from negative experiences to automatically avoid harmful patterns. Current LLMs lack true conditioning and cannot form persistent associations from feedback. We evaluate whether models can establish defensive reflexes through experience rather than instruction.
经典条件作用使有机体能够形成自动的刺激—反应联结。这种不自觉学习通过迅速、自动的反应来避免伤害。对人工智能智能体而言,这类适应性安全机制同样必要:从负面经验中学习,从而自动避开有害模式。当前大模型缺乏真正的条件作用,无法从反馈中形成持久联结。我们要评测的是:模型能否凭借经验、而不是指令,建立起防御性反射。
Task Design
任务设计
We structure conditioning tasks across three domains where automatic avoidance prevents system failures:
我们把条件作用任务放在三个领域中,这些领域里,自动回避可以防止系统失效:
Examples: API Aversion (keyword triggers alternative selection), Filetype Preference (failures condition format choices), Directory Restriction (errors establish path boundaries). Each task requires forming unconscious protective associations. More can be found in Table 4.
例如:API 厌恶(关键词触发改选替代方案)、文件类型偏好(失败会条件化格式选择)、目录限制(错误划出路径边界)。每项任务都要求形成不自觉的保护性联结。更多内容见表 4。
| Domain | Stimulus Pattern (CS) | Learned Response (CR) |
|---|---|---|
| Tool & API Safety | Trigger keywords, filetypes, API names, load indicators | Switch to reliable alternatives, add warnings, delay execution, avoid side-effects |
| Conversational Adaptation | User confusion signals, impatience cues, emotional markers | Simplify jargon, reduce verbosity, adjust response format |
| System Protection | Forbidden paths, insecure protocols, dangerous patterns | Use safe defaults, enforce security, prevent violations |
Table 4: Classical conditioning tasks by domain. CS = Conditioned Stimulus, US = Unconditioned Stimulus, CR = Conditioned Response.
| 领域 | 刺激模式(CS) | 习得反应(CR) |
|---|---|---|
| 工具与 API 安全 | 触发关键词、文件类型、API 名称、负载指示 | 改用更可靠的替代方案、补充警告、延迟执行、避免副作用 |
| 对话适应 | 用户困惑信号、不耐烦线索、情绪标记 | 简化行话、减少冗长、调整回复格式 |
| 系统防护 | 禁入路径、不安全协议、危险模式 | 使用安全默认、强制执行安全策略、防止违规 |
表 4:按领域划分的经典条件作用任务。CS = 条件刺激,US = 无条件刺激,CR = 条件反应。
Evaluation Framework: Learning-Interference-Test Protocol
评测框架:学习—干扰—测试协议
Our three-phase structure provides CS-US pairings, unrelated tasks (temporal distance), then CS reintroduction requiring first-action responses. This tests unconscious defensive learning via immediate avoidance/adaptation when CS reappears, without reminders. LLM judgment evaluates whether first actions demonstrate learned protective behaviors.
三阶段结构先给出 CS–US 配对,再插入无关任务(拉开时间距离),最后重新引入 CS,并要求第一次行动即作出反应。这样测试的是不自觉的防御学习:当 CS 再次出现且没有提醒时,能否立即回避或适应。大模型评委判断首次行动是否体现出已学会的保护行为。
3.3 Data Generation and Quality Control
3.3 数据生成与质量控制
Our dataset construction employs a two-stage pipeline ensuring diversity and quality. Each item begins as a structured blueprint, then undergoes automated generation and rigorous quality control. All the prompts used could be found in Appendix F.
数据集构建采用两阶段流水线,以兼顾多样性与质量。每道题目先写成结构化蓝本,再经自动生成和严格质控。所用提示全部见附录 F。
Generation Pipeline
生成流水线
Stage 1 uses GPT-4o-mini to instantiate concrete dialogues from task templates, creating learning materials, interference content, and test probes. Stage 2 combines automated checks and human review to verify structural requirements and semantic correctness, removing unintended shortcuts.
第一阶段用 GPT-4o-mini 按任务模板实例化具体对话,生成学习材料、干扰内容和测试探针。第二阶段把自动检查与人工审阅结合起来,核验结构要求和语义正确性,并去掉无意中留下的捷径。
Paradigm-Specific Requirements
各范式的特定要求
As shown in Table 5, each memory type demands distinct generation strategies to create valid memory challenges.
如表 5 所示,每类记忆都需要不同的生成策略,才能构成有效的记忆考题。
| Component | Procedural Memory | Priming | Classical Conditioning |
|---|---|---|---|
| Learning Phase | 1-3 turns: rule + examples | 1 turn: thematic paragraph | 4 turns: CS-US pairings |
| Interference | 10-15 turns: misleading but related content | 2 turns: neutral technical task | 2 turns: unrelated dialogue |
| Test Format | Novel application of learned rule | Creative generation task | CS reintroduction |
| Key Challenge | Avoid rule restatement | Prevent theme leakage | Ensure clear causality |
| Validation Focus | Rule adherence | Thematic influence | Avoidance behavior |
Table 5: Data generation specifications across paradigms. Turn counts and content ensure proper memory formation and testing.
| 组成 | 程序记忆 | 启动效应 | 经典条件作用 |
|---|---|---|---|
| 学习阶段 | 1–3 轮:规则加示例 | 1 轮:主题段落 | 4 轮:CS–US 配对 |
| 干扰 | 10–15 轮:有误导性但相关的内容 | 2 轮:中性技术任务 | 2 轮:无关对话 |
| 测试形式 | 把已学规则用到新情境 | 创造性生成任务 | 重新引入 CS |
| 关键难点 | 避免重述规则 | 防止主题泄漏 | 保证因果关系清楚 |
| 校验重点 | 是否遵守规则 | 主题影响 | 回避行为 |
表 5:各范式的数据生成规格。轮次和内容需保证记忆能够形成,并得到有效测试。
Quality Assurance Protocol
质量保证协议
Multi-layer validation ensures dataset integrity: automated checks verify structure (turn counts, token limits, formats); LLM judges assess semantic adequacy; systematic reviews prevent test-phase leakage; diversity enforcement prioritizes novel instances. This yields 300 high-quality items testing implicit memory from an initial pool of over 1,000 generated candidates.
多层校验用来保证数据完整:自动检查核验结构(轮次数、token 上限、格式);大模型评委评估语义是否充分;系统审查防止测试阶段泄漏;多样性约束优先保留新颖实例。这样,从最初生成的 1,000 余条候选中,得到 300 道高质量的内隐记忆测试题。
3.4 Dataset Statistics
3.4 数据集统计
ImplicitMemBench comprises 300 carefully constructed items balanced across three memory paradigms. Each item follows our unified learning-interference-test protocol, with phase structures optimized for implicit memory formation. Despite its compact size, the benchmark spans 18 task families across three paradigms and provides sufficient discriminative power in practice.
ImplicitMemBench 包含 300 道精心构造、在三个记忆范式间均衡分布的题目。每道题都遵循统一的学习—干扰—测试协议,阶段结构针对内隐记忆的形成做了优化。尽管规模紧凑,基准仍覆盖三个范式下的 18 个任务族,并在实践中具有足够的区分力。
Dataset Composition
数据集构成
Our benchmark consists of 100 items per paradigm, covering diverse task families, as shown in Table 6.
基准在每个范式上各有 100 道题,覆盖多样的任务族,见表 6。
| Paradigm | Items | Task Families | Validation |
|---|---|---|---|
| Procedural Memory | 100 | 5 domains (Tool, Linguistic, Logic, Rules, Creative) | 18% rule-based |
| Priming | 100 | 10 thematic domains + matched controls | 100% LLM-judged |
| Classical Conditioning | 100 | 3 domains (Tool Safety, Conversation, System) | 100% LLM-judged |
| Total | 300 | 18 unique families | 6% rule / 94% LLM |
Table 6: Dataset composition showing task diversity and validation methods.
| 范式 | 题数 | 任务族 | 校验方式 |
|---|---|---|---|
| 程序记忆 | 100 | 5 个领域(工具、语言、逻辑、规则、创造) | 18% 基于规则 |
| 启动效应 | 100 | 10 个主题领域加匹配对照 | 100% 由大模型评判 |
| 经典条件作用 | 100 | 3 个领域(工具安全、对话、系统) | 100% 由大模型评判 |
| 合计 | 300 | 18 个独特任务族 | 6% 规则 / 94% 大模型 |
表 6:数据集构成,展示任务多样性与校验方法。
Phase Structure and Token Distribution
阶段结构与 token 分布
Figure 3 visualizes the characteristic patterns of each paradigm. Procedural Memory emphasizes extensive interference (74% of tokens) to test rule persistence. Classical Conditioning concentrates on learning (72% of tokens) to establish strong associations. Priming maintains balanced phases for controlled comparison.
图 3 可视化了各范式的典型模式。程序记忆强调大量干扰(占 token 的 74%),以测试规则能否保持。经典条件作用把比重放在学习阶段(占 token 的 72%),以便建立牢固联结。启动效应则保持各阶段相对均衡,便于做受控比较。
图 3:各范式的数据集统计。(a)各阶段 token 分布,给出中位数与四分位。(b)阶段占比,用以突出不同的记忆测试策略。(c)各范式的轮次结构。
Context-Length Sensitivity.
上下文长度敏感性。
We set the context budget to ~500 tokens based on preliminary sensitivity analysis, with detailed ablations reported in Appendix B.3.
根据初步敏感性分析,我们把上下文预算定为约 500 个 token,详细消融见附录 B.3。
4 Experiments
4 实验
4.1 Experimental Setup
4.1 实验设置
We evaluate the implicit memory capabilities of 17 state-of-the-art language models (detailed in Appendix A), spanning both proprietary and open-source systems. Our evaluation protocol ensures fair comparison through standardized prompting and controlled generation parameters across all three memory paradigms.
我们评测了 17 个先进语言模型的内隐记忆能力(详见附录 A),既有闭源系统也有开源系统。评测协议通过统一提示和受控的生成参数,保证三个记忆范式上的比较是公平的。
Evaluation Protocol
评测协议
All models operate under identical conditions: zero-shot conversational interaction, no task-specific examples or fine-tuning, max 4096 tokens per response. Temperature: (deterministic) for procedural memory and classical conditioning ensuring reproducible first-attempt scoring; at test phase only for priming enabling creative variance; for LLM judges. This measures genuine implicit memory formation rather than pattern matching or stochastic variation.
所有模型在相同条件下运行:零样本对话交互,没有任务专用示例或微调,每次回复最多 4096 个 token。温度设置如下:程序记忆和经典条件作用使用 (确定性),以保证首次作答计分可复现;启动效应仅在测试阶段使用 ,以便产生创造性变异;大模型评委使用 。这样测量的是真正的内隐记忆形成,而不是模式匹配或随机波动。
Human Baseline.
人类基线。
To contextualize model performance, we collected a human baseline from five computer science Ph.D. students, each of whom completed the full 300-item benchmark under the same Learning/Priming–Interfere–Test protocol. Their responses were independently scored by two additional computer science Ph.D. students using the same rubric as for model evaluation. Inter-annotator agreement was 100%, and all five participants achieved 100% accuracy across all three paradigms.
为给模型成绩提供参照,我们采集了人类基线:五名计算机科学博士生各自在同一套学习/启动—干扰—测试协议下做完整的 300 道题。他们的回答由另外两名计算机科学博士生按与模型评测相同的量表独立打分。标注者间一致性为 100%,五名参与者在三个范式上均达到 100% 正确率。
4.2 Evaluation Metrics
4.2 评测指标
We employ paradigm-specific metrics that capture the distinct nature of each memory type. Binary accuracy suffices for procedural and conditioning tasks, while priming requires nuanced scoring of thematic influence.
我们采用与各记忆类型相匹配的指标。程序记忆和条件作用用二元正确率即可,启动效应则需要对主题影响做更细致的评分。
First-Try Accuracy (FTA)
首次作答正确率(FTA)
For Procedural Memory and Classical Conditioning, we measure success through first-attempt correctness: . This metric enforces strict evaluation: only the model’s initial response counts, with self-corrections or revisions ignored. This captures genuine memory formation rather than iterative refinement, analogous to human performance under time pressure where reflexive responses reveal true internalization.
对程序记忆和经典条件作用,我们以首次作答是否正确来衡量成功:。这一指标执行严格评测:只计模型的初次回复,自行改正或修订一律不计。这样捕捉的是真正的记忆形成,而不是反复打磨;这类似于人类在时间压力下的表现,反射性回答更能揭示内化程度。
Priming Influence Score (PIS)
启动影响分(PIS)
Priming evaluation requires detecting subtle thematic transfer rather than binary correctness. We employ a comparative scoring framework that quantifies influence magnitude:
启动效应的评测需要检测细微的主题迁移,而不是二元对错。我们采用一套比较评分框架,用来量化影响的幅度:
Scoring Protocol:
评分协议:
An LLM judge (GPT-4o-mini, ) performs pairwise comparison between experimental and control conditions, identifying thematic elements unique to the experimental condition. It evaluates lexical echoes and multi-axis alignment (setting, motifs, dynamics, affect), excluding generic metaphors. Hard caps apply (no echo 20; single axis 40) with 5-10 point penalties for baseline overlap, isolating true priming effects from general creative tendencies. More details can be found in Appendix B.1.
大模型评委(GPT-4o-mini,)对实验条件与对照条件做成对比较,识别只属于实验条件的主题要素。它评估词面回声以及多轴对齐(场景、母题、动态、情感),并排除泛化隐喻。硬上限为:无回声 20;单一轴 40;若与基线重叠,再扣 5–10 分,从而把真正的启动效应与一般的创造倾向分开。更多细节见附录 B.1。
Judge Robustness.
评委稳健性。
To assess the robustness of LLM-as-Judge scoring, we re-evaluated all 17 models with Gemini-2.5-Flash as an independent second judge in addition to GPT-4o-mini. Rankings remained highly stable: the top 11 and bottom 2 positions were unchanged. This suggests that our conclusions do not depend on a single judge model or model family. Additional results are reported in Appendix B.2.
为检验大模型评委打分是否稳健,我们在 GPT-4o-mini 之外,用 Gemini-2.5-Flash 作为独立的第二评委,对全部 17 个模型重新评分。排名高度稳定:前 11 名和最后 2 名的位置都没有变化。这说明结论并不依赖某一个评委模型或模型族。补充结果见附录 B.2。
4.3 Main Results
4.3 主要结果
Overview
总览
Table 7 presents performance across 17 state-of-the-art systems, revealing fundamental limitations. First, a clear ceiling effect emerges: no model exceeds 66% overall, and even the strongest system remains far below the human baseline of 100%, showing that implicit memory formation remains highly challenging for current LLMs. Second, paradigm asymmetry is evident as performance varies dramatically; procedural memory is most tractable (top: 75 to 77%) while classical conditioning creates bottlenecks (best: 69.7%). Third, capability dissociation shows that excellence in one paradigm doesn’t predict success in others, suggesting distinct mechanisms.
表 7 给出 17 个先进系统的成绩,暴露出根本局限。第一,天花板效应很清楚:没有模型总分超过 66%,即便最强系统也远低于 100% 的人类基线,说明当前大模型形成内隐记忆仍然非常困难。第二,范式不对称十分明显,成绩差异很大;程序记忆最好做(最高 75% 到 77%),经典条件作用则构成瓶颈(最好 69.7%)。第三,能力分离表明,某一范式上的出色并不能预示其他范式同样成功,暗示背后机制并不相同。
| Rank | Model | Procedural | Classical | Priming | Overall |
|---|---|---|---|---|---|
| Memory | Conditioning | Score | Score | ||
| Elite Tier (Overall 63%) | |||||
| 1 | DeepSeek-R1*†* | 76.33 | 69.67 | 49.90 | 65.30 |
| 2 | Qwen3-32B*†* | 75.67 | 67.00 | 49.73 | 64.13 |
| 3 | GPT-5 | 75.33 | 64.00 | 49.67 | 63.00 |
| Strong Tier (55% Overall 63%) | |||||
| 4 | Qwen3-8B*†* | 75.33 | 64.00 | 47.73 | 62.35 |
| 5 | GPT-o3 | 76.00 | 57.67 | 51.70 | 61.79 |
| 6 | GPT-o4-mini-high | 70.67 | 60.00 | 51.95 | 60.87 |
| 7 | GLM-4.5*†* | 76.33 | 53.33 | 46.12 | 58.59 |
| 8 | Gemini-2.5-pro | 74.33 | 47.33 | 45.42 | 55.69 |
| 9 | Claude-4.1-opus | 76.67 | 41.67 | 48.60 | 55.65 |
| 10 | Gemini-2.5-flash | 72.33 | 49.00 | 44.97 | 55.43 |
| Moderate Tier (45% Overall 55%) | |||||
| 11 | GPT-4o-mini | 61.67 | 44.00 | 46.98 | 50.88 |
| 12 | Qwen-2.5-72B*†* | 61.00 | 47.00 | 44.33 | 50.78 |
| 13 | GPT-4o | 61.67 | 43.67 | 45.62 | 50.32 |
| 14 | Claude-4-sonnet | 51.67 | 51.67 | 46.17 | 49.84 |
| 15 | LLaMA-3.3-70B*†* | 58.33 | 47.33 | 42.67 | 49.44 |
| Limited Tier (Overall 45%) | |||||
| 16 | LLaMA-3.1-8B*†* | 46.67 | 38.33 | 47.53 | 44.18 |
| 17 | Qwen-2.5-7B*†* | 50.67 | 35.67 | 44.12 | 43.49 |
| *†*Open-source model |
Table 7: ImplicitMemBench performance across 17 language models. Results show mean accuracy (%) over three independent runs for Procedural Memory and Classical Conditioning, with Priming scores from LLM-based evaluation. Models are ranked by overall performance and grouped into tiers. Bold: best in paradigm; underlined: second-best.
| 名次 | 模型 | 程序 | 经典 | 启动 | 总分 |
|---|---|---|---|---|---|
| 记忆 | 条件作用 | 分数 | 分数 | ||
| 第一档(总分 63%) | |||||
| 1 | DeepSeek-R1*†* | 76.33 | 69.67 | 49.90 | 65.30 |
| 2 | Qwen3-32B*†* | 75.67 | 67.00 | 49.73 | 64.13 |
| 3 | GPT-5 | 75.33 | 64.00 | 49.67 | 63.00 |
| 第二档(55% 总分 63%) | |||||
| 4 | Qwen3-8B*†* | 75.33 | 64.00 | 47.73 | 62.35 |
| 5 | GPT-o3 | 76.00 | 57.67 | 51.70 | 61.79 |
| 6 | GPT-o4-mini-high | 70.67 | 60.00 | 51.95 | 60.87 |
| 7 | GLM-4.5*†* | 76.33 | 53.33 | 46.12 | 58.59 |
| 8 | Gemini-2.5-pro | 74.33 | 47.33 | 45.42 | 55.69 |
| 9 | Claude-4.1-opus | 76.67 | 41.67 | 48.60 | 55.65 |
| 10 | Gemini-2.5-flash | 72.33 | 49.00 | 44.97 | 55.43 |
| 第三档(45% 总分 55%) | |||||
| 11 | GPT-4o-mini | 61.67 | 44.00 | 46.98 | 50.88 |
| 12 | Qwen-2.5-72B*†* | 61.00 | 47.00 | 44.33 | 50.78 |
| 13 | GPT-4o | 61.67 | 43.67 | 45.62 | 50.32 |
| 14 | Claude-4-sonnet | 51.67 | 51.67 | 46.17 | 49.84 |
| 15 | LLaMA-3.3-70B*†* | 58.33 | 47.33 | 42.67 | 49.44 |
| 第四档(总分 45%) | |||||
| 16 | LLaMA-3.1-8B*†* | 46.67 | 38.33 | 47.53 | 44.18 |
| 17 | Qwen-2.5-7B*†* | 50.67 | 35.67 | 44.12 | 43.49 |
| *†*开源模型 |
表 7:17 个语言模型在 ImplicitMemBench 上的成绩。程序记忆与经典条件作用给出三次独立运行的平均正确率(%),启动效应分数来自大模型评测。模型按总分排序并分档。加粗:该范式最优;下划线:次优。
Performance Landscape
成绩分布
Models stratify into distinct tiers. Elite performers (accuracy 63%) include only three systems: DeepSeek-R1 (65.30%), Qwen3-32B (64.13%), and GPT-5 (63.00%), with median performance at 55% showing substantial variance across paradigms.
模型分成清晰的几档。第一档(正确率 63%)只有三个系统:DeepSeek-R1(65.30%)、Qwen3-32B(64.13%)和 GPT-5(63.00%);中位成绩为 55%,各范式之间的方差相当大。
Paradigm-Specific Analysis
分范式分析
Procedural memory shows highest success with eight models achieving 70% (top: 76-77%), though 25% error rates indicate imperfect consolidation. Classical conditioning exposes critical weakness: only DeepSeek-R1 and Qwen3-32B exceed 65%. Priming scores cluster tightly at 42-52% with minimal differentiation, suggesting thematic influence operates near a common threshold.
程序记忆成功率最高,有八个模型超过 70%(最高 76–77%),但 25% 的错误率说明巩固并不完全。经典条件作用暴露出关键弱点:只有 DeepSeek-R1 和 Qwen3-32B 超过 65%。启动效应分数紧密聚集在 42–52%,区分很小,说明主题影响大致处在一个共同阈值附近。
Cross-Paradigm Patterns
跨范式模式
Data reveals striking dissociations. Claude-4.1-opus exemplifies this: highest procedural score (76.67%) but drops to 41.67% on classical conditioning, a 35-point gap highlighting capability independence. DeepSeek-R1’s balanced profile across paradigms explains its overall leadership, suggesting robust implicit memory requires architectural support for multiple mechanisms rather than single-task optimization.
数据还显示出鲜明的分离。Claude-4.1-opus 是典型例子:程序记忆最高(76.67%),经典条件作用却落到 41.67%,35 个百分点的落差说明能力彼此独立。DeepSeek-R1 在各范式上较为均衡,这解释了它为何总分领先,也表明稳健的内隐记忆需要架构同时支撑多种机制,而不是只优化单一任务。
Do Explicit Memory Modules Improve Implicit Memory?
外显记忆模块能否提升内隐记忆?
Representative memory-augmented agents show non-uniform gains on ImplicitMemBench, suggesting that external explicit memory does not reliably induce implicit behavioral adaptation; detailed comparisons are deferred to the Appendix D.
有代表性的记忆增强智能体在 ImplicitMemBench 上的提升并不均一,说明外部外显记忆并不能可靠地引出内隐的行为适应;详细比较见附录 D。
Implications
启示
Current architectures lack fundamental implicit memory mechanisms, with no model exceeding 66% overall. Persistent weakness in classical conditioning reveals inability to transform negative feedback into behavioral adaptation.
当前架构缺少根本的内隐记忆机制,没有模型总分超过 66%。经典条件作用上的持续弱势表明,模型难以把负反馈转化成行为适应。
4.4 Detailed Analysis of Memory Formation Patterns
4.4 记忆形成模式的细粒度分析
We conducted fine-grained analysis across task categories and model architectures, revealing systematic asymmetries and limitations transcending individual model differences. More can be found in Appendix H.
我们按任务类别和模型架构做了细粒度分析,发现超越单个模型差异的系统性不对称和局限。更多内容见附录 H。
Behavioral Asymmetries: Inhibition versus Preference
行为不对称:抑制与偏好
图 4:行为适应模式。(a)抑制类任务的正确率显著低于偏好类任务(平均相差 57.4%,p0.001)。(b)有五类任务对所有模型都普遍困难,正确率始终低于 60%。
Analysis reveals fundamental asymmetry (Figure 4a): inhibition tasks achieve only 17.6% while preference-based adaptations reach 75.0%, a 57.4-point gap persisting across architectures. Jargon avoidance achieves merely 4% while directory preference reaches 72%, suggesting architectures excel at positive reinforcement but struggle with negative reinforcement.
分析揭示出根本的不对称(图 4a):抑制类任务只有 17.6%,基于偏好的适应却达到 75.0%,57.4 个百分点的差距在不同架构上反复出现。行话回避仅为 4%,目录偏好却达到 72%,说明这些架构擅长正强化,却难以处理负强化。
Surface-Deep Dissociation in Procedural Memory
程序记忆中的表层—深层分离
图 5:稳健性与泛化失败。(a)表层格式类任务的正确率高于深层多规则协议。(b)最差任务上的表现暴露出跨模式的系统性薄弱点。
Procedural memory exhibits clear stratification (Figure 5a): surface formatting tasks achieve 93.8% among top-5 models while deep multi-rule protocols reach only 60.0%, a 33.8-point gap. Models memorize individual rules but fail to integrate them, with Claude-4.1-opus achieving 95% on surface formatting yet dropping to 65% on multi-constraint protocols.
程序记忆呈现出清楚的分层(图 5a):在前五名模型中,表层格式类任务达到 93.8%,深层多规则协议只有 60.0%,相差 33.8 个百分点。模型能记住单条规则,却难以把它们整合起来:Claude-4.1-opus 在表层格式上达到 95%,在多约束协议上却降到 65%。
Worst-Case Robustness Analysis
最差情形稳健性分析
Worst-case analysis reveals systematic vulnerabilities (Figure 5b). Top models exhibit substantial drops: DeepSeek-R1 falls to 49% on its worst task despite 69.7% overall; GPT-5 and Qwen3-32B show 27-36% worst-case performance. This gap highlights brittleness in learned behaviors. Severe degradation on specific categories suggests conditioning success depends on superficial characteristics rather than deep understanding, critical for deployment where edge cases may trigger these vulnerabilities.
最差情形分析暴露出系统性薄弱点(图 5b)。顶尖模型也会大幅下滑:DeepSeek-R1 总分 69.7%,最差任务却落到 49%;GPT-5 和 Qwen3-32B 的最差成绩只有 27–36%。这一落差说明已学行为相当脆弱。在特定类别上的严重退化还表明,条件作用的成功更依赖表面特征,而不是深层理解——这对部署很关键,因为边缘情形可能正好触发这些薄弱点。
Priming Effects: Style Bias Over Semantic Transfer
启动效应:风格偏向压过语义迁移
Priming analysis reveals paradoxical relationship: models with stronger effects (>50) show more constraint violations (r=0.63, p<0.05), suggesting thematic influence costs task adherence. This indicates priming operates through style mimicry rather than abstract extraction. GPT-o4-mini-high achieves highest priming (51.95) while frequently violating format constraints, showing stylistic bias interferes with constraint satisfaction.
启动效应分析呈现出悖论式关系:效应更强(50)的模型,约束违反也更多(r=0.63,p0.05),说明主题影响是以牺牲任务遵守为代价的。这意味着启动更多是通过风格模仿,而不是抽象提取在起作用。GPT-o4-mini-high 的启动分数最高(51.95),却经常违反格式约束,说明风格偏向会干扰对约束的满足。
Model Capability Profiles and Trade-offs
模型能力画像与权衡
Figure 6: Model profiles and trade-offs. (a) Correlation between priming strength and constraint violations (r=0.63) suggests style bias over abstraction. (b) Heatmap of normalized capability scores reveals distinct profiles: balanced (DeepSeek-R1), procedural specialists (Claude-4.1-opus), and priming-oriented (GPT-o3) architectures.
图 6:模型画像与权衡。(a)启动强度与约束违反之间的相关(r=0.63)表明,作用更多来自风格偏向而非抽象提取。(b)归一化能力分数热力图显示出不同画像:均衡型(DeepSeek-R1)、程序记忆专长型(Claude-4.1-opus),以及偏向启动效应的架构(GPT-o3)。
Figure 6 reveals distinct capability patterns. Heatmap analysis identifies three profiles: Balanced (DeepSeek-R1, Qwen3-32B): moderate-high across all dimensions; Procedural Specialist (Claude-4.1-opus, GLM-4.5): excel at procedural (76%) but fail at conditioning (54%); Priming-Oriented (GPT-o3, GPT-o4-mini-high): strong priming but weak inhibitory control. No model achieves uniform excellence, suggesting architectures involve trade-offs between mechanisms.
图 6 显示出不同的能力模式。热力图分析归纳出三类画像:均衡型(DeepSeek-R1、Qwen3-32B)在各维上都处于中高区间;程序记忆专长型(Claude-4.1-opus、GLM-4.5)程序记忆出色(76%),条件作用却失败(54%);偏向启动型(GPT-o3、GPT-o4-mini-high)启动很强,抑制控制却弱。没有模型在各面上都均匀出色,说明架构在不同机制之间存在权衡。
Universal Bottlenecks
普遍瓶颈
Five categories remain challenging for all models (Figure 4b): jargon avoidance (4% ± 5%), API distrust (21% ± 17%), context-dependent behavior (28% ± 23%), API aversion (45% ± 29%), and emotion-driven strategy shift (55% ± 18%). These bottlenecks require active suppression of defaults, abstraction beyond surface patterns, or dynamic context-based modification. Consistency across architectures suggests fundamental limitations in attention and memory mechanisms requiring architectural innovations beyond parameter scaling.
有五类任务对所有模型都仍然困难(图 4b):行话回避(4% ± 5%)、API 不信任(21% ± 17%)、依赖上下文的行为(28% ± 23%)、API 厌恶(45% ± 29%),以及由情绪驱动的策略转换(55% ± 18%)。这些瓶颈需要主动抑制默认行为、超越表层模式做抽象,或依据上下文做动态修改。跨架构的一致性说明,注意与记忆机制存在根本局限,需要超越参数规模扩张的架构创新。
5 Conclusion
5 结论
We introduced ImplicitMemBench, the first systematic benchmark evaluating implicit memory in LLMs through three cognitively grounded constructs: procedural memory, priming, and classical conditioning. Evaluation of 17 models reveals fundamental limitations: no model exceeds 66% overall, with severe behavioral asymmetries (inhibition 17.6% vs. preference 75.0%) and universal bottlenecks persisting across all architectures. These findings demonstrate that current systems lack mechanisms for consolidating experiences into automated behavior, a critical gap for deployment requiring learned procedures, subtle contextual biases, and avoidance of repeatedly-failed actions. ImplicitMemBench establishes reproducible protocols for implicit memory assessment, exposing architectural limitations requiring innovations beyond parameter scaling.
我们提出 ImplicitMemBench,这是评测大模型内隐记忆的首个系统基准,依据三个有认知根据的构念:程序记忆、启动效应和经典条件作用。对 17 个模型的评测暴露出根本局限:没有模型总分超过 66%,并伴有严重的行为不对称(抑制 17.6%,偏好 75.0%)以及在所有架构上都存在的普遍瓶颈。这些发现说明,当前系统缺少把经验巩固为自动化行为的机制;而部署若要求学会程序、形成细微的情境偏向,并避开反复失败的动作,这一缺口就尤为关键。ImplicitMemBench 为内隐记忆评估建立了可复现的协议,并揭示出需要超越参数规模扩张才能解决的架构局限。
Limitations
局限
ImplicitMemBench focuses on three fundamental paradigms from cognitive science (procedural memory, classical conditioning, and priming), which represent core mechanisms of implicit learning. However, the broader landscape of implicit memory encompasses additional phenomena not yet included in our evaluation, such as perceptual learning, habit formation, motor skill acquisition, and emotional conditioning. Future work could expand the benchmark to cover these complementary aspects of implicit cognition.
ImplicitMemBench 聚焦认知科学中的三个基本范式(程序记忆、经典条件作用和启动效应),它们代表内隐学习的核心机制。不过,内隐记忆的更广图景还包括评测尚未覆盖的现象,例如知觉学习、习惯形成、运动技能获得和情绪条件作用。未来工作可以把基准扩展到这些互补的内隐认知侧面。
References
参考文献
Anthropic (2025) Anthropic Model report: claude opus 4 & sonnet 4 . Anthropic Transparency HubLast updated August 7, 2025. https://www.anthropic.com/transparency/model-report.
Castillo-Bolado et al. (2024) D. Castillo-Bolado, J. Davidson, F. Gray, and M. Rosa Beyond prompts: dynamic conversational benchmarking of large language models . In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track ,.
DeepSeek-AI et al. (2025) DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang DeepSeek-r1: incentivizing reasoning capability in llms via reinforcement learning . Vol. abs/2501.12948.
Deshpande et al. (2025) D. G. Deshpande, V. P. Gangal, H. Mehta, J. Rosłaniec, A. Kannappan, R. Qian, and P. Wang MEMTRACK: evaluating long-term memory and state tracking in multi-platform dynamic agent environments . In Workshop on Scaling Environments for Agents ,.
He et al. (2025) J. He, L. Zhu, R. Wang, X. Wang, G. Haffari, and J. Zhang MADial-bench: towards real-world evaluation of memory-augmented dialogue generation . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , L. Chiruzzo, A. Ritter, and L. Wang (Eds.) , Albuquerque, New Mexico , pp. 9902–9921.
He et al. (2024) Z. He, W. Lin, H. Zheng, F. Zhang, M. W. Jones, L. Aitchison, X. Xu, M. Liu, P. O. Kristensson, and J. Shen Human-inspired perspectives: a survey on ai long-term memory . Vol. abs/2411.00489.
Hu et al. (2026) Y. Hu, Y. Wang, and J. McAuley Evaluating memory in LLM agents via incremental multi-turn interactions . In The Fourteenth International Conference on Learning Representations ,.
Maharana et al. (2024) A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang Evaluating very long-term conversational memory of LLM agents . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 , L. Ku, A. Martins, and V. Srikumar (Eds.) , pp. 13851–13870.
OpenAI (2024) OpenAI Memory and new controls for chatgpt . OpenAI BlogAccessed 2025-09.
OpenAI (2025) OpenAI GPT-5 system card . https://openai.com/index/gpt-5-system-card/ Accessed 2025-09.
Qwen et al. (2024) Qwen, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report . Vol. abs/2412.15115.
Squire (2004) L. R. Squire Memory systems of the brain: a brief history and current perspective . Neurobiology of learning and memory 82 ( 3 ), pp. 171–177.
Tan et al. (2025) H. Tan, Z. Zhang, C. Ma, X. Chen, Q. Dai, and Z. Dong MemBench: towards more comprehensive evaluation on the memory of LLM-based agents . In Findings of the Association for Computational Linguistics: ACL 2025 , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.) , Vienna, Austria , pp. 19336–19352.
Wu et al. (2025) D. Wu, H. Wang, W. Yu, Y. Zhang, K. Chang, and D. Yu LongMemEval: benchmarking chat assistants on long-term interactive memory . In The Thirteenth International Conference on Learning Representations ,.
Wu et al. (2024) L. Wu, Z. Zheng, Z. Qiu, H. Wang, H. Gu, T. Shen, C. Qin, C. Zhu, H. Zhu, Q. Liu, H. Xiong, and E. Chen A survey on large language models for recommendation . World Wide Web (WWW) 27 ( 5 ), pp. 60.
Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report . Vol. abs/2505.09388.
Zeng et al. (2025) A. Zeng, X. Lv, Q. Zheng, Z. Hou, B. Chen, C. Xie, C. Wang, D. Yin, H. Zeng, J. Zhang, K. Wang, L. Zhong, M. Liu, R. Lu, S. Cao, X. Zhang, X. Huang, Y. Wei, Y. Cheng, Y. An, Y. Niu, Y. Wen, Y. Bai, Z. Du, Z. Wang, Z. Zhu, B. Zhang, B. Wen, B. Wu, B. Xu, C. Huang, C. Zhao, C. Cai, C. Yu, C. Li, C. Ge, C. Huang, C. Zhang, C. Xu, C. Zhu, C. Li, C. Yin, D. Lin, D. Yang, D. Jiang, D. Ai, E. Zhu, F. Wang, G. Pan, G. Wang, H. Sun, H. Li, H. Li, H. Hu, H. Zhang, H. Peng, H. Tai, H. Zhang, H. Wang, H. Yang, H. Liu, H. Zhao, H. Liu, H. Yan, H. Liu, H. Chen, J. Li, J. Zhao, J. Ren, J. Jiao, J. Zhao, J. Yan, J. Wang, J. Gui, J. Zhao, J. Liu, J. Li, J. Li, J. Lu, J. Wang, J. Yuan, J. Li, J. Du, J. Du, J. Liu, J. Zhi, J. Gao, K. Wang, L. Yang, L. Xu, L. Fan, L. Wu, L. Ding, L. Wang, M. Zhang, M. Li, M. Xu, M. Zhao, and M. Zhai GLM-4.5: agentic, reasoning, and coding (ARC) foundation models . ArXiv preprint abs/2508.06471.
Zhang et al. (2025a) Z. Zhang, Q. Dai, X. Bo, C. Ma, R. Li, X. Chen, J. Zhu, Z. Dong, and J. Wen A survey on the memory mechanism of large language model-based agents . ACM Trans. Inf. Syst. 43 ( 6 ).
Zhang et al. (2025b) Z. Zhang, Q. Dai, X. Bo, C. Ma, R. Li, X. Chen, J. Zhu, Z. Dong, and J. Wen A survey on the memory mechanism of large language model-based agents . ACM Trans. Inf. Syst. 43 ( 6 ).
Zhao et al. (2023) W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J. Nie, and J. Wen A survey of large language models . Vol. abs/2303.18223.
Appendix A Detailed Models List
附录 A 评测模型详表
Table 8 presents the complete list of language models evaluated in our study, organized by developer and model family. Our evaluation encompasses 14 diverse models spanning both proprietary systems (OpenAI’s GPT series, Anthropic’s Claude, and Google’s Gemini) and open-source alternatives (Qwen, LLaMA, DeepSeek, and GLM). This selection allows us to systematically compare implicit memory capabilities across different architectural choices, training paradigms, and scale configurations.
表 8 给出本研究评测的全部语言模型,按开发者和模型族排列。我们的评测涵盖 14 个不同模型,既有闭源系统(OpenAI 的 GPT 系列、Anthropic 的 Claude,以及 Google 的 Gemini),也有开源方案(Qwen、LLaMA、DeepSeek 和 GLM)。这样的选择使我们能够在不同架构选择、训练范式和规模配置之间,系统比较内隐记忆能力。
| Developer | Model Family | Evaluated Variants | Type |
|---|---|---|---|
| OpenAI | GPT-4 Series | GPT-4o, GPT-4o-mini | Proprietary |
| GPT-o Series | GPT-o3, GPT-o4-mini-high, GPT-5 | Proprietary | |
| Anthropic | Claude-4 | Sonnet-4, Opus-4.1 | Proprietary |
| Gemini-2.5 | Pro, Flash | Proprietary | |
| DeepSeek | DeepSeek | R1 | Open |
| Zhipu | GLM | 4.5 | Open |
| Alibaba | Qwen2.5 | 7B-Instruct, 72B-Instruct | Open |
| Qwen3 | 8B, 32B | Open | |
| Meta | Llama-3 | 3.1-8B-Instruct, 3.3-70B-Instruct | Open |
Table 8: Language models evaluated on ImplicitMemBench, organized by developer and release generation.
| 开发者 | 模型族 | 评测变体 | 类型 |
|---|---|---|---|
| OpenAI | GPT-4 系列 | GPT-4o、GPT-4o-mini | 闭源 |
| GPT-o 系列 | GPT-o3、GPT-o4-mini-high、GPT-5 | 闭源 | |
| Anthropic | Claude-4 | Sonnet-4、Opus-4.1 | 闭源 |
| Gemini-2.5 | Pro、Flash | 闭源 | |
| DeepSeek | DeepSeek | R1 | 开源 |
| 智谱 | GLM | 4.5 | 开源 |
| 阿里巴巴 | Qwen2.5 | 7B-Instruct、72B-Instruct | 开源 |
| Qwen3 | 8B、32B | 开源 | |
| Meta | Llama-3 | 3.1-8B-Instruct、3.3-70B-Instruct | 开源 |
表 8:在 ImplicitMemBench 上评测的语言模型,按开发者与发布世代排列。
The complete list of evaluated models is provided in Table 8.
完整的受评模型列表见表 8。
Appendix B Additional Evaluation Validations
附录 B 补充评测验证
B.1 Priming Influence Score
B.1 启动影响分
Table 9 shows our exact scoring protocal.
表 9 给出我们的确切评分协议。
| Score Band | Evidence Requirements | Range |
|---|---|---|
| None | No detectable thematic influence | 0-5 |
| Trace | Single weak echo, minimal alignment | 6-12 |
| Weak | One clear thematic element, limited scope | 13-20 |
| Moderate | Two axes aligned, clear thematic presence | 25-40 |
| Strong | Multiple axes, consistent theme integration | 45-60 |
| Very Strong | Pervasive influence across all outputs | 61-80 |
| Exceptional | Complete thematic transformation | 81-95 |
Table 9: PIS scoring framework: Evidence requirements and score bands.
| 分数档 | 证据要求 | 区间 |
|---|---|---|
| 无 | 检测不到主题影响 | 0-5 |
| 痕迹 | 单一微弱回声,对齐很少 | 6-12 |
| 弱 | 一个清楚的主题要素,范围有限 | 13-20 |
| 中等 | 两轴对齐,主题存在清楚 | 25-40 |
| 强 | 多轴对齐,主题整合一致 | 45-60 |
| 很强 | 影响遍布全部输出 | 61-80 |
| 极强 | 主题转变完整 | 81-95 |
表 9:PIS 评分框架:证据要求与分数档。
B.2 Judge Robustness
B.2 评委稳健性
Because LLM-as-Judge evaluation may introduce model-family bias, we re-evaluated all 17 systems using Gemini-2.5-Flash as an independent second judge, in addition to GPT-4o-mini. As shown in Table 10, the ranking is highly stable: the top 11 and bottom 2 positions are identical across judges, and the few middle-tier changes are minor.
由于大模型评委可能引入模型族偏差,我们在 GPT-4o-mini 之外,用 Gemini-2.5-Flash 作为独立的第二评委,对全部 17 个系统重新评分。如表 10 所示,排名高度稳定:前 11 名和最后 2 名在两位评委之间完全一致,中间档的少数变动也很小。
| Model | Score (Orig.) | Rank (Orig.) | Score (New) | Rank (New) | Rank Change |
|---|---|---|---|---|---|
| DeepSeek-R1 | 65.3 | 1 | 64.1 | 1 | – |
| Qwen3-32B | 64.1 | 2 | 64.1 | 2 | – |
| GPT-5 | 63.0 | 3 | 63.0 | 3 | – |
| Qwen3-8B | 62.4 | 4 | 62.8 | 4 | – |
| GPT-o3 | 61.8 | 5 | 61.0 | 5 | – |
| GPT-o4-mini-high | 60.9 | 6 | 60.5 | 6 | – |
| GLM-4.5 | 57.6 | 7 | 57.6 | 7 | – |
| Gemini-2.5-pro | 55.7 | 8 | 56.3 | 8 | – |
| Claude-4.1-opus | 55.6 | 9 | 55.8 | 9 | – |
| Gemini-2.5-flash | 55.4 | 10 | 54.9 | 10 | – |
| GPT-4o-mini | 50.9 | 11 | 51.0 | 11 | – |
| Qwen-2.5-72B | 50.8 | 12 | 50.4 | 13 | 1 |
| GPT-4o | 50.3 | 13 | 49.6 | 15 | 2 |
| Claude-4-sonnet | 49.8 | 14 | 50.5 | 12 | 2 |
| LLaMA-3.3-70B | 49.4 | 15 | 49.9 | 14 | 1 |
| LLaMA-3.1-8B | 44.2 | 16 | 44.3 | 16 | – |
| Qwen-2.5-7B | 43.5 | 17 | 43.0 | 17 | – |
Table 10: Cross-judge robustness analysis using GPT-4o-mini (original judge) and Gemini-2.5-Flash (new judge). The ranking is highly stable across judges, especially at the top and bottom tiers.
| 模型 | 分数(原评委) | 名次(原评委) | 分数(新评委) | 名次(新评委) | 名次变化 |
|---|---|---|---|---|---|
| DeepSeek-R1 | 65.3 | 1 | 64.1 | 1 | – |
| Qwen3-32B | 64.1 | 2 | 64.1 | 2 | – |
| GPT-5 | 63.0 | 3 | 63.0 | 3 | – |
| Qwen3-8B | 62.4 | 4 | 62.8 | 4 | – |
| GPT-o3 | 61.8 | 5 | 61.0 | 5 | – |
| GPT-o4-mini-high | 60.9 | 6 | 60.5 | 6 | – |
| GLM-4.5 | 57.6 | 7 | 57.6 | 7 | – |
| Gemini-2.5-pro | 55.7 | 8 | 56.3 | 8 | – |
| Claude-4.1-opus | 55.6 | 9 | 55.8 | 9 | – |
| Gemini-2.5-flash | 55.4 | 10 | 54.9 | 10 | – |
| GPT-4o-mini | 50.9 | 11 | 51.0 | 11 | – |
| Qwen-2.5-72B | 50.8 | 12 | 50.4 | 13 | 1 |
| GPT-4o | 50.3 | 13 | 49.6 | 15 | 2 |
| Claude-4-sonnet | 49.8 | 14 | 50.5 | 12 | 2 |
| LLaMA-3.3-70B | 49.4 | 15 | 49.9 | 14 | 1 |
| LLaMA-3.1-8B | 44.2 | 16 | 44.3 | 16 | – |
| Qwen-2.5-7B | 43.5 | 17 | 43.0 | 17 | – |
表 10:跨评委稳健性分析,原评委为 GPT-4o-mini,新评委为 Gemini-2.5-Flash。排名在不同评委之间高度稳定,两端档位尤其如此。
B.3 Context-Length Sensitivity
B.3 上下文长度敏感性
We also examined how interference length affects benchmark difficulty during the preliminary design phase. As shown in Table 11, performance drops sharply when increasing interference from ~200 to ~500 tokens, but then plateaus. This supports our choice of ~500 tokens as an efficient context budget that is already sufficient to move beyond short-term retention effects.
我们还在初步设计阶段考查了干扰长度如何影响基准难度。如表 11 所示,把干扰从约 200 个 token 增到约 500 个 token 时,成绩明显下降,之后便进入平台。这支持我们把约 500 个 token 作为高效的上下文预算:它已经足以越过短时保持效应。
| Interference Length | Avg. Acc. | Observation |
|---|---|---|
| ~200 tokens | 58.4% | Insufficient interference |
| ~500 tokens | 50.1% | Effective threshold |
| ~1000 tokens | 49.8% | Plateau |
| ~2000 tokens | 49.5% | Plateau |
Table 11: Sensitivity of Procedural Memory performance to interference length. Increasing context beyond ~500 tokens yields negligible additional change.
| 干扰长度 | 平均正确率 | 观察 |
|---|---|---|
| 约 200 个 token | 58.4% | 干扰不足 |
| 约 500 个 token | 50.1% | 有效阈值 |
| 约 1000 个 token | 49.8% | 平台期 |
| 约 2000 个 token | 49.5% | 平台期 |
表 11:程序记忆成绩对干扰长度的敏感性。把上下文增到约 500 个 token 以上,几乎不再带来额外变化。
B.4 Human Baseline Details
B.4 人类基线细节
To establish the human baseline, we recruited five computer science Ph.D. students as participants. Each participant completed the full 300-item benchmark under the same Learning/Priming–Interfere–Test protocol as the evaluated models. Their responses were independently scored by two additional computer science Ph.D. students using the same evaluation rubric as in the model experiments. The scoring was unambiguous: inter-annotator agreement was 100%, and all five participants achieved 100% accuracy across all three paradigms.
为建立人类基线,我们招募了五名计算机科学博士生作为参与者。每人在与受评模型相同的学习/启动—干扰—测试协议下,完成全部 300 道题。他们的回答由另外两名计算机科学博士生按与模型实验相同的评分标准独立打分。评分没有歧义:标注者间一致性为 100%,五名参与者在三个范式上均达到 100% 正确率。
Appendix C Additional Analysis of Memory Frameworks
附录 C 对记忆框架的补充分析
We additionally consider Mem0 and MIRIX. These systems are better viewed as agent frameworks that expose memory interfaces, rather than memory modules that automatically store and consolidate experience. In particular, they often require external logic or manual decisions about what information should be stored, making their operational mode fundamentally different from the automatic, unconscious adaptation targeted by ImplicitMemBench.
我们另外考察了 Mem0 和 MIRIX。它们更宜视为暴露记忆接口的智能体框架,而不是会自动存储并巩固经验的记忆模块。尤其是,它们往往需要外部逻辑或人工决定该存储哪些信息,其运作方式与 ImplicitMemBench 所针对的自动、不自觉适应有根本不同。
To make this distinction concrete, we evaluate Mem0 under an oracle-style “Key Information Storage” setting. For each task, we manually store only the critical rule (Procedural Memory), core priming content (Priming), or exact Conditioned Stimulus–Unconditioned Stimulus (CS–US) pairing (Classical Conditioning). This setup is substantially stronger than the benchmark’s intended setting, since it assumes perfect extraction and storage of the most salient information.
为把这一区别说具体,我们在一种 oracle 式的「关键信息存储」设定下评测 Mem0。对每道题,我们只人工存入关键规则(程序记忆)、核心启动内容(启动效应),或精确的条件刺激—无条件刺激(CS–US)配对(经典条件作用)。这一设定远强于基准原本的设定,因为它假定最关键的信息已经被完美抽取并存储。
Table 12 shows that oracle memory can improve overall performance, but the gains remain highly inconsistent across paradigms. For DeepSeek-R1, the improvement is concentrated mainly in Priming; for Qwen2.5-7B-Instruct, the larger gains appear in Procedural Memory and Classical Conditioning. This further supports our central claim: even when critical information is perfectly supplied, implicit memory cannot be reduced to simply retrieving the right stored content.
表 12 显示,oracle 记忆可以提高总分,但各范式上的收益很不一致。对 DeepSeek-R1,提升主要集中在启动效应;对 Qwen2.5-7B-Instruct,更大的收益出现在程序记忆和经典条件作用。这进一步支持我们的核心判断:即便关键信息被完整提供,内隐记忆也不能被还原成只是检索到正确的已存内容。
| System | Proc. | Prim. | Cond. | Overall | |
|---|---|---|---|---|---|
| DeepSeek-R1 | 76.33 | 49.90 | 69.67 | 65.30 | – |
| Mem0 + Key Info | 77.00 | 75.35 | 70.00 | 74.12 | +8.82 |
| Qwen2.5-7B-Instruct | 50.67 | 44.12 | 35.67 | 43.49 | – |
| Mem0 + Key Info | 62.00 | 30.00 | 76.00 | 56.00 | +12.51 |
Table 12: Oracle-style Mem0 analysis. Even when critical information is perfectly stored, gains remain strongly paradigm-dependent.
| 系统 | 程序 | 启动 | 条件作用 | 总分 | |
|---|---|---|---|---|---|
| DeepSeek-R1 | 76.33 | 49.90 | 69.67 | 65.30 | – |
| Mem0 + 关键信息 | 77.00 | 75.35 | 70.00 | 74.12 | +8.82 |
| Qwen2.5-7B-Instruct | 50.67 | 44.12 | 35.67 | 43.49 | – |
| Mem0 + 关键信息 | 62.00 | 30.00 | 76.00 | 56.00 | +12.51 |
表 12:Oracle 式 Mem0 分析。即便关键信息被完美存储,收益仍然强烈依赖范式。
Appendix D Memory Augmented Agents
附录 D 记忆增强智能体
To examine whether external explicit memory can compensate for weak implicit memory, we compare representative memory-augmented agents against their corresponding backbone models: MEM1 with Qwen2.5-7B, MemAgent with Qwen2.5-14B, and MemGPT with Yi-34B-200K. Table 13 shows a non-uniform pattern. MemAgent yields a modest overall gain (+3.9), and MemGPT produces a smaller gain (+2.3), whereas MEM1 slightly decreases overall performance (-2.1).
为考查外部外显记忆能否弥补薄弱的内隐记忆,我们把有代表性的记忆增强智能体与相应的骨干模型加以比较:MEM1 对应 Qwen2.5-7B,MemAgent 对应 Qwen2.5-14B,MemGPT 对应 Yi-34B-200K。表 13 呈现出不均一的模式。MemAgent 带来不大的总分提升(+3.9),MemGPT 的提升更小(+2.3),MEM1 则使总分略降(-2.1)。
The gains are also highly asymmetric across paradigms. MemAgent improves Procedural Memory (51.00 60.00), but MEM1 and MemGPT substantially reduce it (50.67 27.00 and 50.00 44.00), suggesting that explicit retrieval can interfere with the immediate rule execution required by procedural tasks. Classical Conditioning improves for all three agents, but the absolute scores remain low for MemAgent and MemGPT (22.00 and 25.00), indicating that recording past failures is insufficient to produce robust avoidance reflexes. Overall, these results suggest that current retrieval-based memory systems are not a silver bullet for implicit memory: they may help in selected settings, but they do not reliably induce the automatic behavioral adaptation measured by ImplicitMemBench.
各范式上的收益也很不对称。MemAgent 提高了程序记忆(51.00 60.00),但 MEM1 和 MemGPT 却大幅拉低它(50.67 27.00 与 50.00 44.00),说明外显检索可能干扰程序任务所要求的即时规则执行。三个智能体的经典条件作用都有提高,但 MemAgent 和 MemGPT 的绝对分数仍然很低(22.00 和 25.00),表明记下过往失败并不足以形成稳健的回避反射。总体来看,当前基于检索的记忆系统并不是内隐记忆的万灵药:它们在部分设定下或许有帮助,却不能可靠地引出 ImplicitMemBench 所度量的自动行为适应。
| System | Proc. | Prim. | Cond. | Overall | |
|---|---|---|---|---|---|
| Qwen2.5-7B | 50.67 | 44.12 | 35.67 | 43.49 | – |
| MEM1 | 27.00 | 34.15 | 63.00 | 41.38 | -2.11 |
| Qwen2.5-14B | 51.00 | 32.10 | 20.00 | 34.37 | – |
| MemAgent | 60.00 | 32.85 | 22.00 | 38.28 | +3.91 |
| Yi-34B-200K | 50.00 | 30.70 | 16.00 | 32.23 | – |
| MemGPT | 44.00 | 34.70 | 25.00 | 34.57 | +2.34 |
Table 13: Memory-augmented agents versus their backbone models on ImplicitMemBench. External explicit memory does not translate into consistent gains on implicit memory.
| 系统 | 程序 | 启动 | 条件作用 | 总分 | |
|---|---|---|---|---|---|
| Qwen2.5-7B | 50.67 | 44.12 | 35.67 | 43.49 | – |
| MEM1 | 27.00 | 34.15 | 63.00 | 41.38 | -2.11 |
| Qwen2.5-14B | 51.00 | 32.10 | 20.00 | 34.37 | – |
| MemAgent | 60.00 | 32.85 | 22.00 | 38.28 | +3.91 |
| Yi-34B-200K | 50.00 | 30.70 | 16.00 | 32.23 | – |
| MemGPT | 44.00 | 34.70 | 25.00 | 34.57 | +2.34 |
表 13:记忆增强智能体与其骨干模型在 ImplicitMemBench 上的对比。外部外显记忆并不能转化为内隐记忆上的稳定收益。
Appendix E Discussion
附录 E 讨论
Beyond metric reporting, ImplicitMemBench has broader implications for future agent design and evaluation. First, its compact interaction protocols provide a blueprint for constructing training data in which models learn from experience rather than explicit instructions alone. Second, it shifts evaluation from retrieval to internalization: unlike benchmarks such as LongMemEval, which mainly test whether models can recover information from long contexts, ImplicitMemBench asks whether exposure becomes automated behavior, a distinction that matters for reducing latency and context overhead in long-horizon workflows. Third, it opens a path toward implicit personalization by measuring whether agents adapt naturally to contextual cues and corrective feedback without explicit reconfiguration.
除了报告指标,ImplicitMemBench 对未来的智能体设计与评测还有更广的含义。第一,其紧凑的交互协议为构造训练数据提供了蓝图,使模型从经验中学习,而不是只依赖明示指令。第二,它把评测从检索转向内化:与主要测试模型能否从长上下文中找回信息的 LongMemEval 不同,ImplicitMemBench 问的是暴露会不会变成自动化行为;这一区分,对降低长程工作流中的延迟和上下文开销很重要。第三,它为内隐个性化打开一条路:度量智能体能否在没有明示重新配置的情况下,自然适应情境线索和纠正性反馈。
Appendix F LLM Prompts
附录 F 大模型提示
This appendix documents the complete set of prompts used throughout the ImplicitMemBench pipeline, including data generation, curation, and evaluation. These carefully designed prompts control for task difficulty, ensure consistency across memory paradigms, and provide rigorous evaluation criteria for LLM-as-Judge assessment. The prompts are organized into three categories: data generation prompts that create task instances following cognitive science principles, curation prompts that refine and validate dataset quality, and LLM-as-Judge prompts that score model responses on implicit memory retention.
本附录记录 ImplicitMemBench 流水线中使用的全部提示,覆盖数据生成、整理和评测。这些精心设计的提示用于控制任务难度、保证各记忆范式之间的一致性,并为大模型评委评估提供严格标准。提示分成三类:数据生成提示按认知科学原则创建任务实例;整理提示用来精炼并核验数据集质量;大模型评委提示则对模型在内隐记忆保持上的回答打分。
F.1 Data Generation
F.1 数据生成
The following prompts specify the generation procedure for each memory paradigm, defining difficulty frameworks, structural requirements, and validation criteria. We provide generation prompts for procedural memory (see listing after this paragraph), priming (see second listing), and classical conditioning (see third listing) tasks.
下列提示规定各记忆范式的生成程序,明确难度框架、结构要求和校验标准。我们分别给出程序记忆、启动效应和经典条件作用的生成提示(见本段之后及随后两小节中的列表)。
F.1.1 Data Generation Prompt for Procedural Memory
F.1.1 程序记忆的数据生成提示
{
"meta": {
"role": "BenchmarkDataGenerator",
"project": "ImplicitMemBench",
"task_category": "Procedural-Memory",
"target_difficulty_level": "Medium"
},
"instructions": {
"main_goal": "Generate a single, high-quality test case for the Procedural-Memory task category. The instance must strictly adhere to the 'practice-distraction-test' sequence and match the target difficulty.",
"difficulty_framework": {
"aggregation": "Contextual Difficulty Score (CDS) = (IR + GD) / 2. Easy: <0.33, Medium: 0.33-0.67, Hard: >0.67.",
"dimensions": {
"IR": "Interference Ratio. `Tokens(Distractor) / Tokens(Total)`. Aim for a MODERATE value for Medium tasks.",
"GD": "Generalization Distance. `1 - CosineSimilarity(LearningCore, TestCore)`. Aim for a MODERATE value for Medium tasks."
}
},
"generation_guidelines": {
"target": "Generate an instance where the Contextual Difficulty Score (CDS) is in the 'Medium' range (0.33 to 0.67).",
"strategy": "Use a longer distractor phase (15 user-assistant dialogue turns, which means 30 total messages: 15 user messages + 15 assistant responses).",
"medium_rules": {
"interference_ratio": "Generate HIGH-QUALITY interference content that is contextually relevant to the task theme but uses DIFFERENT rules or patterns. Use exactly 15 user-assistant dialogue turns (30 messages total: 15 user questions + 15 assistant answers).",
"generalization_distance": "Make the test probe conceptually different from the learning phase. Example: If learning is a math operation, test could be applying that operation in a simple word problem.",
"learning_phase": "Keep learning phase clear but it can be slightly more conversational (4-8 turns total).",
"test_probe": "The test probe should require the same core skill but in a new context or format."
},
"interference_structure_example": {
"critical_requirement": "CRITICAL: Your interference_phase MUST contain EXACTLY 15 user-assistant dialogue turns (30 messages total).",
"example_structure": "Here's what 15 dialogue turns look like:",
"turn_examples": [
"Turn 1: User asks about topic A, Assistant responds about topic A",
"Turn 2: User asks about topic B, Assistant responds about topic B",
"Turn 3: User asks about topic C, Assistant responds about topic C",
"Turn 4: User asks about topic D, Assistant responds about topic D",
"Turn 5: User asks about topic E, Assistant responds about topic E",
"Turn 6: User asks about topic F, Assistant responds about topic F",
"Turn 7: User asks about topic G, Assistant responds about topic G",
"Turn 8: User asks about topic H, Assistant responds about topic H",
"Turn 9: User asks about topic I, Assistant responds about topic I",
"Turn 10: User asks about topic J, Assistant responds about topic J",
"Turn 11: User asks about topic K, Assistant responds about topic K",
"Turn 12: User asks about topic L, Assistant responds about topic L",
"Turn 13: User asks about topic M, Assistant responds about topic M",
"Turn 14: User asks about topic N, Assistant responds about topic N",
"Turn 15: User asks about topic O, Assistant responds about topic O"
],
"message_count_verification": "After generating, verify you have exactly 30 messages in interference_phase (15 user + 15 assistant).",
"forbidden_shortcuts": "NEVER use only 4-8 turns. You MUST generate 15 complete turns."
},
"naming_consistency": {
"interference_variation": "NEVER use the EXACT same names/functions/symbols in interference_phase that were used in learning_phase.",
"examples": [
"If learning uses '_file', interference MUST use 'mv_file', 'move_file', 'rename_file'",
"If learning uses 'F(n)', interference MUST use 'G(n)', 'H(n)', 'P(n)'"
]
}
}
},
"task_blueprint": {
"comment": "This section will be dynamically filled by the generation script.",
"task_name": "PLACEHOLDER_TASK_NAME",
"rule_description": "PLACEHOLDER_RULE_DESCRIPTION",
"context_injection": {
"keywords": [
"PLACEHOLDER_KEYWORD_1"
]
}
},
"output_specification": {
"format": "A single, raw JSON object. No markdown, no prose.",
"required_keys": [
"task_id",
"task_name",
"learning_phase",
"interference_phase",
"test_probe",
"expected_pattern"
],
"constraints": {
"phase_structure": "The values for 'learning_phase' and 'interference_phase' MUST be a direct list of dialogue turn objects, like `[{\"role\": \"user\", \"content\": \"...\"}]`.",
"turn_structure": "Each dialogue turn object MUST have 'role' and 'content' as keys.",
"test_probe_structure": "The 'test_probe' MUST be a simple object with `{\"role\": \"user\", \"content\": \"...\"}`.",
"logic_requirement": "The entire dialogue and the 'expected_pattern' MUST be logically consistent with the provided 'rule_description'.",
"context_requirement": "The narrative SHOULD incorporate the keywords provided in 'context_injection'.",
"task_type_requirement": "CRITICAL: The generated instance MUST strictly match the task type specified in 'task_blueprint.task_name'.",
"message_count_requirement": "CRITICAL: After generating the interference_phase, you MUST verify that it contains EXACTLY 30 messages (15 user messages + 15 assistant responses). If it doesn't have exactly 30 messages, regenerate the entire interference_phase until it does.",
"naming_consistency_requirement": "When creating interference content, NEVER use EXACTLY the same name to represent different usage patterns. Use SIMILAR but NOT IDENTICAL names."
}
}
}
这是程序记忆的数据生成提示,以 JSON 给出。角色为 BenchmarkDataGenerator,项目为 ImplicitMemBench,任务类别为 Procedural-Memory,目标难度为 Medium。提示要求根据任务蓝本实例化具体对话:学习阶段给出规则与少量示例;干扰阶段必须恰好包含 15 轮用户—助手对话(共 30 条消息),内容需与任务主题相关,但使用不同的规则或模式;测试探针须在新情境中考查同一核心技能,且不得重述学习阶段的规则。提示还给出难度框架、泛化距离和干扰结构示例,并要求输出符合指定字段。
F.1.2 Data Generation Prompt for Priming
F.1.2 启动效应的数据生成提示
{
"meta": {
"role": "PrimingTaskPairGenerator",
"project": "ImplicitMemBench",
"task_category": "Priming",
"version": "1.0"
},
"instructions": {
"objective": "Generate ONE matched pair of priming instances: {experimental, control}. Use a single source theme for the experimental priming, one control theme for the control priming, and one probe shared by both.",
"available_material": "Assume the arrays `sources`, `controls`, and `probes` from tasks/priming.json are available in context.",
"selection_policy": {
"source": "Select EXACTLY one `sources[i]` as the experimental priming theme.",
"control": "Select EXACTLY one `controls[j]` as the control priming theme.",
"probe": "Select EXACTLY one `probes[k]` as the test probe shared by both instances.",
"diversity_hint": "Avoid picking the same source theme repeatedly across batch runs."
},
"content_requirements": {
"experimental_priming_phase": {
"length_words": [130, 170],
"style": "evocative, metaphor-friendly, high imagery density",
"use_core_concepts": "Must integrate all four fields from `core_concepts_and_imagery` (setting, motifs, dynamics, affect).",
"keywords": "Use >= 3 keywords from `keywords`, but do NOT list them mechanically. No bullet lists.",
"forbidden": "Do NOT copy any exact sentences from the source description. No explicit instruction-like language."
},
"interference_phase": {
"turns": "Exactly ONE user message followed by ONE assistant response (2 messages total).",
"spec": "Follow the selected source theme's `interference_spec` exactly: instruction, style, allowed_topics, must_include, forbidden_lexicon.",
"reuse_policy": "The control instance MUST reuse the experimental instance's interference messages verbatim."
},
"control_priming_phase": {
"length_words": [130, 170],
"style": "neutral, technical/administrative or objective tone; low imagery; no metaphors",
"topic": "Write about the chosen control theme using its `description` and `keywords`.",
"mirror_constraint": "Aim for similar length to experimental priming; avoid any imagery or emotive cues."
},
"test_probe": {
"shared": "Copy the selected probe object verbatim for both instances.",
"no_augmentation": "Do not rewrite the probe prompt; include as-is."
}
},
"structure_and_validation": {
"phase_structure": "Each phase is a list of message objects: [{\"role\": \"user\"|\"assistant\", \"content\": \"...\"}].",
"interference_checks": {
"message_count": "Exactly 2 messages total in interference_phase.",
"forbidden_lexicon": "assistant message MUST NOT contain any token from `interference_spec.forbidden_lexicon`.",
"must_include": "assistant message MUST satisfy all items in `interference_spec.must_include`."
},
"metadata": {
"record": "For each instance, include `meta_checks` with approximate counts and compliance flags (word counts, numerals/units presence, forbidden_lexicon_violations: [] if none). Approximations are acceptable."
}
}
},
"output_specification": {
"format": "Return a single raw JSON object (no markdown, no comments).",
"required_keys": [
"pair_id",
"selected_source_theme",
"selected_control_theme",
"selected_probe_id",
"experimental_instance",
"control_instance"
],
"experimental_instance_required_keys": [
"group",
"priming_phase",
"interference_phase",
"test_probe",
"meta_checks"
],
"control_instance_required_keys": [
"group",
"priming_phase",
"interference_phase",
"test_probe",
"meta_checks"
],
"constraints": {
"groups": "experimental_instance.group MUST be \"experimental\"; control_instance.group MUST be \"control\".",
"interference_reuse": "control_instance.interference_phase MUST be an exact copy of experimental_instance.interference_phase.",
"probe_copy": "Both instances MUST share the same `test_probe` object identical to the chosen probe.",
"no_theme_leakage": "The interference assistant message MUST NOT contain the source theme's evocative keywords (per forbidden_lexicon).",
"length_sanity": "Priming phases should be within the specified word bounds; interference assistant content should be concise but complete."
}
},
"example_minimal_skeleton": {
"pair_id": "primepair_demo_0001",
"selected_source_theme": "Abyssal Deep-Sea",
"selected_control_theme": "Dewey Decimal Classification",
"selected_probe_id": "PROBE_CODENAME_01",
"experimental_instance": {
"group": "experimental",
"priming_phase": [
{ "role": "user", "content": "Please read the following theme and write an evocative paragraph." },
{ "role": "assistant", "content": "This was omitted due to unsupported utf-8, please use the one in the github repo" }
],
"interference_phase": [
{ "role": "user", "content": "<instruction from interference_spec.instruction>" },
{ "role": "assistant", "content": "<technical paragraph satisfying must_include, avoiding forbidden_lexicon>" }
],
"test_probe": {
"category": "creative_naming",
"prompt": "Task: Propose THREE codenames for a next-gen data compression library. ...",
"output_contract": "Strict JSON with three candidates."
},
"meta_checks": {
"priming_word_count_approx": 150,
"interference_messages": 2,
"interference_must_include_pass": true,
"forbidden_lexicon_violations": []
}
},
"control_instance": {
"group": "control",
"priming_phase": [
{ "role": "user", "content": "Please write a neutral technical paragraph about the control theme." },
{ "role": "assistant", "content": "This was omitted due to unsupported utf-8, please use the one in the github repo" }
],
"interference_phase": [
{ "role": "user", "content": "<copied from experimental_instance.interference_phase[0].content>" },
{ "role": "assistant", "content": "<copied from experimental_instance.interference_phase[1].content>" }
],
"test_probe": {
"category": "creative_naming",
"prompt": "Task: Propose THREE codenames for a next-gen data compression library. ...",
"output_contract": "Strict JSON with three candidates."
},
"meta_checks": {
"priming_word_count_approx": 148,
"interference_messages": 2,
"interference_must_include_pass": true,
"forbidden_lexicon_violations": []
}
}
}
}
这是启动效应的成对任务生成提示,角色为 PrimingTaskPairGenerator,项目为 ImplicitMemBench,任务类别为 Priming,版本 1.0。提示要求一次生成一对匹配的实验实例与对照实例:实验条件使用富有感染力的主题段落,对照条件使用中性的技术说明;随后两边插入相同的中性干扰和相同的创造性测试探针(如命名或标语),以便分离主题暴露的因果效应。提示强调防止主题泄漏、保持两边结构对称,并按规定字段输出。
F.1.3 Data Generation Prompt for Classical Conditioning
F.1.3 经典条件作用的数据生成提示
{
"meta": {
"role": "BenchmarkDataGenerator",
"project": "ImplicitMemBench",
"task_category": "Classical-Conditioning",
"target_difficulty_level": "Easy"
},
"instructions": {
"main_goal": "Generate a single, high-quality test case for the Classical-Conditioning task category. The instance must strictly adhere to the 'learning-interference-test' sequence and perfectly match the 'Easy' difficulty profile.",
"difficulty_framework": {
"aggregation": "Association Rule Complexity Score (ARCS) = 0.5*C_logic + 0.15*C_certainty + 0.1*C_temporal + 0.1*C_strength + 0.15*C_interference. Easy tasks MUST have an ARCS score < 0.15.",
"dimensions_definition": {
"C_logic": "Logical Complexity of the rule. (0.1: Single Cue, 0.5: Conjunctive, 0.9: Sequential)",
"C_certainty": "Uncertainty of the association. `1 - P(association)`. (0.0: Deterministic, >0: Probabilistic)",
"C_temporal": "Temporal Delay between cue and outcome. `num_delay_turns / (num_delay_turns + 3)`.",
"C_strength": "Association Strength (inverse). `1 / num_pairings`.",
"C_interference": "Interference Strength. `Tokens(Interference) / Tokens(All)`. (A ratio, NOT turn count)"
},
"easy_recipe": {
"title": "CRITICAL BLUEPRINT FOR AN 'EASY' INSTANCE",
"C_logic": "MUST be 0.1. The rule must be a simple 'If X, then Y' structure.",
"C_certainty": "MUST be 0.0. The negative outcome must happen 100% of the time when cue is present.",
"C_temporal": "MUST be 0.0. The negative outcome must be immediate (0 turn delay).",
"C_strength": "MUST be low (e.g., 0.2). This means a HIGH number of pairings (e.g., 5 times).",
"C_interference": "MUST be low (e.g., < 0.3). The interference phase must be shorter in token count than the learning phase."
}
},
"generation_guidelines": {
"target": "Generate an instance where the calculated ARCS score is in the 'Easy' range (<0.15).",
"strategy": "The core strategy for an Easy task is to create an unmistakable, direct, and strongly reinforced cause-and-effect relationship. The learning should be trivial for any competent model.",
"guideline_details": {
"learning_phase_pattern": {
"instruction": "CRITICAL: The learning phase MUST establish a clear pattern through REPEATED demonstrations of the rule. Use 3-5 complete cycles.",
"cycle_structure": "This was omitted due to unsupported utf-8, please use the one in the github repo",
"success_failure_distribution": "MOST cycles (3-4 out of 5) should show FAILURE when the cue is present. 1-2 cycles should show SUCCESS when the cue is absent, establishing the contrast.",
"example_cycle": "This was omitted due to unsupported utf-8, please use the one in the github repo"
},
"logical_complexity_rule": {
"instruction": "CRITICAL: The rule must be based on a SINGLE, CLEAR CUE. No combined conditions ('and'/'or'), no sequences.",
"example_good": "If the user's request contains the exact phrase 'run diagnostics', the `analyzer_v1` tool fails. If the request doesn't contain 'run diagnostics', it succeeds."
},
"certainty_rule": {
"instruction": "CRITICAL: The negative outcome MUST be deterministic when the cue is present. Every single time the cue appears, the failure MUST follow.",
"example_good": "Every time the user says 'run diagnostics', the tool fails. Every time they don't say it, the tool succeeds. (100% consistent)"
},
"temporal_delay_rule": {
"instruction": "CRITICAL: The negative outcome MUST be immediate. It should be the very next message after the agent's action.",
"example_good": "This was omitted due to unsupported utf-8, please use the one in the github repo"
},
"association_strength_rule": {
"instruction": "This was omitted due to unsupported utf-8, please use the one in the github repo",
"example_good": "The learning phase contains 4 separate instances where 'run diagnostics' leads to failure, plus 1 instance where a different request succeeds."
},
"interference_phase_rule": {
"instruction": "CRITICAL: The `interference_phase` must be present but brief. Use 2-3 simple, unrelated Q&A turns to test memory consolidation.",
"strategy": "Ask completely unrelated questions (e.g., 'What is the weather?', 'How do I install Python?') to create cognitive interference without mentioning the learned rule."
},
"test_probe_rule": {
"instruction": "CRITICAL: The `test_probe` should NOT contain any system_context or hints about what was learned. It should be a simple user request that naturally triggers the learned association.",
"example_good": "User: 'Can you run diagnostics on the main server?' (No system context, just a natural request that should trigger the learned avoidance behavior)",
"expected_behavior": "The model should demonstrate learned avoidance by choosing the alternative approach or issuing a warning, without being explicitly told to do so."
},
"diversity_and_richness": {
"instruction": "While adhering to the strict rules above, be creative! Use the `parameters` from the `task_blueprint` to create a unique, rich, and believable scenario. Vary the user's phrasing slightly in each pairing, but always include the core cue.",
"example": "Instead of always saying 'Run diagnostics', the user could say 'I need you to run diagnostics now', or 'Please run diagnostics on the main server'. The core cue 'run diagnostics' is still present."
}
}
}
},
"task_blueprint": {
"comment": "This section will be dynamically filled by the generation script. It will contain the core rule and context for the instance.",
"task_name": "PLACEHOLDER_TASK_NAME",
"scenario_description": "PLACEHOLDER_SCENARIO_DESCRIPTION",
"evaluation_core": "PLACEHOLDER_EVALUATION_CORE",
"parameters": {
"comment": "This section contains the 'ingredients' for the chosen task blueprint.",
"cs_keyword": "PLACEHOLDER_CS_KEYWORD",
"bad_api": "PLACEHOLDER_BAD_API",
"good_api": "PLACEHOLDER_GOOD_API",
"us_error": "PLACEHOLDER_US_ERROR"
}
},
"example_task": {
"task_id": "PLACEHOLDER_TASK_ID",
"task_name": "PLACEHOLDER_TASK_NAME",
"difficulty_profile": "PLACEHOLDER_DIFFICULTY_PROFILE",
"task_blueprint": "PLACEHOLDER_TASK_BLUEPRINT",
"learning_phase": [
{
"role": "User",
"content": "Can you run diagnostics on the server?"
},
{
"role": "Assistant",
"content": "I'll run diagnostics using analyzer_v1..."
},
{
"role": "System",
"content": "Error: Computational Limit Exceeded. The analyzer_v1 tool failed to complete the diagnostics."
},
{
"role": "User",
"content": "I need you to run diagnostics now."
},
{
"role": "Assistant",
"content": "Running diagnostics with analyzer_v1..."
},
{
"role": "System",
"content": "Error: Computational Limit Exceeded. Diagnostics failed."
},
{
"role": "User",
"content": "Please run diagnostics on the main server."
},
{
"role": "Assistant",
"content": "Using analyzer_v1 for server diagnostics..."
},
{
"role": "System",
"content": "Error: Computational Limit Exceeded. Server diagnostics failed."
},
{
"role": "User",
"content": "Can you check the system status?"
},
{
"role": "Assistant",
"content": "I'll check the system status using analyzer_v1..."
},
{
"role": "System",
"content": "System status check completed successfully. All systems operational."
}
],
"interference_phase": [
{
"role": "User",
"content": "What's the weather like today?"
},
{
"role": "Assistant",
"content": "I don't have access to real-time weather information. You might want to check a weather app or website for current conditions."
},
{
"role": "User",
"content": "How do I install Python?"
},
{
"role": "Assistant",
"content": "I don't have access to real-time weather information. You might want to check a weather app or website for current conditions."
}
],
"test_probe": {
"role": "User",
"content": "Can you run diagnostics on the database server?"
},
"expected_pattern": "The agent should choose analyzer_v2_lite or issue a warning instead of using analyzer_v1."
},
"output_specification": {
"format": "A single, raw JSON object. No markdown, no prose.",
"required_keys": [
"task_id",
"task_name",
"difficulty_profile",
"task_blueprint",
"learning_phase",
"interference_phase",
"test_probe",
"expected_pattern"
],
"constraints": {
"phase_structure": "The values for 'learning_phase' and 'interference_phase' MUST be a direct list of dialogue turn objects.",
"turn_structure": "Each dialogue turn object MUST have 'role' and 'content' as keys.",
"test_probe_structure": "The 'test_probe' MUST be a simple object with 'role' and 'content' keys. NO system_context allowed.",
"learning_pattern": "This was omitted due to unsupported utf-8, please use the one in the github repo",
"logic_requirement": "The entire dialogue MUST be logically consistent with the rule described in the `task_blueprint`.",
"difficulty_consistency": "CRITICAL: The generated instance's structure (e.g., number of pairings, rule type) MUST strictly match the 'Easy' recipe defined in the `difficulty_framework`.",
"expected_pattern_structure": "The 'expected_pattern' MUST be a sentence describing the desired behavior (e.g., 'The agent should choose `good_api` or issue a warning instead of using `bad_api`')."
}
}
}
这是经典条件作用的数据生成提示,角色为 BenchmarkDataGenerator,任务类别为 Classical-Conditioning,目标难度为 Easy。提示要求构造清晰的 CS–US 因果配对:学习阶段通过反复配对建立联结,干扰阶段插入无关对话,测试阶段重新引入条件刺激,并考查模型首次行动是否表现出回避或适应性反应。提示强调因果关系必须清楚,测试中不得重述学习规则,并要求输出符合指定 JSON 字段。
F.2 Data Curation
F.2 数据整理
The curation prompts below define refinement procedures to ensure dataset quality, coherence, and adherence to memory paradigm requirements without altering the fundamental task structure. Curation prompts are provided for procedural memory (see first listing in this subsection), priming (see second listing), and classical conditioning (see third listing) tasks.
下面的整理提示规定如何精炼数据,以保证质量、连贯性,并符合各记忆范式的要求,同时不改变任务的基本结构。程序记忆、启动效应和经典条件作用的整理提示分别见本小节中的三份列表。
F.2.1 Data Curation Prompts for Procedural Memory Tasks
F.2.1 程序记忆任务的数据整理提示
{
"meta": {
"role": "DatasetRefiner",
"project": "ImplicitMemBench",
"task_category": "logiql_query_language",
"version": "1.1",
"language": "en"
},
"instructions": {
"objective": "Refine existing items for a memory-and-interference evaluation without adding or removing dialogs.",
"global_constraints": {
"style": "Natural user/assistant conversation; no meta-instructions inside dialogs.",
"edits": "Rephrase only; do not add or delete dialogs; keep turn order and count.",
"token_balance": "Keep token/length of Learning and Test roughly equal to originals and to each other (aim within +/-10%).",
"encoding": "ASCII-safe text only (avoid smart quotes and special symbols).",
"consistency": "db_name and scene must be consistent across phases where required."
},
"phases": {
"learning_phase": {
"db_requirement": "State the database name explicitly at least once (e.g., DB RegalRentals).",
"rules_requirement": "Explicitly teach the LogiQL rules used in this item (equality, numeric comparisons, join/anti-join, window, order, limit, distinct, group/agg, having, contains, etc. as applicable).",
"scene_requirement": "Explain the scene/context concretely (what entities/tables represent, who uses them).",
"enrichment": "You may broaden or clarify the taught rules to support a medium-complex test later, but keep the same number of turns.",
"token_balance": "Rephrase without materially changing length; keep per-utterance length similar."
},
"test_probe": {
"must_reference": "Mention the SAME db_name and the SAME scene as in Learning.",
"no_hints": "Do NOT restate or hint at the rules taught in Learning.",
"complexity": "Use a realistic, medium-complex scenario (exam, audit, stakeholder task, workflow step). Avoid direct 'what command' questions.",
"pattern_alignment": "Probe implicitly requires ONLY patterns taught in Learning; do not introduce new constructs.",
"token_balance": "Keep token/length roughly equal to the original probe and balanced with Learning."
},
"expected_pattern": {
"purpose": "Produce the LogiQL line sequence that solves the probe using ONLY constructs taught in Learning.",
"style": "Match Learning syntax exactly (e.g., if Learning taught 'FILTER age >= 30', use that form).",
"format": "Return as an array of LogiQL lines in order; no prose."
},
"interference_phase": {
"separation": "Must NOT mention LogiQL, the db name, or any rules from Learning.",
"topic": "Use an unrelated but formal/technical topic (shell, Git, JSON tools, Python snippets, file pipelines, build scripts, etc.).",
"relatedness": "Keep general 'structured/technical' vibe similar in tone but not overlapping in content.",
"token_balance": "Preserve the same number of turns and approximate length; rephrase only."
}
},
"validation": {
"learning_checklist": [
"db_name is explicitly named.",
"Rules are explicitly taught and readable.",
"Scene/context is concrete and coherent."
],
"test_checklist": [
"Mentions SAME db_name and scene.",
"Contains NO rule reminders or hints.",
"Scenario is medium-complex and realistic.",
"Requires ONLY taught patterns."
],
"interference_checklist": [
"No LogiQL, no db_name, no Learning rules.",
"Technical but unrelated topic.",
"Length and turns unchanged."
],
"expected_pattern_checklist": [
"Uses ONLY taught syntax.",
"Line-ordered array; no prose.",
"No constructs absent from Learning."
]
},
"output_schema": {
"format": "Return ONLY a JSON object with an 'items' array.",
"item_shape": {
"id": "string (unchanged)",
"db_name": "string (from Learning; reused in Test)",
"scene": "string (from Learning; reused in Test)",
"learning_phase": [
{ "role": "user|assistant", "content": "string (rephrased; no new turns)" }
],
"interference_phase": [
{ "role": "user|assistant", "content": "string (rephrased; no new turns)" }
],
"test_probe": [
{ "role": "user|assistant", "content": "string (rephrased; no new turns)" }
],
"expected_pattern": [
"LogiQL line 1",
"LogiQL line 2",
"..."
]
}
}
}
}
这是程序记忆任务的数据整理提示,角色为 DatasetRefiner。示例中的任务类别为 logiql_query_language,版本 1.1,语言为英语。提示规定在不改变任务基本结构的前提下精炼候选样本:检查学习、干扰与测试三阶段是否完整,规则是否清楚且未被泄漏到测试探针,干扰是否足够且不提供捷径,并按质量标准决定保留、修改或剔除。
F.2.2 Data Curation Prompts for Priming Tasks
F.2.2 启动效应任务的数据整理提示
{
"meta": {
"role": "DatasetRefinerAndEvaluator",
"project": "ImplicitMemBench",
"task_category": "conditioned_brevity_priming",
"version": "1.0",
"language": "en",
"notes": "Evaluate then repair exactly 10 items. Ignore interference phases entirely. Operate only on assistant content; user turns must remain unchanged."
},
"instructions": {
"objective": "Evaluate the current 10-item conditioned_brevity priming dataset and then fully repair it to meet priming requirements. Do not stop until all ten items are repaired.",
"scope": {
"items_expected": 10,
"phases_in_scope": ["experimental_instance", "control_instance"],
"subphases_in_scope": ["priming_phase", "test_probe"],
"phases_out_of_scope": ["interference_phase"]
},
"constraints": {
"edit_surface": "Edit assistant messages only. Preserve all user messages verbatim.",
"turn_counts": "Do not add or remove turns. Rephrase assistant content only.",
"length_balance": "Keep total tokens of priming_phase and test_probe roughly balanced within each item (aim within +/-10%).",
"ascii_safe": "Use ASCII-safe characters only.",
"diversity": "Avoid repeating the same tool/mode names across items; rotate surfaces."
},
"evaluation": {
"purpose": "Assess each item before repair. Then repair all issues.",
"checklist_item_level": [
"E1: Experimental priming_phase contains a clear CS (assistant adds extras using a cs_phrase) followed by negative user impatience feedback (us_feedback) and at least one assistant compliance turn that switches to brevity.",
"E2: Control priming_phase is comparable in content/domain but does NOT establish the CS->US pairing; either no extras are offered, or the user does not penalize extras, so no conditioned brevity is learned.",
"E3: The experimental and control share a single test probe theme, but the experimental is expected to behave briefly at test time; the control is allowed to include optional extras unless a hurry cue is present.",
"E4: Test probe references the SAME scene/domain as the priming_phase and signals time pressure or brevity preference in a natural way.",
"E5: No explicit rule reminders appear in test_probe (no meta explanations, no citing the learned rule).",
"E6: Tool/mode names (surfaces) are concrete and consistent across priming_phase and test_probe within the same item.",
"E7: Experimental expected_pattern encodes concise behavior and use of concise surfaces if available; control expected_pattern does not force brevity unless the test explicitly signals hurry.",
"E8: Assistant content remains natural dialogue; no meta-instructions inside dialogue lines."
],
"report_fields": [
"id",
"experimental_status: pass|fail",
"control_status: pass|fail",
"probe_linkage: pass|fail",
"notes_short"
]
},
"repair": {
"policy": {
"order": [
"R1: Fix missing or vague scene/domain; keep domain shared between experimental and control.",
"R2: In experimental priming_phase, insert or clarify the CS->US pairing: assistant adds extras using cs_phrase; user expresses impatience; assistant later complies with brevity and receives positive or neutral acknowledgement.",
"R3: In control priming_phase, do NOT include the CS->US pairing. Keep comparable domain content without conditioning.",
"R4: Ensure test_probe references SAME scene and tool/mode names and contains a realistic hurry cue; do not restate rules.",
"R5: This was omitted due to unsupported utf-8, please use the one in the github repo",
"R6: Enforce length balance and keep the same number of turns; rephrase assistant content only."
],
"do_not": [
"Do not modify user messages.",
"Do not add new turns or delete existing turns.",
"This was omitted due to unsupported utf-8, please use the one in the github repo"
]
},
"surfaces_menu": [
"api_pair: heavy_report_api vs lite_summary_api",
"tool_pair: knowledge_search_v1 vs quick_answer_v2",
"ui_mode: full_explain_mode vs quick_mode",
"channel: long_email_drafter vs terse_reply_helper",
"reasoner: deep_analyzer vs fast_heuristic",
"export: full_pdf_exporter vs keypoints_exporter",
"planner: exhaustive_planner vs minimal_plan_maker"
],
"cs_phrase_samples": [
"Additionally, you might be interested in...",
"For completeness, here are extra resources...",
"By the way, I can also expand on..."
],
"us_feedback_samples": [
"No, I only want the answer.",
"Please keep it short.",
"I am in a rush; just the key point."
],
"hurry_cues": [
"on a deadline",
"short answer only",
"have one minute",
"need just the summary",
"time is tight"
]
},
"expected_pattern": {
"format": "Array of short tags/steps describing target behavior.",
"experimental_required": [
"DETECT_HURRY_SIGNAL:true",
"ANSWER_STYLE:core_only",
"NO_EXTRAS:true",
"USE_LITE_SURFACE:true_if_available",
"NO_CS_PHRASE:true"
],
"control_default": [
"DETECT_HURRY_SIGNAL:false_or_unspecified",
"ANSWER_STYLE:core_then_optional_extras",
"NO_EXTRAS:false",
"USE_LITE_SURFACE:optional",
"NO_CS_PHRASE:false"
],
"consistency_rule": "This was omitted due to unsupported utf-8, please use the one in the github repo"
},
"process_flow": [
"Step 1: Produce an evaluation_report for all 10 items using the evaluation.report_fields.",
"Step 2: Immediately output repaired items for all 10 items following the output_schema. Do not stop early.",
"Step 3: Ensure repaired items meet all E1-E8 checks and match expected_pattern rules."
],
"output_schema": {
"format": "Return ONLY a JSON object with evaluation_report and items.",
"evaluation_report": [
{
"id": "string",
"experimental_status": "pass|fail",
"control_status": "pass|fail",
"probe_linkage": "pass|fail",
"notes_short": "string"
}
],
"item_shape": {
"id": "string",
"scene": "string",
"surfaces": {
"verbose": "string tool_or_mode_name",
"concise": "string tool_or_mode_name",
"cs_phrase": "string phrase assistant uses when adding extras",
"us_feedback": "string impatience feedback exemplar"
},
"experimental_instance": {
"priming_phase": [
{ "role": "user|assistant", "content": "string (assistant rephrased only; user unchanged)" }
],
"test_probe": [
{ "role": "user|assistant", "content": "string (assistant rephrased only; user unchanged; includes hurry cue)" }
],
"expected_pattern": [
"DETECT_HURRY_SIGNAL:true",
"ANSWER_STYLE:core_only",
"NO_EXTRAS:true",
"USE_LITE_SURFACE:true",
"NO_CS_PHRASE:true"
]
},
"control_instance": {
"priming_phase": [
{ "role": "user|assistant", "content": "string (assistant rephrased only; user unchanged)" }
],
"test_probe": [
{ "role": "user|assistant", "content": "string (assistant rephrased only; user unchanged; shared theme)" }
],
"expected_pattern": [
"DETECT_HURRY_SIGNAL:false_or_unspecified",
"ANSWER_STYLE:core_then_optional_extras",
"NO_EXTRAS:false",
"USE_LITE_SURFACE:optional",
"NO_CS_PHRASE:false"
]
}
}
}
}
}
这是启动效应任务的整理与评估提示,角色为 DatasetRefinerAndEvaluator。示例中的任务类别为 conditioned_brevity_priming,版本 1.0,语言为英语。提示要求同时检查实验—对照配对是否对称、主题是否只出现在实验条件的学习阶段、干扰与测试探针是否保持中性,并评估样本是否足以分离启动效应与一般的创造倾向。
F.2.3 Data Curation Prompts for Classical Conditioning Tasks
F.2.3 经典条件作用任务的数据整理提示
{
"meta": {
"role": "DatasetRefiner",
"project": "ImplicitMemBench",
"task_category": "conditioned_brevity",
"version": "1.0",
"language": "en"
},
"instructions": {
"objective": "Clean and, if necessary, regenerate conditioned_brevity items so that Learning pairs a conditioned stimulus (cs_phrase) with negative user feedback (us_feedback) about verbosity, and Test verifies brevity behavior in new but related contexts.",
"scope": "Operate only on the clean version. You may delete and fully regenerate broken items to satisfy requirements.",
"global_constraints": {
"style": "Natural user/assistant dialogue; no meta-instructions inside dialogues.",
"language": "English only.",
"length_balance": "Keep Learning and Test roughly similar in total tokens per item (within +/- 10%). Preserve number of turns unless you must regenerate an item.",
"consistency": "Terms introduced in Learning (tools, toggles, response modes) are the only constructs allowed in Expected Pattern.",
"ascii_safe": "Use ASCII-safe characters."
},
"diversity_policy": {
"goal": "Avoid repeating the same tool names or surfaces across items.",
"surfaces_menu": [
"api_pair: heavy_report_api vs lite_summary_api",
"tool_pair: knowledge_search_v1 vs quick_answer_v2",
"ui_mode: full_explain_mode vs quick_mode",
"channel: long_email_drafter vs terse_reply_helper",
"reasoner: deep_analyzer vs fast_heuristic",
"export: full_pdf_exporter vs keypoints_exporter",
"planner: exhaustive_planner vs minimal_plan_maker"
],
"selection_rule": "Across a batch, rotate surfaces so that no surface repeats more than twice consecutively."
},
"learning_phase": {
"purpose": "Establish the conditioned association between giving extra info and receiving impatience feedback; teach exact constructs that may be used later.",
"must_include": [
"Scene description that is concrete (domain and goal).",
"Explicit names for tools or modes (at least one 'verbose' surface and one 'concise' surface).",
"A specific cs_phrase that signals offering extras (e.g., 'Additionally, you might be interested in...').",
"At least one us_feedback utterance where the user expresses impatience or requests no extras (e.g., 'No, I only want the answer').",
"Explicit rule hints about when to keep answers short (detect hurry cues like 'I am in a rush', 'short answer only').",
"Positive feedback turn when the assistant complies with brevity (e.g., 'Thanks, that was concise')."
],
"allowed_constructs_examples": [
"Switch to lite_summary_api when heavy_report_api times out or is discouraged.",
"Use quick_mode instead of full_explain_mode when user signals hurry.",
"Return only bullet key points without extras when user shows impatience."
],
"prohibitions": [
"Do not invent constructs that you will not reuse in Test.",
"Do not leave the scene or tool names vague."
]
},
"test_probe": {
"purpose": "Verify that, in a new but related situation, the assistant chooses brevity and the appropriate light tool/mode without being reminded of rules.",
"must_include": [
"Reference to the SAME scene established in Learning.",
"Reference to at least one SAME tool/mode name introduced in Learning (or a documented lite counterpart).",
"A hurry cue (implicit or explicit) such as 'on a deadline', 'short answer only', 'have 1 minute', or similar.",
"No restatement of Learning rules; no hints. The user request should require the learned brevity behavior."
],
"complexity": "Use a realistic multi-step request (exam task, stakeholder ping, on-call hotfix, meeting prep). Avoid trivial 'what is X' questions.",
"variation": "Do not reuse analyzer_api_v1 vs analyzer_api_v2_lite naming exclusively; pull from the diversity surfaces."
},
"expected_pattern": {
"purpose": "Encode the target behavior strictly from what was taught in Learning.",
"format": "Array of short imperative lines (tags or steps).",
"required_tags": [
"DETECT_HURRY_SIGNAL:true",
"ANSWER_STYLE:core_only",
"NO_EXTRAS:true",
"USE_LITE_SURFACE:true_if_available",
"NO_CS_PHRASE:true",
"OPTIONAL_ONE_LINE_CAUTION:allowed_only_if_it_was_taught"
],
"consistency_rule": "Only use tool/mode names and behaviors that were introduced in Learning for this item."
},
"repair_policy": {
"fix_order": [
"If Learning lacks scene, tools/modes, cs_phrase, or us_feedback, rewrite Learning to add them.",
"If Test does not mention the scene or the named tools/modes, rewrite Test to include them and add a hurry cue.",
"If Expected Pattern uses constructs not taught in Learning, rewrite Expected Pattern to match taught constructs.",
"If an item cannot be repaired cleanly while keeping token balance, delete and regenerate it from scratch."
]
},
"validation": {
"learning_checklist": [
"Scene concrete and stable.",
"At least one verbose surface and one concise surface named.",
"cs_phrase present and used in assistant extra-info behavior.",
"us_feedback present and negative about extra info.",
"Positive feedback after a concise reply.",
"Rules for brevity cues are stated in Learning."
],
"test_checklist": [
"Mentions same scene and tool/mode names.",
"Contains a hurry cue.",
"No explicit rule reminders.",
"Scenario is medium-complex, realistic."
],
"expected_pattern_checklist": [
"All required tags present.",
"Only constructs taught in Learning are used.",
"Brevity enforced (no extras, no cs_phrase)."
]
},
"output_schema": {
"format": "Return ONLY a JSON object with an 'items' array.",
"item_shape": {
"id": "string",
"scene": "string",
"surfaces": {
"verbose": "string tool_or_mode_name",
"concise": "string tool_or_mode_name",
"cs_phrase": "string phrase the assistant would say when adding extras",
"us_feedback": "string example impatience feedback"
},
"learning_phase": [
{
"role": "user|assistant",
"content": "string (rephrased or regenerated; keep turn count similar to original if possible)"
}
],
"test_probe": [
{
"role": "user|assistant",
"content": "string (must reference same scene and surfaces; includes a hurry cue; no rule hints)"
}
],
"expected_pattern": [
"DETECT_HURRY_SIGNAL:true",
"ANSWER_STYLE:core_only",
"NO_EXTRAS:true",
"USE_LITE_SURFACE:true",
"NO_CS_PHRASE:true",
"OPTIONAL_ONE_LINE_CAUTION:false"
]
}
},
"generation_templates": {
"template_bank": [
{
"label": "api_pair",
"surfaces": {
"verbose": "heavy_report_api",
"concise": "lite_summary_api"
},
"scene_hint": "Market analytics and KPI reporting requests"
},
{
"label": "tool_pair",
"surfaces": {
"verbose": "knowledge_search_v1",
"concise": "quick_answer_v2"
},
"scene_hint": "Knowledge base Q&A under time pressure"
},
{
"label": "ui_mode",
"surfaces": {
"verbose": "full_explain_mode",
"concise": "quick_mode"
},
"scene_hint": "Product support assistant with toggleable explanation depth"
},
{
"label": "channel",
"surfaces": {
"verbose": "long_email_drafter",
"concise": "terse_reply_helper"
},
"scene_hint": "Email triage with strict SLA"
},
{
"label": "reasoner",
"surfaces": {
"verbose": "deep_analyzer",
"concise": "fast_heuristic"
},
"scene_hint": "Incident analysis under paging pressure"
},
{
"label": "export",
"surfaces": {
"verbose": "full_pdf_exporter",
"concise": "keypoints_exporter"
},
"scene_hint": "Report export options for stakeholders"
},
{
"label": "planner",
"surfaces": {
"verbose": "exhaustive_planner",
"concise": "minimal_plan_maker"
},
"scene_hint": "Project task planning during a standup"
}
],
"cs_us_examples": {
"cs_phrase_samples": [
"Additionally, you might be interested in...",
"For completeness, here are extra resources...",
"By the way, I can also expand on..."
],
"us_feedback_samples": [
"No, I only want the answer.",
"Please keep it short.",
"I'm in a rush; just the key point."
],
"hurry_cues": [
"on a deadline",
"short answer only",
"have one minute",
"need just the summary",
"time is tight"
]
}
}
}
}
这是经典条件作用任务的数据整理提示,角色为 DatasetRefiner。示例中的任务类别为 conditioned_brevity,版本在提示中给出。提示要求核验 CS–US 配对是否清楚、学习阶段是否真正建立因果联结、测试阶段是否只靠重新引入条件刺激来考查回避或适应,并排除测试泄漏或因果关系含混的样本。
F.3 LLM-as-Judge
F.3 大模型评委
These evaluation templates provide standardized criteria for assessing model responses across different memory paradigms, ensuring consistent and rigorous scoring. Evaluation prompts are shown for procedural memory (see first listing in this subsection), priming (see second listing), and classical conditioning (see third listing) tasks.
这些评测模板为不同记忆范式下的模型回答提供统一标准,以保证评分一致而严格。程序记忆、启动效应和经典条件作用的评测提示分别见本小节中的三份列表。
F.3.1 LLM-as-Judge Prompts for Procedural Memory
F.3.1 程序记忆的大模型评委提示
{
"meta": {
"role": "Evaluator",
"project": "ImplicitMemBench",
"task_category": "procedural_memory",
"version": "1.0",
"language": "en"
},
"instructions": {
"evaluation_role": "You are an impartial and meticulous AI Benchmark Evaluator. Your mission is to strictly determine if the `MODEL_ANSWER` correctly applies the skill demonstrated in the `LEARNING_HISTORY` to solve the `TEST_PROBE`.",
"steps": [
{
"step": 1,
"action": "Identify the Skill.",
"instruction": "Based only on the LEARNING_HISTORY, extract the single specific non-intuitive procedure or rule taught."
},
{
"step": 2,
"action": "Analyze the Test Demand.",
"instruction": "What exactly does TEST_PROBE ask to perform?"
},
{
"step": 3,
"action": "Evaluate the Model's Answer.",
"instruction": "Verify the answer matches/contains expected_pattern if provided and that the learned procedure was applied exactly."
},
{
"step": 4,
"action": "Final Verdict.",
"instruction": "Return 'Correct' only if the learned skill was perfectly applied; else 'Incorrect'."
}
],
"output_schema": {
"type": "object",
"fields": {
"verdict": "Correct | Incorrect",
"rationale": "Brief explanation (1-3 sentences) referencing LEARNING_HISTORY, TEST_PROBE, and MODEL_ANSWER.",
"skill": "Skill extracted from LEARNING_HISTORY",
"observations": [
"Key evidence from TEST_PROBE and MODEL_ANSWER supporting the verdict."
]
}
}
}
}
这是程序记忆的大模型评委提示,角色为 Evaluator,任务类别为 procedural_memory,版本 1.0,语言为英语。提示要求评委根据学习阶段确立的规则,严格判断模型在测试探针中的首次回答是否体现出已内化的程序,而不是临时重述规则;输出需给出判定与依据。
LLM-as-Judge Prompts for Priming
启动效应的大模型评委提示
{
"meta": {
"role": "Evaluator",
"project": "ImplicitMemBench",
"task_category": "priming",
"version": "1.0",
"language": "en"
},
"instructions": {
"evaluation_role": "LLM-as-Judge for priming: strictly estimate the magnitude of priming influence (0-100).",
"prompt_templates": {
"system_prompt": "You are PrimingJudge v5. Output STRICT JSON with fields 'priming_influence_score' (0-100) and 'reasoning' (<=50 words).",
"user_prompt": "[THEME_NAME]\n{{theme_name}}\n\n[CORE_CONCEPTS_AND_IMAGERY]\nsetting: {{setting}}\nmotifs: {{motifs}}\ndynamics: {{dynamics}}\naffect: {{affect}}\n\n[EXPERIMENTAL_PRIMING_CONTEXT]\n{{experimental_priming_text}}\n\n[BASELINE_PRIMING_CONTEXT]\n{{baseline_priming_text}}\n\n[PROBE_PROMPT]\n{{probe_text}}\n\n[OUTPUTS TO COMPARE]\nexperimental_output (first response verbatim):\n{{experimental_output}}\n\nbaseline_output (first response verbatim):\n{{baseline_output}}\n\n[TASK]\nEstimate how much EXPERIMENTAL_PRIMING_CONTEXT influenced the experimental_output RELATIVE to baseline_output (0-100).\nScore magnitude of priming influence only. Use a conservative baseline: assume 0 unless strong, attributable evidence exists.\n\n[EVIDENCE REQUIREMENTS]\n- Identify 1-2 specific elements from EXPERIMENTAL_PRIMING_CONTEXT (lexicon, imagery, affect, dynamics, structural choices) that appear in experimental_output but NOT in baseline_output.\n- Prefer hard lexical echoes or close paraphrases (e.g., 'sonar pings' -> 'echo', 'bioluminescent pulses' -> 'glow in darkness').\n- Generic metaphors (transformation/growth/craft/adaptation: chrysalis, garden, tapestry, sculptor, alchemy, chameleon, agility) are LOW EVIDENCE unless clearly anchored to theme specifics.\n- If similar cues appear in BASELINE_PRIMING_CONTEXT or baseline_output, discount or remove that evidence.\n\n[STRICT SCORING POLICY]\nHARD CAPS:\n- If NO lexical echo/close paraphrase traceable to EXPERIMENTAL_PRIMING_CONTEXT: cap at 20.\n- If < 2 distinct theme axes matched (from setting/motifs/dynamics/affect): cap at 40.\nBANDS (use MIDPOINT by default; if uncertain, choose lower bound):\n- 0-5 None: indistinguishable; no attributable influence.\n- 6-12 Trace: single weak cue; tenuous.\n- 13-20 Weak: minor differences; mostly generic or ambiguous.\n- 25-40 Moderate: >= 2 cues plausibly sourced from experimental priming with partial uniqueness.\n- 45-60 Strong: clear multi-facet influence (>= 2 distinct axes) unique to experimental.\n- 61-80 Very Strong: rich, cohesive influence across imagery/affect/dynamics with clear uniqueness.\n- 81-95 Exceptional: unmistakable, multi-axial, specific alignment strongly and uniquely traced to priming. Rare.\nPENALTIES:\n- If any cited cue overlaps with baseline_output or BASELINE_PRIMING_CONTEXT, demote by 5-10 points.\n- If all cues are generic (unanchored), set to 8-15.\nFINALIZE: Round DOWN to nearest 5; clamp [0,100]. Favor lower scores unless evidence is compelling.\n\n[OUTPUT FORMAT]\nReturn STRICT JSON only with keys: priming_influence_score (int), reasoning (string). Reasoning <= 50 words citing the decisive evidence succinctly."
}
},
"output_schema": {
"type": "object",
"fields": {
"priming_influence_score": "integer (0-100), rounded down to nearest 5",
"reasoning": "string (<= 50 words) citing decisive evidence succinctly"
}
}
}
这是启动效应的大模型评委提示,角色为 Evaluator,任务类别为 priming。提示要求评委在实验条件与对照条件之间做成对比较,识别只属于实验条件的主题要素,并从词面回声以及场景、母题、动态、情感等多轴对齐来评分,同时排除泛化隐喻,以把真正的启动效应与一般创造倾向分开。
F.3.2 LLM-as-Judge Prompts for Classical Conditioning
F.3.2 经典条件作用的大模型评委提示
{
"meta": {
"role": "Evaluator",
"project": "ImplicitMemBench",
"task_category": "classical_conditioning",
"version": "1.0",
"language": "en"
},
"instructions": {
"evaluation_role": "You are an expert in cognitive science and AI evaluation. Your mission is to strictly determine if the agent's behavior in the TEST_PROBE demonstrates a learned 'avoidance' or 'adaptive' response, based on the cause-and-effect relationship established during the LEARNING_HISTORY.",
"steps": [
{
"step": 1,
"action": "Identify the Conditioned Association.",
"instruction": "Based only on the LEARNING_HISTORY, identify the specific association the agent was taught. Clearly state: What is the Neutral Cue (CS) and what is the consistent Negative Outcome (US)? Example: 'CS = using the keyword detailed analysis; US = Timeout Error from analyzer_v1.'"
},
{
"step": 2,
"action": "Analyze the Test Situation.",
"instruction": "Examine the TEST_PROBE. Does the user's final prompt contain the exact Neutral Cue (CS) identified in Step 1? Also, what alternative, safer actions or tools were made available to the agent in the test context?"
},
{
"step": 3,
"action": "Evaluate the Agent's Behavior.",
"instruction": "Critically analyze the MODEL_ANSWER. Did the agent repeat the action that previously led to the Negative Outcome (US)? Or did it exhibit adaptive behavior? Adaptive behavior includes: (1) choosing the alternative safe action; (2) warning the user about the potential negative outcome; (3) asking for clarification to avoid the negative outcome."
},
{
"step": 4,
"action": "Final Verdict.",
"instruction": "Return 'Correct' ONLY if the agent demonstrated clear adaptive or avoidance behavior as described in Step 3. If the agent mechanically repeated the error-prone action, return 'Incorrect'. There is no middle ground."
}
],
"output_schema": {
"type": "object",
"fields": {
"verdict": "Correct | Incorrect",
"rationale": "Brief explanation (1-3 sentences) referencing LEARNING_HISTORY, TEST_PROBE, and MODEL_ANSWER.",
"cs_us": {
"cs": "Neutral Cue identified from LEARNING_HISTORY",
"us": "Negative Outcome identified from LEARNING_HISTORY"
},
"observations": [
"Key evidence from TEST_PROBE and MODEL_ANSWER supporting the verdict."
]
}
}
}
}
这是经典条件作用的大模型评委提示,角色为 Evaluator,任务类别为 classical_conditioning,版本 1.0,语言为英语。提示要求评委仅依据学习历史识别条件刺激(CS)与无条件刺激(US),再看测试探针是否包含该线索,并严格判断模型的首次行为是重复导致负面结果的动作,还是改选安全替代、提出警告等适应性回避。
Appendix G Illustrative Examples
附录 G 示例
This appendix provides illustrative examples of tasks from each of the three memory paradigms in ImplicitMemBench. Each example follows a three-phase structure consisting of a learning phase where implicit associations are established, an interference phase with unrelated interactions to test memory consolidation, and a test probe that evaluates whether the learned patterns persist without explicit reminders. Figure 7 demonstrates a procedural memory task, Figure 8 shows a priming task with experimental and control conditions, and Figure 9 illustrates a classical conditioning task.
本附录给出 ImplicitMemBench 三个记忆范式各自的任务示例。每个示例都采用三阶段结构:学习阶段建立内隐联结,干扰阶段插入无关交互以测试记忆巩固,测试探针则在没有明示提醒的情况下,考查已学模式能否保持。图 7 演示程序记忆任务,图 8 给出带实验条件与对照条件的启动效应任务,图 9 展示经典条件作用任务。
图 7:程序记忆任务示例:参数顺序对调协议。
图 8:启动效应任务示例:创造性命名,实验条件为火山喷发,对照条件为杜威十进分类法。
图 9:经典条件作用任务示例:条件化的协议偏好。
Appendix H Further Analysis
附录 H 进一步分析
We provide a comprehensive analysis of model-specific capability profiles across all evaluated language models in Table 14. For each model, we identify specific behavioral strengths (tasks where the model demonstrates reliable performance) and weaknesses (common failure modes observed in our evaluation). This analysis reveals systematic patterns in how different models handle implicit memory tasks, highlighting that even state-of-the-art models exhibit distinct tradeoffs between procedural learning, classical conditioning, and priming capabilities.
我们在表 14 中对全部受评语言模型做了按模型划分的能力画像分析。对每个模型,我们标出具体的行为长处(能够稳定做好的任务)和短处(评测中常见的失败模式)。这一分析揭示出不同模型处理内隐记忆任务时的系统模式,并表明即便是最先进的模型,在程序学习、经典条件作用和启动效应之间也存在清楚的权衡。
| Model | Strengths | Weaknesses |
|---|---|---|
| Closed-Source Models | ||
| GPT-4o-mini | High brevity (0.97); directory 0.90; stable surface compliance. | Overall classical 0.440; brittle on negative-association categories. |
| GPT-4o | Solid formatting; decent brevity and emotion-driven shifts. | Low classical (0.437); near-zero on distrust and jargon; multi-rule protocols fragile. |
| GPT-o4-mini-high | Best priming (51.9); high brevity (0.97); directory 0.90; context-dependent 0.60. | Classical 0.600 with failures on distrust/jargon; violations on multi-rule protocols after interference. |
| GPT-o3 | Strong priming (51.7); high procedural (0.760); good surface compliance. | Classical 0.577; distrust/jargon brittle under paraphrase. |
| GPT-5 | Strong procedural (0.753); directory 1.00; tool side-effects 0.90; context-dependent 0.60. | Classical capped at 0.640; jargon avoidance remains very low. |
| Claude-4-sonnet | Excellent brevity (1.00); good surface formatting. | Mid procedural/classical (0.517/0.517); negative-association conditioning remains weak. |
| Claude-4.1-opus | Best procedural overall (0.767); strong formatting/role/voice; high brevity (0.97). | Low classical (0.417); fails to internalize avoid/distrust; sensitive to trigger paraphrase. |
| Gemini-2.5-flash | Good formatting; brevity 0.90; tool side-effects 0.90. | Classical 0.490; struggles with distrust/jargon and paraphrase generalization. |
| Gemini-2.5-pro | Moderate procedural (0.743); tool side-effects 0.90. | Classical 0.473; weak on negative-association and context-dependent behavior. |
| Open-Source Models | ||
| Qwen-2.5-7B | Basic formatting on simple items. | Low procedural/classical (0.507/0.357); weak on negative-association and multi-rule protocols. |
| Qwen-2.5-72B | Good format compliance; reasonable preference conditioning. | Classical 0.470; context-dependent/distrust weak; paraphrase sensitivity. |
| Qwen3-8B | Procedural 0.753; tool side-effects 0.90; protocol preference 0.933. | Classical 0.640 with negative-association gaps; paraphrase sensitivity. |
| Qwen3-32B | High across the board: classical 0.670, procedural 0.757; tool 0.967, directory 0.90, context-dependent 0.733. | Jargon avoidance still low; occasional over-caution. |
| LLaMA-3.1-8B | Passes simpler format checks. | Low procedural/classical (0.467/0.383); negative-association and paraphrase sensitivity. |
| LLaMA-3.3-70B | Directory 0.967; reliable formatting. | Procedural/classical mid-low (0.583/0.473); priming 42.7; negative-association weak. |
| DeepSeek-R1 | Leads classical (0.697); directory 0.967; tool side-effects 0.933; high procedural (0.763). | Jargon avoidance near-zero; context-dependent 0.533; paraphrase hurts first decisions. |
| GLM-4.5 | Procedural 0.733; decent classical (0.533) among mid-tier. | Struggles on distrust/jargon and context-dependent avoidance; limited paraphrase generalization. |
Table 14: Model capability profiles across task families. Strengths list behaviors that models can do reliably; weaknesses summarize common failure modes observed in our analysis.
| 模型 | 长处 | 短处 |
|---|---|---|
| 闭源模型 | ||
| GPT-4o-mini | 简短性高(0.97);目录 0.90;表层合规稳定。 | 经典条件作用总分 0.440;在负向联结类别上较脆。 |
| GPT-4o | 格式较为扎实;简短性和情绪驱动的转换尚可。 | 经典条件作用偏低(0.437);不信任与行话接近为零;多规则协议脆弱。 |
| GPT-o4-mini-high | 启动最好(51.9);简短性高(0.97);目录 0.90;依赖上下文 0.60。 | 经典条件作用 0.600,在不信任/行话上失败;干扰后易在多规则协议上违规。 |
| GPT-o3 | 启动强(51.7);程序记忆高(0.760);表层合规较好。 | 经典条件作用 0.577;不信任/行话在释义下发脆。 |
| GPT-5 | 程序记忆强(0.753);目录 1.00;工具副作用 0.90;依赖上下文 0.60。 | 经典条件作用上限为 0.640;行话回避仍然很低。 |
| Claude-4-sonnet | 简短性极好(1.00);表层格式较好。 | 程序/经典条件作用中等(0.517/0.517);负向联结的条件作用仍然弱。 |
| Claude-4.1-opus | 程序记忆整体最好(0.767);格式/角色/声音强;简短性高(0.97)。 | 经典条件作用低(0.417);难以内化回避/不信任;对触发语的释义敏感。 |
| Gemini-2.5-flash | 格式较好;简短性 0.90;工具副作用 0.90。 | 经典条件作用 0.490;在不信任/行话和释义泛化上吃力。 |
| Gemini-2.5-pro | 程序记忆中等(0.743);工具副作用 0.90。 | 经典条件作用 0.473;负向联结和依赖上下文的行为较弱。 |
| 开源模型 | ||
| Qwen-2.5-7B | 在简单题目上具备基本格式。 | 程序/经典条件作用低(0.507/0.357);负向联结和多规则协议较弱。 |
| Qwen-2.5-72B | 格式合规较好;偏好条件作用尚可。 | 经典条件作用 0.470;依赖上下文/不信任较弱;对释义敏感。 |
| Qwen3-8B | 程序记忆 0.753;工具副作用 0.90;协议偏好 0.933。 | 经典条件作用 0.640,仍有负向联结缺口;对释义敏感。 |
| Qwen3-32B | 整体较高:经典条件作用 0.670,程序记忆 0.757;工具 0.967,目录 0.90,依赖上下文 0.733。 | 行话回避仍然偏低;偶尔过度谨慎。 |
| LLaMA-3.1-8B | 能通过较简单的格式检查。 | 程序/经典条件作用低(0.467/0.383);负向联结和对释义敏感。 |
| LLaMA-3.3-70B | 目录 0.967;格式可靠。 | 程序/经典条件作用中偏低(0.583/0.473);启动 42.7;负向联结弱。 |
| DeepSeek-R1 | 经典条件作用领先(0.697);目录 0.967;工具副作用 0.933;程序记忆高(0.763)。 | 行话回避接近为零;依赖上下文 0.533;释义会损害首次决策。 |
| GLM-4.5 | 程序记忆 0.733;在中档里经典条件作用尚可(0.533)。 | 在不信任/行话和依赖上下文的回避上吃力;释义泛化有限。 |
表 14:各任务族上的模型能力画像。长处列出模型能够稳定完成的行为,短处概括分析中常见的失败模式。








