Memory efficiency and resource-rational encoding in sentence processing

句子加工中的记忆效率与资源理性编码

Weijie Xu
University of California, Irvine
weijie.xu@uci.edu

Brian Dillon
University of Massachusetts Amherst
bwdillon@umass.edu

Richard Futrell
University of California, Irvine
rfutrell@uci.edu

Abstract

摘要

There is a growing consensus that, in order to serve as models of human language processing, language models (LMs) need to be constrained in their use of memory for context, the analogue to human working memory (WM). Here we take a novel yet simple approach to constraining WM in language models, in a way that reflects models of human cognition where memory is treated as a limited resource and deployed strategically. In order to capture this constraint on memory encoding, we inject noise into the hidden representations of Transformer-based LMs at tunable rates. Then we train the models with a hybrid objective, such that they learn to maximize the performance of next-word prediction subject to explicit constraints on the total encoding precision. We find that explicit WM constraints improve the model’s alignment with human reading times. More importantly, we find that the need to manage encoding precision reshapes the nature of the models’ context representations, making them more compressed and categorical. Our results show how resource-rational models of WM allocation can be implemented in neural models simply and successfully, and point to a dissociation between WM retrieval mechanisms and the underlying memory representations in models of human sentence processing.

现有共识正逐渐形成：语言模型（LM）若要充当人类语言加工的模型，就必须限制其用于上下文的记忆，这相当于人类的工作记忆（WM）。本文采用一种新的、却也简洁的方法来约束语言模型中的工作记忆，其思路对应把记忆视为有限资源、并加以策略性配置的人类认知模型。为捕捉这种加在记忆编码上的约束，我们以可调节的速率，向基于 Transformer 的语言模型的隐表征注入噪声。随后用混合目标训练模型，使其在总编码精度受到显式约束的前提下，尽量提高下一个词预测的表现。我们发现，显式的工作记忆约束能够提升模型与人类阅读时间的对齐程度。更重要的是，管理编码精度的需求会改变模型上下文表征的形态，使其更压缩、也更范畴化。结果表明，工作记忆分配的资源理性模型可以简单而有效地在神经模型中实现，并提示：在人类句子加工的模型里，工作记忆的检索机制与底层记忆表征之间可以相互分离。

1 Introduction

1 引言

Human sentence processing is incremental, continuously integrating linguistic signals into contextual representations as the sentence unfolds. This incremental process requires the support of working memory (WM), since a previous input is not perceptually accessible anymore at a future time point. However, our WM capacity is strikingly limited, both in terms of how many items can be simultaneously stored (Miller, 1956; Lewis, 1996; McElree, 2006), and in terms of how long it can be faithfully represented before being forgotten (Gibson, 1998; Futrell et al., 2020). Despite considerable empirical support for this cognitive limitation in sentence processing, theoretically, the exact representational nature of WM constraints remains an open question. This paper attempts to answer: What are WM resource constraints? How can we simulate them at the computational level? How do they shape the representational space in sentence processing? To answer these questions, we propose to integrate WM constraints into Transformer language models (LMs), in a way that allows us to generate and evaluate hypotheses for human WM processes in a stimulus computable fashion (Frank and Goodman, 2025). Previous work has proposed numerous ad-hoc ways to constrain the memory in LMs, including reduced size of the model or training data (Warstadt et al., 2023) and locally biased attention (De Varda and Marelli, 2024; Clark et al., 2025). Here we pursue a more principled approach grounded in resource-rational cognitive theories (Lewis et al., 2014; Lieder and Griffiths, 2020), aiming to advance understanding of both psycholinguistic theories of WM constraint and the impact of our psycholinguistically inspired architectural choice on LMs’ performance. We conceptualize WM constraints focusing on the representational quality of information (McElree, 2006). We argue that greater memory resources result in lower representational uncertainty in memory encodings of linguistic inputs (Ma et al., 2014; Bates and Jacobs, 2020; Bays et al., 2024; Xu and Futrell, 2026). To capture representational uncertainty, we inject noise into Transformer’s self-attention value vectors, such that the encoding precision at a key position exponentially decays as a function of its distance with the query position. Given our conceptualization, models with greater WM resources at their disposal can encode linguistic inputs with greater precision in their value vectors. We then train LMs with varying WM constraints in a resource-rational fashion:

人类的句子加工是增量的：随着句子展开，语言信号被持续整合进上下文表征。这一增量过程需要工作记忆（WM）的支撑，因为先前的输入在后续时刻已不再可被感知到。然而，我们的工作记忆容量极为有限：既能同时保存的项目数量有限（Miller, 1956; Lewis, 1996; McElree, 2006），表征能在被遗忘前忠实保持的时间也短（Gibson, 1998; Futrell et al., 2020）。尽管句子加工中这一认知限制已有大量经验支持，工作记忆约束在表征上究竟意味着什么，理论上仍是开放问题。本文试图回答：工作记忆资源约束是什么？如何在计算层上模拟它们？它们如何塑造句子加工中的表征空间？为回答这些问题，我们提议把工作记忆约束写入 Transformer 语言模型（LM），从而能以刺激可计算（stimulus-computable）的方式提出并检验关于人类工作记忆过程的假说（Frank and Goodman, 2025）。先前工作提出过多种临时性的记忆约束办法，包括缩小模型或训练数据（Warstadt et al., 2023），以及带局部偏向的注意力（De Varda and Marelli, 2024; Clark et al., 2025）。我们则采取一条更有原则的路线，扎根于资源理性的认知理论（Lewis et al., 2014; Lieder and Griffiths, 2020），既希望推进关于工作记忆约束的心理语言学理论，也希望看清这种受心理语言学启发的架构选择会如何影响语言模型的表现。我们把工作记忆约束主要放在信息的表征质量上（McElree, 2006）。我们认为，记忆资源越充足，语言输入在记忆编码中的表征不确定性就越低（Ma et al., 2014; Bates and Jacobs, 2020; Bays et al., 2024; Xu and Futrell, 2026）。为刻画表征不确定性，我们向 Transformer 自注意力的价值向量注入噪声，使某一键位置上的编码精度随其与查询位置的距离按指数衰减。按这一构想，可支配工作记忆资源更多的模型，就能以更高的精度把语言输入编码进价值向量。随后，我们以资源理性的方式，在不同工作记忆约束下训练语言模型：

Figure 1: a. Greater WM resources result in lower representational noise for the encoded linguistic input. b. Injecting Gaussian noise into Transformers’ self-attention value vectors v, whose precision decays as a function of query-key distance controlled by θ. c. Precision decay rate θ learned by models across attention heads per layer.

图 1：a. 工作记忆资源越多，已编码语言输入的表征噪声越低。b. 向 Transformer 自注意力的价值向量 $v$ 注入高斯噪声，其精度随查询–键距离衰减，由 $\theta$ 控制。c. 模型在各层各个注意力头上学到的精度衰减速率 $\theta$。

The models learn to maximize the performance of next-word prediction task under the constraint of limited WM resources for precise encoding (Hahn et al., 2022). We first observe an asymmetrical decay of encoding precision: lower-layer attention heads undergo faster precision decay under the WM constraint. We then evaluate our models by using their surprisal estimates to predict human reading times (RTs). Consistent with previous studies, we find that explicit WM constraints can help the model’s surprisal estimates to better predict RTs. More importantly, the model’s output distributions reveal that stronger constraints lead to a more compressed representational space in next-word predictions, resulting in more categorical predictions with less fine-grained distinctions across different contextual inputs, a novel prediction to be examined in future experimental work of human sentence processing.

模型要在用于精确编码的工作记忆资源有限的约束下，尽量提高下一个词预测任务的表现（Hahn et al., 2022）。我们首先观察到编码精度的不对称衰减：在工作记忆约束下，较低层的注意力头精度衰减更快。接着，我们用模型给出的 surprisal（意外度）估计去预测人类阅读时间（RT）。与已有研究一致，显式的工作记忆约束能够让模型的 surprisal 估计更好地预测阅读时间。更重要的是，模型的输出分布表明，更强的约束会把下一个词预测的表征空间压得更紧，预测也更范畴化，不同上下文输入之间的细粒度差别变少。这是一条新的预测，有待未来的人类句子加工实验加以检验。

2 Background

2 背景

2.1 WM constraints at the computational level

2.1 计算层上的工作记忆约束

In order to understand WM processes (Oberauer, 2009), two questions must be addressed: (i) what kinds of information are encoded in memory; (ii) how do we make use of such information? In sentence processing, computational models and theories have focused primarily on the latter, targeting the retrieval mechanism of WM (Lewis and Vasishth, 2005; Lewis et al., 2006). The former has, however, long been the elephant in the room, until a more recent and growing body of work has started to address the representational nature of WM encoding (e.g., Smith and Vasishth, 2020; Hahn et al., 2022; Keshev et al., 2025; Xu and Futrell, 2026). WM resources, under retrieval-based models, are often conceived as attentional resources, whose distribution determines the extent to which a previous input is retrieved and incorporated into the computation of representation at a future time point (Lewis and Vasishth, 2005; Lewis et al., 2006; Ryu and Lewis, 2021, 2025; Oh and Schuler, 2022). For most studies applying these ideas in LMs, this WM constraint has been interpreted as a locality bias in attention (De Varda and Marelli, 2024; Clark et al., 2025; Mita et al., 2025) or as limited length of context window (Kuribayashi et al., 2022), a design choice that is grounded in the empirical observation where information in the recent past is more salient and better remembered (Gibson et al., 1996; Gibson, 1998; Bartek et al., 2011). This retrieval-centered theorizing of WM constraint leaves unaddressed how such a constraint influences what we have encoded in memory in the first place. Here we pursue a different approach to conceptualize WM resources and constraints: instead of viewing them as attentional resources, we focus on the representational nature of the encoded information itself, and directly link WM resources to representational quality (McElree, 2006). In an interesting exception, Timkey and Linzen (2023) propose an implementation of WM constraint that is not grounded in the locality bias. Instead, the constraint consists in the limited number of computational modules (that is, the number of attention heads).

要理解工作记忆过程（Oberauer, 2009），必须回答两个问题：（i）记忆里编码的是哪类信息；（ii）我们如何使用这些信息？在句子加工中，计算模型与理论主要关注后者，把目标对准工作记忆的检索机制（Lewis and Vasishth, 2005; Lewis et al., 2006）。前者却长期像房间里那头无人提及的大象，直到近年才有越来越多的工作开始讨论工作记忆编码的表征性质（例如 Smith and Vasishth, 2020; Hahn et al., 2022; Keshev et al., 2025; Xu and Futrell, 2026）。在基于检索的模型里，工作记忆资源往往被看成注意资源：它如何分配，决定了先前输入在未来时刻被检索出来、并进入表征计算的程度（Lewis and Vasishth, 2005; Lewis et al., 2006; Ryu and Lewis, 2021, 2025; Oh and Schuler, 2022）。多数把这些想法用于语言模型的研究，把这一工作记忆约束理解成注意力上的局部偏向（De Varda and Marelli, 2024; Clark et al., 2025; Mita et al., 2025），或理解成有限的上下文窗口长度（Kuribayashi et al., 2022）；这一设计选择的依据是经验观察：近期过去的信息更显著，也被记得更好（Gibson et al., 1996; Gibson, 1998; Bartek et al., 2011）。这种以检索为中心的工作记忆约束理论，没有回答该约束最初如何影响我们编码进记忆的内容。这里我们改用另一种方式来构想工作记忆资源与约束：不再把它们看成注意资源，而是着眼于已编码信息本身的表征性质，并把工作记忆资源直接与表征质量联系起来（McElree, 2006）。一个有意思的例外是 Timkey and Linzen (2023)：他们提出的工作记忆约束实现并不建立在局部偏向上，约束体现为计算模块数量有限（也就是注意力头的数量）。

Human memory representations are noisy, as the received information always undergoes a certain degree of unavoidable perturbation during transmission in the neural system (e.g., Ma et al., 2014; Futrell et al., 2020; Hahn et al., 2022; Brady et al., 2024; Bays et al., 2024). We thus interpret representational quality as representational uncertainty, or the level of noise in memory encoding. As shown in Figure 1a, we argue that greater resources result in lower representational uncertainty, an assumption entertained by more and more recent studies (e.g., Ma et al., 2006; Bays et al., 2009; Ma et al., 2014; Bates and Jacobs, 2020; Bays et al., 2024; Xu and Futrell, 2026). Our conceptualization of WM resource constraint attempts to address the first question raised at the beginning of this section. That is, we aim to understand the relationship between memory constraints and the content of information encoded in memory, and we offer a computational-level analysis of this question (Marr, 1982). This does not stand in opposition to retrieval-focused work, in that the emergence of retrieval bias may be shaped by the representational quality of what is encoded in the first place. We revisit this point in Section 7.

人类的记忆表征是带噪声的，因为接收到的信息在神经系统的传递中总会受到一定程度、且不可避免的扰动（例如 Ma et al., 2014; Futrell et al., 2020; Hahn et al., 2022; Brady et al., 2024; Bays et al., 2024）。我们因而把表征质量理解成表征不确定性，也就是记忆编码中的噪声水平。如图 1a 所示，我们认为资源越充足，表征不确定性就越低；这一假定正被越来越多的近期研究采用（例如 Ma et al., 2006; Bays et al., 2009; Ma et al., 2014; Bates and Jacobs, 2020; Bays et al., 2024; Xu and Futrell, 2026）。我们对工作记忆资源约束的构想，意在回答本节开头提出的第一个问题，也就是弄清记忆约束与编码进记忆的信息内容之间的关系，并对此给出计算层分析（Marr, 1982）。这并不与聚焦检索的工作相对立：检索偏向的出现，可能首先受到已编码内容之表征质量的塑造。我们将在第 7 节再回到这一点。

2.2 Efficient use of limited WM resources

2.2 有限工作记忆资源的高效使用

What is the optimal strategy to efficiently use limited WM resources in language processing? The answer to this question requires an explicitly defined processing task (Anderson, 1990), and implies a trade-off between the task performance and the amount of available resources (Simon, 1955; Lewis et al., 2014; Gershman et al., 2015; Lieder and Griffiths, 2020). In other words, as a resource-rational language user, the goal is to find what kinds of representation for the encoding of linguistic inputs require the least amount of resources yet good enough to accomplish certain processing task. Following Hahn et al. (2022), we take next-word prediction as one such processing task. Theoretically, next-word prediction has been argued to be a causal bottleneck for the processing of both lexical and structural features (Hale, 2001; Levy, 2008). Empirically, the great success of language modeling suggests that complex linguistic features can emerge simply from the prediction task (Mahowald et al., 2024; Futrell and Mahowald, 2025). In the resource-rational model of Hahn et al. (2022), inputs in the past are assigned a probability of erasure dependent on their lexical identity and the distance from the current time point. The model learns how to assign this erasure probability to maximize the performance of next-word prediction, subject to a resource limit on how many words are retained. However, this binary erasure in their model (either erased or not) does not capture the intuition that the representation of a past input is often subtly distorted rather than being entirely forgotten.

在语言加工中，高效使用有限工作记忆资源的最优策略是什么？回答这个问题，需要明确指定一项加工任务（Anderson, 1990），并意味着任务表现与可用资源数量之间存在权衡（Simon, 1955; Lewis et al., 2014; Gershman et al., 2015; Lieder and Griffiths, 2020）。换句话说，作为资源理性的语言使用者，目标是找到这样的表征：编码语言输入所需的资源尽量少，却又足够好地完成特定加工任务。沿用 Hahn et al. (2022)，我们把下一个词预测当作这样一项加工任务。在理论上，下一个词预测被论证为词汇特征与结构特征加工的因果瓶颈（Hale, 2001; Levy, 2008）。在经验上，语言建模的巨大成功表明，复杂的语言特征可以单单从预测任务中涌现出来（Mahowald et al., 2024; Futrell and Mahowald, 2025）。在 Hahn et al. (2022) 的资源理性模型里，过去的输入会按其词汇身份以及与当前时刻的距离，被赋予一个擦除概率。模型学习如何分配这一擦除概率，以便在“能保留多少词”这一资源上限之下，尽量提高下一个词预测的表现。不过，他们模型中的二元擦除（要么擦掉，要么保留）并不能捕捉这样一种直觉：过去输入的表征往往只是被轻微扭曲，而并非被彻底遗忘。

[note] One way to view cognitive resources is in terms of the number of perceptual samples available: more perceptual samples are needed to form a more precise encoding distribution (Norris, 2006). Moreover, at the neural level, from the perspective of probabilistic population coding, the overall amplitude of the population activity is inversely proportional to the variance of the represented distribution (Ma et al., 2006), raising a link between the cognitive resources and the lower-level biological energy. Our use of the term “computational-level” slightly deviates from the one posited by Marr (1982), and our approach is situated at the boundary between the computational and algorithmic levels under Marr’s three-level framework. Following the resource-rational approach to cognition (Section 2.2), we view our model as going beyond the computational level (i.e., idealized processing model shaped only by the cognitive function), pushing the theorizing towards the algorithmic level by considering more and more realistic cognitive architecture and resources (Lieder and Griffiths, 2020).

注：看待认知资源的一种方式，是看可用的知觉样本有多少：要形成更精确的编码分布，就需要更多知觉样本（Norris, 2006）。此外，在神经层面，从概率群体编码的角度看，群体活动的总体幅度与所表征分布的方差成反比（Ma et al., 2006），从而把认知资源与更底层的生物能量联系起来。我们使用的“计算层”一语，与 Marr (1982) 的原义略有偏离；按 Marr 的三层框架，我们的做法处在计算层与算法层的交界。沿用认知的资源理性路径（第 2.2 节），我们认为本模型已越过计算层（即仅由认知功能塑造的理想化加工模型），通过纳入越来越贴近现实的认知架构与资源，把理论推向算法层（Lieder and Griffiths, 2020）。

2.3 The current study

2.3 本研究

To model the link between representational uncertainty and WM resources, we inject Gaussian noise into self-attention value vectors of Transformers. This method lets us manipulate WM constraints: models with greater WM resources have a lower level of noise, allowing them to more precisely encode linguistic inputs. Compared to binary noise manipulations (Futrell et al., 2020; Hahn et al., 2022), Gaussian noise on the distributional representational space of Transformers implements more subtle perturbations on the encoded information. From an information-theoretic perspective, additive white Gaussian noise (AWGN) is a commonly used noise model for communication channels, whose capacity is determined by the signal-to-noise power ratio (SNR). The Gaussian noise we inject into Transformers can therefore be interpreted in terms of modifying the SNR under which the information in each layer is transmitted to the next. In neuroscience, AWGN channels are also widely used to model information encoding in neurons (Stone, 2018). In addition to Gaussian noise injection, we adopt a hybrid training objective: models learn to maximize the performance of next-word prediction with explicit penalty on the model’s encoding precision, so that they learn to strike a balance on the trade-off between prediction and memory (Still, 2014; Hahn and Futrell, 2019).

为刻画表征不确定性与工作记忆资源之间的联系，我们向 Transformer 自注意力的价值向量注入高斯噪声。这一方法使我们能够调控工作记忆约束：工作记忆资源更充足的模型噪声更低，从而能更精确地编码语言输入。与二元噪声操作（Futrell et al., 2020; Hahn et al., 2022）相比，在 Transformer 的分布表征空间上施加高斯噪声，会对已编码信息造成更细微的扰动。从信息论的角度看，加性白高斯噪声（AWGN）是通信信道常用的噪声模型，其容量由信噪功率比（SNR）决定。因此，我们注入 Transformer 的高斯噪声，可以理解成在改变各层信息向下一层传递时所处的信噪比。在神经科学中，AWGN 信道也被广泛用来为神经元中的信息编码建模（Stone, 2018）。除注入高斯噪声外，我们还采用混合训练目标：模型在下一个词预测上尽量提高表现，同时又对编码精度施加显式惩罚，从而在预测与记忆的权衡中学会取得平衡（Still, 2014; Hahn and Futrell, 2019）。

3 Modeling framework

3 建模框架

3.1 Injecting noise into self-attention

3.1 在自注意力中注入噪声

As illustrated in Figure 1b, we modify the self-attention mechanism (Vaswani et al., 2017) in Transformer-based LMs such that the noisy version of the value vector of the token ˜v ∈R at a previous key position k given the query position q is sampled from a Gaussian distribution
(1) ˜v ∼N v, τI , where τ is the encoding precision, which is the inverse of variance 1/σ. We assume an exponential decay on the encoding precision τ, such that the precision of a key token k is a function of its distance with the query position q and is controlled by the decay rate θ (2) τ = τ e, where τ is the highest achievable precision in memory encoding. The maximal precision τ is a hyperparameter manually specified. The specification in Eq. 2 implies that linguistic inputs can never be perfectly encoded without any noise, and τ is the base level of irreducible noise, which in our model is specified to be encoding precision of the token at query position itself (when q −k = 0). This imperfect encoding is not necessarily a flaw in our modeling strategy in that representations are always noisy during neural transmission even at the perceptual level as mentioned above.

如图 1b 所示，我们修改基于 Transformer 的语言模型中的自注意力机制（Vaswani et al., 2017），使得在给定查询位置 $q$ 时，此前键位置 $k$ 上词的价值向量之带噪版本 $\tilde{v}_{qk} \in \mathbb{R}^{d}$，从如下高斯分布中采样：

$\tilde{v}_{qk} \sim \mathcal{N}\!\left(v_{qk},\, \tau^{-1} I\right), \quad (1)$

其中 $\tau$ 是编码精度，也就是方差的倒数 $1/\sigma^{2}$。我们假定编码精度 $\tau$ 按指数衰减，从而使键词 $k$ 的精度成为它与查询位置 $q$ 之距离的函数，并由衰减速率 $\theta$ 控制：

$\tau_{qk} = \tau_{0}\, e^{-\theta (q-k)}, \quad (2)$

其中 $\tau_{0}$ 是记忆编码所能达到的最高精度。最大精度 $\tau_{0}$ 是手工设定的超参数。式 (2) 的设定意味着语言输入永远无法在完全无噪声的情况下被完美编码；$\tau_{0}$ 对应不可消除噪声的基线水平。在我们的模型里，它被规定为查询位置上该词自身的编码精度（当 $q-k=0$ 时）。这种不完美的编码并不必然是建模策略上的缺陷，因为如前所述，即便在知觉层面，表征在神经传递中也总是带有噪声。

The output z of each attention head at query position q is thus a weighted sum over noisy value vectors of all previous key positions k X (3) a˜v, z = whose weights a are given by  Kq (4) √ . a = softmax d We then apply a reparameterization for this weighted sum of Gaussian distributions for efficient stochastic optimization. The attention head output A small value of 0.0001 is added to every τ in implementation to avoid numerical underflow. As mentioned earlier, what actually matters for WM constraint is the signal-to-noise ratio (SNR), which is currently manipulated by τ assuming constant power of the signal. Future work can consider manipulating SNR more directly. z is decomposed into two components, one deterministic and one capturing the Gaussian noise: (5) + ˜z. z = z The deterministic part z is computed as the standard self-attention as if there were no noise. The noisy part is a unit Gaussian distribution ϵ ∼N (0, I) scaled by the standard deviation σ of ˜z: (6) ˜z = σϵ, Assuming that the noise components of all previous tokens are independent of each other, the variance of ˜z after rescaling is X X (aσ) = (7) a τ . = σ We inject noise into self-attention instead of the output embedding of Transformer blocks for two reasons. First, theoretically, the mechanism of self-attention better resembles WM processes, where input representations at different time points communicate with each other. Second, for our purpose, the noisy representation of a token depends on its position relative to the query position of memory retrieval. This design is naturally realized by self-attention with parallelized computation.

因此，各注意力头在查询位置 $q$ 上的输出 $z$，是此前所有键位置 $k$ 上带噪价值向量的加权和：

$z_{q} = \sum_{k} a_{qk}\tilde{v}_{qk}, \quad (3)$

其权重 $a$ 由下式给出：

$a_{q} = \mathrm{softmax}\!\left(\dfrac{K^{\top} q}{\sqrt{d}}\right). \quad (4)$

为了对这一高斯分布的加权和做高效的随机优化，我们再做一次重参数化。注意力头的输出 $z$ 被分解为两部分，一部分是确定性的，另一部分捕捉高斯噪声：

$z_{q} = z^{\mathrm{det}}_{q} + \tilde{z}_{q}. \quad (5)$

注：实现时会给每个 $\tau_{qk}$ 加上一个很小的值 0.0001，以避免数值下溢。如前所述，对工作记忆约束真正起作用的是信噪比（SNR）；目前是在假定信号功率恒定的前提下，通过 $\tau$ 来调控。未来工作可以考虑更直接地调控信噪比。

确定性部分 $z^{\mathrm{det}}$ 按标准自注意力计算，相当于没有噪声。噪声部分则是由 $\tilde{z}$ 的标准差 $\sigma$ 缩放的单位高斯分布 $\epsilon \sim \mathcal{N}(0, I)$：

$\tilde{z}_{q} = \sigma_{q}\epsilon, \quad (6)$

假定此前所有词的噪声分量彼此独立，则缩放后 $\tilde{z}_{q}$ 的方差为：

$\sigma_{q}^{2} = \sum_{k}(a_{qk}\sigma_{qk})^{2} = \sum_{k} a_{qk}^{2}\tau_{qk}^{-1}. \quad (7)$

我们把噪声加在自注意力上，而不是 Transformer 块的输出嵌入上，出于两点考虑。第一，在理论上，自注意力机制更接近工作记忆过程：不同时刻的输入表征会彼此交流。第二，就我们的目的而言，一个词的带噪表征取决于它相对于记忆检索查询位置的位置。这一设计可以由带并行计算的自注意力自然实现。

3.2 Manipulating WM capacity constraint

3.2 调控工作记忆容量约束

In order to manipulate WM constraints, we train the models with a hybrid loss function: X X X L = − log p (y | y) + λ τ z , | {z } | {z } (8) where z is a normalization constant given by z = L H T (T −1) (9) .

为调控工作记忆约束，我们用如下混合损失函数训练模型：

$L = -\sum_{y \in S}\sum_{t=1}^{T}\log p(y_{t}\mid y_{<t}) + \lambda \sum_{hltk}\tau_{hltk}\, z^{-1}, \quad (8)$

其中第一项为交叉熵损失，第二项为记忆容量项；$z$ 是归一化常数，由下式给出：

$z = \dfrac{L H T (T-1)}{2}. \quad (9)$

This loss function consists of two components. The first is the cross-entropy loss, which measures the model’s performance in the next-word prediction task on training examples S. The second component represents the average encoding precision τ of past tokens over all attention heads h and layers l, which corresponds to the amount of WM resources. WM constraint is represented by λ: higher λ means stronger constraint, applying stronger penalty to models with higher encoding precision.

该损失函数由两部分组成。第一项是交叉熵损失，衡量模型在训练样本 $S$ 上下一个词预测任务中的表现。第二项表示在所有注意力头 $h$ 和层 $l$ 上，过去词的平均编码精度 $\tau$，对应所消耗的工作记忆资源。工作记忆约束由 $\lambda$ 表示：$\lambda$ 越大，约束越强，对编码精度更高的模型施加的惩罚也更重。

4 Experiment 1: Memory capacity and encoding precision decay

4 实验 1：记忆容量与编码精度衰减

4.1 Language Models

4.1 语言模型

We train on the Wikitext-103 dataset (Merity et al., 2016) a sequence of five noisy LMs adapted from the architecture of GPT-2 small (Radford et al., 2019), corresponding to five WM capacity constraints λ = [0, 0.001, 0.01, 0.1, 1]. The model architecture is the same as GPT-2 small, except that the block size is 512 (instead of 1024 in standard GPT-2 small) and that the self-attention heads include noisy encoding as outlined in Section 3.1. Each attention head at each layer gets its own precision decay parameter θ to be learned from training data. The highest achievable encoding precision (τ in Eq. 2) is set to 10000 (equivalent to σ = 0.01). See Appendix A for model training details.

我们在 Wikitext-103 数据集（Merity et al., 2016）上训练一组共五个带噪语言模型，架构改编自 GPT-2 small（Radford et al., 2019），分别对应五种工作记忆容量约束 $\lambda = [0, 0.001, 0.01, 0.1, 1]$。模型架构与 GPT-2 small 相同，只是块大小为 512（标准 GPT-2 small 为 1024），并且自注意力头包含第 3.1 节所述的带噪编码。每一层的每一个注意力头都有自己的精度衰减参数 $\theta$，由训练数据学习得到。可达到的最高编码精度（式 (2) 中的 $\tau_{0}$）设为 10000（相当于 $\sigma = 0.01$）。模型训练细节见附录 A。

4.2 Results and discussion

4.2 结果与讨论

Figure 1c shows the precision decay rate θ learned by each attention head per layer. With no constraint on WM capacity (λ = 0), all heads learn a decay rate close to 0. Intuitively, this is consistent with the memory-surprisal trade-off proposed in Hahn et al. (2021), in the sense that it is useful to use all the information from the preceding context to maximize the accuracy of next-word prediction. Importantly, as WM constraint gets stronger, there is a clear pattern of asymmetrical decay. With lower-to-mid memory constraint (0 < λ ≤0.01), precision decay first begins among lower-layer attention heads, which often take charge of processing morphosyntactic features where information dependency is more local (Vig and Belinkov, 2019). Then, when WM constraint further increases (λ > 0.01), the decay starts to diffuse to other heads at higher layers, which is often considered to process longer discourse with deeply contextualized representations (Kuribayashi et al., 2025). Models and code are available at: https://github.com/weijiexu-charlie/resource-rational-encoding. Our implementation is based on nanoGPT (Karpathy, 2023), whose backbone reflects the architecture of GPT-2.. This prioritization of higher layers mirrors humans’ memory strategy, in the sense that humans tend to remember the semantic or discourse gist rather than the exact surface form (Sachs, 1974).

图 1c 展示了各层每一个注意力头学到的精度衰减速率 $\theta$。在工作记忆容量不受约束时（$\lambda = 0$），所有头学到的衰减速率都接近 0。直观上，这与 Hahn et al. (2021) 提出的记忆–surprisal 权衡一致：为尽量提高下一个词预测的准确性，把前文的全部信息都用上是有益的。重要的是，随着工作记忆约束变强，出现了清晰的不对称衰减模式。在偏低到中等的记忆约束下（$0 < \lambda \leq 0.01$），精度衰减首先出现在较低层的注意力头上；这些头往往负责加工形态句法特征，其信息依赖也更局部（Vig and Belinkov, 2019）。当约束进一步加大（$\lambda > 0.01$）时，衰减开始扩散到更高层的其他头；这些更高层通常被认为在处理更长的语篇，并形成深度语境化的表征（Kuribayashi et al., 2025）。模型和代码见：https://github.com/weijiexu-charlie/resource-rational-encoding。我们的实现基于 nanoGPT（Karpathy, 2023），其主干反映的是 GPT-2 的架构。这种对更高层的优先保留，接近人类的记忆策略：人更常记住语义或语篇大意，而不是原句的精确表层形式（Sachs, 1974）。

5 Experiment 2: Psychometric predictive power on human reading times

5 实验 2：对人类阅读时间的心理测量预测力

Experiment 2 examines the alignment between our models and human reading behaviors. We assume that models better aligning with human reading behaviors should have higher psychometric predictive power for its surprisal estimates on reading times (RTs). Following previous work, the predictive power is quantified as the difference of log-likelihood ∆LL between regression models fit to RTs with and without surprisal terms (Wilcox et al., 2023; Xu et al., 2023; Frank and Bod, 2011). Higher ∆LL indicates better alignment.

实验 2 考察我们的模型与人类阅读行为的对齐程度。我们假定：与人类阅读行为对齐更好的模型，其 surprisal 估计对阅读时间（RT）应有更高的心理测量预测力。沿用已有工作，预测力量化为：带与不带 surprisal 项的回归模型在拟合阅读时间时对数似然之差 $\Delta$LL（Wilcox et al., 2023; Xu et al., 2023; Frank and Bod, 2011）。$\Delta$LL 越高，对齐越好。

5.1 Experiment setup

5.1 实验设置

We evaluate the model predictive power on three English reading-time corpora: (1) Provo (Luke and Christianson, 2018): An eye-tracking corpus with 55 short passages collected from 84 participants. (2) SPR Natural Stories Corpus (SPRNSC) (Futrell et al., 2021): A self-paced reading corpus with 10 long stories (approx 1000 words each) collected from 181 participants. (3) A-Maze Natural Stories (MazeNSC) (Boyce and Levy, 2023): Same text material as SPRNSC but collected from 100 participants using A-Maze paradigm (Boyce et al., 2020).

我们在三个英语阅读时间语料上评估模型的预测力：（1）Provo（Luke and Christianson, 2018）：眼动语料，含 55 篇短文，采集自 84 名被试。（2）SPR Natural Stories Corpus（SPRNSC）（Futrell et al., 2021）：自定步进阅读语料，含 10 篇长故事（每篇约 1000 词），采集自 181 名被试。（3）A-Maze Natural Stories（MazeNSC）（Boyce and Levy, 2023）：文本材料与 SPRNSC 相同，但用 A-Maze 范式从 100 名被试处采集（Boyce et al., 2020）。

RT measures. For Provo, we examine model predictive power on two eye-tracking RT measures separately, namely the first fixation and the total time. We exclude eye-tracking RTs that are faster than 100ms or slower than 2000ms. For SPRNSC, due to the strong spillover effect in the self-paced reading paradigm, we use RTs of both the critical region and the spillover region. For MazeNSC, we focus on the critical region, and exclude RT observations where participants chose a wrong word in their first try. We also exclude RTs that are shorter than 100ms or longer than 3000ms in MazeNSC. The NSC corpus intentionally includes rare constructions. The duration of the first fixation on the word of interest. The sum of durations of all fixations on a word. We adopt a wider RT exclusion window for MazeNSC, since the A-Maze paradigm often induces longer RTs.

阅读时间指标。对 Provo，我们分别考察模型对两项眼动阅读时间指标的预测力，即首次注视和总注视时间。我们剔除快于 100ms 或慢于 2000ms 的眼动阅读时间。对 SPRNSC，由于自定步进阅读范式存在较强的溢出效应，我们同时使用关键区和溢出区的阅读时间。对 MazeNSC，我们关注关键区，并剔除被试第一次选错词的阅读时间观察；同时剔除短于 100ms 或长于 3000ms 的阅读时间。注：NSC 语料有意纳入了罕见构式。首次注视指对目标词的第一次注视持续时间。总注视时间指落在某一词上的全部注视持续时间之和。我们对 MazeNSC 采用更宽的阅读时间剔除窗口，因为 A-Maze 范式往往会使阅读时间更长。

Figure 2: a. Psychometric predictive power of word surprisal estimates on RTs (measured as ∆LL) across WM constraints; Bar height is the mean ∆LL over 10 held-out folds with standard errors; Dashed lines are the result of the baseline GPT-2. b. Model perplexity on RT corpora. c. Distribution of surprisal estimates over six bins for RT corpora. d. Attention score over key tokens as a function of query-key distance, aggregated over all query tokens, attention heads and layers; passages in Provo corpus have much shorter context than NSC corpus.

图 2：a. 词 surprisal 估计对阅读时间的心理测量预测力（以 $\Delta$LL 衡量）随工作记忆约束变化；柱高为 10 个留出折上 $\Delta$LL 的均值，并给出标准误；虚线是基线 GPT-2 的结果。b. 模型在阅读时间语料上的困惑度。c. 阅读时间语料上 surprisal 估计在六个区间上的分布。d. 注意力分数随查询–键距离的变化，对所有查询词、注意力头和层聚合；Provo 语料中的段落上下文远短于 NSC 语料。

Surprisal estimates. Following Goldstein et al. (2022), we use a sliding window paradigm to generate the surprisal estimate of each word in RT corpora, with the maximal context window (511 preceding tokens). In addition to the five noisy LMs trained in Experiment 1, we also train a baseline GPT-2 model on the same dataset without noise injection. All other hyperparameters and training procedures are identical to the noisy models. We examine the predictive power of all six models.

surprisal 估计。沿用 Goldstein et al. (2022)，我们用滑动窗口范式为阅读时间语料中的每个词生成 surprisal 估计，最大上下文窗口为 511 个前文词元。除实验 1 中训练的五个带噪语言模型外，我们还在同一数据集上训练了一个不加噪声的基线 GPT-2。其余超参数和训练程序与带噪模型相同。我们考察全部六个模型的预测力。

Regression models. In order to get ∆LL, for each model, we fit two linear mixed-effects regression models (Baayen et al., 2008) to predict RTs: M: Null model including word position in the sentence, word length, and log-transformed word frequency from Speer (2022) M: M plus terms of word surprisal To account for the spillover effect typically observed in reading experiments, we also include the word length, word frequency, and the surprisal of two previous words as control predictors. The predictive power is thus measured as the difference of log-likelihood ∆LL between $M_0$ and $M_1$. Regression models are fit using 10-fold cross validation, with ∆LL calculated from the held-out set.

回归模型。为得到 $\Delta$LL，对每个模型我们拟合两个线性混合效应回归模型（Baayen et al., 2008）来预测阅读时间：

$M_{0}$：空模型，包含词在句中的位置、词长，以及 Speer (2022) 给出的对数词频

$M_{1}$：在 $M_{0}$ 之上加入词 surprisal 项

为控制阅读实验中常见的溢出效应，我们还把前两个词的词长、词频和 surprisal 作为控制预测变量纳入。预测力因而衡量为 $M_{0}$ 与 $M_{1}$ 之间的对数似然之差 $\Delta$LL。回归模型用十折交叉验证拟合，$\Delta$LL 在留出集上计算。

5.2 Results and discussion

5.2 结果与讨论

Better alignment with human RTs with WM constraint. As shown in Figure 2a, explicit WM constraint in principle can improve the model-generated surprisals’ psychometric predictive power on human RTs. This is especially true for the SPR NSC corpus, where an enhanced predictive power is observed with moderate WM constraint (λ = 0.01) compared to the baseline GPT-2 without noise injection. We also observe a mild increasing trend of predictive power for the first-fixation durations of the Provo corpus. Moreover, as indicated by Figure 2b, models with explicit WM constraint yields higher perplexity on RT corpora, echoing recent findings where models that are exceedingly accurate in next-word predictions tend to show worse alignment with human reading behaviors (Oh and Schuler, 2023; Oh and Linzen, 2025). It is also worth noting that we only include random intercept for all regression models, which is the maximal random structure that allows all models to converge without singular fit. Note that the length of each passage in the Provo corpus is much shorter than in the NSC corpus, which may potentially explain the divergent pattern of predictive power between these two corpora.

带工作记忆约束时，与人类阅读时间的对齐更好。如图 2a 所示，显式的工作记忆约束原则上可以提高模型生成的 surprisal 对人类阅读时间的心理测量预测力。这一点在 SPR NSC 语料上尤其明显：与不加噪声的基线 GPT-2 相比，中等约束（$\lambda = 0.01$）带来了更高的预测力。在 Provo 语料的首次注视上，我们也观察到预测力有温和上升的趋势。此外，如图 2b 所示，带显式工作记忆约束的模型在阅读时间语料上困惑度更高，这与近期发现相呼应：下一个词预测过准的模型，往往与人类阅读行为对齐更差（Oh and Schuler, 2023; Oh and Linzen, 2025）。还需要说明的是，所有回归模型都只纳入了随机截距，这是能让全部模型在不出现奇异拟合的情况下收敛的最大随机结构。另外，Provo 语料中每篇段落的长度远短于 NSC 语料，这或许有助于解释两套语料上预测力模式的分歧。

Figure 3: a. Generating the representational space of predictions from next-word log probabilities over the vocab; dimensionality reduced using PCA (50 PCs). b. Representational space across WM constraints, visualized from the first 2 PCs; each point represents an item of noun phrase (labels include Noun1 and Prep). c. Mean Euclidean distance (based on all 50 PCs) with standard error, over all pairs of items and within each animacy category.

图 3：a. 用词表上下一个词的对数概率生成预测的表征空间；再用 PCA 降维（50 个主成分）。b. 不同工作记忆约束下的表征空间，由前 2 个主成分可视化；每个点表示一个名词短语项目（标签含 Noun1 和 Prep）。c. 基于全部 50 个主成分的平均欧氏距离及标准误，计算所有项目对以及各生命性类别内部。

It is also worth noting that we do not expect the RT alignment to necessarily increase monotonically as a function of WM constraint, since realistically human WM must not be infinitely constrained, a point we revisit in the Limitations section.

还需要说明的是，我们并不预期阅读时间对齐必然随工作记忆约束单调增强，因为现实中人类工作记忆不可能受到无限约束。这一点我们会在“局限”一节再谈。

Less extreme surprisal estimates with memory constraint. As shown in Figure 2c, the model generates fewer extreme surprisal estimates at the lower end with stronger memory constraint λ. Instead, more and more estimates with higher λ are at the moderate level [5, 15). We speculate that the memory constraint may have encouraged the model to adopt a processing strategy that focuses more on making next-word predictions based on general semantic categories rather than on fine-grained distinctions, a hypothesis that we explore in Section 6.

记忆约束下，极端 surprisal 估计更少。如图 2c 所示，记忆约束 $\lambda$ 越强，模型在低端产生的极端 surprisal 估计就越少。取而代之的是，随着 $\lambda$ 增大，越来越多的估计落在中等区间 $[5, 15)$。我们推测，记忆约束可能促使模型采取这样一种加工策略：更多地依据较粗的语义范畴来做下一个词预测，而不是依据细粒度差别。这一假说我们将在第 6 节探讨。

Emergent but dissociable locality bias in attention. Figure 2d shows attention scores on key tokens, aggregated over all heads and layers. Compared to baseline GPT-2, all noisy models have more locally biased attention, with higher weights on local tokens. However, this locality bias does not monotonically increase in WM constraint λ. Although moderate constraint (λ = 0.001) indeed encourages the model to assign higher attention scores on local tokens, this effect is weakened for mid-to-higher constraint (λ ≥ 0.01). (The enhanced predictive power is not observed in eye-tracking total time, nor in Maze RT. For total time, the lack of a clear pattern is possibly because total time mixes early and late (re-)reading of a word, potentially indexing multiple different underlying processes (Clifton Jr et al., 2007; Schotter and Dillon, 2025). For A-Maze, this experimental paradigm is typically memory demanding, which potentially explains the reduced predictive power in Maze RT.) We speculate that the poorer representational quality due to faster precision decay may have encouraged the model to more uniformly distribute its attention weights as it becomes less certain about which token to attend to in the past context. This pattern potentially points to a dissociation between WM encoding and retrieval, suggesting that stronger WM resource constraint does not necessarily lead to more locally biased retrieval in all situations.

注意力中涌现、却可与编码相分离的局部偏向。图 2d 展示了对所有头和层聚合后、落在键词上的注意力分数。与基线 GPT-2 相比，所有带噪模型的注意力都更偏向近处，局部词上的权重更高。不过，这种局部偏向并不随工作记忆约束 $\lambda$ 单调增强。中等偏弱的约束（$\lambda = 0.001$）的确促使模型给局部词分配更高的注意力分数，但到了中等到更强的约束（$\lambda \geq 0.01$），这一效应反而减弱。（在眼动总注视时间和 Maze 阅读时间上，并未观察到预测力的提升。对总注视时间而言，模式不清晰，可能是因为它混合了对一个词的早期注视和后期（再）阅读，从而可能对应多种不同的底层过程（Clifton Jr et al., 2007; Schotter and Dillon, 2025）。对 A-Maze 而言，这一实验范式通常对记忆要求较高，或许可以解释 Maze 阅读时间上预测力的下降。）我们推测，精度衰减更快所导致的表征质量下降，可能促使模型把注意力权重铺得更均匀，因为它对过去上下文中该关注哪一个词变得更不确定。这一模式或许指向工作记忆编码与检索之间的分离：更强的工作记忆资源约束，并不必然在所有情形下都带来更偏向近处的检索。

6 Experiment 3: Representational space of next-word predictions

6 实验 3：下一个词预测的表征空间

Previous studies have argued that LMs tend to make sharper predictions than humans with more fine-grained distinctions across contextual inputs, possibly due to the models’ powerful superhuman memory (Oh and Linzen, 2025). This raises the expectation that models with more human-like constraints should make predictions that are less fine-grained, reflecting broader semantic categories. In Experiment 3, we aim to examine this claim, looking at how WM constraint shapes the model’s representational space when making next-word predictions. We hypothesize that stronger WM constraint may result in: (i) more compressed representational space, such that different linguistic inputs are encoded more similarly, with more similar linguistic predictions across contexts; (ii) stronger categorical bias, such that different categories of certain linguistic feature (e.g., animate vs. inanimate) are represented more distinctly.

已有研究认为，语言模型往往比人类做出更尖的预测，并在不同上下文输入之间做出更细的区分，这可能是因为模型拥有超乎人类的强大记忆（Oh and Linzen, 2025）。由此可以预期，带有更接近人类之约束的模型，其预测应不那么细碎，而更能反映较宽的语义范畴。在实验 3 中，我们想检验这一主张，观察工作记忆约束如何塑造模型在做下一个词预测时的表征空间。我们的假说是，更强的工作记忆约束可能导致：（i）更压缩的表征空间，使不同的语言输入被编码得更相似，不同上下文中的语言预测也更接近；（ii）更强的范畴偏向，使某一语言特征的不同类别（例如有生命 vs. 无生命）被表征得更加分明。

6.1 Experiment setup

6.1 实验设置

As in Figure 3a, we first construct a set of 1000 items in English in the form of Det Noun1 Prep Det Noun2 Adv. The animacy of Noun1 is manipulated such that 500 items are animate and the other 500 are inanimate. We manipulate the animacy feature for two reasons. First, it has been shown to actively serve as a processing cue in language comprehension (Trueswell et al., 1994). Second, animacy is grammatically marked on pronominal elements in English. Therefore, we consider the animacy feature a potential candidate for the categorical bias to emerge with memory constraint. Then, using the models trained in Experiment 1, the initial representational space of next-word prediction is constructed from the log probabilities over possible continuations w generated at the Adv. In the end, we reduce the dimensionality on this initial representational space using Principle Component Analysis (PCA). The ultimate representational space of next-word predictions is taken from the first 50 PCs. In order to evaluate this representational space, for each pair of items i and j, we calculate their Euclidean distance d given by q (10) (x −x)(x −x). d = Higher item-to-item distance thus indicates that their next-word predictions are less similar.

如图 3a 所示，我们首先构造一组 1000 条英语项目，形式为“限定词–名词1–介词–限定词–名词2–副词”。我们对名词1 的生命性做了操纵：500 条有生命，另外 500 条无生命。操纵生命性特征有两点考虑。第一，已有研究表明，生命性在语言理解中会积极充当加工线索（Trueswell et al., 1994）。第二，在英语中，生命性会在代词性成分上得到语法标记。因此，我们认为生命性特征有可能在记忆约束下显现出范畴偏向。随后，使用实验 1 中训练的模型，下一个词预测的初始表征空间由副词位置上对可能后续词 $w$ 给出的对数概率构成。最后，我们用主成分分析（PCA）对这一初始表征空间降维，并把前 50 个主成分作为最终的下一个词预测表征空间。为评估该表征空间，对每一对项目 $i$ 和 $j$，我们计算其欧氏距离 $d_{ij}$：

$d_{ij} = \sqrt{(x_{i}-x_{j})^{\top}(x_{i}-x_{j})}. \quad (10)$

项目之间的距离越大，说明它们的下一个词预测越不相似。

6.2 Results and discussion

6.2 结果与讨论

More compressed representational space with higher WM constraint. Figure 3b shows the representational space of next-word predictions visualized from the first 2 PCs, and Figure 3c summarizes the item-to-item Euclidean distance based on all 50 PCs. Without explicit WM constraint (λ = 0), the representational space is more spread out, with higher item-to-item distance, indicating that next-word predictions are relatively distinct from each other across different items. (Noun2 and Adv are kept identical across all items so these tokens more local to the prediction site do not mask the effect of our manipulation on earlier ones. Similar to Experiment 2, here for each item we took the average of 100 samples of log probability distributions. It is worth noting that this is not the internal representational space encoded by the model. Instead, what we look into here is the representational space implied by the language modeling head output distributions for $w_{t+1}$. Small standard error in Fig. 3c due to the large number of item pairs.)

工作记忆约束越强，表征空间越压缩。图 3b 用前 2 个主成分可视化下一个词预测的表征空间，图 3c 则汇总了基于全部 50 个主成分的项目间欧氏距离。在没有显式工作记忆约束时（$\lambda = 0$），表征空间铺得更开，项目间距离更大，表明不同项目的下一个词预测相对彼此有别。（所有项目中的名词2 和副词保持相同，以免更靠近预测位置的这些词元掩盖我们对更早成分所做操纵的效应。与实验 2 类似，这里对每个项目取 100 次对数概率分布样本的平均。需要说明的是，这并不是模型内部编码的表征空间，而是由语言建模头对 $w_{t+1}$ 的输出分布所隐含的表征空间。图 3c 中的标准误很小，是因为项目对的数量很大。）

Figure 4: Mean silhouette scores with standard error for the subject-noun animacy (animate vs. inanimate) and the preposition (temporal vs. spatial) categories.

图 4：主语名词生命性（有生命 vs. 无生命）与介词类别（时间 vs. 空间）的平均轮廓系数及标准误。

As WM constraint gets stronger, the representational space becomes increasingly concentrated with lower item-to-item distance. This reduced representational distance with higher λ suggests that the model’s predictions become more similar to each other across items.

随着工作记忆约束变强，表征空间日益集中，项目间距离下降。$\lambda$ 增大时表征距离缩小，说明模型在不同项目上的预测变得更加彼此相似。

Emergence of categorical encoding. Although a categorical bias is not observed for the animacy feature on the subject noun of our stimuli, interestingly, there are indeed separate clusters of next-word predictions that start to emerge with stronger WM constraint λ, as indicated in Figure 3b (λ ≥0.01). Visual inspection reveals that models start to tease apart noun phrases with prepositions before and after from others. Compared to other prepositions in our stimuli, before and after tend to function as temporal prepositions, which express temporal relations semantically and more naturally take clausal complements syntactically (Dubinsky and Williams, 1995). This emergent categorical encoding is corroborated by the result of silhouette scores in Figure 4, which is a metric that quantifies how well the two categories can be separated from each other in the representational space (see Appendix C). This result indicates that at least some linguistic features (e.g., prepositions in our case) are encoded more categorically with stronger WM constraint, shifting away from fine-grained semantic distinctions towards higher-level generalization (Alvarez, 2011; Bates and Jacobs, 2020; Brady and Tenenbaum, 2013; Zaslavsky et al., 2018). Future work may consider a more thorough investigation on what kinds of linguistic features tend to be categorically encoded under WM constraints. See Appendix D for the PCA visualization with 200 randomly sampled noun-phrase labels.

范畴化编码的出现。虽然在我们刺激的主语名词生命性特征上并未观察到范畴偏向，有意思的是，随着工作记忆约束 $\lambda$ 变强，下一个词预测确实开始出现相互分离的簇，如图 3b 所示（$\lambda \geq 0.01$）。目视检查表明，模型开始把带介词 before 和 after 的名词短语与其他项目区分开来。与刺激中的其他介词相比，before 和 after 更常充当时间介词：在语义上表达时间关系，在句法上也更自然地带小句补足语（Dubinsky and Williams, 1995）。图 4 中的轮廓系数结果支持了这种涌现出的范畴化编码；该指标衡量两个类别在表征空间中能被分开得有多好（见附录 C）。这一结果表明，至少有一部分语言特征（在我们这里是介词）会在更强的工作记忆约束下被更范畴化地编码，从细粒度的语义差别转向更高层的概括（Alvarez, 2011; Bates and Jacobs, 2020; Brady and Tenenbaum, 2013; Zaslavsky et al., 2018）。未来工作可以更系统地考察，在工作记忆约束下，哪些类型的语言特征倾向于被范畴化编码。带 200 个随机抽取的名词短语标签的 PCA 可视化见附录 D。

7 General discussion

7 综合讨论

In this paper, we introduce a conceptualization of WM constraint at the computational level. Focusing on the task nature of information processing, our model characterizes what kinds of representations would be encoded under WM constraint in order to accomplish the processing task in a good enough way. In information-theoretic terms, we view memory as a communicative channel that transmits information from the past to the future, and its constraint as the information bottleneck during this transmission (Still, 2014; Hahn and Futrell, 2019; Bates and Jacobs, 2020). Our model, therefore, is agnostic to how these representational properties and memory efficiency are realized at the algorithmic level. As noted above, one potential algorithmic-level implementation of WM efficiency is locally biased retrieval (De Varda and Marelli, 2024; Clark et al., 2025). Intuitively, this retrieval bias may naturally arise from locally biased encoding precision, in that the model may prioritize the attention to more precisely encoded input. In this sense, our focus on representational quality for WM constraint does not argue against accounts focusing on retrieval bias. Instead, in our view, the emergence of locally biased retrieval may be driven by locally biased encoding precision, as what is retrieved ultimately depends on what is encoded in the first place. However, our result also reveals a dissociation: while moderate WM constraints indeed enhance the locality bias in attention weights, this effect diminishes with more stringent constraint. This dissociation suggests that the locality bias in retrieval mechanism alone is not sufficient to fully characterize WM processes. In order to develop an explanatory account for WM efficiency, it is important to target the representational properties more directly, identifying what kinds of information are encoded and why they are encoded for certain processing tasks. Moreover, recent advances in mechanistic interpretability (Arora et al., 2024), psycholinguistically inspired probing (Hu et al., 2020), as well as our own Experiment 3, all serve as potential methods to examine how the interpretable linguistic features are encoded in our models. We believe that our computational-level manipulation of WM constraints, powered by these methods, provides a promising pathway for analyses on how different interpretable linguistic features are encoded and prioritized for more efficient processing. We believe such an analysis can provide more detailed insight into human memory efficiency, establishing closer connection to traditional linguistic and cognitive theories. Finally, our focus on the representational quality highlights a connection to model quantization (e.g., Dettmers et al., 2022; Frantar et al., 2022; Dettmers et al., 2023; Lin et al., 2024), which can be viewed as another algorithmic-level realization of WM constraint. As a technique developed to reduce GPU memory demand, quantization approximates continuous values by mapping them to a small set of discrete levels, preserving as much task-relevant information as possible despite certain degree of degradation of precision. From this perspective, stronger WM constraint corresponds to more aggressive quantization: more fine-grained distinctions in the original continuous space are lost, making it increasingly difficult to reconstruct detailed linguistic representations. This connection to quantization thus points to a promising route for translating WM constraint, a classic notion in cognitive psychology, to its implementation in the engineering of human-like artificial intelligent systems.

本文在计算层上提出一种对工作记忆约束的构想。着眼于信息加工的任务性质，我们的模型刻画的是：在工作记忆约束下，会编码出什么样的表征，才能足够好地完成加工任务。用信息论的话说，我们把记忆看成一条把信息从过去传到未来的通信信道，并将其约束看成这一传递过程中的信息瓶颈（Still, 2014; Hahn and Futrell, 2019; Bates and Jacobs, 2020）。因此，我们的模型并不预先规定这些表征性质与记忆效率在算法层上如何实现。如前所述，工作记忆效率在算法层上的一种可能实现，是带局部偏向的检索（De Varda and Marelli, 2024; Clark et al., 2025）。直观上，这种检索偏向可能自然来自编码精度上的局部偏向：模型或许会优先把注意力投向编码得更精确的输入。就此而言，我们强调工作记忆约束的表征质量，并不反对聚焦检索偏向的解释。相反，在我们看来，局部偏向检索的出现，可能正是由编码精度上的局部偏向所驱动的，因为最终能被检索到的，首先取决于当初被写成了什么样。不过，我们的结果也揭示出一种分离：中等的工作记忆约束的确增强了注意力权重上的局部偏向，但约束再收紧时，这一效应反而减弱。这种分离表明，单靠检索机制中的局部偏向，并不足以完整刻画工作记忆过程。若要为工作记忆效率给出解释性说明，就需要更直接地对准表征性质，弄清编码的是哪类信息，以及为什么会为特定加工任务编码这些信息。此外，机制可解释性的近期进展（Arora et al., 2024）、受心理语言学启发的探测（Hu et al., 2020），以及我们自己的实验 3，都可以作为潜在方法，用来考察可解释的语言特征如何被编码进模型。我们相信，在这些方法的配合下，对工作记忆约束做计算层操纵，为分析不同可解释语言特征如何被编码、并被优先用于更高效的加工，提供了一条有前景的路径。我们相信，这样的分析能够为人类记忆效率提供更细致的洞察，并与传统的语言学、认知理论建立更紧密的联系。最后，我们对表征质量的关注，也凸显了与模型量化之间的联系（例如 Dettmers et al., 2022; Frantar et al., 2022; Dettmers et al., 2023; Lin et al., 2024）；量化可以看成工作记忆约束在算法层上的另一种实现。作为一项为降低 GPU 显存需求而发展出来的技术，量化把连续值映射到一小集离散水平来做近似，在精度有所下降的情况下，尽可能保留与任务相关的信息。从这一角度看，更强的工作记忆约束对应于更激进的量化：原始连续空间中更细的差别会丢失，要重建细致的语言表征也变得越来越困难。与量化的这一联系，因而指出一条有前景的路线：把工作记忆约束这一认知心理学中的经典概念，转写到类人人工智能系统的工程实现之中。

8 Conclusion

8 结论

In this paper, we manipulate WM constraint at the computational level in a resource-rational fashion, such that models with greater WM resources encode linguistic inputs more precisely. We show that explicit WM constraints improve the model’s alignment with human RTs, a pattern that can be possibly explained by more compressed and categorical representational space in next-word predictions. For psycholinguistics, our work provides a new way to use LLMs as a tool to understand WM processes in human sentence processing, highlighting a dissociation between WM retrieval and the underlying representation of the encoded information. For the engineering of human-like artificial intelligent systems, our conceptualization of WM constraint highlights a connection between this classic notion in psychological theories and algorithmic-level techniques such as quantization.

本文以资源理性的方式，在计算层上调控工作记忆约束，使工作记忆资源更充足的模型能更精确地编码语言输入。我们表明，显式的工作记忆约束能够提升模型与人类阅读时间的对齐；这一模式或许可以由下一个词预测中更压缩、更范畴化的表征空间来解释。对心理语言学而言，我们的工作提供了一种新的途径，把大语言模型用作理解人类句子加工中工作记忆过程的工具，并突出了工作记忆检索与已编码信息之底层表征之间的分离。对类人人工智能系统的工程而言，我们对工作记忆约束的构想，凸显了这一心理学理论中的经典概念与量化等算法层技术之间的联系。

Limitations

局限

Although our theorizing of WM constraint focuses on the representational nature of memory encoding at the computational level, our model is adapted from the architecture of Transformer-based LMs (GPT-2 small specifically), and therefore has committed to the algorithmic-level mechanisms implemented in Transformers. For example, the internal representations of Transformers are computed in a parallel manner, a feature that makes model training more efficient. However, this parallel computation is not truly incremental. In human sentence processing, rather than taking the entire sequence as a whole, linguistic inputs are processed word by word and the memory state is updated incrementally at each time point, a process that resembles the information flow in Recurrent Neural Networks (RNNs) (Elman, 1990; Hochreiter and Schmidhuber, 1997). Although traditional RNNs are not efficient in terms of model training due to its sequential nature, this issue has been addressed by recent advances such as State Space Models (SSMs) (Dao and Gu, 2024), making it possible to efficiently train the model while maintaining its sequential nature. Future work may consider using SSMs to model incremental human sentence processing in a more realistic way. Our model also assumes that the ordinal position relative to the retrieval site is a primitive dimension of attention and decay, an assumption that does not fully align with human sentence processing. Linguistic information is hierarchically structured, where individual units are grouped into constituents that in turn form larger constituents (Chomsky, 1957). As a result, units that are proximate in the linear sequence order can actually take very different hierarchical levels in mental representation. A substantial body of work has established that such hierarchical organization plays a crucial role in language processing (e.g., Caucheteux et al., 2023; Regev et al., 2024; Zhao et al., 2025; Gwilliams et al., 2025). However, in our model, the decay of encoding precision is specified purely as a function of the linear distance between tokens, and therefore is insensitive to the hierarchical structure of linguistic representations. As mentioned in Section 5.2, the predictive power on human RTs does not necessarily increase monotonically as a function of the model’s WM constraint λ, since the WM capacity in humans should still be powerful enough to support sentence processing. Therefore, it is more likely for λ to exhibit the Goldilocks effect, where the predictive power on human RTs peaks at a moderate level of λ (e.g., λ = 0.01 for the SPR NSC corpus). However, there is a lack of linking hypothesis between λ and the actual level of human WM constraint, making it difficult to predict which specific value of λ should best align with human behaviors. In the end, the current work is based on GPT-2 small trained on an English-biased dataset (i.e., Wikitext-103). In order to argue for a domain-general implementation of WM constraints, future work may consider crosslinguistic examination on our models, with more thorough investigation across different model sizes and architectures.

尽管我们对工作记忆约束的理论思考聚焦于计算层上记忆编码的表征性质，模型本身改编自基于 Transformer 的语言模型架构（具体是 GPT-2 small），因而已经承诺了 Transformer 中实现的那些算法层机制。例如，Transformer 的内部表征是并行计算的，这一特性使模型训练更高效。但这种并行计算并不是真正增量的。在人类句子加工中，语言输入并非把整个序列一次性纳入，而是一词一词地加工，记忆状态在每个时刻增量更新；这一过程更接近循环神经网络（RNN）中的信息流动（Elman, 1990; Hochreiter and Schmidhuber, 1997）。传统 RNN 因其序列性质，在模型训练上并不高效，但这一问题已被状态空间模型（SSM）等近期进展所缓解（Dao and Gu, 2024），使模型既能高效训练，又能保持其序列性质。未来工作可以考虑用 SSM 以更贴近现实的方式为增量的人类句子加工建模。我们的模型还假定，相对于检索位置的序次位置是注意力与衰减的一个基元维度；这一假定与人类句子加工并不完全一致。语言信息是按层次组织的：个别单位被组合成成分，这些成分再构成更大的成分（Chomsky, 1957）。因此，在线性序列顺序上邻近的单位，在心理表征中实际上可以处于很不同的层次。大量工作已经表明，这种层次组织在语言加工中起着关键作用（例如 Caucheteux et al., 2023; Regev et al., 2024; Zhao et al., 2025; Gwilliams et al., 2025）。但在我们的模型里，编码精度的衰减被规定为仅仅是词元之间线性距离的函数，因而对语言表征的层次结构并不敏感。如第 5.2 节所述，对人类阅读时间的预测力并不必然随模型的工作记忆约束 $\lambda$ 单调上升，因为人类的工作记忆容量仍须强大到足以支撑句子加工。因此，$\lambda$ 更可能呈现金发姑娘效应：对人类阅读时间的预测力在中等水平的 $\lambda$ 上达到峰值（例如，在 SPR NSC 语料上为 $\lambda = 0.01$）。不过，$\lambda$ 与人类工作记忆约束的实际水平之间还缺少连接假设，因而很难预测哪一个具体的 $\lambda$ 值应与人类行为对齐最好。最后，当前工作基于在偏英语的数据集（即 Wikitext-103）上训练的 GPT-2 small。若要主张工作记忆约束的实现具有领域一般性，未来工作可以对我们的模型做跨语言考察，并在不同模型规模与架构上做更充分的检验。

Acknowledgments

致谢

This work is supported by NSF #1947307 to RF, NSF-IIS #2504954 to BD, as well as a Samuel F. Conti Faculty Fellowship from the University of Massachusetts, Amherst. We are also grateful to Jonathan Webster for his excellent research assistance, as well as the reviewers and audience at the ACL Rolling Review and the 2026 HSP Conference on Human Sentence Processing for helpful discussion and feedback.

本工作得到 NSF #1947307（授予 RF）、NSF-IIS #2504954（授予 BD），以及马萨诸塞大学阿默斯特分校 Samuel F. Conti 教师研究员基金的资助。我们也感谢 Jonathan Webster 出色的研究协助，以及 ACL Rolling Review 与 2026 年 HSP 人类句子加工会议的审稿人和听众所给予的讨论与反馈。

References

参考文献

George A Alvarez. 2011. Representing multiple objects as an ensemble enhances visual cognition. Trends in Cognitive Sciences, 15(3):122–131.

John Robert Anderson. 1990. The Adaptive Character of Thought.

Psychology Press.

Aryaman Arora, Dan Jurafsky, and Christopher Potts. 2024. CausalGym: Benchmarking causal interpretability methods on linguistic tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14638–14663. R Harald Baayen, Douglas J Davidson, and Douglas M Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4):390–412.

Brian Bartek, Richard L Lewis, Shravan Vasishth, and Mason R Smith. 2011. In search of on-line locality effects in sentence comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5):1178. Christopher J Bates and Robert A Jacobs. 2020. Efficient data compression in perception and perceptual memory.

Psychological Review, 127(5):891. Paul M Bays, Raquel FG Catalao, and Masud Husain. 2009. The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9(10):7–7. Paul M Bays, Sebastian Schneegans, Wei Ji Ma, and Timothy F Brady. 2024. Representation and computation in visual working memory.

Nature Human Behaviour, pages 1–19.

Veronica Boyce, Richard Futrell, and Roger P Levy. 2020. Maze made easy: Better and easier measurement of incremental processing difficulty. Journal of Memory and Language, 111:104082.

Veronica Boyce and Roger Levy. 2023. A-maze of Natural Stories: Comprehension and surprisal in the Maze task.

Glossa Psycholinguistics, 2(1). Timothy F Brady, Maria M Robinson, and Jamal R Williams. 2024. Noisy and hierarchical visual memory across timescales.

Nature Reviews Psychology, pages 1–17. Timothy F Brady and Joshua B Tenenbaum. 2013. A probabilistic model of visual working memory: Incorporating higher order regularities into working memory capacity estimates.

Psychological Review, 120(1):85.

Charlotte Caucheteux, Alexandre Gramfort, and JeanRémi King. 2023. Evidence of a predictive coding hierarchy in the human brain listening to speech.

Nature Human Behaviour, 7(3):430–441.

Noam Chomsky. 1957. Syntactic structures. Mouton de Gruyter.

Christian Clark, Byung-Doh Oh, and William Schuler. 2025. Linear recency bias during training improves transformers’ fit to reading times. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7735–7747.

Charles Clifton Jr, Adrian Staub, and Keith Rayner. 2007. Eye movements in reading words and sentences. Eye movements, pages 341–371.

Tri Dao and Albert Gu. 2024. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, pages 10041–10071.

Andrea De Varda and Marco Marelli. 2024. Locally biased transformers better align with human reading times. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 30– 36.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318– 30332.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115.

Stanley Dubinsky and Kemp Williams. 1995. Recategorization of prepositions as complementizers: The case of temporal prepositions in english.

Linguistic Inquiry, 26(1):125–137. Jeffrey L Elman. 1990. Finding structure in time.

Cognitive Science, 14(2):179–211. Michael C Frank and Noah D Goodman. 2025. Cognitive modeling using artificial intelligence.

Annual Review of Psychology, 77. Stefan L Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science, 22(6):829–834.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

Richard Futrell, Edward Gibson, and Roger P Levy. 2020. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive Science, 44(3):e12814. Richard Futrell, Edward Gibson, Harry J Tily, Idan Blank, Anastasia Vishnevetsky, Steven T Piantadosi, and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructions.

Language Resources and Evaluation, 55:63–77.

Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. arXiv preprint arXiv:2501.17047. Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. 2015. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines.

Science, 349(6245):273–278.

Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1–76.

Edward Gibson, Neal Pearlmutter, Enriqueta CansecoGonzalez, and Gregory Hickok. 1996. Recency preference in the human sentence processing mechanism. Cognition, 59(1):23–59. Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, and 1 others. 2022. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3):369–380.

Laura Gwilliams, Alec Marantz, David Poeppel, and Jean-Rémi King. 2025. Hierarchical dynamic coding coordinates speech comprehension in the human brain. Proceedings of the National Academy of Sciences, 122(42):e2422097122.

Michael Hahn, Judith Degen, and Richard Futrell. 2021. Modeling word and morpheme order in natural language as an efficient trade-off of memory and surprisal.

Psychological Review, 128(4):726.

Michael Hahn and Richard Futrell. 2019. Estimating predictive rate–distortion curves via neural variational inference.

Entropy, 21(7):640.

Michael Hahn, Richard Futrell, Roger Levy, and Edward Gibson. 2022. A resource-rational model of human processing of recursive linguistic structure. Proceedings of the National Academy of Sciences, 119(43):e2122602119.

John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780.

Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 1725–1744.

Andrej Karpathy. 2023. nanoGPT. https://github. com/karpathy/nanoGPT.

Maayan Keshev, Mandy Cartner, Aya Meltzer-Asscher, and Brian Dillon. 2025. A working memory model of sentence processing as binding morphemes to syntactic positions. Topics in Cognitive Science, 17(1):88– 105.

Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context limitations make neural language models more human-like. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10421–10436.

Tatsuki Kuribayashi, Yohei Oseki, Souhaib Ben Taieb, Kentaro Inui, and Timothy Baldwin. 2025. Large language models are human-like internally. Transactions of the Association for Computational Linguistics, 13:1743–1766.

Roger Levy. 2008. Expectation-based syntactic comprehension.

Cognition, 106(3):1126–1177. Richard L Lewis. 1996. Interference in short-term memory: The magical number two (or three) in sentence processing. Journal of Psycholinguistic Research, 25(1):93–115. Richard L Lewis, Andrew Howes, and Satinder Singh. 2014. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in Cognitive Science, 6(2):279–311. Richard L Lewis and Shravan Vasishth. 2005. An activation-based model of sentence processing as skilled memory retrieval.

Cognitive Science, 29:375– 419. Richard L Lewis, Shravan Vasishth, and Julie A Van Dyke. 2006. Computational principles of working memory in sentence comprehension. Trends in Cognitive Sciences, 10(10):447–454.

Falk Lieder and Thomas L Griffiths. 2020. Resourcerational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and Brain Sciences, 43:e1. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, WeiMing Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware weight quantization for ondevice LLM compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pages 87–100. Steven G Luke and Kiel Christianson. 2018. The provo corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2):826– 833.

Wei Ji Ma, Jeffrey M Beck, Peter E Latham, and Alexandre Pouget. 2006. Bayesian inference with probabilistic population codes.

Nature Neuroscience, 9(11):1432–1438.

Wei Ji Ma, Masud Husain, and Paul M Bays. 2014. Changing concepts of working memory. Nature Neuroscience, 17(3):347–356. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences.

David Marr. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information.

Henry Holt and Co., Inc., New York, NY, USA.

Brian McElree. 2006. Accessing recent events. Psychology of Learning and Motivation, 46:155–200.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information.

Psychological Review, 63(2):81.

Masato Mita, Ryo Yoshida, and Yohei Oseki. 2025. Developmentally-plausible working memory shapes a critical period for language acquisition. arXiv preprint arXiv:2502.04795.

Dennis Norris. 2006. The Bayesian reader: Explaining word recognition as an optimal Bayesian decision process.

Psychological Review, 113(2):327.

Klaus Oberauer. 2009. Design for a working memory. Psychology of Learning and Motivation, 51:45–100.

Byung-Doh Oh and Tal Linzen. 2025. To model human linguistic prediction, make LLMs less superhuman. arXiv preprint arXiv:2510.05141.

Byung-Doh Oh and William Schuler. 2022. Entropyand distance-based predictors from GPT-2 attention patterns predict reading times over and above GPT-2 surprisal. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9324–9334.

Byung-Doh Oh and William Schuler. 2023. Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:336–350. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. Tamar I Regev, Colton Casto, Eghbal A Hosseini, Markus Adamek, Anthony L Ritaccio, Jon T Willie, Peter Brunner, and Evelina Fedorenko. 2024. Neural populations in the language network differ in the size of their temporal receptive windows.

Nature Human Behaviour, 8(10):1924–1942. Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65.

Soo Hyun Ryu and Richard L Lewis. 2021. Accounting for agreement phenomena in sentence comprehension with transformer language models: Effects of similarity-based interference on surprisal and attention. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 61– 71.

Soo Hyun Ryu and Richard L Lewis. 2025. Memory for prediction: A transformer-based theory of sentence processing. Journal of Memory and Language, 145:104670. Jacqueline S Sachs. 1974. Memory in reading and listening to discourse. Memory & Cognition, 2(1):95–100. Elizabeth R Schotter and Brian Dillon. 2025. A beginner’s guide to eye tracking for psycholinguistic studies of reading.

Behavior Research Methods, 57(2):68. Herbert A Simon. 1955. A behavioral model of rational choice.

The Quarterly Journal of Economics, pages 99–118.

Garrett Smith and Shravan Vasishth. 2020. A principled approach to feature selection in models of sentence processing.

Cognitive Science, 44(12):e12918.

Robyn Speer. 2022. rspeer/wordfreq: v3.0.

Susanne Still. 2014. Information bottleneck approach to predictive inference.

Entropy, 16(2):968–989. James V Stone. 2018. Principles of neural information theory: Computational neuroscience and metabolic efficiency.

Sebtel Press.

William Timkey and Tal Linzen. 2023. A language model with limited memory capacity captures interference in human sentence processing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8705–8720. John C Trueswell, Michael K Tanenhaus, and Susan M Garnsey. 1994. Semantic influences on parsing: Use of thematic role information in syntactic ambiguity resolution. Journal of Memory and Language, 33(3):285–318. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30.

Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and 1 others. 2023. Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning. Ethan G Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P Levy. 2023. Testing the predictions of surprisal theory in 11 languages. Transactions of the Association for Computational Linguistics, 11:1451–1470.

Weijie Xu, Jason Chon, Tianran Liu, and Richard Futrell. 2023. The linearity of the effect of surprisal on reading times across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15711–15721.

Weijie Xu and Richard Futrell. 2026. Strategic resource allocation in memory encoding: An efficiency principle shaping language processing. Journal of Memory and Language, 146:104706.

Noga Zaslavsky, Charles Kemp, Terry Regier, and Naftali Tishby. 2018. Efficient compression in color naming and its evolution. Proceedings of the National Academy of Sciences, 115(31):7937–7942.

Junyuan Zhao, Ruimin Gao, and Jonathan R Brennan. 2025. Decoding the neural dynamics of headed syntactic structure building. Journal of Neuroscience, 45(17).

A Model training

A 模型训练

Training procedure. Models are trained on Wikitext-103 dataset (Merity et al., 2016) with 16000 training steps. The training procedure largely follows nanoGPT (Karpathy, 2023). Each batch contains 256 examples with sequences of 512 tokens randomly sampled from training data.

训练程序。模型在 Wikitext-103 数据集（Merity et al., 2016）上训练，共 16000 个训练步。训练程序大体沿用 nanoGPT（Karpathy, 2023）。每个批次包含 256 个样本，序列长度为 512 个词元，从训练数据中随机抽取。

Figure 5: Learning trajectories of model training

图 5：模型训练的学习轨迹

Models are trained using AdamW optimizer, with a learning rate of 0.0006. The learning rate was linearly warmed up over the first 3% of the maximal training steps (i.e., 500 steps), and was cosine annealed to a minimum of 0.00006. The decay rates θ are initialized randomly from Gaussian distribution with µ = 1 and σ = 0.001.

模型使用 AdamW 优化器训练，学习率为 0.0006。学习率在最大训练步数的前 3%（即 500 步）内线性预热，再按余弦退火降至最低值 0.00006。衰减速率 $\theta$ 从均值为 $\mu = 1$、标准差为 $\sigma = 0.001$ 的高斯分布中随机初始化。

Learning trajectories. Figure 5 shows the learning trajectories of model training for the cross-entropy component. With stronger memory constraint λ, cross-entropy loss decreases more slowly and ends up with higher final cross-entropy loss.

学习轨迹。图 5 展示了交叉熵分量在模型训练中的学习轨迹。记忆约束 $\lambda$ 越强，交叉熵损失下降得越慢，最终的交叉熵损失也越高。

B Per-layer attention scores for RT corpora in Experiment 2

B 实验 2 中阅读时间语料的逐层注意力分数

Figure 6 and 7 show the attention scores on key tokens as a function of key-query distance for NSC and Provo corpus respectively.

图 6 和图 7 分别展示了 NSC 语料与 Provo 语料上，键词注意力分数随键–查询距离的变化。

C Silhouette scores in Experiment 3

C 实验 3 中的轮廓系数

For each item, we calculate its silhouette score (Rousseeuw, 1987) with respect to the categories of interest. Taking the animacy feature as an example (which only has two categories: animate vs. inanimate), if item i is from the animate category, its silhouette score s is given by b −a (11) s = max(a, b), where a is the mean distance with all other items from the same animate category (i.e., within-category distance), and b is the mean distance Then, the separatability between the two cate-

对每个项目，我们计算其相对于所关注类别的轮廓系数（Rousseeuw, 1987）。以生命性特征为例（该特征只有两个类别：有生命 vs. 无生命），若项目 $i$ 来自有生命类别，其轮廓系数 $s_{i}$ 由下式给出：

$s_{i} = \dfrac{b_{i}-a_{i}}{\max(a_{i}, b_{i})}, \quad (11)$

其中 $a_{i}$ 是它与同属有生命类别的所有其他项目的平均距离（即类内距离），$b_{i}$ 是它与无生命类别中所有项目的平均距离（即类间距离）。随后，某一语言特征两个类别之间的可分性，取为所有项目轮廓系数的均值。平均轮廓系数越高，可分性越好。

D PCA visualization with 200 NP labels in Experiment 3

D 实验 3 中带 200 个名词短语标签的 PCA 可视化

Figure 8 shows the representational space in Experiment 3 visualized from the first two PCs with 200 randomly sampled noun-phrase labels (including Noun1 and Prep). When λ ≥0.01, there emerges a separate cluster for items with temporal prepositions before and after.

图 8 展示了实验 3 中由前两个主成分可视化的表征空间，并标注了随机抽取的 200 个名词短语标签（含 Noun1 和 Prep）。当 $\lambda \geq 0.01$ 时，带时间介词 before 和 after 的项目会分离出一个独立的簇。

Figure 6: Aggregated attention score per layer on NSC corpus

图 6：NSC 语料上逐层聚合的注意力分数

Figure 7: Aggregated attention score per layer on Provo corpus

图 7：Provo 语料上逐层聚合的注意力分数

Figure 8: Representational space of next-word prediction visualized from the first two PCs with 200 randomly sampled noun-phrase labels including Noun1 and Prep.

图 8：由前两个主成分可视化的下一个词预测表征空间，并标注随机抽取的 200 个名词短语标签（含 Noun1 和 Prep）。