Superposition Yields Robust Neural Scaling

Yizhou Liu Thanks:  liuyz@mit.edu    Ziming Liu       Jeff Gore Affiliation:  Massachusetts Institute of Technology Affiliation:  Cambridge, MA 02139

Abstract

The success of today’s large language models (LLMs) depends on the observation that larger models perform better. However, the origin of this neural scaling law — the finding that loss decreases as a power law with model size — remains unclear. Starting from two empirical principles — that LLMs represent more things than the model dimensions (widths) they have (i.e., representations are superposed), and that words or concepts in language occur with varying frequencies — we constructed a toy model to study the loss scaling with model size. We found that when superposition is weak, meaning only the most frequent features are represented without interference, the scaling of loss with model size depends on the underlying feature frequency; if feature frequencies follow a power law, so does the loss. In contrast, under strong superposition, where all features are represented but overlap with each other, the loss becomes inversely proportional to the model dimension across a wide range of feature frequency distributions. This robust scaling behavior is explained geometrically: when many more vectors are packed into a lower dimensional space, the interference (squared overlaps) between vectors scales inversely with that dimension. We then analyzed four families of open-sourced LLMs and found that they exhibit strong superposition and quantitatively match the predictions of our toy model. The Chinchilla scaling law turned out to also agree with our results. We conclude that representation superposition is an important mechanism underlying the observed neural scaling laws. We anticipate that these insights will inspire new training strategies and model architectures to achieve better performance with less computation and fewer parameters.

当今大语言模型（LLM）的成功，依赖「更大模型表现更好」这一观察。但神经缩放律——损失随模型规模呈幂律下降——的起源仍不清楚。我们从两条经验原则出发：LLM 表示的事物多于其模型维数（宽度），即表征处于叠加（superposition）；以及语言中词或概念出现频率不等。据此构造玩具模型，研究损失随模型规模的缩放。发现当叠加较弱——仅最频繁特征被无干扰地表示——时，损失随模型规模的缩放依赖底层特征频率：若特征频率服从幂律，损失亦然。相反，在强叠加下——所有特征都被表示但彼此重叠——在广泛的特征频率分布上，损失与模型维数成反比。这一稳健缩放可从几何解释：当远多于维数的向量被塞进低维空间时，向量间干扰（重叠平方）随该维数反比缩放。我们随后分析四个开源 LLM 家族，发现它们呈强叠加，并与玩具模型预测定量吻合；Chinchilla 缩放律也与结果一致。结论是：表征叠加是观测到的神经缩放律背后的重要机制。我们预期这些洞见将启发新的训练策略与模型结构，以更少计算与参数达到更好表现。

1 Introduction

The remarkable success of large language models (LLMs) has been driven by the empirical observation that increasing model size, training data, and compute consistently leads to better performance [1, 2, 3, 4]. Across a wide range of tasks — including language understanding [1, 5, 6], math [7, 8, 9, 10], and code generation [11, 12] — larger models achieve lower loss, higher accuracy, and greater generalization abilities [2, 13]. This consistent trend, known as neural scaling laws, has been observed across multiple model families and architectures, fueling the development of increasingly large models [2, 3, 4]. These scaling laws have not only shaped the current strategies for building better models but have also raised fundamental questions about why such simple and universal patterns emerge in complex learning systems.

大语言模型（LLM）的显著成功，由「增大模型规模、训练数据与算力会持续带来更好表现」这一经验观察驱动 [1, 2, 3, 4]。在语言理解 [1, 5, 6]、数学 [7, 8, 9, 10] 与代码生成 [11, 12] 等广泛任务上，更大模型损失更低、准确率更高、泛化更强 [2, 13]。这一一致趋势被称为神经缩放律，已在多个模型家族与架构上被观测到，并推动了越来越大的模型 [2, 3, 4]。缩放律不仅塑造了当前造更好模型的策略，也提出了根本问题：为何复杂学习系统中会出现如此简单而普遍的模式。

Figure 1: Superposition leads to robust and fast power-law loss decay with model size. (a) Illustration of no superposition where a three-dimensional space can at most represent three features without any interference (overlap). (b) Toy model results in the regime of weak superposition, where we set data dimension $n=10240$ (number of features). The toy model will be introduced; more details are in Appendix D.1. (c) Illustration of superposition: there are more features than the dimension of the space. (d) The same toy models in the strong superposition regime show lower losses, which are on power laws with model dimension and have exponents close to $1$ (color coding same as panel b). The gray points are from actual LLMs, which seem to have a similar power-law relation with exponent $\sim 1$.

图 1：叠加带来随模型规模稳健且快速的幂律损失衰减。（a）无叠加示意：三维空间最多无干扰（重叠）地表示三个特征。（b）弱叠加区玩具模型结果，数据维 $n=10240$（特征数）。玩具模型将在后文引入；更多细节见附录 D.1。（c）叠加示意：特征数多于空间维数。（d）同一玩具模型在强叠加区损失更低，相对模型维呈幂律且指数接近 $1$（配色同面板 b）。灰色点来自真实 LLM，似乎也有指数 $\sim 1$ 的类似幂律关系。

The scaling of loss with model size plays a central role in both the practical design and the theoretical understanding of large-scale machine learning systems, yet its origin remains inconclusive [3, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]. Various explanations have been proposed, drawing from statistical learning theory and empirical phenomenological models, including improved function or manifold approximation in larger models [14, 15], and enhanced representation or skill learning in larger models [18, 19, 20, 21]. In the limit of infinite data, many of these explanations predict a power-law decay of loss with model size, provided the underlying data distribution also follows a power law. However, the predicted scaling exponents depend sensitively on the properties of the data distribution. Moreover, the connection between these mechanistic explanations and the behavior of actual LLMs needs further exploration.

损失随模型规模的缩放，在大规模机器学习系统的实践设计与理论理解中都居中心地位，但其起源仍无定论 [3, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]。已有解释来自统计学习理论与经验唯象模型，包括更大模型中更好的函数或流形逼近 [14, 15]，以及更强的表征或技能学习 [18, 19, 20, 21]。在无限数据极限下，许多解释在数据分布也服从幂律时，预测损失随模型规模幂律衰减；但预测的缩放指数对数据分布性质敏感。这些机制解释与真实 LLM 行为之间的联系仍需进一步探索。

When considering LLMs specifically, it becomes clear that representation learning can be a limiting factor, which is closely related to a phenomenon called superposition [25, 26], yet this aspect has not been thoroughly studied. LLMs must learn embedding vectors for tokens, process these representations through transformer layers to predict the next token, and use a final projection (the language model head) to generate the output. Conceptually, fitting functions or manifolds and learning skills or grammars are primarily tasks of the transformer layers, while representation learning is more directly tied to the embedding matrix and the language model head. To represent more than fifty thousand tokens — or even more abstract concepts — within a hidden space of at most a few thousand dimensions, the quality of representations is inevitably constrained, contributing to the final loss. Although models can represent more features than their dimensionality would suggest through a mechanism known as superposition [26], prior work on neural scaling laws has implicitly assumed weak superposition [15, 16, 17, 18, 19], which may be less relevant to the regime where LLMs operate. To address this gap, we systematically study here how representation superposition and data structure together influence the scaling of loss with model size.

具体到 LLM，表征学习可以成为限制因素，这与称为 叠加 的现象密切相关 [25, 26]，但该方面尚未被充分研究。LLM 须为 token 学习嵌入向量，经 transformer 层处理这些表征以预测下一 token，再用最终投影（语言模型头）生成输出。概念上，拟合函数或流形、学习技能或语法主要是 transformer 层的任务，而表征学习更直接系于嵌入矩阵与语言模型头。要在至多数千维的隐空间中表示五万以上 token——乃至更多抽象概念——表征质量不可避免受约束，并贡献于最终损失。虽可通过叠加机制表示多于维数的特征 [26]，先前神经缩放律工作常隐含假设弱叠加 [15, 16, 17, 18, 19]，这可能与 LLM 实际所处区间不太相关。为填补缺口，我们在此系统研究表征叠加与数据结构如何共同影响损失随模型规模的缩放。

We adopt a toy model construction similar to [26] to study how superposition affects neural scaling laws. In the toy model, representations are learned by recovering data, each composed of multiple latent features. These features in data have different frequencies of occurrence, reflecting their relative importance. In the absence of superposition, only the most frequent features are perfectly represented, while the others are ignored. As illustrated in Figure 1a, the first three of six features are represented in the three-dimensional space without interference, and the remaining three are omitted. We find that in the weak superposition regime, the scaling of loss with model dimension depends sensitively on how feature frequency decays with rank: the loss follows a power law with model size only if the feature frequencies themselves follow a power law, provided that $m$ is sufficiently large (Figure 1b). By contrast, strong superposition allows many more features to be represented, albeit with overlap in the representation (Figure 1c). In this regime, the model displays a robust behavior: loss scales inversely with model dimension, independent of the data frequency distribution (Figure 1d). Remarkably, we find that actual LLMs follow the same rule. Comparing models with the same dimension but different superposition degrees, the ones with stronger superposition also have a lower loss. Building on the basic phenomena, we provide the main takeaways: (i) In the strong superposition regime, loss due to imperfect representation scales inversely with model dimension, largely independent of feature frequency distributions; (ii) Scaling behaviors in the weak superposition regime are brittle and depend on the number of activated but unlearned features, and in the strong superposition regime are robust and can be understood via generic geometric facts; (iii) LLMs exhibit strong superposition and agree quantitatively with our toy model predictions.

我们采用类似 [26] 的玩具模型构造，研究叠加如何影响神经缩放律。玩具模型通过恢复数据来学习表征，每条数据由多个潜在特征组成；这些特征出现频率不同，反映相对重要性。无叠加时，仅最频繁特征被完美表示，其余被忽略。如图 1a：六特征中前三在三维空间无干扰表示，后三省略。我们发现：在弱叠加区，损失随模型维的缩放对特征频率随秩衰减的方式敏感——仅当特征频率本身服从幂律且 $m$ 足够大时，损失才随模型规模呈幂律（图 1b）。相反，强叠加允许表示远更多特征，但表征有重叠（图 1c）。在此区间模型表现稳健：损失与模型维成反比，且独立于数据频率分布（图 1d）。更惊人的是，真实 LLM 遵循同一规则；同维但叠加程度不同时，叠加更强者损失更低。基于这些基本现象，主要要点为：（i）强叠加下，不完美表征导致的损失与模型维成反比，大体独立于特征频率分布；（ii）弱叠加区缩放脆弱，依赖被激活但未学习的特征数，强叠加区稳健，可借通用几何事实理解；（iii）LLM 呈强叠加，并与玩具模型预测定量一致。

The rest of the paper will elaborate on the takeaways. In Section 2, we introduce the toy model, describe the data sampling procedure, and explain how we control the degree of superposition. Section 3 presents the detailed results and quantitative analysis of how superposition and data distribution influence loss scaling. In Section 4, we compare our findings to related works. Finally, Section 5 summarizes our conclusions and discusses limitations and future directions. Codes are available at https://github.com/liuyz0/SuperpositionScaling.

余下篇幅展开这些要点。第 2 节引入玩具模型、数据采样，以及如何控制叠加程度。第 3 节给出叠加与数据分布如何影响损失缩放的详细结果与定量分析。第 4 节与相关工作比较。第 5 节总结结论并讨论局限与未来方向。代码见 https://github.com/liuyz0/SuperpositionScaling。

2 Methods

Figure 2: Toy model of superposition. (a) Architecture and loss of the toy model. (b and c) Rows of the weight matrix $W$, denoted by $W_{i}$, are representations for features. (b) No superposition represented the most frequent features, i.e., the first three, without interference ($WW^{T}$ has zero off-diagonal elements). In the frequency-rank plot, height means frequency, and color means $\|W_{i}\|_{2}$. (c) With superposition, more features are represented, while the vectors $W_{i}$ overlap.

图 2：叠加玩具模型。（a）玩具模型结构与损失。（b 与 c）权矩阵 $W$ 的行 $W_{i}$ 是特征的表征。（b）无叠加：表示最频繁特征（前三）且无干扰（$WW^{T}$ 非对角为零）。频率–秩图中高度为频率，颜色为 $\|W_{i}\|_{2}$。（c）有叠加时表示更多特征，同时向量 $W_{i}$ 重叠。

To gain understanding of the relationship between superposition and data structure, we need a toy model on learning representation simple enough yet not simpler — two key principles need to be reflected, (i) there are more features to represent than the dimension of the model, and (ii) features occur in data with different frequencies. Later, we will discuss how the loss due to representation studied here may affect the overall final loss in LLMs.

要理解叠加与数据结构的关系，需要一个足够简单却不过度简化的表征学习玩具模型——须体现两条关键原则：（i）待表示特征多于模型维数；（ii）特征在数据中出现频率不等。后文再讨论此处研究的表征损失如何影响 LLM 的总体最终损失。

We adopt the toy model of superposition from Anthropic [26] with minor modifications (Figure 2a). Input data $x\in\mathbb{R}^{n}$ is a vector with data dimension $n$ being the number of atomic (or irreducible) features. Each element $x_{i}$ in $x$ is interpreted as the activation of this data point at feature $i$, which is sampled as

我们采用 Anthropic [26] 的叠加玩具模型并略作修改（图 2a）。输入 $x\in\mathbb{R}^{n}$ 为向量，数据维 $n$ 即原子（或不可约）特征数。$x$ 中每个元素 $x_{i}$ 解释为该数据点在特征 $i$ 上的激活，采样为

$$x_{i}=u_{i}v_{i},~u_{i}\sim\mathrm{Bernoulli}(p_{i})~\&~v_{i}\sim U(0,2). \tag{1}$$

Here, $u_{i}$ sampled from a Bernoulli distribution controls whether the feature $i$ is activated, and $v_{i}$ sampled from a uniform distribution controls the activation strength once feature $i$ is activated. The frequency of feature $i$ to appear in the data is $p_{i}$. Without loss of generality, we make the indices of features the same as their frequency or importance rank. The data structure is then about how $p_{i}$ decreases with rank $i$. The expected number of activations in one data will be referred to as activation density: $E=\sum_{i=1}^{n}p_{i}$. The model learns hidden representations by recovering the data, which cannot be done perfectly because the model dimension $m$ is much smaller than the number of possible features in the data $n$. The trainable parameters are a weight matrix $W\in\mathbb{R}^{n\times m}$ and a bias vector $b\in\mathbb{R}^{n}$. The weight matrix embeds data $x$ into a hidden space with dimension $m$, $h=W^{T}x$, with $m\ll n$. In practice, we fix $n$ as a large number and change the model dimension $m$. We use $W$ to read out the embedding, where $y=\mathrm{ReLU}(Wh+b)$. The loss is defined as the difference between the recovered $y$ and the original $x$, $L=\langle\|y-x\|_{2}^{2}\rangle_{x}$, where $\langle\cdot\rangle_{x}$ means average over $x$ distribution.

其中，$u_{i}$ 取自伯努利分布，控制特征 $i$ 是否被激活；$v_{i}$ 取自均匀分布，控制激活后的强度。特征 $i$ 在数据中出现的频率为 $p_{i}$。不失一般性，令特征下标与其频率或重要性秩相同。数据结构于是关于 $p_{i}$ 如何随秩 $i$ 下降。一条数据中激活数的期望称为激活密度：$E=\sum_{i=1}^{n}p_{i}$。模型通过恢复数据学习隐表征——因模型维 $m$ 远小于可能特征数 $n$，无法完美恢复。可训参数为权矩阵 $W\in\mathbb{R}^{n\times m}$ 与偏置 $b\in\mathbb{R}^{n}$。$W$ 把数据 $x$ 嵌入维数为 $m$ 的隐空间，$h=W^{T}x$，且 $m\ll n$。实践中固定较大的 $n$，改变模型维 $m$。用同一 $W$ 读出嵌入：$y=\mathrm{ReLU}(Wh+b)$。损失定义为恢复的 $y$ 与原 $x$ 之差，$L=\langle\|y-x\|_{2}^{2}\rangle_{x}$，其中 $\langle\cdot\rangle_{x}$ 表示对 $x$ 分布取平均。

We can now formally introduce superposition. Note that $W_{i}$ is the representation of feature $i$ in the hidden space, where we use $W_{i}$ to denote the $i$th row of the $W$ matrix, in our toy model, feature $i$ is well represented when the norm $\|W_{i}\|_{2}$ is close to $1$. No superposition ideally means the first $m$ rows of $W$ form an orthogonal basis (i.e., the first $m$ most important features represented perfectly) and the rest of the rows are zero (i.e., rest of the features ignored or lost), as illustrated in Figure 2b. Superposition means that there are more than $m$ rows in $W$ with norms close to $1$ (Figure 2c). Conceptually, superposition is preferred when features are more equal in importance ($p_{i}$ are more uniform) and when features are more sparse ($E/n$ is smaller) [26].

现在可形式引入叠加。注意 $W_{i}$ 是特征 $i$ 在隐空间中的表征（$W_{i}$ 为 $W$ 的第 $i$ 行）；在玩具模型中，当范数 $\|W_{i}\|_{2}$ 接近 1 时，特征 $i$ 被良好表示。理想的无叠加意味着：$W$ 的前 $m$ 行构成正交基（前 $m$ 个最重要特征被完美表示），其余行为零（其余特征被忽略或丢失），如图 2b。叠加则意味着有多于 $m$ 行的范数接近 1（图 2c）。概念上，当特征重要性更均等（$p_{i}$ 更均匀）且更稀疏（$E/n$ 更小）时，叠加更受青睐 [26]。

Figure 3: Weight decay can robustly tune the degree of superposition. (a) Given data dimension $n=1000$ and feature frequency $p_{i}\propto 1/i^{\alpha}$ with $\alpha=1$, we tested model dimension $m$ from $10$ to $100$ and weight decay from $-1$ to $1$. The frequency-rank plot shows that important features tend to be represented (color means $\|W_{i}\|_{2}$ and colorbar same as Figure 2b). The row norm distribution is bimodal, making $\phi_{1/2}$ a good measure for the fraction represented. Smaller weight decay leads to larger $\phi_{1/2}$ or stronger superposition. (b) After changing $\alpha$, the ability for small weight decay to reach strong superposition is not changed. We show the result at $m=39$ here, more in Appendix D.3.

图 3：权重衰减可稳健调节叠加程度。（a）给定数据维 $n=1000$、特征频率 $p_{i}\propto 1/i^{\alpha}$ 且 $\alpha=1$，测试模型维 $m$ 从 10 到 100、权重衰减从 $-1$ 到 $1$。频率–秩图显示重要特征倾向于被表示（颜色为 $\|W_{i}\|_{2}$，色条同图 2b）。行范数分布呈双峰，使 $\phi_{1/2}$ 成为已表示比例的良好度量。更小权重衰减对应更大 $\phi_{1/2}$ 或更强叠加。（b）改变 $\alpha$ 后，小权重衰减达到强叠加的能力不变。此处展示 $m=39$；更多见附录 D.3。

We found that it is possible to control superposition independently of data properties by modifying the optimizer with a weight decay (or growth) term:

我们发现：通过在优化器中加入权重衰减（或增长）项，可以独立于数据性质控制叠加：

$$W_{i,t+1}=\left{\begin{aligned} &W_{i,t}-\eta_{t}\gamma W_{i,t},\gamma\geq 0,\
&W_{i,t}-\eta_{t}\gamma W_{i,t}(1/|W_{i,t}|_{2}-1),\gamma<0,\end{aligned}\right. \tag{2}$$

where $\eta_{t}$ is the learning rate and $W{i,t}$ is the $i$th row of the weight matrix at step $t$ (vector operations are element-wise). For weight decay $\gamma<0$, the update corresponds to gradient descent on $(|W_{i,t}|_{2}-1)^{2}$, encouraging unit-norm rows. We implement this in a modified AdamW [27] optimizer with a warm-up and cosine decay learning rate schedule. Training runs for 20k–80k steps with batch sizes of 2048–8192, ensuring models reach saturation. At each step, we sample new data using probabilities $p_{i}$. We observe that important features are represented as expected, and row norms of $W$ become bimodal, clustering near 0 and 1 (Figure 3a). This allows us to define the fraction of represented features as

其中 $\eta_{t}$ 为学习率，$W{i,t}$ 为第 $t$ 步权矩阵第 $i$ 行（向量运算逐元素）。对权重衰减 $\gamma<0$，更新对应在 $(|W_{i,t}|_{2}-1)^{2}$ 上的梯度下降，鼓励单位范数行。我们在带预热与余弦衰减学习率日程的改动 AdamW [27] 中实现。训练 20k–80k 步，批大小 2048–8192，确保模型达饱和。每步按概率 $p_{i}$ 采样新数据。我们观察到重要特征如预期被表示，$W$ 的行范数呈双峰，聚在 0 与 1 附近（图 3a）。于是可定义已表示特征比例为

$$\phi_{1/2}=|\{i:\|W_{i}\|_{2}>1/2\}|/n, \tag{3}$$

namely, the fraction of rows with norm larger than $1/2$. We found that weight decay can tune superposition smoothly, with small weight decay $\gamma$ giving strong superposition, i.e., $\phi_{1/2}\approx 1\gg m/n$, and large weight decay corresponding to weak superposition, i.e., $\phi_{1/2}\sim m/n$ (Figure 3a). The ability of weight decay to tune superposition is robust to a variety of feature frequency distributions (Figure 3b), where $p_{i}\propto 1/i^{\alpha}$ and we control the decay speed of $p_{i}$ via the data exponent $\alpha$. Thus, weight decay provides a practical knob for controlling superposition in our study.

即范数大于 $1/2$ 的行所占比例。我们发现权重衰减可平滑调节叠加：小 $\gamma$ 给出强叠加，即 $\phi_{1/2}\approx 1\gg m/n$；大权重衰减对应弱叠加，即 $\phi_{1/2}\sim m/n$（图 3a）。该调节能力对多种特征频率分布稳健（图 3b），其中 $p_{i}\propto 1/i^{\alpha}$，我们用数据指数 $\alpha$ 控制 $p_{i}$ 衰减速度。因此，权重衰减为本研究提供了实用的叠加旋钮。

Our toy model differs from LLMs in architecture, data, and loss. Since we focus on representations rather than next-token prediction, we omit transformer layers. Conceptually, LLMs map a document to a token, with inputs and outputs in different spaces, while the toy model operates within a single shared space. Despite this, the toy model captures key aspects of language structure through engineered sparsity and feature importance, making its data structure aligned with that of LLMs at a higher level. While LLMs use cross-entropy loss and the toy model uses squared error, we will show that this does not affect the scaling behavior of the loss. Thus, the toy model is a suitable abstraction for studying representation-limited scaling.

玩具模型在结构、数据与损失上不同于 LLM。因聚焦表征而非下一 token 预测，我们省略 transformer 层。概念上，LLM 把文档映射到 token，输入与输出在不同空间；玩具模型则在单一共享空间内运作。尽管如此，玩具模型通过工程化的稀疏性与特征重要性捕捉语言结构的关键方面，使其数据结构在更高层次与 LLM 对齐。LLM 用交叉熵、玩具模型用平方误差，但我们将表明这不影响损失的缩放行为。因此，玩具模型是研究「表征受限缩放」的合适抽象。

3 Results

Figure 4: Rich scaling phenomena arise when we change the degree of superposition and data structures. (a) In the same experiments ($n=10240$) in Figure 1, where $p_{i}\propto\exp(-i/400)$ (blue), $p_{i}\propto 1/i^{1.2}$ (yellow), and $p_{i}\propto n-i$ (green), we use $\gamma=-1$ to reach strong superposition and $\gamma=0.1$ for weak superposition. R-squared value of the fitting is used to measure how likely the fitted part is a power law (Appendix D.4). R-squared values closer to 1 mean that the data are more similar to a power law. We found that at strong superposition, power laws are robust across different underlying feature distributions. Yet, at weak superposition, only power-law feature frequencies can lead to power-law losses. (b) When changing superposition by weight decay and varying feature frequency decay by $\alpha$ given $p_{i}\propto 1/i^{\alpha}$, we found roughly three distinct behaviors. For (b), $n=1000$ and $m=10\sim 100$ (more in Appendix D.4).

图 4：改变叠加程度与数据结构时出现丰富缩放现象。（a）与图 1 相同实验（$n=10240$），$p_{i}\propto\exp(-i/400)$（蓝）、$p_{i}\propto 1/i^{1.2}$（黄）、$p_{i}\propto n-i$（绿）；用 $\gamma=-1$ 达强叠加、$\gamma=0.1$ 达弱叠加。用拟合的 R 方度量拟合段有多像幂律（附录 D.4）；越接近 1 越像。强叠加下，对不同底层特征分布幂律都稳健；弱叠加下仅幂律特征频率能导致幂律损失。（b）用权重衰减改变叠加、用 $\alpha$ 改变 $p_{i}\propto 1/i^{\alpha}$ 的衰减时，大致出现三种不同行为。对（b），$n=1000$，$m=10\sim 100$（更多见附录 D.4）。

In the toy model, we found rich phenomena by varying superposition and feature importance. Reviewing the experiments in Figure 1, we set weight decay $\gamma=-1$ to have strong superposition and $\gamma=0.1$ to have weak superposition. We compute $R$ squared values from linear fits in log-log plots to quantify scaling behavior, assessing how closely the loss follows a power law to model dimension. We can see that at strong superposition, the losses are close to power laws, regardless of the underlying feature frequencies, yet the loss is a power law at weak superposition if the feature frequency $p_{i}$ is a power law with rank $i$ (Figure 4a). For a systematic scan, we next set $p_{i}\propto 1/i^{\alpha}$ and can vary the data exponent $\alpha$ to change how fast $p_{i}$ decays consistently.[^1] Unless otherwise specified, we set activation density $E$ as $1$ by default, whose value will be shown not to affect the scaling. Assuming a power-law form for the final test loss, $L\propto 1/m^{\alpha_{m}}$, we extract the model exponent $\alpha_{m}$ from the empirical fit. We fit the loss with a power law in all cases. The fitted $\alpha_{m}$ reveals how fast losses decay, even in the weak superposition regime where the power law fit is poor (Appendix D.4). Roughly, three distinct patterns emerge: (1) under weak superposition (positive $\gamma$, yellow box in Figure 4b), $\alpha_{m}$ is small, indicating slow loss decay; (2) under strong superposition and a wide range of small data exponents (red box), $\alpha_{m}$ remains robustly near 1; (3) for strong superposition with large data exponents (blue box), $\alpha_{m}$ increases with $\alpha$. By interpreting these three patterns, we aim to understand when loss follows a power law with model dimension, and what determines the exponent when it does.

在玩具模型中，改变叠加与特征重要性可得到丰富现象。回顾图 1 实验：取 $\gamma=-1$ 为强叠加、$\gamma=0.1$ 为弱叠加。在双对数图上做线性拟合，用 $R$ 方量化缩放行为，评估损失相对模型维有多接近幂律。可见强叠加下损失接近幂律，不论底层特征频率如何；弱叠加下仅当 $p_{i}$ 随秩 $i$ 呈幂律时损失才是幂律（图 4a）。系统扫描中，接着取 $p_{i}\propto 1/i^{\alpha}$，用数据指数 $\alpha$ 一致改变 $p_{i}$ 衰减速度。[^1] 除非另说明，默认激活密度 $E=1$，其取值后文将表明不影响缩放。假设最终测试损失呈幂律 $L\propto 1/m^{\alpha_{m}}$，从经验拟合提取模型指数 $\alpha_{m}$。所有情形都用幂律拟合损失；即便弱叠加区幂律拟合较差，$\alpha_{m}$ 仍揭示损失衰减快慢（附录 D.4）。大致出现三种模式：（1）弱叠加（正 $\gamma$，图 4b 黄框）下 $\alpha_{m}$ 小，损失衰减慢；（2）强叠加且数据指数在较宽的小范围内（红框），$\alpha_{m}$ 稳健地接近 1；（3）强叠加且数据指数大（蓝框）时，$\alpha_{m}$ 随 $\alpha$ 增大。通过解读这三种模式，我们试图理解：损失何时相对模型维呈幂律，以及呈幂律时指数由什么决定。

Figure 5: Loss at weak superposition can be well described by the expected number of activated but unlearned features. (a) The actual loss and theory at weak superposition (Equation (4)) agree when weight decay is positive. (b) For those closest to the ideal no superposition case (Appendix D.5), we expect $\alpha_{m}=\alpha-1$, which is close to measured values. Error bars are standard errors.

图 5：弱叠加下的损失可由「被激活但未学习」特征的期望数很好描述。（a）弱叠加下实际损失与理论（式 (4)）在权重衰减为正时吻合。（b）对最接近理想无叠加的情形（附录 D.5），期望 $\alpha_{m}=\alpha-1$，与测得值接近。误差棒为标准误。

We begin by analyzing the weak superposition regime. Consider an idealized case where the top $\phi_{1/2}n$ most frequent features are perfectly represented, where $\phi_{1/2}$ is the fraction of represented features. The loss can then be written as:

我们先分析弱叠加区。考虑理想情形：最频繁的前 $\phi_{1/2}n$ 个特征被完美表示，$\phi_{1/2}$ 为已表示特征比例。损失可写为：

$$L=\sum_{i=1}^{\phi_{1/2}n}\langle(y_{i}-x_{i})^{2}\rangle_{x}+\sum_{i=\phi_{1/2}n+1}^{n}\langle(y_{i}-x_{i})^{2}\rangle_{x}=\sum_{i=\phi_{1/2}n+1}^{n}\langle x_{i}^{2}\rangle_{x}=\langle v^{2}\rangle\sum_{i=\phi_{1/2}n+1}^{n}p_{i}. \tag{4}$$

The first equality follows from the definition of loss. The second holds because the first $\phi_{1/2}n$ features are assumed to be perfectly reconstructed, and the remaining features are ignored (i.e., $y_{i}=0$ for $i>\phi_{1/2}n$). The final expression uses the definition of $x_{i}=u_{i}v_{i}$, where $v\sim U(0,2)$, giving $\langle v^{2}\rangle=4/3$. To approximate the remaining sum, we use the integral $\int_{\phi_{1/2}n}^{n}p_{i}\mathrm{d}i$ and refer to this estimate as the loss by unlearned features. We find that, in the weak superposition regime, the actual losses closely match this prediction (Figure 5a). Slight discrepancies — where actual losses are slightly higher — can be attributed to the fact that models do not exactly learn the first $\phi_{1/2}n$ features. In contrast, in the strong superposition regime, the predicted loss by unlearned features is nearly zero, yet the actual loss remains non-zero, indicating additional sources of error. Focusing on cases closest to the ideal no-superposition scenario — where $\phi_{1/2}n=m$ and the first $m$ features are represented — we observe that such cases occur when $\alpha>1$ and yield a model exponent $\alpha_{m}\approx\alpha-1$ (Figure 5b). This matches the theoretical expectation that $\int_{m}^{n}p_{i}\mathrm{d}i\propto m^{-\alpha+1}$ when $n\gg m$ and $\alpha>1$. Thus, in the weak superposition regime, loss scaling is well described by the contribution of unlearned features, that is, the total frequency of features not represented by the model.

第一个等号来自损失定义。第二个成立是因为假定前 $\phi_{1/2}n$ 个特征被完美重建，其余被忽略（即 $i>\phi_{1/2}n$ 时 $y_{i}=0$）。最终表达式用 $x_{i}=u_{i}v_{i}$，且 $v\sim U(0,2)$，故 $\langle v^{2}\rangle=4/3$。对其余和用积分 $\int_{\phi_{1/2}n}^{n}p_{i}\mathrm{d}i$ 逼近，称此估计为「未学习特征带来的损失」。我们发现弱叠加下实际损失与该预测密切吻合（图 5a）。略高的微小偏差可归因于模型并非精确学会前 $\phi_{1/2}n$ 个特征。相反，强叠加下按未学习特征预测的损失近乎为零，实际损失仍非零，说明还有其他误差来源。聚焦最接近理想无叠加——$\phi_{1/2}n=m$ 且前 $m$ 个特征被表示——的情形，我们观察到它们出现在 $\alpha>1$ 时，并给出模型指数 $\alpha_{m}\approx\alpha-1$（图 5b）。这与理论期望一致：当 $n\gg m$ 且 $\alpha>1$ 时，$\int_{m}^{n}p_{i}\mathrm{d}i\propto m^{-\alpha+1}$。因此弱叠加区的损失缩放，可由未学习特征的贡献——即模型未表示特征的总频率——很好描述。

We can now explain when loss follows a power law in the weak superposition regime. When feature frequencies follow a power law, $p_{i}\propto 1/i^{\alpha}$ with $\alpha>1$, the integral $\int_{\phi_{1/2}n}^{n}p_{i}\mathrm{d}i$ becomes a power law in $\phi_{1/2}$. And if the $\phi_{1/2}$ scales approximately as a power of model size $m$ (ideally it is $m/n$). Then the loss also scales as a power law in $m$. Consequently, the deviations from power-law behavior seen in Figure 1b — for non-power-law $p_{i}$ — are consistent with this explanation. Moreover, when $p_{i}\propto 1/i^{\alpha}$ with $\alpha>1$, we expect the model exponent to be $\alpha_{m}\approx\alpha-1$, which matches the relatively small values of $\alpha_{m}$ observed on the right side of Figure 4b. In summary, the weak superposition regime reveals a strong dependence of loss vs. model dimension scaling on the structure of feature frequencies in the data.

现在可解释弱叠加下损失何时呈幂律。当特征频率服从幂律 $p_{i}\propto 1/i^{\alpha}$ 且 $\alpha>1$ 时，积分 $\int_{\phi_{1/2}n}^{n}p_{i}\mathrm{d}i$ 成为 $\phi_{1/2}$ 的幂律；若 $\phi_{1/2}$ 又近似随模型规模 $m$ 呈幂（理想为 $m/n$），则损失也随 $m$ 呈幂律。因此图 1b 中非幂律 $p_{i}$ 偏离幂律的行为与此解释一致。此外，当 $p_{i}\propto 1/i^{\alpha}$ 且 $\alpha>1$ 时，期望模型指数 $\alpha_{m}\approx\alpha-1$，与图 4b 右侧相对较小的 $\alpha_{m}$ 吻合。总之，弱叠加区显示：损失相对模型维的缩放强烈依赖数据中特征频率的结构。

Figure 6: Loss scaling at strong superposition is mainly explained via geometry. (a) The more frequent the features are, the more likely their norms are greater than $1$ (purple). (b) The row norm distribution is bimodal around $1$. Related results can be found in Appendix D.6. (c) Fraction of strongly represented features ($\|W_{i}\|_{2}>1$) at stong superposition ($\gamma=-1$) is around $\min\{1,m^{2}/2n\}$ (fast increasing dashed line), which is much larger than $m$ (slowly increasing dashed line). (d) The strongly represented features have decreasing overlaps as expected, where we plotted all the data when $\gamma<0$, and the dashed line is $\kappa^{2}$. (e) At strong superposition, $\alpha_{m}=1$ (horizontal line) if the strongly represented features having small overlaps dominate, which is true for a wide range of $\alpha$. But $\alpha_{m}=2(\alpha-1)$ (fast increasing dashed line) if weakly represented ones with large overlaps dominate, explaining the measured $\alpha_{m}$ when $\alpha$ is large. Error bars are standard errors.

图 6：强叠加下的损失缩放主要由几何解释。（a）特征越频繁，其范数越可能大于 $1$（紫）。（b）行范数分布在 $1$ 附近呈双峰。相关结果见附录 D.6。（c）强叠加（$\gamma=-1$）下强表示特征比例（$\|W_{i}\|_{2}>1$）约在 $\min\{1,m^{2}/2n\}$（快速上升虚线），远大于 $m$（缓慢上升虚线）。（d）强表示特征的重叠如预期下降；图中画出 $\gamma<0$ 的全部数据，虚线为 $\kappa^{2}$。（e）强叠加下，若小重叠的强表示特征占主导，则 $\alpha_{m}=1$（水平线）——这对较宽的 $\alpha$ 范围成立；若大重叠的弱表示特征占主导，则 $\alpha_{m}=2(\alpha-1)$（快速上升虚线），解释 $\alpha$ 较大时测得的 $\alpha_{m}$。误差棒为标准误。

We now turn to the strong superposition regime, where all features are represented. In this regime, the loss arises primarily from the non-zero overlaps between representation vectors. For instance, consider the case where only feature $j$ is activated in the data. Even for $i\neq j$, the output $y_{j}$ receives contributions on the order of $W_{i}\cdot W_{j}$, leading to a loss that scales as $(W_{i}\cdot W_{j})^{2}$ due to the squared error objective. The nonlinearity $\mathrm{ReLU}$ nonlinearity and bias terms do not affect the scaling behavior (Appendix A.1). Consequently, we study the overlaps between representation vectors when the number of vectors exceeds the dimensionality of the space.

现在转向强叠加区——所有特征都被表示。此区间损失主要来自表征向量间非零重叠。例如数据中仅特征 $j$ 被激活时，即便 $i\neq j$，输出 $y_{j}$ 也会收到量级为 $W_{i}\cdot W_{j}$ 的贡献，平方误差目标下损失按 $(W_{i}\cdot W_{j})^{2}$ 缩放。$\mathrm{ReLU}$ 非线性与偏置项不影响缩放行为（附录 A.1）。因此，我们在向量数超过空间维数时研究表征向量间的重叠。

We now provide theoretical expectations for vector overlaps. Consider $\nu$ unit vectors $w_{i}\in\mathbb{R}^{m}$ with $\nu\geq m$. It can be shown that [28]

下面给出向量重叠的理论期望。考虑 $\nu$ 个单位向量 $w_{i}\in\mathbb{R}^{m}$ 且 $\nu\geq m$。可以证明 [28]

$$\max_{i\neq j}|w_{i}\cdot w_{j}|\geq\sqrt{\frac{\nu-m}{m(\nu-1)}}\equiv\kappa. \tag{5}$$

This lower bound, $\kappa$, approaches $\sqrt{1/m}$ when $\nu\gg m$. The bound is met when the vectors form an equal angle tight frame (ETF) [29, 30, 31], which minimizes interference and appears in contexts such as quantum measurements [32, 33, 34, 35] and neural collapse [36, 37]. However, ETFs in real space can only exist if $\nu\leq m(m+1)/2$ [29, 30, 31]. In our toy model, feature frequencies vary, so exact equal-angle configurations are suboptimal — important features may benefit from smaller overlaps. Nonetheless, when feature frequencies are nearly uniform (e.g., small data exponent $\alpha$), $\sqrt{1/m}$ remains a good estimate. Although the rows $W_{i}$ are not strictly unit vectors, they are close enough that the scaling behavior remains valid. In summary, we expect that in the strong superposition regime: (i) Many representation vectors can approximate a mutually orthogonal configuration, with typical overlaps scaling as $\sqrt{1/m}$; (ii) The number of such vectors can be much larger than $m$ but still bounded (e.g., $\sim m^{2}/2$ for ETFs), so when $n$ is too large, some feature vectors cannot be in such configurations, having larger overlaps.

该下界 $\kappa$ 在 $\nu\gg m$ 时趋近 $\sqrt{1/m}$。当向量构成等角紧框架（ETF）[29, 30, 31] 时达到该界——ETF 最小化干扰，并出现在量子测量 [32, 33, 34, 35] 与神经坍缩 [36, 37] 等语境。但实空间中 ETF 仅当 $\nu\leq m(m+1)/2$ 时可能存在 [29, 30, 31]。玩具模型中特征频率不等，精确等角配置并非最优——重要特征或受益于更小重叠。尽管如此，当特征频率近乎均匀（例如数据指数 $\alpha$ 小）时，$\sqrt{1/m}$ 仍是良好估计。虽行 $W_{i}$ 并非严格单位向量，但足够接近，缩放行为仍成立。总之，强叠加区我们期望：（i）许多表征向量可近似相互正交配置，典型重叠按 $\sqrt{1/m}$ 缩放；（ii）此类向量数可远大于 $m$ 但仍有界（例如 ETF 约 $m^{2}/2$），故当 $n$ 过大时，部分特征向量无法处于此类配置，重叠更大。

With these theoretical expectations in hand, we evaluate our trained toy models. The first step is to test for the presence of a nearly orthogonal configuration among representation vectors. A preliminary clue comes from examining the norms of the rows $W_{i}$, which exhibit a bimodal distribution: one peak slightly below 1, corresponding to less frequent features, and another above 1, corresponding to the most frequent ones (Figure 6, a and b). This suggests that the most important features are represented in a nearly orthogonal manner, while less important ones exhibit greater overlap. To quantify this, we compute the fraction of rows with norm exceeding 1:

有了这些理论期望，我们评估已训玩具模型。第一步检验表征向量中是否存在近正交配置。初步线索来自行 $W_{i}$ 的范数：呈双峰分布——一峰略低于 1，对应较少频特征；另一峰高于 1，对应最频繁特征（图 6a、b）。这表明最重要特征以近正交方式表示，较不重要者重叠更大。为量化，我们计算范数超过 1 的行所占比例：

$$\phi_{1}=|\{i:\|W_{i}\|_{2}>1\}|/n, \tag{6}$$

which significantly exceeds $m/n$ and is around the ETF expectation $m^{2}/2n$ (Figure 6c). For these rows, the mean squared overlaps match the theoretical bound $\kappa^{2}$ well (Figure 6d). Although these vectors do not form a perfect ETF, their configuration is much closer to an ETF than random initialization (Appendix D.6). In contrast, rows with norms less than $1$ exhibit larger overlaps on average (Appendix D.6). We refer to rows with $\|W_{i}\|_{2}>1$ as strongly represented features and those with smaller norms as weakly represented. These results confirm our expectations: strongly represented features attempt to form an approximately orthogonal configuration with typical overlaps scaling as $\sqrt{1/m}$; their number exceeds $m$ a lot but remains bounded, necessitating the presence of weakly represented features with larger overlaps.

该比例显著超过 $m/n$，并接近 ETF 期望 $m^{2}/2n$（图 6c）。对这些行，平均重叠平方与理论界 $\kappa^{2}$ 吻合良好（图 6d）。虽非完美 ETF，其配置远比随机初始化更接近 ETF（附录 D.6）。相反，范数小于 $1$ 的行平均重叠更大（附录 D.6）。我们称 $\|W_{i}\|_{2}>1$ 的行为强表示特征，范数更小的为弱表示。结果确认期望：强表示特征试图形成典型重叠按 $\sqrt{1/m}$ 缩放的近似正交配置；其数量远超 $m$ 但仍有界，因而必然存在重叠更大的弱表示特征。

Figure 7: Activation density does not affect scaling exponents in our tests. (a) Loss is roughly proportional to activation density given the degree of superposition ($m=63$, $n=1000$). (b) So, $E$ will only affect the coefficient but not the exponent when considering the power law with model dimension. We plot the evidence $\alpha_{m}\approx 1$ at strong superposition, see more in Appendix D.7.

图 7：激活密度在我们的测试中不影响缩放指数。（a）给定叠加程度时，损失大致与激活密度成正比（$m=63$，$n=1000$）。（b）因此考虑相对模型维的幂律时，$E$ 只影响系数不影响指数。我们画出强叠加下 $\alpha_{m}\approx 1$ 的证据；更多见附录 D.7。

In the strong superposition regime, the loss is mainly determined by a weighted sum of squared overlaps between representation vectors. For strongly represented features, where overlaps scale as $\sqrt{1/m}$, the loss contribution scales as $1/m$. In contrast, weakly represented features can have large, constant overlaps. In the worst case, their total contribution is proportional to $\sum_{i=\phi_{1}n}^{n}p_{i}\sim m^{-2(\alpha-1)}$, where we approximate $\phi_{1}n\approx m^{2}/2$. (See Appendix A.1 for loss decomposition details) If the loss is dominated by the strongly represented features, we expect the overall loss to scale as $1/m$. In the opposite limit — when weakly represented features dominate — the loss scales as $m^{-2(\alpha-1)}$, meaning the effective model exponent becomes $\alpha_{m}\approx 2(\alpha-1)$. Although we do not solve the toy model analytically, we hypothesize that as $\alpha$ increases (i.e., feature frequencies become more skewed), optimization prioritizes minimizing the loss of the most frequent features. As a result, the remaining loss is mostly due to the weakly represented ones. Empirically, we find a simple and elegant rule (Figure 6e):

强叠加区损失主要由表征向量间重叠平方的加权和决定。对强表示特征，重叠按 $\sqrt{1/m}$ 缩放，损失贡献按 $1/m$ 缩放。弱表示特征则可有大的常值重叠；最坏情形下其总贡献正比于 $\sum_{i=\phi_{1}n}^{n}p_{i}\sim m^{-2(\alpha-1)}$，其中近似 $\phi_{1}n\approx m^{2}/2$。（损失分解细节见附录 A.1）若损失由强表示特征主导，期望总体损失按 $1/m$ 缩放；相反若弱表示主导，损失按 $m^{-2(\alpha-1)}$ 缩放，有效模型指数变为 $\alpha_{m}\approx 2(\alpha-1)$。虽未解析求解玩具模型，我们假设：随 $\alpha$ 增大（特征频率更偏斜），优化优先最小化最频繁特征的损失，剩余损失主要来自弱表示者。经验上我们找到一条简单优雅的规则（图 6e）：

$$\alpha_{m}\approx\max\{1,2(\alpha-1)\}. \tag{7}$$

In short, the optimization selects whether strongly or weakly represented features dominate the loss, whichever leads to faster decay with model size $m$. We now have a full picture that the origin of $\alpha_{m}=1$ lies in the geometric constraints on vector overlaps, and this scaling is robust as long as the feature frequency distribution is not too skewed.

简言之，优化选择让强表示或弱表示特征主导损失——取随模型规模 $m$ 衰减更快的那一方。至此全貌清楚：$\alpha_{m}=1$ 的起源在于向量重叠的几何约束，只要特征频率分布不太偏斜，该缩放就稳健。

We also studied the effect of the number of expected activated features or activation density $E$, which was set to $1$. By fixing data exponent $\alpha=1$, which will be shown to be relevant to natural language, we can scan different superposition degrees and activation densities. Since $p_{i}\leq 1$ is required, which is equivalent to $p_{1}\leq 1$, we have $E\leq\sum_{i=1}^{n}1/i^{\alpha}$, setting the upper bound for our scanning. We found that loss is approximately proportional to activation density $E$ (Figure 7a). This fact suggests that the power law exponent should not change, which we confirmed (Figure 7b). Under a controlled superposition degree, activation density linearly increases loss and thus does not affect the scaling exponents in our experiments.

我们还研究了期望激活特征数即激活密度 $E$ 的影响，此前设为 $1$。固定数据指数 $\alpha=1$（后文将表明与自然语言相关），可扫描不同叠加程度与激活密度。因要求 $p_{i}\leq 1$（等价于 $p_{1}\leq 1$），有 $E\leq\sum_{i=1}^{n}1/i^{\alpha}$，构成扫描上界。我们发现损失近似与激活密度 $E$ 成正比（图 7a），暗示幂律指数不应改变——已确认（图 7b）。在受控叠加程度下，激活密度线性抬高损失，因而不影响我们实验中的缩放指数。

Figure 8: Superposition may explain the neural scaling law observed in actual LLMs. We evaluate four open-sourced model classes, Opt [38], GPT2 [39], Qwen [40], and Pythia [41], which have model sizes from around 100M to 70B (details in Appendix C). (a) We found the mean square overlaps of $W_{i}/\|W_{i}\|_{2}$ roughly follow $1/m$ scaling, where $W$ is the language model head. (b) The model class is reflected by color as panel a, while we use shapes for evaluation datasets [42, 43, 44, 45]. The loss related to model size is fitted as a power law, yielding empirical $\alpha_{m}=0.91\pm 0.04$ close to $1$ (details in Appendix D.8).

图 8：叠加或可解释真实 LLM 中观测到的神经缩放律。我们评估四个开源模型类：Opt [38]、GPT2 [39]、Qwen [40] 与 Pythia [41]，规模约从 100M 到 70B（细节见附录 C）。（a）发现 $W_{i}/\|W_{i}\|_{2}$ 的均方重叠大致服从 $1/m$ 缩放，其中 $W$ 为语言模型头。（b）模型类用与面板 a 相同的颜色，评估数据集 [42, 43, 44, 45] 用形状。与模型规模相关的损失按幂律拟合，得经验 $\alpha_{m}=0.91\pm 0.04$，接近 $1$（细节见附录 D.8）。

Finally, we explore how the phenomena found in our toy model might be relevant to real LLMs [38, 39, 40, 41]. As a simple mapping, we treat tokens as atomic features, with data dimension $n$ equal to the vocabulary size. The model dimension $m$ for LLMs is known. We analyze the language model head, denoted by the weight matrix $W$. Through the norm and interference distributions of the rows of $W$, we claim LLMs are in superposition (Appendix D.8). If we measure token frequency, it follows a power law with exponent $\alpha$ close to $1$ (Appendix D.8). We conclude, based on the knowledge from toy models, that LLMs operate in a superposition regime, and expect loss to be dominated by small overlaps of the strongly represented $\sim\sqrt{1/m}$. We next calculated the mean squared overlaps of normalized rows $W_{i}/\|W_{i}\|_{2}$, and found they roughly obey $1/m$ scaling (Figure 8a). We argue that cross-entropy loss, given that the overlaps are small in absolute value, can be expanded and approximately scales as the mean square overlaps (Appendix A.2). We therefore expect the loss of representation-limited LLMs to have $1/m$ scaling. LLM losses are close to a linear function of $1/m$ (Appendix D.8). Yet when $m\to\infty$, the extrapolation of losses does not hit $0$. The non-zero intersection can be due to intrinsic uncertainty in language. Increasing model sizes decreases “wrong" interferences but cannot eliminate uncertainty in the data. So, as in previous papers where loss is decomposed into model size part, dataset size part, and a constant [3], we fit our loss values by the following,

最后，我们探索玩具模型中的现象如何与真实 LLM 相关 [38, 39, 40, 41]。简单映射下，把 token 当作原子特征，数据维 $n$ 等于词表大小；LLM 的模型维 $m$ 已知。我们分析语言模型头权矩阵 $W$。通过 $W$ 行的范数与干扰分布，我们主张 LLM 处于叠加（附录 D.8）。若度量 token 频率，它服从指数 $\alpha$ 接近 $1$ 的幂律（附录 D.8）。基于玩具模型知识，我们结论：LLM 在叠加区运行，并期望损失由强表示者的小重叠 $\sim\sqrt{1/m}$ 主导。接着计算归一化行 $W_{i}/\|W_{i}\|_{2}$ 的均方重叠，发现大致服从 $1/m$ 缩放（图 8a）。我们论证：在重叠绝对值较小时，交叉熵可展开并近似按均方重叠缩放（附录 A.2）。因此期望表征受限 LLM 的损失具有 $1/m$ 缩放。LLM 损失接近 $1/m$ 的线性函数（附录 D.8）；但当 $m\to\infty$ 时，外推并不过零。非零截距可来自语言内在不确定性——增大模型减小「错误」干扰，却无法消除数据中的不确定性。故如先前把损失分解为模型规模部分、数据集规模部分与常数 [3]，我们按下式拟合损失：

$$L=C_{m}/m^{\alpha_{m}}+L_{\backslash m}, \tag{8}$$

where the model size part $C_{m}/m^{\alpha_{m}}$ is universal (model size is a function of $m$), and $L_{\backslash m}$ contains loss irrelevant to model size, depending on the evaluation dataset and model class. The fitting yields $\alpha_{m}=0.91\pm 0.04$ (Figure 8b). We inferred from the Chinchilla models [3] that due to model size $N\propto m^{2.52\pm 0.03}$ (Appendix D.8), $\alpha_{m}=(2.52\pm 0.03)\times\alpha_{N}=0.88\pm 0.06$, where $\alpha_{N}=0.35\pm 0.02$ [46] is the power-law exponent of loss with model size. The exponents $\alpha_{m}$ from LLMs are close to 1.

其中模型规模部分 $C_{m}/m^{\alpha_{m}}$ 具有普适性（模型规模是 $m$ 的函数），$L_{\backslash m}$ 含与模型规模无关的损失，依赖评估数据集与模型类。拟合得 $\alpha_{m}=0.91\pm 0.04$（图 8b）。从 Chinchilla 模型 [3] 推断：因模型规模 $N\propto m^{2.52\pm 0.03}$（附录 D.8），$\alpha_{m}=(2.52\pm 0.03)\times\alpha_{N}=0.88\pm 0.06$，其中 $\alpha_{N}=0.35\pm 0.02$ [46] 为损失随模型规模的幂律指数。LLM 的 $\alpha_{m}$ 均接近 1。

In conclusion, we presented a comprehensive picture through the toy model, at different superposition degrees and data structures, of when the loss scales as a power law with model dimension or size, and what governs the scaling exponent. We confirmed that LLMs operate in the strong superposition regime. Moreover, LLMs [38, 39, 40, 41] quantitatively match toy model predictions: the model exponent $\alpha_{m}$ is close to $1$, and representation overlap scales as $\sqrt{1/m}$. The inferred exponent $\alpha_{m}$ for Chinchilla models [3] is also close to 1. Superposition is therefore an important mechanism underlying the observed scaling laws.

总之，我们通过玩具模型在不同叠加程度与数据结构下给出全景：损失何时随模型维或规模呈幂律，以及缩放指数由什么决定。我们确认 LLM 处于强叠加区；且 LLM [38, 39, 40, 41] 与玩具模型预测定量吻合：模型指数 $\alpha_{m}$ 接近 $1$，表征重叠按 $\sqrt{1/m}$ 缩放。从 Chinchilla 模型 [3] 推断的 $\alpha_{m}$ 也接近 1。因此叠加是观测到的缩放律背后的重要机制。

4 Related works

Neural scaling laws were first characterized empirically [2], demonstrating that for LLMs, the cross-entropy loss improves predictably as a power-law with increased model size (parameters), dataset size, or compute, over multiple orders of magnitude. This finding is built on earlier observations (e.g. [47]) that deep learning performance scales in a smooth power-law fashion with data and model growth. Key works showed the surprisingly universal nature of such scaling behaviors across architectures and tasks [2, 3, 4], directing further development of LLMs.

神经缩放律首先被经验刻画 [2]：对 LLM，交叉熵损失随模型规模（参数）、数据集规模或算力增大，在多个数量级上可预测地按幂律改善。该发现建立在更早观察之上（如 [47]）：深度学习表现随数据与模型增长呈平滑幂律。关键工作展示了此类缩放行为跨架构与任务的惊人普适性 [2, 3, 4]，并指引了 LLM 的进一步发展。

Several heuristic toy models have been proposed to explain neural scaling laws. One common view is that models aim to fit data manifolds or functions, and the scaling exponents depend strongly on the structure of the data [14, 15]. Another group of models assumes the network learns discrete features or skills [18, 19], whose importance follows a power-law distribution, giving results the same as ours in the weak superposition regime. One toy model predicts that loss scales inversely with model width [24], arguing that parameters independently perform the same task with noise, and the scaling follows from the central limit theorem. However, this model applies mainly in the overparameterized regime.

若干启发式玩具模型被提出以解释神经缩放律。一种常见观点是模型旨在拟合数据流形或函数，缩放指数强烈依赖数据结构 [14, 15]。另一组模型假定网络学习离散特征或技能 [18, 19]，其重要性服从幂律分布，结果与我们在弱叠加区相同。有玩具模型预测损失与模型宽度成反比 [24]，论证参数独立完成同一任务并带噪声，缩放来自中心极限定理；但该模型主要适用于过参数化区间。

More formal approaches from statistical learning theory still rely on similar heuristics. The scaling behavior depends on how the limits of dataset size and model size are taken. When the dataset is fixed and model size grows to infinity, the system is variance-limited, and loss scales as $1/m$ by central limit theorem arguments [15]. When the dataset size grows to infinity first, the loss scaling enters the resolution-limited regime. In kernel methods, this leads to $\alpha_{m}=\alpha-1$ [15, 16, 17], consistent with our weak superposition regime. Here, $\alpha$ is the exponent of the power-law decay of kernel eigenvalues, which can be interpreted as abstract feature importance. This regime has also been described as fitting the data manifold [15]. Thus, previous studies mainly address weak superposition or overfitting regimes, while our work reveals new behaviors in the underfitting (where the number of features far exceeds the model dimension) and strong superposition regime, which might be more relevant to LLMs.

统计学习理论中更形式的方法仍依赖类似启发式。缩放行为取决于如何取数据集规模与模型规模的极限。当数据集固定、模型规模趋于无穷时，系统方差受限，由中心极限定理损失按 $1/m$ 缩放 [15]。当数据集规模先趋于无穷，损失进入分辨率受限区间。在核方法中，这导致 $\alpha_{m}=\alpha-1$ [15, 16, 17]，与我们的弱叠加区一致；此处 $\alpha$ 是核特征值幂律衰减的指数，可解释为抽象特征重要性。该区间也被描述为拟合数据流形 [15]。因此先前研究主要处理弱叠加或过拟合区间，而我们的工作揭示了欠拟合（特征数远超模型维）与强叠加区的新行为——这可能与 LLM 更相关。

Our toy model is Anthropic’s model of superposition [26], with changes in data sampling. The original study explored how data structure influences superposition but did not explicitly control it. Related models have appeared in compressed sensing [48, 49, 50, 51] and neural information processing [52, 53], though key differences arise due to distinct contexts and objectives.

我们的玩具模型是 Anthropic 的叠加模型 [26]，数据采样有改动。原研究探索了数据结构如何影响叠加，但未显式控制叠加。相关模型亦出现在压缩感知 [48, 49, 50, 51] 与神经信息处理 [52, 53] 中，但因语境与目标不同而有关键差异。

5 Discussion

We acknowledge that our work is built on observations of the toy model and shallow theoretical analysis without rigorously solving the toy model. We are thus limited to explaining some deeper questions. For example, Equation (7) is an observation. Although we explained the two extremes, $\alpha_{m}=1$ and $\alpha_{m}=2(\alpha-1)$, and provided heuristics for a transition, we cannot predict the exact transition.

我们承认工作建立在玩具模型观察与浅层理论分析之上，并未严格求解玩具模型，因而对某些更深层问题解释有限。例如式 (7) 是观察结果：虽解释了 $\alpha_{m}=1$ 与 $\alpha_{m}=2(\alpha-1)$ 两个极端并给出过渡启发式，但无法预测精确过渡点。

Neural scaling laws also include scaling laws with dataset size and with training steps, which we did not study. At each step, a fixed number of new data are used for optimization. So, we expect the scaling with the total data amount and that with training steps will be the same, similar to the results at weak superposition [19]. However, in the strong superposition regime, data or training step scaling is related to angle distribution and how angles between representations evolve, which cannot be easily explained without rigorous solving. Future works with better theoretical analysis of our toy model are needed to answer these detailed or extended questions.

神经缩放律还包括随数据集规模与训练步数的缩放，本文未研究。每步用固定数量的新数据做优化，故期望随总数据量与随训练步数的缩放相同，类似弱叠加下的结果 [19]。但在强叠加区，数据或训练步缩放与角度分布以及表征间角度如何演化相关，不经严格求解不易解释。需要未来对玩具模型做更好理论分析，以回答这些细节或扩展问题。

Our analysis suggests LLMs are in the strong superposition regime, but the underlying reasons were not studied. Specifically, LLMs do not need weight growth $\gamma<0$ to have strong superposition. The use of cross-entropy loss or softmax function may be important — minimizing the interference from an unimportant token prefers to make the representation non-zero with negative overlaps with others. Another contributing factor may be that $m^{2}/2>n$ in LLMs, which could make it easier for a nearly mutually orthogonal configuration to form, thereby favoring superposition.

我们的分析表明 LLM 处于强叠加区，但底层原因未研究。具体地，LLM 无需权重增长 $\gamma<0$ 即可有强叠加。交叉熵损失或 softmax 函数可能重要——最小化来自不重要 token 的干扰时，偏好让表征非零且与其他表征负重叠。另一可能因素是 LLM 中 $m^{2}/2>n$，更易形成近相互正交配置，从而青睐叠加。

We focused on representation loss in this study, yet LLMs should also have losses due to parsing or processing in the transformer layers. As a conceptual discussion, we imagine that the loss associated with model size can be written as

本研究聚焦表征损失，但 LLM 还应有 transformer 层中解析或处理带来的损失。作为概念讨论，我们设想与模型规模相关的损失可写为

$$C_{m}/m^{\alpha_{m}}=f_{m}(m)+f_{\ell}(\ell), \tag{9}$$

where $\ell$ is the depth of the LLM, $f_{m}$ and $f_{\ell}$ are two functions capturing the loss due to representation and parsing, respectively. We can write the equality because $\ell$ depends on $m$ in LLMs [38, 39, 40, 41]. Given model size $N$, $m$ and $\ell$ are constrained (roughly, $N\propto m^{2}\ell$). There is an optimal $m$-$\ell$ relationship such that the loss $f_{m}(m)+f_{\ell}(\ell)$ can be minimized given $N$ [54]. At this optimal $m$-$\ell$ relationship, $f_{m}(m)$ and $f_{\ell}(\ell)$ should be balanced. One can see that the empirical $m$-$\ell$ relationship is not optimal according to [54] but is close. Therefore, following such $m$-$\ell$ relation, we expect $f_{\ell}(\ell)$ to be similar to $f_{m}(m)$. And if $f_{m}(m)\sim 1/m$ due to superposition and $f_{\ell}(\ell)$ is similar, we can measure an empirical $\alpha_{m}\approx 1$ from data, which is true. So, we conclude that superposition is an important mechanism underlying the loss and scaling laws we see. However, since we do not know the function $f_{\ell}(\ell)$, we cannot predict the optimal $m$-$\ell$ relation or the exponent $\alpha_{N}$.

其中 $\ell$ 为 LLM 深度，$f_{m}$ 与 $f_{\ell}$ 分别刻画表征与解析带来的损失。可写成等式是因为 LLM 中 $\ell$ 依赖 $m$ [38, 39, 40, 41]。给定模型规模 $N$，$m$ 与 $\ell$ 受约束（大致 $N\propto m^{2}\ell$）。存在最优 $m$–$\ell$ 关系，使给定 $N$ 时损失 $f_{m}(m)+f_{\ell}(\ell)$ 可最小化 [54]。在该最优关系上，$f_{m}(m)$ 与 $f_{\ell}(\ell)$ 应平衡。可见经验 $m$–$\ell$ 关系按 [54] 并非最优但接近。因此沿该关系，期望 $f_{\ell}(\ell)$ 与 $f_{m}(m)$ 相似；若因叠加有 $f_{m}(m)\sim 1/m$ 且 $f_{\ell}(\ell)$ 类似，则可从数据测得经验 $\alpha_{m}\approx 1$——事实如此。故我们结论叠加是所见损失与缩放律背后的重要机制。但因不知函数 $f_{\ell}(\ell)$，无法预测最优 $m$–$\ell$ 关系或指数 $\alpha_{N}$。

A natural future direction is to study the parsing-limited scaling (i.e, $f_{\ell}(\ell)$ function). We hypothesize that parsing-limited scaling may resemble the scaling behavior observed when fitting functions or manifolds. The open-source LLMs we examined, as well as those studied in prior neural scaling law papers [2, 3], span model sizes from 100M to 70B parameters, where width (model dimension) varies. However, in the regime from 70B to 700B [38, 39, 40, 41], model scaling seems to focus on increasing depth. We therefore expect parsing-limited scaling to become important for models larger than 70B. It is also plausible that the observed scaling of inference time [55] is connected to this parsing-limited regime. The study of neural scaling laws can be refined by distinguishing between width-limited and depth-limited regimes. In each regime, there should be loss decay behaviors with model size, dataset size, and training steps, highlighting the need for further investigation.

自然的未来方向是研究解析受限缩放（即 $f_{\ell}(\ell)$ 函数）。我们假设解析受限缩放可能类似拟合函数或流形时观测到的缩放。我们所检开源 LLM 以及先前神经缩放律论文 [2, 3] 中的模型，规模从 100M 到 70B，宽度（模型维）在变；而在 70B 到 700B 区间 [38, 39, 40, 41]，模型缩放似乎聚焦加深。因此期望解析受限缩放对大于 70B 的模型变得重要。观测到的推理时间缩放 [55] 也可能与该解析受限区间相关。神经缩放律研究可通过区分宽度受限与深度受限区间来细化；每个区间都应有随模型规模、数据集规模与训练步数的损失衰减行为，凸显进一步研究的必要。

Beyond explaining existing phenomena, our results may offer guidance for future LLM development and training strategies. If our framework accounts for the observed neural scaling laws [3], we suggest that this kind of scaling is reaching its limits — not because increasing model dimension is impossible, but because it is inefficient. Recognizing that superposition benefits LLMs, we propose that encouraging superposition could enable smaller models to match the performance of larger ones and make training more efficient. Recent architectures such as nGPT [56], which constrain hidden states and weight matrix rows to lie on the unit sphere, demonstrate improved performance and faster training. We interpret these methods as analogous to our weight growth mechanism, both of which promote superposition. Conversely, weight decay is expected to suppress superposition. Prior work has shown that weight decay in LLMs is used primarily for training stability rather than performance gains. Alternative optimization techniques that stabilize training without weight decay have shown promising results [57], potentially due to enhanced superposition. In summary, promoting strong superposition may be a fruitful direction for the design of both LLM architectures and optimization algorithms.

除解释既有现象外，结果或可为未来 LLM 发展与训练策略提供指引。若我们的框架解释了观测到的神经缩放律 [3]，我们建议这类缩放正接近极限——并非不能再增大模型维，而是效率低下。认识到叠加有利于 LLM，我们提议鼓励叠加或可使更小模型匹配更大模型的表现，并使训练更高效。近期如 nGPT [56] 等架构把隐状态与权矩阵行约束在单位球上，展示更好表现与更快训练；我们将其解读为类似我们的权重增长机制，二者都促进叠加。相反，权重衰减预期抑制叠加。先前工作显示 LLM 中权重衰减主要用于训练稳定而非性能增益；不靠权重衰减也能稳定训练的替代优化技术已有有希望结果 [57]，或因增强了叠加。总之，促进强叠加或是设计 LLM 架构与优化算法的有益方向。

As a side note, while pre-training loss is a key indicator of model performance, it should not be the sole metric of interest. At the same loss level, LLMs with different degrees of superposition may exhibit differences in emergent abilities such as reasoning or trainability via reinforcement learning [58]. Understanding these differences is an important direction for future research.

附带一提：预训练损失虽是模型表现的关键指标，不应是唯一关注度量。在相同损失水平下，叠加程度不同的 LLM 在推理或经强化学习可训性等涌现能力上或有差异 [58]。理解这些差异是重要未来方向。

In conclusion, we demonstrated that superposition of representations can give rise to fast power law decay of loss with model size — a phenomenon that holds across a wide range of data structures and may help explain the neural scaling laws observed in LLMs [3]. Our results contribute to a deeper understanding of modern artificial intelligence systems. This work also opens rich directions for future research. We hope our insights will support the continued development and training of more capable and efficient LLMs.

总之，我们展示表征叠加可带来损失随模型规模的快速幂律衰减——该现象在广泛数据结构上成立，或有助于解释 LLM 中观测到的神经缩放律 [3]。结果加深对现代人工智能系统的理解，并开辟丰富未来研究方向。希望这些洞见支持继续发展与训练更强、更高效的 LLM。

References

[1] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877–1901, 2020.

[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

[3] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Jack W Rae, Alana Lai, Joel Wang, Katie Millican, Susannah Young, Olivier Tieleman, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 , 2022.

[4] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, 2020.

[5] OpenAI. Gpt-4 technical report, 2023. https://openai.com/research/gpt-4.

[6] Haotong Qin, Ge-Peng Ji, Salman Khan, Deng-Ping Fan, Fahad Shahbaz Khan, and Luc Van Gool. How good is google bard’s visual understanding? an empirical study on open challenges. arXiv preprint arXiv:2307.15016 , 2023. https://arxiv.org/abs/2307.15016.

[7] Aitor Lewkowycz, Pasquale Minervini, Jacob Andreas, et al. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858 , 2022. https://arxiv.org/abs/2206.14858.

[8] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Andrew Hartshorn, Elvis Saravia, Andrew Poulton, et al. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 , 2022. https://arxiv.org/abs/2211.09085.

[9] Stephen Wolfram. Wolfram|alpha as the computation engine for gpt models, 2023. https://www.wolfram.com/wolfram-alpha-openai-plugin.

[10] Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature , 2024. https://www.nature.com/articles/s41586-024-00001-2.

[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021. https://arxiv.org/abs/2107.03374.

[12] GitHub. Github copilot: Your ai pair programmer, 2022. https://github.com/features/copilot.

[13] Jack W Rae, Sebastian Borgeaud, Trevor Cai, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 , 2021. https://arxiv.org/abs/2112.11446.

[14] Utkarsh Sharma and Jared Kaplan. Neural scaling laws: A low-dimensional perspective. Journal of Machine Learning Research , 23(228):1–29, 2022.

[15] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences , 121(27):e2311878121, 2024.

[16] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning , pages 1024–1034. PMLR, 2020.

[17] Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859 , 2022.

[18] Marcus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074 , 2021.

[19] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems , 36:28699–28722, 2023.

[20] Ziming Liu, Yizhou Liu, Eric J Michaud, Jeff Gore, and Max Tegmark. Physics of skill learning. arXiv preprint arXiv:2501.12391 , 2025.

[21] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205.10487 , 2022.

[22] Ari Brill. Neural scaling laws rooted in the data distribution. arXiv preprint arXiv:2412.07942 , 2024.

[23] Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm. Journal of Statistical Mechanics: Theory and Experiment , 2020(12):124001, 2020.

[24] Jinyeop Song, Ziming Liu, Max Tegmark, and Jeff Gore. A resource model for neural scaling law, 2024.

[25] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy, 2018.

[26] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread , 2022.

[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

[28] L. Welch. Lower bounds on the maximum cross correlation of signals. IEEE Transactions on Information Theory , 20(3):397–399, 1974.

[29] Peter G Casazza and Gitta Kutyniok. Finite frames: Theory and applications . Springer Science & Business Media, 2012.

[30] Thomas Strohmer and Robert W Heath. Grassmannian frames with applications to coding and communication. Applied and Computational Harmonic Analysis , 14(3):257–275, 2003.

[31] Matthew Fickus, Dustin G Mixon, and Janet C Tremain. Steiner equiangular tight frames. Linear algebra and its applications , 436(5):1014–1027, 2012.

[32] Joseph M Renes, Robin Blume-Kohout, A J Scott, and Carlton M Caves. Symmetric informationally complete quantum measurements. Journal of Mathematical Physics , 45(6):2171–2180, 2004.

[33] Yizhou Liu and John B. DeBrota. Relating measurement disturbance, information, and orthogonality. Phys. Rev. A , 104:052216, Nov 2021.

[34] Yizhou Liu and Shunlong Luo. Quantifying unsharpness of measurements via uncertainty. Phys. Rev. A , 104:052227, Nov 2021.

[35] Yizhou Liu, Shunlong Luo, and Yuan Sun. Total, classical and quantum uncertainties generated by channels. Theoretical and Mathematical Physics , 213(2):1613–1631, 2022.

[36] Vardan Papyan, X.Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652–24663, 2020.

[37] Vignesh Kothapalli. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041 , 2022.

[38] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.

[39] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

[40] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 , 2024.

[41] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.

[42] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 , 2016.

[43] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

[44] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1–67, 2020.

[45] Yukun Zhu, Ryan Kiros, Richard S Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv preprint arXiv:1506.06724 , 2015.

[46] Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102 , 2024.

[47] Joel Hestness, Sharan Narang, Niki Ardalani, Greg Diamos, Heewoo Jun, Hossein Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yan Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 , 2017.

[48] David L Donoho. Compressed sensing. IEEE Transactions on Information Theory , 52(4):1289–1306, 2006.

[49] Emmanuel J Candès, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory , 52(2):489–509, 2006.

[50] Richard G Baraniuk. Compressive sensing. IEEE Signal Processing Magazine , 24(4):118–121, 2007.

[51] Madhu S Advani and Surya Ganguli. Statistical mechanics of optimal convex inference in high dimensions. Physical Review X , 6(3):031034, 2016.

[52] Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature , 381(6583):607–609, 1996.

[53] Behtash Babadi and Haim Sompolinsky. Sparseness and expansion in sensory representations. Neuron , 83(5):1213–1226, 2014.

[54] Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467 , 2020.

[55] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 , 2024.

[56] Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. ngpt: Normalized transformer with representation learning on the hypersphere. arXiv preprint arXiv:2410.01131 , 2024.

[57] Yizhou Liu, Ziming Liu, and Jeff Gore. Focus: First order concentrated updating scheme. arXiv preprint arXiv:2501.12243 , 2025.

[58] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Appendix A Theoretical analysis

A.1 Toy model loss

We provide a simple analysis for toy model loss scaling. The expected loss in the weak superposition regime is well explained by Equation (4). We do not need to repeat it here.

我们对玩具模型损失缩放给出简单分析。弱叠加区的期望损失已由式 (4) 很好解释，此处不重复。

In the strong superposition regime, we consider an even special data sampling, where data $x$ is sampled such that each data point has and only has one activated feature. The frequency for feature $i$ to be activated is still $p_{i}$. After determining which feature is activated, say $i$, we still sample $x_{i}$ as $v_{i}$ from $U(0,2)$. This sampling is different from the experiments. Yet, since we learned that activation density does not affect scaling exponent (Figure 7), we expected this analysis to predict at least the scaling exponent. Under the assumptions, we have

在强叠加区，我们考虑更特殊的数据采样：每条数据 $x$ 有且仅有一个被激活特征；特征 $i$ 被激活的频率仍为 $p_{i}$。确定激活特征为 $i$ 后，仍从 $U(0,2)$ 采样 $x_{i}$ 为 $v_{i}$。该采样与实验不同；但因已知激活密度不影响缩放指数（图 7），我们期望该分析至少能预测缩放指数。在这些假设下，有

$$L=\sum_{i=1}^{n}p_{i}\Big\langle\sum_{j\neq i}\mathrm{ReLU}^{2}(W_{j}\cdot W_{i}v_{i}+b_{j})+(\mathrm{ReLU}(W_{i}\cdot W_{i}v_{i}+b_{i})-v_{i})^{2}\Big\rangle_{v_{i}}. \tag{10}$$

We are unable to solve for the optimal $W$ and $b$ such that this loss $L$ is minimized. Yet, it is easy to see that we want $\|W_{i}\|_{2}^{2}$ to be close to $1$, $W_{j}\cdot W_{i}$ to be as small as possible, and $b_{j}$ to be small negative values of the same order of magnitude as $W_{i}\cdot W_{j}$, such that the interference terms $\mathrm{ReLU}(W_{j}\cdot W_{i}v_{i}+b_{j})$ may vanish and the recovered feature value $\mathrm{ReLU}(W_{i}\cdot W_{i}v_{i}+b_{i})$ can be close to the real one $v_{i}$.

我们无法求解使损失 $L$ 最小的最优 $W$ 与 $b$。但不难看出：希望 $\|W_{i}\|_{2}^{2}$ 接近 $1$，$W_{j}\cdot W_{i}$ 尽量小，且 $b_{j}$ 为与 $W_{i}\cdot W_{j}$ 同量级的小负值，从而使干扰项 $\mathrm{ReLU}(W_{j}\cdot W_{i}v_{i}+b_{j})$ 可能消失，恢复的特征值 $\mathrm{ReLU}(W_{i}\cdot W_{i}v_{i}+b_{i})$ 接近真实 $v_{i}$。

We now review our proposed configuration, consisting of strongly represented and weakly represented features. The first $\phi_{1}n$ most important features are considered to be strongly represented, whose overlap with any other representation scales as $\sqrt{1/m}$. The rest of the features are weakly represented and squeezed into a small angle such that they have small overlaps with the strongly represented, while they can have large overlaps with each other. With such a configuration, the first $\phi_{1}n$ terms in the summation of Equation (10) will scale as $1/m$ since each term $\langle\cdots\rangle_{v_{i}}$ scales as $1/m$. The rest of the terms in Equation (10), in the worst scenario that the terms $\langle\cdots\rangle_{v_{i}}$ do not decrease obviously with $m$, will be proportional to $\sum_{i=\phi_{i}n}^{n}p_{i}$. If the strongly represented features dominate, we can have $1/m$ scaling for the loss. On the contrary, when the weakly represented dominate, we expect the loss to have scaling like $\sum_{i=\phi_{i}n}^{n}p_{i}$. More specifically, if $p_{i}\sim 1/i^{\alpha}$ ($\alpha>1$) and we use $m^{2}/2$ to approximate $\phi_{i}n$, the loss scales as $1/m^{2(\alpha-1)}$.

现在回顾我们提出的配置，由强表示与弱表示特征组成。最重要的前 $\phi_{1}n$ 个特征视为强表示，其与任意其他表征的重叠按 $\sqrt{1/m}$ 缩放；其余特征弱表示，被挤入小角度，使它们与强表示重叠小，彼此之间则可有大重叠。在此配置下，式 (10) 求和中前 $\phi_{1}n$ 项按 $1/m$ 缩放，因每项 $\langle\cdots\rangle_{v_{i}}$ 按 $1/m$ 缩放。其余项在最坏情形——$\langle\cdots\rangle_{v_{i}}$ 不随 $m$ 明显下降——下正比于 $\sum_{i=\phi_{i}n}^{n}p_{i}$。若强表示主导，损失可有 $1/m$ 缩放；若弱表示主导，期望损失按 $\sum_{i=\phi_{i}n}^{n}p_{i}$ 缩放。更具体地，若 $p_{i}\sim 1/i^{\alpha}$（$\alpha>1$）并用 $m^{2}/2$ 近似 $\phi_{i}n$，损失按 $1/m^{2(\alpha-1)}$ 缩放。

A.2 Cross-entropy loss

We provide the reason why cross-entropy loss also scales as squared overlaps. We consider the last hidden state after going through the normalization layer is $W_{i}/\|W_{i}\|_{2}$, such that the output should be the $i$th token. By constructing such an example, we ignore possible loss due to parsing but focus on the loss just due to representation. The loss from this data point is

我们给出交叉熵损失也按重叠平方缩放的理由。考虑经归一化层后的最后隐状态为 $W_{i}/\|W_{i}\|_{2}$，使得输出应为第 $i$ 个 token。通过构造该例子，我们忽略可能的解析损失，只聚焦纯表征损失。该数据点的损失为

$$L=-\ln\frac{e^{\|W_{i}\|_{2}}}{\sum_{j}e^{W_{i}\cdot W_{j}/\|W_{i}\|_{2}}}=\ln\Big[1+\sum_{j\neq i}e^{W_{i}\cdot W_{j}/\|W_{i}\|_{2}-\|W_{i}\|_{2}}\Big]. \tag{11}$$

We assume that $W_{i}\cdot W_{j}/\|W_{i}\|_{2}$ is much smaller than $1$ since we know the overlap scale as $1/m$. We then approximate the loss via Taylor expansion

因已知重叠按 $1/m$ 缩放，假定 $W_{i}\cdot W_{j}/\|W_{i}\|_{2}$ 远小于 $1$。于是用 Taylor 展开逼近损失

$$L=\ln\Big[1+(n-1)e^{-\|W_{i}\|_{2}}+\sum_{j\neq i}[W_{i}\cdot W_{j}/\|W_{i}\|_{2}+(W_{i}\cdot W_{j}/\|W_{i}\|_{2})^{2}/2]e^{-\|W_{i}\|_{2}}\Big]. \tag{12}$$

In the first thought, the summation $\sum_{j\neq i}W_{i}\cdot W_{j}$ should be zero since there are positive and negative overlaps distributed evenly if the vectors span the whole space. But in language, one sentence can have different continuations, connecting different tokens. For example, both putting “cats" or “dogs" after “I like" are legit. The existence of data “I like" then will tend to squeeze different tokens closer to each other. The summation $W_{i}\cdot W_{j}$ should be a small positive constant $\epsilon_{D,i}$ related to the correlation in data. The reason we keep the second-order term is clear now as they are the lowest order terms related to model sizes. We keep expanding the $\ln$ function and have

乍看，$\sum_{j\neq i}W_{i}\cdot W_{j}$ 应为零——若向量张满整个空间，正负重叠均匀分布。但在语言中，同一句子可有不同续写，连接不同 token。例如「I like」后接「cats」或「dogs」都合理。存在数据「I like」会倾向把不同 token 挤近。故求和 $W_{i}\cdot W_{j}$ 应为与数据相关相关的小正常数 $\epsilon_{D,i}$。保留二阶项的理由现在清楚：它们是与模型规模相关的最低阶项。继续展开 $\ln$ 得

$$L=(n-1)e^{-\|W_{i}\|_{2}}+\frac{\epsilon_{D,i}e^{-\|W_{i}\|_{2}}}{\|W_{i}\|_{2}}+\frac{1}{2}\sum_{j\neq i}\left(\frac{W_{i}\cdot W_{j}}{\|W_{i}\|_{2}}\right)^{2}e^{-\|W_{i}\|_{2}}. \tag{13}$$

The part related to the model size is mainly