Not All Tokens Are What You Need for Pretraining

Rho-1: Not All Tokens Are What You Need

Rho-1：并非所有 token 都是预训练所需

Zhenghao Lin Zhibin Gou Yeyun Gong Xiao Liu Yelong Shen † † thanks: Equal contribution. See author contributions for details. Work done during their internships at Microsoft Research Asia. 🖂:˜ zhenghaolin@stu.xmu.edu.cn ;˜˜ zebgou@gmail.com † † thanks: Correspondence authors. Affiliation: Microsoft https://aka.ms/rho Ruochen Xu Chen Lin Yujiu Yang Jian Jiao Nan Duan Weizhu Chen Affiliation: Microsoft https://aka.ms/rho Affiliation: Xiamen University Affiliation: Tsinghua University Affiliation: Shanghai AI Laboratory

Abstract

摘要

Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that “Not all tokens in a corpus are equally important for language model training” . Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called Rho-1. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring tokens using a reference model, and then training the language model with a focused loss on tokens with higher scores. When continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively — matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when continual pretraining on 80B general tokens, Rho-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both data efficiency and performance of the language model pre-training.

以往语言模型预训练方法对所有训练 token 一视同仁地施加下一 token 预测损失。我们质疑这一惯例，并提出： 「语料中并非所有 token 对语言模型训练同等重要」 。我们首先分析语言模型在 token 级的训练动态，发现不同 token 呈现出截然不同的损失轨迹。基于这些观察，我们提出新的语言模型 Rho-1。与在语料上学习预测每一个下一 token 的传统 LM 不同，Rho-1 采用选择性语言建模（Selective Language Modeling，SLM），只在与目标分布对齐的有用 token 上训练。方法先用参考模型给 token 打分，再仅对高分 token 施加聚焦损失。在 15B OpenWebMath 语料上持续预训练时，Rho-1 在 9 项数学任务上的少样本准确率最高可绝对提升 30%。微调后，Rho-1-1B 与 7B 在 MATH 上分别达到 40.6% 与 51.8% 的当时最优——仅用 DeepSeekMath 约 3% 的预训练 token 即与之持平。此外，在 80B 通用 token 上持续预训练时，Rho-1 在 15 项多样任务上平均提升 6.8%，同时提高了语言模型预训练的数据效率与性能。

Figure 1: We continual pretrain 1B and 7B LMs with 15B OpenWebMath tokens. Rho-1 is trained with our proposed Selective Language Modeling (SLM), while baselines are trained using causal language modeling. SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster.

图 1：我们用 15B OpenWebMath token 持续预训练 1B 与 7B 语言模型。Rho-1 采用本文提出的选择性语言建模（SLM），基线采用因果语言建模。SLM 将 GSM8k 与 MATH 上的平均少样本准确率提升逾 16%，并以 5-10 倍速度达到基线性能。

Figure 2: Upper: Even an extensively filtered pretraining corpus contains token-level noise. Left: Previous Causal Language Modeling (CLM) trains on all tokens. Right: Our proposed Selective Language Modeling (SLM) selectively applies loss on those useful and clean tokens.

图 2： 上 ：即便经过充分过滤的预训练语料仍含 token 级噪声。 左 ：以往因果语言建模（CLM）对所有 token 训练。 右 ：本文提出的选择性语言建模（SLM）只对那些有用且干净的 token 施加损失。

1 Introduction

1 引言

Scaling up model parameters and dataset size has consistently elevated the next-token prediction accuracy in large language models, yielding significant advancements in artificial intelligence (Kaplan et al., 2020; Brown et al., 2020; OpenAI, 2023; Team et al., 2023). However, training on all available data is not always optimal or feasible. As a result, the practice of data filtering has become crucial, using various heuristics and classifiers (Brown et al., 2020; Wenzek et al., 2019) to select training documents. These techniques significantly improve data quality and boost model performance.

扩大模型参数与数据集规模持续提高了大语言模型的下一 token 预测准确率，推动了人工智能的显著进展（Kaplan 等，2020；Brown 等，2020；OpenAI，2023；Team 等，2023）。然而，对全部可得数据一视同仁地训练并非总是最优或可行。因此，数据过滤成为关键实践：用各种启发式规则与分类器（Brown 等，2020；Wenzek 等，2019）挑选训练文档。这些技术显著提升数据质量并增强模型性能。

However, despite thorough document-level filtering, high-quality datasets still contain many noisy tokens that can negatively affect training, as illustrated in Figure 2 (Upper). Removing such tokens might alter the text’s meaning, while overly strict filtering could exclude useful data (Welbl et al., 2021; Muennighoff et al., 2024) and lead to biases (Dodge et al., 2021; Longpre et al., 2023). Furthermore, research indicates that the distribution of web data does not inherently align with the ideal distribution for downstream applications (Tay et al., 2022; Wettig et al., 2023). For example, common corpus at the token level may include undesirable content like hallucinations or highly ambiguous tokens that are hard to predict. Applying the same loss to all tokens can lead to inefficient computation on non-essential tokens, potentially restricting LLMs from achieving more advanced levels of intelligence.

即便文档级过滤很彻底，高质量数据集中仍有大量噪声 token，可能损害训练，如图 2（上）所示。直接删除这些 token 可能改变文本含义，而过滤过严又会排除有用数据（Welbl 等，2021；Muennighoff 等，2024）并引入偏差（Dodge 等，2021；Longpre 等，2023）。此外，研究表明：网络数据的分布本身并不天然对齐下游应用的理想分布（Tay 等，2022；Wettig 等，2023）。例如，常见语料在 token 级可能包含幻觉内容，或难以预测的高歧义 token。对所有 token 施加相同损失，会在非必要 token 上浪费计算，可能限制 LLM 达到更高智能水平。

To explore how language models learn at the token level, we initially examined training dynamics, particularly how the token-level loss evolves during usual pretraining. In §2.1, we evaluated the model’s token perplexity at different checkpoints and categorized tokens into different types. Our findings reveal that significant loss reduction is limited to a select group of tokens. Many tokens are “easy tokens” that are already learned, and some are “hard tokens” that exhibit variable losses and resist convergence. These tokens can lead to numerous ineffective gradient updates.

为探究语言模型如何在 token 级学习，我们首先考察训练动态，尤其是常规预训练中 token 级损失如何演化。在第 2.1 节，我们评估不同检查点上的 token 困惑度，并将 token 分为若干类型。结果表明：显著的损失下降仅发生在一小部分 token 上。许多 token 是「容易 token」，早已学会；另一些是「困难 token」，损失波动大、难以收敛。这些 token 会带来大量无效梯度更新。

Based on these analyses, we introduce Rho-1 models trained with a novel Selective Language Modeling (SLM) objective [^1]. As shown in Figure 2 (Right), this approach inputs the full sequence into the model and selectively removes the loss of undesired tokens. The detailed pipeline is depicted in Figure 4: First, SLM trains a reference language model on high-quality corpora. This model establishes utility metrics to score tokens according to the desired distribution, naturally filtering out unclean and irrelevant tokens. Second, SLM uses the reference model to score each token in a corpus using its loss (§2.2). Finally, we train a language model only on those tokens that exhibit a high excess loss between the reference and the training model, selectively learning the tokens that best benefit downstream applications (§2.2).

基于上述分析，我们提出以新颖的选择性语言建模（SLM）目标训练的 Rho-1 模型[^1]。如图 2（右）所示，该方法把完整序列输入模型，再选择性地去掉不期望 token 上的损失。完整流程见图 4：首先，SLM 在高质量语料上训练参考语言模型；该模型按目标分布建立效用度量并为 token 打分，自然滤除不干净与不相关的 token。其次，SLM 用参考模型按损失对语料中每个 token 打分（第 2.2 节）。最后，我们仅在参考模型与训练模型之间超额损失（excess loss）较高的那些 token 上训练语言模型，有选择地学习最有利于下游应用的 token（第 2.2 节）。

We show through comprehensive experiments that SLM significantly enhances token efficiency during training and improves performance on downstream tasks. Furthermore, our findings indicate that SLM effectively identifies tokens relevant to the target distribution, resulting in improved perplexity scores on benchmarks for models trained with the selected tokens. §3.2 shows the effectiveness of SLM on math continual pretraining: both 1B and 7B Rho-1 outperform CLM-trained baselines by over 16% on the GSM8k and MATH datasets. SLM reaches baseline accuracy up to 10x faster, as shown in Figure 1. Remarkably, Rho-1-7B matches the state-of-the-art performance of DeepSeekMath-7B using only 15B tokens, compared to the 500B tokens required by DeepSeekMath. Upon fine-tuning, Rho-1-1B and 7B achieve 40.6% and 51.8% on MATH, respectively. Notably, Rho-1-1B is the first 1B LM to exceed 40% accuracy, nearing the early GPT-4’s CoT performance of 42.5%. §3.3 confirms the efficacy of SLM in general continual pretraining: Training Tinyllama-1B on 80B tokens with SLM improves 6.8% on average across 15 benchmarks, with gains over 10% in code and math tasks. In §3.4, we demonstrate that in settings without high-quality reference data, we can use SLM for self-referencing, leading to an average improvement of up to 3.3% in downstream tasks.

全面实验表明，SLM 显著提升训练时的 token 效率，并改善下游任务表现。我们还发现，SLM 能有效识别与目标分布相关的 token，使按所选 token 训练的模型在基准上的困惑度更好。第 3.2 节展示 SLM 在数学持续预训练上的效果：1B 与 7B 的 Rho-1 在 GSM8k 与 MATH 上均比 CLM 基线高出逾 16%。如图 1 所示，SLM 最高可快约 10 倍达到基线准确率。更值得注意的是，Rho-1-7B 仅用 15B token（相较 DeepSeekMath 所需的 500B）即达到与 DeepSeekMath-7B 相当的当时最优水平。微调后，Rho-1-1B 与 7B 在 MATH 上分别达到 40.6% 与 51.8%。尤其是 Rho-1-1B 成为首个准确率超过 40% 的 1B 级 LM，接近早期 GPT-4 的 CoT 表现 42.5%。第 3.3 节确认 SLM 在通用持续预训练中的效力：用 SLM 在 80B token 上训练 Tinyllama-1B，相对直接持续预训练在 15 项基准上平均提升 6.8%，代码与数学任务增益超过 10%。第 3.4 节表明：在没有高质量参考数据时，可用 SLM 做自参考（self-referencing），下游任务平均最多提升 3.3%。

Figure 3: The loss of four categories of tokens during pretraining. (a) shows the loss of H\toH, L\toH, H\toL, and L\toL tokens during pretraining. (b) and (c) show three cases of fluctuating tokens’ loss in L\toL and H\toH during pretraining, respectively.

图 3： 预训练过程中四类 token 的损失。（a）展示 H\toH、L\toH、H\toL 与 L\toL 四类 token 在预训练中的损失。（b）与（c）分别展示 L\toL 与 H\toH 中波动 token 损失的三个案例 。

2 Selective Language Modeling

2 选择性语言建模

2.1 Not All Tokens Are Equal: Training Dynamics of Token Loss Our investigation begins with a critical look at how individual tokens’ losses evolve during standard pre-training. We continue pre-training Tinyllama-1B with 15B tokens from OpenWebMath, saving checkpoints after every 1B tokens. We then evaluate token-level loss at these intervals using the validation set of approximately 320,000 tokens. Figure 3(a) reveals a striking pattern: tokens fall into four categories based on their loss trajectory—persistent high loss (H\toH), increasing loss (L\toH), decreasing loss (H\toL), and consistent low loss (L\toL). For further details on these categories, see §D.1. Our analysis uncovers that a mere 26% of tokens show a notable loss reduction (H\toL), while the majority (51%) remain in the L\toL category, indicating they have already been learned. Interestingly, 11% of the tokens are persistently challenging (H\toH), likely due to high aleatoric uncertainty (Hüllermeier and Waegeman, 2021). Additionally, 12% of tokens experience an unexpected loss increase (L\toH) during training.

2.1 并非所有 token 平等：token 损失的训练动态 我们首先仔细考察标准预训练中单个 token 损失如何演化。用 OpenWebMath 的 15B token 持续预训练 Tinyllama-1B，每 1B token 保存一次检查点；再在约 32 万 token 的验证集上评估这些间隔处的 token 级损失。图 3(a) 呈现出鲜明模式：按损失轨迹，token 落入四类——持续高损失（H\toH）、损失上升（L\toH）、损失下降（H\toL）与持续低损失（L\toL）。类别细节见第 D.1 节。分析表明：仅约 26% 的 token 出现显著损失下降（H\toL），多数（51%）留在 L\toL，说明它们已被学会。有趣的是，11% 的 token 持续困难（H\toH），很可能源于较高的偶然不确定性（aleatoric uncertainty）（Hüllermeier 与 Waegeman，2021）。另有 12% 的 token 在训练中出现出人意料的损失上升（L\toH）。

Our second observation is that a significant number of token losses exhibit persistent fluctuations, and resist convergence. The loss of many L\toL and H\toH tokens, as depicted in Figure 3 (b) and (c), show high variance during training. In §D.2, we visualize and analyze the content of these tokens and find that many of them are noisy, which is consistent with our hypothesis.

第二个观察是：大量 token 损失持续波动、难以收敛。如图 3(b)(c) 所示，许多 L\toL 与 H\toH token 的损失在训练中方差很大。第 D.2 节可视化并分析这些 token 的内容，发现其中许多是噪声，与我们的假设一致。

Consequently, we learn that the loss associated with each token during training does not decrease smoothly like the overall loss; instead, there is a complex training dynamic among different tokens. If we can select the appropriate tokens for the model to focus on during training, we may be able to stabilize the trajectory of the model’s training and enhance its data efficiency.

因此我们认识到：训练中每个 token 的损失并不像整体损失那样平滑下降；不同 token 之间存在复杂的训练动态。若能在训练中为模型选出合适的 token 去聚焦，便可能稳住训练轨迹并提升数据效率。

Overview Inspired by the practice of reference model in document-level filtering, we propose a simple pipeline of token-level data selection, termed “Selective Language Modeling (SLM)”. Our method comprises three steps, as depicted in Figure 4. We begin by training a reference model on a curated, high-quality dataset. This model then assesses the loss of each token within the pretraining corpus. In the final phase, we train the language model selectively, focusing on tokens with high excess loss between the training and reference model. The intuition is that tokens with high excess loss are more learnable and better aligned with the desired distribution, naturally excluding tokens that are either irrelevant or of low quality. Below, we provide a detailed description of each step.

概览 受文档级过滤中参考模型做法启发，我们提出简洁的 token 级数据选择流程，称为「选择性语言建模（SLM）」。方法分三步，如图 4 所示。先在精选的高质量数据上训练参考模型；再用该模型评估预训练语料中每个 token 的损失；最后阶段选择性训练语言模型，聚焦训练模型与参考模型之间超额损失较高的 token。直觉是：超额损失高的 token 更可学、也更对齐目标分布，从而自然排除无关或低质量 token。下文详述各步。

\mathcal{L}_{\text{RM}}

x_{i}

\mathcal{L}_{\text{RM}}(x_{i})=-\log P(x_{i}|x_{<i}) \tag{1}

\mathcal{L}_{\text{RM}}

\mathcal{L}_{\text{RM}}

Figure 4: The pipeline of Selective Language Modeling (SLM). SLM optimizes language model performance by concentrating on valuable, clean tokens during pre-training. It involves three steps: (Step 1) Initially, train a reference model on high-quality data. (Step 2) Then, score each token’s loss in a corpus using the reference model. (Step 3) Finally, selectively train the language model on tokens that have higher scores.

图 4： 选择性语言建模（SLM）流程。SLM 在预训练中聚焦有价值、干净的 token，以优化语言模型性能。共三步：（步骤 1）先在高质量数据上训练参考模型；（步骤 2）再用参考模型对语料中每个 token 的损失打分；（步骤 3）最后仅在得分较高的 token 上选择性训练语言模型 。

Selective Pretraining Note that Causal Language Modeling (CLM) employs the cross-entropy loss:

选择性预训练 注意，因果语言建模（CLM）采用交叉熵损失：

\mathcal{L}_{\text{CLM}}(\theta)=-\frac{1}{N}\sum_{i=1}^{N}\log P(x_{i}|x_{<i};\theta) \tag{2}

\mathcal{L_{\text{CLM}}}(\theta)

\mathcal{L_{\text{CLM}}}(\theta)

\mathcal{L}_{\Delta}(x_{i})=\mathcal{L}_{\theta}(x_{i})-\mathcal{L}_{\text{RM}}(x_{i}) \tag{3}

k\%

k\%

\mathcal{L}_{\text{SLM}}(\theta)=-\frac{1}{N*k\%}\sum_{i=1}^{N}I_{k\%}(x_{i})\cdot\log P(x_{i}|x_{<i};\theta) \tag{4}

N*k\%

N*k\%

I_{k%}(x_{i})=\begin{cases}1&\text{if }x_{i}\text{ ranks in the top }k%\text{ by }S(x_{i})\0&\text{otherwise}\end{cases} \tag{5}

\mathcal{L}_{\Delta}

\mathcal{L}_{\Delta}

3 Experiments

3 实验

We continually pretrained models in both mathematical and general domain and designed ablation and analysis experiments to understand the effectiveness of SLM.

我们在数学与通用两个领域持续预训练模型，并设计消融与分析实验以理解 SLM 的有效性。

Reference Model Training To train our mathematical reference model, we gathered a dataset of 0.5B high-quality, math-related tokens. This dataset is a blend of synthetic data from GPT (Yu et al., 2024; Huang et al., 2024) and manually curated data (Yue et al., 2024; Ni et al., 2024). For the general reference model, we compiled a corpus of 1.9B tokens from open-source datasets, such as Tulu-v2 (Ivison et al., 2023) and OpenHermes-2.5 (Teknium, 2023). We trained the reference models for 3 epochs. The maximum learning rate was set at 5e-5 for 1B models and 1e-5 for 7B models, applying a cosine decay schedule. We set the maximum sequence lengths to 2048 for 1B models and 4096 for 7B models, packing multiple samples into these lengths for model input. In all main experiments, we initialized the continual pretraining model and the reference model with the same base model.

参考模型训练 为训练数学参考模型，我们收集了 0.5B 高质量、数学相关 token，混合来自 GPT 的合成数据（Yu 等，2024；Huang 等，2024）与人工整理数据（Yue 等，2024；Ni 等，2024）。通用参考模型则从开源数据（如 Tulu-v2（Ivison 等，2023）与 OpenHermes-2.5（Teknium，2023））汇编 1.9B token。参考模型训练 3 个 epoch。1B 模型最大学习率 5e-5，7B 为 1e-5，采用余弦衰减。1B 最大序列长度 2048，7B 为 4096，并将多样本打包到这些长度作为输入。所有主实验中，持续预训练模型与参考模型均用 同一 基座模型初始化。

|\bm{\theta}|

Uniq.
Toks∗

Train
Toks

MMLU
STEM

|\bm{\theta}|

模型	$\|\bm{\theta}\|$	数据	唯一点 Token∗	训练 Token	GSM8K	MATH†	SVAMP	ASDiv	MAWPS	TAB	MQA	MMLU STEM	SAT‡	AVG
1–2B 基座模型
Tinyllama	1.1B	-	-	-	2.9	3.2	11.0	18.1	20.4	12.5	14.6	16.1	21.9	13.4
Phi-1.5	1.3B	-	-	-	32.4	4.2	43.4	53.1	66.2	24.4	14.3	21.8	18.8	31.0
Qwen1.5	1.8B	-	-	-	36.1	6.8	48.5	63.6	79.0	29.2	25.1	31.3	40.6	40.0
Gemma	2.0B	-	-	-	18.8	11.4	38.0	56.6	72.5	36.9	26.8	34.4	50.0	38.4
DeepSeekLLM	1.3B	OWM	14B	150B	11.5	8.9	-	-	-	-	-	29.6	31.3	-
DeepSeekMath	1.3B	-	120B	150B	23.8	13.6	-	-	-	-	-	33.1	56.3	-
在 Tinyllama-1B 上持续预训练
Tinyllama-CT	1.1B	OWM	14B	15B	6.4	2.4	21.7	36.7	47.7	17.9	13.9	23.0	25.0	21.6
Rho-1-Math	1.1B	OWM	14B	9B	29.8	14.0	49.2	61.4	79.8	25.8	30.4	24.7	28.1	38.1
$\Delta$				-40%	+23.4	+11.6	+27.5	+24.7	+32.1	+7.9	+16.5	+1.7	+3.1	+16.5
Rho-1-Math	1.1B	OWM	14B	30B	36.2	15.6	52.1	67.0	83.9	29.0	32.5	23.3	28.1	40.9
≥7B 基座模型
LLaMA-2	7B		-	-	14.0	3.6	39.5	51.7	63.5	30.9	12.4	32.7	34.4	31.4
Mistral	7B		-	-	41.2	11.6	64.7	68.5	87.5	52.9	33.0	49.5	59.4	52.0
Minerva	8B	-	39B	164B	16.2	14.1	-	-	-	-	-	35.6	-	-
Minerva	62B	-	39B	109B	52.4	27.6	-	-	-	-	-	53.9	-	-
Minerva	540B	-	39B	26B	58.8	33.6	-	-	-	-	-	63.9	-	-
LLemma	7B	PPile	55B	200B	38.8	17.2	56.1	69.1	82.4	48.7	41.0	45.4	59.4	50.9
LLemma	34B	PPile	55B	50B	54.2	23.0	67.9	75.7	90.1	57.0	49.8	54.7	68.8	60.1
Intern-Math	7B	-	31B	125B	41.8	14.4	61.6	66.8	83.7	50.0	57.3	24.8	37.5	48.7
Intern-Math	20B	-	31B	125B	65.4	30.0	75.7	79.3	94.0	50.9	38.5	53.1	71.9	62.1
DeepSeekMath	7B	-	120B	500B	64.1	34.2	74.0	83.9	92.4	63.4	62.4	56.4	84.4	68.4
在 Mistral-7B 上持续预训练
Mistral-CT	7B	OWM	14B	15B	42.9	22.2	68.6	71.0	86.1	45.1	47.7	52.6	65.6	55.8
Rho-1-Math	7B	OWM	14B	10.5B	66.9	31.0	77.8	79.0	93.9	49.9	58.7	54.6	84.4	66.2
$\Delta$				-30%	+24.0	+8.8	+9.2	+8.0	+7.8	+4.8	+11.0	+2.0	+18.8	+10.4

Pretraining Corpus For mathematical reasoning, we utilize the OpenWebMath (OWM) dataset (Paster et al., 2023), which comprises approximately 14B tokens sourced from math-related web pages in the Common Crawl. In the general domain, we combine the SlimPajama (Daria et al., 2023) and StarCoderData (Li et al., 2023a) (both part of the Tinyllama corpus) with OpenWebMath, training on a total of 80 billion tokens with a mix ratio of 6:3:1.

预训练语料 数学推理方面，我们使用 OpenWebMath（OWM）数据集（Paster 等，2023），约含 14B 来自 Common Crawl 数学相关网页的 token。通用领域则把 SlimPajama（Daria 等，2023）与 StarCoderData（Li 等，2023a）（二者均为 Tinyllama 语料的一部分）与 OpenWebMath 按 6:3:1 混合，共训练 800 亿 token。

Pretraining Setting For math pretraining, we continue pretraining on the Tinyllama-1.1B model (Zhang et al., 2024) and the Mistral-7B model (Jiang et al., 2023) with learning rates of 8e-5 and 2e-5, respectively. For the 1.1B model, we conducted our training on 32 \times H100 80G GPUs. This configuration allowed us to train approximately 15 billion tokens in around 3.5 hours and 50 billion tokens in about 12 hours. In the case of the 7B model, training the same 15 billion tokens took approximately 18 hours under similar hardware conditions. For general domain, we set the learning rate for Tinyllama-1.1B model to 1e-4 and train 80B tokens under the same hardware conditions, which takes approximately 19 hours. The batch size is uniformly set to 1M tokens for both domains. Regarding the token selection ratio, we use 60% for the Tinyllama-1.1B model and 70% for the Mistral-7B model.

预训练设定 数学预训练在 Tinyllama-1.1B（Zhang 等，2024）与 Mistral-7B（Jiang 等，2023）上持续预训练，学习率分别为 8e-5 与 2e-5。1.1B 模型在 32 \times H100 80G GPU 上训练：约 15B token 需约 3.5 小时，50B token 约 12 小时。7B 模型在相近硬件上训同样 15B token 约需 18 小时。通用领域将 Tinyllama-1.1B 学习率设为 1e-4，同硬件训 80B token 约 19 小时。两领域 batch size 统一为 1M token。token 选择比例：Tinyllama-1.1B 用 60%，Mistral-7B 用 70%。

Baseline Setting We use models that have been continually pretrained (Tinyllama-CT and Mistral-CT) through regular causal language modeling as baselines. Moreover, we compare Rho-1 with well-known and top-performing baselines, including Gemma (Team et al., 2024), Qwen1.5 (Bai et al., 2023), Phi-1.5 (Li et al., 2023b), DeepSeekLLM (DeepSeek-AI, 2024), DeepSeekMath (Shao et al., 2024), CodeLlama (Roziere et al., 2023), Mistral (Jiang et al., 2023), Minerva (Lewkowycz et al., 2022), Tinyllama (Zhang et al., 2024), LLemma (Azerbayev et al., 2023), and InternLM2-Math (Ying et al., 2024). For fine-tuning results, we also compare with previous best models MAmmoTH(Yue et al., 2024) and ToRA(Gou et al., 2024).

基线设定 我们以经常规因果语言建模持续预训练的模型（Tinyllama-CT 与 Mistral-CT）为基线，并与知名或表现突出的基线比较，包括 Gemma（Team 等，2024）、Qwen1.5（Bai 等，2023）、Phi-1.5（Li 等，2023b）、DeepSeekLLM（DeepSeek-AI，2024）、DeepSeekMath（Shao 等，2024）、CodeLlama（Roziere 等，2023）、Mistral（Jiang 等，2023）、Minerva（Lewkowycz 等，2022）、Tinyllama（Zhang 等，2024）、LLemma（Azerbayev 等，2023）与 InternLM2-Math（Ying 等，2024）。微调结果还与此前最优的 MAmmoTH（Yue 等，2024）和 ToRA（Gou 等，2024）比较。

Evaluation Setup To comprehensively evaluate pretrained models, we compare their few-shot capabilities and fine-tuning performance across a variety of tasks. We adopt the lm-eval-harness[^2] (Gao et al., 2023) for general tasks, and develop math evaluation suite[^3] for math tasks. We use vllm (v0.3.2) (Kwon et al., 2023) to speed up inference. Further details on our evaluation can be found in Appendix E.

评测设定 为全面评估预训练模型，我们在多种任务上比较少样本能力与微调表现。通用任务采用 lm-eval-harness[^2]（Gao 等，2023），数学任务使用自建数学评测套件[^3]。推理加速使用 vllm（v0.3.2）（Kwon 等，2023）。更多评测细节见附录 E。

\Delta

Model	Size	Tools	SFT Data	GSM8k	MATH	SVAMP	ASDiv	MAWPS	TAB	GSM-H	AVG
Used for SFT?	✓	✓	✗	✗	✗	✗	✗
Previous Models
GPT4-0314	-	✗	-	92.0	42.5	93.1	91.3	97.6	67.1	64.7	78.3
GPT4-0314 (PAL)	-	✓	-	94.2	51.8	94.8	92.6	97.7	95.9	77.6	86.4
MAmmoTH	70B	✓	MI-260k	76.9	41.8	82.4	-	-	-	-	-
ToRA	7B	✓	ToRA-69k	68.8	40.1	68.2	73.9	88.8	42.4	54.6	62.4
ToRA	70B	✓	ToRA-69k	84.3	49.7	82.7	86.8	93.8	74.0	67.2	76.9
DeepSeekMath	7B	✓	ToRA-69k	79.8	52.0	80.1	87.1	93.8	85.8	63.1	77.4
Our Pretrained Models
TinyLlama-CT	1B	✓	ToRA-69k	51.4	38.4	53.4	66.7	81.7	20.5	42.8	50.7
Rho-1-Math	1B	✓	ToRA-69k	59.4	40.6	60.7	74.2	88.6	26.7	48.1	56.9
$\Delta$				+8.0	+2.2	+7.3	+7.5	+6.9	+6.2	+5.3	+6.2
Mistral-CT	7B	✓	ToRA-69k	77.5	48.4	76.9	83.8	93.4	67.5	60.4	72.6
Rho-1-Math	7B	✓	ToRA-69k	81.3	51.8	80.8	85.5	94.5	70.1	63.1	75.3
$\Delta$				+3.8	+3.4	+3.9	+1.7	+1.1	+2.6	+2.7	+2.7

\Delta

模型	规模	工具	SFT 数据	GSM8k	MATH	SVAMP	ASDiv	MAWPS	TAB	GSM-H	AVG
是否用于 SFT？	✓	✓	✗	✗	✗	✗	✗
以往模型
GPT4-0314	-	✗	-	92.0	42.5	93.1	91.3	97.6	67.1	64.7	78.3
GPT4-0314 (PAL)	-	✓	-	94.2	51.8	94.8	92.6	97.7	95.9	77.6	86.4
MAmmoTH	70B	✓	MI-260k	76.9	41.8	82.4	-	-	-	-	-
ToRA	7B	✓	ToRA-69k	68.8	40.1	68.2	73.9	88.8	42.4	54.6	62.4
ToRA	70B	✓	ToRA-69k	84.3	49.7	82.7	86.8	93.8	74.0	67.2	76.9
DeepSeekMath	7B	✓	ToRA-69k	79.8	52.0	80.1	87.1	93.8	85.8	63.1	77.4
我们的预训练模型
TinyLlama-CT	1B	✓	ToRA-69k	51.4	38.4	53.4	66.7	81.7	20.5	42.8	50.7
Rho-1-Math	1B	✓	ToRA-69k	59.4	40.6	60.7	74.2	88.6	26.7	48.1	56.9
$\Delta$				+8.0	+2.2	+7.3	+7.5	+6.9	+6.2	+5.3	+6.2
Mistral-CT	7B	✓	ToRA-69k	77.5	48.4	76.9	83.8	93.4	67.5	60.4	72.6
Rho-1-Math	7B	✓	ToRA-69k	81.3	51.8	80.8	85.5	94.5	70.1	63.1	75.3
$\Delta$				+3.8	+3.4	+3.9	+1.7	+1.1	+2.6	+2.7	+2.7

Few-shot CoT Reasoning Results We evalute base models prompting with few-shot chain-of-thought (CoT) (Wei et al., 2022a) examples following previous works (Lewkowycz et al., 2022; Azerbayev et al., 2023; Shao et al., 2024). As results shown in Table 1, in comparison to continue pretraining directly, Rho-1-Math has achieved the average few-shot accuracy improvement of 16.5% on 1B models and 10.4% on 7B models. Furthermore, after training for multiple epochs on OpenWebMath, we find that Rho-1 could further increase the average few-shot accuracy to 40.9%. Compared to DeepSeekMath-7B, which pretrained on 500 billion math-related tokens, Rho-1-7B pretrained on only 15 billion tokens (selecting 10.5 billion tokens) achieved comparable results, demonstrating the efficiency of our approach.

少样本 CoT 推理结果 我们按以往工作（Lewkowycz 等，2022；Azerbayev 等，2023；Shao 等，2024），用少样本思维链（CoT）（Wei 等，2022a）示例提示评测基座模型。如表 1 所示，相对直接持续预训练，Rho-1-Math 在 1B 模型上平均少样本准确率提升 16.5%，在 7B 上提升 10.4%。在 OpenWebMath 上多 epoch 训练后，Rho-1 还可把平均少样本准确率进一步推到 40.9%。相较在 5000 亿数学相关 token 上预训练的 DeepSeekMath-7B，Rho-1-7B 仅在 150 亿 token（选出 105 亿）上预训练即达到可比结果，体现了方法效率。

Tool-Integrated Reasoning Results We fine-tune Rho-1 and baseline models on 69k ToRA corpus (Gou et al., 2024), consisting of 16k GPT-4-generated trajectories in a tool-integrated reasoning format, and 53k answer-augmented samples using LLaMA. As presented in Table 2, Rho-1-1B and Rho-1-7B achieved a state-of-the-art 40.6% and 51.8% on MATH dataset, respectively. On some unseen tasks (e.g., TabMWP and GSM-Hard), Rho-1 also demonstrates a certain degree of generalizability, with an average few-shot accuracy improvement of 6.2% on the Rho-1-Math-1B and 2.7% on Rho-1-Math-7B.

工具集成推理结果 我们在 69k ToRA 语料（Gou 等，2024）上微调 Rho-1 与基线：其中含 16k 条 GPT-4 生成的工具集成推理轨迹，以及 53k 条用 LLaMA 做答案增强的样本。如表 2，Rho-1-1B 与 Rho-1-7B 在 MATH 上分别达到当时最优的 40.6% 与 51.8%。在部分未见任务（如 TabMWP 与 GSM-Hard）上，Rho-1 也表现出一定泛化：Rho-1-Math-1B 平均少样本准确率提升 6.2%，Rho-1-Math-7B 提升 2.7%。

Figure 5: General pretraining results. We continual pretraining Tinyllama-1B on 80G general tokens. Tinyllama-CT is etrained with CLM, while Rho-1 is trained with our proposed SLM.

图 5： 通用预训练结果 。我们在 80G 通用 token 上持续预训练 Tinyllama-1B。Tinyllama-CT 用 CLM 训练，Rho-1 用本文提出的 SLM。

3.3 General Pre-training Results We confirm the efficacy of the SLM in general pretraining by continual training Tinyllama-1.1B on 80 billion tokens. The results depicted in Figure 5 indicate that although Tinyllama has already undergone extensive training on the majority of these tokens, the application of SLM yields an average enhancement of 6.8% across 15 benchmarks compared to direct continual pretraining. The improvements were especially pronounced in code and math tasks, exceeding 10%.

3.3 通用预训练结果 我们通过在 800 亿 token 上持续训练 Tinyllama-1.1B，确认 SLM 在通用预训练中的效力。图 5 显示：尽管 Tinyllama 已在这些 token 的大部分上充分训练过，相对直接持续预训练，应用 SLM 仍在 15 项基准上平均提升 6.8%。代码与数学任务上的提升尤为明显，超过 10%。

\mathcal{L}_{\text{RM}}

Score
Function

Uniq.
Toks

Train
Toks

\mathcal{L}_{\text{RM}}

模型	打分函数	数据	唯一点 Token	训练 Token	GSM8K	MATH	SVAMP	ASDiv	MAWPS	MQA	AVG
Tinyllama-CT (RM)	-	OWM	14B	15B	6.3	2.6	21.7	36.7	47.7	13.9	21.5
Tinyllama-SLM	$\mathcal{L}_{\text{RM}}$	OWM	14B	10.5B	6.7	4.6	23.3	40.0	54.5	14.3	23.9
Tinyllama-SLM	$\mathcal{H}_{\text{RM}}$	OWM	14B	10.5B	7.0	4.8	23.0	39.3	50.5	13.5	23.0
Tinyllama-SLM	$\mathcal{L}_{\text{RM}}\cap\mathcal{H}_{\text{RM}}$	OWM	14B	9B	7.1	5.0	23.5	41.2	53.8	18.0	24.8
Tinyllama-CT	-	PPile	55B	52B	8.0	6.6	23.8	41.0	54.7	14.2	24.7
Tinyllama-SLM	$\mathcal{L}_{\text{RM}}\cap\mathcal{H}_{\text{RM}}$	PPile	55B	36B	8.6	8.4	24.4	43.6	57.9	16.1	26.5

\mathcal{L}_{\text{RM}}

\mathcal{L}_{\text{RM}}

The experimental results, as shown in Table 3, indicate that using only the OWM-trained reference model can effectively guide the model in pre-training on the same corpus, improving average downstream performance by +2.4%. Using only the information entropy as the score function brought about a similar improvement. Additionally, we considered training on the intersection of tokens selected by the two scoring functions and found better performance, with a 40% reduction in tokens and +3.3% performance. Furthermore, training the SLM on the PPile, despite only using the OWM subset to train the reference model, still achieved a 1.8% improvement with 30% fewer tokens used. For more details, please refer to Appendix H.

实验结果如表 3：仅用 OWM 训练的参考模型即可有效引导在同一语料上的预训练，下游平均表现提升 +2.4%。仅用信息熵作打分函数也能带来相近提升。此外，取两种打分函数所选 token 的交集再训练，表现更好：token 减少 40%，性能 +3.3%。即便参考模型只在 OWM 子集上训练，在 PPile 上做 SLM 仍以少用 30% token 取得 1.8% 提升。更多细节见附录 H。

Figure 6: The dynamics of pretraining loss and downstream loss. (a) and (c) represent the loss of tokens selected/unselected by SLM during pretraining in both SLM and CLM methods, while (b) represents the loss of the SLM and CLM methods on MetaMath (Yu et al., 2024). We tested the above results through the process of pretraining with a total of 4 billion tokens.

图 6： 预训练损失与下游损失的动态。（a）与（c）分别表示 SLM 与 CLM 方法中，被 SLM 选中/未选中 token 在预训练中的损失；（b）表示 SLM 与 CLM 在 MetaMath（Yu 等，2024）上的损失。上述结果在总共 40 亿 token 的预训练过程中测得 。

Selected Token Loss Aligns Better with Downstream Performance We utilized the reference model to filter tokens and assess their impact on validation and downstream losses after training. As depicted in Figure 6, we pretrained on 4B tokens and tracked loss variations across methods and validation sets. The Rho-1 showed greater loss reduction on selected tokens than regular pretraining. Cross-referencing figures (a), (b), and (c) reveals that selected-token pretraining substantially lowers downstream loss, while traditional pretraining’s effect on downstream loss is less pronounced despite initial loss reductions. Therefore, we expect that selecting tokens for pretraining is more efficient.

所选 token 损失与下游表现更对齐 我们用参考模型过滤 token，并在训练后评估它们对验证损失与下游损失的影响。如图 6，我们在 4B token 上预训练并跟踪各方法与验证集上的损失变化。相对常规预训练，Rho-1 在所选 token 上的损失下降更大。交叉对照图 (a)(b)(c) 可见：在所选 token 上预训练能显著降低下游损失；而传统预训练尽管初期损失下降，对下游损失的影响却不那么明显。因此我们预期：为预训练挑选 token 更高效。

Figure 7: The relationship between the selected tokens / unselected tokens loss in SLM and downstream task performance. The y-axis represents the average few-shot accuracy on GSM8k and MATH. The x-axis represents the average loss on selected tokens / unselected tokens at corresponding checkpoint (2B, 5B, 8B, 11B, and 14B).

图 7： SLM 中已选 / 未选 token 损失与下游任务表现的关系。纵轴为 GSM8k 与 MATH 上的平均少样本准确率；横轴为对应检查点（2B、5B、8B、11B、14B）上已选 / 未选 token 的平均损失 。

In Figure 7, we demonstrate that the loss of selected tokens correlates with downstream task performance, following a power law similar to recent findings (Gadre et al., 2024). Our analysis shows that tokens selected by SLM positively impact performance, while those not selected have a negative impact. Thus, reducing loss across all tokens is not imperative for improved model performance. Refer to Appendix F for further details.

图 7 表明：所选 token 的损失与下游任务表现相关，并遵循类似近期发现的幂律关系（Gadre 等，2024）。分析显示，SLM 选出的 token 对性能有正向影响，未选中的则有负向影响。因此，降低所有 token 的损失并非提升模型表现的必要前提。更多细节见附录 F。

Figure 8: The PPL of tokens selected by different checkpoint. We test the PPL of the tokens selected at 2B, 5B, 8B, 11B, and 14B.

图 8： 不同检查点所选 token 的 PPL。我们测试在 2B、5B、8B、11B、14B 处所选 token 的 PPL 。

What Tokens are Selected with SLM? We aim to analyze the tokens selected by the SLM method in pretraining to further explore its working mechanism. To this end, we visualize the token selection process during the training of Rho-1 using the OpenWebMath. In §G.1, we have highlighted in blue the tokens that were retained during actual pretraining. We observe that the majority of tokens chosen by the SLM method are closely related to mathematics, effectively training the model on the parts of the original corpus that are pertinent to mathematical content.

SLM 选出了哪些 token？ 我们希望分析预训练中 SLM 所选 token，以进一步探究其机制。为此，我们可视化 Rho-1 在 OpenWebMath 上训练时的 token 选择过程。第 G.1 节用蓝色标出实际预训练中保留的 token。我们观察到：SLM 所选多数 token 与数学密切相关，有效地把模型训练集中在原语料中与数学内容相关的部分。

Furthermore, we investigated the differences in token filtering across various checkpoints during the training process and tested the perplexity of these tokens on different checkpoints. As illustrated in Figure 8, we found that the tokens selected by later checkpoints tend to have higher perplexity towards the later stages of training and lower perplexity in the earlier stages. This may suggest that the model first optimizes tokens with a larger learnable space, thereby increasing learning efficiency. Moreover, we noticed a sample-wise “double descent” (Nakkiran et al., 2021) on the loss of selected tokens, where the select token’s perplexity initially increases before decreases. This might be an effect of selecting tokens based on excess loss, targeting those most in need at each checkpoint.

我们还考察了训练过程中不同检查点在 token 过滤上的差异，并在各检查点测试这些 token 的困惑度。如图 8：较晚检查点选出的 token，在训练后期往往有更高困惑度、在早期则更低。这可能说明模型先优化可学空间更大的 token，从而提高学习效率。此外，我们在所选 token 损失上观察到样本级「双重下降」（double descent）（Nakkiran 等，2021）——所选 token 的困惑度先升后降。这或许是按超额损失选 token、在每个检查点瞄准「最急需」者所带来的效应。

Effect of Token Select Ratio We investigate the impact of token selecting ratios of the SLM. Generally, the selecting ratio is defined by heuristic rules, similar to the approach previously employed in the training of Masked Language Models (MLMs) (Devlin et al., 2019; Liu et al., 2019). As shown in Figure 9, the selected tokens is suitable for accounting for about 60% of the original tokens.

token 选择比例的影响 我们研究 SLM 的 token 选择比例。一般地，选择比例由启发式规则定义，类似此前掩码语言模型（MLM）训练中的做法（Devlin 等，2019；Liu 等，2019）。如图 9，所选 token 约占原 token 的 60% 较为合适。

4 Conclusion

4 结论

In this paper, we propose using Selective Language Modeling(SLM) to train Rho-1, which select more suitable tokens for current pretraining stage. We conducted the detailed analysis of the loss of tokens during the pretraining process and found that not all tokens are equal during pretraining. Our experiments and analysis in the fields of mathematics and general have demonstrated the effectiveness of the SLM method, emphasizing the importance of token level in the LLM pretraining process. In the future, how to improve pretraining of LLMs from the perspective of token level worthy of in-depth research.

本文提出用选择性语言建模（SLM）训练 Rho-1，为当前预训练阶段挑选更合适的 token。我们对预训练过程中的 token 损失做了细致分析，发现预训练中并非所有 token 平等。在数学与通用领域的实验与分析表明 SLM 有效，也凸显了 LLM 预训练过程中 token 级视角的重要性。未来，如何从 token 级改进 LLM 预训练值得深入研究。

Acknowledgments Zhenghao Lin and Chen Lin were supported by National Key R&D Program of China (No. 2022ZD0160501), the Natural Science Foundation of China (No.62372390,62432011). Zhibin Gou and Yujiu Yang were supported by the Shenzhen Science and Technology Program (JCYJ20220818101001004) and the “Graph Neural Network Project” of Ping An Technology (Shenzhen) Co., Ltd.

致谢 Zhenghao Lin 与 Chen Lin 受国家重点研发计划（No. 2022ZD0160501）与国家自然科学基金（No.62372390、62432011）支持。Zhibin Gou 与 Yujiu Yang 受深圳市科技计划（JCYJ20220818101001004）以及平安科技（深圳）有限公司「图神经网络项目」支持。

References

参考文献

Guo et al. [2024] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196 , 2024. Li et al. [2023f] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In ACL (1) , pages 12286-12312. Association for Computational Linguistics, 2023f. Wan et al. [2024] Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=jiDsk12qcz . Fu et al. [2023] Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning , pages 10421-10430. PMLR, 2023.

Appendix

附录

Appendix A Author Contributions

附录 A 作者贡献

Zhenghao Lin designed and implemented detailed token selection process, conducted extensive preliminary experiments, developed the pre-training and evaluation pipeline, conducted most of the pre-training experiments and analysis, implemented baselines, and significantly contributed to the writing. Zhibin Gou presented a preliminary proposal, introduced the method of using excess loss for reweighting tokens, compiled high-quality corpora, trained reference models, set up the fine-tuning and evaluation pipelines, designed the experimental analysis, and significantly contributed to the writing. Yeyun Gong proposed the initial project and co-led the project with Weizhu Chen, they offered extensive advice and guidance on experiments and writing, and oversaw team collaboration and resource management. Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, and Nan Duan offered research mentorship, coordinated the project, and contributed to the writing.

Zhenghao Lin 设计并实现了详细的 token 选择流程，开展大量前期实验，搭建预训练与评测管线，完成大部分预训练实验与分析，实现基线，并在写作上贡献显著。Zhibin Gou 提出初步方案，引入用超额损失对 token 重加权的方法，汇编高质量语料，训练参考模型，搭建微调与评测管线，设计实验分析，并在写作上贡献显著。Yeyun Gong 提出初始项目，与 Weizhu Chen 共同领导项目；二人在实验与写作上提供大量建议与指导，并统筹团队协作与资源管理。Xiao Liu、Yelong Shen、Ruochen Xu、Chen Lin、Yujiu Yang、Jian Jiao 与 Nan Duan 提供研究指导，协调项目，并参与写作。

Appendix B Related Works

附录 B 相关工作

B.1 Pretraining Data Optimization The objective of optimizing pre-training corpora is to maximize the performance and efficiency of language model training by improving the quality and scale of the pretrain data mixture. This includes data collecting through crawling (Raffel et al., 2020) or synthesis (Polu and Sutskever, 2020; Gunasekar et al., 2023), de-duplication (Lee et al., 2021; Kandpal et al., 2022; Tirumala et al., 2023), filtering and selection (Xie et al., 2024a; Albalak et al., 2024), as well as data composition (Xie et al., 2024b) and curriculum (Chen et al., 2024; Ma et al., 2024).

B.1 预训练数据优化 优化预训练语料的目标，是通过提升预训练数据混合的质量与规模，最大化语言模型训练的性能与效率。这包括经由爬取（Raffel 等，2020）或合成（Polu 与 Sutskever，2020；Gunasekar 等，2023）收集数据、去重（Lee 等，2021；Kandpal 等，2022；Tirumala 等，2023）、过滤与选择（Xie 等，2024a；Albalak 等，2024），以及数据组合（Xie 等，2024b）与课程（Chen 等，2024；Ma 等，2024）。

B.2 Data Selection Data selection for fine-tuning has been extensively studied, focusing on improving quality (Li et al., 2023c), diversity (Liu et al., 2024), and distribution matching (Li et al., 2023d; Xia et al., 2024; Ni et al., 2024; Kang et al., 2024). For pretraining, various lightweight filters are utilized (Albalak et al., 2024), including heuristic-based (e.g., language and item count filtering), classifier-based (Brown et al., 2020), and perplexity or loss-based approaches (Wenzek et al., 2019; Qin et al., 2024). The massive public RedPajama-Data-v2 dataset (Computer, 2023), for example, leverages over 40 quality indicators for data filtering and reweighting. Nevertheless, strict filtering like blocklist (Raffel et al., 2020) and Safety API filtering (Welbl et al., 2021), have been found to hurt evaluation loss or induce bias (Dodge et al., 2021).

B.2 数据选择 微调阶段的数据选择已有大量研究，聚焦提升质量（Li 等，2023c）、多样性（Liu 等，2024）与分布匹配（Li 等，2023d；Xia 等，2024；Ni 等，2024；Kang 等，2024）。预训练则多用各类轻量过滤器（Albalak 等，2024），包括启发式（如语言与条目计数过滤）、基于分类器（Brown 等，2020），以及基于困惑度或损失的方法（Wenzek 等，2019；Qin 等，2024）。例如大规模公开 RedPajama-Data-v2 数据集（Computer，2023）利用逾 40 项质量指标做过滤与重加权。然而，黑名单（Raffel 等，2020）与 Safety API 过滤（Welbl 等，2021）等严格过滤，已被发现会损害评测损失或引入偏差（Dodge 等，2021）。

Sample-level selection has been extensively studied in previous research (Sener and Savarese, 2017; Killamsetty et al., 2021), particularly through online batch selection (Loshchilov and Hutter, 2015; Schaul et al., 2015; Chang et al., 2017; Katharopoulos and Fleuret, 2018; Jiang et al., 2019). These approaches have been applied to various classification tasks (Song et al., 2020; Mindermann et al., 2022) and language modeling (Fan and Jaggi, 2023). However, Kaddour et al. (2023) find that batch selection is not computationally efficient.

样本级选择在以往研究中被广泛探讨（Sener 与 Savarese，2017；Killamsetty 等，2021），尤其是在线 batch 选择（Loshchilov 与 Hutter，2015；Schaul 等，2015；Chang 等，2017；Katharopoulos 与 Fleuret，2018；Jiang 等，2019）。这些方法已用于多种分类任务（Song 等，2020；Mindermann 等，2022）与语言建模（Fan 与 Jaggi，2023）。但 Kaddour 等（2023）发现 batch 选择在计算上并不高效。

Many previous works have employed the general idea of using a reference model as a proxy for data selection. For instance, Selection Via Proxy trains a proxy model to select samples with high uncertainty (Coleman et al., 2019). Xie et al. (2024a) and Engstrom et al. (2024) utilize n-gram models or datamodels with a target dataset to estimate importance weights. Additionally, Xie et al. (2024b) optimize the worst-case excess loss (Oren et al., 2019) relative to a reference model to determine domain weights. One of SLM’s scoring functions is excess loss, and the most relevant work related to excess loss is RHO-LOSS (Mindermann et al., 2022), which trains a small model on a holdout set and uses the difference between training loss and holdout loss to select in-batch samples. Although excess loss is mathematically identical to RHO-LOSS, SLM differs in three important ways: 1) The focus is distinct. Motivated by the training dynamics of token loss, the core idea of SLM is to select useful tokens for pre-training. Its score functions are highly flexible and not limited to excess loss (see Appendix H for other functions). In contrast, RHO-LOSS aims to mathematically derive a reducible holdout loss to minimize generalization loss. 2) The meaning and training procedure of the proxy model are different. SLM trains a reference model on high-quality data to reflect the desired data distribution, whereas RHO-LOSS trains a small model on a random holdout set. 3) The selection scale and granularity vary. RHO-LOSS selects sample-level data on a small scale (typically 1K-1M samples) for task-specific fine-tuning tasks such as MNIST (LeCun et al., 1998) and SST-2 (Socher et al., 2013). In contrast, SLM conducts fine-grained token-level selection on large-scale language model pre-training, involving up to 80B tokens.

许多以往工作采用「用参考模型作数据选择代理」这一通用思路。例如 Selection Via Proxy 训练代理模型选择高不确定性样本（Coleman 等，2019）。Xie 等（2024a）与 Engstrom 等（2024）利用 n-gram 模型或 datamodel，结合目标数据集估计重要性权重。此外，Xie 等（2024b）相对参考模型优化最坏情形超额损失（Oren 等，2019）以确定领域权重。SLM 的打分函数之一是超额损失；与超额损失最相关的工作是 RHO-LOSS（Mindermann 等，2022），它在 holdout 集上训练小模型，并用训练损失与 holdout 损失之差选择 batch 内样本。尽管超额损失在数学上与 RHO-LOSS 相同，SLM 有三点重要不同：1）侧重点不同。受 token 损失训练动态启发，SLM 的核心是为预训练挑选有用 token；其打分函数高度灵活，不限于超额损失（其他函数见附录 H）。相较之下，RHO-LOSS 旨在数学上导出可约减的 holdout 损失以最小化泛化损失。2）代理模型的含义与训练过程不同。SLM 在高质量数据上训练参考模型以反映期望数据分布，而 RHO-LOSS 在随机 holdout 集上训练小模型。3）选择尺度与粒度不同。RHO-LOSS 在小规模（通常 1K-1M 样本）上做样本级选择，面向 MNIST（LeCun 等，1998）、SST-2（Socher 等，2013）等任务特定微调；SLM 则在大规模语言模型预训练上做细粒度 token 级选择，涉及多达 80B token。

Token-level training strategies have also been explored, especially for the pre-training of BERT-like models using Masked Language Modeling (MLM) (Devlin et al., 2018). Specifically, “selective masking” involves masking important tokens in the input to focus on learning tokens that are more relevant to downstream tasks (Gu et al., 2020; Lad et al., 2022), whereas “token dropping” aims to reduce training costs by omitting less important tokens (Zhong et al., 2023a; Hou et al., 2022). (Li et al., 2023e) assesses the quality of each token based on the skewness of its predicted distribution and truncates the noisy tokens during training. Additionally, some research has approached the analysis and detection of under-trained tokens from a tokenization perspective (Rumbelow and Watkins, 2023; Land and Bartolo, 2024). To our knowledge, we are the first to explore token-level data selection for large language model training, aimed at enhancing data quality and information density at the most fundamental granularity.

token 级训练策略也有探索，尤其是用掩码语言建模（MLM）预训练类 BERT 模型时（Devlin 等，2018）。具体地，「选择性掩码」对输入中重要 token 掩码，以聚焦学习与下游任务更相关的 token（Gu 等，2020；Lad 等，2022）；而「token 丢弃」则通过省略次要 token 降低训练成本（Zhong 等，2023a；Hou 等，2022）。Li 等（2023e）根据预测分布的偏度评估每个 token 的质量，并在训练中截断噪声 token。另有研究从分词视角分析并检测训练不足的 token（Rumbelow 与 Watkins，2023；Land 与 Bartolo，2024）。据我们所知，我们是首次为大语言模型训练探索 token 级数据选择，旨在从最基础粒度提升数据质量与信息密度。

B.3 Language Model Training Dynamics Investigating the training dynamics of language models is essential for understanding their behavior throughout the training process. This research includes studying internal representations (Saphra and Lopez, 2018), the acquisition of linguistic knowledge (Choshen et al., 2021; Liu et al., 2021), and the phenomenon of grokking (Power et al., 2022). The analysis by Xia et al. (2022) is the most related to ours, which examines token-level training trajectories in models of varying sizes. Our findings, however, diverge from those of Xia et al. (2022), who posit that tokens with little change in perplexity are “already learned”. We identify a spectrum of token patterns, including “easy tokens” and “hard tokens” that resist convergence. Recognizing this, we propose a method of selective language modeling that targets the influential tokens, optimizing the learning process.

B.3 语言模型训练动态 研究语言模型的训练动态，对理解其在整个训练过程中的行为至关重要。该方向包括内部表征（Saphra 与 Lopez，2018）、语言知识获取（Choshen 等，2021；Liu 等，2021）以及 grokking 现象（Power 等，2022）。与我们最相关的是 Xia 等（2022）的分析，他们考察不同规模模型中的 token 级训练轨迹。但我们的发现与 Xia 等（2022）不同：他们主张困惑度几乎不变的 token「早已学会」；我们则识别出一系列 token 模式，包括「容易 token」以及难以收敛的「困难 token」。据此，我们提出选择性语言建模，瞄准有影响力的 token，以优化学习过程。

B.4 Scaling Laws Scaling laws guide us in discovering the impact of factors such as parameter count, data size, and compute on language model performance and behavior. These studies usually focus on predicable scaling though power law (Kaplan et al., 2020; Hernandez et al., 2021), optimal resource allocation (Hoffmann et al., 2022), downstream tasks (Wei et al., 2022b; Isik et al., 2024; Gadre et al., 2024), architectures (Tay et al., 2022), memorization (Tirumala et al., 2022; Carlini et al., 2022; Henighan et al., 2023; Biderman et al., 2024), and repeating data (Hernandez et al., 2022; Muennighoff et al., 2024; Xue et al., 2024). Most scaling laws on model performance study cross-entory loss on all training tokens, while we focus on the tokens loss of desired distributions.

B.4 缩放定律 缩放定律帮助我们发现参数量、数据规模与算力等因素对语言模型性能与行为的影响。这些研究通常关注可预测的幂律缩放（Kaplan 等，2020；Hernandez 等，2021）、最优资源分配（Hoffmann 等，2022）、下游任务（Wei 等，2022b；Isik 等，2024；Gadre 等，2024）、架构（Tay 等，2022）、记忆（Tirumala 等，2022；Carlini 等，2022；Henighan 等，2023；Biderman 等，2024）以及数据重复（Hernandez 等，2022；Muennighoff 等，2024；Xue 等，2024）。多数关于模型性能的缩放定律研究所有训练 token 上的交叉熵损失，而我们聚焦期望分布上的 token 损失。

Appendix C Limitations and Future Work

附录 C 局限与未来工作

Generalizability In math continual pretraining, as depicted in Figure 6, training exclusively with SLM leads to quickly convergence to the domain focused by the reference model, accompanied by a significant rise in the loss of unselected tokens. Although no adverse effects, like biases, have been observed from the increased loss yet, a general pretraining loss on text and code may prevent overfitting (Goodhart and Goodhart, 1984), as suggested by Ouyang et al. (2022) and Azerbayev et al. (2023). Furthermore, future efforts could broaden the corpus scope of the reference model, and enlarge the pretraining data size, as exemplified by DeepSpeedMath (Shao et al., 2024).

可泛化性 在数学持续预训练中，如图 6 所示，仅用 SLM 训练会迅速收敛到参考模型所聚焦的领域，同时未选 token 的损失显著上升。尽管尚未观察到因损失升高带来的偏差等负面影响，但文本与代码上的通用预训练损失或可防止过拟合（Goodhart 与 Goodhart，1984），如 Ouyang 等（2022）与 Azerbayev 等（2023）所建议。此外，未来工作可扩大参考模型的语料范围，并增大预训练数据规模，如 DeepSeekMath（Shao 等，2024）所示。

Scalability Due to budget constraints, we have only verified the effectiveness of our method on smaller models (<=7B parameters) and smaller datasets (<100B tokens). Smaller models benefit significantly from removing the loss of irrelevant tokens and focusing on important ones. However, it’s possible that very large models trained on extensive corpora may naturally develop this inductive bias to compress useful data (i.e., compressing everything), although it may sounds inefficient for now. Therefore, future works should study whether this selective language modeling technique can scale to very large models and data (Kaplan et al., 2020).

可扩展性 受预算限制，我们仅在较小模型（\leq7B 参数）与较小数据集（<100B token）上验证了方法有效性。较小模型从去掉无关 token 损失、聚焦重要 token 中获益显著。但也可能：在超大规模语料上训练的超大模型会自然形成压缩有用数据的归纳偏置（即「压缩一切」），尽管眼下听起来仍低效。因此，未来工作应研究这种选择性语言建模技术能否扩展到超大模型与数据（Kaplan 等，2020）。

Is training a reference model necessary? To score tokens, we need a high-quality reference model. This could be a base model trained with a small amount of high-quality data, or a performant open-source model. In fact, since we only need input logprobs or perplexity from reference model, we could even utilize more powerful proprietary model APIs. We can input tokens and use the log probabilities of the input returned by the API as reference scores. We leave this for future works.

是否必须训练参考模型？ 要为 token 打分，需要高质量参考模型。它可以是用少量高质量数据训练的基座模型，也可以是性能良好的开源模型。事实上，由于我们只需参考模型的输入 logprob 或困惑度，甚至可以利用更强的专有模型 API：输入 token，把 API 返回的输入对数概率当作参考分数。我们把这一点留给未来工作。

How to improve upon SLM? There are many natural extensions of SLM, e.g., reweighting tokens instead of selecting may improve robustness; using a reference model as a reward model to guide pretraining with reinforcement learning; adopting multiple reference models to reduce overfitting; designing token-level curriculum learning and iterative strategies for continuous improvements, etc .

如何在 SLM 之上改进？ SLM 有许多自然扩展，例如：对 token 重加权而非硬选择或可提升鲁棒性；把参考模型当奖励模型，用强化学习引导预训练；采用多个参考模型以降低过拟合；设计 token 级课程学习与迭代策略以持续改进，等等。

Expanding the use of SLM SLM may be extended to supervised fine-tuning to address the noise and distribution mismatches in many SFT datasets. Another potential application is alignment, e.g., by training a reference model to emphasize helpfulness, truthfulness, and harmlessness, we may obtain a base model that is natively aligned during the pretraining stage. Meanwhile, we believe that the idea of SLM may find broader applications in multimodal data such as images, videos, and speech, which have a high noise-to-information ratio than text.

拓展 SLM 的用途 SLM 可扩展到监督微调，以应对许多 SFT 数据集中的噪声与分布失配。另一潜在应用是对齐：例如，训练强调有用性、真实性与无害性的参考模型，或可在预训练阶段就得到原生对齐的基座模型。同时，我们认为 SLM 的思路也可更广泛地用于图像、视频与语音等多模态数据——它们相对文本有更高的噪声-信息比。

Appendix D Analysis and Visualization of Tokens in Pretraining

附录 D 预训练中 token 的分析与可视化

Figure 10: The loss of four categories of tokens during Mistral-7B pretraining on OpenWebMath. (a) shows the loss of H\toH, L\toH, H\toL, and L\toL tokens during pretraining. (b) and (c) show three cases of fluctuating tokens’ loss in L\toL and H\toH during pretraining, respectively.

图 10： Mistral-7B 在 OpenWebMath 上预训练时四类 token 的损失。（a）展示 H\toH、L\toH、H\toL 与 L\toL 四类 token 在预训练中的损失。（b）与（c）分别展示 L\toL 与 H\toH 中波动 token 损失的三个案例 。

D.1 More Details of Four Categories Tokens We categorize tokens into four categories: H\toH, L\toH, H\toL, L\toL. During the training process, we collected the loss of each token after training on each 1 billion tokens training data. We then used linear fitting and took the difference in loss between the first and last points as evidence of whether the loss decreased during the training process.

D.1 四类 token 的更多细节 我们将 token 分为四类：H\toH、L\toH、H\toL、L\toL。训练过程中，每训练 10 亿 token 后收集每个 token 的损失；再做线性拟合，并以首末点损失差作为训练中损失是否下降的依据。

(l_{0},l_{1},...,l_{n})

(l_{0},l_{1},...,l_{n})

f(a,b)=\text{minimize}\sum_{i=0}^{n}(l_{i}-(ax_{i}+b))^{2}, \tag{6}

x_{0}=0

x_{0}=0

\Delta\mathcal{L}

\Delta\mathcal{L}

D.2 Non-Converging Tokens in Pretrainig In §2.1, we mentioned that during the training process, only a minority of tokens belong to the H\toL category. Among the remaining categories of H\toH and L\toL tokens, there are tokens that exhibit significant fluctuations during training. Furthermore, there are instances where H\toL tokens are not effectively learned. Therefore, in our analysis, we specifically select those tokens from these categories that demonstrate considerable variability and distinct loss. We visualize these tokens that exhibit abnormal behavior during the training process. As illustrated in Figure 12, we find that the majority of these tokens originate from rather chaotic corpora. For instance, the corpora may include a mix of custom symbols, unintelligible gibberish, and information such as timetables and bibliographic references. Within a segment of normal text, there may also be fluctuations in the usage of common conjunctions, word suffixes, and punctuation marks. The latter may not necessarily be disastrous for training; in fact, it could represent a normal occurrence. However, if we can effectively mitigate the losses caused by the former, it might lead to more stable and efficient model training.

D.2 预训练中不收敛的 token 第 2.1 节提到：训练过程中仅少数 token 属于 H\toL。在其余的 H\toH 与 L\toL 中，有些 token 在训练中波动显著。此外，也有 H\toL token 并未被有效学会。因此，我们从这些类别中专门挑出变异大、损失形态鲜明的 token，可视化其训练中的异常行为。如图 12 所示，我们发现多数这类 token 来自相当杂乱的语料：例如混有自定义符号、难以理解的乱码，以及时刻表、文献引用等信息。即便在一段正常文本中，常见连词、词缀与标点的用法也可能出现波动。后者对训练未必灾难，甚至可能是正常现象；但若能有效缓解前者带来的损失，或可让模型训练更稳、更高效。

Appendix E Evalution Details

附录 E 评测细节

E.1 Math Evalution We conducted a comprehensive evaluation of the model across various math reasoning benchmarks, encompassing a range of difficulties from elementary to university level, multiple mathematical domains, and diverse question types including multiple-choice and open-ended questions. Our benchmarks include GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), GSM-Hard (Gao et al., 2022), SVAMP (Patel et al., 2021), ASDIV (Miao et al., 2020), MAWPS (Koncel-Kedziorski et al., 2016), TabMWP (TAB) (Lu et al., 2023), MathQA (MQA) (Amini et al., 2019), MMLU-STEM (Hendrycks et al., 2020), and SAT (Azerbayev et al., 2023).

E.1 数学评测 我们在多种数学推理基准上全面评测模型，覆盖从小学到大学的难度、多个数学领域，以及选择题与开放题等题型。基准包括 GSM8k（Cobbe 等，2021）、MATH（Hendrycks 等，2021）、GSM-Hard（Gao 等，2022）、SVAMP（Patel 等，2021）、ASDIV（Miao 等，2020）、MAWPS（Koncel-Kedziorski 等，2016）、TabMWP（TAB）（Lu 等，2023）、MathQA（MQA）（Amini 等，2019）、MMLU-STEM（Hendrycks 等，2020）与 SAT（Azerbayev 等，2023）。

E.2 General Evalution In the evaluation of general domain, we followed the lm-evaluation-harness (Gao et al., 2023) and evalute model on MMLU (Hendrycks et al., 2020), BBH (Suzgun et al., 2022), AGIEval (Zhong et al., 2023b), ARC-Easy and ARC-Challenge (Clark et al., 2018), BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), Hellaswag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), OpenBookQA (Mihaylov et al., 2018). On HumanEval (Zheng et al., 2023) and TydiQA (Clark et al., 2020), we follow the evaluation pipeline of open-instrcut (Ivison et al., 2023) and report Pass@1 and Pass@10 for HumanEval and F1 for TydiQA. For MBPP (Austin et al., 2021) benchmark, we follow the evaluation pipeline of DeepSeek-Coder (Guo et al., 2024), and report Pass@1 and Pass@10.

E.2 通用评测 通用领域评测遵循 lm-evaluation-harness（Gao 等，2023），在 MMLU（Hendrycks 等，2020）、BBH（Suzgun 等，2022）、AGIEval（Zhong 等，2023b）、ARC-Easy 与 ARC-Challenge（Clark 等，2018）、BoolQ（Clark 等，2019）、PIQA（Bisk 等，2020）、Hellaswag（Zellers 等，2019）、WinoGrande（Sakaguchi 等，2021）、OpenBookQA（Mihaylov 等，2018）上评测。在 HumanEval（Zheng 等，2023）与 TydiQA（Clark 等，2020）上，我们遵循 open-instruct（Ivison 等，2023）管线，报告 HumanEval 的 Pass@1 与 Pass@10，以及 TydiQA 的 F1。对 MBPP（Austin 等，2021），遵循 DeepSeek-Coder（Guo 等，2024）管线，报告 Pass@1 与 Pass@10。

Appendix F Relate the Selected Tokens’ Loss to Downstream Task Performance

附录 F 将所选 token 损失与下游任务表现关联

In this section, we declare the details about correlating the loss of selected tokens with the performance of downstream tasks. Concurrent study has explored similar methods to study the impact of scaling laws with the performance of models in downstream tasks (Gadre et al., 2024). Our analysis here differs in that it aims to elucidate the relationship between the decrease/increase in loss for selected/unselected tokens and the model’s performance on downstream tasks.

本节说明如何把所选 token 的损失与下游任务表现相关联。同期研究已探索用类似方法研究缩放定律对下游任务模型表现的影响（Gadre 等，2024）。我们这里的分析目标不同：阐明已选 / 未选 token 损失的下降 / 上升与模型下游表现之间的关系。

We use the average accuracy of MATH and GSM8K as the standard for measuring downstream tasks performance of model. Based on the trend of data points in Figure 7, we propose the relationship between the average accuracy of downstream tasks and selected/unselected tokens’ loss,

我们以 MATH 与 GSM8K 的平均准确率作为衡量模型下游任务表现的标准。基于图 7 中数据点趋势，我们提出下游任务平均准确率与已选 / 未选 token 损失之间的关系：

Acc(\mathcal{L})=\log(a*\mathcal{L}+c) \tag{7}

a

a

Appendix G Examples of Tokens Selected by SLM

附录 G SLM 所选 token 示例

G.1 Token Selected Examples In Figure 13, we present several examples of tokens selected by the SLM method, with content marked in blue indicating the tokens actually chosen during the pretraining process.

G.1 所选 token 示例 图 13 给出若干 SLM 所选 token 的例子，蓝色标记表示预训练过程中实际被选中的 token。

G.2 Dynamic Token Selected In Figure 14, we display the dynamic changes in token selection tendencies throughout the SLM training process. We chose four checkpoints during the training process (0%, 33%, 66%, and 100%) to analyze the current tendencies in token selection. The preferences for token selection are indicated by different colors, ranging from high to low preference, typically represented as deep blue, blue, black, orange, and dark orange, respectively.

G.2 动态 token 选择 图 14 展示 SLM 训练过程中 token 选择倾向的动态变化。我们选取训练过程中四个检查点（0%、33%、66%、100%）分析当时的选择倾向。偏好由高到低通常用深蓝、蓝、黑、橙、深橙表示。

Appendix H Self-Reference Setting

附录 H 自参考设定

\mathcal{L}_{\text{RM}}(x_{i})

Score Function	Score	Function	Select Ratio	Select	Ratio	GSM8K	MATH	SVAMP
Score
Function
Select
Ratio
-	100%	6.3	2.6	21.7	36.7	47.7	13.9	21.5
$\mathcal{L}_{\text{RM}}(x_{i})$	90%	7.4	4.4	23.4	38.7	51.9	14.4	23.4
80%	6.4	4.6	23.1	39.7	52.0	14.3	23.4
70%	6.7	4.6	23.3	40.0	54.5	14.3	23.9
60%	7.0	4.6	22.2	38.5	52.2	13.7	23.0
50%	5.7	4.2	20.7	36.7	46.7	10.3	20.7
$\mathcal{H}_{\text{RM}}(x_{i})$	90%	6.7	3.0	23.7	40.3	52.3	13.1	23.2
80%	6.8	3.6	22.5	40.6	52.9	13.6	23.3
70%	7.0	4.8	23.0	39.3	50.5	13.5	23.0
60%	6.5	4.8	26.5	37.3	49.7	15.6	23.4
50%	4.7	5.8	20.9	33.8	42.5	11.1	19.8
$\mathcal{H}_{\text{RM}}(x_{i})\cup\mathcal{L}_{\text{RM}}(x_{i})$	$50\%\cup 70\%(80\%)$	6.4	3.6	22.7	38.4	52.6	15.3	23.2
$70\%\cup 60\%(77\%)$	6.3	4.6	24.4	39.6	51.4	16.3	23.8
$70\%\cup 50\%(75\%)$	6.9	5.6	23.2	39.9	52.9	12.6	23.5
$60\%\cup 60\%(70\%)$	6.7	5.2	24.7	39.2	50.6	14.6	23.5
$60\%\cup 50\%(68\%)$	7.1	5.8	21.7	37.3	49.6	15.3	22.8
	$60\%\cup 40\%(65\%)$	7.3	6.0	23.6	36.9	48.6	13.1	22.6
$\mathcal{H}_{\text{RM}}(x_{i})\cap\mathcal{L}_{\text{RM}}(x_{i})$	$80\%\cap 90\%(76\%)$	6.0	4.4	23.7	38.5	51.2	13.3	22.8
$75\%\cap 75\%(72\%)$	7.8	5.2	24.2	39.4	54.9	14.7	24.4
$70\%\cap 90\%(68\%)$	6.8	4.6	22.2	40.3	53.0	14.8	23.6
$80\%\cap 80\%(67\%)$	8.2	6.4	21.2	39.1	53.4	15.0	23.9
$70\%\cap 70\%(60\%)$	7.1	5.0	23.5	41.2	53.8	18.0	24.8

Score
Function

Select
Ratio

\mathcal{L}_{\text{RM}}(x_{i})

打分函数	选择比例	GSM8K	MATH	SVAMP	ASDiv	MAWPS	MQA	AVG
-	100%	6.3	2.6	21.7	36.7	47.7	13.9	21.5
$\mathcal{L}_{\text{RM}}(x_{i})$	90%	7.4	4.4	23.4	38.7	51.9	14.4	23.4
	80%	6.4	4.6	23.1	39.7	52.0	14.3	23.4
	70%	6.7	4.6	23.3	40.0	54.5	14.3	23.9
	60%	7.0	4.6	22.2	38.5	52.2	13.7	23.0
	50%	5.7	4.2	20.7	36.7	46.7	10.3	20.7
$\mathcal{H}_{\text{RM}}(x_{i})$	90%	6.7	3.0	23.7	40.3	52.3	13.1	23.2
	80%	6.8	3.6	22.5	40.6	52.9	13.6	23.3
	70%	7.0	4.8	23.0	39.3	50.5	13.5	23.0
	60%	6.5	4.8	26.5	37.3	49.7	15.6	23.4
	50%	4.7	5.8	20.9	33.8	42.5	11.1	19.8
$\mathcal{H}_{\text{RM}}(x_{i})\cup\mathcal{L}_{\text{RM}}(x_{i})$	$50\%\cup 70\%(80\%)$	6.4	3.6	22.7	38.4	52.6	15.3	23.2
	$70\%\cup 60\%(77\%)$	6.3	4.6	24.4	39.6	51.4	16.3	23.8
	$70\%\cup 50\%(75\%)$	6.9	5.6	23.2	39.9	52.9	12.6	23.5
	$60\%\cup 60\%(70\%)$	6.7	5.2	24.7	39.2	50.6	14.6	23.5
	$60\%\cup 50\%(68\%)$	7.1	5.8	21.7	37.3	49.6	15.3	22.8
	$60\%\cup 40\%(65\%)$	7.3	6.0	23.6	36.9	48.6	13.1	22.6
$\mathcal{H}_{\text{RM}}(x_{i})\cap\mathcal{L}_{\text{RM}}(x_{i})$	$80\%\cap 90\%(76\%)$	6.0	4.4	23.7	38.5	51.2	13.3	22.8
	$75\%\cap 75\%(72\%)$	7.8	5.2	24.2	39.4	54.9	14.7	24.4
	$70\%\cap 90\%(68\%)$	6.8	4.6	22.2	40.3	53.0	14.8	23.6
	$80\%\cap 80\%(67\%)$	8.2	6.4	21.2	39.1	53.4	15.0	23.9
	$70\%\cap 70\%(60\%)$	7.1	5.0	23.5	41.2	53.8	18.0	24.8

\mathcal{L}_{\text{RM}}(x_{i})

\mathcal{L}_{\text{RM}}(x_{i})

\mathcal{H}_{\text{RM}}(x_{i})=-\sum_{k=1}^{V}P(t_{k}|x_{<i})\log P(t_{k}|x_{<i}), \tag{8}

t_{k}

t_{k}

Appendix I Weak-to-Strong Generalization

附录 I 弱到强泛化

Model Train Toks GSM8K MATH SVAMP ASDiv MAWPS TAB MQA MMLU STEM MMLU STEM SAT AVG MMLU STEM Llama-2-7B-CT 15B 28.4 13.6 50.3 62.8 79.5 37.6 34.1 41.6 43.5 43.5 Llama-2-7B-CT w/ 1B RM 10.5B 29.8 16.0 55.5 63.7 80.4 37.9 34.3 38.2 43.8 44.4 MMLU STEM Table 5: Weak-to-Strong generalization result on math benchmark.

MMLU
STEM

模型 训练 Token GSM8K MATH SVAMP ASDiv MAWPS TAB MQA MMLU STEM SAT AVG Llama-2-7B-CT 15B 28.4 13.6 50.3 62.8 79.5 37.6 34.1 41.6 43.5 43.5 Llama-2-7B-CT w/ 1B RM 10.5B 29.8 16.0 55.5 63.7 80.4 37.9 34.3 38.2 43.8 44.4 表 5： 数学基准上的弱到强泛化结果 。

模型	训练 Token	GSM8K	MATH	SVAMP	ASDiv	MAWPS	TAB	MQA	MMLU STEM	SAT	AVG
Llama-2-7B-CT	15B	28.4	13.6	50.3	62.8	79.5	37.6	34.1	41.6	43.5	43.5
Llama-2-7B-CT w/ 1B RM	10.5B	29.8	16.0	55.5	63.7	80.4	37.9	34.3	38.2	43.8	44.4

Apart from the main experiments where we use the same base model for the reference and continual pretraining, we also investigate if a smaller reference model can effectively guide the pretraining of a larger model. We use Tinyllama-1.1B as reference model and continual pretraining Llama-2-7B on 15B OpenWebMath tokens. Results presented in Table 5 indicate that, despite the considerable gap between the small and large models (Li et al., 2023f), employing the small reference model to token selection can still yield benefits to the pre-training of the larger model. If reference and training models have different vocabularies, one can consider performing token alignment (Wan et al., 2024; Fu et al., 2023), which we leave for future work.

除主实验中参考模型与持续预训练使用同一基座外，我们还考察较小参考模型能否有效引导更大模型的预训练。以 Tinyllama-1.1B 为参考模型，在 15B OpenWebMath token 上持续预训练 Llama-2-7B。表 5 结果表明：尽管小模型与大模型之间差距可观（Li 等，2023f），用小参考模型做 token 选择仍能为更大模型的预训练带来收益。若参考模型与训练模型词表不同，可考虑做 token 对齐（Wan 等，2024；Fu 等，2023），我们留给未来工作。

Figure 11: Sample text containing four categories of tokens. Among them, blue represents tokens of categorie H\toL, green indicates tokens of categorie L\toL, yellow signifies tokens of categorie H\toH, and red denotes tokens of categorie L\toH.

图 11： 含四类 token 的样例文本。其中蓝色为 H\toL，绿色为 L\toL，黄色为 H\toH，红色为 L\toH 。

Figure 12: An example of an abnormal state of token perplexity during pretrainig process. The tokens highlighted in orange represent tokens that were significant abnormalities during the pretraining process.

图 12： 预训练过程中 token 困惑度异常状态的示例。橙色高亮表示预训练中显著异常的 token 。

Figure 13: Specific examples of selecting tokens during the selective pretraining process of the Rho-1. The tokens marked in blue represent the actual tokens trained during the training process, while the remaining black tokens are not trained during the training process.

图 13： Rho-1 选择性预训练过程中选 token 的具体例子。蓝色标记为训练中实际被训练的 token，其余黑色 token 在训练中不被训练 。

Figure 14: An example of dynamic token selection changes during the training process, which illustrated with five different score levels represented by deep blue, light blue, black, light orange, and dark orange. The bluer the color indicates a higher tendency for the token to be selected, while the more orange the color suggests a lower tendency for the token to be selected.

图 14： 训练过程中动态 token 选择变化的示例，用深蓝、浅蓝、黑、浅橙、深橙五个得分等级示意。颜色越蓝表示越倾向被选中，越橙表示越不倾向被选中 。

Rho-1: Not All Tokens Are What You Need

Rho-1：并非所有 token 都是预训练所需

Abstract

摘要

1 Introduction

1 引言

2 Selective Language Modeling

2 选择性语言建模

2.1 Not All Tokens Are Equal: Training Dynamics of Token Loss

2.1 并非所有 token 平等：token 损失的训练动态

Overview

概览

Reference Modeling

参考建模

Selective Pretraining

选择性预训练

3 Experiments

3 实验

Reference Model Training

参考模型训练

Pretraining Corpus

预训练语料

Pretraining Setting

预训练设定

Baseline Setting

基线设定

Evaluation Setup

评测设定

Few-shot CoT Reasoning Results

少样本 CoT 推理结果

Tool-Integrated Reasoning Results

工具集成推理结果

3.3 General Pre-training Results

3.3 通用预训练结果

3.4 Self-Reference Results

3.4 自参考结果

Selected Token Loss Aligns Better with Downstream Performance

所选 token 损失与下游表现更对齐

What Tokens are Selected with SLM?

SLM 选出了哪些 token？

Effect of Token Select Ratio

token 选择比例的影响

4 Conclusion

4 结论

Acknowledgments

致谢

References

参考文献

Appendix

附录

Appendix A Author Contributions

附录 A 作者贡献

Appendix B Related Works

附录 B 相关工作

B.1 Pretraining Data Optimization

B.1 预训练数据优化

B.2 Data Selection

B.2 数据选择

B.3 Language Model Training Dynamics

B.3 语言模型训练动态

B.4 Scaling Laws

B.4 缩放定律

Appendix C Limitations and Future Work

附录 C 局限与未来工作

Generalizability

可泛化性

Scalability

可扩展性

Is training a reference model necessary?

是否必须训练参考模型？

How to improve upon SLM?

如何在 SLM 之上改进？

Expanding the use of SLM

拓展 SLM 的用途

Appendix D Analysis and Visualization of Tokens in Pretraining

附录 D 预训练中 token 的分析与可视化

D.1 More Details of Four Categories Tokens

D.1 四类 token 的更多细节

D.2 Non-Converging Tokens in Pretrainig

D.2 预训练中不收敛的 token