Scaling Data-Constrained Language Models

Niklas Muennighoff     Alexander M. Rush      Boaz Barak      Teven Le Scao     Aleksandra Piktus     Nouamane Tazi     Sampo Pyysalo     Thomas Wolf     Colin Raffel      Hugging Face    Harvard University    University of Turku n.muennighoff@gmail.com

Abstract

摘要

The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

当前放大语言模型的趋势是同时增加参数量与训练数据集规模。外推这一趋势会发现：训练数据规模很快可能受限于互联网上可用文本量。受此限制驱动，我们研究数据受限情形下的语言模型缩放。具体地，我们运行一大组实验，改变数据重复程度与算力预算，最大到 9000 亿训练 token 与 90 亿参数模型。我们发现：在固定算力预算且数据受限时，用重复数据最多训到约 4 个 epoch，相对拥有唯一数据时损失变化可忽略。但重复更多后，继续加算力的收益最终衰减到零。我们提出并经验验证一条计入重复 token 与多余参数价值递减的算力最优缩放律。最后，我们尝试缓解数据稀缺的方法，包括用代码数据增强训练集，或去掉常用过滤。逾 400 次训练运行的模型与数据集公开于 https://github.com/huggingface/datablations。

1 Introduction

1 引言

Recent work on compute-optimal language models [42] shows that many previously trained large language models (LLMs, which we define as having more than one billion parameters) could have attained better performance for a given compute budget by training a smaller model on more data. Notably, the 70-billion parameter Chinchilla model [42] outperforms the 280-billion parameter Gopher model [89] while using a similar compute budget by being trained on four times more data. Extrapolating these laws for compute allocation (hereafter "Chinchilla scaling laws") to a 530 billion parameter model, such as the under-trained MT-NLG model [99], would require training on a massive 11 trillion tokens, corresponding to more than 30 terabytes of text data. For most languages, available data is several orders of magnitude smaller, meaning that LLMs in those languages are already data-constrained. Villalobos et al. 2022 estimate that even high-quality English language data will be exhausted by the year 2024 given the Chinchilla scaling laws and the trend of training ever-larger models. This motivates the question [112, 81]: what should we do when we run out of data?

近期关于算力最优语言模型的工作 [42] 表明：许多先前训练的大规模语言模型（LLM，我们定义为超过十亿参数）若在给定算力预算下训更小模型、看更多数据，本可获得更好表现。值得注意的是，700 亿参数的 Chinchilla [42] 以相近算力预算、通过在约四倍数据上训练，优于 2800 亿参数的 Gopher [89]。把这些算力分配律（下文称「Chinchilla 缩放律」）外推到如训练不足的 MT-NLG [99] 这样的 5300 亿参数模型，将需要在约 11 万亿 token 上训练，对应超过 30 TB 文本。对多数语言，可用数据小几个数量级，意味着这些语言上的 LLM 已经数据受限。Villalobos et al. 2022 估计：按 Chinchilla 缩放律与不断训练更大模型的趋势，即便高质量英语数据也会在 2024 年耗尽。这引出问题 [112, 81]：数据用尽后我们该怎么办？

In this work we investigate scaling large language models in a data-constrained regime, and whether training an LLM with multiple epochs of repeated data impacts scaling. Using multiple epochs is, of course, standard in machine learning generally; however, most prior large language models have been trained for a single epoch [51, 15] and some work explicitly advocates against reusing data [40]. An exception is the recent Galactica models [108] that were trained for 4.25 epochs and exhibit continually decreasing validation loss and improving downstream performance throughout training. However, the experiments of Galactica do not compare this setup to an alternative non-data-constrained model trained for one epoch on unique data. Without this comparison, it is difficult to quantify the trade-off between additional compute versus additional data collection.

本文研究数据受限情形下大规模语言模型的缩放，以及用重复数据多 epoch 训练 LLM 是否影响缩放。多 epoch 在机器学习中本属常规；但多数先前大语言模型只训单个 epoch [51, 15]，也有工作明确主张不要复用数据 [40]。例外是近期 Galactica 模型 [108]：训了 4.25 个 epoch，训练全程验证损失持续下降、下游表现持续改进。但 Galactica 的实验未把该设定与「单 epoch、不数据受限、看唯一数据」的对照模型比较。没有这一比较，很难量化「加算力」相对「再收集数据」的权衡。

Our main focus is to quantify the impact of multiple epochs in LLM training such that practitioners can decide how to allocate compute when scaling models. Toward this end, we assembled a battery of empirical training runs of varying data and compute constraints. Specifically, we train more than 400 models ranging from 10 million to 9 billion parameters for up to 1500 epochs and record final test loss. We use these results to fit a new data-constrained scaling law that generalizes the Chinchilla scaling law [42] to the repeated data regime and yields a better prediction of loss in this setting. Figure 1 summarizes our main results targeting the value of repeated data (Return) and optimal allocation of resources in that regime (Allocation). We find that, while models trained for a single epoch consistently have the best validation loss per compute, differences tend to be insignificant among models trained for up to 4 epochs and do not lead to differences in downstream task performance. Additional epochs continue to be beneficial, but returns eventually diminish to zero. We find that, in the data-constrained regime, allocating new compute to both more parameters and epochs is necessary, and that epochs should be scaled slightly faster. These findings suggest a simple way to continue scaling total training compute budgets further ahead in the future than the previously anticipated limits.

我们的主要目标是量化 LLM 训练中多 epoch 的影响，使实践者在放大模型时能决定如何分配算力。为此，我们组装了一批改变数据与算力约束的经验训练运行。具体地，我们训练逾 400 个模型，参数从 1000 万到 90 亿，最多 1500 个 epoch，并记录最终测试损失。用这些结果拟合新的数据受限缩放律，把 Chinchilla 缩放律 [42] 推广到重复数据情形，并在该设定下更好预测损失。图 1 汇总主要结果：针对重复数据价值（Return）与该情形下资源最优分配（Allocation）。我们发现：虽单 epoch 模型在每算力验证损失上始终最好，但最多约 4 个 epoch 的模型之间差异往往不显著，也不会带来下游任务表现差异。继续增加 epoch 仍有益，但收益最终趋于零。我们还发现：在数据受限情形下，新增算力需同时分配给更多参数与更多 epoch，且 epoch 应略快放大。这些发现给出一种简单方式，使总训练算力预算能比先前预期的极限走得更远。

Finally, given the challenges imposed by data constraints, we consider methods complementary to repeating for improving downstream accuracy without adding new natural language data. Experiments consider incorporating code tokens and relaxing data filtering. For code, English LLMs, such as PaLM [19] or Gopher [89], are trained on a small amount of code data alongside natural language data, though no benchmarking was reported to justify that decision. We investigate training LLMs on a mix of language data and Python data at 10 different mixing rates and find that mixing in code is able to provide a 2$\times$ increase in effective tokens even when evaluating only natural language tasks. For filtering, we revisit perplexity and deduplication filtering strategies on both noisy and clean datasets and find that data filtering is primarily effective for noisy datasets.

最后，鉴于数据约束带来的挑战，我们考虑与重复互补、且不新增自然语言数据即可改进下游准确率的方法。实验考察纳入代码 token 与放宽数据过滤。对代码：PaLM [19]、Gopher [89] 等英语 LLM 会在自然语言旁混入少量代码，但未见支撑该决策的基准报告。我们在 10 种混合比例下把 LLM 训在语言数据与 Python 数据的混合上，发现即便只评自然语言任务，混入代码也能带来约 $2\times$ 的有效 token 增加。对过滤：我们在嘈杂与干净数据集上重访困惑度与去重过滤，发现数据过滤主要对嘈杂数据集有效。

2 Background

2 背景

Predicting the scaling behavior of large models is critical when deciding on training resources. Specifically, two questions are of interest: (Allocation) What is the optimal balance of resources? (Return) What is the expected value of additional resources? For scaling LLMs, the resource is compute (measured in FLOPs), and it can be allocated to training a larger model or training for more steps.[^1] The metric used to quantify progress is the model’s loss on held-out data, i.e. the ability to predict the underlying data as measured in the model’s cross-entropy [2, 42]. We aim to minimize the loss ($L$) subject to a compute resource constraint ($C$) via optimal allocation to $N$ and $D$ as:

决定训练资源时，预测大模型缩放行为至关重要。具体有两个问题值得关心：(Allocation) 资源的最优平衡是什么？(Return) 追加资源的期望收益是什么？对放大 LLM，资源是算力（以 FLOPs 计），可分配给训更大模型或训更多步。[^1]量化进展的指标是模型在留出数据上的损失，即用交叉熵衡量的对底层数据的预测能力 [2, 42]。我们旨在算力资源约束 ($C$) 下，通过对 $N$ 与 $D$ 的最优分配最小化损失 ($L$)：

$$\argmin_{N,D}L(N,D)\text{ s.t. }\text{FLOPs}(N,D)=C \tag{1}$$

Currently, there are established best practices for scaling LLMs. Return follows a power-law: loss scales as a power-law with the amount of compute used for training [39, 46, 6, 35, 7, 41]. Allocation is balanced: resources are divided roughly equally between scaling of parameters and data [42]. These scaling laws were established empirically by training LLMs and carefully extrapolating behavior.

当前 LLM 缩放已有成熟实践。Return 遵循幂律：损失随用于训练的算力量呈幂律缩放 [39, 46, 6, 35, 7, 41]。Allocation 保持平衡：资源在参数放大与数据放大之间大致均分 [42]。这些缩放律由训练 LLM 并谨慎外推行为经验建立。

Chinchilla [42] uses three methods for making scaling predictions:

Chinchilla [42] 用三种方法做缩放预测：

• • (Fixed Parameters) Train with a fixed model size but on varying amounts of data.

• • (固定参数) 固定模型规模，改变数据量训练。

• • (Fixed FLOPs) Train with fixed computation while parameters and training tokens vary.

• • (固定 FLOPs) 固定算力，改变参数与训练 token。

• • (Parametric Fit) Derive and fit a formula for the loss.

• • (参数化拟合) 推导并拟合损失公式。

For the parametric fit, the loss ($L$) is a function of parameters ($N$) and training tokens ($D$):

对参数化拟合，损失 ($L$) 是参数 ($N$) 与训练 token ($D$) 的函数：

$$L(N,D)=\frac{A}{N^{\alpha}}+\frac{B}{D^{\beta}}+E \tag{2}$$

Where $\{A,\alpha,B,\beta,E\}$ are learned variables fit using the training runs from the first two approaches [42]. Using these learned variables, they propose calculating the optimal allocation of compute ($C$) to $N$ and $D$ as follows:

其中 $\{A,\alpha,B,\beta,E\}$ 是用前两种方法的训练运行拟合的学习变量 [42]。利用这些学习变量，他们提出把算力 ($C$) 最优分配给 $N$ 与 $D$ 的计算如下：

$$\displaystyle N_{opt}(C)=G{\left({C}/{6}\right)}^{a}\hskip 10.00002ptD_{opt}(C)=G^{-1}{\left({C}/{6}\right)}^{b} \\ \displaystyle\hskip 10.00002pt\text{ where } \\ \displaystyle G={\left(\frac{\alpha A}{\beta B}\right)}^{\frac{1}{\alpha+\beta}}\hskip 10.00002pta=\frac{\beta}{\alpha+\beta}\hskip 10.00002ptb=\frac{\alpha}{\alpha+\beta}$$

These methods lead to the conclusion that $\alpha\approx\beta$ and hence $N$ and $D$ should be scaled proportionally for compute-optimal training. As loss can be an imperfect proxy for performance on natural language tasks [123, 97, 105], they also validate their conclusions on various downstream tasks.

这些方法得出 $\alpha\approx\beta$，因而算力最优训练应等比例放大 $N$ 与 $D$。由于损失可能是自然语言任务表现的不完美代理 [123, 97, 105]，他们也在多种下游任务上验证结论。

3 Method: Data-Constrained Scaling Laws

3 方法：数据受限缩放律

We are interested in scaling behavior in the data-constrained regime. Specifically, given a limited amount of unique data, what is the best Allocation of and Return for computational resources. Prior work [46, 42] assumes that the necessary data to support scaling is unlimited. Our aim is therefore to introduce a modified version of Equation 2 that accounts for data constraints and fit the terms in the modified scaling law to data from a large body of experiments.

我们关心数据受限情形下的缩放行为。具体地，给定有限唯一数据量，算力资源的最佳Allocation与Return是什么。先前工作 [46, 42] 假定支撑缩放所需的数据无限。因此我们的目标是引入计入数据约束的式 2 修改版，并把修改后缩放律中的各项拟合到大量实验数据。

The primary method we consider is repeating data, i.e. allocating FLOPs to multiple epochs on the same data. Given a budget of unique data $D_{C}$, we split the Chinchilla total data term $D$ into two parts: the number of unique tokens used, $U_{D}$, and the number of repetitions, $R_{D}$ (i.e. epochs - 1). Given total training tokens $D$ and data budget $D_{C}$ these terms are simply computed as $U_{D}=\min\{D_{C},D\}$ and $R_{D}=(D/U_{D})-1$. When training for a single epoch like done in prior scaling studies, $R_{D}=0$. We are thus interested in minimizing Equation 1 with the additional constraint of a data budget $D_{C}$:

我们主要考虑的方法是重复数据，即把 FLOPs 分配到同一数据上的多个 epoch。给定唯一数据预算 $D_{C}$，我们把 Chinchilla 的总数据项 $D$ 拆成两部分：所用唯一 token 数 $U_{D}$，以及重复次数 $R_{D}$（即 epochs $-$ 1）。给定总训练 token $D$ 与数据预算 $D_{C}$，这两项简单算为 $U_{D}=\min\{D_{C},D\}$、$R_{D}=(D/U_{D})-1$。像先前缩放研究那样只训单 epoch 时，$R_{D}=0$。因此我们关心在额外数据预算 $D_{C}$ 约束下最小化式 1：

$$\argmin_{N,D}L(N,D)\text{ s.t. }\text{FLOPs}(N,D)=C,U_{D}\leq D_{C} \tag{4}$$

Symmetrically, for mathematical convenience, we split the parameter term $N$ into two parts: the base number of parameters needed to optimally fit the unique tokens $U_{N}$, and the number of times to “repeat” this initial allocation, $R_{N}$. We compute $U_{N}$ by first rearranging Equation 3 to find the optimal compute budget for the unique tokens used ($U_{D}$). We input this value into the $N_{opt}$ formula of Equation 3 to get $U_{N}=\min\{N_{opt},N\}$. $U_{N}$ thus corresponds to the compute-optimal number of parameters for $U_{D}$ or less if $N<N_{opt}$. Once we have $U_{N}$, we compute the repeat value as $R_{N}=(N/U_{N})-1$.

对称地，为数学方便，我们把参数项 $N$ 也拆成两部分：最优拟合唯一 token $U_{D}$ 所需的基础参数量 $U_{N}$，以及把这份初始分配「重复」的次数 $R_{N}$。计算 $U_{N}$ 时，先重排式 3 得到所用唯一 token（$U_{D}$）的最优算力预算，再代入式 3 的 $N_{opt}$ 公式得 $U_{N}=\min\{N_{opt},N\}$。因而 $U_{N}$ 对应 $U_{D}$ 的算力最优参数量，或在 $N<N_{opt}$ 时取更小的 $N$。得到 $U_{N}$ 后，重复值算为 $R_{N}=(N/U_{N})-1$。

To empirically explore the scaling behavior in a data-limited setting we train LLMs under these constraints. We consider three different experimental protocols in this work:

为经验探索数据受限设定下的缩放行为，我们在这些约束下训练 LLM。本文考虑三种实验协议：

• • (Fixed Unique Data) In §5 we fix the data constraint $D_{C}$ and train models varying epochs and parameters. These experiments target Allocation, specifically tradeoff of $D$ and $N$.

• • (固定唯一数据) 第 5 节固定数据约束 $D_{C}$，改变 epoch 与参数训练。这些实验针对Allocation，具体是 $D$ 与 $N$ 的权衡。

• • (Fixed FLOPs) In §6 we fix the computation available and vary $D_{C}$ (and thus also $U_{D}$ and $U_{N}$). These experiments target Return, i.e. how well does repeating scale compared to having more unique data.

• • (固定 FLOPs) 第 6 节固定可用算力，改变 $D_{C}$（因而也改变 $U_{D}$ 与 $U_{N}$）。这些实验针对Return，即重复相对拥有更多唯一数据缩放得如何。

• • (Parametric Fit) We fit a formula introduced in §3.1 on all our training runs and evaluate its predictive capability throughout §5 and §6.

• • (参数化拟合) 我们在全部训练运行上拟合第 3.1 节引入的公式，并在第 5、6 节评估其预测能力。

Before discussing experimental results we describe the parametric assumptions.

在讨论实验结果前，先描述参数化假设。

3.1 Parametric Fit

To extrapolate scaling curves, it is necessary to incorporate repetition into the Chinchilla formula (Equation 2). We generalize Equation 2 by replacing $D$ and $N$ with terms corresponding to the effective data ($D^{\prime}$) and effective model parameters ($N^{\prime}$).

3.1 参数化拟合

要外推缩放曲线，必须把重复纳入 Chinchilla 公式（式 2）。我们通过把 $D$ 与 $N$ 替换为对应有效数据（$D^{\prime}$）与有效模型参数（$N^{\prime}$）的项，推广式 2。

$$L(N,D)=\frac{A}{N^{\prime\alpha}}+\frac{B}{D^{\prime\beta}}+E$$

Intuitively, $D^{\prime}$ should be smaller or equal to $D$ where $D$ is the total number of processed tokens since repeated tokens provide less useful information to the model than new ones. We use an exponential decay formulation, where the value of a data token processed loses roughly $(1-1/R^{*}_{D})$ fraction of its value per repetition, where $R^{*}_{D}$ is a learned constant. After some derivations and approximations (see Appendix A), this boils down to

直观上，$D^{\prime}$ 应小于或等于总处理 token 数 $D$，因为重复 token 提供的有用信息少于新 token。我们采用指数衰减形式：每重复一次，已处理数据 token 大约失去 $(1-1/R^{*}_{D})$ 比例的价值，其中 $R^{*}_{D}$ 为学习常数。经若干推导与近似（见附录 A）后，得到

$$D^{\prime}=U_{D}+U_{D}R_{D}^{*}(1-e^{\frac{-R_{D}}{R_{D}^{*}}})\;. \tag{5}$$

Note that for $R_{D}=0$ (no repetitions), $D^{\prime}=U_{D}=D$. For $R_{D}\ll R^{*}_{D}$, $e^{-R_{D}/R^{*}_{D}}\approx 1-\tfrac{R_{D}}{R^{*}_{D}}$ and so

注意：当 $R_{D}=0$（无重复）时，$D^{\prime}=U_{D}=D$。当 $R_{D}\ll R^{*}_{D}$ 时，$e^{-R_{D}/R^{*}_{D}}\approx 1-\tfrac{R_{D}}{R^{*}_{D}}$，因而

$$D^{\prime}\approx U_{D}+U_{D}R^{*}_{D}(1-1+R_{D}/R^{*}_{D})=U_{D}(1+R_{D})=D$$

and hence in this case, repeated data is worth almost the same as fresh data. (This is also consistent with the predictions of the “deep bootstrap” framework [76].) As $R_{D}$ grows, the value of repeated tokens tends to zero, and the effective data $D^{\prime}$ becomes much smaller than $D$. The formula implies that no matter how many times we repeat the data, we will not get a better loss than could be obtained with a single epoch on $U_{D}+U_{D}R^{*}_{D}$ fresh tokens.

此时重复数据几乎与新鲜数据等价。（这也与「深度 bootstrap」框架的预测一致 [76]。）随 $R_{D}$ 增大，重复 token 价值趋于零，有效数据 $D^{\prime}$ 远小于 $D$。公式意味着：无论重复多少次，损失都不会好过在 $U_{D}+U_{D}R^{*}_{D}$ 个新鲜 token 上单 epoch 所能达到的损失。

Just as processing repeated tokens yields a diminishing return, both intuitively and empirically, models with sizes that vastly outstrip the available data also offer diminishing returns per parameter. Hence we use a symmetric formula for the number of effective parameters, where again $R^{*}_{N}$ is learned,

正如处理重复 token 收益递减，模型规模远超可用数据时，每参数收益也递减——这既合直觉也合经验。因此我们对有效参数数使用对称公式，同样学习 $R^{*}_{N}$，

$$N^{\prime}=U_{N}+U_{N}R_{N}^{*}(1-e^{\frac{-R_{N}}{R_{N}^{*}}})\;. \tag{6}$$

The learned constants $R^{*}_{D}$, $R^{*}_{N}$ roughly correspond to the “half-life” of repeated data and excess parameters. For example, at $R_{D}=R^{*}_{D}$, the number of effective tokens $D^{\prime}$ is $U_{D}+U_{D}R_{D}(1-e^{-1})$ which means that the $U_{D}R_{D}$ repeated tokens are worth on average $1-1/e$ fraction of fresh ones.

学习常数 $R^{*}_{D}$、$R^{*}_{N}$ 大致对应重复数据与多余参数的「半衰期」。例如在 $R_{D}=R^{*}_{D}$ 时，有效 token 数 $D^{\prime}$ 为 $U_{D}+U_{D}R_{D}(1-e^{-1})$，意味着 $U_{D}R_{D}$ 个重复 token 平均只值新鲜 token 的 $1-1/e$。

Using a methodology similar to [42], $R_{N}^{*}$ and $R_{D}^{*}$ can be fit on empirical measurements, which yields data-driven estimates. See Appendix A for more details on the derivations and the fitting procedure.

采用与 [42] 类似的方法，可在经验测量上拟合 $R_{N}^{*}$ 与 $R_{D}^{*}$，得到数据驱动估计。推导与拟合流程见附录 A。

4 Experimental Setup

4 实验设定

Figure 2: Dataset setup. We ensure that runs using less data (more epochs) always use a subset of the data used in runs with more data (fewer epochs).

图 2：数据集设定。我们确保：使用更少数据（更多 epoch）的运行，始终使用更多数据（更少 epoch）运行所用数据的子集。

For all experiments, we train transformer language models with the GPT-2 architecture and tokenizer [88]. Models have up to 8.7 billion parameters and are trained for up to 900 billion total tokens. Following [42] we use cosine learning rate schedules that decay 10$\times$ over the course of training for each model (different schedules led to different estimates in [46]). Unlike [46], we do not use early stopping to also explore the extent of overfitting when repeating. Other hyperparameters are based on prior work [89, 42] and detailed in Appendix S. Models are trained on subsets of C4 [90]. The data constraints are carefully defined to ensure maximal overlap as shown in Figure 2. Unlike [40], we always repeat the entire available data rather than subsets of it. Data is shuffled after each epoch. As repeating data can result in extreme overfitting (see Appendix H), we report loss on a held-out test set unless otherwise specified (see Appendix K). This contrasts training loss used in [42], but should not alter our findings as the held-out data stems from the same underlying dataset.

全部实验中，我们训练带 GPT-2 架构与分词器的 Transformer 语言模型 [88]。模型最多 87 亿参数，最多训 9000 亿总 token。遵循 [42]，我们对每个模型使用在训练过程中衰减 $10\times$ 的余弦学习率日程（不同日程在 [46] 中导致不同估计）。与 [46] 不同，我们不做早停，以便同时探索重复时的过拟合程度。其余超参基于先前工作 [89, 42]，细节见附录 S。模型在 C4 [90] 的子集上训练。数据约束经仔细定义以保证最大重叠，见图 2。与 [40] 不同，我们始终重复全部可用数据而非其子集会。每个 epoch 后打乱数据。由于重复数据可能导致极端过拟合（见附录 H），除非另有说明，我们报告留出测试集上的损失（见附录 K）。这与 [42] 使用的训练损失不同，但不应改变我们的发现，因为留出数据来自同一底层数据集。

5 Results: Resource Allocation for Data-Constrained Scaling

5 结果：数据受限缩放的资源分配

Figure 3: IsoLoss contours for 100 million unique tokens. (Left): 93 models trained with varying parameters and epochs on a fixed dataset. Contours show an interpolation of results with the same final test loss. (Right): Comparison with the loss predictions from our proposed scaling laws for the same budget of 100 million unique tokens and the predicted efficient frontier. The diminishing returns from training on repeated data can be seen in the increase in distance of the contour curves.

图 3：1 亿唯一 token 的 IsoLoss 等高线。*(左)：在固定数据集上以不同参数与 epoch 训练的 93 个模型。等高线为相同最终测试损失结果的插值。(右)：*与我们提出的缩放律在同一 1 亿唯一 token 预算下的损失预测及预测高效前沿比较。等高线间距增大，可见重复数据训练的收益递减。

Our first experimental setting considers scaling in a setting where all models have the same data constraint. For these experiments, the unique training data budget $D_{C}$ is fixed at either 100M, 400M or 1.5B tokens. For each data budget, we train a set of language models with increasing amounts of compute that is allocated to either more parameters or more epochs on the unique training data.

第一种实验设定考察所有模型具有相同数据约束时的缩放。这些实验中，唯一训练数据预算 $D_{C}$ 固定为 100M、400M 或 1.5B token。对每个数据预算，我们训练一组语言模型，把递增算力分配给更多参数，或在唯一训练数据上训更多 epoch。

Figure 3(left) shows the main results for scaling with 100M unique tokens[^2] (see Appendix C for 400M and 1.5B tokens). For 100M tokens, the corresponding one-epoch compute-optimal model according to scaling laws from [42] has $U_{N}$ of approximately 7M parameters (see Appendix B for the scaling coefficients we use). Results show that more than a 50% reduction in loss can be attained by training for several epochs ($R_{D}>0$) and increasing model size beyond what would be compute-optimal for 100M tokens ($R_{N}>0$). We find the best loss to be at around 20-60$\times$ more parameters and epochs, which corresponds to spending around 7000$\times$ more FLOPs. These results suggest that one-epoch models significantly under-utilize their training data and more signal can be extracted by repeating data and adding parameters at the cost of sub-optimal compute utilization.

图 3（左）展示在 1 亿唯一 token 上缩放的主要结果[^2]（400M 与 1.5B token 见附录 C）。对 1 亿 token，按 [42] 缩放律对应的单 epoch 算力最优模型，$U_{N}$ 约 700 万参数（我们使用的缩放系数见附录 B）。结果表明：通过训多个 epoch（$R_{D}>0$）并把模型规模增到超过对 1 亿 token 算力最优的规模（$R_{N}>0$），损失可降低超过 50%。我们发现最佳损失大约在参数与 epoch 各多约 20–60$\times$ 处，对应多花约 7000$\times$ FLOPs。这些结果提示：单 epoch 模型显著未充分利用其训练数据；以次优算力利用为代价，通过重复数据并增加参数，可以提取更多信号。

Figure 3(right) shows the predicted contours created by fitting our data-constrained scaling laws on 182 training runs. In the single-epoch case ($R_{D}=0$) with near compute-optimal parameters ($R_{N}=0$) our scaling equation (§3.1) reduces to the Chinchilla equation. In this case, both formulas predict the optimal allocation of compute to parameters and data to be the same, resulting in overlapping efficient frontiers. As data is repeated for more than a single epoch, our fit predicts that excess parameters decay faster in value than repeated data ($R_{N}^{*}<R_{D}^{*}$). As a result, the data-constrained efficient frontier suggests allocating most additional compute to more epochs rather than more parameters. This contrasts the Chinchilla scaling laws [42], which suggest equally scaling both. However, note that they do not repeat the entire training data and their parametric fit explicitly relies on the assumption that models are trained for a single epoch only. Thus, there is no guarantee that their scaling predictions hold for repeated data.

图 3（右）展示在 182 次训练运行上拟合数据受限缩放律得到的预测等高线。在单 epoch（$R_{D}=0$）且近算力最优参数（$R_{N}=0$）时，我们的缩放方程（§3.1）退化为 Chinchilla 方程。此时两公式对参数与数据的最优算力分配预测相同，高效前沿重叠。当数据重复超过单 epoch 时，我们的拟合预测多余参数价值衰减快于重复数据（$R_{N}^{*}<R_{D}^{*}$）。因此，数据受限高效前沿建议把多数追加算力分给更多 epoch 而非更多参数。这与 Chinchilla 缩放律 [42]「二者同等放大」的建议相反。但注意：他们并不重复全部训练数据，且参数化拟合显式依赖模型只训单 epoch 的假设。因此，其缩放预测对重复数据并无保证。

FLOP budget ($C$)	Parameters ($N$)	Training tokens ($D$)	Data budget ($D_{C}$)
$9.3\times 10^{20}$	$2.8$B	55B	$\{\ 55,28,18,14,11,9,4,1.25\}$B
$2.1\times 10^{21}$	$4.2$B	84B	$\{\ 84,42,28,21,17,12,6,1.9\}$B
$9.3\times 10^{21}$	$8.7$B	178B	$\{178,88,58,44,35,25,13,4\}$B

Figure 4: Validation Loss for Different Data Constraints (IsoFLOP). Each curve represents the same number of FLOPs spent on an equal size model. Colors represent different numbers of epochs due to repeating because of data constraints. Parameters and training tokens are set to match the single-epoch compute-optimal configurations for the given FLOPs. Models trained on data that is repeated for multiple epochs have consistently worse loss and diverge if too many epochs are used.

FLOP 预算 ($C$)	参数 ($N$)	训练 token ($D$)	数据预算 ($D_{C}$)
$9.3\times 10^{20}$	$2.8$B	55B	$\{55,28,18,14,11,9,4,1.25\}$B
$2.1\times 10^{21}$	$4.2$B	84B	$\{84,42,28,21,17,12,6,1.9\}$B
$9.3\times 10^{21}$	$8.7$B	178B	$\{178,88,58,44,35,25,13,4\}$B

图 4：不同数据约束下的验证损失（IsoFLOP）。每条曲线表示在同等规模模型上花费相同 FLOPs。颜色表示因数据约束而重复导致的不同 epoch 数。参数与训练 token 设为匹配给定 FLOPs 的单 epoch 算力最优配置。在因重复而多 epoch 的数据上训练的模型损失一贯更差；epoch 过多时会发散。

For all three data budgets, our results suggest that Allocation is optimized by scaling epochs faster than additional parameters. We confirm this at scale by training the data-constrained compute-optimal model for $9.3\times 10^{21}$ FLOPs and 25 billion unique tokens as suggested by our efficient frontier. Despite having 27% less parameters, this model achieves better loss than the model suggested by the Chinchilla scaling laws (Figure 1, right). Similarly, the 120 billion parameter Galactica model trained on repeated data should have been significantly smaller according to data-constrained scaling laws (Appendix G). An additional benefit of using a smaller model is cheaper inference, though adding parameters can make it easier to parallelize training across GPUs.

对全部三个数据预算，我们的结果都建议：Allocation 通过比追加参数更快地放大 epoch 来优化。我们按高效前沿建议，在 $9.3\times 10^{21}$ FLOPs 与 250 亿唯一 token 上训练数据受限算力最优模型，在大规模上确认这一点。尽管参数少 27%，该模型损失仍优于 Chinchilla 缩放律建议的模型（图 1，右）。类似地，按数据受限缩放律，在重复数据上训练的 1200 亿参数 Galactica 模型本应显著更小（附录 G）。使用更小模型的额外好处是推理更便宜，尽管增加参数可使跨 GPU 并行训练更容易。

Adding parameters and epochs causes the loss to decrease and eventually increase again, suggesting that too much compute can hurt performance. Results from [46] also show that loss can increase when too many parameters are used, even with early stopping. However, we expect that appropriate regularization (such as simply removing all excess parameters as an extreme case) could prevent this behavior. Thus, our formula presented in §3 and its predicted isoLoss contours in Figure 3 do not model the possibility that excess epochs or parameters could hurt performance.

增加参数与 epoch 会使损失先降后升，提示算力过多可能损害性能。[46] 的结果也显示：即便早停，参数过多时损失仍可能上升。不过我们预期适当正则（极端情形下干脆去掉全部多余参数）可防止该行为。因此，第 3 节公式及其在图 3 中预测的 isoLoss 等高线，并不建模多余 epoch 或参数可能损害性能的可能性。

6 Results: Resource Return for Data-Constrained Scaling

6 结果：数据受限缩放的资源回报

Figure 5: Empirical and Extrapolated loss with constrained data. (Left): Loss as a function of repeated tokens for three different training budgets each with fixed number of parameters. Loss curves predicted by our data-constrained scaling laws are shifted to exactly match the loss at 100% unique data. Return on FLOPs decays with repeated data in a regular pattern. (Right): Extrapolating from the proposed data-constrained scaling law shows that at small numbers epochs are benign, but at large number of epochs loss stops improving.

图 5：约束数据下的经验与外推损失。*(左)：三组不同训练预算、各自固定参数量下，损失随重复 token 的变化。数据受限缩放律预测的损失曲线平移以精确匹配 100% 唯一数据处的损失。重复数据上 FLOPs 的回报按规律衰减。(右)：*按提出的数据受限缩放律外推显示：少量 epoch 无害，但 epoch 很多时损失停止改进。

Next, consider the question of Return on scaling. To quantify this value, we run experiments with three FLOP budgets across eight respective data budgets to compare return on FLOPs.

接下来考虑缩放的Return问题。为量化该价值，我们在三组 FLOP 预算、各自八组数据预算上做实验，比较 FLOPs 的回报。

Figure 4shows the configurations and validation curves for models trained on the same number of total tokens. Conforming to intuition and prior work on deduplication [55], repeated data is worth less, thus models trained on less unique data (and, correspondingly, more epochs) have consistently higher loss. However, the loss difference for a few epochs is negligible. For example, the $N=8.7$ billion parameter model trained for four epochs ($D_{C}=44$ billion unique tokens) finishes training with only 0.5% higher validation loss than the single-epoch model ($D_{C}=178$ billion unique tokens).

图 4 展示在相同总 token 数上训练的模型配置与验证曲线。符合直觉与去重相关先前工作 [55]：重复数据价值更低，因而在更少唯一数据（相应更多 epoch）上训练的模型损失一贯更高。但对少量 epoch，损失差异可忽略。例如，$N=87$ 亿参数、训四个 epoch（$D_{C}=440$ 亿唯一 token）的模型，最终验证损失仅比单 epoch 模型（$D_{C}=1780$ 亿唯一 token）高 0.5%。

In Figure 5 (left), we compare the final test loss of each model to predictions from our parametric fit. The data-constrained scaling laws can accurately measure the decay in the value of repeated data as seen by the proximity of empirical results (dots) and parametric fit (lines). We note however that it significantly underestimates the final test loss of failing models where loss increases midway through training, such as models trained for 44 epochs (not depicted).

图 5（左）把每个模型的最终测试损失与参数化拟合预测比较。数据受限缩放律能准确度量重复数据价值的衰减，可见经验结果（点）与参数化拟合（线）接近。但我们注意到：对训练中途损失上升的失败模型（如训 44 个 epoch，图中未画），它显著低估最终测试损失。

In Figure 5 (right), we extrapolate the three budgets by further scaling compute while keeping the data constraints ($D_{C}$) at 55B, 84B, and 178B tokens, respectively. The parameter $R_{D}^{*}$ introduced in §3 represents roughly the “half-life” of epochs: specifically the point where repeated tokens have lost $\tfrac{1}{e}$ of their value. Through our fitting in Appendix A, we found $R_{D}^{*}\approx 15$, corresponding to 15 repetitions (or 16 epochs). Graphically, this can be seen by the stark diminishing returns in the proximity of the 16-epoch marker and the flattening out soon after.

图 5（右）在保持数据约束（$D_{C}$）分别为 55B、84B 与 178B token 的前提下进一步放大算力，外推三组预算。第 3 节引入的参数 $R_{D}^{*}$ 大致表示 epoch 的「半衰期」：具体是重复 token 失去 $\tfrac{1}{e}$ 价值的点。经附录 A 的拟合，我们得到 $R_{D}^{*}\approx 15$，对应 15 次重复（或 16 个 epoch）。图上可在 16-epoch 标记附近看到收益急剧递减，并很快趋平。

Overall, the Return when repeating data is relatively good. Meaningful gains from repeating data can be made up to around 16 epochs ($R_{D}^{*}$) beyond which returns diminish extremely fast.

总体而言，重复数据的Return相对不错。重复数据带来的有意义收益大约可持续到 16 个 epoch（$R_{D}^{*}$），之后收益衰减极快。

7 Results: Complementary Strategies for Obtaining Additional Data

7 结果：获取额外数据的互补策略

Figure 6: Strategies for data-constrained settings and their downstream performance. (Left): Schematic showing alternative data use strategies of code filling and filtering. (Right): $N=4.2$ billion parameter models trained for a total of $D=84$ billion tokens with varying budgets $D_{C}$. For repeating and filling with code, five models with different seeds are trained for each dot and the standard deviation is visualized as the shaded area.

图 6：数据受限设定下的策略及其下游表现。(左)：示意代码填充与过滤等替代数据使用策略。(右)：$N=42$ 亿参数模型，总训 $D=840$ 亿 token，改变预算 $D_{C}$。对重复与用代码填充，每个点训五个不同种子模型，阴影为标准差。

While repeating data is effective, it has diminishing returns. We therefore consider strategies for scaling $D$ targeting improved downstream performance as opposed to directly minimizing loss.

尽管重复有效，但收益递减。因此我们考虑以改进下游表现为目标、而非直接最小化损失的放大 $D$ 策略。

Figure 6 (left) illustrates the strategies: (a) Code augmentation: We use Python code from The Stack [49] to make up for missing natural language data. The combined dataset consisting of code and natural language samples is shuffled randomly. (b) Adapting filtering: We investigate the performance impact of deduplication and perplexity filtering, two common filtering steps that can severely limit available data. Removing such filtering steps can free up additional training data.

图 6（左）示意策略：(a) 代码增强：用 The Stack [49] 的 Python 代码补足缺失的自然语言数据。由代码与自然语言样本组成的合并数据集随机打乱。(b) 调整过滤：我们考察去重与困惑度过滤这两类常会严重限制可用数据的过滤步骤的性能影响。去掉此类过滤可释放额外训练数据。

For these experiments, we set a maximum data budget ($D_{C}$) of 84 billion tokens. For repetition and code filling, only a subset of $D_{C}$ is available and the rest needs to be compensated for via repeating or adding code. For both filtering methods, we start out with approximately twice the budget (178 billion tokens), as it is easier to gather noisy data and filter it than it is to gather clean data for training. For perplexity filtering, we select the top 25% samples with the lowest perplexity according to a language model trained on Wikipedia. This results in 44 billion tokens that are repeated for close to two epochs to reach the full data budget. For deduplication filtering, all samples with a 100-char overlap are removed resulting in 21 billion tokens that are repeated for four epochs during training. See Appendix N for more details on the filtering procedures.

这些实验中，我们设最大数据预算（$D_{C}$）为 840 亿 token。对重复与代码填充，只有 $D_{C}$ 的子集可用，其余需通过重复或加代码补齐。对两种过滤方法，我们从大约两倍预算（1780 亿 token）起步，因为收集嘈杂数据再过滤，往往比收集干净训练数据更容易。对困惑度过滤，我们按在 Wikipedia 上训练的语言模型选取困惑度最低的前 25% 样本，得到 440 亿 token，再重复近两个 epoch 达到完整数据预算。对去重过滤，去掉有 100 字符重叠的全部样本，得到 210 亿 token，训练中重复四个 epoch。过滤流程细节见附录 N。

When comparing across data strategies, loss ceases to be a good evaluation metric as the models are trained on different data distributions. We thus evaluate models on 19 natural language tasks with zero to five in-context few-shot exemplars [15] producing 114 scores per model. As our evaluation tasks cover different metrics and random baselines, we re-scale all scores to be in the same range to better reflect performance ranges before averaging. Details on the evaluation datasets are in Appendix K.

跨数据策略比较时，损失不再是好评测指标，因为模型训在不同数据分布上。因此我们在 19 个自然语言任务上、以零到五个上下文少样本样例评测模型 [15]，每个模型产生 114 个分数。由于评测任务覆盖不同指标与随机基线，我们在平均前把所有分数重缩放到同一区间，以更好反映表现范围。评测数据集细节见附录 K。

In Figure 6 (right) we compare the downstream performance of all strategies. For repeating data, differences in downstream performance are insignificant for up to around 4 epochs (25% budget) and then start dropping, which aligns with our results on test loss in §6. Filling up to 50% of data with code (42 billion tokens) also shows no deterioration. Beyond that, performance decreases quickly on natural language tasks. However, adding more code data may benefit non-natural language tasks, which are not considered in the benchmarking. Two of the tasks benchmarked, WebNLG [17, 34], a generation task, and bAbI [122, 57], a reasoning task, see jumps in performance as soon as code is added, possibly due to code enabling models to learn long-range state-tracking capabilities beneficial for these tasks.

图 6（右）比较全部策略的下游表现。对重复数据，最多约 4 个 epoch（25% 预算）下游表现差异不显著，之后开始下降，与第 6 节测试损失结果一致。用代码填充最多 50% 数据（420 亿 token）也未见恶化。超过该比例，自然语言任务表现迅速下降。不过，加入更多代码数据可能有利于未纳入基准的非自然语言任务。所测任务中，生成任务 WebNLG [17, 34] 与推理任务 bAbI [122, 57] 在一加入代码后表现即跃升，可能因代码使模型学到对这类任务有益的长程状态跟踪能力。

Of the filtering approaches, we find perplexity-filtering to be effective, while deduplication does not help. Prior work found deduplication was able to improve perplexity [55]; however, it did not evaluate on downstream tasks. Deduplication may have value not captured in our benchmark, such as reducing memorization [45, 40, 16, 10]. We also investigate filtering on a different noisier dataset in Appendix O, where we find it to be more effective. Overall, in a data-constrained regime, we recommend reserving filtering for noisy datasets and using both code augmentation and repeating to increase data tokens. For example, first doubling the available data by adding code and then repeating the new dataset for four epochs results in 8$\times$ more training tokens that are expected to be just as good as having had 8$\times$ more unique data from the start.

在过滤方法中，我们发现困惑度过滤有效，而去重无帮助。先前工作发现去重能改进困惑度 [55]；但未在下游任务上评测。去重可能具有我们基准未捕捉的价值，如减少记忆 [45, 40, 16, 10]。我们也在附录 O 考察了另一更嘈杂数据集上的过滤，发现更有效。总体而言，在数据受限情形下，我们建议把过滤留给嘈杂数据集，并用代码增强与重复二者增加数据 token。例如：先加代码使可用数据翻倍，再把新数据集重复四个 epoch，可得到 $8\times$ 训练 token，预期与一开始就有 $8\times$ 唯一数据一样好。

8 Related Work

8 相关工作

Large language models Scaling up transformer language models [111] across parameter count and training data has been shown to result in continuous performance gains [19]. Starting with the 1.4 billion parameter GPT-2 model [88], a variety of scaled-up language models have been trained, commonly referred to as large language models (LLMs). They can be grouped into dense models [15, 47, 58, 89, 20, 13, 132, 109, 103, 108, 130, 95, 56, 65] and sparse models [30, 131, 28, 135] depending on whether each forward pass makes use of all parameters. These models are generally pre-trained to predict the next token in a sequence, which makes them applicable to various language tasks directly after pre-training [15, 118, 50, 71, 102] by reformulating said NLP tasks as context continuation tasks (see [67] for an earlier proposal on this topic). We focus on the most common scenario, where a dense transformer model is trained to do next-token prediction on a large corpus and evaluated directly after pre-training using held-out loss or zero- to few-shot prompting.

大规模语言模型 在参数量与训练数据上放大 Transformer 语言模型 [111] 已被表明带来持续性能增益 [19]。自 14 亿参数的 GPT-2 [88] 起，已训练多种放大语言模型，通常称为大规模语言模型（LLM）。按每次前向是否使用全部参数，可分为稠密模型 [15, 47, 58, 89, 20, 13, 132, 109, 103, 108, 130, 95, 56, 65] 与稀疏模型 [30, 131, 28, 135]。这些模型一般预训练为预测序列中下一 token，因而可在预训练后直接用于多种语言任务 [15, 118, 50, 71, 102]，做法是把 NLP 任务改写为上下文续写任务（该主题的更早提议见 [67]）。我们聚焦最常见情形：稠密 Transformer 在大语料上做下一 token 预测，并在预训练后用留出损失或零到少样本提示直接评测。

Scaling laws Prior work has estimated an optimal allocation of compute for the training of LLMs. Kaplan et al. 2020 suggested a 10$\times$ increase in compute should be allocated to a 5.5$\times$ increase in model size and a 1.8$\times$ increase in training tokens. This first scaling law has led to the creation of very large models trained on relatively little data, such as the 530 billion parameter MT-NLG model trained on 270 billion tokens [99]. More recent work [42], however, showed that model size and training data should rather be scaled in equal proportions. These findings called for a renewed focus on the scaling of pre-training data rather than scaling model size via complex parallelization strategies [98, 91, 9, 78]. Up-sampling is often employed when pre-training data is partly limited, such as data from a high-quality domain like Wikipedia or text in a rare language for training multilingual LLMs [60, 82]. Hernandez et al. 2022 study up-sampling of data subsets and find that repeating only 0.1% of training data 100 times significantly degrades performance. In contrast, our work focuses on repeating the entire pre-training corpus for multiple epochs rather than up-sampling parts of it.

缩放律 先前工作估计了 LLM 训练的最优算力分配。Kaplan et al. 2020 建议算力增加 $10\times$ 时应把模型规模增约 $5.5\times$、训练 token 增约 $1.8\times$。第一条缩放律催生了在相对较少数据上训练的很大模型，如在 2700 亿 token 上训练的 5300 亿参数 MT-NLG [99]。但更近期工作 [42] 表明，模型规模与训练数据应按等比例放大。这些发现要求重新关注预训练数据的放大，而非仅靠复杂并行策略放大模型规模 [98, 91, 9, 78]。当预训练数据部分受限时（如 Wikipedia 等高质领域，或训练多语 LLM 时的稀有语言文本）常采用上采样 [60, 82]。Hernandez et al. 2022 研究数据子集上采样，发现仅把 0.1% 训练数据重复 100 次就会显著损害表现。相较之下，我们的工作聚焦把整个预训练语料重复多个 epoch，而非上采样其中部分。

Alternative data strategies Large pre-training datasets are commonly filtered to remove undesired samples or reduce noise [101]. Perplexity-based filtering, whereby a trained model is used to filter out samples with high perplexity, has been found beneficial to reduce noise in web-crawled datasets [121]. Mixing of data is employed for the pre-training data of multilingual LLMs, where text data from different languages is combined [23, 126, 100, 74]. However, both for code and natural language models, mixing different (programming) languages has been reported to under-perform monolingual models [80, 113]. Some work has investigated mixing code and natural language data for prediction tasks, such as summarizing code snippets [44] or predicting function names [4]. Several pre-training datasets for LLMs include low amounts of code data [31, 89, 95]. However, these past works generally do not provide any ablation on the drawbacks of including code or the benefits for natural language task performance. We perform a detailed benchmarking of mixing Python and natural language in LLM pre-training at 10 different mixing rates.

替代数据策略 大型预训练数据集常被过滤以去除不良样本或降低噪声 [101]。基于困惑度的过滤——用训练好的模型滤掉高困惑度样本——被发现有助于降低网络抓取数据集的噪声 [121]。多语 LLM 的预训练数据会混合不同语言的文本 [23, 126, 100, 74]。但对代码与自然语言模型，混合不同（编程）语言都被报告弱于单语模型 [80, 113]。一些工作研究为摘要代码片段 [44] 或预测函数名 [4] 等预测任务混合代码与自然语言。若干 LLM 预训练数据集包含少量代码 [31, 89, 95]。但这些先前工作通常不提供纳入代码的代价或对自然语言任务收益的消融。我们在 10 种混合比例下，对 LLM 预训练中混合 Python 与自然语言做了详细基准评测。

9 Conclusion

9 结论

This work studies data-constrained scaling, focusing on the optimal use of computational resources when unique data is limited. We propose an extension to the Chinchilla scaling laws that takes into account the decay in value of repeated data, and we fit this function using a large set of controlled experiments. We find that despite recommendations of earlier work, training large language models for multiple epochs by repeating data is beneficial and that scaling laws continue to hold in the multi-epoch regime, albeit with diminishing returns. We also consider complementary approaches to continue scaling models, and find that code gives the ability to scale an additional 2$\times$. We believe that our findings will enable further scaling of language models to unlock new capabilities with current data. However, our work also indicates that there are limits on the scaling horizon. In addition to collecting additional data, researchers should explore using current data in a more effective manner.

本文研究数据受限缩放，聚焦唯一数据有限时计算资源的最优使用。我们提出对 Chinchilla 缩放律的扩展，计入重复数据价值衰减，并用大量受控实验拟合该函数。我们发现：尽管早期工作有不同建议，通过重复数据多 epoch 训练大规模语言模型是有益的，且缩放律在多 epoch 情形下仍然成立，只是收益递减。我们还考虑继续放大模型的互补方法，发现代码能额外提供约 $2\times$ 的缩放空间。我们相信这些发现将使语言模型在现有数据上进一步放大、解锁新能力。但工作也表明缩放视野存在极限。除收集更多数据外，研究者还应探索更有效地使用当前数据。

Acknowledgments and Disclosure of Funding

致谢与资助披露

This work was co-funded by the European Union under grant agreement No 101070350. The authors wish to acknowledge CSC – IT Center for Science, Finland, for generous computational resources on the LUMI supercomputer.[^3] We are thankful for the immense support from teams at LUMI and AMD, especially Samuel Antao. Hugging Face provided storage and additional compute instances. This work was supported by a Simons Investigator Fellowship, NSF grant DMS-2134157, DARPA grant W911NF2010021, and DOE grant DE-SC0022199. We are grateful to Harm de Vries, Woojeong Kim, Mengzhou Xia and the EleutherAI community for exceptional feedback. We thank Loubna Ben Allal for help with the Python data and Big Code members for insightful discussions on scaling laws. We thank Thomas Wang, Helen Ngo and TurkuNLP members for support on early experiments.

本工作由欧盟资助协议 No 101070350 共同资助。作者感谢芬兰 CSC – IT Center for Science 在 LUMI 超级计算机上提供的慷慨算力资源。[^3]我们感谢 LUMI 与 AMD 团队尤其是 Samuel Antao 的大力支持。Hugging Face 提供了存储与额外计算实例。本工作得到 Simons Investigator Fellowship、NSF 资助 DMS-2134157、DARPA 资助 W911NF2010021 与 DOE 资助 DE-SC0022199 支持。感谢 Harm de Vries、Woojeong Kim、Mengzhou Xia 与 EleutherAI 社区的出色反馈。感谢 Loubna Ben Allal 在 Python 数据上的帮助，以及 Big Code 成员关于缩放律的深入讨论。感谢 Thomas Wang、Helen Ngo 与 TurkuNLP 成员对早期实验的支持。

References

参考文献

Aghajanyan et al. [2023] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. 2023. Scaling Laws for Generative Mixed-Modal Language Models. arXiv preprint arXiv:2301.03728 .

Alabdulmohsin et al. [2022] Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. 2022. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems , 35:22300–22312.

Allal et al. [2023] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. SantaCoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988 .

Allamanis et al. [2015] Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015. Suggesting accurate method and class names. In Proceedings of the 2015 10th joint meeting on foundations of software engineering , pages 38–49.

Bach et al. [2022] Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts .

Bahri et al. [2021] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2021. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701 .

Bansal et al. [2022] Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. 2022. Data scaling laws in NMT: The effect of noise and architecture. In International Conference on Machine Learning , pages 1466–1482. PMLR.

Bender et al. [2021] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 610–623.

Bian et al. [2021] Zhengda Bian, Hongxin Liu, Boxiang Wang, Haichen Huang, Yongbin Li, Chuanrui Wang, Fan Cui, and Yang You. 2021. Colossal-AI: A unified deep learning system for large-scale parallel training. arXiv preprint arXiv:2110.14883 .

Biderman et al. [2023a] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. 2023a. Emergent and Predictable Memorization in Large Language Models. arXiv preprint arXiv:2304.11158 .

Biderman et al. [2023b] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023b. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373 .

Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about Physical Commonsense in Natural Language. In Thirty-Fourth AAAI Conference on Artificial Intelligence .

Black et al. [2022] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. GPT-NeoX-20B: An Open-Source Autoregressive Language Model. arXiv preprint arXiv:2204.06745 .

Black et al. [2021] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata , 58.

Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901.

Carlini et al. [2022] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 .

Castro Ferreira et al. [2020] Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task Overview and Evaluation Results (WebNLG+ 2020). In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020) , pages 55–76, Dublin, Ireland (Virtual). Association for Computational Linguistics.

Chen et al. [2020] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative pretraining from pixels. In International conference on machine learning , pages 1691–1703. PMLR.

Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 .

Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling Instruction-Finetuned Language Models . arXiv preprint arXiv:2210.11416 .

Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACL .

Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457v1 .

Conneau et al. [2019] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised Cross-lingual Representation Learning at Scale. arXiv preprint arXiv:1911.02116 .

Dagan et al. [2006] Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers , pages 177–190. Springer.

De Marneffe et al. [2019] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung , volume 23, pages 107–124.

Dehghani et al. [2023] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. 2023. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442 .

Dehghani et al. [2018] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. 2018. Universal transformers. arXiv preprint arXiv:1807.03819 .

Du et al. [2022] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning , pages 5547–5569. PMLR.

Dušek et al. [2020] Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge . Computer Speech & Language , 59:123–156.

Fedus et al. [2021] William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res , 23:1–40.

Gao et al. [2020] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027 .

Gao et al. [2021] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation .

Gehman et al. [2020] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 .

Gehrmann et al. [2021] Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. 2021. The gem benchmark: Natural language generation, its evaluation and metrics. arXiv preprint arXiv:2102.01672 .

Ghorbani et al. [2021] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. 2021. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740 .

Gupta et al. [2023] Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, Mutsumi Nakamura, Arindam Mitra, Santosh Mashetty, and Chitta Baral. 2023. Instruction Tuned Models are Quick Learners. arXiv preprint arXiv:2306.05539 .

Heafield [2011] Kenneth Heafield. 2011. KenLM: Faster and Smaller Language Model Queries . In Proceedings of the Sixth Workshop on Statistical Machine Translation , pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics.

Henderson et al. [2022] Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Advances in Neural Information Processing Systems , 35:29217–29234.

Henighan et al. [2020] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701 .

Hernandez et al. [2022] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. 2022. Scaling Laws and Interpretability of Learning from Repeated Data. arXiv preprint arXiv:2205.10487 .

Hernandez et al. [2021] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. arXiv preprint arXiv:2102.01293 .

Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training Compute-Optimal Large Language Models. arXiv preprint arXiv:2203.15556 .

Huang et al. [2018] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. 2018. Music transformer. arXiv preprint arXiv:1809.04281 .

Iyer et al. [2016] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2073–2083.

Kandpal et al. [2022] Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating Training Data Mitigates Privacy Risks in Language Models .

Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 .

Khrushchev et al. [2022] Mikhail Khrushchev, Ruslan Vasilev, Alexey Petrov, and Nikolay Zinov. 2022. YaLM 100B .

Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Kocetkov et al. [2022] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. 2022. The Stack: 3 TB of permissively licensed source code. arXiv preprint arXiv:2211.15533 .

Kojima et al. [2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 .

Komatsuzaki [2019] Aran Komatsuzaki. 2019. One epoch is all you need. arXiv preprint arXiv:1906.06669 .

Kudo and Richardson [2018] Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.

Ladhak et al. [2020] Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. arXiv preprint arXiv:2010.03093 .

Laurençon et al. [2022] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The BigScience ROOTS Corpus: A 1.6 TB Composite Multilingual Dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track .

Lee et al. [2021] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021. Deduplicating Training Data Makes Language Models Better. arXiv preprint arXiv:2107.06499 .

Li et al. [2023] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.06161 .

Liang et al. [2022] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 .

Lieber et al. [2021] Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs , 1.

Lin [2004] Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81.

Lin et al. [2021] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668 .

Lin et al. [2019] Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. 2019. Choosing transfer languages for cross-lingual learning. arXiv preprint arXiv:1905.12688 .

Longpre et al. [2023a] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023a. The Flan Collection: Designing Data and Methods for Effective Instruction Tuning. arXiv preprint arXiv:2301.13688 .

Longpre et al. [2023b] Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, Deb Roy, and Sara Hooker. 2023b. The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI .

Longpre et al. [2023c] Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2023c. A Pretrainer’s Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity .

Luukkonen et al. [2023] Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Le Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. 2023. FinGPT: Large Generative Models for a Small Language. In The 2023 Conference on Empirical Methods in Natural Language Processing .

Madani et al. [2020] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. 2020. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.03497 .

McCann et al. [2018] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The Natural Language Decathlon: Multitask Learning as Question Answering . CoRR , abs/1806.08730.

Min et al. [2021] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943 .

Mostafazadeh et al. [2017] Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics , pages 46–51.

Muennighoff [2020] Niklas Muennighoff. 2020. Vilio: State-of-the-art visio-linguistic models applied to hateful memes. arXiv preprint arXiv:2012.07788 .

Muennighoff [2022] Niklas Muennighoff. 2022. SGPT: GPT Sentence Embeddings for Semantic Search. arXiv preprint arXiv:2202.08904 .

Muennighoff et al. [2023] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023. OctoPack: Instruction Tuning Code Large Language Models. arXiv preprint arXiv:2308.07124 .

Muennighoff et al. [2022a] Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022a. MTEB: Massive Text Embedding Benchmark . arXiv preprint arXiv:2210.07316 .

Muennighoff et al. [2022b] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022b. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786 .

Nakkiran et al. [2021a] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. 2021a. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003.

Nakkiran et al. [2021b] Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. 2021b. The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net.

Narayan et al. [2018] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , Brussels, Belgium.

Narayanan et al. [2021] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1–15.

Nie et al. [2020] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics.

Nijkamp et al. [2022] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 .

nostalgebraist [2022] nostalgebraist. 2022. chinchilla’s wild implications. lesswrong .

Orlanski et al. [2023] Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishah Singh, and Michele Catasta. 2023. Measuring The Impact Of Programming Language Distribution. arXiv preprint arXiv:2302.01973 .

Ortiz Su’arez et al. [2020] Pedro Javier Ortiz Su’arez, Laurent Romary, and Benoit Sagot. 2020. A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 1703–1714, Online. Association for Computational Linguistics.

Ortiz Su’arez et al. [2019] Pedro Javier Ortiz Su’arez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures . Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 – 16, Mannheim. Leibniz-Institut f"ur Deutsche Sprache.

Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 .

Penedo et al. [2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv preprint arXiv:2306.01116 .

Prabhumoye et al. [2023] Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models. arXiv preprint arXiv:2302.07388 .

Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.

Rae et al. [2021] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 .

Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21(140):1–67.

Rasley et al. [2020] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 3505–3506.

Roemmele et al. [2011] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning , pages 90–95.

Sakaguchi et al. [2021] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99–106.

Sanh et al. [2022] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In The Tenth International Conference on Learning Representations .

Scao et al. [2022a] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022a. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 .

Scao et al. [2022b] Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. 2022b. What Language Model to Train if You Have One Million GPU Hours? arXiv preprint arXiv:2210.15424 .

Shin et al. [2022] Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, et al. 2022. On the effect of pretraining corpora on in-context learning by a large-scale language model. arXiv preprint arXiv:2204.13509 .

Shoeybi et al. [2019] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 .

Smith et al. [2022] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.11990 .

Soltan et al. [2022] Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, et al. 2022. Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model. arXiv preprint arXiv:2208.01448 .

Sorscher et al. [2022] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. 2022. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486 .

Srivastava et al. [2022] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 .

Su et al. [2022] Hui Su, Xiao Zhou, Houjing Yu, Yuwen Chen, Zilin Zhu, Yang Yu, and Jie Zhou. 2022. WeLM: A Well-Read Pre-trained Language Model for Chinese. arXiv preprint arXiv:2209.10372 .

Tay et al. [2022a] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. 2022a. Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? arXiv preprint arXiv:2207.10551 .

Tay et al. [2021] Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2021. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109.10686 .

Tay et al. [2022b] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022b. Unifying Language Learning Paradigms. arXiv preprint arXiv:2205.05131 .

Tay et al. [2022c] Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. 2022c. Transcending scaling laws with 0.1% extra compute. arXiv preprint arXiv:2210.11399 .

Taylor et al. [2022] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 .

Thoppilan et al. [2022] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 .

Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 .

Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems , 30.

Villalobos et al. [2022] Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. 2022. Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning. arXiv preprint arXiv:2211.04325 .

Virtanen et al. [2019] Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076 .

Wang et al. [2019] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems . CoRR , abs/1905.00537.

Wang et al. [2023] Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. arXiv preprint arXiv:2306.04751 .

Wang et al. [2022] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv preprint arXiv:2212.10560 .

Wei et al. [2021] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 .

Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 .

Weidinger et al. [2021] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 .

Welbl et al. [2017] Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209 .

Wenzek et al. [2019] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019. CCNet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359 .

Weston et al. [2015] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698 .

Xia et al. [2022] Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2022. Training Trajectories of Language Models Across Scales. arXiv preprint arXiv:2212.09803 .

Xia et al. [2021] Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, and Ahmed Hassan Awadallah. 2021. MetaXL: Meta representation transformation for low-resource cross-lingual learning. arXiv preprint arXiv:2104.07908 .

Xu et al. [2023] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 .

Xue et al. [2020] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934 .

Yang et al. [2021] Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. 2021. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems , 34:17084–17097.

Yong et al. [2022] Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, et al. 2022. BLOOM+ 1: Adding Language Support to BLOOM for Zero-Shot Prompting. arXiv preprint arXiv:2212.09535 .

Zellers et al. [2019] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics .

Zeng et al. [2022] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. GLM-130B: An Open Bilingual Pre-trained Model. arXiv preprint arXiv:2210.02414 .

Zeng et al. [2021] Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. PanGu-alpha: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation. arXiv preprint arXiv:2104.12369 .

Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 .

Zhao et al. [2023] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 .

Zhou et al. [2023] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 .

Zoph et al. [2022] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906 .

Appendix

附录

Appendix A Derivation of Data-Constrained Scaling Laws

附录 A 数据受限缩放律的推导

Let $N$ be the number of model parameters, $D$ be the training tokens and $U$ be the "unique" training tokens i.e. the size of the dataset that is to be trained on for one or more epochs. Chinchilla [42] only deals with non-repeated tokens, thus $D=U$ and we can write their formula (“Approach 3”) as:

令 $N$ 为模型参数量，$D$ 为训练 token，$U$ 为「唯一」训练 token，即要以一个或多个 epoch 训练的数据集大小。Chinchilla [42] 只处理不重复 token，因而 $D=U$，可把其公式（「方法 3」）写成：

$$L(N,U)=\tfrac{A}{N^{\alpha}}+\tfrac{B}{U^{\beta}}+E \tag{7}$$

where $E$ represents the irreducible loss. $A$, $B$, $\alpha$ and $\beta$ are learned parameters.

其中 $E$ 表示不可约损失。$A$、$B$、$\alpha$ 与 $\beta$ 为学习参数。

We now want to generalize this expression to multiple epochs where tokens are repeated. We repeat the data $R_{D}$ times, where $R_{D}=0$ corresponds to the base case of a single epoch. We let $D^{\prime}$ be the “effective data size”: the number of unique data needed to get the same value as repeating $U$ unique tokens for $R_{D}$ repeats. Hence, if $R_{D}=0$, the effective data is the same as the total data processed. Intuitively, each time a sample is repeated, it is worth less as the model has already learned some of its information. Assume that each time a model trains on a token, it learns a $1-\delta$ fraction of the information in it for some constant $0\leq\delta\leq 1$. (Thus, if $\delta=0$ repeated tokens are as good as new ones, and if $\delta=1$, repeated tokens are worth nothing.) In other words, we expect the decrease in value of each repetition to be proportional to the value of the prior repetition, which is equivalent to exponential decay. As we would like to sum up the value of all repetitions, we temporarily assume an integral number of repeats and express it as a geometric series:

我们现在要把该表达式推广到 token 被重复的多 epoch。数据重复 $R_{D}$ 次，$R_{D}=0$ 对应单 epoch 基线。令 $D^{\prime}$ 为「有效数据大小」：为获得与把 $U$ 个唯一 token 重复 $R_{D}$ 次相同的价值，所需的唯一数据量。因此若 $R_{D}=0$，有效数据等于总处理数据。直观上，样本每被重复一次，因其部分信息已被模型学到，价值更低。假定模型每在一个 token 上训练一次，就学到其中 $1-\delta$ 比例的信息，其中常数 $0\leq\delta\leq 1$。（因而若 $\delta=0$，重复 token 与新 token 一样好；若 $\delta=1$，重复 token 毫无价值。）换言之，我们预期每次重复价值的下降与前一次重复的价值成正比，等价于指数衰减。由于要对全部重复的价值求和，我们暂时假定整数次重复，并写成几何级数：

$$D^{\prime}=U+(1-\delta)U+(1-\delta)^{2}U+\cdots+(1-\delta)^{R_{D}}U \tag{8}$$

We know that the sum $S$ of a geometric series with a common ratio $r$ is:

我们知道公比为 $r$ 的几何级数之和 $S$ 为：

$$S=\frac{a(1-r^{n})}{1-r} \tag{9}$$

where $a$ is the first term and $n$ the number of terms in the series. As $r=(1-\delta)$ and $a=(1-\delta)U$:

其中 $a$ 为首项，$n$ 为项数。由于 $r=(1-\delta)$ 且 $a=(1-\delta)U$：

$$D^{\prime}=U+U\sum_{k=1}^{R_{D}}(1-\delta)^{k}=U+(1-\delta)U\tfrac{(1-(1-\delta)^{R_{D}})}{\delta} \tag{10}$$

Note that Equation 10 can also be used with a non-integer number of repetitions. We can directly use Equation 10 as our effective data and learn $\delta$ but for convenience and interpretability, we redefine it in terms of the number of epochs beyond which repeating does not help. Note that as more data is repeated, the right-hand side tends to $\tfrac{(1-\delta)U}{\delta}$, as $\lim_{R_{D}\to\infty}(1-(1-\delta)^{R_{D}})=1$. Let $R^{*}_{D}=\tfrac{1-\delta}{\delta}$, hence $D^{\prime}$ “plateaus” at $U+R^{*}_{D}U$ as $R_{D}$ goes to infinity.

注意式 10 也可用于非整数次重复。我们可以直接把式 10 当作有效数据并学习 $\delta$，但为方便与可解释性，我们用「超过该 epoch 数后重复不再有帮助」来重定义。注意随更多数据被重复，右边趋于 $\tfrac{(1-\delta)U}{\delta}$，因为 $\lim_{R_{D}\to\infty}(1-(1-\delta)^{R_{D}})=1$。令 $R^{*}_{D}=\tfrac{1-\delta}{\delta}$，因而当 $R_{D}$ 趋于无穷时，$D^{\prime}$ 「平台」在 $U+R^{*}_{D}U$。

If we assume $\delta$ to be small, ${1-\delta}$ tends to one and we can approximate $1/R^{*}_{D}=\tfrac{\delta}{1-\delta}\approx\delta$.

若假定 $\delta$ 很小，${1-\delta}$ 趋于一，可近似 $1/R^{*}_{D}=\tfrac{\delta}{1-\delta}\approx\delta$。

Next, define $e^{x}$ in terms of its Taylor series expansion:

接下来用泰勒展开定义 $e^{x}$：

$$e^{x}=1+x+\frac{x^{2}}{2!}+\frac{x^{3}}{3!}+\dots\approx 1+x \tag{11}$$

If $x$ is small later terms become increasingly small, thus $e^{x}\approx 1+x$. As we have assumed $\delta$ to be small, let $x=-\delta$, which yields

若 $x$ 很小，后面各项越来越小，因而 $e^{x}\approx 1+x$。因已假定 $\delta$ 很小，令 $x=-\delta$，得到

$$(1+x)=(1-\delta)\approx e^{-\delta}\approx e^{-1/R^{*}_{D}} \tag{12}$$

Now inserting $(1-\delta)/\delta=R_{D}^{*}$ and $(1-\delta)^{R_{D}}=e^{(-1/R^{*}_{D})^{R_{D}}}$ into Equation 10 we get our final equation representing the effective data:

现把 $(1-\delta)/\delta=R_{D}^{*}$ 与 $(1-\delta)^{R_{D}}=e^{(-1/R^{*}_{D})^{R_{D}}}$ 代入式 10，得到表示有效数据的最终方程：

$$D^{\prime}=U+U\cdot R_{D}^{*}\cdot(1-e^{-R_{D}/R_{D}^{*}}) \tag{13}$$

where $U$ and $R_{D}$ are given while $R_{D}^{*}$ is a learned constant. If no repeats are done, the second part of the sum is zero and the term simplifies to the single-epoch scaling laws from Equation 7. While $R_{D}\ll R_{D}^{*}$, the second term is approximated as $U\cdot R_{D}$ and for $R_{D}\gg R_{D}^{*}$, it plateaus at $U\cdot R_{D}^{*}$. Hence $R^{*}_{D}$ corresponds to the number of times we can repeat tokens before seeing sharply diminishing returns.

其中 $U$ 与 $R_{D}$ 给定，而 $R_{D}^{*}$ 为学习常数。若不做重复，和式第二项为零，该项退化为式 7 的单 epoch 缩放律。当 $R_{D}\ll R_{D}^{*}$ 时，第二项近似为 $U\cdot R_{D}$；当 $R_{D}\gg R_{D}^{*}$ 时，平台在 $U\cdot R_{D}^{*}$。因此 $R^{*}_{D}$ 对应在看到急剧递减收益之前可重复 token 的次数。

Let us consider a concrete example to show that Equation 13 is a very good approximation of Equation 10 and make the equations more intuitive. Suppose repeated data retains 75% of its value ($\delta=0.25$) and we train on a single token or data unit ($U=1$) for five epochs, i.e. we repeat it four times ($R_{D}=4$). In that case Equation 10 yields $D^{\prime}=U+(1-\delta)U\tfrac{(1-(1-\delta)^{R_{D}})}{\delta}=1+(0.75)*4*(1-0.75^{4})=3.05$. Thus despite training for 5 total units (4 of which are repetitions), we only get the value equivalent to $3.05$ units. As we have defined $R_{D}^{*}=(1-\delta)/\delta$, the corresponding $R_{D}^{*}$ value is $3$. Setting $R_{D}^{*}=3$ in Equation 13 yields $D^{\prime}=U+U\cdot R_{D}^{*}\cdot(1-e^{-R_{D}/R_{D}^{*}})=1+3*(1-e^{-4/3})=3.21$. Due to our approximations, the results are not the same, i.e. $3.21$ is slightly higher than $3.05$. However, note that the data term is additionally raised to a power of $\beta=0.353$ (see Equation 7; Appendix B), thus the actual difference calculated as $((3.21^{0.353})/(3.05^{0.353}))-1$ is a mere 1.8% despite this relatively large $\delta$ of $0.25$. Equation 13 has the benefit that we can interpret $R_{D}^{*}$ as the number of repetitions beyond which repeating yields sharply diminishing returns and flattens out soon after. Consider $R_{D}=100$ then $D^{\prime}=1+3*(1-e^{-100/3})=3.99$. No matter how many repeats are done the effective data will never exceed $4$ i.e. it plateaus at $U+R_{D}^{*}U$ as $R_{D}$ tends to infinity.

我们看一个具体例子，说明式 13 是式 10 的很好近似，并让方程更直观。假定重复数据保留 75% 价值（$\delta=0.25$），我们在单个 token 或数据单元（$U=1$）上训五个 epoch，即重复四次（$R_{D}=4$）。此时式 10 给出 $D^{\prime}=U+(1-\delta)U\tfrac{(1-(1-\delta)^{R_{D}})}{\delta}=1+(0.75)*4*(1-0.75^{4})=3.05$。因而尽管总共训了 5 个单位（其中 4 个是重复），我们只得到等价于 $3.05$ 个单位的价值。由于定义 $R_{D}^{*}=(1-\delta)/\delta$，对应 $R_{D}^{*}$ 值为 $3$。在式 13 中设 $R_{D}^{*}=3$ 得 $D^{\prime}=U+U\cdot R_{D}^{*}\cdot(1-e^{-R_{D}/R_{D}^{*}})=1+3*(1-e^{-4/3})=3.21$。由于近似，结果不完全相同，即 $3.21$ 略高于 $3.05$。但注意数据项还要再取 $\beta=0.353$ 次幂（见式 7；附录 B），因而按 $((3.21^{0.353})/(3.05^{0.353}))-1$ 计算的实际差异，即便在此相对较大的 $\delta=0.25$ 下也仅 1.8%。式 13 的好处是：可以把 $R_{D}^{*}$ 解释为超过该重复次数后重复收益急剧递减并很快趋平的次数。考虑 $R_{D}=100$，则 $D^{\prime}=1+3*(1-e^{-100/3})=3.99$。无论重复多少次，有效数据都不会超过 $4$，即当 $R_{D}$ 趋于无穷时平台在 $U+R_{D}^{*}U$。

Similarly, we consider repeating parameters. Symmetric to seeing the same data, excess parameters learn the same features and do not add any value in the extreme. For the Chinchilla equation (Equation 7) increasing parameters from 1 billion to 10 billion yields the same absolute decrease in loss regardless of whether the dataset is a single token or 1 billion tokens. However, intuition and our data (Appendix F) suggest that in the first case, adding parameters should not decrease loss at all, as the additional 9 billion parameters cannot possibly learn anything from the single token that the first 1 billion parameters have not already learned. Thus, to allow excess parameters to decay to adding nothing, we also replace $N$ with a symmetric version of Equation 13 yielding our final equation:

类似地，我们考虑重复参数。与看相同数据对称，多余参数学到相同特征，极端情形下不增加任何价值。对 Chinchilla 方程（式 7），把参数从 10 亿增到 100 亿，无论数据集是单个 token 还是 10 亿 token，损失的绝对下降相同。但直觉与我们的数据（附录 F）提示：第一种情形下增加参数不应降低损失，因为额外 90 亿参数不可能从那单个 token 学到前 10 亿参数尚未学到的东西。因此，为使多余参数衰减到「什么也不加」，我们用式 13 的对称版本替换 $N$，得到最终方程：

$$\displaystyle L(U_{N},U_{D},R_{N},R_{D})=\frac{A}{(U_{N}+U_{N}R_{N}^{*}(1-e^{\frac{-R_{N}}{R_{N}^{*}}}))^{\alpha}}+\frac{B}{(U_{D}+U_{D}R_{D}^{*}(1-e^{\frac{-R_{D}}{R_{D}^{*}}}))^{\beta}}+E$$

We define $U_{N}$, as the number of "unique" parameters that provide an optimal fit for $U_{D}$. Additional parameters decay with a symmetric version of the expression for repeated data. $R_{N}$ is the number that the "unique" parameters are repeated i.e. $R_{N}=\max\{(N/U_{N})-1,0\}$. If $R_{N}^{*}=\infty$, additional parameters do not decay at all and $(U_{N}+U_{N}R_{N}^{*}(1-e^{\frac{-R_{N}}{R_{N}^{*}}}))$ reduces to $N$. We compute $U_{N}$ from $U_{D}$ by setting $D_{opt}=U_{D}$ and rearranging Equation 3 to map from $D_{opt}$ to $N_{opt}$. $U_{N}$ is then $\min\{N_{opt},N\}$. This is equivalent to the following:

我们把 $U_{N}$ 定义为给 $U_{D}$ 提供最优拟合的「唯一」参数量。额外参数按与重复数据对称的表达式衰减。$R_{N}$ 是「唯一」参数被重复的次数，即 $R_{N}=\max\{(N/U_{N})-1,0\}$。若 $R_{N}^{*}=\infty$，额外参数完全不衰减，$(U_{N}+U_{N}R_{N}^{*}(1-e^{\frac{-R_{N}}{R_{N}^{*}}}))$ 退化为 $N$。我们通过令 $D_{opt}=U_{D}$ 并重排式 3 从 $D_{opt}$ 映射到 $N_{opt}$，由 $U_{D}$ 计算 $U_{N}$。$U_{N}$ 随后取 $\min\{N_{opt},N\}$。这等价于：

$$\displaystyle U_{N}=\min\{((U_{D}\cdot G)^{\beta/\alpha})\cdot G,N\}\hskip 10.00002pt\text{ where }\hskip 10.00002ptG={\left(\frac{\alpha A}{\beta B}\right)}^{\frac{1}{\alpha+\beta}}$$

Equation 14is a generalization of Equation 7: It provides the same estimates for optimal model and data size in the single epoch case, but allows for decay in the value of parameters and tokens, thus generalizing to training for multiple epochs and with excess parameters. It can thus be used as a direct replacement of Equation 7. If $R_{N}^{*}$ and $R_{D}^{*}$ are unknown, one can simply set them to infinity by default, which will make Equation 14 completely equivalent to Equation 7.

式 14 是式 7 的推广：它在单 epoch 情形给出相同的最优模型与数据规模估计，但允许参数与 token 价值衰减，从而推广到多 epoch 训练与多余参数。因此可直接替换式 7。若 $R_{N}^{*}$ 与 $R_{D}^{*}$ 未知，可默认设为无穷，使式 14 与式 7 完全等价。

To learn the parameters $R_{N}^{*}$ and $R_{D}^{*}$, we largely follow the approach from [42]. We fix $a$, $b$, $e$, $\alpha$, $\beta$ to the values learned on C4 in Appendix B and minimize:

为学习参数 $R_{N}^{*}$ 与 $R_{D}^{*}$，我们大体遵循 [42] 的做法。把 $a$、$b$、$e$、$\alpha$、$\beta$ 固定为附录 B 在 C4 上学习的值，并最小化：

$$\displaystyle\min_{R_{N}^{*},R_{D}^{*}}\sum_{\text{Run }i}\text{Huber}_{\delta}\Big( \\ \displaystyle\text{LSE}\big(a-\alpha\log(U_{N}^{i}+U_{N}^{i}R_{N}^{*}(1-e^{\frac{-R_{N}^{i}}{R_{N}^{*}}})), \\ \displaystyle b-\beta\log(U_{D}^{i}+U_{D}^{i}R_{D}^{*}(1-e^{\frac{-R_{D}^{i}}{R_{D}^{*}}})),e\big)-\log L^{i}\Big)$$

We use the LBFGS algorithm to find local minima of the objective above, started on a grid of initialization given by: $R_{N}^{*}\in\{0.,4.,\dots,20.\}$ and $R_{D}^{*}\in\{0.,4.,\dots,20.\}$. We fit on 182 samples with parameters varying from 7 million up to 9 billion and epochs ranging from 1 to 500. We removed outliers referenced in Appendix F from our fitting, as our formulas do not allow for excess parameters or excess epochs to negatively impact performance. We assume excess parameters or epochs only cause performance to plateau but never to worsen. However, it is difficult to identify all samples where excess parameters or epochs hurt, as for some data budgets we only train a single model, thus we do not know if the loss of that model is already in the range where it starts to increase again. Further, there are samples where loss initially increases and then decreases as a function of epochs (double descent, see Appendix D), which further contributes to noise in the fitting. Nevertheless, we are able to get a fairly stable fit resulting in $R_{N}^{*}=5.309743$ and $R_{D}^{*}=15.387756$. Since $R_{D}^{*}>R_{N}^{*}$, excess parameters decay faster. Hence, the data-constrained efficient frontiers in Figures 1,3 suggest scaling compute allocated to epochs faster than to parameters. This value of $R_{D}^{*}$ yields $\delta\approx 6*10^{-2}$ ($0.19$ for $R_{N}^{*}$), which respects the assumption that $\delta$ is small. Inserting these learned parameters and the parameters from Appendix B, and simplifying Equation 15 yields the precise formulation we use to predict loss ($L$) given unique tokens ($U_{N}$), parameter repetitions ($R_{N}$) and data repetitions ($R_{D}$):

我们用 LBFGS 算法寻找上述目标的局部极小，并在初始化网格上启动：$R_{N}^{*}\in\{0.,4.,\dots,20.\}$，$R_{D}^{*}\in\{0.,4.,\dots,20.\}$。在 182 个样本上拟合，参数从 700 万到 90 亿，epoch 从 1 到 500。我们从拟合中去掉附录 F 提到的离群点，因为我们的公式不允许多余参数或多余 epoch 负面损害性能。我们假定多余参数或 epoch 只会使性能平台，从不恶化。但很难识别全部「多余参数或 epoch 已伤害」的样本：对某些数据预算我们只训单个模型，因而无法知道该模型损失是否已进入再次上升的区间。此外，有些样本损失随 epoch 先升后降（双重下降，见附录 D），进一步给拟合带来噪声。尽管如此，我们仍得到相当稳定的拟合：$R_{N}^{*}=5.309743$，$R_{D}^{*}=15.387756$。由于 $R_{D}^{*}>R_{N}^{*}$，多余参数衰减更快。因此图 1、3 中的数据受限高效前沿建议把算力更快分配给 epoch 而非参数。该 $R_{D}^{*}$ 值对应 $\delta\approx 6*10^{-2}$（对 $R_{N}^{*}$ 为 $0.19$），符合 $\delta$ 很小的假定。代入这些学习参数与附录 B 参数并化简式 15，得到我们用来在给定唯一 token（$U_{N}$）、参数重复（$R_{N}$）与数据重复（$R_{D}$）时预测损失（$L$）的精确形式：

$$\displaystyle L(U_{D},R_{N},R_{D})= \\ \displaystyle\frac{521}{(U_{N}+5.3\cdot U_{N}(1-e^{\frac{-R_{N}}{5.3}}))^{0.35}}+\frac{1488}{(U_{D}+15.4\cdot U_{D}(1-e^{\frac{-R_{D}}{15.4}}))^{0.35}}+1.87 \\ \displaystyle\text{ where }U_{N}=U_{D}\cdot 0.051$$

Parametric Fit	$R_{D}^{*}$	$R_{N}^{*}$	Loss	$R^{2}$
No decay	-	-	-	0.1430
Equation 14 but only decay $N$	-	713.0015	0.0241	0.1671
Equation 14 but only decay $D$	2.9157	-	0.0169	0.7395
Equation 14	15.3878	5.3097	0.0158	0.7810
Equation 10 for both $N$ and $D$	0.0104	0.3676	0.0155	0.8062
Equation 18 for both $N$ and $D$	0.0105	0.3676	0.0155	0.8061

Table 1: Comparison of different versions of our parametric fit. All versions are fitted on the same 182 samples. We report the fitting loss and the $R^{2}$ on those samples. No decay corresponds to assuming Chinchilla holds for repeated data without modification necessary. For Equation 10, we use the same equation for $D$ and $N$ renaming the $\delta$ to $R_{D}^{*}$ and $R_{N}^{*}$.

参数化拟合	$R_{D}^{*}$	$R_{N}^{*}$	损失	$R^{2}$
无衰减	-	-	-	0.1430
式 14 但只衰减 $N$	-	713.0015	0.0241	0.1671
式 14 但只衰减 $D$	2.9157	-	0.0169	0.7395
式 14	15.3878	5.3097	0.0158	0.7810
式 10 对 $N$ 与 $D$	0.0104	0.3676	0.0155	0.8062
式 18 对 $N$ 与 $D$	0.0105	0.3676	0.0155	0.8061

表 1：不同版本参数化拟合的比较。所有版本在同一 182 个样本上拟合。报告拟合损失与这些样本上的 $R^{2}$。无衰减对应假定 Chinchilla 对重复数据无需修改仍成立。对式 10，我们对 $D$ 与 $N$ 使用同一方程，把 $\delta$ 重命名为 $R_{D}^{*}$ 与 $R_{N}^{*}$。

We experiment with different versions of our formula and display the learned values in Table 1. No decay or decaying only $D$ or $N$ of Equation 14 leads to worse loss and $R^{2}$ than Equation 14. Thus, it is important to decay both the value of excess parameters and data repetitions. We also consider an explicit exponential where $D^{\prime}=\sum_{k=0}^{R_{D}}U*e^{-R_{D}^{*}k}$, hence from Equation 9 it follows:

我们尝试公式的不同版本，学习值见表 1。无衰减，或式 14 只衰减 $D$ 或只衰减 $N$，损失与 $R^{2}$ 都差于完整式 14。因此，同时衰减多余参数与数据重复的价值很重要。我们还考虑显式指数形式 $D^{\prime}=\sum_{k=0}^{R_{D}}U*e^{-R_{D}^{*}k}$，因而由式 9 有：

$$\displaystyle D^{\prime}=U\tfrac{1-(e^{-R_{D}^{*}})^{R_{D}+1}}{1-e^{-R_{D}^{*}}}$$

This explicit decay, Equation 10, and Equation 14 all yield similar results with $R^{2}$ around 80. Equation 14 fits the data slightly worse than Equation 10, likely due to our approximations. Nevertheless, we use Equation 14 throughout as it has fewer terms, and we find it easier to interpret.

该显式衰减、式 10 与式 14 都给出相近结果，$R^{2}$ 约 80。式 14 对数据的拟合略差于式 10，可能因近似所致。尽管如此，我们全文使用式 14，因其项更少，也更易解释。

A.1 Analytical properties of compute-optimal point

Figure 7: A cartoon of how the compute-optimal tradeoff deviates from Chinchilla as we increase the number of epochs. Initially the model size and tokens processed grow proportionally ($R_{N}=R_{D}$) but since $R^{*}_{N}<R^{*}_{D}$, at some point adding parameters offers worse returns compared to increasing the number of tokens processed, and hence we deviate from the Chinchilla curve.

A.1 算力最优点的解析性质

图 7：随 epoch 增加，算力最优权衡如何偏离 Chinchilla 的示意。起初模型规模与处理 token 等比例增长（$R_{N}=R_{D}$）；但由于 $R^{*}_{N}<R^{*}_{D}$，到某点增加参数的回报劣于增加处理 token 数，因而偏离 Chinchilla 曲线。

In our case, consider the setting of a fixed compute budget $C$ and a fixed budget of unique tokens $U_{D}$ implying a set of unique parameters $U_{N}$. Let $R_{D}$ denote the number of times we repeat data (we assume that we are in the multi-epoch regime and hence $R_{D}>0$).

在我们的情形，考虑固定算力预算 $C$ 与固定唯一 token 预算 $U_{D}$（隐含一组唯一参数 $U_{N}$）。令 $R_{D}$ 表示重复数据的次数（假定已处于多 epoch 情形，故 $R_{D}>0$）。

Write $U_{D}=cU_{N}$ (for Chinchilla $c\approx 20$). When $R_{D}\ll R^{*}_{D}$ and $R_{N}\ll R^{*}_{N}$, our scaling agrees with Chinchilla, and so the point $(U_{N},U_{D})$, corresponding to $R_{D}=R_{N}=0$ is on the optimal compute curve. Increasing $R_{D}$ by $\epsilon$ corresponds to increasing the number of tokens by $\epsilon U_{D}=\epsilon cU_{N}$, while increasing $R_{N}$ by $\epsilon$ corresponds to increasing the number of parameters by $\epsilon U_{N}$. For small positive $R_{D},R_{N}$, our curve agrees with Chinchilla and so we need to increase $R_{N},R_{D}$ by the same amount to maintain the proportionality. Hence up to some value $r>0$, the optimal compute curve corresponds to $R_{N}=R_{D}=r$. Our curve differs from Chinchilla when $r$ gets closer to either $R^{*}_{N}$ or $R^{*}_{D}$. At this point, we start to see sharply diminishing returns.