Scaling Laws for Neural Language Models

神经语言模型的缩放定律（Scaling Laws for Neural Language Models）

Jared Kaplan † † thanks: Equal contribution. Contributions: Jared Kaplan and Sam McCandlish led the research. Tom Henighan contributed the LSTM experiments. Tom Brown, Rewon Child, and Scott Gray, and Alec Radford developed the optimized Transformer implementation. Jeff Wu, Benjamin Chess, and Alec Radford developed the text datasets. Dario Amodei provided guidance throughout the project. Affiliation: Johns Hopkins University, OpenAI Email: jaredk@jhu.edu    Sam McCandlish ∗ Affiliation: OpenAI Email: sam@openai.com    Tom Henighan Affiliation: OpenAI Email: henighan@openai.com    Tom B. Brown Affiliation: OpenAI Email: tom@openai.com    Benjamin Chess Affiliation: OpenAI Email: bchess@openai.com    Rewon Child Affiliation: OpenAI Email: rewon@openai.com    Scott Gray Affiliation: OpenAI Email: scott@openai.com    Alec Radford Affiliation: OpenAI Email: alec@openai.com    Jeffrey Wu Affiliation: OpenAI Email: jeffwu@openai.com    Dario Amodei Affiliation: OpenAI Email: damodei@openai.com

Jared Kaplan* Sam McCandlish* Tom Henighan Tom B. Brown Benjamin Chess Rewon Child Scott Gray Alec Radford Jeffrey Wu Dario Amodei
所属机构：Johns Hopkins University；OpenAI

• 同等贡献。Kaplan 与 McCandlish 主导研究；Henighan 贡献 LSTM 实验；Brown、Child、Gray、Radford 开发优化 Transformer 实现；Wu、Chess、Radford 开发文本数据集；Amodei 提供全程指导。

Abstract

摘要

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

我们研究语言模型在交叉熵损失上的经验缩放定律。损失随模型规模、数据集规模与训练所用算力呈幂律缩放，部分趋势跨越超过七个数量级。网络宽度或深度等其他架构细节在相当宽的范围内影响很小。简单方程刻画了过拟合对模型/数据规模的依赖，以及训练速度对模型规模的依赖。据此可确定固定算力预算的最优分配。更大的模型显著更样本高效，因此算力最优的训练方式是：在相对适中的数据量上训练非常大的模型，并在远未收敛时显著提前停止。

1 Introduction

1 引言

Contents

Language provides a natural domain for the study of artificial intelligence, as the vast majority of reasoning tasks can be efficiently expressed and evaluated in language, and the world’s text provides a wealth of data for unsupervised learning via generative modeling. Deep learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY+19, LOG+19, RSR+19] approaching human-level performance on many specific tasks [WPN+19], including the composition of coherent multi-paragraph prompted text samples [RWC+19].

从一开始就是《公约》缔约国。

语言为人工智能的研究提供了自然领域,因为绝大多数推理任务可以用语言有效表达和评价,世界文本通过生成模型学为无监督的学习提供了丰富的数据. 深度学习近来在语言建模方面迅速取得进展,最先进的艺术模型[RNSS18,DCLT18,YDY+19,LOG+19,RSR+19]在许多具体任务上接近人类层面的性能[WPN+19],包括连贯多段激发文本样本的构成[RWC+19].

One might expect language modeling performance to depend on model architecture, the size of neural models, the computing power used to train them, and the data available for this training process. In this work we will empirically investigate the dependence of language modeling loss on all of these factors, focusing on the Transformer architecture [VSP+17, LSP+18]. The high ceiling and low floor for performance on language tasks allows us to study trends over more than seven orders of magnitude in scale.

人们可能期望语言建模的性能取决于模型架构,神经模型的大小,用来训练这些模型的计算力,以及这个训练过程可用的数据. 在这项工作中,我们将实证地调查语言模型丢失对所有这些因素的依赖性,重点是Transformer架构[VSP+17,LSP+18]. 语文任务执行的上限高,下限低,使我们能够研究七个以上规模的趋势。

Throughout we will observe precise power-law scalings for performance as a function of training time, context length, dataset size, model size, and compute budget.

在整个过程中,我们将根据训练时间、背景长度、数据集大小、模型大小和预算计算情况,对业绩进行精确的电法缩放。

Figure 1: Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute[^1] used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.

图1:随着我们增加模型大小,数据集大小,以及用于培训的计算量[^1],语言模型的性能得到顺利改善. 为了取得最佳业绩,所有三个因素都必须同步扩大。 经验性表现在不受到其他两个因素的束缚时,与每个个体因素都有幂律关系.

1.1 Summary

1.1 总结

Our key findings for Transformer language models are are as follows:

我们对变形语模式的主要发现如下:

Performance depends strongly on scale, weakly on model shape:

Model performance depends most strongly on scale, which consists of three factors: the number of model parameters $N$ (excluding embeddings), the size of the dataset $D$, and the amount of compute $C$ used for training. Within reasonable limits, performance depends very weakly on other architectural hyperparameters such as depth vs. width. (Section 3)

性能主要取决于规模,弱于模型形状:

模型性能最依赖于尺度,由三个因素组成:模型参数$N$(不包括嵌入)的数量,数据集$D$的大小,以及用于培训的计算$C$的金额. 在合理的限度内,性能非常弱依赖于其他建筑超参数,如深度对宽度. (第3节)

Smooth power laws:

Performance has a power-law relationship with each of the three scale factors $N,D,C$ when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1). We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3)

平滑幂律:

性能与$N,D,C$的三个尺度因子在不受其他两个因子的束缚时,每个因子都具有幂律关系,其趋势超过六个规模(见图1)。 我们没有观察到在上端偏离这些趋势的迹象,尽管在达到零损失之前,业绩必须最终平息下来。 (第3节)

Universality of overfitting:

Performance improves predictably as long as we scale up $N$ and $D$ in tandem, but enters a regime of diminishing returns if either $N$ or $D$ is held fixed while the other increases. The performance penalty depends predictably on the ratio $N^{0.74}/D$, meaning that every time we increase the model size 8x, we only need to increase the data by roughly 5x to avoid a penalty. (Section 4)

• 过于适应的普遍性:

只要我们同步扩大$N$和$D$,性能就可以预测得到改善,但如果$N$或$D$被固定,而其他增加则进入递减回报的制度。 性能处罚可以预测取决于$N^{0.74}/D$的比例,这意味着每次增加模型尺寸8x,我们只需要增加大约5x的数据来避免惩罚. (第4节)

Universality of training:

Training curves follow predictable power-laws whose parameters are roughly independent of the model size. By extrapolating the early part of a training curve, we can roughly predict the loss that would be achieved if we trained for much longer. (Section 5)

培训的普遍性:

训练曲线遵循可预测的幂律,其参数大致独立于模型大小。 通过推断一个训练曲线的早期部分,我们可以大致预测如果我们训练更长的时间将会实现的损失. (第5节)

Transfer improves with test performance:

When we evaluate models on text with a different distribution than they were trained on, the results are strongly correlated to those on the training validation set with a roughly constant offset in the loss – in other words, transfer to a different distribution incurs a constant penalty but otherwise improves roughly in line with performance on the training set. (Section 3.2.2)

QQ 传输随着测试性能的改善:

当我们对文本促成了不同分布的模型进行评估时,结果与培训鉴定组的模型有着密切的联系,而且损失被大致固定地抵消了,换句话说,转移到不同分布的模型将不断受到惩罚,但除此之外,结果与培训组的绩效大致相符。 (第3.2.2节)

Sample efficiency:

Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps (Figure 2) and using fewer data points (Figure 4).

• 样本效率:

大型模型比小型模型更具有样本效率,以较少的优化步骤(图2)达到相同的性能水平,并使用较少的数据点(图4).

Figure 2: We show a series of language model training runs, with models ranging in size from $10^{3}$ to $10^{9}$ parameters (excluding embeddings).

图2:我们显示了一系列语言模型训练运行,模型大小从$10^{3}$到$10^{9}$参数(不包括嵌入)不等.

Figure 3: As more compute becomes available, we can choose how much to allocate towards training larger models, using larger batches, and training for more steps. We illustrate this for a billion-fold increase in compute. For optimally compute-efficient training, most of the increase should go towards increased model size. A relatively small increase in data is needed to avoid reuse. Of the increase in data, most can be used to increase parallelism through larger batch sizes, with only a very small increase in serial training time required.

图3 随着更多的计算,我们可以选择分配多少用于培训更大的模型,使用更大的批次,以及培训更多的步骤。 我们用计算数的十亿倍来说明这一点。 为了优化计算效率的培训,增加的多数应增加模式规模。 为了避免重复使用,需要较少增加数据。 在增加的数据中,大多数可以通过较大的批量大小来增加并行性,只需要极小的增加序列训练时间.

Convergence is inefficient:

When working within a fixed compute budget $C$ but without any other restrictions on the model size $N$ or available data $D$, we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3). Maximally compute-efficient training would therefore be far more sample efficient than one might expect based on training small models to convergence, with data requirements growing very slowly as $D\sim C^{0.27}$ with training compute. (Section 6)

校对:Soup

当在固定计算预算$C$范围内工作,但对模型大小$N$或现有数据$D$没有任何其他限制时,我们通过培训非常大的模型 和阻止* 严重缺乏趋同性* 来达到最佳性能(见图3)。 因此,最高效的计算培训的样本效率将远远高于基于培训小模式的预期,随着培训计算,数据要求增长非常缓慢。 (第6节)

Optimal batch size:

The ideal batch size for training these models is roughly a power of the loss only, and continues to be determinable by measuring the gradient noise scale [MKAT18]; it is roughly 1-2 million tokens at convergence for the largest models we can train. (Section 5.1)

最佳批量大小 :

欢迎各位来到美国, (第5.1节)

Taken together, these results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute. We expect that larger language models will perform better and be more sample efficient than current models.

综合起来,这些结果表明,随着我们适当扩大模型大小、数据和计算,语言模型的性能可以顺利和预测地改进。 我们预计,更大的语言模型将比目前的模型发挥更好和更有效率的样本。

1.2 Summary of Scaling Laws

1.2 缩放定律摘要

The test loss of a Transformer trained to autoregressively model language can be predicted using a power-law when performance is limited by only either the number of non-embedding parameters $N$, the dataset size $D$, or the optimally allocated compute budget $C_{\rm min}$ (see Figure 1):

当性能仅受非嵌入参数$N$,数据集大小$D$的限制,或者最优化分配的计算预算$C_{\rm min}$的限制时,可以使用幂律来预测测试器的丢失:

1. For models with a limited number of parameters, trained to convergence on sufficiently large datasets: $$L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\penalty\ \penalty\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\penalty\ \text{(non-embedding parameters)}$$ (1.1)
2. For large models trained with a limited dataset with early stopping: $$L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\penalty\ \penalty\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\penalty\ \text{(tokens)}$$ (1.2)
3. When training with a limited amount of compute, a sufficiently large dataset, an optimally-sized model, and a sufficiently small batch size (making optimal[^2] use of compute): $$L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\penalty\ \penalty\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\penalty\ \text{(PF-days)}$$ (1.3)

1. 对于参数数量有限的模型,经培训可在足够大的数据集上趋同:$$L(N)=\left(N_{\mathrm{c}}/N\right)^{\alpha_{N}};\penalty\ \penalty\ \alpha_{N}\sim 0.076,\quad N_{\mathrm{c}}\sim 8.8\times 10^{13}\penalty\ \text{(non-embedding parameters)}$$(1.1)
2. 受过有限数据集培训并提前停止的大型模型:$$L(D)=\left(D_{\mathrm{c}}/D\right)^{\alpha_{D}};\penalty\ \penalty\ \alpha_{D}\sim 0.095,\quad D_{\mathrm{c}}\sim 5.4\times 10^{13}\penalty\ \text{(tokens)}$$(1.2)
3. 当训练使用有限的计算量时,一个足够大的数据集,一个最优化的大小模型和一个足够小的批量大小(使计算量得到最佳[^2]的使用):$$L(C_{\rm min})=\left(C_{\mathrm{c}}^{\rm min}/C_{\rm min}\right)^{\alpha_{C}^{\rm min}};\penalty\ \penalty\ \alpha_{C}^{\rm min}\sim 0.050,\quad C_{\mathrm{c}}^{\rm min}\sim 3.1\times 10^{8}\penalty\ \text{(PF-days)}$$ (1.3)

These relations hold across eight orders of magnitude in $C_{\rm min}$, six orders of magnitude in $N$, and over two orders of magnitude in $D$. They depend very weakly on model shape and other Transformer hyperparameters (depth, width, number of self-attention heads), with specific numerical values associated with the Webtext2 training set [RWC+19]. The power laws $\alpha_{\rm N},\alpha_{\rm D},\alpha_{C}^{\rm min}$ specify the degree of performance improvement expected as we scale up $N$, $D$, or $C_{\rm min}$; for example, doubling the number of parameters yields a loss that is smaller by a factor $2^{-\alpha_{N}}=0.95$. The precise numerical values of $N_{\mathrm{c}},C_{\rm c}^{\rm min},$ and $D_{\mathrm{c}}$ depend on the vocabulary size and tokenization and hence do not have a fundamental meaning.

这些关系在$C_{\rm min}$有8个数量级,在$N$有6个数量级,在$D$有2个以上数量级。 它们非常微弱地依赖于模型形状和其他Transformer超参数(深度,宽度,自注意力头数),具体数值与Webtext2训练集[RWC+19]相关.$\alpha_{\rm N},\alpha_{\rm D},\alpha_{C}^{\rm min}$我就用电法规定,当我们扩大$N$、$D$或$C_{\rm min}$时,预期的性能改善程度;例如,将参数数翻一番,损失率比$2^{-\alpha_{N}}=0.95$小。 $N_{\mathrm{c}},C_{\rm c}^{\rm min},$和$D_{\mathrm{c}}$的精确数值取决于词汇大小和符号化,因此没有根本意义.

The critical batch size, which determines the speed/efficiency tradeoff for data parallelism ([MKAT18]), also roughly obeys a power law in $L$:

关键批量大小决定了数据并行性的速度/效率权衡([MKAT18]),也大致遵守$L$中的功率定律:

$$B_{\rm crit}\left(L\right)=\frac{B_{\ast}}{L^{1/\alpha_{B}}},\qquad B_{\ast}\sim 2\cdot 10^{8}\text{ tokens},\ \ \alpha_{B}\sim 0.21 \tag{1.4}$$

Figure 4: Left: The early-stopped test loss $L(N,D)$ varies predictably with the dataset size $D$ and model size $N$ according to Equation (1.5). Right: After an initial transient period, learning curves for all model sizes $N$ can be fit with Equation (1.6), which is parameterized in terms of $S_{\rm min}$, the number of steps when training at large batch size (details in Section 5.1).

图4:左: 早期停止的测试损失$L(N,D)$随着数据集大小$D$和模型大小$N$根据方程式(1.5)的预测而变化. ** 权利**: 经过最初的短暂时期后,所有型号的$N$的学习曲线可以与方程式(1.6)相适应,方程式以$S_{\rm min}$为参数,大批量规模的训练时的步骤数(详见第5.1节).

Equation (1.1) and (1.2) together suggest that as we increase the model size, we should increase the dataset size sublinearly according to $D\propto N^{\frac{\alpha_{N}}{\alpha_{D}}}\sim N^{0.74}$. In fact, we find that there is a single equation combining (1.1) and (1.2) that governs the simultaneous dependence on $N$ and $D$ and governs the degree of overfitting:

方程式(1.1)和(1.2)一起表明,随着模型规模的扩大,我们应当根据$D\propto N^{\frac{\alpha_{N}}{\alpha_{D}}}\sim N^{0.74}$线性地增加数据集大小。 事实上,我们发现有一个将1.1和1.2合并在一起的单一方程式,用以规范同时依赖$N$和$D$,并规范过拟合的程度:

$$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} \tag{1.5}$$

with fits pictured on the left in figure 4. We conjecture that this functional form may also parameterize the trained log-likelihood for other generative modeling tasks.

图4中左侧的图片。 我们推测,这种功能形式也可以将训练过的日志相似性参数化,用于其他生成模型制作任务.

When training a given model for a finite number of parameter update steps $S$ in the infinite data limit, after an initial transient period, the learning curves can be accurately fit by (see the right of figure 4)

当在无限数据限值中为有限数量的参数更新步骤$S$培训特定模型时,在初始瞬间后,学习曲线可以通过(见图4的权利)准确匹配.

$$L(N,S)=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(\frac{S_{c}}{S_{\rm min}(S)}\right)^{\alpha_{S}} \tag{1.6}$$

where $S_{c}\approx 2.1\times 10^{3}$ and $\alpha_{S}\approx 0.76$, and $S_{\rm min}(S)$ is the minimum possible number of optimization steps (parameter updates) estimated using Equation (5.4).

其中$S_{c}\approx 2.1\times 10^{3}$和$\alpha_{S}\approx 0.76$,和$S_{\rm min}(S)$是使用方程式估算的最小可能的优化步骤(参数更新)数(5.4).

When training within a fixed compute budget $C$, but with no other constraints, Equation (1.6) leads to the prediction that the optimal model size $N$, optimal batch size $B$, optimal number of steps $S$, and dataset size $D$ should grow as

当在固定计算预算$C$范围内进行训练但没有其他限制时,方程式(1.6)导致预测最佳模型大小$N$,最佳批量大小$B$,最佳步骤数量$S$,以及数据集大小$D$应该随着增长而增长.

$$N\propto C^{\alpha_{C}^{\rm min}/\alpha_{N}},\quad B\propto C^{\alpha_{C}^{\rm min}/\alpha_{B}},\quad S\propto C^{\alpha_{C}^{\rm min}/\alpha_{S}},\quad D=B\cdot S\quad \tag{1.7}$$

with

与

$$\alpha_{C}^{\rm min}=1/\left(1/\alpha_{S}+1/\alpha_{B}+1/\alpha_{N}\right) \tag{1.8}$$

which closely matches the empirically optimal results $N\propto C_{\rm min}^{0.73}$, $B\propto C_{\rm min}^{0.24}$, and $S\propto C_{\rm min}^{0.03}$. As the computational budget $C$ increases, it should be spent primarily on larger models, without dramatic increases in training time or dataset size (see Figure 3). This also implies that as models grow larger, they become increasingly sample efficient. In practice, researchers typically train smaller models for longer than would be maximally compute-efficient because of hardware constraints. Optimal performance depends on total compute as a power law (see Equation (1.3)).

(b) 与经验最佳结果$N\propto C_{\rm min}^{0.73}$、$B\propto C_{\rm min}^{0.24}$和$S\propto C_{\rm min}^{0.03}$密切匹配。 随着计算预算$C$的增加,它应主要用于更大的模型,而不大幅度增加训练时间或数据集的规模(见图3)。 这还意味着,随着模型的扩大,它们越来越有效率。 在实践中,研究人员通常对较小的模型进行培训的时间长于由于硬件限制而产生最大计算效率的水平。 最佳性能取决于作为动力定律的总计算(见Equation (1.3)).

We provide some basic theoretical motivation for Equation (1.5), an analysis of learning curve fits and their implications for training time, and a breakdown of our results per token. We also make some brief comparisons to LSTMs and recurrent Transformers [DGV+18].

我们为方程式学提供了一些基本的理论动力(1.5)求职,对学习曲线及其对训练时间持久的和平进行分析,并对我们的成果逐个进行细分。 我们还与LSTMs和反复出现的变换器[DGV+18]进行一些简短的比较.

1.3 Notation

1.3 记号

We use the following notation:

我们使用下列标记:

• • $L$ – the cross entropy loss in nats. Typically it will be averaged over the tokens in a context, but in some cases we report the loss for specific tokens within the context.
• • $N$ – the number of model parameters, excluding all vocabulary and positional embeddings
• • $C\approx 6NBS$ – an estimate of the total non-embedding training compute, where $B$ is the batch size, and $S$ is the number of training steps (ie parameter updates). We quote numerical values in PF-days, where one PF-day $=10^{15}\times 24\times 3600=8.64\times 10^{19}$ floating point operations.
• • $D$ – the dataset size in tokens
• • $B_{\rm crit}$ – the critical batch size [MKAT18], defined and discussed in Section 5.1. Training at the critical batch size provides a roughly optimal compromise between time and compute efficiency.
• • $C_{\rm min}$ – an estimate of the minimum amount of non-embedding compute to reach a given value of the loss. This is the training compute that would be used if the model were trained at a batch size much less than the critical batch size.
• • $S_{\rm min}$ – an estimate of the minimal number of training steps needed to reach a given value of the loss. This is also the number of training steps that would be used if the model were trained at a batch size much greater than the critical batch size.
• • $\alpha_{X}$ – power-law exponents for the scaling of the loss as $L(X)\propto 1/X^{\alpha_{X}}$ where $X$ can be any of $N,D,C,S,B,C^{\rm min}$.

-$L$– Nats的交叉 en损. 通常情况下,在上下文中,它将平均于物证,但在某些情况下,我们报告在上下文中的具体物证的损失。
-$N$- 模型参数的数目,* 不包括所有词汇和位置嵌入*

• $C\approx 6NBS$——对非嵌入式训练总计算的估计,其中$B$为批量规模,$S$为训练步骤的数量(i参数更新). 我们引用PF日的数值,其中一个PF日的$=10^{15}\times 24\times 3600=8.64\times 10^{19}$浮点操作.
  -$D$-- -- 代号数据集大小
• $B_{\rm crit}$——第5.1节界定和讨论的关键批量尺寸[MKAT18]。 关键批量规模的培训在时间和计算效率之间提供了大致最佳的妥协。
  -$C_{\rm min}$-- -- 估计非嵌入计算得出某一损失价值的最低数额。 如果该型号的批号比批号要小得多,则使用这种训练计算法。
  -$S_{\rm min}$-- --啪声 -- 估计达到一定损失价值所需的最低限度训练步骤。 这也是如果该型号的分批培训比关键分批培训大得多,将使用的培训步骤的数目。
  -$\alpha_{X}$– 作为$L(X)\propto 1/X^{\alpha_{X}}$将损失缩放为$X$的任何一种的幂律促进者.

2 Background and Methods

2 背景与方法

We train language models on WebText2, an extended version of the WebText [RWC+19] dataset, tokenized using byte-pair encoding [SHB15] with a vocabulary size $n_{\rm vocab}=50257$. We optimize the autoregressive log-likelihood (i.e. cross-entropy loss) averaged over a 1024-token context, which is also our principal performance metric. We record the loss on the WebText2 test distribution and on a selection of other text distributions. We primarily train decoder-only [LSP+18, RNSS18] Transformer [VSP+17] models, though we also train LSTM models and Universal Transformers [DGV+18] for comparison.

我们在WebText2上培训语言模型,这是WebText[RWC+19]数据集的扩展版本,以字节-pair编码[SHB15]为标志,词汇大小为$n_{\rm vocab}=50257$. 我们优化了自递式的类似日志(即交叉输入损失),平均在1024个上下文上,这也是我们的主要业绩衡量标准。 我们在WebText2测试分发和其他文本分发选集中记录损失. 我们主要只培训解码器[LSP+18,RNSS18]变压器[VSP+17]型号,虽然我们也培训LSTM型号和通用变压器[DGV+18]进行对比.

2.1 Parameter and Compute Scaling of Transformers

2.1 Transformer 的参数与算力缩放

Operation	Parameters	FLOPs per Token
Embed	$\left(n_{{\rm vocab}}+n_{{\rm ctx}}\right)d_{{\rm model}}$	$4d_{{\rm model}}$
Attention: QKV	$n_{{\rm layer}}d_{{\rm model}}3d_{{\rm attn}}$	$2n_{{\rm layer}}d_{{\rm model}}3d_{{\rm attn}}$
Attention: Mask	—	$2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm attn}}$
Attention: Project	$n_{{\rm layer}}d_{{\rm attn}}d_{{\rm model}}$	$2n_{{\rm layer}}d_{{\rm attn}}d_{{\rm embd}}$
Feedforward	$n_{{\rm layer}}2d_{{\rm model}}d_{{\rm ff}}$	$2n_{{\rm layer}}2d_{{\rm model}}d_{{\rm ff}}$
De-embed	—	$2d_{{\rm model}}n_{{\rm vocab}}$
Total (Non-Embedding)	$N=2d_{{\rm model}}n_{{\rm layer}}\left(2d_{{\rm attn}}+d_{{\rm ff}}\right)$	$C_{\mathrm{forward}}=2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm attn}}$

Table 1: Parameter counts and compute (forward pass) estimates for a Transformer model. Sub-leading terms such as nonlinearities, biases, and layer normalization are omitted.

** 行动**	** 参数**	** 按托肯计算的供资比率**
内嵌	第二十届	$4d_{{\rm model}}$
注意:QKV	第二十届	第二十届
注意:面具	. . . . . .	第二十届
注意:项目	互信	互信
向导	第二十届	第二十届
卸载	. . . . . .	第二十届
** 总计(不计费用)**	第二十届	第二十届

表1:变压器模型参数计算和计算(前传)估计数。 省略了非线性、偏差和层层正常化等次要术语。

We parameterize the Transformer architecture using hyperparameters $n_{\rm layer}$ (number of layers), $d_{{\rm model}}$ (dimension of the residual stream), $d_{\rm ff}$ (dimension of the intermediate feed-forward layer), $d_{\rm attn}$ (dimension of the attention output), and $n_{\rm heads}$ (number of attention heads per layer). We include $n_{\rm ctx}$ tokens in the input context, with $n_{\rm ctx}=1024$ except where otherwise noted.

我们使用超采取协调一致的参数来参数化Transformer架构 $n_{\rm layer}$(层数), $d_{{\rm model}}$(残流的分数), $d_{\rm ff}$(中间向导层的分数), $d_{\rm attn}$(注意力输出的分数),和 $n_{\rm heads}$(每层注意力头数). 我们在输入语境中包括$n_{\rm ctx}$令牌,除另有说明外,还有$n_{\rm ctx}=1024$.

We use $N$ to denote the model size, which we define as the number of non-embedding parameters

我们使用$N$来表示模型大小,我们将其定义为既没有嵌入参数的数量

$$\displaystyle N \\ \displaystyle\approx 2d_{{\rm model}}n_{{\rm layer}}\left(2d_{{\rm attn}}+d_{{\rm ff}}\right) \\ \displaystyle=12n_{\rm layer}d_{{\rm model}}^{2}\quad\text{ with the standard }\quad d_{\rm attn}=d_{\rm ff}/4=d_{{\rm model}} \tag{2.1}$$

where we have excluded biases and other sub-leading terms. Our models also have $n_{\rm vocab}d_{{\rm model}}$ parameters in an embedding matrix, and use $n_{\rm ctx}d_{{\rm model}}$ parameters for positional embeddings, but we do not include these when discussing the ‘model size’ $N$; we will see that this produces significantly cleaner scaling laws.

我们排除了偏见和其他次要条件。 我们的模型在嵌入式矩阵中也有$n_{\rm vocab}d_{{\rm model}}$参数,并使用$n_{\rm ctx}d_{{\rm model}}$参数进行位置嵌入,但我们在讨论`模型大小 ' $N$时并不包括这些参数;我们将看到这会产生显著的更清洁的缩放法则.

Evaluating a forward pass of the Transformer involves roughly

评估Transformer的前传涉及大致

$$C_{\rm forward}\approx 2N+2n_{{\rm layer}}n_{{\rm ctx}}d_{{\rm model}} \tag{2.2}$$

add-multiply operations, where the factor of two comes from the multiply-accumulate operation used in matrix multiplication. A more detailed per-operation parameter and compute count is included in Table 1.

加乘运算,其中2的系数来自矩阵乘法中使用的乘积运算. 表1列有更详细的每个操作参数和计算数。

For contexts and models with $d_{{\rm model}}>n_{\rm ctx}/12$, the context-dependent computational cost per token is a relatively small fraction of the total compute. Since we primarily study models where $d_{\rm model}\gg n_{\rm ctx}/12$, we do not include context-dependent terms in our training compute estimate. Accounting for the backwards pass (approximately twice the compute as the forwards pass), we then define the estimated non-embedding compute as $C\approx 6N$ floating point operators per training token.

对于具有$d_{{\rm model}}>n_{\rm ctx}/12$的上下文和模型,每令牌依赖上下文的计算成本是计算总成本的相对一小部分. 由于我们主要研究$d_{\rm model}\gg n_{\rm ctx}/12$的模型,因此我们的培训计算估计数中并不包括取决于具体情况的术语。 计算后传(大约为前传的两倍计算),然后将估计的非嵌入式计算定义为每个训练楷书的$C\approx 6N$浮点运算符.

2.2 Training Procedures

2.2 训练流程

Unless otherwise noted, we train models with the Adam optimizer [KB14] for a fixed $2.5\times 10^{5}$ steps with a batch size of $512$ sequences of $1024$ tokens. Due to memory constraints, our largest models (more than 1B parameters) were trained with Adafactor [SS18]. We experimented with a variety of learning rates and schedules, as discussed in Appendix D.6. We found that results at convergence were largely independent of learning rate schedule. Unless otherwise noted, all training runs included in our data used a learning rate schedule with a 3000 step linear warmup followed by a cosine decay to zero.

除非另有说明,否则 这里是使用亚当优化器[KB14]为固定的$2.5\times 10^{5}$步骤训练模型,其批量大小为$512$序列为$1024$令牌. 由于内存的限制,我们最大的模型(超过1B参数)在Aductor[SS18]训练. 如附录D.6所述,我们试验了各种学习率和学习时间表。 我们发现,趋同的结果在很大程度上独立于学习率时间表。 除非另有说明,我们数据中包括的所有培训都采用了学习速度表,采用3000步线性热速率,然后将余弦衰减到零。

2.3 Datasets

2.3 数据集

We train our models on an extended version of the WebText dataset described in [RWC+19]. The original WebText dataset was a web scrape of outbound links from Reddit through December 2017 which received at least 3 karma. In the second version, WebText2, we added outbound Reddit links from the period of January to October 2018, also with a minimum of 3 karma. The karma threshold served as a heuristic for whether people found the link interesting or useful. The text of the new links was extracted with the Newspaper3k python library. In total, the dataset consists of 20.3M documents containing 96 GB of text and $1.62\times 10^{10}$ words (as defined by wc). We then apply the reversible tokenizer described in [RWC+19], which yields $2.29\times 10^{10}$ tokens. We reserve $6.6\times 10^{8}$ of these tokens for use as a test set, and we also test on similarly-prepared samples of Books Corpus [ZKZ+15], Common Crawl [Fou], English Wikipedia, and a collection of publicly-available Internet Books.

我们用[RWC+19]中描述的WebText数据集的扩展版本来训练我们的模型. 最初的WebText数据集是一个从Reddit到2017年12月的输出链接的网络刮痕,它至少收到了3个因果. 在第二个版本WebText2中,我们增加了从2018年1月至10月的外出Reddit链接,同样最少3个因果. 因果门槛对于人们是否觉得联系有趣或有用,起到了推力作用. 与Newspaper3k python库一起提取了新链接的文本. 数据集总共包括包含96GB文本的20.3M文档和$1.62\times 10^{10}$词(由wc定义). 然后,我们使用[RWC+19]中描述的可逆的代币,该代币产生$2.29\times 10^{10}$. 我们保留其中的$6.6\times 10^{8}$作为测试集使用,我们还测试了类似准备的Books Corpus [ZKZ+15],Common Crawl [Fou],英语维基百科,以及可公开查阅的互联网图书集.

3 Empirical Results and Basic Power Laws

3 经验结果与基本幂律

To characterize language model scaling we train a wide variety of models, varying a number of factors including:

为了描述语言模型的规模,我们培训了各种各样的模型,其中有若干因素,包括:

• • Model size (ranging in size from 768 to 1.5 billion non-embedding parameters)
• • Dataset size (ranging from 22 million to 23 billion tokens)
• • Shape (including depth, width, attention heads, and feed-forward dimension)
• • Context length (1024 for most runs, though we also experiment with shorter contexts)
• • Batch size ($2^{19}$ for most runs, but we also vary it to measure the critical batch size)

• 模型尺寸(大小从768亿到15亿非嵌入参数不等)
• 数据集大小(从2 200万到230亿令牌不等)
• 形状(包括深度、宽度、注意力头和向导维度)

• 背景长度(大多数时间为1024次,尽管我们也试验时间较短)
• 批量体积(大多数批量体积为$2^{19}$,但我们也为了测量批量体积而改变)

In this section we will display data along with empirically-motivated fits, deferring theoretical analysis to later sections.

在本节中,我们将展示数据以及以经验为动力的适合情况,将理论分析推迟到以后各节。

3.1 Approximate Transformer Shape and Hyperparameter Independence

3.1 近似 Transformer 形状与超参数无关性

Figure 5: Performance depends very mildly on model shape when the total number of non-embedding parameters $N$ is held fixed. The loss varies only a few percent over a wide range of shapes. Small differences in parameter counts are compensated for by using the fit to $L(N)$ as a baseline. Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an $(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288)$ reaches a loss within 3% of the $(48,1600)$ model used in [RWC+19].

图5:当$N$的非嵌入参数总数被固定时,性能非常温和地依赖于模型形状. 损失在各种各样的形状上只有很小的差别。 参数计数的微小差异通过将适值与$L(N)$作为基线予以补偿。 光谱比尤其可以因40而异,而只略微影响性能;$(n_{\mathrm{layer}},d_{\mathrm{model}})=(6,4288)$在[RWC+19]中使用的$(48,1600)$模型的3%范围内达到损失.

Transformer performance depends very weakly on the shape parameters $n_{\rm layer},n_{\rm heads}$, and $d_{\rm ff}$ when we hold the total non-embedding parameter count $N$ fixed. To establish these results we trained models with fixed size while varying a single hyperparameter. This was simplest for the case of $n_{\rm heads}$. When varying $n_{\rm layer}$, we simultaneously varied $d_{{\rm model}}$ while keeping $N\approx 12n_{\rm layer}d_{{\rm model}}^{2}$ fixed. Similarly, to vary $d_{\rm ff}$ at fixed model size we also simultaneously varied the $d_{{\rm model}}$ parameter, as required by the parameter counts in Table 1. Independence of $n_{\rm layers}$ would follow if deeper Transformers effectively behave as ensembles of shallower models, as has been suggested for ResNets [VWB16]. The results are shown in Figure 5.

Transformer的性能非常微弱地依赖于形状参数$n_{\rm layer},n_{\rm heads}$,当我们持有总的非嵌入参数计数$N$固定时的$d_{\rm ff}$. 为了确定这些结果,我们培训了固定尺寸的模型,同时采用了一个超参数。 这对$n_{\rm heads}$一案最为简单。 当改变$n_{\rm layer}$时,我们同时改变$d_{{\rm model}}$,同时保持$N\approx 12n_{\rm layer}d_{{\rm model}}^{2}$的固定. 同样,为了在固定模型大小上改变$d_{\rm ff}$,我们也同时按照表1参数计数的要求改变$d_{{\rm model}}$参数. 如果更深的变换器能像ResNets[VWB16]所建议的那样,有效地发挥更浅的模型的组合作用,则$n_{\rm layers}$的独立将随之而来。 结果见图5。

3.2 Performance with Non-Embedding Parameter Count $N$

3.2 相对非嵌入参数量 $N$ 的性能

Figure 6: Left: When we include embedding parameters, performance appears to depend strongly on the number of layers in addition to the number of parameters. Right: When we exclude embedding parameters, the performance of models with different depths converge to a single trend. Only models with fewer than 2 layers or with extreme depth-to-width ratios deviate significantly from the trend.

图6:左: 当我们列入嵌入参数时,性能似乎除了参数数量外,还严重依赖于层数. ** 权利**: 当我们排除嵌入参数时,不同深度的模型的性能会趋同于单一的趋势. 只有少于2层或极深比比宽的模型,才明显偏离趋势.

In Figure 6 we display the performance of a wide variety of models, ranging from small models with shape $(n_{\rm layer},d_{{\rm model}})=(2,128)$ through billion-parameter models, ranging in shape from $(6,4288)$ through $(207,768)$. Here we have trained to near convergence on the full WebText2 dataset and observe no overfitting (except possibly for the very largest models).

在图6中,我们展示了各种模型的性能,从形状为$(n_{\rm layer},d_{{\rm model}})=(2,128)$的小模型到10亿参数模型,形状从$(6,4288)$到$(207,768)$. 在这里,我们训练了在完整的WebText2数据集上接近趋同,并且观测到没有过拟合(除了可能是最大的模型).

As shown in Figure 1, we find a steady trend with non-embedding parameter count $N$, which can be fit to the first term of Equation (1.5), so that

如图1所示,我们发现一个稳定的趋势,非嵌入参数计数为$N$,这可以适应方程式的第一个术语(1.5),因此,

$$L(N)\approx\left(\frac{N_{c}}{N}\right)^{\alpha_{N}} \tag{3.1}$$

To observe these trends it is crucial to study performance as a function of $N$; if we instead use the total parameter count (including the embedding parameters) the trend is somewhat obscured (see Figure 6). This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [LCG+19].

要观察这些趋势,必须研究作为$N$函数的性能;如果我们使用总参数计数(包括嵌入参数),趋势就有些模糊(见图6)。 这表明嵌入式矩阵可以变小而不影响性能,这在近期的作品[LCG+19]中已经看到.

Although these models have been trained on the WebText2 dataset, their test loss on a variety of other datasets is also a power-law in $N$ with nearly identical power, as shown in Figure 8.

虽然这些模型在WebText2数据集上接受了培训,但是它们在各种其他数据集上的测试丢失也是$N$中功率几乎相同的功率定律,如图8所示.

3.2.1 Comparing to LSTMs and Universal Transformers

Figure 7:

3.2.1 与 LSTM 和通用Transformer进行比较

图7:

In Figure 7 we compare LSTM and Transformer performance as a function of non-embedding parameter count $N$. The LSTMs were trained with the same dataset and context length. We see from these figures that the LSTMs perform as well as Transformers for tokens appearing early in the context, but cannot match the Transformer performance for later tokens. We present power-law relationships between performance and context position Appendix D.5, where increasingly large powers for larger models suggest improved ability to quickly recognize patterns.

在图7中,我们比较了LSTM和Transformer的性能,作为非嵌入参数计算$N$的一个函数. LSTMs接受了同样的数据集和上下文长度的培训. 我们从这些数字中看到,LSTM和Transformer在上下文中早期出现的代号上都表现出来,但不能与Transformer的性能相匹配来表示后来的代号. 我们提出业绩与背景立场之间的幂律关系附录D.5,其中对较大模型的日益强大的权力表明,快速识别模式的能力得到了提高。

We also compare the performance of standard Transformers to recurrent Transformers [DGV+18] in Figure 17 in the appendix. These models re-use parameters, and so perform slightly better as a function of $N$, at the cost of additional compute per-parameter.

我们还将标准变换器的性能与附录图17中的反复变换器[DGV+18]进行比较.的案例重新使用参数,因此作为$N$的函数性能稍好一些,成本为每参数的额外计算.

3.2.2 Generalization Among Data Distributions

We have also tested our models on a set of additional text data distributions. The test loss on these datasets as a function of model size is shown in Figure 8; in all cases the models were trained only on the WebText2 dataset. We see that the loss on these other data distributions improves smoothly with model size, in direct parallel with the improvement on WebText2. We find that generalization depends almost exclusively on the in-distribution validation loss, and does not depend on the duration of training or proximity to convergence. We also observe no dependence on model depth (see Appendix D.8).

3.2.2 数据分配的概况

我们还测试了一套额外的文本数据分布模型。 这些数据集作为 58. 模型大小函数的测试损失见图8;在所有情况下,模型都是在WebText2数据集中培训的。 我们看到这些其它数据分布的损失随着模型大小的改善而顺利改善,与WebText2上的改进直接平行. 我们认为,一般化几乎完全取决于分配中的验证损失,并不取决于培训的持续时间或接近趋同的程度。 我们还观察到没有依赖模型深度(见附录D.8)。

Figure 8: Left: Generalization performance to other data distributions improves smoothly with model size, with only a small and very slowly growing offset from the WebText2 training distribution. Right: Generalization performance depends only on training distribution performance, and not on the phase of training. We compare generalization of converged models (points) to that of a single large model (dashed curves) as it trains.

图8:左: 通向其他数据分布的通用性能随着模型大小的改善而顺利改善,只有很小且非常缓慢增长的WebText2训练分布抵消. ** 权利**: 一般化的表现只取决于培训分配的表现,而不取决于培训阶段。 我们把趋同模型(点)的概括性与训练时的单一大模型(凹陷曲线)的概括性进行比较。

3.3 Performance with Dataset Size and Compute

3.3 相对数据集规模与算力的性能

We display empirical trends for the test loss as a function of dataset size $D$ (in tokens) and training compute $C$ in Figure 1.

我们显示测试损失的经验趋势,作为数据集大小$D$(以符号表示)和训练计算$C$在图1中的函数.

For the trend with $D$ we trained a model with $(n_{\rm layer},n_{\rm embd})=(36,1280)$ on fixed subsets of the WebText2 dataset. We stopped training once the test loss ceased to decrease. We see that the resulting test losses can be fit with simple power-law

关于$D$的趋势,我们在WebText2数据集的固定子集上用$(n_{\rm layer},n_{\rm embd})=(36,1280)$培训了一个模型。 一旦测试损失不再减少,我们就停止了训练。 我们发现 由此造成的测试损失 能够适应简单的电法

$$L(D)\approx\left(\frac{D_{c}}{D}\right)^{\alpha_{D}} \tag{3.2}$$

in the dataset size. The data and fit appear in Figure 1.

在数据集大小中。 数据和适用情况见图1。

The total amount of non-embedding compute used during training can be estimated as $C=6NBS$, where $B$ is the batch size, $S$ is the number of parameter updates, and the factor of $6$ accounts for the forward and backward passes. Thus for a given value of $C$ we can scan over all models with various $N$ to find the model with the best performance on step $S=\frac{C}{6BS}$. Note that in these results the batch size $B$ remains fixed for all models, which means that these empirical results are not truly optimal. We will account for this in later sections using an adjusted $C_{\rm min}$ to produce cleaner trends.

培训期间使用的非依循计算总数可估算为$C=6NBS$,其中$B$为批量大小,$S$为参数更新数,$6$因数为前向和后向通行证. 因此,对于给定的$C$值,我们可以用各种$N$扫描所有模型,找到在$S=\frac{C}{6BS}$步骤上表现最好的模型. 请注意,在这些结果中,* 批量大小$B$ 对所有模型仍然固定*,这意味着这些经验性结果并非真正最佳. 我们将在以后各节中说明这一点,使用经调整的$C_{\rm min}$来产生更清洁的趋势。

The result appears as the heavy black line on the left-hand plot in Figure 1. It can be fit with

结果以图1中左手图中的重黑线出现. 它可以适应

$$L(C)\approx\left(\frac{C_{c}}{C}\right)^{\alpha_{C}} \tag{3.3}$$

The figure also includes images of individual learning curves to clarify when individual models are optimal. We will study the optimal allocation of compute more closely later on. The data strongly suggests that sample efficiency improves with model size, and we also illustrate this directly in Figure 19 in the appendix.

该图还包括个人学习曲线的图像,以澄清单个模型何时是最佳. 我们稍后将更仔细地研究计算的最佳分配。 数据强烈表明,样本效率随着模型大小的提高而提高,我们在附录图19中也直接说明了这一点.

4 Charting the Infinite Data Limit and Overfitting

4 刻画无限数据极限与过拟合

In Section 3 we found a number of basic scaling laws for language modeling performance. Here we will study the performance of a model of size $N$ trained on a dataset with $D$ tokens while varying $N$ and $D$ simultaneously. We will empirically demonstrate that the optimally trained test loss accords with the scaling law of Equation (1.5). This provides guidance on how much data we would need to train models of increasing size while keeping overfitting under control.

在第3节中,我们发现了一些关于语言模型表现的基本缩放法。 在此,我们将研究一个大小为$N$的模型的性能,该模型在使用$D$令牌的数据集上培训,同时不同大小为$N$和$D$。 我们将实证地证明,经过最佳训练的测试损失符合方程式定律(1.5)。 这提供了指导,说明我们需要多少数据来训练规模越来越大的模型,同时保持过度调整的控制。

4.1 Proposed $L(N,D)$ Equation

4.1 提议的 $L(N,D)$ 方程

Figure 9: The early-stopped test loss $L(N,D)$ depends predictably on the dataset size $D$ and model size $N$ according to Equation (1.5). Left: For large $D$, performance is a straight power law in $N$. For a smaller fixed $D$, performance stops improving as $N$ increases and the model begins to overfit. (The reverse is also true, see Figure 4.) Right: The extent of overfitting depends predominantly on the ratio $N^{\frac{\alpha_{N}}{\alpha_{D}}}/D$, as predicted in equation (4.3). The line is our fit to that equation.

图9:早期停止的测试损失卷宗$L(N,D)$ắn根据方程式(1.5)预测取决于数据集大小$D$和模型大小$N$. ** 左侧: 对于大型的$D$,性能是$N$中的直电法. 对于一个较小的固定的$D$,随着$N$的增加,性能停止改善,模型开始过拟合. (反之亦然,见图4) ** 权利**: 过于适应的程度主要取决于$N^{\frac{\alpha_{N}}{\alpha_{D}}}/D$的比例,如方程式预测(4.3). 这条线是我们适合这个方程式的。

We have chosen the parameterization (1.5) (repeated here for convenience):

1929. 我们选择了参数化(1.5)(为方便起见在此重复):

$$L(N,D)=\left[\left(\frac{N_{c}}{N}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}+\frac{D_{c}}{D}\right]^{\alpha_{D}} \tag{4.1}$$

using three principles:

采用三项原则:

1. Changes in vocabulary size or tokenization are expected to rescale the loss by an overall factor. The parameterization of $L(N,D)$ (and all models of the loss) must naturally allow for such a rescaling.
2. Fixing $D$ and sending $N\to\infty$, the overall loss should approach $L(D)$. Conversely, fixing $N$ and sending $D\to\infty$ the loss must approach $L(N)$.
3. $L(N,D)$ should be analytic at $D=\infty$, so that it has a series expansion in $1/D$ with integer powers. Theoretical support for this principle is significantly weaker than for the first two.

1. 词汇大小或符号化的变化预计将按总体因素调整损失规模。 $L(N,D)$的参数化(以及损失的所有模型)必须自然允许这种缩放.
2. 修复$D$和发送$N\to\infty$,总损失应接近$L(D)$。 相反,确定$N$和发送$D\to\infty$的损失必须接近$L(N)$。
3. $L(N,D)$应在$D=\infty$进行分析,使其在$1/D$具有整数功率的系列扩展. 对这一原则的理论支持明显弱于前两项.

Our choice of $L(N,D)$ satisfies the first requirement because we can rescale $N_{c},D_{c}$ with changes in the vocabulary. This also implies that the values of $N_{c},D_{c}$ have no fundamental meaning.

我们对$L(N,D)$的选择满足了第一个要求,因为随着词汇的变化,我们可以对$N_{c},D_{c}$进行调整. 这也意味着$N_{c},D_{c}$的价值没有根本意义.

Since we stop training early when the test loss ceases to improve and optimize all models in the same way, we expect that larger models should always perform better than smaller models. But with fixed finite $D$, we also do not expect any model to be capable of approaching the best possible loss (ie the entropy of text). Similarly, a model with fixed size will be capacity-limited. These considerations motivate our second principle. Note that knowledge of $L(N)$ at infinite $D$ and $L(D)$ at infinite $N$ fully determines all the parameters in $L(N,D)$.

由于在测试损失停止以同样的方式改进和优化所有模型时,我们很早就停止了训练,我们期望更大的模型总是比较小的模型表现更好. 但是,有了固定的有限$D$,我们也并不期望任何模型能够接近尽可能最好的损失(即文字的 en). 同样,固定规模的模型也将是能力有限的。 这些考虑推动了我们的第二项原则。 注意对$L(N)$在无限$D$和对$L(D)$在无限$N$的知识完全决定了$L(N,D)$中的所有参数.

The third principle is more speculative. There is a simple and general reason one might expect overfitting to scale $\propto 1/D$ at very large $D$. Overfitting should be related to the variance or the signal-to-noise ratio of the dataset [AS17], and this scales as $1/D$. This expectation should hold for any smooth loss function, since we expect to be able to expand the loss about the $D\to\infty$ limit. However, this argument assumes that $1/D$ corrections dominate over other sources of variance, such as the finite batch size and other limits on the efficacy of optimization. Without empirical confirmation, we would not be very confident of its applicability.

第三项原则是更投机。 一个简单和普遍的理由,是人们可能期望在非常大范围的$D$中超标为$\propto 1/D$。 过于适应应当与数据集[AS17]的差值或信号与噪声之比有关,而这个尺度作为$1/D$. 这种期望应维持任何顺畅的损失功能,因为我们期望能够扩大有关$D\to\infty$限制的损失。 然而,这一论点假定,$1/D$更正在其它差异源之上占主导地位,例如有限批量大小和对优化效果的其他限制。 如果没有经验的证实,我们不会非常相信它的适用性。

Our third principle explains the asymmetry between the roles of $N$ and $D$ in Equation (1.5). Very similar symmetric expressions[^3] are possible, but they would not have a $1/D$ expansion with integer powers, and would require the introduction of an additional parameter.

我们的第三项原则解释了$N$和$D$在方程式(1.5)中的角色不对称. 非常相似的对称表达式[^3]是可能的,但它们不会有一个具有整数功率的$1/D$扩展,需要引入额外的参数.

In any case, we will see that our equation for $L(N,D)$ fits the data well, which is the most important justification for our $L(N,D)$ ansatz.

无论如何,我们将看到,我们的$L(N,D)$等式非常符合数据,这是我们$L(N,D)$ ansatz最重要的理由。

4.2 Results

4.2 结果

We regularize all our models with 10% dropout, and by tracking test loss and stopping once it is no longer decreasing. The results are displayed in Figure 9, including a fit to the four parameters $\alpha_{N},\alpha_{D},N_{c},D_{c}$ in Equation (1.5):

我们把所有的模型规范化 10%的辍学, 通过跟踪测试损失, 停止一旦它不再下降。 图9显示这些结果,包括与方程式(1.5)中的$\alpha_{N},\alpha_{D},N_{c},D_{c}$四个参数相符的结果:

Parameter	$\alpha_{N}$	$\alpha_{D}$	$N_{c}$	$D_{c}$
Value	$0.076$	$0.103$	$6.4\times 10^{13}$	$1.8\times 10^{13}$

Table 2: Fits to $L(N,D)$

参数	$\alpha_{N}$	$\alpha_{D}$	$N_{c}$	$D_{c}$
数值	$0.076$	$0.103$	$6.4\times 10^{13}$	$1.8\times 10^{13}$

表 2: 适合第$L(N,D)$号决定

We obtain an excellent fit, with the exception of the runs where the dataset has been reduced by a factor of $1024$, to about $2\times 10^{7}$ tokens. With such a small dataset, an epoch consists of only 40 parameter updates. Perhaps such a tiny dataset represents a different regime for language modeling, as overfitting happens very early in training (see Figure 16). Also note that the parameters differ very slightly from those obtained in Section 3, as here we are fitting the full $L(N,D)$ rather than just $L(N,\infty)$ or $L(\infty,D)$.

我们得到了一个绝佳的合身,除了数据集被缩小为$1024$倍的运行,大约是$2\times 10^{7}$令牌. 由于数据集如此小,一个时代只包含40个参数更新. 也许如此微小的数据集代表了不同的语言建模制度,因为过拟合在培训初期就发生了(见图16)。 那是因为这里我们配有完整的$L(N,D)$, 而不仅仅是$L(N,\infty)$或$L(\infty,D)$。

To chart the borderlands of the infinite data limit, we can directly study the extent of overfitting. For all but the largest models, we see no sign of overfitting when training with the full 22B token WebText2 dataset, so we can take it as representative of $D=\infty$. Thus we can compare finite $D$ to the infinite data limit by defining

为了绘制无限数据极限的边界地带,我们可以直接研究过拟合的程度. 除了最大的型号外,我们没有看到训练时使用完整的22B令牌WebText2数据集时过于匹配的迹象,因此我们可以把它当作$D=\infty$的代表. 因此,我们可以通过定义:

$$\delta L(N,D)\equiv\frac{L(N,D)}{L(N,\infty)}-1 \tag{4.2}$$

and studying it as a function of $N,D$. In fact, we see empirically that $\delta L$ depends only a specific combination of $N$ and $D$, as shown in Figure 16. This follows from the scaling law of Equation (1.5), which implies

并且作为$N,D$的一个函数来研究. 事实上,我们实证地看到,$\delta L$只取决于$N$和$D$的具体组合,如图16所示. 这源于方程式定律(1.5),这意味着

$$\delta L\approx\left(1+\left(\frac{N}{N_{c}}\right)^{\frac{\alpha_{N}}{\alpha_{D}}}\frac{D_{c}}{D}\right)^{\alpha_{D}}-1 \tag{4.3}$$

Note that at large $D$ this formula also has a series expansion in powers of $1/D$.

可在一般情况下,$D$这一公式还具有$1/D$的一系列权力扩张.

We estimate that the variation in the loss with different random seeds is roughly $0.02$, which means that to avoid overfitting when training to within that threshold of convergence we require

我们估计,使用不同随机种子的损失变化大致为$0.02$,这意味着为了避免在训练时过于适应我们要求达到的趋同阈值。

$$D\gtrsim(5\times 10^{3})\,N^{0.74} \tag{4.4}$$

With this relation, models smaller than $10^{9}$ parameters can be trained with minimal overfitting on the 22B token WebText2 dataset, but our largest models will encounter some mild overfitting. More generally, this relation shows that dataset size may grow sub-linearly in model size while avoiding overfitting. Note however that this does not typically represent maximally compute-efficient training. We should also emphasize that we have not optimized regularization (eg the dropout probability) while varying dataset and model size.

在这种关系下,小于$10^{9}$参数的模型可以在22B令牌WebText2数据集上进行最小的超配训练,但我们最大的模型会遇到一些轻微的超配. 更一般地说,这种关系表明,数据集大小可能在模型大小中子线性增长,同时避免过于适应. 但请注意,这通常并不代表计算效率最高的培训。 我们还应强调,我们没有优化正规化(如辍学概率),而不同的数据集和模型规模。

5 Scaling Laws with Model Size and Training Time

5 相对模型规模与训练时间的缩放定律

In this section we will demonstrate that a simple scaling law provides a good description for the loss as a function of model size $N$ and training time. First we will explain how to use the results of [MKAT18] to define a universal training step $S_{\rm min}$, which accounts for the fact that most of our models have not been trained at an optimal batch size. Then we will demonstrate that we can fit the model size and training time dependence of the loss using Equation (1.6). Later we will use these results to predict the optimal allocation of training compute between model size and training time, and then confirm that prediction.

在本节中,我们将表明,一个简单的缩放法很好地描述了损失作为模型大小$N$和训练时间的函数。 首先,我们将解释如何利用[MKAT18]的结果来定义一个通用培训步骤$S_{\rm min}$,这说明我们大多数的模型没有以最佳的批量规模进行训练。 然后,我们将证明,我们可以使用方程式(1.6)来适应损失的模型大小和训练时间依赖。 后来我们将利用这些结果来预测在模型大小和训练时间之间计算出的训练的最佳分配,然后确认这一预测。

5.1 Adjustment for Training at $B_{\rm crit}(L)$

5.1 在 $B_{\rm crit}(L)$ 下训练的修正

Figure 10: The critical batch size $B_{\rm crit}$ follows a power law in the loss as performance increase, and does not depend directly on the model size. We find that the critical batch size approximately doubles for every $13\%$ decrease in loss. $B_{\rm crit}$ is measured empirically from the data shown in Figure 18, but it is also roughly predicted by the gradient noise scale, as in [MKAT18].

图10:关键批量尺寸$B_{\rm crit}$在性能增加时在损失中遵循动力定律,不直接取决于模型大小. 我们发现,每$13\%$损失减少一倍左右的批量。 $B_{\rm crit}$从图18显示的数据中经验性地进行了测量,但也大致通过梯度噪声尺度来预测,如[MKAT18].

A simple empirical theory for the batch size dependence of training was developed in [MKAT18] (see also [SLA+18, ZLN+19]). It was argued that there is a critical batch size $B_{\rm crit}$ for training; for $B$ up to $B_{\rm crit}$ the batch size can be increased with very minimal degradation in compute-efficiency, whereas for $B>B_{\rm crit}$ increases in $B$ result in diminishing returns. It was also argued that the gradient noise scale provides a simple prediction for $B_{\rm crit}$, and that neither depends directly on model size except through the value of the loss that has been attained. These results can be used to predict how training time and compute will vary with the batch size. To utilize both training time and compute as effectively as possible, it is best to train with a batch size $B\approx B_{\rm crit}$. Training at $B\gg B_{\rm crit}$ minimizes the number of training steps, while $B\ll B_{\rm crit}$ minimizes the use of compute.

在MKAT18中为培训的批量大小依赖性开发了一个简单的经验理论. 据认为,培训有关键的批量尺寸$B_{\rm crit}$;对于$B$至$B_{\rm crit}$,批量尺寸可以增加,计算效率的降解程度极低,而对于$B>B_{\rm crit}$,$B$的增加导致回报下降。 也有人认为,梯度噪声尺度为$B_{\rm crit}$提供了一个简单的预测,除了通过已经实现的损失值外,两者都不直接取决于模型大小. 这些结果可用于预测训练时间和计算会如何随批量大小而变化. 为了尽可能有效地利用训练时间和计算,最好采用第$B\approx B_{\rm crit}$号批量培训。 $B\gg B_{\rm crit}$的培训尽量减少了培训步骤的数量,而$B\ll B_{\rm crit}$则尽量减少了计算的使用.

More specifically, it was demonstrated that for a wide variety of neural network tasks, the number of training steps $S$ and the number of data examples processed $E=BS$ satisfy the simple relation

更具体地说,证明对于多种神经网络任务来说,训练步骤的数量$S$和处理的数据实例的数量满足了简单的关系.

$$\left(\frac{S}{S_{\rm min}}-1\right)\left(\frac{E}{E_{\rm min}}-1\right)=1 \tag{5.1}$$

when training to any fixed value of the loss $L$. Here $S_{\rm min}$ is the minimum number of steps necessary to reach $L$, while $E_{\rm min}$ is the minimum number of data examples that must be processed.

当训练达到损失的任何固定价值时 这里的$S_{\rm min}$是达到$L$所需的最低步骤数,而$E_{\rm min}$是必须处理的数据示例的最低数目.

We demonstrate the relation (5.1) for Transformers in Figure 18 in the appendix. This relation defines the critical batch size

我们在附录图18中展示了Transformer的关系(5.1)。 此关系定义了批量批量大小

$$B_{\rm crit}(L)\equiv\frac{E_{\rm min}}{S_{\rm min}} \tag{5.2}$$

which is a function of the target value of the loss. Training at the critical batch size makes a roughly optimal time/compute tradeoff, requiring $2S_{\rm min}$ training steps and processing $E=2E_{\rm min}$ data examples.

这是损失目标值的函数。 关键批量规模的培训可以作出大致最佳的时间/计算权衡,需要$2S_{\rm min}$培训步骤和处理$E=2E_{\rm min}$数据实例。

In Figure 10 we have plotted the critical batch size and gradient noise scale[^4] as a function of training loss for two different models. We see that $B_{\rm crit}(L)$ is independent of model size, and only depends on the loss $L$. So the predictions of [MKAT18] continue to hold for Transformer language models. The critical batch size can be fit with a power-law in the loss

在图10中,我们绘制了临界批量提供关于梯度噪声的尺度[^4],作为两种不同模型培训损失的一个函数。 我们看到,$B_{\rm crit}(L)$独立于模型大小,仅取决于损失$L$. 因此,[MKAT18]的预测继续为Transformer语言模型所持有. 关键批量的尺寸与损失中的电源法相符

$$B_{\rm crit}(L)\approx\frac{B_{*}}{L^{1/\alpha_{B}}} \tag{5.3}$$

where $B_{*}\approx 2\times 10^{8}$ and $\alpha_{B}\approx 0.21$.

竿头:

We have chosen this parameterization for $B_{\rm crit}(L)$ because as the loss approaches its minimum value $L_{\rm min}$, the gradient noise scale is expected to diverge, and we expect $B_{\rm crit}$ to track this noise scale. We do not know $L_{\rm min}$, as we see no sign that our models are approaching it, but $L_{\rm min}>0$ since the entropy of natural language is non-zero. Since apparently $L_{\rm min}$ is much smaller than the values of $L$ we have achieved, we used a parameterization where $B_{\rm crit}$ diverges as $L\to 0$.

我们为$B_{\rm crit}(L)$选择了这个参数化,因为随着损失接近其最低值$L_{\rm min}$,梯度噪声尺度预计会有所不同,我们期望$B_{\rm crit}$跟踪这个噪声尺度. 我们不知道$L_{\rm min}$,因为我们没有看到我们的模型接近它的迹象,但是$L_{\rm min}>0$因为自然语言的 en是非零的. 由于$L_{\rm min}$显然比我们所实现的$L$的值要小得多,我们使用了一个参数化方法,其中$B_{\rm crit}$与$L\to 0$不同。

We will use $B_{\rm crit}(L)$ to estimate the relation between the number of training steps $S$ while training at batch size $B=2^{19}$ tokens and the number of training steps while training at $B\gg B_{\rm crit}$. This is simply

我们将使用$B_{\rm crit}(L)$来估计在$S$培训时的训练步骤数量与在$B\gg B_{\rm crit}$培训时的训练步骤数量之间的关系。 这很简单

$$S_{\rm min}(S)\equiv\frac{S}{1+B_{\rm crit}(L)/B}\qquad(\text{minimum steps, at }B\gg B_{\rm crit}) \tag{5.4}$$

for any given target value $L$ for the loss. This also defines a critical value of the compute needed to train to $L$ with a model of size $N$ if we were to train at $B\ll B_{\rm crit}(L)$. This is

有关损失的$L$。 这还定义了如果我们要在$B\ll B_{\rm crit}(L)$进行训练,需要使用大小为$N$的型号来进行$L$训练的计算的关键值. 这是

$$C_{\rm min}(C)\equiv\frac{C}{1+B/B_{\rm crit}(L)}\qquad(\text{minimum compute, at }B\ll B_{\rm crit}) \tag{5.5}$$

where $C=6NBS$ estimates the (non-embedding) compute used at batch size $B$.

其中$C=6NBS$估算了按批号$B$使用的(非嵌入式)计算.

5.2 Results for $L(N,S_{\rm min})$ and Performance with Model Size and Compute

5.2 $L(N,S_{\rm min})$ 的结果以及相对模型规模与算力的性能

Figure 11: When we hold either total compute or number of training steps fixed, performance follows $L(N,S)$ from Equation (5.6). Each value of compute budget has an associated optimal model size that maximizes performance. Mediocre fits at small $S$ are unsurprising, as the power-law equation for the learning curves breaks down very early in training.

图11:当我们掌握总计算数或固定培训步骤数时,业绩与方程式的$L(N,S)$(5.6)。 计算预算的每个价值都有一个相关的最佳模型规模,可以最大限度地提高绩效。 在小的$S$中,平庸的配方并不奇怪,因为学习曲线的幂律方程在训练初期就崩溃了.

Now we will use $S_{\rm min}$ defined in Equation (5.4) to obtain a simple and universal fit for the dependence of the loss on model size and training time in the infinite data limit. We will fit the stable, Adam-optimized training runs using Equation (1.6), repeated here for convenience:

现在,我们将使用在方程式(5.4)中定义的$S_{\rm min}$来获得一个简单和通用的适应,以适应在无限数据限制中对模型大小和训练时间的依赖. 使用方程式(1.6)进行最优化的稳定训练,

$$L(N,S_{\rm min})=\left(\frac{N_{c}}{N}\right)^{\alpha_{N}}+\left(\frac{S_{c}}{S_{\rm min}}\right)^{\alpha_{S}} \tag{5.6}$$

for the loss. We include all training steps after the warmup period of the learning rate schedule, and find a fit to the data with the parameters:

为损失。 我们包括学习率表的热化期之后的所有培训步骤,并找到与参数相匹配的数据:

Parameter	$\alpha_{N}$	$\alpha_{S}$	$N_{c}$	$S_{c}$
Value	$0.077$	$0.76$	$6.5\times 10^{13}$	$2.1\times 10^{3}$

Table 3: Fits to $L(N,S)$