Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction

Roll the dice & look before you leap : Going beyond the creative limits of next-token prediction

掷骰子，三思而后行 ：超越下一词预测的创造极限

Vaishnavh Nagarajan Affiliation: Google Research, US Correspondence to: vaishnavh@google.com Chen Henry Wu Affiliation: Carnegie Mellon University, Pittsburgh, US Correspondence to: chenwu2@cs.cmu.edu Charles Ding Affiliation: Carnegie Mellon University, Pittsburgh, US Aditi Raghunathan Affiliation: Carnegie Mellon University, Pittsburgh, US

Abstract

摘要

We design a suite of minimal algorithmic tasks that are a loose abstraction of open-ended real-world tasks. This allows us to cleanly and controllably quantify the creative limits of the present-day language model. Much like real-world tasks that require a creative, far-sighted leap of thought, our tasks require an implicit, open-ended stochastic planning step that either (a) discovers new connections in an abstract knowledge graph (like in wordplay, drawing analogies, or research) or (b) constructs new patterns (like in designing math problems or new proteins). In these tasks, we empirically and conceptually argue how next-token learning is myopic and memorizes excessively; multi-token approaches, namely teacherless training and diffusion models, comparatively excel in producing diverse and original output. Secondly, to elicit randomness without hurting coherence, we find that injecting noise at the input layer (dubbed as seed-conditioning) works surprisingly as well as (and in some conditions, better than) temperature sampling from the output layer. Thus, our work offers a principled, minimal test-bed for analyzing open-ended creative skills, and offers new arguments for going beyond next-token learning and temperature sampling. We make part of the code available under https://github.com/chenwu98/algorithmic-creativity

我们设计一套极简算法任务，作为 开放式 真实任务的松散抽象，从而干净、可控地量化当代语言模型的创造极限。与需要富有创意、目光长远的思维跃迁的真实任务类似，这些任务要求隐含、开放的 随机 规划步骤：要么 (a) 在抽象知识图中发现新联系（如文字游戏、类比、科研），要么 (b) 构造新模式（如设计数学题或新蛋白）。我们从经验与概念上论证：下一词学习目光短浅且过度记忆；多词元方法——无教师训练与扩散模型——在产出多样、原创输出上相对更强。其次，为在不伤连贯性的前提下引出随机性，我们发现在输入层注入噪声（称为 种子条件化 ）的效果出人意料地不亚于（在某些条件下甚至优于）输出层温度采样。因此，本文为分析开放式创造技能提供原则性极简试验台，并为超越下一词学习与温度采样提供新论据。部分代码见 https://github.com/chenwu98/algorithmic-creativity

1 Introduction

1 引言

Keywords: Not all forms of intelligence are solely about being correct or wrong. In open-ended tasks, what also matters is finding creative ways to satisfy a request, making surprising and fresh connections never seen before. For instance, consider responding to highly under-specified prompts like “Generate a challenging high-school word problem involving the Pythagoras Theorem.” or “Suggest some candidate therapeutic antibodies targeting the HER2 antigen.” or “Provide a vivid analogy to differentiate quantum and classical mechanics.” Creativity in such tasks requires generating responses that are not just correct or coherent, but are also diverse across responses and are original compared to the training data. These currently-sidelined desiderata will rise to prominence as we explore LLMs for open-ended scientific discovery (Gruver et al. 2023; Romera-Paredes et al. 2024; Si et al. 2024; Lu et al. 2024a), for generating novel training data (Yu et al. 2024; Yang et al. 2024c; Wang et al. 2023), and as we scale up test-time compute approaches that benefit from diversity in exploration, such as best-of-N (Cobbe et al. 2021; Chow et al. 2024; Dang et al. 2025) and long chain-of-thought reasoning (OpenAI 2024; DeepSeek-AI 2025; Snell et al. 2024; Wu et al. 2024).

关键词： 并非所有智能都只关对错。在 开放式 任务中，同样重要的是找到满足请求的创意方式，建立从未见过的惊奇新联系。例如回应高度欠定的提示：「生成一道涉及勾股定理的高中难题。」「给出若干靶向 HER2 抗原的候选治疗性抗体。」「用生动类比区分量子与经典力学。」此类任务上的创造力，要求回复不仅 正确 或 连贯 ，还要跨回复 多样 ，相对训练数据 原创 。这些目前被边缘化的诉求，会随我们用 LLM 做开放式科学发现（Gruver et al. 2023；Romera-Paredes et al. 2024；Si et al. 2024；Lu et al. 2024a）、生成新颖训练数据（Yu et al. 2024；Yang et al. 2024c；Wang et al. 2023），以及扩大受益于探索多样性的测试时算力方法——如 best-of-N（Cobbe et al. 2021；Chow et al. 2024；Dang et al. 2025）与长思维链推理（OpenAI 2024；DeepSeek-AI 2025；Snell et al. 2024；Wu et al. 2024）——而愈发突出。

Unlike simple open-ended tasks like generating names and basic sentences (Zhang et al. 2024b; Hopkins et al. 2023), many creative tasks (like designing a clever Olympiad problem) are said to involve a random flash of creative insight termed variously as a leap of thought (Wang et al. 2024a; Talmor et al. 2020; Zhong et al. 2024), a “eureka” moment (Bubeck et al. 2023), a mental leap (Holyoak & Thagard 1995; Callaway 2013; Hofstadter 1995) or an incubation step (Varshney et al. 2019). The thesis of this paper is that learning to solve such creative leap-of-thought tasks (defined shortly) is misaligned with the current language modeling paradigm (a) in terms of next-token learning, and (b) in how randomness is elicited. We articulate these two concerns by designing a suite of algorithmic tasks inspired by such creative tasks. We then demonstrate how the creativity of language models suffers in these tasks, and how this can be alleviated (to an extent, within our tasks).

不同于生成名字与简单句子等简单开放任务（Zhang et al. 2024b；Hopkins et al. 2023），许多创造任务（如设计巧妙奥赛题）据说涉及随机灵光一现，被称作 思维跃迁 （Wang et al. 2024a；Talmor et al. 2020；Zhong et al. 2024）、「尤里卡」时刻（Bubeck et al. 2023）、心智跃迁（Holyoak & Thagard 1995；Callaway 2013；Hofstadter 1995）或孵化步骤（Varshney et al. 2019）。本文论点是：学习这类创造跃迁任务（定义见下）与当前语言建模范式错位——(a) 在下一词学习上，(b) 在如何引出随机性上。我们设计受此类创造任务启发的算法任务套件，展示语言模型在此的创造力如何受损，以及如何（在任务范围内）有所缓解。

Concretely, for the scope of this paper, a creative leap-of-thought task refers to tasks that involve a search-and-plan process; crucially, this process orchestrates multiple random decisions in advance before generating the output. Typically, such a leap of thought is highly implicit in the text — to infer it, one has to deeply engage with the text and detect higher-order patterns. We can think of tasks like designing satisfying math problems, generating worthwhile research ideas, or drawing surprising analogies as examples of such tasks.

具体而言，本文范围内的创造跃迁任务指需要搜索与规划的过程；关键是，该过程在生成输出前须 预先 协调多个 随机 决策。通常，这种跃迁在文本中 高度隐含 ——要推断它，必须深入文本、察觉高阶模式。设计令人满意的数学题、提出有价值的研究想法、画出惊奇类比，都可视为此类任务。

Ideally, one would directly study these real-world tasks to quantify the limits of language models. Indeed, a flurry of recent works report that LLM-generated research ideas tend to be rephrased from existing ideas (Gupta & Pruthi 2025; Beel et al. 2025) and that LLM outputs tend to be less creative than humans e.g., Chakrabarty et al. 2024; Lu et al. 2024b (See §J). While assessing real-world tasks is a lofty goal, the evaluations are subjective (Wang et al. 2024b; Runco & Jaeger 2012), and when the model has been exposed to all of the internet, originality is hard to ascertain. Thus, the conclusions will inevitably invite debate (such as Si et al. 2024 vs. Gupta & Pruthi 2025 or Lu et al. 2024a vs. Beel et al. 2025).

理想情况应直接在真实任务上量化语言模型极限。确有一批近作报告：LLM 生成的研究想法往往是对已有想法的改写（Gupta & Pruthi 2025；Beel et al. 2025），LLM 输出也常不如人类有创意，例如 Chakrabarty et al. 2024；Lu et al. 2024b（见 §J）。评估真实任务目标远大，但评价主观（Wang et al. 2024b；Runco & Jaeger 2012）；当模型已见过整个互联网，原创性也难确认。结论难免引发争论（如 Si et al. 2024 对 Gupta & Pruthi 2025，或 Lu et al. 2024a 对 Beel et al. 2025）。

In search of more definitive conclusions, we approach from a different angle: we study minimal and controllable tasks that are loose abstractions of real-world tasks and yet allow one to rigorously quantify originality and diversity. This follows along the lines of recent works that have studied the diversity of models in graph path-finding (Khona et al. 2024) and generating challenging CFGs (Allen-Zhu & Li 2023b). Broadly, we refer to such tasks as open-ended algorithmic tasks. Our aim is to design tasks more minimal than these prior tasks, and crucially, tease apart distinct computational skills required for creativity. This will allow us to systematically investigate issues in the current paradigm of model training and propose alternatives.

为求更确定的结论，我们换一角度：研究作为真实任务松散抽象、却能严格量化原创与多样性的极简可控任务。这延续近期在图路径寻找（Khona et al. 2024）与生成困难 CFG（Allen-Zhu & Li 2023b）上研究模型多样性的工作。我们统称此类任务为开放式算法任务。目标是设计比先前更极简、并能拆开创造力所需不同计算技能的任务，从而系统考察当前训练范式的问题并提出替代。

As our first main contribution, we draw inspiration from cognitive science literature (Boden 2003) (see also Franceschelli & Musolesi 2023) to design algorithmic tasks isolating two distinct types of creative leaps of thought. The first class of tasks involves combinational creativity: drawing novel connections in knowledge, like in research, wordplay or drawing analogies (see Fig 1 for task description). The second class of tasks involves exploratory creativity: constructing fresh patterns subject to certain rules, like in designing problems and suspense (see Fig 2). In these tasks, we can precisely evaluate models for the fraction of generations that are coherent, unique and original (not present in training set). We term this metric “algorithmic creativity” to denote that it is solely evaluates the computational aspects of creativity.

作为第一项主要贡献，我们受认知科学文献（Boden 2003）（亦见 Franceschelli & Musolesi 2023）启发，设计算法任务以隔离两类不同的创造跃迁。第一类是 组合创造 ：在知识中建立新联系，如科研、文字游戏或类比（任务描述见图 1）。第二类是 探索创造 ：在规则约束下构造新模式，如出题与悬念设计（见图 2）。在这些任务上，我们可精确评估生成中 连贯、唯一且原创 （不在训练集中）的比例。我们称该指标为「算法创造力」，表示它只评估创造力的计算侧面。

(a) Sibling Discovery

(a) 兄弟姐妹发现

(a) Circle Construction

(a) 圆构造

Figure 3: Multi-token teacherless finetuning improves algorithmic creativity (top; Eq 1) and reduces memorization (bottom; fraction of generations seen during training) on our four open-ended algorithmic tasks for a Gemma v1 (2B) model.

图 3： 无教师多词元微调提升算法创造力（上；式 1），并降低记忆（下；训练中见过的生成比例），在四项开放式算法任务上对 Gemma v1（2B）模型 。

Within this framework, we articulate two creative limits of the current language modeling paradigm. First, we empirically find that next-token learning achieves lower algorithmic creativity (and higher memorization) compared to multi-token approaches, namely, teacherless training (Bachmann & Nagarajan 2024; Monea et al. 2023; Tschannen et al. 2023) and diffusion models (Hoogeboom et al. 2021; Austin et al. 2021; Lou et al. 2023) (see Fig 3 and Fig 4). Our argument is that in all our tasks, inferring the latent leap of thought requires observing global higher-order patterns rather than local next-token patterns in the sequence.

在此框架内，我们阐明当前语言建模范式的两点创造局限。第一，经验上发现下一词学习的算法创造力低于多词元方法——无教师训练（Bachmann & Nagarajan 2024；Monea et al. 2023；Tschannen et al. 2023）与扩散模型（Hoogeboom et al. 2021；Austin et al. 2021；Lou et al. 2023）（见图 3、图 4），记忆也更高。我们的论证是：在全部任务中，推断潜在跃迁需要观察序列中的全局高阶模式，而非局部下一词模式。

Next, we turn to how we elicit randomness from a Transformer. While the de facto approach is to elicit randomness at the output — temperature sampling — we contrast this with injecting randomness into the model. Concretely, we study seed-conditioning, where we train with a random prefix string per example. Surprisingly, we find that seed-conditioning induces non-trivial algorithmic creativity even with deterministic, greedy decoding . In fact, seed-conditioning is competitive with and in some cases, more creative than temperature sampling. Intuitively, maximizing diversity at the output-token-level is computationally burdensome: it requires simultaneously processing a diverse set of leaps of thoughts to compute a marginalized token distribution. It is easier to first sample a single latent leap of thought, and then compute the token conditioned on that one leap. We conjecture that seed-conditioning enables a latent-leap-conditioned token generation.

接着讨论如何从 Transformer 引出随机性。默认做法是在输出端用温度采样；我们对比向模型注入随机性。具体研究 种子条件化 ：为每个训练样例配一个随机前缀串。令人惊讶的是，即便 确定性贪心解码 ，种子条件化也能诱导不可忽略的算法创造力；事实上它与温度采样相当，有时更有创意。直观上，在输出词元级最大化多样性计算负担重：须同时处理多样跃迁以计算边缘化词元分布。更容易的是先采样单个潜在跃迁，再据此计算条件词元。我们猜想种子条件化实现了以潜在跃迁为条件的词元生成。

Overall, we hope our study advances the field in two directions. First, we provide a new angle to advocate for multi-token approaches, orthogonal to the “path-star” example in Bachmann & Nagarajan 2024 (or 6 in short). Whereas, the path-star example portrays a gap in correctness of reasoning, ours shows a gap in diversity of open-ended thinking. We note though that 6 is an impossibility result where next-token learning breaks down spectacularly (unless there is exponential data or compute), while ours is a data-inefficiency result (where next-token learning occurs but is mediocre). Next, the gap we show appears even in 2-token-lookahead tasks as against the many-token-lookahead path-star task. Third, and perhaps most conceptually important is the fact that, while the path-star task is amenable to next-token prediction upon reversing the tokens, we identify tasks where no re-ordering is friendly towards next-token prediction — the optimal thing to do is to globally learn higher-order patterns implicit in the whole future sequence . This presents a challenge to recent proposals that aim to fix next-token prediction via permutations (Pannatier et al. 2024; Thankaraj et al. 2025) or partial lookaheads (Bavarian et al. 2022; Fried et al. 2022; Kitouni et al. 2024; Nolte et al. 2024).

总体希望沿两个方向推进领域。第一，为多词元方法提供新论据，正交于 Bachmann & Nagarajan 2024 的「路径星」例子（下称 6）。路径星刻画 正确性 推理差距，我们展示开放式思维 多样性 差距。注意 6 是不可能性结果（除非指数数据或算力，下一词学习彻底崩坏），我们则是数据低效结果（下一词仍在学，但平庸）。其次，我们在仅需 2 词元前瞻的任务上也能看到差距，而路径星需多词元前瞻。第三、或许概念上最重要的是：路径星在反转词元顺序后可对下一词预测友好，我们找到无论何种重排都不友好下一词预测的任务—— 最优做法是全局学习整段未来序列中隐含的高阶模式 。这对靠置换（Pannatier et al. 2024；Thankaraj et al. 2025）或部分前瞻（Bavarian et al. 2022；Fried et al. 2022；Kitouni et al. 2024；Nolte et al. 2024）修补下一词预测的近期方案提出挑战。

As a second direction of progress, we hope our work provides a foundation to think about open-ended tasks which are extremely hard to quantify in the wild. This may spur more algorithmic explorations on improving diversity (such as our approach of seed-conditioning) and on curbing verbatim memorization in language models.

作为第二方向，希望为极难在野外量化的开放式任务提供思考基础，从而推动更多提升多样性的算法探索（如种子条件化）以及抑制语言模型逐字记忆。

Our contributions:

我们的贡献 ：

We create minimal, controlled and easy-to-quantify open-ended algorithmic tasks. These tasks isolate, and loosely capture two fundamental modes of creativity.

创建极简、可控、易量化的开放式算法任务，隔离并松散刻画两种基本创造模式。

We find that multi-token prediction through teacherless training or diffusion, results in significantly increased algorithmic creativity and reduced memorization in our tasks compared to next-token prediction.

发现通过无教师训练或扩散做多词元预测，相对下一词预测显著提高算法创造力并降低记忆。

Our argument provides new support for multi-token prediction, going beyond (6). We show a gap in creativity in an open-ended task (rather than correctness in a deterministic one), in much simpler 2-token-lookahead tasks, and in tasks where no token permutation is friendly to next-token-learning.

论证为多词元预测提供超越 (6) 的新支持：在开放任务上展示创造力差距（而非确定性任务上的正确性），任务更简单（2 词元前瞻），且无任何词元置换对下一词学习友好。

We find that input-randomization via seed-conditioning achieves algorithmic creativity competitive with conventional output-randomization via temperature sampling.

发现输入随机化的种子条件化，其算法创造力可与传统输出随机化的温度采样匹敌。

2 Open-ended algorithmic tasks & two types of creativity

2 开放式算法任务与两类创造力

We are interested in designing simple algorithmic tasks that are loosely inspired by endeavors such as generating scientific ideas, wordplay, narration, or problem-set design, where one needs to generate strings that are both “interesting” and never seen before. In all these tasks, before generating the output, one requires a (creative) leap of thought, a process that (a) is implicit i.e., is not spelled out in token space (or is even inherently hard to spell out), (b) involves discrete random choices (c) and together, those choices must be coherent in that they are carefully planned to satisfy various non-trivial, discrete constraints. These constraints fundamentally define the task and make it interesting e.g., a word problem should be solvable by arithmetic rules, or a pun must deliver a surprising punchline. The goal in such open-ended tasks is not just coherence though, but also diversity and novelty — generations must be as varied as possible and must not be regurgitated training data. Before we design tasks that require the aforementioned leap of thought, we first clarify what tasks do not require it.

我们关注设计简单算法任务，松散启发自生成科学想法、文字游戏、叙事或题集设计等：需要生成既「有趣」又从未见过的字符串。在这些任务中，生成输出前需要一次*（创造）思维跃迁*，该过程 (a) 隐含，即未在词元空间写明（甚至本质难写明）；(b) 涉及离散 随机 选择；(c) 这些选择须 连贯 ，经仔细规划以满足各种非平凡离散约束。约束从根本上定义任务并使之有趣，例如应用题须可用算术规则求解，双关须交付惊奇包袱。开放式任务的目标不只是连贯，还有多样性与新颖——生成应尽量多样，且不能是训练数据的复读。在设计需要上述跃迁的任务前，先澄清哪些任务 不需要 它。

Open-ended tasks that do not require a leap of thought. One simple open-ended task that may come to mind is generating uniformly-random known entities, like celebrity names (Zhang et al. 2024b). However, there is no opportunity to create a novel string here. A more interesting example may be generating grammatically coherent PCFG strings following a subject verb object format e.g., the cat chased a rat (Hopkins et al. 2023). While novel strings become possible here, no sophisticated leaps of thought are involved; each token can be generated on the fly, satisfying a local next-token constraint to be coherent.

不需要思维跃迁的开放式任务 。 一个简单例子是均匀随机生成已知实体，如名人名字（Zhang et al. 2024b）。但这里没有创造新串的空间。更有趣的是按主谓宾生成语法连贯的 PCFG 串，例如 the cat chased a rat（Hopkins et al. 2023）。虽可出现新串，却无复杂跃迁；每个词元可即时生成，满足局部下一词约束即可连贯。

In light of this, we can rephrase our goal as designing open-ended, creative tasks where coherence requires satisfying more interesting, “global” constraints. To build this systematically, we draw inspiration from literature in cognitive science (Boden 2003). Boden 2003 argues that fundamentally, there are three forms of creativity in that order: combinational, exploratory and transformative . We elaborate on the first two (the last, we do not look at).

据此，目标可改写为：设计连贯性要求满足更有趣「全局」约束的开放、创造任务。为系统构建，我们受认知科学文献（Boden 2003）启发。Boden 2003 认为创造力从根本上有三种，依次为 组合、探索与变革 。我们展开前两种（第三种本文不涉及）。

2.1 The fundamental types of creativity (Boden 2003)

2.1 创造力的基本类型（Boden 2003）

Combinational creativity. Consider rudimentary wordplay of the form “What musical genre do balloons enjoy? Pop music.” or “What kind of shoes do spies wear? Sneakers.” There is a global structure here: two unrelated entities (genre & balloons) are related eventually through a punchline (pop); the punchline itself is a mutual neighbor on a semantic graph. More broadly, Boden 2003 argues that many tasks, like the above, involve “making unfamiliar combinations of familiar ideas” or the “unexpected juxtaposition of [known] ideas”. Other tasks include drawing analogies, or finding connections between disparate ideas in science. All these tasks involve a leap of thought that in effect searches and plans over a space of known facts and combines them.

组合创造 。 考虑初级文字游戏：「气球喜欢什么音乐类型？流行音乐。」或「间谍穿什么鞋？运动鞋。」这里有全局结构：两个不相关实体（类型与气球）最终通过包袱（pop）联系起来；包袱本身是语义图上的共同邻居。更广地，Boden 2003 认为许多任务涉及「熟悉观念的陌生组合」或「[已知]观念的意外并置」。其他任务包括画类比，或在科学中发现迥异想法之间的联系。这些任务的跃迁本质上是在已知事实空间上搜索并规划、再加以组合。

Exploratory creativity. Consider the act of developing a mystery or designing logical puzzles. These endeavors are not as knowledge-heavy. What they crucially require is constructing fresh patterns that satisfy some highly non-trivial global constraint e.g., being resolvable as per some rules (e.g., logic). Such endeavors fall into a second class of exploratory creativity in Boden 2003. This includes much grander forms of exploration e.g., exploring various forms of outputs within a stylistic constraint, or exploring various corollaries within a theoretical paradigm in physics or chemistry. The leap of thought here requires searching over all possible sequences, constrained by a set of rules.

探索创造 。 考虑构思悬疑或设计逻辑谜题。这些活动不那么依赖知识存量；关键是构造满足高度非平凡全局约束的新模式，例如可按某些规则（如逻辑）求解。此类努力落入 Boden 2003 的第二类 探索 创造，也包括更宏大的探索：在风格约束内探索各种输出形式，或在物理、化学某理论范式内探索各种推论。此处的跃迁需要在规则约束下搜索所有可能序列。

In the upcoming sections, we will attempt to capture some core computations of rudimentary instances within the two classes of creative skills above. We emphasize that by no means does our minimal algorithmic setup intend to capture the human values that go into these endeavors; nor do they capture the rich array of creative acts that Boden 2003 discusses within these categories. (See limitations in §6)

后续各节尝试捕捉上述两类创造技能中初级实例的一些核心计算。我们强调：极简算法设定绝无意捕捉这些活动中的人类价值，也未覆盖 Boden 2003 在各类下讨论的丰富创造行为。（局限见 §6）

2.2 The basic setting and notations

2.2 基本设定与记号

\mathcal{D}

m

\texttt{coh}:\mathbb{V}^{L}\mapsto\{\texttt{true},\texttt{false}\}

\texttt{coh}:\mathbb{V}^{L}\mapsto\{\texttt{true},\texttt{false}\}

T

T

\texttt{mem}_{S}(\bm{s})

\texttt{mem}_{S}(\bm{s})

\hat{\texttt{cr}}_{N}(T)=\frac{\texttt{uniq}(\{\bm{s}\in T|\neg\texttt{mem}_{S}(\bm{s})\land\texttt{coh}(\bm{s})\})}{|T|}. \tag{1}

Our setup models an in-distribution form of novelty as it offers a rigorous way to study the problem. Admittedly, this is a far simpler form of novelty than what is expected in the real-world. Nevertheless, even this simple setting will help us appreciate the limits of current language models.

我们的设定刻画分布内新颖性，以便严格研究。诚然，这比真实世界期望的新颖简单得多。即便如此，这一简单设定仍有助于认识当前语言模型的极限。

2.3 Tasks inspired by combinational creativity

2.3 受组合创造启发的任务

Combinational creativity requires a recall-and-search through entities from memory, constrained to relate with each other in an interesting way. We abstract this through tasks that discover structures from an in-weights graph i.e., a graph stored in the model weights, not reveal in context.

组合创造需要从记忆中回忆并搜索实体，并约束它们以有趣方式相互关联。我们用从 权重内 图——存在模型权重中、不在上下文揭示的图——发现结构的任务来抽象这一点。

\mathcal{G}

\mathcal{G}

\Gamma

\Gamma

(\Gamma,\gamma,\gamma^{\prime})

(\Gamma,\gamma,\gamma^{\prime})

\bm{s}=(\gamma,\gamma^{\prime},\Gamma

\bm{s}=(\gamma,\gamma^{\prime},\Gamma)

\mathcal{G}=({V},E)

\mathcal{G}=({V},E)

This type of a higher-order planning task can be thought of an abstraction of more complex wordplay (like antanaclasis, where a word must repeat in two different senses in a sentence, while still being coherently related to the rest of the sentence) or creating word games (like crosswords) or discovering contradictions or feedback loops in a body of knowledge, an essential research skill — see §B.3.

这类高阶规划可视为更复杂文字游戏（如 同词异义 ：一词须在句中以两种不同意义出现，同时与句中其余部分连贯相关）、字谜（如填字）或在知识体中发现矛盾与反馈环——重要研究技能——的抽象，见 §B.3。

2.4 Tasks inspired by exploratory creativity

2.4 受探索创造启发的任务

We are also interested in the less-knowledge-intensive creativity involved in tasks like designing word problems that demand novel solutions. We capture this skill minimally through tasks that construct new structures. No knowledge graph is involved in these tasks.

我们也关心设计要求新解法的应用题等不那么知识密集的创造。用构造新结构的任务极简刻画该技能；这些任务不涉及知识图。

N

N

\pi

\pi

\pi

\pi

2.5 Permutation-invariance of our tasks

2.5 任务的置换不变性

We emphasize a key novel aspect in our last three tasks. Many algorithmic tasks in literature like addition (Lee et al. 2024), or the path-star task (6) or Sibling Discovery have a natural ordering in which the tokens can be learned and generated, even if not left to right. However, the Triangle Discovery, Line Construction and Circle Construction tasks are permutation-invariant — no token is more privileged than the other, and all tokens must be “simultaneously learned” to infer the underlying process. We view this as an abstraction of real-world tasks where the creative process is highly implicit, and not immediate from the text. These tasks offer a test-bed even for non-next-token approaches that rely on re-permuting the tokens (Pannatier et al. 2024; Thankaraj et al. 2025) or predicting only a part of the future (Kitouni et al. 2024; Nolte et al. 2024; Bavarian et al. 2022; Fried et al. 2022).

我们强调后三项任务的一个关键新侧面。文献中许多算法任务如加法（Lee et al. 2024）、路径星任务 (6) 或兄弟姐妹发现，即便不是从左到右，也有词元可被学习与生成的自然顺序。然而三角形发现、线构造与圆构造是置换不变的——无词元比其他更特权，必须「同时学习」所有词元才能推断底层过程。我们视之为真实任务的抽象：创造过程高度隐含，无法从文本立即看出。这些任务也为依赖重排词元（Pannatier et al. 2024；Thankaraj et al. 2025）或只预测部分未来（Kitouni et al. 2024；Nolte et al. 2024；Bavarian et al. 2022；Fried et al. 2022）的非下一词方法提供试验台。

2.6 How next-token learning may suffer in our tasks

2.6 下一词学习在我们任务上可能如何吃亏

\bm{z}

\bm{z}

(\gamma,\gamma^{\prime},\Gamma)

(\gamma,\gamma^{\prime},\Gamma)

\bm{z}:=\Gamma

\bm{z}:=\Gamma

p(\gamma^{\prime}|\bm{z}:=\Gamma)

p(\gamma^{\prime}|\bm{z}:=\Gamma)

p(\bm{z})

p(\bm{z})

3 Training and Inference

3 训练与推理

\bm{p}

\bm{p}

Diffusion models. Rather than sequentially predicting each token conditioned on previously generated tokens, discrete diffusion models (Hoogeboom et al. 2021; Austin et al. 2021) iteratively add noise to all tokens and then learn to denoise them in reverse. This strategy allows the model to capture global dependencies among tokens during training, making it an example of a multi-token objective. In our experiments, we used the score entropy discrete diffusion model (SEDD, Lou et al. 2023), which starts generation from a sequence of fully masked tokens. Over multiple steps, the model simultaneously predicts and unmasks these tokens, progressively refining the entire sequence.

扩散模型。 离散扩散模型（Hoogeboom et al. 2021；Austin et al. 2021）不是按已生成词元顺序条件预测每个词元，而是迭代地对所有词元加噪，再学习反向去噪。该策略使模型在训练中捕捉词元间全局依赖，是多词元目标的一例。实验中我们使用分数熵离散扩散模型（SEDD，Lou et al. 2023），从全掩码词元序列开始生成；经多步同时预测并揭开这些词元，逐步精炼整段序列。

Inference. In all the above techniques, we extract each sample independently from the model (as against say, extracting them in continuous succession in the same context). For Transformers, during inference, we perform standard autoregression in both the next- and multi-token trained settings. We do this either with greedy decoding or with nucleus sampling (Finlayson et al. 2024).

推理 。 上述技术均从模型 独立 抽取每个样本（而非在同一上下文中连续抽取）。对 Transformer，推理时在下一词与多词元训练设定下都做标准自回归，可用贪心解码或核采样（Finlayson et al. 2024）。

3.1 Seed-conditioning

3.1 种子条件化

Unlike closed-ended tasks where a prompt (i.e., a prefix) maps to a unique correct answer, our open-ended tasks are prompt-free. As an alternative to this, we also tried prepending a prompt of pause tokens (Goyal et al. 2024) to all datapoints — both during training and during inference — to enable extra computation before the model emits its outputs. Next, we tried a more sophisticated alternative we term seed-conditioning . Here, we associate each training point with an arbitrary (meaningless) prefix to the model (e.g., of random characters) called a seed; during test-time, we prompt with novel seeds hoping to extract novel test data. We provide hypotheses motivating this procedure in §5.1.

封闭任务中提示（即前缀）映到唯一正确答案；我们的开放任务无提示。作为替代，我们尝试给所有数据点——训练与推理——前置停顿词元提示（Goyal et al. 2024），以便在模型发出输出前多算几步。接着尝试更精巧的替代，称为 种子条件化 ：为每个训练点配一个任意（无意义）前缀（如随机字符）称为种子；测试时用新颖种子提示，希望抽出新颖测试数据。动机假设见 §5.1。

4 Experimental results

4 实验结果

Key details. Part of our experiments are performed for a Gemma v1 (2B) pre-trained model (Gemma Team et al. 2024), averaged over 4 runs. For diffusion, we use a 90M (non-embedding) parameters Score Entropy Discrete Diffusion model (SEDD; Lou et al. 2023). For a fair comparison against NTP, we use a 86M (non-embedding) parameters GPT-2 model (Radford et al. 2019). In all our experiments, we finetune the models until it is clear that algorithmic creativity (Eq. 1) has saturated. All values are reported from this checkpoint. Finally, since our best Transformer results were under noise-conditioning (for both next- and mult-token training), our main results are reported under that training setting; we provide various ablations without that as well. Please see §D for more experimental details, and §C for precise dataset details (e.g., how the graph is constructed, how sequences are formatted etc.,).

关键细节。 部分实验在 Gemma v1（2B）预训练模型上完成（Gemma Team et al. 2024），对 4 次运行取平均。扩散用 90M（非嵌入）参数的分数熵离散扩散模型（SEDD；Lou et al. 2023）。为公平对照 NTP，使用 86M（非嵌入）参数 GPT-2 模型（Radford et al. 2019）。全部实验微调到算法创造力（式 1）饱和为止，指标均取该检查点。由于最佳 Transformer 结果均在噪声条件化下（下一词与多词元训练皆然），主结果在该训练设定下报告；我们也提供无噪声条件化的多种消融。更多实验细节见 §D，精确数据细节（如图如何构造、序列如何格式化等）见 §C。

4.1 Observations

4.1 观察

Multi-token prediction improves algorithmic creativity significantly. In all our datasets, the algorithmic creativity of the Gemma v1 (2B) model increases significantly under multi-token prediction ( Fig 3), with nearly a 5x factor for the discovery datasets. Note that for this, we have selected the learning rate favorable towards next-token prediction; tuning for multi-token yields further gains (Fig 15).

多词元预测显著提升算法创造力 。 在全部数据集上，Gemma v1（2B）在多词元预测下算法创造力显著上升（图 3），发现类数据集近 5 倍。注意此处学习率是对下一词预测有利的选择；为多词元调参还可再涨（图 15）。

Figure 4: Multi-token diffusion training improves algorithmic creativity (top; Eq 1) on our four open-ended tasks, and reduces memorization on discovery tasks but not construction tasks (bottom).

图 4： 多词元扩散训练在四项开放式任务上提升算法创造力（上；式 1），并在发现类任务上降低记忆，但在构造类任务上不然（下） 。

In Fig 4, we report performance for the diffusion model against next-token & teacherless training of similar-sized Transformers. We see that diffusion models are consistently better than next-token training, achieving up to 5x higher algorithmic creativity. However, the gains are much smaller or absent with teacherless training. This echoes prior discussions suggesting that the teacherless objective is a hard objective to optimize for smaller Transformers (6); other multi-token approaches (Gloeckle et al. 2024) are even known to hurt small models of the order of 300M.

图 4 报告扩散模型相对同规模 Transformer 的下一词与无教师训练。可见扩散相对下一词训练持续更好，算法创造力最高约 5 倍。但相对无教师训练，增益小得多或没有。这呼应先前讨论：无教师目标对较小 Transformer 难优化 (6)；其他多词元方法（Gloeckle et al. 2024）甚至已知会 伤害 约 300M 量级的小模型。

Multi-token prediction reduces memorization significantly. Algorithmic creativity may suffer either because the model outputs incoherent garbage, or because it repeats the same original output, or because it simply parrots out the training data. In almost all settings, it is the last reason that dominates: across the board (in Fig 3, Fig 4 bottom), next-token prediction is significantly prone to memorizing the data, while multi-token methods are highly resistant. As foreshadowed in §2.6, we hypothesize that this is because NTP memorizes the earlier training tokens without a global plan, having fit the later tokens via local coherence rules (because of Clever Hans cheats à la 6). Note that an exception to this is the smaller models (especially for diffusion) in our construction tasks, where memorization increases under the multi-token objectives; but this increase is mild and crucially, does not hurt algorithmic creativity.

多词元预测显著降低记忆 。 算法创造力受损，要么因输出不连贯垃圾，要么因重复同一原创输出，要么因简单复读训练数据。几乎所有设定中，最后一项占主导：全面来看（图 3、图 4 下），下一词预测极易记忆数据，而多词元方法高度抵抗。如 §2.6 所预示，我们假设这是因为 NTP 在靠局部连贯规则拟合较后词元（因聪明汉斯作弊 à la 6）后，无全局计划地记忆较早训练词元。例外是构造任务上的较小模型（尤其扩散），多词元目标下记忆上升；但升幅温和，且关键是不伤算法创造力。

We point the reader to §B.4 for further empirical evidence supporting our argument about NTP from §2.6, including experiments on token-reordering and experiments ruling out other hypotheses.

请读者参阅 §B.4，获取支持 §2.6 中 NTP 论证的进一步经验证据，包括词元重排实验与排除其他假说的实验。

Figure 5: Seed-conditioning significantly improves algorithmic creativity of both next- and multi-token prediction on Gemma v1 (2B) model. The X-axis labels denote the prefix (at training and inference) and the temperature (at inference).

图 5： 种子条件化显著提升 Gemma v1（2B）上下一词与多词元预测的算法创造力。横轴标签表示（训练与推理时的）前缀以及（推理时的）温度 。

Figure 6: Seed-conditioning improves algorithmic creativity of the GPT-2 (86M) model: All models are trained with NTP. The color denotes temperature, while the X-axis, the prefix. Remarkably, seed-conditioning achieves high creativity even with greedy decoding. However, note that for the discovery datasets, top-K sampling must be enabled to see a positive gap between seed- and null- settings.

图 6： 种子条件化提升 GPT-2（86M）模型的算法创造力：全部模型以 NTP 训练。颜色表示温度，横轴表示前缀。值得注意的是，即便贪心解码，种子条件化也能达到高创造力。但注意在发现类数据集上，须启用 top-K 采样才能看到种子与空设定之间的正向差距 。

Seed-conditioning improves algorithmic creativity for Transformers. Orthogonal to the effect of multi-token vs. next-token objectives, we point out the effect of seed-conditioning a Transformer. First, and most remarkably, with seed-conditioning, there is no need for temperature: even deterministic generation via greedy decoding generates diverse outputs that have highly non-trivial creativity values (Fig 5, Fig 6). This means that, even though the seeds we provide are arbitrarily paired with the training datapoint, the models have learned to transform the arbitrary noise into meaningful randomness. We also find that increasing the seed lengths consistently boosts algorithmic creativity for both next-token and multi-token approaches (Fig 5, 23). Thus, for Transformers, we propose viewing seed-conditioning as a knob for diversity distinct from temperature sampling.

种子条件化提升 Transformer 的算法创造力 。 与多词元相对下一词目标的效应正交，我们指出对 Transformer 做种子条件化的效应。首先也最引人注目：有种子条件化时不必用温度——即便贪心解码的确定性生成，也能产出多样性高、创造力值不可忽略的输出（图 5、图 6）。这意味着，尽管种子与训练点任意配对，模型已学会把任意噪声变成有意义的随机性。我们还发现加长种子持续提升下一词与多词元方法的算法创造力（图 5、23）。因此对 Transformer，我们建议把种子条件化视为有别于温度采样的多样性旋钮。

Next, in comparison to temperature sampling, in our settings, seed-conditioning results in the highest algorithmic creativity in both the small models (Fig 6) and the larger models (Fig 5). In fact, in our larger models, the null and pause token prefix with temperature sampling (Fig 18) exhibit almost no algorithmic creativity (they are mode collapsed, see Fig 19, 20). Besides, for any fixed temperature, prefixing a seed only improves performance over a null prefix (Fig 6, 18).

其次，相对温度采样，在我们设定中种子条件化在小模型（图 6）与较大模型（图 5）上都给出最高算法创造力。事实上，较大模型上带温度采样的空与停顿词元前缀（图 18）几乎没有算法创造力（它们模式崩塌，见图 19、20）。此外，对任何固定温度，前置种子只相对空前缀提升表现（图 6、18）。

K

K

Robustness to hyperparameters. In §E and Fig 22, we do sensitivity analysis on all the datasets. We report how our above findings are robust to the choice of learning rate, batch-size, number of training steps, weight given to the multi-token objective, varying sampling conditions and reasonable changes to the complexity of the dataset and training set size (as per our argument in §2.6, we do expect the next-vs. multi-token gap to diminish for larger dataset size).

对超参的稳健性 。 在 §E 与图 22，我们对全部数据集做敏感性分析，报告上述发现对学习率、批大小、训练步数、多词元目标权重、不同采样条件以及数据集复杂度与训练集大小合理变化的稳健性（按 §2.6 的论证，我们预期下一词相对多词元的差距会随数据增多而缩小）。

4.2 An initial exploration of real-world summarization

4.2 真实摘要任务的初步探索

5

为更现实地审视发现，我们在摘要任务（XSUM、CNN/DailyMail）上对用 NTP 与多词元无教师目标微调的 GPT 模型做初步调查。对任意给定提示，通过生成 5 个不同补全并计算 Self-Bleu（Zhu et al. 2018）衡量多样性。诚然摘要不如我们希望的那么开放：质量更高（即 Rouge 更高；Lin 2004）必然意味着多样性更低。为此我们绘制多样性如何随模型质量随时间演变；图 7 显示在给定质量下，较大的多词元模型多样性更高（尽管幅度不大）。该增幅在小模型上不成立，在 CNN/DailyMail 上也不总明显（见 §I）。有趣的是，无教师训练持续提升按 Rouge 衡量的摘要质量。

Figure 7: Multi-token training improves diversity scores for XSUM summarization for large GPT-2 models: Here, we plot diversity and quality as measured over multiple checkpoints during finetuning, and observe differences in diversity for a fixed quality.

图 7： 多词元训练提升大型 GPT-2 模型在 XSUM 摘要上的多样性分数：我们绘制微调期间多检查点的多样性与质量，并观察固定质量下的多样性差异 。

5 Discussion

5 讨论

5.1 Intuition about seed-conditioning

5.1 关于种子条件化的直觉

Seed-conditioning produces non-trivial algorithmic creativity even with greedy decoding. Perhaps, one could view the seed prefixes as a simpler alternative to varying the wordings of a prompt Li et al. 2023; Lau et al. 2024; Naik et al. 2024 or tuning a soft-prompt (Wang et al. 2024c), both of which are known to induce diversity. However, it is unexpected that the seeds provide any meaningful randomness at all, given they are arbitrarily picked with no semantic relationship to the training point. We do not understand this presently.

种子条件化即便在贪心解码下也能产生不可忽略的算法创造力。或许可将种子前缀视为变换提示措辞（Li et al. 2023；Lau et al. 2024；Naik et al. 2024）或调软提示（Wang et al. 2024c）的更简替代——后两者已知能诱导多样性。但鉴于种子任意选取、与训练点无语义关系，它们竟能提供任何有意义随机性，仍出人意料。我们目前尚不理解。

K

K

Thus, we speculatively put forth two more arguments. First, is a representational one. Fixing a random seed upfront may help the model flesh out (i.e., compute the tokens of) one thought per sample, as against maintaining a running set of multiple thoughts and computing a distribution over all their tokens at each step. A similar point is made in a concurrent position paper (Jahrens & Martinetz 2025). The second argument is specific to next-token prediction on open-ended planning tasks: fixing a random seed upfront may help the model co-ordinate multiple interlocking random decisions in advance rather than deciding them on the fly.

因此我们试探性地提出另外两点。第一是表示层面的：事先固定随机种子可能帮助模型为每个样本充实（即计算词元）一个想法，而不是维护多个想法的运行集合并在每步对它们的全部词元计算分布。同期立场论文（Jahrens & Martinetz 2025）也提出类似观点。第二点专属于开放式规划任务上的下一词预测：事先固定随机种子可能帮助模型预先协调多个环环相扣的随机决策，而非即时决定。

Finally, there are also optimization aspects of how seed-conditioning works that we do not understand (see §B.1). Regardless, it remains to be seen whether seed-conditioning is useful in tasks beyond the minimal ones we design.

最后，种子条件化如何工作还有我们不理解的优化侧面（见 §B.1）。无论如何，它在超出我们设计的极简任务之外是否有用，仍待观察。

5.2 Effects of reasoning-enhancing methods.

5.2 增强推理方法的影响

Our argument is limited to learning open-ended tasks in a supervised manner. While we do not comment on how well other approaches like RL (DeepSeek-AI 2025), chain-of-thought (CoT; Wei et al. 2022), and scaling test-time compute (OpenAI 2024) would fare, we remark that these methods are designed to enhance the quality of a single example. It is unclear how to design them to maximize originality against a training set, and diversity over multiple responses. Furthermore, there is a profound question as to whether merely spelling out a model’s thought in token space can be an efficient way to search and maximize diversity. This may require enumerating all possible candidates by trial and error, an impossible feat when the search space is large.

我们的论证限于以监督方式学习开放式任务。虽不评论 RL（DeepSeek-AI 2025）、思维链（CoT；Wei et al. 2022）与扩展测试时算力（OpenAI 2024）等其他方法表现如何，但指出这些方法旨在提升单个样例的质量。如何设计它们以相对训练集最大化原创性、并在多份回复上最大化多样性，尚不清楚。此外有一个深刻问题：仅把模型想法写在词元空间，是否是搜索并最大化多样性的高效方式。当搜索空间大时，这可能要求通过试错枚举所有候选——几乎不可能。

We present more discussions in §B.

更多讨论见 §B。

6 Limitations

6 局限

We enumerate in detail the limitations of our work in terms of our experimental conclusions and in terms of our general approach to an abstraction of creativity.

我们从实验结论与对创造力抽象的一般进路两方面，详细列举工作局限。

6.1 Limitations of our experimental conclusions

6.1 实验结论的局限

There may be many ways to improve upon next-token prediction for a minimal task. Unfortunately, success here does not necessarily guarantee success on more complex tasks. Conversely, minimal tasks are more valuable as a failure base case: failure here guarantees failure in more complex tasks.

对极简任务改进下一词预测的方式可能很多。可惜此处的成功未必保证更复杂任务上的成功。反过来，极简任务作为失败基线更有价值：此处失败保证更复杂任务也会失败。

Our examples do not preclude the existence of tasks where next-token prediction will outperform multi-token prediction; multi-token prediction is simply a more general-purpose objective suitable to lookahead tasks.

我们的例子不排除存在下一词预测优于多词元预测的任务；多词元预测只是更适合前瞻任务的通用目标。

The teacherless multi-token prediction technique we explore as an alternative is generally harder to optimize than next-token prediction, especially for smaller models.

作为替代探索的无教师多词元预测，一般比下一词预测更难优化，尤其对小模型。

Even if multi-token approaches outperform next-token prediction relatively, in some of our simple tasks, all algorithms are far from delivering a sufficiently diverse model.

即便多词元相对优于下一词，在我们某些简单任务上，所有算法距离交付足够多样的模型仍很远。

Although our tasks are minimal, we note that there is a certain range of hyperparameters (e.g., high degree or edge count) beyond which the models can struggle to learn them. We find that Triangle Discovery in particular is a challenging task, especially for smaller models. We also note that the models are curiously sensitive to the way the edges are formatted (see §F.4).

尽管任务极简，我们注意到存在一定超参范围（如高度数或边数）之外模型会学得很吃力。尤其三角形发现对小模型具有挑战。模型对边的格式化方式也出奇敏感（见 §F.4）。

While seed-conditioning appears to be a competitive alternative to temperature sampling, there are caveats here: (a) In our GPT-2 (86M) experiments for Sibling Discovery and Triangle Discovery, the gap between seed-conditioning and temperature-sampling is contingent on top-K sampling (which is turned on by default in the library as it is standard practice in real-world applications). Although this does not seem to be a requirement in our other settings (and is immaterial for the other datasets which have a smaller vocabulary), it is not clear how crucial this factor would be for demonstrating the same gap in real settings. (b) It is not clear how well this idea may generalize to realistic data. (c) Unfortunately, seed-conditioning requires special training of the model, which is not as appealing as the far simpler temperature sampling.

虽种子条件化看似可与温度采样竞争，仍有附带条件：(a) 在 GPT-2（86M）的兄弟姐妹发现与三角形发现实验中，种子相对温度的差距依赖于 top-K 采样（库默认开启，因真实应用中常见）。虽其他设定似乎不要求（且对词表更小的其他数据集无关紧要），但在真实设定中要展示同样差距时该因素有多关键，尚不清楚。(b) 该想法对现实数据泛化如何也不清。(c) 遗憾的是种子条件化需要特殊训练，不如温度采样简单诱人。

6.2 Our approach to creativity

6.2 我们对创造力的进路

Below, we enumerate some important limitations of our approach towards building abstract and minimal models of creative tasks.

以下列举我们构建创造任务抽象极简模型时的若干重要局限。

The skills we capture in our tasks are only (a subset of) computational skills necessary for creativity; these are far from being sufficient .

任务捕捉的技能只是创造力 必要 计算技能的（子集）；远非 充分 。

The type of algorithmic tasks we study capture only a tiny subset of creative tasks that fall under the taxonomy in Boden 2003. There is yet another class called transformative creativity that we do not look at, and also other important taxonomies such as the Big-C/little-c creativity (Csikszentmihalyi 1996). Big-C Creativity corresponds breakthroughs and world-changing ideas; what we focus on is adjacent to a class of little-c creativity tasks. Relatedly, many real-world creative tasks appear to be “out-of-distribution” in nature, which we do not capture.

所研究的算法任务类型只覆盖 Boden 2003 分类下创造任务的极小子集。还有一类 变革 创造我们未涉及，以及其他重要分类如大 C / 小 c 创造力（Csikszentmihalyi 1996）。大 C 对应突破与改变世界的想法；我们关注的邻近小 c 创造任务。相关地，许多真实创造任务呈现「分布外」性质，我们未刻画。

Real-world creative tasks also apply over much larger context length and require drawing connections from a significantly larger memory (literally, the set of all things a human may know about). Our algorithmic tasks are tiny in comparison (although deliberately so).

真实创造任务也作用在长得多的上下文上，并需从显著更大的记忆（字面上一人类可能知道的一切）中建立联系。我们的算法任务相比之下极小（虽是有意为之）。

Our empirical measure of creativity for algorithmic tasks is only a computationally-efficient proxy. Achieving an absolute high algorithmic creativity score does not imply a complete coverage of the space.

对算法任务的经验创造力度量只是计算高效的代理。算法创造力分数绝对高，并不意味着完全覆盖空间。

As stated earlier, we study abstract tasks that are inspired by the computations involved in creative tasks. Our study is not intended to capture the subjective, social, cultural and personal values integral to many creative tasks.

如前所述，我们研究受创造任务所涉计算启发的抽象任务。研究无意捕捉许多创造任务中不可或缺的主观、社会、文化与个人价值。

7 Related Work

7 相关工作

\approx 100m

开放式算法任务。 与我们直接相关的是 Khona et al. 2024；Allen-Zhu & Li 2023b，他们在开放式算法任务上研究下一词训练模型的多样性。Khona et al. 2024 展示知识图路径连通性在温度缩放下的多样性-准确率权衡。我们展示该权衡如何在替代训练方法下改善。Allen-Zhu & Li 2023b 经验表明，在「无限」数据体制（约 1 亿词元）下下一词预测器 能够 学会合成困难 CFG。除兄弟姐妹发现外，我们的数据不是 CFG。我们的否定结果不与其矛盾，因我们展示的是较小数据体制下 NTP 的次优性。工作还把上述研究延伸到更极简、仅需 2 跳前瞻的任务上的局限。另有在非开放图算法任务上研究 Transformer 的工作，见 §J。

Diversity in generative models. Generative diversity has long been a major goal, at least until the revolution in (deterministic) reasoning of language models. Much work has gone into concerns such as mode collapse (Che et al. 2017) or posterior collapse (Bowman et al. 2016) and memorization (Carlini et al. 2020; Carlini et al. 2023; Nasr et al. 2023). Temperature sampling has been known to have weak correlation with creativity, often inadvertently introducing incoherence (Peeperkorn et al. 2024; Chen & Ding 2023; Chung et al. 2023). Our results on seed-conditioning are also reminiscent of a line of work on reinforcement learning (RL) showing that adding noises to the policy model parameters enables more efficient exploration than directly adding noises to the output space (Plappert et al. 2017; Fortunato et al. 2017). We defer discussion of theoretical studies of diversity and memorization and with empirical studies on natural-language creativity in §J.

生成模型中的多样性。 生成多样性长期是重要目标，至少直到语言模型（确定性）推理革命之前。大量工作关注模式崩塌（Che et al. 2017）或后验崩塌（Bowman et al. 2016）以及记忆（Carlini et al. 2020；Carlini et al. 2023；Nasr et al. 2023）。已知温度采样与创造力相关弱，常无意引入不连贯（Peeperkorn et al. 2024；Chen & Ding 2023；Chung et al. 2023）。种子条件化结果也让人联想到强化学习（RL）中向策略模型参数加噪比直接向输出空间加噪更能高效探索的工作线（Plappert et al. 2017；Fortunato et al. 2017）。多样性与记忆的理论研究及自然语言创造力经验研究的讨论延至 §J。

Going beyond next-token prediction (NTP). There has been a recent emerging discussion surrounding the role of NTP as foundational piece in developing intelligent models. On the critical side, arguments have been made about the inference-time issues with auto-regression (Dziri et al. 2024; LeCun 2024; Kääriäinen 2006; Ross & Bagnell 2010). Others have reported the planning and arithmetic limitations of next-token trained models (McCoy et al. 2023; Momennejad et al. 2023; Valmeekam et al. 2023a; Valmeekam et al. 2023b; Valmeekam et al. 2023c; Bachmann & Nagarajan 2024) where the goal is accuracy, not diversity. As for diffusion as an alternative to NTP, our findings parallel that of Ye et al. 2024 who show that their variant of diffusion is able to solve the challenging path-star task of 6. We provide references to more lines of multi-token prediction work in §J. There are also other Transformer failures such as the reversal curse (Allen-Zhu & Li 2023a) or shortcut-learning (Dziri et al. 2024; Zhang et al. 2023; Liu et al. 2023; Young & You 2022; Lai et al. 2021; Ranaldi & Zanzotto 2023), however these are out-of-distribution failures; the sub-optimality we show is in-distribution, like in 6.

超越下一词预测（NTP）。 围绕 NTP 作为发展智能模型基础构件的角色，近期讨论兴起。批评方面，有人论证自回归的推理时问题（Dziri et al. 2024；LeCun 2024；Kääriäinen 2006；Ross & Bagnell 2010）。另有报告下一词训练模型在规划与算术上的局限（McCoy et al. 2023；Momennejad et al. 2023；Valmeekam et al. 2023a；Valmeekam et al. 2023b；Valmeekam et al. 2023c；Bachmann & Nagarajan 2024），其目标是准确率而非多样性。作为 NTP 替代的扩散，我们的发现平行于 Ye et al. 2024：其扩散变体能解决 6 的困难路径星任务。更多多词元预测工作线见 §J。另有反转诅咒（Allen-Zhu & Li 2023a）或捷径学习（Dziri et al. 2024；Zhang et al. 2023；Liu et al. 2023；Young & You 2022；Lai et al. 2021；Ranaldi & Zanzotto 2023）等 Transformer 失败，但那些是分布外失败；我们展示的次优是分布内的，与 6 类似。

Injecting noise into a Transformer. Most related to seed-conditioning is DeSalvo et al. 2024 who induce diversity by varying a soft -prompt learned using a reconstruction loss. Our approach requires no modification to the architecture or the loss; however, we train the whole model, which is more expensive. A concurrent position paper (Jahrens & Martinetz 2025) conceptually suggests injecting noise with the same motivation as us. The benefits of seed-conditioning may also be related to the fact that varying the wording in a prompt is known to induce diverse outputs (Li et al. 2023; Lau et al. 2024; Naik et al. 2024). Various works also inject noise into a Transformer, in a different form (e.g., inducing Gaussian noise), and for a different function such as quality, robustness (Hua et al. 2022; Jain et al. 2024) or efficiency (Wang et al. 2024c).

向 Transformer 注入噪声。 与种子条件化最相关的是 DeSalvo et al. 2024：通过变化用重构损失学到的 软 提示诱导多样性。我们的方法无需改架构或损失；但训练整个模型，更昂贵。同期立场论文（Jahrens & Martinetz 2025）概念上建议以与我们相同的动机注入噪声。种子条件化的益处也可能与变换提示措辞已知能诱导多样输出有关（Li et al. 2023；Lau et al. 2024；Naik et al. 2024）。另有工作以不同形式（如高斯噪声）向 Transformer 注入噪声，服务于质量、稳健性（Hua et al. 2022；Jain et al. 2024）或效率（Wang et al. 2024c）等不同功能。

8 Conclusions

8 结论

This work provides a minimal test-bed of tasks abstracting distinct modes of creativity. While these tasks are admittedly an extreme caricaturization of real-world tasks, they enable us to quantify otherwise elusive metrics like originality and diversity. They also enable us to control and investigate distinct parts of the current apparatus for language modeling (next-token learning and softmax-based temperature sampling) and advocate for alternatives (multi-token learning and seed-conditioning). The surprising effectiveness of seed-conditioning raises various open questions (§B.1). There are also other profound questions as to whether reasoning-enhancing methods like RL and CoT are optimal for enhancing open-ended diversity and originality (§5.2). Overall, we hope our work inspires discussion in the various directions of multi-token prediction, creativity and planning.

本文提供抽象不同创造模式的极简任务试验台。虽这些任务无疑是对真实任务的极端漫画化，却使我们能量化原本难以捉摸的原创性与多样性指标。它们也使我们能控制并考察当前语言建模装置的不同部分（下一词学习与基于 softmax 的温度采样），并倡导替代方案（多词元学习与种子条件化）。种子条件化出人意料的有效性引出诸多开放问题（§B.1）。另有深刻问题：RL 与 CoT 等增强推理方法对提升开放式多样性与原创性是否最优（§5.2）。总体希望本文激发多词元预测、创造力与规划等方向的讨论。

9 Impact Statement

9 影响声明

This paper presents work whose goal is to advance the field of Machine Learning through the study of simple algorithmic tasks inspired by creativity. There are many potential societal consequences of our work — especially if one applies AI to real-world creative endeavors — none which we feel must be specifically highlighted in our focused algorithmic study.

本文通过研究受创造力启发的简单算法任务推进机器学习。工作可能有多种社会后果——尤其若将 AI 用于真实创造活动——但在我们聚焦的算法研究中，我们认为无需特别强调其中哪一项。

References

参考文献

Narayan et al. (2018) Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, 2018. URL https://arxiv.org/abs/1808.08745 . Nasr et al. (2023) Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models. ArXiv , 2023. Nolte et al. (2024) Nolte, N., Kitouni, O., Williams, A., Rabbat, M., and Ibrahim, M. Transformers can navigate mazes with multi-step prediction. CoRR , abs/2412.05117, 2024. doi: 10.48550/ARXIV.2412.05117 . URL https://doi.org/10.48550/arXiv.2412.05117 . OpenAI (2024) OpenAI. Openai o1 system card. ArXiv , 2024. Padmakumar & He (2024) Padmakumar, V. and He, H. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=Feiz5HtCD0 . Pannatier et al. (2024) Pannatier, A., Courdier, E., and Fleuret, F. σ \sigma -gpts: A new approach to autoregressive models. In Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2024, Vilnius, Lithuania, September 9-13, 2024, Proceedings, Part VII , volume 14947 of Lecture Notes in Computer Science , pp. 143–159. Springer, 2024. Peeperkorn et al. (2024) Peeperkorn, M., Kouwenhoven, T., Brown, D., and Jordanous, A. Is temperature the creativity parameter of large language models? abs/2405.00492, 2024. Plappert et al. (2017) Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M. Parameter space noise for exploration. ArXiv , abs/1706.01905, 2017. URL https://api.semanticscholar.org/CorpusID:2971655 . Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. Ranaldi & Zanzotto (2023) Ranaldi, L. and Zanzotto, F. M. Hans, are you clever? clever hans effect analysis of neural systems, 2023.

Romera-Paredes et al. (2024) Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., Kohli, P., and Fawzi, A. Mathematical discoveries from program search with large language models. Nat. , 625(7995), 2024. Ross & Bagnell (2010) Ross, S. and Bagnell, D. Efficient reductions for imitation learning. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010 , volume 9 of JMLR Proceedings , 2010. Runco & Jaeger (2012) Runco, M. A. and Jaeger, G. J. The standard definition of creativity. Creativity Research Journal , 24(1):92-96, 2012. Sanford et al. (2024) Sanford, C., Fatemi, B., Hall, E., Tsitsulin, A., Kazemi, S. M., Halcrow, J., Perozzi, B., and Mirrokni, V. Understanding transformer reasoning capabilities via graph algorithms. abs/2405.18512, 2024. Saparov et al. (2024) Saparov, A., Pawar, S., Pimpalgaonkar, S., Joshi, N., Pang, R. Y., Padmakumar, V., Kazemi, S. M., Kim, N., and He, H. Transformers struggle to learn to search, 2024. URL https://arxiv.org/abs/2412.04703 . (96) Schnitzler, J., Ho, X., Huang, J., Boudin, F., Sugawara, S., and Aizawa, A. Morehopqa: More than multi-hop reasoning. abs/2406.13397. doi: 10.48550/ARXIV.2406.13397 . Shannon (1948) Shannon, C. E. A mathematical theory of communication. The Bell System Technical Journal , 27(3):379-423, 1948. Shannon (1951) Shannon, C. E. Prediction and entropy of printed english. The Bell System Technical Journal , 30(1):50-64, 1951. Shlegeris et al. (2022) Shlegeris, B., Roger, F., Chan, L., and McLean, E. Language models are better than humans at next-token prediction. arXiv preprint arXiv:2212.11281 , 2022. Si et al. (2024) Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? A large-scale human study with 100+ NLP researchers. 2024.

Zhong et al. (2024) Zhong, S., Huang, Z., Gao, S., Wen, W., Lin, L., Zitnik, M., and Zhou, P. Let’s think outside the box: Exploring leap-of-thought in large language models with creative humor generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024 , 2024. Zhu et al. (2018) Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models. The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , 2018.

Appendix A Transformer Training Objectives

附录 A Transformer 训练目标

\texttt{LM}_{\theta}

\texttt{LM}_{\theta}

\displaystyle\mathcal{J}_{\texttt{next-token}}(\theta) \\ \displaystyle=\mathbb{E}_{\mathcal{D}}\Big[\sum_{i=1}^{L_{\texttt{resp}}}\log{\texttt{LM}}_{\theta}\left(\hat{r}_{i}={r}_{i};\bm{p},\bm{r}_{<i}\right)\Big] \tag{2}

\bm{\$}

\bm{\$}

\displaystyle\mathcal{J}_{\texttt{multi-token}}(\theta) \\ \displaystyle=\mathbb{E}_{\mathcal{D}}\Big[\sum_{i=1}^{L_{\texttt{resp}}}\log{\texttt{LM}}_{\theta}\left(\hat{r}_{i}={r}_{i};\bm{p},\bm{\$}_{<i}\right)\Big] \tag{3}

Appendix B Further discussion

附录 B 进一步讨论

B.1 Style of noise-injection

B.1 噪声注入方式

Our technique of injecting noise into the model is somewhat different from how noise is introduced in traditional VAEs (Kingma & Welling 2014) or GANs (Goodfellow et al. 2020), and this difference is worth noticing. In traditional approaches, although the model learns a noise-output mapping, this mapping is enforced only at a distribution level i.e., the distribution of noise vectors must map to a distribution of real vectors. However, in our approach we arbitrarily enforce what noise vector goes to what real datapoint, at a pointwise level. This raises the open questions of why seed-conditioning works in the first place — surprisingly, without breaking optimization or generalization — and whether there is a way to enforce it at distribution-level, and whether that can provide even greater improvements.

我们向模型注入噪声的技术，与传统 VAE（Kingma & Welling 2014）或 GAN（Goodfellow et al. 2020）引入噪声的方式有所不同，值得注意。传统方法中，虽模型学习噪声-输出映射，但该映射仅在分布层面强制，即噪声向量分布须映到真实向量分布。我们的方法则在点对点层面任意强制哪个噪声向量对应哪个真实数据点。这引出开放问题：种子条件化为何以出人意料地不破坏优化或泛化的方式首先有效；是否能在分布层面强制；以及那样能否带来更大提升。

B.2 In-weights vs in-context graphs for combinational creativity

B.2 组合创造的权重内图与上下文内图

Combinational creativity requires searching through known entities. In abstracting this, there is an interesting choice to be made as to whether the relevant search space is retrieved and spelled out in-context or whether it remains in-weights (like in Sibling Discovery and Triangle Discovery). We argue that the in-context version does not capture the creative skills required in many real-world tasks. For instance, discovering a fresh and surprising analogy necessitates noticing similarities from sufficiently distinct parts of one’s vast, rich space of memory. Thus, the core challenge here lies in retrieving from the entirety of one’s memory. If one were to faithfully simulate this an in-context version of this in a model, one would have to provide the entirety of the model’s pretraining data in context.

组合创造需要在已知实体中搜索。抽象时有一有趣选择：相关搜索空间是检索并写在上下文中，还是留在权重内（如兄弟姐妹发现与三角形发现）。我们认为上下文版本捕捉不了许多真实任务所需的创造技能。例如，发现新鲜惊奇的类比，需要从广阔丰富记忆中足够不同的部分察觉相似。因此核心挑战在于从记忆整体检索。若要在模型中忠实地用上下文版本模拟，就必须把模型全部预训练数据放进上下文。

B.3 Examples of Triangle Discovery

B.3 三角形发现的例子

Although we presented this task as a more complex, higher-order counterpart to Sibling Discovery, we retrospectively identify some real-world examples that resemble the higher-order search skill involved in this task.

虽将该任务表述为兄弟姐妹发现更复杂的高阶对应物，我们事后识别出一些类似该任务所涉高阶搜索技能的真实例子。

Discovering contradictions: Consider identifying non-trivial contradictions within (a large body) of knowledge (like a legal system, or a proof based on many lemmas, or the literature spanning many papers in a certain field). This may require identifying two or more facts that together result in an implication that contradicts another fact.

发现矛盾 ： 考虑在（大块）知识中识别非平凡矛盾（如法律体系、基于诸多引理的证明，或某领域跨多篇论文的文献）。这可能需要识别两个或多个事实，它们合在一起推出与另一事实矛盾的含义。

Discovering feedback loops: Fields like biology, ecology, climate science and economics may involve discovering non-trivial feedback loops. Unlike feedback loops where two events encourage each other, a non-trivial loop would be one where an Event A encourages Event B, that in turn encourages Event C that in turn encourages Event A.

发现反馈环 ： 生物、生态、气候科学与经济学等领域可能涉及发现非平凡反馈环。不同于两事件相互促进的反馈，非平凡环可以是事件 A 促进事件 B，B 促进事件 C，C 又促进事件 A。

\texttt{sound}_{1}

\texttt{sound}_{1}

\{\texttt{Ninja},\texttt{(Some assassin in Japan)}_{1},\texttt{(Some assassin in Japan)}_{2}\}

\{\texttt{Ninja},\texttt{(Some assassin in Japan)}_{1},\texttt{(Some assassin in Japan)}_{2}\}

B.4 Further evidence of our argument in §2.6

B.4 对 §2.6 论证的进一步证据

Below we provide two more pieces of evidence affirming the failure mechanism of next-token prediction outlined in §2.6.

以下再提供两则证据，肯定 §2.6 概述的下一词预测失败机制。

Improved algorithmic creativity is not due to some form of capacity control. While §2.6 argues that multi-token prediction should help creativity by providing critical lookahead capabilities, it is also possible that it simply acts as a form of capacity control that prevents memorization. We rule this out in Fig 8: even as memorization computed on unseen seeds is controlled, the multi-token model perfectly reproduces the training data on seen seeds. We term this seed-memorization . An exact equivalence of this phenomenon was noticed in GANs in Nagarajan et al. 2018, where the generator can be trained on specific latent vectors to memorize the mapping on those, and yet produce fresh samples outside of those latent vectors.

算法创造力提升并非某种容量控制 。 §2.6 论证多词元预测应通过提供关键前瞻能力帮助创造力，但也可能只是防止记忆的容量控制。图 8 排除这一点：即便在 未见 种子上计算的记忆受控，多词元模型仍在 已见 种子上完美复现训练数据。我们称之为 种子记忆 。Nagarajan et al. 2018 在 GAN 中注意到完全等价现象：生成器可在特定潜向量上训练以记忆那些映射，却仍在那些潜向量之外产出新鲜样本。

Figure 8: Even if multi-token prediction reduces memorization (on unseen seeds), it has enough capacity to memorize training data on the seen seeds (denoted by seed-memorization). Note that the best algorithmic creativity for NTP and MTP are achieved at step 10k and 40k, respectively, which are the checkpoints we used to report metrics in Fig 4.

图 8： 即便多词元预测降低（未见种子上的）记忆，它仍有足够容量在已见种子上记忆训练数据（记为种子记忆）。注意 NTP 与 MTP 的最佳算法创造力分别在第 10k 与 40k 步达到，即我们用来报告图 4 指标的检查点 。

Effect of token reordering. The implication of our argument in §2.6 is that next-token learning would benefit from reversing the token ordering of the Sibling Discovery task (i.e., parent appears before siblings). Indeed, we find this to be the case in Fig 12 and Fig 22. Interestingly, we find that the reverse-trained NTP model is still far from the original multi-token teacherless model. More surprisingly, a teacherless model trained on the reversed data, achieves even higher algorithmic creativity of all training methods here. Note that in all other datasets, no reordering of the tokens should make any change to the training.

词元重排的影响 。 §2.6 论证的含义是：下一词学习会受益于反转兄弟姐妹发现的词元顺序（即父出现在兄弟姐妹之前）。的确，图 12 与图 22 如此。有趣的是，反转训练的 NTP 模型仍远逊于原始多词元无教师模型。更惊人的是，在反转数据上训练的无教师模型，在此处所有训练方法中算法创造力最高。注意在所有其他数据集上，任何词元重排都不应改变训练。

(a) Sibling Discovery

(a) 兄弟姐妹发现

(a) Circle Construction

(a) 圆构造

Appendix C Description of datasets

附录 C 数据集描述

C.1 Datasets inspired by combinational creativity

C.1 受组合创造启发的数据集

\mathcal{G}

\mathcal{G}

m=|\mathscr{V}|

m=|\mathscr{V}|

|\mathscr{V}|=5

|\mathscr{V}|=5

\mathcal{G}=({V},E)

\mathcal{G}=({V},E)

\leq 1.2\cdot\texttt{deg}

\leq 1.2\cdot\texttt{deg}

1/3

(u,v,w)

|{V}|=999

|{V}|=999

C.2 Datasets inspired by exploratory creativity

C.2 受探索创造启发的数据集

N

N

N=9,M=15

N=9,M=15

E

E

N=9,M=15

N=9,M=15

Appendix D Further experimental details

附录 D 进一步实验细节

4

4

0

0

5\times 10^{-4}

Hyperparameter	Sibling Discovery	Triangle Discovery	Circle Construction	Line Construction
Max. Learning Rate	$5\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-5}$
Model Seq. Len.	$32$	$32$	$2048$	$2048$
Training steps	$7500$	$10k$	$15k$	$15k$
Training size	$50k$	$15k$	$10k$	$10k$
Weight given to multi-token obj.	$0.5$	$0.5$	$0.75$	$0.75$

5\times 10^{-4}

超参	兄弟姐妹发现	三角形发现	圆构造	线构造
Max. Learning Rate	$5\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-5}$
Model Seq. Len.	$32$	$32$	$2048$	$2048$
Training steps	$7500$	$10k$	$15k$	$15k$
Training size	$50k$	$15k$	$10k$	$10k$
Weight given to multi-token obj.	$0.5$	$0.5$	$0.75$	$0.75$

10^{-4}

10^{-4}

10^{-4}

10^{-4}

Appendix E Sensitivity analyses for Gemma v1 (2B)

附录 E Gemma v1（2B）的敏感性分析

In this section, we report that our observations are robust to the choice of various hyper-parameters. First, we present a series of plots for the Gemma v1 (2B) model; each group of plots reports varying one hyperparameter for all the datasets. Fig 11 for train set size, Fig 12 for task complexity, Fig 13 for the weight given to the multi-token objective (and Fig 14 correspondingly for memorization), Fig 15 for learning rates, Fig 16 for number of training steps and Fig 17 for batch size. In §E.1, we report analyses for varying sampling conditions. It is worth noting that the occasional exceptions to our trends generally come from Line Construction, suggesting that this task is most friendly towards next-token prediction of the four we study.

本节报告观察对各种超参选择的稳健性。首先给出 Gemma v1（2B）的系列图；每组图对全部数据集变化一个超参。图 11 为训练集大小，图 12 为任务复杂度，图 13 为多词元目标权重（图 14 对应记忆），图 15 为学习率，图 16 为训练步数，图 17 为批大小。§E.1 报告不同采样条件的分析。值得注意：趋势的偶发例外一般来自线构造，暗示四项中它最对下一词预测友好。

Note on task-complexity. In Fig 12, we report robustness of our results to variations in the task complexity (e.g., degree, path length etc.,). Note that the variations we have explored are within reasonable factors. If we vastly increase certain factors (e.g., increase the degree of the vertices), we expect learning to become either highly trivial or non-trivial (see §C for some reasoning). Besides, as discussed in the main paper, teacherless training is a hard objective to optimize especially for smaller models; thus, we expect increasing the task complexity beyond a point to hurt the teacherless model for a fixed model size (crucially, for optimization reasons, not generalization reasons).

关于任务复杂度 。 图 12 报告结果对任务复杂度变化（如度数、路径长度等）的稳健性。注意所探索变化在合理因子内。若大幅增加某些因子（如提高顶点度数），预期学习会变得极平凡或极非平凡（理由见 §C）。此外如正文所述，无教师训练尤其对小模型难优化；因此预期超过某点后提高任务复杂度会伤害固定规模下的无教师模型（关键是优化原因，而非泛化原因）。

Figure 11: Training size and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token prediction across various training set sizes. Note though that, in our examples, we except the gap to diminish eventually with sufficiently many training datapoints (this is unlike the failure of next-token prediction in 6).

图 11： Gemma v1（2B）的训练规模与算法创造力：在多种训练集大小下，多词元预测均提升算法创造力。但注意在我们的例子中，预期差距最终会随足够多训练点缩小（这不同于 6 中下一词预测的失败） 。

Figure 12: Task complexity and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token prediction across (reasonable) variations in the dataset parameters (as described in §C).

图 12： Gemma v1（2B）的任务复杂度与算法创造力：在数据参数的（合理）变化下（见 §C），多词元预测均提升算法创造力 。

Figure 13: Weight given to multi-token objective and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token prediction across various weights given to the multi-token component of the objective, barring some deviations for Line Construction.

图 13： Gemma v1（2B）的多词元目标权重与算法创造力：在多词元分量的多种权重下，多词元预测均提升算法创造力，线构造有个别偏离 。

Figure 14: Weight given to multi-token objective and memorization score for Gemma v1 (2B): Memorization reduces under multi-token prediction across various weights given to the multi-token component of the objective.

图 14： Gemma v1（2B）的多词元目标权重与记忆分数：在多词元分量的多种权重下，多词元预测均降低记忆 。

Figure 15: Learning rate and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token prediction across various learning rates.

图 15： Gemma v1（2B）的学习率与算法创造力：在多种学习率下，多词元预测均提升算法创造力 。

Figure 16: Training steps and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity under multi-token prediction across lengths of training.

图 16： Gemma v1（2B）的训练步数与算法创造力：跨不同训练长度，多词元预测下的算法创造力 。

Figure 17: Batch size and algorithmic creativity for Gemma v1 (2B): Algorithmic creativity increases under multi-token prediction across various batch sizes. Note that here batch size is effectively proportional to the model sequence length, since we pack multiple finetuning examples into the sequence.

图 17： Gemma v1（2B）的批大小与算法创造力：在多种批大小下，多词元预测均提升算法创造力。注意此处批大小实际与模型序列长度成正比，因我们把多个微调样例打包进序列 。

E.1 Varying sampling methods

E.1 变化采样方法

Fig 18, Fig 19, and Fig 20 report creativity, memorization and coherence (i.e., fraction of generated strings that are coherent) for various sampling methods (greedy decoding and nucleus sampling) with various prefix conditionings (namely, null, pause and seed).

图 18、图 19 与图 20 报告在各种采样方法（贪心解码与核采样）与各种前缀条件化（空、停顿与种子）下的创造力、记忆与连贯性（即生成串中连贯的比例）。

Figure 18: Algorithmic creativity under various sampling conditions for Gemma v1 (2B): Across all conditions, and in almost all datasets (with a few exceptions in Line Construction), multi-token prediction improves creativity. Furthermore, seed-conditioning achieves best algorithmic creativity, with a longer seed helping more.

图 18： Gemma v1（2B）在各种采样条件下的算法创造力：几乎全部条件与数据集上（线构造有少量例外），多词元预测提升创造力。此外种子条件化达到最佳算法创造力，更长种子帮助更大 。

Figure 19: Memorization under various sampling conditions for Gemma v1 (2B): Barring a few conditions, the most prominent trend is that memorization reduces under multi-token prediction for various sampling conditions. Observe that the null and pause-conditioned models do produce some memorized output while their creativity was non-existent.

图 19： Gemma v1（2B）在各种采样条件下的记忆：除少数条件外，最突出趋势是多词元预测在各种采样条件下降低记忆。注意空与停顿条件化模型 确实 产生一些记忆输出，而其创造力却不存在 。

Figure 20: Coherence under various sampling conditions for Gemma v1 (2B): Surprisingly, coherence of all models is high or at least noticeable, across various sampling conditions. This suggests that the low algorithmic creativity of the null-conditioned models in the previous plots arises from model collapsing to single original point.

图 20： Gemma v1（2B）在各种采样条件下的连贯性：令人惊讶的是，所有模型在各种采样条件下连贯性都高或至少可察觉。这表明前图中空条件化模型算法创造力低，源于模型崩塌到单一原创点 。

Appendix F Additional experiments in SEDD (90M)vs. GPT-2 (86M)

附录 F SEDD（90M）对 GPT-2（86M）的额外实验

F.1 Ablation studies

F.1 消融研究

In this section, we first provide additional ablation studies for SEDD (90M)vs GPT-2 (86M)with different training and dataset settings (Fig 21 and Fig 22).

本节先给出 SEDD（90M）对 GPT-2（86M）在不同训练与数据设定下的额外消融（图 21 与图 22）。

Figure 21: Learning rates and algorithmic creativity for the SEDD (90M)model vs. GPT-2 (86M): MTP achieves higher algorithmic creativity than NTP when both are trained at their optimal learning rates.

图 21： SEDD（90M）对 GPT-2（86M）的学习率与算法创造力：当二者均在各自最优学习率训练时，MTP 的算法创造力高于 NTP 。

Figure 22: Task complexity and algorithmic creativity of SEDD (90M) model vs. GPT-2 (86M): MTP consistently outperforms NTP under varying task configurations, with some exceptions in the Line Constructionand Circle Constructiondatasets.

图 22： SEDD（90M）对 GPT-2（86M）的任务复杂度与算法创造力：在变化的任务配置下 MTP 持续优于 NTP，线构造与圆构造有个别例外 。

F.2 Effect of seed length

F.2 种子长度的影响

We provide an ablation study on the seed length for NTP vs MTP on the Sibling Discovery task (Fig 23 (left)). We see that longer seeds lead to higher algorithmic creativity.

我们在兄弟姐妹发现任务上对 NTP 对 MTP 做种子长度消融（图 23 左）。可见更长种子带来更高算法创造力。

Figure 23: GPT-2 (86M) Transformer achieves higher algorithmic creativity with longer seeds (top): We report algorithmic creativity with seeds of length 4 and 10, with both NTP and teacherless MTP. we do not see gains of seed-conditioning when it comes to diffusion training (right): We report algorithmic creativity of the SEDD (90M) model with or without seed-conditioning.

图 23： GPT-2（86M）Transformer 在更长种子下算法创造力更高（上）：报告长度 4 与 10 的种子下 NTP 与无教师 MTP 的算法创造力。扩散训练上看不到种子条件化增益（右）：报告 SEDD（90M）有无种子条件化的算法创造力 。

F.3 Seed-conditioning for diffusion training

F.3 扩散训练的种子条件化

In Fig 23 (right), we apply seed-conditioning to the SEDD (90M) on the Sibling Discovery task. Unlike what we see on transformers, we do not observe improved creativity when applying seed-conditioning to diffusion training.

图 23（右）在兄弟姐妹发现上对 SEDD（90M）施加种子条件化。与 Transformer 上所见不同，扩散训练施加种子条件化未见创造力提升。

F.4 Format sensitivity for Triangle Discovery

F.4 三角形发现的格式敏感性

Recall that our input format for Triangle Discovery follows the edge list representation of triangles (§C, Fig. 10). For instance, triangle ABC is represented as AB, BC, CA. This format explicitly lists the edges of the triangle, making it easier for the model to attend to edge-level patterns during learning.

回顾三角形发现的输入格式遵循三角形的 边列表表示 （§C，图 10）。例如三角形 ABC 表示为 AB, BC, CA。该格式显式列出三角形的边，便于模型在学习中关注边级模式。

We also experimented with an alternative node-based representation, where triangle ABC is represented more compactly as ABC, without making the edges explicit. We note in Fig 24 that the models are curiously sensitive to the way the triangles are formatted. Models trained on the node-based format perform equally badly with all training objectives, while the diffusion model outperforms NTP by a large margin with the edge list representation.

我们也试验了替代的 基于节点的表示 ：三角形 ABC 更紧凑地表示为 ABC，不显式给出边。图 24 显示模型对三角形格式化方式出奇敏感。在基于节点的格式上，所有训练目标下模型都同样差；而在边列表表示下，扩散模型大幅优于 NTP。

Figure 24: Sensitivity to formatting of the sequence in Triangle Discovery: We find that all our small models perform equally poorly with a node-wise representation of the input sequence, whereas there was a stark difference in performance with the edge-wise representation.

图 24： 三角形发现中序列格式的敏感性：发现我们所有小模型在节点式输入表示上同样差，而在边式表示上表现差异鲜明 。

Appendix G Additional experiments with medium-sized Transformer and SEDD

附录 G 中等规模 Transformer 与 SEDD 的额外实验

\sim

我们在更大模型规模（约 400M 参数）上复现 SEDD（90M）与 GPT-2（86M）实验。图 25 显示与较小模型规模（图 4）相似的趋势。

\sim

图 25： 在中等规模（约 400M）模型上，多词元扩散训练在四项开放式算法任务上提升式 1 的算法创造力（上） 。

Appendix H Decomposing creativity

附录 H 分解创造力

Through following experiments on the GPT-2 (86M) model in the Sibling Discovery task, we try to understand the dynamics between two important quantities that affect algorithmic creativity: diversity/duplication and originality/memorization

通过 GPT-2（86M）在兄弟姐妹发现上的下列实验，我们试图理解影响算法创造力的两个重要量之间的动态：多样性/重复与原创性/记忆。

H.1 Diversity score

H.1 多样性分数

Equation (1) defines our algorithmic creativity by rewarding samples that are both unique and novel. A higher score can be achieved either by enhancing diversity or by reducing memorization. In the following section, we examine this decomposition using the Sibling Discovery task. Formally, we define the diversity score as:

式 (1) 通过奖励既唯一又新颖的样本定义算法创造力。更高分数可通过增强多样性或降低记忆达到。下节用兄弟姐妹发现任务考察该分解。形式上定义多样性分数为：

\hat{\texttt{dv}}_{N}(T)=\frac{\texttt{uniq}(\{\bm{s}\in T|\texttt{coh}(\bm{s})\})}{|T|}. \tag{4}

We first demonstrate that creativity and diversity are not necessarily correlated, and next, that MTP particularly improves creativity (while achieving lower diversity than NTP). To show this, we report the algorithmic creativity and diversity scores along training in Fig 26. We see that for NTP, the diversity score keeps increasing and stays high, while algorithmic creativity increases in the first 10k steps and starts to decrease. For teacherless training, both scores increase throughout training. While the diversity of MTP is surprisingly lower than NTP throughout training, the creativity of MTP surpasses NTP at 20k steps.

我们先证明创造力与多样性未必相关，再证明 MTP 尤其提升创造力（同时多样性低于 NTP）。为此在图 26 报告训练过程中的算法创造力与多样性分数。可见对 NTP，多样性分数持续上升并保持高位，而算法创造力在前 10k 步上升后开始下降。对无教师训练，两分数在整个训练中都上升。虽 MTP 的多样性在整个训练中出人意料地低于 NTP，但 MTP 的创造力在 20k 步超越 NTP。

Figure 26: Algorithmic creativity and diversity are not necessarily correlated, exhibiting distinct dynamics: We find that NTP has a high diversity score through training, even higher than MTP. However, its algorithmic creativity reaches only a mediocre peak before descending, when MTP starts surpassing it.

图 26： 算法创造力与多样性未必相关，呈现不同动态：发现 NTP 在整个训练中多样性分数高，甚至高于 MTP。但其算法创造力只达到平庸峰值后下降，而 MTP 开始超越它 。

H.2 Decomposing algorithmic creativity as diversity and memorization

H.2 将算法创造力分解为多样性与记忆

Better creativity can be achieved either by enhancing diversity or by reducing memorization - we try to disentangle these factors in this section. In Fig 27, we plot the algorithmic creativity, diversity, and memorization scores at the checkpoint of best algorithmic creativity. We see that seed-conditioning contributes to higher diversity but does little to bring down memorization; however, teacherless training contributes to higher diversity and also to reducing memorization. In Fig 26, we see that the best creativity and best diversity are not achieved at the same checkpoint.

更好的创造力可通过增强多样性或降低记忆达到——本节试图拆开这些因素。图 27 在最佳算法创造力检查点绘制算法创造力、多样性与记忆分数。可见种子条件化贡献更高多样性，但对降低记忆帮助不大；而无教师训练既提高多样性也降低记忆。图 26 显示最佳创造力与最佳多样性并非同一检查点。

Figure 27: Decomposition of algorithmic creativity for GPT-2 (86M) in Sibling Discovery: We report algorithmic creativity, diversity and memorization at the checkpoint of best algorithmic creativity. We see that seed-conditioning contributes to higher diversity but does not help bring down memorization; teacherless training helps both diversity and in bringing down memorization.

图 27： GPT-2（86M）在兄弟姐妹发现上算法创造力的分解：在最佳算法创造力检查点报告算法创造力、多样性与记忆。可见种子条件化贡献更高多样性但不帮助降低记忆；无教师训练既帮助多样性也帮助降低记忆 。

Figure 28: Data scaling curve for algorithmic creativity and diversity: As we increase the training data (for a fixed underlying graph), the theoretically expected maximum algorithmic creativity decreases as expected, while the theoretically expected maximum diversity stays the same. NTP tails to achieve the theoretically expected algorithmic creativity, while MTP almost achieves the theoretically expected performance at scale.

图 28： 算法创造力与多样性的数据缩放曲线：在固定底层图上增加训练数据时，理论期望的最大算法创造力如预期下降，而理论期望的最大多样性保持不变。NTP 未能达到理论期望的算法创造力，而 MTP 在规模上几乎达到理论期望表现 。

H.3 Data scaling for algorithmic creativity

H.3 算法创造力的数据缩放

T

T

Appendix I Experiments on summarization

附录 I 摘要实验

0.05

0.05

32

Hyperparameter	XSUM	CNN/DailyMail
Batch Size	$32$	$32$
Max. Learning Rate	$5\times 10^{-5}$	$3\times 10^{-6}$
Warmup Steps	$338$	$124$
Training Steps	$7778$	$2486$
Training Size	$248906$	$79552$

32

超参	XSUM	CNN/DailyMail
Batch Size	$32$	$32$
Max. Learning Rate	$5\times 10^{-5}$	$3\times 10^{-6}$
Warmup Steps	$338$	$124$
Training Steps	$7778$	$2486$
Training Size	$248906$	$79552$

(0.5,0.5,0,0)

(0.5,0.5,0,0)

I.1 Additional graphs for effect of multi-token training

I.1 多词元训练效应的额外图

Fig 29 shows the diversity and quality graphs on the smaller-sized GPT-2 models on XSUM, and Fig 30 for CNN/DailyMail. While we consistently see improved quality from the multi-token model across the board, we don’t see an increased diversity for fixed Rouge scores anymore.

图 29 给出较小 GPT-2 模型在 XSUM 上的多样性与质量图，图 30 为 CNN/DailyMail。虽全面持续看到多词元模型质量提升，但固定 Rouge 下不再看到多样性增加。

Figure 29: Multi-Token Objective has no effect on diversity for smaller GPT models on XSUM.

图 29： 多词元目标对较小 GPT 模型在 XSUM 上的多样性无影响 。

Figure 30: Multi-Token Objective increases diversity for GPT-L and GPT-M but not for GPT-XL or GPT-S on CNN/DailyMail

图 30： 多词元目标提升 GPT-L 与 GPT-M 在 CNN/DailyMail 上的多样性，但对 GPT-XL 或 GPT-S 不然

I.2 Effect of seed-conditioning

I.2 种子条件化的影响

10

10

Figure 31: Seed-conditioning has no effect on diversity for GPT models on XSUM summarization with next-token prediction.

图 31： 种子条件化对 GPT 模型在 XSUM 摘要上下一词预测的多样性无影响 。

Figure 32: Seed-conditioning has no effect on diversity for GPT models on XSUM summarization with multi-token prediction.

图 32： 种子条件化对 GPT 模型在 XSUM 摘要上多词元预测的多样性无影响 。

Appendix J More related works

附录 J 更多相关工作

Empirical studies of creativity in LLMs. There is a long line of recent works that measure novelty and creativity of LLMs and LLM-assisted users. (Chakrabarty et al. 2024; Lu et al. 2024b) quantitatively evaluate and report that models vastly underperform under expert human evaluation against human writers. Zhang et al. 2024a argue that finetuning methods such as RLHF and DPO, are limited when applied to creative humor-generation tasks. Likewise models like GPT4 and Claude currently underperform top human contestants in generating humorous captions. In poetry, Walsh et al. argue that there are certain characterstic styles that ChatGPT restricts itself to. Even assisted-writing can reduce diversity (Padmakumar & He 2024) or produce bland writing (Mirowski et al. 2024). On the positive side, Si et al. 2024 report that LLMs surprisingly generate novel research ideas, although these are less feasible. Anderson et al. 2024 find that users tend to produce more divergent ideas when assisted by ChatGPT (although at a group level, ideas tend to homogenize). Another line of works (Wang et al. 2024a; Talmor et al. 2020; Zhong et al. 2024) has proposed algorithmic improvements involve creative leaps-of-thought for real-world tasks.

LLM 创造力的经验研究。 近期有一长串工作衡量 LLM 与 LLM 辅助用户的新颖性与创造力。(Chakrabarty et al. 2024；Lu et al. 2024b) 定量评估并报告：在专家人类评价下模型相对人类写作者大幅落后。Zhang et al. 2024a 论证 RLHF 与 DPO 等微调方法用于创造幽默生成任务时受限。同样，GPT-4 与 Claude 等模型目前在生成幽默字幕上不如顶尖人类选手。诗歌方面，Walsh et al. 认为 ChatGPT 局限于某些特征风格。即便辅助写作也可能降低多样性（Padmakumar & He 2024）或产出平淡文字（Mirowski et al. 2024）。积极一面，Si et al. 2024 报告 LLM 出人意料地能生成新颖研究想法，尽管可行性较低。Anderson et al. 2024 发现用户在 ChatGPT 辅助下倾向产出更多发散想法（但在群体层面想法趋于同质化）。另一工作线（Wang et al. 2024a；Talmor et al. 2020；Zhong et al. 2024）为真实任务提出涉及创造跃迁的算法改进。

Other studies have proposed benchmarks for evaluating creativity. AidanBench (McLaughlin et al. 2024) and NoveltyBench (Zhang et al. 2025) evaluate LMs on their ability to produce diverse and coherent responses by penalizing repetition across generations. However, they do not measure originality relative to training data, leaving open whether outputs are genuinely novel or simply unseen paraphrases/recombinations. Zhao et al. 2024 evaluate LM creativity using the Torrance Tests of Creative Thinking, a standard in human psychometrics. Another line of work such as Alchemy (Wang et al. 2021), IVRE (Xu et al. 2022), and DiscoveryWorld (Jansen et al. 2024) present simulations with hidden facts and rules, requiring LMs to explore, hypothesize, and test through interaction. While these simulations focus on pretrained models rather than examining how training shapes creative capabilities, they serve as valuable and realistic benchmarks for assessing the role of creativity in scientific discovery.

另有研究提出评估创造力的基准。AidanBench（McLaughlin et al. 2024）与 NoveltyBench（Zhang et al. 2025）通过惩罚跨生成重复，评估 LM 产出多样连贯回复的能力。但它们不衡量相对训练数据的原创性，输出究竟是真正新颖还是未见过的改写/重组仍开放。Zhao et al. 2024 用人类心理测量中的托兰斯创造性思维测验评估 LM 创造力。另一线工作如 Alchemy（Wang et al. 2021）、IVRE（Xu et al. 2022）与 DiscoveryWorld（Jansen et al. 2024）提供含隐藏事实与规则的仿真，要求 LM 通过交互探索、假设与检验。这些仿真聚焦预训练模型而非考察训练如何塑造创造能力，但仍是评估创造力在科学发现中角色的有价值且现实的基准。

Finally, we refer the reader to Franceschelli & Musolesi 2023 for a rigorous treatment of philosophical questions surrounding creativity in LLMs. We also a refer to Wang et al. 2024b for a theoretical treatment of how to formalize subjectivity in creativity.

最后请读者参阅 Franceschelli & Musolesi 2023，以严格处理围绕 LLM 创造力的哲学问题；以及 Wang et al. 2024b，以理论处理如何形式化创造力中的主观性。

The next-token prediction debate. In support of next-token prediction, there are arguments (Shannon 1948; Shannon 1951; Alabdulmohsin et al. 2024) that claim that language is captured by NTP with models even superceding humans (Shlegeris et al. 2022) at NTP. There are also theoretical results emphasizing the expressivity (Merrill & Sabharwal 2024; Feng et al. 2023) and learnability (Malach 2023; Wies et al. 2023) of autoregressive Transformers as long as there is a sufficiently long chain of thought.

下一词预测之争。 支持下一词预测方面，有论证（Shannon 1948；Shannon 1951；Alabdulmohsin et al. 2024）称语言由 NTP 捕捉，模型甚至在 NTP 上超越人类（Shlegeris et al. 2022）。也有理论结果强调：只要思维链足够长，自回归 Transformer 的表达力（Merrill & Sabharwal 2024；Feng et al. 2023）与可学性（Malach 2023；Wies et al. 2023）。

Multi-token training. While these methods employ diverse strategies, a common feature is their reliance on multi-token objectives that capture broader dependencies across entire sequences. Representative examples include teacherless training (Bachmann & Nagarajan 2024; Monea et al. 2023; Tschannen et al. 2023) and independent output heads or modules (Gloeckle et al. 2024; DeepSeek-AI et al. 2024) or inserting a lookahead attention (Du et al. 2023). Another line of research is discrete diffusion models (Hoogeboom et al. 2021; Austin et al. 2021; Gong et al. 2023; Lou et al. 2023), which avoid strict left-to-right factorization by iteratively refining an entire sequence at multiple positions. There are other models as well, such as energy-based models (Dawid & LeCun 2023) and non-autoregressive models or (Gu et al. 2018).

多词元训练。 这些方法策略多样，共同特征是依赖捕捉整段序列更广依赖的多词元目标。代表性例子包括无教师训练（Bachmann & Nagarajan 2024；Monea et al. 2023；Tschannen et al. 2023）、独立输出头或模块（Gloeckle et al. 2024；DeepSeek-AI et al. 2024），或插入前瞻注意力（Du et al. 2023）。另一研究线是离散扩散模型（Hoogeboom et al. 2021；Austin et al. 2021；Gong et al. 2023；Lou et al. 2023），通过在多位置迭代精炼整段序列避免严格从左到右分解。另有基于能量的模型（Dawid & LeCun 2023）与非自回归模型（Gu et al. 2018）等。

Transformers and graph algorithmic tasks. Graph tasks have been used to understand various limitations of Transformers in orthogonal settings. Bachmann & Nagarajan 2024; Saparov et al. 2024 report that Transformers are limited in terms of learning to search tasks on graphs, while Sanford et al. 2024 provide positive expressivity results for a range of algorithmic tasks that process an graph. These works differ from our study of combinational creativity since their graphs are provided in-context and the tasks have a unique answer. Other works (Schnitzler et al.; Yang et al. 2024a; Yang et al. 2024b) study multi-hop question answering on a knowledge graph; however, this does not require planning.

Transformer 与图算法任务。 图任务被用于理解 Transformer 在正交设定下的多种局限。Bachmann & Nagarajan 2024；Saparov et al. 2024 报告 Transformer 在学习图上的搜索任务方面受限，而 Sanford et al. 2024 为处理图的一系列算法任务给出正面表达力结果。这些工作与我们对组合创造的研究不同：其图在上下文给出，且任务有唯一答案。另有工作（Schnitzler et al.；Yang et al. 2024a；Yang et al. 2024b）研究知识图上的多跳问答；但这不需要规划。

Diversity of generative models. One line of work relevant to us in the history of generative models is RNN-based VAE for text data (Bowman et al. 2016). The motivation, like in our work, was to learn high-level semantic features rather than next-token features with the hope of producing more novel sentences. However, this suffered from posterior collapse, where the model ignores the latent variable altogether inspiring various solutions (Yang et al. 2017; Goyal et al. 2017). Our results on noise-conditioning are also reminiscent of a line of work on exploration in reinforcement learning (RL), where it has been shown that adding noises to the policy model parameters enables more efficient exploration than directly adding noises to the output space (Plappert et al. 2017; Fortunato et al. 2017).

生成模型的多样性。 生成模型史上与我们相关的一条线是面向文本数据的基于 RNN 的 VAE（Bowman et al. 2016）。动机与我们类似：希望学习高层语义特征而非下一词特征，以产出更新颖句子。但这遭受后验崩塌——模型完全忽略潜变量——并激发多种解法（Yang et al. 2017；Goyal et al. 2017）。噪声条件化结果也让人联想到强化学习中的探索工作线：向策略模型参数加噪比直接向输出空间加噪更能高效探索（Plappert et al. 2017；Fortunato et al. 2017）。

Learning-theoretic studies of diversity in LLMs. Various theoretical works provide rigorous arguments for how preventing hallucination and maximizing the model’s coverage are at odds with each other in abstract settings (Kalai & Vempala 2024; Kalavasis et al. 2024; Kleinberg & Mullainathan 2024). We clarify that this tension does not apply in our concrete settings. In those abstract settings, the strings in the support can be arbitrary and adversarially chosen whereas, our strings are generated by a simple rule (which can be learned).

LLM 多样性的学习理论研究。 多项理论工作在抽象设定中严格论证：防止幻觉与最大化模型覆盖相互对立（Kalai & Vempala 2024；Kalavasis et al. 2024；Kleinberg & Mullainathan 2024）。我们澄清：该张力不适用于我们的具体设定。在那些抽象设定中，支撑中的串可任意且对抗选取；而我们的串由可学习的简单规则生成。

Another theoretical question underlying generative models is that the optimum of their objectives are attained at perfect memorization; yet they tend to produce novel examples e.g., this question has been posed for GANs in Nagarajan et al. 2018 and for diffusion in Nakkiran et al. 2024 (see “remarks on generalization”) or Kamb & Ganguli 2024. Of relevance to us is, Kamb & Ganguli 2024 who provide a theoretical and empirical argument for how image diffusion models are able to generate combinatorially many creative outputs; theirs however do not require the type of planning our tasks do.

生成模型背后的另一理论问题是：其目标的最优在完美记忆处达到，却往往产出新颖样例；例如 Nagarajan et al. 2018 对 GAN、Nakkiran et al. 2024（见「泛化评注」）或 Kamb & Ganguli 2024 对扩散提出该问题。与我们相关的是 Kamb & Ganguli 2024：他们从理论与经验论证图像扩散模型如何能生成组合爆炸多的创造输出；但其任务不要求我们任务所需的那类规划。

Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction

掷骰子，三思而后行：超越下一词预测的创造极限

Abstract

摘要

1 Introduction

1 引言

Keywords:

关键词：

2 Open-ended algorithmic tasks & two types of creativity

2 开放式算法任务与两类创造力

2.1 The fundamental types of creativity (Boden 2003)

2.1 创造力的基本类型（Boden 2003）

2.2 The basic setting and notations

2.2 基本设定与记号

2.3 Tasks inspired by combinational creativity

2.3 受组合创造启发的任务

2.3.1 Sibling discovery

2.3.1 兄弟姐妹发现

2.3.2 Triangle discovery

2.3.2 三角形发现

2.4 Tasks inspired by exploratory creativity

2.4 受探索创造启发的任务

2.4.1 Circle construction

2.4.1 圆构造

2.4.2 Line construction

2.4.2 线构造

2.5 Permutation-invariance of our tasks

2.5 任务的置换不变性

2.6 How next-token learning may suffer in our tasks

2.6 下一词学习在我们任务上可能如何吃亏

3 Training and Inference

3 训练与推理

Transformers.

Transformer。

Diffusion models.

扩散模型。

3.1 Seed-conditioning

3.1 种子条件化

4 Experimental results

4 实验结果

Key details.

关键细节。

4.1 Observations

4.1 观察

4.2 An initial exploration of real-world summarization

4.2 真实摘要任务的初步探索

5 Discussion

5 讨论

5.1 Intuition about seed-conditioning

5.1 关于种子条件化的直觉

5.2 Effects of reasoning-enhancing methods.

5.2 增强推理方法的影响

6 Limitations

6 局限

6.1 Limitations of our experimental conclusions

6.1 实验结论的局限

6.2 Our approach to creativity

6.2 我们对创造力的进路

7 Related Work

7 相关工作

Open-ended algorithmic tasks.

开放式算法任务。

Diversity in generative models.

生成模型中的多样性。

Going beyond next-token prediction (NTP).

超越下一词预测（NTP）。

Injecting noise into a Transformer.

向 Transformer 注入噪声。

8 Conclusions

8 结论

9 Impact Statement

9 影响声明

References

参考文献

Appendix A Transformer Training Objectives

附录 A Transformer 训练目标

Appendix B Further discussion

附录 B 进一步讨论

B.1 Style of noise-injection

B.1 噪声注入方式