The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

marginparsep has been altered. topmargin has been altered. marginparpush has been altered.

marginparsep 已被修改。topmargin 已被修改。marginparpush 已被修改。

The page layout violates the ICML style.

页面布局违反 ICML 样式。

Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you.

请勿更改页面布局，也不要引入 geometry、savetrees 或 fullpage 等会改变布局的宏包。

We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again.

我们无法可靠撤销对样式的任意改动。请移除 offending 宏包或改变布局的命令后重试。

The Flexibility Trap: Why Arbitrary Order Limits Reasoning Potential in Diffusion Language Models

灵活性陷阱：为何任意序 限制 扩散语言模型的推理潜力

Zanlin Ni${}^{\,1\,}$**, Shenzhi Wang1, Yang Yue1, Tianyu Yu2, Weilin Zhao2, Yeguo Hua3, Tianyi Chen3, Jun Song4, Cheng Yu4, Bo Zheng4, Gao Huang${}^{\,1\,\textrm{\Letter}}$**

${}^{1\,}$ LeapLab, Tsinghua University  ${}^{2\,}$ NLPLab, Tsinghua University  ${}^{3\,}$ Tsinghua University  ${}^{4\,}$ Alibaba Group

nzl22@mails.tsinghua.edu.cn, gaohuang@tsinghua.edu.cn

Figure 1: Less flexibility unlocks better reasoning potential. Left: We observe a counter-intuitive phenomenon where restricting dLLMs to standard Autoregressive (AR) order expands the reasoning solution space. Right: Motivated by this, we propose “JustGRPO”. By foregoing complex arbitrary-order adaptations and adopting just GRPO, we elicit the reasoning capability of dLLMs more effectively. Results reported with generation length of 256, more results in Table 1.

图 1：更少灵活性反而解锁更好的推理潜力。 左：我们观察到反直觉现象——把 dLLM 限制为标准自回归（AR）顺序会扩大推理解空间。右：据此提出 “JustGRPO”。放弃复杂的任意序适配、直接采用 GRPO，能更有效激发 dLLM 推理能力。报告生成长度 256 的结果，更多见表 1。

1 Introduction

1 引言

Recent research has witnessed a surge in Diffusion Large Language Models (dLLMs) Nie et al. 2025; Ye et al. 2025; Zhu et al. 2025; Zhao et al. 2025, which challenge the dominant autoregressive (AR) paradigm Brown et al. 2020; Achiam et al. 2023 by treating sequence generation as a discrete denoising process. Central to the appeal of dLLMs is their theoretical flexibility, which offers two distinct advantages over the strict left-to-right causal chain of AR models: efficient parallel decoding and the capability for arbitrary-order generation.

近期研究见证扩散大语言模型（dLLM）热潮 Nie 等 2025；Ye 等 2025；Zhu 等 2025；Zhao 等 2025，它们把序列生成视为离散去噪过程，挑战主导的自回归（AR）范式 Brown 等 2020；Achiam 等 2023。dLLM 的吸引力在于理论灵活性，相对 AR 严格从左到右因果链有两大优势：高效并行解码，以及任意序生成能力。

While the efficiency gains of parallel decoding have been well-established Wu et al. 2025b; Labs et al. 2025; DeepMind 2025; Song et al. 2025; Wu et al. 2025a, the implications of arbitrary-order generation remain less explored Ye et al. 2024; Kim et al. 2025. In theory, the unconstrained generation order constitutes a superset of the fixed autoregressive trajectory, unlocking non-sequential problem-solving paths. Ideally, such advantage could facilitate more advanced problem-solving abilities that benefit general reasoning tasks like mathematics and coding. As a result, recent works have increasingly adopted RL to elicit reasoning capabilities of dLLMs Zhao et al. 2025; Gong et al. 2025; Wang et al. 2025a.

并行解码的效率收益已被充分确立 Wu 等 2025b；Labs 等 2025；DeepMind 2025；Song 等 2025；Wu 等 2025a，任意序生成的含义则探索较少 Ye 等 2024；Kim 等 2025。理论上，无约束生成顺序是固定自回归轨迹的超集，可解锁非顺序解题路径，理想上应利于数学与代码等通用推理。因此近期工作越来越多用 RL 激发 dLLM 推理 Zhao 等 2025；Gong 等 2025；Wang 等 2025a。

In this paper, we present a counter-intuitive observation: arbitrary-order generation, in its current form, narrows rather than expands the reasoning potential elicitable by RL. To rigorously assess this, we employ Pass@$k$ Chen 2021, which measures the coverage of solution space. Recent studies suggest that RL primarily acts to sharpen the base model’s distribution; consequently, the Pass@$k$ performance of the base model effectively sets the upper bound for the reasoning capability of the model after RL training Yue et al. 2025; Liu et al. 2025; Zhang et al. 2025. Under this metric, we compare the reasoning potential of LLaDA Nie et al. 2025 with arbitrary-order generation against standard AR decoding. As shown in Figure 1 (Left), restricting a dLLM to standard AR order in fact yields a higher Pass@$k$, and consequently a higher reasoning boundary, than its flexible counterpart.

本文给出反直觉观察：以当前形式，任意序生成被 RL 激发的推理潜力是收窄而非扩大。为严格评估，我们采用 Pass@$k$ Chen 2021 度量解空间覆盖。近期研究表明 RL 主要锐化基座分布；因而基座的 Pass@$k$ 实质上设定 RL 后推理能力上界 Yue 等 2025；Liu 等 2025；Zhang 等 2025。在该指标下比较 LLaDA Nie 等 2025 的任意序与标准 AR 解码。如图 1（左），把 dLLM 限制为标准 AR 顺序反而得到更高 Pass@$k$，从而更高的推理边界。

Figure 2: Confronting vs. bypassing uncertainty. (a) AR order preserves reasoning space by forcing decisions at uncertain tokens. (b) Arbitrary order bypasses uncertainty and resolves easier tokens first. Once future context is established, the original forks collapse, prematurely narrowing the solution space.

图 2：直面 对 绕开不确定性。 (a) AR 顺序 迫使在不确定词元处做决策，从而保留推理空间。(b) 任意序 绕开不确定性、先解更容易词元。一旦未来上下文确立，原有分叉塌缩，过早收窄解空间。

We attribute this counter-intuitive phenomenon to the way model handles uncertainty. Reasoning is inherently non-uniform: it typically hinges on sparse “forking tokens”, i.e., connectives like “Therefore” or “Since” which do not merely continue a sentence but fundamentally steer the logical trajectory into distinct branches Wang et al. 2025c; Cheng et al. 2025; Huang et al. 2025a. At these forks, the reasoning path diverges, naturally manifesting as localized spikes in entropy Wang et al. 2025c. Standard AR decoding compels the model to confront this uncertainty (Figure 2a). By sampling exactly at the fork, the model is able to explore different reasoning paths, thereby preserving the diversity of the generated rationales. Arbitrary-order generation, however, allows the model to bypass these hard decisions (Figure 2b). It prioritizes low-uncertainty completions first. By the time the model returns to infill the bypassed forks, the established future context has already severely constrained the potential branches. The ambiguity is prematurely resolved, and the original high entropy is suppressed. We term this phenomenon entropy degradation. Effectively, the model trades the exploration of diverse reasoning paths for the greedy optimization of local consistency.

我们把这一反直觉现象归因于模型处理不确定性的方式。推理本质非均匀：常依赖稀疏的「分叉词元」，如 “Therefore”“Since” 等连接词——它们不只续句，更从根本上把逻辑轨迹导入不同分支 Wang 等 2025c；Cheng 等 2025；Huang 等 2025a。分叉处路径分岔，自然表现为局部熵尖峰 Wang 等 2025c。标准 AR 解码迫使模型直面不确定性（图 2a）：恰在分叉处采样，可探索不同推理路径，从而保留生成推理链的多样性。任意序则允许绕开难题（图 2b）：优先完成低不确定补全；待回头填充分叉时，已确立的未来上下文严重约束潜在分支，歧义被过早消解，原本高熵被压制。我们称之为 熵退化。实质上，模型用探索多样推理路径去换局部一致性的贪心优化。

The above observations motivate a rethink of RL for dLLMs. Current methods operate under the assumption that preserving arbitrary-order flexibility is essential. This commitment incurs a heavy tax: algorithms must grapple with a combinatorial explosion of denoising trajectories Zhao et al. 2025; Yang et al. 2025; Gong et al. 2025 and intractable marginal likelihoods Ou et al. 2025, forcing reliance on unstable approximations Wang et al. 2025a; Ou et al. 2025; Rojas et al. 2025. However, if arbitrary order is non-essential, or even detrimental for eliciting reasoning potential, this complexity is unjustified.

上述观察促使重新思考 dLLM 的 RL。现有方法假定必须保留任意序灵活性，代价沉重：算法须面对去噪轨迹组合爆炸 Zhao 等 2025；Yang 等 2025；Gong 等 2025 与难处理边缘似然 Ou 等 2025，被迫依赖不稳近似 Wang 等 2025a；Ou 等 2025；Rojas 等 2025。若任意序对激发推理非必要甚至有害，这种复杂度便无正当性。

To this end, we propose a return to simplicity with JustGRPO. We demonstrate that eliciting reasoning potential for general reasoning tasks does not require complex, diffusion-specific RL adaptations. Instead, it can be better achieved by simply treating the dLLM as an AR model during RL training. This allows us to apply standard GRPO Shao et al. 2024 without bells and whistles, turning an otherwise intractable optimization with unstable approximations into a well-defined task.

为此我们提出回归简洁的 JustGRPO。表明对通用推理任务激发潜力，不必做复杂、扩散专用的 RL 适配；RL 训练时把 dLLM 当作 AR 模型即可。从而可无花哨地套用标准 GRPO Shao 等 2024，把原本难处理、依赖不稳近似的优化变成定义良好的任务。

JustGRPO is surprisingly effective. On complex reasoning benchmarks, it achieves competitive results (e.g., 89.1% accuracy on GSM8K, 45.1% on MATH-500), surpassing methods that rely on complex diffusion-specific RL. Crucially, while we train with AR constraints to maximize reasoning potential, the model retains dLLMs’ ability of efficient parallel decoding at inference time. By returning to basic left-to-right ordering, this work encourages a re-evaluation of arbitrary versus AR order in the development of next-generation language models.

JustGRPO 出奇有效。在复杂推理基准上取得有竞争力结果（如 GSM8K 89.1%、MATH-500 45.1%），超过依赖复杂扩散专用 RL 的方法。关键是：虽用 AR 约束训练以最大化推理潜力，推理时仍保留 dLLM 高效并行解码能力。回归基本从左到右顺序，本文鼓励在下一代语言模型发展中重新评估任意序与 AR 顺序。

2 Preliminaries

2 预备知识

2.1 Diffusion Large Language Models

2.1 扩散大语言模型

Diffusion Large Language Models (dLLMs), particularly Masked Diffusion Models (MDMs), generate sequences by iteratively denoising a masked state $x_{t}$ initialized from fully masked tokens. The process is indexed by a continuous time variable $t\in[0,1]$, representing the masking ratio. Given a clean sequence $x_{0}$, the forward process independently masks each token with probability $t$:

扩散大语言模型（dLLM），尤其掩码扩散模型（MDM），通过迭代去噪从全掩码初始化的掩码状态 $x_{t}$ 生成序列。过程由连续时间 $t\in[0,1]$ 索引，表示掩码比例。给定干净序列 $x_{0}$，前向过程以概率 $t$ 独立掩码各词元：

$$q(x_{t,k}\mid x_{0,k})=\begin{cases}\texttt{[MASK]},&\text{with prob }t,\
x_{0,k},&\text{with prob }1-t,\end{cases} \tag{1}$$

where $k$ is the token index. A neural network $p_{\theta}(x_{0}\mid x_{t})$ estimates the original token distribution at masked positions. During inference, generation starts from $x_{1}$ (all [MASK]) and iteratively unmasks a subset of tokens based on heuristics such as confidence scores, updating $x_{t}\rightarrow x_{t-\Delta t}$ until completion. As a special case, dLLMs can also perform generation autoregressively by always unmasking the leftmost token. The model is trained by minimizing the Negative Evidence Lower Bound, which reduces to a weighted cross entropy loss over masked tokens:

其中 $k$ 为词元下标。神经网络 $p_{\theta}(x_{0}\mid x_{t})$ 估计掩码位置的原始词元分布。推理从 $x_{1}$（全 [MASK]）出发，按置信度等启发式迭代去掩部分词元，更新 $x_{t}\rightarrow x_{t-\Delta t}$ 直至完成。特例是始终去掩最左词元，从而自回归生成。训练最小化负证据下界，化简为掩码词元上的加权交叉熵：

$$\mathcal{L}_{\mathrm{MDM}}(\theta)=-\mathbb{E}_{t\sim\mathcal{U}[0,1],\,x_{t}\sim q(x_{t}\mid x_{0})}\left[\frac{1}{t}\sum_{k=1}^{L}\mathbf{1}[x_{t,k}=\texttt{[MASK]}]\,\log p_{\theta}(x_{0,k}\mid x_{t})\right]. \tag{2}$$

2.2 Group Relative Policy Optimization

2.2 组相对策略优化

Group Relative Policy Optimization (GRPO) is a reinforcement learning algorithm that avoids value function estimation by using group level reward normalization. It is typically applied to autoregressive policies $\pi_{\theta}(o_{k}\mid o_{<k},q)$. For each query $q$, GRPO samples a group of $G$ outputs $\{o_{1},\dots,o_{G}\}$ from the old policy $\pi_{\theta_{\text{old}}}$. An advantage $\hat{A}_{i,k}$ is computed by standardizing the reward $r(o_{i})$ against the group statistics: $\hat{A}_{i,k}=(r(o_{i})-\mu_{G})/\sigma_{G}$, where $\mu_{G}$ and $\sigma_{G}$ are the group mean and standard deviation. The GRPO objective maximizes a clipped surrogate function with a KL regularization term:

组相对策略优化（GRPO）是强化学习算法，用组级奖励归一化避免价值函数估计，通常用于自回归策略 $\pi_{\theta}(o_{k}\mid o_{<k},q)$。对每个查询 $q$，从旧策略 $\pi_{\theta_{\text{old}}}$ 采样一组 $G$ 个输出 $\{o_{1},\dots,o_{G}\}$。优势 $\hat{A}_{i,k}$ 由奖励 $r(o_{i})$ 相对组统计标准化：$\hat{A}_{i,k}=(r(o_{i})-\mu_{G})/\sigma_{G}$。目标最大化带 KL 正则的裁剪代理：

$$\mathcal{J}(\theta)=\mathbb{E}_{q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{k=1}^{|o_{i}|}\left(\min\left(\rho_{i,k}\hat{A}_{i,k},\text{clip}(\rho_{i,k},1-\epsilon,1+\epsilon)\hat{A}_{i,k}\right)-\beta D_{\text{KL}}\right)\right] \tag{3}$$

with the token level importance ratio $\rho_{i,k}=\frac{\pi_{\theta}(o_{i,k}\mid o_{i,<k},q)}{\pi_{\theta_{\text{old}}}(o_{i,k}\mid o_{i,<k},q)}$.

其中词元级重要性比 $\rho_{i,k}=\frac{\pi_{\theta}(o_{i,k}\mid o_{i,<k},q)}{\pi_{\theta_{\text{old}}}(o_{i,k}\mid o_{i,<k},q)}$。

2.3 Pass@$k$ as a Proxy for Reasoning Potential

2.3 以 Pass@$k$ 代理推理潜力

To rigorously quantify the reasoning capability boundaries of dLLMs, we adopt the Pass@$k$ metric Chen 2021; Yue et al. 2025. In the context of Reinforcement Learning with Verifiable Rewards (RLVR), which is currently the dominant paradigm for enhancing reasoning capabilities, exploration is a prerequisite for improvement. An RL agent can only reinforce correct reasoning paths if it is capable of sampling them during the exploration phase to obtain a positive reward signal.

为严格量化 dLLM 推理能力边界，采用 Pass@$k$ Chen 2021；Yue 等 2025。在当前主导的可验证奖励强化学习（RLVR）中，探索是改进前提：只有探索阶段能采样到正确推理路径并获正奖励，智能体才能强化它们。

Accordingly, Pass@$k$ has been widely established as a standard measure of a model’s reasoning potential Yue et al. 2025; Liu et al. 2025; Zhang et al. 2025. It measures the probability that at least one correct solution is generated within $k$ independent samples, effectively representing the upper bound of the solution space accessible to RL optimization. Formally, following the unbiased estimator formulation Chen 2021; Yue et al. 2025, given $n$ samples where $c$ are correct, Pass@$k$ is calculated as:

因此 Pass@$k$ 被广泛作为模型推理潜力的标准度量 Yue 等 2025；Liu 等 2025；Zhang 等 2025。它度量 $k$ 次独立采样中至少一次正确的概率，实质上表示 RL 优化可达解空间上界。形式化地，按无偏估计 Chen 2021；Yue 等 2025，给定 $n$ 次采样中 $c$ 次正确：

$$\text{Pass@}k=\mathbb{E}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right] \tag{4}$$

A high Pass@$k$ indicates that the correct reasoning trajectory lies within the model’s sampling distribution and is thus learnable via RL optimization. Conversely, if a model consistently fails to yield a solution despite a vast sampling budget, it suggests the problem lies beyond its intrinsic reasoning boundary. In such scenarios, standard RLVR methodologies are fundamentally limited by the absence of positive exploration signals Yue et al. 2025.

高 Pass@$k$ 表示正确推理轨迹落在采样分布内，因而可通过 RL 学习。反之，若大采样预算仍始终无解，则问题超出内在推理边界；此类场景下标准 RLVR 因缺乏正探索信号而根本受限 Yue 等 2025。

Figure 3: Reasoning potential measured by Pass@$k$. While arbitrary order is competitive in single-shot settings ($k=1$), it exhibits flatter scaling curves compared to AR Order.

图 3：用 Pass@$k$ 度量推理潜力。 任意序在单次设置（$k=1$）有竞争力，但相对 AR 顺序缩放曲线更平。

3 The Flexibility Trap

3 灵活性陷阱

In this section, we rigorously test whether the flexibility of arbitrary-order generation translates into higher reasoning potential. We compare two decoding modes: Arbitrary Order, which follows standard diffusion decoding with low-confidence remasking Nie et al. 2025; Zhao et al. 2025; Wu et al. 2025b, and AR Order, where arbitrary-order flexibility is disabled and generation is strictly constrained to left-to-right decoding. We adopt the commonly used experimental setup in prior diffusion LLM work: a maximum of 256 tokens decoded over 256 steps, a semi-autoregressive block size of 32. Sampling temperature is set to 0.6 as in Yue et al. 2025. The prompt template follows Zhao et al. 2025. Results under alternative temperatures and sampling configurations are deferred to Appendix B.

本节严格检验任意序灵活性是否转化为更高推理潜力。比较两种解码：任意序，即标准扩散解码加低置信重掩码 Nie 等 2025；Zhao 等 2025；Wu 等 2025b；AR 顺序，关闭任意序、严格从左到右。沿用先前扩散 LLM 常用设置：最多 256 词元、256 步、半自回归块大小 32，采样温度 0.6（同 Yue 等 2025），提示模板同 Zhao 等 2025。其他温度与采样配置见附录 B。

3.1 Arbitrary Order Limits Reasoning Potential

3.1 任意序限制推理潜力

Pass@$k$ analysis.

We first evaluate the reasoning potential using the Pass@$k$ metric on three representative dLLMs: LLaDA-Instruct Nie et al. 2025, Dream-Instruct Ye et al. 2025, and LLaDA 1.5 Zhu et al. 2025 on four reasoning benchmarks: GSM8K, MATH-500, HumanEval, and MBPP. As shown in Figure 3, while arbitrary order often achieves competitive performance at $k=1$, it exhibits a notably flatter scaling curve compared to the AR mode. As $k$ increases, the AR mode demonstrates a stronger capacity to uncover correct solutions.

Pass@$k$ 分析。

先在 GSM8K、MATH-500、HumanEval、MBPP 四个推理基准上，对 LLaDA-Instruct Nie 等 2025、Dream-Instruct Ye 等 2025、LLaDA 1.5 Zhu 等 2025 用 Pass@$k$ 评估。如图 3，任意序常在 $k=1$ 有竞争力，但相对 AR 模式缩放曲线明显更平；$k$ 增大时 AR 更能挖出正确解。

Figure 4: Solution space coverage measured by Pass@$1024$. The problems solvable by arbitrary order (AO) are largely a subset of those solved by AR Order.

图 4：解空间覆盖，以 Pass@$1024$ 度量。任意序（AO）可解问题大体是 AR 顺序可解问题的子集。

Solution space coverage.

One might wonder whether arbitrary order explores a different solution space, albeit less efficiently, which could account for its lower reasoning potential. We test this by analyzing solution coverage at $k=1024$ using LLaDA-Instruct. Figure 4 shows that the problems solvable by arbitrary order are largely a subset of those solved by AR. On HumanEval, 21.3% of problems are solved exclusively by AR, whereas the reverse is negligible (0.6%). This indicates that the flexible decoding process rarely unlocks genuinely new solutions. Instead, it appears to retreat into a more conservative subset of the AR solution space.

解空间覆盖。

或问任意序是否探索了不同但效率较低的解空间。我们用 LLaDA-Instruct 在 $k=1024$ 分析覆盖。图 4 显示任意序可解问题大体是 AR 的子集。HumanEval 上 21.3% 问题仅 AR 可解，反向可忽略（0.6%）。说明灵活解码很少解锁真正新解，更像退守到 AR 解空间更保守的子集。

3.2 Mechanism: The Entropy Degradation

3.2 机制：熵退化

Figure 5: Frequently bypassed tokens in arbitrary order, measured on MATH-500, are typically logical connectors and transition words.

图 5：任意序中常被绕开的词元（MATH-500 上统计）多为逻辑连接与过渡词。

Adaptive decoding bypasses logical forks.

To understand why the theoretically superior solution space of dLLMs collapses in practice, we examine more closely how the two decoding modes behave. In AR order, the model is constrained to strictly resolve the left-most unknown token at each step, forcing the model to confront uncertainty as it arises. In contrast, arbitrary order adaptively selects tokens to update based on model confidence, preferentially generating “easy” tokens with high certainty while bypassing “hard” ones. Inspecting the frequently bypassed tokens reveals a clear pattern: As shown in Figure 5, the diffusion sampler disproportionately defers logical connectives and transition markers such as “Therefore”, “Thus”, and “Since”. Prior work has shown that such tokens often have high entropy (which also holds true in dLLMs; see Figure 6), and act as “reasoning sparks” or “logical forks”, functioning as branching points that determine subsequent reasoning directions Wang et al. 2025c; Cheng et al. 2025; Wang et al. 2025b; Huang et al. 2025a. In conventional language models, keeping these tokens in high-entropy state is critical for effective exploration of the reasoning space Wang et al. 2025c.

自适应解码绕开逻辑分叉。

为理解理论更优的解空间为何实践中塌缩，细看两种解码。AR 顺序每步必须解析最左未知词元，迫使直面不确定性；任意序则按模型置信自适应选更新词元，优先高确定性「易」词、绕开「难」词。查看常被绕开词元有清晰规律：如图 5，扩散采样器不成比例地推迟 “Therefore”“Thus”“Since” 等逻辑连接与过渡标记。先前工作表明此类词元常高熵（dLLM 中亦然，见图 6），充当「推理火花」或「逻辑分叉」，决定后续方向 Wang 等 2025c；Cheng 等 2025；Wang 等 2025b；Huang 等 2025a。传统语言模型中保持这些词元高熵对有效探索推理空间至关重要 Wang 等 2025c。

Figure 6: Entropy degradation. While the global average token entropy (dashed lines) of arbitrary order remains comparable to AR, the entropy at logical forks (blue bars) drops significantly.

图 6：熵退化。 任意序全局平均词元熵（虚线）与 AR 相当，但逻辑分叉处熵（蓝柱）显著下降。

The “entropy degradation” phenomenon.

How does the adaptive behavior of arbitrary order affect these logical forking tokens? We measure the entropy of them when they are decoded in Figure 6 (more results in Appendix C). In AR order, these tokens generally maintain high entropy when decoded, indicating a relatively fruitful branching point where multiple reasoning paths remain viable Wang et al. 2025c; Cheng et al. 2025. In contrast, arbitrary order exhibits a sharp decrease in entropy. By deferring the logical connectors, the model prioritizes generating easier future tokens before deciding the logic connections that lead to them. When the model eventually returns to fill in the bypassed connectors, the navigational freedom is notably constrained; the process acts less as an open-ended navigational decision at a fork, and more as a retrospective alignment to bridge the gap to the pre-generated conclusion. We term this phenomenon entropy degradation.

「熵退化」现象。

任意序自适应行为如何影响这些逻辑分叉词元？图 6 测量它们被解码时的熵（更多结果见附录 C）。AR 顺序下这些词元解码时一般保持高熵，表明仍是多路径可行的丰饶分支点 Wang 等 2025c；Cheng 等 2025。任意序则熵急剧下降：推迟逻辑连接词后，模型优先生成更容易的未来词元，再决定通向它们的逻辑连接。回头填充分叉时，导航自由度明显受限；过程更像事后对齐以桥接预生成结论，而非分叉处开放导航决策。我们称之为 熵退化。

Conclusion.

In summary, the flexibility of arbitrary order serves as a mechanism for inference-time exploitation rather than reasoning exploration. By bypassing high-uncertainty tokens, the model effectively collapses the solution space to a safer, lower-entropy path, squeezing out slightly better single-shot coherence at the expense of reasoning potential. Autoregressive ordering, by contrast, lack this bypassing capability and are therefore forced to sample directly from high-entropy distributions at logical forks. It is precisely this inability to circumvent critical decision points that prevents the premature narrowing of the search space and preserves the reasoning potential.

结论。

总之，任意序灵活性在推理时更像利用机制而非探索：绕开高不确定词元，把解空间塌缩到更安全、更低熵路径，略换更好的单次连贯，却牺牲推理潜力。自回归顺序无法绕开，被迫在逻辑分叉处直接从高熵分布采样；正是不能规避关键决策点，才避免搜索空间过早收窄并保留推理潜力。

4 “Just GRPO” for Diffusion Language Models

4 扩散语言模型的 “Just GRPO”

The findings in Section 3 suggest that arbitrary order actually limits the reasoning potential accessible to RL. Despite this, current RL methods for dLLMs remain heavily burdened by the need to preserve this specific flexibility. In this section, we uncover the heavy “tax” imposed by this flexibility (Section 4.1) and show that discarding it enables a minimalist yet surprisingly effective solution: JustGRPO (Section 4.2).

第 3 节表明任意序实际限制了 RL 可达的推理潜力。尽管如此，现有 dLLM 的 RL 方法仍被保留该灵活性拖累。本节揭示这一灵活性施加的沉重「税」（第 4.1 节），并说明放弃它可得到极简却出奇有效的 JustGRPO（第 4.2 节）。

4.1 The Flexibility Tax in dLLMs’ RL

4.1 dLLM 强化学习中的灵活性税

Existing diffusion RL methods operate under the premise that the policy must accommodate the combinatorial space of denoising trajectories $\mathcal{T}$ to preserve arbitrary order generation. This design choice, while conceptually general, introduces several fundamental challenges:

现有扩散 RL 方法假定策略必须容纳去噪轨迹组合空间 $\mathcal{T}$ 以保留任意序。这一设计概念上一般，却引入若干根本挑战：

Ambiguity in token-level decomposition.

In dLLMs, a generation state $s_{t}$ is a noisy sequence conditioned on a stochastic unmasking trajectory $\tau$. Unlike autoregressive models, dLLMs do not admit a unique, index-aligned conditional probability of the form $\pi(o_{t}\mid s_{t})$, making token-level credit assignment ambiguous and rendering the standard importance ratio $\rho_{t}=\frac{\pi_{\theta}(o_{t}\mid s_{t})}{\pi_{\text{old}}(o_{t}\mid s_{t})}$ hard to define.

词元级分解的歧义。

在 dLLM 中，生成状态 $s_{t}$ 是条件于随机去掩轨迹 $\tau$ 的噪声序列。与自回归不同，dLLM 不具有唯一、按下标对齐的条件概率形式 $\pi(o_{t}\mid s_{t})$，使词元级信用分配含糊，标准重要性比 $\rho_{t}=\frac{\pi_{\theta}(o_{t}\mid s_{t})}{\pi_{\text{old}}(o_{t}\mid s_{t})}$ 难定义。

Intractable sequence likelihood.

Autoregressive models factorize the sequence likelihood as $\log\pi(o)=\sum_{t}\log\pi(o_{t}\mid o_{<t},q)$, whereas dLLMs require marginalization over all valid denoising trajectories, $\pi_{\theta}(o\mid q)=\sum_{\tau\in\mathcal{T}}\pi_{\theta}(o,\tau\mid q)$. For a sequence of length $N$, the trajectory space grows as $|\mathcal{T}|=O(N!)$, rendering exact likelihood computation infeasible and forcing existing methods to rely on ELBO-based surrogates rather than the original objective Wang et al. 2025a; Ou et al. 2025.

难处理的序列似然。

自回归将序列似然分解为 $\log\pi(o)=\sum_{t}\log\pi(o_{t}\mid o_{<t},q)$；dLLM 需对所有合法去噪轨迹边缘化，$\pi_{\theta}(o\mid q)=\sum_{\tau\in\mathcal{T}}\pi_{\theta}(o,\tau\mid q)$。长度 $N$ 时 $|\mathcal{T}|=O(N!)$，精确似然不可行，现有方法被迫用基于 ELBO 的代理而非原目标 Wang 等 2025a；Ou 等 2025。

Sampler-learner mismatch.

Even with an accurate likelihood approximation, a more subtle issue persists. In practice, rollout samples are typically produced by confidence-based generation $o\sim\pi_{\theta}^{\text{conf}}(o\mid q)$ to navigate the combinatorial space. However, the ELBO objective still targets the likelihood of the original model distribution $\pi_{\theta}(o\mid q)$, rather than that of the actual sampling policy $\pi_{\theta}^{\text{conf}}(o\mid q)$, leading to a mismatch between rollout and optimization that can degrade performance Schulman et al. 2015.

采样器—学习器失配。

即便似然近似准确，更微妙问题仍在。实践中 rollout 常由基于置信的生成 $o\sim\pi_{\theta}^{\text{conf}}(o\mid q)$ 以导航组合空间；ELBO 目标却仍对准原始模型分布 $\pi_{\theta}(o\mid q)$ 的似然，而非实际采样策略 $\pi_{\theta}^{\text{conf}}(o\mid q)$，导致 rollout 与优化失配、损害性能 Schulman 等 2015。

4.2 JustGRPO

We propose a return to simplicity. Since pure autoregressive order yields better reasoning potential (Section 3), we explicitly forgo arbitrary-order generation during the RL stage. This constraint transforms the dLLM from a chaotic sequence denoiser into a well-defined autoregressive policy $\pi_{\theta}^{AR}$.

我们主张回归简洁。因纯自回归顺序给出更好推理潜力（第 3 节），RL 阶段显式放弃任意序生成。该约束把 dLLM 从混乱序列去噪器变为定义良好的自回归策略 $\pi_{\theta}^{AR}$。

Formulation.

Standard GRPO assumes a policy $\pi(o_{k}|o_{<k},q)$ accepting a partial sequence $o_{<k}$ with all tokens observed and predicts one token $o_{k}$ at a time, where $q$ is the query. Diffusion language models, however, are architected as sequence-level denoisers that accept a full sequence with mixed observed and masked tokens and predict the original values for all masked positions simultaneously.

形式化。

标准 GRPO 假定策略 $\pi(o_{k}|o_{<k},q)$ 接受部分序列 $o_{<k}$（全部已观测）并每次预测一个词元 $o_{k}$。扩散语言模型则架构为序列级去噪器：接受混合已观测与掩码词元的完整序列，并同时预测所有掩码位置的原值。

By forgoing arbitrary-order generation, we are able to bridge the above gap and rigorously define an AR policy $\pi_{\theta}^{\text{AR}}$ for dLLMs. To obtain the probability of the next token $o_{k}$ given history $o_{<k}$, we construct $\tilde{x}_{k}$ where the past is observed and the future is masked:

放弃任意序后，可弥合上述鸿沟并严格定义 dLLM 的 AR 策略 $\pi_{\theta}^{\text{AR}}$。为得给定历史 $o_{<k}$ 下下一词元 $o_{k}$ 的概率，构造过去已观测、未来掩码的 $\tilde{x}_{k}$：

$$\tilde{x}_{k}=[\underbrace{o_{1},\dots,o_{k-1}}_{\text{Observed}},\underbrace{\texttt{[MASK]},\dots,\texttt{[MASK]}}_{\text{Masked}}]. \tag{5}$$

Although the diffusion language model outputs predictions for all masked positions, the autoregressive policy concerns only the next token $o_{k}$. We define $\pi_{\theta}^{\text{AR}}(\cdot|o_{<k},q)$ as follows:

虽扩散模型对所有掩码位置输出预测，自回归策略只关心下一词元 $o_{k}$。定义 $\pi_{\theta}^{\text{AR}}(\cdot|o_{<k},q)$ 如下：

$$\pi_{\theta}^{\text{AR}}(\cdot|o_{<k},q)\triangleq\text{Softmax}(f_{\theta,k}(\tilde{x}_{k},q)), \tag{6}$$

where $f_{\theta,k}$ denotes the model logits at position $k$. Consequently, the likelihood of a complete reasoning chain $o$ is exactly computable as:

其中 $f_{\theta,k}$ 为位置 $k$ 的模型 logits。因而完整推理链 $o$ 的似然可精确计算为：

$$\pi_{\theta}^{\text{AR}}(o|q)=\prod_{k=1}^{|o|}\pi_{\theta}^{\text{AR}}(o_{k}|o_{<k},q). \tag{7}$$

Optimization.

The above formulation enables the direct application of standard GRPO to diffusion language models. For each query $q$, we sample a group of outputs $\{o_{i}\}_{i=1}^{G}$ using the old policy $\pi_{\theta_{\text{old}}}^{\text{AR}}$. The objective is:

优化。

上述形式化使标准 GRPO 可直接用于扩散语言模型。对每个查询 $q$，用旧策略 $\pi_{\theta_{\text{old}}}^{\text{AR}}$ 采样一组输出 $\{o_{i}\}_{i=1}^{G}$。目标为：

$$\displaystyle\mathcal{J}(\theta) \\ \displaystyle=\mathbb{E}_{q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}^{\text{AR}}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{k=1}^{|o_{i}|}\left(\min\left(\rho_{i,k}\hat{A}_{i,k},\text{clip}(\rho_{i,k},1-\varepsilon,1+\varepsilon)\hat{A}_{i,k}\right)-\beta\mathbb{D}_{\text{KL}}\right)\right], \tag{8}$$

where $\rho_{i,k}=\frac{\pi_{\theta}^{\text{AR}}(o_{i,k}\mid o_{i,<k},q)}{\pi_{\theta_{\text{old}}}^{\text{AR}}(o_{i,k}\mid o_{i,<k},q)}$ is the probability ratio between the current and old policies and $\hat{A}_{i,k}$ denotes the group-standardized advantage.

其中 $\rho_{i,k}=\frac{\pi_{\theta}^{\text{AR}}(o_{i,k}\mid o_{i,<k},q)}{\pi_{\theta_{\text{old}}}^{\text{AR}}(o_{i,k}\mid o_{i,<k},q)}$ 为当前与旧策略概率比，$\hat{A}_{i,k}$ 为组标准化优势。

Remarks.

One might worry whether training in AR mode turns the diffusion language model into a standard autoregressive model. This is not the case. The AR constraint is applied only during training to correctly assign credit. It refines the model’s joint distribution without altering the underlying architecture. At inference time, the model retains its conditional independence properties, still allowing us to employ parallel samplers Ben-Hamu et al. 2025 to accelerate decoding. JustGRPO thus benefits from reasoning exploration of AR order while preserving the inference speed of dLLMs (see Section 5.2).

备注。

或担心 AR 模式训练会把扩散语言模型变成标准自回归模型。并非如此。AR 约束仅在训练施加以正确分配信用， refinement 联合分布而不改底层架构。推理时模型仍保留条件独立性，仍可用并行采样器 Ben-Hamu 等 2025 加速解码。JustGRPO 因而兼得 AR 顺序的推理探索与 dLLM 的推理速度（见第 5.2 节）。

5 Experiments

5 实验

We evaluate JustGRPO on standard mathematical reasoning and coding benchmarks. Our experimental design aims to verify two hypotheses: (i) that enforcing autoregressive (AR) order during RL training elicits superior reasoning capabilities compared to complex arbitrary-order approximations, and (ii) that this constraint applies only to the optimization objective, leaving the diffusion model’s parallel decoding capabilities intact at inference.

我们在标准数学推理与代码基准上评估 JustGRPO。实验设计旨在验证两点：(i) RL 训练时强制自回归（AR）顺序相对复杂任意序近似能激发更强推理；(ii) 该约束仅作用于优化目标，推理时扩散模型并行解码能力完好。

Experimental Setups.

We apply JustGRPO on LLaDA-Instruct Nie et al. 2025 and evaluate its effectiveness on four standard reasoning and coding benchmarks: GSM8K, MATH-500, HumanEval, and MBPP. For mathematical tasks, we train on the official training split of each dataset following Zhao et al. 2025; Ou et al. 2025. For coding tasks, we train on a subset of AceCoder-87K Zeng et al. 2025 following Gong et al. 2025; Ou et al. 2025. We apply JustGRPO directly to the models without additional task-specific SFT. Our training recipe largely follows Huang et al. 2025b. To evaluate the trained models, we follow the standard LLaDA evaluation protocol Nie et al. 2025, which applies low-confidence remasking together with semi-autoregressive decoding in blocks of 32 tokens, using a sampling temperature of 0. We evaluate all benchmarks at generation lengths of 128, 256, and 512 following Zhao et al. 2025; Ou et al. 2025. More details on training are provided in Appendix A.

实验设置。

在 LLaDA-Instruct Nie 等 2025 上应用 JustGRPO，评估 GSM8K、MATH-500、HumanEval、MBPP。数学任务按 Zhao 等 2025；Ou 等 2025 在各数据集官方训练集上训练；代码任务按 Gong 等 2025；Ou 等 2025 在 AceCoder-87K Zeng 等 2025 子集上训练。直接对模型做 JustGRPO，无额外任务 SFT。训练配方大体遵循 Huang 等 2025b。评估遵循标准 LLaDA 协议 Nie 等 2025：低置信重掩码加块大小 32 的半自回归解码，温度 0。按 Zhao 等 2025；Ou 等 2025 在生成长度 128、256、512 上评估全部基准。训练细节见附录 A。

5.1 Main Results

5.1 主要结果

Table 1 reports the system-level comparison. We observe that simplifying the RL objective to a standard GRPO formulation yields consistent improvements over methods specifically designed for dLLMs.

表 1 报告系统级对比。可见把 RL 目标简化为标准 GRPO，相对专为 dLLM 设计的方法仍持续改进。

Performance on reasoning tasks.

On GSM8K, JustGRPO achieves 89.1% accuracy (seq len 256), improving upon the previous best method, SPG, by a non-trivial margin (3.0%). This trend generalizes to the more challenging MATH-500 benchmark, where our approach outperforms ESPO by 6.1%. These results challenge the prevailing assumption that RL for diffusion models requires optimizing over the full combinatorial space of denoising trajectories. Instead, treating the dLLM as a sequential generator during training appears more effective for credit assignment in general reasoning tasks.

推理任务表现。

GSM8K 上 JustGRPO 达 89.1%（序列长 256），相对此前最佳 SPG 提升非琐碎的 3.0%。趋势推广到更具挑战的 MATH-500，相对 ESPO 高 6.1%。这些结果挑战「扩散模型 RL 必须在完整去噪轨迹组合空间上优化」的通行假设；训练时把 dLLM 当作顺序生成器，对通用推理任务的信用分配似乎更有效。

Robustness across generation budgets.

A potential concern is that AR constraints might overfit the model to specific trajectory lengths. However, we observe robust performance gains across varying sequence lengths (128, 256, 512). This stability suggests that the policy has improved its underlying reasoning capability rather than merely memorizing fixed-length patterns.

跨生成预算的稳健性。

潜在担心是 AR 约束会使模型过拟合特定轨迹长度。但我们在不同序列长度（128、256、512）上均见稳健增益，说明策略提升了底层推理能力，而非仅记忆定长模式。

	GSM8K	MATH-500	HumanEval	MBPP
Model / Seq Len	128	256	512	128	256	512	128	256	512	128	256	512
LLaDA-1.5 Zhu et al. 2025	-	83.3	-	-	-	-	29.3	39.6	51.9	39.6	39.9	38.8
LLaDOU Huang et al. 2025b	-	88.1	-	-	41.1	-	-	59.1	-	-	51.6	-
d1 Zhao et al. 2025	73.2	81.1	82.1	33.8	38.6	40.2	-	-	-	-	-	-
wd1 Tang et al. 2025	77.2	80.8	82.3	33.3	34.4	39.0	-	-	-	-	-	-
$d$-TreeRPO Pan et al. 2025	-	81.2	82.6	-	37.7	38.9	-	-	-	-	-	-
ESPO Ou et al. 2025	80.0	82.3	83.7	36.0	39.0	43.4	28.1	42.1	50.0	47.4	44.6	44.2
GDPO Rojas et al. 2025	78.4	82.8	84.5	33.2	39.6	41.4	26.2	39.6	39.0	43.6	50.6	47.1
SPG Wang et al. 2025a	78.5	86.1	84.5	33.4	40.0	41.8	-	-	-	-	-	-
JustGRPO	83.8	89.1	89.8	39.0	45.1	45.2	37.8	49.4	48.7	50.6	52.4	49.0

Table 1: System-level comparison of RL post-training approaches on LLaDA-Instruct. JustGRPO achieves better performance than existing diffusion-specific RL methods. LLaDA-1.5 and LLADOU are excluded from the comparison as LLaDA-1.5 is trained on a privately collected dataset at a significantly larger scale, while LLaDOU modifies the base architecture with an auxiliary module.

	GSM8K	MATH-500	HumanEval	MBPP
Model / Seq Len	128	256	512	128	256	512	128	256	512	128	256	512
LLaDA-1.5 Zhu et al. 2025	-	83.3	-	-	-	-	29.3	39.6	51.9	39.6	39.9	38.8
LLaDOU Huang et al. 2025b	-	88.1	-	-	41.1	-	-	59.1	-	-	51.6	-
d1 Zhao et al. 2025	73.2	81.1	82.1	33.8	38.6	40.2	-	-	-	-	-	-
wd1 Tang et al. 2025	77.2	80.8	82.3	33.3	34.4	39.0	-	-	-	-	-	-
$d$-TreeRPO Pan et al. 2025	-	81.2	82.6	-	37.7	38.9	-	-	-	-	-	-
ESPO Ou et al. 2025	80.0	82.3	83.7	36.0	39.0	43.4	28.1	42.1	50.0	47.4	44.6	44.2
GDPO Rojas et al. 2025	78.4	82.8	84.5	33.2	39.6	41.4	26.2	39.6	39.0	43.6	50.6	47.1
SPG Wang et al. 2025a	78.5	86.1	84.5	33.4	40.0	41.8	-	-	-	-	-	-
JustGRPO	83.8	89.1	89.8	39.0	45.1	45.2	37.8	49.4	48.7	50.6	52.4	49.0

表 1：在 LLaDA-Instruct 上 RL 后训练方法的系统级对比。JustGRPO 优于现有扩散专用 RL。LLaDA-1.5 与 LLaDOU 不参与对比：前者在显著更大规模私有数据上训练，后者用辅助模块改基座架构。

5.2 JustGRPO Preserves Parallel Decoding

5.2 JustGRPO 保留并行解码

Figure 7: JustGRPO preserves the parallel decoding capability of dLLMs. Interestingly, when compared to original instruct model, accuracy gains are larger with more parallel tokens, likely due to more robust reasoning capabilities after JustGRPO training. We adopt training-free EB-sampler Ben-Hamu et al. 2025 for parallel decoding. Generation length is set to 256.

图 7：JustGRPO 保留 dLLM 的并行解码能力。 有趣的是，相对原 instruct 模型，并行词元越多精度增益越大，或因 JustGRPO 训练后推理更稳健。并行解码采用免训练 EB-sampler Ben-Hamu 等 2025。生成长度 256。

We further investigate whether the AR training compromises dLLMs’ parallel decoding capabilities. We employ the training-free Entropy Bounded (EB) Sampler Ben-Hamu et al. 2025 to evaluate inference performance under varying degrees of parallelism (tokens per step).

我们进一步检验 AR 训练是否损害 dLLM 并行解码。用免训练熵有界（EB）采样器 Ben-Hamu 等 2025，在不同并行度（每步词元数）下评估推理表现。

Figure 7 shows that our model retains full compatibility with parallel decoding and exhibits a strictly superior trade-off between speed and accuracy than the original LLaDA-Instruct model. Interestingly, the performance gain becomes more pronounced as parallelism increases. While the baseline’s performance degrades sharply with high parallel steps, our model maintains relatively more stable. For instance, on MBPP, the accuracy gap expands from +10.6% at conservative settings (1 token/step) to +25.5% at aggressive settings ($\sim$5 tokens/step).

图 7 显示模型与并行解码完全兼容，并在速度—精度权衡上严格优于原 LLaDA-Instruct。有趣的是，并行度越高增益越明显。基线在高并行步数下急剧退化，我们的模型相对更稳。例如 MBPP 上，保守设置（1 词元/步）精度差距 +10.6%，激进设置（$\sim$5 词元/步）扩大到 +25.5%。

This observation suggests that JustGRPO does not merely fit a specific decoding path; rather, it learns a more robust reasoning manifold that is resilient to the approximation errors inherent in parallel sampling. In other words, the AR formulation acts as a training “scaffold” that refines the joint distribution $p(o)$ more effectively, providing a more stable foundation for inference.

这表明 JustGRPO 并非仅拟合特定解码路径，而是学到对并行采样固有近似误差更有韧性的稳健推理流形。换言之，AR 形式化充当训练「脚手架」，更有效地精炼联合分布 $p(o)$，为推理提供更稳基础。

6 Related Work

6 相关工作

Diffusion language models.

Inspired by the success of diffusion models in continuous image domains Ho et al. 2020; Rombach et al. 2022, recent work has extended diffusion to discrete text generation. Early approaches operating in continuous embedding spaces Li et al. 2022; Gong et al. 2022; Han et al. 2022 suffered from optimization and discretization issues. In contrast, masked diffusion models Lou et al. 2023; Sahoo et al. 2024; Shi et al. 2024; Ou et al. 2024, which define the diffusion process directly in the discrete token space via random masking, have become the dominant paradigm. Notably, recent large-scale models such as LLaDA Nie et al. 2025 and Dream Ye et al. 2025 demonstrate scalability and performance competitive with autoregressive (AR) models. The emergence of such large-scale diffusion language models has spurred a growing body of follow-up studies exploring their distinctive modeling and inference properties. Among various advantages discussed in the literature, two aspects have received particular attention. First, dLLMs naturally support parallel decoding, enabling significant inference acceleration Wu et al. 2025b; Ben-Hamu et al. 2025; Labs et al. 2025; DeepMind 2025; Song et al. 2025. Second, their non-autoregressive formulation allows for arbitrary-order token generation, which has been hypothesized to benefit complex reasoning by relaxing strict left-to-right constraints Ye et al. 2024; Kim et al. 2025.

扩散语言模型。

受连续图像域扩散成功启发 Ho 等 2020；Rombach 等 2022，近期工作把扩散扩展到离散文本。早期在连续嵌入空间操作的方法 Li 等 2022；Gong 等 2022；Han 等 2022 受优化与离散化问题困扰。相比之下，直接在离散词元空间经随机掩码定义扩散的掩码扩散模型 Lou 等 2023；Sahoo 等 2024；Shi 等 2024；Ou 等 2024 成为主导范式。大规模模型如 LLaDA Nie 等 2025 与 Dream Ye 等 2025 展示出可扩展性及与自回归（AR）竞争的性能。大规模扩散语言模型催生大量后续研究探索其建模与推理特性。文献尤关注两点：其一，dLLM 天然支持并行解码、显著加速推理 Wu 等 2025b；Ben-Hamu 等 2025；Labs 等 2025；DeepMind 2025；Song 等 2025；其二，非自回归形式允许任意序词元生成，被假设可通过放松严格从左到右约束利于复杂推理 Ye 等 2024；Kim 等 2025。

The value of order arbitrariness.

While early studies validated the utility of arbitrary order generation in constrained tasks like Sudoku and Zebra Puzzles Ye et al. 2024; Nie et al. 2025; Kim et al. 2025, recent research has begun to investigate its value in general reasoning domains. One line of research, exemplified by Dream-Coder Xie et al. 2025 and DiffuCoder Gong et al. 2025, shows that diffusion models can naturally exhibit non-standard, human-like decoding behaviors (e.g., sketch-first reasoning) without any explicit supervision on generation order. DiffuCoder Gong et al. 2025 further quantifies this phenomenon by demonstrating that higher sampling temperatures reduce AR-ness, with increased order randomness correlating with improved output diversity. Another line of research, exemplified by P2 Peng et al. 2025 and LLaDOU Huang et al. 2025b, explicitly optimizes the decoding strategy for better generation order, thus better performance. Despite these advances, it remains unclear whether the observed gains primarily arise from better exploitation of existing solution patterns encoded in the data and model, or whether order arbitrariness itself enables qualitatively new reasoning strategies that are unattainable under a purely autoregressive decoding regime.

顺序任意性的价值。

早期研究在数独、斑马谜等约束任务验证任意序效用 Ye 等 2024；Nie 等 2025；Kim 等 2025，近期开始考察通用推理领域。一条线以 Dream-Coder Xie 等 2025、DiffuCoder Gong 等 2025 为代表，显示扩散模型可自然呈现非标准、类人解码行为（如先草图再推理），无需对生成顺序显式监督。DiffuCoder 进一步量化：更高采样温度降低 AR 性，顺序随机性增加与输出多样性相关。另一条线如 P2 Peng 等 2025、LLaDOU Huang 等 2025b 显式优化解码策略以改善顺序与性能。尽管有进展，仍不清楚收益主要来自更好利用数据与模型中已有解题模式，还是顺序任意性本身启用了纯自回归体制下不可及的质性新推理策略。

Reinforcement learning for diffusion language models.

Reinforcement learning for dLLMs faces structural optimization hurdles distinct from the autoregressive paradigm, primarily stemming from the combinatorial explosion of denoising trajectories. While early attempts Zhao et al. 2025; Yang et al. 2025; Gong et al. 2025 sought to adapt token-level formulations directly, they were fundamentally limited by ill-defined state transitions, necessitating reliance on unstable mean-field approximations. Consequently, the field has shifted toward sequence-level perspectives Zhu et al. 2025; Wang et al. 2025a; Rojas et al. 2025; Ou et al. 2025, employing various surrogates to approximate the intractable marginal likelihood. Yet, a critical off-policy misalignment persists across these methods: the heuristic-guided sampling required for efficient exploration diverges from the underlying diffusion prior, rendering gradients biased without principled correction Schulman et al. 2015. Two notable exceptions partially address these issues. LLaDOU Huang et al. 2025b explicitly models token position selection via an auxiliary policy, enabling direct estimation of trajectory likelihoods, while TraceRL Wang et al. 2025d aligns optimization with inference traces through a shrinkage-based step-wise MDP formulation. Nevertheless, these approaches remain committed to preserving the full arbitrary-order generation mechanism, implicitly treating its structural complexity as indispensable, whereas we question whether effective RL training can be achieved more simply by reexamining the necessity of arbitrary order itself.

扩散语言模型的强化学习。

dLLM 的 RL 面临与自回归范式不同的结构优化障碍，主因是去噪轨迹组合爆炸。早期尝试 Zhao 等 2025；Yang 等 2025；Gong 等 2025 直接适配词元级形式化，却受病态状态转移限制，不得不依赖不稳的平均场近似。领域遂转向序列级视角 Zhu 等 2025；Wang 等 2025a；Rojas 等 2025；Ou 等 2025，用各类代理逼近难处理边缘似然。但这些方法仍存在关键 off-policy 失配：高效探索所需的启发式引导采样偏离底层扩散先验，无原则校正则梯度有偏 Schulman 等 2015。两个例外部分应对：LLaDOU Huang 等 2025b 用辅助策略显式建模词元位置选择以直接估计轨迹似然；TraceRL Wang 等 2025d 用基于收缩的逐步 MDP 使优化与推理轨迹对齐。不过它们仍执着保留完整任意序机制，隐含视其结构复杂度为不可或缺；我们则质疑：通过重新审视任意序必要性，有效 RL 训练是否可以更简单。

7 Conclusion

7 结论

The intuitive appeal of diffusion language models (dLLMs) lies in their order arbitrariness, often perceived as a superior mechanism for navigating complex reasoning paths. Our study reveals a counter-intuitive reality: this unrestricted flexibility in fact narrows the reasoning potential. By allowing the model to bypass high-entropy tokens, effectively skipping the most demanding logical branches, arbitrary-order generation acts as an exploitation mechanism that prioritizes greedy optimization of individual trajectories at the expense of broader solution coverage.

扩散语言模型（dLLM）的直观吸引力在于顺序任意性，常被视为导航复杂推理路径的优越机制。我们的研究揭示反直觉现实：这种无约束灵活性实际上收窄推理潜力。允许模型绕开高熵词元、跳过最吃力的逻辑分支，任意序生成充当利用机制，优先贪心优化单条轨迹，牺牲更广解覆盖。

Therefore, the elicitation of reasoning capability for dLLMs can be simpler. By operating dLLMs in a standard autoregressive manner, we enable the direct application of Group Relative Policy Optimization (GRPO) without any complex adaptations tailored for order arbitrariness. This intentional constraint paradoxically yields a significant improvement in reasoning performance, while fully preserving the parallel decoding capabilities of dLLMs. By returning to the basic, natural left-to-right order of language modeling, we hope to encourage a re-examination of its real value in the development of next-generation language models.

因此，激发 dLLM 推理能力可以更简单。以标准自回归方式运行 dLLM，即可直接应用组相对策略优化（GRPO），无需为顺序任意性做复杂适配。这一有意约束反而显著提升推理表现，同时完整保留 dLLM 并行解码能力。回归语言建模基本的、自然的从左到右顺序，我们希望促使重新审视其在下一代语言模型发展中的真实价值。

References

参考文献

Achiam et al. (2023) Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Ben-Hamu et al. (2025) Ben-Hamu, H., Gat, I., Severo, D., Nolte, N., and Karrer, B. Accelerated sampling from masked diffusion models via entropy bounded unmasking. arXiv preprint arXiv:2505.24857 , 2025.

Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020.

Chen (2021) Chen, M. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Cheng et al. (2025) Cheng, D., Huang, S., Zhu, X., Dai, B., Zhao, W. X., Zhang, Z., and Wei, F. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506.14758 , 2025.

DeepMind (2025) DeepMind. Gemini diffusion, 2025. URL https://deepmind.google/models/gemini-diffusion/ .

Gong et al. (2022) Gong, S., Li, M., Feng, J., Wu, Z., and Kong, L. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933 , 2022.

Gong et al. (2025) Gong, S., Zhang, R., Zheng, H., Gu, J., Jaitly, N., Kong, L., and Zhang, Y. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint arXiv:2506.20639 , 2025.

Han et al. (2022) Han, X., Kumar, S., and Tsvetkov, Y. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17432 , 2022.

Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Huang et al. (2025a) Huang, G., Xu, T., Wang, M., Yi, Q., Gong, X., Li, S., Xiong, R., Li, K., Jiang, Y., and Zhou, B. Low-probability tokens sustain exploration in reinforcement learning with verifiable reward. arXiv preprint arXiv:2510.03222 , 2025a.

Huang et al. (2025b) Huang, Z., Chen, Z., Wang, Z., Li, T., and Qi, G.-J. Reinforcing the diffusion chain of lateral thought with diffusion language models. arXiv preprint arXiv:2505.10446 , 2025b.

Kim et al. (2025) Kim, J., Shah, K., Kontonis, V., Kakade, S., and Chen, S. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768 , 2025.

Labs et al. (2025) Labs, I., Khanna, S., Kharbanda, S., Li, S., Varma, H., Wang, E., Birnbaum, S., Luo, Z., Miraoui, Y., Palrecha, A., et al. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506.17298 , 2025.

Li et al. (2022) Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-lm improves controllable text generation. Advances in neural information processing systems , 35:4328–4343, 2022.

Liu et al. (2025) Liu, M., Diao, S., Lu, X., Hu, J., Dong, X., Choi, Y., Kautz, J., and Dong, Y. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.24864 , 2025.

Lou et al. (2023) Lou, A., Meng, C., and Ermon, S. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834 , 2023.

Nie et al. (2025) Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. Large language diffusion models. arXiv preprint arXiv:2502.09992 , 2025.

Ou et al. (2024) Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736 , 2024.

Ou et al. (2025) Ou, J., Han, J., Xu, M., Xu, S., Xie, J., Ermon, S., Wu, Y., and Li, C. Principled rl for diffusion llms emerges from a sequence-level perspective. arXiv preprint arXiv:2512.03759 , 2025.

Pan et al. (2025) Pan, L., Tao, S., Zhai, Y., Fu, Z., Fang, L., He, M., Zhang, L., Liu, Z., Ding, B., Liu, A., et al. d-treerpo: Towards more reliable policy optimization for diffusion language models. arXiv preprint arXiv:2512.09675 , 2025.

Peng et al. (2025) Peng, Z., Bezemek, Z., Patel, S., Rector-Brooks, J., Yao, S., Tong, A., and Chatterjee, P. Path planning for masked diffusion model sampling. ArXiv , abs/2502.03540, 2025. URL https://api.semanticscholar.org/CorpusID:276161145 .

Rojas et al. (2025) Rojas, K., Lin, J., Rasul, K., Schneider, A., Nevmyvaka, Y., Tao, M., and Deng, W. Improving reasoning for diffusion language models via group diffusion policy optimization. arXiv preprint arXiv:2510.08554 , 2025.

Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Sahoo et al. (2024) Sahoo, S., Arriola, M., Schiff, Y., Gokaslan, A., Marroquin, E., Chiu, J., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems , 37:130136–130184, 2024.

Schulman et al. (2015) Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning , pp. 1889–1897. PMLR, 2015.

Shao et al. (2024) Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Xiao, M., Li, Y. K., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.

Shi et al. (2024) Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems , 37:103131–103167, 2024.

Song et al. (2025) Song, Y., Zhang, Z., Luo, C., Gao, P., Xia, F., Luo, H., Li, Z., Yang, Y., Yu, H., Qu, X., et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02193 , 2025.

Tang et al. (2025) Tang, X., Dolga, R., Yoon, S., and Bogunovic, I. wd1: Weighted policy optimization for reasoning in diffusion language models. arXiv preprint arXiv:2507.08838 , 2025.

Wang et al. (2025a) Wang, C., Rashidinejad, P., Su, D., Jiang, S., Wang, S., Zhao, S., Zhou, C., Shen, S. Z., Chen, F., Jaakkola, T., et al. Spg: Sandwiched policy gradient for masked diffusion language models. arXiv preprint arXiv:2510.09541 , 2025a.

Wang et al. (2025b) Wang, J., Liu, R., Zhang, F., Li, X., and Zhou, G. Stabilizing knowledge, promoting reasoning: Dual-token constraints for rlvr. arXiv preprint arXiv:2507.15778 , 2025b.

Wang et al. (2025c) Wang, S., Yu, L., Gao, C., Zheng, C., Liu, S., Lu, R., Dang, K., Chen, X., Yang, J., Zhang, Z., et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939 , 2025c.

Wang et al. (2025d) Wang, Y., Yang, L., Li, B., Tian, Y., Shen, K., and Wang, M. Revolutionizing reinforcement learning framework for diffusion large language models. arXiv preprint arXiv:2509.06949 , 2025d.

Wu et al. (2025a) Wu, C., Zhang, H., Xue, S., Diao, S., Fu, Y., Liu, Z., Molchanov, P., Luo, P., Han, S., and Xie, E. Fast-dllm v2: Efficient block-diffusion llm. arXiv preprint arXiv:2509.26328 , 2025a.

Wu et al. (2025b) Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., and Xie, E. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618 , 2025b.

Xie et al. (2025) Xie, Z., Ye, J., Zheng, L., Gao, J., Dong, J., Wu, Z., Zhao, X., Gong, S., Jiang, X., Li, Z., et al. Dream-coder 7b: An open diffusion language model for code. arXiv preprint arXiv:2509.01142 , 2025.

Yang et al. (2025) Yang, L., Tian, Y., Li, B., Zhang, X., Shen, K., Tong, Y., and Wang, M. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505.15809 , 2025.

Ye et al. (2024) Ye, J., Gao, J., Gong, S., Zheng, L., Jiang, X., Li, Z., and Kong, L. Beyond autoregression: Discrete diffusion for complex reasoning and planning. arXiv preprint arXiv:2410.14157 , 2024.

Ye et al. (2025) Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487 , 2025.

Yue et al. (2025) Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Song, S., and Huang, G. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837 , 2025.

Zeng et al. (2025) Zeng, H., Jiang, D., Wang, H., Nie, P., Chen, X., and Chen, W. Acecoder: Acing coder rl via automated test-case synthesis. arXiv preprint arXiv:2502.01718 , 2025.

Zhang et al. (2025) Zhang, C., Neubig, G., and Yue, X. On the interplay of pre-training, mid-training, and rl on reasoning language models. arXiv preprint arXiv:2512.07783 , 2025.

Zhao et al. (2025) Zhao, S., Gupta, D., Zheng, Q., and Grover, A. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216 , 2025.

Zhu et al. (2025) Zhu, F., Wang, R., Nie, S., Zhang, X., Wu, C., Hu, J., Zhou, J., Chen, J., Lin, Y., Wen, J.-R., et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505.19223 , 2025.

Appendix

附录

Appendix A Experimental Details

附录 A 实验细节

A.1 Data Preparation

A.1 数据准备

For mathematical reasoning tasks, we train on the official training split of each dataset, following the standard protocol in Zhao et al. 2025; Ou et al. 2025. For code generation tasks, we adopt the AceCoder-87K dataset Zeng et al. 2025. Following the data processing pipeline of DiffuCoder Gong et al. 2025, we further select 21K challenging samples from AceCoder-87K that are equipped with verifiable unit tests.

数学推理任务在各数据集官方训练集上训练，遵循 Zhao 等 2025；Ou 等 2025 标准协议。代码生成采用 AceCoder-87K Zeng 等 2025；按 DiffuCoder Gong 等 2025 数据处理流程，进一步选出 21K 带可验证单元测试的困难样本。

A.2 Training Configuration

A.2 训练配置

Our training setup largely follows Huang et al. 2025b. Unlike Huang et al. 2025b, we perform reinforcement learning directly on the dLLM without introducing training modules. During the rollout phase, we adopt exact autoregressive sampling, which allows direct log-probability computation under the standard GRPO formulation. We also reduce the total number of training steps, as JustGRPO exhibits fast and stable convergence. All experiments are conducted on 16$\times$ NVIDIA H100 GPUs. Detailed hyperparameters are reported in Table 2.

训练设置大体遵循 Huang 等 2025b。与之不同，我们直接在 dLLM 上做强化学习、不引入训练模块。rollout 阶段采用精确自回归采样，从而可在标准 GRPO 形式下直接计算对数概率。亦减少总训练步数，因 JustGRPO 收敛快且稳。全部实验在 16$\times$ NVIDIA H100 上完成。超参见表 2。

Hyperparameter	Value
Base Model	LLaDA 8B Instruct
RL Algorithm	GRPO
Optimizer	AdamW
Learning Rate	$5\times 10^{-6}$
LR Scheduler	Constant
Weight Decay	0.0
Optimizer Betas $(\beta_{1},\beta_{2})$	$(0.9,0.999)$
Global Batch Size	64
Group Size ($G$)	16
Policy Update Steps	1
Training Steps	125
Max Completion Length	256
Sampling Temperature	1.0
KL Penalty Coefficient	0.0

Table 2: Training hyperparameters for JustGRPO.

Hyperparameter	Value
Base Model	LLaDA 8B Instruct
RL Algorithm	GRPO
Optimizer	AdamW
Learning Rate	$5\times 10^{-6}$
LR Scheduler	Constant
Weight Decay	0.0
Optimizer Betas $(\beta_{1},\beta_{2})$	$(0.9,0.999)$
Global Batch Size	64
Group Size ($G$)	16
Policy Update Steps	1
Training Steps	125
Max Completion Length	256
Sampling Temperature	1.0
KL Penalty Coefficient	0.0

表 2：JustGRPO 的训练超参。

A.3 Reward Function

A.3 奖励函数

Mathematical Reasoning Tasks.

We employ a binary reward scheme. Each completion receives a reward of 1 if and only if the final answer is mathematically equivalent to the ground-truth solution, and 0 otherwise, following Huang et al. 2025b.

数学推理任务。

采用二元奖励：当且仅当最终答案与标准解数学等价时奖励为 1，否则 0，遵循 Huang 等 2025b。

Code Generation Tasks.

For code generation, the total reward $r$ is defined as a weighted sum of a correctness reward $r_{\text{code}}$ and a format reward $r_{\text{format}}$:

代码生成任务。

总奖励 $r$ 定义为正确性奖励 $r_{\text{code}}$ 与格式奖励 $r_{\text{format}}$ 的加权和：

$$r=r_{\text{code}}+r_{\text{format}}. \tag{9}$$

• • Correctness reward ($r_{\text{code}}$): Defined as the pass rate (ranging from 0 to 1) of the generated code on the provided unit tests. This term is only evaluated when $r_{\text{format}}=1$.

• • 正确性奖励（$r_{\text{code}}$）：生成代码在给定单元测试上的通过率（0 到 1）。仅当 $r_{\text{format}}=1$ 时评估该项。

• • Format reward ($r_{\text{format}}$): A heuristic reward designed to encourage syntactically valid outputs. – $1.0$: A valid Markdown code block with correct Python syntax. – $0.5$: A valid Markdown code block that contains syntax errors. – $0.0$: Failure to generate a valid Markdown code block.

• • 格式奖励（$r_{\text{format}}$）：鼓励句法合法输出的启发式。– $1.0$：合法 Markdown 代码块且 Python 句法正确。– $0.5$：合法 Markdown 代码块但有句法错误。– $0.0$：未能生成合法 Markdown 代码块。

Appendix B More Analysis Results on Reasoning Boundary

附录 B 推理边界的更多分析结果

To further validate the robustness of our findings about reasoning boundary in Section 3, we conduct extended analyses on the HumanEval benchmark using the LLaDA-Instruct model Nie et al. 2025. We investigate the impact of temperature, sampling strategies, and decoding block sizes on reasoning performance.

为进一步验证第 3 节关于推理边界发现的稳健性，我们在 HumanEval 上用 LLaDA-Instruct Nie 等 2025 做扩展分析，考察温度、采样策略与解码块大小对推理表现的影响。

B.1 Temperature analysis.

B.1 温度分析。

Figure 8: Pass@$k$ comparison with different temperatures.

图 8：不同温度下的 Pass@$k$ 对比。

As shown in Figure 8, the AR mode exhibits a standard pattern: performance peaks at moderate temperatures ($T\approx 0.6$) and degrades when $T>1.0$, whereas the Arbitrary Order mode attains its peak performance at higher temperatures.

如图 8，AR 模式呈标准形态：中等温度（$T\approx 0.6$）表现最好，$T>1.0$ 时退化；任意序则在更高温度达到峰值。

This observation aligns with the entropy degradation mechanism discussed in Section 3: the diffusion sampler inherently suppresses uncertainty at critical branching points, thereby requiring higher temperatures to induce sufficient exploration. Crucially, even under these optimized settings, the arbitrary order mode fails to match the reasoning potential of the AR mode. As illustrated in the “Optimal” comparison setting (bottom right of Figure 8), the best-performing AR configuration still outperforms the optimal arbitrary order baseline in scaling behavior. A plausible explanation is that excessively high temperatures in arbitrary order decoding inject noise into tokens that require high determinism (e.g., code syntax or mathematical suffixes), leading to nonsensical outputs and degraded results Wang et al. 2025c.

这与第 3 节讨论的熵退化机制一致：扩散采样器在关键分支点固有地压制不确定性，因而需要更高温度才能诱导足够探索。关键的是，即便在这些优化设置下，任意序仍赶不上 AR 的推理潜力。如图 8 右下「Optimal」对比，最佳 AR 配置在缩放行为上仍优于最优任意序基线。一种合理解释是：任意序中过高温度会向需要高确定性的词元（如代码句法或数学后缀）注入噪声，导致无意义输出与性能下降 Wang 等 2025c。

B.2 Different sampling algorithms.

B.2 不同采样算法。

We also experiment with different sampling algorithms, such as negative entropy sampling (Neg-Entropy)  Ye et al. 2025 and top-$k$ margin sampling (Margin)  Kim et al. 2025 in Figure 9(a).

我们还实验负熵采样（Neg-Entropy） Ye 等 2025 与 top-$k$ 间隔采样（Margin） Kim 等 2025，见图 9(a)。

Figure 9: (a) Sampling algorithm comparison. (b) Correlation between different sampling algorithms and AR in per-problem accuracy. Algorithms with higher Pass@$k$ performance tend to show higher correlation with AR in problem-level performance.

图 9：(a) 采样算法对比。(b) 不同采样算法与 AR 在逐题精度上的相关性。Pass@$k$ 更高的算法往往与 AR 在题目级表现上相关性更高。

The results show that although more sophisticated sampling algorithms can achieve better Pass@$k$ performance than default confidence-based sampling, they still cannot catch up with AR order. Meanwhile, these better Pass@$k$ algorithms also show slightly worse Pass@$1$ performance compared to confidence-based sampling, making the overall Pass@$k$ curve closer to AR order’s Pass@$k$ curve. To investigate this similarity, we calculate the per-problem accuracy correlation between different sampling algorithms and the AR mode (Figure 9(b)). We observe that algorithms with higher scaling potential (higher Pass@$128$) consistently show stronger correlation with AR, with the most effective method (Neg-Entropy) achieving a correlation of $0.970$. This suggests that sampling algorithms with better Pass@$k$ tend to behave more like AR in performance characteristics.

结果表明，更复杂采样算法可比默认基于置信的采样取得更好 Pass@$k$，但仍赶不上 AR 顺序。同时这些更好 Pass@$k$ 的算法相对置信采样的 Pass@$1$ 略差，使整体 Pass@$k$ 曲线更接近 AR。为考察相似性，我们计算各采样算法与 AR 模式的逐题精度相关（图 9(b)）。观察到缩放潜力更高（更高 Pass@$128$）的算法与 AR 相关更强，最有效的 Neg-Entropy 相关达 $0.970$。说明更好 Pass@$k$ 的采样算法在表现特征上更像 AR。

B.3 Block size analysis.

B.3 块大小分析。

Figure 10: Pass@$k$ comparison with different semi-autoregressive block sizes of arbitrary order (AO) generation. Smaller block sizes explicitly restrict the model’s order flexibility, leading to a more AR-like behavior and better Pass@$k$ performance.

图 10：任意序（AO）生成在不同半自回归块大小下的 Pass@$k$ 对比。更小块大小显式限制顺序灵活性，行为更像 AR，Pass@$k$ 更好。

We analyze the effect of the semi-autoregressive block size in Figure 10. The AR Order (which is effectively block size of 1) maintains a clear advantage across all tested block sizes. Furthermore, we observe a trend of improvement in Pass@$k$ as the block size decreases (from 128 to 8). Since smaller block sizes explicitly restrict the model’s order flexibility, this trend is aligned with our finding that less order flexibility leads to better reasoning potential.

我们在图 10 分析半自回归块大小影响。AR 顺序（等效块大小 1）在所有测试块大小上保持明显优势。此外，块大小从 128 降到 8 时 Pass@$k$ 呈改善趋势。更小块显式限制顺序灵活性，与「更少顺序灵活性带来更好推理潜力」的发现一致。

Appendix C More Results on Entropy Degradation

附录 C 熵退化的更多结果

Figure 11: Entropy comparison results on more forking tokens.

图 11：更多分叉词元上的熵对比结果。

To validate the robustness of the “entropy degradation” phenomenon observed in Section 3, we extended our analysis to a wider range of logical connectors. We conducted experiments on a comprehensive set of common logical connectors that typically serve as pivotal decision points in reasoning chains.

为验证第 3 节「熵退化」现象的稳健性，我们将分析扩展到更广范围的逻辑连接词，在推理链中通常充当关键决策点的常见连接词集合上实验。

The results, shown in Figure 11, demonstrate that the phenomenon is consistent across these diverse tokens. Similar to the primary findings, the AR order maintains higher average entropy at these forks, indicating active reasoning and decision-making. Conversely, the Arbitrary Order consistently results in lower entropy, confirming the premature collapse of branching possibilities.

图 11 结果说明现象在这些多样词元上一致。与主要发现类似，AR 顺序在这些分叉处保持更高平均熵，表明主动推理与决策；任意序则一致导致更低熵，确认分支可能性过早塌缩。

The specific forking tokens evaluated in this extended experiment include: “Therefore”, “Thus”, “So”, “Since”, “When”, “Given”, “However”, “Let”, “First”, “Then”, “Next”, “Finally”, “Now”, “Similarly”, “Calculate”, “Solving”, “Notice”, “Specifically”, “Follows”, “Because”, “But”, “Or”, “Consider”, “Also”, “Express”, and “Write”.

本扩展实验评估的具体分叉词元包括：“Therefore”、“Thus”、“So”、“Since”、“When”、“Given”、“However”、“Let”、“First”、“Then”、“Next”、“Finally”、“Now”、“Similarly”、“Calculate”、“Solving”、“Notice”、“Specifically”、“Follows”、“Because”、“But”、“Or”、“Consider”、“Also”、“Express” 与 “Write”。

Appendix D Training Efficiency Analysis of JustGRPO

附录 D JustGRPO 训练效率分析

Figure 12: Training efficiency on GSM8K (Wall-clock Time vs. Accuracy). The approximation-based baseline (ESPO) suffers from early saturation. In contrast, JustGRPO (Ours) exhibits better accuracy-wall time trade-off despite the theoretical overhead of exact likelihood estimation. Moreover, Ours-Fast (gradient update on top-25% entropy tokens) illustrates the potential for further acceleration via simple engineering optimizations. Wall-clock time is measured on 16$\times$H100 GPUs.

图 12：GSM8K 上的训练效率（墙钟时间 对 精度）。基于近似的基线（ESPO）早期饱和。相比之下，JustGRPO（Ours）尽管精确似然估计有理论开销，仍展现更好的精度—墙钟权衡。此外 Ours-Fast（仅对熵最高 25% 词元做梯度更新）说明经简单工程优化仍可进一步加速。墙钟时间在 16$\times$H100 上测量。

Applying GRPO to dLLMs presents a structural trade-off. Unlike autoregressive models that benefit from causal masking, dLLMs require independent likelihood evaluation at each position, inherently incurring additional computational overhead. While approximation-based methods Zhao et al. 2025; Ou et al. 2025 bypass this cost by forgoing exact estimation, we demonstrate that adhering to exact likelihoods yields a superior trade-off between accuracy and wall-clock time.

对 dLLM 应用 GRPO 存在结构权衡。自回归可受益于因果掩码，dLLM 需在每位独立评估似然，固有额外开销。基于近似的方法 Zhao 等 2025；Ou 等 2025 通过放弃精确估计绕开该成本；我们表明坚持精确似然在精度与墙钟时间上给出更优权衡。

Figure 12 compares JustGRPO against ESPO Ou et al. 2025, a representative approximation-based baseline. It can be observed that ESPO saturates earlier, effectively plateauing around 10 hours. In contrast, JustGRPO exhibits a superior efficiency frontier: it surpasses the baseline’s peak accuracy in less time and maintains a continuous scaling trend. Furthermore, motivated by the observation that reasoning is primarily steered by a subset of high-entropy forking tokens Wang et al. 2025c, we demonstrate that JustGRPO allows for further acceleration via a simple heuristic: restricting gradient computation to the top-25% high-entropy tokens (Ours-Fast). This modification yields faster convergence and even slightly better performance. We prioritize simplicity in the current implementation and leave further efficiency optimizations for future work.

图 12 将 JustGRPO 与代表性近似基线 ESPO Ou 等 2025 对比。可见 ESPO 更早饱和，约 10 小时即平台化。JustGRPO 展现更优效率前沿：更短时间超过基线峰值精度，并保持持续缩放。此外，受「推理主要由高熵分叉词元子集主导」观察启发 Wang 等 2025c，我们演示可用简单启发式进一步加速：仅对熵最高 25% 词元计算梯度（Ours-Fast）。该修改收敛更快，表现甚至略好。当前实现优先简洁，进一步效率优化留待未来。