Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

marginparsep has been altered. topmargin has been altered. marginparpush has been altered.

marginparsep 已被改动。topmargin 已被改动。marginparpush 已被改动。

The page layout violates the ICML style.

页面布局违反了 ICML 样式。

Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you.

请不要更改页面布局，也不要引入 geometry、savetrees 或 fullpage 等会替你改布局的宏包。

We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again.

我们无法可靠地撤销对样式的任意改动。请移除有问题的宏包或会改变布局的命令后再试。

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

强化学习真的能把 LLM 的推理能力激励到超越基座模型吗？

Yang Yue${}^{\,1\,*\,\dagger\,}$****, Zhiqi Chen${}^{\,1\,*}$, Rui Lu${}^{\,1\,}$, Andrew Zhao${}^{\,1\,}$, Zhaokai Wang${}^{\,2\,}$, Yang Yue${}^{\,1\,}$, Shiji Song${}^{\,1\,}$, and Gao Huang${}^{\,1\,\textrm{\Letter}}$

${}^{1\,}$LeapLab, Tsinghua University  ${}^{2\,}$Shanghai Jiao Tong University

∗ Equal Contribution  † Project Lead  ${}^{\textrm{\Letter}}$ Corresponding Author

Figure 1: (Left) The effect of current RLVR on LLM’s reasoning ability. Search trees are generated by repeated sampling from the base and RLVR-trained models for a given problem. Grey indicates paths that are unlikely to be sampled by the model, while black indicates paths that are likely to be sampled. Green indicates correct paths, which has positive rewards. Our key finding is that all reasoning paths in the RLVR model are already present in the base model. For certain problems like Problem A, RLVR training biases the distribution toward rewarded paths, improving sampling efficiency. However, this comes at the cost of reduced scope of reasoning capacity: For other problems like Problem B, the base model contains the correct path, whereas that of the RLVR model does not. (Right) As RLVR training progresses, the average performance (i.e., pass@1) improves, but the coverage of solvable problems (i.e., pass@256) decreases, indicating a reduction in LLM’s reasoning boundary.

图 1：**（左）当前 RLVR 对 LLM 推理能力的影响。对给定问题，通过从基座模型与 RLVR 训练模型反复采样生成搜索树。灰色表示模型不太可能采样到的路径，黑色表示很可能采样到的路径。绿色表示有正奖励的正确路径。关键发现是：RLVR 模型中的所有推理路径在基座模型中业已存在。对问题 A 这类情形，RLVR 训练把分布偏向有奖励的路径，提高采样效率；代价是推理能力覆盖变窄：对问题 B，基座模型含正确路径，而 RLVR 模型的搜索树中没有。（右）**随着 RLVR 训练推进，平均表现（即 pass@1）上升，但可解问题覆盖（即 pass@256）下降，表明 LLM 的推理边界在收缩。

1 Introduction

1 引言

The development of reasoning-centric large language models (LLMs), such as OpenAI-o1 Jaech et al. (2024), DeepSeek-R1 Guo et al. (2025), and Kimi-1.5 Team et al. (2025), has significantly advanced the frontier of LLM capabilities, particularly in solving complex logical tasks involving mathematics and programming. In contrast to traditional instruction-tuned approaches that rely on human-curated annotations Achiam et al. (2023); Grattafiori et al. (2024), the key driver behind this leap forward is large-scale Reinforcement Learning with Verifiable Rewards (RLVR) Lambert et al. (2024); Guo et al. (2025). RLVR starts with a pretrained base model or one fine-tuned on long chains of thought (CoT) data, optimizing it via reinforcement learning based on simple, automatically computable rewards. These rewards are determined by whether the model’s output matches a ground-truth solution in mathematics or passes unit tests in code, thus enabling scalability without human labeling. This framework has gained significant attention due to its simplicity and practical effectiveness. In traditional RL settings such as game playing (e.g., Atari, Go), agents often autonomously discover new strategies and surpass even human-level performance through self-improvement Mnih et al. (2015); Silver et al. (2017). Inspired by this success, it is widely believed that RLVR similarly enables LLMs to autonomously develop novel reasoning patterns, including enumeration, self-reflection, and iterative refinement, surpassing the capabilities of their base models Guo et al. (2025). Consequently, RLVR has been considered a promising path toward continuously self-evolving LLMs, potentially bringing us closer to more powerful intelligence Guo et al. (2025).

以推理为中心的大语言模型（LLM）——如 OpenAI-o1（Jaech 等，2024）、DeepSeek-R1（Guo 等，2025）与 Kimi-1.5（Team 等，2025）——显著推进了 LLM 能力边界，尤其是在数学与编程等复杂逻辑任务上。相对依赖人工标注的传统指令微调（Achiam 等，2023；Grattafiori 等，2024），这一跃迁的关键驱动力是大规模可验证奖励强化学习（Reinforcement Learning with Verifiable Rewards，RLVR）（Lambert 等，2024；Guo 等，2025）。RLVR 从预训练基座模型、或在长思维链（CoT）数据上微调过的模型出发，用简单、可自动计算的奖励做强化学习：奖励由模型输出是否与数学标准答案一致、或是否通过代码单元测试决定，因而无需人工标注即可扩展。该框架因其简洁与实效受到广泛关注。在传统 RL 设定如游戏（Atari、围棋）中，智能体常能通过自我改进自主发现新策略，甚至超越人类水平（Mnih 等，2015；Silver 等，2017）。受此成功启发，人们普遍相信 RLVR 同样能让 LLM 自主发展出枚举、自我反思、迭代精炼 等新推理模式，从而超越基座模型能力（Guo 等，2025）。因此，RLVR 被视为通向持续自我演化 LLM、并更接近更强智能的有希望路径（Guo 等，2025）。

However, despite its empirical success, the underlying effectiveness of current RLVR remains underexamined. This raises a fundamental question: Does current RLVR genuinely enable LLMs to acquire novel reasoning abilities–similar to how traditional RL discovers new strategies through exploration–or does it simply utilize reasoning patterns already in the base model?

然而，尽管经验上很成功，当前 RLVR 的底层效力仍未得到充分审视。这引出一个根本问题：当前 RLVR 是否真的让 LLM 获得了新的推理能力——类似传统 RL 通过探索发现新策略——还是只是在利用基座模型中已有的推理模式？

To rigorously answer this question, we must first assess the reasoning capability boundaries of both base and RLVR-trained models. Traditional evaluation metrics rely on average score from greedy decoding or nucleus sampling Holtzman et al. (2020), which reflects average-case behavior. However, these metrics risk underestimating the true potential of a model, especially if it fails on difficult problems after limited attempts, despite being capable of solving them with more sampling. To overcome this limitation, we adopt the pass@k metric Brown et al. (2024), where a problem is considered solved if any of the $k$ sampled outputs is correct. By allowing multiple attempts, pass@k reveals whether a model has the potential to solve a problem. The average pass@k across a dataset thus reflects the proportion of problems a model can potentially solve within $k$ trials, offering a more robust view of its reasoning boundary. This provides a rigorous test on whether the RLVR training yields fundamentally transcending capacity, enabling the model to solve problems that the base model cannot.

要严谨回答，须先评估基座模型与 RLVR 训练模型的推理能力边界。传统评测依赖贪心解码或核采样（Holtzman 等，2020）的平均分，反映平均情形行为。但若模型在有限次尝试后在难题上失败、却本有能力用更多采样解出，这些指标会低估真实潜力。为克服这一局限，我们采用 pass@$k$ 指标（Brown 等，2024）：只要 $k$ 个采样输出中有任一正确，即视该题为已解。允许多次尝试后，pass@$k$ 能揭示模型是否有潜力解某题。数据集上的平均 pass@$k$ 因而反映模型在 $k$ 次试验内可解问题的比例，对推理边界给出更稳健的刻画。这也为检验 RLVR 训练是否带来根本性能力超越——使模型解出基座无法解的题——提供了严格测试。

Using the pass@k metric, we conduct extensive experiments across various benchmarks, covering multiple LLM families, model sizes, and RLVR algorithms to compare base models with their RLVR-trained counterparts. We uncover several surprising findings that offer a more comprehensive assessment of the effectiveness of current RLVR training and reveal the gap between existing RLVR methods and the ideal goals of RL-discovering genuinely new reasoning strategies:

用 pass@$k$ 指标，我们在多种基准、覆盖多个 LLM 家族、模型规模与 RLVR 算法上，系统比较基座模型与其 RLVR 训练对应版本。我们得到若干出人意料的发现，更全面地评估当前 RLVR 训练的效力，并揭示现有 RLVR 方法与「RL 通过探索发现真正新推理策略」这一理想目标之间的差距：

$\bullet$Current RLVR models often exhibit narrower reasoning coverage than their base models. In pass@$k$ curves, although RLVR models outperform their base models at small $k$, it is surprising that base models consistently surpass RLVR models across all benchmarks and LLM families as $k$ increases. This suggests that current RLVR training does not expand, and even reduce the scope of reasoning over solvable problems. Manual inspection of model responses shows that, for most problems, the base model can produce at least one correct CoT, implying that it can already generate correct reasoning paths for problems that were previously considered only solvable for RLVR models.

$\bullet$当前 RLVR 模型的推理覆盖往往比其基座模型更窄。在 pass@$k$ 曲线上，尽管小 $k$ 时 RLVR 优于基座，但当 $k$ 增大时，基座模型在所有基准与 LLM 家族上 始终反超 RLVR——说明当前 RLVR 训练并未扩展、甚至缩小了可解问题上的推理范围。人工检查模型回答显示：对多数问题，基座模型至少能产出一条正确 CoT，意味着对那些此前被认为「只有 RLVR 才能解」的题，基座其实已能生成正确推理路径。

$\bullet$Reasoning paths generated by current RLVR model already exist in its base model. To further investigate this phenomenon, we analyze the accuracy distribution. The results show that although RLVR improves average performance (i.e., pass@1) by sampling more efficiently on problems already solvable by the base model, it does not enable the model to solve new problems. Further perplexity analysis reveals that the reasoning paths produced by RLVR models already exist within the output distribution of the base model. These findings indicate that RLVR does not introduce fundamentally new reasoning capabilities and that the reasoning capacity of current RLVR models remains bounded by that of its base model. This effect of RLVR is illustrated in Figure 1 (left).

$\bullet$当前 RLVR 模型生成的推理路径，在其基座模型中业已存在。为进一步探究，我们分析准确率分布：结果显示，尽管 RLVR 通过在基座已能解的题上更高效地采样来提升平均表现（即 pass@1），它并不能让模型解出新题。进一步的困惑度分析表明，RLVR 模型产生的推理路径已落在基座模型的输出分布内。这些发现说明：RLVR 并未引入根本性的新推理能力，当前 RLVR 模型的推理能力仍受其基座模型所界定。图 1（左）示意了这一效应。

$\bullet$Current RLVR algorithms perform similarly and remain far from optimal. Treating the base model as an upper bound, we define the sampling efficiency gap ($\Delta_{\text{SE}}$), shown in Figure 8 (top), as the difference between an RL model’s pass@1 and the base model’s pass@$k$ (with $k=256$ as a proxy for upper-bound performance). This metric quantifies how closely an RL algorithm approaches the optimal bound. Across all algorithms (e.g., PPO, GRPO, Reinforce++), $\Delta_{\text{SE}}$ shows only minor variation yet remains consistently large, suggesting that current RLVR methods, while improving sampling efficiency, are still far from optimal.

$\bullet$当前 RLVR 算法表现相近，且远未达最优。以基座模型为上界，我们定义采样效率缺口（$\Delta_{\text{SE}}$），见图 8（上）：即 RL 模型的 pass@1 与基座模型 pass@$k$（取 $k=256$ 作为上界性能代理）之差。该指标量化 RL 算法距最优边界有多近。跨所有算法（如 PPO、GRPO、Reinforce++），$\Delta_{\text{SE}}$ 仅有微小差异，却始终很大——说明当前 RLVR 方法虽提升采样效率，仍远未达最优。

$\bullet$RLVR and distillation are fundamentally different. While RLVR improves reasoning scores by more efficiently sampling high-reward outputs, it does not elicit new reasoning capabilities and remains constrained within the base model’s capacity. In contrast, distillation can transfer new reasoning patterns from a stronger teacher to the student. As a result, distilled models often demonstrate an expanded reasoning scope beyond that of the base model.

$\bullet$RLVR 与蒸馏本质不同。RLVR 通过更高效地采样高奖励输出提升推理分数，但并不引出新推理能力，仍受基座能力约束。相对地，蒸馏可把更强教师的新推理模式迁移到学生。因此，蒸馏模型往往展现出超越基座模型的推理范围。

In conclusion, our findings show that current RLVR methods, while improving sampling efficiency, rarely elicit novel reasoning beyond the base model. This highlights a gap between existing RLVR methods and the goals of reinforcement learning, underscoring the need for improved RL paradigms such as better exploration, continual data scaling, fine-grained process signal, and multi-turn agent interaction.

总之，我们的发现表明：当前 RLVR 方法虽提升采样效率，却很少引出超越基座的新推理。这凸显了现有 RLVR 与强化学习目标之间的差距，也强调需要更好的 RL 范式——如更强的探索、持续数据扩展、细粒度过程信号，以及多轮智能体交互。

2 Preliminaries

2 预备知识

In this section, we first outline the fundamentals of RLVR, then introduce the pass@$k$ metric to evaluate reasoning boundaries, and explain why it is preferred over alternatives like best-of-$N$.

本节先概述 RLVR 基础，再引入用于评估推理边界的 pass@$k$ 指标，并说明为何它优于 best-of-$N$ 等替代方案。

Figure 2: Pass@$k$ curves of base models and their RLVR-trained counterparts across multiple mathematical benchmarks. When $k$ is small, RL-trained models outperform their base versions. However, as $k$ increases to the tens or hundreds, base models consistently catch up and surpass RL-trained models. More results on GSM8K and AMC23 can be found at Figure 10.

图 2：基座模型与其 RLVR 训练对应版本在多个数学基准上的 pass@$k$ 曲线。小 $k$ 时，RL 训练模型优于基座版本；但当 $k$ 增至数十或数百，基座模型 始终赶上并超越 RL 训练模型。GSM8K 与 AMC23 上更多结果见图 10。

2.1 Reinforcement Learning with Verifiable Rewards

2.1 可验证奖励强化学习

Verifiable Rewards. Let $\pi_{\theta}$ be an LLM with parameters $\theta$ that generates a token sequence $\mathbf{y}=(y_{1},\dots,y_{T})$ conditioned on a natural-language prompt $x$. A deterministic verifier $\mathcal{V}$ returns a binary reward: $r=\mathcal{V}(x,\mathbf{y})\in\{0,1\},$ where $r=1$ if and only if the model’s final answer is exactly correct. A format reward may also be added to encourage the model to explicitly separate the reasoning process from the final answer. The goal of RL is to learn a policy to maximize the expected reward: $J(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(\cdot|x)}[\,r\,]\right]$ , where $\mathcal{D}$ is the distribution of prompts.

可验证奖励。设 $\pi_{\theta}$ 为参数 $\theta$ 的 LLM，在自然语言提示 $x$ 条件下生成 token 序列 $\mathbf{y}=(y_{1},\dots,y_{T})$。确定性验证器 $\mathcal{V}$ 返回二值奖励：$r=\mathcal{V}(x,\mathbf{y})\in\{0,1\}$，当且仅当模型最终答案完全正确时 $r=1$。也可加入格式奖励，鼓励模型把推理过程与最终答案显式分开。RL 的目标是学习最大化期望奖励的策略：$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\mathbb{E}_{\mathbf{y}\sim\pi_{\theta}(\cdot|x)}[\,r\,]\right]$，其中 $\mathcal{D}$ 为提示分布。

RLVR Algorithms. Proximal Policy Optimization (PPO) Schulman et al. (2017) proposed using the following clipped surrogate to maximize the objective:

RLVR 算法。近端策略优化（PPO）（Schulman 等，2017）提出用如下裁剪代理目标最大化目标：

$$\mathcal{L}_{\text{CLIP}}=\mathbb{E}\left[\min\!\left(r_{t}(\theta)A_{t},\;\operatorname{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)A_{t}\right)\right], \tag{1}$$

where $r_{t}(\theta)=\pi_{\theta}(y_{t}|x,\mathbf{y}_{<t})/\pi_{\theta_{\text{old}}}(y_{t}|x,\mathbf{y}_{<t})$, and $A_{t}$ is the advantage estimated by a value network $V_{\phi}$. KL divergence term is optionally applied, to constrain the model from deviating too far from the original policy. More algorithms are introduced in Section C.5.

其中 $r_{t}(\theta)=\pi_{\theta}(y_{t}|x,\mathbf{y}_{<t})/\pi_{\theta_{\text{old}}}(y_{t}|x,\mathbf{y}_{<t})$，$A_{t}$ 为由价值网络 $V_{\phi}$ 估计的优势。可选地施加 KL 散度项，以约束模型不要偏离原策略过远。更多算法见第 C.5 节。

Policy Gradient. PPO and its variants belong to the policy gradient class of RL  Williams (1992); Sutton et al. (1998). These methods learn exclusively from on-policy samples, i.e., samples generated by the current LLM. In the context of verifiable rewards, the training objective generally maximizes the log-likelihood of samples with correct answers and minimizes the likelihood of those with incorrect answers.

策略梯度。PPO 及其变体属于策略梯度类 RL（Williams，1992；Sutton 等，1998）。这些方法仅从on-policy 样本——即当前 LLM 生成的样本——中学习。在可验证奖励语境下，训练目标一般是最大化答对样本的对数似然，最小化答错样本的似然。

Zero RL Training applies RL directly to the base model without any supervised fine-tuning (SFT) Guo et al. (2025). To clearly study the effect of RLVR, we follow this zero-RL setting for all math tasks using pretrained models as start model. However, for coding and visual reasoning tasks, open-source work typically uses instruction-tuned models as starting points, primarily due to the training instability and limited effectiveness of using a pure zero-RL setting. Following this convention, we compare the finetuned model with its RLVR-trained counterpart to focus solely on the effect of RLVR.

Zero RL 训练把 RL 直接施加于基座模型，不做任何监督微调（SFT）（Guo 等，2025）。为清晰研究 RLVR 效应，所有数学任务我们遵循该 zero-RL 设定，以预训练模型为起点。但对编码与视觉推理任务，开源工作通常以指令微调模型为起点，主要因纯 zero-RL 训练不稳定且效果有限。遵循这一惯例，我们比较微调模型与其 RLVR 训练对应版本，以单独聚焦 RLVR 的效应。

2.2 Metrics for LLM Reasoning Capacity Boundary

2.2 LLM 推理能力边界的度量

Pass@$k$ Metrics. Accurately measuring the reasoning ability boundary of base and RL models is challenging, as methods like greedy decoding or the average of nucleus samplings Holtzman et al. (2020) only reflect average-case performance. To accurately measure the reasoning ability boundary, we extend the commonly used pass@k metric from code generation Chen et al. (2021) to all tasks with verifiable rewards. Given a problem, we sample $k$ outputs from the model. The pass@$k$ value for this question is 1 if at least one of the $k$ samples passes verification; otherwise, it is 0. The average pass@$k$ value over the dataset reflects the proportion of problems in the dataset that the model can solve within $k$ trials, providing a rigorous evaluation of the reasoning capacity coverage of LLMs. We adopt an unbiased, low-variance estimator for computing to calculate pass@$k$, as detailed in Section A.2.

Pass@$k$ 指标。准确度量基座与 RL 模型的推理能力边界并不容易：贪心解码或核采样平均（Holtzman 等，2020）只反映平均情形表现。为准确度量推理能力边界，我们把代码生成中常用的 pass@$k$ 指标（Chen 等，2021）扩展到所有带可验证奖励的任务。对给定问题，从模型采样 $k$ 个输出；若至少有一个通过验证，则该题的 pass@$k$ 为 1，否则为 0。数据集上的平均 pass@$k$ 反映模型在 $k$ 次试验内可解问题的比例，从而对 LLM 推理能力覆盖给出严格评估。我们采用无偏、低方差的估计器计算 pass@$k$，细节见第 A.2 节。

Comparison with Best-of-$N$ and Majority Voting. Best-of-$N$ Cobbe et al. (2021) and majority voting are practical methods for selecting correct answers, but they may overlook a model’s full reasoning potential. In contrast, we use pass@$k$ not to assess practical utility but to investigate the boundaries of reasoning capacity. If a model produces a correct solution in any of the $k$ samples, we treat the problem as within its potential scope. Thus, if RL enhances reasoning, the RL-trained model should succeed in more such problems than the base model. Methods like Best-of-$N$ or majority voting may miss these successes if the correct answer is not selected by the verifier or voting.

与 Best-of-$N$ 及多数投票的比较。Best-of-$N$（Cobbe 等，2021）与多数投票是选出正确答案的实用方法，但可能忽略模型的完整推理潜力。相对地，我们用 pass@$k$并非评估实用效用，而是探究推理能力边界。只要 $k$ 个样本中任一样本给出正解，就视该题落在模型的潜在范围内。因此，若 RL 增强了推理，RL 训练模型应在更多此类问题上成功。Best-of-$N$ 或多数投票若未由验证器或投票选中正解，则可能漏掉这些成功。

Random Guessing Issue. For coding tasks, where a compiler and predefined unit test cases are used as verifiers, the pass@$k$ value can accurately reflect whether the model can solve the problem. In mathematics, the issue of “guessing” can become pronounced as $k$ increases, where a model may generate an incorrect CoT but still accidentally arrive at the correct answer. To address this, we manually check the correctness of CoT for a subset of model outputs as detailed in Section 3.1. By combining results on math with manually checking and coding, we rigorously evaluate the scope of LLM’s reasoning capacity. Another caveat is that, with an astronomically large $k$, even uniform sampling over the token dictionary would stumble upon the correct reasoning path–though this is infeasible within today’s time and compute budgets. Crucially, we find that the base model already produces correct outputs at realistic values ($k=128$ or $1024$), well within practical resource limits.

随机猜测问题。对编码任务，用编译器与预定义单元测试作验证器时，pass@$k$ 能准确反映模型是否能解题。在数学中，随 $k$ 增大，「猜测」问题会变得突出：模型可能生成错误 CoT 却偶然得到正确答案。为此，我们对手工检查部分模型输出的 CoT 正确性，细节见第 3.1 节。结合经人工核查的数学结果与编码结果，我们严格评估 LLM 推理能力的范围。另一点需注意：若 $k$ 天文量级，即便在词表上均匀采样也可能碰巧撞上正确推理路径——但这在今天的时间与算力预算内不可行。关键是：我们发现基座模型已在现实取值（$k=128$ 或 $1024$）下产出正确输出，完全落在实用资源限度内。

Task	Start Model	RL Framework	RL Algorithm(s)	Benchmark(s)
Mathematics	LLaMA-3.1-8B Qwen2.5-7B/14B/32B-Base Qwen2.5-Math-7B	SimpleRLZoo Oat-Zero DAPO	GRPO	GSM8K, MATH500 Minerva, Olympiad AIME24, AMC23
Code Generation	Qwen2.5-7B-Instruct DeepSeek-R1-Distill-Qwen-14B	Code-R1 DeepCoder	GRPO	LiveCodeBench HumanEval+
Visual Reasoning	Qwen2.5-VL-7B	EasyR1	GRPO	MathVista MathVision
Deep Analysis	Qwen2.5-7B-Base Qwen2.5-7B-Instruct DeepSeek-R1-Distill-Qwen-7B	VeRL	PPO, GRPO Reinforce++ RLOO, ReMax, DAPO	Omni-Math-Rule MATH500

Table 1: Experimental setup for assessing RLVR’s effect on the reasoning boundaries of LLMs.

任务	起点模型	RL 框架	RL 算法	基准
数学	LLaMA-3.1-8B；Qwen2.5-7B/14B/32B-Base；Qwen2.5-Math-7B	SimpleRLZoo；Oat-Zero；DAPO	GRPO	GSM8K，MATH500；Minerva，Olympiad；AIME24，AMC23
代码生成	Qwen2.5-7B-Instruct；DeepSeek-R1-Distill-Qwen-14B	Code-R1；DeepCoder	GRPO	LiveCodeBench；HumanEval+
视觉推理	Qwen2.5-VL-7B	EasyR1	GRPO	MathVista；MathVision
深度分析	Qwen2.5-7B-Base；Qwen2.5-7B-Instruct；DeepSeek-R1-Distill-Qwen-7B	VeRL	PPO，GRPO；Reinforce++；RLOO，ReMax，DAPO	Omni-Math-Rule；MATH500

表 1：评估 RLVR 对 LLM 推理边界影响的实验设定。

3 RLVR’s Effect on Reasoning Capacity Boundary

3 RLVR 对推理能力边界的影响

With the evaluation metrics for reasoning boundaries established, we now conduct a comprehensive evaluation of the base and RLVR models through extensive experiments. Our analysis is organized by task category, covering three representative domains: mathematics, code generation, and visual reasoning. The overall experimental setup is summarized in Table 1.

有了推理边界的评测指标后，我们通过大量实验对基座与 RLVR 模型做全面评估。分析按任务类别组织，覆盖三个代表性领域：数学、代码生成与视觉推理。总体实验设定汇总于表 1。

Evaluation Protocol. For sampling procedures for both base and RLVR models, we use a temperature of 0.6 and a top-$p$ value of 0.95, allowing a maximum generation of 16,384 tokens. We also show the effect of different temperature settings in Figure 17. For evaluation of the base model, a common practice is to include few-shot examples in the prompt to guide the output Grattafiori et al. (2024); Yang et al. (2024); Liu et al. (2024). However, to ensure a fair and unbiased comparison, we deliberately avoid using few-shot prompts for base models, eliminating any potential confounding effects on reasoning that might be introduced by in-context examples. For evaluating both the base and RLVR models, we use the same zero-shot prompt as in RLVR training, or the default prompt provided by the benchmark, ensuring a consistent setup across both models. Interestingly, although base models often produce unformatted or non-sensical responses without few-shot guidance, we observe that with sufficient sampling, they are still capable of generating correctly formatted outputs and successfully solving complex problems. Prompt templates for training and evaluation are provided in Appendix D.

评测协议。对基座与 RLVR 模型的采样，温度取 0.6、top-$p$ 取 0.95，最大生成 16,384 token。不同温度设定的影响见图 17。评测基座模型时，常见做法是在提示中加入少样本示例以引导输出（Grattafiori 等，2024；Yang 等，2024；Liu 等，2024）。为保证公平无偏比较，我们有意不对基座使用少样本提示，以排除上下文示例可能引入的推理混杂效应。评测基座与 RLVR 时，使用与 RLVR 训练相同的零样本提示，或基准默认提示，保证两模型设定一致。有趣的是：尽管没有少样本引导时基座常产出格式混乱或无意义回答，我们观察到只要采样足够多，它们仍能生成格式正确的输出并成功解出复杂问题。训练与评测提示模板见附录 D。

3.1 RLVR for Mathematical Reasoning

3.1 面向数学推理的 RLVR

Models and Benchmarks. In math problems, models are required to generate a reasoning process (i.e., CoT) along with the final answer. To ensure the robustness of conclusions, we experiment with multiple LLM families, primarily Qwen2.5 (7B/14B/32B base variants) Yang et al. (2024) and additionally LLaMA-3.1-8B Grattafiori et al. (2024). We adopt RLVR models released by SimpleRLZoo Zeng et al. (2025), which train zero-RL models using GRPO on GSM8K and the MATH training set, with correctness reward only, excluding any format-based reward. We compare the pass@$k$ curves of base and zero-RL models on benchmarks of varying difficulty: GSM8K Cobbe et al. (2021), MATH500 Hendrycks et al. (2021), Minerva Lewkowycz et al. (2022), Olympiad He et al. (2024), AIME24, and AMC23. Additionally, we include the RLVR model Oat-Zero-7B and DAPO-32B Liu et al. (2025b); Yu et al. (2025). These two models are characterized by strong performance on the challenging AIME24 benchmark.

模型与基准。数学题要求模型生成推理过程（即 CoT）与最终答案。为保证结论稳健，我们实验多个 LLM 家族，主要是 Qwen2.5（7B/14B/32B 基座变体）（Yang 等，2024），并额外包含 LLaMA-3.1-8B（Grattafiori 等，2024）。RLVR 模型采用 SimpleRLZoo（Zeng 等，2025）发布版本：在 GSM8K 与 MATH 训练集上用 GRPO 训练 zero-RL 模型，仅用正确性奖励、不含格式奖励。我们在难度各异的基准上比较基座与 zero-RL 的 pass@$k$ 曲线：GSM8K（Cobbe 等，2021）、MATH500（Hendrycks 等，2021）、Minerva（Lewkowycz 等，2022）、Olympiad（He 等，2024）、AIME24 与 AMC23。此外还包含 RLVR 模型 Oat-Zero-7B 与 DAPO-32B（Liu 等，2025b；Yu 等，2025），二者在高难度 AIME24 上表现突出。

The Effect of RLVR: Increased Likelihood of Correct Samples, Decreased Coverage of Solvable Problems. As shown in Figure 2, we consistently observe a contrasting trend between small and large $k$ values. When $k$ is small (e.g., $k=1$, equivalent to average-case accuracy), RL-trained models outperform their base counterparts. This aligns with the common observation that RL improves performance, suggesting that RLVR makes models significantly more likely to sample correct responses. However, as $k$ increases, with steeper curves, base models consistently catch up to and eventually surpass RL-trained models across all benchmarks, indicating their broader coverage of solvable problems. For example, on the Minerva benchmark with a 32B-sized model, the base model outperforms the RL-trained model by approximately 9% at $k=128$, implying that it can solve around 9% more problems in the validation set.

RLVR 的效应：正确样本似然上升，可解问题覆盖下降。如图 2 所示，我们 始终观察到小 $k$ 与大 $k$ 之间的对比趋势。小 $k$（如 $k=1$，等价于平均准确率）时，RL 训练模型优于基座——这与「RL 提升表现」的常见观察一致，说明 RLVR 使模型显著更可能采样到正确回答。但随着 $k$ 增大，基座曲线更陡，并在所有基准上赶上并最终超越 RL 训练模型，表明其可解问题覆盖更广。例如，在 Minerva 上用 32B 模型，基座在 $k=128$ 时约领先 RL 训练模型 9%，意味着验证集上约多解 9% 的题。

We further examine RL models trained with Oat-Zero and DAPO. As shown in Figure 11, although the RL model initially demonstrates a strong performance, nearly 30% higher than the base model, it is eventually surpassed by the base model. Based on these results, we conclude that RLVR increases the likelihood of sampling correct responses at low $k$, but narrows the model’s overall coverage. We further analyze the root cause of this phenomenon in Section 4.1.

我们进一步考察用 Oat-Zero 与 DAPO 训练的 RL 模型。如图 11，尽管 RL 模型起初表现很强——几乎比基座高 30%——最终仍被基座超越。基于这些结果，我们结论：RLVR 在低 $k$ 提高正确回答的采样似然，但收窄模型的整体覆盖。第 4.1 节进一步分析该现象的根因。

CoT Case Analysis. We present the correct CoTs sampled from the base model in Figure 20 and Figure 21, manually selected from 2048 samplings for the hardest questions in AIME24. The responses from the base model tend to be long CoTs and exhibit reflective behavior, highlighting the strong reasoning ability inherent in the base model.

CoT 案例分析。图 20 与图 21 展示从基座模型采样到的正确 CoT，人工选自 AIME24 最难题上的 2048 次采样。基座回答往往是长 CoT，并表现出反思行为，凸显基座模型固有的强推理能力。

Validity of Chain-of-Thought. For mathematical problems, the common evaluation is based solely on the correctness of the final answer, with the risk of “hacking”. To accurately reflect the reasoning ability boundary using pass@$k$, it is important to assess how many solved problems result from sampling genuinely correct CoTs, rather than from lucky guesses. Following Brown et al. (2024), we manually inspect all CoTs that led to correct answers to the most challenging solvable problems in the GSM8k dataset – those with an average accuracy below 5% but above 0%. The base model answered 25 such questions, with 24 containing at least one correct CoT. Similarly, the RL-trained model answered 25 questions, 23 of which included at least one correct CoT. We also manually check the CoTs for problems in the challenging AIME24 benchmark with an average accuracy below 5%. Details can be found in Section C.2. The base model answered 7 such questions, with 5 out of 6 containing at least one correct CoT (excluding one ambiguous case of correctness due to skipped reasoning steps). Similarly, the RL-trained model answered 6 questions, 4 of which included at least one correct CoT. These results suggest that the base model can sample valid reasoning paths to solve the problems.

思维链的有效性。数学题的常见评测只看最终答案对错，存在「被猜对」的风险。要用 pass@$k$ 准确反映推理能力边界，须评估有多少「解出的题」来自真正正确的 CoT，而非幸运猜测。遵循 Brown 等（2024），我们对手工检查 GSM8k 中最具挑战、可解问题的全部导向正解的 CoT——即平均准确率低于 5% 但高于 0% 的题。基座解出 25 道此类题，其中 24 道含至少一条正确 CoT；RL 训练模型同样解出 25 道，其中 23 道含至少一条正确 CoT。我们也对手工检查 AIME24 上平均准确率低于 5% 的题的 CoT，细节见第 C.2 节。基座解出 7 道，6 道中有 5 道含至少一条正确 CoT（排除一道因跳步导致正确性存疑）；RL 训练模型解出 6 道，其中 4 道含至少一条正确 CoT。这些结果表明：基座模型能够采样到有效推理路径来解题。

3.2 RLVR for Code Generation

3.2 面向代码生成的 RLVR

Figure 3: RLVR for Coding.

图 3：面向编码的 RLVR。

Models and Benchmarks. We adopt the open-sourced RLVR-trained model, CodeR1-Zero-Qwen2.5-7B Liu and Zhang (2025), which trains zero-RL models on 12K LeetCode and TACO samples over 832 steps, based on Qwen2.5-7B-Instruct-1M Yang et al. (2025b). For evaluation, models are assessed on LiveCodeBench v5, comprising 279 problems that span from August 2024 to January 2025 Jain et al. (2025), as well as HumanEval+ and MBPP+ Liu et al. (2023). We also evaluate the most powerful open-source RLVR-trained coding LLM, DeepCoder-14B Luo et al. (2025), built on DeepSeek-R1-Distill-Qwen-14B. Here both models take 32k response length. Due to their high computational cost, we evaluate them only on LiveCodeBench as a representative benchmark.

模型与基准。我们采用开源 RLVR 训练模型 CodeR1-Zero-Qwen2.5-7B（Liu 与 Zhang，2025）：基于 Qwen2.5-7B-Instruct-1M（Yang 等，2025b），在 12K 条 LeetCode 与 TACO 样本上训练 zero-RL 共 832 步。评测在 LiveCodeBench v5（含 2024 年 8 月至 2025 年 1 月的 279 题）（Jain 等，2025）以及 HumanEval+ 与 MBPP+（Liu 等，2023）上进行。我们还评估目前最强的开源 RLVR 编码 LLM DeepCoder-14B（Luo 等，2025），基于 DeepSeek-R1-Distill-Qwen-14B。二者响应长度均为 32k；因计算成本高，仅以 LiveCodeBench 为代表基准评测。

The Effect of RLVR. Since passing all unit tests is nearly impossible to achieve by guesswork, pass@$k$ provides a reliable measure of a model’s reasoning boundary. As shown in Figure 3, Figure 12, and Figure 4 (left), the effects of RLVR on three code generation benchmarks exhibit trends that are highly consistent with those observed in mathematical benchmarks.

RLVR 的效应。要通过全部单元测试几乎不可能靠猜，因此 pass@$k$ 能可靠度量模型的推理边界。如图 3、图 12 与图 4（左）所示，RLVR 在三个代码生成基准上的效应与数学基准高度一致。

3.3 RLVR for Visual Reasoning

3.3 面向视觉推理的 RLVR

Models and Benchmarks. In visual reasoning, models must jointly interpret visual and textual inputs to solve complex reasoning problems. This has gained significant attention in the multimodal community since the rise of LLM reasoning Chen et al. (2025a); Shen et al. (2025); Zheng et al. (2025). For our experiments, we select math within visual contexts as a representative task. We use the EasyR1 framework Zheng et al. (2025) to train Qwen2.5-VL-7B Bai et al. (2025) on Geometry3K Lu et al. (2021), and evaluate its visual reasoning capabilities on filtered MathVista-TestMini Lu et al. (2024) and MathVision-TestMini Wang et al. (2024), where multiple-choice questions are removed.

模型与基准。视觉推理要求模型联合理解视觉与文本输入以解决复杂推理问题。自 LLM 推理兴起以来，这在多模态社区受到广泛关注（Chen 等，2025a；Shen 等，2025；Zheng 等，2025）。实验中我们选取视觉情境下的数学作为代表任务。用 EasyR1 框架（Zheng 等，2025）在 Geometry3K（Lu 等，2021）上训练 Qwen2.5-VL-7B（Bai 等，2025），并在过滤后的 MathVista-TestMini（Lu 等，2024）与 MathVision-TestMini（Wang 等，2024）上评估视觉推理能力（已去掉选择题）。

The Effect of RLVR. As shown in Figure 4 (right), the effects of RLVR on visual reasoning are highly consistent with those observed in math and coding benchmarks. This suggests that the original model has broader coverage of solvable questions even in multimodal tasks.

RLVR 的效应。如图 4（右），RLVR 对视觉推理的效应与数学、编码基准高度一致。这说明即便在多模态任务中，原模型的可解问题覆盖也更广。

Validity of Chain-of-Thought. Similarly, we manually inspect a subset of the most challenging problems, i.e.those with an average accuracy below 5%. We find that for both the original and RL models, 7 out of 8 problems have at least one correct CoT. These results support the validity of CoTs.

思维链的有效性。类似地，我们对手工检查最具挑战的子集——平均准确率低于 5% 的题。发现对原模型与 RL 模型，8 道中有 7 道含至少一条正确 CoT。这些结果支持 CoT 的有效性。

Figure 4: Pass@$k$ curves of base and RLVR models. (Left) Code Generation. (Right) Visual Reasoning.

图 4：基座与 RLVR 模型的 pass@$k$ 曲线。**（左）代码生成。（右）**视觉推理。

4 Deep Analysis

4 深度分析

In this section, we conduct a deeper analysis of the effects of current RLVR training. We also highlight the distinct characteristics of distillation in comparison to RLVR. In addition, we design controlled experiments to examine the impact of different RL algorithms and design choices.

本节更深入分析当前 RLVR 训练的效应，并突出蒸馏相对 RLVR 的不同特征。此外，我们设计受控实验考察不同 RL 算法与设计选择的影响。

4.1 Reasoning Paths Already Present in Base Models

4.1 推理路径在基座模型中业已存在

Figure 5: Qwen2.5-7B Accuracy Histogram on Minerva.

图 5：Qwen2.5-7B 在 Minerva 上的准确率直方图。

Accuracy Distribution Analysis. Experiments in Section 3 reveal a surprising trend: the base model covers a wider range of solvable problems than the RLVR-trained model. To better understand this, we analyze how the accuracy distribution changes before and after RLVR training. As shown in Figure 5, RLVR increases the frequency of high accuracies near 1.0 and reduces the frequency of low accuracies (e.g., 0.1, 0.2). However, a deviation from this trend is the increased frequency at accuracy 0 — indicating that RLVR leads to more unsolvable problems. This also explains the improvement of RLVR in average scores, driven not by solving new problems but by improving sampling efficiency on problems already solvable by the base model. Additional accuracy histograms are provided in Figure 14.

准确率分布分析。第 3 节实验揭示出人意料的趋势：基座模型的可解问题覆盖比 RLVR 训练模型更广。为更好理解，我们分析 RLVR 训练前后准确率分布如何变化。如图 5，RLVR 提高了接近 1.0 的高准确率频次，降低了低准确率（如 0.1、0.2）频次。但偏离该趋势的是准确率为 0 的频次上升——表明 RLVR 带来了更多不可解问题。这也解释了 RLVR 平均分的提升：并非靠解出新题，而是靠在基座已能解的题上提高采样效率。更多准确率直方图见图 14。

Base	SimpleRLZoo	AIME24	MATH500
✓	✓	63.3%	92.4%
✓	✗	13.3%	3.6%
✗	✓	0.0%	1.0%
✗	✗	23.3%	3.0%

Table 2: We evaluate on AIME24 ($k=1024$) and MATH500 ($k=128$). The table reports the solvable/unsolvable fraction of problems falling into four categories.

基座	SimpleRLZoo	AIME24	MATH500
✓	✓	63.3%	92.4%
✓	✗	13.3%	3.6%
✗	✓	0.0%	1.0%
✗	✗	23.3%	3.0%

表 2：我们在 AIME24（$k=1024$）与 MATH500（$k=128$）上评估。表中报告落入四类情形的可解 / 不可解问题占比。

Solvable-Problem Coverage Analysis. To further investigate, we compare the set of solvable questions for both the base model and its corresponding RL-trained version on AIME24 and MATH500. We find that there are many cases where the base model solves a problem but the RLVR model fails, and very few where RLVR succeeds while the base model does not, as shown in Table 2. Details can be found at Section C.7. As shown in Table 5, the set of problems solved by the RL-trained model is nearly a subset of those solvable by the base model. A similar trend is observed in coding tasks as shown in Table 6. This raises the natural question: Do all reasoning paths generated by RL-trained models already exist within the output distribution of their base models?

可解问题覆盖分析。为进一步探究，我们在 AIME24 与 MATH500 上比较基座模型与其对应 RL 训练版本的可解题集合。发现大量情形是基座解出而 RLVR 失败，而 RLVR 成功、基座失败的情形极少，如表 2 所示。细节见第 C.7 节。如表 5，RL 训练模型解出的题集几乎是基座可解题集的子集。编码任务上有类似趋势，见表 6。这自然引出问题：RL 训练模型生成的所有推理路径，是否已存在于其基座模型的输出分布中？

Perplexity Analysis. To answer this question, we utilize the metric perplexity. Given a model $m$, a problem $x$, and a response $\mathbf{Y}=(y_{1},\dots,y_{T})$ (can be generated by the same model, another model, or humans), the perplexity is defined as the exponentiated average negative log-likelihood of a sequence:

困惑度分析。为回答该问题，我们使用困惑度指标。给定模型 $m$、问题 $x$ 与回答 $\mathbf{Y}=(y_{1},\dots,y_{T})$（可由同一模型、另一模型或人类生成），困惑度定义为序列负对数似然指数平均：

$$\text{PPL}_{m}(\mathbf{Y}\mid x)=\exp\left(-\frac{1}{T}\sum_{t=1}^{T}\log P(y_{t}\mid x,y_{1},\dots,y_{t-1})\right),$$

which reflects the model’s ability to predict the given response $\mathbf{Y}$ conditioned on the prompt $x$. Lower perplexity indicates that the model has a higher likelihood of generating this response.

它反映模型在提示 $x$ 条件下预测给定回答 $\mathbf{Y}$ 的能力。困惑度越低，表示模型生成该回答的似然越高。

Figure 6: Perplexity distribution of responses. The conditioning problem $x$ is omitted in the figure.

图 6：回答的困惑度分布。图中省略条件问题 $x$。

We randomly sample two problems from AIME24 and employ Qwen2.5-7B-Base and SimpleRL-Qwen2.5-7B-Base to generate 16 responses for each problem, denoted as $\mathbf{Y}_{\text{base}}$ and $\mathbf{Y}_{\text{RL}}$, respectively. We also let OpenAI-o1 Jaech et al. (2024) generate 8 responses, denoted as $\mathbf{Y}_{\text{GT}}$. As shown in Figure 6, the distribution of $\text{PPL}_{\text{Base}}(\mathbf{Y}_{\text{RL}}|x)$ closely matches the lower portion of the $\text{PPL}_{\text{Base}}(\mathbf{Y}_{\text{Base}}|x)$ distribution, corresponding to responses that the base model tends to generate. This suggests that the responses from RL-trained models are highly likely to be generated by the base model. In Section C.4, we show that $\text{PPL}_{\text{Base}}(\mathbf{Y}_{\text{RL}}|x)$ gradually decreases as RL training progresses, indicating that RLVR mainly sharpens the distribution within the base model’s prior rather than expanding beyond it.

我们从 AIME24 随机采样两道题，分别用 Qwen2.5-7B-Base 与 SimpleRL-Qwen2.5-7B-Base 各生成 16 条回答，记为 $\mathbf{Y}_{\text{base}}$ 与 $\mathbf{Y}_{\text{RL}}$；并让 OpenAI-o1（Jaech 等，2024）生成 8 条，记为 $\mathbf{Y}_{\text{GT}}$。如图 6，$\text{PPL}_{\text{Base}}(\mathbf{Y}_{\text{RL}}|x)$ 的分布与 $\text{PPL}_{\text{Base}}(\mathbf{Y}_{\text{Base}}|x)$ 分布的较低部分密切吻合，对应该基座倾向于生成的回答。这说明 RL 训练模型的回答很可能由基座模型生成。第 C.4 节显示，随着 RL 训练推进，$\text{PPL}_{\text{Base}}(\mathbf{Y}_{\text{RL}}|x)$ 逐渐下降，表明 RLVR 主要是在基座先验内锐化分布，而非向外扩展。

Summary. Combining the above analyses, we arrive at three key observations. First, problems solved by the RLVR model are also solvable by the base model; the observed improvement in average scores stems from more efficient sampling on these already solvable problems, rather than learning to solve new problems. Second, after RLVR training, the model often exhibits narrower reasoning coverage compared to its base model. Third, all the reasoning paths exploited by the RLVR model are already present in the sampling distribution of the base model. These findings indicate that RLVR does not introduce fundamentally new reasoning capabilities and that the reasoning capacity of the trained model remains bounded by that of its base model.

小结。综合上述分析，得到三点关键观察。第一，RLVR 模型解出的题，基座也能解；平均分提升来自在这些已可解题上更高效的采样，而非学会解新题。第二，RLVR 训练后，模型的推理覆盖往往比基座更窄。第三，RLVR 模型所利用的全部推理路径，已存在于基座模型的采样分布中。这些发现表明：RLVR 并未引入根本性的新推理能力，训练后模型的推理能力仍受其基座所界定。

4.2 Distillation Expands the Reasoning Boundary

4.2 蒸馏扩展推理边界

Figure 7: pass@$k$ of base, Instruct, RLVR, and distilled models.

图 7：基座、Instruct、RLVR 与蒸馏模型的 pass@$k$。

In addition to direct RL training, another effective approach to improving the reasoning ability of small base models is distillation from a powerful reasoning model Guo et al. (2025). This process is analogous to instruction-following fine-tuning in post-training. However, instead of using short instruction-response pairs, the training data consist of long CoT reasoning traces generated by the teacher model. Given the limitations of current RLVR in expanding reasoning capabilities, it is natural to ask whether distillation exhibits similar behavior. We focus on a representative model, DeepSeek-R1-Distill-Qwen-7B, which distills DeepSeek-R1 into Qwen2.5-Math-7B. We compare it with the base model Qwen2.5-Math-7B and its RL-trained counterpart Qwen2.5-Math-7B-Oat-Zero and include Qwen2.5-Math-7B-Instruct as an additional baseline. As shown in Figure 7, the pass@$k$ curve of the distilled model is consistently and significantly above that of the base model. This indicates that, unlike RL that is fundamentally bounded by the reasoning capacity of the base model, distillation introduces new reasoning patterns learned from a stronger teacher model. As a result, the distilled model is capable of surpassing the reasoning boundary of the base model.

除直接 RL 训练外，提升小基座推理能力的另一有效途径是从强推理模型蒸馏（Guo 等，2025）。该过程类似后训练中的指令跟随微调；但训练数据不是短指令–回答对，而是教师模型生成的长 CoT 推理轨迹。鉴于当前 RLVR 在扩展推理能力上的局限，自然要问蒸馏是否表现类似。我们聚焦代表模型 DeepSeek-R1-Distill-Qwen-7B：把 DeepSeek-R1 蒸馏到 Qwen2.5-Math-7B。将其与基座 Qwen2.5-Math-7B 及其 RL 训练对应版本 Qwen2.5-Math-7B-Oat-Zero 比较，并纳入 Qwen2.5-Math-7B-Instruct 作为额外基线。如图 7，蒸馏模型的 pass@$k$ 曲线 始终且显著高于基座。这表明：与从根本上受基座推理能力界定的 RL 不同，蒸馏引入了从更强教师学到的新推理模式，从而能使蒸馏模型超越基座的推理边界。

4.3 Effects of Different RL Algorithms

4.3 不同 RL 算法的效应

As discussed previously, the primary effect of RL is to enhance sampling efficiency rather than to expand a model’s reasoning capacity. To quantify this, we propose the Sampling Efficiency Gap ($\Delta_{\text{SE}}$), defined as the difference between the RL-trained model’s pass@1 and the base model’s pass@$k$ (we use $k=256$ in our evaluation). Lower $\Delta_{\text{SE}}$ is better. Here we conduct clean experiments to study the effect of different RL algorithms in enhancing sampling efficiency.

如前所述，RL 的主要效应是提升采样效率，而非扩展模型推理能力。为量化这一点，我们提出采样效率缺口（$\Delta_{\text{SE}}$），定义为 RL 训练模型的 pass@1 与基座模型 pass@$k$ 之差（评测中取 $k=256$）。$\Delta_{\text{SE}}$ 越低越好。这里我们做干净实验，研究不同 RL 算法在提升采样效率上的效应。

Experiment Setup. We re-implement popular RL algorithms using the VeRL framework Sheng et al. (2024) for fair comparison, including PPO Schulman et al. (2017), GRPO Shao et al. (2024), Reinforce++ Hu (2025), RLOO Ahmadian et al. (2024), ReMax Li et al. (2024), and DAPO Yu et al. (2025). Following DAPO Yu et al. (2025) and Oat-Zero Liu et al. (2025b), we remove the KL term to avoid constraining model learning. During training, we use the AdamW optimizer Loshchilov and Hutter (2017) with a constant learning rate of $10^{-6}$. For rollout, we employ a prompt batch size of 256 and generate 8 responses per prompt. The maximum rollout length is set to 8,192 tokens, and the sampling temperature is set as 1.0. We use a PPO mini-batch size of 256.

实验设定。我们用 VeRL 框架（Sheng 等，2024）重实现流行 RL 算法以便公平比较，包括 PPO（Schulman 等，2017）、GRPO（Shao 等，2024）、Reinforce++（Hu，2025）、RLOO（Ahmadian 等，2024）、ReMax（Li 等，2024）与 DAPO（Yu 等，2025）。遵循 DAPO（Yu 等，2025）与 Oat-Zero（Liu 等，2025b），去掉 KL 项以免约束模型学习。训练使用 AdamW 优化器（Loshchilov 与 Hutter，2017），恒定学习率 $10^{-6}$。rollout 时提示 batch 大小 256，每提示生成 8 条回答；最大 rollout 长度 8,192 token，采样温度 1.0；PPO mini-batch 大小 256。

To assess in-domain and out-of-domain generalization under RLVR, we split Omni-MATH-Rule, a subset of Omni-MATH Gao et al. (2025) containing verifiable problems, into a training set (2,000 samples) and an in-domain test set (821 samples), and use MATH500 as the out-of-domain benchmark.

为评估 RLVR 下的域内与域外泛化，我们将 Omni-MATH-Rule——Omni-MATH（Gao 等，2025）中含可验证问题的子集——划分为训练集（2,000 样本）与域内测试集（821 样本），并用 MATH500 作为域外基准。

Results. As shown in Figure 8 (top), although different RL algorithms exhibit slight variations in both pass@1 and pass@256, these differences are not fundamental. Different RL algorithms yield slightly different $\Delta_{\text{SE}}$ values (i.e., ranging from GRPO’s 43.9 to RLOO’s best 42.6 on the in-domain test set). Furthermore, we observe that $\Delta_{\text{SE}}$ remains consistently above 40 points across different algorithms, highlighting that existing RL methods are still far from achieving optimal sampling efficiency. This suggests that novel RL algorithms or entirely new paradigms may be necessary to approach the upper bound. Additional observations can be found at Section C.5.

结果。如图 8（上），尽管不同 RL 算法在 pass@1 与 pass@256 上略有差异，这些差异并非根本性的。不同算法给出略有不同的 $\Delta_{\text{SE}}$（例如域内测试集上从 GRPO 的 43.9 到 RLOO 最优的 42.6）。此外，我们观察到跨算法 $\Delta_{\text{SE}}$ 始终高于 40 分，凸显现有 RL 方法距最优采样效率仍远。这表明或许需要全新的 RL 算法或范式才能逼近上界。更多观察见第 C.5 节。

Figure 8: (Top) Different RL algorithms. (Bottom) Different RL training steps. The detailed values for each point at pass@1 and pass@256 are provided in Table 3 and Table 4.

图 8：**（上）不同 RL 算法。（下）**不同 RL 训练步数。pass@1 与 pass@256 各点的详细数值见表 3 与表 4。

4.4 Effects of RL Training

4.4 RL 训练的效应

Asymptotic Effects. Based on the setup in Section 4.3, we investigate the effect of the training steps on the asymptotic performance of the model. As shown in Figure 1 (right), as RL training progresses, pass@1 on the training set consistently improves from 26.1 to 42.5. However, as RLVR training progresses, pass@256 progressively decreases, indicating a reduced reasoning boundary.

渐近效应。基于第 4.3 节设定，我们考察训练步数对模型渐近表现的影响。如图 1（右），随着 RL 训练推进，训练集 pass@1 从 26.1 持续提升到 42.5；但 pass@256 逐步下降，表明推理边界在收缩。

Effect of Number of Rollouts $n$. The training hyperparameter $n$, the number of responses per prompt, can affect pass@$k$ by enabling broader exploration during training. We increase $n$ from 8 to 32. As shown in Figure 16, pass@$k$ improves slightly over $n=8$, but the RL-trained model is still eventually outperformed by the base model. We leave the question of whether scaling RLVR training can eventually surpass the base model to future investigation.

rollout 数 $n$ 的效应。训练超参 $n$（每提示的回答数）可通过在训练中实现更广探索影响 pass@$k$。我们将 $n$ 从 8 增至 32。如图 16，相对 $n=8$，pass@$k$ 略有改善，但 RL 训练模型最终仍被基座超越。RLVR 训练规模扩大后能否最终超越基座，留给未来研究。

Effect of KL Loss. To control model deviation, some prior work adds a KL penalty. We ablate this by applying a KL term with coefficient 0.001. As shown in Figure 16, the KL-regularized model achieves similar pass@1 to GRPO without KL, but with a much lower pass@128.

KL 损失的效应。为控制模型偏离，部分先前工作加入 KL 惩罚。我们以系数 0.001 施加 KL 项做消融。如图 16，带 KL 正则的模型 pass@1 与无 KL 的 GRPO 相近，但 pass@128 低得多。

4.5 Effects of Entropy

4.5 熵的效应

As RL training progresses, the model’s output entropy typically decreases Yu et al. (2025), which may contribute to a reduced reasoning boundary due to less diverse output. To investigate this factor, we increase the generation temperature of the RLVR-trained model to match the output entropy of the base model at $T=0.6$. As shown in Figure 18, although the RLVR model performs slightly better pass@$k$ at higher temperatures compared to its own performance at $T=0.6$, it still underperforms the base model across pass@$k$. This suggests that while reduced entropy contributes to the narrowing of the reasoning boundary, it alone does not fully account for the reduction.

随着 RL 训练推进，模型输出熵通常下降（Yu 等，2025），可能因输出多样性降低而促成推理边界收缩。为探究该因素，我们提高 RLVR 训练模型的生成温度，使其输出熵与基座在 $T=0.6$ 时匹配。如图 18，尽管 RLVR 模型在较高温度下的 pass@$k$ 略好于其自身在 $T=0.6$ 的表现，跨 pass@$k$ 仍低于基座。这说明：熵降低有助于解释推理边界收窄，但 单凭这一点并不能完全解释这一收缩。

4.6 Effects of Model Size Scaling

4.6 模型规模缩放的效应

Scaling plays a central role in the capabilities of contemporary LLMs. It remains an important question whether the conclusions drawn continue to hold as model size increases. For many large models, isolating the effect of RLVR is not feasible. For example, in the case of GPT-o1, the base model is not publicly accessible. Qwen3-235B Yang et al. (2025a) is trained through multiple stages, including RLVR and long-context CoT supervised fine-tuning, which makes it impossible to disentangle the impact of RLVR alone. For Deepseek-R1-Zero, the absence of a publicly hosted API forced us to self-host the model, but throughput was limited to around 50 tokens per second at a maximum sequence length of 32k, rendering pass@$k$ evaluation currently impractical. As a more tractable alternative, we selected the Magistral-Medium-2506 API to conduct a preliminary set of experiments. This model is trained using pure RL, with Mistral-Medium-3-2505 as the starting model Rastogi et al. (2025). Although the model size is not disclosed, Magistral-Medium performs comparably to Deepseek-R1 and is positioned near the frontier in terms of reasoning capability.

缩放在当代 LLM 能力中居核心地位。随模型变大，既有结论是否仍然成立，是重要问题。对许多大模型，难以孤立 RLVR 的效应。例如 GPT-o1 的基座未公开。Qwen3-235B（Yang 等，2025a）经多阶段训练，含 RLVR 与长上下文 CoT 监督微调，无法单独拆开 RLVR 影响。对 Deepseek-R1-Zero，因缺少公开托管 API，我们不得不自托管，但在最大序列长度 32k 时吞吐仅约每秒 50 token，使 pass@$k$ 评测目前不现实。作为更可行的替代，我们选择 Magistral-Medium-2506 API 做一组初步实验。该模型用纯 RL 训练，起点为 Mistral-Medium-3-2505（Rastogi 等，2025）。尽管规模未披露，Magistral-Medium 表现可比 Deepseek-R1，在推理能力上接近前沿。

Figure 9: pass@$k$ curves of Magistral-Medium.

图 9：Magistral-Medium 的 pass@$k$ 曲线。

We queried the models using a maximum context length of 40k as the original paper does. Once again, we observed that RLVR provides significant gains at low $k$, but little or no improvement at higher $k$. Specifically, at $k=1$, the RLVR-enhanced model solves approximately 7 more problems on AIME24 and 8 more on AIME25 compared to its base version. However, as $k$ increases, the performance gap steadily narrows. These observations suggest that our conclusion continues to hold even for current, highly capable, near-frontier reasoning models. Whether this trend persists as more compute, such as pre-training scale budgets, is dedicated to RL training remains a critical question for the future of LLM reasoning.

我们按原论文设定，以最大上下文长度 40k 查询模型。再次观察到：RLVR 在低 $k$ 带来显著增益，在高 $k$ 则几乎没有提升。具体地，在 $k=1$ 时，相对基座版本，RLVR 增强模型在 AIME24 上大约多解 7 题、在 AIME25 上大约多解 8 题；但随 $k$ 增大，性能差距稳步收窄。这些观察表明：即便对当前高度能干、接近前沿的推理模型，我们的结论仍然成立。把更多算力（例如预训练级预算）投入 RL 训练后该趋势是否持续，仍是 LLM 推理未来的关键问题。

5 Discussion

5 讨论

In Section 3 and Section 4, we identified key limitations of RLVR in enhancing LLM reasoning capabilities. In this section, we explore possible underlying factors that may explain why RLVR remains bounded by the reasoning capacity of the base model.

在第 3 节与第 4 节，我们指出了 RLVR 在增强 LLM 推理能力上的关键局限。本节探讨可能解释「为何 RLVR 仍受基座推理能力界定」的底层因素。

Discussion 1: Key Differences Between Traditional RL and RLVR for LLMs are Vast Action Space and Pretrained Priors. Traditional RL such as AlphaGo Zero and the DQN series Silver et al. (2017); Mnih et al. (2015); Yue et al. (2023) can continuously improve the performance of a policy in environments like Go and Atari games without an explicit upper bound. There are two key differences between traditional RL and RLVR for LLMs. First, the action space in language models is exponentially larger than that of Go or Atari games Ramamurthy et al. (2023). RL algorithms were not originally designed to handle such a vast action space, which makes it nearly impossible to explore the reward signal effectively if training starts from scratch. Therefore, the second distinction is that RLVR for LLMs starts with a pretrained base model with useful prior, whereas traditional RL in Atari and GO games often begins from scratch. This pretrained prior guides the LLM in generating reasonable responses, making the exploration process significantly easier, and the policy can receive positive reward feedback.

讨论 1：传统 RL 与面向 LLM 的 RLVR 的关键差异，是巨大动作空间与预训练先验。传统 RL 如 AlphaGo Zero 与 DQN 系列（Silver 等，2017；Mnih 等，2015；Yue 等，2023）能在围棋与 Atari 等环境中没有显式上界地持续提升策略表现。传统 RL 与面向 LLM 的 RLVR 有两点关键差异。第一，语言模型的动作空间比围棋或 Atari 指数级更大（Ramamurthy 等，2023）。RL 算法最初并非为如此巨大的动作空间设计，若从零开始训练，几乎不可能有效探索到奖励信号。因此第二点差异是：面向 LLM 的 RLVR 从带有有用先验的预训练基座出发，而 Atari 与围棋中的传统 RL 常从零开始。该预训练先验引导 LLM 生成合理回答，使探索显著更容易，策略也能收到正奖励反馈。

Discussion 2: Priors as a Double-Edged Sword in This Vast Action Space. Since the sampling of responses is guided by the pretrained prior, the policy may struggle to explore new reasoning patterns beyond what the prior already provides. Specifically, in such a complex and highly combinatorial space, most responses generated by naive token-level sampling exploration are constrained by the base model’s prior. Any sample deviating from the prior is highly likely to produce invalid or non-sensical outputs, leading to negative outcome reward. As discussed in Section 2.1, policy gradient algorithms aim to maximize the log-likelihood of responses within the prior that receive positive rewards, while minimizing the likelihood of responses outside the prior that receive negative rewards. As a result, the trained policy tends to produce responses already present in the prior, constraining its reasoning ability within the boundaries of the base model. From this perspective, training RL models from a distilled model may temporarily provide a beneficial solution, as distillation helps inject a better prior.

讨论 2：在此巨大动作空间中，先验是双刃剑。由于回答采样受预训练先验引导，策略可能难以探索先验之外的新推理模式。具体地，在如此复杂且高度组合的空间中，朴素 token 级采样探索生成的大多数回答受基座先验约束；任何偏离先验的样本都极可能产出无效或无意义输出，从而得到负的结果奖励。如第 2.1 节所述，策略梯度算法旨在最大化先验内获正奖励回答的对数似然，同时最小化先验外获负奖励回答的似然。结果是：训练后的策略倾向于产出先验中已有的回答，把推理能力约束在基座模型边界内。从这个角度看，从蒸馏模型出发训练 RL 或许能暂时提供有益方案，因为蒸馏有助于注入更好的先验。

Possible Future Work. As discussed above, inefficient exploration mechanisms in a vast action space and the reliance on binary outcome rewards may be the root causes of the limitations observed in current RLVR settings. To fundamentally address these challenges, several directions may be worth exploring:

可能的未来工作。如上所述，巨大动作空间中低效的探索机制，以及对二值结果奖励的依赖，或是当前 RLVR 设定中所观察局限的根因。要从根本上应对这些挑战，若干方向值得探索：

• • Efficient exploration strategies in high-level abstraction. High-level exploration mechanisms such as AlphaEvolve Novikov et al. (2025), which perform self-evolution in a program-level abstraction space, may be crucial for navigating the vast action space. Such strategies could facilitate the discovery of out-of-prior reasoning patterns and previously unseen knowledge structures.
• • Data scale via curriculum. A curriculum can begin by training on easier subproblems, allowing the model to improve sampling efficiency and acquire essential meta-skills. By increasing success rates on simpler tasks before tackling harder ones, such a curriculum may hierarchically reduce the exploration space and lift performance from nearly zero to non-zero on challenging parent tasks, thereby enabling RLVR to obtain meaningful rewards Zhang et al. (2025); Li et al. (2025). Although traces of such hierarchical relationships occasionally appear in current RLVR training data, and their effects have been observed in recent work Chen et al. (2025b), realizing their full potential will require a more deliberate, large-scale data-RL iteration pipeline that ensures sufficient coverage of meta-skills as well as appropriate relationships between easy and hard problems.
• • Process reward and fine-grained credit assignment. Compared to purely binary outcome rewards, incorporating intermediate signals to guide the reasoning trajectory may significantly improve exploration efficiency and steer exploration toward more promising solution paths.
• • Agentic RL. Current RLVR reasoning are limited to single-turn response, whereas iterative refinement based on feedback is crucial for IMO-level reasoning Huang and Yang (2025). It also lacks the ability to actively collect new information by using search tools or conducting experiments. A multi-turn agentic RL paradigm, featuring richer interactions with environment feedback, could allow models to generate novel experiences and learn from them. This emerging agent framework has been described as the beginning of an "era of experience" Silver and Sutton (2025).

• • 高层抽象上的高效探索策略。诸如 AlphaEvolve（Novikov 等，2025）这类在程序级抽象空间中自我演化的高层探索机制，对驾驭巨大动作空间可能至关重要。此类策略或有助于发现先验之外的推理模式与此前未见的知识结构。
• • 经由课程的数据规模。课程可先从较易子问题训练，让模型提升采样效率并获得必要元技能。在挑战性父任务上把成功率从近零抬到非零之前，先提高简单任务成功率，可分层缩小探索空间，从而使 RLVR 获得有意义的奖励（Zhang 等，2025；Li 等，2025）。尽管当前 RLVR 训练数据中偶有此类层级关系的痕迹，且近期工作已观察到其效应（Chen 等，2025b），要充分释放潜力，仍需更有意、大规模的数据–RL 迭代管线，确保元技能覆盖充分，以及易–难题之间关系恰当。
• • 过程奖励与细粒度信用分配。相对纯二值结果奖励，引入中间信号引导推理轨迹，或可显著提升探索效率，并把探索导向更有希望的解题路径。
• • 智能体式 RL。当前 RLVR 推理限于单轮回答，而基于反馈的迭代精炼 对 IMO 级推理至关重要（Huang 与 Yang，2025）。它也缺乏主动用搜索工具或做实验收集新信息的能力。多轮智能体式 RL 范式——与环境反馈有更丰富交互——或能让模型生成新经验并从中学习。这一新兴智能体框架被描述为「经验时代」的开端（Silver 与 Sutton，2025）。

6 Related Work

6 相关工作

We summarize key related works on the analysis of RLVR here and provide a more comprehensive discussion in Appendix B. While recent RLVR methods have achieved impressive empirical results Guo et al. (2025); Lambert et al. (2024), their fundamental impact on reasoning remains underexplored. Several studies Liu et al. (2025a); Zhao et al. (2025b); Shah et al. (2025) suggest that reflective behaviors in RLVR models originate from the base models rather than being learned through reinforcement learning. Dang et al.  Dang et al. (2025) observed a decline in pass@$k$ performance post-RLVR training, but their analysis was limited in scope. More importantly, they did not explore the relationship between the base model and the RL model. Deepseek-Math Shao et al. (2024) also observed similar trends, but their study was limited to a single instruction-tuned model and two math benchmarks. In contrast, our work systematically investigates a wide range of models, tasks, and RL algorithms to accurately assess the effects of current RLVR methods and models. We further provide in-depth analyses, including accuracy distributions, reasoning coverage, perplexity trends, and comparison against distilled models, offering a comprehensive understanding of RLVR’s capabilities and limitations.

我们在此总结关于 RLVR 分析的关键相关工作，更全面的讨论见附录 B。尽管近期 RLVR 方法取得了令人印象深刻的经验结果（Guo 等，2025；Lambert 等，2024），其对推理的根本影响仍探索不足。若干研究（Liu 等，2025a；Zhao 等，2025b；Shah 等，2025）指出：RLVR 模型中的反思行为源于基座模型，而非通过强化学习习得。Dang 等（Dang 等，2025）观察到 RLVR 训练后 pass@$k$ 下降，但分析范围有限；更重要的是，他们未探究基座与 RL 模型之间的关系。Deepseek-Math（Shao 等，2024）也观察到类似趋势，但其研究限于单一指令微调模型与两个数学基准。相对地，我们的工作系统考察广泛模型、任务与 RL 算法，以准确评估当前 RLVR 方法与模型的效应；并进一步提供准确率分布、推理覆盖、困惑度趋势以及与蒸馏模型比较等深入分析，从而对 RLVR 的能力与局限给出全面理解。

7 Conclusion and Limitations

7 结论与局限

RLVR is widely regarded as a promising approach to enable LLMs to continuously self-improve and acquire novel reasoning capabilities. In this paper, we systematically investigate the effect of current RLVR methods on the reasoning capacity boundaries of LLMs. Surprisingly, our findings reveal that current RLVR rarely elicits fundamentally new reasoning patterns; instead, the reasoning capabilities of RLVR-trained models remain bounded by those of their base models. These results indicate that current RLVR methods have not fully realized the potential of reinforcement learning to elicit novel reasoning abilities in LLMs through exploration and exploitation. This limitation may stem from the lack of effective exploration strategies in the vast language space as we discussed in Section 5. Exploration in high-level abstraction, fine-grained credit assignment, and multi-turn agent-environment interactions may alleviate this problem. We hope the community will continue developing methods along these dimensions to unlock the potential of reinforcement learning to discover genuinely novel reasoning strategies.

RLVR 被广泛视为让 LLM 持续自我改进、获得新推理能力的有希望途径。本文系统考察当前 RLVR 方法对 LLM 推理能力边界的影响。令人惊讶的是，我们发现当前 RLVR 很少引出根本性的新推理模式；相反，RLVR 训练模型的推理能力仍受其基座所界定。这些结果表明：当前 RLVR 方法尚未充分实现「通过探索与利用引出 LLM 新推理能力」这一强化学习潜力。该局限或源于第 5 节所讨论的、在巨大语言空间中缺乏有效探索策略。高层抽象上的探索、细粒度信用分配，以及多轮智能体–环境交互，或可缓解这一问题。我们希望社区继续沿这些维度发展方法，以解锁强化学习发现真正新颖推理策略的潜力。

Despite our best efforts, this study has several limitations. Although we have attempted to evaluate as many strong, publicly available pure-RLVR models as possible, our analysis is constrained by the fact that many of the most capable models and training pipelines remain proprietary. Moreover, RL for LLM is rapidly evolving, and emerging techniques may mitigate some of the limitations identified here. Consequently, our conclusions should be interpreted with awareness of these practical constraints.

尽管已尽最大努力，本研究仍有若干局限。虽已尝试评估尽可能多的强、公开可用的纯 RLVR 模型，但许多最能干模型与训练管线仍属专有，分析因此受限。此外，面向 LLM 的 RL 演进迅速，新兴技术或许能缓解此处指出的部分局限。因此，解读我们的结论时须意识到这些现实约束。

Author Contributions

作者贡献

All authors made valuable contributions to the experimental design, analysis, and iteration, as well as to the writing, editing, and overall management of the project.

全体作者在实验设计、分析与迭代，以及写作、编辑与项目整体管理上均作出有价值贡献。

• • Yang Yue **(乐洋)**led the project, first discovered the phenomenon where RL pass@k is surpassed by the base model, and proposed the idea; designed the experiments and partially conducted experiments; took primary responsibility for writing the manuscript.
• • Zhiqi Chen conducted substantial experiments, including pass@k evaluation across models and benchmarks, and the perplexity analysis; contributed to discussions, figure creation, and manuscript review.
• • Rui Lu contributed to inspiration of the idea and conceptualization of the project, story writing and manual check of AI reasoning trajectory.
• • Andrew Zhao contributed to discussions on experimental design, proposed the perplexity-based analysis, and contributed to the early implementation of the RL training code.
• • Zhaokai Wang contributed to discussions of RLVR’s effect on reasoning boundary, writing, proofreading, and comprehensive manuscript review.
• • Yang Yue **(乐阳)**contributed to the training of visual reasoning model, discussions, proofreading and figure refinement.
• • Gao Huang & Shiji Song supervised the research, and assisted in writing the paper.

• • Yang Yue **（乐洋）**领导项目，首先发现 RL 的 pass@k 被基座模型超越的现象并提出想法；设计实验并部分执行实验；主要负责撰写手稿。
• • Zhiqi Chen 完成大量实验，包括跨模型与基准的 pass@k 评测以及困惑度分析；参与讨论、制图与手稿审阅。
• • Rui Lu 贡献想法灵感与项目概念化、故事写作，以及对 AI 推理轨迹的人工核查。
• • Andrew Zhao 参与实验设计讨论，提出基于困惑度的分析，并贡献 RL 训练代码的早期实现。
• • Zhaokai Wang 参与关于 RLVR 对推理边界影响的讨论、写作、校对与全面手稿审阅。
• • Yang Yue **（乐阳）**贡献视觉推理模型训练、讨论、校对与图示润色。
• • Gao Huang 与 Shiji Song 指导研究，并协助论文写作。

Acknowledgements

致谢

This work is supported in part by the National Key R&D Program of China under Grant 2022ZD0114903, the National Natural Science Foundation of China under Grants 42327901 and U24B20173, and the Scientific Research Innovation Capability Support Project for Young Faculty under Grant ZYGXQNJSKYCXNLZCXM-I20.

本工作部分受国家重点研发计划（批准号 2022ZD0114903）、国家自然科学基金（批准号 42327901 与 U24B20173），以及青年教师科研创新能力支持项目（批准号 ZYGXQNJSKYCXNLZCXM-I20）支持。

References

参考文献

Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report . arXiv preprint arXiv:2303.08774 . Cited by: §1 .

Ahmadian et al. (2024) A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstun, and S. Hooker Back to basics: revisiting reinforce style optimization for learning from human feedback in llms . ACL . Cited by: §A.1 , §4.3 .

Bai et al. (2025) S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2.5-vl technical report . arXiv preprint arXiv:2502.13923 . Cited by: §3.3 .

Brown et al. (2024) B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini Large language monkeys: scaling inference compute with repeated sampling . arXiv preprint arXiv:2407.21787 . Cited by: §1 , §3.1 .

Chen et al. (2025a) L. Chen, L. Li, H. Zhao, Y. Song, and Vinci R1-v: reinforcing super generalization ability in vision-language models with less than $3 . Note: https://github.com/Deep-Agent/R1-V Accessed: 2025-02-02 Cited by: Appendix B , §3.3 .

Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code . arXiv preprint arXiv:2107.03374 . Cited by: §A.2 , §2.2 .

Chen et al. (2025b) Y. Chen, Z. Yang, Z. Liu, C. Lee, P. Xu, M. Shoeybi, B. Catanzaro, and W. Ping Acereason-nemotron: advancing math and code reasoning through reinforcement learning . arXiv preprint arXiv:2505.16400 . Cited by: 2nd item .

Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168 . Cited by: §2.2 , §3.1 .

Dang et al. (2025) X. Dang, C. Baek, J. Z. Kolter, and A. Raghunathan Assessing diversity collapse in reasoning . In Scaling Self-Improving Foundation Models without Human Supervision , External Links: Link Cited by: Appendix B , §6 .

Gao et al. (2025) B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, Z. Tang, B. Wang, D. Zan, S. Quan, G. Zhang, L. Sha, Y. Zhang, X. Ren, T. Liu, and B. Chang Omni-math: a universal olympiad level mathematic benchmark for large language models . Cited by: §4.3 .

Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models . arXiv preprint arXiv:2407.21783 . Cited by: §1 , §3.1 , §3 .

Gulcehre et al. (2023) C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling . arXiv preprint arXiv:2308.08998 . Cited by: Appendix B .

Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv:2501.12948 . Cited by: Appendix B , §1 , §2.1 , §4.2 , §6 .

He et al. (2024) C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: a challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems . ACL . Cited by: §3.1 .

Hendrycks et al. (2021) D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt Measuring mathematical problem solving with the math dataset . NeurIPS . Cited by: §3.1 .

Holtzman et al. (2020) A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi The curious case of neural text degeneration . ICLR . Cited by: §1 , §2.2 .

Hu (2025) J. Hu REINFORCE++: a simple and efficient approach for aligning large language models . arXiv preprint arXiv:2501.03262 . Cited by: §4.3 .

Huang and Yang (2025) Y. Huang and L. F. Yang Winning gold at imo 2025 with a model-agnostic verification-and-refinement pipeline . arXiv preprint arXiv:2507.15855 . Cited by: 4th item .

Jaech et al. (2024) A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card . arXiv preprint arXiv:2412.16720 . Cited by: Appendix B , §1 , §4.1 .

Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code . ICLR . Cited by: §3.2 .

Lambert et al. (2024) N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. Tulu 3: pushing frontiers in open language model post-training . arXiv preprint arXiv:2411.15124 . Cited by: Appendix B , §1 , §6 .

Lewkowycz et al. (2022) A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al. Solving quantitative reasoning problems with language models . NeurIPS . Cited by: §3.1 .

Li et al. (2025) J. Li, H. Lin, H. Lu, K. Wen, Z. Yang, J. Gao, Y. Wu, and J. Zhang Questa: expanding reasoning capacity in llms via question augmentation . arXiv preprint arXiv:2507.13266 . Cited by: 2nd item .

Li et al. (2024) Z. Li, T. Xu, Y. Zhang, Z. Lin, Y. Yu, R. Sun, and Z. Luo Remax: a simple, effective, and efficient reinforcement learning method for aligning large language models . ICML . Cited by: §4.3 .

Liu et al. (2024) A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report . arXiv preprint arXiv:2412.19437 . Cited by: §3 .

Liu et al. (2023) J. Liu, C. S. Xia, Y. Wang, and L. Zhang Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation . In NeurIPS , Cited by: §3.2 .

Liu and Zhang (2025) J. Liu and L. Zhang Code-r1: reproducing r1 for code with reliable rewards . Note: https://github.com/ganler/code-r1 GitHub repository Cited by: Appendix B , §3.2 .

Liu et al. (2025a) Z. Liu, C. Chen, W. Li, T. Pang, C. Du, and M. Lin There may not be aha moment in r1-zero-like training – a pilot study . Note: https://oatllm.notion.site/oat-zero Notion Blog Cited by: Appendix B , §6 .

Liu et al. (2025b) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective . arXiv preprint arXiv:2503.20783 . Cited by: Appendix B , §3.1 , §4.3 .

Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter Decoupled weight decay regularization . In ICLR , Cited by: §4.3 .

Lu et al. (2024) P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K. Chang, M. Galley, and J. Gao MathVista: evaluating mathematical reasoning of foundation models in visual contexts . In ICLR , Cited by: §3.3 .

Lu et al. (2021) P. Lu, R. Gong, S. Jiang, L. Qiu, S. Huang, X. Liang, and S. Zhu Inter-gps: interpretable geometry problem solving with formal language and symbolic reasoning . In ACL , Cited by: §3.3 .

Luo et al. (2025) M. Luo, S. Tan, R. Huang, A. Patel, A. Ariyak, Q. Wu, X. Shi, R. Xin, C. Cai, M. Weber, C. Zhang, L. E. Li, R. A. Popa, and I. Stoica DeepCoder: a fully open-source 14b coder at o3-mini level . Note: Notion Blog Cited by: §3.2 .

Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning . nature 518 ( 7540 ), pp. 529–533 . Cited by: §1 , §5 .

Novikov et al. (2025) A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. Ruiz, A. Mehrabian, et al. AlphaEvolve: a coding agent for scientific and algorithmic discovery . arXiv preprint arXiv:2506.13131 . Cited by: 1st item .

Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback . NeurIPS . Cited by: Appendix B .

Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model . NeurIPS . Cited by: Appendix B .

Ramamurthy et al. (2023) R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi Is reinforcement learning (not) for natural language processing: benchmarks, baselines, and building blocks for natural language policy optimization . In ICLR , Cited by: §5 .

Rastogi et al. (2025) A. Rastogi, A. Q. Jiang, A. Lo, G. Berrada, G. Lample, J. Rute, J. Barmentlo, K. Yadav, K. Khandelwal, K. R. Chandu, et al. Magistral . arXiv preprint arXiv:2506.10910 . Cited by: §4.6 .

Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms . arXiv preprint arXiv:1707.06347 . Cited by: §2.1 , §4.3 .

Shah et al. (2025) D. J. Shah, P. Rushton, S. Singla, M. Parmar, K. Smith, Y. Vanjani, A. Vaswani, A. Chaluvaraju, A. Hojel, A. Ma, et al. Rethinking reflection in pre-training . arXiv preprint arXiv:2504.04022 . Cited by: Appendix B , §6 .

Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models . arXiv preprint arXiv:2402.03300 . Cited by: §A.1 , Appendix B , §4.3 , §6 .

Shen et al. (2025) H. Shen, Z. Zhang, K. Zhao, Q. Zhang, R. Xu, and T. Zhao VLM-r1: a stable and generalizable r1-style large vision-language model . Note: https://github.com/om-ai-lab/VLM-R1 Accessed: 2025-02-15 Cited by: Appendix B , §3.3 .

Sheng et al. (2024) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu HybridFlow: a flexible and efficient rlhf framework . arXiv preprint arXiv: 2409.19256 . Cited by: §4.3 .

Silver et al. (2017) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge . nature 550 ( 7676 ), pp. 354–359 . Cited by: §1 , §5 .

Silver and Sutton (2025) D. Silver and R. S. Sutton Welcome to the era of experience . Google AI . Cited by: 4th item .

Sutton et al. (1998) R. S. Sutton A. G. Barto et al. Reinforcement learning: an introduction . Vol. 1 , MIT press Cambridge . Cited by: §2.1 .

Team et al. (2025) K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. 5: scaling reinforcement learning with llms . arXiv preprint arXiv:2501.12599 . Cited by: §1 .

Wang et al. (2024) K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li Measuring multimodal mathematical reasoning with math-vision dataset . In NeurIPS Datasets and Benchmarks Track , Cited by: §3.3 .

Wang et al. (2025) S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, et al. Beyond the 80/20 rule: high-entropy minority tokens drive effective reinforcement learning for llm reasoning . NeurIPS . Cited by: Appendix B .

Wang et al. (2023) Y. Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources . NeurIPS . Cited by: Appendix B .

Williams (1992) R. J. Williams Simple statistical gradient-following algorithms for connectionist reinforcement learning . Machine learning 8 , pp. 229–256 . Cited by: §2.1 .

Yang et al. (2025a) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report . arXiv preprint arXiv:2505.09388 . Cited by: §4.6 .

Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report . arXiv preprint arXiv:2412.15115 . Cited by: §3.1 , §3 .

Yang et al. (2025b) A. Yang, B. Yu, C. Li, D. Liu, F. Huang, H. Huang, J. Jiang, J. Tu, J. Zhang, J. Zhou, et al. Qwen2.5-1m technical report . arXiv preprint arXiv:2501.15383 . Cited by: §3.2 .

Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale . arXiv preprint arXiv:2503.14476 . Cited by: Appendix B , §3.1 , §4.3 , §4.5 .

Yue et al. (2023) Y. Yue, B. Kang, Z. Xu, G. Huang, and S. Yan Value-consistent representation learning for data-efficient reinforcement learning . In AAAI , Cited by: §5 .

Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman Star: bootstrapping reasoning with reasoning . NeurIPS . Cited by: Appendix B .

Zeng et al. (2025) W. Zeng, Y. Huang, Q. Liu, W. Liu, K. He, Z. Ma, and J. He SimpleRL-zoo: investigating and taming zero reinforcement learning for open base models in the wild . arXiv preprint arXiv:2503.18892 . Cited by: Appendix B , §3.1 .

Zhang et al. (2025) K. Zhang, A. Lv, J. Li, Y. Wang, F. Wang, H. Hu, and R. Yan StepHint: multi-level stepwise hints enhance reinforcement learning to reason . arXiv preprint arXiv:2507.02841 . Cited by: 2nd item .

Zhao et al. (2025a) A. Zhao, Y. Wu, Y. Yue, T. Wu, Q. Xu, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang Absolute zero: reinforced self-play reasoning with zero data . NeurIPS . Cited by: Appendix B .

Zhao et al. (2025b) R. Zhao, A. Meterez, S. Kakade, C. Pehlevan, S. Jelassi, and E. Malach Echo chamber: rl post-training amplifies behaviors learned in pretraining . arXiv preprint arXiv:2504.07912 . Cited by: Appendix B , §6 .

Zheng et al. (2025) Y. Zheng, J. Lu, S. Wang, Z. Feng, D. Kuang, and Y. Xiong EasyR1: an efficient, scalable, multi-modality rl training framework . Note: https://github.com/hiyouga/EasyR1 Cited by: Appendix B , §3.3 .

Appendix

附录

Appendix Contents

附录目录

Appendix A Implementation Details

附录 A 实现细节

A.1 RLVR Algorithms

A.1 RLVR 算法

To reduce memory and computational overhead, several critic-free variants have been proposed. GRPO Shao et al. (2024) estimates the advantage with a normalized reward within a group of responses to the same question: $A_{i}=[r_{i}-\operatorname{mean}(\textbf{r})]/{\operatorname{std}(\textbf{r})}$, where $\textbf{r}=\{r_{1},\ldots,r_{G}\}$ denotes the set of rewards for a group of $G$ sampled responses. RLOO Ahmadian et al. (2024) instead adopts a leave-one-out baseline within each batch $\mathcal{B}$. Its advantage is defined as $A_{i}=r_{i}-\tfrac{1}{|\mathcal{B}|-1}\sum_{j\neq i}r_{j}$.

为降低内存与计算开销，已提出若干无 critic 变体。GRPO（Shao 等，2024）用同一问题下一组回答内的归一化奖励估计优势：$A_{i}=[r_{i}-\operatorname{mean}(\textbf{r})]/{\operatorname{std}(\textbf{r})}$，其中 $\textbf{r}=\{r_{1},\ldots,r_{G}\}$ 表示一组 $G$ 个采样回答的奖励集合。RLOO（Ahmadian 等，2024）则在每个 batch $\mathcal{B}$ 内采用 leave-one-out 基线，其优势定义为 $A_{i}=r_{i}-\tfrac{1}{|\mathcal{B}|-1}\sum_{j\neq i}r_{j}$。

A.2 Low-Variance pass@$k$ Estimation

A.2 低方差 pass@$k$ 估计

Directly computing pass@$k$ using only $k$ sampled outputs per problem can lead to high variance. To mitigate this, we follow the unbiased estimation method proposed by Chen et al.  Chen et al. (2021). Specifically, for each problem $x_{i}$ from the evaluation dataset $\mathcal{D}$, we generate $n$ samples ($n\geq k$) and count the number of correct samples as $c_{i}$. The unbiased estimator of pass@$k$ over the dataset is given by:

若每题仅用 $k$ 个采样输出直接计算 pass@$k$，方差可能很高。为缓解，我们遵循 Chen 等（Chen 等，2021）提出的无偏估计方法。具体地，对评测集 $\mathcal{D}$ 中每题 $x_{i}$，生成 $n$ 个样本（$n\geq k$），并把正确样本数记为 $c_{i}$。数据集上 pass@$k$ 的无偏估计为：

$$\text{pass@}k:=\mathbb{E}_{x_{i}\sim\mathcal{D}}\left[1-\frac{\binom{n-c_{i}}{k}}{\binom{n}{k}}\right] \tag{2}$$

With this formulation, we can easily estimate pass@$k$ with low variance across all $k\leq n$.

有了该形式，我们可轻松以低方差估计所有 $k\leq n$ 的 pass@$k$。

In our experiments, we set $n$ to the largest (i.e., rightmost) $k$ value in the pass@$k$ curves, typically 128, 256, or 1024. For example, in Figure 2, we use $n=128$ for MATH500, Minerva, and GSM8K, and $n=1024$ for AMC23 and AIME24. For the Olympiad benchmark, we set $n=128$ for the Qwen models and $n=1024$ for LLaMA-3.1-8B, due to its relatively lower base model capacity.

实验中，我们将 $n$ 设为 pass@$k$ 曲线最右（最大）$k$ 值，通常为 128、256 或 1024。例如图 2 中，MATH500、Minerva 与 GSM8K 取 $n=128$，AMC23 与 AIME24 取 $n=1024$。Olympiad 基准上，因 LLaMA-3.1-8B 基座能力相对较低，对 Qwen 模型取 $n=128$，对 LLaMA-3.1-8B 取 $n=1024$。

Appendix B More Related Works

附录 B 更多相关工作

Reinforcement Learning for LLM Reasoning. Since the emergence of LLMs, the post-training phase has proven crucial to enhance problem solving and reasoning abilities Ouyang et al. (2022). This stage typically falls into three main categories: supervised fine-tuning using human-curated or distilled data Wang et al. (2023), self-improvement iteration Zelikman et al. (2022); Gulcehre et al. (2023), and reinforcement learning Ouyang et al. (2022). Previously, a reward model or preferences between responses were employed for reward modeling Ouyang et al. (2022); Rafailov et al. (2023). Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has gained significant traction as a method to improve the reasoning capabilities of LLMs in domains such as mathematics and programming Lambert et al. (2024); Shao et al. (2024). An encouraging landmark work is OpenAI’s o1 model Jaech et al. (2024), which was among the first large-scale applications of RL for reasoning, achieving state-of-the-art results at the time of its release. Following this, Deepseek-R1 Guo et al. (2025) became the first open-weight model to match or surpass the performance of o1. A significant innovation introduced with R1 is the “zero” setting, where reinforcement learning is applied directly to the base LLM, bypassing any intermediate supervised tuning. This approach inspired a wave of open-source efforts to replicate or extend R1’s methodology and improve RL algorithms Zeng et al. (2025); Liu et al. (2025b); Yu et al. (2025); Liu and Zhang (2025); Zhao et al. (2025a); Wang et al. (2025). In parallel, reinforcement learning has also gained attention in the multimodal domain, driving advancements in multimodal reasoning Chen et al. (2025a); Shen et al. (2025); Zheng et al. (2025).

面向 LLM 推理的强化学习。自 LLM 兴起以来，后训练阶段被证明对增强解题与推理能力至关重要（Ouyang 等，2022）。该阶段通常落入三类：用人工整理或蒸馏数据做监督微调（Wang 等，2023）、自我改进迭代（Zelikman 等，2022；Gulcehre 等，2023），以及强化学习（Ouyang 等，2022）。此前，奖励建模多依赖奖励模型或回答间偏好（Ouyang 等，2022；Rafailov 等，2023）。近来，可验证奖励强化学习（RLVR）作为提升 LLM 在数学与编程等领域推理能力的方法受到广泛关注（Lambert 等，2024；Shao 等，2024）。一个鼓舞人心的里程碑是 OpenAI 的 o1 模型（Jaech 等，2024），它是最早大规模将 RL 用于推理的应用之一，发布时达到当时最优。随后，Deepseek-R1（Guo 等，2025）成为首个在性能上匹配或超越 o1 的开权模型。R1 引入的重要创新是「zero」设定：把强化学习直接施加于基座 LLM，跳过中间的监督微调。该方法激发一波开源努力，以复现或扩展 R1 的方法论并改进 RL 算法（Zeng 等，2025；Liu 等，2025b；Yu 等，2025；Liu 与 Zhang，2025；Zhao 等，2025a；Wang 等，2025）。与此并行，强化学习在多模态领域也受到关注，推动多模态推理进展（Chen 等，2025a；Shen 等，2025；Zheng 等，2025）。

Analysis of RLVR. Although there are many excellent open-source works and algorithmic designs in the field of RLVR, there remains a lack of deep understanding regarding the root effects of RLVR on LLM reasoning abilities and its limitations when starting from the base model. Several studies Liu et al. (2025a); Zhao et al. (2025b); Shah et al. (2025) highlight that the reflective behaviors observed in R1-like models actually emerge from the base models, rather than being introduced by RLVR training. Dang et al.  Dang et al. (2025) observed a phenomenon similar to our findings: Pass@k deteriorates rapidly and fails to recover with reinforcement learning, but this was seen only in a limited experimental setup with Qwen-2.5-0.5B on GSM8K. More importantly, they did not explore the relationship between the base model and the RL model. In contrast, our paper conducts systematic and rigorous experiments to show that not only reflective behaviors but all reasoning paths are already embedded in the base model. We further demonstrate that RLVR does not elicit any new reasoning abilities beyond the base model.

RLVR 的分析。尽管 RLVR 领域有许多优秀开源工作与算法设计，从基座出发时 RLVR 对 LLM 推理能力的根效应及其局限仍缺乏深入理解。若干研究（Liu 等，2025a；Zhao 等，2025b；Shah 等，2025）强调：类 R1 模型中观察到的反思行为实际来自基座模型，而非由 RLVR 训练引入。Dang 等（Dang 等，2025）观察到与我们类似的现象：Pass@k 在强化学习后迅速恶化且难以恢复，但仅见于 Qwen-2.5-0.5B 在 GSM8K 上的有限实验设定；更重要的是，他们未探究基座与 RL 模型的关系。相对地，本文通过系统、严格的实验表明：不仅反思行为，全部推理路径都已嵌入基座模型。我们进一步证明，RLVR 并未引出任何超越基座的新推理能力。

Appendix C Detailed Experimental Results

附录 C 详细实验结果

C.1 More Results on Mathematics and Coding

C.1 数学与编码上的更多结果

Figure 10: More results of SimpleRLZoo on GSM8K and AMC23.

图 10：SimpleRLZoo 在 GSM8K 与 AMC23 上的更多结果。

Figure 11: Oat-Zero-7B and DAPO-32B are evaluated on AIME24 and compared against their respective base models.

图 11：在 AIME24 上评测 Oat-Zero-7B 与 DAPO-32B，并与各自基座模型比较。

Figure 12: Coder-R1 on LiveCodeBench.

图 12：Coder-R1 在 LiveCodeBench 上的结果。

C.2 Validity of Chain-of-Thought on AIME24

C.2 AIME24 上思维链的有效性

We manually check the CoTs for the most challenging AIME24 benchmark. To begin, we introduce a filtering mechanism designed to eliminate easily guessable problems. Specifically, we prompt Qwen2.5-7B-Base to answer questions directly, without using chain-of-thought reasoning, and sample answers multiple times. If a problem can be answered correctly with a low but non-zero probability (e.g., <5%), we consider it to be guessable and remove it. Problems that can be directly answered correctly with a high probability are retained, as they are likely easier and solvable using valid CoTs.

我们对手工检查最具挑战的 AIME24 基准上的 CoT。首先引入过滤机制，剔除易被猜中的题。具体地，提示 Qwen2.5-7B-Base 直接答题、不使用思维链推理，并多次采样答案。若某题能以低但非零概率（如 <5%）直接答对，则视其为可猜并剔除；能以高概率直接答对的题予以保留，因为它们很可能较易、可用有效 CoT 解出。

Figure 13: Pass@$k$ curves of the base and SimpleRLZoo-7B models in the filtered AIME24.

图 13：过滤后 AIME24 上基座与 SimpleRLZoo-7B 模型的 pass@$k$ 曲线。

The base and RL model pass@$k$ curves on this filtered AIME24 can be found in Figure 13, showing a similar trending to previous results. Although this filtering method is heuristic, it proves to be effective. Applying it to AIME24 (30 questions) results in a subset of 18 questions. We then prompt the models to answer these filtered questions using CoT reasoning. Then we perform a manual inspection of all CoTs that led to correct answers on the hardest problems – those with an average accuracy below 5%. The base model answered 7 such questions, with 5 out of 6 containing at least one correct CoT (excluding one ambiguous case of correctness due to skipped reasoning steps). Similarly, the RL-trained model answered 6 questions, 4 of which included at least one correct CoT. These results suggest that even for the hardest questions in the challenging AIME24, base model can sample valid reasoning paths to solve the problems.

过滤后 AIME24 上基座与 RL 模型的 pass@$k$ 曲线见图 13，趋势与先前结果相似。尽管该过滤方法是启发式的，却证明有效：应用于 AIME24（30 题）得到 18 题子集。再提示模型用 CoT 推理回答这些过滤后的题，并对最难题——平均准确率低于 5%——上导向正解的全部 CoT 做人工检查。基座解出 7 道此类题，6 道中有 5 道含至少一条正确 CoT（排除一道因跳步导致正确性存疑）；RL 训练模型解出 6 道，其中 4 道含至少一条正确 CoT。这些结果表明：即便对挑战性 AIME24 中最难的题，基座模型也能采样到有效推理路径来解题。

C.3 Accuracy Distribution Visulization

C.3 准确率分布可视化

Figure 14: Accuracy histogram before and after RLVR with SimpleRLZoo models.

图 14：SimpleRLZoo 模型在 RLVR 前后的准确率直方图。

C.4 Perplexity Analysis

C.4 困惑度分析

To analyze how perplexity evolves over the course of RLVR training, we evaluated three RLVR checkpoints–early, middle, and final in Section 4.3 RL training. For each checkpoint, we sampled 32 responses per problem, computed the median among 32 perplexity values, and reported the average over the first 10 problems in the table. As expected, we observed that $\text{PPL}_{\text{Base}}(\mathbf{Y}_{\text{RL}}|x)$ gradually decreases as RL training progresses, indicating that RLVR mainly sharpens the distribution within the base model’s prior rather than expanding beyond it.

为分析 RLVR 训练过程中困惑度如何演化，我们在第 4.3 节 RL 训练中评估了早、中、晚三个 RLVR 检查点。对每个检查点，每题采样 32 条回答，计算 32 个困惑度值的中位数，并报告表中前 10 题的平均。如预期，我们观察到随 RL 训练推进，$\text{PPL}_{\text{Base}}(\mathbf{Y}_{\text{RL}}|x)$ 逐渐下降，表明 RLVR 主要是在基座先验内锐化分布，而非向外扩展。

Figure 15: Perplexity Evolution during RL Training.

图 15：RL 训练过程中的困惑度演化。

C.5 Different RLVR Algorithms

C.5 不同 RLVR 算法

We report several additional observations on different RLVR algorithms in Figure 8. First, DAPO achieves slightly higher pass@1 scores across all three datasets; however, its dynamic sampling strategy requires approximately $3\sim 6\times$ more samples per batch during training compared to other algorithms. Moreover, its performance drops significantly at $k=256$. Second, RLOO and Reinforce++ perform consistently well across the entire $k$ range (from 1 to 256), while maintaining efficient training costs, achieving a good balance between effectiveness and efficiency. Third, ReMax shows lower performance at both pass@1 and pass@256. We hypothesize that this is due to its use of the greedy response reward as the advantage baseline, which in the RLVR setting is binary (0 or 1) and highly variable. This likely results in unstable gradient updates during training.

我们报告图 8 中关于不同 RLVR 算法的若干额外观察。第一，DAPO 在三个数据集上的 pass@1 略高；但其动态采样策略在训练中每 batch 约需其他算法 $3\sim 6\times$ 的样本数，且在 $k=256$ 时表现显著下降。第二，RLOO 与 Reinforce++ 在整个 $k$ 范围（1 到 256）表现 始终良好，同时保持高效训练成本，在效力与效率间取得良好平衡。第三，ReMax 在 pass@1 与 pass@256 上均较低；我们猜测因其用贪心回答奖励作优势基线，而在 RLVR 设定中该奖励是二值（0 或 1）且高度可变，可能导致训练中梯度更新不稳定。

Model	Omni-MATH-Train	Omni-MATH-Test	MATH500
	pass@1	pass@256	pass@1	pass@256	pass@1	pass@256
Qwen2.5-7B	9.9	67.2	10.2	69.1	34.5	96.2
GRPO	26.1	66.3	25.1	68.3	74.4	97.2
PPO	27.2	65.8	26.8	69.2	75.2	97.2
ReMax	24.4	65.5	23.8	67.5	73.5	96.6
RLOO	28.6	66.4	28.1	69.2	75.0	97.4
Reinforce++	28.2	67.7	28.0	69.7	75.4	96.8
DAPO	31.4	66.1	26.5	67.0	75.6	96.4

Table 3: Detailed values for each point at pass@1 and pass@256 across different RL algorithms in Figure 8.

模型	Omni-MATH-Train		Omni-MATH-Test		MATH500
	pass@1	pass@256	pass@1	pass@256	pass@1	pass@256
Qwen2.5-7B	9.9	67.2	10.2	69.1	34.5	96.2
GRPO	26.1	66.3	25.1	68.3	74.4	97.2
PPO	27.2	65.8	26.8	69.2	75.2	97.2
ReMax	24.4	65.5	23.8	67.5	73.5	96.6
RLOO	28.6	66.4	28.1	69.2	75.0	97.4
Reinforce++	28.2	67.7	28.0	69.7	75.4	96.8
DAPO	31.4	66.1	26.5	67.0	75.6	96.4

表 3：图 8 中不同 RL 算法在 pass@1 与 pass@256 各点的详细数值。

Model	Omni-MATH-Train	Omni-MATH-Test	MATH500
	pass@1	pass@256	pass@1	pass@256	pass@1	pass@256
Qwen2.5-7B	9.9	67.2	10.2	69.1	34.5	96.2
GRPO-step150	26.1	66.3	25.1	68.3	74.4	97.2
GRPO-step300	33.6	65.3	27.1	66.6	75.4	96.0
GRPO-step450	42.5	64.3	28.3	63.9	76.3	95.4

Table 4: Detailed values at pass@1 and pass@256 across different RL training steps in Figure 1 (right).

模型	Omni-MATH-Train		Omni-MATH-Test		MATH500
	pass@1	pass@256	pass@1	pass@256	pass@1	pass@256
Qwen2.5-7B	9.9	67.2	10.2	69.1	34.5	96.2
GRPO-step150	26.1	66.3	25.1	68.3	74.4	97.2
GRPO-step300	33.6	65.3	27.1	66.6	75.4	96.0
GRPO-step450	42.5	64.3	28.3	63.9	76.3	95.4

表 4：图 1（右）中不同 RL 训练步数在 pass@1 与 pass@256 的详细数值。

C.6 Effects of KL and Rollout Number

C.6 KL 与 rollout 数的效应

Figure 16: Ablation Study on KL Loss and Rollout Number $n$. For increasing $n$ from 8 to 32, we keep the prompt batch size unchanged, which results in increased computation per training step. Due to resource constraints, we train for only 220 steps under this setting, leading to lower pass@1 as the model has not yet converged. Nevertheless, the model with $n=32$ achieves a higher pass@128, highlighting the positive effect of larger rollout numbers in improving pass@$k$ at higher values of $k$.

图 16：关于 KL 损失与 rollout 数 $n$ 的消融。将 $n$ 从 8 增至 32 时保持提示 batch 大小不变，导致每训练步计算量增加。受资源限制，该设定下仅训练 220 步，因模型尚未收敛，pass@1 较低。尽管如此，$n=32$ 的模型取得更高的 pass@128，凸显更大 rollout 数在提升高 $k$ 处 pass@$k$ 上的正向效应。

C.7 Solvable Problem Coverage Analysis

C.7 可解问题覆盖分析

Table 2 reports the fraction of problems categorized as four conditions: (1) both models solve the problem at least once, (2) only the base model solves it, (3) only the RLVR model solves it, and (4) neither model solves it in any of the $k$ samples. It highlights that there are many cases where the base model solves a problem but RLVR fails (type 2), and very few where RLVR succeeds while the base does not (type 3). Even in the rare type 3 cases (e.g., 1% or about 5 problems in MATH500), the base model is able to solve all of them when sampling 1024 times. These results support our conclusion that RLVR rarely solves problems the base model cannot and generally results in reduced coverage.

表 2 报告四类情形下的问题占比：（1）两模型至少各解出一次；（2）仅基座解出；（3）仅 RLVR 解出；（4）在 $k$ 次采样中两模型皆未解出。它凸显：基座解出而 RLVR 失败的情形很多（类型 2），而 RLVR 成功、基座失败的情形极少（类型 3）。即便在罕见的类型 3 情形中（例如 MATH500 上约 1%，约 5 题），当采样 1024 次时基座也能全部解出。这些结果支持我们的结论：RLVR 很少解出基座不能解的题，并通常导致覆盖缩小。

Models	Problem Indices
Qwen2.5-7B-Base	0, 1, 4, 6, 7, 8, 9, 11, 12, 14, 15, 16, 17, 18, 19, 22, 23, 24, 25, 26, 27, 28, 29
SimpleRL-Qwen2.5-7B	0, 1, 6, 7, 8, 9, 12, 14, 15, 16, 18, 22, 23, 24, 25, 26, 27, 28, 29

Table 5: Indices of solvable problems in AIME24 (starting from 0). An approximate subset relationship can be observed: most problems solved by the RL model are also solvable by the base model.

模型	问题索引
Qwen2.5-7B-Base	0, 1, 4, 6, 7, 8, 9, 11, 12, 14, 15, 16, 17, 18, 19, 22, 23, 24, 25, 26, 27, 28, 29
SimpleRL-Qwen2.5-7B	0, 1, 6, 7, 8, 9, 12, 14, 15, 16, 18, 22, 23, 24, 25, 26, 27, 28, 29

表 5：AIME24 中可解问题的索引（从 0 起）。可观察到近似子集关系：RL 模型解出的多数题，基座也能解。

Model	Solvable Problem Indices
Qwen2.5-7B-Instruct-1M	400, 402, 403, 407, 409, 412, 413, 417, 418, 419, 422, 423, 427, 432, 433, 436, 438, 439, 440, 444, 445, 448, 449
Coder-R1	400, 402, 403, 407, 412, 413, 417, 418, 419, 422, 423, 427, 430, 433, 438, 439, 440, 444, 445, 449

Table 6: Indices of solvable problems in LiveCodeBench (ranging from 400 to 450, starting from 0).

模型	可解问题索引
Qwen2.5-7B-Instruct-1M	400, 402, 403, 407, 409, 412, 413, 417, 418, 419, 422, 423, 427, 432, 433, 436, 438, 439, 440, 444, 445, 448, 449
Coder-R1	400, 402, 403, 407, 412, 413, 417, 418, 419, 422, 423, 427, 430, 433, 438, 439, 440, 444, 445, 449

表 6：LiveCodeBench 中可解问题的索引（范围 400 到 450，从 0 起）。

C.8 Temperature and Entropy Analysis

C.8 温度与熵分析

Figure 17: We found that the base model’s performance drops when the temperature exceeds 1.0, as it tends to generate more random and less coherent tokens. In contrast, the RL model’s performance remains relatively stable across different temperature settings. Therefore, we use $T=0.6$ in the main experiments, as it allows both models to demonstrate their best reasoning performance.

图 17：我们发现温度超过 1.0 时基座模型表现下降，因其倾向于生成更随机、更不连贯的 token。相对地，RL 模型表现在不同温度设定下相对稳定。因此主实验使用 $T=0.6$，使两模型都能展现其最佳推理表现。

Figure 18: Comparison of Base and RLVR Models with Matched Output Entropy. We evaluate the base model (Qwen2.5-7B) on each dataset using temperature $T=0.6$ and report its output entropy $E_{\text{base}}$ in the title of each figure. To enable a fair comparison, we increase the temperature of the RLVR model (SimpleRLZoo) until its output entropy approximately matches $E_{\text{base}}$. For example, on AMC23, we set $T=0.9$ to achieve $E_{\text{RL}}=0.47$. We also include RLVR results at $T=0.6$ as an additional baseline, which has lower entropy—e.g., 0.22 on AMC23 and 0.33 on MATH500.

图 18：匹配输出熵时基座与 RLVR 模型的比较。我们在各数据集上用温度 $T=0.6$ 评测基座（Qwen2.5-7B），并在每张图标题中报告其输出熵 $E_{\text{base}}$。为公平比较，提高 RLVR 模型（SimpleRLZoo）的温度，直至其输出熵近似匹配 $E_{\text{base}}$。例如在 AMC23 上取 $T=0.9$ 以达到 $E_{\text{RL}}=0.47$。我们也纳入 RLVR 在 $T=0.6$ 的结果作为额外基线，其熵更低——例如 AMC23 上为 0.22，MATH500 上为 0.33。

C.9 Training Dynamics

C.9 训练动态

Figure 19: The curves of training reward, response length, and generation entropy during training, corresponding to experiments in Section 4.

图 19：训练过程中训练奖励、回答长度与生成熵的曲线，对应第 4 节实验。

C.10 CoT Case Analysis

C.10 CoT 案例分析

Figure 20: Qwen2.5-Base-7B Correct Response - Case 1.

图 20：Qwen2.5-Base-7B 正确回答——案例 1。

Figure 21: Qwen2.5-Base-7B Correct Response - Case 2.

图 21：Qwen2.5-Base-7B 正确回答——案例 2。

Appendix D Prompt Templates

附录 D 提示模板

We provide the prompt templates used for training and evaluation in our experiments. The prompt for SimpleRL training and evaluation is shown in Figure 22, while the prompt for Oat-Zero is shown in Figure 23. For Code-R1 training, prompt in Figure 24 is adopted. For Code-R1 evaluation, we follow the original codebase and adopt the default templates from the benchmarks, including LiveCodeBench prompt (Figure 25), HumanEval+, and MBPP+ prompt (Figure 26). The prompt used for EasyR1 training and evaluation is shown in Figure 27. For VeRL-trained RL models, as discussed in Section 4.3 and Section 4.4, the training and evaluation prompts are provided in Figure 28. For evaluating Mistral and Magistral models on AIME24/25, prompts are provided in Figure 29. To ensure a fair comparison, the base models use the same prompts as their corresponding RL-trained counterparts during evaluation.

我们给出实验中用于训练与评测的提示模板。SimpleRL 训练与评测提示见图 22，Oat-Zero 提示见图 23。Code-R1 训练采用图 24 的提示；Code-R1 评测遵循原代码库，采用各基准默认模板，包括 LiveCodeBench 提示（图 25）、HumanEval+ 与 MBPP+ 提示（图 26）。EasyR1 训练与评测提示见图 27。对 VeRL 训练的 RL 模型，如第 4.3 与 4.4 节所述，训练与评测提示见图 28。在 AIME24/25 上评测 Mistral 与 Magistral 模型的提示见图 29。为公平比较，评测时基座模型使用与其对应 RL 训练版本相同的提示。

Figure 22: Prompt for SimpleRL Training and Evaluation. The base model uses the same prompt as the RL model during evaluation.

图 22：SimpleRL 训练与评测提示。评测时基座模型使用与 RL 模型相同的提示。

Figure 23: Prompt for Oat-Zero training and evaluation.

图 23：Oat-Zero 训练与评测提示。

Figure 24: Prompt for Code-R1 training.

图 24：Code-R1 训练提示。

Figure 25: Since Code-R1 does not specify an evaluation prompt, we adopt the original LiveCodeBench evaluation prompt. To encourage both the base and RL-trained models to generate code, we append ‘‘‘python to the end of the prompt. Using this setup, we reproduce a pass@$1$ score of 28.6, which is close to the reported 29.7.

图 25：因 Code-R1 未指定评测提示，我们采用原始 LiveCodeBench 评测提示。为鼓励基座与 RL 训练模型都生成代码，在提示末尾追加 ‘‘‘python。用该设定我们复现了 pass@$1$ 分数 28.6，接近所报告的 29.7。

Figure 26: Prompt for Code-R1 Evaluation on HumanEval+ and MBPP+.

图 26：Code-R1 在 HumanEval+ 与 MBPP+ 上的评测提示。

Figure 27: Prompt for EasyR1 training and evaluation.

图 27：EasyR1 训练与评测提示。

Figure 28: Prompt for VeRL training on Omni-math-train and evaluation on Omni-math-eval and MATH500.

图 28：VeRL 在 Omni-math-train 上训练、在 Omni-math-eval 与 MATH500 上评测的提示。

Figure 29: Prompt for Mistral and Magistral model evaluation.

图 29：Mistral 与 Magistral 模型评测提示。

Appendix E Broader Impacts

附录 E 更广泛影响

The potential negative social impacts of our method align with those typically associated with general LLM reasoning technologies. We emphasize the importance of adhering to the principles of fair and safe deployment in LLM systems.

我们方法潜在的负面社会影响，与一般 LLM 推理技术通常相关的影响一致。我们强调在 LLM 系统中坚持公平与安全部署原则的重要性。