CURE: Critique-Driven Unified Reinforcement Learning for Test-Time Self-Improvement
CURE: Critique-Driven Unified Reinforcement Learning for Test-Time Self-Improvement
CURE:面向测试时自我改进的批评驱动统一强化学习
Gaoling School of Artificial Intelligence, Renmin University of China
WeChat, Tencent Inc.
{aaaguirong, yankailin}@ruc.edu.cn
Equal Contribution. Project Lead. Corresponding Author.
Abstract
摘要
The evolution paradigm of Large Language Models (LLMs) is shifting from scaling training compute to scaling inference-time compute. While Reinforcement Learning with Verifiable Rewards (RLVR) has become a key engine for this transition, standard approaches often fail to equip models with the autonomous improvement capabilities required for test-time scaling. Existing critique-guided methods attempt to mitigate this by leveraging external feedback or ground-truth signals; however, these dependencies are unavailable at test time, fundamentally limiting the model’s capacity for continuous self-improvement. To bridge this gap, we propose CURE (Critique-driven Unified REinforcement Learning), a framework that jointly optimizes a single policy for standard solving, critiquing, and guided re-exploration. Uniquely, CURE facilitates re-exploration by generating strategic hints while discarding initial incorrect solutions to mitigate anchoring bias. Empirical results across diverse mathematical reasoning and code generation benchmarks demonstrate that CURE not only maintains competitive single-turn performance but, more importantly, unlocks effective inference-time scaling, enabling the model to significantly boost accuracy through iterative self-improvement.
大语言模型(LLM)的演进范式,正在从扩大训练计算转向扩大推理时计算。尽管带可验证奖励的强化学习(RLVR)已成为这一转变的关键引擎,标准方法却常常无法让模型具备测试时缩放所需的自主改进能力。已有的批评引导方法试图借助外部反馈或真值信号来缓解这一问题;然而这些依赖在测试时并不存在,从根本上限制了模型持续自我改进的能力。为填补这一缺口,我们提出 CURE(Critique-driven Unified REinforcement Learning),一个把单一策略同时优化于常规求解、批评和引导再探索的框架。CURE 的独特之处在于,它通过生成策略性提示来促进再探索,同时丢掉最初的错误解,以减轻锚定偏差。在多样的数学推理和代码生成基准上的实证结果表明,CURE 不仅保持有竞争力的单轮表现,更重要的是打开了有效的推理时缩放,使模型能够通过迭代自我改进显著提高准确率。
1 Introduction
1 引言
The evolution of Large Language Models (LLMs) (Ouyang et al., 2022; Yang et al., 2024) is witnessing a paradigm shift from scaling training-time compute to scaling test-time compute. Recent breakthroughs suggest that allocating more computational resources during inference can yield remarkable reasoning gains. This generally manifests in two paradigms: (1) Parallel Scaling, which improves performance by generating and evaluating multiple candidate solutions simultaneously (Wang et al., 2022; Snell et al., 2024); and (2) Sequential Scaling, which enhances reasoning by extending the chain of thought (CoT) to incorporate more deliberate reasoning behaviors (Yang et al., 2025d), typically induced through the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm (Jaech et al., 2024; Guo et al., 2025).
大语言模型(LLM)(Ouyang et al., 2022; Yang et al., 2024)的演进,正在经历从扩大训练时计算到扩大测试时计算的范式转变。近期突破表明,在推理阶段分配更多计算资源可以带来可观的推理收益。这通常表现为两种范式:(1)并行缩放,通过同时生成并评估多个候选解来提高表现(Wang et al., 2022; Snell et al., 2024);(2)顺序缩放,通过拉长思维链(CoT)以纳入更审慎的推理行为来增强推理(Yang et al., 2025d),这类行为通常由带可验证奖励的强化学习(RLVR)范式诱导(Jaech et al., 2024; Guo et al., 2025)。
However, current approaches to inference-time scaling remain relatively primitive and fall short of supporting human-like sequential refinement, characterized by iterative self-verification, critique, and correction. On one hand, parallel scaling merely samples independent solutions rather than iteratively improving them. On the other hand, current large reasoning models based on sequential scaling still struggle to perform reliable self-verification and correction (Kang et al., 2025), primarily because these capabilities are not explicitly optimized during training.
然而,现有的推理时缩放方法仍然相对初级,还不足以支撑人那样的顺序精化,即通过迭代的自我校验、批评和改正来推进。一方面,并行缩放只是采样相互独立的解,而不是对它们做迭代改进。另一方面,当前基于顺序缩放的大型推理模型仍然难以进行可靠的自检和改正(Kang et al., 2025),主要是因为这些能力在训练中并未被明确优化。
To address this, recent works have explored critique-guided exploration to improve the test-time improvement performance of LLMs, utilizing external feedback from stronger teachers (Xi et al., 2024; Yang et al., 2025b) or training specific critique models (Xi et al., 2025). Nevertheless, existing critique-guided methods suffer from fundamental dependencies that break the self-improvement loop during inference. They either rely on stronger external supervision that is impractical or prohibitively expensive, or depend on ground-truth signals to trigger the critique process—signals that are unavailable at test time. As a result, the policy model lacks the autonomy to solve, verify, and re-explore in a continuous loop, failing to realize the true potential of inference-time scaling (Zuo et al., 2025).
为应对这一点,近期工作探索了批评引导的探索,以改进 LLM 的测试时改进表现,要么利用更强教师的外部反馈(Xi et al., 2024; Yang et al., 2025b),要么训练专门的批评模型(Xi et al., 2025)。尽管如此,已有批评引导方法仍受制于会在推理阶段打断自我改进回路的根本依赖。它们要么依赖并不实际或代价过高的更强外部监督,要么依赖真值信号来触发批评过程——而这些信号在测试时并不可得。结果是,策略模型缺少在连续回路中求解、校验并再探索的自主性,无法实现推理时缩放的真正潜力(Zuo et al., 2025)。
To bridge this gap, we propose CURE (Critique-driven Unified REinforcement Learning), a unified framework that enables models to explore and self-improve continuously at test time. As shown in Figure 1, our framework jointly optimizes three atomic capabilities: (1) Solving: The fundamental ability to generate solutions for complex problems. (2) Critiquing: The capacity to verify the correctness of the model’s own outputs (Verification), identify potential flaws and generate strategic hints for improvement (Hint Generation). (3) Critique-Guided Re-Exploration: The ability to leverage these self-generated hints to reconstruct reasoning paths. Through this unified training process, the model integrates these capabilities into a cohesive system, forming a self-contained improvement loop that enables robust inference-time scaling performance.
为填补这一缺口,我们提出 CURE(Critique-driven Unified REinforcement Learning),一个使模型能够在测试时持续探索并自我改进的统一框架。如图 1 所示,我们的框架联合优化三项基本能力:(1)求解:为复杂问题生成解答的基本能力。(2)批评:校验模型自身输出正确性(校验)、识别潜在缺陷并生成用于改进的策略性提示(提示生成)的能力。(3)批评引导的再探索:利用这些自行生成的提示来重构推理路径的能力。通过这一统一训练过程,模型把这些能力整合进一个连贯的系统,形成一个自包含的改进回路,从而实现稳健的推理时缩放表现。
Specifically, our training process integrates these capabilities into a unified RLVR pipeline. A critical design choice in CURE is our handling of the re-exploration context. Unlike prior works (Zhang et al., 2025b; Xi et al., 2025) that append the entire history of incorrect attempts, we deliberately discard the initial incorrect solution during re-generation. Our pilot study (Figure 2) suggests that this design mitigates anchoring bias and prevents the model from repeating previous mistakes. Instead, we constrain the model to generate high-level strategic hints and force it to restart reasoning from a fresh state conditioned only on these hints. Finally, we optimize these capabilities within a unified objective: (1) Solving is enhanced by replaying successful re-exploration trajectories during RLVR; (2) Critiquing is optimized via direct supervised learning of golden verification judgments and RL with delayed accuracy-based rewards that prioritize hint utility; and (3) Re-exploration is aligned to effectively follow the generated strategic hints.
具体而言,我们的训练过程把这些能力整合进统一的 RLVR 流程。CURE 中一个关键的设计选择,是对再探索上下文的处理。与把全部错误尝试历史都拼接上去的已有工作(Zhang et al., 2025b; Xi et al., 2025)不同,我们在再生成时有意丢掉最初的错误解。我们的预实验(图 2)表明,这一设计可以减轻锚定偏差,并防止模型重复先前的错误。相反,我们约束模型生成高层策略性提示,并迫使它从只以这些提示为条件的全新状态重新开始推理。最后,我们在统一目标下优化这些能力:(1)求解通过在 RLVR 中回放成功的再探索轨迹来增强;(2)批评通过直接监督学习黄金校验判断,以及带延迟的、基于准确率、优先考虑提示效用的强化学习来优化;(3)再探索被对齐到有效遵循所生成的策略性提示。
Extensive experiments on math and code tasks show that CURE achieves competitive single-turn performance and superior scalability. On AIME25, CURE already improves by 2.8% at the second turn and reaches a total gain of 4.2% through iterative self-improvement. These results suggest that CURE successfully instills a robust self-improvement mechanism into LLMs.
在数学和代码任务上的大量实验表明,CURE 达到了有竞争力的单轮表现,以及更好的可缩放性。在 AIME25 上,CURE 在第二轮就已提高 2.8%,并通过迭代自我改进达到总共 4.2% 的收益。这些结果表明,CURE 成功地向 LLM 注入了稳健的自我改进机制。
2 Related Work
2 相关工作
RLVR for LLM Reasoning As a novel training paradigm for LLM reasoning, RLVR leverages verifiable reward functions, where they provide explicit binary feedback, e.g., assigning 1 to correct outputs and 0 to incorrect ones (Jaech et al., 2024; Guo et al., 2025; Team et al., 2025). Compared to reinforcement learning from human feedback (RLHF) (Ziegler et al., 2019; Ouyang et al., 2022), RLVR avoids reliance on subjective human evaluations or complex reward models, thereby making the training process more transparent and efficient. Consequently, RLVR is particularly well-suited to domains with crisp, executable evaluation protocols, such as mathematical reasoning (Hu et al., 2025; Zeng et al., 2025; He et al., 2025; Yang et al., 2025c) and code generation (Kumar et al., 2024; Xie et al., 2025), where solutions can be checked by symbolic solvers, compilers, or unit tests.
面向 LLM 推理的 RLVR 作为 LLM 推理的一种新训练范式,RLVR 利用可验证的奖励函数,提供显式的二元反馈,例如对正确输出给 1、对错误输出给 0(Jaech et al., 2024; Guo et al., 2025; Team et al., 2025)。与来自人类反馈的强化学习(RLHF)(Ziegler et al., 2019; Ouyang et al., 2022)相比,RLVR 避免依赖主观的人类评价或复杂的奖励模型,从而使训练过程更透明、更高效。因此,RLVR 特别适合带有清晰、可执行评估规程的领域,例如数学推理(Hu et al., 2025; Zeng et al., 2025; He et al., 2025; Yang et al., 2025c)和代码生成(Kumar et al., 2024; Xie et al., 2025),在这些领域中,解答可以由符号求解器、编译器或单元测试来检查。
Reliable Critique for LLM Reasoning LLM-based critics offer a promising way to provide automated and scalable supervision for LLM outputs (Luo et al., 2023; McAleese et al., 2024; Ke et al., 2024). Existing approaches can be broadly categorized into two paradigms: (1) one line of work decouples the actor and critic, training a dedicated critique model to evaluate and comment on the actor’s outputs (Xi et al., 2024, 2025; Yang et al., 2025b); (2) while another line unifies generation and critique within a single model, using self-generated feedback to drive refinement and learn how to revise responses accordingly (Zhang et al., 2025b; Li et al., 2025; Tang et al., 2025). In this work, we adopt the latter paradigm to equip a single model with unified capabilities for test-time self-improvement.
面向 LLM 推理的可靠批评 基于 LLM 的批评者,为 LLM 输出提供自动化、可扩展的监督开辟了一条有希望的路径(Luo et al., 2023; McAleese et al., 2024; Ke et al., 2024)。已有方法大致可分为两种范式:(1)一条路线把行动者与批评者分开,训练专门的批评模型来评估并评论行动者的输出(Xi et al., 2024, 2025; Yang et al., 2025b);(2)另一条路线把生成与批评统一在同一个模型里,用自行生成的反馈来驱动精化,并学习如何据此修改回答(Zhang et al., 2025b; Li et al., 2025; Tang et al., 2025)。在本工作中,我们采用后一种范式,使单一模型具备用于测试时自我改进的统一能力。
Guided Exploration for LLM Reasoning For complex and challenging tasks, prior works have shown that sampling a policy model multiple times may still fail to yield correct outcomes (Yue et al., 2025; Zhang et al., 2025b), motivating the need for more effective and guided exploration. For clarity, we categorize existing approaches into two groups. (1) The first group incorporates expert trajectories during training to shape priors over the search space (Yan et al., 2025; Huang et al., 2025; Ma et al., 2025; Fu et al., 2025; Zhang et al., 2025a), thus improving sample efficiency. However, these methods require costly, hard-to-scale expert supervision and may suffer from a distributional mismatch between expert demonstrations and the policy model’s sampling distribution. (2) The second group guides exploration with external textual signals such as critiques or hints (Tang et al., 2025; Wang et al., 2025a; Yang et al., 2025b; Xi et al., 2025), e.g., by highlighting errors in previous responses to steer subsequent attempts. Our work builds on the second approach, but uses high-level hints without appending incorrect solutions, in order to avoid anchoring bias induced by erroneous initial solutions and improve re-exploration.
面向 LLM 推理的引导探索 对于复杂而有挑战的任务,已有工作表明,即使对策略模型多次采样,仍可能得不到正确答案(Yue et al., 2025; Zhang et al., 2025b),这促使人们寻求更有效、更有引导的探索。为清楚起见,我们把已有方法分成两组。(1)第一组在训练中纳入专家轨迹,以塑造搜索空间上的先验(Yan et al., 2025; Huang et al., 2025; Ma et al., 2025; Fu et al., 2025; Zhang et al., 2025a),从而提高样本效率。不过,这些方法需要代价高、难以扩展的专家监督,并且可能受到专家示范与策略模型采样分布之间失配的影响。(2)第二组用批评或提示这类外部文本信号来引导探索(Tang et al., 2025; Wang et al., 2025a; Yang et al., 2025b; Xi et al., 2025),例如通过标出先前回答中的错误来引导后续尝试。我们的工作建立在第二种方法上,但使用高层提示,并且不拼接错误解,以避免由最初错误解引起的锚定偏差,并改进再探索。
图 1:我们的方法 CURE 总览。左图展示整体流程,并标出基线与 CURE 再探索策略之间的关键差异。右图给出在单一策略内联合优化多种能力的统一训练框架。
3 CURE: Critique-Driven Unified Reinforcement Learning
3 CURE:批评驱动的统一强化学习
Reinforcement Learning Objective We formulate the reasoning process as a Markov Decision Process (MDP), where a policy generates a response given a query . The goal of RL is to maximize the expected reward provided by the environment, subject to a Kullback-Leibler (KL) divergence constraint to maintain training stability. The standard objective is defined as:
where is the KL penalty coefficient and is a fixed reference policy (typically the initial model) used to prevent the tuned policy from deviating excessively from its original distribution.
强化学习目标 我们把推理过程表述为马尔可夫决策过程(MDP):策略 在给定查询 时生成回答 。强化学习的目标是最大化环境提供的期望奖励 ,并加以 Kullback-Leibler(KL)散度约束以保持训练稳定。标准目标定义为:
其中 是 KL 惩罚系数, 是固定的参照策略(通常是初始模型),用来防止调好的策略过度偏离其原始分布。
Group Relative Policy Optimization (GRPO) GRPO (Shao et al., 2024) has been widely adopted in RLVR to enhance LLM reasoning. To optimize Eq. (1), policy gradient methods require estimating an advantage function . GRPO leverages group-relative statistics to compute advantages, incurring low computational and memory overhead. Specifically, for each query , GRPO samples a group of outputs from the old policy . The advantage for all tokens in the -th output is derived by standardizing its reward against the group's distribution:
where denotes the outcome-based reward. The PPO-style surrogate objective used in GRPO is formulated as:
where limits the update step size to ensure training stability, and represents the importance sampling ratio.
组相对策略优化(GRPO) GRPO(Shao et al., 2024)已在 RLVR 中被广泛采用,以增强 LLM 推理。为优化式 (1),策略梯度方法需要估计优势函数 。GRPO 借助组内相对统计量来计算优势,计算和内存开销较低。具体而言,对每个查询 ,GRPO 从旧策略 采样一组 个输出 。第 个输出中所有 token 的优势 ,通过把它的奖励对该组分布做标准化得到:
其中 表示基于结果的奖励。GRPO 中使用的 PPO 风格代理目标表述为:
其中 限制更新步长以保证训练稳定,而 表示重要性采样比。
3.1 Framework Overview
3.1 框架总览
As highlighted in Section 1, current critique-guided RLVR methods typically depend on external supervision or ground-truth signals to trigger critiques, which are either unavailable or impractical during inference. This dependency fundamentally restricts the model’s ability to perform autonomous self-improvement at test time. To bridge this gap, we propose the Critique-driven Unified REinforcement Learning (CURE) framework. Unlike prior approaches that treat critique as a mere auxiliary task for exploration, CURE is designed to integrate three critical capabilities—Solving, Critiquing, and Critique-Guided Re-Exploration—into a single, unified policy. The core intuition is that even when a model fails to solve a problem directly, it often possesses the latent capacity to recognize errors (Liu et al., 2025; Feng et al., 2025) and generate actionable hints. By explicitly optimizing the model to critique its own outputs and use those critiques to guide subsequent re-generation, we equip the model with an intrinsic self-improvement mechanism that functions independently of external feedback.
如第 1 节所强调的,当前批评引导的 RLVR 方法通常依赖外部监督或真值信号来触发批评,而这些信号在推理时要么不可得,要么不实际。这种依赖从根本上限制了模型在测试时进行自主自我改进的能力。为填补这一缺口,我们提出批评驱动的统一强化学习(CURE)框架。与把批评仅仅当作探索的辅助任务的已有方法不同,CURE 被设计成把三项关键能力——求解、批评,以及批评引导的再探索——整合进单一的统一策略。核心直觉是:即使模型不能直接解出一道题,它也往往具备识别错误(Liu et al., 2025; Feng et al., 2025)并生成可操作提示的潜在能力。通过明确优化模型去批评自身输出,并用这些批评引导后续再生成,我们使模型具备一套不依赖外部反馈、可独立运作的内在自我改进机制。
Our framework establishes a co-training process that achieves two objectives: (1) Autonomous Self-Improvement Loop: We train the model not only to solve problems but also to autonomously verify its answers and generate strategic hints. This removes the reliance on ground truths, enabling the self-improvement loop to operate seamlessly during inference. (2) Unbiased Re-Exploration: Distinct from methods that condition re-generation on incorrect initial solutions (Jiang et al., 2025; Zhang et al., 2025b), we propose to discard the faulty reasoning path during re-exploration, and leverage high-level strategic hints to guide exploration from a fresh context, avoiding anchoring bias.
我们的框架建立了一套协同训练过程,以实现两个目标:(1)自主自我改进回路:我们训练模型不仅解题,还要自主校验答案并生成策略性提示。这去掉了对真值的依赖,使自我改进回路能在推理时顺畅运转。(2)无偏的再探索:与把再生成条件放在最初错误解上的方法(Jiang et al., 2025; Zhang et al., 2025b)不同,我们提出在再探索时丢掉有缺陷的推理路径,并借助高层策略性提示,从全新上下文引导探索,以避免锚定偏差。
Overview As illustrated in Figure 1, during each online RL iteration, the model first generates diverse solutions. We then selectively sample solution pairs to generate critiques containing verification judgments and strategic hints. For the guided re-generation, we concatenate the generated hint with the original problem—explicitly discarding the incorrect initial solution. Finally, the model is updated via a unified GRPO objective that jointly optimizes these capabilities.
总览 如图 1 所示,在每次在线强化学习迭代中,模型先生成多样的解。我们再有选择地采样成对的解,以生成包含校验判断和策略性提示的批评。对于引导再生成,我们把生成的提示与原问题拼接——并明确丢掉最初的错误解。最后,模型通过联合优化这些能力的统一 GRPO 目标来更新。
3.2 Training Pipeline
3.2 训练流程
Each CURE training iteration consists of three phases: Initial Solution Generation, Critique Generation, and Critique-Guided Re-Generation.
Phase 1: Initial Solution Generation For each problem , we first sample a group of initial solutions from the current policy . These solutions are evaluated using an outcome verifier to obtain binary rewards , where .
CURE 的每次训练迭代由三个阶段组成:初始解生成、批评生成,以及批评引导的再生成。
阶段 1:初始解生成 对每个问题 ,我们首先从当前策略 采样一组 个初始解 。这些解用结果校验器评估,得到二元奖励 ,其中 。
Phase 2: Critique Generation Rather than critiquing every response, we employ a Selective Sampling Strategy to construct a critique batch aimed at maximizing exploration efficiency while maintaining training stability. Specifically, we categorize queries based on their group-level accuracy () and prioritize sampling from “informative” instances—namely, Zero-pass () and Partial-pass () groups—where the model either fails completely or exhibits inconsistency. Crucially, we enforce a balanced 1:1 ratio of positive samples (correct, ) and negative samples (incorrect, ) to prevent the policy from collapsing into trivial verification patterns, such as consistently predicting a single class. Once the batch is constructed, for each sampled pair , the model generates a critique . As detailed in Appendix A, this critique comprises a verification judgment and a strategic hint , which serves as a conceptual guide for the subsequent re-exploration.
阶段 2:批评生成 我们并不对每一个回答都做批评,而是采用选择性采样策略来构造批评批次 ,目的是在保持训练稳定的同时最大化探索效率。具体而言,我们按组级准确率()对查询分类,并优先从“有信息量”的实例中采样——即零通过()和部分通过()组——在这些组里模型要么完全失败,要么表现不一致。关键的是,我们强制正样本(正确,)与负样本(错误,)保持 1:1 平衡,以防止策略塌缩到琐碎的校验模式,例如始终预测同一类。批次构造完成后,对每个采样对 ,模型生成批评 。如附录 A 所述,该批评包括校验判断 和策略性提示 ,后者作为后续再探索的概念引导。
Phase 3: Critique-Guided Re-Generation The strategic hint serves as a scaffold for re-exploration. We construct a guided prompt by concatenating the original problem and the hint: (see Appendix A). A key distinction of CURE is the deliberate exclusion of the initial incorrect solution . Unlike prior self-correction paradigms that append the wrong reasoning trace to the context, we force the model to reconstruct the reasoning path based on the problem and high-level hint only. This design effectively mitigates the anchoring bias caused by initial errors. As supported by our pilot experiments in Figure 2, discarding the incorrect context can reduce the repetition of wrong answers (“SameRatio”) and improve the overall re-generation accuracy compared to appending it (see Appendix C for details).
Conditioned on this fresh context, the model generates a group of guided solutions . These solutions are evaluated against the ground truth to obtain binary outcome rewards . Crucially, we employ the outcome of this re-exploration to assign a Delayed Reward to the critique itself. The total critique reward is defined as a combination of the guided generation accuracy and structural constraints:
阶段 3:批评引导的再生成 策略性提示 作为再探索的支架。我们把原问题和提示拼接,构造引导提示:(见附录 A)。CURE 的一个关键区别,是有意排除最初的错误解 。与把错误推理轨迹拼进上下文的已有自我改正范式不同,我们迫使模型只根据问题和高层提示来重构推理路径。这一设计有效减轻了由最初错误引起的锚定偏差。如图 2 中的预实验所支持的,丢掉错误上下文可以减少错误答案的重复(“SameRatio”),并且相对把它拼接上去,还能提高整体再生成准确率(细节见附录 C)。
以这一全新上下文为条件,模型生成一组 个引导解 。这些解对照真值评估,得到二元结果奖励 。关键的是,我们用这次再探索的结果,给批评 本身分配延迟奖励。总的批评奖励 定义为引导生成准确率与结构约束的组合:
Figure 2: Pilot Experiment Results. SameRatio denotes the proportion of re-exploration attempts that overlap with the initial incorrect solution, reflecting the model’s tendency to repeat its previous errors. Accuracy is the percentage of problems solved correctly, either during the initial generation (Phase 1) or re-generation (Phase 3).
图 2:预实验结果。SameRatio 表示与最初错误解发生重叠的再探索尝试所占比例,反映模型重复先前错误的倾向。Accuracy 是在初始生成(阶段 1)或再生成(阶段 3)中正确解出的问题百分比。
Here, represents the average success rate of the re-exploration group, aligning the critique objective with the goal of reasoning: a valid critique is defined strictly by its utility in leading to a correct solution. The auxiliary term ensures the structural validity of the critique by enforcing adherence to the predefined output schema (e.g., requiring specific keys like “High_Level_Hint”). We define the penalty function as:
where and are hyperparameters penalizing format violations and trivial hints (e.g., length ), respectively.
这里, 表示再探索组的平均成功率,从而把批评目标与推理的目标对齐:一个有效的批评,严格由其能否导向正确解答的效用来定义。辅助项 通过强制遵守预定义的输出模式(例如要求包含 “High_Level_Hint” 这类特定键)来保证批评的结构有效性。我们把罚项函数定义为:
其中 和 是分别惩罚格式违规和琐碎提示(例如长度 )的超参数。
3.3 Unified Training Objective
3.3 统一训练目标
Our framework optimizes a unified objective that jointly enhances the model’s capabilities in standard solving, critiquing, and guided solving. Each task is formulated as a GRPO objective (Eq. (3)), differing only in input conditioning and task-specific advantage definitions. The total objective is defined as an additive combination:
我们的框架优化一个统一目标,以同时增强模型在常规求解、批评和引导求解上的能力。每个任务都被表述为 GRPO 目标(式 (3)),差别只在输入条件和任务特定的优势定义。总目标定义为相加组合:
Solving Objective with Experience Replay The solving objective optimizes the standard generation policy . To effectively propagate the “0-to-1” breakthroughs achieved during re-exploration, we employ an Experience Replay strategy. Specifically, we selectively incorporate successful guided solutions back into the initial solution group , treating them as if they were generated directly from the problem (i.e., by masking the hint context). The augmented group facilitates a critical re-calibration of advantages. The successful guided trajectories, serving as positive anchors with reward , significantly elevate the group mean reward, particularly for previously zero-pass groups. Under GRPO’s normalization (Eq. (2)), this shift assigns strongly negative advantages to the original incorrect solutions while assigning positive advantages to the replayed ones. This mechanism provides the necessary gradient signal to disincentivize failure modes and reinforce correct reasoning paths.
带经验回放的求解目标 求解目标 优化标准生成策略 。为了有效传播再探索中取得的“从 0 到 1”突破,我们采用经验回放策略。具体而言,我们有选择地把成功的引导解 并回初始解组 ,把它们当作直接由问题 生成(也即把提示上下文掩掉)。增强后的组 便于对优势做一次关键的重新校准。成功的引导轨迹作为奖励 的正锚点,会显著抬高组平均奖励,尤其是对先前的零通过组。在 GRPO 的标准化(式 (2))下,这一偏移给原先的错误解分配强烈的负优势,同时给回放的解分配正优势。这一机制提供了必要的梯度信号,以抑制失败模式并强化正确的推理路径。
Critique Objective with Hybrid Advantages The critique objective optimizes the critique generation policy . Given the composite structure of the critique—comprising a verification judgment and a strategic hint —we design a Hybrid Advantage Function that adapts to the token segment:
where serves as the baseline, representing the average accuracy of the initial solution group.
For the Verification Segment, we assign a fixed positive advantage (1.0). In implementation, we explicitly prepend the ground-truth judgment (e.g., Error_Found: true, derived from the solution’s correctness) to the beginning of the critique response . This effectively transforms the verification task into a supervised optimization (akin to behavior cloning), ensuring the model grounds its critique on correct judgments and improving learning efficiency.
For the subsequent Hint Segment, we utilize the centered advantage . This formulation provides a rigorous quality control mechanism: it yields positive advantage values only when the hint leads to a guided performance exceeding the model’s baseline average (). Conversely, misleading or trivial hints that fail to improve the outcome receive negative or zero advantages. This contrastive signal effectively suppresses the generation of unhelpful text, incentivizing the model to produce strategic, actionable guidance.
带混合优势的批评目标 批评目标 优化批评生成策略 。鉴于批评的复合结构——包括校验判断 和策略性提示 ——我们设计了随 token 片段变化的混合优势函数 :
其中 作为基线,表示初始解组的平均准确率。
对校验片段,我们分配固定的正优势(1.0)。实现上,我们把真值判断(例如由解答正确性得到的 Error_Found: true)显式前置到批评回答 的开头。这实际上把校验任务变成了有监督优化(类似于行为克隆),确保模型把批评建立在正确判断上,并提高学习效率。
对随后的提示片段,我们使用中心化优势 。这一表述提供了严格的质量控制机制:只有当提示使引导后的表现超过模型基线平均()时,才给出正优势。相反,未能改进结果的误导或琐碎提示会得到负优势或零优势。这一对比信号有效抑制无益文本的生成,激励模型产出有策略、可操作的引导。
Guided Solving Objective Finally, the guided solving objective optimizes the policy to generate reasoning paths conditioned on the generated hints. This term ensures that the model remains compliant with its own strategic advice. Similar to the standard solving task, we apply the group-relative advantage normalization (Eq. (2)) within each guided solution group to facilitate stable policy updates.
引导求解目标 最后,引导求解目标 优化策略 ,以在所生成提示的条件下生成推理路径。这一项确保模型仍然遵循它自己的策略性建议。与标准求解任务类似,我们在每个引导解组 内应用组相对优势标准化(式 (2)),以便做稳定的策略更新。
Optimization Details. In practice, we implement the unified objective as follows. First, we compute advantages independently within each sub-task group, which standardizes the optimization signals across groups and prevents any group from dominating training. Second, to handle variable output lengths, we average the GRPO loss of each training sample over its generated tokens, as reflected in Eq. (3). Finally, we form the overall objective by summing the three task losses with equal weights, using a fixed 1 : 1 : 1 weighting scheme. This simple strategy works well for two reasons: (1) group-wise reward normalization together with per-token loss averaging naturally keeps the optimization scales of different samples comparable; and (2) the three sub-tasks use distinct prompt templates, which separate them in the input space and reduce destructive interference during joint optimization.
优化细节。 实践中,我们按如下方式实现统一目标。第一,在每个子任务组内独立计算优势,从而在各组之间标准化优化信号,并防止任一分组主导训练。第二,为处理可变的输出长度,我们把每个训练样本的 GRPO 损失对其生成的 token 取平均,如式 (3) 所示。最后,我们以固定的 1 : 1 : 1 加权,把三项任务损失相加,形成总体目标。这一简单策略效果很好,原因有二:(1)组内奖励标准化与按 token 平均损失,自然使不同样本的优化尺度可比;(2)三个子任务使用不同的提示模板,在输入空间中把它们分开,从而减少联合优化时的破坏性干扰。
4 Experiments
4 实验
4.1 Experimental Setup
4.1 实验设置
Base Models and Baselines We conduct experiments on two target models: Qwen2.5-7B-Instruct (Yang et al., 2024) and Qwen3-4B-Base (Yang et al., 2025a). This selection covers both instruction-tuned and pre-trained models across different scales, allowing us to evaluate the generalization of CURE. We employ the GRPO (Shao et al., 2024) algorithm as our primary baseline.
基座模型与基线 我们在两个目标模型上做实验:Qwen2.5-7B-Instruct(Yang et al., 2024)和 Qwen3-4B-Base(Yang et al., 2025a)。这一选择覆盖了不同规模的指令微调模型和预训练模型,使我们能够评估 CURE 的泛化能力。我们把 GRPO(Shao et al., 2024)算法作为主要基线。
Training Settings For mathematical reasoning, we construct our training dataset by downsampling a subset from DeepMath (He et al., 2025). To evaluate generalization, we also extend CURE to the code generation domain, following the setup in Wang et al. (2025b) to construct a training set from CodeContests (Li et al., 2022). All experiments are implemented using the verl framework (Sheng et al., 2025). For the GRPO baseline, we sample 8 responses per prompt. To ensure a fair comparison, we align the sampling budget for CURE by generating 4 initial solutions and 4 guided solutions per prompt, along with a single critique response for the selected batch. Complete hyperparameters and implementation details for both domains are provided in Appendix D.
训练设置 对数学推理,我们通过从 DeepMath(He et al., 2025)下采样一个子集来构造训练数据。为评估泛化能力,我们还把 CURE 扩展到代码生成领域,按 Wang et al.(2025b)的设置,从 CodeContests(Li et al., 2022)构造训练集。全部实验用 verl 框架(Sheng et al., 2025)实现。对 GRPO 基线,每个提示采样 8 个回答。为公平比较,我们把 CURE 的采样预算对齐为:每个提示生成 4 个初始解和 4 个引导解,并对所选批次再生成一条批评回答。两个领域的完整超参数与实现细节见附录 D。
Evaluation Settings We evaluate performance across two domains: (1) Mathematics: MATH500 (Hendrycks et al., 2020), AMC23, AIME24 (Zhang and Math-AI, 2024), AIME25 (Zhang and Math-AI, 2025), and OlympiadBench (He et al., 2024); (2) Code Generation: Codeforces (Penedo et al., 2025), MBPP (Austin et al., 2021), and LiveCodeBench (Jain et al., 2024). During inference, we set the sampling temperature to 0.6. We report two key metrics: (1) Standard Single-turn Accuracy: We report Avg@32 for smaller math datasets and Avg@2 for larger math and all code datasets to ensure reliability. (2) Inference-time Self-Improvement: This metric measures the final accuracy via our critique-guided loop. Specifically, we execute the re-generation process for up to 8 rounds (including the initial attempt), where the model re-generates only if its verification predicts “Incorrect”. (see Appendix F.2 for a comparison with unconditional strategy). Detailed self-verification results are provided in Appendix F.1.
评估设置 我们在两个领域上评估表现:(1)数学:MATH500(Hendrycks et al., 2020)、AMC23、AIME24(Zhang and Math-AI, 2024)、AIME25(Zhang and Math-AI, 2025)以及 OlympiadBench(He et al., 2024);(2)代码生成:Codeforces(Penedo et al., 2025)、MBPP(Austin et al., 2021)和 LiveCodeBench(Jain et al., 2024)。推理时,采样温度设为 0.6。我们报告两项关键指标:(1)标准单轮准确率:较小的数学数据集报告 Avg@32,较大的数学数据集以及全部代码数据集报告 Avg@2,以保证可靠性。(2)推理时自我改进:这一指标通过我们的批评引导回路衡量最终准确率。具体而言,再生成过程最多执行 8 轮(包括最初尝试),模型仅在其校验预测为 “Incorrect” 时才再生成(与无条件策略的比较见附录 F.2)。详细的自我校验结果见附录 F.1。
4.2 Main Results
4.2 主要结果
Table 1 summarizes the performance of CURE across mathematical reasoning and code generation benchmarks.
表 1 汇总了 CURE 在数学推理和代码生成基准上的表现。
Table 1: Main results on mathematical reasoning and code generation benchmarks. We report the accuracy for standard generation and the performance with inference-time self-improvement (up to rounds). Olym.: OlympiadBench; LCB: LiveCodeBench; CF: CodeForces.
| Method | AIME24 | AIME25 | AMC | MATH | Olym. | Avg. | MBPP | LCB | CF | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | ||||||||||
| Base Model | 11.3 | 8.9 | 52.2 | 74.9 | 39.4 | 37.3 | 64.0 | 25.4 | 5.8 | 31.7 |
| GRPO | 14.2 | 10.6 | 57.5 | 78.8 | 44.1 | 41.0 | 83.5 | 29.1 | 8.2 | 40.3 |
| + Self-Improve () | 13.3 | 10.5 | 57.7 | 79.2 | 46.4 | 41.4 | 83.2 | 30.6 | 9.5 | 41.1 |
| CURE (Ours) | 14.1 | 11.8 | 58.4 | 79.7 | 45.8 | 42.0 | 84.3 | 27.3 | 10.0 | 40.5 |
| + Self-Improve () | 16.0 | 16.0 | 60.9 | 82.8 | 48.7 | 44.9 | 85.6 | 28.5 | 13.0 | 42.4 |
| Qwen3-4B-Base | ||||||||||
| Base Model | 9.4 | 5.9 | 40.2 | 67.3 | 35.8 | 31.7 | 62.4 | 6.3 | 1.8 | 23.5 |
| GRPO | 20.9 | 17.5 | 65.7 | 85.3 | 51.8 | 48.2 | 78.3 | 19.2 | 12.4 | 36.6 |
| + Self-Improve () | 20.8 | 18.3 | 66.7 | 85.2 | 53.2 | 48.8 | 76.3 | 18.7 | 12.8 | 35.9 |
| CURE (Ours) | 21.4 | 17.3 | 66.8 | 84.5 | 52.3 | 48.5 | 79.4 | 23.5 | 12.5 | 38.5 |
| + Self-Improve () | 23.5 | 19.7 | 70.1 | 85.1 | 54.1 | 50.5 | 82.1 | 25.9 | 13.1 | 40.4 |
表 1:数学推理与代码生成基准上的主要结果。我们报告标准生成的准确率,以及带推理时自我改进(最多 轮)的表现。Olym.:OlympiadBench;LCB:LiveCodeBench;CF:CodeForces。
| Method | AIME24 | AIME25 | AMC | MATH | Olym. | Avg. | MBPP | LCB | CF | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | ||||||||||
| Base Model | 11.3 | 8.9 | 52.2 | 74.9 | 39.4 | 37.3 | 64.0 | 25.4 | 5.8 | 31.7 |
| GRPO | 14.2 | 10.6 | 57.5 | 78.8 | 44.1 | 41.0 | 83.5 | 29.1 | 8.2 | 40.3 |
| + Self-Improve () | 13.3 | 10.5 | 57.7 | 79.2 | 46.4 | 41.4 | 83.2 | 30.6 | 9.5 | 41.1 |
| CURE (Ours) | 14.1 | 11.8 | 58.4 | 79.7 | 45.8 | 42.0 | 84.3 | 27.3 | 10.0 | 40.5 |
| + Self-Improve () | 16.0 | 16.0 | 60.9 | 82.8 | 48.7 | 44.9 | 85.6 | 28.5 | 13.0 | 42.4 |
| Qwen3-4B-Base | ||||||||||
| Base Model | 9.4 | 5.9 | 40.2 | 67.3 | 35.8 | 31.7 | 62.4 | 6.3 | 1.8 | 23.5 |
| GRPO | 20.9 | 17.5 | 65.7 | 85.3 | 51.8 | 48.2 | 78.3 | 19.2 | 12.4 | 36.6 |
| + Self-Improve () | 20.8 | 18.3 | 66.7 | 85.2 | 53.2 | 48.8 | 76.3 | 18.7 | 12.8 | 35.9 |
| CURE (Ours) | 21.4 | 17.3 | 66.8 | 84.5 | 52.3 | 48.5 | 79.4 | 23.5 | 12.5 | 38.5 |
| + Self-Improve () | 23.5 | 19.7 | 70.1 | 85.1 | 54.1 | 50.5 | 82.1 | 25.9 | 13.1 | 40.4 |
In the standard single-turn setting, CURE consistently maintains comparable or superior performance relative to the GRPO baseline. For example, on Qwen2.5-7B-Instruct, CURE achieves an average accuracy improvement of +1.0% on math tasks and competitive performance gains on coding tasks (e.g., +1.8% on CodeForces). We attribute these gains primarily to the Experience Replay mechanism. By selectively replaying successful re-generated solutions for previously zero-pass groups, CURE effectively injects positive learning signals into the sparse reward landscape of hard problem clusters, thereby transforming zero-gradient stagnation into effective learning.
在标准单轮设置下,CURE 相对 GRPO 基线始终保持可比或更好的表现。例如,在 Qwen2.5-7B-Instruct 上,CURE 在数学任务上平均准确率提高 +1.0%,在编程任务上也有有竞争力的提升(例如在 CodeForces 上 +1.8%)。我们把这些收益主要归因于经验回放机制。通过对先前零通过组有选择地回放再生成成功的解,CURE 有效地把正学习信号注入难题簇稀疏的奖励景观,从而把零梯度停滞转化为有效学习。
The primary contribution of CURE lies in its ability to unlock effective test-time self-improvement. A central hypothesis of this work is that CURE instills a dynamic self-improvement mechanism that can be leveraged during inference. As shown in the “+ Self-Improve” rows of Table 1, CURE demonstrates a significant and sustained performance boost through iterative self-improvement. On Qwen2.5-7B-Instruct, inference-time scaling elevates the average accuracy from 42.0% to 44.9%, significantly surpassing the baseline’s scaling ceiling of 41.4%. Notably, this advantage is domain-agnostic: on CodeForces, scaling with CURE yields a substantial +3.0% improvement over its single-turn baseline, whereas GRPO shows minimal gain (+1.3%). These results confirm that CURE instills a robust policy that can perform effective test-time self-improvement across diverse reasoning domains.
CURE 的主要贡献在于,它能够解锁有效的测试时自我改进。这项工作的一个核心假设是:CURE 向模型注入了一种可在推理时调用的动态自我改进机制。如表 1 中 “+ Self-Improve” 各行所示,CURE 通过迭代自我改进展现出显著且持续的性能提升。在 Qwen2.5-7B-Instruct 上,推理时扩展把平均准确率从 42.0% 提到 44.9%,明显超过基线 41.4% 的扩展上限。值得注意的是,这一优势并不限于某一领域:在 CodeForces 上,用 CURE 做扩展相对其单轮基线有 +3.0% 的明显提升,而 GRPO 的收益很小(+1.3%)。这些结果确认,CURE 注入的是一套稳健的策略,能够在多样的推理领域上做有效的测试时自我改进。
Figure 3: Inference-time self-improvement results on mathematical (a, b) and coding (c) benchmarks using Qwen2.5-7B-Instruct. The plots track accuracy changes over iterative rounds. For the coding task, results are reported only for GRPO and CURE, as ablation variants are primarily analyzed within the mathematical domain.
图 3:在 Qwen2.5-7B-Instruct 上,数学(a、b)与编程(c)基准的推理时自我改进结果。图中跟踪 轮迭代中准确率的变化。对编程任务,只报告 GRPO 和 CURE 的结果,因为消融变体主要在数学领域内分析。
The trajectory analysis in Figure 3 further reveals the qualitative difference between methods. CURE extends the effective scaling window to 5-6 rounds, achieving a higher and more stable peak accuracy. In contrast, the GRPO baseline exhibits only a marginal initial rise followed by stagnation or oscillation. This disparity highlights that standard RLVR fails to equip the model with a constructive self-improvement mechanism. By contrast, our unified training framework successfully converts test-time compute into tangible performance gains.
图 3 中的轨迹分析进一步揭示了各方法之间的质的差异。CURE 把有效扩展窗口拉长到 5–6 轮,并达到更高、更稳定的峰值准确率。相比之下,GRPO 基线只在开始时略有上升,随后陷入停滞或振荡。这一反差说明,标准 RLVR 未能让模型具备建设性的自我改进机制。相比之下,我们的统一训练框架成功把测试时计算转化为实在的性能收益。
图 4:多轮自我改进案例研究。
To further inspect the self-improvement process, we present a concrete case study in Figure 4. As shown, from Turn 1 to Turn 4, the model consistently misinterprets the geometric configuration across different aspects. Guided by self-generated hints, the model is prompted to re-evaluate the relationships among segments, ultimately arriving at the correct solution in Turn 5.
为进一步检视自我改进过程,我们在图 4 中给出一个具体案例。如图所示,从第 1 轮到第 4 轮,模型在不同方面持续误读几何配置。在自行生成的提示引导下,模型被促使重新评估各线段之间的关系,最终在第 5 轮得到正确解答。
4.3 Inference-Time Scaling Analysis
4.3 推理时扩展分析
To further analyze how CURE uses additional test-time budget, we compare several inference strategies on MATH500 with Qwen2.5-7B-Instruct under the same maximum generation budget. Specifically, we consider standard decoding (Pass@1), majority voting, Best-of- with an external reward model, and our iterative self-improvement. Here, denotes the number of parallel samples for majority voting and Best-of-, while denotes the maximum number of sequential refinement turns for iterative self-improvement. Since these strategies allocate the generation budget differently (parallel sampling versus sequential refinement), we further report token and latency statistics in Table 3. For Best-of-, we use Math-Shepherd-7B as the external reward model for candidate selection.
为进一步分析 CURE 如何使用额外的测试时预算,我们在 MATH500 上,用 Qwen2.5-7B-Instruct,在相同的最大生成预算下比较若干推理策略。具体而言,我们考虑标准解码(Pass@1)、多数投票、带外部奖励模型的 Best-of-,以及我们的迭代自我改进。这里, 表示多数投票和 Best-of- 的并行样本数,而 表示迭代自我改进的最大顺序精炼轮数。由于这些策略以不同方式分配生成预算(并行采样对顺序精炼),我们还在表 3 中报告 token 与延迟统计。对 Best-of-,我们用 Math-Shepherd-7B 作为候选选择的外部奖励模型。
Table 2: Comparison of inference-time strategies on MATH with Qwen2.5-7B-Instruct (%). For majority voting and Best-of-, the budget is the number of sampled candidates ; for iterative self-improvement, it is the maximum number of refinement turns .
| Inference Strategy | Budget | GRPO | CURE |
|---|---|---|---|
| Standard (Pass@1) | 1 | 78.8 | 79.7 |
| Majority Voting | 4 | 81.4 | 82.2 |
| 8 | 81.8 | 82.6 | |
| Best-of- (Math-Shepherd-7B) | 4 | 82.6 | 81.2 |
| 8 | 82.6 | 81.0 | |
| Iterative Self-Improve (Ours) | 4 | 79.4 | 82.4 |
| 8 | 79.2 | 82.8 |
表 2:在 MATH 上用 Qwen2.5-7B-Instruct 比较推理时策略(%)。对多数投票和 Best-of-,预算是采样候选数 ;对迭代自我改进,预算是最大精炼轮数 。
| Inference Strategy | Budget | GRPO | CURE |
|---|---|---|---|
| Standard (Pass@1) | 1 | 78.8 | 79.7 |
| Majority Voting | 4 | 81.4 | 82.2 |
| 8 | 81.8 | 82.6 | |
| Best-of- (Math-Shepherd-7B) | 4 | 82.6 | 81.2 |
| 8 | 82.6 | 81.0 | |
| Iterative Self-Improve (Ours) | 4 | 79.4 | 82.4 |
| 8 | 79.2 | 82.8 |
As shown in Table 2, CURE consistently outperforms GRPO under iterative self-improvement and majority voting strategies. Notably, under the Best-of- strategy, CURE’s performance slightly drops and underperforms GRPO. We hypothesize this is primarily due to a distribution shift: the external reward model (Math-Shepherd-7B) is largely trained on standard, straightforward reasoning trajectories. Consequently, it may struggle to accurately evaluate—or might actively penalize—the unique self-correction patterns, reflection steps, or revised trajectories inherently generated by CURE. This limitation of relying on an out-of-distribution external reward model further highlights the advantage of our Iterative Self-Improvement approach. By relying on the model’s intrinsic ability to refine its own outputs, our strategy avoids the format-alignment issues of external rerankers and achieves the highest overall performance.
如表 2 所示,在迭代自我改进和多数投票策略下,CURE 始终优于 GRPO。值得注意的是,在 Best-of- 策略下,CURE 的表现略有下降,并弱于 GRPO。我们猜想这主要是由于分布偏移:外部奖励模型(Math-Shepherd-7B)大多在标准、直截了当的推理轨迹上训练。因此,它可能难以准确评估——甚至可能主动惩罚——CURE 固有生成的独特自我改正模式、反思步骤或修订轨迹。依赖分布外外部奖励模型的这一局限,进一步凸显了我们迭代自我改进方法的优势。凭借模型改进自身输出的内在能力,我们的策略避开了外部重排器的格式对齐问题,并取得了最高的总体表现。
Table 3: Efficiency and latency trade-off on MATH500 under the same maximum generation budget. Avg. Tokens denotes the total number of generated tokens per query. For BoN, we conservatively exclude the additional latency of the external reward model used for reranking.
| Method | Acc. (%) | Avg. Tokens | Latency (s) |
|---|---|---|---|
| GRPO + BoN () | 82.6 | 6400 | 0.44 |
| CURE + Iter. () | 82.8 | 1827 | 0.43 |
表 3:在相同最大生成预算下,MATH500 上的效率与延迟权衡。Avg. Tokens 表示每个查询生成的 token 总数。对 BoN,我们保守地未计入用于重排的外部奖励模型的额外延迟。
| Method | Acc. (%) | Avg. Tokens | Latency (s) |
|---|---|---|---|
| GRPO + BoN () | 82.6 | 6400 | 0.44 |
| CURE + Iter. () | 82.8 | 1827 | 0.43 |
To further quantify the practical trade-off between accuracy and deployment cost, we compare GRPO + Best-of- () against CURE + iterative self-improvement () in terms of average generated tokens and latency per query on MATH500. As shown in Table 3, CURE achieves slightly higher accuracy while using substantially fewer generated tokens, with comparable reported latency in this setup. This indicates that CURE converts additional inference-time computation into more efficient self-improvement rather than merely increasing parallel sampling. We further provide the corresponding efficiency–latency comparison on OlympiadBench in Appendix F.4. On this harder benchmark, CURE still uses fewer generated tokens, but its sequential refinement loop incurs higher latency, highlighting that the latency advantage is task-dependent rather than universal.
为进一步量化准确率与部署成本之间的实际权衡,我们在 MATH500 上,按每个查询的平均生成 token 数和延迟,比较 GRPO + Best-of-()与 CURE + 迭代自我改进()。如表 3 所示,CURE 在准确率略高的同时,使用的生成 token 明显更少,并且在这一设置下报告的延迟相当。这表明,CURE 是把额外的推理时计算转化为更高效的自我改进,而不仅仅是增加并行采样。我们还在附录 F.4 中给出了 OlympiadBench 上对应的效率–延迟比较。在这一更难的基准上,CURE 仍然使用更少的生成 token,但其顺序精炼回路会带来更高延迟,说明延迟优势是依赖任务的,而非普遍成立。
4.4 Analysis of Exploration Dynamics
4.4 探索动态分析
To evaluate CURE’s ability to overcome exploration stagnation, we visualize the training dynamics of zero-pass problems—queries for which no correct solutions are obtained across all attempts—in Figure 5. For such samples, standard GRPO receives no effective learning signal.
为评估 CURE 克服探索停滞的能力,我们在图 5 中可视化 零通过问题——即在所有尝试中都没有得到正确解的查询——的训练动态。对这类样本,标准 GRPO 得不到有效的学习信号。
图 5:训练过程中零通过问题比例的动态(在 Qwen2.5-7B-Instruct 上,经 EMA 平滑)。
Although CURE initially shows a higher failure rate due to unstable critiques, it exhibits a crossover and sustained decline after about 200 steps, eventually dropping significantly below the GRPO baseline. This trajectory confirms that once the critique capability is grounded and continuously optimized, the guided re-exploration mechanism effectively converts hard failures into successes, unlocking effective learning signals for these hard samples. Furthermore, the ablation variant that does not optimize the hint generation ability (w/o Hint Optimization) plateaus at a higher error rate than CURE, underscoring the necessity of explicit hint optimization for sustaining long-term exploration gains.
尽管 CURE 起初因批评尚不稳定而表现出更高的失败率,但大约 200 步之后出现交叉并持续下降,最终明显低于 GRPO 基线。这一轨迹确认:一旦批评能力被锚定并持续优化,引导再探索机制就能有效地把困难失败转化为成功,为这些难样本解锁有效的学习信号。此外,不优化提示生成能力的消融变体(w/o Hint Optimization)会在高于 CURE 的错误率上趋于平稳,这凸显了显式优化提示对于维持长期探索收益的必要性。
4.5 Ablation Studies
4.5 消融实验
To disentangle the contributions of key components in our CURE framework, we conduct a series of ablation studies. We first analyze the two sub-components of the Critique—Verification and Hint—by selectively removing their optimization objectives. Subsequently, we examine the impact of the Experience Replay strategy.
为拆开 CURE 框架中关键组件各自的贡献,我们做了一系列消融实验。我们首先分析批评的两个子组件——校验与提示——方法是有选择地去掉它们的优化目标。随后,我们考察经验回放策略的影响。
Impact of Verification Optimization We first examine the variant w/o Verification Optimization, where the advantage for the verification segment is set to 0 (i.e., not trained). As shown in Figure 3, this model maintains a similar single-turn accuracy to the full CURE, suggesting that verification supervision does not significantly affect the base solving capability. However, its scaling trajectory is nearly flat. Lacking a calibrated verification ability, the model cannot effectively distinguish between correct and incorrect solutions during inference, rendering the conditional re-generation strategy ineffective. This underscores that a robust self-verification capability is crucial for successful inference-time self-improvement.
校验优化的影响 我们首先考察变体 w/o Verification Optimization,其中校验片段的优势被设为 0(即不训练)。如图 3 所示,这一模型与完整 CURE 保持相近的单轮准确率,说明校验监督并不显著影响基础求解能力。然而,它的扩展轨迹几乎是平的。由于缺少校准过的校验能力,模型在推理时无法有效区分正确解与错误解,使有条件的再生成策略失效。这凸显稳健的自我校验能力对成功的推理时自我改进至关重要。
Impact of Hint Optimization Next, we evaluate the variant w/o Hint Optimization, where the hint generation is not explicitly optimized via RL (the verification is still trained). We observe two key phenomena: (1) Lower Initial Performance: Particularly on harder benchmarks like OlympiadBench, the starting accuracy is noticeably lower than CURE. This confirms that without optimizing the hint generation policy, the quality of feedback during training is suboptimal, limiting the model’s ability to explore complex solution spaces. (2) Limited Scaling Potential: Although this variant shows some initial improvement in the first 3-4 rounds, it peaks earlier and achieves a lower total gain compared to the full CURE. This indicates that explicitly optimizing hint generation is essential for sustaining long-term growth during inference-time scaling.
提示优化的影响 接下来,我们评估变体 w/o Hint Optimization,其中提示生成未经强化学习显式优化(校验仍然训练)。我们观察到两个关键现象:(1)初始表现更低:尤其在 OlympiadBench 这类更难的基准上,起始准确率明显低于 CURE。这确认,若不优化提示生成策略,训练期间反馈的质量就会不够好,从而限制模型探索复杂解空间的能力。(2)扩展潜力有限:尽管这一变体在前 3–4 轮有一些初始提升,但相对完整 CURE,它更早见顶,总收益也更低。这表明,显式优化提示生成,对于在推理时扩展中维持长期增长是必要的。
Impact of Experience Replay Strategy Finally, we investigate whether to incorporate all successful guided solutions into corresponding solution groups or replaying only those from zero-pass groups. Our empirical results indicate that prioritizing the latter yields better training stability and final performance. We put the detailed results and comparisons in Appendix F.6.
经验回放策略的影响 最后,我们考察是把所有成功的引导解并入相应的解组,还是只回放来自零通过组的那些。经验结果表明,优先采用后者会带来更好的训练稳定性和最终表现。详细结果与比较见附录 F.6。
5 Conclusion
5 结论
In this work, we introduce CURE, a unified RL framework that empowers LLMs with autonomous self-improvement capabilities. By jointly optimizing solving, critiquing, and re-exploration within a single policy, CURE eliminates dependencies on external supervision during test-time exploration. Our experiments in math and code domains demonstrate that CURE not only maintains competitive single-turn performance but also unlocks effective inference-time scaling potential, enabling models to continuously perform a solve–verify–critique–explore loop. Future work may extend CURE to open-ended domains and scale the iterative loop to more complex, multi-turn reasoning chains.
在这项工作中,我们提出 CURE,一个统一的强化学习框架,使大语言模型具备自主自我改进能力。通过在单一策略内联合优化求解、批评和再探索,CURE 消除了测试时探索对外部监督的依赖。我们在数学和代码领域的实验表明,CURE 不仅保持有竞争力的单轮表现,还解锁了有效的推理时扩展潜力,使模型能够持续执行求解–校验–批评–探索回路。未来工作可以把 CURE 扩展到开放域,并把迭代回路扩展到更复杂的多轮推理链。
Limitations
局限
Though effective in enhancing the test-time self-improvement abilities of LLMs, our work has some limitations: (1) We conduct experiments only in verifiable domains (i.e., mathematics and code), where deterministic reward signals can be naturally leveraged to optimize each capability within our framework. Extending our method to open-domain tasks, where supervision is less explicit and rewards are inherently noisy, remains a practical yet challenging direction for future work. (2) Compared with vanilla RLVR, our framework incurs higher computational costs at each optimization step due to the joint optimization of multiple model capabilities. However, these additional costs enable the model to achieve sustained performance gains at test time through continuous self-improvement, outperforming vanilla baselines.
尽管在增强大语言模型的测试时自我改进能力上有效,我们的工作仍有一些局限:(1)我们只在可验证领域(即数学和代码)做实验,这些领域可以自然地利用确定性奖励信号,来优化框架中的各项能力。把方法扩展到开放域任务——那里监督更不明确、奖励本质上带噪声——仍然是未来工作中切实但有挑战的方向。(2)与普通 RLVR 相比,由于要联合优化模型的多种能力,我们的框架在每一步优化上会带来更高的计算成本。不过,这些额外成本使模型能够通过持续的自我改进,在测试时获得持续的性能收益,并超过普通基线。
Ethics Statement
伦理声明
In this work, we introduce a unified training framework that endows a single policy model with standard solving, self-critiquing, and critique-guided re-exploration abilities. Our method enables the optimized model to achieve continual test-time self-improvement via an iterative loop of proposing, verifying, critiquing, and re-exploring new solutions. Our work makes a positive step toward effective self-evolution of LLMs.
在这项工作中,我们提出一个统一训练框架,使单一策略模型具备常规求解、自我批评,以及批评引导的再探索能力。我们的方法使优化后的模型能够通过提出、校验、批评并再探索新解的迭代回路,实现持续的测试时自我改进。我们的工作朝着大语言模型有效的自我演化迈出了积极一步。
Acknowledgments
致谢
We sincerely thank all the anonymous reviewers and AC(s) for their helpful feedback and constructive suggestions. This work was supported by The National Natural Science Foundation of China (No. 62376273) and Beijing Natural Science Foundation (L253001).
我们衷心感谢所有匿名审稿人和领域主席(AC)提供的有益反馈与建设性建议。本工作得到国家自然科学基金(编号 62376273)和北京市自然科学基金(L253001)的资助。
References
参考文献
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
Zhangying Feng, Qianglong Chen, Ning Lu, Yongqian Li, Siqi Cheng, Shuangmu Peng, Duyu Tang, Shengcai Liu, and Zhirui Zhang. 2025. Is prm necessary? problem-solving rl implicitly induces prm capability in llms. arXiv preprint arXiv:2505.11227.
Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. 2025. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506.19767.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3828–3850.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, and 1 others. 2025. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290.
Hsiu-Yuan Huang, Chenming Tang, Weijie Liu, Saiyong Yang, and Yunfang Wu. 2025. Think outside the policy: In-context steered policy optimization. arXiv preprint arXiv:2510.26519.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv:2412.16720.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
Yuhua Jiang, Yuwen Xiong, Yufeng Yuan, Chao Xin, Wenyuan Xu, Yu Yue, Qianchuan Zhao, and Lin Yan. 2025. Pag: Multi-turn reinforced llm self-correction with policy as generative verifier. arXiv preprint arXiv:2506.10406.
Liwei Kang, Yue Deng, Yao Xiao, Zhanfeng Mo, Wee Sun Lee, and Lidong Bing. 2025. First try matters: Revisiting the role of reflection in reasoning models. arXiv preprint arXiv:2510.08308.
Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, and 1 others. 2024. Critiquellm: Towards an informative critique generation model for evaluation of large language model generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13034–13054.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, and 1 others. 2024. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
Ang Li, Yifei Wang, Zhihang Yuan, Stefanie Jegelka, and Yisen Wang. 2025. Lanpo: Bootstrapping language and numerical feedback for reinforcement learning in llms. arXiv preprint arXiv:2510.16552.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
Xiaoyuan Liu, Tian Liang, Zhiwei He, Jiahao Xu, Wenxuan Wang, Pinjia He, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. Trust, but verify: A self-verification approach to reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505.13445.
Liangchen Luo, Zi Lin, Yinxiao Liu, Lei Shu, Yun Zhu, Jingbo Shang, and Lei Meng. 2023. Critique ability of large language models. arXiv preprint arXiv:2310.04815.
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, and 1 others. 2025. Learning what reinforcement learning can’t: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527.
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. 2024. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. 2025. Codeforces. https://huggingface.co/datasets/open-r1/codeforces.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pages 1279–1297.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.
Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Saiyong Yang, and Yunfang Wu. 2025. Do not step into the same river twice: Learning to reason from trial and error. arXiv preprint arXiv:2510.26109.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599.
Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun Li, Jiaqing Liang, Sihang Jiang, Zhaoqian Dai, Shuguang Ma, Fei Yu, and Yanghua Xiao. 2025a. Hint: Helping ineffective rollouts navigate towards effectiveness. arXiv preprint arXiv:2510.09388.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. 2025b. Co-evolving llm coder and unit tester via reinforcement learning. arXiv preprint arXiv:2506.03136.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. 2025c. Octothinker: Mid-training incentivizes reinforcement learning scaling. arXiv preprint arXiv:2506.20512.
Zhiheng Xi, Jixuan Huang, Xin Guo, Boyang Hong, Dingwen Yang, Xiaoran Fan, Shuo Li, Zehui Chen, Junjie Ye, Siyu Yuan, and 1 others. 2025. Critique-rl: Training language models for critiquing through two-stage reinforcement learning. arXiv preprint arXiv:2510.24320.
Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, and 1 others. 2024. Enhancing llm reasoning via critique models with test-time and training-time supervision. arXiv preprint arXiv:2411.16579.
Zhihui Xie, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong, and 1 others. 2025. Teaching language models to critique via reinforcement learning. arXiv preprint arXiv:2502.03492.
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.
Wenkai Yang, Jingwen Chen, Yankai Lin, and Ji-Rong Wen. 2025b. Deepcritic: Deliberate critique with large language models. arXiv preprint arXiv:2505.00662.
Wenkai Yang, Weijie Liu, Ruobing Xie, Yiju Guo, Lulu Wu, Saiyong Yang, and Yankai Lin. 2025c. Laser: Reinforcement learning with last-token self-rewarding. arXiv preprint arXiv:2510.14943.
Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. 2025d. Towards thinking-optimal scaling of test-time compute for LLM reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892.
Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. 2025a. Stephint: Multi-level stepwise hints enhance reinforcement learning to reason. arXiv preprint arXiv:2507.02841.
Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, and Helen Meng. 2025b. Critique-grpo: Advancing llm reasoning with natural language and numerical feedback. arXiv preprint arXiv:2506.03106.
Yifan Zhang and Team Math-AI. 2024. American invitational mathematics examination (aime) 2024.
Yifan Zhang and Team Math-AI. 2025. American invitational mathematics examination (aime) 2025.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, and 1 others. 2025. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084.
A Prompt Templates
A 提示模板
A.1 Prompt Templates for Main Experiments
A.1 主实验的提示模板
Critique Prompt Template
批评提示模板
You are given a problem and an AI-generated solution.
Problem:
{question}
AI Solution:
{solution}
Your tasks:
- Carefully read Problem and the entire AI Solution from start to finish.
- Identify the single earliest step or statement whose error affects the final answer.
- Ignore any corrected or irrelevant intermediate errors.
- If the final answer is missing, this must be considered an error.
- If multiple issues propagate, choose the earliest one.
- If no errors propagate to the final answer, set ERROR_FOUND: false and proceed to task 5.
- Quote the minimal snippet that contains this issue.
- Briefly explain why this issue affects the final answer.
- Produce a high-level hint that helps a student avoid making the same kind of error (if any error was found) or ensure correctness (if the solution was flawless).
- Do NOT provide the next step or the correct answer.
- The hint must be strategic and grounded in the problem’s context, but avoid referencing specific numbers, symbols, or lines from the solution.
Rules:
- Do not rewrite or solve the problem.
- The hint must be self-contained and avoid specific computations or targeted fixes.
- Even when ERROR_FOUND: false, you should still provide a general hint relevant to the overall approach.
- You must output all four fields below in order.
Output format:
ERROR_FOUND: true/false
ERROR_QUOTE: <the minimal snippet or “”>
WHY_IT_MATTERS: <1–2 sentence explanation or “”>
HIGH_LEVEL_HINT:
你将得到一道问题和一份由人工智能生成的解答。
Problem:
{question}
AI Solution:
{solution}
你的任务:
- 从头到尾仔细阅读问题和整份人工智能解答。
- 找出影响最终答案的、最早的那一个出错步骤或陈述。
- 忽略任何随后已被改正或与结果无关的中间错误。
- 若最终答案缺失,必须视为错误。
- 若有多个问题向前传播,选择最早的那一个。
- 若没有任何错误传播到最终答案,将 ERROR_FOUND 设为 false,并进入任务 5。
- 引用包含该问题的最短片段。
- 简要说明该问题为何影响最终答案。
- 给出一条高层提示:若发现错误,帮助学生避免同类错误;若解答无误,则帮助确保正确性。
- 不要提供下一步,也不要给出正确答案。
- 提示必须有策略性,并立足于问题的语境,但避免引用解答中的具体数字、符号或行文。
规则:
- 不要改写或求解该问题。
- 提示必须自成一体,并避免具体计算或针对性修补。
- 即使 ERROR_FOUND: false,仍应给出与总体思路相关的一般提示。
- 必须按顺序输出下面全部四个字段。
输出格式:
ERROR_FOUND: true/false
ERROR_QUOTE: <最短片段或 “”>
WHY_IT_MATTERS: <一两句说明或 “”>
HIGH_LEVEL_HINT: <不含具体步骤或答案的一般策略提示>
Critique-Guided Generation Prompt Template
批评引导生成的提示模板
You are given a Problem and a Hint.
Task:
- Generate a new solution to the Problem.
- Carefully incorporate the guidance from the Hint, without making any explicit reference to the Hint in your output.
- Ensure the reasoning is consistent, clear, and logically sound.
- The final output should be a complete step-by-step solution to the Problem.
Input:
Problem:
{problem}
Hint:
{hint}
Output:
<your solution here, put your final answer within \boxed{}>
你将得到一道问题和一条提示。
任务:
- 为该问题生成一个新的解答。
- 仔细吸收提示中的引导,但不要在输出中显式提及该提示。
- 确保推理前后一致、清晰,并且逻辑站得住。
- 最终输出应是对该问题的完整逐步解答。
输入:
Problem:
{problem}
Hint:
{hint}
输出:
<在此给出解答,并把最终答案放在 \boxed{} 中>
A.2 Prompt Templates for Pilot Study
A.2 预实验的提示模板
Prompt Template for Generating High Level Hint
生成高层提示的提示模板
You are given a problem and an AI-generated solution.
Problem:
{problem}
Please reason step by step, and put your final answer within \boxed{}.
AI Solution:
{initial response}
Your tasks:
- Carefully read Problem and the entire AI Solution from start to finish.
- Identify the single earliest step or statement whose mistake survives to the final answer (i.e., it is NOT fully corrected later).
- If an earlier mistake is later corrected and no longer affects the final answer, ignore it.
- If multiple issues survive, choose the earliest one.
- Quote the minimal snippet that contains this issue.
- Briefly explain why this issue affects the final answer.
- Produce a high-level, general hint that helps a student avoid making the same kind of mistake.
- Do NOT provide the next step or the correct answer.
- Keep the hint general and transferable, not tied to specific numbers, symbols, or lines.
Rules:
- Do not rewrite or solve the problem.
- The hint should be strategic, 1–3 sentences, and avoid specific computations or targeted fixes.
Output format:
ERROR_QUOTE: <the minimal snippet or “ ” >
WHY_IT_MATTERS: <1–2 sentence explanation>
HIGH_LEVEL_HINT:
你将得到一道问题和一份由人工智能生成的解答。
Problem:
{problem}
请逐步推理,并把最终答案放在 \boxed{} 中。
AI Solution:
{initial response}
你的任务:
- 从头到尾仔细阅读问题和整份人工智能解答。
- 找出其错误一直保留到最终答案的、最早的那一个步骤或陈述(也即后来没有被完全改正)。
- 若更早的错误后来被改正,且不再影响最终答案,则忽略它。
- 若有多个问题保留下来,选择最早的那一个。
- 引用包含该问题的最短片段。
- 简要说明该问题为何影响最终答案。
- 给出一条高层、一般性的提示,帮助学生避免同类错误。
- 不要提供下一步,也不要给出正确答案。
- 提示应保持一般、可迁移,不要绑定到具体数字、符号或行文。
规则:
- 不要改写或求解该问题。
- 提示应有策略性,长度为 1–3 句,并避免具体计算或针对性修补。
输出格式:
ERROR_QUOTE: <最短片段或 “ ” >
WHY_IT_MATTERS: <一两句说明>
HIGH_LEVEL_HINT: <不含具体步骤或答案的一般策略提示>
Prompt Template for Hint-guided Re-exploration
提示引导再探索的提示模板
You are given a Problem and a Hint.
Task:
- Generate a new solution to the Problem.
- Carefully incorporate the guidance from the Hint.
- Ensure the reasoning is consistent, clear, and logically sound.
- The final output should be a complete step-by-step solution to the Problem.
Input:
Problem:
{problem}
Hint:
{hint}
Output:
<your improved solution here, put your final answer within \boxed{}>
你将得到一道问题和一条提示。
任务:
- 为该问题生成一个新的解答。
- 仔细吸收提示中的引导。
- 确保推理前后一致、清晰,并且逻辑站得住。
- 最终输出应是对该问题的完整逐步解答。
输入:
Problem:
{problem}
Hint:
{hint}
输出:
<在此给出改进后的解答,并把最终答案放在 \boxed{} 中>
Prompt Template for Generating Specific Critique
生成具体批评的提示模板
You are a mathematics expert. A student is trying to solve a question, and the student’s solution contains mistakes.
Task:
Briefly and step-by-step determine whether the student’s solution is correct.
If you find an error, ONLY point out where the error is and why it is wrong.
Do NOT continue solving the problem, do NOT provide the correct method, and do NOT give the correct final answer.
Question: {problem}
Student’s Solution: {initial response}
Critique:
你是一位数学专家。一名学生正在解题,且该学生的解答含有错误。
任务:
简要地、逐步地判断该学生的解答是否正确。
若发现错误,只指出错误在哪里以及为何是错的。
不要继续求解该问题,不要提供正确方法,也不要给出正确的最终答案。
Question: {problem}
Student’s Solution: {initial response}
Critique:
Prompt Template for Specific Critique-guided Re-exploration
具体批评引导再探索的提示模板
You are given a question, a previous wrong solution, and a critique that identifies where the previous solution went wrong.
Your task is to write a fully revised, correct solution.
Inputs:
Question:
{problem}
Previous Solution:
{initial response}
Critique:
{critique}
Please re-answer by:
- Use the critique to fix all identified mistakes. Ensure the revised solution addresses all issues raised in the critique.
- Provide a complete, clear, step-by-step solution from scratch.
- Do NOT quote or restate the previous solution. Do NOT reproduce the incorrect steps.
- Placing your final answer within \boxed{}.
你将得到一道题、一份先前的错误解答,以及一份指出先前解答错在何处的批评。
你的任务是写出一份全面修订后的正确解答。
输入:
Question:
{problem}
Previous Solution:
{initial response}
Critique:
{critique}
请按如下方式重新作答:
- 用批评修正所有已指出的错误。确保修订后的解答处理批评中提出的全部问题。
- 从头提供一份完整、清晰的逐步解答。
- 不要引用或复述先前解答。不要再现错误步骤。
- 把最终答案放在 \boxed{} 中。
B Usage of AI Assistants
B 人工智能助手的使用
We used AI assistants (specifically Gemini 3) solely for the purpose of refining the clarity, grammar, and flow of the manuscript’s writing. All scientific concepts, experimental designs, data analyses, and conclusions presented in this work are original and were conducted by the authors.
我们使用人工智能助手(具体为 Gemini 3)仅用于润色文稿的清晰度、语法与行文流畅。本工作中提出的全部科学概念、实验设计、数据分析和结论都是原创的,并由作者完成。
C Pilot Study
C 预实验
To investigate how different re-exploration strategies influence model performance, we conducted a pilot experiment comparing four critique-guided configurations against a re-sampling baseline. The four guided setups—Self-Specific, Self-Hint, Teacher-Specific, and Teacher-Hint—are constructed by crossing two models (Self vs. Teacher) with two strategies (Task-specific vs. High-level hints). For each question, we first sampled four candidate responses from the policy model and evaluated their correctness. Focusing on the zero-pass subset, we apply our four guided configurations. Each configuration generates two independent critiques, each leading to four refined responses, totaling eight refinements. These are compared against a Re-Sampling baseline of eight direct samples without feedback.
为考察不同再探索策略如何影响模型表现,我们做了一项预实验,把四种批评引导配置与再采样基线加以比较。四种引导设置——Self-Specific、Self-Hint、Teacher-Specific 和 Teacher-Hint——由两种模型(自身对教师)与两种策略(任务具体对高层提示)交叉构成。对每一道题,我们先从策略模型采样四个候选回答,并评估其正确性。我们把注意力放在零通过子集上,并应用这四种引导配置。每种配置生成两条独立批评,每条批评再引出四个精炼回答,合计八次精炼。这些结果与不带反馈的八次直接采样的再采样基线相比较。
The evaluation metrics include SameRatio and Accuracy. SameRatio measures the proportion of the eight newly generated responses that exactly replicate the originally selected incorrect answer. A higher SameRatio indicates a stronger tendency to persist in the same error during re-exploration. Accuracy is defined as the overall correctness rate aggregated across both the initial responses and the newly generated responses.
评估指标包括 SameRatio 和准确率。SameRatio 衡量八个新生成回答中,与原先选定的错误答案完全相同的比例。更高的 SameRatio 表明再探索时更倾向于停留在同一错误上。准确率定义为在初始回答和新生成回答上汇总的总体正确率。
We conducted our pilot experiments using the Qwen2.5-7B-Instruct and Qwen3-4B models across three challenging mathematical benchmarks: MATH, AIME25, and OlympiadBench. We use Qwen2.5-72B-Instruct as the teacher model. For the Qwen2.5-7B-Instruct model, we set the temperature to 1 and top- to 0.95. For the Qwen3-4B model, the temperature was set to 0.7 and top- to 0.8.
我们用 Qwen2.5-7B-Instruct 和 Qwen3-4B 模型,在三个有挑战性的数学基准上做预实验:MATH、AIME25 和 OlympiadBench。教师模型采用 Qwen2.5-72B-Instruct。对 Qwen2.5-7B-Instruct,温度设为 1,top- 设为 0.95。对 Qwen3-4B,温度设为 0.7,top- 设为 0.8。
As illustrated in Figure 6, across multiple datasets and both evaluated models, the Specific Critique Guide yielded a high SameRatio and lower Accuracy than random sampling, even with prompts explicitly identifying errors. This validates the tendency for models to inadvertently replicate incorrect responses. Our approach effectively mitigates this issue, achieving a lower SameRatio and higher Accuracy, which in turn facilitates more efficient re-exploration.
如图 6 所示,在多个数据集和两个被评估模型上,具体批评引导都比随机采样得到更高的 SameRatio 和更低的准确率,即便提示已明确指出错误。这验证了模型会不经意复现错误回答的倾向。我们的方法有效缓解了这一问题,得到更低的 SameRatio 和更高的准确率,从而有助于更高效的再探索。
图 6:在 Qwen2.5-7B-Instruct 和 Qwen3-4B 模型上,比较不同再探索策略的 SameRatio 与准确率。
D Training Details
D 训练细节
D.1 Dataset Construction
D.1 数据构造
Mathematical Reasoning To construct a training set conducive to complex reasoning, we pre-process the DeepMath (He et al., 2025) dataset by rigorously filtering out binary classification problems (e.g., True/False questions). These instances are excluded because their simplistic answer space makes them highly susceptible to random guessing, which can yield spurious rewards that poorly reflect actual reasoning ability—thereby introducing noise into the RL signal. From the remaining pool, we randomly downsample 76,800 instances. Given a global training batch size of 128, this dataset size corresponds to exactly 600 training steps.
数学推理 为构造有利于复杂推理的训练集,我们预处理 DeepMath(He et al., 2025)数据集,严格过滤掉二元分类问题(例如正误判断题)。排除这些实例,是因为它们过于简单的答案空间很容易被随机猜测命中,从而产生不能很好反映真实推理能力的虚假奖励——并把噪声引入强化学习信号。我们从剩余集合中随机下采样 76,800 条。在全局训练批次大小为 128 时,这一数据规模恰好对应 600 个训练步。
Code Generation For the coding domain, we follow the setup in Wang et al. (2025b), utilizing their curated subset of CodeContests which contains approximately 4.5k training instances.
代码生成 对编程领域,我们遵循 Wang et al.(2025b)的设置,使用他们整理的 CodeContests 子集,其中约有 4,500 条训练实例。
D.2 Optimization and Hyperparameters
D.2 优化与超参数
Both CURE and the GRPO baseline are implemented using the verl framework and optimized via AdamW. For CURE, parameter updates are performed once per step using a unified batch that aggregates data from initial solutions, critiques, and guided solutions.
CURE 与 GRPO 基线都用 verl 框架实现,并用 AdamW 优化。对 CURE,每一步用一个统一批次做一次参数更新,该批次汇总来自初始解、批评和引导解的数据。
For mathematical tasks, we train for 600 steps with a global batch size of 128. For coding tasks, we train for a total of 300 steps with a global batch size of 32. All other optimization hyperparameters remain consistent with the math experiments. For the Qwen3-4B-Base model, we employ a brief warmup phase: we first train using standard GRPO for 50 steps to stabilize instruction following, and then use this checkpoint to initialize CURE training.
对数学任务,我们训练 600 步,全局批次大小为 128。对编程任务,一共训练 300 步,全局批次大小为 32。其余优化超参数与数学实验保持一致。对 Qwen3-4B-Base 模型,我们采用一段短暂的预热:先用标准 GRPO 训练 50 步以稳定指令跟随,再用该检查点初始化 CURE 训练。
Table 4 lists the common training hyperparameters shared by both methods. Additionally, Table 5 details the specific hyperparameters introduced by our CURE framework. All experiments were conducted on a server equipped with 4 NVIDIA A800 GPUs.
表 4 列出两种方法共用的训练超参数。此外,表 5 详列我们的 CURE 框架引入的特定超参数。全部实验在配备 4 块 NVIDIA A800 GPU 的服务器上完成。
Table 4: Detailed training hyperparameters for both CURE and the GRPO baseline.
| Hyperparameter | Value |
|---|---|
| Train Batch Size | 128 |
| Mini-Batch Size | 128 |
| Learning Rate (LR) | |
| KL Coefficient () | 0.0 |
| Rollout () | 8 |
| Temperature | 1.0 |
| Top- | 1.0 |
| Max Prompt Length | 6144 |
| Max Response Length | 4096 |
表 4:CURE 与 GRPO 基线共用的详细训练超参数。
| Hyperparameter | Value |
|---|---|
| Train Batch Size | 128 |
| Mini-Batch Size | 128 |
| Learning Rate (LR) | |
| KL Coefficient () | 0.0 |
| Rollout () | 8 |
| Temperature | 1.0 |
| Top- | 1.0 |
| Max Prompt Length | 6144 |
| Max Response Length | 4096 |
Table 5: CURE-specific hyperparameters.
| Hyperparameter | Value |
|---|---|
| Critique Batch Size | 256 |
| Rollout (Initial ) | 4 |
| Rollout (Guided ) | 4 |
| Rollout (Critique) | 1 |
| Format Penalty (, Missing Keys) | 0.5 |
| Format Penalty (, Length) | 0.25 |
表 5:CURE 特有的超参数。
| Hyperparameter | Value |
|---|---|
| Critique Batch Size | 256 |
| Rollout (Initial ) | 4 |
| Rollout (Guided ) | 4 |
| Rollout (Critique) | 1 |
| Format Penalty (, Missing Keys) | 0.5 |
| Format Penalty (, Length) | 0.25 |
D.3 Training Cost Analysis
D.3 训练成本分析
Compared with standard RLVR, CURE introduces additional training overhead due to its multi-stage generation process, which includes initial solution generation, critique generation, and critique-guided re-generation within each optimization step. To quantify this overhead, we report the wall-clock training cost measured on mathematical tasks using Qwen2.5-7B-Instruct on a server with 4 NVIDIA A800 GPUs.
与标准 RLVR 相比,CURE 因其多阶段生成过程而带来额外的训练开销:在每一步优化中都包括初始解生成、批评生成,以及批评引导的再生成。为量化这一开销,我们报告在配备 4 块 NVIDIA A800 GPU 的服务器上,用 Qwen2.5-7B-Instruct 在数学任务上测得的墙钟训练成本。
Table 7 compares the per-step cost of the GRPO baseline and CURE. Rollout Time denotes the total time spent on response generation within one training step. For CURE, this includes all three generation phases. Total Step Time denotes the end-to-end wall-clock time of one complete training iteration, including both rollout and optimization.
表 7 比较 GRPO 基线与 CURE 的每步成本。Rollout Time 表示一个训练步内花在回答生成上的总时间。对 CURE,这包括全部三个生成阶段。Total Step Time 表示一次完整训练迭代的端到端墙钟时间,包括 rollout 与优化。
Table 7: Training cost comparison per optimization step on mathematical tasks using Qwen2.5-7B-Instruct with 4 NVIDIA A800 GPUs.
| Metric | GRPO | CURE (Ours) |
|---|---|---|
| Rollout Time | 70.2s | 117.1s |
| Total Step Time | 191.0s | 267.6s |
表 7:在配备 4 块 NVIDIA A800 GPU 的环境下,用 Qwen2.5-7B-Instruct 在数学任务上比较每步优化成本。
| Metric | GRPO | CURE (Ours) |
|---|---|---|
| Rollout Time | 70.2s | 117.1s |
| Total Step Time | 191.0s | 267.6s |
As shown in Table 7, CURE increases rollout time relative to GRPO due to its richer training pipeline, while the increase in overall step time is more moderate. We note that this overhead can be alleviated in practice through pipelined execution across batches, where solution generation, critique generation, and guided re-generation are overlapped.
如表 7 所示,由于训练流程更丰富,CURE 相对 GRPO 增加了 rollout 时间,而整体步进时间的增加更为温和。我们指出,这一开销在实践中可以通过跨批次的流水执行来缓解,使解的生成、批评生成和引导再生成相互重叠。
E Evaluation Details
E 评估细节
During inference, we set the sampling temperature to 0.6 and the maximum generation length to 8192 tokens across all tasks. For the code generation benchmark LiveCodeBench, we utilize Version 2, which consists of 511 problems.
推理时,我们在全部任务上把采样温度设为 0.6,最大生成长度设为 8192 个 token。对代码生成基准 LiveCodeBench,我们使用第 2 版,其中包含 511 道题。
To evaluate the test-time self-improvement potential, we implement an iterative self-improvement loop with a maximum depth of rounds. The process proceeds as follows for each query:
Initialization: The model generates an initial solution using the standard sampling parameters.
Iterative Re-generation: In each round :
The model generates a critique for the current solution .
We parse the verification judgment from . If the judgment is “Error_Found: False” (i.e., predicted correct), the loop effectively pauses, and the current solution is carried forward as the result for all subsequent rounds up to .
If the judgment is “Error_Found: True” (i.e., predicted incorrect), the model utilizes the generated hint to produce a new guided solution .
- Performance Aggregation: For smaller benchmarks (AMC and AIME), we perform this scaling process in parallel over 32 independent trajectories and report the average accuracy (Avg@32) at each round . For larger benchmarks (MATH, Olympiad, and all code tasks), we report Avg@2.
为评估测试时自我改进的潜力,我们实现了一个最大深度为 轮的迭代自我改进回路。对每个查询,过程如下:
初始化:模型用标准采样参数生成初始解 。
迭代再生成:在每一轮 中:
模型为当前解 生成批评 。
我们从 中解析校验判断。若判断为 “Error_Found: False”(即预测为正确),回路实际上暂停,当前解 被沿用为直到 的所有后续轮次的结果。
若判断为 “Error_Found: True”(即预测为错误),模型利用生成的提示 产生新的引导解 。
- 表现汇总:对较小的基准(AMC 和 AIME),我们在 32 条独立轨迹上并行做这一扩展过程,并报告每一轮 的平均准确率(Avg@32)。对较大的基准(MATH、Olympiad 以及全部代码任务),我们报告 Avg@2。
F Additional Results
F 补充结果
F.1 Detailed Self-Verification Results
F.1 自我校验的详细结果
We evaluate the self-verification capability of the models using the F1 score. Specifically, the F1 score is computed as the harmonic mean of verification accuracy on self-generated correct and incorrect solutions. A higher F1 score indicates a stronger ability to distinguish correct answers from incorrect ones, which is a prerequisite for effective inference-time self-improvement.
我们用 F1 分数评估模型的自我校验能力。具体而言,F1 分数是模型在自行生成的正确解与错误解上校验准确率的调和平均。更高的 F1 分数表明更强的区分正确答案与错误答案的能力,而这是有效的推理时自我改进的前提。
Table 6 reports self-verification performance across all benchmarks. CURE consistently outperforms the GRPO baseline on both mathematical reasoning and code generation tasks. These results demonstrate that our unified training objective effectively grounds critique generation in factual correctness.
表 6 报告了全部基准上的自我校验表现。在数学推理和代码生成任务上,CURE 都始终优于 GRPO 基线。这些结果表明,我们的统一训练目标有效地把批评生成建立在事实正确性之上。
Table 6: Self-verification performance (F1 score, %) on mathematical reasoning and code generation benchmarks.
| Method | AIME24 | AIME25 | AMC | MATH | Olym. | Avg. | MBPP | LCB | CF | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | ||||||||||
| Base Model | 44.5 | 36.8 | 68.7 | 64.5 | 64.5 | 55.8 | 18.0 | 62.2 | 60.5 | 46.9 |
| GRPO | 53.3 | 38.3 | 73.0 | 76.3 | 73.6 | 62.9 | 16.3 | 64.3 | 47.0 | 42.5 |
| CURE (Ours) | 72.4 | 80.7 | 75.5 | 77.9 | 74.9 | 76.3 | 49.7 | 68.3 | 60.6 | 59.4 |
| Qwen3-4B-Base | ||||||||||
| Base Model | 22.9 | 23.9 | 25.5 | 25.9 | 24.8 | 24.6 | 20.0 | 18.6 | 17.8 | 18.8 |
| GRPO | 42.2 | 50.4 | 49.7 | 56.9 | 46.8 | 49.2 | 52.0 | 54.3 | 35.9 | 47.4 |
| CURE (Ours) | 67.9 | 77.3 | 71.4 | 71.5 | 71.3 | 71.9 | 51.5 | 68.9 | 55.6 | 58.7 |
表 6:数学推理与代码生成基准上的自我校验表现(F1 分数,%)。
| Method | AIME24 | AIME25 | AMC | MATH | Olym. | Avg. | MBPP | LCB | CF | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | ||||||||||
| Base Model | 44.5 | 36.8 | 68.7 | 64.5 | 64.5 | 55.8 | 18.0 | 62.2 | 60.5 | 46.9 |
| GRPO | 53.3 | 38.3 | 73.0 | 76.3 | 73.6 | 62.9 | 16.3 | 64.3 | 47.0 | 42.5 |
| CURE (Ours) | 72.4 | 80.7 | 75.5 | 77.9 | 74.9 | 76.3 | 49.7 | 68.3 | 60.6 | 59.4 |
| Qwen3-4B-Base | ||||||||||
| Base Model | 22.9 | 23.9 | 25.5 | 25.9 | 24.8 | 24.6 | 20.0 | 18.6 | 17.8 | 18.8 |
| GRPO | 42.2 | 50.4 | 49.7 | 56.9 | 46.8 | 49.2 | 52.0 | 54.3 | 35.9 | 47.4 |
| CURE (Ours) | 67.9 | 77.3 | 71.4 | 71.5 | 71.3 | 71.9 | 51.5 | 68.9 | 55.6 | 58.7 |
F.2 Analysis of Unconditional Scaling Strategy
F.2 无条件扩展策略分析
In the main text, we presented results using the conditional scaling strategy (Strategy A), where the model only re-generates a solution if it predicts an error. Here, we analyze an alternative Unconditional Re-generation strategy, where the model iteratively generates a new solution based on the critique at every round, regardless of the verification judgment.
在正文中,我们报告的是有条件扩展策略(策略 A)的结果,即模型仅在预测存在错误时才再生成解答。这里,我们分析另一种无条件再生成策略:模型在每一轮都根据批评迭代生成新解,而不论校验判断如何。
图 7:在 Qwen2.5-7B-Instruct 上,数学基准的无条件自我改进扩展结果。图中跟踪在没有校验过滤的情况下, 轮迭代中准确率的变化。
Figure 7 illustrates the scaling trajectories of this unconditional strategy on MATH500 and OlympiadBench using Qwen2.5-7B-Instruct. We observe two key phenomena:
- Initial Improvement: Similar to the conditional re-generation approach, the unconditional strategy achieves accuracy gains in the first 3-4 rounds. This confirms that the generated hints are intrinsically constructive and can guide the model towards better solutions even without explicit filtering.
- Instability in Later Stages: Unlike the conditional strategy which maintains a stable peak, the unconditional strategy exhibits performance oscillation after the initial rise. This instability arises because the model lacks a stopping mechanism, leading it to more frequently overwrite correct solutions with incorrect ones in later turns than the conditional strategy.
图 7 展示了这一无条件策略在 MATH500 和 OlympiadBench 上、用 Qwen2.5-7B-Instruct 得到的扩展轨迹。我们观察到两个关键现象:
- 初期提升:与有条件再生成方法类似,无条件策略在前 3–4 轮取得准确率收益。这确认,生成的提示本身具有建设性,即使没有显式过滤,也能引导模型走向更好的解。
- 后期不稳定:与保持稳定峰值的有条件策略不同,无条件策略在最初上升之后会出现表现振荡。这种不稳定源于模型缺少停止机制,从而使它在后续轮次中比有条件策略更频繁地用错误解覆盖正确解。
Comparing these strategies highlights the importance of both capabilities in CURE: while the hint provides the necessary gradient for upward improvement (Exploration), the verification capability acts as a crucial gatekeeper (Exploitation), locking in correct solutions to convert volatile gains into a stable trajectory.
比较这些策略凸显了 CURE 中两项能力的重要性:提示为向上改进提供必要的梯度(探索),而校验能力则充当关键的守门人(利用),把正确解锁定下来,从而把易变的收益转化为稳定的轨迹。
F.3 Additional Scaling Results
F.3 补充的扩展结果
Additionally, we provide the scaling trajectories for the Qwen3-4B-Base model in Figure 8. Consistent with the Qwen2.5 results, CURE demonstrates robust self-improvement on both MATH500 and OlympiadBench.
此外,我们在图 8 中给出 Qwen3-4B-Base 模型的扩展轨迹。与 Qwen2.5 的结果一致,CURE 在 MATH500 和 OlympiadBench 上都表现出稳健的自我改进。
图 8:Qwen3-4B-Base 上的推理时扩展结果。
F.4 Efficiency and Latency on OlympiadBench
F.4 OlympiadBench 上的效率与延迟
To complement the main-text analysis on MATH500, we further compare GRPO + Best-of- () and CURE + iterative self-improvement () on OlympiadBench. Table 8 reports the accuracy, average generated tokens per query, and latency per query. Similar to the MATH500 results, CURE achieves slightly better accuracy while using substantially fewer generated tokens. However, because OlympiadBench is more challenging and often requires more refinement turns before early stopping, the sequential iterative process leads to higher latency in this setting.
为补充正文中对 MATH500 的分析,我们进一步在 OlympiadBench 上比较 GRPO + Best-of-()与 CURE + 迭代自我改进()。表 8 报告准确率、每个查询的平均生成 token 数,以及每个查询的延迟。与 MATH500 的结果类似,CURE 在准确率略好的同时,使用的生成 token 明显更少。不过,由于 OlympiadBench 更难,并且在提前停止之前往往需要更多精炼轮次,顺序迭代过程在这一设置下会带来更高延迟。
Table 8: Efficiency and latency trade-off on OlympiadBench under the same maximum generation budget. Avg. Tokens denotes the total number of generated tokens per query. For BoN, we conservatively exclude the additional latency of the external reward model used for reranking.
| Method | Acc. (%) | Avg. Tokens | Latency (s) |
|---|---|---|---|
| GRPO + BoN () | 48.2 | 10678 | 0.66 |
| CURE + Iter. () | 48.7 | 4211 | 0.95 |
表 8:在相同最大生成预算下,OlympiadBench 上的效率与延迟权衡。Avg. Tokens 表示每个查询生成的 token 总数。对 BoN,我们保守地未计入用于重排的外部奖励模型的额外延迟。
| Method | Acc. (%) | Avg. Tokens | Latency (s) |
|---|---|---|---|
| GRPO + BoN () | 48.2 | 10678 | 0.66 |
| CURE + Iter. () | 48.7 | 4211 | 0.95 |
F.5 Results on an Additional Backbone Model
F.5 在额外主干模型上的结果
To further evaluate the robustness of CURE across model families, we additionally conduct experiments on OctoThinker-3B-Short-Base (Wang et al., 2025c), a LLaMA-based model that undergoes mid-training. The results are reported in Table 9.
为进一步评估 CURE 跨模型家族的稳健性,我们还在 OctoThinker-3B-Short-Base(Wang et al., 2025c)上做了实验,这是一个经过中间训练的、基于 LLaMA 的模型。结果见表 9。
Due to the relatively limited capacity of this 3B model, it produces too few correct solutions on the challenging AIME24 and AIME25 benchmarks for stable evaluation. We therefore focus on three representative benchmarks: MATH500, OlympiadBench, and AMC23.
由于这一 3B 模型容量相对有限,它在有挑战性的 AIME24 和 AIME25 基准上产生的正确解过少,难以做稳定评估。因此我们把重点放在三个有代表性的基准上:MATH500、OlympiadBench 和 AMC23。
Table 9: Results on OctoThinker-3B-Short-Base.
| Method | MATH500 | OlympiadBench | AMC23 | Avg. |
|---|---|---|---|---|
| Base Model | 3.5 | 1.2 | 1.1 | 1.9 |
| GRPO (Baseline) | 41.8 | 13.2 | 20.9 | 25.3 |
| + Self-Improve () | 42.2 | 12.3 | 21.1 | 25.2 |
| CURE (Ours) | 42.0 | 14.3 | 20.4 | 25.6 |
| + Self-Improve () | 43.6 | 14.8 | 21.0 | 26.5 |
表 9:OctoThinker-3B-Short-Base 上的结果。
| Method | MATH500 | OlympiadBench | AMC23 | Avg. |
|---|---|---|---|---|
| Base Model | 3.5 | 1.2 | 1.1 | 1.9 |
| GRPO (Baseline) | 41.8 | 13.2 | 20.9 | 25.3 |
| + Self-Improve () | 42.2 | 12.3 | 21.1 | 25.2 |
| CURE (Ours) | 42.0 | 14.3 | 20.4 | 25.6 |
| + Self-Improve () | 43.6 | 14.8 | 21.0 | 26.5 |
As shown in Table 9, CURE maintains competitive single-turn performance over the GRPO baseline and achieves stronger inference-time self-improvement under iterative scaling. In particular, with test-time self-improvement up to rounds, CURE improves the average accuracy from 25.6% to 26.5%, while the GRPO baseline exhibits almost no gain (25.3% to 25.2%). These results further support that the benefits of CURE generalize beyond the Qwen family to other backbone architectures.
如表 9 所示,CURE 相对 GRPO 基线保持有竞争力的单轮表现,并在迭代扩展下取得更强的推理时自我改进。具体而言,在最多 轮的测试时自我改进下,CURE 把平均准确率从 25.6% 提到 26.5%,而 GRPO 基线几乎没有收益(25.3% 到 25.2%)。这些结果进一步支持,CURE 的收益可以超越 Qwen 家族,推广到其他主干架构。
F.6 Impact of Experience Replay Strategy
F.6 经验回放策略的影响
We investigated two strategies for experience replay: (1) Selective Replay (CURE): Only replaying successful guided solutions from “zero-pass” groups (i.e., where the model initially failed completely). (2) Full Replay: Replaying all successful guided solutions, including those from groups that already contained correct initial solutions.
我们考察了两种经验回放策略:(1)选择性回放(CURE):只回放来自“零通过”组(也即模型起初完全失败的组)的成功引导解。(2)全量回放:回放全部成功的引导解,包括那些来自已经含有正确初始解的组。
As shown in Table 10, while Full Replay yields a marginal scaling improvement (41.8%), it significantly degrades the base single-turn reasoning capability (40.1%) below the GRPO baseline (41.0%). In contrast, Selective Replay achieves optimal performance on both metrics.
如表 10 所示,尽管全量回放带来轻微的扩展提升(41.8%),它却显著削弱了基础单轮推理能力(40.1%),并落到 GRPO 基线(41.0%)之下。相比之下,选择性回放在两项指标上都取得最优表现。
Table 10: Ablation study on Experience Replay Strategy (Average accuracy across 5 math benchmarks).
| Method | Single-turn Avg. | Scaling Avg. () |
|---|---|---|
| GRPO Baseline | 41.0 | 41.4 |
| Full Replay | 40.1 | 41.8 |
| Selective Replay (CURE) | 42.0 | 44.9 |
表 10:经验回放策略的消融实验(五个数学基准上的平均准确率)。
| Method | Single-turn Avg. | Scaling Avg. () |
|---|---|---|
| GRPO Baseline | 41.0 | 41.4 |
| Full Replay | 40.1 | 41.8 |
| Selective Replay (CURE) | 42.0 | 44.9 |
We attribute the degradation in Full Replay to excessive distribution shift. Since guided solutions are conditioned on hints (Query + Hint), they deviate from the standard inference distribution. Indiscriminately treating them as standard training samples introduces off-policy noise, destabilizing the optimization landscape. As visualized in Figure 9, the Full Replay strategy (Pink Line) exhibits a sudden spike in the actor’s gradient norm, indicating training instability. By restricting replay to “zero-pass” groups, CURE injects high-value learning signals only where they are strictly necessary, preserving stability while breaking exploration stagnation.
我们把全量回放的退化归因于过度的分布偏移。由于引导解以提示为条件(查询 + 提示),它们偏离了标准推理分布。不加区分地把它们当作标准训练样本,会引入离策略噪声,使优化景观变得不稳定。如图 9 所示,全量回放策略(粉线)在策略梯度范数上出现突然尖峰,表明训练不稳定。通过把回放限制在“零通过”组,CURE 只在真正必要之处注入高价值学习信号,从而在打破探索停滞的同时保持稳定。
图 9:梯度范数动态。
G Licenses
G 许可
Datasets and Benchmarks. The training dataset is derived from DeepMath (MIT License) and CodeForces (CC-BY-4.0). We evaluate on: MATH 500 (MIT License), AIME 2024/2025 (CC0: Public Domain), AMC 2023 (Apache License 2.0), Olympiad Bench (MIT License), CodeForces (CC-BY-4.0), MBPP (CC-BY-4.0), and LiveCodeBench (CC License).
数据集与基准。 训练数据来自 DeepMath(MIT License)和 CodeForces(CC-BY-4.0)。我们评估所用的是:MATH 500(MIT License)、AIME 2024/2025(CC0:公共领域)、AMC 2023(Apache License 2.0)、Olympiad Bench(MIT License)、CodeForces(CC-BY-4.0)、MBPP(CC-BY-4.0)以及 LiveCodeBench(CC License)。
Framework. RL training is based on verl v0.5 (Apache-2.0 license). Evaluation is performed using vllm framework (Kwon et al., 2023) v0.8.5 (Apache-2.0 License).
框架。 强化学习训练基于 verl v0.5(Apache-2.0 license)。评估使用 vllm 框架(Kwon et al., 2023)v0.8.5(Apache-2.0 License)。
H Qualitative Comparison of Hint Quality
H 提示质量的定性比较
Figure 10 presents a qualitative comparison between CURE and the GRPO baseline on a number theory problem. The initial solution correctly identifies the smallest solution (1, 2, 3) but fails to consider permutations. The GRPO baseline model, lacking explicit critique training, generates a hint that is technically sounding but semantically irrelevant to the actual error. Consequently, the guided re-generation fails to break out of the initial incorrect path, repeating the same incomplete answer. In contrast, CURE generates a strategic hint that directly challenges the flaw in the reasoning logic. Crucially, this hint does not leak the answer but instead prompts the model to reconsider its assumptions. It successfully triggers a new line of reasoning (“Symmetry and Permutations”), enabling the model to discover the complete set of solutions. This case exemplifies how CURE’s critique capability acts as an effective scaffold for escaping local optima.
图 10 在一道数论问题上,对 CURE 与 GRPO 基线做了定性比较。最初的解正确找出了最小解 (1, 2, 3),但没有考虑排列。缺少显式批评训练的 GRPO 基线模型,生成了一条听起来技术正确、但在语义上与实际错误无关的提示。结果,引导再生成未能跳出最初的错误路径,重复了同一个不完整的答案。相比之下,CURE 生成了一条直接挑战推理逻辑缺陷的策略性提示。关键的是,这条提示并不泄露答案,而是促使模型重新考虑其假设。它成功触发了一条新的推理路线(“对称性与排列”),使模型发现完整的解集。这一案例表明,CURE 的批评能力可以作为逃离局部最优的有效支架。
图 10:所生成提示的定性比较。









