Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms

可验证隐藏动力学对局：从已解机制生成智能体强化学习环境

Xinjie Shen Affiliation:  Georgia Institute of Technology    Wei Fan Affiliation:  Alibaba Token Foundry, Alibaba Group    Xudong Guo Affiliation:  Alibaba Token Foundry, Alibaba Group    Jianhong Tu Affiliation:  Alibaba Token Foundry, Alibaba Group    Yang Su Affiliation:  Alibaba Token Foundry, Alibaba Group    Chuqiao Kuang Affiliation:  Alibaba Token Foundry, Alibaba Group    Yinger Zhang Affiliation:  Alibaba Token Foundry, Alibaba Group    Lianghao Deng Affiliation:  Alibaba Token Foundry, Alibaba Group    Dayiheng Liu Affiliation:  Alibaba Token Foundry, Alibaba Group

Abstract

摘要

Language-model agents increasingly face long-horizon tasks with evolving state, interdependent decisions, and delayed outcomes. Scaling their training requires diverse agentic environments, dependable outcome signals, and low extension cost. Existing generation pipelines commonly construct an environment before defining its outcome rule or annotating its trajectories, leaving dynamics and evaluation to be aligned post hoc. VHD-Play reverses this dependency by sampling and solving a mathematical model before a corpus-grounded setter renders its decision process as stateful tools. The executable dynamics and trajectory-scoring reference are inherited from the same solved model. The pipeline produces 3,300 diverse agentic environments at a cost of a few cents each. Training Qwen3.6-35B-A3B on three families raises its mean agentic score from 0.204 to 0.815 in a five-family diagnostic. Gains also appear on held-out instances from all three training families and eight unseen mechanism families, then extend beyond the generated substrate to external benchmarks for general function calling, travel planning, and 365-day e-commerce. On E-Commerce Bench, the trained checkpoint completes every run without bankruptcy and exceeds Qwen3.7-Max. We compare written-out problems with stateful versions that reveal or hide their parameters. The comparison shows that most of the learnable gap lies in stateful interaction rather than underlying problem solving. A frozen 35B setter realizes larger environments, and scale-matched training retains gains as mechanism size and horizon grow, indicating the potential for an evolving training substrate.

语言模型智能体日益面对状态演化、决策相互依赖、结果延后的长程任务。扩展其训练需要多样的智能体环境、可靠的结果信号，以及低扩展成本。现有生成流水线通常先构造环境，再定义结果规则或标注轨迹，把动力学与评测留到事后对齐。VHD-Play 反转这一依赖：先抽样并求解数学模型，再由语料锚定的设定器（setter）将其决策过程渲染为有状态工具。可执行动力学与轨迹评分参考均继承自同一已解模型。流水线以每个约几美分的成本产出 3,300 个多样智能体环境。在三个族上训练 Qwen3.6-35B-A3B，使其在五族诊断中的平均智能体分数从 0.204 提升到 0.815。增益也出现在全部三个训练族的持出实例以及八个未见机制族上，并延伸到生成基底之外的通用函数调用、旅行规划与 365 天电商等外部基准。在 E-Commerce Bench 上，训练后检查点每次运行都完成且未破产，并超过 Qwen3.7-Max。我们比较纸面写出的问题与揭示或隐藏参数的有状态版本。比较表明，可学习差距的大部分在有状态交互，而非底层解题能力。冻结的 35B 设定器能实现更大规模环境，规模匹配的训练在机制规模与视野增长时仍保持增益，显示可演化训练基底的潜力。

1 Introduction

1 引言

As language-model agents reach broader deployment, they are increasingly expected to complete diverse, end-to-end workflows rather than answer isolated requests. Emerging applications ask them to repair software through repeated iterations (Orlanski et al., 2026), resolve customer requests across tools (Yao et al., 2024), and operate simulated businesses over extended periods (Backlund and Petersson, 2025). Such workflows place the model inside a process whose state evolves as it acts (He et al., 2026). Information gathered through one tool call can determine a later decision, while an earlier edit, purchase, or commitment can change the options that remain  (Wang et al., 2026b). These workflows therefore extend beyond fully specified question answering to agentic, interactive tasks. Fig. 1(a) shows a marked performance gap between the written-out and agentic forms, while broader evaluations find that reliability declines as the expert time and interaction horizon required by these workflows grow (Kwa et al., 2025; Rabanser et al., 2026).

随着语言模型智能体部署范围扩大，人们越来越期望它们完成多样的端到端工作流，而不是回答孤立请求。新兴应用要求它们通过反复迭代修复软件（Orlanski et al., 2026）、跨工具处理客户请求（Yao et al., 2024），以及在较长时段内运营模拟业务（Backlund and Petersson, 2025）。这类工作流把模型置于随其行动而演化的过程之中（He et al., 2026）。一次工具调用收集到的信息可以决定后续决策，而更早的编辑、采购或承诺会改变仍可选择的选项（Wang et al., 2026b）。因此这些工作流从完全给定的问答，扩展到智能体式、交互式任务。图 1(a) 显示纸面形式与智能体形式之间存在明显性能差距；更广的评测也发现，随着这些工作流所需的专家时间与交互视野增长，可靠性会下降（Kwa et al., 2025; Rabanser et al., 2026）。

Figure 1: (a) The same optimization problem appears as written-out QA or embedded in a stateful agentic environment, but performance does not transfer automatically. (b) Agentic training improves held-out instances from trained families, unseen optimization families (near-OOD), and families with different decision or observation structures (far-OOD). DP, LP, and QP denote dynamic, linear, and quadratic programming.

图 1：（a）同一优化问题既可以写成纸面问答，也可以嵌入有状态智能体环境，但性能不会自动迁移。（b）智能体训练提升训练族持出实例、未见优化族（近 OOD），以及决策或观测结构不同的族（远 OOD）。DP、LP、QP 分别表示动态规划、线性规划与二次规划。

Meeting this expectation creates a joint environment–signal construction problem. Every training or evaluation instance needs an executable environment and a dependable signal tied to the outcome produced within it. Written-out mathematics and reasoning tasks  (DeepSeek-AI et al., 2025; Hu et al., 2025; Stojanovski et al., 2025), as well as search tasks with answer-based rewards (Jin et al., 2025), provide cheap, exact signals but present the relevant information in the prompt rather than require the model to recover it while acting in a changing state. Hand-engineered simulators  (Hubbs et al., 2020) restore that interaction and can expose exact outcomes, but specialists must implement the dynamics and evaluator for each new family (Zeng et al., 2026). Human-feedback methods  (Christiano et al., 2017; Ouyang et al., 2022) reach less structured workflows at the cost of repeated expert annotation. Learned judges and rubrics (Gunjal et al., 2025; Lyu et al., 2026) reduce that burden but introduce an evaluator whose validity must itself be established (Norman et al., 2026). Systems now generate tools and interactive environments (Cai et al., 2025; Song et al., 2026; Tu et al., 2026; Wang et al., 2026c), sometimes together with executable checks (Gao et al., 2026), broadening task coverage and reducing authoring work. Together, these approaches address only different parts of producing diverse agentic environments with dependable outcome signals at low extension cost. Yet generative pipelines commonly construct the task or environment before fixing its reward assignment or evaluation rule, leaving dynamics and evaluation to be aligned afterward.

要满足这一期望，就形成了环境与信号的联合构造问题。每个训练或评测实例都需要可执行环境，以及与其中产生的结果绑定的可靠信号。纸面数学与推理任务（DeepSeek-AI et al., 2025; Hu et al., 2025; Stojanovski et al., 2025），以及带答案奖励的搜索任务（Jin et al., 2025），能提供廉价且精确的信号，但相关信息直接出现在提示中，并不要求模型在变化状态中行动时再去恢复。手工仿真器（Hubbs et al., 2020）恢复了这种交互，也能暴露精确结果，但每个新族都需要专家实现动力学与评测器（Zeng et al., 2026）。人类反馈方法（Christiano et al., 2017; Ouyang et al., 2022）能覆盖更少结构化的工作流，代价是反复专家标注。学习型评判与量规（Gunjal et al., 2025; Lyu et al., 2026）减轻了负担，却引入了自身有效性仍待确立的评测器（Norman et al., 2026）。现有系统会生成工具与交互环境（Cai et al., 2025; Song et al., 2026; Tu et al., 2026; Wang et al., 2026c），有时还带可执行检查（Gao et al., 2026），拓宽了任务覆盖并降低了编写成本。合在一起，这些方法只分别解决了「以低扩展成本产出多样智能体环境与可靠结果信号」的不同侧面。而生成式流水线通常仍先构造任务或环境，再固定奖励分配或评测规则，把动力学与评测留到事后对齐。

VHD-Play reverses this order. We first sample and solve a mathematical problem before generating the environment. A frozen setter then uses a sample from a diverse corpus of real-world documents to seed the scenario and implements the problem’s decision process through stateful tools. The sampled problem governs how the environment evolves, while its solution supplies the outcome signal. The player sees neither the parameters nor the solution and must recover the relevant information through interaction. Established mathematical model families, such as optimization, offer mature solvers and a ready source of tasks, while new parameter draws produce additional environments within each family at low marginal cost. Fig. 2 illustrates the construction order.

VHD-Play 反转了这一顺序。我们先抽样并求解数学问题，再生成环境。冻结的设定器从多样真实世界文档语料中抽样，为场景赋种子，并通过有状态工具实现该问题的决策过程。抽样问题支配环境如何演化，其解则提供结果信号。玩家既看不到参数也看不到解，必须通过交互恢复相关信息。优化等成熟数学模型族提供了现成求解器与任务来源，而在每个族内重新抽样参数，就能以很低的边际成本再产出环境。图 2 展示了这一构造顺序。

Figure 2: Construction order. Environment-first pipelines construct the agent-facing environment $E$ before defining its outcome rule or annotating trajectories. VHD-Play first solves a sampled mathematical model $M(\theta)$, whose reference outcomes $z_{\theta}$ fix $\mathcal{R}_{\theta}$ before the executable dynamics $D(\theta)$ are realized and wrapped as environment $E(\theta)$.

图 2：构造顺序。环境优先流水线先构造面向智能体的环境 $E$，再定义结果规则或标注轨迹。VHD-Play 先求解抽样数学模型 $M(\theta)$，由其参考结果 $z_{\theta}$ 固定 $\mathcal{R}_{\theta}$，再实现可执行动力学 $D(\theta)$ 并包装为环境 $E(\theta)$。

The paper makes three contributions. (1) We introduce mechanism-first construction, deriving an agentic environment and its outcome signal from the same pre-solved problem. (2) We realize it as a corpus-grounded generation and replay pipeline with information-asymmetric interfaces and automatic admission, producing 3,300 admitted environments at low extension cost. (3) Using these environments to train a Qwen3.6-35B-A3B model raises its mean agentic score across five optimization families from $0.204$ to $0.815$. The checkpoint improves on all three held-out training and all eight unseen families. Performance remains strong as the same construction scales tasks and horizons, pointing toward an evolving training substrate. Externally, it reaches $3.4\times$ the base ending balance and surpasses Qwen3.7-Max on a 365-day storefront (Fan et al., 2026), improves ten interaction-focused BFCL V4 cells by $2.84$ points (Patil et al., 2025), and preserves written-out problem solving.

本文有三项贡献。（1）提出机制优先构造：从同一预解问题导出智能体环境及其结果信号。（2）将其实现为语料锚定的生成与回放流水线，配合信息不对称接口与自动准入，以低扩展成本产出 3,300 个准入环境。（3）用这些环境训练 Qwen3.6-35B-A3B，使五个优化族上的平均智能体分数从 $0.204$ 提升到 $0.815$。检查点在全部三个持出训练族与全部八个未见族上均有提升；同一构造在任务与视野放大时表现仍强，指向可演化的训练基底。外部方面，它在 365 天店面任务上达到基线期末余额的 $3.4\times$ 并超过 Qwen3.7-Max（Fan et al., 2026），在十个交互向 BFCL V4 单元上提升 $2.84$ 分（Patil et al., 2025），并保持纸面解题能力。

2 Related Work

2 相关工作

Formal and verifiable reasoning. Written-out mathematics QA pairs each problem with inexpensive checks (DeepSeek-AI et al., 2025; Stojanovski et al., 2025), and search agents can likewise receive answer-based rewards (Jin et al., 2025). These settings score final responses without requiring agents to alter and operate within a changing task state. SATLM and LLM+P improve reliability by translating problems into logic or planning languages and delegating inference to a solver (Ye et al., 2023; Liu et al., 2023), but still target an answer or plan rather than an agentic training environment. Formal or human-designed frameworks instead provide exact dynamics through text games (Côté et al., 2018), PDDL environments (Silver and Chitnis, 2020), or language-rendered planning domains (Stein et al., 2023). Procedural generation varies instances under fixed rules  (Cobbe et al., 2020; Hubbs et al., 2020). Thus verifiability either stops at an answer or plan or retains a domain specification. VHD-Play keeps the computable reference while using sampled mechanisms and corpus grounding to produce diverse agentic instances.

形式化与可验证推理。 纸面数学问答为每个问题配上廉价检查（DeepSeek-AI et al., 2025; Stojanovski et al., 2025），搜索智能体同样可接受基于答案的奖励（Jin et al., 2025）。这些设定只对最终响应打分，不要求智能体在变化的任务状态中改写并操作。SATLM 与 LLM+P 通过把问题译成逻辑或规划语言并交给求解器推理来提高可靠性（Ye et al., 2023; Liu et al., 2023），但目标仍是答案或计划，而非智能体训练环境。形式化或人工设计的框架则通过文本游戏（Côté et al., 2018）、PDDL 环境（Silver and Chitnis, 2020）或语言渲染的规划域（Stein et al., 2023）提供精确动力学。过程化生成在固定规则下变化实例（Cobbe et al., 2020; Hubbs et al., 2020）。于是可验证性要么停在答案或计划，要么仍保留领域规范。VHD-Play 保留可计算参考，同时用抽样机制与语料锚定产出多样智能体实例。

Generated agentic environments. As target workflows grow more varied and longer, this per-domain specification cost becomes harder to sustain. Recent systems reduce it by synthesizing tools and task environments (Cai et al., 2025; Song et al., 2026; Tu et al., 2026), or by generating world models and simulators  (Wang et al., 2026c; Lyu et al., 2026; Wang et al., 2025). Across agent training, feedback comes from process or turn-level rewards (Liu et al., 2025; Tao et al., 2026; Chae et al., 2025), learned reward models  (Christiano et al., 2017; Ouyang et al., 2022), generated rubrics (Gunjal et al., 2025), and executable verifiers (Gao et al., 2026; Zeng et al., 2026). Generation can therefore broaden agency and semantic diversity at lower authoring cost. Yet learned or generated graders require validation (Norman et al., 2026), and agreement between generated dynamics and evaluation is commonly established afterward (Zhang et al., 2025). Work on partial observability and information gathering (Kaelbling et al., 1998; Zhou et al., 2025; Huang et al., 2025) and long-horizon execution (Sinha et al., 2026; Wang et al., 2026a) sharpens the behavioral target but often assumes fixed environments. The two lines therefore cover complementary parts: formal methods secure outcomes but retain domain authoring, whereas generation reduces authoring but reopens outcome grounding. VHD-Play bridges them by pre-solving each sampled mechanism and retaining its reference for hidden dynamics and graded scoring.

生成式智能体环境。 目标工作流越来越多样、越来越长，按域编写规范的成本越来越难维持。近期系统通过合成工具与任务环境（Cai et al., 2025; Song et al., 2026; Tu et al., 2026），或生成世界模型与仿真器（Wang et al., 2026c; Lyu et al., 2026; Wang et al., 2025）来降低成本。智能体训练中的反馈来自过程或回合级奖励（Liu et al., 2025; Tao et al., 2026; Chae et al., 2025）、学习型奖励模型（Christiano et al., 2017; Ouyang et al., 2022）、生成量规（Gunjal et al., 2025）以及可执行验证器（Gao et al., 2026; Zeng et al., 2026）。因此生成能以更低编写成本拓宽能动性与语义多样性。但学习型或生成的打分器需要验证（Norman et al., 2026），生成动力学与评测之间的一致也常事后建立（Zhang et al., 2025）。部分可观测与信息收集（Kaelbling et al., 1998; Zhou et al., 2025; Huang et al., 2025）以及长程执行（Sinha et al., 2026; Wang et al., 2026a）的工作锐化了行为目标，却往往假定环境固定。两条线索因而互补：形式方法保住结果但仍需领域编写，生成降低编写却重新打开结果锚定。VHD-Play 通过预解每个抽样机制并保留其参考，用于隐藏动力学与分级评分，从而桥接二者。

3 Formulation

3 形式化

Our goal is to construct diverse agentic environments together with dependable outcome signals, without repeating the full authoring effort for every instance. The two artifacts form a joint problem because an environment determines which trajectories can occur, while its evaluator determines what those trajectories are worth. Producing them as separate artifacts leaves their agreement to be established after generation.

我们的目标是构造多样的智能体环境，并配上可靠的结果信号，而不必为每个实例重复全部编写工作。这两类产物构成联合问题：环境决定哪些轨迹可能发生，评测器则决定这些轨迹值多少。若把它们当作分开的产物来造，一致性只能留到生成之后再确立。

An agentic environment is a process rather than a prompt. Observations depend on state, actions change that state, and their consequences unfold along a trajectory. We use dynamics broadly for the relationships among states, actions, observations, and outcomes. In a real environment, these dynamics may be unknown and need not admit an explicit mathematical form. An effective policy nevertheless needs some useful internal proxy for them, even if it never recovers a set of equations. Operations research often abstracts a concrete decision problem and its operating conditions into a mathematical model for analysis and policy design  (Hubbs et al., 2020). We take a similar view of agentic environments, treating their behavior as governed by an underlying model. To construct an environment together with a verifiable outcome signal, VHD-Play reverses the usual direction. We begin with a mathematical model that usually comes with a verifiable reference solution, then render its state transitions, information structure, constraints, and objective as a new environment. The model thereby becomes the common source of the interactive process and its evaluation.

智能体环境是一个过程，而不是一条提示。观测依赖状态，动作改变状态，后果沿轨迹展开。我们广义地用动力学指状态、动作、观测与结果之间的关系。在真实环境中，这些动力学可能未知，也不必有显式数学形式。有效策略仍需要某种有用的内部代理，即便从不恢复方程组。运筹学常把具体决策问题及其运行条件抽象成数学模型，用于分析与策略设计（Hubbs et al., 2020）。我们对智能体环境取类似视角，将其行为视为由底层模型支配。为同时构造环境与可验证结果信号，VHD-Play 反转通常方向：从通常带有可验证参考解的数学模型出发，再将其状态转移、信息结构、约束与目标渲染为新环境。模型因此成为交互过程及其评测的共同来源。

3.1 Construction order

3.1 构造顺序

Once an environment $E$ and its outcome rule $\mathcal{R}_{E}$ are fixed, policy learning is conceptually straightforward. A policy $\pi$ acts in $E$ to produce a trajectory $\tau_{\pi}$, and $\mathcal{R}_{E}$ maps the realized outcome to a signal for improving $\pi$. This describes how an environment is used for learning. Policy optimization itself takes its construction as given. For an existing environment, supervision can be added either by defining $\mathcal{R}_{E}$ or by assigning an annotation $y_{\pi}$ to a sampled trajectory $\tau_{\pi}$. When the environment itself is generated, it is still commonly constructed before either form of supervision. We call this order environment-first. The interactive process $E$, including whatever transition structure governs it, is fixed before an outcome rule is defined or trajectories from it are annotated. This construction need not isolate the transition structure as a separate object, even when that structure is known.

一旦环境 $E$ 及其结果规则 $\mathcal{R}_{E}$ 固定，策略学习在概念上就直截了当。策略 $\pi$ 在 $E$ 中行动产生轨迹 $\tau_{\pi}$，$\mathcal{R}_{E}$ 把实现结果映射为改进 $\pi$ 的信号。这描述的是环境如何被用于学习；策略优化本身把环境构造当作给定。对已有环境，监督既可以定义 $\mathcal{R}_{E}$，也可以给抽样轨迹 $\tau_{\pi}$ 赋标注 $y_{\pi}$。当环境本身被生成时，仍常在两种监督之前先构造环境。我们称此顺序为环境优先。交互过程 $E$（含支配它的任何转移结构）在定义结果规则或标注其轨迹之前就已固定。即便转移结构已知，这种构造也不必把它隔离成单独对象。

VHD-Play instead makes the upstream mechanism explicit. Let $\theta$ denote sampled parameters and $M(\theta)$ the corresponding mathematical model. Solving the model yields reference outcomes $z_{\theta}$ and fixes the outcome rule $\mathcal{R}_{\theta}(\,\cdot\,;z_{\theta})$. We write $D(\theta)$ for the model’s executable realization, which governs state transitions and utility, and $E(\theta)$ for the agent-facing environment obtained by wrapping $D(\theta)$ with an interaction interface. For a policy $\pi$, $\tau_{\pi}$ is the resulting trajectory and $y_{\pi}$ its outcome signal. The arrows below record which artifact is available when the next is constructed.

VHD-Play 则把上游机制显式化。令 $\theta$ 表示抽样参数，$M(\theta)$ 为对应数学模型。求解模型得到参考结果 $z_{\theta}$，并固定结果规则 $\mathcal{R}_{\theta}(\,\cdot\,;z_{\theta})$。我们写 $D(\theta)$ 表示模型的可执行实现（支配状态转移与效用），$E(\theta)$ 表示用交互接口包装 $D(\theta)$ 后得到的面向智能体环境。对策略 $\pi$，$\tau_{\pi}$ 为所得轨迹，$y_{\pi}$ 为其结果信号。下方箭头记录构造下一产物时哪些产物已可用。

$$\displaystyle\text{Environment-first} \\ \displaystyle E\xrightarrow{\ \mathrm{define}\ }\mathcal{R}_{E}\quad\text{or}\quad(E,\pi)\xrightarrow{\ \mathrm{interact}\ }\tau_{\pi}\xrightarrow{\ \mathrm{annotate}\ }y_{\pi}, \\ \displaystyle\text{VHD-Play} \\ \displaystyle M(\theta)\xrightarrow{\ \mathrm{solve}\ }z_{\theta}\xrightarrow{\ \mathrm{fix}\ }\mathcal{R}_{\theta}(\,\cdot\,;z_{\theta}), \\ \displaystyle M(\theta)\xrightarrow{\ \mathrm{realize}\ }D(\theta)\xrightarrow{\ \mathrm{wrap}\ }E(\theta), \\ \displaystyle\tau_{\pi}\sim E(\theta),\qquad y_{\pi}=\mathcal{R}_{\theta}\!\left(\tau_{\pi};z_{\theta}\right).$$

Environment-first construction may contain rich or even known dynamics. The distinction is that they are already packaged in $E$ when supervision is supplied. In VHD-Play, both the executable process and its evaluation descend from $M(\theta)$. Solving first fixes $z_{\theta}$ and the corresponding outcome rule before realization supplies the stateful dynamics and wrapping exposes them to a policy. During training, $E(\theta)$ generates $\tau_{\pi}$, while the already fixed $\mathcal{R}_{\theta}$ evaluates it against $z_{\theta}$. Fig. 2 illustrates this reversal, and App. E.1 places common methods within the same construction view. We next unpack how the construction inherits the useful properties of the solved mechanism.

环境优先构造也可以包含丰富甚至已知的动力学。区别在于：提供监督时，它们已经封装进 $E$。在 VHD-Play 中，可执行过程及其评测都来自 $M(\theta)$。先求解固定 $z_{\theta}$ 与对应结果规则，再由实现提供有状态动力学，并由包装暴露给策略。训练时 $E(\theta)$ 生成 $\tau_{\pi}$，而已固定的 $\mathcal{R}_{\theta}$ 对照 $z_{\theta}$ 评测它。图 2 展示了这一反转，附录 E.1 把常见方法放进同一构造视角。接下来我们拆解构造如何继承已解机制的有用性质。

3.2 Model, dynamics, and interface

3.2 模型、动力学与接口

The mechanism-first construction separates the mathematical model, its executable dynamics, and the interface exposed to the policy. A mechanism family $f$ defines a distribution $P_{f}$ over multi-period decision problems, and a draw $\theta\sim P_{f}$ fixes one instance. The mathematical model $M(\theta)$ specifies its decisions, constraints, objective, and feasible policies. Its executable dynamics $D(\theta)$ contain a state space $\mathcal{S}$, a transition $\Lambda_{\theta}$, and a utility function $U_{\theta}$. The environment $E(\theta)$ wraps these dynamics with an observation map $\Omega_{\theta}$, an action set $\mathcal{A}$, and a budget of $T$ turns. Together these objects define the feedback process below, while Fig. 3(a) places the $M(\theta)\!\to\!D(\theta)\!\to\!E(\theta)$ realization beside the interaction and evaluation paths from the same model.

机制优先构造把数学模型、其可执行动力学，以及暴露给策略的接口分开。机制族 $f$ 定义多期决策问题上的分布 $P_{f}$，一次抽样 $\theta\sim P_{f}$ 固定一个实例。数学模型 $M(\theta)$ 规定决策、约束、目标与可行策略。其可执行动力学 $D(\theta)$ 含状态空间 $\mathcal{S}$、转移 $\Lambda_{\theta}$ 与效用函数 $U_{\theta}$。环境 $E(\theta)$ 用观测映射 $\Omega_{\theta}$、动作集 $\mathcal{A}$ 与 $T$ 回合预算包装这些动力学。这些对象共同定义下方反馈过程；图 3(a) 把 $M(\theta)\!\to\!D(\theta)\!\to\!E(\theta)$ 实现与同一模型的交互、评测路径并置。

$$\begin{gathered}D(\theta)=(\mathcal{S},\Lambda_{\theta},U_{\theta}),\qquad E(\theta)=\operatorname{Wrap}!\big(D(\theta);\Omega_{\theta},\mathcal{A},T\big),\[-1.0pt]
a_{t}=\pi(o_{\leq t}),\qquad s_{t+1}=\Lambda_{\theta}(s_{t},a_{t}),\qquad o_{t+1}=\Omega_{\theta}(s_{t+1},a_{t}),\[-1.0pt]
u(\pi;\theta)=U_{\theta}(\tau_{\pi}),\qquad 0\leq t<T.\end{gathered} \tag{2}$$

Figure 3: Mechanism-first construction from one solved instance to a training substrate. (a) The model fixes dynamics and evaluation while the interface controls the policy view. (b) Corpus-grounded draws undergo execution and supported reference checks before training.

图 3：从单个已解实例到训练基底的机制优先构造。（a）模型固定动力学与评测，接口控制策略视角。（b）语料锚定的抽样在训练前经过执行与支持的参考检查。

A turn is one answered interface call. A notable feature of agentic environments is that a policy may need to explore the environment before it can act effectively. Under partial observability and over long horizons, an agent may spend turns acquiring task-relevant information before making commitments whose effects persist. Let $\mathcal{A}_{\mathrm{probe}}\subseteq\mathcal{A}$ denote these information-acquisition actions. A probe returns a designated view of hidden state or parameters, whereas a decision action changes state through $\Lambda_{\theta}$. Both consume the shared budget $T$. Probes are not required in every agentic environment, but instantiate the recurring coupling between gathering information and acting on it. The interface may also provide state-independent computation, which neither reads nor changes task state. Withholding parameters yields a partially observed process whose latent state includes the draw (Kaelbling et al., 1998). The policy need not reconstruct $M(\theta)$ symbolically, but effective action requires a useful proxy for the exposed dynamics.

一个回合是一次得到应答的接口调用。智能体环境的一个显著特征是：策略可能需要先探索环境，才能有效行动。在部分可观测与长视野下，智能体可能先花回合获取任务相关信息，再做出效果会持续的承诺。令 $\mathcal{A}_{\mathrm{probe}}\subseteq\mathcal{A}$ 表示这些信息获取动作。探询返回隐藏状态或参数的指定视图，而决策动作通过 $\Lambda_{\theta}$ 改变状态。二者共享预算 $T$。并非每个智能体环境都需要探询，但它体现了收集信息与据此行动之间的反复耦合。接口也可提供与状态无关的计算，既不读也不改任务状态。扣留参数得到潜在状态包含该抽样的部分可观测过程（Kaelbling et al., 1998）。策略不必符号化重建 $M(\theta)$，但有效行动需要暴露动力学的有用代理。

The builder $G$ receives the complete draw and the corpus seed, whereas the policy receives only observations published through $E(\theta)$:

构建器 $G$ 收到完整抽样与语料种子，而策略只收到经 $E(\theta)$ 发布的观测：

$$G\colon(\theta,c)\ \mapsto\ \big(D(\theta),E(\theta)\big)\qquad\text{whereas}\qquad\pi\colon o_{\leq t}\ \mapsto\ a_{t}. \tag{3}$$

Since $M(\theta)$ and $z_{\theta}$ already fix the governing structure and evaluation, $G$ is asked to realize and wrap them rather than invent them jointly. This narrower role reduces the capacity required of the setter. Sec. 4 gives the realization, and App. C.3 shows that the starting Qwen3.6-35B-A3B checkpoint, which is later optimized as the player, can already act as the setter and produce valid agentic environments. A stronger setter primarily improves yield and interface refinement. The asymmetry lies in what the interface publishes, not in whether the environment depends on $\theta$. Because $\Omega_{\theta}$ and $\mathcal{A}$ belong to the wrapper, they provide an explicit control point over which components of the draw appear initially, which require interaction to reveal, and which remain latent. In the agentic form, the default interface offers no direct read of the complete draw or either reference value, so the policy can obtain only the views returned by its calls. Fig. 1(a) shows the resulting agentic difficulty. Models perform strongly on the underlying problems in written-out form but substantially worse in their stateful agentic form. Because the model and interface are separate, the same draw can be presented in written-out form without changing the underlying problem or its evaluation. Resampling $\theta$ and varying its range or horizon produces further instances, while changing the corpus seed varies their setting and language. These choices provide a controllable information boundary and low-cost extension once the family-level components exist.

由于 $M(\theta)$ 与 $z_{\theta}$ 已固定支配结构与评测，$G$ 被要求实现并包装它们，而不是联合发明。这一更窄角色降低了对设定器容量的要求。第 4 节给出实现；附录 C.3 表明，随后作为玩家被优化的起始 Qwen3.6-35B-A3B 检查点，已能充当设定器并产出有效智能体环境。更强设定器主要提高产率与接口打磨。不对称在于接口发布什么，而不在环境是否依赖 $\theta$。因为 $\Omega_{\theta}$ 与 $\mathcal{A}$ 属于包装器，它们提供显式控制点：抽样的哪些成分一开始可见、哪些需交互揭示、哪些保持潜在。在智能体形式下，默认接口不能直接读取完整抽样或任一参考值，策略只能获得其调用返回的视图。图 1(a) 显示由此带来的智能体难度：模型在纸面形式上对底层问题表现很强，在有状态智能体形式上却差很多。由于模型与接口分离，同一抽样可以纸面呈现而不改变底层问题或评测。重新抽样 $\theta$ 并改变其范围或视野可再产实例，更换语料种子则改变设定与语言。一旦族级组件存在，这些选择提供可控的信息边界与低成本扩展。

Once $E(\theta)$ is obtained, policy learning shifts from solving a fully specified written-out problem to acting online through sequential, state-dependent decisions from interface observations, under the same objective and fixed reference $z_{\theta}$. Inventory control gives a concrete example. Per-product demand, one shared capacity, and a joint ordering fee define $M(\theta)$. The state $s_{t}$ records stock on hand, prices and a forecast appear through $\Omega_{\theta}$, and the demand table remains hidden. Stock bought early occupies capacity later, and no action recovers a lost sale. The resulting interface therefore retains both information gathering and binding commitments. App.  G gives the corresponding objects for all eleven families.

一旦得到 $E(\theta)$，策略学习就从求解完全给定的纸面问题，转向在同一目标与固定参考 $z_{\theta}$ 下，依据接口观测做在线的、顺序的、依赖状态的决策。库存控制是一个具体例子。各产品需求、一个共享容量与联合订货费定义 $M(\theta)$。状态 $s_{t}$ 记录在手库存，价格与预测经 $\Omega_{\theta}$ 出现，需求表保持隐藏。早买的库存占用后期容量，且没有任何动作能挽回已失销量。因此所得接口同时保留信息收集与有约束力的承诺。附录 G 给出全部十一个族的对应对象。

3.3 Outcome evaluation

3.3 结果评测

The same pre-generation solution fixes how a completed trajectory is evaluated. A family-specific solver $S_{f}$ computes $z_{\theta}=(u^{*}(\theta),u_{0}(\theta))$, where $u^{*}(\theta)$ is the optimal value of $M(\theta)$ and $u_{0}(\theta)$ is the value of a fixed default policy. Our realization instantiates the general signal $y_{\pi}$ as a scalar episode-level reward:

同一预生成解固定了完成后轨迹如何被评测。族专用求解器 $S_{f}$ 计算 $z_{\theta}=(u^{*}(\theta),u_{0}(\theta))$，其中 $u^{*}(\theta)$ 是 $M(\theta)$ 的最优值，$u_{0}(\theta)$ 是固定默认策略的值。我们的实现把一般信号 $y_{\pi}$ 实例化为标量回合级奖励：

$$S_{f}\colon\theta\ \mapsto\ z_{\theta}=\big(u^{*}(\theta),\,u_{0}(\theta)\big),\qquad\qquad r(\pi;\theta)\;=\;\operatorname{clip}_{[0,1]}\!\left(\frac{u(\pi;\theta)-u_{0}(\theta)}{u^{*}(\theta)-u_{0}(\theta)}\right). \tag{4}$$

The references use closed-form arithmetic, enumeration, or a numerical solver. No language model estimates or judges either endpoint, and both are fixed before the opening observation. The verified optimum gives the scale an upper anchor, while the default policy gives it a lower anchor. Eq. 4 therefore maps the default to $r=0$, the full-information optimum to $r=1$, and clips outcomes to this meaningful range rather than to an arbitrary numerical interval. The score is arithmetic over realized utility and fixed references, so no learned evaluator enters the scoring path. Under partial information, $u^{*}(\theta)$ is an upper reference and need not be attainable by an online policy. Sec. 4 describes the implementation’s equivalent shift of origin.

参考值用闭式算术、枚举或数值求解器得到。没有任何语言模型估计或评判任一端点，二者都在开场观测之前固定。已验证最优给出尺度上锚，默认策略给出下锚。因此式 4 把默认映到 $r=0$，把全信息最优映到 $r=1$，并把结果裁剪到这一有意义区间，而非任意数值区间。分数是对实现效用与固定参考的算术运算，学习型评测器不进入评分路径。在部分信息下，$u^{*}(\theta)$ 是上参考，在线策略未必可达。第 4 节描述实现中等价的原点平移。

Taken together, this construction yields stateful agentic interaction, a verifiable optimum and reference-grounded score, an explicit information boundary, and diverse instances obtained by resampling rather than reauthoring. Sec. 4 realizes these principles through sampling, solving, corpus-grounded synthesis, and behavioral admission. Sec. 5 then tests whether the resulting policies improve on new draws, mechanism families, and larger horizons, and whether written-out and agentic forms expose a distinct operating shortfall.

合在一起，这一构造给出有状态智能体交互、可验证最优与参考锚定分数、显式信息边界，以及通过重抽样而非重写得到的多样实例。第 4 节通过抽样、求解、语料锚定合成与行为准入实现这些原则。第 5 节再检验所得策略是否在新抽样、机制族与更长视野上改进，以及纸面与智能体形式是否暴露出不同的操作短板。

4 Producing environments at scale

4 规模化产出环境

Sec. 3 formalizes how a solved mechanism jointly determines the environment dynamics and its outcome signal. We now turn this principle into a scalable generation pipeline. Fig. 3(b) summarizes the construction, while Tab. 1 reports the resulting substrate.

第 3 节形式化了已解机制如何联合决定环境动力学与结果信号。我们现在把该原则做成可扩展的生成流水线。图 3(b) 概括构造，表 1 报告所得基底。

Sampling and solving. Each instance begins with $\theta\sim P_{f}$, followed by the family solver $S_{f}$. Closed-form procedures, enumeration, dynamic programming, or numerical optimization compute $z_{\theta}$ without a language model and thereby fix the normalized outcome rule in Eq. 4. To make $M(\theta)$ executable, the pipeline maps its decision variables to state updates, enforces its constraints in the transition function, and accumulates its objective as terminal utility. The resulting $D(\theta)$ instantiates the same realization logic across parameter draws, while corpus grounding varies its setting and interface. The inventory example in App. D.1 shows this mapping concretely, with orders updating stock, capacity limiting transitions, and revenue and costs determining utility. Established mathematical and operations-research models make candidate mechanisms straightforward to source and instantiate. App. G catalogues the formulations, decision structures, and reference procedures of all eleven reported families.

抽样与求解。 每个实例从 $\theta\sim P_{f}$ 开始，再经族求解器 $S_{f}$。闭式程序、枚举、动态规划或数值优化在不用语言模型的情况下计算 $z_{\theta}$，从而固定式 4 中的归一化结果规则。为使 $M(\theta)$ 可执行，流水线把决策变量映射为状态更新，在转移函数中强制约束，并把目标累积为终端效用。所得 $D(\theta)$ 在各参数抽样上复用同一实现逻辑，语料锚定则改变其设定与接口。附录 D.1 的库存例子具体展示该映射：订单更新库存、容量限制转移、收入与成本决定效用。成熟数学与运筹模型使候选机制易于来源与实例化。附录 G 编目全部十一个报告族的表述、决策结构与参考程序。

Dimension	Measured scale	Structure
Scenario seeds	28 topical domains; no admitted domain above 4.7%	Real-world documents vary settings, entities, relations, and language
Mechanisms	3 training families; 11 evaluated families	Repeated parameter draws from inventory DP, routing, knapsack, LP/QP, scheduling, search, and related models
Environments	3,300 admitted; $\geq\!10$ tools each; $\approx$$0.01–$0.03 per admitted record	Stateful dynamics, domain-specific interfaces, and player-facing instructions

Table 1: Scale and structure of the generated substrate. Scenario seeds diversify the setting and language, while mechanism families determine the dynamics and outcome semantics. Apps. G, B, C.3, and I report family definitions, verification coverage, the cost reconstruction, and full distributions.

维度	实测规模	结构
场景种子	28 个主题域；无一准入域超过 4.7%	真实世界文档变化设定、实体、关系与语言
机制	3 个训练族；11 个评测族	库存 DP、路由、背包、LP/QP、调度、搜索及相关模型上的反复参数抽样
环境	3,300 个准入；各 $\geq\!10$ 工具；每条准入记录约 $0.01$–$0.03$ 美元	有状态动力学、域专用接口与面向玩家指令

表 1：生成基底的规模与结构。场景种子多样化设定与语言，机制族决定动力学与结果语义。附录 G、B、C.3 与 I 报告族定义、验证覆盖、成本重建与完整分布。

Figure 4: External transfer. (a) Ten interaction-focused BFCL V4 cells gain $2.84$ points. (b) Dots show five storefront runs per arm, horizontal segments mark means, and $\times$ marks bankruptcy. The trained checkpoint completes all five and reaches $3.4\times$ the Base mean, above Qwen3.7-Max. (c) TravelBench plan quality rises from $0.700$ to $0.794$, below the Qwen3.7-Max reference of $0.891$.

图 4：外部迁移。（a）十个交互向 BFCL V4 单元提升 $2.84$ 分。（b）点表示每臂五次店面运行，横线段标均值，$\times$ 标破产。训练检查点五次全部完成，达到 Base 均值的 $3.4\times$，高于 Qwen3.7-Max。（c）TravelBench 计划质量从 $0.700$ 升至 $0.794$，低于 Qwen3.7-Max 参考 $0.891$。

Corpus-grounded realization. For each environment, a frozen language-model setter receives the complete draw and one independently sampled passage from a diverse corpus of real-world documents, then generates the scenario, relational database, domain-specific tool schemas and bodies, and player instruction. The passage supplies the entities, relations, and domain language, while $M(\theta)$ determines the decision process and $z_{\theta}$ fixes its evaluation. The Qwen3.6-35B-A3B starting checkpoint already produces valid agent-facing environments in this role, while a stronger setter mainly improves yield and interface refinement. App. C.2 details the realization and record structure, and App. C.3 gives the comparison. Each record retains the complete draw inside $D(\theta)$ and publishes through $E(\theta)$ only the observations and actions defined by its interface. The player therefore acts on state revealed through catalogue, probe, decision, or clock operations rather than reading the latent parameters or references directly. App. C.4 details this isolation and information boundary.

语料锚定实现。 对每个环境，冻结的语言模型设定器收到完整抽样，以及从多样真实世界文档语料中独立抽样的一段文字，再生成场景、关系数据库、域专用工具模式与实现，以及玩家指令。段落提供实体、关系与领域语言，$M(\theta)$ 决定决策过程，$z_{\theta}$ 固定评测。起始 Qwen3.6-35B-A3B 检查点在该角色下已能产出有效的面向智能体环境；更强设定器主要提高产率与接口打磨。附录 C.2 详述实现与记录结构，附录 C.3 给出比较。每条记录把完整抽样保留在 $D(\theta)$ 内，经 $E(\theta)$ 只发布其接口定义的观测与动作。因此玩家依据目录、探询、决策或时钟操作揭示的状态行动，而不是直接读取潜在参数或参考。附录 C.4 详述该隔离与信息边界。

Admission and scale. Admission tests whether a generated candidate executes and whether its realized behavior agrees with the precomputed construction. Failed candidates are regenerated or rejected before training. Across the held-out audit, every sampled environment executed successfully, repeated outcomes agreed, and realized outcomes remained within their stored references. App. B reports the admission criteria, coverage, and audit statistics. The admitted environments draw scenario seeds from 28 topical domains with normalized entropy 0.997. Across them, realizations range from hardware replenishment and education logistics to adaptive-sports planning. App. D traces these cases from mechanism to interface and admission, while App. I reports the complete environment metadata, family composition, and admission yield. Episodes in these environments average 59.7 assistant turns.

准入与规模。 准入检验生成候选是否可执行，以及其实现行为是否与预计算构造一致。失败候选在训练前被再生或拒绝。在持出审计中，每个抽样环境都成功执行，重复结果一致，实现结果落在存储参考之内。附录 B 报告准入准则、覆盖与审计统计。准入环境的场景种子来自 28 个主题域，归一化熵 0.997。实现覆盖从硬件补货、教育物流到适应性运动规划。附录 D 从机制到接口与准入追踪这些案例；附录 I 报告完整环境元数据、族构成与准入产率。这些环境中的回合平均 59.7 次助手轮次。

Policy optimization. Each admitted environment supplies trajectories and the fixed episode-level signal from Eq. 4. Our realization optimizes groups of rollouts from the same instance with GRPO (Shao et al., 2024). The mechanism-first construction can be paired with other policy optimizers, as formalized in App. E.3. App. C.5 gives the objective and run configuration, while App. F reports the episode, token, and context budgets.

策略优化。 每个准入环境提供轨迹以及式 4 的固定回合级信号。我们的实现用 GRPO（Shao et al., 2024）优化同一实例上成组的 rollout。机制优先构造也可与其他策略优化器配对，见附录 E.3 的形式化。附录 C.5 给出目标与运行配置，附录 F 报告回合、词元与上下文预算。

5 Experiments

5 实验

The experiments address three research questions. RQ1. Do policy gains extend to held-out and unseen mechanisms, external interfaces, and substantially longer horizons? RQ2. If a model can solve the written-out problem, why does stateful interaction still leave substantial room for policy improvement? RQ3. Can environment generation and policy learning co-scale as mechanisms grow?

实验围绕三个研究问题。RQ1. 策略增益是否延伸到持出与未见机制、外部接口以及显著更长的视野？RQ2. 若模型能解纸面问题，为何有状态交互仍留下大量策略改进空间？RQ3. 环境生成与策略学习能否随机制增长而共同扩展？

Experimental setup. The starting policy is Qwen3.6-35B-A3B (Base). GRPO produces the trained policy. The pipeline produces 3,300 admitted environments. Generated-family evaluation covers held-out environments from all three training families and eight unseen families. Generated-family scores use Eq. 4, while external benchmarks retain their native metrics. The primary comparison is between the starting and trained checkpoints under the same evaluation protocol. Qwen3.7-Max provides an additional within-model reference point under its API protocol. App. F specifies the sampling, budget, and execution settings. App. H provides the full result grids and training diagnostics.

实验设置。 起始策略为 Qwen3.6-35B-A3B（Base）。GRPO 产出训练后策略。流水线产出 3,300 个准入环境。生成族评测覆盖全部三个训练族的持出环境以及八个未见族。生成族分数用式 4，外部基准保留其原生指标。主比较是同一评测协议下起始与训练检查点。Qwen3.7-Max 在其 API 协议下提供额外的同系列参考。附录 F 规定抽样、预算与执行设置；附录 H 提供完整结果网格与训练诊断。

5.1 Training gains and transfer

5.1 训练增益与迁移

Across generated mechanism families. At the agentic level, training improves every evaluated family. Fig. 1(b) organizes eleven families by what changes from training. Held-out training families gain $+0.56$ on average. Near-OOD families are unseen optimization families and gain $+0.63$. Far-OOD families change the decision structure or restrict probing and gain $+0.24$ and $+0.16$ across the two groups. App. H reports the complete family grid.

跨生成机制族。 在智能体层面上，训练改进每一个评测族。图 1(b) 按相对训练的变化组织十一个族。持出训练族平均增益 $+0.56$。近 OOD 族是未见优化族，增益 $+0.63$。远 OOD 族改变决策结构或限制探询，两组分别增益 $+0.24$ 与 $+0.16$。附录 H 报告完整族网格。

Beyond the generated substrate. We evaluate the trained checkpoint on three external agentic or tool-use benchmarks that retain their native protocols and share neither instances nor interfaces with the generated training environments. For general function calling, we report BFCL V4’s ten Multi-Turn, Agentic, and Hallucination cells, matching our focus on stateful tool use (Patil et al., 2025). TravelBench asks agents to use travel tools to assemble multi-day itineraries satisfying coupled timing, budget, commonsense, and personalized constraints (Cheng et al., 2025). E-Commerce Bench runs a 365-day autonomous business in which an agent manages multiple storefronts, customer orders, returns, market events, and delayed cash flow (Fan et al., 2026).

超出生成基底。 我们在三个外部智能体或工具使用基准上评测训练检查点；它们保留原生协议，且与生成训练环境既无共享实例也无共享接口。通用函数调用方面，我们报告 BFCL V4 的十个 Multi-Turn、Agentic 与 Hallucination 单元，以匹配我们对有状态工具使用的关注（Patil et al., 2025）。TravelBench 要求智能体用旅行工具组装满足耦合时间、预算、常识与个性化约束的多日行程（Cheng et al., 2025）。E-Commerce Bench 运行 365 天自主业务，智能体管理多个店面、客户订单、退货、市场事件与延迟现金流（Fan et al., 2026）。

Fig. 4 summarizes the external results, with the clearest change on the longest-horizon benchmark. Across five reported storefront runs per arm, the base checkpoint completes four and goes bankrupt once. The trained 35B-A3B checkpoint completes all five without bankruptcy over $1{,}147$–$1{,}825$ assistant turns, far beyond the 59.7-turn generated-family average in Sec. 4. Its mean ending balance rises from $54{,}294$ to $182{,}844$, or $3.4\times$, and exceeds the Qwen3.7-Max reference of $165{,}224$. The unweighted mean over this BFCL V4 subset rises from $61.25$ to $64.08$, and TravelBench plan quality increases from $0.700$ to $0.794$, below the Qwen3.7-Max reference of $0.891$. These results show transfer to general tool use, constrained planning, and substantially longer interaction. App. H reports the remaining native metrics and cell-level breakdowns.

图 4 概括外部结果，最长视野基准上的变化最清晰。每臂五次报告的店面运行中，基线检查点完成四次、破产一次。训练后的 35B-A3B 检查点在 $1{,}147$–$1{,}825$ 次助手轮次内五次全部完成且未破产，远超第 4 节生成族平均 59.7 轮。其平均期末余额从 $54{,}294$ 升至 $182{,}844$，即 $3.4\times$，并超过 Qwen3.7-Max 参考 $165{,}224$。该 BFCL V4 子集的未加权均值从 $61.25$ 升至 $64.08$，TravelBench 计划质量从 $0.700$ 升至 $0.794$，低于 Qwen3.7-Max 参考 $0.891$。这些结果表明向通用工具使用、约束规划与显著更长交互的迁移。附录 H 报告其余原生指标与单元级分解。

5.2 A learnable gap under stateful interaction

5.2 有状态交互下的可学习差距

	Complete Problem	Parameters Revealed	Parameters Acquired
Family	Written-Out $F$	Informed-Agentic $I$	Agentic $A$
Inventory DP	0.976 $\rightarrow$ 0.999	0.469 $\rightarrow$ 0.968	0.449 $\rightarrow$ 0.912
Knapsack	0.984 $\rightarrow$ 0.996	0.132 $\rightarrow$ 0.948	0.117 $\rightarrow$ 0.843
Linear Programming	0.978 $\rightarrow$ 1.000	0.161 $\rightarrow$ 0.732	0.143 $\rightarrow$ 0.661
Quadratic Programming	0.975 $\rightarrow$ 1.000	0.168 $\rightarrow$ 0.836	0.162 $\rightarrow$ 0.795
Routing	0.897 $\rightarrow$ 0.965	0.226 $\rightarrow$ 0.889	0.148 $\rightarrow$ 0.866
Family Macro	0.962 $\rightarrow$ 0.992	0.231 $\rightarrow$ 0.875	0.204 $\rightarrow$ 0.815

Table 2: Five optimization families under written-out ($F$), informed-agentic ($I$), and agentic ($A$) presentations. Scores show Base $\rightarrow$ trained under the same solver-derived outcome scale.

	完整问题	参数已揭示	参数需获取
族	纸面 $F$	知情智能体 $I$	智能体 $A$
库存 DP	0.976 $\rightarrow$ 0.999	0.469 $\rightarrow$ 0.968	0.449 $\rightarrow$ 0.912
背包	0.984 $\rightarrow$ 0.996	0.132 $\rightarrow$ 0.948	0.117 $\rightarrow$ 0.843
线性规划	0.978 $\rightarrow$ 1.000	0.161 $\rightarrow$ 0.732	0.143 $\rightarrow$ 0.661
二次规划	0.975 $\rightarrow$ 1.000	0.168 $\rightarrow$ 0.836	0.162 $\rightarrow$ 0.795
路由	0.897 $\rightarrow$ 0.965	0.226 $\rightarrow$ 0.889	0.148 $\rightarrow$ 0.866
族宏观	0.962 $\rightarrow$ 0.992	0.231 $\rightarrow$ 0.875	0.204 $\rightarrow$ 0.815

表 2：五个优化族在纸面（$F$）、知情智能体（$I$）与智能体（$A$）呈现下的结果。分数为同一求解器导出结果尺度下的 Base $\rightarrow$ 训练后。

Across five optimization families, we compare each problem in written-out ($F$), informed-agentic ($I$), and agentic ($A$) forms under the same solver-derived outcome scale. In $F$, the model reasons over the complete instance and returns an answer, with the Python tool available. In $I$ and $A$, the problem and evaluation stay fixed, but the model must instead act as a policy through the same multi-step environment. $I$ reveals all sampled parameters upfront, while $A$ exposes them only through interaction. In inventory control, for example, $I$ shows the demand for each product before the episode, whereas $A$ requires the model to collect and analyze demand information before deciding how much to order.

在五个优化族上，我们在同一求解器导出的结果尺度下比较每个问题的纸面（$F$）、知情智能体（$I$）与智能体（$A$）形式。在 $F$ 中，模型对完整实例推理并返回答案，可用 Python 工具。在 $I$ 与 $A$ 中，问题与评测保持固定，但模型必须作为策略在同一多步环境中行动。$I$ 预先揭示全部抽样参数，而 $A$ 只通过交互暴露它们。以库存控制为例，$I$ 在回合开始前展示各产品需求，而 $A$ 要求模型先收集并分析需求信息，再决定订多少货。

The construction preserves problem-solving competence while exposing a large, learnable policy gap. The base checkpoint scores $0.962$ in $F$ but falls to $0.231$ in $I$, although the sampled parameters remain available. It must now carry decisions through state changes, shared constraints, and commitments whose consequences persist across the horizon. Requiring those parameters to be acquired through the interface further lowers $A$ to $0.204$. Training changes $(F,I,A)$ by $(+0.030,+0.644,+0.611)$, closing $84\%$ of the parameter-revealed stateful gap and $77\%$ of the full written-out-to-agentic gap.

该构造保持解题能力，同时暴露出大而可学习的策略差距。 基线检查点在 $F$ 上得 $0.962$，在 $I$ 上却落到 $0.231$，尽管抽样参数仍可用。它现在必须把决策贯穿状态变化、共享约束，以及后果跨视野持续的承诺。要求经接口获取这些参数，进一步把 $A$ 降到 $0.204$。训练使 $(F,I,A)$ 变化 $(+0.030,+0.644,+0.611)$，弥合参数已揭示有状态差距的 $84\%$，以及完整纸面到智能体差距的 $77\%$。

Representative trajectories in App. H.3 make the gap concrete. In a seven-period allocation task, both checkpoints reveal all eight items and invoke the code tool. The base policy nevertheless writes “For each period, find best combination,” spends 27 of 29 shared-budget units in the first three periods, and ends with score 0. The trained policy instead writes “I need to plan across all 7 periods,” constructs one horizon-wide allocation, and reaches score 1. The paired excerpts in App. H.3.1 show that the failure is not recognition of the local knapsack but construction of the coupled horizon-wide model. In the observation case reported in App. H.3.2, the base policy states that it needs to probe the items but acts after revealing only four of eight. The trained policy reveals all eight before planning and again reaches score 1. These traces expose myopic optimization under persistent cross-period state and premature commitment under incomplete observation. App. H.3.3 gives a third paired case in which both policies identify the same leading option, but only the trained policy stops acquiring information before its cost overwhelms the outcome.

附录 H.3 的代表轨迹使差距具体化。在七期分配任务中，两个检查点都揭示全部八个物品并调用代码工具。基线策略仍写「对每一期找最佳组合」，在前三期花掉 29 个共享预算单位中的 27 个，最终得分为 0。训练后策略则写「我需要跨全部 7 期规划」，构造一次全视野分配，并达到分数 1。附录 H.3.1 的成对摘录表明，失败不在识别局部背包，而在构造耦合的全视野模型。附录 H.3.2 报告的观测案例中，基线策略声称需要探询物品，却在只揭示八个中的四个后就行动。训练后策略在规划前揭示全部八个，并再次达到分数 1。这些轨迹暴露了在持续跨期状态下的短视优化，以及在不完整观测下的过早承诺。附录 H.3.3 给出第三组成对案例：两个策略识别同一领先选项，但只有训练后策略在信息成本淹没结果之前停止获取信息。

The gain is not explained by only learning to call the code tool. The fixed-stratum analysis in App. H.5 attributes $+0.030$ of a $+0.443$ agentic gain to increased adoption across the eight families with both tool-use strata, leaving $+0.413$ within fixed strata. Consistent with a capability-elicitation interpretation, the gain coincides with a modest policy shift of $D_{\mathrm{KL}}(\pi_{\mathrm{trained}}\|\pi_{\mathrm{Base}})=0.089$ nats per assistant token on held-out trajectory states, with the estimation protocol and breakdown reported in App. H.2. The construction therefore preserves the mathematical problem while creating substantial room to learn information acquisition, state-dependent decisions, and sustained execution. The unseen-family and external gains in Sec. 5.1 show how far that policy improvement extends beyond the original mechanisms and interfaces.

增益不能仅用学会调用代码工具来解释。附录 H.5 的固定层分析把 $+0.443$ 智能体增益中的 $+0.030$ 归因于在同时具有工具使用层的八个族上采纳增加，余下 $+0.413$ 落在固定层内。与能力激发解释一致，增益伴随温和的策略偏移：$D_{\mathrm{KL}}(\pi_{\mathrm{trained}}\|\pi_{\mathrm{Base}})=0.089$ nats/助手词元（持出轨迹状态上），估计协议与分解见附录 H.2。因此该构造保持数学问题，同时为学习信息获取、依赖状态的决策与持续执行留下大量空间。第 5.1 节的未见族与外部增益显示，该策略改进能延伸到原始机制与接口之外多远。

5.3 Co-scaling environment and policy frontiers

5.3 环境与策略前沿的共同扩展

The construction opens learnable space beyond the current policy. RQ3 asks whether environment generation and policy learning can continue to advance as mechanism size and horizon grow. Fig. 5 increases task complexity across four configurations by scaling both the number of items and the decision horizon, from 6 items over 5 periods to 11 items over 11 periods. We use two copies of Qwen3.6-35B-A3B. One remains frozen and realizes admitted environments from the sampled mathematical models and their reference outcomes under a fixed interface, solver, and evaluator. At each scale, the other starts as Base, trains under a matched budget, and is evaluated on fresh held-out environments at the corresponding scale. Each filled point represents one such scale-matched checkpoint.

该构造打开超出当前策略的可学习空间。 RQ3 问：环境生成与策略学习能否随机制规模与视野增长继续推进。图 5 通过同时放大物品数与决策视野，在四个配置上提高任务复杂度，从 6 物品×5 期到 11 物品×11 期。我们使用两份 Qwen3.6-35B-A3B。一份保持冻结，在固定接口、求解器与评测器下，从抽样数学模型及其参考结果实现准入环境。在每个尺度上，另一份从 Base 起步，在匹配预算下训练，并在对应尺度的新鲜持出环境上评测。每个实心点表示这样一个尺度匹配检查点。

Figure 5: Environment–policy co-scaling over four knapsack configurations. Each filled point is a separate checkpoint trained and evaluated on fresh environments at that scale. Labels give its gain over the common Base.

图 5：四个背包配置上的环境–策略共同扩展。每个实心点是在该尺度上训练并在新鲜环境上评测的单独检查点。标签给出相对共同 Base 的增益。

The shared starting checkpoint makes this asymmetry concrete. Across all four scales, its frozen construction copy continues to realize admitted environments that leave substantial room for its Base policy to improve. At $6\times 5$, the Base and trained policies score $0.432$ and $0.843$. At $11\times 11$, they score $0.085$ and $0.946$. The gains across the sweep are $+0.412$, $+0.551$, $+0.675$, and $+0.861$, with the two larger configurations averaging $+0.768$ compared with $+0.482$ for the two smaller configurations. The mathematical model makes the complete structure and reference outcomes available during construction, while the policy receives only interface observations and must recover the relevant information through interaction. The same checkpoint can consequently construct an environment that is harder than it can initially navigate as a policy, and the growing gains show that this opened space remains learnable as mechanism scale increases. This pattern shows the potential for environment construction and policy learning to co-scale under a shared evaluator. App. C.3 reports complementary construction results, and App. J gives the exact level-wise values.

共享起始检查点使这种不对称具体化。在全部四个尺度上，其冻结构造副本继续实现准入环境，给 Base 策略留下大量改进空间。在 $6\times 5$ 上，Base 与训练策略得分为 $0.432$ 与 $0.843$。在 $11\times 11$ 上，分别为 $0.085$ 与 $0.946$。扫描上的增益为 $+0.412$、$+0.551$、$+0.675$ 与 $+0.861$，两个更大配置平均 $+0.768$，两个更小配置平均 $+0.482$。数学模型在构造时提供完整结构与参考结果，而策略只收到接口观测，必须通过交互恢复相关信息。因此同一检查点能构造出比它作为策略起初能导航的更难环境，而不断增大的增益表明，随机制规模增大，这一打开的空间仍可学习。该模式显示在共享评测器下环境构造与策略学习共同扩展的潜力。附录 C.3 报告互补构造结果，附录 J 给出精确的分层数值。

6 Conclusion

6 结论

Mechanism-first construction turns solved models into diverse stateful environments with solver-derived outcome signals. Corpus grounding supplies the setting, while asymmetric interfaces require information acquisition and consequential action. Across the mechanisms and external benchmarks tested here, the resulting training improves policy performance under changing state while preserving written-out problem solving. The setter and scale experiments further show that a fixed setter can realize new draws and that increasing mechanism size and horizon continues to provide useful training headroom under the same evaluator. By deriving environment dynamics and outcome rules from a shared solved mechanism, VHD-Play extends verifiable reinforcement learning from answers toward trajectories and offers a practical framework for expanding agentic training environments as policy capabilities develop across broader task settings.

机制优先构造把已解模型变成多样的有状态环境，并配上求解器导出的结果信号。语料锚定提供设定，不对称接口要求信息获取与有后果的行动。在本文测试的机制与外部基准上，所得训练在状态变化下改进策略表现，同时保持纸面解题能力。设定器与尺度实验进一步表明，固定设定器能实现新抽样，而增大机制规模与视野在同一评测器下继续提供有用的训练余量。通过从共享已解机制导出环境动力学与结果规则，VHD-Play 把可验证强化学习从答案延伸到轨迹，并为策略能力在更广任务设定上发展时扩展智能体训练环境提供了实用框架。

References

参考文献

Appendix

附录

Backlund and Petersson (2025) A. Backlund and L. Petersson Vending-bench: a benchmark for long-term coherence of autonomous agents . arXiv preprint arXiv:2502.15840 . Cited by: §1 .

Cai et al. (2025) S. Cai, R. Fang, J. Wu, et al. AutoForge: automated environment synthesis for agentic reinforcement learning . arXiv preprint arXiv:2512.22857 . Cited by: §E.2 , §1 , §2 .

Chae et al. (2025) H. Chae, S. Kim, J. Cho, S. Kim, S. Moon, G. Hwangbo, D. Lim, M. Kim, et al. Web-Shepherd: advancing PRMs for reinforcing web agents . In Advances in Neural Information Processing Systems (NeurIPS) , Note: Spotlight. arXiv:2505.15277 Cited by: §2 .

Cheng et al. (2025) X. Cheng, Y. Hu, X. Zhang, L. Xu, L. Tan, Z. Pan, X. Li, and Y. Liu Beyond itinerary planning: a real-world benchmark for multi-turn and tool-using travel tasks . arXiv preprint arXiv:2512.22673 . Note: Accepted to ACL 2026 Cited by: §5.1 .

Christiano et al. (2017) P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei Deep reinforcement learning from human preferences . Advances in Neural Information Processing Systems . Cited by: §E.1 , §1 , §2 .

Cobbe et al. (2020) K. Cobbe, C. Hesse, J. Hilton, and J. Schulman Leveraging procedural generation to benchmark reinforcement learning . In Proceedings of the 37th International Conference on Machine Learning , Proceedings of Machine Learning Research , Vol. 119 , pp. 2048–2056 . Cited by: §2 .

Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168 . Cited by: §E.3 .

Côté et al. (2018) M. Côté, Á. Kádár, X. Yuan, B. Kybartas, T. Barnes, E. Fine, J. Moore, R. Y. Tao, M. Hausknecht, L. El Asri, M. Adada, W. Tay, and A. Trischler TextWorld: a learning environment for text-based games . arXiv preprint arXiv:1806.11532 . Cited by: §2 .

DeepSeek-AI et al. (2025) DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning . Nature 645 , pp. 633–638 . External Links: Document Cited by: Appendix A , §1 , §2 .

Fan et al. (2026) W. Fan, X. Shen, X. Guo, J. Tu, Y. Su, Y. Zhang, L. Deng, F. Wang, B. Dong, Y. Song, and D. Liu E-commerce bench: evaluating llm agents on long-horizon autonomous business operation . Cited by: §1 , §5.1 .

Gao et al. (2026) J. Gao, J. Chen, C. He, S. Xu, D. Jin, and Y. Wu From self-evolving synthetic data to verifiable-reward rl: post-training multi-turn interactive tool-using agents . arXiv preprint arXiv:2601.22607 . Cited by: §1 , §2 .

Gunjal et al. (2025) A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. Hendryx Rubrics as rewards: reinforcement learning beyond verifiable domains . arXiv preprint arXiv:2507.17746 . Cited by: §E.1 , §1 , §2 .

He et al. (2026) Z. He, Y. Wang, C. Zhi, Y. Hu, T. Chen, L. Yin, Z. Chen, T. A. Wu, et al. MemoryArena: benchmarking agent memory in interdependent multi-session agentic tasks . arXiv preprint arXiv:2602.16313 . Cited by: §1 .

Hu et al. (2025) J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum Open-reasoner-zero: an open source approach to scaling up reinforcement learning on the base model . arXiv preprint arXiv:2503.24290 . Cited by: Appendix A , §1 .

Huang et al. (2025) T. Huang, S. Chen, M. Chen, J. May, L. Yang, M. Wan, and P. Zhou Teaching language models to gather information proactively . In Findings of the Association for Computational Linguistics: EMNLP 2025 , pp. 15588–15599 . External Links: Document Cited by: §2 .

Hubbs et al. (2020) C. D. Hubbs, H. D. Perez, O. Sarwar, N. V. Sahinidis, I. E. Grossmann, and J. M. Wassick OR-Gym: a reinforcement learning library for operations research problems . arXiv preprint arXiv:2008.06319 . Cited by: §1 , §2 , §3 .

Jin et al. (2025) B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han Search-R1: training LLMs to reason and leverage search engines with reinforcement learning . arXiv preprint arXiv:2503.09516 . Cited by: §1 , §2 .

Kaelbling et al. (1998) L. P. Kaelbling, M. L. Littman, and A. R. Cassandra Planning and acting in partially observable stochastic domains . Artificial Intelligence 101 ( 1–2 ), pp. 99–134 . External Links: Document Cited by: §2 , §3.2 .

Kwa et al. (2025) T. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, et al. Measuring AI ability to complete long software tasks . In Advances in Neural Information Processing Systems (NeurIPS) , Note: arXiv:2503.14499 Cited by: §1 .

Lightman et al. (2023) H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe Let’s verify step by step . arXiv preprint arXiv:2305.20050 . Cited by: §E.3 .

Liu et al. (2023) B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone LLM+P: empowering large language models with optimal planning proficiency . arXiv preprint arXiv:2304.11477 . Cited by: §2 .

Liu et al. (2025) X. Liu, K. Wang, Y. Wu, F. Huang, Y. Li, J. Zhang, and J. Jiao Agentic reinforcement learning with implicit step rewards . arXiv preprint arXiv:2509.19199 . Cited by: §E.3 , §2 .

Lyu et al. (2026) Y. Lyu, C. Wang, L. Shen, J. Huang, and T. Xu Mock worlds, real skills: building small agentic language models with synthetic tasks, simulated environments, and rubric-based rewards . arXiv preprint arXiv:2601.22511 . Cited by: §1 , §2 .

Norman et al. (2026) J. D. Norman, M. U. Rivera, and D. A. Hughes Reliability without validity: a systematic, large-scale evaluation of LLM-as-a-judge models across agreement, consistency, and bias . arXiv preprint arXiv:2606.19544 . Cited by: §E.1 , §1 , §2 .

Orlanski et al. (2026) G. Orlanski, D. Roy, A. Yun, C. Shin, A. Gu, A. Ge, D. Adila, N. Roberts, et al. SlopCodeBench: benchmarking how coding agents degrade over long-horizon iterative tasks . arXiv preprint arXiv:2603.24755 . Cited by: §1 .

Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, et al. Training language models to follow instructions with human feedback . Advances in Neural Information Processing Systems . Cited by: §E.1 , §1 , §2 .

Patil et al. (2025) S. G. Patil, H. Mao, F. Yan, C. C. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez The Berkeley function calling leaderboard (BFCL): from tool use to agentic evaluation of large language models . In Proceedings of the 42nd International Conference on Machine Learning , Proceedings of Machine Learning Research , Vol. 267 . Cited by: §1 , §5.1 .

Rabanser et al. (2026) S. Rabanser, S. Kapoor, P. Kirgis, K. Liu, S. Utpala, and A. Narayanan Towards a science of AI agent reliability . In Proceedings of the 43rd International Conference on Machine Learning (ICML) , Note: arXiv:2602.16666 Cited by: §1 .

Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms . arXiv preprint arXiv:1707.06347 . Cited by: §E.3 .

Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models . arXiv preprint arXiv:2402.03300 . Cited by: §E.3 , §4 .

Silver and Chitnis (2020) T. Silver and R. Chitnis PDDLGym: gym environments from PDDL problems . arXiv preprint arXiv:2002.06432 . Cited by: §2 .

Sinha et al. (2026) A. Sinha, A. Arun, S. Goel, S. Staab, and J. Geiping The illusion of diminishing returns: measuring long horizon execution in LLMs . In International Conference on Learning Representations (ICLR) , Note: arXiv:2509.09677 Cited by: §2 .

Song et al. (2026) X. Song, H. Chang, G. Dong, Y. Zhu, J. Wen, and Z. Dou EnvScaler: scaling tool-interactive environments for llm agent via programmatic synthesis . arXiv preprint arXiv:2601.05808 . Cited by: §E.2 , §E.2 , §E.2 , Table 7 , Table 7 , §1 , §2 .

Stein et al. (2023) K. Stein, D. Fišer, J. Hoffmann, and A. Koller Automating the generation of prompts for LLM-based action choice in PDDL planning . arXiv preprint arXiv:2311.09830 . Cited by: §2 .

Stojanovski et al. (2025) Z. Stojanovski, O. Stanley, J. Sharratt, R. Jones, A. Adefioye, J. Kaddour, and A. Köpf Reasoning gym: reasoning environments for reinforcement learning with verifiable rewards . arXiv preprint arXiv:2505.24760 . Cited by: Appendix A , §1 , §2 .

Tao et al. (2026) L. Tao, B. Peng, W. Yao, T. Ge, H. Cheng, M. H. Wang, J. Gao, and S. Li TRACE: turn-level reward assignment via credit estimation for long-horizon agents . arXiv preprint arXiv:2607.13988 . Cited by: §2 .

Tu et al. (2026) D. Tu, H. Hao, H. Yang, et al. ScaleEnv: scaling environment synthesis from scratch for generalist interactive tool-use agent training . arXiv preprint arXiv:2602.06820 . Cited by: §E.2 , §1 , §2 .

Wang et al. (2026a) X. J. Wang, H. Bai, Y. Sun, et al. The long-horizon task mirage? diagnosing where and why agentic systems break . arXiv preprint arXiv:2604.11978 . Cited by: §2 .

Wang et al. (2025) Y. Wang, D. Yin, Y. Cui, et al. LLMs as scalable, general-purpose simulators for evolving digital agent training . arXiv preprint arXiv:2510.14969 . Cited by: §2 .

Wang et al. (2026b) Z. Wang, F. Wu, H. Wang, X. Tang, B. Li, Z. Yin, Y. Ma, Y. Li, et al. Why reasoning fails to plan: a planning-centric analysis of long-horizon decision making in LLM agents . arXiv preprint arXiv:2601.22311 . Cited by: §1 .

Wang et al. (2026c) Z. Wang, C. Xu, B. Liu, et al. Agent world model: infinity synthetic environments for agentic reinforcement learning . arXiv preprint arXiv:2602.10090 . Cited by: §1 , §2 .

Wei et al. (2025) Q. Wei, S. Zeng, C. Li, Z. Wang, W. Brown, O. Frunza, W. Deng, A. Schneider, Y. Nevmyvaka, Y. K. Zhao, A. Garcia, and M. Hong Reinforcing multi-turn reasoning in LLM agents via fine-grained reward structure and credit assignment . arXiv preprint arXiv:2505.11821 . Cited by: §E.3 .

Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan τ \tau -bench: a benchmark for tool-agent-user interaction in real-world domains . arXiv preprint arXiv:2406.12045 . Cited by: §1 .

Ye et al. (2023) X. Ye, Q. Chen, I. Dillig, and G. Durrett SatLM: satisfiability-aided language models using declarative prompting . In Advances in Neural Information Processing Systems , Note: arXiv:2305.09656 Cited by: §2 .

Zeng et al. (2026) Z. Zeng, H. Ivison, Y. Wang, L. Yuan, S. S. Li, Z. Ye, S. Li, J. He, et al. RLVE: scaling up reinforcement learning for language models with adaptive verifiable environments . In Proceedings of the 43rd International Conference on Machine Learning (ICML) , Note: arXiv:2511.07317 Cited by: §1 , §2 .

Zhang et al. (2025) J. Zhang, Y. Peng, F. Kong, C. Yang, Y. Wu, Z. Yu, J. Xiang, J. Ruan, et al. AutoEnv: automated environments for measuring cross-environment agent learning . arXiv preprint arXiv:2511.19304 . Cited by: §2 .

Zhou et al. (2025) Z. Zhou, X. Feng, Z. Zhu, J. Yao, S. Koyejo, and B. Han From passive to active reasoning: can large language models ask the right questions under incomplete information? . In Proceedings of the 42nd International Conference on Machine Learning (ICML) , Note: arXiv:2506.08295 Cited by: §2 .

Contents

目录

Appendix A Discussion

附录 A 讨论

Scaling agentic training requires diverse settings, realistic stateful interaction, and dependable outcome signals to grow together. VHD-Play assigns these roles within one construction. Corpus seeds supply settings and language, the underlying mechanism specifies state transitions and outcome semantics, and agent-facing interfaces determine what the policy can observe before consequential action. The precomputed reference anchors the resulting outcome signal.

扩展智能体训练需要多样设定、真实的有状态交互，以及可靠的结果信号共同增长。VHD-Play 在同一构造中分配这些角色。语料种子提供设定与语言，底层机制规定状态转移与结果语义，面向智能体的接口决定策略在采取有后果行动前能观测到什么。预计算参考锚定所得结果信号。

The comparison across the three presentations exposes a gap between mathematical competence and agentic policy. The starting checkpoint remains near ceiling when the complete problem is given, yet falls sharply when the same mechanism unfolds through changing state, shared constraints, and persistent commitments. Most of this gap remains when the parameters are revealed. Training raises both stateful forms while preserving written-out problem solving, a pattern consistent with capability elicitation. Representative trajectories connect this change to information acquisition, cross-period decisions, and the cost of continued exploration. App. H.2 measures the policy shift, and App. H.5 separates it from increased code-tool use. Improvement also appears on unseen mechanisms and external stateful tasks (Sec. 5.1).

三种呈现方式的比较暴露了数学能力与智能体策略之间的差距。起始检查点在完整问题给定时仍接近上限，但当同一机制经变化状态、共享约束与持续承诺展开时急剧下降。即便参数已揭示，这一差距的大部分仍然存在。训练抬高两种有状态形式，同时保持纸面解题，这一模式与能力激发一致。代表轨迹把这一变化连接到信息获取、跨期决策与继续探索的成本。附录 H.2 度量策略偏移，附录 H.5 将其与增加的代码工具使用分开。增益也出现在未见机制与外部有状态任务上（第 5.1 节）。

More broadly, we view the learnable space in agentic training as arising from an asymmetry between the structure available when an environment is constructed and what a policy can recover through interaction. This asymmetry may come from human design or the capabilities of a stronger setter. In VHD-Play, much of the structure behind it is already supplied by the mathematical model through latent quantities, coupled constraints, and delayed consequences. The interface determines how this structure is revealed to the policy. The starting 35B checkpoint can already realize valid new draws, while mechanism parameters extend scale, horizon, and information cost under the same evaluator. The setter and frontier results suggest that this substrate can expand with policy competence. App. K summarizes the scope of the evidence.

更广地说，我们把智能体训练中的可学习空间视为来自一种不对称：构造环境时可获得的结构，与策略通过交互能恢复的内容之间的不对称。它可能来自人工设计，也可能来自更强设定器的能力。在 VHD-Play 中，其背后的大量结构已由数学模型通过潜在量、耦合约束与延迟后果提供。接口决定这些结构如何向策略揭示。起始 35B 检查点已能实现有效的新抽样，而机制参数在同一评测器下扩展规模、视野与信息成本。设定器与前沿结果表明，该基底可随策略能力扩展。附录 K 概括证据范围。

This construction suggests a broader role for formal decision models in agentic learning. Written-out mathematics has provided language models with a scalable substrate for reasoning because structured problems admit inexpensive verification (DeepSeek-AI et al., 2025; Hu et al., 2025; Stojanovski et al., 2025). Agentic learning requires analogous structure not only in problems and answers, but also in observations, actions, and consequences across trajectories. Operations research offers a direct route because established models encode objectives, constraints, resource coupling, and sequential decisions, while mature solution procedures provide fixed reference outcomes. Their realization and interface turn this structure into an interactive process. We do not claim that operations research is the only substrate for this transition. More generally, any formal mechanism that supports parameterized variation, executable realization, controlled information asymmetry, and an outcome standard fixed before interaction could play a similar role. Operations research is useful here because established families provide these properties together. The present experiments establish the feasibility and learnability of this construction over the evaluated families, not that the policy explicitly reconstructs the underlying model or that operations research is sufficient for agentic competence more generally.

这一构造暗示形式决策模型在智能体学习中可扮演更广角色。纸面数学之所以为语言模型提供可扩展的推理基底，是因为结构化问题容许廉价验证（DeepSeek-AI et al., 2025; Hu et al., 2025; Stojanovski et al., 2025）。智能体学习需要类似结构，不仅在问题与答案中，也在跨轨迹的观测、动作与后果中。运筹学提供直接路径：成熟模型编码目标、约束、资源耦合与顺序决策，而成熟求解程序提供固定参考结果。其实现与接口把这一结构变成交互过程。我们并不声称运筹学是这一过渡的唯一基底。更一般地，任何支持参数化变化、可执行实现、可控信息不对称，以及在交互前固定结果标准的形式机制，都可扮演类似角色。运筹学在此有用，是因为成熟族把这些性质一并提供。当前实验确立的是：在评测族上该构造可行且可学习；并不表明策略显式重建底层模型，也不表明运筹学对更一般的智能体能力已足够。

Appendix B Admission and verification

附录 B 准入与验证

Admission evaluates three criteria during generation and replay. Each can reject a record, subject to the replay coverage specified below. A failed generation check triggers another attempt. Write $u^{*}(\theta)$ for the optimal value and $u_{0}(\theta)$ for the default one, both computed by the sampler before the dynamics $D(\theta)$ existed. Replaying a policy $\pi$ through the generated code accumulates a terminal cumulative utility $u(\pi;\theta)$. Where the verifier has compatible drivers, it uses up to three: $\pi^{*}$ solving the family’s program, $\pi_{0}$ greedy on the visible values, and $\pi_{\varnothing}$ committing nothing at all. Criteria below appear in the order they apply.

准入在生成与回放期间评估三项准则。每一项都可拒绝一条记录，受下文规定的回放覆盖约束。生成检查失败会触发再次尝试。记 $u^{*}(\theta)$ 为最优值、$u_{0}(\theta)$ 为默认值，二者都在动力学 $D(\theta)$ 存在之前由抽样器算好。把策略 $\pi$ 经生成代码回放，会累积终端累计效用 $u(\pi;\theta)$。在验证器具有兼容驱动时，最多使用三个：$\pi^{*}$ 求解该族程序、$\pi_{0}$ 对可见值贪婪，以及 $\pi_{\varnothing}$ 完全不做承诺。下列准则按适用顺序出现。

C1, executability. The generated dynamics and declared interface tools must initialize and execute without exception. Where compatible replay exists, each available reference policy must reach a numeric terminal outcome.

C1，可执行性。 生成的动力学与声明的接口工具必须初始化并执行且不抛异常。在存在兼容回放处，每个可用参考策略都必须达到数值终端结果。

C2, a valid default region. The default outcome must lie in a family-defined admissible region appropriate to the objective’s scale and direction. A candidate outside this region is regenerated.

C2，有效默认区域。 默认结果必须落在与目标尺度与方向相适的族定义可接受区域内。落在该区域外的候选将被再生。

C3, reward agreement at the optimum. For replay-supported families, admission requires the reference driver to reproduce the precomputed optimum within a family-specific numerical tolerance. Routing uses a separate driver that acts through the environment interface and applies the same agreement check to the optimum and default outcomes.

C3，最优处的奖励一致。 对支持回放的族，准入要求参考驱动在族专用数值容差内复现预计算最优。路由使用单独驱动，经环境接口行动，并对最优与默认结果施加同一致性检查。

A stratified post-generation audit drew ten held-out environments from each of the eight replay-supported families and ran oracle, default, and idle policies twice from fresh state. Every family passed all ten environments. Across 480 executions, none raised an exception, all 240 repeated terminal outcomes agreed, and no replayed raw utility exceeded its precomputed $u^{*}(\theta)$. App. D shows the corresponding mechanism, interface, and outcome structure for inventory, routing, and knapsack.

分层的生成后审计从八个支持回放的族中各抽取十个持出环境，并从新鲜状态两次运行 oracle、默认与空闲策略。每个族的十个环境全部通过。在 480 次执行中，无一抛出异常，全部 240 次重复终端结果一致，且没有任何回放原始效用超过其预计算 $u^{*}(\theta)$。附录 D 展示库存、路由与背包对应的机制、接口与结果结构。

Post-generation replay selects a driver from the interface roles declared by each record rather than from its family label. The shared driver covers seven reported optimization families, namely inventory DP, facility location, budget-coupled knapsack, linear and quadratic programming, scheduling, and scheduling with tardiness. It executes a family solution obtained in closed form, by enumeration, by dynamic programming, or from a numerical solver, then compares the realized outcome with the precomputed reference.

生成后回放从每条记录声明的接口角色中选择驱动，而不是从其族标签。共享驱动覆盖七个报告优化族，即库存 DP、设施选址、预算耦合背包、线性与二次规划、调度，以及带拖期的调度。它执行以闭式、枚举、动态规划或数值求解器得到的族解，再把实现结果与预计算参考比较。

Routing uses the separate direct-interface replay described under C3. The three other reported families also have computable references. Negotiation uses backward induction to obtain a session oracle. Sequential stopping and sequential search use hindsight ceilings together with their default references. These quantities enter the outcome rule directly. What varies across families is how the reference is obtained and whether the shared driver can execute a corresponding trajectory, not whether a reference exists.

路由使用 C3 所述的单独直接接口回放。其余三个报告族也有可计算参考。谈判用逆向归纳得到会话 oracle。顺序停止与顺序搜索使用事后上界及其默认参考。这些量直接进入结果规则。各族之间变化的是参考如何获得，以及共享驱动能否执行对应轨迹，而不是参考是否存在。

Appendix C Implementation and record structure

附录 C 实现与记录结构

C.1 Reference computation

C.1 参考计算

The eleven reported families compute their references by a closed-form procedure, exact enumeration, dynamic programming, or numerical optimization. Inventory replenishment uses backward Bellman recursion over joint inventory states. Routing uses Held-Karp recursion over served-customer subsets, facility location enumerates open-facility subsets, and tardiness scheduling applies exponential recursion over job subsets. Linear programming uses HiGHS and the separable quadratic programme uses SLSQP through scipy.optimize. No language model is called during reference computation.

十一个报告族通过闭式程序、精确枚举、动态规划或数值优化计算参考。库存补货对联合库存状态做反向 Bellman 递归。路由对已服务客户子集做 Held-Karp 递归，设施选址枚举开放设施子集，拖期调度对作业子集做指数递归。线性规划用 HiGHS，可分二次规划经 scipy.optimize 用 SLSQP。参考计算期间不调用任何语言模型。

All later components receive the same parameter dictionary and reference pair.

后续所有组件收到同一参数字典与参考对。

Family registration.

A mechanism family is registered through a parameter sampler, reference procedure, and reusable realization or replay adapter. The reported families adapt established mathematical and operations-research formulations and solver routines for these components. Once registered, the same components serve repeated parameter and corpus draws, while environment generation and admission remain automated.

族注册。

机制族通过参数抽样器、参考程序，以及可复用实现或回放适配器注册。报告族为这些组件适配成熟数学与运筹表述及求解例程。一旦注册，同一组件服务反复的参数与语料抽样，而环境生成与准入保持自动化。

C.2 Setter passes

C.2 设定器遍次

The setter receives an independently sampled real-world source passage, its topical label, and the complete parameter vector. It emits an entity and relationship model, a backing database, at least ten executable tools, the player instruction, an executable default policy, and the required output format. Candidates that fail execution or interface alignment are refined or regenerated before admission. The source corpus spans 28 topical domains, and each environment is grounded in a separately sampled passage.

设定器收到独立抽样的真实世界源段落、其主题标签，以及完整参数向量。它发出实体与关系模型、底层数据库、至少十个可执行工具、玩家指令、可执行默认策略，以及所需输出格式。执行或接口对齐失败的候选在准入前被打磨或再生。源语料覆盖 28 个主题域，每个环境锚定在单独抽样的段落上。

Reusable family dynamics are supplied by the sampler where available, while the setter instantiates dynamics for families without that reusable implementation. The reference pair remains sampler-computed in every environment, regardless of which component wrote the dynamics.

可复用的族动力学在可用时由抽样器提供；对没有该可复用实现的族，由设定器实例化动力学。无论哪个组件写了动力学，参考对在每个环境中都由抽样器计算。

setter	family	emitted	replay	attempts	refined
Qwen3.6-35B-A3B	inventory DP	10/10	9/10	2.70	40%
routing	8/10	8/8	2.00	50%
tardiness	4/10	4/4	3.00	50%
Qwen3.7-Max	inventory DP	10/10	10/10	1.10	0%
routing	10/10	10/10	1.00	0%
tardiness	10/10	10/10	1.00	0%

Table 3: Setter-capacity feasibility check over three mechanism families. Attempts are averaged over emitted records, and refined is the share whose tool interface was revised during synthesis. The comparison uses the same families but not identical parameter draws.

设定器	族	发出	回放	尝试次数	打磨
Qwen3.6-35B-A3B	库存 DP	10/10	9/10	2.70	40%
路由	8/10	8/8	2.00	50%
拖期	4/10	4/4	3.00	50%
Qwen3.7-Max	库存 DP	10/10	10/10	1.10	0%
路由	10/10	10/10	1.00	0%
拖期	10/10	10/10	1.00	0%

表 3：三个机制族上的设定器容量可行性检查。尝试次数在发出记录上平均；打磨为合成期间工具接口被修订的份额。比较使用同一族但非相同参数抽样。

C.3 Setter capacity

C.3 设定器容量

In Tab 3, we test whether mechanism-first realization itself requires a frontier setter by using a frozen copy of the Qwen3.6-35B-A3B starting checkpoint, without environment-specific training, on ten requested records from each of inventory DP, routing, and tardiness scheduling. We compare its aggregate generation and verification outcomes with existing Qwen3.7-Max records from the same families. The parameter draws are not paired, so the comparison measures feasibility and realization burden rather than a causal model effect.

在表 3 中，我们用未经环境专用训练的冻结 Qwen3.6-35B-A3B 起始检查点副本，在库存 DP、路由与拖期调度上各请求十条记录，检验机制优先实现本身是否需要前沿设定器。我们将其总体生成与验证结果与同族已有的 Qwen3.7-Max 记录比较。参数抽样未配对，因此比较度量可行性与实现负担，而非因果模型效应。

The starting checkpoint emits 22 of 30 requested records, and 21 of its 22 emitted environments preserve the reference under compatible replay, giving an effective admission rate of 70%. Qwen3.7-Max emits and passes all 30 records, giving 100%, while reducing the mean number of attempts from 2.50 to 1.03 and the interface-refinement rate from 45.5% to zero. The smaller setter therefore establishes feasibility, while the stronger setter improves generation efficiency and interface alignment.

起始检查点发出 30 条请求中的 22 条，其中 21 条在兼容回放下保持参考，有效准入率 70%。Qwen3.7-Max 发出并通过全部 30 条，达 100%，同时把平均尝试次数从 2.50 降到 1.03，接口打磨率从 45.5% 降到零。因此较小设定器确立可行性，较强设定器改进生成效率与接口对齐。

These additional attempts erase the smaller setter’s per-token price advantage in the available reconstruction. At the lowest public rates we found, both setters cost roughly $\$0.01$–$\$0.03$ per admitted environment, and Qwen3.6-35B-A3B is not cheaper per admission despite its lower token rate.[^1]

这些额外尝试抹去了较小设定器在可用重建中的每词元价格优势。按我们找到的最低公开费率，两个设定器每个准入环境大约 $\$0.01$–$\$0.03$，且尽管 Qwen3.6-35B-A3B 词元费率更低，按准入计并不更便宜。[^1]

C.4 Environment and interpreter isolation

C.4 环境与解释器隔离

Each environment retains the complete sampled draw and reference pair inside its executable dynamics. The player-facing interface exposes only the declared catalogue, probe, commitment, and clock operations, without access to latent values or references. Every presentation provides the same state-independent computation utility, which cannot access environment internals. The written-out and agentic forms therefore differ in parameter access rather than in access to computation.

每个环境把完整抽样与参考对保留在其可执行动力学内。面向玩家的接口只暴露声明的目录、探询、承诺与时钟操作，不能访问潜在值或参考。每种呈现都提供同一与状态无关的计算效用，且无法访问环境内部。因此纸面与智能体形式的差别在于参数访问，而非计算访问。

C.5 Training implementation

C.5 训练实现

Training uses GRPO with 64 prompts per step and 16 retained rollouts per prompt, oversampled to 18 attempts. The advantage is the return minus its group mean, without standard-deviation normalization, a KL penalty, or a value model. One inner epoch consumes each batch at learning rate $2\times 10^{-6}$ and clip ratio $4\times 10^{-3}$. The reported checkpoint follows 34 optimizer steps, consumes 2,176 environments from the 2,200-environment training partition, and produces 34,816 scored rollouts.

训练使用 GRPO，每步 64 条提示，每提示保留 16 条 rollout，过采样至 18 次尝试。优势为回报减去其组均值，不做标准差归一化、KL 惩罚或价值模型。一个内 epoch 以学习率 $2\times 10^{-6}$、裁剪比 $4\times 10^{-3}$ 消费每批。报告检查点经 34 次优化器步，消费 2,200 环境训练划分中的 2,176 个，并产出 34,816 条已评分 rollout。

Tab. 9 reports the resource and optimization settings.

表 9 报告资源与优化设置。

Appendix D Representative generated environments

附录 D 代表性生成环境

This section traces three admitted records from their corpus setting and sampled mechanism through the generated interface and fixed evaluation. Two belong to the training split and one to an unseen evaluation family. Fig. 6 shows how inventory control, routing, and knapsack become operational tasks in hardware, education, and adaptive sports. Aggregate statistics in Tab. 1 characterize diversity across the admitted set, while these cases make the mechanism-to-interface mapping concrete.

本节追踪三条准入记录：从语料设定与抽样机制，到生成接口与固定评测。两条属训练划分，一条属未见评测族。图 6 展示库存控制、路由与背包如何在硬件、教育与适应性运动中成为操作任务。表 1 的总体统计刻画准入集上的多样性，而这些案例使机制到接口的映射具体化。

Figure 6: Representative generated environments across three corpus domains and mechanism families. Each panel shows the information-acquisition and commitment pattern exposed to the policy, the state carried across the horizon, and the exact solver-backed advantage over the fixed default.

图 6：三个语料域与机制族上的代表性生成环境。每幅面板展示暴露给策略的信息获取与承诺模式、跨视野携带的状态，以及相对固定默认的精确求解器支撑优势。

Tab. 4 records the reference procedure, raw default-to-optimum gap, and admission evidence for each example. All three have a positive gap, and compatible replay reproduces the reference through the generated interface.

表 4 记录每个例子的参考程序、原始默认到最优间隙，以及准入证据。三者都有正间隙，兼容回放经生成接口复现参考。

mechanism	reference computation	raw default $\rightarrow$ optimum	gap	admission evidence
inventory DP	backward DP over joint inventory states	$22.74\rightarrow 45.01$	$22.27$	executable; reference replay passed
routing	Held–Karp tours with DP over backlog	$-117.81\rightarrow 36.40$	$154.21$	executable; reference replay passed
two-dimensional knapsack	per-period knapsack DP with DP over the shared budget	$371.45\rightarrow 1{,}340.96$	$969.51$	executable; reference replay passed

Table 4: Solver and admission evidence for the representative records. The positive raw gaps establish that the instances are not solved by reproducing the fixed default policy.

机制	参考计算	原始默认 $\rightarrow$ 最优	间隙	准入证据
库存 DP	联合库存状态上的反向 DP	$22.74\rightarrow 45.01$	$22.27$	可执行；参考回放通过
路由	Held–Karp 巡游，并在积压上做 DP	$-117.81\rightarrow 36.40$	$154.21$	可执行；参考回放通过
二维背包	每期背包 DP，并在共享预算上做 DP	$371.45\rightarrow 1{,}340.96$	$969.51$	可执行；参考回放通过

表 4：代表记录的求解器与准入证据。正的原始间隙表明这些实例不能靠复现固定默认策略来解决。

D.1 Inventory control as hardware replenishment

D.1 作为硬件补货的库存控制

The inventory example is grounded in the hardware domain and rendered as six days of replenishment at the fictional Shenzhen Bay Hardware Distribution Center. Its generated instruction asks the agent to manage an Industrial Servo Motor, a Precision Optical Encoder, and a Thermal Management Unit. The visible description states that one truck carries at most six units per day, an order of any size incurs a joint setup cost of $37.69$, each product can hold and receive at most five units, unsold inventory carries forward, and unmet demand incurs a lost-sales penalty. This is operational language for one sampled joint-replenishment model, rather than an independently authored reward.

库存例子锚定在硬件域，渲染为虚构深圳湾硬件分拨中心的六日补货。生成指令要求智能体管理工业伺服电机、精密光电编码器与热管理单元。可见描述写明：一辆卡车每天最多运六件；任意规模订单产生 $37.69$ 的联合启动成本；每种产品最多持有与接收五件；未售库存结转；未满足需求产生失销惩罚。这是一次抽样联合补货模型的操作语言，而非独立编写的奖励。

Tab. 5 gives the model’s forecast and hidden demand trajectories. Each tuple in the economics column gives unit price, order cost, holding cost, and lost-sales penalty.

表 5 给出模型的预测与隐藏需求轨迹。经济列中的每个元组给出单价、订货成本、持有成本与失销惩罚。

product	economics	visible forecast	hidden demand
Servo Motor X12	$(9.37,2.61,0.89,0.46)$	$(5,5,5,2,4,1)$	$(6,5,4,2,4,2)$
Optical Encoder Z8	$(12.05,3.13,1.07,0.30)$	$(1,5,2,5,1,2)$	$(2,3,3,4,2,2)$
Thermal Unit T5	$(11.21,3.72,1.19,1.62)$	$(3,5,5,5,2,5)$	$(3,6,3,5,4,5)$

Table 5: The sampled quantities underlying the hardware inventory example. The forecast is available through the interface; probing a product reveals its full hidden demand trajectory.

产品	经济	可见预测	隐藏需求
Servo Motor X12	$(9.37,2.61,0.89,0.46)$	$(5,5,5,2,4,1)$	$(6,5,4,2,4,2)$
Optical Encoder Z8	$(12.05,3.13,1.07,0.30)$	$(1,5,2,5,1,2)$	$(2,3,3,4,2,2)$
Thermal Unit T5	$(11.21,3.72,1.19,1.62)$	$(3,5,5,5,2,5)$	$(3,6,3,5,4,5)$

表 5：硬件库存例子背后的抽样量。预测经接口可用；探询某产品揭示其完整隐藏需求轨迹。

Concretely, the sampler renders order quantities as action inputs, inventory balance as persistent state, capacity and setup costs as transition constraints, and the objective terms as accumulated utility. The generated interface exposes catalogue and inventory views, a per-product demand probe, a replenishment action, a day-advance action, and helpers for capacity, holding cost, setup cost, and horizon planning. The dynamics update inventory only after an order and day advance, charge the joint setup cost once on any ordering day, carry unsold stock forward, and make lost sales irreversible. The sampler computes a raw default value of $22.74$ and a raw optimum of $45.01$; after the implementation’s shift of origin, the stored reference pair is $u_{0}=0$ and $u^{*}=22.27$. The inventory reference driver reproduces the solver-derived outcome through the generated interface.

具体地，抽样器把订货量渲染为动作输入，把库存余额渲染为持续状态，把容量与启动成本渲染为转移约束，并把目标项累积为效用。生成接口暴露目录与库存视图、按产品需求探询、补货动作、日推进动作，以及容量、持有成本、启动成本与视野规划助手。动力学仅在订单与日推进后更新库存，在任意订货日只收取一次联合启动成本，未售库存结转，失销不可逆。抽样器计算原始默认值 $22.74$ 与原始最优 $45.01$；经实现的原点平移后，存储参考对为 $u_{0}=0$ 与 $u^{*}=22.27$。库存参考驱动经生成接口复现求解器导出结果。

D.2 Routing as education logistics

D.2 作为教育物流的路由

The generated interface supports probing, route dispatch, capacity and distance checks, backlog accounting, site profiles, and horizon planning. The exact solver combines Held–Karp routing with dynamic programming over backlog. Its raw default and optimum are $-117.81$ and $36.40$, yielding a non-trivial gap of $154.21$. The candidate executes successfully, and its reference policy reproduces the optimum through the generated tools.

生成接口支持探询、路线派遣、容量与距离检查、积压核算、站点画像与视野规划。精确求解器把 Held–Karp 路由与积压上的动态规划结合。其原始默认与最优为 $-117.81$ 与 $36.40$，得到非平凡间隙 $154.21$。候选成功执行，其参考策略经生成工具复现最优。

D.3 Knapsack allocation as adaptive-sports planning

D.3 作为适应性运动规划的背包分配

The generated interface supports item inspection, revenue probing, shelf-feasibility checks, subset selection, budget tracking, and day advancement. An exact two-level dynamic program computes the raw default and optimum, $371.45$ and $1{,}340.96$, giving a non-trivial gap of $969.51$. The candidate executes successfully, and its reference policy reproduces the optimum through the generated tools.

生成接口支持物品检查、收入探询、货架可行性检查、子集选择、预算跟踪与日推进。精确的两层动态规划计算原始默认与最优 $371.45$ 与 $1{,}340.96$，得到非平凡间隙 $969.51$。候选成功执行，其参考策略经生成工具复现最优。

Appendix E Construction, feedback, and optimization

附录 E 构造、反馈与优化

Sec. 3 separates the provenance of an interactive task from the feedback used to train on it. The separation yields four axes, summarized in Tab. 6. Environment construction determines where the dynamics and interface originate. Outcome provenance determines which object assigns meaning to successful behavior. Feedback localization determines where that assessment is attached to a trajectory. Policy optimization determines how the resulting quantities update a model. The four choices need not move together. Together with Eq. 1, they form a common map. A method is located by the source of its environment, the provenance and placement of its feedback, and the optimizer that consumes that feedback.

第 3 节把交互任务的来源与用于其上训练的反馈分开。这一分离得到四个轴，概括于表 6。环境构造决定动力学与接口源自何处。结果来源决定哪个对象赋予成功行为以意义。反馈定位决定该评估附着在轨迹何处。策略优化决定所得量如何更新模型。这四项选择不必一起移动。连同式 1，它们构成一张共同地图。一种方法由其环境来源、反馈的来源与放置，以及消费该反馈的优化器定位。

Construction	Environment source	Outcome provenance	Localization	Optimizer
Environment-first	existing or generated dynamics and interface	fitted reward model, rubric model, or judge	prefix, turn, or terminal	independent choice
Environment-first	existing or generated task or program	authored or generated checker attached to the task	usually terminal	independent choice
Mechanism-first	realization generated from sampled $\theta$	references computed from the same $\theta$ before realization	independent choice, terminal here	independent choice, GRPO here

Table 6: Four axes that are often conflated in descriptions of agent training. The representative rows show that feedback localization and policy optimization can change without changing the provenance from which an environment and its outcome were constructed.

构造	环境来源	结果来源	定位	优化器
环境优先	已有或生成的动力学与接口	拟合奖励模型、量规模型或评判	前缀、回合或终端	独立选择
环境优先	已有或生成的任务或程序	附着于任务的编写或生成检查器	通常终端	独立选择
机制优先	从抽样 $\theta$ 生成的实现	实现前由同一 $\theta$ 计算的参考	独立选择，此处为终端	独立选择，此处为 GRPO

表 6：智能体训练描述中常被混为一谈的四个轴。代表性行表明，反馈定位与策略优化可以改变，而不改变环境及其结果的构造来源。

E.1 Construction order and method coverage

E.1 构造顺序与方法覆盖

Within this map, the principal distinction introduced by VHD-Play is the construction order in Eq. 1. Its arrows record which artifact is available when the next is constructed. Along the environment-first branch, $E$ exists before either $\mathcal{R}_{E}$ is defined or sampled trajectories receive annotations $y_{\pi}$. Along the VHD-Play branch, solving $M(\theta)$ first yields $z_{\theta}$ and fixes $\mathcal{R}_{\theta}$. Realization then produces $D(\theta)$ and $E(\theta)$. The mechanism guides realization, while the precomputed standard provides a reference for checking it. The provenance, verification, and extension differences in Tab. 7 follow from this reversal.

在这张地图中，VHD-Play 引入的主要区分是式 1 的构造顺序。其箭头记录构造下一产物时哪些产物已可用。沿环境优先分支，$E$ 在定义 $\mathcal{R}_{E}$ 或给抽样轨迹赋标注 $y_{\pi}$ 之前就存在。沿 VHD-Play 分支，先求解 $M(\theta)$ 得到 $z_{\theta}$ 并固定 $\mathcal{R}_{\theta}$，再实现 $D(\theta)$ 与 $E(\theta)$。机制引导实现，预计算标准提供检查参考。表 7 中的来源、验证与扩展差异都来自这一反转。

For an environment $E$ that is already available, a learned evaluator uses $\hat{r}_{\psi}(\tau)=J_{\psi}(E,\tau)$, with validity resting on the fitted model or judge  (Christiano et al., 2017; Ouyang et al., 2022; Norman et al., 2026). An executable verifier uses $r(\tau)=V_{E}(\tau)$, where $V_{E}$ may be authored or generated after the task, while a sampled trajectory may instead receive an annotation $y_{\pi}$. These choices change the provenance and placement of feedback while retaining $E$ as the upstream object.

对已可用的环境 $E$，学习型评测器用 $\hat{r}_{\psi}(\tau)=J_{\psi}(E,\tau)$，有效性依赖拟合模型或评判（Christiano et al., 2017; Ouyang et al., 2022; Norman et al., 2026）。可执行验证器用 $r(\tau)=V_{E}(\tau)$，其中 $V_{E}$ 可在任务之后编写或生成；抽样轨迹也可改为接收标注 $y_{\pi}$。这些选择改变反馈的来源与放置，同时仍把 $E$ 保留为上游对象。

Rubric-based rewards refine the learned-evaluator case. A rubric supplies criteria $q_{1},\ldots,q_{m}$, a grader assigns criterion scores $J_{\psi,j}(E,\tau,q_{j})$, and an aggregation rule forms $\hat{r}_{\mathrm{rubric}}(\tau)=\sum_{j=1}^{m}w_{j}J_{\psi,j}(E,\tau,q_{j})$  (Gunjal et al., 2025). This changes the structure and coverage of feedback while its semantics still come from the rubric, grader, and aggregation rule introduced for $E$. Thus outcome type and feedback localization can vary without changing the construction dependency in Eq. 1. Learned reward or judge pipelines, rubric-based supervision, trajectory annotation, and executable verifiers are therefore represented by different settings of the same four axes. Mechanism-first construction adds the alternate upstream branch in Eq. 1. The change concerns construction provenance rather than feedback localization or policy optimization.

基于量规的奖励细化学习型评测器情形。量规提供准则 $q_{1},\ldots,q_{m}$，打分器给出准则分数 $J_{\psi,j}(E,\tau,q_{j})$，聚合规则形成 $\hat{r}_{\mathrm{rubric}}(\tau)=\sum_{j=1}^{m}w_{j}J_{\psi,j}(E,\tau,q_{j})$（Gunjal et al., 2025）。这改变反馈的结构与覆盖，但其语义仍来自为 $E$ 引入的量规、打分器与聚合规则。因此结果类型与反馈定位可以变化，而不改变式 1 的构造依赖。学习型奖励或评判管线、基于量规的监督、轨迹标注与可执行验证器，因而由同一四个轴的不同设置表示。机制优先构造增加式 1 中的另一上游分支。变化关乎构造来源，而非反馈定位或策略优化。

E.2 A released environment-first comparison

E.2 已发布的环境优先对照

EnvScaler provides a strong public instance of the environment-first branch in Eq. 1  (Song et al., 2026). It first generates a reusable program skeleton containing state, rules, and tools, then instantiates its database and task. This yields the executable environment $E$, whose released training scenarios contain many API-based interactions. EnvScaler subsequently generates the checklist and final-state validators that define the scenario-specific outcome rule $\mathcal{R}_{E}$. Its construction therefore follows $E\rightarrow\mathcal{R}_{E}$ in our definition. Both substrates provide stateful interaction, diverse APIs, and computed rewards. VHD-Play instead fixes its objective and references before realizing the environment. Tab. 7 compares the two orders directly.

EnvScaler 是式 1 环境优先分支的一个强公开实例（Song et al., 2026）。它先生成含状态、规则与工具的可复用程序骨架，再实例化其数据库与任务。这得到可执行环境 $E$，其发布的训练场景含大量基于 API 的交互。EnvScaler 随后生成清单与终态验证器，定义场景专用结果规则 $\mathcal{R}_{E}$。因此其构造按我们的定义遵循 $E\rightarrow\mathcal{R}_{E}$。两种基底都提供有状态交互、多样 API 与计算奖励。VHD-Play 则在实现环境之前固定其目标与参考。表 7 直接比较两种顺序。

Dimension	Mechanism-first (VHD-Play)	Environment-first (EnvScaler)
Construction order	Solve $M(\theta)$ to fix $z_{\theta}$ and $\mathcal{R}_{\theta}$ before realizing $D(\theta)$ and wrapping $E(\theta)$.	Generate a stateful program skeleton, database, and task, then derive the checklist and final-state validators.
Dynamics and APIs	The mechanism fixes transition semantics. The setter renders corpus-grounded relational data and executable APIs.	An LLM generates the state schema, business rules, and Python tools.
Outcome signal	Realized terminal utility is normalized between a default and a precomputed reference from the same mechanism.	Reward is the fraction of LLM-generated Boolean checklist functions satisfied by the final state.
Verification	Reference replay compares generated dynamics with outcomes computed before realization.	AST checks and a 100-round testing–checking agent loop filter skeletons by execution behavior.
Information boundary	The builder sees the draw and references. The player sees interface views, and three forms vary this boundary.	State is hidden behind tools. A conversation mode reveals task information through a simulated user.
Extension unit	Register a sampler, reference procedure, and reusable realization once per family. Then resample parameters and corpus seeds or increase size and horizon.	Generate databases, tasks, and validators within a skeleton. A new skeleton adds discovery, synthesis, and assessment.
Reusable setup	Standard mathematical formulations and solution procedures are already available. Integrating a family requires one reusable sampler and realization/replay adapter.	Each program skeleton is generated and assessed. EnvScaler reports approximately $1.024 per skeleton (Song et al., 2026).
Human input	Once a family is registered, draws, realization, and admission require no per-environment labels.	The main generation loop uses no per-scenario human labels. Manual judgments are reported for comparison with a model-based quality audit; their scale and labor are unspecified.
Marginal cost	Approximately $0.01–$0.03 per admitted environment under the reconstruction in App. I.	$0.0635 per scenario, or approximately$0.08–$0.09 after amortizing reported skeleton generation (Song et al., 2026).

Table 7: Construction-order comparison. Both substrates provide executable state, diverse APIs, and computed terminal feedback. EnvScaler constructs the executable substrate and task before generating its validators. VHD-Play fixes the outcome standard before realization, yielding different provenance, verification, and extension paths.

维度	机制优先（VHD-Play）	环境优先（EnvScaler）
构造顺序	先求解 $M(\theta)$ 以固定 $z_{\theta}$ 与 $\mathcal{R}_{\theta}$，再实现 $D(\theta)$ 并包装 $E(\theta)$。	生成有状态程序骨架、数据库与任务，再导出清单与终态验证器。
动力学与 API	机制固定转移语义。设定器渲染语料锚定的关系数据与可执行 API。	LLM 生成状态模式、业务规则与 Python 工具。
结果信号	实现终端效用在同一机制的默认与预计算参考之间归一化。	奖励为终态满足的 LLM 生成布尔清单函数比例。
验证	参考回放比较生成动力学与实现前计算的结果。	AST 检查与 100 轮测试–检查智能体环按执行行为过滤骨架。
信息边界	构建器看到抽样与参考。玩家看到接口视图，三种形式变化该边界。	状态隐藏在工具后。对话模式经模拟用户揭示任务信息。
扩展单元	每族一次注册抽样器、参考程序与可复用实现。再重抽样参数与语料种子，或增大规模与视野。	在骨架内生成数据库、任务与验证器。新骨架增加发现、合成与评估。
可复用设置	标准数学表述与求解程序已可用。接入一族只需一个可复用抽样器与实现/回放适配器。	每个程序骨架被生成并评估。EnvScaler 报告约每骨架 $1.024（Song et al., 2026）。
人工输入	族一旦注册，抽样、实现与准入无需逐环境标签。	主生成环不使用逐场景人工标签。报告人工判断用于与基于模型的质量审计比较；其规模与劳动未说明。
边际成本	按附录 I 重建，每个准入环境约 $0.01–$0.03。	每场景 $0.0635，或摊销报告骨架生成后约$0.08–$0.09（Song et al., 2026）。

表 7：构造顺序比较。两种基底都提供可执行状态、多样 API 与计算终端反馈。EnvScaler 先构造可执行基底与任务，再生成验证器。VHD-Play 在实现前固定结果标准，带来不同的来源、验证与扩展路径。

Among the systems reviewed in Sec. 2, EnvScaler’s combination of executable skeletons, task scenarios, validator code, an RL corpus, training infrastructure, and trained checkpoints makes it one of the most complete public environment-first baselines available (Cai et al., 2025; Song et al., 2026; Tu et al., 2026).

在第 2 节综述的系统中，EnvScaler 把可执行骨架、任务场景、验证器代码、RL 语料、训练基础设施与训练检查点合在一起，是目前最完整的公开环境优先基线之一（Cai et al., 2025; Song et al., 2026; Tu et al., 2026）。

Tab. 7 compares the closest marginal units, an EnvScaler scenario and a VHD-Play admitted environment. Both pipelines aim to replace recurring human trajectory labels with executable signals. EnvScaler further calibrates a model-based quality audit against manual judgments, while VHD-Play checks generated behavior against precomputed references.

表 7 比较最接近的边际单元：一个 EnvScaler 场景与一个 VHD-Play 准入环境。两条流水线都旨在用可执行信号替代反复的人工轨迹标签。EnvScaler 进一步用人工判断校准基于模型的质量审计，而 VHD-Play 对照预计算参考检查生成行为。

To compare the resulting training substrates under a common learner, we adapt 2,200 released EnvScaler RL scenarios drawn across 51 program skeletons while preserving their native environments and checklist rewards. The control starts from the same Qwen3.6-35B-A3B checkpoint and uses the same GRPO optimizer, batch size, rollout group size, and 34-step budget as VHD-Play. The released RL scenarios form its training substrate, and step 34 is fixed before external evaluation. Benchmark instances and inference settings are then shared across the two trained checkpoints. BFCL, TravelBench, and E-Commerce Bench are external to both training corpora.

为在共同学习者下比较所得训练基底，我们适配跨 51 个程序骨架抽取的 2,200 个已发布 EnvScaler RL 场景，同时保留其原生环境与清单奖励。对照从同一 Qwen3.6-35B-A3B 检查点起步，并使用与 VHD-Play 相同的 GRPO 优化器、批大小、rollout 组大小与 34 步预算。已发布 RL 场景构成其训练基底，在外部评测前固定步 34。随后两个训练检查点共享基准实例与推理设置。BFCL、TravelBench 与 E-Commerce Bench 对两个训练语料都是外部的。

EnvScaler’s original experiments show substantial gains for smaller policies from lower starting scores. Its Qwen3-8B SFT-and-RL checkpoint raises BFCL-v3 Multi-Turn from 28.88 to 41.88 (Song et al., 2026). Our matched control evaluates a different operating point with a stronger 35B starting policy. Training raises its native reward over 34 steps, improves TravelBench plan quality from 0.700 to 0.753, and brings all five storefront runs to day 365. Aggregate transfer is less uniform. Its BFCL ten-cell mean is 58.56 against 61.25 for Base, while mean storefront balance remains close to Base at 52,661 versus 54,294. Under the same 35B starting checkpoint and update budget, VHD-Play improves all three external aggregates. Fig. 7 and Tab. 8 summarize the comparison. The two regimes suggest that the large gains from a lower 8B baseline do not automatically translate into uniform gains when continuing from a stronger 35B policy.

EnvScaler 的原始实验显示，较小策略从更低起始分数获得大幅增益。其 Qwen3-8B SFT-and-RL 检查点把 BFCL-v3 Multi-Turn 从 28.88 提到 41.88（Song et al., 2026）。我们的匹配对照评估的是更强 35B 起始策略下的不同工作点。训练在 34 步内抬高其原生奖励，把 TravelBench 计划质量从 0.700 提到 0.753，并使五次店面运行都到第 365 天。总体迁移不够均匀。其 BFCL 十单元均值为 58.56，对照 Base 的 61.25；平均店面余额仍接近 Base，为 52,661 对 54,294。在同一 35B 起始检查点与更新预算下，VHD-Play 改进全部三个外部总体指标。图 7 与表 8 概括比较。两种体制表明：从更低 8B 基线的大幅增益，并不自动转化为从更强 35B 策略继续训练时的均匀增益。

Figure 7: Matched environment-first control. Panel (a) reports EnvScaler’s native training reward over 34 GRPO steps. Panel (b) reports change from Base on ten BFCL cells. Panel (c) shows five 365-day storefront runs per model, with horizontal segments marking means and crosses marking bankruptcy. Panel (d) reports TravelBench plan quality.

图 7：匹配的环境优先对照。面板 (a) 报告 EnvScaler 在 34 次 GRPO 步上的原生训练奖励。面板 (b) 报告十个 BFCL 单元相对 Base 的变化。面板 (c) 显示每模型五次 365 天店面运行，横线段标均值，叉号标破产。面板 (d) 报告 TravelBench 计划质量。

Benchmark	Cell or measure	Base	EnvScaler	VHD-Play
BFCL V4	Multi-turn base	65.00	57.00	70.50
Missing function	44.50	43.00	50.50
Missing parameter	50.00	38.50	51.00
Long context	55.50	48.00	55.50
Key-value memory	60.00	61.29	64.52
Vector memory	65.16	70.97	67.74
Summary memory	59.35	63.23	59.35
Web search	57.00	49.50	59.00
Relevance	75.00	77.08	81.25
Irrelevance	80.97	77.04	81.52
	Ten-cell mean	61.25	58.56	64.08
TravelBench	Plan score	0.700	0.753	0.794
Storefront	Mean ending balance	54,294	52,661	182,844
Day-365 completion	4/5	5/5	5/5
Bankruptcies	1/5	0/5	0/5

Table 8: Matched environment-first control. BFCL reports all ten interaction-focused cells. Both trained checkpoints start from the same base model and use the same GRPO setting, 2,200 training records, and 34-step update budget.

基准	单元或度量	Base	EnvScaler	VHD-Play
BFCL V4	Multi-turn base	65.00	57.00	70.50
Missing function	44.50	43.00	50.50
Missing parameter	50.00	38.50	51.00
Long context	55.50	48.00	55.50
Key-value memory	60.00	61.29	64.52
Vector memory	65.16	70.97	67.74
Summary memory	59.35	63.23	59.35
Web search	57.00	49.50	59.00
Relevance	75.00	77.08	81.25
Irrelevance	80.97	77.04	81.52
	十单元均值	61.25	58.56	64.08
TravelBench	计划分数	0.700	0.753	0.794
Storefront	平均期末余额	54,294	52,661	182,844
第 365 天完成	4/5	5/5	5/5
破产	1/5	0/5	0/5

表 8：匹配的环境优先对照。BFCL 报告全部十个交互向单元。两个训练检查点从同一基座模型起步，使用相同 GRPO 设置、2,200 条训练记录与 34 步更新预算。

E.3 Feedback and optimization

E.3 反馈与优化

Mechanism-first construction determines the provenance of an environment and its evaluative standard. It does not prescribe when feedback is attached to a trajectory or how that feedback updates a policy. Once the construction from $M(\theta)$ has supplied $E(\theta)$, $z_{\theta}$, and $\mathcal{R}_{\theta}$, the downstream choices can be written as

机制优先构造决定环境及其评测标准的来源。它不规定反馈何时附着到轨迹，也不规定该反馈如何更新策略。一旦从 $M(\theta)$ 的构造已提供 $E(\theta)$、$z_{\theta}$ 与 $\mathcal{R}_{\theta}$，下游选择可写为

$$\begin{gathered}y_{\pi}^{\mathrm{term}}=\mathcal{R}{\theta}(\tau{\pi,\leq T};z_{\theta}),\qquad y_{\pi,t}^{\mathrm{proc}}=\mathcal{R}{\theta,t}(\tau{\pi,\leq t};z_{\theta}),\[-1.0pt]
\pi^{+}=\operatorname{Update}(\pi;\tau_{\pi},y_{\pi}).\end{gathered} \tag{5}$$

Here $y_{\pi}$ denotes either the terminal scalar or the sequence of process signals. Outcome models, process reward models, turn-level checks, and generated verifiers instantiate different choices of $\mathcal{R}$ and its input (Cobbe et al., 2021; Lightman et al., 2023; Liu et al., 2025; Wei et al., 2025). The update may likewise use PPO, GRPO, or another optimizer (Schulman et al., 2017; Shao et al., 2024). The reported realization sets $y_{\pi}$ to the normalized scalar terminal reward in Eq. 4 and instantiates $\operatorname{Update}$ with GRPO. App. C.5 gives the corresponding objective and run configuration.

此处 $y_{\pi}$ 表示终端标量或过程信号序列。结果模型、过程奖励模型、回合级检查与生成验证器实例化 $\mathcal{R}$ 及其输入的不同选择（Cobbe et al., 2021; Lightman et al., 2023; Liu et al., 2025; Wei et al., 2025）。更新同样可用 PPO、GRPO 或其他优化器（Schulman et al., 2017; Shao et al., 2024）。报告实现把 $y_{\pi}$ 设为式 4 的归一化标量终端奖励，并用 GRPO 实例化 $\operatorname{Update}$。附录 C.5 给出对应目标与运行配置。

Appendix F Evaluation settings

附录 F 评测设置

Tab. 9 reports the resource and policy-optimization settings. We set the maximum episode length to 1,000 turns, the per-turn generation cap to 16,384 tokens, the cumulative generation cap to 120,000 tokens, the prompt cap to 8,192 tokens, and the managed context window to 65,536 tokens. We count one sampled mechanism, its realized interface, and its fixed outcome rule as one environment instance. The reported set contains 3,300 such environments. Of these, 2,200 form the training partition, 300 provide held-out evaluation for the three training families, and 800 provide evaluation for the eight unseen families. Each evaluation family contributes 100 environments. Across the trained checkpoint’s agentic generated-family exports, episodes have median $31$, mean $59.7$, and 95th percentile $211$ assistant turns.

表 9 报告资源与策略优化设置。我们将最大回合长度设为 1,000 轮，每轮生成上限 16,384 词元，累计生成上限 120,000 词元，提示上限 8,192 词元，托管上下文窗口 65,536 词元。我们把一次抽样机制、其实现接口与其固定结果规则计为一个环境实例。报告集合含 3,300 个此类环境。其中 2,200 构成训练划分，300 为三个训练族提供持出评测，800 为八个未见族提供评测。每个评测族贡献 100 个环境。在训练检查点的智能体生成族导出中，回合的中位数为 $31$，均值为 $59.7$，第 95 百分位为 $211$ 次助手轮次。

Setting	Value
Resource Settings
Maximum Turns per Episode	1,000
Generation Cap per Turn	16,384 tokens
Maximum Cumulative Generation	120,000 tokens
Prompt Cap	8,192 tokens
Managed Context Window	65,536 tokens
Observed Assistant Turns, Median / Mean / P95	31 / 59.7 / 211
Policy Optimization
Algorithm	GRPO
Prompts per Step	64
Rollouts per Prompt	16 retained, oversampled to 18
Learning Rate	$2\times 10^{-6}$
Clip Ratio	$4\times 10^{-3}$
Epochs per Batch	1
KL Penalty	none
Advantage Standardization	none
Reported Checkpoint	step 34

Table 9: Resource and policy-optimization settings used in the reported runs.

设置	值
资源设置
每回合最大轮次	1,000
每轮生成上限	16,384 词元
最大累计生成	120,000 词元
提示上限	8,192 词元
托管上下文窗口	65,536 词元
观测助手轮次，中位 / 均值 / P95	31 / 59.7 / 211
策略优化
算法	GRPO
每步提示数	64
每提示 rollout	保留 16，过采样至 18
学习率	$2\times 10^{-6}$
裁剪比	$4\times 10^{-3}$
每批 epoch	1
KL 惩罚	无
优势标准化	无
报告检查点	步 34

表 9：报告运行所用的资源与策略优化设置。

Appendix G Mechanism families

附录 G 机制族

Tab. 10 catalogues the eleven families the paper draws instances from. Each row gives the readable name, the mathematical problem its sampler draws, the decision structure an episode presents, the procedure that computes $u^{*}(\theta)$, and the evaluation band. Across the optimization families the default policy behind $u_{0}(\theta)$ follows one recipe. Act on the sticker numbers, split any shared budget evenly over the horizon, never probe, and let the true parameters settle what that earns. The linear and the quadratic rows call the same numerical routine as their reference optimum, one period at a time on the visible coefficients, while facility location enumerates open subsets on the advertised serving costs. The reference low end is therefore a solved decision taken on misleading data. Outside that group the default is set per family. Bargaining concedes ten percent per round, sequential stopping commits to the first sealed option, and sequential search inspects everything before claiming the best of what it saw.

表 10 编目论文抽取实例的十一个族。每行给出可读名称、抽样器抽取的数学问题、回合呈现的决策结构、计算 $u^{*}(\theta)$ 的程序，以及评测带。在优化族中，$u_{0}(\theta)$ 背后的默认策略遵循同一配方：按标价数字行动，把任何共享预算在视野上均分，从不探询，让真实参数决定那能挣多少。线性与二次行调用与其参考最优相同的数值例程，按可见系数逐期求解；设施选址则在标示服务成本上枚举开放子集。因此参考低端是在误导数据上做出的已解决策。该组之外，默认按族设定：议价每轮让步百分之十，顺序停止承诺第一个密封选项，顺序搜索在认领所见最佳之前检查全部。

family	mathematical problem	decision structure	reference optimum computed by	band
inventory DP	multi-product capacitated joint replenishment	agentic horizon, inventory carries between periods, one shared order capacity, joint setup charged once per ordering period, demand revealed by probe	backward dynamic program over the joint integer inventory state	in-domain held out
routing	multi-period capacitated vehicle routing with backlog	agentic; one vehicle leaves the depot each period and tours the chosen subset, unserved demand carries as backlog and is charged, demand revealed by probe	exact Held-Karp tour per subset, with a dynamic program over the backlog carry	in-domain held out
negotiation	sequential bilateral bargaining under a cash ledger	agentic; about 25 deals against a counterpart kernel, bankroll carries, per-deal round budget, bankruptcy at zero	backward induction per deal, then the best attainable session total	in-domain held out
knapsack	multi-period two-dimensional 0/1 knapsack with a shared budget	agentic; a shelf-space cap and a stocking-cost cap each period, one procurement budget over the whole horizon, probe reveals an item’s profit trajectory	two-dimensional knapsack dynamic program per period, plus a dynamic program over the budget split	near-OOD
linear programming	multi-period linear program with a shared budget	agentic; constraint matrix disclosed, drifting objective coefficients hidden and revealed by probe	the stacked all-periods program solved by HiGHS	near-OOD
quadratic programming	multi-period separable concave quadratic program with a shared budget	agentic; curvature disclosed, drifting linear coefficients hidden and revealed by probe	sequential quadratic programming on the stacked coupled program, global under concavity	near-OOD
sequential stopping	optimal stopping over sealed options	strictly sequential; reveal the next option or commit the last revealed one, and a passed option cannot be revisited	the largest value in the sequence, a hindsight ceiling no online policy attains	far-OOD, distinct decision structure
sequential search	Weitzman search with per-option inspection costs	inspect any option at its own charge, then claim one, and every inspection is paid for	the best single inspection chosen in hindsight, again not attainable online	far-OOD, distinct decision structure
facility location	uncapacitated facility location, one instance per period	agentic with a probe budget of six tenths of the items; opening costs and demands disclosed, serving costs hidden and drifting, periods independent	enumeration over open subsets, each customer assigned to its cheapest open facility	far-OOD, probe-budgeted
scheduling	single machine, weighted completion time	agentic with a probe budget; processing times disclosed, weights hidden and drifting, a full permutation committed each period	Smith’s rule, a sort by weight over processing time, provably optimal	far-OOD, probe-budgeted
scheduling (tardiness)	single machine, weighted tardiness, NP-hard	agentic with a probe budget; processing times and due dates disclosed, weights hidden and drifting	exact dynamic program over job subsets	far-OOD, probe-budgeted

Table 10: Every mechanism family the paper uses, with the mathematics it samples, the decision structure an episode presents, the procedure that computes the reference optimum, and the evaluation band. Sequential stopping and sequential search are scored against a hindsight bound no online policy attains, so their rows compare the two arms rather than measure distance to an attainable optimum.

族	数学问题	决策结构	参考最优计算方法	带
库存 DP	多产品有容量联合补货	智能体视野，库存跨期结转，一个共享订货容量，联合启动在每订货期收取一次，需求经探询揭示	联合整数库存状态上的反向动态规划	域内持出
路由	带积压的多期有容量车辆路径	智能体；每期一辆车离开仓库巡游所选子集，未服务需求作为积压结转并计费，需求经探询揭示	每子集精确 Held-Karp 巡游，并对积压结转做动态规划	域内持出
谈判	现金账本下的顺序双边议价	智能体；约 25 笔交易对对手核，资金结转，每笔交易有回合预算，零时破产	每笔交易逆向归纳，再取可达会话总和最优	域内持出
背包	带共享预算的多期二维 0/1 背包	智能体；每期货架空间上限与备货成本上限，整视野一个采购预算，探询揭示物品利润轨迹	每期二维背包动态规划，再对预算分割做动态规划	近 OOD
线性规划	带共享预算的多期线性规划	智能体；约束矩阵公开，漂移目标系数隐藏并经探询揭示	用 HiGHS 求解堆叠的全期程序	近 OOD
二次规划	带共享预算的多期可分凹二次规划	智能体；曲率公开，漂移线性系数隐藏并经探询揭示	在堆叠耦合程序上做序列二次规划，凹性下全局	近 OOD
顺序停止	密封选项上的最优停止	严格顺序；揭示下一选项或承诺上次已揭示者，已过选项不可再访	序列中最大值，在线策略不可达的事后上界	远 OOD，决策结构不同
顺序搜索	带每选项检查成本的 Weitzman 搜索	按各自费用检查任意选项，再认领一个，每次检查都付费	事后选中的单次最佳检查，同样在线不可达	远 OOD，决策结构不同
设施选址	无容量设施选址，每期一个实例	智能体，探询预算为物品的十分之六；开放成本与需求公开，服务成本隐藏并漂移，期独立	枚举开放子集，每位客户分配到最便宜开放设施	远 OOD，探询有预算
调度	单机，加权完工时间	智能体，有探询预算；加工时间公开，权重隐藏并漂移，每期提交完整排列	Smith 规则，按权重/加工时间排序，可证最优	远 OOD，探询有预算
调度（拖期）	单机，加权拖期，NP-hard	智能体，有探询预算；加工时间与交货期公开，权重隐藏并漂移	作业子集上的精确动态规划	远 OOD，探询有预算

表 10：论文使用的每一个机制族，含其抽样数学、回合呈现的决策结构、计算参考最优的程序，以及评测带。顺序停止与顺序搜索对照在线策略不可达的事后界打分，因此其行比较两臂，而非度量到可达最优的距离。

Appendix H Additional experimental results

附录 H 补充实验结果

H.1 Training dynamics

H.1 训练动态

Fig. 8 plots its per-family rewards for the reported training families against optimizer step. Inventory DP climbs from $0.408$ to $0.904$ and routing from $0.150$ to $0.898$, while negotiation reaches $0.470$ from a near-zero start.

图 8 相对优化器步绘制报告训练族的各族奖励。库存 DP 从 $0.408$ 升到 $0.904$，路由从 $0.150$ 升到 $0.898$，谈判从接近零起步达到 $0.470$。

The curve establishes only that optimization proceeded. Reward is defined on generated environments, so its rise cannot distinguish learning a stateful policy from fitting training instances. Step $34$ supplies every trained-model number in the paper, while held-out evaluation provides the transfer evidence.

曲线只确立优化在进行。奖励定义在生成环境上，其上升不能区分是在学有状态策略还是在拟合训练实例。步 $34$ 提供文中每一个训练模型数字，而持出评测提供迁移证据。

Figure 8: Per-family training reward through the reported checkpoint. These values are measured on training instances and document optimization rather than transfer; held-out evaluation provides the transfer evidence.

图 8：到报告检查点为止的各族训练奖励。这些值在训练实例上测量，记录的是优化而非迁移；持出评测提供迁移证据。

H.2 Generated-family results

H.2 生成族结果

Fig. 1(a) summarizes the three-arm written-out-to-agentic comparison; the corresponding family-level cells appear in Tab. 11. Tab. 12 gives the complete base-to-trained grid behind the transfer bands in Fig. 1(b). Reading its agentic columns downward recovers the per-family results of Sec. 5.1, while reading across a row compares written-out and agentic forms.

图 1(a) 概括三臂纸面到智能体比较；对应的族级单元见表 11。表 12 给出图 1(b) 迁移带背后完整的 Base 到训练网格。向下读其智能体列可恢复第 5.1 节的各族结果；横向读一行则比较纸面与智能体形式。

	Base	VHD-Play (step 34)	Qwen3.7-Max
family	written	agentic	written	agentic	written	agentic
inventory DP	0.976	0.449	0.999	0.912	1.000	0.489
knapsack	0.984	0.117	0.996	0.843	1.000	0.525
linear programming	0.978	0.143	1.000	0.661	1.000	0.399
quadratic programming	0.975	0.162	1.000	0.795	1.000	0.588
routing	0.897	0.148	0.965	0.866	0.996	0.142
mean	0.962	0.204	0.992	0.815	0.999	0.429
shortfall	$0.758$	$\mathbf{0.177}$	$0.571$

Table 11: Five optimization families in written-out and agentic form, scored by Eq. 4. Written out minus agentic defines the shortfall. Training reduces the mean shortfall from $0.758$ to $0.177$ while retaining near-ceiling written-out performance. Bold marks the better of the base and trained checkpoints.

	Base	VHD-Play（步 34）	Qwen3.7-Max
族	纸面	智能体	纸面	智能体	纸面	智能体
库存 DP	0.976	0.449	0.999	0.912	1.000	0.489
背包	0.984	0.117	0.996	0.843	1.000	0.525
线性规划	0.978	0.143	1.000	0.661	1.000	0.399
二次规划	0.975	0.162	1.000	0.795	1.000	0.588
路由	0.897	0.148	0.965	0.866	0.996	0.142
均值	0.962	0.204	0.992	0.815	0.999	0.429
缺口	$0.758$	$\mathbf{0.177}$	$0.571$

表 11：五个优化族在纸面与智能体形式上的结果，按式 4 计分。纸面减智能体定义缺口。训练把平均缺口从 $0.758$ 降到 $0.177$，同时保持接近上限的纸面表现。粗体标出 Base 与训练检查点中更好者。

	written out ($F$)	agentic ($A$)
family	Base	VHD-Play	Base	VHD-Play	$\Delta A$
inventory DP	0.976	0.999	0.449	0.912	$+0.463$
knapsack	0.984	0.996	0.117	0.843	$+0.726$
linear programming	0.978	1.000	0.143	0.661	$+0.518$
quadratic programming	0.975	1.000	0.162	0.795	$+0.633$
routing	0.897	0.965	0.148	0.866	$+0.718$
facility location	1.000	1.000	0.390	0.646	$+0.256$
scheduling	1.000	1.000	0.522	0.607	$+0.085$
scheduling (tardiness)	1.000	1.000	0.390	0.537	$+0.147$
negotiation	n/a	n/a	0.047	0.555	$+0.509$
sequential stopping	n/a	n/a	0.350	0.767	$+0.417$
sequential search	n/a	n/a	0.310	0.376	$+0.066$
mean, five-family subset	0.962	0.992	0.204	0.815	$\mathbf{+0.611}$

Table 12: Every reported generated-family result for the base model and trained checkpoint. Written out exposes all parameters in the prompt and permits Python-tool use, whereas agentic form exposes them through the environment. Negotiation, sequential stopping and sequential search were evaluated only in agentic form, so their written-out cells are absent rather than zero. The mean row covers the five families evaluated in both forms.