CollabLLM: From Passive Responders to Active Collaborators

CollabLLM：从被动应答者到主动协作者

Shirley Wu Affiliation:  Stanford University    Michel Galley Affiliation:  Microsoft    Baolin Peng Affiliation:  Microsoft    Hao Cheng Affiliation:  Microsoft    Gavin Li Affiliation:  Stanford University    Yao Dou Affiliation:  Georgia Institute of Technology    Weixin Cai Affiliation:  Stanford University    James Zou Affiliation:  Stanford University    Jure Leskovec Affiliation:  Stanford University    Jianfeng Gao Correspondence to: shirwu@cs.stanford.edu Correspondence to: mgalley@microsoft.com Affiliation:  Microsoft

Abstract

摘要

Large Language Models are typically trained with next-turn rewards, limiting their ability to optimize for long-term interaction. As a result, they often respond passively to ambiguous or open-ended user requests, failing to help users reach their ultimate intents and leading to inefficient conversations. To address these limitations, we introduce CollabLLM, a novel and general training framework that enhances multiturn human-LLM collaboration. Its key innovation is a collaborative simulation that estimates the long-term contribution of responses using Multiturn-aware Rewards. By reinforcement fine-tuning these rewards, CollabLLM goes beyond responding to user requests, and actively uncovers user intent and offers insightful suggestions—a key step towards more human-centered AI. We also devise a multiturn interaction benchmark with three challenging tasks such as document creation. CollabLLM significantly outperforms our baselines with averages of 18.5% higher task performance and 46.3% improved interactivity by LLM judges. Finally, we conduct a large user study with 201 judges, where CollabLLM increases user satisfaction by 17.6% and reduces user spent time by 10.4%.

大语言模型通常按下一轮奖励训练，难以面向长期交互做优化。结果是：面对含糊或开放式请求，模型往往被动应答，帮不上用户真正意图，对话也更低效。为此我们提出 CollabLLM——一套面向多轮人机协作的通用训练框架。核心创新是协作仿真：用多轮感知奖励（Multiturn-aware Rewards）估计回复的长期贡献，再经强化微调，使模型不只应答请求，还能主动澄清意图、给出有洞见的建议——向更以人为中心的 AI 迈出关键一步。我们还设计了含文档撰写等三项高难度任务的多轮交互基准。按 LLM 评判，CollabLLM 相对基线平均任务表现高 18.5%，交互性高 46.3%。最后在 201 名评测者的真人研究中，用户满意度提升 17.6%，耗时减少 10.4%。

Keywords:

Figure 1: CollabLLM Framework: Given a context 1, the model generates a response 2 to maximize long-term collaboration gains, termed Multiturn-aware Rewards (MR). During training, MRs are estimated via 3 collaborative simulation, which forward-samples conversations with simulated users. Finally, 4 reinforcement fine-tuning is applied using the MRs.

关键词：

图 1：CollabLLM 框架。给定上下文 1，模型生成回复 2，以最大化长期协作收益，即多轮感知奖励（MR）。训练时，MR 经由 3 协作仿真估计——向前采样与模拟用户的对话。最后用 MR 做 4 强化微调。

Figure 2: Real examples from CollabLLM and non-collaborative LLM fine-tuing. (a) Non-collaborative LLM fine-tuing relies single-turn rewards on immediate responses, which exhibits passive behaviors that follow the user’s requests, leading to user frustration, less efficient process, and less satisfactory results. (b) CollabLLM incorporates Multiturn-aware Rewards from collaborative simulation, enabling forward-looking strategies. This results in more high-performing, efficient, and interactive conversations that anticipate future needs, propose timely clarification, and provide insightful suggestions.

图 2：CollabLLM 与非协作 LLM 微调的真实样例。(a) 非协作微调依赖对即时回复的单轮奖励，行为被动、跟从用户请求，导致挫败感、流程低效、结果难满意。(b) CollabLLM 用协作仿真得到多轮感知奖励，支持前瞻策略：对话更高产、更高效、更可交互，能预判后续需求、及时澄清，并给出有洞见的建议。

1 Introduction

1 引言

Modern Large Language Models (LLMs) excel at generating high-quality single-turn responses when given well-specified inputs. However, real-world users often do not fully articulate their intents and sometimes initiate conversations with an imprecise understanding of their own needs (Taylor 1968). As a result, users routinely refine their requests post hoc through iterative corrections, which can increase frustration, hinder effective task completion, and reduce conversational efficiency (Amershi et al. 2019; Zamfirescu-Pereira et al. 2023; Wang et al. 2024; Kim et al. 2024). Therefore, an open problem is to train models that actively guide users in clarifying and refining their intents, and helps them achieve their goals. This key challenge would improve user satisfaction and efficiency and streamline human-LLM interactions—especially as LLMs are being applied to real-world tasks that are increasingly complex and open-ended.

当代大语言模型在输入写清时，单轮高质量回复已很强。但真实用户往往说不全意图，有时连自己要什么都不清楚（Taylor 1968）。于是事后靠反复纠正来改请求，挫败感上升、任务难完成、对话更低效（Amershi et al. 2019；Zamfirescu-Pereira et al. 2023；Wang et al. 2024；Kim et al. 2024）。开放问题是：如何训练能主动引导用户澄清、细化意图并达成目标的模型。这会抬高满意度与效率，也让人机交互更顺——尤其 LLM 正进入越来越复杂、开放的真实任务。

A notable limitation of established fine-tuning techniques, such as Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al. 2022), is that they primarily reward LLMs for immediate, single-turn responses, reducing their incentive to seek clarification or assist users in refining their intents or preferences. As a result, commonly used LLMs tend to prioritize direct answers, even though seeking additional context would enhance task completion and increase user satisfaction (Kim et al. 2024).

既有微调如基于人类反馈的强化学习（RLHF）（Ouyang et al. 2022）的一大局限是：主要奖励即时单轮回复，模型缺少动机去追问或帮用户细化意图与偏好。结果是常用 LLM 倾向直接给答案，即便多要一点上下文本可更好完成任务、提高满意度（Kim et al. 2024）。

Here we introduce CollabLLM, a novel and general training framework that improves the ability of LLMs to effectively collaborate with humans in multiturn scenarios (Gao et al. 2019; Balog & Zhai 2023; Rahmani et al. 2023). The key innovation of CollabLLM is to promote LLMs’ forward-looking behavior that leads to long-term collaboration gains (Figure 1). We introduce a collaborative simulation module that samples future conversations with users to estimate the long-term impact of model responses across multiple turns, a measure we term the Multiturn-aware Reward (MR). The MR function evaluates responses by incorporating both extrinsic metrics, such as task-specific success, and intrinsic metrics, such as efficiency, to holistically assess collaboration quality (*cf.*Section 3). By fine-tuning with established RL algorithms (Rafailov et al. 2023; Schulman et al. 2017) on MRs, CollabLLM promotes responses that lead to better task completion and efficiency in later conversation stages. As shown in Figure 2b, the fine-tuned model goes beyond simply responding to user requests in Figure 2a—it actively collaborates by asking follow-up questions about the writing tone, generating targeted content about the role of optimism, and offering insightful suggestions such as adding anecdotes.

本文提出 CollabLLM：一套提升 LLM 在多轮场景与人有效协作的通用训练框架（Gao et al. 2019；Balog & Zhai 2023；Rahmani et al. 2023）。关键在于推动模型的前瞻行为，以获得长期协作收益（图 1）。我们引入协作仿真模块：采样与用户的未来对话，估计模型回复跨多轮的长期影响，并称之为多轮感知奖励（MR）。MR 同时纳入任务成功等外在指标，以及效率等内在指标，从整体评估协作质量（见第 3 节）。再在 MR 上用既有 RL 算法（Rafailov et al. 2023；Schulman et al. 2017）微调，CollabLLM 会偏向在后续对话中更好完成任务、更省交互的回复。如图 2b，微调后的模型不止于图 2a 那种跟从请求——它会追问文风、生成关于乐观作用的定向内容，并建议加入轶事等洞见。

We also introduce three challenging multiturn tasks for training and evaluation in simulated environments: MediumDocEdit-Chat, BigCodeBench-Chat, and MATH-Chat, which respectively encompass document creation, code generation, and multiturn question answering. On the three test sets, our approach improves task accuracy metrics by 18.5% and interactivity by 46.3% on average compared to our best baselines, according to LLM judges. Beyond the tasks that the CollabLLMs are fine-tuned on, we show CollabLLMs are highly generalizable to other data domains.

我们还提出三项高难度多轮任务，用于仿真环境中的训练与评测：MediumDocEdit-Chat、BigCodeBench-Chat 与 MATH-Chat，分别覆盖文档撰写、代码生成与多轮问答。在三个测试集上，相对最佳基线，按 LLM 评判平均任务准确指标提升 18.5%，交互性提升 46.3%。除微调所用任务外，CollabLLM 对其他数据域也表现出较强泛化。

Moreover, we perform a large-scale and real-world user study with 201 Amazon Mechanical Turkers (MTurkers), who are asked to write documents with the help of anonymous AI assistants, either CollabLLM or non-collaboratively trained LLMs. CollabLLM achieves impressive improvement with 17.6% increase in user satisfaction and yield user time savings of 10.4% on average. The qualitative analysis from MTurkers confirms our observations: non-collaboratively-trained LLMs passively agree with users, while CollabLLM actively provide insightful questions and suggestions to guide writing processes.

此外，我们做了大规模真人研究：201 名 Amazon Mechanical Turk 工作者（MTurker）在匿名 AI 助手（CollabLLM 或非协作训练 LLM）帮助下写文档。CollabLLM 将用户满意度提升 17.6%，平均节省时间 10.4%。定性反馈印证观察：非协作训练模型被动附和用户，而 CollabLLM 会主动提出有洞见的问题与建议，引导写作。

2 Problem Formulation

2 问题形式化

In contrast to many existing tasks that are single-turn and require no human involvement beyond the initial query, our problem formulation reflects a real-world setting in which a user’s underlying (implicit) goal is defined as $g$ in a multiturn conversational task. The conversation unfolds over multiple turns $t_{j}=\{u_{j},m_{j}\}$, where $u_{j}$ is the user input and $m_{j}$ is the model’s response at each turn $j=1,\dots,K$, where $K$ is the number of turns in the conversation.

许多既有任务是单轮的，除初始查询外无需人参与；我们的设定更贴近真实：用户隐含目标记为 $g$，在多轮对话任务中展开。对话由多轮 $t_{j}=\{u_{j},m_{j}\}$ 组成，其中 $u_{j}$ 为用户输入，$m_{j}$ 为模型在第 $j=1,\dots,K$ 轮的回复，$K$ 为总轮数。

At the $j$-th turn, the model generates its response based on the previous conversation turns $t_{1:j-1}=\{t_{1},\dots,t_{j-1}\}$ and the current user response $u_{j}$. For simplicity, we define historical conversation at $j$-th turn as $t^{h}_{j}=t_{1:j-1}\cup\{u_{j}\}$, therefore, $m_{j}=M(t^{h}_{j})$. The objective is to generate a sequence of model responses $\{m_{j}\}_{j=1}^{K}$ that effectively and efficiently achieve for goal $g$, e.g.,answering a math question, where goal achievement is assessed based on user satisfaction or an external evaluation function, such as accuracy by LLM judge. Formally, we define the objective as $R^{}(t_{1:K}\mid g)$, where$R^{}$ incorporate the achievement of task success and user experience factors such as time cost.

第 $j$ 轮时，模型基于此前各轮 $t_{1:j-1}=\{t_{1},\dots,t_{j-1}\}$ 与当前用户输入 $u_{j}$ 生成回复。为简便，记第 $j$ 轮历史为 $t^{h}_{j}=t_{1:j-1}\cup\{u_{j}\}$，故 $m_{j}=M(t^{h}_{j})$。目标是生成回复序列 $\{m_{j}\}_{j=1}^{K}$，有效且高效地达成目标 $g$（例如解一道数学题）；目标达成可由用户满意度或外部评估（如 LLM 评判的准确率）衡量。形式上目标为 $R^{*}(t_{1:K}\mid g)$，其中 $R^{*}$ 综合任务成功与耗时等用户体验因素。

3 Unified Collaborative LLM Training

3 统一的协作式 LLM 训练

Key Motivations. Established LLM training frameworks, such as Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al. 2022), focus on maximizing immediate rewards for single-turn tasks. This cause a misalignment between their single-turn objective and real-world multiturn objective $R^{*}(t_{1:K}\mid g)$. Precisely, the model’s accumulative single-turn reward $\sum_{j=1}^{j=K}R(m_{j}\mid t_{j}^{h})$ may not imply a higher final reward $R^{*}(t_{1:K}\mid g)$. In fact, achieving high single-turn rewards at each turn may not imply a higher final reward. For example, consider a task where the user’s goal $g$ is to write an engaging article. A model trained with traditional RLHF might generate isolated responses, like drafting an introduction or listing conclusions. While these responses are helpful in isolation, they fail to consider how the sections flow together, resulting in an article that might not be cohesive and aligned with the user’s goal.

关键动机。既有 LLM 训练框架如 RLHF（Ouyang et al. 2022）主要最大化单轮任务的即时奖励，这与真实多轮目标 $R^{*}(t_{1:K}\mid g)$ 错位。精确地说，模型累积的单轮奖励 $\sum_{j=1}^{j=K}R(m_{j}\mid t_{j}^{h})$ 并不意味着最终奖励 $R^{*}(t_{1:K}\mid g)$ 更高。事实上，每一轮都刷高单轮奖励，未必带来更高的最终奖励。例如用户目标 $g$ 是写一篇吸引人的文章时，传统 RLHF 训出的模型可能分别起草引言或罗列结论；这些回复单独看有用，却未考虑各段如何衔接，文章可能既不连贯也与目标不对齐。

Instead, effective multiturn collaboration requires model responses that optimally contribute to the final reward. The model should aim to align its responses with the user’s goal $g$ by considering their impact on the entire conversation trajectory $t_{1:K}$. In the previous example, instead of generating a conclusion, asking, “Should I maintain an engaging tone in the conclusion like the introduction?” offers better long-term alignment with the goal.

相反，有效的多轮协作需要回复对最终奖励有最优贡献。 模型应考虑回复对整段轨迹 $t_{1:K}$ 的影响，以对齐用户目标 $g$。在上例中，与其直接写结论，不如问：「结论是否应像引言一样保持吸引人的语气？」——这更利于长期对齐。

3.1 Multiturn-aware Rewards

3.1 多轮感知奖励

In Figure 1, our key insight is that effective multiturn collaboration relies on forward-looking strategies. Given a context 1, the model should consider how its response 2 influences the subsequent turns of the conversation. To capture this, we design a 3 collaborative simulation module to estimate this impact. By 4 fine-tuning to distinguish between potential future conversations resulting from different responses, the model generates responses that align better with the overarching goal $g$.

如图 1，关键洞见是：有效多轮协作依赖前瞻策略。给定上下文 1，模型应考虑回复 2 如何影响后续各轮。为此我们设计 3 协作仿真模块估计该影响；再经 4 微调，使模型区分不同回复导致的未来对话，从而生成更对齐总体目标 $g$ 的回复。

This high-level design naturally aligns with causal effect estimation (Pearl 2009; Pearl et al. 2016), which evaluates the interventional effects of an action in sequential decision-making. Appendix A provides further discussion on the connection between causal effect estimation and our approach. More specifically, we define the Multiturn-aware Reward:

这一高层设计自然对应因果效应估计（Pearl 2009；Pearl et al. 2016）：评估序贯决策中某动作的干预效应。附录 A 进一步讨论因果效应估计与本文方法的联系。更具体地，我们定义多轮感知奖励：

Multiturn-aware Reward (MR): The multiturn-aware reward for model response $m_{j}$ at the $j$-th turn is given by:

多轮感知奖励（MR）：第 $j$ 轮模型回复 $m_{j}$ 的多轮感知奖励为：

$$\displaystyle\text{{MR} }(m_{j}\mid t_{j}^{h},g) \\ \displaystyle= \\ \displaystyle\mathbb{E}_{t_{j}^{f}\sim P(t_{j+1:K}\mid t_{j}^{h}\cup\{m_{j}\})}R^{*}(t_{j}^{h}\cup\{m_{j}\}\cup t_{j}^{f}\mid g) \\ \displaystyle= \\ \displaystyle\mathbb{E}_{t_{j}^{f}\sim P(t_{j}^{f}\mid t_{1:j})}R^{*}(t_{1:j}\cup t_{j}^{f}\mid g),$$

where $t_{1:j}$ denotes the conversation history up to and including the $j$-th turn, and $t_{j}^{f}=t_{j+1:K}$ represents the forward trajectory of turns following the $j$-th turn. The distribution $P(t_{j}^{f}\mid t_{1:j})$ models the possible forward conversations conditioned on the prior conversation history.

其中 $t_{1:j}$ 表示含第 $j$ 轮在内的对话历史，$t_{j}^{f}=t_{j+1:K}$ 为第 $j$ 轮之后的前向轨迹。分布 $P(t_{j}^{f}\mid t_{1:j})$ 刻画在已有历史上可能的前向对话。

However, computing Equation 1 remains challenging as it requires the following components: (a) A conversation-level reward function, $R^{*}(t\mid g)$, for evaluating an arbitrary multiturn conversation $t$, and (b) a sampling strategy for obtaining forward conversations $P(t_{j}^{f}\mid t_{1:j})$, which represents the forward conversation distribution. We elaborate on the two components in Section 3.1.1 and 3.1.2.

但计算式 (1) 仍难，因为它需要：(a) 对话级奖励函数 $R^{*}(t\mid g)$，用于评估任意多轮对话 $t$；(b) 前向对话的采样策略 $P(t_{j}^{f}\mid t_{1:j})$。我们在第 3.1.1 与 3.1.2 节分别展开这两部分。

3.1.1 Conversation-level Reward Function

We approximate the conversation-level reward $R^{*}(t\mid g)$ with a combination of extrinsic (goal-specific) and intrinsic (goal-agnostic) metrics:

3.1.1 对话级奖励函数

$$R^{*}(t\mid g)\simeq R_{\text{ext}}(t,g)+R_{\text{int}}(t), \tag{2}$$

where $R_{\text{ext}}(t,g)$ focuses on task-specific success, and $R_{\text{int}}(t)$ evaluates user experience including efficiency and engagement.

其中 $R_{\text{ext}}(t,g)$ 关注任务成功，$R_{\text{int}}(t)$ 评估效率与投入感等用户体验。

• • Extrinsic Reward $R_{\text{ext}}(t,g)$ measures how well the conversation achieves the user’s goal $g$. Formally: $$R_{\text{ext}}(t,g)=S(\operatorname{Extract}(t),y_{g}),$$ (3) where $\operatorname{Extract}(t)$ extracts the final solution or response from the conversation $t$, especially for tasks requiring revisions or multi-step answers. $y_{g}$ is the reference solution for the goal $g$, *e.g.,*the ground truth solution for a math problem. And $S(\cdot,\cdot)$ evaluates task-specific metrics like accuracy or similarity. This ensures the conversation contributes directly to achieving the desired goal.

• • 外在奖励 $R_{\text{ext}}(t,g)$ 衡量对话对用户目标 $g$ 的达成程度。形式化为：$$R_{\text{ext}}(t,g)=S(\operatorname{Extract}(t),y_{g}),$$ (3) 其中 $\operatorname{Extract}(t)$ 从对话 $t$ 抽出最终解或回复（尤其适合需修订或多步作答的任务）；$y_{g}$ 为目标 $g$ 的参考解（如数学题标准答案）；$S(\cdot,\cdot)$ 为准确率或相似度等任务指标。这保证对话直接服务于期望目标。

• • Intrinsic Reward $R_{\text{int}}(t)$ prioritizes conversations that enhance user experience, defined as: $$R_{\text{int}}(t)=-\min[\lambda\cdot\text{TokenCount}(t),1]+R_{\text{LLM}}(t),$$ (4) where we encourage conversational efficiency by penalizing excessive tokens that users read and write, with $\lambda$ controlling the penalty severity. This efficiency measure is bounded by 1 to maintain balance with other metrics. The second term, $R_{\text{LLM}}(t)$, is assigned by an LLM-based judge (Zheng et al. 2023) on a 0–1 scale, evaluating user-valued objectives such as engagement / interactivity. Notably, additional conversational aspects, such as clarity, can be further integrated into the objective.

• • 内在奖励 $R_{\text{int}}(t)$ 偏好提升用户体验的对话，定义为：$$R_{\text{int}}(t)=-\min[\lambda\cdot\text{TokenCount}(t),1]+R_{\text{LLM}}(t),$$ (4) 其中用惩罚用户读写过多 token 来鼓励效率，$\lambda$ 控制惩罚强度；效率项上界为 1，以与其他指标平衡。第二项 $R_{\text{LLM}}(t)$ 由基于 LLM 的评判器（Zheng et al. 2023）在 0–1 上打分，评估投入感 / 交互性等用户看重的目标。清晰度等其他对话维度也可并入目标。

The conversation-level reward incorporates task-specific and human-centered metrics, encouraging the model to balance goal achievement, efficiency, and engagement.

对话级奖励综合任务指标与以人为中心的指标，促使模型在目标达成、效率与投入感之间取得平衡。

Figure 3: Simulated Multiturn Environment for Evaluation. Our evaluation pipeline simulates real-world collaborations by prompting an user simulator LLM to emulate diverse behaviors and personalities in multiturn conversations.

图 3：用于评测的仿真多轮环境。评测流水线通过提示用户仿真 LLM 扮演多样行为与人格，在多轮对话中模拟真实协作。

3.1.2 Forward Sampling

To compute Eq. 1, we require samples from $P(t_{j}^{f}\mid t_{1:j})$, the distribution of forward conversation conditioned on the conversation history. A simple approach is to use Monte Carlo sampling, where the conversation is extended turn-by-turn until it concludes. However, this can be computationally expensive for computing reward for every model response. For a scalable approximation, we introduce a window size $w$ as a hyperparameter to limit the maximum number of forward turns considered in $t_{j}^{f}$. This reduces the computational cost while maintaining sufficient context.

3.1.2 前向采样

More importantly, while real-world conversations could be gathered from human participants, sampling multiple forward conversations during training is costly and impractical. To further reduce cost and ensure scalability, we introduce a user simulator $U$.

为计算式 (1)，需要从 $P(t_{j}^{f}\mid t_{1:j})$——以对话历史为条件的前向对话分布——取样。简单做法是蒙特卡洛：一轮轮延展对话直至结束；但对每个模型回复都算奖励时开销很大。为可扩展近似，我们引入窗口大小超参 $w$，限制 $t_{j}^{f}$ 最多考虑的前向轮数，在保有足够上下文的同时降低计算。

User Simulator:. A user simulator $U:\mathcal{T}\rightarrow\mathcal{U}$ is a function that maps a given conversation history $t\in\mathcal{T}$ to a user response $u\in\mathcal{U}$. Specifically, $U$ generates a probabilistic distribution $P(u\mid t)$ over possible user responses conditioned on the conversation history $t$, simulating realistic user behavior.

用户仿真器：用户仿真器 $U:\mathcal{T}\rightarrow\mathcal{U}$ 将给定对话历史 $t\in\mathcal{T}$ 映到用户回复 $u\in\mathcal{U}$。具体地，$U$ 生成以历史 $t$ 为条件的用户回复分布 $P(u\mid t)$，以模拟真实用户行为。

Specifically, we prompt an LLM to role-play as users, explicitly asking the LLM to follow the same language style as the previous user turns, and injecting typical user behaviors. The user simulator operates with an implicit goal $g$, which it seeks to achieve over the course of the conversation. This design emulates real-world scenarios where users may have evolving needs, limited background knowledge, or require clarification, resulting in naturally unfolding multiturn conversations (Park et al. 2024).

我们提示 LLM 角色扮演用户：明确要求沿用此前用户轮的语言风格，并注入典型用户行为。仿真器带有隐含目标 $g$，并在对话过程中试图达成。该设计模拟真实场景中需求演变、背景知识有限或需要澄清等情况，使多轮对话自然展开（Park et al. 2024）。

3.2 Optimization & Synthetic Datasets

3.2 优化与合成数据

With the conversation-level reward function and forward sampling strategy, we can compute MR for any model response without requiring an additional reward model, which is often costly and slow to train. Unlike traditional single-turn reward approaches, MR explicitly accounts for the impact of a response on future conversations, promoting long-term collaboration.

有了对话级奖励与前向采样，即可对任意模型回复计算 MR，而无需额外训练往往昂贵缓慢的奖励模型。与传统单轮奖励不同，MR 显式考虑回复对未来对话的影响，从而推动长期协作。

Further, we employ reinforcement learning (RL) methods such as PPO (Schulman et al. 2017) and DPO (Rafailov et al. 2023) to guide the model in navigating complex conversations. By optimizing for higher MR, the model learns to generate responses that enhance overall effectiveness and efficiency by the end of the conversation.

进而我们用 PPO（Schulman et al. 2017）、DPO（Rafailov et al. 2023）等强化学习方法，引导模型应对复杂对话。优化更高的 MR，使模型学会在对话结束时整体更有效、更高效的回复。

Moreover, MR can generate high-quality synthetic conversations (*cf.*Figure 8 in Appendix B) for both supervised fine-tuning (SFT) and DPO. For SFT, it iteratively selects top-ranked responses to build realistic, goal-directed conversation histories. For DPO, it constructs pairwise comparisons by ranking responses at each turn, distinguishing “chosen” and “rejected” pairs based on MR scores. The generated synthetic data aligns with multiturn objectives.

此外，MR 还能生成高质量合成对话（见附录 B 图 8），供监督微调（SFT）与 DPO 使用。对 SFT，它迭代选取排名靠前的回复，构建贴近真实、指向目标的对话历史；对 DPO，它在每轮按 MR 排序构造成对比较，区分「选中」与「拒绝」对。生成的合成数据与多轮目标对齐。

Overall, CollabLLM enables scalable dataset generation and online RL training without human annotation, making it generalizable across diverse tasks. In Appendix 7, we compare CollabLLM with related prompting- and training-based approaches, highlighting its contributions.

总体而言，CollabLLM 可在无需人工标注的情况下扩展数据生成与在线 RL 训练，并泛化到多样任务。附录 7 将 CollabLLM 与相关的提示式、训练式方法对比，突出其贡献。

4 Experimental Setup[^*]

4 实验设置[^*]

For fine-tuning and evaluation, we create three multiturn datasets using publicly available data across diverse domains (Hendrycks et al. 2021; Zhuo et al. 2024; Chiusano 2024): collaborative document editing, coding problem assistance, and multiturn mathematics problem solving.

为微调与评测，我们用公开数据跨多领域构建三套多轮数据集（Hendrycks et al. 2021；Zhuo et al. 2024；Chiusano 2024）：协作文档编辑、编程题协助，以及多轮数学解题。

To build a multiturn environment (Figure 3), we employ GPT-4o-mini as a user simulator LLM to role-play realistic user behaviors, given the target problem and conversation history. Our simulation-based evaluations are designed to closely mimic real-world interactions (Park et al. 2024). Unlike traditional single-turn tasks, our setup requires dynamic interactions over multiple turns to achieving a goal. The three interactive datasets are:

为搭建多轮环境（图 3），我们用 GPT-4o-mini 作为用户仿真 LLM，在给定目标题与对话历史时角色扮演真实用户行为。基于仿真的评测旨在贴近真实交互（Park et al. 2024）。与传统单轮任务不同，本设定需多轮动态交互才能达成目标。三套交互数据集为：

MediumDocEdit-Chat: Document editing requires iterative feedback and refinements across multiple turns to ensure coherence and alignment with user intent. We sample 100 Medium articles as goal documents, which are summarized into target problems to guide the user simulator. After each interaction, task performance is evaluated using the BLEU score, measuring similarity between the extracted document and the original articles.

MediumDocEdit-Chat：文档编辑需多轮迭代反馈与修订，以保证连贯并对齐用户意图。我们采样 100 篇 Medium 文章作为目标文档，再摘要成目标题以引导用户仿真器。每次交互后，用 BLEU 衡量抽出文档与原文的相似度作为任务表现。

BigCodeBench-Chat: Coding tasks inherently require multiturn interactions, such as clarifying requirements and debugging. We sample 600 coding problems from BigCodeBench (Zhuo et al. 2024) as the target problems given to the user simulator. For evaluation, we compute the average Pass Rate (PR) of code at the end of the interactions.

BigCodeBench-Chat：编程任务天然需要澄清需求、调试等多轮交互。我们从 BigCodeBench（Zhuo et al. 2024）采样 600 道编程题作为给用户仿真器的目标题。评测取交互结束时代码的平均 通过率（PR）。

MATH-Chat: Math problem solving often requires addressing implicit assumptions, verifying intermediate steps, and clarifying reasoning. We sample 200 level-5 math problems from MATH (Hendrycks et al. 2021) to prompt the user simulator, which interacts with the LLMs. Task success is measured by the accuracy (ACC) of the final solution, as evaluated by an LLM judge.

MATH-Chat：数学题常需处理隐含假设、核对中间步骤并澄清推理。我们从 MATH（Hendrycks et al. 2021）采样 200 道 5 级题提示用户仿真器与 LLM 交互。任务成功由 LLM 评判的最终解 准确率（ACC） 衡量。

In addition to the above task-specific metrics, we incorporate two task-agnostic scores across all datasets: 1) Average Token Count, which quantifies the average number of tokens generated by the LLM per conversation, reflecting interaction efficiency. 2) Interactivity (ITR), which evaluates engagement levels using an LLM judge (Claude-3.5-Sonnet), with scores rescaled to the range [0, 1].

除上述任务指标外，全部数据集还纳入两项与任务无关的分数：1）平均 Token 数，衡量每次对话 LLM 平均生成的 token 量，反映交互效率；2）交互性（ITR），由 LLM 评判器（Claude-3.5-Sonnet）评估投入程度，分数缩放到 [0, 1]。

	MediumDocEdit-Chat	BigCodeBench-Chat	MATH-Chat
	BLEU $\uparrow$	#Tokens$(k)\downarrow$	ITR $\uparrow$	PR $\uparrow$	#Tokens$(k)\downarrow$	ITR $\uparrow$	ACC $\uparrow$	#Tokens$(k)\downarrow$	ITR $\uparrow$
Base	32.2	2.49	46.0	9.3	1.59	22.0	11.0	3.40	44.0
Proactive Base	35.0	2.18	62.0	11.0	1.51	33.7	12.5	2.90	46.0
SFT	35.2	2.21	68.0	11.7	1.35	42.0	13.5	2.88	58.0
PPO	38.5	2.00	78.0	14.0	1.35	40.7	13.0	2.59	52.0
Offline DPO	36.4	2.15	82.0	12.3	1.35	46.7	15.5	2.40	50.0
Online DPO	36.8	2.00	92.0	13.0	1.31	52.0	16.5	2.37	60.0
Rel. Improv.	5.14%	8.25%	48.3%	18.2%	13.2%	54.3%	32.0%	18.3%	36.4%

Table 1: Evaluation results on our multiturn datasets. Green zone: Baselines; Orange zone: Variants of CollabLLMs.

	MediumDocEdit-Chat	BigCodeBench-Chat	MATH-Chat
	BLEU $\uparrow$	#Tokens$(k)\downarrow$	ITR $\uparrow$	PR $\uparrow$	#Tokens$(k)\downarrow$	ITR $\uparrow$	ACC $\uparrow$	#Tokens$(k)\downarrow$	ITR $\uparrow$
Base	32.2	2.49	46.0	9.3	1.59	22.0	11.0	3.40	44.0
Proactive Base	35.0	2.18	62.0	11.0	1.51	33.7	12.5	2.90	46.0
SFT	35.2	2.21	68.0	11.7	1.35	42.0	13.5	2.88	58.0
PPO	38.5	2.00	78.0	14.0	1.35	40.7	13.0	2.59	52.0
Offline DPO	36.4	2.15	82.0	12.3	1.35	46.7	15.5	2.40	50.0
Online DPO	36.8	2.00	92.0	13.0	1.31	52.0	16.5	2.37	60.0
Rel. Improv.	5.14%	8.25%	48.3%	18.2%	13.2%	54.3%	32.0%	18.3%	36.4%

表 1：多轮数据集上的评测结果。绿色区：基线；橙色区：CollabLLM 各变体。

Figure 4: Ablation Study of Reward Mechanisms on MediumDocEdit-Chat. This figure compares three immediate reward mechanisms with three MR variants. The results demonstrate that MR consistently improves task-specific performance (BLEU), conversational efficiency (# Tokens), and interactivity (ITR) compared to immediate reward mechanisms.

图 4：MediumDocEdit-Chat 上奖励机制消融。对比三种即时奖励与三种 MR 变体。结果显示，相对即时奖励，MR 持续提升任务表现（BLEU）、对话效率（# Tokens）与交互性（ITR）。

Fine-tuning CollabLLMs. CollabLLMs are based on Llama-3.1-8B (Llama Team 2024) with LoRA finetuning (Hu et al. 2022). We train four model variants: 1) Offline models: SFT and Offline DPO are fine-tuned on pre-generated multiturn conversational datasets guided by Multiturn-aware Rewards (MR) (*cf.*Section 3.2). 2) Online models: PPO and Online DPO are further trained from the SFT and Offline DPO models, respectively. The model during online fine-tuning is involved in the collaborative simulation to compute MRs, which, in turn, dynamically adjust the model preference.

微调 CollabLLM。CollabLLM 基于 Llama-3.1-8B（Llama Team 2024），用 LoRA 微调（Hu et al. 2022）。我们训练四种变体：1）离线模型：SFT 与 Offline DPO 在由多轮感知奖励（MR）引导预生成的多轮对话数据上微调（见第 3.2 节）；2）在线模型：PPO 与 Online DPO 分别从 SFT 与 Offline DPO 继续训练。在线微调时模型参与协作仿真以计算 MR，并据此动态调整偏好。

Baselines. We compare CollabLLMs against (1) the pretrained Llama-3.1-8B (Base), (2) the base model with proactive prompt engineering (Proactive Base), which encourages follow-up and clarification questions.

基线。对照包括：(1) 预训练 Llama-3.1-8B（Base）；(2) 带主动提示工程的基座（Proactive Base），鼓励追问与澄清。

Figure 5: Case study on BigCodeBench-Chat. The non-collaborative LLM assumes user needs, adding unnecessary steps like punctuation and stopword removal. In contrast, CollabLLM clarifies tokenizer preferences, error handling, and package installation, leading to a solution that precisely aligns with user intent.

图 5：BigCodeBench-Chat 案例。非协作 LLM 臆测用户需求，加入标点与停用词去除等多余步骤；CollabLLM 则澄清分词器偏好、错误处理与包安装，使最终方案精确对齐用户意图。

	Action-level Accuracy	Macro Metric
	Ambiguous	Non-Ambiguous	Accuracy	F1
GPT-4o	15.44%	95.60%	55.52%	56.62%
Base (Llama-3.1-8B)	16.26%	90.40%	53.33%	53.31%
CollabLLM	52.84%	72.32%	62.58%	55.08%

Figure 6: Reward comparison for response A and B of Figure 5 shows different preferences.

	动作级准确率	宏平均指标
	含糊	不含糊	准确率	F1
GPT-4o	15.44%	95.60%	55.52%	56.62%
Base (Llama-3.1-8B)	16.26%	90.40%	53.33%	53.31%
CollabLLM	52.84%	72.32%	62.58%	55.08%

图 6：对图 5 中回复 A 与 B 的奖励对比，显示不同偏好。

5 Results of Simulated Experiments

5 仿真实验结果

We present the results in Table 1 and the takeaways are:

结果见表 1，要点如下：

Prompt engineering is helpful, but limited in terms of performance gains and flexibility. Proactive Base improves base model performance by encouraging follow-up questions and clarifications. For example, it increases BLEU on MediumDocEdit-Chat from 32.2% to 35.0% and reduces read tokens by 0.31k compared to the base model. However, these gains are modest and do not fully address the challenges of multiturn collaboration. We observe that prompting strategies remain rigid, relying on predefined instructions rather than adapting dynamically to user needs. For instance, the model sometimes asks clarification questions even when unnecessary, leading to redundant interactions that disrupt conversation flow.

提示工程有用，但增益与灵活性有限。Proactive Base 通过鼓励追问与澄清提升基座表现：例如 MediumDocEdit-Chat 上 BLEU 从 32.2% 提到 35.0%，相对基座少读 0.31k token。但增益有限，不足以充分应对多轮协作。我们观察到提示策略偏僵硬，依赖预设指令而非动态适应用户：模型有时在不必澄清时仍提问，造成冗余、打断对话节奏。

CollabLLM increases task performance, conversational efficiency, and engagement. CollabLLM achieves 18.5% superior task-specific performance, 13.3% more efficient conversations, and 46.3% enhanced interactivity compared to the best baselines. We highlight that CollabLLM engage in more meaningful collaborations, with ITR shows substantial gains. For MediumDocEdit-Chat, the Online DPO model increases ITR from 0.46 to 0.92. Moreover, our framework significantly improves conversational efficiency by minimizing the content users need to review to arrive at the final solution. For MATH-Chat, Online DPO decreases token count per conversation by 1.03k compared to the base model.

CollabLLM 提升任务表现、对话效率与投入感。相对最佳基线，CollabLLM 任务指标平均高 18.5%，对话更高效 13.3%，交互性高 46.3%。尤其 ITR 提升显著：MediumDocEdit-Chat 上 Online DPO 将 ITR 从 0.46 提到 0.92。框架还通过减少用户为得到最终解需审阅的内容，显著提高效率：MATH-Chat 上 Online DPO 相对基座每对话少 1.03k token。

5.1 Ablations on Reward Mechanisms (Figure 4)

5.1 奖励机制消融（图 4）

To investigate how components contribute to CollabLLM’s superior performance, we conduct an ablation study focusing on the reward mechanisms used during fine-tuning. We evaluate the following reward mechanisms:

为弄清各组件如何贡献 CollabLLM 的优势，我们围绕微调所用奖励机制做消融，评估：

• • Variants of Multiturn-aware Reward: We vary the forward sampling window size $w=1,2,3$ to assess their ability to capture long-term conversational effects through simulated collaborations.

• • 多轮感知奖励变体：将前向采样窗口取 $w=1,2,3$，评估其通过仿真协作捕捉长期对话效应的能力。

• • Immediate Rewards evaluate the model’s immediate response based on: 1) Helpfulness: Assessed by an LLM judge; 2) Extrinsic Reward: Focuses on task-specific metrics like BLEU while ignoring intrinsic factors such as efficiency; 3) Extrinsic + Intrinsic Reward: Combines task-specific metrics with efficiency and interactivity measures. This can be seen as a special case of the multiturn-aware reward function with $w=0$.

• • 即时奖励按模型当轮回复评估：1) 有用性：LLM 评判；2) 外在奖励：只看 BLEU 等任务指标，忽略效率等内在因素；3) 外在 + 内在奖励：任务指标叠加效率与交互性。可视为多轮感知奖励在 $w=0$ 时的特例。

Figure 7: Our real-world user study includes 201 participants interacting with an anonymized AI assistant randomly sampled from Base, Proactive Base, and CollabLLM. Participants rate (a) document quality and (b) overall interaction experience, with additional assessments (d) every three turns. We also measure (c) user spent time to evaluate efficiency.

图 7：真人研究含 201 名参与者，与从 Base、Proactive Base、CollabLLM 中随机抽取的匿名 AI 助手交互。参与者对 (a) 文档质量与 (b) 整体交互体验打分，并每三轮做一次 (d) 评估；同时记录 (c) 耗时以衡量效率。

We present results in Figure 4. Interestingly, expanding the forward sampling window $w$ within the range generally enhances performance and efficiency by better capturing future interactions. Notably, MR with $w=2$ balances the gains and additional costs to conduct forward sampling, making it well-suited for large-scale fine-tuning. In contrast, immediate rewards, even with extrinsic and intrinsic components, fall short as they ignore long-term impact. These findings validate the positive impact of the forward sampling strategy in MRs.

结果见图 4。有意思的是，在考察范围内扩大前向窗口 $w$，一般能更好捕捉未来交互，从而提升表现与效率。尤其 $w=2$ 的 MR 在收益与前向采样额外开销之间较均衡，适合大规模微调。相比之下，即便同时含外在与内在分量，即时奖励仍因忽视长期影响而不足。这些发现验证了 MR 中前向采样策略的积极作用。

5.2 Case Study (Figure 5 &  6)

5.2 案例研究（图 5 与图 6）

We now offer a deeper insight into CollabLLM’s behavior as shown in Figure 5. In this example, the user request to tokenize a text file is inherently open-ended due to unspecified factors, such as the NLTK environment, tokenizer selection, and optional preprocessing steps. The base LLM makes several arbitrary assumptions, applying lowercase conversion and stopword removal without user confirmation. The user simulator later corrects these assumptions, but the final solution remains incorrect due to missing stopwords. In contrast, CollabLLM actively clarifies user intent by seeking confirmation on key decisions, ensuring an aligned final solution with a 100% Pass Rate. This approach also reduces user effort with lower token usage.

图 5 给出更细的行为解读。用户请求对文本文件做分词，但 NLTK 环境、分词器选择与可选预处理均未说明，请求本身开放。基座 LLM 擅自做小写转换与去停用词等假设；用户仿真器事后纠正，但因停用词缺失最终解仍错。CollabLLM 则主动澄清关键决策，保证最终方案对齐，通过率 100%，且 token 更少、用户负担更轻。

In Figure 6, we compare different reward mechanisms for responses A and B of Figure 5, to confirm that these rewards work as intended. The helpfulness rewards favor response A due to its seemingly more well-round output. Extrinsic rewards assign zero scores to both, as A provides an incorrect solution and B defers answering. Extrinsic + Intrinsic rewards slightly favor B for efficiency and engagement. Interestingly, MR assigns significantly higher rewards to B, especially at $w=2$ and $w=3$, since the response obtains useful information and provide a precise answer within the future interaction window.

图 6 对比图 5 中回复 A 与 B 的不同奖励机制，以确认奖励是否按预期工作。有用性奖励偏好看似更全面的 A；外在奖励对两者都给零——A 解错，B 暂缓作答；外在 + 内在略偏 B，因其效率与投入感更好。有趣的是 MR 显著更偏 B，尤其在 $w=2$ 与 $w=3$ 时：该回复在未来交互窗口内能拿到有用信息并给出精确答案。

Model	Strengths	Weaknesses
Base	“Follows great instruction and does exactly what I’m asking it to do.”, “It can create a nice form of an outline to work with.”	“The AI just agreed with me on pretty much everything. There was no discussion”, “I didn’t really like that it kept coming up with different options”
Proactive Base	“It is very organized and it actually asks you for feedback after writing the revision.”	“The AI seemed to be very redundant and asked me the same questions over and over.”
Collab LLM	“Asking questions and making you think of things you never thought of”, “The AI really helped me with focusing on one part of the story at a time.”, “It helped really well to navigate what to say and what information is needed”	“The AI assistant was not up to date enough to help with this recent sporting event. The AI assistant also asked me to repeat information I had already given it.”

Table 3: Representative Feedback from Human Participants.

模型	优点	不足
Base	「能很好跟指令，按我说的做。」「能做出不错的大纲。」	「AI 几乎事事附和，没什么讨论。」「不太喜欢它老换各种选项。」
Proactive Base	「很有条理，改完还会问你反馈。」	「显得很啰嗦，同一问题问了又问。」
Collab LLM	「会提问，让你想到从没想过的点。」「帮我一次只盯故事的一部分。」「很会引导该说什么、需要哪些信息。」	「对最近这场体育赛事不够新，帮不上；有时让我重复已经给过的信息。」

表 3：真人参与者代表性反馈。

5.3 Model Generalization (Table 2)

5.3 模型泛化（表 2）

Modern foundation models are expected to generalize across a diverse range of tasks beyond their training domain. A key question is whether collaborative behaviors learned by CollabLLM during fine-tuning transfer effectively to new tasks without additional adaptation.

当代基础模型被期待能泛化到训练域之外的多样任务。关键问题是：CollabLLM 微调时学到的协作行为，能否在无需额外适配时迁移到新任务。

We assess CollabLLM, trained with online DPO on BigCodeBench-Chat (the coding assistance task), on Abg-CoQA (Guo et al. 2021), a question-answering (QA) benchmark where questions are labeled as ambiguous or non-ambiguous (*cf.*Appendix D). We categorize the model’s responses into two actions—asking a clarifying question or providing a direct answer—and evaluate action-level accuracy within each question type. As shown in Table 2, GPT-4o and Llama-3.1-8B rarely ask clarifying questions regardless of ambiguity. In contrast, CollabLLM proactively asks questions about 50% of the time while maintaining high accuracy on unambiguous inputs. This behavior leads to the highest Macro Accuracy across both ambiguous and non-ambiguous sets and improves Macro F1 over the base model, while leaving room for further improvement against GPT-4o. These results suggest that CollabLLM effectively generalizes its learned collaborative strategies beyond its training domain.

我们将在 BigCodeBench-Chat（编程协助）上用 online DPO 训练的 CollabLLM，放到 Abg-CoQA（Guo et al. 2021）上评估：该问答基准将问题标为含糊或不含糊（见附录 D）。把模型回复分为两类动作——提出澄清问题或直接作答——并在各类问题内评估动作级准确率。如表 2，GPT-4o 与 Llama-3.1-8B 无论是否含糊都很少澄清；CollabLLM 约一半时间主动提问，同时在不含糊输入上仍保持较高准确率。该行为使其在含糊与不含糊集合上取得最高宏准确率，相对基座提升宏 F1，相对 GPT-4o 仍有改进空间。结果表明 CollabLLM 能把学到的协作策略有效泛化到训练域之外。

6 Real-world User Study

6 真人用户研究

Setup. We conduct a large-scale user study using Amazon Mechanical Turk with 201 participants. Each participant is assigned a document type—randomly selected to be either blog post, creative writing, or personal statement—and chooses a topic from a predefined set. To simulate real-world scenarios where users have only a rough idea of the task, they are first asked to provide brief responses to topic-related questions. Participants then engage in at least eight turns of conversation with an anonymized AI assistant, which can be Base, Proactive Base, or CollabLLM. Every three turns, they provide an interaction rating based on their experience so far. After the conversation, participants rate the final document quality and overall interaction. All ratings are in a scale from 1 to 10. We also record the total interaction duration to assess efficiency. The detailed user study setup is provided in Appendix E.

设置。我们在 Amazon Mechanical Turk 上做大规模用户研究，共 201 名参与者。每人被随机分配一种文档类型（博客、创意写作或个人陈述），再从预设集合选题。为模拟用户对任务只有粗略想法的真实场景，先请他们就话题相关问题做简短回答，再与匿名 AI 助手（Base、Proactive Base 或 CollabLLM）至少对话八轮。每三轮基于当前体验打一次交互分；结束后再评最终文档质量与整体交互。评分均为 1–10。同时记录总交互时长以衡量效率。细节见附录 E。

Quantitative Results (Figure 7). Across multiple metrics, CollabLLM consistently outperforms the baselines. It achieves an average document quality score of 8.50. Specifically, 91.4% of participants rate CollabLLM’s document quality as “good” (score 8–9), and 56.9% as “very good” (score 9–10), compared to 88.5% and 39.3% for Base (Llama-3.1-8B), respectively. Similarly, 63.8% of participants find CollabLLM highly engaging, while only 42.6% report the same for Llama-3.1-8B.

定量结果（图 7）。多项指标上 CollabLLM 持续优于基线。文档质量均分 8.50：91.4% 的参与者将 CollabLLM 的文档质量评为「好」（8–9 分），56.9% 评为「很好」（9–10 分）；Base（Llama-3.1-8B）对应为 88.5% 与 39.3%。同样，63.8% 认为 CollabLLM 高度有投入感，Llama-3.1-8B 仅 42.6%。

Interestingly, for multiturn interaction, the Base model shows a declining trend in ratings from turns 6–9, indicating reduced user experience in longer conversations. In contrast, both CollabLLM and Proactive Base exhibit increasing ratings over time, with CollabLLM consistently achieving higher average ratings every three turns compared to Proactive Base. This suggests that CollabLLM maintains sustained engagement more effectively.

有趣的是，在多轮交互上，Base 在第 6–9 轮评分呈下降趋势，说明长对话体验变差；CollabLLM 与 Proactive Base 评分随时间上升，且 CollabLLM 每三轮平均分始终高于 Proactive Base。这表明 CollabLLM 更能维持持续投入。

Moreover, CollabLLM improves task efficiency, reducing time spent by 10.4% compared to the Base model and by 15.6% relative to Proactive Base. While Proactive Base is prompted to maintain conciseness, it frequently asks unnecessary questions, causing lower efficiency. In contrast, CollabLLM strikes a more streamlined user experience.

此外 CollabLLM 提升任务效率：相对 Base 耗时少 10.4%，相对 Proactive Base 少 15.6%。Proactive Base 虽被提示保持简洁，却常问多余问题，效率更低；CollabLLM 的用户体验更干练。

Qualitative Results (Table 3). We collected a total of 180 strengths and 180 weaknesses across the three models. Table 3 presents representative feedback, while we summarize here the mddels’ strengths and weaknesses: The base model generates coherent content while effectively follow user instructions, but it sometimes struggles with maintaining context in long texts, and can be overly verbose or repetitive in its responses. Proactive Base excels in responsiveness and adapting to user input but struggles with memory retention, and could produce repetitive or overly structured content. On the other hand, CollabLLM is highly engaging, effectively guiding users through writing, adapting seamlessly to feedback. However, users also point out that CollabLLM can occasionally feel bland, lack of up to date information, and require additional effort to personalize the output. Overall, CollabLLM enhances collaboration by guiding users through an interactive and iterative refinement process, yet future improvements should focus on increasing personalization, creativity, and real-time knowledge integration to further optimize human-LLM collaboration.

定性结果（表 3）。三模型共收集 180 条优点与 180 条不足。表 3 给出代表性反馈，此处概括：基座能跟指令、内容连贯，但长文难保上下文，有时啰嗦或重复；Proactive Base 响应快、能适应用户输入，但记忆差，内容可能重复或过于程式化；CollabLLM 投入感强，能引导写作并顺畅适配反馈，但用户也指出偶尔平淡、知识不够新、个性化需额外费力。总体看，CollabLLM 通过交互式迭代修订增强协作；未来应加强个性化、创造力与实时知识整合，进一步优化人机协作。

7 Related Work

7 相关工作

	Task-Agnostic	Versatile Interaction	User-Centric	Causal & Objective-Aligned Reward
ClarifyGPT (Mu et al. 2023)	✗	✗	✗	-
STaR-GATE (Andukuri et al. 2024)	✔	✗	✗	-
Multiturn RLHF (Shani et al. 2024)	✔	✔	✗	✗
CollabLLM	✔	✔	✔	✔

Table 4: Compare CollabLLM with Selected Works. (1) Task-Agnostic, assessing whether the approach applies across diverse domains rather than being task-specific; (2) Versatile Interaction, evaluating its ability to support diverse strategies for intent discovery and efficient task completion beyond predefined behaviors; (3) User-Centric, determining whether engagement, efficiency, and intent discovery are explicitly considered; and (4) Causal & Objective-Aligned Reward, measuring whether reward estimation captures causal effects on future interactions and optimizes for long-term task success.

	任务无关	多样交互	用户中心	因果且目标对齐的奖励
ClarifyGPT（Mu et al. 2023）	✗	✗	✗	-
STaR-GATE（Andukuri et al. 2024）	✔	✗	✗	-
Multiturn RLHF（Shani et al. 2024）	✔	✔	✗	✗
CollabLLM	✔	✔	✔	✔

表 4：CollabLLM 与若干工作对比。(1) 任务无关：是否跨领域通用而非绑定单一任务；(2) 多样交互：是否支持超越预设行为的多种意图发现与高效完成策略；(3) 用户中心：是否显式考虑投入感、效率与意图发现；(4) 因果且目标对齐的奖励：奖励估计是否捕捉对未来交互的因果效应，并优化长期任务成功。

Non-collaborative LLM training. Existing LLM training frameworks, including pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL) (Rafailov et al. 2023; Schulman et al. 2017; Ouyang et al. 2022; Lee et al. 2024), primarily optimize for next-turn response quality. Standard RL methods such as Proximal Policy Optimization (PPO) (Schulman et al. 2017) apply rewards to individual model responses without accounting for their long-term impact on conversation trajectories. While effective for single-turn objectives, these approaches fail to capture how responses influence user intent discovery and long-term task success (Amershi et al. 2019; Zamfirescu-Pereira et al. 2023; Wang et al. 2024; Kim et al. 2024; Lu et al. 2024).

非协作式 LLM 训练。既有 LLM 训练——预训练、监督微调（SFT）与强化学习（RL）（Rafailov et al. 2023；Schulman et al. 2017；Ouyang et al. 2022；Lee et al. 2024）——主要优化下一轮回复质量。PPO（Schulman et al. 2017）等标准 RL 把奖励加在单条回复上，不计其对对话轨迹的长期影响。这对单轮目标有效，却抓不住回复如何影响意图发现与长期任务成功（Amershi et al. 2019；Zamfirescu-Pereira et al. 2023；Wang et al. 2024；Kim et al. 2024；Lu et al. 2024）。

Prompting techniques for multiturn interaction. Prior work has explored prompting strategies to enhance LLM interactivity, particularly for clarification questions (Keh et al. 2024; Mu et al. 2023; Zhang & Choi 2023; Chi et al. 2024; Kim et al. 2023; Deng et al. 2023b; Zhao & Dou 2024) and mixed-initiative dialogues (Deng et al. 2023a; Chen et al. 2023; Liao et al. 2023). For instance, Mu et al. 2023 prompt LLMs to ask clarification questions when code generation requests are ambiguous. However, such prompting-based approaches are constrained by predefined interaction patterns, limiting adaptability across different tasks and conversation stages. Moreover, their reliance on fixed prompts reduces generalization, as demonstrated in our experiments where proactive prompting fails to match the effectiveness of our fine-tuned models.

多轮交互的提示技术。先前工作用提示提升 LLM 交互性，尤其是澄清问题（Keh et al. 2024；Mu et al. 2023；Zhang & Choi 2023；Chi et al. 2024；Kim et al. 2023；Deng et al. 2023b；Zhao & Dou 2024）与混合主动对话（Deng et al. 2023a；Chen et al. 2023；Liao et al. 2023）。例如 Mu et al. 2023 在代码生成请求含糊时提示模型提问。但提示式方法受预设交互模式约束，跨任务与对话阶段适应性有限；依赖固定提示也削弱泛化——我们实验中主动提示远不及微调模型。

Learning-based methods for multiturn interaction.

面向多轮交互的学习方法。

• • LLMs for generating clarification questions: Beyond prompting, prior studies have explored supervised fine-tuning (Andukuri et al. 2024), RL fine-tuning (Chen et al. 2024; Zamani et al. 2020; Erbacher & Soulier 2023), and active learning (Pang et al. 2024) to train models to ask clarification questions. For example, Chen et al. 2024 use Direct Preference Optimization (DPO) to encourage models to identify ambiguous user queries and request clarifications. However, like prompting approaches, these methods primarily focus on clarification questions and do not generalize to broader multiturn collaboration strategies.

• • 用于生成澄清问题的 LLM： 除提示外，已有监督微调（Andukuri et al. 2024）、RL 微调（Chen et al. 2024；Zamani et al. 2020；Erbacher & Soulier 2023）与主动学习（Pang et al. 2024）训练模型提问。例如 Chen et al. 2024 用 DPO 鼓励识别含糊查询并请求澄清。但与提示类似，这些方法主要盯着澄清问题，难以泛化到更广的多轮协作策略。

• • Multiturn training for LLMs: Several studies extend RLHF to multiturn settings by optimizing trajectory-level rewards (Shani et al. 2024; Zhou et al. 2024; Gao et al. 2024; Shi et al. 2024b). Other works (Xu et al. 2023; Deng et al. 2024) leverage self-chat or self-play to enhance model adaptation to dialogue. See Zhang et al. 2025 for a comprehensive survey. However, these methods primarily rely on post-hoc trajectory-level data, learning from observed conversations rather than explicitly modeling the causal effect of individual responses on long-term task success (see Appendix A for further details). Additionally, they often overlook open-ended tasks such as document generation (Faltings et al. 2023; Jiang et al. 2024), where real-world user responses can be highly variable, and users may have limited capacity to read and refine lengthy model outputs.

• • LLM 的多轮训练： 若干工作把 RLHF 扩到多轮，优化轨迹级奖励（Shani et al. 2024；Zhou et al. 2024；Gao et al. 2024；Shi et al. 2024b）；另一些用自对话或自对弈增强对话适应（Xu et al. 2023；Deng et al. 2024）。综述见 Zhang et al. 2025。但这些方法多依赖事后轨迹级数据，从已观测对话学习，而非显式建模单条回复对长期任务成功的因果效应（细节见附录 A）。它们也常忽略文档生成等开放任务（Faltings et al. 2023；Jiang et al. 2024）——真实用户回复高度多变，且阅读、修订长回复的能力有限。

In Table 4, we compare CollabLLM with related methods across four key dimensions. CollabLLM is a general, user-centric, and multiturn-aware framework that leverages more accurate reward estimation to better align with real-world objectives, enhancing user satisfaction and streamlining human-LLM interactions.

表 4 从四个关键维度对比 CollabLLM 与相关方法。CollabLLM 是通用、用户中心且多轮感知的框架，用更准确的奖励估计对齐真实目标，提升满意度并理顺人机交互。

8 Conclusion

8 结论

Multiturn human-LLM collaborations are increasingly prevalent in real-world applications. We believe future foundation models should act as collaborators rather than passive responders, actively uncovering user intents in open-ended and complex tasks—an area where current LLMs fall short. The key insight of CollabLLM is making LLMs more multiturn-aware by using forward sampling to estimate the long-term impact of responses. Through extensive simulated and real-world evaluations, we demonstrate that CollabLLM is highly effective, efficient, and engaging, while also generalizing well to new tasks and interactions, advancing the frontiers of human-centered LLMs.

多轮人机协作在真实应用中日益普遍。我们认为未来基础模型应充当协作者而非被动应答者，在开放复杂任务中主动揭示用户意图——而这正是当前 LLM 的短板。CollabLLM 的关键是通过前向采样估计回复的长期影响，使 LLM 更具多轮意识。经充分的仿真与真人评测，我们表明 CollabLLM 有效、高效且有投入感，并能泛化到新任务与新交互，推进以人为中心的 LLM 边界。

Impact Statement

影响声明

This paper presents work aimed at making AI more user- and human-centric, which, in our view, yields a positive societal impact. Most current work on AI and its evaluation focuses on fully automated tasks, with no user involvement in solving the task or optimization for a collaborative experience with users. This has serious societal drawbacks, given issues such as AI hallucinations (Huang et al. 2025), biases (Gallegos et al. 2024), and unsafe language (Shi et al. 2024a) that arise from a lack of human oversight. The common focus on having AI models autonomously complete tasks also ignores the reality that many scenarios have humans present regardless of the level of automation, and that not priming AI models to proactively seek human help, feedback, or clarifications misses an opportunity to make generative AI more accurate, effective, and safe. This consideration would also help increase the adoption of AI in safety-critical scenarios, such as medical decision-making tasks (Liu et al. 2024), in which we believe AI models should be inclined to seek confirmation or verification (Gero et al. 2023) from an expert in case of uncertainty—a behavior that is mostly absent in current state-of-the-art LLMs.

本文旨在让 AI 更以用户与人为中心，我们认为这具有积极社会影响。当前多数 AI 工作与评测聚焦全自动任务，解题过程无用户参与，也不优化协作体验。鉴于幻觉（Huang et al. 2025）、偏见（Gallegos et al. 2024）与不安全语言（Shi et al. 2024a）等因缺乏人类监督而生的问题，这有严重社会弊端。一味让模型自主完成任务，也忽略了许多场景中无论自动化程度如何都有人在场的现实；若不引导模型主动寻求帮助、反馈或澄清，就错失让生成式 AI 更准确、有效与安全的机会。这也有助于提升 AI 在医疗决策等安全关键场景（Liu et al. 2024）中的采用——我们相信模型在不确定时应倾向于向专家确认或核验（Gero et al. 2023），而当前前沿 LLM 大多缺少这种行为。

This work presents one of the first attempts to train LLMs in such human-centric environments. To promote future research in this societally beneficial direction, we plan to release all the code, models, data, benchmarks, and user simulators described in this work.

本文是在此类以人为中心环境中训练 LLM 的早期尝试之一。为推动这一有益方向，我们计划公开文中所述全部代码、模型、数据、基准与用户仿真器。

The data collected in our study involves human participants recruited through Mechanical Turk. We took several measures to ensure the privacy of these workers in the document creation tasks. First, we asked workers to confirm that they were willing to share the text they wrote as part of a public dataset. Second, we urged them not to include any personally identifiable information (PII) in their writings and to focus only on topics of public knowledge or fictitious stories. Third, we scanned the collected data to ensure that no PII was included. For the final version of the dataset, we will recruit additional workers to manually review each collected conversation to ensure that no PII or other safety issues (e.g., offensive language) exist in the data. Mechanical Turk workers were paid $10 per conversation. Given that conversations averaged 28.4 minutes, including break times, this means workers were paid more than$20 per hour on average—above the minimum wage in the country where the data was collected.

研究数据来自 Mechanical Turk 招募的参与者。在文档撰写任务中我们采取多项隐私措施：请工作者确认愿意将所写文本纳入公开数据集；敦促勿写入可识别个人信息（PII），只写公共知识或虚构故事；并扫描数据确认无 PII。最终版数据集还将另雇人手审每段对话，确保无 PII 或其他安全问题（如冒犯性语言）。每位工作者每段对话报酬 10 美元；对话平均 28.4 分钟（含休息），折合时薪逾 20 美元——高于数据采集所在国最低工资。

Acknowledgments

致谢

We thank Vishal Chowdhary, Hoifung Poon, Swadheen Shukla, Chandan Singh and Chenglong Wang, as well as members of the Deep Learning and Health Futures groups at Microsoft Research for helpful discussions. We thank lab members in Leskovec and Zou’s labs for discussions and for providing feedback.

感谢 Vishal Chowdhary、Hoifung Poon、Swadheen Shukla、Chandan Singh 与 Chenglong Wang，以及微软研究院 Deep Learning 与 Health Futures 组同事的讨论。感谢 Leskovec 与 Zou 实验室成员的讨论与反馈。

References

参考文献

Amershi et al. (2019) Amershi, S., Weld, D. S., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S. T., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., and Horvitz, E. Guidelines for human-ai interaction. In CHI . ACM, 2019.

Andukuri et al. (2024) Andukuri, C., Fränken, J., Gerstenberg, T., and Goodman, N. D. Star-gate: Teaching language models to ask clarifying questions. CoRR , abs/2403.19154, 2024.

Balog & Zhai (2023) Balog, K. and Zhai, C. Rethinking conversational agents in the era of llms: Proactivity, non-collaborativity, and beyond. In SIGIR-AP , 2023.

Chen et al. (2023) Chen, M., Yu, X., Shi, W., Awasthi, U., and Yu, Z. Controllable mixed-initiative dialogue generation through prompting. In ACL , 2023.

Chen et al. (2024) Chen, M., Sun, R., Arik, S. Ö., and Pfister, T. Learning to clarify: Multi-turn conversations with action-based contrastive self-training. CoRR , abs/2406.00222, 2024.

Chi et al. (2024) Chi, Y., Lin, J., Lin, K., and Klein, D. CLARINET: augmenting language models to ask clarification questions for retrieval. CoRR , abs/2405.15784, 2024.

Chiusano (2024) Chiusano, F. Medium articles dataset. https://huggingface.co/datasets/fabiochiu/medium-articles , 2024. A dataset of Medium articles collected through web scraping, including metadata such as titles, authors, publication dates, tags, and content.

Deng et al. (2023a) Deng, Y., Liao, L., Chen, L., Wang, H., Lei, W., and Chua, T. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. In EMNLP , 2023a.

Deng et al. (2023b) Deng, Y., Zhang, W., Chen, Z., and Gu, Q. Rephrase and respond: Let large language models ask better questions for themselves. CoRR , abs/2311.04205, 2023b.

Deng et al. (2024) Deng, Y., Zhang, W., Lam, W., Ng, S., and Chua, T. Plug-and-play policy planner for large language model powered dialogue agents. In ICLR , 2024.

Erbacher & Soulier (2023) Erbacher, P. and Soulier, L. CIRCLE: multi-turn query clarifications with reinforcement learning. CoRR , abs/2311.02737, 2023.

Faltings et al. (2023) Faltings, F., Galley, M., Brantley, K., Peng, B., Cai, W., Zhang, Y., Gao, J., and Dolan, B. Interactive text generation. In EMNLP , 2023.

Gallegos et al. (2024) Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., and Ahmed, N. K. Bias and fairness in large language models: A survey, 2024. URL https://arxiv.org/abs/2309.00770 .

Gao et al. (2019) Gao, J., Galley, M., and Li, L. Neural approaches to conversational AI. Found. Trends Inf. Retr. , 13, 2019.

Gao et al. (2024) Gao, Z., Zhan, W., Chang, J. D., Swamy, G., Brantley, K., Lee, J. D., and Sun, W. Regressing the relative future: Efficient policy optimization for multi-turn rlhf, 2024.

Gero et al. (2023) Gero, Z., Singh, C., Cheng, H., Naumann, T., Galley, M., Gao, J., and Poon, H. Self-verification improves few-shot clinical information extraction, 2023. URL https://arxiv.org/abs/2306.00024 .

Guo et al. (2021) Guo, M., Zhang, M., Reddy, S., and Alikhani, M. Abg-coqa: Clarifying ambiguity in conversational question answering. In AKBC , 2021.

Hendrycks et al. (2021) Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS , 2021.

Hu et al. (2022) Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In ICLR , 2022.

Huang et al. (2025) Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., and Liu, T. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems , 43(2):1–55, January 2025. ISSN 1558-2868. doi: 10.1145/3703155 . URL http://dx.doi.org/10.1145/3703155 .

Jiang et al. (2024) Jiang, Y., Shao, Y., Ma, D., Semnani, S. J., and Lam, M. S. Into the unknown unknowns: Engaged human learning through participation in language model agent conversations. arXiv , abs/2408.15232, 2024.

Keh et al. (2024) Keh, S., Chiu, J. T., and Fried, D. Asking more informative questions for grounded retrieval. In FNAACL , 2024.

Kim et al. (2023) Kim, G., Kim, S., Jeon, B., Park, J., and Kang, J. Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models. In EMNLP , 2023.

Kim et al. (2024) Kim, Y., Lee, J., Kim, S., Park, J., and Kim, J. Understanding users’ dissatisfaction with chatgpt responses: Types, resolving tactics, and the effect of knowledge level. In IUI . ACM, 2024.

Lee et al. (2024) Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In ICML , 2024.

Liao et al. (2023) Liao, L., Yang, G. H., and Shah, C. Proactive conversational agents in the post-chatgpt world. In SIGIR , 2023.

Liu et al. (2024) Liu, L., Yang, X., Lei, J., Shen, Y., Wang, J., Wei, P., Chu, Z., Qin, Z., and Ren, K. A survey on medical large language models: Technology, application, trustworthiness, and future directions, 2024. URL https://arxiv.org/abs/2406.03712 .

Llama Team (2024) Llama Team, A. . M. The llama 3 herd of models, 2024.

Lu et al. (2024) Lu, Y., Yang, S., Qian, C., Chen, G., Luo, Q., Wu, Y., Wang, H., Cong, X., Zhang, Z., Lin, Y., Liu, W., Wang, Y., Liu, Z., Liu, F., and Sun, M. Proactive agent: Shifting LLM agents from reactive responses to active assistance. CoRR , abs/2410.12361, 2024.

Mu et al. (2023) Mu, F., Shi, L., Wang, S., Yu, Z., Zhang, B., Wang, C., Liu, S., and Wang, Q. Clarifygpt: Empowering llm-based code generation with intention clarification. CoRR , abs/2310.10996, 2023.

Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , 2022.

Pang et al. (2024) Pang, J., Fan, H., Wang, P., Xiao, J., Tang, N., Yang, S., Jia, C., Huang, S., and Yu, Y. Empowering language models with active inquiry for deeper understanding. CoRR , abs/2402.03719, 2024.

Park et al. (2024) Park, J. S., Zou, C. Q., Shaw, A., Hill, B. M., Cai, C., Morris, M. R., Willer, R., Liang, P., and Bernstein, M. S. Generative agent simulations of 1,000 people, 2024.

Pearl (2009) Pearl, J. Causality: Models, Reasoning and Inference . Cambridge University Press, 2nd edition, 2009.

Pearl et al. (2016) Pearl, J., Glymour, M., and Jewell, N. P. Causal Inference in Statistics: A Primer . John Wiley & Sons, 2016.

Rafailov et al. (2023) Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS , 2023.

Rahmani et al. (2023) Rahmani, H. A., Wang, X., Feng, Y., Zhang, Q., Yilmaz, E., and Lipani, A. A survey on asking clarification questions datasets in conversational systems. In ACL , 2023.

Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

Shani et al. (2024) Shani, L., Rosenberg, A., Cassel, A. B., Lang, O., Calandriello, D., Zipori, A., Noga, H., Keller, O., Piot, B., Szpektor, I., Hassidim, A., Matias, Y., and Munos, R. Multi-turn reinforcement learning from preference human feedback. CoRR , abs/2405.14655, 2024.

Shi et al. (2024a) Shi, D., Shen, T., Huang, Y., Li, Z., Leng, Y., Jin, R., Liu, C., Wu, X., Guo, Z., Yu, L., Shi, L., Jiang, B., and Xiong, D. Large language model safety: A holistic survey, 2024a. URL https://arxiv.org/abs/2412.17686 .

Shi et al. (2024b) Shi, W., Yuan, M., Wu, J., Wang, Q., and Feng, F. Direct multi-turn preference optimization for language agents. In EMNLP . Association for Computational Linguistics, 2024b.

Taylor (1968) Taylor, R. S. Question-negotiation and information seeking in libraries. College & research libraries , 29(3):178–194, 1968.

Wang et al. (2024) Wang, J., Ma, W., Sun, P., Zhang, M., and Nie, J. Understanding user experience in large language model interactions. CoRR , abs/2401.08329, 2024.

Xu et al. (2023) Xu, C., Guo, D., Duan, N., and McAuley, J. J. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. In EMNLP , 2023.

Zamani et al. (2020) Zamani, H., Dumais, S. T., Craswell, N., Bennett, P. N., and Lueck, G. Generating clarifying questions for information retrieval. In WWW , 2020.

Zamfirescu-Pereira et al. (2023) Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., and Yang, Q. Why johnny can’t prompt: How non-ai experts try (and fail) to design LLM prompts. In CHI . ACM, 2023.

Zhang et al. (2025) Zhang, C., Dai, X., Wu, Y., Yang, Q., Wang, Y., Tang, R., and Liu, Y. A survey on multi-turn interaction capabilities of large language models. arXiv preprint arXiv:2501.09959 , 2025.

Zhang & Choi (2023) Zhang, M. J. Q. and Choi, E. Clarify when necessary: Resolving ambiguity through interaction with lms. CoRR , abs/2311.09469, 2023.

Zhao & Dou (2024) Zhao, Z. and Dou, Z. Generating multi-turn clarification for web information seeking. In WWW , 2024.

Zheng et al. (2023) Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS , 2023.

Zhou et al. (2024) Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. Archer: Training language model agents via hierarchical multi-turn RL. In ICML , 2024.

Zhuo et al. (2024) Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 , 2024.

Claim of Public Resource Release

公开资源声明

The following components will be made publicly available to facilitate reproducibility and further research:

为便于复现与后续研究，将公开以下组件：

• • Code: The full implementation, including dataset generation and training pipelines necessary for reproducing the reported results in our simulated conversational environments will be made publicly available.

• • 代码： 完整实现，含在仿真对话环境中复现报告结果所需的数据生成与训练流水线，将公开。

• • Training Datasets: The offline datasets used for Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) will be released, subject to licensing agreements of the original datasets.

• • 训练数据集： 用于监督微调（SFT）与直接偏好优化（DPO）的离线数据将发布，但须遵守原始数据集许可协议。

• • Models: We plan to release the fine-tuned CollabLLMs, following a thorough safety assessment and compliance review to ensure responsible deployment.

• • 模型： 计划在充分安全评估与合规审查后发布微调后的 CollabLLM，以确保负责任部署。

Appendix A Supplementary Discussion

附录 A 补充讨论

A.1 Connection Between Multiturn-aware Reward and Causal Inference

A.1 多轮感知奖励与因果推断的联系

Our approach naturally aligns with causal inference principles, as it aims to quantify how a model’s response influences the future trajectory of a conversation. This aligns with the fundamental goal of causal effect estimation, which seeks to isolate the impact of an intervention—in this case, a model response—on long-term outcomes.

我们的方法自然对齐因果推断原则：目标是量化模型回复如何影响对话的未来轨迹。这与因果效应估计的基本目标一致——隔离干预（此处即模型回复）对长期结果的影响。

From a causal perspective, given a conversation history $t^{h}_{j}$ at turn $j$, the causal effect of a model response $m_{j}$ on the final conversation trajectory can be expressed using front-door adjustment (Pearl 2009; Pearl et al. 2016):

从因果视角，给定第 $j$ 轮历史 $t^{h}_{j}$，模型回复 $m_{j}$ 对最终对话轨迹的因果效应可用前门调整（Pearl 2009；Pearl et al. 2016）表示为：

$$\sum R^{*}(t_{1:K}\mid g)P(t_{1:K}\mid t^{h}_{j})P(t^{h}_{j})=\sum R^{*}(t_{1:K}\mid g)P(t_{1:K}\mid t^{h}_{j})=\mathbb{E}_{t_{1:K}\sim P(t_{1:K}\mid t^{h}_{j})}R^{*}(t_{1:K}\mid g). \tag{5}$$

This equation captures the expected long-term reward of a conversation conditioned on the model’s response at turn $j$. It explicitly accounts for how $m_{j}$ intervenes in the conversation, influencing future turns and, ultimately, task success.

该式刻画以第 $j$ 轮模型回复为条件的对话期望长期奖励，显式计入 $m_{j}$ 如何介入对话、影响后续轮次并最终影响任务成功。

A.2 Distinction from Other Multiturn Training Frameworks

A.2 与其他多轮训练框架的区别

Existing multiturn trajectory-based training frameworks (Shani et al. 2024; Zhou et al. 2024; Gao et al. 2024) primarily rely on learning from observed trajectory-level rewards. These methods estimate the utility of responses by assigning rewards post hoc to completed conversations, training models to imitate historically successful responses. However, this approach is fundamentally correlational—it captures statistical associations between responses and final outcomes but does not explicitly model how different responses causally influence future turns.

既有基于轨迹的多轮训练框架（Shani et al. 2024；Zhou et al. 2024；Gao et al. 2024）主要从观测到的轨迹级奖励学习：事后给完整对话赋奖，训练模型模仿历史上成功的回复。该方法本质是相关的——捕捉回复与最终结果的统计关联，却不显式建模不同回复如何因果地影响后续轮次。

In contrast, our Multiturn-aware Reward (MR) framework intervenes on model responses and uses forward sampling to explicitly simulate alternative future trajectories. This enables the model to estimate the counterfactual impact of different responses and adjust its behavior accordingly. By leveraging causal effect estimation, MR training moves beyond passive imitation of high-reward conversations and instead actively optimizes responses to maximize long-term goal attainment. This distinction is crucial in dynamic human-LLM interactions where user needs evolve throughout a conversation.

相比之下，我们的多轮感知奖励（MR）框架对模型回复做干预，并用前向采样显式仿真替代未来轨迹，使模型估计不同回复的反事实影响并据此调整行为。借助因果效应估计，MR 训练超越对高奖励对话的被动模仿，主动优化回复以最大化长期目标达成。在用户需求随对话演变的动态人机交互中，这一区别至关重要。

Appendix B Dataset and Training Details

附录 B 数据与训练细节

B.1 Dataset Generation for Offline Training

B.1 离线训练的数据生成

Figure 8: Generating high-quality conversation data with Multiturn-aware Rewards (MR).

图 8：用多轮感知奖励（MR）生成高质量对话数据。

The Multiturn-aware Reward (MR) function enables the generation of high-quality synthetic conversation datasets for training. Given a user query, multiple LLM responses are sampled and ranked based on their MR scores, with higher-ranked responses designated as Chosen and lower-ranked as Rejected. To simulate natural conversational flow, the first turn from the chosen response’s forward interaction window is appended to the prompt for the next turn, iteratively extending the conversation until completion. Solid red arrows denote data collection for Supervised Fine-Tuning (SFT), while dashed blue arrows indicate preference data construction for Direct Preference Optimization (DPO). This approach systematically curates multiturn conversations that enhance both response quality and collaborative efficiency, both of which are explicitly captured by MR.

多轮感知奖励（MR）可用于生成高质量合成对话训练数据。给定用户查询，采样多个 LLM 回复并按 MR 排序，排名高者为Chosen，低者为Rejected。为模拟自然对话流，把选中回复前向交互窗口的第一轮接到下一轮提示，迭代延展直至结束。实心红箭头表示监督微调（SFT）的数据收集，虚线蓝箭头表示直接偏好优化（DPO）的偏好数据构建。该方法系统化整理多轮对话，同时提升回复质量与协作效率——二者均由 MR 显式刻画。

Given (1) a user simulator LLM, *e.g.,*GPT-4o-mini, (2) an assistant LLM, GPT-4o, and (3) arbitrary tasks with defined task-specific metric, we can simulated and generate high-quality conversations following Figure 8. We create the following training datasets in this simulated environments.

给定 (1) 用户仿真 LLM（如 GPT-4o-mini）、(2) 助手 LLM（GPT-4o）以及 (3) 带有任务指标的任意任务，即可按图 8 仿真并生成高质量对话。我们在该仿真环境中构建下列训练数据。

	# Train	# Turns	Average # Turns	Chosen MR	Rejected MR
MediumDocEdit-Chat	500	2,303	4.61	0.312 $\pm$0.104	0.246 $\pm$0.113
BigCodeBench-Chat	500	2,627	5.25	0.494 $\pm$0.621	0.207 $\pm$0.763
MATH-Chat	500	2,527	5.05	0.863 $\pm$0.524	0.547 $\pm$0.502

Table 5: Statistics of conversational datasets created from MR. Chosen/Rejected MR indicates the mean and standard deviation (mean $\pm$ std) of MRs for chosen and rejected responses (*cf.*Figure 8).

	# Train	# Turns	Average # Turns	Chosen MR	Rejected MR
MediumDocEdit-Chat	500	2,303	4.61	0.312 $\pm$0.104	0.246 $\pm$0.113
BigCodeBench-Chat	500	2,627	5.25	0.494 $\pm$0.621	0.207 $\pm$0.763
MATH-Chat	500	2,527	5.05	0.863 $\pm$0.524	0.547 $\pm$0.502

表 5：由 MR 构建的对话数据统计。Chosen/Rejected MR 表示选中与拒绝回复 MR 的均值 ± 标准差（见图 8）。

B.2 Training Details

B.2 训练细节

We provide the hyperparameters for CollabLLM fine-tuning in Table 6.

表 6 给出 CollabLLM 微调超参。

Notably, CollabLLM relies on a minimal set of hyperparameters, using the same window size and sample size for computing MRs across multiple datasets. The penalty factor on token count, $\lambda$, is set lower for MediumDocEdit-Chat compared to BigCodeBench-Chat and MATH-Chat, as document lengths in MediumDocEdit-Chat can vary significantly and may be easily bounded by 1 in Eq. 4 if $\lambda$ is too large.

值得注意的是，CollabLLM 超参很少：多数据集计算 MR 时使用相同窗口与采样数。token 数惩罚因子 $\lambda$ 在 MediumDocEdit-Chat 上设得低于 BigCodeBench-Chat 与 MATH-Chat，因前者文档长度差异大，若 $\lambda$ 过大，式 (4) 中效率项易被 1 截断。

LoRA Configuration
Rank $r$	32
Scaling factor $\alpha$	16
Dropout	0.1
Bias	False

Fine-Tuning Hyperparameters
	SFT	Offline DPO	Online DPO	PPO
Learning rate	1e-5	5e-6	5e-6	2e-6
Total batch size	64	64	32	64
Number of epochs	3	8	1	5

CollabLLM-specific Hyperparameters
	MediumDocEdit-Chat	BigCodeBench-Chat	MATH-Chat
Window size $w$	2	2	2
Sample size for MR	3	3	3
Penalty $\lambda$	1e-4	5e-4	5e-4

Table 6: Hyperparameters for LoRA configuration, different stages of fine-tuning, and CollabLLM-specific fine-tuning.

LoRA 配置
Rank $r$	32
Scaling factor $\alpha$	16
Dropout	0.1
Bias	False

微调超参
	SFT	Offline DPO	Online DPO	PPO
Learning rate	1e-5	5e-6	5e-6	2e-6
Total batch size	64	64	32	64
Number of epochs	3	8	1	5

CollabLLM 专用超参
	MediumDocEdit-Chat	BigCodeBench-Chat	MATH-Chat
Window size $w$	2	2	2
Sample size for MR	3	3	3
Penalty $\lambda$	1e-4	5e-4	5e-4

表 6：LoRA 配置、各阶段微调及 CollabLLM 专用超参。

Appendix C Prompts

附录 C 提示词

C.1 User Simulator

C.1 用户仿真器

C.2 Prompt for Proactive Base

C.2 Proactive Base 的提示

C.3 System Prompt

C.3 系统提示

C.4 Interactivity Metric by LLM Judge

C.4 LLM 评判的交互性指标

C.5 Helpfulness Reward by LLM Judge

C.5 LLM 评判的有用性奖励

Appendix D Question Template and Example on Abg-CoQA

附录 D Abg-CoQA 的问题模板与样例

We use the following prompt format for the LLMs to answer the question given a story.

我们用下列提示格式，让 LLM 在给定故事后回答问题。

For example:

例如：

The label of the above question is ambiguous since the user’s query about ‘‘Where did they go when they left?’’ could mean ‘‘Where did they go when they left the zoo?’’ or ‘‘Where did the grandparents go when they left me?’’.

上例问题标为含糊：用户问「他们离开后去了哪里？」既可能指「离开动物园后去哪？」，也可能指「祖父母离开我之后去哪？」。

Appendix E User Study

附录 E 用户研究

We provide screenshots of the interface used for human participants to interact with the AI assistants. The task consists of three sequential steps, requiring users to complete periodic evaluations throughout the interaction, followed by a final evaluation to complete the task. All data collection is fully anonymized to ensure user privacy.

我们提供真人与 AI 助手交互所用界面的截图。任务分三步：交互过程中周期性评价，最后再做总评以完成任务。数据采集全程匿名，以保护隐私。