LLMs Get Lost In Multi-Turn Conversation

LLMs Get Lost In Multi-Turn Conversation

大模型会在多轮对话中迷路

Philippe Laban † † thanks: Equal Contributions Affiliation: Microsoft Research Email: plaban@microsoft.com Hiroaki Hayashi Affiliation: Salesforce Research Email: jenneville@microsoft.com Yingbo Zhou Affiliation: Salesforce Research Email: hiroakihayashi@salesforce.com Jennifer Neville Affiliation: Microsoft Research Email: yingbo.zhou@salesforce.com

Abstract

摘要

Large Language Models (LLMs) are conversational interfaces. As such, LLMs have the potential to assist their users not only when they can fully specify the task at hand, but also to help them define, explore, and refine what they need through multi-turn conversational exchange. Although analysis of LLM conversation logs has confirmed that underspecification occurs frequently in user instructions, LLM evaluation has predominantly focused on the single-turn, fully-specified instruction setting. In this work, we perform large-scale simulation experiments to compare LLM performance in single- and multi-turn settings. Our experiments confirm that all the top open- and closed-weight LLMs we test exhibit significantly lower performance in multi-turn conversations than single-turn, with an average drop of 39% across six generation tasks. Analysis of 200,000+ simulated conversations decomposes the performance degradation into two components: a minor loss in aptitude and a significant increase in unreliability. We find that LLMs often make assumptions in early turns and prematurely attempt to generate final solutions, on which they overly rely. In simpler terms, we discover that when LLMs take a wrong turn in a conversation, they get lost and do not recover .

大语言模型（LLM）本身就是对话接口。因此，它们不仅应能在用户把任务一次性说清楚时提供帮助，还应能通过多轮交谈，帮用户定义、探索并逐步明确自己到底需要什么。对大模型对话日志的分析已经证实，用户指令里经常出现欠指定；可现有评测仍主要盯着单轮、指令完整的设定。本文用大规模模拟实验，比较大模型在单轮与多轮设定下的表现。实验证实，我们测试的所有顶尖开源权重与闭源权重模型，在多轮对话中的表现都明显低于单轮，六个生成任务上平均下降 39%。对 20 万段以上模拟对话的分析，把这种下降拆成两部分：能力上限（aptitude）只有小幅损失，不可靠性（unreliability）则显著上升。我们还发现，模型常常在早期轮次里做出假设，过早尝试生成最终解答，并过度依赖这些解答。说得更直白一些： 一旦大模型在对话中走错一步，它们就会迷路，而且回不来 。

Microsoft/lost_in_conversation datasets/Microsoft/lost_in_conversation

Figure 1: In this work, we simulate single- and multi-turn conversations for six generation tasks. The 15 LLMs we test perform much worse in multi-turn settings (-35%) explained by some loss in aptitude, and large losses in reliability. Aptitude is defined as performance in best-case conversation simulation, and unreliability as the gap between best- and worst-case performance. In short, we find that LLMs get lost in multi-turn, underspecified conversation.

图 1：本文为六项生成任务模拟了单轮与多轮对话。我们测试的 15 个大模型在多轮设定下差得多（-35%），原因是能力上限有所下降，可靠性则大幅变差。能力上限定义为最佳情形模拟中的表现，不可靠性定义为最佳与最差情形之间的差距。简而言之，我们发现大模型会在多轮、欠指定的对话中迷路。

1 Introduction

1 引言

Today’s large language models (LLMs) function as conversational interfaces (e.g., ChatGPT, Gemini, Claude), enabling users to interact with the LLM through multiple conversation turns. Such interaction promises to help users not only when they know what they need (i.e., they can fully specify their requirements in an instruction), but also when they don’t. In such cases, users might start with an underspecified instruction and further clarify their needs through turn interactions. Though studies of LLM conversation logs have confirmed that underspecification in user instructions is prevalent [27], LLM systems are typically evaluated in single-turn, fully-specified settings.

今天的大语言模型（LLM）本身就是对话接口（ 例如 ChatGPT、Gemini、Claude），用户可以通过多轮对话与模型交互。这种交互不仅承诺在用户已经知道自己需要什么时提供帮助（也就是能够把要求完整写进一条指令），也承诺在用户还说不清楚时提供帮助。后一种情形里，用户可能先给出一条欠指定的指令，再通过后续轮次澄清需求。尽管对大模型对话日志的研究已经证实，用户指令中的欠指定非常普遍 [27]，大模型系统的评测却通常仍采用单轮、指令完整的设定。

Even though a growing body of work proposes to evaluate LLMs in a multi-turn fashion, we identify in our review (Section 2) that most prior work treats the conversation as episodic : conversation turns might relate to each other, but the conversation can effectively be decomposed as an array of subtasks that can be evaluated in isolation. We argue that episodic tasks move away from what is prevalent in human conversation: underspecification [91, 27].

尽管已有越来越多的工作主张用 多轮 方式评测大模型，我们在综述（第 2 节）中指出，此前大多数工作把对话当作 情节式 （episodic）来处理：各轮之间或许相关，但整段对话实质上可以拆成一组能单独打分的子任务。我们认为，情节式任务偏离了人类对话中更常见的现象：欠指定 [91, 27]。

In this work, we close this gap by creating a simulation environment for multi-turn underspecified conversations - sharded simulation - that leverages existing instructions from high-quality single-turn benchmarks. At a high level, the sharding process we propose transforms existing single-turn instructions into sharded instructions, a set of smaller instructions that jointly deliver the same information as the original instruction. Sharded simulation then ensures that each turn of conversation reveals at most one shard of information per conversation turn, enforcing that the instruction is gradually revealed through the conversation.

本文通过构建一个面向多轮欠指定对话的模拟环境来填补这一空白。这套分片模拟（sharded simulation）复用高质量单轮基准里的现成指令。概括地说，我们提出的分片过程把原有的单轮指令改写成 分片指令 ，也就是一组更小的指令，合在一起传递与原指令相同的信息。分片模拟随后保证每一轮对话最多揭示一片信息，从而迫使指令在对话过程中被逐步揭开。

On the set of tasks that we experimented on, we observed that models engaged in multi-turn underspecified conversations achieved an average performance of 65%-a 25-point drop from single-turn performances of 90% when they receive the entire instruction at the beginning of the conversation. Notably, we observe this drop in performance even in two-turn conversations, and across all LLMs we test, from small open-weights (LLama3.1-8B-Instruct) to state-of-the-art (Gemini 2.5 Pro).

在我们实验过的任务上，进入多轮欠指定对话的模型平均成绩为 65%，比它们在对话一开始就收到完整指令时的单轮成绩 90% 低了 25 个百分点。值得注意的是，即便只有两轮对话，我们也能观察到这种下降，而且它出现在我们测试的所有大模型上，从较小的开源权重模型（Llama3.1-8B-Instruct）到最强的模型（Gemini 2.5 Pro）都是如此。

Furthermore, we decompose the performance degradation into two components: (1) loss in aptitude, and (2) increase in unreliability. We find that in single-turn settings, models with higher aptitude tend to be more reliable (e.g., GPT-4.1, Gemini 2.5 Pro). On the other hand, all LLMs exhibit very high unreliability in multi-turn settings, regardless of aptitude. We refer to this as the lost in conversation phenomenon : when LLMs take a wrong turn in multi-turn conversation, they get lost and do not recover.

我们进一步把成绩下降拆成两个部分：（1）能力上限的损失，以及（2）不可靠性的上升。我们发现，在单轮设定里，能力上限更高的模型往往也更可靠（ 例如 GPT-4.1、Gemini 2.5 Pro）。另一方面，所有大模型在多轮设定里都表现出很高的不可靠性，与能力上限无关。我们把这称作 对话迷路现象 ：大模型一旦在多轮对话中走错一步，就会迷路，而且回不来。

We investigate several explanations for this effect and show that the LLMs tend to (1) generate overly verbose responses, leading them to (2) propose final solutions prematurely in conversation, (3) make incorrect assumptions about underspecified details, and (4) rely too heavily on previous (incorrect) answer attempts.

我们考察了几种可能的解释，并表明大模型往往会（1）生成过于冗长的回复，进而（2）过早在对话中提出最终解答，（3）对欠指定的细节做出错误假设，以及（4）过度依赖此前（并不正确的）答题尝试。

Our findings highlight a gap between how LLMs are used in practice and how the models are being evaluated. Ubiquitous performance degradation over multi-turn interactions is likely a reason for low uptake of AI systems [73, 4, 28], particularly with novice users who are less skilled at providing complete, detailed instructions from the onset of conversation [87, 35].

这些发现凸显了大模型在实际中的用法与模型被评测的方式之间的差距。多轮交互中几乎无处不在的成绩下降，很可能是人工智能系统采用率偏低的一个原因 [73, 4, 28]，对那些不擅长在对话一开始就给出完整、细致指令的新手用户尤其如此 [87, 35]。

The rest of the paper is structured as follows: Section 2 situates our work with respect to prior work on multi-turn evaluation. In Section 3, we describe the simulation environment we built for both single- and multi-turn conversations on a diverse set of generation tasks. We introduce the six tasks and the metrics we use to evaluate the aptitude and reliability of models in Section 4.1. Sections 5-6 define our main experiment involving 15 LLMs, and analyze the main findings. Finally, the Implications section (Section 7) discusses the ramifications of the work, from the perspective of organizations that are building LLM-based conversation products, to that of end-users of the LLM-based systems. We provide actionable recommendations based on small-scale experiments and make a concrete call-to-action to LLM builders, urging them to prioritize multi-turn reliability in conjunction with aptitude in future model iterations.

后文安排如下：第 2 节把本文放在已有多轮评测工作的脉络里。第 3 节描述我们为多样化生成任务构建的单轮与多轮对话模拟环境。第 4.1 节介绍六项任务，以及我们用来衡量模型能力上限与可靠性的指标。第 5-6 节给出涵盖 15 个大模型的主实验，并分析主要发现。最后，影响一节（第 7 节）从正在打造大模型对话产品的组织，一直讨论到这些系统的终端用户。我们依据小规模实验给出可操作的建议，并向大模型构建者发出明确呼吁：在后续模型迭代中，应把多轮可靠性与能力上限放在一起优先考虑。

2 Background and Related Work

2 背景与相关工作

Previous-generation language models (e.g., BART [45], GPT-2 [65], or T5 [66]) were not equipped to handle multi-turn conversations, which led evaluation to focus on single-turn tasks [79]. Conversational AI was typically implemented as specialized systems that leveraged language models as components [36], and were evaluated through human protocols [17, 42, 21, 54], or competitions like Amazon’s Alex Prize [67].

上一代语言模型（例如 BART [45]、GPT-2 [65] 或 T5 [66]）还不具备处理多轮对话的能力，评测因此集中在单轮任务上 [79]。对话式人工智能通常被做成专用系统，把语言模型当作其中的组件来用 [36]，评测则依靠人工协议 [17, 42, 21, 54]，或像 Amazon 的 Alexa Prize 这样的竞赛 [67]。

As the meteoric rise of ChatGPT led to increased interest in multi-turn evaluation, initial popular efforts such as MT-bench [89] leveraged crowd-sourced annotations to evaluate LLM-as-a-judge ability. Follow-up works expanded on MT-bench, for instance to include longer conversations [37, 18], increase evaluation granularity [2], or to tackle different aspects such as naturalness [72] or tool use [85, 80].

随着 ChatGPT 迅速走红，多轮评测的兴趣随之上升。较早的热门工作如 MT-bench [89] 借助众包标注，来衡量用大模型当评判者的能力。后续工作在 MT-bench 上继续扩展，例如纳入更长的对话 [37, 18]、提高评测粒度 [2]，或处理自然度 [72]、工具使用 [85, 80] 等不同侧面。

Crucially, such works typically simulate episodic conversations: each turn in the conversation introduces a subtask that relates to previous conversation turns, but can be evaluated in isolation. In this work, we find that episodic tasks overestimate LLM performance in multi-turn conversations (see Section 7.3). In short, although episodic tasks require some level of multi-turn context understanding, they do not involve actively fusing the information to answer underspecified user instructions. Underspecified user instructions are not only common in real-world human-AI communication [27], but also a natural tendency in conversations, termed “the principle of least effort” [91]. We show that underspecification in multi-turn conversations leads to large and universal performance degradations: LLMs make early assumptions to fill in for missing information, prematurely attempt to propose finalized solutions, and have difficulty adapting and course-correcting when provided with new information. We make underspecification the central element of our evaluation setting.

关键在于，这类工作通常模拟的是 情节式 对话：每一轮引入一个与前面轮次相关的子任务，但这个子任务可以单独打分。本文发现，情节式任务会高估大模型在多轮对话中的表现（见第 7.3 节）。简而言之，情节式任务虽然要求一定程度的多轮上下文理解，却并不要求模型主动把信息融合起来，去回答 欠指定 的用户指令。欠指定的用户指令不仅在真实的人机沟通中很常见 [27]，也是对话中一种自然倾向，被称为“最小努力原则” [91]。我们表明，多轮对话中的欠指定会带来巨大且普遍的成绩下降：大模型会用早期假设去填补缺失信息，过早尝试提出最终解答，并且在获得新信息时难以调整和纠偏。我们把欠指定作为评测设定的核心要素。

Multi-turn episodic evaluation is sometimes framed as a way to evaluate multi-turn model capabilities with higher granularity. Categories of subtasks (such as refinement, follow-up, expansion, etc.) allow for the study of more specific LLM behavior [2, 37, 74, 19, 16, 48, 25]. According to such framing, multi-turn tasks differ from single-turn tasks and are not evaluated on the same set of tasks. We argue that this framing is artificial and limits the scope of multi-turn evaluation, restricting the direct comparison of multi-turn and single-turn abilities of LLMs. In our work, we conduct both single-turn and multi-turn conversation simulations on a common set of tasks : controlled experiments that precisely allow us to identify performance degradations from single- to multi-turn settings.

多轮情节式评测有时被说成是一种能以更高粒度衡量多轮能力的办法。子任务类别（如 refinement、follow-up、expansion 等）让人可以更具体地观察大模型行为 [2, 37, 74, 19, 16, 48, 25]。按这种说法，多轮任务与单轮任务 不同 ，也不会在同一组任务上评测。我们认为这种框架是人为的，它限制了多轮评测的范围，也限制了人们直接比较大模型的多轮能力与单轮能力。本文在 同一组任务 上同时做单轮和多轮对话模拟：正是这种受控实验，让我们能够精确识别从单轮到多轮的成绩下降。

Evaluating LLMs in multi-turn settings is a challenge because conversational trajectories diverge far more than in a single-turn. Thus, most previous studies have focused on classification or short-form tasks, with more straightforward evaluation settings. However, the predominant use cases for LLMs are generative in nature, both for programming (e.g., coding assistants) and natural language (e.g., writing, summarizing) [88, 26]. Long-form evaluation in the multi-turn setting is therefore essential, as it assesses models’ ability to flexibly adapt and refine the response as the users provide more information. In this work, we focus exclusively on generation tasks that capture widely used scenarios in both programming and natural language domains.

在多轮设定里评测大模型很难，因为对话轨迹比单轮分叉得多。因此，以往研究大多聚焦分类或短答案任务，评测设定更直截了当。可大模型的主流用途是生成性的，既包括编程（ 例如 编程助手），也包括自然语言（ 例如 写作、摘要）[88, 26]。因此，多轮设定下的长文本评测必不可少，因为它衡量的是：当用户不断补充信息时，模型能否灵活调整并改进回复。本文只关注生成任务，这些任务对应编程和自然语言两个领域里广泛使用的场景。

Scaling multi-turn experimentation requires simulating a user. Existing studies implemented such user simulation in different ways: relying on templates [12, 68, 39, 16], using an LLM [63, 46, 7, 48], involving human annotators [21, 7], or real users as part of a study [67, 38, 11]. Although involving real users leads to the most natural and realistic conversations, it comes at the cost of scalability and reproducibility. In this work, we adopt an LLM-based simulator to enable controlled flexibility and divergence. Nevertheless, a fully automated simulation limits the scope of our findings: the conversations we simulate are not representative of human-AI conversations. We therefore frame the simulation as a tool to study the LLM behavior in the multi-turn setting rather than user behavior. In addition, as detailed in the Limitations Section (Section 9), we argue that our simulation framework is simplistic and idealized. For example, the conversations are guaranteed to end with sufficient information to solve the tasks, and the simulator limits unexpected behavior (e.g., derailing) that can occur in real-world settings. We suggest these choices imply that degradations observed in this work are most likely underestimates of what occurs in real-world, underspecified multi-turn Human-AI conversations. Appendix A introduces other related work specifically focused on underspecified communication.

要把多轮实验做大规模，就需要模拟用户。已有研究用不同方式实现这种用户模拟：依赖模板 [12, 68, 39, 16]，使用大模型 [63, 46, 7, 48]，请标注员参与 [21, 7]，或在研究中直接使用真实用户 [67, 38, 11]。请真实用户参与能得到最自然、最贴近现实的对话，代价是难以规模化和复现。本文采用基于大模型的模拟器，以便在受控条件下保留灵活性和分叉。不过，完全自动化的模拟也限制了发现的适用范围：我们模拟出的对话并不能代表真实的人机对话。因此，我们把模拟定位为研究大模型在多轮设定中 行为 的工具，而不是研究用户行为的工具。此外，正如局限一节（第 9 节）所详述的，我们认为这套模拟框架偏简单、偏理想化。例如，对话被保证会在信息足以完成任务时结束，模拟器也限制了现实中可能出现的意外行为（ 例如 跑题）。我们认为，这些选择意味着本文观察到的下降，很可能低估了真实世界里欠指定的多轮人机对话中会发生的情况。附录 A 介绍了其他专门讨论欠指定沟通的相关工作。

3 Simulating Underspecified, Multi-Turn Conversation

3 模拟欠指定的多轮对话

To assess LLM performance in multi-turn, underspecified conversation, we develop a simulation environment that repurposes existing tasks from single-turn benchmarks. First, we apply a sharding process to transform original fully-specified instructions into sharded instructions . Second, we implement a sharding simulation environment that carries out a multi-turn conversation based on a sharded instruction.

为了评估大模型在多轮、欠指定对话中的表现，我们开发了一个模拟环境，把单轮基准中的现成任务重新利用起来。首先，我们用 分片过程 把原先完整指定的指令改写成 分片指令 。其次，我们实现一个分片模拟环境，依据分片指令展开多轮对话。

3.1 Sharding Process: From Fully-Specified to Sharded Instructions An original, fully-specified instruction from GSM8K [14] and the equivalent sharded instruction are listed in Figure 2.

3.1 分片过程：从完整指定到分片指令 图 2 列出了来自 GSM8K [14] 的一条原始完整指令，以及与之对应的分片指令。

The original instruction is a single, long utterance that introduces all the content at once: a high-level question (i.e., “How long will it take [\dots]”), context, and conditions. The sharded instruction is composed of a set of shards, each introducing a single element from the original instruction. More specifically, the first shard (Shard 1) of a sharded instruction always introduces the high-level intent for the instruction, and subsequent shards each provide clarification to the instruction. Taken jointly, the set of shards reflects the same information provided in the fully-specified instruction, with the information explicitly divided across shards.

原始指令是一段一次性给出全部内容的长话语：一个高层问题（ 即 “需要多久\dots\dots”）、上下文和条件。分片指令由 一组碎片 组成，每一片只引入原指令中的一个要素。更具体地说，分片指令的第一片（Shard 1）总是引入该指令的高层意图，后续各片分别提供澄清。合在一起，这组碎片反映的信息与完整指定的指令相同，只是这些信息被明确拆到了不同碎片里。

In Appendix B, we provide a more precise and mathematical definition of a sharded instruction in relation to the original fully-specified instruction, and define five key properties a sharded instruction must satisfy to be considered valid.

在附录 B 中，我们给出分片指令相对于原始完整指令的更精确、更形式化的定义，并给出一条有效的分片指令必须满足的五条关键性质。

As part of our work, we developed a semi-automatic sharding process to scale the creation of sharded instructions. This process, described in depth in Appendix C, ensured that the experiments we carried out used sharded instructions that adhered to the properties we defined.

作为工作的一部分，我们开发了一套半自动分片流程，以便规模化地制作分片指令。该流程在附录 C 中有详细说明，它保证实验所用的分片指令符合我们定义的性质。

Fully-Specified Instruction (original) Jay is making snowballs to prepare for a snowball fight with his sister. He can build 20 snowballs in an hour, but 2 melt every 15 minutes. How long will it take before he has 60 snowballs? Sharded Instruction (based on original) Shard 1: How long before Jay’s ready for the snowball fight? Shard 2: He’s preparing for a snowball fight with his sister. Shard 3: He can make 20 snowballs per hour. Shard 4: He’s trying to get to 60 total. Shard 5: The problem is that 2 melt every 15 minutes. (a) Original GSM8K instruction.

完整指定的指令（原始） Jay 正在堆雪球，准备和妹妹打雪仗。他一小时能堆 20 个雪球，但每 15 分钟会融化 2 个。他要过多久才能有 60 个雪球？ 分片指令（基于原始指令） 碎片 1：Jay 还要多久才能准备好打雪仗？ 碎片 2：他正准备和妹妹打雪仗。 碎片 3：他每小时能做 20 个雪球。 碎片 4：他想凑到总共 60 个。 碎片 5：问题是每 15 分钟会融化 2 个。 (a) 原始 GSM8K 指令。

3.2 Simulating Sharded Conversations Figure 3: Sharded Conversation Simulation Diagram. The subject for the simulation is highlighted in red.

3.2 模拟分片对话 图 3：分片对话模拟示意图。模拟中的受试对象用红色标出。

Figure 3 depicts the process of simulating a multi-turn, underspecified conversation based on a sharded instruction. At a high-level, the conversation involves three parties: the assistant is the LLM being evaluated in the simulation, the user (simulated by an LLM) who has access to the entirety of the sharded instruction and is in charge of revealing shards during turns of the conversation, and the system which categorizes and evaluates assistant responses.

图 3 描绘了如何基于一条分片指令，模拟一段多轮、欠指定的对话。概括地说，对话涉及三方： 助手 是模拟中被评测的大模型； 用户 （由大模型模拟）掌握整条分片指令，负责在各轮中揭示碎片； 系统 则对助手回复分类并打分。

On the first turn, the user simulator reveals the first shard of the instruction (i.e., Shard 1) to the assistant, which then generates a free text response. The system processes the assistant’s response into one of seven possible response strategies: clarification, refusal, hedging, interrogation, discussion, missing, or answer attempt,[^1] based on Herlihy et al. [27]’s LLM response categorization. If the assistant generates an answer attempt (i.e., proposing an explicit, full-form solution), then the answer extractor component determines the span that corresponds to the answer within the assistant’s free-form response (e.g., code snippet, number). This step is required because LLMs often pad answer attempts with additional information, such as a natural-language explanation or a follow-up question, which could hinder evaluation. Finally, the extracted answer is scored by a task-specific evaluator function. Subsequent turns follow a similar pattern: at each turn, the user simulator reveals at most one shard of information, the assistant responds freely, which gets evaluated if the response is classified as an answer attempt. The conversation ends if one of two conditions is met: (1) the task-evaluator assesses that an assistant answer attempt is correct, or (2) if at the start of a new turn, the user simulator has run out of shards to reveal in the conversation.

第一轮，用户模拟器向助手揭示指令的第一片（ 即 Shard 1），助手随后生成自由文本回复。系统把助手的回复归入七种可能的回复策略之一： 澄清 、 拒绝 、 含糊其辞 、 追问 、 讨论 、 缺失 或 答题尝试 ，[^1] 依据是 Herlihy 等人 [27] 对大模型回复的分类。如果助手生成了答题尝试（ 即 提出一个明确、完整形式的解答），答案抽取组件就从助手的自由文本回复中确定对应答案的片段（ 例如 代码片段、数字）。这一步是必要的，因为大模型常常会在答题尝试外再垫上额外信息，比如自然语言解释或后续提问，这可能妨碍评测。最后，抽取出的答案由任务专用的评估函数打分。后续轮次遵循类似模式：每一轮，用户模拟器最多揭示一片信息，助手自由作答；若该回复被判定为答题尝试，就会被打分。对话在满足以下两个条件之一时结束：（1）任务评估器判定某次助手答题尝试是正确的，或（2）新的一轮开始时，用户模拟器已经没有可揭示的碎片了。

Preliminary experiments revealed that during simulation, evaluated assistants often asked clarification questions that related to specific shards of the instruction. As such, deciding which shard to reveal next in the conversation (the role of the user simulator) is non-trivial, as it should take into account the state of the conversation so far. We instantiate the user simulator as a low-cost LLM (specifically, GPT-4o-mini) that has access to the entire sharded instruction and the state of the conversation so far, tasking it with deciding the next shard to reveal that fits most naturally in the ongoing simulated conversation. The user simulator is also tasked with rephrasing the shard to fit naturally within the conversation without modifying its informational content. See Appendix J for an example simulated sharded conversation.

初步实验显示，模拟过程中，被评测的助手经常提出与指令中特定碎片相关的澄清问题。因此，决定接下来揭示哪一片（这是用户模拟器的职责）并不简单，因为它应当考虑对话到目前为止的状态。我们把用户模拟器实例化为一个低成本大模型（具体是 GPT-4o-mini），它能看到整条分片指令以及截至目前的对话状态，并负责选择在当前模拟对话中显得最自然的下一片。用户模拟器还需要改写该碎片，使其自然地嵌入对话，同时不改变其信息内容。分片对话的模拟示例见附录 J。

Besides user messages, the assistant receives a minimal system instruction (before the first turn) that provides the necessary context to accomplish the task (such as a database schema or a list of available API tools). Importantly, the assistant is not explicitly informed that it is participating in a multi-turn, underspecified conversation and is not encouraged to pursue specific conversational strategies. Although such additional instructions would likely alter model behavior, we argue that such changes are not realistic, as such information is not available a priori in practical settings. In summary, we provide no information about the setting to the evaluated assistant model during simulation, aiming to assess default model behavior.

除了用户消息，助手还会在第一轮之前收到一条极简的系统指令，提供完成任务所需的必要上下文（例如数据库模式，或可用的 API 工具列表）。重要的是，助手并没有被明确告知自己正在参与一段多轮、欠指定的对话，也没有被鼓励去采取特定的对话策略。尽管这类额外指令很可能改变模型行为，我们认为这样的改动并不现实，因为在实际设定中事先得不到这些信息。总之，模拟过程中我们不向被评测的助手模型提供关于设定的信息，目的是评估模型的默认行为。

Apart from the user simulator, the strategy classifier and answer extractor components are also implemented with prompt-based GPT-4o-mini. While the choice of LLM-based components in the simulator allows for dynamic choices that provide a more realistic simulation, they also unavoidably lead to simulation errors, which can affect the validity of experiments. To understand the scope of simulation errors and their effect on simulation validity, we conducted an in-depth manual annotation of several hundred simulatesouthworth2023developingd conversations. The annotation effort and its findings are detailed in Appendix D. In summary, we found that errors introduced by the user simulator, strategy classifier, or answer extraction occurred in less than 5% of inspected conversations and that these errors disfavored the assistant model in less than 2% of the conversations. We believe the process described above can accurately simulate multi-turn, underspecified conversations based on sharded instructions, and we rely on it to simulate conversations for our experiments.

除用户模拟器外，策略分类器和答案抽取组件同样用基于提示的 GPT-4o-mini 实现。模拟器里这些基于大模型的组件虽然能做出更动态、因而也更接近真实的选择，但不可避免地会引入模拟误差，并可能影响实验有效性。为了弄清模拟误差的范围及其对有效性的影响，我们对数百段模拟对话做了深入的人工标注。标注工作及其发现详见附录 D。概括地说，我们发现用户模拟器、策略分类器或答案抽取引入的错误出现在不到 5% 的受检对话中，这些错误不利于助手模型的比例不到 2%。我们相信上述流程能够依据分片指令准确模拟多轮、欠指定的对话，并依靠它为实验模拟对话。

3.3 Simulation Types Figure 4: Conversation simulation types based on sharded instructions. Once an original fully-specified instruction (blue block) is sharded (set of yellow blocks), the “shards” can be used to simulate single-turn (Full, Concat) or multi-turn (Sharded, Recap, Snowball) conversations, affecting the pace of information disclosure.

3.3 模拟类型 图 4：基于分片指令的对话模拟类型。一旦一条原始的完整指定指令（蓝色块）被分片（一组黄色块），这些“碎片”就可以用来模拟单轮（Full、Concat）或多轮（Sharded、Recap、Snowball）对话，从而改变信息披露的节奏。

We leverage sharded instructions to simulate five types of single- or multi-turn conversations, as illustrated in Figure 4. We now introduce each one and explain its purpose in our experiments.

我们利用分片指令模拟五种单轮或多轮对话，如图 4 所示。下面依次介绍每一种，并说明它在实验中的用途。

Fully-specified (short-form: Full) simulates single-turn, fully-specified conversations in which the original instruction is provided to the LLM in the first turn. This simulation type evaluates baseline model performance on the tasks.

完整指定 （简称 Full）模拟单轮、完整指定的对话：原始指令在第一轮就提供给大模型。这种模拟类型评估模型在这些任务上的基线表现。

Sharded simulates multi-turn, underspecified conversations as outlined above. Sharded simulations are our primary tool to evaluate model performance in underspecified, multi-turn conversations.

分片 模拟上文所述的多轮、欠指定对话。分片模拟是我们评估模型在欠指定多轮对话中表现的主要工具。

Concat simulates single-turn, fully-specified conversation based on the sharded instruction. The shards are concatenated into a single instruction in bullet-point form (with one shard per line), preceded by an instruction to complete the task taking into account all bullet-points. The Concat simulation is a logical mid-point between full and sharded, in which underspecification is removed (like Full) but the rephrasing that occurred during instruction sharding is preserved (like Sharded). Concat is intended as a verification baseline: a model that succeeds at both Full and Concat, but not at Sharded, struggles specifically because of underspecification and the multi-turn nature of the conversation, and not due to the rephrasing that occurred during the sharding process, which may have led to information loss.

拼接 基于分片指令模拟单轮、完整指定的对话。各碎片被拼成一条条目式指令（每行一片），前面再加上一条要求综合所有条目完成任务的说明。Concat 模拟是 Full 与 Sharded 之间的逻辑中点：欠指定像 Full 一样被去掉了，但分片过程中发生的改写像 Sharded 一样被保留。Concat 意在作为核验基线：如果一个模型在 Full 和 Concat 上都能成功，却在 Sharded 上不行，那它卡住的原因就具体是欠指定和对话的多轮性质，而不是分片过程中的改写——那种改写有可能导致信息损失。

Recap simulates a Sharded conversation, and adds a final recapitulation turn which restates all the shards of the instruction in a single turn, giving the LLM one final attempt at responding. Recap is a combination of the Sharded simulation followed by a Concat turn, and is explored as a method in Section 7.1 to evaluate whether such a conceptually simple agent-like intervention can mitigate the loss in performance observed in Sharded conversations.

回顾 在模拟一段 Sharded 对话之后，再增加最后一轮 总述轮 ，把指令的全部碎片在单轮里重述一遍，给大模型最后一次作答机会。Recap 是 Sharded 模拟后再接一个 Concat 轮次的组合，我们在第 7.1 节把它作为一种方法来考察：这种概念上很简单、类似智能体的干预，能否缓解 Sharded 对话中观察到的成绩损失。

Snowball takes the Recap simulation a step further, implementing turn-level recapitulation. At each turn, the user simulator introduces a new shard, but also restates all the shards that have been revealed so far in the conversation, producing a snowball effect as each turn reveals all the information from the previous turn, plus one additional shard. The redundancy implemented in the Snowball simulation is also explored as a method in Section 7.1 to study whether turn-level reminders help alleviate the need for LLMs to recall information across multiple turns of context.

滚雪球 把 Recap 模拟再推进一步，实现轮次级的总述。每一轮，用户模拟器引入一片新碎片，同时也重述对话中到目前为止已揭示的全部碎片，形成 滚雪球 效应：每一轮都给出上一轮的全部信息，再加上一片新碎片。Snowball 模拟中的这种冗余，同样在第 7.1 节被当作一种方法来考察：轮次级的提醒，是否有助于减轻大模型跨多轮上下文回忆信息的负担。

4 Task and Metric Selection

4 任务与指标选择

4.1 Task Selection Figure 5: Six sharded tasks included in our experiments. We purposefully include tasks that involve generating programming and natural language. For each task, an illustrative fully-specified instruction and its sharded counterpart. We sharded 90-120 instructions based on high-quality datasets (Instruction Origin), re-purposing existing evaluation.

4.1 任务选择 图 5：实验中纳入的六项分片任务。我们有意同时纳入涉及生成程序和生成自然语言的任务。对每项任务，图中给出一条示意性的完整指令及其分片对应物。我们基于高质量数据集（指令来源）对 90-120 条指令做了分片，从而复用了已有评测。

We constructed sharded instructions for six tasks that we use in a large-scale simulation experiment. For each task, we selected instructions from one or two high-quality single-turn, fully-specified benchmarks, and implemented a semi-automatic sharding process. The process relied first on an LLM (GPT-4o) to propose and verify sharding candidates, which were then reviewed and edited (when necessary) by the authors of the work. The sharding process (outlined in detail in Appendix C) allowed us to scale the construction of sharded instruction corpora while ensuring validity of the underlying instructions. For each task, we prepared 90-120 sharded instructions (each paired with the original single-turn instructions), which required between 1-4 hours of manual inspection and annotation.

我们为六项任务构建了分片指令，并用于大规模模拟实验。对每项任务，我们从一到两个高质量的单轮、完整指定基准中选取指令，并实施半自动分片流程。该流程首先让一个大模型（GPT-4o）提出并核验分片候选，再由本文作者审阅，并在必要时编辑。分片流程（附录 C 有详细说明）让我们能够规模化地构建分片指令语料，同时保证底层指令的有效性。每项任务我们准备了 90-120 条分片指令（每条都配有原始单轮指令），所需的人工检查与标注时间在 1 到 4 小时之间。

We carefully selected popular and diverse generation tasks across programming and non-programming use cases. Figure 5 provides an example of an original and sharded instruction for each task, which we now introduce.

我们仔细挑选了编程与非编程场景中流行且多样的生成任务。图 5 为每项任务提供了一条原始指令和一条分片指令的示例，下面依次介绍。

Code The assistant must help the user write a function in the Python programming language. The original instructions were sourced from the HumanEval [10] and LiveCodeBench [31] datasets, two popular benchmarks used to evaluate LLM programming aptitude.

代码 助手需要帮助用户用 Python 编写一个函数。原始指令来自 HumanEval [10] 和 LiveCodeBench [31] 这两个常用于评估大模型编程能力的基准。

Database The assistant is provided with the schema of an SQL database and a user query in natural language, and must produce an SQL query that retrieves the requested information from the database (a.k.a., text-to-SQL). The original instructions and databases were sourced from the popular Spider dataset [86].

数据库 助手会得到一个 SQL 数据库的模式，以及一条自然语言形式的用户查询，并必须写出能从数据库中取出所需信息的 SQL 查询（也就是 text-to-SQL）。原始指令和数据库来自流行的 Spider 数据集 [86]。

Actions The assistant is provided with a set of API (Application Programming Interface) schemas, and a user instruction that requires API use, and must generate the programmatic API commands that match the user request. We sourced API schemas and user instructions from the Berkeley Function Calling Leaderboard (BFCL) [85], a popular benchmark used to measure LLM ability at API function calling.

动作 助手会得到一组 API（应用程序编程接口）模式，以及一条需要使用 API 的用户指令，并必须生成与用户请求相符的程序化 API 命令。我们从 Berkeley Function Calling Leaderboard（BFCL）[85] 获取 API 模式和用户指令，这是衡量大模型 API 函数调用能力的常用基准。

Math The assistant is provided with an elementary math word problem, and must perform a series of calculations using basic arithmetic operations to reach a numerical answer. We sourced problems from the GSM8K dataset [14].

数学 助手会得到一道小学数学应用题，必须用基本算术运算完成一系列计算，得到一个数值答案。题目来自 GSM8K 数据集 [14]。

Data-to-text The assistant is provided tabular data and several elements of related metadata, and must produce a caption (natural language sentence) describing the underlying data. We leverage the ToTTo [59] dataset to formulate sharded instructions.

数据到文本 助手会得到表格数据以及若干相关元数据，并必须写出一句描述底层数据的说明（自然语言句子）。我们借助 ToTTo [59] 数据集来构造分片指令。

Summary The assistant receives a corpus of around twenty documents and a user query, and must generate a summary with citations that addresses the query based on the documents. We re-purpose the instructions from Summary of a Haystack [40]. The summary task is the only task we include that tests long-context capabilities, with instructions spanning several tens of thousands of tokens, which is known to deteriorate model performance [29, 32, 33].

摘要 助手会收到大约二十篇文档组成的语料以及一条用户查询，并必须生成带引用的摘要，依据这些文档回应用户查询。我们复用了 Summary of a Haystack [40] 中的指令。摘要是我们纳入的唯一一项测试长上下文能力的任务，指令长度可达数万 token，而已知这会损害模型表现 [29, 32, 33]。

For each task, we reuse the metrics used in the original benchmarks. More specifically, the first four tasks (Code, Database, Actions, and Math) are evaluated for binary correctness, either by executing an answer attempt (code, SQL query), or validating semantic equivalence to a reference answer (API call, numerical answer). The last two tasks (Data-to-Text and Summary) are refinement tasks, which get scored on a continuous range (0-100). Data-to-text uses the BLEU metric [58], and Summary uses a custom LLM-as-a-judge metric (“Joint Score”) built to measure information coverage and attribution accuracy of the summary [40]. We map binary accuracy in the range of 0-100 (0 = failure, 100 = success) so that all tasks produce scores on a common scale, facilitating aggregation.

对每项任务，我们沿用原基准使用的指标。更具体地说，前四项任务（代码、数据库、动作和数学）按二元正确性评估，要么执行一次答题尝试（代码、SQL 查询），要么核验其与参考答案在语义上是否等价（API 调用、数值答案）。后两项任务（数据到文本和摘要）是* refinement 任务*，得分落在连续区间（0-100）。数据到文本使用 BLEU 指标 [58]，摘要使用定制的“大模型当评判者”指标（“Joint Score”），用来衡量摘要的信息覆盖率和归因准确性 [40]。我们把二元正确率映射到 0-100（0 = 失败，100 = 成功），以便所有任务在同一尺度上出分，便于汇总。

Appendix I lists implementation details of the sharding process for each task, including the sample selection process and any task-specific logic that was implemented to facilitate reproducibility. Even though we intended for the six selected tasks to be representative of a wide range of LLM use cases, we put effort into making the sharding process efficient and reproducible, as we see the process itself as a contribution of our work. We envision that future LLM evaluation practitioners can shard their own dataset artifacts to study LLM multi-turn behavior in more diverse and unique settings.

附录 I 列出了每项任务分片流程的实现细节，包括样本选择过程，以及为便于复现而实现的任何任务专用逻辑。尽管我们希望所选的六项任务能代表大模型的广泛用途，我们仍花力气让分片流程高效且可复现，因为我们把流程本身也视为本文的一项贡献。我们设想，未来的大模型评测实践者可以对自己的数据资产做分片，从而在更多样、更独特的设定中研究大模型的多轮行为。

S_{i}

T=1.0

\overline{P}=\displaystyle\sum_{i=1}^{N}S_{i}\big/N

A^{90}=\operatorname{percentile}_{90}(S)

U_{10}^{90}=\operatorname{percentile}_{90}(S)-\operatorname{percentile}_{10}(S).

\overline{P}

\overline{P}

R_{10}^{90}=100-U_{10}^{90}

R_{10}^{90}=100-U_{10}^{90}

\overline{P}

\overline{P}

5 Simulation Scale and Parameters

5 模拟规模与参数

N=10

N=10

\overline{P}

\overline{P}

We selected a total of 15 LLMs from eight model families: OpenAI (GPT-4o-mini, GPT-4o [30], o3 [57], and GPT-4.1), Anthropic (Claude 3 Haiku, Claude 3.7 Sonnet), Google’s Gemini (Gemini 2.5 Flash, Gemini 2.5 Pro) [75], Meta’s Llama (Llama3.1-8B-Instruct, Llama3.3-70B-Instruct, Llama 4 Scout) [23], AI2 OLMo-2-13B [56], Microsoft Phi-4 [1], Deepseek-R1 [24], and Cohere Command-A [15]. This selection prioritizes the evaluation of state-of-the-art models, including both small (8B) and large models (300B+). We purposefully include both open- and closed-weights models, as well as two reasoning models (o3, R1) to study the effect additional thinking (test-time compute) has on multi-turn conversation capability. Details on model versioning and access are listed in Appendix H. We estimate the total cost of conducting simulations to be around $5,000.

我们总共选取了来自八个模型系列的 15 个大模型：OpenAI（GPT-4o-mini、GPT-4o [30]、o3 [57] 和 GPT-4.1），Anthropic（Claude 3 Haiku、Claude 3.7 Sonnet），Google 的 Gemini（Gemini 2.5 Flash、Gemini 2.5 Pro）[75]，Meta 的 Llama（Llama3.1-8B-Instruct、Llama3.3-70B-Instruct、Llama 4 Scout）[23]，AI2 的 OLMo-2-13B [56]，Microsoft 的 Phi-4 [1]，Deepseek-R1 [24]，以及 Cohere 的 Command-A [15]。这一选择优先评测最强的模型，既包括较小的模型（8B），也包括很大的模型（300B+）。我们有意同时纳入开源权重与闭源权重模型，以及两个推理模型（o3、R1），以便研究额外思考（测试时计算）对多轮对话能力的影响。模型版本与访问方式详见附录 H。我们估计开展这些模拟的总成本约为 5000 美元。

6 Results

6 结果

\overline{P}

\overline{P}

6.1 Average Performance Findings Table 1 summarizes results from the simulation. At a high level, every model sees its performance degrade on every task when comparing Full and Sharded performance, with an average degradation of -39% . We name this phenomenon Lost in Conversation: models that achieve stellar (90%+) performance in the lab-like setting of fully-specified, single-turn conversation struggle on the exact same tasks in a more realistic setting when the conversation is underspecified and multi-turn.

6.1 平均表现方面的发现 表 1 汇总了模拟结果。概括地说， 比较 Full 与 Sharded 表现时，每个模型在每项任务上都会下降，平均下降幅度为 -39% 。我们把这种现象称为 Lost in Conversation：在完整指定的单轮对话这种实验室式设定里能拿到极高（90%+）分数的模型，在更现实的、对话欠指定且为多轮的设定里，会在 完全相同的任务 上陷入困境。

In comparison, models perform roughly equivalently in the Concat setting, with Concat performance averaging 95.1% of the Full counterpart. This implies that the loss in performance for Sharded is not explained by potential loss of information in sharded instructions, as such a loss would be reflected in lower Concat performance. We observe that smaller models (Llama3.1-8B-Instruct, OLMo-2-13B, Claude 3 Haiku) have more pronounced Concat degradations (86-92), and interpret this as indicating that smaller models struggle to generalize as well as larger models: benign rephrasing affects performance more than for larger, more robust models. This lack of robustness to paraphrasing can be observed visually in Table 1: Concat degradation (red background) is more pronounced in the top rows (weaker models) than the bottom rows (stronger models).

相比之下，模型在 Concat 设定中的表现大致相当，Concat 成绩平均相当于 Full 对应成绩的 95.1%。这意味着 Sharded 的成绩损失并不能用分片指令可能丢失信息来解释，因为那样的损失会表现为更低的 Concat 成绩。我们观察到，较小的模型（Llama3.1-8B-Instruct、OLMo-2-13B、Claude 3 Haiku）在 Concat 上的下降更明显（86-92），并将其解释为：较小模型的泛化不如较大模型稳定，无害的改写对它们成绩的影响大于对更大、更稳健模型的影响。这种对改写缺乏稳健性，可以在表 1 中直观看到：Concat 下降（红色背景）在靠上的行（较弱模型）比靠下的行（较强模型）更明显。

The last column of the Table ( / ) aggregates performance degradation across the six tasks, summarizing the magnitude of the Lost in Conversation effect for each model. Surprisingly, more performant models (Claude 3.7 Sonnet, Gemini 2.5, GPT-4.1) get equally lost in conversation compared to smaller models (Llama3.1-8B-Instruct, Phi-4), with average degradations of 30-40%. This is in part due to metric definitions. Since smaller models achieve lower absolute scores in Full, they have less scope for degradation than the better models. In short, no matter how strong an LLM’s single-turn performance is, we observe large performance degradations in the multi-turn setting.

表的最后一列（ / ）汇总六个任务上的成绩下降，概括每个模型身上 Lost in Conversation 效应的幅度。令人惊讶的是， 更强的模型（Claude 3.7 Sonnet、Gemini 2.5、GPT-4.1）与较小模型（Llama3.1-8B-Instruct、Phi-4）一样会在对话中迷路 ，平均下降 30-40%。这在一定程度上来自指标定义。较小模型在 Full 上的绝对分数更低，可供下降的空间也就小于更好的模型。简而言之，无论一个大模型的单轮表现有多强，我们都在多轮设定中观察到大幅成绩下降。

When looking at the task-specific breakdown, some models see more muted degradations in certain tasks. For instance, Command-A sees the least degradation on the Actions task, while Claude 3.7 Sonnet and GPT-4.1 conserve performance well on Code, and Gemini 2.5 Pro in the Data-to-Text task. This finding indicates that the multi-turn capabilities of models are not uniform across domains and validates the importance of benchmarking models across a wide variety of tasks to investigate model capabilities.

按任务拆开看，有些模型在某些任务上的下降更温和。例如，Command-A 在动作任务上下降最少，Claude 3.7 Sonnet 和 GPT-4.1 在代码任务上较能保住成绩，Gemini 2.5 Pro 则在数据到文本任务上较能保住成绩。这一发现表明，模型的多轮能力在不同领域并不均匀，也说明有必要在多种多样的任务上做基准测试，才能考察模型能力。

Additional test-time compute (reasoning tokens) does not help models navigate multi-turn underspecification, as the two reasoning models included in the experiment (o3, Deepseek-R1) deteriorate in similar ways to non-reasoning models. This result confirms that additional test-time compute does not, on its own, allow models to strategize over multi-turn conversation . The analysis we conduct identifies a potential root cause: reasoning models tend to generate lengthier responses (on avg. 33% longer than non-reasoning LLMs). As we find in Appendix F, longer assistant responses tend to contain more assumptions, which can derail the conversation by confusing the model on what requirements were posed by the user vs. its own previous turn responses.

额外的测试时计算（推理 token）并不能帮助模型应对多轮欠指定，因为实验中纳入的两个推理模型（o3、Deepseek-R1）以与非推理模型相似的方式下降。这一结果证实， 额外的测试时计算本身，并不能让模型在多轮对话上形成策略 。我们做的分析指出一个可能的根因：推理模型往往生成更长的回复（平均比非推理大模型长 33%）。正如我们在附录 F 中发现的，更长的助手回复往往包含更多假设，这些假设可能让模型分不清哪些要求来自用户、哪些来自它自己先前轮次的回复，从而把对话带偏。

6.2 Aptitude vs. Reliability Analysis (a) Visualizing Aptitude and Unreliability.

6.2 能力上限与可靠性分析 (a) 能力上限与不可靠性的可视化。

\overline{P}

\overline{P}

\overline{P}

\overline{P}

Appendix F explores potential root causes for models getting lost in conversations. We identify four specific causes: (1) LLMs prematurely propose full answer attempts, making assumptions about problem specifications that lead to confusion (Appendix F.1), (2) they overly rely on previous (incorrect) answer attempts leading to lengthier “bloated” answers (Section F.2), (3) LLMs overly adjust their answers based on the first and last turn of conversation, evidenced by a loss-of-middle-turns phenomenon (Appendix F.3), and (4) they produce overly verbose answers, which likely introduces assumptions that detract attention from user utterances (Section F.4).

附录 F 探讨了模型在对话中迷路的可能根因。我们识别出四条具体原因：（1）大模型过早提出完整的答题尝试，并对问题规格做出假设，从而带来混乱（附录 F.1）；（2）它们过度依赖此前（并不正确的）答题尝试，导致答案越来越长、出现“膨胀”（第 F.2 节）；（3）大模型过分根据对话的第一轮和最后一轮调整答案，表现为中间轮次丢失现象（附录 F.3）；（4）它们产生过于冗长的答案，这很可能引入假设，并把注意力从用户话语上引开（第 F.4 节）。

6.3 Gradual Sharding Experiment The multi-turn conversations simulated based on sharded conversations are not representative of underspecified conversations that users might have with LLMs in realistic settings. In particular, the fact that sharded instructions must be maximal (property P3) and that the simulated user must reveal at most one shard of information per turn (Section 3.2) can seem unrealistic and adversarial. In fact, prior work has found that minor and severe underspecification appear in equal proportions in public LLM chat logs [27]. To explore the relationship between the granularity of sharding and the lost in conversation phenomenon, we propose the gradual sharding experiment.

6.3 渐进分片实验 基于分片指令模拟出的多轮对话，并不能代表用户在现实设定中可能与大模型进行的欠指定对话。尤其是，分片指令必须是极大的（性质 P3），而且模拟用户每轮最多揭示一片信息（第 3.2 节），这可能显得不现实，甚至带有对抗性。事实上，已有工作发现，在公开的大模型聊天日志里，轻微与严重的欠指定出现比例相当 [27]。为了考察分片粒度与对话迷路现象之间的关系，我们提出渐进分片实验。

In the gradual sharding experiment, we selected 31 instructions from our original experiment across multiple tasks, and expanded each sharded instruction into seven sharded instructions, with the shard-set size growing from 2 to 8 shards. The instruction selection and sharding process are detailed in Appendix K. The process ensured that at each shard set size (from 1 to 8), task complexity is fixed, and the only modified factor is the granularity of sharding.

在渐进分片实验中，我们从原实验里跨多项任务选出 31 条指令，并把每条分片指令扩展成七条分片指令，碎片集合大小从 2 增加到 8。指令选择与分片过程详见附录 K。该流程保证在每个碎片集合大小（从 1 到 8）上，任务复杂度保持不变，唯一被改动的因素是分片的粒度。

We ran simulations for the gradual sharding experiments with two models (GPT-4o and GPT-4o-mini), with results summarized in Figure 6(c). We find that both models get lost in conversation (a minor degradation in aptitude and a large increase in unreliability) with two-shard instructions and beyond. In other words, the gradual sharding experiment indicates that any conversation that involves underspecification and occurs in two or more turns leads to models getting lost in conversation . For users, the granularity at which information is specified does not majorly impact reliability: providing all the information at once (1-shard) is the only effective method to improve reliability.

我们用两个模型（GPT-4o 和 GPT-4o-mini）运行了渐进分片实验的模拟，结果汇总于图 6(c)。我们发现，从两片指令开始往后，两个模型都会在对话中迷路（能力上限小幅下降，不可靠性大幅上升）。换句话说，渐进分片实验表明， 任何涉及欠指定、并且发生在两轮或更多轮中的对话，都会导致模型迷路 。对用户而言，信息被指定的粒度并不会显著影响可靠性：一次性提供全部信息（1 片）是提高可靠性的唯一有效办法。

7 Implications

7 影响

7.1 Implications for System and Agent Builders Simulation Type Model 4o-mini 86.8 84.4 50.4 66.5 61.8 4o 93.0 90.9 59.1 76.6 65.3 Table 2: Experimental Results with additional simulation types: Recap and Snowball. Both strategies involve repeating user-turn information to mitigate models getting lost in conversations.

	Simulation Type
Model
4o-mini	86.8	84.4	50.4	66.5	61.8
4o	93.0	90.9	59.1	76.6	65.3

7.1 对系统与智能体构建者的影响 模拟类型 模型 4o-mini 86.8 84.4 50.4 66.5 61.8 4o 93.0 90.9 59.1 76.6 65.3 表 2：增加 Recap 和 Snowball 这两种模拟类型后的实验结果。两种策略都通过重复用户轮次中的信息，来缓解模型在对话中迷路。

	模拟类型
模型
4o-mini	86.8	84.4	50.4	66.5	61.8
4o	93.0	90.9	59.1	76.6	65.3

Building LLM-based applications typically involves complex processes: decomposition of problems, retrieval of relevant information, use of tools, and calling of actions. Such processes are typically orchestrated by an agentic framework (such as Autogen [84] or LangChain [8]) that allows system builders to compose workflows with LLM calls as individual blocks. As such, an argument could be made that multi-turn capabilities are not a necessary feature of LLMs, as it can be offloaded to the agent framework. In other words, do we need native multi-turn support in LLMs when an agent framework can orchestrate interactions with users and leverage LLMs only as single-turn operators?

构建基于大模型的应用通常涉及复杂流程：问题分解、检索相关信息、使用工具，以及调用动作。这些流程通常由智能体框架（例如 Autogen [84] 或 LangChain [8]）来编排，系统构建者可以把带有大模型调用的工作流像搭积木一样组合起来。因此可以提出这样一种论点：多轮能力并不是大模型的必要特性，因为它可以交给智能体框架去处理。换句话说，当智能体框架可以编排与用户的交互、并只把大模型当作单轮算子来用时，我们还需要大模型原生支持多轮吗？

To answer this question, we implemented two agent-style conversation simulation types: Recap and Snowball. Both preprocess user utterances before sending them to the LLM. In Recap, a conversation proceeds in the same way as Sharded, but a user turn is added at the end, which recapitulates all the previous user turns. Snowball is a more gradual recapitulation: at each turn, the user simulator reveals a new shard, and repeats all previously revealed shards at that point. Both simulation types repeat the past user’s turn information to make it more prominent and give the LLM a chance to leverage the redundancy to improve its responses. We include the experimental detail in Appendix M.

为了回答这个问题，我们实现了两种智能体式的对话模拟类型：Recap 和 Snowball。二者都会在把用户话语送给大模型之前先做预处理。在 Recap 中，对话进程与 Sharded 相同，但最后会增加一个用户轮次，把此前所有用户轮次总述一遍。Snowball 是更渐进的总述：每一轮，用户模拟器揭示一片新碎片，并同时重复到该点为止已揭示的全部碎片。两种模拟类型都重复过去的用户轮次信息，使其更突出，并给大模型一个利用冗余来改进回复的机会。实验细节见附录 M。

Table 2 summarizes the results on all instructions for four tasks (Code, Database, Math, Actions) for two tested models (GPT-4o, GPT-4o-mini). Both Recap and Snowball demonstrate some level of success, with improvements over Sharded simulations, but the performance still lags behind Full or Concat. While Recap outperforms Snowball, we note that Recap is an unrealistic setting because the intervention is conducted on the last turn of the conversation, which is not known a priori when conversation unfolds with a real user. Snowball gives a sense of realistic performance gains achievable through user-turn repetition: it can mitigate the Full-to-Sharded performance deterioration by 15-20%. In short, relying on an agent-like framework to process information might be limiting, and we argue LLMs should natively support multi-turn interaction.

表 2 汇总了两个受试模型（GPT-4o、GPT-4o-mini）在全部指令、四项任务（代码、数据库、数学、动作）上的结果。Recap 和 Snowball 都表现出一定成效，相对 Sharded 模拟有所提升，但成绩仍落后于 Full 或 Concat。虽然 Recap 优于 Snowball，我们指出 Recap 是一种不现实的设定，因为干预发生在对话的 最后一轮 ，而与真实用户展开对话时事先并不知道何时是最后一轮。Snowball 给出了通过重复用户轮次所能实现的、较为现实的收益：它大约能缓解 Full 到 Sharded 成绩恶化的 15-20%。简而言之，依赖类似智能体的框架来处理信息可能存在局限，我们认为大模型应当原生支持多轮交互。

7.2 Implications for LLM Builders A lot of effort has been put in improving LLM aptitude : demonstrating that LLMs can accomplish tasks of increasing intellectual complexity, with recent results showing LLMs can compete in mathematics Olympiads, or solve Ph.D.-level technical questions in a benchmark aptly named Humanity’s Last Exam [62].

7.2 对大模型构建者的影响 人们已经在提升大模型的 能力上限 上投入了大量努力：展示大模型能够完成智力复杂度不断提高的任务，最近的结果还表明大模型可以参加数学奥林匹克，或在一个恰如其分地命名为 Humanity’s Last Exam 的基准上解答博士水平的技术问题 [62]。

In this work, we call on LLM builders to prioritize reliability of the models they build, as our experiments demonstrate that the randomness involved in generating text with LLMs leads to catastrophic unreliability in all the models we tested, degrading the quality of responses the average LLM users see.

在本文中，我们呼吁大模型构建者优先考虑其所建模型的 可靠性 ，因为我们的实验表明，用大模型生成文本时所涉及的随机性，会在我们测试的所有模型中导致灾难性的不可靠，从而降低普通大模型用户看到的回复质量。

T=0

T=0

To evaluate this argument, we conducted a supplementary experiment in which the assistant’s temperature for generating responses (AT) was varied to three values: 1.0, 0.5, and 0.0. Additionally, since Sharded simulation uses an LLM-based user simulator, we also varied the user’s temperature (UT) with the same three values. Further details on the experiment, including sample selection and simulation scale, are in Appendix L.

为了评估这一论点，我们做了一项补充实验，把助手生成回复的温度（AT）设为三个取值：1.0、0.5 和 0.0。此外，由于 Sharded 模拟使用基于大模型的用户模拟器，我们也用同样的三个取值改变用户温度（UT）。关于实验的更多细节，包括样本选择和模拟规模，见附录 L。

4o-mini 4o Simulation AT=1.0 AT=0.5 AT=0.0 width 7pt AT=1.0 AT=0.5 AT=0.0 Full 16.0 15.0 6.8 width 7pt 17.8 8.0 2.8 Concat 20.2 17.8 9.5 width 7pt 20.2 17.8 5.8 UT=1.0 49.8 46.8 51.0 width 7pt 41.0 43.8 31.8 UT=0.5 31.7 34.0 40.5 width 7pt 39.5 40.8 31.8 UT=0.0 38.5 28.0 30.5 width 7pt 35.8 38.0 29.7 Table 3: Unreliability of models when changing assistant temperature (AT) and user temperature (UT) in Full, Concat and Sharded settings. The lower the number the more reliable the assistant is.

	4o-mini	4o
Simulation	AT=1.0	AT=0.5	AT=0.0 width 7pt	AT=1.0	AT=0.5	AT=0.0
Full	16.0	15.0	6.8 width 7pt	17.8	8.0	2.8
Concat	20.2	17.8	9.5 width 7pt	20.2	17.8	5.8
UT=1.0	49.8	46.8	51.0 width 7pt	41.0	43.8	31.8
UT=0.5	31.7	34.0	40.5 width 7pt	39.5	40.8	31.8
UT=0.0	38.5	28.0	30.5 width 7pt	35.8	38.0	29.7

4o-mini 4o 模拟 AT=1.0 AT=0.5 AT=0.0 AT=1.0 AT=0.5 AT=0.0 Full 16.0 15.0 6.8 17.8 8.0 2.8 Concat 20.2 17.8 9.5 20.2 17.8 5.8 UT=1.0 49.8 46.8 51.0 41.0 43.8 31.8 UT=0.5 31.7 34.0 40.5 39.5 40.8 31.8 UT=0.0 38.5 28.0 30.5 35.8 38.0 29.7 表 3：在 Full、Concat 和 Sharded 设定中改变助手温度（AT）和用户温度（UT）时模型的不可靠性。数字越低，助手越可靠。

	4o-mini	4o
模拟	AT=1.0	AT=0.5	AT=0.0	AT=1.0	AT=0.5	AT=0.0
Full	16.0	15.0	6.8	17.8	8.0	2.8
Concat	20.2	17.8	9.5	20.2	17.8	5.8
UT=1.0	49.8	46.8	51.0	41.0	43.8	31.8
UT=0.5	31.7	34.0	40.5	39.5	40.8	31.8
UT=0.0	38.5	28.0	30.5	35.8	38.0	29.7

U_{10}^{90}

U_{10}^{90}

U_{10}^{90}<15

U_{10}^{90}<15

7.3 Implications for NLP Practitioners Our experiments demonstrate that model behavior in single- and multi-turn settings on the same underlying set of instructions can diverge in important ways, for example, with large observed degradations in performance and reliability.

7.3 对自然语言处理实践者的影响 我们的实验表明，在同一组底层指令上，模型在单轮与多轮设定中的行为可能以重要方式分叉，例如在表现和可靠性上观察到大幅下降。

We selected the initial six tasks to span a wide range of generation tasks, from programming to multi-document summarization. Yet this set of tasks is limited across multiple dimensions, such as focusing on English-language instructions and analytical (i.e., non-creative) tasks. We put effort into making the sharding process scalable by automating portions that could be handled by an LLM, while manually validating and finalizing samples for quality control. The sharding process - detailed in Appendix C - required an average of three hours of manual work (prompt engineering or inspection) from an author to prepare and finalize 100 sharded instructions.

我们最初选择的六项任务，意在覆盖从编程到多文档摘要的广泛生成任务。但这组任务在多个维度上仍有局限，例如只关注英语指令和分析型（也就是非创意型）任务。我们努力让分片流程可扩展：能由大模型处理的部分实现自动化，同时通过人工核验和定稿来控制质量。分片流程——详见附录 C——平均需要一位作者投入三小时的人工（提示工程或检查），来准备并定稿 100 条分片指令。

We encourage NLP practitioners to experiment with sharding and release sharded versions of their tasks and instructions alongside fully specified ones.

我们鼓励自然语言处理实践者尝试分片，并在发布完整指定的任务与指令时，同时发布它们的分片版本。

Translation Model 4o-mini 41.7 43.4 42.1 4o 35.9 38.5 40.9 Table 4: Performance on the translation task for Full, Concat, and Sharded simulations.

	Translation
Model
4o-mini	41.7	43.4	42.1
4o	35.9	38.5	40.9

翻译 模型 4o-mini 41.7 43.4 42.1 4o 35.9 38.5 40.9 表 4：翻译任务在 Full、Concat 和 Sharded 模拟中的表现。

	翻译
模型
4o-mini	41.7	43.4	42.1
4o	35.9	38.5	40.9

To illustrate the feasibility of sharding new tasks, and understand compatibility requirements for sharding, we prepared sharded instructions for a seventh task: Translation. The task consists of translating an entire document (10 sentences) from German to English, leveraging paired documents from WMT 2019 on document-level translation [70]. In the Sharded setting, each turn reveals two additional sentences from the source document and requires the assistant to translate all sentences provided so far, whereas the Full and Concat settings reveal the entire document in the first turn. Evaluation is conducted with the standard BLEU metric [58]. We describe practical implementation details in Appendix I.

为了说明给新任务做分片是可行的，并弄清分片需要满足哪些兼容条件，我们为第七项任务准备了分片指令：翻译。该任务是把整篇文档（10 个句子）从德语译成英语，语料来自 WMT 2019 文档级翻译的平行文档 [70]。在 Sharded 设定中，每一轮从源文档额外揭示两个句子，并要求助手翻译到目前为止提供的全部句子；而 Full 和 Concat 设定则在第一轮揭示整篇文档。评估使用标准的 BLEU 指标 [58]。实践中的实现细节见附录 I。

Results from Full, Concat, and Sharded simulations are summarized in Table 4. Both models we tested - GPT-4o-mini and GPT-4o - do not exhibit degradation in performance in the Sharded setting, with BLEU scores being within 10% difference of each other in all settings. We believe this result reflects that the task can largely be accomplished at the sentence-level despite some prior work has framed translation at the document-level [64], and that the BLEU score does not adequately capture document-level nuances [52]. In other words, if a task is episodic (i.e., it can be decomposed into turn-level subtasks), the models can avoid getting lost in conversation by completing each subtask without having to handle multi-turn context. In short, the Sharded Translation task simulates multi-turn conversations that are not underspecified.

Full、Concat 和 Sharded 模拟的结果汇总于表 4。我们测试的两个模型——GPT-4o-mini 和 GPT-4o——在 Sharded 设定中 并没有 表现出成绩下降，各设定中的 BLEU 分数彼此相差不超过 10%。我们认为这一结果反映的是：尽管已有工作把翻译框定为文档级任务 [64]，该任务很大程度上仍可在句子级完成；而且 BLEU 分数并不能充分捕捉文档级的细微差别 [52]。换句话说，如果一项任务是情节式的（也就是可以分解成轮次级的子任务），模型就可以通过完成每个子任务来避免在对话中迷路，而不必处理多轮上下文。简而言之，分片翻译任务模拟的是并不欠指定的多轮对话。

1+1

1+1

7.4 Implications for Users of Conversational Systems Users of LLM-based products should be aware of the lack of reliability of LLMs, particularly when used in multi-turn settings. Generally available generative technology is new, and prior work has identified the randomness in LLM-generated text as a point of confusion for users [55, 81, 77, 43]. We make two practical recommendations that can help users of LLM-based systems get the most out of their exchanges.

7.4 对对话系统用户的影响 基于大模型的产品的用户应当意识到大模型缺乏可靠性，在多轮设定中尤其如此。面向大众的生成技术还很新，已有工作指出，大模型生成文本中的随机性会让用户感到困惑 [55, 81, 77, 43]。我们给出两条实用建议，可以帮助大模型系统的用户从交谈中获得更好的结果。

If time allows, try again. If a conversation with an LLM did not lead to expected outcomes, starting a new conversation that repeats the same information might yield significantly better outcomes than continuing an ongoing conversation. This is because current LLMs can get lost in the conversation, and our experiments show that persisting in a conversation with the model is ineffective. In addition, since LLMs generate text with randomness, a new conversation may lead to improved outcomes.

如果时间允许，再试一次。 如果与大模型的一段对话没有带来预期结果，新开一段对话、重复相同信息，可能比继续当前对话得到明显更好的结果。这是因为当前的大模型可能已经在对话中迷路，而我们的实验表明，坚持在同一段对话里继续纠偏并不有效。此外，由于大模型生成文本带有随机性，新的对话也可能带来更好的结果。

Consolidate before retrying. Since LLMs are ineffective at dealing with information dispersed across multiple turns, consolidating instruction requirements into a single instruction is an effective strategy to improve the model’s aptitude and reliability (as shown by the Concat experiments). When a user notices that a model is lost in conversation, they can ask the LLM: ‘‘Please consolidate everything I’ve told you so far,’’ then bring the response to a new conversation, alleviating the need for manual consolidation. In practice, there is anecdotal evidence that early adopters of LLM-based applications are aware that LLMs get lost in conversation. For example, users of the Cursor LLM-based coding environment report that frequently creating new conversations ‘‘whenever they can’’ is a recommended strategy to ensure high quality responses even though the tool allows to keep conversations going indefinitely.[^2]

重试前先把要求收拢到一处。 由于大模型不擅长处理分散在多轮中的信息，把指令要求收拢成一条指令，是提高模型能力上限和可靠性的有效策略（Concat 实验已表明这一点）。当用户注意到模型已经在对话中迷路时，可以先问大模型：「请把到目前为止我告诉你的一切汇总起来」，再把回复带到一段新对话中，从而减轻手工汇总的负担。实践中，已有轶事证据表明，大模型应用的早期采纳者已经意识到模型会在对话中迷路。例如，基于大模型的编程环境 Cursor 的用户报告说，尽管该工具允许对话无限继续，但「一有机会就」频繁开新对话，仍是保证回复质量的推荐策略。[^2]

These two recommendations remain cumbersome for users and can only offer patched solutions rather than a principled approach. Once future LLMs can more reliably handle multi-turn conversations, the need for such recommendations should be alleviated, allowing users to communicate underspecified instructions over multiple turns naturally with less risk of the model getting lost in conversation.

这两条建议对用户来说仍然麻烦，也只能提供打补丁式的办法，而不是原则性的方案。一旦未来的大模型能更可靠地处理多轮对话，这类建议的必要性就应当降低，从而让用户能够自然地用多轮传递欠指定的指令，而不那么担心模型会在对话中迷路。

8 Conclusion

8 结论

In this work, we conduct a large-scale simulation of single- and multi-turn conversations with LLMs, and find that on a fixed set of tasks, LLM performance degrades significantly in multi-turn, underspecified settings. LLMs get lost in conversation, which materializes as a significant decrease in reliability as models struggle to maintain context across turns, make premature assumptions, and over-rely on their previous responses. Additional experiments reveal that known remediations that work for simpler settings (such as agent-like concatenation or decreasing temperature during generation) are ineffective in multi-turn settings, and we call on LLM builders to prioritize the reliability of models in multi-turn settings.

本文对大模型的单轮与多轮对话做了大规模模拟，并发现：在固定的一组任务上，大模型在多轮、欠指定设定中的表现会显著下降。大模型会在对话中迷路，具体表现为可靠性显著下降，因为模型难以在各轮之间保持上下文，会过早做出假设，并过度依赖自己先前的回复。更多实验表明，在更简单设定中行之有效的已知补救办法（例如类似智能体的拼接，或在生成时降低温度）在多轮设定中并不有效。我们呼吁大模型构建者优先考虑模型在多轮设定中的可靠性。

9 Limitations

9 局限

我们工作的第一项局限，是依赖完全自动化的模拟。借助大模型来模拟用户话语，我们才能把实验做大规模，包括多次运行同一种模拟——若换成真实用户，成本将高得难以承受。可我们得到的模拟并不能代表自然的人机对话。分片过程的性质（定义见附录 C）以及模拟环境的性质（见第 3.2 节）保证模拟对话遵循相当狭窄的结构，很可能没有刻画出在庞大、多样的用户群体中会出现的全部对话动态。例如，模拟过程保证每一轮都会揭示一片新信息，并且对话的最后一轮已经指定完成任务所需的全部信息，而真实用户未必如此。分片过程的性质 P1、P2 和 P5 也限制了对话范围，因为分片指令紧密对应一条现成的完整指定指令，高层意图总是在对话第一轮就被点明。碎片的最小性同样不现实，甚至可能带有对抗性；不过渐进分片实验发现，不同的分片粒度会带来相似的成绩下降，只要对话发生在两轮或更多轮。除了分片粒度，自动模拟也缺少真人参与对话时可能出现的细微之处，包括术语上的误解、因系统失败而沮丧放弃 [82]，或某些对话根本没有可行的终点（例如用户想要一个尚未解决的问题的答案）。由于这些因素，我们认为所做模拟代表的是一块相对温和的测试场，用来考察大模型的多轮能力。 正因为模拟条件过于简化，我们认为实验中观察到的下降，最可能低估了大模型的不可靠性，以及大模型在现实设定中迷路的频繁程度 。 这些实验为在多轮设定中研究大模型提供了一个可扩展、低成本的实验环境。

A second limitation of our work is the focus on analytical tasks. Although we selected a diverse set of both programming and natural language tasks, we restricted experiments to tasks that involve an analytical solution. This restriction limits the scope of our findings, as we do not establish whether models get lost in conversation on more open-ended tasks, such as creative writing [5]. This was a conscious choice: though there has been some progress on creative writing evaluation, it is still an active area of research [6], and we relied on more established tasks and metrics for the initial set of experiments. Determining whether degradation occurs - and if so, identifying the magnitude - on creative tasks is an important direction for future work.

我们工作的第二项局限，是聚焦于分析型任务。尽管我们选择了编程和自然语言两类多样化任务，实验仍限于涉及分析性解答的任务。这一限制缩小了发现的适用范围，因为我们没有确定模型是否也会在更开放的任务上迷路，例如创意写作 [5]。这是有意识的选择：尽管创意写评测已有一些进展，它仍是一个活跃的研究方向 [6]，而我们在最初这组实验中依赖更成熟的任务和指标。确定创意任务上是否发生下降——如果发生，幅度有多大——是未来工作的一个重要方向。

A third limitation of the work is the focus on text-only tasks in the English language. Establishing whether models get lost in conversation in other languages, or in tasks that involve multiple modalities in either user or assistant utterances, could help establish the scope of the degradation observed in LLM multi-turn capabilities.

工作的第三项局限，是聚焦于英语的纯文本任务。确定模型是否会在其他语言中迷路，或在用户或助手话语涉及多种模态的任务中迷路，将有助于界定我们在大模型多轮能力上观察到的下降范围。

References

参考文献

[21] S. E. Finch, J. D. Finch, and J. D. Choi. Don’t forget your abc’s: Evaluating the state-of-the-art in chat-oriented dialogue systems. In The 61st Annual Meeting Of The Association For Computational Linguistics , 2023. [22] S. Frisson. Semantic underspecification in language processing. Lang. Linguistics Compass , 3:111-127, 2009. URL https://api.semanticscholar.org/CorpusID:13384476 . [23] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024. [24] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025. [25] C. Han. Can language models follow multiple turns of entangled instructions? arXiv preprint arXiv:2503.13222 , 2025. [26] K. Handa, A. Tamkin, M. McCain, S. Huang, E. Durmus, S. Heck, J. Mueller, J. Hong, S. Ritchie, T. Belonax, et al. Which economic tasks are performed with ai? evidence from millions of claude conversations. arXiv preprint arXiv:2503.04761 , 2025. [27] C. Herlihy, J. Neville, T. Schnabel, and A. Swaminathan. On overcoming miscalibrated conversational priors in llm-based chatbots. arXiv preprint arXiv:2406.01633 , 2024. [28] M. C. Horowitz, L. Kahn, J. Macdonald, and J. Schneider. Adopting ai: how familiarity breeds both trust and contempt. AI & society , 39(4):1721-1735, 2024. [29] K.-H. Huang, P. Laban, A. R. Fabbri, P. K. Choubey, S. Joty, C. Xiong, and C.-S. Wu. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news articles. arXiv preprint arXiv:2309.09369 , 2023. [30] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024.

[61] S. Pezzelle. Dealing with semantic underspecification in multimodal nlp. arXiv preprint arXiv:2306.05240 , 2023. [62] L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam. arXiv preprint arXiv:2501.14249 , 2025. [63] C. Poelitz and N. McKenna. Synthetic clarification and correction dialogues about data-centric tasks-a teacher-student approach. arXiv preprint arXiv:2503.14167 , 2025. [64] M. Post and M. Junczys-Dowmunt. Escaping the sentence-level paradigm in machine translation. arXiv preprint arXiv:2304.12959 , 2023. [65] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. [66] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020. [67] A. Ram, R. Prasad, C. Khatri, A. Venkatesh, R. Gabriel, Q. Liu, J. Nunn, B. Hedayatnia, M. Cheng, A. Nagar, et al. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:1801.03604 , 2018. [68] S. Reddy, D. Chen, and C. D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics , 7:249-266, 2019. [69] R. Sarkar, B. Sarrafzadeh, N. Chandrasekaran, N. Rangan, P. Resnik, L. Yang, and S. K. Jauhar. Conversational user-ai intervention: A study on prompt rewriting for improved llm response generation. ArXiv , abs/2503.16789, 2025. URL https://api.semanticscholar.org/CorpusID:277244656 . [70] Y. Scherrer, J. Tiedemann, and S. Loáiciga. Analysing concatenation approaches to document-level nmt in two different domains. In Proceedings of the Third Workshop on Discourse in Machine Translation , Hong-Kong, Nov. 2019. Association for Computational Linguistics.

[91] G. K. Zipf. Human behavior and the principle of least effort: An introduction to human eoclogy . Addison-Wesley Press, 1949.

Appendices

附录

Appendix A Related work on Underspecification

附录 A 关于欠指定的相关工作

The Background (Section 2) reviews the most directly related prior work, focused on multi-turn evaluation. We now cover other related prior works that have studied underspecification.

背景一节（第 2 节）回顾了最直接相关的已有工作，焦点是多轮评测。我们现在介绍其他研究过欠指定的相关工作。

Prior work on communication and linguistics has identified underspecification as a common feature of human language [41, 20, 22, 61].

沟通与语言学方面的已有工作已经指出，欠指定是人类语言的常见特征 [41, 20, 22, 61]。

Understanding how LLMs handle underspecified instructions is crucial towards improving conversational capabilities. To this end, Herlihy et al. [27] identified common response patterns such as hedging, refusal, clarification, and interrogation when underspecified queries are presented to conversational LLM systems, and proposed mechanisms to recover from them. Malaviya et al. [53] highlighted the importance of supporting context for more accurate and principled evaluation of LLM responses on underspecified queries, and Sarkar et al. [69] showed that a system that proactively rewrites user instructions to account for underspecification leads to improved LLM response. Shaikh et al. [71] studied the degree of grounding (i.e., clarifications and follow-up questions) that LLMs perform in conversation logs and observed that they significantly lack in generating follow-up questions, where humans are 15 times more likely to do so. Chang et al. [7] hired annotators to manually reproduce fully-specified instructions through a chat interface, and found that the users reveal the entirety of the instruction in 34% of the time, leaving some detail underspecified a majority of the time.

理解大模型如何处理欠指定指令，对于改进对话能力至关重要。为此，Herlihy 等人 [27] 识别出当欠指定查询被交给对话式大模型系统时常见的回复模式，如含糊其辞、拒绝、澄清和追问，并提出从这些模式中恢复的机制。Malaviya 等人 [53] 强调，要更准确、更有原则地评估大模型对欠指定查询的回复，支持性上下文很重要；Sarkar 等人 [69] 则表明，一个会主动改写用户指令以照顾欠指定的系统，能带来更好的大模型回复。Shaikh 等人 [71] 研究了大模型在对话日志中进行 grounding 的程度（ 即 澄清和后续提问），并观察到它们明显缺少后续提问，而人类提出后续问题的可能性要高 15 倍。Chang 等人 [7] 聘请标注员通过聊天界面手工复现完整指定的指令，并发现用户只有 34% 的时间会揭示全部指令，大多数时候都会留下一些细节欠指定。

Several works have explored direct tasks to evaluate model ability when dealing with underspecification. Liu et al. [49] introduced AmbiEnt, a natural language inference benchmark, which revealed that understanding ambiguous statements is still a challenge even to the state-of-the-art LLMs. Wildenburg et al. [83] created the DUST task, which requires the language model to determine the relative levels of specifications between two sentences, finding that when interpreting underspecified sentences, LMs exhibit little uncertainty. Vijayvargiya et al. [78] evaluated LLM agents for GitHub issue resolution in an underspecified setting, showing that follow-up interactions to supplement information helps improve the resolve rate but detecting the ambiguities in the instructions remains a challenge.

已有若干工作探索了直接评测模型处理欠指定能力的任务。Liu 等人 [49] 提出自然语言推理基准 AmbiEnt，表明即便对最强的大模型，理解含糊陈述仍是挑战。Wildenburg 等人 [83] 创建了 DUST 任务，要求语言模型判断两个句子之间的相对指定程度，并发现在解释欠指定句子时，语言模型表现出的不确定性很小。Vijayvargiya 等人 [78] 在欠指定设定中评估用于解决 GitHub issue 的大模型智能体，表明通过后续交互补充信息有助于提高解决率，但检测指令中的歧义仍然困难。

Prior work has classified different root causes for underspecification. First, task underspecification occurs when humans provide incomplete descriptions of the task at hand, which is prominent in “specification-heavy tasks” [60]. Second, intent misalignment occur when the AI fails to understand the user’s intent or motivation, and is one of the common sources of user dissatisfaction [34, 76]. Finally, Chaturvedi et al. [9] discuss location and and reference ambiguity, in emboddied settings that involve physical spaces such as a Minecraft game.

已有工作把欠指定的不同根因做了分类。第一，任务欠指定发生在人类对当前任务的描述不完整时，这在“规格很重的任务”中尤为突出 [60]。第二，意图错位发生在人工智能未能理解用户的意图或动机时，也是用户不满意的常见来源之一 [34, 76]。最后，Chaturvedi 等人 [9] 讨论了位置歧义和指称歧义，它们出现在涉及物理空间的具身设定中，例如 Minecraft 游戏。

Appendix B Precise Definition of Sharded Instructions

附录 B 分片指令的精确定义

Section 3.1 introduces the concept of sharding at a high level. This Appendix offers a more precise definition by first defining mathematical terminology, and then defining properties that a sharded instruction must satisfy to be considered valid.

第 3.1 节从较高层面介绍了分片概念。本附录给出更精确的定义：先定义数学用语，再定义一条分片指令要被视为有效所必须满足的性质。

q

q

I(q)=[\mathcal{I},({c}_{1},\cdots,c_{m})]

\mathcal{I}

\mathcal{I}

q

q

q^{\prime}=[s_{1},\cdots s_{k}]\;\mbox{ s.t. }I(q)=I(q^{\prime})

s_{j}

s_{j}

q^{\prime}

q^{\prime}

I(q)=I(q^{\prime})

I(q)=I(q^{\prime})

\mathcal{I}_{q}=\mathcal{I}_{q^{\prime}}

\mathcal{I}_{q}=\mathcal{I}_{q^{\prime}}

\rho(\mathbf{s}_{2..k})

\rho(\mathbf{s}_{2..k})

k

k

[s_{1},\cdots s_{k}]

[s_{1},\cdots s_{k}]

Appendix C Semi-Automatic Sharding Process

附录 C 半自动分片流程

Figure 7: Process diagram of the four-step semi-automatic process to transform fully-specified instructions into a sharded instruction. The first three steps (segmentation, rephrasing, verification) are automated, while the fourth (inspect and edit) was manually completed by the authors of the work. The last row represents the rejection criteria for a sample.

图 7：把完整指定指令转变成分片指令的四步半自动流程示意图。前三步（切分、改写、核验）是自动的，第四步（检查与编辑）由本文作者手工完成。最后一行表示一条样本的拒绝标准。

We rely on a semi-automatic process to transform fully-specified instructions into their sharded equivalents. The process - illustrated in Figure 7 - consists of a sequence of three automated steps (Segmentation, Rephrasing, Verification) followed by a manual step that was conducted by an author of the paper.

我们依靠一套半自动流程，把完整指定的指令转变成对应的分片形式。该流程——如图 7 所示——由三个自动步骤（切分、改写、核验）依次构成，随后是由论文一位作者完成的人工步骤。

We now detail each step of the process, then go over task-specific details we implemented as needed. We note that as part of our open-source release, we provide all the prompts used in the first three LLM-based steps.

我们现在详细说明流程的每一步，再介绍我们按需实现的任务专用细节。需要指出的是，作为开源发布的一部分，我们提供了前三个基于大模型的步骤中使用的全部提示。

Step 1: Segmentation Given an original fully-specified instruction (left-most column in Figure 7), the LLM is prompted to extract segments of the instructions. Segments are intended to correspond to the atomic content units (defined in Appendix B). In particular, the prompt indicates that segments must not overlap, and that not all words in the original instruction must belong to a segment. Prompts are task-specific and incorporate at least three few-shot examples of segmentation, to allow for the concept of segmentation to be illustrated through examples. At this stage, any instruction that yields fewer than three segments are filtered out and does not proceed to the next stage.

第 1 步：切分 给定一条原始的完整指定指令（图 7 最左列），大模型被提示去抽取指令中的 片段 。这些片段意在对应原子内容单元（定义见附录 B）。提示特别指出，片段不得重叠，而且原指令中并非所有词都必须属于某个片段。提示是任务专用的，并至少包含三个切分的少样本示例，以便通过例子说明切分的概念。在这一阶段，任何产生少于三个片段的指令都会被过滤掉，不再进入下一阶段。

Step 2: Rephrasing Given the original fully-specified instruction and the extracted segments, this stage consists in rewriting each segment to be decontextualized [13] and conversational. In other words, dependencies between segments are resolved, and the ordering is changed such that obtained shards adhere to properties P2 and P5. In the example above, the fourth segment (highlighted in orange) becomes the first shard as it reveals the overall intent, and light rephrasing occur in other shards. The rephrasing prompt is task-specific, and includes few-shot examples of rephrasing segmented instructions.

第 2 步：改写 给定原始完整指定指令以及抽出的片段，这一阶段要把每个片段改写为去语境化 [13] 且口语化的形式。换句话说，片段之间的依赖被消除，顺序也被调整，使得到的 碎片 符合性质 P2 和 P5。在上面的例子中，第四个片段（用橙色标出）变成第一片，因为它揭示了总体意图，其他碎片则发生轻度改写。改写提示是任务专用的，并包含对已切分指令进行改写的少样本示例。

\overline{P}

\overline{P}

\displaystyle\overline{P}_{\textsc{Concat}} \\ \displaystyle\geq 0.8~\overline{P}_{\textsc{Full}} \\ \displaystyle\overline{P}_{\textsc{Shuffle-concat}} \\ \displaystyle\geq 0.8~\overline{P}_{\textsc{Full}},

\overline{P}_{\textsc{X}}

\overline{P}_{\textsc{X}}

Step 4: Inspect and Edit Even though the first three steps define the sharding process and implement some level of quality assurance, they do not guarantee the level of quality required for precise and large-scale experiments due to relying on LLM outputs. To obtain high-quality shards, we reserve step 4 for manual inspection and validation. To facilitate the procedure, we developed a web-based annotation interface. In the interface, an annotator can review a pair of fully-specified and sharded instructions, edit, add, or remove individual shards, and decide to accept or reject sharded instructions. Sharded instructions included in our experiments were all manually reviewed by two authors of the work. The amount of editing and filtering required in this final stage varied by task.

第 4 步：检查与编辑 尽管前三步定义了分片流程并实现了一定程度的质量保证，但由于依赖大模型输出，它们并不能保证精确、大规模实验所需的质量水平。为了得到高质量的碎片，我们把第 4 步留给人工检查与核验。为便于操作，我们开发了一个基于网页的标注界面。在该界面中，标注员可以审阅一对完整指定指令与分片指令，编辑、添加或删除单个碎片，并决定接受或拒绝分片指令。纳入我们实验的分片指令全部由两位作者人工审阅。这一最后阶段所需的编辑和过滤量因任务而异。

Inspecting and editing an auto-generated instruction typically requires 1-3 minutes per instruction, an order of magnitude less than it would require for authors to write the sharded instructions de-novo from a given fully-specified instruction. As part of our open-source release, we provide all the prompts used during sharding, which we hope can facilitate the sharding of additional tasks.

检查并编辑一条自动生成的指令，通常每条需要 1-3 分钟，比作者从一条给定的完整指定指令从头撰写分片指令所需的时间低一个数量级。作为开源发布的一部分，我们提供分片过程中使用的全部提示，希望这能便于为更多任务做分片。

Appendix D Inspection of Simulated Sharded Conversation

附录 D 对模拟分片对话的检查

Inspection All Tasks Actions Code Math Db Shard Fully Revealed 96.0 98.3 94.9 93.4 100.0 Shard Contextualized 98.4 98.3 98.3 98.3 98.6 Strategy Accuracy 95.2 94.7 95.5 95.6 94.7 Extraction Success 97.0 100.0 93.4 98.4 100.0 Overall Success 97.8 100.0 96.0 96.0 100.0 Table 5: Results of the inspection of 100 simulated sharded conversations across four tasks: Actions, Code, Math, and Database. The first column aggregates annotation results on the four tasks.

Inspection	All Tasks	Actions	Code	Math	Db
Shard Fully Revealed	96.0	98.3	94.9	93.4	100.0
Shard Contextualized	98.4	98.3	98.3	98.3	98.6
Strategy Accuracy	95.2	94.7	95.5	95.6	94.7
Extraction Success	97.0	100.0	93.4	98.4	100.0
Overall Success	97.8	100.0	96.0	96.0	100.0

检查项 全部任务 动作 代码 数学 数据库 碎片被完整揭示 96.0 98.3 94.9 93.4 100.0 碎片被恰当语境化 98.4 98.3 98.3 98.3 98.6 策略准确率 95.2 94.7 95.5 95.6 94.7 抽取成功 97.0 100.0 93.4 98.4 100.0 总体成功 97.8 100.0 96.0 96.0 100.0 表 5：对四个任务（动作、代码、数学和数据库）上 100 段模拟分片对话的检查结果。第一列汇总四个任务上的标注结果。

检查项	全部任务	动作	代码	数学	数据库
碎片被完整揭示	96.0	98.3	94.9	93.4	100.0
碎片被恰当语境化	98.4	98.3	98.3	98.3	98.6
策略准确率	95.2	94.7	95.5	95.6	94.7
抽取成功	97.0	100.0	93.4	98.4	100.0
总体成功	97.8	100.0	96.0	96.0	100.0

The sharding simulation environment (described in Section 3) relies on LLM components to simulate the user, classify assistant responses, and extract answers from free-text responses. LLM-based components are likely to fail, and we performed an inspection of 200 simulated Sharded conversations to understand the level of simulation error and the potential effect on estimating the performance of the assistant LLMs due to the error.

分片模拟环境（见第 3 节）依靠大模型组件来模拟用户、对助手回复分类，并从自由文本回复中抽取答案。基于大模型的组件很可能失败，因此我们对 200 段模拟的 Sharded 对话做了检查，以了解模拟误差的水平，以及这种误差可能如何影响对助手大模型表现的估计。

For each inspected conversation, we annotated user turns, assistant turns, and the overall conversation with five specific elements.

对每段受检对话，我们用五个具体要素标注用户轮次、助手轮次以及整段对话。

For user utterances, we annotated whether the utterance revealed exactly the information from one shard in the sharded instruction (Shard Fully Revealed). Specifically, we flagged turns that revealed more than one shard, and turns that revealed a shard only partially. We also annotated each user’s turn for whether it is appropriately contextualized in the conversation (Shard Contextualized). For example, if the previous assistant’s turn asked a binary clarification question (yes/no), then proper contextualization would require a Yes/No response to directly address the assistant’s response.

对用户话语，我们标注该话语是否恰好揭示了分片指令中一片的信息（碎片被完整揭示）。具体地说，我们标出揭示了不止一片的轮次，以及只部分揭示了一片的轮次。我们还标注每一轮用户话语是否在对话中被恰当语境化（碎片被恰当语境化）。例如，如果上一轮助手问了一个是非澄清问题，那么恰当的语境化就要求用“是/否”直接回应该助手回复。

For assistant utterances, we annotated whether the classified strategy was accurate (Strategy Accuracy). For example, if the response is labeled as a clarification, we confirm if it poses a clarification question to the user. When assistant utterances were labeled as answer attempts, we further labeled whether the answer extraction step was successful (Extraction Success).

对助手话语，我们标注所分类的策略是否准确（策略准确率）。例如，如果回复被标为澄清，我们就确认它是否向用户提出了澄清问题。当助手话语被标为答题尝试时，我们进一步标注答案抽取步骤是否成功（抽取成功）。

Upon completing the inspection of each user and assistance utterance, we assigned a global label to the entire conversation on whether or not the errors that occurred during simulation (if any) affected the overall validity of the simulation. If not, the simulation was marked as successful (Overall Success).

在完成对每轮用户和助手话语的检查后，我们给整段对话打一个全局标签，说明模拟过程中发生的错误（如果有）是否影响了模拟的总体有效性。如果没有，该模拟就被标为成功（总体成功）。

We inspected conversations for four tasks: Actions, Code, Math and Database. The other two (Summary and Data-to-text) are refining tasks that require an answer attempt at each turn, and do not rely on an LLM-based user simulator. As such, they have limited scope for simulation error.

我们检查了四项任务的对话：动作、代码、数学和数据库。另外两项（摘要和数据到文本）是 refinement 任务，每一轮都需要一次答题尝试，并且不依赖基于大模型的用户模拟器。因此，它们可供模拟误差发生的空间有限。

Table 5 summarizes the results of the inspection annotation. Overall, the simulation environment is highly reliable, with roughly 98% of inspected conversations labeled as successful. Some errors occur in each component. With user simulation, a single shard is fully revealed around 96% of the time, and properly contextualized 98% of the time. The processing of assistant responses also leads to errors: the turn strategy classification is only 95% accurate, and extraction of answer attempts has an accuracy of 97%.

表 5 汇总了检查标注的结果。总体而言，模拟环境非常可靠，大约 98% 的受检对话被标为成功。每个组件都会出现一些错误。在用户模拟方面，大约 96% 的时间会完整揭示单片，98% 的时间会被恰当语境化。对助手回复的处理也会导致错误：轮次策略分类的准确率只有 95%，答题尝试的抽取准确率为 97%。

Utterance-level errors did not always affect the validity of the overall simulation. In some cases, we observed that the user simulator would correct an error in an early turn, subsequently in the conversation, or that an error in answer extraction on the wrong answer attempt would occur at a turn, but the extraction would be successful later on. In summary, we empirically find that the simulation environment is largely accurate: though some errors occur, large drops of performance in the Sharded setting (beyond 2%) are not due to errors caused by the simulator.

话语级错误并不总会影响整段模拟的有效性。在某些情形中，我们观察到用户模拟器会在对话后续轮次中纠正早期轮次的错误，或者错误答案尝试上的抽取错误发生在某一轮，但随后抽取又成功了。概括地说，我们从经验上发现模拟环境大体准确：尽管会出现一些错误，Sharded 设定中超过 2% 的大幅成绩下降，并不是由模拟器造成的错误所致。

Appendix E Concrete Example of Loss in Aptitude vs. Reliability

附录 E 能力上限损失与可靠性损失的具体例子

N=10

N=10

Finally, let’s assume that the Full performance is achieved by having perfect performance (i.e., success in 10/10 randomly sampled runs) on 9 instructions, and failing on all the sampled simulations of the last, tenth instruction. In other words:

最后，假设 Full 表现是这样取得的：在 9 条指令上表现完美（ 即 在 10/10 次随机抽样运行中都成功），而在最后第十条指令的全部抽样模拟中都失败。也就是说：

S^{\textsc{Full}}_{ij}=\begin{cases}100,&\text{if }i\in{1,\dots,9}\0,&\text{if }i=10\end{cases},

S^{\textsc{Full}}_{ij}

S^{\textsc{Full}}_{ij}

\overline{P}=60

\overline{P}=60

\overline{P}=60

\overline{P}=60

Situation 1: Drop in Aptitude. The LLM achieves perfect performance on six of the ten instructions:

情形 1：能力上限下降。 大模型在十条指令中的六条上取得完美表现：

S^{\textsc{Sharded}}_{ij}=\begin{cases}100,&\text{if }i\in{1,\dots,6}\0,&\text{if }i\in{7,\dots,10}\end{cases}.

\overline{P}=60

\overline{P}=60

Situation 2: Drop in Reliability. The LLM achieves mixed performance (6-7 perfect scores per instruction) on nine of the 10 instructions:

情形 2：可靠性下降。 大模型在 10 条指令中的九条上取得混合表现（每条指令有 6-7 个完美分数）：

S^{\textsc{Sharded}}_{ij}=\begin{cases}100,&\text{if }1\leq i\leq 3,1\leq j\leq 6\100,&\text{if }4\leq i\leq 9,1\leq j\leq 7\0,&\text{otherwise}\end{cases}.

\overline{P}=60

\overline{P}=60

Situations 1 and 2 illustrate extreme scenarios where the average drop in performance is entirely explained by a drop in aptitude or reliability, but in practice a combination is more likely to occur, as in situation 3.

情形 1 和情形 2 说明的是极端情景：平均成绩下降完全由能力上限或可靠性的下降来解释。但在实践中，更可能出现的是二者的组合，如情形 3。

Situation 3: Combined drop in Aptitude and Reliability. The LLM achieves perfect performance on three instructions, and mixed performance (6 perfect scores per instruction) on five of the 10 instructions:

情形 3：能力上限与可靠性同时下降。 大模型在三条指令上取得完美表现，并在 10 条指令中的五条上取得混合表现（每条指令有 6 个完美分数）：

S^{\textsc{Sharded}}_{ij}=\begin{cases}100,&\text{if }1\leq i\leq 3\100,&\text{if }4\leq i\leq 8,1\leq j\leq 6\0,&\text{otherwise}\end{cases}.

\overline{P}=60

\overline{P}=60

Finally, we note that though this concrete example we provide uses binary scores (0 and 100) for simulated conversation outcomes, aptitude (A) and unreliability (U) can equally be applied to continuous metrics (such as BLEU).

最后我们指出，尽管这里给出的具体例子对模拟对话结果使用二元分数（0 和 100），能力上限（A）和不可靠性（U）同样可以用于连续指标（如 BLEU）。

Appendix F Qualitative Analyses of Simulation Logs

附录 F 模拟日志的定性分析

In the following subsections, we report qualitative analyses on the corpus of simulations from the main experiment (Section 6.1). The purpose of the analyses is to discern root causes in model behavior that lead to performance degradation. We identify four behaviors below and provide the analysis for each item in the rest of the section:

在以下各小节中，我们报告对主实验（第 6.1 节）模拟语料的定性分析。分析的目的是辨认导致成绩下降的模型行为根因。我们识别出下面四种行为，并在本节其余部分为每一项提供分析：

LLMs attempt to answer the entire problem prematurely.

大模型过早尝试回答整个问题。

LLMs overly rely on previous (incorrect) answer attempts, leading to lengthier “bloated” answers.

大模型过度依赖此前（并不正确的）答题尝试，导致答案越来越长、出现“膨胀”。

LLMs overly adjust their answers based on the last conversation turn, materialized by a pronounced forgetting of middle-turns.

大模型过分根据对话的最后一轮调整答案，表现为对中间轮次明显的遗忘。

LLMs produce answers that are overly verbose, which likely introduce problem assumptions that detract attention from user-utterances.

大模型产生过于冗长的答案，这很可能引入问题假设，并把注意力从用户话语上引开。

\overline{P}

	Conversation Progress At First Answer Attempt
Model	0-20%	20-40%	40-60%	60-80%	80-100%
First answer attempt is …	earliest	early	midway	late	latest
3.1-8B	16.1	24.0	35.3	39.6	39.7
OLMo2	17.6	32.7	37.7	47.3	26.4
3-Haiku	27.1	35.6	47.4	58.9	70.3
4o-mini	30.2	39.2	48.4	58.2	59.9
3.3-70B	33.3	40.1	51.2	60.0	69.3
Phi-4	25.7	33.1	47.0	53.0	57.9
CMD-A	38.0	42.9	56.5	65.5	73.5
4-Scout	39.8	36.8	51.0	57.9	64.8
o3	21.0	37.9	51.9	58.4	68.0
3.7-Sonnet	29.2	35.6	55.3	68.0	71.6
R1	39.5	43.1	53.5	66.4	50.2
4o	36.0	41.4	56.2	65.6	90.4
2.5-Flash	39.0	48.6	60.2	70.8	74.6
4.1	33.9	52.7	60.6	69.0	78.6
2.5-Pro	41.1	45.7	53.5	64.6	63.8
Average	30.9	40.5	51.7	60.4	64.4

\overline{P}

	首次答题尝试时的对话进度
模型	0-20%	20-40%	40-60%	60-80%	80-100%
首次答题尝试……	最早	较早	中途	较晚	最晚
3.1-8B	16.1	24.0	35.3	39.6	39.7
OLMo2	17.6	32.7	37.7	47.3	26.4
3-Haiku	27.1	35.6	47.4	58.9	70.3
4o-mini	30.2	39.2	48.4	58.2	59.9
3.3-70B	33.3	40.1	51.2	60.0	69.3
Phi-4	25.7	33.1	47.0	53.0	57.9
CMD-A	38.0	42.9	56.5	65.5	73.5
4-Scout	39.8	36.8	51.0	57.9	64.8
o3	21.0	37.9	51.9	58.4	68.0
3.7-Sonnet	29.2	35.6	55.3	68.0	71.6
R1	39.5	43.1	53.5	66.4	50.2
4o	36.0	41.4	56.2	65.6	90.4
2.5-Flash	39.0	48.6	60.2	70.8	74.6
4.1	33.9	52.7	60.6	69.0	78.6
2.5-Pro	41.1	45.7	53.5	64.6	63.8
平均	30.9	40.5	51.7	60.4	64.4

During Sharded simulation, responses are classified according to a seven-class conversation strategy categorization. In particular, each assistant response is tagged as being a formal answer attempt or not (as answer attempts require further processing: extraction and evaluation by the task-specific evaluator).

在 Sharded 模拟中，回复会按七类对话策略分类。尤其是，每条助手回复都会被标记为是否构成正式的 答题尝试 （因为答题尝试需要进一步处理：由任务专用评估器抽取并打分）。

On the onset of conversation, LLMs have the least amount of information (highest level of underspecification), and are least likely to formulate correct answer attempts. Proposing a solution early might therefore plant certain incorrect elements in it, which wrongly influences the interaction later in the conversation.

在对话开始时，大模型掌握的信息最少（欠指定程度最高），也最不可能形成正确的答题尝试。因此，过早提出解答可能会把某些不正确的要素种进去，并在之后的对话中错误地影响交互。

To evaluate this hypothesis, we bin all simulated conversations from our experiments based on how early in the conversation the first answer attempt is generated by the LLM. Specifically, we create five bins: 0-20% if the first answer attempt occurs within the first 20% turns of the conversation, and 20-40%, 40-60%, 60-80%, and 80-100% if it occurs in later turns of the conversation. Of the six tasks included in our experiments, only two (Math and Code) observed a significant range in LLM behavior for answer attempt timing. For the other four tasks, models attempt an answer from the first turn in most of the time, rendering analysis on this parameter impossible.

为了评估这一假设，我们根据大模型首次生成答题尝试的早晚，把实验中的全部模拟对话分箱。具体地说，我们创建五个箱子：若首次答题尝试发生在对话前 20% 的轮次内，则记为 0-20%；若发生在更靠后的轮次，则记为 20-40%、40-60%、60-80% 和 80-100%。在我们纳入实验的六项任务中，只有两项（数学和代码）在答题尝试时机上观察到模型行为有显著范围。对其余四项任务，模型大多数时候从第一轮就开始尝试作答，使得无法就这一参数做分析。

Analysis results for the two remaining tasks are presented in Table 6. We observe that for every single model, conversations with a later first answer attempt lead to higher averaged performance. Across all models, conversations with the first attempt being made in the first 20% of conversations achieve a score of 30.9, less than half of the 64.4 when the LLM waits for the last 20% of the conversation to make an answer attempt.

对其余两项任务的分析结果见表 6。我们观察到，对每一个模型，首次答题尝试更晚的对话都会带来更高的平均表现。在所有模型上，首次尝试发生在对话前 20% 的对话得分为 30.9，不到大模型等到对话最后 20% 才做答题尝试时所得 64.4 的一半。

In other words, we find that premature answer attempts detract LLM performance. Conversations where the model clarifies user instructions or discusses the problem at a high-level before moving to generating complete answer attempts lead to higher performance. We hypothesize that this is due to the model making incorrect assumptions in premature solutions, which conflict with subsequent user instructions in later turns.

换句话说，我们发现过早的答题尝试会损害大模型表现。模型先澄清用户指令或在较高层面讨论问题、然后再生成完整答题尝试的对话，会带来更高表现。我们推测，这是因为模型在过早的解答中做出了不正确的假设，这些假设与后续轮次中的用户指令发生冲突。

F.2 Answer Bloat in Multi-Turn Conversation Figure 9: Average length (in number of characters) of answer attempts across four tasks (Code, Database, Data-to-text, and Summary) in Sharded conversations. Answer attempts in the Full and Concat settings tend to be shorter on average than those from Sharded setting. Sharded answer attempts increase in length as the LLMs make more answer attempts.

F.2 多轮对话中的答案膨胀 图 9：四项任务（代码、数据库、数据到文本和摘要）在 Sharded 对话中答题尝试的平均长度（以字符数计）。Full 和 Concat 设定中的答题尝试平均往往短于 Sharded 设定。随着大模型做出更多答题尝试，Sharded 答题尝试的长度会增加。

In multi-turn conversation simulations, the LLM might make multiple answer attempts, with each subsequent attempt being potentially based on previous attempts. In contrast, single-turn conversations constrain conversation dynamics, with the LLM making a single, first-and-final answer attempt.

在多轮对话模拟中，大模型可能做出多次答题尝试，每一次后续尝试都可能基于先前的尝试。相比之下，单轮对话约束了对话动态，大模型只做一次、也是最终一次答题尝试。

To understand multi-turn conversation dynamics, we calculate the average length of answer attempts in each simulation type. For the Sharded setting, we calculate average length for each attempt within a simulation (i.e., average length of the first attempt, second attempt, third attempt, etc.). We note for readers here that the analysis is conducted on extracted answer attempts (output of the Answer Extractor module in Figure 3) rather than the entire assistant responses. The extracted answer more accurately measures dynamics in answer attempts (i.e., generated SQL query, or Python function) rather than the entire responses, which might contain varying amounts of unrelated content.

为了理解多轮对话动态，我们计算每种模拟类型中答题尝试的平均长度。对 Sharded 设定，我们计算一次模拟中每一次尝试的平均长度（ 即 第一次尝试、第二次尝试、第三次尝试等的平均长度）。这里提醒读者，分析是在抽出的答题尝试上进行的（也就是图 3 中答案抽取模块的输出），而不是整段助手回复。抽出的答案能更准确地衡量答题尝试中的动态（即生成的 SQL 查询或 Python 函数），而不是整段回复——后者可能包含数量不等的无关内容。

Results of the analysis are plotted in Figure 9. Across the four tasks, we find that answer lengths in the Full and Concat settings tend to be similar, typically within 2-10% of each other. On three of the analyzed tasks (Code, Database, Summary), the first answer attempt in the Sharded setting has a similar length to Full and Concat counterparts, yet for each subsequent answer attempt, we observe an increase in average answer length. The effect is such that the final answer attempts in Sharded conversations (right portion of the four plots) tend to be 20-300% longer than the solutions generated in the Full and Concat settings. We name this observation the answer bloat effect : as a multi-turn conversation progresses, the LLM generates incorrect answer attempts, making assumptions about portions of the instruction that remain unspecified. As the user reveals additional information in succeeding turns, the LLM does not successfully invalidate its prior assumptions and overly relies on its previous attempts. Answer bloat in multi-turn, underspecified conversation leads to longer solutions compared to single-turn equivalents.

分析结果绘制于图 9。在四项任务上，我们发现 Full 和 Concat 设定中的答案长度往往相近，通常彼此相差 2-10%。在所分析的三项任务（代码、数据库、摘要）上，Sharded 设定中的第一次答题尝试长度与 Full 和 Concat 的对应物相近，但每一次后续答题尝试，我们都观察到平均答案长度增加。其效果是：Sharded 对话中的最后一次答题尝试（四张图的右侧部分）往往比 Full 和 Concat 设定中生成的解答长 20-300%。我们把这一观察称为 答案膨胀效应 ：随着多轮对话推进，大模型生成不正确的答题尝试，并对指令中尚未指定的部分做出假设。当用户在后续轮次中揭示更多信息时，大模型未能成功推翻自己先前的假设，并过度依赖先前的尝试。多轮、欠指定对话中的答案膨胀，导致解答比单轮对应物更长。

\overline{P}

\overline{P}

F.3 Over-adjust based on Last Turn of Conversation Because the summary task requires the assistant to attribute its summary back to documents through citation, the task offers a unique opportunity to analyze what turns of information LLMs pay attention to as the multi-turn conversation progresses. As a reminder, the summary task involves a user introducing new documents at each turn. The focus of our analysis is therefore to understand whether document introduction order (across turns) affects the likelihood of the LLM citing a document.

F.3 过分根据对话最后一轮做调整 因为摘要任务要求助手通过引用把摘要归因到文档，该任务提供了一个独特机会，用来分析随着多轮对话推进，大模型把注意力放在哪些轮次的信息上。提醒一下，摘要任务中用户会在每一轮引入新文档。因此，我们分析的焦点是：文档被引入的顺序（跨轮次）是否影响大模型引用某篇文档的可能性。

In Figure 10, we plot the the results of our analysis. Each row corresponds to the analysis of summaries generated at a given turn in the sharded simulation. At turn 1 (top row), 96% of the cited documents were introduced in the first turn. The missing 4% correspond to hallucinated citation to documents that were not introduced, and explains why none of the rows’ distribution sum to 100%. At turn two (second row from the top), summaries include citation in roughly equal proportion for turn-1 and turn-2 documents (i.e., 48% and 49%).

在图 10 中，我们绘出分析结果。每一行对应分片模拟中某一轮所生成摘要的分析。在第 1 轮（最上一行），96% 被引用的文档是在第一轮引入的。缺失的 4% 对应于对尚未引入文档的幻觉引用，这也解释了为什么没有任何一行的分布加总为 100%。在第二轮（从上数第二行），摘要对第 1 轮和第 2 轮文档的引用比例大致相当（即 48% 和 49%）。

Figure 10: Analysis of citation patterns in summaries generated by LLMs with the Sharded simulation. At each turn, the LLM generates an updated summary (y-axis), which includes citations from the documents that have been revealed up to this turn. Percentages in a row do not add up to 100% due to citation hallucinations that occur for some models.

图 10：对大模型在 Sharded 模拟中生成的摘要所做的引用模式分析。每一轮，大模型生成一份更新后的摘要（纵轴），其中包含到该轮为止已揭示文档的引用。一行中的百分比加总不为 100%，是因为某些模型会出现引用幻觉。

We interpret this to mean that in 2-turn conversations, LLMs pay roughly equal attention to documents in either turn. Analysis of summaries generated in turns 3-8 of sharded simulations reveal an imbalance in the documents the LLM cites to. In eighth-turn summaries, 20% of citations are to documents introduced in turn 8, compared to 8% from turn 2 and 3 (150% difference). At a high-level, as the conversation progresses, LLMs are most likely to cite either documents in the first or last turns, and less likely to cite documents introduced in intermediary (middle) turns. This finding mirrors findings of a loss-in-the-middle phenomena of LLMs paying more attention to documents at the start or end of their provided context, at the cost of middle-context content [29, 50, 40]. In short, we observe that the lost-in-the-middle phenomena occurs not only in single-turn long-context settings, but also in multi-turn conversation. We name this phenomenon loss-in-middle-turns .

我们将其解释为：在两轮对话中，大模型对两轮文档的关注大致相当。对分片模拟第 3-8 轮所生成摘要的分析，则揭示出大模型引用文档时的不平衡。在第八轮摘要中，20% 的引用指向第 8 轮引入的文档，而第 2 轮和第 3 轮各只有 8%（相差 150%）。概括地说，随着对话推进，大模型最可能引用第一轮或最后一轮中的文档，而不太可能引用中间轮次引入的文档。这一发现呼应了 中间丢失 （loss-in-the-middle）现象：大模型对所给上下文开头或结尾的文档关注更多，代价是中间上下文的内容 [29, 50, 40]。简而言之，我们观察到中间丢失现象不仅出现在单轮长上下文设定中，也出现在多轮对话中。我们把这一现象称为 中间轮次丢失 （loss-in-middle-turns）。

We note that the analysis presented in Figure 10 averages numbers across the 15 LLMs included in our main experiment. Even though we observe some loss-in-middle-turns in all models, the magnitude of the effect varies across models, typically with more performant models having a more muted effect, showing they have better capabilities of handling attribution across multiple turns of context. We do not include model-specific analyses in this work and leave it for future work.

我们指出，图 10 给出的分析是对主实验所纳入 15 个大模型的数字取平均。尽管我们在所有模型上都观察到一定程度的中间轮次丢失，效应幅度因模型而异，通常更强的模型效应更弱，表明它们更有能力处理跨多轮上下文的归因。本文不纳入按模型拆分的分析，将其留待未来工作。

\overline{P}

	Relative Assistant Verbosity
Task	0-20%	20-40%	40-60%	60-80%	80-100%
Assistants responses are …	shortest	short	median	long	longest
Code	55.3	52.3	48.9	46.9	42.5
Math	62.9	64.0	62.1	60.9	56.1
Database	43.8	40.0	37.3	34.3	31.3
Actions	41.5	49.6	54.2	53.6	50.8
Data-to-Text	25.0	24.3	24.0	23.1	21.8
Summary	15.4	14.7	13.5	12.0	10.3
Average	40.7	40.8	40.1	38.6	35.6

\overline{P}

	助手的相对冗长度
任务	0-20%	20-40%	40-60%	60-80%	80-100%
助手回复……	最短	较短	中位	较长	最长
代码	55.3	52.3	48.9	46.9	42.5
数学	62.9	64.0	62.1	60.9	56.1
数据库	43.8	40.0	37.3	34.3	31.3
动作	41.5	49.6	54.2	53.6	50.8
数据到文本	25.0	24.3	24.0	23.1	21.8
摘要	15.4	14.7	13.5	12.0	10.3
平均	40.7	40.8	40.1	38.6	35.6

When simulating multiple conversations based on a common instruction, we observe variation in responses, particularly in the length of the response generated by the LLM. To understand how verbosity (length of a response) affects model performance, we perform a verbosity analysis.

在基于同一条指令模拟多段对话时，我们观察到回复存在变异，尤其是大模型生成回复的长度。为了理解冗长度（回复长度）如何影响模型表现，我们做了一项冗长度分析。

N=10

N=10

On five of the six tasks, performance is 10-50% higher in simulated conversations with shortest response length, compared to conversations with longest response length. As assistant responses get longer (left to right in the Table), performances gradually drop. The Actions task is the only task where such an effect is not observed, and where shortest response length from the assistant is detrimental.

在六项任务中的五项上，回复长度最短的模拟对话，成绩比回复长度最长的对话高 10-50%。随着助手回复变长（表中从左到右），成绩逐渐下降。动作任务是唯一没有观察到这种效应的任务，在该任务上助手回复最短反而有害。

Predominantly however, models achieve higher performance when they generate shorter responses. We hypothesize that deterioration due to over-verbosity is due to longer responses typically containing more assumptions or hypotheses from the assistant, which can lead to confusion in following turns. On the other hand, short turns tend to be focused (e.g, a single clarification question), and keep the conversation on track.

不过总体而言，模型在生成较短回复时成绩更高。我们推测，过度冗长导致的恶化，是因为更长的回复通常包含助手更多的假设或猜想，这可能在后续轮次中带来混乱。另一方面，短轮次往往更聚焦（例如只提出一个澄清问题），并能让对话保持在轨道上。

Deterioration due to over-verbosity is note-worthy, as besides deteriorating underlying model performance, longer responses also take longer for users to read, which is undesirable. The finding therefore indicates that longer LLM responses are bad both for models and end-users.

过度冗长造成的恶化值得注意，因为除了损害底层模型表现外，更长的回复也需要用户花更长时间阅读，而这并不可取。因此，这一发现表明，更长的大模型回复对模型和对终端用户都不利。

Appendix G Assistant Response Categorization

附录 G 助手回复的分类

We categorize each assistant response into one of the seven categories to capture the answer attempt and evaluate if that is the case, as well as to understand the model behavior tendency. Herlihy et al. [27] defined seven turn categories for LLM responses and classified them using LLM, uncovering that GPT-4 prefers answering directly even when the query is underspecified. Motivated by this study, we similarly define seven response categories which we list in Table 8, together with example responses. Key differences are discussion and answer attempt; we observed many responses containing large body of text formulating the question in our preliminary experiments, which led to redefining “Miscellaneous” from [27] into “Discussion” in our experiment. “Direct Response” in [27] corresponds to our “Answer Attempt.”

我们把每条助手回复归入七类之一，以便捕捉答题尝试并判断是否确实如此，同时也便于理解模型的行为倾向。Herlihy 等人 [27] 为大模型回复定义了七种轮次类别，并用大模型加以分类，发现即便查询欠指定，GPT-4 也更倾向于直接作答。受该研究启发，我们同样定义了七种回复类别，连同示例回复列于表 8。关键差别在于讨论和答题尝试；在初步实验中，我们观察到许多回复包含大段文字来表述问题，于是把 [27] 中的“Miscellaneous”重新定义为实验中的“Discussion”。[27] 中的“Direct Response”对应于我们的“Answer Attempt”。

\ldots

Name	Description	Example
Answer attempt	The response contains a complete answer attempt to the question that can be extracted verbatim.	The dog is 50 meters away from the house.
Clarification	The response is a brief single question that directly inquires about one aspect of the query.	To calculate the distance, I need to know how long the dog ran. Could you provide more information about that?
Interrogation	The response contains multiple questions addressed to the user.	I cannot answer the question without knowing (1) speed, (2) duration, and (3) starting position. Please tell me about these points and I can calculate the distance!
Discussion	The response discusses the question in detail without answering, asking, or refusing to answer.	The question is trying to measure the distance between the dog and the house. We can calculate based on this equation: [Equation]. [ $\ldots$ ]
Hedging	The response provides multiple answer candidates based on hypotheticals (ifs, cases).	1. If the dog was originally in the house, it would be 50 meters away now. 2. If the dog was at the park, it would be 100 meters away from the house now.
Refusal	The response refuses to answer the question without a follow-up question or a request.	I can’t answer your question because I don’t have sufficient information.
Missing	The response is empty.	[blank]

\ldots

名称	描述	示例
答题尝试	回复包含对该问题的一次完整答题尝试，并且可以按原文抽出。	那条狗离房子有 50 米。
澄清	回复是一个简短的单一问题，直接询问查询的某一个方面。	要计算距离，我需要知道那条狗跑了多久。你能再提供一些相关信息吗？
追问	回复包含向用户提出的多个问题。	在不知道 (1) 速度、(2) 持续时间和 (3) 起始位置的情况下，我无法回答这个问题。请告诉我这些要点，我就能计算距离！
讨论	回复详细讨论该问题，但既不回答，也不提问，也不拒绝回答。	这个问题试图测量狗和房子之间的距离。我们可以基于这个方程来计算：[方程]。[ $\ldots$ ]
含糊其辞	回复基于假设情形（如果、分情况）给出多个候选答案。	1. 如果那条狗原来在房子里，它现在会在 50 米外。2. 如果那条狗原来在公园，它现在会离房子 100 米。
拒绝	回复拒绝回答该问题，且没有后续提问或请求。	我无法回答你的问题，因为我没有足够的信息。
缺失	回复为空。	[空白]

Appendix H Model Access

附录 H 模型访问

T=1.0

T=1.0

Short Form Name Version Access Provider 4o GPT-4o gpt-4o-2024-11-20 OpenAI / Microsoft API 4o-mini GPT-4o-mini gpt-4o-mini-2024-07-18 OpenAI API 4.1 GPT-4.1 gpt-4.1-2025-04-14 OpenAI / Microsoft API o3 o3 o3-2025-04-16 OpenAI / Microsoft API 3-Haiku Claude 3 Haiku claude-3-haiku-20240307 Amazon Bedrock 3.7-Sonnet Claude 3.7 Sonnet claude-3-7-sonnet-20250219 Amazon Bedrock 2.5-Flash Gemini 2.5 Flash gemini-2.5-flash-preview-04-17 Gemini API 2.5-Pro Gemini 2.5 Pro gemini-2.5-pro-preview-03-25 Gemini API 3.1-8B Llama-3.1-8B-Instruct N/A Local Ollama 3.3-70B Llama-3.3-70B-Instruct N/A Amazon Bedrock 4-Scout Llama-4-Scout-17B-16E N/A Together AI CMD-A Command-A command-a-03-2025 Cohere API R1 Deepseek-R1 N/A Amazon Bedrock OLMo2 OLMo2-13B N/A Local Ollama Phi-4 Phi-4 N/A Local Ollama Table 9: Specific model versions used as part of our experiments. For each model, we define the exact Version of the model accessed (for models that have versioning) and the Access Provider to facilitate result reproducibility.

Short Form	Name	Version	Access Provider
4o	GPT-4o	gpt-4o-2024-11-20	OpenAI / Microsoft API
4o-mini	GPT-4o-mini	gpt-4o-mini-2024-07-18	OpenAI API
4.1	GPT-4.1	gpt-4.1-2025-04-14	OpenAI / Microsoft API
o3	o3	o3-2025-04-16	OpenAI / Microsoft API
3-Haiku	Claude 3 Haiku	claude-3-haiku-20240307	Amazon Bedrock
3.7-Sonnet	Claude 3.7 Sonnet	claude-3-7-sonnet-20250219	Amazon Bedrock
2.5-Flash	Gemini 2.5 Flash	gemini-2.5-flash-preview-04-17	Gemini API
2.5-Pro	Gemini 2.5 Pro	gemini-2.5-pro-preview-03-25	Gemini API
3.1-8B	Llama-3.1-8B-Instruct	N/A	Local Ollama
3.3-70B	Llama-3.3-70B-Instruct	N/A	Amazon Bedrock
4-Scout	Llama-4-Scout-17B-16E	N/A	Together AI
CMD-A	Command-A	command-a-03-2025	Cohere API
R1	Deepseek-R1	N/A	Amazon Bedrock
OLMo2	OLMo2-13B	N/A	Local Ollama
Phi-4	Phi-4	N/A	Local Ollama

简称 名称 版本 访问提供方 4o GPT-4o gpt-4o-2024-11-20 OpenAI / Microsoft API 4o-mini GPT-4o-mini gpt-4o-mini-2024-07-18 OpenAI API 4.1 GPT-4.1 gpt-4.1-2025-04-14 OpenAI / Microsoft API o3 o3 o3-2025-04-16 OpenAI / Microsoft API 3-Haiku Claude 3 Haiku claude-3-haiku-20240307 Amazon Bedrock 3.7-Sonnet Claude 3.7 Sonnet claude-3-7-sonnet-20250219 Amazon Bedrock 2.5-Flash Gemini 2.5 Flash gemini-2.5-flash-preview-04-17 Gemini API 2.5-Pro Gemini 2.5 Pro gemini-2.5-pro-preview-03-25 Gemini API 3.1-8B Llama-3.1-8B-Instruct N/A Local Ollama 3.3-70B Llama-3.3-70B-Instruct N/A Amazon Bedrock 4-Scout Llama-4-Scout-17B-16E N/A Together AI CMD-A Command-A command-a-03-2025 Cohere API R1 Deepseek-R1 N/A Amazon Bedrock OLMo2 OLMo2-13B N/A Local Ollama Phi-4 Phi-4 N/A Local Ollama 表 9：实验中使用的具体模型版本。对每个模型，我们给出所访问模型的确切 Version（对有版本管理的模型）以及 Access Provider，以便于结果复现。

简称	名称	版本	访问提供方
4o	GPT-4o	gpt-4o-2024-11-20	OpenAI / Microsoft API
4o-mini	GPT-4o-mini	gpt-4o-mini-2024-07-18	OpenAI API
4.1	GPT-4.1	gpt-4.1-2025-04-14	OpenAI / Microsoft API
o3	o3	o3-2025-04-16	OpenAI / Microsoft API
3-Haiku	Claude 3 Haiku	claude-3-haiku-20240307	Amazon Bedrock
3.7-Sonnet	Claude 3.7 Sonnet	claude-3-7-sonnet-20250219	Amazon Bedrock
2.5-Flash	Gemini 2.5 Flash	gemini-2.5-flash-preview-04-17	Gemini API
2.5-Pro	Gemini 2.5 Pro	gemini-2.5-pro-preview-03-25	Gemini API
3.1-8B	Llama-3.1-8B-Instruct	N/A	Local Ollama
3.3-70B	Llama-3.3-70B-Instruct	N/A	Amazon Bedrock
4-Scout	Llama-4-Scout-17B-16E	N/A	Together AI
CMD-A	Command-A	command-a-03-2025	Cohere API
R1	Deepseek-R1	N/A	Amazon Bedrock
OLMo2	OLMo2-13B	N/A	Local Ollama
Phi-4	Phi-4	N/A	Local Ollama

Appendix I Task-specific Implementation details

附录 I 任务专用实现细节

We provide task implementation details. For each task, we specify: (1) the selection of original single-turn fully-specified instruction, (2) the evaluation metric that was repurposed from the original dataset, (3) and what the initial system messages consists of (if any).

我们给出任务实现细节。对每项任务，我们说明：（1）原始单轮完整指定指令的选取，（2）从原数据集复用的评估指标，（3）以及初始系统消息由什么构成（如果有的话）。

I.1 Code The Code instructions are sourced from a combination of HumanEval [10], a dataset of 164 basic Python programming problems given the function header and the docstring that specifies the problem, and LiveCodeBench [31], an evolving dataset of Python algorithmic challenges. In particular, we source from the “call-based” problem subset in LiveCodeBench v5, with the difficulty of either “Easy” and “Medium”, to align the solution formats between the two sources.

I.1 代码 代码指令来自 HumanEval [10] 与 LiveCodeBench [31] 的组合。前者是 164 个基础 Python 编程问题，给定函数头和说明问题的文档字符串；后者是不断演进的 Python 算法挑战数据集。我们特别从 LiveCodeBench v5 的“call-based”问题子集中取材，难度为“Easy”或“Medium”，以便让两种来源的解答格式对齐。

We first sharded all HumanEval problems following the protocol mentioned in Appendix C, obtaining 45 high quality sets of shards that meet the criteria. The rest of the dataset were discarded because of being simplistic, leaving little room to construct sufficient number of shards for a problem. Subsequently, we shuffled and sharded the aforementioned subset from LiveCodeBench until obtaining 100 valid sharded instructions.

我们首先按附录 C 所述协议对全部 HumanEval 问题做分片，得到 45 组满足标准的高质量碎片。数据集的其余部分因过于简单而被丢弃，几乎没有空间为一道题构造足够数量的碎片。随后，我们打乱并分片上述 LiveCodeBench 子集，直到得到 100 条有效的分片指令。

We follow the original prompts used by the benchmark authors as much as possible for the single-turn (Full and Concat) evaluation. Specifically, Full prompt from HumanEval includes the function header and the docstring provided as prompt in HumanEval dataset, and Full & Concat from LiveCodeBench includes starter_code consisting of the function signature.

对单轮（Full 和 Concat）评估，我们尽可能沿用基准作者使用的原始提示。具体地说，来自 HumanEval 的 Full 提示包含 HumanEval 数据集中作为提示提供的函数头和文档字符串；来自 LiveCodeBench 的 Full 与 Concat 则包含由函数签名构成的 starter_code。

Both HumanEval- and LiveCodeBench-derived problems come with test cases which we use to compute the functional accuracy of the answer attempt by the LLMs. We re-use the evaluation codebase maintained by Jain et al. [31], which (1) wraps the candidate function in a test module, (2) execute given the inputs, and (3) checks the equivalence of the output from the expected output, with a default timeout set to prevent the evaluator from getting trapped during evaluation (e.g., brute-force implementation may not pass under the set time budget). In case when multiple code blocks are present in a response, the answer extraction module selects the last function definition in the last markdown code block.

源自 HumanEval 和 LiveCodeBench 的问题都带有测试用例，我们用它们来计算大模型答题尝试的功能正确率。我们复用 Jain 等人 [31] 维护的评估代码库，它会（1）把候选函数包进一个测试模块，（2）用给定输入执行，以及（3）检查输出是否与期望输出等价，并设置默认超时，以免评估器在评测中卡住（ 例如 暴力实现可能无法在既定时限内通过）。当一条回复中出现多个代码块时，答案抽取模块会选择最后一个 markdown 代码块中的最后一个函数定义。

I.2 Database The Database instructions are sourced from the validation portion of the Spider dataset [86]. We note that though a more recent version of Spider has been released (Spider 2.0 [44]), the instructions in the second iteration are more advanced and represent less typical database use, and we select instructions from the more realistic Spider 1.0.

I.2 数据库 数据库指令来自 Spider 数据集的验证部分 [86]。我们指出，尽管已经发布了更新版本的 Spider（Spider 2.0 [44]），第二版中的指令更偏高级，也不那么代表典型的数据库用法，因此我们从更贴近现实的 Spider 1.0 中选取指令。

The authors of Spider categorized queries into four levels of difficulty (EASY, MEDIUM, HARD, XHARD), based on the syntax complexity of a reference SQL query. We filtered out queries of EASY complexity, as they tended to yield fewer than three shards when processed. The rest of the 433 natural language queries in Spider were gradually sharded until reaching a total of 107 valid sharded instructions.

Spider 的作者根据参考 SQL 查询的句法复杂度，把查询分成四个难度等级（EASY、MEDIUM、HARD、XHARD）。我们过滤掉 EASY 复杂度的查询，因为它们在处理时往往产生少于三片。Spider 中其余 433 条自然语言查询被逐步分片，直到一共得到 107 条有效的分片指令。

Each original instruction in Spider supplies a database schema, represented in SQL as a series of table schema (i.e., each define a series of columns including name, type, and optional index). We include the database schema as part of the system message (i.e., prior to the first turn of conversation), and informing the LLM that users will provide natural-language queries that must be answered using a database with the provided schema.

Spider 中的每条原始指令都提供一个数据库模式，以 SQL 表示为一系列表模式（即每个定义一系列列，包括名称、类型和可选索引）。我们把数据库模式放进系统消息（也就是在对话第一轮之前），并告知大模型：用户将提供必须用带有所给模式的数据库来回答的自然语言查询。

Each original instruction in Spider is paired with a reference SQL solution. We follow Zhong et al. [90] for the evaluation methodology. For a given original instruction, the candidate and reference SQL queries are executed on a fixed set of databases, and exact match of the results on all databases is required to mark the candidate as successful (Score = 100). If a discrepancy is observed on any test database, the candidate is incorrect (Score = 0). One limitation of SQL execution is that false positives can occur: two queries can return the same output on a given database, even when they are not semantically equivalent. Zhong et al. [90] found that by evaluating on an increased number of databases, false positives become negligible. Finally, any invalid candidate that does not successfully execute (e.g., syntax error) is considered incorrect (Score = 0).

Spider 中的每条原始指令都配有一条参考 SQL 解答。我们遵循 Zhong 等人 [90] 的评估方法。对一条给定的原始指令，候选 SQL 查询和参考 SQL 查询会在一组固定的数据库上执行，只有在所有数据库上结果都精确匹配时，候选才被标为成功（Score = 100）。如果在任何一个测试数据库上观察到差异，候选就是不正确的（Score = 0）。SQL 执行的一项局限是可能出现假阳性：两条查询可以在给定数据库上返回相同输出，即使它们在语义上并不等价。Zhong 等人 [90] 发现，通过在更多数据库上评估，假阳性会变得可以忽略。最后，任何未能成功执行的无效候选（例如句法错误）都被视为不正确（Score = 0）。

I.3 Actions The Actions instructions are sourced from the released test portion of the Berkeley Function Calling Leaderboard V3 (BFCL) [85]. BFCL V3 consists of three sub-genre of instructions: (1) Parallel, (2) Multiple, and (3) Multiple-Parallel. Initial experimentation with the sub-genres identified Parallel as the most suited for sharding, as Parallel instructions specify multiple subtasks that should be used and combined into a single action that accomplishes the entirety of the instruction. We shuffled all the BFCL V3 Parallel instructions, and sharded gradually until we obtained 105 valid sharded instructions.

I.3 动作 动作指令来自 Berkeley Function Calling Leaderboard V3（BFCL）[85] 已发布的测试部分。BFCL V3 由三类指令构成：（1）Parallel，（2）Multiple，以及（3）Multiple-Parallel。对这些子类的初步实验表明，Parallel 最适合分片，因为 Parallel 指令指定多个子任务，这些子任务应被组合进一个能完成整条指令的单一动作。我们打乱全部 BFCL V3 Parallel 指令，并逐步分片，直到得到 105 条有效的分片指令。

We note that though a more recent iteration of BFCL includes multi-turn instructions, it differs from sharding experiments as it does not involve underspecification, with each turn having an independent intermediate solution (which we call episodic multi-turn conversations). Our implementation in comparison shards original instructions allowing us to simulate multi-turn underspecified conversations for this task setting. The Background section (Section 2) discusses the relationship between episodic and underspecified multi-turn conversation more in-depth.

我们指出，尽管更新一版的 BFCL 包含多轮指令，它与分片实验不同，因为它不涉及欠指定，每一轮都有独立的中间解答（我们称之为情节式多轮对话）。相比之下，我们的实现对原始指令做分片，从而可以在这一任务设定中模拟多轮欠指定对话。背景一节（第 2 节）更深入地讨论了情节式多轮对话与欠指定多轮对话之间的关系。

Each instruction in BFCL comes with tool set documentation, a JSON object that specifies the set of available actions (APIs) for the assistant to complete user instructions. We include the tool set documentation as part of the system message, along with a message indicating that user queries will require the use of the provided tools to be completed.

BFCL 中的每条指令都带有工具集文档，这是一个 JSON 对象，指定助手可用来完成用户指令的一组可用动作（API）。我们把工具集文档放进系统消息，并附带一条说明：用户查询将需要使用所提供的工具才能完成。

Each instruction in BFCL comes with a reference answer, consisting of the API calls that should be called to accomplish the user instruction. The maintainers of BFCL have released an evaluation toolkit that assesses semantic equivalence between a candidate answer and the reference answer. We leverage the official evaluation toolkit, assigning a score of S=100 for candidate answers that are considered semantically equivalent to the reference answer, and a score of S=0 otherwise. When the evaluation toolkit is not able to parse a candidate answer (e.g., a syntax error), the candidate is considered incorrect (S=0).

BFCL 中的每条指令都带有参考答案，由应当调用以完成用户指令的 API 调用构成。BFCL 的维护者发布了一个评估工具包，用来判断候选答案与参考答案之间的语义等价。我们使用官方评估工具包，对被认为与参考答案语义等价的候选答案赋分 S=100，否则赋分 S=0。当评估工具包无法解析候选答案时（例如句法错误），该候选被视为不正确（S=0）。

I.4 Math The Math instructions are sourced from the “main” portion of the GSM8K dataset [14]. We did not perform a filter on the original 8,700 instructions. We shuffled the instructions and sharded incrementally until we obtained 103 valid sharded instructions. Each GSM8K is paired with a numerical reference answer. We used the official toolkit released alongside GSM8K to standardize numerical answers (i.e., strip formatting, etc.). Standardized candidate numerical answers can then be compared through exact match to the reference answer. If the toolkit detects a match, the candidate answer is considered correct (Score=100), and incorrect otherwise (Score = 0). A short, single-sentence system prompt is used to indicate to the assistant that it will be solving mathematical problems.

I.4 数学 数学指令来自 GSM8K 数据集的“main”部分 [14]。我们没有对原来的 8,700 条指令做过滤。我们打乱这些指令并逐步分片，直到得到 103 条有效的分片指令。每条 GSM8K 都配有一个数值参考答案。我们使用随 GSM8K 一起发布的官方工具包来标准化数值答案（即去掉格式等）。标准化后的候选数值答案再通过精确匹配与参考答案比较。如果工具包检测到匹配，候选答案被视为正确（Score=100），否则不正确（Score = 0）。我们使用一句简短的单句系统提示，告知助手它将求解数学问题。

I.5 Data-to-Text The Data-to-Text instructions are based on instructions in the released test set ToTTo dataset [59]. In ToTTo, fully-specified instructions have the following information elements: (1) a HTML-formatted table extracted from a Wikipedia page, (2) a subset of cells in the table that have been highlighted, (3) the name of the Wikipedia page that included the Table, (4) the name of the Section in the Wikipedia page that included the Table. Given these elements, the task objective is to generate a caption for the Table specifically focusing on the highlighted cells and considering the available meta-data. Instructions were shuffled and sharded incrementally until we obtained 120 valid sharded instructions.

I.5 数据到文本 数据到文本指令基于 ToTTo 数据集已发布测试集中的指令 [59]。在 ToTTo 中，完整指定的指令包含以下信息要素：（1）从某个维基百科页面抽出的、以 HTML 格式给出的表格，（2）表格中一组被 高亮 的单元格，（3）包含该表的维基百科页面名称，（4）包含该表的维基百科页面中的章节名称。给定这些要素，任务目标是为该表生成一句说明，并特别聚焦于高亮单元格，同时考虑可用的元数据。指令被打乱并逐步分片，直到我们得到 120 条有效的分片指令。

For each instruction, we generate sharded instructions by assigning different information elements to individual shards. The first shard consists of the initial HTML-formatted table without highlighting. The second shard provides an updated table with the highlighting present, the third shard provides the Wikipedia page name, the fourth shard provides the Wikipedia Section name. Finally, a fifth shard provides a fixed set of 10 randomly-selected example captions from the training set of the ToTTo dataset.

对每条指令，我们通过把不同信息要素分配给各个碎片来生成分片指令。第一片由最初的、没有高亮的 HTML 格式表格构成。第二片提供带高亮的更新表格，第三片提供维基百科页面名称，第四片提供维基百科章节名称。最后，第五片提供从 ToTTo 数据集训练集中随机选出的固定 10 条示例说明。

Each instruction in ToTTo is assigned one to three reference captions that were collected by authors of the original dataset. Evaluation on a candidate caption calculates the BLEU score [58] between the candidate and the set of available references, following the evaluation methodology from the original paper.

ToTTo 中的每条指令被赋予一到三条参考说明，由原数据集作者收集。对候选说明的评估计算该候选与可用参考集合之间的 BLEU 分数 [58]，评测方法沿用原论文。

The Data-to-Text is a refinement task; at each turn, the model is provided an additional shard of information, and is explicitly told to update its response considering all the information provided so far. As a refinement task, assistant responses at each turn are automatically categorized as answer attempts, and the extracted answer is considered to be the entire response. The system instruction informs the model that its response should consist solely of a table caption, without additional text (such as intro, outro, or politeness wording).

数据到文本是一项 refinement 任务；在每一轮，模型会得到额外一片信息，并被明确告知要根据到目前为止提供的全部信息更新回复。作为 refinement 任务，每一轮的助手回复都会被自动归类为答题尝试，抽出的答案被视为整段回复。系统指令告知模型：它的回复应当只由一句表格说明构成，不要附加其他文字（例如开场白、收尾或客套用语）。

I.6 Summary The Summary instructions are based on samples of the Summary of a Haystack dataset [40]. We reuse the entire instructions from Summary of a Haystack to produce 92 sharded instructions. The original instructions each consist of a haystack - 100 documents for a total of 100,000 tokens of content - and a user query. The goal of the task is to generate a bullet-point-formatted summary of the query-relevant insights that occur in the collection of documents, and use citation to attribute information in each of the bullet points back to the source documents.

I.6 摘要 摘要指令基于 Summary of a Haystack 数据集中的样本 [40]。我们复用 Summary of a Haystack 的全部指令，生成 92 条分片指令。原始指令各自由一个 haystack ——100 篇文档，合计 100,000 个 token 的内容——以及一条用户查询构成。任务目标是生成一份条目式摘要，概括文档集合中与查询相关的见解，并用引用把每个条目中的信息归因到源文档。

The original setting of the Summary of a Haystack purposefully includes a large amount of redundancy (each insight is repeated across at least 6 documents) to evaluate LLMs’ ability to thoroughly cite sources. However, we simplify the task for the multi-turn setting, as the 100,000-token haystacks restrict the variety of models we can evaluate. We instead follow subsequent work in selecting smaller Haystacks (“mini-Haystacks”) [3]. Mini-Haystacks consist of 20 documents and ensure that each reference insight is repeated across three documents. For each instruction, we produce ten shards by randomly assigning two documents per shard. The initial shard further specifies high-level task instruction, by specifying the user query, the expected bullet-point format, with a formatted citation.

Summary of a Haystack 的原始设定有意包含大量冗余（每条见解至少在 6 篇文档中重复），以评估大模型全面引用源的能力。不过，我们为多轮设定简化了该任务，因为 100,000 token 的 haystack 会限制我们能评估的模型种类。我们转而沿用后续工作，选用更小的 Haystack（“mini-Haystacks”）[3]。Mini-Haystacks 由 20 篇文档构成，并保证每条参考见解在三篇文档中重复。对每条指令，我们通过随机把两篇文档分配给每一片，产生十片。初始的一片还进一步说明高层任务指令，给出用户查询、期望的条目格式，以及格式化的引用方式。

Summary of a Haystack relies on an LLM-based metric (Joint Score) to compute the quality of the summary in terms of both the relevance of the candidate bullet points (coverage) and the quality of the generated attribution within the bullet points (citation). The authors note that the metric is recall-based, such that longer summaries are likely to score higher than shorter ones. To account for length bias, the original task instructs models to generate summaries of at most 300 words, which we include in our experiments as well. Specifically, models are instructed in all settings to generate summaries of up to 300 words. We observed that in multi-turn settings, models often forget this instruction, leading to non-adherence to the instruction. To avoid penalizing models that correctly remain within the 300-word limit, we truncate summaries that go beyond the limit, removing words in equal proportion from summary bullet points, such that evaluated summaries all respect the 300-word limit. We note that this tendency for LLMs to go beyond is further discussed in Appendix F, where we observe that across tasks, model answer attempts get “bloated” over turns of conversations. In single-turn settings (full, concat), LLMs largely respect the 300-word length limit.

Summary of a Haystack 依靠一项基于大模型的指标（Joint Score）来计算摘要质量，既看候选条目的相关性（覆盖率），也看条目内生成归因的质量（引用）。作者指出该指标基于召回，因此较长的摘要往往比较短的摘要得分更高。为了校正长度偏差，原任务要求模型生成至多 300 词的摘要，我们也把这一点纳入实验。具体地说，在所有设定中，模型都被要求生成不超过 300 词的摘要。我们观察到，在多轮设定中，模型常常 忘记 这条指令，从而导致不遵守该要求。为了避免惩罚那些正确保持在 300 词限制内的模型，我们对超出限制的摘要做截断，按相同比例从各摘要条目中删词，使被评估的摘要都遵守 300 词限制。我们指出，大模型这种超出限制的倾向在附录 F 中有进一步讨论，我们在那里观察到：跨各项任务，模型的答题尝试会随着对话轮次“膨胀”。在单轮设定（full、concat）中，大模型大体遵守 300 词的长度限制。

The summary task is a refinement task. Assistant responses at each turn are automatically categorized as answer attempts, and the entire response is considered to be the extracted answer.

摘要任务是一项 refinement 任务。每一轮的助手回复都会被自动归类为答题尝试，整段回复被视为抽出的答案。

\to

\to

In initial experiments, we experimented with other sharding strategies, including breaking shards at a specific number of words (rather than sentence boundary), and increasing the length of documents (from 10 to 20 sentences), without observing significant differences in results. This led us to adopt the setting we describe: sharding every two sentences, and truncating at 10-sentences.

在初步实验中，我们还尝试了其他分片策略，包括按特定词数而不是按句界断开碎片，以及把文档长度从 10 句增加到 20 句，但没有观察到结果上的显著差异。这让我们采用这里描述的设定：每两个句子分一片，并截到 10 句。

We evaluated performance with the BLEU metric[58], the standard metric for translation tasks, which was used as well in the original WMT 2019 competition.

我们用 BLEU 指标 [58] 评估表现，这是翻译任务的标准指标，原 WMT 2019 竞赛也使用了它。

Appendix J Example Simulated Conversation

附录 J 模拟对话示例

Figure 11: Example simulated multi-turn conversation for the Math task. This conversation simulation was with assistant model Llama3.1-8B-Instruct. The sharded instruction consists of six shards. The correct answer to the instruction is 85,000 calories.

图 11：数学任务上一段模拟多轮对话的示例。这段对话模拟使用的助手模型是 Llama3.1-8B-Instruct。分片指令由六片构成。该指令的正确答案是 85,000 卡路里。

Figure 11 provides an example conversation that was simulated during our experiments in the sharded setting. The simulation was conducted on the Math task, with a 6-shard instruction, and using the Llama3.1-8B-Instruct as the assistant. This conversation illustrates the following properties described in the rest of the paper: (1) the LLM makes assumptions early in the conversation (in Turn 1, describing four pastries that are irrelevant), (2) although it correctly interprets user-provided information, it also unnecessarily updates the information for assumptions it made (Turn 4), (3) this leads to unnecessary complexity, and the model ultimately forgets that the initial instruction was to calculate total calorie count, and returns only half of the calculation (just for Mini Blueberry Muffin). In short, this conversation illustrates the lost in conversation phenomenon: when the user instruction is underspecified (Turns 1-4), the LLM makes assumptions that detract from the conversation and lead to incorrect or incomplete answers.

图 11 给出了我们在实验的分片设定中模拟出的一段对话示例。模拟在数学任务上进行，指令有 6 片，助手使用 Llama3.1-8B-Instruct。这段对话展示了正文其余部分描述的如下性质：（1）大模型在对话早期就做出假设（在第 1 轮描述了四种并不相关的糕点）；（2）尽管它正确理解了用户提供的信息，却也在不必要地更新自己先前做出的假设（第 4 轮）；（3）这带来了不必要的复杂性，模型最终忘记了初始指令是计算总卡路里，只返回了计算的一半（只算了 Mini Blueberry Muffin）。简而言之，这段对话展示了对话迷路现象：当用户指令欠指定时（第 1-4 轮），大模型会做出把对话带偏的假设，并导致不正确或不完整的答案。

Appendix K Gradual Sharding Implementation

附录 K 渐进分片的实现

To evaluate the effect of instruction granularity on performance degradations, we conducted the gradual sharding experiment .

为了评估指令粒度对成绩下降的影响，我们开展了 渐进分片实验 。

We selected sharded instructions that had exactly eight shards, leading to a total of eight instructions across three tasks (Code, Math, Data-to-Text). We then leveraged an LLM (GPT-4o) to expand each instruction into 7 variants with differing number of shards. The LLM was instructed to merge the original sharded instruction into a smaller sharded instruction with two to seven shards. The instruction authorized minor rephrasing to allow for individual shards to be fluent, but encouraged the LLM to remain as close as possible to the original instruction in wording.

我们选取恰好有八片的分片指令，在三项任务（代码、数学、数据到文本）上共得到八条指令。然后我们借助一个大模型（GPT-4o）把每条指令扩展成 7 个碎片数量不同的变体。该大模型被要求把原始分片指令 合并 成碎片数为二到七的更小分片指令。指令允许做轻度改写，以便各片读起来通顺，但鼓励大模型在措辞上尽可能接近原始指令。

As such, each of the original instruction can be paired to: (1) a concat instruction (one-shard), and (2) 7 sharded instructions, ranging from two to eight shards. Applying this method to the 31 instructions yields a total of 248 instructions, with an equal number for the number of shards (from 1 to 8) and on the identical underlying problems.

这样，每条原始指令都可以对应到：（1）一条 concat 指令（一片），以及（2）7 条分片指令，碎片数从两片到八片。把这种方法应用到 31 条指令上，一共得到 248 条指令，各种碎片数（从 1 到 8）数量相等，并且对应完全相同的底层问题。

We ran simulations using the 248 instructions, simulating 10 conversations per instruction and model for two models: GPT-4o and GPT-4o-mini. Findings of the gradual sharding experiment are described in Section 6.3.

我们用这 248 条指令运行模拟，对两个模型（GPT-4o 和 GPT-4o-mini）的每条指令、每个模型各模拟 10 段对话。渐进分片实验的发现见第 6.3 节。

Appendix L Temperature Experiment Implementation

附录 L 温度实验的实现

To evaluate the effect of temperature on aptitude and reliability of LLMs in single- and multi-turn settings, we conducted the following temperature experiment .

为了评估温度对大模型在单轮与多轮设定中能力上限和可靠性的影响，我们开展了如下 温度实验 。

We selected 10 instructions from each of four tasks: Code, Database, Actions, and Math (for a total of 40). We ran experiments with two models (GPT-4o and GPT-4o-mini). For each instruction and each temperature combination, we conducted simulations for three conversation settings: full, concat, and sharded. For each conversation setting, we varied temperature parameters to three values: 0.0, 0.5, and 1.0. For the full and concat setings, this corresponds to three temperature combinations (as only the assistant temperature can be modified), whereas there are a total of nine combinations for the sharded setting, as both the assistant and user temperature is varied.

我们从四项任务中各选取 10 条指令：代码、数据库、动作和数学（合计 40 条）。我们用两个模型（GPT-4o 和 GPT-4o-mini）做实验。对每条指令和每种温度组合，我们在三种对话设定中进行模拟：full、concat 和 sharded。对每种对话设定，我们把温度参数设为三个取值：0.0、0.5 和 1.0。对 full 和 concat 设定，这对应于三种温度组合（因为只能修改助手温度）；而对 sharded 设定则一共有九种组合，因为助手温度和用户温度都会变化。

We chose to increase the number of simulations to 20 runs per condition (compared to 10 in the main experiment), as the focus of the experiment is to measure variations in model aptitude and reliability, and added simulation runs lead to better percentile estimates used in calculating metrics. This added requirement was not computationally expensive as the temperature experiment involved a limited number of models (2 vs. 15) and instructions (40 vs. 600) in comparison to our main experiment.

我们选择把每种条件下的模拟次数增加到 20 次（主实验中为 10 次），因为该实验的重点是衡量模型能力上限和可靠性的变异，增加模拟次数能改善用于计算指标的百分位估计。这一额外要求在计算上并不昂贵，因为与主实验相比，温度实验涉及的模型更少（2 对 15）、指令也更少（40 对 600）。

Findings of the experiments are described in Section 7.2.

实验发现见第 7.2 节。

Appendix M Recap & Snowball Experiment Implementation

附录 M Recap 与 Snowball 实验的实现

N=10

N=10

Appendix N On obtaining deterministic outputs from LLMs

附录 N 关于从大模型获得确定性输出

As we demonstrated in our experimental results, setting the temperatures to zero still leads to high unreliability, due to compounding effect of subtle non-determinism over tokens and turns.

正如我们在实验结果中表明的，把温度设为零仍然会带来很高的不可靠性，原因是细微的非确定性会在 token 和轮次上产生复合效应。

T=0

T=0

Notable model providers acknowledge the non-determinism implicitly or explicitly; Anthropic recommends sampling multiple times to cross-validate output consistency,[^3] Google also highlights that their model outputs are mostly deterministic,[^4] and OpenAI recommends setting seed parameter to further reduce the non-determinism.[^5]

主要的模型提供方以或隐或显的方式承认这种非确定性；Anthropic 建议多次采样以交叉核验输出一致性，[^3] Google 也强调其模型输出 大体 是确定性的，[^4] OpenAI 则建议设置 seed 参数以进一步降低非确定性。[^5]

Nevertheless, we caution users that multi-turn conversations can be increasingly unreliable owing to divergent LLM responses.

尽管如此，我们仍提醒用户：由于大模型回复会分叉，多轮对话可能会变得越来越不可靠。

Appendix O Prompts

附录 O 提示

O.1 Sharding We show the prompts for the sharding process below, using Math as an example task. Double-bracketed terms are placeholders that get replaced with the actual data. Other tasks share the same outline with different exemplars and rules to enforce stable outputs. We refer the readers to the GitHub repository for the exact prompts on other tasks.

O.1 分片 我们在下面展示分片流程的提示，并以数学作为示例任务。双方括号中的词是会被实际数据替换的占位符。其他任务共享同一提纲，只是示例和用来约束稳定输出的规则不同。其他任务的准确提示，请读者参阅 GitHub 仓库。

切分 你会得到一条完整指定的指令，任务是把该指令切分成若干信息单元，每一单元只揭示指令中的一条信息。 你必须以如下 JSON 格式输出一份片段列表： [{"segment": "[指令中的原句摘录]"}, {"segment": "[指令中的原句摘录]"}, \dots] 规则： [互不重叠] 各片段必须互不重叠，并覆盖整条指令。对于原指令中非必要的部分（分隔符、标题等），你可以选择留下一些空隙。 [尽量最小] 你应当把片段中的信息拆得尽可能小。如果遇到并列表达式（X 和 Y），应当拆成两个片段。每个片段应代表一个信息单元。 示例查询： What are the names and locations of the stadiums that had concerts that occurred in both 2014 and 2015? 输出： {"segments": [{"segment": "names and locations"}, {"segment": "stadiums"}, {"segment": "concerts"}, {"segment": "in both 2014"}, {"segment": "and 2015"}]} 现在请为下面这条完整指定的指令完成该任务： [[INSTRUCTION]]

O.2 Experiments The experiments involve several LLM calls with specific prompts to simulate the conversation, which we list below. We refer readers to the GitHub repository for how they are incorporated.

O.2 实验 实验中会用若干带有特定提示的大模型调用来模拟对话，我们列在下面。它们如何被接入，请读者参阅 GitHub 仓库。

User simulator You are simulating a user of an interactive LLM system (like ChatGPT). You are simulating a user of an interactive LLM system (like ChatGPT). The user is inherently lazy, and answers in short form, providing only minimal information to the system. You should not be proactive. The user is inherently lazy, and answers in short form, providing only minimal information to the system. You should not be proactive. Here’s the conversation so far: Here’s the conversation so far: [[CONVERSATION_SO_FAR]] [[CONVERSATION_SO_FAR]] Here are the shards that have already been revealed: Here are the shards that have already been revealed: [[SHARDS_REVEALED]] [[SHARDS_REVEALED]] Here are all the shards that have not been revealed yet: Here are all the shards that have not been revealed yet: [[SHARDS_NOT_REVEALED]] [[SHARDS_NOT_REVEALED]] You must generate a response to the conversation so far. Here are the rules: You must generate a response to the conversation so far. Here are the rules: [Providing a shard] You can reveal the content of a shard to the system in your response if it will help the system move closer to answering the problem. You should select the shard to reveal that is most "basic" and currently the most relevant. [Providing a shard] You can reveal the content of a shard to the system in your response if it will help the system move closer to answering the problem. You should select the shard to reveal that is most "basic" and currently the most relevant. [One Shard at a Time] You should only reveal at most one shard at a time. [One Shard at a Time] You should only reveal at most one shard at a time. [Reveal Entire Shard] If you reveal a shard, you must make sure to include all the information in the shard. For example, if the shard is "your symptoms are that you have a headache in the mornings", your response can’t just be yeah I have headaches”, you must say yup mostly headaches in the mornings``. [Reveal Entire Shard] If you reveal a shard, you must make sure to include all the information in the shard. For example, if the shard is "your symptoms are that you have a headache in the mornings", your response can’t just be yeah I have headaches”, you must say yup mostly headaches in the mornings``. [Irrelevant Clarifications] If the system asks you a question irrelevant to the shards, asks you a generic question (Can you give me a hint?), you should respond with an answer that does not provide a shard. (I don’t know, Is that really important?, etc.) You should not reveal any information beyond what is available in the shards. [Irrelevant Clarifications] If the system asks you a question irrelevant to the shards, asks you a generic question (Can you give me a hint?), you should respond with an answer that does not provide a shard. (I don’t know, Is that really important?, etc.) You should not reveal any information beyond what is available in the shards. [No Repeated Shards] You should not reveal the same shard more than once. Carefully review the already revealed shards, and only reveal a shard if its shard_id is not on the list. [No Repeated Shards] You should not reveal the same shard more than once. Carefully review the already revealed shards, and only reveal a shard if its shard_id is not on the list. [Rephrase Shards] If you reveal a shard, you should rephrase it in a conversational way. Do not copy the shard verbatim. [Rephrase Shards] If you reveal a shard, you should rephrase it in a conversational way. Do not copy the shard verbatim. [Do Not Ask Questions] Your response should always be declarative sentences, and not questions. [Do Not Ask Questions] Your response should always be declarative sentences, and not questions. [Brevity of Response] You should favor being succint. Your answer can also have typos, improper grammar, capitalization, etc. You are simulating a real person talking to an AI, who is in a hurry. [Brevity of Response] You should favor being succint. Your answer can also have typos, improper grammar, capitalization, etc. You are simulating a real person talking to an AI, who is in a hurry. [Format] Your response should be formatted as a JSON object with the following keys: [Format] Your response should be formatted as a JSON object with the following keys: response: The response to the conversation so far. response: The response to the conversation so far. shard_id: The shard you are revealing to the system. The shard_id can be an integer, or -1 if you did not reveal any shards. shard_id: The shard you are revealing to the system. The shard_id can be an integer, or -1 if you did not reveal any shards. For example: For example: {"response": "I don’t know", "shard_id": -1} {"response": "I don’t know", "shard_id": -1} or: or: {"response": "yeah I want it to […]", "shard_id": 1} {"response": "yeah I want it to […]", "shard_id": 1}

用户模拟器 你正在模拟一个交互式大模型系统（类似 ChatGPT）的用户。 该用户天生懒惰，只用简短形式作答，只向系统提供最少的信息。你不应当主动推进。 到目前为止的对话如下： [[CONVERSATION_SO_FAR]] 以下是已经揭示的碎片： [[SHARDS_REVEALED]] 以下是尚未揭示的全部碎片： [[SHARDS_NOT_REVEALED]] 你必须针对到目前为止的对话生成一条回复。规则如下： [提供一片] 如果揭示某片内容能帮助系统更接近解答该问题，你可以在回复中向系统揭示该片。你应当选择当前最“基本”、也最相关的那一片来揭示。 [一次只给一片] 你一次最多只应揭示一片。 [完整揭示该片] 如果你揭示一片，就必须确保包含该片中的全部信息。例如，如果该片是“你的症状是早上会头疼”，你的回复不能只是“yeah I have headaches”，而必须说“yup mostly headaches in the mornings”。 [无关澄清] 如果系统问你一个与这些碎片无关的问题，或问你一个泛泛的问题（“Can you give me a hint?”），你应当给出不提供任何碎片的回答。（“I don’t know”、“Is that really important?” 等。）你不应当揭示碎片中没有的信息。 [不要重复碎片] 你不应当把同一片揭示超过一次。请仔细核对应经揭示的碎片，只有当其 shard_id 不在列表上时才揭示该片。 [改写碎片] 如果你揭示一片，应当用口语化的方式改写它。不要逐字抄写该片。 [不要提问] 你的回复应当始终是陈述句，而不是问句。 [回复简短] 你应当尽量简洁。你的回答也可以有拼写错误、不规范的语法和大小写等。你在模拟一个真实的人，正匆忙地和一个人工智能交谈。 [格式] 你的回复应格式化为带有以下键的 JSON 对象： response：对到目前为止对话的回复。 shard_id：你正在向系统揭示的那一片。shard_id 可以是一个整数；如果你没有揭示任何碎片，则为 -1。 例如： {"response": "I don’t know", "shard_id": -1} 或： {"response": "yeah I want it to […]", "shard_id": 1}

	Lost in Conversation Experiment
Model	Full	Concat	Sharded	Overall
					width 7pt						width 7pt						width 7pt	/	/
3.1-8B	27.4	64.1	82.9	13.7	63.9	07.6 width 7pt	21.2	47.7	83.0	15.7	62.6	06.5 width 7pt	21.7	25.9	45.5	13.3	37.4	03.4 width 7pt	91.6	62.5
OLMo2	18.8	54.8	56.1	17.2	80.0	- width 7pt	16.3	40.5	49.8	14.3	80.1	- width 7pt	14.4	22.4	13.8	09.0	46.3	- width 7pt	86.5	50.5
3-Haiku	44.8	85.0	83.5	29.8	73.9	11.6 width 7pt	36.3	76.5	80.2	30.1	76.1	09.2 width 7pt	31.5	31.8	55.9	18.6	47.1	01.6 width 7pt	91.6	52.4
4o-mini	75.9	89.3	94.1	35.9	88.1	14.9 width 7pt	66.7	90.7	92.2	31.2	88.0	12.5 width 7pt	50.3	40.2	52.4	19.8	58.7	07.2 width 7pt	93.0	56.2
3.3-70B	72.0	91.1	95.0	34.1	91.7	15.8 width 7pt	52.7	87.9	97.0	32.0	91.8	14.7 width 7pt	51.6	35.4	71.0	22.4	61.5	10.5 width 7pt	93.2	64.2
Phi-4	53.2	87.6	82.7	23.9	89.2	- width 7pt	48.4	79.6	76.0	28.6	90.4	- width 7pt	39.1	33.1	34.1	23.2	52.5	- width 7pt	99.0	61.7
CMD-A	72.0	91.9	98.5	27.7	94.5	24.3 width 7pt	61.6	86.1	98.4	33.2	91.9	21.3 width 7pt	44.9	33.6	72.0	27.9	66.0	04.9 width 7pt	97.3	60.4
4-Scout	73.9	92.7	98.0	35.2	96.3	13.7 width 7pt	60.3	81.5	98.3	28.2	92.9	13.7 width 7pt	46.4	27.1	69.9	26.1	67.0	12.3 width 7pt	91.0	66.1
o3	86.4	92.0	89.8	40.2	81.6	30.7 width 7pt	87.2	83.3	91.5	39.4	80.0	30.4 width 7pt	53.0	35.4	60.2	21.7	63.1	26.5 width 7pt	98.1	64.1
3.7-Sonnet	78.0	93.9	95.4	45.6	85.4	29.3 width 7pt	76.2	81.5	96.0	53.3	87.2	28.9 width 7pt	65.6	34.9	33.3	35.1	70.0	23.6 width 7pt	100.4	65.9
R1	99.4	92.1	97.0	27.0	95.5	26.1 width 7pt	97.1	89.9	97.0	36.7	92.9	24.4 width 7pt	70.9	31.5	47.5	20.0	67.3	17.2 width 7pt	103.6	60.8
4o	88.4	93.6	96.1	42.1	93.8	23.9 width 7pt	82.9	91.7	97.1	32.2	91.9	23.9 width 7pt	61.3	42.3	65.0	20.5	67.9	10.6 width 7pt	94.5	57.9
2.5-Flash	97.0	96.3	88.4	51.2	90.6	29.1 width 7pt	92.5	95.5	89.2	51.9	88.4	29.4 width 7pt	68.3	51.3	42.6	31.0	66.1	26.1 width 7pt	99.3	65.8
4.1	96.6	93.0	94.7	54.6	91.7	26.5 width 7pt	88.7	86.5	98.5	54.4	89.7	26.8 width 7pt	72.6	46.0	62.9	28.6	70.7	13.3 width 7pt	97.9	61.8
2.5-Pro	97.4	97.3	97.8	54.8	90.2	31.2 width 7pt	95.7	94.9	98.1	56.9	89.3	31.8 width 7pt	68.1	43.8	36.3	46.2	64.3	24.9 width 7pt	100.1	64.5

	Lost in Conversation 实验
模型	Full	Concat	Sharded	总体
																		/	/
3.1-8B	27.4	64.1	82.9	13.7	63.9	07.6	21.2	47.7	83.0	15.7	62.6	06.5	21.7	25.9	45.5	13.3	37.4	03.4	91.6	62.5
OLMo2	18.8	54.8	56.1	17.2	80.0	-	16.3	40.5	49.8	14.3	80.1	-	14.4	22.4	13.8	09.0	46.3	-	86.5	50.5
3-Haiku	44.8	85.0	83.5	29.8	73.9	11.6	36.3	76.5	80.2	30.1	76.1	09.2	31.5	31.8	55.9	18.6	47.1	01.6	91.6	52.4
4o-mini	75.9	89.3	94.1	35.9	88.1	14.9	66.7	90.7	92.2	31.2	88.0	12.5	50.3	40.2	52.4	19.8	58.7	07.2	93.0	56.2
3.3-70B	72.0	91.1	95.0	34.1	91.7	15.8	52.7	87.9	97.0	32.0	91.8	14.7	51.6	35.4	71.0	22.4	61.5	10.5	93.2	64.2
Phi-4	53.2	87.6	82.7	23.9	89.2	-	48.4	79.6	76.0	28.6	90.4	-	39.1	33.1	34.1	23.2	52.5	-	99.0	61.7
CMD-A	72.0	91.9	98.5	27.7	94.5	24.3	61.6	86.1	98.4	33.2	91.9	21.3	44.9	33.6	72.0	27.9	66.0	04.9	97.3	60.4
4-Scout	73.9	92.7	98.0	35.2	96.3	13.7	60.3	81.5	98.3	28.2	92.9	13.7	46.4	27.1	69.9	26.1	67.0	12.3	91.0	66.1
o3	86.4	92.0	89.8	40.2	81.6	30.7	87.2	83.3	91.5	39.4	80.0	30.4	53.0	35.4	60.2	21.7	63.1	26.5	98.1	64.1
3.7-Sonnet	78.0	93.9	95.4	45.6	85.4	29.3	76.2	81.5	96.0	53.3	87.2	28.9	65.6	34.9	33.3	35.1	70.0	23.6	100.4	65.9
R1	99.4	92.1	97.0	27.0	95.5	26.1	97.1	89.9	97.0	36.7	92.9	24.4	70.9	31.5	47.5	20.0	67.3	17.2	103.6	60.8
4o	88.4	93.6	96.1	42.1	93.8	23.9	82.9	91.7	97.1	32.2	91.9	23.9	61.3	42.3	65.0	20.5	67.9	10.6	94.5	57.9
2.5-Flash	97.0	96.3	88.4	51.2	90.6	29.1	92.5	95.5	89.2	51.9	88.4	29.4	68.3	51.3	42.6	31.0	66.1	26.1	99.3	65.8
4.1	96.6	93.0	94.7	54.6	91.7	26.5	88.7	86.5	98.5	54.4	89.7	26.8	72.6	46.0	62.9	28.6	70.7	13.3	97.9	61.8
2.5-Pro	97.4	97.3	97.8	54.8	90.2	31.2	95.7	94.9	98.1	56.9	89.3	31.8	68.1	43.8	36.3	46.2	64.3	24.9	100.1	64.5

LLMs Get Lost In Multi-Turn Conversation

大模型会在多轮对话中迷路

Abstract

摘要

1 Introduction

1 引言

2 Background and Related Work

2 背景与相关工作

3 Simulating Underspecified, Multi-Turn Conversation

3 模拟欠指定的多轮对话

3.1 Sharding Process: From Fully-Specified to Sharded Instructions

3.1 分片过程：从完整指定到分片指令

3.2 Simulating Sharded Conversations

3.2 模拟分片对话

3.3 Simulation Types

3.3 模拟类型

Fully-specified

完整指定

Sharded

分片

Concat

拼接

Recap

回顾

Snowball

滚雪球

4 Task and Metric Selection

4 任务与指标选择

4.1 Task Selection

4.1 任务选择

Code

代码

Database

数据库

Actions

动作

Math

数学

Data-to-text

数据到文本

Summary

摘要

4.2 Metric Selection

4.2 指标选择

5 Simulation Scale and Parameters

5 模拟规模与参数

6 Results

6 结果

6.1 Average Performance Findings

6.1 平均表现方面的发现

6.2 Aptitude vs. Reliability Analysis

6.2 能力上限与可靠性分析

6.3 Gradual Sharding Experiment

6.3 渐进分片实验

7 Implications

7 影响

7.1 Implications for System and Agent Builders

7.1 对系统与智能体构建者的影响

7.2 Implications for LLM Builders

7.2 对大模型构建者的影响

7.3 Implications for NLP Practitioners

7.3 对自然语言处理实践者的影响

7.4 Implications for Users of Conversational Systems

7.4 对对话系统用户的影响

If time allows, try again.

如果时间允许，再试一次。

Consolidate before retrying.

重试前先把要求收拢到一处。

8 Conclusion

8 结论

9 Limitations

9 局限

References

参考文献

Appendices

附录

Appendix A Related work on Underspecification

附录 A 关于欠指定的相关工作

Appendix B Precise Definition of Sharded Instructions

附录 B 分片指令的精确定义