Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interaction

Audio MultiChallenge: 面向自然人类交互的口语对话系统多轮评测

Advait Gosai ∗ Affiliation:  Scale AI    Tyler Vuong ∗ Affiliation:  Scale AI    Utkarsh Tyagi Affiliation:  Scale AI    Steven Li Affiliation:  Scale AI    Wenjia You Affiliation:  Scale AI    Miheer Bavare Affiliation:  Scale AI    Arda Uçar Affiliation:  Scale AI    Zhongwang Fang Affiliation:  Scale AI    Brian Jang Affiliation:  Scale AI    Bing Liu Affiliation:  Scale AI    Yunzhong He Affiliation:  Scale AI

Abstract

摘要

End-to-end (E2E) spoken dialogue systems are increasingly replacing cascaded pipelines for voice-based human-AI interaction, processing raw audio directly without intermediate transcription. Existing benchmarks primarily evaluate these models on synthetic speech and single-turn tasks, leaving realistic multi-turn conversational ability underexplored. We introduce Audio MultiChallenge, an open-source benchmark to evaluate E2E spoken dialogue systems under natural multi-turn interaction patterns. Building on the text-based MultiChallenge framework, which evaluates Inference Memory, Instruction Retention, and Self Coherence, we introduce a new axis Voice Editing that tests robustness to mid-utterance speech repairs and backtracking. We further augment each axis to the audio modality, such as introducing Audio-Cue challenges for Inference Memory that require recalling ambient sounds and paralinguistic signals beyond semantic content. We curate 452 conversations from 47 speakers with 1,712 instance-specific rubrics through a hybrid audio-native agentic and human-in-the-loop pipeline that exposes model failures at scale while preserving natural disfluencies found in unscripted human speech. Our evaluation of proprietary and open-source models reveals that even frontier models struggle on our benchmark, with Gemini 3 Pro Preview (Thinking), our highest-performing model achieving a 54.65% pass rate. Error analysis shows that models fail most often on our new axes and that Self Coherence degrades with longer audio context. These failures reflect difficulty of tracking edits, audio cues, and long-range context in natural spoken dialogue. Audio MultiChallenge provides a reproducible testbed to quantify them and drive improvements in audio-native multi-turn interaction capability.

端到端（E2E）口语对话系统正越来越多地取代级联流水线，直接处理原始音频，不必再经过中间转写，用于语音形式的人机交互。现有基准主要在合成语音和单轮任务上评测这些模型，对真实的多轮对话能力关注不够。我们提出开源基准 Audio MultiChallenge，用于在自然的多轮交互模式下降评测端到端口语对话系统。它建立在文本版 MultiChallenge 框架之上——该框架评测推理记忆（Inference Memory）、指令保持（Instruction Retention）和自我连贯（Self Coherence）——并新增一条轴语音编辑（Voice Editing），用来检验模型对句中自我修正和撤回的稳健程度。我们还把各条轴扩展到音频模态，例如为推理记忆引入音频线索（Audio-Cue）挑战，要求模型记住环境声和副语言信号，而不能只靠语义内容。我们通过一套混合的、面向音频的智能体与人工在环流水线，从 47 位说话人中整理出 452 段对话，并配有 1,712 条实例级评分细则：这套流程能大规模暴露模型失败，同时保留无脚本真人语音中的自然不流畅现象。对专有和开源模型的评测表明，即便前沿模型在本基准上也相当吃力；表现最好的 Gemini 3 Pro Preview（Thinking）通过率也只有 54.65%。错误分析显示，模型在我们新设的轴上失败最多，而且自我连贯会随音频上下文变长而下降。这些失败反映出，在自然口语对话中追踪编辑、音频线索和长程上下文仍然困难。Audio MultiChallenge 提供了一个可复现的试验台，用来量化这些问题，并推动音频原生多轮交互能力的改进。

advait.gosai@scale.com   huggingface.co/ScaleAI/audiomc   scale.com/leaderboard/audiomc [^*]

Figure 1: Samples from Audio MultiChallenge. Highlighted text shows key interaction patterns per axis.

图 1：Audio MultiChallenge 中的样本。高亮文字标出各条轴上的关键交互模式。

1 Introduction

1 引言

Spoken dialogue systems [26] are evolving from cascaded pipelines comprising Automatic Speech Recognition (ASR) [35], text-based Large Language Models (LLMs) [8], and Text-to-Speech (TTS) systems [41] to native end-to-end (E2E) architectures that map input speech directly to conversational responses without an intermediate transcription step [34, 5, 45]. Speech-to-speech (S2S) models and broader audio language models (Audio LMs) process raw or minimally processed acoustic features and generate spoken or textual outputs, often operating over shared latent representations or learned discrete speech units rather than relying solely on text tokens [34, 47, 7, 14, 13]. This integration can reduce response latency and mitigate error propagation, while better preserving paralinguistic and acoustic information that is often lost in cascaded systems [12, 25].

口语对话系统 [26] 正在从由自动语音识别（ASR）[35]、文本大语言模型（LLM）[8] 和文本转语音（TTS）系统 [41] 组成的级联流水线，转向原生的端到端（E2E）架构：后者把输入语音直接映射为对话回复，不再经过中间转写步骤 [34, 5, 45]。语音到语音（S2S）模型以及更广泛的音频语言模型（Audio LM）处理原始或仅做少量预处理的声学特征，并生成语音或文本输出，往往在共享的潜在表示或学到的离散语音单元上运作，而不是只依赖文本标记 [34, 47, 7, 14, 13]。这种一体化可以降低响应时延、减轻错误传递，也能更好地保留级联系统中常常丢失的副语言和声学信息 [12, 25]。

As E2E systems move toward real-world user-facing interaction, their evaluation must reflect the demands of extended, spontaneous dialogue rather than isolated prompt–response pairs or short, synthetic TTS inputs common in existing benchmarks [9, 40, 22]. Evidence from text-only LLMs shows that strong single-turn performance does not guarantee robust multi-turn behavior [15, 28, 6]. These weaknesses are amplified in the audio modality. Audio is temporally continuous and often represented as high-frame-rate token streams, which rapidly consume context and make long-horizon state tracking harder than in text [7, 12]. Consequently, while existing Audio LM benchmarks suggest strong performance on short, clean, or synthetic inputs, robustness in realistic multi-turn settings remains understudied. Even recent multi-turn efforts are limited to short dialogues [18], leaving unclear whether systems can handle key interaction patterns such as maintaining self-consistency, following user constraints, leveraging audio cues over long contexts, and recovering from user repairs like mid-utterance self-corrections and barge-ins.

随着端到端系统走向面向真实用户的交互，评测也必须反映拉长的、自发的对话所提出的要求，而不能只看现有基准中常见的孤立问答对，或短的、合成的 TTS 输入 [9, 40, 22]。来自纯文本 LLM 的证据已经表明，单轮表现强，并不保证多轮行为同样稳健 [15, 28, 6]。这些弱点在音频模态上会被放大。音频在时间上是连续的，又常常被表示成高帧率的标记流，会很快占满上下文，使长程状态跟踪比文本更难 [7, 12]。因此，尽管现有 Audio LM 基准在短、干净或合成的输入上显得很强，真实多轮设定下的稳健性仍缺少研究。即便最近的多轮工作也往往限于较短对话 [18]，系统能否处理一些关键交互模式仍不清楚，例如保持自我一致、遵守用户约束、在长上下文中利用音频线索，以及从用户的句中自我修正和插话中恢复。

To rigorously evaluate these dimensions, we open-source Audio MultiChallenge, an extension of the text-based MultiChallenge evaluation framework [15] to the audio modality through three key contributions:

为了严格评测这些方面，我们开源 Audio MultiChallenge。它把文本版 MultiChallenge 评测框架 [15] 扩展到音频模态，主要有三项贡献：

• • Audio-native axes and tasks. We introduce a novel Voice Editing axis that evaluates robustness to mid-utterance speech repairs and backtracking, capturing natural human behaviors such as self-corrections that are common in spontaneous dialogue. We also add Audio-Cue Inference Memory tasks that require recalling ambient sounds or paralinguistic signals from previous turns, rather than only semantic content, and adapt the remaining Instruction Retention and Self Coherence axes to better reflect the complexities of voice interactions.

• • 面向音频的轴与任务。 我们提出新的语音编辑轴，评测模型对句中语音修正和撤回的稳健程度，捕捉自发对话里常见的自我纠正等自然行为。我们还加入音频线索推理记忆任务，要求回想先前轮次中的环境声或副语言信号，而不能只靠语义内容；并将其余的指令保持和自我连贯轴加以调整，以便更好地反映语音交互的复杂性。

• • Hybrid data generation for scalability and diversity. We introduce a data-generation pipeline that combines multi-agent synthetic component with a human-in-the-loop process to produce our final multi-turn conversation prompts. Rather than providing human contributors with verbatim scripts, we provide high-level blueprints and interaction goals derived from a scalable agentic loop designed to elicit diverse failure modes. These blueprints encourage improvisational interactions that surface realistic failures, making our dataset sufficiently adversarial while remaining entirely unscripted and preserving paralinguistics and natural human disfluencies.

• • 兼顾规模与多样性的混合数据生成。 我们提出一套数据生成流水线，把多智能体合成部分与人工在环过程结合起来，得到最终的多轮对话提示。我们并不给人类贡献者逐字脚本，而是提供由可扩展的智能体循环导出的高层蓝图和交互目标，用于诱发多样的失败模式。这些蓝图鼓励即兴交互，把真实失败暴露出来，使数据集既有足够的对抗性，又完全不按脚本朗读，从而保留副语言特征和自然的人类不流畅现象。

• • Fine-grained rubric-based evaluation. We replace coarse binary success metrics with 1,712 atomic, instance-specific rubrics across the dataset that decompose each ideal response into critical sub-requirements, and our LLM-as-a-judge method using these rubrics achieves 93% agreement rate with human evaluators. While satisfying all corresponding rubrics is required to pass a task, individual rubrics provide high-resolution attribution of common model failures.

• • 细粒度的细则评测。 我们不再使用粗糙的二元成败指标，而是在整个数据集上使用 1,712 条原子的、针对实例的评分细则，把每个理想回复拆成关键子要求；用这些细则做大模型评委，与人类评估者的一致率为 93%。要通过一项任务，必须满足该任务对应的全部细则，但单条细则也能高分辨率地归因常见的模型失败。

Audio MultiChallenge comprises 452 multi-turn conversations from 47 unique speakers, summing up to 15 hours of user audio recorded without post-processing to retain ambient acoustic variability.

Audio MultiChallenge 包含来自 47 位不同说话人的 452 段多轮对话，用户音频合计约 15 小时；录音不做后期处理，以保留环境声学上的变化。

Our analysis of Audio MultiChallenge reveals substantial gaps in current E2E capabilities across frontier models. Even the highest-performing model, Gemini 3 Pro Preview (Thinking), achieves only 54.65% pass rate, with other model families, especially open-source ones lagging considerably behind. We find that Self Coherence degrades consistently as cumulative user audio duration increases, dropping from 33.3% on shorter tasks to 20.0% on tasks with 3–5 minutes of audio. Models also struggle to recall paralinguistic and ambient signals from previous turns, scoring 36.5% lower on Audio-Cue Inference Memory than on semantic memory tasks. Voice Editing proves to be the most challenging axis, where behaviors such as mid-utterance self-corrections and prior-turn edits frequently trigger failures that are largely absent in cleaner text baselines. Together, our results suggest the need for audio-native training and evaluation that target realistic multi-turn spoken interaction.

我们对 Audio MultiChallenge 的分析表明，当前端到端能力在前沿模型之间仍有明显差距。即便表现最好的 Gemini 3 Pro Preview（Thinking），通过率也只有 54.65%；其他模型族，尤其是开源模型，落后更多。我们发现，自我连贯会随着用户音频累计时长增加而持续下降：较短任务上为 33.3%，在有 3–5 分钟音频的任务上降到 20.0%。模型也难以回想先前轮次中的副语言和环境信号，在音频线索推理记忆上的得分比语义记忆任务平均低 36.5%。语音编辑是最难的一条轴：句中自我修正和跨轮编辑等行为，常常触发在更干净的文本基线里几乎看不到的失败。综合来看，这些结果说明，有必要开展面向真实多轮口语交互的音频原生训练与评测。

2 Audio MultiChallenge

Conversations	452
Inference Memory	132
Instruction Retention	120
Self Coherence	117
Voice Editing	83
Turn Count	3-8
Rubrics	1,712
Speakers	47
User Audio Duration	14.99 h
Sampling Rate	48 kHz
Language	English

Table 1: Dataset Statistics

对话数	452
推理记忆	132
指令保持	120
自我连贯	117
语音编辑	83
轮次数	3-8
评分细则	1,712
说话人	47
用户音频时长	14.99 小时
采样率	48 kHz
语言	English

表 1：数据集统计

In Audio MultiChallenge, we focus on four core axes designed to test the multi-turn capabilities E2E spoken dialogue systems: Inference Memory, Instruction Retention, Self Coherence, and Voice Editing. The fourth axis, Voice Editing, is a novel addition specific to the speech domain. We introduce this axis to address a fundamental difference between modalities. Unlike text inputs where errors are typically backspaced prior to submission, speech is a continuous stream where the editing process is audible. This necessitates that models dynamically filter retracted information to isolate the user’s final intent. Collectively, these categories represent practical, real-world conversational hurdles that remain challenging for current frontier models and have historically been evaluated primarily on single-turn tasks. For each category, we release a set of human-recorded, non-scripted test examples curated according to the specific axis definition.

在 Audio MultiChallenge 中，我们关注四条核心轴，用来检验端到端口语对话系统的多轮能力：推理记忆、指令保持、自我连贯和语音编辑。第四条轴语音编辑是专门针对语音领域的新增内容。我们引入这条轴，是为了对应模态之间的一个根本差别。文本输入里，错误通常在提交前就被退格改掉；语音则是连续的流，编辑过程本身是听得见的。因此模型必须动态过滤被撤回的信息，才能分离出用户最终的意图。合在一起，这些类别代表了当前前沿模型仍然感到困难、而以往又主要在单轮任务上评测的、贴近真实对话的障碍。对每个类别，我们都发布一组按该轴定义整理的、人类录制且非脚本化的测试样本。

We explicitly distinguish Audio MultiChallenge from text-based MultiChallenge to address the unique requirements of spoken interaction. Although MultiChallenge offers a strong foundation, simply applying TTS to the existing text dataset is insufficient. For instance, many text-based interactions such as writing code or formatting text do not naturally translate to voice. Audio MultiChallenge instead captures the inherent complexity of spontaneous human speech by curating real recordings, which unlike synthetic or scripted data, include natural disfluencies, backtracking, and third-party interruptions or cross-talk. Our approach allows us to evaluate model robustness in terms of semantic understanding as well as against real-world acoustics, varying accents, and complex paralinguistic cues which text-derived benchmarks fail to capture. We define our axes in detail below.

我们明确把 Audio MultiChallenge 与文本版 MultiChallenge 区分开来，以应对口语交互的特殊要求。尽管 MultiChallenge 提供了很好的基础，只对已有文本数据做 TTS 是不够的。例如，许多文本交互（如写代码或排版）并不能自然地转到语音。Audio MultiChallenge 转而通过整理真实录音来捕捉自发人类语音固有的复杂性：与合成或照稿朗读的数据不同，这些录音包含自然的不流畅、撤回，以及第三方插话或串音。这样，我们既能评测语义理解上的稳健性，也能评测模型对真实声学、不同口音和复杂副语言线索的稳健性，而这些是从文本派生的基准难以捕捉的。下面我们详细定义各条轴。

2.1 Axes

2.1 各条轴

Inference Memory

Inference Memory evaluates the model’s capacity to recall and synthesize scattered user-specified details from previous turns that are implicitly required to satisfy the user’s final request. Unlike direct queries (e.g., "What did I say earlier?"), these tasks demand that the model proactively reallocate attention to prior context to inform its reasoning. For example, a standard Semantic test might require remembering a stated nut allergy when generating a dessert recipe. Audio MultiChallenge extends this definition to include Audio-Cue Gated Inference Memory. In these scenarios, the critical context to be retained is not explicitly spoken, but is embedded in the acoustic environment or paralinguistic signals. An Audio-Cue gated test might feature the background sound of heavy rain in an initial turn, implicitly constraining a later request for an outfit recommendation. This evaluates the model’s ability to retain and integrate both semantic content and acoustic cues over a multi-turn conversation.

推理记忆

推理记忆评测模型能否回想并综合先前轮次中、为满足用户最终请求所隐含需要的、分散的用户细节。与直接提问（例如“我刚才说了什么？”）不同，这类任务要求模型主动把注意力重新分配到先前上下文，以支撑当前推理。例如，一个标准的语义测试可能要求在生成甜点食谱时记住用户提过的坚果过敏。Audio MultiChallenge 把这一定义扩展到包含音频线索门控的推理记忆。在这些情形里，需要保留的关键上下文并没有被明确说出来，而是嵌在声学环境或副语言信号中。一个音频线索门控测试可能在最初一轮里出现大雨的背景声，从而隐含地约束后续的穿衣建议。这评测的是模型在多轮对话中同时保持并整合语义内容和声学线索的能力。

Instruction Retention

Instruction Retention measures a model’s ability to follow explicit user instructions consistently across an entire multi-turn conversation. In Audio MultiChallenge, we broaden the instruction space beyond simple stylistic or structural constraints, to stacked and conditional rule sets. These include multi-constraint directives and trigger-based behaviors, for example changing the response format only after a specific phrase is spoken. We also broaden conversation topics here to include more realistic voice applications such as debate and roleplay. This axis evaluates whether an Audio LM can maintain and correctly apply precise, evolving instructions over long dialogue histories, a failure mode that is frequently observed in speech interactions [30, 31].

指令保持

指令保持衡量模型能否在整段多轮对话中一贯地遵守用户给出的显式指令。在 Audio MultiChallenge 中，我们把指令空间从简单的风格或结构约束，拓宽到叠放和条件规则集。其中包括多约束指令和触发式行为，例如只有在听到某个特定短语之后才改变回复格式。我们还把对话主题拓宽到更贴近真实语音应用的辩论和角色扮演。这条轴评测的是 Audio LM 能否在很长的对话历史上维持并正确应用精确、不断变化的指令——这是语音交互中经常观察到的失败模式 [30, 31]。

Figure 2: Topic Distribution. Detailed breakdown in Appendix A.7

图 2：主题分布。详细分解见附录 A.7。

Self Coherence

Self Coherence evaluates the model’s ability to maintain internal consistency regarding factual assertions, opinions, and established personas throughout a dialogue. A common failure mode in frontier models is contradicting its prior correct responses to align with a user who questions or challenges them. For Audio MultiChallenge, we refine this category by excluding contrived, adversarial red-teaming prompts in favor of realistic conversational scenarios. While adaptive reasoning is desirable in light of new evidence, we specifically target unwarranted contradictions where the model invalidates its own previous statements without justification.

自我连贯

自我连贯评测模型能否在整段对话中，就事实断言、观点和已确立的人设保持内部一致。前沿模型一个常见的失败模式是：当用户质疑或挑战它时，它会推翻自己先前正确的回答去迎合用户。对 Audio MultiChallenge，我们收紧了这一类别，去掉刻意的对抗性红队提示，改用更贴近真实的对话场景。面对新证据时做适应性推理是可取的，但我们专门针对那些没有正当理由就否定自己先前陈述的、站不住脚的自相矛盾。

Voice Editing

Voice Editing is a novel axis introduced specifically to mimic speech interactions with a personal assistant. We define this axis as the model’s ability to recognize and apply immediate, spontaneous speech repairs, such as mid-utterance self-corrections or implemented planned edits that span multiple turns. This axis targets a unique challenge for audio-native models. While LLMs are trained on refined prompts where errors are backspaced or redrafted prior to submission, speech is a continuous stream where the editing process is audible. Consequently, models must dynamically filter out retracted content (e.g., "Let’s make it four yuccas, hmm no six.") and resolve implicit contradictions that arise within consecutive turns. Evaluation here focuses on the model’s robustness to these natural disfluencies and its capacity to discern the user’s final intent amidst the messiness of spoken interaction.

语音编辑

语音编辑是专门为模拟与个人助手的语音交互而新设的轴。我们把它定义为：模型识别并落实即时、自发的语音修正的能力，例如句中的自我纠正，或跨多轮实施的计划性修改。这条轴针对音频原生模型的一个独特难点。LLM 训练时看到的往往是经过润色的提示，错误在提交前就被退格或重写；语音则是连续的流，编辑过程听得见。因此，模型必须动态过滤被撤回的内容（例如“那就四个丝兰，嗯不，六个。”），并化解相邻轮次里出现的隐含矛盾。这里的评测关注模型对这些自然不流畅现象的稳健程度，以及它能否在杂乱的口语中分辨用户的最终意图。

2.2 Dataset Curation

2.2 数据整理

Producing naturalistic, diverse, and challenging test examples for Audio MultiChallenge is resource intensive. We optimize our dataset curation using a two-stage hybrid approach that combines (i) a multi-agent pipeline to synthetically explore Audio LM failure modes at scale with a (ii) human-in-the-loop stage that translates these failure blueprints into realistic conversations and corresponding evaluation rubrics. An overview is shown in Figure 3, and we describe each step below.

为 Audio MultiChallenge 制作自然、多样且有难度的测试样本耗费资源。我们用两阶段混合方法来优化数据整理：其一是多智能体流水线，用于大规模、合成地探索 Audio LM 的失败模式；其二是人工在环阶段，把这些失败蓝图改写成真实对话和相应的评测细则。流程概览见图 3，各步骤如下。

Figure 3: Audio MultiChallenge dataset generation process that utilizes an agentic blueprint generation followed by human-in-the-loop multi-turn conversation generation and quality check. Conversation topic seeds, generated human-friendly blueprints and data samples are provided in the Appendix.

图 3：Audio MultiChallenge 的数据生成过程：先由智能体生成蓝图，再经人工在环进行多轮对话采集和质量检查。对话主题种子、面向人类的蓝图以及数据样本见附录。

Generating Synthetic Audio LM Failures

We implement an automated multi-agent framework to generate synthetic conversations that elicit failures along a targeted axis. This approach follows the synthetic generation methodology of MultiChallenge, adapted to the audio modality.

生成合成的 Audio LM 失败

我们实现了一个自动化的多智能体框架，用于生成沿着指定轴诱发失败的合成对话。这一做法沿用 MultiChallenge 的合成生成方法，并适配到音频模态。

As shown in Figure 3, the process begins by sampling a target Evaluation Axis $A$ (e.g. Inference Memory) along with a seed topic $T$ (e.g. DietaryRestrictions) and persona $P$ (e.g. Content creator) from curated taxonomies. A Planner Agent (text LLM) then devises an initial conversation strategy aimed at triggering a specific failure mode while adhering to the sampled topic. Based on this strategy, a Tester Agent (text LLM) generates initial user prompt text, which is converted to speech using TTS to simulate audio input. This audio is then fed to the Target Agent (Audio LM). The Planner Agent evaluates the Target Agent’s output to detect failure. If the model succeeds, the Planner updates the strategy to introduce higher complexity in the subsequent turn. This cycle repeats until a failure is detected or a maximum turn count is reached. Once a failure is detected, the Planner Agent analyzes the history $H$ and extracts the core logic or trick that caused the model to fail along the axis. This is synthesized into a concise Human-friendly Blueprint $B_{final}$, serving as a high-level strategic guide for human contributors. In this study, we use gpt-4o-mini-tts by OpenAI as the TTS system, randomly sample between GPT 4o Audio Preview and Gemini 2.5 Pro for the Target Agent, and utilize the reasoning capability of OpenAI o3 as the Planner that guides GPT 4o as the User Agent. The full algorithm is provided in Appendix A.6.

如图 3 所示，过程首先从整理好的分类体系中采样目标评测轴 $A$（例如推理记忆）、种子主题 $T$（例如 DietaryRestrictions）和人设 $P$（例如内容创作者）。规划智能体（文本 LLM）随后拟定初始对话策略，目标是在遵守所采样主题的同时触发特定失败模式。基于该策略，测试智能体（文本 LLM）生成最初的用户提示文本，再用 TTS 转成语音，以模拟音频输入。这段音频被送给目标智能体（Audio LM）。规划智能体再评估目标智能体的输出，以检测是否失败。若模型成功了，规划器就更新策略，在下一轮引入更高复杂度。这一循环重复进行，直到检测到失败，或达到最大轮次数。一旦检测到失败，规划智能体分析历史 $H$，抽出导致模型在该轴上失败的核心逻辑或“机关”，并把它浓缩成一份简明的、面向人类的蓝图 $B_{final}$，作为给人类贡献者的高层策略指南。本研究中，TTS 系统使用 OpenAI 的 gpt-4o-mini-tts，目标智能体在 GPT 4o Audio Preview 与 Gemini 2.5 Pro 之间随机抽取，规划器则利用 OpenAI o3 的推理能力来指导作为用户智能体的 GPT 4o。完整算法见附录 A.6。

Human-In-The-Loop Conversation Generation

Using axis-specific blueprints as seeds, we collect audio recordings from human contributors. Each contributor attempts to induce a verifiable failure within 3 to 8 turns. Contributors receive the blueprint and a detailed definition of the target axis, but the blueprint is only strategic guidance. They are instructed to rewrite the plan in their own words, introduce new constraints, or pivot topics as needed to expose model weaknesses. This encourages spontaneous, in-the-wild interaction, including pauses, hesitations, rambling, and mid-utterance self-corrections. On average, 65% of attempted conversations leveraged a synthetic blueprint, highlighting their effectiveness. We provide blueprint samples in Appendix A.2.

人工在环的对话生成

以针对各轴的蓝图为种子，我们向人类贡献者采集录音。每位贡献者尝试在 3 到 8 轮内诱发一次可核验的失败。贡献者会收到蓝图以及目标轴的详细定义，但蓝图只是策略指导。我们要求他们用自己的话改写计划，按需要引入新约束或转换主题，以暴露模型弱点。这会鼓励自发的、接近真实场景的交互，包括停顿、犹豫、东拉西扯，以及句中的自我修正。平均而言，65% 的尝试对话用到了合成蓝图，说明它们是有效的。蓝图样本见附录 A.2。

To collect natural speech data, contributors interact with an Audio LM (either GPT 4o Audio Preview and Gemini 2.5 Pro) in a turn-based manner until a failure is manually detected. For each turn, the contributor records an audio prompt, which is processed by the Target Audio LM. The model output is returned as text for immediate review. After each turn, contributors check that the interaction remains challenging and coherent, avoids premature failures unrelated to the assigned axis, and continues the conversation naturally. If these conditions are met, the contributor proceeds to the next turn. Once an objective failure is triggered under the assigned axis, the conversation ends.

为采集自然语音数据，贡献者与 Audio LM（GPT 4o Audio Preview 或 Gemini 2.5 Pro）按轮交互，直到人工检测到一次失败。每一轮，贡献者录制一段音频提示，由目标 Audio LM 处理，模型输出以文本返回以便即时检查。每轮之后，贡献者要确认交互仍然有挑战性且连贯，避免出现与指定轴无关的过早失败，并让对话自然进行。若这些条件满足，贡献者进入下一轮。一旦在指定轴上触发了客观失败，对话即结束。

Figure 4: We utilize a fixed-context evaluation protocol where each model receives identical conversation history and is evaluated on its response to the final user turn. An LLM-as-a-judge is subsequently used to evaluate the text stream of the model response against our human-written rubrics.

图 4：我们采用固定上下文评测协议：每个模型收到相同的对话历史，并仅就其针对最后一轮用户输入的回复接受评测。随后用大模型评委，对照我们人工撰写的细则，评估模型回复的文本流。

Human-Authored Rubrics for Model Evaluation

Once the conversation is marked as a failure along the specific axis, contributors formulate a set of atomic, binary criteria specifically designed to evaluate the model response of the final turn where the failure occurred. Rather than creating a single, unstructured ideal rubric, we require contributors to decompose the ideal response into a set of multiple discrete rubrics. To ensure these rubrics strictly measure the targeted axes, we instruct contributors to generate criteria that correspond exclusively to both the assigned axis and the final user turn. General conversational requirements related to only the final user turn or generic axis definitions related to the assigned axis only are explicitly excluded. By breaking down the ideal response into self-contained, binary statements, our approach allows an LLM-as-a-judge to verify individual rubric criterion with higher precision than when scoring against a single reference text.

用于模型评测的人工撰写细则

一旦对话被标为沿特定轴失败，贡献者会制定一组原子的、二元的标准，专门用来评测失败发生的那一轮、也就是最后一轮的模型回复。我们并不要求写一条结构松散的“理想细则”，而是要求贡献者把理想回复拆成多条离散细则。为确保这些细则严格衡量目标轴，我们要求标准必须同时只对应指定轴和最后一轮用户输入。仅与最后一轮用户输入有关的一般性对话要求，或仅与指定轴的一般定义有关的要求，都被明确排除。通过把理想回复拆成自含的二元陈述，我们的方法可以让大模型评委逐条核验细则，精度高于对照单一参考文本打分。

Quality Control

We construct Audio MultiChallenge with highly trained core contributors and a three-stage review process. Each task is audited by specialized reviewers, an independent quality control team, and our internal research team. We evaluate quality along conversation naturalness, failure validity, prompt diversity, audio quality, and rubric quality, as summarized in Figure 3.

质量控制

Audio MultiChallenge 由训练有素的核心贡献者构建，并经过三阶段审阅。每项任务都由专门审阅者、独立的质量控制团队以及我们的内部研究团队审核。我们从对话自然度、失败有效性、提示多样性、音频质量和细则质量几方面评估质量，如图 3 所概括。

3 Evaluation

3 评测

Fixed-Context Evaluation Protocol

To ensure a standardized evaluation, we adopt a fixed-context protocol, as depicted in Figure 4. Specifically, for each test example, we pre-seed the full conversation history (both user and assistant turns) and evaluate only the model’s response to the final turn using our LLM-as-a-judge approach. User turns are provided in context as base64-encoded audio files, while assistant turns are provided as raw text. This contrasts with sequential evaluation methods used in prior work, where a model generates responses turn-by-turn to pre-recorded user prompts [18]. We avoid the sequential approach because it can suffer from conversation drift: if a model’s response in the first turn diverges even slightly from the reference transcript, subsequent pre-recorded user follow-ups and their associated rubrics may become semantically invalid. Our approach encapsulates more natural back-and-forth interaction patterns and ensures that every model receives an identical context state, enabling a fair, reproducible comparison.

固定上下文评测协议

为了保证评测标准一致，我们采用图 4 所示的固定上下文协议。具体而言，对每个测试样本，我们预先置入完整的对话历史（用户轮次和助手轮次都包括），只用大模型评委方法评测模型对最后一轮的回复。用户轮次以 base64 编码的音频文件放入上下文，助手轮次则以原始文本提供。这与先前工作中的顺序评测不同：那里模型要一轮轮地对预先录好的用户提示生成回复 [18]。我们避免顺序做法，是因为它会出现对话漂移：如果模型第一轮的回复与参考转写哪怕只偏一点，后面预先录好的用户追问及其对应细则在语义上就可能失效。我们的做法既能容纳更自然的来回交互模式，又保证每个模型面对同一上下文状态，从而可以进行公平、可复现的比较。

Evaluation Metrics

We utilize the Average Pass Rate (APR) metric [20] to evaluate model performance on Audio MultiChallenge. Concretely, a model’s response is considered successful if and only if it satisfies all corresponding rubrics; otherwise, it is marked as a failure. We report the Overall APR as the percentage of tasks successfully completed across the dataset. This methodology aligns with MultiChallenge [15], where a single, binary weighted target question was defined to grade each tasks. Instead of restricting our data to one rubric criteria, we allow for multiple binary weighted atomic rubrics for increased LLM judge reliability, and equivalently calculate model performance as their APR. Additionally, we report an Average Rubric Score (ARS) [20] or a rubric compliance [37] score for each model which represents the average percentage of satisfied rubrics per task. We define our evaluation metrics formally in Appendix A.4.

评测指标

我们用平均通过率（Average Pass Rate，APR）[20] 来衡量模型在 Audio MultiChallenge 上的表现。具体来说，模型的回复当且仅当满足全部对应细则时才算成功，否则记为失败。我们把总体 APR 报告为整个数据集中成功完成的任务百分比。这一方法与 MultiChallenge [15] 一致，后者为每项任务定义一个二元加权的目标问题来打分。我们不把数据限制为只有一条细则，而是允许每项任务有多条二元加权的原子细则，以提高大模型评委的可靠性，并同样用 APR 计算模型表现。此外，我们还报告平均细则得分（Average Rubric Score，ARS）[20]，也就是每个模型的细则符合率 [37]，表示每项任务中被满足细则的平均百分比。指标的形式化定义见附录 A.4。

Model	Output Modality	Overall	Per-Axis APR (%)
APR (%)	ARS (%)	IM	IR	SC	VE
Gemini 3 Pro Preview (Thinking)	Text	54.65	82.54	57.58	65.283	40.96	49.57
Gemini 2.5 Pro (Thinking) [10]	Text	46.90	78.91	44.70	56.67	40.96	43.59
Gemini 2.5 Flash (Thinking)	Text	40.04	74.59	30.30	37.50	54.22	43.59
Gemini 2.5 Flash	Text	26.11	65.42	19.70	29.17	31.33	26.50
GPT Realtime[^1]	Text	23.45	65.95	21.97	26.67	22.89	22.22
GPT Realtime	Audio	20.35	63.03	19.70	20.00	26.51	17.09
GPT 4o Audio Preview [24]	Text	25.44	67.58	23.48	27.50	31.33	21.37
GPT 4o Audio Preview	Audio	23.23	64.25	21.21	30.83	24.10	17.09
GPT 4o Mini Audio Preview	Text	14.82	57.01	15.15	16.67	20.48	8.55
GPT 4o Mini Audio Preview	Audio	13.05	54.91	9.85	12.50	21.69	11.11
Qwen 3 Omni [45]	Audio	24.34	63.43	20.45	27.50	30.12	21.37
Qwen 2.5 Omni [44]	Text	11.95	45.39	12.88	13.33	18.07	5.13
Voxtral Small 24B [29]	Text	26.33	66.59	20.45	25.00	27.71	33.33
Mimo Audio 7B Instruct (Thinking) [43] [^1]	Text	19.69	55.78	19.70	22.50	27.71	11.11
Mimo Audio 7B Instruct	Text	18.58	53.86	17.42	20.00	27.71	11.97
Phi 4 MM Instruct [1]	Text	15.49	46.96	16.67	16.67	28.92	3.42
Gemma 3n E4B IT [39]	Text	15.49	43.11	14.39	22.50	26.51	1.71
Kimi Audio 7B Instruct [16]	Text	13.72	48.36	14.39	16.67	20.48	5.13
Kimi Audio 7B Instruct	Audio	10.40	22.08	15.91	6.67	13.25	5.98
LFM2 Audio 1.5B [3]	Audio	9.29	19.10	15.15	7.50	15.66	0.00

Table 2: Overall and Per-Axis scores on Audio MultiChallenge per model and output modality. APR serves as final scores. IM: Inference Memory, IR: Instruction Retention, SC: Self Coherence, VE: Voice Editing. Green highlights indicate best performance with output modality as Text; blue highlights indicate best Audio output performance.

模型	输出模态	总体	分轴 APR（%）
APR（%）	ARS（%）	IM	IR	SC	VE
Gemini 3 Pro Preview (Thinking)	Text	54.65	82.54	57.58	65.283	40.96	49.57
Gemini 2.5 Pro (Thinking) [10]	Text	46.90	78.91	44.70	56.67	40.96	43.59
Gemini 2.5 Flash (Thinking)	Text	40.04	74.59	30.30	37.50	54.22	43.59
Gemini 2.5 Flash	Text	26.11	65.42	19.70	29.17	31.33	26.50
GPT Realtime[^1]	Text	23.45	65.95	21.97	26.67	22.89	22.22
GPT Realtime	Audio	20.35	63.03	19.70	20.00	26.51	17.09
GPT 4o Audio Preview [24]	Text	25.44	67.58	23.48	27.50	31.33	21.37
GPT 4o Audio Preview	Audio	23.23	64.25	21.21	30.83	24.10	17.09
GPT 4o Mini Audio Preview	Text	14.82	57.01	15.15	16.67	20.48	8.55
GPT 4o Mini Audio Preview	Audio	13.05	54.91	9.85	12.50	21.69	11.11
Qwen 3 Omni [45]	Audio	24.34	63.43	20.45	27.50	30.12	21.37
Qwen 2.5 Omni [44]	Text	11.95	45.39	12.88	13.33	18.07	5.13
Voxtral Small 24B [29]	Text	26.33	66.59	20.45	25.00	27.71	33.33
Mimo Audio 7B Instruct (Thinking) [43] [^1]	Text	19.69	55.78	19.70	22.50	27.71	11.11
Mimo Audio 7B Instruct	Text	18.58	53.86	17.42	20.00	27.71	11.97
Phi 4 MM Instruct [1]	Text	15.49	46.96	16.67	16.67	28.92	3.42
Gemma 3n E4B IT [39]	Text	15.49	43.11	14.39	22.50	26.51	1.71
Kimi Audio 7B Instruct [16]	Text	13.72	48.36	14.39	16.67	20.48	5.13
Kimi Audio 7B Instruct	Audio	10.40	22.08	15.91	6.67	13.25	5.98
LFM2 Audio 1.5B [3]	Audio	9.29	19.10	15.15	7.50	15.66	0.00

表 2：各模型、各输出模态在 Audio MultiChallenge 上的总体与分轴得分。APR 作为最终成绩。IM：推理记忆，IR：指令保持，SC：自我连贯，VE：语音编辑。绿色高亮表示文本输出下的最佳成绩；蓝色高亮表示音频输出下的最佳成绩。

3.1 Results

3.1 结果

We present scores on Audio MultiChallenge across a variety of E2E architectures in Table 2, along with their performance per-axis. We observe that Gemini 3 Pro Preview (Thinking) achieves the highest overall score of 54.65%, followed by its counterparts Gemini 2.5 Pro (Thinking) and 2.5 Flash (Thinking). In comparison, the remainder of proprietary and open-source models perform modestly, with GPT 4o Audio Preview and Voxtral Small 24B leading, scoring a 25.44% and 26.33% APR respectively, closely followed by GPT Realtime with an APR of 23.45% (output modality set to Text). Open-source S2S architectures in particular struggle the most, with no model surpassing Qwen 3 Omni which achieves a modest 24.34% APR. Analyzing per-axis scores, our newly adapted Voice Editing and Inference Memory prove to be the most challenging tasks for frontier models, with them scoring 17.99% and 21.55% on average, respectively. We highlight the following key findings and provide samples of model failures in Appendix A.1.

表 2 给出了多种端到端架构在 Audio MultiChallenge 上的得分，以及分轴表现。我们看到，Gemini 3 Pro Preview（Thinking）取得最高的总体成绩 54.65%，其后是同系列的 Gemini 2.5 Pro（Thinking）和 2.5 Flash（Thinking）。相比之下，其余专有和开源模型表现平平：GPT 4o Audio Preview 与 Voxtral Small 24B 相对靠前，APR 分别为 25.44% 和 26.33%，紧随其后的是输出模态设为文本的 GPT Realtime，APR 为 23.45%。开源 S2S 架构尤其吃力，没有任何模型超过 Qwen 3 Omni 那不算高的 24.34% APR。从分轴得分看，我们新适配的语音编辑和推理记忆对前沿模型最难，平均分别只有 17.99% 和 21.55%。下面我们强调几项主要发现，模型失败样本见附录 A.1。

Performance varies over output modalities

We observe a "modality gap" when evaluating models that support both text and audio output streams. Under a Text only output configuration, these models score 19.36% on average, however, when the same models are configured to output Audio (or joint Audio and Text depending on their architecture), their performance degrades to 16.76%. These findings are independently reported in other recent works [43], highlighting the need for improved S2S post-training data to overcome this gap in performance.

表现随输出模态而变化

在同时支持文本和音频输出流的模型上，我们观察到一种“模态差距”。在仅输出文本的配置下，这些模型平均得分 19.36%；但当同一批模型被配置为输出音频（或按架构输出音频与文本的组合）时，表现降到 16.76%。其他近期工作也独立报告了类似发现 [43]，说明需要改进 S2S 的后期训练数据，才能弥补这一差距。

Audio-Cue Inference Memory lags substantially

We classify Audio-Cue gated Inference Memory tasks ($n=42$) as ones where the information the model must remember from a previous turn is contained only in the audio signal rather than in the words spoken, as described in Section 2.1. Our examples span both paralinguistic cues such as the speaker’s tone and ambient cues such as background sound. Figure 5 shows a significant gap in remembering these audio cues in our multi-turn setup. Across most models, performance drops substantially when the task requires inferring paralinguistic features or ambient sounds from previous turns, with ARS falling by 36.5% relatively compared to semantic memory on average across models. This unique subset of our data highlights the limitations of current E2E spoken dialogue systems when it comes to audio-native state tracking and recall, which is fundamental to realtime and multi-turn voice conversations.

音频线索推理记忆明显落后

我们把音频线索门控的推理记忆任务（$n=42$）定义为：模型必须记住的、来自先前轮次的信息，只包含在音频信号里，而不在说出的语词中，如第 2.1 节所述。我们的例子既包括说话人语气之类的副语言线索，也包括背景声之类的环境线索。图 5 显示，在我们的多轮设定下，记住这些音频线索存在明显差距。对大多数模型而言，当任务要求从先前轮次推断副语言特征或环境声时，表现会明显下降：各模型平均而言，ARS 相对语义记忆下降 36.5%。这一独特子集凸显了当前端到端口语对话系统在音频原生的状态跟踪与回想上的局限，而这正是实时、多轮语音对话的基本能力。

Figure 5: APR (%) scores on Audio-Cue vs. Semantic Inference Memory tasks. (T) indicates text output, (A) indicates audio output.

图 5：音频线索推理记忆与语义推理记忆任务上的 APR（%）得分。(T) 表示文本输出，(A) 表示音频输出。

Conditional Instructions cause model failures

Frontier models score 25.08% APR and 46.11% ARS on average on Instruction Retention (IR) tasks. To analyze the main sources of error, we automatically cluster each IR rubric item into two instruction types: Fixed, which are unconditional, persistent constraints meant to hold throughout the conversation (e.g., response style or roleplay prompts), and Conditional, which are context-dependent instructions that only apply in specific future-turn scenarios. Figure 6 shows that each model, regardless of their overall performance, demonstrates worse performance in conditional IR. These results support treating conditional instruction following as a distinct, harder sub-problem within IR, and they motivate future work on reliably retrieving and applying conditional rules rather than optimizing only for conventional instruction following.

条件指令会带来模型失败

前沿模型在指令保持（IR）任务上平均 APR 为 25.08%，ARS 为 46.11%。为分析主要错误来源，我们把每条 IR 细则自动聚类成两类指令：固定指令是无条件、持续有效、应在整段对话中保持的约束（例如回复风格或角色扮演提示）；条件指令则依赖上下文，只在未来某些特定轮次的情形下才适用。图 6 显示，无论总体表现如何，每个模型在条件 IR 上都更差。这些结果支持把条件指令遵循当作 IR 内部一个更难的独立子问题，并促使后续工作去可靠地检索和施用条件规则，而不是只优化常规的指令遵循。

Figure 6: ARS (%) on Instruction Retention tasks classified by rubric conditionality. ARS is reported instead of APR since the classification is at the rubric level – whereas a single task can have both conditional and fixed rubrics. (T) indicates text output, (A) indicates audio output.

图 6：按细则条件性分类的指令保持任务上的 ARS（%）。由于分类是在细则级别进行的——同一任务可以同时有条件细则和固定细则——这里报告 ARS 而非 APR。(T) 表示文本输出，(A) 表示音频输出。

Self Coherence linearly declines on tasks with increased context

Since the models we benchmark use different audio tokenizers, raw token counts are not directly comparable across systems. We therefore use cumulative user-audio duration per task as a proxy for audio context length. Self Coherence degrades sharply as cumulative user audio increases. Figure 8b shows that APR on Self Coherence drops from 33.3% for tasks with 0–60 seconds of cumulative user audio to 20.0% for tasks with 3–5 minutes, and further to 13.3% for tasks exceeding 5 minutes (though this final bin contains only 3 samples) when averaged across all models. We analyze duration effects on other axes in Section 3.2.

自我连贯随上下文增加而近似线性下降

由于我们所评测的模型使用不同的音频分词器，原始标记数在系统之间不能直接比较。因此我们用每项任务的用户音频累计时长，作为音频上下文长度的代理。随着用户音频累计增加，自我连贯明显变差。图 8b 显示，在所有模型平均下，自我连贯的 APR 从累计用户音频为 0–60 秒的任务上的 33.3%，降到 3–5 分钟任务上的 20.0%，在超过 5 分钟的任务上进一步降到 13.3%（不过最后这一档只有 3 个样本）。其他轴上的时长效应见第 3.2 节。

Mid-Utterance and Prior-Turn Voice Edits hurt performance

Voice Editing remains our most challenging axis, with an average APR of 17.99%. Most failures on Voice Editing occur when users introduce multiple mid-utterance (in-turn speech repairs) and prior-turn edits throughout a conversation, with the final prompt typically requesting a summary or revision that integrates all previously specified changes. Successfully completing these tasks requires strong state tracking and multi-constraint integration at the semantic level, as well as robustness to user backchannels and disfluencies at the acoustic level. This helps explain why stronger or reasoning-heavy models can be similarly robust to mid-utterance and prior-turn edits and to unedited rubric items corresponding to simple instructions that were never modified, as shown in Figure 7. Our ablation studies in Section 3.4 further highlight the difficulty of this axis.

句中编辑和跨轮语音编辑会损害表现

语音编辑仍是我们最难的一条轴，平均 APR 为 17.99%。该轴上的多数失败出现在用户在对话中引入多次句中（同一轮内的语音修正）和跨轮编辑，而最后一轮提示通常要求做一份把先前所有修改都整合进去的摘要或修订。要成功完成这些任务，既需要语义层面的强状态跟踪和多约束整合，也需要声学层面对用户附和与不流畅现象的稳健性。这也有助于解释：为何更强或更偏推理的模型，对句中编辑、跨轮编辑，以及对从未被修改过的简单指令所对应的未编辑细则，可以表现出相近的稳健程度，如图 7 所示。第 3.4 节的消融进一步凸显了这条轴的难度。

Figure 7: ARS (%) on Voice Editing tasks classified by the edit type made by the user for each rubric. Unedited refers to statements that the user did not backtrack or revise either within the same turn or in future turns. (T) indicates text output, (A) indicates audio output.

图 7：按用户对每条细则所做编辑类型分类的语音编辑任务 ARS（%）。未编辑指用户既未在同一轮内、也未在后续轮次中撤回或修改的陈述。(T) 表示文本输出，(A) 表示音频输出。

3.2 Analysis

3.2 分析

Turn Count

Figure 8a shows no meaningful relationship between turn count and performance: the scores in Audio MultiChallenge remain largely stable as the number of turns increases. Although the benchmark is multi-turn, the conversations are short relative to the context windows of modern models, so adding turns does not mechanically increase difficulty. This mirrors findings from prior work using fixed-context evaluation protocols [15]. Since each turn’s length is also arbitrarily defined, it is a weak measure for conversation histories that models must parse through. Furthermore, since the history for all previous turns is provided to the model and is audited (i.e., free of assistant hallucinations or errors), models can often leverage additional in-context examples at higher turn counts to respond consistently. We observe similarly weak trends when analyzing performance by axis.

轮次数

图 8a 显示，轮次数与表现之间没有有意义的关系：Audio MultiChallenge 上的得分随轮次增加大体保持稳定。尽管这是多轮基准，对话相对现代模型的上下文窗口仍然偏短，因此增加轮次并不会机械地提高难度。这与先前使用固定上下文评测协议的发现一致 [15]。此外，每一轮的长度本身也是任意划分的，作为模型必须解析的对话历史的度量并不强。再者，此前所有轮次的历史都提供给了模型，并且经过审核（即不含助手幻觉或错误），模型在更高轮次时往往还能利用额外的上下文样例来保持回复一致。按轴分析时，我们也看到类似的弱趋势。

Figure 8: APR (%) across (a) number of turns, (b) binned total user audio duration (in seconds)

图 8：APR（%）随 (a) 轮次数、(b) 分箱后的用户音频总时长（秒）的变化

Audio Duration

Unlike turn count, total user-audio duration is a more direct proxy for effective context length. Figure 8b shows a clear degradation in Self Coherence as audio duration increases, consistent with the trend discussed above. Voice Editing also worsens with longer histories as average APR falls from 17.5% in the 3–5 minute bin to 1.2% and 0% for 5-8 minute and 8+ minute tasks respectively (though these bins have low sample sizes $n=4,2$). Inference Memory declines more modestly with duration, driven primarily by Semantic Inference Memory, where longer contexts make targeted recall harder. By contrast, Audio-Cue Inference Memory shows no clear trend with duration; qualitatively, we observe performance is often bottlenecked first by audio perception (e.g., reliably identifying paralinguistic cues or background sounds) rather than retention per se, leading to failures even on short clips. Interestingly, Instruction Retention remains steady or even improves on longer audio tasks because many instruction-following tasks introduce conditional or fixed constraints early in the dialogue that must be followed throughout. Under our fixed-context protocol, the model also receives audited prior assistant turns that already follow these constraints, providing additional in-context demonstrations it can imitate in later turns, consistent with in-context learning effects observed on instruction following in LLMs [17].

音频时长

与轮次数不同，用户音频总时长是有效上下文长度更直接的代理。图 8b 显示，自我连贯随音频时长增加而明显下降，与上文趋势一致。语音编辑在更长历史上也会变差：平均 APR 从 3–5 分钟档的 17.5%，分别降到 5–8 分钟和 8 分钟以上任务的 1.2% 和 0%（不过这两档样本很少，$n=4,2$）。推理记忆随时长下降较为温和，主要由语义推理记忆驱动——更长上下文使有针对性的回想更难。相比之下，音频线索推理记忆与时长没有清晰趋势；从定性上看，我们观察到表现往往首先受制于音频感知（例如能否可靠识别副语言线索或背景声），而不是记忆本身，因此即使在短片段上也会失败。有意思的是，指令保持在更长音频任务上保持稳定，甚至有所提高，因为许多指令遵循任务会在对话较早阶段引入必须全程遵守的条件或固定约束。在我们的固定上下文协议下，模型还会收到已经遵守这些约束的、经过审核的先前助手轮次，从而获得可在后续轮次中模仿的额外上下文示范，这与在 LLM 指令遵循中观察到的上下文学习效应一致 [17]。

Figure 9: Rubric compliance (ARS) as a function of the model’s average response length in tokens.

图 9：细则符合率（ARS）与模型平均回复长度（以标记计）的关系。

Response Length

One hypothesis for why models score higher on rubric-based benchmarks is their scores may correlate with response length [37]. Since rubrics evaluate against an explicit list of criteria, an overly verbose model could satisfy more rubric items even if the output is not realistic or human-desirable. A common strategy to mitigate this is to include negatively weighted criteria for incorrect or undesirable content. For Audio MultiChallenge, we adapt this approach by utilizing exclusion rubrics (e.g., “Avoids mentioning leather products”) that explicitly check whether the model omits information the user instructed not to mention, later edited away, or signaled a preference to avoid. Figure 9 plots rubric compliance (ARS) against each model’s average response length. Response length is measured by tokenizing each model’s text output stream with the tiktoken library. Despite these controls, we observe a strong, approximately linear relationship between response length and ARS. To assess whether this reflects score hacking, we independently review responses from the Gemini family and find no evidence that their higher scores are driven primarily by verbosity. Instead, their outputs more consistently reflect stronger audio and semantic understanding across axes and more reliable state tracking.

回复长度

关于模型为何在基于细则的基准上得分更高，一种假设是分数可能与回复长度相关 [37]。由于细则是对照一份显式标准清单来评的，过于冗长的模型即使输出并不真实或不符合人的期望，也可能满足更多细则条目。常见的缓解办法是为错误或不受欢迎的内容加入负向加权标准。对 Audio MultiChallenge，我们借用这一思路，使用排除性细则（例如“避免提及皮革制品”），明确检查模型是否省略了用户要求不要提及、后来已编辑掉、或表示希望避开的信息。图 9 画出细则符合率（ARS）与各模型平均回复长度的关系。回复长度用 tiktoken 库对每个模型的文本输出流分词得到。即便有了这些控制，我们仍观察到回复长度与 ARS 之间存在很强的、近似线性的关系。为判断这是否属于刷分，我们独立审阅了 Gemini 系列的回复，没有发现其更高分数主要来自啰嗦。相反，它们的输出在各条轴上更稳定地体现出更强的音频与语义理解，以及更可靠的状态跟踪。

3.3 LLM-as-a-Judge

3.3 大模型评委

Judge	Cohen’s $\kappa$	Macro F1
o4 Mini	0.873	0.937
GPT 5 Mini	0.870	0.935
GPT 4.1	0.811	0.906
Claude Haiku 4.5	0.808	0.904
Mistral Medium	0.787	0.894
DeepSeek V3.1	0.765	0.882

Table 3: LLM Judge agreement with humans.

评委	Cohen’s $\kappa$	宏平均 F1
o4 Mini	0.873	0.937
GPT 5 Mini	0.870	0.935
GPT 4.1	0.811	0.906
Claude Haiku 4.5	0.808	0.904
Mistral Medium	0.787	0.894
DeepSeek V3.1	0.765	0.882

表 3：大模型评委与人类的一致性。

To evaluate our LLM-as-a-judge rubric-grading setup, we measure inter-rater agreement between model-generated and human-assigned labels across the full dataset. Human annotators are instructed to grade randomly sampled GPT 4o Audio Preview or Gemini 2.5 Pro responses according to their corresponding written rubrics.

为评估我们用大模型评委按细则打分的设置，我们在整个数据集上测量模型生成标签与人工标注之间的评分者一致性。人类标注者被要求按相应的书面细则，为随机抽样的 GPT 4o Audio Preview 或 Gemini 2.5 Pro 回复打分。

We aggregate these rubric grades into a set of 1,712 observations and compute Cohen’s $\kappa$ and Macro F1 scores calculated following Arora et al. [4]. To mitigate the self-enhancement bias known to affect LLM judges [48, 42], we exclude our evaluated ALMs which are capable of text-based prompts from judging. Our results, presented in Table 3, show that all judges achieve extremely high agreement with human graders, indicating the effectiveness of our rubric creation and overall LLM-as-a-Judge setup. Among the tested judges, o4 Mini attains the highest performance, with a Macro F1 score of 0.937, and is therefore selected as our primary LLM judge. We provide our LLM-as-a-Judge setup in Appendix A.3.

我们把这些细则评分汇总成 1,712 条观测，并按 Arora 等人 [4] 的做法计算 Cohen’s $\kappa$ 和宏平均 F1。为减轻已知会影响大模型评委的自我增强偏差 [48, 42]，我们不让所评测的、能够处理文本提示的音频语言模型担任评委。表 3 显示，所有评委与人类打分者都达到极高的一致性，说明我们的细则撰写和大模型评委整体设置是有效的。在受测评委中，o4 Mini 表现最好，宏平均 F1 为 0.937，因此被选为主要的大模型评委。大模型评委的设置见附录 A.3。

3.4 Ablation Studies

3.4 消融实验

TTS vs. Human User Audio

To isolate the impact of spontaneous and disfluent speech, we evaluate our models on synthetic versions of our human user audio, generated by first performing ASR using whisper-large-v3 followed by TTS using gpt-4o-mini-tts. We exclude Audio-Cue IM and tasks that contain interruptions or sidebar conversations, resulting in a filtered subset of 385 samples for this study. Results in Table 4a depict models configured for text outputs achieve a 7.5% relative performance improvement when using TTS speech. In contrast, the same architectures configured for audio outputs exhibit a 2.5% relative decline, suggesting their post-training may be optimized for real human input.

TTS 与真人用户音频

为单独考察自发和不流畅语音的影响，我们在由真人用户音频生成的合成版本上评测模型：先用 whisper-large-v3 做 ASR，再用 gpt-4o-mini-tts 做 TTS。我们排除音频线索推理记忆，以及含插话或旁支对话的任务，得到用于本研究的 385 条过滤子集。表 4a 显示，配置为文本输出的模型在使用 TTS 语音时相对提高 7.5%；相反，同一批架构在配置为音频输出时相对下降 2.5%，说明它们的后期训练可能更针对真实的人类输入做了优化。

Table 4b shows that Voice Editing benefits the most from TTS under text output configurations. We attribute this effect to the inability of ASR and TTS to fully capture human-like hesitations and unintelligible segments in highly spontaneous speech. The resulting input is slower and more structured, making Voice Editing tasks easier to follow. However, the performance degradation observed for audio output configurations on Voice Editing and Instruction Retention when using TTS requires further investigation. Finally, while this study applies TTS to transcripts derived from unscripted human speech, our internal findings indicate that TTS inputs boost ALM performance even strongly for scripted or fully synthetic dialogue, underscoring the importance of collecting real human recordings to accurately evaluate model performance.

表 4b 显示，在文本输出配置下，语音编辑从 TTS 中获益最多。我们将这一效应归因于：ASR 和 TTS 无法完整捕捉高度自发语音里类似真人的犹豫和含混片段。得到的输入更慢、更有条理，因而语音编辑任务更容易跟上。不过，音频输出配置在 TTS 上于语音编辑和指令保持上出现的表现下降，还需要进一步研究。最后，虽然本研究是把 TTS 用在由无脚本真人语音得到的转写上，我们内部的发现表明，对照稿或完全合成的对话，TTS 输入对音频语言模型表现的提升甚至更强，这更说明，要准确评测模型表现，采集真实的人类录音十分重要。

Model	Output Modality	Human APR (%)	TTS APR (%)	$\Delta$Relative (%)
Gemini 3 Pro Preview	Text	55.58	56.62	+1.9
Gemini 2.5 Pro	Text	47.53	49.09	+3.3
Gemini 2.5 Flash Thinking	Text	41.56	42.86	+3.1
Gemini 2.5 Flash	Text	25.97	27.79	+7.0
GPT-4o Audio Preview	Text	27.27	28.05	+2.9
GPT-4o Audio Preview	Audio	24.42	23.64	-3.2
GPT-4o Mini Audio Preview	Text	15.84	17.92	+13.1
GPT-4o Mini Audio Preview	Audio	14.03	13.77	-1.9
GPT Realtime	Text	24.16	29.35	+21.5
GPT Realtime	Audio	20.52	20.00	-2.5

Axis	Output Modality	Human APR (%)	TTS APR (%)	$\Delta$Relative (%)
Inference Memory	Text	34.05	34.88	+2.4
Inference Memory	Audio	19.38	20.54	+6.0
Instruction Retention	Text	37.12	38.95	+4.9
Instruction Retention	Audio	20.80	18.52	-11.0
Self Coherence	Text	34.18	35.62	+4.2
Self Coherence	Audio	24.47	26.58	+8.6
Voice Editing	Text	30.24	33.70	+11.5
Voice Editing	Audio	14.89	12.94	-13.0

(a) APR (%) on Human vs. TTS user audio per model.

模型	输出模态	真人 APR（%）	TTS APR（%）	$\Delta$相对（%）
Gemini 3 Pro Preview	Text	55.58	56.62	+1.9
Gemini 2.5 Pro	Text	47.53	49.09	+3.3
Gemini 2.5 Flash Thinking	Text	41.56	42.86	+3.1
Gemini 2.5 Flash	Text	25.97	27.79	+7.0
GPT-4o Audio Preview	Text	27.27	28.05	+2.9
GPT-4o Audio Preview	Audio	24.42	23.64	-3.2
GPT-4o Mini Audio Preview	Text	15.84	17.92	+13.1
GPT-4o Mini Audio Preview	Audio	14.03	13.77	-1.9
GPT Realtime	Text	24.16	29.35	+21.5
GPT Realtime	Audio	20.52	20.00	-2.5

轴	输出模态	真人 APR（%）	TTS APR（%）	$\Delta$相对（%）
推理记忆	Text	34.05	34.88	+2.4
推理记忆	Audio	19.38	20.54	+6.0
指令保持	Text	37.12	38.95	+4.9
指令保持	Audio	20.80	18.52	-11.0
自我连贯	Text	34.18	35.62	+4.2
自我连贯	Audio	24.47	26.58	+8.6
语音编辑	Text	30.24	33.70	+11.5
语音编辑	Audio	14.89	12.94	-13.0

(a) 各模型在真人与 TTS 用户音频上的 APR（%）。

E2E vs. Cascaded Systems

We benchmark frontier LLMs in a cascaded pipeline (ASR + LLM) to test their performance on our dataset. For all cascaded experiments, we use the same transcripts obtained from whisper-large-v3 as the text input to the LLMs. Table 5a reports APR (%) on our previously filtered subset that excludes audio-cue and interruption tasks. We observe cascaded systems using GPT 5 and Claude Opus 4.5 perform strongly relative to most E2E baselines, though they do not surpass all Gemini models. Notably, cascading reduces performance for Gemini 3 Pro Preview and Gemini 2.5 Pro, with relative drops of 10.3% and 4.9%, while Gemini 2.5 Flash Thinking improves by 10.0%. A possible hypothesis is that newer Gemini models are post-trained more extensively on audio-native data to function as E2E multimodal architectures, rather than solely being strong LLMs.

端到端与级联系统

我们在级联流水线（ASR + LLM）中评测前沿 LLM，以检验它们在本数据集上的表现。所有级联实验都使用由 whisper-large-v3 得到的同一批转写，作为 LLM 的文本输入。表 5a 报告了先前已过滤、排除音频线索和插话任务的子集上的 APR（%）。我们看到，使用 GPT 5 和 Claude Opus 4.5 的级联系统相对大多数端到端基线表现较强，但并未超过所有 Gemini 模型。值得注意的是，级联会降低 Gemini 3 Pro Preview 和 Gemini 2.5 Pro 的表现，相对降幅分别为 10.3% 和 4.9%，而 Gemini 2.5 Flash Thinking 则提高 10.0%。一种可能的解释是，较新的 Gemini 模型在后期训练中更充分地使用了音频原生数据，以便作为端到端多模态架构来工作，而不仅仅是很强的 LLM。

Table 5b reports performance on the excluded audio specific subset containing audio cues or interruptions ($n=67$). As expected, cascaded performance drops sharply for Gemini 3 Pro Preview and Gemini 2.5 Pro, with relative declines of 30.3% and 24.1%. In contrast, Gemini 2.5 Flash Thinking shows no change between E2E and cascaded on this subset. While GPT 5, Claude Opus 4.5, and GPT 4o cascaded systems also drop in APR on this subset, they still achieve nontrivial scores. This pattern is consistent with recent findings that audio understanding can improve through text-only post training [36], and suggests that reasoners may partially infer some audio cues, particularly paralinguistic and emotional context, from transcripts and surrounding dialogue.

表 5b 报告了被排除的、含音频线索或插话的音频特定子集（$n=67$）上的表现。正如预期，Gemini 3 Pro Preview 和 Gemini 2.5 Pro 的级联表现大幅下降，相对降幅分别为 30.3% 和 24.1%。相比之下，Gemini 2.5 Flash Thinking 在该子集上端到端与级联之间没有变化。虽然 GPT 5、Claude Opus 4.5 和 GPT 4o 的级联系统在该子集上的 APR 也有下降，但仍然取得了不可忽视的分数。这一模式与近期发现一致：音频理解可以通过纯文本后期训练得到提升 [36]，并表明推理模型或许能从转写和周围对话中，部分推断出某些音频线索，尤其是副语言和情绪上下文。

Model	E2E APR (%)	Cascaded APR (%)	$\Delta$Relative (%)
Gemini 3 Pro Preview	55.58	49.87	-10.3
Gemini 2.5 Pro	47.53	45.19	-4.9
Gemini 2.5 Flash Thinking	41.56	45.71	+10.0
GPT 5	–	51.17	–
Claude Opus 4.5	–	39.22	–
GPT 4o	–	28.57	–
GPT 4o Audio Preview (Text)	27.27	–	–
GPT Realtime (Text)	24.16	–	–

Model	E2E APR (%)	Cascaded APR (%)	$\Delta$Relative (%)
Gemini 3 Pro Preview	49.25	34.33	-30.3
Gemini 2.5 Pro	43.28	32.84	-24.1
Gemini 2.5 Flash Thinking	31.34	31.34	0.0
GPT 5	–	35.82	–
Claude Opus 4.5	–	25.37	–
GPT 4o	–	23.88	–
GPT Realtime (Text)	19.40	–	–
GPT 4o Audio Preview (Text)	14.93	–	–

(a) APR (%) on tasks excluding audio-cues & interruptions

模型	端到端 APR（%）	级联 APR（%）	$\Delta$相对（%）
Gemini 3 Pro Preview	55.58	49.87	-10.3
Gemini 2.5 Pro	47.53	45.19	-4.9
Gemini 2.5 Flash Thinking	41.56	45.71	+10.0
GPT 5	–	51.17	–
Claude Opus 4.5	–	39.22	–
GPT 4o	–	28.57	–
GPT 4o Audio Preview (Text)	27.27	–	–
GPT Realtime (Text)	24.16	–	–

模型	端到端 APR（%）	级联 APR（%）	$\Delta$相对（%）
Gemini 3 Pro Preview	49.25	34.33	-30.3
Gemini 2.5 Pro	43.28	32.84	-24.1
Gemini 2.5 Flash Thinking	31.34	31.34	0.0
GPT 5	–	35.82	–
Claude Opus 4.5	–	25.37	–
GPT 4o	–	23.88	–
GPT Realtime (Text)	19.40	–	–
GPT 4o Audio Preview (Text)	14.93	–	–

(a) 排除音频线索与插话任务后的 APR（%）

4 Related Work

4 相关工作

Single-Turn Speech Evaluation

Existing single-turn speech-to-speech (S2S) and audio–language model (ALM) benchmarks primarily target isolated utterances or short clips. VoiceBench [9] and AudioBench [40] cover a range of speech understanding, audio scene understanding, and paralinguistic tasks where prompts are typically brief and close-ended. Big Bench Audio [22] similarly adapts Big Bench Hard [38] reasoning tasks to the audio modality using TTS to synthesize its queries. Recent works on closing the text–speech understanding gap [11] and stress-sensitive prosody reasoning [46] likewise rely on synthetic or carefully read speech and focus on low-level capabilities (e.g., ASR, speaker traits, prosodic robustness) rather than interactive dialogue. Furthermore, audio reasoning benchmarks such as MMAU-Pro [27] and MMAR [32] broaden coverage to human speech, environmental sounds, and music but generally adopt a single-turn audio QA format, rather than testing conversational behavior. Long-form audio benchmarks such as BLAB [2], AudioMarathon [21], and LongAudioBench [19] construct hour-scale recordings and show that model performance degrades as audio inputs become longer and sparser. However, they too treat each recording as a single comprehension task and do not require models to track user preferences, edits, or self-consistency which are common patterns in human-assistant interaction.

单轮语音评测

现有的单轮语音到语音（S2S）和音频语言模型（ALM）基准，主要针对孤立话语或短片段。VoiceBench [9] 和 AudioBench [40] 覆盖一系列语音理解、音频场景理解和副语言任务，提示通常简短且封闭。Big Bench Audio [22] 同样把 Big Bench Hard [38] 的推理任务改到音频模态，并用 TTS 合成问句。近期关于缩小文本—语音理解差距 [11] 以及对应激敏感的韵律推理 [46] 的工作，同样依赖合成或仔细朗读的语音，焦点也在底层能力（如 ASR、说话人特质、韵律稳健性），而不是交互式对话。此外，MMAU-Pro [27] 和 MMAR [32] 等音频推理基准把覆盖面拓宽到人类语音、环境声和音乐，但一般仍采用单轮音频问答形式，而不检验对话行为。BLAB [2]、AudioMarathon [21] 和 LongAudioBench [19] 等长音频基准构造小时级录音，并表明随着音频输入变长、变稀疏，模型表现会下降。不过它们同样把每段录音当作单一理解任务，并不要求模型跟踪在人机助手交互中常见的用户偏好、编辑或自我一致性。

Multi-Turn LLM Evaluation

Multi-turn evaluations have been explored more deeply in text-only settings. MultiChallenge [15] builds synthetic conversations targeting instruction retention, user information recall, versioned editing, and self-coherence, and finds that frontier models struggle despite near-saturation on earlier multi-turn benchmarks. Further studies report that LLM performance drops significantly when moving from single-turn to multi-turn settings across long-context generation, state tracking, reasoning, and aggregation tasks [28, 6].

多轮 LLM 评测

多轮评测在纯文本设定下已被更深入地探讨。MultiChallenge [15] 构建针对指令保持、用户信息回想、版本化编辑和自我连贯的合成对话，并发现尽管前沿模型在更早的多轮基准上已接近饱和，在这里仍然吃力。后续研究还报告，从单轮转到多轮后，LLM 在长上下文生成、状态跟踪、推理和聚合任务上的表现会明显下降 [28, 6]。

Multi-Turn Speech Evaluation

In speech-native settings, however, multi-turn evaluation remains comparatively underexplored. MTalk-Bench [18] introduces multi-turn S2S dialogues spanning semantic content, paralinguistic cues, and ambient sound, combining arena-style pairwise comparison with rubric-based absolute scoring, but is limited to 2–3 turns and uses scripted dialogue. Our work targets this gap by focusing on extended multi-turn S2S or S2T interaction, long-horizon instruction retention, inference memory, and robustness to non-monotonic user behavior. Furthermore, to the best of our knowledge, there is currently no speech-native benchmark that systematically tests Voice Editing in multi-turn interaction.

多轮语音评测

然而在语音原生设定下，多轮评测仍然相对不足。MTalk-Bench [18] 引入覆盖语义内容、副语言线索和环境声的多轮 S2S 对话，把竞技场式成对比较与基于细则的绝对打分结合起来，但限于 2–3 轮，并且使用照稿对话。我们的工作针对这一空白，关注更长的多轮 S2S 或语音到文本交互、长程指令保持、推理记忆，以及对非单调用户行为的稳健性。此外，据我们所知，目前还没有语音原生基准系统地检验多轮交互中的语音编辑。

5 Conclusion and Future Work

5 结论与未来工作

We introduce Audio MultiChallenge, a collection of 452 realistic and challenging multi-turn conversations between humans and spoken dialogue systems. We curate 1,712 instance-specific rubrics to evaluate four core competencies central to voice-based assistants, Inference Memory, Instruction Retention, Self Coherence, and Voice Editing. Our results reveal substantial gaps in frontier model performance on audio-cue retention, conditional instruction following, long-context self-consistency, and mid-utterance edits. We open-source Audio MultiChallenge to support reproducible evaluation and to drive progress toward more capable E2E architectures in real-world multi-turn settings.

我们提出 Audio MultiChallenge，这是 452 段人类与口语对话系统之间真实、有难度的多轮对话合集。我们整理了 1,712 条实例级细则，用来评测语音助手的四项核心能力：推理记忆、指令保持、自我连贯和语音编辑。结果表明，前沿模型在音频线索保持、条件指令遵循、长上下文自我一致性以及句中编辑上，仍有明显差距。我们开源 Audio MultiChallenge，以支持可复现的评测，并推动端到端架构在真实多轮场景中走向更强的能力。

A key limitation of this work is that our evaluation does not operate directly on raw audio outputs. Future work should explore improved hybrid data curation pipelines, longer conversational contexts, and open-ended audio-native evaluation frameworks. Such methods could enable principled assessment of paralinguistic instruction retention and surface open questions around rubric design, verification, and the role of audio-native judges.

这项工作的一个主要局限是，评测并不直接作用于原始音频输出。未来工作应探索改进的混合数据整理流水线、更长的对话上下文，以及开放式的音频原生评测框架。这类方法有助于对副语言指令保持做有原则的评估，并引出围绕细则设计、核验以及音频原生评委角色的开放问题。

Acknowledgment

致谢

We thank Kaustubh Deshpande for his insights on our methodology, and Ashan Panduwawala for his engineering support in setting up our human-in-the-loop data collection. We also thank Rani Alsaberi and Chanyu Li for their contributions towards our data curation and quality control.

我们感谢 Kaustubh Deshpande 对方法提出的见解，以及 Ashan Panduwawala 在搭建人工在环数据采集上提供的工程支持。我们也感谢 Rani Alsaberi 和 Chanyu Li 在数据整理与质量控制方面的贡献。

References

参考文献

[1] A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V. Chaudhary, C. Chen, et al. (2025) Phi-4-mini technical report: compact yet powerful multimodal language models via mixture-of-loras . arXiv preprint arXiv:2503.01743 . Cited by: Table 2 .

[2] O. Ahia, M. Bartelds, K. Ahuja, H. Gonen, V. Hofmann, S. Arora, S. S. Li, V. Puttagunta, M. Adeyemi, C. Buchireddy, B. Walls, N. Bennett, S. Watanabe, N. A. Smith, Y. Tsvetkov, and S. Kumar (2025) BLAB: brutally long audio bench . arXiv preprint arXiv:2505.03054 . Cited by: §4 .

[3] A. Amini, A. Banaszak, H. Benoit, A. Böök, T. Dakhran, S. Duong, A. Eng, F. Fernandes, M. Härkönen, A. Harrington, et al. (2025) LFM2 technical report . arXiv preprint arXiv:2511.23404 . Cited by: Table 2 .

[4] R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal (2025) HealthBench: evaluating large language models towards improved human health . External Links: 2505.08775 , Link Cited by: §A.3 , §3.3 .

[5] S. Arora, J. Tian, H. Futami, J. Jung, J. Shi, Y. Kashiwagi, E. Tsunoo, and S. Watanabe (2025) Chain-of-thought training for open e2e spoken dialogue systems . arXiv preprint arXiv:2506.00722 . Cited by: §1 .

[6] A. Bertsch, A. Pratapa, T. Mitamura, G. Neubig, and M. R. Gormley (2025) Oolong: evaluating long context reasoning and aggregation capabilities . arXiv preprint arXiv:2511.02817 . Cited by: §1 , §4 .

[7] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour (2023) AudioLM: a language modeling approach to audio generation . External Links: 2209.03143 , Link Cited by: §1 , §1 .

[8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners . External Links: 2005.14165 , Link Cited by: §1 .

[9] Y. Chen, X. Yue, C. Zhang, X. Gao, R. T. Tan, and H. Li (2024) VoiceBench: benchmarking llm-based voice assistants . arXiv preprint arXiv:2410.17196 . Cited by: §1 , §4 .

[10] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities . arXiv preprint arXiv:2507.06261 . Cited by: Table 2 .

[11] S. Cuervo, S. Seto, M. de Seyssel, R. H. Bai, Z. Gu, T. Likhomanenko, N. Jaitly, and Z. Aldeneh (2025) Closing the gap between text and speech understanding in llms . External Links: 2510.13632 , Link Cited by: §4 .

[12] W. Cui, D. Yu, X. Jiao, Z. Meng, G. Zhang, Q. Wang, S. Y. Guo, and I. King (2025) Recent advances in speech language models: a survey . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.) , Vienna, Austria , pp. 13943–13970 . External Links: Link , Document , ISBN 979-8-89176-251-0 Cited by: §1 , §1 .

[13] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi (2022) High fidelity neural audio compression . External Links: 2210.13438 , Link Cited by: §1 .

[14] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour (2024) Moshi: a speech-text foundation model for real-time dialogue . External Links: 2410.00037 , Link Cited by: §1 .

[15] K. Deshpande, V. Sirdeshmukh, J. B. Mols, L. Jin, E. Hernandez-Cardona, D. Lee, J. Kritz, W. E. Primack, S. Yue, and C. Xing (2025) MultiChallenge: a realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs . In Findings of the Association for Computational Linguistics: ACL 2025 , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.) , Vienna, Austria , pp. 18632–18702 . External Links: Link , Document , ISBN 979-8-89176-256-5 Cited by: §1 , §1 , §3 , §3.2 , §4 .

[16] D. Ding, Z. Ju, Y. Leng, S. Liu, T. Liu, Z. Shang, K. Shen, W. Song, X. Tan, H. Tang, et al. (2025) Kimi-audio technical report . arXiv preprint arXiv:2504.18425 . Cited by: Table 2 .

[17] Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu, B. Chang, X. Sun, L. Li, and Z. Sui (2024) A survey on in-context learning . External Links: 2301.00234 , Link Cited by: §3.2 .

[18] Y. Du, Q. Huang, G. Zhu, Z. Dai, S. Chen, Q. Zhu, Y. Zhang, L. Zhou, and B. Wang (2025) MTalk-Bench: evaluating speech-to-speech models in multi-turn dialogues via arena-style and rubrics protocols . arXiv preprint arXiv:2508.18240 . Cited by: §1 , §3 , §4 .

[19] S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro (2025) Audio flamingo 2: an audio-language model with long-audio understanding and expert reasoning abilities . External Links: 2503.03983 , Link Cited by: §4 .

[20] X. Guo, U. Tyagi, A. Gosai, P. Vergara, J. Park, E. G. H. Montoya, C. B. C. Zhang, B. Hu, Y. He, B. Liu, and R. S. Srinivasa (2025) Beyond seeing: evaluating multimodal llms on tool-enabled image perception, transformation, and reasoning . External Links: 2510.12712 , Link Cited by: §A.4 , §3 .

[21] P. He, Z. Wen, Y. Wang, Y. Wang, X. Liu, J. Huang, Z. Lei, Z. Gu, X. Jin, J. Yang, K. Li, Z. Liu, W. Li, C. Wang, C. He, and L. Zhang (2025) AudioMarathon: a comprehensive benchmark for long-context audio understanding and efficiency in audio LLMs . arXiv preprint arXiv:2510.07293 . Cited by: §4 .

[22] M. Hill-Smith, G. Cameron, and A. Analysis (2024) Evaluating audio reasoning with big bench audio . Note: https://huggingface.co/blog/big-bench-audio-release Hugging Face blog Cited by: §1 , §4 .

[23] W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed (2021) HuBERT: self-supervised speech representation learning by masked prediction of hidden units . External Links: 2106.07447 , Link Cited by: §A.5 .

[24] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. (2024) Gpt-4o system card . arXiv preprint arXiv:2410.21276 . Cited by: Table 2 .

[25] S. Ji, Y. Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Cheng, X. Yang, Z. Wang, Q. Yang, J. Li, Y. Jiang, J. He, Y. Chu, J. Xu, and Z. Zhao (2024) WavChat: a survey of spoken dialogue models . External Links: 2411.13577 , Link Cited by: §1 .

[26] K. Jokinen and M. McTear (2010) Spoken dialogue systems . Synthesis lectures on human language technologies , Springer International Publishing , Cham ( en ). Cited by: §1 .

[27] S. Kumar, Š. Sedláček, V. Lokegaonkar, F. López, W. Yu, N. Anand, H. Ryu, L. Chen, M. Plička, M. Hlaváček, et al. (2025) MMAU-pro: a challenging and comprehensive benchmark for holistic evaluation of audio general intelligence . arXiv preprint arXiv:2508.13992 . Cited by: §4 .

[28] P. Laban, H. Hayashi, Y. Zhou, and J. Neville (2025) LLMs get lost in multi-turn conversation . arXiv preprint arXiv:2505.06120 . Cited by: §1 , §4 .

[29] A. H. Liu, A. Ehrenberg, A. Lo, C. Denoix, C. Barreau, G. Lample, J. Delignon, K. R. Chandu, P. von Platen, P. R. Muddireddy, et al. (2025) Voxtral . arXiv preprint arXiv:2507.13264 . Cited by: Table 2 .

[30] K. Lu, Z. Chen, S. Fu, C. H. Yang, J. Balam, B. Ginsburg, Y. F. Wang, and H. Lee (2024) DeSTA2: developing instruction-following speech language model without speech instruction-tuning data . arXiv preprint arXiv:2409.20007 . Cited by: §2.1 .

[31] K. Lu, C. Kuan, and H. Lee (2025) Speech-ifeval: evaluating instruction-following and quantifying catastrophic forgetting in speech-aware language models . arXiv preprint arXiv:2505.19037 . Cited by: §2.1 .

[32] Z. Ma, Y. Ma, Y. Zhu, C. Yang, Y. Chao, R. Xu, et al. (2025) MMAR: a challenging benchmark for deep reasoning in speech, audio, music, and their mix . arXiv preprint arXiv:2505.13032 . Cited by: §4 .

[33] L. McInnes, J. Healy, and J. Melville (2020) UMAP: uniform manifold approximation and projection for dimension reduction . External Links: 1802.03426 , Link Cited by: §A.5 .

[34] T. A. Nguyen, E. Kharitonov, J. Copet, Y. Adi, W. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, et al. (2023) Generative spoken dialogue language modeling . Transactions of the Association for Computational Linguistics 11 , pp. 250–266 . Cited by: §1 .

[35] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2022) Robust speech recognition via large-scale weak supervision . In International Conference on Machine Learning , External Links: Link Cited by: §1 .

[36] A. Rouditchenko, S. Bhati, E. Araujo, S. Thomas, H. Kuehne, R. Feris, and J. Glass (2025) Omni-r1: do you really need audio to fine-tune your audio llm? . External Links: 2505.09439 , Link Cited by: §3.4 .

[37] M. Sharma, C. B. C. Zhang, C. Bandi, C. Wang, A. Aich, H. Nghiem, T. Rabbani, Y. Htet, B. Jang, S. Basu, A. Balwani, D. Peskoff, M. Ayestaran, S. M. Hendryx, B. Kenstler, and B. Liu (2025) ResearchRubrics: a benchmark of prompts and rubrics for evaluating deep research agents . External Links: 2511.07685 , Link Cited by: §3 , §3.2 .

[38] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, and J. Wei (2022) Challenging big-bench tasks and whether chain-of-thought can solve them . arXiv preprint arXiv:2210.09261 . Cited by: §4 .

[39] G. Team (2025) Gemma 3n . Google . External Links: Link Cited by: Table 2 .

[40] B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen (2025) AudioBench: a universal benchmark for audio large language models . In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics , Cited by: §1 , §4 .

[41] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous (2017) Tacotron: towards end-to-end speech synthesis . External Links: 1703.10135 , Link Cited by: §1 .

[42] K. Wataoka, T. Takahashi, and R. Ri (2025) Self-preference bias in llm-as-a-judge . External Links: 2410.21819 , Link Cited by: §3.3 .

[43] L. Xiaomi (2025) MiMo-audio: audio language models are few-shot learners . External Links: Link Cited by: §3.1 , Table 2 .

[44] J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, et al. (2025) Qwen2. 5-omni technical report . arXiv preprint arXiv:2503.20215 . Cited by: Table 2 .

[45] J. Xu, Z. Guo, H. Hu, Y. Chu, X. Wang, J. He, Y. Wang, X. Shi, T. He, X. Zhu, Y. Lv, Y. Wang, D. Guo, H. Wang, L. Ma, P. Zhang, X. Zhang, H. Hao, Z. Guo, B. Yang, B. Zhang, Z. Ma, X. Wei, S. Bai, K. Chen, X. Liu, P. Wang, M. Yang, D. Liu, X. Ren, B. Zheng, R. Men, F. Zhou, B. Yu, J. Yang, L. Yu, J. Zhou, and J. Lin (2025) Qwen3-omni technical report . arXiv preprint arXiv:2509.17765 . Cited by: §1 , Table 2 .

[46] I. Yosha, G. Maimon, and Y. Adi (2025) StressTest: can your speech lm handle the stress? . External Links: 2505.22765 , Link Cited by: §4 .

[47] D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu (2023) SpeechGPT: empowering large language models with intrinsic cross-modal conversational abilities . External Links: 2305.11000 , Link Cited by: §1 .

[48] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging llm-as-a-judge with mt-bench and chatbot arena . External Links: 2306.05685 , Link Cited by: §3.3 .

Appendix A Appendix

附录 A 附录

A.1 Sample Tasks and Model Failures

A.1 样本任务与模型失败

A.2 Sample Human-Friendly Blueprints

A.2 面向人类的蓝图样本

A.3 LLM-as-a-Judge Prompt

A.3 大模型评委提示

A.1.4 Voice Editing

We use the following prompt for our LLM-as-a-Judge rubric grading, adapted from Arora et al., 2025 [4]. We provide further implementation details in our dataset card for reproducibility of our evaluation pipeline.

A.1.4 语音编辑

我们使用以下提示来做大模型评委的细则打分，改编自 Arora 等人，2025 [4]。更多实现细节见我们的数据卡，以便复现评测流水线。

A.4 Evaluation Metrics

A.4 评测指标

In this section, we formally define the two primary metrics used to evaluate model performance on Audio MultiChallenge: Average Pass Rate (APR) and Average Rubric Score (ARS) as adapted from Guo et. al, 2025 [20]. Let $\mathcal{T}=\{1,2,\dots,N\}$ denote the set of tasks in the benchmark. For each task $i$, there exists a set of atomic rubric criteria indexed by $j\in\{1,2,\dots,N_{i}\}$. Let $\mathbb{1}_{r_{ij}}\in\{0,1\}$ be the binary indicator representing whether the model’s response satisfies criterion $j$ for task $i$.

本节形式化定义用于评测模型在 Audio MultiChallenge 上表现的两项主要指标：平均通过率（APR）和平均细则得分（ARS），二者改编自 Guo 等人，2025 [20]。令 $\mathcal{T}=\{1,2,\dots,N\}$ 表示基准中的任务集合。对每项任务 $i$，存在一组由 $j\in\{1,2,\dots,N_{i}\}$ 索引的原子细则标准。令 $\mathbb{1}_{r_{ij}}\in\{0,1\}$ 为二元指示，表示模型回复是否满足任务 $i$ 的第 $j$ 条标准。

Average Pass Rate (APR).

The APR is a strict metric measuring the percentage of tasks where the model satisfies all associated rubric criteria. We define a binary pass indicator $p_{i}$ for task $i$, where $p_{i}=1$ if $\prod_{j=1}^{N_{i}}\mathbb{1}_{r_{ij}}=1$, and $0$ otherwise. The overall APR is the mean of these binary indicators:

平均通过率（APR）。

APR 是一项严格指标，衡量模型满足全部相关细则标准的任务百分比。我们为任务 $i$ 定义二元通过指示 $p_{i}$：若 $\prod_{j=1}^{N_{i}}\mathbb{1}_{r_{ij}}=1$，则 $p_{i}=1$，否则为 0。总体 APR 是这些二元指示的均值：

$$\text{APR}=\frac{1}{N}\sum_{i=1}^{N}p_{i}=\frac{1}{N}\sum_{i=1}^{N}\left(\prod_{j=1}^{N_{i}}\mathbb{1}_{r_{ij}}\right) \tag{1}$$

Average Rubric Score (ARS).

To provide a more granular assessment of partial success, we utilize the ARS. This metric calculates the average proportion of rubric criteria satisfied per task, treating all rubrics within a task with equal weight. We define the task-level score $s_{i}\in[0,1]$ as:

平均细则得分（ARS）。

为更细致地评估部分成功，我们使用 ARS。该指标计算每项任务中被满足细则标准的平均比例，任务内各条细则等权。我们将任务级得分 $s_{i}\in[0,1]$ 定义为：

$$s_{i}=\frac{1}{N_{i}}\sum_{j=1}^{N_{i}}\mathbb{1}_{r_{ij}} \tag{2}$$

Similar to APR, the final ARS for the benchmark is the mean of these task-level scores:

与 APR 类似，基准的最终 ARS 是这些任务级得分的均值：

$$\text{ARS}=\frac{1}{N}\sum_{i=1}^{N}s_{i} \tag{3}$$

A.5 Dataset Diversity

A.5 数据多样性

Figure 10: HuBERT features UMAP

图 10：HuBERT 特征的 UMAP

We assess acoustic diversity using self-supervised speech representations from HuBERT [23] . For each task, we randomly sample one utterance (k = 1) to avoid bias from variable task lengths, extract mean-pooled embeddings from the final transformer layer, and project them to two dimensions using UMAP [33] with cosine distance. We observe a noticeable dispersion consistent with heterogeneous speaker and style characteristics across tasks. Quantitatively, the average pairwise cosine distance between embeddings is 0.57, indicating a moderate degree of acoustic variability in our dataset across different speakers and audio-cues.

我们用 HuBERT [23] 的自监督语音表示来评估声学多样性。对每项任务，我们随机抽取一条话语（k = 1），以避免任务长度不等带来的偏差，从最后一层 Transformer 提取均值池化嵌入，再用余弦距离的 UMAP [33] 投影到二维。我们观察到明显的分散，这与任务之间异质的说话人和风格特征相一致。定量上看，嵌入之间的平均成对余弦距离为 0.57，表明本数据集在不同说话人和音频线索上具有中等程度的声学变化。

A.6 Synthetic Audio LM Failure Discovery

A.6 合成 Audio LM 失败发现

1: Axis List $\mathcal{A}$, Topic List $\mathcal{T}$, Persona List $\mathcal{P}$, Planner $M_{plan}$, Tester $M_{tester}$, Target $M_{target}$, TTS Model $\text{TTS}(\cdot)$
2: A human-friendly blueprint $B_{final}$ targeting axis $a$
3: $a\sim\text{Uniform}(\mathcal{A})$ $\triangleright$ Sample targeted axis
4: $t,p,\text{max\_turns}\sim\text{Sample}(\mathcal{T},\mathcal{P},\text{TURN\_OPTIONS})$
5: $H_{target}\leftarrow\emptyset$; $H_{tester}\leftarrow\emptyset$ $\triangleright$ Initialize Empty Conversation Histories
6: $B\leftarrow M_{plan}.\text{InitialBlueprint}(a,t,p,\text{max\_turns})$
7: $\text{break\_found}\leftarrow\text{False}$
8: while $\neg\text{break\_found}$ and $|H_{target}|<\text{max\_turns}$ do
9:   $u_{text}\leftarrow M_{tester}.\text{GenerateTurn}(H_{tester},B)$ $\triangleright$ Tester generates turn based on current blueprint
10:   $u_{audio}\leftarrow\text{TTS}(u_{text})$
11:   $H_{target}\leftarrow H_{target}\cup\{u_{audio}\}$
12:   $r_{text}\leftarrow M_{target}(H_{target})$ $\triangleright$ Audio LM generates text output
13:   $H_{target}\leftarrow H_{target}\cup\{r_{text}\}$
14:   $H_{tester}\leftarrow H_{tester}\cup\{u_{text}\/,r_{text}\}$
15:   $B,\text{break\_found},\text{break\_reason}\leftarrow M_{plan}.\text{UpdatePlan}(H_{tester},B,a)$
16: end while
17: if break_found then
18:   $\triangleright$ Distill raw history into a clear guide for humans
19:   $B_{final}\leftarrow M_{plan}.\text{SummarizeForHumans}(H_{tester},B,\text{break\_reason})$
20:   return $B_{final}$
21: end if
22: return Null

Algorithm 1 Synthetic Audio LM Failure Discovery

1: 轴列表 $\mathcal{A}$，主题列表 $\mathcal{T}$，人设列表 $\mathcal{P}$，规划器 $M_{plan}$，测试器 $M_{tester}$，目标 $M_{target}$，TTS 模型 $\text{TTS}(\cdot)$
2: 针对轴 $a$ 的一份面向人类的蓝图 $B_{final}$
3: $a\sim\text{Uniform}(\mathcal{A})$ $\triangleright$ 采样目标轴
4: $t,p,\text{max\_turns}\sim\text{Sample}(\mathcal{T},\mathcal{P},\text{TURN\_OPTIONS})$
5: $H_{target}\leftarrow\emptyset$；$H_{tester}\leftarrow\emptyset$ $\triangleright$ 初始化空的对话历史
6: $B\leftarrow M_{plan}.\text{InitialBlueprint}(a,t,p,\text{max\_turns})$
7: $\text{break\_found}\leftarrow\text{False}$
8: while $\neg\text{break\_found}$ and $|H_{target}|<\text{max\_turns}$ do
9:   $u_{text}\leftarrow M_{tester}.\text{GenerateTurn}(H_{tester},B)$ $\triangleright$ 测试器根据当前蓝图生成一轮
10:   $u_{audio}\leftarrow\text{TTS}(u_{text})$
11:   $H_{target}\leftarrow H_{target}\cup\{u_{audio}\}$
12:   $r_{text}\leftarrow M_{target}(H_{target})$ $\triangleright$ Audio LM 生成文本输出
13:   $H_{target}\leftarrow H_{target}\cup\{r_{text}\}$
14:   $H_{tester}\leftarrow H_{tester}\cup\{u_{text}\/,r_{text}\}$
15:   $B,\text{break\_found},\text{break\_reason}\leftarrow M_{plan}.\text{UpdatePlan}(H_{tester},B,a)$
16: end while
17: if break_found then
18:   $\triangleright$ 把原始历史提炼成给人类的清晰指南
19:   $B_{final}\leftarrow M_{plan}.\text{SummarizeForHumans}(H_{tester},B,\text{break\_reason})$
20:   return $B_{final}$
21: end if
22: return Null

算法 1 合成 Audio LM 失败发现

A.7 Conversation Topics

A.7 对话主题

Our multi-agen Audio LM failure discovery loop is seeded with a randomly sampled conversation topic to its Planner Agent. We showcase demonstrative samples of these topic seeds below.

我们的多智能体 Audio LM 失败发现循环，会为规划智能体随机采样一个对话主题作为种子。下面展示这些主题种子的示例。

Voice Editing
Daily Schedules	DoctorAppointment	User mentions a doctor appointment conversationally, correcting details mid-speech.
Daily Schedules	AppointmentMixup	User briefly mixes details between appointments, then immediately self-corrects.
Daily Schedules	MeetingCoordination	User talks through meeting details naturally, making spontaneous corrections.
Daily Schedules	TimeCorrectionMidSpeech	User states a time but immediately corrects it while speaking.
Daily Schedules	DurationAdjustment	User estimates a duration aloud, then revises it mid-thought.
Daily Schedules	TimeOfDaySwitch	User specifies morning/afternoon but quickly corrects themselves.
Daily Schedules	TimingAdjustment	User mentions schedule timing and adjusts it immediately.
Numbers	PhoneNumberCorrection	User shares a phone number conversationally with immediate self-corrections.
Numbers	NumberSegmentFix	User gives a full number, then corrects a mistaken segment.
Numbers	ContactDetailsBuildup	User provides contact details incrementally across turns.
Numbers	TrackingNumberRecall	User recalls a tracking number while checking it, correcting digits aloud.
Numbers	OrderReferenceBuildup	User shares order details gradually, adding or fixing parts.
Numbers	ConfirmationCodeMixup	User says a code but corrects characters mid-speech.
Numbers	PolicyNumberCorrection	User reads a policy number and fixes errors as they notice them.
Numbers	AccountInfoSharing	User mentions account details and immediately corrects numbers or dates.
Numbers	DocumentNumberRecall	User recalls document numbers while verifying, correcting sequences aloud.
Shopping Lists	GroceryListCorrection	User talks through a shopping list with natural self-corrections.
Shopping Lists	ShoppingListAdjustment	User adds, removes, or swaps items mid-list.
Shopping Lists	PriceCorrection	User states prices conversationally and corrects amounts immediately.
Message Drafts	TextMessageFix	User dictates a message and revises wording mid-sentence.
Message Drafts	RecipientCorrection	User starts a message, then immediately switches the recipient.
Message Drafts	MessageDrafting	User builds message content naturally over multiple turns.
Event Details	PartyPlanning	User discusses event details with multiple spontaneous corrections.
Event Details	GuestListAdjustment	User lists attendees and immediately corrects names.
Event Details	EventTimeChange	User states an event time and fixes it right away.
Event Details	ItemSwapping	User names an item but replaces it mid-thought.
To Do Lists	TodoListCorrections	User describes tasks conversationally, correcting actions or wording.
To Do Lists	DeadlineAdjustment	User mentions a due date and immediately revises it.
To Do Lists	MultiTaskBuildup	User thinks through tasks incrementally, adding more as remembered.
Travel Itineraries	FlightDetailCorrection	User discusses flight details and corrects dates or times mid-speech.
Travel Itineraries	TravelDateSwitch	User states travel dates but catches an error immediately.
Travel Itineraries	HotelReservation	User talks through hotel plans with natural corrections.
Travel Itineraries	FlightInfoSharing	User shares flight info conversationally, fixing numbers or dates.
Travel Itineraries	DestinationSwitch	User names a destination, then corrects it immediately.
Travel Itineraries	MultiStopAdjustment	User describes a route and adds or changes stops mid-speech.
Instruction Retention
Response Structure	IncludeFunFact	Include a fun fact related to the topic in each response.
Response Structure	RephraseQuestions	Clarify the user request in every response before responding.
Response Structure	LimitedWords	Answer every question using only a set amount of words.
Response Structure	CounterArgumentIncluded	Begin every response with a specific counterargument.
ResponseStyle	StylisticChoice	Adhere to specific response styles (e.g., Haiku, Poem).
ResponseStyle	NoPersonalPronouns	Do not use any personal pronouns.
Personalization	BeginnerLevel	Treat the user as a beginner when explaining concepts.
Personalization	RephraseQuestions	Rephrase unclear questions to confirm intent.
Personalization	ExampleHeavy	Use more examples to support explanations.
Personalization	ConfirmationGating	Ask for confirmation before proceeding with assumptions.
Role Play	FitnessInstructor	Adopt the persona of a fitness instructor.
Role Play	Tutor	Adopt the persona of a tutor.
Role Play	MotivationalCoach	Adopt the persona of a motivational coach.
Role Play	FictionalCharacter	Adopt the persona of a specific fictional character.
Role Play	HistoricFigure	Adopt the persona of a specific historic figure.
Role Play	Interviewer	Adopt the persona of an interviewer.
Content Restrictions	AvoidSpecificWords	Avoid using a particular word provided in context.
Content Restrictions	NoNegatives	Never use negative words or phrases.
Inference Memory
Food & Drink	DietaryRestrictions	Remember user dietary restrictions (e.g., allergies) for future suggestions.
Food & Drink	FavoriteCuisine	Recall user’s favorite cuisine for restaurant/dish recommendations.
Food & Drink	SpecificTastePreference	Remember specific likes/dislikes (e.g., dislikes bitter foods).
Schedule & Time	EventDate	Recall specific dates mentioned for events when planning.
Schedule & Time	TimeConflictManagement	Adjust suggestions based on previously mentioned availability or conflicts.
Schedule & Time	RecurringEventRecognition	Implicitly refer to recurring events (e.g., weekly meetings) in future turns.
Personal Details	RelationshipDetails	Recall details about user’s relationships (e.g., partner’s preference).
Personal Details	GiftPreferences	Incorporate stated gift preferences into future options.
Personal Details	ImportantDate	Remember significant dates and offer advice closer to that date.
Location & Travel	TravelDestination	Recall planned destinations for activity recommendations.
Location & Travel	DistanceConsideration	Remember distance preferences (e.g., walking distance).
Location & Travel	PreviousTripComparison	Implicitly refer back to past trips when comparing destinations.
Work & Projects	ProjectDeadlines	Recall deadlines for time management advice.
Work & Projects	TaskCompletionStatus	Recollect status updates when suggesting next steps.
Work & Projects	CollaborationDetails	Refer to specific colleagues mentioned when offering advice.
Hobby & Interest	HobbyDetails	Recall hobbies (e.g., photography) for related suggestions.
Hobby & Interest	OngoingProject	Refer to personal projects (e.g., building a model) when offering ideas.
Hobby & Interest	SeasonalActivityPreference	Recollect seasonal preferences when the season approaches.
Shopping & Purchase	PreferredBrands	Remember brand preferences for future suggestions.
Shopping & Purchase	PreviousPurchaseFeedback	Recall feedback on past purchases for similar recommendations.
Shopping & Purchase	PriceSensitivity	Keep budget/price preferences in mind for future recommendations.
Emotional State	EmotionalState	Implicitly reference user’s mood (e.g., stressed) in supportive suggestions.
Emotional State	MentalHealthGoals	Recall goals (e.g., reducing anxiety) for wellness suggestions.
Emotional State	RecentEmotionalExperience	Recollect recent experiences to offer empathy in future conversations.
Behavioral Patterns	RecurringHesitation	Recall frequent hesitation points to offer reassurance.
Behavioral Patterns	ProblemSolvingApproach	Remember preferred solution style when giving advice.
Behavioral Patterns	ImplicitKnowledgeGap	Recall common knowledge gaps to clarify proactively.
Self Coherence
Calculations	BudgetOrCostEstimates	Model must not provide conflicting financial figures later in the conversation.
Calculations	TimeCalculations	Model must not give contradictory time estimates later.
Factual Statements	HistoricalDates	Model must not contradict specific dates provided for historical events.
Factual Statements	ScientificFacts	Model must ensure scientific facts are not contradicted in future statements.
Factual Statements	GeographicalDetails	Model must not contradict geographical details (e.g., population) later.
Recommendations	ProductRecommendations	Model must not contradict a product recommendation by suggesting something totally different later without cause.
Recommendations	DietOrHealthAdvice	Model must avoid contradicting previously given dietary or health advice.
Recommendations	TravelAdvice	Model must avoid contradictory travel suggestions.
Contextualization	LanguageAndTone	Model must avoid contradictory shifts in tone (e.g., formal to friendly) unless prompted.
Reasoning	CausalReasoning	Model must not contradict previously stated cause-and-effect explanations later.
Reasoning	ArgumentativeCoherence	Model must not undermine earlier arguments with inconsistent logic later.
Reasoning	ProConSynthesis	Model must not reverse previously stated trade-offs without justification.
Plan	AdherenceToProposedPlan	Model must not forget details from a proposed plan.
Plan Contradictions	GivenAdviceConsistency	Model must not give advice that conflicts with earlier guidance.
Personas	RoleOrPersonaAdherence	Model must not break from an established role or persona mid-conversation.
Personas	ScenarioConsistency	Model must not contradict established scenario assumptions later.
Personas	AnalogyConsistency	Model must not change analogy mappings inconsistently once introduced.

Table 6: Taxonomy of Topics and Instructions

语音编辑
日程安排	DoctorAppointment	用户在对话中提到医生预约，并在说话中途修正细节。
日程安排	AppointmentMixup	用户短暂把不同预约的细节弄混，随即自我纠正。
日程安排	MeetingCoordination	用户自然地谈会议细节，并做自发修正。
日程安排	TimeCorrectionMidSpeech	用户说出一个时间，但立刻在说话过程中改正。
日程安排	DurationAdjustment	用户口头估计一段时间，然后在思考中途改口。
日程安排	TimeOfDaySwitch	用户指定上午或下午，但很快纠正自己。
日程安排	TimingAdjustment	用户提到日程时间并立即调整。
数字	PhoneNumberCorrection	用户以对话方式报电话号码，并立即自我纠正。
数字	NumberSegmentFix	用户给出完整号码，然后改正其中一段。
数字	ContactDetailsBuildup	用户分多轮逐步提供联系方式。
数字	TrackingNumberRecall	用户一边核对一边回忆运单号，并口头改正数字。
数字	OrderReferenceBuildup	用户逐步分享订单细节，补充或修正其中部分。
数字	ConfirmationCodeMixup	用户说出一个代码，但在说话中途改正字符。
数字	PolicyNumberCorrection	用户读保单号，并在发现错误时改正。
数字	AccountInfoSharing	用户提到账户信息，并立即改正数字或日期。
数字	DocumentNumberRecall	用户一边核验一边回忆文件编号，并口头改正序列。
购物清单	GroceryListCorrection	用户边说购物清单边做自然的自我纠正。
购物清单	ShoppingListAdjustment	用户在列举过程中增删或替换物品。
购物清单	PriceCorrection	用户以对话方式报价格，并立即改正金额。
消息草稿	TextMessageFix	用户口述一条消息，并在句中修改措辞。
消息草稿	RecipientCorrection	用户开始写一条消息，随即改换收件人。
消息草稿	MessageDrafting	用户在多轮中自然地组织消息内容。
活动细节	PartyPlanning	用户讨论活动细节，并多次自发修正。
活动细节	GuestListAdjustment	用户列举出席者并立即改正姓名。
活动细节	EventTimeChange	用户说出活动时间并马上改正。
活动细节	ItemSwapping	用户说出一件物品，随即在思考中途替换。
待办清单	TodoListCorrections	用户以对话方式描述任务，并改正动作或措辞。
待办清单	DeadlineAdjustment	用户提到截止日期并立即改口。
待办清单	MultiTaskBuildup	用户边想边增加任务，想起一件补一件。
旅行行程	FlightDetailCorrection	用户讨论航班细节，并在说话中途改正日期或时间。
旅行行程	TravelDateSwitch	用户说出旅行日期，但立刻发现错误并改正。
旅行行程	HotelReservation	用户谈酒店安排，并做自然修正。
旅行行程	FlightInfoSharing	用户以对话方式分享航班信息，并改正数字或日期。
旅行行程	DestinationSwitch	用户说出目的地，随即立即改正。
旅行行程	MultiStopAdjustment	用户描述路线，并在说话中途增加或更改经停点。
指令保持
回复结构	IncludeFunFact	每条回复都要包含一条与主题相关的有趣事实。
回复结构	RephraseQuestions	每条回复在作答前先澄清用户请求。
回复结构	LimitedWords	每个问题都只用规定字数作答。
回复结构	CounterArgumentIncluded	每条回复都以一条指定的反驳开头。
回复风格	StylisticChoice	遵守特定的回复风格（例如俳句、诗歌）。
回复风格	NoPersonalPronouns	不要使用任何人称代词。
个性化	BeginnerLevel	在解释概念时把用户当作初学者。
个性化	RephraseQuestions	改述不清楚的问题以确认意图。
个性化	ExampleHeavy	用更多例子来支撑解释。
个性化	ConfirmationGating	在按假定继续之前先征求确认。
角色扮演	FitnessInstructor	采用健身教练的人设。
角色扮演	Tutor	采用辅导老师的人设。
角色扮演	MotivationalCoach	采用激励教练的人设。
角色扮演	FictionalCharacter	采用某个特定虚构人物的人设。
角色扮演	HistoricFigure	采用某个特定历史人物的人设。
角色扮演	Interviewer	采用采访者的人设。
内容限制	AvoidSpecificWords	避免使用上下文中给出的某个特定词。
内容限制	NoNegatives	绝不使用否定词或否定说法。
推理记忆
饮食	DietaryRestrictions	记住用户的饮食限制（例如过敏），供后续建议使用。
饮食	FavoriteCuisine	回想用户最喜欢的菜系，用于餐厅或菜品推荐。
饮食	SpecificTastePreference	记住具体的好恶（例如不喜欢苦的食物）。
日程与时间	EventDate	在做规划时回想提到过的具体活动日期。
日程与时间	TimeConflictManagement	根据先前提到的空闲或冲突来调整建议。
日程与时间	RecurringEventRecognition	在后续轮次中隐含地提及例行活动（例如每周例会）。
个人细节	RelationshipDetails	回想用户人际关系的细节（例如伴侣的偏好）。
个人细节	GiftPreferences	把已说明的礼物偏好纳入后续选项。
个人细节	ImportantDate	记住重要日期，并在临近时提供建议。
地点与旅行	TravelDestination	回想计划中的目的地，用于活动推荐。
地点与旅行	DistanceConsideration	记住距离偏好（例如步行可及）。
地点与旅行	PreviousTripComparison	在比较目的地时隐含地回顾以往旅行。
工作与项目	ProjectDeadlines	回想截止日期，用于时间管理建议。
工作与项目	TaskCompletionStatus	在建议下一步时回想进度更新。
工作与项目	CollaborationDetails	在给建议时提及先前说过的具体同事。
爱好与兴趣	HobbyDetails	回想爱好（例如摄影），用于相关建议。
爱好与兴趣	OngoingProject	在提供想法时提及个人项目（例如做模型）。
爱好与兴趣	SeasonalActivityPreference	在相应季节临近时回想季节性偏好。
购物与购买	PreferredBrands	记住品牌偏好，供后续建议使用。
购物与购买	PreviousPurchaseFeedback	回想对以往购买的反馈，用于类似推荐。
购物与购买	PriceSensitivity	在后续推荐中记住预算或价格偏好。
情绪状态	EmotionalState	在支持性建议中隐含地提及用户心情（例如压力大）。
情绪状态	MentalHealthGoals	回想目标（例如减轻焦虑），用于健康建议。
情绪状态	RecentEmotionalExperience	回想近期经历，以便在后续对话中表达共情。
行为模式	RecurringHesitation	回想经常犹豫的地方，以便给予安抚。
行为模式	ProblemSolvingApproach	在给建议时记住偏好的解决风格。
行为模式	ImplicitKnowledgeGap	回想常见的知识缺口，以便主动澄清。
自我连贯
计算	BudgetOrCostEstimates	模型随后不得给出相互冲突的财务数字。
计算	TimeCalculations	模型随后不得给出相互矛盾的时间估计。
事实陈述	HistoricalDates	模型不得与先前给出的历史事件具体日期相矛盾。
事实陈述	ScientificFacts	模型必须确保科学事实在后续陈述中不被推翻。
事实陈述	GeographicalDetails	模型随后不得与地理细节（例如人口）相矛盾。
推荐	ProductRecommendations	模型不得在没有缘由的情况下，后来推荐完全不同的产品，从而否定先前建议。
推荐	DietOrHealthAdvice	模型必须避免与先前给出的饮食或健康建议相矛盾。
推荐	TravelAdvice	模型必须避免相互矛盾的旅行建议。
情境化	LanguageAndTone	除非受到提示，模型必须避免语气上自相矛盾的转变（例如从正式变为亲热）。
推理	CausalReasoning	模型随后不得与先前陈述的因果解释相矛盾。
推理	ArgumentativeCoherence	模型随后不得用不一致的逻辑削弱先前论点。
推理	ProConSynthesis	模型不得在没有理由的情况下推翻先前陈述的利弊权衡。
计划	AdherenceToProposedPlan	模型不得忘记已提出计划中的细节。
计划矛盾	GivenAdviceConsistency	模型不得给出与先前指导冲突的建议。
人设	RoleOrPersonaAdherence	模型不得在对话中途脱离已确立的角色或人设。
人设	ScenarioConsistency	模型随后不得与已确立的情境假定相矛盾。
人设	AnalogyConsistency	模型一旦引入类比映射，就不得不一致地更改它。

表 6：主题与指令的分类体系