Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
摘要1 引言2 相关工作2.1 语音语言模型Thinker-Talker 架构。原生端到端架构。2.2 全双工语音交互全双工对话系统。时分复用(Time-Division Multiplexing, TDM)。信道分复用(Channel-Division Multiplexing, CDM)。3 分层声学-语义建模3.1 模态干扰的优化动力学优化方向分裂。语义稀释。3.2 核心架构创新:分离与对齐共享主干基础。分层参数分离。语义对齐通道。4 实验4.1 基线系统级全双工模型原生全双工模型4.2 训练数据构建4.3 实现细节4.4 口语问答指标。结果。消融。讨论。4.5 全双工对话指标。结果。延迟。4.6 语音生成指标。结果。5 几何分析的推广6 结论局限性致谢参考文献附录 A 层数消融附录 B 案例研究B 案例研究:插话与附和附录 C 错误分析附录 D 全局梯度影响分析附录 E 实时推理算法附录 F 数据合成流水线细节F.1 智能体架构F.2 交互行为建模插话生成。附和注入。\n\n附和在后处理阶段按概率注入($p=0.5$)。F.3 质量控制F.4 提示模板Prompt 1:用户角色扮演指令Prompt 2:插话规划(动机与位置)Prompt 3:插话语句生成Prompt 4:用户附和生成Prompt 5:最终对话质量评审
返回论文列表
目录

Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

收录于 ACL 2026 Outstanding Paper被引 5 次
打开论文来源

Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

分层声学-语义建模:面向全双工语音语言模型的模态分离与语义连贯

Zhenyu Liu^{1,2}, Xuanyu Zhang^{1,2}, Yunxin Li^{1,2}, Qixun Teng^{1}, Shenyuan Jiang^{1}, Haolan Chen, Minjun Zhao, Fanbo Meng, Yu Xu, Yancheng He, Baotian Hu^{1,2,*}, Haizhou Li^{2,3}, Min Zhang^{1,2}

^{1}School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China

^{2}Center for Language, Intelligence and Machines, Shenzhen Loop Area Institute, Shenzhen, China

^{3}School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China

Correspondence: liuzhenyuhit@gmail.com, 25s151197@stu.hit.edu.cn, liyx@hit.edu.cn
hubaotian@hit.edu.cn, zhangmin2021@hit.edu.cn, haizhouli@cuhk.edu.cn

  • Corresponding author.

Abstract

摘要

Developing seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge and long-standing goal for the speech and NLP community. Despite notable progress, recent endeavors are fundamentally constrained by severe modality interference, which causes substantial knowledge degradation and compromises semantic integrity — ultimately making full-duplex SLMs feel unnatural and unintelligent. In this paper, through an exhaustive fine-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises from inherent gradient conflicts between acoustic and semantic modeling when the two modalities are forced to share a deep parameter space. Guided by this key insight, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic alignment channel. Extensive experiments on multiple full-duplex benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both speech intelligence (+7.4% on Spoken QA) and full-duplex interaction fluidity (+28.5% on FullDuplexBench 1.5) without compromising inference efficiency. To the best of our knowledge, this work is the first to achieve two key advances: 1) uncovering and elucidating the root cause of modality interference in full-duplex SLMs, and 2) designing an elegant hierarchical model together with a practical solution for seamless, high-performance, native intelligent full-duplex SLMs.

开发流畅、高性能、原生智能的全双工语音语言模型(Spoken Language Models, SLMs),仍是语音与自然语言处理领域的一项关键挑战,也是长期目标。尽管已有可观进展,近期工作在根本上仍受严重的模态干扰制约:它会带来明显的知识退化,并损害语义完整性——最终让全双工 SLM 显得不自然、也不够智能。本文通过对模型优化动力学的细致分析,找出了这一性能下降的根源:当声学建模与语义建模被迫共享深层参数空间时,二者之间存在固有的梯度冲突,模态干扰由此产生。基于这一认识,我们提出 Lychee-FD,一种用于缓解模态干扰的原生端到端全双工框架。我们给出分层参数分离策略,在深层把相互冲突的模态拆开,同时用专门的语义对齐通道保持跨模态连贯。在多个全双工基准上的大量实验表明,该方法显著推进了现有水平:口语问答的语音智力提升 7.4%,FullDuplexBench 1.5 上的全双工交互流畅度提升 28.5%,且不牺牲推理效率。据我们所知,这项工作首次同时做到两点:1)揭示并阐明全双工 SLM 中模态干扰的根源;2)设计一种简洁的分层模型,并为流畅、高性能、原生智能的全双工 SLM 提供可行方案。

1 Introduction

1 引言

The rapid evolution of Large Language Models (LLMs) has fundamentally reshaped our daily lives, establishing them as ubiquitous assistants capable of complex reasoning and instruction following. Within this landscape, Spoken Language Models (SLMs) represent a significant paradigm shift from text-based to voice-based interaction. Despite recent advancements in Omni-modal models capable of seamless voice interaction (Hurst et al., 2024; Zhan et al., 2024; Li et al., 2025b; Kaplan et al., 2020; Hoffmann et al., 2022; Xu et al., 2023; Gao et al., 2025), a critical disparity remains between artificial agents and authentic human conversation. Currently, most voice interactions are constrained to a rigid half-duplex mode, where the system strictly alternates between listening and speaking in a sequential manner. In contrast, authentic human conversation is inherently full-duplex, requiring the ability to continuously process incoming audio streams while concurrently generating responses (Sacks et al., 1974; Fu et al., 2025; Ji et al., 2024). The forced turn-taking of half-duplex systems disrupts the fluidity of interaction, creating an artificial barrier between user and agent.

大语言模型(Large Language Models, LLMs)的迅速发展,已经深刻改变了日常生活,使它们成为能够进行复杂推理和指令跟随的普遍助手。在这一图景中,语音语言模型(Spoken Language Models, SLMs)标志着从文本交互转向语音交互的重要范式变化。尽管近期全模态模型已能提供较为顺畅的语音交互(Hurst et al., 2024; Zhan et al., 2024; Li et al., 2025b; Kaplan et al., 2020; Hoffmann et al., 2022; Xu et al., 2023; Gao et al., 2025),人工智能体与真人对话之间仍有一条关键差距。眼下大多数语音交互仍被限制在僵硬的半双工模式里:系统必须先听后说、按顺序轮换。真人对话却天然是全双工的,需要一边持续处理输入音频流,一边同步生成回复(Sacks et al., 1974; Fu et al., 2025; Ji et al., 2024)。半双工系统强制轮换发言,会打断交互的流畅性,在用户与智能体之间筑起一道人为的隔阂。

全双工语音语言模型在推理开销与准确率之间的权衡,含 Lychee-FD

Figure 1: Visualization of the efficiency and intelligence trade-off. Existing paradigms face a dilemma when extending a half-duplex SLM (black star) to a full-duplex one. Native End-to-End models (blue diamond) sacrifice accuracy for efficiency, while Thinker-Talker models (red diamond) preserve knowledge but incur prohibitive inference costs. In contrast, our proposed Hierarchical framework (green diamond) combines low latency with high accuracy, significantly outperforming all full-duplex SLM baselines.

图 1:效率与智力权衡的可视化。把半双工 SLM(黑星)扩展成全双工时,现有范式陷入两难。原生端到端模型(蓝色菱形)为了效率牺牲准确率,Thinker-Talker 模型(红色菱形)能保住知识,却带来过高的推理开销。相比之下,我们提出的分层框架(绿色菱形)把低延迟和高准确率合在一起,明显优于所有全双工 SLM 基线。

To bridge this gap, developing native Full-Duplex SLMs (FDSLMs) has emerged as a critical challenge and a long-standing goal for the speech and NLP communities (Sacks et al., 1974; Arora et al., 2025; Nguyen et al., 2023; Lin et al., 2022). This full-duplex paradigm demands complex pragmatic reasoning, enabling the agent to handle spontaneous interruptions, inject natural backchannels, and dynamically manage turn-taking without explicit system-level triggers (Hu et al., 2025; Liu et al., 2025). Consequently, achieving seamless full-duplex communication is widely regarded as the next critical milestone in human-machine interaction, promising to unlock a truly natural, fluid, and immersive conversation experience (Lin et al., 2025b; Fu et al., 2025; Lin et al., 2025a).

为弥补这一差距,开发原生全双工 SLM(Full-Duplex SLMs, FDSLMs)已成为语音与自然语言处理社区的关键挑战和长期目标(Sacks et al., 1974; Arora et al., 2025; Nguyen et al., 2023; Lin et al., 2022)。这种全双工范式需要复杂的语用推理,使智能体能够处理即兴插话、插入自然的附和,并在没有系统级显式触发的情况下动态管理话轮(Hu et al., 2025; Liu et al., 2025)。因此,实现流畅的全双工交流被广泛视为人机交互的下一个关键节点,有望打开真正自然、流畅、沉浸式的对话体验(Lin et al., 2025b; Fu et al., 2025; Lin et al., 2025a)。

Despite progress in FDSLM development (Fu et al., 2024; Zeng et al., 2025), current methods still suffer from severe Modality Interference. As illustrated in Figure 1, adapting a half-duplex base model (black star) into a native End-to-End architecture (blue diamond) precipitates significant knowledge degradation. This degradation is further corroborated by state-of-the-art models such as Moshi (Défossez et al., 2024), which reports a significant drop in speech intelligence (-12.7% on LlamaQ) after full-duplex alignment. Historically, this type of interference has been a notorious and persistent optimization bottleneck across deep learning (Ruder, 2017; Caruana, 1997; Yu et al., 2020; Sener and Koltun, 2018). While some approaches (Wang et al., 2025; Chen et al., 2025b) attempt to circumvent this interference by adopting Thinker-Talker architectures (red diamond), they require intricate multi-stage training and introduce significant latency. These limitations indicate that existing paradigms either fail in knowledge retention or inference efficiency. Consequently, the central research question of this work is: How can we resolve this modality interference to simultaneously achieve high inference efficiency and robust knowledge retention in full-duplex SLMs?

尽管 FDSLM 的研制已有进展(Fu et al., 2024; Zeng et al., 2025),现有方法仍深受严重的模态干扰之苦。如图 1 所示,把半双工基座模型(黑星)改成原生端到端架构(蓝色菱形),会带来明显的知识退化。这一现象也被 Moshi(Défossez et al., 2024)等最强模型所印证:在做完全双工对齐后,语音智力明显下降(LlamaQ 上为 −12.7%)。这类干扰在深度学习中一直是棘手且反复出现的优化瓶颈(Ruder, 2017; Caruana, 1997; Yu et al., 2020; Sener and Koltun, 2018)。有些方法(Wang et al., 2025; Chen et al., 2025b)试图用 Thinker-Talker 架构(红色菱形)绕开干扰,但又需要复杂的多阶段训练,并引入明显延迟。这些局限说明,现有范式要么保不住知识,要么推理效率不够。因此,本文的核心问题是:如何化解这种模态干扰,使全双工 SLM 同时具备高推理效率和稳健的知识保持?

To uncover the root cause of modality interference, we conduct an exhaustive fine-grained model optimization dynamics analysis (shown in Figure 2). By quantifying the geometric relationships between the gradient vectors of semantic and acoustic objectives, we demonstrate the inherent gradient conflicts that fundamentally cause this interference. First, by calculating the layer-wise gradient cosine similarity, we reveal severe optimization divergence in gradient directions. While the gradient directions of text and speech objectives are synergistic in shallow layers, they become increasingly orthogonal and negative in deeper layers. This empirically demonstrates that forcing acoustic and semantic modeling to update within a shared parameter space inevitably fractures the optimization trajectory. Second, by evaluating the gradient magnitude ratio, we identify severe semantic dilution. The temporal alignment of sparse text tokens (e.g., 3Hz) with dense audio frames (e.g., 25Hz) via padding tokens consistently suppresses the magnitude of semantic gradients across all layers. Consequently, the optimization landscape becomes overwhelmingly dominated by acoustic modeling, effectively suppressing semantic modeling.

为找出模态干扰的根源,我们对模型优化动力学做了细致分析(见图 2)。通过量化语义目标与声学目标梯度向量之间的几何关系,我们揭示了从根本上造成干扰的固有梯度冲突。首先,计算逐层梯度余弦相似度后,我们发现梯度方向上存在严重的优化方向分裂。文本与语音目标的梯度在浅层是协同的,到了深层却越来越正交,甚至转为负值。这从经验上说明,强迫声学建模和语义建模在同一组共享参数里更新,优化轨迹必然被撕开。其次,评估梯度幅度比后,我们识别出严重的语义稀释。用填充符把稀疏的文本词元(例如 3Hz)与密集的音频帧(例如 25Hz)做时间对齐,会在各层持续压低语义梯度的幅度。结果是,优化景观几乎被声学建模主导,语义建模受到抑制。

Inspired by these insights, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference with two architectural innovations. First, we propose a hierarchical parameter separation strategy for optimization divergence in deep layers. Specifically, we separate the deep layers into independent acoustic and semantic heads. By executing these heads in parallel, we maintain the original model depth, thereby preserving inference efficiency. Second, to counter semantic dilution, we introduce a semantic alignment channel to generate coherent internal monologues. By utilizing continuous textual supervision as a semantic anchor, we preserve the robustness of semantic modeling during training. Extensive experiments demonstrate that Lychee-FD achieves the state-of-the-art performance, specifically delivering an average 7.4% improvement on Spoken QA tasks and a 28.5% gain on FullDuplexBench 1.5. Ultimately, our approach effectively mitigates modality interference, simultaneously achieving high inference efficiency and robust knowledge retention.

受这些观察启发,我们提出 Lychee-FD,一种用两项架构创新来缓解模态干扰的原生端到端全双工框架。第一,针对深层的优化方向分裂,我们提出分层参数分离策略。具体而言,把深层拆成独立的声学头和语义头;两头并行执行,从而保持原有模型深度,也就保住了推理效率。第二,为对抗语义稀释,我们引入语义对齐通道,用于生成连贯的内部独白。用连续的文本监督作为语义锚,训练时即可保持语义建模的稳健性。大量实验表明,Lychee-FD 达到了当前最优水平:口语问答任务平均提升 7.4%,FullDuplexBench 1.5 提升 28.5%。该方法有效缓解了模态干扰,同时实现了高推理效率和稳健的知识保持。

Our contributions are summarized as follows:

本文贡献概括如下:

  • • For the first time, we uncover the root cause of modality interference in full-duplex SLMs. Supported by an exhaustive fine-grained analysis of model optimization dynamics, we reveal that these obstacles stem from inherent gradient conflicts between acoustic and semantic modeling when forced into a shared deep parameter space.
  • • 我们首次揭示了全双工 SLM 中模态干扰的根源。通过对模型优化动力学的细致分析,我们说明:这些障碍来自声学建模与语义建模在共享深层参数空间中的固有梯度冲突。
  • • We present Lychee-FD, the first fully native intelligent full-duplex framework. We propose a hierarchical parameter separation strategy that decouples acoustic and semantic modeling, bridged by an elegantly designed semantic alignment channel to preserve coherent internal monologues and robust semantics.
  • • 我们提出 Lychee-FD,第一个完全原生的智能全双工框架。我们设计分层参数分离策略,把声学建模与语义建模拆开,并用精心设计的语义对齐通道来保持连贯的内部独白和稳健语义。
  • • We advance the state-of-the-art across multiple full-duplex benchmarks. To forge ahead the full-duplex research in the communities, we open-source our framework, training pipeline, and model weights at https://github.com/HITsz-TMG/Lychee-FD.
  • • 我们在多个全双工基准上推进了现有最佳水平。为推动社区的全双工研究,我们在 https://github.com/HITsz-TMG/Lychee-FD 开源了框架、训练流程和模型权重。

2 Related Work

2 相关工作

2.1 Spoken Language Model

2.1 语音语言模型

SLMs have evolved from cascading ASR-LLM-TTS pipelines (Zhang et al., 2023; An et al., 2024; Borsos et al., 2023; Chen et al., 2025c; Zeghidour et al., 2022) to unified architectures. Current methods are mainly grouped into two paradigms:

SLM 已从级联的 ASR-LLM-TTS 流水线(Zhang et al., 2023; An et al., 2024; Borsos et al., 2023; Chen et al., 2025c; Zeghidour et al., 2022)演进到统一架构。当前方法主要分成两种范式:

Thinker-Talker Architectures.

This paradigm decouples acoustic generation from the LLM backbone to mitigate modality interference (Xu et al., 2025; Wang et al., 2025; Fang et al., 2025). For instance, Xu et al. (2025) adopts a dual-track autoregressive architecture. Despite their stability, these systems often require intricate, multi-stage training curricula and suffer from inference bottlenecks caused by the separate generation modules.

Thinker-Talker 架构。

这一范式把声学生成从 LLM 主干中拆出,以减轻模态干扰(Xu et al., 2025; Wang et al., 2025; Fang et al., 2025)。例如,Xu et al. (2025) 采用双轨自回归架构。这类系统虽然稳定,却往往需要复杂的多阶段训练课程,并且会因独立的生成模块而出现推理瓶颈。

Native End-to-End Architectures.

Conversely, these architectures embed speech and text into a shared semantic space, enabling direct speech-to-speech reasoning (Li et al., 2025c; Xie and Wu, 2024b; Mitsui et al., 2024; Zeng et al., 2025). For examples, Wu et al. (2025) introduces latent audio encoding to capture paralinguistic cues.

原生端到端架构。

与此相对,这类架构把语音和文本嵌入同一语义空间,从而直接做语音到语音的推理(Li et al., 2025c; Xie and Wu, 2024b; Mitsui et al., 2024; Zeng et al., 2025)。例如,Wu et al. (2025) 引入潜在音频编码,以捕捉副语言线索。

Crucially, while existing SLMs remain fundamentally bottlenecked by rigid, half-duplex turn-taking mechanisms, our work advances the field by establishing a native Full-Duplex framework. By enabling simultaneous listening and speaking, we aim to unlock a truly natural, fluid, and immersive paradigm for future human-machine interaction.

需要强调的是,现有 SLM 根本上仍受僵硬的半双工轮换机制制约;我们的工作则进一步建立了原生全双工框架。通过同时听和说,我们希望为未来的人机交互打开一种真正自然、流畅、沉浸式的范式。

2.2 Full Duplex Speech Interaction

2.2 全双工语音交互

To achieve turn-free and fluid conversation, recent research has explored full-duplex paradigms that transcend rigid turn-taking. These efforts can be broadly categorized into three paradigms:

为了实现不依赖固定轮换的流畅对话,近期研究探索了超越僵硬话轮的全双工范式。这些工作大体可分为三类:

Full-Duplex Dialogue System.

Early efforts primarily leverage Voice Activity Detection (VAD) as a dialogue manager to control the speaking process of half-duplex SLMs (Chen et al., 2025a; Liao et al., 2025; Li et al., 2025a; Zhang et al., 2025a). Although making some progress, these cascaded systems suffer from high latency and error propagation, prompting a shift towards unified modeling.

全双工对话系统。

早期工作主要把语音活动检测(Voice Activity Detection, VAD)当作对话管理器,用来控制半双工 SLM 的发言过程(Chen et al., 2025a; Liao et al., 2025; Li et al., 2025a; Zhang et al., 2025a)。尽管有所进展,这类级联系统延迟高、误差会层层传递,于是研究转向了统一建模。

Time-Division Multiplexing (TDM).

To address these limitations, recent works use the SLM itself as the dialogue manager. TDM approaches flatten listening and speaking tokens into a single temporal sequence. However, this leads to increasing computational complexity and limits long-context interaction (Zhang et al., 2025b; Veluri et al., 2024; Zhang et al., 2024; Yu et al., 2025).

时分复用(Time-Division Multiplexing, TDM)。

为克服上述限制,近期工作让 SLM 自身充当对话管理器。TDM 方法把听和说的词元展平到同一时间序列中。但这会提高计算复杂度,并限制长上下文交互(Zhang et al., 2025b; Veluri et al., 2024; Zhang et al., 2024; Yu et al., 2025)。

Channel-Division Multiplexing (CDM).

Instead of flattening, CDM approaches explicitly model concurrent input and output streams, offering the most integrated form of interaction (Zhang et al., 2025c; Nguyen et al., 2023; Xie and Wu, 2024a). For example, Défossez et al. (2024) and Chen et al. (2025b) integrate time-aligned text and audio streams to provide explicit semantic guidance for speech generation. However, these fully shared architectures entangle distinct modalities in a single parameter space, leaving them highly susceptible to severe knowledge degradation.

信道分复用(Channel-Division Multiplexing, CDM)。

CDM 方法不再做展平,而是显式建模并行的输入和输出流,提供最完整的交互形式(Zhang et al., 2025c; Nguyen et al., 2023; Xie and Wu, 2024a)。例如,Défossez et al. (2024) 与 Chen et al. (2025b) 把时间对齐的文本流和音频流结合在一起,为语音生成提供显式的语义引导。不过,这些完全共享的架构把不同模态缠在同一参数空间里,因而很容易出现严重的知识退化。

Unlike previous works that struggle with modality interference, Lychee-FD provides the first fundamental solution to the core problem within FDSLMs. By resolving the inherent deep-layer gradient conflicts between acoustic and semantic modeling, our native end-to-end framework successfully achieves ultra-low latency with robust knowledge retention, substantially pushing forward the frontier of research for the speech and NLP communities.

与此前在模态干扰上陷入困境的工作不同,Lychee-FD 为 FDSLM 的核心问题提供了第一种根本性的解决方案。通过化解声学建模与语义建模在深层的固有梯度冲突,我们的原生端到端框架在保持稳健知识的同时实现了极低延迟,显著推进了语音与自然语言处理领域的研究前沿。

文本与语音任务的逐层梯度余弦相似度和梯度幅度比

Figure 2: Optimization Dynamics Visualization. (a) Gradient Cosine Similarity: The transition to negative values in deep layers reveals conflicting optimization directions between semantic and acoustic modeling, motivating our hierarchical parameter separation. (b) Gradient Magnitude Ratio: The consistently lower ratio for “Aligned” (Red) compared to “Dense” (Blue) indicates that sparse alignment dilutes semantic supervision, motivating our semantic alignment channel.

图 2:优化动力学可视化。(a) 梯度余弦相似度:深层转为负值,说明语义建模与声学建模的优化方向相互冲突,这促使我们提出分层参数分离。(b) 梯度幅度比:“Aligned”(红)始终低于“Dense”(蓝),说明稀疏对齐会稀释语义监督,这促使我们提出语义对齐通道。

3 Hierarchical Acoustic-Semantic Modeling

3 分层声学-语义建模

3.1 Optimization Dynamics of Modality Interference

3.1 模态干扰的优化动力学

To empirically investigate the fundamental root cause of the modality interference and understand the learning process within FDSLMs, we conducted an in-depth model optimization dynamics analysis, shown in Figure 2. We utilized a native CDM architecture initialized with weights from StepAudio-2-mini (Wu et al., 2025). We performed forward passes on 1K samples from the training set to accumulate gradients for the cross-entropy losses of both text token generation (Ltext\mathcal{L}_{\text{text}}Ltext​) and speech token generation (Lspeech\mathcal{L}_{\text{speech}}Lspeech​), without updating the model parameters. Specifically, the layer-wise gradient vectors are formally defined as follows:

为从经验上考察模态干扰的根本原因,并理解 FDSLM 内部的学习过程,我们做了深入的模型优化动力学分析,见图 2。我们采用以 StepAudio-2-mini(Wu et al., 2025)权重初始化的原生 CDM 架构,在训练集的 1K 条样本上做前向计算,分别累积文本词元生成损失(Ltext\mathcal{L}_{\text{text}}Ltext​)和语音词元生成损失(Lspeech\mathcal{L}_{\text{speech}}Lspeech​)的交叉熵梯度,但不更新模型参数。逐层梯度向量形式化定义如下:

gtext(l)=∇θ(l)Ltext,(1)\mathbf{g}_{\text{text}}^{(l)}=\nabla_{\theta^{(l)}}\mathcal{L}_{\text{text}}, \tag{1}gtext(l)​=∇θ(l)​Ltext​,(1)

gspeech(l)=∇θ(l)Lspeech,(2)\mathbf{g}_{\text{speech}}^{(l)}=\nabla_{\theta^{(l)}}\mathcal{L}_{\text{speech}}, \tag{2}gspeech(l)​=∇θ(l)​Lspeech​,(2)

where ∇θ(l)\nabla_{\theta^{(l)}}∇θ(l)​ denotes the gradient operator with respect to the layer parameters θ(l)\theta^{(l)}θ(l), and the resulting tensors are flattened into vectors. By analyzing the geometric relationships of these gradient vectors, we quantitatively uncover the fundamental interaction dynamics between the two modalities during the entire learning process.

其中 ∇θ(l)\nabla_{\theta^{(l)}}∇θ(l)​ 表示对第 lll 层参数 θ(l)\theta^{(l)}θ(l) 的梯度算子,得到的张量被展平为向量。通过分析这些梯度向量的几何关系,我们定量揭示了两种模态在整个学习过程中的基本交互动力学。

Optimization Divergence.

We investigated the compatibility of the two optimization objectives by calculating the cosine similarity S(l)\mathcal{S}^{(l)}S(l) between text and speech gradient vectors:

优化方向分裂。

我们通过计算文本与语音梯度向量的余弦相似度 S(l)\mathcal{S}^{(l)}S(l),考察两个优化目标是否相容:

S(l)=cos⁡(gtext(l),gspeech(l)).(3)\mathcal{S}^{(l)}=\cos(\mathbf{g}_{\text{text}}^{(l)},\mathbf{g}_{\text{speech}}^{(l)}). \tag{3}S(l)=cos(gtext(l)​,gspeech(l)​).(3)

As shown in Figure 2a, the similarity reveals a distinct layer-wise pattern. In the shallow layers (0-9), the cosine similarity is positive, indicating that the two modalities share synergistic optimization directions, focusing on common low-level features processing. However, as the depth increases, the similarity drops sharply, turning negative and fluctuating in the deeper layers. This trend empirically confirms our hypothesis regarding the dual nature of speech: while shallow layers can share representations, the deep layers face a fundamental conflict between acoustic and semantic modeling. Forcing a unified set of parameters to resolve these opposing gradient directions inevitably leads to sub-optimal performance, validating the root cause of the observed modality interference. To further substantiate these findings, we extend our gradient cosine similarity analysis to the Moshi (Défossez et al., 2024) architecture in Appendix 5, and provide a global gradient influence analysis in Appendix D to demonstrate the causal destructive impact of this modality conflict.

如图 2a 所示,相似度呈现出清晰的逐层规律。在浅层(第 0–9 层),余弦相似度为正,说明两种模态共享协同的优化方向,共同处理低层特征。随着深度增加,相似度急剧下降,在深层转为负值并上下波动。这一趋势从经验上证实了我们对语音双重性质的假设:浅层可以共享表示,深层却面临声学建模与语义建模的根本冲突。强迫同一组参数去调和方向相反的梯度,必然导致次优表现,这也验证了所观察到的模态干扰的根源。为进一步支撑这些发现,我们在附录 5 中把梯度余弦相似度分析推广到 Moshi(Défossez et al., 2024)架构,并在附录 D 中给出全局梯度影响分析,以说明这种模态冲突会造成因果性的破坏性影响。

Thinker-Talker、原生端到端与 Lychee-FD 三种架构对比

Figure 3: Two mainstream architecture paradigms of SLMs and our proposed Lychee-FD. Our design features a hierarchical parameter separation strategy to resolve deep-layer modality conflicts and a semantic alignment channel to enforce robust knowledge retention without sacrificing inference efficiency.

图 3:SLM 的两种主流架构范式,以及我们提出的 Lychee-FD。我们的设计用分层参数分离策略化解深层模态冲突,并用语义对齐通道在不牺牲推理效率的前提下强制保持稳健的知识。

Semantic Dilution.

Prevalent Full-Duplex SLMs typically address the frequency mismatch between text (approximately 3Hz) and audio (typically 25Hz) by interleaving padding tokens to enforce temporal alignment (Défossez et al., 2024; Wu et al., 2025; Chen et al., 2025b). To evaluate the impact of this alignment on optimization, we compared the ratio R(l)\mathcal{R}^{(l)}R(l) of gradient magnitudes between the two modalities:

语义稀释。

常见的全双工 SLM 通常用插入填充符的办法,把文本(大约 3Hz)与音频(通常 25Hz)的频率不匹配对齐(Défossez et al., 2024; Wu et al., 2025; Chen et al., 2025b)。为评估这种对齐对优化的影响,我们比较了两种模态的梯度幅度比 R(l)\mathcal{R}^{(l)}R(l):

R(l)=∣∣gtext(l)∣∣/∣∣gspeech(l)∣∣.(4)\mathcal{R}^{(l)}=||\mathbf{g}_{\text{text}}^{(l)}||/||\mathbf{g}_{\text{speech}}^{(l)}||. \tag{4}R(l)=∣∣gtext(l)​∣∣/∣∣gspeech(l)​∣∣.(4)

As illustrated in Figure 2b, we observe a substantial disparity between the continuous text supervision (Dense) and the sparse text with padding (Aligned). Specifically, the gradient magnitude ratio in the Aligned setting is consistently suppressed across all layers, suggesting that the introduction of padding tokens effectively dilutes the density of semantic supervision. Consequently, the optimization dynamics become dominated by acoustic reconstruction, which drives the observed degradation in knowledge retention.

如图 2b 所示,连续文本监督(Dense)与带填充的稀疏文本(Aligned)之间存在明显差距。具体而言,Aligned 设定下的梯度幅度比在各层都被压低,说明引入填充符实际上稀释了语义监督的密度。结果是,优化动力学被声学重建主导,从而造成我们所看到的知识保持退化。

3.2 Core Architectural Innovations: Separation and Alignment

3.2 核心架构创新:分离与对齐

Guided by these insights, we introduce Lychee-FD, a native end-to-end framework designed to resolve modality interference. Rather than circumventing these optimization bottlenecks via high-latency Thinker-Talker models (as illustrated in Figure 3), our architecture fundamentally tackles them via two targeted innovations: a Hierarchical Parameter Separation strategy to disentangle conflicting optimization directions, and a Semantic Alignment Channel to counteract semantic dilution and enforce robust knowledge retention.

基于这些认识,我们提出 Lychee-FD,一种用于化解模态干扰的原生端到端框架。我们没有像图 3 那样用高延迟的 Thinker-Talker 模型绕开这些优化瓶颈,而是用两项针对性创新从根本上手:用分层参数分离策略拆开相互冲突的优化方向,并用语义对齐通道对抗语义稀释、强制保持稳健的知识。

Shared Backbone Foundation.

We choose StepAudio-2-mini (Wu et al., 2025) as our half-duplex backbone due to its public availability. We employ the Whisper-v3-large encoder for input audio processing. For acoustic output, we adopt the CosyVoice2 tokenizer to convert audio into discrete speech tokens at a 25Hz frame rate. Crucially, to ensure precise temporal alignment for full-duplex interaction, we utilize a 25Hz frame rate, distinct from the setting adopted by Step-Audio-2.

共享主干基础。

我们选择公开可获取的 StepAudio-2-mini(Wu et al., 2025)作为半双工骨干。输入音频用 Whisper-v3-large 编码器处理。声学输出方面,我们采用 CosyVoice2 分词器,把音频转成 25Hz 帧率的离散语音词元。关键的是,为了给全双工交互提供精确的时间对齐,我们采用 25Hz 帧率,这与 Step-Audio-2 原来的设定不同。

Hierarchical Parameter Separation.

Guided by the observation that acoustic and semantic gradients become orthogonal in deeper layers, we design a hierarchical Transformer architecture. We retain a unified Transformer backbone for the shallow layers to leverage shared low-level feature processing. Formally, given the input embedding sequence E∈RN×d\mathbf{E}\in\mathbb{R}^{N\times d}E∈RN×d, the shared representation Hshared\mathbf{H}_{\text{shared}}Hshared​ is mathematically formulated as follows:

分层参数分离。

鉴于声学梯度与语义梯度在深层变得正交,我们设计了分层 Transformer 架构。浅层仍保留统一的 Transformer 主干,以利用共享的低层特征处理。形式化地,给定输入嵌入序列 E∈RN×d\mathbf{E}\in\mathbb{R}^{N\times d}E∈RN×d,共享表示 Hshared\mathbf{H}_{\text{shared}}Hshared​ 写为:

Hshared=Fshared(E;θshared),(5)\mathbf{H}_{\text{shared}}=\mathcal{F}_{\text{shared}}(\mathbf{E};\theta_{\text{shared}}), \tag{5}Hshared​=Fshared​(E;θshared​),(5)

where Fshared\mathcal{F}_{\text{shared}}Fshared​ denotes the stack of shared Transformer layers parameterized by θshared\theta_{\text{shared}}θshared​.

其中 Fshared\mathcal{F}_{\text{shared}}Fshared​ 表示由 θshared\theta_{\text{shared}}θshared​ 参数化的共享 Transformer 层堆叠。

In the deeper layers, we physically disentangle the parameters into three specialized heads: the Semantic Head for text generation, the Acoustic Head for speech synthesis, and the Control Head for interaction management (e.g., stop/start signals). These heads operate in parallel as follows:

在深层,我们把参数物理拆成三个专用头:用于文本生成的语义头、用于语音合成的声学头,以及用于交互管理(例如起止信号)的控制头。这些头并行工作,如下所示:

Om=Fheadm(Hshared;θm),m∈{T,A,C}(6)\mathbf{O}^{m}=\mathcal{F}_{\text{head}}^{m}(\mathbf{H}_{\text{shared}};\theta_{m}),\quad m\in\{T,A,C\} \tag{6}Om=Fheadm​(Hshared​;θm​),m∈{T,A,C}(6)

where mmm represents the modality (Text, Acoustic, Control), and Om\mathbf{O}^{m}Om denotes the output logits for each head. The total cross-entropy loss L\mathcal{L}L is computed as the summation of the next-token prediction losses for each specific head as follows:

其中 mmm 表示模态(文本、声学、控制),Om\mathbf{O}^{m}Om 是各头的输出 logits。总交叉熵损失 L\mathcal{L}L 为各专用头下一词元预测损失之和:

L=−∑m∈{T,A,C}∑tlog⁡P(ytm∣y<t,E;θ),(7)\mathcal{L}=-\sum_{m\in\{T,A,C\}}\sum_{t}\log P(y_{t}^{m}|y_{<t},\mathbf{E};\theta), \tag{7}L=−m∈{T,A,C}∑​t∑​logP(ytm​∣y<t​,E;θ),(7)

where ytmy_{t}^{m}ytm​ denotes the ground-truth token for modality mmm at step ttt. This hierarchical split effectively isolates the conflicting optimization objectives, allowing the model to articulate high-fidelity acoustic responses without corrupting its underlying semantic modeling.

其中 ytmy_{t}^{m}ytm​ 表示模态 mmm 在第 ttt 步的真值词元。这种分层拆分有效隔离了相互冲突的优化目标,使模型能够给出高保真的声学回应,而不破坏其底层的语义建模。

Semantic Alignment Channel.

To counter the semantic dilution caused by sparse alignment in speech-native tasks, we incorporate a semantic alignment channel to generate coherent internal monologues. During training, these monologues serve as a semantic anchor, maintaining high-magnitude gradient flow for the language modeling objective. Specifically, we organize the parallel generation streams as follows:

语义对齐通道。

为对抗语音原生任务中稀疏对齐造成的语义稀释,我们加入语义对齐通道,用来生成连贯的内部独白。训练时,这些独白充当语义锚,为语言建模目标维持高幅度的梯度流。具体而言,我们把并行生成流组织如下:

$$\begin{aligned}
Y^{\text{T}} &= [t_{1},t_{2},\cdots,t_{n},\texttt{},\texttt{},\cdots,\texttt{}], \
Y^{\text{A}} &= [a_{1},a_{2},\cdots,a_{n},a_{n+1},a_{n+2},\cdots,\texttt{}], \
Y^{\text{C}} &= [\texttt{},c_{1},\cdots,\cdots,\cdots,\texttt{}],
\end{aligned} \tag{8}$$

where YCY^{\text{C}}YC employs special tokens (e.g., and ) to manage the onset and offset of the model’s speech. By explicitly modeling the text channel alongside the acoustic channel, we ensure high-magnitude gradient flow for the language modeling objective, thereby preserving robust knowledge retention.

其中 YCY^{\text{C}}YC 用特殊词元(例如 和 )管理模型语音的起止。通过在声学通道之外显式建模文本通道,我们保证语言建模目标保有高幅度梯度流,从而维持稳健的知识保持。

Model Type LlamaQ S→TS{\rightarrow}TS→T LlamaQ S→SS{\rightarrow}SS→S WebQ S→TS{\rightarrow}TS→T WebQ S→SS{\rightarrow}SS→S TriviaQA S→TS{\rightarrow}TS→T TriviaQA S→SS{\rightarrow}SS→S Avg. S→TS{\rightarrow}TS→T Avg. S→SS{\rightarrow}SS→S Takeover Rate
Freeze-Omni System-level 71.3 50.7 38.3 25.8 24.3 23.9 44.6 33.4 99.6
VITA 1.5 System-level 75.7 51.0 41.8 29.2 35.0 26.0 50.8 35.4 100
dGSLM Native – 1.3 – 0.2 – 0.4 – 0.6 100
FLM-audio Native 41.3 36.7 15.6 14.5 10.5 10.4 22.4 20.5 99.5
Moshi Native 62.3 54.7 25.3 19.6 19.1 17.4 35.5 30.5 93.8
SALMONN-omni∗^*∗ Native 67.0 61.7 33.7 28.1 32.9 24.2 44.5 38.0 99.9
Fun-Audio-Chat Native 72.3 64.3 26.2 24.4 29.6 27.7 42.7 38.8 99.9
StepAudio-2-mini Half-duplex 74.7 62.0 39.9 30.8 39.5 29.8 51.3 40.9 –
Lychee-FD (Ours) Native 73.7 65.3 38.3 33.9 42.5 39.4 51.5 46.2 100
\quad w/o SemSemSem-ChannelChannelChannel 69.3 61.0 34.1 31.5 34.2 30.1 45.9 40.8 99.6
\quad w/o ParamParamParam-SepSepSep 67.0 36.0 34.6 22.5 36.6 24.2 46.1 27.6 98.5

Table 1: Performance comparison on spoken question answering benchmarks. We report accuracy (Acc) in both speech-to-text (S→TS{\rightarrow}TS→T) and speech-to-speech (S→SS{\rightarrow}SS→S) settings. We also report average takeover rate (TOR) across three benchmarks. Bold denotes best results and underlined denotes second best. ∗^*∗ denotes our implementation.

模型 类型 LlamaQ S→TS{\rightarrow}TS→T LlamaQ S→SS{\rightarrow}SS→S WebQ S→TS{\rightarrow}TS→T WebQ S→SS{\rightarrow}SS→S TriviaQA S→TS{\rightarrow}TS→T TriviaQA S→SS{\rightarrow}SS→S 平均 S→TS{\rightarrow}TS→T 平均 S→SS{\rightarrow}SS→S 接管率
Freeze-Omni 系统级 71.3 50.7 38.3 25.8 24.3 23.9 44.6 33.4 99.6
VITA 1.5 系统级 75.7 51.0 41.8 29.2 35.0 26.0 50.8 35.4 100
dGSLM 原生 – 1.3 – 0.2 – 0.4 – 0.6 100
FLM-audio 原生 41.3 36.7 15.6 14.5 10.5 10.4 22.4 20.5 99.5
Moshi 原生 62.3 54.7 25.3 19.6 19.1 17.4 35.5 30.5 93.8
SALMONN-omni∗^*∗ 原生 67.0 61.7 33.7 28.1 32.9 24.2 44.5 38.0 99.9
Fun-Audio-Chat 原生 72.3 64.3 26.2 24.4 29.6 27.7 42.7 38.8 99.9
StepAudio-2-mini 半双工 74.7 62.0 39.9 30.8 39.5 29.8 51.3 40.9 –
Lychee-FD(本文) 原生 73.7 65.3 38.3 33.9 42.5 39.4 51.5 46.2 100
\quad w/o SemSemSem-ChannelChannelChannel 69.3 61.0 34.1 31.5 34.2 30.1 45.9 40.8 99.6
\quad w/o ParamParamParam-SepSepSep 67.0 36.0 34.6 22.5 36.6 24.2 46.1 27.6 98.5

表 1:口语问答基准上的性能对比。我们分别报告语音到文本(S→TS{\rightarrow}TS→T)和语音到语音(S→SS{\rightarrow}SS→S)设定下的准确率(Acc),以及三个基准上的平均接管率(TOR)。粗体为最优结果,下划线为次优。∗^*∗ 表示由我们复现。

4 Experiments

4 实验

4.1 Baselines

4.1 基线

To ensure a comprehensive evaluation, we compare our proposed method against a diverse set of representative and competitive full-duplex SLMs. These baselines cover the primary architectural paradigms currently explored in the field:

为了做全面评估,我们把提出的方法与一组有代表性、也有竞争力的全双工 SLM 进行比较。这些基线覆盖了该领域目前主要的架构范式:

System-Level Full-Duplex Models

include Freeze-Omni (Wang et al., 2025) and VITA-1.5 (Fu et al., 2025), which integrate an external VAD module to manage the dialogue state of a standard half-duplex SLM.

系统级全双工模型

包括 Freeze-Omni(Wang et al., 2025)和 VITA-1.5(Fu et al., 2025)。它们接入外部 VAD 模块,来管理标准半双工 SLM 的对话状态。

Native Full-Duplex Models

include dGSLM (Nguyen et al., 2023), FLM-Audio (Yao et al., 2025), Moshi (Défossez et al., 2024) and Fun-Audio-Chat (Chen et al., 2025b). Specifically, Fun-Audio-Chat adopts a Thinker-Talker architecture, while the others utilize a CDM architecture.

原生全双工模型

包括 dGSLM(Nguyen et al., 2023)、FLM-Audio(Yao et al., 2025)、Moshi(Défossez et al., 2024)和 Fun-Audio-Chat(Chen et al., 2025b)。其中 Fun-Audio-Chat 采用 Thinker-Talker 架构,其余采用 CDM 架构。

4.2 Training Data Construction

4.2 训练数据构建

Given the scarcity of open-source full-duplex datasets, we developed an automated pipeline to synthesize high-quality training data covering three key interaction behaviors: Interruptions, User Backchannels, and AI Backchannels. We employed multiple agents to simulate realistic User-Assistant dialogues, injecting rule-based constraints to trigger diverse interruption types (e.g., topic switching, follow-up queries) and natural backchannels. To ensure acoustic robustness, we synthesized speech from the transcripts via CosyVoice 2 Du et al. (2024), coupled with 80K predefined voice prompts for zero-shot cloning. After rigorous filtering to remove samples with logical inconsistencies or low audio quality, we curated a final dataset of approximately 140K full-duplex dialogue instances, providing a diverse and reliable foundation for our experiments. We provide more details of our data pipeline in Appendix F.

由于公开的全双工数据集很少,我们开发了一套自动流水线,合成覆盖三类关键交互行为的高质量训练数据:插话、用户附和与 AI 附和。我们用多个智能体模拟真实的用户–助手对话,并注入基于规则的约束,以触发多样的插话类型(例如话题切换、追问)和自然的附和。为了保证声学稳健性,我们用 CosyVoice 2(Du et al., 2024)根据誊写合成语音,并配上 8 万条预定义音色提示做零样本克隆。经过严格过滤,去掉逻辑不一致或音质差的样本后,最终得到约 14 万条全双工对话实例,为实验提供了多样且可靠的基础。数据流水线的更多细节见附录 F。

4.3 Implementation Details

4.3 实现细节

We optimize our model using AdamW (Loshchilov and Hutter, 2019) with a cosine learning rate scheduler. All experiments are conducted on 8 NVIDIA H20 GPUs, with a global batch size of 32 and a learning rate of 3e-6. We set the warmup ratio to 0.1 and train for 1 epoch, which takes approximately 16 hours. For inference, we employ greedy sampling for both text and speech token generation. We evaluate our model with three random seeds and report their average performance. Regarding the hierarchical architecture configuration, unless otherwise specified, we utilize a shared backbone of 24 Transformer layers. The specialized heads are configured with 4 layers for the text channel, 4 layers for the speech channel, and 2 layers for the control channel. This yields approximately 10B total parameters, a choice supported by the marginal gain analysis in Appendix A.

我们用 AdamW(Loshchilov and Hutter, 2019)和余弦学习率调度来优化模型。全部实验在 8 张 NVIDIA H20 上完成,全局批量大小为 32,学习率为 3e-6。预热比例设为 0.1,训练 1 个 epoch,大约需要 16 小时。推理时,文本和语音词元都采用贪心采样。我们用三个随机种子评估模型,并报告平均表现。关于分层架构的配置,除非另有说明,共享主干为 24 层 Transformer。专用头的配置是:文本通道 4 层,语音通道 4 层,控制通道 2 层。总参数大约 100 亿,这一选择由附录 A 中的边际收益分析所支持。

4.4 Spoken Question Answering

4.4 口语问答

Metrics.

To evaluate the speech intelligence capabilities of our model, we follow previous work (Défossez et al., 2024) and utilize three standard spoken question answering benchmarks: LlamaQ, WebQ, and TriviaQA. We report the accuracy (Acc) under both speech-to-text (S→TS\rightarrow TS→T) and speech-to-speech (S→SS\rightarrow SS→S) settings. For speech-to-speech setting, we leverage Whisper-large-v3 (Radford et al., 2023) to obtain the transcription of generated speech. Additionally, we report the takeover rate (TOR) across three benchmarks to quantify the frequency of model responses, serving as an indicator of the model’s turn-taking behavior.

指标。

为评估模型的语音智力,我们沿用既往工作(Défossez et al., 2024),使用三个标准口语问答基准:LlamaQ、WebQ 和 TriviaQA。我们分别报告语音到文本(S→TS\rightarrow TS→T)和语音到语音(S→SS\rightarrow SS→S)设定下的准确率(Acc)。在语音到语音设定中,我们用 Whisper-large-v3(Radford et al., 2023)转写生成的语音。此外,我们还报告三个基准上的接管率(takeover rate, TOR),用来衡量模型作出回应的频率,作为话轮行为的指标。

Result.

As presented in Table 1, Lychee-FD demonstrates superior spoken question answering capabilities. It achieves the highest average accuracy across both speech-to-text (S→TS\rightarrow TS→T) and speech-to-speech (S→SS\rightarrow SS→S) settings. Compared to the previous SOTA native full-duplex model, Fun-Audio-Chat, Lychee-FD delivers a substantial improvement of 7.4% in S→SS\rightarrow SS→S accuracy and 8.8% in S→TS\rightarrow TS→T accuracy. Even when compared to system-level pipelines like VITA-1.5, our end-to-end approach demonstrates superior reasoning capabilities (10.8% in S→SS\rightarrow SS→S and 0.7% in S→TS\rightarrow TS→T), validating the effectiveness of our framework. Furthermore, Lychee-FD maintains a perfect TOR of 100%, confirming that this exceptional knowledge retention is achieved without compromising seamless, real-time interaction stability.

结果。

如表 1 所示,Lychee-FD 表现出更强的口语问答能力。它在语音到文本(S→TS\rightarrow TS→T)和语音到语音(S→SS\rightarrow SS→S)两种设定上都取得了最高的平均准确率。与此前最强的原生全双工模型 Fun-Audio-Chat 相比,Lychee-FD 在 S→SS\rightarrow SS→S 准确率上提高 7.4%,在 S→TS\rightarrow TS→T 上提高 8.8%。即便与 VITA-1.5 这类系统级流水线相比,我们的端到端方法在推理能力上也更好(S→SS\rightarrow SS→S 高 10.8%,S→TS\rightarrow TS→T 高 0.7%),说明框架是有效的。此外,Lychee-FD 保持了 100% 的完美 TOR,说明这种出色的知识保持并没有牺牲流畅、实时的交互稳定性。

Model FDBench SRR↑\uparrow↑ SIR↑\uparrow↑ EIR↓\downarrow↓ SRIR↑\uparrow↑ FSED.↓\downarrow↓ IRD↓\downarrow↓ FullDuplexBench 1.0 I-TOR↑\uparrow↑ B-Freq↑\uparrow↑ B-TOR↓\downarrow↓ T-TOR↑\uparrow↑ P-TOR↓\downarrow↓ Stop↓\downarrow↓ FullDuplexBench 1.5 IRR↑\uparrow↑ BRR↑\uparrow↑ Stop↓\downarrow↓ Lat.↓\downarrow↓
dGSLM – – – – – – 91.7 1.5 69.1 97.5 93.5 2523 – – – –
Freeze-Omni 12.9 57.2 25.7 29.5 667 5413 77.5 0.1 63.6 33.6 46.3 1380 27.0 63.0 660 2066
VITA 1.5 21.0 46.1 16.3 78.3 3036 9925 99.5 2.5 81.8 58.8 88.8 1523 6.0 38.0 1222 2140
FLM-audio 7.5 69.4 1.0 0.9 989 3408 91.0 0.3 61.8 96.6 56.5 4579 10.0 43.0 2439 983
Moshi 41.4 78.8 22.1 73.9 1895 1421 87.5 5.1 36.4 76.4 54.1 885 61.0 26.0 1071 3034
Lychee-FD (Ours) 86.3 99.7 0.4 95.8 637 1210 94.5 14.6 23.4 98.3 10.0 840 78.0 69.0 570 826

Table 2: Performance comparison of full-duplex interaction capabilities and efficiency. Despite interaction behavior metrics, we also report efficiency metrics of each benchmark (FSED, IRD, Lat. Stop.), measured in milliseconds (ms). ↑\uparrow↑ indicates higher is better. Bold denotes best results and underlined denotes second best.

模型 FDBench SRR↑\uparrow↑ SIR↑\uparrow↑ EIR↓\downarrow↓ SRIR↑\uparrow↑ FSED.↓\downarrow↓ IRD↓\downarrow↓ FullDuplexBench 1.0 I-TOR↑\uparrow↑ B-Freq↑\uparrow↑ B-TOR↓\downarrow↓ T-TOR↑\uparrow↑ P-TOR↓\downarrow↓ Stop↓\downarrow↓ FullDuplexBench 1.5 IRR↑\uparrow↑ BRR↑\uparrow↑ Stop↓\downarrow↓ Lat.↓\downarrow↓
dGSLM – – – – – – 91.7 1.5 69.1 97.5 93.5 2523 – – – –
Freeze-Omni 12.9 57.2 25.7 29.5 667 5413 77.5 0.1 63.6 33.6 46.3 1380 27.0 63.0 660 2066
VITA 1.5 21.0 46.1 16.3 78.3 3036 9925 99.5 2.5 81.8 58.8 88.8 1523 6.0 38.0 1222 2140
FLM-audio 7.5 69.4 1.0 0.9 989 3408 91.0 0.3 61.8 96.6 56.5 4579 10.0 43.0 2439 983
Moshi 41.4 78.8 22.1 73.9 1895 1421 87.5 5.1 36.4 76.4 54.1 885 61.0 26.0 1071 3034
Lychee-FD(本文) 86.3 99.7 0.4 95.8 637 1210 94.5 14.6 23.4 98.3 10.0 840 78.0 69.0 570 826

表 2:全双工交互能力与效率对比。除交互行为指标外,我们还报告各基准的效率指标(FSED、IRD、Lat. Stop.),单位为毫秒(ms)。↑\uparrow↑ 表示越高越好。粗体为最优结果,下划线为次优。

Ablation.

To systematically investigate the individual contributions of our two proposed innovations, we conducted ablation studies on two variants: w/o Sem-Channel, which replaces the semantic alignment channel with sparse time-aligned text, and w/o Param-Sep, which removes the hierarchical parameter separation to employ a fully shared architecture. As shown in Table 1, when applying the time-aligned text, we observe a significant performance decline in both S→TS\rightarrow TS→T (5.6%) and S→SS\rightarrow SS→S (5.4%) settings. This parallel degradation confirms our hypothesis regarding semantic dilution: the sparse supervision provided by time-aligned text fails to sustain robust linguistic modeling, which in turn causes knowledge degradation. In contrast, the w/o Param-Sep setting reveals severe modality interference. While its text generation capability remains relatively stable, its speech accuracy suffers a catastrophic drop to 27.6%. This disparity validates our optimization dynamics analysis (as illustrated in Figure 2): without physically disentangling the shared parameters, the optimization landscape becomes dominated by semantic modeling, effectively suppressing the learning of acoustic features in deep layers.

消融。

为系统考察两项创新各自的贡献,我们对两个变体做了消融:w/o Sem-Channel 用稀疏的时间对齐文本替换语义对齐通道;w/o Param-Sep 去掉分层参数分离,改用完全共享的架构。如表 1 所示,改用时间对齐文本后,S→TS\rightarrow TS→T 下降 5.6%,S→SS\rightarrow SS→S 下降 5.4%。这种同步下降证实了关于语义稀释的假设:时间对齐文本提供的稀疏监督撑不起稳健的语言建模,进而造成知识退化。相比之下,w/o Param-Sep 设定暴露出严重的模态干扰。它的文本生成能力相对还稳,语音准确率却急剧掉到 27.6%。这一反差验证了我们的优化动力学分析(见图 2):如果不把共享参数从物理上拆开,优化景观就会被语义建模主导,从而抑制深层声学特征的学习。

Discussion.

We highlight two critical phenomena that distinguish Lychee-FD from existing paradigms. First, our model not only recovers the performance of its half-duplex backbone (StepAudio-2-mini) but surpasses it in both S→TS\rightarrow TS→T (+0.2%) and S→SS\rightarrow SS→S (+5.3%) settings. This performance gain, achieved without increasing model depth, strongly validates that our framework realizes robust knowledge retention while maintaining the inference efficiency of the native end-to-end model. This underscores the pivotal role of explicit semantic modeling in driving model intelligence, offering valuable insights for future human-machine interaction systems. Second, Lychee-FD achieves the smallest modality gap among all native CDM models (e.g., FLM-Audio and Moshi) and remains competitive with Thinker-Talker architectures, without requiring intricate multi-stage training. This demonstrates that by physically decoupling semantic and acoustic modeling, Lychee-FD effectively resolves modality interference. Consequently, our approach allows the model to articulate nature acoustic responses without corrupting its underlying semantic logic, providing a streamlined solution to the efficiency-intelligence trade-off.

讨论。

我们强调两个使 Lychee-FD 有别于现有范式的关键现象。第一,我们的模型不仅找回了半双工骨干 StepAudio-2-mini 的表现,还在 S→TS\rightarrow TS→T(+0.2%)和 S→SS\rightarrow SS→S(+5.3%)上都超过了它。这一提升并未增加模型深度,有力说明框架在保持原生端到端推理效率的同时实现了稳健的知识保持。这也凸显了显式语义建模对模型智力的关键作用,为未来的人机交互系统提供了有价值的启示。第二,在所有原生 CDM 模型(如 FLM-Audio 和 Moshi)中,Lychee-FD 的模态差距最小,并且可以与 Thinker-Talker 架构相竞争,却不需要复杂的多阶段训练。这说明,从物理上把语义建模与声学建模拆开,Lychee-FD 有效化解了模态干扰。因此,该方法使模型能够给出自然的声学回应,而不破坏其底层语义逻辑,为效率与智力的权衡提供了一种更干净的方案。

4.5 Full-duplex Chatting

4.5 全双工对话

Metrics.

For the Full-duplex Chatting, we select three mainstream benchmarks: FDBench (Peng et al., 2025), FullDuplexBench 1.0, and FullDuplexBench 1.5 (Lin et al., 2025a). We follow the recommended settings of these benchmarks for evaluating both baselines and our method. Specifically, FDBench assesses turn-taking and interruption behaviors using Success-Replies Rate (SRR), Success-Interrupts Rate (SIR), Early-Interrupts Rate (EIR), and Success-Replies-to-Interrupts Rate (SRIR), alongside timing metrics such as First-Speech-Emit-Delay (FSED) and Interrupt-Response-Delay (IRD). FullDuplexBench 1.0 consists of four subsets: interruption (I), assistant backchannel (B), turn-taking (T), and user pause (P). In addition to the takeover rate (TOR) and Interruption stop delay (Stop.), we also report the frequency of generated backchannels during user speech (B-Freq). Finally, FullDuplexBench 1.5 utilizes GPT-4o-1124 to classify model responses to user interruptions and backchannels into four categories (Response, Resume, Uncertain, or Unknown), reporting the Interruption-Response Rate (IRR), Backchannel-Resume Rate (BRR), as well as the interruption stop delay and response latency.

指标。

全双工对话方面,我们选用三个主流基准:FDBench(Peng et al., 2025)、FullDuplexBench 1.0 和 FullDuplexBench 1.5(Lin et al., 2025a)。评估基线和我们的方法时,均遵循各基准的推荐设定。具体而言,FDBench 用成功回复率(Success-Replies Rate, SRR)、成功打断率(Success-Interrupts Rate, SIR)、过早打断率(Early-Interrupts Rate, EIR)和打断后成功回复率(Success-Replies-to-Interrupts Rate, SRIR)评估话轮与插话行为,并辅以首次出声延迟(First-Speech-Emit-Delay, FSED)和插话响应延迟(Interrupt-Response-Delay, IRD)等时间指标。FullDuplexBench 1.0 包含四个子集:插话(I)、助手附和(B)、话轮转换(T)和用户停顿(P)。除接管率(TOR)和插话停止延迟(Stop.)外,我们还报告用户说话期间模型生成附和的频率(B-Freq)。最后,FullDuplexBench 1.5 用 GPT-4o-1124 把模型对用户插话和附和的回应分成四类(回应、继续、不确定或未知),并报告插话回应率(Interruption-Response Rate, IRR)、附和后续接率(Backchannel-Resume Rate, BRR),以及插话停止延迟和响应时延。

Result.

As presented in Table 2, Lychee-FD achieves state-of-the-art performance across 10 of the 11 evaluated interaction metrics, demonstrating superior capability in managing complex conversational dynamics, including interruption, backchanneling, dynamic turn-taking, and pause handling. While VITA-1.5 exhibits a marginally higher I-TOR on FullDuplexBench 1.0, its consistently high B-TOR and P-TOR reveal a tendency towards aggressive, indiscriminate speech rather than intelligent turn-taking. In contrast, Lychee-FD maintains a balanced interaction profile, effectively distinguishing between user pauses and interruptions. Notably, on the challenging FullDuplexBench 1.5, our model delivers a substantial 28.5% average improvement over system-level baselines (e.g., Freeze-Omni) that rely on external VAD. Crucially, this empirical breakthrough directly validates our core scientific insight: by resolving the inherent gradient conflicts that plague fully shared architectures, Lychee-FD preserves robust semantic awareness during simultaneous listening and speaking. Consequently, this result not only proves that our model realizes truly natural, fluid, and immersive interaction, but also confirms that LLMs can intrinsically function as highly effective dialog managers. By eliminating handcrafted signal processing modules, our approach provides an efficient, practical solution to the core problem of full-duplex communication, substantially pushing forward the frontier of native end-to-end SLMs.

结果。

如表 2 所示,Lychee-FD 在 11 项交互指标中的 10 项上达到当前最优,显示出更强的复杂对话动态管理能力,包括插话、附和、动态话轮转换和停顿处理。VITA-1.5 在 FullDuplexBench 1.0 上的 I-TOR 略高,但它的 B-TOR 和 P-TOR 一直偏高,说明它更倾向于不加区分地抢话,而不是做有判断的话轮管理。相比之下,Lychee-FD 保持了更均衡的交互画像,能够有效区分用户停顿和插话。值得注意的是,在更具挑战的 FullDuplexBench 1.5 上,我们的模型相对依赖外部 VAD 的系统级基线(例如 Freeze-Omni)平均提升 28.5%。这一经验突破直接印证了我们的核心认识:化解完全共享架构中固有的梯度冲突后,Lychee-FD 在边听边说时仍能保持稳健的语义觉察。因此,这一结果不仅说明模型实现了真正自然、流畅、沉浸式的交互,也确认大语言模型可以本身就充当高效的对话管理器。通过去掉手工的信号处理模块,我们的方法为全双工交流的核心问题提供了高效、可行的方案,显著推进了原生端到端 SLM 的研究前沿。

Latency.

We evaluate inference efficiency through two categories: first response latency (FSED, Lat.) and interruption response latency (IRD, Stop.). Regarding the former, Lychee-FD achieves the lowest latency across all benchmarks. Since our hierarchical parameter separation strategy introduces no additional model depth, we can leverage standard pipeline parallelism for speedup, demonstrating the efficiency of our architectures. Further details about this real-time parallel inference are provided in Appendix E. For Interruption Response Latency, our model demonstrates an even greater advantage, achieving the lowest stop latency when meeting an interruption (e.g., 570ms Stop. on FullDuplexBench 1.5). It is worth noting that interruption latency is a function of both model processing speed and interaction accuracy—since a model must first correctly identify an interruption before it can stop generating. The superior performance confirms that our architecture incurs no computational overhead, and that its robust semantic awareness enables rapid, accurate reactions to user interventions.

延迟。

我们从两类指标评估推理效率:首次回应延迟(FSED、Lat.)和插话回应延迟(IRD、Stop.)。就前者而言,Lychee-FD 在所有基准上都取得了最低延迟。由于分层参数分离并不增加模型深度,我们可以用标准的流水线并行来加速,这说明架构是高效的。实时并行推理的更多细节见附录 E。在插话回应延迟上,我们的模型优势更大,遇到插话时停止延迟最低(例如 FullDuplexBench 1.5 上的 Stop. 为 570 毫秒)。需要指出,插话延迟既取决于模型处理速度,也取决于交互准确性——模型必须先正确识别插话,才能停止生成。出色的表现说明,我们的架构没有带来额外计算开销,而其稳健的语义觉察使它能够对用户介入作出迅速、准确的反应。

4.6 Speech Generation

4.6 语音生成

Lychee-FD 与基线的 UTMOS 语音质量对比

Figure 4: Comparison of speech synthesis quality via UTMOS. Lychee-FD (w/o Param-Sep) denotes the variant without hierarchical parameter separation strategy.

图 4:用 UTMOS 对比语音合成质量。Lychee-FD(w/o Param-Sep)表示去掉分层参数分离策略的变体。

Moshi 架构上的逐层梯度余弦相似度

Figure 5: Gradient Cosine Similarity on the Moshi architecture. The transition from task synergy in shallow layers to task conflict in deep layers confirms that modality interference is a universal phenomenon in fully shared end-to-end SLMs.

图 5:Moshi 架构上的梯度余弦相似度。从浅层的任务协同转向深层的任务冲突,说明模态干扰是完全共享的端到端 SLM 中的普遍现象。

Metrics.

To investigate the quality of the generated speech, we evaluate the content consistency and speech naturalness of the model on LlamaQ dataset. Specifically, we report the Word Error Rate (WER) between the generated text and the transcribed speech to measure content consistency. Additionally, we employ UTMOS (Saeki et al., 2022), a trained speech quality assessment model, to score the naturalness of the generated audio.

指标。

为考察生成语音的质量,我们在 LlamaQ 数据集上评估模型的内容一致性和语音自然度。具体而言,我们报告生成文本与转写语音之间的词错误率(Word Error Rate, WER),用来衡量内容一致性。此外,我们用经过训练的语音质量评估模型 UTMOS(Saeki et al., 2022)为生成音频的自然度打分。

Result.

High-fidelity speech synthesis serves as the cornerstone of voice interaction, significantly elevating the immersive quality of full-duplex conversations. As illustrated in Figure 4, Lychee-FD achieves the highest UTMOS score of 4.50, surpassing Freeze-Omni, Moshi and even its half-duplex backbone. This result empirically validates that our proposed hierarchical parameter separation strategy effectively prevents acoustic modeling from semantic interference, thereby preserving fine-grained prosodic details that are often lost in shared architectures. When the parameter separation is removed, we observe a significant decline in speech quality. This finding corroborates our hypothesis regarding modality interference from an acoustic perspective, demonstrating that forcing the acoustic head to share deep layers with text processing objectives inevitably degrades audio fidelity. Ultimately, by physically disentangling the modeling pathways, Lychee-FD not only improves generation accuracy but also synthesizes speech with richer acoustic details, delivering a more natural and enjoyable conversational experience.

结果。

高保真语音合成是语音交互的基石,能明显提升全双工对话的沉浸感。如图 4 所示,Lychee-FD 取得了最高的 UTMOS 分数 4.50,超过了 Freeze-Omni、Moshi,甚至超过了它的半双工骨干。这一结果从经验上说明,分层参数分离策略有效避免了声学建模受到语义干扰,从而保住了共享架构中常常丢失的细粒度韵律细节。完全去掉参数分离后,语音质量明显下降。这一发现从声学角度印证了关于模态干扰的假设:强迫声学头与文本处理目标共享深层,必然会损害音频保真度。最终,通过从物理上拆开建模通路,Lychee-FD 不仅提高了生成准确率,还能合成带有更丰富声学细节的语音,带来更自然、更好听的对话体验。

5 Geometric Analysis Extension

5 几何分析的推广

To verify the universality of our findings, we extend the gradient cosine similarity analysis to Moshi (Défossez et al., 2024), investigating whether deep-layer gradient conflicts are architecture-specific artifacts or a fundamental bottleneck across different FDSLMs. Following our established methodology, we find that Moshi’s optimization dynamics exhibit a strikingly similar trend (Figure 5). Shallow layers (0-19) maintain positive similarity (S(l)>0\mathcal{S}^{(l)}>0S(l)>0), indicating task synergy. Conversely, as depth increases, the similarity sharply declines to distinctly negative values in deep layers (23-31). This highlights the optimization divergence between text and speech tasks. This cross-architecture validation empirically confirms that inherent gradient conflicts between semantic and acoustic modeling are not isolated anomalies, but a universal bottleneck in fully shared paradigms. Consequently, it firmly substantiates the necessity and generalizability of our proposed Hierarchical Parameter Separation.

为验证发现是否具有普遍性,我们把梯度余弦相似度分析推广到 Moshi(Défossez et al., 2024),考察深层梯度冲突究竟只是特定架构的产物,还是原生端到端 FDSLM 中的根本瓶颈。沿用既有方法,我们发现 Moshi 的优化动力学呈现出极为相似的趋势(图 5)。浅层到中层(第 0–19 层)保持正相似度(S(l)>0\mathcal{S}^{(l)}>0S(l)>0),说明任务是协同的。随着深度增加,相似度急剧下降,在深层(第 23–31 层)明确转为负值。这凸显了文本任务与语音任务之间的优化方向分裂。这一跨架构验证从经验上确认:语义建模与声学建模之间的固有梯度冲突不是孤立异常,而是完全共享范式中的普遍瓶颈。因此,它有力支撑了我们所提出的分层参数分离的必要性与可推广性。

6 Conclusion

6 结论

In this paper, we address a critical challenge and a long-standing goal for the speech and NLP communities: developing seamless, high-performance native full-duplex spoken language models. To the best of our knowledge, we are the first to uncover the fundamental root cause of modality interference. Through an in-depth analysis of model optimization dynamics, we reveal that the inability to simultaneously listen and speak stems from inherent gradient conflicts within a shared deep parameter space, which specifically manifest as optimization divergence and semantic dilution. Inspired by these observations, we introduce Lychee-FD. Our framework elegantly resolves these bottlenecks by decoupling conflicting modalities via a hierarchical parameter separation strategy, coupled with a semantic alignment channel to enforce robust knowledge retention. Extensive experiments demonstrate that Lychee-FD significantly advances the state-of-the-art performance, successfully reconciling ultra-low latency interaction with deep speech intelligence. Ultimately, our work substantially pushes forward the frontier of full-duplex SLMs research, paving the way for the next generation of natural, fluid, and immersive human-machine interaction.

本文面向语音与自然语言处理领域的一项关键挑战和长期目标:开发流畅、高性能的原生全双工语音语言模型。据我们所知,我们首次揭示了模态干扰的根本原因。通过对优化动力学的深入分析,我们说明:无法同时听和说,根源在于共享深层参数空间中的固有梯度冲突,具体表现为优化方向分裂和语义稀释。受这些观察启发,我们提出 Lychee-FD。该框架用分层参数分离策略把相互冲突的模态拆开,并配合语义对齐通道来强制保持稳健的知识,从而干净地化解了这些瓶颈。大量实验表明,Lychee-FD 达到了当前最优水平,把极低延迟的交互与深入的语音智力统一了起来。这项工作为全双工 SLM 研究推进了边界,也为下一代自然、流畅、沉浸式的人机交互铺平了道路。

Limitations

局限性

While Lychee-FD enables the development of a high-performance native full-duplex framework with highly responsive acoustic processing capabilities, deploying such systems in unconstrained, real-world open-mic scenarios remains a promising yet challenging next research frontier. At present, our model implements highly sensitive interruption detection and can effectively suspend speech output upon any user barge-in. However, discriminating between intentional user instructions and incidental background side-talk (as illustrated in Appendix C) remains a non-trivial challenge to address. This limitation is tentatively attributed primarily to the current data synthesis paradigm, which is largely centered on direct two-party interactions and lacks detailed intent annotations for complex multi-speaker scenarios. This issue points to a critical gap in data coverage within the field, rather than stemming from an architectural bottleneck of the hierarchical framework we propose. Moving forward, we hope our preliminary work can provide a modest impetus for subsequent research in the community. By constructing diverse open-mic datasets and integrating prosodic features for intent disambiguation, future studies may build upon this initial framework to develop truly context-aware and adaptive selective interruption mechanisms for next-generation SLMs.

Lychee-FD 虽然已经能够支撑高性能、对声学处理高度灵敏的原生全双工框架,但把这类系统部署到不受约束的真实开放麦克风场景中,仍是下一阶段既有前景、也有难度的研究前沿。目前,我们的模型实现了高度灵敏的插话检测,一旦用户插入,就能有效暂停语音输出。不过,要区分有意的用户指令和偶然的背景边聊(如附录 C 所示),仍然不是一件容易的事。我们初步把这一局限主要归因于当前的数据合成范式:它大体围绕直接的双方交互,缺少对复杂多方场景的细致意图标注。这一问题指向的是该领域在数据覆盖上的缺口,而不是我们提出的分层框架在架构上的瓶颈。展望未来,我们希望这项初步工作能为社区后续研究提供一点推动。通过构建多样的开放麦克风数据集,并结合韵律特征来消歧意图,后续研究或许可以在这一初始框架之上,为下一代 SLM 发展真正有语境意识、能够自适应选择是否打断的机制。

Acknowledgments

致谢

This work is jointly supported by grants: National Natural Science Foundation of China (Grant No. 62422603), Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024B0101050003) and Shenzhen Science and Technology Program (Grant No. ZDSYS20230626091203008).

本工作得到以下资助的共同支持:国家自然科学基金(批准号 62422603)、广东省基础与应用基础研究基金(批准号 2024B0101050003),以及深圳市科技计划(批准号 ZDSYS20230626091203008)。

References

参考文献

An et al. (2024) K. An, Q. Chen, C. Deng, Z. Du, C. Gao, Z. Gao, Y. Gu, T. He, H. Hu, K. Hu, S. Ji, Y. Li, Z. Li, H. Lu, H. Luo, X. Lv, B. Ma, Z. Ma, C. Ni, C. Song, J. Shi, X. Shi, H. Wang, W. Wang, Y. Wang, Z. Xiao, Z. Yan, Y. Yang, B. Zhang, Q. Zhang, S. Zhang, N. Zhao, and S. Zheng FunAudioLLM: voice understanding and generation foundation models for natural interaction between humans and llms . CoRR abs/2407.04051 . External Links: Link , Document , 2407.04051 Cited by: §2.1 .

Arora et al. (2025) S. Arora, K. Chang, C. Chien, Y. Peng, H. Wu, Y. Adi, E. Dupoux, H. Lee, K. Livescu, and S. Watanabe On the landscape of spoken language models: A comprehensive survey . Trans. Mach. Learn. Res. 2025 . External Links: Link Cited by: §1 .

Borsos et al. (2023) Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour AudioLM: A language modeling approach to audio generation . IEEE ACM Trans. Audio Speech Lang. Process. 31 , pp. 2523–2533 . External Links: Link , Document Cited by: §2.1 .

Caruana (1997) R. Caruana Multitask learning . Machine Learning 28 ( 1 ), pp. 41–75 . External Links: ISSN 1573-0565 , Document , Link Cited by: §1 .

Chen et al. (2025a) J. Chen, Y. Hu, J. Li, K. Li, K. Liu, W. Li, X. Li, Z. Li, F. Shen, X. Tang, M. Wei, Y. Wu, F. Xie, K. Xu, and K. Xie FireRedChat: A pluggable, full-duplex voice interaction system with cascaded and semi-cascaded implementations . CoRR abs/2509.06502 . External Links: Link , Document , 2509.06502 Cited by: §2.2 .

Chen et al. (2025b) Q. Chen, L. Cheng, C. Deng, X. Li, J. Liu, C. Tan, W. Wang, J. Xu, J. Ye, Q. Zhang, Q. Zhang, and J. Zhou Fun-audio-chat technical report . External Links: 2512.20156 , Link Cited by: §1 , §2.2 , §3.1 , §4.1 .

Chen et al. (2025c) S. Chen, C. Wang, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei Neural codec language models are zero-shot text to speech synthesizers . IEEE Transactions on Audio, Speech and Language Processing 33 ( ), pp. 705–718 . External Links: Document Cited by: §2.1 .

Défossez et al. (2024) A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour Moshi: a speech-text foundation model for real-time dialogue . CoRR abs/2410.00037 . External Links: Link , Document , 2410.00037 Cited by: §1 , §2.2 , §3.1 , §4.1 , §4.4 , §5 .

Du et al. (2024) Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang, F. Yu, H. Liu, Z. Sheng, Y. Gu, C. Deng, W. Wang, S. Zhang, Z. Yan, and J. Zhou CosyVoice 2: scalable streaming speech synthesis with large language models . CoRR abs/2412.10117 . External Links: Link , Document , 2412.10117 Cited by: §4.2 .

Fang et al. (2025) Q. Fang, S. Guo, Y. Zhou, Z. Ma, S. Zhang, and Y. Feng LLaMA-omni: seamless speech interaction with large language models . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 , External Links: Link Cited by: §2.1 .

Fu et al. (2024) C. Fu, H. Lin, Z. Long, Y. Shen, M. Zhao, Y. Zhang, X. Wang, D. Yin, L. Ma, X. Zheng, R. He, R. Ji, Y. Wu, C. Shan, and X. Sun VITA: towards open-source interactive omni multimodal LLM . CoRR abs/2408.05211 . External Links: Link , Document , 2408.05211 Cited by: §1 .

Fu et al. (2025) C. Fu, H. Lin, X. Wang, Y. Zhang, Y. Shen, X. Liu, H. Cao, Z. Long, H. Gao, K. Li, L. Ma, X. Zheng, R. Ji, X. Sun, C. Shan, and R. He VITA-1.5: towards gpt-4o level real-time vision and speech interaction . CoRR abs/2501.01957 . External Links: Link , Document , 2501.01957 Cited by: §1 , §1 , §4.1 .

Gao et al. (2025) H. Gao, H. Shao, X. Wang, C. Qiu, Y. Shen, S. Cai, Y. Shi, Z. Xu, Z. Long, Y. Zhang, S. Dong, C. Fu, K. Li, L. Ma, and X. Sun LUCY: linguistic understanding and control yielding early stage of her . CoRR abs/2501.16327 . External Links: Link , Document , 2501.16327 Cited by: §1 .

Hoffmann et al. (2022) J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L. Sifre Training compute-optimal large language models . In Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS ’22 , Red Hook, NY, USA . External Links: ISBN 9781713871088 Cited by: §1 .

Hu et al. (2025) K. Hu, E. Hosseini-Asl, C. Chen, E. Casanova, S. Ghosh, P. Zelasko, Z. Chen, J. Li, J. Balam, and B. Ginsburg Efficient and direct duplex modeling for speech-to-speech language model . In 26th Annual Conference of the International Speech Communication Association, Interspeech 2025, Rotterdam, The Netherlands, 17-21 August 2025 , O. Scharenborg, C. Oertel, and K. Truong (Eds.) , External Links: Link , Document Cited by: §1 .

Hurst et al. (2024) O. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Mkadry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. Passos, A. Kirillov, A. Christakis, A. Conneau, A. Kamali, A. Jabri, A. Moyer, A. Tam, A. Crookes, A. Tootoochian, A. Tootoonchian, A. Kumar, A. Vallone, A. Karpathy, A. Braunstein, A. Cann, A. Codispoti, A. Galu, A. Kondrich, A. Tulloch, A. Mishchenko, A. Baek, A. Jiang, A. Pelisse, A. Woodford, A. Gosalia, A. Dhar, A. Pantuliano, A. Nayak, A. Oliver, B. Zoph, B. Ghorbani, B. Leimberger, B. Rossen, B. Sokolowsky, B. Wang, B. Zweig, B. Hoover, B. Samic, B. McGrew, B. Spero, B. Giertler, B. Cheng, B. Lightcap, B. Walkin, B. Quinn, B. Guarraci, B. Hsu, B. Kellogg, B. Eastman, C. Lugaresi, C. L. Wainwright, C. Bassin, C. Hudson, C. Chu, C. Nelson, C. Li, C. J. Shern, C. Conger, C. Barette, C. Voss, C. Ding, C. Lu, C. Zhang, C. Beaumont, C. Hallacy, C. Koch, C. Gibson, C. Kim, C. Choi, C. McLeavey, C. Hesse, C. Fischer, C. Winter, C. Czarnecki, C. Jarvis, C. Wei, C. Koumouzelis, D. Sherburn, D. Kappler, D. Levin, D. Levy, D. Carr, D. Farhi, D. Mély, D. Robinson, D. Sasaki, D. Jin, D. Valladares, D. Tsipras, D. Li, P. D. Nguyen, D. Findlay, E. Oiwoh, E. Wong, E. Asdar, E. Proehl, E. M. Yang, E. Antonow, E. Kramer, E. Peterson, E. Sigler, E. Wallace, E. Brevdo, E. Mays, F. Khorasani, F. P. Such, F. Raso, F. Zhang, F. von Lohmann, F. Sulit, G. Goh, G. Oden, G. Salmon, G. Starace, G. Brockman, H. Salman, H. Bao, H. Hu, H. Wong, H. Wang, H. Schmidt, H. Whitney, H. Jun, H. Kirchner, H. P. de Oliveira Pinto, H. Ren, H. Chang, H. W. Chung, I. Kivlichan, I. R. O’Connell, I. Osband, I. Silber, I. Sohl, I. Okuyucu, I. Lan, I. Kostrikov, I. Sutskever, I. Kanitscheider, I. Gulrajani, J. Coxon, J. Menick, J. W. Pachocki, J. Aung, J. Betker, J. Crooks, J. Lennon, J. R. Kiros, J. Leike, J. Park, J. Kwon, J. Phang, J. Teplitz, J. Wei, J. Wolfe, J. Chen, J. Harris, J. Varavva, J. G. Lee, J. Shieh, J. Lin, J. Yu, J. Weng, J. Tang, J. Yu, J. Jang, J. Q. Candela, J. Beutler, J. Landers, J. Parish, J. Heidecke, J. Schulman, J. Lachman, J. Mckay, J. Uesato, J. Ward, J. W. Kim, J. Huizinga, J. Sitkin, J. Kraaijeveld, J. Gross, J. Kaplan, J. Snyder, J. Achiam, J. Jiao, J. Lee, J. Zhuang, J. Harriman, K. Fricke, K. Hayashi, K. Singhal, K. Shi, K. Karthik, K. Wood, K. Rimbach, K. Hsu, K. Nguyen, K. Gu-Lemberg, K. Button, K. Liu, K. Howe, K. Muthukumar, K. Luther, L. Ahmad, L. Kai, L. Itow, L. Workman, L. Pathak, L. Chen, L. Jing, L. Guy, L. Fedus, L. Zhou, L. Mamitsuka, L. Weng, L. McCallum, L. Held, O. Long, L. Feuvrier, L. Zhang, L. Kondraciuk, L. Kaiser, L. Hewitt, L. Metz, L. Doshi, M. Aflak, M. Simens, M. Boyd, M. Thompson, M. Dukhan, M. Chen, M. Gray, M. Hudnall, M. Zhang, M. Aljubeh, M. Litwin, M. Zeng, M. Johnson, M. Shetty, M. M. Gupta, M. Shah, M. A. Yatbaz, M. Yang, M. Zhong, M. Glaese, M. Chen, M. Janner, M. Lampe, M. Petrov, M. Wu, M. Wang, M. Fradin, M. Pokrass, M. Castro, M. Castro, M. Pavlov, M. Brundage, M. Wang, M. Khan, M. Murati, M. Bavarian, M. Lin, M. Yesildal, N. Soto, N. Gimelshein, N. Cone, N. M. Staudacher, N. Summers, N. LaFontaine, N. Chowdhury, N. Ryder, N. Stathas, N. Turley, N. A. Tezak, N. Felix, N. Kudige, N. S. Keskar, N. Deutsch, N. Bundick, N. Puckett, O. Nachum, O. E. Okelola, O. Boiko, O. Murk, O. Jaffe, O. Watkins, O. Godement, O. Campbell-Moore, P. Chao, P. McMillan, P. Belov, P. Su, P. Bak, P. Bakkum, P. Deng, P. Dolan, P. Hoeschele, P. Welinder, P. Tillet, P. Pronin, P. Tillet, P. Dhariwal, Q. Yuan, R. Dias, R. Lim, R. Arora, R. Troll, R. Lin, R. G. Lopes, R. Puri, R. Miyara, R. H. Leike, R. Gaubert, R. Zamani, R. B. Wang, R. Donnelly, R. Honsby, R. Smith, R. Sahai, R. Ramchandani, R. Huet, R. Carmichael, R. Zellers, R. Chen, R. Chen, R. R. Nigmatullin, R. Cheu, S. Jain, S. Altman, S. Schoenholz, S. Toizer, S. Miserendino, S. Agarwal, S. Culver, S. Ethersmith, S. Gray, S. Grove, S. Metzger, S. Hermani, S. Jain, S. Zhao, S. Wu, S. Jomoto, S. Wu, S. Xia, S. Phene, S. Papay, S. Narayanan, S. Coffey, S. Y. L. Lee, S. Hall, S. Balaji, T. Broda, T. Stramer, T. Xu, T. Gogineni, T. Christianson, T. Sanders, T. Patwardhan, T. Cunninghman, T. Degry, T. Dimson, T. Raoux, T. Shadwell, T. Zheng, T. Underwood, T. Markov, T. Sherbakov, T. Rubin, T. Stasi, T. Kaftan, T. Heywood, T. A. Peterson, T. Walters, T. Eloundou, V. Qi, V. Moeller, V. Monaco, V. Kuo, V. Fomenko, W. H. Chang, W. Zheng, W. Zhou, W. Manassra, W. Sheu, W. Zaremba, Y. Patil, Y. Qian, Y. Kim, Y. Cheng, Y. Zhang, Y. He, Y. Zhang, Y. Jin, Y. Dai, and Y. Malkov GPT-4o system card . External Links: Link Cited by: §1 .

Ji et al. (2024) S. Ji, Y. Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Cheng, X. Yang, Z. Wang, Q. Yang, J. Li, Y. Jiang, J. He, Y. Chu, J. Xu, and Z. Zhao WavChat: A survey of spoken dialogue models . CoRR abs/2411.13577 . External Links: Link , Document , 2411.13577 Cited by: §1 .

Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models . CoRR abs/2001.08361 . External Links: Link , 2001.08361 Cited by: §1 .

Lei et al. (2026) Y. Lei, S. He, J. Hu, D. Zhang, X. Luo, D. Zhu, S. Feng, R. Liu, J. He, Y. Sun, H. Wu, and H. Wang MoE adapter for large audio language models: sparsity, disentanglement, and gradient-conflict-free . CoRR abs/2601.02967 . External Links: Link , Document , 2601.02967 Cited by: §9 .

Li et al. (2025a) G. Li, C. Wang, H. Xue, S. Wang, D. Gao, Z. Zhang, Y. Lin, W. Li, L. Xiao, Z. Fu, and L. Xie Easy turn: integrating acoustic and linguistic modalities for robust turn-taking in full-duplex spoken dialogue systems . CoRR abs/2509.23938 . External Links: Link , Document , 2509.23938 Cited by: §2.2 .

Li et al. (2025b) Y. Li, X. Chen, S. Jiang, H. Shi, Z. Liu, X. Zhang, N. Deng, Z. Xu, Y. Ma, M. Zhang, et al. Uni-moe-2.0-omni: scaling language-centric omnimodal large model with advanced moe, training and data . arXiv preprint arXiv:2511.12609 . Cited by: §1 .

Li et al. (2025c) Y. Li, Z. Liu, Z. Li, X. Zhang, Z. Xu, X. Chen, H. Shi, S. Jiang, X. Wang, J. Wang, et al. Perception, reason, think, and plan: a survey on large multimodal reasoning models . arXiv preprint arXiv:2505.04921 . Cited by: §2.1 .

Liao et al. (2025) B. Liao, Y. Xu, J. Ou, K. Yang, W. Jian, P. Wan, and D. Zhang FlexDuo: A pluggable system for enabling full-duplex capabilities in speech dialogue systems . CoRR abs/2502.13472 . External Links: Link , Document , 2502.13472 Cited by: §2.2 .

Lin et al. (2025a) G. Lin, J. Lian, T. Li, Q. Wang, G. Anumanchipalli, A. H. Liu, and H. Lee Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities . CoRR abs/2503.04721 . External Links: Link , Document , 2503.04721 Cited by: §1 , §4.5 .

Lin et al. (2022) T. Lin, Y. Wu, F. Huang, L. Si, J. Sun, and Y. Li Duplex conversation: towards human-like interaction in spoken dialogue systems . In KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022 , A. Zhang and H. Rangwala (Eds.) , pp. 3299–3308 . External Links: Link , Document Cited by: §1 .

Lin et al. (2025b) Y. Lin, C. Yang, W. Chen, C. Li, C. Huang, X. Chen, and H. Lee A preliminary exploration with gpt-4o voice mode . CoRR abs/2502.09940 . External Links: Link , Document , 2502.09940 Cited by: §1 .

Liu et al. (2025) C. Liu, M. Su, Y. Xiang, Y. Huang, Y. Yang, K. Zhang, and M. Fan Toward enabling natural conversation with older adults via the design of llm-powered voice agents that support interruptions and backchannels . In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI 2025, YokohamaJapan, 26 April 2025- 1 May 2025 , N. Yamashita, V. Evers, K. Yatani, S. X. Ding, B. Lee, M. Chetty, and P. O. T. Dugas (Eds.) , pp. 163:1–163:22 . External Links: Link , Document Cited by: §1 .

Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization . In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , External Links: Link Cited by: §4.3 .

Mitsui et al. (2024) K. Mitsui, K. Mitsuda, T. Wakatsuki, Y. Hono, and K. Sawada PSLM: parallel generation of text and speech with llms for low-latency spoken dialogue systems . In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024 , Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.) , pp. 2692–2700 . External Links: Link , Document Cited by: §2.1 .

Nguyen et al. (2023) T. A. Nguyen, E. Kharitonov, J. Copet, Y. Adi, W. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, and E. Dupoux Generative spoken dialogue language modeling . Trans. Assoc. Comput. Linguistics 11 , pp. 250–266 . External Links: Link , Document Cited by: §1 , §2.2 , §4.1 .

Peng et al. (2025) Y. Peng, Y. Chao, D. Ng, Y. Ma, C. Ni, B. Ma, and E. S. Chng FD-bench: A full-duplex benchmarking pipeline designed for full duplex spoken dialogue systems . In 26th Annual Conference of the International Speech Communication Association, Interspeech 2025, Rotterdam, The Netherlands, 17-21 August 2025 , O. Scharenborg, C. Oertel, and K. Truong (Eds.) , External Links: Link , Document Cited by: §4.5 .

Radford et al. (2023) A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever Robust speech recognition via large-scale weak supervision . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett (Eds.) , Proceedings of Machine Learning Research , Vol. 202 , pp. 28492–28518 . External Links: Link Cited by: §4.4 .

Ruder (2017) S. Ruder An overview of multi-task learning in deep neural networks . CoRR abs/1706.05098 . External Links: Link , 1706.05098 Cited by: §1 .

Sacks et al. (1974) H. Sacks, E. A. Schegloff, and G. Jefferson A simplest systematics for the organization of turn-taking for conversation . Language 50 ( 4 ), pp. 696–735 . External Links: ISSN 00978507, 15350665 , Link Cited by: §1 , §1 .

Saeki et al. (2022) T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari UTMOS: utokyo-sarulab system for voicemos challenge 2022 . In 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022 , H. Ko and J. H. L. Hansen (Eds.) , pp. 4521–4525 . External Links: Link , Document Cited by: §4.6 .

Sener and Koltun (2018) O. Sener and V. Koltun Multi-task learning as multi-objective optimization . In Advances in Neural Information Processing Systems , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.) , Vol. 31 , pp.  . External Links: Link Cited by: §1 .

Veluri et al. (2024) B. Veluri, B. N. Peloquin, B. Yu, H. Gong, and S. Gollakota Beyond turn-based interfaces: synchronous llms as full-duplex dialogue agents . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 , Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.) , pp. 21390–21402 . External Links: Link , Document Cited by: §2.2 .

Wang et al. (2025) X. Wang, Y. Li, C. Fu, Y. Zhang, Y. Shen, L. Xie, K. Li, X. Sun, and L. Ma Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen LLM . In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025 , External Links: Link Cited by: §1 , §2.1 , §4.1 .

Wu et al. (2025) B. Wu, C. Yan, C. Hu, C. Yi, C. Feng, F. Tian, F. Shen, G. Yu, H. Zhang, J. Li, M. Chen, P. Liu, W. You, X. T. Zhang, X. Li, X. Yang, Y. Deng, Y. Huang, Y. Li, Y. Zhang, Z. You, B. Li, C. Wan, H. Hu, J. Zhen, S. Chen, S. Yuan, X. Zhang, Y. Jiang, Y. Zhou, Y. Yang, B. Li, B. Ma, C. Song, D. Pang, G. Hu, H. Sun, K. An, N. Wang, S. Gao, W. Ji, W. Li, W. Sun, X. Wen, Y. Ren, Y. Ma, Y. Lu, B. Wang, B. Li, C. Miao, C. Liu, C. Xu, D. Shi, D. Hu, D. Wu, E. Liu, G. Huang, G. Yan, H. Zhang, N. Hao, H. Jia, H. Zhou, J. Sun, J. Wu, J. Wu, J. Yang, J. Yang, J. Lin, K. Li, L. Yang, L. Shi, L. Zhou, L. Gu, M. Li, M. Li, M. Li, N. Wu, Q. Han, Q. Tan, S. Pang, S. Fan, S. Liu, T. Cao, W. Lu, W. He, W. Xie, X. Zhao, X. Li, Y. Yu, Y. Yang, Y. Liu, Y. Lu, Y. Wang, Y. Ding, Y. Liang, Y. Lu, Y. Luo, Y. Yin, Y. Zhan, and Y. Zhang Step-audio 2 technical report . CoRR abs/2507.16632 . External Links: Link , Document , 2507.16632 Cited by: §2.1 , §3.1 , §3.1 , §3.2 .

Xie and Wu (2024a) Z. Xie and C. Wu Mini-omni2: towards open-source gpt-4o with vision, speech and duplex capabilities . CoRR abs/2410.11190 . External Links: Link , Document , 2410.11190 Cited by: §2.2 .

Xie and Wu (2024b) Z. Xie and C. Wu Mini-omni: language models can hear, talk while thinking in streaming . CoRR abs/2408.16725 . External Links: Link , Document , 2408.16725 Cited by: §2.1 .

Xu et al. (2025) J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin Qwen2.5-omni technical report . CoRR abs/2503.20215 . External Links: Link , Document , 2503.20215 Cited by: §2.1 .

Xu et al. (2023) P. Xu, X. Zhu, and D. A. Clifton Multimodal learning with transformers: A survey . IEEE Trans. Pattern Anal. Mach. Intell. 45 ( 10 ), pp. 12113–12132 . External Links: Link , Document Cited by: §1 .

Yao et al. (2025) Y. Yao, X. Li, X. Jiang, X. Fang, N. Yu, W. Ma, A. Sun, and Y. Wang FLM-audio: natural monologues improves native full-duplex chatbots via dual training . CoRR abs/2509.02521 . External Links: Link , Document , 2509.02521 Cited by: §4.1 .

Yu et al. (2020) T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn Gradient surgery for multi-task learning . In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.) , Vol. 33 , pp. 5824–5836 . External Links: Link Cited by: §1 .

Yu et al. (2025) W. Yu, S. Wang, X. Yang, X. Chen, X. Tian, J. Zhang, G. Sun, L. Lu, Y. Wang, and C. Zhang SALMONN-omni: a standalone speech llm without codec injection for full-duplex conversation . In Advances in Neural Information Processing Systems , Cited by: §2.2 .

Zeghidour et al. (2022) N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi SoundStream: an end-to-end neural audio codec . IEEE ACM Trans. Audio Speech Lang. Process. 30 , pp. 495–507 . External Links: Link , Document Cited by: §2.1 .

Zeng et al. (2025) A. Zeng, Z. Du, M. Liu, L. Zhang, S. Jiang, Y. Dong, and J. Tang Scaling speech-text pre-training with synthetic interleaved data . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 , External Links: Link Cited by: §1 , §2.1 .

Zhan et al. (2024) J. Zhan, J. Dai, J. Ye, Y. Zhou, D. Zhang, Z. Liu, X. Zhang, R. Yuan, G. Zhang, L. Li, H. Yan, J. Fu, T. Gui, T. Sun, Y. Jiang, and X. Qiu AnyGPT: unified multimodal LLM with discrete sequence modeling . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 , L. Ku, A. Martins, and V. Srikumar (Eds.) , pp. 9637–9662 . External Links: Link , Document Cited by: §1 .

Zhang et al. (2023) D. Zhang, S. Li, X. Zhang, J. Zhan, P. Wang, Y. Zhou, and X. Qiu SpeechGPT: empowering large language models with intrinsic cross-modal conversational abilities . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , H. Bouamor, J. Pino, and K. Bali (Eds.) , pp. 15757–15773 . External Links: Link , Document Cited by: §2.1 .

Zhang et al. (2025a) H. Zhang, W. Li, R. Chen, V. Kothapally, M. Yu, and D. Yu LLM-enhanced dialogue management for full-duplex spoken dialogue systems . CoRR abs/2502.14145 . External Links: Link , Document , 2502.14145 Cited by: §2.2 .

Zhang et al. (2025b) Q. Zhang, L. Cheng, C. Deng, Q. Chen, W. Wang, S. Zheng, J. Liu, H. Yu, C. Tan, Z. Du, and S. Zhang OmniFlatten: an end-to-end GPT model for seamless voice conversation . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2025 , W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.) , pp. 14570–14580 . External Links: Link Cited by: §2.2 .

Zhang et al. (2025c) X. L. T. D. Zhang, G. Wang, J. Xue, K. Fang, L. Zhao, R. Ma, S. Ren, S. Liu, T. Guo, W. Zhuang, X. Zhang, X. Song, Y. Yan, Y. He, Cici, B. Shen, C. Zhu, C. Ma, C. Chen, H. Chen, J. Li, L. Li, M. Zhu, P. Li, Q. Wang, S. Deng, W. Xiong, W. Huang, W. Yang, Y. Jiang, Y. Yang, Y. Tian, Y. Ma, Y. Yu, Z. Zhang, Z. Yue, B. Xiao, B. Xia, B. Gao, B. Ye, C. Cai, C. Liu, C. He, C. Li, D. Zhu, D. Zhang, F. Shi, G. Wang, H. Zhang, H. Lv, H. Li, H. Tian, H. Qu, H. Xu, H. Zhang, H. Liu, J. Duo, J. Zuo, J. Wei, J. Xiao, J. Dong, J. Shi, J. Hu, K. Bao, K. Zhou, L. Zhang, M. Chen, N. Chen, P. Zhang, Q. Chen, Q. Wang, R. Li, S. Liu, S. Wang, S. Li, S. Yu, S. Cao, S. Chen, S. Gu, W. Wang, W. Ma, X. Deng, X. Yong, X. Zhang, X. Wang, Y. Song, Y. Zhao, Y. Zhao, Y. Gao, Y. Cheng, Y. Tu, Y. Wang, Z. Huang, Z. Tang, Z. Lin, Z. Song, Z. Xu, Z. Zheng, and Z. Jiang MiMo-audio: audio language models are few-shot learners . ArXiv abs/2512.23808 . External Links: Link Cited by: §2.2 .

Zhang et al. (2024) X. Zhang, Y. Chen, S. Hu, X. Han, Z. Xu, Y. Xu, W. Zhao, M. Sun, and Z. Liu Beyond the turn-based game: enabling real-time conversations with duplex models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 , Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.) , pp. 11543–11557 . External Links: Link , Document Cited by: §2.2 .

Appendix A Layer Ablation

附录 A 层数消融

Lychee-FD 的层数消融:准确率与计算量

Figure 6: Layer ablation study on model performance and computation cost. Accuracy on LlamaQ saturates after 4 separated layers, while computational overhead grows linearly. This indicates that 4 layers represent the optimal trade-off between performance gains and inference efficiency.

图 6:层数消融研究:模型表现与计算开销。LlamaQ 上的准确率在分离 4 层后趋于饱和,而计算开销继续线性增长。这说明 4 层是性能收益与推理效率之间的最佳折中。

To explore the impact of the number of separated layers, we conduct a layer ablation study on both model performance on LlamaQ and static inference cost for speech, as illustrated in Figure 6. We observe that accuracy follows a steep upward trajectory in the initial stages, surging from 36.0 to 65.4 as the depth increases to 4 layers. This indicates that a relatively shallow separation is sufficient to resolve the primary modality conflicts. However, beyond this point, the performance gain saturates, yielding diminishing returns, while the computational overhead continues to grow linearly. Consequently, we identify 4 layers as the optimal configuration. This choice represents the most favorable trade-off, securing the vast majority of the performance gain while minimizing the additional parameter budget, thereby ensuring high inference efficiency.

为考察分离层数的影响,我们在 LlamaQ 上的模型表现和语音的静态推理开销上都做了层数消融,见图 6。我们观察到,准确率在最初阶段上升很快,分离深度增加到 4 层时,从 36.0 升到 65.4。这说明较浅的分离就足以化解主要的模态冲突。过了这一点,性能收益趋于饱和,回报递减,计算开销却继续线性增长。因此,我们把 4 层定为最优配置。这一选择代表了最有利的折中:拿下绝大部分性能收益,同时尽量压低额外参数预算,从而保证高推理效率。

Appendix B Case Study

附录 B 案例研究

To intuitively demonstrate the interaction quality of Lychee-FD, we move beyond isolated examples and analyze user interventions through two basic dimensions: the presence of voice activity and the user’s actual intent to intervene. Since the default state (no voice activity and no intent) simply requires the model to continue listening or speaking, we focus on the remaining three representative scenarios: intentional interruption (voice with strong intent to intervene), passive backchannel (voice with intent to encourage continuation), and side-talk (voice with no intent towards the agent).

为直观展示 Lychee-FD 的交互质量,我们不再只看孤立例子,而是从两个基本维度分析用户介入:是否存在语音活动,以及用户是否真的打算介入。默认状态(没有语音活动、也没有介入意图)只要求模型继续听或继续说,因此我们聚焦其余三种有代表性的情形:有意插话(有语音,且有强烈介入意图)、被动附和(有语音,意图是鼓励对方继续),以及边聊(有语音,但并不是对智能体说的)。

Lychee-FD 在讲碳对面方面对插话与附和的案例

Figure 7: A case study demonstrating Lychee-FD’s capability in handling complex turn-taking dynamics. Lychee-FD successfully generates backchannels, halts immediately upon interruption, and provides a contextually accurate response to the user’s specific query without misinterpreting passive feedback.

图 7:案例研究,展示 Lychee-FD 处理复杂话轮动态的能力。Lychee-FD 能够成功生成附和,在遇到插话时立刻停下,并对用户的具体问题给出符合语境的准确回答,而不会把被动反馈误判为插话。

Lychee-FD 对边聊(披萨)误接话的错误分析

Figure 8: An error analysis illustrating a common limitation in handling side-talk during full-duplex interaction. While the model correctly detects voice activity and halts its speech, it fails to recognize that the user’s utterance is directed at a third party, highlighting a key area for future community research.

图 8:错误分析,说明全双工交互中处理边聊时的常见局限。模型虽然正确检测到语音活动并停止了自己的发言,却没能认识到用户的话是对第三者说的,这是社区后续研究的一个关键方向。

B Case Study: Interruption and Backchannel

B 案例研究:插话与附和

In Figure 7, we present a real-world conversation sample that illustrates the first two scenarios. As the model begins explaining a recipe, it first employs a natural backchannel (“Uh-huh”) to acknowledge the user’s start. Crucially, the model exhibits two key capabilities when facing different user interventions: (1) Intentional Interruption: When the user interrupts with a specific clarification question (“what exactly is guanciale?”), Lychee-FD halts its speech output almost instantaneously, avoiding the awkward talking-over phenomenon common in half-duplex systems. (2) Passive Backchannel: Later, as the model explains the definition of guanciale, the user interjects with a short backchannel (“I see”). Here, Lychee-FD demonstrates precise intent understanding. Instead of misinterpreting this acoustic signal as a barge-in command—a common failure in fully-shared models due to semantic dilution—the model correctly identifies it as a passive signal of agreement. Benefiting from our proposed Semantic Alignment Channel, which preserves robust language modeling during simultaneous listening, Lychee-FD seamlessly resumes its explanation without unnecessary pauses or topic fragmentation. This interaction confirms that our Lychee-FD effectively maintains the model’s language capabilities even during rapid turn-switching, enabling a fluid, seamless, and truly natural conversational experience.

在图 7 中,我们给出一个真实对话样本,用来说明前两种情形。模型开始讲解一道菜谱时,先用一句自然的附和(“Uh-huh”)回应了用户的开口。面对不同的用户介入,模型表现出两项关键能力:(1)有意插话: 当用户插入一个具体的澄清问题(“what exactly is guanciale?”,意式培根是什么)时,Lychee-FD 几乎立刻停止语音输出,避免了半双工系统常见的双方抢话。(2)被动附和: 随后,模型在解释 guanciale 的定义时,用户插入一句短附和(“I see”)。这里 Lychee-FD 表现出准确的意图理解。它没有把这一声学信号误判为抢话指令——这是完全共享模型因语义稀释而常见的失败——而是正确识别为表示认同的被动信号。得益于我们提出的语义对齐通道在边听边说时仍能保持稳健的语言建模,Lychee-FD 顺畅地继续讲解,没有不必要的停顿,也没有把话题割裂。这一交互说明,即便在快速切换话轮时,Lychee-FD 仍能保住模型的语言能力,从而实现流畅、连贯、真正自然的对话体验。

Appendix C Error Analysis

附录 C 错误分析

Despite the strong performance in standard interactions, we observe certain limitations in handling complex acoustic environments, particularly regarding side-talk. In a native full-duplex setting, the model continuously processes incoming audio. As shown in Figure 8, when the user briefly speaks to a third party in the background (e.g., asking a roommate about food), the model incorrectly interprets this background conversation as a direct interruption. Consequently, it halts its current explanation and attempts to respond to the irrelevant query.

尽管在标准交互中表现很好,我们仍观察到模型在处理复杂声学环境时有一些局限,尤其是边聊。在原生全双工设定下,模型会持续处理输入音频。如图 8 所示,当用户短暂地对后台第三者说话(例如问室友吃不吃饭)时,模型错误地把这段背景对话当成直接插话,于是停下当前讲解,并试图回答这个无关的问题。

This error indicates that while Lychee-FD is highly responsive to voice activity, its ability to distinguish between user-to-agent commands and user-to-human side-talk remains constrained. We argue that this is not an architectural limitation of our framework, but rather a common challenge for current native SLMs, largely due to the lack of multi-speaker intent annotations in existing data synthesis paradigms. By highlighting this scenario, we hope to provide a blueprint for the community. Future work will focus on integrating intent detection or utilizing prosodic cues to improve the model’s robustness in open-mic, multi-speaker environments.

这一错误说明,Lychee-FD 对语音活动虽然高度灵敏,但在区分“对智能体的指令”和“对人的边聊”上仍然受限。我们认为,这并不是我们框架的架构局限,而是当前原生 SLM 的共同挑战,很大程度上是因为现有数据合成范式缺少多方意图标注。指出这一情形,是希望给社区提供一个参照。后续工作将侧重于融入意图检测,或利用韵律线索,提高模型在开放麦克风、多方场景中的稳健性。

Appendix D Global Gradient Influence Analysis

附录 D 全局梯度影响分析

全共享基线与分层参数分离的全局梯度影响分数

Figure 9: Global gradient influence scores among Text and Speech tasks. Left: The fully shared baseline suffers from severe destructive interference between semantic and acoustic modeling (negative scores). Right: Our Hierarchical Parameter Separation not only eliminates this conflict but also fosters constructive synergy (positive scores) between modalities.

图 9:文本任务与语音任务之间的全局梯度影响分数。 左:完全共享的基线在语义建模与声学建模之间出现严重的破坏性干扰(负分)。右:我们的分层参数分离不仅消除了这一冲突,还促进了模态之间的建设性协同(正分)。

To further validate the effectiveness of our proposed Hierarchical Parameter Separation, we investigate modality interference from a causal perspective. Inspired by Lei et al. (2026), we quantify how updating parameters for one task causally affects the performance of another. Specifically, we calculate the normalized global Influence Score Im→iI_{m\rightarrow i}Im→i​ of task mmm on task iii (m,i∈{T,A}m,i\in\{T,A\}m,i∈{T,A}):

为进一步验证分层参数分离的有效性,我们从因果角度考察模态干扰。受 Lei et al. (2026) 启发,我们量化为一项任务更新参数时,会怎样因果地影响另一项任务的表现。具体而言,我们计算任务 mmm 对任务 iii 的归一化全局影响分数 Im→iI_{m\rightarrow i}Im→i​(m,i∈{T,A}m,i\in\{T,A\}m,i∈{T,A}):

Im→i=Li(θ)−Li(θ−ηgm)Li(θ)−Li(θ−ηgi),(9)I_{m\rightarrow i}=\frac{\mathcal{L}_{i}(\theta)-\mathcal{L}_{i}(\theta-\eta\mathbf{g}_{m})}{\mathcal{L}_{i}(\theta)-\mathcal{L}_{i}(\theta-\eta\mathbf{g}_{i})}, \tag{9}Im→i​=Li​(θ)−Li​(θ−ηgi​)Li​(θ)−Li​(θ−ηgm​)​,(9)

where θ\thetaθ denotes the global model parameters, and gm=∇θLm\mathbf{g}_{m}=\nabla_{\theta}\mathcal{L}_{m}gm​=∇θ​Lm​ is the gradient derived solely from the objective of task mmm. The denominator represents the loss reduction when task iii is updated using its own gradient, serving as a normalization factor. A negative score (Im→i<0I_{m\rightarrow i}<0Im→i​<0) indicates destructive interference, meaning that optimizing task mmm degrades the performance of task iii. Conversely, a positive score (Im→i>0I_{m\rightarrow i}>0Im→i​>0) implies constructive synergy.

其中 θ\thetaθ 表示全局模型参数,gm=∇θLm\mathbf{g}_{m}=\nabla_{\theta}\mathcal{L}_{m}gm​=∇θ​Lm​ 是仅由任务 mmm 的目标得到的梯度。分母表示任务 iii 用自身梯度更新时的损失下降,用作归一化因子。负分(Im→i<0I_{m\rightarrow i}<0Im→i​<0)表示破坏性干扰,也就是优化任务 mmm 会损害任务 iii 的表现。相反,正分(Im→i>0I_{m\rightarrow i}>0Im→i​>0)表示建设性协同。

Figure 9 visualizes the influence scores between the text and speech tasks. In the fully shared baseline (left), the update of the text task exerts a negative influence on the speech task (-0.129). This empirically confirms our hypothesis: forcing conflicting modalities to update within a shared parameter space causes destructive interference, where acoustic learning actively corrupts semantic modeling. In contrast, under our Hierarchical Parameter Separation (right), this destructive interference is entirely resolved. Notably, the influence score of text on speech shifts to a positive value (0.205). Furthermore, the constructive synergy from speech to text is significantly amplified (from 0.312 to 0.773). This demonstrates that our architecture effectively disentangles conflicting optimization dynamics, allowing the model to acquire robust speech capabilities while mutually reinforcing its language intelligence.

图 9 可视化了文本任务与语音任务之间的影响分数。在完全共享的基线中(左),文本任务的更新对语音任务产生负向影响(−0.129)。这从经验上证实了我们的假设:强迫相互冲突的模态在共享参数空间里更新,会造成破坏性干扰,声学学习会主动损害语义建模。相比之下,在我们的分层参数分离下(右),这种破坏性干扰被完全化解。值得注意的是,文本对语音的影响分数转为正值(0.205)。此外,从语音到文本的建设性协同也被明显放大(从 0.312 升到 0.773)。这说明我们的架构有效拆开了相互冲突的优化动力学,使模型在获得稳健语音能力的同时,也能相互加强其语言智力。

Appendix E Real-Time Inference Algorithm

附录 E 实时推理算法

Deploying Lychee-FD for real-time full-duplex interaction presents a unique system-level challenge. Standard LLM inference engines, such as vLLM, are fundamentally designed around a linear layer topology (L1→L2→⋯→LDL_{1}\rightarrow L_{2}\rightarrow\dots\rightarrow L_{D}L1​→L2​→⋯→LD​) and a single autoregressive output stream. However, our proposed hierarchical architecture requires the simultaneous generation of text, audio, and control signals. Forcing this multi-head architecture into a single GPU or a linear pipeline would result in the sequential execution of the specialized heads, introducing severe latency bottlenecks that disrupt the fluidity of real-time conversation.

把 Lychee-FD 部署到实时全双工交互中,会带来独特的系统级挑战。vLLM 等标准 LLM 推理引擎,从根本上是按线性层拓扑(L1→L2→⋯→LDL_{1}\rightarrow L_{2}\rightarrow\dots\rightarrow L_{D}L1​→L2​→⋯→LD​)和单一自回归输出流来设计的。而我们提出的分层架构需要同时生成文本、音频和控制信号。如果把这种多头架构硬塞进单卡或线性流水线,专用头就只能串行执行,从而引入严重的延迟瓶颈,破坏实时对话的流畅性。

To bridge the gap between theoretical modality decoupling and practical deployment, we introduce Directed Acyclic Graph Pipeline Parallelism (DAG-PP) by customizing the vLLM engine. Instead of linear tensor passing, we implement a 1-to-N tensor broadcast mechanism at the branching point of the shared backbone. As illustrated in Algorithm 1, the intermediate hidden states (Hshared\mathbf{H}_{\mathrm{shared}}Hshared​) are broadcasted via NCCL to multiple GPUs. This allows the Semantic, Acoustic, and Control heads to execute strictly in parallel across different devices. Crucially, this parallel execution ensures that the effective model depth on the critical path remains unchanged compared to the half-duplex backbone, significantly reducing the inference latency bottleneck of multi-head architectures. A distributed synchronization barrier is then employed to collect the multi-stream logits for synchronous sampling before the next autoregressive step.

为了把理论上的模态解耦落到实际部署,我们通过定制 vLLM 引擎,引入有向无环图流水线并行(Directed Acyclic Graph Pipeline Parallelism, DAG-PP)。我们不再做线性的张量传递,而是在共享主干的分叉点实现 1 对 N 的张量广播机制。如算法 1 所示,中间隐状态(Hshared\mathbf{H}_{\mathrm{shared}}Hshared​)通过 NCCL 广播到多张 GPU。这样,语义头、声学头和控制头就可以在不同设备上严格并行执行。关键的是,这种并行执行保证关键路径上的有效模型深度与半双工骨干相同,从而显著减轻多头架构的推理延迟瓶颈。 随后再用分布式同步屏障收集多路 logits,以便在下一步自回归之前做同步采样。

This algorithm-system co-design demonstrates that our hierarchical parameter separation not only resolves gradient conflicts during training but also unlocks strict physical parallelism during inference. By hiding the computational overhead of multi-modal generation behind parallel execution, Lychee-FD achieves the state-of-the-art interaction intelligence while strictly adhering to the ultra-low latency constraints of full-duplex spoken dialogues.

这种算法与系统的协同设计说明,分层参数分离不仅在训练时化解了梯度冲突,也在推理时打开了严格的物理并行。通过把多模态生成的计算开销隐藏到并行执行之后,Lychee-FD 在严格满足全双工口语对话极低延迟约束的同时,达到了当前最优的交互智力。

Input: Text tokens XT\mathbf{X}^{T}XT, Acoustic tokens XA\mathbf{X}^{A}XA, Control tokens XC\mathbf{X}^{C}XC, User speech U\mathbf{U}U
Output: Text logits OT\mathbf{O}^{T}OT, Acoustic logits OA\mathbf{O}^{A}OA, Control logits OC\mathbf{O}^{C}OC
/* Initial Embedding (GPU 0) /
Em←Embeddingm(Xm),∀m∈{T,A,C}\mathbf{E}^{m}\leftarrow\mathrm{Embedding}_{m}(\mathbf{X}^{m}),\quad\forall m\in\{T,A,C\}Em←Embeddingm​(Xm),∀m∈{T,A,C};
EU←AudioEncoder(U)\mathbf{E}^{U}\leftarrow\mathrm{AudioEncoder}(\mathbf{U})EU←AudioEncoder(U);
E←ET+EA+EC+EU\mathbf{E}\leftarrow\mathbf{E}^{T}+\mathbf{E}^{A}+\mathbf{E}^{C}+\mathbf{E}^{U}E←ET+EA+EC+EU;
/
Shared Backbone Execution (GPU 0) /
H(0)←E\mathbf{H}^{(0)}\leftarrow\mathbf{E}H(0)←E;
for l←1l\leftarrow 1l←1 to LsharedL_{\mathrm{shared}}Lshared​ do
H(l)←Flshared(H(l−1))\mathbf{H}^{(l)}\leftarrow\mathcal{F}^{\mathrm{shared}}_{l}(\mathbf{H}^{(l-1)})H(l)←Flshared​(H(l−1));
/
DAG Branching /
NCCL_Broadcast(Hshared→GPUT,GPUA,GPUC\mathbf{H}_{\mathrm{shared}}\rightarrow\mathrm{GPU}_{T},\mathrm{GPU}_{A},\mathrm{GPU}_{C}Hshared​→GPUT​,GPUA​,GPUC​);
/
Specialized Heads Execution /
parallel for
m∈{T,A,C}m\in\{T,A,C\}m∈{T,A,C} on GPUm\mathrm{GPU}_{m}GPUm​
Hm←Hshared\mathbf{H}^{m}\leftarrow\mathbf{H}_{\mathrm{shared}}Hm←Hshared​;
for k←1k\leftarrow 1k←1 to LmL_{m}Lm​ do
Hm←Fkm(Hm)\mathbf{H}^{m}\leftarrow\mathcal{F}^{m}_{k}(\mathbf{H}^{m})Hm←Fkm​(Hm);
/
Distributed Synchronization */
Barrier_Synchronize();
return (OT,OA,OC)(\mathbf{O}^{T},\mathbf{O}^{A},\mathbf{O}^{C})(OT,OA,OC);

Algorithm 1 Directed Acyclic Graph Pipeline Parallelism (DAG-PP)

算法 1 有向无环图流水线并行(DAG-PP)

输入: 文本词元 XT\mathbf{X}^{T}XT,声学词元 XA\mathbf{X}^{A}XA,控制词元 XC\mathbf{X}^{C}XC,用户语音 U\mathbf{U}U
输出: 文本 logits OT\mathbf{O}^{T}OT,声学 logits OA\mathbf{O}^{A}OA,控制 logits OC\mathbf{O}^{C}OC
/* 初始嵌入(GPU 0) /
Em←Embeddingm(Xm),∀m∈{T,A,C}\mathbf{E}^{m}\leftarrow\mathrm{Embedding}_{m}(\mathbf{X}^{m}),\quad\forall m\in\{T,A,C\}Em←Embeddingm​(Xm),∀m∈{T,A,C};
EU←AudioEncoder(U)\mathbf{E}^{U}\leftarrow\mathrm{AudioEncoder}(\mathbf{U})EU←AudioEncoder(U);
E←ET+EA+EC+EU\mathbf{E}\leftarrow\mathbf{E}^{T}+\mathbf{E}^{A}+\mathbf{E}^{C}+\mathbf{E}^{U}E←ET+EA+EC+EU;
/
共享主干执行(GPU 0) /
H(0)←E\mathbf{H}^{(0)}\leftarrow\mathbf{E}H(0)←E;
for l←1l\leftarrow 1l←1 to LsharedL_{\mathrm{shared}}Lshared​ do
H(l)←Flshared(H(l−1))\mathbf{H}^{(l)}\leftarrow\mathcal{F}^{\mathrm{shared}}_{l}(\mathbf{H}^{(l-1)})H(l)←Flshared​(H(l−1));
/
DAG 分叉 /
NCCL_Broadcast(Hshared→GPUT,GPUA,GPUC\mathbf{H}_{\mathrm{shared}}\rightarrow\mathrm{GPU}_{T},\mathrm{GPU}_{A},\mathrm{GPU}_{C}Hshared​→GPUT​,GPUA​,GPUC​);
/
专用头执行 /
parallel for
m∈{T,A,C}m\in\{T,A,C\}m∈{T,A,C} on GPUm\mathrm{GPU}_{m}GPUm​
Hm←Hshared\mathbf{H}^{m}\leftarrow\mathbf{H}_{\mathrm{shared}}Hm←Hshared​;
for k←1k\leftarrow 1k←1 to LmL_{m}Lm​ do
Hm←Fkm(Hm)\mathbf{H}^{m}\leftarrow\mathcal{F}^{m}_{k}(\mathbf{H}^{m})Hm←Fkm​(Hm);
/
分布式同步 */
Barrier_Synchronize();
return (OT,OA,OC)(\mathbf{O}^{T},\mathbf{O}^{A},\mathbf{O}^{C})(OT,OA,OC);

Appendix F Data Synthesis Pipeline Details

附录 F 数据合成流水线细节

To address the scarcity of full-duplex interaction data, we developed an automated data synthesis pipeline. This pipeline orchestrates interactions between a User Agent and an Assistant Agent, managed by a Conversation Conductor. The process explicitly models complex conversational behaviors including interruptions and backchannels.

为缓解全双工交互数据的稀缺,我们开发了一套自动数据合成流水线。该流水线调度**用户智能体(User Agent)与助手智能体(Assistant Agent)之间的交互,并由对话协调器(Conversation Conductor)**管理。过程显式建模插话和附和等复杂对话行为。

F.1 Agent Architecture

F.1 智能体架构

  • • User Agent is initialized with a specific Persona (randomly sampled from a pool of diverse profiles) and a Speaking Style (sampled from 19 distinct styles such as Concise and logical, Impatient, Humorous and witty). The agent is instructed to act authentically rather than helpfully.
  • • 用户智能体(User Agent) 会先被赋予一个特定的人设(Persona,从多样档案池中随机抽样)和一种说话风格(Speaking Style,从 19 种不同风格中抽样,例如简洁而有条理、不耐烦、幽默机智)。该智能体被要求表现得真实,而不是一味帮忙。
  • • Assistant Agent generates responses based on the conversation history to simulate a realistic AI assistant.
  • • 助手智能体(Assistant Agent) 根据对话历史生成回复,以模拟真实的 AI 助手。
  • • Reviewer Agent evaluates dialogue turns based on persona adherence, event execution quality, and logical flow.
  • • 评审智能体(Reviewer Agent) 根据人设遵循程度、事件执行质量和逻辑流畅性评估对话轮次。

F.2 Interaction Behavior Modeling

F.2 交互行为建模

Interruption Generation.

We implemented a two-stage mechanism to generate naturalistic interruptions. Random interruptions are scheduled between turns 2 and 4.

插话生成。

我们实现了两阶段机制来生成自然的插话。随机插话安排在第 2 轮到第 4 轮之间。

  1. Planning Phase: The User Agent analyzes the Assistant’s current response context to determine a valid Interruption Motivation (Correction, Deeper Inquiry, Topic Shift, Strong Emotional Reaction, or Impatience). It then inserts a placeholder tag at the precise logical point within the Assistant’s text.
  1. 规划阶段: 用户智能体分析助手当前回复的语境,确定一个合理的插话动机(Interruption Motivation:纠正、深入追问、话题转移、强烈情绪反应,或不耐烦)。然后在助手文本中逻辑上最合适的位置插入占位标签 。
  1. Execution Phase: Conditioned on the chosen motivation and the context prior to the interruption point, the User Agent generates the specific interruption utterance.
  1. 执行阶段: 以选定的动机以及插话点之前的语境为条件,用户智能体生成具体的插话语句。

Backchannel Injection.

Backchannels are injected probabilistically (p=0.5p=0.5p=0.5) during post-processing.

附和注入。\n\n附和在后处理阶段按概率注入(p=0.5p=0.5p=0.5)。

  • • User Backchannels: The User Agent reviews the Assistant’s response to insert feedback signals (uh-huh, gotcha) wrapped in tags.
  • • 用户附和: 用户智能体审看助手的回复,插入包在 标签里的反馈信号(uh-huh, gotcha)。
  • • AI Backchannels: Similarly, the system generates backchannels for the User’s speech to simulate active listening by the Assistant.
  • • AI 附和: 类似地,系统为用户的发言生成附和,以模拟助手的积极倾听。

F.3 Quality Control

F.3 质量控制

We employ a rigorous filtering process. A Reviewer Agent scores the final dialogue on a scale of 1-5 across three dimensions: Persona Consistency, Quality of Interruption Event, and Naturalness of Backchannels. Dialogues with low logical consistency or failed event executions are discarded.

我们采用严格的过滤流程。评审智能体按 1–5 分,从三个维度给最终对话打分:人设一致性、插话事件质量和附和自然度。逻辑一致性差或事件执行失败的对话会被丢弃。

F.4 Prompt Templates

F.4 提示模板

We provide the core system prompts used in our pipeline below.

我们在下面给出流水线中使用的核心系统提示。

Prompt 1: User Role-Play Instruction

Prompt 1:用户角色扮演指令

System Instruction: You are a person in a real-time voice conversation. In the conversation history, your lines are marked with "speaker": "You". You are talking to the person marked "speaker": "Other".

You are NOT an AI assistant. Your task is to speak naturally based on your persona. React authentically, don't try to be helpful.

Persona Details:

  • Persona: {persona}
  • Communication Style: {style}

Conversation Rules: Speak, don't write: Use filler words (e.g., "um", "uh", "like"), hesitations, and natural phrasing. Stay in character: Let your persona guide your responses.

Task: Now, it's your turn. Generate your next response as "You".

系统指令:你是实时语音对话中的一个人。在对话历史里,你的台词标为 "speaker": "You"。你正在与标为 "speaker": "Other" 的人交谈。

你不是 AI 助手。你的任务是根据自己的人设自然说话。要真实地作出反应,不要一心想着帮忙。

人设细节:

  • 人设:{persona}
  • 沟通风格:{style}

对话规则:开口说,而不是书面写:使用填充词(例如 "um"、"uh"、"like")、犹豫和自然的措辞。保持角色:让人设引导你的回答。

任务:现在轮到你了。以 "You" 的身份生成下一句回应。

Prompt 2: Interruption Planning (Motivation & Placement)

Prompt 2:插话规划(动机与位置)

Context: The assistant is currently saying: “{context}”

语境:助手当前正在说:“{context}”

Task: Plan and Place the Interruption. Your goal is to find the perfect moment to interrupt, driven by your persona.

任务:规划并安放插话。你的目标是在人设的驱动下,找到最合适的插话时机。

Action 1: Plan the Interruption’s Motivation. First, think about why your persona would interrupt here. Choose a motivation that fits your character:

行动 1:规划插话动机。先想清楚,你的人设为什么要在这里插话。选择一个符合角色的动机:

  • • Correction: The assistant’s response contains a point that may not be entirely accurate, and you want to clarify or refine it.
  • • 纠正(Correction):助手的回答里有一点可能并不完全准确,你想澄清或修正。
  • • Deeper Inquiry: You need to ask for clarification on a key point before they move on.
  • • 深入追问(Deeper Inquiry):你需要在对方说下去之前,先把一个关键点问清楚。
  • • Topic Shift: What they said reminds you of something else, and you want to change the subject.
  • • 话题转移(Topic Shift):对方说的话让你想到了别的事情,你想换个话题。
  • • Strong Emotional Reaction: You are surprised, excited, or disagree strongly and can’t hold it in.
  • • 强烈情绪反应(Strong Emotional Reaction):你感到惊讶、兴奋,或强烈不同意,忍不住要说出来。
  • • Impatience: You want to cut to the chase or stop a lengthy explanation.
  • • 不耐烦(Impatience):你想直奔主题,或打断一段冗长的解释。

Action 2: Place the Interruption Marker. Based on your chosen motivation, find the most natural point in the assistant’s speech to jump in. Insert ONLY the empty tag pair at that precise spot.

行动 2:安放插话标记。根据选定的动机,在助手的话语中找到最自然的切入点。只在那一处插入空标签对 。

Prompt 3: Interruption Utterance Generation

Prompt 3:插话语句生成

Context: You just decided to interrupt the assistant while they were saying: “{context}” Your motivation for interrupting is: {motivation}

语境:你刚刚决定在助手正说“{context}”时打断对方。你插话的动机是:{motivation}

Task: Deliver the Interruption. Now, say the words you would use to interrupt. Your utterance must sound spontaneous and directly reflect your motivation.

任务:给出这句插话。现在说出你用来打断的话。你的语句必须听起来是即兴的,并且直接体现你的动机。

[Detailed examples for motivations provided to the model: Correction, Deeper Inquiry, Topic Shift, Strong Emotional Reaction, Impatience]

[向模型提供的各动机的详细示例:纠正、深入追问、话题转移、强烈情绪反应、不耐烦]

Output: Generate ONLY the interrupting phrase itself.

输出:只生成插话短语本身。

Prompt 4: User Backchannel Generation

Prompt 4:用户附和生成

Task: As the assistant is speaking, you want to show you’re listening. The assistant’s last utterance was: “{context}”.

任务:助手正在说话时,你想表现出自己在听。助手上一句是:“{context}”。

  • • Action 1: Think of a short, spoken backchannel phrase (e.g., "uh-huh", "gotcha", "right", "mhm").
  • • 行动 1:想一句简短的口语附和(例如 "uh-huh"、"gotcha"、"right"、"mhm")。
  • • Action 2: Find the most natural point in the assistant’s speech to insert this backchannel, wrapped in tags.
  • • 行动 2:在助手的话语中找到最自然的插入点,把这句附和包在 标签里插入。

Prompt 5: Final Dialogue Quality Review

Prompt 5:最终对话质量评审

Role: You are a meticulous evaluator of simulated spoken dialogues.

角色:你是模拟口语对话的细致评估者。

Scoring Criteria:

评分标准:

  1. Persona Consistency & Depth (1-5): Does the user’s speech effectively embody the assigned persona ({persona}) and style ({style})?
  1. 人设一致性与深度(1–5):用户的发言是否有效体现了指定的人设({persona})和风格({style})?
  1. Quality of Interruption Event (1-5): Is the interruption timed perfectly and motivated by the persona? Does it feel natural or forced?
  1. 插话事件质量(1–5):插话时机是否恰到好处,并且由人设驱动?它听起来自然,还是显得生硬?
  1. Naturalness of Backchannels (1-5): Are backchannels subtle and placed to improve flow, or are they distracting/robotic?
  1. 附和自然度(1–5):附和是否含蓄、并且被放在能改善节奏的位置,还是显得干扰或机械?

Output: Provide scores and a detailed justification explaining your reasoning.

输出:给出分数,并提供详细理由,说明你的判断依据。