MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery
MauBERT: Universal Phonetic Inductive Biases for Few-Shot Acoustic Units Discovery
MauBERT：面向少样本声学单元发现的通用语音归纳偏置
Angelo Ortiz Tandazo ⋄† Manel Khentout ⋄ Email: angelo.ortiz.tandazo@ens.psl.eu Youssef Benchekroun Thomas Hueber †* † † thanks: * Equally contributed as senior authors. Affiliation: Meta AI Research, France Emmanuel Dupoux ⋄ ENS, PSL Research University, EHESS, CNRS, Paris, France † Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, Grenoble, France
Abstract
摘要
This paper introduces MauBERT, a multilingual extension of HuBERT that leverages articulatory features for robust cross-lingual phonetic representation learning. We continue HuBERT pre-training with supervision based on a phonetic-to-articulatory feature mapping in languages. Our models learn from multilingual data to predict articulatory features or phones, resulting in language-independent representations that capture multilingual phonetic properties. Through comprehensive ABX discriminability testing, we show MauBERT models produce more context-invariant representations than state-of-the-art multilingual self-supervised learning models. Additionally, the models effectively adapt to unseen languages and casual speech with minimal self-supervised fine-tuning ( hours of speech). This establishes an effective approach for instilling linguistic inductive biases in self-supervised speech models.
本文介绍 MauBERT，它是 HuBERT 的多语言扩展，借助发音特征（articulatory features）学习稳健的跨语言语音表征。我们在 55 种语言上，按音子到发音特征的映射继续对 HuBERT 做有监督预训练。模型从多语言数据中学习预测发音特征或音子，从而得到能够刻画多语言语音属性的、与具体语言无关的表征。通过系统的 ABX 可区分性测试，我们表明 MauBERT 模型比当前最强的多语言自监督学习模型产生更少受语境影响的表征。此外，只需少量自监督微调（10 小时语音），这些模型就能有效适应未见语言和自然口语。这说明，把语言学上的归纳偏置注入自监督语音模型，是一条可行的路径。
1 Introduction
1 引言
Is it possible to automatically discover the linguistic units of an unknown language from raw audio only? Doing so would be of great help to linguists or speech technologists working on low-resource or unwritten languages (chen2024evaluating; mohamed_22_review_ssl; zelasko22_asrinventories; chen_23_s2s_translation; zhang_21_uwspeech), or to cognitive modellers trying to understand how children learn their native language before learning to read and write (kuhl_1993_nlm; werker_07_phonetic). This question has been addressed using a variety of approaches under the Zero Resource Speech Challenge series (versteegh2015zero; dunbar17_zerospeech; dunbar_22_zrc_lessons), yielding impressive progress alongside unresolved questions.
能否只凭原始音频，就自动发现一种未知语言的语言学单元？这对从事低资源或无文字语言工作的语言学家和语音技术研究者会很有帮助（chen2024evaluating; mohamed_22_review_ssl; zelasko22_asrinventories; chen_23_s2s_translation; zhang_21_uwspeech），也对试图理解儿童在学会读写之前如何掌握母语的认知建模者有用（kuhl_1993_nlm; werker_07_phonetic）。Zero Resource Speech Challenge 系列从多种路径探讨了这个问题（versteegh2015zero; dunbar17_zerospeech; dunbar_22_zrc_lessons），既取得了显著进展，也留下了尚未解决的问题。
Much of this progress stems from advances in self-supervised learning (SSL) techniques (oord2018representation; baevski2020wav2vec; hsu21_hubert), which have produced speech representations that capture phonetic structure better than traditional features like MFCCs or mel filterbanks. This is evidenced by improved discriminability in the learnt representation spaces: two instances of the syllable ‘bit’ lie closer together than one instance of ‘bit’ and one instance of ‘bet’, even across different speakers (schatz16_phdthesis; schatz13_interspeech). Further evidence comes from the success of quantisation of these representations, yielding low-bitrate discrete codes suitable for training generative language models that produce novel utterances in the target language (lakhotia2021generative; borsos_23_audiolm; defossez_24_moshi; simon_25_calm).
这些进展很大程度上来自自监督学习（SSL）技术（oord2018representation; baevski2020wav2vec; hsu21_hubert）。它们得到的语音表征，比 MFCC 或梅尔滤波器组等传统特征更能抓住语音结构。一个直接证据是学到的表征空间里可区分性更好：同一个音节 “bit” 的两个实例，会比一个 “bit” 和一个 “bet” 更靠近，即使说话人不同也是如此（schatz16_phdthesis; schatz13_interspeech）。进一步的证据来自对这些表征做量化：得到的低码率离散编码，可以用来训练生成式语言模型，并在目标语言中产生新的语句（lakhotia2021generative; borsos_23_audiolm; defossez_24_moshi; simon_25_calm）。
Figure 1: a. Multilingual training. MauBERT-feat is trained to recognise ternary-valued articulatory features and phones using an encoder (HuBERT-base), downstream modules (weighted sum, up-projection, two-layer BLSTM, feature projection), and a phone model (two-layer perceptron); the feature states receive no gradients from the phone recognition loss due to the stop-gradient operator (sg). b. Self-supervised fine-tuning. Top: Offline clustering is applied to one of the layers of MauBERT, the teacher network, on an unseen language; bottom: the MauBERT encoder, the student network, is then trained to predict the corresponding clusters of masked input.
图 1：a. 多语言训练。 MauBERT-feat 用编码器（HuBERT-base）、下游模块（加权求和、升维、两层 BLSTM、特征投影）以及音子模型（两层感知机）来识别取三值的发音特征和音子；由于停止梯度算子（sg），特征状态不会接收来自音子识别损失的梯度。b. 自监督微调。 上： 在未见语言上，对教师网络 MauBERT 的某一层做离线聚类；下： 再训练学生网络，即 MauBERT 编码器，去预测被掩码输入所对应的簇。
However, current approaches face two limitations. First, units discovered through speech SSL do not correspond one-to-one with linguistic units like phones, syllables or words. After clustering, these units are typically shorter and more numerous than standard linguistic units: long vs. for phonemes, and vs. for phonemes (lavechin_25_simulating; schatz_21_pnas). Moreover, they lack full invariance to speaker identity (deseyssel22_probing; mohamed24_interspeech) and phonetic context (hallap23_any-ctx), suggesting they capture acoustic events rather than abstract linguistic units. As a result, they produce codes with higher bitrates than phonemic transcriptions: about versus (lakhotia2021generative; dunbar_22_zrc_lessons). Second, current SSL algorithms require massive amounts of clean speech: hsu21_hubert uses hours of clean English audio, zanonboito24_mhubert uses hours, and chen24_xeus uses hours. Such quantities are unavailable for low-resource languages, and notably, children acquire their language’s phonetics with far less than of much noisier input.
不过，现有方法仍有两点不足。第一，语音 SSL 发现的单元并不能与音子、音节或词等语言学单元一一对应。聚类之后，这些单元通常比标准语言学单元更短、数量也更多：长度约为 –，而音素约为 ；单元数 –，而音素约为 –（lavechin_25_simulating; schatz_21_pnas）。它们对说话人身份（deseyssel22_probing; mohamed24_interspeech）和语音语境（hallap23_any-ctx）也还没有完全不变，说明它们捕捉的更像是声学事件，而不是抽象的语言学单元。结果是，它们产生的编码码率高于音素转写：大约 –，而音素转写约为 –（lakhotia2021generative; dunbar_22_zrc_lessons）。第二，现有 SSL 算法需要大量干净语音：hsu21_hubert 用了 小时干净英语音频，zanonboito24_mhubert 用了 小时，chen24_xeus 用了 小时。低资源语言没有这样的规模；而且儿童掌握母语语音时，所用输入远少于 ，噪声也大得多。
One avenue for improving SSL models involves pre-training universal models (conneau2020unsupervised), with recent work expanding both language coverage and training data (babu22_xls-r; zanonboito24_mhubert; pratap24_mms; chen24_xeus). Inspired by the International Phonetic Alphabet (IPA), another research direction explores how phonetically-informed target signals influence learnt representations and their cross-lingual transferability (wang22m_interspeech; ma23b_interspeech; feng23_interspeech), suggesting that explicit phonological supervision enhances speech models’ cross-lingual capabilities.
改进 SSL 模型的一条路径，是预训练通用模型（conneau2020unsupervised）。近期工作同时扩大了语言覆盖和训练数据（babu22_xls-r; zanonboito24_mhubert; pratap24_mms; chen24_xeus）。受国际音标（IPA）启发，另一条路线则考察带语音信息的目标信号如何影响学到的表征及其跨语言可迁移性（wang22m_interspeech; ma23b_interspeech; feng23_interspeech），结果表明，显式的音系监督可以增强语音模型的跨语言能力。
In this paper, we explore the hypothesis that standard SSL algorithms lack strong inductive biases necessary for learning invariant speech representations from limited audio data in new languages. Following the universal pre-training and phonetically-informed research lines, we propose transforming a monolingual pre-trained SSL model (specifically, HuBERT-base trained on English) into a universal SSL model with strong inductive biases by fine-tuning it on universal IPA phonemes and features across diverse languages. By directly addressing the limitations of prior work, such as the need for large datasets and the absence of explicit phonological supervision, our method provides a practical and linguistically informed solution for both speech technology and language documentation in low-resource settings. We evaluate this model, coined MauBERT, on the ZRC challenge, which presents 5 languages with less than of training data (English, French, German, Mandarin, Wolof). To increase the evaluation’s diversity and validity, we extend the ZRC benchmark with typologically diverse languages (Swahili, Tamil, Thai, Turkish, Ukrainian). Evaluation employs the within- and across-speaker ABX metrics from ZRC, supplemented with metrics measuring invariance to contextual allophony (hallap23_any-ctx).
本文要检验的假设是：标准 SSL 算法缺少足够强的归纳偏置，因而难以仅凭新语言里有限的音频，就学会具有不变性的语音表征。沿着通用预训练和语音信息监督这两条线索，我们提出把一个单语预训练 SSL 模型（具体是在英语上训练的 HuBERT-base）转变成带有强归纳偏置的通用 SSL 模型，方法是在 种多样化语言上，用通用的 IPA 音素和特征对它做微调。先前工作往往需要大规模数据，又缺少显式音系监督；我们直接针对这两点，为低资源场景下的语音技术和语言记录提供一条既实用、又有语言学依据的路径。我们在 ZRC 挑战上评估这一名为 MauBERT 的模型：该挑战包含 5 种训练数据不足 的语言（英语、法语、德语、普通话、沃洛夫语）。为提高评测的多样性和有效性，我们又为 ZRC 基准补充了 种类型学上多样化的语言（斯瓦希里语、泰米尔语、泰语、土耳其语、乌克兰语）。评测采用 ZRC 的说话人内和跨说话人 ABX 指标，并补充衡量对语境音位变体不变性的指标（hallap23_any-ctx）。
Our main contributions are twofold: (i) We demonstrate that multilingual supervised fine-tuning of HuBERT for articulatory feature or phone prediction creates robust multilingual phonetic representations with strong zero-shot transfer capabilities. (ii) Our resulting models enable effective adaptation to unseen languages and casual speech with minimal self-supervised fine-tuning, achieving strong speaker and contextual invariance in new languages with only of unlabelled data. As a by-product, our method also yields candidate phoneme and feature sets for unseen languages, with potential applications for linguistic analyses of low-resource languages. The code with the data processing, methods and baselines can be found at https://github.com/bootphon/maubert.
主要贡献有两点：（i）我们表明，为预测发音特征或音子而对 HuBERT 做多语言有监督微调，能够得到稳健的多语言语音表征，并具有较强的零样本迁移能力。（ii）得到的模型只需少量自监督微调，就能有效适应未见语言和自然口语，仅用 无标注数据，便在新语言上对说话人和语境都表现出较强的不变性。作为副产品，该方法还能为未见语言给出候选音素和特征集合，有望用于低资源语言的语言学分析。数据处理、方法和基线的代码见 https://github.com/bootphon/maubert。
2 Related Work
2 相关工作
Multilingual Speech Representation Learning.
The field of multilingual speech processing has grown rapidly with large-scale semi- or self-supervised learning models that showed the potential for cross-lingual representation learning with little to no supervision (wang2021unispeech; conneau2020unsupervised). Recent studies have expanded language coverage (babu22_xls-r), diversified data sources (pratap24_mms), and improved efficiency (zanonboito24_mhubert) and robustness to noise (chen24_xeus). These multilingual SSL models build upon foundational work in self-supervised speech representation learning (baevski2020wav2vec; hsu21_hubert; chen_22_wavlm) and have been evaluated with multi-task frameworks like SUPERB (yang21c_superb; shi23g_ml_superb). Meanwhile, other studies have explored the impact of phonetically-informed targets on learnt representations and their cross-lingual transferability (wang22m_interspeech; ma23b_interspeech; feng23_interspeech). Inspired by the downstream framework of SUPERB, this work extends HuBERT-base for articulatory feature prediction.
多语言语音表征学习。
多语言语音处理发展很快，大规模半监督或自监督学习模型已经表明，即使只有很少或完全没有监督，也有可能学到跨语言表征（wang2021unispeech; conneau2020unsupervised）。近期研究扩大了语言覆盖（babu22_xls-r），拓宽了数据来源（pratap24_mms），并改进了效率（zanonboito24_mhubert）和对噪声的稳健性（chen24_xeus）。这些多语言 SSL 模型建立在自监督语音表征学习的基础工作之上（baevski2020wav2vec; hsu21_hubert; chen_22_wavlm），并用 SUPERB 这类多任务框架来评估（yang21c_superb; shi23g_ml_superb）。与此同时，另一些研究考察了带语音信息的目标对学到表征及其跨语言迁移的影响（wang22m_interspeech; ma23b_interspeech; feng23_interspeech）。受 SUPERB 下游框架启发，本工作扩展 HuBERT-base，用于发音特征预测。
Articulatory Features in Speech Processing.
Early work established the use of articulatory features (AFs) in speech processing (deng_91_hmm; elenius91_eurospeech; eide_93_lingfeats), demonstrated that supervised learning can be used to automatically extract phonological features from raw (and continuous) speech (papcun1992inferring; king00_phonfeats_rnn) and produced robust articulatory/phonological feature-based speech technologies (kirchhoff1999robust; livescu2007articulatory; frankel2007articulatory). The development of systematic feature inventories, particularly PanPhon (mortensen16_panphon), has provided practical computational tools for cross-linguistic analysis. This has enabled recent efforts to explore AFs in multilingual contexts, demonstrating their effectiveness for zero-shot multilingual speech synthesis (staib20_interspeech) or showing their utility for cross-lingual speech recognition in low-resource languages (feng23_interspeech).
语音处理中的发音特征。
早期工作确立了发音特征（AF）在语音处理中的用法（deng_91_hmm; elenius91_eurospeech; eide_93_lingfeats），表明可以用监督学习从原始（且连续的）语音中自动提取音系特征（papcun1992inferring; king00_phonfeats_rnn），并发展出稳健的、基于发音/音系特征的语音技术（kirchhoff1999robust; livescu2007articulatory; frankel2007articulatory）。系统化的特征清单，尤其是 PanPhon（mortensen16_panphon），为跨语言分析提供了可用的计算工具。这使近期工作能够在多语言场景中考察发音特征，例如证明它们对零样本多语言语音合成有效（staib20_interspeech），或显示它们有助于低资源语言的跨语言语音识别（feng23_interspeech）。
Evaluation of Speech Representations.
More specific subtasks have been developed as alternatives to downstream-based evaluation, offering clearer insights into unsupervised language learning. A prominent example is the ABX discriminability evaluation (schatz16_phdthesis), which assesses whether learnt representations can distinguish between different phonetic units in a way that reflects human perceptual boundaries. The Zero Resource Speech Challenge series (versteegh2015zero; dunbar17_zerospeech) has systematically applied ABX evaluation to assess unsupervised speech representations, establishing benchmarks for phonetic discrimination across diverse languages and speakers. While ABX testing shows sufficient correlation with downstream performance to serve as a model comparison proxy, traditional ABX evaluation has not assessed other types of invariance, like speaking rate or speech style variations (dunbar_22_zrc_lessons). A recent extension has begun addressing this limitation by measuring context invariance (hallap23_any-ctx). The present work builds on this extension and adds the comparison between read and casual speech.
语音表征的评测。
作为基于下游任务评测的替代，人们发展了更专门的子任务，以便更清楚地观察无监督的语言学习。一个突出的例子是 ABX 可区分性评测（schatz16_phdthesis）：它检验学到的表征能否区分不同的语音单元，并且这种区分是否与人类知觉边界相符。Zero Resource Speech Challenge 系列（versteegh2015zero; dunbar17_zerospeech）系统地用 ABX 评估无监督语音表征，为跨语言、跨说话人的语音判别建立了基准。ABX 测试与下游表现有足够相关性，因此可以作为比较模型的代理；但传统 ABX 评测并未衡量语速或语体变化等其他不变性（dunbar_22_zrc_lessons）。最近的一个扩展开始补上这一点，用于衡量语境不变性（hallap23_any-ctx）。本工作建立在这一扩展之上，并加入朗读语音与自然口语的对照。
3 MauBERT
In this section, we introduce our Multilingual articulatory hidden-unit BERT (MauBERT) models (Figure 1). We describe the base architecture for multilingual training (§3.1), and the self-supervised fine-tuning approach (§3.2).
本节介绍我们的 Multilingual articulatory hidden-unit BERT（MauBERT）模型（图 1）。我们先说明多语言训练的基础结构（§3.1），再说明自监督微调方法（§3.2）。
3.1 Multilingual Pre-Training
3.1 多语言预训练
MauBERT models are based on multilingual, continual learning of a pre-trained self-supervised speech model for articulatory feature (AF) or phone recognition (Figure 1a). We re-train HuBERT (hsu21_hubert) using the VoxCommunis Corpus (ahn22_voxcommunis), and the associated featural annotations extracted with PanPhon (mortensen16_panphon).
MauBERT 模型的基础，是对一个预训练自监督语音模型做多语言持续学习，用于发音特征（AF）或音子识别（图 1a）。我们用 VoxCommunis 语料库（ahn22_voxcommunis）以及由 PanPhon（mortensen16_panphon）提取的特征标注，重新训练 HuBERT（hsu21_hubert）。
We propose two versions of MauBERT: feat and phone. The former incorporates an AF bottleneck (Figure 1a), while the latter directly predicts phones without intermediate AFs[^1].
我们提出两个版本的 MauBERT：feat 和 phone。前者加入发音特征瓶颈（图 1a），后者则直接预测音子，不再经过中间的发音特征[^1]。
Encoder.
We use the pre-trained HuBERT-base model as our encoder. The convolutional feature extractor is kept frozen, but the Transformer encoder is trainable. We extract the feature extractor’s output after layer normalisation and dropout, as well as the outputs from each of the Transformer encoder layers. The input masking is disabled during continual pre-training following SUPERB’s downstream framework (yang21c_superb).
编码器。
我们用预训练的 HuBERT-base 模型作为编码器。卷积特征提取器保持冻结，Transformer 编码器则可训练。我们取出特征提取器在层归一化和 dropout 之后的输出，以及 个 Transformer 编码层各自的输出。持续预训练期间关闭输入掩码，这遵循 SUPERB 的下游框架（yang21c_superb）。
Downstream modules.
Preliminary experiments revealed that a simple linear layer on top of HuBERT was insufficient for the feature recognition task. Following the SUPERB framework for ASR (yang21c_superb), we instead leverage a weighted sum of HuBERT’s intermediate representations fed into a BLSTM. Ablation experiments confirmed the importance of this contextual architecture: replacing the BLSTM with non-contextual networks consistently degraded performance on both the multilingual recognition tasks and the phonetic probing, motivating our adoption of the SUPERB design.
下游模块。
预备实验表明，只在 HuBERT 上加一层简单的线性层，不足以完成特征识别任务。于是我们沿用 SUPERB 中用于自动语音识别的框架（yang21c_superb），对 HuBERT 的中间表征做加权求和，再送入 BLSTM。消融实验证实了这一带上下文结构的重要性：若把 BLSTM 换成无上下文网络，多语言识别和语音探测上的表现都会持续变差，因此我们采用了 SUPERB 的设计。
Concretely, we first compute a weighted sum of the intermediate representations from our encoder and up-project them to a -dimensional space. These representations are then processed through a bidirectional two-layer LSTM. Finally, we down-project the concatenated forward and backward output states into task-specific spaces: a -dimensional AF space for MauBERT-feat and a -dimensional phone space for MauBERT-phone.
具体来说，我们先对编码器的中间表征做加权求和，再升维到 维空间。这些表征随后经过一个双向两层 LSTM。最后，把拼接后的前向和后向输出状态投影到任务特定空间：MauBERT-feat 是 维发音特征空间，MauBERT-phone 是 维音子空间。
| Eval. lang. | MauBERT variant | Feat. acc. | Phone acc. | PER |
|---|---|---|---|---|
| Train | feat | |||
| phone | ||||
| Dev | feat | |||
| phone |
Table 1: Feature and phone evaluation of MauBERT on the held-out test set of the training languages and zero-shot performance on the development languages. All scores are in %.
| 评测语言 | MauBERT 变体 | 特征准确率 | 音子准确率 | PER |
|---|---|---|---|---|
| 训练 | feat | |||
| phone | ||||
| 开发 | feat | |||
| phone |
表 1：MauBERT 在 种训练语言留出测试集上的特征和音子评测，以及在 种开发语言上的零样本表现。所有分数均为 %。
Phone model.
Given the non-injective nature of the feature-to-phone mapping, for MauBERT-feat, we jointly learn a phone model consisting of a two-layer perceptron. Since we want the pre-training to be led by the feature recognition task only, a stop gradient operator prevents the feature hidden states from receiving any gradients from the phone recognition loss.
音子模型。
由于特征到音子的映射不是单射，对 MauBERT-feat，我们联合学习一个由两层感知机组成的音子模型。我们希望预训练只由特征识别任务主导，因此用停止梯度算子阻止特征隐状态接收来自音子识别损失的任何梯度。
3.2 Self-Supervised Fine-Tuning
3.2 自监督微调
We employ self-supervised fine-tuning to adapt MauBERT models to unseen languages with limited or no labelled data. This approach generates pseudo-labels through clustering of learnt representations and applies masked language modelling (Figure 1b), enabling MauBERT to adapt to the acoustic patterns of new languages.
我们用自监督微调，让 MauBERT 模型在标注有限或全无标注的未见语言上适应。该方法通过对学到的表征聚类来生成伪标签，再做掩码语言建模（图 1b），从而使 MauBERT 能够适应新语言的声学模式。
We use four methods to generate pseudo-labels: K-means, frequent features, frequent phones and all phones. As hsu21_hubert, we apply K-means clustering with to representations from encoder Transformer layers (HuBERT-base, MauBERT) or downstream module layers (MauBERT variants). For MauBERT-feat, we extract the top most frequent feature vectors (feat. freq.) from the articulatory feature space (top of Figure 1b). For both MauBERT variants, we extract the top most frequent phones (phone freq.) or all phones from pre-training data (all phones). See Appendix C for details.
我们用四种方法生成伪标签：K-means、高频特征、高频音子，以及全部音子。与 hsu21_hubert 一样，我们在编码器 Transformer 层（HuBERT-base、MauBERT）或下游模块层（MauBERT 变体）的表征上做 的 K-means 聚类。对 MauBERT-feat，我们从发音特征空间中取出频次最高的 个特征向量（feat. freq.，见图 1b 上部）。对两个 MauBERT 变体，我们取出频次最高的 个音子（phone freq.），或取出预训练数据中的全部音子（all phones）。细节见附录 C。
| Model | # Params | # Langs | # Hours | Seen | Seen |
|---|---|---|---|---|---|
| dev | test | ||||
| MMS | |||||
| XEUS | |||||
| mHuBERT- | |||||
| HuBERT-base | |||||
| MauBERT (ours) | [^2] |
Table 2: Comparison of speech models by number of parameters, number of languages, training data size, and development and test languages seen during training (continual learning for MauBERT).
| 模型 | 参数量 | 语言数 | 小时数 | 见过 | 见过 |
|---|---|---|---|---|---|
| 开发 | 测试 | ||||
| MMS | |||||
| XEUS | |||||
| mHuBERT- | |||||
| HuBERT-base | |||||
| MauBERT（本文） | – | [^2] |
表 2：各语音模型在参数量、语言数、训练数据规模，以及训练期间见过的开发和测试语言上的比较（MauBERT 为持续学习）。
4 MauBERT Multilingual Pre-Training
4 MauBERT 的多语言预训练
This section describes the multilingual training and evaluation of MauBERT variants for articulatory feature and phone recognition.
本节说明 MauBERT 各变体在发音特征和音子识别上的多语言训练与评测。
4.1 Data Processing
4.1 数据处理
We use the VoxCommunis Corpus, which provides phone-level annotations for a subset of Common Voice (ardila20_common_voice) obtained with Montreal Forced Aligner (mcauliffe17_mfa). Of the covered languages, are used for supervised articulatory feature prediction (totalling hours), serve as development languages, and are discarded as they were test languages. (Refer to §5.3 for the development and test languages and to Appendix A for more data processing details.)
我们使用 VoxCommunis 语料库。它为 Common Voice（ardila20_common_voice）的一个子集提供音子级标注，标注由 Montreal Forced Aligner（mcauliffe17_mfa）得到。在所覆盖的 种语言中， 种用于有监督的发音特征预测（合计 ）， 种作为开发语言， 种因属于测试语言而被去掉。（开发和测试语言见 §5.3，更多数据处理细节见附录 A。）
Using PanPhon’s feature table[^3], ternary-feature[^4] annotations are derived from the phone-level annotations. Annotated segments incompatible with PanPhon are manually fixed (e.g. [tS] [t͡S], [bH] [*̈b], [] [g]). Finally, we collapse the IPA table by keeping only distinct feature vectors (e.g. [æ̈], [eQ], [5Q], [∥‘æ] and [æ] are all represented by the same feature vector), which reduces the table size from to segment representatives. These representatives are then used for both phone recognition and feature recognition (underlying feature values).
利用 PanPhon 的特征表[^3]，我们从音子级标注导出三值特征[^4]标注。与 PanPhon 不兼容的标注片段会手工修正（例如 [tS] [t͡S]，[bH] [̈b]，[] [g]）。最后，我们压缩 IPA 表，只保留互不相同的特征向量*（例如 [æ̈]、[eQ]、[5Q]、[∥‘æ] 和 [æ] 都由同一特征向量表示），从而把表的规模从 降到 个音段代表。这些代表随后同时用于音子识别和特征识别（底层特征值）。
| Systems | Development languages | Test languages (ZRC) | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Layer | # units | triphone | phoneme ABX | triphone ABX | |||||||||||||||||||
| ABX | within ctx | any ctx | avg. | avg. | ||||||||||||||||||||
| WS | AS | WS | AS | WS | AS | WS | AS | WS | AS | WS | AS | |||||||||||||
| Zero-shot | ||||||||||||||||||||||||
| MFCC | - | |||||||||||||||||||||||
| MMS-B | ||||||||||||||||||||||||
| XEUS | ||||||||||||||||||||||||
| mHuBERT- | ||||||||||||||||||||||||
| HuBERT-base | ||||||||||||||||||||||||
| MauBERT | ||||||||||||||||||||||||
| feat | ||||||||||||||||||||||||
| phone | proj | |||||||||||||||||||||||
| supervised FT (10 h) | ||||||||||||||||||||||||
| HuBERT-base | ||||||||||||||||||||||||
| + PR | ws | |||||||||||||||||||||||
| + MPR | ||||||||||||||||||||||||
| MauBERT | ||||||||||||||||||||||||
| feat + MPR | ||||||||||||||||||||||||
| phone + MPR | ||||||||||||||||||||||||
| self-supervised FT (10 h) | ||||||||||||||||||||||||
| HuBERT-base | ||||||||||||||||||||||||
| + K-means (L) | ||||||||||||||||||||||||
| MauBERT-feat | ||||||||||||||||||||||||
| + K-means (L) | ||||||||||||||||||||||||
| + K-means (feat) | ||||||||||||||||||||||||
| + feat. freq. | ||||||||||||||||||||||||
| + phone freq. | ||||||||||||||||||||||||
| MauBERT-phone | ||||||||||||||||||||||||
| + K-means (proj) | ||||||||||||||||||||||||
| + K-means (phone) | ||||||||||||||||||||||||
| + phone freq. | ||||||||||||||||||||||||
| + all phones |
Table 3: Acoustic discriminability scores (lower is better) over development languages (sw, ta, th, tr, uk) and, as test languages, the languages from the Zero Resource Speech Challenge (en, fr, zh, de, wo). The best layer for each model is selected based on the average ABX score on the development languages. The best scores are in bold and the second best are underlined.
| 系统 | 开发语言 | 测试语言 (ZRC) | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 模型 | 层 | 单元数 | triphone | 音素 ABX | 三音子 ABX | |||||||||||||||||||
| ABX | 语境内 | 任意语境 | avg. | avg. | ||||||||||||||||||||
| 说话人内 | 跨说话人 | 说话人内 | 跨说话人 | 说话人内 | 跨说话人 | 说话人内 | 跨说话人 | 说话人内 | 跨说话人 | 说话人内 | 跨说话人 | |||||||||||||
| 零样本 | ||||||||||||||||||||||||
| MFCC | - | |||||||||||||||||||||||
| MMS-B | ||||||||||||||||||||||||
| XEUS | ||||||||||||||||||||||||
| mHuBERT- | ||||||||||||||||||||||||
| HuBERT-base | ||||||||||||||||||||||||
| MauBERT | ||||||||||||||||||||||||
| feat | ||||||||||||||||||||||||
| phone | proj | |||||||||||||||||||||||
| 有监督微调（10 小时） | ||||||||||||||||||||||||
| HuBERT-base | ||||||||||||||||||||||||
| + PR | ws | |||||||||||||||||||||||
| + MPR | ||||||||||||||||||||||||
| MauBERT | ||||||||||||||||||||||||
| feat + MPR | ||||||||||||||||||||||||
| phone + MPR | ||||||||||||||||||||||||
| 自监督微调（10 小时） | ||||||||||||||||||||||||
| HuBERT-base | ||||||||||||||||||||||||
| + K-means (L) | ||||||||||||||||||||||||
| MauBERT-feat | ||||||||||||||||||||||||
| + K-means (L) | ||||||||||||||||||||||||
| + K-means (feat) | ||||||||||||||||||||||||
| + feat. freq. | ||||||||||||||||||||||||
| + phone freq. | ||||||||||||||||||||||||
| MauBERT-phone | ||||||||||||||||||||||||
| + K-means (proj) | ||||||||||||||||||||||||
| + K-means (phone) | ||||||||||||||||||||||||
| + phone freq. | ||||||||||||||||||||||||
| + all phones |
表 3：在 种开发语言（sw、ta、th、tr、uk）以及作为测试语言的 Zero Resource Speech Challenge 中 种语言（en、fr、zh、de、wo）上的声学可区分性分数（越低越好）。各模型的最佳层按开发语言上的平均 ABX 分数选定。最佳分数用粗体，次优分数加下划线。
4.2 Training Details
4.2 训练细节
We train MauBERT variants for feature or phone recognition across the languages drawn from the VoxCommunis Corpus. Due to PanPhon’s ternary feature representation, we exclude MauBERT-feat predictions that correspond to zero-valued target features. Furthermore, to handle multiphthongs (hypernym of diphthong), we use a uniform heuristic so that the duration of the resulting monophthongs is roughly the same.
我们在取自 VoxCommunis 语料库的 种语言上，训练 MauBERT 变体做特征或音子识别。由于 PanPhon 采用三值特征表示，MauBERT-feat 中与取值为零的目标特征相对应的预测会被排除。此外，为了处理复合元音（multiphthong，是双元音的上位词），我们采用一条统一的启发式规则，使得到的单元音时长大致相同。
The feat and phone variants are trained to minimise binary and multiclass cross-entropy losses, respectively, at the frame level with the Adam optimiser (kingma_15_adam). We use one V GPU for steps with a tri-stage learning rate schedule ( for warmup and for decay) that peaks at . Following conneau2020unsupervised, we employ a language up-sampling strategy to balance the amount of data between low-resource and high-resource languages. (See Appendix A for more details.)
feat 和 phone 两个变体分别在帧级最小化二元和多类交叉熵损失，优化器为 Adam（kingma_15_adam）。我们使用一块 V GPU，训练 步，学习率采用三阶段日程（ 步预热， 步衰减），峰值为 。沿用 conneau2020unsupervised，我们采用语言上采样策略，以平衡低资源和高资源语言的数据量。（更多细节见附录 A。）
4.3 Evaluation and Results
4.3 评测与结果
We evaluate our MauBERT models using three speech recognition metrics: frame-wise feature accuracy, frame-wise phone accuracy and phone error rate (PER) without deduplication heuristics. For feature accuracy, we compute scores over non-zero features only, excluding zero-valued target features as in training. Since MauBERT-phone lacks an explicit feature space, we extract feature vectors from predicted phones using PanPhon’s feature table.
我们用三项语音识别指标评估 MauBERT 模型：帧级特征准确率、帧级音子准确率，以及不做去重启发式的音子错误率（PER）。特征准确率只在非零特征上计算，与训练时一样排除取值为零的目标特征。由于 MauBERT-phone 没有显式的特征空间，我们用 PanPhon 的特征表，从预测出的音子中提取特征向量。
Table 1 shows results for both MauBERT variants on held-out test sets from the training languages and the development languages. Both variants exhibit superior performance on training languages, particularly for phone-level metrics. When transitioning from training to development languages, phone accuracy drops by about pp and PER increases by approximately pp, while feature accuracy shows more resilience with only pp degradation. The feat variant consistently outperforms the phone variant in articulatory feature prediction across language sets and features (see Figure 2). However, this advantage does not translate to improved phone recognition performance. The phone variant leads by pp in phone accuracy and, since frame-level errors accumulate over the sequence, the gap widens to about pp in PER. Note that both gaps are stable across language sets.
表 1 给出两个 MauBERT 变体在 种训练语言留出测试集和 种开发语言上的结果。两个变体在训练语言上表现都更好，尤其是音子级指标。从训练语言转到开发语言时，音子准确率大约下降 个百分点，PER 大约上升 个百分点，而特征准确率更稳健，只下降约 – 个百分点。feat 变体在各语言集合和各项特征上的发音特征预测都持续优于 phone 变体（见图 2）。不过，这一优势并没有转化为更好的音子识别。phone 变体在音子准确率上领先 个百分点；由于帧级错误会在序列上累积，PER 上的差距扩大到大约 个百分点。需要说明的是，这两项差距在各语言集合上都保持稳定。
Figure 2: Classification errors across articulatory features () for both MauBERT variants on the development languages. All values are in %.
图 2：两个 MauBERT 变体在 种开发语言上、各发音特征的分类错误率（）。所有数值均为 %。
5 Few-shot Language Adaptation
5 少样本语言适应
In this section, we assess the linguistic relevance of MauBERT’s learnt representations by evaluating their phonetic invariance across languages and speaking styles, in a zero-shot or few-shot setting.
本节评估 MauBERT 所学表征的语言学相关性：在零样本或少样本设定下，考察它们跨语言、跨语体的语音不变性。
5.1 Language Adaptation Setting
5.1 语言适应设定
Modes.
We compare how SSL models encode speech in a new language in three modes: zero-shot, supervised fine-tuning and self-supervised fine-tuning. All the baselines and our two MauBERT models are evaluated in zero-shot mode, while only the monolingual baseline and our two models are evaluated in the fine-tuning modes (on the training split) for fairness[^5]. In the supervised mode, the models are trained to predict the ground-truth phones of masked inputs (MPR) or without masking at all (PR). In the self-supervised mode, a clustering step first produces discrete pseudo-labels, which are later used as targets for masked prediction.
模式。
我们比较 SSL 模型在三种模式下如何编码新语言的语音：零样本、有监督微调和自监督微调。所有基线与两个 MauBERT 模型都在零样本模式下评测；出于公平，只有单语基线和我们的两个模型进入微调模式（在 训练划分上）[^5]。有监督模式下，模型被训练去预测被掩码输入的真实音子（MPR），或完全不加掩码（PR）。自监督模式下，先通过聚类得到离散伪标签，再作为掩码预测的目标。
Baselines.
We compare MauBERT against several baselines, including traditional acoustic features (MFCCs), the monolingual HuBERT-base backbone, and three self-supervised models trained on massively multilingual data: MMS-B (pratap24_mms), mHuBERT- (zanonboito24_mhubert), and XEUS (chen24_xeus). Table 2 shows a brief comparison of the training data between the baselines and our models.
基线。
我们将 MauBERT 与若干基线比较，包括传统声学特征（MFCC）、单语的 HuBERT-base 骨干，以及三个在大规模多语言数据上训练的自监督模型：MMS-B（pratap24_mms）、mHuBERT-（zanonboito24_mhubert）和 XEUS（chen24_xeus）。表 2 简要比较了各基线与我们模型的训练数据。
Figure 3: Reduction of the triphone ABX error rates across the development languages and test languages between the base HuBERT model, and MauBERT, tested in zero-shot and after masked fine-tuning () with or without labels on a new language. The two speaker conditions are averaged, and the subset is chosen for the test languages. ∗Mandarin and Wolof only have of training data, resp.
图 3：在 种开发语言和 种测试语言上，基础 HuBERT 模型与 MauBERT 之间三音子 ABX 错误率的下降；MauBERT 分别在零样本下，以及在新语言上用 做有标注或无标注的掩码微调后测试。两种说话人条件取平均，测试语言选用 子集。∗普通话和沃洛夫语的训练数据分别只有 –。
Implementation.
For the supervised fine-tuning, we train the HuBERT modules[^6] for steps on one V GPU with a tri-stage learning rate schedule ( for warmup and for decay). We use the Adam optimiser with a peak learning rate at . For the self-supervised fine-tuning, we train these modules for steps on one H GPU. We use the Adam optimiser with a linear decay schedule ( for warmup, then linear decay back to zero) that peaks at .
实现。
有监督微调时，我们在一块 V GPU 上训练 HuBERT 模块[^6] 共 步，学习率采用三阶段日程（ 步预热， 步衰减）。优化器为 Adam，峰值学习率为 。自监督微调时，这些模块在一块 H GPU 上训练 步。优化器仍为 Adam，学习率采用线性衰减日程（ 步数用于预热，随后线性衰减至零），峰值为 。
5.2 Metric
5.2 指标
We employ the ABX discriminability test to measure phonetic invariance (schatz16_phdthesis). It evaluates speech representations by comparing distances between three triphones: A, X (same linguistic unit as A), and B (different unit). The test is considered successful when the distance between A and X is smaller than that between A and B. The test comprises two variants: a triphone-based version that examines complete triphone representations, and a phoneme-based version that focuses exclusively on central phone representations.
我们用 ABX 可区分性测试来衡量语音不变性（schatz16_phdthesis）。它通过比较三个三音子之间的距离来评估语音表征：A、X（与 A 属于同一语言学单元）和 B（不同单元）。当 A 与 X 的距离小于 A 与 B 的距离时，测试视为成功。该测试有两个变体：基于三音子的版本考察完整的三音子表征，基于音素的版本则只关注中心音子的表征。
The speaker condition varies between two scenarios: within-speaker (all triphones share the speaker) and across-speaker (only A and B share the speaker). In addition, contextual conditions across all three items (A, B, and X) can be manipulated: within-context (where all items share identical surrounding phonetic context) versus any-context (where surrounding contexts may differ).
说话人条件分两种：说话人内（所有三音子来自同一说话人）和跨说话人（只有 A 和 B 来自同一说话人）。此外，还可以操纵三个项目（A、B 和 X）的语境条件：语境内（所有项目共享相同的前后语音语境）与 任意语境（前后语境可以不同）。
We compute all the ABX scores with the CPU backend of fastabx (poli25_fastabx).
所有 ABX 分数均用 fastabx 的 CPU 后端计算（poli25_fastabx）。
5.3 Language Data
5.3 语言数据
Following the Zero Resource Speech Challenge dunbar17_zerospeech, we curate ABX-ready datasets for five development languages from the VoxCommunis Corpus: Swahili, Tamil, Thai, Turkish and Ukrainian. The ABX datasets consist of three splits for each language: a training set, a validation set and a test set. We select the best parameters, hyperparameters and layers of the various models according to their impact on the average ABX score (triphone-based ABX, within-context phoneme ABX and any-context phoneme ABX) on the ABX test sets.
沿用 Zero Resource Speech Challenge （dunbar17_zerospeech），我们从 VoxCommunis 语料库为五种开发语言整理可用于 ABX 的数据集：斯瓦希里语、泰米尔语、泰语、土耳其语和乌克兰语。每种语言的 ABX 数据包含三个划分： 训练集、验证集和测试集。我们根据各模型在 ABX 测试集上的平均 ABX 分数（基于三音子的 ABX、语境内音素 ABX 以及任意语境音素 ABX）来选择最佳参数、超参数和层。
We use both the development and surprise languages from the aforementioned Zero Resource Speech Challenge , namely English, French, Mandarin, German and Wolof, as test languages (hereafter referred to as ZRC). The amount of speech in the original training set ranges from for Mandarin to for English. We thus extract training and validation splits of up to . In line with the desiderata of the challenge, we keep the original test subsets of differing length (, and ) to evaluate the effect of context length (triphone-based ABX only). We evaluate only the best configuration for each model on these languages.
我们把上述 Zero Resource Speech Challenge 的开发语言和惊喜语言，即英语、法语、普通话、德语和沃洛夫语，用作测试语言（下文称作 ZRC）。原始训练集中的语音量从普通话的 到英语的 不等。因此我们提取至多 的训练和验证划分。按照该挑战的要求，我们保留长度不同的原始测试子集（、 和 ），以评估语境长度的影响（仅用于基于三音子的 ABX）。在这些语言上，我们只评估各模型的最佳配置。
Additionally, we curate an ABX dataset of casual speech in English and French, sourcing high-quality recorded conversations of native speakers. The dataset possesses the same three-split structure as the development languages.
此外，我们还整理了英语和法语自然口语的 ABX 数据集，来源是母语者高质量录制的对话。该数据集与开发语言一样，采用三分结构。
| Systems | Read | Casual | |||
|---|---|---|---|---|---|
| WS | AS | WS | AS | ||
| MMS | |||||
| XEUS | |||||
| mHuBERT-147 | |||||
| HuBERT-base | |||||
| ours | |||||
| MauBERT-feat | |||||
| + K-means (L) | |||||
| MauBERT-phone | |||||
| + phone freq. |
Table 4: Triphone-based ABX error rates across registers (read vs. spontaneous) for English and French in zero-shot mode. Our two MauBERT variants are also tested after self-supervised fine-tuning on .
| 系统 | 朗读 | 口语 | |||
|---|---|---|---|---|---|
| 说话人内 | 跨说话人 | 说话人内 | 跨说话人 | ||
| MMS | |||||
| XEUS | |||||
| mHuBERT-147 | |||||
| HuBERT-base | |||||
| 本文 | |||||
| MauBERT-feat | |||||
| + K-means (L) | |||||
| MauBERT-phone | |||||
| + phone freq. |
表 4：英语和法语在零样本模式下、跨语体（朗读对即时口语）的三音子 ABX 错误率。我们的两个 MauBERT 变体也在 自监督微调之后做了测试。
5.4 Few-shot Language Adaptation Results
5.4 少样本语言适应结果
Our experimental results demonstrate the competitive performance of our MauBERT models across multiple evaluation scenarios.
实验结果表明，我们的 MauBERT 模型在多种评测情景下都具有竞争力。
Multilingual Training Benefits.
The top of Table 3 illustrates the phonetic invariance performance in zero-shot mode achieved by MauBERT models through multilingual pre-training. Our models attain particularly strong results in the any-context phoneme-based ABX tasks, and the MauBERT-phone model delivers the best overall zero-shot performance ( against for XEUS). Further, Figure 3 confirms the performance improvements of the multilingual pre-training as well as the proposed self-supervised fine-tuning: for the HuBERT-base baseline vs. for MauBERT-phone in zero-shot and after phone freq. MPR in triphone ABX with similar audio lengths.
多语言训练的收益。
表 3 上半部分展示了 MauBERT 模型通过多语言预训练在零样本模式下获得的语音不变性。我们的模型在任意语境、基于音素的 ABX 任务上尤其强，MauBERT-phone 给出了最佳的总体零样本结果（，XEUS 为 ）。图 3 也证实了多语言预训练以及所提出的自监督微调带来的提升：在音频长度相近的三音子 ABX 上，HuBERT-base 基线为 ，MauBERT-phone 零样本为 ，经过 phone freq. MPR 后为 。
Cross-linguistic Performance Patterns.
Table 3 also shows that development languages present greater challenges than test languages when evaluated under comparable conditions (triphone ABX with similar audio lengths) across models and modes. This pattern indicates varying degrees of phonetic complexity across language families and suggests that our model selection strategy (detailed in §5.3) based on development languages’ performance provides a robust foundation for cross-lingual generalisation. Figure 3 reinforces this observation, with development languages (shown with dotted lines) generally exhibiting higher error rates and more variable performance across the training progression compared to test languages (solid lines), suggesting the former present more challenging phonetic discrimination tasks and may represent more diverse or complex phonological systems.
跨语言表现模式。
表 3 还显示，在可比条件（音频长度相近的三音子 ABX）下，无论模型还是模式，开发语言都比测试语言更难。这一模式表明不同语系的语音复杂度不同，也说明我们按开发语言表现来选择模型的策略（详见 §5.3）为跨语言泛化提供了稳健基础。图 3 进一步支持这一观察：开发语言（虚线）的错误率通常更高，并且在训练进程中波动更大，测试语言（实线）则更平稳，说明前者的语音判别任务更难，其音系系统也可能更多样或更复杂。
Supervised Fine-tuning Efficacy.
Supervised fine-tuning yields substantial improvements in ABX error rates. Particularly striking is the effectiveness of predicting the ground-truth phones of masked inputs (MPR), which reduces ABX error rates compared to standard phone prediction (PR), especially for triphone-based ABX. The MauBERT-phone + MPR configuration achieves the best supervised performance ( on development languages, on test languages), representing a significant relative improvement over the zero-shot baseline. Figure 3 illustrates this systematic improvement pattern across all languages, with supervised fine-tuning showing the most notable gains ( average ABX score). Remarkably, fine-tuning effectiveness appears largely independent of training data quantity: low-resource language Wolof achieves comparable error rates to high-resource languages, indicating robust few-shot adaptation capabilities.
有监督微调的效果。
有监督微调使 ABX 错误率大幅下降。尤其突出的是，预测被掩码输入的真实音子（MPR）比标准音子预测（PR）更能降低 ABX 错误率，在基于三音子的 ABX 上尤为明显。MauBERT-phone + MPR 配置取得了最佳有监督结果（开发语言 ，测试语言 ），相对零样本基线有约 的相对提升。图 3 展示了这一在所有语言上系统改善的模式，有监督微调带来的增益最为显著（平均 ABX 分数 ）。值得注意的是，微调效果似乎在很大程度上不取决于训练数据量：低资源语言沃洛夫语达到了与高资源语言相当的错误率，说明少样本适应能力是稳健的。
Self-supervised Fine-tuning Analysis.
While self-supervised fine-tuning approaches show consistent improvements over zero-shot performance, a performance gap remains compared to the fully-supervised standard. Among the clustering strategies, our phone frequency-based approach demonstrates some gains over standard K-means clustering, particularly excelling in phoneme-level discrimination tasks and longer temporal contexts ( and triphone ABX). MauBERT-phone with phone frequency clustering achieves the best self-supervised performance ( average ABX score), highlighting the value of linguistically-informed clustering strategies.
自监督微调分析。
自监督微调相对零样本表现有持续提升，但与完全有监督的标准相比仍有差距。在各聚类策略中，我们基于音子频率的方法相对标准 K-means 聚类有一定收益，尤其是在音素级判别任务和更长的时间语境（ 和 三音子 ABX）上。MauBERT-phone 配上按音子频率聚类，取得了最佳自监督结果（平均 ABX 分数 ），说明有语言学依据的聚类策略是有价值的。
Speech Register Adaptation Results.
Table 4 reveals nuanced domain-specific patterns across read versus casual speech. In zero-shot mode, our models perform slightly better than multilingual baselines on read speech (MauBERT-phone: vs. XEUS: ) but show reversed performance on casual speech ( vs. for XEUS), reflecting the inherent difficulty of spontaneous speech processing with its increased phonetic variability and reduced articulatory precision. However, self-supervised fine-tuning not only amplifies our advantage on read speech () but also recovers competitive performance on casual speech (), demonstrating the robustness of our adaptation approach across speech domains.
语体适应结果。
表 4 揭示了朗读与自然口语之间细致的领域差异。零样本时，我们的模型在朗读上略好于多语言基线（MauBERT-phone：，XEUS：），在自然口语上则相反（，XEUS 为 ），这反映出即时口语本身更难处理：语音变异更大，发音也更不精确。不过，自监督微调不仅扩大了我们在朗读上的优势（），也在自然口语上回到了有竞争力的水平（），说明这一适应方法在不同语音领域上都比较稳健。
5.5 Phonetic Inventory Discovery Results
5.5 音位清单发现结果
Two of the MauBERT SSL methods consist in assigning a feature or a phoneme set to a new language as a target for SSL fine-tuning. These methods amount to discovering the phonetic inventories of previously unseen languages[^7]. Following zelasko22_asrinventories, we leverage the frequency distribution of (discrete) articulatory feature vectors produced by MauBERT-feat, where high-frequency combinations likely correspond to actual phones in the language inventory[^8].
MauBERT 的两种 SSL 方法，是把一个特征集合或音素集合指派给新语言，作为 SSL 微调的目标。这些方法相当于在发现先前未见语言的语音清单[^7]。沿用 zelasko22_asrinventories，我们利用 MauBERT-feat 产生的（离散）发音特征向量的频率分布：高频组合很可能对应该语言清单中的实际音子[^8]。
Table 7 reveals a clear trade-off between precision and recall across different threshold strategies. The top- approach achieves consistently high recall (at least for four out of five languages), successfully capturing most phonemes in the target inventories. However, this comes at the cost of precision (), indicating substantial inclusion of spurious feature vectors. Conversely, the optimised frequency threshold approach significantly improves precision () while maintaining reasonable recall (), suggesting more accurate phonetic identification with fewer false positives.
表 7 清楚显示了不同阈值策略在精确率与召回率之间的权衡。取频次最高的 个向量时，召回率一贯较高（五种语言中有四种至少为 ），能够覆盖目标清单中的大部分音素。但这以精确率为代价（–），说明混入了不少虚假特征向量。相反，优化后的频率阈值显著提高了精确率（–），同时保持了合理的召回率（–），意味着语音识别更准确，假阳性也更少。
The superior F performance of optimised thresholds over fixed thresholds underscores the importance of adaptive, data-driven approaches to inventory discovery. (See Table 8 for some inventory examples with F-optimal thresholds.)
优化阈值在 F 上优于固定阈值，说明自适应、由数据驱动的清单发现方法很重要。（采用 F 最优阈值的若干清单示例见表 8。）
6 Discussion
6 讨论
Broader Impact.
Our demonstration that effective phonetic models can be developed for low-resource languages with minimal training data (as evidenced by Wolof performance with less than of data) is an encouraging signal towards more linguistic inclusion in computational models. In addition, our frequency-based methodology offers particular value for endangered language documentation, where traditional phonological analysis may be impractical, providing linguists with not only a multilingual articulatory feature recogniser but also an automated tool for initial phonetic hypothesis generation that can guide subsequent detailed analysis. However, the superior performance of high-resource languages like English also highlights the importance of linguistic diversity in training data, since the imbalance thereof could persist through evaluation.
更广泛的影响。
我们表明，只需极少的训练数据，就能为低资源语言建立起有效的语音模型（沃洛夫语在不足 数据上的表现即是例证），这对提高计算模型的语言包容性是一个积极信号。此外，基于频率的方法对濒危语言记录尤其有价值：传统音系分析在这些场景里往往难以实施。语言学家由此不仅得到一个多语言发音特征识别器，还能用自动化工具生成初步的语音假设，以指导后续的细致分析。不过，英语等高资源语言表现更好，也说明训练数据中的语言多样性很重要，因为这种不平衡可能一直延续到评测。
Future Work.
Several promising research directions emerge from our findings. The counter-intuitive relationship between training data quantity and fine-tuning effectiveness suggests that investigation into optimal data selection strategies could yield significant improvements, potentially focusing on phonetically diverse rather than simply large datasets. Given that speaker and content information are encoded at different layers of HuBERT’s encoder, selectively targeting a subset of hidden layers during multilingual pre-training is another promising avenue worth exploring. The domain adaptation capabilities demonstrated in our casual speech experiments indicate potential for developing more robust models through multi-domain training paradigms. Furthermore, extending the self-supervised fine-tuning beyond the encoder to encompass the entire MauBERT architecture could address current limitations by enabling end-to-end adaptation of both the pre-trained representations and the downstream articulatory feature prediction modules, potentially leading to improved performance on target languages and domains, and better phonetic inventory discovery.
未来工作。
我们的发现引出几条有希望的研究方向。训练数据量与微调效果之间这种有违直觉的关系表明，研究更优的数据选择策略可能会带来明显改进，也许应关注语音上更多样的数据，而不是单纯追求更大规模。由于说话人和内容信息编码在 HuBERT 编码器的不同层，在多语言预训练中有选择地针对一部分隐层，也是值得探索的方向。自然口语实验所展示的领域适应能力，表明通过多领域训练范式，有可能发展出更稳健的模型。进一步地，若把自监督微调从编码器扩展到整个 MauBERT 结构，就有可能同时适应预训练表征和下游的发音特征预测模块，从而缓解当前局限，在目标语言和领域上取得更好表现，并改进音位清单发现。
7 Conclusion
7 结论
This work presents MauBERT, a multilingual extension of HuBERT that demonstrates competitive phonetic discrimination capabilities across diverse languages while revealing important insights about cross-lingual representation learning. Our results establish that multilingual supervised pre-training creates robust phonetic foundations that enable effective few-shot adaptation to new languages ( hours of speech) with or without supervision. The demonstrated effectiveness on both read and spontaneous speech, coupled with strong performance on low-resource languages, positions this work as a significant step towards more inclusive multilingual speech technologies.
本文提出 MauBERT，它是 HuBERT 的多语言扩展，在多样语言上展现出有竞争力的语音判别能力，也为跨语言表征学习提供了重要认识。我们的结果表明，多语言有监督预训练能够打下稳健的语音基础，从而在有监督或无监督的情况下，都能有效地以少样本适应新语言（ 小时语音）。该方法在朗读和即时口语上都有效，并在低资源语言上表现较强，是迈向更具包容性的多语言语音技术的重要一步。
Limitations
局限
The evaluation is constrained to the ABX discrimination task, which, while established as a standard phonetic benchmark, may not fully capture the nuanced linguistic representations as required for other linguistic levels (e.g. syntax and semantics). The performance gap between self-supervised and supervised fine-tuning methods suggests that clustering- or frequency-based approaches, despite their linguistic motivation, remain suboptimal compared to gold-standard supervision.
评测限于 ABX 判别任务。虽然它已被视为标准的语音基准，但未必能充分刻画其他语言层面（例如句法和语义）所需的细致语言表征。自监督微调与有监督微调之间的表现差距说明，尽管聚类或基于频率的方法有语言学动机，它们仍不如金标准监督。
Acknowledgments
致谢
We thank the reviewers for their valuable feedback, which helped improve the clarity and quality of the paper. This work was granted access to the HPC resources of IDRIS under the allocation 2024-AD011014739R1 made by GENCI, and was supported in part by Agence Nationale de Recherche (ANR-17-EURE-0017 FrontCog, ANR-10-IDEX-0001-02 PSL and ANR-23-IACL-0006 France 2030). ED in his EHESS role and MK were funded by an ERC grant (InfantSimulator). Views and opinions expressed are those of the authors only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
感谢审稿人提供的宝贵意见，帮助改进了本文的清晰度和质量。本研究获准使用 IDRIS 的高性能计算资源，分配号为 GENCI 提供的 2024-AD011014739R1；并部分得到法国国家科研署资助（ANR-17-EURE-0017 FrontCog、ANR-10-IDEX-0001-02 PSL 以及 ANR-23-IACL-0006 France 2030）。Dupoux 以其 EHESS 身份与 Khentout 的工作由一项 ERC 资助（InfantSimulator）支持。文中观点仅代表作者本人，不一定反映欧盟或欧洲研究理事会的立场。欧盟及资助机构对此不承担责任。
References
参考文献
Ahn and Chodroff (2022) E. Ahn and E. Chodroff VoxCommunis: a corpus for cross-linguistic phonetic analysis . In Proceedings of the Thirteenth Language Resources and Evaluation Conference , N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, and S. Piperidis (Eds.) , Marseille, France , pp. 5286–5294 . External Links: Link Cited by: Appendix A , §2 , §3.1 .
Ardila et al. (2020) R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber Common Voice: a massively-multilingual speech corpus . In Proceedings of the Twelfth Language Resources and Evaluation Conference , N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis (Eds.) , Marseille, France , pp. 4218–4222 ( eng ). External Links: Link , ISBN 979-10-95546-34-4 Cited by: Appendix A , §4.1 .
Babu et al. (2022) A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino, A. Baevski, A. Conneau, and M. Auli XLS-R: self-supervised cross-lingual speech representation learning at scale . In Interspeech 2022 , pp. 2278–2282 . External Links: Document , ISSN 2958-1796 Cited by: §1 , §2 .
Baevski et al. (2020) A. Baevski, Y. Zhou, A. Mohamed, and M. Auli Wav2vec 2.0: a framework for self-supervised learning of speech representations . In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.) , Vol. 33 , pp. 12449–12460 . External Links: Link Cited by: §1 , §2 .
Borsos et al. (2023) Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour AudioLM: a language modeling approach to audio generation . IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 ( ), pp. 2523–2533 . External Links: Document Cited by: §1 .
Chen et al. (2024a) C. Chen, W. Chen, R. J. Zevallos, and J. E. Ortega Evaluating self-supervised speech representations for indigenous American languages . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , N. Calzolari, M. Kan, V. Hoste, A. Lenci, S. Sakti, and N. Xue (Eds.) , pp. 6444–6450 . External Links: Link Cited by: §1 .
Chen et al. (2023) P. Chen, K. Tran, Y. Yang, J. Du, J. Kao, Y. Chung, P. Tomasello, P. Duquenne, H. Schwenk, H. Gong, H. Inaguma, S. Popuri, C. Wang, J. Pino, W. Hsu, and A. Lee Speech-to-speech translation for a real-world unwritten language . In Findings of the Association for Computational Linguistics: ACL 2023 , A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.) , Toronto, Canada , pp. 4969–4983 . External Links: Link , Document Cited by: §1 .
Chen et al. (2022) S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei WavLM: large-scale self-supervised pre-training for full stack speech processing . IEEE Journal of Selected Topics in Signal Processing 16 ( 6 ), pp. 1505–1518 . External Links: Document Cited by: §2 .
Chen et al. (2024b) W. Chen, W. Zhang, Y. Peng, X. Li, J. Tian, J. Shi, X. Chang, S. Maiti, K. Livescu, and S. Watanabe Towards robust speech representation learning for thousands of languages . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.) , Miami, Florida, USA , pp. 10205–10224 . External Links: Link , Document Cited by: §1 , §1 , §2 , §5.1 .
Conneau et al. (2021) A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli Unsupervised cross-lingual representation learning for speech recognition . In Interspeech 2021 , pp. 2426–2430 . External Links: Document , ISSN 2958-1796 Cited by: §1 , §2 , §4.2 .
de Seyssel et al. (2022) M. de Seyssel, M. Lavechin, Y. Adi, E. Dupoux, and G. Wisniewski Probing phoneme, language and speaker information in unsupervised speech representations . In Interspeech 2022 , pp. 1402–1406 . External Links: Document , ISSN 2958-1796 Cited by: §1 .
Défossez et al. (2024) A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour Moshi: a speech-text foundation model for real-time dialogue . Note: Version 2 External Links: 2410.00037 , Link Cited by: §1 .
Deng and Erler (1991) L. Deng and K. Erler Microstructural speech units and their hmm representation for discrete utterance speech recognition . In [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing , Vol. , pp. 193–196 vol.1 . External Links: Document Cited by: §2 .
Dunbar et al. (2017) E. Dunbar, X. N. Cao, J. Benjumea, J. Karadayi, M. Bernard, L. Besacier, X. Anguera, and E. Dupoux The zero resource speech challenge 2017 . In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) , Vol. , pp. 323–330 . External Links: Document Cited by: Appendix A , §1 , §2 , §5.3 .
Dunbar et al. (2022) E. Dunbar, N. Hamilakis, and E. Dupoux Self-supervised language learning from raw audio: lessons from the zero resource speech challenge . IEEE Journal of Selected Topics in Signal Processing 16 ( 6 ), pp. 1211–1226 . External Links: Document Cited by: §1 , §1 , §2 .
Eide et al. (1993) E. Eide, J. R. Rohlicek, H. Gish, and S. Mitter A linguistic feature representation of the speech waveform . In 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing , Vol. 2 , pp. 483–486 vol.2 . External Links: Document Cited by: §2 .
Elenius and Takacs (1991) K. Elenius and G. Takacs Phoneme recognition with an artificial neural network . In 2nd European Conference on Speech Communication and Technology (Eurospeech 1991) , pp. 121–124 . External Links: Document , ISSN 1018-4074 Cited by: §2 .
Feng et al. (2023) S. Feng, M. Tu, R. Xia, C. Huang, and Y. Wang Language-universal phonetic representation in multilingual speech pretraining for low-resource speech recognition . In Interspeech 2023 , pp. 1384–1388 . External Links: Document , ISSN 2958-1796 Cited by: §1 , §2 , §2 .
Frankel et al. (2007) J. Frankel, M. Magimai-Doss, S. King, K. Livescu, and Ö. Çetin Articulatory feature classifiers trained on 2000 hours of telephone speech . In 8th Annual Conference of the International Speech Communication Association, INTERSPEECH 2007, Antwerp, Belgium, August 27-31, 2007 , pp. 2485–2488 . External Links: Link , Document Cited by: §2 .
Hallap et al. (2023) M. Hallap, E. Dupoux, and E. Dunbar Evaluating context-invariance in unsupervised speech representations . In Interspeech 2023 , pp. 2973–2977 . External Links: Document , ISSN 2958-1796 Cited by: §1 , §1 , §2 .
Hsu et al. (2021) W. Hsu, B. Bolte, Y. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed HuBERT: self-supervised speech representation learning by masked prediction of hidden units . IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 ( ), pp. 3451–3460 . External Links: Document Cited by: §1 , §1 , §2 , §3.1 , §3.2 .
King and Taylor (2000) S. King and P. Taylor Detection of phonological features in continuous speech using neural networks . Computer Speech & Language 14 ( 4 ), pp. 333–353 . External Links: ISSN 0885-2308 , Document , Link Cited by: §2 .
Kingma and Ba (2015) D. P. Kingma and J. Ba Adam: A method for stochastic optimization . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , Y. Bengio and Y. LeCun (Eds.) , External Links: Link Cited by: §4.2 .
Kirchhoff (1999) K. Kirchhoff Robust Speech Recognition Using Articulatory Information . PhD dissertation , University of Bielefeld . External Links: Link Cited by: §2 .
Kuhl (1993) P. K. Kuhl Innate predispositions and the effects of experience in speech perception: the native language magnet theory . In Developmental Neurocognition: Speech and Face Processing in the First Year of Life , B. de Boysson-Bardies, S. de Schonen, P. Jusczyk, P. McNeilage, and J. Morton (Eds.) , pp. 259–274 . External Links: ISBN 978-94-015-8234-6 , Document , Link Cited by: §1 .
Lakhotia et al. (2021) K. Lakhotia, E. Kharitonov, W. Hsu, Y. Adi, A. Polyak, B. Bolte, T. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux On generative spoken language modeling from raw audio . Transactions of the Association for Computational Linguistics 9 , pp. 1336–1354 . External Links: Link , Document Cited by: §1 , §1 .
Lavechin et al. (2025) M. Lavechin, M. de Seyssel, H. Titeux, G. Wisniewski, H. Bredin, A. Cristia, and E. Dupoux Simulating early phonetic and word learning without linguistic categories . Developmental Science 28 ( 2 ), pp. e13606 . External Links: Document , Link , https://onlinelibrary.wiley.com/doi/pdf/10.1111/desc.13606 Cited by: §1 .
Livescu et al. (2007) K. Livescu, O. Cetin, M. Hasegawa-Johnson, S. King, C. Bartels, N. Borges, A. Kantor, P. Lal, L. Yung, A. Bezman, S. Dawson-Haggerty, B. Woods, J. Frankel, M. Magimai-Doss, and K. Saenko Articulatory feature-based methods for acoustic and audio-visual speech recognition: summary from the 2006 JHU summer workshop . In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07 , Vol. 4 , pp. IV–621–IV–624 . External Links: Document Cited by: §2 .
Ma et al. (2023) Z. Ma, Z. Zheng, G. Yang, Y. Wang, C. Zhang, and X. Chen Pushing the limits of unsupervised unit discovery for SSL speech representation . In Interspeech 2023 , pp. 1269–1273 . External Links: Document , ISSN 2958-1796 Cited by: §1 , §2 .
McAuliffe et al. (2017) M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi . In Proc. Interspeech 2017 , pp. 498–502 . External Links: Document Cited by: Appendix A .
Mohamed et al. (2022) A. Mohamed, H. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S. Li, K. Livescu, L. Maaløe, T. N. Sainath, and S. Watanabe Self-supervised speech representation learning: a review . IEEE Journal of Selected Topics in Signal Processing 16 ( 6 ), pp. 1179–1210 . External Links: Document Cited by: §1 .
Mohamed et al. (2024) M. Mohamed, O. D. Liu, H. Tang, and S. Goldwater Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations . In Interspeech 2024 , pp. 3625–3629 . External Links: Document , ISSN 2958-1796 Cited by: §1 .
Mortensen et al. (2016) D. R. Mortensen, P. Littell, A. Bharadwaj, K. Goyal, C. Dyer, and L. Levin PanPhon: a resource for mapping IPA segments to articulatory feature vectors . In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers , Y. Matsumoto and R. Prasad (Eds.) , Osaka, Japan , pp. 3475–3484 . External Links: Link Cited by: Appendix A , §2 , §3.1 .
Oord et al. (2019) A. v. d. Oord, Y. Li, and O. Vinyals Representation learning with contrastive predictive coding . Note: Version 2 External Links: 1807.03748 , Link Cited by: §1 .
Papcun et al. (1992) G. Papcun, J. Hochberg, T. R. Thomas, F. Laroche, J. Zacks, and S. Levy Inferring articulation and recognizing gestures from acoustics with a neural network trained on x-ray microbeam data . The Journal of the Acoustical Society of America 92 ( 2 ), pp. 688–700 . External Links: Document Cited by: §2 .
Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay Scikit-learn: machine learning in Python . Journal of Machine Learning Research 12 ( 85 ), pp. 2825–2830 . External Links: Link Cited by: Appendix C .
Poli et al. (2025) M. Poli, E. Chemla, and E. Dupoux Fastabx: a library for efficient computation of abx discriminability . External Links: 2505.02692 , Link Cited by: §5.2 .
Pratap et al. (2024) V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W. Hsu, A. Conneau, and M. Auli Scaling speech technology to 1,000+ languages . Journal of Machine Learning Research 25 ( 97 ), pp. 1–52 . External Links: Link Cited by: §1 , §2 , §5.1 .
Rouard et al. (2025) S. Rouard, M. Orsini, A. Roebel, N. Zeghidour, and A. Défossez Continuous audio language models . Note: Version 2 External Links: 2509.06926 , Link Cited by: §1 .
Schatz et al. (2021) T. Schatz, N. H. Feldman, S. Goldwater, X. Cao, and E. Dupoux Early phonetic learning without phonetic categories: insights from large-scale simulations on realistic input . Proceedings of the National Academy of Sciences 118 ( 7 ), pp. e2001844118 . External Links: Document , Link , https://www.pnas.org/doi/pdf/10.1073/pnas.2001844118 Cited by: §1 .
Schatz et al. (2013) T. Schatz, V. Peddinti, F. Bach, A. Jansen, H. Hermansky, and E. Dupoux Evaluating speech features with the minimal-pair abx task: analysis of the classical mfc/plp pipeline . In Interspeech 2013 , pp. 1781–1785 . External Links: Document , ISSN 2958-1796 Cited by: §1 .
Schatz (2016) T. Schatz ABX-Discriminability Measures and Applications . Theses , Université Paris 6 (UPMC) . External Links: Link Cited by: §1 , §2 , §5.2 .
Shi et al. (2023) J. Shi, D. Berrebbi, W. Chen, E. Hu, W. Huang, H. Chung, X. Chang, S. Li, A. Mohamed, H. Lee, and S. Watanabe ML-SUPERB: multilingual speech universal performance benchmark . In Interspeech 2023 , pp. 884–888 . External Links: Document , ISSN 2958-1796 Cited by: §2 .
Staib et al. (2020) M. Staib, T. H. Teh, A. Torresquintero, D. S. R. Mohan, L. Foglianti, R. Lenain, and J. Gao Phonological features for 0-shot multilingual speech synthesis . In Interspeech 2020 , pp. 2942–2946 . External Links: Document , ISSN 2958-1796 Cited by: §2 .
Versteegh et al. (2015) M. Versteegh, R. Thiollière, T. Schatz, X. N. Cao, X. Anguera, A. Jansen, and E. Dupoux The zero resource speech challenge 2015 . In Interspeech 2015 , pp. 3169–3173 . External Links: Document , ISSN 2958-1796 Cited by: §1 , §2 .
Wang et al. (2022) C. Wang, Y. Wang, Y. Wu, S. Chen, J. Li, S. Liu, and F. Wei Supervision-guided codebooks for masked prediction in speech pre-training . In Interspeech 2022 , pp. 2643–2647 . External Links: Document , ISSN 2958-1796 Cited by: §1 , §2 .
Wang et al. (2021) C. Wang, Y. Wu, Y. Qian, K. Kumatani, S. Liu, F. Wei, M. Zeng, and X. Huang UniSpeech: unified speech representation learning with labeled and unlabeled data . In Proceedings of the 38th International Conference on Machine Learning , M. Meila and T. Zhang (Eds.) , Proceedings of Machine Learning Research , Vol. 139 , pp. 10937–10947 . External Links: Link Cited by: §2 .
Werker et al. (2007) J. F. Werker, F. Pons, C. Dietrich, S. Kajikawa, L. Fais, and S. Amano Infant-directed speech supports phonetic category learning in English and Japanese . Cognition 103 ( 1 ), pp. 147–162 . External Links: ISSN 0010-0277 , Document , Link Cited by: §1 .
Yang et al. (2021) S. Yang, P. Chi, Y. Chuang, C. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G. Lin, T. Huang, W. Tseng, K. Lee, D. Liu, Z. Huang, S. Dong, S. Li, S. Watanabe, A. Mohamed, and H. Lee SUPERB: speech processing universal performance benchmark . In Interspeech 2021 , pp. 1194–1198 . External Links: Document , ISSN 2958-1796 Cited by: §2 , §3.1 .
Zanon Boito et al. (2024) M. Zanon Boito, V. Iyer, N. Lagos, L. Besacier, and I. Calapodescu mHuBERT-147: a compact multilingual hubert model . In Interspeech 2024 , pp. 3939–3943 . External Links: Document , ISSN 2958-1796 Cited by: §1 , §1 , §2 , §5.1 .
Żelasko et al. (2022) P. Żelasko, S. Feng, L. Moro Velázquez, A. Abavisani, S. Bhati, O. Scharenborg, M. Hasegawa-Johnson, and N. Dehak Discovering phonetic inventories with crosslingual automatic speech recognition . Computer Speech & Language 74 , pp. 101358 . External Links: ISSN 0885-2308 , Document , Link Cited by: §1 , §5.5 .
Zhang et al. (2021) C. Zhang, X. Tan, Y. Ren, T. Qin, K. Zhang, and T. Liu UWSpeech: speech to speech translation for unwritten languages . Proceedings of the AAAI Conference on Artificial Intelligence 35 ( 16 ), pp. 14319–14327 . External Links: Link , Document Cited by: §1 .
Appendix A Data
附录 A 数据
| IETF code | Language | # Hours |
|---|---|---|
| ab | Abkhaz | |
| am | Amharic | |
| ba | Bashkir | |
| be | Belarusian | |
| bg | Bulgarian | |
| bn | Bengali | |
| ca | Catalan | |
| ckb | Central kurdish | |
| cs | Czech | |
| cv | Chuvash | |
| dv | Maldivian | |
| el | Greek | |
| eu | Basque | |
| gn | Guarani | |
| ha | Hausa | |
| hi | Hindi | |
| hsb | Upper sorbian | |
| hu | Hungarian | |
| hy-AM | Armenian |
| IETF code | Language | # Hours |
|---|---|---|
| id | Indonesian | |
| it | Italian | |
| ja | Japanese | |
| ka | Georgian | |
| kk | Kazakh | |
| kmr | Northern kurdish | |
| ko | Korean | |
| ky | Kyrgyz | |
| lij | Ligurian | |
| lt | Lithuanian | |
| ml | Malayalam | |
| mn | Mongolian | |
| mr | Marathi | |
| mt | Maltese | |
| myv | Erzya | |
| nan-tw | Taiwanese hokkien | |
| pa-IN | Punjabi | |
| nl | Dutch |
| IETF code | Language | # Hours |
|---|---|---|
| pl | Polish | |
| pt | Portuguese | |
| ro | Romanian | |
| ru | Russian | |
| rw | Kinyarwanda | |
| sk | Slovak | |
| sl | Slovenian | |
| sq | Albanian | |
| sr | Serbian | |
| sv-SE | Swedish | |
| tk | Turkmen | |
| tt | Tatar | |
| ug | Uyghur | |
| ur | Urdu | |
| uz | Uzbek | |
| vi | Vietnamese | |
| yo | Yoruba | |
| yue | Cantonese | |
| Total |
Table 5: List of languages with their amount of speech included in the training set.
| IETF 代码 | 语言 | 小时数 |
|---|---|---|
| ab | Abkhaz | |
| am | Amharic | |
| ba | Bashkir | |
| be | Belarusian | |
| bg | Bulgarian | |
| bn | Bengali | |
| ca | Catalan | |
| ckb | Central kurdish | |
| cs | Czech | |
| cv | Chuvash | |
| dv | Maldivian | |
| el | Greek | |
| eu | Basque | |
| gn | Guarani | |
| ha | Hausa | |
| hi | Hindi | |
| hsb | Upper sorbian | |
| hu | Hungarian | |
| hy-AM | Armenian |
| IETF 代码 | 语言 | 小时数 |
|---|---|---|
| id | Indonesian | |
| it | Italian | |
| ja | Japanese | |
| ka | Georgian | |
| kk | Kazakh | |
| kmr | Northern kurdish | |
| ko | Korean | |
| ky | Kyrgyz | |
| lij | Ligurian | |
| lt | Lithuanian | |
| ml | Malayalam | |
| mn | Mongolian | |
| mr | Marathi | |
| mt | Maltese | |
| myv | Erzya | |
| nan-tw | Taiwanese hokkien | |
| pa-IN | Punjabi | |
| nl | Dutch |
| IETF 代码 | 语言 | 小时数 |
|---|---|---|
| pl | Polish | |
| pt | Portuguese | |
| ro | Romanian | |
| ru | Russian | |
| rw | Kinyarwanda | |
| sk | Slovak | |
| sl | Slovenian | |
| sq | Albanian | |
| sr | Serbian | |
| sv-SE | Swedish | |
| tk | Turkmen | |
| tt | Tatar | |
| ug | Uyghur | |
| ur | Urdu | |
| uz | Uzbek | |
| vi | Vietnamese | |
| yo | Yoruba | |
| yue | Cantonese | |
| Total |
表 5：训练集所含 种语言及其语音时长。
We use the annotations of Common Voice (ardila20_common_voice) made by ahn22_voxcommunis. At the time of download (21 August 2024), the dataset consisted of languages. Five languages (Swahili, Tamil, Thai, Turkish and Ukrainian) were held out for hyperparameter tuning, and three languages (French, Mandarin and Hong Kong Mandarin) were discarded because of their presence in the test set. Table 5 contains the list of the languages kept for training.
我们使用 ahn22_voxcommunis 为 Common Voice（ardila20_common_voice）制作的标注。下载时（2024 年 8 月 21 日），该数据集包含 种语言。五种语言（斯瓦希里语、泰米尔语、泰语、土耳其语和乌克兰语）留出用于超参数调节，三种语言（法语、普通话和香港普通话）因出现在测试集中而被去掉。表 5 列出了保留用于训练的 种语言。
Training languages.
We filter out utterances containing spn segments, which indicate alignment errors from Montreal Forced Aligner (mcauliffe17_mfa), or those that are excessively long (non-silent phones exceeding ). We retain only utterances lasting between . We fix misplaced diacritics that were incorrectly attached to adjacent phones in four languages. We also handle IPA characters that PanPhon (mortensen16_panphon) does not recognise by mapping them to their proper equivalents (e.g. [] becomes [g]). To prevent high-resource languages from dominating the training data for MauBERT, we limit each language to a maximum of , yielding a total of of multilingual training speech.
训练语言。
我们滤除含有 spn 片段的语句（这表示 Montreal Forced Aligner（mcauliffe17_mfa）出现了对齐错误），以及过长的语句（非静音音子超过 ）。我们只保留时长在 – 之间的语句。对四种语言中错误附着在相邻音子上的变音符号做了修正。对于 PanPhon（mortensen16_panphon）无法识别的 IPA 字符，我们把它们映射到合适的对应形式（例如 [] 变为 [g]）。为避免高资源语言在 MauBERT 的训练数据中占主导，我们将每种语言限制在最多 ，最终得到合计 的多语言训练语音。
Development languages.
We apply the same preprocessing pipeline used for the training languages to the development languages. Next, we combine the three original Common Voice splits (train, dev, and test) and create three new splits with the following specifications for each language: (i) the test set contains of audio uniformly distributed across speakers, (ii) the training set contains uniformly distributed across speakers, and (iii) the validation set contains hours of audio. We ensure that speakers are completely disjoint across all newly created splits.
开发语言。
我们对开发语言采用与训练语言相同的预处理流程。随后，把 Common Voice 原有的三个划分（train、dev 和 test）合并，再为每种语言新建三个划分，规格如下：（i）测试集包含 – 音频，均匀分配给 位说话人；（ii）训练集包含 –，均匀分配给 位说话人；（iii）验证集包含 – 音频。我们确保新建各划分的说话人完全不交叉。
Test languages.
We use the five languages from the Zero Resource Challenge dunbar17_zerospeech: English, French, Mandarin, German, and Wolof. From the original long-form recordings (each corresponding to a different speaker), we extract[^9] training and validation splits of up to per language through the following process: (i) apply voice activity detection using official challenge alignments, (ii) segment recordings into clips including silences of up to , and (iii) assign each speaker’s clips exclusively to either training or validation splits to ensure speaker disjointness. This yields training sets of and validation sets of , with Mandarin having the smallest splits and European languages having the largest.
测试语言。
我们使用 Zero Resource Challenge （dunbar17_zerospeech）中的五种语言：英语、法语、普通话、德语和沃洛夫语。从原始的长录音（每条对应不同说话人）中，我们按如下步骤提取[^9]每种语言至多 的训练和验证划分：（i）用官方挑战对齐做语音活动检测；（ii）把录音切成 – 的片段，其中可包含至多 的静音；（iii）把每位说话人的片段只分到训练或验证中的一个划分，以保证说话人不交叉。由此得到 – 的训练集和 – 的验证集，其中普通话的划分最小，欧洲语言的划分最大。
Appendix B Training
附录 B 训练
Table 6 lists the (hyper-) parameters used for multilingual feature recognition. All the (hyper-) parameters for self-supervised and supervised fine-tuning can be found in the released code.
表 6 列出了用于多语言特征识别的（超）参数。自监督和有监督微调的全部（超）参数见已发布的代码。
Language Up-sampling.
During multilingual pre-training, we draw from the multinomial distribution , where is the number of audios of language , is the training set size, and is the up-sampling factor controlling the importance between high- and low-resource languages.
语言上采样。
多语言预训练期间，我们从多项式分布 中抽样，其中 是语言 的音频条数， 是训练集规模， 是上采样因子，用来调节高资源和低资源语言的相对权重。
Length grouping.
To reduce unused representations in batches, we split the multilingual data into buckets of audio of roughly the same length.
按长度分组。
为了减少批次中未使用的表征，我们把多语言数据按音频长度大致相近的原则分到不同桶中。
Appendix C Clustering methods
附录 C 聚类方法
Ground-truth phones.
We use the collapsed list of segments from PanPhon for the development languages, and the list of unique phonemes from the official alignments for the test languages.
真实音子。
对开发语言，我们使用来自 PanPhon 的压缩音段列表；对测试语言，则使用官方对齐中的唯一音素列表。
K-means.
We run the MiniBatchKMeans algorithm from scikit-learn (pedregosa_11_scikit) on the training set for each development and test language. We select three different representations: (i) the best-performing layer from zero-shot mode, (ii) the feature logits for MauBERT-feat, (iii) and the phone logits (after reducing to only the phones seen during training) for MauBERT-phone.
K-means。
我们在每种开发语言和测试语言的训练集上运行 scikit-learn（pedregosa_11_scikit）中的 MiniBatchKMeans 算法。我们选取三种不同表征：（i）零样本模式下表现最好的层，（ii）MauBERT-feat 的特征 logits，（iii）以及 MauBERT-phone 的音子 logits（先把输出限制为训练期间见过的音子）。
Predicted phones.
First, we determine the most likely phone label for each frame in the training set. We then prune the phone prediction layer by removing the output heads corresponding to phones absent from the training set (all phones labels). For the phone freq. labels, we further restrict the prediction layer to the most frequent phones in the training set, discarding the remaining heads and fine-tuning the layer to produce highly confident predictions for these phones.
预测音子。
首先，我们为训练集中的每一帧确定最可能的音子标签。然后修剪音子预测层，去掉训练集中未出现的音子所对应的输出头（all phones 标签）。对 phone freq. 标签，我们进一步把预测层限制为训练集中频次最高的 个音子，丢掉其余输出头，并微调该层，使它对这 个音子给出高置信度预测。
Predicted features.
For the feat. freq. labels, we extract predicted articulatory feature logits for each frame in the training set, apply a sigmoid activation, and hard-threshold the resulting values at to obtain binary feature vectors. We then compute the frequency of each unique binary vector across the training set and retain the most common ones. For pseudo-labelling, each frame is assigned to its nearest neighbour among these frequent vectors under the distance, with ties broken in favour of vectors with fewer zero-valued features.
预测特征。
对 feat. freq. 标签，我们提取训练集中每一帧的预测发音特征 logits，经过 sigmoid 激活后，以 为硬阈值得到二值特征向量。随后统计训练集中每个独特二值向量的频率，保留最常见的 个。做伪标注时，每一帧在 距离下被指派给这 个高频向量中的最近邻；若出现平局，则优先选择取值为零的特征更少的向量。
| Paramenters | Value |
|---|---|
| Model | |
| Up-projection dimension | |
| BLSTM layers | |
| BLSTM dimension | |
| BLSTM dropout | |
| BLSTM layer normalisation | No |
| Phone MLP hidden dimension | |
| Phone MLP activation function | GELU |
| Features | |
| Diphthong feature strategy | split |
| Zero values loss | ignore |
| Hyper-Paramenters | Value |
|---|---|
| Data | |
| Up-sample factor () | |
| Batch size | |
| Optimizer | |
| Name | Adam |
| Peak learning rate | |
| Betas | (, ) |
| Weight decay | No |
| Epsilon | |
| Warmup steps | |
| Hold steps | |
| Decay steps | |
| Mixed precision | fp16 |
Table 6: Model parameters and training hyper-parameters used for MauBERT-feat.
| 参数 | 取值 |
|---|---|
| 模型 | |
| 升维维度 | |
| BLSTM 层数 | |
| BLSTM 维度 | |
| BLSTM dropout | |
| BLSTM 层归一化 | No |
| 音子 MLP 隐层维度 | |
| 音子 MLP 激活函数 | GELU |
| 特征 | |
| 双元音特征策略 | split |
| 零值损失 | ignore |
| Hyper-参数 | 取值 |
|---|---|
| 数据 | |
| 上采样因子 () | |
| 批次大小 | |
| 优化器 | |
| 名称 | Adam |
| 峰值学习率 | |
| Betas | (, ) |
| 权重衰减 | No |
| Epsilon | |
| 预热步数 | |
| 保持步数 | |
| 衰减步数 | |
| 混合精度 | fp16 |
表 6：MauBERT-feat 所用的模型参数与训练超参数。
Appendix D Inventory discovery supplement
附录 D 清单发现补充
| Language | Inventory size | Top | F1-optimal | |||||
|---|---|---|---|---|---|---|---|---|
| Prec. | Recall | F1 | Prec. | Recall | F1 | |||
| Swahili | ||||||||
| Tamil | ||||||||
| Thai | ||||||||
| Turkish | ||||||||
| Ukrainian |
Table 7: Precision, recall and F score for the inventory discovery on the development languages for the top- threshold and the best threshold for F score.
| 语言 | 清单规模 | Top | F1-optimal | |||||
|---|---|---|---|---|---|---|---|---|
| 精确率 | 召回率 | F1 | 精确率 | 召回率 | F1 | |||
| Swahili | ||||||||
| Tamil | ||||||||
| Thai | ||||||||
| Turkish | ||||||||
| Ukrainian |
表 7：在开发语言上做清单发现时，取频次最高的 个以及 F 最优阈值下的精确率、召回率和 F 分数。
| Lang. | Correctly predicted phones | Missing phones |
|---|---|---|
| th | m, i, k, j, u, a, p, w, n, t, l, s, b, N, e, o, h, d, f, E, O, i:, a:, u:, R, e:, kh, ph, th, E:, O:, t͡C, 7, 0: | P, o:, W, t͡Ch, W:, 7:, 0, a ı |
| tr | m, i, k, j, u, a, p, b, e, o, g, h, f, t͡S, S, d͡Z, R, t, n, d, W, y, s, l, z | m:, k:, j:, p:, b:, g:, h:, f:, t͡S:, S:, d͡Z:, v, v:, R:, t:, n:, Z, d:, s:, œ, l:, z: |
| uk | m, i, k, j, u, p, b, r, E, S, t, x, t͡S, n, Z, d, a, s, tj, lj, v, z, t͡s, sj, rj, l, nj, t͡sj | g, f, O, d͡Z, I, H, d͡z, dj, zj, d͡zj |
Table 8: Phonetic inventory prediction using an F1-optimal threshold for Thai, Turkish and Ukrainian. The language inventories comprise all the phones observed in the alignments from VoxCommunis.
| 语言 | 正确预测的音子 | 缺失的音子 |
|---|---|---|
| th | m, i, k, j, u, a, p, w, n, t, l, s, b, N, e, o, h, d, f, E, O, i:, a:, u:, R, e:, kh, ph, th, E:, O:, t͡C, 7, 0: | P, o:, W, t͡Ch, W:, 7:, 0, a ı |
| tr | m, i, k, j, u, a, p, b, e, o, g, h, f, t͡S, S, d͡Z, R, t, n, d, W, y, s, l, z | m:, k:, j:, p:, b:, g:, h:, f:, t͡S:, S:, d͡Z:, v, v:, R:, t:, n:, Z, d:, s:, œ, l:, z: |
| uk | m, i, k, j, u, p, b, r, E, S, t, x, t͡S, n, Z, d, a, s, tj, lj, v, z, t͡s, sj, rj, l, nj, t͡sj | g, f, O, d͡Z, I, H, d͡z, dj, zj, d͡zj |
表 8：在泰语、土耳其语和乌克兰语上，用 F1 最优阈值预测音位清单。语言清单包含 VoxCommunis 对齐中观测到的全部音子。
Appendix E Spoken language modelling evaluation
附录 E 口语语言建模评测
To complement the ABX phonetic probing results, we assess whether the discrete units produced by MauBERT lead to better spoken language models (sLMs) through using these units as input features to a spoken LM trained in a larger dataset ( hours of Libri-Light).
为补充 ABX 语音探测结果，我们考察 MauBERT 产生的离散单元能否带来更好的口语语言模型（sLM）：把这些单元作为输入特征，在更大规模的数据集（Libri-Light 的 小时）上训练口语语言模型。
We use our zero-shot models and SSL models fine-tuned on the English adaptation set () as frame-level encoders. A K-means model (with clusters) is trained on the resulting embeddings to produce discrete tokens, which are then used to train an OPT-M language model (zhang2022optopenpretrainedtransformer) with fairseq2 (balioglu2023fairseq2), following the architectural choices of hassid2023textually and poli2025spidr. Training uses the hours subset of Libri-Light (9052942) on GPUs, with a context length of tokens, batches of at most tokens, for steps. The learning rate is set to with a linear warmup of steps and cosine annealing; remaining hyper-parameters follow OPT-M defaults. We choose the checkpoint with the lowest validation loss.
我们用零样本模型，以及在英语适应集（）上微调过的 SSL 模型，作为帧级编码器。再在得到的嵌入上训练一个 K-means 模型（ 个簇）以产生离散标记，随后用 fairseq2（balioglu2023fairseq2）训练一个 OPT-M 语言模型（zhang2022optopenpretrainedtransformer），结构选择遵循 hassid2023textually 和 poli2025spidr。训练使用 Libri-Light 的 小时子集（9052942），在 块 GPU 上进行，上下文长度为 个标记，每批至多 个标记，共 步。学习率设为 ，线性预热 步后再做余弦退火；其余超参数沿用 OPT-M 的默认值。我们选择验证损失最低的检查点。
We compare MauBERT against the monolingual SSL models: wav2vec 2.0 (baevski2020wav2vec), HuBERT-base (hsu21_hubert), WavLM-base chen_22_wavlm and DinoSR liu2023dinosr, all trained on hours of English. We evaluate sLMs on three tasks: sWUGGY (lexical), sBLIMP (syntactic), and tSC (semantic consistency), reporting scores as percentages. We consider two vocabulary sizes: (Table 9) and (Table 10). For Table 9, the MauBERT-phone zero-shot (all phones) variant uses a one-hot encoding over all observed phones in all our data; for Table 10, the MauBERT-phone SSL fine-tuned (logits) variant uses a one-hot encoding over all observed phones in the English adaptation data.
我们将 MauBERT 与单语 SSL 模型比较：wav2vec 2.0（baevski2020wav2vec）、HuBERT-base（hsu21_hubert）、WavLM-base（chen_22_wavlm）和 DinoSR（liu2023dinosr），它们都在 小时英语上训练。我们在三项任务上评估口语语言模型：sWUGGY（词汇）、sBLIMP（句法）和 tSC（语义一致性），分数以百分比报告。我们考虑两种词表规模：（表 9）和 （表 10）。在表 9 中，MauBERT-phone 的零样本（全部音子）变体对数据中观测到的全部 个音子使用独热编码；在表 10 中，MauBERT-phone 经 SSL 微调（logits）的变体，对英语适应数据中观测到的全部 个音子使用独热编码。
| Model | Units | sWUGGY | sBLIMP | tSC | |
|---|---|---|---|---|---|
| all | in-vocab | ||||
| Monolingual models | |||||
| wav2vec 2.0† | K-means (L) | ||||
| WavLM-base | K-means (L) | ||||
| HuBERT-base | K-means (L) | ||||
| DinoSR† | codebook (L) | ||||
| SpidR† | codebook (L) | ||||
| Ours | |||||
| MauBERT-feat | |||||
| zero-shot | K-means (L) | ||||
| zero-shot | feat. freq. | ||||
| + self-supervised FT | K-means (L) | ||||
| MauBERT-phone | |||||
| zero-shot | K-means (proj) | ||||
| zero-shot⋄ | one-hot (all phones) | ||||
| + self-supervised FT | K-means (L) | 57.34 | |||
| † Fetched from poli2025spidr. |
Table 9: Spoken language modelling scores in English (). Monolingual self-supervised learning baselines on top. ⋄All phones seen in the data are kept (). Best scores in bold, second best underlined.
| 模型 | 单元 | sWUGGY | sBLIMP | tSC | |
|---|---|---|---|---|---|
| 全部 | 词表内 | ||||
| 单语模型 | |||||
| wav2vec 2.0† | K-means (L) | ||||
| WavLM-base | K-means (L) | ||||
| HuBERT-base | K-means (L) | ||||
| DinoSR† | codebook (L) | ||||
| SpidR† | codebook (L) | ||||
| 本文 | |||||
| MauBERT-feat | |||||
| 零样本 | K-means (L) | ||||
| 零样本 | feat. freq. | ||||
| + 自监督微调 | K-means (L) | ||||
| MauBERT-phone | |||||
| 零样本 | K-means (proj) | ||||
| 零样本⋄ | one-hot (all phones) | ||||
| + 自监督微调 | K-means (L) | 57.34 | |||
| † 取自 poli2025spidr. |
表 9：英语口语语言建模分数（）。 上方为单语自监督学习基线。⋄保留数据中见到的全部音子（）。最佳分数用粗体，次优分数加下划线。
Results.
Tables 9 and 10 reveal a nuanced picture that does not always mirror the ABX probing results. In zero-shot mode, MauBERT variants underperform the monolingual HuBERT-base and WavLM-base baselines on sWUGGY and sBLIMP, despite their superior phonetic discriminability in ABX. This gap is consistent across both vocabulary sizes and suggests that the richer cross-lingual phonetic structure encoded by MauBERT does not translate straightforwardly into better lexical or syntactic priors for English-specific language modelling. After self-supervised fine-tuning, however, both MauBERT variants recover competitive performance, approaching WavLM-base on sWUGGY ( vs. ) and matching or exceeding HuBERT-base across all three tasks. This mirrors the pattern observed in ABX scores, where self-supervised fine-tuning substantially closes the gap with toplines. The MauBERT-phone variant with K-means fine-tuning achieves the strongest sLM performance among our models at (Table 10: sWUGGY, sBLIMP), consistent with its leading ABX scores in Table 3.
结果。
表 9 和表 10 呈现出一幅并不总是与 ABX 探测结果一致的细致图景。零样本时，尽管 MauBERT 变体在 ABX 上的语音可区分性更强，它们在 sWUGGY 和 sBLIMP 上仍落后于单语的 HuBERT-base 和 WavLM-base 基线。这一差距在两种词表规模下都存在，说明 MauBERT 所编码的更丰富的跨语言语音结构，并不能直接转化成针对英语的、更好的词汇或句法先验。不过，经过自监督微调后，两个 MauBERT 变体都恢复到有竞争力的水平：在 sWUGGY 上接近 WavLM-base（– 对 ），并在全部三项任务上达到或超过 HuBERT-base。这与 ABX 分数中的模式一致，即自监督微调会大幅缩小与上限之间的差距。在 时，MauBERT-phone 配上 K-means 微调在我们的模型中取得了最强的口语语言建模表现（表 10： sWUGGY， sBLIMP），也与它在表 3 中领先的 ABX 分数相符。
Taken together, these results suggest that while MauBERT’s multilingual phonetic inductive biases do not directly boost English sLM performance in zero-shot mode, the representations become competitive after even minimal self-supervised adaptation, paralleling the conclusions drawn from ABX probing. This conclusion is only tentative, because the base speech model was trained in English itself and is therefore difficult to disentangle the effect of the pre-training from our fine-tuning. Further studies should start with a base universal model trained on languages other than English, and/or tested on a larger set of held-out languages.
综合来看，这些结果表明：MauBERT 的多语言语音归纳偏置并不能在零样本下直接提升英语口语语言建模的表现，但只需少量自监督适应，表征就能变得有竞争力，这与从 ABX 探测中得出的结论是平行的。这一结论还只是初步的，因为基础语音模型本身就是在英语上训练的，因而很难把预训练的作用和我们的微调分开。后续研究应当从在英语以外的语言上训练的通用基础模型出发，并且/或者在更大的留出语言集合上测试。
| Model | Units | sWUGGY | sBLIMP | tSC | |
|---|---|---|---|---|---|
| all | in-vocab | ||||
| MauBERT-feat | |||||
| zero-shot | K-means (L) | ||||
| zero-shot | feat. freq. | ||||
| + self-supervised FT | K-means (L) | ||||
| + self-supervised FT | one-hot (logits) | ||||
| MauBERT-phone | |||||
| zero-shot | K-means (proj) | ||||
| zero-shot | phone freq. | ||||
| + self-supervised FT | K-means (L) | ||||
| + self-supervised FT⋄ | one-hot (logits) |
Table 10: Spoken language modelling scores in English (). ⋄Only the phones seen during fine-tuning are used as vocabulary (). Best scores in bold, second best underlined.
| 模型 | 单元 | sWUGGY | sBLIMP | tSC | |
|---|---|---|---|---|---|
| 全部 | 词表内 | ||||
| MauBERT-feat | |||||
| 零样本 | K-means (L) | ||||
| 零样本 | feat. freq. | ||||
| + 自监督微调 | K-means (L) | ||||
| + 自监督微调 | one-hot (logits) | ||||
| MauBERT-phone | |||||
| 零样本 | K-means (proj) | ||||
| 零样本 | phone freq. | ||||
| + 自监督微调 | K-means (L) | ||||
| + 自监督微调⋄ | one-hot (logits) |
表 10：英语口语语言建模分数（）。 ⋄只用微调期间见过的音子作为词表（）。最佳分数用粗体，次优分数加下划线。
Appendix F DiscoPhon benchmark
附录 F DiscoPhon 基准
DiscoPhon (poli2026_discophon) is a benchmark designed to evaluate the ability of speech representation models to encode phonemic information on unlabelled data. It consists of 6 development languages (German, Swahili, Tamil, Thai, Turkish and Ukrainian) and 6 test languages (Mandarin, English, Basque, French, Japanese and Wolof), selected to span a diverse range of phonological categories. These language splits differ from those used in §5. While MauBERT has not seen any of the development languages, it retains some English influence from the HuBERT backbone and some Japanese bias from the of Japanese in the training data.
DiscoPhon（poli2026_discophon）是一个用于评估语音表征模型能否在无标注数据上编码音素信息的基准。它包含 6 种开发语言（德语、斯瓦希里语、泰米尔语、泰语、土耳其语和乌克兰语）和 6 种测试语言（普通话、英语、巴斯克语、法语、日语和沃洛夫语），选择时覆盖了多样的音系范畴。这些语言划分与 §5 中使用的不同。MauBERT 没有见过其中任何一种开发语言，但由于 HuBERT 骨干，它仍带有一些英语影响，又因训练数据中有 日语而带有一定的日语偏向。
We compare MauBERT against the same probing baselines as in §5: MMS-1B, XEUS, mHuBERT-147, and HuBERT-base, using the best layers identified in Table 3. We report results under three conditions: (i) many-to-one with units, where each unit resulting from the clustering process is mapped to the most probable phoneme (Table 11); (ii) one-to-one with ground-truth vocabulary size, where each phoneme is assigned a unique unit (Table 12); and (iii) one-to-one with fine-tuned models predicting the ground-truth vocabulary size, yielding a phoneme–unit bijection (Table 13). We evaluate using Phone Error Rate (PER), -value, , Phoneme-Normalised Mutual Information (PNMI), and triphone-based ABX discriminability on continuous representations.
我们把 MauBERT 与 §5 中相同的探测基线进行比较：MMS-1B、XEUS、mHuBERT-147 和 HuBERT-base，并采用表 3 中确定的最佳层。我们在三种条件下报告结果：（i） 个单元的多对一，聚类得到的每个单元被映射到最可能的音素（表 11）；（ii）词表规模为真实值的一对一，每个音素被指派一个唯一单元（表 12）；（iii）微调模型预测真实词表规模的一对一，从而形成音素与单元的双射（表 13）。评估指标包括音子错误率（PER）、 值、、音素归一化互信息（PNMI），以及连续表征上基于三音子的 ABX 可区分性。
| dev languages | test languages | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PER | -value | PNMI | ABX c. | PER | -value | PNMI | ABX c. | |||
| Zero-shot | ||||||||||
| MMS-1B | ||||||||||
| XEUS | ||||||||||
| mHuBERT-147 | ||||||||||
| HuBERT | ||||||||||
| MauBERT-feat | ||||||||||
| MauBERT-phone | ||||||||||
| Self-supervised FT (10 h) | ||||||||||
| HuBERT + K-means (L11) | ||||||||||
| MauBERT-feat | ||||||||||
| + K-means (L9) | ||||||||||
| + K-means (feat) | ||||||||||
| + feat. freq. | ||||||||||
| + phone freq. | ||||||||||
| MauBERT-phone | ||||||||||
| + K-means (proj) | ||||||||||
| + K-means (phone) | ||||||||||
| + phone freq. | ||||||||||
| + all phones |
Table 11: DiscoPhon benchmark’s many-to-one scores ( units). Fine-tuned models predict or clusters (MauBERT-phone + all phones MPR predicts all the seen phones). Units are mapped to the most probable phoneme for evaluation. Target layer as in Table 3. Results (in ) averaged across dev. and test languages, resp. Triphone-based ABX averaged between within- and across-speaker conditions. K-means clustering: layer for pseudo-labels to be predicted between parentheses. Best scores in bold, second best underlined.
| 开发语言 | 测试语言 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PER | -value | PNMI | ABX c. | PER | -value | PNMI | ABX c. | |||
| 零样本 | ||||||||||
| MMS-1B | ||||||||||
| XEUS | ||||||||||
| mHuBERT-147 | ||||||||||
| HuBERT | ||||||||||
| MauBERT-feat | ||||||||||
| MauBERT-phone | ||||||||||
| 自监督微调（10 小时） | ||||||||||
| HuBERT + K-means (L11) | ||||||||||
| MauBERT-feat | ||||||||||
| + K-means (L9) | ||||||||||
| + K-means (feat) | ||||||||||
| + feat. freq. | ||||||||||
| + phone freq. | ||||||||||
| MauBERT-phone | ||||||||||
| + K-means (proj) | ||||||||||
| + K-means (phone) | ||||||||||
| + phone freq. | ||||||||||
| + all phones |
表 11：DiscoPhon 基准的多对一分数（ 个单元）。 微调模型预测 或 个簇（MauBERT-phone + all phones MPR 预测所有见过的音子）。评测时将单元映射到最可能的音素。目标层同表 3。结果（以 计）分别在开发语言和测试语言上取平均。基于三音子的 ABX 在说话人内与跨说话人条件之间取平均。K-means 聚类：括号中为待预测伪标签所在的层。最佳分数用粗体，次优分数加下划线。
| dev languages | test languages | |||||||
|---|---|---|---|---|---|---|---|---|
| PER | -value | PNMI | PER | -value | PNMI | |||
| Zero-shot | ||||||||
| MMS-1B | ||||||||
| XEUS | ||||||||
| mHuBERT-147 | ||||||||
| HuBERT | ||||||||
| MauBERT-feat | ||||||||
| MauBERT-phone | ||||||||
| Self-supervised FT (10 h) | ||||||||
| HuBERT + K-means (L11) | ||||||||
| MauBERT-feat | ||||||||
| + K-means (L9) | ||||||||
| + K-means (feat) | ||||||||
| + feat. freq. | ||||||||
| + phone freq. | ||||||||
| MauBERT-phone | ||||||||
| + K-means (proj) | ||||||||
| + K-means (phone) | ||||||||
| + phone freq. | ||||||||
| + all phones |
Table 12: DiscoPhon benchmark’s one-to-one scores (() units). Fine-tuned models predict or clusters (MauBERT-phone + all phones MPR predicts all the seen phones). Each phoneme is mapped to a single unit. Target layer as in Table 3, best scores in bold, second best underlined.
| 开发语言 | 测试语言 | |||||||
|---|---|---|---|---|---|---|---|---|
| PER | -value | PNMI | PER | -value | PNMI | |||
| 零样本 | ||||||||
| MMS-1B | ||||||||
| XEUS | ||||||||
| mHuBERT-147 | ||||||||
| HuBERT | ||||||||
| MauBERT-feat | ||||||||
| MauBERT-phone | ||||||||
| 自监督微调（10 小时） | ||||||||
| HuBERT + K-means (L11) | ||||||||
| MauBERT-feat | ||||||||
| + K-means (L9) | ||||||||
| + K-means (feat) | ||||||||
| + feat. freq. | ||||||||
| + phone freq. | ||||||||
| MauBERT-phone | ||||||||
| + K-means (proj) | ||||||||
| + K-means (phone) | ||||||||
| + phone freq. | ||||||||
| + all phones |
表 12：DiscoPhon 基准的一对一分数（() 个单元）。 微调模型预测 或 个簇（MauBERT-phone + all phones MPR 预测所有见过的音子）。每个音素映射到一个单元。目标层同表 3。最佳分数用粗体，次优分数加下划线。
Results.
The DiscoPhon results largely corroborate and extend the ABX findings from §5. In zero-shot mode, both MauBERT variants substantially outperform all baselines across PER, -value, , and PNMI in the many-to-one condition (Table 11), with MauBERT-feat and MauBERT-phone reducing PER by at least relative to the best multilingual baseline on development (mHuBERT-147) on test languages (XEUS). This advantage is consistent with the phonetic invariance improvements observed in Table 3, and reinforces the conclusion that multilingual articulatory supervision yields genuinely better phoneme-level representations, not merely better triphone discriminability.
结果。
DiscoPhon 的结果大体上印证并扩展了 §5 中的 ABX 发现。零样本时，在多对一条件（表 11）下，两个 MauBERT 变体在 PER、 值、 和 PNMI 上都大幅优于所有基线：相对开发语言上最好的多语言基线（mHuBERT-147）和测试语言上最好的基线（XEUS），MauBERT-feat 和 MauBERT-phone 至少把 PER 相对降低了 。这一优势与表 3 中观察到的语音不变性提升一致，也进一步说明，多语言发音监督带来的是真正更好的音素级表征，而不仅仅是更好的三音子可区分性。
After self-supervised fine-tuning, MauBERT variants further strengthen their lead. In the many-to-one condition (Table 11), MauBERT-phone with phone frequency clustering achieves the best PER and -value across both development and test languages ( PER, -value on dev; PER, -value on test), paralleling its top ABX performance. The feature frequency and phone frequency methods consistently outperform standard K-means, echoing the pattern from §5.4, where linguistically motivated clustering strategies showed advantages in longer temporal contexts.
经过自监督微调后，MauBERT 变体进一步扩大领先。在多对一条件（表 11）下，MauBERT-phone 配上按音子频率聚类，在开发语言和测试语言上都取得了最佳的 PER 和 值（开发集 PER ， 值 ；测试集 PER ， 值 ），与它在 ABX 上的领先表现一致。基于特征频率和音子频率的方法持续优于标准 K-means，这与 §5.4 中的模式相呼应：有语言学动机的聚类策略在更长的时间语境上更有优势。
The one-to-one condition (Tables 12 and 13) is considerably more demanding, as each phoneme must be assigned a dedicated unit. Here, the ranking among methods is preserved, but absolute scores deteriorate sharply for all models, highlighting the difficulty of achieving a clean phoneme–unit bijection without gold supervision. Notably, the K-means (phone) variant incurs a significant ABX cost in Table 13, suggesting that forcing a bijective mapping via K-means can hurt representational quality despite improving phoneme assignment metrics. Nonetheless, MauBERT variants with phone- or feature-frequency-based fine-tuning remain the strongest systems (Table 12: MauBERT-phone + phone freq. reaches PER, -value on dev), and the ground-truth vocabulary fine-tuning condition (Table 13) shows that phone-frequency MPR yields the best trade-off between PER and -value.
一对一条件（表 12 和表 13）要求高得多，因为每个音素都必须被指派一个专用单元。在这里，各方法的排序仍然保持，但所有模型的绝对分数都急剧下降，说明在没有金标准监督的情况下，要得到干净的音素–单元双射非常困难。值得注意的是，K-means（phone）变体在表 13 中付出了明显的 ABX 代价，表明通过 K-means 强制做双射映射，尽管能改善音素指派指标，却可能损害表征质量。尽管如此，采用基于音子或特征频率微调的 MauBERT 变体仍是最强的系统（表 12：MauBERT-phone + phone freq. 在开发集上达到 PER 、 值 ）；而在真实词表微调条件（表 13）中，基于音子频率的 MPR 在 PER 与 值之间给出了最好的折中。
| dev languages | test languages | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PER | -value | PNMI | ABX c. | PER | -value | PNMI | ABX c. | |||
| HuBERT + K-means (L11) | ||||||||||
| MauBERT-feat | ||||||||||
| + K-means (L9) | ||||||||||
| + K-means (feat) | ||||||||||
| + feat. freq. | ||||||||||
| + phone freq. | ||||||||||
| MauBERT-phone | ||||||||||
| + K-means (proj) | ||||||||||
| + K-means (phone) | ||||||||||
| + phone freq. |
Table 13: DiscoPhon benchmark’s one-to-one scores ( units). Fine-tuned models predict the ground-truth vocabulary size, resulting in a phoneme-unit bijection. The target layer corresponds to the logits from the MPR task (on top of HuBERT’s last layer). K-means clustering: layer for pseudo-labels to be predicted between parentheses. Best scores in bold, second best underlined.
| 开发语言 | 测试语言 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PER | -value | PNMI | ABX c. | PER | -value | PNMI | ABX c. | |||
| HuBERT + K-means (L11) | ||||||||||
| MauBERT-feat | ||||||||||
| + K-means (L9) | ||||||||||
| + K-means (feat) | ||||||||||
| + feat. freq. | ||||||||||
| + phone freq. | ||||||||||
| MauBERT-phone | ||||||||||
| + K-means (proj) | ||||||||||
| + K-means (phone) | ||||||||||
| + phone freq. |
表 13：DiscoPhon 基准的一对一分数（ 个单元）。 微调模型预测真实词表规模，从而形成音素与单元的双射。目标层对应 MPR 任务的 logits（加在 HuBERT 最后一层之上）。K-means 聚类：括号中为待预测伪标签所在的层。最佳分数用粗体，次优分数加下划线。