Librispeech: An ASR corpus based on public domain audio books
LIBRISPEECH: AN ASR CORPUS BASED ON PUBLIC DOMAIN AUDIO BOOKS
∗Center for Language and Speech Processing & Human Language Technology Center of Excellence
The Johns Hopkins University, Baltimore, MD 21218, USA
{vassil.panayotov,dpovey}@gmail.com, {guoguo,khudanpur}@jhu.edu
ABSTRACT
摘要
This paper introduces a new corpus of read English speech, suitable for training and evaluating speech recognition systems. The LibriSpeech corpus is derived from audiobooks that are part of the LibriVox project, and contains 1000 hours of speech sampled at 16 kHz. We have made the corpus freely available for download, along with separately prepared language-model training data and pre-built language models. We show that acoustic models trained on LibriSpeech give lower error rate on the Wall Street Journal (WSJ) test sets than models trained on WSJ itself. We are also releasing Kaldi scripts that make it easy to build these systems.
本文介绍一套新的英语朗读语音语料,适合用来训练和评测语音识别系统。LibriSpeech 语料来自 LibriVox 项目中的有声书,包含 1000 小时、采样率为 16 kHz 的语音。我们已把语料免费开放下载,并一并提供单独准备好的语言模型训练数据以及预先建好的语言模型。我们表明,在 LibriSpeech 上训练的声学模型,在华尔街日报(WSJ)测试集上的错误率低于在 WSJ 本身上训练的模型。我们同时也发布 Kaldi 脚本,方便搭建这些系统。
Index Terms— Speech Recognition, Corpus, LibriVox
索引词— 语音识别,语料,LibriVox
1. INTRODUCTION
1. 引言
The rapid increase in the amount of multimedia content on the Internet in recent years makes it feasible to automatically collect data for the purpose of training statistical models. This is particularly true when the source data is already organized into well curated, machine readable collections. The LibriVox project, a volunteer effort, is currently responsible for the creation of approximately 8000 public domain audio books, the majority of which are in English. Most of the recordings are based on texts from Project Gutenberg, also in the public domain.
近年来互联网上多媒体内容迅速增加,已经使自动收集数据来训练统计模型变得可行。当源数据本身已经被整理成质量较好、机器可读的合集时,尤其如此。志愿者项目 LibriVox 目前已制作大约 8000 本进入公有领域的有声书,其中大多数是英语。这些录音大多依据同样进入公有领域的古腾堡计划 文本。
Although the use of audio books for building synthetic voices [1, 2] has previously been investigated, we are not aware of any freely available read speech corpus in English that is suitable for training and testing speech recognition systems, and which is as large scale as the one we present here. The volunteer-supported speech-gathering effort Voxforge, on which the acoustic models we used for alignment were trained, contains a certain amount of LibriVox audio, but the dataset is much smaller than the one we present here, with around 100 hours of English speech, and suffers from major gender and per-speaker duration imbalances.
虽然用有声书来构建合成语音 [1, 2] 此前已有人研究,但我们并不知道有哪一套可自由获取的英语朗读语料,既适合训练和测试语音识别系统,规模又能与本文给出的这套相当。志愿者支持的语音采集项目 VoxForge(我们用于对齐的声学模型就是在它上面训练的)包含一定数量的 LibriVox 音频,但数据量比这里给出的要小得多,英语语音大约只有 100 小时,并且在性别和每个说话人的时长上存在明显不均。
This paper presents the LibriSpeech corpus, which is a read speech data set based on LibriVox's audio books. The corpus is freely available under the very permissive CC BY 4.0 license [3] and there are example scripts in the open source Kaldi ASR toolkit [4] that demonstrate how high quality acoustic models can be trained on this data.
本文介绍 LibriSpeech 语料,这是一套基于 LibriVox 有声书的朗读语音数据。语料按非常宽松的 CC BY 4.0 许可 [3] 免费提供,开放源代码的 Kaldi 语音识别工具包 [4] 中也有示例脚本,用来说明如何在这套数据上训练高质量的声学模型。
Section 2 presents the long audio alignment procedure that we used in the creation of this corpus. Section 3 describes the structure of the corpus. In Section 4 we describe the process we used to build the language models, which we make available with this corpus. Finally in Section 5 we present experimental results on models trained on this data set, using both the LibriSpeech dev and test sets and Wall Street Journal (WSJ) [5] test sets.
第 2 节介绍我们在创建这套语料时使用的长音频对齐流程。第 3 节描述语料的结构。第 4 节说明我们用来构建语言模型的过程,这些语言模型也随语料一并提供。最后,第 5 节给出在这套数据上训练的模型的实验结果,评测同时使用 LibriSpeech 的开发和测试集,以及华尔街日报(WSJ)[5] 测试集。
2. AUDIO ALIGNMENT
2. 音频对齐
Most acoustic model training procedures expect that the training data come in the form of relatively short utterances, usually up to few tens of seconds in length, each with corresponding text. Therefore we need to align the audio recordings with the corresponding texts, and split them into short segments. We also aim to exclude segments of audio that might not correspond exactly with the aligned text. Our procedure is similar to that described in [6], and consists of two stages. (Note: we have since become aware of a different, phone-based approach [7]).
大多数声学模型训练流程都假定训练数据是较短的语句,长度通常不超过几十秒,并且每句都有对应文本。因此我们需要把音频录音与相应文本对齐,再切成短段。我们也希望排除那些可能与对齐文本并不严格对应的音频片段。我们的流程与 [6] 中描述的相近,分成两个阶段。(注:我们后来才了解到另一种基于音素的做法 [7]。)
2.1. Text preprocessing, lexicon and LM creation
2.1. 文本预处理、词典与语言模型的构建
Each book's text is normalized by converting it into upper-case, removing the punctuation, and expanding common abbreviations and non-standard words [8]. Then the SRILM toolkit [9] is used to train a Witten-Bell [10] smoothed bigram language model on the text of that book. We base our lexicon on CMUdict, from which we remove the numeric stress markers; the pronunciations for out-of-vocabulary (OOV) words are generated with the Sequitur G2P toolkit [11]. In order to avoid possible problems with recognizing excessively long audio recordings, the audio chapters are split into segments of up to 30 minutes in length. The audio is then recognized using the gmm-decode-faster decoder from the Kaldi toolkit, trained on the VoxForge dataset. For this first decoding pass we use a triphone model discriminatively trained with Boosted MMI [12], based on MFCC [13] features processed with frame-splicing over 7 frames, followed by LDA, followed by a global semi-tied covariance (STC) transform [14].
每本书的文本都先做规整:改成大写、去掉标点,并展开常见缩写和不规范词 [8]。随后用 SRILM 工具包 [9] 在该书文本上训练带 Witten-Bell [10] 平滑的二元语言模型。词典以 CMUdict 为底,并去掉其中的数字重音标记;未登录(OOV)词的发音用 Sequitur 字形到音位(G2P)工具包 [11] 生成。为避免识别过长录音可能带来的问题,音频章节被切成最长 30 分钟的片段。随后用在 VoxForge 数据上训练、来自 Kaldi 工具包的 gmm-decode-faster 解码器做识别。第一遍解码使用以 Boosted MMI [12] 判别训练的三音素模型,特征是对 7 帧做拼接处理后的 MFCC [13],再接 LDA,再接全局半绑定协方差(STC)变换 [14]。
2.2. First alignment stage
2.2. 第一阶段对齐
We use the Smith-Waterman alignment algorithm [15] to find the best single region of alignment between the recognized audio and the chapter text. This is like doing Levenshtein alignment [16], except we do not require it to consume the whole reference or hypothesis from the beginning to end, and it also has tunable rather than fixed weights for the different kinds of errors. From this we take the largest single region of similarity (which in most cases would be the entire chapter) and discard the rest, if any. Within that region of similarity, we mark a transcript word as being part of an “island of confidence” if it is part of an exact match with the reference whose length is 12 phones or more. We now split the audio into shorter segments, of 35 seconds or less, using a dynamic programming algorithm. We only allow it to split on silence intervals whose length is at least 0.5 second and which are inside an island of confidence. This allows us to generate, with reasonable confidence, a candidate text for each split piece of audio.
我们用 Smith-Waterman 对齐算法 [15] 在识别出的音频和章节文本之间找出最佳的单一对齐区间。这类似于做 Levenshtein 对齐 [16],不同的是我们并不要求它从开头到结尾用尽整个参考或假设,而且各类错误的权重可以调节,而不是固定不变。由此我们取最大的一段相似区间(多数情况下会是整章),其余若有则丢掉。在该相似区间内,若某个转写词属于与参考完全匹配、且长度不少于 12 个音素的片段,我们就把它标为“信心岛”的一部分。接着我们用动态规划算法把音频切成不超过 35 秒的较短片段,并且只允许在长度至少 0.5 秒、且位于信心岛内部的静音区间处切开。这样就能以比较有把握的方式,为切出的每一段音频生成候选文本。
2.3. Second alignment stage
2.3. 第二阶段对齐
The goal of the second stage is to filter out segments where the candidate text obtained by the first stage has a high likelihood of being inaccurate. Possible sources of text-audio mismatch include inaccuracies in Project Gutenberg texts, reader-introduced insertions, deletions, substitutions and transpositions, and involuntary disfluencies [1, 17]. Other significant sources of mismatch that we noticed are inaccurate text normalization and grapheme-to-phoneme errors in the automatically generated pronunciations.
第二阶段的目标,是滤掉第一阶段得到的候选文本很可能不准的那些片段。文本与音频对不上的可能来源包括:古腾堡计划文本中的不准确之处,朗读者引入的插入、删除、替换和换位,以及不自觉的不流畅 [1, 17]。我们还注意到其他重要的不一致来源:文本规整不准确,以及自动生成发音时的字形到音位错误。
In this second stage of alignment, we use a custom-generated decoding graph for each segment. The decoding graph, diagrammed in Figure 1, is formed from a combination of the linear sequence of words in the transcript with a generic phone-level bigram language model. Our aim is to use the phone-level bigram to allow arbitrary insertions between words in the transcript, or replacement of words in the transcript; we will reject any utterance whose decoding shows any deviation from the transcript. We also experimented with a single-phone filler model to model errors, but found the phone-level bigram was more effective at finding segments with inaccurate transcripts.
在这第二阶段对齐中,我们为每一段单独生成解码图。如图 1 所示,该解码图由转写中的线性词序列与一套通用的音素级二元语言模型组合而成。我们的目的是用音素级二元模型,允许在转写的词与词之间任意插入,或替换转写中的词;一旦某句的解码结果显示出任何偏离转写之处,我们就拒绝该句。我们也试验过用单音素填充模型来刻画错误,但发现音素级二元模型在找出转写不准的片段上更有效。
The most obvious way to generate the decoding graph would be to include multiple copies of the phone-level bigram graph, but this would lead to very large decoding graphs. Instead we use a single copy of the bigram part of the decoding graph (Figure 1), but we modify the decoder so that after entering the bigram part of the model from word-position in the transcript, we may only return at position (corresponding to an insertion between words) or position (corresponding to a substitution or deletion of a word). This is like a pushdown transducer that can only store one item in the pushdown store.
生成这张解码图最直接的办法,是放入多份音素级二元图,但这样会得到非常大的解码图。我们改为只使用一份解码图中的二元部分(图 1),同时修改解码器:从转写中的词位置 进入模型的二元部分之后,只允许在位置 返回(对应词与词之间的插入),或在位置 返回(对应一个词的替换或删除)。这就像一个下推转换器,下推栈里只能存放一项。
图 1. 第二阶段对齐算法所用的示例文法(G)接受器
In this second decoding stage, we use a speaker-adapted model [18, 19] with fMLLR transforms estimated at the speaker level, based on the transcript generated by the first decoding pass.
在这第二阶段解码中,我们使用说话人自适应模型 [18, 19],并在说话人一级估计 fMLLR 变换,依据是第一遍解码生成的转写。
In most cases this algorithm succeeds in detecting text-audio mismatches, especially for native speakers. There are also instances of false rejections. A common problem is, for example, the “assimilation” of a short, 1-2 phone word into a neighboring silence period, which leads to an erroneous detection of deletion from the audio. However, since the original amount of audio in the audiobooks is so large, we can afford to lose a certain percentage of it. Figure 2 shows examples of the kinds of errors that we typically find by applying this method.
在多数情况下,这套算法能够检出文本与音频的不一致,对母语者尤其如此。也会出现误拒。一个常见问题是,例如,把一个只有 1–2 个音素的短词“同化”进邻近的静音段,从而导致错误地判定音频中发生了删除。不过,有声书中的原始音频量很大,我们承受得起丢掉其中一定比例。图 2 给出了用这种方法通常会发现的几类错误的例子。
Fig. 2. Examples of typical text-audio discrepancies detected in second stage decoding. a) Chapter 1 of “Northanger Abbey” by J. Austen, read by Kara Shallenberg; b) Chapter 23 of “Frankenstein” by M. Shelley, read by Hugh McGuire; c) Chapter 15 of “Frankenstein” by M. Shelley, read by Gord Mackenzie
图 2. 第二阶段解码中检出的典型文本–音频不一致示例。a) 奥斯汀《诺桑觉寺》第 1 章,由 Kara Shallenberg 朗读;b) 雪莱《弗兰肯斯坦》第 23 章,由 Hugh McGuire 朗读;c) 雪莱《弗兰肯斯坦》第 15 章,由 Gord Mackenzie 朗读
The whole alignment process took approximately 65 hours on two Amazon EC2 cc2.8xlarge instances, to produce an initial set of aligned audio of size approximately 1200 hours.
整个对齐过程在两台 Amazon EC2 cc2.8xlarge 实例上大约用了 65 小时,得到大约 1200 小时的初始对齐音频。
2.4. Data segmentation
2.4. 数据切分
The second stage of alignment, which we described above, gives us a subset of the audio segments of length up to 35 seconds, that have a good likelihood of having accurate transcripts. Next we break these long segments up into smaller segments. We used two different methods for this. For training data, our rule was to split on any silence interval longer than 0.3 seconds. For test data, we only allowed splits if those intervals coincided with a sentence break in the reference text. The idea was that data split at sentence breaks is likely to be easier to recognize from a language modeling point of view.
上面介绍的第二阶段对齐,为我们提供了一组长度不超过 35 秒、转写很可能准确的音频片段。接下来再把这些较长的片段切得更短。我们为此用了两种不同的方法。对训练数据,规则是在任何长于 0.3 秒的静音区间处切开。对测试数据,只有当这些区间恰好落在参考文本的句子边界上时,才允许切开。这样做的想法是:按句界切开的数据,从语言建模的角度看往往会更容易识别。
3. DATA SELECTION AND CORPUS STRUCTURE
3. 数据选择与语料结构
3.1. Data selection
3.1. 数据选择
To select the audio recordings for inclusion into the corpus we use LibriVox's API to collect information about the readers, the audio book projects in which they participated, and the chapters of books that they read. The URLs for audio files and reference texts were obtained by matching the information from LibriVox's API with the metadata records from the Internet Archive and Project Gutenberg's RDF/XML files. For a small fraction of audiobooks no exact match for the title was found in Project Gutenberg, so to improve coverage we allowed a fuzzy matching of titles.
为了选出收入语料的音频录音,我们使用 LibriVox 的 API 收集关于朗读者、他们参与的有声书项目、以及所读章节的信息。音频文件和参考文本的 URL,是通过把 LibriVox API 中的信息与互联网档案馆 的元数据记录以及古腾堡计划的 RDF/XML 文件 相对齐得到的。对一小部分有声书,古腾堡计划中找不到标题的精确匹配,为提高覆盖率,我们允许对标题做模糊匹配。
In order to guarantee that there was no speaker overlap between the training, development and test sets, we wanted to ensure that each recording is unambiguously attributable to a single speaker. To that end we exclude such LibriVox genres as, for example, “Dramatic Reading”, which include predominantly multi-reader audio chapters. As an extra precaution, in the final post-processing step of the alignment processing the recordings are processed with the LIUM speaker diarization toolkit [20] to automatically detect multi-speaker chapters. A custom GUI application was written, that makes use of the text-audio alignment information and the speaker diarization information, to allow for quick inspection and filtering out of the remaining multi-speaker recordings. This application also made it possible to quickly produce gender information for the speakers and to discard a small number of recordings that had excessive audio quality problems.
为了保证训练集、开发集和测试集之间没有说话人重叠,我们希望确保每条录音都能明确归到一名说话人。为此我们排除了例如 “Dramatic Reading” 这类以多读者章节为主的 LibriVox 类型。作为额外的防范,在对齐流程的最后后处理步骤中,录音还要用 LIUM 说话人分离工具包 [20] 处理,以自动检测多说话人章节。我们编写了一个定制的图形界面程序,利用文本–音频对齐信息和说话人分离信息,便于快速检查并过滤掉仍含多名说话人的录音。该程序也使我们可以迅速补上说话人的性别信息,并丢掉少数音质问题过大的录音。
We ensured a gender balance at the speaker level and in terms of the amount of data available for each gender.
我们在说话人数量上,以及每个性别可用的数据量上,都保证了性别平衡。
3.2. Corpus partitions
3.2. 语料划分
The size of the corpus makes it impractical, or at least inconvenient for some users, to distribute it as a single large archive. Thus the training portion of the corpus is split into three subsets, with approximate size 100, 360 and 500 hours respectively. A simple automatic procedure was used to select the audio in the first two sets to be, on average, of higher recording quality and with accents closer to US English. An acoustic model was trained on WSJ's si-84 data subset and was used to recognize the audio in the corpus, using a bigram LM estimated on the text of the respective books. We computed the Word Error Rate (WER) of this automatic transcript relative to our reference transcripts obtained from the book texts.
语料的规模使得把它作为单一的大压缩包来分发并不现实,至少对部分用户来说不方便。因此语料的训练部分被拆成三个子集,大小大约分别是 100、360 和 500 小时。我们用一种简单的自动流程,使前两个子集中的音频在平均意义上录音质量更高、口音也更接近美国英语。我们在 WSJ 的 si-84 数据子集上训练了一个声学模型,并用在各书文本上估计的二元语言模型去识别语料中的音频。再把这份自动转写相对于我们从书稿得到的参考转写,计算词错误率(WER)。
The speakers in the corpus were ranked according to the WER of the WSJ model's transcripts, and were divided roughly in the middle, with the lower-WER speakers designated as “clean” and the higher-WER speakers designated as “other”. From the “clean” pool, 20 male and 20 female speakers were drawn at random and assigned to a development set. The same was repeated to form a test set. For each dev or test set speaker, approximately eight minutes of speech are used, for total of approximately 5 hours and 20 minutes each. Note that, as mentioned in Section 2.4, we use a different segmentation procedure for development and test data, than for training data.
语料中的说话人按 WSJ 模型转写的词错误率排序,并大致从中间分开:错误率较低的称为 “clean”,较高的称为 “other”。从 “clean” 池中随机抽取 20 名男性和 20 名女性说话人,划入开发集。再用同样办法组成测试集。每名开发集或测试集说话人大约使用 8 分钟语音,每个集合总共大约 5 小时 20 分钟。需要注意的是,如第 2.4 节所述,开发集和测试数据的切分办法与训练数据不同。
The rest of the audio in the “clean” pool was randomly split into two training sets with approximate size 100 and 360 hours respectively. For each speaker in these training sets the amount of speech was limited to 25 minutes, in order to avoid major imbalances in per-speaker audio duration.
“clean” 池中其余的音频被随机拆成两个训练集,大小大约分别是 100 小时和 360 小时。这些训练集中每名说话人的语音量限制在 25 分钟,以避免每个说话人的音频时长出现严重不均。
The “other” pool was similarly split into test and development sets, and a single training set of approximately 500 hours. For this pool, however we did not choose the development and test sets at random; instead we deliberately chose more challenging data. The WER we computed using the WSJ models was used to rank the speakers in order of increasing difficulty, and the speakers for the test and development set were randomly chosen from the third quartile of this sorted list. Table 1 provides a summary of all subsets in the corpus.
“other” 池同样被拆成测试集、开发集,以及一个大约 500 小时的训练集。不过对这个池,我们并没有随机选取开发和测试集,而是有意挑选更有挑战性的数据。我们用 WSJ 模型算出的词错误率,按难度递增给说话人排序,测试集和开发集的说话人再从这份排序表的第三个四分位中随机抽取。表 1 汇总了语料中的全部子集。
| subset | hours | per-spk minutes | female spkrs | male spkrs | total spkrs |
|---|---|---|---|---|---|
| dev-clean | 5.4 | 8 | 20 | 20 | 40 |
| test-clean | 5.4 | 8 | 20 | 20 | 40 |
| dev-other | 5.3 | 10 | 16 | 17 | 33 |
| test-other | 5.1 | 10 | 17 | 16 | 33 |
| train-clean-100 | 100.6 | 25 | 125 | 126 | 251 |
| train-clean-360 | 363.6 | 25 | 439 | 482 | 921 |
| train-other-500 | 496.7 | 30 | 564 | 602 | 1166 |
Table 1. Data subsets in LibriSpeech
| 子集 | 小时数 | 每名说话人分钟数 | 女性说话人 | 男性说话人 | 说话人合计 |
|---|---|---|---|---|---|
| dev-clean | 5.4 | 8 | 20 | 20 | 40 |
| test-clean | 5.4 | 8 | 20 | 20 | 40 |
| dev-other | 5.3 | 10 | 16 | 17 | 33 |
| test-other | 5.1 | 10 | 17 | 16 | 33 |
| train-clean-100 | 100.6 | 25 | 125 | 126 | 251 |
| train-clean-360 | 363.6 | 25 | 439 | 482 | 921 |
| train-other-500 | 496.7 | 30 | 564 | 602 | 1166 |
表 1. LibriSpeech 中的数据子集
4. LANGUAGE MODELS
4. 语言模型
To make it easy to reproduce the results we report here, we have released language model training data and pre-built language models online, along with the text data that we used to build the language models. The language model training material is carefully selected to avoid any overlap with the texts which appear in the test and development sets.
为了便于复现我们在此报告的结果,我们已在网上发布了语言模型的训练数据和预先建好的语言模型,以及我们用来构建这些语言模型的文本数据。语言模型的训练材料经过仔细挑选,以避免与测试集和开发集中出现的文本有任何重叠。
The source material for these language models is Project Gutenberg books. All books, in their entirety, on which the test and development sets are based were filtered out, as well as any book whose title has cosine similarity, over letter 3-grams, greater than 0.7 with any of the titles of these books. After filtering on titles, the text of approximately 22 000 candidate books was downloaded from Project Gutenberg. An inverted index of all 5-grams, with stop words deleted, is built for the books in the test and development sets. All candidate books are then checked against this index and each book for which more than one percent of the 5-grams which appear in it, appear in any of the books in the test and development sets, is removed from the candidate set. The method is effective for finding shared text such as, for example, popular fairy tales which are present in more than one fairy tales collection, long citations of poems in other works, and so on. We used other heuristics to also filter out texts such as numeric tables, sequences from the Human Genome Project, and other types of documents that were deemed inappropriate for language model training.
这些语言模型的源材料是古腾堡计划中的书籍。开发和测试集所依据的那些书,其全文都被过滤掉,标题在字母三元组上与这些书中任一标题的余弦相似度大于 0.7 的书也被去掉。按标题过滤之后,大约 22000 本候选书的文本从古腾堡计划下载下来。我们为测试集和开发集中的书建立了一个去掉停用词后全部五元组的倒排索引。再拿所有候选书对照该索引检查:若一本书中出现的五元组有超过百分之一也出现在测试集或开发集的任一本书里,该书就从候选集中删除。这种方法能有效找出共享文本,例如出现在不止一部童话集中的流行童话、其他作品中对诗的大段引用,等等。我们还用其他启发式办法,过滤掉数字表、人类基因组计划中的序列,以及其他被认为不宜用来训练语言模型的各类文档。
After the above steps, approximately 14 500 public domain books, containing around 803 million tokens in total and 900 000 unique words, remained.
经过上述步骤之后,大约留下 14500 本进入公有领域的书,总共约有 8.03 亿个记号、90 万个不同的词。
To select a lexicon, the words in the corpus were ranked by frequency, and the 200 000 most frequent words were selected. Around one third of these words are present in the CMU pronunciation dictionary, accounting for around 97.5% of all tokens in the evaluation sets; we generated pronunciations for the remaining words using the Sequitur G2P toolkit [11]. Modified Kneser-Ney smoothed 3- and 4-grams [21, 22] are trained. The perplexity for the 3-gram model is 170, and the out of vocabulary token rate is approximately 0.4% on average. For the 4-gram language model the perplexity is around 150.
为了选定词表,语料中的词按频率排序,并取频次最高的 20 万个词。这些词中大约三分之一出现在 CMU 发音词典里,覆盖评测集中大约 97.5% 的记号;其余词的发音我们用 Sequitur 字形到音位工具包 [11] 生成。我们训练了修正 Kneser-Ney 平滑的三元和四元模型 [21, 22]。三元模型的困惑度是 170,未登录记号率平均大约为 0.4%。四元语言模型的困惑度大约是 150。
5. EXPERIMENTS
5. 实验
In this section we present decoding results using models trained using various amounts of LibriSpeech data, and on WSJ data, on both LibriSpeech and WSJ test sets. The recordings available from LibriVox are not completely ideal for training acoustic models for other domains, because the audio is MP3-compressed and because the site's guidelines for upload recommend noise removal and volume normalization. These practices are not consistently enforced, however, so there is a significant fraction of noisy and non-processed audio available, combined with audio that has been subjected to automatic noise removal.
在本节中,我们给出用不同数量的 LibriSpeech 数据以及 WSJ 数据训练的模型,分别在 LibriSpeech 和 WSJ 测试集上的解码结果。LibriVox 上可获得的录音,并不完全适合用来为其他领域训练声学模型,因为音频经过了 MP3 压缩,而且该站点的上传指南建议去除噪声 并做音量归一。不过这些做法执行得并不一律,因此既有相当一部分带噪声、未经处理的音频,也有经过自动去噪的音频。
In order to assess the performance of the acoustic models on non-compressed audio we use the Wall Street Journal read speech corpus [5], as a baseline. We employ language models, trained on the text material the WSJ corpus provides, in conjunction with acoustic models trained on the LibriSpeech data to decode WSJ's test sets, and compare the results with those for state-of-the-art models trained on WSJ's own si-284 set (which contains 82 hours of speech data). The WSJ results we present in Table 2 are for the “open-vocabulary” (60K) test condition, using not the standard 60K word dictionary supplied with WSJ but an extended version that we built to cover more of the words that appear in the WSJ language models. For the language model we used a pruned version of the standard trigram language model that is distributed with the WSJ corpus. The acoustic models, referred to as SAT in the tables, are speaker-adapted GMM models [18, 19], and those referred to as DNN, are based on deep neural networks with p-norm non-linearities [23], trained and tested on top of fMLLR features. The models estimated on LibriSpeech's training data are named after the amount of audio they were built on. The models marked with 460h are trained on the union of the “train-clean-100” and “train-clean-360” subsets, and those marked with 960h are trained on all of LibriSpeech's training sets.
为了评估这些声学模型在未压缩音频上的表现,我们把华尔街日报朗读语音语料 [5] 作为基线。我们使用在 WSJ 语料所提供的文本材料上训练的语言模型,配合在 LibriSpeech 数据上训练的声学模型,去解码 WSJ 的测试集,并与在 WSJ 自己的 si-284 集合(含 82 小时语音数据)上训练的当时一流模型相比较。表 2 中给出的 WSJ 结果对应“开放词表”(60K)测试条件,所用的不是 WSJ 附带的标准 60K 词词典,而是我们为覆盖 WSJ 语言模型中出现的更多词而构建的扩展版本。语言模型方面,我们使用随 WSJ 语料分发的标准三元语言模型的剪枝版本。表中称为 SAT 的声学模型是说话人自适应的高斯混合模型 [18, 19],称为 DNN 的则基于带 p-norm 非线性的深度神经网络 [23],在 fMLLR 特征上训练和测试。在 LibriSpeech 训练数据上估计的模型,按所使用的音频量命名。标注为 460h 的模型是在 “train-clean-100” 与 “train-clean-360” 子集的并集上训练的,标注为 960h 的模型则使用了 LibriSpeech 的全部训练集。
| Acoustic model | eval’92 | dev’93 | eval’93 | |
|---|---|---|---|---|
| LS | SAT 100h | 5.72 | 10.10 | 9.14 |
| LS | SAT 460h | 5.49 | 8.96 | 7.69 |
| LS | SAT 960h | 5.33 | 8.87 | 8.32 |
| LS | DNN 100h | 4.08 | 7.31 | 6.73 |
| LS | DNN 460h | 3.90 | 6.75 | 5.95 |
| LS | DNN 960h | 3.63 | 6.52 | 5.66 |
| WSJ | SAT si-284 | 6.26 | 9.39 | 9.19 |
| WSJ | DNN si-284 | 3.92 | 6.97 | 5.74 |
Table 2. WERs on WSJ's test sets under the “open vocabulary” (60K) test condition
| 声学模型 | eval’92 | dev’93 | eval’93 | |
|---|---|---|---|---|
| LS | SAT 100h | 5.72 | 10.10 | 9.14 |
| LS | SAT 460h | 5.49 | 8.96 | 7.69 |
| LS | SAT 960h | 5.33 | 8.87 | 8.32 |
| LS | DNN 100h | 4.08 | 7.31 | 6.73 |
| LS | DNN 460h | 3.90 | 6.75 | 5.95 |
| LS | DNN 960h | 3.63 | 6.52 | 5.66 |
| WSJ | SAT si-284 | 6.26 | 9.39 | 9.19 |
| WSJ | DNN si-284 | 3.92 | 6.97 | 5.74 |
表 2. 在“开放词表”(60K)测试条件下,WSJ 测试集上的词错误率
Similarly LibriSpeech's language models are used with WSJ acoustic models to decode LibriSpeech's test sets. For these tests the results in Table 3 were obtained by rescoring with the full 4-gram language model from Section 4.
类似地,LibriSpeech 的语言模型也与 WSJ 声学模型一起,用来解码 LibriSpeech 的测试集。对这些测试,表 3 中的结果是用第 4 节中的完整四元语言模型做重打分得到的。
| Acoustic model | dev-clean | test-clean | dev-other | test-other | |
|---|---|---|---|---|---|
| LS | SAT 100h | 8.19 | 9.32 | 29.31 | 31.52 |
| LS | SAT 460h | 7.26 | 8.34 | 26.27 | 28.11 |
| LS | SAT 960h | 7.08 | 8.04 | 21.14 | 22.65 |
| LS | DNN 100h | 5.93 | 6.59 | 20.42 | 22.52 |
| LS | DNN 460h | 5.27 | 5.78 | 17.67 | 19.12 |
| LS | DNN 960h | 4.90 | 5.51 | 12.98 | 13.97 |
| WSJ | SAT si-284 | 10.87 | 12.44 | 39.44 | 41.26 |
| WSJ | DNN si-284 | 7.80 | 8.49 | 27.39 | 30.01 |
Table 3. WERs on LibriSpeech's test sets; all results are obtained by rescoring with a 4-gram language model.
| 声学模型 | dev-clean | test-clean | dev-other | test-other | |
|---|---|---|---|---|---|
| LS | SAT 100h | 8.19 | 9.32 | 29.31 | 31.52 |
| LS | SAT 460h | 7.26 | 8.34 | 26.27 | 28.11 |
| LS | SAT 960h | 7.08 | 8.04 | 21.14 | 22.65 |
| LS | DNN 100h | 5.93 | 6.59 | 20.42 | 22.52 |
| LS | DNN 460h | 5.27 | 5.78 | 17.67 | 19.12 |
| LS | DNN 960h | 4.90 | 5.51 | 12.98 | 13.97 |
| WSJ | SAT si-284 | 10.87 | 12.44 | 39.44 | 41.26 |
| WSJ | DNN si-284 | 7.80 | 8.49 | 27.39 | 30.01 |
表 3. LibriSpeech 测试集上的词错误率;全部结果都是用四元语言模型重打分得到的。
In order to be able to rescore lattices using large language models in a memory efficient manner, we implemented a new rescoring tool, which is now part of the Kaldi toolkit. Table 4 shows the word error rates for language models of different size. The first pass decoding is performed using the 3-gram model pruned with threshold using SRILM's pruning method; the other numbers are obtained through lattice rescoring.
为了能够以节省内存的方式,用大型语言模型对晶格做重打分,我们实现了一个新的重打分工具,该工具现已进入 Kaldi 工具包。表 4 给出了不同规模语言模型的词错误率。第一遍解码使用按 SRILM 剪枝方法、以阈值 剪枝后的三元模型;其余数字通过晶格重打分得到。
| Language model | dev-clean | test-clean | dev-other | test-other |
|---|---|---|---|---|
| 3-gram prn. thresh. 3e-7 | 7.54 | 8.02 | 18.51 | 19.41 |
| 3-gram prn. thresh. 1e-7 | 6.57 | 7.21 | 16.72 | 17.66 |
| 3-gram full | 5.14 | 5.74 | 13.89 | 14.77 |
| 4-gram full | 4.90 | 5.51 | 12.98 | 13.97 |
Table 4. LM rescoring results for the 960 hour DNN model
| 语言模型 | dev-clean | test-clean | dev-other | test-other |
|---|---|---|---|---|
| 3-gram prn. thresh. 3e-7 | 7.54 | 8.02 | 18.51 | 19.41 |
| 3-gram prn. thresh. 1e-7 | 6.57 | 7.21 | 16.72 | 17.66 |
| 3-gram full | 5.14 | 5.74 | 13.89 | 14.77 |
| 4-gram full | 4.90 | 5.51 | 12.98 | 13.97 |
表 4. 960 小时 DNN 模型的语言模型重打分结果
6. CONCLUSIONS
6. 结论
We have automatically aligned and segmented English read speech from audiobooks with the corresponding book text, and filtered out segments with noisy transcripts, in order to produce a corpus of English read speech suitable for training speech recognition systems. We have demonstrated that models trained with our corpus do better on the standard Wall Street Journal (WSJ) test sets than models built on WSJ itself – the large size of our corpus (1000 hours, versus the 82 hours of WSJ's si-284 data) outweighs the audio mismatch. We are releasing this corpus online and have introduced scripts into the Kaldi speech recognition toolkit so that others can easily replicate these results.
我们把有声书中的英语朗读语音与相应的书稿文本自动对齐并切分,同时滤掉转写嘈杂不准的片段,从而得到一套适合训练语音识别系统的英语朗读语料。我们表明,用这套语料训练的模型,在标准的华尔街日报(WSJ)测试集上优于在 WSJ 本身上构建的模型——语料的大体量(1000 小时,相对 WSJ 的 si-284 数据只有 82 小时)压过了音频上的差异。我们正在把这套语料放到网上发布,并已在 Kaldi 语音识别工具包中加入脚本,方便他人复现这些结果。
7. REFERENCES
7. 参考文献
[1] K. Prahallad, Automatic building of synthetic voices from audio books, Ph.D. thesis, CMU, Pittsburgh, 2010.
[2] S. King and V. Karaiskos, “The Blizzard Challenge 2012,” in Proceedings Blizzard Workshop, 2012.
[3] “Creative Commons Attribution 4.0 International Public License,” https://creativecommons.org/licenses/by/4.0/, November 2013.
[4] D. Povey, A. Ghoshal, et al., “The Kaldi Speech Recognition Toolkit,” in Proc. ASRU, 2011.
[5] D. B. Paul and J. M. Baker, “The design for the Wall Street Journal-based CSR corpus,” in Proceedings of the workshop on Speech and Natural Language. Association for Computational Linguistics, 1992, pp. 357–362.
[6] T. J. Hazen, “Automatic alignment and error correction of human generated transcripts for long speech recordings,” in In Proc. Interspeech, 2006.
[7] X. Anguera, J. Luque, and C. Gracia, “Audio-to-text alignment for speech recognition with very limited resources,” in Interspeech, 2014.
[8] R. Sproat et al., “Normalization of non-standard words,” Computer Speech & Language, vol. 15, no. 3, pp. 287–333, 2001.
[9] A. Stolcke, “SRILM - An Extensible Language Modeling Toolkit,” in ICSLP, 2002.
[10] I.H. Witten and T.C. Bell, “The zero-frequency problem: Estimating the probabilities of novel events in adaptive text compression,” IEEE Transactions on Information Theory, vol. 37, no. 4, 1991.
[11] M. Bisani and H. Ney, “Joint-sequence models for grapheme-to-phoneme conversion.,” Speech Communication, vol. 50, no. 5, pp. 434–451, 2008.
[12] D. Povey and D. Kanevsky and B. Kingsbury and B. Ramabhadran and G. Saon and K. Visweswariah, “Boosted MMI for Feature and Model Space Discriminative Training,” in ICASSP, 2008.
[13] S. Davis and P. Mermelstein, “Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences,” Acoustics, Speech and Signal Processing, IEEE Transactions on, vol. 28, no. 4, pp. 357–366, 1980.
[14] M.J.F. Gales, “Semi-tied covariance matrices for hidden markov models,” IEEE Transactions on Speech and Audio Processing, vol. 7, pp. 272–281, 1999.
[15] T. Smith and M. Waterman, “Identification of common molecular subsequences,” Journal of Molecular Biology, vol. 147, no. 1, pp. 195–197, 1981.
[16] V.I. Levenshtein, “Binary Codes Capable of Correcting Deletions, Insertions and Reversals,” Soviet Physics Doklady, vol. 10, pp. 707, 1966.
[17] N. Braunschweiler, M. J. F. Gales, and S. Buchholz, “Lightly supervised recognition for automatic alignment of large coherent speech recordings.,” in INTERSPEECH. 2010, pp. 2222–2225, ISCA.
[18] M. J. F. Gales and P. C. Woodland, “Mean and Variance Adaptation Within the MLLR Framework,” Computer Speech and Language, vol. 10, pp. 249–264, 1996.
[19] T. Anastasakos, J. McDonough, R. Schwartz, and J. Makhoul, “A Compact Model for Speaker-Adaptive Training,” in ICSLP, 1996.
[20] S. Meignier and T. Merlin, “LIUM SpkDiarization: an open source toolkit for diarization,” in CMU SPUD Workshop, Dallas (Texas, USA), March 2010.
[21] R. Kneser and H. Ney, “Improved backing-off for m-gram language modeling,” in ICASSP, 1995, vol. 1, pp. 181–184.
[22] S. F. Chen and J. Goodman, “An empirical study of smoothing techniques for language modeling,” in Proceedings of the 34th annual meeting on Association for Computational Linguistics. Association for Computational Linguistics, 1996, pp. 310–318.
[23] X. Zhang, J. Trmal, D. Povey, and S. Khudanpur, “Improving deep neural network acoustic models using generalized maxout networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2014, Florence, Italy, May 4-9, 2014, 2014, pp. 215–219.

