Audio Set: An ontology and human-labeled dataset for audio events
AUDIO SET: AN ONTOLOGY AND HUMAN-LABELED DATASET FOR AUDIO EVENTS
Google, Inc., Mountain View, CA, and New York, NY, USA
{jgemmeke,dpwe,freedmand,arenjansen,wadelawrence,channingmoore,plakal,marvinritter}@google.com
ABSTRACT
摘要
Audio event recognition, the human-like ability to identify and relate sounds from audio, is a nascent problem in machine perception. Comparable problems such as object detection in images have reaped enormous benefits from comprehensive datasets – principally ImageNet. This paper describes the creation of Audio Set, a large-scale dataset of manually-annotated audio events that endeavors to bridge the gap in data availability between image and audio research. Using a carefully structured hierarchical ontology of 632 audio classes guided by the literature and manual curation, we collect data from human labelers to probe the presence of specific audio classes in 10 second segments of YouTube videos. Segments are proposed for labeling using searches based on metadata, context (e.g., links), and content analysis. The result is a dataset of unprecedented breadth and size that will, we hope, substantially stimulate the development of high-performance audio event recognizers.
音频事件识别,也就是像人一样从声音里辨认并关联各种声响,在机器感知里还是一个刚刚起步的问题。与之相近的问题,例如图像中的物体检测,已经从全面的数据集——主要是 ImageNet——中获得了巨大收益。本文介绍 Audio Set 的构建:这是一个大规模的人工标注音频事件数据集,旨在缩小图像研究与音频研究在数据可得性上的差距。我们在文献和人工整理的指导下,建立了一个包含 632 个音频类别、结构严谨的层次本体,并请人工标注员判断这些类别是否出现在 YouTube 视频的 10 秒片段中。待标注片段通过基于元数据、上下文(例如链接)和内容分析的检索来提名。我们希望,由此得到的这份广度与规模都前所未有的数据集,能够有力地推动高性能音频事件识别器的发展。
Index Terms— Audio event detection, sound ontology, audio databases, data collection
索引词— 音频事件检测,声音本体,音频数据库,数据收集
1. INTRODUCTION
1. 引言
Within the domain of machine perception, we are interested in artificial sound understanding. We would like to produce an audio event recognizer that can label hundreds or thousands of different sound events in real-world recordings with a time resolution better than one second – just as human listeners can recognize and relate the sounds they hear.
在机器感知这个领域里,我们关心的是人工的声音理解。我们希望做出一个音频事件识别器,能够在真实世界的录音中为成百上千种不同的声音事件打上标签,时间分辨率好于一秒——就像人类听者能够认出并关联自己听到的声音一样。
Recently, there have been astonishing results in the analogous problem of image recognition [1, 2, 3]. These make use of the ImageNet dataset, which provides more than 1 million images labeled with 1000 object categories [4]. ImageNet appears to have been a major factor driving these developments, yet nothing of this scale exists for sound sources.
近年来,在与之类似的图像识别问题上已经出现了令人惊讶的结果 [1, 2, 3]。这些工作利用了 ImageNet 数据集,其中提供了超过 100 万张带有 1000 个物体类别标签的图像 [4]。ImageNet 看来是推动这些进展的一个主要因素,然而在声源这一侧,还不存在同等规模的资源。
This paper describes the creation of Audio Set, a dataset and ontology of audio events that endeavors to provide comprehensive coverage of real-world sounds at ImageNet-like scale. Our intention is to use the dataset to create automatic audio event recognition systems that are comparable to the state-of-the-art in recognizing objects in real-world images.
本文介绍 Audio Set 的创建。它是一套音频事件的数据集与本体,试图以接近 ImageNet 的尺度,对真实世界的声音给出尽量全面的覆盖。我们的意图是用这份数据去训练自动音频事件识别系统,使其能够与真实图像中物体识别的当时最好水平相比。
Audio event recognition has been studied from perceptual and engineering perspectives. Warren & Verbrugge [5] were among the first to connect perceptual properties to acoustic features in their study of bouncing and breaking sounds. Ballas [6] probed the perception of 41 short events obtained from a sound effects collection by relating identification time to acoustic features, measures of familiarity, and environmental prevalence. Gygi, Kidd & Watson [7] used multidimensional scaling to identify acoustic factors such as temporal envelope and pitch measures that predicted similarity ratings among 50 environmental sounds. LeMaitre and Heller [8] proposed a taxonomy of sound events distinguishing objects and actions, and used identification time and priming effects to show that listeners find a “middle range” of abstraction most natural.
音频事件识别一直从知觉和工程两条线索被研究。Warren 与 Verbrugge [5] 在对弹跳和破裂声音的研究中,较早地把知觉属性与声学特征联系起来。Ballas [6] 从音效库中取出 41 个短事件,把识别时间与声学特征、熟悉度以及环境中的常见程度联系起来,以此探查知觉。Gygi、Kidd 与 Watson [7] 用多维尺度法找出能够预测 50 种环境声相似性评分的声学因素,例如时间包络和音高度量。LeMaitre 与 Heller [8] 提出了一个区分物体与动作的声音事件分类体系,并利用识别时间和启动效应表明,听者觉得处于“中间层次”的抽象最为自然。
Engineering-oriented taxonomies and datasets began with Gaver [9] who used perceptual factors to guide the design of synthetic sound effects conveying different actions and materials (tapping, scraping, etc.). Nakatani & Okuno [10] devised a sound ontology to support real-world computational auditory scene analysis. Burger et al. [11] developed a set of 42 “noisemes” (by analogy with phonemes) to provide a practical basis for fine-grained manual annotation of 5.6 hours of web video soundtrack. Sharing many of the motivations of this paper, Salamon et al. [12] released a dataset of 18.5 hours of urban sound recordings selected from freesound.org, labeled at fine temporal resolution with 10 low-level sound categories chosen from their urban sound taxonomy of 101 categories. Most recently, Säger et al. [13] systematically constructed adjective-noun and verb-noun pairs from tags applied to entire freesound.org recordings to construct AudioSentiBank, 1,267 hours of audio labeled with 1,123 adjective-noun or verb-noun “sentiment” tags. The labels’ time resolution is limited whole clips, which can be up to 15 min long, and there is no guarantee that the word-pairs actually belong together – e.g., “talking bird” yields mostly tracks containing both “talking” and “bird”, but very few birds doing the talking.
面向工程的分类体系与数据集,是从 Gaver [9] 开始的:他用知觉因素来指导合成音效的设计,以传达不同的动作和材料(敲击、刮擦等)。Nakatani 与 Okuno [10] 设计了一个声音本体,用来支持真实世界中的计算听觉场景分析。Burger 等人 [11] 提出了 42 个“noiseme”(类比于音素),为 5.6 小时网络视频声轨的细粒度人工标注提供了可操作的基础。与本文动机相近,Salamon 等人 [12] 从 freesound.org 选出 18.5 小时城市声音录音并予以发布,按精细的时间分辨率,用他们 101 类城市声音分类中选出的 10 个低层类别来标注。最近,Säger 等人 [13] 从整段 freesound.org 录音的标签出发,系统构造形容词–名词和动词–名词对,由此建成 AudioSentiBank:1,267 小时音频,带有 1,123 个形容词–名词或动词–名词的“情感”标签。这些标签的时间分辨率只到整段录音,长度最长可达 15 分钟,而且并不能保证词对真的是一起出现的——例如,“talking bird”大多会给出同时包含“talking”和“bird”的音轨,但几乎没有正在说话的鸟。
Automatic systems for audio event classification go back to the Muscle Fish content-based sound effects retrieval system [14]. EU Project CHIL conducted an Acoustic Event Detection evaluation [15] over examples of 16 “meeting room acoustic events” comparing results of three systems. A similar task was included in the IEEE-sponsored DCASE 2013 [16] which attracted 7 submissions to detect 16 “office” audio events. F-measures (harmonic mean of precision and recall of detected events) were below 0.2, leaving much room to improve. DCASE 2016 [17], includes an audio event detection task based on 90 minutes of real home and outdoor recordings, carefully annotated with 13 audio event categories spanning “bird singing” to “washing dishes”.
音频事件分类的自动系统,可以追溯到 Muscle Fish 基于内容的音效检索系统 [14]。欧盟项目 CHIL 做过一次声学事件检测评测 [15],在 16 类“会议室声学事件”的例子上比较了三个系统的结果。IEEE 赞助的 DCASE 2013 [16] 也包括类似任务,收到 7 份提交,用于检测 16 类“办公室”音频事件。F 值(所检测事件的精确率与召回率的调和平均)低于 0.2,改进空间还很大。DCASE 2016 [17] 包含一项音频事件检测任务,基于 90 分钟真实的家庭和户外录音,仔细标注了 13 类音频事件,范围从“鸟鸣”到“洗碗”。
Unlike previous work, Audio Set considers all sound events rather than a limited domain. We believe that a large-scale task, in terms of both categories and data, will enable more powerful learning techniques, and hence a step-change in system quality.
与以往工作不同,Audio Set 考虑的是全部声音事件,而不是某个受限领域。我们相信,在类别和数据两方面都做成大规模任务,将使更有力的学习技术成为可能,从而带来系统质量的阶跃式变化。
2. THE AUDIO SET ONTOLOGY
2. AUDIO SET 本体
Given the goal of creating a general-purpose audio event recognizer, we need to define the set of events the system should recognize. This set of classes will allow us to collect labeled data for training and evaluation. In turn, human-labeled data provides a crisp definition of our research goal: to create a system able to predict the human labelings from audio alone.
既然目标是做一个通用的音频事件识别器,我们就需要先定义系统应当识别的事件集合。这组类别将使我们能够收集用于训练和评价的标注数据。反过来,人工标注数据也给研究目标提供了一个清晰的定义:做出一个只凭音频就能预测人工标注的系统。
Rather than a flat list of audio events, we want to have events structured into an abstraction hierarchy. In training, this indicates classes with nonexclusive relationships; for instance we do not want to our classifiers to attempt to separate “dog sounds” from “bark”. During recognition, hierarchical relations allow backing-off to more general descriptions when the classifier encounters ambiguity among several subcategories (e.g., a sound that is recognized ambiguously as “growl”, “bark” and “howl” can fall back to a classification of “dog sounds”). Finally, a well-structured hierarchy can aid human labeling by allowing a labeler to quickly and directly find the set of terms that best describe a sound; this was also important during the development of the event set when trying to add categories without overlap and duplication.
我们不想只要一张扁平的音频事件清单,而希望把事件组织成抽象层次。在训练中,这标示出具有非互斥关系的类别;例如,我们不希望分类器试图把“狗的声音”和“吠叫”分开。在识别时,如果分类器在几个子类之间遇到含混(例如一段声音被含混地识别为“低吼”“吠叫”和“嚎叫”),层次关系允许回退到更一般的描述,如“狗的声音”。最后,结构良好的层次也能帮助人工标注:标注员可以迅速、直接地找到最能描述该声音的一组词语;在扩展事件集合、尽量避免重叠和重复时,这一点同样重要。
This structured set of audio event categories is called the Audio Set Ontology. The specific principles that guided its development were:
这套结构化的音频事件类别称为 Audio Set 本体。指导其发展的具体原则是:
• The categories should provide a comprehensive set that can be used to describe the audio events encountered in real-world recordings.
• 这些类别应当构成一个全面的集合,能够用来描述在真实世界录音中遇到的音频事件。
• The category of a particular audio event should correspond to the idea or understanding that immediately comes to the mind of a listener hearing the sound.
• 某个音频事件所属的类别,应当对应于听者听到该声音时立刻想到的概念或理解。
• Individual categories should be distinguishable by a “typical” listener. That is, if two categories correspond to sounds that a listener cannot easily or reliably distinguish, the categories should be merged. Sounds that can be distinguished only by an expert (such as particular bird species, or fine distinctions in musical instruments) should not be separated. This is a natural condition to prevent the set becoming unwieldy, although we recognize the possibility of extensions that expand certain nodes with more detailed, expert, distinctions.
• 各个类别应当能被“普通”听者区分。也就是说,如果两个类别对应的声音听者不容易或不可靠地区分开,就应当合并。只有专家才能区分的声音(例如特定鸟种,或乐器上的细微差别)不应被拆开。这是防止集合变得臃肿的自然条件,不过我们也承认,将来可以用更细的、面向专家的区分去扩展某些节点。
• Ideally, individual categories are distinct based on their sound alone, i.e. without relying on accompanying visual information or details of context. Thus, a sound is a “thump”, rather than “bare foot stamping on a wooden floor”.
• 理想情况下,各个类别只凭声音本身就能区分,也就是不依赖伴随的视觉信息或细节情境。因此,一个声音是“闷响”,而不是“赤脚踩在木地板上”。
• The hierarchical structure allows annotators to identify the best, most specific categories for given audio events as easily as possible. This means that the hierarchy should not be too deep, while the number of children in any node should rarely be more than 10, to facilitate rapid scanning.
• 层次结构应让标注员尽可能容易地为给定音频事件找出最好、最具体的类别。这意味着层次不宜过深,而任一节点的子类数目应很少超过 10 个,以便快速扫读。
To avoid biasing the category set by the orientation of a particular researcher, or the limited diversity of sounds in a particular dataset or task, we sought to start from a neutral, large-scale analysis of web text. We seeded our audio event lexicon using a modified form of the “Hearst patterns” [18] to identify hyponyms of “sound”, i.e. terms that frequently occur in constructions of the form “... sounds, such as X and Y ...” or “X, Y, and other sounds ...”, etc. Applying these rules over web-scale text yields a very large set of terms, which are then sorted in terms of how well they represent sounds – calculated as a combination of overall frequency of occurrence with how exclusively they are identified as hyponyms of “sound” rather than other terms. This gave us a starting list of over 3000 terms; as part of the matching process, they were also resolved against Freebase/Knowledge Graph entity/machine IDs (MIDs) [19], which we use as stable identifiers.
为了避免类别集合被某一研究者的倾向,或被某一数据集、某一任务中有限的声音多样性所带偏,我们希望从一个中性的、大规模的网页文本分析出发。我们用一种改写过的“Hearst 模式” [18] 来播种音频事件词表,以找出 “sound” 的下位词,也就是经常出现在 “... sounds, such as X and Y ...” 或 “X, Y, and other sounds ...” 这类结构中的词语。把这些规则用在网络尺度的文本上,会得到一个非常大的词集合,再按它们代表声音的程度排序——该程度由总体出现频率,以及它们在多大程度上被专属地识别为 “sound” 而不是其他词的下位词,二者结合起来计算。由此得到一份超过 3000 个词的起始清单;在匹配过程中,它们还被对齐到 Freebase / Knowledge Graph 的实体/机器编号(MID)[19],我们把这些编号当作稳定的标识符。
Starting from the top of the sorted list, we manually assembled a hierarchy to contain these terms in a way that best agreed with our intuitive understanding of the sounds. This step is subjective, but it best captures the role of the hierarchy in supporting human labelers. We stopped when the list seemed to be supplying terms that were all obscure or ill-defined (e.g., “Wilhelm scream”, “The Oak Ridge Boys”, “Earcon”, “Whump”). The resulting structure is not a strict hierarchy as nodes may occur in several places; for instance, “Hiss” appears under “Cat”, “Steam”, and “Onomatopoeia”. There are in total 33 categories that appear more than once.
我们从排序表的顶端开始,手工搭出一个层次,把这些词以最符合我们对声音直觉理解的方式放进去。这一步是主观的,但它最能体现层次在支持人工标注员方面的作用。当我们觉得清单开始提供的都是冷僻或定义不清的词时(例如 “Wilhelm scream”、“The Oak Ridge Boys”、“Earcon”、“Whump”),便停了下来。得到的结构并不是严格的层次,因为节点可能出现在多处;例如,“Hiss” 出现在 “Cat”、“Steam” 和 “Onomatopoeia” 之下。一共有 33 个类别出现不止一次。
We refined this initial structure by comparing it with existing audio event lists or taxonomies, including [9, 20, 11, 12, 21, 8, 17]. Our hope was that the initial list would subsume these earlier efforts; in fact, numerous gaps were exposed, although eventually we reached a point where nearly every class from other sets was covered. Some classes from other sets were not incorporated because they were too specific, or otherwise did not meet our criteria of being readily identifiable. For instance, the urban sounds taxonomy of [12] includes “Car radio” as a source of recorded music, which makes sense when labeling city street ambience recordings but is too specialized or context-dependent for our set (although we do have “Radio” for sounds that are clearly being produced by a radio). The 165 verified-distinct sound clips used in [21] include detailed examples such “Trumpet jazz solo” and “Walking on leaves”; in our scheme, these would be simply “Trumpet” and “Walk, footsteps”, respectively.
我们随后把这一初始结构与已有的音频事件清单或分类体系作比较,包括 [9, 20, 11, 12, 21, 8, 17],并加以改进。我们原本希望初始清单能够覆盖这些较早的工作;事实上暴露出许多缺口,不过最终几乎覆盖了其他集合中的每一类。其他集合中有些类别没有被纳入,因为它们过于具体,或在其他方面不满足我们“容易辨认”的标准。例如,[12] 的城市声音分类把 “Car radio” 当作录制音乐的一个来源,这在标注城市街道环境录音时说得通,但对我们的集合来说过于专门、过于依赖情境(尽管我们确实有 “Radio”,用于明显由收音机发出的声音)。[21] 中使用的 165 段经过核实、彼此可分的声音片段,包含诸如 “Trumpet jazz solo” 和 “Walking on leaves” 这样的细例;在我们的方案里,它们会分别只是 “Trumpet” 和 “Walk, footsteps”。
We then began using this set to label recordings, as well as trying to find examples for every category (as described in section 2.1 below). Feedback from the larger group of people involved led to further modifications to the category set.
我们随后开始用这套类别去标注录音,并尝试为每个类别找到例子(如下文 2.1 节所述)。参与其中的更大一批人给出的反馈,又带来了对类别集合的进一步修改。
Our final list contains 632 audio event categories, arranged in a hierarchy with a maximum depth of 6 levels; an example from among the eight level-6 nodes is “Sounds of things” → “Vehicle” → “Motor vehicle” → “Emergency vehicle” → “Siren” → “Ambulance (siren)” (a category that was appropriated from the Urban Sound taxonomy [12]). Figure 1 shows the 50 first- and second-level nodes in this hierarchy.
我们的最终清单包含 632 个音频事件类别,按最多 6 层的层次排列;在 8 个第 6 层节点中,有一个例子是 “Sounds of things” → “Vehicle” → “Motor vehicle” → “Emergency vehicle” → “Siren” → “Ambulance (siren)”(这一类别借自 Urban Sound 分类 [12])。图 1 给出了这一层次中的 50 个第一层和第二层节点。
图 1:Audio Set 本体的上两层。
2.1. Ontology Data Release
2.1. 本体数据发布
The ontology is released as a JSON file containing the following fields for each category:
本体以 JSON 文件的形式发布,每个类别包含下列字段:
g.co/audioset
• ID: The Knowledge Graph Machine ID (MID) best matching the sound or source, used as the primary identifier for the class. In Knowledge Graph, many MIDs refer to specific objects (e.g., /m/0gy1t2s, “Bicycle bell” or /m/02y 763, “Sliding door”); when used in the ontology, they are understood to mean “the sound readily associated with the specified object”. For instance, in the case of “Sliding door”, the sound made by someone crashing into a closed door would be emanating from the door; however, it would not be the characterisic short burst of bearing noise that suggests “Sliding door”, so it should instead be labeled “Smash” or “Thump”.
• ID:与该声音或声源最匹配的 Knowledge Graph 机器编号(MID),用作该类的主标识符。在 Knowledge Graph 中,许多 MID 指的是具体物体(例如 /m/0gy1t2s,“Bicycle bell”,或 /m/02y 763,“Sliding door”);用在本体里时,它们被理解为“与所指物体容易联系在一起的那种声音”。例如,就 “Sliding door” 而言,有人撞上一扇关着的门所发出的声音确实来自这扇门;但它并不是令人联想到 “Sliding door” 的那种短促、典型的轴承噪声,因此应改标为 “Smash” 或 “Thump”。
• Display name: A brief one or two word name for the sound, sometimes with a small number of comma-separated alternatives (e.g., “Burst, pop”), and sometimes with parenthesized disambiguation (e.g. “Fill (with liquid)”). Display names are intended to be unambiguous even when shown without their ancestor nodes.
• Display name:该声音的简短名称,通常一两个词,有时带少量逗号分隔的别名(例如 “Burst, pop”),有时带括号里的消歧说明(例如 “Fill (with liquid)”)。显示名即便在不展示祖先节点时,也应当是无歧义的。
• Description: A longer description, typically one or two sentences, used to provide more explanation of the meaning and limits of the class. In many cases these are based on Wikipedia or WordNet descriptions (with appropriate citation URLs), adapted to emphasize their specific use for audio events.
• Description:一段较长的说明,通常一两句话,用来进一步解释该类的含义和边界。许多情况下,它们基于 Wikipedia 或 WordNet 的描述(并附上相应的引用 URL),再改写以强调其在音频事件上的具体用法。
• Examples: As an alternative to the textual descriptions, we also collected at least one example of the sound (excepting “Abstract” classes, described below). At present, all examples are provided as URLs indicating short excerpts from public YouTube videos. The exercise of finding concrete examples for each class was helpful in flushing out indistinct or ambiguous categories from earlier versions of the ontology.
• Examples:作为文本说明之外的补充,我们还为每种声音收集了至少一个例子(下文所述的 “Abstract” 类除外)。目前,所有例子都以 URL 的形式给出,指向公开 YouTube 视频中的短片段。为每个类寻找具体例子,也有助于把本体较早版本中含混或有歧义的类别清理出去。
• Children: The hierarchy is encoded by including within each node the MIDs of all the immediate children of that category.
• Children:层次关系的编码方式,是在每个节点中列入该类别全部直接子节点的 MID。
• Restrictions: Of the 632 categories, 56 are “blacklisted”, meaning they are not exposed to labelers because they have turned out to be obscure (e.g., “Alto saxophone”) or confusing (e.g., “Sounds of things”). Another 22 nodes are marked “Abstract” (e.g., “Onomatopoeia”), meaning that they exist purely as intermediate nodes to help structure the ontology, and are not expected to ever be used directly as labels. These flags appear in the Restrictions field.
• Restrictions:在 632 个类别中,有 56 个被标为 “blacklisted”(黑名单),也就是不展示给标注员,因为它们后来被发现过于冷僻(例如 “Alto saxophone”)或容易造成混淆(例如 “Sounds of things”)。另有 22 个节点被标为 “Abstract”(例如 “Onomatopoeia”),意思是它们只作为中间节点来帮助组织本体,并不预期被直接当作标签使用。这些标记出现在 Restrictions 字段中。
Figure 2 shows the complete record for a single example category, “Bird vocalization, bird call, bird song”.
图 2 给出了一个示例类别 “Bird vocalization, bird call, bird song” 的完整记录。
图 2:Audio Set 本体中一个声音类别的完整数据记录。子节点分别对应 “Tweet”、“Chirp” 和 “Squawk”。
3. AUDIO SET DATASET
3. AUDIO SET 数据集
The Audio Set YouTube Corpus consists of labeled YouTube segments, structured as a CSV file comprising YouTube identifiers, start time, end time and one or more labels. Dataset segments are all 10 seconds long (except when that exceeds the length of the underlying video). Each dataset segment carries one or more ontology class labels.
Audio Set YouTube Corpus 由带标注的 YouTube 片段组成,以 CSV 文件 的形式组织,其中包括 YouTube 标识符、起始时间、结束时间以及一个或多个标签。数据集中的片段都是 10 秒长(若超过底层视频长度则除外)。每条片段带有一个或多个本体类别标签。
g.co/audioset
3.1. Human rating
3.1. 人工评分
Human raters were presented with a 10-second segments including both the video and audio components, but did not have access to the title or other meta-information of the underlying YouTube video. If the duration of the video was less than 10 seconds, the entire video was used for rating. We experimented both with shorter segments and audio-only presentation, but raters found these conditions far more difficult, possibly due to the fine-grained nature of our audio event ontology.
人工评分员看到的是包含视频和音频两部分的 10 秒片段,但不能看到底层 YouTube 视频的标题或其他元信息。如果视频时长不足 10 秒,就用整段视频来评分。我们也试验过更短的片段,以及只给音频的呈现方式,但评分员觉得这些条件要困难得多,这可能是因为我们的音频事件本体粒度很细。
For each segment, raters were asked to independently rate the presence of one or more labels. The possible ratings were “present”, “not present” and “unsure”. Each segment was rated by three raters and a majority vote was required to record an overall rating. For speed, a segment’s third rating was not collected if the first two ratings agreed for all labels.
对于每个片段,评分员被要求独立判断一个或多个标签是否存在。可选的评分是 “present”(存在)、“not present”(不存在)和 “unsure”(不确定)。每个片段由三名评分员评分,需要多数票才能记下一个总体评分。为了加快进度,如果前两次评分在全部标签上都已一致,就不再收集该片段的第三次评分。
The raters were unanimous in 76.2% of votes. The “unsure” rating was rare, representing only 0.5% of responses, so 2:1 majority votes account for 23.6% of the decisions. Categories that achieved the highest rater agreement include “Christmas music”, “Accordion” and “Babbling” (> 0.92); while some categories with low agreement include “Basketball bounce”, “Boiling” and “Bicycle” (< 0.17).
评分员在 76.2% 的投票中意见一致。“unsure” 很少见,只占全部回答的 0.5%,因此 2:1 的多数票占决策的 23.6%。评分员一致性最高的类别包括 “Christmas music”、“Accordion” 和 “Babbling”(> 0.92);一致性较低的类别则包括 “Basketball bounce”、“Boiling” 和 “Bicycle”(< 0.17)。
Spot checking revealed a small number of labeling errors which were mostly attributed to: 1) confusing labels, 2) human error, and 3) difference in detection of faint/non-salient audio events. As an additional check we analyzed correlations between “present” labels and words in the video’s metadata. This exposed some commonly-misinterpreted labels, which were then removed from the ontology. Due to the scale of the data and since majority agreement was very high, no other other corrective actions were taken.
抽查发现了少量标注错误,主要归因为:1)容易混淆的标签,2)人为失误,以及 3)对微弱或不显著音频事件的检测差异。作为额外检查,我们分析了 “present” 标签与视频元数据中词语之间的相关。这暴露出一些经常被误解的标签,随后便从本体中删除。由于数据规模很大,而且多数一致率已经很高,没有再采取其他纠正措施。
3.2. Selecting segments for rating
3.2. 选择送去评分的片段
In order to obtain a sufficient number of positive ratings from a moderate volume of labeling effort, it is important to send for rating only those that have a good chance of containing particular audio events.
为了在中等规模的标注工作量下得到足够多的正例评分,重要的是只把那些较有可能包含特定音频事件的片段送去评分。
We used a variety of methods to identify videos that were likely to be relevant to particular labels: About half of the audio events corresponded to labels already predicted by an internal video-level automatic annotation system, and we used the videos bearing the label to provide segments for rating. The labels from the internal video labeling system are automatically generated and thus inevitably imperfect. They are, however, based on multiple complementary sources of information, such as metadata, anchor text of incoming links, comments, and user engagement signals, and their reliability is greatest for popular videos which have more of these attributes. We therefore select only videos with at least 1,000 views.
我们用多种方法来找出可能与特定标签相关的视频:大约一半的音频事件,对应于内部视频级自动标注系统已经预测出的标签,我们便用带有该标签的视频来提供待评分片段。内部视频标注系统给出的标签是自动生成的,因而难免不完美。不过,它们基于多种互补的信息来源,例如元数据、入链锚文本、评论以及用户参与信号,并且在拥有更多这类属性的热门视频上最为可靠。因此我们只选择播放量至少为 1,000 的视频。
In addition, videos were selected by searching titles and other metadata for relevant keywords. We used a ranking-based approach where videos were scored by matching not only against the display name of the audio event, but also, with decreasing weight, its parents in the ontology. Including the term “sound” as a root node in these searches improved the precision of results. Temporal selection of segments with the returned videos was arbitrary, typically taking segments with a start time at 30 sec to avoid any introduction or channel branding at the start of the video. For some of the data, temporal selection was based on content analysis such as nearest-neighbour search between the manually-verified clips clips and the label-specific candidate videos.
此外,还通过在标题和其他元数据中检索相关关键词来选择视频。我们采用基于排序的方法:视频的得分不仅来自与音频事件显示名的匹配,也以递减的权重匹配其在本体中的父节点。在这些检索中把 “sound” 当作根节点纳入,提高了结果的精度。对返回视频中片段的时间选择是任意的,通常取起始时间为 30 秒的片段,以避免视频开头的引子或频道品牌宣传。对部分数据,时间选择基于内容分析,例如在人工核实过的片段与特定标签的候选视频之间做近邻检索。
Using all these techniques, there were some categories for which we were unable to find enough positive examples to fully populate the dataset, but this proportion is now very small (and shrinking).
即使用上所有这些技术,仍有一些类别我们找不到足够的正例来把数据集完全填满,不过这一比例现在已经很小(而且还在缩小)。
In general we found that segments nominated by the automatic video annotation system performed the best (49% of segments rated present, versus 41% for the metadata-based approach), with the caveat that not all audio classes are included. For audio classes not included in the automatic annotations, 36% of the metadata-based segments were rated present. The effectiveness of the content-similarity approaches varied widely depending on the quality of the example and the candidate set of videos.
总体来看,由自动视频标注系统提名的片段表现最好(49% 的片段被评为存在,基于元数据的方法则为 41%),需要说明的是,并非所有音频类别都被包含在内。对于自动标注未覆盖的音频类别,基于元数据的片段中有 36% 被评为存在。基于内容相似性的方法,其效果随例子质量和候选视频集合的不同而差异很大。
Regardless of the mechanism by which a segment was nominated, we gathered ratings for all labels associated with the segment by any method. Additionally, we always collected ratings for “Speech” and “Music” (excepting a few experimental runs).
无论片段是由哪种机制提名的,我们都会为该片段上通过任何方法关联到的全部标签收集评分。此外,我们几乎总是同时收集 “Speech” 和 “Music” 的评分(少数实验轮次除外)。
3.3. Dataset construction and characteristics
3.3. 数据集的构建与特征
The released dataset constitutes a subset of the collected material: Only “present” ratings are represented in the release. Rating was conducted with an effort to maximize balance across audio event labels. However, since segments can be labeled with multiple audio events (including the always-rated “Speech” and “Music”), certain labels appear much more frequently. A second objective was to avoid drawing more than one segment from any given video, to avoid correlation of examples.
发布的数据集是所收集材料的一个子集:发布中只保留 “present” 的评分。评分过程中尽量在各音频事件标签之间取得平衡。不过,由于一条片段可以被标上多个音频事件(包括总是会被评分的 “Speech” 和 “Music”),某些标签会频繁得多。第二个目标是避免从同一视频中抽取多于一个片段,以免例子之间相关。
These objectives were achieved by iteratively adding segments for the least-represented class (for which further examples are available). Out of the set of candidate segments for this audio event class, preference is given to segments bearing the greatest number of labels. We also provide maximally-balanced train and test subsets (from disjoint videos), chosen to provide at least 50 positive examples (in both subsets) for as many classes as possible. These sets were constructed by first collecting examples for the rarest classes, then moving on to less-rare classes and adding more segments only where they had not already passed the threshold of 50. Even so, very common labels such as “Music” ended up with more than 5000 labels.
这些目标是通过迭代实现的:每次为当前表示最少、而且还能找到更多例子的类别添加片段。在该类的候选片段集合中,优先选择带有最多标签的片段。我们还提供尽量平衡的训练和测试子集(来自互不相交的视频),目的是为尽可能多的类别在两个子集中都提供至少 50 个正例。构造时先为最稀有的类别收集例子,再转向不那么稀有的类别,并且只在尚未达到 50 的阈值时才继续添加片段。即便如此,像 “Music” 这样非常常见的标签,最终还是超过了 5000 个标签实例。
The resulting dataset includes 1,789,621 segments (4,971 hours), comprising at least 100 instances for 485 audio event categories. The remaining categories are either excluded (blacklisted / abstract as described in section 2.1), or difficult to find using our current approaches. We will continue to develop methods for proposing segments for rating, and aim eventually to cover all non-excluded classes. The unbalanced train set contains 1,771,873 segments and the evaluation set contains 17,748. Because single segments can have multiple labels (on average 2.7 labels per segment), the overall count of labels is not uniform, and is distributed as shown in Fig. 3. “Music” is particularly common, present in 56% of the segments.
最终的数据集包含 1,789,621 个片段(4,971 小时),其中 485 个音频事件类别至少各有 100 个实例。其余类别或被排除(如 2.1 节所述被列入黑名单或标为抽象),或按我们目前的方法难以找到。我们将继续改进提名待评分片段的方法,目标是最终覆盖所有未被排除的类别。非平衡训练集包含 1,771,873 个片段,评价集包含 17,748 个。由于单条片段可以带有多个标签(平均每段 2.7 个标签),标签的总体计数并不均匀,其分布如图 3 所示。“Music” 特别常见,出现在 56% 的片段中。
图 3:整个 1,789,621 段数据集上的标签计数直方图。
4. BENCHMARK
4. 基准
To give a sense of the performance possible with this data, we have trained a simple baseline system. Using the embedding layer representation of a deep-network classifier trained on a large set of generic video topic labels [22], we used the training portion of the Audio Set YouTube Corpus to train a shallow fully-connected neural network classifier for the 485 categories in the released segments. We evaluated on the test partition by applying the classifier to 1 sec frames taken from each segment, averaging the scores, then for each category ranking all segments by their scores. This system gave a balanced mean Average Precision across the 485 categories of 0.314, and an average AUC of 0.959 (corresponding to a d-prime class separation of 2.452). The category with the best AP was “Music” with AP / AUC / d-prime of 0.896 / 0.951 / 2.34 (reflecting its high prior); the worst AP was for “Rattle” with 0.020 / 0.796 / 1.168.
为了让人感受一下用这份数据能达到怎样的性能,我们训练了一个简单的基线系统。我们使用一个在大规模通用视频主题标签上训练的深度网络分类器的嵌入层表示 [22],再利用 Audio Set YouTube Corpus 的训练部分,为发布片段中的 485 个类别训练一个浅层全连接神经网络分类器。评价在测试划分上进行:把分类器应用到从每条片段中取出的 1 秒帧,将分数取平均,再按每个类别用这些分数对全部片段排序。该系统在 485 个类别上的平衡平均精度(mean Average Precision)为 0.314,平均 AUC 为 0.959(对应于 2.452 的 d-prime 类分离度)。AP 最好的类别是 “Music”,其 AP / AUC / d-prime 为 0.896 / 0.951 / 2.34(反映了它很高的先验);AP 最差的是 “Rattle”,为 0.020 / 0.796 / 1.168。
5. CONCLUSION
5. 结论
We have introduced the Audio Set dataset of generic audio events, comprising an ontology of 632 audio event categories and a collection of 1,789,621 labeled 10 sec excerpts from YouTube videos. The ontology is hierarchically structured with the goal of covering all acoustic distinctions made by a ‘typical’ listener. We are releasing this data to accelerate research in the area of acoustic event detection, just as ImageNet has driven research in image understanding. In the future, we hope to be able to make larger and improved releases, ideally including contributions from the wider community.
我们介绍了面向通用音频事件的 Audio Set 数据集,其中包括一个包含 632 个音频事件类别的本体,以及 1,789,621 条带标注的、来自 YouTube 视频的 10 秒片段。本体按层次组织,目标是覆盖“普通”听者所能做出的全部声学区分。我们发布这份数据,是为了加速声学事件检测领域的研究,正如 ImageNet 推动了图像理解研究一样。未来,我们希望能够发布更大、更好的版本,最好也能纳入更广泛社区的贡献。
6. REFERENCES
6. 参考文献
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, 2012, pp. 1097–1105.
[2] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1–9.
[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” arXiv preprint arXiv:1512.03385, 2015.
[4] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al., “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
[5] William H Warren and Robert R Verbrugge, “Auditory perception of breaking and bouncing events: a case study in ecological acoustics,” Journal of Experimental Psychology: Human perception and performance, vol. 10, no. 5, pp. 704, 1984.
[6] James A Ballas, “Common factors in the identification of an assortment of brief everyday sounds,” Journal of experimental psychology: human perception and performance, vol. 19, no. 2, pp. 250, 1993.
[7] Brian Gygi, Gary R Kidd, and Charles S Watson, “Similarity and categorization of environmental sounds,” Perception & psychophysics, vol. 69, no. 6, pp. 839–855, 2007.
[8] Guillaume Lemaitre and Laurie M. Heller, “Evidence for a basic level in a taxonomy of everyday action sounds,” Experimental Brain Research, vol. 226, pp. 253–264, 2013.
[9] William W Gaver, “What in the world do we hear? an ecological approach to auditory event perception,” Ecological psychology, vol. 5, no. 1, pp. 1–29, 1993.
[10] Tohomiro Nakatani and Hiroshi G Okuno, “Sound ontology for computational auditory scene analysis,” in Proceeding for the 1998 conference of the American Association for Artificial Intelligence, 1998.
[11] Susanne Burger, Qin Jin, Peter F Schulam, and Florian Metze, “Noisemes: Manual annotation of environmental noise in audio streams,” Tech. Rep. CMU-LTI-12-07, Carnegie Mellon University, 2012.
[12] Justin Salamon, Christopher Jacoby, and Juan Pablo Bello, “A dataset and taxonomy for urban sound research,” in Proceedings of the 22nd ACM international conference on Multimedia. ACM, 2014, pp. 1041–1044.
[13] Sebastian Sager, Damian Borth, Benjamin Elizalde, Christian Schulze, Bhiksha Raj, Ian Lane, and Andreas Dengel, “Audiosentibank: Large-scale semantic ontology of acoustic concepts for audio content analysis,” arXiv preprint arXiv:1607.03766, 2016.
[14] Erling Wold, Thom Blum, Douglas Keislar, and James Wheaten, “Content-based classification, search, and retrieval of audio,” IEEE multimedia, vol. 3, no. 3, pp. 27–36, 1996.
[15] Andrey Temko, Robert Malkin, Christian Zieger, Dušan Macho, Climent Nadeu, and Maurizio Omologo, “Clear evaluation of acoustic event detection and classification systems,” in International Evaluation Workshop on Classification of Events, Activities and Relationships. Springer, 2006, pp. 311–322.
[16] Dan Stowell, Dimitrios Giannoulis, Emmanouil Benetos, Mathieu Lagrange, and Mark D Plumbley, “Detection and classification of acoustic scenes and events,” IEEE Transactions on Multimedia, vol. 17, no. 10, pp. 1733–1746, 2015.
[17] Annamaria Mesaros, Toni Heittola, and Tuomas Virtanen, “TUT database for acoustic scene classification and sound event detection,” in 24th European Signal Processing Conference 2016 (EUSIPCO 2016), Budapest, Hungary, 2016, http://www.cs.tut.fi/sgn/arg/dcase2016/.
[18] Marti A Hearst, “Automatic acquisition of hyponyms from large text corpora,” in Proceedings of the 14th conference on Computational linguistics-Volume 2. Association for Computational Linguistics, 1992, pp. 539–545.
[19] Amit Singhal, “Introducing the knowledge graph: things, not strings,” 2012, Official Google blog, https://googleblog.blogspot.com/2012/05/introducing-knowledge-graph-things-not.html.
[20] George A Miller, “WordNet: a lexical database for English,” Communications of the ACM, vol. 38, no. 11, pp. 39–41, 1995.
[21] Sam Norman-Haignere, Nancy G Kanwisher, and Josh H McDermott, “Distinct cortical pathways for music and speech revealed by hypothesis-free voxel decomposition,” Neuron, vol. 88, no. 6, pp. 1281–1296, 2015.
[22] Shawn Hershey, Sourish Chaudhury, Daniel P. W. Ellis, Jort Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Rif A. Saurous, Brian Seybold, Malcolm Slaney, and Ron Weiss, “CNN architectures for large-scale audio classification,” in IEEE ICASSP 2017, New Orleans, 2017.


