HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Abstract
摘要
Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models are publicly available [1].
学习文本–视频嵌入,通常需要一批带人工描述的视频片段。不过,这类数据集既贵又费时,很难做到很大的规模。本文转而利用视频里已经带有的自然语言标注,也就是自动转写的解说,来学习这类嵌入。工作有三方面贡献。第一,我们发布 HowTo100M:一个大规模数据集,包含 1.36 亿个视频片段,来自 122 万段配有解说的网络教学视频,刻画人们完成并口述两万三千多种不同视觉任务的过程。采集流程快速、可扩展,也不再需要额外的人工标注。第二,我们表明,在这批数据上训练出的文本–视频嵌入,在 YouCook2、CrossTask 这类教学视频数据集上,为文本到视频检索和动作定位带来了当时最好的结果。最后,我们说明这一嵌入也能迁到其他领域:在通用 YouTube 视频(MSR-VTT)和电影(LSMDC)上微调后,效果超过只在这些数据集上训练的模型。数据集、代码和模型均已公开 [1]。
图 1:我们通过观看人们完成各类视觉任务、并配有解说的数百万个视频片段,来学习文本–视频联合嵌入。学到的嵌入能够较好地迁移到其他教学与非教学的文本–视频数据集上。
- Introduction
Communicating about the visual world using language is a key ability of humans as intelligent beings. A three year old child can manipulate objects, observe its own actions and describe them to others using language; while adults can learn new skills by reading books or watching videos. This interplay between video and language extends naturally to artificial agents that need to understand the visual world and communicate about it with people. Examples of tasks that still represent a significant challenge for current artificial systems include text-to-video retrieval [25, 32, 54, 55, 63], text-based action or event localization [15], video captioning [36, 61], and video question answering [51, 63]. Yet, progress on these problems is important for a host of applications from searching video archives to human-robot communication.
用语言谈论所见的视觉世界,是人类作为智能生物的一项关键能力。三岁的孩子已经能摆弄物体、观察自己的动作,再用语言讲给别人听;成年人则可以通过读书或看视频学会新的技能。视频与语言之间的这种互动,很自然地延伸到需要理解视觉世界、并与人就此交流的智能体。对当前人工系统来说,仍相当困难的任务包括文本到视频检索 [25, 32, 54, 55, 63]、基于文本的动作或事件定位 [15]、视频描述 [36, 61],以及视频问答 [51, 63]。但把这些问题向前推进一步,对从视频档案检索到人机交流等一系列应用都很重要。
A common approach to model visual concepts described with language is to learn a mapping of text and video into a shared embedding space, where related text fragments and video clips are close to each other [15, 32, 37, 38, 59]. Learning a good representation often requires a large set of paired video clips and text captions. In fact, given the huge variability of video scenes and their textual descriptions, learning a generic embedding space may require millions of paired video clips and text captions. However, existing datasets (e.g. MSR-VTT [58], DiDeMo [15], EPIC-KITCHENS [7]), are on the scale of tens to hundreds of thousands of such pairs that have been annotated manually. Manual collection of such datasets is expensive and hard to scale. It is also subjective since video annotation can often be an ill-defined task with low annotator consistency [58].
要对语言所描述的视觉概念建模,一条常用途径是学习把文本和视频映到同一个嵌入空间,使相关的文本片段和视频片段彼此靠近 [15, 32, 37, 38, 59]。要学到好的表示,往往需要大量成对的视频片段与文本描述。事实上,考虑到视频场景及其文字描述的巨大差异,要学一个通用的嵌入空间,可能需要数百万对样本。然而,现有数据集(如 MSR-VTT [58]、DiDeMo [15]、EPIC-KITCHENS [7])的规模大约只有数万到数十万对,而且都靠人工标注。人工收集这类数据既昂贵,也很难扩大规模。它还带有主观性,因为视频标注常常本身就不那么好定义,标注者之间的一致性也偏低 [58]。
In this work, we explore a different source of supervision to obtain paired video clips and text captions for learning joint representations of video and language. We observe that narrated instructional videos are available in large quantities (e.g. on YouTube) and provide a large amount of visual and language data. In particular, instructional videos [2, 30, 68] often contain narration with an explicit intention of explaining the visual content on screen. To leverage this rich source of data, we collect a new large-scale dataset containing 136 million video clips sourced from 1.22 million narrated instructional videos depicting humans performing more than 23,000 different tasks. Each clip is paired with a text annotation in the form of an automatically transcribed narration.
本文探索另一种监督来源,以得到成对的视频片段和文本描述,用来学习视频与语言的联合表示。我们注意到,配有解说的教学视频数量很大(例如在 YouTube 上),能够提供大量视觉和语言数据。尤其是,教学视频 [2, 30, 68] 里的解说往往带有明确意图,就是要把屏幕上的视觉内容讲清楚。为利用这一丰富来源,我们收集了一个新的大规模数据集,包含 1.36 亿个视频片段,来自 122 万段配有解说的教学视频,刻画人们完成两万三千多项不同任务的过程。每个片段都配有一条文本标注,形式是自动转写的解说。
Contributions. The contributions of this work are three-fold. First, we collect a new dataset of close-captioned video clips, HowTo100M, that is orders of magnitude larger than any other existing video-text datasets (Section 3). Second, we show that such data can be used to learn powerful video-language representations. Our model (Section 4), trained on HowTo100M, sets a new state-of-the-art for text-based action localization and text-to-video retrieval on existing datasets of instructional videos, YouCook2 [67] and CrossTask [68]. Finally, we explore the ability of models trained on our data to transfer to non-instructional videos. In particular, we demonstrate that models pretrained on HowTo100M can be successfully transferred by fine tuning on the MSR-VTT dataset (generic Youtube videos) and the LSMDC dataset (movies).
贡献。本文有三方面贡献。第一,我们收集了一个带字幕的视频片段新数据集 HowTo100M,规模比当时已有的任何视频–文本数据集都高出数个数量级(第 3 节)。第二,我们表明这类数据可以用来学习有力的视频–语言表示。我们的模型(第 4 节)在 HowTo100M 上训练后,在已有的教学视频数据集 YouCook2 [67] 和 CrossTask [68] 上,为基于文本的动作定位和文本到视频检索树立了新的最好结果。最后,我们考察在这批数据上训练的模型迁移到非教学视频的能力。具体而言,我们表明在 HowTo100M 上预训练的模型,可以通过在 MSR-VTT(通用 YouTube 视频)和 LSMDC(电影)上微调,成功地完成迁移。
- Related work
A significant number of computer vision applications rely on a joint understanding of visual and textual cues. These applications include automatic image and video captioning [20, 36, 60, 61], visual question answering [9, 29, 51, 63], visual content retrieval based on textual queries [32, 56, 63], temporal localization of events in videos using natural language [15, 26] or video summarization with natural language [38].
相当多的计算机视觉应用,都依赖于对视觉线索和文本线索的联合理解。这些应用包括自动图像与视频描述 [20, 36, 60, 61]、视觉问答 [9, 29, 51, 63]、基于文本查询的视觉内容检索 [32, 56, 63]、用自然语言在视频中做事件的时间定位 [15, 26],以及用自然语言做视频摘要 [38]。
Vision, language and speech. A common approach to model vision and language is learning a joint embedding space where visual and textual cues are adjacent if and only if they are semantically similar [6, 8, 10, 11, 25, 32, 35, 37, 38, 59, 54, 55, 57]. Most of these works rely on medium scale well annotated datasets in which descriptive captions are collected for each video clip. This process is costly as it requires considerable human annotation effort making these datasets hard to scale (see Table 1). In this work, we train a joint video and language model without a single manually annotated video description by leveraging automatically transcribed narrated videos. Using the spoken text from narrated videos to supervise vision models has seen some recent interest [2, 5, 13, 30, 45, 62]. Harwath et al. [13] utilize the raw speech waveform to supervise the visual model, however, their method does not scale as annotators were paid to record audio descriptions for thousands of images. Chen et al. [5] use subtitles from documentaries to automatically obtain object labels, but their focus is on learning object detectors rather than text-video embeddings and their dataset contains only 9 documentary movies, compared to about 15 years of video content considered in this work.
视觉、语言与语音。对视觉和语言建模的一条常用途径,是学习一个联合嵌入空间,使视觉线索和文本线索在且仅在语义相近时彼此相邻 [6, 8, 10, 11, 25, 32, 35, 37, 38, 59, 54, 55, 57]。这些工作大多依赖中等规模、标注较好的数据集,其中每个视频片段都收集了描述性字幕。这一过程代价很高,需要大量人工标注,因而很难把数据集做大(见表 1)。本文则利用自动转写的解说视频,在完全没有人工撰写视频描述的情况下,训练联合的视频–语言模型。用解说视频里的口头文本来监督视觉模型,近期已有一些关注 [2, 5, 13, 30, 45, 62]。Harwath 等人 [13] 用原始语音波形监督视觉模型,但他们的方法难以扩大规模,因为是付费请标注者为数千张图像录制音频描述。Chen 等人 [5] 用纪录片字幕自动得到物体标签,但他们的重点是学习物体检测器而非文本–视频嵌入,而且数据集只有 9 部纪录片,相比之下,本文考虑的视频内容大约相当于 15 年。
| Dataset | Clips | Captions | Videos | Duration | Source | Year |
|---|---|---|---|---|---|---|
| Charades [48] | 10k | 16k | 10,000 | 82h | Home | 2016 |
| MSR-VTT [58] | 10k | 200k | 7,180 | 40h | Youtube | 2016 |
| YouCook2 [67] | 14k | 14k | 2,000 | 176h | Youtube | 2018 |
| EPIC-KITCHENS [7] | 40k | 40k | 432 | 55h | Home | 2018 |
| DiDeMo [15] | 27k | 41k | 10,464 | 87h | Flickr | 2017 |
| M-VAD [52] | 49k | 56k | 92 | 84h | Movies | 2015 |
| MPII-MD [43] | 69k | 68k | 94 | 41h | Movies | 2015 |
| ANet Captions [26] | 100k | 100k | 20,000 | 849h | Youtube | 2017 |
| TGIF [27] | 102k | 126k | 102,068 | 103h | Tumblr | 2016 |
| LSMDC [44] | 128k | 128k | 200 | 150h | Movies | 2017 |
| How2 [45] | 185k | 185k | 13,168 | 298h | Youtube | 2018 |
| HowTo100M | 136M | 136M | 1.221M | 134,472h | Youtube | 2019 |
Table 1: Comparison of existing video description datasets. The size of our new HowTo100M dataset bypasses the size of largest available datasets by three orders of magnitude. M denotes million while k denotes thousand.
| 数据集 | 片段 | 描述 | 视频 | 时长 | 来源 | 年份 |
|---|---|---|---|---|---|---|
| Charades [48] | 10k | 16k | 10,000 | 82h | Home | 2016 |
| MSR-VTT [58] | 10k | 200k | 7,180 | 40h | Youtube | 2016 |
| YouCook2 [67] | 14k | 14k | 2,000 | 176h | Youtube | 2018 |
| EPIC-KITCHENS [7] | 40k | 40k | 432 | 55h | Home | 2018 |
| DiDeMo [15] | 27k | 41k | 10,464 | 87h | Flickr | 2017 |
| M-VAD [52] | 49k | 56k | 92 | 84h | Movies | 2015 |
| MPII-MD [43] | 69k | 68k | 94 | 41h | Movies | 2015 |
| ANet Captions [26] | 100k | 100k | 20,000 | 849h | Youtube | 2017 |
| TGIF [27] | 102k | 126k | 102,068 | 103h | Tumblr | 2016 |
| LSMDC [44] | 128k | 128k | 200 | 150h | Movies | 2017 |
| How2 [45] | 185k | 185k | 13,168 | 298h | Youtube | 2018 |
| HowTo100M | 136M | 136M | 1.221M | 134,472h | Youtube | 2019 |
表 1:现有视频描述数据集的比较。新的 HowTo100M 在规模上比当时最大的可用数据集高出三个数量级。M 表示百万,k 表示千。
Learning from instructional videos. Instructional videos are rising in popularity in the context of learning steps of complex tasks [2, 16, 41, 42, 46, 68], visual-linguistic reference resolution [17, 18], action segmentation in long untrimmed videos [66] and joint learning of object states and actions [3]. Related to our work, [2, 30, 62] also consider automatically generated transcription of narrated instructional videos as a source of supervision. However as opposed to our work, these works typically extract from transcriptions only a small number of predefined labels. Numerous datasets of web instructional videos were proposed over the past years [2, 30, 45, 47, 50, 67, 68]. Among the first to harvest instructional videos, Sener et al. [47] use WikiHow, an encyclopedia of how to articles, to collect 17 popular physical tasks, and obtain videos by querying these tasks on YouTube. In a similar vein, COIN [50] and CrossTask [68] datasets are collected by first searching for tasks on WikiHow and then videos for each task on YouTube. We use the same approach for collecting HowTo100M. The main distinction between our dataset and previous efforts is the unprecedented scale both in terms of variety (more than 23,000 tasks from 12 different domains) and size (136 million clips sourced from 1.2 million instructional videos).
从教学视频中学习。在学习复杂任务的步骤 [2, 16, 41, 42, 46, 68]、视觉–语言指代消解 [17, 18]、在未剪辑的长视频中做动作分割 [66],以及联合学习物体状态与动作 [3] 这些问题上,教学视频正越来越受关注。与本文相关的是,[2, 30, 62] 同样把配解说教学视频的自动转写当作监督来源。但与本文不同,这些工作通常只从转写里抽出少量预先定义的标签。过去几年里,已经有不少网络教学视频数据集被提出 [2, 30, 45, 47, 50, 67, 68]。较早开始收集教学视频的 Sener 等人 [47],利用 WikiHow 这一 How to 条目百科,选出 17 项常见的动手任务,再在 YouTube 上检索这些任务得到视频。与此类似,COIN [50] 和 CrossTask [68] 也是先在 WikiHow 上检索任务,再为每个任务在 YouTube 上找视频。我们用同样的方式收集 HowTo100M。本数据集与以往工作的主要区别,在于前所未有的规模:既体现在多样性上(来自 12 个不同领域的两万三千多项任务),也体现在体量上(从 120 万段教学视频中得到 1.36 亿个片段)。
Figure 2: Examples of clip-caption pairs retrieved with the help of our joint embedding. Pairs are selected based on the similarity between visual appearance and corresponding narration, while they are arranged based on linguistic similarity across pairs. Examples are taken from 4 distinct clusters, corresponding to Knitting, Woodwork/Measuring, Cooking/Seasoning and Electric maintenance.
图 2:借助我们的联合嵌入检索到的片段–描述配对示例。配对按视觉外观与对应解说之间的相似度选出,再按各对之间的语言相似度排列。例子来自 4 个不同簇,分别对应编织、木工/测量、烹饪/调味,以及电气维修。
Large scale data for model pretraining. The use of large scale and potentially noisy data from the web is an exciting prospect to pretrain language and vision models. In natural language processing, BERT [19], GPT [39], and GPT-2 [40] are examples of language models trained on large-scale data that achieve state-of-the-art for many tasks. In fact, training GPT-2 on WebText [40] a dataset of 40GB of text from Reddit achieves state-of-the-art even in zero-shot settings. In vision, [28, 49] explore the use of image metadata such as Instagram hashtags to pretrain image classifiers. We are inspired by these works and focus our efforts on learning a strong embedding for joint understanding of video and language. We demonstrate that our video-language embedding learned from millions of YouTube videos not only outperforms previous work on tasks related to instructional videos without fine-tuning, but also generalizes well to non-instructional videos with some fine-tuning. We release our dataset, feature extraction pipeline, and model parameters as a resource that the video and language community can build on.
用于模型预训练的大规模数据。用网上大规模、可能带噪声的数据来预训练语言和视觉模型,是一条令人振奋的思路。在自然语言处理中,BERT [19]、GPT [39] 和 GPT-2 [40] 都是在大规模数据上训练、并在许多任务上取得当时最好结果的语言模型。事实上,在来自 Reddit、规模为 40GB 文本的 WebText [40] 上训练 GPT-2,即便在零样本设定下也能达到最好水平。在视觉方面,[28, 49] 探索用 Instagram 标签这类图像元数据来预训练图像分类器。我们受到这些工作的启发,把力气放在学习一个用于视频与语言联合理解的有力嵌入上。我们表明,从数百万段 YouTube 视频中学到的视频–语言嵌入,不仅在与教学视频相关的任务上无需微调就超过已有工作,而且在稍作微调后,对非教学视频也有很好的泛化。我们公开数据集、特征提取流程和模型参数,供视频与语言社区继续使用。
- The HowTo100M dataset
We collect a new dataset of narrated videos with an emphasis on instructional videos where content creators teach complex tasks. This ensures that most narrations describe the observed visual content. HowTo100M features 1.22 million videos from YouTube, with activities from domains such as cooking, hand crafting, personal care, gardening, etc. Each video is associated with a narration available as subtitles that are either written manually or are the output of an Automatic Speech Recognition (ASR) system.
我们收集了一个新的解说视频数据集,重点放在内容创作者教授复杂任务的教学视频上。这样可以保证大多数解说都在描述屏幕上看到的视觉内容。HowTo100M 包含来自 YouTube 的 122 万段视频,活动领域包括烹饪、手工、个人护理、园艺等。每段视频都配有可以当作字幕使用的解说,这些字幕或是人工撰写的,或是自动语音识别(ASR)系统的输出。
3.1. Data collection
Visual tasks. With an aim to obtain instructional videos that describe how to perform certain activities, we first start by acquiring a large list of activities using WikiHow – an online resource that contains 120,000 articles on How to ... for a variety of domains ranging from cooking to human relationships structured in a hierarchy. We are primarily interested in "visual tasks" that involve some interaction with the physical world (e.g. Making peanut butter, Pruning a tree) as compared to others that are more abstract (e.g. Ending a toxic relationship, Choosing a gift). To obtain predominantly visual tasks, we limit them to one of 12 categories (listed in Table 2). We exclude categories such as Relationships and Finance and Business, that may be more abstract. We further refine the set of tasks, by filtering them in a semi-automatic way. In particular, we restrict the primary verb to physical actions, such as make, build and change, and discard non-physical verbs, such as be, accept and feel. This procedure yields 23,611 visual tasks in total.
视觉任务。为了得到描述如何完成某些活动的教学视频,我们先借助 WikiHow 获取一份很长的活动清单。WikiHow 是一个在线资源,收录了 12 万篇「How to …」文章,领域从烹饪到人际关系,并按层级组织。我们主要关心那些需要与物理世界发生某种互动的「视觉任务」(例如做花生酱、修剪树木),而不是更抽象的任务(例如结束一段有害的关系、挑选礼物)。为得到以视觉为主的任务,我们把范围限制在 12 个类目之内(见表 2),并排除人际关系、金融与商业这类可能更抽象的类目。我们还以半自动的方式进一步筛选任务:把中心动词限制为 make、build、change 这类身体动作,去掉 be、accept、feel 这类非身体动词。这一流程最终得到 23,611 项视觉任务。
Instructional videos. We search for YouTube videos related to the task by forming a query with how to preceding the task name (e.g. how to paint furniture). We choose videos that have English subtitles - either uploaded manually, generated automatically by YouTube ASR, or generated automatically after translation from a different language by YouTube API.
教学视频。我们用 how to 加上任务名称构成查询(例如 how to paint furniture),在 YouTube 上检索与该任务相关的视频。我们只选用带英文字幕的视频——字幕可能是人工上传的,也可能由 YouTube 的自动语音识别生成,或先由其他语言再经 YouTube API 自动翻译得到。
We improve the quality and consistency of the dataset, by adopting the following criteria. We restrict to the top 200 search results, as the latter ones may not be related to the query task. Videos with less than 100 views are removed as they are often of poor quality or are amateurish. We also ignore videos that have less than 100 words as that may be insufficient text to learn a good video-language embedding. Finally, we remove videos longer than 2,000 seconds.
为提高数据集的质量和一致性,我们采用如下标准。只保留检索结果的前 200 条,因为更靠后的结果可能与查询任务无关。播放量不足 100 的视频会被去掉,因为它们往往质量较差,或显得不够专业。字幕少于 100 词的视频我们也不采用,因为文本量可能不足以学到好的视频–语言嵌入。最后,我们去掉长度超过 2,000 秒的视频。
| Category | Tasks | Videos | Clips |
|---|---|---|---|
| Food and Entertaining | 11504 | 497k | 54.4M |
| Home and Garden | 5068 | 270k | 29.5M |
| Hobbies and Crafts | 4273 | 251k | 29.8M |
| Cars & Other Vehicles | 810 | 68k | 7.8M |
| Pets and Animals | 552 | 31k | 3.5M |
| Holidays and Traditions | 411 | 27k | 3.0M |
| Personal Care and Style | 181 | 16k | 1.6M |
| Sports and Fitness | 205 | 16k | 2.0M |
| Health | 172 | 15k | 1.7M |
| Education and Communications | 239 | 15k | 1.6M |
| Arts and Entertainment | 138 | 10k | 1.2M |
| Computers and Electronics | 58 | 5k | 0.6M |
| Total | 23.6k | 1.22M | 136.6M |
Table 2: Number of tasks, videos and clips within each category.
| 类别 | 任务 | 视频 | 片段 |
|---|---|---|---|
| Food and Entertaining | 11504 | 497k | 54.4M |
| Home and Garden | 5068 | 270k | 29.5M |
| Hobbies and Crafts | 4273 | 251k | 29.8M |
| Cars & Other Vehicles | 810 | 68k | 7.8M |
| Pets and Animals | 552 | 31k | 3.5M |
| Holidays and Traditions | 411 | 27k | 3.0M |
| Personal Care and Style | 181 | 16k | 1.6M |
| Sports and Fitness | 205 | 16k | 2.0M |
| Health | 172 | 15k | 1.7M |
| Education and Communications | 239 | 15k | 1.6M |
| Arts and Entertainment | 138 | 10k | 1.2M |
| Computers and Electronics | 58 | 5k | 0.6M |
| 合计 | 23.6k | 1.22M | 136.6M |
表 2:各类目中的任务数、视频数与片段数。
As some videos may appear in several tasks, we de-duplicate videos based on YouTube IDs. However, note that the dataset may still contain duplicates if a video was uploaded several times or edited and re-uploaded. Nevertheless, this is not a concern at our scale.
由于同一段视频可能出现在多项任务中,我们按 YouTube ID 做了去重。不过需要说明,如果同一视频被多次上传,或经过剪辑后再上传,数据集里仍可能有重复。但在我们这样的规模下,这并不会构成问题。
3.2. Paired video clips and captions
Subtitles are often organized as a list of text chunks (lines), and need not form complete sentences. Each line is associated with a time interval in the video, typically the duration in which the line is uttered. We select each line of the subtitles as a caption, and pair it with the video clip from the time interval corresponding to the line. We show some examples from our clip-caption pairs in Figure 2.
字幕往往整理成一串文本块(行),不一定构成完整句子。每一行对应视频中的一段时间,通常就是这行话被说出的区间。我们把字幕的每一行当作一条描述,并与该行所对应时间区间里的视频片段配对。图 2 给出了若干片段–描述配对的例子。
Different from other datasets with clip-caption pairs (e.g. MSR-VTT), our captions are not manually annotated, but automatically obtained through the narration. Thus, they can be thought of as weakly paired. Typical examples of incoherence include the content producer asking viewers to subscribe to their channel, talking about something unrelated to the video, or describing something before or after it happens. Furthermore, our captions are often incomplete, lack punctuation, or are grammatically incorrect sentences, as they come from continuous narration and often ASR. We have manually inspected 400 randomly sampled clip-caption pairs and found that in 51 %, at least one object or action mention in the caption is visually seen in the video clip.
与其他带有片段–描述配对的数据集(例如 MSR-VTT)不同,我们的描述不是人工标注的,而是从解说中自动得到的,因此可以看作弱配对。常见的不一致包括:内容创作者请观众订阅频道、谈论与视频无关的事情,或在事情发生之前或之后才去描述。此外,这些描述常常不完整、缺少标点,或不合语法,因为它们来自连续解说,而且往往来自自动语音识别。我们人工检查了随机抽取的 400 对片段–描述,发现其中 51% 至少能在视频片段里看到描述所提到的一个物体或动作。
Statistics. The initial set of visual tasks are obtained by focusing on 12 WikiHow categories. Table 2 shows the number of collected WikiHow tasks and corresponding videos and clips per category. In Appendix A [33], we show the first two levels of the WikiHow hierarchy: the twelve categories and their subcategories along with the number of chosen tasks and corresponding videos in our dataset. We compare the sizes of existing clip-caption paired datasets in Table 1. HowTo100M is several orders of magnitude larger than existing datasets and contains an unprecedented duration (15 years) of video data. However, unlike previous datasets, HowTo100M does not have clean annotated captions. As the videos contain complex activities, they are relatively long with an average duration of 6.5 minutes. On average, a video produces 110 clip-caption pairs, with an average duration of 4 seconds per clip and 4 words (after excluding stop-words) per caption. For more details, we show in Appendix A [33] the distribution of nouns and verbs. Our data collection procedure assumes that searching with How to queries on YouTube would result in mostly instructional videos. We verify this by randomly selecting 100 videos and labeling their type. 71% of the videos are found to be instructional, 12% are vlogs, and another 7% are product reviews or advertisements. Note that vlogs, reviews and ads may also contain correspondences between visual content and narration. In particular, we noticed that objects shown on screen are often mentioned in narration. We do not discard such non-instructional videos, as they may still be useful for the learning the joint embedding.
统计。最初的视觉任务集合,是通过聚焦 WikiHow 的 12 个类目得到的。表 2 给出了各类目中收集到的 WikiHow 任务数,以及对应的视频数和片段数。在附录 A [33] 中,我们展示了 WikiHow 层级的前两级:这十二个类目及其子类,以及本数据集中入选的任务数和对应视频数。表 1 比较了现有片段–描述配对数据集的规模。HowTo100M 比已有数据集高出数个数量级,并包含前所未有的视频时长(15 年)。不过,与以往数据集不同,HowTo100M 并没有干净的人工标注描述。由于视频包含复杂活动,它们相对较长,平均时长为 6.5 分钟。平均每段视频产生 110 对片段–描述,每个片段平均 4 秒,每条描述在去掉停用词后平均 4 个词。更多细节见附录 A [33] 中的名词与动词分布。我们的采集流程假定,用 How to 查询在 YouTube 上检索,得到的大多会是教学视频。我们随机抽取 100 段视频并标注类型来核实这一点:其中 71% 是教学视频,12% 是 Vlog,另有 7% 是产品评测或广告。需要说明的是,Vlog、评测和广告里也可能存在视觉内容与解说的对应。我们尤其注意到,屏幕上出现的物体常常会在解说里被提到。我们没有丢掉这类非教学视频,因为它们对学习联合嵌入仍可能有用。
- Text-video joint embedding model
We now present our model to learn a joint text-video embedding from the automatically paired video clips and captions in our dataset. More formally, we are given a set of video clips and associated captions . We denote by and the and dimensional feature representation of a video clip and caption , respectively. Given this, our goal is to learn two mapping functions: and that respectively embed video and caption features into a common -dimensional space, such that the cosine similarity
接下来给出我们的模型,用于从数据集里自动配对的视频片段和描述中学习文本–视频联合嵌入。更形式地说,给定 个视频片段及其对应描述 。我们分别用 和 表示视频片段 与描述 的 维和 维特征。在此基础上,目标是学习两个映射: 和 ,分别把视频特征和描述特征嵌入到同一个 维空间,使得余弦相似度
is high when caption describes the video clip , and low otherwise.
在描述 讲的是视频片段 时取较高的值,否则取较低的值。
In this work, we use the class of non-linear embedding functions used in [32], which are given by:
本文采用 [32] 中使用的那类非线性嵌入函数,形式如下:
where , , , are learnable parameters, is an element-wise sigmoid activation and is the element-wise multiplication (Hadamard product). In practice, , and resulting in a model composed of 67M parameters. Note that the first term on the right-hand side in Equations (2) and (3) is a linear fully-connected layer and the second term corresponds to a context gating function [31] with an output ranging between 0 and 1, which role is to modulate the output of the linear layer. As a result, this embedding function can model non-linear multiplicative interactions between the dimensions of the input feature vector which has proven effective in other text-video embedding applications [32].
其中 、、, 是可学习参数, 是按元素的 sigmoid 激活, 是按元素相乘(Hadamard 积)。实际中取 、、,模型共有 6700 万个参数。需要说明的是,式 (2) 和式 (3) 右边的第一项是线性全连接层,第二项对应输出介于 0 和 1 之间的 context gating 函数 [31],其作用是调节线性层的输出。这样,该嵌入函数能够刻画输入特征向量各维之间的非线性乘法交互,这一点在其他文本–视频嵌入应用中已被证明有效 [32]。
Loss. We train our embedding model using the max-margin ranking loss [21, 32, 54, 55, 64]. At each iteration of our training algorithm, we sample a mini-batch of caption-clip training pairs , and update the model parameters with a gradient step of the following loss:
损失。我们用最大间隔排序损失 [21, 32, 54, 55, 64] 训练嵌入模型。在训练算法的每一步,我们采样一个由描述–片段训练对 构成的小批量 ,并用下列损失做一次梯度更新:
where is the similarity score (1) between video clip and caption , is a set of negative pairs for caption-clip and is the margin. The first term in Equation (B) corresponds to the ranking loss when sampling a negative caption, while the second term corresponds to sampling a negative video clip. We fix in practice. Our model parameters are updated using Adam [23] with a learning rate of . Implementation details of the loss are provided in Appendix B [33].
其中 是视频片段 与描述 之间的相似度(式 (1)), 是描述–片段 的负样本集合, 是间隔。Equation (B) 中的第一项对应采样一条负描述时的排序损失,第二项对应采样一个负视频片段。实践中我们固定 。模型参数用 Adam [23] 更新,学习率为 。损失的实现细节见附录 B [33]。
Sampling strategy. Similar to [15], we apply an intra-video negative sampling strategy to define . We show in Section 5.3 that this approach is critical for good performance. More precisely, half of our negative pairs , are selected such that the video clip and the caption belong to the same original YouTube video (as ), while the other half are sampled from other YouTube videos. We apply intra-negative sampling to ensure that the learned embedding focuses on relevant aspects of the video clip (e.g. the hands of the person showing how to knead dough) rather than irrelevant background features (e.g. the kitchen). In Appendix C [33], we also provide an empirical analysis of the positive pair sampling strategy. We show that even though the training data is noisy, our attempts to automatically select correct positive pairs during training did not yield improvements so far. We think this could be attributed to the fact our model is shallow and is trained on a large amount of data.
采样策略。与 [15] 类似,我们用视频内负采样策略来定义 。第 5.3 节表明,这一做法对取得好的效果至关重要。更具体地说,一半负样本对 的选取方式,是让视频片段 与描述 来自同一条原始 YouTube 视频(与 相同),另一半则从其他 YouTube 视频中采样。采用视频内负采样,是为了让学到的嵌入把注意力放在片段中真正相关的部分(例如演示揉面的那双手),而不是无关的背景特征(例如厨房)。在附录 C [33] 中,我们还对正样本对的采样策略做了实证分析。结果表明,尽管训练数据是有噪声的,我们在训练过程中自动筛选正确正样本的尝试,迄今并未带来提升。我们认为,这可能是因为模型本身较浅,而又是在大量数据上训练的。
Clip and caption representation. The clip feature consists of temporally max-pooled pre-extracted CNN features. The caption feature is the output of a shallow 1D-CNN on top of pre-computed word embeddings. More details are given in Section 5.1.
片段与描述表示。片段特征 由预先提取的 CNN 特征经时间维最大池化得到。描述特征 是在预先计算的词嵌入之上,用一个浅层一维卷积神经网络得到的输出。更多细节见第 5.1 节。
- Experiments
In this section, we demonstrate that a strong joint representation for video and text can be learned from our unlabeled HowTo100M dataset. We provide experimental results for a variety of domains ranging from instructional videos in CrossTask [68], cooking videos in YouCook2 [67], generic YouTube videos in MSR-VTT [58] to movie video clips in LSMDC [44]. Specifically, we evaluate our learned embedding on the tasks of localizing steps in instructional videos of CrossTask [68] and text-based video retrieval on YouCook2 [67], MSR-VTT [58] and LSMDC [44] datasets.
本节表明,可以从未标注的 HowTo100M 数据集中,学到有力的视频–文本联合表示。我们在多种领域上给出实验结果,从 CrossTask [68] 中的教学视频、YouCook2 [67] 中的烹饪视频、MSR-VTT [58] 中的通用 YouTube 视频,到 LSMDC [44] 中的电影片段。具体来说,我们在 CrossTask [68] 教学视频上的步骤定位,以及 YouCook2 [67]、MSR-VTT [58] 和 LSMDC [44] 上的基于文本的视频检索任务上,评估学到的嵌入。
Our key findings are the following: (i) For instructional video datasets, such as CrossTask [68] and YouCook2 [67], our off-the-shelf embedding trained on HowTo100M significantly outperforms state-of-the-art models trained on much smaller and manually-annotated datasets. (ii) On generic YouTube videos (MSR-VTT [58]), our HowTo100M embedding provides competitive retrieval performance compared to state-of-the-art methods trained on MSR-VTT. Moreover, we show that fine-tuning our pre-trained embedding model on just a fifth of annotated videos from MSR-VTT outperforms state-of-the-art. (iii) We show that fine-tuning our embedding on LSMDC enables generalization to movie videos and scripts despite the large domain gap. (iv) Finally, we demonstrate the importance of scale in HowTo100M to learn better joint video-text embeddings.
主要发现如下:(i) 在 CrossTask [68] 和 YouCook2 [67] 这类教学视频数据集上,仅在 HowTo100M 上训练、可直接拿来使用的嵌入,显著超过了在小得多、且经过人工标注的数据集上训练的当时最好模型。(ii) 在通用 YouTube 视频(MSR-VTT [58])上,HowTo100M 嵌入的检索表现,可以和直接在 MSR-VTT 上训练的最好方法相比。此外,我们表明,只需用 MSR-VTT 中五分之一的标注视频来微调预训练嵌入模型,就能超过当时最好结果。(iii) 我们表明,在 LSMDC 上微调该嵌入,尽管领域差异很大,仍能泛化到电影视频和剧本。(iv) 最后,我们说明 HowTo100M 的规模对于学到更好的视频–文本联合嵌入是重要的。
5.1. Implementation details
Video features. We extract frame-level and video-level features with pre-trained 2D and 3D CNNs. 2D features are extracted with the ImageNet pre-trained Resnet-152 [14] at the rate of one frame per second. 3D features are extracted with the Kinetics [4] pre-trained ResNeXt-101 16-frames model [12] to obtain 1.5 features per second. We aggregate features from longer video clips by the temporal max-pooling and concatenate 2D and 3D features to form a single 4096 dimensional vector for each video clip.
视频特征。我们用预训练的二维和三维卷积神经网络提取帧级和视频级特征。二维特征用在 ImageNet 上预训练的 Resnet-152 [14] 提取,速率为每秒一帧。三维特征用在 Kinetics [4] 上预训练的 ResNeXt-101 16 帧模型 [12] 提取,得到每秒 1.5 个特征。对于更长的视频片段,我们用时间维最大池化聚合特征,再把二维和三维特征拼接,为每个视频片段形成一个 4096 维向量。
Text pre-processing. We preprocess transcribed video narrations by discarding common English stop-words. For the word representations, we use the GoogleNews pre-trained word2vec embedding model [34].
文本预处理。我们先去掉转写解说中常见的英语停用词。词表示采用在 GoogleNews 上预训练的 word2vec 嵌入模型 [34]。
Training time. Once the video and text features are extracted, training our embedding model on the full HowTo100M dataset is relatively fast and takes less than three days on a single Tesla P100 GPU.
训练时间。一旦提取好视频和文本特征,在完整 HowTo100M 上训练嵌入模型相对较快,在单块 Tesla P100 GPU 上不到三天即可完成。
5.2. Datasets and evaluation setups
Action step localization. We evaluate localization of action steps in instructional videos on the recent CrossTask dataset [68]. CrossTask includes 18 tasks and 2.7k instructional videos with manually annotated action segments. Each video may contain multiple segments, corresponding to different actions. It also provides an ordered list of action steps with short natural language descriptions for each task. We apply our model trained only on HowTo100M to the problem of step localization by computing similarity between every frame in the video and the action label names of CrossTask. In order to compare to [68], we follow a similar inference procedure. We use the same recall metric as in [68], which is defined by the number of step assignments that fall into the correct ground truth interval, divided by the total number of steps in the video. Videos from the test set of CrossTask are removed from the HowTo100M training set to ensure that they are not observed at training time.
动作步骤定位。我们在较新的 CrossTask 数据集 [68] 上,评估教学视频中的动作步骤定位。CrossTask 包含 18 项任务和 2700 段带人工标注动作片段的教学视频。每段视频可以包含多个片段,对应不同动作。它还为每项任务提供了带简短自然语言描述的有序步骤列表。我们把只在 HowTo100M 上训练的模型用于步骤定位:计算视频中每一帧与 CrossTask 动作标签名称之间的相似度。为了与 [68] 比较,我们采用类似的推断流程,并使用与 [68] 相同的召回率指标,即落入正确真值区间的步骤分配数,除以视频中的步骤总数。CrossTask 测试集中的视频已从 HowTo100M 训练集中剔除,以确保训练时不会看到它们。
Text-based video retrieval. We also evaluate our learned embedding on the task of video clip retrieval using natural language queries. Given a textual description, the goal is to retrieve representative video clips from a large pool of videos. We evaluate our learned embedding using the standard recall metrics R@1, R@5, R@10 and the median rank (Median R). We provide experimental results for the following domain-specific video description datasets.
基于文本的视频检索。我们还在用自然语言查询检索视频片段这一任务上评估学到的嵌入。给定一段文字描述,目标是从大量视频中检索出有代表性的片段。我们用标准的召回率指标 R@1、R@5、R@10 以及中位排名(Median R)来评估。下面给出若干领域相关视频描述数据集上的实验结果。
YouCook2 [67] is a cooking video dataset collected from YouTube. It features 89 different recipes and 14k video clips all annotated with textual descriptions collected from paid human workers. Since no descriptions are provided for the test set clips, we evaluate YouCook2 clip retrieval task on the validation clips (3.5k in total). Note that we have taken care to remove the few validation YouCook2 videos that are also present in HowTo100M.
YouCook2 [67] 是一个从 YouTube 收集的烹饪视频数据集,包含 89 种不同食谱和 1.4 万个视频片段,全部配有付费标注者撰写的文字描述。由于测试集片段没有提供描述,我们在验证集片段(共 3500 个)上评估 YouCook2 的片段检索。需要说明的是,我们也已把同时出现在 HowTo100M 中的少数 YouCook2 验证视频去掉。
MSR-VTT [58] is a dataset of generic videos collected from 257 popular video queries depicting 20 categories (including music, sports, movie, etc.) from YouTube. It contains 200k unique video clip-caption pairs, all annotated by paid human workers. We evaluate our model on the MSR-VTT clip retrieval test set used in [63] as performance of several other methods is reported on it.
MSR-VTT [58] 是一个通用视频数据集,来自 YouTube 上 257 个热门检索词,覆盖 20 个类目(包括音乐、体育、电影等)。它包含 20 万对独特的视频片段–描述,全部由付费标注者标注。我们在 [63] 所使用的 MSR-VTT 片段检索测试集上评估模型,因为若干其他方法的结果也是在该测试集上报告的。
LSMDC [44] is a dataset of movie clips. It features 101k unique video clip-caption pairs. All clips are associated with a description that either comes from the movie script or the audio description. We evaluate our model on the official LSMDC test set that contains 1000 video-caption pairs.
LSMDC [44] 是一个电影片段数据集,包含 10.1 万对独特的视频片段–描述。每个片段都配有一条描述,来自电影剧本或音频解说。我们在官方 LSMDC 测试集 上评估模型,该测试集包含 1000 对视频–描述。
5.3. Study of negative pair sampling strategy
We first study the effect of alternative strategies for sampling negative caption-video clip pairs when training our embedding. Table 3 shows that using negatives from the same video (intra-negatives) is beneficial as compared to randomly sampling them from other YouTube videos. The improvement is particularly significant on YouCook2 and CrossTask which are more fine-grained datasets than MSR-VTT and LSMDC. For the rest of the paper, we report numbers using our model trained with the intra-negative sampling strategy.
我们首先考察,在训练嵌入时,用不同策略采样负的描述–视频片段对会有什么影响。表 3 表明,使用来自同一视频的负样本(视频内负样本),比从其他 YouTube 视频中随机采样更有帮助。这一提升在比 MSR-VTT 和 LSMDC 更细粒度的 YouCook2 和 CrossTask 上尤为明显。在后文中,我们报告的都是采用视频内负采样策略训练的模型结果。
| Negative sampling | M (R@10) | L (R@10) | Y (R@10) | C (AVG Recall) |
|---|---|---|---|---|
| No intra-negative | 30.1 | 12.3 | 18.1 | 25.7 |
| With intra-negative | 29.6 | 14.0 | 24.8 | 33.6 |
Table 3: Impact of intra-video negative pairs during training. M: MSR-VTT, L: LSMDC, Y: YouCook2, C: CrossTask.
| 负样本采样 | M(R@10) | L(R@10) | Y(R@10) | C(平均召回) |
|---|---|---|---|---|
| 无视频内负样本 | 30.1 | 12.3 | 18.1 | 25.7 |
| 有视频内负样本 | 29.6 | 14.0 | 24.8 | 33.6 |
表 3:训练时使用视频内负样本对的影响。M:MSR-VTT,L:LSMDC,Y:YouCook2,C:CrossTask。
5.4. Scale matters
A natural question is whether the large scale of our dataset is truly required to achieve high performance. To answer this, we train our embedding model on smaller subsets of our dataset. These smaller subsets of HowTo100M are created by gradually decreasing the allowed Youtube search rank (see the paragraph on data collection in Section 3.1 for more details) for training videos. We experiment with the following rank thresholds: top 2 (15k videos), top 3 (28k videos), top 5 (52k videos), top 10 (104k videos), top 20 (197k videos), top 40 (364k videos), top 80 (648k videos) and top 200 (entire HowTo100M dataset). This process ensures that we subsample training videos that are more likely to be relevant to the queried task as we reduce the size of the training dataset. Figure 3 shows average recall on CrossTask and the R@10 clip retrieval results on LSMDC, MSR-VTT and YouCook2 when varying the size of the training dataset. There is a clear improvement over all evaluated tasks with the gradual increase in the amount of training data. Interestingly, we do not observe any saturation, hence we can expect further improvements by collecting even more readily-available and unlabeled video data.
一个很自然的问题是:要取得较高的性能,是否真的需要数据集这么大的规模。为回答这一点,我们在数据集的更小子集上训练嵌入模型。这些更小的 HowTo100M 子集,是通过逐步降低训练视频所允许的 YouTube 检索排名得到的(详见第 3.1 节关于数据采集的段落)。我们实验了如下排名阈值:前 2 名(1.5 万段视频)、前 3 名(2.8 万段)、前 5 名(5.2 万段)、前 10 名(10.4 万段)、前 20 名(19.7 万段)、前 40 名(36.4 万段)、前 80 名(64.8 万段)以及前 200 名(完整 HowTo100M)。这样做可以保证,在缩小训练集时,留下来的训练视频仍然更可能与查询任务相关。图 3 给出了在改变训练数据规模时,CrossTask 上的平均召回,以及 LSMDC、MSR-VTT 和 YouCook2 上的 R@10 片段检索结果。随着训练数据逐渐增多,所有评估任务都有明显提升。有意思的是,我们没有观察到饱和,因此可以预期,再收集更多现成、未标注的视频数据,还有进一步提升的空间。
图 3:改变 HowTo100M 训练集规模时的检索与步骤定位结果。
5.5. Comparison with state-of-the-art
CrossTask. We compare our off-the-shelf embedding trained on HowTo100M against methods proposed by Alayrac et al. [2] and Zhukov et al. [68] which is the current state-of-the-art on CrossTask for weakly supervised methods. Note that Zhukov et al. [68] have access to the ordered list of action labels at the task level and narrations are the only form of supervision during training. We also report the fully-supervised upper-bound from [68] obtained with a model that has been trained on action segments with ground truth annotation. The results are shown in Table 4. Our approach significantly outperforms the state-of-the-art, even though it has not been specifically designed for the task of step localization in videos. The improvement made by our method is consistent across all tasks (with the exception of Make Meringue), showing that the trained model is not biased towards any specific domain. The recall is above 30% for most tasks with the significant improvement observed for the "Add Oil to a Car" task (6.4% to 30.7% boost in recall). Note that our method also outperforms the fully-supervised upper bound [68] on average. Thus, we conclude that training on a large amount of narrated videos is better than training a step localization model on a small but carefully annotated training set.
CrossTask。我们把仅在 HowTo100M 上训练、可直接使用的嵌入,与 Alayrac 等人 [2] 以及 Zhukov 等人 [68] 的方法进行比较;后者是当时 CrossTask 上弱监督方法中的最好结果。需要说明,Zhukov 等人 [68] 在任务层面可以使用有序的动作标签列表,训练时的监督只来自解说。我们还报告了 [68] 中的全监督上界,该上界来自在带真值标注的动作片段上训练的模型。结果见表 4。我们的方法显著超过了当时最好水平,尽管它并不是专门为视频中的步骤定位设计的。除「做蛋白糖霜」(Make Meringue)外,这一提升在所有任务上都一致,说明训练出的模型并没有偏向某个特定领域。大多数任务上的召回率都高于 30%,其中「给汽车加油」(Add Oil to a Car)的提升最为明显(召回率从 6.4% 提高到 30.7%)。还应注意,我们的方法在平均值上也超过了全监督上界 [68]。因此我们得出结论:在大量配解说的视频上训练,要好过在一个虽经仔细标注、但规模较小的训练集上训练步骤定位模型。
| Make Kimchi Rice | Pickle Cucumber | Make Banana Ice Cream | Grill Steak | Jack Up Car | Make Jello Shots | Change Tire | Make Lemonade | Add Oil to Car | Make Latte | Build Shelves | Make Taco Salad | Make French Toast | Make Irish Coffee | Make Strawberry Cake | Make Pancakes | Make Meringue | Make Fish Curry | Average | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Fully-supervised upper-bound [68] | 19.1 | 25.3 | 38.0 | 37.5 | 25.7 | 28.2 | 54.3 | 25.8 | 18.3 | 31.2 | 47.7 | 12.0 | 39.5 | 23.4 | 30.9 | 41.1 | 53.4 | 17.3 | 31.6 |
| Alayrac et al. [2] | 15.6 | 10.6 | 7.5 | 14.2 | 9.3 | 11.8 | 17.3 | 13.1 | 6.4 | 12.9 | 27.2 | 9.2 | 15.7 | 8.6 | 16.3 | 13.0 | 23.2 | 7.4 | 13.3 |
| Zhukov et al. [68] | 13.3 | 18.0 | 23.4 | 23.1 | 16.9 | 16.5 | 30.7 | 21.6 | 4.6 | 19.5 | 35.3 | 10.0 | 32.3 | 13.8 | 29.5 | 37.6 | 43.0 | 13.3 | 22.4 |
| Ours trained on HowTo100M only | 33.5 | 27.1 | 36.6 | 37.9 | 24.1 | 35.6 | 32.7 | 35.1 | 30.7 | 28.5 | 43.2 | 19.8 | 34.7 | 33.6 | 40.4 | 41.6 | 41.9 | 27.4 | 33.6 |
Table 4: Step localization results on CrossTask [68] instructional video dataset.
| Make Kimchi Rice | Pickle Cucumber | Make Banana Ice Cream | Grill Steak | Jack Up Car | Make Jello Shots | Change Tire | Make Lemonade | Add Oil to Car | Make Latte | Build Shelves | Make Taco Salad | Make French Toast | Make Irish Coffee | Make Strawberry Cake | Make Pancakes | Make Meringue | Make Fish Curry | 平均 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 全监督上界 [68] | 19.1 | 25.3 | 38.0 | 37.5 | 25.7 | 28.2 | 54.3 | 25.8 | 18.3 | 31.2 | 47.7 | 12.0 | 39.5 | 23.4 | 30.9 | 41.1 | 53.4 | 17.3 | 31.6 |
| Alayrac et al. [2] | 15.6 | 10.6 | 7.5 | 14.2 | 9.3 | 11.8 | 17.3 | 13.1 | 6.4 | 12.9 | 27.2 | 9.2 | 15.7 | 8.6 | 16.3 | 13.0 | 23.2 | 7.4 | 13.3 |
| Zhukov et al. [68] | 13.3 | 18.0 | 23.4 | 23.1 | 16.9 | 16.5 | 30.7 | 21.6 | 4.6 | 19.5 | 35.3 | 10.0 | 32.3 | 13.8 | 29.5 | 37.6 | 43.0 | 13.3 | 22.4 |
| 本文(仅在 HowTo100M 上训练) | 33.5 | 27.1 | 36.6 | 37.9 | 24.1 | 35.6 | 32.7 | 35.1 | 30.7 | 28.5 | 43.2 | 19.8 | 34.7 | 33.6 | 40.4 | 41.6 | 41.9 | 27.4 | 33.6 |
表 4:CrossTask [68] 教学视频数据集上的步骤定位结果。
YouCook2 [67] does not provide an official benchmark nor any reported number for clip retrieval. As a consequence, we have applied a state-of-the-art text-video embedding model from Klein et al. [25] (HGLMM FV CCA) on YouCook2 using our features. We also report results of our model trained on YouCook2 instead of HowTo100M in Table 5. First, we notice that our off-the-shelf model trained on HowTo100M significantly outperforms both the exact same model directly trained on YouCook2 and [25]. Furthermore, fine-tuning our model pre-trained on HowTo100M on YouCook2 results in a significant improvement of 13.7 % in R@10 against [25]. In conclusion, we show that the off-the-shelf HowTo100M trained model can outperform state-of-the-art on this domain specific instructional video dataset. Moreover, we demonstrate that our model can get further benefits from fine-tuning.
YouCook2 [67] 没有提供官方的片段检索基准,也没有已报告的数字。因此,我们用自己的特征,在 YouCook2 上运行了 Klein 等人 [25] 的当时最好文本–视频嵌入模型(HGLMM FV CCA)。表 5 中我们也报告了不在 HowTo100M、而在 YouCook2 上训练的本模型结果。首先可以看到,仅在 HowTo100M 上训练、可直接使用的模型,显著超过了在 YouCook2 上直接训练的同一模型,也超过了 [25]。此外,把在 HowTo100M 上预训练的模型再在 YouCook2 上微调,R@10 相对 [25] 有 13.7 个百分点的明显提升。总之,我们表明,仅在 HowTo100M 上训练的现成模型,就能在这一领域相关的教学视频数据集上超过当时最好水平。同时,微调还能给模型带来进一步的好处。
| Method | Trainset | R@1 | R@5 | R@10 | Median R |
|---|---|---|---|---|---|
| Random | None | 0.03 | 0.15 | 0.3 | 1675 |
| HGLMM FV CCA [25] | YouCook2 | 4.6 | 14.3 | 21.6 | 75 |
| Ours | YouCook2 | 4.2 | 13.7 | 21.5 | 65 |
| Ours | HowTo100M | 6.1 | 17.3 | 24.8 | 46 |
| Ours | PT: HowTo100MFT: YouCook2 | 8.2 | 24.5 | 35.3 | 24 |
Table 5: YouCook2 clip retrieval results. PT denotes: pre-trained, while FT denotes: fine-tuned.
| 方法 | 训练集 | R@1 | R@5 | R@10 | 中位排名 |
|---|---|---|---|---|---|
| 随机 | 无 | 0.03 | 0.15 | 0.3 | 1675 |
| HGLMM FV CCA [25] | YouCook2 | 4.6 | 14.3 | 21.6 | 75 |
| 本文 | YouCook2 | 4.2 | 13.7 | 21.5 | 65 |
| 本文 | HowTo100M | 6.1 | 17.3 | 24.8 | 46 |
| 本文 | 预训练:HowTo100M微调:YouCook2 | 8.2 | 24.5 | 35.3 | 24 |
表 5:YouCook2 片段检索结果。PT 表示预训练,FT 表示微调。
MSR-VTT. We compare our model trained on (i) HowTo100M only, (ii) MSR-VTT only and (iii) pre-trained on HowTo100M and then fine-tuned on MSR-VTT against prior work that directly uses MSR-VTT for training (reproduced in [63]) in Table 6. Our off-the-shelf HowTo100M model outperforms [22, 24, 53, 64, 65] that are directly trained on MSR-VTT. Here again, after fine-tuning the HowTo100M pre-trained model on MSR-VTT, we observe a significant improvement over the state-of-the-art JSFusion [63] trained on MSR-VTT. However, as opposed to instructional videos (CrossTask) and cooking videos (YouCook2), training our model directly on MSR-VTT performs better than our off-the-shelf model trained on HowTo100M. We believe this is due to MSR-VTT videos being generic Youtube videos that are different from the instructional or VLOG type of videos that dominate HowTo100M. In Figure 4, we also investigate the impact on performance at various amounts of supervision when fine-tuning our pre-trained model. It shows that state-of-the-art performance [63] can be attained with only 20% of MSR-VTT samples. This has great practical implications as comparable performance can be obtained using significantly reduced annotation.
MSR-VTT。在表 6 中,我们把三种训练方式下的本模型——(i) 只在 HowTo100M 上训练,(ii) 只在 MSR-VTT 上训练,(iii) 先在 HowTo100M 上预训练再在 MSR-VTT 上微调——与直接用 MSR-VTT 训练的已有工作(结果转引自 [63])进行比较。仅在 HowTo100M 上训练的现成模型,已经超过了直接在 MSR-VTT 上训练的 [22, 24, 53, 64, 65]。同样地,把 HowTo100M 预训练模型在 MSR-VTT 上微调后,相对直接在 MSR-VTT 上训练的当时最好方法 JSFusion [63],我们又看到了明显提升。不过,与教学视频(CrossTask)和烹饪视频(YouCook2)不同,在 MSR-VTT 上直接训练本模型,要好过只在 HowTo100M 上训练的现成模型。我们认为,这是因为 MSR-VTT 是通用的 YouTube 视频,与在 HowTo100M 中占主导的教学或 Vlog 类视频并不相同。在图 4 中,我们还考察了微调预训练模型时,不同监督数量对性能的影响。结果显示,只用 MSR-VTT 中 20% 的样本,就能达到当时最好水平 [63]。这一点很有实际意义,因为用显著更少的标注就能得到相当的性能。
| Method | Trainset | R@1 | R@5 | R@10 | Median R |
|---|---|---|---|---|---|
| Random | None | 0.1 | 0.5 | 1.0 | 500 |
| C+LSTM+SA+FC7 [53] | MSR-VTT | 4.2 | 12.9 | 19.9 | 55 |
| VSE-LSTM [24] | MSR-VTT | 3.8 | 12.7 | 17.1 | 66 |
| SNUVL [64] | MSR-VTT | 3.5 | 15.9 | 23.8 | 44 |
| Kaufman et al. [22] | MSR-VTT | 4.7 | 16.6 | 24.1 | 41 |
| CT-SAN [65] | MSR-VTT | 4.4 | 16.6 | 22.3 | 35 |
| JSFusion [63] | MSR-VTT | 10.2 | 31.2 | 43.2 | 13 |
| Ours | HowTo100M | 7.5 | 21.2 | 29.6 | 38 |
| Ours | MSR-VTT | 12.1 | 35.0 | 48.0 | 12 |
| Ours | PT: HowTo100MFT: MSR-VTT | 14.9 | 40.2 | 52.8 | 9 |
Table 6: MSR-VTT clip retrieval results. PT denotes: pre-trained, while FT denotes: fine-tuned.
| 方法 | 训练集 | R@1 | R@5 | R@10 | 中位排名 |
|---|---|---|---|---|---|
| 随机 | 无 | 0.1 | 0.5 | 1.0 | 500 |
| C+LSTM+SA+FC7 [53] | MSR-VTT | 4.2 | 12.9 | 19.9 | 55 |
| VSE-LSTM [24] | MSR-VTT | 3.8 | 12.7 | 17.1 | 66 |
| SNUVL [64] | MSR-VTT | 3.5 | 15.9 | 23.8 | 44 |
| Kaufman et al. [22] | MSR-VTT | 4.7 | 16.6 | 24.1 | 41 |
| CT-SAN [65] | MSR-VTT | 4.4 | 16.6 | 22.3 | 35 |
| JSFusion [63] | MSR-VTT | 10.2 | 31.2 | 43.2 | 13 |
| 本文 | HowTo100M | 7.5 | 21.2 | 29.6 | 38 |
| 本文 | MSR-VTT | 12.1 | 35.0 | 48.0 | 12 |
| 本文 | 预训练:HowTo100M微调:MSR-VTT | 14.9 | 40.2 | 52.8 | 9 |
表 6:MSR-VTT 片段检索结果。PT 表示预训练,FT 表示微调。
图 4:用不同数量的 MSR-VTT 监督,微调 HowTo100M 预训练模型,并评估文本到视频片段检索。
LSMDC. Finally, we compare to state-of-the-art on LSMDC in Table 7. This dataset is even more challenging as movie clips are quite distinct from HowTo100M videos. We compare against several other prior works that have been reproduced in [63] and are trained directly on LSMDC. Here again, we see that pre-training our model on HowTo100M and fine-tuning it on LSMDC also provides improvements upon a model directly trained on LSMDC. This finding is interesting and shows that a HowTo100M pre-trained model can still be useful when fine-tuned on videos from a different domain.
LSMDC。最后,我们在表 7 中与 LSMDC 上的当时最好方法进行比较。这一数据集更具挑战性,因为电影片段与 HowTo100M 中的视频差异相当大。比较对象是 [63] 中复现、并直接在 LSMDC 上训练的若干已有工作。同样地,我们看到,先在 HowTo100M 上预训练再在 LSMDC 上微调,也比直接在 LSMDC 上训练的模型更好。这一发现很有意思,它说明 HowTo100M 预训练模型在不同领域的视频上微调后,仍然可以发挥作用。
| Method | Trainset | R@1 | R@5 | R@10 | Median R |
|---|---|---|---|---|---|
| Random | None | 0.1 | 0.5 | 1.0 | 500 |
| C+LSTM+SA+FC7 [53] | LSMDC | 4.3 | 12.6 | 18.9 | 98 |
| VSE-LSTM [24] | LSMDC | 3.1 | 10.4 | 16.5 | 79 |
| SNUVL [64] | LSMDC | 3.6 | 14.7 | 23.9 | 50 |
| Kaufman et al. [22] | LSMDC | 4.7 | 15.9 | 23.4 | 64 |
| CT-SAN [65] | LSMDC | 4.5 | 14.1 | 20.9 | 67 |
| JSFusion [63] | LSMDC | 9.1 | 21.2 | 34.1 | 36 |
| Ours | HowTo100M | 4.0 | 9.8 | 14.0 | 137 |
| Ours | LSMDC | 7.2 | 18.3 | 25.0 | 44 |
| Ours | PT: HowTo100MFT: LSMDC | 7.1 | 19.6 | 27.9 | 40 |
Table 7: LSMDC clip retrieval results. PT denotes: pre-trained, while FT denotes: fine-tuned.
| 方法 | 训练集 | R@1 | R@5 | R@10 | 中位排名 |
|---|---|---|---|---|---|
| 随机 | 无 | 0.1 | 0.5 | 1.0 | 500 |
| C+LSTM+SA+FC7 [53] | LSMDC | 4.3 | 12.6 | 18.9 | 98 |
| VSE-LSTM [24] | LSMDC | 3.1 | 10.4 | 16.5 | 79 |
| SNUVL [64] | LSMDC | 3.6 | 14.7 | 23.9 | 50 |
| Kaufman et al. [22] | LSMDC | 4.7 | 15.9 | 23.4 | 64 |
| CT-SAN [65] | LSMDC | 4.5 | 14.1 | 20.9 | 67 |
| JSFusion [63] | LSMDC | 9.1 | 21.2 | 34.1 | 36 |
| 本文 | HowTo100M | 4.0 | 9.8 | 14.0 | 137 |
| 本文 | LSMDC | 7.2 | 18.3 | 25.0 | 44 |
| 本文 | 预训练:HowTo100M微调:LSMDC | 7.1 | 19.6 | 27.9 | 40 |
表 7:LSMDC 片段检索结果。PT 表示预训练,FT 表示微调。
5.6. Cross-dataset fine-tuning evaluation
In this section, we evaluate the advantage of HowTo100M for pre-training compared to pre-training on other smaller datasets. Figure 5 shows evaluation on YouCook2, MSR-VTT and LSMDC clip retrieval (R@10) using no pre-training (No PT), using pre-training on YouCook2, MSR-VTT, LSMDC and HowTo100M datasets while fine-tuning to the target dataset. For all evaluated datasets, pre-training on HowTo100M prior to fine-tuning on the target dataset consistently yields best results.
本节评估 HowTo100M 用于预训练,相比在其他更小数据集上预训练,有怎样的优势。图 5 给出了在 YouCook2、MSR-VTT 和 LSMDC 片段检索(R@10)上的结果,设定包括不做预训练(No PT),以及分别在 YouCook2、MSR-VTT、LSMDC 和 HowTo100M 上预训练后再微调到目标数据集。在所有评估的数据集上,先在 HowTo100M 上预训练再微调到目标数据集,都一致地取得最好结果。
图 5:在不同数据集上预训练后再做片段检索的结果。评估数据集为 LSMDC、YouCook2 和 MSR-VTT。
5.7. Qualitative results
图 6:用训练好的联合嵌入,在 HowTo100M 上做视频片段检索的示例。
Figure 6 illustrates examples of retrieved video clips from HowTo100M using our trained joint text-video embedding. For example, our learned representation can correctly distinguish between queries Cut paper and Cut wood. A demo of the retrieval system is available online [1].
图 6 展示了用训练好的文本–视频联合嵌入,从 HowTo100M 中检索到的视频片段示例。例如,学到的表示能够正确区分 Cut paper(裁纸)和 Cut wood(锯木头)这两种查询。检索系统的演示可在网上获得 [1]。
- Conclusion
We have introduced HowTo100M, a video dataset with more than 130M video clips, extracted from 1.2M narrated web videos of people performing complex visual tasks. Our data collection method is fast, scalable and does not require any manual annotation. We use this dataset to learn a joint text-video embedding by leveraging more than 130M video clip-caption pairs. We have shown through various experiments that our learned embedding can perform better compared to models trained on existing carefully annotated but smaller video description datasets. The dataset, pre-trained models and code are available at [1].
我们发布了 HowTo100M,这是一个包含一亿三千多万个视频片段的数据集,这些片段提取自 120 万段配有解说、展示人们完成复杂视觉任务的网络视频。采集方法快速、可扩展,也不需要任何人工标注。我们利用其中一亿三千多万对视频片段–描述,来学习文本–视频联合嵌入。通过多项实验,我们表明学到的嵌入可以超过在现有虽经仔细标注、但规模较小的视频描述数据集上训练的模型。数据集、预训练模型和代码见 [1]。
Acknowledgements
致谢
The project was partially supported by Antoine Miech Google PhD fellowship, the MSR-Inria joint lab, the Louis Vuitton - ENS Chair on Artificial Intelligence, the ERC grant LEAP (No. 336845), the CIFAR Learning in Machines&Brains program, and the European Regional Development Fund under the project IMPACT (reg. no. CZ.02.1.01/0.0/0.0/15 003/0000468).
本项目部分得到 Antoine Miech 的 Google 博士奖学金、MSR-Inria 联合实验室、Louis Vuitton - ENS 人工智能讲席、ERC 项目 LEAP(编号 336845)、CIFAR Learning in Machines&Brains 项目,以及欧洲区域发展基金 IMPACT 项目(登记号 CZ.02.1.01/0.0/0.0/15 003/0000468)的资助。
References
参考文献
[1] Project webpage. https://www.di.ens.fr/willow/research/howto100m/, 2019. 1, 8
[2] Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Ivan Laptev, Josef Sivic, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In CVPR, 2016. 2, 6, 7
[3] Jean-Baptiste Alayrac, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Joint discovery of object states and manipulation actions. In ICCV, 2017. 2
[4] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. 5
[5] Kai Chen, Hang Song, Chen Change Loy, and Dahua Lin. Discover and learn new objects from documentaries. In CVPR, 2017. 2
[6] Mithun Chowdhury, Panda Rameswar, Evangelos Papalexakis, and Amit Roy-Chowdhury. Webly supervised joint embedding for cross-modal image-text retrieval. In ACM International Conference on Multimedia, 2018. 2
[7] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018. 2
[8] Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. Dual encoding for zero-example video retrieval. In CVPR, 2019. 2
[9] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In EMNLP, pages 457–468, 2016. 2
[10] Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. A multi-view embedding space for modeling internet images, tags, and their semantics. IJCV, 2014. 2
[11] Yunchao Gong, Liwei Wang, Micah Hodosh, Julia Hockenmaier, and Svetlana Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In ECCV, 2014. 2
[12] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In CVPR, 2018. 5
[13] David Harwath, Adrià Recasens, Dídac Surís, Galen Chuang, Antonio Torralba, and James Glass. Jointly discovering visual objects and spoken words from raw sensory input. In ECCV, 2018. 2
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016. 5
[15] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. ICCV, 2017. 1, 2, 5, 12
[16] De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Connectionist temporal modeling for weakly supervised action labeling. In ECCV, 2016. 2
[17] De-An Huang, Joseph J. Lim, Li Fei-Fei, and Juan Carlos Niebles. Unsupervised visual-linguistic reference resolution in instructional videos. In CVPR, 2017. 2
[18] De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. Finding "it": Weakly-supervised reference-aware visual grounding in instructional video. In CVPR, 2018. 2
[19] Kenton Lee Jacob Devlin, Ming-Wei Chang and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. preprint, 2018. 3
[20] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR, 2016. 2
[21] Andrej Karpathy, Armand Joulin, and Fei Fei F Li. Deep fragment embeddings for bidirectional image sentence mapping. In NIPS, 2014. 5
[22] Dotan Kauman, Gil Levi, Tal Hassner, and Lior Wolf. Temporal tessellation: A unified approach for video analysis. In ICCV, 2017. 7, 8
[23] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 5
[24] Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. TACL, 2014. 7, 8
[25] Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Associating neural word embeddings with deep image representations using fisher vectors. In CVPR, 2015. 1, 2, 7
[26] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, 2017. 2
[27] Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo. TGIF: A New Dataset and Benchmark on Animated GIF Description. In CVPR, 2016. 2
[28] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018. 3
[29] Mateusz Malinowski, Marcus Rohrbach, and Mario Fritz. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 2015. 2
[30] Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Murphy. What's cookin'? interpreting cooking videos using text, speech and vision. NAACL, 2015. 2
[31] Antoine Miech, Ivan Laptev, and Josef Sivic. Learnable pooling with context gating for video classification. arXiv preprint arXiv:1706.06905, 2017. 5
[32] Antoine Miech, Ivan Laptev, and Josef Sivic. Learning a Text-Video Embedding from Incomplete and Heterogeneous Data. arXiv:1804.02516, 2018. 1, 2, 4, 5
[33] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100M: Learning a text-video embedding by watching hundred million narrated video clips. arXiv preprint arXiv:1906.03327, 2019. 4, 5
[34] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. 5
[35] Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. Learning joint embedding with multimodal cues for cross-modal video-text retrieval. In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval. ACM, 2018. 2
[36] Pingbo Pan, Zhongwen Xu, Yi Yang, Fei Wu, and Yueting Zhuang. Hierarchical recurrent neural encoder for video representation with application to captioning. In CVPR, pages 1029–1038, 2016. 1, 2
[37] Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR, 2016. 1, 2
[38] Bryan A Plummer, Matthew Brown, and Svetlana Lazebnik. Enhancing video summarization via vision-language embedding. In CVPR, 2017. 1, 2
[39] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understandingby Generative Pre-Training. preprint, 2018. 3
[40] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. preprint, 2019. 3
[41] Alexander Richard, Hilde Kuehne, and Juergen Gall. Weakly supervised action learning with rnn based fine-to-coarse modeling. In CVPR, 2017. 2
[42] Alexander Richard, Hilde Kuehne, and Juergen Gall. Action sets: Weakly supervised action segmentation without ordering constraints. In CVPR, 2018. 2
[43] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In CVPR, 2015. 2
[44] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. IJCV, 2017. 2, 5, 6
[45] Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze. How2: a large-scale dataset for multimodal language understanding. In Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL). NeurIPS, 2018. 2
[46] Fadime Sener and Angela Yao. Unsupervised learning and segmentation of complex activities from video. In CVPR, 2018. 2
[47] Ozan Sener, Amir R. Zamir, Silvio Savarese, and Ashutosh Saxena. Unsupervised semantic parsing of video collections. In The IEEE International Conference on Computer Vision (ICCV), December 2015. 2
[48] Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, 2016. 2
[49] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017. 3
[50] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019. 2
[51] Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through questionanswering. In CVPR, 2016. 1, 2
[52] Atousa Torabi, Christopher Pal, Hugo Larochelle, and Aaron Courville. Using descriptive video services to create a large data source for video annotation research. arXiv preprint arXiv:1503.01070, 2015. 2
[53] Atousa Torabi, Niket Tandon, and Leonid Sigal. Learning language-visual embedding for movie understanding with natural-language. arXiv preprint arXiv:1609.08124, 2016. 7, 8
[54] Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text matching tasks. PAMI, 2018. 1, 2, 5
[55] Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings. In CVPR, pages 5005–5013, 2016. 1, 2, 5
[56] Xin Wang, Jiawei Wu, Da Zhang, Yu Su, and William Yang Wang. Learning to compose topic-aware mixture of experts for zero-shot video captioning. In AAAI, 2018. 2
[57] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. ICCV, 2017. 2
[58] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. 1, 2, 5, 6
[59] Ran Xu, Caiming Xiong, Wei Chen, and Jason J Corso. Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In AAAI, volume 5, page 6, 2015. 1, 2
[60] Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In CVPR, pages 4651–4659, 2016. 2
[61] Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. In CVPR, pages 4584–4593, 2016. 1, 2
[62] Shoou-I Yu, Lu Jiang, and Alexander Hauptmann. Instructional videos for unsupervised harvesting and learning of action examples. In ACM, 2014. 2
[63] Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In ECCV, 2018. 1, 2, 6, 7, 8
[64] Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. Video captioning and retrieval models with semantic attention. In ECCV LSMDC2016 Workshop, 2016. 5, 7, 8
[65] Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. End-to-end concept word detection for video captioning, retrieval, and question answering. In CVPR, 2017. 7, 8
[66] Luowei Zhou, Xu Chenliang, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018. 2
[67] Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018. 2, 5, 6, 7
[68] Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Crosstask weakly supervised learning from instructional videos. In CVPR, 2019. 2, 5, 6, 7





