Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

看、注意、讲述：带视觉注意力的神经图像描述生成（Show, Attend and Tell: Neural Image Caption Generation with Visual Attention）

Kelvin Xu kelvin.xu@umontreal.ca

Jimmy Lei Ba jimmy@psi.utoronto.ca

Ryan Kiros rkiros@cs.toronto.edu

Kyunghyun Cho kyunghyun.cho@umontreal.ca

Aaron Courville aaron.courville@umontreal.ca

Ruslan Salakhutdinov rsalakhu@cs.toronto.edu

Richard S. Zemel zemel@cs.toronto.edu

Yoshua Bengio find-me@the.web

Abstract

摘要

Inspired by recent work in machine translation and object detection, we introduce an attention based model that automatically learns to describe the content of images. We describe how we can train this model in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound. We also show through visualization how the model is able to automatically learn to fix its gaze on salient objects while generating the corresponding words in the output sequence. We validate the use of attention with state-of-the-art performance on three benchmark datasets: Flickr8k, Flickr30k and MS COCO.

受机器翻译与目标检测近期工作启发，我们引入一种基于注意力的模型，能自动学习描述图像内容。我们说明如何用标准反向传播以确定性方式训练该模型，以及如何通过最大化变分下界以随机方式训练。我们还展示如何在生成描述的过程中可视化注意力的“注视”位置。在 Flickr8k、Flickr30k 与 MS COCO 三个基准数据集上，我们用 BLEU 与 METEOR 指标达到当时最好结果。对注意力的定性分析表明，所学对齐与人类直觉高度吻合。

1 Introduction

1 引言

Automatically generating captions of an image is a task very close to the heart of scene understanding — one of the primary goals of computer vision. Not only must caption generation models be powerful enough to solve the computer vision challenges of determining which objects are in an image, but they must also be capable of capturing and expressing their relationships in a natural language. For this reason, caption generation has long been viewed as a difficult problem. It is a very important challenge for machine learning algorithms, as it amounts to mimicking the remarkable human ability to compress huge amounts of salient visual infomation into descriptive language.

自动生成图像的字幕是一项非常接近场景理解核心的任务——计算机视觉的主要目标之一. 不仅必须使字幕生成模型 我们也能够用自然语言捕捉和表达它们的关系。 因此,标题生成长期以来一直被视为一个困难的问题。 这对机器学习算法来说是一个非常重要的挑战, 如果是模仿人类将大量突出的视觉信息压缩成描述性语言的显著能力。

Despite the challenging nature of this task, there has been a recent surge of research interest in attacking the image caption generation problem. Aided by advances in training neural networks (Krizhevsky et al., 2012) and large classification datasets (Russakovsky et al., 2014), recent work has significantly improved the quality of caption generation using a combination of convolutional neural networks (convnets) to obtain vectorial representation of images and recurrent neural networks to decode those representations into natural language sentences (see Sec. 2).

尽管这一任务很有挑战性，最近仍出现大量研究兴趣。借助深度神经网络（卷积网络与循环网络）的进展，以及用于训练它们的大型分类数据集（Krizhevsky 等, 2012；Ilya Sutskever & Vinyals, 2014），模型现在可以不必再严重依赖复杂的流水线与传统的计算机视觉与自然语言处理技术。

Figure 1: Our model learns a words/image alignment. The visualized attentional maps (3) are explained in section 3.1 & 5.4

图 1：我们的模型学习词与图像的对齐。可视化的注意力图（3）在第 3.1 与 5.4 节解释。

Figure 2: Attention over time. As the model generates each word, its attention changes to reflect the relevant parts of the image. “soft” (top row) vs “hard” (bottom row) attention. (Note that both models generated the same captions in this example.)

图 2：注意力随时间变化。模型每生成一个词，注意力就转向图像中相关的部分。“软”注意力（上行）与“硬”注意力（下行）。（本例中两个模型生成了相同的描述。）

Figure 3: Examples of attending to the correct object (white indicates the attended regions, underlines indicated the corresponding word)

图 3：关注到正确物体的例子（白色表示被关注区域，下划线标出对应的词）

One of the most curious facets of the human visual system is the presence of attention (Rensink, 2000; Corbetta & Shulman, 2002). Rather than compress an entire image into a static representation, attention allows for salient features to dynamically come to the forefront as needed. This is especially important when there is a lot of clutter in an image. Using representations (such as those from the top layer of a convnet) that distill information in image down to the most salient objects is one effective solution that has been widely adopted in previous work. Unfortunately, this has one potential drawback of losing information which could be useful for richer, more descriptive captions. Using more low-level representation can help preserve this information. However working with these features necessitates a powerful mechanism to steer the model to information important to the task at hand.

人类视觉系统最令人好奇的一面是注意力的存在（Rensink, 2000；Corbetta & Shulman, 2002）。注意力并不把整幅图像压成静态表示，而是让显著特征按需动态地走到前台。图像里杂乱物体很多时，这一点尤其重要。用把图像信息蒸馏到最显著物体上的表示（例如卷积网络顶层表示）是先前工作广泛采用的有效做法。但这样做的一个缺点是：一旦信息在这一步被丢掉，后面就无法恢复。

In this paper, we describe approaches to caption generation that attempt to incorporate a form of attention with two variants: a “hard” attention mechanism and a “soft” attention mechanism. We also show how one advantage of including attention is the ability to visualize what the model “sees”. Encouraged by recent advances in caption generation and inspired by recent success in employing attention in machine translation (Bahdanau et al., 2014) and object recognition (Ba et al., 2014; Mnih et al., 2014), we investigate models that can attend to salient part of an image while generating its caption.

本文描述把注意力纳入描述生成的方法，包含两种变体：“硬”注意力机制与“软”注意力机制。我们还说明，加入注意力的一个好处是能够可视化模型“看到”了什么。在描述生成与机器翻译的近期进展鼓励下，我们给出一种可学习的对齐框架，把图像区域与生成词对应起来。

The contributions of this paper are the following:

本文件的贡献如下:

• • We introduce two attention-based image caption generators under a common framework (Sec. 3.1): 1) a “soft” deterministic attention mechanism trainable by standard back-propagation methods and 2) a “hard” stochastic attention mechanism trainable by maximizing an approximate variational lower bound or equivalently by REINFORCE (Williams, 1992).

• • 我们在同一框架下引入两种基于注意力的图像描述生成器（第 3.1 节）：1）可用标准反向传播训练的“软”确定性注意力机制；2）可通过最大化近似变分下界、或等价地用 REINFORCE（Williams, 1992）训练的“硬”随机注意力机制。

• • We show how we can gain insight and interpret the results of this framework by visualizing “where” and “what” the attention focused on. (see Sec. 5.4)

• • 我们说明如何可视化模型在生成每个词时关注的图像区域，并表明所学对齐与人类直觉高度一致（第 5.4 节）。

• • Finally, we quantitatively validate the usefulness of attention in caption generation with state of the art performance (Sec. 5.3) on three benchmark datasets: Flickr8k (Hodosh et al., 2013) , Flickr30k (Young et al., 2014) and the MS COCO dataset (Lin et al., 2014).

• • 我们在 Flickr8k、Flickr30k 与 MS COCO 上取得当时最好的 BLEU / METEOR 结果，且使用单模型、不做集成（第 5.3 节）。

2 Related Work

2 相关工作

In this section we provide relevant background on previous work on image caption generation and attention. Recently, several methods have been proposed for generating image descriptions. Many of these methods are based on recurrent neural networks and inspired by the successful use of sequence to sequence training with neural networks for machine translation (Cho et al., 2014; Bahdanau et al., 2014; Sutskever et al., 2014). One major reason image caption generation is well suited to the encoder-decoder framework (Cho et al., 2014) of machine translation is because it is analogous to “translating” an image to a sentence.

在本节中,我们提供了以往关于图像标题生成和关注工作的相关背景。 最近,提出了几种生成图像描述的方法. 许多这些方法都基于反复出现的神经网络,并受到成功及早利用序列与神经网络进行序列培训的启发,用于机器翻译(C 这不是 51.Cho等人,2014年;Bahdanau等人,2014年;Sutskever等人,2014年)。 一个ẫn 原因 的某些方面 图像标题生成非常适合机器翻译的编码器-解码器框架(Cho等人,2014年),因为它类似于将图像“转换”成 你自己的句子。

The first approach to use neural networks for caption generation was Kiros et al. (2014a), who proposed a multimodal log-bilinear model that was biased by features from the image. This work was later followed by Kiros et al. (2014b) whose method was designed to explicitly allow a natural way of doing both ranking and generation. Mao et al. (2014) took a similar approach to generation but replaced a feed-forward neural language model with a recurrent one. Both Vinyals et al. (2014) and Donahue et al. (2014) use LSTM RNNs for their models. Unlike Kiros et al. (2014a) and Mao et al. (2014) whose models see the image at each time step of the output word sequence, Vinyals et al. (2014) only show the image to the RNN at the beginning. Along with images, Donahue et al. (2014) also apply LSTMs to videos, allowing their model to generate video descriptions.

Kiros等人(2014年a)是使用神经网络生成标题的第一个方法提倡采用城里图像特征偏颇的多式对数比线模型行为的受害者。 这部作品后来被正在为明确允许自然的排名和生成本来就允许的方法的基罗斯等人(2014年b)所遵循. Mao等人(2014年)对生成采取了类似的做法,但用反复出现的神经语言模型巷道取代了向导神经语言模型. Vinyals等人(2014年)和Donahue等人(2014年)都使用LSTM RNNs作为模型. 事实上,与基罗斯等人(2014年a)和毛泽东等人(2014年ẩn)不同的是,他们的模型在输出词序列的每个步骤都能看到图像,Vinyals等人(2014年)只在开头向RNN显示图像. Donahue等人(2014年)也将LSTMs应用于视频,使其模型能够生成视频描述.

All of these works represent images as a single feature vector from the top layer of a pre-trained convolutional network. Karpathy & Li (2014) instead proposed to learn a joint embedding space for ranking and generation whose model learns to score sentence and image similarity as a function of R-CNN object detections with outputs of a bidirectional RNN. Fang et al. (2014) proposed a three-step pipeline for generation by incorporating object detections. Their model first learn detectors for several visual concepts based on a multi-instance learning framework. A language model trained on captions was then applied to the detector outputs, followed by rescoring from a joint image-text embedding space. Unlike these models, our proposed attention framework does not explicitly use object detectors but instead learns latent alignments from scratch. This allows our model to go beyond “objectness” and learn to attend to abstract concepts.

所有这些作品都代表着图像作为来自一个预先株式会方面取得了重大进展的演聆网络顶层的单一特征矢量. Karpathy & Li (2014) 转而提议学习一个排名正在开展和生成的联合嵌入空间,其模型学习nake 句子和图像相似度作为R-CNN对象检测功能,其输出为双向RNN.踊跃向Fang等人(2014年)提议采用三步式管道,通过安装物体探测器进行生成。 他们的模型首先撬动基于多层学习框架的几种视觉概念的探测器. 然后,呱嬷在探测器输出才能够使用经过字幕培训的语言模型,然后从联合图像文本嵌入带动空间之间的互动疮。 与这些标明模型不同,我们贯彻的注意框架并没有明确地使用物体探测的基本原理,而是从零开始学习潜在的对齐. 这使得我们的模型能够超越“目标”,؟抽象的概念。

Prior to the use of neural networks for generating captions, two main approaches were dominant. The first involved generating caption templates which were filled in based on the results of object detections and attribute discovery (Kulkarni et al. (2013), Li et al. (2011), Yang et al. (2011), Mitchell et al. (2012), Elliott & Keller (2013)). The second approach was based on first retrieving similar captioned images from a large database then modifying these retrieved captions to fit the query (Kuznetsova et al., 2012; Kuznetsova et al., 2014). These approaches typically involved an intermediate “generalization” step to remove the specifics of a caption that are only relevant to the retrieved image, such as the name of a city. Both of these approaches have since fallen out of favour to the now dominant neural network methods.

在使用神经网络生成标题之前,主要有两种方法。 首先是生成根据物体检测和属性发现结果(Kulkarni等人,2013年;Li等人,2011年;Yang等人,2011年;Mitchell等人,2012年;Elliott & Keller,2013年)填报的字幕对发展的模板。 第二种方法是首先从一个大型数据库检索类似的标题图像,通过以下方式修改这些还将检索到的标题,以适应查询(Kuznetsova等人,2012年;Kuznetsova等人, 2014年)。 这些办法通常涉及ằ 哪一种的结论,即采取中间“一般化”步骤,删除实质内容中仅与取回的图像有关的具体内容, 现在我们可以从城市名称中取出。 这两种方法后来都不再十分重视的目的是现在占主导地位的神经网络方法。

There has been a long line of previous work incorporating attention into neural networks for vision related tasks. Some that share the same spirit as our work include Larochelle & Hinton (2010); Denil et al. (2012); Tang et al. (2014). In particular however, our work directly extends the work of Bahdanau et al. (2014); Mnih et al. (2014); Ba et al. (2014).

过去的工作很长,把注意力纳入神经网络,以完成与愿景有关的任务。 一些与我们的工作具有相同精神的国家包括Larochelle & Hinton(2010年);Denil等人(2012年);Tang等人(2014年)。 然而,我们的工作尤其直接扩展了Bahdanau等人(2014年)、Mnih等人(2014年)、Ba等人(2014年) 明天的工作。

3 Image Caption Generation with Attention Mechanism

3 带注意力机制的图像描述生成

3.1 Model Details

3.1 模型细节

In this section, we describe the two variants of our attention-based model by first describing their common framework. The main difference is the definition of the $\phi$ function which we describe in detail in Section 4. We denote vectors with bolded font and matrices with capital letters. In our description below, we suppress bias terms for readability.

在本节中,我们首先描述了我们以关注为基础的模式的两个变体。 主要的区别在于对$\phi$函数的定义,我们在 我不禁在第4节中详细描述了这一定义. 我们用粗体表示无法确定的矢量,用大写字母表示矩阵。 在下文的描述中无法确定可读性的偏差术语。

3.1.1 Encoder: Convolutional Features

Our model takes a single raw image and generates a caption $\mathbf{y}$ encoded as a sequence of 1-of-$K$ encoded words.

的一名

我们 为什么该模型罢了? 我们的模型基本上是一个昙花一现的图像 并生成一个被编码为$\mathbf{y}$的字符串。

$$y=\left\{\mathbf{y}_{1},\ldots,\mathbf{y}_{C}\right\},\mbox{ }\mathbf{y}_{i}\in\mathbb{R}^{K}$$

where $K$ is the size of the vocabulary and $C$ is the length of the caption.

其中ằng XX 我以前是词汇的年龄,而$C$是标题的长度。

We use a convolutional neural network in order to extract a set of feature vectors which we refer to as annotation vectors. The extractor produces $L$ vectors, each of which is a D-dimensional representation corresponding to a part of the image.

我们用卷积神经网络提取一组特征向量，称作标注向量。提取器产生 $L$ 个向量，每个是对应图像某一部分的 $D$ 维表示。

$$\displaystyle a=\left\{\mathbf{a}_{1},\ldots,\mathbf{a}_{L}\right\},\mbox{ }\mathbf{a}_{i}\in\mathbb{R}^{D}$$

In order to obtain a correspondence between the feature vectors and portions of the 2-D image, we extract features from a lower convolutional layer unlike previous work which instead used a fully connected layer. This allows the decoder to selectively focus on certain parts of an image by selecting a subset of all the feature vectors.

为了让特征向量与二维图像的局部区域对应起来，我们从较低的卷积层提取特征，而不是像先前工作那样使用全连接层。这样，解码器可以通过选择全部特征向量的一个子集，有选择地聚焦图像的某些部分。

Figure 4: A LSTM cell, lines with bolded squares imply projections with a learnt weight vector. Each cell learns how to weigh its input components (input gate), while learning how to modulate that contribution to the memory (input modulator). It also learns weights which erase the memory cell (forget gate), and weights which control how this memory should be emitted (output gate).

图 4：一个 LSTM 单元；带粗方框的线表示带可学习权重向量的投影。每个单元学习如何对输入分量加权（输入门），如何调节其对记忆的贡献（输入调制），如何擦除记忆（遗忘门），以及如何把记忆暴露给隐状态（输出门）。

3.1.2 Decoder: Long Short-Term Memory Network

We use a long short-term memory (LSTM) network (Hochreiter & Schmidhuber, 1997) that produces a caption by generating one word at every time step conditioned on a context vector, the previous hidden state and the previously generated words. Our implementation of LSTM closely follows the one used in Zaremba et al. (2014) (see Fig. 4). Using $T_{s,t}:\mathbb{R}^{s}\rightarrow\mathbb{R}^{t}$ to denote a simple affine transformation with parameters that are learned,

3.1.2 解轩:长期短期记忆网.

我们使用一个长期的短期内存(LSTM)网络(H在制订Hochreiter & Schmidhuber,1997年), 未按上下文向量、 先前隐藏状态囤积的状态和先前生成的单词的每个步骤生成一个单词,从而产生一个标题。对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对法律依据对对对对对对对对对对对对对对对对对对对对对对对对对对对的民主和对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对的议程项目对对对对对对对对对对所表明的对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对 Anh对对对对对对对一会对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对对展开了对对对对对对对对ụ对对对对对对对对对对对 使用$T_{s,t}:\mathbb{R}^{s}\rightarrow\mathbb{R}^{t}$表示带有所学参数的简单近缘转换,

$$\displaystyle\begin{pmatrix}\mathbf{i}{t}\
\mathbf{f}{t}\
\mathbf{o}{t}\
\mathbf{g}{t}\
\end{pmatrix}= \ \displaystyle\begin{pmatrix}\sigma\
\sigma\
\sigma\
\tanh\
\end{pmatrix}T_{D+m+n,n}\begin{pmatrix}\mathbf{E}\mathbf{y}{t-1}\
\mathbf{h}{t-1}\
\hat{\mathbf{z}{t}}\
\end{pmatrix} \ \displaystyle\mathbf{c}{t} \ \displaystyle=\mathbf{f}{t}\odot\mathbf{c}{t-1}+\mathbf{i}{t}\odot\mathbf{g}{t} \ \displaystyle\mathbf{h}{t} \ \displaystyle=\mathbf{o}{t}\odot\tanh(\mathbf{c}_{t}). \tag{1}$$

Here, $\mathbf{i}_{t}$, $\mathbf{f}_{t}$, $\mathbf{c}_{t}$, $\mathbf{o}_{t}$, $\mathbf{h}_{t}$ are the input, forget, memory, output and hidden state of the LSTM, respectively. The vector $\hat{\mathbf{z}}\in\mathbb{R}^{D}$ is the context vector, capturing the visual information associated with a particular input location, as explained below. $\mathbf{E}\in\mathbb{R}^{m\times K}$ is an embedding matrix. Let $m$ and $n$ denote the embedding and LSTM dimensionality respectively and $\sigma$ and $\odot$ be the logistic sigmoid activation and element-wise multiplication respectively.

这里，$\mathbf{i}_{t}$、$\mathbf{f}_{t}$、$\mathbf{c}_{t}$、$\mathbf{o}_{t}$、$\mathbf{h}_{t}$ 分别是 LSTM 的输入门、遗忘门、记忆、输出门与隐状态。向量 $\hat{\mathbf{z}}\in\mathbb{R}^{D}$ 是上下文向量，捕获与特定输入位置相关的视觉信息，下文说明。$\mathbf{E}\in\mathbb{R}^{m\times K}$ 是嵌入矩阵，$m$ 是嵌入维数。在条件语言模型语境中，可把投影矩阵 $T$、偏置与嵌入矩阵 $E$ 理解为解码器的参数。

In simple terms, the context vector $\hat{\mathbf{z}}_{t}$ (equations (1)–(3)) is a dynamic representation of the relevant part of the image input at time $t$. We define a mechanism $\phi$ that computes $\hat{\mathbf{z}}_{t}$ from the annotation vectors $\mathbf{a}_{i},i=1,\ldots,L$ corresponding to the features extracted at different image locations. For each location $i$, the mechanism generates a positive weight $\alpha_{i}$ which can be interpreted either as the probability that location $i$ is the right place to focus for producing the next word (the “hard” but stochastic attention mechanism), or as the relative importance to give to location $i$ in blending the $a_{i}$’s together. The weight $\alpha_{i}$ of each annotation vector $a_{i}$ is computed by an attention model $f_{\mbox{att}}$ for which we use a multilayer perceptron conditioned on the previous hidden state $h_{t-1}$. The soft version of this attention mechanism was introduced by Bahdanau et al. (2014). For emphasis, we note that the hidden state varies as the output RNN advances in its output sequence: “where” the network looks next depends on the sequence of words that has already been generated.

简单地说，上下文向量 $\hat{\mathbf{z}}_{t}$（式 (1)–(3)）是时刻 $t$ 与输入图像相关部分对应的动态表示。我们定义机制 $\phi$，根据对应不同图像位置提取的标注向量 $\mathbf{a}_{i},i=1,\ldots,L$，计算 $\hat{\mathbf{z}}_{t}$。

$$\displaystyle e_{ti}= \\ \displaystyle f_{\mbox{att}}(\mathbf{a}_{i},\mathbf{h}_{t-1}) \\ \displaystyle\alpha_{ti}= \\ \displaystyle\frac{\exp(e_{ti})}{\sum_{k=1}^{L}\exp(e_{tk})}. \tag{4}$$

Once the weights (which sum to one) are computed, the context vector $\hat{z}_{t}$ is computed by

一旦加权数(总和为1)计算出来,则上下文矢量$\hat{z}_{t}$由

$$\displaystyle\hat{\mathbf{z}}_{t}=\phi\left(\left\{\mathbf{a}_{i}\right\},\left\{\alpha_{i}\right\}\right), \tag{6}$$

where $\phi$ is a function that returns a single vector given the set of annotation vectors and their corresponding weights. The details of $\phi$ function are discussed in Sec. 4.

其中 ′$\phi$是返回单个矢量的函数,以对应的注释矢量及其相应的权重。 43 第4节讨论了$\phi$函数的细节。

The initial memory state and hidden state of the LSTM are predicted by an average of the annotation vectors fed through two separate MLPs (init,c and init,h):

LSTM 的初始记忆状态与隐状态，由标注向量的平均经两个独立的 MLP（init,c 与 init,h）预测得到：

$$\displaystyle\mathbf{c}_{0}=f_{\text{init,c}}(\frac{1}{L}\sum_{i}^{L}\mathbf{a}_{i}) \\ \displaystyle\mathbf{h}_{0}=f_{\text{init,h}}(\frac{1}{L}\sum_{i}^{L}\mathbf{a}_{i})$$

In this work, we use a deep output layer (Pascanu et al., 2014) to compute the output word probability given the LSTM state, the context vector and the previous word:

在这部作品中,我们使用深输出层(Pascanu et al., 2014)来计算输出单词概率,考虑到LSTM状态,上下文矢量和上ح的单词:

$$\displaystyle p(\mathbf{y}_{t}|\mathbf{a},\mathbf{y}_{1}^{t-1})\propto\exp(\mathbf{L}_{o}(\mathbf{E}\mathbf{y}_{t-1}+\mathbf{L}_{h}\mathbf{h}_{t}+\mathbf{L}_{z}\hat{\mathbf{z}}_{t})) \tag{7}$$

Where $\mathbf{L}_{o}\in\mathbb{R}^{K\times m}$, $\mathbf{L}_{h}\in\mathbb{R}^{m\times n}$, $\mathbf{L}_{z}\in\mathbb{R}^{m\times D}$, and $\mathbf{E}$ are learned parameters initialized randomly.

其中 $\mathbf{L}_{o}\in\mathbb{R}^{K\times m}$，$\mathbf{L}_{h}\in\mathbb{R}^{m\times n}$，$\mathbf{L}_{z}\in\mathbb{R}^{m\times D}$，以及 $\mathbf{E}$ 是随机初始化的可学习参数。

4 Learning Stochastic “Hard” vs Deterministic “Soft” Attention

4 学习随机“硬”注意力与确定性“软”注意力

In this section we discuss two alternative mechanisms for the attention model $f_{\mbox{att}}$: stochastic attention and deterministic attention.

实际上是 我只是 听 59/250 上帝 酝 那么你 所有的 你应该 罗 马上 知道了 开始进行

Figure 5: Examples of mistakes where we can use attention to gain intuition into what the model saw.

图 5：可用注意力理解模型“看错了什么”的出错例子。

4.1 Stochastic “Hard” Attention

4.1 随机“硬”注意力

We represent the location variable $s_{t}$ as where the model decides to focus attention when generating the $t^{th}$ word. $s_{t,i}$ is an indicator one-hot variable which is set to 1 if the $i$-th location (out of $L$) is the one used to extract visual features. By treating the attention locations as intermediate latent variables, we can assign a multinoulli distribution parametrized by $\{\alpha_{i}\}$, and view $\hat{z}_{t}$ as a random variable:

我们代表的是位置变量$s_{t}$,即模型在生成的主题$t^{th}$词时决定集中关注的位置. $s_{t,i}$是一个指标一热变量,如果XXMA▪TH3XX - 第1位(在$L$中)是用来提取视觉特征的,则设定为1 没有. 通过将注意力位置作为中间潜伏变量处理,我们可以指定一个由$\{\alpha_{i}\}$化的多诺利分布 天呐,并视$\hat{z}_{t}$为随机变量:

$$\displaystyle p( \\ \displaystyle s_{t,i}=1\mid s_{j<t},\mathbf{a})=\alpha_{t,i} \\ \displaystyle\hat{\mathbf{z}}_{t} \\ \displaystyle=\sum_{i}{s}_{t,i}\mathbf{a}_{i}. \tag{8}$$

We define a new objective function $L_{s}$ that is a variational lower bound on the marginal log-likelihood $\log p(\mathbf{y}\mid\mathbf{a})$ of observing the sequence of words $\mathbf{y}$ given image features $\mathbf{a}$. The learning algorithm for the parameters $W$ of the models can be derived by directly optimizing $L_{s}$:

我们定义新目标函数 $L_{s}$，它是在给定图像特征 $\mathbf{a}$ 时观测到词序列 $\mathbf{y}$ 的边缘对数似然 $\log p(\mathbf{y}\mid\mathbf{a})$ 的变分下界。模型参数 $W$ 的学习规则可通过直接优化 $L_{s}$ 得到：

$$\displaystyle L_{s}= \\ \displaystyle\sum_{s}p(s\mid\mathbf{a})\log p(\mathbf{y}\mid s,\mathbf{a}) \\ \displaystyle\leq \\ \displaystyle\log\sum_{s}p(s\mid\mathbf{a})p(\mathbf{y}\mid s,\mathbf{a}) \\ \displaystyle= \\ \displaystyle\log p(\mathbf{y}\mid\mathbf{a}) \tag{10}$$

$$\frac{\partial L_{s}}{\partial W}=\sum_{s}p(s\mid\mathbf{a})\bigg[\frac{\partial\log p(\mathbf{y}\mid s,\mathbf{a})}{\partial W}+\
\log p(\mathbf{y}\mid s,\mathbf{a})\frac{\partial\log p(s\mid\mathbf{a})}{\partial W}\bigg]. \tag{11}$$

Equation 11 suggests a Monte Carlo based sampling approximation of the gradient with respect to the model parameters. This can be done by sampling the location $s_{t}$ from a multinouilli distribution defined by Equation 8.

方程式11建议对模型参数的梯度进行基于蒙特卡洛的采样近似。 这可以通过从 我还是方程式8定义能够获得的多努伊利分布物中取样来进行。

$$\displaystyle\tilde{s_{t}}\sim\text{Multinoulli}_{L}(\{\alpha_{i}\})$$

$$\frac{\partial L_{s}}{\partial W}\approx\frac{1}{N}\sum_{n=1}^{N}\bigg[\frac{\partial\log p(\mathbf{y}\mid\tilde{s}^{n},\mathbf{a})}{\partial W}+\
\log p(\mathbf{y}\mid\tilde{s}^{n},\mathbf{a})\frac{\partial\log p(\tilde{s}^{n}\mid\mathbf{a})}{\partial W}\bigg] \tag{12}$$

A moving average baseline is used to reduce the variance in the Monte Carlo estimator of the gradient, following Weaver & Tao (2001). Similar, but more complicated variance reduction techniques have previously been used by Mnih et al. (2014) and Ba et al. (2014). Upon seeing the $k^{th}$ mini-batch, the moving average baseline is estimated as an accumulated sum of the previous log likelihoods with exponential decay:

采用了移动平均基线,以减少Weaver & Tao (2001年) 难道在蒙特卡洛梯度估计值中的差异吗? Mnih等人(2014年)和Ba等人(2014ằng,2014年)以前 你先使用过类似但更为复杂的减差技术。 在看到$k^{th}$小批量后,移动女友平均基线估计为指数衰减的 这样以前的的步伐:

$$\displaystyle b_{k}=0.9\times b_{k-1}+0.1\times\log p(\mathbf{y}\mid\tilde{s}_{k},\mathbf{a})$$

To further reduce the estimator variance, an entropy term on the multinouilli distribution $H[s]$ is added. Also, with probability 0.5 for a given image, we set the sampled attention location $\tilde{s}$ to its expected value $\alpha$. Both techniques improve the robustness of the stochastic attention learning algorithm. The final learning rule for the model is then the following:

为了进一步减少估计提交其差异,在多努伊利分配$H[s]$上 Adriannp憧憬词被添加. 另外,如果精灵图 请将本函及其附件作为安全理事会值为0.5,我们将把 我从没被抽样的注意位置确定为$\tilde{s}$。 这两种技术都提高了斯图克注意力学习算法的稳健性. 模型的最后学习规则是:

$$\frac{\partial L_{s}}{\partial W}\approx\frac{1}{N}\sum_{n=1}^{N}\bigg[\frac{\partial\log p(\mathbf{y}\mid\tilde{s}^{n},\mathbf{a})}{\partial W}+\
\lambda_{r}(\log p(\mathbf{y}\mid\tilde{s}^{n},\mathbf{a})-b)\frac{\partial\log p(\tilde{s}^{n}\mid\mathbf{a})}{\partial W}+\lambda_{e}\frac{\partial H[\tilde{s}^{n}]}{\partial W}\bigg] \tag{13}$$

where, $\lambda_{r}$ and $\lambda_{e}$ are two hyper-parameters set by cross-validation. As pointed out and used in Ba et al. (2014) and Mnih et al. (2014), this is formulation is equivalent to the REINFORCE learning rule (Williams, 1992), where the reward for the attention choosing a sequence of actions is a real value proportional to the log likelihood of the target sentence under the sampled attention trajectory.

其中,鼎 ($\lambda_{r}$)和鼎XX($\lambda_{e}$)是通过交叉验证设定的两个超参数. 正如Ba方面的努力是吗( 我们去2014年)和Mnih等人的报告(2014年)所指出和使用的那样,这种提法相当于REINFORCE学习规则(Williams,1992年),其中对注意选择一系列行动给予的奖励是相对于抽样注意轨迹下目标句 我们走吧对数可能性的实际价值。

In making a hard choice at every point, $\phi\left(\left\{\mathbf{a}_{i}\right\},\left\{\alpha_{i}\right\}\right)$ from Equation 6 is a function that returns a sampled $\mathbf{a}_{i}$ at every point in time based upon a multinouilli distribution parameterized by $\alpha$.

我应该在每点进行硬性选择时,从方程式6的$\phi\left(\left\{\mathbf{a}_{i}\right\},\left\{\alpha_{i}\right\}\right)$是一个函数,它会基于由$\alpha$ 化的多努伊利分布参数,在每点返回一个样本的$\mathbf{a}_{i}$. 我们有这样的功能.

4.2 Deterministic “Soft” Attention

4.2 确定性“软”注意力

Learning stochastic attention requires sampling the attention location $s_{t}$ each time, instead we can take the expectation of the context vector $\hat{\mathbf{z}}_{t}$ directly,

学习随机注意力需要每次采样注意力位置 $s_{t}$；我们也可以直接取上下文向量 $\hat{\mathbf{z}}_{t}$ 的期望，

$$\displaystyle\mathbb{E}_{p(s_{t}|a)}[\hat{\mathbf{z}}_{t}]=\sum_{i=1}^{L}\alpha_{t,i}\mathbf{a}_{i} \tag{14}$$

and formulate a deterministic attention model by computing a soft attention weighted annotation vector $\phi\left(\left\{\mathbf{a}_{i}\right\},\left\{\alpha_{i}\right\}\right)=\sum_{i}^{L}\alpha_{i}\mathbf{a}_{i}$ as introduced by Bahdanau et al. (2014). This corresponds to feeding in a soft $\alpha$ weighted context into the system. The whole model is smooth and differentiable under the deterministic attention, so learning end-to-end is trivial by using standard back-propagation.

并像 Bahdanau 等人（2014）那样，用软注意力加权的标注向量 $\phi\left(\left\{\mathbf{a}_{i}\right\},\left\{\alpha_{i}\right\}\right)=\sum_{i}^{L}\alpha_{i}\mathbf{a}_{i}$ 构成确定性注意力模型。这相当于把经软 $\alpha$ 加权的上下文送入系统。整个模型因此可微，可用标准反向传播训练。

Learning the deterministic attention can also be understood as approximately optimizing the marginal likelihood in Equation 10 under the attention location random variable $s_{t}$ from Sec. 4.1. The hidden activation of LSTM $\mathbf{h}_{t}$ is a linear projection of the stochastic context vector $\hat{\mathbf{z}}_{t}$ followed by $\tanh$ non-linearity. To the first order Taylor approximation, the expected value $\mathbb{E}_{p(s_{t}|a)}[\mathbf{h}_{t}]$ is equal to computing $\mathbf{h}_{t}$ using a single forward prop with the expected context vector $\mathbb{E}_{p(s_{t}|a)}[\hat{\mathbf{z}}_{t}]$. Considering Eq. 7, let $\mathbf{n}_{t}=\mathbf{L}_{o}(\mathbf{E}\mathbf{y}_{t-1}+\mathbf{L}_{h}\mathbf{h}_{t}+\mathbf{L}_{z}\hat{\mathbf{z}}_{t})$, $\mathbf{n}_{t,i}$ denotes $\mathbf{n}_{t}$ computed by setting the random variable $\hat{\mathbf{z}}$ value to $\mathbf{a}_{i}$. We define the normalized weighted geometric mean for the softmax $k^{th}$ word prediction:

学习确定性注意力也可以理解为：在第 4.1 节的注意力位置随机变量 $s_{t}$ 下，近似优化式 (10) 中的边缘似然。LSTM 的隐激活 $\mathbf{h}_{t}$ 是随机上下文向量 $\hat{\mathbf{z}}_{t}$ 的线性投影再接 $\tanh$ 非线性。

$$\displaystyle NWGM[p(y_{t}=k\mid\mathbf{a})] \\ \displaystyle={\prod_{i}\exp(n_{t,k,i})^{p(s_{t,i}=1|a)}\over\sum_{j}\prod_{i}\exp(n_{t,j,i})^{p(s_{t,i}=1|a)}} \\ \displaystyle={\exp(\mathbb{E}_{p(s_{t}|a)}[n_{t,k}])\over\sum_{j}\exp(\mathbb{E}_{p(s_{t}|a)}[n_{t,j}])}$$

The equation above shows the normalized weighted geometric mean of the caption prediction can be approximated well by using the expected context vector, where $\mathbb{E}[\mathbf{n}_{t}]=\mathbf{L}_{o}(\mathbf{E}\mathbf{y}_{t-1}+\mathbf{L}_{h}\mathbb{E}[\mathbf{h}_{t}]+\mathbf{L}_{z}\mathbb{E}[\hat{\mathbf{z}}_{t}])$. It shows that the NWGM of a softmax unit is obtained by applying softmax to the expectations of the underlying linear projections. Also, from the results in Baldi & Sadowski (2014), $NWGM[p(y_{t}=k\mid\mathbf{a})]\approx\mathbb{E}[p(y_{t}=k\mid\mathbf{a})]$ under softmax activation. That means the expectation of the outputs over all possible attention locations induced by random variable $s_{t}$ is computed by simple feedforward propagation with expected context vector $\mathbb{E}[\hat{\mathbf{z}}_{t}]$. In other words, the deterministic attention model is an approximation to the marginal likelihood over the attention locations.

以上方程显示,通过使用预期的上下文矢量,可以很好地大致得出标题预测的正常加权几何平均值。 它表明,实施一项软马克就其单位的NWGM是通过将软马克应用到联合国系统各基础线性预测的预期中来获得的. 另外,根据Baldi &政府提供的结果好几两族(2014年 我保证),$NWGM[p(y_{t}=k\mid\mathbf{a})]\approx\mathbb{E}[p(y_{t}=k\mid\mathbf{a})]$在软马克斯激活下. 他们是通过使用预期的上下文 什么事? 换句话说,决定性关注模型是相对于关注地点的边际可能性的近似.

4.2.1 Doubly Stochastic Attention

By construction, $\sum_{i}\alpha_{ti}=1$ as they are the output of a softmax. In training the deterministic version of our model we introduce a form of doubly stochastic regularization, where we also encourage $\sum_{t}\alpha_{ti}\approx 1$. This can be interpreted as encouraging the model to pay equal attention to every part of the image over the course of generation. In our experiments, we observed that this penalty was important quantitatively to improving overall BLEU score and that qualitatively this leads to more rich and descriptive captions. In addition, the soft attention model predicts a gating scalar $\beta$ from previous hidden state $\mathbf{h}_{t-1}$ at each time step $t$, such that, $\phi\left(\left\{\mathbf{a}_{i}\right\},\left\{\alpha_{i}\right\}\right)=\beta\sum_{i}^{L}\alpha_{i}\mathbf{a}_{i}$, where $\beta_{t}=\sigma(f_{\beta}(\mathbf{h}_{t-1}))$. We notice our attention weights put more emphasis on the objects in the images by including the scalar $\beta$.

4.2.1 双重随机注意力

由构造可知，$\sum_{i}\alpha_{ti}=1$，因为它们是 softmax 的输出。在训练确定性版本时，我们引入一种双重随机正则：同时鼓励 $\sum_{t}\alpha_{ti}\approx 1$。这可以理解为鼓励模型在整句生成过程中对每个空间位置给予大致均等的总关注，避免注意力塌缩到少数区域。

		BLEU
Dataset	Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR
Flickr8k	Google NIC(Vinyals et al., 2014)†Σ Log Bilinear (Kiros et al., 2014a)∘ Soft-Attention Hard-Attention	Google NIC(Vinyals et al., 2014)†Σ	Log Bilinear (Kiros et al., 2014a)∘	Soft-Attention	Hard-Attention	63 65.6 67 67	63	65.6	67	67	41 42.4 44.8 45.7	41	42.4	44.8	45.7	27 27.7 29.9 31.4	27	27.7	29.9	31.4	— 17.7 19.5 21.3	—	17.7	19.5	21.3	— 17.31 18.93 20.30	—	17.31	18.93	20.30
Google NIC(Vinyals et al., 2014)†Σ
Log Bilinear (Kiros et al., 2014a)∘
Soft-Attention
Hard-Attention
63
65.6
67
67
41
42.4
44.8
45.7
27
27.7
29.9
31.4
—
17.7
19.5
21.3
—
17.31
18.93
20.30
Flickr30k	Google NIC†∘Σ Log Bilinear Soft-Attention Hard-Attention	Google NIC†∘Σ	Log Bilinear	Soft-Attention	Hard-Attention	66.3 60.0 66.7 66.9	66.3	60.0	66.7	66.9	42.3 38 43.4 43.9	42.3	38	43.4	43.9	27.7 25.4 28.8 29.6	27.7	25.4	28.8	29.6	18.3 17.1 19.1 19.9	18.3	17.1	19.1	19.9	— 16.88 18.49 18.46	—	16.88	18.49	18.46
Google NIC†∘Σ
Log Bilinear
Soft-Attention
Hard-Attention
66.3
60.0
66.7
66.9
42.3
38
43.4
43.9
27.7
25.4
28.8
29.6
18.3
17.1
19.1
19.9
—
16.88
18.49
18.46
COCO	CMU/MS Research (Chen & Zitnick, 2014)a MS Research (Fang et al., 2014)†a BRNN (Karpathy & Li, 2014)∘ Google NIC†∘Σ Log Bilinear∘ Soft-Attention Hard-Attention	CMU/MS Research (Chen & Zitnick, 2014)a	MS Research (Fang et al., 2014)†a	BRNN (Karpathy & Li, 2014)∘	Google NIC†∘Σ	Log Bilinear∘	Soft-Attention	Hard-Attention	— — 64.2 66.6 70.8 70.7 71.8	—	—	64.2	66.6	70.8	70.7	71.8	— — 45.1 46.1 48.9 49.2 50.4	—	—	45.1	46.1	48.9	49.2	50.4	— — 30.4 32.9 34.4 34.4 35.7	—	—	30.4	32.9	34.4	34.4	35.7	— — 20.3 24.6 24.3 24.3 25.0	—	—	20.3	24.6	24.3	24.3	25.0	20.41 20.71 — — 20.03 23.90 23.04	20.41	20.71	—	—	20.03	23.90	23.04
CMU/MS Research (Chen & Zitnick, 2014)a
MS Research (Fang et al., 2014)†a
BRNN (Karpathy & Li, 2014)∘
Google NIC†∘Σ
Log Bilinear∘
Soft-Attention
Hard-Attention
—
—
64.2
66.6
70.8
70.7
71.8
—
—
45.1
46.1
48.9
49.2
50.4
—
—
30.4
32.9
34.4
34.4
35.7
—
—
20.3
24.6
24.3
24.3
25.0
20.41
20.71
—
—
20.03
23.90
23.04

Google NIC(Vinyals et al., 2014)†Σ
Log Bilinear (Kiros et al., 2014a)∘
Soft-Attention
Hard-Attention

63
65.6
67
67

41
42.4
44.8
45.7

27
27.7
29.9
31.4

—
17.7
19.5
21.3

—
17.31
18.93
20.30

Google NIC†∘Σ
Log Bilinear
Soft-Attention
Hard-Attention

66.3
60.0
66.7
66.9

42.3
38
43.4
43.9

27.7
25.4
28.8
29.6

18.3
17.1
19.1
19.9

—
16.88
18.49
18.46

CMU/MS Research (Chen & Zitnick, 2014)a
MS Research (Fang et al., 2014)†a
BRNN (Karpathy & Li, 2014)∘
Google NIC†∘Σ
Log Bilinear∘
Soft-Attention
Hard-Attention

—
—
64.2
66.6
70.8
70.7
71.8

—
—
45.1
46.1
48.9
49.2
50.4

—
—
30.4
32.9
34.4
34.4
35.7

—
—
20.3
24.6
24.3
24.3
25.0

20.41
20.71
—
—
20.03
23.90
23.04

Table 1: BLEU-1,2,3,4/METEOR metrics compared to other methods, $\dagger$ indicates a different split, (—) indicates an unknown metric, $\circ$ indicates the authors kindly provided missing metrics by personal communication, $\Sigma$ indicates an ensemble, $a$ indicates using AlexNet