Efficient Estimation of Word Representations in Vector Space
Efficient Estimation of Word Representations in Vector Space
词向量空间中词表示的高效估计（Efficient Estimation of Word Representations in Vector Space）
Abstract
摘要
We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.
我们提出两种新的模型架构，用于从超大规模数据集中计算词的连续向量表示。这些表示的质量在词相似度任务上衡量，并与此前基于不同类型神经网络、表现最好的技术进行比较。我们观察到，在计算成本低得多的情况下准确率有大幅提升，也就是说，从 16 亿词的数据集中学习高质量词向量用不了一天。此外，我们还表明，这些向量在我们用于衡量句法和语义词相似度的测试集上达到了当时最好的水平。
1 Introduction
1 引言
Many current NLP systems and techniques treat words as atomic units - there is no notion of similarity between words, as these are represented as indices in a vocabulary. This choice has several good reasons - simplicity, robustness and the observation that simple models trained on huge amounts of data outperform complex systems trained on less data. An example is the popular N-gram model used for statistical language modeling - today, it is possible to train N-grams on virtually all available data (trillions of words [3]).
当前许多 NLP 系统和技术把词当作原子单元——词被表示成词汇表中的索引，不存在词与词之间的相似度概念。这一选择有若干充分理由：简单、稳健，以及一个观察——在海量数据上训练的简单模型，往往优于在较少数据上训练的复杂系统。一个例子是用于统计语言建模的流行 N-gram 模型——如今已有可能在几乎所有可得数据（万亿词量级）上训练 N-gram [3]。
However, the simple techniques are at their limits in many tasks. For example, the amount of relevant in-domain data for automatic speech recognition is limited - the performance is usually dominated by the size of high quality transcribed speech data (often just millions of words). In machine translation, the existing corpora for many languages contain only a few billions of words or less. Thus, there are situations where simple scaling up of the basic techniques will not result in any significant progress, and we have to focus on more advanced techniques.
然而，在许多任务上，简单技术已接近极限。例如，自动语音识别中相关的领域内数据量有限——性能通常由高质量转写语音数据的规模主导（往往只有百万词量级）。在机器翻译中，许多语言现有语料只有几十亿词甚至更少。因此，存在一些情形，单纯放大基本技术不会带来显著进展，我们必须转向更先进的技术。
With progress of machine learning techniques in recent years, it has become possible to train more complex models on much larger data set, and they typically outperform the simple models. Probably the most successful concept is to use distributed representations of words [10]. For example, neural network based language models significantly outperform N-gram models [1, 27, 17].
随着近年来机器学习技术的进步，已有可能在大得多的数据集上训练更复杂的模型，它们通常优于简单模型。最成功的概念大概是使用词的分布式表示 [10]。例如，基于神经网络的语言模型显著优于 N-gram 模型 [1, 27, 17]。
1.1 Goals of the Paper
1.1 本文目标
The main goal of this paper is to introduce techniques that can be used for learning high-quality word vectors from huge data sets with billions of words, and with millions of words in the vocabulary. As far as we know, none of the previously proposed architectures has been successfully trained on more than a few hundred of millions of words, with a modest dimensionality of the word vectors between 50 - 100.
本文的主要目标是介绍若干技术，用于从包含数十亿词、词汇表达数百万词的超大规模数据集中学习高质量词向量。据我们所知，此前提出的架构都未能成功训练超过几亿词，且词向量维度较为适中，在 50–100 之间。
We use recently proposed techniques for measuring the quality of the resulting vector representations, with the expectation that not only will similar words tend to be close to each other, but that words can have multiple degrees of similarity [20]. This has been observed earlier in the context of inflectional languages - for example, nouns can have multiple word endings, and if we search for similar words in a subspace of the original vector space, it is possible to find words that have similar endings [13, 14].
我们使用近来提出的技术来衡量所得向量表示的质量，并期望不仅相似词会彼此靠近，而且词可以具有多种不同程度的相似性 [20]。这一点早在屈折语语境中就被观察到——例如，名词可以有多种词尾，若在原向量空间的某个子空间中搜索相似词，就有可能找到词尾相似的词 [13, 14]。
Somewhat surprisingly, it was found that similarity of word representations goes beyond simple syntactic regularities. Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that vector(”King”) - vector(”Man”) + vector(”Woman”) results in a vector that is closest to the vector representation of the word Queen [20].
有些出人意料的是，人们发现词表示的相似性超出了简单的句法规律。使用词偏移技术，对词向量做简单的代数运算，例如可以证明 vector("King") − vector("Man") + vector("Woman") 得到的向量最接近 Queen 的向量表示 [20]。
In this paper, we try to maximize accuracy of these vector operations by developing new model architectures that preserve the linear regularities among words. We design a new comprehensive test set for measuring both syntactic and semantic regularities¹, and show that many such regularities can be learned with high accuracy. Moreover, we discuss how training time and accuracy depends on the dimensionality of the word vectors and on the amount of the training data.
在本文中，我们试图通过开发能保留词之间线性规律的新模型架构，来最大化这类向量运算的准确率。我们设计了一套新的综合测试集，用于同时衡量句法和语义规律¹，并表明许多此类规律可以以很高的准确率被学到。此外，我们还讨论训练时间和准确率如何依赖于词向量的维度以及训练数据量。
1 The test set is available at www.fit.vutbr.cz/~imikolov/rnnlm/word-test.v1.txt
1.2 Previous Work
1.2 相关工作
Representation of words as continuous vectors has a long history [10, 26, 8]. A very popular model architecture for estimating neural network language model (NNLM) was proposed in [1], where a feedforward neural network with a linear projection layer and a non-linear hidden layer was used to learn jointly the word vector representation and a statistical language model. This work has been followed by many others.
把词表示为连续向量已有很长的历史 [10, 26, 8]。一种非常流行的估计神经网络语言模型（NNLM）的模型架构由 [1] 提出，其中使用带线性投影层和非线性隐层的前馈神经网络，联合学习词向量表示和统计语言模型。此后有许多工作沿这一方向展开。
Another interesting architecture of NNLM was presented in [13, 14], where the word vectors are first learned using neural network with a single hidden layer. The word vectors are then used to train the NNLM. Thus, the word vectors are learned even without constructing the full NNLM. In this work, we directly extend this architecture, and focus just on the first step where the word vectors are learned using a simple model.
另一种有趣的 NNLM 架构见 [13, 14]：词向量先用只有一个隐层的神经网络学习，再用这些词向量去训练 NNLM。因此，即便不构造完整的 NNLM，也能学到词向量。本文直接扩展这一架构，只关注用简单模型学习词向量的第一步。
It was later shown that the word vectors can be used to significantly improve and simplify many NLP applications [4, 5, 29]. Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 29, 23, 19, 9], and some of the resulting word vectors were made available for future research and comparison². However, as far as we know, these architectures were significantly more computationally expensive for training than the one proposed in [13], with the exception of certain version of log-bilinear model where diagonal weight matrices are used [23].
后来有人表明，词向量可以显著改进并简化许多 NLP 应用 [4, 5, 29]。词向量本身的估计使用了不同的模型架构，并在各种语料上训练 [4, 29, 23, 19, 9]，其中一些所得词向量已公开供后续研究与比较²。然而据我们所知，这些架构的训练计算代价显著高于 [13] 所提出的方法，只有某些使用对角权重矩阵的对数双线性模型版本例外 [23]。
2 Model Architectures
2 模型架构
Many different types of models were proposed for estimating continuous representations of words, including the well-known Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA). In this paper, we focus on distributed representations of words learned by neural networks, as it was previously shown that they perform significantly better than LSA for preserving linear regularities among words [20, 31]; LDA moreover becomes computationally very expensive on large data sets.
已有许多不同类型的模型被提出用于估计词的连续表示，包括著名的潜在语义分析（LSA）和潜在狄利克雷分配（LDA）。本文关注由神经网络学习的词的分布式表示，因为此前已表明它们在保留词之间的线性规律方面显著优于 LSA [20, 31]；此外，LDA 在大数据集上会变得计算上非常昂贵。
Similar to [18], to compare different model architectures we define first the computational complexity of a model as the number of parameters that need to be accessed to fully train the model. Next, we will try to maximize the accuracy, while minimizing the computational complexity.
与 [18] 类似，为比较不同模型架构，我们首先把模型的计算复杂度定义为完全训练该模型所需访问的参数个数。接下来，我们将在最小化计算复杂度的同时尽量提高准确率。
For all the following models, the training complexity is proportional to
对以下所有模型，训练复杂度正比于
where is number of the training epochs, is the number of the words in the training set and is defined further for each model architecture. Common choice is and up to one billion. All models are trained using stochastic gradient descent and backpropagation [26].
其中 是训练轮数， 是训练集中的词数， 对每种模型架构另行定义。常见选择是 –， 可达十亿。所有模型都用随机梯度下降和反向传播训练 [26]。
2.1 Feedforward Neural Net Language Model (NNLM)
2.1 前馈神经网络语言模型（NNLM）
The probabilistic feedforward neural network language model has been proposed in [1]. It consists of input, projection, hidden and output layers. At the input layer, previous words are encoded using 1-of- coding, where is size of the vocabulary. The input layer is then projected to a projection layer that has dimensionality , using a shared projection matrix. As only inputs are active at any given time, composition of the projection layer is a relatively cheap operation.
概率前馈神经网络语言模型由 [1] 提出。它由输入层、投影层、隐层和输出层组成。在输入层， 个先前的词用 1-of- 编码，其中 是词汇表大小。输入层再通过共享投影矩阵投影到维度为 的投影层 。由于任一时刻只有 个输入是激活的，投影层的组合是相对便宜的操作。
The NNLM architecture becomes complex for computation between the projection and the hidden layer, as values in the projection layer are dense. For a common choice of , the size of the projection layer () might be 500 to 2000, while the hidden layer size is typically 500 to 1000 units. Moreover, the hidden layer is used to compute probability distribution over all the words in the vocabulary, resulting in an output layer with dimensionality . Thus, the computational complexity per each training example is
NNLM 架构在投影层与隐层之间的计算变得复杂，因为投影层中的值是稠密的。以常见的 为例，投影层大小（）可能是 500 到 2000，而隐层大小 通常是 500 到 1000 个单元。此外，隐层还要用来计算词汇表中所有词上的概率分布，从而使输出层维度为 。因此，每个训练样本的计算复杂度为
where the dominating term is . However, several practical solutions were proposed for avoiding it; either using hierarchical versions of the softmax [25, 23, 18], or avoiding normalized models completely by using models that are not normalized during training [4, 9]. With binary tree representations of the vocabulary, the number of output units that need to be evaluated can go down to around . Thus, most of the complexity is caused by the term .
其中占主导的是 项。不过已有若干实用方案来规避它：要么使用层次 Softmax 的版本 [25, 23, 18]，要么完全避开归一化模型，改用训练时不归一化的模型 [4, 9]。若用词汇表的二叉树表示，需要求值的输出单元数可降到大约 。这样，大部分复杂度就由 项造成。
In our models, we use hierarchical softmax where the vocabulary is represented as a Huffman binary tree. This follows previous observations that the frequency of words works well for obtaining classes in neural net language models [16]. Huffman trees assign short binary codes to frequent words, and this further reduces the number of output units that need to be evaluated: while balanced binary tree would require outputs to be evaluated, the Huffman tree based hierarchical softmax requires only about . For example when the vocabulary size is one million words, this results in about two times speedup in evaluation. While this is not crucial speedup for neural network LMs as the computational bottleneck is in the term, we will later propose architectures that do not have hidden layers and thus depend heavily on the efficiency of the softmax normalization.
在我们的模型中，我们使用层次 Softmax，并把词汇表表示成 Huffman 二叉树。这遵循此前的观察：词频很适合用来在神经网络语言模型中获得类别 [16]。Huffman 树给频繁词分配较短的二进制编码，这进一步减少了需要求值的输出单元数：平衡二叉树需要求值 个输出，而基于 Huffman 树的层次 Softmax 大约只需要 。例如当词汇表大小为一百万词时，这大约能带来两倍的求值加速。对神经网络语言模型而言这并非关键加速，因为计算瓶颈在 项；但我们稍后会提出没有隐层的架构，因而严重依赖 Softmax 归一化的效率。
2.2 Recurrent Neural Net Language Model (RNNLM)
2.2 循环神经网络语言模型（RNNLM）
Recurrent neural network based language model has been proposed to overcome certain limitations of the feedforward NNLM, such as the need to specify the context length (the order of the model ), and because theoretically RNNs can efficiently represent more complex patterns than the shallow neural networks [15, 2]. The RNN model does not have a projection layer; only input, hidden and output layer. What is special for this type of model is the recurrent matrix that connects hidden layer to itself, using time-delayed connections. This allows the recurrent model to form some kind of short term memory, as information from the past can be represented by the hidden layer state that gets updated based on the current input and the state of the hidden layer in the previous time step.
基于循环神经网络的语言模型被提出来克服前馈 NNLM 的某些局限，例如需要指定上下文长度（模型阶数 ），并且理论上 RNN 能比浅层神经网络更高效地表示更复杂的模式 [15, 2]。RNN 模型没有投影层；只有输入、隐层和输出层。这类模型的特殊之处在于把隐层连到自身的循环矩阵，使用延时连接。这使循环模型能形成某种短期记忆，因为来自过去的信息可由隐层状态表示，而该状态根据当前输入和上一时间步的隐层状态更新。
The complexity per training example of the RNN model is
RNN 模型每个训练样本的复杂度为
where the word representations have the same dimensionality as the hidden layer . Again, the term can be efficiently reduced to by using hierarchical softmax. Most of the complexity then comes from .
其中词表示 与隐层 维度相同。同样， 项可通过层次 Softmax 高效降为 。大部分复杂度于是来自 。
2.3 Parallel Training of Neural Networks
2.3 神经网络的并行训练
To train models on huge data sets, we have implemented several models on top of a large-scale distributed framework called DistBelief [6], including the feedforward NNLM and the new models proposed in this paper. The framework allows us to run multiple replicas of the same model in parallel, and each replica synchronizes its gradient updates through a centralized server that keeps all the parameters. For this parallel training, we use mini-batch asynchronous gradient descent with an adaptive learning rate procedure called Adagrad [7]. Under this framework, it is common to use one hundred or more model replicas, each using many CPU cores at different machines in a data center.
为了在超大规模数据集上训练模型，我们在名为 DistBelief [6] 的大规模分布式框架上实现了若干模型，包括前馈 NNLM 以及本文提出的新模型。该框架允许并行运行同一模型的多个副本，每个副本通过保存全部参数的中央服务器同步其梯度更新。对于这种并行训练，我们使用带自适应学习率过程 Adagrad [7] 的小批量异步梯度下降。在此框架下，通常使用一百个或更多模型副本，每个副本在数据中心不同机器上使用许多 CPU 核心。
3 New Log-linear Models
3 新的对数线性模型
In this section, we propose two new model architectures for learning distributed representations of words that try to minimize computational complexity. The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model. While this is what makes neural networks so attractive, we decided to explore simpler models that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently.
本节我们提出两种新的模型架构，用于学习词的分布式表示，并尽量降低计算复杂度。上一节的主要观察是：大部分复杂度由模型中的非线性隐层造成。尽管正是这一点使神经网络如此有吸引力，我们仍决定探索更简单的模型——它们或许不能像神经网络那样精确地表示数据，但有可能在多得多的数据上被高效训练。
The new architectures directly follow those proposed in our earlier work [13, 14], where it was found that neural network language model can be successfully trained in two steps: first, continuous word vectors are learned using simple model, and then the N-gram NNLM is trained on top of these distributed representations of words. While there has been later substantial amount of work that focuses on learning word vectors, we consider the approach proposed in [13] to be the simplest one. Note that related models have been proposed also much earlier [26, 8].
新架构直接沿用我们早期工作 [13, 14] 中提出的结构，当时发现神经网络语言模型可以分两步成功训练：先用简单模型学习连续词向量，再在这些词的分布式表示之上训练 N-gram NNLM。尽管后来有大量工作聚焦于学习词向量，我们仍认为 [13] 提出的方法是最简单的。注意，相关模型更早也有人提出过 [26, 8]。
3.1 Continuous Bag-of-Words Model
3.1 连续词袋模型
The first proposed architecture is similar to the feedforward NNLM, where the non-linear hidden layer is removed and the projection layer is shared for all words (not just the projection matrix); thus, all words get projected into the same position (their vectors are averaged). We call this architecture a bag-of-words model as the order of words in the history does not influence the projection. Furthermore, we also use words from the future; we have obtained the best performance on the task introduced in the next section by building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word. Training complexity is then
第一种提出的架构类似于前馈 NNLM，但去掉了非线性隐层，并且投影层对所有词共享（不只是投影矩阵共享）；因此，所有词都被投影到同一位置（它们的向量被平均）。我们称这一架构为词袋模型，因为历史中词的顺序不影响投影。此外，我们还使用来自未来的词；在下一节引入的任务上，我们通过构建一个对数线性分类器获得了最佳性能，该分类器以四个未来词和四个历史词为输入，训练准则是正确分类当前（中间）词。于是训练复杂度为
We denote this model further as CBOW, as unlike standard bag-of-words model, it uses continuous distributed representation of the context. The model architecture is shown at Figure 1. Note that the weight matrix between the input and the projection layer is shared for all word positions in the same way as in the NNLM.
我们进一步将此模型记为 CBOW，因为与标准词袋模型不同，它使用上下文的连续分布式表示。模型架构见图 1。注意，输入层与投影层之间的权重矩阵对所有词位置共享，方式与 NNLM 中相同。
Figure 1: New model architectures. The CBOW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word.
图 1：新的模型架构。CBOW 架构根据上下文预测当前词，Skip-gram 则在给定当前词时预测周围的词。
3.2 Continuous Skip-gram Model
3.2 连续 Skip-gram 模型
The second architecture is similar to CBOW, but instead of predicting the current word based on the context, it tries to maximize classification of a word based on another word in the same sentence. More precisely, we use each current word as an input to a log-linear classifier with continuous projection layer, and predict words within a certain range before and after the current word. We found that increasing the range improves quality of the resulting word vectors, but it also increases the computational complexity. Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples.
第二种架构与 CBOW 类似，但不是根据上下文预测当前词，而是试图基于同一句子中的另一个词来最大化对某个词的分类。更精确地说，我们把每个当前词作为带连续投影层的对数线性分类器的输入，并预测当前词之前和之后一定范围内的词。我们发现增大范围会改善所得词向量的质量，但也会增加计算复杂度。由于较远的词通常与当前词的相关性低于较近的词，我们在训练样本中从那些较远的词中采样更少，从而给它们更低的权重。
The training complexity of this architecture is proportional to
该架构的训练复杂度正比于
where is the maximum distance of the words. Thus, if we choose , for each training word we will select randomly a number in range , and then use words from history and words from the future of the current word as correct labels. This will require us to do word classifications, with the current word as input, and each of the words as output. In the following experiments, we use .
其中 是词的最大距离。因此，若选择 ，对每个训练词我们会在区间 内随机选一个数 ，然后把当前词历史中的 个词和未来中的 个词当作正确标签。这将要求我们做 次词分类，以当前词为输入，以这 个词各自为输出。在后续实验中，我们使用 。
4 Results
4 结果
To compare the quality of different versions of word vectors, previous papers typically use a table showing example words and their most similar words, and understand them intuitively. Although it is easy to show that word France is similar to Italy and perhaps some other countries, it is much more challenging when subjecting those vectors in a more complex similarity task, as follows. We follow previous observation that there can be many different types of similarities between words, for example, word big is similar to bigger in the same sense that small is similar to smaller. Example of another type of relationship can be word pairs big - biggest and small - smallest [20]. We further denote two pairs of words with the same relationship as a question, as we can ask: ”What is the word that is similar to small in the same sense as biggest is similar to big?”
为比较不同版本词向量的质量，以往论文通常用一张表展示若干示例词及其最相似的词，并凭直觉理解它们。尽管很容易说明词 France 与 Italy 以及或许其他一些国家相似，但当把这些向量放到如下更复杂的相似度任务中时，挑战要大得多。我们沿用此前的观察：词之间可以有许多不同类型的相似性，例如词 big 与 bigger 相似，其意义正如 small 与 smaller 相似。另一类关系的例子可以是词对 big – biggest 和 small – smallest [20]。我们进一步把具有相同关系的两对词称为一个问题，因为可以问：“哪个词与 small 相似的意义，正如 biggest 与 big 相似？”
Somewhat surprisingly, these questions can be answered by performing simple algebraic operations with the vector representation of words. To find a word that is similar to small in the same sense as biggest is similar to big, we can simply compute vector . Then, we search in the vector space for the word closest to measured by cosine distance, and use it as the answer to the question (we discard the input question words during this search). When the word vectors are well trained, it is possible to find the correct answer (word ) using this method.
有些出人意料的是，这些问题可以通过对词的向量表示做简单代数运算来回答。要找到一个与 small 相似、其意义正如 biggest 与 big 相似的词，我们可以直接计算向量 。然后在向量空间中按余弦距离搜索最接近 的词，并把它作为问题的答案（搜索时丢弃输入问题中的词）。当词向量训练良好时，用这种方法有可能找到正确答案（词 ）。
Finally, we found that when we train high dimensional word vectors on a large amount of data, the resulting vectors can be used to answer very subtle semantic relationships between words, such as a city and the country it belongs to, e.g. France is to Paris as Germany is to Berlin. Word vectors with such semantic relationships could be used to improve many existing NLP applications, such as machine translation, information retrieval and question answering systems, and may enable other future applications yet to be invented.
最后，我们发现，当我们在大量数据上训练高维词向量时，所得向量可用于回答词之间非常微妙的语义关系，例如城市与所属国家，如 France 之于 Paris 正如 Germany 之于 Berlin。具有此类语义关系的词向量可用于改进许多现有 NLP 应用，例如机器翻译、信息检索和问答系统，并可能催生尚未发明的其他未来应用。
4.1 Task Description
4.1 任务描述
To measure quality of the word vectors, we define a comprehensive test set that contains five types of semantic questions, and nine types of syntactic questions. Two examples from each category are shown in Table 1. Overall, there are 8869 semantic and 10675 syntactic questions. The questions in each category were created in two steps: first, a list of similar word pairs was created manually. Then, a large list of questions is formed by connecting two word pairs. For example, we made a list of 68 large American cities and the states they belong to, and formed about 2.5K questions by picking two word pairs at random. We have included in our test set only single token words, thus multi-word entities are not present (such as New York).
为衡量词向量的质量，我们定义了一套综合测试集，包含五类语义问题和九类句法问题。每类各两个例子见表 1。总体有 8869 道语义题和 10675 道句法题。每类中的问题分两步创建：先手工制作相似词对列表，再把两对词连起来形成大量问题。例如，我们制作了 68 个美国大城市及其所属州的列表，并随机选取两对词形成约 2.5K 道题。测试集中只包含单词词元，因此没有多词实体（如 New York）。
We evaluate the overall accuracy for all question types, and for each question type separately (semantic, syntactic). Question is assumed to be correctly answered only if the closest word to the vector computed using the above method is exactly the same as the correct word in the question; synonyms are thus counted as mistakes. This also means that reaching 100% accuracy is likely to be impossible, as the current models do not have any input information about word morphology. However, we believe that usefulness of the word vectors for certain applications should be positively correlated with this accuracy metric. Further progress can be achieved by incorporating information about structure of words, especially for the syntactic questions.
我们评估所有问题类型的总体准确率，以及每一类（语义、句法）的准确率。只有当用上述方法算出的向量所对应的最近词与问题中的正确词完全相同，该问题才被认为回答正确；因此同义词也计为错误。这也意味着达到 100% 准确率很可能不可能，因为当前模型没有任何关于词形态的输入信息。不过，我们相信词向量对某些应用的有用性应与这一准确率指标正相关。进一步进展可通过纳入关于词结构的信息来实现，尤其是对句法问题。
Type of relationship Word Pair 1 Word Pair 2 Common capital city Athens Greece Oslo Norway All capital cities Astana Kazakhstan Harare Zimbabwe Currency Angola kwanza Iran rial City-in-state Chicago Illinois Stockton California Man-Woman brother sister grandson granddaughter Adjective to adverb apparent apparently rapid rapidly Opposite possibly impossibly ethical unethical Comparative great greater tough tougher Superlative easy easiest lucky luckiest Present Participle think thinking read reading Nationality adjective Switzerland Swiss Cambodia Cambodian Past tense walking walked swimming swam Plural nouns mouse mice dollar dollars Plural verbs work works speak speaks
Table 1: Examples of five types of semantic and nine types of syntactic questions in the Semantic-Syntactic Word Relationship test set.
| 关系类型 | 词对 1 | 词对 2 |
|---|---|---|
| Common capital city | Athens Greece | Oslo Norway |
| All capital cities | Astana Kazakhstan | Harare Zimbabwe |
| Currency | Angola kwanza | Iran rial |
| City-in-state | Chicago Illinois | Stockton California |
| Man-Woman | brother sister | grandson granddaughter |
| Adjective to adverb | apparent apparently | rapid rapidly |
| Opposite | possibly impossibly | ethical unethical |
| Comparative | great greater | tough tougher |
| Superlative | easy easiest | lucky luckiest |
| Present Participle | think thinking | read reading |
| Nationality adjective | Switzerland Swiss | Cambodia Cambodian |
| Past tense | walking walked | swimming swam |
| Plural nouns | mouse mice | dollar dollars |
| Plural verbs | work works | speak speaks |
表 1：语义—句法词关系测试集中五类语义问题和九类句法问题的例子。
4.2 Maximization of Accuracy
4.2 准确率最大化
We have used a Google News corpus for training the word vectors. This corpus contains about 6B tokens. We have restricted the vocabulary size to 1 million most frequent words. Clearly, we are facing time constrained optimization problem, as it can be expected that both using more data and higher dimensional word vectors will improve the accuracy. To estimate the best choice of model architecture for obtaining as good as possible results quickly, we have first evaluated models trained on subsets of the training data, with vocabulary restricted to the most frequent 30k words. The results using the CBOW architecture with different choice of word vector dimensionality and increasing amount of the training data are shown in Table 2.
我们使用 Google News 语料训练词向量。该语料约有 60 亿词元。我们将词汇表大小限制为最频繁的 100 万词。显然，我们面对的是时间受限的优化问题，因为可以预期：使用更多数据以及更高维的词向量都会提升准确率。为估计在尽快获得尽可能好的结果时模型架构的最佳选择，我们先在训练数据的子集上评估模型，词汇表限制为最频繁的 3 万词。使用 CBOW 架构、不同词向量维度、并逐步增加训练数据量的结果见表 2。
It can be seen that after some point, adding more dimensions or adding more training data provides diminishing improvements. So, we have to increase both vector dimensionality and the amount of the training data together. While this observation might seem trivial, it must be noted that it is currently popular to train word vectors on relatively large amounts of data, but with insufficient size (such as 50 - 100). Given Equation 4, increasing amount of training data twice results in about the same increase of computational complexity as increasing vector size twice.
可以看出，过了某一点之后，再增加维度或再增加训练数据，改进会递减。因此，我们必须同时增加向量维度和训练数据量。尽管这一观察看似平凡，但必须指出：当前流行的做法是在相对大量的数据上训练词向量，但维度不足（如 50–100）。由式 4，把训练数据量增加一倍，所带来的计算复杂度增加，大约与把向量维度增加一倍相当。
Dimensionality / Training words 24M 49M 98M 196M 391M 783M 50 13.4 15.7 18.6 19.1 22.5 23.2 100 19.4 23.1 27.8 28.7 33.4 32.2 300 23.2 29.2 35.3 38.6 43.7 45.9 600 24.0 30.1 36.5 40.8 46.6 50.4
Table 2: Accuracy on subset of the Semantic-Syntactic Word Relationship test set, using word vectors from the CBOW architecture with limited vocabulary. Only questions containing words from the most frequent 30k words are used.
| 维度 / 训练词数 | 24M | 49M | 98M | 196M | 391M | 783M |
|---|---|---|---|---|---|---|
| 50 | 13.4 | 15.7 | 18.6 | 19.1 | 22.5 | 23.2 |
| 100 | 19.4 | 23.1 | 27.8 | 28.7 | 33.4 | 32.2 |
| 300 | 23.2 | 29.2 | 35.3 | 38.6 | 43.7 | 45.9 |
| 600 | 24.0 | 30.1 | 36.5 | 40.8 | 46.6 | 50.4 |
表 2：在语义—句法词关系测试集子集上的准确率，使用来自词汇表受限的 CBOW 架构的词向量。只使用包含最频繁 3 万词中的词的问题。
For the experiments reported in Tables 2 and 4, we used three training epochs with stochastic gradient descent and backpropagation. We chose starting learning rate 0.025 and decreased it linearly, so that it approaches zero at the end of the last training epoch.
对表 2 和表 4 所报告的实验，我们使用随机梯度下降和反向传播训练三个 epoch。我们选择起始学习率 0.025，并线性递减，使其在最后一个训练 epoch 结束时接近零。
4.3 Comparison of Model Architectures
4.3 模型架构比较
First we compare different model architectures for deriving the word vectors using the same training data and using the same dimensionality of 640 of the word vectors. In the further experiments, we use full set of questions in the new Semantic-Syntactic Word Relationship test set, i.e. unrestricted to the 30k vocabulary. We also include results on a test set introduced in [20] that focuses on syntactic similarity between words³.
我们首先在相同训练数据、相同词向量维度 640 的条件下，比较用于导出词向量的不同模型架构。在后续实验中，我们使用新的语义—句法词关系测试集的全部问题，即不限于 3 万词词汇表。我们还纳入了 [20] 引入的、聚焦于词之间句法相似度的测试集上的结果³。
3 We thank Geoff Zweig for providing us the test set.
3 感谢 Geoff Zweig 为我们提供测试集。
The training data consists of several LDC corpora and is described in detail in [18] (320M words, 82K vocabulary). We used these data to provide a comparison to a previously trained recurrent neural network language model that took about 8 weeks to train on a single CPU. We trained a feedforward NNLM with the same number of 640 hidden units using the DistBelief parallel training [6], using a history of 8 previous words (thus, the NNLM has more parameters than the RNNLM, as the projection layer has size ).
训练数据由若干 LDC 语料组成，详见 [18]（3.2 亿词，8.2 万词词汇表）。我们使用这些数据，以便与此前训练的、在单 CPU 上大约花了 8 周的循环神经网络语言模型进行比较。我们用 DistBelief 并行训练 [6] 训练了具有相同 640 个隐单元的前馈 NNLM，历史为 8 个先前词（因此该 NNLM 的参数比 RNNLM 多，因为投影层大小为 ）。
In Table 3, it can be seen that the word vectors from the RNN (as used in [20]) perform well mostly on the syntactic questions. The NNLM vectors perform significantly better than the RNN - this is not surprising, as the word vectors in the RNNLM are directly connected to a non-linear hidden layer. The CBOW architecture works better than the NNLM on the syntactic tasks, and about the same on the semantic one. Finally, the Skip-gram architecture works slightly worse on the syntactic task than the CBOW model (but still better than the NNLM), and much better on the semantic part of the test than all the other models.
表 3 可见，来自 RNN 的词向量（如 [20] 所用）主要在句法问题上表现良好。NNLM 向量显著优于 RNN——这并不奇怪，因为 RNNLM 中的词向量直接连到非线性隐层。CBOW 架构在句法任务上优于 NNLM，在语义任务上大致相当。最后，Skip-gram 架构在句法任务上略逊于 CBOW 模型（但仍优于 NNLM），而在测试集的语义部分上远优于所有其他模型。
Model Semantic-Syntactic Word Relationship test set MSR Word Relatedness Architecture Semantic Accuracy [%] Syntactic Accuracy [%] Test Set [20] RNNLM 9 36 35 NNLM 23 53 47 CBOW 24 64 61 Skip-gram 55 59 56
Table 3: Comparison of architectures using models trained on the same data, with 640-dimensional word vectors. The accuracies are reported on our Semantic-Syntactic Word Relationship test set, and on the syntactic relationship test set of [20]
| 模型架构 | 语义—句法词关系测试集 语义准确率 [%] | 语义—句法词关系测试集 句法准确率 [%] | MSR 词相关测试集 [20] |
|---|---|---|---|
| RNNLM | 9 | 36 | 35 |
| NNLM | 23 | 53 | 47 |
| CBOW | 24 | 64 | 61 |
| Skip-gram | 55 | 59 | 56 |
表 3：在相同数据上训练、640 维词向量的架构比较。准确率报告在我们的语义—句法词关系测试集以及 [20] 的句法关系测试集上。
Next, we evaluated our models trained using one CPU only and compared the results against publicly available word vectors. The comparison is given in Table 4. The CBOW model was trained on subset of the Google News data in about a day, while training time for the Skip-gram model was about three days.
接下来，我们评估仅用一个 CPU 训练的模型，并把结果与公开可得的词向量比较。比较见表 4。CBOW 模型在 Google News 数据的子集上大约训练了一天，而 Skip-gram 模型的训练时间大约是三天。
Model Vector Training Accuracy [%] Dimensionality words Semantic Syntactic Total Collobert-Weston NNLM 50 660M 9.3 12.3 11.0 Turian NNLM 50 37M 1.4 2.6 2.1 Turian NNLM 200 37M 1.4 2.2 1.8 Mnih NNLM 50 37M 1.8 9.1 5.8 Mnih NNLM 100 37M 3.3 13.2 8.8 Mikolov RNNLM 80 320M 4.9 18.4 12.7 Mikolov RNNLM 640 320M 8.6 36.5 24.6 Huang NNLM 50 990M 13.3 11.6 12.3 Our NNLM 20 6B 12.9 26.4 20.3 Our NNLM 50 6B 27.9 55.8 43.2 Our NNLM 100 6B 34.2 64.5 50.8 CBOW 300 783M 15.5 53.1 36.1 Skip-gram 300 783M 50.0 55.9 53.3
Table 4: Comparison of publicly available word vectors on the Semantic-Syntactic Word Relationship test set, and word vectors from our models. Full vocabularies are used.
| 模型 | 向量维度 | 训练词数 | 语义准确率 [%] | 句法准确率 [%] | 总准确率 [%] |
|---|---|---|---|---|---|
| Collobert-Weston NNLM | 50 | 660M | 9.3 | 12.3 | 11.0 |
| Turian NNLM | 50 | 37M | 1.4 | 2.6 | 2.1 |
| Turian NNLM | 200 | 37M | 1.4 | 2.2 | 1.8 |
| Mnih NNLM | 50 | 37M | 1.8 | 9.1 | 5.8 |
| Mnih NNLM | 100 | 37M | 3.3 | 13.2 | 8.8 |
| Mikolov RNNLM | 80 | 320M | 4.9 | 18.4 | 12.7 |
| Mikolov RNNLM | 640 | 320M | 8.6 | 36.5 | 24.6 |
| Huang NNLM | 50 | 990M | 13.3 | 11.6 | 12.3 |
| Our NNLM | 20 | 6B | 12.9 | 26.4 | 20.3 |
| Our NNLM | 50 | 6B | 27.9 | 55.8 | 43.2 |
| Our NNLM | 100 | 6B | 34.2 | 64.5 | 50.8 |
| CBOW | 300 | 783M | 15.5 | 53.1 | 36.1 |
| Skip-gram | 300 | 783M | 50.0 | 55.9 | 53.3 |
表 4：公开可得词向量与我们模型所得词向量在语义—句法词关系测试集上的比较。使用完整词汇表。
For experiments reported further, we used just one training epoch (again, we decrease the learning rate linearly so that it approaches zero at the end of training). Training a model on twice as much data using one epoch gives comparable or better results than iterating over the same data for three epochs, as is shown in Table 5, and provides additional small speedup.
对后续报告的实验，我们只使用一个训练 epoch（同样，我们线性递减学习率，使其在训练结束时接近零）。如表 5 所示，在两倍数据上用一个 epoch 训练模型，给出的结果与在同一数据上迭代三个 epoch 相当或更好，并额外提供少量加速。
Model Vector Training Accuracy [%] Training time Dimensionality words [days] Semantic Syntactic Total 3 epoch CBOW 300 783M 15.5 53.1 36.1 1 3 epoch Skip-gram 300 783M 50.0 55.9 53.3 3 1 epoch CBOW 300 783M 13.8 49.9 33.6 0.3 1 epoch CBOW 300 1.6B 16.1 52.6 36.1 0.6 1 epoch CBOW 600 783M 15.4 53.3 36.2 0.7 1 epoch Skip-gram 300 783M 45.6 52.2 49.2 1 1 epoch Skip-gram 300 1.6B 52.2 55.1 53.8 2 1 epoch Skip-gram 600 783M 56.7 54.5 55.5 2.5
Table 5: Comparison of models trained for three epochs on the same data and models trained for one epoch. Accuracy is reported on the full Semantic-Syntactic data set.
| 模型 | 向量维度 | 训练词数 | 语义 [%] | 句法 [%] | 总计 [%] | 训练时间 [天] |
|---|---|---|---|---|---|---|
| 3 epoch CBOW | 300 | 783M | 15.5 | 53.1 | 36.1 | 1 |
| 3 epoch Skip-gram | 300 | 783M | 50.0 | 55.9 | 53.3 | 3 |
| 1 epoch CBOW | 300 | 783M | 13.8 | 49.9 | 33.6 | 0.3 |
| 1 epoch CBOW | 300 | 1.6B | 16.1 | 52.6 | 36.1 | 0.6 |
| 1 epoch CBOW | 600 | 783M | 15.4 | 53.3 | 36.2 | 0.7 |
| 1 epoch Skip-gram | 300 | 783M | 45.6 | 52.2 | 49.2 | 1 |
| 1 epoch Skip-gram | 300 | 1.6B | 52.2 | 55.1 | 53.8 | 2 |
| 1 epoch Skip-gram | 600 | 783M | 56.7 | 54.5 | 55.5 | 2.5 |
表 5：在相同数据上训练三个 epoch 的模型与训练一个 epoch 的模型的比较。准确率报告在完整语义—句法数据集上。
4.4 Large Scale Parallel Training of Models
4.4 模型的大规模并行训练
As mentioned earlier, we have implemented various models in a distributed framework called DistBelief. Below we report the results of several models trained on the Google News 6B data set, with mini-batch asynchronous gradient descent and the adaptive learning rate procedure called Adagrad [7]. We used 50 to 100 model replicas during the training. The number of CPU cores is an estimate since the data center machines are shared with other production tasks, and the usage can fluctuate quite a bit. Note that due to the overhead of the distributed framework, the CPU usage of the CBOW model and the Skip-gram model are much closer to each other than their single-machine implementations. The result are reported in Table 6.
如前所述，我们在名为 DistBelief 的分布式框架中实现了各种模型。下面报告在 Google News 60 亿词数据集上、用小批量异步梯度下降和自适应学习率过程 Adagrad [7] 训练的若干模型的结果。训练中我们使用 50 到 100 个模型副本。CPU 核心数是估计值，因为数据中心机器与其他生产任务共享，用量可能有较大波动。注意，由于分布式框架的开销，CBOW 模型与 Skip-gram 模型的 CPU 用量比它们的单机实现彼此接近得多。结果见表 6。
Model Vector Training Accuracy [%] Training time Dimensionality words [days x CPU cores] Semantic Syntactic Total NNLM 100 6B 34.2 64.5 50.8 14 x 180 CBOW 1000 6B 57.3 68.9 63.7 2 x 140 Skip-gram 1000 6B 66.1 65.1 65.6 2.5 x 125
Table 6: Comparison of models trained using the DistBelief distributed framework. Note that training of NNLM with 1000-dimensional vectors would take too long to complete.
| 模型 | 向量维度 | 训练词数 | 语义 [%] | 句法 [%] | 总计 [%] | 训练时间 [天 × CPU 核] |
|---|---|---|---|---|---|---|
| NNLM | 100 | 6B | 34.2 | 64.5 | 50.8 | 14 × 180 |
| CBOW | 1000 | 6B | 57.3 | 68.9 | 63.7 | 2 × 140 |
| Skip-gram | 1000 | 6B | 66.1 | 65.1 | 65.6 | 2.5 × 125 |
表 6：使用 DistBelief 分布式框架训练的模型比较。注意，训练 1000 维向量的 NNLM 耗时过长而无法完成。
4.5 Microsoft Research Sentence Completion Challenge
4.5 微软研究院句子补全挑战
The Microsoft Sentence Completion Challenge has been recently introduced as a task for advancing language modeling and other NLP techniques [32]. This task consists of 1040 sentences, where one word is missing in each sentence and the goal is to select word that is the most coherent with the rest of the sentence, given a list of five reasonable choices. Performance of several techniques has been already reported on this set, including N-gram models, LSA-based model [32], log-bilinear model [24] and a combination of recurrent neural networks that currently holds the state of the art performance of 55.4% accuracy on this benchmark [19].
微软句子补全挑战近来被提出，作为推进语言建模及其他 NLP 技术的一项任务 [32]。该任务包含 1040 个句子，每个句子缺少一个词，目标是在五个合理选项中选出与句子其余部分最连贯的词。已有若干技术在该集合上报告了性能，包括 N-gram 模型、基于 LSA 的模型 [32]、对数双线性模型 [24]，以及目前保持该基准最好成绩 55.4% 准确率的循环神经网络组合 [19]。
We have explored the performance of Skip-gram architecture on this task. First, we train the 640-dimensional model on 50M words provided in [32]. Then, we compute score of each sentence in the test set by using the unknown word at the input, and predict all surrounding words in a sentence. The final sentence score is then the sum of these individual predictions. Using the sentence scores, we choose the most likely sentence.
我们探索了 Skip-gram 架构在该任务上的表现。首先，我们在 [32] 提供的 5000 万词上训练 640 维模型。然后，对测试集中每个句子，我们把未知词放在输入端，预测句子中所有周围词，计算得分。最终句子得分是这些单独预测的总和。利用句子得分，我们选择最可能的句子。
A short summary of some previous results together with the new results is presented in Table 7. While the Skip-gram model itself does not perform on this task better than LSA similarity, the scores from this model are complementary to scores obtained with RNNLMs, and a weighted combination leads to a new state of the art result 58.9% accuracy (59.2% on the development part of the set and 58.7% on the test part of the set).
表 7 简要汇总了若干先前结果以及新结果。虽然 Skip-gram 模型本身在该任务上并不比 LSA 相似度更好，但该模型的得分与 RNNLM 所得得分互补，加权组合带来了新的最好结果：58.9% 准确率（开发集部分 59.2%，测试集部分 58.7%）。
Architecture Accuracy [%] 4-gram [32] 39 Average LSA similarity [32] 49 Log-bilinear model [24] 54.8 RNNLMs [19] 55.4 Skip-gram 48.0 Skip-gram + RNNLMs 58.9
Table 7: Comparison and combination of models on the Microsoft Sentence Completion Challenge.
| 架构 | 准确率 [%] |
|---|---|
| 4-gram [32] | 39 |
| Average LSA similarity [32] | 49 |
| Log-bilinear model [24] | 54.8 |
| RNNLMs [19] | 55.4 |
| Skip-gram | 48.0 |
| Skip-gram + RNNLMs | 58.9 |
表 7：微软句子补全挑战上各模型的比较与组合。
5 Examples of the Learned Relationships
5 学到的关系示例
Table 8 shows words that follow various relationships. We follow the approach described above: the relationship is defined by subtracting two word vectors, and the result is added to another word. Thus for example, Paris - France + Italy = Rome. As it can be seen, accuracy is quite good, although there is clearly a lot of room for further improvements (note that using our accuracy metric that assumes exact match, the results in Table 8 would score only about 60%). We believe that word vectors trained on even larger data sets with larger dimensionality will perform significantly better, and will enable the development of new innovative applications. Another way to improve accuracy is to provide more than one example of the relationship. By using ten examples instead of one to form the relationship vector (we average the individual vectors together), we have observed improvement of accuracy of our best models by about 10% absolutely on the semantic-syntactic test.
表 8 展示了遵循各种关系的词。我们沿用上文方法：关系由两个词向量相减定义，再把结果加到另一个词上。因此例如 Paris − France + Italy = Rome。可以看出准确率相当好，尽管显然仍有很大改进空间（注意，若使用我们要求精确匹配的准确率指标，表 8 的结果大约只得 60%）。我们相信，在更大的数据集上用更大维度训练的词向量会显著更好，并将推动新的创新应用的发展。提高准确率的另一途径是提供不止一个关系例子。用十个例子而不是一个来形成关系向量（我们把各个向量平均起来），我们观察到最好模型在语义—句法测试上的准确率绝对提升约 10%。
It is also possible to apply the vector operations to solve different tasks. For example, we have observed good accuracy for selecting out-of-the-list words, by computing average vector for a list of words, and finding the most distant word vector. This is a popular type of problems in certain human intelligence tests. Clearly, there is still a lot of discoveries to be made using these techniques.
也可以把向量运算用于解决不同任务。例如，我们观察到在选出列表外的词方面有很好的准确率：计算词列表的平均向量，再找最远的词向量。这是某些人类智力测验中常见的一类问题。显然，用这些技术还有许多发现有待做出。
Relationship Example 1 Example 2 Example 3 France - Paris Italy: Rome Japan: Tokyo Florida: Tallahassee big - bigger small: larger cold: colder quick: quicker Miami - Florida Baltimore: Maryland Dallas: Texas Kona: Hawaii Einstein - scientist Messi: midfielder Mozart: violinist Picasso: painter Sarkozy - France Berlusconi: Italy Merkel: Germany Koizumi: Japan copper - Cu zinc: Zn gold: Au uranium: plutonium Berlusconi - Silvio Sarkozy: Nicolas Putin: Medvedev Obama: Barack Microsoft - Windows Google: Android IBM: Linux Apple: iPhone Microsoft - Ballmer Google: Yahoo IBM: McNealy Apple: Jobs Japan - sushi Germany: bratwurst France: tapas USA: pizza
Table 8: Examples of the word pair relationships, using the best word vectors from Table 4 (Skip-gram model trained on 783M words with 300 dimensionality).
| 关系 | 例子 1 | 例子 2 | 例子 3 |
|---|---|---|---|
| France − Paris | Italy: Rome | Japan: Tokyo | Florida: Tallahassee |
| big − bigger | small: larger | cold: colder | quick: quicker |
| Miami − Florida | Baltimore: Maryland | Dallas: Texas | Kona: Hawaii |
| Einstein − scientist | Messi: midfielder | Mozart: violinist | Picasso: painter |
| Sarkozy − France | Berlusconi: Italy | Merkel: Germany | Koizumi: Japan |
| copper − Cu | zinc: Zn | gold: Au | uranium: plutonium |
| Berlusconi − Silvio | Sarkozy: Nicolas | Putin: Medvedev | Obama: Barack |
| Microsoft − Windows | Google: Android | IBM: Linux | Apple: iPhone |
| Microsoft − Ballmer | Google: Yahoo | IBM: McNealy | Apple: Jobs |
| Japan − sushi | Germany: bratwurst | France: tapas | USA: pizza |
表 8：词对关系示例，使用表 4 中最好的词向量（在 7.83 亿词上训练的 300 维 Skip-gram 模型）。
6 Conclusion
6 结论
In this paper we studied the quality of vector representations of words derived by various models on a collection of syntactic and semantic language tasks. We observed that it is possible to train high quality word vectors using very simple model architectures, compared to the popular neural network models (both feedforward and recurrent). Because of the much lower computational complexity, it is possible to compute very accurate high dimensional word vectors from a much larger data set. Using the DistBelief distributed framework, it should be possible to train the CBOW and Skip-gram models even on corpora with one trillion words, for basically unlimited size of the vocabulary. That is several orders of magnitude larger than the best previously published results for similar models.
本文研究了各种模型在一组句法和语义语言任务上导出的词向量表示的质量。我们观察到，与流行的神经网络模型（前馈和循环）相比，可以用非常简单的模型架构训练出高质量词向量。由于计算复杂度低得多，有可能从大得多的数据集中计算出非常准确的高维词向量。使用 DistBelief 分布式框架，应当有可能甚至在万亿词语料上训练 CBOW 和 Skip-gram 模型，词汇表大小基本上不受限。这比此前发表的类似模型的最好结果大几个数量级。
An interesting task where the word vectors have recently been shown to significantly outperform the previous state of the art is the SemEval-2012 Task 2 [11]. The publicly available RNN vectors were used together with other techniques to achieve over 50% increase in Spearman’s rank correlation over the previous best result [31]. The neural network based word vectors were previously applied to many other NLP tasks, for example sentiment analysis [12] and paraphrase detection [28]. It can be expected that these applications can benefit from the model architectures described in this paper.
一个近来已表明词向量显著优于此前最好结果的有趣任务是 SemEval-2012 Task 2 [11]。公开可得的 RNN 向量与其他技术一起使用，使 Spearman 秩相关相对此前最好结果提升超过 50% [31]。基于神经网络的词向量此前已被应用于许多其他 NLP 任务，例如情感分析 [12] 和释义检测 [28]。可以预期，这些应用能从本文描述的模型架构中获益。
Our ongoing work shows that the word vectors can be successfully applied to automatic extension of facts in Knowledge Bases, and also for verification of correctness of existing facts. Results from machine translation experiments also look very promising. In the future, it would be also interesting to compare our techniques to Latent Relational Analysis [30] and others. We believe that our comprehensive test set will help the research community to improve the existing techniques for estimating the word vectors. We also expect that high quality word vectors will become an important building block for future NLP applications.
我们正在进行的工作表明，词向量可以成功用于知识库中事实的自动扩展，以及验证已有事实的正确性。机器翻译实验的结果也看起来很有希望。未来，将我们的技术与潜在关系分析 [30] 等进行比较也会很有意思。我们相信，我们的综合测试集将帮助研究社区改进估计词向量的现有技术。我们也预期，高质量词向量将成为未来 NLP 应用的重要构建块。
7 Follow-Up Work
7 后续工作
After the initial version of this paper was written, we published single-machine multi-threaded C++ code for computing the word vectors, using both the continuous bag-of-words and skip-gram architectures⁴. The training speed is significantly higher than reported earlier in this paper, i.e. it is in the order of billions of words per hour for typical hyperparameter choices. We also published more than 1.4 million vectors that represent named entities, trained on more than 100 billion words. Some of our follow-up work will be published in an upcoming NIPS 2013 paper [21].
在本文初稿写完之后，我们发布了用于计算词向量的单机多线程 C++ 代码，同时支持连续词袋和 skip-gram 架构⁴。训练速度显著高于本文先前报告的结果，即在典型超参数选择下约为每小时数十亿词。我们还发布了超过 140 万个表示命名实体的向量，它们在超过 1000 亿词上训练。我们的一些后续工作将发表在即将到来的 NIPS 2013 论文中 [21]。
4 The code is available at https://code.google.com/p/word2vec/
References
参考文献
[1] Y. Bengio, R. Ducharme, P. Vincent. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.
[2] Y. Bengio, Y. LeCun. Scaling learning algorithms towards AI. In: Large-Scale Kernel Machines, MIT Press, 2007.
[3] T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. Large language models in machine translation. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, 2007.
[4] R. Collobert and J. Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In International Conference on Machine Learning, ICML, 2008.
[5] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa. Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research, 12:2493-2537, 2011.
[6] J. Dean, G.S. Corrado, R. Monga, K. Chen, M. Devin, Q.V. Le, M.Z. Mao, M.A. Ranzato, A. Senior, P. Tucker, K. Yang, A. Y. Ng., Large Scale Distributed Deep Networks, NIPS, 2012.
[7] J.C. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011.
[8] J. Elman. Finding Structure in Time. Cognitive Science, 14, 179-211, 1990.
[9] Eric H. Huang, R. Socher, C. D. Manning and Andrew Y. Ng. Improving Word Representations via Global Context and Multiple Word Prototypes. In: Proc. Association for Computational Linguistics, 2012.
[10] G.E. Hinton, J.L. McClelland, D.E. Rumelhart. Distributed representations. In: Parallel distributed processing: Explorations in the microstructure of cognition. Volume 1: Foundations, MIT Press, 1986.
[11] D.A. Jurgens, S.M. Mohammad, P.D. Turney, K.J. Holyoak. Semeval-2012 task 2: Measuring degrees of relational similarity. In: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), 2012.
[12] A.L. Maas, R.E. Daly, P.T. Pham, D. Huang, A.Y. Ng, and C. Potts. Learning word vectors for sentiment analysis. In Proceedings of ACL, 2011.
[13] T. Mikolov. Language Modeling for Speech Recognition in Czech, Masters thesis, Brno University of Technology, 2007.
[14] T. Mikolov, J. Kopecký, L. Burget, O. Glembek and J. Černocký. Neural network based language models for higly inflective languages, In: Proc. ICASSP 2009.
[15] T. Mikolov, M. Karafiát, L. Burget, J. Černocký, S. Khudanpur. Recurrent neural network based language model, In: Proceedings of Interspeech, 2010.
[16] T. Mikolov, S. Kombrink, L. Burget, J. Černocký, S. Khudanpur. Extensions of recurrent neural network language model, In: Proceedings of ICASSP 2011.
[17] T. Mikolov, A. Deoras, S. Kombrink, L. Burget, J. Černocký. Empirical Evaluation and Combination of Advanced Language Modeling Techniques, In: Proceedings of Interspeech, 2011.
[18] T. Mikolov, A. Deoras, D. Povey, L. Burget, J. Černocký. Strategies for Training Large Scale Neural Network Language Models, In: Proc. Automatic Speech Recognition and Understanding, 2011.
[19] T. Mikolov. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012.
[20] T. Mikolov, W.T. Yih, G. Zweig. Linguistic Regularities in Continuous Space Word Representations. NAACL HLT 2013.
[21] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed Representations of Words and Phrases and their Compositionality. Accepted to NIPS 2013.
[22] A. Mnih, G. Hinton. Three new graphical models for statistical language modelling. ICML, 2007.
[23] A. Mnih, G. Hinton. A Scalable Hierarchical Distributed Language Model. Advances in Neural Information Processing Systems 21, MIT Press, 2009.
[24] A. Mnih, Y.W. Teh. A fast and simple algorithm for training neural probabilistic language models. ICML, 2012.
[25] F. Morin, Y. Bengio. Hierarchical Probabilistic Neural Network Language Model. AISTATS, 2005.
[26] D. E. Rumelhart, G. E. Hinton, R. J. Williams. Learning internal representations by back-propagating errors. Nature, 323:533.536, 1986.
[27] H. Schwenk. Continuous space language models. Computer Speech and Language, vol. 21, 2007.
[28] R. Socher, E.H. Huang, J. Pennington, A.Y. Ng, and C.D. Manning. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In NIPS, 2011.
[29] J. Turian, L. Ratinov, Y. Bengio. Word Representations: A Simple and General Method for Semi-Supervised Learning. In: Proc. Association for Computational Linguistics, 2010.
[30] P. D. Turney. Measuring Semantic Similarity by Latent Relational Analysis. In: Proc. International Joint Conference on Artificial Intelligence, 2005.
[31] A. Zhila, W.T. Yih, C. Meek, G. Zweig, T. Mikolov. Combining Heterogeneous Models for Measuring Relational Similarity. NAACL HLT 2013.
[32] G. Zweig, C.J.C. Burges. The Microsoft Research Sentence Completion Challenge, Microsoft Research Technical Report MSR-TR-2011-129, 2011.