GloVe: Global Vectors for Word Representation

GloVe：词表征的全局向量

Jeffrey Pennington, Richard Socher, Christopher D. Manning
Computer Science Department, Stanford University, Stanford, CA 94305
jpennin@stanford.edu, richard@socher.org, manning@stanford.edu

Abstract

摘要

Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic, but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word cooccurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful substructure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition.

近期学习词向量空间表征的方法，已能用向量运算捕捉细粒度语义与句法规律，但这些规律从何而来仍不透明。我们分析并明确指出：要让这类规律在词向量中出现，模型需要具备哪些性质。结果是一个新的全局对数双线性回归模型，兼取文献中两大模型族之长——全局矩阵分解与局部上下文窗口方法。模型只在词–词共现矩阵的非零元上训练，而不必扫整张稀疏矩阵或语料里每一个上下文窗口，从而高效利用统计信息。所得向量空间具有有意义的子结构：在近期词类比任务上达到 75%；在相似度任务与命名实体识别上也优于相关模型。

1 Introduction

1 引言

Semantic vector space models of language represent each word with a real-valued vector. These vectors can be used as features in a variety of applications, such as information retrieval (Manning et al., 2008), document classification (Sebastiani, 2002), question answering (Tellex et al., 2003), named entity recognition (Turian et al., 2010), and parsing (Socher et al., 2013). Most word vector methods rely on the distance or angle between pairs of word vectors as the primary method for evaluating the intrinsic quality of such a set of word representations. Recently, Mikolov et al. (2013c) introduced a new evaluation scheme based on word analogies that probes the finer structure of the word vector space by examining not the scalar distance between word vectors, but rather their various dimensions of difference. For example, the analogy “king is to queen as man is to woman” should be encoded in the vector space by the vector equation king − queen = man − woman. This evaluation scheme favors models that produce dimensions of meaning, thereby capturing the multi-clustering idea of distributed representations (Bengio, 2009).

语言的语义向量空间模型用实值向量表示每个词。这些向量可用作多种应用的特征，如信息检索（Manning et al., 2008）、文档分类（Sebastiani, 2002）、问答（Tellex et al., 2003）、命名实体识别（Turian et al., 2010）以及句法分析（Socher et al., 2013）。多数词向量方法主要靠词向量对之间的距离或夹角来评估表征内在质量。近来，Mikolov et al. (2013c) 提出基于词类比的新评测：不只看标量距离，而考察差分的各个维度。例如，类比「king 之于 queen 如同 man 之于 woman」应在向量空间中由方程 king − queen = man − woman 编码。该方案偏好能产出「意义维度」的模型，从而体现分布式表征的多聚类思想（Bengio, 2009）。

The two main model families for learning word vectors are: 1) global matrix factorization methods, such as latent semantic analysis (LSA) (Deerwester et al., 1990) and 2) local context window methods, such as the skip-gram model of Mikolov et al. (2013c). Currently, both families suffer significant drawbacks. While methods like LSA efficiently leverage statistical information, they do relatively poorly on the word analogy task, indicating a sub-optimal vector space structure. Methods like skip-gram may do better on the analogy task, but they poorly utilize the statistics of the corpus since they train on separate local context windows instead of on global co-occurrence counts.

学习词向量的两大模型族是：1）全局矩阵分解方法，如潜在语义分析（LSA）（Deerwester et al., 1990）；2）局部上下文窗口方法，如 Mikolov et al. (2013c) 的 skip-gram。两族目前都有明显短板。LSA 一类能高效利用统计信息，但在词类比上较差，说明向量空间结构欠佳；skip-gram 一类类比更好，却因在各局部窗口上分别训练、未直接用全局共现计数，而未能充分利用语料统计。

In this work, we analyze the model properties necessary to produce linear directions of meaning and argue that global log-bilinear regression models are appropriate for doing so. We propose a specific weighted least squares model that trains on global word-word co-occurrence counts and thus makes efficient use of statistics. The model produces a word vector space with meaningful substructure, as evidenced by its state-of-the-art performance of 75% accuracy on the word analogy dataset. We also demonstrate that our methods outperform other current methods on several word similarity tasks, and also on a common named entity recognition (NER) benchmark.

本文分析产生线性「意义方向」所需的模型性质，并论证全局对数双线性回归模型适于此事。我们提出一种在全局词–词共现计数上训练的加权最小二乘模型，从而高效利用统计。该模型得到具有有意义子结构的词向量空间，在词类比数据集上以 75% 准确率达到当时最优；在若干词相似度任务以及常见命名实体识别（NER）基准上也优于当时方法。

We provide the source code for the model as well as trained word vectors at http://nlp. stanford.edu/projects/glove/.

模型源码与训好的词向量见 http://nlp.stanford.edu/projects/glove/。

2 Related Work

2 相关工作

Matrix Factorization Methods. Matrix factorization methods for generating low-dimensional word representations have roots stretching as far back as LSA. These methods utilize low-rank approximations to decompose large matrices that capture statistical information about a corpus. The particular type of information captured by such matrices varies by application. In LSA, the matrices are of “term-document” type, i.e., the rows correspond to words or terms, and the columns correspond to different documents in the corpus. In contrast, the Hyperspace Analogue to Language (HAL) (Lund and Burgess, 1996), for example, utilizes matrices of “term-term” type, i.e., the rows and columns correspond to words and the entries correspond to the number of times a given word occurs in the context of another given word. A main problem with HAL and related methods is that the most frequent words contribute a disproportionate amount to the similarity measure: the number of times two words co-occur with the or and, for example, will have a large effect on their similarity despite conveying relatively little about their semantic relatedness. A number of techniques exist that addresses this shortcoming of HAL, such as the COALS method (Rohde et al., 2006), in which the co-occurrence matrix is first transformed by an entropyor correlation-based normalization. An advantage of this type of transformation is that the raw co-occurrence counts, which for a reasonably sized corpus might span 8 or 9 orders of magnitude, are compressed so as to be distributed more evenly in a smaller interval. A variety of newer models also pursue this approach, including a study (Bullinaria and Levy, 2007) that indicates that positive pointwise mutual information (PPMI) is a good transformation. More recently, a square root type transformation in the form of Hellinger PCA (HPCA) (Lebret and Collobert, 2014) has been suggested as an effective way of learning word representations. Shallow Window-Based Methods. Another approach is to learn word representations that aid in making predictions within local context windows. For example, Bengio et al. (2003) introduced a model that learns word vector representations as part of a simple neural network architecture for language modeling. Collobert and Weston (2008) decoupled the word vector training from the downstream training objectives, which paved the way for Collobert et al. (2011) to use the full context of a word for learning the word representations, rather than just the preceding context as is the case with language models. Recently, the importance of the full neural network structure for learning useful word representations has been called into question. The skip-gram and continuous bag-of-words (CBOW) models of Mikolov et al. (2013a) propose a simple single-layer architecture based on the inner product between two word vectors. Mnih and Kavukcuoglu (2013) also proposed closely-related vector log-bilinear models, vLBL and ivLBL, and Levy et al. (2014) proposed explicit word embeddings based on a PPMI metric. In the skip-gram and ivLBL models, the objective is to predict a word’s context given the word itself, whereas the objective in the CBOW and vLBL models is to predict a word given its context. Through evaluation on a word analogy task, these models demonstrated the capacity to learn linguistic patterns as linear relationships between the word vectors. Unlike the matrix factorization methods, the shallow window-based methods suffer from the disadvantage that they do not operate directly on the co-occurrence statistics of the corpus. Instead, these models scan context windows across the entire corpus, which fails to take advantage of the vast amount of repetition in the data.

矩阵分解方法。 用矩阵分解生成低维词表征可追溯到 LSA。这类方法对捕获语料统计信息的大矩阵做低秩近似；矩阵所含信息因应用而异。LSA 用「词–文档」型矩阵：行对应词/术语，列对应语料中不同文档。相对地，如 Hyperspace Analogue to Language（HAL）（Lund and Burgess, 1996）使用「词–词」型矩阵：行列皆为词，条目为某词出现在另一词上下文中的次数。HAL 及相关方法的主要问题是：最频繁词对相似度度量贡献过大——例如两词与 the、and 的共现次数会强烈影响相似度，却几乎不传达语义相关性。已有若干技术应对，如 COALS（Rohde et al., 2006）先对共现矩阵做基于熵或相关的归一化。这类变换的优点是：合理规模语料上可跨 8–9 个数量级的原始共现计数，被压到更均匀的较小区间。更新的模型也走类似路线，包括表明正点互信息（PPMI）是良好变换的研究（Bullinaria and Levy, 2007）。更近则有人以 Hellinger PCA（HPCA）形式的平方根类变换学习词表征（Lebret and Collobert, 2014）。浅层窗口方法。 另一路线是学习有助于在局部上下文窗口内做预测的词表征。例如 Bengio et al. (2003) 在简单神经语言模型结构中一并学习词向量；Collobert and Weston (2008) 把词向量训练与下游目标解耦，为 Collobert et al. (2011) 用词的完整上下文（而非语言模型仅用前文）学习表征铺平了道路。近来，完整神经网络结构对学有用词表征是否必要受到质疑。Mikolov et al. (2013a) 的 skip-gram 与连续词袋（CBOW）提出基于两词向量内积的简单单层结构；Mnih and Kavukcuoglu (2013) 提出相近的向量对数双线性模型 vLBL 与 ivLBL；Levy et al. (2014) 则基于 PPMI 度量提出显式词嵌入。skip-gram 与 ivLBL 的目标是给定词预测其上下文，CBOW 与 vLBL 则是给定上下文预测词。经词类比评估，这些模型展现出把语言模式学成词向量间线性关系的能力。与矩阵分解不同，浅层窗口方法的缺点是不直接操作语料共现统计，而是在整语料上扫描上下文窗口，未能利用数据中大量重复。

3 The GloVe Model

3 GloVe 模型

The statistics of word occurrences in a corpus is the primary source of information available to all unsupervised methods for learning word representations, and although many such methods now exist, the question still remains as to how meaning is generated from these statistics, and how the resulting word vectors might represent that meaning. In this section, we shed some light on this question. We use our insights to construct a new model for word representation which we call GloVe, for Global Vectors, because the global corpus statistics are captured directly by the model. First we establish some notation. Let the matrix of word-word co-occurrence counts be denoted by X, whose entries X i j tabulate the number of times word j occurs in the context of word i. Let X i = P k X ik be the number of times any word appears in the context of word i. Finally, let Pi j = P( j |i) = X i j /X i be the probability that word j appear in the

词在语料中的出现统计，是所有无监督词表征学习方法的主要信息来源。尽管方法已很多，意义如何从这些统计中生成、所得词向量又如何表示该意义，问题仍在。本节对此作一些澄清，并据此构造新模型，称为 GloVe（Global Vectors），因其直接捕获全局语料统计。先约定记号。令词–词共现计数矩阵为 $X$，条目 $X_{ij}$ 记录词 $j$ 出现在词 $i$ 上下文中的次数。令 $X_i=\sum_k X_{ik}$ 为任意词出现在词 $i$ 上下文中的次数。最后令 $P_{ij}=P(j|i)=X_{ij}/X_i$ 为词 $j$ 出现在词 $i$ 上下文中的概率。

Table 1: Co-occurrence probabilities for target words billion token corpus. Only in the ratio does noise fro cancel out, so that large values (much greater than 1 small values (much less than 1) correlate well with p Probability and Ratio

表 1：目标词 ice 与 steam 与选自约 60 亿词元语料的若干上下文词的共现概率。只有在比率中，来自 water、fashion 等非判别词的噪声才会抵消；远大于 1 的值与 ice 特有属性相关，远小于 1 的值与 steam 特有属性相关。

context of word i. We begin with a simple example that showcases how certain aspects of meaning can be extracted directly from co-occurrence probabilities. Consider two words i and j that exhibit a particular aspect of interest; for concreteness, suppose we are interested in the concept of thermodynamic phase, for which we might take i = ice and j = steam. The relationship of these words can be examined by studying the ratio of their co-occurrence probabilities with various probe words, k. For words k related to ice but not steam, say k = solid, we expect the ratio Pik /P j k will be large. Similarly, for words k related to steam but not ice, say k = gas, the ratio should be small. For words k like water or fashion, that are either related to both ice and steam, or to neither, the ratio should be close to one. Table 1 shows these probabilities and their ratios for a large corpus, and the numbers confirm these expectations. Compared to the raw probabilities, the ratio is better able to distinguish relevant words (solid and gas) from irrelevant words (water and fashion) and it is also better able to discriminate between the two relevant words. The above argument suggests that the appropriate starting point for word vector learning should be with ratios of co-occurrence probabilities rather than the probabilities themselves. Noting that the ratio Pik /P j k depends on three words i, j, and k, the most general model takes the form, F (wi , w j , w̃k ) =

我们从简单例子出发，展示某些意义侧面可直接从共现概率中提取。考虑展现某一关注侧面的两词 $i$ 与 $j$；具体地，取热力学相变概念，$i=\mathrm{ice}$，$j=\mathrm{steam}$。可通过考察它们与各探测词 $k$ 的共现概率之比来研究关系。对与 ice 相关而与 steam 无关的词（如 $k=\mathrm{solid}$），期望比率 $P_{ik}/P_{jk}$ 较大；对与 steam 相关而与 ice 无关的词（如 $k=\mathrm{gas}$），比率应较小；对 water 或 fashion 这类与二者都相关或都不相关的词，比率应接近 1。表 1 给出大规模语料上的概率与比率，数字印证了这些预期。相对原始概率，比率更能把相关词（solid、gas）与无关词（water、fashion）分开，也能更好区分两个相关词。上述论证表明：词向量学习的合适起点应是共现概率的比率而非概率本身。注意到比率 $P_{ik}/P_{jk}$ 依赖三词 $i,j,k$，最一般模型取形式 $F(w_i,w_j,\tilde{w}_k)=P_{ik}/P_{jk}$。

where w ∈ Rd are word vectors and w̃ ∈ Rd are separate context word vectors whose role will be discussed in Section 4.2. In this equation, the right-hand side is extracted from the corpus, and F may depend on some as-of-yet unspecified parameters. The number of possibilities for F is vast, but by enforcing a few desiderata we can select a unique choice. First, we would like F to encode

其中 $w\in\mathbb{R}^d$ 为词向量，$\tilde{w}\in\mathbb{R}^d$ 为单独的上下文词向量（作用见第 4.2 节）。方程右边由语料提取，$F$ 还可依赖尚未指定的参数。$F$ 的选择空间巨大，但施加若干期望性质后可唯一确定。首先，我们希望 $F$ 能编码

the information present the ratio Pik /P j k in the word vector space. Since vector spaces are inherently linear structures, the most natural way to do this is with vector differences. With this aim, we can restrict our consideration to those functions F that depend only on the difference of the two target words, modifying Eqn. (1) to, F (wi − w j , w̃k ) =

比率 $P_{ik}/P_{jk}$ 所含信息于词向量空间。向量空间本质是线性结构，最自然的做法是用向量差。因此可把考虑范围限制为只依赖两目标词之差的 $F$，将式 (1) 改为 $F(w_i-w_j,\tilde{w}_k)=P_{ik}/P_{jk}$。

Next, we note that the arguments of F in Eqn. (2) are vectors while the right-hand side is a scalar. While F could be taken to be a complicated function parameterized by, e.g., a neural network, doing so would obfuscate the linear structure we are trying to capture. To avoid this issue, we can first take the dot product of the arguments,   Pik F (wi − w j )T w̃k = , Pj k

接着注意：式 (2) 中 $F$ 的自变量是向量，右边却是标量。虽可把 $F$ 取成由神经网络等参数化的复杂函数，但这会模糊我们试图捕获的线性结构。为避免此问题，可先对自变量取点积：

which prevents F from mixing the vector dimensions in undesirable ways. Next, note that for word-word co-occurrence matrices, the distinction between a word and a context word is arbitrary and that we are free to exchange the two roles. To do so consistently, we must not only exchange w ↔ w̃ but also X ↔ X T . Our final model should be invariant under this relabeling, but Eqn. (3) is not. However, the symmetry can be restored in two steps. First, we require that F be a homomorphism between the groups (R, +) and (R>0 , ×), i.e.,   F (wTi w̃k ) F (wi − w j )T w̃k = , F (wTj w̃k )

这可防止 $F$ 以不良方式混合向量各维。其次，对词–词共现矩阵而言，词与上下文词的区分是任意的，二者角色可交换。要一致地交换，不仅需 $w\leftrightarrow\tilde{w}$，还要 $X\leftrightarrow X^T$。最终模型应在此重标记下不变，但式 (3) 并非如此。对称性可分两步恢复。首先要求 $F$ 是群 $(\mathbb{R},+)$ 与 $(\mathbb{R}_{>0},\times)$ 之间的同态，即

which, by Eqn. (3), is solved by, F (wTi w̃k ) = Pik =

由式 (3)，解为 $F(w_i^T \tilde{w}_k)=P_{ik}$。

The solution to Eqn. (4) is F = exp, or,

式 (4) 的解是 $F=\exp$，亦即

wTi w̃k = log(Pik ) = log(X ik ) − log(X i ) .

$$w_i^T \tilde{w}_k = \log(P_{ik}) = \log(X_{ik}) - \log(X_i).$$

Next, we note that Eqn. (6) would exhibit the exchange symmetry if not for the log(X i ) on the right-hand side. However, this term is independent of k so it can be absorbed into a bias bi for wi . Finally, adding an additional bias b̃k for w̃k restores the symmetry, wTi w̃k + bi + b̃k = log(X ik ) .

接着注意：若非右边的 $\log(X_i)$，式 (6) 将具有交换对称性。该项与 $k$ 无关，可吸收进 $w_i$ 的偏置 $b_i$。最后为 $\tilde{w}_k$ 再加偏置 $\tilde{b}_k$，对称性恢复：$w_i^T \tilde{w}_k + b_i + \tilde{b}_k = \log(X_{ik})$。

Eqn. (7) is a drastic simplification over Eqn. (1), but it is actually ill-defined since the logarithm diverges whenever its argument is zero. One resolution to this issue is to include an additive shift in the logarithm, log(X ik ) → log(1 + X ik ), which maintains the sparsity of X while avoiding the divergences. The idea of factorizing the log of the co-occurrence matrix is closely related to LSA and we will use the resulting model as a baseline in our experiments. A main drawback to this model is that it weighs all co-occurrences equally, even those that happen rarely or never. Such rare cooccurrences are noisy and carry less information than the more frequent ones — yet even just the zero entries account for 75–95% of the data in X, depending on the vocabulary size and corpus. We propose a new weighted least squares regression model that addresses these problems. Casting Eqn. (7) as a least squares problem and introducing a weighting function f (X i j ) into the cost function gives us the model J=

式 (7) 相对式 (1) 大幅简化，但其实不适定：对数在零处发散。一种解决是把对数改成 $\log(X_{ik})\rightarrow\log(1+X_{ik})$，既保持 $X$ 稀疏又避免发散。对共现矩阵取对数再分解的想法与 LSA 密切相关，我们将用所得模型作实验基线。该模型的主要缺点是对所有共现一视同仁，包括稀有或从未出现的；稀有共现噪声大、信息少——而仅零条目就占 $X$ 中 75%–95% 的数据（视词表与语料而定）。我们提出新的加权最小二乘回归模型来应对。把式 (7) 写成最小二乘并在代价中引入加权函数 $f(X_{ij})$，得到模型

  2 f X i j wTi w̃ j + bi + b̃ j − log X i j ,

$$J=\sum_{i,j=1}^{V} f(X_{ij})\big(w_i^T \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij}\big)^2,$$

(8) where V is the size of the vocabulary. The weighting function should obey the following properties: 1. f (0) = 0. If f is viewed as a continuous function, it should vanish as x → 0 fast enough that the lim x→0 f (x) log2 x is finite. 2. f (x) should be non-decreasing so that rare co-occurrences are not overweighted. 3. f (x) should be relatively small for large values of x, so that frequent co-occurrences are not overweighted. Of course a large number of functions satisfy these properties, but one class of functions that we found to work well can be parameterized as, ( (x/x max ) α if x < x max f (x) = (9) 1 otherwise .

其中 $V$ 为词表大小。加权函数应满足：1. $f(0)=0$；若视 $f$ 为连续函数，当 $x\rightarrow 0$ 时应衰减得足够快，使 $\lim_{x\rightarrow 0} f(x)\log^2 x$ 有限。2. $f(x)$ 应非减，以免稀有共现被过重加权。3. 对大的 $x$，$f(x)$ 应相对较小，以免频繁共现被过重加权。满足这些性质的函数很多，我们发现一类参数化形式效果良好：若 $x<x_{\max}$ 则 $f(x)=(x/x_{\max})^\alpha$，否则为 1。

Figure 1: Weighting function f with α = 3/4. The performance of the model depends weakly on the cutoff, which we fix to x max = 100 for all our experiments. We found that α = 3/4 gives a modest improvement over a linear version with α = 1. Although we offer only empirical motivation for choosing the value 3/4, it is interesting that a similar fractional power scaling was found to give the best performance in (Mikolov et al., 2013a). 3.1

图 1：加权函数 $f$（$\alpha=3/4$）。模型表现对截断点依赖较弱，实验中一律取 $x_{\max}=100$。我们发现 $\alpha=3/4$ 相对线性版本 $\alpha=1$ 有适度提升。虽仅有经验动机，有趣的是 Mikolov et al. (2013a) 也发现类似分数幂缩放效果最好。

Because all unsupervised methods for learning word vectors are ultimately based on the occurrence statistics of a corpus, there should be commonalities between the models. Nevertheless, certain models remain somewhat opaque in this regard, particularly the recent window-based methods like skip-gram and ivLBL. Therefore, in this subsection we show how these models are related to our proposed model, as defined in Eqn. (8). The starting point for the skip-gram or ivLBL methods is a model Q i j for the probability that word j appears in the context of word i. For concreteness, let us assume that Q i j is a softmax, exp(wTi w̃ j ) Q i j = PV . T k=1 exp(wi w̃k )

因为所有无监督词向量方法最终都基于语料出现统计，模型之间应有共通处。不过某些模型在此方面仍较不透明，尤其是 skip-gram、ivLBL 等近期窗口方法。因此本小节说明这些模型与我们在式 (8) 中提出的模型如何相关。skip-gram 或 ivLBL 的起点是词 $j$ 出现在词 $i$ 上下文中的概率模型 $Q_{ij}$。具体地，设 $Q_{ij}$ 为 softmax：$Q_{ij}=\exp(w_i^T \tilde{w}_j)/\sum_k\exp(w_i^T \tilde{w}_k)$。

Most of the details of these models are irrelevant for our purposes, aside from the the fact that they attempt to maximize the log probability as a context window scans over the corpus. Training proceeds in an on-line, stochastic fashion, but the implied global objective function can be written as, X J=− log Q i j . (11) i ∈corpus j ∈context(i)

对这些模型，多数细节与我们无关，关键是它们在上下文窗口扫过语料时试图最大化对数概率。训练以在线随机方式进行，但隐含的全局目标可写成 $J=-\sum_{i\in\mathrm{corpus}}\sum_{j\in\mathrm{context}(i)}\log Q_{ij}$。

Evaluating the normalization factor of the softmax for each term in this sum is costly. To allow for efficient training, the skip-gram and ivLBL models introduce approximations to Q i j . However, the sum in Eqn. (11) can be evaluated much more efficiently if we first group together those terms that have the same values for i and j, J=−

对和中每一项都计算 softmax 归一化因子代价很高。为高效训练，skip-gram 与 ivLBL 对 $Q_{ij}$ 引入近似。然而，若先把 $i,j$ 相同的项归并，式 (11) 的求和可以高效得多。

where we have used the fact that the number of like terms is given by the co-occurrence matrix X. P Recalling our notation for X i = k X ik and Pi j = X i j /X i , we can rewrite J as, J=−

其中同类项的个数由共现矩阵 $X$ 给出。记 $X_i=\sum_k X_{ik}$，$P_{ij}=X_{ij}/X_i$，可将 $J$ 重写。

(13) where H (Pi ,Q i ) is the cross entropy of the distributions Pi and Q i , which we define in analogy to X i . As a weighted sum of cross-entropy error, this objective bears some formal resemblance to the weighted least squares objective of Eqn. (8). In fact, it is possible to optimize Eqn. (13) directly as opposed to the on-line training methods used in the skip-gram and ivLBL models. One could interpret this objective as a “global skip-gram” model, and it might be interesting to investigate further. On the other hand, Eqn. (13) exhibits a number of undesirable properties that ought to be addressed before adopting it as a model for learning word vectors. To begin, cross entropy error is just one among many possible distance measures between probability distributions, and it has the unfortunate property that distributions with long tails are often modeled poorly with too much weight given to the unlikely events. Furthermore, for the measure to be bounded it requires that the model distribution Q be properly normalized. This presents a computational bottleneck owing to the sum over the whole vocabulary in Eqn. (10), and it would be desirable to consider a different distance measure that did not require this property of Q. A natural choice would be a least squares objective in which normalization factors in Q and P are discarded, X
Jˆ = X i P̂i j − Q̂ i j 2 (14) i, j

其中 $H(P_i,Q_i)$ 为分布 $P_i$ 与 $Q_i$ 的交叉熵。作为交叉熵误差的加权和，该目标与式 (8) 的加权最小二乘在形式上有些相似。事实上可直接优化该式，而非 skip-gram/ivLBL 的在线训练；可将其理解为「全局 skip-gram」，值得进一步研究。另一方面，该式有若干不良性质：交叉熵只是分布距离之一，长尾分布往往拟合差、对罕见事件权重过大；且度量有界要求模型分布 $Q$ 正确归一化，这因式 (10) 对全词表求和而成为计算瓶颈。自然的选择是丢掉 $Q$ 与 $P$ 中归一化因子的最小二乘目标。

where P̂i j = X i j and Q̂ i j = exp(wTi w̃ j ) are the unnormalized distributions. At this stage another problem emerges, namely that X i j often takes very large values, which can complicate the optimization. An effective remedy is to minimize the squared error of the logarithms of P̂ and Q̂ instead, X
Jˆ = X i log P̂i j − log Q̂ i j 2 i, j

其中 $\hat{P}_{ij}=X_{ij}$，$\hat{Q}_{ij}=\exp(w_i^T \tilde{w}_j)$ 为未归一化分布。此时另一问题出现：$X_{ij}$ 常取很大值，优化困难。有效办法是改为最小化对数的平方误差。

X i wTi w̃ j − log X i j 2 .

$$\sum_{i,j} X_i\big(w_i^T \tilde{w}_j - \log X_{ij}\big)^2.$$

Finally, we observe that while the weighting factor X i is preordained by the on-line training method inherent to the skip-gram and ivLBL models, it is by no means guaranteed to be optimal. In fact, Mikolov et al. (2013a) observe that performance can be increased by filtering the data so as to reduce the effective value of the weighting factor for frequent words. With this in mind, we introduce a more general weighting function, which we are free to take to depend on the context word as well. The result is, X
Jˆ = (16) f (X i j ) wTi w̃ j − log X i j 2 , i, j

最后注意到：加权因子 $X_i$ 由 skip-gram/ivLBL 固有的在线训练方式预先定下，未必最优。事实上 Mikolov et al. (2013a) 观察到过滤数据以降低频繁词有效权重可提升表现。据此我们引入更一般的加权函数，也可依赖上下文词。结果为 $\hat{J}=\sum_{i,j} f(X_{ij})(w_i^T \tilde{w}_j-\log X_{ij})^2$，等价于先前推导的式 (8)。

which is equivalent1 to the cost function of Eqn. (8), which we derived previously. 3.2

3.2 模型复杂度

As can be seen from Eqn. (8) and the explicit form of the weighting function f (X ), the computational complexity of the model depends on the number of nonzero elements in the matrix X. As this number is always less than the total number of entries of the matrix, the model scales no worse than O(|V | 2 ). At first glance this might seem like a substantial improvement over the shallow windowbased approaches, which scale with the corpus size, |C|. However, typical vocabularies have hundreds of thousands of words, so that |V | 2 can be in the hundreds of billions, which is actually much larger than most corpora. For this reason it is important to determine whether a tighter bound can be placed on the number of nonzero elements of X. In order to make any concrete statements about the number of nonzero elements in X, it is necessary to make some assumptions about the distribution of word co-occurrences. In particular, we will assume that the number of co-occurrences of word i with word j, X i j , can be modeled as a power-law function of the frequency rank of that word pair, ri j : k Xi j = . (17) (r i j ) α

由式 (8) 与加权函数 $f(X)$ 的显式形式可见，模型计算复杂度取决于矩阵 $X$ 的非零元个数；该数总小于矩阵总条目数，故规模不差于 $O(|V|^2)$。乍看相对随语料规模 $|C|$ 增长的浅层窗口方法是大幅改进。但典型词表有数十万词，$|V|^2$ 可达千亿量级，实际远大于多数语料。因此有必要对 $X$ 非零元个数给出更紧界。为此需对词共现分布作假设：设词对共现次数 $X_{ij}$ 可建模为其频次秩 $r_{ij}$ 的幂律函数。

1 We could also include bias terms in Eqn. (16).

注：式 (16) 中也可加入偏置项。

The total number of words in the corpus is proportional to the sum over all elements of the cooccurrence matrix X, |C| ∼

语料总词数与共现矩阵 $X$ 所有元素之和成正比。

where we have rewritten the last sum in terms of the generalized harmonic number Hn, m . The upper limit of the sum, |X |, is the maximum frequency rank, which coincides with the number of nonzero elements in the matrix X. This number is also equal to the maximum value of r in Eqn. (17) such that X i j ≥ 1, i.e., |X | = k 1/α . Therefore we can write Eqn. (18) as, |C| ∼ |X | α H | X |,α .

其中最后一和用广义调和数改写。求和上限 $|X|$ 为最大频次秩，亦即矩阵 $X$ 非零元个数；也等于式 (17) 中使 $X_{ij}\ge 1$ 的最大 $r$，即 $|X|=k^{1/\alpha}$。

We are interested in how |X | is related to |C| when both numbers are large; therefore we are free to expand the right hand side of the equation for large |X |. For this purpose we use the expansion of generalized harmonic numbers (Apostol, 1976), H x, s = giving,

我们关心 $|X|$ 与 $|C|$ 都很大时二者关系，因而可对大 $|X|$ 展开右边。利用广义调和数展开（Apostol, 1976）。

where ζ (s) is the Riemann zeta function. In the limit that X is large, only one of the two terms on the right hand side of Eqn. (21) will be relevant, and which term that is depends on whether α > 1, ( O(|C|) if α < 1, |X | = (22) O(|C| 1/α ) if α > 1. For the corpora studied in this article, we observe that X i j is well-modeled by Eqn. (17) with α = 1.25. In this case we have that |X | = O(|C| 0.8 ). Therefore we conclude that the complexity of the model is much better than the worst case O(V 2 ), and in fact it does somewhat better than the on-line window-based methods which scale like O(|C|).

其中 $\zeta$ 为黎曼 zeta 函数。在 $X$ 很大的极限下，式 (21) 右边两项只有一项主导，取决于 $\alpha$ 是否大于 1：若 $\alpha<1$ 则 $|X|=O(|C|)$；若 $\alpha>1$ 则 $|X|=O(|C|^{1/\alpha})$。本文语料上观察到 $X_{ij}$ 可用 $\alpha=1.25$ 的式 (17) 良好建模，此时 $|X|=O(|C|^{0.8})$。因此模型复杂度远好于最坏情形 $O(V^2)$，且略优于规模为 $O(|C|)$ 的在线窗口方法。

4 Experiments

4 实验

4.1 Evaluation methods

4.1 评估方法

We conduct experiments on the word analogy task of Mikolov et al. (2013a), a variety of word similarity tasks, as described in (Luong et al., 2013), and on the CoNLL-2003 shared benchmark

我们在 Mikolov et al. (2013a) 的词类比任务、Luong et al. (2013) 所述多种词相似度任务，以及 CoNLL-2003 NER 共享任务基准上做实验。

Table 2: Results on the word analogy task, given as percent accuracy. Underlined scores are best within groups of similarly-sized models; bold scores are best overall. HPCA vectors are publicly available2 ; (i)vLBL results are from (Mnih et al., 2013); skip-gram (SG) and CBOW results are from (Mikolov et al., 2013a,b); we trained SG† and CBOW† using the word2vec tool3 . See text for details and a description of the SVD models. Model ivLBL HPCA GloVe SG CBOW vLBL ivLBL GloVe SVD SVD-S SVD-L CBOW† SG† GloVe CBOW SG SVD-L GloVe

表 2：词类比任务结果（准确率百分比）。下划线为相近规模模型组内最佳，粗体为总体最佳。

dataset for NER (Tjong Kim Sang and De Meulder, 2003). Word analogies. The word analogy task consists of questions like, “a is to b as c is to ?” The dataset contains 19,544 such questions, divided into a semantic subset and a syntactic subset. The semantic questions are typically analogies about people or places, like “Athens is to Greece as Berlin is to ?”. The syntactic questions are typically analogies about verb tenses or forms of adjectives, for example “dance is to dancing as fly is to ?”. To correctly answer the question, the model should uniquely identify the missing term, with only an exact correspondence counted as a correct match. We answer the question “a is to b as c is to ?” by finding the word d whose representation wd is closest to wb − wa + wc according to the cosine similarity.4 2 http://lebret.ch/words/

词类比。 任务形如「$a$ 之于 $b$ 如同 $c$ 之于？」数据集含 19,544 题，分语义与句法子集。语义题多为人物或地点类比（如「Athens 之于 Greece 如同 Berlin 之于？」）；句法题多为动词时态或形容词形态（如「dance 之于 dancing 如同 fly 之于？」）。正确作答须唯一识别缺失项，仅精确匹配计对。我们找表示 $w_d$ 按余弦相似度最接近 $w_b-w_a+w_c$ 的词 $d$ 作为答案。

Figure 2: Accuracy on the analogy task as function o trained on the 6 billion token corpus. In (a), the wind Word similarity. While the analogy task is our primary focus since it tests for interesting vector space substructures, we also evaluate our model on a variety of word similarity tasks in Table 3. These include WordSim-353 (Finkelstein et al., 2001), MC (Miller and Charles, 1991), RG (Rubenstein and Goodenough, 1965), SCWS (Huang et al., 2012), and RW (Luong et al., 2013). Named entity recognition. The CoNLL-2003 English benchmark dataset for NER is a collection of documents from Reuters newswire articles, annotated with four entity types: person, location, organization, and miscellaneous. We train models on CoNLL-03 training data on test on three datasets: 1) ConLL-03 testing data, 2) ACE Phase 2 (2001-02) and ACE-2003 data, and 3) MUC7 Formal Run test set. We adopt the BIO2 annotation standard, as well as all the preprocessing steps described in (Wang and Manning, 2013). We use a comprehensive set of discrete features that comes with the standard distribution of the Stanford NER model (Finkel et al., 2005). A total of 437,905 discrete features were generated for the CoNLL2003 training dataset. In addition, 50-dimensional vectors for each word of a five-word context are added and used as continuous features. With these features as input, we trained a conditional random field (CRF) with exactly the same setup as the CRFjoin model of (Wang and Manning, 2013). 4.2 Corpora and training details We trained our model on five corpora of varying sizes: a 2010 Wikipedia dump with 1 billion tokens; a 2014 Wikipedia dump with 1.6 billion tokens; Gigaword 5 which has 4.3 billion tokens; the combination Gigaword5 + Wikipedia2014, which the analogy task. This number is evaluated on a subset of the dataset so it is not included in Table 2. 3C OS M UL performed worse than cosine similarity in almost all of our experiments.

图 2：类比任务准确率随向量维度与窗口大小/类型的变化。所有模型在 60 亿词元语料上训练。词相似度。 类比是主焦点（检验有趣的向量空间子结构），我们也在表 3 的多种词相似度任务上评估，包括 WordSim-353、MC、RG、SCWS 与 RW。命名实体识别。 CoNLL-2003 英文 NER 基准来自路透社新闻标注，含人物、地点、组织与杂类四类实体。我们在 CoNLL-03 训练数据上训练，并在 CoNLL-03 测试、ACE 以及 MUC7 上测试；采用 BIO2 标注与 Wang and Manning (2013) 的预处理。离散特征来自 Stanford NER 标准配置（CoNLL-2003 训练集约 437,905 维），并加入五词窗口内各词的 50 维向量作连续特征，训练与 Wang and Manning (2013) CRF-join 相同设置的 CRF。

of vector size and window size/type. All models are dow size is 10. In (b) and (c), the vector size is 100. has 6 billion tokens; and on 42 billion tokens of web data, from Common Crawl5 . We tokenize and lowercase each corpus with the Stanford tokenizer, build a vocabulary of the 400,000 most frequent words6 , and then construct a matrix of cooccurrence counts X. In constructing X, we must choose how large the context window should be and whether to distinguish left context from right context. We explore the effect of these choices below. In all cases we use a decreasing weighting function, so that word pairs that are d words apart contribute 1/d to the total count. This is one way to account for the fact that very distant word pairs are expected to contain less relevant information about the words’ relationship to one another. For all our experiments, we set x max = 100, α = 3/4, and train the model using AdaGrad (Duchi et al., 2011), stochastically sampling nonzero elements from X, with initial learning rate of 0.05. We run 50 iterations for vectors smaller than 300 dimensions, and 100 iterations otherwise (see Section 4.6 for more details about the convergence rate). Unless otherwise noted, we use a context of ten words to the left and ten words to the right. The model generates two sets of word vectors, W and W̃ . When X is symmetric, W and W̃ are equivalent and differ only as a result of their random initializations; the two sets of vectors should perform equivalently. On the other hand, there is evidence that for certain types of neural networks, training multiple instances of the network and then combining the results can help reduce overfitting and noise and generally improve results (Ciresan et al., 2012). With this in mind, we choose to use 5 To demonstrate the scalability of the model, we also trained it on a much larger sixth corpus, containing 840 billion tokens of web data, but in this case we did not lowercase the vocabulary, so the results are not directly comparable. 6 For the model trained on Common Crawl data, we use a larger vocabulary of about 2 million words.

4.2 语料与训练细节

我们在五种不同规模语料上训练：2010 维基（10 亿词元）、2014 维基（16 亿）、Gigaword5（43 亿）、Gigaword5+Wiki2014（60 亿），以及 Common Crawl 网页数据（420 亿词元）。用 Stanford 分词器分词并小写，取词频最高的 400,000 词建词表，再构造共现矩阵 $X$。构造 $X$ 时需选择上下文窗口大小及是否区分左右上下文。一律使用递减加权，使相距 $d$ 的词对贡献 $1/d$。实验中取 $x_{\max}=100$、$\alpha=3/4$，用 AdaGrad 随机采样 $X$ 非零元训练，初学习率 0.05；维度小于 300 时迭代 50 次，否则 100 次。除非另说明，上下文为左右各十词。模型产生两套词向量 $W$ 与 $\tilde{W}$；当 $X$ 对称时二者等价、仅随机初始化不同。借鉴多网络集成可减过拟合的证据，我们取 $W+\tilde{W}$ 作为最终词向量，通常小幅提升表现，语义类比上增益最大。

the sum W + W̃ as our word vectors. Doing so typically gives a small boost in performance, with the biggest increase in the semantic analogy task. We compare with the published results of a variety of state-of-the-art models, as well as with our own results produced using the word2vec tool and with several baselines using SVDs. With word2vec, we train the skip-gram (SG† ) and continuous bag-of-words (CBOW† ) models on the 6 billion token corpus (Wikipedia 2014 + Gigaword 5) with a vocabulary of the top 400,000 most frequent words and a context window size of 10. We used 10 negative samples, which we show in Section 4.6 to be a good choice for this corpus. For the SVD baselines, we generate a truncated matrix Xtrunc which retains the information of how frequently each word occurs with only the top 10,000 most frequent words. This step is typical of many matrix-factorization-based methods as the extra columns can contribute a disproportionate number of zero entries and the methods are otherwise computationally expensive. The singular vectors of this matrix constitute the baseline “SVD”. We also evaluate two related baselines: “SVD-S” in which we take the SVD of √ Xtrunc , and “SVD-L” in which we take the SVD of log(1+ Xtrunc ). Both methods help compress the otherwise large range of values in X.7 4.3 Results We present results on the word analogy task in Table 2. The GloVe model performs significantly better than the other baselines, often with smaller vector sizes and smaller corpora. Our results using the word2vec tool are somewhat better than most of the previously published results. This is due to a number of factors, including our choice to use negative sampling (which typically works better than the hierarchical softmax), the number of negative samples, and the choice of the corpus. We demonstrate that the model can easily be trained on a large 42 billion token corpus, with a substantial corresponding performance boost. We note that increasing the corpus size does not guarantee improved results for other models, as can be seen by the decreased performance of the SVD7 We also investigated several other weighting schemes for transforming X; what we report here performed best. Many weighting schemes like PPMI destroy the sparsity of X and therefore cannot feasibly be used with large vocabularies. With smaller vocabularies, these information-theoretic transformations do indeed work well on word similarity measures, but they perform very poorly on the word analogy task.

4.3 结果

表 2 给出词类比结果。GloVe 显著优于其他基线，且常以更小向量维度与更小语料取胜。我们用 word2vec 工具得到的结果也略好于多数已发表数字，原因包括选用负采样、负样本数与语料选择等。模型可轻松在 420 亿词元语料上训练并带来明显提升；增大语料并不保证其他模型也提升——SVD 在更大规模上表现下降即为一例，也进一步说明我们这类加权方案的必要。表 3 为五个词相似度数据集上的 Spearman 等级相关：先对词表内各特征归一化再算余弦，再与人类评分相关。GloVe 在不到一半规模的语料上优于 word2vec 网站提供的含短语向量的 CBOW*（约 1000 亿词新闻）。表 4 为基于 CRF 的 NER：GloVe 在几乎所有指标上优于其他方法，仅 CoNLL 测试集上 HPCA 略好。结论是 GloVe 向量对下游 NLP 任务有用。

Table 3: Spearman rank correlation on word similarity tasks. All vectors are 300-dimensional. The CBOW∗ vectors are from the word2vec website and differ in that they contain phrase vectors. Model Size WS353 MC SVD 6B 35.3 35.1 SVD-S 6B 56.5 71.5 SVD-L 6B 65.7 72.7 † CBOW 6B 57.2 65.6 SG† 6B 62.8 65.2 GloVe 6B 65.8 72.7 SVD-L 42B 74.0 76.4 GloVe 42B 75.9 83.6 CBOW∗ 100B 68.4 79.6

表 3：词相似度任务上的 Spearman 等级相关。向量均为 300 维。

L model on this larger corpus. The fact that this basic SVD model does not scale well to large corpora lends further evidence to the necessity of the type of weighting scheme proposed in our model. Table 3 shows results on five different word similarity datasets. A similarity score is obtained from the word vectors by first normalizing each feature across the vocabulary and then calculating the cosine similarity. We compute Spearman’s rank correlation coefficient between this score and the human judgments. CBOW∗ denotes the vectors available on the word2vec website that are trained with word and phrase vectors on 100B words of news data. GloVe outperforms it while using a corpus less than half the size. Table 4 shows results on the NER task with the CRF-based model. The L-BFGS training terminates when no improvement has been achieved on the dev set for 25 iterations. Otherwise all configurations are identical to those used by Wang and Manning (2013). The model labeled Discrete is the baseline using a comprehensive set of discrete features that comes with the standard distribution of the Stanford NER model, but with no word vector features. In addition to the HPCA and SVD models discussed previously, we also compare to the models of Huang et al. (2012) (HSMN) and Collobert and Weston (2008) (CW). We trained the CBOW model using the word2vec tool8 . The GloVe model outperforms all other methods on all evaluation metrics, except for the CoNLL test set, on which the HPCA method does slightly better. We conclude that the GloVe vectors are useful in downstream NLP tasks, as was first 8 We use the same parameters as above, except in this case

基本 SVD 难以良好扩展到大语料，进一步说明模型中加权方案的必要性。表 3 显示五个词相似度数据集结果。表 4 显示 NER 任务：除 CoNLL 测试集上 HPCA 略优外，GloVe 在其余指标上全面领先。

Table 4: F1 score on NER task with 50d vectors. Discrete is the baseline without word vectors. We use publicly-available vectors for HPCA, HSMN, and CW. See text for details. Model Dev Test ACE MUC7 Discrete 91.0 85.4 77.4 73.4 SVD 90.8 85.7 77.3 73.7 SVD-S 91.0 85.5 77.6 74.3 SVD-L 90.5 84.8 73.6 71.5 HPCA 92.6 88.7 81.7 80.7 HSMN 90.5 85.7 78.7 74.7 CW 92.2 87.4 81.7 80.2 CBOW 93.1 88.2 82.2 81.1 GloVe 93.2 88.3 82.9 82.2 shown for neural vectors in (Turian et al., 2010). 4.4 Model Analysis: Vector Length and Context Size In Fig. 2, we show the results of experiments that vary vector length and context window. A context window that extends to the left and right of a target word will be called symmetric, and one which extends only to the left will be called asymmetric. In (a), we observe diminishing returns for vectors larger than about 200 dimensions. In (b) and (c), we examine the effect of varying the window size for symmetric and asymmetric context windows. Performance is better on the syntactic subtask for small and asymmetric context windows, which aligns with the intuition that syntactic information is mostly drawn from the immediate context and can depend strongly on word order. Semantic information, on the other hand, is more frequently non-local, and more of it is captured with larger window sizes. 4.5 Model Analysis: Corpus Size In Fig. 3, we show performance on the word analogy task for 300-dimensional vectors trained on different corpora. On the syntactic subtask, there is a monotonic increase in performance as the corpus size increases. This is to be expected since larger corpora typically produce better statistics. Interestingly, the same trend is not true for the semantic subtask, where the models trained on the smaller Wikipedia corpora do better than those trained on the larger Gigaword corpus. This is likely due to the large number of cityand countrybased analogies in the analogy dataset and the fact that Wikipedia has fairly comprehensive articles for most such locations. Moreover, Wikipedia’s

表 4：50 维向量在 NER 任务上的 F1。Discrete 为无词向量的基线。

Figure 3: Accuracy on the analogy task for 300dimensional vectors trained on different corpora. entries are updated to assimilate new knowledge, whereas Gigaword is a fixed news repository with outdated and possibly incorrect information. 4.6

图 3：在不同语料上训练的 300 维向量在类比任务上的准确率。

The total run-time is split between populating X and training the model. The former depends on many factors, including window size, vocabulary size, and corpus size. Though we did not do so, this step could easily be parallelized across multiple machines (see, e.g., Lebret and Collobert (2014) for some benchmarks). Using a single thread of a dual 2.1GHz Intel Xeon E5-2658 machine, populating X with a 10 word symmetric context window, a 400,000 word vocabulary, and a 6 billion token corpus takes about 85 minutes. Given X, the time it takes to train the model depends on the vector size and the number of iterations. For 300-dimensional vectors with the above settings (and using all 32 cores of the above machine), a single iteration takes 14 minutes. See Fig. 4 for a plot of the learning curve. 4.7

4.4 模型分析：向量长度与上下文大小

图 2 给出改变向量长度与上下文窗口的实验。向目标词左右延伸的窗口称对称窗口，仅向左延伸称非对称。图 (a) 显示约 200 维以上收益递减。图 (b)(c) 考察对称与非对称窗口大小：句法子任务上，小窗口与非对称窗口更好——符合句法信息多来自邻近上下文且强依赖词序的直觉；语义信息则更常非局部，更大窗口捕获更多。

4.5 模型分析：语料规模

图 3 给出不同语料上 300 维向量的类比表现。句法子任务随语料增大单调上升，符合更大语料统计更好的预期；语义子任务则不然——较小维基语料上的模型优于更大的 Gigaword，可能因类比集中含大量城邦类题，而维基对地点条目更全面且持续更新，Gigaword 则是固定新闻库。

A rigorous quantitative comparison of GloVe with word2vec is complicated by the existence of many parameters that have a strong effect on performance. We control for the main sources of variation that we identified in Sections 4.4 and 4.5 by setting the vector length, context window size, corpus, and vocabulary size to the configuration mentioned in the previous subsection. The most important remaining variable to control for is training time. For GloVe, the relevant parameter is the number of training iterations. For word2vec, the obvious choice would be the number of training epochs. Unfortunately, the code is currently designed for only a single epoch:

4.6 模型分析：运行时间

总运行时间分为填充 $X$ 与训练模型。前者依赖窗口、词表与语料规模，可跨机并行。在双路 2.1GHz Xeon E5-2658 单线程上，对称 10 词窗口、40 万词表、60 亿词元语料填充 $X$ 约 85 分钟；给定 $X$ 后，300 维向量用满 32 核单次迭代约 14 分钟。学习曲线见图 4。

Figure 4: Overall accuracy on the word analogy task the number of iterations for GloVe and by the numbe (b). In all cases, we train 300-dimensional vectors Gigaword 5) with the same 400,000 word vocabulary it specifies a learning schedule specific to a single pass through the data, making a modification for multiple passes a non-trivial task. Another choice is to vary the number of negative samples. Adding negative samples effectively increases the number of training words seen by the model, so in some ways it is analogous to extra epochs. We set any unspecified parameters to their default values, assuming that they are close to optimal, though we acknowledge that this simplification should be relaxed in a more thorough analysis. In Fig. 4, we plot the overall performance on the analogy task as a function of training time. The two x-axes at the bottom indicate the corresponding number of training iterations for GloVe and negative samples for word2vec. We note that word2vec’s performance actually decreases if the number of negative samples increases beyond about 10. Presumably this is because the negative sampling method does not approximate the target probability distribution well.9 For the same corpus, vocabulary, window size, and training time, GloVe consistently outperforms word2vec. It achieves better results faster, and also obtains the best results irrespective of speed.

图 4：词类比总体准确率随训练时间变化（由 GloVe 迭代次数与 word2vec 负样本数控制）。

4.7 模型分析：与 word2vec 的比较

与 word2vec 做严格定量比较因参数众多而复杂。我们控制向量长度、窗口、语料与词表后，最重要的剩余变量是训练时间。对 GloVe 即迭代次数；对 word2vec，负样本数增加可视为有效训练词数增加，某种程度上类似额外 epoch。图 4 显示：在相同语料、词表、窗口与训练时间下，GloVe 持续优于 word2vec——更快达到更好结果，且不论速度也取得最佳结果。word2vec 在负样本约超过 10 后表现反而下降，推测因负采样未能很好近似目标概率分布。

Recently, considerable attention has been focused on the question of whether distributional word representations are best learned from count-based 9 In contrast, noise-contrastive estimation is an approxi-

5 结论

近来，分布式词表征是否优于基于计数的方法受到较多关注。本文论证这两族并非迥异：对 skip-gram 与 LBL 模型的全局目标稍加修改，即可得到显式加权最小二乘回归形式，而加权方案可显著提升性能。

mation which improves with more negative samples. In Table 1 of (Mnih et al., 2013), accuracy on the analogy task is a non-decreasing function of the number of negative samples.

所得模型 GloVe 兼取计数模型高效利用统计与预测模型灵活加权之长，在词类比、词相似度与 NER 任务上相对相关模型均表现强劲。

as a function of training time, which is governed by er of negative samples for CBOW (a) and skip-gram s on the same 6B token corpus (Wikipedia 2014 + y, and use a symmetric context window of size 10. methods or from prediction-based methods. Currently, prediction-based models garner substantial support; for example, Baroni et al. (2014) argue that these models perform better across a range of tasks. In this work we argue that the two classes of methods are not dramatically different at a fundamental level since they both probe the underlying co-occurrence statistics of the corpus, but the efficiency with which the count-based methods capture global statistics can be advantageous. We construct a model that utilizes this main benefit of count data while simultaneously capturing the meaningful linear substructures prevalent in recent log-bilinear prediction-based methods like word2vec. The result, GloVe, is a new global log-bilinear regression model for the unsupervised learning of word representations that outperforms other models on word analogy, word similarity, and named entity recognition tasks.

源代码与数据见项目页。我们希望 GloVe 能为后续词向量研究提供坚实基线，并促使社区更多关注模型目标中的加权机制。

Acknowledgments

致谢

We thank the anonymous reviewers for their valuable comments. Stanford University gratefully acknowledges the support of the Defense Threat Reduction Agency (DTRA) under Air Force Research Laboratory (AFRL) contract no. FA865010-C-7020 and the Defense Advanced Research Projects Agency (DARPA) Deep Exploration and Filtering of Text (DEFT) Program under AFRL contract no. FA8750-13-2-0040. Any opinions, findings, and conclusion or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of the DTRA, AFRL, DEFT, or the US government.

感谢匿名审稿人的有益意见。本工作部分得到 Defense Advanced Research Projects Agency（DARPA）Deep Exploration and Filtering of Text（DEFT）Program 支持（AFRL 资助 FA8750-13-2-0040），以及 NSF IIS-1159679 资助。任何观点、发现、结论或建议均属作者，不一定代表资助方。

References

参考文献

Tom M. Apostol. 1976. Introduction to Analytic Number Theory. Introduction to Analytic Number Theory. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL. Yoshua Bengio. 2009. Learning deep architectures for AI. Foundations and Trends in Machine Learning. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. JMLR, 3:1137–1155. John A. Bullinaria and Joseph P. Levy. 2007. Extracting semantic representations from word cooccurrence statistics: A computational study. Behavior Research Methods, 39(3):510–526. Dan C. Ciresan, Alessandro Giusti, Luca M. Gambardella, and Jürgen Schmidhuber. 2012. Deep neural networks segment neuronal membranes in electron microscopy images. In NIPS, pages 2852–2860. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Almost) from Scratch. JMLR, 12:2493–2537. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12. Lev Finkelstein, Evgenly Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th international conference on World Wide Web, pages 406–414. ACM. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving

Word Representations via Global Context and Multiple Word Prototypes. In ACL. Rémi Lebret and Ronan Collobert. 2014. Word embeddings through Hellinger PCA. In EACL. Omer Levy, Yoav Goldberg, and Israel RamatGan. 2014. Linguistic regularities in sparse and explicit word representations. CoNLL-2014. Kevin Lund and Curt Burgess. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instrumentation, and Computers, 28:203–208. Minh-Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word representations with recursive neural networks for morphology. CoNLL-2013. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In ICLR Workshop Papers. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In NIPS, pages 3111–3119. Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013c. Linguistic regularities in continuous space word representations. In HLTNAACL. George A. Miller and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and cognitive processes, 6(1):1–28. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In NIPS. Douglas L. T. Rohde, Laura M. Gonnerman, and David C. Plaut. 2006. An improved model of semantic similarity based on lexical co-occurence. Communications of the ACM, 8:627–633. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. ACM Computing Surveys, 34:1–47. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing With Compositional Vector Grammars. In ACL.

Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron Fernandes, and Gregory Marton. 2003. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the SIGIR Conference on Research and Development in Informaion Retrieval. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In CoNLL-2003. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of ACL, pages 384–394. Mengqiu Wang and Christopher D. Manning. 2013. Effect of non-linear deep architecture in sequence labeling. In Proceedings of the 6th International Joint Conference on Natural Language Processing (IJCNLP).