Distilling the Knowledge in a Neural Network
Distilling the Knowledge in a Neural Network
蒸馏神经网络中的知识（Distilling the Knowledge in a Neural Network）
Geoffrey Hinton Google Inc.Mountain View geoffhinton@google.com Thanks: Also affiliated with the University of Toronto and the Canadian Institute for Advanced Research. Thanks: Equal contribution. Oriol VinyalsGoogle Inc.Mountain View vinyals@google.com Jeff DeanGoogle Inc.Mountain View jeff@google.com
Geoffrey Hinton Oriol Vinyals Jeff Dean
Google Inc.，Mountain View
geoffhinton@google.com vinyals@google.com jeff@google.com
致谢：Hinton 同时任职于多伦多大学与加拿大高级研究所（CIFAR）。Vinyals 与 Dean 贡献相同。
Abstract
摘要
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions [3]. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators [1] have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
提高几乎任意机器学习算法性能的一种很简单办法，是在同一数据上训练许多不同模型，再平均它们的预测 [3]。不幸的是，用整个集成做预测很麻烦，计算开销也可能大到无法部署给大量用户，尤其当单个模型已是大型神经网络时。Caruana 及其合作者 [1] 已表明，可以把集成中的知识压缩进一个更易部署的单模型；我们用另一种压缩技术把这一思路推进了一步。我们在 MNIST 上得到一些出人意料的结果，并表明通过把集成中的知识蒸馏进单模型，可以显著改进一个广泛使用的商用系统的声学模型。我们还引入一类新的集成：由一个或多个完整模型与许多专家模型组成，专家专门区分完整模型容易混淆的细粒度类别。与专家混合（mixture of experts）不同，这些专家模型可以快速、并行地训练。
1 Introduction
1 引言
Many insects have a larval form that is optimized for extracting energy and nutrients from the environment and a completely different adult form that is optimized for the very different requirements of traveling and reproduction. In large-scale machine learning, we typically use very similar models for the training stage and the deployment stage despite their very different requirements: For tasks like speech and object recognition, training must extract structure from very large, highly redundant datasets but it does not need to operate in real time and it can use a huge amount of computation. Deployment to a large number of users, however, has much more stringent requirements on latency and computational resources. The analogy with insects suggests that we should be willing to train very cumbersome models if that makes it easier to extract structure from the data. The cumbersome model could be an ensemble of separately trained models or a single very large model trained with a very strong regularizer such as dropout [9]. Once the cumbersome model has been trained, we can then use a different kind of training, which we call “distillation” to transfer the knowledge from the cumbersome model to a small model that is more suitable for deployment. A version of this strategy has already been pioneered by Rich Caruana and his collaborators [1]. In their important paper they demonstrate convincingly that the knowledge acquired by a large ensemble of models can be transferred to a single small model.
许多昆虫的幼虫形态最优化,可以从环境中提取能量和养分,而成人形态则完全不同,对旅行和繁殖的要求也大不相同. 在大规模机器学习中,我们通常在训练阶段和部署阶段使用非常相似的模型,尽管它们的要求非常不同: 对于语音和对象识别等任务,训练必须从非常大,高度冗余的数据集中提取结构,但不需要实时操作,可以使用大量的计算. 但是,向大量用户部署人员对延迟和计算资源的要求要严格得多。 与昆虫的类比表明,我们应当愿意训练非常繁琐的模型,这样就更容易从数据中提取结构. 繁琐的模型可以是一组经过单独训练的模型,也可以是单个经过非常强正则项如退学器训练的非常大的模型[9]. 一旦对繁琐的模型进行了培训,我们就可以使用另一种类型的培训,我们称之为“沉淀”,将知识从繁琐的模型转移到更适合部署的小型模型。 这一策略咀嚼的版本已经由里希·卡鲁阿纳及其合作者开创[1]. 在他们的重要论文中,他们令人信服地表明,一个庞大的模型组合所获得的知识可以转移到一个单一的小型模型上.
A conceptual block that may have prevented more investigation of this very promising approach is that we tend to identify the knowledge in a trained model with the learned parameter values and this makes it hard to see how we can change the form of the model but keep the same knowledge. A more abstract view of the knowledge, that frees it from any particular instantiation, is that it is a learned mapping from input vectors to output vectors. For cumbersome models that learn to discriminate between a large number of classes, the normal training objective is to maximize the average log probability of the correct answer, but a side-effect of the learning is that the trained model assigns probabilities to all of the incorrect answers and even when these probabilities are very small, some of them are much larger than others. The relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize. An image of a BMW, for example, may only have a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot.
一个可能阻止了对这一非常有希望的方法进行更多调查的概念障碍是,我们倾向于在经过训练的模型中用学到的参数值来识别知识,这使我们很难看到我们如何能够改变模型的形式,但保持同样的知识。 对知识的一种更为抽象的观点,即将其从任何特定的即时现象中解脱出来,就是它是一种从输入向量到输出向量的学习图. 对于学习区分大量班级的繁琐模型,正常的训练目标是尽量扩大正确答案的平均日志概率,但学习的一个副作用是,经过训练的模型为所有不正确的答案分配概率,即使这些概率很小,也有一些比其他的要大得多. 错误答案的相对概率告诉我们很多关于繁琐的模式如何倾向于概括. 例如,宝马的形象可能只有很小的机会被误认为垃圾车,但这一错误的可能性仍然比误认为胡萝卜要大很多倍.
It is generally accepted that the objective function used for training should reflect the true objective of the user as closely as possible. Despite this, models are usually trained to optimize performance on the training data when the real objective is to generalize well to new data. It would clearly be better to train models to generalize well, but this requires information about the correct way to generalize and this information is not normally available. When we are distilling the knowledge from a large model into a small one, however, we can train the small model to generalize in the same way as the large model. If the cumbersome model generalizes well because, for example, it is the average of a large ensemble of different models, a small model trained to generalize in the same way will typically do much better on test data than a small model that is trained in the normal way on the same training set as was used to train the ensemble.
普遍接受的看法是,用于培训的客观职能应尽量反映用户的真正目标。 尽管如此,当真正的目标是很好地概括到新数据时,模型通常都会被训练来优化培训数据的性能. 显然,最好对模型进行培训,使其普遍化,但这需要关于普遍化正确方法的信息,而这种信息通常不可用。 然而,当我们把知识从一个大模型中提炼成一个小模型时,我们可以训练这个小模型,以和大模型相同的方式进行概括. 如果繁琐的模型能很好地概括,例如因为它是不同模型的大型集合体的平均值,那么经过同样方式的概括化训练的小型模型在测试数据上一般会比在训练集合体的相同训练组上以正常方式训练的小型模型做得更好.
An obvious way to transfer the generalization ability of the cumbersome model to a small model is to use the class probabilities produced by the cumbersome model as “soft targets” for training the small model. For this transfer stage, we could use the same training set or a separate “transfer” set. When the cumbersome model is a large ensemble of simpler models, we can use an arithmetic or geometric mean of their individual predictive distributions as the soft targets. When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases, so the small model can often be trained on much less data than the original cumbersome model and using a much higher learning rate.
将繁琐模型的通俗化能力转移到一个小模型的一个明显方法是将繁琐模型所产生的阶级概率作为培训小模型的“软目标”。 对于这一移交阶段,我们可以使用同样的培训组合或单独的“移交”组合。 当繁琐的模型是一大群较简单的模型时,我们可以使用其单个预测分布的算术或几何平均值作为软目标. 当软目标有很高的Entropy时,它们提供的每个训练案例的信息比硬目标要多得多,训练案例之间的梯度差异要小得多,所以小模型往往可以比最初的繁琐模型少得多的数据来训练,并且使用高得多的学习率.
For tasks like MNIST in which the cumbersome model almost always produces the correct answer with very high confidence, much of the information about the learned function resides in the ratios of very small probabilities in the soft targets. For example, one version of a 2 may be given a probability of of being a 3 and of being a 7 whereas for another version it may be the other way around. This is valuable information that defines a rich similarity structure over the data (i. e. it says which 2’s look like 3’s and which look like 7’s) but it has very little influence on the cross-entropy cost function during the transfer stage because the probabilities are so close to zero. Caruana and his collaborators circumvent this problem by using the logits (the inputs to the final softmax) rather than the probabilities produced by the softmax as the targets for learning the small model and they minimize the squared difference between the logits produced by the cumbersome model and the logits produced by the small model. Our more general solution, called “distillation”, is to raise the temperature of the final softmax until the cumbersome model produces a suitably soft set of targets. We then use the same high temperature when training the small model to match these soft targets. We show later that matching the logits of the cumbersome model is actually a special case of distillation.
对于像MNIST这样的任务,繁琐的模型几乎总是以非常高的自信产生正确的答案,许多关于学到的功能的信息都存在于软目标的极小概率的比例上. 例如,一个版本的2可能会得到成为3的概率,而成为7的概率,而另一个版本则可能相反。 这是有价值的信息,它定义了与数据丰富的相似性结构(,即),它说2的哪一个看起来像3's,哪一个看起来像7's),但是在转移阶段对交叉交替成本函数影响很小,因为概率接近于零。 卡鲁阿纳和他的合作者通过使用对数(对最终软马克斯的投入)而不是软马克斯产生的概率作为学习小模型的目标来规避这个问题,他们将繁琐模型产生的对数和小模型产生的对数之间的平方差最小化. 我们更一般的解决方案,称为“蒸馏”,是提高最终软马克的温度,直到繁琐的模式产生一套适当的软目标为止。 然后在训练小模型时使用同样的高温来配合这些软目标. 我们稍后显示,匹配繁琐模型的对数实际上是蒸馏的特殊案例。
The transfer set that is used to train the small model could consist entirely of unlabeled data [1] or we could use the original training set. We have found that using the original training set works well, especially if we add a small term to the objective function that encourages the small model to predict the true targets as well as matching the soft targets provided by the cumbersome model. Typically, the small model cannot exactly match the soft targets and erring in the direction of the correct answer turns out to be helpful.
用于训练小模型的转录集可以完全由未贴标签的数据组成[1],也可以使用原来的训练集. 我们发现,使用原有的训练集效果很好,特别是如果我们在目标函数中增加一个小词,鼓励小模型预测真正的目标,以及匹配繁琐模型提供的软目标. 典型的情况是,小模型不能完全匹配软目标,而错误地朝着正确答案的方向走,结果证明是有用的.
2 Distillation
2 蒸馏
Neural networks typically produce class probabilities by using a “softmax” output layer that converts the logit, , computed for each class into a probability, , by comparing with the other logits.
神经网络通常通过使用一个“软max”输出层来产生类概率,该层将每个类别计算成概率的, 将与其他对数进行比较。
where is a temperature that is normally set to . Using a higher value for produces a softer probability distribution over classes.
其中XXMAT的实质性温度为 。 使用更高值的产生更软的概率分布于各类.
In the simplest form of distillation, knowledge is transferred to the distilled model by training it on a transfer set and using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax. The same high temperature is used when training the distilled model, but after it has been trained it uses a temperature of 1.
法律依据: 以最简单的蒸馏形式,知识通过对蒸馏模型进行转录集培训,并在转录集中采用软目标分配,在转录集中采用软模中温度高的繁琐模型生成每个案例. 在训练蒸馏模型时也使用同样的高温,但经过训练后使用温度为1.
When the correct labels are known for all or some of the transfer set, this method can be significantly improved by also training the distilled model to produce the correct labels. One way to do this is to use the correct labels to modify the soft targets, but we found that a better way is to simply use a weighted average of two different objective functions. The first objective function is the cross entropy with the soft targets and this cross entropy is computed using the same high temperature in the softmax of the distilled model as was used for generating the soft targets from the cumbersome model. The second objective function is the cross entropy with the correct labels. This is computed using exactly the same logits in softmax of the distilled model but at a temperature of 1. We found that the best results were generally obtained by using a condiderably lower weight on the second objective function. Since the magnitudes of the gradients produced by the soft targets scale as it is important to multiply them by when using both hard and soft targets. This ensures that the relative contributions of the hard and soft targets remain roughly unchanged if the temperature used for distillation is changed while experimenting with meta-parameters.
当正确的标签被所有或部分的传输集所熟知时,这个方法还可以通过训练蒸馏模型来产生正确的标签来大大改进. 这样做的方法之一是使用正确的标签来修改软目标,但我们发现更好的方法是简单地使用两个不同客观函数的加权平均值. 第一个客观的功能是带软目标的横py,这种横py的计算使用蒸馏模型软体轴上相同的高温,与从繁琐模型中生成软目标时所使用的温度相同. 第二个目标函数是带有正确标签的交叉 en. 在蒸馏模型的软最大值中使用完全相同的对数计算,但温度为1。 我们发现,一般通过对第二个目标功能使用相对较低的权重来取得最佳结果。 由于软目标尺度产生的梯度是,因此在使用硬目标和软目标时必须将其乘以。 这确保了如果在试验元参数时改变蒸馏所用的温度,则硬和软目标的相对贡献大致保持不变。
2.1 Matching logits is a special case of distillation
2.1 匹配 logits 是蒸馏的特例
Each case in the transfer set contributes a cross-entropy gradient, , with respect to each logit, of the distilled model. If the cumbersome model has logits which produce soft target probabilities and the transfer training is done at a temperature of , this gradient is given by:
转录集中的每个案例都提供了一个跨直径梯度(),与蒸馏模型的每个对数()有关. 如果繁琐的模型有产生软目标概率的logits,并且转移训练是在的温度下完成的,这个梯度通过:
If the temperature is high compared with the magnitude of the logits, we can approximate:
如果温度比对数的大小高,我们可以大致:
If we now assume that the logits have been zero-meaned separately for each transfer case so that Eq. 3 simplifies to:
如果我们现在假设每个转移个案的对数是零的,因此 Eq. 3简化为:
So in the high temperature limit, distillation is equivalent to minimizing , provided the logits are zero-meaned separately for each transfer case. At lower temperatures, distillation pays much less attention to matching logits that are much more negative than the average. This is potentially advantageous because these logits are almost completely unconstrained by the cost function used for training the cumbersome model so they could be very noisy. On the other hand, the very negative logits may convey useful information about the knowledge acquired by the cumbersome model. Which of these effects dominates is an empirical question. We show that when the distilled model is much too small to capture all of the knowledege in the cumbersome model, intermediate temperatures work best which strongly suggests that ignoring the large negative logits can be helpful.
因此,在高温限值中,蒸馏等效于最小化,前提是每个转录例的对数是零的。 在较低的温度下,蒸馏对匹配比平均值更负的对数的注意要小得多. 这有可能是有利的,因为这些日志几乎完全不受训练繁琐模型的成本功能的限制,因此它们可能非常吵闹。 另一方面,非常负面的日志可能传达关于繁琐模式所获得知识的有用信息。 其中哪些影响占主导地位是一个经验问题。 我们方面也表明,当蒸馏模型是否可以在繁琐的模型中捕捉到所有知觉模型时,中间温度最有效,这强烈地表明,忽略大型负对数可能是有益的。
3 Preliminary experiments on MNIST
3 MNIST 上的初步实验
To see how well distillation works, we trained a single large neural net with two hidden layers of 1200 rectified linear hidden units on all 60,000 training cases. The net was strongly regularized using dropout and weight-constraints as described in [5]. Dropout can be viewed as a way of training an exponentially large ensemble of models that share weights. In addition, the input images were jittered by up to two pixels in any direction. This net achieved 67 test errors whereas a smaller net with two hidden layers of 800 rectified linear hidden units and no regularization achieved 146 errors. But if the smaller net was regularized solely by adding the additional task of matching the soft targets produced by the large net at a temperature of 20, it achieved 74 test errors. This shows that soft targets can transfer a great deal of knowledge to the distilled model, including the knowledge about how to generalize that is learned from translated training data even though the transfer set does not contain any translations.
为了了解蒸馏效果如何,我们训练了一个大型神经网,上面有两层1200个修正线性隐藏单元,覆盖所有6万个培训案例。 如[5]所述,通过辍学和体重限制,该网已严格规范化。 辍学可被视为一种培训一种规模庞大、能分享重量的模型的方法。 此外,输入的图像在任意方向上被最多两个像素所震动. 这一净额实现了67个测试错误,而一个有两层800个修正线性隐藏单元的较小的网则实现了146个错误。 但如果将较小的网只通过增加额外任务,在20度温度下匹配大网产生的软目标来规范化,则达到了74个测试错误. 这表明软 你为什么要将大量知识转移到蒸馏模型,包括如何概括从翻译培训数据中学到的知识,尽管转移集没有包含任何翻译.
When the distilled net had 300 or more units in each of its two hidden layers, all temperatures above 8 gave fairly similar results. But when this was radically reduced to 30 units per layer, temperatures in the range 2.5 to 4 worked significantly better than higher or lower temperatures.
当蒸馏网在它的两个隐藏层中每个层有300个或更多的单位时,所有温度超过8的温度都给出了相当相似的结果. 但是,当这种温度急剧下降到每层30个单位时,2.5至4之间的温度比高温或低温高得多。
We then tried omitting all examples of the digit 3 from the transfer set. So from the perspective of the distilled model, 3 is a mythical digit that it has never seen. Despite this, the distilled model only makes 206 test errors of which 133 are on the 1010 threes in the test set. Most of the errors are caused by the fact that the learned bias for the 3 class is much too low. If this bias is increased by 3.5 (which optimizes overall performance on the test set), the distilled model makes 109 errors of which 14 are on 3s. So with the right bias, the distilled model gets 98.6% of the test 3s correct despite never having seen a 3 during training. If the transfer set contains only the 7s and 8s from the training set, the distilled model makes 47.3% test errors, but when the biases for 7 and 8 are reduced by 7.6 to optimize test performance, this falls to 13.2% test errors.
之后我们尝试在传输集中省略所有数字3的例子. 因此从蒸馏模型的角度来看,3是一个从未见过的神话数字. 尽管如此,蒸馏模型只制造了206个测试错误,其中133个在测试集的1010个3上. 大多数错误是由于3级的学习偏差太低所致. 如果这种偏差增加3.5(在测试集上优化了整体性能),蒸馏模型就会产生109个错误,其中14个在3s上. 因此在正确的偏差下,蒸馏模型得到98.6%的测试3s正确,尽管在训练期间从未见过3个. 如果转录集只包含**训练集中的7和8,蒸馏模型的测试误差为47.3%,但当7和8的偏差减少7.6以优化测试性能时,则下降到13.2%的测试误差.
4 Experiments on speech recognition
4 语音识别实验
In this section, we investigate the effects of ensembling Deep Neural Network (DNN) acoustic models that are used in Automatic Speech Recognition (ASR). We show that the distillation strategy that we propose in this paper achieves the desired effect of distilling an ensemble of models into a single model that works significantly better than a model of the same size that is learned directly from the same training data.
在本节中,我们调查在自动语音识别(ASR)中使用的全美深神经网络(DNN)声学模型的效果. 我们表明,我们在本文中提出的蒸馏战略实现了将一组模型蒸馏成一个单一模型的预期效果,这个模型比直接从同一培训数据中汲取的同样大小的模型大得多。
State-of-the-art ASR systems currently use DNNs to map a (short) temporal context of features derived from the waveform to a probability distribution over the discrete states of a Hidden Markov Model (HMM) [4]. More specifically, the DNN produces a probability distribution over clusters of tri-phone states at each time and a decoder then finds a path through the HMM states that is the best compromise between using high probability states and producing a transcription that is probable under the language model.
目前最先进的ASR系统利用DNNs将波形产生的特征的(短)时间背景映射到隐藏的Markov模型的离散状态上的概率分布[4]. 更具体地说,DNN每次在三声道状态的集群中产生概率分布,然后一个解码器通过HMM状态找到一条路径,这是在使用高概率状态和产生语言模型下可能存在的转录之间达成的最佳妥协.
Although it is possible (and desirable) to train the DNN in such a way that the decoder (and, thus, the language model) is taken into account by marginalizing over all possible paths, it is common to train the DNN to perform frame-by-frame classification by (locally) minimizing the cross entropy between the predictions made by the net and the labels given by a forced alignment with the ground truth sequence of states for each observation:
虽然通过在所有可能路径上边际化来考虑解码器(以及因此,语言模型)的方式可以(和可取的)对DNN进行训练,但通常通过(局部)最小化网的预测与强制调整每个观测点的状态的真值序列而给出的标签之间的交叉 en度来对DNN进行逐帧分类:
where are the parameters of our acoustic model which maps acoustic observations at time , , to a probability, , of the “correct” HMM state , which is determined by a forced alignment with the correct sequence of words. The model is trained with a distributed stochastic gradient descent approach.
其中是我国声学模型的参数,该模型将当时,的声学观测图显示为的概率,即"正确"的HMM状态,由强制与正确的词序对齐决定. 该模型采用分布式斜度下降法培训。
We use an architecture with 8 hidden layers each containing 2560 rectified linear units and a final softmax layer with 14,000 labels (HMM targets ). The input is 26 frames of 40 Mel-scaled filterbank coefficients with a 10ms advance per frame and we predict the HMM state of 21st frame. The total number of parameters is about 85M. This is a slightly outdated version of the acoustic model used by Android voice search, and should be considered as a very strong baseline. To train the DNN acoustic model we use about 2000 hours of spoken English data, which yields about 700M training examples. This system achieves a frame accuracy of 58.9%, and a Word Error Rate (WER) of 10.9% on our development set.
我们使用一个包含8个隐藏层的架构,每个层包含2560个修正的线性单元和14,000个标签(HMM目标 )的最终软马克斯层. 输入为26帧40Mel比例的滤波库系数,每帧预留10ms,我们预测HMM状态为21st帧. 参数总数约为85M. 这是Android语音搜索所使用的声学模型略为过时的版本,应当视为非常强的基线. 为了训练DNN声学模型,我们使用大约20小时的英语口语数据,这产生了大约700M的训练实例. 该系统的帧精度为58.9%,而我们的开发集的文字误差率为10.9%。
4.1 Results
4.1 结果
We trained 10 separate models to predict , using exactly the same architecture and training procedure as the baseline. The models are randomly initialized with different initial parameter values and we find that this creates sufficient diversity in the trained models to allow the averaged predictions of the ensemble to significantly outperform the individual models. We have explored adding diversity to the models by varying the sets of data that each model sees, but we found this to not significantly change our results, so we opted for the simpler approach. For the distillation we tried temperatures of and used a relative weight of 0.5 on the cross-entropy for the hard targets, where bold font indicates the best value that was used for table 1 .
我们培训了10个单独的模型,以预测,使用与基线完全相同的架构和训练流程。 这些模型以不同的初始参数值随机初始化,我们发现这在训练有素的模型中创造了足够的多样性,使得对综艺的平均值预测大大超过单个模型. 我们已经通过改变每个模型所看到的数据集来探索在模型中增加多样性,但我们发现这并没有显著改变我们的结果,所以我们选择了更简单的方法. 在蒸馏过程中,我们尝试了视情况而定的温度,并在硬目标上采用了0.5的相对重量,粗体字体表示表1所使用的最佳值。
Table 1 shows that, indeed, our distillation approach is able to extract more useful information from the training set than simply using the hard labels to train a single model. More than 80% of the improvement in frame classification accuracy achieved by using an ensemble of 10 models is transferred to the distilled model which is similar to the improvement we observed in our preliminary experiments on MNIST. The ensemble gives a smaller improvement on the ultimate objective of WER (on a 23K-word test set) due to the mismatch in the objective function, but again, the improvement in WER achieved by the ensemble is transferred to the distilled model.
表1显示,事实上,我们的蒸馏方法能够从培训中获取比仅仅使用硬标签来培训单一模型更有用的信息。 通过使用10种模型的组合而实现的框架分类精度改进的80%以上转移到了蒸馏模型,这与我们在MNIST的初步实验中观察到的改进类似。 综艺对WER(在23K字测试集上)的最终目标有较小的改进,因为目标函数不匹配,但同样,综艺实现的WER的改进被转移到蒸馏模型上.
| System | Test Frame Accuracy | WER |
|---|---|---|
| Baseline | 58.9% | 10.9% |
| 10xEnsemble | 61.1% | 10.7% |
| Distilled Single model | 60.8% | 10.7% |
Table 1: Frame classification accuracy and WER showing that the distilled single model performs about as well as the averaged predictions of 10 models that were used to create the soft targets.
| 系统 | 测试框架精确度 | 微风 |
|---|---|---|
| 基线 | 58.9% | 10.9% |
| 10x 组合 | 61.1% | 10.7% |
| 单一型号 | 60.8% | 10.7% |
表1:框架分类精度和WER显示,蒸馏的单一模型对用于创建软目标的10个模型进行大约以及平均预测。
We have recently become aware of related work on learning a small acoustic model by matching the class probabilities of an already trained larger model [8]. However, they do the distillation at a temperature of 1 using a large unlabeled dataset and their best distilled model only reduces the error rate of the small model by 28% of the gap between the error rates of the large and small models when they are both trained with hard labels.
我们最近认识到,通过匹配一个已经受过训练的较大模型的班级概率,在学习一个小声学模型方面开展了相关工作[8]。 然而,他们使用一个大的无标签数据集在1的温度下进行蒸馏,而他们最好的蒸馏模型在同时使用硬标签进行训练时,只会将小模型的误差率降低28%.
5 Training ensembles of specialists on very big datasets
5 在超大数据集上训练专家集成
Training an ensemble of models is a very simple way to take advantage of parallel computation and the usual objection that an ensemble requires too much computation at test time can be dealt with by using distillation. There is, however, another important objection to ensembles: If the individual models are large neural networks and the dataset is very large, the amount of computation required at training time is excessive, even though it is easy to parallelize.
训练一组模型是利用平行计算的一种非常简单的方法,通常的反对意见是,一个集合在测试时需要过多的计算,可以通过蒸馏处理. 然而,还有另一项对综艺的重要反对意见: 如果单个模型是大型神经网络,数据集非常大,那么在训练时所需的计算量是过多的,尽管很容易平行.
In this section we give an example of such a dataset and we show how learning specialist models that each focus on a different confusable subset of the classes can reduce the total amount of computation required to learn an ensemble. The main problem with specialists that focus on making fine-grained distinctions is that they overfit very easily and we describe how this overfitting may be prevented by using soft targets.
在本节中,我们举出一个这样的数据集的例子,我们显示,学习专家模型如何能减少学习综艺所需的计算总数。 专家专注于细微区分的主要问题是,这些区分非常容易适应,我们描述了如何通过使用软目标来防止这种过于适应。
5.1 The JFT dataset
5.1 JFT 数据集
JFT is an internal Google dataset that has 100 million labeled images with 15,000 labels. When we did this work, Google’s baseline model for JFT was a deep convolutional neural network [7] that had been trained for about six months using asynchronous stochastic gradient descent on a large number of cores. This training used two types of parallelism [2]. First, there were many replicas of the neural net running on different sets of cores and processing different mini-batches from the training set. Each replica computes the average gradient on its current mini-batch and sends this gradient to a sharded parameter server which sends back new values for the parameters. These new values reflect all of the gradients received by the parameter server since the last time it sent parameters to the replica. Second, each replica is spread over multiple cores by putting different subsets of the neurons on each core. Ensemble training is yet a third type of parallelism that can be wrapped around the other two types, but only if a lot more cores are available. Waiting for several years to train an ensemble of models was not an option, so we needed a much faster way to improve the baseline model.
JFT是一个Google内部数据集,拥有1亿个标签的图像,有15,000个标签.各种不同的神经网络, 已经训练了大约六个月, 这种训练采用了两种平行主义[2]. 首先,许多神经网的复制品运行在不同组核上,并处理与训练组不同的小型枪管. 每个复制件会计算当前小批次的平均梯度, 并将这个梯度发送给一个已磨损的参数服务器, 它会给参数发送新的值 。 这些新值反映了参数服务器自上次向复制人发送参数以来收到的所有梯度. 其次,每个复制品通过在每个核心上放置不同的神经元子集来分散在多个核心上. 集成训练(英語:Consult training)尚是第三种平行主义,可以包圍其他两种类型,但只有有更多核心可用. 等待几年来训练一组模型并不是一个选择,所以我们需要一个更快的方法来改进基线模型。
5.2 Specialist Models
5.2 专家模型
When the number of classes is very large, it makes sense for the cumbersome model to be an ensemble that contains one generalist model trained on all the data and many “specialist” models, each of which is trained on data that is highly enriched in examples from a very confusable subset of the classes (like different types of mushroom). The softmax of this type of specialist can be made much smaller by combining all of the classes it does not care about into a single dustbin class.
当课程数量非常大时,繁琐的模型就是一个综艺节目,包含一个对所有数据进行培训的通论模型和许多“专家”模型,每个模型都受过数据培训,从一个非常混杂的子集(如不同类型的蘑菇)中以实例高度丰富。 这种类型的专家的软马克斯可以通过将它不关心的所有类别合并成一个单一的粉末类而变得小得多.
To reduce overfitting and share the work of learning lower level feature detectors, each specialist model is initialized with the weights of the generalist model. These weights are then slightly modified by training the specialist with half its examples coming from its special subset and half sampled at random from the remainder of the training set. After training, we can correct for the biased training set by incrementing the logit of the dustbin class by the log of the proportion by which the specialist class is oversampled.
为了减少过拟合和分享学习低级特征探测器的工作,每个专家模型都以通用模型的权重初始化. 然后,通过对专家进行培训,对这些权重略作修改,其半数实例来自专家的特殊子集,一半随机抽样来自培训的其余部分。 训练后,我们可以纠正所设置的有偏见的训练,通过专家班被过度抽查的比例的对数,增加尘斌班的登录数。
5.3 Assigning classes to specialists
5.3 为专家分配类别
In order to derive groupings of object categories for the specialists, we decided to focus on categories that our full network often confuses. Even though we could have computed the confusion matrix and used it as a way to find such clusters, we opted for a simpler approach that does not require the true labels to construct the clusters.
为了为专家得出对象类别分组,我们决定侧重于我们整个网络经常混淆的类别。 尽管我们可以计算混乱矩阵,并将其作为找到这种集群的方法,但我们选择了一种简单的方法,不需要真正的标签来构建集群。
| JFT 1: Tea party; Easter; Bridal shower; Baby shower; Easter Bunny; … |
|---|
| JFT 2: Bridge; Cable-stayed bridge; Suspension bridge; Viaduct; Chimney; … |
| JFT 3: Toyota Corolla E100; Opel Signum; Opel Astra; Mazda Familia; … |
Table 2: Example classes from clusters computed by our covariance matrix clustering algorithm
| JFT 1: 茶会;复活节;新娘淋浴;婴儿淋浴;率先复活节兔子;. |
|---|
| ** JFT 2**:桥;有线悬浮桥;悬浮桥;蒸汽;烟囱;. |
| JFT3:丰田科罗拉E100;Opel Signum;Opel Astra;Mazda Familia;. |
表2: 我们的共变矩阵群集算法计算出来的群集示例类
In particular, we apply a clustering algorithm to the covariance matrix of the predictions of our generalist model, so that a set of classes that are often predicted together will be used as targets for one of our specialist models, . We applied an on-line version of the K-means algorithm to the columns of the covariance matrix, and obtained reasonable clusters (shown in Table 2). We tried several clustering algorithms which produced similar results.
特别是,我们对可予概括的模型的预测的共变矩阵采用了一个集合算法,这样,一组常常一起预测的类将被用作我们的一个专家模型的目标. 我们在共变矩阵的列上应用了K度算法的在线版本,并获得了合理的集群(在表2中显示). 我们尝试了几种集群算法 产生类似的结果。
5.4 Performing inference with ensembles of specialists
5.4 用专家集成做推理
Before investigating what happens when specialist models are distilled, we wanted to see how well ensembles containing specialists performed. In addition to the specialist models, we always have a generalist model so that we can deal with classes for which we have no specialists and so that we can decide which specialists to use. Given an input image , we do top-one classification in two steps:
在调查专家模型蒸馏后会发生什么之前,我们想看看包含专家的组合表现如何。 除了专家模式之外,我们总是有一个通用模式,这样我们就可以处理那些我们没有专家的班级,并且可以决定使用哪个专家. 鉴于输入图像,我们分两个步骤进行头一个分类:
Step 1: For each test case, we find the most probable classes according to the generalist model. Call this set of classes . In our experiments, we used .
第1步:对于每个测试案例,我们发现最可能按照通论模式分类. 调用此类。 在我们的实验中,我们使用了.
Step 2: We then take all the specialist models, , whose special subset of confusable classes, , has a non-empty intersection with and call this the active set of specialists (note that this set may be empty). We then find the full probability distribution over all the classes that minimizes:
第二步:我们然后采取所有的专家模型,,其特殊子集的可混淆类,,与有一个非空的交叉点,并称这组活跃的专家为(注意此集可能是空的). 然后,我们将发现在所有类别中的全部概率分布,以尽量减少:
where denotes the KL divergence, and denote the probability distribution of a specialist model or the generalist full model. The distribution is a distribution over all the specialist classes of plus a single dustbin class, so when computing its KL divergence from the full distribution we sum all of the probabilities that the full distribution assigns to all the classes in ’s dustbin.
其中表示KL差异, 表示专家模型或通论者全模型的概率分布. 209. 20世纪80年代,在20世纪80年代,在20世纪70年代,在20世纪70年代,在20世纪80年代,在20世纪70年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪90年代,在20世纪70年代,在20世纪中,在20世纪50年代,在20世纪中,在20世纪中,在20世纪90年代,在20世纪90年代中,在20世纪90年代,在20世纪中,在20世纪90年代,在20世纪在20世纪中,在20世纪,在20世纪中,在20世纪90年代
Eq. 5 does not have a general closed form solution, though when all the models produce a single probability for each class the solution is either the arithmetic or geometric mean, depending on whether we use or ). We parameterize (with ) and we use gradient descent to optimize the logits w.r.t. eq. 5. Note that this optimization must be carried out for each image.
Eq. 5没有一般的闭合形式溶液,但当所有模型为每个类产生单一概率时,溶液要么是算术,要么是几何平均值,这取决于我们是使用还是. 我们参数化了(与),我们使用梯度下降来优化logitsw.r.t.eq. 5. 请注意,必须针对每个图像进行这种优化.
5.5 Results
5.5 结果
| System | Conditional Test Accuracy | Test Accuracy |
|---|---|---|
| Baseline | 43.1% | 25.0% |
| + 61 Specialist models | 45.9% | 26.1% |
Table 3: Classification accuracy (top 1) on the JFT development set.
| 系统 | 有条件测试准确性 | 测试精确度 |
|---|---|---|
| 基线 | 43.1% | 25.0% |
| +61 专家模型 | 45.9% | 26.1% |
表3:JFT开发集的分类精度(上1).
Starting from the trained baseline full network, the specialists train extremely fast (a few days instead of many weeks for JFT). Also, all the specialists are trained completely independently. Table 3 shows the absolute test accuracy for the baseline system and the baseline system combined with the specialist models. With 61 specialist models, there is a 4.4% relative improvement in test accuracy overall. We also report conditional test accuracy, which is the accuracy by only considering examples belonging to the specialist classes, and restricting our predictions to that subset of classes.
从训练有素的基线全网络开始,专家的训练速度非常快(对JFT来说是几天而不是多星期)。 此外,所有专家都接受完全独立的培训。 表3显示基线系统和基线系统的绝对测试精度,以及专家模型。 拥有61个专业型号,测试精度总体相对提高4.4%. 我们还报告有条件的测试准确性,即只考虑属于专家类的事例,将我们的预测局限于该类。
| # of specialists covering | # of test examples | delta in top1 correct | relative accuracy change |
|---|---|---|---|
| 0 | 350037 | 0 | 0.0% |
| 1 | 141993 | +1421 | +3.4% |
| 2 | 67161 | +1572 | +7.4% |
| 3 | 38801 | +1124 | +8.8% |
| 4 | 26298 | +835 | +10.5% |
| 5 | 16474 | +561 | +11.1% |
| 6 | 10682 | +362 | +11.3% |
| 7 | 7376 | +232 | +12.8% |
| 8 | 4703 | +182 | +13.6% |
| 9 | 4706 | +208 | +16.6% |
| 10 or more | 9082 | +324 | +14.1% |
Table 4: Top 1 accuracy improvement by # of specialist models covering correct class on the JFT test set.
| 专家人数 | # 测试实例 | 顶部三角洲正确 | 相对准确性 32. |
|---|---|---|---|
| 0 | 350037 | 0 | 0.0% |
| 1 | 141993 | +1421 | +3.4% |
| 2 | 67161 | +1572 | +7.4% |
| 3 | 38801 | +1124 | +8.8% |
| 4 | 26298 | +835 | +10.5% |
| 5 | 16474 | +561 | +11.1% |
| 6 | 10682 | +362 | +11.3% |
| 7 | 7376 | +232 | +12.8% |
| 8 | 4703 | +182 | +13.6% |
| 9 | 4706 | +208 | +16.6% |
| 10岁或以上 | 9082 | +324 | +14.1% |
表4:通过##在JFT测试集中覆盖正确类的专家模型,提高最高1级的精度.
For our JFT specialist experiments, we trained 61 specialist models, each with 300 classes (plus the dustbin class). Because the sets of classes for the specialists are not disjoint, we often had multiple specialists covering a particular image class. Table 4 shows the number of test set examples, the change in the number of examples correct at position 1 when using the specialist(s), and the relative percentage improvement in top1 accuracy for the JFT dataset broken down by the number of specialists covering the class. We are encouraged by the general trend that accuracy improvements are larger when we have more specialists covering a particular class, since training independent specialist models is very easy to parallelize.
为了我们的JFT专家实验,我们培训了61个专家模型,每个模型都有300个班级(加上尘滩班)。 由于为专家开设的系列课程并非不相干,我们经常有多个专家负责一个特定的图像课程。 表4显示测试集实例的数量、在使用专家时在位置1正确实例数量的变化,以及按涵盖该类的专家人数分列的JFT数据集在顶级精确度方面的相对百分比提高。 令我们感到鼓舞的是,总的趋势是,当我们有更多的专家负责某一特定课程时,精确度的提高会更大,因为培训独立的专家模式非常容易平行。
6 Soft Targets as Regularizers
6 软目标作为正则
One of our main claims about using soft targets instead of hard targets is that a lot of helpful information can be carried in soft targets that could not possibly be encoded with a single hard target. In this section we demonstrate that this is a very large effect by using far less data to fit the 85M parameters of the baseline speech model described earlier. Table 5 shows that with only 3% of the data (about 20M examples), training the baseline model with hard targets leads to severe overfitting (we did early stopping, as the accuracy drops sharply after reaching 44.5%), whereas the same model trained with soft targets is able to recover almost all the information in the full training set (about 2% shy). It is even more remarkable to note that we did not have to do early stopping: the system with soft targets simply “converged” to 57%. This shows that soft targets are a very effective way of communicating the regularities discovered by a model trained on all of the data to another model.
我们关于使用软目标而不是硬目标的主要说法之一是,许多有益的信息可以在软目标中携带,而软目标不可能用单一硬目标编码. 在本节中,我们通过使用的数据少得多来适应前面所述的基线语音模型的85M参数,表明这是一个非常大的影响。 表5显示,只有3%的数据(约20M例),用硬目标培训基线模型会导致严重过度调整(我们确实提前停止了,因为精度在达到44.5%后急剧下降),而同样受过软目标培训的模型则能够在完整的培训集中回收几乎所有信息(约2%害羞). 更值得注意的是,我们不必提前停止:软目标系统只是“缩合”到57%。 这表明软目标是一种非常有效的方法,可以将一个对所有数据受过训练的模型所发现的规律性传达给另一个模型.
| System & training set | Train Frame Accuracy | Test Frame Accuracy |
|---|---|---|
| Baseline (100% of training set) | 63.4% | 58.9% |
| Baseline (3% of training set) | 67.3% | 44.5% |
| Soft Targets (3% of training set) | 65.4% | 57.0% |
Table 5: Soft targets allow a new model to generalize well from only 3% of the training set. The soft targets are obtained by training on the full training set.
| 系统和培训集 | 列车框架精确度 | 测试框架精确度 |
|---|---|---|
| 基线(整套培训的100%) | 63.4% | 58.9% |
| 基线(成套培训的3%) | 67.3% | 44.5% |
| 软目标(占整套培训的3%) | 65.4% | 57.0% |
表5:软目标使新模式能够从仅占培训组合的3%的高度概括。 软目标是通过全套培训的培训获得的。
6.1 Using soft targets to prevent specialists from overfitting
6.1 用软目标防止专家过拟合
The specialists that we used in our experiments on the JFT dataset collapsed all of their non-specialist classes into a single dustbin class. If we allow specialists to have a full softmax over all classes, there may be a much better way to prevent them overfitting than using early stopping. A specialist is trained on data that is highly enriched in its special classes. This means that the effective size of its training set is much smaller and it has a strong tendency to overfit on its special classes. This problem cannot be solved by making the specialist a lot smaller because then we lose the very helpful transfer effects we get from modeling all of the non-specialist classes.
我们在JFT数据集实验中使用的专家 把他们所有的非专家阶层 都变成了一个单一的垃圾桶阶层 如果我们允许专家在所有班级都拥有一个完整的软顶,也许有更好的方法来防止他们过拟合,而不是使用早停. 专家接受特殊课程中高度丰富的数据培训。 这意味着其培训集的有效规模要小得多,而且其特别班级往往过于适合。 这个问题不能通过使专家小很多的方法来解决,因为那样我们就会失去我们从模拟所有非专家班级中获得的非常有益的转移效应.
Our experiment using 3% of the speech data strongly suggests that if a specialist is initialized with the weights of the generalist, we can make it retain nearly all of its knowledge about the non-special classes by training it with soft targets for the non-special classes in addition to training it with hard targets. The soft targets can be provided by the generalist. We are currently exploring this approach.
我们用3%的语音数据进行的实验强烈地表明,如果一位专家以通论者的权重初始化,我们就可以通过培训非特殊课程的软目标,以及培训硬目标,使其保留几乎所有关于非特殊课程的知识。 软目标可以由通论者提供. 我们目前正在探讨这种做法。
7 Relationship to Mixtures of Experts
7 与专家混合的关系
The use of specialists that are trained on subsets of the data has some resemblance to mixtures of experts [6] which use a gating network to compute the probability of assigning each example to each expert. At the same time as the experts are learning to deal with the examples assigned to them, the gating network is learning to choose which experts to assign each example to based on the relative discriminative performance of the experts for that example. Using the discriminative performance of the experts to determine the learned assignments is much better than simply clustering the input vectors and assigning an expert to each cluster, but it makes the training hard to parallelize: First, the weighted training set for each expert keeps changing in a way that depends on all the other experts and second, the gating network needs to compare the performance of different experts on the same example to know how to revise its assignment probabilities. These difficulties have meant that mixtures of experts are rarely used in the regime where they might be most beneficial: tasks with huge datasets that contain distinctly different subsets.
使用接受过数据子集培训的专家,与专家的混合物有些相似[6],这些专家使用一个绘图网络计算将每个实例分配给每个专家的概率。 在专家学习处理分配给他们的例子的同时,定位网络正在学习根据专家的相对歧视性表现来选择哪些专家分配每个例子。 利用专家的差别性表现来确定所学到的任务要比简单地将输入矢量组合起来并指派一名专家到每个集群要好得多,但这使得培训难以平行化:第一,对每个专家的加权培训不断以取决于所有其他专家的方式变化,第二,配对网络需要用同一例子比较不同专家的成绩,以了解如何修改任务概率。 这些困难意味着专家的混合体很少用于可能最有利的制度:包含不同子集的巨大数据集的任务。
It is much easier to parallelize the training of multiple specialists. We first train a generalist model and then use the confusion matrix to define the subsets that the specialists are trained on. Once these subsets have been defined the specialists can be trained entirely independently. At test time we can use the predictions from the generalist model to decide which specialists are relevant and only these specialists need to be run.
在培训多位专家的同时,要容易得多。 我们首先训练一个通用模型,然后使用混淆矩阵来定义专家所训练的子集. 一旦确定了这些子集,专家就可以完全独立地接受培训。 在试验时,我们可以利用通用模型的预测来决定哪些专家是相关的,只有这些专家需要运行.
8 Discussion
8 讨论
We have shown that distilling works very well for transferring knowledge from an ensemble or from a large highly regularized model into a smaller, distilled model. On MNIST distillation works remarkably well even when the transfer set that is used to train the distilled model lacks any examples of one or more of the classes. For a deep acoustic model that is version of the one used by Android voice search, we have shown that nearly all of the improvement that is achieved by training an ensemble of deep neural nets can be distilled into a single neural net of the same size which is far easier to deploy.
我们已经表明,蒸馏非常有助于将知识从一个综艺或从一个高度规范的大型模型转移到一个较小的、蒸馏模型。 在MNIST蒸馏上,即使用于训练蒸馏模型的调试装置缺乏一个或多个类别的例子,效果也非常好。 对于Android语音搜索所使用的深音学模型, 我们已经表明, 通过训练一组深神经网而实现的几乎所有改进, 都可以被蒸馏成一个大小相同的单一神经网, 而这个神经网的部署要容易得多。
For really big neural networks, it can be infeasible even to train a full ensemble, but we have shown that the performance of a single really big net that has been trained for a very long time can be significantly improved by learning a large number of specialist nets, each of which learns to discriminate between the classes in a highly confusable cluster. We have not yet shown that we can distill the knowledge in the specialists back into the single large net.
对于真正的大型神经网络来说,甚至训练一个完整的综艺节目都可能不可行,但我们已经表明,通过学习大量专家网,可以大大改进一个经过很长一段时间训练的单一的真正的大型网的性能,每个网都学会在一个高度混杂的集群中区分班级. 我们尚未表明,我们能够将专家的知识重新提炼成单一的大网。
Acknowledgments
We thank Yangqing Jia for assistance with training models on ImageNet and Ilya Sutskever and Yoram Singer for helpful discussions.
++ 承认
我们感谢Yangqing Jia在ImageNet、Ilya Sutskever和Yoram Singer的培训模式方面给予的协助。
References
参考文献
[1] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD ’06, pages 535–541, New York, NY, USA, 2006. ACM.
[2] J. Dean, G. S. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In NIPS , 2012.
[3] T. G. Dietterich. Ensemble methods in machine learning. In Multiple classifier systems , pages 1–15. Springer, 2000.
[4] G. E. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE , 29(6):82–97, 2012.
[5] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 , 2012.
[6] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79–87, 1991.
[7] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems , pages 1097–1105, 2012.
[8] J. Li, R. Zhao, J. Huang, and Y. Gong. Learning small-size dnn with output-distribution-based criteria. In Proceedings Interspeech 2014 , pages 1910–1914, 2014.
[9] N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929–1958, 2014.