Quantum-chemical insights from deep tensor neural networks | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
AbstractIntroductionResultsMolecular deep tensor neural networksLearning molecular energiesLocal chemical potentialEnergy predictions for isomersDiscussionMethods1. Assign initial atomic descriptors2. Gaussian feature expansion of the inter-atomic distances3. Perform $T$ interaction passes4. Predict energy contributions5. Obtain the molecular energy $E = \sumi Ei$Additional informationAcknowledgementsAuthor contributionsReferences
返回论文列表
目录

Quantum-chemical insights from deep tensor neural networks

收录于 Nature Communications 期刊论文;Nature Communications 8, 13890 (2017);DOI 10.1038/ncomms13890被引 1,474 次
打开论文来源

Quantum-chemical insights from deep tensor neural networks

Received 24 Jun 2016 | Accepted 9 Nov 2016 | Published 9 Jan 2017

DOI: 10.1038/ncomms13890 OPEN

Kristof T. Schütt^{1}, Farhad Arbabzadah^{1}, Stefan Chmiela^{1}, Klaus R. Müller^{1,2} & Alexandre Tkatchenko^{3,4}

^{1} Machine Learning Group, Technische Universität Berlin, Marchstr. 23, 10587 Berlin, Germany. ^{2} Department of Brain and Cognitive Engineering, Korea University, Anam-dong, Seongbuk-gu, Seoul 136-713, Republic of Korea. ^{3} Theory Department, Fritz-Haber-Institut der Max-Planck-Gesellschaft, Faradayweg 4-6, D-14195 Berlin, Germany. ^{4} Physics and Materials Science Research Unit, University of Luxembourg, Luxembourg,, L-1511 Luxembourg. Correspondence and requests for materials should be addressed to K.R.M. (email: klaus-robert.mueller@tu-berlin.de) or to A.T. (email: alexandre.tkatchenko@uni.lu).

Abstract

Learning from data has led to paradigm shifts in a multitude of disciplines, including web, text and image search, speech recognition, as well as bioinformatics. Can machine learning enable similar breakthroughs in understanding quantum many-body systems? Here we develop an efficient deep learning approach that enables spatially and chemically resolved insights into quantum-mechanical observables of molecular systems. We unify concepts from many-body Hamiltonians with purpose-designed deep tensor neural networks, which leads to size-extensive and uniformly accurate (1 kcal mol^{−1}) predictions in compositional and configurational chemical space for molecules of intermediate size. As an example of chemical relevance, the model reveals a classification of aromatic rings with respect to their stability. Further applications of our model for predicting atomic energies and local chemical potentials in molecules, reliable isomer energies, and molecules with peculiar electronic structure demonstrate the potential of machine learning for revealing insights into complex quantum-chemical systems.

从数据中学习,已经在许多学科里带来了范式转变,包括网络、文本和图像检索、语音识别以及生物信息学。机器学习是否也能在理解量子多体体系上带来类似突破?我们在此发展一种高效的深度学习方法,能够对分子体系的量子力学可观测量给出空间分辨和化学分辨的认识。我们把多体哈密顿量中的概念与专门设计的深度张量神经网络统一起来,从而在中等大小分子的组成与构型化学空间中,得到具有规模广延性、并且一致准确(1 kcal mol^{−1})的预测。作为一个有化学意义的例子,该模型揭示了按稳定性对芳环的一种分类。我们还用它预测分子中的原子能量和局域化学势、可靠的同分异构体能量,以及具有特殊电子结构的分子,以此表明机器学习有潜力揭示复杂量子化学体系中的洞见。

Introduction

Chemistry permeates all aspects of our life, from the development of new drugs to the food that we consume and materials we use on a daily basis. Chemists rely on empirical observations based on creative and painstaking experimentation that leads to eventual discoveries of molecules and materials with desired properties and mechanisms to synthesize them. Many discoveries in chemistry can be guided by searching large databases of experimental or computational molecular structures and properties by using concepts based on chemical similarity. Because the structure and properties of molecules are determined by the laws of quantum mechanics, ultimately chemical discovery must be based on fundamental quantum principles. Indeed, electronic structure calculations and intelligent data analysis (machine learning) have recently been combined aiming towards the goal of accelerated discovery of chemicals with desired properties^{1,2,3,4,5,6,7,8}. However, so far the majority of these pioneering efforts have focused on the construction of reduced models trained on large data sets of density-functional theory calculations.

化学渗透在我们生活的各个方面,从新药研发,到日常食用的食物,再到每天使用的材料。化学家依靠创造性而又艰苦的实验所得到的经验观察,最终发现具有所需性质的分子与材料,以及合成它们的途径。许多化学发现可以通过在大型实验或计算分子结构与性质数据库中,借助化学相似性概念进行检索来加以引导。由于分子的结构和性质由量子力学定律决定,化学发现最终必须建立在基本的量子原理之上。事实上,电子结构计算与智能数据分析(机器学习)近来已被结合起来,目标是加速发现具有所需性质的化学品^{1,2,3,4,5,6,7,8}。不过到目前为止,这些开创性努力大多仍集中于在大型密度泛函理论计算数据集上训练得到简化模型。

In this work, we develop an efficient deep learning approach that enables spatially and chemically resolved insights into quantum-mechanical properties of molecular systems beyond those trivially contained in the training dataset. Obviously, computational models are not predictive if they lack accuracy. In addition to being interpretable, size-extensive and efficient, our deep tensor neural network (DTNN) approach is uniformly accurate (1 kcal mol^{−1}) throughout compositional and configurational chemical space. On the more fundamental side, the mathematical construction of the DTNN model provides statistically rigorous partitioning of extensive molecular properties into atomic contributions—a long-standing challenge for quantum-mechanical calculations of molecules.

在这项工作中,我们发展一种高效的深度学习方法,能够对分子体系的量子力学性质给出空间分辨和化学分辨的认识,而且这些认识超出了训练数据中直接包含的内容。计算模型如果不够准确,显然就不具有预测能力。除了可解释、具有规模广延性并且高效之外,我们的深度张量神经网络(DTNN)方法在组成与构型化学空间中都达到一致的准确度(1 kcal mol^{−1})。在更根本的层面上,DTNN 模型的数学构造提供了一种把广延分子性质划分到原子贡献上的统计上严谨的方案——这是分子量子力学计算中一个长期存在的难题。

Results

Molecular deep tensor neural networks

It is common to use a carefully chosen representation of the problem at hand as a basis for machine learning^{9,10,11}. For example, molecules can be represented as Coulomb matrices^{7,12,13}, scattering transforms^{14}, bags of bonds^{15}, smooth overlap of atomic positions^{16,17} or generalized symmetry functions^{18,19}. Kernel-based learning of molecular properties transforms these representations non-linearly by virtue of kernel functions. In contrast, deep neural networks^{20} are able to infer the underlying regularities and learn an efficient representation in a layer-wise fashion^{21}.

通常会先为手头的问题精心选择一种表示,再把它作为机器学习的基础^{9,10,11}。例如,分子可以表示为库仑矩阵^{7,12,13}、散射变换^{14}、键袋(bags of bonds)^{15}、原子位置平滑重叠(smooth overlap of atomic positions)^{16,17},或广义对称函数^{18,19}。基于核的分子性质学习借助核函数对这些表示做非线性变换。相比之下,深度神经网络^{20}能够推断其中的规律,并以逐层的方式学到一种高效表示^{21}。

Molecular properties are governed by the laws of quantum mechanics, which yield the remarkable flexibility of chemical systems, but also impose constraints on the behaviour of bonding in molecules. The approach presented here utilizes the many-body Hamiltonian concept for the construction of the DTNN architecture (Fig. 1), embracing the principles of quantum chemistry, while maintaining the full flexibility of a complex data-driven learning machine.

分子性质受量子力学定律支配,这些定律既带来了化学体系惊人的灵活性,也对分子中成键的行为施加了约束。本文给出的方法在构造 DTNN 结构时借用了多体哈密顿量的概念(图 1),既接纳量子化学的原理,又保留复杂数据驱动学习机器的全部灵活性。

Figure 1 | Prediction and explanation of molecular energies with a deep tensor neural network. (a) Molecules are encoded as input for the neural network by a vector of nuclear charges and an inter-atomic distance matrix. This description is complete and invariant to rotation and translation. (b) Illustration of the network architecture. Each atom type corresponds to a vector of coefficients ci(0)\mathbf{c}_i^{(0)}ci(0)​, which is repeatedly refined by interactions vij\mathbf{v}_{ij}vij​. The interactions depend on the current representation ci(t)\mathbf{c}_i^{(t)}ci(t)​, as well as the distance DijD_{ij}Dij​ to an atom jjj. After TTT iterations, an energy contribution EiE_iEi​ is predicted for the final coefficient vector ci(T)\mathbf{c}_i^{(T)}ci(T)​. The molecular energy EEE is the sum over these atomic contributions. (c) Mean absolute errors of predictions for the GDB-9 dataset of 133,885 molecules as a function of the number of atoms. The employed neural network uses two interaction passes (T=2T=2T=2) and 50,000 reference calculation during training. The inset shows the error of an equivalent network trained on 5,000 GDB-9 molecules with 20 or more atoms, as small molecules with 15 or less atoms are added to the training set. (d) Extract from the calculated (black) and predicted (orange) molecular dynamics trajectory of toluene. The curve on the right shows the agreement of the predicted and calculated energy distributions. (e) Energy contribution EprobeE_{\mathrm{probe}}Eprobe​ (or local chemical potential ΩHM(r)\Omega_H^M(\mathbf{r})ΩHM​(r), see text) of a hydrogen test charge on a ∑i∥r−ri∥−2\sum_i\|\mathbf{r}-\mathbf{r}_i\|^{-2}∑i​∥r−ri​∥−2 isosurface for various molecules from the GDB-9 dataset for a DTNN model with T=2T=2T=2.

DTNN Figure 1: prediction and explanation of molecular energies

图 1|用深度张量神经网络预测并解释分子能量。(a) 分子被编码为核电荷向量和核间距矩阵,作为神经网络的输入。这一描述是完备的,并且对旋转和平移不变。(b) 网络结构示意。每种原子对应一个系数向量 ci(0)\mathbf{c}_i^{(0)}ci(0)​,并经由相互作用 vij\mathbf{v}_{ij}vij​ 反复 refinement。这些相互作用依赖于当前表示 ci(t)\mathbf{c}_i^{(t)}ci(t)​,以及到原子 jjj 的距离 DijD_{ij}Dij​。经过 TTT 次迭代后,由最终系数向量 ci(T)\mathbf{c}_i^{(T)}ci(T)​ 预测能量贡献 EiE_iEi​。分子能量 EEE 是这些原子贡献之和。(c) 在包含 133,885 个分子的 GDB-9 数据集上,预测的平均绝对误差随原子数的变化。所用神经网络包含两次相互作用传递(T=2T=2T=2),训练时使用 50,000 个参考计算。插图显示:在一个用 5,000 个含 20 个或更多原子的 GDB-9 分子训练的同等网络上,把含 15 个或更少原子的小分子逐步加入训练集后误差的变化。(d) 甲苯分子动力学轨迹的一个片段:计算值(黑)与预测值(橙)。右侧曲线显示预测与计算能量分布的符合程度。(e) 在来自 GDB-9 的若干分子上,氢测试电荷在 ∑i∥r−ri∥−2\sum_i\|\mathbf{r}-\mathbf{r}_i\|^{-2}∑i​∥r−ri​∥−2 等值面上的能量贡献 EprobeE_{\mathrm{probe}}Eprobe​(或局域化学势 ΩHM(r)\Omega_H^M(\mathbf{r})ΩHM​(r),见正文);所用 DTNN 模型为 T=2T=2T=2。

DTNN receives molecular structures through a vector of nuclear charges ZZZ and a matrix of atomic distances DDD ensuring rotational and translational invariance by construction (Fig. 1a). The distances are expanded in a Gaussian basis, yielding a feature vector d^ij\hat{\mathbf{d}}_{ij}d^ij​, which accounts for the different nature of interactions at various distance regimes. Similar approaches have been applied to the entries of the Coulomb matrix for the prediction of molecular properties before^{12}.

DTNN 通过核电荷向量 ZZZ 和核间距矩阵 DDD 接收分子结构,从而在构造上保证旋转和平移不变性(图 1a)。这些距离在高斯基中展开,得到特征向量 d^ij\hat{\mathbf{d}}_{ij}d^ij​,用以刻画不同距离区间上相互作用的不同性质。类似做法此前已被用于库仑矩阵的矩阵元,以预测分子性质^{12}。

The total energy EME^MEM for the molecule MMM composed of NNN atoms is written as a sum over NNN atomic energy contributions EiE_iEi​, thus satisfying permutational invariance with respect to atom indexing. Each atom iii is represented by a coefficient vector ci∈RB\mathbf{c}_i \in \mathbb{R}^Bci​∈RB, where BBB is the number of basis functions, or features. Motivated by quantum-chemical atomic basis set expansions, we assign an atom type-specific descriptor vector to these coefficients ci(0)=cZi\mathbf{c}_i^{(0)}=\mathbf{c}_{Z_i}ci(0)​=cZi​​. Subsequently, this atomic expansion is repeatedly refined by pairwise interactions with the surrounding atoms

由 NNN 个原子组成的分子 MMM,其总能量 EME^MEM 写成 NNN 个原子能量贡献 EiE_iEi​ 之和,从而对原子编号满足置换不变性。每个原子 iii 由系数向量 ci∈RB\mathbf{c}_i \in \mathbb{R}^Bci​∈RB 表示,其中 BBB 是基函数(或特征)的数目。受量子化学原子基组展开的启发,我们为这些系数赋予一个与原子类型有关的描述向量 ci(0)=cZi\mathbf{c}_i^{(0)}=\mathbf{c}_{Z_i}ci(0)​=cZi​​。随后,这一原子展开通过与周围原子的成对相互作用被反复 refinement

cit+1=cit+∑j≠ivij.(1)\mathbf{c}_i^{t+1} = \mathbf{c}_i^t + \sum_{j \neq i} \mathbf{v}_{ij}. \quad (1)cit+1​=cit​+j=i∑​vij​.(1)

where the interaction term vij\mathbf{v}_{ij}vij​ reflects the influence of atom jjj at a distance DijD_{ij}Dij​ on atom iii. Note that this refinement step is seamlessly integrated into the architecture of the molecular DTNN, and is therefore adapted throughout the learning process. In Supplementary Discussion, we show the relation to convolutional neural networks that have been applied to images, speech and text with great success because of their ability to capture local structure^{22,23,24,25,26,27}. Considering a molecule as a graph, TTT refinements of the coefficient vectors are comprised of all walks of length TTT through the molecule ending at the corresponding atom^{28,29}. From the point of view of many-body interatomic interactions, subsequent refinement steps ttt correlate atomic neighbourhoods with increasing complexity.

其中相互作用项 vij\mathbf{v}_{ij}vij​ 反映原子 jjj 在距离 DijD_{ij}Dij​ 处对原子 iii 的影响。需要注意的是,这一 refinement 步骤被无缝地写进分子 DTNN 的结构之中,因此会在整个学习过程中一起被调整。在 Supplementary Discussion 中,我们说明了它与卷积神经网络的关系;后者因能够捕捉局部结构,已在图像、语音和文本上取得很大成功^{22,23,24,25,26,27}。若把分子看作图,对系数向量做 TTT 次 refinement,便包含了分子中所有长度为 TTT、并在相应原子处终止的通路^{28,29}。从多体原子间相互作用的角度看,随后的 refinement 步骤 ttt 会把越来越复杂的原子邻域关联起来。

While the initial atomic representations only consider isolated atoms, the interaction terms characterize how the basis functions of two atoms overlap with each other at a certain distance. Each refinement step is supposed to reduce these overlaps, thereby embedding the atoms of the molecule into their chemical environment. Following this procedure, the DTNN implicitly learns an atom-centered basis that is unique and efficient with respect to the property to be predicted.

初始的原子表示只考虑孤立原子,而相互作用项则刻画两个原子的基函数在某一距离上如何相互重叠。每一步 refinement 都应减小这些重叠,从而把分子中的原子嵌入到它们的化学环境中。按这一步骤,DTNN 隐式地学到一种以原子为中心的基,它对所要预测的性质而言既独特又高效。

Non-linear coupling between the atomic vector features and the interatomic distances is achieved by a tensor layer^{30,31,32}, such that the coefficient kkk of the refinement is given by

原子向量特征与核间距之间的非线性耦合由一个张量层实现^{30,31,32},从而 refinement 的第 kkk 个系数由下式给出

vijk=tanh⁡(bk+∑lmVklmcjl d^ijm+∑lWklccjl+∑mWkmdd^ijm).(2)v_{ijk} = \tanh\left( b_k + \sum_{lm} V_{klm} c_{jl}\, \hat{d}_{ijm} + \sum_l W^c_{kl} c_{jl} + \sum_m W^d_{km} \hat{d}_{ijm} \right). \quad (2)vijk​=tanh(bk​+lm∑​Vklm​cjl​d^ijm​+l∑​Wklc​cjl​+m∑​Wkmd​d^ijm​).(2)

where bkb_kbk​ is the bias of feature kkk and WcW^cWc and WdW^dWd are the weights of atom representation and distance, respectively. The slice VkV_kVk​ of the parameter tensor VVV combines the inputs multiplicatively. Since VVV incorporates many parameters, using this kind of layer is both computationally expensive as well as prone to overfitting. Therefore, we employ a low-rank tensor factorization, as described in (ref. 33), such that

其中 bkb_kbk​ 是特征 kkk 的偏置,WcW^cWc 和 WdW^dWd 分别是原子表示和距离的权重。参数张量 VVV 的切片 VkV_kVk​ 以相乘的方式组合输入。由于 VVV 含有许多参数,使用这类层既计算昂贵,也容易过拟合。因此我们采用文献 33 中所述的低秩张量分解,使得

vij=tanh⁡[Wfc((Wcfcj+bf1)∘(Wdfd^ij+bf2))],(3)\mathbf{v}_{ij} = \tanh\left[ W^{fc} \left( (W^{cf} \mathbf{c}_j + \mathbf{b}^{f1}) \circ (W^{df} \hat{\mathbf{d}}_{ij} + \mathbf{b}^{f2}) \right) \right], \quad (3)vij​=tanh[Wfc((Wcfcj​+bf1)∘(Wdfd^ij​+bf2))],(3)

where ‘∘\circ∘’ represents element-wise multiplication, while WcfW^{cf}Wcf, bf1\mathbf{b}^{f1}bf1, WdfW^{df}Wdf, bf2\mathbf{b}^{f2}bf2 and WfcW^{fc}Wfc are the weight matrices and corresponding biases of atom representations, distances and resulting factors, respectively. As the dimensionality of WcfcjW^{cf}\mathbf{c}_jWcfcj​ and Wdfd^ijW^{df}\hat{\mathbf{d}}_{ij}Wdfd^ij​ corresponds to the number of factors, choosing only a few drastically decreases the number of parameters, thus solving both issues of the tensor layer at once.

其中 ‘∘\circ∘’ 表示逐元素相乘,而 WcfW^{cf}Wcf、bf1\mathbf{b}^{f1}bf1、WdfW^{df}Wdf、bf2\mathbf{b}^{f2}bf2 和 WfcW^{fc}Wfc 分别是原子表示、距离以及所得因子的权重矩阵和相应偏置。由于 WcfcjW^{cf}\mathbf{c}_jWcfcj​ 和 Wdfd^ijW^{df}\hat{\mathbf{d}}_{ij}Wdfd^ij​ 的维数对应于因子数目,只选取少数几个因子就会大大减少参数量,从而同时解决张量层的这两个问题。

Arriving at the final embedding after a given number of interaction refinements, two fully-connected layers predict an energy contribution from each atomic coefficient vector, such that their sum corresponds to the total molecular energy EME^MEM. Therefore, the DTNN architecture scales with the number of atoms in a molecule, fully capturing the extensive nature of the energy. All weights, biases, as well as the atom type-specific descriptors were initialized randomly and trained using stochastic gradient descent.

在经过给定次数的相互作用 refinement 后得到最终嵌入,再用两个全连接层从每个原子系数向量预测能量贡献,使它们的总和对应于分子总能量 EME^MEM。因此,DTNN 结构随分子中的原子数扩展,完整抓住了能量的广延性质。所有权重、偏置以及与原子类型有关的描述符都随机初始化,并用随机梯度下降训练。

Learning molecular energies

To demonstrate the versatility of the proposed DTNN, we train models with up to three interaction passes T=3T=3T=3 for both compositional and configurational degrees of freedom in molecular systems. The DTNN accuracy saturates at T=3T=3T=3, and leads to a strong correlation between atoms in molecules, as can be visualized by the complexity of the potential learned by the network (Fig. 1e). For training, we employ chemically diverse data sets of equilibrium molecular structures, as well as molecular dynamics (MD) trajectories for small molecules. We employ two subsets of the GDB-13 database^{34,35} referred to as GDB-7, including >7,000 molecules with up to seven heavy (C, N, O, F) atoms, and GDB-9, consisting of 133,885 molecules with up to nine heavy atoms^{36}. In both cases, the learning task is to predict the molecular total energy calculated with density-functional theory (DFT). All GDB molecules are stable and synthetically accessible according to organic chemistry rules^{35}. Molecular features such as functional groups or signatures include single, double and triple bonds; (hetero-) cycles, carboxy, cyanide, amide, amine, alcohol, epoxy, sulphide, ether, ester, chloride, aliphatic and aromatic groups. For each of the many possible stoichiometries, many constitutional isomers are considered, each being represented only by a low-energy conformational isomer.

为了说明所提出的 DTNN 的通用性,我们在分子体系的组成自由度和构型自由度上,训练最多包含三次相互作用传递(T=3T=3T=3)的模型。DTNN 的精度在 T=3T=3T=3 时趋于饱和,并使分子中的原子之间产生强关联,这一点可以从网络所学势的复杂程度中看出(图 1e)。训练时,我们使用化学上多样的平衡分子结构数据集,以及小分子的分子动力学(MD)轨迹。我们采用 GDB-13 数据库^{34,35} 的两个子集:称为 GDB-7 的子集包含 7,000 多个最多含 7 个重原子(C、N、O、F)的分子;GDB-9 则包含 133,885 个最多含 9 个重原子的分子^{36}。在两种情形下,学习任务都是预测用密度泛函理论(DFT)算出的分子总能量。按照有机化学规则,所有 GDB 分子都是稳定且可合成的^{35}。分子特征(如官能团或签名)包括单键、双键和三键;(杂)环、羧基、氰基、酰胺、胺、醇、环氧、硫醚、醚、酯、氯化物,以及脂肪族和芳香族基团。对每一种可能的化学计量,都考虑了许多构造异构体,而每个构造异构体只由一个低能构象异构体代表。

As Supplementary Table 1 demonstrates, DTNN achieves a mean absolute error of 1.0 kcal mol^{−1} on both GDB data sets, training on 5.8 k GDB-7 (80%) and 25 k (20%) GDB-9 reference calculations, respectively. Figure 1c shows the performance on GDB-9 depending on the size of the molecule. We observe that larger molecules have lower errors because of their abundance in the training data. However, when predicting larger molecules than present in the training set, the errors increase. This is because the molecules in the GDB-9 set are quite small, so we considered all atoms to be in each other's chemical environment. Imposing a distance cutoff to interatomic interactions of 3 Å leads to a 0.1 kcal mol^{−1} increase in the error. However, this distance cutoff restricts only the direct interactions considered in the refinement steps. With multiple refinements, the effective cutoff increases by a factor of TTT because of indirect interactions over multiple atoms. Given large enough molecules, so that a reasonable distance cutoff can be chosen, scaling to larger molecules will require only to have well-represented local environments. For now, we observe that at least a few larger molecules are needed to achieve a good prediction accuracy. Following this train of thought, we trained the network on a restricted subset of 5 k molecules with >20 atoms. By adding smaller molecules to the training set, we are able to reduce the test error from 2.1 kcal mol^{−1} to <1.5 kcal mol^{−1} (see inset in Fig. 1c). This result demonstrates that our model is able to transfer knowledge learned from small molecules to larger molecules with diverse functional groups.

正如 Supplementary Table 1 所示,DTNN 在两个 GDB 数据集上的平均绝对误差都可以达到 1.0 kcal mol^{−1},训练分别使用 5.8 k 个 GDB-7 参考计算(占 80%)和 25 k 个 GDB-9 参考计算(占 20%)。图 1c 给出了 GDB-9 上的表现随分子大小的变化。我们看到,较大分子的误差更低,因为它们在训练数据中更为丰富。不过,当预测比训练集中出现过的分子更大的分子时,误差会上升。这是因为 GDB-9 中的分子相当小,我们把所有原子都视为处于彼此的化学环境之中。若对原子间相互作用施加 3 Å 的距离截断,误差会增加 0.1 kcal mol^{−1}。不过,这一距离截断只限制 refinement 步骤中考虑的直接相互作用。经过多次 refinement 后,由于经由多个原子的间接相互作用,有效截断会增大到 TTT 倍。如果分子足够大,从而可以选择一个合理的距离截断,那么扩展到更大分子时,只需要局部环境得到充分表示。就目前而言,我们看到至少需要一些较大分子,才能获得较好的预测精度。沿着这一思路,我们在一个受限子集上训练网络,该子集包含 5 k 个原子数大于 20 的分子。通过把较小分子加入训练集,我们能够把测试误差从 2.1 kcal mol^{−1} 降到 <1.5 kcal mol^{−1}(见图 1c 插图)。这一结果表明,我们的模型能够把从小分子学到的知识迁移到具有多样官能团的较大分子上。

While only encompassing conformations of a single molecule, reproducing MD simulation trajectories poses a radically different challenge to predicting energies of purely equilibrium structures. We learned potential energies for MD trajectories of benzene, toluene, malonaldehyde and salicylic acid, carried out at a rather high temperature of 500 K to achieve exhaustive exploration of the potential-energy surface of such small molecules. The neural network yields mean absolute errors of 0.05, 0.18, 0.17 and 0.39 kcal mol^{−1} for these molecules, respectively (Supplementary Table 1). Figure 1d shows the excellent agreement between the DFT and DTNN MD trajectory of toluene, as well as the corresponding energy distributions. The DTNN errors are much smaller than the energy of thermal fluctuations at room temperature (∼0.6 kcal mol^{−1}), meaning that DTNN potential-energy surfaces can be utilized to calculate accurate molecular thermodynamic properties by virtue of Monte Carlo simulations.

尽管只涵盖单个分子的构象,再现 MD 模拟轨迹与预测纯粹平衡结构的能量是性质完全不同的挑战。我们学习了苯、甲苯、丙二醛和水杨酸的 MD 轨迹上的势能;这些轨迹在相当高的 500 K 下进行,以便对这类小分子的势能面做充分探索。神经网络对这四个分子给出的平均绝对误差分别为 0.05、0.18、0.17 和 0.39 kcal mol^{−1}(Supplementary Table 1)。图 1d 显示了甲苯的 DFT 与 DTNN 分子动力学轨迹以及相应能量分布之间的良好符合。DTNN 的误差远小于室温下热涨落的能量(约 0.6 kcal mol^{−1}),这意味着可以借助蒙特卡洛模拟,用 DTNN 势能面计算准确的分子热力学性质。

Supplementary Figs 1 and 2 illustrate how the performance of DTNN depends on the number of employed reference calculations and refinement steps (Supplementary Discussion). The ability of DTNN to accurately describe equilibrium structures within the GDB-9 database and MD trajectories of selected molecules of chemical relevance demonstrates the feasibility of developing a universal machine learning architecture that can capture compositional as well as configurational degrees of freedom in the vast chemical space. While the employed architecture of the DTNN is universal, the learned coefficients are different for GDB-9 and MD trajectories of single molecules.

Supplementary Figs 1 和 2 说明了 DTNN 的表现如何依赖于所用参考计算的数量和 refinement 步数(Supplementary Discussion)。DTNN 能够准确描述 GDB-9 数据库中的平衡结构,以及若干有化学意义的分子的 MD 轨迹,这表明有可能发展一种通用的机器学习结构,在广阔的化学空间中同时捕捉组成自由度和构型自由度。尽管所采用的 DTNN 结构是通用的,但对 GDB-9 和单分子 MD 轨迹学到的系数是不同的。

Local chemical potential

Beyond predicting accurate energies, the true power of DTNN lies in its ability to provide novel quantum-chemical insights. In the context of DTNN, we define a local chemical potential ΩAM(r)\Omega_A^M(\mathbf{r})ΩAM​(r) as an energy of a certain atom type AAA, located at a position r\mathbf{r}r in the molecule MMM. While the DTNN models the interatomic interactions, we only allow the atoms of the molecule act on the probe atom, while the probe does not influence the molecule. The spatial and chemical sensitivity provided by our DTNN approach is shown in Fig. 1e for a variety of fundamental molecular building blocks. In this case, we employed hydrogen as a test charge, while the results for A={C,N,O,H}A=\{\mathrm{C,N,O,H}\}A={C,N,O,H} are shown in Fig. 2. Despite being trained only on total energies of molecules, the DTNN approach clearly grasps fundamental chemical concepts such as bond saturation and different degrees of aromaticity. For example, the DTNN model predicts the C6O3H6\mathrm{C_6O_3H_6}C6​O3​H6​ molecule to be ‘more aromatic’ than benzene or toluene (Fig. 1e). Remarkably, it turns out that C6O3H6\mathrm{C_6O_3H_6}C6​O3​H6​ does have higher ring stability than both benzene and toluene and DTNN predicts it to be the molecule with the most stable aromatic carbon ring among all molecules in the GDB-9 database (Fig. 3). Further chemical effects learned by the DTNN model are shown in Fig. 2 that demonstrates the differences in the chemical potential distribution of H, C, N and O atoms in benzene, toluene, salicylic acid and malonaldehyde. For example, the chemical potentials of different atoms over an aromatic ring are qualitatively different for H, C, N and O atoms—an evident fact for a trained chemist. However, the subtle chemical differences described by DTNN are accompanied by chemically accurate predictions—a challenging task for humans.

除了准确预测能量之外,DTNN 真正的力量在于它能够提供新的量子化学认识。在 DTNN 的语境中,我们把局域化学势 ΩAM(r)\Omega_A^M(\mathbf{r})ΩAM​(r) 定义为某一原子类型 AAA 位于分子 MMM 中位置 r\mathbf{r}r 处的能量。尽管 DTNN 建模的是原子间相互作用,我们只允许分子中的原子作用于探针原子,而探针并不影响分子。图 1e 展示了我们的 DTNN 方法对多种基本分子结构单元所给出的空间与化学灵敏度。这里我们用氢作为测试电荷,而 A={C,N,O,H}A=\{\mathrm{C,N,O,H}\}A={C,N,O,H} 的结果则示于图 2。尽管只在分子总能量上训练,DTNN 方法显然已经抓住了键饱和以及不同程度的芳香性这类基本化学概念。例如,DTNN 模型预测 C6O3H6\mathrm{C_6O_3H_6}C6​O3​H6​ 分子比苯或甲苯“更芳香”(图 1e)。值得注意的是,C6O3H6\mathrm{C_6O_3H_6}C6​O3​H6​ 的环稳定性确实高于苯和甲苯,并且 DTNN 预测它是 GDB-9 数据库所有分子中芳香碳环最稳定的分子(图 3)。图 2 进一步展示了模型学到的化学效应,给出了氢、碳、氮、氧原子在苯、甲苯、水杨酸和丙二醛中的化学势分布差异。例如,不同原子在芳环上方的化学势,对 H、C、N、O 是定性不同的——这对受过训练的化学家来说是显然的。不过,DTNN 所描述的这些细微化学差别,是与达到化学精度的预测相伴随的——这对人类而言是一项有挑战的任务。

Figure 2 | Chemical potentials ΩAM(r)\Omega_A^M(\mathbf{r})ΩAM​(r) for A={C,N,O,H}A=\{\mathrm{C, N, O, H}\}A={C,N,O,H} atoms. The isosurface was generated for ∑i∥r−ri∥−2=3.8 A˚−2\sum_i \|\mathbf{r}-\mathbf{r}_i\|^{-2}=3.8\,\mathrm{\AA}^{-2}∑i​∥r−ri​∥−2=3.8A˚−2 (the index iii is used to sum over all atoms of the corresponding molecule). The molecules shown are (in order from top to bottom of the figure): benzene, toluene, salicylic acid and malondehyde. Atom colouring: carbon = black, hydrogen = white, oxygen = red.

DTNN Figure 2: chemical potentials for C, N, O, H atoms

图 2|A={C,N,O,H}A=\{\mathrm{C, N, O, H}\}A={C,N,O,H} 原子的化学势 ΩAM(r)\Omega_A^M(\mathbf{r})ΩAM​(r)。等值面按 ∑i∥r−ri∥−2=3.8 A˚−2\sum_i \|\mathbf{r}-\mathbf{r}_i\|^{-2}=3.8\,\mathrm{\AA}^{-2}∑i​∥r−ri​∥−2=3.8A˚−2 生成(指标 iii 用于对相应分子的全部原子求和)。所示分子(按图中从上到下的顺序)为:苯、甲苯、水杨酸和丙二醛(原文拼写为 malondehyde)。原子着色:碳为黑色,氢为白色,氧为红色。

Figure 3 | Classification of molecular carbon ring stability. Shown are 20 molecules (10 most stable and 10 least stable) with respect to the energy of the carbon ring predicted by the DTNN model. Atom colouring: carbon = black; hydrogen = white; oxygen = red; nitrogen = blue; fluorine = yellow.

DTNN Figure 3: classification of molecular carbon ring stability

图 3|分子碳环稳定性的分类。图中给出 20 个分子(最稳定的 10 个和最不稳定的 10 个),依据是 DTNN 模型预测的碳环能量。原子着色:碳为黑色;氢为白色;氧为红色;氮为蓝色;氟为黄色。

Because DTNN provides atomic energies by construction, it allows us to classify molecules by the stability of different building blocks, for example aromatic rings or methyl groups. An example of such classification is shown in Fig. 3, where we plot the molecules with most stable and least stable carbon aromatic rings in GDB-9. The distribution of atomic energies is shown in Supplementary Fig. 3, while Supplementary Fig. 4 lists the full stability ranking. The DTNN classification leads to interesting stability trends, notwithstanding the intrinsic non-uniqueness of atomic energy partitioning. However, unlike atomic projections employed in electronic-structure calculations, the DTNN approach has a firm foundation in statistical learning theory. In quantum-chemical calculations, every molecule would correspond to a different partitioning depending on its self-consistent electron density. In contrast, the DTNN approach learns the partitioning on a large molecular dataset, generating a transferable and global ‘dressed atom’ representation of molecules in chemical space. Recalling that DTNN exhibits errors below 1 kcal mol^{−1}, the classification shown in Fig. 3 can provide useful guidance for the chemical discovery of molecules with desired properties. Analytical gradients of the DTNN model with respect to chemical composition or ci(t)\mathbf{c}_i^{(t)}ci(t)​ could also aid in the exploration of chemical compound space^{37}.

由于 DTNN 在构造上就给出原子能量,它使我们能够按不同结构单元的稳定性对分子分类,例如芳环或甲基。图 3 给出了这类分类的一个例子,其中画出了 GDB-9 中碳芳环最稳定和最不稳定的分子。原子能量的分布见 Supplementary Fig. 3,完整的稳定性排序见 Supplementary Fig. 4。尽管原子能量划分本质上并不唯一,DTNN 的分类仍然给出了有意义的稳定性趋势。不过,与电子结构计算中采用的原子投影不同,DTNN 方法在统计学习理论中有坚实基础。在量子化学计算中,每个分子都会因其自洽电子密度而对应不同的划分。相反,DTNN 方法在大型分子数据集上学习这一划分,从而在化学空间中生成一种可迁移的、全局的“dress 原子”表示。考虑到 DTNN 的误差低于 1 kcal mol^{−1},图 3 所示的分类可以为寻找具有所需性质的分子提供有用的引导。DTNN 模型对化学组成或 ci(t)\mathbf{c}_i^{(t)}ci(t)​ 的解析梯度,也可能有助于探索化合物空间^{37}。

Energy predictions for isomers

The quantitative accuracy achieved by DTNN and its size extensivity paves the way to the calculation of configurational and conformational energy differences—a long-standing challenge for machine learning approaches^{7,12,13,38}. The reliability of DTNN for isomer energy predictions is demonstrated by the energy distribution in Fig. 4 for molecular isomers with C7O2H10\mathrm{C_7O_2H_{10}}C7​O2​H10​ chemical formula (a total of 6,095 isomers in the GDB-9 data set).

DTNN 所达到的定量精度及其规模广延性,为计算构型和构象能量差铺平了道路——这对机器学习方法而言是一个长期存在的挑战^{7,12,13,38}。图 4 中化学式为 C7O2H10\mathrm{C_7O_2H_{10}}C7​O2​H10​ 的分子同分异构体(GDB-9 数据集中共 6,095 个)的能量分布,表明了 DTNN 在同分异构体能量预测上的可靠性。

Figure 4 | Isomer energies with chemical formula C7O2H10\mathrm{C_7O_2H_{10}}C7​O2​H10​. DTNN trained on the GDB-9 database is able to acurately discriminate between 6,095 different isomers of C7O2H10\mathrm{C_7O_2H_{10}}C7​O2​H10​, which exhibit a non-trivial spectrum of relative energies.

DTNN Figure 4: isomer energies of C7O2H10

图 4|化学式为 C7O2H10\mathrm{C_7O_2H_{10}}C7​O2​H10​ 的同分异构体能量。在 GDB-9 数据库上训练的 DTNN 能够准确区分 6,095 种不同的 C7O2H10\mathrm{C_7O_2H_{10}}C7​O2​H10​ 同分异构体,它们呈现出并非平凡的相对能量谱。

Training a common model for chemical as well as conformational freedoms requires a more complex model. Furthermore, it comes with technical challenges like sampling and multiscale issues since the MD trajectories form clusters of small variation within the chemical compound space. As a proof of principle, we trained the DTNN to predict various MD trajectories of the C7O2H10\mathrm{C_7O_2H_{10}}C7​O2​H10​ isomers. To this end, we calculated short MD trajectories of 5,000 steps each for 113 randomly picked isomers as well as consistent total energies for all equilbrium structures. The training set is composed of all isomers in equilibrium as well as 50% of each MD trajectory. The remaining MD calculations are used for validation and testing. Despite the added complexity, our model achieves a mean absolute error of 1.7 kcal mol^{−1}.

若要为化学自由度和构象自由度训练一个共同模型,就需要更复杂的模型。此外还会带来采样和多尺度这类技术挑战,因为 MD 轨迹会在化合物空间中形成变差很小的团簇。作为原理验证,我们训练 DTNN 来预测 C7O2H10\mathrm{C_7O_2H_{10}}C7​O2​H10​ 同分异构体的多条 MD 轨迹。为此,我们为随机选取的 113 个同分异构体各计算了 5,000 步的短 MD 轨迹,并为所有平衡结构计算了一致的总能量。训练集由全部平衡同分异构体以及每条 MD 轨迹的 50% 组成。其余 MD 计算用于验证和测试。尽管复杂度有所增加,我们的模型仍达到 1.7 kcal mol^{−1} 的平均绝对误差。

Discussion

DTNNs provide an efficient way to represent chemical environments allowing for chemically accurate predictions. To this end, an implicit, atom-centered basis is learned from reference calculations. Employing this representation, atoms can be embedded in their chemical environment within a few refinement steps. Furthermore, DTNNs have the advantage that the embedding is built recursively from pairwise distances. Therefore, all necessary invariances (translation, rotation, permutation) are guaranteed to be exploited by the model. In addition, the learned embedding can be used to generate alchemical reaction paths (Supplementary Fig. 5).

DTNN 提供了一种表示化学环境的高效方式,从而能够做出达到化学精度的预测。为此,模型从参考计算中学习一种隐式的、以原子为中心的基。利用这一表示,原子可以在少数几步 refinement 中被嵌入到它们的化学环境里。此外,DTNN 的一个优点是,这种嵌入是由成对距离递归构造的。因此,模型保证会利用全部必要的不变性(平移、旋转、置换)。此外,学到的嵌入还可用来生成炼金术式的反应路径(Supplementary Fig. 5)。

In previous approaches, potential-energy surfaces were constructed by fitting many-body expansions with neural networks^{39,40,41}. However, these methods require a separate NN for each non-equivalent many-body term in the expansion. Since DTNN learns a common basis in which the atom interact, higher-order interactions can obtained more efficiently without separate treament.

在以往的方法中,势能面是通过用神经网络拟合多体展开来构造的^{39,40,41}。不过,这些方法需要为展开中每一个不等价的多体项分别设置一个神经网络。由于 DTNN 学到的是原子在其中发生相互作用的公共基,更高阶的相互作用可以更高效地得到,而不必分别处理。

Approaches like smooth overlap of atomic positions^{16,17} or manually crafted atom-centered symmetry functions^{18,19,42} are, like DTNN, based on representing chemical environments. All these approaches have in common that size-extensivity regarding the number of atoms is achieved by predicting atomic energy contributions using a non-linear regression method (for example, neural networks or kernel ridge regression). However, the previous approaches have a fixed set of basis functions describing the atomic environments. In contrast, DTNNs are able to adapt to the problem at hand in a data-driven fashion. Beyond the obvious advantage of not having to manually select symmetry functions and carefully tune hyper-parameters of the representation, this property of the DTNN makes it possible to gain quantum-chemical insights by analysing the learned representation.

诸如原子位置平滑重叠^{16,17}或手工构造的原子中心对称函数^{18,19,42}这类方法,与 DTNN 一样,都建立在表示化学环境之上。这些方法的共同之处在于,对原子数的规模广延性是通过用非线性回归方法(例如神经网络或核岭回归)预测原子能量贡献来实现的。不过,以往方法用一组固定的基函数来描述原子环境。相比之下,DTNN 能够以数据驱动的方式适应手头的问题。除了不必手工选择对称函数、也不必仔细调节表示的超参数这一明显优点之外,DTNN 的这一性质还使通过分析学到的表示来获得量子化学认识成为可能。

Obviously, more work is required to extend this predictive power for larger molecules, where the DTNN model will have to be combined with a reliable model for long-range interatomic (van der Waals) interactions. The intrinsic interpolation smoothness achieved by the DTNN model can also be used to identify molecules with peculiar electronic structure. Supplementary Fig. 6 shows a list of molecules with the largest DTNN errors compared with reference DFT calculations. It is noteworthy that most molecules in this figure are characterized by unconventional bonding and the electronic structure of these molecules has potential multi-reference character. The large prediction errors could stem from these molecules being not sufficiently represented by the training data. On the other hand, DTNN predictions might turn out to be closer to the correct answer because of its smooth interpolation in chemical space. Higher-level quantum-chemical calculations would be required to investigate this interesting hypothesis in the future.

显然,要把这种预测能力扩展到更大的分子,还需要更多工作;那时 DTNN 模型必须与可靠的长程原子间(范德华)相互作用模型结合起来。DTNN 模型所具有的内在插值平滑性,也可用来识别具有特殊电子结构的分子。Supplementary Fig. 6 列出了与参考 DFT 计算相比 DTNN 误差最大的分子。值得注意的是,该图中的大多数分子都具有非常规成键,其电子结构可能具有多参考态特征。较大的预测误差可能源于这些分子在训练数据中没有得到充分表示。另一方面,由于 DTNN 在化学空间中的平滑插值,其预测最终也可能更接近正确答案。要考察这一有意义的假设,未来还需要更高水平的量子化学计算。

We have proposed and developed a deep tensor neural network that enables understanding of quantum-chemical many-body systems beyond properties contained in the training dataset. The DTNN model is scalable with molecular size, efficient, and achieves uniform accuracy of 1 kcal mol^{−1} throughout compositional and configuration space for molecules of intermediate size. The DTNN model leads to novel insights into chemical systems, a fact that we illustrated on the example of relative aromatic ring stability, local molecular chemical potentials, relative isomer energies and the identification of molecules with peculiar electronic structure.

我们提出并发展了一种深度张量神经网络,使人们能够理解量子化学多体体系,而且所得到的认识超出了训练数据中包含的性质。DTNN 模型可随分子大小扩展,效率高,并在中等大小分子的组成空间和构型空间中达到 1 kcal mol^{−1} 的一致精度。DTNN 模型能够带来对化学体系的新认识,我们以相对芳环稳定性、分子局域化学势、相对同分异构体能量,以及识别具有特殊电子结构的分子为例作了说明。

Many avenues remain for improving the DTNN model on multiple fronts. Among these we mention the extension of the model to increasingly larger molecules, predicting atomic forces and frequencies, and non-extensive electronic and optical properties. We propose the DTNN model as a versatile framework for understanding complex quantum-mechanical systems based on high-throughput electronic structure calculations.

在多个方面,改进 DTNN 模型仍有许多途径。这里我们提到:把模型扩展到越来越大的分子,预测原子力和频率,以及非广延的电子与光学性质。我们把 DTNN 模型作为一个通用框架提出,用于在高通量电子结构计算的基础上理解复杂的量子力学体系。

Methods

Reference data sets. We employ two subsets of the GDB database^{34}, referred to in this paper as GDB-7 and GDB-9. GDB-7 contains 7,211 molecules with up to seven heavy atoms out of the elements C, N, O, S and Cl, saturated with hydrogen^{12}. Similarly, GDB-9 includes 133,885 molecules with up to 9 heavy atoms out of C, O, N, F (ref. 36). Both data sets include calculations of atomization energies employing density-functional theory^{43} with the PBE0 (ref. 44) and B3LYP (ref. 45–49) exchange-correlation potential, respectively.

参考数据集。我们使用 GDB 数据库^{34} 的两个子集,本文分别称为 GDB-7 和 GDB-9。GDB-7 包含 7,211 个分子,最多含有 7 个来自 C、N、O、S 和 Cl 的重原子,并由氢饱和^{12}。类似地,GDB-9 包含 133,885 个分子,最多含有 9 个来自 C、O、N、F 的重原子(文献 36)。两个数据集都包含用密度泛函理论^{43} 计算的原子化能量,交换关联势分别采用 PBE0(文献 44)和 B3LYP(文献 45–49)。

The molecular dynamics trajectories are calculated at a temperature of 500 K and resolution of 0.5 fs using density-functional theory with the PBE exchange-correlation potential^{50}. The data sets for benzene, toluene, malonaldehyde and salicylic acid consist of 627, 442, 993 and 320 k time steps, respectively. In the presented experiments, we predict the potential energy of the MD geometries.

分子动力学轨迹在温度为 500 K、时间分辨为 0.5 fs 的条件下,用采用 PBE 交换关联势^{50} 的密度泛函理论计算。苯、甲苯、丙二醛和水杨酸的数据集分别包含 627k、442k、993k 和 320k 个时间步。在本文实验中,我们预测这些 MD 几何构型的势能。

The molecular energies of the various data sets are predicted using a deep tensor neural network. The core idea is to represent atoms in the molecule as vectors depending on their type and to subsequently refine the representation by embedding the atoms in their neighbourhood. This is done in a sequence of interaction passes, where the atom representations influence each other in a pair-wise fashion. While each of these refinements depends only on the pair-wise atomic distances, multiple passes enable the architecture to also take angular information into account. Because of this decomposition of atomic interactions, an efficient representation of embedded atoms is learned following quantum-chemical principles.

各数据集上的分子能量用深度张量神经网络来预测。核心想法是:把分子中的原子按其类型表示为向量,随后通过把原子嵌入到它们的邻域中来 refinement 这一表示。这是在一连串相互作用传递中完成的,其中原子表示以成对的方式相互影响。尽管每一步 refinement 只依赖于成对的核间距,多次传递使这一结构也能够考虑角度信息。由于对原子相互作用做了这种分解,模型按照量子化学原理学到一种嵌入原子的高效表示。

In the following, we describe the deep tensor neural network step-by-step, including hyper-parameters used in our experiments.

下面我们逐步描述深度张量神经网络,并给出实验中使用的超参数。

1. Assign initial atomic descriptors

We assign an initial coefficient vector to each atom iii of the molecule according to its nuclear charge ZiZ_iZi​:

我们根据核电荷 ZiZ_iZi​,为分子中的每个原子 iii 赋予一个初始系数向量:

ci(0)=cZi∈RB,(4)\mathbf{c}_i^{(0)} = \mathbf{c}_{Z_i} \in \mathbb{R}^B, \quad (4)ci(0)​=cZi​​∈RB,(4)

where BBB is the number of basis functions. All presented models use atomic descriptors with 30 coefficients. We initialize each coefficient randomly following cz∼N(0,1/B)c_z \sim \mathcal{N}(0, 1/\sqrt{B})cz​∼N(0,1/B​).

其中 BBB 是基函数的数目。本文给出的全部模型都使用含 30 个系数的原子描述符。我们按 cz∼N(0,1/B)c_z \sim \mathcal{N}(0, 1/\sqrt{B})cz​∼N(0,1/B​) 随机初始化每一个系数。

2. Gaussian feature expansion of the inter-atomic distances

The inter-atomic distances DijD_{ij}Dij​ are spread across many dimensions by a uniform grid of Gaussians

核间距 DijD_{ij}Dij​ 通过一组均匀网格上的高斯函数被展开到许多维上

d^ijk=exp⁡(−(Dij−(μmin⁡+kΔμ))22σ2),0≤k≤μmax⁡/Δμ,(5)\hat{d}_{ij}^k = \exp\left( -\frac{\left(D_{ij} - (\mu_{\min} + k\Delta\mu)\right)^2}{2\sigma^2} \right), \qquad 0 \le k \le \mu_{\max}/\Delta\mu, \quad (5)d^ijk​=exp(−2σ2(Dij​−(μmin​+kΔμ))2​),0≤k≤μmax​/Δμ,(5)

with Δμ\Delta\muΔμ being the gap between two Gaussians of width σ\sigmaσ.

其中 Δμ\Delta\muΔμ 是宽度为 σ\sigmaσ 的相邻两个高斯之间的间隔。

In our experiments, we set both to 0.2 Å. The centre of the first Gaussian μmin⁡\mu_{\min}μmin​ was set to −1, while μmax⁡\mu_{\max}μmax​ was chosen depending on the range of distances in the data (10 Å for GDB-7 and benzene, 15 Å for toluene, malonaldehyde and salicylic acid and 20 Å for GDB-9).

在我们的实验中,这两项都被设为 0.2 Å。第一个高斯的中心 μmin⁡\mu_{\min}μmin​ 设为 −1,而 μmax⁡\mu_{\max}μmax​ 则根据数据中的距离范围选取(GDB-7 和苯为 10 Å,甲苯、丙二醛和水杨酸为 15 Å,GDB-9 为 20 Å)。

3. Perform TTT interaction passes

Each coefficient vector ci(t)\mathbf{c}_i^{(t)}ci(t)​, corresponding to atom iii after ttt passes, is corrected by the interactions with the other atoms of the molecule:

每个系数向量 ci(t)\mathbf{c}_i^{(t)}ci(t)​ 对应于经过 ttt 次传递后的原子 iii,它由与分子中其他原子的相互作用来修正:

ci(t+1)=ci(t)+∑j≠ivij.(6)\mathbf{c}_i^{(t+1)} = \mathbf{c}_i^{(t)} + \sum_{j \neq i} \mathbf{v}_{ij}. \quad (6)ci(t+1)​=ci(t)​+j=i∑​vij​.(6)

Here, we model the interaction v\mathbf{v}v as follows:

在这里,我们把相互作用 v\mathbf{v}v 建模为:

vij=tanh⁡[Wfc((Wcfcj+bf1)∘(Wdfd^ij+bf2))],(7)\mathbf{v}_{ij} = \tanh\left[ W^{fc} \left( (W^{cf}\mathbf{c}_j + \mathbf{b}^{f1}) \circ (W^{df}\hat{\mathbf{d}}_{ij} + \mathbf{b}^{f2}) \right) \right], \quad (7)vij​=tanh[Wfc((Wcfcj​+bf1)∘(Wdfd^ij​+bf2))],(7)

where the circle (∘\circ∘) represents the element-wise matrix product. The factor representation in the presented models employs 60 neurons.

其中圆圈(∘\circ∘)表示逐元素矩阵乘积。本文模型中的因子表示使用 60 个神经元。

4. Predict energy contributions

Finally, we predict the energy contributions EiE_iEi​ from each atom iii. Employing two fully-connected layers, for each atom a scaled energy contribution E^i\hat{E}_iE^i​ is predicted:

最后,我们从每个原子 iii 预测能量贡献 EiE_iEi​。使用两个全连接层,为每个原子预测一个经过缩放的能量贡献 E^i\hat{E}_iE^i​:

oi=tanh⁡(Wout1ci(T)+bout1),(8)\mathbf{o}_i = \tanh\left( W^{\mathrm{out}1} \mathbf{c}_i^{(T)} + \mathbf{b}^{\mathrm{out}1} \right), \quad (8)oi​=tanh(Wout1ci(T)​+bout1),(8)

E^i=Wout2oi+bout2.(9)\hat{E}_i = W^{\mathrm{out}2} \mathbf{o}_i + b^{\mathrm{out}2}. \quad (9)E^i​=Wout2oi​+bout2.(9)

In our experiments, the hidden layer oi\mathbf{o}_ioi​ possesses 15 neurons. To obtain the final contributions, E^i\hat{E}_iE^i​ is shifted to the mean EμE_\muEμ​ and scaled by the s.d. EσE_\sigmaEσ​ of the energy per atom estimated on the training set.

在我们的实验中,隐层 oi\mathbf{o}_ioi​ 有 15 个神经元。为得到最终贡献,把 E^i\hat{E}_iE^i​ 平移到训练集上估计的每原子能量均值 EμE_\muEμ​,并按标准差 EσE_\sigmaEσ​ 缩放。

Ei=EσE^i+Eμ.(10)E_i = E_\sigma \hat{E}_i + E_\mu. \quad (10)Ei​=Eσ​E^i​+Eμ​.(10)

This procedure ensures a good starting point for the training.

这一做法为训练提供了一个良好的起点。

5. Obtain the molecular energy E=∑iEiE = \sum_i E_iE=∑i​Ei​

The bias parameters as well as Wout2W^{\mathrm{out}2}Wout2 are initially set to zero. All other weight matrices are initialized drawing from a uniform distribution according to (ref. 51). Neural network code is available.

偏置参数以及 Wout2W^{\mathrm{out}2}Wout2 最初被设为零。所有其他权重矩阵按照文献 51 从均匀分布中抽取初始化。神经网络代码已经公开。

The deep tensor neural networks have been trained for 3,000 epochs minimizing the squared error, using stochastic gradient descent with 0.9 momentum and a constant learning rate^{52}. The final results are taken from the models with the best validation error in early stopping.

深度张量神经网络训练了 3,000 个 epoch,最小化平方误差,使用动量为 0.9、学习率为常数的随机梯度下降^{52}。最终结果取自早停时验证误差最好的模型。

All DTNN models were trained and executed on an NVIDIA Tesla K40 GPU. The computational cost of the employed models depends on the number of reference calculations, the number of interaction passes as well as the number of atoms per molecule. The training times for all models and data sets are shown in Supplementary Table 2, ranging from 6 h for 5.768 reference calculations of GDB-7 with one interaction pass, to 162 h for 100,000 reference calculations of the GDB-9 data set with three interaction passes.

所有 DTNN 模型都在一块 NVIDIA Tesla K40 GPU 上训练和运行。所用模型的计算代价取决于参考计算的数量、相互作用传递的次数以及每个分子的原子数。全部模型和数据集的训练时间见 Supplementary Table 2,短则对 5.768 个 GDB-7 参考计算、一次相互作用传递用 6 小时,长则对 100,000 个 GDB-9 参考计算、三次相互作用传递用 162 小时。

On the other hand, the prediction is instantaneous: all models predict examples from the employed data sets in <1 ms. Supplementary Fig. 7 shows the scaling of the prediction time with the number of atoms and interaction layers. Even for a molecule with 100 atoms, a DTNN with three interaction layers requires <5 ms for a prediction.

另一方面,预测是瞬时完成的:所有模型对所用数据集中的样本都能在 <1 ms 内给出预测。Supplementary Fig. 7 给出了预测时间随原子数和相互作用层数的变化。即使对含 100 个原子的分子,一个有三层相互作用的 DTNN 做一次预测也只需 <5 ms。

The prediction as well as the training steps scale linearly with the number of interaction passes and quadratically with the number of atoms, since the pairwise atomic distances are required for the interactions. For large molecules it is reasonable to introduce a distance cutoff. In that case, the DTNN will also scale linearly with the number of atoms.

预测和训练步骤都随相互作用传递次数线性增长,并随原子数二次增长,因为相互作用需要成对的核间距。对大分子来说,引入距离截断是合理的。在那种情形下,DTNN 也会随原子数线性扩展。

Given a trained neural network as described in the previous section, one can extract the coefficients vectors ci(t)\mathbf{c}_i^{(t)}ci(t)​ for each atom iii and each interaction pass ttt for a molecule of interest. From each final representation ci(T)\mathbf{c}_i^{(T)}ci(T)​, the energy contribution EiE_iEi​ of the corresponding atom to the molecular energy can be obtained. Instead, we let the molecule act on a probe atom, described by its charge zzz and the pairwise distances d1,…,dnd_1,\ldots,d_nd1​,…,dn​ to the atoms of the molecule:

有了上一节所述的训练好的神经网络,就可以对感兴趣的分子提取每个原子 iii 和每次相互作用传递 ttt 的系数向量 ci(t)\mathbf{c}_i^{(t)}ci(t)​。从每个最终表示 ci(T)\mathbf{c}_i^{(T)}ci(T)​,可以得到相应原子对分子能量的贡献 EiE_iEi​。我们改为让分子作用于一个探针原子,该探针由其电荷 zzz 以及到分子中各原子的成对距离 d1,…,dnd_1,\ldots,d_nd1​,…,dn​ 来描述:

cprobe(t+1)=cprobe(t)+∑j=1nvj,(11)\mathbf{c}_{\mathrm{probe}}^{(t+1)} = \mathbf{c}_{\mathrm{probe}}^{(t)} + \sum_{j=1}^{n} \mathbf{v}_j, \quad (11)cprobe(t+1)​=cprobe(t)​+j=1∑n​vj​,(11)

with vj=tanh⁡(Wfc((Wcfcj+bf1)∘(Wdfd^j+bf2)))\mathbf{v}_j = \tanh\big(W^{fc}\big((W^{cf}\mathbf{c}_j + \mathbf{b}^{f1}) \circ (W^{df}\hat{\mathbf{d}}_j + \mathbf{b}^{f2})\big)\big)vj​=tanh(Wfc((Wcfcj​+bf1)∘(Wdfd^j​+bf2))). While this is equivalent to how the coefficient vectors of the molecule are corrected, here, the molecule does not get to be influenced by the probe. Now, the energy of the probe atom is predicted as usual from the final representation cprobe(T)\mathbf{c}_{\mathrm{probe}}^{(T)}cprobe(T)​.

其中 vj=tanh⁡(Wfc((Wcfcj+bf1)∘(Wdfd^j+bf2)))\mathbf{v}_j = \tanh\big(W^{fc}\big((W^{cf}\mathbf{c}_j + \mathbf{b}^{f1}) \circ (W^{df}\hat{\mathbf{d}}_j + \mathbf{b}^{f2})\big)\big)vj​=tanh(Wfc((Wcfcj​+bf1)∘(Wdfd^j​+bf2)))。这虽然等价于分子系数向量被修正的方式,但在这里分子不会受到探针的影响。随后,探针原子的能量仍按通常方式由最终表示 cprobe(T)\mathbf{c}_{\mathrm{probe}}^{(T)}cprobe(T)​ 预测。

Interpreting this as a local potential ΩAM(r)\Omega_A^M(\mathbf{r})ΩAM​(r) generated by the molecule, we can use the neural network to visualize the learned interactions as illustrated in Supplementary Fig. 8. The presented energy surfaces show the potential for different probe atoms plotted on an isosurface of ∑i=1ndi−2\sum_{i=1}^{n} d_i^{-2}∑i=1n​di−2​. We used Mayavi^{53} for the visualization of the surfaces.

若把它解释为分子产生的局域势 ΩAM(r)\Omega_A^M(\mathbf{r})ΩAM​(r),就可以用神经网络把学到的相互作用可视化,如 Supplementary Fig. 8 所示。所给出的能量曲面展示了不同探针原子的势,画在 ∑i=1ndi−2\sum_{i=1}^{n} d_i^{-2}∑i=1n​di−2​ 的等值面上。我们使用 Mayavi^{53} 来可视化这些曲面。

Data availability. The GDB-9 data set is available under the DOI 10.6084/m9.figshare.978904. All data sets used in this work are available at http://quantum-machine.org/datasets/.

数据可用性。GDB-9 数据集可通过 DOI 10.6084/m9.figshare.978904 获得。本工作使用的全部数据集可在 http://quantum-machine.org/datasets/ 获取。

Additional information

Supplementary Information accompanies this paper at http://www.nature.com/naturecommunications

Competing financial interests: The authors declare no competing financial interests.

Reprints and permission information is available online at http://npg.nature.com/reprintsandpermissions/

How to cite this article: Schütt, K. T. et al. Quantum-chemical insights from deep tensor neural networks. Nat. Commun. 8, 13890 doi: 10.1038/ncomms13890 (2017).

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

© The Author(s) 2017

Acknowledgements

We thank Huziel Sauceda for providing molecular dynamics trajectories for C7O2H10 isomers. K.T.S. and K.R.M. thank the Einstein Foundation for generously funding the ETERNAL project. Additional support was provided by the DFG (MU 987/20-1) and the Federal Ministry of Education and Research (BMBF) for the Berlin Big Data Center BBDC (01IS14013A). K.R.M. gratefully acknowledges the BK21 program funded by Korean National Research Foundation grant (No. 2012-005741). Part of this research was performed while the authors were visiting the Institute for Pure and Applied Mathematics (IPAM), which is supported by the National Science Foundation (NSF).

我们感谢 Huziel Sauceda 为 C7O2H10 同分异构体提供了分子动力学轨迹。K.T.S. 和 K.R.M. 感谢爱因斯坦基金会为 ETERNAL 项目提供的慷慨资助。德意志研究联合会(DFG,MU 987/20-1)以及联邦教育与研究部(BMBF)对柏林大数据中心 BBDC(01IS14013A)的支持也为本工作提供了额外资助。K.R.M. 感谢由韩国国家研究基金资助的 BK21 项目(编号 2012-005741)。部分研究是作者访问纯粹与应用数学研究所(IPAM)期间完成的,该研究所得到美国国家科学基金会(NSF)的支持。

Author contributions

K.T.S. conceived the DTNN, performed analyses and prepared the figures, K.T.S., F.A., K.R.M. and A.T. developed the theory, K.R.M. and A.T. designed the analyses, S.S. helped with the MD predictions, K.T.S., K.R.M. and A.T. wrote the paper. All authors discussed results and commented on the manuscript.

References

[1] Kang, B. & Ceder, G. Battery materials for ultrafast charging and discharging. Nature 458, 190–193 (2009).

[2] Nørskov, J. K., Bligaard, T., Rossmeisl, J. & Christensen, C. H. Towards the computational design of solid catalysts. Nat. Chem. 1, 37–46 (2009).

[3] Hachmann, J. et al. The Harvard clean energy project: large-scale computational screening and design of organic photo-voltaics on the world community grid. J. Phys. Chem. Lett. 2, 2241–2251 (2011).

[4] Pyzer-Knapp, E. O., Suh, C., Gomez-Bombarelli, R., Aguilera-Iparraguirre, J. & Aspuru-Guzik, A. What is high-throughput virtual screening? A perspective from organic materials discovery. Annu. Rev. Mater. Res. 45, 195–216 (2015).

[5] Curtarolo, S. et al. The high-throughput highway to computational materials design. Nat. Mater. 12, 191–201 (2013).

[6] Snyder, J. C., Rupp, M., Hansen, K., Müller, K.-R. & Burke, K. Finding density functionals with machine learning. Phys. Rev. Lett. 108, 253002 (2012).

[7] Rupp, M., Tkatchenko, A., Müller, K.-R. & Von Lilienfeld, O. A. Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett. 108, 058301 (2012).

[8] Ramakrishnan, R., Dral, P. O., Rupp, M. & von Lilienfeld, O. A. Big data meets quantum chemistry approximations: the Δ-machine learning approach. J. Chem. Theory Comput. 11, 2087–2096 (2015).

[9] Bishop, C. M. Pattern Recognition and Machine Learning Springer (2006).

[10] Ghiringhelli, L. M., Vybiral, J., Levchenko, S. V., Draxl, C. & Scheffler, M. Big data of materials science: critical role of the descriptor. Phys. Rev. Lett. 114, 105503 (2015).

[11] Schütt, K. et al. How to represent crystal structures for machine learning: towards fast prediction of electronic properties. Phys. Rev. B 89, 205118 (2014).

[12] Montavon, G. et al. Machine learning of molecular electronic properties in chemical compound space. New J. Phys. 15, 095003 (2013).

[13] Hansen, K. et al. Assessment and validation of machine learning methods for predicting molecular atomization energies. J. Chem. Theory Comput. 9, 3404–3419 (2013).

[14] Hirn, M., Poilvert, N. & Mallat, S. Quantum energy regression using scattering transforms. Preprint at https://arxiv.org/abs/1502.02077 (2015).

[15] Hansen, K. et al. Machine learning predictions of molecular properties: accurate many-body potentials and nonlocality in chemical space. J. Phys. Chem. Lett. 6, 2326 (2015).

[16] Bartók, A. P., Kondor, R. & Csanyi, G. On representing chemical environments. Phys. Rev. B 87, 184115 (2013).

[17] Bartók, A. P., Payne, M. C., Kondor, R. & Csanyi, G. Gaussian approximation potentials: the accuracy of quantum mechanics, without the electrons. Phys. Rev. Lett. 104, 136403 (2010).

[18] Behler, J. Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys. 134, 074106 (2011).

[19] Behler, J. Neural network potential-energy surfaces in chemistry: a tool for large-scale simulations. Phys. Chem. Chem. Phys. 13, 17930–17955 (2011).

[20] LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

[21] Montavon, G., Braun, M. L. & Müller, K.-R. Kernel analysis of deep networks. J. Mach. Learn. Res. 12, 2563–2581 (2011). MathSciNet MATH

[22] Ciresan, D., Meier, U. & Schmidhuber, J. Multi-column deep neural networks for image classification. In Proc. Conference on Computer Vision and Pattern Recognition. 3642–3649 (2012).

[23] Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In Proc. Advances in Neural Information Processing Systems. 25, 1097–1105 (2012).

[24] LeCun, Y. & Bengio, Y. in The Handbook of Brain Theory and Neural Networks (ed. Arbib M.A.) 255–257 (The MIT Press, Cambridge, MA, USA, 1995).

[25] Hinton, G. et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29, 82–97 (2012).

[26] Sainath, T. N. et al. Deep convolutional neural networks for large-scale speech tasks. Neural Netw. 64, 39–48 (2015).

[27] Collobert, R. & Weston, J. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proc. 25th International Conference on Machine Learning. 160–167 (2008).

[28] Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M. & Mon-fardini, G. The graph neural network model. IEEE Trans. Neural Netw. 20, 61–80 (2009).

[29] Duvenaud, D. K. et al. Convolutional networks on graphs for learning molecular fingerprints. In Proc. Advances in Neural Information Processing Systems. 28, 2224–2232 (2015).

[30] Socher, R. et al. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of the conference on empirical methods in natural language processing (EMNLP) 1631–1642 (2013).

[31] Sutskever, I., Martens, J. & Hinton, G. E. Generating text with recurrent neural networks. Proc. 28th Annu. Int. Conf. Mach. Learn. 1017–1024 (2011).

[32] Socher, R., Chen, D., Manning, C. D. & Ng, A. Reasoning with neural tensor networks for knowledge base completion. In Proc. Advances in Neural Information Processing Systems. 26, 926–934 (2013).

[33] Taylor, G. W. & Hinton, G. E. Factored conditional restricted Boltzmann machines for modeling motion style. In Proc. 26th Annual International Conference on Machine Learning. 1025–1032 (2009).

[34] Blum, L. C. & Reymond, J.-L. 970 Million Druglike Small Molecules for Virtual Screening in the Chemical Universe Database GDB-13. J. Am. Chem. Soc. 131, 8732 (2009).

[35] Reymond, J.-L. The chemical space project. Acc. Chem. Res. 48, 722–730 (2015).

[36] Ramakrishnan, R., Dral, P. O., Rupp, M. & von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data 1, 140022 (2014).

[37] von Lilienfeld, O. A. First principles view on chemical compound space: gaining rigorous atomistic control of molecular properties. Int. J. Quantum Chem. 113, 1676–1689 (2013).

[38] De, S., Bartok, A. P., Csanyi, G. & Ceriotti, M. Comparing molecules and solids across structural and alchemical space. Phys. Chem. Chem. Phys. 18, 13754–13769 (2016).

[39] Malshe, M. et al. Development of generalized potential-energy surfaces using many-body expansions, neural networks, and moiety energy approximations. J. Chem. Phys. 130, 184102 (2009).

[40] Manzhos, S. & Carrington, T. Jr A random-sampling high dimensional model representation neural network for building potential energy surfaces. J. Chem. Phys. 125, 084109 (2006).

[41] Manzhos, S. & Carrington, T. Jr Using neural networks, optimized coordinates, and high-dimensional model representations to obtain a vinyl bromide potential surface. J. Chem. Phys. 129, 224104 (2008).

[42] Behler, J. & Parrinello, M. Generalized neural-network representation of high-dimensional potential-energy surfaces. Phys. Rev. Lett. 98, 146401 (2007).

[43] Hohenberg, P. & Kohn, W. Inhomogeneous electron gas. Phys. Rev. 136, B864–B871 (1964). MathSciNet

[44] Perdew, J. P., Ernzerhof, M. & Burke, K. Rationale for mixing exact exchange with density functional approximations. J. Chem. Phys. 105, 9982–9985 (1996).

[45] Becke, A. D. Density-functional exchange-energy approximation with correct asymptotic behavior. Phys. Rev. A 38, 3098–3100 (1988).

[46] Lee, C., Yang, W. & Parr, R. G. Development of the Colle- Salvetti correlation-energy formula into a functional ofthe electron density. Phys. Rev. B 37, 785–789 (1988).

[47] Vosko, S. H., Wilk, L. & Nusair, M. Accurate spin-dependent electron liquid correlation energies for local spin density calculations: a critical analysis. Can. J. Phys. 58, 1200–1211 (1980).

[48] Stephens, P., Devlin, F., Chabalowski, C. & Frisch, M. J. Ab initio calculation of vibrational absorption and circular dichro-ism spectra using density functional force fields. J. Phys. Chem. 98, 11623–11627 (1994).

[49] Becke, A. d. Beckes 3 parameter functional combined with the non-local correlation LYP. J. Chem. Phys. 98, 5648 (1993).

[50] Perdew, J. P., Burke, K. & Ernzerhof, M. Generalized gradient approximation made simple. Phys. Rev. Lett. 77, 3865–3868 (1996).

[51] Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. 13th International Conference on Artificial Intelligence and Statistics. 249–256 (2010).

[52] LeCun, Y. A., Bottou, L., Orr, G. B. & Müller, K.-R. in Neural Networks: Tricks of the Trade 9–48 (Springer, 2012).

[53] Ramachandran, P. & Varoquaux, G. Mayavi: 3D visualization of scientific data. Comput. Sci. Eng. 13, 40–51 (2011).