SchNet: A continuous-filter convolutional neural network for modeling quantum interactions

SchNet：用于建模量子相互作用的连续滤波卷积神经网络（SchNet: A continuous-filter convolutional neural network for modeling quantum interactions）

K. T. Schütt 1 1 footnotemark: 1 Affiliation:  Machine Learning Group, Technische Universität Berlin, Germany    P.-J. Kindermans Affiliation:  Machine Learning Group, Technische Universität Berlin, Germany    H. E. Sauceda Affiliation:  Theory Department, Fritz-Haber-Institut der Max-Planck-Gesellschaft, Berlin, Germany    S. Chmiela Affiliation:  Machine Learning Group, Technische Universität Berlin, Germany    A. Tkatchenko Affiliation:  Physics and Materials Science Research Unit, University of Luxembourg, Luxembourg    K.-R. Müller 2 2 footnotemark: 2 Affiliation:  Machine Learning Group, Technische Universität Berlin, Germany Affiliation:  Max-Planck-Institut für Informatik, Saarbrücken, Germany Affiliation:  Dept. of Brain and Cognitive Engineering, Korea University, Seoul, South Korea    * kristof.schuett@tu-berlin.de  klaus-robert.mueller@tu-berlin.de

Abstract

摘要

Deep learning has the potential to revolutionize quantum chemistry as it is ideally suited to learn representations for structured data and speed up the exploration of chemical space. While convolutional neural networks have proven to be the first choice for images, audio and video data, the atoms in molecules are not restricted to a grid. Instead, their precise locations contain essential physical information, that would get lost if discretized. Thus, we propose to use continuous-filter convolutional layers to be able to model local correlations without requiring the data to lie on a grid. We apply those layers in SchNet: a novel deep learning architecture modeling quantum interactions in molecules. We obtain a joint model for the total energy and interatomic forces that follows fundamental quantum-chemical principles. Our architecture achieves state-of-the-art performance for benchmarks of equilibrium molecules and molecular dynamics trajectories. Finally, we introduce a more challenging benchmark with chemical and structural variations that suggests the path for further work.

深度学习有潜力变革量子化学，因为它很适合学习原子体系的表征，并在计算上要求极高的量子化学计算之外进行泛化。我们提出 SchNet：一种专门设计用于预测小有机分子到大型化合物等各种分子体系性质的深度架构。关键要素是：在整个模型中始终遵循原子尺度输入表示所需的基本量子化学原则，以及一种在学习局部相互作用时不依赖离散网格的新型连续滤波卷积层。我们进一步给出 SchNet 的解释性分析，展示它能够学习符合化学直觉的有意义表征。最后，我们在分子体系中化学与构象自由度相关的多种基准上展示了 SchNet 的池化能力，并获得了当时领先的性能。

1 Introduction

1 引言

The discovery of novel molecules and materials with desired properties is crucial for applications such as batteries, catalysis and drug design. However, the vastness of chemical compound space and the computational cost of accurate quantum-chemical calculations prevent an exhaustive exploration. In recent years, there have been increased efforts to use machine learning for the accelerated discovery of molecules and materials with desired properties [1, 2, 3, 4, 5, 6, 7, 8, 9]. However, these methods are only applied to stable systems in so-called equilibrium, i.e., local minima of the potential energy surface $E(\mathbf{r}_{1},\dots,\mathbf{r}_{n})$ where $\mathbf{r}_{i}$ is the position of atom $i$. Data sets such as the established QM9 benchmark [10] contain only equilibrium molecules. Predicting stable atom arrangements is in itself an important challenge in quantum chemistry and material science.

发现具有理想特性的新分子和材料对于电池、催化物和药物设计等应用至关重要。 然而,化学化合物空间的广阔性以及精确量子化学计算的成本使得无法进行详尽无遗的探索. 近年来,人们更加努力地利用机器学习来加速发现具有理想特性的分子和材料[1,2,3,4,5,6,7,8,9]. 然而,这些方法只适用于所谓的“平衡”中的稳定系统,即潜在能量表面的局部微型$E(\mathbf{r}_{1},\dots,\mathbf{r}_{n})$,其中$\mathbf{r}_{i}$是原子$i$的位置。 既定的QM9基准[10]等数据集只包含平衡分子. 预测稳定的原子安排本身是量子化学和材料科学的一个重要挑战。

In general, it is not clear how to obtain equilibrium conformations without optimizing the atom positions. Therefore, we need to compute both the total energy $E(\mathbf{r}_{1},\dots,\mathbf{r}_{n})$ and the forces acting on the atoms

通常,不能达到最优的(分寸),而能达到最优的(分寸)者,是分明的。 因此,我们需要计算总能量$E(\mathbf{r}_{1},\dots,\mathbf{r}_{n})$和在原子上作用的力量。

$$\textbf{F}_{i}(\mathbf{r}_{1},\dots,\mathbf{r}_{n})=-\frac{\partial E}{\partial\mathbf{r}_{i}}(\mathbf{r}_{1},\dots,\mathbf{r}_{n}). \tag{1}$$

One possibility is to use a less computationally costly, however, also less accurate quantum-chemical approximation. Instead, we choose to extend the domain of our machine learning model to both compositional (chemical) and configurational (structural) degrees of freedom.

一种可能性是采用成本较低的计算方法,但也采用精确度较低的量子化学近似法。 相反,我们选择将机器学习模型的范畴扩展到自由的成分(化学)和配置(结构)程度。

In this work, we aim to learn a representation for molecules using equilibrium and non-equilibrium conformations. Such a general representation for atomistic systems should follow fundamental quantum-mechanical principles. Most importantly, the predicted force field has to be curl-free. Otherwise, it would be possible to follow a circular trajectory of atom positions such that the energy keeps increasing, i.e., breaking the law of energy conservation. Furthermore, the potential energy surface as well as its partial derivatives have to be smooth, e.g., in order to be able to perform geometry optimization. Beyond that, it is beneficial that the model incorporates the invariance of the molecular energy with respect to rotation, translation and atom indexing. Being able to model both chemical and conformational variations constitutes an important step towards ML-driven quantum-chemical exploration.

在这项工作中,我们的目标是学习分子使用均衡性和非均匀性配体的表示. 这种对原子主义体系的一般代表应该遵循基本的量子机制原则。 最重要的是,预测的力量场必须是无卷曲的. 否则,就有可能沿着原子位置的循环轨道走,使能量不断增加,即违反节能法则。 此外,潜在的能量表面及其部分衍生物必须光滑,例如,才能进行几何优化。 除此之外,该模型在旋转、翻译和原子指数化方面吸收分子能量的不惯性也是有益的。 能够模拟化学和成像变异,是朝向由ML驱动的量子化学勘探迈出的重要一步。

This work provides the following key contributions:

这项工作作出了以下主要贡献:

• • We propose continuous-filter convolutional (cfconv) layers as a means to move beyond grid-bound data such as images or audio towards modeling objects with arbitrary positions such as astronomical observations or atoms in molecules and materials.
• • We propose SchNet: a neural network specifically designed to respect essential quantum-chemical constraints. In particular, we use the proposed cfconv layers in $\mathbb{R}^{3}$ to model interactions of atoms at arbitrary positions in the molecule. SchNet delivers both rotationally invariant energy prediction and rotationally equivariant force predictions. We obtain a smooth potential energy surface and the resulting force-field is guaranteed to be energy-conserving.
• • We present a new, challenging benchmark -- ISO17 -- including both chemical and conformational changes[^3]. We show that training with forces improves generalization in this setting as well.

• 我们建议将“连续过滤”层(cfconv)层(cfconv)层(Cfconv)层(civil-filter convolutional)层(civil)层(civil-filter convolutional)层(Cfconv)层(Cfconv)层(Cfconv)层)作为一种手段,将图像或音频等网格数据转移到具有天文观测或分子和材料中的原子等任意位置的建模物体上。
• 我们提议SchNet:一个专门为尊重重要的量子化学限制而设计的神经网络。 特别是,我们利用$\mathbb{R}^{3}$中拟议的Cfconv层来模拟原子在分子中任意位置上的相互作用。 SchNet既提供旋转无变能量预测,又提供旋转等效力预测. 我们得到一个平滑的潜在能量表面,由此形成的力场保证能节能.
• 我们提出了一个具有挑战性的新的基准 -- -- ISO17 -- -- 包括化学和符合性的变化[^3]。 我们表明,在这种环境下,与部队进行训练也会加强普遍化。

2 Related work

2 相关工作

Previous work has used neural networks and Gaussian processes applied to hand-crafted features to fit potential energy surfaces [11, 12, 13, 14, 15, 16]. Graph convolutional networks for circular fingerprint [17] and molecular graph convolutions [18] learn representations for molecules of arbitrary size. They encode the molecular structure using neighborhood relationships as well as bond features, e.g., one-hot encodings of single, double and triple bonds. In the following, we briefly review the related work that will be used in our empirical evaluation: gradient domain machine learning (GDML), deep tensor neural networks (DTNN) and enn-s2s.

之前的作品使用了神经网络和应用于手动特性的高斯工艺,以适应潜在的能量表面[11,12,13,14,15,16]. 循环指纹的图卷积网络[17]和分子图演化[18]为任意大小的分子学习表达. 它们利用邻里关系以及债券特征来编码分子结构,例如单倍和三倍债券的单热编码. 在下文中,我们简要回顾了将在我们的经验性评估中使用的相关工作:梯度域机学习(GDML),深伸缩神经网络(DTNN)和enn-s2s.

Gradient-domain machine learning (GDML)

[19] proposed GDML as a method to construct force fields that explicitly obey the law of energy conservation. GDML captures the relationship between energy and interatomic forces (see Eq. 1) by training the gradient of the energy estimator. The functional relationship between atomic coordinates and interatomic forces is thus learned directly and energy predictions are obtained by re-integration. However, GDML does not scale well due to its kernel matrix growing quadratically with the number of atoms as well as the number of examples. Beyond that, it is not designed to represent different compositions of atom types unlike SchNet, DTNN and enn-s2s.

渐变域机器学习(GDML)

[19] 提出GDML作为建造明确遵守节能法的力场的方法. GDML通过训练能量估计器的梯度来捕捉能量与原子间力的关系(见Eq.1). 因此原子坐标和原子间力之间的功能关系是直接学习的,能量预测是通过再融合获得的. 然而,GDML由于内核矩阵随原子数量和实例数量而呈四倍增长,因此其缩放度不高. 除此之外,它的设计并不代表原子类型的不同成分,不同于SchNet,DTNN和enn-s2s.

Deep tensor neural networks (DTNN)

[20] proposed the DTNN for molecules that are inspired by the many-body Hamiltonian applied to the interactions of atoms. They have been shown to reach chemical accuracy on a small set of molecular dynamics trajectories as well as QM9. Even though the DTNN shares the invariances with our proposed architecture, its interaction layers lack the continuous-filter convolution interpretation. It falls behind in accuracy compared to SchNet and enn-s2s.

• 深度神经网络(DTNN)

[20]为受多体汉密尔顿尼用於原子相互作用的分子提出了DTNN. 已经证明,在一小组分子动力学轨迹以及QM9上它们都达到了化学精度. 尽管DTNN与我们提议的架构有相同的偏差,但其袍子的相互作用层缺乏连续的过滤器演化解释. 它在准确度上落后于SchNet和enn-s2s.

enn-s2s

[21] proposed the enn-s2s as a variant of message-passing neural networks that uses bond type features in addition to interatomic distances. It achieves state-of-the-art performance on all properties of the QM9 benchmark [21]. Unfortunately, it cannot be used for molecular dynamics predictions (MD-17). This is caused by discontinuities in their potential energy surface due to the discreteness of the one-hot encodings in their input. In contrast, SchNet does not use such features and yields a continuous potential energy surface by using continuous-filter convolutional layers.

enn-s2s

[21]提出了en-s帘网(enn-s2s),作为信息传递神经网络的一种变体,除了原子间距离之外,还使用了债券类型特征. 它在QM9基准的所有属性上实现了最先进的性能[21]. 不幸的是,它不能用于分子动力学预测(MD-17). 这是由于其潜在的能量表面因单热编码在输入中的离散而出现不连续. 相比之下,SchNet没有使用这种特性,通过使用连续滤波层产生连续潜在能量表面.

3 Continuous-filter convolutions

3 连续滤波卷积

Figure 1: The discrete filter (left) is not able to capture the subtle positional changes of the atoms resulting in discontinuous energy predictions $\hat{E}$ (bottom left). The continuous filter captures these changes and yields smooth energy predictions (bottom right).

图1:离散滤波器(左)无法捕捉原子的微妙位置变化,导致不连续的能量预测$\hat{E}$(左下). 连续的滤波器能捕捉这些变化并产生光滑的能量预测(下方右侧).

In deep learning, convolutional layers operate on discretized signals such as image pixels [22, 23], video frames [24] or digital audio data [25]. While it is sufficient to define the filter on the same grid in these cases, this is not possible for unevenly spaced inputs such as the atom positions of a molecule (see Fig. 1). Other examples include astronomical observations [26], climate data [27] and the financial market [28]. Commonly, this can be solved by a re-sampling approach defining a representation on a grid [29, 30, 7]. However, choosing an appropriate interpolation scheme is a challenge on its own and, possibly, requires a large number of grid points. Therefore, various extensions of convolutional layers even beyond the Euclidean space exist, e.g., for graphs [31, 32] and 3d shapes[33]. Analogously, we propose to use continuous filters that are able to handle unevenly spaced data, in particular, atoms at arbitrary positions.

在深度学习中,转录层在像素像素[22,23],视频帧[24]或数字音频数据[25]等盘状信号上运行. 虽然在这种情况下定义同一网格上的滤波器就足够了,但是对于分子的原子位置等不均匀的间距输入,这是不可能的(见图1)。 其他例子包括天文观测[26],气候数据[27]和金融市场[28]. 通常,这可以通过重新取样的方法来解决,在网格上定义一个代表[29,30,7]. 然而,选择适当的插值办法本身就是一个挑战,可能还需要大量的网点。 因此,甚至超过欧几利得空间的旋 快层存在各种延伸,例如,对于图[31,32]和3d形状[33]. 同样,我们提议使用能够处理不均匀空间数据的连续过滤器,特别是任意位置的原子。

Given the feature representations of $n$ objects $X^{l}=(\mathbf{x}^{l}_{1},\ldots,\mathbf{x}^{l}_{n})$ with $\mathbf{x}^{l}_{i}\in\mathbb{R}^{F}$ at locations $R=(\mathbf{r}_{1},\ldots,\mathbf{r}_{n})$ with $\mathbf{r}_{i}\in\mathbb{R}^{D}$, the continuous-filter convolutional layer $l$ requires a filter-generating function

鉴于$n$对象$X^{l}=(\mathbf{x}^{l}_{1},\ldots,\mathbf{x}^{l}_{n})$与$\mathbf{x}^{l}_{i}\in\mathbb{R}^{F}$位于$R=(\mathbf{r}_{1},\ldots,\mathbf{r}_{n})$与$\mathbf{r}_{i}\in\mathbb{R}^{D}$位置的特征表示, 连续过滤层$l$需要一个过滤生成功能

$$W^{l}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{F},$$

that maps from a position to the corresponding filter values. This constitutes a generalization of a filter tensor in discrete convolutional layers. As in dynamic filter networks [34], this filter-generating function is modeled with a neural network. While dynamic filter networks generate weights restricted to a grid structure, our approach generalizes this to arbitrary position and number of objects. The output $\mathbf{x}_{i}^{l+1}$ for the convolutional layer at position $\mathbf{r}_{i}$ is then given by

从位置映射到相应的过滤值。 这构成了滤波器在离散的卷积层中的概括. 如同动态滤波器网络[34],这种产生滤波器的功能是以神经网络为模型的. 虽然动态过滤网络产生仅限于网格结构的权重,但我们的方法将它概括为任意位置和对象数量. 在$\mathbf{r}_{i}$位置的转录层的输出$\mathbf{x}_{i}^{l+1}$由

$$\mathbf{x}_{i}^{l+1}=(X^{l}*W^{l})_{i}=\sum_{j}\mathbf{x}^{l}_{j}\circ W^{l}(\mathbf{r}_{i}-\mathbf{r}_{j}), \tag{2}$$

where "$\circ$" represents the element-wise multiplication. We apply these convolutions feature-wise for computational efficiency [35]. The interactions between feature maps are handled by separate object-wise or, specifically, atom-wise layers in SchNet.

其中 "$\circ$" 表示元素乘法。 为了计算效率,我们采用这些演化特性[35]。 特征图之间的相互作用由SchNet中单独的对象处理,具体来说,是由原子处理层处理.

4 SchNet

Figure 2: Illustration of SchNet with an architectural overview (left), the interaction block (middle) and the continuous-filter convolution with filter-generating network (right). The shifted softplus is defined as $\text{ssp}(x)=\ln(0.5e^{x}+0.5)$.

图 2: 以建筑概况(左),互动块(中)和连续滤波器与滤波器生成网络(右)的演化为SchNet的图解. 转动软加定义为$\text{ssp}(x)=\ln(0.5e^{x}+0.5)$.

SchNet is designed to learn a representation for the prediction of molecular energies and atomic forces. It reflects fundamental physical laws including invariance to atom indexing and translation, a smooth energy prediction w.r.t. atom positions as well as energy-conservation of the predicted force fields. The energy and force predictions are rotationally invariant and equivariant, respectively.

SchNet旨在学习分子能量和原子力预测的表示. 它反映了基本的物理定律,包括对原子索引和翻译的偏差、对原子位置的平稳能量预测以及预测力场的能量保护。 能量和力的预测分别是旋转不定的和相等的.

4.1 Architecture

4.1 架构

Fig. 2 shows an overview of the SchNet architecture. At each layer, the molecule is represented atom-wise analogous to pixels in an image. Interactions between atoms are modeled by the three interaction blocks. The final prediction is obtained after atom-wise updates of the feature representation and pooling of the resulting atom-wise energy. In the following, we discuss the different components of the network.

图2显示了SchNet架构的概况. 在每一层中,分子在图像中以原子感表示,类似于像素。 原子之间的相互作用由三个相互作用块模拟. 最终的预测是在对特征表现进行原子感更新并汇集由此产生的原子感能后获得的. 下面我们讨论网络的不同组成部分。

Molecular representation

A molecule in a certain conformation can be described uniquely by a set of $n$ atoms with nuclear charges $Z=(Z_{1},\dots,Z_{n})$ and atomic positions $R=(\mathbf{r}_{1},\dots\mathbf{r}_{n})$. Through the layers of the neural network, we represent the atoms using a tuple of features $X^{l}=(\mathbf{x}_{1}^{l},\dots\mathbf{x}_{n}^{l})$, with $\mathbf{x}^{l}_{i}\in\mathbb{R}^{F}$ with the number of feature maps $F$, the number of atoms $n$ and the current layer $l$. The representation of atom $i$ is initialized using an embedding dependent on the atom type $Z_{i}$:

分子表示

某种配位态中的分子可以被一组具有核电荷的$n$原子和原子位置的$R=(\mathbf{r}_{1},\dots\mathbf{r}_{n})$独占地描述. 通过神经网络的层层,我们用Tweple的特征$X^{l}=(\mathbf{x}_{1}^{l},\dots\mathbf{x}_{n}^{l})$代表原子,其中$\mathbf{x}^{l}_{i}\in\mathbb{R}^{F}$具有特征地图$F$,原子数量$n$和当前 的层$l$. 原子$i$的表示是使用一个依赖于原子类型$Z_{i}$的嵌入方式初始化的 :

$$\mathbf{x}^{0}_{i}=\mathbf{a}_{Z_{i}}. \tag{3}$$

The atom type embeddings $\mathbf{a}_{Z}$ are initialized randomly and optimized during training.

嵌入的原子类型$\mathbf{a}_{Z}$在训练中随机初始化并优化.

Atom-wise layers

A recurring building block in our architecture are atom-wise layers. These are dense layers that are applied separately to the representation $\mathbf{x}^{l}_{i}$ of atom $i$:

++ 原子层

我们建筑中一个反复出现的构件是原子层。 这些是密集层,分别适用于原子$\mathbf{x}^{l}_{i}$的表示:

$$\mathbf{x}^{l+1}_{i}=W^{l}\mathbf{x}^{l}_{i}+\mathbf{b}^{l}$$

These layers is responsible for the recombination of feature maps. Since weights are shared across atoms, our architecture remains scalable with respect to the size of the molecule.

这些层负责特征地图的重组. 由于权重是跨原子共享的,所以我们的架构在分子大小上仍然可以伸缩.

Interaction

The interaction blocks, as shown in Fig. 2 (middle), are responsible for updating the atomic representations based on the molecular geometry $R=(\mathbf{r}_{1},\dots\mathbf{r}_{n})$. We keep the number of feature maps constant at $F=64$ throughout the interaction part of the network. In contrast to MPNN and DTNN, we do not use weight sharing across multiple interaction blocks.

互动

如图2(middle)所示,相互作用块负责根据分子几何$R=(\mathbf{r}_{1},\dots\mathbf{r}_{n})$来更新原子表达. 我们在整个网络互动部分将特征地图的数量保持在$F=64$。 与MPNN和DTNN相比,我们不使用在多个交互块之间的重量共享.

The blocks use a residual connection inspired by ResNet [36]:

区块使用由ResNet[36]启发的剩余连接:

$$\mathbf{x}_{i}^{l+1}=\mathbf{x}_{i}^{l}+\mathbf{v}_{i}^{l}.$$

As shown in the interaction block in Fig. 2, the residual $\mathbf{v}_{i}^{l}$ is computed through an atom-wise layer, an interatomic continuous-filter convolution (cfconv) followed by two more atom-wise layers with a softplus non-linearity in between. This allows for a flexible residual that incorporates interactions between atoms and feature maps.

如图2中的相互作用块所示,残余的$\mathbf{v}_{i}^{l}$是通过一个原子偏导层,即原子间连续滤波演化(Cfconv),然后是两个原子偏导层,在两者之间具有软加非线性。 这使得能够有灵活的剩余部分,纳入原子和特征图之间的相互作用。

Filter-generating networks

(a) 1${}^{\text{st}}$ interaction block

过滤器生成网络

(a) 1${}^{\text{st}}$互动区块

The cfconv layer including its filter-generating network are depicted at the right panel of Fig. 2. In order to satisfy the requirements for modeling molecular energies, we restrict our filters for the cfconv layers to be rotationally invariant. The rotational invariance is obtained by using interatomic distances

Cfconv层包括其产生过滤器的网络在Fig的右面板上描绘. 2. 为了满足分子能量建模的要求,我们限制滤波器,使Cfconv层具有自转性。 利用原子间距离获得自转偏移

$$d_{ij}=\|\mathbf{r}_{i}-\mathbf{r}_{j}\|$$

as input for the filter network. Without further processing, the filters would be highly correlated since a neural network after initialization is close to linear. This leads to a plateau at the beginning of training that is hard to overcome. We avoid this by expanding the distance with radial basis functions

作为过滤网络的输入。 如果不进一步处理,滤波器将高度相关,因为初始化后的神经网络接近线性. 这导致在训练开始时出现一个难以克服的高原. 我们通过扩大射线基础功能的距离来避免这种情况

$$e_{k}(\mathbf{r}_{i}-\mathbf{r}_{j})=\exp(-\gamma\|d_{ij}-\mu_{k}\|^{2})$$

located at centers $0\text{\AA}\leq\mu_{k}\leq 30\text{\AA}$ every $0.1$Å with $\gamma=10$Å. This is chosen such that all distances occurring in the data sets are covered by the filters. Due to this additional non-linearity, the initial filters are less correlated leading to a faster training procedure. Choosing fewer centers corresponds to reducing the resolution of the filter, while restricting the range of the centers corresponds to the filter size in a usual convolutional layer. An extensive evaluation of the impact of these variables is left for future work. We feed the expanded distances into two dense layers with softplus activations to compute the filter weight $W(\mathbf{r}_{i}-\mathbf{r}_{j})$ as shown in Fig. 2 (right).

位于$0\text{\AA}\leq\mu_{k}\leq 30\text{\AA}$中心,每$0.1$与$\gamma=10$相邻。选择时,数据集中的所有距离都由过滤器覆盖。 由于这种额外的非线性,初始过滤器的关联性较小,导致训练程序更快。 选择较少的中枢对应于降低滤波器的分辨率,而限制中枢的范围对应于通常的卷积层中的滤波器大小. 对这些变量的影响的广泛评价留待今后工作处理。 我们用软加活化法将扩大的距离输入两个密集层,以计算图2(右)所示的滤波器重$W(\mathbf{r}_{i}-\mathbf{r}_{j})$。

Fig 3 shows 2d-cuts through generated filters for all three interaction blocks of SchNet trained on an ethanol molecular dynamics trajectory. We observe how each filter emphasizes certain ranges of interatomic distances. This enables its interaction block to update the representations according to the radial environment of each atom. The sequential updates from three interaction blocks allow SchNet to construct highly complex many-body representations in the spirit of DTNNs [20] while keeping rotational invariance due to the radial filters.

图3显示SchNet所有三个相互作用区块经过乙醇分子动力学轨迹训练后,通过生成的过滤器进行2d剪切。 我们观察每个过滤器是如何强调原子间距离的。 这使得它的相互作用块能够根据每个原子的射线环境更新表示. 从三个相互作用块进行顺序更新,使SchNet能够按照DTNNs[20]的精神构造高度复杂的多体表示力,同时由于射线滤波器而保持旋转的不变化.

4.2 Training with energies and forces

4.2 用能量与力训练

As described above, the interatomic forces are related to the molecular energy, so that we can obtain an energy-conserving force model by differentiating the energy model w.r.t. the atom positions

如上所述,原子间力与分子能量有关,这样我们就可以通过区分能量模型w.r.t.原子位置来获得能保全力模型.

$$\hat{\textbf{F}}_{i}(Z_{1},\dots,Z_{n},\mathbf{r}_{1},\dots,\mathbf{r}_{n})=-\frac{\partial\hat{E}}{\partial\mathbf{r}_{i}}(Z_{1},\dots,Z_{n},\mathbf{r}_{1},\dots,\mathbf{r}_{n}). \tag{4}$$

[19] pointed out that this leads to an energy-conserving force-field by construction. As SchNet yields rotationally invariant energy predictions, the force predictions are rotationally equivariant by construction. The model has to be at least twice differentiable to allow for gradient descent of the force loss. We chose a shifted softplus $\text{ssp}(x)=\ln(0.5e^{x}+0.5)$ as non-linearity throughout the network in order to obtain a smooth potential energy surface. The shifting ensures that $\text{ssp}(0)=0$ and improves the convergence of the network. This activation function shows similarity to ELUs [37], while having infinite order of continuity.

[19]指出,这导致通过建筑而形成一个节能的力场. 由于SchNet产生旋转无变能量预测,力预测通过构造来旋转等效. 该模型必须至少有两次不同,才能使力损失发生梯度下降。 我们选择了一个转向软加$\text{ssp}(x)=\ln(0.5e^{x}+0.5)$作为整个网络的非线性,以便获得一个平滑的潜在能量表面. 这种转变确保了$\text{ssp}(0)=0$,并改善了网络的趋同。 这个激活函数显示出与ELUs的相似性[37],同时具有无限的连续性顺序.

We include the total energy $E$ as well as forces $\mathbf{F}_{i}$ in the training loss to train a neural network that performs well on both properties:

我们把总能量$E$以及力量$\mathbf{F}_{i}$纳入训练损失中,以训练在两种特性上都表现良好的神经网络:

$$\ell(\hat{E},(E,\mathbf{F}_{1},\dots,\mathbf{F}_{n}))=\rho\|E-\hat{E}\|^{2}+\frac{1}{n}\sum_{i=0}^{n}\left\|\mathbf{F}_{i}-\left(-\frac{\partial\hat{E}}{\partial\mathbf{R}_{i}}\right)\right\|^{2}. \tag{5}$$

This kind of loss has been used before for fitting a restricted potential energy surfaces with MLPs [38]. In our experiments, we use $\rho=0.01$ for combined energy and force training. The value of $\rho$ was optimized empirically to account for different scales of energy and forces.

此类损失以前曾用于装配受限的潜在能量表面MLP[38]. 在我们的实验中,我们使用$\rho=0.01$进行综合能量和力学训练. 根据经验优化了$\rho$的价值,以计入不同的能量和力量尺度。

Due to the relation of energies and forces reflected in the model, we expect to see improved generalization, however, at a computational cost. As we need to perform a full forward and backward pass on the energy model to obtain the forces, the resulting force model is twice as deep and, hence, requires about twice the amount of computation time.

然而,由于模型中所反映的能量和力量之间的关系,我们期望以计算成本看到通用化的改善。 由于我们需要完全向前和向后传递能量模型以获取能量,由此产生的力模型是深度的两倍,因此需要大约两倍的计算时间。

Even though the GDML model captures this relationship between energies and forces, it is explicitly optimized to predict the force field while the energy prediction is a by-product. Models such as circular fingerprints [17], molecular graph convolutions or message-passing neural networks[21] for property prediction across chemical compound space are only concerned with equilibrium molecules, i.e., the special case where the forces are vanishing. They can not be trained with forces in a similar manner, as they include discontinuities in their predicted potential energy surface caused by discrete binning or the use of one-hot encoded bond type information.

尽管GDML模型捕捉到能量与力之间的这种关系,但它被明确优化以预测力场,而能量预测则是副产品. 循环指纹[17],分子图演化或信息传递神经网络[21]等模型用于跨化学化合物空间的属性预测,仅涉及平衡分子,即力量消失的特殊情况. 他们不能以类似的方式接受力学训练,因为他们包括因离散的宾加法或使用一热编码的债券类型信息而导致的预测潜在能量表面不连续.

5 Experiments and results

5 实验与结果

In this section, we apply the SchNet to three different quantum chemistry datasets: QM9, MD17 and ISO17. We designed the experiments such that each adds another aspect towards modeling chemical space. While QM9 only contains equilibrium molecules, for MD17 we predict conformational changes of molecular dynamics of single molecules. Finally, we present ISO17 combining both chemical and structural changes.

在本节中,我们将SchNet应用于三个不同的量子化学数据集:QM9,MD17和ISO17. 我们设计了实验,使每个实验都为模型化空间增加了另一个方面。 虽然QM9只包含平衡分子,但对MD17来说,我们预测单分子分子动力学的配位变化. 最后,我们提出ISO17,结合化学和结构变化。

For all datasets, we report mean absolute errors in kcal/mol for the energies and in kcal/mol/Å for the forces. The architecture of the network was fixed after an evaluation on the MD17 data sets for benzene and ethanol (see supplement). In each experiment, we split the data into a training set of given size $N$ and use a validation set of 1,000 examples for early stopping. The remaining data is used as test set. All models are trained with SGD using the ADAM optimizer [39] with 32 molecules per mini-batch. We use an initial learning rate of $10^{-3}$ and an exponential learning rate decay with ratio $0.96$ every 100,000 steps. The model used for testing is obtained using an exponential moving average over weights with decay rate 0.99.

对于所有数据集,我们报告能量的千卡/摩尔和能量的千卡/摩尔/奥的绝对误差。 网络的结构在对苯和乙醇的MD17数据集进行评估后被固定下来(见补编). 39. 在每一个实验中,我们把数据分成一个给定大小的 XMATH0XX 的训练集,并使用一套由1000个实例组成的验证集来进行早期停止。 其余数据作为测试集使用。 所有型号都使用SGD使用ADAM优化器进行训练[39],每小批量32分子. 我们采用$10^{-3}$的初始学习率和每10万步的$0.96$比率的指数化学习率衰减。 用于测试的模型采用指数移动平均值大于加权,衰减率为0.99。

5.1 QM9 – chemical degrees of freedom

5.1 QM9——化学自由度

$N$		SchNet	DTNN [20]	enn-s2s [21]		enn-s2s-ens5 [21]
50,000		0.59	0.94	–		–
100,000		0.34	0.84	–		–
110,462		0.31	–	0.45		0.33

Table 1: Mean absolute errors for energy predictions in kcal/mol on the QM9 data set with given training set size $N$. Best model in bold.

$N$		图标网	DTNN [20] (英语).	enn-s2s [21] (英语).		enn-s2s-ens5 [21] (英语).
50,000		页:1	0.94	- 说吧		- 说吧
100,000		页:1	0.84	- 说吧		- 说吧
110,462		页:1	- 说吧	0.45		0.33

表1:在QM9数据集中以kcal/mol表示的能量预测的平均绝对误差,特定训练集尺寸为$N$ . 最佳模型的粗体。

QM9 is a widely used benchmark for the prediction of various molecular properties in equilibrium [10, 40, 41]. Therefore, the forces are zero by definition and do not need to be predicted. In this setting, we train a single model that generalizes across different compositions and sizes.

QM9是一个广泛使用的基准,用于预测平衡中的各种分子性质[10,40,41]. 因此,根据定义,这些力量是零的,不需要预测。 在这个背景下,我们训练一个单一的模型,将不同的组成和大小概括起来。

QM9 consists of $\approx$130k organic molecules with up to 9 heavy atoms of the types $\{$C, O, N, F$\}$. As the size of the training set varies across previous work, we trained our models each of these experimental settings. Table 1 shows the performance of various competing methods for predicting the total energy (property $U_{0}$ in QM9). We provide comparisons to the DTNN [20] and the best performing MPNN configuration denoted enn-s2s and an ensemble of MPNNs (enn-s2s-ens5) [21]. SchNet consistently obtains state-of-the-art performance with an MAE of 0.31 kcal/mol at 110k training examples.

QM9由$\approx$ 130k有机分子组成,其重原子可达9个,类型为$\{$ C,O,N,F $\}$. 由于训练的大小因以往的工作而异, 我们训练了我们的模型 每一个实验环境。 表1显示预测总能量的各种相互竞争方法(质量M9中的财产$U_{0}$)的表现。 我们提供与 DTNN [20] 的比较,最出色的 MPNN 配置表示 *enn-s2s * 和一个 MPNN 的组合(enn-s2s-ens5) [21]. SchNet在110k训练实例下,以0.31 kcal/mol的MAE持续获得最新性能.

5.2 MD17 – conformational degrees of freedom

5.2 MD17——构象自由度

			$N$ = 1,000		$N$ = 50,000
			GDML [19]	SchNet		DTNN [20]	SchNet
			forces	energy	both		energy	energy	both
Benzene	energy		0.07	1.19	0.08		0.04	0.08	0.07
forces		0.23	14.12	0.31		–	1.23	0.17
Toluene	energy		0.12	2.95	0.12		0.18	0.16	0.09
forces		0.24	22.31	0.57		–	1.79	0.09
Malonaldehyde	energy		0.16	2.03	0.13		0.19	0.13	0.08
forces		0.80	20.41	0.66		–	1.51	0.08
Salicylic acid	energy		0.12	3.27	0.20		0.41	0.25	0.10
forces		0.28	23.21	0.85		–	3.72	0.19
Aspirin	energy		0.27	4.20	0.37		–	0.25	0.12
forces		0.99	23.54	1.35		–	7.36	0.33
Ethanol	energy		0.15	0.93	0.08		–	0.07	0.05
forces		0.79	6.56	0.39		–	0.76	0.05
Uracil	energy		0.11	2.26	0.14		–	0.13	0.10
forces		0.24	20.08	0.56		–	3.28	0.11
Naphtalene	energy		0.12	3.58	0.16		–	0.20	0.11
forces		0.23	25.36	0.58		–	2.58	0.11

Table 2: Mean absolute errors for energy and force predictions in kcal/mol and kcal/mol/Å, respectively. GDML and SchNet test errors for training with 1,000 and 50,000 examples of molecular dynamics simulations of small, organic molecules are shown. SchNets were trained only on energies as well as energies and forces combined. Best results in bold.

			$N$ = 1,000		$N$ = 50,000
			** GDML** [19]	** 科学网**		页:1	** 科学网**
			部队全盘	* 在我生命中,	两者		瞭力	瞭力	两者
** 苯基**	* 能源*		页:1	1.19	0.08		页:1	0.08	0.07
部队		页:1	14.12	0.31		- 说吧	1.23	0.17 (单位:千美元)
** 甲苯**	* 能源*		页:1	2.95	页:1		0.18	0.16	页:1
部队		页:1	22.31	0.57		- 说吧	1.79	页:1
** 马拉诺醛**	* 能源*		0.16	2.03	从现在起		0.19	0.13	0.0 0.0 0.0
部队		0.80	20.41	页:1		- 说吧	1.51	页:1
** 盐酸**	* 能源*		页:1	3.27	0.20		0.41	0.25	页:1
部队		页:1	23.21	0.85		- 说吧	3.72	页:1
** 阿斯匹林**	* 能源*		页:1	4.20	0.37		- 说吧	0.25	页:1
部队		页:1	23.54	1.35		- 说吧	7.36	页:1
** 乙醇**	* 能源*		0.15	0.93	页:1		- 说吧	0.07	** 0.05** (中文(简体) ).
部队		0.79	6.56	页:1		- 说吧	0.76	合并 合并 合并
(单位:百万)	* 能源*		页:1	2.26	0.14		- 说吧	0.13	页:1
部队		页:1	20.08	0.56		- 说吧	3.28	页:1
** 纳氟化物**	* 能源*		页:1	3.58	0.16		- 说吧	0.20	页:1
部队		页:1	25.36	0.58		- 说吧	2.58	页:1

表2: kcal/mol和kcal/mol/ Ö的能量和力预测平均绝对误差。 GDML和SchNet测试错误用于训练,并展示了1000和50,000个小有机分子分子动力学模拟实例. SchNet只接受了关于能量以及能量和力量组合的训练。 最佳成绩用粗体表示.

MD17 is a collection of eight molecular dynamics simulations for small organic molecules. These data sets were introduced by [19] for prediction of energy-conserving force fields using GDML. Each of these consists of a trajectory of a single molecule covering a large variety of conformations. Here, the task is to predict energies and forces using a separate model for each trajectory. This molecule-wise training is motivated by the need for highly-accurate force predictions when doing molecular dynamics.

MD17是8个分子动力学模拟小有机分子的集合. [19] 引入了这些数据集,用于使用GDML预测节能力场. 每一个都由一个单分子的轨迹组成,覆盖了各种各样的配体. 在这里,任务是对每个轨迹使用单独的模型来预测能量和力. 这种分子性举例说明的动机苔藓在进行分子动力学时需要高度精确的力预测.

Table 2 shows the performance of SchNet using 1,000 and 50,000 training examples in comparison with GDML and DTNN. Using the smaller data set, GDML achieves remarkably accurate energy and force predictions despite being only trained on forces. The energies are only used to fit the integration constant. As mentioned before, GDML does not scale well with the number of atoms and training examples. Therefore, it cannot be trained on 50,000 examples. The DTNN was evaluated only on four of these MD trajectories using the larger training set [20]. Note that the enn-s2s cannot be used on this dataset due to discontinuities in its inferred potential energy surface.

表2显示与GDML和DTNN相比,SchNet使用1 000和50 000个训练实例的绩效。 使用较小的数据集,GDML尽管只接受过部队训练,但能实现显著的精确能量和力量预测. 能量只用于适应整合常数. 如前所述,GDML在原子数量和训练实例方面没有很好的比例。 因此,它不能接受50 000个实例的训练。 对DTNN只使用更大的训练集[20]对这些MD轨迹中的四个进行了评价. 请注意,由于所推断的潜在能量表面不连续,所以不能在本数据集中使用enn-s2s。

We trained SchNet using just energies and using both energies and forces. While the energy-only model shows high errors for the small training set, the model including forces achieves energy predictions comparable to GDML. In particular, we observe that SchNet outperforms GDML on the more flexible molecules malonaldehyde and ethanol, while GDML reaches much lower force errors on the remaining MD trajectories that all include aromatic rings.

我们用能量和力量训练SchNet。 虽然仅能模型显示出小训练集的高度错误,但包括力在内的模型实现了与GDML相当的能量预测. 特别是,我们观察到,SchNet在更灵活的分子麦隆甲醛和乙醇上的GDML表现优于GDML,而GDML在其余的MD轨迹上的力差要小得多,这些轨迹都包含芳香环.

The real strength of SchNet is its scalability, as it outperforms the DTNN in three of four data sets using 50,000 training examples using only energies in training. Including force information, SchNet consistently obtains accurate energies and forces with errors below 0.12 kcal/mol and 0.33 kcal/mol/Å, respectively. Remarkably, when training on energies and forces using 1,000 training examples, SchNet performs better than training the same model on energies alone for 50,000 examples.

SchNet的真正优点在于其可扩展性,因为它在使用50 000个训练实例的四套数据中,仅利用训练方面的能量,在其中三套数据中超过了DTNN。 包括力量信息,SchNet持续获得准确的能量和力,误差分别低于0.12千卡/摩尔和0.33千卡/摩尔/奥. 值得注意的是,当使用1 000个训练实例进行关于能量和力量的训练时,SchNet的表现优于仅就50 000个实例进行关于能量的同一模式的训练。

5.3 ISO17 – chemical and conformational degrees of freedom

5.3 ISO17——化学与构象自由度

		mean predictor	SchNet
			energy	energy+forces
known molecules /	energy	14.89	0.52	0.36
unknown conformation	forces	19.56	4.13	1.00
unknown molecules /	energy	15.54	3.11	2.40
unknown conformation	forces	19.15	5.71	2.18

Table 3: Mean absolute errors on C7O2H10 isomers in kcal/mol.

		** 平均预测器**	** 科学网**
			能源在这一问题上,	能源+力量
** 已知分子/**	* 能源*	14.89	0.52	页:1
** 不详的规格**	部队	19.56	4.13	的某些方面
** 未知分子/**	* 能源*	15.54	3.11	页:1
** 不详的规格**	部队	19.15	5.71	页:1

表3:kcal/mol中C7O2H10异构体的平均绝对误差。

As the next step towards quantum-chemical exploration, we demonstrate the capability of SchNet to represent a complex potential energy surface including conformational and chemical changes. We present a new dataset – ISO17 – where we consider short MD trajectories of 129 isomers, i.e., chemically different molecules with the same number and types of atoms. In contrast to MD17, we train a joint model across different molecules. We calculate energies and interatomic forces from short MD trajectories of 129 molecules drawn randomly from the largest set of isomers in QM9. While the composition of all included molecules is C7O2H10, the chemical structures are fundamentally different. With each trajectory consisting of 5,000 conformations, the data set consists of 645,000 labeled examples.

作为进行量子化学勘探的下一步,我们证明SchNet有能力代表一个复杂的潜在能量表面,包括配位和化学变化。 我们提出了一个新的数据集 — ISO17 — 我们考虑129个异构体的MD短轨,即原子数量和类型相同的化学不同分子。 与MD17形成对比的是,我们在不同分子之间训练一种联合模型. 我们通过从QM9中最大的一组异构体随机抽取的129分子的MD短轨计算能量和原子间力. 虽然所有包含的分子的成分都是C7O2H10,但化学结构却根本不同. 每一轨迹包括5 000个轨迹,数据集包括645 000个有标签的例子。

We consider two scenarios with this dataset: In the first variant, the molecular graph structures present in training are also present in the test data. This demonstrates how well our model is able to represent a complex potential energy surface with chemical and conformational changes. In the more challenging scenario, the test data contains a different subset of molecules. Here we evaluate the generalization of our model to previously unseen chemical structures. We predict forces and energies in both cases and compare to the mean predictor as a baseline. We draw a subset of 4,000 steps from 80% of the MD trajectories for training and validation. This leaves us with a separate test set for each scenario: (1) the unseen 1,000 conformations of molecule trajectories included in the training set and (2) all 5,000 conformations of the remaining 20% of molecules not included in training.

我们考虑使用这一数据集的两种情况: 在第一个变体中,训练中存在的分子图结构也存在于测试数据中. 这表明我们的模型能够很好地通过化学和成像变化代表一个复杂的潜在能量表面。 在更具挑战性的情景中,测试数据包含了不同的分子子集. 在这里,我们评价我们的模型被概括到以前未见的化学结构。 我们预测两种情况下的力量和能量,并将平均预测器作为基线进行比较。 我们从80%的MD轨迹中抽取4000个步骤用于训练和验证. 这使得我们对每种情况都有一个单独的测试套:(1) 训练套中包含的未见的1,000个分子轨迹的轨迹;(2) 其余20%的分子未纳入训练的所有5,000个轨迹。

Table 3 shows the performance of the SchNet on both test sets. Our proposed model reaches chemical accuracy for the prediction of energies and forces for the test set of known molecules. Including forces in the training improves the performance here as well as on the set of unseen molecules. This shows that using force information does not only help to accurately predict nearby conformations of a single molecule, but indeed helps to generalize across chemical compound space.

表3显示SchNet在两套测试机上的性能. 我们提出的模型达到化学精确度,用于预测已知分子的能量和试验组的能量。 将力量纳入训练会提高这里的性能 以及一组看不见的分子 这表明,使用强力信息不仅有助于准确预测单个分子的近似性,而且确实有助于跨化学化合物空间的通用.

6 Conclusions

6 结论

We have proposed continuous-filter convolutional layers as a novel building block for deep neural networks. In contrast to the usual convolutional layers, these can model unevenly spaced data as occurring in astronomy, climate reasearch and, in particular, quantum chemistry. We have developed SchNet to demonstrate the capabilities of continuous-filter convolutional layers in the context of modeling quantum interactions in molecules. Our architecture respects quantum-chemical constraints such as rotationally invariant energy predictions as well as rotationally equivariant, energy-conserving force predictions.

我们提出 连续的过滤层 革命层 作为新的构件 深层神经网络。 与通常的卷积层形成对比的是,它们可以模拟在天文学,气候再研究,特别是量子化学中发生的不均匀的空间数据. 我们开发了SchNet,以在分子中的量子相互作用建模的背景下展示连续滤层的演化能力. 我们的架构尊重量子化学限制,如旋转无变能量预测,以及旋转等效、能动力预测。

We have evaluated our model in three increasingly challenging experimental settings. Each brings us one step closer to practical chemical exploration driven by machine learning. SchNet improves the state-of-the-art in predicting energies for molecules in equilibrium of the QM9 benchmark. Beyond that, it achieves accurate predictions for energies and forces for all molecular dynamics trajectories in MD17. Finally, we have introduced ISO17 consisting of 645,000 conformations of various C7O2H10 isomers. While we achieve promising results on this new benchmark, modeling chemical and conformational variations remains difficult and needs further improvement. For this reason, we expect that ISO17 will become a new standard benchmark for modeling quantum interactions with machine learning.

我们在三种日益具有挑战性的实验环境中评估了我们的模型。 每一个都使我们更接近机器学习驱动的实际化学探索。 SchNet改进了在QM9基准平衡中预测分子能量的先进水平. 除此之外,它还实现了对MD17中所有分子动力学轨迹的能量和力的准确预测. 最后,我们引入了ISO17,由各种C7O2H10异构体的645 000个配体组成。 虽然我们在这一新的基准上取得了有希望的成果,但模拟化学和符合性的变化仍然是困难的,需要进一步改进。 为此,我们期望ISO17将成为模拟量子相互作用与机器学习的新标准基准.

Acknowledgments

This work was supported by the Federal Ministry of Education and Research (BMBF) for the Berlin Big Data Center BBDC (01IS14013A). Additional support was provided by the DFG (MU 987/20-1) and from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement NO 657679. K.R.M. gratefully acknowledges the BK21 program funded by Korean National Research Foundation grant (No. 2012-005741) and the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (no. 2017-0-00451).

所表明的事实

这项工作得到了联邦教育和研究部(BMBF)对柏林大数据中心BBDC(01IS14013A)的支持. DFG(MU 987/20-1)和欧洲联盟根据Marie Sklodowska-Curie赠款协议第657679号开展的2020年地平线研究和创新方案提供了额外支助。 K.R.M.感谢韩国国家研究基金会赠款(第2012-005741号)和韩国政府供资的信息和通信技术促进研究所赠款(第2017-0-00451号)资助的BK21方案。

References

参考文献

[1] M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. Von Lilienfeld. Fast and accurate modeling of molecular atomization energies with machine learning. Phys. Rev. Lett. , 108(5):058301, 2012.

[2] G. Montavon, M. Rupp, V. Gobre, A. Vazquez-Mayagoitia, K. Hansen, A. Tkatchenko, K.-R. Müller, and O. A. von Lilienfeld. Machine learning of molecular electronic properties in chemical compound space. New J. Phys. , 15(9):095003, 2013.

[3] K. Hansen, G. Montavon, F. Biegler, S. Fazli, M. Rupp, M. Scheffler, O. A. Von Lilienfeld, A. Tkatchenko, and K.-R. Müller. Assessment and validation of machine learning methods for predicting molecular atomization energies. J. Chem. Theory Comput. , 9(8):3404–3419, 2013.

[4] K. T. Schütt, H. Glawe, F. Brockherde, A. Sanna, K.-R. Müller, and EKU Gross. How to represent crystal structures for machine learning: Towards fast prediction of electronic properties. Phys. Rev. B , 89(20):205118, 2014.

[5] K. Hansen, F. Biegler, R. Ramakrishnan, W. Pronobis, O. A. von Lilienfeld, K.-R. Müller, and A. Tkatchenko. Machine learning predictions of molecular properties: Accurate many-body potentials and nonlocality in chemical space. J. Phys. Chem. Lett. , 6:2326, 2015.

[6] F. A. Faber, L. Hutchison, B. Huang, Ju. Gilmer, S. S. Schoenholz, G. E. Dahl, O. Vinyals, S. Kearnes, P. F. Riley, and O. A. von Lilienfeld. Fast machine learning models of electronic and energetic properties consistently reach approximation errors better than dft accuracy. arXiv preprint arXiv:1702.05532 , 2017.

[7] F. Brockherde, L. Voigt, L. Li, M. E. Tuckerman, K. Burke, and K.-R. Müller. Bypassing the Kohn-Sham equations with machine learning. Nature Communications , 8(872), 2017.

[8] W. Boomsma and J. Frellsen. Spherical convolutions and their application in molecular modelling. In Advances in Neural Information Processing Systems 30 , pages 3436–3446. 2017.

[9] M. Eickenberg, G. Exarchakis, M. Hirn, and S. Mallat. Solid harmonic wavelet scattering: Predicting quantum molecular energy from invariant descriptors of 3d electronic densities. In Advances in Neural Information Processing Systems 30 , pages 6543–6552. 2017.

[10] R. Ramakrishnan, P. O. Dral, M. Rupp, and O. A. von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data , 1, 2014.

[11] S. Manzhos and T. Carrington Jr. A random-sampling high dimensional model representation neural network for building potential energy surfaces. J. Chem. Phys. , 125(8):084109, 2006.

[12] R. Malshe, M .and Narulkar, L. M. Raff, M. Hagan, S. Bukkapatnam, P. M. Agrawal, and R. Komanduri. Development of generalized potential-energy surfaces using many-body expansions, neural networks, and moiety energy approximations. J. Chem. Phys. , 130(18):184102, 2009.

[13] J. Behler and M. Parrinello. Generalized neural-network representation of high-dimensional potential-energy surfaces. Phys. Rev. Lett. , 98(14):146401, 2007.

[14] A. P. Bartók, M. C. Payne, R. Kondor, and G. Csányi. Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Phys. Rev. Lett. , 104(13):136403, 2010.

[15] J. Behler. Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys. , 134(7):074106, 2011.

[16] A. P. Bartók, R. Kondor, and G. Csányi. On representing chemical environments. Phys. Rev. B , 87(18):184115, 2013.

[17] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, NIPS , pages 2224–2232, 2015.

[18] S. Kearnes, K. McCloskey, M. Berndl, V. Pande, and P. F. Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of Computer-Aided Molecular Design , 30(8):595–608, 2016.

[19] S. Chmiela, A. Tkatchenko, H. E. Sauceda, I. Poltavsky, K. T. Schütt, and K.-R. Müller. Machine learning of accurate energy-conserving molecular force fields. Science Advances , 3(5):e1603015, 2017.

[20] K. T. Schütt, F. Arbabzadah, S. Chmiela, K.-R. Müller, and A. Tkatchenko. Quantum-chemical insights from deep tensor neural networks. Nature Communications , 8(13890), 2017.

[21] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning , pages 1263–1272, 2017.

[22] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541–551, 1989.

[23] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097–1105, 2012.

[24] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , pages 1725–1732, 2014.

[25] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop , pages 125–125, 2016.

[26] W. Max-Moerbeck, J. L. Richards, T. Hovatta, V. Pavlidou, T. J. Pearson, and A. C. S. Readhead. A method for the estimation of the significance of cross-correlations in unevenly sampled red-noise time series. Monthly Notices of the Royal Astronomical Society , 445(1):437–459, 2014.

[27] K. B. Ólafsdóttir, M. Schulz, and M. Mudelsee. Redfit-x: Cross-spectral analysis of unevenly spaced paleoclimate time series. Computers & Geosciences , 91:11–18, 2016.

[28] L. E. Nieto-Barajas and T. Sinha. Bayesian interpolation of unequally spaced time series. Stochastic environmental research and risk assessment , 29(2):577–587, 2015.

[29] J. C. Snyder, M. Rupp, K. Hansen, K.-R. Müller, and K. Burke. Finding density functionals with machine learning. Physical review letters , 108(25):253002, 2012.

[30] M. Hirn, S. Mallat, and N. Poilvert. Wavelet scattering regression of quantum chemical energies. Multiscale Modeling & Simulation , 15(2):827–863, 2017.

[31] J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun. Spectral networks and locally connected networks on graphs. In ICLR , 2014.

[32] M. Henaff, J. Bruna, and Y. LeCun. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163 , 2015.

[33] J. Masci, D. Boscaini, M. Bronstein, and P. Vandergheynst. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops , pages 37–45, 2015.

[34] X. Jia, B. De Brabandere, T. Tuytelaars, and L. V. Gool. Dynamic filter networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29 , pages 667–675. 2016.

[35] F. Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv preprint arXiv:1610.02357 , 2016.

[36] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770–778, 2016.

[37] D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

[38] A Pukrittayakamee, M Malshe, M Hagan, LM Raff, R Narulkar, S Bukkapatnum, and R Komanduri. Simultaneous fitting of a potential-energy surface and its corresponding force fields using feedforward neural networks. The Journal of chemical physics , 130(13):134101, 2009.

[39] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

[40] L. C. Blum and J.-L. Reymond. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc. , 131:8732, 2009.

[41] J.-L. Reymond. The chemical space project. Acc. Chem. Res. , 48(3):722–730, 2015.