Equivariant message passing for the prediction of tensorial properties and molecular spectra | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表
Abstract1. Introduction2. Related work3. Equivariant message passing3.1. Message passing for 3d-embedded graphs3.2. Building equivariant MPNNs3.3. Limits of rotationally invariant representations4. Polarizable atom interaction neural network (PAINN)4.1. Representation4.2. Prediction of tensorial properties5. Results5.1. Chemical compound space5.2. Molecular dynamics trajectories5.3. Advantages of equivariant features5.4. Molecular Spectra6. ConclusionsAcknowledgementsReferences
返回论文列表
目录

Equivariant message passing for the prediction of tensorial properties and molecular spectra

收录于 ICML 2021 会议论文 / Spotlight;Proceedings of the 38th International Conference on Machine Learning,PMLR 139:9377-9388;arXiv:2102.03150被引 376 次
打开论文来源

Equivariant Message Passing for the Prediction of Tensorial Properties and Molecular Spectra

Kristof T. Schütt^{1 2} Oliver T. Unke^{1 2} Michael Gastegger^{1 3}

^{1}Machine Learning Group, Technische Universität Berlin, 10587 Berlin, Germany
^{2}Berlin Institute for the Foundations of Learning and Data, 10587 Berlin, Germany
^{3}BASLEARN – TU Berlin / BASF Joint Lab for Machine Learning, 10587 Berlin, Germany.
Correspondence to: Kristof T. Schütt kristof.schuett@tu-berlin.de, Michael Gastegger michael.gastegger@tu-berlin.de.

Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021. Copyright 2021 by the author(s).

Abstract

Message passing neural networks have become a method of choice for learning on graphs, in particular the prediction of chemical properties and the acceleration of molecular dynamics studies. While they readily scale to large training data sets, previous approaches have proven to be less data efficient than kernel methods. We identify limitations of invariant representations as a major reason and extend the message passing formulation to rotationally equivariant representations. On this basis, we propose the polarizable atom interaction neural network (PAINN) and improve on common molecule benchmarks over previous networks, while reducing model size and inference time. We leverage the equivariant atomwise representations obtained by PAINN for the prediction of tensorial properties. Finally, we apply this to the simulation of molecular spectra, achieving speedups of 4-5 orders of magnitude compared to the electronic structure reference.

消息传递神经网络已成为图学习的常用方法,尤其是用于预测化学性质和加速分子动力学研究。它们虽然容易扩展到大规模训练集,但以往方法的数据效率已被证明不如核方法。我们把不变表示的局限视为一个主要原因,并把消息传递形式推广到旋转等变表示。在此基础上,我们提出可极化原子相互作用神经网络(PAINN),在常见分子基准上超过此前的网络,同时减小了模型规模并缩短了推断时间。我们利用 PAINN 得到的等变原子表示来预测张量性质。最后,我们将其用于分子光谱模拟,与电子结构参考相比实现了 4–5 个数量级的加速。

1. Introduction

Studying dynamics of chemical systems allows insight into processes such as reactions or the folding of proteins, and constitutes a fundamental challenge in computational chemistry. Since the motion of atoms is governed by the laws of quantum mechanics, accurate ab initio molecular dynamics (MD) simulations may require solving the Schrödinger equation for millions of time steps. While the exact solution is infeasible to compute for all but the smallest systems, even fast approximations such as density functional theory quickly become prohibitive for large systems and the prediction of accurate spectra.

研究化学体系的动力学,可以揭示反应或蛋白质折叠等过程,也是计算化学中的一个基本难题。由于原子运动受量子力学定律支配,准确的从头算分子动力学(MD)模拟可能需要求解数百万个时间步的薛定谔方程。除了最小的体系外,精确求解都不可行;即便是密度泛函理论这类较快的近似,在大体系和准确光谱预测上也会很快变得难以承受。

Recently, machine learning potentials (Behler, 2016; Unke et al., 2020; von Lilienfeld et al., 2020) have gained popularity for studying systems ranging from small molecules at high levels of theory (Chmiela et al., 2018; Westermayr et al., 2020) to systems with thousands or millions of atoms (Morawietz et al., 2016; Bartók et al., 2018; Lu et al., 2020). In particular, message-passing neural networks (Gilmer et al., 2017) (MPNNs) yield accurate predictions for chemical properties across chemical compound space and can handle large amounts of training data. Albeit MPNNs have significantly increased in accuracy over the years (as well as in computational cost), kernel methods with manually crafted features (Chmiela et al., 2017; Christensen et al., 2020; Bartók et al., 2010) have still proven to perform better when only small training sets are available.

近年来,机器学习势(Behler, 2016; Unke et al., 2020; von Lilienfeld et al., 2020)在研究各类体系上日益流行,既包括在高水平理论上的小分子(Chmiela et al., 2018; Westermayr et al., 2020),也包括含有数千乃至数百万原子的体系(Morawietz et al., 2016; Bartók et al., 2018; Lu et al., 2020)。尤其是消息传递神经网络(Gilmer et al., 2017)(MPNN),能够在化学空间中准确预测化学性质,并能处理大量训练数据。尽管 MPNN 这些年来精度明显提高(计算代价也一并上升),但在只有小训练集时,带手工特征的核方法(Chmiela et al., 2017; Christensen et al., 2020; Bartók et al., 2010)仍然被证明表现更好。

While molecules are often represented as graphs, they are in fact interacting particles in a continuous 3d space. Consequently, SchNet (Schütt et al., 2017) modeled message passes as continuous-filter convolutions over that space, albeit with rotationally invariant filters. As Miller et al. (2020) pointed out, this leads to a loss of relevant directional, equivariant information. Klicpera et al. (2020a) have introduced directional message-passing, the angular information here is restricted to the messages while the representation of nodes (atoms) remains rotationally invariant. While equivariant convolutions have been successfully applied in computer vision (Cohen & Welling, 2017; Weiler et al., 2018b; Worrall & Brostow, 2018), previous approaches to molecular prediction (Thomas et al., 2018; Anderson et al., 2019) have not reached the accuracy of their rotationally invariant counterparts.

分子虽然常被表示为图,但它们实际上是连续三维空间中的相互作用粒子。因此,SchNet(Schütt et al., 2017)把消息传递建模为在该空间上的连续滤波器卷积,不过所用滤波器是旋转不变的。正如 Miller et al.(2020)所指出的,这会丢失相关的方向性等变信息。Klicpera et al.(2020a)引入了方向性消息传递,但这里的角度信息只限于消息,节点(原子)的表示仍然是旋转不变的。等变卷积虽已在计算机视觉中成功应用(Cohen & Welling, 2017; Weiler et al., 2018b; Worrall & Brostow, 2018),但此前用于分子预测的方法(Thomas et al., 2018; Anderson et al., 2019)尚未达到其旋转不变对照方法的精度。

In this work, we propose rotationally equivariant message passing and the polarizable atom interaction neural network (PAINN) architecture as one instance of it. We examine the limited capability of rotation-invariant representations to propagate directional information and show that equivariant representations do not suffer from this issue. PAINN outperforms invariant message passing networks on common molecular benchmarks and performs at small sample sizes on par with kernel methods that have been deemed to be more data-efficient than neural networks. Beyond that, the rotationally equivariant representation of PAINN enables the prediction of tensorial properties which we apply to the ring-polymer MD simulation of infrared and Raman spectra. By an acceleration of 4-5 orders of magnitude, PAINN makes the simulation of these spectra feasible, reducing the runtime in one case from projected 25 years to one hour.

本文提出旋转等变的消息传递,并以可极化原子相互作用神经网络(PAINN)结构作为其中一个实例。我们考察了旋转不变表示在传播方向信息上的局限,并表明等变表示没有这一问题。PAINN 在常见分子基准上超过不变的消息传递网络,并在小样本设定下与被认为比神经网络更省数据的核方法表现相当。不仅如此,PAINN 的旋转等变表示还使预测张量性质成为可能,我们将其用于红外和拉曼光谱的环聚合物分子动力学模拟。凭借 4–5 个数量级的加速,PAINN 使这些光谱的模拟成为可行,在其中一个例子中把按进度估计的 25 年运行时间缩短到 1 小时。

2. Related work

Behler & Parrinello (2007) introduced neural network potentials taking atom-centered symmetry functions based on distances and angles as features. Graph neural networks (Scarselli et al., 2008) for molecular graphs (Duvenaud et al., 2015; Kearnes et al., 2016) and 3d geometries (Schütt et al., 2017; Gilmer et al., 2017; Schütt et al., 2018; Unke & Meuwly, 2019; Lubbers et al., 2018) do not require such manually crafted features, but learn embeddings of atom types and use graph convolutions or, more general, message passing (Gilmer et al., 2017), to model atom interactions based on interatomic distances. Klicpera et al. (2020a) introduced directional message passing by including additional angular information in the message.

Behler & Parrinello(2007)提出了以基于距离和角度的原子中心对称函数为特征的神经网络势。用于分子图(Duvenaud et al., 2015; Kearnes et al., 2016)和三维几何(Schütt et al., 2017; Gilmer et al., 2017; Schütt et al., 2018; Unke & Meuwly, 2019; Lubbers et al., 2018)的图神经网络(Scarselli et al., 2008)不需要这类手工特征,而是学习原子类型嵌入,并用图卷积或更一般的消息传递(Gilmer et al., 2017),基于核间距来建模原子相互作用。Klicpera et al.(2020a)通过在消息中加入额外的角度信息,引入了方向性消息传递。

Steerable CNNs (Cohen & Welling, 2017; Weiler et al., 2018a) allow to build equivariant filter banks according to known symmetries of associated feature types. While these approaches work on grids, architectures such as Tensor Field Networks (Thomas et al., 2018), Cormorant (Anderson et al., 2019) and NEQUIP (Batzner et al., 2021) use equivariant convolutions based on spherical harmonics (SH) and Clebsch-Gordon (CG) transforms for point clouds. Kondor & Trivedi (2018) have described a general framework on equivariance and convolutions in neural networks focusing on irreducible representations. In contrast, PAINN models equivariant interactions in Cartesian space which is conceptually simpler and does not require tensor contractions with CG coefficients. A similar approach was proposed by Jing et al. (2021) with the GVP-GNN. Both approaches are designed for distinct applications resulting in crucial differences in the design of the neural network architectures, most notably the message functions. The GVP-GNN has been designed for single-point protein sequence predictions, e.g., allowing the use of non-smooth components. In contrast, PAINN is designed for simulations with millions of inference steps, requiring a fast message function and a smoothly differentiable model.

可转向 CNN(Cohen & Welling, 2017; Weiler et al., 2018a)允许根据相关特征类型已知的对称性来构建等变滤波器组。这些方法作用在网格上,而 Tensor Field Networks(Thomas et al., 2018)、Cormorant(Anderson et al., 2019)和 NEQUIP(Batzner et al., 2021)这类结构则对点云使用基于球谐(SH)和 Clebsch-Gordon(CG)变换的等变卷积。Kondor & Trivedi(2018)描述了一个关于神经网络中等变与卷积的一般框架,重点在不可约表示。相比之下,PAINN 在笛卡尔空间中建模等变相互作用,概念上更简单,也不需要对 CG 系数做张量收缩。Jing et al.(2021)的 GVP-GNN 提出了类似思路。两种方法面向不同应用,因而在网络结构设计上有关键差异,尤其是消息函数。GVP-GNN 是为蛋白质序列的单点预测设计的,例如允许使用非光滑组件。而 PAINN 面向需要数百万次推断步的模拟,因此需要快速的消息函数以及可光滑微分的模型。

3. Equivariant message passing

Notation. To clearly distinguish between the two concepts, we write feature vectors as x∈RF×1x \in \mathbb{R}^{F \times 1}x∈RF×1 and vectors in 3d coordinate space as r⃗∈R1×3\vec{r} \in \mathbb{R}^{1 \times 3}r∈R1×3. Vectorial features will be written as x⃗∈RF×3\vec{x} \in \mathbb{R}^{F \times 3}x∈RF×3. Norms ∥⋅∥\|\cdot\|∥⋅∥, scalar products ⟨⋅,⋅⟩\langle \cdot, \cdot \rangle⟨⋅,⋅⟩ and tensor products ⊗\otimes⊗ are calculated along the spatial dimension. All other operations are calculated along the feature dimension, if not stated otherwise. We write the Hadamard product as ∘\circ∘.

记号。为了清楚区分这两个概念,我们把特征向量写成 x∈RF×1x \in \mathbb{R}^{F \times 1}x∈RF×1,把三维坐标空间中的向量写成 r⃗∈R1×3\vec{r} \in \mathbb{R}^{1 \times 3}r∈R1×3。矢量特征写成 x⃗∈RF×3\vec{x} \in \mathbb{R}^{F \times 3}x∈RF×3。范数 ∥⋅∥\|\cdot\|∥⋅∥、标量积 ⟨⋅,⋅⟩\langle \cdot, \cdot \rangle⟨⋅,⋅⟩ 和张量积 ⊗\otimes⊗ 沿空间维计算。若无另行说明,其余运算沿特征维计算。Hadamard 积写成 ∘\circ∘。

3.1. Message passing for 3d-embedded graphs

MPNNs build complex representations of nodes within their local neighborhood in a graph through a repeated exchange of messages followed by updates of node features. Here, we consider graphs embbedded in 3d Euclidean space, where edges are specified by the relative positions r⃗ij=r⃗j−r⃗i\vec{r}_{ij} = \vec{r}_j - \vec{r}_irij​=rj​−ri​ of nodes i,ji, ji,j within a local neighborhood N(i)={j∣∥r⃗ij∥≤rcut}N(i) = \{j \mid \|\vec{r}_{ij}\| \le r_{\mathrm{cut}}\}N(i)={j∣∥rij​∥≤rcut​}. In a chemistry context, this agrees with the fact that a large part of the energy variations can be attributed to local interactions, often conceptualized as bond lengths and angles. Thus, only nodes within that range can interact directly, so the number of messages does not scale quadratically with the number of the nodes. A general MPNN for embedded graphs can be written as

MPNN 通过反复交换消息并更新节点特征,在图的局部邻域内为节点构建复杂表示。这里我们考虑嵌入三维欧氏空间中的图(原文作 embbedded),其中边由局部邻域 N(i)={j∣∥r⃗ij∥≤rcut}N(i) = \{j \mid \|\vec{r}_{ij}\| \le r_{\mathrm{cut}}\}N(i)={j∣∥rij​∥≤rcut​} 内节点 i,ji,ji,j 的相对位置 r⃗ij=r⃗j−r⃗i\vec{r}_{ij} = \vec{r}_j - \vec{r}_irij​=rj​−ri​ 确定。在化学语境中,这与下述事实相符:能量变化的很大一部分可归因于局域相互作用,通常被概念化为键长和键角。因此,只有该范围内的节点才能直接相互作用,消息数量也就不会随节点数二次增长。嵌入图上的一般 MPNN 可以写成

mit+1=∑j∈N(i)Mt(sit,sjt,r⃗ij)(1)m_i^{t+1} = \sum_{j \in N(i)} M_t(s_i^t, s_j^t, \vec{r}_{ij}) \tag{1}mit+1​=j∈N(i)∑​Mt​(sit​,sjt​,rij​)(1)

sit+1=Ut(sit,mit+1).(2)s_i^{t+1} = U_t(s_i^t, m_i^{t+1}). \tag{2}sit+1​=Ut​(sit​,mit+1​).(2)

with the update and message functions UtU_tUt​ and MtM_tMt​, respectively (Gilmer et al., 2017). Many neural network potentials and even conventional CNNs can be cast in this general framework. Rotational invariance of the representation can be ensured by choosing rotationally invariant message and update functions, which by definition need to fulfill

其中 UtU_tUt​ 和 MtM_tMt​ 分别是更新函数和消息函数(Gilmer et al., 2017)。许多神经网络势,甚至常规 CNN,都可以纳入这一一般框架。若选择旋转不变的消息函数和更新函数,表示的旋转不变性就能得到保证;按定义,它们需要满足

f(x⃗)=f(Rx⃗),(3)f(\vec{x}) = f(R\vec{x}), \tag{3}f(x)=f(Rx),(3)

for any rotation matrix R∈R3×3R \in \mathbb{R}^{3 \times 3}R∈R3×3.

其中 R∈R3×3R \in \mathbb{R}^{3 \times 3}R∈R3×3 为任意旋转矩阵。

3.2. Building equivariant MPNNs

To obtain more expressive representations of local environments, neurons do not have to be scalar, but can be geometric objects such as vectors and tensors (Hinton et al., 2011; Cohen & Welling, 2017; Thomas et al., 2018; Anderson et al., 2019). For the purpose of this work, we restrict ourselves to scalar and vectorial representations sits_i^tsit​ and v⃗it\vec{v}_i^tvit​, respectively, such that a corresponding message pass can be written as

为了得到对局部环境更具表达力的表示,神经元不必是标量,也可以是向量和张量这类几何对象(Hinton et al., 2011; Cohen & Welling, 2017; Thomas et al., 2018; Anderson et al., 2019)。就本文而言,我们限于标量表示 sits_i^tsit​ 和矢量表示 v⃗it\vec{v}_i^tvit​,相应的一次消息传递可以写成

m⃗v,it+1=∑j∈N(i)M⃗t(sit,sjt,v⃗it,v⃗jt,r⃗ij)(4)\vec{m}_{v,i}^{t+1} = \sum_{j \in N(i)} \vec{M}_t(s_i^t, s_j^t, \vec{v}_i^t, \vec{v}_j^t, \vec{r}_{ij}) \tag{4}mv,it+1​=j∈N(i)∑​Mt​(sit​,sjt​,vit​,vjt​,rij​)(4)

Message and update functions for scalar features can be defined analogously. Rotational equivariance of vector features v⃗it+1\vec{v}_i^{t+1}vit+1​ can be ensured by employing rotationally equivariant functions U⃗t\vec{U}_tUt​ and M⃗t\vec{M}_tMt​, fulfilling

标量特征的消息函数和更新函数可以类似地定义。矢量特征 v⃗it+1\vec{v}_i^{t+1}vit+1​ 的旋转等变性,可通过采用旋转等变的函数 U⃗t\vec{U}_tUt​ 和 M⃗t\vec{M}_tMt​ 来保证,它们满足

Rf⃗(x⃗)=f⃗(Rx⃗)(5)R\vec{f}(\vec{x}) = \vec{f}(R\vec{x}) \tag{5}Rf​(x)=f​(Rx)(5)

for any rotation matrix R∈R3×3R \in \mathbb{R}^{3 \times 3}R∈R3×3, where the matrix-vector product is applied over the spatial dimension. This constitutes essentially a linearity constraint for directional information. Therefore, any nonlinearities of the model need to be applied to scalar features. In particular, equivariant MPNNs may use the following operations:

其中 R∈R3×3R \in \mathbb{R}^{3 \times 3}R∈R3×3 为任意旋转矩阵,矩阵–向量乘法作用在空间维上。这本质上是对方向信息的一种线性约束。因此,模型中的任何非线性都必须作用在标量特征上。具体而言,等变 MPNN 可以使用以下运算:

• Any (nonlinear) function of scalars: f(s)f(s)f(s)

• 标量的任意(非线性)函数:f(s)f(s)f(s)

• Scaling of vectors: s∘v⃗s \circ \vec{v}s∘v

• 向量的缩放:s∘v⃗s \circ \vec{v}s∘v

• Linear combinations of equivariant vectors: Wv⃗W\vec{v}Wv

• 等变向量的线性组合:Wv⃗W\vec{v}Wv

• Scalar products: s=∥v⃗∥2s = \|\vec{v}\|^2s=∥v∥2, s=⟨v⃗1,v⃗2⟩s = \langle \vec{v}_1, \vec{v}_2 \rangles=⟨v1​,v2​⟩

• 标量积:s=∥v⃗∥2s = \|\vec{v}\|^2s=∥v∥2,s=⟨v⃗1,v⃗2⟩s = \langle \vec{v}_1, \vec{v}_2 \rangles=⟨v1​,v2​⟩

• Vector products: v⃗1×v⃗2\vec{v}_1 \times \vec{v}_2v1​×v2​

• 向量积:v⃗1×v⃗2\vec{v}_1 \times \vec{v}_2v1​×v2​

Thus, directional information is preserved during message passing while invariant representations can be recovered from scalar products to obtain rotationally invariant predictions.

这样,方向信息在消息传递中得以保留,同时可以从标量积中恢复不变表示,从而得到旋转不变的预测。

3.3. Limits of rotationally invariant representations

In the following, we examine the expressiveness of invariant and equivariant representations at the example of simple 2d molecular structures. To ensure rotational invariance of a predicted property, the message function is often restricted to depend only on rotationally invariant inputs such as distances (Schütt et al., 2017; Lubbers et al., 2018) or angles (Klicpera et al., 2020a;b). While a single pass is limited to interactions within a local environment, successive message passes are supposed to construct more complex representations and propagate local information beyond the neighborhood. This raises the question whether rotationally invariant representations of atomic environments hih_ihi​ are sufficient here.

下面我们以简单的二维分子结构为例,考察不变表示和等变表示的表达能力。为了保证所预测性质的旋转不变性,消息函数常常被限制为只依赖于旋转不变的输入,例如距离(Schütt et al., 2017; Lubbers et al., 2018)或角度(Klicpera et al., 2020a;b)。单次传递只限于局部环境内的相互作用,但连续多次消息传递本应构建更复杂的表示,并把局部信息传播到邻域之外。这就提出一个问题:原子环境的旋转不变表示 hih_ihi​ 在这里是否足够。

Table 1. Comparison of expressiveness and computational complexity of distances, angles and directions for a simple message function of the oxygen atom of a water molecule.

Features Distances Angles Directions
Message MMM at atom iii ∑j∈Ni∣r⃗ij∣\sum_{j \in N_i} |\vec{r}_{ij}|∑j∈Ni​​∣rij​∣ ∑j∈Ni∑k∈Niαjik\sum_{j \in N_i} \sum_{k \in N_i} \alpha_{jik}∑j∈Ni​​∑k∈Ni​​αjik​ ∑j∈Nir⃗ij/∣r⃗ij∣\sum_{j \in N_i} \vec{r}_{ij}/|\vec{r}_{ij}|∑j∈Ni​​rij​/∣rij​∣
Scaling with neighbors O(∣N∣)O(|N|)O(∣N∣) O(∣N∣2)O(|N|^2)O(∣N∣2) O(∣N∣)O(|N|)O(∣N∣)
Resolve change of ∣r⃗1j∣|\vec{r}_{1j}|∣r1j​∣ yes no no
Resolve change of α213\alpha_{213}α213​ no yes yes

表 1. 对水分子中氧原子的一个简单消息函数,比较距离、角度和方向的表达能力与计算复杂度。

特征 距离 角度 方向
原子 iii 处的消息 MMM ∑j∈Ni∣r⃗ij∣\sum_{j \in N_i} |\vec{r}_{ij}|∑j∈Ni​​∣rij​∣ ∑j∈Ni∑k∈Niαjik\sum_{j \in N_i} \sum_{k \in N_i} \alpha_{jik}∑j∈Ni​​∑k∈Ni​​αjik​ ∑j∈Nir⃗ij/∣r⃗ij∣\sum_{j \in N_i} \vec{r}_{ij}/|\vec{r}_{ij}|∑j∈Ni​​rij​/∣rij​∣
随邻居数的缩放 O(∣N∣)O(|N|)O(∣N∣) O(∣N∣2)O(|N|^2)O(∣N∣2) O(∣N∣)O(|N|)O(∣N∣)
能否分辨 ∣r⃗1j∣|\vec{r}_{1j}|∣r1j​∣ 的变化 是 否 否
能否分辨 α213\alpha_{213}α213​ 的变化 否 是 是

Illustration of message passing using angles and directions for two molecular structures

Figure 1. Illustration of message passing using angles and directions for two structures. All edges within the cutoff range (dashed lines) have equal length. The representations of the blue and red node are the same using angles (left), while directions allow to distinguish both structures (right).

图 1. 对两种结构使用角度和方向做消息传递的示意。截断范围内(虚线)的所有边长度相等。使用角度时,蓝色和红色节点的表示相同(左);而方向则可以把两种结构区分开(右)。

Tab. 1 compares three simplified message functions for the example of a water molecule. Using a distance-based message function, the representation of the atom 1 (oxygen) is able to resolve changing bond lengths to atoms 2 and 3 (hydrogens), however it is not sensitive to changes of the bond angle. On the other hand, using the angle directly as part of the message function can not resolve the distances. Therefore, a combination of distances and angles is required to obtain a more expressive message function (Klicpera et al., 2020a). Unfortunately, including angles in the messages scales O(∣N∣2)O(|N|^2)O(∣N∣2) with the number of neighbors. Alternatively, directions to neighboring atoms may be used as messages M⃗t(r⃗ij)=r⃗ij/∥r⃗ij∥\vec{M}_t(\vec{r}_{ij}) = \vec{r}_{ij}/\|\vec{r}_{ij}\|Mt​(rij​)=rij​/∥rij​∥. Employing the update function Ut(m)=∥m∥2U_t(m) = \|m\|^2Ut​(m)=∥m∥2, this is related to angles as follows:

表 1 以水分子为例,比较了三种简化的消息函数。使用基于距离的消息函数时,原子 1(氧)的表示能够分辨它与原子 2、3(氢)之间键长的变化,但对键角变化不敏感。另一方面,把角度直接作为消息函数的一部分,又无法分辨距离。因此,要得到表达力更强的消息函数,需要把距离和角度结合起来(Klicpera et al., 2020a)。遗憾的是,在消息中加入角度会随邻居数按 O(∣N∣2)O(|N|^2)O(∣N∣2) 缩放。另一种做法是把指向相邻原子的方向用作消息 M⃗t(r⃗ij)=r⃗ij/∥r⃗ij∥\vec{M}_t(\vec{r}_{ij}) = \vec{r}_{ij}/\|\vec{r}_{ij}\|Mt​(rij​)=rij​/∥rij​∥。若采用更新函数 Ut(m)=∥m∥2U_t(m) = \|m\|^2Ut​(m)=∥m∥2,则它与角度有如下关系:

∥∑j=1Nr⃗ij∥r⃗ij∥∥2=∑j,k⟨r⃗ij∥r⃗ij∥,r⃗ik∥r⃗ik∥⟩=∑j=1N∑k=1Ncos⁡αjik.\left\|\sum_{j=1}^{N} \frac{\vec{r}_{ij}}{\|\vec{r}_{ij}\|}\right\|^2 = \sum_{j,k} \left\langle \frac{\vec{r}_{ij}}{\|\vec{r}_{ij}\|}, \frac{\vec{r}_{ik}}{\|\vec{r}_{ik}\|} \right\rangle = \sum_{j=1}^{N} \sum_{k=1}^{N} \cos \alpha_{jik}.​j=1∑N​∥rij​∥rij​​​2=j,k∑​⟨∥rij​∥rij​​,∥rik​∥rik​​⟩=j=1∑N​k=1∑N​cosαjik​.

Thus, using equivariant messages, the runtime complexity remains O(∣N∣)O(|N|)O(∣N∣) while angular information can be resolved. Note that this update function contracts the equivariant messages to a rotationally invariant representation.

因此,使用等变消息时,运行时复杂度仍是 O(∣N∣)O(|N|)O(∣N∣),同时又能分辨角度信息。需要注意,这一更新函数会把等变消息收缩成旋转不变表示。

Beyond the computational benefits, equivariant representations allow to propagate directional information beyond the neighborhood which is not possible in the invariant case. Fig. 1 illustrates this at a minimal example, where four atoms are arranged in an equidistant chain with the cutoff chosen such that only neighboring atoms are within range. We observe that for the two arrangements the angles are equal as well (Fig. 1, left). Therefore, they are indistinguishable for invariant message passing with distances and angles. In contrast, the equivariant representations differ in sign of their components (Fig. 1, right). When contracting them to invariant representations, as in the previous example, this information is lost. However, we may instead retain equivariance in the representation and design a message function ecMtec{M}_tecMt​ that does not only propagate directions to neighboring atoms, but those of equivariant representations as well. This enables the efficient propagation of directional information by scaling linearly with the number of neighbors and keeping the required cutoff small.

除了计算上的好处外,等变表示还允许把方向信息传播到邻域之外,这在不变情形下做不到。图 1 用一个最小例子说明了这一点:四个原子排成等距链,截断半径选得使只有相邻原子落在范围内。我们看到,两种排列的角度也相同(图 1,左)。因此,用距离和角度做不变消息传递时,它们无法区分。相比之下,等变表示在分量符号上不同(图 1,右)。若像上例那样把它们收缩成不变表示,这一信息就会丢失。不过,我们也可以在表示中保持等变性,并设计一种消息函数 ecMtec{M}_tecMt​,它不仅把方向传播给相邻原子,也传播等变表示本身的方向。这样就能按邻居数线性缩放,并把所需截断保持得较小,从而有效传播方向信息。

Note that neither many-body representations, using angles, dihedral angles etc., nor equivariant representations corresponding to a multipole expansion are complete. It has been shown that even when including up to 4-body invariants, there are structures with as few as eight atoms which cannot be distinguished (Pozdnyakov et al., 2020) and that a many-body expansion up to nnn-body contributions is necessary to guarantee convergence for a system consisting of nnn atoms (Hermann et al., 2007). The same holds for multipole expansions, where scalars and vectors correspond to the 0th and 1st order. Thus, even spherical harmonics expansions with less than infinite degree are unable to represent arbitrary equivariant nnn-body functions. Instead, a practically sufficient and computationally efficient approach for the problem at hand is desirable.

需要指出,无论是使用角度、二面角等的多体表示,还是对应于多极展开的等变表示,都不是完备的。已经表明,即便计入直至 4 体的不变量,仍然存在少至八个原子就无法被区分的结构(Pozdnyakov et al., 2020);而对由 nnn 个原子组成的体系,要保证收敛,多体展开需要直到 nnn 体贡献(Hermann et al., 2007)。多极展开也是如此,其中标量和矢量分别对应 0 阶和 1 阶。因此,即便是次数低于无穷的球谐展开,也无法表示任意的等变 nnn 体函数。更可取的是,针对手头问题给出一种在实践上足够、计算上也高效的方法。

In the following, we propose a neural network architecture that takes advantage of these properties. It overcomes the limitations of invariant representations discussed above, which we will demonstrate at the example of an organometallic compound in Section 5.3.2.

下面我们提出一种利用这些性质的神经网络结构。它克服了上文讨论的不变表示的局限,我们将在第 5.3.2 节以一个金属有机化合物为例加以说明。

4. Polarizable atom interaction neural network (PAINN)

The potential energy surface E(Z1,…,ZN,r⃗1,…,r⃗N)E(Z_1, \ldots, Z_N, \vec{r}_1, \ldots, \vec{r}_N)E(Z1​,…,ZN​,r1​,…,rN​), with nuclear charges Zi∈NZ_i \in \mathbb{N}Zi​∈N and atom positions r⃗i∈R3\vec{r}_i \in \mathbb{R}^3ri​∈R3 exhibits certain symmetries. Generally, they include invariance of the energy towards the permutation of atom indices iii, as well as rotations and translations of the molecule. A neural network potential should encode these constraints to ensure the symmetries of the predicted energy surface and increase data efficiency. A common inductive bias of the neural network potential is a decomposition of the energy into atomwise contributions E=∑i=1Nϵ(si)E = \sum_{i=1}^{N} \epsilon(s_i)E=∑i=1N​ϵ(si​), where an output network ϵ\epsilonϵ predicts energy contributions from atoms embedded within their chemical environment, represented by si∈RFs_i \in \mathbb{R}^Fsi​∈RF (Behler & Parrinello, 2007; Bartók et al., 2013).

势能面 E(Z1,…,ZN,r⃗1,…,r⃗N)E(Z_1, \ldots, Z_N, \vec{r}_1, \ldots, \vec{r}_N)E(Z1​,…,ZN​,r1​,…,rN​) 具有某些对称性,其中核电荷 Zi∈NZ_i \in \mathbb{N}Zi​∈N,原子位置 r⃗i∈R3\vec{r}_i \in \mathbb{R}^3ri​∈R3。一般而言,它们包括能量对原子指标 iii 的置换不变,以及对分子旋转和平移的不变。神经网络势应当编码这些约束,以保证所预测能量面的对称性,并提高数据效率。神经网络势常见的归纳偏置,是把能量分解为原子贡献 E=∑i=1Nϵ(si)E = \sum_{i=1}^{N} \epsilon(s_i)E=∑i=1N​ϵ(si​),其中输出网络 ϵ\epsilonϵ 根据嵌入在化学环境中的原子(由 si∈RFs_i \in \mathbb{R}^Fsi​∈RF 表示)来预测能量贡献(Behler & Parrinello, 2007; Bartók et al., 2013)。

While properties of chemical compounds may be such rotationally invariant scalars, they can also be equivariant tensorial properties, e.g. the multipole expansion of the electron density

化学化合物的性质可以是这类旋转不变的标量,也可以是等变的张量性质,例如电子密度的多极展开

n(r⃗)=q+μ⃗⊤r⃗+r⃗⊤Qr⃗+…,(6)n(\vec{r}) = q + \vec{\mu}^{\top} \vec{r} + \vec{r}^{\top} Q \vec{r} + \ldots, \tag{6}n(r)=q+μ​⊤r+r⊤Qr+…,(6)

with charge qqq, dipole μ⃗\vec{\mu}μ​, quadrupole QQQ and so on. Similarly, one can interpret invariant and equivariant atomwise representations as local charges and dipole moments of atom-centered multipole expansions (Gastegger et al., 2020). We leverage this in Section 4.2 to predict tensorial molecular properties. We coin our proposed architecture polarizable atom interaction neural network (PAINN).

其中 qqq 为电荷,μ⃗\vec{\mu}μ​ 为偶极矩,QQQ 为四极矩,以此类推。类似地,可以把不变和等变的原子表示解释为原子中心多极展开中的局部电荷和偶极矩(Gastegger et al., 2020)。我们在第 4.2 节利用这一点来预测分子的张量性质。我们把所提出的结构称为可极化原子相互作用神经网络(PAINN)。

4.1. Representation

The inputs to PAINN are the nuclear charges Zi∈NZ_i \in \mathbb{N}Zi​∈N and positions r⃗i∈R3\vec{r}_i \in \mathbb{R}^3ri​∈R3 for each atom iii. Similar to previous approaches, the invariant atom representations are initialized to learned embeddings of the atom type si0=aZi∈RF×1s_i^0 = a_{Z_i} \in \mathbb{R}^{F \times 1}si0​=aZi​​∈RF×1. We keep the number of features FFF constant throughout the network. The equivariant representations are set to v⃗i0=0⃗∈RF×3\vec{v}_i^0 = \vec{0} \in \mathbb{R}^{F \times 3}vi0​=0∈RF×3, since there is no directional information available initially.

PAINN 的输入是每个原子 iii 的核电荷 Zi∈NZ_i \in \mathbb{N}Zi​∈N 和位置 r⃗i∈R3\vec{r}_i \in \mathbb{R}^3ri​∈R3。与以往方法类似,不变的原子表示初始化为原子类型的可学习嵌入 si0=aZi∈RF×1s_i^0 = a_{Z_i} \in \mathbb{R}^{F \times 1}si0​=aZi​​∈RF×1。我们在整个网络中保持特征数 FFF 不变。等变表示设为 v⃗i0=0⃗∈RF×3\vec{v}_i^0 = \vec{0} \in \mathbb{R}^{F \times 3}vi0​=0∈RF×3,因为一开始还没有方向信息。

Next, we define message and update functions as introduced in Sec. 3. We use a residual structure of interchanging message and update blocks (Fig. 2a), resulting in coupled scalar and vectorial representations. For the residual of the scalar message function, we adopt the feature-wise, continuous-filter convolutions introduced by Schütt et al. (2017)

接下来,我们定义第 3 节中引入的消息函数和更新函数。我们使用消息块与更新块交替的残差结构(图 2a),从而得到耦合的标量和矢量表示。对标量消息函数的残差,我们采用 Schütt et al.(2017)提出的按特征的连续滤波器卷积

Δsim=(ϕs(s)∗Ws)i=∑jϕs(sj)∘Ws(∥r⃗ij∥),(7)\Delta s_i^{m} = (\phi_s(s) * W_s)_i = \sum_{j} \phi_s(s_j) \circ W_s(\|\vec{r}_{ij}\|), \tag{7}Δsim​=(ϕs​(s)∗Ws​)i​=j∑​ϕs​(sj​)∘Ws​(∥rij​∥),(7)

where ϕs\phi_sϕs​ consists of atomwise layers as shown in Fig. 2b. The roationally-invariant filters WsW_sWs​ are linear combinations of radial basis functions sin⁡(nπ/rcut∥r⃗ij∥)/∥r⃗ij∥\sin(n\pi / r_{\mathrm{cut}} \|\vec{r}_{ij}\|) / \|\vec{r}_{ij}\|sin(nπ/rcut​∥rij​∥)/∥rij​∥ as proposed by Klicpera et al. (2020a) with 1≤n≤201 \le n \le 201≤n≤20. Additionally, we apply a cosine cutoff to the filters (Behler, 2011).

其中 ϕs\phi_sϕs​ 由按原子的层组成,如图 2b 所示。旋转不变滤波器 WsW_sWs​(原文作 roationally-invariant)是径向基函数 sin⁡(nπ/rcut∥r⃗ij∥)/∥r⃗ij∥\sin(n\pi / r_{\mathrm{cut}} \|\vec{r}_{ij}\|) / \|\vec{r}_{ij}\|sin(nπ/rcut​∥rij​∥)/∥rij​∥ 的线性组合,采用 Klicpera et al.(2020a)的做法,其中 1≤n≤201 \le n \le 201≤n≤20。此外,我们还对滤波器施加余弦截断(Behler, 2011)。

Analogously, we use continuous-filter convolutions for the residual of the equivariant message function

类似地,我们对等变消息函数的残差使用连续滤波器卷积

Δv⃗im=∑jv⃗j∘ϕvv(sj)∘Wvv(∥r⃗ij∥)+∑jϕvs(sj)∘Wvs′(∥r⃗ij∥)r⃗ij∥r⃗ij∥,(8)\Delta \vec{v}_i^{m} = \sum_{j} \vec{v}_j \circ \phi_{vv}(s_j) \circ W_{vv}(\|\vec{r}_{ij}\|) + \sum_{j} \phi_{vs}(s_j) \circ W'_{vs}(\|\vec{r}_{ij}\|) \frac{\vec{r}_{ij}}{\|\vec{r}_{ij}\|}, \tag{8}Δvim​=j∑​vj​∘ϕvv​(sj​)∘Wvv​(∥rij​∥)+j∑​ϕvs​(sj​)∘Wvs′​(∥rij​∥)∥rij​∥rij​​,(8)

where the first term is a convolution of an invariant filter with scaled, equivariant features ∑jv⃗j∘ϕs(sj)\sum_{j} \vec{v}_j \circ \phi_s(s_j)∑j​vj​∘ϕs​(sj​), related to the gating proposed by Weiler et al. (2018a) as an equivariant nonlinearity. This propagates directional information obtained in previous message passes to neighboring atoms. The second term is a convolution of invariant features with an equivariant filter. This can be derived as the gradient of an invariant filter ∇Wvs(∥r⃗ij∥)=Wvs′(∥r⃗ij∥)r⃗ij/∥r⃗ij∥\nabla W_{vs}(\|\vec{r}_{ij}\|) = W'_{vs}(\|\vec{r}_{ij}\|) \vec{r}_{ij}/\|\vec{r}_{ij}\|∇Wvs​(∥rij​∥)=Wvs′​(∥rij​∥)rij​/∥rij​∥. Since Wvs′(∥r⃗ij∥)W'_{vs}(\|\vec{r}_{ij}\|)Wvs′​(∥rij​∥) is another invariant filter, we can model it directly without taking the derivative. Furthermore, we use a shared network ϕ\phiϕ to perform the transform of all message functions and split the features afterwards (see Fig. 2b).

其中第一项是不变滤波器与经过缩放的等变特征 ∑jv⃗j∘ϕs(sj)\sum_{j} \vec{v}_j \circ \phi_s(s_j)∑j​vj​∘ϕs​(sj​) 的卷积,这与 Weiler et al.(2018a)作为等变非线性提出的门控相关。它把此前消息传递中得到的方向信息传播给相邻原子。第二项是不变特征与等变滤波器的卷积。它可以导出为不变滤波器的梯度 ∇Wvs(∥r⃗ij∥)=Wvs′(∥r⃗ij∥)r⃗ij/∥r⃗ij∥\nabla W_{vs}(\|\vec{r}_{ij}\|) = W'_{vs}(\|\vec{r}_{ij}\|) \vec{r}_{ij}/\|\vec{r}_{ij}\|∇Wvs​(∥rij​∥)=Wvs′​(∥rij​∥)rij​/∥rij​∥。由于 Wvs′(∥r⃗ij∥)W'_{vs}(\|\vec{r}_{ij}\|)Wvs′​(∥rij​∥) 本身又是一个不变滤波器,我们可以直接对它建模,而不必再求导。此外,我们用一个共享网络 ϕ\phiϕ 来完成所有消息函数的变换,然后再把特征拆开(见图 2b)。

After the features-wise message blocks, the update blocks are applied atomwise across features. The residual of the scalar update function is given by

在按特征的消息块之后,更新块按原子跨特征施加。标量更新函数的残差为

Δsiu=ass(si,∥Vv⃗i∥)+asv(si,∥Vv⃗i∥)⟨Uv⃗i,Vv⃗i⟩.(9)\Delta s_i^{u} = a_{ss}(s_i, \|V\vec{v}_i\|) + a_{sv}(s_i, \|V\vec{v}_i\|) \langle U\vec{v}_i, V\vec{v}_i \rangle. \tag{9}Δsiu​=ass​(si​,∥Vvi​∥)+asv​(si​,∥Vvi​∥)⟨Uvi​,Vvi​⟩.(9)

PaiNN architecture with full network, message block, and update block

Figure 2. The architecture of PAINN with the full architecture (a) as well as the message (b) and update blocks (c) of the equivariant message passing. In all experiments, we use 128 features for sis_isi​ and v⃗i\vec{v}_ivi​ throughout the architecture. Other layer sizes are annotated in grey.

图 2. PAINN 的结构:完整结构 (a),以及等变消息传递中的消息块 (b) 和更新块 (c)。在全部实验中,我们在整个结构里对 sis_isi​ 和 v⃗i\vec{v}_ivi​ 都使用 128 维特征。其他层大小用灰色标出。

Again, we use a scaling functions computed by a shared network a(si,∥Vv⃗i∥)a(s_i, \|V\vec{v}_i\|)a(si​,∥Vvi​∥) as nonlinearity. In this case, the norm of a linear combination of features is also used to obtain the scaling, thus, coupling the scalar representations with contracted equivariant features. In a second term, we use the scalar product of two linear combinations of equivariant features. Similarly, we define the residual for the equivariant features

我们再次使用由共享网络 a(si,∥Vv⃗i∥)a(s_i, \|V\vec{v}_i\|)a(si​,∥Vvi​∥) 计算出的缩放函数作为非线性。在这种情形下,特征线性组合的范数也被用来得到缩放,从而把标量表示与收缩后的等变特征耦合起来。在第二项中,我们使用等变特征的两个线性组合的标量积。类似地,我们定义等变特征的残差

Δv⃗iu=avv(si,∥Vv⃗i∥)Uv⃗i,(10)\Delta \vec{v}_i^{u} = a_{vv}(s_i, \|V\vec{v}_i\|) U\vec{v}_i, \tag{10}Δviu​=avv​(si​,∥Vvi​∥)Uvi​,(10)

which is again a nonlinear scaling of linearly combined equivariant features.

它同样是对线性组合后的等变特征做非线性缩放。

4.2. Prediction of tensorial properties

PAINN is able to predict scalar properties using atomwise output layers of rotationally invariant representations sis_isi​, in the same manner as previous approaches (Schütt et al., 2017; Unke & Meuwly, 2019). Beyond that, the additional equivariant representations enable the prediction of equivariant tensors of order MMM that can be constructed using a rank-1 tensor decomposition

PAINN 能够用旋转不变表示 sis_isi​ 的按原子输出层来预测标量性质,方式与以往方法相同(Schütt et al., 2017; Unke & Meuwly, 2019)。不仅如此,额外的等变表示还使预测 MMM 阶等变张量成为可能,这类张量可用秩 1 张量分解来构造

T=∑i=1N∑k=1Rλ(si) ν⃗(v⃗i)k,1⊗⋯⊗ν⃗(v⃗i)k,M.(11)T = \sum_{i=1}^{N} \sum_{k=1}^{R} \lambda(s_i)\, \vec{\nu}(\vec{v}_i)_{k,1} \otimes \cdots \otimes \vec{\nu}(\vec{v}_i)_{k,M}. \tag{11}T=i=1∑N​k=1∑R​λ(si​)ν(vi​)k,1​⊗⋯⊗ν(vi​)k,M​.(11)

Gated equivariant block used in PaiNN output layers

Figure 3. Gated equivariant block.

图 3. 门控等变块。

The scalar and vectorial components of this decomposition can be obtained from an output network of gated equivariant blocks, as shown in Fig. 3. These components again make use gated equivariant nonlinearities (Weiler et al., 2018b) and show similarities to geometric vector perceptrons (GVP) (Jing et al., 2021). However, while GVP blocks are used as message functions, PAINN keeps the pairwise message functions comparatively lightweight. Instead complex transformations are restricted to the atomwise update function and tensor output network.

这一分解中的标量和矢量分量,可由门控等变块组成的输出网络得到,如图 3 所示。这些组件再次使用门控等变非线性(Weiler et al., 2018b),并与几何向量感知机(GVP)(Jing et al., 2021)有相似之处。不过,GVP 块是作为消息函数使用的,而 PAINN 把成对的消息函数保持得相对轻量。更为复杂的变换则限制在按原子的更新函数和张量输出网络中。

Depending on the chemical property of interest, one may want to include additional constraints or replace ν⃗k,m\vec{\nu}_{k,m}νk,m​ with the atom positions to include the global molecular geometry. In the following, we demonstrate how this can be employed for two tensorial properties that are important for the simulation of molecular spectra.

根据所关心的化学性质,人们可能希望加入额外约束,或用原子位置替换 ν⃗k,m\vec{\nu}_{k,m}νk,m​,以纳入分子的整体几何。下面我们说明,如何把这一点用于对分子光谱模拟很重要的两种张量性质。

The molecular dipole moment μ⃗\vec{\mu}μ​ is the response of the molecular energy to an electric field ∇F⃗E\nabla_{\vec{F}} E∇F​E and, at the same time, the first moment of the electron density (see Eq. 6). It is often predicted using latent atomic charges (Gastegger et al., 2017):

分子偶极矩 μ⃗\vec{\mu}μ​ 是分子能量对电场的响应 ∇F⃗E\nabla_{\vec{F}} E∇F​E,同时也是电子密度的一阶矩(见式 6)。它常常用潜在的原子电荷来预测(Gastegger et al., 2017):

μ⃗=∑i=1Nqatom(si) r⃗i.(12)\vec{\mu} = \sum_{i=1}^{N} q_{\mathrm{atom}}(s_i)\, \vec{r}_i. \tag{12}μ​=i=1∑N​qatom​(si​)ri​.(12)

This assumes the center of mass at r⃗=0⃗\vec{r} = \vec{0}r=0 for brevity. While this only leverages invariant representations, we can take advantage of the equivariant features of PAINN. In this picture, the molecular dipole is constructed from polarized atoms, i.e. both atomic dipoles and charges (Veit et al., 2020), yielding

为简洁起见,这里假定质心在 r⃗=0⃗\vec{r} = \vec{0}r=0。这种方法只用到了不变表示,而我们还可以利用 PAINN 的等变特征。在这一图景中,分子偶极由极化原子构成,也即同时使用原子偶极和电荷(Veit et al., 2020),从而得到

μ⃗=∑i=1Nμ⃗atom(v⃗i)+qatom(si) r⃗i.(13)\vec{\mu} = \sum_{i=1}^{N} \vec{\mu}_{\mathrm{atom}}(\vec{v}_i) + q_{\mathrm{atom}}(s_i)\, \vec{r}_i. \tag{13}μ​=i=1∑N​μ​atom​(vi​)+qatom​(si​)ri​.(13)

In terms of Eq. 11, this is a simple case with M=1M = 1M=1, R=2R = 2R=2, where the first term predicts the latent local dipoles and the second term computes the dipole generated by the local charges.

就式 11 而言,这是 M=1M = 1M=1、R=2R = 2R=2 的简单情形:第一项预测潜在的局部偶极,第二项计算由局部电荷产生的偶极。

As an example with order M=2M = 2M=2, we consider the polarizability tensor, which describes the response of the molecular dipole to an electric field JF⃗(μ⃗)=HF⃗(E)J_{\vec{F}}(\vec{\mu}) = H_{\vec{F}}(E)JF​(μ​)=HF​(E). We construct polarizability tensors using

作为 M=2M = 2M=2 阶的例子,我们考虑极化率张量,它描述分子偶极对电场的响应 JF⃗(μ⃗)=HF⃗(E)J_{\vec{F}}(\vec{\mu}) = H_{\vec{F}}(E)JF​(μ​)=HF​(E)。我们用下式构造极化率张量

α=∑i=1Nα0(si) I3+ν⃗(v⃗i)⊗r⃗i+r⃗i⊗ν⃗(v⃗i),(14)\alpha = \sum_{i=1}^{N} \alpha_0(s_i)\, I_3 + \vec{\nu}(\vec{v}_i) \otimes \vec{r}_i + \vec{r}_i \otimes \vec{\nu}(\vec{v}_i), \tag{14}α=i=1∑N​α0​(si​)I3​+ν(vi​)⊗ri​+ri​⊗ν(vi​),(14)

where the first term of the sum models isotropic, atomwise polarizabilities. The other two terms add the anisotropic components of the polarizability tensor. Similar to the charges in Eq. 13, here the atom positions are used to incorporate the global structure of the molecule.

其中求和的第一项建模各向同性的原子极化率。另外两项加入极化率张量的各向异性分量。与式 13 中的电荷类似,这里用原子位置来纳入分子的整体结构。

5. Results

PAINN has been implemented using PYTORCH (Paszke et al., 2019) and SCHNETPACK (Schütt et al., 2018). All models use F=128F = 128F=128 and two output layers as shown in Fig. 2 (blue) with the type of output layers depending on the property of interest. They were trained using the Adam optimizer (Kingma & Ba, 2014), the squared loss and weight decay λ=0.01\lambda = 0.01λ=0.01 (Loshchilov & Hutter, 2019), if not stated otherwise. We decay the learning rate by a factor of 0.5 if the validation loss plateaus. We apply exponential smoothing with factor 0.9 to the validation loss to reduce the impact of fluctuations which are particularly common when training with both energies and forces. Please refer to the supplement for further details on training parameters.

PAINN 已用 PYTORCH(Paszke et al., 2019)和 SCHNETPACK(Schütt et al., 2018)实现。所有模型均使用 F=128F = 128F=128,以及图 2(蓝色)所示的两层输出,输出层类型取决于所关注的性质。若无另行说明,训练使用 Adam 优化器(Kingma & Ba, 2014)、平方损失和权重衰减 λ=0.01\lambda = 0.01λ=0.01(Loshchilov & Hutter, 2019)。若验证损失进入平台,我们将学习率衰减 0.5 倍。我们对验证损失做因子为 0.9 的指数平滑,以减轻波动的影响,这种波动在同时训练能量和力时尤为常见。训练参数的更多细节请见补充材料。

5.1. Chemical compound space

We use the QM9 dataset of ≈130\approx 130≈130k small organic molecules (Ramakrishnan et al., 2014) with up to nine heavy atoms to evaluate the performance of PAINN for the prediction of scalar properties across chemical compound space. We predict the magnitude of the dipole moment using Eq. 13 and the electronic spatial extent by

我们使用约 13 万个最多含 9 个重原子的有机小分子组成的 QM9 数据集(Ramakrishnan et al., 2014),评估 PAINN 在化学空间中预测标量性质的表现。偶极矩的大小用式 13 预测,电子空间范围则由下式给出

⟨R2⟩=∑i=1Nqatom(si) ∥r⃗i∥2,\langle R^2 \rangle = \sum_{i=1}^{N} q_{\mathrm{atom}}(s_i)\, \|\vec{r}_i\|^2,⟨R2⟩=i=1∑N​qatom​(si​)∥ri​∥2,

as implemented by SCHNETPACK. The remaining properties are predicted as sums over atomic contributions. PAINN is trained on 110k examples while 10k molecules are used as a validation set for decaying the learning rate and early stopping. The remaining data is used as test set and results are averaged over three random splits. For the isotropic polarizability α\alphaα, we first observed validation MAEs of 0.054 a00.054\, a_00.054a0​. Upon closer inspection, we notice that for this property both the squared loss as well as the MAE can be reduced when minimizing the MAE directly (as done by Klicpera et al. (2020b)). This yields both validation and test MAEs of 0.045 a00.045\, a_00.045a0​ that are comparable to those of DIMENET++.

具体实现见 SCHNETPACK。其余性质则预测为原子贡献之和。PAINN 在 11 万个样本上训练,另用 1 万个分子作为验证集,用于衰减学习率和早停。剩余数据用作测试集,结果在三次随机划分上取平均。对于各向同性极化率 α\alphaα,我们最初观察到验证 MAE 为 0.054 a00.054\, a_00.054a0​。进一步检查后发现,对这一性质,若像 Klicpera et al.(2020b)那样直接最小化 MAE,平方损失和 MAE 都可以降低。由此得到的验证和测试 MAE 均为 0.045 a00.045\, a_00.045a0​,与 DIMENET++ 相当。

Tab. 2 shows the mean absolute error (MAE) of PAINN for 12 target properties of QM9 in comparison with previous approaches. SCHNET (Schütt et al., 2017) and PHYSNET (Unke & Meuwly, 2019) are MPNNs with distance-based interactions, DIMENET++ (Klicpera et al., 2020b) includes additional angular information and is an improved variant of DIMENET (Klicpera et al., 2020a). L1NET (Miller et al., 2020) and CORMORANT (Anderson et al., 2019) are equivariant neural networks based on spherical harmonics and Clebsch-Gordon coefficients.

表 2 给出了 PAINN 在 QM9 的 12 个目标性质上的平均绝对误差(MAE),并与以往方法比较。SCHNET(Schütt et al., 2017)和 PHYSNET(Unke & Meuwly, 2019)是基于距离相互作用的 MPNN,DIMENET++(Klicpera et al., 2020b)加入了额外的角度信息,是 DIMENET(Klicpera et al., 2020a)的改进变体。L1NET(Miller et al., 2020)和 CORMORANT(Anderson et al., 2019)是基于球谐函数和 Clebsch-Gordon 系数的等变神经网络。

Table 2. Mean absolute errors on QM9 dataset for various chemical properties. Results for PAINN are averaged over three random splits. Best in bold.

Target Unit SCHNET PHYSNET DIMENET++ CORMORANT L1NET PAINN
μ\muμ D 0.033 0.053 0.030 0.038 0.043 0.012
α\alphaα a03a_0^3a03​ 0.235 0.062 0.044 0.085 0.088 0.045
ϵHOMO\epsilon_{\mathrm{HOMO}}ϵHOMO​ meV 41 32.9 24.6 34 46.0 27.6
ϵLUMO\epsilon_{\mathrm{LUMO}}ϵLUMO​ meV 34 24.7 19.5 38 34.6 20.4
Δϵ\Delta\epsilonΔϵ meV 63 42.5 32.6 38 67.5 45.7
⟨R2⟩\langle R^2 \rangle⟨R2⟩ a02a_0^2a02​ 0.073 0.765 0.331 0.961 0.354 0.066
ZPVE meV 1.7 1.39 1.21 2.03 1.56 1.28
U0U_0U0​ meV 14 8.15 6.32 22 13.46 5.85
UUU meV 19 8.34 6.28 21 13.83 5.83
HHH meV 14 8.42 6.53 21 14.36 5.98
GGG meV 14 9.40 7.56 20 13.99 7.35
cvc_vcv​ cal/mol K\mathrm{cal}/\mathrm{mol}\,\mathrm{K}cal/molK 0.033 0.028 0.023 0.026 0.031 0.024

表 2. QM9 数据集上各化学性质的平均绝对误差。PAINN 的结果对三次随机划分取平均。最优结果加粗。

Target Unit SCHNET PHYSNET DIMENET++ CORMORANT L1NET PAINN
μ\muμ D 0.033 0.053 0.030 0.038 0.043 0.012
α\alphaα a03a_0^3a03​ 0.235 0.062 0.044 0.085 0.088 0.045
ϵHOMO\epsilon_{\mathrm{HOMO}}ϵHOMO​ meV 41 32.9 24.6 34 46.0 27.6
ϵLUMO\epsilon_{\mathrm{LUMO}}ϵLUMO​ meV 34 24.7 19.5 38 34.6 20.4
Δϵ\Delta\epsilonΔϵ meV 63 42.5 32.6 38 67.5 45.7
⟨R2⟩\langle R^2 \rangle⟨R2⟩ a02a_0^2a02​ 0.073 0.765 0.331 0.961 0.354 0.066
ZPVE meV 1.7 1.39 1.21 2.03 1.56 1.28
U0U_0U0​ meV 14 8.15 6.32 22 13.46 5.85
UUU meV 19 8.34 6.28 21 13.83 5.83
HHH meV 14 8.42 6.53 21 14.36 5.98
GGG meV 14 9.40 7.56 20 13.99 7.35
cvc_vcv​ cal/mol K\mathrm{cal}/\mathrm{mol}\,\mathrm{K}cal/molK 0.033 0.028 0.023 0.026 0.031 0.024

PAINN achieves state-of-the-art results in six target properties and yields comparable results to DIMENET++ on another two targets. On the remaining properties, PAINN achieves the second best results after DIMENET++. Note that PAINN using about 600k parameters is significantly smaller than DIMENET++ with about 1.8M parameters. For random batches of 50 molecules from QM9, the inference time is reduced from 45 ms to 13 ms, i.e. an improvement of more than 70%, when comparing PAINN to the reference implementation of DIMENET++^{1} using an NVIDIA V100.

PAINN 在六个目标性质上达到当时最优,并在另外两个目标上与 DIMENET++ 相当。在其余性质上,PAINN 的结果仅次于 DIMENET++,位列第二。需要注意的是,PAINN 约有 60 万参数,明显小于约 180 万参数的 DIMENET++。在 QM9 上随机抽取 50 个分子的批次中,与 DIMENET++ 的参考实现^{1} 相比,在一块 NVIDIA V100 上,推断时间从 45 ms 降到 13 ms,即改进超过 70%。

^{1}https://github.com/klicperajo/dimenet

5.2. Molecular dynamics trajectories

We evaluate the ability to predict combined energies and forces on the MD17 benchmark (Chmiela et al., 2017) including molecular dynamics trajectories of small organic molecules. While the atomic forces could be predicted directly from vectorial features, we employ the gradients of the energy model F⃗i=−∂E/∂r⃗i\vec{F}_i = -\partial E / \partial \vec{r}_iFi​=−∂E/∂ri​ to ensure conservation of energy. This property is crucial to run stable molecular dynamics simulations. To demonstrate the data efficiency of PAINN, we use the more challenging setting with 1k known structures of which we use 950 for training and 50 for validation, where a separate model is trained for each trajectory. Tab. 3 shows the comparison with SGDML (Chmiela et al., 2018) and NEQUIP (Batzner et al., 2021), which were trained on forces only, as well as SCHNET, PHYSNET, DIMENET and FCHL19 (Christensen et al., 2020), that were trained on a combined loss of energies and forces. Christensen & von Lilienfeld (2020) have found that the energies of MD17 are noisy. Thus, depending on the molecule and chosen tradeoff, using energies for training is not always beneficial. For this reason, we train two PAINN models per trajectory: only on forces and on a combined loss including energy with a force error weight of ρ=0.95\rho = 0.95ρ=0.95. PAINN achieves the lowest mean absolute errors for 12 out 14 targets on models trained only on forces and exhibits errors in a similar range as Gaussian regression with the FCHL19 kernel. Overall, PAINN performs best or equal to FCHL19 on 9 out of 14 targets. This demonstrates that equivariant neural networks approaches such as PAINN are able to compete with kernel methods in the small data regime, while being able to scale to large data sets at the same time.

我们在包含有机小分子分子动力学轨迹的 MD17 基准(Chmiela et al., 2017)上,评估同时预测能量和力的能力。原子力虽然可以直接从矢量特征预测,但我们采用能量模型的梯度 F⃗i=−∂E/∂r⃗i\vec{F}_i = -\partial E / \partial \vec{r}_iFi​=−∂E/∂ri​,以保证能量守恒。这一性质对运行稳定的分子动力学模拟至关重要。为了展示 PAINN 的数据效率,我们采用更具挑战性的设定:已知结构只有 1000 个,其中 950 个用于训练、50 个用于验证,并且为每条轨迹单独训练一个模型。表 3 给出了与只在力上训练的 SGDML(Chmiela et al., 2018)和 NEQUIP(Batzner et al., 2021),以及在能量与力的组合损失上训练的 SCHNET、PHYSNET、DIMENET 和 FCHL19(Christensen et al., 2020)的比较。Christensen & von Lilienfeld(2020)发现 MD17 的能量是有噪声的。因此,视分子和所选权衡而定,用能量来训练并不总是有益。出于这一原因,我们为每条轨迹训练两个 PAINN 模型:一个只在力上训练,一个在包含能量的组合损失上训练,力误差权重为 ρ=0.95\rho = 0.95ρ=0.95。在只在力上训练的模型中,PAINN 在 14 个目标里的 12 个上取得最低平均绝对误差(原文作 12 out 14),误差范围与使用 FCHL19 核的高斯回归相近。总体上看,PAINN 在 14 个目标中的 9 个上优于或持平于 FCHL19。这表明,PAINN 这类等变神经网络方法既能在小数据设定下与核方法竞争,又能同时扩展到大规模数据集。

Table 3. Mean absolute errors on MD17 dataset for energy and force predictions in kcal/mol and kcal/mol/Å, respectively. Batzner et al. (2021) only reported force errors for NEQUIP. Results for PAINN are averaged over three random splits. Best in bold.

SGDML NEQUIP PAINN SCHNET PHYSNET DIMENET FCHL19 PAINN
trained on forces only trained on energies & forces
ASPIRIN energy 0.19 – 0.167 0.37 0.230 0.204 0.182 0.159
forces 0.68 0.348 0.338 1.35 0.605 0.499 0.478 0.371
ETHANOL energy 0.07 – 0.064 0.08 0.059 0.064 0.054 0.063
forces 0.33 0.208 0.224 0.39 0.160 0.230 0.136 0.230
MALONDIALDEHYDE energy 0.10 – 0.100 0.13 0.094 0.104 0.081 0.091
forces 0.41 0.337 0.344 0.66 0.319 0.383 0.245 0.319
NAPHTHALENE energy 0.12 – 0.116 0.16 0.142 0.122 0.117 0.117
forces 0.11 0.096 0.077 0.58 0.310 0.215 0.151 0.083
SALICYLIC ACID energy 0.12 – 0.116 0.20 0.126 0.134 0.114 0.114
forces 0.28 0.238 0.195 0.85 0.337 0.374 0.221 0.209
TOLUENE energy 0.10 – 0.095 0.12 0.100 0.102 0.098 0.097
forces 0.14 0.101 0.094 0.57 0.191 0.216 0.203 0.102
URACIL energy 0.11 – 0.106 0.14 0.108 0.115 0.104 0.104
forces 0.24 0.172 0.139 0.56 0.218 0.301 0.105 0.140

表 3. MD17 数据集上能量和力预测的平均绝对误差,单位分别为 kcal/mol 和 kcal/mol/Å。Batzner et al.(2021)只报告了 NEQUIP 的力误差。PAINN 的结果对三次随机划分取平均。最优结果加粗。

SGDML NEQUIP PAINN SCHNET PHYSNET DIMENET FCHL19 PAINN
trained on forces only trained on energies & forces
ASPIRIN energy 0.19 – 0.167 0.37 0.230 0.204 0.182 0.159
forces 0.68 0.348 0.338 1.35 0.605 0.499 0.478 0.371
ETHANOL energy 0.07 – 0.064 0.08 0.059 0.064 0.054 0.063
forces 0.33 0.208 0.224 0.39 0.160 0.230 0.136 0.230
MALONDIALDEHYDE energy 0.10 – 0.100 0.13 0.094 0.104 0.081 0.091
forces 0.41 0.337 0.344 0.66 0.319 0.383 0.245 0.319
NAPHTHALENE energy 0.12 – 0.116 0.16 0.142 0.122 0.117 0.117
forces 0.11 0.096 0.077 0.58 0.310 0.215 0.151 0.083
SALICYLIC ACID energy 0.12 – 0.116 0.20 0.126 0.134 0.114 0.114
forces 0.28 0.238 0.195 0.85 0.337 0.374 0.221 0.209
TOLUENE energy 0.10 – 0.095 0.12 0.100 0.102 0.098 0.097
forces 0.14 0.101 0.094 0.57 0.191 0.216 0.203 0.102
URACIL energy 0.11 – 0.106 0.14 0.108 0.115 0.104 0.104
forces 0.24 0.172 0.139 0.56 0.218 0.301 0.105 0.140

5.3. Advantages of equivariant features

5.3.1. ABLATION STUDIES

We evaluate the impact of equivariant vector features at the example of the aspirin MD trajectory from the previous section. Compared to the full model, we remove the scalar product of vector features in Eq. 9 from the update block and the convolution over vector features in Eq. 8 (i.e., Wvv=0W_{vv} = 0Wvv​=0).

我们以前一节中的阿司匹林 MD 轨迹为例,评估等变矢量特征的影响。与完整模型相比,我们从更新块中去掉式 9 里矢量特征的标量积,并去掉式 8 中对矢量特征的卷积(即 Wvv=0W_{vv} = 0Wvv​=0)。

Table 4. Ablation study for the prediction of energies [kcal/mol] and forces [kcal/mol/Å] for aspirin trajectories from MD17.

Ablation # params FFF energy MAE force MAE
no ablation 588.3k 128 0.159 0.371
no scalar product of vector features in Eq. 8 589.1k 134 0.173 0.420
no vector propagation (Wvv=0\mathcal{W}_{vv} = 0Wvv​=0 in Eq. 7) 589.2k 135 0.183 0.441
remove both 590.1k 142 0.200 0.507
no vector features 590.3k 174 0.449 1.194

表 4. 在 MD17 阿司匹林轨迹上预测能量 [kcal/mol] 和力 [kcal/mol/Å] 的消融实验。

Ablation # params FFF energy MAE force MAE
no ablation 588.3k 128 0.159 0.371
no scalar product of vector features in Eq. 8 589.1k 134 0.173 0.420
no vector propagation (Wvv=0\mathcal{W}_{vv} = 0Wvv​=0 in Eq. 7) 589.2k 135 0.183 0.441
remove both 590.1k 142 0.200 0.507
no vector features 590.3k 174 0.449 1.194

Table 4 show the results for the various ablations. The number of parameters is kept approximately constant by raising the number of node features FFF accordingly. We observe that all ablated components contribute to the final accuracy of the model, where the convolution over equivariant features in the message function has a slightly larger impact. This component also enables the propagation of directional information, which will be examined in Sec. 5.3.2. Finally, we remove all vector features from the model, resulting in an invariant model. Despite keeping the number of parameters constant by increasing the number of atoms features to F=174F = 174F=174, the mean absolute error of the forces increases beyond 1 kcal/mol/Å.

表 4 给出了各项消融的结果(原文作 Table 4 show)。通过相应提高节点特征数 FFF,参数量大致保持不变。我们观察到,所有被消融的组件都对模型的最终精度有贡献,其中消息函数里对等变特征的卷积影响略大。这一组件也使方向信息的传播成为可能,将在第 5.3.2 节考察。最后,我们从模型中去掉全部矢量特征,得到一个不变模型。尽管通过把原子特征数提高到 F=174F = 174F=174 来保持参数量不变(原文作 atoms features),力的平均绝对误差仍升至超过 1 kcal/mol/Å。

Rotational energy profile of substituted ferrocene as a function of rotation angle theta

Figure 4. Rotational energy profile of substituted ferrocene obtained by varying the rotation angle θ\thetaθ while keeping all bond distances fixed at their equilibrium values (H: white, C: black, F: purple, Fe: grey). For small cutoffs rcutr_{\mathrm{cut}}rcut​ (dashed grey circles), MPNNs with scalar feature representations are unable to represent information about θ\thetaθ.

图 4. 取代二茂铁的转动能剖面:改变转动角 θ\thetaθ,同时把所有键长固定在平衡值(H:白,C:黑,F:紫,Fe:灰)。当截断 rcutr_{\mathrm{cut}}rcut​ 较小(灰色虚线圆)时,使用标量特征表示的 MPNN 无法表示关于 θ\thetaθ 的信息。

5.3.2. PROPAGATION OF DIRECTIONAL INFORMATION IN SUBSTITUTED FERROCENE

To demonstrate the advantages of equivariant over invariant representations in practice, we consider a ferrocene derivative where one hydrogen atom in each cyclopentadienyl ring is substituted by fluorine (see Fig. 4). This molecule has been chosen as it features small energy fluctuations (<1<1<1 kcal/mol) when the rings rotate relative to each other. Since the torsional energy profile depends mainly on the orientation of the distant fluorine atoms (measured by the rotation angle θ\thetaθ), it is a challenging prediction target for models without equivariant representations. Fig. 4 shows the predicted energy profiles for a full rotation of the cyclopentadienyl ring using cutoffs rcut∈{2.5,3.0,4.0}r_{\mathrm{cut}} \in \{2.5, 3.0, 4.0\}rcut​∈{2.5,3.0,4.0} Å. All models are trained on energies of 10k structures sampling thermal fluctuations (300 K) and ring rotations of substituted ferrocene. Another 1k structures are used for validation. SchNet (Schütt et al., 2017), which uses scalar features, predicts a flat energy profile for rcut≤3.0r_{\mathrm{cut}} \le 3.0rcut​≤3.0 Å, because it is unable to resolve θ\thetaθ. Although it is possible to encode the relevant information when using a larger cutoff, the learned representation seems to be susceptible to noise, as indicated by the deviations of the predicted energy profile. Since DimeNet (Klicpera et al., 2020a) includes terms that explicitly depend on angles between triplets of atoms, θ\thetaθ can be resolved for smaller cutoffs, but still fails for rcut=2.5r_{\mathrm{cut}} = 2.5rcut​=2.5 Å. In contrast, the equivariant environment representation of PAINN allows to faithfully reproduce the torsional energy profile even for very small cutoffs.

为了在实际中说明等变表示相对不变表示的优势,我们考虑一种二茂铁衍生物:每个环戊二烯基环上有一个氢原子被氟取代(见图 4)。选择这一分子,是因为它在两个环相对转动时能量波动很小(<1<1<1 kcal/mol)。扭转能剖面主要取决于远处氟原子的取向(用转动角 θ\thetaθ 度量),因此对没有等变表示的模型来说是一个有挑战性的预测目标。图 4 给出了在截断 rcut∈{2.5,3.0,4.0}r_{\mathrm{cut}} \in \{2.5, 3.0, 4.0\}rcut​∈{2.5,3.0,4.0} Å 下,环戊二烯基环完整旋转一圈的预测能剖面。所有模型都在 1 万个结构的能量上训练,这些结构采样了取代二茂铁的热涨落(300 K)和环转动。另用 1000 个结构做验证。使用标量特征的 SchNet(Schütt et al., 2017)在 rcut≤3.0r_{\mathrm{cut}} \le 3.0rcut​≤3.0 Å 时预测出平坦的能剖面,因为它无法分辨 θ\thetaθ。虽然使用更大截断时有可能编码相关信息,但学到的表示似乎对噪声敏感,这从预测能剖面的偏差可以看出。由于 DimeNet(Klicpera et al., 2020a)包含显式依赖于三原子夹角的项,θ\thetaθ 可以在较小截断下被分辨,但在 rcut=2.5r_{\mathrm{cut}} = 2.5rcut​=2.5 Å 时仍然失败。相比之下,PAINN 的等变环境表示即使在非常小的截断下,也能忠实再现扭转能剖面。

5.4. Molecular Spectra

We apply PAINN to the efficient computation of infrared and Raman spectra of ethanol and aspirin. Although these spectra can in principle be obtained from a single molecular structure via the harmonic oscillator approximation, such an approach tends to neglect important effects, e.g. different molecular conformations (Thomas et al., 2013). In order to obtain high quality spectra, molecular dynamics simulations have to be performed, requiring the prediction of forces at each time step. Additionally, one needs to compute dipole moments and polarizabilities along the generated trajectory. The infrared and Raman spectra are obtained as the Fourier transform of the time auto-correlation functions of the respective property. Using electronic structure methods, such simulations become prohibitively expensive due to the large number of successive computations required. Since nuclear quantum effects (NQE) need to be considered to obtain high quality spectra (Sauceda et al., 2021), ring-polymer molecular dynamics (RPMD) simulations need to be performed, which treat multiple copies of a molecule simultaneously and further increase computational cost.

我们将 PAINN 用于高效计算乙醇和阿司匹林的红外光谱与拉曼光谱。虽然这些光谱原则上可以通过谐振子近似从单一分子结构得到,但这种方法往往会忽略重要效应,例如不同的分子构象(Thomas et al., 2013)。为了得到高质量光谱,必须进行分子动力学模拟,从而需要在每个时间步预测力。此外,还需要沿生成的轨迹计算偶极矩和极化率。红外和拉曼光谱分别由相应性质的时间自相关函数经傅里叶变换得到。若使用电子结构方法,由于需要大量连续计算,这类模拟会变得过于昂贵。又因要得到高质量光谱必须考虑核量子效应(NQE)(Sauceda et al., 2021),还需要进行环聚合物分子动力学(RPMD)模拟,它同时处理分子的多个复本,从而进一步提高计算代价。

We train a joined model for energies, forces, dipole moments and polarizability tensors on 8k ethanol conformations using additional 1k molecules each as validation and test set. A second model is trained for aspirin with 16k training examples and 2k molecules for validation and testing. Energies and forces are predicted as described above for MD17. The dipole moments and polarizability tensors are obtained as described in Section 4.2, employing two gated equivariant blocks each yielding atomwise scalars and vectors, that are used to compute the outputs as described in Eqs. 13 and 14. The joined model exhibits accurate predictions for energy, forces, dipole moments and polarizabilities, as shown in the inset table of Fig 5. To evaluate the importance of equivariant features for the dipole prediction, we have trained an alternative model using only latent partial charges, but no local dipoles. The results show that predicting molecular dipoles using vector features (Eq. 13) reduces the mean abs. error by more than 50%, reaching 0.009 D compared to 0.020 D using only scalars and positions (Eq. 12). Simulated infrared and Raman spectra using classical MD and RPMD with 64 replicas and the respective PAINN models are shown for both molecules in Fig. 5, alongside harmonic spectra obtained with the electronic structure reference and, for ethanol, experimental spectra recorded in the gas phase (Linstrom & Mallard, 2020; Kiefer, 2017).

我们在 8000 个乙醇构象上训练一个同时预测能量、力、偶极矩和极化率张量的联合模型(原文作 joined model),并另用各 1000 个分子作为验证集和测试集。第二个模型针对阿司匹林训练,训练集 1.6 万个样本,验证和测试各 2000 个分子。能量和力的预测方式与上文 MD17 相同。偶极矩和极化率张量按第 4.2 节所述得到,各使用两个门控等变块,分别产生按原子的标量和矢量,再按式 13 和式 14 计算输出。如图 5 内嵌表所示,这一联合模型对能量、力、偶极矩和极化率都给出了准确预测。为了评估等变特征对偶极预测的重要性,我们还训练了一个只用潜在部分电荷、不用局部偶极的对照模型。结果表明,用矢量特征(式 13)预测分子偶极,可使平均绝对误差降低一半以上,达到 0.009 D,而只用标量和位置(式 12)时为 0.020 D。图 5 给出了两种分子在经典 MD 以及 64 个复本的 RPMD 下、用相应 PAINN 模型模拟得到的红外和拉曼光谱,并附上电子结构参考的谐波光谱;对乙醇还给出了气相实验光谱(Linstrom & Mallard, 2020; Kiefer, 2017)。

IR and Raman spectra of ethanol and aspirin compared with harmonic reference and experiment

Figure 5. IR (top) and Raman (bottom) spectra of ethanol and aspirin. Spectra calculated with the reference method using the harmonic oscillator approximation are shown in black (QM harmonic). The inset table shows the mean absolute errors on the respective test set.

图 5. 乙醇和阿司匹林的红外(上)与拉曼(下)光谱。用参考方法在谐振子近似下计算的光谱以黑色给出(QM harmonic)。内嵌表给出相应测试集上的平均绝对误差。

The peak positions and intensities of ethanol infrared spectra (Fig. 5a, top) computed with classical MD and PAINN agree closely with the static electronic structure reference, e.g. in the C-H and O-H stretching regions at 3000 cm−1^{-1}−1 and 3900 cm−1^{-1}−1. This indicates, that PAINN faithfully reproduces the original electronic structure method in general. However, when compared to experiment, both spectra are shifted towards higher frequencies. The PAINN RPMD spectrum, on the other hand shows excellent agreement with experiment, illustrating the importance of NQEs. Similar observations can be made for the ethanol Raman spectrum (Fig. 5a, bottom), where the PAINN RMPD spectrum once again offers the most faithful reproduction of the experiment. We observe similar trends for aspirin (Fig. 5b). RPMD spectra are once again shifted to lower frequencies while the classical MD closely resembles the static electronic structure spectrum.

用经典 MD 和 PAINN 计算的乙醇红外光谱(图 5a,上)在峰位和强度上与静态电子结构参考密切吻合,例如在 3000 cm−1^{-1}−1 和 3900 cm−1^{-1}−1 的 C–H 与 O–H 伸缩区。这表明,总体而言 PAINN 忠实地再现了原来的电子结构方法。不过,与实验相比,两条光谱都向更高频率偏移。另一方面,PAINN 的 RPMD 光谱与实验吻合极好,说明了核量子效应的重要性。乙醇拉曼光谱(图 5a,下)也可看到类似现象,其中 PAINN 的 RMPD 光谱(原文作 RMPD)再次最忠实地再现了实验。我们在阿司匹林(图 5b)上观察到类似趋势:RPMD 光谱再次向更低频率移动,而经典 MD 则与静态电子结构光谱十分接近。

PAINN reduces the time required to obtain these spectra by several orders of magnitude. For ethanol, an individual reference computation takes ≈140\approx 140≈140 seconds compared to 14 ms for a PAINN evaluation on a V100 GPU. This corresponds to a reduction of the overall simulation time from 400 days to approximately one hour. The speedup offered by PAINN is even more pronounced for aspirin, where a simulation that would have taken 25 years (3140 seconds / step) now takes one hour (15 ms / step).

PAINN 把获得这些光谱所需的时间缩短了数个数量级。对乙醇而言,一次参考计算约需 140 秒,而在 V100 GPU 上一次 PAINN 求值只需 14 ms。这相当于把总体模拟时间从 400 天降到大约 1 小时。PAINN 带来的加速在阿司匹林上更为显著:一场本来需要 25 年的模拟(3140 秒/步),现在只需 1 小时(15 ms/步)。

6. Conclusions

We have given general guidelines to design equivariant MPNNs and discussed the advantages of equivariant representations over angular features in terms of computational efficiency as well as their ability to propagate directional information. On this basis, we have proposed PAINN that yields fast and accurate predictions of scalar and tensorial molecular properties. Thereby, equivariant message passing allows us to significantly reduce both model size and inference time compared to directional message-passing while retaining accuracy. Finally, we have demonstrated that PAINN can be applied to the prediction of tensorial properties, which we leverage to accelerate the simulation of molecular spectra by 4-5 orders of magnitude – from years to hours.

我们给出了设计等变 MPNN 的一般原则,并从计算效率以及传播方向信息的能力两方面,讨论了等变表示相对角度特征的优势。在此基础上,我们提出了 PAINN,能够快速、准确地预测标量和张量分子性质。由此,与方向性消息传递相比,等变消息传递使我们能够在保持精度的同时显著减小模型规模并缩短推断时间。最后,我们证明 PAINN 可用于预测张量性质,并据此把分子光谱模拟加速了 4–5 个数量级——从数年降到数小时。

In future work, the equivariant representation of PAINN as well as the ability to predict tensorial properties may be leveraged in generative models of 3d geometries (Gebauer et al., 2019; Köhler et al., 2019; Simm et al., 2020) or the prediction of wavefunctions (Hegde & Bowen, 2017; Schütt et al., 2019; Hermann et al., 2020). We see further applications of equivariant message passing in 3d shape recognition and graph embedding (Goyal & Ferrara, 2018).

在未来工作中,PAINN 的等变表示以及预测张量性质的能力,可用于三维几何的生成模型(Gebauer et al., 2019; Köhler et al., 2019; Simm et al., 2020),或用于波函数预测(Hegde & Bowen, 2017; Schütt et al., 2019; Hermann et al., 2020)。我们还看到等变消息传递在三维形状识别和图嵌入中的进一步应用(Goyal & Ferrara, 2018)。

Many challenges remain for the fast and accurate prediction of molecular properties, e.g. modeling of enzymatic active sites or surface reactions. To describe such phenomena, highly accurate reference methods are required. Due to their computational cost, reference data generation can become a bottleneck, making data-efficient MPNNs such as PAINN invaluable for future chemistry research.

要快速而准确地预测分子性质,仍有许多挑战,例如酶活性位点或表面反应的建模。要描述这类现象,需要高精度的参考方法。由于其计算代价,参考数据的生成可能成为瓶颈,这使得 PAINN 这类数据高效的 MPNN 对未来的化学研究尤为宝贵。

Acknowledgements

KTS and OTU acknowledge support by the Federal Ministry of Education and Research (BMBF) for BIFOLD (01IS18037A). OTU acknowledges funding from the Swiss National Science Foundation (Grant No. P2BSP2 188147). MG works at the BASLEARN – TU Berlin/BASF Joint Lab for Machine Learning, co-financed by TU Berlin and BASF SE.

KTS 和 OTU 感谢德国联邦教育与研究部(BMBF)对 BIFOLD(01IS18037A)的支持。OTU 感谢瑞士国家科学基金会的资助(资助号 P2BSP2 188147)。MG 就职于 BASLEARN——柏林工业大学/巴斯夫机器学习联合实验室,由柏林工业大学与 BASF SE 共同资助。

References

Anderson, B., Hy, T. S., and Kondor, R. Cormorant: Covariant Molecular Neural Networks. Advances in Neural Information Processing Systems, 32:14537–14546, 2019.

Bartók, A. P., Payne, M. C., Kondor, R., and Csányi, G. Gaussian approximation potentials: The accuracy of quantum mechanics, without the electrons. Phys. Rev. Lett., 104(13):136403, 2010.

Bartók, A. P., Kondor, R., and Csányi, G. On representing chemical environments. Phys. Rev. B, 87(18):184115, 2013.

Bartók, A. P., Kermode, J., Bernstein, N., and Csányi, G. Machine learning a general-purpose interatomic potential for silicon. Phys. Rev. X, 8(4):041048, 2018.

Batzner, S., Smidt, T. E., Sun, L., Mailoa, J. P., Kornbluth, M., Molinari, N., and Kozinsky, B. SE (3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials. arXiv preprint arXiv:2101.03164, 2021.

Behler, J. Atom-centered symmetry functions for constructing high-dimensional neural network potentials. J. Chem. Phys, 134(7):074106, 2011.

Behler, J. Perspective: Machine learning potentials for atomistic simulations. J. Chem. Phys, 145(17):170901, 2016.

Behler, J. and Parrinello, M. Generalized neural-network representation of high-dimensional potential-energy surfaces. Phys. Rev. Lett., 98(14):146401, 2007.

Chmiela, S., Tkatchenko, A., Sauceda, H. E., Poltavsky, I., Schütt, K. T., and Müller, K.-R. Machine Learning of Accurate Energy-Conserving Molecular Force Fields. Sci. Adv., 3(5):e1603015, 2017.

Chmiela, S., Sauceda, H. E., Müller, K.-R., and Tkatchenko, A. Towards exact molecular dynamics simulations with machine-learned force fields. Nat. Commun., 9(1):3887, 2018. doi: 10.1038/s41467-018-06169-2.

Christensen, A. S. and von Lilienfeld, O. A. On the role of gradients for machine learning of molecular energies and forces. Mach. Learn.: Sci. Technol., 1(4):045018, 2020.

Christensen, A. S., Bratholm, L. A., Faber, F. A., and Anatole von Lilienfeld, O. FCHL revisited: Faster and more accurate quantum machine learning. J. Chem. Phys, 152(4):044107, 2020.

Cohen, T. S. and Welling, M. Steerable CNNs. International Conference on Learning Representations, 2017.

Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A., and Adams, R. P. Convolutional networks on graphs for learning molecular fingerprints. Advances in Neural Information Processing Systems, 28:2224–2232, 2015.

Gastegger, M., Behler, J., and Marquetand, P. Machine learning molecular dynamics for the simulation of infrared spectra. Chem. Sci., 8(10):6924–6935, 2017.

Gastegger, M., Schütt, K. T., and Müller, K.-R. Machine learning of solvent effects on molecular spectra and reactions. arXiv preprint arXiv:2010.14942, 2020.

Gebauer, N., Gastegger, M., and Schütt, K. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in Neural Information Processing Systems, pp. 7564–7576, 2019.

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chemistry. Interational Conference on Machine Learning, 2017.

Goyal, P. and Ferrara, E. Graph embedding techniques, applications, and performance: A survey. Knowledge-Based Systems, 151:78–94, 2018.

Hegde, G. and Bowen, R. C. Machine-learned approximations to Density Functional Theory Hamiltonians. Sci. Rep., 7(1):1–11, 2017.

Hermann, A., Krawczyk, R. P., Lein, M., Schwerdtfeger, P., Hamilton, I. P., and Stewart, J. J. Convergence of the many-body expansion of interaction potentials: From van der waals to covalent and metallic systems. Physical Review A, 76(1):013202, 2007.

Hermann, J., Schätzle, Z., and Noé, F. Deep-neural-network solution of the electronic Schrödinger equation. Nat. Chem., 12(10):891–897, 2020.

Hinton, G. E., Krizhevsky, A., and Wang, S. D. Transforming auto-encoders. International Conference on Artificial Neural Networks, pp. 44–51, 2011.

Jing, B., Eismann, S., Suriana, P., Townshend, R. J. L., and Dror, R. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=1YLJDvSx6J4.

Kearnes, S., McCloskey, K., Berndl, M., Pande, V., and Riley, P. Molecular graph convolutions: moving beyond fingerprints. J. Comput. Aided Mol. Des., 30(8):595–608, 2016.

Kiefer, J. Simultaneous acquisition of the polarized and depolarized Raman signal with a single detector. Anal. Chem., 89(11):5725–5728, 2017.

Kingma, D. P. and Ba, J. Adam: A method for Stochastic Optimization. arXiv preprint arXiv:1412.6980, 2014.

Klicpera, J., Groß, J., and Günnemann, S. Directional message passing for molecular graphs. International Conference on Learning Representations, 2020a.

Klicpera, J., Groß, J., and Günnemann, S. Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules. Machine Learning for Molecules Workshop at NeurIPS, 2020b.

Köhler, J., Klein, L., and Noé, F. Equivariant Flows: sampling configurations for multi-body systems with symmetric energies. Proceedings of the 37th International Conference on Machine Learning, 2019.

Kondor, R. and Trivedi, S. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. International Conference on Machine Learning, pp. 2747–2755, 2018.

Linstrom, P. and Mallard, W. G. NIST Chemistry WebBook NIST Standard Reference Database Number 69. National Institute of Standards and Technology, doi:10.18434/T4D303, (retrieved September 24, 2020), Gaithersburg MD, 20899, 2020.

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. International Conference on Learning Representations, 2019.

Lu, D., Wang, H., Chen, M., Liu, J., Lin, L., Car, R., Jia, W., Zhang, L., et al. 86 PFLOPS Deep Potential Molecular Dynamics simulation of 100 million atoms with ab initio accuracy. arXiv preprint arXiv:2004.11658, 2020.

Lubbers, N., Smith, J. S., and Barros, K. Hierarchical modeling of molecular energies using a deep neural network. J. Chem. Phys, 148(24):241715, 2018.

Miller, B. K., Geiger, M., Smidt, T. E., and Noé, F. Relevance of rotationally equivariant convolutions for predicting molecular properties. arXiv preprint arXiv:2008.08461, 2020.

Morawietz, T., Singraber, A., Dellago, C., and Behler, J. How van der Waals interactions determine the unique properties of water. Proc. Natl. Acad. Sci., 113(30):8368–8373, 2016.

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.

Pozdnyakov, S. N., Willatt, M. J., Bartók, A. P., Ortner, C., Csányi, G., and Ceriotti, M. Incompleteness of atomic structure representations. Physical Review Letters, 125(16):166001, 2020.

Ramakrishnan, R., Dral, P. O., Rupp, M., and Von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Sci. Data, 1(1):1–7, 2014.

Sauceda, H. E., Vassilev-Galindo, V., Chmiela, S., Müller, K.-R., and Tkatchenko, A. Dynamical strengthening of covalent and non-covalent molecular interactions by nuclear quantum effects at finite temperature. Nat. Commun., 12(1):1–10, 2021.

Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. The graph neural network model. IEEE Trans. Neural Netw., 20(1):61–80, 2008.

Schütt, K., Kindermans, P.-J., Sauceda, H. E., Chmiela, S., Tkatchenko, A., and Müller, K.-R. SchNet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in Neural Information Processing Systems, pp. 991–1001, 2017.

Schütt, K., Kessel, P., Gastegger, M., Nicoli, K., Tkatchenko, A., and Müller, K.-R. SchNetPack: A deep learning toolbox for atomistic systems. J. Chem. Theory Comput., 15(1):448–455, 2018.

Schütt, K., Gastegger, M., Tkatchenko, A., Müller, K.-R., and Maurer, R. J. Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions. Nat. Commun., 10(1):1–10, 2019.

Schütt, K. T., Arbabzadah, F., Chmiela, S., Müller, K. R., and Tkatchenko, A. Quantum-chemical insights from deep tensor neural networks. Nat. Commun., 8, 2017. doi: 10.1038/ncomms13890.

Schütt, K. T., Sauceda, H. E., Kindermans, P.-J., Tkatchenko, A., and Müller, K.-R. SchNet–A deep learning architecture for molecules and materials. J. Chem. Phys, 148(24):241722, 2018.

Simm, G. N., Pinsler, R., Csányi, G., and Hernández-Lobato, J. M. Symmetry-Aware Actor-Critic for 3D Molecular Design. arXiv preprint arXiv:2011.12747, 2020.

Thomas, M., Brehm, M., Fligg, R., Vöhringer, P., and Kirchner, B. Computing vibrational spectra from ab initio molecular dynamics. Phys. Chem. Chem. Phys., 15(18):6608–6622, 2013.

Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-and translation-equivariant neural networks for 3D point clouds. arXiv preprint arXiv:1802.08219, 2018.

Unke, O. T. and Meuwly, M. PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments, and Partial Charges. J. Chem. Theory Comput., 15(6):3678–3693, 2019.

Unke, O. T., Chmiela, S., Sauceda, H. E., Gastegger, M., Poltavsky, I., Schütt, K. T., Tkatchenko, A., and Müller, K.-R. Machine Learning Force Fields. arXiv preprint arXiv:2010.07067, 2020.

Veit, M., Wilkins, D. M., Yang, Y., DiStasio Jr, R. A., and Ceriotti, M. Predicting molecular dipole moments by combining atomic partial charges and atomic dipoles. J. Chem. Phys., 153(2):024113, 2020.

von Lilienfeld, O. A., Müller, K.-R., and Tkatchenko, A. Exploring chemical compound space with quantum-based machine learning. Nat. Rev. Chem., 4(7):347–358, 2020.

Weiler, M., Geiger, M., Welling, M., Boomsma, W., and Cohen, T. S. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. Advances in Neural Information Processing Systems, 31:10381–10392, 2018a.

Weiler, M., Hamprecht, F. A., and Storath, M. Learning Steerable Filters for Rotation Equivariant CNNs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 849–858, 2018b.

Westermayr, J., Gastegger, M., and Marquetand, P. Combining SchNet and SHARC: The SchNarc machine learning approach for excited-state dynamics. J. Phys. Chem. Lett., 11(10):3828–3834, 2020.

Worrall, D. and Brostow, G. CubeNet: Equivariance to 3D Rotation and Translation. Proceedings of the European Conference on Computer Vision (ECCV), pp. 567–584, 2018.