SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
Abstract
摘要
We introduce the SE(3)-Transformer, a variant of the self-attention module for 3D point clouds and graphs, which is equivariant under continuous 3D roto-translations. Equivariance is important to ensure stable and predictable performance in the presence of nuisance transformations of the data input. A positive corollary of equivariance is increased weight-tying within the model. The SE(3)-Transformer leverages the benefits of self-attention to operate on large point clouds and graphs with varying number of points, while guaranteeing SE(3)-equivariance for robustness. We evaluate our model on a toy -body particle simulation dataset, showcasing the robustness of the predictions under rotations of the input. We further achieve competitive performance on two real-world datasets, ScanObjectNN and QM9. In all cases, our model outperforms a strong, non-equivariant attention baseline and an equivariant model without attention.
我们提出 SE(3)-Transformer,这是用于三维点云和图的自注意力模块的一个变体,它对连续的三维旋转-平移是等变的。等变性很重要,因为它保证在输入数据出现干扰变换时,性能仍然稳定、可预期。等变性的一个积极推论,是模型内部的权重绑定增加。SE(3)-Transformer 借助自注意力的优点,可以处理点数可变的大规模点云和图,同时保证 SE(3) 等变以获得稳健性。我们在一个玩具 体粒子模拟数据集上评估该模型,展示预测在输入旋转下的稳健性。我们还在两个真实数据集 ScanObjectNN 和 QM9 上取得了有竞争力的结果。在所有情形中,我们的模型都超过了一个较强的非等变注意力基线,以及一个不使用注意力的等变模型。
1 Introduction
1 引言
Self-attention mechanisms [31] have enjoyed a sharp rise in popularity in recent years. Their relative implementational simplicity coupled with high efficacy on a wide range of tasks such as language modeling [31], image recognition [18], or graph-based problems [32], make them an attractive component to use. However, their generality of application means that for specific tasks, knowledge of existing underlying structure is unused. In this paper, we propose the SE(3)-Transformer shown in Fig. 1, a self-attention mechanism specifically for 3D point cloud and graph data, which adheres to equivariance constraints, improving robustness to nuisance transformations and general performance.
自注意力机制[31]近年迅速流行。它实现相对简单,又在语言建模[31]、图像识别[18]以及基于图的问题[32]等广泛任务上效果很好,因此是很有吸引力的组件。不过,正因为应用面很广,在特定任务上往往没有用上已有的底层结构。本文提出图 1 所示的 SE(3)-Transformer,这是专门用于三维点云和图数据的自注意力机制,它遵守等变约束,从而提高对干扰变换的稳健性,并改善总体性能。
Point cloud data is ubiquitous across many fields, presenting itself in diverse forms such as 3D object scans [29], 3D molecular structures [21], or -body particle simulations [14]. Finding neural structures which can adapt to the varying number of points in an input, while respecting the irregular sampling of point positions, is challenging. Furthermore, an important property is that these structures should be invariant to global changes in overall input pose; that is, 3D translations and rotations of the input point cloud should not affect the output. In this paper, we find that the explicit imposition of equivariance constraints on the self-attention mechanism addresses these challenges. The SE(3)-Transformer uses the self-attention mechanism as a data-dependent filter particularly suited for sparse, non-voxelised point cloud data, while respecting and leveraging the symmetries of the task at hand.
点云数据在许多领域都很常见,形式多样,例如三维物体扫描[29]、三维分子结构[21],或 体粒子模拟[14]。要找到既能适应输入点数变化、又尊重点位置不规则采样的神经结构,并不容易。此外,一个重要性质是这些结构应当对输入整体姿态的全局变化保持不变,也就是说,对输入点云做三维平移和旋转不应当影响输出。本文发现,把等变约束显式加到自注意力机制上,可以应对这些挑战。SE(3)-Transformer 把自注意力当作一种数据依赖的滤波器,特别适合稀疏、非体素化的点云数据,同时又尊重并利用当前任务的对称性。
equal contribution
work done while at the Bosch Center for Artificial Intelligence
34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.
Figure 1: A) Each layer of the SE(3)-Transformer maps from a point cloud to a point cloud (or graph to graph) while guaranteeing equivariance. For classification, this is followed by an invariant pooling layer and an MLP. B) In each layer, for each node, attention is performed. Here, the red node attends to its neighbours. Attention weights (indicated by line thickness) are invariant w.r.t. input rotation.
图 1:A)SE(3)-Transformer 的每一层都把点云映到点云(或把图映到图),同时保证等变。若做分类,后面再接一个不变的池化层和一层 MLP。B)在每一层中,对每个节点做注意力。这里红色节点关注它的邻居。注意力权重(用线的粗细表示)对输入旋转是不变的。
Self-attention itself is a pseudo-linear map between sets of points. It can be seen to consist of two components: input-dependent attention weights and an embedding of the input, called a value embedding. In Fig. 1, we show an example of a molecular graph, where attached to every atom we see a value embedding vector and where the attention weights are represented as edges, with width corresponding to the attention weight magnitude. In the SE(3)-Transformer, we explicitly design the attention weights to be invariant to global pose. Furthermore, we design the value embedding to be equivariant to global pose. Equivariance generalises the translational weight-tying of convolutions. It ensures that transformations of a layer’s input manifest as equivalent transformations of the output. SE(3)-equivariance in particular is the generalisation of translational weight-tying in 2D known from conventional convolutions to roto-translations in 3D. This restricts the space of learnable functions to a subspace which adheres to the symmetries of the task and thus reduces the number of learnable parameters. Meanwhile, it provides us with a richer form of invariance, since relative positional information between features in the input is preserved.
自注意力本身是点集合之间的一种伪线性映射。可以看成由两部分组成:依赖输入的注意力权重,以及对输入的嵌入,称为值嵌入。图 1 给出一个分子图的例子:每个原子上都附着一个值嵌入向量,注意力权重表示成边,宽度对应权重大小。在 SE(3)-Transformer 中,我们把注意力权重显式设计成对整体姿态不变,并把值嵌入设计成对整体姿态等变。等变性推广了卷积中的平移权重绑定。它保证对一层输入做变换,会在输出上体现为等价的变换。特别地,SE(3) 等变是把常规卷积中已知的二维平移权重绑定,推广到三维中的旋转-平移。这把可学习函数限制在遵守任务对称性的子空间里,从而减少可学习参数的数目。与此同时,它提供了一种更丰富的不变性,因为输入中特征之间的相对位置信息被保留下来。
The works closest related to ours are tensor field networks (TFN) [28] and their voxelised equivalent, 3D steerable CNNs [37]. These provide frameworks for building SE(3)-equivariant convolutional networks operating on point clouds. Employing self-attention instead of convolutions has several advantages. (1) It allows a natural handling of edge features extending TFNs to the graph setting. (2) This is one of the first examples of a nonlinear equivariant layer. In Section 3.2, we show our proposed approach relieves the strong angular constraints on the filter compared to TFNs, therefore adding representational capacity. This constraint has been pointed out in the equivariance literature to limit performance severely [36]. Furthermore, we provide a more efficient implementation, mainly due to a GPU accelerated version of the spherical harmonics. The TFN baselines in our experiments leverage this and use significantly scaled up architectures compared to the ones used in [28].
与我们最接近的工作是张量场网络(TFN)[28],以及它的体素化对应物,三维可转向卷积网络[37]。它们提供了在点云上构建 SE(3) 等变卷积网络的框架。用自注意力代替卷积有若干好处。(1) 它可以自然地处理边特征,从而把 TFN 推广到图的设定。(2) 这是非线性等变层的早期例子之一。在第 3.2 节中,我们表明,与 TFN 相比,本文方法放开了滤波器上很强的角向约束,因而增加了表示能力。等变文献已经指出,这一约束会严重限制性能[36]。此外,我们提供了更高效的实现,主要得益于在 GPU 上加速的球谐函数。实验中的 TFN 基线也使用了这一实现,并采用了比 [28] 中显著扩大的结构。
Our contributions are the following:
我们的贡献如下:
- We introduce a novel self-attention mechanism, guaranteeably invariant to global rotations and translations of its input. It is also equivariant to permutations of the input point labels.
- 我们提出一种新的自注意力机制,可以保证对输入的整体旋转和平移不变。它对输入点标签的排列也是等变的。
- We show that the SE(3)-Transformer resolves an issue with concurrent SE(3)-equivariant neural networks, which suffer from angularly constrained filters.
- 我们表明,SE(3)-Transformer 解决了当时 SE(3) 等变神经网络的一个问题,那些网络的滤波器受到角向约束。
- We introduce a Pytorch implementation of spherical harmonics, which is 10x faster than Scipy on CPU and faster on GPU. This directly addresses a bottleneck of TFNs [28]. E.g., for a ScanObjectNN model, we achieve speed up of the forward pass compared to a network built with SH from the lielearn library (see Appendix C).
- 我们给出球谐函数的 Pytorch 实现,在 CPU 上比 Scipy 快 10 倍,在 GPU 上快 。这直接针对 TFN 的一个瓶颈[28]。例如,对一个 ScanObjectNN 模型,与使用 lielearn 库中球谐实现的网络相比,前向过程加快约 (见附录 C)。
- Code available at https://github.com/FabianFuchsML/se3-transformer-public
2 Background And Related Work
2 背景与相关工作
In this section we introduce the relevant background materials on self-attention, graph neural networks, and equivariance. We are concerned with point cloud based machine learning tasks, such as object classification or segmentation. In such a task, we are given a point cloud as input, represented as a collection of coordinate vectors with optional per-point features .
本节介绍自注意力、图神经网络和等变性的相关背景。我们关心基于点云的机器学习任务,例如物体分类或分割。在这类任务中,输入是点云,表示成 个坐标向量 的集合,并可选地带有逐点特征 。
2.1 The Attention Mechanism
2.1 注意力机制
The standard attention mechanism [31] can be thought of as consisting of three terms: a set of query vectors for , a set of key vectors for , and a set of value vectors for , where and are the dimensions of the low dimensional embeddings. We commonly interpret the key and the value as being ‘attached’ to the same point . For a given query , the attention mechanism can be written as
标准注意力机制[31]可以看成由三项组成:一组查询向量 (),一组键向量 (),以及一组值向量 (),其中 和 是低维嵌入的维数。我们通常把键 和值 理解成“附着”在同一个点 上。对给定查询 ,注意力机制可以写成
where we used a softmax as a nonlinearity acting on the weights. In general, the number of query vectors does not have to equal the number of input points [16]. In the case of self-attention the query, key, and value vectors are embeddings of the input features, so
其中我们对权重使用 softmax 作为非线性。一般而言,查询向量的数目不必等于输入点的数目[16]。在自注意力中,查询、键和值向量都是输入特征的嵌入,因此
where are, in the most general case, neural networks [30]. For us, query is associated with a point in the input, which has a geometric location . Thus if we have points, we have possible queries. For query , we say that node attends to all other nodes .
其中 在最一般的情形下是神经网络[30]。对我们来说,查询 与输入中的点 相关联,该点有几何位置 。因此若有 个点,就有 个可能的查询。对查询 ,我们说节点 关注所有其他节点 。
Motivated by a successes across a wide range of tasks in deep learning such as language modeling [31], image recognition [18], graph-based problems [32], and relational reasoning [30, 9], a recent stream of work has applied forms of self-attention algorithms to point cloud data [44, 42, 16]. One such example is the Set Transformer [16]. When applied to object classification on ModelNet40 [41], the input to the Set Transformer are the cartesian coordinates of the points. Each layer embeds this positional information further while dynamically querying information from other points. The final per-point embeddings are downsampled and used for object classification.
受深度学习中语言建模[31]、图像识别[18]、基于图的问题[32]以及关系推理[30, 9]等一系列任务成功的推动,近期有一批工作把各种自注意力算法用到点云数据上[44, 42, 16]。Set Transformer [16] 就是其中一个例子。把它用到 ModelNet40 [41] 上的物体分类时,输入是点的笛卡尔坐标。每一层进一步嵌入这些位置信息,同时动态地从其他点查询信息。最后的逐点嵌入经下采样后用于物体分类。
Permutation equivariance A key property of self-attention is permutation equivariance. Permutations of point labels lead to permutations of the self-attention output. This guarantees the attention output does not depend arbitrarily on input point ordering. Wagstaff et al. [33] recently showed that this mechanism can theoretically approximate all permutation equivariant functions. The SE(3)-transformer is a special case of this attention mechanism, inheriting permutation equivariance. However, it limits the space of learnable functions to rotation and translation equivariant ones.
排列等变性 自注意力的一个关键性质是排列等变。点标签 的排列,会导致自注意力输出也跟着排列。这保证注意力输出不会任意依赖于输入点的次序。Wagstaff 等人[33]最近表明,这一机制在理论上可以逼近所有排列等变函数。SE(3)-Transformer 是这一注意力机制的特例,因而继承了排列等变性。不过,它把可学习函数限制在对旋转和平移等变的那些函数上。
2.2 Graph Neural Networks
2.2 图神经网络
Attention scales quadratically with point cloud size, so it is useful to introduce neighbourhoods: instead of each point attending to all other points, it only attends to its nearest neighbours. Sets with neighbourhoods are naturally represented as graphs. Attention has previously been introduced on graphs under the names of intra-, self-, vertex-, or graph-attention [17, 31, 32, 12, 26]. These methods were unified by Wang et al. [34] with the non-local neural network. This has the simple form
注意力随点云规模二次增长,因此引入邻域是有用的:每个点不再关注所有其他点,而只关注最近邻。带邻域的集合可以自然地表示成图。注意力此前已在图上出现,名称包括 intra-、self-、vertex- 或 graph-attention [17, 31, 32, 12, 26]。这些方法被 Wang 等人[34]用非局部神经网络统一起来。它具有如下简单形式
where and are neural networks and normalises the sum as a function of all features in the neighbourhood . This has a similar structure to attention, and indeed we can see it as performing attention per neighbourhood. While non-local modules do not explicitly incorporate edge-features, it is possible to add them, as done in Veličković et al. [32] and Hoshen [12].
其中 和 是神经网络, 作为邻域 中所有特征的函数,对求和做归一化。这与注意力结构相似,的确可以看成在每个邻域上做注意力。非局部模块本身并不显式纳入边特征,但可以加上它们,正如 Veličković 等人[32]和 Hoshen [12] 所做的那样。
2.3 Equivariance
2.3 等变性
Given a set of transformations for , where is an abstract group, a function is called equivariant if for every there exists a transformation such that
给定一组变换 (),其中 是一个抽象群,函数 称为等变,如果对每个 都存在变换 ,使得
The indices can be considered as parameters describing the transformation. Given a pair , we can solve for the family of equivariant functions satisfying Equation 4. Furthermore, if are linear and the map is also linear, then a very rich and developed theory already exists for finding [6]. In the equivariance literature, deep networks are built from interleaved linear maps and equivariant nonlinearities. In the case of 3D roto-translations it has already been shown that a suitable structure for is a tensor field network [28], explained below. Note that Romero et al. [24] recently introduced a 2D roto-translationally equivariant attention module for pixel-based image data.
指标 可以看成描述变换的参数。给定一对 ,可以求解满足方程 4 的等变函数族 。此外,若 是线性的,并且映射 也是线性的,那么已经有一套非常丰富、成熟的理论可以用来寻找 [6]。在等变文献中,深度网络由交错的线性映射 和等变非线性构成。对三维旋转-平移,已经表明 的一种合适结构是张量场网络[28],下文会说明。注意 Romero 等人[24]最近为基于像素的图像数据引入了一个二维旋转-平移等变的注意力模块。
Group Representations In general, the transformations are called group representations. Formally, a group representation is a map from a group to the set of invertible matrices . Critically is a group homomorphism; that is, it satisfies the following property for all . Specifically for 3D rotations , we have a few interesting properties: 1) its representations are orthogonal matrices, 2) all representations can be decomposed as
群表示 一般而言,变换 称为群表示。形式上看,群表示 是从群 到 可逆矩阵集合 的映射。关键在于 是群同态,也就是对所有 满足 。特别对三维旋转 ,有几条有趣的性质:1) 它的表示是正交矩阵;2) 所有表示都可以分解为
where is an orthogonal, , change-of-basis matrix [5]; each for is a matrix known as a Wigner-D matrix; and the is the direct sum or concatenation of matrices along the diagonal. The Wigner-D matrices are irreducible representations of —think of them as the ‘smallest’ representations possible. Vectors transforming according to (i.e. we set ), are called type- vectors. Type-0 vectors are invariant under rotations and type-1 vectors rotate according to 3D rotation matrices. Note, type- vectors have length . They can be stacked, forming a feature vector transforming according to Eq. (5).
其中 是一个正交的 基变换矩阵[5];每个 ()是一个 的矩阵,称为 Wigner-D 矩阵; 是直和,也就是把矩阵沿对角线拼接。Wigner-D 矩阵是 的不可约表示,可以把它们想成“尽可能小”的表示。按 变换的向量(即取 )称为 type- 向量。type-0 向量在旋转下不变,type-1 向量按三维旋转矩阵旋转。注意,type- 向量的长度是 。它们可以叠在一起,形成按方程 (5) 变换的特征向量 。
The ‘D’ stands for Darstellung, German for representation
这里的 “D” 来自德语 Darstellung,意为表示。
Tensor Field Networks Tensor field networks (TFN) [28] are neural networks, which map point clouds to point clouds under the constraint of SE(3)-equivariance, the group of 3D rotations and translations. For point clouds, the input is a vector field of the form
张量场网络 张量场网络(TFN)[28] 是在 SE(3) 等变约束下把点云映到点云的神经网络,SE(3) 即三维旋转和平移构成的群。对点云而言,输入是如下形式的向量场
where is the Dirac delta function, are the 3D point coordinates and are point features, representing such quantities as atomic number or point identity. For equivariance to be satisfied, the features of a TFN transform under Eq. (5), where . Each is a concatenation of vectors of different types, where a subvector of type- is written . A TFN layer computes the convolution of a continuous-in-space, learnable weight kernel from type- features to type- features. The type- output of the TFN layer at position is
其中 是狄拉克函数, 是三维点坐标, 是点特征,表示原子序数或点身份一类量。为满足等变,TFN 的特征按方程 (5) 变换,且 。每个 是不同类型向量的拼接,其中 type- 的子向量写成 。TFN 层计算一个在空间上连续、可学习的权重核 的卷积,该核把 type- 特征映到 type- 特征。TFN 层在位置 处的 type- 输出是
We can also include a sum over input channels, but we omit it here. Weiler et al. [37], Thomas et al. [28] and Kondor [15] showed that the kernel lies in the span of an equivariant basis . The kernel is a linear combination of these basis kernels, where the th coefficient is a learnable function of the radius . Mathematically this is
我们也可以对输入通道求和,但这里略去。Weiler 等人[37]、Thomas 等人[28]和 Kondor [15] 表明,核 落在一组等变基 的张成空间中。该核是这些基核的线性组合,其中第 个系数是半径 的可学习函数 。数学上写成
Each basis kernel is formed by taking a linear combination of Clebsch-Gordan matrices of shape , where the th linear combination coefficient is the th dimension of the th spherical harmonic . Each basis kernel completely constrains the form of the learned kernel in the angular direction, leaving the only learnable degree of freedom in the radial direction. Note that only when and , which reduces the kernel to a scalar multiplied by the identity, , referred to as self-interaction [28]. As such we can rewrite the TFN layer as
每个基核 由形状为 的 Clebsch-Gordan 矩阵 做线性组合得到,其中第 个线性组合系数是第 个球谐函数 的第 维。每个基核 在角向上完全限制了所学习核的形式,唯一可学习的自由度只剩下径向。注意 仅当 且 时成立,此时核退化为一个标量 乘以单位阵,,称为自我交互[28]。于是可以把 TFN 层改写为
Figure 2: Updating the node features using our equivariant attention mechanism in four steps. A more detailed description, especially of step 2, is provided in the Appendix. Steps 3 and 4 visualise a graph network perspective: features are passed from nodes to edges to compute keys, queries and values, which depend both on features and relative positions in a rotation-equivariant manner.
图 2:用我们的等变注意力机制分四步更新节点特征。更详细的说明,尤其是第 2 步,见附录。第 3、4 步展示了图网络的视角:特征从节点传到边上,用来计算键、查询和值,它们以旋转等变的方式同时依赖于特征和相对位置。
3 Method
3 方法
Here, we present the SE(3)-Transformer. The layer can be broken down into a procedure of steps as shown in Fig. 2, which we describe in the following section. These are the construction of a graph from a point cloud, the construction of equivariant edge functions on the graph, how to propagate SE(3)-equivariant messages on the graph, and how to aggregate them. We also introduce an alternative for the self-interaction layer, which we call attentive self-interaction.
这里我们给出 SE(3)-Transformer。如图 2 所示,该层可以拆成若干步骤,我们将在下面一节描述。这些步骤包括:由点云构造图,在图上构造等变的边函数,如何在图上传播 SE(3) 等变的消息,以及如何聚合这些消息。我们还为自我交互层引入一种替代方案,称为带注意力的自我交互。
3.1 Neighbourhoods
3.1 邻域
Given a point cloud , we first introduce a collection of neighbourhoods , one centered on each point . These neighbourhoods are computed either via the nearest-neighbours methods or may already be defined. For instance, molecular structures have neighbourhoods defined by their bonding structure. Neighbourhoods reduce the computational complexity of the attention mechanism from quadratic in the number of points to linear. The introduction of neighbourhoods converts our point cloud into a graph. This step is shown as Step 1 of Fig. 2.
给定点云 ,我们首先引入一组邻域 ,每个邻域以点 为中心。这些邻域可以用最近邻方法计算,也可能本来就已经给定。例如,分子结构的邻域由其成键结构定义。引入邻域后,注意力机制的计算复杂度从点数的平方降到线性。引入邻域也就把点云变成了图。这一步对应图 2 的第 1 步。
3.2 The SE(3)-Transformer
3.2 SE(3)-Transformer
The SE(3)-Transformer itself consists of three components. These are 1) edge-wise attention weights , constructed to be SE(3)-invariant on each edge , 2) edge-wise SE(3)-equivariant value messages, propagating information between nodes, as found in the TFN convolution of Eq. (7), and 3) a linear/attentive self-interaction layer. Attention is performed on a per-neighbourhood basis as follows:
SE(3)-Transformer 本身由三个部件组成。它们是:1) 逐边的注意力权重 ,构造为在每条边 上对 SE(3) 不变;2) 逐边的 SE(3) 等变值消息,在节点之间传递信息,与方程 (7) 中的 TFN 卷积相同;3) 线性或带注意力的自我交互层。注意力在每个邻域上按如下方式进行:
These components are visualised in Fig. 2. If we remove the attention weights then we have a tensor field convolution, and if we instead remove the dependence of on , we have a conventional attention mechanism. Provided that the attention weights are invariant, Eq. (10) is equivariant to SE(3)-transformations. This is because it is just a linear combination of equivariant value messages. Invariant attention weights can be achieved with a dot-product attention structure shown in Eq. (11). This mechanism consists of a normalised inner product between a query vector at node and a set of key vectors along each edge in the neighbourhood where
这些部件在图 2 中可视化。若去掉注意力权重,就得到张量场卷积;若去掉 对 的依赖,就得到常规注意力机制。只要注意力权重 是不变的,方程 (10) 就对 SE(3) 变换等变。这是因为它不过是等变值消息的线性组合。不变的注意力权重可以用方程 (11) 所示的点积注意力结构得到。该机制由节点 处的查询向量 ,与邻域 中每条边 上的一组键向量 之间的归一化内积组成,其中
is the direct sum, i.e. vector concatenation in this instance. The linear embedding matrices and are of TFN type (c.f. Eq. (8)). The attention weights are invariant for the following reason. If the input features are SO(3)-equivariant, then the query and key vectors are also SE(3)-equivariant, since the linear embedding matrices are of TFN type. The inner product of SO(3)-equivariant vectors, transforming under the same representation is invariant, since if and , then , because of the orthonormality of representations of SO(3), mentioned in the background section. We follow the common practice from the self-attention literature [31, 16], and chosen a softmax nonlinearity to normalise the attention weights to unity, but in general any nonlinear function could be used.
是直和,在这里也就是向量拼接。线性嵌入矩阵 和 属于 TFN 类型(参见方程 (8))。注意力权重 之所以不变,理由如下。若输入特征 是 SO(3) 等变的,则查询 和键向量 也是 SE(3) 等变的,因为线性嵌入矩阵属于 TFN 类型。在同一表示 下变换的 SO(3) 等变向量,其内积是不变的:若 且 ,则 ,这是因为背景一节提到的 SO(3) 表示具有正交性。我们沿用自注意力文献中的常见做法[31, 16],选择 softmax 非线性把注意力权重归一化为 1,但一般而言也可以使用任意非线性函数。
Aside: Angular Modulation The attention weights add extra degrees of freedom to the TFN kernel in the angular direction. This is seen when Eq. (10) is viewed as a convolution with a data-dependent kernel . In the literature, SO(3) equivariant kernels are decomposed as a sum of products of learnable radial functions and non-learnable angular kernels (c.f. Eq. (8)). The fixed angular dependence of is a strange artifact of the equivariance condition in noncommutative algebras and while necessary to guarantee equivariance, it is seen as overconstraining the expressiveness of the kernels. Interestingly, the attention weights introduce a means to modulate the angular profile of , while maintaining equivariance.
插语:角向调制 注意力权重在角向上为 TFN 核增加了额外的自由度。把方程 (10) 看成与数据依赖核 的卷积时,这一点可以看出来。文献中,SO(3) 等变核被分解为可学习径向函数 与不可学习角向核 的乘积之和(参见方程 (8))。 固定的角向依赖,是非交换代数中等变条件的一个奇特产物;虽然为保证等变所必需,但也被视为过度限制了核的表达力。有意思的是,注意力权重 提供了一种在保持等变的同时,调制 角向轮廓的手段。
Channels, Self-interaction Layers, and Non-Linearities Analogous to conventional neural networks, the SE(3)-Transformer can straightforwardly be extended to multiple channels per representation degree , so far omitted for brevity. This sets the stage for self-interaction layers. The attention layer (c.f. Fig. 2 and circles 1 and 2 of Eq. (10)) aggregates information over nodes and input representation degrees . In contrast, the self-interaction layer (c.f. circle 3 of Eq. (10)) exchanges information solely between features of the same degree and within one node—much akin to 1x1 convolutions in CNNs. Self-interaction is an elegant form of learnable skip connection, transporting information from query point in layer to query point in layer . This is crucial since, in the SE(3)-Transformer, points do not attend to themselves. In our experiments, we use two different types of self-interaction layer: (1) linear and (2) attentive, both of the form
通道、自我交互层与非线性 与常规神经网络类似,SE(3)-Transformer 可以直截了当地为每个表示次数 扩展出多个通道,前文为简洁起见略去了这一点。这就为自我交互层做好了准备。注意力层(参见图 2 以及方程 (10) 中的圆圈 1 和 2)在节点和输入表示次数 上聚合信息。相对地,自我交互层(参见方程 (10) 中的圆圈 3)只在同一节点、同一次数的特征之间交换信息,很像 CNN 中的 卷积。自我交互是一种优雅的可学习跳跃连接,把第 层查询点 的信息传到第 层的查询点 。这一点很关键,因为在 SE(3)-Transformer 中,点并不关注自身。实验中我们使用两种自我交互层:(1) 线性的和 (2) 带注意力的,两者都具有如下形式
Linear: Following Schütt et al. [25], output channels are a learned linear combination of input channels using one set of weights per representation degree, shared across all points. As proposed in Thomas et al. [28], this is followed by a norm-based non-linearity.
线性: 沿用 Schütt 等人[25],输出通道是输入通道的可学习线性组合,对每个表示次数使用一组在所有点上共享的权重 。按照 Thomas 等人[28]的建议,后面再接一个基于范数的非线性。
Attentive: We propose an extension of linear self-interaction, attentive self-interaction, combining self-interaction and nonlinearity. We replace the learned scalar weights with attention weights output from an MLP, shown in Eq. (13) ( means concatenation.). These weights are SE(3)-invariant due to the invariance of inner products of features, transforming under the same representation.
带注意力的: 我们提出线性自我交互的一个扩展,即带注意力的自我交互,把自我交互和非线性结合在一起。我们用 MLP 输出的注意力权重替换可学习的标量权重 ,见方程 (13)( 表示拼接)。这些权重是 SE(3) 不变的,因为在同一表示下变换的特征,其内积不变。
3.3 Node and Edge Features
3.3 节点特征与边特征
Point cloud data often has information attached to points (node-features) and connections between points (edge-features), which we would both like to pass as inputs into the first layer of the network. Node information can directly be incorporated via the tensors in Eqs. (6) and (10). For incorporating edge information, note that is part of multiple neighbourhoods. One can replace with in Eq. (10). Now, can carry different information depending on which neighbourhood we are currently performing attention over. In other words, can carry information both about node but also about edge . Alternatively, if the edge information is scalar, it can be incorporated into the weight matrices and as an input to the radial network (see step 2 in Fig. 2).
点云数据常常在点上带有信息(节点特征),在点与点的连接上带有信息(边特征),我们希望把这两类信息都作为网络第一层的输入。节点信息可以直接通过方程 (6) 和 (10) 中的张量 纳入。至于纳入边信息,注意 属于多个邻域。可以在方程 (10) 中把 换成 。此时 可以根据当前正在哪个邻域 上做注意力,而携带不同的信息。换句话说, 既可以携带关于节点 的信息,也可以携带关于边 的信息。另一种做法是,若边信息是标量,可以把它作为径向网络的输入,纳入权重矩阵 和 (见图 2 的第 2 步)。
Table 1: Predicting future locations and velocities in an electron-proton simulation.
| Linear | DeepSet [46] | Tensor Field [28] | Set Transformer [16] | SE(3)-Transformer | ||
|---|---|---|---|---|---|---|
| Position | MSE | 0.0691 | 0.0639 | 0.0151 | 0.0139 | 0.0076 |
| std | — | 0.0086 | 0.0011 | 0.0004 | 0.0002 | |
| — | 0.038 | 0.167 | ||||
| Velocity | MSE | 0.261 | 0.246 | 0.125 | 0.101 | 0.075 |
| std | — | 0.017 | 0.002 | 0.004 | 0.001 | |
| — | 1.11 | 0.37 |
表 1:在电子-质子模拟中预测未来位置和速度。
| Linear | DeepSet [46] | Tensor Field [28] | Set Transformer [16] | SE(3)-Transformer | ||
|---|---|---|---|---|---|---|
| 位置 | MSE | 0.0691 | 0.0639 | 0.0151 | 0.0139 | 0.0076 |
| std | — | 0.0086 | 0.0011 | 0.0004 | 0.0002 | |
| — | 0.038 | 0.167 | ||||
| 速度 | MSE | 0.261 | 0.246 | 0.125 | 0.101 | 0.075 |
| std | — | 0.017 | 0.002 | 0.004 | 0.001 | |
| — | 1.11 | 0.37 |
4 Experiments
4 实验
We test the efficacy of the SE(3)-Transformer on three datasets, each testing different aspects of the model. The -body problem is an equivariant task: rotation of the input should result in rotated predictions of locations and velocities of the particles. Next, we evaluate on a real-world object classification task. Here, the network is confronted with large point clouds of noisy data with symmetry only around the gravitational axis. Finally, we test the SE(3)-Transformer on a molecular property regression task, which shines light on its ability to incorporate rich graph structures. We compare to publicly available, state-of-the-art results as well as a set of our own baselines. Specifically, we compare to the Set-Transformer [16], a non-equivariant attention model, and Tensor Field Networks [28], which is similar to SE(3)-Transformer but does not leverage attention.
我们在三个数据集上检验 SE(3)-Transformer 的效力,每个数据集测试模型的不同方面。 体问题是等变任务:输入旋转后,对粒子位置和速度的预测也应当跟着旋转。接下来,我们在一个真实世界的物体分类任务上评价。这里网络面对的是大规模、带噪声的点云,对称性只绕重力轴存在。最后,我们在分子性质回归任务上测试 SE(3)-Transformer,考察它纳入丰富图结构的能力。我们与公开的、当时最好的结果以及一组自己的基线进行比较。具体来说,对照包括非等变的注意力模型 Set-Transformer [16],以及与 SE(3)-Transformer 类似但不使用注意力的张量场网络[28]。
Similar to [27, 39], we measure the exactness of equivariance by applying uniformly sampled SO(3)-transformations to input and output. The distance between the two, averaged over samples, yields the equivariance error. Note that, unlike in Sosnovik et al. [27], the error is not squared:
与 [27, 39] 类似,我们通过对输入和输出施加均匀采样的 SO(3) 变换,来衡量等变的精确程度。两者之间的距离对样本取平均,得到等变误差。注意,与 Sosnovik 等人[27]不同,这里的误差没有平方:
4.1 N-Body Simulations
4.1 体模拟
In this experiment, we use an adaptation of the dataset from Kipf et al. [14]. Five particles each carry either a positive or a negative charge and exert repulsive or attractive forces on each other. The input to the network is the position of a particle in a specific time step, its velocity, and its charge. The task of the algorithm is then to predict the relative location and velocity 500 time steps into the future. We deliberately formulated this as a regression problem to avoid the need to predict multiple time steps iteratively. Even though it certainly is an interesting direction for future research to combine equivariant attention with, e.g., an LSTM, our goal here was to test our core contribution and compare it to related models. This task sets itself apart from the other two experiments by not being invariant but equivariant: When the input is rotated or translated, the output changes respectively (see Fig. 3).
在这一实验中,我们使用对 Kipf 等人[14]数据集的改编。五个粒子各自带有正电荷或负电荷,彼此施加斥力或引力。网络的输入是某个时间步上粒子的位置、速度和电荷。算法的任务是预测 500 个时间步之后的相对位置和速度。我们有意把它写成回归问题,以避免需要逐步预测多个时间步。尽管把等变注意力与例如 LSTM 结合起来,无疑是有意思的未来研究方向,但这里的目标是检验我们的核心贡献,并与相关模型比较。这一任务与另外两个实验的不同之处在于,它不是不变的而是等变的:输入旋转或平移时,输出会相应地改变(见图 3)。
We trained an SE(3)-Transformer with 4 equivariant layers, each followed by an attentive self-interaction layer (details are provided in the Appendix). Table 1 shows quantitative results. Our model outperforms both an attention-based, but not rotation-equivariant approach (Set Transformer) and a equivariant approach which does not levarage attention (Tensor Field). The equivariance error shows that our approach is indeed fully rotation equivariant up to the precision of the computations.
我们训练了一个带 4 个等变层的 SE(3)-Transformer,每层后面接一个带注意力的自我交互层(细节见附录)。表 1 给出定量结果。我们的模型既超过了基于注意力、但对旋转不等变的方法(Set Transformer),也超过了等变但不使用注意力的方法(Tensor Field)。等变误差表明,我们的方法在计算精度范围内确实是完全旋转等变的。
Figure 3: A model based on conventional self-attention (left) and our rotation-equivariant version (right) predict future locations and velocities in a 5-body problem. The respective left-hand plots show input locations at time step , ground truth locations at , and the respective predictions. The right-hand plots show predicted locations and velocities for rotations of the input in steps of 10 degrees. The dashed curves show the predicted locations of a perfectly equivariant model.
图 3:基于常规自注意力的模型(左)和我们的旋转等变版本(右),在五体问题中预测未来的位置和速度。各自左侧图给出 时的输入位置、 时的真实位置,以及相应的预测。右侧图给出输入按 10 度步进旋转时,预测的位置和速度。虚线给出完全等变模型的预测位置。
Figure 4: ScanObjectNN: -axis shows data augmentation on the test set. The -value corresponds to the maximum rotation around a random axis in the --plane. If both training and test set are not rotated ( in a), breaking the symmetry of the SE(3)-Transformer by providing the -component of the coordinates as an additional, scalar input improves the performance significantly. Interestingly, the model learns to ignore the additional, symmetry-breaking input when the training set presents a rotation-invariant problem (strongly overlapping dark red circles and dark purple triangles in b).
图 4:ScanObjectNN: 轴表示测试集上的数据增强。 值对应绕 - 平面内随机轴的最大旋转角。若训练集和测试集都不旋转(图 a 中 ),把坐标的 分量作为额外的标量输入,从而打破 SE(3)-Transformer 的对称性,会显著提高性能。有意思的是,当训练集呈现的是旋转不变问题时,模型会学会忽略这一额外的、破坏对称性的输入(图 b 中深红色圆点和深紫色三角几乎重合)。
Table 2: Classification accuracy on the 'object only' category of the ScanObjectNN dataset. The performance of the SE(3)-Transformer is averaged over 5 runs (standard deviation 0.7%).
| Tensor Field | DeepSet | SE(3)-Transf. | 3DmFV | Set Transformer | PointNet | SpiderCNN | Tensor Field +z | PointNet++ | SE(3)-Transf.+z | PointCNN | DGCNN | PointGLR | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No. Points | 128 | 1024 | 128 | 1024 | 1024 | 1024 | 1024 | 128 | 1024 | 128 | 1024 | 1024 | 1024 |
| Accuracy | 63.1% | 71.4% | 72.8% | 73.8% | 74.1% | 79.2% | 79.5% | 81.0% | 84.3% | 85.0% | 85.5% | 86.2% | 87.2% |
表 2:在 ScanObjectNN 数据集“仅物体”类别上的分类准确率。SE(3)-Transformer 的性能对 5 次运行取平均(标准差 0.7%)。
| Tensor Field | DeepSet | SE(3)-Transf. | 3DmFV | Set Transformer | PointNet | SpiderCNN | Tensor Field +z | PointNet++ | SE(3)-Transf.+z | PointCNN | DGCNN | PointGLR | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 点数 | 128 | 1024 | 128 | 1024 | 1024 | 1024 | 1024 | 128 | 1024 | 128 | 1024 | 1024 | 1024 |
| 准确率 | 63.1% | 71.4% | 72.8% | 73.8% | 74.1% | 79.2% | 79.5% | 81.0% | 84.3% | 85.0% | 85.5% | 86.2% | 87.2% |
4.2 Real-World Object Classification on ScanObjectNN
4.2 ScanObjectNN 上的真实世界物体分类
Object classification from point clouds is a well-studied task. Interestingly, the vast majority of current neural network methods work on scalar coordinates without incorporating vector specific inductive biases. Some recent works explore rotation invariant point cloud classification [45, 47, 4, 22]. Our method sets itself apart by using roto-translation equivariant layers acting directly on the point cloud without prior projection onto a sphere [22, 45, 7]. This allows for weight-tying and increased sample efficiency while transporting information about local and global orientation through the network - analogous to the translation equivariance of CNNs on 2D images. To test our method, we choose ScanObjectNN, a recently introduced dataset for real-world object classification. The benchmark provides point clouds of 2902 objects across 15 different categories. We only use the coordinates of the points as input and object categories as training labels. We train an SE(3)-Transformer with 4 equivariant layers with linear self-interaction followed by max-pooling and an MLP. Interestingly, the task is not fully rotation invariant, in a statistical sense, as the objects are aligned with respect to the gravity axis. This results in a performance loss when deploying a fully SO(3) invariant model (see Fig. 4a). In other words: when looking at a new object, it helps to know where ‘up’ is. We create an SO(2) invariant version of our algorithm by additionally feeding the -component as an type-0 field and the position as an additional type-1 field (see Appendix). We dub this model SE(3)-Transformer +z. This way, the model can ‘learn’ which symmetries to adhere to by suppressing and promoting different inputs (compare Fig. 4a and Fig. 4b). In Table 2, we compare our model to the current state-of-the-art in object classification. Despite the dataset not playing to the strengths of our model (full SE(3)-invariance) and a much lower number of input points, the performance is competitive with models specifically designed for object classification - PointGLR [23], for instance, is pre-trained on the larger ModelNet40 dataset [41]. For a discussion of performance vs. number of input points used, see Appendix D.1.2.
从点云做物体分类是一项研究充分的任务。有意思的是,当前绝大多数神经网络方法都在标量坐标上工作,而没有纳入向量特有的归纳偏置。一些近期工作探索旋转不变的点云分类[45, 47, 4, 22]。我们的方法与它们的不同之处在于,使用旋转-平移等变层直接作用在点云上,而无需事先投影到球面[22, 45, 7]。这允许权重绑定并提高样本效率,同时把关于局部和整体取向的信息在网络中传递下去,类似于 CNN 在二维图像上的平移等变性。为测试我们的方法,我们选择 ScanObjectNN,这是一个最近提出的真实世界物体分类数据集。该基准提供跨越 15 个不同类别的 2902 个物体的点云。我们只把点的坐标作为输入,把物体类别作为训练标签。我们训练一个带 4 个等变层的 SE(3)-Transformer,使用线性自我交互,再接最大化池化和一层 MLP。有意思的是,这一任务在统计意义上并非完全旋转不变,因为物体是相对于重力轴对齐的。因此,部署完全 SO(3) 不变的模型会带来性能损失(见图 4a)。换句话说:看到一个新物体时,知道哪一边是“上”是有帮助的。我们通过额外把 分量作为 type-0 场、把 位置作为额外的 type-1 场输入,构造了算法的 SO(2) 不变版本(见附录)。我们把这一模型称为 SE(3)-Transformer +z。这样,模型可以通过抑制或提升不同输入,来‘学习’应当遵守哪些对称性(比较图 4a 和图 4b)。在表 2 中,我们把模型与当时物体分类的最好方法进行比较。尽管该数据集并不发挥我们模型的长处(完全的 SE(3) 不变),而且输入点数少得多,其性能仍能与专门为物体分类设计的模型相竞争。例如 PointGLR [23] 是在更大的 ModelNet40 数据集[41]上预训练的。关于性能与所用输入点数的讨论,见附录 D.1.2。
At time of submission, PointGLR was a recently published preprint [23]. The performance of the following models was taken from the official benchmark of the dataset as of June 4th, 2020 (https://hkust-vgd.github.io/benchmark/): 3DmFV [3], PointNet [19], SpiderCNN [43], PointNet++ [20], DGCNN [35].
4.3 QM9
The QM9 regression dataset [21] is a publicly available chemical property prediction task. There are 134k molecules with up to 29 atoms per molecule. Atoms are represented as a 5 dimensional one-hot node embeddings in a molecular graph connected by 4 different chemical bond types (more details in Appendix). ‘Positions’ of each atom are provided. We show results on the test set of Anderson et al. [1] for 6 regression tasks in Table 3. Lower is better. The table is split into non-equivariant (top) and equivariant models (bottom). Our nearest neighbors are Cormorant and TFN (own implementation). We see that while not state-of-the-art, we offer competitive performance, especially against Cormorant and TFN, which transform under irreducible representations of SE(3) (like us), unlike LieConv(T3), using a left-regular representation of SE(3), which may explain its success.
QM9 回归数据集[21]是一个公开的化学性质预测任务。其中有 13.4 万个分子,每个分子最多 29 个原子。原子在分子图中表示成 5 维独热节点嵌入,由 4 种不同的化学键类型连接(更多细节见附录)。每个原子的“位置”是给定的。我们在表 3 中给出 Anderson 等人[1]测试集上 6 项回归任务的结果。数值越低越好。表格分成非等变模型(上)和等变模型(下)。与我们最近的对照是 Cormorant 和 TFN(我们自己的实现)。可以看到,尽管不是当时最好,我们仍给出了有竞争力的结果,尤其是相对于同样在 SE(3) 不可约表示下变换的 Cormorant 和 TFN。这与使用 SE(3) 左正则表示的 LieConv(T3) 不同,后者的成功或许可以由此解释。
Table 3: QM9 Mean Absolute Error. Top: Non-equivariant models. Bottom: Equivariant models. SE(3)-Tr. is averaged over 5 runs.
| TASK | ||||||
|---|---|---|---|---|---|---|
| UNITS | bohr | meV | meV | meV | D | cal/mol K |
| WaveScatt [11] | .160 | 118 | 85 | 76 | .340 | .049 |
| NMP [10] | .092 | 69 | 43 | 38 | .030 | .040 |
| SchNet [25] | .235 | 63 | 41 | 34 | .033 | .033 |
| Cormorant [1] | .085 | 61 | 34 | 38 | .038 | .026 |
| LieConv(T3) [8] | .084 | 49 | 30 | 25 | .032 | .038 |
| TFN [28] | .223 | 58 | 40 | 38 | .064 | .101 |
| SE(3)-Transformer |
表 3:QM9 平均绝对误差。上:非等变模型。下:等变模型。SE(3)-Tr. 对 5 次运行取平均。
| 任务 | ||||||
|---|---|---|---|---|---|---|
| 单位 | bohr | meV | meV | meV | D | cal/mol K |
| WaveScatt [11] | .160 | 118 | 85 | 76 | .340 | .049 |
| NMP [10] | .092 | 69 | 43 | 38 | .030 | .040 |
| SchNet [25] | .235 | 63 | 41 | 34 | .033 | .033 |
| Cormorant [1] | .085 | 61 | 34 | 38 | .038 | .026 |
| LieConv(T3) [8] | .084 | 49 | 30 | 25 | .032 | .038 |
| TFN [28] | .223 | 58 | 40 | 38 | .064 | .101 |
| SE(3)-Transformer |
5 Conclusion
5 结论
We have presented an attention-based neural architecture designed specifically for point cloud data. This architecture is guaranteed to be robust to rotations and translations of the input, obviating the need for training time data augmentation and ensuring stability to arbitrary choices of coordinate frame. The use of self-attention allows for anisotropic, data-adaptive filters, while the use of neighbourhoods enables scalability to large point clouds. We have also introduced the interpretation of the attention mechanism as a data-dependent nonlinearity, adding to the list of equivariant nonlinearities which we can use in equivariant networks. Furthermore, we provide code for a speed up of spherical harmonics computation of up to 3 orders of magnitudes. This speed-up allowed us to train significantly larger versions of both the SE(3)-Transformer and the Tensor Field network [28] and to apply these models to real-world datasets.
我们提出了一种专门为点云数据设计的、基于注意力的神经结构。这一结构保证对输入的旋转和平移稳健,从而不必在训练时做数据增强,并保证对任意坐标系选择保持稳定。使用自注意力可以得到各向异性、随数据自适应的滤波器,而使用邻域则使模型可以扩展到大规模点云。我们还把注意力机制解释成一种数据依赖的非线性,从而扩充了等变网络可以使用的等变非线性清单。此外,我们提供的代码可以把球谐函数的计算加快最多 3 个数量级。这一加速使我们能够训练显著更大的 SE(3)-Transformer 和张量场网络[28],并把这些模型用到真实数据集上。
Our experiments showed that adding attention to a roto-translation-equivariant model consistently led to higher accuracy and increased training stability. Specifically for large neighbourhoods, attention proved essential for model convergence. On the other hand, compared to conventional attention, adding the equivariance constraints also increases performance in all of our experiments while at the same time providing a mathematical guarantee for robustness with respect to rotations of the input data.
实验表明,把注意力加到旋转-平移等变模型上,会稳定地带来更高的准确率和更好的训练稳定性。特别是在邻域较大时,注意力对模型收敛至关重要。另一方面,与常规注意力相比,加上等变约束也在我们的全部实验中提高了性能,同时又为输入数据旋转下的稳健性提供了数学保证。
Broader Impact
更广泛的影响
The main contribution of the paper is a mathematically motivated attention mechanism which can be used for deep learning on point cloud based problems. We do not see a direct potential of negative impact to the society. However, we would like to stress that this type of algorithm is inherently suited for classification and regression problems on molecules. The SE(3)-Transformer therefore lends itself to application in drug research. One concrete application we are currently investigating is to use the algorithm for early-stage suitability classification of molecules for inhibiting the reproductive cycle of the coronavirus. While research of this sort always requires extensive testing in wet labs, computer algorithms can be and are being used to filter out particularly promising compounds from large databases of millions of molecules.
本文的主要贡献是一种有数学动机的注意力机制,可用于基于点云的深度学习问题。我们没有看到它对社会有直接的负面影响。不过我们想强调,这类算法本质上适合分子上的分类和回归问题。因此 SE(3)-Transformer 可以用于药物研究。我们目前正在考察的一个具体应用,是用该算法对分子做早期适用性分类,看它们是否适合抑制冠状病毒的繁殖周期。尽管这类研究始终需要在湿实验室中进行广泛测试,但计算机算法可以、并且已经被用来从数百万分子的大型数据库中筛出特别有希望的化合物。
Acknowledgements and Funding Disclosure
致谢与资助声明
We would like to express our gratitude to the Bosch Center for Artificial Intelligence and Koninklijke Philips N.V. for funding our work and contributing to open research by publishing our paper. Fabian Fuchs worked on this project while on a research sabbatical at the Bosch Center for Artificial Intelligence. His PhD is funded by the EPSRC AIMS Centre for Doctoral Training at Oxford University.
我们感谢 Bosch Center for Artificial Intelligence 与 Koninklijke Philips N.V. 资助我们的工作,并通过发表本文为开放研究做出贡献。Fabian Fuchs 是在 Bosch Center for Artificial Intelligence 做研究休假期间参与本项目的。他的博士由牛津大学 EPSRC AIMS Centre for Doctoral Training 资助。
References
参考文献
[1] Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. In Advances in Neural Information Processing Systems (NeurIPS). 2019.
[2] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv Preprint, 2016.
[3] Yizhak Ben-Shabat, Michael Lindenbaum, and Anath Fischer. 3dmfv: Three-dimensional point cloud classification in realtime using convolutional neural networks. IEEE Robotics and Automation Letters, 2018.
[4] Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Meng Wang, and Liang Lin. ClusterNet: Deep Hierarchical Cluster Network with Rigorously Rotation-Invariant Representation for Point Cloud Analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[5] Gregory S Chirikjian, Alexander B Kyatkin, and AC Buckingham. Engineering applications of noncommutative harmonic analysis: with emphasis on rotation and motion groups. Appl. Mech. Rev., 54(6):B97–B98, 2001.
[6] Taco S. Cohen and Max Welling. Steerable cnns. International Conference on Learning Representations (ICLR), 2017.
[7] Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling. Spherical cnns. In International Conference on Learning Representations (ICLR), 2018.
[8] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. Proceedings of the International Conference on Machine Learning, ICML, 2020.
[9] Fabian B. Fuchs, Adam R. Kosiorek, Li Sun, Oiwi Parker Jones, and Ingmar Posner. End-to-end recurrent multi-object tracking and prediction with relational reasoning. arXiv preprint, 2020.
[10] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, and Oriol Vinyals George E. Dahl. Neural message passing for quantum chemistry. Proceedings of the International Conference on Machine Learning, ICML, 2020.
[11] Matthew J. Hirn, Stéphane Mallat, and Nicolas Poilvert. Wavelet scattering regression of quantum chemical energies. Multiscale Model. Simul., 15(2):827–863, 2017.
[12] Yedid Hoshen. Vain: Attentional multi-agent predictive modeling. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[13] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, ICLR, 2015.
[14] Thomas N. Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard S. Zemel. Neural relational inference for interacting systems. In Proceedings of the International Conference on Machine Learning, ICML, 2018.
[15] Risi Kondor. N-body networks: a covariant hierarchical neural network architecture for learning atomic potentials. arXiv preprint, 2018.
[16] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning, ICML, 2019.
[17] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. International Conference on Learning Representations (ICLR), 2017.
[18] Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In Advances in Neural Information Processing System (NeurIPS), 2019.
[19] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[20] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[21] Raghunathan Ramakrishnan, Pavlo Dral, Matthias Rupp, and Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 08 2014.
[22] Yongming Rao, Jiwen Lu, and Jie Zhou. Spherical fractal convolutional neural networks for point cloud recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[23] Yongming Rao, Jiwen Lu, and Jie Zhou. Global-local bidirectional reasoning for unsupervised representation learning of 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[24] David W. Romero, Erik J. Bekkers, Jakub M. Tomczak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. Proceedings of the International Conference on Machine Learning (ICML), 2020.
[25] K. T. Schütt, P.-J. Kindermans, H. E. Sauceda, S. Chmiela1, A. Tkatchenko, and K.-R. Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[26] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), 2018.
[27] Ivan Sosnovik, Michał Szmaja, and Arnold Smeulders. Scale-equivariant steerable networks. International Conference on Learning Representations (ICLR), 2020.
[28] Nathaniel Thomas, Tess Smidt, Steven M. Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotationand translation-equivariant neural networks for 3d point clouds. ArXiv Preprint, 2018.
[29] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In International Conference on Computer Vision (ICCV), 2019.
[30] Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. International Conference on Learning Representations (ICLR), 2018.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[32] Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations (ICLR), 2018.
[33] Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, and Michael A. Osborne. On the limitations of representing functions on sets. International Conference on Machine Learning (ICML), 2019.
[34] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[35] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG), 2019.
[36] Maurice Weiler and Gabriele Cesa. General E(2)-Equivariant Steerable CNNs. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
[37] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[38] Marysia Winkels and Taco S. Cohen. 3d g-cnns for pulmonary nodule detection. 1st Conference on Medical Imaging with Deep Learning (MIDL), 2018.
[39] Daniel E. Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[40] Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[41] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[42] Saining Xie, Sainan Liu, and Zeyu Chen Zhuowen Tu. Attentional shapecontextnet for point cloud recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[43] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spidercnn: Deep learning on point sets with parameterized convolutional filters. European Conference on Computer Vision (ECCV), 2018.
[44] Jiancheng Yang, Qiang Zhang, and Bingbing Ni. Modeling point clouds with self-attention and gumbel subset sampling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[45] Yang You, Yujing Lou, Qi Liu, Yu-Wing Tai, Lizhuang Ma, Cewu Lu, and Weiming Wang. Pointwise Rotation-Invariant Network with Adaptive Sampling and 3D Spherical Voxel Convolution. In AAAI Conference on Artificial Intelligence, 2019.
[46] Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander Smola. Deep Sets. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[47] Zhiyuan Zhang, Binh-Son Hua, David W. Rosen, and Sai-Kit Yeung. Rotation invariant convolutions for 3d point clouds deep learning. In International Conference on 3D Vision (3DV), 2019.



