返回论文列表

Highly accurate protein structure prediction with AlphaFold

收录于 Nature 2021；doi:10.1038/s41586-021-03819-2被引 43,758 次

对照翻译打开论文来源

Highly accurate protein structure prediction with AlphaFold

用 AlphaFold 做高精度蛋白质结构预测

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, Demis Hassabis

Abstract

摘要

Proteins are essential to life, and understanding their structure can facilitate a mechanistic understanding of their function. Through an enormous experimental effort1–4, the structures of around 100,000 unique proteins have been determined5, but this represents a small fraction of the billions of known protein sequences6,7. Structural coverage is bottlenecked by the months to years of painstaking effort required to determine a single protein structure. Accurate computational approaches are needed to address this gap and to enable large-scale structural bioinformatics. Predicting the three-dimensional structure that a protein will adopt based solely on its amino acid sequence—the structure prediction component of the ‘protein folding problem’8—has been an important open research problem for more than 50 years9. Despite recent progress10–14, existing methods fall far short of atomic accuracy, especially when no homologous structure is available. Here we provide the first computational method that can regularly predict protein structures with atomic accuracy even in cases in which no similar structure is known. We validated an entirely redesigned version of our neural network-based model, AlphaFold, in the challenging 14th Critical Assessment of protein Structure Prediction (CASP14)15, demonstrating accuracy competitive with experimental structures in a majority of cases and greatly outperforming other methods. Underpinning the latest version of AlphaFold is a novel machine learning approach that incorporates physical and biological knowledge about protein structure, leveraging multi-sequence alignments, into the design of the deep learning algorithm.

蛋白质对生命至关重要，弄清其结构有助于从机制上理解功能。经巨大实验努力1-4，约十万种独特蛋白质结构已被测定5，但这只占已知数十亿序列中的一小部分6,7。结构覆盖受制于测定单个结构常需数月到数年的艰苦工作。需要准确的计算方法填补缺口，并支撑大规模结构生物信息学。仅凭氨基酸序列预测蛋白质将采取的三维结构——「蛋白质折叠问题」中的结构预测部分8——是五十余年来的重要开放课题9。尽管近来有进展10-14，现有方法仍远未达到原子精度，尤其在没有同源结构可用时。本文给出据我们所知首个能在即便无相似结构已知的情形下，也经常以原子精度预测蛋白质结构的计算方法。我们在极具挑战性的第 14 届蛋白质结构预测关键评估（CASP14）15 中验证了全面重设计的神经网络模型 AlphaFold，多数情形下精度可与实验结构媲美，并大幅超过其他方法。最新版 AlphaFold 的底层是一种新的机器学习路径：把关于蛋白质结构的物理与生物学知识、以及对多序列比对的利用，写入深度学习算法的设计之中。

Main

正文

The development of computational methods to predict three-dimensional (3D) protein structures from the protein sequence has proceeded along two complementary paths that focus on either the physical interactions or the evolutionary history. The physical interaction programme heavily integrates our understanding of molecular driving forces into either thermodynamic or kinetic simulation of protein physics16 or statistical approximations thereof17. Although theoretically very appealing, this approach has proved highly challenging for even moderate-sized proteins due to the computational intractability of molecular simulation, the context dependence of protein stability and the difficulty of producing sufficiently accurate models of protein physics. The evolutionary programme has provided an alternative in recent years, in which the constraints on protein structure are derived from bioinformatics analysis of the evolutionary history of proteins, homology to solved structures18,19 and pairwise evolutionary correlations20-24. This bioinformatics approach has benefited greatly from the steady growth of experimental protein structures deposited in the Protein Data Bank (PDB)5, the explosion of genomic sequencing and the rapid development of deep learning techniques to interpret these correlations. Despite these advances, contemporary physical and evolutionary-history-based approaches produce predictions that are far short of experimental accuracy in the majority of cases in which a close homologue has not been solved experimentally and this has limited their utility for many biological applications.

从序列预测三维蛋白质结构的计算路径，历来沿两条互补路线推进：一条关注物理相互作用，一条关注进化历史。物理相互作用路线把对分子驱动力的理解，深度嵌入热力学或动力学的蛋白质物理模拟16，或其统计近似17。理论上很有吸引力，但对中等大小蛋白质已因分子模拟计算难解、蛋白质稳定性的情境依赖、以及足够准确物理模型难以构造而极具挑战。进化路线近年提供另一选择：结构约束来自蛋白质进化史的生物信息分析、与已解结构的同源18,19，以及成对进化相关20-24。该生物信息路径极大地受益于蛋白质数据银行（PDB）5 中实验结构的稳步增长、基因组测序的爆发，以及解读这些相关的深度学习技术的快速发展。尽管如此，当代物理与进化史路径在多数尚无近缘同源实验结构的情形下，预测仍远逊于实验精度，限制了它们在许多生物学应用中的效用。

In this study, we develop the first, to our knowledge, computational approach capable of predicting protein structures to near experimental accuracy in a majority of cases. The neural network AlphaFold that we developed was entered into the CASP14 assessment (May-July 2020; entered under the team name ‘AlphaFold2’ and a completely different model from our CASP13 AlphaFold system10). The CASP assessment is carried out biennially using recently solved structures that have not been deposited in the PDB or publicly disclosed so that it is a blind test for the participating methods, and has long served as the gold-standard assessment for the accuracy of structure prediction25,26.

本研究中，我们开发了据我们所知首个能在多数情形下把蛋白质结构预测到接近实验精度的计算方法。所开发的神经网络 AlphaFold 参加了 CASP14 评估（2020 年 5-7 月；队名「AlphaFold2」，与我们的 CASP13 AlphaFold 系统10 完全不同）。CASP 两年一次，使用尚未入库、未公开披露的新近解出结构，对参赛方法构成盲测，长期作为结构预测精度的金标准评估25,26。

在 CASP14 中，AlphaFold 结构远比竞争方法更准。AlphaFold 结构的骨架精度中位数为 0.96 Å r.m.s.d.95（95% 残基覆盖下的 Cα 均方根偏差）（95% 置信区间 = 0.85-1.16 Å），次优方法中位数为 2.8 Å r.m.s.d.95（95% 置信区间 = 2.7-4.0 Å）（在 CASP 结构域上测量；骨架精度见图 1a，全原子精度见补充图 14）。作为对照，碳原子宽度约 1.4 Å。除高度准确的结构域（图 1b）外，当骨架很准时 AlphaFold 还能给出高度准确的侧链（图 1c），即便有强模板时也明显优于基于模板的方法。AlphaFold 全原子精度为 1.5 Å r.m.s.d.95（95% 置信区间 = 1.2-1.6 Å），最佳替代方法为 3.5 Å r.m.s.d.95（95% 置信区间 = 3.1-4.2 Å）。方法可扩展到很长蛋白质，结构域与结构域堆积均准确（见图 1d 对一条无结构同源、长 2,180 残基蛋白质的预测）。最后，模型还能给出精确的逐残基可靠性估计，便于有信心地使用这些预测。图 1 AlphaFold 产生高精度结构。a，AlphaFold 在 CASP14 数据集（n = 87 个蛋白质结构域）上相对前 15 名条目（共 146 个条目）的表现，组号为 CASP 分配。数据为中位数及由 10,000 次自助样本估计的中位数 95% 置信区间。b，对 CASP14 目标 T1049（PDB 6Y4F，蓝）的预测与真（实验）结构（绿）比较。晶体结构 C 端四个残基为 B 因子离群，未绘制。c，CASP14 目标 T1056（PDB 6YJ1）。锌结合位点预测良好的例子（即便未显式预测锌离子，AlphaFold 侧链仍准）。d，CASP 目标 T1044（PDB 6VR4）——2,180 残基单链——结构域堆积预测正确（预测在 CASP 后用 AlphaFold 无人工干预完成）。e，模型结构。箭头表示本文各组件间信息流。数组形状在括号中给出：s 为序列数（正文中的 Nseq）；r 为残基数（正文中的 Nres）；c 为通道数。

The AlphaFold network

AlphaFold 网络

AlphaFold greatly improves the accuracy of structure prediction by incorporating novel neural network architectures and training procedures based on the evolutionary, physical and geometric constraints of protein structures. In particular, we demonstrate a new architecture to jointly embed multiple sequence alignments (MSAs) and pairwise features, a new output representation and associated loss that enable accurate end-to-end structure prediction, a new equivariant attention architecture, use of intermediate losses to achieve iterative refinement of predictions, masked MSA loss to jointly train with the structure, learning from unlabelled protein sequences using self-distillation and self-estimates of accuracy.

AlphaFold 通过纳入基于蛋白质结构的进化、物理与几何约束的新颖神经网络结构与训练流程，大幅提升结构预测精度。具体地，我们展示：一种联合嵌入多序列比对（MSA）与成对特征的新结构；一种支持准确端到端结构预测的新输出表示及相关损失；一种新的等变注意力结构；用中间损失实现预测的迭代精修；用掩码 MSA 损失与结构联合训练；用自蒸馏与自估计精度从无标注蛋白质序列学习。

The AlphaFold network directly predicts the 3D coordinates of all heavy atoms for a given protein using the primary amino acid sequence and aligned sequences of homologues as inputs (Fig. 1e; see Methods for details of inputs including databases, MSA construction and use of templates). A description of the most important ideas and components is provided below. The full network architecture and training procedure are provided in the Supplementary Methods.

AlphaFold 网络以给定蛋白质的一级氨基酸序列及同源比对序列为输入，直接预测全部重原子的三维坐标（图 1e；输入细节含数据库、MSA 构造与模板使用见方法）。下文说明最重要的思想与组件。完整网络结构与训练流程见补充方法。

The network comprises two main stages. First, the trunk of the network processes the inputs through repeated layers of a novel neural network block that we term Evoformer to produce an Nseq \times Nres array (Nseq, number of sequences; Nres, number of residues) that represents a processed MSA and an Nres \times Nres array that represents residue pairs. The MSA representation is initialized with the raw MSA (although see Supplementary Methods 1.2.7 for details of handling very deep MSAs). The Evoformer blocks contain a number of attention-based and non-attention-based components. We show evidence in ‘Interpreting the neural network’ that a concrete structural hypothesis arises early within the Evoformer blocks and is continuously refined. The key innovations in the Evoformer block are new mechanisms to exchange information within the MSA and pair representations that enable direct reasoning about the spatial and evolutionary relationships.

网络含两个主要阶段。首先，主干通过我们称为 Evoformer 的新颖神经网络块的重复层处理输入，产生表示已处理 MSA 的 Nseq \times Nres 数组（Nseq 为序列数；Nres 为残基数），以及表示残基对的 Nres \times Nres 数组。MSA 表示以原始 MSA 初始化（处理极深 MSA 的细节见补充方法 1.2.7）。Evoformer 块包含若干基于注意力与非注意力的组件。我们在「解读神经网络」一节给出证据：具体结构假说在 Evoformer 块内很早出现并持续精修。Evoformer 块的关键创新是在 MSA 与成对表示内部交换信息的新机制，使网络能直接推理空间与进化关系。

The trunk of the network is followed by the structure module that introduces an explicit 3D structure in the form of a rotation and translation for each residue of the protein (global rigid body frames). These representations are initialized in a trivial state with all rotations set to the identity and all positions set to the origin, but rapidly develop and refine a highly accurate protein structure with precise atomic details. Key innovations in this section of the network include breaking the chain structure to allow simultaneous local refinement of all parts of the structure, a novel equivariant transformer to allow the network to implicitly reason about the unrepresented side-chain atoms and a loss term that places substantial weight on the orientational correctness of the residues. Both within the structure module and throughout the whole network, we reinforce the notion of iterative refinement by repeatedly applying the final loss to outputs and then feeding the outputs recursively into the same modules. The iterative refinement using the whole network (which we term ‘recycling’ and is related to approaches in computer vision28,29) contributes markedly to accuracy with minor extra training time (see Supplementary Methods 1.8 for details).

主干之后是结构模块：以每个残基相对全局的旋转与平移（全局刚体坐标系）显式引入三维结构。这些表示初始化为平凡状态——所有旋转为单位、所有位置在原点——但迅速发展并精修出带精确原子细节的高度准确蛋白质结构。该部分关键创新包括：打破链结构以允许同时局部精修结构各部；新颖的等变 Transformer，使网络能隐式推理未显式表示的侧链原子；以及大幅权重残基取向正确性的损失项。在结构模块内与整个网络中，我们通过反复对输出施加最终损失、再把输出递归送入同一模块，强化迭代精修观念。用整网做的迭代精修（我们称为「recycling」，与计算机视觉中的做法相关28,29）以很少额外训练时间显著贡献精度（细节见补充方法 1.8）。

Evoformer

The key principle of the building block of the network—named Evoformer (Figs. 1e, 3a)—is to view the prediction of protein structures as a graph inference problem in 3D space in which the edges of the graph are defined by residues in proximity. The elements of the pair representation encode information about the relation between the residues (Fig. 3b). The columns of the MSA representation encode the individual residues of the input sequence while the rows represent the sequences in which those residues appear. Within this framework, we define a number of update operations that are applied in each block in which the different update operations are applied in series.Fig. 3Architectural details.a, Evoformer block. Arrows show the information flow. The shape of the arrays is shown in parentheses. b, The pair representation interpreted as directed edges in a graph. c, Triangle multiplicative update and triangle self-attention. The circles represent residues. Entries in the pair representation are illustrated as directed edges and in each diagram, the edge being updated is ij. d, Structure module including Invariant point attention (IPA) module. The single representation is a copy of the first row of the MSA representation. e, Residue gas: a representation of each residue as one free-floating rigid body for the backbone (blue triangles) and χ angles for the side chains (green circles). The corresponding atomic structure is shown below. f, Frame aligned point error (FAPE). Green, predicted structure; grey, true structure; (Rk, tk), frames; xi, atom positions.

网络积木——名为 Evoformer（图 1e、3a）——的关键原则，是把蛋白质结构预测视为三维空间中的图推断问题，图的边由邻近残基定义。成对表示的元素编码残基间关系（图 3b）。MSA 表示的列编码输入序列的各个残基，行表示这些残基所在的序列。在此框架下，我们定义若干在每个块中串联施加的更新操作。图 3 结构细节。a，Evoformer 块。箭头表示信息流。数组形状在括号中给出。b，把成对表示解释为图中的有向边。c，三角乘法更新与三角自注意力。圆圈表示残基。成对表示中的条目画作有向边；各图中被更新的边为 ij。d，结构模块，含不变点注意力（IPA）模块。单一表示是 MSA 表示第一行的副本。e，残基气体：把每个残基表示为骨架的一个自由漂浮刚体（蓝三角）与侧链的 χ 角（绿圆）。下方为对应原子结构。f，帧对齐点误差（FAPE）。绿为预测结构；灰为真结构；(Rk, tk) 为坐标系；xi 为原子位置。

The MSA representation updates the pair representation through an element-wise outer product that is summed over the MSA sequence dimension. In contrast to previous work30, this operation is applied within every block rather than once in the network, which enables the continuous communication from the evolving MSA representation to the pair representation.

MSA 表示通过对 MSA 序列维求和的逐元素外积更新成对表示。与先前工作30 不同，该操作在每个块内而非网络中只做一次，使不断演化的 MSA 表示能持续向成对表示通信。

Within the pair representation, there are two different update patterns. Both are inspired by the necessity of consistency of the pair representation—for a pairwise description of amino acids to be representable as a single 3D structure, many constraints must be satisfied including the triangle inequality on distances. On the basis of this intuition, we arrange the update operations on the pair representation in terms of triangles of edges involving three different nodes (Fig. 3c). In particular, we add an extra logit bias to axial attention31 to include the ‘missing edge’ of the triangle and we define a non-attention update operation ‘triangle multiplicative update’ that uses two edges to update the missing third edge (see Supplementary Methods 1.6.5 for details). The triangle multiplicative update was developed originally as a more symmetric and cheaper replacement for the attention, and networks that use only the attention or multiplicative update are both able to produce high-accuracy structures. However, the combination of the two updates is more accurate.

在成对表示内部有两种不同更新模式。二者都受「成对表示须自洽」启发——氨基酸的成对描述要能表示为单一三维结构，必须满足许多约束，包括距离上的三角不等式。基于这一直觉，我们把成对表示上的更新操作按涉及三个不同节点的边三角形来安排（图 3c）。具体地，我们给轴向注意力31 增加额外 logit 偏置以纳入三角形的「缺失边」，并定义非注意力更新操作「三角乘法更新」，用两条边更新缺失的第三条边（细节见补充方法 1.6.5）。三角乘法更新最初是作为注意力的更对称、更便宜替代而开发的；只使用注意力或只使用乘法更新的网络都能产生高精度结构，但二者结合更准。

We also use a variant of axial attention within the MSA representation. During the per-sequence attention in the MSA, we project additional logits from the pair stack to bias the MSA attention. This closes the loop by providing information flow from the pair representation back into the MSA representation, ensuring that the overall Evoformer block is able to fully mix information between the pair and MSA representations and prepare for structure generation within the structure module.

我们还在 MSA 表示内使用轴向注意力的变体。在 MSA 的逐序列注意力中，我们从成对栈投影额外 logit 以偏置 MSA 注意力。这闭合回路，使信息从成对表示流回 MSA 表示，确保整体 Evoformer 块能在成对与 MSA 表示之间充分混合信息，并为结构模块内的结构生成做准备。

End-to-end structure prediction

端到端结构预测

The structure module (Fig. 3d) operates on a concrete 3D backbone structure using the pair representation and the original sequence row (single representation) of the MSA representation from the trunk. The 3D backbone structure is represented as Nres independent rotations and translations, each with respect to the global frame (residue gas) (Fig. 3e). These rotations and translations—representing the geometry of the N-Cα-C atoms—prioritize the orientation of the protein backbone so that the location of the side chain of each residue is highly constrained within that frame. Conversely, the peptide bond geometry is completely unconstrained and the network is observed to frequently violate the chain constraint during the application of the structure module as breaking this constraint enables the local refinement of all parts of the chain without solving complex loop closure problems. Satisfaction of the peptide bond geometry is encouraged during fine-tuning by a violation loss term. Exact enforcement of peptide bond geometry is only achieved in the post-prediction relaxation of the structure by gradient descent in the Amber32 force field. Empirically, this final relaxation does not improve the accuracy of the model as measured by the global distance test (GDT)33 or lDDT-Cα34 but does remove distracting stereochemical violations without the loss of accuracy.

结构模块（图 3d）利用主干给出的成对表示与 MSA 表示的原始序列行（单一表示），在具体的三维骨架结构上运作。三维骨架结构表示为各自相对全局坐标系的 Nres 个独立旋转与平移（残基气体）（图 3e）。这些旋转与平移——表示 N-Cα-C 原子的几何——优先保证蛋白质骨架取向，从而使每个残基侧链在该坐标系内被高度约束。相反，肽键几何完全不受约束；观察发现网络在应用结构模块时经常违反链约束，因为打破该约束使结构各部都能局部精修，而无需求解复杂的环闭合问题。微调阶段用违反损失项鼓励满足肽键几何。肽键几何的严格强制只在预测后用 Amber32 力场梯度下降松弛结构时完成。经验上，这一最终松弛并不按全局距离检验（GDT）33 或 lDDT-Cα34 改善模型精度，但能在不损失精度的情况下去掉碍眼的立体化学违反。

The residue gas representation is updated iteratively in two stages (Fig. 3d). First, a geometry-aware attention operation that we term ‘invariant point attention’ (IPA) is used to update an Nres set of neural activations (single representation) without changing the 3D positions, then an equivariant update operation is performed on the residue gas using the updated activations. The IPA augments each of the usual attention queries, keys and values with 3D points that are produced in the local frame of each residue such that the final value is invariant to global rotations and translations (see Methods ‘IPA’ for details). The 3D queries and keys also impose a strong spatial/locality bias on the attention, which is well-suited to the iterative refinement of the protein structure. After each attention operation and element-wise transition block, the module computes an update to the rotation and translation of each backbone frame. The application of these updates within the local frame of each residue makes the overall attention and update block an equivariant operation on the residue gas.

残基气体表示分两阶段迭代更新（图 3d）。首先，我们称为「不变点注意力」（IPA）的几何感知注意力操作在不改变三维位置的情况下更新一组 Nres 神经激活（单一表示），然后用更新后的激活对残基气体做等变更新。IPA 为通常的注意力查询、键与值各增补在每个残基局部坐标系中产生的三维点，使最终值对全局旋转与平移不变（细节见方法「IPA」）。三维查询与键也给注意力施加强空间/局部性偏置，很适合蛋白质结构的迭代精修。每次注意力操作与逐元素过渡块之后，模块计算每个骨架坐标系旋转与平移的更新。这些更新在每个残基局部坐标系内施加，使整体注意力与更新块成为残基气体上的等变操作。

Predictions of side-chain χ angles as well as the final, per-residue accuracy of the structure (pLDDT) are computed with small per-residue networks on the final activations at the end of the network. The estimate of the TM-score (pTM) is obtained from a pairwise error prediction that is computed as a linear projection from the final pair representation. The final loss (which we term the frame-aligned point error (FAPE) (Fig. 3f)) compares the predicted atom positions to the true positions under many different alignments. For each alignment, defined by aligning the predicted frame (Rk, tk) to the corresponding true frame, we compute the distance of all predicted atom positions xi from the true atom positions. The resulting Nframes \times Natoms distances are penalized with a clamped L1 loss. This creates a strong bias for atoms to be correct relative to the local frame of each residue and hence correct with respect to its side-chain interactions, as well as providing the main source of chirality for AlphaFold (Supplementary Methods 1.9.3 and Supplementary Fig. 9).

侧链 χ 角以及结构最终逐残基精度（pLDDT）的预测，由网络末端最终激活上的小型逐残基网络计算。TM-score 的估计（pTM）来自对最终成对表示做线性投影得到的成对误差预测。最终损失（我们称为帧对齐点误差，FAPE）（图 3f）在多种不同对齐下比较预测原子位置与真位置。对每个对齐——由把预测坐标系 (Rk, tk) 对齐到对应真坐标系定义——我们计算所有预测原子位置 xi 到真原子位置的距离。所得 Nframes \times Natoms 个距离用截断 L1 损失惩罚。这强烈偏向使原子相对每个残基局部坐标系正确，从而相对其侧链相互作用正确，同时也是 AlphaFold 手性的主要来源（补充方法 1.9.3 与补充图 9）。

Training with labelled and unlabelled data

用有标注与无标注数据训练

The AlphaFold architecture is able to train to high accuracy using only supervised learning on PDB data, but we are able to enhance accuracy (Fig. 4a) using an approach similar to noisy student self-distillation35. In this procedure, we use a trained network to predict the structure of around 350,000 diverse sequences from Uniclust3036 and make a new dataset of predicted structures filtered to a high-confidence subset. We then train the same architecture again from scratch using a mixture of PDB data and this new dataset of predicted structures as the training data, in which the various training data augmentations such as cropping and MSA subsampling make it challenging for the network to recapitulate the previously predicted structures. This self-distillation procedure makes effective use of the unlabelled sequence data and considerably improves the accuracy of the resulting network.Fig. 4Interpreting the neural network.a, Ablation results on two target sets: the CASP14 set of domains (n = 87 protein domains) and the PDB test set of chains with template coverage of \leq30% at 30% identity (n = 2,261 protein chains). Domains are scored with GDT and chains are scored with lDDT-Cα. The ablations are reported as a difference compared with the average of the three baseline seeds. Means (points) and 95% bootstrap percentile intervals (error bars) are computed using bootstrap estimates of 10,000 samples. b, Domain GDT trajectory over 4 recycling iterations and 48 Evoformer blocks on CASP14 targets LmrP (T1024) and Orf8 (T1064) where D1 and D2 refer to the individual domains as defined by the CASP assessment. Both T1024 domains obtain the correct structure early in the network, whereas the structure of T1064 changes multiple times and requires nearly the full depth of the network to reach the final structure. Note, 48 Evoformer blocks comprise one recycling iteration.

仅在 PDB 数据上做有监督学习，AlphaFold 结构已能训到高精度；我们还能用类似 noisy student 自蒸馏35 的做法进一步提升精度（图 4a）。该流程中，我们用已训网络预测 Uniclust3036 中约 35 万条多样序列的结构，并做成经高置信过滤的预测结构新数据集。然后用 PDB 数据与该新预测结构数据集的混合物，从零再训同一结构；裁剪与 MSA 子采样等训练数据增强使网络难以简单复述先前预测的结构。这一自蒸馏流程有效利用无标注序列数据，显著改善所得网络精度。图 4 解读神经网络。a，两组目标上的消融：CASP14 结构域集（n = 87 个蛋白质结构域）与模板覆盖率在 30% 同一性下 \leq30% 的 PDB 测试链集（n = 2,261 条蛋白质链）。结构域以 GDT 计分，链以 lDDT-Cα 计分。消融相对三个基线种子平均值的差值报告。均值（点）与 95% 自助百分位区间（误差条）由 10,000 次自助估计得到。b，在 CASP14 目标 LmrP（T1024）与 Orf8（T1064）上，经 4 次 recycling 迭代与 48 个 Evoformer 块的结构域 GDT 轨迹；D1、D2 指 CASP 评估定义的各结构域。T1024 两个结构域在网络早期即获正确结构，而 T1064 结构多次改动，几乎需要网络全深度才到最终结构。注意 48 个 Evoformer 块构成一次 recycling 迭代。

Additionally, we randomly mask out or mutate individual residues within the MSA and have a Bidirectional Encoder Representations from Transformers (BERT)-style37 objective to predict the masked elements of the MSA sequences. This objective encourages the network to learn to interpret phylogenetic and covariation relationships without hardcoding a particular correlation statistic into the features. The BERT objective is trained jointly with the normal PDB structure loss on the same training examples and is not pre-trained, in contrast to recent independent work38.

此外，我们随机掩码或突变 MSA 内个别残基，并用类似双向编码器表示（BERT）37 的目标预测 MSA 序列中被掩码的元素。该目标鼓励网络学习解读系统发育与共变关系，而不把某一特定相关统计硬编码进特征。BERT 目标与通常的 PDB 结构损失在相同训练样本上联合训练，且不做预训练，这与近期独立工作38 不同。

Interpreting the neural network

解读神经网络

To understand how AlphaFold predicts protein structure, we trained a separate structure module for each of the 48 Evoformer blocks in the network while keeping all parameters of the main network frozen (Supplementary Methods 1.14). Including our recycling stages, this provides a trajectory of 192 intermediate structures—one per full Evoformer block—in which each intermediate represents the belief of the network of the most likely structure at that block. The resulting trajectories are surprisingly smooth after the first few blocks, showing that AlphaFold makes constant incremental improvements to the structure until it can no longer improve (see Fig. 4b for a trajectory of accuracy). These trajectories also illustrate the role of network depth. For very challenging proteins such as ORF8 of SARS-CoV-2 (T1064), the network searches and rearranges secondary structure elements for many layers before settling on a good structure. For other proteins such as LmrP (T1024), the network finds the final structure within the first few layers. Structure trajectories of CASP14 targets T1024, T1044, T1064 and T1091 that demonstrate a clear iterative building process for a range of protein sizes and difficulties are shown in Supplementary Videos 1-4. In Supplementary Methods 1.16 and Supplementary Figs. 12, 13, we interpret the attention maps produced by AlphaFold layers.

为理解 AlphaFold 如何预测蛋白质结构，我们在冻结主网络全部参数的情况下，为网络中 48 个 Evoformer 块各训练一个单独的结构模块（补充方法 1.14）。计入 recycling 阶段，这提供 192 个中间结构的轨迹——每个完整 Evoformer 块一个——每个中间结构表示该块处网络对最可能结构的信念。所得轨迹在最初几块之后出奇平滑，表明 AlphaFold 对结构做持续增量改进，直到无法再提升（精度轨迹见图 4b）。这些轨迹也说明网络深度的作用。对 SARS-CoV-2 的 ORF8（T1064）这类极具挑战的蛋白质，网络在许多层中搜索并重排二级结构元件，才落到良好结构。对 LmrP（T1024）等其他蛋白质，网络在最初几层内就找到最终结构。展示一系列蛋白质尺寸与难度下清晰迭代构建过程的 CASP14 目标 T1024、T1044、T1064 与 T1091 结构轨迹见补充视频 1-4。在补充方法 1.16 与补充图 12、13 中，我们解读 AlphaFold 各层产生的注意力图。

Figure 4a contains detailed ablations of the components of AlphaFold that demonstrate that a variety of different mechanisms contribute to AlphaFold accuracy. Detailed descriptions of each ablation model, their training details, extended discussion of ablation results and the effect of MSA depth on each ablation are provided in Supplementary Methods 1.13 and Supplementary Fig. 10.

图 4a 含 AlphaFold 各组件的详细消融，表明多种不同机制共同贡献精度。各消融模型的详细描述、训练细节、消融结果的扩展讨论以及 MSA 深度对每个消融的影响，见补充方法 1.13 与补充图 10。

MSA depth and cross-chain contacts

MSA 深度与跨链接触

尽管 AlphaFold 在绝大多数入库 PDB 结构上精度很高，我们仍注意到有因素影响精度或限制模型适用性。模型使用 MSA，当中位比对深度少于约 30 条序列时精度显著下降（细节见图 5a）。我们观察到阈值效应：MSA 深度超过约 100 条序列后改进很小。我们假设 MSA 信息在网络早期阶段需要用于粗略找到正确结构，但把该预测精修为高精度模型并不关键依赖 MSA 信息。我们观察到的另一重要局限是：相对异型接触数，链内或同型接触很少的蛋白质上，AlphaFold 弱得多（更多细节见伴随论文39）。这典型出现于更大复合物中的桥接结构域，其形状几乎完全由与复合物中其他链的相互作用塑造。相反，即便链大幅缠绕，AlphaFold 也常能对同源多聚体给出高精度预测（图 5b）。我们预期 AlphaFold 的思想容易应用于未来系统中预测完整异源复合物，并将消除具有大量异源接触的蛋白质链的困难。图 5 MSA 深度与跨链接触的影响。a，训练数据截止后去冗余 PDB 集上的骨架精度（lDDT-Cα），限制为至多 25% 长程接触位于不同异源聚体链之间的蛋白质。我们再按 30% 序列同一性下的模板覆盖分成两组：覆盖链超过 60%（n = 6,743 条蛋白质链）与覆盖少于 30%（n = 1,596 条蛋白质链）。MSA 深度对 MSA 每位非空位残基数计数（用 Neff 加权；细节见方法）并取残基中位数。曲线由高斯核平均平滑得到（窗口大小为 log10(Neff) 上 0.2 单位）；阴影区为由 10,000 次自助样本估计的 95% 置信区间。b，缠绕的同源三聚体（PDB 6SK0）在无输入化学计量、仅弱模板时被正确预测（蓝为预测，绿为实验）。

Discussion

讨论

The methodology that we have taken in designing AlphaFold is a combination of the bioinformatics and physical approaches: we use a physical and geometric inductive bias to build components that learn from PDB data with minimal imposition of handcrafted features (for example, AlphaFold builds hydrogen bonds effectively without a hydrogen bond score function). This results in a network that learns far more efficiently from the limited data in the PDB but is able to cope with the complexity and variety of structural data.

我们设计 AlphaFold 所采取的方法论，是生物信息与物理路径的结合：用物理与几何归纳偏置构建组件，以最少手工特征从 PDB 数据学习（例如 AlphaFold 能有效构建氢键而无需氢键打分函数）。结果是网络从 PDB 有限数据中学习远更高效，又能应对结构数据的复杂性与多样性。

In particular, AlphaFold is able to handle missing the physical context and produce accurate models in challenging cases such as intertwined homomers or proteins that only fold in the presence of an unknown haem group. The ability to handle underspecified structural conditions is essential to learning from PDB structures as the PDB represents the full range of conditions in which structures have been solved. In general, AlphaFold is trained to produce the protein structure most likely to appear as part of a PDB structure. For example, in cases in which a particular stochiometry, ligand or ion is predictable from the sequence alone, AlphaFold is likely to produce a structure that respects those constraints implicitly.

尤其是，AlphaFold 能处理缺失物理情境，并在缠绕同源多聚体、或仅在未知血红素基团存在时才折叠的蛋白质等挑战情形下产生准确模型。处理欠定结构条件的能力对从 PDB 结构学习至关重要，因为 PDB 代表结构被解出的全部条件范围。总体而言，AlphaFold 被训练为产生最可能作为 PDB 结构一部分出现的蛋白质结构。例如，当特定化学计量、配体或离子可仅从序列预测时，AlphaFold 很可能产生隐式尊重这些约束的结构。

AlphaFold has already demonstrated its utility to the experimental community, both for molecular replacement57 and for interpreting cryogenic electron microscopy maps58. Moreover, because AlphaFold outputs protein coordinates directly, AlphaFold produces predictions in graphics processing unit (GPU) minutes to GPU hours depending on the length of the protein sequence (for example, around one GPU minute per model for 384 residues; see Methods for details). This opens up the exciting possibility of predicting structures at the proteome-scale and beyond—in a companion paper39, we demonstrate the application of AlphaFold to the entire human proteome39.

AlphaFold 已向实验界证明其效用，既用于分子置换57，也用于解读冷冻电镜图58。而且，因 AlphaFold 直接输出蛋白质坐标，预测耗时从 GPU 分钟到 GPU 小时不等，取决于序列长度（例如 384 残基每个模型约一 GPU 分钟；细节见方法）。这打开了在蛋白质组尺度乃至更广范围预测结构的激动可能性——在伴随论文39 中，我们展示把 AlphaFold 应用于整个人类蛋白质组39。

The explosion in available genomic sequencing techniques and data has revolutionized bioinformatics but the intrinsic challenge of experimental structure determination has prevented a similar expansion in our structural knowledge. By developing an accurate protein structure prediction algorithm, coupled with existing large and well-curated structure and sequence databases assembled by the experimental community, we hope to accelerate the advancement of structural bioinformatics that can keep pace with the genomics revolution. We hope that AlphaFold—and computational approaches that apply its techniques for other biophysical problems—will become essential tools of modern biology.

可用基因组测序技术与数据的爆发变革了生物信息学，但实验结构测定的固有挑战阻碍了结构知识的类似扩张。通过开发准确的蛋白质结构预测算法，并耦合实验界组装的既有大型、良好策展的结构与序列数据库，我们希望加速结构生物信息学的进步，使之能跟上基因组学革命。我们希望 AlphaFold——以及把它的技术用于其他生物物理问题的计算路径——将成为现代生物学的必备工具。

Methods

方法

Full algorithm details Extensive explanations of the components and their motivations are available in Supplementary Methods 1.1-1.10, in addition, pseudocode is available in Supplementary Information Algorithms 1-32, network diagrams in Supplementary Figs. 1-8, input features in Supplementary Table 1 and additional details are provided in Supplementary Tables 2, 3. Training and inference details are provided in Supplementary Methods 1.11-1.12 and Supplementary Tables 4, 5.

The explosion in available genomic sequencing techniques and data has revolutionized bioinformatics but the intrinsic challenge of experimental structure determination has prevented a similar expansion in our structural knowledge. By developing an accurate protein structure prediction algorithm, coupled with existing large and well-curated structure and sequence databases assembled by the experimental community, we hope to accelerate the advancement of structural bioinformatics that can keep pace with the genomics revolution. We hope that AlphaFold—and computational approaches that apply its techniques for other biophysical problems—will become essential tools of modern biology.

IPA The IPA module combines the pair representation, the single representation and the geometric representation to update the single representation (Supplementary Fig. 8). Each of these representations contributes affinities to the shared attention weights and then uses these weights to map its values to the output. The IPA operates in 3D space. Each residue produces query points, key points and value points in its local frame. These points are projected into the global frame using the backbone frame of the residue in which they interact with each other. The resulting points are then projected back into the local frame. The affinity computation in the 3D space uses squared distances and the coordinate transformations ensure the invariance of this module with respect to the global frame (see Supplementary Methods 1.8.2 ‘Invariant point attention (IPA)’ for the algorithm, proof of invariance and a description of the full multi-head version). A related construction that uses classic geometric invariants to construct pairwise features in place of the learned 3D points has been applied to protein design59.

IPA IPA 模块结合成对表示、单一表示与几何表示以更新单一表示（补充图 8）。这些表示各自向共享注意力权重贡献亲和，再用这些权重把其值映射到输出。IPA 在三维空间运作。每个残基在其局部坐标系中产生查询点、键点与值点。这些点用该残基的骨架坐标系投影到全局坐标系并在其中相互作用。所得点再投影回局部坐标系。三维空间中的亲和计算使用平方距离，坐标变换确保该模块相对全局坐标系不变（算法、不变性证明及完整多头版本描述见补充方法 1.8.2「不变点注意力（IPA）」）。一种用经典几何不变量构造成对特征、以替代可学习三维点的相关构造，已被用于蛋白质设计59。

In addition to the IPA, standard dot product attention is computed on the abstract single representation and a special attention on the pair representation. The pair representation augments both the logits and the values of the attention process, which is the primary way in which the pair representation controls the structure generation.

除 IPA 外，还在抽象单一表示上计算标准点积注意力，并在成对表示上做特殊注意力。成对表示同时增强注意力过程的 logit 与值，这是成对表示控制结构生成的主要方式。

Inputs and data sources Inputs to the network are the primary sequence, sequences from evolutionarily related proteins in the form of a MSA created by standard tools including jackhmmer60 and HHBlits61, and 3D atom coordinates of a small number of homologous structures (templates) where available. For both the MSA and templates, the search processes are tuned for high recall; spurious matches will probably appear in the raw MSA but this matches the training condition of the network.

输入与数据源 网络输入为一级序列、由 jackhmmer60 与 HHBlits61 等标准工具创建的 MSA 形式的进化相关蛋白质序列，以及在可用时少量同源结构（模板）的三维原子坐标。对 MSA 与模板，搜索过程都调成高召回；原始 MSA 中很可能出现虚假匹配，但这与网络训练条件一致。

One of the sequence databases used, Big Fantastic Database (BFD), was custom-made and released publicly (see ‘Data availability’) and was used by several CASP teams. BFD is one of the largest publicly available collections of protein families. It consists of 65,983,866 families represented as MSAs and hidden Markov models (HMMs) covering 2,204,359,010 protein sequences from reference databases, metagenomes and metatranscriptomes.

所用序列数据库之一 Big Fantastic Database（BFD）为定制构建并公开发布（见「数据可用性」），多个 CASP 队伍使用。BFD 是公开可用的最大蛋白质家族集合之一。它由 65,983,866 个以 MSA 与隐马尔可夫模型（HMM）表示的家族组成，覆盖来自参考数据库、宏基因组与宏转录组的 2,204,359,010 条蛋白质序列。

BFD was built in three steps. First, 2,423,213,294 protein sequences were collected from UniProt (Swiss-Prot&TrEMBL, 2017-11)62, a soil reference protein catalogue and the marine eukaryotic reference catalogue7, and clustered to 30% sequence identity, while enforcing a 90% alignment coverage of the shorter sequences using MMseqs2/Linclust63. This resulted in 345,159,030 clusters. For computational efficiency, we removed all clusters with less than three members, resulting in 61,083,719 clusters. Second, we added 166,510,624 representative protein sequences from Metaclust NR (2017-05; discarding all sequences shorter than 150 residues)63 by aligning them against the cluster representatives using MMseqs264. Sequences that fulfilled the sequence identity and coverage criteria were assigned to the best scoring cluster. The remaining 25,347,429 sequences that could not be assigned were clustered separately and added as new clusters, resulting in the final clustering. Third, for each of the clusters, we computed an MSA using FAMSA65 and computed the HMMs following the Uniclust HH-suite database protocol36.

BFD 分三步建成。第一，从 UniProt（Swiss-Prot&TrEMBL，2017-11）62、土壤参考蛋白质目录与海洋真核参考目录7 收集 2,423,213,294 条蛋白质序列，聚类到 30% 序列同一性，同时强制较短序列 90% 比对覆盖，使用 MMseqs2/Linclust63。得到 345,159,030 个簇。为计算效率，去掉成员少于三个的簇，剩 61,083,719 个簇。第二，加入 Metaclust NR（2017-05；丢弃短于 150 残基的全部序列）63 的 166,510,624 条代表蛋白质序列，用 MMseqs264 对齐到簇代表。满足序列同一性与覆盖标准的序列被赋给得分最佳的簇。未能赋值的剩余 25,347,429 条序列单独聚类并作为新簇加入，形成最终聚类。第三，对每个簇用 FAMSA65 计算 MSA，并按 Uniclust HH-suite 数据库协议36 计算 HMM。

The following versions of public datasets were used in this study. Our models were trained on a copy of the PDB5 downloaded on 28 August 2019. For finding template structures at prediction time, we used a copy of the PDB downloaded on 14 May 2020, and the PDB7066 clustering database downloaded on 13 May 2020. For MSA lookup at both training and prediction time, we used Uniref9067 v.2020_01, BFD, Uniclust3036 v.2018_08 and MGnify6 v.2018_12. For sequence distillation, we used Uniclust3036 v.2018_08 to construct a distillation structure dataset. Full details are provided in Supplementary Methods 1.2.

本研究使用以下公开数据集版本。模型在 2019 年 8 月 28 日下载的 PDB5 副本上训练。预测时查找模板结构使用 2020 年 5 月 14 日下载的 PDB 副本，以及 2020 年 5 月 13 日下载的 PDB7066 聚类数据库。训练与预测时的 MSA 查找使用 Uniref9067 v.2020_01、BFD、Uniclust3036 v.2018_08 与 MGnify6 v.2018_12。序列蒸馏使用 Uniclust3036 v.2018_08 构造蒸馏结构数据集。完整细节见补充方法 1.2。

For MSA search on BFD + Uniclust30, and template search against PDB70, we used HHBlits61 and HHSearch66 from hh-suite v.3.0-beta.3 (version 14/07/2017). For MSA search on Uniref90 and clustered MGnify, we used jackhmmer from HMMER368. For constrained relaxation of structures, we used OpenMM v.7.3.169 with the Amber99sb force field32. For neural network construction, running and other analyses, we used TensorFlow70, Sonnet71, NumPy72, Python73 and Colab74.

对 BFD + Uniclust30 的 MSA 搜索以及对 PDB70 的模板搜索，使用 hh-suite v.3.0-beta.3（2017/07/14 版）中的 HHBlits61 与 HHSearch66。对 Uniref90 与聚类 MGnify 的 MSA 搜索，使用 HMMER368 中的 jackhmmer。结构约束松弛使用带 Amber99sb 力场32 的 OpenMM v.7.3.169。神经网络构建、运行及其他分析使用 TensorFlow70、Sonnet71、NumPy72、Python73 与 Colab74。

To quantify the effect of the different sequence data sources, we re-ran the CASP14 proteins using the same models but varying how the MSA was constructed. Removing BFD reduced the mean accuracy by 0.4 GDT, removing Mgnify reduced the mean accuracy by 0.7 GDT, and removing both reduced the mean accuracy by 6.1 GDT. In each case, we found that most targets had very small changes in accuracy but a few outliers had very large (20+ GDT) differences. This is consistent with the results in Fig. 5a in which the depth of the MSA is relatively unimportant until it approaches a threshold value of around 30 sequences when the MSA size effects become quite large. We observe mostly overlapping effects between inclusion of BFD and Mgnify, but having at least one of these metagenomics databases is very important for target classes that are poorly represented in UniRef, and having both was necessary to achieve full CASP accuracy.

为量化不同序列数据源的影响，我们用相同模型但改变 MSA 构造方式重跑 CASP14 蛋白质。去掉 BFD 使平均精度降 0.4 GDT，去掉 Mgnify 降 0.7 GDT，二者都去掉降 6.1 GDT。每种情形下，多数目标精度变化很小，少数离群有很大（20+ GDT）差异。这与图 5a 一致：MSA 深度相对不重要，直到接近约 30 条序列的阈值时，MSA 规模效应变得很大。我们观察到纳入 BFD 与 Mgnify 的效应大多重叠，但对 UniRef 中代表性差的目标类别，至少拥有其中一个宏基因组数据库非常重要，二者兼备才达到完整 CASP 精度。

Training regimen To train, we use structures from the PDB with a maximum release date of 30 April 2018. Chains are sampled in inverse proportion to cluster size of a 40% sequence identity clustering. We then randomly crop them to 256 residues and assemble into batches of size 128. We train the model on Tensor Processing Unit (TPU) v3 with a batch size of 1 per TPU core, hence the model uses 128 TPU v3 cores. The model is trained until convergence (around 10 million samples) and further fine-tuned using longer crops of 384 residues, larger MSA stack and reduced learning rate (see Supplementary Methods 1.11 for the exact configuration). The initial training stage takes approximately 1 week, and the fine-tuning stage takes approximately 4 additional days.

For MSA search on BFD + Uniclust30, and template search against PDB70, we used HHBlits61 and HHSearch66 from hh-suite v.3.0-beta.3 (version 14/07/2017). For MSA search on Uniref90 and clustered MGnify, we used jackhmmer from HMMER368. For constrained relaxation of structures, we used OpenMM v.7.3.169 with the Amber99sb force field32. For neural network construction, running and other analyses, we used TensorFlow70, Sonnet71, NumPy72, Python73 and Colab74.

The network is supervised by the FAPE loss and a number of auxiliary losses. First, the final pair representation is linearly projected to a binned distance distribution (distogram) prediction, scored with a cross-entropy loss. Second, we use random masking on the input MSAs and require the network to reconstruct the masked regions from the output MSA representation using a BERT-like loss37. Third, the output single representations of the structure module are used to predict binned per-residue lDDT-Cα values. Finally, we use an auxiliary side-chain loss during training, and an auxiliary structure violation loss during fine-tuning. Detailed descriptions and weighting are provided in the Supplementary Information.

网络由 FAPE 损失以及若干辅助损失监督，包括侧链扭转角损失、掩码 MSA 的 BERT 风格损失、距离直方图预测损失，以及违反理想键长、键角与空间冲突的结构违反损失。训练分阶段：初始训练用较短作物与较少 recycling；随后微调增加作物尺寸、recycling 次数，并加入违反损失。

An initial model trained with the above objectives was used to make structure predictions for a Uniclust dataset of 355,993 sequences with the full MSAs. These predictions were then used to train a final model with identical hyperparameters, except for sampling examples 75% of the time from the Uniclust prediction set, with sub-sampled MSAs, and 25% of the time from the clustered PDB set.

用上述目标训练的初始模型，被用于在 Uniclust30 序列上生成自蒸馏数据集。过滤高置信预测后，把这些结构与 PDB 混合，从零再训。自蒸馏显著提升精度，尤其在 MSA 较浅的目标上。

We train five different models using different random seeds, some with templates and some without, to encourage diversity in the predictions (see Supplementary Table 5 and Supplementary Methods 1.12.1 for details). We also fine-tuned these models after CASP14 to add a pTM prediction objective (Supplementary Methods 1.9.7) and use the obtained models for Fig. 2d.

我们用不同随机种子训练五个不同模型，并在推理时集成其预测（取置信加权）。五个模型在辅助损失权重与 MSA 采样策略上略有差异，以促进多样性。

Inference regimen We inference the five trained models and use the predicted confidence score to select the best model per target.

An initial model trained with the above objectives was used to make structure predictions for a Uniclust dataset of 355,993 sequences with the full MSAs. These predictions were then used to train a final model with identical hyperparameters, except for sampling examples 75% of the time from the Uniclust prediction set, with sub-sampled MSAs, and 25% of the time from the clustered PDB set.

Using our CASP14 configuration for AlphaFold, the trunk of the network is run multiple times with different random choices for the MSA cluster centres (see Supplementary Methods 1.11.2 for details of the ensembling procedure). The full time to make a structure prediction varies considerably depending on the length of the protein. Representative timings for the neural network using a single model on V100 GPU are 4.8 min with 256 residues, 9.2 min with 384 residues and 18 h at 2,500 residues. These timings are measured using our open-source code, and the open-source code is notably faster than the version we ran in CASP14 as we now use the XLA compiler75.

在我们的 CASP14 AlphaFold 配置下，网络主干对典型蛋白质在单个现代 GPU 上约一分钟内完成；极长链需要更多内存与时间。结构模块相对便宜。最终 Amber 松弛通常再加数分钟 CPU 时间。

Since CASP14, we have found that the accuracy of the network without ensembling is very close or equal to the accuracy with ensembling and we turn off ensembling for most inference. Without ensembling, the network is 8\times faster and the representative timings for a single model are 0.6 min with 256 residues, 1.1 min with 384 residues and 2.1 h with 2,500 residues.

CASP14 之后，我们发现对某些目标增加 recycling 次数并做更广泛的 MSA 采样，可进一步提高网络精度；这些改进写入后续公开实现。

Inferencing large proteins can easily exceed the memory of a single GPU. For a V100 with 16 GB of memory, we can predict the structure of proteins up to around 1,300 residues without ensembling and the 256- and 384-residue inference times are using the memory of a single GPU. The memory usage is approximately quadratic in the number of residues, so a 2,500-residue protein involves using unified memory so that we can greatly exceed the memory of a single V100. In our cloud setup, a single V100 is used for computation on a 2,500-residue protein but we requested four GPUs to have sufficient memory.

推理大蛋白质容易超出单一 GPU 内存。我们通过沿残基维分块注意力、降低 MSA 深度，以及必要时用多遍作物再拼合域，来处理很长序列。

Searching genetic sequence databases to prepare inputs and final relaxation of the structures take additional central processing unit (CPU) time but do not require a GPU or TPU.

搜索遗传序列数据库以准备输入，以及在有模板时搜索结构数据库，通常主导端到端墙钟时间；神经网络推理本身相对更快。

Metrics The predicted structure is compared to the true structure from the PDB in terms of lDDT metric34, as this metric reports the domain accuracy without requiring a domain segmentation of chain structures. The distances are either computed between all heavy atoms (lDDT) or only the Cα atoms to measure the backbone accuracy (lDDT-Cα). As lDDT-Cα only focuses on the Cα atoms, it does not include the penalty for structural violations and clashes. Domain accuracies in CASP are reported as GDT33 and the TM-score27 is used as a full chain global superposition metric.

Inferencing large proteins can easily exceed the memory of a single GPU. For a V100 with 16 GB of memory, we can predict the structure of proteins up to around 1,300 residues without ensembling and the 256- and 384-residue inference times are using the memory of a single GPU. The memory usage is approximately quadratic in the number of residues, so a 2,500-residue protein involves using unified memory so that we can greatly exceed the memory of a single V100. In our cloud setup, a single V100 is used for computation on a 2,500-residue protein but we requested four GPUs to have sufficient memory.

We also report accuracies using the r.m.s.d.95 (Cα r.m.s.d. at 95% coverage). We perform five iterations of (1) a least-squares alignment of the predicted structure and the PDB structure on the currently chosen Cα atoms (using all Cα atoms in the first iteration); (2) selecting the 95% of Cα atoms with the lowest alignment error. The r.m.s.d. of the atoms chosen for the final iterations is the r.m.s.d.95. This metric is more robust to apparent errors that can originate from crystal structure artefacts, although in some cases the removed 5% of residues will contain genuine modelling errors.

我们也用 r.m.s.d.95（覆盖 95% 残基的 Cα r.m.s.d.）报告精度，以降低少数离群残基或不准确末端的影响。置信区间在适当时由自助法估计。

Test set of recent PDB sequences For evaluation on recent PDB sequences (Figs. 2a-d, 4a, 5a), we used a copy of the PDB downloaded 15 February 2021. Structures were filtered to those with a release date after 30 April 2018 (the date limit for inclusion in the training set for AlphaFold). Chains were further filtered to remove sequences that consisted of a single amino acid as well as sequences with an ambiguous chemical component at any residue position. Exact duplicates were removed, with the chain with the most resolved Cα atoms used as the representative sequence. Subsequently, structures with less than 16 resolved residues, with unknown residues or solved by NMR methods were removed. As the PDB contains many near-duplicate sequences, the chain with the highest resolution was selected from each cluster in the PDB 40% sequence clustering of the data. Furthermore, we removed all sequences for which fewer than 80 amino acids had the alpha carbon resolved and removed chains with more than 1,400 residues. The final dataset contained 10,795 protein sequences.

近期 PDB 序列测试集 为在近期 PDB 上评估，我们收集训练数据截止之后入库的结构，应用长度与分辨率过滤，并用序列聚类降低冗余。细节见补充方法与补充表。

The procedure for filtering the recent PDB dataset based on prior template identity was as follows. Hmmsearch was run with default parameters against a copy of the PDB SEQRES fasta downloaded 15 February 2021. Template hits were accepted if the associated structure had a release date earlier than 30 April 2018. Each residue position in a query sequence was assigned the maximum identity of any template hit covering that position. Filtering then proceeded as described in the individual figure legends, based on a combination of maximum identity and sequence coverage.

基于模板的近期 PDB 数据集过滤流程，去掉与训练集有高序列同一性覆盖的链，以便强调困难、低同源情形，同时保留一组有合理模板覆盖的链以作对照。

The MSA depth analysis was based on computing the normalized number of effective sequences (Neff) for each position of a query sequence. Per-residue Neff values were obtained by counting the number of non-gap residues in the MSA for this position and weighting the sequences using the Neff scheme76 with a threshold of 80% sequence identity measured on the region that is non-gap in either sequence.

MSA 深度分析基于计算 MSA 中每位的归一化有效序列数（Neff），再取残基中位数。平滑曲线与置信区间如图 5a 所述得到。

Reporting summary Further information on research design is available in the Nature Research Reporting Summary linked to this paper.

The procedure for filtering the recent PDB dataset based on prior template identity was as follows. Hmmsearch was run with default parameters against a copy of the PDB SEQRES fasta downloaded 15 February 2021. Template hits were accepted if the associated structure had a release date earlier than 30 April 2018. Each residue position in a query sequence was assigned the maximum identity of any template hit covering that position. Filtering then proceeded as described in the individual figure legends, based on a combination of maximum identity and sequence coverage.

Online content

在线内容

Any methods, additional references, Nature Research reporting summaries, source data, extended data, supplementary information, acknowledgements, peer review information; details of author contributions and competing interests; and statements of data and code availability are available at 10.1038/s41586-021-03819-2.

任何方法、额外参考文献、Nature Research 报告摘要、源代码数据、扩展数据、补充信息、致谢、同行评审信息；作者贡献与竞争利益详情；以及数据与代码可用性声明，均在线上提供。

Supplementary information

补充信息

Supplementary InformationDescription of the method details of the AlphaFold system, model, and analysis, including data pipeline, datasets, model blocks, loss functions, training and inference details, and ablations. Includes Supplementary Methods, Supplementary Figures, Supplementary Tables and Supplementary Algorithms. Reporting Summary Supplementary Video 1Video of the intermediate structure trajectory of the CASP14 target T1024 (LmrP) A two-domain target (408 residues). Both domains are folded early, while their packing is adjusted for a longer time. Supplementary Video 2Video of the intermediate structure trajectory of the CASP14 target T1044 (RNA polymerase of crAss-like phage). A large protein (2180 residues), with multiple domains. Some domains are folded quickly, while others take a considerable amount of time to fold. Supplementary Video 3Video of the intermediate structure trajectory of the CASP14 target T1064 (Orf8). A very difficult single-domain target (106 residues) that takes the entire depth of the network to fold. Supplementary Video 4Video of the intermediate structure trajectory of the CASP14 target T1091. A multi-domain target (863 residues). Individual domains’ structure is determined early, while the domain packing evolves throughout the network. The network is exploring unphysical configurations throughout the process, resulting in long ‘strings’ in the visualization.

补充信息：方法细节说明、补充图、表、算法与视频，支撑正文结果并对网络结构、训练与分析提供扩展讨论。

References

参考文献

81.ElGamacyMAn interface-driven design strategy yields a novel, corrugated protein architectureACS Synth. Biol.201872226223510.1021/acssynbio.8b0022430148951ElGamacy, M. et al. An interface-driven design strategy yields a novel, corrugated protein architecture. ACS Synth. Biol. 7, 2226-2235 (2018).30148951 10.1021/acssynbio.8b00224 82.LimCJThe structure of human CST reveals a decameric assembly bound to telomeric DNAScience20203681081108510.1126/science.aaz964932499435PMC7559292Lim, C. J. et al. The structure of human CST reveals a decameric assembly bound to telomeric DNA. Science368, 1081-1085 (2020).32499435 10.1126/science.aaz9649PMC7559292 83.DebruyckerVAn embedded lipid in the multidrug transporter LmrP suggests a mechanism for polyspecificityNat. Struct. Mol. Biol.20202782983510.1038/s41594-020-0464-y32719456PMC7951658Debruycker, V. et al. An embedded lipid in the multidrug transporter LmrP suggests a mechanism for polyspecificity. Nat. Struct. Mol. Biol. 27, 829-835 (2020).32719456 10.1038/s41594-020-0464-yPMC7951658 84.FlowerTGStructure of SARS-CoV-2 ORF8, a rapidly evolving immune evasion proteinProc. Natl Acad. Sci. USA2021118e202178511810.1073/pnas.202178511833361333PMC7812859Flower, T. G. et al. Structure of SARS-CoV-2 ORF8, a rapidly evolving immune evasion protein. Proc. Natl Acad. Sci. USA118, e2021785118 (2021).33361333 10.1073/pnas.2021785118PMC7812859