Neural Networks and Physical Systems with Emergent Collective Computational Abilities

Proc. Natl. Acad. Sci. USA, Vol. 79, pp. 2554–2558, April 1982, Biophysics

《美国国家科学院院刊》（Proc. Natl. Acad. Sci. USA）第 79 卷，第 2554–2558 页，1982 年 4 月，生物物理学

Neural networks and physical systems with emergent collective computational abilities

具有涌现集体计算能力的神经网络与物理系统

(associative memory/parallel processing/categorization/content-addressable memory/fail-soft devices)

（联想记忆／并行处理／归类／内容寻址存储器／软失效器件）

J. J. Hopfield

Division of Chemistry and Biology, California Institute of Technology, Pasadena, California 91125; and Bell Laboratories, Murray Hill, New Jersey 07974

Contributed by John J. Hopfield, January 15, 1982

J. J. Hopfield

加州理工学院化学与生物学部，加利福尼亚州帕萨迪纳 91125；贝尔实验室，新泽西州默里山 07974

由 John J. Hopfield 提交，1982 年 1 月 15 日

Abstract

摘要

Computational properties of use to biological organisms or to the construction of computers can emerge as collective properties of systems having a large number of simple equivalent components (or neurons). The physical meaning of content-addressable memory is described by an appropriate phase space flow of the state of a system. A model of such a system is given, based on aspects of neurobiology but readily adapted to integrated circuits. The collective properties of this model produce a content-addressable memory which correctly yields an entire memory from any subpart of sufficient size. The algorithm for the time evolution of the state of the system is based on asynchronous parallel processing. Additional emergent collective properties include some capacity for generalization, familiarity recognition, categorization, error correction, and time sequence retention. The collective properties are only weakly sensitive to details of the modeling or the failure of individual devices.

对生物体或计算机构造有用的计算性质，可以作为由大量简单、等同的组件（或神经元）构成的系统的集体性质而涌现出来。内容寻址存储器的物理含义，可以用系统状态在相空间中的适当流动来描述。本文给出这样一个系统的模型，它依据神经生物学的某些方面，同时很容易改造为集成电路。该模型的集体性质产生了一种内容寻址存储器：只要给出足够大的任意一部分，它就能正确给出整条记忆。系统状态随时间演化的算法基于异步并行处理。其他涌现的集体性质还包括一定的泛化能力、熟悉度识别、归类、纠错以及时间序列的保持。这些集体性质对建模细节和个别器件的失效只有很弱的敏感性。

Given the dynamical electrochemical properties of neurons and their interconnections (synapses), we readily understand schemes that use a few neurons to obtain elementary useful biological behavior (1–3). Our understanding of such simple circuits in electronics allows us to plan larger and more complex circuits which are essential to large computers. Because evolution has no such plan, it becomes relevant to ask whether the ability of large collections of neurons to perform "computational" tasks may in part be a spontaneous collective consequence of having a large number of interacting simple neurons.

知道了神经元及其相互连接（突触）的动态电化学性质，我们不难理解那些用少数几个神经元实现基本而有用的生物行为的方案（1–3）。我们对电子学中这类简单电路的理解，使我们能够规划大型计算机所必需的更大、更复杂的电路。进化并没有这样的规划，因此值得一问：大量神经元执行“计算”任务的能力，是否部分是大量相互作用的简单神经元自发产生的集体结果？

In physical systems made from a large number of simple elements, interactions among large numbers of elementary components yield collective phenomena such as the stable magnetic orientations and domains in a magnetic system or the vortex patterns in fluid flow. Do analogous collective phenomena in a system of simple interacting neurons have useful "computational" correlates? For example, are the stability of memories, the construction of categories of generalization, or time-sequential memory also emergent properties and collective in origin? This paper examines a new modeling of this old and fundamental question (4–8) and shows that important computational properties spontaneously arise.

在由大量简单单元构成的物理系统中，大量基本组件之间的相互作用会产生集体现象，例如磁性系统中稳定的磁化取向和磁畴，或流体中的涡旋图样。由简单神经元相互作用构成的系统中，类似的集体现象是否对应着有用的“计算”功能？例如，记忆的稳定性、泛化类别的构建、时间序列记忆，是否也是涌现性质、源于集体行为？本文对这个古老而基本的问题（4–8）提出一种新的建模，并表明重要的计算性质会自发出现。

All modeling is based on details, and the details of neuroanatomy and neural function are both myriad and incompletely known (9). In many physical systems, the nature of the emergent collective properties is insensitive to the details inserted in the model (e.g., collisions are essential to generate sound waves, but any reasonable interatomic force law will yield appropriate collisions). In the same spirit, I will seek collective properties that are robust against change in the model details.

一切建模都建立在细节之上，而神经解剖和神经功能的细节既繁多又尚未完全弄清（9）。在许多物理系统中，涌现的集体性质对模型中放入的细节并不敏感（例如，产生声波离不开碰撞，但任何合理的原子间作用力定律都能给出合适的碰撞）。本着同样的思路，我将寻找对模型细节变化保持稳健的集体性质。

The model could be readily implemented by integrated circuit hardware. The conclusions suggest the design of a delocalized content-addressable memory or categorizer using extensive asynchronous parallel processing.

这个模型很容易用集成电路硬件实现。结论提示了一种设计：利用大规模异步并行处理，构建非局域化的内容寻址存储器或归类器。

The publication costs of this article were defrayed in part by page charge payment. This article must therefore be hereby marked "advertisement" in accordance with 18 U. S. C. §1734 solely to indicate this fact.

本文的出版费用部分由版面费支付。因此，依据《美国法典》第 18 编第 1734 条，本文须在此标注为“广告”，仅用以说明这一事实。

The general content-addressable memory of a physical system

物理系统的一般内容寻址存储器

Suppose that an item stored in memory is "H. A. Kramers & G. H. Wannier Phys. Rev. 60, 252 (1941)." A general content-addressable memory would be capable of retrieving this entire memory item on the basis of sufficient partial information. The input "& Wannier, (1941)" might suffice. An ideal memory could deal with errors and retrieve this reference even from the input "Vannier, (1941)". In computers, only relatively simple forms of content-addressable memory have been made in hardware (10, 11). Sophisticated ideas like error correction in accessing information are usually introduced as software (10).

假设存储器中存有一条记录：“H. A. Kramers & G. H. Wannier Phys. Rev. 60, 252 (1941).” 一般的内容寻址存储器应能凭足够的部分信息取回整条记录，输入“& Wannier, (1941)”或许就够了。理想的存储器还能处理错误，即使输入的是“Vannier, (1941)”，也能取回这条文献。在计算机中，硬件只实现过比较简单的内容寻址存储器（10, 11）；访问信息时的纠错等复杂想法，通常以软件形式引入（10）。

There are classes of physical systems whose spontaneous behavior can be used as a form of general (and error-correcting) content-addressable memory. Consider the time evolution of a physical system that can be described by a set of general coordinates. A point in state space then represents the instantaneous condition of the system. This state space may be either continuous or discrete (as in the case of $N$ Ising spins).

有几类物理系统，它们的自发行为可以用作一般的（且能纠错的）内容寻址存储器。考虑一个可以用一组广义坐标描述的物理系统随时间的演化，状态空间中的一个点就代表系统的瞬时状况。这个状态空间可以是连续的，也可以是离散的（如 $N$ 个伊辛自旋的情形）。

The equations of motion of the system describe a flow in state space. Various classes of flow patterns are possible, but the systems of use for memory particularly include those that flow toward locally stable points from anywhere within regions around those points. A particle with frictional damping moving in a potential well with two minima exemplifies such a dynamics.

系统的运动方程描述了状态空间中的一种流动。可能的流动模式有许多类，而可用作记忆的系统，尤其包括那些从局部稳定点周围区域内任意位置都会流向该点的系统。在有两个极小值的势阱中运动、带摩擦阻尼的粒子，就是这种动力学的一个例子。

If the flow is not completely deterministic, the description is more complicated. In the two-well problems above, if the frictional force is characterized by a temperature, it must also produce a random driving force. The limit points become small limiting regions, and the stability becomes not absolute. But as long as the stochastic effects are small, the essence of local stable points remains.

如果流动不是完全确定性的，描述就更复杂。在上面的双势阱问题中，如果摩擦力用一个温度来刻画，它就必然同时产生随机驱动力。极限点变成了小的极限区域，稳定性也不再是绝对的。但只要随机效应很小，局部稳定点的本质就依然存在。

Consider a physical system described by many coordinates $X_1 \cdots X_N$, the components of a state vector $X$. Let the system have locally stable limit points $X_a, X_b, \cdots$. Then, if the system is started sufficiently near any $X_a$, as at $X = X_a + \Delta$, it will proceed in time until $X \approx X_a$. We can regard the information stored in the system as the vectors $X_a, X_b, \cdots$. The starting point $X = X_a + \Delta$ represents a partial knowledge of the item $X_a$, and the system then generates the total information $X_a$.

考虑一个由许多坐标 $X_1 \cdots X_N$ 描述的物理系统，这些坐标是状态向量 $X$ 的分量。设系统有局部稳定的极限点 $X_a, X_b, \cdots$。那么，只要系统从足够靠近某个 $X_a$ 的位置出发，例如 $X = X_a + \Delta$，它就会随时间演化，直到 $X \approx X_a$。我们可以把系统中存储的信息看作向量 $X_a, X_b, \cdots$。起点 $X = X_a + \Delta$ 代表对条目 $X_a$ 的部分了解，系统随后生成完整的信息 $X_a$。

Any physical system whose dynamics in phase space is dominated by a substantial number of locally stable states to which it is attracted can therefore be regarded as a general content-addressable memory. The physical system will be a potentially useful memory if, in addition, any prescribed set of states can readily be made the stable states of the system.

因此，任何一个物理系统，只要它在相空间中的动力学由相当数量的、吸引它前往的局部稳定状态所主导，就可以看作一般的内容寻址存储器。如果此外还能很容易地让任意指定的一组状态成为系统的稳定状态，这个物理系统就有望成为有用的存储器。

The model system

模型系统

The processing devices will be called neurons. Each neuron $i$ has two states like those of McCullough and Pitts (12): $V_i = 0$ ("not firing") and $V_i = 1$ ("firing at maximum rate"). When neuron $i$ has a connection made to it from neuron $j$, the strength of connection is defined as $T_{ij}$. (Nonconnected neurons have $T_{ij} \equiv 0$.) The instantaneous state of the system is specified by listing the $N$ values of $V_i$, so it is represented by a binary word of $N$ bits.

处理器件称为神经元。每个神经元 $i$ 像 McCullough 和 Pitts（12）的神经元一样有两种状态：$V_i = 0$（“不放电”）和 $V_i = 1$（“以最大频率放电”）。若神经元 $j$ 向神经元 $i$ 有连接，连接强度定义为 $T_{ij}$（没有连接的神经元 $T_{ij} \equiv 0$）。系统的瞬时状态由 $N$ 个 $V_i$ 的取值列出，因此可以用一个 $N$ 位的二进制字表示。

The state changes in time according to the following algorithm. For each neuron $i$ there is a fixed threshold $U_i$. Each neuron $i$ readjusts its state randomly in time but with a mean attempt rate $W$, setting

状态按下面的算法随时间变化。每个神经元 $i$ 有一个固定阈值 $U_i$。每个神经元 $i$ 在随机的时刻调整自己的状态，平均尝试频率为 $W$，规则是

$$V_i \to 1 \ \text{if} \ \sum_{j \neq i} T_{ij} V_j > U_i, \qquad V_i \to 0 \ \text{if} \ \sum_{j \neq i} T_{ij} V_j < U_i \qquad [1]$$

Thus, each neuron randomly and asynchronously evaluates whether it is above or below threshold and readjusts accordingly. (Unless otherwise stated, we choose $U_i = 0$.)

也就是说，每个神经元随机、异步地判断自己是在阈值之上还是之下，并据此调整状态。（除非另有说明，我们取 $U_i = 0$。）

Although this model has superficial similarities to the Perceptron (13, 14) the essential differences are responsible for the new results. First, Perceptrons were modeled chiefly with neural connections in a "forward" direction $A \to B \to C \to D$. The analysis of networks with strong backward coupling $A \rightleftarrows B \rightleftarrows C$ proved intractable. All our interesting results arise as consequences of the strong back-coupling. Second, Perceptron studies usually made a random net of neurons deal directly with a real physical world and did not ask the questions essential to finding the more abstract emergent computational properties. Finally, Perceptron modeling required synchronous neurons like a conventional digital computer. There is no evidence for such global synchrony and, given the delays of nerve signal propagation, there would be no way to use global synchrony effectively. Chiefly computational properties which can exist in spite of asynchrony have interesting implications in biology.

这个模型虽然表面上与感知机（13, 14）有些相似，但新结果正来自两者的本质差别。第一，感知机主要按“前向”方向 $A \to B \to C \to D$ 建模神经连接，而带有强反向耦合 $A \rightleftarrows B \rightleftarrows C$ 的网络，事实证明难以分析。我们所有有意思的结果都是强反向耦合的产物。第二，感知机研究通常让一个随机的神经元网络直接面对真实的物理世界，并没有提出那些对发现更抽象的涌现计算性质至关重要的问题。最后，感知机的建模要求神经元像传统数字计算机那样同步工作。没有证据表明存在这种全局同步，而且考虑到神经信号传播的延迟，也无法有效利用全局同步。主要是那些在异步条件下仍能存在的计算性质，才对生物学有意义。

The information storage algorithm

信息存储算法

Suppose we wish to store the set of states $V^s$, $s = 1 \cdots n$. We use the storage prescription (15, 16)

假设我们希望存储一组状态 $V^s$，$s = 1 \cdots n$。我们使用如下存储规则（15, 16）：

$$T_{ij} = \sum_s (2V_i^s - 1)(2V_j^s - 1) \qquad [2]$$

but with $T_{ii} = 0$. From this definition

但取 $T_{ii} = 0$。由这一定义可得

$$\sum_j T_{ij} V_j^{s'} = \sum_s (2V_i^s - 1)\left[\sum_j V_j^{s'}(2V_j^s - 1)\right] \equiv H_j^{s'} \qquad [3]$$

The mean value of the bracketed term in Eq. 3 is 0 unless $s = s'$, for which the mean is $N/2$. This pseudoorthogonality yields

式 3 中方括号项的平均值为 0，除非 $s = s'$，此时平均值为 $N/2$。这种准正交性给出

$$\sum_j T_{ij} V_j^{s'} \equiv \langle H_i^{s'} \rangle \approx (2V_i^{s'} - 1) N/2 \qquad [4]$$

and is positive if $V_i^{s'} = 1$ and negative if $V_i^{s'} = 0$. Except for the noise coming from the $s \neq s'$ terms, the stored state would always be stable under our processing algorithm.

当 $V_i^{s'} = 1$ 时它为正，当 $V_i^{s'} = 0$ 时它为负。若不考虑来自 $s \neq s'$ 各项的噪声，被存储的状态在我们的处理算法下总是稳定的。

Such matrices $T_{ij}$ have been used in theories of linear associative nets (15–19) to produce an output pattern from a paired input stimulus, $S_1 \to O_1$. A second association $S_2 \to O_2$ can be simultaneously stored in the same network. But the confusing simulus $0.6\,S_1 + 0.4\,S_2$ will produce a generally meaningless mixed output $0.6\,O_1 + 0.4\,O_2$. Our model, in contrast, will use its strong nonlinearity to make choices, produce categories, and regenerate information and, with high probability, will generate the output $O_1$ from such a confusing mixed stimulus.

这样的矩阵 $T_{ij}$ 曾用于线性联想网络的理论（15–19），由配对的输入刺激产生输出模式，$S_1 \to O_1$。同一个网络里还可以同时存储第二组联想 $S_2 \to O_2$。但令人混淆的刺激 $0.6\,S_1 + 0.4\,S_2$ 会产生一个通常毫无意义的混合输出 $0.6\,O_1 + 0.4\,O_2$。相比之下，我们的模型会利用自身很强的非线性作出选择、形成类别、重建信息，面对这样令人混淆的混合刺激，它会以很高的概率产生输出 $O_1$。

A linear associative net must be connected in a complex way with an external nonlinear logic processor in order to yield true computation (20, 21). Complex circuitry is easy to plan but more difficult to discuss in evolutionary terms. In contrast, our model obtains its emergent computational properties from simple properties of many cells rather than circuitry.

线性联想网络必须以复杂的方式与外部的非线性逻辑处理器相连，才能进行真正的计算（20, 21）。复杂电路容易规划，却更难从进化的角度来讨论。相比之下，我们的模型的涌现计算性质来自大量细胞的简单性质，而不是电路。

The biological interpretation of the model

模型的生物学解释

Most neurons are capable of generating a train of action potentials—propagating pulses of electrochemical activity—when the average potential across their membrane is held well above its normal resting value. The mean rate at which action potentials are generated is a smooth function of the mean membrane potential, having the general form shown in Fig. 1.

当膜两侧的平均电位保持在远高于正常静息值的水平时，大多数神经元能够产生一串动作电位，也就是传播的电化学活动脉冲。动作电位产生的平均频率是平均膜电位的光滑函数，其大致形状如图 1 所示。

The biological information sent to other neurons often lies in a short-time average of the firing rate (22). When this is so, one can neglect the details of individual action potentials and regard Fig. 1 as a smooth input–output relationship. [Parallel pathways carrying the same information would enhance the ability of the system to extract a short-term average firing rate (23, 24).]

发送给其他神经元的生物信息，往往体现在放电频率的短时平均上（22）。在这种情况下，可以忽略单个动作电位的细节，把图 1 看作一条光滑的输入—输出关系。［携带相同信息的并行通路会增强系统提取短时平均放电频率的能力（23, 24）。］

A study of emergent collective effects and spontaneous computation must necessarily focus on the nonlinearity of the input–output relationship. The essence of computation is nonlinear logical operations. The particle interactions that produce true collective effects in particle dynamics come from a nonlinear dependence of forces on positions of the particles. Whereas linear associative networks have emphasized the linear central region (14–19) of Fig. 1, we will replace the input–output relationship by the dot-dash step. Those neurons whose operation is dominantly linear merely provide a pathway of communication between nonlinear neurons. Thus, we consider a network of "on or off" neurons, granting that some of the interconnections may be by way of neurons operating in the linear regime.

研究涌现的集体效应和自发计算，必然要关注输入—输出关系的非线性。计算的本质是非线性的逻辑运算。在粒子动力学中，产生真正集体效应的粒子相互作用，来自力对粒子位置的非线性依赖。线性联想网络强调的是图 1 中间的线性区域（14–19），我们则把输入—输出关系替换为点划线所示的阶跃。那些以线性工作为主的神经元，只是在非线性神经元之间提供通信通路。因此，我们考虑由“开或关”神经元组成的网络，同时承认其中一部分互连可能要经过工作在线性区的神经元。

Delays in synaptic transmission (of partially stochastic character) and in the transmission of impulses along axons and dendrites produce a delay between the input of a neuron and the generation of an effective output. All such delays have been modeled by a single parameter, the stochastic mean processing time $1/W$.

突触传递中的延迟（部分带有随机性）以及冲动沿轴突和树突传递的延迟，使神经元从接收输入到产生有效输出之间存在延迟。所有这些延迟都用一个参数来建模，即随机的平均处理时间 $1/W$。

The input to a particular neuron arises from the current leaks of the synapses to that neuron, which influence the cell mean potential. The synapses are activated by arriving action potentials. The input signal to a cell $i$ can be taken to be

某个神经元的输入来自通向它的各个突触的电流泄漏，这些电流影响细胞的平均电位。突触由到达的动作电位激活。细胞 $i$ 的输入信号可以取为

$$\sum_j T_{ij} V_j \qquad [5]$$

where $T_{ij}$ represents the effectiveness of a synapse. Fig. 1 thus becomes an input–output relationship for a neuron.

其中 $T_{ij}$ 表示突触的效能。于是图 1 就成为一个神经元的输入—输出关系。

Fig. 1. Firing rate versus membrane voltage for a typical neuron (solid line), dropping to 0 for large negative potentials and saturating for positive potentials. The broken lines show approximations used in modeling.

图 1　典型神经元的放电频率随膜电压的变化（实线）：在较大的负电位处降为 0，在正电位处饱和。虚线表示建模中使用的近似（点划线为本文模型，短划线为线性建模）。

Little, Shaw, and Roney (8, 25, 26) have developed ideas on the collective functioning of neural nets based on "on/off" neurons and synchronous processing. However, in their model the relative timing of action potential spikes was central and resulted in reverberating action potential trains. Our model and theirs have limited formal similarity, although there may be connections at a deeper level.

Little、Shaw 和 Roney（8, 25, 26）基于“开/关”神经元和同步处理，提出了关于神经网络集体运作的想法。不过在他们的模型中，动作电位尖峰的相对时序是核心，并导致动作电位串的反复回响。我们的模型与他们的模型形式上的相似之处有限，尽管在更深的层面上可能存在联系。

Most modeling of neural learning networks has been based on synapses of a general type described by Hebb (27) and Eccles (28). The essential ingredient is the modification of $T_{ij}$ by correlations like

大多数神经学习网络的建模，都基于 Hebb（27）和 Eccles（28）描述的一般类型的突触。其核心要素是按如下相关性修改 $T_{ij}$：

$$\Delta T_{ij} = [V_i(t) V_j(t)]_{\text{average}} \qquad [6]$$

where the average is some appropriate calculation over past history. Decay in time and effects of $[V_i(t)]_{\text{avg}}$ or $[V_j(t)]_{\text{avg}}$ are also allowed. Model networks with such synapses (16, 20, 21) can construct the associative $T_{ij}$ of Eq. 2. We will therefore initially assume that such a $T_{ij}$ has been produced by previous experience (or inheritance). The Hebbian property need not reside in single synapses; small groups of cells which produce such a net effect would suffice.

其中的平均是对过去历史的某种适当计算，也允许随时间衰减以及 $[V_i(t)]_{\text{avg}}$ 或 $[V_j(t)]_{\text{avg}}$ 的影响。具有这类突触的模型网络（16, 20, 21）能够构造出式 2 中的联想矩阵 $T_{ij}$。因此我们一开始先假定这样的 $T_{ij}$ 已由以往的经验（或遗传）产生。Hebb 性质不必存在于单个突触中，只要有能产生这种净效应的小细胞群就够了。

The network of cells we describe performs an abstract calculation and, for applications, the inputs should be appropriately coded. In visual processing, for example, feature extraction should previously have been done. The present modeling might then be related to how an entity or Gestalt is remembered or categorized on the basis of inputs representing a collection of its features.

我们描述的细胞网络执行的是一种抽象计算，用于实际时，输入需要经过适当编码。例如在视觉处理中，特征提取应当事先完成。这样，本文的建模就可能与下述问题相关：如何根据代表某个实体或格式塔（Gestalt）一组特征的输入，来记住它或对它归类。

Studies of the collective behaviors of the model

模型集体行为的研究

The model has stable limit points. Consider the special case $T_{ij} = T_{ji}$, and define

这个模型有稳定的极限点。考虑 $T_{ij} = T_{ji}$ 的特殊情形，并定义

$$E = -\frac{1}{2} \sum\sum_{i \neq j} T_{ij} V_i V_j \qquad [7]$$

$\Delta E$ due to $\Delta V_i$ is given by

由 $\Delta V_i$ 引起的 $\Delta E$ 为

$$\Delta E = -\Delta V_i \sum_{j \neq i'} T_{ij} V_j \qquad [8]$$

Thus, the algorithm for altering $V_i$ causes $E$ to be a monotonically decreasing function. State changes will continue until a least (local) $E$ is reached. This case is isomorphic with an Ising model. $T_{ij}$ provides the role of the exchange coupling, and there is also an external local field at each site. When $T_{ij}$ is symmetric but has a random character (the spin glass) there are known to be many (locally) stable states (29).

因此，修改 $V_i$ 的算法使 $E$ 成为单调递减的函数。状态变化会一直持续，直到 $E$ 达到（局部）最小。这种情形与伊辛模型同构：$T_{ij}$ 起交换耦合的作用，每个格点上还有一个外部局部场。已知当 $T_{ij}$ 对称但具有随机性时（自旋玻璃），存在许多（局部）稳定状态（29）。

Monte Carlo calculations were made on systems of $N = 30$ and $N = 100$, to examine the effect of removing the $T_{ij} = T_{ji}$ restriction. Each element of $T_{ij}$ was chosen as a random number between −1 and 1. The neural architecture of typical cortical regions (30, 31) and also of simple ganglia of invertebrates (32) suggests the importance of 100–10,000 cells with intense mutual interconnections in elementary processing, so our scale of $N$ is slightly small.

为考察去掉 $T_{ij} = T_{ji}$ 限制的影响，我们对 $N = 30$ 和 $N = 100$ 的系统做了蒙特卡罗计算。$T_{ij}$ 的每个元素取 −1 到 1 之间的随机数。典型皮层区域（30, 31）以及无脊椎动物简单神经节（32）的神经结构表明，在基本处理中，100 到 10000 个紧密互连的细胞很重要，所以我们取的 $N$ 规模略小。

The dynamics algorithm was initiated from randomly chosen initial starting configurations. For $N = 30$ the system never displayed an ergodic wandering through state space. Within a time of about $4/W$ it settled into limiting behaviors, the commonest being a stable state. When 50 trials were examined for a particular such random matrix, all would result in one of two or three end states. A few stable states thus collect the flow from most of the initial state space. A simple cycle also occurred occasionally—for example, $\cdots A \to B \to A \to B \cdots$.

动力学算法从随机选取的初始构型开始运行。$N = 30$ 时，系统从未在状态空间中表现出遍历式的游荡。大约在 $4/W$ 的时间内，它就进入极限行为，最常见的是稳定状态。对某个这样的随机矩阵考察 50 次试验，全部结果都落在两三个终态之一。可见少数几个稳定状态汇集了来自大部分初始状态空间的流动。偶尔也会出现简单的循环，例如 $\cdots A \to B \to A \to B \cdots$。

The third behavior seen was chaotic wandering in a small region of state space. The Hamming distance between two binary states $A$ and $B$ is defined as the number of places in which the digits are different. The chaotic wandering occurred within a short Hamming distance of one particular state. Statistics were done on the probability $p_i$ of the occurrence of a state in a time of wandering around this minimum, and an entropic measure of the available states $M$ was taken

观察到的第三种行为，是在状态空间的一个小区域内混沌游荡。两个二进制状态 $A$ 和 $B$ 之间的汉明距离，定义为两者对应位数字不同的位置数。混沌游荡发生在离某个特定状态很短的汉明距离之内。我们统计了在这个极小值附近游荡期间各状态出现的概率 $p_i$，并取可达状态数 $M$ 的熵度量

$$\ln M = -\sum p_i \ln p_i \qquad [9]$$

A value of $M = 25$ was found for $N = 30$. The flow in phase space produced by this model algorithm has the properties necessary for a physical content-addressable memory whether or not $T_{ij}$ is symmetric.

$N = 30$ 时得到 $M = 25$。无论 $T_{ij}$ 是否对称，这个模型算法在相空间中产生的流动都具备物理内容寻址存储器所需的性质。

Simulations with $N = 100$ were much slower and not quantitatively pursued. They showed qualitative similarity to $N = 30$.

$N = 100$ 的模拟慢得多，没有做定量研究，定性上与 $N = 30$ 相似。

Why should stable limit points or regions persist when $T_{ij} \neq T_{ji}$? If the algorithm at some time changes $V_i$ from 0 to 1 or vice versa, the change of the energy defined in Eq. 7 can be split into two terms, one of which is always negative. The second is identical if $T_{ij}$ is symmetric and is "stochastic" with mean 0 if $T_{ij}$ and $T_{ji}$ are randomly chosen. The algorithm for $T_{ij} \neq T_{ji}$ therefore changes $E$ in a fashion similar to the way $E$ would change in time for a symmetric $T_{ij}$ but with an algorithm corresponding to a finite temperature.

为什么在 $T_{ij} \neq T_{ji}$ 时，稳定的极限点或区域仍会存在？如果算法某一时刻把 $V_i$ 从 0 改为 1 或反之，式 7 定义的能量的变化可以拆成两项，其中一项总是负的。第二项在 $T_{ij}$ 对称时与第一项相同；若 $T_{ij}$ 和 $T_{ji}$ 随机选取，它则是均值为 0 的“随机”项。因此，在 $T_{ij} \neq T_{ji}$ 时，算法改变 $E$ 的方式，类似于对称 $T_{ij}$ 下 $E$ 随时间的变化，只不过对应的是有限温度下的算法。

About 0.15 N states can be simultaneously remembered before error in recall is severe. Computer modeling of memory storage according to Eq. 2 was carried out for $N = 30$ and $N = 100$. $n$ random memory states were chosen and the corresponding $T_{ij}$ was generated. If a nervous system preprocessed signals for efficient storage, the preprocessed information would appear random (e.g., the coding sequences of DNA have a random character). The random memory vectors thus simulate efficiently encoded real information, as well as representing our ignorance. The system was started at each assigned nominal memory state, and the state was allowed to evolve until stationary.

在回忆错误变得严重之前，大约可以同时记住 0.15 N 个状态。 我们按式 2 对 $N = 30$ 和 $N = 100$ 做了记忆存储的计算机建模：随机选取 $n$ 个记忆状态，生成相应的 $T_{ij}$。如果神经系统为了高效存储而对信号做了预处理，预处理后的信息看起来就会是随机的（例如 DNA 的编码序列具有随机性）。因此，随机记忆向量既模拟了经过高效编码的真实信息，也代表了我们的无知。系统从每个指定的名义记忆状态出发，任其演化直到静止。

Typical results are shown in Fig. 2. The statistics are averages over both the states in a given matrix and different matrices. With $n = 5$, the assigned memory states are almost always stable (and exactly recallable). For $n = 15$, about half of the nominally remembered states evolved to stable states with less than 5 errors, but the rest evolved to states quite different from the starting points.

典型结果见图 2，统计量是对同一矩阵中的各个状态以及不同矩阵所取的平均。$n = 5$ 时，指定的记忆状态几乎总是稳定的（并且能被准确回忆）。$n = 15$ 时，名义上记住的状态中约有一半演化到错误少于 5 个的稳定状态，其余则演化到与起点相差很远的状态。

These results can be understood from an analysis of the effect of the noise terms. In Eq. 3, $H_i^{s'}$ is the "effective field" on neuron $i$ when the state of the system is $s'$, one of the nominal memory states. The expectation value of this sum, Eq. 4, is $\pm N/2$ as appropriate. The $s \neq s'$ summation in Eq. 2 contributes no mean, but has a rms noise of $[(n-1)N/2]^{1/2} \equiv \sigma$. For $nN$ large, this noise is approximately Gaussian and the probability of an error in a single particular bit of a particular memory will be

分析噪声项的影响，就能理解这些结果。在式 3 中，当系统处于名义记忆状态之一 $s'$ 时，$H_i^{s'}$ 是作用在神经元 $i$ 上的“有效场”。这个和的期望值（式 4）相应地为 $\pm N/2$。式 2 中 $s \neq s'$ 的求和不贡献均值，但有均方根噪声 $[(n-1)N/2]^{1/2} \equiv \sigma$。当 $nN$ 很大时，这一噪声近似服从高斯分布，某个特定记忆中某一特定比特出错的概率为

$$P = \frac{1}{\sqrt{2\pi\sigma^2}} \int_{N/2}^{\infty} e^{-x^2/2\sigma^2}\, dx \qquad [10]$$

For the case $n = 10$, $N = 100$, $P = 0.0091$, the probability that a state had no errors in its 100 bits should be about $e^{-0.91} \approx 0.40$. In the simulation of Fig. 2, the experimental number was 0.6.

在 $n = 10$、$N = 100$ 的情形下，$P = 0.0091$，一个状态的 100 个比特全部无误的概率应约为 $e^{-0.91} \approx 0.40$。在图 2 的模拟中，实验值为 0.6。

The theoretical scaling of $n$ with $N$ at fixed $P$ was demonstrated in the simulations going between $N = 30$ and $N = 100$. The experimental results of half the memories being well retained at $n = 0.15\,N$ and the rest badly retained is expected to be true for all large $N$. The information storage at a given level of accuracy can be increased by a factor of 2 by a judicious choice of individual neuron thresholds. This choice is equivalent to using variables $\mu_i = \pm 1$, $T_{ij} = \sum_s \mu_i^s \mu_j^s$, and a threshold level of 0.

在从 $N = 30$ 到 $N = 100$ 的模拟中，验证了固定 $P$ 时 $n$ 随 $N$ 变化的理论标度关系。在 $n = 0.15\,N$ 时一半记忆保持良好、其余保持很差，这一实验结果预计对所有大的 $N$ 都成立。巧妙选取各神经元的阈值，可以使给定精度下的信息存储量提高到 2 倍。这种选择等价于使用变量 $\mu_i = \pm 1$、$T_{ij} = \sum_s \mu_i^s \mu_j^s$，并取阈值为 0。

Fig. 2. The probability distribution of the occurrence of errors in the location of the stable states obtained from nominally assigned memories. (Panels: $n = 5$, $N = 100$; $n = 10$, $N = 100$; $n = 15$, $N = 100$. Horizontal axis: $N_{\text{err}}$ = Number of Errors in State; vertical axis: Probability.)

图 2　从名义指定的记忆出发得到的稳定状态，其位置上出现错误数的概率分布。（三个子图分别为 $n = 5$、$N = 100$；$n = 10$、$N = 100$；$n = 15$、$N = 100$。横轴：$N_{\text{err}}$ = 状态中的错误数；纵轴：概率。）

Given some arbitrary starting state, what is the resulting final state (or statistically, states)? To study this, evolutions from randomly chosen initial states were tabulated for $N = 30$ and $n = 5$. From the (inessential) symmetry of the algorithm, if (101110···) is an assigned stable state, (010001···) is also stable. Therefore, the matrices had 10 nominal stable states. Approximately 85% of the trials ended in assigned memories, and 10% ended in stable states of no obvious meaning. An ambiguous 5% landed in stable states very near assigned memories. There was a range of a factor of 20 of the likelihood of finding these 10 states.

给定任意一个起始状态，最终会得到什么状态（或在统计意义上，哪些状态）？为研究这个问题，我们对 $N = 30$、$n = 5$ 的情形，把从随机初始状态出发的演化结果列表统计。由于算法的（非本质的）对称性，如果 (101110···) 是指定的稳定状态，那么 (010001···) 也是稳定的，因此这些矩阵有 10 个名义稳定状态。约 85% 的试验终止于指定的记忆，10% 终止于没有明显意义的稳定状态，另有含义不明的 5% 落在离指定记忆很近的稳定状态。找到这 10 个状态的可能性彼此相差最多 20 倍。

The algorithm leads to memories near the starting state. For $N = 30$, $n = 5$, partially random starting states were generated by random modification of known memories. The probability that the final state was that closest to the initial state was studied as a function of the distance between the initial state and the nearest memory state. For distance ≤ 5, the nearest state was reached more than 90% of the time. Beyond that distance, the probability fell off smoothly, dropping to a level of 0.2 (2 times random chance) for a distance of 12.

算法会把系统引向起始状态附近的记忆。对 $N = 30$、$n = 5$，我们随机修改已知记忆，生成部分随机的起始状态，研究最终状态恰为离初始状态最近的记忆的概率，与初始状态到最近记忆状态距离之间的关系。距离 ≤ 5 时，超过 90% 的情况会到达最近的状态。超过这个距离，概率平滑下降，在距离为 12 时降到 0.2（随机水平的 2 倍）。

The phase space flow is apparently dominated by attractors which are the nominally assigned memories, each of which dominates a substantial region around it. The flow is not entirely deterministic, and the system responds to an ambiguous starting state by a statistical choice between the memory states it most resembles.

*相空间中的流动显然由吸引子主导，而这些吸引子就是名义指定的记忆，*每个吸引子都支配着它周围相当大的一片区域。流动并不完全是确定性的，面对含义不明的起始状态，系统会在它最相像的几个记忆状态之间作出统计性的选择。

Were it desired to use such a system in an Si-based content-addressable memory, the algorithm should be used and modified to hold the known bits of information while letting the others adjust.

如果想在硅基内容寻址存储器中使用这样的系统，就应当采用这一算法并加以修改：固定已知的信息位，让其余各位自行调整。

The model was studied by using a "clipped" $T_{ij}$, replacing $T_{ij}$ in Eq. 3 by ±1, the algebraic sign of $T_{ij}$. The purposes were to examine the necessity of a linear synapse supposition (by making a highly nonlinear one) and to examine the efficiency of storage. Only $N(N/2)$ bits of information can possibly be stored in this symmetric matrix. Experimentally, for $N = 100$, $n = 9$, the level of errors was similar to that for the ordinary algorithm at $n = 12$. The signal-to-noise ratio can be evaluated analytically for this clipped algorithm and is reduced by a factor of $(2/\pi)^{1/2}$ compared with the unclipped case. For a fixed error probability, the number of memories must be reduced by $2/\pi$.

我们还用“截断”的 $T_{ij}$ 研究了这个模型，即把式 3 中的 $T_{ij}$ 换成它的代数符号 ±1。目的是检验线性突触假设是否必要（换成高度非线性的突触），并考察存储效率。这个对称矩阵最多只能存储 $N(N/2)$ 比特信息。实验上，$N = 100$、$n = 9$ 时的错误水平，与普通算法在 $n = 12$ 时相近。截断算法的信噪比可以解析求出，比未截断的情形降低 $(2/\pi)^{1/2}$ 倍。若保持错误概率不变，记忆数量必须减少到 $2/\pi$ 倍。

With the $\mu$ algorithm and the clipped $T_{ij}$, both analysis and modeling showed that the maximal information stored for $N = 100$ occurred at about $n = 13$. Some errors were present, and the Shannon information stored corresponded to about $N(N/8)$ bits.

采用 $\mu$ 算法和截断的 $T_{ij}$ 时，分析和建模都表明，$N = 100$ 的最大存储信息量出现在 $n = 13$ 左右。此时存在一些错误，所存储的香农信息约相当于 $N(N/8)$ 比特。

New memories can be continually added to $T_{ij}$. The addition of new memories beyond the capacity overloads the system and makes all memory states irretrievable unless there is a provision for forgetting old memories (16, 27, 28).

新的记忆可以不断加入 $T_{ij}$。超出容量继续加入新记忆会使系统过载，导致所有记忆状态都无法取回，除非设有遗忘旧记忆的机制（16, 27, 28）。

The saturation of the possible size of $T_{ij}$ will itself cause forgetting. Let the possible values of $T_{ij}$ be 0, ±1, ±2, ±3, and $T_{ij}$ be freely incremented within this range. If $T_{ij} = 3$, a next increment of +1 would be ignored and a next increment of −1 would reduce $T_{ij}$ to 2. When $T_{ij}$ is so constructed, only the recent memory states are retained, with a slightly increased noise level. Memories from the distant past are no longer stable. How far into the past are states remembered depends on the digitizing depth of $T_{ij}$, and 0, ···, ±3 is an appropriate level for $N = 100$. Other schemes can be used to keep too many memories from being simultaneously written, but this particular one is attractive because it requires no delicate balances and is a consequence of natural hardware.

$T_{ij}$ 可取值范围的饱和本身就会导致遗忘。设 $T_{ij}$ 的可能取值为 0、±1、±2、±3，并可在此范围内自由增减。若 $T_{ij} = 3$，下一次 +1 的增量会被忽略，而下一次 −1 的增量会把 $T_{ij}$ 降为 2。这样构造 $T_{ij}$ 时，只有最近的记忆状态被保留，噪声水平略有上升，久远的记忆不再稳定。能记住多久以前的状态，取决于 $T_{ij}$ 的数字化深度；对 $N = 100$，0, ···, ±3 是合适的水平。也可以用其他方案防止同时写入过多记忆，但这种方案很有吸引力，因为它不需要精细的平衡，而且是天然硬件的自然结果。

Real neurons need not make synapses both of $i \to j$ and $j \to i$. Particular synapses are restricted to one sign of output. We therefore asked whether $T_{ij} = T_{ji}$ is important. Simulations were carried out with only one $ij$ connection: if $T_{ij} \neq 0$, $T_{ji} = 0$. The probability of making errors increased, but the algorithm continued to generate stable minima. A Gaussian noise description of the error rate shows that the signal-to-noise ratio for given $n$ and $N$ should be decreased by the factor $1/\sqrt{2}$, and the simulations were consistent with such a factor. This same analysis shows that the system generally fails in a "soft" fashion, with signal-to-noise ratio and error rate increasing slowly as more synapses fail.

真实的神经元不一定同时有 $i \to j$ 和 $j \to i$ 两个方向的突触，而且特定突触的输出只限于一种符号。因此我们考察了 $T_{ij} = T_{ji}$ 是否重要。在模拟中每对 $ij$ 只保留一条连接：若 $T_{ij} \neq 0$，则 $T_{ji} = 0$。出错的概率增加了，但算法仍能产生稳定的极小值。用高斯噪声描述错误率可知，给定 $n$ 和 $N$ 时信噪比应降低为原来的 $1/\sqrt{2}$，模拟结果与这一因子相符。同样的分析还表明，系统通常以“软”方式失效：随着失效的突触增多，信噪比下降、错误率上升都是缓慢的。

Memories too close to each other are confused and tend to merge. For $N = 100$, a pair of random memories should be separated by 50 ± 5 Hamming units. The case $N = 100$, $n = 8$, was studied with seven random memories and the eighth made up a Hamming distance of only 30, 20, or 10 from one of the other seven memories. At a distance of 30, both similar memories were usually stable. At a distance of 20, the minima were usually distinct but displaced. At a distance of 10, the minima were often fused.

彼此过于接近的记忆会被混淆，并趋于合并。$N = 100$ 时，一对随机记忆之间的距离应为 50 ± 5 个汉明单位。我们研究了 $N = 100$、$n = 8$ 的情形：七个记忆随机选取，第八个与其余七个之一的汉明距离只有 30、20 或 10。距离为 30 时，两个相似的记忆通常都是稳定的；距离为 20 时，两个极小值通常仍可区分，但位置有所偏移；距离为 10 时，两个极小值常常融合在一起。

The algorithm categorizes initial states according to the similarity to memory states. With a threshold of 0, the system behaves as a forced categorizer.

算法根据初始状态与各记忆状态的相似程度对初始状态归类。阈值为 0 时，系统表现为一个强制归类器。

The state 00000··· is always stable. For a threshold of 0, this stable state is much higher in energy than the stored memory states and very seldom occurs. Adding a uniform threshold in the algorithm is equivalent to raising the effective energy of the stored memories compared to the 0000 state, and 0000 also becomes a likely stable state. The 0000 state is then generated by any initial state that does not resemble adequately closely one of the assigned memories and represents positive recognition that the starting state is not familiar.

状态 00000··· 总是稳定的。阈值为 0 时，这个稳定状态的能量比存储的记忆状态高得多，极少出现。在算法中加入统一的阈值，相当于提高存储记忆相对于 0000 状态的有效能量，于是 0000 也成为一个很可能出现的稳定状态。这时，任何与指定记忆都不够相像的初始状态都会演化到 0000 状态，它表示明确识别出起始状态是陌生的。

Familiarity can be recognized by other means when the memory is drastically overloaded. We examined the case $N = 100$, $n = 500$, in which there is a memory overload of a factor of 25. None of the memory states assigned were stable. The initial rate of processing of a starting state is defined as the number of neuron state readjustments that occur in a time $1/2W$. Familiar and unfamiliar states were distinguishable most of the time at this level of overload on the basis of the initial processing rate, which was faster for unfamiliar states. This kind of familiarity can only be read out of the system by a class of neurons or devices abstracting average properties of the processing group.

在记忆严重过载时，还可以用其他方式识别熟悉度。我们考察了 $N = 100$、$n = 500$ 的情形，记忆过载达 25 倍，所有指定的记忆状态都不稳定。把一个起始状态的初始处理速率定义为在 $1/2W$ 时间内发生的神经元状态调整次数。在这种过载水平下，根据初始处理速率，大多数时候能够区分熟悉和陌生的状态：陌生状态的处理速率更快。这种熟悉度只能由一类提取处理群体平均性质的神经元或器件从系统中读出。

For the cases so far considered, the expectation value of $T_{ij}$ was 0 for $i \neq j$. A set of memories can be stored with average correlations, and $\bar{T}_{ij} = C_{ij} \neq 0$ because there is a consistent internal correlation in the memories. If now a partial new state $X$ is stored

在目前考虑过的情形中，$i \neq j$ 时 $T_{ij}$ 的期望值为 0。也可以存储一组带有平均相关性的记忆，这时由于记忆内部存在一致的相关性，$\bar{T}_{ij} = C_{ij} \neq 0$。如果现在存储一个部分的新状态 $X$：

$$\Delta T_{ij} = (2X_i - 1)(2X_j - 1) \qquad i, j \leq k < N \qquad [11]$$

using only $k$ of the neurons rather than $N$, an attempt to reconstruct it will generate a stable point for all $N$ neurons. The values of $X_{k+1} \cdots X_N$ that result will be determined primarily from the sign of

即只用 $k$ 个神经元而不是全部 $N$ 个，那么尝试重建它时，会得到全部 $N$ 个神经元的一个稳定点。由此得到的 $X_{k+1} \cdots X_N$ 的取值，主要由下式的符号决定：

$$\sum_{j=1}^{k} c_{ij} x_j \qquad [12]$$

and $X$ is completed according to the mean correlations of the other memories. The most effective implementation of this capacity stores a large number of correlated matrices weakly followed by a normal storage of $X$.

$X$ 就按其他记忆的平均相关性被补全。实现这种能力最有效的办法，是先弱地存储大量相关的矩阵，再正常存储 $X$。

A nonsymmetric $T_{ij}$ can lead to the possibility that a minimum will be only metastable and will be replaced in time by another minimum. Additional nonsymmetric terms which could be easily generated by a minor modification of Hebb synapses

非对称的 $T_{ij}$ 可能使某个极小值只是亚稳的，并随时间被另一个极小值取代。我们在 $T_{ij}$ 中加入了一些额外的非对称项，它们只需对 Hebb 突触稍作修改就很容易产生：

$$\Delta T_{ij} = A \sum_s (2V_i^{s+1} - 1)(2V_j^s - 1) \qquad [13]$$

were added to $T_{ij}$. When $A$ was judiciously adjusted, the system would spend a while near $V_s$ and then leave and go to a point near $V_{s+1}$. But sequences longer than four states proved impossible to generate, and even these were not faithfully followed.

适当调节 $A$ 时，系统会在 $V_s$ 附近停留一阵，然后离开，去往 $V_{s+1}$ 附近的某一点。但事实证明无法产生超过四个状态的序列，即便是这样的序列，也不能被忠实地遵循。

Discussion

讨论

In the model network each "neuron" has elementary properties, and the network has little structure. Nonetheless, collective computational properties spontaneously arose. Memories are retained as stable entities or Gestalts and can be correctly recalled from any reasonably sized subpart. Ambiguities are resolved on a statistical basis. Some capacity for generalization is present, and time ordering of memories can also be encoded. These properties follow from the nature of the flow in phase space produced by the processing algorithm, which does not appear to be strongly dependent on precise details of the modeling. This robustness suggests that similar effects will obtain even when more neurobiological details are added.

在这个模型网络中，每个“神经元”只有最基本的性质，网络也几乎没有结构。尽管如此，集体计算性质还是自发出现了。记忆作为稳定的实体或格式塔被保留下来，可以从任何大小合理的部分正确地回忆出来。歧义在统计的基础上得到消解。网络有一定的泛化能力，记忆的时间顺序也可以被编码。这些性质源于处理算法在相空间中产生的流动的本性，而这种流动似乎并不强烈依赖于建模的精确细节。这种稳健性提示，即使加入更多神经生物学细节，类似的效应仍会出现。

Much of the architecture of regions of the brains of higher animals must be made from a proliferation of simple local circuits with well-defined functions. The bridge between simple circuits and the complex computational properties of higher nervous systems may be the spontaneous emergence of new computational capabilities from the collective behavior of large numbers of simple processing elements.

高等动物大脑各区域的结构，很大一部分必定是由大量功能明确的简单局部电路增殖而成。连接简单电路与高级神经系统复杂计算性质的桥梁，可能正是从大量简单处理单元的集体行为中自发涌现出的新的计算能力。

Implementation of a similar model by using integrated circuits would lead to chips which are much less sensitive to element failure and soft-failure than are normal circuits. Such chips would be wasteful of gates but could be made many times larger than standard designs at a given yield. Their asynchronous parallel processing capability would provide rapid solutions to some special classes of computational problems.

用集成电路实现类似的模型，可以得到对元件失效和软失效远比普通电路不敏感的芯片。这种芯片会浪费一些逻辑门，但在给定良率下可以做得比标准设计大许多倍。它们的异步并行处理能力，能为某些特殊类别的计算问题提供快速的解。

The work at California Institute of Technology was supported in part by National Science Foundation Grant DMR-8107494. This is contribution no. 6580 from the Division of Chemistry and Chemical Engineering.

加州理工学院的工作部分由美国国家科学基金会资助（项目号 DMR-8107494）。本文为化学与化学工程学部第 6580 号成果。

References

参考文献

1. Willows, A. O. D., Dorsett, D. A. & Hoyle, G. (1973) J. Neurobiol. 4, 207–237, 255–285.

2. Kristan, W. B. (1980) in Information Processing in the Nervous System, eds. Pinsker, H. M. & Willis, W. D. (Raven, New York), 241–261.

3. Knight, B. W. (1975) Lect. Math. Life Sci. 5, 111–144.

4. Smith, D. R. & Davidson, C. H. (1962) J. Assoc. Comput. Mach. 9, 268–279.

5. Harmon, L. D. (1964) in Neural Theory and Modeling, ed. Reiss, R. F. (Stanford Univ. Press, Stanford, CA), pp. 23–24.

6. Amari, S.-I. (1977) Biol. Cybern. 26, 175–185.

7. Amari, S.-I. & Akikazu, T. (1978) Biol. Cybern. 29, 127–136.

8. Little, W. A. (1974) Math. Biosci. 19, 101–120.

9. Marr, J. (1969) J. Physiol. 202, 437–470.

10. Kohonen, T. (1980) Content Addressable Memories (Springer, New York).

11. Palm, G. (1980) Biol. Cybern. 36, 19–31.

12. McCulloch, W. S. & Pitts, W. (1943) Bull. Math Biophys. 5, 115–133.

13. Minsky, M. & Papert, S. (1969) Perceptrons: An Introduction to Computational Geometry (MIT Press, Cambridge, MA).

14. Rosenblatt, F. (1962) Principles of Perceptrons (Spartan, Washington, DC).

15. Cooper, L. N. (1973) in Proceedings of the Nobel Symposium on Collective Properties of Physical Systems, eds. Lundqvist, B. & Lundqvist, S. (Academic, New York), 252–264.

16. Cooper, L. N., Liberman, F. & Oja, E. (1979) Biol. Cybern. 33, 9–28.

17. Longuet-Higgins, J. C. (1968) Proc. Roy. Soc. London Ser. B 171, 327–334.

18. Longuet-Higgins, J. C. (1968) Nature (London) 217, 104–105.

19. Kohonen, T. (1977) Associative Memory—A System-Theoretic Approach (Springer, New York).

20. Willwacher, G. (1976) Biol. Cybern. 24, 181–198.

21. Anderson, J. A. (1977) Psych. Rev. 84, 413–451.

22. Perkel, D. H. & Bullock, T. H. (1969) Neurosci. Res. Symp. Summ. 3, 405–527.

23. John, E. R. (1972) Science 177, 850–864.

24. Roney, K. J., Scheibel, A. B. & Shaw, G. L. (1979) Brain Res. Rev. 1, 225–271.

25. Little, W. A. & Shaw, G. L. (1978) Math. Biosci. 39, 281–289.

26. Shaw, G. L. & Roney, K. J. (1979) Phys. Rev. Lett. 74, 146–150.

27. Hebb, D. O. (1949) The Organization of Behavior (Wiley, New York).

28. Eccles, J. G. (1953) The Neurophysiological Basis of Mind (Clarendon, Oxford).

29. Kirkpatrick, S. & Sherrington, D. (1978) Phys. Rev. 17, 4384–4403.

30. Mountcastle, V. B. (1978) in The Mindful Brain, eds. Edelman, G. M. & Mountcastle, V. B. (MIT Press, Cambridge, MA), pp. 36–41.

31. Goldman, P. S. & Nauta, W. J. H. (1977) Brain Res. 122, 393–413.

32. Kandel, E. R. (1979) Sci. Am. 241, 61–70.