DARTS: Differentiable Architecture Search

Hanxiao Liu Thanks:  Current affiliation: Google Brain. Affiliation:  CMU Email:  hanxiaol@cs.cmu.com    Karen Simonyan Affiliation:  DeepMind Email:  simonyan@google.com    Yiming Yang Affiliation:  CMU Email:  yiming@cs.cmu.edu

Abstract

摘要

This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Our implementation has been made publicly available to facilitate further research on efficient architecture search algorithms.

本文把架构搜索表述为可微问题，以应对可扩展性挑战。与在离散、不可微搜索空间上做进化或强化学习不同，我们的方法对架构表示做连续松弛，从而可用梯度下降高效搜索。在 CIFAR-10、ImageNet、Penn Treebank 与 WikiText-2 上的大量实验表明，该算法既能发现高性能卷积结构用于图像分类、也能发现循环结构用于语言建模，速度比当时最优的不可微技术快数个数量级。实现已公开，以便继续研究高效架构搜索。

1 Introduction

1 引言

Discovering state-of-the-art neural network architectures requires substantial effort of human experts. Recently, there has been a growing interest in developing algorithmic solutions to automate the manual process of architecture design. The automatically searched architectures have achieved highly competitive performance in tasks such as image classification (Zoph & Le 2017; Zoph et al. 2018; Liu et al. 2018b; Liu et al. 2018a; Real et al. 2018) and object detection (Zoph et al. 2018).

发现前沿神经网络结构需要人类专家大量投入。近来，用算法自动完成结构设计的兴趣日益增长。自动搜索到的架构已在图像分类（Zoph & Le 2017；Zoph 等 2018；Liu 等 2018b；Liu 等 2018a；Real 等 2018）与目标检测（Zoph 等 2018）等任务上取得很强竞争力。

The best existing architecture search algorithms are computationally demanding despite their remarkable performance. For example, obtaining a state-of-the-art architecture for CIFAR-10 and ImageNet required 2000 GPU days of reinforcement learning (RL) (Zoph et al. 2018) or 3150 GPU days of evolution (Real et al. 2018). Several approaches for speeding up have been proposed, such as imposing a particular structure of the search space (Liu et al. 2018b; Liu et al. 2018a), weights or performance prediction for each individual architecture (Brock et al. 2018; Baker et al. 2018) and weight sharing/inheritance across multiple architectures (Elsken et al. 2017; Pham et al. 2018b; Cai et al. 2018; Bender et al. 2018), but the fundamental challenge of scalability remains. An inherent cause of inefficiency for the dominant approaches, e.g. based on RL, evolution, MCTS (Negrinho & Gordon 2017), SMBO (Liu et al. 2018a) or Bayesian optimization (Kandasamy et al. 2018), is the fact that architecture search is treated as a black-box optimization problem over a discrete domain, which leads to a large number of architecture evaluations required.

尽管表现出色，现有最优架构搜索算法计算代价极高。例如，为 CIFAR-10 与 ImageNet 得到前沿结构，强化学习约需 2000 GPU 天（Zoph 等 2018），进化约需 3150 GPU 天（Real 等 2018）。已有多种加速思路：约束搜索空间结构（Liu 等 2018b；Liu 等 2018a）、预测各架构权重或性能（Brock 等 2018；Baker 等 2018）、跨架构共享/继承权重（Elsken 等 2017；Pham 等 2018b；Cai 等 2018；Bender 等 2018），但可扩展性这一根本挑战仍在。主流方法（基于 RL、进化、MCTS、SMBO 或贝叶斯优化等）低效的内在原因，是把架构搜索当成离散域上的黑盒优化，因而需要大量架构评估。

In this work, we approach the problem from a different angle, and propose a method for efficient architecture search called DARTS (Differentiable ARchiTecture Search). Instead of searching over a discrete set of candidate architectures, we relax the search space to be continuous, so that the architecture can be optimized with respect to its validation set performance by gradient descent. The data efficiency of gradient-based optimization, as opposed to inefficient black-box search, allows DARTS to achieve competitive performance with the state of the art using orders of magnitude less computation resources. It also outperforms another recent efficient architecture search method, ENAS (Pham et al. 2018b). Notably, DARTS is simpler than many existing approaches as it does not involve controllers (Zoph & Le 2017; Baker et al. 2017; Zoph et al. 2018; Pham et al. 2018b; Zhong et al. 2018), hypernetworks (Brock et al. 2018) or performance predictors (Liu et al. 2018a), yet it is generic enough handle both convolutional and recurrent architectures.

本文换一角度，提出高效架构搜索方法 DARTS（Differentiable ARchiTecture Search）。我们不在离散候选集上搜索，而是把搜索空间松弛为连续，从而可用梯度下降按验证集表现优化架构。相对低效的黑盒搜索，基于梯度的数据效率使 DARTS 能以低数个数量级的算力达到与前沿相当的性能，并优于另一高效方法 ENAS（Pham 等 2018b）。DARTS 也更简单：不需要控制器、超网络或性能预测器，却通用到可同时处理卷积与循环架构。

The idea of searching architectures within a continuous domain is not new (Saxena & Verbeek 2016; Ahmed & Torresani 2017; Veniat & Denoyer 2017; Shin et al. 2018), but there are several major distinctions. While prior works seek to fine-tune a specific aspect of an architecture, such as filter shapes or branching patterns in a convolutional network, DARTS is able to learn high-performance architecture building blocks with complex graph topologies within a rich search space. Moreover, DARTS is not restricted to any specific architecture family, and is applicable to both convolutional and recurrent networks.

在连续域中搜索架构并非新想法（Saxena & Verbeek 2016；Ahmed & Torresani 2017；Veniat & Denoyer 2017；Shin 等 2018），但有若干重要区别。先前工作多微调结构的某一方面（如卷积核形状或分支模式），而 DARTS 能在丰富搜索空间中学习带复杂图拓扑的高性能积木单元。此外 DARTS 不局限于某一架构族，卷积与循环网络皆可。

In our experiments (Sect. 3) we show that DARTS is able to design a convolutional cell that achieves 2.76 $\pm$ 0.09% test error on CIFAR-10 for image classification using 3.3M parameters, which is competitive with the state-of-the-art result by regularized evolution (Real et al. 2018) obtained using three orders of magnitude more computation resources. The same convolutional cell also achieves 26.7% top-1 error when transferred to ImageNet (mobile setting), which is comparable to the best RL method (Zoph et al. 2018). On the language modeling task, DARTS efficiently discovers a recurrent cell that achieves 55.7 test perplexity on Penn Treebank (PTB), outperforming both extensively tuned LSTM (Melis et al. 2018) and all the existing automatically searched cells based on NAS (Zoph & Le 2017) and ENAS (Pham et al. 2018b).

实验（第 3 节）表明：DARTS 设计的卷积 cell 在 CIFAR-10 上以 3.3M 参数达到 $2.76\pm0.09\%$ 测试错误，与正则化进化（Real 等 2018）的前沿结果相当，但算力少约三个数量级；同一 cell 迁移到 ImageNet（mobile）top-1 为 26.7%，可与最优 RL 方法（Zoph 等 2018）相比。语言建模上，DARTS 高效发现的循环 cell 在 Penn Treebank（PTB）上测试困惑度 55.7，优于充分调参的 LSTM（Melis 等 2018）以及基于 NAS、ENAS 的已有自动搜索 cell。

Our contributions can be summarized as follows:

贡献可概括如下：

• • We introduce a novel algorithm for differentiable network architecture search based on bilevel optimization, which is applicable to both convolutional and recurrent architectures.

• • 提出基于双层优化的可微网络架构搜索新算法，适用于卷积与循环结构。

• • Through extensive experiments on image classification and language modeling tasks we show that gradient-based architecture search achieves highly competitive results on CIFAR-10 and outperforms the state of the art on PTB. This is a very interesting result, considering that so far the best architecture search methods used non-differentiable search techniques, e.g. based on RL (Zoph et al. 2018) or evolution (Real et al. 2018; Liu et al. 2018b).

• • 在图像分类与语言建模上的大量实验表明：基于梯度的架构搜索在 CIFAR-10 上极具竞争力，在 PTB 上超过当时最优。鉴于此前最优方法多用不可微技术（如 RL 或进化），这一结果很有意思。

• • We achieve remarkable efficiency improvement (reducing the cost of architecture discovery to a few GPU days), which we attribute to the use of gradient-based optimization as opposed to non-differentiable search techniques.

• • 效率显著提升（把架构发现代价降到数个 GPU 天），我们归因于用梯度优化替代不可微搜索。

• • We show that the architectures learned by DARTS on CIFAR-10 and PTB are transferable to ImageNet and WikiText-2, respectively.

• • 表明在 CIFAR-10 与 PTB 上学到的结构可分别迁移到 ImageNet 与 WikiText-2。

The implementation of DARTS is available at https://github.com/quark0/darts

DARTS 实现见 https://github.com/quark0/darts

2 Differentiable Architecture Search

2 可微架构搜索

We describe our search space in general form in Sect. 2.1, where the computation procedure for an architecture (or a cell in it) is represented as a directed acyclic graph. We then introduce a simple continuous relaxation scheme for our search space which leads to a differentiable learning objective for the joint optimization of the architecture and its weights (Sect. 2.2). Finally, we propose an approximation technique to make the algorithm computationally feasible and efficient (Sect. 2.3).

第 2.1 节以一般形式描述搜索空间：架构（或其 cell）的计算过程表示为有向无环图。第 2.2 节引入简单的连续松弛，使架构与权重可联合、可微地优化。第 2.3 节提出近似技术，使算法在计算上可行且高效。

2.1 Search Space

2.1 搜索空间

Following Zoph et al. 2018; Real et al. 2018; Liu et al. 2018a; Liu et al. 2018b, we search for a computation cell as the building block of the final architecture. The learned cell could either be stacked to form a convolutional network or recursively connected to form a recurrent network.

沿用 Zoph 等 2018；Real 等 2018；Liu 等 2018a；Liu 等 2018b，我们搜索作为最终架构积木的计算 cell。学到的 cell 既可堆叠成卷积网络，也可递归连接成循环网络。

A cell is a directed acyclic graph consisting of an ordered sequence of $N$ nodes. Each node $x^{(i)}$ is a latent representation (e.g. a feature map in convolutional networks) and each directed edge $(i,j)$ is associated with some operation $o^{(i,j)}$ that transforms $x^{(i)}$. We assume the cell to have two input nodes and a single output node. For convolutional cells, the input nodes are defined as the cell outputs in the previous two layers (Zoph et al. 2018). For recurrent cells, these are defined as the input at the current step and the state carried from the previous step. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes.

一个 cell 是由有序的 $N$ 个节点组成的有向无环图。每个节点 $x^{(i)}$ 是潜表示（如卷积网络中的特征图），每条有向边 $(i,j)$ 关联某个把 $x^{(i)}$ 变换的运算 $o^{(i,j)}$。假设 cell 有两个输入节点与一个输出节点。对卷积 cell，输入节点定义为前两层 cell 的输出（Zoph 等 2018）；对循环 cell，则定义为当前步输入与上一步携带的状态。cell 输出通过对全部中间节点做归约（如拼接）得到。

Each intermediate node is computed based on all of its predecessors:

每个中间节点由其全部前驱计算：

$$x^{(j)}=\sum_{i<j}o^{(i,j)}(x^{(i)}) \tag{1}$$

A special zero operation is also included to indicate a lack of connection between two nodes. The task of learning the cell therefore reduces to learning the operations on its edges.

还包含特殊的 zero 运算，表示两节点间无连接。因此学习 cell 就归结为学习边上的运算。

2.2 Continuous Relaxation and Optimization

2.2 连续松弛与优化

Let $\mathcal{O}$ be a set of candidate operations (e.g., convolution, max pooling, zero) where each operation represents some function $o(\cdot)$ to be applied to $x^{(i)}$. To make the search space continuous, we relax the categorical choice of a particular operation to a softmax over all possible operations:

设 $\mathcal{O}$ 为候选运算集合（如卷积、最大池化、zero），每个运算是作用在 $x^{(i)}$ 上的函数 $o(\cdot)$。为使搜索空间连续，我们把「选某一个运算」的类别选择松弛为对所有可能运算的 softmax：

$$\displaystyle\bar{o}^{(i,j)}(x)=\sum_{o\in\mathcal{O}}\frac{\exp(\alpha_{o}^{(i,j)})}{\sum_{o^{\prime}\in\mathcal{O}}\exp(\alpha_{o^{\prime}}^{(i,j)})}o(x) \tag{2}$$

where the operation mixing weights for a pair of nodes $(i,j)$ are parameterized by a vector $\alpha^{(i,j)}$ of dimension $|\mathcal{O}|$. The task of architecture search then reduces to learning a set of continuous variables $\alpha=\big\{\alpha^{(i,j)}\big\}$, as illustrated in Fig. 1. At the end of search, a discrete architecture can be obtained by replacing each mixed operation $\bar{o}^{(i,j)}$ with the most likely operation, i.e., $o^{(i,j)}=\mathrm{argmax}_{o\in\mathcal{O}}\enskip\alpha^{(i,j)}_{o}$. In the following, we refer to $\alpha$ as the (encoding of the) architecture.

其中节点对 $(i,j)$ 的运算混合权重由维数为 $|\mathcal{O}|$ 的向量 $\alpha^{(i,j)}$ 参数化。架构搜索于是归结为学习一组连续变量 $\alpha=\{\alpha^{(i,j)}\}$，见图 1。搜索结束时，用最可能的运算替换每个混合运算 $\bar{o}^{(i,j)}$，即 $o^{(i,j)}=\mathrm{argmax}_{o\in\mathcal{O}}\alpha^{(i,j)}_{o}$，得到离散架构。下文称 $\alpha$ 为（编码后的）架构。

Figure 1: An overview of DARTS: (a) Operations on the edges are initially unknown. (b) Continuous relaxation of the search space by placing a mixture of candidate operations on each edge. (c) Joint optimization of the mixing probabilities and the network weights by solving a bilevel optimization problem. (d) Inducing the final architecture from the learned mixing probabilities.

图 1：DARTS 概览：(a) 边上运算最初未知。(b) 在每条边上放置候选运算的混合，对搜索空间做连续松弛。(c) 通过求解双层优化问题，联合优化混合概率与网络权重。(d) 由学到的混合概率导出最终架构。

After relaxation, our goal is to jointly learn the architecture $\alpha$ and the weights $w$ within all the mixed operations (e.g. weights of the convolution filters). Analogous to architecture search using RL (Zoph & Le 2017; Zoph et al. 2018; Pham et al. 2018b) or evolution (Liu et al. 2018b; Real et al. 2018) where the validation set performance is treated as the reward or fitness, DARTS aims to optimize the validation loss, but using gradient descent.

松弛之后，目标是联合学习架构 $\alpha$ 与所有混合运算内的权重 $w$（如卷积核权重）。类比用 RL 或进化做架构搜索时把验证集表现当奖励或适应度，DARTS 也优化验证损失，但用的是梯度下降。

Denote by $\mathcal{L}_{train}$ and $\mathcal{L}_{val}$ the training and the validation loss, respectively. Both losses are determined not only by the architecture $\alpha$, but also the weights $w$ in the network. The goal for architecture search is to find $\alpha^{*}$ that minimizes the validation loss $\mathcal{L}_{val}(w^{*},\alpha^{*})$, where the weights $w^{*}$ associated with the architecture are obtained by minimizing the training loss $w^{*}=\mathrm{argmin}_{w}\ \mathcal{L}_{train}(w,\alpha^{*})$.

记训练与验证损失为 $\mathcal{L}_{train}$、$\mathcal{L}_{val}$。二者都由架构 $\alpha$ 与网络权重 $w$ 共同决定。架构搜索的目标是找到使验证损失 $\mathcal{L}_{val}(w^{*},\alpha^{*})$ 最小的 $\alpha^{*}$，其中与该架构对应的权重 $w^{*}$ 由最小化训练损失得到：$w^{*}=\mathrm{argmin}_{w}\mathcal{L}_{train}(w,\alpha^{*})$。

This implies a bilevel optimization problem (Anandalingam & Friesz 1992; Colson et al. 2007) with $\alpha$ as the upper-level variable and $w$ as the lower-level variable:

这蕴含一个双层优化问题（Anandalingam & Friesz 1992；Colson 等 2007），上层变量为 $\alpha$，下层为 $w$：

$$\displaystyle\min_{\alpha}\quad \\ \displaystyle\mathcal{L}_{val}(w^{*}(\alpha),\alpha) \\ \displaystyle w^{*}(\alpha)=\mathrm{argmin}_{w}\enskip\mathcal{L}_{train}(w,\alpha) \tag{3}$$

The nested formulation also arises in gradient-based hyperparameter optimization (Maclaurin et al. 2015; Pedregosa 2016; Franceschi et al. 2018), which is related in a sense that the architecture $\alpha$ could be viewed as a special type of hyperparameter, although its dimension is substantially higher than scalar-valued hyperparameters such as the learning rate, and it is harder to optimize.

这种嵌套形式也出现在基于梯度的超参优化中（Maclaurin 等 2015；Pedregosa 2016；Franceschi 等 2018）。在某种意义上 $\alpha$ 可视为特殊超参，但其维数远高于学习率这类标量超参，也更难优化。

Create a mixed operation $\bar{o}^{(i,j)}$ parametrized by $\alpha^{(i,j)}$ for each edge $(i,j)$
while not converged do

1. Update architecture $\alpha$ by descending $\nabla_{\alpha}\mathcal{L}_{val}(w-\xi\nabla_{w}\mathcal{L}_{train}(w,\alpha),\alpha)$
   ($\xi=0$ if using first-order approximation)
2. Update weights $w$ by descending $\nabla_{w}\mathcal{L}_{train}(w,\alpha)$
   Derive the final architecture based on the learned $\alpha$.

Algorithm 1 DARTS – Differentiable Architecture Search

为每条边 $(i,j)$ 创建由 $\alpha^{(i,j)}$ 参数化的混合运算 $\bar{o}^{(i,j)}$
while 未收敛 do

1. 沿 $\nabla_{\alpha}\mathcal{L}_{val}(w-\xi\nabla_{w}\mathcal{L}_{train}(w,\alpha),\alpha)$ 下降以更新架构 $\alpha$
   （一阶近似时 $\xi=0$）
2. 沿 $\nabla_{w}\mathcal{L}_{train}(w,\alpha)$ 下降以更新权重 $w$
   根据学到的 $\alpha$ 导出最终架构。

算法 1 DARTS — 可微架构搜索

2.3 Approximate Architecture Gradient

2.3 近似架构梯度

Evaluating the architecture gradient exactly can be prohibitive due to the expensive inner optimization. We therefore propose a simple approximation scheme as follows:

精确计算架构梯度会因内层优化昂贵而难以承受。因此我们提出如下简单近似：

$$\displaystyle\nabla_{\alpha}\mathcal{L}_{val}(w^{*}(\alpha),\alpha) \\ \displaystyle\approx \\ \displaystyle\nabla_{\alpha}\mathcal{L}_{val}(w-\xi\nabla_{w}\mathcal{L}_{train}(w,\alpha),\alpha) \tag{5}$$

where $w$ denotes the current weights maintained by the algorithm, and $\xi$ is the learning rate for a step of inner optimization. The idea is to approximate $w^{}(\alpha)$by adapting$w$using only a single training step*, without solving the inner optimization (equation 4) completely by training until convergence. Related techniques have been used in meta-learning for model transfer (Finn et al. 2017), gradient-based hyperparameter tuning (Luketina et al. 2016) and unrolled generative adversarial networks (Metz et al. 2017). Note equation 6 will reduce to$\nabla_{\alpha}\mathcal{L}{val}(w,\alpha)$if$w$is already a local optimum for the inner optimization and thus$\nabla{w}\mathcal{L}_{train}(w,\alpha)=0$.

其中 $w$ 为算法当前维护的权重，$\xi$ 为内层一步的学习率。想法是只用一步训练适配 $w$ 来近似 $w^{}(\alpha)$*，而不把内层优化（式 4）训到收敛。相关技巧见于模型迁移的元学习（Finn 等 2017）、基于梯度的超参调节（Luketina 等 2016）以及展开的生成对抗网络（Metz 等 2017）。若$w$已是内层局部最优从而$\nabla_{w}\mathcal{L}{train}(w,\alpha)=0$，则式 6 退化为$\nabla{\alpha}\mathcal{L}_{val}(w,\alpha)$。

The iterative procedure is outlined in Alg. 1. While we are not currently aware of the convergence guarantees for our optimization algorithm, in practice it is able to reach a fixed point with a suitable choice of $\xi$[^1]. We also note that when momentum is enabled for weight optimisation, the one-step unrolled learning objective in equation 6 is modified accordingly and all of our analysis still applies.

迭代过程见算法 1。我们目前尚不知该优化算法的收敛保证，但实践中选取合适的 $\xi$ 能到达不动点[^1]。另需说明：若权重优化启用动量，则式 6 中的一步展开目标需相应修改，但分析仍然成立。

Applying chain rule to the approximate architecture gradient (equation 6) yields

对近似架构梯度（式 6）应用链式法则得到

$$\nabla_{\alpha}\mathcal{L}_{val}(w^{\prime},\alpha)-\xi\nabla^{2}_{\alpha,w}\mathcal{L}_{train}(w,\alpha)\nabla_{w^{\prime}}\mathcal{L}_{val}(w^{\prime},\alpha) \tag{7}$$

where $w^{\prime}=w-\xi\nabla_{w}\mathcal{L}_{train}(w,\alpha)$ denotes the weights for a one-step forward model. The expression above contains an expensive matrix-vector product in its second term. Fortunately, the complexity can be substantially reduced using the finite difference approximation. Let $\epsilon$ be a small scalar[^2] and $w^{\pm}=w\pm\epsilon\nabla_{w^{\prime}}\mathcal{L}_{val}(w^{\prime},\alpha)$. Then:

其中 $w^{\prime}=w-\xi\nabla_{w}\mathcal{L}_{train}(w,\alpha)$ 表示一步前向模型的权重。上式第二项含昂贵的矩阵–向量积。幸运的是可用有限差分大幅降低复杂度。设 $\epsilon$ 为小标量[^2]，$w^{\pm}=w\pm\epsilon\nabla_{w^{\prime}}\mathcal{L}_{val}(w^{\prime},\alpha)$。则：

$$\nabla^{2}_{\alpha,w}\mathcal{L}_{train}(w,\alpha)\nabla_{w^{\prime}}\mathcal{L}_{val}(w^{\prime},\alpha)\approx\frac{\nabla_{\alpha}\mathcal{L}_{train}(w^{+},\alpha)-\nabla_{\alpha}\mathcal{L}_{train}(w^{-},\alpha)}{2\epsilon} \tag{8}$$

Evaluating the finite difference requires only two forward passes for the weights and two backward passes for $\alpha$, and the complexity is reduced from $O(|\alpha||w|)$ to $O(|\alpha|+|w|)$.

有限差分只需对权重做两次前向、对 $\alpha$ 做两次反向，复杂度从 $O(|\alpha||w|)$ 降到 $O(|\alpha|+|w|)$。

First-order Approximation

When $\xi=0$, the second-order derivative in equation 7 will disappear. In this case, the architecture gradient is given by $\nabla_{\alpha}\mathcal{L}_{val}(w,\alpha)$, corresponding to the simple heuristic of optimizing the validation loss by assuming the current $w$ is the same as $w^{*}(\alpha)$. This leads to some speed-up but empirically worse performance, according to our experimental results in Table 1 and Table 2. In the following, we refer to the case of $\xi=0$ as the first-order approximation, and refer to the gradient formulation with $\xi>0$ as the second-order approximation.

一阶近似

当 $\xi=0$ 时，式 7 中的二阶导消失。此时架构梯度为 $\nabla_{\alpha}\mathcal{L}_{val}(w,\alpha)$，对应「假定当前 $w$ 已等于 $w^{*}(\alpha)$」去优化验证损失的简单启发式。这会带来一定加速，但据表 1、表 2 的实验，性能更差。下文称 $\xi=0$ 为一阶近似，$\xi>0$ 的梯度形式为二阶近似。

2.4 Deriving Discrete Architectures

2.4 导出离散架构

To form each node in the discrete architecture, we retain the top-$k$ strongest operations (from distinct nodes) among all non-zero candidate operations collected from all the previous nodes. The strength of an operation is defined as $\frac{\exp(\alpha_{o}^{(i,j)})}{\sum_{o^{\prime}\in\mathcal{O}}\exp(\alpha_{o^{\prime}}^{(i,j)})}$. To make our derived architecture comparable with those in the existing works, we use $k=2$ for convolutional cells (Zoph et al. 2018; Liu et al. 2018a; Real et al. 2018) and $k=1$ for recurrent cells (Pham et al. 2018b).

在离散架构中形成每个节点时，我们在来自所有前驱节点的非零候选运算中，保留强度最高的 top-$k$ 个（来自不同节点）。运算强度定义为 $\frac{\exp(\alpha_{o}^{(i,j)})}{\sum_{o^{\prime}\in\mathcal{O}}\exp(\alpha_{o^{\prime}}^{(i,j)})}$。为与已有工作导出的结构可比，卷积 cell 取 $k=2$（Zoph 等 2018；Liu 等 2018a；Real 等 2018），循环 cell 取 $k=1$（Pham 等 2018b）。

The zero operations are excluded in the above for two reasons. First, we need exactly $k$ non-zero incoming edges per node for fair comparison with the existing models. Second, the strength of the zero operations is underdetermined, as increasing the logits of zero operations only affects the scale of the resulting node representations, and does not affect the final classification outcome due to the presence of batch normalization (Ioffe & Szegedy 2015).

上述过程排除 zero 运算，原因有二。其一，为与现有模型公平对比，每个节点需要恰好 $k$ 条非零入边。其二，zero 运算的强度欠定：增大其 logit 只影响节点表示的尺度，因有批归一化（Ioffe & Szegedy 2015），并不影响最终分类结果。

Figure 2: Learning dynamics of our iterative algorithm when $\mathcal{L}_{val}(w,\alpha)=\alpha w-2\alpha+1$ and $\mathcal{L}_{train}(w,\alpha)=w^{2}-2\alpha w+\alpha^{2}$, starting from $(\alpha^{(0)},w^{(0)})=(2,-2)$. The analytical solution for the corresponding bilevel optimization problem is $(\alpha^{*},w^{*})=(1,1)$, which is highlighted in the red circle. The dashed red line indicates the feasible set where constraint equation 4 is satisfied exactly (namely, weights in $w$ are optimal for the given architecture $\alpha$). The example shows that a suitable choice of $\xi$ helps to converge to a better local optimum.

图 2：当 $\mathcal{L}_{val}(w,\alpha)=\alpha w-2\alpha+1$、$\mathcal{L}_{train}(w,\alpha)=w^{2}-2\alpha w+\alpha^{2}$，从 $(\alpha^{(0)},w^{(0)})=(2,-2)$ 出发时，迭代算法的学习动态。对应双层优化问题的解析解为 $(\alpha^{*},w^{*})=(1,1)$，以红圈标出。红色虚线表示约束式 4 精确成立的可行集（即对给定 $\alpha$，$w$ 已最优）。该例表明选取合适的 $\xi$ 有助于收敛到更好的局部最优。

3 Experiments and Results

3 实验与结果

Our experiments on CIFAR-10 and PTB consist of two stages, architecture search (Sect. 3.1) and architecture evaluation (Sect. 3.2). In the first stage, we search for the cell architectures using DARTS, and determine the best cells based on their validation performance. In the second stage, we use these cells to construct larger architectures, which we train from scratch and report their performance on the test set. We also investigate the transferability of the best cells learned on CIFAR-10 and PTB by evaluating them on ImageNet and WikiText-2 (WT2) respectively.

CIFAR-10 与 PTB 上的实验分两阶段：架构搜索（第 3.1 节）与架构评估（第 3.2 节）。第一阶段用 DARTS 搜索 cell，并按验证表现选定最佳 cell；第二阶段用这些 cell 构造更大的网络，从头训练并报告测试集表现。我们也考察 CIFAR-10 / PTB 上最佳 cell 向 ImageNet / WikiText-2（WT2）的可迁移性。

3.1 Architecture Search

3.1 架构搜索

3.1.1 Searching for Convolutional Cells on CIFAR-10

We include the following operations in $\mathcal{O}$: $3\times 3$ and $5\times 5$ separable convolutions, $3\times 3$ and $5\times 5$ dilated separable convolutions, $3\times 3$ max pooling, $3\times 3$ average pooling, identity, and $zero$. All operations are of stride one (if applicable) and the convolved feature maps are padded to preserve their spatial resolution. We use the ReLU-Conv-BN order for convolutional operations, and each separable convolution is always applied twice (Zoph et al. 2018; Real et al. 2018; Liu et al. 2018a).

3.1.1 在 CIFAR-10 上搜索卷积 cell

$\mathcal{O}$ 中包含：$3\times3$ 与 $5\times5$ 可分离卷积、$3\times3$ 与 $5\times5$ 膨胀可分离卷积、$3\times3$ 最大池化、$3\times3$ 平均池化、恒等与 zero。所有运算步幅为 1（如适用），卷积特征图经填充以保持空间分辨率。卷积运算采用 ReLU-Conv-BN 顺序，每个可分离卷积总是应用两次（Zoph 等 2018；Real 等 2018；Liu 等 2018a）。

Our convolutional cell consists of $N=7$ nodes, among which the output node is defined as the depthwise concatenation of all the intermediate nodes (input nodes excluded). The rest of the setup follows Zoph et al. 2018; Liu et al. 2018a; Real et al. 2018, where a network is then formed by stacking multiple cells together. The first and second nodes of cell $k$ are set equal to the outputs of cell $k-2$ and cell $k-1$, respectively, and $1\times 1$ convolutions are inserted as necessary. Cells located at the $1/3$ and $2/3$ of the total depth of the network are reduction cells, in which all the operations adjacent to the input nodes are of stride two. The architecture encoding therefore is $(\alpha_{normal},\alpha_{reduce})$, where $\alpha_{normal}$ is shared by all the normal cells and $\alpha_{reduce}$ is shared by all the reduction cells.

卷积 cell 含 $N=7$ 个节点，输出节点定义为全部中间节点（不含输入）的深度拼接。其余设置沿用 Zoph 等 2018；Liu 等 2018a；Real 等 2018：堆叠多个 cell 形成网络。cell $k$ 的前两个节点分别等于 cell $k-2$ 与 cell $k-1$ 的输出，必要时插入 $1\times1$ 卷积。位于网络总深度 $1/3$ 与 $2/3$ 处的 cell 为归约 cell，与输入节点相邻的运算步幅均为 2。因此架构编码为 $(\alpha_{normal},\alpha_{reduce})$，分别由所有普通 cell 与所有归约 cell 共享。

Detailed experimental setup for this section can be found in Sect. A.1.1.

本节详细实验设置见附录 A.1.1。

3.1.2 Searching for Recurrent Cells on Penn Treebank

Our set of available operations includes linear transformations followed by one of $\tanh$, $\mathrm{relu}$, $\mathrm{sigmoid}$ activations, as well as the identity mapping and the zero operation. The choice of these candidate operations follows Zoph & Le 2017; Pham et al. 2018b.

3.1.2 在 Penn Treebank 上搜索循环 cell

可用运算包括：线性变换后接 $\tanh$、$\mathrm{relu}$、$\mathrm{sigmoid}$ 之一，以及恒等与 zero。候选选择沿用 Zoph & Le 2017；Pham 等 2018b。

Our recurrent cell consists of $N=12$ nodes. The very first intermediate node is obtained by linearly transforming the two input nodes, adding up the results and then passing through a $\tanh$ activation function, as done in the ENAS cell (Pham et al. 2018b). The rest of the cell is learned. Other settings are similar to ENAS, where each operation is enhanced with a highway bypass (Zilly et al. 2016) and the cell output is defined as the average of all the intermediate nodes. As in ENAS, we enable batch normalization in each node to prevent gradient explosion during architecture search, and disable it during architecture evaluation. Our recurrent network consists of only a single cell, i.e. we do not assume any repetitive patterns within the recurrent architecture.

循环 cell 含 $N=12$ 个节点。第一个中间节点对两个输入节点分别线性变换后相加，再经 $\tanh$，与 ENAS cell（Pham 等 2018b）相同；其余部分可学习。其他设置类似 ENAS：每个运算加 highway 旁路（Zilly 等 2016），cell 输出为全部中间节点的平均。搜索阶段在每个节点启用批归一以防梯度爆炸，评估阶段关闭。循环网络仅含单个 cell，即不假设循环结构内部有重复模式。

Detailed experimental setup for this section can be found in Sect. A.1.2.

本节详细实验设置见附录 A.1.2。

Figure 3: Search progress of DARTS for convolutional cells on CIFAR-10 and recurrent cells on Penn Treebank. We keep track of the most recent architectures over time. Each architecture snapshot is re-trained from scratch using the training set (for 100 epochs on CIFAR-10 and for 300 epochs on PTB) and then evaluated on the validation set. For each task, we repeat the experiments for 4 times with different random seeds, and report the median and the best (per run) validation performance of the architectures over time. As references, we also report the results (under the same evaluation setup; with comparable number of parameters) of the best existing cells discovered using RL or evolution, including NASNet-A (Zoph et al. 2018) (2000 GPU days), AmoebaNet-A (3150 GPU days) (Real et al. 2018) and ENAS (0.5 GPU day) (Pham et al. 2018b).

图 3：DARTS 在 CIFAR-10 上搜索卷积 cell、在 Penn Treebank 上搜索循环 cell 的进度。我们随时间跟踪最近架构：每个快照在训练集上从头重训（CIFAR-10 100 epoch，PTB 300 epoch）再在验证集评估。每项任务用不同随机种子重复 4 次，报告架构验证表现随时间的中位数与（每次运行的）最佳值。作为参照，也报告在相同评估设置、相近参数量下，用 RL 或进化发现的已有最佳 cell：NASNet-A（2000 GPU 天）、AmoebaNet-A（3150 GPU 天）与 ENAS（0.5 GPU 天）。

Figure 4: Normal cell learned on CIFAR-10.

图 4：在 CIFAR-10 上学到的普通 cell。

3.2 Architecture Evaluation

3.2 架构评估

To determine the architecture for final evaluation, we run DARTS four times with different random seeds and pick the best cell based on its validation performance obtained by training from scratch for a short period (100 epochs on CIFAR-10 and 300 epochs on PTB). This is particularly important for recurrent cells, as the optimization outcomes can be initialization-sensitive (Fig. 3).

为确定最终评估用的架构，我们用不同随机种子跑四次 DARTS，并按短训后的验证表现（CIFAR-10 100 epoch，PTB 300 epoch）挑选最佳 cell。这对循环 cell 尤其重要，因为优化结果对初始化敏感（图 3）。

To evaluate the selected architecture, we randomly initialize its weights (weights learned during the search process are discarded), train it from scratch, and report its performance on the test set. We note the test set is never used for architecture search or architecture selection.

评估选定架构时，随机初始化权重（丢弃搜索阶段学到的权重），从头训练，并报告测试集表现。注意测试集从不用于架构搜索或选择。

Detailed experimental setup for architecture evaluation on CIFAR-10 and PTB can be found in Sect. A.2.1 and Sect. A.2.2, respectively. Besides CIFAR-10 and PTB, we further investigated the transferability of our best convolutional cell (searched on CIFAR-10) and recurrent cell (searched on PTB) by evaluating them on ImageNet (mobile setting) and WikiText-2, respectively. More details of the transfer learning experiments can be found in Sect. A.2.3 and Sect. A.2.4.

CIFAR-10 与 PTB 上架构评估的详细设置分别见附录 A.2.1 与 A.2.2。除二者外，我们还把在 CIFAR-10 / PTB 上搜到的最佳卷积 / 循环 cell 分别迁到 ImageNet（mobile）与 WikiText-2 评估，细节见 A.2.3 与 A.2.4。

[b] Architecture Test Error Params Search Cost #ops Search (%) (M) (GPU days) Method DenseNet-BC (Huang et al. 2017) 3.46 25.6 – – manual NASNet-A + cutout (Zoph et al. 2018) 2.65 3.3 2000 13 RL NASNet-A + cutout (Zoph et al. 2018)† 2.83 3.1 2000 13 RL BlockQNN (Zhong et al. 2018) 3.54 39.8 96 8 RL AmoebaNet-A (Real et al. 2018) 3.34 $\pm$ 0.06 3.2 3150 19 evolution AmoebaNet-A + cutout (Real et al. 2018)† 3.12 3.1 3150 19 evolution AmoebaNet-B + cutout (Real et al. 2018) 2.55 $\pm$ 0.05 2.8 3150 19 evolution Hierarchical evolution (Liu et al. 2018b) 3.75 $\pm$ 0.12 15.7 300 6 evolution PNAS (Liu et al. 2018a) 3.41 $\pm$ 0.09 3.2 225 8 SMBO ENAS + cutout (Pham et al. 2018b) 2.89 4.6 0.5 6 RL ENAS + cutout (Pham et al. 2018b)* 2.91 4.2 4 6 RL Random search baseline*‡* + cutout 3.29 $\pm$ 0.15 3.2 4 7 random DARTS (first order) + cutout 3.00 $\pm$ 0.14 3.3 1.5 7 gradient-based DARTS (second order) + cutout 2.76 $\pm$ 0.09 3.3 4 7 gradient-based

Obtained by repeating ENAS for 8 times using the code publicly released by the authors. The cell for final evaluation is chosen according to the same selection protocol as for DARTS.

Obtained by training the corresponding architectures using our setup.

Best architecture among 24 samples according to the validation error after 100 training epochs.

Table 1: Comparison with state-of-the-art image classifiers on CIFAR-10 (lower error rate is better). Note the search cost for DARTS does not include the selection cost (1 GPU day) or the final evaluation cost by training the selected architecture from scratch (1.5 GPU days).

架构	测试错误 (%)	参数 (M)	搜索代价 (GPU 天)	#ops	搜索方法
DenseNet-BC (Huang 等 2017)	3.46	25.6	–	–	手工
NASNet-A + cutout (Zoph 等 2018)	2.65	3.3	2000	13	RL
NASNet-A + cutout †	2.83	3.1	2000	13	RL
BlockQNN (Zhong 等 2018)	3.54	39.8	96	8	RL
AmoebaNet-A (Real 等 2018)	3.34±0.06	3.2	3150	19	进化
AmoebaNet-A + cutout †	3.12	3.1	3150	19	进化
AmoebaNet-B + cutout	2.55±0.05	2.8	3150	19	进化
Hierarchical evolution (Liu 等 2018b)	3.75±0.12	15.7	300	6	进化
PNAS (Liu 等 2018a)	3.41±0.09	3.2	225	8	SMBO
ENAS + cutout	2.89	4.6	0.5	6	RL
ENAS + cutout *	2.91	4.2	4	6	RL
随机搜索基线 ‡ + cutout	3.29±0.15	3.2	4	7	随机
DARTS（一阶）+ cutout	3.00±0.14	3.3	1.5	7	基于梯度
DARTS（二阶）+ cutout	2.76±0.09	3.3	4	7	基于梯度

用作者公开代码将 ENAS 重复 8 次；最终评估 cell 的选择协议与 DARTS 相同。
† 在我们的设置下训练对应架构得到。
‡ 在 100 个训练 epoch 后按验证错误，从 24 个样本中取最佳架构。

表 1：CIFAR-10 上与前沿图像分类器对比（错误率越低越好）。注意 DARTS 的搜索代价不含选择代价（1 GPU 天）与最终从头评估代价（1.5 GPU 天）。

[b] Architecture Perplexity Params Search Cost #ops Search valid test (M) (GPU days) Method Variational RHN (Zilly et al. 2016) 67.9 65.4 23 – – manual LSTM (Merity et al. 2018) 60.7 58.8 24 – – manual LSTM + skip connections (Melis et al. 2018) 60.9 58.3 24 – – manual LSTM + 15 softmax experts (Yang et al. 2018) 58.1 56.0 22 – – manual NAS (Zoph & Le 2017) – 64.0 25 1e4 CPU days 4 RL ENAS (Pham et al. 2018b)* 68.3 63.1 24 0.5 4 RL ENAS (Pham et al. 2018b)† 60.8 58.6 24 0.5 4 RL Random search baseline*‡* 61.8 59.4 23 2 4 random DARTS (first order) 60.2 57.6 23 0.5 4 gradient-based DARTS (second order) 58.1 55.7 23 1 4 gradient-based

Obtained using the code (Pham et al. 2018a) publicly released by the authors.

Obtained by training the corresponding architecture using our setup.

Best architecture among 8 samples according to the validation perplexity after 300 training epochs.

Table 2: Comparison with state-of-the-art language models on PTB (lower perplexity is better). Note the search cost for DARTS does not include the selection cost (1 GPU day) or the final evaluation cost by training the selected architecture from scratch (3 GPU days).

架构	困惑度 valid/test	参数 (M)	搜索代价	#ops	方法
Variational RHN	67.9 / 65.4	23	–	–	手工
LSTM (Merity 等 2018)	60.7 / 58.8	24	–	–	手工
LSTM + skip (Melis 等 2018)	60.9 / 58.3	24	–	–	手工
LSTM + 15 softmax experts	58.1 / 56.0	22	–	–	手工
NAS (Zoph & Le 2017)	– / 64.0	25	1e4 CPU 天	4	RL
ENAS *	68.3 / 63.1	24	0.5	4	RL
ENAS †	60.8 / 58.6	24	0.5	4	RL
随机搜索基线 ‡	61.8 / 59.4	23	2	4	随机
DARTS（一阶）	60.2 / 57.6	23	0.5	4	基于梯度
DARTS（二阶）	58.1 / 55.7	23	1	4	基于梯度

表 2：PTB 上与前沿语言模型对比（困惑度越低越好）。DARTS 搜索代价不含选择（1 GPU 天）与最终评估（3 GPU 天）。

3.3 Results Analysis

3.3 结果分析

The CIFAR-10 results for convolutional architectures are presented in Table 1. Notably, DARTS achieved comparable results with the state of the art (Zoph et al. 2018; Real et al. 2018) while using three orders of magnitude less computation resources (i.e. 1.5 or 4 GPU days vs 2000 GPU days for NASNet and 3150 GPU days for AmoebaNet). Moreover, with slightly longer search time, DARTS outperformed ENAS (Pham et al. 2018b) by discovering cells with comparable error rates but less parameters. The longer search time is due to the fact that we have repeated the search process four times for cell selection. This practice is less important for convolutional cells however, because the performance of discovered architectures does not strongly depend on initialization (Fig. 3).

卷积架构的 CIFAR-10 结果见表 1。值得注意的是，DARTS 在算力少约三个数量级的情况下（1.5 或 4 GPU 天 vs NASNet 2000、AmoebaNet 3150）达到与前沿相当的结果。搜索稍久时，DARTS 还以相近错误率、更少参数超过 ENAS。更长搜索时间是因为我们为选 cell 重复了四次；对卷积 cell 这一点不那么关键，因为发现结构的表现对初始化依赖不强（图 3）。

Alternative Optimization Strategies

To better understand the necessity of bilevel optimization, we investigated a simplistic search strategy, where $\alpha$ and $w$ are jointly optimized over the union of the training and validation sets using coordinate descent. The resulting best convolutional cell (out of 4 runs) yielded 4.16 $\pm$ 0.16% test error using 3.1M parameters, which is worse than random search. In the second experiment, we optimized $\alpha$ simultaneously with $w$ (without alteration) using SGD, again over all the data available (training + validation). The resulting best cell yielded 3.56 $\pm$ 0.10% test error using 3.0M parameters. We hypothesize that these heuristics would cause $\alpha$ (analogous to hyperparameters) to overfit the training data, leading to poor generalization. Note that $\alpha$ is not directly optimized on the training set in DARTS.

替代优化策略

为更好理解双层优化的必要性，我们考察了一种简化搜索：在训练+验证并集上用坐标下降联合优化 $\alpha$ 与 $w$。四次运行中最佳卷积 cell 以 3.1M 参数得到 $4.16\pm0.16\%$ 测试错误，差于随机搜索。第二次实验中，用 SGD 同时（不交替）优化 $\alpha$ 与 $w$，同样用全部数据；最佳 cell 为 $3.56\pm0.10\%$、3.0M 参数。我们猜测这些启发式会使 $\alpha$（类似超参）过拟合训练数据，从而泛化差。注意 DARTS 中 $\alpha$ 并不直接在训练集上优化。

Table 2 presents the results for recurrent architectures on PTB, where a cell discovered by DARTS achieved the test perplexity of 55.7. This is on par with the state-of-the-art model enhanced by a mixture of softmaxes (Yang et al. 2018), and better than all the rest of the architectures that are either manually or automatically discovered. Note that our automatically searched cell outperforms the extensively tuned LSTM (Melis et al. 2018), demonstrating the importance of architecture search in addition to hyperparameter search. In terms of efficiency, the overall cost (4 runs in total) is within 1 GPU day, which is comparable to ENAS and significantly faster than NAS (Zoph & Le 2017).

表 2 给出 PTB 上循环架构结果：DARTS 发现的 cell 测试困惑度 55.7，与带 softmax 混合的前沿模型（Yang 等 2018）相当，并优于其余手工或自动发现的结构。自动搜索 cell 超过充分调参的 LSTM（Melis 等 2018），说明除超参搜索外架构搜索本身重要。效率上，四次运行总代价在 1 GPU 天内，与 ENAS 相当、远快于 NAS。

It is also interesting to note that random search is competitive for both convolutional and recurrent models, which reflects the importance of the search space design. Nevertheless, with comparable or less search cost, DARTS is able to significantly improve upon random search in both cases (2.76 $\pm$ 0.09 vs 3.29 $\pm$ 0.15 on CIFAR-10; 55.7 vs 59.4 on PTB).

有趣的是，随机搜索对卷积与循环模型都有竞争力，反映出搜索空间设计的重要性。尽管如此，在相近或更低搜索代价下，DARTS 在两种情形都显著优于随机搜索（CIFAR-10：$2.76\pm0.09$ vs $3.29\pm0.15$；PTB：55.7 vs 59.4）。

Results in Table 3 show that the cell learned on CIFAR-10 is indeed transferable to ImageNet. It is worth noticing that DARTS achieves competitive performance with the state-of-the-art RL method (Zoph et al. 2018) while using three orders of magnitude less computation resources.

表 3 表明在 CIFAR-10 上学到的 cell 确实可迁移到 ImageNet。值得注意的是，DARTS 以少约三个数量级的算力达到与前沿 RL 方法相当的表现。

Architecture	Test Error (%)	Params	$+\times$	Search Cost	Search
top-1	top-5	(M)	(M)	(GPU days)	Method
Inception-v1 (Szegedy et al. 2015)	30.2	10.1	6.6	1448	–	manual
MobileNet (Howard et al. 2017)	29.4	10.5	4.2	569	–	manual
ShuffleNet 2$\times$ ($g=3$) (Zhang et al. 2017)	26.3	–	$\sim$5	524	–	manual
NASNet-A (Zoph et al. 2018)	26.0	8.4	5.3	564	2000	RL
NASNet-B (Zoph et al. 2018)	27.2	8.7	5.3	488	2000	RL
NASNet-C (Zoph et al. 2018)	27.5	9.0	4.9	558	2000	RL
AmoebaNet-A (Real et al. 2018)	25.5	8.0	5.1	555	3150	evolution
AmoebaNet-B (Real et al. 2018)	26.0	8.5	5.3	555	3150	evolution
AmoebaNet-C (Real et al. 2018)	24.3	7.6	6.4	570	3150	evolution
PNAS (Liu et al. 2018a)	25.8	8.1	5.1	588	$\sim$225	SMBO
DARTS (searched on CIFAR-10)	26.7	8.7	4.7	574	4	gradient-based

Table 3: Comparison with state-of-the-art image classifiers on ImageNet in the mobile setting.

架构	top-1	top-5	参数 (M)	+× (M)	搜索代价	方法
Inception-v1	30.2	10.1	6.6	1448	–	手工
MobileNet	29.4	10.5	4.2	569	–	手工
ShuffleNet 2× (g=3)	26.3	–	~5	524	–	手工
NASNet-A	26.0	8.4	5.3	564	2000	RL
NASNet-B	27.2	8.7	5.3	488	2000	RL
NASNet-C	27.5	9.0	4.9	558	2000	RL
AmoebaNet-A	25.5	8.0	5.1	555	3150	进化
AmoebaNet-B	26.0	8.5	5.3	555	3150	进化
AmoebaNet-C	24.3	7.6	6.4	570	3150	进化
PNAS	25.8	8.1	5.1	588	~225	SMBO
DARTS（CIFAR-10 搜索）	26.7	8.7	4.7	574	4	基于梯度

表 3：ImageNet mobile 设置下与前沿图像分类器对比。

[b] Architecture Perplexity Params Search Cost Search valid test (M) (GPU days) Method LSTM + augmented loss (Inan et al. 2017) 91.5 87.0 28 – manual LSTM + continuous cache pointer (Grave et al. 2016) – 68.9 – – manual LSTM (Merity et al. 2018) 69.1 66.0 33 – manual LSTM + skip connections (Melis et al. 2018) 69.1 65.9 24 – manual LSTM + 15 softmax experts (Yang et al. 2018) 66.0 63.3 33 – manual ENAS (Pham et al. 2018b)† (searched on PTB) 72.4 70.4 33 0.5 RL DARTS (searched on PTB) 71.2 69.6 33 1 gradient-based

Obtained by training the corresponding architecture using our setup.

Table 4: Comparison with state-of-the-art language models on WT2.

架构	困惑度 valid/test	参数 (M)	搜索代价	方法
LSTM + augmented loss	91.5 / 87.0	28	–	手工
LSTM + continuous cache	– / 68.9	–	–	手工
LSTM (Merity 等 2018)	69.1 / 66.0	33	–	手工
LSTM + skip	69.1 / 65.9	24	–	手工
LSTM + 15 softmax experts	66.0 / 63.3	33	–	手工
ENAS †（PTB 搜索）	72.4 / 70.4	33	0.5	RL
DARTS（PTB 搜索）	71.2 / 69.6	33	1	基于梯度

表 4：WT2 上与前沿语言模型对比。

Table 4 shows that the cell identified by DARTS transfers to WT2 better than ENAS, although the overall results are less strong than those presented in Table 2 for PTB. The weaker transferability between PTB and WT2 (as compared to that between CIFAR-10 and ImageNet) could be explained by the relatively small size of the source dataset (PTB) for architecture search. The issue of transferability could potentially be circumvented by directly optimizing the architecture on the task of interest.

表 4 显示 DARTS 识别的 cell 迁移到 WT2 优于 ENAS，但整体结果不如表 2 在 PTB 上那么强。PTB 与 WT2 之间较弱的可迁移性（相对 CIFAR-10↔ImageNet）可能因为架构搜索的源数据集（PTB）相对较小。可迁移性问题或可通过直接在目标任务上优化架构来规避。

4 Conclusion

4 结论

We presented DARTS, a simple yet efficient architecture search algorithm for both convolutional and recurrent networks. By searching in a continuous space, DARTS is able to match or outperform the state-of-the-art non-differentiable architecture search methods on image classification and language modeling tasks with remarkable efficiency improvement by several orders of magnitude.

我们提出了 DARTS：一种简单而高效、同时面向卷积与循环网络的架构搜索算法。通过在连续空间中搜索，DARTS 能在图像分类与语言建模上匹配或超过当时最优的不可微方法，并以数个数量级的幅度提升效率。

There are many interesting directions to improve DARTS further. For example, the current method may suffer from discrepancies between the continuous architecture encoding and the derived discrete architecture. This could be alleviated, e.g., by annealing the softmax temperature (with a suitable schedule) to enforce one-hot selection. It would also be interesting to investigate performance-aware architecture derivation schemes based on the shared parameters learned during the search process.

还有许多有趣方向可继续改进 DARTS。例如，当前方法可能受连续架构编码与导出离散架构之间差异的影响；可通过（合适日程下）退火 softmax 温度以强制 one-hot 选择来缓解。基于搜索阶段学到的共享参数、设计性能感知的架构导出方案，也很有意思。

Acknowledgements

致谢

The authors would like to thank Zihang Dai, Hieu Pham and Zico Kolter for useful discussions.

作者感谢 Zihang Dai、Hieu Pham 与 Zico Kolter 的有益讨论。

References

参考文献

Ahmed & Torresani (2017) Karim Ahmed and Lorenzo Torresani. Connectivity learning in multi-branch networks. arXiv preprint arXiv:1709.09582 , 2017.

Anandalingam & Friesz (1992) G Anandalingam and TL Friesz. Hierarchical optimization: An introduction. Annals of Operations Research , 34(1):1–11, 1992.

Baker et al. (2017) Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. ICLR , 2017.

Baker et al. (2018) Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. ICLR Workshop , 2018.

Bender et al. (2018) Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understanding and simplifying one-shot architecture search. In International Conference on Machine Learning , pp. 549–558, 2018.

Brock et al. (2018) Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. ICLR , 2018.

Cai et al. (2018) Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. AAAI , 2018.

Colson et al. (2007) Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research , 153(1):235–256, 2007.

DeVries & Taylor (2017) Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 , 2017.

Elsken et al. (2017) Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. arXiv preprint arXiv:1711.04528 , 2017.

Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , pp. 1126–1135, 2017.

Franceschi et al. (2018) Luca Franceschi, Paolo Frasconi, Saverio Salzo, and Massimilano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. ICML , 2018.

Gal & Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In NIPS , pp. 1019–1027, 2016.

Grave et al. (2016) Edouard Grave, Armand Joulin, and Nicolas Usunier. Improving neural language models with a continuous cache. arXiv preprint arXiv:1612.04426 , 2016.

Howard et al. (2017) Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Huang et al. (2017) Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. In CVPR , 2017.

Inan et al. (2017) Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. ICLR , 2017.

Ioffe & Szegedy (2015) Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015.

Kandasamy et al. (2018) Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric Xing. Neural architecture search with bayesian optimisation and optimal transport. NIPS , 2018.

Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Krause et al. (2017) Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. arXiv preprint arXiv:1709.07432 , 2017.

Liu et al. (2018a) Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. ECCV , 2018a.

Liu et al. (2018b) Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. ICLR , 2018b.

Loshchilov & Hutter (2016) Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Luketina et al. (2016) Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient-based tuning of continuous regularization hyperparameters. In ICML , pp. 2952–2960, 2016.

Maclaurin et al. (2015) Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In ICML , pp. 2113–2122, 2015.

Melis et al. (2018) Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. ICLR , 2018.

Merity et al. (2018) Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm language models. ICLR , 2018.

Metz et al. (2017) Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. ICLR , 2017.

Negrinho & Gordon (2017) Renato Negrinho and Geoff Gordon. Deeparchitect: Automatically designing and training deep architectures. arXiv preprint arXiv:1704.08792 , 2017.

Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W , 2017.

Pedregosa (2016) Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In ICML , 2016.

Pham et al. (2018a) Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Authors’ implementation of “Efficient Neural Architecture Search via Parameter Sharing”. https://github.com/melodyguan/enas/tree/2734eb2657847f090e1bc5c51c2b9cbf0be51887 , 2018a. Accessed: 2018-04-05.

Pham et al. (2018b) Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. ICML , 2018b.

Polyak & Juditsky (1992) Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization , 30(4):838–855, 1992.

Real et al. (2018) Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548 , 2018.

Saxena & Verbeek (2016) Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. In NIPS , pp. 4053–4061, 2016.

Shin et al. (2018) Richard Shin, Charles Packer, and Dawn Song. Differentiable neural network architecture search. In ICLR Workshop , 2018.

Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich, Jen-Hao Rick Chang, et al. Going deeper with convolutions. In CVPR , 2015.

Veniat & Denoyer (2017) Tom Veniat and Ludovic Denoyer. Learning time/memory-efficient deep architectures with budgeted super networks. arXiv preprint arXiv:1706.00046 , 2017.

Yang et al. (2018) Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: a high-rank rnn language model. ICLR , 2018.

Zhang et al. (2017) Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083 , 2017.

Zhong et al. (2018) Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 2423–2432, 2018.

Zilly et al. (2016) Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. arXiv preprint arXiv:1607.03474 , 2016.

Zoph & Le (2017) Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. ICLR , 2017.

Zoph et al. (2018) Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. CVPR , 2018.

Appendix A Experimental Details

附录 A 实验细节

A.1 Architecture Search

A.1 架构搜索

A.1.1 CIFAR-10

Since the architecture will be varying throughout the search process, we always use batch-specific statistics for batch normalization rather than the global moving average. Learnable affine parameters in all batch normalizations are disabled during the search process to avoid rescaling the outputs of the candidate operations.

A.1.1 CIFAR-10

由于搜索过程中架构不断变化，批归一化始终使用批次统计而非全局滑动平均。搜索阶段关闭所有批归一中的可学习仿射参数，以免重缩放候选运算输出。

To carry out architecture search, we hold out half of the CIFAR-10 training data as the validation set. A small network of 8 cells is trained using DARTS for 50 epochs, with batch size $64$ (for both the training and validation sets) and the initial number of channels $16$. The numbers were chosen to ensure the network can fit into a single GPU. We use momentum SGD to optimize the weights $w$, with initial learning rate $\eta_{w}=0.025$ (annealed down to zero following a cosine schedule without restart (Loshchilov & Hutter 2016)), momentum $0.9$, and weight decay $3\times 10^{-4}$. We use zero initialization for architecture variables (the $\alpha$’s in both the normal and reduction cells), which implies equal amount of attention (after taking the softmax) over all possible ops. At the early stage this ensures weights in every candidate op to receive sufficient learning signal (more exploration). We use Adam (Kingma & Ba 2014) as the optimizer for $\alpha$, with initial learning rate $\eta_{\alpha}=3\times 10^{-4}$, momentum $\beta=(0.5,0.999)$ and weight decay $10^{-3}$. The search takes one day on a single GPU[^3].

架构搜索时，把 CIFAR-10 训练数据一半留作验证集。用 DARTS 训练含 8 个 cell 的小网络共 50 个 epoch，批量 64（训练与验证皆然），初始通道数 16——这些数字保证网络能装进单卡。权重 $w$ 用动量 SGD 优化，初始学习率 $\eta_{w}=0.025$（按无重启余弦退火到 0），动量 0.9，权重衰减 $3\times10^{-4}$。架构变量（普通与归约 cell 中的 $\alpha$）零初始化，意味着 softmax 后对所有运算同等关注，早期保证每个候选运算都能收到足够学习信号（更多探索）。$\alpha$ 用 Adam 优化，初始学习率 $\eta_{\alpha}=3\times10^{-4}$，动量 $\beta=(0.5,0.999)$，权重衰减 $10^{-3}$。单卡搜索约一天[^3]。

A.1.2 PTB

For architecture search, both the embedding and the hidden sizes are set to 300. The linear transformation parameters across all incoming operations connected to the same node are shared (their shapes are all 300 $\times$ 300), as the algorithm always has the option to focus on one of the predecessors and mask away the others. Tying the weights leads to memory savings and faster computation, allowing us to train the continuous architecture using a single GPU. Learnable affine parameters in batch normalizations are disabled, as we did for convolutional cells. The network is then trained for 50 epochs using SGD without momentum, with learning rate $\eta_{w}=20$, batch size 256, BPTT length 35, and weight decay $5\times 10^{-7}$. We apply variational dropout (Gal & Ghahramani 2016) of $0.2$ to word embeddings, $0.75$ to the cell input, and $0.25$ to all the hidden nodes. A dropout of $0.75$ is also applied to the output layer. Other training settings are identical to those in Merity et al. 2018; Yang et al. 2018. Similarly to the convolutional architectures, we use Adam for the optimization of $\alpha$ (initialized as zeros), with initial learning rate $\eta_{\alpha}=3\times 10^{-3}$, momentum $\beta=(0.9,0.999)$ and weight decay $10^{-3}$. The search takes 6 hours on a single GPU.

A.1.2 PTB

搜索时嵌入与隐层大小均为 300。连到同一节点的所有入运算共享线性变换参数（形状皆为 $300\times300$），因为算法总可以选择聚焦某一前驱并屏蔽其余。绑权节省显存并加速计算，使连续架构可在单卡上训练。批归一的可学习仿射参数关闭，与卷积 cell 相同。网络用无动量 SGD 训 50 个 epoch，学习率 $\eta_{w}=20$，批量 256，BPTT 长度 35，权重衰减 $5\times10^{-7}$。对词嵌入 / cell 输入 / 全部隐节点分别施加 0.2 / 0.75 / 0.25 的变分 dropout，输出层再加 0.75 dropout。其余训练设置同 Merity 等 2018；Yang 等 2018。$\alpha$（零初始化）用 Adam，初始学习率 $\eta_{\alpha}=3\times10^{-3}$，动量 $\beta=(0.9,0.999)$，权重衰减 $10^{-3}$。单卡搜索约 6 小时。

A.2 Architecture Evaluation

A.2 架构评估

A.2.1 CIFAR-10

A large network of 20 cells is trained for 600 epochs with batch size 96. The initial number of channels is increased from 16 to 36 to ensure our model size is comparable with other baselines in the literature (around 3M). Other hyperparameters remain the same as the ones used for architecture search. Following existing works (Pham et al. 2018b; Zoph et al. 2018; Liu et al. 2018a; Real et al. 2018), additional enhancements include cutout (DeVries & Taylor 2017), path dropout of probability $0.2$ and auxiliary towers with weight $0.4$. The training takes 1.5 days on a single GPU with our implementation in PyTorch (Paszke et al. 2017). Since the CIFAR results are subject to high variance even with exactly the same setup (Liu et al. 2018b), we report the mean and standard deviation of 10 independent runs for our full model.

A.2.1 CIFAR-10

含 20 个 cell 的大网络训 600 个 epoch，批量 96。初始通道从 16 增到 36，使模型大小与文献基线（约 3M）可比。其余超参与搜索阶段相同。沿用已有工作，额外增强包括 cutout、概率 0.2 的路径 dropout，以及权重 0.4 的辅助塔。我们的 PyTorch 实现单卡约 1.5 天。因即便设置完全相同 CIFAR 结果方差仍大，完整模型报告 10 次独立运行的均值与标准差。

To avoid any discrepancy between different implementations or training settings (e.g. the batch sizes), we incorporated the NASNet-A cell (Zoph et al. 2018) and the AmoebaNet-A cell (Real et al. 2018) into our training framework and reported their results under the same settings as our cells.

为避免不同实现或训练设置（如批量）的差异，我们把 NASNet-A 与 AmoebaNet-A cell 纳入同一训练框架，并在与我们 cell 相同的设置下报告结果。

A.2.2 PTB

A single-layer recurrent network with the discovered cell is trained until convergence with batch size 64 using averaged SGD (Polyak & Juditsky 1992) (ASGD), with learning rate $\eta_{w}=20$ and weight decay $8\times 10^{-7}$. To speedup, we start with SGD and trigger ASGD using the same protocol as in Yang et al. 2018; Merity et al. 2018. Both the embedding and the hidden sizes are set to 850 to ensure our model size is comparable with other baselines. The token-wise dropout on the embedding layer is set to 0.1. Other hyperparameters remain exactly the same as those for architecture search. For fair comparison, we do not finetune our model at the end of the optimization, nor do we use any additional enhancements such as dynamic evaluation (Krause et al. 2017) or continuous cache (Grave et al. 2016). The training takes 3 days on a single 1080Ti GPU with our PyTorch implementation. To account for implementation discrepancies, we also incorporated the ENAS cell (Pham et al. 2018b) into our codebase and trained their network under the same setup as our discovered cells.

A.2.2 PTB

用发现的 cell 构成单层循环网络，批量 64，用平均 SGD（ASGD）训至收敛，学习率 $\eta_{w}=20$，权重衰减 $8\times10^{-7}$。为加速，先用 SGD，再按 Yang 等 2018；Merity 等 2018 的协议切换到 ASGD。嵌入与隐层大小取 850 以与基线可比。嵌入层 token-wise dropout 为 0.1；其余超参与搜索阶段完全相同。为公平对比，优化结束不做微调，也不用动态评估或 continuous cache 等额外增强。我们的 PyTorch 实现在单块 1080Ti 上约 3 天。为计入实现差异，也把 ENAS cell 纳入代码库并在相同设置下训练。

A.2.3 ImageNet

We consider the mobile setting where the input image size is 224$\times$224 and the number of multiply-add operations in the model is restricted to be less than 600M.

A.2.3 ImageNet

我们考虑 mobile 设置：输入 224×224，模型乘加次数限制在 600M 以下。

A network of 14 cells is trained for 250 epochs with batch size 128, weight decay $3\times 10^{-5}$ and initial SGD learning rate 0.1 (decayed by a factor of 0.97 after each epoch). Other hyperparameters follow Zoph et al. 2018; Real et al. 2018; Liu et al. 2018a[^4]. The training takes 12 days on a single GPU.

含 14 个 cell 的网络训 250 个 epoch，批量 128，权重衰减 $3\times10^{-5}$，初始 SGD 学习率 0.1（每 epoch 乘 0.97）。其余超参沿用 Zoph 等 2018；Real 等 2018；Liu 等 2018a[^4]。单卡训练约 12 天。

A.2.4 WikiText-2

We use embedding and hidden sizes 700, weight decay $5\times 10^{-7}$, and hidden-node variational dropout 0.15. Other hyperparameters remain the same as in our PTB experiments.

A.2.4 WikiText-2

嵌入与隐层大小 700，权重衰减 $5\times10^{-7}$，隐节点变分 dropout 0.15。其余超参与 PTB 实验相同。

Appendix B Search with Increased Depth

附录 B 加深搜索

To better understand the effect of depth for architecture search, we conducted architecture search on CIFAR-10 by increasing the number of cells in the stack from 8 to 20. The initial number of channels is reduced from 16 to 6 due to memory budget of a single GPU. All the other hyperparameters remain the same. The search cost doubles and the resulting cell achieves 2.88 $\pm$ 0.09% test error, which is slightly worse than 2.76 $\pm$ 0.09% obtained using a shallower network. This particular setup may have suffered from the enlarged discrepancy of the number of channels between architecture search and final evaluation. Moreover, searching with a deeper model might require different hyperparameters due to the increased number of layers to back-prop through.

为更好理解深度对架构搜索的影响，我们在 CIFAR-10 上把堆叠 cell 数从 8 增到 20。因单卡显存限制，初始通道从 16 降到 6，其余超参不变。搜索代价翻倍，所得 cell 测试错误 $2.88\pm0.09\%$，略差于用更浅网络得到的 $2.76\pm0.09\%$。该设置可能受搜索与最终评估之间通道数差异放大的影响；用更深模型搜索也可能因反向传播层数增加而需要不同超参。

Appendix C Complexity Analysis

附录 C 复杂度分析

In this section, we analyze the complexity of our search space for convolutional cells.

本节分析卷积 cell 搜索空间的复杂度。

Each of our discretized cell allows $\prod_{k=1}^{4}\frac{(k+1)k}{2}\times(7^{2})\approx 10^{9}$ possible DAGs without considering graph isomorphism (recall we have 7 non-zero ops, 2 input nodes, 4 intermediate nodes with 2 predecessors each). Since we are jointly learning both normal and reduction cells, the total number of architectures is approximately $(10^{9})^{2}=10^{18}$. This is greater than the  $5.6\times 10^{14}$ of PNAS (Liu et al. 2018a) which learns only a single type of cell.

不计图同构，每个离散化 cell 允许约 $\prod_{k=1}^{4}\frac{(k+1)k}{2}\times(7^{2})\approx10^{9}$ 种可能 DAG（7 种非零运算、2 个输入节点、4 个中间节点且各有 2 个前驱）。由于联合学习普通与归约 cell，总架构数约 $(10^{9})^{2}=10^{18}$，大于只学单一类型 cell 的 PNAS 的约 $5.6\times10^{14}$。

Also note that we retained the top-2 predecessors per node only at the very end, and our continuous search space before this final discretization step is even larger. Specifically, each relaxed cell (a fully connected graph) contains $2+3+4+5=14$ learnable edges, allowing $(7+1)^{14}\approx 4\times 10^{12}$ possible configurations ($+1$ to include the zero op indicating a lack of connection). Again, since we are learning both normal and reduction cells, the total number of architectures covered by the continuous space before discretization is $(4\times 10^{12})^{2}\approx 10^{25}$.

另需注意：我们仅在最后保留每节点 top-2 前驱，此前的连续搜索空间更大。具体地，每个松弛 cell（全连接图）含 $2+3+4+5=14$ 条可学习边，允许 $(7+1)^{14}\approx4\times10^{12}$ 种配置（+1 含表示无连接的 zero）。同时学习普通与归约 cell 时，离散化前连续空间覆盖的架构总数约 $(4\times10^{12})^{2}\approx10^{25}$。