End-to-End Training of Deep Visuomotor Policies

深度视觉运动策略的端到端训练（End-to-End Training of Deep Visuomotor Policies）

Sergey Levine svlevine@eecs.berkeley.edu    Chelsea Finn cbfinn@eecs.berkeley.eduTrevor Darrell trevor@eecs.berkeley.eduPieter Abbeel pabbeel@eecs.berkeley.edu    Division of Computer Science University of California Berkeley, CA 94720-1776, USA These authors contributed equally.

Sergey Levine（svlevine@eecs.berkeley.edu）、Chelsea Finn（cbfinn@eecs.berkeley.edu）、Trevor Darrell（trevor@eecs.berkeley.edu）、Pieter Abbeel（pabbeel@eecs.berkeley.edu）
加州大学伯克利分校计算机科学系，Berkeley, CA 94720-1776, USA
†这些作者贡献相同。

Abstract

摘要

Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot’s motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods.

策略搜索方法可以让机器人为广泛任务学习控制策略，但实际应用中策略搜索往往仍需要为感知、状态估计与底层控制手写组件。本文旨在回答：把感知与控制系统联合端到端训练，是否比分开训练各组件表现更好？为此，我们提出一种方法，可学习把原始图像观测直接映射到机器人电机力矩的策略。策略由约 92,000 参数的深度卷积神经网络（CNN）表示，并用引导策略搜索训练——该方法把策略搜索转化为监督学习，监督来自简单的轨迹中心强化学习。我们在一系列需要视觉与控制紧密配合的真实世界操作任务上评估（例如把瓶盖旋到瓶子上），并给出与多种先前策略搜索方法的仿真比较。

1 Introduction

1 引言

keywords

关键词

Robots can perform impressive tasks under human control, including surgery (Lanfranco et al. 2004) and household chores (Wyrobek et al. 2008). However, designing the perception and control software for autonomous operation remains a major challenge, even for basic tasks. Policy search methods hold the promise of allowing robots to automatically learn new behaviors through experience (Kober et al. 2010b; Deisenroth et al. 2011; Kalakrishnan et al. 2011; Deisenroth et al. 2013). However, policies learned using such methods often rely on a number of hand-engineered components for perception and control, so as to present the policy with a more manageable and low-dimensional representation of observations and actions. The vision system in particular can be complex and prone to errors, and it is typically not improved during policy training, nor adapted to the goal of the task.

机器人可以在人类控制下完成令人印象深刻的任务，包括手术（Lanfranco 等，2004）与家务（Wyrobek 等，2008）。然而，即便对基本任务，为自主运行设计感知与控制软件仍是重大挑战。策略搜索方法有望让机器人通过经验自动学习新行为（Kober 等，2010b；Deisenroth 等，2011；Kalakrishnan 等，2011；Deisenroth 等，2013）。但用这些方法学到的策略往往仍依赖若干手写的感知与控制组件，以便向策略提供更易处理、更低维的观测与动作表示。视觉系统尤其复杂且易出错，在策略训练期间通常既不改进，也不针对任务目标适配。

Figure 1: Our method learns visuomotor policies that directly use camera image observations (left) to set motor torques on a PR2 robot (right).

图 1：我们的方法学习视觉运动策略，直接用相机图像观测（左）在 PR2 机器人（右）上设定电机力矩。

In this article, we aim to answer the following question: can we acquire more effective policies for sensorimotor control if the perception system is trained jointly with the control policy, rather than separately? In order to represent a policy that performs both perception and control, we use deep neural networks. Deep neural network representations have recently seen widespread success in a variety of domains, such as computer vision and speech recognition, and even playing video games. However, using deep neural networks for real-world sensorimotor policies, such as robotic controllers that map image pixels and joint angles to motor torques, presents a number of unique challenges. Successful applications of deep neural networks typically rely on large amounts of data and direct supervision of the output, neither of which is available in robotic control. Real-world robot interaction data is scarce, and task completion is defined at a high level by means of a cost function, which means that the learning algorithm must determine on its own which action to take at each point. From the control perspective, a further complication is that observations from the robot’s sensors do not provide us with the full state of the system. Instead, important state information, such as the positions of task-relevant objects, must be inferred from inputs such as camera images.

本文旨在回答：若感知系统与控制策略联合训练而非分开训练，能否获得更有效的感觉运动控制策略？为表示同时做感知与控制的策略，我们使用深度神经网络。深度网络近来在计算机视觉、语音识别乃至电子游戏等领域取得广泛成功。然而，把深度网络用于真实世界的感觉运动策略——例如把图像像素与关节角映射到电机力矩的机器人控制器——带来若干独特挑战。深度网络的成功应用通常依赖大量数据与对输出的直接监督，这两者在机器人控制中都不具备。真实机器人交互数据稀缺，任务完成由高层代价函数定义，学习算法须自行决定每一步动作。从控制角度看，另一困难是观测可能无法提供系统完整状态的全部信息。

We address these challenges by developing a guided policy search algorithm for sensorimotor deep learning, as well as a novel CNN architecture designed for robotic control. Guided policy search converts policy search into supervised learning, by iteratively constructing the training data using an efficient model-free trajectory optimization procedure. We show that this can be formalized as an instance of Bregman ADMM (BADMM) (Wang and Banerjee 2014), which can be used to show that the algorithm converges to a locally optimal solution. In our method, the full state of the system is observable at training time, but not at test time. For most tasks, providing the full state simply requires positioning objects in one of several known positions for each trial during training. At test time, the learned CNN policy can handle novel, unknown configurations, and no longer requires full state information. Since the policy is optimized with supervised learning, we can use standard methods like stochastic gradient descent for training. Our CNNs have 92,000 parameters and 7 layers, including a novel spatial feature point transformation that provides accurate spatial reasoning and reduces overfitting. This allows us to train our policies with relatively modest amounts of data and only tens of minutes of real-world interaction time.

我们通过为感觉运动深度学习开发引导策略搜索算法，以及面向机器人控制的新型 CNN 架构来应对这些挑战。引导策略搜索把策略搜索转化为监督学习：用高效的免模型轨迹优化迭代构造训练数据。我们证明这可形式化为 Bregman ADMM（BADMM）（Wang and Banerjee，2014）的实例，并据此说明算法收敛到局部最优。在我们的方法中，系统完整状态在训练时可知，测试时不可知。对多数任务，提供完整状态只需目标物体的位姿，可用第二只机械臂在训练时持有目标物体来获得。

We evaluate our method by learning policies for inserting a block into a shape sorting cube, screwing a cap onto a bottle, fitting the claw of a toy hammer under a nail with various grasps, and placing a coat hanger on a rack with a PR2 robot (see Figure 1). These tasks require localization, visual tracking, and handling complex contact dynamics. Our results demonstrate improvements in consistency and generalization from training visuomotor policies end-to-end, when compared to training the vision and control components separately. We also present simulated comparisons that show that guided policy search outperforms a number of prior methods when training high-dimensional neural network policies. Some of the material in this article has previously appeared in two conference papers (Levine and Abbeel 2014; Levine et al. 2015), which we extend to introduce visual input into the policy.

我们通过学习以下策略来评估方法：把积木插入形状分类盒、把瓶盖旋到瓶子上、以多种抓取把玩具锤的爪对准钉子下方，以及用 PR2 把衣架挂到架子上（见图 1）。这些任务需要定位、视觉跟踪并处理复杂接触动力学。结果表明，与分别训练视觉与控制组件相比，端到端训练视觉运动策略在一致性与泛化上有提升。我们也给出仿真比较，显示在训练高维神经网络策略时，引导策略搜索优于若干先前方法。

2 Related Work

2 相关工作

Reinforcement learning and policy search methods (Gullapalli 1990; Williams 1992) have been applied in robotics for playing games such as table tennis (Kober et al. 2010b), object manipulation (Gullapalli 1995; Peters and Schaal 2008; Kober et al. 2010a; Deisenroth et al. 2011; Kalakrishnan et al. 2011), locomotion (Benbrahim and Franklin 1997; Kohl and Stone 2004; Tedrake et al. 2004; Geng et al. 2006; Endo et al. 2008), and flight (Ng et al. 2004). Several recent papers provide surveys of policy search in robotics (Deisenroth et al. 2013; Kober et al. 2013). Such methods are typically applied to one component of the robot control pipeline, which often sits on top of a hand-designed controller, such as a PD controller, and accepts processed input, for example from an existing vision pipeline (Kalakrishnan et al. 2011). Our method learns policies that map visual input and joint encoder readings directly to the torques at the robot’s joints. By learning the entire mapping from perception to control, the perception layers can be adapted to optimize task performance, and the motor control layers can be adapted to imperfect perception.

强化学习与策略搜索方法（Gullapalli，1990；Williams，1992）已广泛用于机器人控制（Peters 与 Schaal，2008；Kober 等，2013；Deisenroth 等，2013）。许多最成功的方法仍依赖手写特征或低维状态表示。尽管如此，已有若干工作探索从视觉等原始感觉输入学习策略。早期工作常在仿真中学习，或依赖预先计算的视觉特征。近期深度强化学习在游戏与仿真中取得显著成功，但样本复杂度通常使真实机器人上的直接应用变得困难。

We represent our policies with convolutional neural networks (CNNs). CNNs have a long history in computer vision and deep learning (Fukushima 1980; LeCun et al. 1989; Schmidhuber 2015), and have recently gained prominence due to excellent results on a number of vision benchmarks (Ciresan et al. 2011; Krizhevsky et al. 2012; Ciresan et al. 2012; Girshick et al. 2014a; Tompson et al. 2014; LeCun et al. 2015; He et al. 2015). Most applications of CNNs focus on classification, where locational information is discarded by means of successive pooling layers to provide for invariance (Lee et al. 2009). Applications to localization typically either use a sliding window (Girshick et al. 2014a) or object proposals (Endres and Hoiem 2010; Uijlings et al. 2013; Girshick et al. 2014b) to localize the object, reducing the task to classification, perform regression to a heatmap of manually labeled keypoints (Tompson et al. 2014), requiring precise knowledge of the object position in the image and camera calibration, or use 3D models to localize previously scanned objects (Pepik et al. 2012; Savarese and Fei-Fei 2007). Many prior robotic applications of CNNs do not directly consider control, but employ CNNs for the perception component of a larger robotic system (Hadsell et al. 2009; Sung et al. 2015; Lenz et al. 2015b; Pinto and Gupta 2015). We use a novel CNN architecture for our policies that automatically learn feature points that capture spatial information about the scene, without any supervision beyond the information from the robot’s encoders and camera.

我们用卷积神经网络（CNN）表示策略。CNN 在计算机视觉中有悠久历史，并构成许多当前最佳视觉系统的骨干。将 CNN 用于控制提出独特要求：策略必须产生精确的空间动作，而非仅类别标签，且必须从相对较少的真实机器人数据中学习。

Applications of deep learning in robotic control have been less prevalent in recent years than in visual recognition. Backpropagation through the dynamics and the image formation process is typically impractical, since they are often non-differentiable, and such long-range backpropagation can lead to extreme numerical instability, since the linearization of a suboptimal policy is likely to be unstable. This issue has also been observed in the related context of recurrent neural networks (Hochreiter et al. 2001; Pascanu and Bengio 2012). The high dimensionality of the network also makes reinforcement learning difficult (Deisenroth et al. 2013). Pioneering early work on neural network control used small, simple networks (Pomerleau 1989; Hunt et al. 1992; Bekey and Goldberg 1992; Lewis et al. 1998; Bakker et al. 2003; Mayer et al. 2006), and has largely been supplanted by methods that use carefully designed policies that can be learned efficiently with reinforcement learning (Kober et al. 2013). More recent work on sensorimotor deep learning has tackled simple task-space motions (Lenz et al. 2015a; Lampe and Riedmiller 2013) and used unsupervised learning to obtain low-dimensional state spaces from images (Lange et al. 2012). Such methods have been demonstrated on tasks with a low-dimensional underlying structure: Lenz et al. 2015a controls the end-effector in 2D space, while Lange et al. 2012 controls a 2-dimensional slot car with 1-dimensional actions. Our experiments include full torque control of 7-DoF robotic arms interacting with objects, with 30-40 state dimensions. In simple synthetic environments, control from images has been addressed with image features (Jodogne and Piater 2007), nonparametric methods (van Hoof et al. 2015), and unsupervised state-space learning (Böhmer et al. 2013; Jonschkowski and Brock 2014). CNNs have also been trained to play video games with Q-learning, Monte Carlo tree search, and stochastic search (Mnih et al. 2013; Koutník et al. 2013; Guo et al. 2014), and have been applied to simple simulated control tasks (Watter et al. 2015; Lillicrap et al. 2015). However, such methods have only been demonstrated on synthetic domains that lack the visual complexity of the real world, and require an impractical number of samples for real-world robotic learning. Our method is sample efficient, requiring only minutes of interaction time. To the best of our knowledge, this is the first method that can train deep visuomotor policies for complex, high-dimensional manipulation skills with direct torque control.

深度学习在机器人控制中的应用近年来相对较少，尽管早期神经控制器工作可追溯到几十年前。若干近期方法在仿真中学习视觉运动策略，或用人类示范。与这些方法相比，我们的算法直接在真实机器人上从力矩级控制学习，并用引导策略搜索降低样本复杂度。

Learning visuomotor policies on a real robot requires handling complex observations and high dimensional policy representations. We tackle these challenges using guided policy search. In guided policy search, the policy is optimized using supervised learning, which scales gracefully with the dimensionality of the policy. The training set for supervised learning can be constructed using trajectory optimization under known dynamics (Levine and Koltun 2013a; Levine and Koltun 2013b; Levine and Koltun 2014; Mordatch and Todorov 2014) and trajectory-centric reinforcement learning methods that operate under unknown dynamics (Levine and Abbeel 2014; Levine et al. 2015), which is the approach taken in this work. In both cases, the supervision is adapted to the policy, to ensure that the final policy can reproduce the training data. The use of supervised learning in the inner loop of iterative policy search has also been proposed in the context of imitation learning (Ross et al. 2011; Ross et al. 2013). However, such methods typically do not address the question of how the supervision should be adapted to the policy.

在真实机器人上学习视觉运动策略需要处理复杂观测与高维动作。若干工作结合视觉伺服与学习，或用位姿估计作为中间表示。我们的目标是去掉对手写中间表示的依赖，联合训练感知与控制。

The goal of our approach is also similar to visual servoing, which performs feedback control on feature points in a camera image (Espiau et al. 1992; Mohta et al. 2014; Wilson et al. 1996). However, our visuomotor policies are entirely learned from real-world data, and do not require feature points or feedback controllers to be specified by hand. This allows our method much more flexibility in choosing how to use the visual signal. Our approach also does not require any sort of camera calibration, in contrast to many visual servoing methods (though not all – see e.g. Jägersand et al. 1997; Yoshimi and Allen 1994).

我们的目标也与视觉伺服相近，后者根据图像特征误差做反馈控制。经典视觉伺服通常需要手选特征与已知几何。学习型方法可减轻部分负担，但端到端从像素到力矩仍少见。我们的工作表明，借助合适的策略搜索与网络架构，这种端到端训练在真实硬件上可行。

3 Background and Overview

3 背景与概述

In this section, we define the visuomotor policy learning problem and present an overview of our approach. The core component of our approach is a guided policy search algorithm that separates the problem of learning visuomotor policies into separate supervised learning and trajectory learning phases, each of which is easier than optimizing the policy directly. We also discuss a policy architecture suitable for end-to-end learning of vision and control, and a training setup that allows our method to be applied to real robotic platforms.

本节定义视觉运动策略学习问题，并概述我们的方法。后续各节给出算法细节、网络架构与实验结果。

3.1 Definitions and Problem Formulation

3.1 定义与问题形式化

In policy search, the goal is to learn a policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ that allows an agent to choose actions $\mathbf{u}_{t}$ in response to observations $\mathbf{o}_{t}$ to control a dynamical system, such as a robot. The policy comes from some parametric class parameterized by $\theta$, which could be, for example, the weights of a neural network. The system is defined by states $\mathbf{x}_{t}$, actions $\mathbf{u}_{t}$, and observations $\mathbf{o}_{t}$. For example, $\mathbf{x}_{t}$ might include the joint angles of the robot, the positions of objects in the world, and their time derivatives, $\mathbf{u}_{t}$ might consist of motor torque commands, and $\mathbf{o}_{t}$ might include an image from the robot’s onboard camera. In this paper, we address finite horizon episodic tasks with $t\in[1,\dots,T]$. The states evolve in time according to the system dynamics $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$, and the observations are, in general, a stochastic consequence of the states, according to $p(\mathbf{o}_{t}|\mathbf{x}_{t})$. Neither the dynamics nor the observation distribution are assumed to be known in general. For notational convenience, we will use $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$ to denote the distribution over actions under the policy conditioned on the state. However, since the policy is conditioned on the observation $\mathbf{o}_{t}$, this distribution is in fact given by $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})=\int\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})p(\mathbf{o}_{t}|\mathbf{x}_{t})d\mathbf{o}_{t}$. The dynamics and $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$ together induce a distribution over trajectories $\tau=\{\mathbf{x}_{1},\mathbf{u}_{1},\mathbf{x}_{2},\mathbf{u}_{2},\dots,\mathbf{x}_{T},\mathbf{u}_{T}\}$:

在策略搜索中，目标是学习策略 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$，它根据观测 $\mathbf{o}_{t}$ 选择动作 $\mathbf{u}_{t}$。系统动力学为 $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$，观测模型为 $p(\mathbf{o}_{t}|\mathbf{x}_{t})$。轨迹分布可写成

$$\pi_{\theta}(\tau)=p(\mathbf{x}_{1})\prod_{t=1}^{T}\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t}).$$

The goal of a task is given by a cost function $\ell(\mathbf{x}_{t},\mathbf{u}_{t})$, and the objective in policy search is to minimize the expectation $E_{\pi_{\theta}(\tau)}[\sum_{t=1}^{T}\ell(\mathbf{x}_{t},\mathbf{u}_{t})]$, which we will abbreviate as $E_{\pi_{\theta}(\tau)}[\ell(\tau)]$. A summary of the notation used in the paper is provided in Table 1.

任务目标由代价函数 $\ell(\mathbf{x}_{t},\mathbf{u}_{t})$ 给出，策略搜索最小化期望总代价。在我们的设定中，训练时可获得完整状态 $\mathbf{x}_{t}$（含目标物体位姿），而测试时策略只接收观测 $\mathbf{o}_{t}$（图像与本体感觉）。

symbol	definition	example/details
$\mathbf{x}_{t}$	Markovian system state at time step $t\in[1,T]$	joint angles, end-effector pose, object positions, and their velocities; dimensionality: 14 to 32
$\mathbf{u}_{t}$	control or action at time step $t\in[1,T]$	joint motor torque commands; dimensionality: 7 (for the PR2 robot)
$\mathbf{o}_{t}$	observation at time step $t\in[1,T]$	RGB camera image, joint encoder readings & velocities, end-effector pose; dimensionality: around 200,000
$\tau$	trajectory: $\tau=\{\mathbf{x}_{1},\mathbf{u}_{1},\mathbf{x}_{2},\mathbf{u}_{2},\dots,\mathbf{x}_{T},\mathbf{u}_{T}\}$	notational shorthand for a sequence of states and actions
$\ell(\mathbf{x}_{t},\mathbf{x}_{t})$	cost function that defines the goal of the task	distance between an object in the gripper and the target
$p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$	unknown system dynamics	physics that govern the robot and any objects it interacts with
$p(\mathbf{o}_{t}|\mathbf{x}_{t})$	unknown observation distribution	stochastic process that produces camera images from system state
$\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$	learned nonlinear global policy parameterized by weights $\theta$	convolutional neural network, such as the one in Figure 2
$\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$	$\int\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})p(\mathbf{o}_{t}|\mathbf{x}_{t})d\mathbf{o}_{t}$	notational shorthand for observation-based policy conditioned on state
$p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$	learned local time-varying linear-Gaussian controller for initial state $\mathbf{x}_{1}^{i}$	time-varying linear-Gaussian controller has form $\mathcal{N}(\mathbf{K}_{ti}\mathbf{x}_{t}+\mathbf{k}_{ti},\mathbf{C}_{ti})$
$\pi_{\theta}(\tau)$	trajectory distribution for $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$: $p(\mathbf{x}_{1})\prod_{t=1}^{T}\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$	notational shorthand for trajectory distribution induced by a policy

Table 1: Summary of the notation frequently used in this article.

符号	定义	示例/细节
$\mathbf{x}_{t}$	马尔可夫状态	关节角、速度、末端执行器位姿、目标物体位姿
$\mathbf{u}_{t}$	动作	关节力矩
$\mathbf{o}_{t}$	观测	RGB 图像与关节配置
$\pi_{\theta}$	策略	将观测映射到力矩的 CNN
$\ell$	代价	到目标点的距离、力矩惩罚等

3.2 Approach Summary

3.2 方法概要

Our methods consists of two main components, which are illustrated in Figure 3. The first is a supervised learning algorithm that trains policies of the form $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})=\mathcal{N}(\mu^{\pi}(\mathbf{o}_{t}),\Sigma^{\pi}(\mathbf{o}_{t}))$, where both $\mu^{\pi}(\mathbf{o}_{t})$ and $\Sigma^{\pi}(\mathbf{o}_{t})$ are general nonlinear functions. In our implementation, $\mu^{\pi}(\mathbf{o}_{t})$ is a deep convolutional neural network, while $\Sigma^{\pi}(\mathbf{o}_{t})$ is an observation-independent learned covariance, though other representations are possible. The second component is a trajectory-centric reinforcement learning (RL) algorithm that generates guiding distributions $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ that provide the supervision used to train the policy. These two components form a policy search algorithm that can be used to learn complex robotic tasks using only a high-level cost function $\ell(\mathbf{x}_{t},\mathbf{u}_{t})$. During training, only samples from the guiding distributions $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ are generated by running rollouts on the physical system, which avoids the need to execute partially trained neural network policies on physical hardware.

我们的方法由两个主要组件组成，如图 3 所示。第一是监督学习算法，训练形式为 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})=\mathcal{N}(\mu^{\pi}(\mathbf{o}_{t}),\Sigma^{\pi}(\mathbf{o}_{t}))$ 的策略，其中 $\mu^{\pi}(\mathbf{o}_{t})$ 与 $\Sigma^{\pi}(\mathbf{o}_{t})$ 均为一般非线性函数。实现中 $\mu^{\pi}(\mathbf{o}_{t})$ 是深度卷积网络，$\Sigma^{\pi}(\mathbf{o}_{t})$ 是与观测无关的可学习协方差，当然也可采用其他表示。第二是轨迹中心的强化学习（RL）算法，生成导引分布，为训练策略提供监督。两组件共同构成策略搜索算法，仅用高层代价函数即可学习复杂机器人任务。训练期间只在物理系统上执行来自导引分布的样本，从而避免在硬件上运行部分训练的神经网络策略。

Supervised learning will not, in general, produce a policy with good long-horizon performance, since a small mistake on the part of the policy will place the system into states that are outside the distribution in the training data, causing compounding errors. To avoid this issue, the training data must come from the policy’s own state distribution (Ross et al. 2011). We achieve this by alternating between trajectory-centric RL and supervised learning. The RL stage adapts to the current policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$, providing supervision at states that are iteratively brought closer to the states visited by the policy. This is formalized as a variant of the BADMM algorithm (Wang and Banerjee 2014) for constrained optimization, which can be used to show that, at convergence, the policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ and the guiding distributions $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ will exhibit the same behavior. This algorithm is derived in Section 4. The guiding distributions are substantially easier to optimize than learning the policy parameters directly (e.g., using model-free reinforcement learning), because they use the full state of the system $\mathbf{x}_{t}$, while the policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ only uses the observations. This means that the method requires the full state to be known during training, but not at test time. This makes it possible to efficiently learn complex visuomotor policies, but imposes additional assumptions on the observability of $\mathbf{x}_{t}$ during training that we discuss in Section 4.

一般而言，监督学习不会产生具有良好长时程表现的策略，因为训练分布未必覆盖策略实际访问的状态。引导策略搜索通过交替优化导引分布与策略来解决这一点，并约束二者保持接近，从而把策略搜索变成带约束的监督学习问题。

Figure 2: Visuomotor policy architecture. The network contains three convolutional layers, followed by a spatial softmax and an expected position layer that converts pixel-wise features to feature points, which are better suited for spatial computations. The points are concatenated with the robot configuration, then passed through three fully connected layers to produce the torques.

图 2：视觉运动策略架构。网络含三个卷积层，后接空间 Softmax 与期望算子以产生特征点，再与机器人配置一并送入全连接层以输出力矩。

When learning visuomotor tasks, the policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ is represented by a novel convolutional neural network (CNN) architecture, which we describe in Section 5.2. CNNs have enjoyed considerable success in computer vision (LeCun et al. 2015), but the most popular architectures rely on large datasets and focus on semantic tasks such as classification, often intentionally discarding spatial information. Our architecture, illustrated in Figure 2, uses a fixed transformation from the last convolutional layer to a set of spatial feature points, which form a concise representation of the visual scene suitable for feedback control. Our network has 7 layers and around 92,000 parameters, which presents a major challenge for standard policy search methods (Deisenroth et al. 2013).

学习视觉运动任务时，策略 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ 必须从图像推断任务相关信息（如目标位置），同时生成稳定控制。这要求感知与控制表示兼容，并在有限真实数据下泛化。

Figure 3: Diagram of our approach, including the main guided policy search phase and initialization phases.

图 3：方法示意图，包括主引导策略搜索阶段与用于加快收敛的初始化。

To reduce the amount of experience needed to train visuomotor policies, we also introduce a pretraining scheme that allows us to train effective policies with a relatively small number of iterations. The pretraining steps are illustrated in Figure 3. The intuition behind our pretraining is that, although we ultimately seek to obtain sensorimotor policies that combine both vision and control, low-level aspects of vision can be initialized independently. To that end, we pretrain the convolutional layers of our network by predicting elements of $\mathbf{x}_{t}$ that are not provided in the observation $\mathbf{o}_{t}$, such as the positions of objects in the scene. We also initially train the guiding trajectory distributions $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ independently of the convolutional network until the trajectories achieve a basic level of competence at the task, and then switch to full guided policy search with end-to-end training of $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$. In our implementation, we also initialize the first layer filters from the model of Szegedy et al. 2014, which is trained on ImageNet (Deng et al. 2009) classification. The initialization and pretraining scheme is described in Section 5.2.

为减少训练视觉运动策略所需的经验量，我们还引入位姿预测预训练与完全观测轨迹预训练，为端到端阶段提供良好初始化。

4 Guided Policy Search with BADMM

4 基于 BADMM 的引导策略搜索

Guided policy search transforms policy search into a supervised learning problem, where the training set is generated by a simple trajectory-centric RL algorithm. This algorithm optimizes linear-Gaussian controllers $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$, and is described in Section 4.2. We refer to the trajectory distribution induced by $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ as $p_{i}(\tau)$. Each $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ succeeds from different initial states. For example, in the task of placing a cap on a bottle, these initial states correspond to different positions of the bottle. By training on trajectories for multiple bottle positions, the final CNN policy can succeed from all initial states, and can generalize to other states from the same distribution.

引导策略搜索把策略搜索转化为监督学习问题，训练集由简单的轨迹中心 RL 算法生成。该算法优化线性-高斯控制器 $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$，见第 4.2 节。我们把由 $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 诱导的轨迹分布记为 $p_{i}(\tau)$。每个 $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 从不同初始状态取得成功。例如在把瓶盖放到瓶子上的任务中，这些初始状态对应瓶子的不同位置。通过在多个瓶子位置的轨迹上训练，最终 CNN 策略可从所有初始状态成功，并泛化到同分布的其他状态。

The final policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ learned with guided policy search is only provided with observations $\mathbf{o}_{t}$ of the full state $\mathbf{x}_{t}$, and the dynamics are assumed to be unknown. A diagram of this method, which corresponds to an expanded version of the guided policy search box in Figure 3, is shown on the right. In the outer loop, we draw sample trajectories $\{\tau_{i}^{j}\}$ for each initial state on the physical system by running the corresponding controller $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$. The samples are used to fit the dynamics $p_{i}(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ that are used to improve $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$, and serve as training data for the policy. The inner loop alternates between optimizing each $p_{i}(\tau)$ and optimizing the policy to match these trajectory distributions. The policy is trained to predict the actions along each trajectory from the observations $\mathbf{o}_{t}$, rather than the full state $\mathbf{x}_{t}$. This allows the policy to directly use raw observations at test time. This alternating optimization can be framed as an instance of the BADMM algorithm (Wang and Banerjee 2014), which converges to a solution where the trajectory distributions and the policy have the same state distribution. This allows greedy supervised training of the policy to produce a policy with good long-horizon performance.

用引导策略搜索学到的最终策略 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ 只获得完整状态 $\mathbf{x}_{t}$ 的观测 $\mathbf{o}_{t}$，并假定动力学未知。该方法示意图（图 3 中引导策略搜索框的展开）见右。外层循环中，在物理系统上运行对应控制器 $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$，为每个初始状态采集样本轨迹 $\{\tau_{i}^{j}\}$。样本用于拟合改进 $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 所需的动力学 $p_{i}(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$，并作为策略的训练数据。内层循环交替优化每个 $p_{i}(\tau)$ 与使策略匹配这些轨迹分布。策略被训练成根据观测 $\mathbf{o}_{t}$ 而非完整状态 $\mathbf{x}_{t}$ 预测各轨迹上的动作，从而使策略在测试时能直接使用原始观测。该交替优化可形式化为 BADMM 算法的实例。

4.1 Algorithm Derivation

4.1 算法推导

Policy search methods minimize the expected cost $E_{\pi_{\theta}}[\ell(\tau)]$, where $\tau=\{\mathbf{x}_{1},\mathbf{u}_{1},\dots,\mathbf{x}_{T},\mathbf{u}_{T}\}$ is a trajectory, and $\ell(\tau)=\sum_{t=1}^{T}\ell(\mathbf{x}_{t},\mathbf{u}_{t})$ is the cost of an episode. In the fully observed case, the expectation is taken under $\pi_{\theta}(\tau)=p(\mathbf{x}_{1})\prod_{t=1}^{T}\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$. The final policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ is conditioned on the observations $\mathbf{o}_{t}$, but $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$ can be recovered as $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})=\int\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})p(\mathbf{o}_{t}|\mathbf{x}_{t})d\mathbf{o}_{t}$. We will present the derivation in this section for $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$, but we do not require knowledge of $p(\mathbf{o}_{t}|\mathbf{x}_{t})$ in the final algorithm. As discussed in Section 4.3, the integral will be evaluated with samples from the real system, which include both $\mathbf{x}_{t}$ and $\mathbf{o}_{t}$. We begin by rewriting the expected cost minimization as a constrained problem:

策略搜索方法最小化期望代价 $E_{\pi_{\theta}}[\ell(\tau)]$，其中 $\tau=\{\mathbf{x}_{1},\mathbf{u}_{1},\dots,\mathbf{x}_{T},\mathbf{u}_{T}\}$ 为轨迹，$\ell(\tau)=\sum_{t=1}^{T}\ell(\mathbf{x}_{t},\mathbf{u}_{t})$ 为回合代价。在完全观测情形下，期望在 $\pi_{\theta}(\tau)=p(\mathbf{x}_{1})\prod_{t=1}^{T}\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ 下计算。最终策略 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ 条件于观测 $\mathbf{o}_{t}$，但可把 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 恢复为 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})=\int\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})p(\mathbf{o}_{t}|\mathbf{x}_{t})d\mathbf{o}_{t}$。本节推导针对 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$，但最终算法不需要知道 $p(\mathbf{o}_{t}|\mathbf{x}_{t})$。如第 4.3 节所述，该积分用真实系统样本评估，样本同时包含 $\mathbf{x}_{t}$ 与 $\mathbf{o}_{t}$。我们首先把期望代价最小化改写为带约束的形式。

$$\min_{p,\pi_{\theta}}E_{p}[\ell(\tau)]\text{ s.t. }p(\mathbf{u}_{t}|\mathbf{x}_{t})=\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})\,\,\forall\,\mathbf{x}_{t},\mathbf{u}_{t},t, \tag{1}$$

where we will refer to $p(\tau)$ as a guiding distribution. This formulation is equivalent to the original problem, since the constraint forces the two distributions to be identical. However, if we approximate the initial state distribution $p(\mathbf{x}_{1})$ with samples $\mathbf{x}_{1}^{i}$, we can choose $p(\tau)$ to be a class of distributions that is much easier to optimize than $\pi_{\theta}$, as we will show later. This will allow us to use simple local learning methods for $p(\tau)$, without needing to train the complex neural network policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ directly with reinforcement learning, which would require a prohibitive amount of experience on real physical systems.

其中我们把 $p(\tau)$ 称为导引分布。该形式与原问题等价，因为约束迫使两分布相同。然而，若用样本 $\mathbf{x}_{1}^{i}$ 近似初始状态分布 $p(\mathbf{x}_{1})$，可以选择比 $\pi_{\theta}$ 容易优化得多的分布类作为 $p(\tau)$，后文将说明。这将允许我们对 $p(\tau)$ 使用简单的局部学习方法，而无需直接用强化学习训练复杂的神经网络策略 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$——后者在真实物理系统上需要禁止数量的经验。

The constrained problem can be solved by a dual descent method, which alternates between minimizing the Lagrangian with respect to the primal variables, and incrementing the Lagrange multipliers by their subgradient. Minimization of the Lagrangian with respect to $p(\tau)$ and $\theta$ is done in alternating fashion: minimizing with respect to $\theta$ corresponds to supervised learning (making $\pi_{\theta}$ match $p(\tau)$), and minimizing with respect to $p(\tau)$ consists of one or more trajectory optimization problems. The dual descent method we use is based on BADMM (Wang and Banerjee 2014), a variant of ADMM (Boyd et al. 2011) that augments the Lagrangian with a Bregman divergence between the constrained variables. We use the KL-divergence as the Bregman constraint, which is particularly convenient for working with probability distributions. We will also modify the constraint $p(\mathbf{u}_{t}|\mathbf{x}_{t})=\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$ by multiplying both sides by $p(\mathbf{x}_{t})$, to get $p(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})=\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})$. This constraint is equivalent, but has the convenient property that we can express the Lagrangian in terms of expectations. The BADMM augmented Lagrangians for $\theta$ and $p$ are therefore given by

约束问题可用对偶下降法求解：交替关于原变量最小化拉格朗日函数，并按次梯度递增拉格朗日乘子。关于 $p(\tau)$ 与 $\theta$ 的拉格朗日最小化交替进行：对 $\theta$ 最小化对应监督学习（使 $\pi_{\theta}$ 匹配 $p(\tau)$），对 $p(\tau)$ 最小化由一个或多个轨迹优化问题组成。我们使用的对偶下降基于 BADMM（Wang and Banerjee，2014），它是 ADMM（Boyd 等，2011）的变体，用约束变量之间的 Bregman 散度增强拉格朗日。我们用 KL 散度作为 Bregman 约束，这对处理概率分布特别方便。我们还将约束 $p(\mathbf{u}_{t}|\mathbf{x}_{t})=\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 两边乘以 $p(\mathbf{x}_{t})$，得到 $p(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})=\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})$。该约束等价，但在表示上更方便。

$$\displaystyle\mathcal{L}_{\theta}(\theta,p) \\ \displaystyle=\sum_{t=1}^{T}E_{p(\mathbf{x}_{t},\mathbf{u}_{t})}[\ell(\mathbf{x}_{t},\mathbf{u}_{t})]+E_{p(\mathbf{x}_{t})\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})}[\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}]-E_{p(\mathbf{x}_{t},\mathbf{u}_{t})}[\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}]+\nu_{t}\phi^{\theta}_{t}(\theta,p) \\ \displaystyle\mathcal{L}_{p}(p,\theta) \\ \displaystyle=\sum_{t=1}^{T}E_{p(\mathbf{x}_{t},\mathbf{u}_{t})}[\ell(\mathbf{x}_{t},\mathbf{u}_{t})]+E_{p(\mathbf{x}_{t})\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})}[\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}]-E_{p(\mathbf{x}_{t},\mathbf{u}_{t})}[\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}]+\nu_{t}\phi^{p}_{t}(\theta,p),$$

where $\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}$ is the Lagrange multiplier for state $\mathbf{x}_{t}$ and action $\mathbf{u}_{t}$ at time $t$, and $\phi^{\theta}_{t}(\theta,p)$ are $\phi^{p}_{t}(\theta,p)$ are expectations of the KL-divergences:

其中 $\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}$ 是时刻 $t$ 状态 $\mathbf{x}_{t}$ 与动作 $\mathbf{u}_{t}$ 的拉格朗日乘子，$\phi^{\theta}_{t}(\theta,p)$ 与 $\phi^{p}_{t}(\theta,p)$ 是 KL 散度的期望：

$$\displaystyle\phi^{p}_{t}(p,\theta) \\ \displaystyle=E_{p(\mathbf{x}_{t})}[D_{\text{KL}}(p(\mathbf{u}_{t}|\mathbf{x}_{t})\|\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t}))] \\ \displaystyle\phi^{\theta}_{t}(\theta,p) \\ \displaystyle=E_{p(\mathbf{x}_{t})}[D_{\text{KL}}(\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t}))\|p(\mathbf{u}_{t}|\mathbf{x}_{t})].$$

Dual descent with alternating primal minimization is then described by the following steps:

带交替原问题最小化的对偶下降由以下步骤描述：

$$\displaystyle\theta \\ \displaystyle\leftarrow\arg\min_{\theta}\sum_{t=1}^{T}E_{p(\mathbf{x}_{t})\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})}[\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}]+\nu_{t}\phi^{\theta}_{t}(\theta,p) \\ \displaystyle p \\ \displaystyle\leftarrow\arg\min_{p}\sum_{t=1}^{T}E_{p(\mathbf{x}_{t},\mathbf{u}_{t})}[\ell(\mathbf{x}_{t},\mathbf{u}_{t})-\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}]+\nu_{t}\phi^{p}_{t}(p,\theta) \\ \displaystyle\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}} \\ \displaystyle\leftarrow\lambda_{\mathbf{x}_{t},\mathbf{u}_{t}}+\alpha\nu_{t}(\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})-p(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})).$$

This procedure is an instance of BADMM, and therefore inherits its convergence guarantees. Note that we drop terms that are independent of the optimization variables on each line. The parameter $\alpha$ is a step size. As with most augmented Lagrangian methods, the weight $\nu_{t}$ is set heuristically, as described in Appendix A.1.

该流程是 BADMM 的实例，因而继承其收敛保证。注意每行我们丢掉与优化变量无关的项。参数 $\alpha$ 是步长。与多数增强拉格朗日方法一样，权重 $\nu_{t}$ 启发式设定，见附录 A.1。

The dynamics only affect the optimization with respect to $p(\tau)$. In order to make this optimization efficient, we choose $p(\tau)$ to be a mixture of $N$ Gaussians $p_{i}(\tau)$, one for each initial state sample $\mathbf{x}_{1}^{i}$. This makes the action conditionals $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ and the dynamics $p_{i}(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ linear-Gaussian, as discussed in Section 4.2. This is a reasonable choice when the system is deterministic, or the noise is Gaussian or small, and we found that this approach is sufficiently tolerant to noise for use on real physical systems. Our choice of $p$ also assumes that the policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ is conditionally Gaussian. This is also reasonable, since the mean and covariance of $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ can be any nonlinear function of the observations $\mathbf{o}_{t}$, which themselves are a function of the unobserved state $\mathbf{x}_{t}$. In Section 4.2, we show how these assumptions enable each $p_{i}(\tau)$ to be optimized very efficiently. We will refer to $p_{i}(\tau)$ as guiding distributions, since they serve to focus the policy on good, low-cost behaviors.

动力学只影响关于 $p(\tau)$ 的优化。为使该优化高效，我们选择 $p(\tau)$ 为 $N$ 个高斯 $p_{i}(\tau)$ 的混合，每个对应一个初始状态样本 $\mathbf{x}_{1}^{i}$。这使得动作条件 $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 与动力学 $p_{i}(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ 为线性-高斯，见第 4.2 节。当系统确定性、或噪声为高斯或较小时，这是合理选择；我们发现该方法对噪声的容忍足以用于真实物理系统。对 $p$ 的选择也假定策略 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ 为条件高斯。这也合理，因为其均值与协方差可以是观测 $\mathbf{o}_{t}$ 的任意非线性函数，而观测本身又是未观测状态 $\mathbf{x}_{t}$ 的函数。第 4.2 节说明这些假定如何使每个 $p_{i}(\tau)$ 能被非常高效地优化。我们称 $p_{i}(\tau)$ 为导引分布，因为它们把策略聚焦到良好行为上。

Aside from learning $p_{i}(\tau)$, we must choose a tractable way to represent the infinite set of constraints $p(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})=\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})$. One approximate approach proposed in prior work is to replace the exact constraints with expectations of features (Peters et al. 2010). When the features consist of linear, quadratic, or higher order monomial functions of the random variable, this can be viewed as a constraint on the moments of the distributions. If we only use the first moment, we get a constraint on the expected action: $E_{p(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})}[\mathbf{u}_{t}]=E_{\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})}[\mathbf{u}_{t}]$. If the stochasticity in the dynamics is low, as we assumed previously, the optimal solution for each $p_{i}(\tau)$ will have low entropy, making this first moment constraint a reasonable approximation. The KL-divergence terms in the augmented Lagrangians will still serve to softly enforce agreement between the higher moments. While this simplification is quite drastic, we found that it was more stable in practice than including higher moments, likely because these higher moments are harder to estimate accurately with a limited number of samples. The alternating optimization is now given by

除学习 $p_{i}(\tau)$ 外，还必须选择可处理的方式表示无穷约束集 $p(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})=\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})$。先前工作提出的一种近似是用特征期望代替精确约束（Peters 等，2010）。当特征由随机变量的线性、二次或更高阶单项式组成时，可视为对分布矩的约束。若只用一阶矩，得到期望动作约束：$E_{p(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})}[\mathbf{u}_{t}]=E_{\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})}[\mathbf{u}_{t}]$。若动力学随机性较低（如前假定），每个 $p_{i}(\tau)$ 的最优解熵较低，使该一阶矩约束成为合理近似。增强拉格朗日中的 KL 项仍会软性地强制更高阶矩一致。尽管这一简化相当剧烈，我们发现它在实践中比包含二阶矩更稳定。

$$\displaystyle\theta \\ \displaystyle\leftarrow\arg\min_{\theta}\sum_{t=1}^{T}E_{p(\mathbf{x}_{t})\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})}[\mathbf{u}_{t}^{\mathrm{T}}\lambda_{\mu t}]+\nu_{t}\phi^{\theta}_{t}(\theta,p) \\ \displaystyle p \\ \displaystyle\leftarrow\arg\min_{p}\sum_{t=1}^{T}E_{p(\mathbf{x}_{t},\mathbf{u}_{t})}[\ell(\mathbf{x}_{t},\mathbf{u}_{t})-\mathbf{u}_{t}^{\mathrm{T}}\lambda_{\mu t}]+\nu_{t}\phi^{p}_{t}(p,\theta) \\ \displaystyle\lambda_{\mu t} \\ \displaystyle\leftarrow\lambda_{\mu t}+\alpha\nu_{t}(E_{\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})}[\mathbf{u}_{t}]-E_{p(\mathbf{u}_{t}|\mathbf{x}_{t})p(\mathbf{x}_{t})}[\mathbf{u}_{t}]), \tag{2}$$

where $\lambda_{\mu t}$ is the Lagrange multiplier on the expected action at time $t$. In the rest of the paper, we will use $\mathcal{L}_{\theta}(\theta,p)$ and $\mathcal{L}_{p}(p,\theta)$ to denote the two augmented Lagrangians in Equations (2) and (3), respectively. In the next two sections, we will describe how $\mathcal{L}_{p}(p,\theta)$ can be optimized with respect to $p$ under unknown dynamics, and how $\mathcal{L}_{\theta}(\theta,p)$ can be optimized for complex, high-dimensional policies. Implementation details of the BADMM optimization are presented in Appendix A.1.

其中 $\lambda_{\mu t}$ 是时刻 $t$ 期望动作上的拉格朗日乘子。后文用 $\mathcal{L}_{\theta}(\theta,p)$ 与 $\mathcal{L}_{p}(p,\theta)$ 分别表示式 (2) 与 (3) 中的两个增强拉格朗日。下面两节说明如何在未知动力学下关于 $p$ 优化 $\mathcal{L}_{p}(p,\theta)$，以及如何对复杂高维策略优化 $\mathcal{L}_{\theta}(\theta,p)$。BADMM 优化的实现细节见附录 A.1。

4.2 Trajectory Optimization under Unknown Dynamics

4.2 未知动力学下的轨迹优化

Since the Lagrangian $\mathcal{L}_{p}(p,\theta)$ in the previous section factorizes over the mixture elements in $p(\tau)=\sum_{i}p_{i}(\tau)$, we describe the trajectory optimization method for a single Gaussian $p(\tau)$. When there are multiple mixture elements, this procedure is applied in parallel to each $p_{i}(\tau)$. Since $p(\tau)$ is Gaussian, the conditionals $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ and $p(\mathbf{u}_{t}|\mathbf{x}_{t})$, which correspond to the dynamics and the controller, are time-varying linear-Gaussian, and given by

由于上一节拉格朗日 $\mathcal{L}_{p}(p,\theta)$ 在混合 $p(\tau)=\sum_{i}p_{i}(\tau)$ 的各成分上因子分解，我们描述单个高斯 $p(\tau)$ 的轨迹优化方法。有多个混合成分时，该流程对每个 $p_{i}(\tau)$ 并行应用。由于 $p(\tau)$ 为高斯，条件分布 $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ 与 $p(\mathbf{u}_{t}|\mathbf{x}_{t})$（对应动力学与控制器）为时变线性-高斯，并由下式给出

$$p(\mathbf{u}_{t}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{K}_{t}\mathbf{x}_{t}+\mathbf{k}_{t},\mathbf{C}_{t})\hskip 32.52127ptp(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})=\mathcal{N}(f_{\mathbf{x}t}\mathbf{x}_{t}+f_{\mathbf{u}t}\mathbf{u}_{t}+f_{ct},\mathbf{F}_{t}).$$

This type of controller can be learned efficiently with a small number of real-world samples, making it a good choice for optimizing the guiding distributions. Since a different set of time-varying linear-Gaussian dynamics is fitted for each initial state, this dynamics representation can model any continuous deterministic system that can be locally linearized. Stochastic dynamics can violate the local linearity assumption in principle, but we found that in practice this representation was well suited for a wide variety of noisy real-world tasks.

这类控制器可用少量真实世界样本高效学习，适合优化导引分布。由于为每个初始状态拟合不同的时变线性-高斯动力学，该动力学表示可建模任何可局部线性化的连续确定性系统。随机动力学原则上可能违反局部线性假定，但实践中我们发现该表示适合多种带噪声的真实世界任务。

The dynamics are determined by the environment. If they are known, $p(\mathbf{u}_{t}|\mathbf{x}_{t})$ can be optimized with a variant of the iterative linear-quadratic-Gaussian regulator (iLQG) (Li and Todorov 2004; Levine and Koltun 2013a), which is a variant of DDP (Jacobson and Mayne 1970). In the case of unknown dynamics, we can fit $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ to sample trajectories sampled from the trajectory distribution at the previous iteration, denoted $\hat{p}(\tau)$. If $\hat{p}(\tau)$ is too different from $p(\tau)$, these samples will not give a good estimate of $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$, and the optimization will diverge. To avoid this, we can bound the change from $\hat{p}(\tau)$ to $p(\tau)$ in terms of their KL-divergence by a step size $\epsilon$, producing the following constrained problem:

动力学由环境决定。若已知，可用迭代线性二次高斯调节器（iLQG）的变体（Li and Todorov，2004；Levine and Koltun，2013a）优化 $p(\mathbf{u}_{t}|\mathbf{x}_{t})$，iLQG 是 DDP（Jacobson and Mayne，1970）的变体。在未知动力学时，可用上一迭代轨迹分布（记为 $\hat{p}(\tau)$）采得的样本轨迹拟合 $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$。若 $\hat{p}(\tau)$ 与 $p(\tau)$ 相差过大，这些样本不能很好估计动力学，优化会发散。为避免这一点，可用 KL 散度约束从 $\hat{p}(\tau)$ 到 $p(\tau)$ 的变化。

$$\min_{p(\tau)\in\mathcal{N}(\tau)}\mathcal{L}_{p}(p,\theta)\text{ s.t. }D_{\text{KL}}(p(\tau)\|\hat{p}(\tau))\leq\epsilon.$$

This type of policy update has previously been proposed by several authors in the context of policy search (Bagnell and Schneider 2003; Peters and Schaal 2008; Peters et al. 2010; Levine and Abbeel 2014). In the case when $p(\tau)$ is Gaussian, this problem can be solved efficiently using dual gradient descent, while the dynamics $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ are fitted to samples gathered by running the previous controller $\hat{p}(\mathbf{u}_{t}|\mathbf{x}_{t})$ on the robot. Fitting a global Gaussian mixture model to tuples $(\mathbf{x}_{t},\mathbf{u}_{t},\mathbf{x}_{t+1})$ and using it as a prior for fitting the dynamics $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ serves to greatly reduce the sample complexity. We describe the dynamics fitting procedure in detail in Appendix A.3.

这类策略更新此前已有若干作者在策略搜索语境中提出（Bagnell and Schneider，2003；Peters and Schaal，2008；Peters 等，2010；Levine and Abbeel，2014）。当 $p(\tau)$ 为高斯时，可用对偶梯度下降高效求解；动力学 $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$ 拟合到在机器人上运行上一控制器 $\hat{p}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 得到的样本。对元组 $(\mathbf{x}_{t},\mathbf{u}_{t},\mathbf{x}_{t+1})$ 拟合全局高斯混合模型并用作拟合动力学的先验，可大幅降低样本复杂度。动力学拟合流程见附录。

Note that the trajectory optimization cost function $\mathcal{L}_{p}(p,\theta)$ also depends on the policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$, while we only have access to $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$. In order to compute a local quadratic expansion of the KL-divergence term $D_{\text{KL}}(p(\mathbf{u}_{t}|\mathbf{x}_{t})\|\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t}))$ inside $\mathcal{L}_{p}(p,\theta)$ for iLQG, we also estimate a linearization of the mean of the conditionally Gaussian policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ with respect to the state $\mathbf{x}_{t}$, using the same procedure that we use to linearize the dynamics. The data for this estimation consists of tuples $\{\mathbf{x}_{t}^{i},E_{\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t}^{i})}[\mathbf{u}_{t}]\}$, which we can obtain because both the states $\mathbf{x}_{t}^{i}$ and the observations $\mathbf{o}_{t}^{i}$ are available for all of the samples evaluated on the real physical system.

注意轨迹优化代价 $\mathcal{L}_{p}(p,\theta)$ 也依赖于策略 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$，而我们只有 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$。为对 $\mathcal{L}_{p}$ 内 KL 项 $D_{\text{KL}}(p(\mathbf{u}_{t}|\mathbf{x}_{t})\|\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t}))$ 做局部二次展开以用于 iLQG，我们还估计条件高斯策略均值关于状态 $\mathbf{x}_{t}$ 的线性化，使用与线性化动力学相同的流程。该估计的数据由元组构成，含状态与策略在对应观测下的期望动作。

This constrained optimization is performed in the “inner loop” of the optimization described in the previous section, and the KL-divergence constraint $D_{\text{KL}}(p(\tau)\|\hat{p}(\tau))\leq\epsilon$ imposes a step size on the trajectory update. The overall algorithm then becomes an instance of generalized BADMM (Wang and Banerjee 2014). Note that the augmented Lagrangian $\mathcal{L}_{p}(p,\theta)$ consists of an expectation under $p(\tau)$ of a quantity that is independent of $p$. We can locally approximate this quantity with a quadratic by using a quadratic expansion of $\ell(\mathbf{x}_{t},\mathbf{u}_{t})$, and fitting a linear-Gaussian to $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$ with the same method we used for the dynamics. We can then solve the primal optimization in the dual gradient descent procedure with a standard LQR backward pass. This is significantly simpler and much faster than the forward-backward dynamic programming procedure employed in previous work (Levine and Abbeel 2014; Levine and Koltun 2014). This improvement is enabled by the use of BADMM, which allows us to always formulate the KL-divergence term in the Lagrangian with the distribution being optimized as the first argument. Since the KL-divergence is convex in its first argument, this makes the corresponding optimization significantly easier. The details of this LQR-based dual gradient descent algorithm are derived in Appendix A.4.

该约束优化在上一节所述优化的“内层循环”中执行，KL 约束 $D_{\text{KL}}(p(\tau)\|\hat{p}(\tau))\leq\epsilon$ 对轨迹更新施加步长。整体算法于是成为广义 BADMM 的实例（Wang and Banerjee，2014）。注意增强拉格朗日 $\mathcal{L}_{p}(p,\theta)$ 由 $p(\tau)$ 下某个与 $p$ 无关的量的期望组成。我们可用 $\ell(\mathbf{x}_{t},\mathbf{u}_{t})$ 的二次展开，并用与动力学相同的方法对 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 拟合线性-高斯，从而局部把该量近似为二次，再在对偶下降中求解原问题。

We can further improve the efficiency of the method by allowing samples from multiple trajectories $p_{i}(\tau)$ to be used to fit a shared dynamics $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$, while the controllers $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ are allowed to vary. This makes sense when the initial states of these trajectories are similar, and they therefore visit similar regions. This allows us to draw just a single sample from each $p_{i}(\tau)$ at each iteration, allowing us to handle many more initial states.

还可通过允许来自多条轨迹 $p_{i}(\tau)$ 的样本拟合共享动力学 $p(\mathbf{x}_{t+1}|\mathbf{x}_{t},\mathbf{u}_{t})$，同时允许控制器 $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 不同，进一步提高效率。当这些轨迹初始状态相近因而访问相似区域时，这是合理的。这使我们每次迭代只需从每个 $p_{i}(\tau)$ 采一个样本，从而能处理更多初始状态。

4.3 Supervised Policy Optimization

4.3 有监督的策略优化

Since the policy parameters $\theta$ participate only in the constraints of the optimization problem in Equation (1), optimizing the policy corresponds to minimizing the KL-divergence between the policy and trajectory distribution, as well as the expectation of $\lambda_{\mu t}^{\mathrm{T}}\mathbf{u}_{t}$. For a conditional Gaussian policy of the form $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})=\mathcal{N}(\mu^{\pi}(\mathbf{o}_{t}),\Sigma^{\pi}(\mathbf{o}_{t}))$, the objective is

由于策略参数 $\theta$ 只参与式 (1) 优化问题的约束，优化策略对应于最小化策略与轨迹分布之间的 KL 散度，以及 $\lambda_{\mu t}^{\mathrm{T}}\mathbf{u}_{t}$ 的期望。对形式为 $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})=\mathcal{N}(\mu^{\pi}(\mathbf{o}_{t}),\Sigma^{\pi}(\mathbf{o}_{t}))$ 的条件高斯策略，目标为

$$\displaystyle\mathcal{L}_{\theta}(\theta,p)\!=\!\!\frac{1}{2N}\!\sum_{i=1}^{N}\!\sum_{t=1}^{T}\!E_{p_{i}(\mathbf{x}_{t},\mathbf{o}_{t})}\!\left[\text{tr}[\mathbf{C}_{ti}^{-1}\Sigma^{\pi}(\mathbf{o}_{t})]\!-\!\log|\Sigma^{\pi}(\mathbf{o}_{t})|\right. \\ \displaystyle\left.+(\mu^{\pi}(\mathbf{o}_{t})\!-\!\mu^{p}_{ti}(\mathbf{x}_{t}))\mathbf{C}_{ti}^{-1}(\mu^{\pi}(\mathbf{o}_{t})\!-\!\mu^{p}_{ti}(\mathbf{x}_{t}))+2\lambda_{\mu t}^{\mathrm{T}}\mu^{\pi}(\mathbf{o}_{t})\right],$$

where $\mu^{p}_{ti}(\mathbf{x}_{t})$ is the mean of $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ and $\mathbf{C}_{ti}$ is the covariance, and the expectation is evaluated using samples from each $p_{i}(\tau)$ with corresponding observations $\mathbf{o}_{t}$. The observations are sampled from $p(\mathbf{o}_{t}|\mathbf{x}_{t})$ by recording camera images on the real system. Since the input to $\mu^{\pi}(\mathbf{o}_{t})$ and $\Sigma^{\pi}(\mathbf{o}_{t})$ is not the state $\mathbf{x}_{t}$, but only an observation $\mathbf{o}_{t}$, we can train the policy to directly use raw observations. Note that $\mathcal{L}_{\theta}(\theta,p)$ is simply a weighted quadratic loss on the difference between the policy mean and the mean action of the trajectory distribution, offset by the Lagrange multiplier. The weighting is the precision matrix of the conditional in the trajectory distribution, which is equal to the curvature of its cost-to-go function (Levine and Koltun 2013a). This has an intuitive interpretation: $\mathcal{L}_{\theta}(\theta,p)$ penalizes deviation from the trajectory distribution, with a penalty that is locally proportional to its cost-to-go. At convergence, when the policy $\pi_{\theta}(\mathbf{u}_{t}|\mathbf{o}_{t})$ takes the same actions as $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$, their Q-functions are equal, and the supervised policy objective becomes equivalent to the policy iteration objective (Levine and Koltun 2014)

其中 $\mu^{p}_{ti}(\mathbf{x}_{t})$ 是 $p_{i}(\mathbf{u}_{t}|\mathbf{x}_{t})$ 的均值，$\mathbf{C}_{ti}$ 是协方差，期望用各 $p_{i}(\tau)$ 的样本及对应观测 $\mathbf{o}_{t}$ 评估。观测通过在真实系统上记录相机图像从 $p(\mathbf{o}_{t}|\mathbf{x}_{t})$ 采样。由于 $\mu^{\pi}(\mathbf{o}_{t})$ 与 $\Sigma^{\pi}(\mathbf{o}_{t})$ 的输入不是状态 $\mathbf{x}_{t}$ 而只是观测 $\mathbf{o}_{t}$，我们可以训练策略直接使用原始观测。注意 $\mathcal{L}_{\theta}(\theta,p)$ 本质上是策略均值与轨迹分布均值动作之差（再偏移拉格朗日乘子）上的加权二次损失。

In this work, we optimize $\mathcal{L}_{\theta}(\theta,p)$ with respect to $\theta$ using stochastic gradient descent (SGD), a standard method for neural network training. The covariance of the Gaussian policy does not depend on the observation in our implementation, though adding this dependence would be straightforward. Since training complex neural networks requires a substantial number of samples, we found it beneficial to include sampled observations from previous iterations into the policy optimization, evaluating the action $\mu^{p}_{ti}(\mathbf{x}_{t})$ at their corresponding states using the current trajectory distributions. Since these samples come from the wrong state distribution, we use importance sampling and weight them according to the ratio of their probability under the current distribution $p(\mathbf{x}_{t})$ and the one they were sampled from, which is straightforward to evaluate under the estimated linear-Gaussian dynamics (Levine and Koltun 2013b).

本文用随机梯度下降（SGD）关于 $\theta$ 优化 $\mathcal{L}_{\theta}(\theta,p)$，这是神经网络训练的标准方法。实现中高斯策略的协方差不依赖于观测，但加入该依赖是直接的。由于训练复杂网络需要大量样本，我们发现把先前迭代的采样观测纳入策略优化有益：用当前轨迹分布在其对应状态上评估动作 $\mu^{p}_{ti}(\mathbf{x}_{t})$。由于这些样本来自错误的状态分布，我们使用重要性采样，按其在当前与先前轨迹分布下的概率比加权。

4.4 Comparison with Prior Guided Policy Search Methods

4.4 与先前引导策略搜索方法的比较

We presented a guided policy search method where the policy is trained on observations, while the trajectories are trained on the full state. The BADMM formulation of guided policy search is new to this work, though several prior guided policy search methods based on constrained optimization have been proposed. Levine and Koltun 2014 proposed a formulation similar to Equation (1), but with a constraint on the KL-divergence between $p(\tau)$ and $\pi_{\theta}$. This results in a more complex, non-convex forward-backward trajectory optimization phase. Since the BADMM formulation solves a convex problem during the trajectory optimization phase, it is substantially faster and easier to implement and use, especially when the number of trajectories $p_{i}(\tau)$ is large.

我们提出一种引导策略搜索方法：策略在观测上训练，轨迹在完整状态上训练。引导策略搜索的 BADMM 形式化是本文新内容，尽管已有若干基于约束优化的先前引导策略搜索方法。Levine 与 Koltun（2014）提出类似式 (1) 的形式，但对 $p(\tau)$ 与 $\pi_{\theta}$ 之间的 KL 散度施加约束。这导致更复杂、非凸的前向-后向轨迹优化阶段。由于 BADMM 形式化在轨迹优化阶段求解凸问题，它显著更快、更易实现与使用，尤其当轨迹数 $p_{i}(\tau)$ 很大时。

The use of ADMM for guided policy search was also proposed by Mordatch and Todorov 2014 for deterministic policies under known dynamics. This approach requires known, deterministic dynamics and trains deterministic policies. Furthermore, because this approach uses a simple quadratic augmented Lagrangian term, it further requires penalty terms on the gradient of the policy to account for local feedback. Our approach enforces this feedback behavior due to the higher moments included in the KL-divergence term, but does not require computing the second derivative of the policy.

Mordatch 与 Todorov（2014）也曾为已知动力学下的确定性策略提出用 ADMM 做引导策略搜索。该方法需要已知、确定性动力学并训练确定性策略。此外，因其使用简单二次增强拉格朗日项，还需要在策略梯度上加惩罚项以刻画局部反馈。我们的方法由于 KL 项包含更高阶矩而强制这种反馈行为，但不需要计算策略的二阶导数。

5 End-to-End Visuomotor Policies

5 端到端视觉运动策略

Guided policy search allows us to optimize complex, high-dimensional policies with raw observations, such as when the input to the policy consists of images from a robot’s onboard camera. However, leveraging this capability to directly learn policies for visuomotor control requires designing a policy representation that is both data-efficient and capable of learning complex control strategies directly from raw visual inputs. In this section, we describe a deep convolutional neural network (CNN) model that is uniquely suited to this task. Our approach combines a novel spatial soft-argmax layer with a pretraining procedure that provides for flexibility and data-efficiency.

引导策略搜索使我们能优化带原始观测的复杂高维策略，例如策略输入为机器人机载相机图像时。然而，要利用这一能力直接学习视觉运动控制策略，需要设计既样本高效、又能直接从原始视觉输入学习复杂控制策略的策略表示。本节描述特别适合该任务的深度卷积网络（CNN）模型。我们的方法结合新颖的空间 soft-argmax 层与提供灵活性与数据效率的预训练流程。

5.1 Visuomotor Policy Architecture

5.1 视觉运动策略架构

Our visuomotor policy runs at 20 Hz on the robot, mapping monocular RGB images and the robot configurations to joint torques on a 7 DoF arm. The configuration includes the angles of the joints and the pose of the end-effector (defined by 3 points in the space of the end-effector), as well as their velocities, but does not include the position of the target object or goal, which must be determined from the image. CNNs often use pooling to discard the locational information that is necessary to determine positions, since it is an irrelevant distractor for tasks such as object classification (Lee et al. 2009). Because locational information is important for control, our policy does not use pooling. Additionally, CNNs built for spatial tasks such as human pose estimation often also rely on the availability of location labels in image-space, such as hand-labeled keypoints (Tompson et al. 2014). We propose a novel CNN architecture capable of estimating spatial information from an image without direct supervision in image space. Our pose estimation experiments, discussed in Section 5.2, show that this network can learn useful visual features using only 3D position information provided by the robot, and no camera calibration. Further training the network with guided policy search to directly output motor torques causes it to acquire task-specific visual features. Our experiments in Section 6.4 show that this improves performance beyond the level achieved with features trained only for pose estimation.

我们的视觉运动策略在机器人上以 20 Hz 运行，把单目 RGB 图像与机器人配置映射到 7 自由度臂的关节力矩。配置包括关节角、末端执行器位姿（由末端执行器空间中的 3 个点定义）及其速度，但不包含目标物体或目标位置——后者必须从图像确定。CNN 常使用池化丢弃确定位置所需的定位信息，因为对物体分类等任务而言它是无关干扰（Lee 等，2009）。由于定位信息对控制很重要，我们的策略不使用池化。此外，面向人体姿态估计等空间任务的 CNN 往往还依赖图像空间中的位置标注，如手标关键点（Tompson 等，2014）。我们提出一种新型 CNN 架构，能在没有图像空间直接监督的情况下从图像估计空间信息。第 5.2 节的位姿估计实验表明，该网络仅用机器人运动学提供的 3D 位置信息即可学到有用视觉特征。

Our network architecture is shown in Figure 2. The visual processing layers of the network consist of three convolutional layers, each of which learns a bank of filters that are applied to patches centered on every pixel of its input. These filters form a hierarchy of local image features. Each convolutional layer is followed by a rectifying nonlinearity of the form $a_{cij}=\max(0,z_{cij})$ for each channel $c$ and each pixel coordinate $(i,j)$. The third convolutional layer contains $32$ response maps with resolution $109\times 109$. These response maps are passed through a spatial softmax function of the form $s_{cij}={e^{a_{cij}}}/{\sum_{i^{\prime}j^{\prime}}{e^{a_{ci^{\prime}j^{\prime}}}}}$. Each output channel of the softmax is a probability distribution over the location of a feature in the image. To convert from this distribution to a coordinate representation $(f_{cx},f_{cy})$, the network calculates the expected image position of each feature, yielding a 2D coordinate for each channel: $f_{cx}=\sum_{ij}s_{cij}x_{ij}$ and $f_{cy}=\sum_{ij}s_{cij}y_{ij}$, where $(x_{ij},y_{ij})$ is the image-space position of the point $(i,j)$ in the response map. Since this is a linear operation, it corresponds to a fixed, sparse fully connected layer with weights $W_{cix}=x_{ij}$ and $W_{cjy}=y_{ij}$. The combination of the spatial softmax and expectation operator implement a kind of soft-argmax. The spatial feature points $(f_{cx},f_{cy})$ are concatenated with the robot’s configuration and fed into two fully connected layers, each with 40 rectified units, followed by linear connections to the torques. The full network contains about 92,000 parameters, of which 86,000 are in the convolutional layers.

网络架构见图 2。视觉处理层由三个卷积层组成，每层学习一组在每个像素为中心的图像块上应用的滤波器，形成局部图像特征的层次。每个卷积层后接形式为 $a_{cij}=\max(0,z_{cij})$ 的整流非线性。第三卷积层含 $32$ 张分辨率为 $109\times 109$ 的响应图。这些响应图通过形式为 $s_{cij}={e^{a_{cij}}}/{\sum_{i^{\prime}j^{\prime}}{e^{a_{ci^{\prime}j^{\prime}}}}}$ 的空间 Softmax。Softmax 的每个输出通道是图像中某特征位置的概率分布。为把该分布转为坐标表示 $(f_{cx},f_{cy})$，网络计算每个特征的期望图像位置，从而为每个通道得到二维坐标：$f_{cx}=\sum_{ij}s_{cij}x_{ij}$，$f_{cy}=\sum_{ij}s_{cij}y_{ij}$，其中 $(x_{ij},y_{ij})$ 是响应图中点 $(i,j)$ 的图像空间位置。

The spatial softmax and the expected position computation serve to convert pixel-wise representations in the convolutional layers to spatial coordinate representations, which can be manipulated by the fully connected layers into 3D positions or motor torques. The softmax also provides lateral inhibition, which suppresses low, erroneous activations, only keeping strong activations that are more likely to be accurate. This makes our policy more robust to distractors, providing generalization to novel visual variation. We compare our architecture with more standard alternatives in Section 6.3 and evaluate robustness to visual distractors in Section 6.4. However, the proposed architecture is also in some sense more specialized for visuomotor control, in contrast to more general standard convolutional networks. For example, not all perception tasks require information that can be coherently summarized by a set of spatial locations.

空间 Softmax 与期望位置计算把卷积层中的逐像素表示转换为空间坐标表示，再由全连接层操作成 3D 位置或电机力矩。Softmax 还提供侧抑制，压制低且错误的激活，只保留更可能准确的强激活。这使策略对干扰物更稳健，从而泛化到新的视觉变化。我们在第 6.3 节与更标准替代架构比较，并在第 6.4 节评估对视觉干扰物的稳健性。不过，所提架构在某种意义上也更专门服务于视觉运动控制，而不同于更通用的标准卷积网络。例如，并非所有感知任务都需要能由一组空间位置连贯概括的信息。

5.2 Visuomotor Policy Training

5.2 视觉运动策略训练

The guided policy search trajectory optimization phase uses the full state of the system, though the final policy only uses the observations. This type of instrumented training is a natural choice for many robotics tasks, where the robot is trained under controlled conditions, but must then act intelligently in uncontrolled, real-world situations. In our tasks, the unobserved variables are the pose of a target object (e.g. the bottle on which a cap must be placed). During training, this target object is typically held in the robot’s left gripper, while the robot’s right arm performs the task, as shown to the right. This allows the robot to move the target through a range of known positions. The final visuomotor policy does not receive this position as input, but must instead use the camera images. Due to the modest amount of training data, distractors that are correlated with task-relevant variables can hamper generalization. For this reason, the left arm is covered with cloth to prevent the policy from associating its appearance with the object’s position.

引导策略搜索的轨迹优化阶段使用系统完整状态，而最终策略只用观测。这种“仪器化训练”对许多机器人任务是自然选择：机器人在受控条件下训练，然后须在不受控的真实情境中智能行动。在我们的任务中，未观测变量是目标物体的位姿（例如须放上瓶盖的瓶子）。训练时该目标物体通常由机器人左夹爪持有，右臂执行任务，如右图所示。这使机器人能把目标移过一系列已知位置。最终视觉运动策略不接收该位置作为输入，而必须使用相机图像。由于训练数据量适中，与任务相关变量相关的干扰物会妨碍泛化。因此左臂用布遮盖，防止策略把它的外观与物体位置关联。

While we can train the visuomotor policy entirely from scratch, the algorithm would spend a large number of iterations learning basic visual features and arm motions that can more efficiently be learned by themselves, before being incorporated into the policy. To speed up learning, we initialize both the vision layers in the policy and the trajectory distributions for guided policy search by leveraging the fully observed training setup. To initialize the vision layers, the robot moves the target object through a range of random positions, recording camera images and the object’s pose, which is computed automatically from the pose of the gripper. This dataset is used to train a pose regression CNN, which consists of the same vision layers as the policy, followed by a fully connected layer that outputs the 3D points that define the target. Since the training set is still small (we use 1000 images collected from random arm motions), we initialize the filters in the first layer with weights from the model of Szegedy et al. 2014, which is trained on ImageNet (Deng et al. 2009) classification. After training on pose regression, the weights in the convolutional layers are transferred to the policy CNN. This enables the robot to learn the appearance of the objects prior to learning the behavior.

虽然可以从头训练视觉运动策略，算法会花费大量迭代学习基本视觉特征与手臂运动——这些本可先单独更高效地学会再并入策略。为加速学习，我们利用完全观测的训练设置，同时初始化策略中的视觉层与引导策略搜索的轨迹分布。为初始化视觉层，机器人把目标物体移过一系列随机位置，记录相机图像与物体位姿（由夹爪位姿自动计算）。该数据集用于训练位姿回归 CNN：与策略相同的视觉层，后接输出定义目标的 3D 点的全连接层。由于训练集仍较小（我们用随机手臂运动采集的 1000 张图像），第一层滤波器用 Szegedy 等（2014）在 ImageNet（Deng 等，2009）分类上训练的模型权重初始化。位姿回归训练后，卷积层权重迁移到策略。

To initialize the linear-Gaussian controllers for each of the initial states, we take 15 iterations of guided policy search without optimizing the visuomotor policy. This allows for much faster training in the early iterations, when the trajectories are not yet successful, and optimizing the full visuomotor policy is unnecessarily time consuming. Since we still want the trajectories to arrive at compatible strategies for each target position, we replace the visuomotor policy during these iterations with a small network that receives the full state, which consisted of two layers with 40 rectified linear hidden units in our experiments. This network serves only to constrain the trajectories and avoid divergent behaviors from emerging for similar initial states, which would make subsequent policy learning difficult.

为初始化各初始状态的线性-高斯控制器，我们先做 15 次不优化视觉运动策略的引导策略搜索迭代。这使早期迭代（轨迹尚未成功、优化完整视觉运动策略不必要地耗时时）训练快得多。由于仍希望各目标位置的轨迹到达相容策略，这些迭代中用接收完整状态的小网络替换视觉运动策略；实验中该网络为两层、每层 40 个整流线性隐单元。该网络只用于约束轨迹，避免对相似初始状态出现发散行为，否则后续策略学习会变难。

After initialization, we train the full visuomotor policy with guided policy search. During the supervised policy optimization phase, the fully connected motor control layers are first optimized by themselves, since they are not initialized with pretraining. This can be done very quickly because these layers are small. Then, the entire network is further optimized end-to-end. We found that first training the upper layers before end-to-end optimization prevented the convolutional layers from forgetting useful features learning during pretraining, when the error signal due to the untrained upper layers is very large. The entire pretraining scheme is summarized in the diagram on the right. Note that the trajectories can be pretrained in parallel with the vision layer pretraining, which does not require access to the physical system. Furthermore, the entire initialization procedure does not use any additional information that is not already available from the robot.

初始化后，用引导策略搜索训练完整视觉运动策略。在有监督策略优化阶段，先单独优化全连接电机控制层，因为它们未经预训练初始化；这些层很小，可很快完成。然后整网进一步端到端优化。我们发现先训练上层再端到端优化，可防止卷积层在未训练上层误差信号很大时忘记预训练学到的有用特征。整个预训练方案概括见右图。注意轨迹可与视觉层预训练并行（后者不需要接触物理系统）。此外，整个初始化流程不使用机器人上本不可得的额外信息。

6 Experimental Evaluation

6 实验评估

In this section, we present a series of experiments aimed at evaluating our approach and answering the following questions:

本节给出一系列实验，旨在评估我们的方法并回答下列问题：

1. How does the guided policy search algorithm compare to other policy search methods for training complex, high-dimensional policies, such as neural networks?
2. Does our trajectory optimization algorithm work on a real robotic platform with unknown dynamics, for a range of different tasks?
3. How does our spatial softmax architecture compare to other, more standard convolutional neural network architectures?
4. Does training the perception and control systems in a visuomotor policy jointly end-to-end provide better performance than training each component separately?

1. 引导策略搜索算法与其他策略搜索方法相比，在训练神经网络等复杂高维策略时表现如何？
2. 我们的轨迹优化算法能否在动力学未知的真实机器人平台上，对一系列不同任务奏效？
3. 我们的空间 Softmax 架构与其他更标准的卷积网络架构相比如何？
4. 在视觉运动策略中联合端到端训练感知与控制系统，是否比分开训练各组件表现更好？

Evaluating a wide range of policy search algorithms on a real robot would be extremely time consuming, particularly for methods that require a large number of samples. We therefore answer question (1) by using a physical simulator and simpler policies that do not use vision. This also allows us to test the generality of guided policy search on tasks that include manipulation, walking, and swimming. To answer question (2), we present a wide range of experiments on a PR2 robot. These experiments allow us to evaluate the sample efficiency of our trajectory optimization algorithm. To address question (3), we compare a range of different policy architectures on the task of localizing a target object (the cube in the shape sorting cube task). Since localizing the target object is a prerequisite for completing the shape sorting cube task, this serves as a good proxy for evaluating different architectures. Finally, we answer the last and most important question (4) by training visuomotor policies for hanging a coat hanger on a clothes rack, inserting a block into a shape sorting cube, fitting the claw of a toy hammer under a nail with various grasps, and screwing on a bottle cap. These tasks are illustrated in Figure 8.

在真实机器人上评估大量策略搜索算法会极其耗时，尤其对需要大量样本的方法。因此我们用物理仿真与不使用视觉的较简单策略回答问题 (1)。这也让我们能在包含操作、行走与游泳的任务上测试引导策略搜索的一般性。为回答问题 (2)，我们在 PR2 上给出大量实验，以评估轨迹优化算法的样本效率。为回答问题 (3)，我们在定位目标物体的任务上比较一系列不同策略架构。为回答问题 (4)，我们在完整视觉运动策略上比较端到端训练与分模块训练。

6.1 Simulated Comparisons to Prior Policy Search Methods

6.1 与先前策略搜索方法的仿真比较

In this section, we compare our method against prior policy search techniques on a range of simulated robotic control tasks. These results previously appeared in our conference paper that introduced the trajectory optimization procedure with local linear models (Levine and Abbeel 2014). In these tasks, the state $\mathbf{x}_{t}$ consists of the joint angles and velocities of each robot, and the actions $\mathbf{u}_{t}$ consist of the torques at each joint. The neural network policies used one hidden layer and soft rectifier nonlinearities of the form $a=\log(1+\exp(z))$. Since these policies use the state as input, they only have a few hundred parameters, far fewer than our visuomotor policies. However, even this number of parameters can pose a major challenge for prior policy search methods (Deisenroth et al. 2013).

本节在一系列仿真机器人控制任务上把我们的方法与先前策略搜索技术比较。这些结果此前出现在引入带局部线性模型的轨迹优化流程的会议论文中（Levine and Abbeel，2014）。这些任务中，状态 $\mathbf{x}_{t}$ 由各机器人关节角与速度组成，动作 $\mathbf{u}_{t}$ 为各关节力矩。神经网络策略使用一个隐层与形式为 $a=\log(1+\exp(z))$ 的软整流非线性。由于这些策略以状态为输入，参数只有几百个，远少于我们的视觉运动策略。

Experimental tasks.

实验任务。

We simulated 2D and 3D peg insertion, octopus arm control, and planar swimming and walking. The difficulty in the peg insertion tasks stems from the need to align the peg with the slot and the complex contacts between the peg and the walls, which result in discontinuous dynamics. Octopus arm control involves moving the tip of a flexible arm to a goal position (Engel et al. 2005). The challenge in this task stems from its high dimensionality: the arm has $25$ degrees of freedom, corresponding to $50$ state dimensions. The swimming task requires controlling a three-link snake, and the walking task requires a seven-link biped to maintain a target velocity. The challenge in these tasks comes from underactuation. Details of the simulation and cost for each task are in Appendix B.1.

我们仿真了 2D 与 3D 插钉、章鱼臂控制以及平面游泳与行走。插钉任务的困难来自需要对齐钉与槽，以及钉与壁之间导致动力学不连续的复杂接触。章鱼臂控制要求把柔性臂尖端移到目标位置（Engel 等，2005），挑战来自高维：$25$ 自由度对应 $50$ 维状态。游泳任务控制三连杆蛇形机器人，行走任务要求七连杆双足保持目标速度。

Prior methods.

先前方法。

We compare to REPS (Peters et al. 2010), reward-weighted regression (RWR) (Peters and Schaal 2007; Kober and Peters 2009), the cross-entropy method (CEM) (Rubinstein and Kroese 2004), and PILCO (Deisenroth and Rasmussen 2011). We also use iLQG (Li and Todorov 2004) with a known model as a baseline, shown as a black horizontal line in all plots. REPS is a model-free method that, like our approach, enforces a KL-divergence constraint between the new and old policy. We compare to a variant of REPS that also fits linear dynamics to generate 500 pseudo-samples (Lioutikov et al. 2014), which we label “REPS (20 + 500).” RWR is an EM algorithm that fits the policy to previous samples weighted by the exponential of their reward, and CEM fits the policy to the best samples in each batch. With Gaussian trajectories, CEM and RWR only differ in the weights. These methods represent a class of RL algorithms that fit the policy to weighted samples, including PoWER and PI2 (Kober and Peters 2009; Theodorou et al. 2010; Stulp and Sigaud 2012). PILCO is a model-based method that uses a Gaussian process to learn a global dynamics model that is used to optimize the policy. We used the open-source implementation of PILCO provided by the authors. Both REPS and PILCO require solving large nonlinear optimizations at each iteration, while our method does not. Our method used $5$ rollouts with the Gaussian mixture model prior, and $20$ without. Due to its computational cost, PILCO was provided with $5$ rollouts per iteration, while other prior methods used $20$ and $100$. For all prior methods with free hyperparameters (such as the fraction of elites for CEM), we performed hyperparameter sweeps and chose the most successful settings for the comparison.